Repository: getnikola/nikola

Issue:
Crash on wordpress import
Doing another issue, I get this on importing the user's dump:

```
Traceback (most recent call last):
  File "/home/ralsina/.virtualenvs/nikola/local/lib/python2.7/site-packages/doit-0.28.0-py2.7.egg/doit/doit_cmd.py", line 165, in run
    return command.parse_execute(args)
  File "/home/ralsina/.virtualenvs/nikola/local/lib/python2.7/site-packages/doit-0.28.0-py2.7.egg/doit/cmd_base.py", line 122, in parse_execute
    return self.execute(params, args)
  File "/home/ralsina/Desktop/proyectos/nikola/master/nikola/plugin_categories.py", line 125, in execute
    self._execute(options, args)
  File "/home/ralsina/Desktop/proyectos/nikola/master/nikola/plugins/command/import_wordpress.py", line 162, in _execute
    self.import_posts(channel)
  File "/home/ralsina/Desktop/proyectos/nikola/master/nikola/plugins/command/import_wordpress.py", line 512, in import_posts
    self.process_item(item)
  File "/home/ralsina/Desktop/proyectos/nikola/master/nikola/plugins/command/import_wordpress.py", line 506, in process_item
    self.import_item(item, wordpress_namespace, 'posts')
  File "/home/ralsina/Desktop/proyectos/nikola/master/nikola/plugins/command/import_wordpress.py", line 484, in import_item
    content = self.transform_content(content)
  File "/home/ralsina/Desktop/proyectos/nikola/master/nikola/plugins/command/import_wordpress.py", line 369, in transform_content
    content = self.transform_code(content)
  File "/home/ralsina/Desktop/proyectos/nikola/master/nikola/plugins/command/import_wordpress.py", line 350, in transform_code
    content = self.code_re3.sub(replacement, content)
  File "/home/ralsina/Desktop/proyectos/nikola/master/nikola/plugins/command/import_wordpress.py", line 341, in replacement
    code = m.group(2)
IndexError: no such group
```

Small issue with new html-tidy filter defaults
Love the html-tidy filter! One issue: by default the option **drop-empty-elements** is set to **yes**. This has the effect of stripping things that look like this out of HTML:

`<i class="fa fa-bomb"></i>`

This should show a bomb from Font Awesome, but it is stripped out by html-tidy5. I added the option
 `--drop-empty-elements no` 
in filters.py and it fixed it right away. I believe that this should be the default going forward.



Discussion:

Use an `<ins>` element for placeholders for insertions. Add a space to make the element non-empty.

@Aeyoun that won't do. According to the bootstrap doc:

Only for use on empty elements
Icon classes should only be used on elements that contain no text content and have no child elements.

http://getbootstrap.com/components/

It's quite an easy fix to add the parameter to filters.py, and it seems to work just fine afterward. Alternate methods would be to create a tidy5 configuration file, or change the makefile to make keeping empty elements the default, but I found it much easier to just modify the last of defaults Nikola sends.

Date: Mon, 8 Jun 2015 05:53:16 -0700
From: notifications@github.com
To: nikola@noreply.github.com
CC: ccrayton@viewkeeper.org
Subject: Re: [nikola] Small issue with new html-tidy filter defaults (#1795)

@Aeyoun that won't do. According to the bootstrap doc:

Only for use on empty elements

Icon classes should only be used on elements that contain no text content and have no child elements.

http://getbootstrap.com/components/

—
Reply to this email directly or view it on GitHub.

:+1: for adding configuration file support.

Any suggestions on how this could look?

We could add a `conf.py` setting to run `tidy5` with a desired config file, and if it’s set, just use that.

(also, adding `--drop-empty-elements no` by default is a good idea IMO)

I was actually thinking of deprecating tidy5 in favor of html5lib (much better performance and serves the same purpose).

How about not adding a new config, but a new meta-filter like:

``` .py
def create_custom_html_tidy(options):
    return lambda infile: _html_tidy_runner(infile, r'-quiet --show-info no --show-warnings no -utf8 ' + options + r' -modify %1')
```

then one can write

```
FILTERS = {
    '.html': [filters.create_custom_html_tidy('-indent --indent-attributes no --sort-attributes alpha --wrap 0 --wrap-sections no --tidy-mark no --drop-empty-elements no')]
}
```

Huge :-1:.

It's cleaner and nicer to use a config file instead of messing with command
line arguments. It also has the added bonus of being usable outside of
Nikola.

We value simplicity over ugly hacks.

## 

Chris Warrick https://chriswarrick.com
Sent from my Galaxy S3.
On 11 Jun 2015 22:59, "felixfontein" notifications@github.com wrote:

> How about not adding a new config, but a new meta-filter like:
> 
> def create_custom_html_tidy(options):
>     return lambda infile: _html_tidy_runner(r'-quiet --show-info no --show-warnings no -utf8 ' + infile + r' -modify %1', options)
> 
> then one can write
> 
> FILTERS = {
>     '.html': [filters.create_custom_html_tidy('-indent --indent-attributes no --sort-attributes alpha --wrap 0 --wrap-sections no --tidy-mark no --drop-empty-elements no')]
> }
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/getnikola/nikola/issues/1795#issuecomment-111275891.


Resolving patch:
diff --git a/nikola/filters.py b/nikola/filters.py
--- a/nikola/filters.py
+++ b/nikola/filters.py
@@ -147,20 +147,24 @@ def jpegoptim(infile):
     return runinplace(r"jpegoptim -p --strip-all -q %1", infile)
 
 
+def html_tidy_withconfig(infile):
+    return _html_tidy_runner(infile, r"-quiet --show-info no --show-warnings no -utf8 -indent -config tidy5.conf -modify %1")
+
+
 def html_tidy_nowrap(infile):
-    return _html_tidy_runner(infile, r"-quiet --show-info no --show-warnings no -utf8 -indent --indent-attributes no --sort-attributes alpha --wrap 0 --wrap-sections no --tidy-mark no -modify %1")
+    return _html_tidy_runner(infile, r"-quiet --show-info no --show-warnings no -utf8 -indent --indent-attributes no --sort-attributes alpha --wrap 0 --wrap-sections no --drop-empty-elements no --tidy-mark no -modify %1")
 
 
 def html_tidy_wrap(infile):
-    return _html_tidy_runner(infile, r"-quiet --show-info no --show-warnings no -utf8 -indent --indent-attributes no --sort-attributes alpha --wrap 80 --wrap-sections no --tidy-mark no -modify %1")
+    return _html_tidy_runner(infile, r"-quiet --show-info no --show-warnings no -utf8 -indent --indent-attributes no --sort-attributes alpha --wrap 80 --wrap-sections no --drop-empty-elements no --tidy-mark no -modify %1")
 
 
 def html_tidy_wrap_attr(infile):
-    return _html_tidy_runner(infile, r"-quiet --show-info no --show-warnings no -utf8 -indent --indent-attributes yes --sort-attributes alpha --wrap 80 --wrap-sections no --tidy-mark no -modify %1")
+    return _html_tidy_runner(infile, r"-quiet --show-info no --show-warnings no -utf8 -indent --indent-attributes yes --sort-attributes alpha --wrap 80 --wrap-sections no --drop-empty-elements no --tidy-mark no -modify %1")
 
 
 def html_tidy_mini(infile):
-    return _html_tidy_runner(infile, r"-quiet --show-info no --show-warnings no -utf8 --indent-attributes no --sort-attributes alpha --wrap 0 --wrap-sections no --tidy-mark no -modify %1")
+    return _html_tidy_runner(infile, r"-quiet --show-info no --show-warnings no -utf8 --indent-attributes no --sort-attributes alpha --wrap 0 --wrap-sections no --tidy-mark no --drop-empty-elements no -modify %1")
 
 
 def _html_tidy_runner(infile, options):
diff --git a/nikola/plugins/command/import_wordpress.py b/nikola/plugins/command/import_wordpress.py
--- a/nikola/plugins/command/import_wordpress.py
+++ b/nikola/plugins/command/import_wordpress.py
@@ -336,9 +336,13 @@ def transform_code(self, content):
         # a ton of things not supported here. We only do a basic [code
         # lang="x"] -> ```x translation, and remove quoted html entities (<,
         # >, &, and ").
-        def replacement(m):
-            language = m.group(1) or ''
-            code = m.group(2)
+        def replacement(m, c=content):
+            if len(m.groups()) == 1:
+                language = ''
+                code = m.group(0)
+            else:
+                language = m.group(1) or ''
+                code = m.group(2)
             code = code.replace('&amp;', '&')
             code = code.replace('&gt;', '>')
             code = code.replace('&lt;', '<')
diff --git a/nikola/plugins/task/posts.py b/nikola/plugins/task/posts.py
--- a/nikola/plugins/task/posts.py
+++ b/nikola/plugins/task/posts.py
@@ -25,9 +25,10 @@
 # SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
 
 from copy import copy
+import os
 
 from nikola.plugin_categories import Task
-from nikola import utils
+from nikola import filters, utils
 
 
 def update_deps(post, lang, task):
@@ -91,7 +92,22 @@ def tl_ch():
                     ] + post.fragment_deps_uptodate(lang),
                     'task_dep': ['render_posts:timeline_changes']
                 }
-                yield task
+
+                # Apply filters specified in the metadata
+                ff = [x.strip() for x in post.meta('filters', lang).split(',')]
+                flist = []
+                for i, f in enumerate(ff):
+                    if not f:
+                        continue
+                    if f.startswith('filters.'):  # A function from the filters module
+                        f = f[8:]
+                        try:
+                            flist.append(getattr(filters, f))
+                        except AttributeError:
+                            pass
+                    else:
+                        flist.append(f)
+                yield utils.apply_filters(task, {os.path.splitext(dest): flist})
 
     def dependence_on_timeline(self, post, lang):
         if "####MAGIC####TIMELINE" not in post.fragment_deps(lang):


Test patch: