Repository: getnikola/nikola

Issue:
Unable to use Tidy on html
Hi, 

I enabled the tidy filter on html file:

from nikola import filters
FILTERS = {
        ".html": [filters.tidy],
        }

and now NIkola refuses to build my site: 

TaskError - taskid:render_tags:../categories/index.html
PythonAction Error
Traceback (most recent call last):
  File "/usr/lib64/python3.3/site-packages/doit/action.py", line 360, in execute
    returned_value = self.py_callable(_self.args, *_kwargs)
  File "/usr/lib64/python3.3/site-packages/nikola/utils.py", line 529, in unlessLink
    action(target)
  File "/usr/lib64/python3.3/site-packages/nikola/filters.py", line 144, in tidy
    for line in output.split("\n"):
TypeError: Type str doesn't support the buffer AP

Is there a problem with tidy ? Can I use another thing as a replacement ?

Unable to use Tidy on html
Hi, 

I enabled the tidy filter on html file:

from nikola import filters
FILTERS = {
        ".html": [filters.tidy],
        }

and now NIkola refuses to build my site: 

TaskError - taskid:render_tags:../categories/index.html
PythonAction Error
Traceback (most recent call last):
  File "/usr/lib64/python3.3/site-packages/doit/action.py", line 360, in execute
    returned_value = self.py_callable(_self.args, *_kwargs)
  File "/usr/lib64/python3.3/site-packages/nikola/utils.py", line 529, in unlessLink
    action(target)
  File "/usr/lib64/python3.3/site-packages/nikola/filters.py", line 144, in tidy
    for line in output.split("\n"):
TypeError: Type str doesn't support the buffer AP

Is there a problem with tidy ? Can I use another thing as a replacement ?



Discussion:
You are using python 3, right? Probably it's just that noone tried this and it's not working right.

Yes, I'm using Python 3.3, but the 2.7 version is also installed. If I have installed nikola using pip, is there a way to change the Python version used ? 

Easiest is just to install it again: pip-2.7 install nikola

Or wait 15' and I'm sure I can find you a simple patch ;-)

I was lying, probably won't have time to look at it today.

I’ll attempt a fix soon.

With all that brokenness in the tool, it's sort of pointless.

So, what is the official resolution?

(a) nuke tidy-in-Nikola altogether; or  
(b) pull in #1167 and mark tidy as “not production-ready as it destroys many HTML5 things (don’t forget it was last updated in 2009)”

(c) Tidy-HTML-Light were we strip empty lines and indenting spaces altogether. Force line breaks before and after closing [sectioning elements](http://www.w3.org/TR/html5/sections.html). Better than nothing.

Google doesn’t seem to know what Tidy-HTML-Light is.  Same goes for DuckDuckGo.  I’m pretty sure it doesn’t exist.

Well, if the goal is to produce pretty HTML, we could just use lxml and pretty-print it.

Two kind of goals here: smaller output (stripped whitespace) or pretty HTML.

Tidy-HTML-Light is better known as regex.

Thanks for the help. I tried using 2.7, but after that, my site refused to build with an UTF-8 error. I will continue 3.3. It was more a test than a real need, so don't worry too much about it for me. 

Do I let the issue open ?

@wichtounet you should create a new issue describing the UTF-8 problem.  We will close this issue if we believe we fixed it well enough (or that we won’t fix it).

@Aeyoun 

For stripping whitespace, yui-compress or whatever does a much better job. For prettier HTML, lxml does a much better job. So, let's fix this by removing tidy.

You are using python 3, right? Probably it's just that noone tried this and it's not working right.

Yes, I'm using Python 3.3, but the 2.7 version is also installed. If I have installed nikola using pip, is there a way to change the Python version used ? 

Easiest is just to install it again: pip-2.7 install nikola

Or wait 15' and I'm sure I can find you a simple patch ;-)

I was lying, probably won't have time to look at it today.

I’ll attempt a fix soon.

With all that brokenness in the tool, it's sort of pointless.

So, what is the official resolution?

(a) nuke tidy-in-Nikola altogether; or  
(b) pull in #1167 and mark tidy as “not production-ready as it destroys many HTML5 things (don’t forget it was last updated in 2009)”

(c) Tidy-HTML-Light were we strip empty lines and indenting spaces altogether. Force line breaks before and after closing [sectioning elements](http://www.w3.org/TR/html5/sections.html). Better than nothing.

Google doesn’t seem to know what Tidy-HTML-Light is.  Same goes for DuckDuckGo.  I’m pretty sure it doesn’t exist.

Well, if the goal is to produce pretty HTML, we could just use lxml and pretty-print it.

Two kind of goals here: smaller output (stripped whitespace) or pretty HTML.

Tidy-HTML-Light is better known as regex.

Thanks for the help. I tried using 2.7, but after that, my site refused to build with an UTF-8 error. I will continue 3.3. It was more a test than a real need, so don't worry too much about it for me. 

Do I let the issue open ?

@wichtounet you should create a new issue describing the UTF-8 problem.  We will close this issue if we believe we fixed it well enough (or that we won’t fix it).

@Aeyoun 

For stripping whitespace, yui-compress or whatever does a much better job. For prettier HTML, lxml does a much better job. So, let's fix this by removing tidy.


Resolving patch:
diff --git a/nikola/filters.py b/nikola/filters.py
--- a/nikola/filters.py
+++ b/nikola/filters.py
@@ -30,7 +30,6 @@
 from functools import wraps
 import os
 import codecs
-import re
 import shutil
 import subprocess
 import tempfile
@@ -144,50 +143,6 @@ def jpegoptim(infile):
     return runinplace(r"jpegoptim -p --strip-all -q %1", infile)
 
 
-def tidy(inplace):
-    # Google site verifcation files are not HTML
-    if re.match(r"google[a-f0-9]+.html", os.path.basename(inplace)) \
-            and open(inplace).readline().startswith(
-                "google-site-verification:"):
-        return
-
-    # Tidy will give error exits, that we will ignore.
-    output = subprocess.check_output(
-        "tidy -m -w 90 --indent no --quote-marks"
-        "no --keep-time yes --tidy-mark no "
-        "--force-output yes '{0}'; exit 0".format(inplace), stderr=subprocess.STDOUT, shell=True)
-
-    for line in output.split("\n"):
-        if "Warning:" in line:
-            if '<meta> proprietary attribute "charset"' in line:
-                # We want to set it though.
-                continue
-            elif '<meta> lacks "content" attribute' in line:
-                # False alarm to me.
-                continue
-            elif '<div> anchor' in line and 'already defined' in line:
-                # Some seeming problem with JavaScript terminators.
-                continue
-            elif '<img> lacks "alt" attribute' in line:
-                # Happens in gallery code, probably can be tolerated.
-                continue
-            elif '<table> lacks "summary" attribute' in line:
-                # Happens for tables, TODO: Check this is normal.
-                continue
-            elif 'proprietary attribute "data-toggle"' in line or \
-                 'proprietary attribute "data-target"':
-                # Some of our own tricks
-                continue
-            else:
-                assert False, (inplace, line)
-        elif "Error:" in line:
-            if '<time> is not recognized' in line:
-                # False alarm, time is proper HTML5.
-                continue
-            else:
-                assert False, line
-
-
 @apply_to_text_file
 def typogrify(data):
     if typo is None:


Test patch: