Repository: getnikola/nikola

Issue:
How to avoid duplicated ids and h1 on the blog index ?
Hi, 

I was trying to optimize my site to fix some issues in HTML and I found out that there were issues on the blog index page because of the duplicated h1 and duplicate id (the ids automatically generated by Nikola on the headers for instance). 

Is there anyway to fix this ?

Thanks
How to avoid duplicated ids and h1 on the blog index ?
Hi, 

I was trying to optimize my site to fix some issues in HTML and I found out that there were issues on the blog index page because of the duplicated h1 and duplicate id (the ids automatically generated by Nikola on the headers for instance). 

Is there anyway to fix this ?

Thanks


Discussion:
Please elaborate. Do you mean that when there are headers in your blog posts, and those have IDs, the IDs are duplicated?

Well, that would be pretty hard to fix, if even doable.
@Kwpolska I edited my post. Sorry, I created it empty by error.
Please show a .html file that exhibits this problem.
The index of my blog: http://baptiste-wicht.com/

There are several duplicated IDs. And several H1 which is not very good for SEO, but not an HTML error as far as I know. 
Yeah, that’s what I was talking about. This is caused by posts having the same headers. This *could* be solved by post-processing the HTML, making a list of all IDs, finding duplicates and assigning new IDs to elements but careful enough not to create new duplicates. I’ll leave this open if anyone wants to fix this, but it’s a really minor bug of the *huge effort — small reward* category in my opinion.

> And several H1 which is not very good for SEO, but not an HTML error as far as I know.

[citation needed]. Nowadays \<h1\> can be duplicated just fine.
Hi, 

Thanks for the answer. 

Wouldn't it simply be possible to generate IDs that are unique directly ? Concatenating the ID of the article (if there is any in nikola) and the name of the header ? 

> [citation needed]. Nowadays h1 can be duplicated just fine.

You're right, it seems it is not much of an issue anymore. 
Well, the slug could be used as an id. Or maybe the gid you use in 
feeds, although that wouldn't be applicable for pages.

There is no simple ID we could use that would (a) look good and (b) be guaranteed to have only characters allowed in id tags.

And Nikola doesn’t generate any of those itself, those come from reST (and possibly other input formats), so fixing this for reST only will only solve part of the problem.
Alright, I would not have think this problem to be so complicated to fix. I think I'll leave with this for now ;)
As usual, we *can* fix this in postprocessing (we already do extensive munching of the generated HTML). If you don't mind having really awful IDs it's not really very hard to do.
I think I can solve this with *nice* IDs, as long as I figure out a way to get all IDs on a page. (Performance might be awful.)
Please elaborate. Do you mean that when there are headers in your blog posts, and those have IDs, the IDs are duplicated?

Well, that would be pretty hard to fix, if even doable.
@Kwpolska I edited my post. Sorry, I created it empty by error.
Please show a .html file that exhibits this problem.
The index of my blog: http://baptiste-wicht.com/

There are several duplicated IDs. And several H1 which is not very good for SEO, but not an HTML error as far as I know. 
Yeah, that’s what I was talking about. This is caused by posts having the same headers. This *could* be solved by post-processing the HTML, making a list of all IDs, finding duplicates and assigning new IDs to elements but careful enough not to create new duplicates. I’ll leave this open if anyone wants to fix this, but it’s a really minor bug of the *huge effort — small reward* category in my opinion.

> And several H1 which is not very good for SEO, but not an HTML error as far as I know.

[citation needed]. Nowadays \<h1\> can be duplicated just fine.
Hi, 

Thanks for the answer. 

Wouldn't it simply be possible to generate IDs that are unique directly ? Concatenating the ID of the article (if there is any in nikola) and the name of the header ? 

> [citation needed]. Nowadays h1 can be duplicated just fine.

You're right, it seems it is not much of an issue anymore. 
Well, the slug could be used as an id. Or maybe the gid you use in 
feeds, although that wouldn't be applicable for pages.

There is no simple ID we could use that would (a) look good and (b) be guaranteed to have only characters allowed in id tags.

And Nikola doesn’t generate any of those itself, those come from reST (and possibly other input formats), so fixing this for reST only will only solve part of the problem.
Alright, I would not have think this problem to be so complicated to fix. I think I'll leave with this for now ;)
As usual, we *can* fix this in postprocessing (we already do extensive munching of the generated HTML). If you don't mind having really awful IDs it's not really very hard to do.
I think I can solve this with *nice* IDs, as long as I figure out a way to get all IDs on a page. (Performance might be awful.)

Resolving patch:
diff --git a/nikola/filters.py b/nikola/filters.py
--- a/nikola/filters.py
+++ b/nikola/filters.py
@@ -442,6 +442,57 @@ def add_header_permalinks(fname, xpath_list=None, file_blacklist=None):
             new_node = lxml.html.fragment_fromstring('<a href="#{0}" class="headerlink" title="Permalink to this heading">¶</a>'.format(hid))
             node.append(new_node)
 
-    data = lxml.html.tostring(doc, encoding="unicode")
-    with io.open(fname, 'w+', encoding='utf-8') as outf:
-        outf.write(data)
+    with io.open(fname, 'w', encoding='utf-8') as outf:
+        outf.write(lxml.html.tostring(doc, encoding="unicode"))
+
+
+@_ConfigurableFilter(top_classes='DEDUPLICATE_IDS_TOP_CLASSES')
+@apply_to_text_file
+def deduplicate_ids(data, top_classes=None):
+    """Post-process HTML via lxml to deduplicate IDs."""
+    if not top_classes:
+        top_classes = ('postpage', 'storypage')
+    doc = lxml.html.document_fromstring(data)
+    elements = doc.xpath('//*')
+    all_ids = [element.attrib.get('id') for element in elements]
+    seen_ids = set()
+    duplicated_ids = set()
+    for i in all_ids:
+        if i is not None and i in seen_ids:
+            duplicated_ids.add(i)
+        else:
+            seen_ids.add(i)
+
+    if duplicated_ids:
+        # Well, that sucks.
+        for i in duplicated_ids:
+            # Results are ordered the same way they are ordered in document
+            offending_elements = doc.xpath('//*[@id="{}"]'.format(i))
+            counter = 2
+            # If this is a story or a post, do it from top to bottom, because
+            # updates to those are more likely to appear at the bottom of pages.
+            # For anything else, including indexes, do it from bottom to top,
+            # because new posts appear at the top of pages.
+            # We also leave the first result out, so there is one element with
+            # "plain" ID
+            if any(doc.find_class(c) for c in top_classes):
+                off = offending_elements[1:]
+            else:
+                off = offending_elements[-2::-1]
+            for e in off:
+                new_id = i
+                while new_id in seen_ids:
+                    new_id = '{0}-{1}'.format(i, counter)
+                    counter += 1
+                e.attrib['id'] = new_id
+                seen_ids.add(new_id)
+                # Find headerlinks that we can fix.
+                headerlinks = e.find_class('headerlink')
+                for hl in headerlinks:
+                    # We might get headerlinks of child elements
+                    if hl.attrib['href'] == '#' + i:
+                        hl.attrib['href'] = '#' + new_id
+                        break
+        return lxml.html.tostring(doc, encoding='unicode')
+    else:
+        return data


Test patch: