Repository: getpelican/pelican

Issue:
Footnote processing returns non-valid html rel="footnote"
When validating pelican-created HTML5, validation fails on the markup of footnotes.

It would appear that the Markdown processor automatically includes the attribute _rel="footnote"_, which attribute is not included in html specification.

Footnote processing returns non-valid html rel="footnote"
When validating pelican-created HTML5, validation fails on the markup of footnotes.

It would appear that the Markdown processor automatically includes the attribute _rel="footnote"_, which attribute is not included in html specification.



Discussion:
You should add `output_format='html5'` to the Markdown class, then the footnote extension will check for this :
http://pythonhosted.org/Markdown/reference.html#output_format
The problem is that there is no way to do this currently. Maybe we should add a `MD_SETTINGS` like the `DOCUTILS_SETTINGS` ? The `MD_EXTENSIONS` is just a subset of the Markdown settings, it could be replaced with:

```
MD_SETTINGS = {
    'extensions': [], # same as MD_EXTENSIONS
    'output_format': 'html5',
   ...
}
```

@saimn: Sounds like a good idea to me!

@shabob: Might you be interested in implementing a solution based on Simon's suggestion?

@Mihara: Since you seem to have run into this issue recently, what do you think of @saimn's suggestion?

I think it's the way to go. Since I don't currently have the time to pick apart pelican's own internals, I hacked up the [markdown_thirdparty plugin](https://github.com/FuzzyWuzzie/markdown_thirdparty) to force the output_format to html5, but a solution for actually configuring markdown settings in a generic way is obviously better.

As a side note, amending that suggestion to also make pelican look for markdown extensions in the same locations it loads plugins from, rather than just site-packages _(as far as I can tell it doesn't, but correct me if I'm wrong)_ would be even better.

Just because it fits with the problem domain "invalid HTML": I've found invalid HTML being generated using reStructuredText (_not_ Markdown) too, and footnote is a matching problem there.

I've mentioned the specific validation error messages in a PR of a Pelican theme: [cid (PR #3)](https://github.com/hdra/Pelican-Cid/pull/3)

Can these problems be fixed using a specific configuration, or do I have to suggest a different HTML output being generated in general at the [Sphinx project](http://sphinx-doc.org/)? Ideas anyone?

You should add `output_format='html5'` to the Markdown class, then the footnote extension will check for this :
http://pythonhosted.org/Markdown/reference.html#output_format
The problem is that there is no way to do this currently. Maybe we should add a `MD_SETTINGS` like the `DOCUTILS_SETTINGS` ? The `MD_EXTENSIONS` is just a subset of the Markdown settings, it could be replaced with:

```
MD_SETTINGS = {
    'extensions': [], # same as MD_EXTENSIONS
    'output_format': 'html5',
   ...
}
```

@saimn: Sounds like a good idea to me!

@shabob: Might you be interested in implementing a solution based on Simon's suggestion?

@Mihara: Since you seem to have run into this issue recently, what do you think of @saimn's suggestion?

I think it's the way to go. Since I don't currently have the time to pick apart pelican's own internals, I hacked up the [markdown_thirdparty plugin](https://github.com/FuzzyWuzzie/markdown_thirdparty) to force the output_format to html5, but a solution for actually configuring markdown settings in a generic way is obviously better.

As a side note, amending that suggestion to also make pelican look for markdown extensions in the same locations it loads plugins from, rather than just site-packages _(as far as I can tell it doesn't, but correct me if I'm wrong)_ would be even better.

Just because it fits with the problem domain "invalid HTML": I've found invalid HTML being generated using reStructuredText (_not_ Markdown) too, and footnote is a matching problem there.

I've mentioned the specific validation error messages in a PR of a Pelican theme: [cid (PR #3)](https://github.com/hdra/Pelican-Cid/pull/3)

Can these problems be fixed using a specific configuration, or do I have to suggest a different HTML output being generated in general at the [Sphinx project](http://sphinx-doc.org/)? Ideas anyone?


Resolving patch:
diff --git a/pelican/readers.py b/pelican/readers.py
--- a/pelican/readers.py
+++ b/pelican/readers.py
@@ -246,8 +246,16 @@ class MarkdownReader(BaseReader):
 
     def __init__(self, *args, **kwargs):
         super(MarkdownReader, self).__init__(*args, **kwargs)
-        self.extensions = self.settings['MD_EXTENSIONS']
-        self.extensions.setdefault('markdown.extensions.meta', {})
+        # make sure 'extension_configs' exists and
+        # and either way 'markdown.extensions.meta' must be in there
+        settings = self.settings['MARKDOWN']
+        settings.setdefault('extension_configs', {})
+        settings['extension_configs'].setdefault(
+            'markdown.extensions.meta', {})
+        settings.setdefault('extensions', [])
+        settings['extensions'].extend(
+            list(settings['extension_configs'].keys()))
+        settings['extensions'] = list(set(settings['extensions']))
         self._source_path = None
 
     def _parse_metadata(self, meta):
@@ -283,8 +291,7 @@ def read(self, source_path):
         """Parse content and metadata of markdown files"""
 
         self._source_path = source_path
-        self._md = Markdown(extensions=self.extensions.keys(),
-                            extension_configs=self.extensions)
+        self._md = Markdown(**self.settings['MARKDOWN'])
         with pelican_open(source_path) as text:
             content = self._md.convert(text)
 
diff --git a/pelican/settings.py b/pelican/settings.py
--- a/pelican/settings.py
+++ b/pelican/settings.py
@@ -101,10 +101,13 @@ def load_source(name, path):
     'PELICAN_CLASS': 'pelican.Pelican',
     'DEFAULT_DATE_FORMAT': '%a %d %B %Y',
     'DATE_FORMATS': {},
-    'MD_EXTENSIONS': {
-        'markdown.extensions.codehilite': {'css_class': 'highlight'},
-        'markdown.extensions.extra': {},
-        'markdown.extensions.meta': {},
+    'MARKDOWN': {
+        'extension_configs': {
+            'markdown.extensions.codehilite': {'css_class': 'highlight'},
+            'markdown.extensions.extra': {},
+            'markdown.extensions.meta': {},
+        },
+        'output_format': 'html5',
     },
     'JINJA_EXTENSIONS': [],
     'JINJA_FILTERS': {},
@@ -368,13 +371,11 @@ def configure_settings(settings):
                            PATH_KEY)
             settings[PATH_KEY] = DEFAULT_CONFIG[PATH_KEY]
 
-    # Save people from declaring MD_EXTENSIONS as a list rather than a dict
-    if not isinstance(settings.get('MD_EXTENSIONS', {}), dict):
-        logger.warning('The format of the MD_EXTENSIONS setting has '
-                       'changed. It should now be a dict mapping '
-                       'fully-qualified extension names to their '
-                       'configurations. Falling back to the default.')
-        settings['MD_EXTENSIONS'] = DEFAULT_CONFIG['MD_EXTENSIONS']
+    # Deprecated warning of MD_EXTENSIONS
+    if 'MD_EXTENSIONS' in settings:
+        logger.warning('MD_EXTENSIONS is deprecated use MARKDOWN '
+                       'instead. Falling back to the default.')
+        settings['MARKDOWN'] = DEFAULT_CONFIG['MARKDOWN']
 
     # Add {PAGE,ARTICLE}_PATHS to {ARTICLE,PAGE}_EXCLUDES
     mutually_exclusive = ('ARTICLE', 'PAGE')


Test patch:
diff --git a/pelican/tests/test_pelican.py b/pelican/tests/test_pelican.py
--- a/pelican/tests/test_pelican.py
+++ b/pelican/tests/test_pelican.py
@@ -208,18 +208,17 @@ def test_write_only_selected(self):
             msg="Writing .*",
             level=logging.INFO)
 
-    def test_md_extensions_list_deprecation(self):
-        """Test that a warning is issued if MD_EXTENSIONS is a list"""
+    def test_md_extensions_deprecation(self):
+        """Test that a warning is issued if MD_EXTENSIONS is used"""
         settings = read_settings(path=None, override={
             'PATH': INPUT_PATH,
             'OUTPUT_PATH': self.temp_path,
             'CACHE_PATH': self.temp_cache,
-            'MD_EXTENSIONS': ['meta'],
+            'MD_EXTENSIONS': {},
         })
         pelican = Pelican(settings=settings)
         mute(True)(pelican.run)()
-        self.assertIsInstance(pelican.settings['MD_EXTENSIONS'], dict)
         self.assertLogCountEqual(
             count=1,
-            msg="The format of the MD_EXTENSIONS setting has changed",
+            msg="MD_EXTENSIONS is deprecated use MARKDOWN instead.",
             level=logging.WARNING)
diff --git a/pelican/tests/test_readers.py b/pelican/tests/test_readers.py
--- a/pelican/tests/test_readers.py
+++ b/pelican/tests/test_readers.py
@@ -405,20 +405,20 @@ def test_article_with_footnote(self):
             _path('article_with_markdown_and_footnote.md'))
         expected_content = (
             '<p>This is some content'
-            '<sup id="fnref:1"><a class="footnote-ref" href="#fn:1" '
-            'rel="footnote">1</a></sup>'
+            '<sup id="fnref-1"><a class="footnote-ref" href="#fn-1"'
+            '>1</a></sup>'
             ' with some footnotes'
-            '<sup id="fnref:footnote"><a class="footnote-ref" '
-            'href="#fn:footnote" rel="footnote">2</a></sup></p>\n'
+            '<sup id="fnref-footnote"><a class="footnote-ref" '
+            'href="#fn-footnote">2</a></sup></p>\n'
 
             '<div class="footnote">\n'
-            '<hr />\n<ol>\n<li id="fn:1">\n'
+            '<hr>\n<ol>\n<li id="fn-1">\n'
             '<p>Numbered footnote&#160;'
-            '<a class="footnote-backref" href="#fnref:1" rev="footnote" '
+            '<a class="footnote-backref" href="#fnref-1" '
             'title="Jump back to footnote 1 in the text">&#8617;</a></p>\n'
-            '</li>\n<li id="fn:footnote">\n'
+            '</li>\n<li id="fn-footnote">\n'
             '<p>Named footnote&#160;'
-            '<a class="footnote-backref" href="#fnref:footnote" rev="footnote"'
+            '<a class="footnote-backref" href="#fnref-footnote"'
             ' title="Jump back to footnote 2 in the text">&#8617;</a></p>\n'
             '</li>\n</ol>\n</div>')
         expected_metadata = {
@@ -482,10 +482,12 @@ def test_article_with_markdown_markup_extension(self):
         # expected
         page = self.read_file(
             path='article_with_markdown_markup_extensions.md',
-            MD_EXTENSIONS={
-                'markdown.extensions.toc': {},
-                'markdown.extensions.codehilite': {},
-                'markdown.extensions.extra': {}
+            MARKDOWN={
+                'extension_configs': {
+                    'markdown.extensions.toc': {},
+                    'markdown.extensions.codehilite': {},
+                    'markdown.extensions.extra': {}
+                }
             }
         )
         expected = ('<div class="toc">\n'