Repository: mkdocs/mkdocs

Issue:
Support Markdown Extension Configs
Mkdocs uses Python-Markdown and offers support for users to specify extensions supported by Python-Markdown. However, currently mkdocs provides no way (sort-of; see below) to pass config settings into the extensions specified. For example, I would often like to use the [TOC](https://pythonhosted.org/Markdown/extensions/toc.html#usage) Extension's `permalink` feature to add the paragraph symbol (¶ or `&para;`) to the end of each header (`h1-6`) as a self-link. A very handy feature in documentation (in fact, it's used by Python-Markdown's docs, which I've linked to). Another use may be to customize how [syntax highlighting](https://pythonhosted.org/Markdown/extensions/code_hilite.html#usage) works. Or non-english users may want to override the default substitutions used by [SmartyPants](https://pythonhosted.org/Markdown/extensions/smarty.html). etc, etc...

Sure, mkdocs could hard-code some specific settings for things that make sense. Granted, not all extensions configs make sense in the context of mkdocs. But do you really want to be fielding bug reports for years to come asking for some additional config setting? And don't forget about [third party extensions](https://github.com/waylan/Python-Markdown/wiki/Third-Party-Extensions). Some of them (like a few of the math extensions) make a lot of sense for some users of mkdocs. But you couldn't possibly expect to provide good support for them without offering a holistic approach.

Interestingly, as mkdocs' config file is YAML and Python-Markdown's [`extension_configs`](https://pythonhosted.org/Markdown/reference.html#extension_configs) keyword accepts a dictionary of dictionaries, it is pretty easy to replicate in the config file. Of note is the recent addition to Python-Markdown's CLI of support for a [YAML config file](https://pythonhosted.org/Markdown/cli.html#using-extensions) for this very thing. One difference I would suggest is that, unlike the CLI feature, there should be no need to also provide a list of extensions separate from the extension configs.  I would expect the implementation to be something like this:

``` python
if isinstance(config['markdown_extensions'], dict):
    extensions = config['markdown_extensions'].keys()
    extensions_configs = config['markdown_extensions']
else:
    # backward compat with old list type
    extensions = config['markdown_extensions']
    extensions_configs = {}

# then later in the code...
md = markdown.Markdown(
    extensions=extensions,
    extensions_configs = extensions_configs
)
html_content = md.convert(markdown_source)
```

Note that support for the existing list of extensions is still supported. The new behavior only happens when the config setting is of the dict type. And the new format in the config file would look something like this:

``` yaml
markdown_extensions:
    markdown.extensions.toc:
        permalink: True
    markdown.extensions.codehilite:
        linenums: True
    markdown.extensions.smarty:
        smart_angled_quotes: True,
        substitutions:
            left-single-quote: '&sbquo;', 
            right-single-quote: '&lsquo;',
            left-double-quote: '&bdquo;',
            right-double-quote: '&ldquo;'
```

A few notes about this proposal:
1. Actually mkdocs does provide a method of sorts for passing in config settings which involves including the settings right in the name of the extension (see the [old docs](https://github.com/waylan/Python-Markdown/blob/e4c13788f1c6f6f204ca7c471b25246f6c156832/docs/reference.txt#L91)). However, that behavior is [deprecated](https://pythonhosted.org/Markdown/release-2.6.html#extension-configuration-as-part-of-extension-name-deprecated) in Python-Markdown 2.6 (released yesterday) and was "pending deprecation" in 2.5. Even in older versions, it is very limited as no spaces are allowed and only string types could be used (booleans were tricky -- usually a 0 or 1 and then special code in the extension to call `int()` on the string). 
2. In previous versions of Python-Markdown (prior to version 2.5), the `extensions_configs` keyword expected a list of tuples for each extension rather than a dictionary (again see the [old docs](https://github.com/waylan/Python-Markdown/blob/e4c13788f1c6f6f204ca7c471b25246f6c156832/docs/reference.txt#L107)). That being the case, the earliest version of Python-Markdown this would work with is 2.5.
3. As of today, mkdocs only supports Python-Markdown version 2.4 as that was the last version to support Python 2.6 (see #165). However, dropping support for Python 2.6 appears to be earmarked for mkdoc 1.0.0 (0.12.0 is currently in development with 1.0.0 being the next milestone).

Given the above, I expect that this feature would be added to 1.0 at the earliest (although I understand if the desire is to not add any new features until after 1.0). Given that fact, I don't see much point in actually creating a pull request just yet (with tests and docs -- the code changes are pretty simple -- see above) as I'm not really interested in continually rebasing it until you guys are ready to accept it. That said, if the idea is accepted, I'll happily do the work when the time comes. If, on the other hand, you guys are not interested in adding support for this, I'll probably look for another project which fits my needs better (not having this is a non-starter for me).



Discussion:
Sorry for the slow reply to this well detailed post. This is quite an intersting one, I have wanted to pass config to the extensions too (also for permalink as it happens) but I couldn't figure out a nice way to do it. It looks like your method would work, but it is rather huge :) This is perhaps one of the reasons Tom was hessitant to let users do this at all. 

That being said, I think it is probably an okay approach, and I'd be +1. You are right, it does sound like we need to do this for the 1.0 milestone (with the python 2.6 drop). Given it is a fairly small change to an existing feature I'd be okay with it being in the 1.0 milestone and not after. Also, it'd be awesome in Python-Markdown used MkDocs :smile:

What would the YAML look like if you had some extensions with config, and some without?

@d0ugal that a good question. As it turns out, YAML is pretty flexable. Consider the following:

``` python
>>> config = """
... markdown_extensions:
...     codehilite:
...         linenums: True
...     toc:
...         permalink: True
...     meta:
...     extra: {}
... """
>>> yaml.load(config)
{'markdown_extensions': {
    'codehilite': {'linenums': True},
    'toc': {'permalink': True},
    'meta': None,
    'extra': {}
}}
```

Remember that YAML is a superset of JSON. So you can use JSON (with curly brackets) as well. And as Python Dicts use the same format as JSON check this out:

``` python
>>> d = yaml.load(config)
>>> s = str(d)
>>> s
"{'markdown_extensions': 'codehilite': {'linenums': True}, 'toc': {'permalink': True}, 'meta': None, 'extra': {}}}"
>>> yaml.load(s)
{'markdown_extensions': 'codehilite': {'linenums': True}, 'toc': {'permalink': True}, 'meta': None, 'extra': {}}}
```

Above, I created a string of the Python dict (which has no line breaks but full use of brackets) and passed that back to YAML and it gave back a dict.

Technically, this is an easy problem to solve. However, the real issue is making sure the format is easy for users who are unfamiliar with YAML, JSON, and/or Python Dicts. That is probably more of a documentation issue. How do we want to document mixing extensions with and without config options?

huh, interesting, I didn't realise that "meta: " was a valid line, for some reason I expected the value to be required.

As for how we want to document it, I'm not sure - but I'm not a fan of the approach in https://github.com/tomchristie/mkdocs/pull/366 :)

I agree. As I noted in a comment on that PR, that is a deprecated feature which will stop working in the next release. I was referring to this in my note 1 in my proposal above.

So I was looking as the mkdocs source to determine where to implement this and I'm not sure where you would prefer this be done. Basically, I'm looking for where I should insert this code (or some variation thereof):

``` python
if isinstance(config['markdown_extensions'], dict):
    extensions = config['markdown_extensions'].keys()
    extensions_configs = config['markdown_extensions']
else:
    # backward compat with old list type
    extensions = config['markdown_extensions']
    extensions_configs = {}
```

My initial thought was to simply add this to `mkdocs.build.convert_markdown` which would work fine and be the least intrusive. However, I noted that there is the `mkdocs.config.validate_config` function which seems like it might be the proper place to do this. However, to do that, then the config (at least internally) would need to also have a `markdown_extension_configs` key which would need to be passed to `mkdocs.build.convert_markdown` from `mkdocs.build._build_page`. Of course, `mkdocs.build.convert_markdown` would also need to have an additional keyword argument to accept the optional `extension_configs`. All doable, but more intrusive and slightly changes the API of `mkdocs.build.convert_markdown`. It just feels more "correct" that way to me.

I note that the internal extensions are all defined within `mkdocs.build.convert_markdown`. It would seem to me that that should all be happening in `mkdocs.config.validate_config` as well... which is why I'm having this quandary. In any event, I suppose the extension config stuff should be in the same place as the internal extensions. 

Yup, I agree. This should all happen in `validate_config` (including the current defaults which are not.) So I would be happy to see the new and old relevant code move there. The API changes shouldn't be a problem other than for the existing tests and we don't consider our the code to be a public API.

My only concern is that `validate_config` is already becoming a nightmare to maintain. I've been looking for a way to better manage that part of the code, but I've not had any particularly good ideas yet. Just splitting it into many smaller methods that validate different areas might be enough.

Moving the extension stuff to `validate_config` is easy enough. But most of the tests in `tests/build_tests.py` fail without creating and passing a config instance in. While it is certainly doable, it makes the build tests mostly feel like config tests. I'm going to leave everything in `convert_markdown` for now and wait for `validate_config` to be improved generally.


Resolving patch:
diff --git a/mkdocs/build.py b/mkdocs/build.py
--- a/mkdocs/build.py
+++ b/mkdocs/build.py
@@ -28,11 +28,18 @@ def convert_markdown(markdown_source, site_navigation=None, extensions=(), stric
     """
 
     # Generate the HTML from the markdown source
+    if isinstance(extensions, dict):
+        user_extensions = list(extensions.keys())
+        extension_configs = dict([(k, v) for k, v in extensions.items() if isinstance(v, dict)])
+    else:
+        user_extensions = list(extensions)
+        extension_configs = {}
     builtin_extensions = ['meta', 'toc', 'tables', 'fenced_code']
     mkdocs_extensions = [RelativePathExtension(site_navigation, strict), ]
-    extensions = builtin_extensions + mkdocs_extensions + list(extensions)
+    extensions = set(builtin_extensions + mkdocs_extensions + user_extensions)
     md = markdown.Markdown(
-        extensions=extensions
+        extensions=extensions,
+        extension_configs=extension_configs
     )
     html_content = md.convert(markdown_source)
 


Test patch:
diff --git a/mkdocs/tests/build_tests.py b/mkdocs/tests/build_tests.py
--- a/mkdocs/tests/build_tests.py
+++ b/mkdocs/tests/build_tests.py
@@ -320,3 +320,22 @@ def test_strict_mode_invalid(self):
         self.assertRaises(
             MarkdownNotFound,
             build.convert_markdown, invalid, site_nav, strict=True)
+
+    def test_extension_config(self):
+        """
+        Test that a dictionary of 'markdown_extensions' is recognized as
+        both a list of extensions and a dictionary of extnesion configs.
+        """
+        markdown_extensions = {
+            'toc': {'permalink': True},
+            'meta': None  # This gets ignored as it is an invalid config
+        }
+        html, toc, meta = build.convert_markdown(dedent("""
+        # A Header
+        """), extensions=markdown_extensions)
+
+        expected_html = dedent("""
+        <h1 id="a-header">A Header<a class="headerlink" href="#a-header" title="Permanent link">&para;</a></h1>
+        """)
+
+        self.assertEqual(html.strip(), expected_html)