Repository: mkdocs/mkdocs

Issue:
Wrong default encoding
I love this project. But i find that, if i use chinese character as the name of markdown file and directory, the encoding will go wrong.
PS: I use python 2.7 on Mac OS X 10.10.3 and Ubuntu 14.04

```
Traceback (most recent call last):
  File "/Users/maemual/Workspace/mkdocs/env/bin/mkdocs", line 9, in <module>
    load_entry_point('mkdocs==0.13.0.dev0', 'console_scripts', 'mkdocs')()
  File "/Users/maemual/Workspace/mkdocs/env/lib/python2.7/site-packages/click/core.py", line 664, in __call__
    return self.main(*args, **kwargs)
  File "/Users/maemual/Workspace/mkdocs/env/lib/python2.7/site-packages/click/core.py", line 644, in main
    rv = self.invoke(ctx)
  File "/Users/maemual/Workspace/mkdocs/env/lib/python2.7/site-packages/click/core.py", line 991, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/maemual/Workspace/mkdocs/env/lib/python2.7/site-packages/click/core.py", line 837, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/maemual/Workspace/mkdocs/env/lib/python2.7/site-packages/click/core.py", line 464, in invoke
    return callback(*args, **kwargs)
  File "/Users/maemual/Workspace/mkdocs/mkdocs/cli.py", line 83, in serve_command
    livereload=livereload,
  File "/Users/maemual/Workspace/mkdocs/mkdocs/serve.py", line 77, in serve
    config = builder()
  File "/Users/maemual/Workspace/mkdocs/mkdocs/serve.py", line 73, in builder
    build(config, live_server=True)
  File "/Users/maemual/Workspace/mkdocs/mkdocs/build.py", line 306, in build
    build_pages(config)
  File "/Users/maemual/Workspace/mkdocs/mkdocs/build.py", line 252, in build_pages
    if not build_template('search.html', env, config, site_navigation):
  File "/Users/maemual/Workspace/mkdocs/mkdocs/build.py", line 166, in build_template
    output_content = template.render(context)
  File "/Users/maemual/Workspace/mkdocs/env/lib/python2.7/site-packages/jinja2/environment.py", line 969, in render
    return self.environment.handle_exception(exc_info, True)
  File "/Users/maemual/Workspace/mkdocs/env/lib/python2.7/site-packages/jinja2/environment.py", line 742, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/maemual/Workspace/mkdocs/mkdocs/themes/readthedocs/search.html", line 1, in top-level template code
    {% extends "base.html" %}
  File "/Users/maemual/Workspace/mkdocs/mkdocs/themes/readthedocs/base.html", line 62, in top-level template code
    <li>{% include "toc.html" %}<li>
  File "/Users/maemual/Workspace/mkdocs/mkdocs/themes/readthedocs/toc.html", line 3, in top-level template code
    <li><span>{{ nav_item.title }}</span></li>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 0: ordinal not in range(128)
```

I use a magic method to fix it which just add next 3 lines of code at the beginning of program.

```
import sys
reload(sys)
sys.setdefaultencoding('utf-8')
```

I know the use of sys.setdefaultencoding() has always been discouraged, and it has become a no-op in py3k. But i really don't know how to fix it correctly.

So, can you help me to fix it?



Discussion:
Ah, interesting. I hadn't considered unicode filenames, and I've certainly not tested that or seen anyone else try. I'll need to do some investigation on how to handle this, it's a new one to me :smile: 

I've marked this as the next release, but as we are close to the end of that development cycle I may have to push it to then next one. It depends how easily resolvable this is.

Does it work if you use Python 3? I've done some initial testing and found that I have the same error on Python 2 but everything works on Python 3

The update to [this comment](http://stackoverflow.com/a/19224331/866026) might be of interest regarding non-ascii file names:

> Note, os.walk returns Unicode strings when the given path is a Unicode string. A normal string path will return binary strings. This is the same behaviour as os.listdir (http://docs.python.org/2/library/os.html#os.listdir).

I can't say that I ever knew that. I suppose as long as you pass unicode strings to `os.walk` ([here](https://github.com/mkdocs/mkdocs/blob/master/mkdocs/config/config_options.py#L255)?) then all the returned file names will already be unicode.?? Which explains why this works fine on Python 3 but fails on Python 2. In Python 3 the strings would all be unicode by default, but not necessarily in Python 2.

Btw, my above comments are based on the assumption that the page title is being generated from the filename. What happens if the page title is explicitly defined in the [config](http://www.mkdocs.org/user-guide/configuration/#documentation-layout)? I agree it should work either way, but at least that might be a workaround till this gets fixed.

It occurs to me that the easy fix may be to force all strings loaded by the YAML config file to be unicode. See https://stackoverflow.com/questions/2890146/how-to-force-pyyaml-to-load-strings-as-unicode-objects. That is, after all, what is already happening in Python 3.

Interesting, I thought pyYAML was already loading everything as Unicode.

> Interesting, I thought pyYAML was already loading everything as Unicode.

Nope, at least not for PY2.  It only returns as unicode for strings that have non-ascii strings as I recall.  You can override it very similar to OrderedDict was done until we removed it.

Sounds like that would be a good idea, to make things consistent.


Resolving patch:
diff --git a/mkdocs/config/config_options.py b/mkdocs/config/config_options.py
--- a/mkdocs/config/config_options.py
+++ b/mkdocs/config/config_options.py
@@ -296,13 +296,16 @@ def run_validation(self, value):
         # TODO: Remove in 1.0
         config_types = set(type(l) for l in value)
 
-        if config_types.issubset(set([str, dict, ])):
+        if config_types.issubset(set([six.text_type, dict, ])):
             return value
 
-        if config_types.issubset(set([str, list, ])):
+        if config_types.issubset(set([six.text_type, list, ])):
             return legacy.pages_compat_shim(value)
 
-        raise ValidationError("Invalid pages config.")
+        raise ValidationError("Invalid pages config. {0} {1}".format(
+            config_types,
+            set([six.text_type, dict, ])
+        ))
 
     def post_validation(self, config, key_name):
 
diff --git a/mkdocs/config/defaults.py b/mkdocs/config/defaults.py
--- a/mkdocs/config/defaults.py
+++ b/mkdocs/config/defaults.py
@@ -1,3 +1,7 @@
+from __future__ import unicode_literals
+
+import six
+
 from mkdocs import utils
 from mkdocs.config import config_options
 
@@ -12,10 +16,10 @@
 DEFAULT_SCHEMA = (
 
     # Reserved for internal use, stores the mkdocs.yml config file.
-    ('config_file_path', config_options.Type(str)),
+    ('config_file_path', config_options.Type(six.string_types)),
 
     # The title to use for the documentation
-    ('site_name', config_options.Type(str, required=True)),
+    ('site_name', config_options.Type(six.string_types, required=True)),
 
     # Defines the structure of the navigation and which markdown files are
     # included in the build.
@@ -26,12 +30,12 @@
 
     # A description for the documentation project that will be added to the
     # HTML meta tags.
-    ('site_description', config_options.Type(str)),
+    ('site_description', config_options.Type(six.string_types)),
     # The name of the author to add to the HTML meta tags
-    ('site_author', config_options.Type(str)),
+    ('site_author', config_options.Type(six.string_types)),
 
     # The path to the favicon for a site
-    ('site_favicon', config_options.Type(str)),
+    ('site_favicon', config_options.Type(six.string_types)),
 
     # The MkDocs theme for the documentation.
     ('theme', config_options.Theme(default='mkdocs')),
@@ -47,14 +51,15 @@
     ('theme_dir', config_options.ThemeDir(exists=True)),
 
     # A copyright notice to add to the footer of documentation.
-    ('copyright', config_options.Type(str)),
+    ('copyright', config_options.Type(six.string_types)),
 
     # set of values for Google analytics containing the account IO and domain,
     # this should look like, ['UA-27795084-5', 'mkdocs.org']
     ('google_analytics', config_options.Type(list, length=2)),
 
     # The address on which to serve the live reloading docs server.
-    ('dev_addr', config_options.Type(str, default='127.0.0.1:8000')),
+    ('dev_addr', config_options.Type(
+        six.string_types, default='127.0.0.1:8000')),
 
     # If `True`, use `<page_name>/index.hmtl` style files with hyperlinks to
     # the directory.If `False`, use `<page_name>.html style file with
@@ -70,7 +75,7 @@
     # A name to use for the link to the project source repo.
     # Default, If repo_url is unset then None, otherwise
     # "GitHub" or "Bitbucket" for known url or Hostname for unknown urls.
-    ('repo_name', config_options.Type(str)),
+    ('repo_name', config_options.Type(six.string_types)),
 
     # Specify which css or javascript files from the docs directory should be
     # additionally included in the site. Default, List of all .css and .js
@@ -109,5 +114,6 @@
     ('extra', config_options.Type(dict, default={})),
 
     # the remote branch to commit to when using gh-deploy
-    ('remote_branch', config_options.Type(str, default='gh-pages')),
+    ('remote_branch', config_options.Type(
+        six.string_types, default='gh-pages')),
 )
diff --git a/mkdocs/nav.py b/mkdocs/nav.py
--- a/mkdocs/nav.py
+++ b/mkdocs/nav.py
@@ -10,6 +10,8 @@
 import logging
 import os
 
+import six
+
 from mkdocs import utils, exceptions
 
 log = logging.getLogger(__name__)
@@ -211,7 +213,7 @@ def _path_to_page(path, title, url_context, use_directory_urls):
 
 def _follow(config_line, url_context, use_dir_urls, header=None, title=None):
 
-    if isinstance(config_line, str):
+    if isinstance(config_line, six.string_types):
         path = os.path.normpath(config_line)
         page = _path_to_page(path, title, url_context, use_dir_urls)
 
@@ -237,7 +239,7 @@ def _follow(config_line, url_context, use_dir_urls, header=None, title=None):
 
     next_cat_or_title, subpages_or_path = next(iter(config_line.items()))
 
-    if isinstance(subpages_or_path, str):
+    if isinstance(subpages_or_path, six.string_types):
         path = subpages_or_path
         for sub in _follow(path, url_context, use_dir_urls, header=header, title=next_cat_or_title):
             yield sub


Test patch:
diff --git a/mkdocs/tests/config/config_options_tests.py b/mkdocs/tests/config/config_options_tests.py
--- a/mkdocs/tests/config/config_options_tests.py
+++ b/mkdocs/tests/config/config_options_tests.py
@@ -1,6 +1,10 @@
+from __future__ import unicode_literals
+
 import os
 import unittest
 
+import six
+
 from mkdocs import utils
 from mkdocs.config import config_options
 
@@ -45,7 +49,7 @@ class TypeTest(unittest.TestCase):
 
     def test_single_type(self):
 
-        option = config_options.Type(str)
+        option = config_options.Type(six.string_types)
         value = option.validate("Testing")
         self.assertEqual(value, "Testing")
 
@@ -62,7 +66,7 @@ def test_multiple_types(self):
                           option.validate, {'a': 1})
 
     def test_length(self):
-        option = config_options.Type(str, length=7)
+        option = config_options.Type(six.string_types, length=7)
 
         value = option.validate("Testing")
         self.assertEqual(value, "Testing")
diff --git a/mkdocs/tests/integration.py b/mkdocs/tests/integration.py
--- a/mkdocs/tests/integration.py
+++ b/mkdocs/tests/integration.py
@@ -14,79 +14,39 @@
     - Build documentation other than just MkDocs as it is relatively simple.
 """
 
-from __future__ import print_function
+from __future__ import print_function, unicode_literals
 
 import click
-import logging
 import os
-import sys
+import subprocess
 
-from mkdocs import cli, config, build, utils
+from mkdocs import utils
 
 DIR = os.path.dirname(__file__)
 MKDOCS_CONFIG = os.path.abspath(os.path.join(DIR, '../../mkdocs.yml'))
 MKDOCS_THEMES = utils.get_theme_names()
-
-
-def silence_logging(is_verbose=False):
-    '''When a --verbose flag is passed, increase the verbosity of mkdocs'''
-
-    logger = logging.getLogger('mkdocs')
-    logger.setLevel(logging.ERROR)
-
-
-def run_build(theme_name, output, config_file, quiet):
-    """
-    Given a theme name and output directory use the configuration
-    for the MkDocs documentation and overwrite the site_dir and
-    theme. If no output is provided, serve the documentation on
-    each theme, one at a time.
-    """
-
-    options = {
-        'theme': theme_name,
-    }
-
-    if config_file is None:
-        config_file = open(MKDOCS_CONFIG, 'rb')
-        if not quiet:
-            print("Using config: {0}".format(config_file.name))
-
-    if not os.path.exists(output):
-        os.makedirs(output)
-    options['site_dir'] = os.path.join(output, theme_name)
-
-    if not quiet:
-        print("Building {0}".format(theme_name))
-
-    try:
-        conf = config.load_config(config_file=config_file, **options)
-        config_file.close()
-        build.build(conf)
-        build.build(conf, dump_json=True)
-    except Exception:
-        print("Error building: {0}".format(theme_name), file=sys.stderr)
-        raise
+TEST_PROJECTS = os.path.abspath(os.path.join(DIR, 'integration'))
 
 
 @click.command()
 @click.option('--output',
               help="The output directory to use when building themes",
-              type=click.Path(file_okay=False, writable=True))
-@click.option('--config-file',
-              help="The MkDocs project config to use.",
-              type=click.File('rb'))
-@click.option('--quiet', is_flag=True)
-def main(output=None, config_file=None, quiet=False):
-
-    if quiet:
-        silence_logging()
-    else:
-        cli.configure_logging()
+              type=click.Path(file_okay=False, writable=True),
+              required=True)
+def main(output=None):
 
     for theme in sorted(MKDOCS_THEMES):
+        project_dir = os.path.dirname(MKDOCS_CONFIG)
+        out = os.path.join(output, theme)
+        command = ['mkdocs', 'build', '--site-dir', out, '--theme', theme]
+        subprocess.check_call(command, cwd=project_dir)
+
+    for project in os.listdir(TEST_PROJECTS):
 
-        run_build(theme, output, config_file, quiet)
+        project_dir = os.path.join(TEST_PROJECTS, project)
+        out = os.path.join(output, project)
+        command = ['mkdocs', 'build', '--site-dir', out]
+        subprocess.check_call(command, cwd=project_dir)
 
     print("The theme builds are available in {0}".format(output))
 
diff --git a/mkdocs/tests/integration/unicode/docs/index.md b/mkdocs/tests/integration/unicode/docs/index.md
new file mode 100644
--- /dev/null
+++ b/mkdocs/tests/integration/unicode/docs/index.md
@@ -0,0 +1,17 @@
+# Welcome to MkDocs
+
+For full documentation visit [mkdocs.org](http://mkdocs.org).
+
+## Commands
+
+* `mkdocs new [dir-name]` - Create a new project.
+* `mkdocs serve` - Start the live-reloading docs server.
+* `mkdocs build` - Build the documentation site.
+* `mkdocs help` - Print this help message.
+
+## Project layout
+
+    mkdocs.yml    # The configuration file.
+    docs/
+        index.md  # The documentation homepage.
+        ...       # Other markdown pages, images and other files.
diff --git "a/mkdocs/tests/integration/unicode/docs/\346\265\213\350\257\225.md" "b/mkdocs/tests/integration/unicode/docs/\346\265\213\350\257\225.md"
new file mode 100644
--- /dev/null
+++ "b/mkdocs/tests/integration/unicode/docs/\346\265\213\350\257\225.md"
@@ -0,0 +1,34 @@
+＃歡迎MkDocs
+
+對於完整的文檔訪問[ mkdocs.org ] （ http://mkdocs.org ） 。
+
+##命令
+
+*` mkdocs新的[ DIR -名稱] ` - 創建一個新的項目。
+*` mkdocs serve` - 啟動Live -重裝文檔服務器​​。
+*` mkdocs build` - 構建文檔站點。
+*` mkdocs help` - 打印此幫助信息。
+
+##項目佈局
+
+    mkdocs.yml ＃配置文件。
+    文檔/
+        index.md ＃文檔主頁。
+        ... ＃其他降價網頁，圖片和其他文件。
+#Huānyíng MkDocs
+
+duìyú wánzhěng de wéndàng fǎngwèn [mkdocs.Org] (http://Mkdocs.Org).
+
+##Mìnglìng
+
+*`mkdocs xīn de [DIR -míngchēng] ` - chuàngjiàn yīgè xīn de xiàngmù.
+*`Mkdocs serve` - qǐdòng Live -zhòng zhuāng wéndàng fúwùqì​​.
+*`Mkdocs build` - gòujiàn wéndàng zhàn diǎn.
+*`Mkdocs help` - dǎyìn cǐ bāngzhù xìnxī.
+
+##Xiàngmù bùjú
+
+    mkdocs.Yml#pèizhì wénjiàn.
+    Wéndàng/
+        index.Md#wéndàng zhǔyè.
+        ... #Qítā jiàngjià wǎngyè, túpiàn hé qítā wénjiàn.
diff --git a/mkdocs/tests/integration/unicode/mkdocs.yml b/mkdocs/tests/integration/unicode/mkdocs.yml
new file mode 100644
--- /dev/null
+++ b/mkdocs/tests/integration/unicode/mkdocs.yml
@@ -0,0 +1 @@
+site_name: My Docs