Repository: mkdocs/mkdocs

Issue:
Pages encoded UTF-8 with BOM not rendered correctly
Hello,

I have a simple index markdown file:

```
# Project

## Overview
```

And it fails to render correctly in any theme.  

**Cinder**
![keqxxqn - imgur](https://cloud.githubusercontent.com/assets/8150232/24431970/c82923c6-1427-11e7-8ae9-12490dbb9dae.png)

**ReadTheDocs**
![amsuaqe - imgur](https://cloud.githubusercontent.com/assets/8150232/24431966/c28d11f2-1427-11e7-9aa3-5b1718bdf11e.png)

Am I doing something wrong?  If I change it to a level 2 heading, the same thing happens. 

I can workaround it by placing a blank line at the start of the file, but then the heading appears too low:

![6xtusui - imgur](https://cloud.githubusercontent.com/assets/8150232/24431972/ceb97c68-1427-11e7-8714-b5c3b5546645.png)

Thanks.


Discussion:
Have you tried including a blank line between your headings? While not strictly necessary in all situations, failing to do so is generally bad form and can result in weird edge cases.
Actually, there is a blank line - that was a typo in writing the issue.  It is the same either way.
It seems to only be occurring on the first page (index.md) in this project, although I seem to recall it doing it on multiple pages in another project (could be wrong).
Same Problem here :( For every document in the repository.

![mkdocs bug](https://cloud.githubusercontent.com/assets/19962677/26756912/dcb89394-48ac-11e7-8ee8-6a152b33573e.PNG)

This is the source of the file:
```markdown
### Bug Template 

'''
[Short description of problem here]

**Reproduction Steps:**

1. [First Step]
2. [Second Step]
3. [Other Steps...]
...
```

Rendered HTML is 

```html
<article class="md-content__inner md-typeset">



  <h1>Bug template</h1>

<p>﻿### Bug Template </p>
```

Tested with default theme and with material in mkdocs, version 0.16.3, running on Anaconda with Python 2.7.2 on Windows 10.
Can you post your mkdocs.yml file?  Maybe this is dependent on certain extensions?  I copied and pasted your example, but the syntax is weird in your example, and it doesn't look to be the full source of the image you posted.  It isn't using backticks for fences, maybe that was a mistake when you posted the example?  When I change the code start and end to ` ``` `, it works fine for me.  But I don't think your source example is right.  Regardless, maybe it has nothing to do with the source and is related to extensions?

![screen shot 2017-06-03 at 4 46 11 pm](https://cloud.githubusercontent.com/assets/1055125/26757542/9b558856-487c-11e7-8626-71cdcb1a92d9.png)

I also can't reproduce the opening post's example in readthedocs. I tired a number of extension combinations and even removed all extensions, and I am not seeing that behavior.
Hmm, sorry, I did only post the header of the document, as I thought the header declaration would suffice. This is the complete document: 

```markdown
### Bug Template 

'''
[Short description of problem here]

**Reproduction Steps:**

1. [First Step]
2. [Second Step]
3. [Other Steps...]

**Expected behavior/content:**

[Describe expected behavior here]

**Observed behavior/content:**

[Describe observed behavior here]

**Screenshots and GIFs**

[Screenshots and GIFs which follow reproduction steps to demonstrate the problem]

**Additional information:**

* Problem started happening recently, didn't happen in an older version: [Yes/No]
* Problem can be reliably reproduced, doesn't happen randomly: [Yes/No]
* Problem happens with all files and projects, not only some files or projects: [Yes/No]
...
'''
```
Note that I have replaced the backticks in the document with `'` so I can paste it correctly ;) 

But this document was only the easiest example to use to provide the error description. The behaviour ocurrs for every document in our repository, regardless of the heading level. Every first heading is rendered as a html header (`<h1>`) following a paragraph containing the actual source heading (`<p>### ...`). The repository is big and contains about 60 - 100 markdown files in various folders and sub folders, so I can't provide a comprehensive example :(

The `yml` file is very basic. I have not created any pages mappings because of the mass of documents, I need to generate mkdocs just for every document in the repository with the given folder structure so I leave that unconfigured. 

These are the settings currently configured:

```yml
site_name: Docs
site_url: url_to_site
repo_url: repo_url
repo_name: Docs
site_description: Docs
site_author: company_name
copyright: company_name
use_directory_urls: false
theme: 'material'
```
Note that I also used the default theme (which can't be used because of the mass of doucments, material works better for that) with the same result.

I will try to generate the pages mapping automatically, perhaps this changes the result. Will keep you posted about that.
I have tested a bit more and could identify that the problem lies with the file encoding. If the file is encoded in UTF-8 BOM (Windows 😞) then the rendering issue ocurrs. If you use *proper* UTF-8 there the files are generated as expected.
Well, that's good news.  At least the issue has been clearly identified.  As far as I know, MkDocs doesn't allow you to specify encoding, so as long as you use *proper* UTF-8, you should be fine.  Maybe this should be more clearly stated in the documentation if it isn't already.
MkDocs could read files with encoding `utf-8-sig` which should strip out BOM **if** it is found, and if it is not found, it would read the file as a normal `utf-8`.  I personally think this would end a lot of confusion in this regard.  Issues like this are always unclear to debug, so just using an encoding that handles with and without BOM may be the easiest path forward to avoid issues like this in the future.

I also think it is perfectly reasonable for MkDocs to just clarify that they only accept `utf-8` without BOM. But I suspect people will run into this again and again and have to have it explained to them; it may just be easier to use `utf-8-sig` and not worry about it.
> If the file is encoded in UTF-8 BOM (Windows :disappointed:) then the rendering issue ocurrs. If you use proper UTF-8 there the files are generated as expected.

That makes sense. Can't believe I didn't think of asking about encoding before.

> I also think it is perfectly reasonable for MkDocs to just clarify that they only accept `utf-8` without BOM. But I suspect people will run into this again and again and have to have it explained to them; it may just be easier to use `utf-8-sig` and not worry about.

You may have a point. I suspect this has not been done given the following in the [Python docs][1]: 

> To increase the reliability with which a `UTF-8` encoding can be detected, Microsoft invented a variant of `UTF-8` (that Python 2.5 calls "`utf-8-sig`") for its Notepad program... In `UTF-8`, the use of the BOM is discouraged and should generally be avoided.

First, I can't imagine anyone actually using Notepad to edit their files. Is there any other editors which actually do that by default? I've never encountered one.

That said, using `utf-8-sig` would just 'do the right thing' whether there is a BOM or not. In fact, a strict reading of the above is that writing the BOM to `utf-8` is discouraged, not accounting for it when reading. As the docs mention:

> On encoding the `utf-8-sig` codec will write `0xe`f, `0xbb`, `0xbf` as the first three bytes to the file. On decoding `utf-8-sig` will skip those three bytes if they appear as the first three bytes in the file. 

So we could read using `utf-8-sig` and then write only with `utf-8`.  That way, we can just ignore any BOM.

[1]: https://docs.python.org/2/library/codecs.html#encodings-and-unicode
> So we could read using utf-8-sig and then write only with utf-8. That way, we can just ignore any BOM.

Yes, that is what I was suggesting, I should have emphasized read, but that was my intention with this suggestion. 
> First, I can't imagine anyone actually using Notepad to edit their files. Is there any other editors which actually do that by default? I've never encountered one.

Well, I used Powershell's `Out-File -Encoding utf8` to modify links in the markdown before calling mkdocs. As it seems Powershell's default setting for utf8 is with BOM ;)
> Well, I used Powershell's Out-File -Encoding utf8 to modify links in the markdown before calling mkdocs. As it seems Powershell's default setting for utf8 is with BOM ;)

Oh course that's what Windows is doing :man_facepalming:.  That's such a Windows thing to do.
> First, I can't imagine anyone actually using Notepad to edit their files. Is there any other editors which actually do that by default? I've never encountered one.

Can't remember what editor i was using (usually.vscode or atom) but do occasionally make minor edits in notepad cause it opens quicker :)  Not sure if autocrlf was interfering or not...

Resolving patch:
diff --git a/mkdocs/nav.py b/mkdocs/nav.py
--- a/mkdocs/nav.py
+++ b/mkdocs/nav.py
@@ -248,7 +248,7 @@ def read_source(self, config):
             'page_read_source', None, config=config, page=self)
         if source is None:
             try:
-                with io.open(self.abs_input_path, 'r', encoding='utf-8') as f:
+                with io.open(self.abs_input_path, 'r', encoding='utf-8-sig') as f:
                     source = f.read()
             except IOError:
                 log.error('File not found: %s', self.abs_input_path)


Test patch:
diff --git a/mkdocs/tests/build_tests.py b/mkdocs/tests/build_tests.py
--- a/mkdocs/tests/build_tests.py
+++ b/mkdocs/tests/build_tests.py
@@ -7,6 +7,7 @@
 import tempfile
 import unittest
 import mock
+import io
 
 try:
     from itertools import izip as zip
@@ -464,3 +465,32 @@ def test_extra_context(self):
         context = build.get_context(mock.Mock(), cfg)
 
         self.assertEqual(context['config']['extra']['a'], 1)
+
+    def test_BOM(self):
+        docs_dir = tempfile.mkdtemp()
+        site_dir = tempfile.mkdtemp()
+        try:
+            # Create an UTF-8 Encoded file with BOM (as Micorsoft editors do). See #1186.
+            f = io.open(os.path.join(docs_dir, 'index.md'), 'w', encoding='utf-8-sig')
+            f.write('# An UTF-8 encoded file with a BOM')
+            f.close()
+
+            cfg = load_config(
+                docs_dir=docs_dir,
+                site_dir=site_dir
+            )
+            build.build(cfg)
+
+            # Verify that the file was generated properly.
+            # If the BOM is not removed, Markdown will return:
+            # `<p>\ufeff# An UTF-8 encoded file with a BOM</p>`.
+            f = io.open(os.path.join(site_dir, 'index.html'), 'r', encoding='utf-8')
+            output = f.read()
+            f.close()
+            self.assertTrue(
+                '<h1 id="an-utf-8-encoded-file-with-a-bom">An UTF-8 encoded file with a BOM</h1>' in output
+            )
+
+        finally:
+            shutil.rmtree(docs_dir)
+            shutil.rmtree(site_dir)