Repository: mkdocs/mkdocs

Issue:
Unexpected template and source code files deployed to site_dir
After upgrading to 0.15.0, the following files get deployed to the `site_dir` after build:

```
site/__init__.py
site/__init__.pyc
site/base.html
site/content.html
site/nav.html
site/toc.html
```

From the `gh-pages` branch, you can see the last clean commit is:

https://github.com/mkdocs/mkdocs/tree/6b7615eee095c8ccd47a106ca29f2f12314d86f5

The most recent commit has the above files included:

https://github.com/mkdocs/mkdocs/tree/ffe467ab2352ec3a0f690024bd908b7dea0443fc



Discussion:
Good spot. I don't think this is actually a problem, so it is a low priority issue but we should try to resolve it.

The tricky thing to realise is, how do we determine which assets should and shouldn't be copied from a theme?

> I don't think this is actually a problem

This _is_ a problem under some specific (but not very unusual) conditions, e.g.:
- have directory `my_project/` in your `PYTHONPATH`
- use the Python 3 standard library package `html`, or any of its subpackages (e.g. `html.entities`)
- use `mkdocs` to generate docs in directory `my_project/html/`

The gratuitous `my_project/html/__init__.py` will make the `html/` directory a Python package that masks the standard library package of the same name, leading to confusing error messages like

```
 ImportError: No module named html.entities
```

Okay, wow. That is an unfortunate set of coincidences. In that case, yes it is a problem.

A simple fix to never copy Python files from a theme's source would probably do the job.

All media files (non markdown and template files) are copied using the [mkdocs.utils.copy_media_files](https://github.com/mkdocs/mkdocs/blob/master/mkdocs/utils/__init__.py#L131) function. We can easily add a line to have it also exclude python files, but I'm thinking we only want to exclude python files from the theme dir, not from the docs_dir. After all, a user could have a python source file they want to host for some reason (presumably with a download link pointing to it in their docs). However, in the code, [copying the themes](https://github.com/mkdocs/mkdocs/blob/master/mkdocs/commands/build.py#L281) and [copying the media files in the docs_dir](https://github.com/mkdocs/mkdocs/blob/master/mkdocs/commands/build.py#L281) is not really differentiated. We could add an `exclude` keyword to `mkdocs.utils.copy_media_files` which accepts a list of file extensions, but I'm not sure we want to go that way.

Yeah, ideally we would refactor this a bit and make it smarter. Maybe we create a class that represents a media directory, then a subclass of it could represent a theme - initially that would just exclude Python files, but later it might do more.

... but I am just thinking "out loud", so I've not thought that through fully.

Right. My thoughts as well. We could do a quick fix (`exclude` keyword) or we could refactor it and make it right. But then how do we refactor it...

I confirm that it can even be worse than masking the `html` standard package, see #878 for example.

It occurs to me that we could require themes to actually be in a subdirectory of the directory which contains the `__init__.py` file (perhaps called `theme`; open to suggestions for alternate/better names). Then we could do `os.path.join(pkgutil.get_loader(theme.module_name).filename, 'theme')` to get the dir of the actual theme and no extra files would be in the actual theme.

So a theme would look like this:

```
- readthedocs/
    - __init__.py
    - README.md
    - theme/
        - css/
        - js/
        - img/
        - base.html
        - ...
```

Of course, to do that, all the existing themes would need to be updated and the change would be backward incompatible. However, if such a change were to be made, it should be done now before more third party themes are developed. Note that this would not effect the `theme_dir` as that is handled separately. Therefore this would only effect themes, not projects - although users would need to update there themes at the same time they update MkDocs. We could issue a warning for a few versions while continuing to support the old scheme. Barring any objection, I'll see what I can come up with.

Or maybe at least check that the target directory is not already in the python path and display a warning about that?

Hmm, interesting. I just noticed that the OP also reported that the template files (`base.html`, etc) are also being copied to the `site_dir`, not just the Python files. I think its clear the we need a different code path for templates than the `doc_dir`. Simply putting the template in a sub-directory won't address that issue.


Resolving patch:
diff --git a/mkdocs/commands/build.py b/mkdocs/commands/build.py
--- a/mkdocs/commands/build.py
+++ b/mkdocs/commands/build.py
@@ -278,11 +278,13 @@ def build(config, live_server=False, dump_json=False, clean_site_dir=False):
         return
 
     # Reversed as we want to take the media files from the builtin theme
-    # and then from the custom theme_dir so the custom versions take take
+    # and then from the custom theme_dir so that the custom versions take
     # precedence.
     for theme_dir in reversed(config['theme_dir']):
         log.debug("Copying static assets from theme: %s", theme_dir)
-        utils.copy_media_files(theme_dir, config['site_dir'])
+        utils.copy_media_files(
+            theme_dir, config['site_dir'], exclude=['*.py', '*.pyc', '*.html']
+        )
 
     log.debug("Copying static assets from the docs dir.")
     utils.copy_media_files(config['docs_dir'], config['site_dir'])
diff --git a/mkdocs/utils/__init__.py b/mkdocs/utils/__init__.py
--- a/mkdocs/utils/__init__.py
+++ b/mkdocs/utils/__init__.py
@@ -16,6 +16,7 @@
 import shutil
 import sys
 import yaml
+import fnmatch
 
 from mkdocs import toc, exceptions
 
@@ -128,16 +129,23 @@ def clean_directory(directory):
             os.unlink(path)
 
 
-def copy_media_files(from_dir, to_dir):
+def copy_media_files(from_dir, to_dir, exclude=None):
     """
-    Recursively copy all files except markdown and HTML into another directory.
+    Recursively copy all files except markdown and exclude[ed] files into another directory.
+
+    `exclude` accepts a list of Unix shell-style wildcards (`['*.py', '*.pyc']`).
+    Note that `exclude` only operates on file names, not directories.
     """
     for (source_dir, dirnames, filenames) in os.walk(from_dir):
         relative_path = os.path.relpath(source_dir, from_dir)
         output_dir = os.path.normpath(os.path.join(to_dir, relative_path))
 
-        # Filter filenames starting with a '.'
-        filenames = [f for f in filenames if not f.startswith('.')]
+        # Filter file names using Unix pattern matching
+        # Always filter file names starting with a '.'
+        exclude_patterns = ['.*']
+        exclude_patterns.extend(exclude or [])
+        for pattern in exclude_patterns:
+            filenames = [f for f in filenames if not fnmatch.fnmatch(f, pattern)]
 
         # Filter the dirnames that start with a '.' and update the list in
         # place to prevent us walking these.


Test patch:
diff --git a/mkdocs/tests/build_tests.py b/mkdocs/tests/build_tests.py
--- a/mkdocs/tests/build_tests.py
+++ b/mkdocs/tests/build_tests.py
@@ -294,7 +294,7 @@ def test_copying_media(self):
         docs_dir = tempfile.mkdtemp()
         site_dir = tempfile.mkdtemp()
         try:
-            # Create a non-empty markdown file, image, dot file and dot directory.
+            # Create a non-empty markdown file, image, html file, dot file and dot directory.
             f = open(os.path.join(docs_dir, 'index.md'), 'w')
             f.write(dedent("""
                 page_title: custom title
@@ -309,6 +309,7 @@ def test_copying_media(self):
             """))
             f.close()
             open(os.path.join(docs_dir, 'img.jpg'), 'w').close()
+            open(os.path.join(docs_dir, 'example.html'), 'w').close()
             open(os.path.join(docs_dir, '.hidden'), 'w').close()
             os.mkdir(os.path.join(docs_dir, '.git'))
             open(os.path.join(docs_dir, '.git/hidden'), 'w').close()
@@ -322,12 +323,48 @@ def test_copying_media(self):
             # Verify only the markdown (coverted to html) and the image are copied.
             self.assertTrue(os.path.isfile(os.path.join(site_dir, 'index.html')))
             self.assertTrue(os.path.isfile(os.path.join(site_dir, 'img.jpg')))
+            self.assertTrue(os.path.isfile(os.path.join(site_dir, 'example.html')))
             self.assertFalse(os.path.isfile(os.path.join(site_dir, '.hidden')))
             self.assertFalse(os.path.isfile(os.path.join(site_dir, '.git/hidden')))
         finally:
             shutil.rmtree(docs_dir)
             shutil.rmtree(site_dir)
 
+    def test_copy_theme_files(self):
+        docs_dir = tempfile.mkdtemp()
+        site_dir = tempfile.mkdtemp()
+        try:
+            # Create a non-empty markdown file.
+            f = open(os.path.join(docs_dir, 'index.md'), 'w')
+            f.write(dedent("""
+                page_title: custom title
+
+                # Heading 1
+
+                This is some text.
+            """))
+            f.close()
+
+            cfg = load_config({
+                'docs_dir': docs_dir,
+                'site_dir': site_dir
+            })
+            build.build(cfg)
+
+            # Verify only theme media are copied, not templates or Python files.
+            self.assertTrue(os.path.isfile(os.path.join(site_dir, 'index.html')))
+            self.assertTrue(os.path.isdir(os.path.join(site_dir, 'js')))
+            self.assertTrue(os.path.isdir(os.path.join(site_dir, 'css')))
+            self.assertTrue(os.path.isdir(os.path.join(site_dir, 'img')))
+            self.assertFalse(os.path.isfile(os.path.join(site_dir, '__init__.py')))
+            self.assertFalse(os.path.isfile(os.path.join(site_dir, '__init__.pyc')))
+            self.assertFalse(os.path.isfile(os.path.join(site_dir, 'base.html')))
+            self.assertFalse(os.path.isfile(os.path.join(site_dir, 'content.html')))
+            self.assertFalse(os.path.isfile(os.path.join(site_dir, 'nav.html')))
+        finally:
+            shutil.rmtree(docs_dir)
+            shutil.rmtree(site_dir)
+
     def test_strict_mode_valid(self):
         pages = [
             'index.md',