Repository: mkdocs/mkdocs

Issue:
Nested dot folders copied upon build
Sometimes it's necessary to include files in a theme which MkDocs should conditionally include in the templates, but not copy to the `site` directory when the documentation is built. As discussed in #1980, MkDocs will ignore files that start with a `.`, but this only applies to top-level directories. It would be great if MkDocs could exclude nested dot directories from the build.

A concrete example: the next version of the Material theme will bundle the FontAwesome files as SVGs for inlining during build time, but MkDocs copies all of them to the `site` directory. The folder structure is:

```
assets/images/icons/fontawesome/...
```

The current workaround is to add a `*.html` to the files bundles with the theme, resulting in a `*.svg.html` extension for each of the 1.500 icons. While this works, it's semantically incorrect.

If MkDocs would ignore nested dot directories, we could just use:

```
assets/images/icons/.fontawesome/...
```

As pointed out by @waylan, the code in question is here:

https://github.com/mkdocs/mkdocs/blob/2fca717794c0c2e581b8ba17149bc292edeb83e4/mkdocs/structure/files.py#L67-L69

I'm not proficient in Python, but since this looks like a glob implementation, I would guess that `**/.*` would solve the problem


Discussion:
> since this looks like a glob implementation, I would guess that `**/.*` would solve the problem

That is correct on both counts. At least I'm assuming `**/.*` would work, although I fear it may be more complicated than that.
This will not work like you expect. I will do a write up. Give me a minute.
@squidfunk it isn't really a glob pattern. It uses Python's fnmatch which is file name centric. Fnmatch doesn't really consider directory slashes:

```py
>>> fnmatch.fnmatch('.test', '.*')
True
>>> fnmatch.fnmatch('.test/something/else', '.*')
True
```

But since we are comparing directories, this breaks:

```py
>>> fnmatch.fnmatch('test/something/.else', '.*')
False
```

Fnmatch doesn't really look at `**`. That's not a thing for Python's fnmatch as `**` is for directories, but maybe this is okay if all paths have their slashes normalized to `/`:

```py
>>> fnmatch.fnmatch('test/something/.else', '*/.*')
True
```

But will fail if not:

```py
>>> fnmatch.fnmatch('test\\something\\.else', '*/.*')
False
```

But I think maybe that all gets normalized based on the OS, so maybe you are okay, but if not, you'd have to account for that:

```py
>>> fnmatch.fnmatch('test/something/.else', '*[/\\].*')
True
```

And maybe that is good enough.

To really use real glob patterns, you need something like a globmatch. I've actually written a library that does that, but maybe you don't really need something like that if the above is sufficient.

```py
from wcmatch import glob
>>> glob.globmatch('test/something/.else', '**/.*', flags=glob.GLOBSTAR)
True
```

Thanks for the rundown @facelessuser. I think any solution that ignores nested dot-directories would be sufficient 😊
Agreed, what is suggested by @waylan may work just fine in this case, depending on how the paths actually get normalized on various OS, it just needs to be tested. But since it is fnmatch `**` is no different than just using `*`.

If `*/.*` isn't sufficient, you could probably just `basename(path)` and compare in the filter function instead of pulling in another dependency (like a library with `globmatch`). I would only suggest that if you needed more complex glob logic. But it looks like they just want to really look at the file name. At least from my quick look.
Yeah, it looks like Windows will normalize things so that `fnmatch.fnmatch('test\\.test', '*/.*')` gets normalized and works fine. But you no longer will match things like `.dotfile` as now it requires the file to lead with `/`.

It may just be best to change filter to use `basename`:

```py
        def filter(name):
            patterns = ['.*', '*.py', '*.pyc', '*.html', '*readme*', 'mkdocs_theme.yml']
            patterns.extend('*{}'.format(x) for x in utils.markdown_extensions)
            patterns.extend(config['theme'].static_templates)
            for pattern in patterns:
                if fnmatch.fnmatch(os.path.basename(name).lower(), pattern):
                    return False
            return True
```
> ```python
> >>> fnmatch.fnmatch('test\\something\\.else', '*/.*')
> False
> ```
> 
> 
> But I think maybe that all gets normalized based on the OS, so maybe you are okay


I just checked and that works correctly on Windows:

```python
>>> fnmatch.fnmatch('test\\something\\.else', '*/.*')
True
>>> fnmatch.fnmatch('test/something/.else', '*/.*')
True
```



So the real question is, do you also get files that have no parent directories?

```py
>>> fnmatch.fnmatch('.dotfile', '*/.*')
False
```
Or can we expect they will always have a leading `/`?

If not, I'd use `basename` to match just what you actually care about.
Using `basename` only works on the filename, but we also want to match dir names at any nesting level.

I think the solution is to use two patterns: `'.*', '*/.*'`. The first pattern checks the root level segment  and the second checks all nested segments of a path.
Ah, so it's looking at parent folders with dots too, then two patterns sounds like the way to go. Yeah, that's the simplest solution I think.

Resolving patch:
diff --git a/mkdocs/structure/files.py b/mkdocs/structure/files.py
--- a/mkdocs/structure/files.py
+++ b/mkdocs/structure/files.py
@@ -64,7 +64,8 @@ def css_files(self):
     def add_files_from_theme(self, env, config):
         """ Retrieve static files from Jinja environment and add to collection. """
         def filter(name):
-            patterns = ['.*', '*.py', '*.pyc', '*.html', '*readme*', 'mkdocs_theme.yml']
+            # '.*' filters dot files/dirs at root level whereas '*/.*' filters nested levels
+            patterns = ['.*', '*/.*', '*.py', '*.pyc', '*.html', '*readme*', 'mkdocs_theme.yml']
             patterns.extend('*{}'.format(x) for x in utils.markdown_extensions)
             patterns.extend(config['theme'].static_templates)
             for pattern in patterns:


Test patch:
diff --git a/mkdocs/tests/structure/file_tests.py b/mkdocs/tests/structure/file_tests.py
--- a/mkdocs/tests/structure/file_tests.py
+++ b/mkdocs/tests/structure/file_tests.py
@@ -323,7 +323,11 @@ def test_files(self):
         'favicon.ico',
         'style.css',
         'foo.md',
-        'README'
+        'README',
+        '.ignore.txt',
+        '.ignore/file.txt',
+        'foo/.ignore.txt',
+        'foo/.ignore/file.txt'
     ])
     def test_add_files_from_theme(self, tdir, ddir):
         config = load_config(docs_dir=ddir, theme={'name': None, 'custom_dir': tdir})