internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: mkdocs/mkdocs

Issue:
allow links to be followed in copy_media_files
I want to set up a small documentation server that checks out all our projects from a local
gitlab repo and link the _docs_ directories of each of them as sub folders of a separate _docs_
folder with its own index.md file and a page for each project in the mkdoc.yml like in this example:

``` yaml
site_name: Our Documentation Collection
pages:
  - Home: index.md
  - project1: project1/index.md
  - project2: project2/index.md  
```

Because _project1_ and _project2_ are logical links to the git clone's docs directories, media files
are not copied to _site_dir_ and thus e.g. images inside the projects not visible in the generated
html. 

To fix it I propose to add  ", followlinks=True" to os.walk in copy_media_files

``` python
def copy_media_files(from_dir, to_dir):
    """
    Recursively copy all files except markdown and HTML into another directory.
    """
    for (source_dir, dirnames, filenames) in os.walk(from_dir, followlinks=True):
```



Discussion:
Hmm, interesting. I wonder if this could cause any issues. I am not sure how I feel about this change, I will give it some thought. It does seem like quite a niche request given it hasn't been raised before.

I think I would be happy for this change to happen, I can't think of any problems with it and there have been no objections.


Resolving patch:
diff --git a/mkdocs/utils/__init__.py b/mkdocs/utils/__init__.py
--- a/mkdocs/utils/__init__.py
+++ b/mkdocs/utils/__init__.py
@@ -136,7 +136,7 @@ def copy_media_files(from_dir, to_dir, exclude=None):
     `exclude` accepts a list of Unix shell-style wildcards (`['*.py', '*.pyc']`).
     Note that `exclude` only operates on file names, not directories.
     """
-    for (source_dir, dirnames, filenames) in os.walk(from_dir):
+    for (source_dir, dirnames, filenames) in os.walk(from_dir, followlinks=True):
         relative_path = os.path.relpath(source_dir, from_dir)
         output_dir = os.path.normpath(os.path.join(to_dir, relative_path))
 


Test patch:
Repository: mkdocs/mkdocs

Issue:
Eliminate duplicate warnings
A few recent updates have added warnings for deprecated behavior (#921 & #947). However, the same warnings are issued once for every single page. The more pages in a project, the more warnings are issued. Each warning only needs to be printed to the console once per build. 

[This](http://stackoverflow.com/a/31953563/866026) may be a potential solution. Will explore when I have some time unless someone beats me to it.



Discussion:

Resolving patch:
diff --git a/mkdocs/commands/build.py b/mkdocs/commands/build.py
--- a/mkdocs/commands/build.py
+++ b/mkdocs/commands/build.py
@@ -16,7 +16,20 @@
 from mkdocs.relative_path_ext import RelativePathExtension
 import mkdocs
 
+
+class DuplicateFilter(object):
+    ''' Avoid logging duplicate messages. '''
+    def __init__(self):
+        self.msgs = set()
+
+    def filter(self, record):
+        rv = record.msg not in self.msgs
+        self.msgs.add(record.msg)
+        return rv
+
+
 log = logging.getLogger(__name__)
+log.addFilter(DuplicateFilter())
 
 
 def convert_markdown(markdown_source, config, site_navigation=None):


Test patch:
Repository: mkdocs/mkdocs

Issue:
Fix Python 3.7 and PEP 479 incompatibility
With [PEP 479](https://www.python.org/dev/peps/pep-0479/) implemented in Python 3.7, raised `StopIteration` is now translated into `RuntimeError`, which breaks `mkdocs build` command.

The recommended way to exit generators is to use `return` instead of raising `StopIteartion`: https://www.python.org/dev/peps/pep-0479/#id39


Discussion:

Resolving patch:
diff --git a/mkdocs/nav.py b/mkdocs/nav.py
--- a/mkdocs/nav.py
+++ b/mkdocs/nav.py
@@ -334,7 +334,7 @@ def _follow(config_line, url_context, config, header=None, title=None):
             header.children.append(page)
 
         yield page
-        raise StopIteration
+        return
 
     elif not isinstance(config_line, dict):
         msg = ("Line in 'page' config is of type {0}, dict or string "
@@ -347,7 +347,7 @@ def _follow(config_line, url_context, config, header=None, title=None):
             "config contains an invalid entry: {0}".format(config_line))
     elif len(config_line) == 0:
         log.warning("Ignoring empty line in the pages config.")
-        raise StopIteration
+        return
 
     next_cat_or_title, subpages_or_path = next(iter(config_line.items()))
 
@@ -355,7 +355,7 @@ def _follow(config_line, url_context, config, header=None, title=None):
         path = subpages_or_path
         for sub in _follow(path, url_context, config, header=header, title=next_cat_or_title):
             yield sub
-        raise StopIteration
+        return
 
     elif not isinstance(subpages_or_path, list):
         msg = ("Line in 'page' config is of type {0}, list or string "


Test patch:
Repository: mkdocs/mkdocs

Issue:
"Files not present in docs directory" message shown for external links
Hi!

### STR
1. `mkvirtualenv mkdocs -p python 3.6`
2. `pip install mkdocs==1.0b1`
3. Update [an existing project's mkdocs.yml](https://github.com/mozilla-neutrino/neutrino-dev/blob/cab8e9e0f50c83fbb5ae0f6fb0e98b286beef049/mkdocs.yml) to use `nav` and take advantage of external link support. eg:

```yaml
docs_dir: 'docs'
site_dir: 'build'
strict: true
theme:
  name: 'material'
# ...

nav:
  - Introduction: './index.md'
  # <snip>
  - v8 Documentation: 'https://neutrinojs.org/'
  - v7 Documentation: 'https://release-v7.neutrinojs.org/'
  - v6 Documentation: 'https://github.com/mozilla-neutrino/neutrino-dev/tree/release/v6/docs'
  - v5 Documentation: 'https://github.com/mozilla-neutrino/neutrino-dev/tree/release/v5/docs'
  - v4 Documentation: 'https://github.com/mozilla-neutrino/neutrino-dev/tree/release/v4/docs'
```

4. `mkdocs serve`

### Expected

Console output doesn't output warnings that give the impression something is broken.

### Actual

```
(mkdocs) neutrino-dev $ mkdocs serve
INFO    -  Building documentation...
INFO    -  Cleaning site directory
INFO    -  The following paths are included in the "nav" configuration, but do not exist in the docs directory:
  - https://neutrinojs.org/
  - https://release-v7.neutrinojs.org/
  - https://github.com/mozilla-neutrino/neutrino-dev/tree/release/v6/docs
  - https://github.com/mozilla-neutrino/neutrino-dev/tree/release/v5/docs
  - https://github.com/mozilla-neutrino/neutrino-dev/tree/release/v4/docs
[I 180723 17:15:03 server:292] Serving on http://127.0.0.1:8000
...
```

For links that are clearly external links (ie contain a full URL), it would be good to not output this message (even at log level INFO), since it's likely to be confusing for end users working on our documentation. (Perhaps log level DEBUG is the most appropriate?)

Many thanks :-)


Discussion:
You make a good point. DEBUG probably makes more sense. Thanks for the feedback.

Resolving patch:
diff --git a/mkdocs/structure/nav.py b/mkdocs/structure/nav.py
--- a/mkdocs/structure/nav.py
+++ b/mkdocs/structure/nav.py
@@ -126,8 +126,7 @@ def get_navigation(files, config):
     links = _get_by_type(items, Link)
     if links:
         # Assume all links are external.
-        # TODO: warn or error on internal links?
-        log.info(
+        log.debug(
             'The following paths are included in the "nav" configuration, '
             'but do not exist in the docs directory:\n  - {}'.format(
                 '\n  - '.join([link.url for link in links]))


Test patch:
Repository: mkdocs/mkdocs

Issue:
link of absolute path without domains will display a warning
If I wrote `[link](/api/file.pdf)` in docs, it would show a warning like this:

```
WARNING -  Documentation file 'forum.md' contains a link to 'api/file.pdf' which is not found in the documentation files.
```

The link works in  the generated docs. The only problem is the warning message.


Discussion:

Resolving patch:
diff --git a/mkdocs/structure/pages.py b/mkdocs/structure/pages.py
--- a/mkdocs/structure/pages.py
+++ b/mkdocs/structure/pages.py
@@ -214,7 +214,8 @@ def run(self, root):
     def path_to_url(self, url):
         scheme, netloc, path, params, query, fragment = urlparse(url)
 
-        if scheme or netloc or not path or AMP_SUBSTITUTE in url or '.' not in os.path.split(path)[-1]:
+        if (scheme or netloc or not path or url.startswith('/')
+                or AMP_SUBSTITUTE in url or '.' not in os.path.split(path)[-1]):
             # Ignore URLs unless they are a relative link to a source file.
             # AMP_SUBSTITUTE is used internally by Markdown only for email.
             # No '.' in the last part of a path indicates path does not point to a file.


Test patch:
diff --git a/mkdocs/tests/structure/page_tests.py b/mkdocs/tests/structure/page_tests.py
--- a/mkdocs/tests/structure/page_tests.py
+++ b/mkdocs/tests/structure/page_tests.py
@@ -772,6 +772,13 @@ def test_external_link(self):
             '<p><a href="http://example.com/index.md">external link</a></p>'
         )
 
+    @mock.patch('io.open', mock.mock_open(read_data='[absolute link](/path/to/file.md)'))
+    def test_absolute_link(self):
+        self.assertEqual(
+            self.get_rendered_result(['index.md']),
+            '<p><a href="/path/to/file.md">absolute link</a></p>'
+        )
+
     @mock.patch('io.open', mock.mock_open(read_data='<mail@example.com>'))
     def test_email_link(self):
         self.assertEqual(
Repository: mkdocs/mkdocs

Issue:
url filter gives the wrong path to local anchor links
When passed "#internal-anchor" on a page other than the top-level index, the `url` filter returns a relative path to the top-level index.

Like, if I have a page `about.md` and the page.url is `about/`; if I pass "#internal-anchor" to the `url` filter, I get `../#internal-anchor`.

The `url` filter should not modify those URLs that are internal to the current page.

(I suffer from this problem when using the material theme; in 3.0.4, in the "Skip to content" link, it passes a toc item's url to the `url` filter, which breaks it on every page except the top-level index.  HTMLProofer complains about the broken link.)


Discussion:

Resolving patch:
diff --git a/mkdocs/utils/__init__.py b/mkdocs/utils/__init__.py
--- a/mkdocs/utils/__init__.py
+++ b/mkdocs/utils/__init__.py
@@ -243,7 +243,7 @@ def normalize_url(path, page=None, base=''):
     path = path_to_url(path or '.')
     # Allow links to be fully qualified URL's
     parsed = urlparse(path)
-    if parsed.scheme or parsed.netloc or path.startswith('/'):
+    if parsed.scheme or parsed.netloc or path.startswith(('/', '#')):
         return path
 
     # We must be looking at a local path.


Test patch:
diff --git a/mkdocs/tests/utils/utils_tests.py b/mkdocs/tests/utils/utils_tests.py
--- a/mkdocs/tests/utils/utils_tests.py
+++ b/mkdocs/tests/utils/utils_tests.py
@@ -104,6 +104,11 @@ def test_create_media_urls(self):
                 'style.css?v=20180308c',
                 'style.css?v=20180308c',
                 '../style.css?v=20180308c'
+            ],
+            '#some_id': [
+                '#some_id',
+                '#some_id',
+                '#some_id'
             ]
         }
 
@@ -160,6 +165,11 @@ def test_create_media_urls_use_directory_urls(self):
                 'style.css?v=20180308c',
                 '../style.css?v=20180308c',
                 '../../style.css?v=20180308c'
+            ],
+            '#some_id': [
+                '#some_id',
+                '#some_id',
+                '#some_id'
             ]
         }
Repository: mkdocs/mkdocs

Issue:
Jinja2 2.10 security vulnerability reported by GitHub
Hi

I just got an alert on my github repo (where we use mkdocs to build our doc) for Jinja2 2.10:

- https://github.com/eclipse/openj9-docs/network/alert/buildenv/requirements.txt/Jinja2/open

Recommendation is to move to 2.10.1.

Are there any known issues with moving to this level?  Are there any plans to update the Mkdocs dependency list to this level? 

Thanks!


Discussion:
This is the first I've heard of it and your link appears to be broken. Could you provide a copy of the alert in a comment here?
With a quick search, I found [this report][1]:

> An issue was discovered in Jinja2 2.10. The from_string function is prone to Server Side Template Injection (SSTI) where it takes the "source" parameter as a template object, renders it, and then returns it. The attacker can exploit it with {{INJECTION COMMANDS}} in a URI.

Note that as MkDocs is a Static Site Generator, there is no template rendering server-side and there should be no way for an attacker to use this exploit with MkDocs. Additionally, we don't specify a specific required version of Jinja2. You are free to update your local copy of Jinja2 to a patched version without issue. That said, we should update our [minimum version][3] from 2.7.1 to a patched version of Jinja2. It appears that [version 2.10.1][2] was released 9 days ago to address this issue specifically.

@SueChaplain thanks for the report.

[2]: https://github.com/pallets/jinja/blob/master/CHANGES.rst#version-2101
[1]: https://nvd.nist.gov/vuln/detail/CVE-2019-8341#vulnCurrentDescriptionTitle
[3]: https://github.com/mkdocs/mkdocs/blob/master/requirements/project-min.txt#L2
>This is the first I've heard of it and your link appears to be broken. Could you provide a copy of the alert in a comment here?

Of course - the report generated by GitHub is private. Sorry! 

The CVE is actually this one: https://nvd.nist.gov/vuln/detail/CVE-2019-10906

*In Pallets Jinja before 2.10.1, str.format_map allows a sandbox escape.*

Not sure whether this would be an exposure in MkDocs or not?

As both issues are patched in 2.10.1, this seems like the appropriate course of action. Thanks for picking this up so quickly.
Thanks for the update @SueChaplain. The analysis is exactly the same: not a problem for a static site generator, but update your local version of Jinja2 is you are concerned.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -55,7 +55,7 @@ def get_packages(package):
     include_package_data=True,
     install_requires=[
         'click>=3.3',
-        'Jinja2>=2.7.1',
+        'Jinja2>=2.10.1',
         'livereload>=2.5.1',
         'lunr[languages]>=0.5.2',
         'Markdown>=2.3.1',


Test patch:
Repository: mkdocs/mkdocs

Issue:
Fix simple typo: seperate -> separate
# Issue Type

[x] Bug (Typo)

# Steps to Replicate

1. Examine mkdocs/commands/serve.py.
2. Search for `seperate`.

# Expected Behaviour

1. Should read `separate`.




Discussion:

Resolving patch:
diff --git a/mkdocs/commands/serve.py b/mkdocs/commands/serve.py
--- a/mkdocs/commands/serve.py
+++ b/mkdocs/commands/serve.py
@@ -80,7 +80,7 @@ def get_web_handlers(self, script):
 
 def _static_server(host, port, site_dir):
 
-    # Importing here to seperate the code paths from the --livereload
+    # Importing here to separate the code paths from the --livereload
     # alternative.
     _init_asyncio_patch()
     from tornado import ioloop


Test patch:
Repository: mkdocs/mkdocs

Issue:
How to add a watched directory
The docs says:

> The serve event is only called when the serve command is used during development. It is passed the Server instance which can be modified before it is activated. For example, additional files or directories could be added to the list of "watched" files for auto-reloading.

How can I add files or directories?

I tried this:

```python
    def on_serve(self, server, config, **kwargs):
        for element in self.config["watch"]:
            server.watch(element)
        return server
```

With this in my `mkdocs.yml`:

```yaml
plugins:
  - search
  - mkdocstrings:
      watch:
        - src/mkdocstrings
```

It detects the changes, but since I gave no function to `server.watch(dir, func)`, the site is not rebuilt.

I checked the source code of `mkdocs`, and I see that you are using a local function that we cannot reuse ourselves without rewriting it:

https://github.com/mkdocs/mkdocs/blob/262c2b70f3b1a450d685530610af3f28e12f9c9f/mkdocs/commands/serve.py#L120-L137

and

https://github.com/mkdocs/mkdocs/blob/262c2b70f3b1a450d685530610af3f28e12f9c9f/mkdocs/commands/serve.py#L69

What would be the best way to add a directory with this same `builder` functionality? Should I simply copy paste it?


Discussion:
> you are using a local function that we cannot reuse ourselves without rewriting it

That would appear to be an oversight on my part. The intention was that this could be done just as you attempted. I missed that the `builder` function needs to be passed to each call to `server.watch`. Therefore, this is a bug.
I managed to hack it anyway, but am waiting for an official change in mkdocs :slightly_smiling_face: 

```python
    def on_serve(self, server, config, **kwargs):
        builder = list(server.watcher._tasks.values())[0]["func"]
        for element in self.config["watch"]:
            server.watch(element, builder)
        return server
```

Resolving patch:
diff --git a/mkdocs/commands/serve.py b/mkdocs/commands/serve.py
--- a/mkdocs/commands/serve.py
+++ b/mkdocs/commands/serve.py
@@ -73,7 +73,7 @@ def get_web_handlers(self, script):
         server.watch(d, builder)
 
     # Run `serve` plugin events.
-    server = config['plugins'].run_event('serve', server, config=config)
+    server = config['plugins'].run_event('serve', server, config=config, builder=builder)
 
     server.serve(root=site_dir, host=host, port=port, restart_delay=0)
 


Test patch:
Repository: mkdocs/mkdocs

Issue:
build --no-directory-urls feature not working
Hello. 
I created new project using 
```
mkdocs new .
mkdocs build --no-directory-urls
```

Still when I try to visit home page I am brought to . directory instead of .\index.html

Sorry for some russian text :D
![image](https://user-images.githubusercontent.com/3143647/114592047-3055ff80-9c93-11eb-9477-e1beabc8fa66.png)

I also tried to modife .yml file like this 
`use_directory_urls: false`

Still not working in "offline" mode

```
pip show mkdocs
Name: mkdocs
Version: 1.1.2
Summary: Project documentation with Markdown.
Home-page: https://www.mkdocs.org
Author: Tom Christie
Author-email: tom@tomchristie.com
License: BSD
Location: /home/demaunt/.local/lib/python3.8/site-packages
Requires: Markdown, lunr, Jinja2, PyYAML, click, livereload, tornado
Required-by: mkdocs-redirects, mkdocs-minify-plugin, mkdocs-material
```




Discussion:
I can verify this is a bug. 

![Unknown](https://user-images.githubusercontent.com/78846/114794973-91490a80-9d5b-11eb-8936-e4ce1c29d283.png)

In the above screen capture, the link circled in red is wrong (`href="."`), whereas the link circled in yellow is correct (`href="index.html"`). Although, I should note that both work fine when behind a server. The first is only broken when browsing local files (`file:///`).

It appears that the issue is related to the `homepage`. The link is created from `nav.homepage`. Yet, I note that when `use_directory_urls` is `False`, the `homepage` class is also missing from the body tag of the homepage. It seems that the homepage is not properly identified.

Resolving patch:
diff --git a/mkdocs/structure/pages.py b/mkdocs/structure/pages.py
--- a/mkdocs/structure/pages.py
+++ b/mkdocs/structure/pages.py
@@ -81,7 +81,7 @@ def is_top_level(self):
 
     @property
     def is_homepage(self):
-        return self.is_top_level and self.is_index and self.file.url == '.'
+        return self.is_top_level and self.is_index and self.file.url in ['.', 'index.html']
 
     @property
     def url(self):


Test patch:
diff --git a/mkdocs/tests/structure/nav_tests.py b/mkdocs/tests/structure/nav_tests.py
--- a/mkdocs/tests/structure/nav_tests.py
+++ b/mkdocs/tests/structure/nav_tests.py
@@ -51,6 +51,7 @@ def test_nav_no_directory_urls(self):
         self.assertEqual(str(site_navigation).strip(), expected)
         self.assertEqual(len(site_navigation.items), 2)
         self.assertEqual(len(site_navigation.pages), 2)
+        self.assertEqual(repr(site_navigation.homepage), "Page(title='Home', url='/index.html')")
 
     def test_nav_missing_page(self):
         nav_cfg = [
Repository: mkdocs/mkdocs

Issue:
file-system race condition while testing with mkdocs serve
when mkdocs serve is shutting down, it cleans up after itself removing the temporary directory created earlier.

the condition whether the directory needs to be removed is unchecked and prone to a file system race condition (shared resource).

given the directory is already removed on the file-system while shutting down, mkdocs serve throws an exception and exits in failure.

this error can be safely defined out of existence by removing the temporary directory only if it exists, will suggest a PR for this report. /E: created #2421


Discussion:

Resolving patch:
diff --git a/mkdocs/commands/serve.py b/mkdocs/commands/serve.py
--- a/mkdocs/commands/serve.py
+++ b/mkdocs/commands/serve.py
@@ -2,7 +2,7 @@
 import shutil
 import tempfile
 
-from os.path import isfile, join
+from os.path import isdir, isfile, join
 from mkdocs.commands.build import build
 from mkdocs.config import load_config
 from mkdocs.exceptions import Abort
@@ -84,4 +84,5 @@ def error_handler(code):
         # Avoid ugly, unhelpful traceback
         raise Abort(str(e))
     finally:
-        shutil.rmtree(site_dir)
+        if isdir(site_dir):
+            shutil.rmtree(site_dir)


Test patch: