Repository: mkdocs/mkdocs

Issue:
Re-building w/ symbolic links stopped working, regression after #2385
Since a444c43 in master using the local development server via `mkdocs serve` updating files that are symbolically linked is not triggering to rebuild (and therefore not reloading browser tabs).

On first glance this is due to the switch to watchdog for detecting file-system changes which needs more guidance to handle this file-type.

Preparing a PR with a patch.

Ref: a444c43474f91dea089922dd8fb188d1db3a4535
restore re-building with symbolic-links, closes #2425
previously (1.1.2 + master at 23e2051) building was triggered by changes
of file-content that was symbolically linked within docs_dir while
`mkdocs serve` was running.

since migrating from livereload>=2.6.1 to watchdog>=2.0.0 to detect
file-system changes (triggering the re-build) it stopped working.

this is because watchdog does not support symbolic links out of the box,
e.g. see [1].

change is to provide additional observe instructions on the realpath [2]
for the following cases:

1. docs_dir & config_file_path path deviation:

   when the absolute path to either the `docs_dir` or the `config_file` is
   different from its realpath, the realpath is added for observing (as
   well).

2. symbolic links within docs_dir:

   if a file within docs_dir is a symbolic link, the files real path
   is added for observing. sub-directories (that are not symbolically
   linked) are traversed up to a depth of nine levels (only if the
   recursive flag is enabled, otherwise no traversal into sub-directories).

Ref: 23e205153f01d24d50fe9ba18e5186cdbc2c2dbe
[1]: https://github.com/gorakhargosh/watchdog/issues/365
[2]: <https://docs.python.org/3.8/library/os.path.html#os.path.realpath>


Discussion:

@ktomk there is no need to open an issue if you are going to open a PR. You can report the issue as part of the PR. To be clear, there is no rule one way or the other. Just trying to save extra work if it is not needed.

@oprypin as this relates to your code, I will give you a chance to review it.
good to know for next time @waylan

yes, please review and bear with me, my python skills are still young @oprypin
 
Thanks for the find. I wasn't considering symlinks at all.

From a quick analysis of the code flow (and I could be mistaken), it seems that the recursion in this code applies to just walking subdirectories, but multiple levels of symlinks don't actually seem to be followed. Is that intentional?

And, this will certainly benefit from tests. There are existing tests that won't be difficult to adapt -- just need to add creation of symlinks at the beginning. And, as per the previous paragraph, a test with a symlink to a directory with a symlink will definitely be needed, regardless if following multiple levels of symlinks is actually done.
> From a quick analysis of the code flow (and I could be mistaken), it seems that the recursion in this code applies to just walking subdirectories, but multiple levels of symlinks don't actually seem to be followed. Is that intentional?

First of all yes, it keeps the logic simple and I don't know if this worked earlier as this is not part of the setup I noticed it in and then checked. Maybe defer unless needed? @oprypin 
Symlinks to symlinks would've worked before. And allowing only 1 level of symlinks just seems arbitrary.
That's about all I have for arguments. I never use symlinks with MkDocs, so I don't know.

In any case, this needs unittests.
> Thanks for the find. I wasn't considering symlinks at all.

If watchdog would just support them out of the box, this would be a no-brainer. You linked livereload impl. earlier, perhaps this should just be ported in-place (for resolution of symlinks only). Was not aware of it, likely the sweet spot.

> Symlinks to symlinks would've worked before.

This then taken care of, too.

> And if we talk about livereload library, well, it just follows all symlinks and all dirs with absolutely no attempt to avoid loops or putting other limits on traversal.
https://github.com/lepture/python-livereload/blob/ecabebaeee2a40743f5a8216e1856c9f9150913f/livereload/watcher.py#L197.

Exactly that. (I just didn't see recursion there but only looked two times, so could be me)

> Ah I think there's no "before" that applies here.

Maybe a miscommunication. Before is at 23e2051 and it exists. Regression test must pass there and fail with feature merge, then must pass for the fix. Give the test a date to be removed in fix-time + one year. Just my 2 cents.

> Regarding loops in file systems.

(also race conditions and that material) Whatever works for you, make the decisions you consider wise. There is something to learn every day in that area and I trust you to handle this well for mkdocs.

> Mhm well I would be glad to take over implementing this. Just didn't want to rush in and be like "no, let me do it".
Please let me know what you think.

Well, do what suites you. I marked the PR as draft and allowed edits, there is nothing that should have prevented you from forking earlier and thanks for being verbal.

So just take the report / PR as inspiration and improve the original feature. Totally fine with that.

I only updated to match with the commit message as change 1. went bonkers, this is edited in.

And you can always ping me for feedback, I might get my mkdocs build pushed which triggered this, so that there is also the use-case available and visible. If I do, I'll comment here with a link.

Today were too many things to do, so I could only summarize here quick and I hope I have addressed all the parts you wanted to have feedback for.



I haven't read through everything here, so maybe this is all covered. Unfortunately, I have very limited time this week (more than usual) and will not be able to give any time to this until next week at the earliest.

So I will just briefly summarize what I expect the behavior should be. Essentially our behavior should exactly match livereload's in this case. Based on reports, some of our users rely heavily on symbolic links and this would be an unacceptable regression for them. Those users would likely argue that we either need to fix this or revert the changes we have now. 

I see three possible solutions:

1. Fix this by modifying Watchdog's behavior locally so that it matches  liverelaod's (no limits at all).
2. Replace watchdog with a competing library which supports this.
3. Revert back to the new server.

My hope is that we can find a reasonable solution for option 1. I would prefer to not need to do option 3. However, if I had recognized this was an issue before, I would have considered this a blocker to accepting that PR. Personally, I don't use symbolic links in my MkDocs projects, but we have always guaranteed they will work, so we can't be breaking them. And we shouldn't be creating any artificial limits either. 
I'm on it

Resolving patch:
diff --git a/mkdocs/livereload/__init__.py b/mkdocs/livereload/__init__.py
--- a/mkdocs/livereload/__init__.py
+++ b/mkdocs/livereload/__init__.py
@@ -4,6 +4,7 @@
 import mimetypes
 import os
 import os.path
+import pathlib
 import re
 import socketserver
 import threading
@@ -76,8 +77,10 @@ def watch(self, path, func=None, recursive=True):
                 stacklevel=2,
             )
 
-        def callback(event):
-            if event.is_directory:
+        def callback(event, allowed_path=None):
+            if isinstance(event, watchdog.events.DirCreatedEvent):
+                return
+            if allowed_path is not None and event.src_path != allowed_path:
                 return
             # Text editors always cause a "file close" event in addition to "modified" when saving
             # a file. Some editors also have "swap" functionality that keeps writing into another
@@ -91,9 +94,43 @@ def callback(event):
                 self._to_rebuild[func] = True
                 self._rebuild_cond.notify_all()
 
-        handler = watchdog.events.FileSystemEventHandler()
-        handler.on_any_event = callback
-        self.observer.schedule(handler, path, recursive=recursive)
+        dir_handler = watchdog.events.FileSystemEventHandler()
+        dir_handler.on_any_event = callback
+
+        seen = set()
+
+        def schedule(path):
+            seen.add(path)
+            if os.path.isfile(path):
+                # Watchdog doesn't support watching files, so watch its directory and filter by path
+                handler = watchdog.events.FileSystemEventHandler()
+                handler.on_any_event = lambda event: callback(event, allowed_path=path)
+
+                parent = os.path.dirname(path)
+                log.debug(f"Watching file '{path}' through directory '{parent}'")
+                self.observer.schedule(handler, parent)
+            else:
+                log.debug(f"Watching directory '{path}'")
+                self.observer.schedule(dir_handler, path, recursive=recursive)
+
+        schedule(os.path.realpath(path))
+
+        def watch_symlink_targets(path_obj):  # path is os.DirEntry or pathlib.Path
+            if path_obj.is_symlink():
+                # The extra `readlink` is needed due to https://bugs.python.org/issue9949
+                target = os.path.realpath(os.readlink(os.fspath(path_obj)))
+                if target in seen or not os.path.exists(target):
+                    return
+                schedule(target)
+
+                path_obj = pathlib.Path(target)
+
+            if path_obj.is_dir() and recursive:
+                with os.scandir(os.fspath(path_obj)) as scan:
+                    for entry in scan:
+                        watch_symlink_targets(entry)
+
+        watch_symlink_targets(pathlib.Path(path))
 
     def serve(self):
         self.observer.start()


Test patch:
diff --git a/mkdocs/tests/base.py b/mkdocs/tests/base.py
--- a/mkdocs/tests/base.py
+++ b/mkdocs/tests/base.py
@@ -70,8 +70,7 @@ def example(self, tdir):
     """
     files = {f: '' for f in files} if isinstance(files, (list, tuple)) else files or {}
 
-    if 'prefix' not in kw:
-        kw['prefix'] = 'mkdocs_test-'
+    kw['prefix'] = 'mkdocs_test-' + kw.get('prefix', '')
 
     def decorator(fn):
         @wraps(fn)
diff --git a/mkdocs/tests/livereload_tests.py b/mkdocs/tests/livereload_tests.py
--- a/mkdocs/tests/livereload_tests.py
+++ b/mkdocs/tests/livereload_tests.py
@@ -3,6 +3,7 @@
 import contextlib
 import email
 import io
+import sys
 import threading
 import time
 import unittest
@@ -36,6 +37,7 @@ def testing_server(root, builder=lambda: None, mount_path="/"):
             port=0,
             root=root,
             mount_path=mount_path,
+            build_delay=0.1,
             bind_and_activate=False,
         )
         server.setup_environ()
@@ -75,28 +77,40 @@ def test_serves_normal_file(self, site_dir):
             self.assertEqual(headers["_status"], "200 OK")
             self.assertEqual(headers.get("content-length"), str(len(output)))
 
-    @tempdir({"foo.docs": "a"})
+    @tempdir({"docs/foo.docs": "docs1", "mkdocs.yml": "yml1"})
     @tempdir({"foo.site": "original"})
-    def test_basic_rebuild(self, site_dir, docs_dir):
+    def test_basic_rebuild(self, site_dir, origin_dir):
+        docs_dir = Path(origin_dir, "docs")
+
         started_building = threading.Event()
 
         def rebuild():
             started_building.set()
-            content = Path(docs_dir, "foo.docs").read_text()
-            Path(site_dir, "foo.site").write_text(content * 5)
+            Path(site_dir, "foo.site").write_text(
+                Path(docs_dir, "foo.docs").read_text() + Path(origin_dir, "mkdocs.yml").read_text()
+            )
 
         with testing_server(site_dir, rebuild) as server:
             server.watch(docs_dir, rebuild)
+            server.watch(Path(origin_dir, "mkdocs.yml"), rebuild)
             time.sleep(0.01)
 
             _, output = do_request(server, "GET /foo.site")
             self.assertEqual(output, "original")
 
-            Path(docs_dir, "foo.docs").write_text("b")
+            Path(docs_dir, "foo.docs").write_text("docs2")
             self.assertTrue(started_building.wait(timeout=10))
+            started_building.clear()
 
             _, output = do_request(server, "GET /foo.site")
-            self.assertEqual(output, "bbbbb")
+            self.assertEqual(output, "docs2yml1")
+
+            Path(origin_dir, "mkdocs.yml").write_text("yml2")
+            self.assertTrue(started_building.wait(timeout=10))
+            started_building.clear()
+
+            _, output = do_request(server, "GET /foo.site")
+            self.assertEqual(output, "docs2yml2")
 
     @tempdir({"foo.docs": "a"})
     @tempdir({"foo.site": "original"})
@@ -408,3 +422,99 @@ def test_redirects_to_mount_path(self, site_dir):
                 headers, _ = do_request(server, "GET /")
             self.assertEqual(headers["_status"], "302 Found")
             self.assertEqual(headers.get("location"), "/mount/path/")
+
+    @tempdir({"mkdocs.yml": "original", "mkdocs2.yml": "original"}, prefix="tmp_dir")
+    @tempdir(prefix="origin_dir")
+    @tempdir({"subdir/foo.md": "original"}, prefix="dest_docs_dir")
+    def test_watches_direct_symlinks(self, dest_docs_dir, origin_dir, tmp_dir):
+        try:
+            Path(origin_dir, "docs").symlink_to(dest_docs_dir, target_is_directory=True)
+            Path(origin_dir, "mkdocs.yml").symlink_to(Path(tmp_dir, "mkdocs.yml"))
+        except NotImplementedError:  # PyPy on Windows
+            self.skipTest("Creating symlinks not supported")
+
+        started_building = threading.Event()
+
+        def wait_for_build():
+            result = started_building.wait(timeout=10)
+            started_building.clear()
+            with self.assertLogs("mkdocs.livereload"):
+                do_request(server, "GET /")
+            return result
+
+        with testing_server(tmp_dir, started_building.set) as server:
+            server.watch(Path(origin_dir, "docs"))
+            server.watch(Path(origin_dir, "mkdocs.yml"))
+            time.sleep(0.01)
+
+            Path(tmp_dir, "mkdocs.yml").write_text("edited")
+            self.assertTrue(wait_for_build())
+
+            Path(dest_docs_dir, "subdir", "foo.md").write_text("edited")
+            self.assertTrue(wait_for_build())
+
+            Path(origin_dir, "unrelated.md").write_text("foo")
+            self.assertFalse(started_building.wait(timeout=0.2))
+
+    @tempdir(["file_dest_1.md", "file_dest_2.md", "file_dest_unused.md"], prefix="tmp_dir")
+    @tempdir(["file_under.md"], prefix="dir_to_link_to")
+    @tempdir()
+    def test_watches_through_symlinks(self, docs_dir, dir_to_link_to, tmp_dir):
+        try:
+            Path(docs_dir, "link1.md").symlink_to(Path(tmp_dir, "file_dest_1.md"))
+            Path(docs_dir, "linked_dir").symlink_to(dir_to_link_to, target_is_directory=True)
+
+            Path(dir_to_link_to, "sublink.md").symlink_to(Path(tmp_dir, "file_dest_2.md"))
+        except NotImplementedError:  # PyPy on Windows
+            self.skipTest("Creating symlinks not supported")
+
+        started_building = threading.Event()
+
+        def wait_for_build():
+            result = started_building.wait(timeout=10)
+            started_building.clear()
+            with self.assertLogs("mkdocs.livereload"):
+                do_request(server, "GET /")
+            return result
+
+        with testing_server(docs_dir, started_building.set) as server:
+            server.watch(docs_dir)
+            time.sleep(0.01)
+
+            Path(tmp_dir, "file_dest_1.md").write_text("edited")
+            self.assertTrue(wait_for_build())
+
+            Path(dir_to_link_to, "file_under.md").write_text("edited")
+            self.assertTrue(wait_for_build())
+
+            Path(tmp_dir, "file_dest_2.md").write_text("edited")
+            self.assertTrue(wait_for_build())
+
+            Path(docs_dir, "link1.md").unlink()
+            self.assertTrue(wait_for_build())
+
+            Path(tmp_dir, "file_dest_unused.md").write_text("edited")
+            self.assertFalse(started_building.wait(timeout=0.2))
+
+    @tempdir()
+    def test_watch_with_broken_symlinks(self, docs_dir):
+        Path(docs_dir, "subdir").mkdir()
+
+        try:
+            if sys.platform != "win32":
+                Path(docs_dir, "subdir", "circular").symlink_to(Path(docs_dir))
+            Path(docs_dir, "self_link").symlink_to(Path(docs_dir, "self_link"))
+
+            Path(docs_dir, "broken_1").symlink_to(Path(docs_dir, "oh no"))
+            Path(docs_dir, "broken_2").symlink_to(Path(docs_dir, "oh no"), target_is_directory=True)
+            Path(docs_dir, "broken_3").symlink_to(Path(docs_dir, "broken_2"))
+        except NotImplementedError:  # PyPy on Windows
+            self.skipTest("Creating symlinks not supported")
+
+        started_building = threading.Event()
+        with testing_server(docs_dir, started_building.set) as server:
+            server.watch(docs_dir)
+            time.sleep(0.01)
+
+            Path(docs_dir, "subdir", "test").write_text("test")
+            self.assertTrue(started_building.wait(timeout=10))