Repository: pallets/werkzeug

Issue:
Stat reloader CPU usage too high on Docker for Mac
After migration from Werkzeug 1.0.1 to 2.0.1 stat reloader consuming 100% of CPU. I used stat reloader because inotify doesn't have good support in docker for mac. Using Werkzeug from django-extensions with runserver_plus command. Returning back to Werkzeug 1.0.1 solves problem (cpu seems ok).

Environment:

- Python version: 3.9.4
- Werkzeug version: 2.0.1
- django-extension: 3.1.3



Discussion:
It seems that the reloader generates a lot of syscalls to `stat()` for testing `isfile()`
Do you think we could cache the result of `isfile(path)` ? That would benefit to StatReloader and WatchdogReloader
Caching calls to `os.path.isfile()` doesn't appear to change the number of `fstatat` syscalls for either reloader.

```python
run_simple("localhost", 5000, app, use_reloader=True, reloader_type="stat")
```

```
$ strace -c -f --trace=%%stat python example.py
```

I think it's because most of the filesystem work is done by `os.walk()` which we can't cache because then new/removed files won't be detected and `os.stat()` to get the mtimes, which is central to how the reloader works and can't be cached.
Werkzeug 2.0 makes the stat reloader more accurate by walking all path directories, rather than only loaded module directories. This makes the stat reloader behavior match the watchdog behavior more, at the expense of higher CPU usage. I'm reluctant to undo that.

Here's some performance stats:

* If I run the reloader for an `example.py` file in the werkzeug repo directory and venv, the stat reloader watches about 300 files every 1 second, at 3.5% CPU.
* If instead I run it from somewhere very generic, such as my home directory with tons of unrelated folders and files, it is understandably much slower, watching 168388 files at 90% CPU.
* If we go back to the Werkzueg directory, but ran `tox` which created a bunch of venvs under `.tox/`, it jumps up to 8000 files, 10% CPU.

There's probably a way for you to optimize what the reloader is watching for your project in your docker container. Perhaps you copied your code to `/`, when it could be installed in a `/project/code` directory that doesn't add a very broad tree to `sys.path`. Or perhaps you have a large amount of files in the tree that can be ignored by pattern, such as `*/.tox/*`. If you're not concerned about installed modules, only your own code, you could add the virtualenv `site-packages` path to be excluded, which will basically go back to 1.0.x behavior.

You can see how many paths the stat reloader is watching by running the following from the same venv and directory your project will run from:

```python
from werkzeug._reloader import _find_stat_paths
print(len(_find_stat_paths(set(), set()))
```

You can print out the list as well and perhaps identify a subtree that has files that aren't relevant to reloader and can be excluded.

You can pass a list of `fnmatch` patterns to `run_simple(..., exclude_patterns=[...])` (or the second `set()` above to test it out) which will tell the reloader to ignore all files matching those patterns. I'm not sure if `runserver_plus` exposes that, but `flask run` will expose that soon, or you can call `run_simple` directly for now.

The reloader already does a bit of optimization by always ignoring `__pycache__`, `.git`, and `.hg` folders. I can add some folders for common Python tools such as tox, nox, pytest, and mypy, as well as the Flask instance folder. Projects will still need to use `exclude_patterns` to tailor the list to their specific project structure.

I'm inclined to close this based on the reasoning and capabilities above, but post back about the number of files watched and if there's any exclude_patterns you can add.
I can do a few things to cut down on the number of files scanned by the stat reloader without meaningfully changing what is watched:

* Ignore common Python tool cache directories (tox, nox, pytest, mypy)
* Always ignore `sys.base_prefix` (it's the system Python files, not the venv, it shouldn't be changing)
* Don't recurse into any directories if they or their parent do not have any Python files (not a package or data within a package)

In the case of running from my home directory, this cuts down from 16k to 800 files. In the case of running from a project directory, it cuts down from 300 (or 8k with tox) to 150.
@davidism thank you for the investigation. In my case, I see the problem in https://github.com/pallets/werkzeug/blob/73ed5efd401bd3b44cfbea67bba642c681fea6b0/src/werkzeug/_reloader.py#L71

os.walk is very CPU extensive because it walks through all frontend files in the project (node_modules). Probably your third suggestion  "Don't recurse into any directories if they or their parent do not have any Python files (not a package or data within a package)" would solve the issue.


Node modules shouldn't be in your flask project. Separate the python and node project files.
It is not Flask, but Django, but it doesn't matter. Yes, there are separate, but obviously not in the way Werkzeug would expect:

```
/app/project/backend/wsgi.py
/app/project/frontend/node_modules/
/app/project/manage.py
```

Would be great if Werkzeug reloader can handle any project structure and ignore folders without Python files. Or at least document how to prevent high CPU usage. For instance, it is possible to re-mount big directories with empty docker volume:

```
volumes:
  - /var/empty:/app/project/frontend/node_modules
```


Resolving patch:
diff --git a/src/werkzeug/_reloader.py b/src/werkzeug/_reloader.py
--- a/src/werkzeug/_reloader.py
+++ b/src/werkzeug/_reloader.py
@@ -11,16 +11,28 @@
 from ._internal import _log
 
 # The various system prefixes where imports are found. Base values are
-# different when running in a virtualenv. The stat reloader won't scan
-# these directories, it would be too inefficient.
-prefix = {sys.prefix, sys.base_prefix, sys.exec_prefix, sys.base_exec_prefix}
+# different when running in a virtualenv. All reloaders will ignore the
+# base paths (usually the system installation). The stat reloader won't
+# scan the virtualenv paths, it will only include modules that are
+# already imported.
+_ignore_always = tuple({sys.base_prefix, sys.base_exec_prefix})
+prefix = {*_ignore_always, sys.prefix, sys.exec_prefix}
 
 if hasattr(sys, "real_prefix"):
     # virtualenv < 20
-    prefix.add(sys.real_prefix)  # type: ignore
+    prefix.add(sys.real_prefix)  # type: ignore[attr-defined]
 
-_ignore_prefixes = tuple(prefix)
+_stat_ignore_scan = tuple(prefix)
 del prefix
+_ignore_common_dirs = {
+    "__pycache__",
+    ".git",
+    ".hg",
+    ".tox",
+    ".nox",
+    ".pytest_cache",
+    ".mypy_cache",
+}
 
 
 def _iter_module_paths() -> t.Iterator[str]:
@@ -29,7 +41,7 @@ def _iter_module_paths() -> t.Iterator[str]:
     for module in list(sys.modules.values()):
         name = getattr(module, "__file__", None)
 
-        if name is None:
+        if name is None or name.startswith(_ignore_always):
             continue
 
         while not os.path.isfile(name):
@@ -67,24 +79,36 @@ def _find_stat_paths(
         if os.path.isfile(path):
             # zip file on sys.path, or extra file
             paths.add(path)
+            continue
+
+        parent_has_py = {os.path.dirname(path): True}
 
         for root, dirs, files in os.walk(path):
-            # Ignore system prefixes for efficience. Don't scan
-            # __pycache__, it will have a py or pyc module at the import
-            # path. As an optimization, ignore .git and .hg since
-            # nothing interesting will be there.
-            if root.startswith(_ignore_prefixes) or os.path.basename(root) in {
-                "__pycache__",
-                ".git",
-                ".hg",
-            }:
+            # Optimizations: ignore system prefixes, __pycache__ will
+            # have a py or pyc module at the import path, ignore some
+            # common known dirs such as version control and tool caches.
+            if (
+                root.startswith(_stat_ignore_scan)
+                or os.path.basename(root) in _ignore_common_dirs
+            ):
                 dirs.clear()
                 continue
 
+            has_py = False
+
             for name in files:
                 if name.endswith((".py", ".pyc")):
+                    has_py = True
                     paths.add(os.path.join(root, name))
 
+            # Optimization: stop scanning a directory if neither it nor
+            # its parent contained Python files.
+            if not (has_py or parent_has_py[os.path.dirname(root)]):
+                dirs.clear()
+                continue
+
+            parent_has_py[root] = has_py
+
     paths.update(_iter_module_paths())
     _remove_by_pattern(paths, exclude_patterns)
     return paths
@@ -267,7 +291,7 @@ def __enter__(self) -> ReloaderLoop:
         return super().__enter__()
 
     def run_step(self) -> None:
-        for name in chain(_find_stat_paths(self.extra_files, self.exclude_patterns)):
+        for name in _find_stat_paths(self.extra_files, self.exclude_patterns):
             try:
                 mtime = os.stat(name).st_mtime
             except OSError:
@@ -311,9 +335,7 @@ def on_any_event(self, event):  # type: ignore
         self.event_handler = EventHandler(
             patterns=["*.py", "*.pyc", "*.zip", *extra_patterns],
             ignore_patterns=[
-                "*/__pycache__/*",
-                "*/.git/*",
-                "*/.hg/*",
+                *[f"*/{d}/*" for d in _ignore_common_dirs],
                 *self.exclude_patterns,
             ],
         )


Test patch:
diff --git a/tests/live_apps/reloader_app.py b/tests/live_apps/reloader_app.py
--- a/tests/live_apps/reloader_app.py
+++ b/tests/live_apps/reloader_app.py
@@ -8,8 +8,8 @@
 # Tox puts the tmp dir in the venv sys.prefix, patch the reloader so
 # it doesn't skip real_app.
 if "TOX_ENV_DIR" in os.environ:
-    _reloader._ignore_prefixes = tuple(
-        set(_reloader._ignore_prefixes) - {sys.prefix, sys.exec_prefix}
+    _reloader._stat_ignore_scan = tuple(
+        set(_reloader._stat_ignore_scan) - {sys.prefix, sys.exec_prefix}
     )