Repository: pallets/werkzeug

Issue:
Reloader option to exclude watch paths
Seems like the server gets reloaded when `gen_py` caching files (created by the `win32com` module) are written or modified (in a temporary directory):

```
 * Detected change in 'C:\\Users\\X\\AppData\\Local\\Temp\\gen_py\\3.6\\C866CA3A-32F7-11D2-9602-00C04F8EE628x0x5x4\\__pycache__\\SpVoice.cpython-36.pyc', reloading
 * Detected change in 'C:\\Users\\X\\AppData\\Local\\Temp\\gen_py\\3.6\\C866CA3A-32F7-11D2-9602-00C04F8EE628x0x5x4\\__pycache__\\_ISpeechVoiceEvents.cpython-36.pyc', reloading
```
This seems to happen quite often. Any way for this to be prevented?

(Using Python `3.6`, Flask `1.0.2`, Werkzeug `0.14.1` on `Win10 64bit`. `FLASK_ENV=debug`, `FLASK_DEBUG=1`)

*Edit*: this also causes occasional cache corruptions (causing runtime errors and requiring wiping the `gen_py` directory to resolve) since it abruptly cuts out the process while it is writing (usually several files) to disk.


Discussion:
set this environment variable: PYTHONDONTWRITEBYTECODE=1
Thanks for the advice. I wiped the temporary `gen_py` directory and I tried it. Seems like `.pyc` files aren't written, however there are still `.py` files being written there. Causing the server to reload:

```
 * Detected change in 'C:\\Users\\X\\AppData\\Local\\Temp\\gen_py\\3.6\\C866CA3A-32F7-11D2-9602-00C04F8EE628x0x5x4\\ISpeechObjectTokens.py', reloading
 * Detected change in 'C:\\Users\\X\\AppData\\Local\\Temp\\gen_py\\3.6\\C866CA3A-32F7-11D2-9602-00C04F8EE628x0x5x4\\SpObjectToken.py', reloading
 * Detected change in 'C:\\Users\\X\\AppData\\Local\\Temp\\gen_py\\3.6\\C866CA3A-32F7-11D2-9602-00C04F8EE628x0x5x4\\ISpeechObjectToken.py', reloading
 * Detected change in 'C:\\Users\\X\\AppData\\Local\\Temp\\gen_py\\3.6\\C866CA3A-32F7-11D2-9602-00C04F8EE628x0x5x4\\SpMemoryStream.py', reloading
 * Detected change in 'C:\\Users\\X\\AppData\\Local\\Temp\\gen_py\\3.6\\C866CA3A-32F7-11D2-9602-00C04F8EE628x0x5x4\\ISpeechMemoryStream.py', reloading
 * Detected change in 'C:\\Users\\X\\AppData\\Local\\Temp\\gen_py\\3.6\\C866CA3A-32F7-11D2-9602-00C04F8EE628x0x5x4\\SpAudioFormat.py', reloading
 * Detected change in 'C:\\Users\\X\\AppData\\Local\\Temp\\gen_py\\3.6\\C866CA3A-32F7-11D2-9602-00C04F8EE628x0x5x4\\ISpeechAudioFormat.py', reloading
```
Does your app successfully reload after seeing those lines above? 
@angadkalra The app reloads (although there is no practical need for it to). Though since the `win32com` module is writing multiple files to disk and is abruptly terminated by the reloader, it occasionally results in a cache corruption (some files successfully written, some don't or only partially). Next time it loads it may sometimes crash the server with a runtime error of the sort `Can't load COM module ...` since the cache has been corrupted and only wiping out the `gen_py` dir would seem to resolve it.

The goal is not to have the server reload when files (including `.py`) are written to cache or temporary directories. It seems reasonable there's already a way to do it but I'm not sure how to achieve it.
I realized this will break the reloader if the virtualenv itself is in the temp folder. None of the tools I commonly use do that by default (pew temp envs, tox, nox), and I don't think Pipenv or Poetry do, but there are a lot of other tools and workflows out there.

Instead, let's allow passing a list of `exclude_paths` like the `extra_files`. I'm not sure of the exact semantics for this. I know there's a difference between the stat and watchdog reloaders about watching paths. I'm not sure what order extra and exclude should be considered in.

Resolving patch:
diff --git a/src/werkzeug/_reloader.py b/src/werkzeug/_reloader.py
--- a/src/werkzeug/_reloader.py
+++ b/src/werkzeug/_reloader.py
@@ -1,11 +1,12 @@
+import fnmatch
 import os
 import subprocess
 import sys
 import threading
 import time
+import typing as t
 from itertools import chain
-from typing import Any
-from typing import Optional
+from pathlib import PurePath
 
 from ._internal import _log
 
@@ -42,7 +43,12 @@ def _iter_module_paths():
             yield name
 
 
-def _find_stat_paths(extra_files):
+def _remove_by_pattern(paths: t.Set[str], exclude_patterns: t.Set[str]) -> None:
+    for pattern in exclude_patterns:
+        paths.difference_update(fnmatch.filter(paths, pattern))
+
+
+def _find_stat_paths(extra_files, exclude_patterns):
     """Find paths for the stat reloader to watch. Returns imported
     module files, Python files under non-system paths. Extra files and
     Python files under extra directories can also be scanned.
@@ -51,12 +57,14 @@ def _find_stat_paths(extra_files):
     such as a project root or ``sys.path.insert``, should be the paths
     of interest to the user anyway.
     """
+    paths = set()
+
     for path in chain(list(sys.path), extra_files):
         path = os.path.abspath(path)
 
         if os.path.isfile(path):
             # zip file on sys.path, or extra file
-            yield path
+            paths.add(path)
 
         for root, dirs, files in os.walk(path):
             # Ignore system prefixes for efficience. Don't scan
@@ -73,12 +81,14 @@ def _find_stat_paths(extra_files):
 
             for name in files:
                 if name.endswith((".py", ".pyc")):
-                    yield os.path.join(root, name)
+                    paths.add(os.path.join(root, name))
 
-    yield from _iter_module_paths()
+    paths.update(_iter_module_paths())
+    _remove_by_pattern(paths, exclude_patterns)
+    return paths
 
 
-def _find_watchdog_paths(extra_files):
+def _find_watchdog_paths(extra_files, exclude_patterns):
     """Find paths for the stat reloader to watch. Looks at the same
     sources as the stat reloader, but watches everything under
     directories instead of individual files.
@@ -96,13 +106,14 @@ def _find_watchdog_paths(extra_files):
     for name in _iter_module_paths():
         dirs.add(os.path.dirname(name))
 
+    _remove_by_pattern(dirs, exclude_patterns)
     return _find_common_roots(dirs)
 
 
 def _find_common_roots(paths):
     root = {}
 
-    for chunks in sorted((x.split(os.path.sep) for x in paths), key=len, reverse=True):
+    for chunks in sorted((PurePath(x).parts for x in paths), key=len, reverse=True):
         node = root
 
         for chunk in chunks:
@@ -117,7 +128,7 @@ def _walk(node, path):
             _walk(child, path + (prefix,))
 
         if not node:
-            rv.add("/".join(path))
+            rv.add(os.path.join(*path))
 
     _walk(root, ())
     return rv
@@ -186,8 +197,14 @@ def _get_args_for_reloading():
 class ReloaderLoop:
     name = ""
 
-    def __init__(self, extra_files=None, interval=1):
+    def __init__(
+        self,
+        extra_files: t.Optional[t.Iterable[str]] = None,
+        exclude_patterns: t.Optional[t.Iterable[str]] = None,
+        interval: t.Union[int, float] = 1,
+    ):
         self.extra_files = {os.path.abspath(x) for x in extra_files or ()}
+        self.exclude_patterns = set(exclude_patterns or ())
         self.interval = interval
 
     def __enter__(self):
@@ -246,7 +263,7 @@ def __enter__(self):
         return super().__enter__()
 
     def run_step(self):
-        for name in chain(_find_stat_paths(self.extra_files)):
+        for name in chain(_find_stat_paths(self.extra_files, self.exclude_patterns)):
             try:
                 mtime = os.stat(name).st_mtime
             except OSError:
@@ -289,7 +306,12 @@ def on_any_event(self, event):
         extra_patterns = [p for p in self.extra_files if not os.path.isdir(p)]
         self.event_handler = EventHandler(
             patterns=["*.py", "*.pyc", "*.zip", *extra_patterns],
-            ignore_patterns=["*/__pycache__/*", "*/.git/*", "*/.hg/*"],
+            ignore_patterns=[
+                "*/__pycache__/*",
+                "*/.git/*",
+                "*/.hg/*",
+                *self.exclude_patterns,
+            ],
         )
         self.should_reload = False
 
@@ -319,7 +341,7 @@ def run(self):
     def run_step(self):
         to_delete = set(self.watches)
 
-        for path in _find_watchdog_paths(self.extra_files):
+        for path in _find_watchdog_paths(self.extra_files, self.exclude_patterns):
             if path not in self.watches:
                 try:
                     self.watches[path] = self.observer.schedule(
@@ -340,7 +362,7 @@ def run_step(self):
                 self.observer.unschedule(watch)
 
 
-reloader_loops: Any = {
+reloader_loops: t.Dict[str, t.Type[ReloaderLoop]] = {
     "stat": StatReloaderLoop,
     "watchdog": WatchdogReloaderLoop,
 }
@@ -374,15 +396,18 @@ def ensure_echo_on():
 
 def run_with_reloader(
     main_func,
-    extra_files: Optional[Any] = None,
-    interval: float = 1,
+    extra_files: t.Optional[t.Iterable[str]] = None,
+    exclude_patterns: t.Optional[t.Iterable[str]] = None,
+    interval: t.Union[int, float] = 1,
     reloader_type: str = "auto",
 ):
     """Run the given function in an independent Python interpreter."""
     import signal
 
-    reloader = reloader_loops[reloader_type](extra_files, interval)
     signal.signal(signal.SIGTERM, lambda *args: sys.exit(0))
+    reloader = reloader_loops[reloader_type](
+        extra_files=extra_files, exclude_patterns=exclude_patterns, interval=interval
+    )
 
     try:
         if os.environ.get("WERKZEUG_RUN_MAIN") == "true":
diff --git a/src/werkzeug/serving.py b/src/werkzeug/serving.py
--- a/src/werkzeug/serving.py
+++ b/src/werkzeug/serving.py
@@ -763,6 +763,7 @@ def run_simple(
     use_debugger=False,
     use_evalex=True,
     extra_files=None,
+    exclude_patterns=None,
     reloader_interval=1,
     reloader_type="auto",
     threaded=False,
@@ -779,6 +780,9 @@ def run_simple(
 
         python -m werkzeug.serving --help
 
+    .. versionchanged:: 2.0
+        Added ``exclude_patterns`` parameter.
+
     .. versionadded:: 0.5
        `static_files` was added to simplify serving of static files as well
        as `passthrough_errors`.
@@ -814,6 +818,9 @@ def run_simple(
     :param extra_files: a list of files the reloader should watch
                         additionally to the modules.  For example configuration
                         files.
+    :param exclude_patterns: List of :mod:`fnmatch` patterns to ignore
+        when running the reloader. For example, ignore cache files that
+        shouldn't reload when updated.
     :param reloader_interval: the interval for the reloader in seconds.
     :param reloader_type: the type of reloader to use.  The default is
                           auto detection.  Valid values are ``'stat'`` and
@@ -926,7 +933,13 @@ def inner():
 
         from ._reloader import run_with_reloader
 
-        run_with_reloader(inner, extra_files, reloader_interval, reloader_type)
+        run_with_reloader(
+            inner,
+            extra_files=extra_files,
+            exclude_patterns=exclude_patterns,
+            interval=reloader_interval,
+            reloader_type=reloader_type,
+        )
     else:
         inner()
 


Test patch:
diff --git a/tests/test_serving.py b/tests/test_serving.py
--- a/tests/test_serving.py
+++ b/tests/test_serving.py
@@ -10,6 +10,8 @@
 import pytest
 
 from werkzeug import run_simple
+from werkzeug._reloader import _find_stat_paths
+from werkzeug._reloader import _find_watchdog_paths
 from werkzeug._reloader import _get_args_for_reloading
 from werkzeug.datastructures import FileStorage
 from werkzeug.serving import make_ssl_devcert
@@ -105,6 +107,16 @@ def test_windows_get_args_for_reloading(monkeypatch, tmp_path):
     assert rv == argv
 
 
+@pytest.mark.parametrize("find", [_find_stat_paths, _find_watchdog_paths])
+def test_exclude_patterns(find):
+    # Imported paths under sys.prefix will be included by default.
+    paths = find(set(), set())
+    assert any(p.startswith(sys.prefix) for p in paths)
+    # Those paths should be excluded due to the pattern.
+    paths = find(set(), {f"{sys.prefix}*"})
+    assert not any(p.startswith(sys.prefix) for p in paths)
+
+
 def test_wrong_protocol(standard_app):
     """An HTTPS request to an HTTP server doesn't show a traceback.
     https://github.com/pallets/werkzeug/pull/838