Repository: pallets/werkzeug

Issue:
Ability to specify locations to watch
When I upgraded werkzeug to 0.10.0, reloading stopped working for my app. In addition to this, when using stat, CPU usage is huge even with an interval of 5. 

If I use werkzeug 0.9.6, CPU usage decreases with about 50% and reloading works.



Discussion:
Which version of watchdog are you using?

0.8.2

There is no such release on PyPI. The dev version from GitHub seems to work fine for me too.

@untitaker just edited :$ was 0.8.2

Hmm, in which way does it not work for you? Could you give me an exact testcase? Note that the behavior of the reloader might have changed, regarding to how much files it watches (it mostly restricts itself to *.py files nowadays)

@untitaker doing `touch app.py` would not trigger reload, nor would actually modifying file content. I'm completely rebuilding my virtualenv now because for some reason it's not picking up watchdog right now.. 

This must have been a glitch in my virtualenv because now it works fine.. Sorry about that.

No problem.

@untitaker So I tracked this down a little further and it turns out that it's because I'm using a vagrant box with an NFS volume with the app. inotify cannot detect changes in network volumes so that's the reason this isn't working.

I am curious about one thing: What's the reason that everything in site-packages is also wachted? Because for a fairly large app with lots of deps, this makes a stat loop extremely expensive.

I can't find the exact issue numbers, but the reloader used to watch too little/break in weird edge cases, so we decided to... watch more instead. Maybe we overdid it, but if the inotify reloader doesn't break in edge cases _and_ is fast, that's obviously a win.

@untitaker I could create a PR with a `reloader_paths` param which defaults to `sys.path`. I can imagine quite a few people use docker or vagrant to avoid installing all the deps for every app on their system so `watchdog` won't be an option then. And when you've got as many files in your `site-packages` as I have the stat loop is extremely expensive

Well, I wouldn't use any of werkzeug.serving for production to begin with. Actually implementing a production-ready reloader is as much out of the scope as having a production ready server in Werkzeug... also, this problem doesn't occur to me since I explicitly restart the services in my deploy scripts (fabric in my case).

@untitaker I wasn't talking about production ;-). I have a mac with a Vagrant box for my app (so I don't have to install redis, pg, rabbitmq etc on my mac). I then mount the `src` folder from my mac fs in the vagrant box using nfs. 

Have you tried the reverse -- mounting part of the vagrant box on the host?

I haven't but it appears to be possible, although it's not quite as easy as the default shared folders setup with vagrant/virtualbox. For now, I'll just do manual restarts but I can imagine more people will run into this issue when they try the new reloader (most devs I know use vagrant or docker for local development).

I am not sure about other solutions. I suppose the reloader could be composed of two parts: One thing that finds the relevant files/folders, another thing that actually watches them. If you want to avoid watching particular files/folders, you would have to provide each of them manually. Is that an acceptable solution?

@untitaker Yup. For me a solution where I can provide a custom list of folders to watch would be perfect.  The default could then still be `sys.path` as it is now.


Resolving patch:
diff --git a/werkzeug/_reloader.py b/werkzeug/_reloader.py
--- a/werkzeug/_reloader.py
+++ b/werkzeug/_reloader.py
@@ -9,7 +9,7 @@
 from werkzeug._compat import PY2, iteritems, text_type
 
 
-def _iter_module_files():
+def _iter_module_files(reloader_paths=None):
     """This iterates over all relevant Python files.  It goes through all
     loaded files from modules, all files in folders of already loaded modules
     as well as all files reachable through a package.
@@ -53,36 +53,45 @@ def _recursive_walk(path_entry):
         except OSError:
             pass
 
-    # The list call is necessary on Python 3 in case the module
-    # dictionary modifies during iteration.
-    for path_entry in list(sys.path):
-        for filename in _recursive_walk(os.path.abspath(path_entry)):
-            yield filename
-
-    for module in list(sys.modules.values()):
-        if module is None:
-            continue
-        filename = _verify_file(getattr(module, '__file__', None))
-        if filename:
-            yield filename
-            for filename in _recursive_walk(os.path.dirname(filename)):
+    if not reloader_paths:
+        # The list call is necessary on Python 3 in case the module
+        # dictionary modifies during iteration.
+        for path_entry in list(sys.path):
+            for filename in _recursive_walk(os.path.abspath(path_entry)):
                 yield filename
-        for package_path in getattr(module, '__path__', ()):
-            for filename in _recursive_walk(os.path.abspath(package_path)):
+
+        for module in list(sys.modules.values()):
+            if module is None:
+                continue
+            filename = _verify_file(getattr(module, '__file__', None))
+            if filename:
+                yield filename
+                for filename in _recursive_walk(os.path.dirname(filename)):
+                    yield filename
+            for package_path in getattr(module, '__path__', ()):
+                for filename in _recursive_walk(os.path.abspath(package_path)):
+                    yield filename
+    else:
+        for path_entry in reloader_paths:
+            for filename in _recursive_walk(os.path.abspath(path_entry)):
                 yield filename
 
 
-def _find_observable_paths(extra_files=None):
+def _find_observable_paths(reloader_paths=None, extra_files=None):
     """Finds all paths that should be observed."""
-    rv = set(os.path.abspath(x) for x in sys.path)
+    if not reloader_paths:
+        rv = set(os.path.abspath(x) for x in sys.path)
+        for module in list(sys.modules.values()):
+            fn = getattr(module, '__file__', None)
+            if fn is None:
+                continue
+            fn = os.path.abspath(fn)
+            rv.add(os.path.dirname(fn))
+    else:
+        rv = set(os.path.abspath(x) for x in reloader_paths)
     for filename in extra_files or ():
         rv.append(os.path.dirname(os.path.abspath(filename)))
-    for module in list(sys.modules.values()):
-        fn = getattr(module, '__file__', None)
-        if fn is None:
-            continue
-        fn = os.path.abspath(fn)
-        rv.add(os.path.dirname(fn))
+
     return rv
 
 
@@ -114,7 +123,8 @@ class ReloaderLoop(object):
     # `eventlet.monkey_patch`) before we get here
     _sleep = staticmethod(time.sleep)
 
-    def __init__(self, extra_files=None, interval=1):
+    def __init__(self, reloader_paths=None, extra_files=None, interval=1):
+        self.reloader_paths = reloader_paths
         self.extra_files = set(os.path.abspath(x)
                                for x in extra_files or ())
         self.interval = interval
@@ -156,7 +166,8 @@ class StatReloaderLoop(ReloaderLoop):
     def run(self):
         mtimes = {}
         while 1:
-            for filename in chain(_iter_module_files(), self.extra_files):
+            for filename in chain(_iter_module_files(self.reloader_paths),
+                                  self.extra_files):
                 try:
                     mtime = os.stat(filename).st_mtime
                 except OSError:
@@ -220,7 +231,7 @@ def run(self):
         while not self.should_reload:
             to_delete = set(watches)
             paths = _find_common_roots(
-                _find_observable_paths(self.extra_files))
+                _find_observable_paths(self.reloader_paths, self.extra_files))
             for path in paths:
                 if path not in watches:
                     try:
@@ -256,10 +267,10 @@ def run(self):
 
 
 def run_with_reloader(main_func, extra_files=None, interval=1,
-                      reloader_type='auto'):
+                      reloader_type='auto', reloader_paths=None):
     """Run the given function in an independent python interpreter."""
     import signal
-    reloader = reloader_loops[reloader_type](extra_files, interval)
+    reloader = reloader_loops[reloader_type](reloader_paths, extra_files, interval)
     signal.signal(signal.SIGTERM, lambda *args: sys.exit(0))
     try:
         if os.environ.get('WERKZEUG_RUN_MAIN') == 'true':
diff --git a/werkzeug/serving.py b/werkzeug/serving.py
--- a/werkzeug/serving.py
+++ b/werkzeug/serving.py
@@ -524,8 +524,8 @@ def is_running_from_reloader():
 def run_simple(hostname, port, application, use_reloader=False,
                use_debugger=False, use_evalex=True,
                extra_files=None, reloader_interval=1,
-               reloader_type='auto', threaded=False, processes=1,
-               request_handler=None, static_files=None,
+               reloader_type='auto', reloader_paths=None, threaded=False,
+               processes=1, request_handler=None, static_files=None,
                passthrough_errors=False, ssl_context=None):
     """Start a WSGI application. Optional features include a reloader,
     multithreading and fork support.
@@ -567,6 +567,9 @@ def run_simple(hostname, port, application, use_reloader=False,
     :param reloader_type: the type of reloader to use.  The default is
                           auto detection.  Valid values are ``'stat'`` and
                           ``'watchdog'``.
+    :param reloader_paths: directories the reloader should watch.  Defaults to
+                           ``None`` which means automatically detect modules.
+                           Valid values are a list of paths or ``None``.
     :param threaded: should the process handle each request in a separate
                      thread?
     :param processes: if greater than 1 then handle each request in a new process
@@ -619,7 +622,7 @@ def inner():
 
         from ._reloader import run_with_reloader
         run_with_reloader(inner, extra_files, reloader_interval,
-                          reloader_type)
+                          reloader_type, reloader_paths)
     else:
         inner()
 


Test patch: