Repository: pallets/werkzeug

Issue:
Werkzeug stat reloader problems
So far two people have run into serious performance issues with the stat reloader. Anybody who has those, **the fix is to install watchdog from pypi**.

If you're interested in fixing this issue, uninstall watchdog again and:
- How does setting `reloader_interval` on `werkzeug.run_simple` or `app.run` affect performance? Try various values:
  
  ```
  app.run(reloader_interval=1)  # the default
  ```
- Run this code inside your virtualenv, post the returned number:
  
  ```
  from werkzeug._reloader import _iter_module_files
  print(len(list(_iter_module_files())))
  ```



Discussion:
cc @joostdevries 

Increasing the reloader_interval only increases the time between cpu usage spikes. With watchdog installed, the cpu usage is almost non-existent!

```
┌[txtsd@dungeon-of-data]─[~/sites/flasktut] 
└─▶ flask/bin/python
Python 3.4.2 (default, Jan 12 2015, 11:38:40) 
[GCC 4.9.2 20141224 (prerelease)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from werkzeug._reloader import _iter_module_files
>>> print(len(list(_iter_module_files())))
10893
>>> 
```

I think it'd be sensible to re-introduce the old behavior of only scanning `sys.modules` for the `stat` reloader. It is incorrect, but more importantly, it is fast without any dependencies. If people encounter issues with unwatched files (as in #569), we can simply tell them to use watchdog.

@untitaker I'd just recommend people to use `reloader_paths` (once released). In quite a lot of cases, `sys.modules` will still yield a huge list of files. I do think there's a regression in the most recent version, in my case a CPU usage increase of about 30%. 

@joostdevries `reloader_paths` is a workaround to a problem that shouldn't have been introduced in the first place. The difference in the algorithm for finding the files to watch is more dramatic than you might think: Werkzeug has to traverse all directories recursively, launching `stat` for anything beneath them.

I noticed this issue right away when my laptop battery lasted only a few hours because werkzeug was constantly taking 20% CPU. Installing watchdog fixed the issue, but would be great to not have defaults with such horrible performance...

Here is the result of `_iter_module_files` on my machine:

```
$ python
Python 2.7.6 (default, Sep  9 2014, 15:04:36)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from werkzeug._reloader import _iter_module_files
>>> print(len(list(_iter_module_files())))
2695
```

Ok, probably makes sense to reintroduce the old behavior. iter-module-files yields 200k items for me and I haven't noticed the perf issue on my desktop.

i do wonder if it would be so bad to depend on a optimized package for reloading in general in any case

On Thu, Mar 05, 2015 at 07:47:06AM -0800, Ronny Pfannschmidt wrote:

> i do wonder if it would be so bad to depend on a optimized package for
> reloading in general in any case

Some filesystems don't have filesystem events (there are open issues about some
setup in Vagrant that is affected by this). I think Watchdog also has a stat
reloader available, but I don't think it's worth depending on Watchdog for that
one.

in that case i do wonder if the relaoding behavior could be enhanced if stuff like dot-files/hidden files and virtualenvs could be ignored along with anything not in the cwd

Hi, I ran into this thread after we noticed high CPU usage figures on idle (30-50%). I tried the watchdog suggestion which worked fine initially, but after a short period of time no pages would be served up by the web server, even after subsequent process restarts and killing zombie python processes.

I have another problem with stat reloader. With version 0.10 if I run this simple app:

``` python
from flask import Flask
import requests
app = Flask(__name__)
@app.route('/')
def index():
     return 'Index'
app.run(debug=True, host='127.0.0.1')
```

then I've got the following traceback:

```
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat
Traceback (most recent call last):
  File "app.py", line 7, in <module>
    app.run(debug = True, host='127.0.0.1')
  File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 772, in run
    run_simple(host, port, self, **options)
  File "/usr/local/lib/python3.4/dist-packages/werkzeug/serving.py", line 622, in run_simple
    reloader_type)
  File "/usr/local/lib/python3.4/dist-packages/werkzeug/_reloader.py", line 269, in run_with_reloader
    reloader.run()
  File "/usr/local/lib/python3.4/dist-packages/werkzeug/_reloader.py", line 159, in run
    for filename in chain(_iter_module_files(), self.extra_files):
  File "/usr/local/lib/python3.4/dist-packages/werkzeug/_reloader.py", line 70, in _iter_module_files
    for package_path in getattr(module, '__path__', ()):
  File "/usr/lib/python3/dist-packages/six.py", line 116, in __getattr__
    _module = self._resolve()
  File "/usr/lib/python3/dist-packages/six.py", line 105, in _resolve
    return _import_module(self.mod)
  File "/usr/lib/python3/dist-packages/six.py", line 76, in _import_module
    __import__(name)
ImportError: No module named 'winreg'
```

If I downgrade to 0.9 or remove "import requests" everything would be fine.

Everybody who has this issue:

With the stat-reloader, could you delete/comment out the following three lines in `werkzeug/_reloader`:

```
for package_path in getattr(module, '__path__', ()):
    for filename in _recursive_walk(os.path.abspath(package_path)):
        yield filename
```

How much does that affect the overall performance of the `stat`-reloader?

@ya-mouse Are you sure this is related to Flask or Werkzeug at all?

@untitaker On Werkzeug 0.10.1, after commenting out those 3 lines, I'm still seeing high CPU usage similar to the usage with the lines intact.

@untitaker I think so. Newer Werkzeug version tries to get __ path __ attribute from six.MovedModule instance (mapping for "tkinter.commondialog") or mapping for "winreg" if python3-tk has been installed.
Not all values in sys.modules are real modules. So, this problem related to Werkzeug 0.10.1.

@ya-mouse I had this problem too, it's an incompatibility between many auto-reloaders (Werkzeug's, Django's, etc) and older versions of `six`. I was able to fix it with this monkey patch in my project: https://gist.github.com/peplin/88ef5f26c4cc53160448 My patch is specific to hiding the old version of six in `pyxb` - you will need to adjust it to patch out `tkinter` instead.

Regarding the six-issue: Please open a separate issue for this.

The six issue has been already opened and was just fixed one hour ago: https://bitbucket.org/gutworth/six/issue/99/six-and-inspect-importerror-_winreg-module


Resolving patch:
diff --git a/werkzeug/_reloader.py b/werkzeug/_reloader.py
--- a/werkzeug/_reloader.py
+++ b/werkzeug/_reloader.py
@@ -14,61 +14,22 @@ def _iter_module_files():
     loaded files from modules, all files in folders of already loaded modules
     as well as all files reachable through a package.
     """
-    found = set()
-    entered = set()
-
-    def _verify_file(filename):
-        if not filename:
-            return
-        filename = os.path.abspath(filename)
-        old = None
-        while not os.path.isfile(filename):
-            old = filename
-            filename = os.path.dirname(filename)
-            if filename == old:
-                break
-        else:
-            if filename[-4:] in ('.pyc', '.pyo'):
-                filename = filename[:-1]
-            if filename not in found:
-                found.add(filename)
-                return filename
-
-    def _recursive_walk(path_entry):
-        if path_entry in entered:
-            return
-        entered.add(path_entry)
-        try:
-            for filename in os.listdir(path_entry):
-                path = os.path.join(path_entry, filename)
-                if os.path.isdir(path):
-                    for filename in _recursive_walk(path):
-                        yield filename
-                else:
-                    if not filename.endswith(('.py', '.pyc', '.pyo')):
-                        continue
-                    filename = _verify_file(path)
-                    if filename:
-                        yield filename
-        except OSError:
-            pass
-
     # The list call is necessary on Python 3 in case the module
     # dictionary modifies during iteration.
-    for path_entry in list(sys.path):
-        for filename in _recursive_walk(os.path.abspath(path_entry)):
-            yield filename
-
     for module in list(sys.modules.values()):
         if module is None:
             continue
-        filename = _verify_file(getattr(module, '__file__', None))
+        filename = getattr(module, '__file__', None)
         if filename:
-            yield filename
-            for filename in _recursive_walk(os.path.dirname(filename)):
-                yield filename
-        for package_path in getattr(module, '__path__', ()):
-            for filename in _recursive_walk(os.path.abspath(package_path)):
+            old = None
+            while not os.path.isfile(filename):
+                old = filename
+                filename = os.path.dirname(filename)
+                if filename == old:
+                    break
+            else:
+                if filename[-4:] in ('.pyc', '.pyo'):
+                    filename = filename[:-1]
                 yield filename
 
 
@@ -83,7 +44,7 @@ def _find_observable_paths(extra_files=None):
             continue
         fn = os.path.abspath(fn)
         rv.add(os.path.dirname(fn))
-    return rv
+    return _find_common_roots(rv)
 
 
 def _find_common_roots(paths):
@@ -219,8 +180,7 @@ def run(self):
 
         while not self.should_reload:
             to_delete = set(watches)
-            paths = _find_common_roots(
-                _find_observable_paths(self.extra_files))
+            paths = _find_observable_paths(self.extra_files)
             for path in paths:
                 if path not in watches:
                     try:
diff --git a/werkzeug/serving.py b/werkzeug/serving.py
--- a/werkzeug/serving.py
+++ b/werkzeug/serving.py
@@ -548,7 +548,7 @@ def run_simple(hostname, port, application, use_reloader=False,
     .. versionadded:: 0.9
        Added command-line interface.
 
-    .. versionadded:: 1.0
+    .. versionadded:: 0.10
        Improved the reloader and added support for changing the backend
        through the `reloader_type` parameter.  See :ref:`reloader`
        for more information.
@@ -566,7 +566,8 @@ def run_simple(hostname, port, application, use_reloader=False,
     :param reloader_interval: the interval for the reloader in seconds.
     :param reloader_type: the type of reloader to use.  The default is
                           auto detection.  Valid values are ``'stat'`` and
-                          ``'watchdog'``.
+                          ``'watchdog'``. See :ref:`reloader` for more
+                          information.
     :param threaded: should the process handle each request in a separate
                      thread?
     :param processes: if greater than 1 then handle each request in a new process


Test patch:
diff --git a/tests/test_serving.py b/tests/test_serving.py
--- a/tests/test_serving.py
+++ b/tests/test_serving.py
@@ -95,13 +95,15 @@ def test_make_ssl_devcert(tmpdir):
     assert os.path.isfile(private_key)
 
 
-@pytest.mark.parametrize('reloader_type', ['watchdog', 'stat'])
-def test_reloader_broken_imports(tmpdir, dev_server, reloader_type):
+@pytest.mark.skipif(watchdog is None, reason='Watchdog not installed.')
+def test_reloader_broken_imports(tmpdir, dev_server):
     # We explicitly assert that the server reloads on change, even though in
     # this case the import could've just been retried. This is to assert
     # correct behavior for apps that catch and cache import errors.
-    if reloader_type == 'watchdog' and watchdog is None:
-        pytest.skip('Watchdog not installed.')
+    #
+    # Because this feature is achieved by recursively watching a large amount
+    # of directories, this only works for the watchdog reloader. The stat
+    # reloader is too inefficient to watch such a large amount of files.
 
     real_app = tmpdir.join('real_app.py')
     real_app.write("lol syntax error")
@@ -116,8 +118,8 @@ def app(environ, start_response):
 
     kwargs['use_reloader'] = True
     kwargs['reloader_interval'] = 0.1
-    kwargs['reloader_type'] = %s
-    ''' % repr(reloader_type))
+    kwargs['reloader_type'] = 'watchdog'
+    ''')
     server.wait_for_reloader_loop()
 
     r = requests.get(server.url)
@@ -135,11 +137,8 @@ def real_app(environ, start_response):
     assert r.content == b'hello'
 
 
-@pytest.mark.parametrize('reloader_type', ['watchdog', 'stat'])
-def test_reloader_nested_broken_imports(tmpdir, dev_server, reloader_type):
-    if reloader_type == 'watchdog' and watchdog is None:
-        pytest.skip('Watchdog not installed.')
-
+@pytest.mark.skipif(watchdog is None, reason='Watchdog not installed.')
+def test_reloader_nested_broken_imports(tmpdir, dev_server):
     real_app = tmpdir.mkdir('real_app')
     real_app.join('__init__.py').write('from real_app.sub import real_app')
     sub = real_app.mkdir('sub').join('__init__.py')
@@ -155,8 +154,8 @@ def app(environ, start_response):
 
     kwargs['use_reloader'] = True
     kwargs['reloader_interval'] = 0.1
-    kwargs['reloader_type'] = %s
-    ''' % repr(reloader_type))
+    kwargs['reloader_type'] = 'watchdog'
+    ''')
     server.wait_for_reloader_loop()
 
     r = requests.get(server.url)