Repository: getpelican/pelican

Issue:
"pelican -lr" fails on Windows
Hi @justinmayer !

I just installed pelican from the [latest commit](https://github.com/getpelican/pelican/commit/8f68551454157aab139f0c59b772fc0b4c635c42) and found `pelican -lr` to not be working on Windows. Either works, on the other hand (`pelican -l` or `pelican -r`).

I get the following output:
~~~
$ pelican -lrD
DEBUG: Pelican version: 3.7.2.dev0
DEBUG: Python version: 3.6.2
DEBUG: Adding current directory to system path
DEBUG: Temporarily adding PLUGIN_PATHS to system path
DEBUG: Restoring system path
CRITICAL: TypeError: can't pickle generator objects
Traceback (most recent call last):
  File "c:\users\jonathan hale\appdata\local\programs\python\python36\lib\runpy.                                                                                              py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\jonathan hale\appdata\local\programs\python\python36\lib\runpy.                                                                                              py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\Jonathan Hale\AppData\Local\Programs\Python\Python36\Scripts\pe                                                                                              lican.exe\__main__.py", line 9, in <module>
  File "c:\users\jonathan hale\appdata\local\programs\python\python36\lib\site-p                                                                                              ackages\pelican\__init__.py", line 563, in main
    p1.start()
  File "c:\users\jonathan hale\appdata\local\programs\python\python36\lib\multip                                                                                              rocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "c:\users\jonathan hale\appdata\local\programs\python\python36\lib\multip                                                                                              rocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "c:\users\jonathan hale\appdata\local\programs\python\python36\lib\multip                                                                                              rocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "c:\users\jonathan hale\appdata\local\programs\python\python36\lib\multip                                                                                              rocessing\popen_spawn_win32.py", line 65, in __init__
    reduction.dump(process_obj, to_child)
  File "c:\users\jonathan hale\appdata\local\programs\python\python36\lib\multip                                                                                              rocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: can't pickle generator objects
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "c:\users\jonathan hale\appdata\local\programs\python\python36\lib\multip                                                                                              rocessing\spawn.py", line 99, in spawn_main
    new_handle = reduction.steal_handle(parent_pid, pipe_handle)
  File "c:\users\jonathan hale\appdata\local\programs\python\python36\lib\multip                                                                                              rocessing\reduction.py", line 87, in steal_handle
    _winapi.DUPLICATE_SAME_ACCESS | _winapi.DUPLICATE_CLOSE_SOURCE)
PermissionError: [WinError 5] Zugriff verweigert

~~~

("Zugriff verweigert" means "Access denied")

This is on a basic `pelican-quickstart` project, no additional plugins, content or custom theme.

Though this may be important for #2162 .

Cheers, Jonathan.


Discussion:
Hi Jonathan. I don't use Windows, so I unfortunately won't be of much assistance. Did you modify the settings file in any way?
No, this is a untouched quickstart project.
The error refers to "pickle", which could be related to content caching, which if memory serves should be disabled by default. You could try explicitly disabling caching, but beyond that I'm out of ideas at the moment.
Disabling caching with

~~~
CACHE_CONTENT = False
LOAD_CONTENT_CACHE = False
~~~

or `--ignore-cache` both did not seem to help. I will see if I can find something later today.
Debugging this results in an exception in multiprocessing/spawn.py around the following code:

~~~python
def _check_not_importing_main():
    if getattr(process.current_process(), '_inheriting', False):
        raise RuntimeError('''
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.''')
~~~

Any ideas? (Adding that "idiom" did not fully fix the error. Now it raises the error in [p1.start()](https://github.com/getpelican/pelican/blob/master/pelican/__init__.py#L563) )
Alright, so it looks like on windows it uses pickle to serialize objects and send them to the child process.
With it the arguments [passed with watchers](https://github.com/getpelican/pelican/blob/master/pelican/__init__.py#L558), which contains `folder watchers`, which are generator objects that are not pickleable, that's why it gives us the

> CRITICAL: TypeError: can't pickle generator objects

Now, should this be handled? ~I have no experience with pickle, it looks a bit as if I manually would serialize/deserialize every attribute... for Generator and every subclass? 🤔~

Generator objects are not related to `Generator` class in `generator.py`. Instead they are some python specific thing. In this case they probably hold a native Windows resource that can not be ~simply~ trivially serialized/deserialized.

(I'm guessing on linux the serialization is handled properly, or the handle to the file watching "thing" is easily shared between processes.... ?)
Alright, so my suggestion is to have a `autoreload_process` function which finds its own watchers and then calls `autoreload` rather than passing `autoreload` directly to `multiprocess.Process(target=here)`. That would omit the watchers argument and we would not need to serialize it.
I sketched this out and it *would* work, but is rather ugly. Potentially better idea:

~~~python
        if args.autoreload and args.listen:
            excqueue = multiprocessing.Queue()
            p1 = multiprocessing.Process(target=autoreload, ...)
            p2 = multiprocessing.Process(target=listen, ...)
            p1.start()
            p2.start()
            exc = excqueue.get()
            p1.terminate()
            p2.terminate()
            logger.critical(exc)
~~~

This currently spawns two processes, one for listen, one for autoreload. How about having one of the two be done by the current process?
Or maybe use threads instead? (Or are they not as portable? 🤔 )

For any of both I could create a pullrequest once we made a decision.
The real error might be `CRITICAL: TypeError: can't pickle generator objects` due to `watchers` containing generator objects.

`multiprocessing` uses `fork` on Unix systems, but on Windows it runs a new process and pickles the arguments.
It looks like pickling is the real issue like @segevfiner said. I did some looking and found [this](https://web.archive.org/web/20160112050840/http://metaoptimize.com/blog/2009/12/22/why-cant-you-pickle-generators-in-python-workaround-pattern-for-saving-training-state/) workaround, and [this](https://peadrop.com/blog/2009/12/29/why-you-cannot-pickle-generators/) response.

The response was written by a CPython dev who recommends refactoring the generator into an iterator if custom `__getstate__` and `__setstate__` methods can't be used.

I'm new to writing iterators, but looking at `file_watcher` and `folder_watcher`, it doesn't look like it would be too difficult to refactor the generators into iterators.
Another option is refactoring so that you only start the generators in the subprocess. Passing the subprocess the configuration for how to start them.

This should be quite sinpler to do I think.
That's a far better idea, I'll try that before trying to write iterators.
I couldn't get it to work by starting the generators in the subprocess, so I tried the iterator method. It worked (ish), but it turns out that you can't pickle `dict_keys` objects either.

Looks like this is gonna be a pain :/
Any news on this?
@danieleteti In a comment above I mentioned I have some changes for something that could work, but it's quite a while ago, so I don't remember any details. I hope I still have those changes lying around in case someone wants me to PR those.

Since back in August the interest in this issue was pretty low I opted for `pelican -Dr` and a local server in the `output/` folder instead. That works just as fine.

Yes, currently I'm working in the same way (`pelican -Dr` and `python -m http.server` in the `output/` folder) but would be nice, expecially for newcomers, to have something that *just works* instead of a workaround.
Do we have any progress on this?
I don't know if this helps anyone else, but I have 2 scripts that contain the (pelican -Dr) and the (python -m http.server) parts build.bat and server.bat.  While the extra command windows are a bit annoying, I find that "start build.bat" and "start server.bat" accomplishes what I need.
Hi @jackdewinter. As with most other projects, if there were any updates regarding this issue, presumably they would be posted here already.

To summarize, I don't believe any Pelican maintainers currently use Windows — and perhaps don't even have access to computers running Windows. Ergo, it's up to the community to decide whether a reasonably elegant solution can be devised. If and when that day comes, we would be happy to review said contributions. Until that time, I think some further documentation is warranted, if only to explain that adding both `--autoreload` and `--listen` to a single `pelican` invocation on Windows does not currently behave as the user might expect. I've taken the liberty of doing that, in the most succinct way possible, via: 9fdcb90
I'm a hugo user, but I prefer python, so I thought I would give this a chance. The fact that this issue is 1.5 years old and not fixed is really kind of surprising. No one has to support windows of course, but i just thought I would mention that the experience on Windows surely is discouraging many potential users. Best wishes, looks like a cool tool from the outside at least...
I'm a former pelican user... for it's lack of interest in supporting windows, I'm switching to Hugo... 
I would to use pelican because it's python, but it's not possible.
@thetanil / @danieleteti: Honestly, folks, I find your comments to be disheartening. At the very least, they are inaccurate and show a lack of understanding as how open-source projects work, and at worst, they exhibit a lack of consideration as to how such comments can hurt the feelings of those who donate, for free, their time and effort to maintain said open-source projects.

Contrary to the casually-flung supposition, as far as I am aware, the experience on Windows is not discouraging potential users in significant numbers. In fact, many Windows users happily use Pelican to publish their sites, every day, without any problems whatsoever. Presumably they do so in the same way that I have personally done for years, which is to regenerate in one terminal pane and serve the output in another. This works beautifully and has been the primary way folks have used Pelican for a very long time.

Then one day someone contributed a pull request that added the ability to serve output from within Pelican. That pull request was reviewed and merged. Yay.

Then @Squareys discovered that combining that feature with the regenerate option on Windows resulted in an error. He not only helpfully reported it, but he spent a considerable amount of time working on a potential solution. I regret that he did not receive timely feedback from anyone on his work. Personally, at the time I had recently relocated and had other priorities to attend to.

To suggest that Pelican maintainers lack interest in supporting Windows is not only false, it’s insulting. If someone submits a pull request that addresses a problem, includes tests, and meets the community’s standards, it is merged. This issue is no different in that respect. Contributions that address this issue are welcome.

In short, please take care with your words. Greater attention to accuracy and empathy in this case would have spared some hurt feelings.
Do any Pelican users here on Windows use the [live-reload Invoke task](https://github.com/getpelican/pelican/pull/2526)? If so, how well does it work for you? If you use Windows and haven't tried it, would you mind giving it a try and reporting the results here?
@justinmayer I really tried to write that comment in the least "choosy begger" way I could. I work open source projects as well, I know how you feel. I think when we work on these projects we also get blind spots because we know how it works. I will try again this way.

My experience as a new user: Find some recommendation, go to the web site, read some docs, follow the quick-start. It doesn't work. Search it up. Land here.

So from my point of view, as a Windows user I did everything right and I still need to build hacky stuff myself to make it work. That's fine, it's open source, I could fix it myself, I understand that. I just thought maybe it's a blind spot and I should at least mention it since the issue is so old.

Just to be clear and perhaps more irritating, I am not even suggesting a technical change. If there was just some advice on the [install docs](https://docs.getpelican.com/en/stable/install.html)

*Windows Users:*
- Don't use `make devserver`
- Don't use pelican -lr
- instead do x/y

And I have to say, even after reading through this issue, as a new user I am unsure what x/y should be. I see there are a couple workarounds, but I do not know what is correct or best.
Now _those_ are useful and actionable suggestions. I agree that should be made clearer. Let's collectively figure out a way to do that.

Since I personally use Invoke rather than Make, I didn't even realize that `pelican -lr` was being called when users run [`make devserver`](https://github.com/getpelican/pelican/blob/6f0743b340cfaa98ced38fc41727c8ea9f1a52f8/pelican/tools/templates/Makefile.jinja2#L110-L115). Windows users who use Make should indeed be warned that task will not behave properly.

As for what Windows users _should_ do, I think the first step is determining whether the aforementioned Invoke `livereload` task works properly on Windows. Anyone care to try and report back?
`invoke livereload` is working for me with Windows 10, Python 3.7, Pelican 4.2.0, livereload 2.6.1 and the output of a fresh pelican-quickstart.

(I did get tripped up for a bit on md/rst files in `content/` not triggering the refresh- #2604.  https://github.com/lepture/python-livereload/pull/203 is still open, but there are workarounds in that issue. I just removed the wildcard from my local `tasks.py` for now since I don't use subdirectories in `content/`)
i tried notes:
- no invoke command. not in pelican docs. found it and installed
- no livereload module. not in docs. found it installed.
- then I got a traceback not implemented. i think it's cause my content folder is messed up locally
- ran out of time. this is just fyi
https://docs.getpelican.com/en/stable/publish.html#invoke
@Squareys: Could you possibly take a look and see whether you still have the changes you worked on related to this issue? They might come in handy. Thanks, Jonathan!

Resolving patch:
diff --git a/pelican/__init__.py b/pelican/__init__.py
--- a/pelican/__init__.py
+++ b/pelican/__init__.py
@@ -24,8 +24,7 @@
 from pelican.readers import Readers
 from pelican.server import ComplexHTTPRequestHandler, RootedHTTPServer
 from pelican.settings import read_settings
-from pelican.utils import (clean_output_dir, file_watcher,
-                           folder_watcher, maybe_pluralize)
+from pelican.utils import (FileSystemWatcher, clean_output_dir, maybe_pluralize)
 from pelican.writers import Writer
 
 try:
@@ -381,65 +380,36 @@ def get_instance(args):
     return cls(settings), settings
 
 
-def autoreload(watchers, args, old_static, reader_descs, excqueue=None):
+def autoreload(args, excqueue=None):
+    print('  --- AutoReload Mode: Monitoring `content`, `theme` and'
+          ' `settings` for changes. ---')
+    pelican, settings = get_instance(args)
+    watcher = FileSystemWatcher(args.settings, Readers, settings)
+    sleep = False
     while True:
         try:
-            # Check source dir for changed files ending with the given
-            # extension in the settings. In the theme dir is no such
-            # restriction; all files are recursively checked if they
-            # have changed, no matter what extension the filenames
-            # have.
-            modified = {k: next(v) for k, v in watchers.items()}
+            # Don't sleep first time, but sleep afterwards to reduce cpu load
+            if sleep:
+                time.sleep(0.5)
+            else:
+                sleep = True
+
+            modified = watcher.check()
 
             if modified['settings']:
                 pelican, settings = get_instance(args)
-
-                # Adjust static watchers if there are any changes
-                new_static = settings.get("STATIC_PATHS", [])
-
-                # Added static paths
-                # Add new watchers and set them as modified
-                new_watchers = set(new_static).difference(old_static)
-                for static_path in new_watchers:
-                    static_key = '[static]%s' % static_path
-                    watchers[static_key] = folder_watcher(
-                        os.path.join(pelican.path, static_path),
-                        [''],
-                        pelican.ignore_files)
-                    modified[static_key] = next(watchers[static_key])
-
-                # Removed static paths
-                # Remove watchers and modified values
-                old_watchers = set(old_static).difference(new_static)
-                for static_path in old_watchers:
-                    static_key = '[static]%s' % static_path
-                    watchers.pop(static_key)
-                    modified.pop(static_key)
-
-                # Replace old_static with the new one
-                old_static = new_static
+                watcher.update_watchers(settings)
 
             if any(modified.values()):
                 print('\n-> Modified: {}. re-generating...'.format(
                     ', '.join(k for k, v in modified.items() if v)))
-
-                if modified['content'] is None:
-                    logger.warning(
-                        'No valid files found in content for '
-                        + 'the active readers:\n'
-                        + '\n'.join(reader_descs))
-
-                if modified['theme'] is None:
-                    logger.warning('Empty theme folder. Using `basic` '
-                                   'theme.')
-
                 pelican.run()
 
-        except KeyboardInterrupt as e:
-            logger.warning("Keyboard interrupt, quitting.")
+        except KeyboardInterrupt:
             if excqueue is not None:
-                excqueue.put(traceback.format_exception_only(type(e), e)[-1])
-            return
+                excqueue.put(None)
+                return
+            raise
 
         except Exception as e:
             if (args.verbosity == logging.DEBUG):
@@ -449,10 +419,8 @@ def autoreload(watchers, args, old_static, reader_descs, excqueue=None):
                 else:
                     raise
             logger.warning(
-                'Caught exception "%s". Reloading.', e)
-
-        finally:
-            time.sleep(.5)  # sleep to avoid cpu load
+                'Caught exception:\n"%s".', e,
+                exc_info=settings.get('DEBUG', False))
 
 
 def listen(server, port, output, excqueue=None):
@@ -476,8 +444,10 @@ def listen(server, port, output, excqueue=None):
         return
 
     except KeyboardInterrupt:
-        print("\nKeyboard interrupt received. Shutting down server.")
         httpd.socket.close()
+        if excqueue is not None:
+            return
+        raise
 
 
 def main(argv=None):
@@ -492,37 +462,11 @@ def main(argv=None):
     try:
         pelican, settings = get_instance(args)
 
-        readers = Readers(settings)
-        reader_descs = sorted(
-            {
-                '%s (%s)' % (type(r).__name__, ', '.join(r.file_extensions))
-                for r in readers.readers.values()
-                if r.enabled
-            }
-        )
-
-        watchers = {'content': folder_watcher(pelican.path,
-                                              readers.extensions,
-                                              pelican.ignore_files),
-                    'theme': folder_watcher(pelican.theme,
-                                            [''],
-                                            pelican.ignore_files),
-                    'settings': file_watcher(args.settings)}
-
-        old_static = settings.get("STATIC_PATHS", [])
-        for static_path in old_static:
-            # use a prefix to avoid possible overriding of standard watchers
-            # above
-            watchers['[static]%s' % static_path] = folder_watcher(
-                os.path.join(pelican.path, static_path),
-                [''],
-                pelican.ignore_files)
-
         if args.autoreload and args.listen:
             excqueue = multiprocessing.Queue()
             p1 = multiprocessing.Process(
                 target=autoreload,
-                args=(watchers, args, old_static, reader_descs, excqueue))
+                args=(args, excqueue))
             p2 = multiprocessing.Process(
                 target=listen,
                 args=(settings.get('BIND'), settings.get('PORT'),
@@ -532,26 +476,19 @@ def main(argv=None):
             exc = excqueue.get()
             p1.terminate()
             p2.terminate()
-            logger.critical(exc)
+            if exc is not None:
+                logger.critical(exc)
         elif args.autoreload:
-            print('  --- AutoReload Mode: Monitoring `content`, `theme` and'
-                  ' `settings` for changes. ---')
-            autoreload(watchers, args, old_static, reader_descs)
+            autoreload(args)
         elif args.listen:
             listen(settings.get('BIND'), settings.get('PORT'),
                    settings.get("OUTPUT_PATH"))
         else:
-            if next(watchers['content']) is None:
-                logger.warning(
-                    'No valid files found in content for '
-                    + 'the active readers:\n'
-                    + '\n'.join(reader_descs))
-
-            if next(watchers['theme']) is None:
-                logger.warning('Empty theme folder. Using `basic` theme.')
-
+            watcher = FileSystemWatcher(args.settings, Readers, settings)
+            watcher.check()
             pelican.run()
-
+    except KeyboardInterrupt:
+        logger.warning('Keyboard interrupt received. Exiting.')
     except Exception as e:
         logger.critical('%s', e)
 
diff --git a/pelican/utils.py b/pelican/utils.py
--- a/pelican/utils.py
+++ b/pelican/utils.py
@@ -734,60 +734,167 @@ def order_content(content_list, order_by='slug'):
     return content_list
 
 
-def folder_watcher(path, extensions, ignores=[]):
-    '''Generator for monitoring a folder for modifications.
-
-    Returns a boolean indicating if files are changed since last check.
-    Returns None if there are no matching files in the folder'''
-
-    def file_times(path):
-        '''Return `mtime` for each file in path'''
-
-        for root, dirs, files in os.walk(path, followlinks=True):
-            dirs[:] = [x for x in dirs if not x.startswith(os.curdir)]
-
-            for f in files:
-                valid_extension = f.endswith(tuple(extensions))
-                file_ignored = any(
-                    fnmatch.fnmatch(f, ignore) for ignore in ignores
-                )
-                if valid_extension and not file_ignored:
-                    try:
-                        yield os.stat(os.path.join(root, f)).st_mtime
-                    except OSError as e:
-                        logger.warning('Caught Exception: %s', e)
-
-    LAST_MTIME = 0
-    while True:
-        try:
-            mtime = max(file_times(path))
-            if mtime > LAST_MTIME:
-                LAST_MTIME = mtime
-                yield True
-        except ValueError:
-            yield None
+class FileSystemWatcher:
+    def __init__(self, settings_file, reader_class, settings=None):
+        self.watchers = {
+            'settings': FileSystemWatcher.file_watcher(settings_file)
+        }
+
+        self.settings = None
+        self.reader_class = reader_class
+        self._extensions = None
+        self._content_path = None
+        self._theme_path = None
+        self._ignore_files = None
+
+        if settings is not None:
+            self.update_watchers(settings)
+
+    def update_watchers(self, settings):
+        new_extensions = set(self.reader_class(settings).extensions)
+        new_content_path = settings.get('PATH', '')
+        new_theme_path = settings.get('THEME', '')
+        new_ignore_files = set(settings.get('IGNORE_FILES', []))
+
+        extensions_changed = new_extensions != self._extensions
+        content_changed = new_content_path != self._content_path
+        theme_changed = new_theme_path != self._theme_path
+        ignore_changed = new_ignore_files != self._ignore_files
+
+        # Refresh content watcher if related settings changed
+        if extensions_changed or content_changed or ignore_changed:
+            self.add_watcher('content',
+                             new_content_path,
+                             new_extensions,
+                             new_ignore_files)
+
+        # Refresh theme watcher if related settings changed
+        if theme_changed or ignore_changed:
+            self.add_watcher('theme',
+                             new_theme_path,
+                             [''],
+                             new_ignore_files)
+
+        # Watch STATIC_PATHS
+        old_static_watchers = set(key
+                                  for key in self.watchers
+                                  if key.startswith('[static]'))
+
+        for path in settings.get('STATIC_PATHS', []):
+            key = '[static]{}'.format(path)
+            if ignore_changed or (key not in self.watchers):
+                self.add_watcher(
+                    key,
+                    os.path.join(new_content_path, path),
+                    [''],
+                    new_ignore_files)
+            if key in old_static_watchers:
+                old_static_watchers.remove(key)
+
+        # cleanup removed static watchers
+        for key in old_static_watchers:
+            del self.watchers[key]
+
+        # update values
+        self.settings = settings
+        self._extensions = new_extensions
+        self._content_path = new_content_path
+        self._theme_path = new_theme_path
+        self._ignore_files = new_ignore_files
+
+    def check(self):
+        '''return a key:watcher_status dict for all watchers'''
+        result = {key: next(watcher) for key, watcher in self.watchers.items()}
+
+        # Various warnings
+        if result.get('content') is None:
+            reader_descs = sorted(
+                {
+                    '%s (%s)' % (type(r).__name__, ', '.join(r.file_extensions))
+                    for r in self.reader_class(self.settings).readers.values()
+                    if r.enabled
+                }
+            )
+            logger.warning(
+                    'No valid files found in content for the active readers:\n'
+                    + '\n'.join(reader_descs))
+
+        if result.get('theme') is None:
+            logger.warning('Empty theme folder. Using `basic` theme.')
+
+        return result
+
+    def add_watcher(self, key, path, extensions=[''], ignores=[]):
+        watcher = self.get_watcher(path, extensions, ignores)
+        if watcher is not None:
+            self.watchers[key] = watcher
+
+    def get_watcher(self, path, extensions=[''], ignores=[]):
+        '''return a watcher depending on path type (file or folder)'''
+        if not os.path.exists(path):
+            logger.warning("Watched path does not exist: %s", path)
+            return None
+
+        if os.path.isdir(path):
+            return self.folder_watcher(path, extensions, ignores)
         else:
-            yield False
+            return self.file_watcher(path)
+
+    @staticmethod
+    def folder_watcher(path, extensions, ignores=[]):
+        '''Generator for monitoring a folder for modifications.
 
+        Returns a boolean indicating if files are changed since last check.
+        Returns None if there are no matching files in the folder'''
 
-def file_watcher(path):
-    '''Generator for monitoring a file for modifications'''
-    LAST_MTIME = 0
-    while True:
-        if path:
+        def file_times(path):
+            '''Return `mtime` for each file in path'''
+
+            for root, dirs, files in os.walk(path, followlinks=True):
+                dirs[:] = [x for x in dirs if not x.startswith(os.curdir)]
+
+                for f in files:
+                    valid_extension = f.endswith(tuple(extensions))
+                    file_ignored = any(
+                        fnmatch.fnmatch(f, ignore) for ignore in ignores
+                    )
+                    if valid_extension and not file_ignored:
+                        try:
+                            yield os.stat(os.path.join(root, f)).st_mtime
+                        except OSError as e:
+                            logger.warning('Caught Exception: %s', e)
+
+        LAST_MTIME = 0
+        while True:
             try:
-                mtime = os.stat(path).st_mtime
-            except OSError as e:
-                logger.warning('Caught Exception: %s', e)
-                continue
-
-            if mtime > LAST_MTIME:
-                LAST_MTIME = mtime
-                yield True
+                mtime = max(file_times(path))
+                if mtime > LAST_MTIME:
+                    LAST_MTIME = mtime
+                    yield True
+            except ValueError:
+                yield None
             else:
                 yield False
-        else:
-            yield None
+
+    @staticmethod
+    def file_watcher(path):
+        '''Generator for monitoring a file for modifications'''
+        LAST_MTIME = 0
+        while True:
+            if path:
+                try:
+                    mtime = os.stat(path).st_mtime
+                except OSError as e:
+                    logger.warning('Caught Exception: %s', e)
+                    continue
+
+                if mtime > LAST_MTIME:
+                    LAST_MTIME = mtime
+                    yield True
+                else:
+                    yield False
+            else:
+                yield None
 
 
 def set_date_tzinfo(d, tz_name=None):


Test patch:
diff --git a/pelican/tests/test_utils.py b/pelican/tests/test_utils.py
--- a/pelican/tests/test_utils.py
+++ b/pelican/tests/test_utils.py
@@ -10,6 +10,7 @@
 
 from pelican import utils
 from pelican.generators import TemplatePagesGenerator
+from pelican.readers import Readers
 from pelican.settings import read_settings
 from pelican.tests.support import (LoggedTestCase, get_article,
                                    locale_available, unittest)
@@ -361,47 +362,91 @@ def test_process_translations(self):
                             self.assertNotIn(a_arts[4], b_arts[5].translations)
                             self.assertNotIn(a_arts[5], b_arts[4].translations)
 
-    def test_watchers(self):
-        # Test if file changes are correctly detected
-        # Make sure to handle not getting any files correctly.
-
-        dirname = os.path.join(os.path.dirname(__file__), 'content')
-        folder_watcher = utils.folder_watcher(dirname, ['rst'])
-
-        path = os.path.join(dirname, 'article_with_metadata.rst')
-        file_watcher = utils.file_watcher(path)
-
-        # first check returns True
-        self.assertEqual(next(folder_watcher), True)
-        self.assertEqual(next(file_watcher), True)
-
-        # next check without modification returns False
-        self.assertEqual(next(folder_watcher), False)
-        self.assertEqual(next(file_watcher), False)
-
-        # after modification, returns True
-        t = time.time()
-        os.utime(path, (t, t))
-        self.assertEqual(next(folder_watcher), True)
-        self.assertEqual(next(file_watcher), True)
-
-        # file watcher with None or empty path should return None
-        self.assertEqual(next(utils.file_watcher('')), None)
-        self.assertEqual(next(utils.file_watcher(None)), None)
-
-        empty_path = os.path.join(os.path.dirname(__file__), 'empty')
-        try:
-            os.mkdir(empty_path)
-            os.mkdir(os.path.join(empty_path, "empty_folder"))
-            shutil.copy(__file__, empty_path)
-
-            # if no files of interest, returns None
-            watcher = utils.folder_watcher(empty_path, ['rst'])
-            self.assertEqual(next(watcher), None)
-        except OSError:
-            self.fail("OSError Exception in test_files_changed test")
-        finally:
-            shutil.rmtree(empty_path, True)
+    def test_filesystemwatcher(self):
+        def create_file(name, content):
+            with open(name, 'w') as f:
+                f.write(content)
+
+        # disable logger filter
+        from pelican.utils import logger
+        logger.disable_filter()
+
+        # create a temp "project" dir
+        root = mkdtemp()
+        content_path = os.path.join(root, 'content')
+        static_path = os.path.join(root, 'content', 'static')
+        config_file = os.path.join(root, 'config.py')
+        theme_path = os.path.join(root, 'mytheme')
+
+        # populate
+        os.mkdir(content_path)
+        os.mkdir(theme_path)
+        create_file(config_file,
+                    'PATH = "content"\n'
+                    'THEME = "mytheme"\n'
+                    'STATIC_PATHS = ["static"]')
+
+        t = time.time() - 1000  # make sure it's in the "past"
+        os.utime(config_file, (t, t))
+        settings = read_settings(config_file)
+
+        watcher = utils.FileSystemWatcher(config_file, Readers, settings)
+        # should get a warning for static not not existing
+        self.assertLogCountEqual(1, 'Watched path does not exist: .*static')
+
+        # create it and update config
+        os.mkdir(static_path)
+        watcher.update_watchers(settings)
+        # no new warning
+        self.assertLogCountEqual(1, 'Watched path does not exist: .*static')
+
+        # get modified values
+        modified = watcher.check()
+        # empty theme and content should raise warnings
+        self.assertLogCountEqual(1, 'No valid files found in content')
+        self.assertLogCountEqual(1, 'Empty theme folder. Using `basic` theme')
+
+        self.assertIsNone(modified['content'])  # empty
+        self.assertIsNone(modified['theme'])  # empty
+        self.assertIsNone(modified['[static]static'])  # empty
+        self.assertTrue(modified['settings'])  # modified, first time
+
+        # add a content, add file to theme and check again
+        create_file(os.path.join(content_path, 'article.md'),
+                    'Title: test\n'
+                    'Date: 01-01-2020')
+
+        create_file(os.path.join(theme_path, 'dummy'),
+                    'test')
+
+        modified = watcher.check()
+        # no new warning
+        self.assertLogCountEqual(1, 'No valid files found in content')
+        self.assertLogCountEqual(1, 'Empty theme folder. Using `basic` theme')
+
+        self.assertIsNone(modified['[static]static'])  # empty
+        self.assertFalse(modified['settings'])  # not modified
+        self.assertTrue(modified['theme'])  # modified
+        self.assertTrue(modified['content'])  # modified
+
+        # change config, remove static path
+        create_file(config_file,
+                    'PATH = "content"\n'
+                    'THEME = "mytheme"\n'
+                    'STATIC_PATHS = []')
+
+        settings = read_settings(config_file)
+        watcher.update_watchers(settings)
+
+        modified = watcher.check()
+        self.assertNotIn('[static]static', modified)  # should be gone
+        self.assertTrue(modified['settings'])  # modified
+        self.assertFalse(modified['content'])  # not modified
+        self.assertFalse(modified['theme'])  # not modified
+
+        # cleanup
+        logger.enable_filter()
+        shutil.rmtree(root)
 
     def test_clean_output_dir(self):
         retention = ()