Repository: gorakhargosh/watchdog (Python)

Issue:
Can't pickle DirectorySnapshots
First of all, I'm not sure if this is an issue, an expected behaviour or just an scenario not aimed to be solved.

## Expected Behavior
I expect to be able to pickle DirectorySnapshot objects, as I can do in the latest release, 0.9.0.

## Current Behavior
I can't pickle a DirectorySnapshot because I'm getting a PicklingError.
It doesn't happen on the 0.9.0 version.

## Possible Solution
One possible workaround is using the `dill` library and it would work, but I find it strange to not make it work with `pickle` like it did before because it breaks existing code.

The following works:
```python
import dill
from src.watchdog.utils.dirsnapshot import DirectorySnapshot

dill.dumps(DirectorySnapshot("/Users/alex/tmp"))
```

## Steps to Reproduce
1. Clone the repository with the latest `master` commit. Currently `7a55f317946953d86a5acea64d9b31931ca25307`.
2. Try to pickle a DirectorySnapshot. Notice the import from `src` directory, not the pypi version.
```python
import pickle
from src.watchdog.utils.dirsnapshot import DirectorySnapshot

pickle.dumps(DirectorySnapshot("/Users/alex/tmp"))
```
3. You will see the following exception:
```python
_pickle.PicklingError: Can't pickle <function DirectorySnapshot.<lambda> at 0x10dfb20d0>: attribute lookup DirectorySnapshot.<lambda> on src.watchdog.utils.dirsnapshot failed
```

## Context (Environment)
I'm facing this problem while using both python 3.7 and python 3.8 in a macOS Mojave 10.14.6.

In my scenario I need to pickle a DirectorySnapshot object to later compare it with the current snapshot and see if there was any change while my program was down.

## Possible Implementation
I just propose to make it work with `pickle`. Maybe the problematic lambda is not required and we can just avoid serializing it using `__getstate__`.

Otherwise it just doesn't have a solution and we have to change to using the `dill` library.




Discussion:
Thanks for the report @Ajordat :)

---

Using `git bisect` shows that the commit that introduced the error is ecaa92756fe78d4771d23550c0935b0190769035: Snapshot: don't walk directories without read permissions (#573).
@danilobellini If I remove the `walker_callback` argument, it works. As this argument is deprecated since 0.7.2, maybe it is also time to remove it, WDYT?
IMO if a deprecated feature is causing problems, it should be removed.

What I don't know is, if it has been deprecated since 0.7.2, why it didn't cause any problems on 0.9.0 and causes them now?
Because with the commit ecaa927 the lambda is now assigned to an attribute of the class. This is causing issues with pickle. I will open a PR to remove the deprecated code :)



Patch:
diff --git a/changelog.rst b/changelog.rst
index e212e70e5..4cb688ad9 100644
--- a/changelog.rst
+++ b/changelog.rst
@@ -13,6 +13,8 @@ Breaking Changes
 
 - Dropped support for Python 2.6, 3.2 and 3.3
 - Emitters that failed to start are now removed
+- [snapshot] Removed the deprecated ``walker_callback`` argument,
+  use ``stat`` instead
 - [watchmedo] The utility is no more installed by default but via the extra
   ``watchdog[watchmedo]``
 
diff --git a/src/watchdog/observers/kqueue.py b/src/watchdog/observers/kqueue.py
index ad5491115..0e4e5bb7f 100644
--- a/src/watchdog/observers/kqueue.py
+++ b/src/watchdog/observers/kqueue.py
@@ -435,12 +435,14 @@ def __init__(self, event_queue, watch, timeout=DEFAULT_EMITTER_TIMEOUT):
         # A collection of KeventDescriptor.
         self._descriptors = KeventDescriptorSet()
 
-        def walker_callback(path, stat_info, self=self):
+        def custom_stat(path, self=self):
+            stat_info = stat(path)
             self._register_kevent(path, stat.S_ISDIR(stat_info.st_mode))
+            return stat_info
 
         self._snapshot = DirectorySnapshot(watch.path,
-                                           watch.is_recursive,
-                                           walker_callback)
+                                           recursive=watch.is_recursive,
+                                           stat=custom_stat)
 
     def _register_kevent(self, path, is_directory):
         """
diff --git a/src/watchdog/utils/dirsnapshot.py b/src/watchdog/utils/dirsnapshot.py
index 40ff95bc0..4083b3fc8 100644
--- a/src/watchdog/utils/dirsnapshot.py
+++ b/src/watchdog/utils/dirsnapshot.py
@@ -226,31 +226,27 @@ class DirectorySnapshot(object):
         snapshot; ``False`` otherwise.
     :type recursive:
         ``bool``
-    :param walker_callback:
-        .. deprecated:: 0.7.2
     :param stat:
         Use custom stat function that returns a stat structure for path.
         Currently only st_dev, st_ino, st_mode and st_mtime are needed.
 
-        A function with the signature ``walker_callback(path, stat_info)``
-        which will be called for every entry in the directory tree.
+        A function taking a ``path`` as argument which will be called
+        for every entry in the directory tree.
     :param listdir:
         Use custom listdir function. For details see ``os.scandir`` if available, else ``os.listdir``.
     """
 
     def __init__(self, path, recursive=True,
-                 walker_callback=(lambda p, s: None),
                  stat=default_stat,
                  listdir=scandir):
         self.recursive = recursive
-        self.walker_callback = walker_callback
         self.stat = stat
         self.listdir = listdir
 
         self._stat_info = {}
         self._inode_to_path = {}
 
-        st = stat(path)
+        st = self.stat(path)
         self._stat_info[path] = st
         self._inode_to_path[(st.st_ino, st.st_dev)] = path
 
@@ -258,7 +254,6 @@ def __init__(self, path, recursive=True,
             i = (st.st_ino, st.st_dev)
             self._inode_to_path[i] = p
             self._stat_info[p] = st
-            walker_callback(p, st)
 
     def walk(self, root):
         try:


Test patch:
diff --git a/tests/test_snapshot_diff.py b/tests/test_snapshot_diff.py
index 9cae27d32..477f04154 100644
--- a/tests/test_snapshot_diff.py
+++ b/tests/test_snapshot_diff.py
@@ -16,12 +16,15 @@
 
 import errno
 import os
+import pickle
 import time
-from .shell import mkdir, touch, mv, rm
+
 from watchdog.utils.dirsnapshot import DirectorySnapshot
 from watchdog.utils.dirsnapshot import DirectorySnapshotDiff
 from watchdog.utils import platform
 
+from .shell import mkdir, touch, mv, rm
+
 
 def wait():
     """
@@ -35,6 +38,13 @@ def wait():
         time.sleep(0.5)
 
 
+def test_pickle(p):
+    """It should be possible to pickle a snapshot."""
+    mkdir(p('dir1'))
+    snasphot = DirectorySnapshot(p('dir1'))
+    pickle.dumps(snasphot)
+
+
 def test_move_to(p):
     mkdir(p('dir1'))
     mkdir(p('dir2'))