Repository: pantsbuild/pants

Issue:
Daemon invalidation bug for created directories
A sequence of events like the following:

1. create new directory under an existing directory (not at the root)
2. create BUILD file in that new directory
3. add a dependency on the BUILD file to some other existing BUILD file

seems to (sometimes?) result in a failure to locate targets in the new BUILD file (most likely because the parent directory of the new directory was not properly invalidated).

I've attached a log from a session that exhibited the problem. Running a command involving the target edited in the equivalent of step 3 (from above) results in an error indicating that the BUILD file from step 2 is not found.
 
[Failure log: pantsd.log.txt](https://github.com/pantsbuild/pants/files/1062789/pantsd.log.txt)



Discussion:
So, it looks like watchman `4.9.0` does not send events for parent directories when they have children created/deleted. The previous ticket on this subject: facebook-watchman-issues-305 (not linking in order to avoid the ping) was supposed to deal with avoiding events for touching of existing children, but it looks like all events for parent directories might have been nuked.

...it's a pretty kneejerk reaction, but I'm going to prototype using the `notify` crate to do this instead.
Had way too much going on today. But at the very least, was talked out of exploring `notify` quite yet. Tomorrow, will revert to invalidating parent directories for changed files as we used to do, and possibly open a ticket for exploring using `notify` post daemon beta.

Resolving patch:
diff --git a/src/python/pants/engine/scheduler.py b/src/python/pants/engine/scheduler.py
--- a/src/python/pants/engine/scheduler.py
+++ b/src/python/pants/engine/scheduler.py
@@ -414,13 +414,12 @@ def root_entries(self, execution_request):
             self._execution_request, execution_request))
       return self._scheduler.root_entries(execution_request)
 
-  def invalidate_files(self, filenames):
+  def invalidate_files(self, direct_filenames):
     """Calls `Graph.invalidate_files()` against an internal product Graph instance."""
-    # NB: Watchman will never trigger an invalidation event for the root directory that
-    # is being watched. Instead, we treat any invalidation of a path directly in the
-    # root directory as an invalidation of the root.
-    if any(os.path.dirname(f) in ('', '.') for f in filenames):
-      filenames = tuple(filenames) + ('', '.')
+    # NB: Watchman no longer triggers events when children are created/deleted under a directory,
+    # so we always need to invalidate the direct parent as well.
+    filenames = set(direct_filenames)
+    filenames.update(os.path.dirname(f) for f in direct_filenames)
     with self._product_graph_lock:
       invalidated = self._scheduler.invalidate(filenames)
       logger.debug('invalidated %d nodes for: %s', invalidated, filenames)


Test patch:
diff --git a/tests/python/pants_test/engine/legacy/test_graph.py b/tests/python/pants_test/engine/legacy/test_graph.py
--- a/tests/python/pants_test/engine/legacy/test_graph.py
+++ b/tests/python/pants_test/engine/legacy/test_graph.py
@@ -126,10 +126,7 @@ def test_invalidate_fsnode_incremental(self):
 
       # Invalidate the '3rdparty/python' DirectoryListing, the `3rdparty` DirectoryListing,
       # and then the root DirectoryListing by "touching" files/dirs.
-      # NB: Invalidation of entries in the root directory is special: because Watchman will
-      # never trigger an event for the root itself, we treat changes to files in the root
-      # directory as events for the root.
-      for filename in ('3rdparty/python/BUILD', '3rdparty/python', 'non_existing_file'):
+      for filename in ('3rdparty/python/BUILD', '3rdparty/jvm', 'non_existing_file'):
         invalidated_count = scheduler.invalidate_files([filename])
         self.assertGreater(invalidated_count,
                            0,