Repository: pantsbuild/pants

Issue:
Hangs and crashes in `./v2 fmt ...`
I use `./v2 fmt src/python/pants/backend/python/rules: src/python/pants/backend/python/subsystems:` for the command on linux with no changed files and I've observed 2 forms of crash as well in hangs in the swim-lanes UI for 6 digests and 2 snapshots. All runs fail in one of these 3 ways.

In order of highest frequency 1st:

1. Panic Snapshotting:
    ```
    19:34:56 [ERROR] panic at 'state = Snapshot { is_running: true, is_notified: false, is_released: false, is_complete: false, is_canceled: false, is_join_interested: false, has_join_waker: false, is_final_ref: false }', /home/jsirois/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.13/src/task/state.rs:227
    ```

2. Panic starting a tokio runtime from within another one:
    ```
    19:33:16 [ERROR] panic at 'Cannot start a runtime from within a runtime. This happens because a function (like `block_on`) attempted to block the current thread while the thread is being used to drive asynchronous tasks.', /home/jsirois/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.13/src/runtime/enter.rs:19
    ```

3. A hang in the v2 ui with 8 swimlanes, 6 DigestFile(...) and 2 Scandir(...) (one for "src/python/pants/backend/python/rules" and one for "src/python/pants/backend/python/subsystems")

A `git bisect` leads to 6db9ab4becf5b520c12db297b10c9e6e78aa994e:
```
jsirois@gill ~/dev/pantsbuild/jsirois-pants ((6db9ab4be...)|BISECTING) $ git bisect bad
6db9ab4becf5b520c12db297b10c9e6e78aa994e is the first bad commit
commit 6db9ab4becf5b520c12db297b10c9e6e78aa994e
Author: Henry Fuller <hfuller@twitter.com>
Date:   Thu Mar 26 12:01:39 2020 -0700

    Add notify fs watcher to engine. (#9318)

...
```


Discussion:
Noting that formatting is an intersting case in the context of file watches since it modifies files in the workspace at the completion of the goal:
https://github.com/pantsbuild/pants/blob/9de623801ae82497e582359040087a7de5ec73bc/src/python/pants/rules/core/fmt.py#L174

It does this via the engine, so presumably there is a crease there to remedy if that is part of the problem.
Interesting, you are seeing some different failures modes than me and Eric. We see the black binary run multiple times occasionally ending with a long exception about computing the Select for Fmt goal. I'm going to try to repro what you're seeing a bit more.

>Noting that formatting is an intersting case in the context of file watches since it modifies files in the workspace at the completion of the goal:

My hunch is that invalidation of the files which were formatted in the build root cause the goal to run again, but I'm not sure exactly why, or where that happens. Are all nodes in the build graph required to be in the completed state for the goal to finish successfully? 
@jsirois are you using a linux machine?
Yes. That was buried in the 1st sentence of the description.
I also get a large Select but elided that output figuring it was not relevant.

So, looks like this in full for me (I hit the most common of the three cases above this time):
```
$ ./v2 fmt src/python/pants/backend/python/rules: src/python/pants/backend/python/subsystems:

All done! ✨ 🍰 ✨
28 files left unchanged.

All done! ✨ 🍰 ✨
28 files left unchanged.

All done! ✨ 🍰 ✨
28 files left unchanged.

All done! ✨ 🍰 ✨
28 files left unchanged.

All done! ✨ 🍰 ✨
28 files left unchanged.

All done! ✨ 🍰 ✨
28 files left unchanged.

All done! ✨ 🍰 ✨
28 files left unchanged.

All done! ✨ 🍰 ✨
28 files left unchanged.

All done! ✨ 🍰 ✨
28 files left unchanged.

22:30:30 [WARN] /home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/base/exception_sink.py:431: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
  process_title=setproctitle.getproctitle(),

timestamp: 2020-03-29T15:30:30.940304
Exception caught: (pants.engine.scheduler.ExecutionError)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/pants_loader.py", line 98, in <module>
    main()
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/pants_loader.py", line 94, in main
    PantsLoader.run()
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/pants_loader.py", line 90, in run
    cls.load_and_execute(entrypoint)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/pants_loader.py", line 83, in load_and_execute
    entrypoint_main()
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/pants_exe.py", line 33, in main
    runner.run(start_time)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/pants_runner.py", line 102, in run
    runner.run()
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/local_pants_runner.py", line 344, in run
    engine_result = self._maybe_run_v2(v2)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/local_pants_runner.py", line 292, in _maybe_run_v2
    return self.graph_session.run_goal_rules(
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/init/engine_initializer.py", line 252, in run_goal_rules
    exit_code = self.scheduler_session.run_goal_rule(goal_product, params)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/scheduler.py", line 518, in run_goal_rule
    self._trace_on_error([exc], request)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/scheduler.py", line 483, in _trace_on_error
    raise ExecutionError(

Exception message: 1 Exception encountered:
Computing Select((<pants.engine.console.Console object at 0x7d7dd75a9c40>, AddressSpecs(dependencies=(SiblingAddresses(directory='src/python/pants/backend/python/rules'), SiblingAddresses(directory='src/python/pants/backend/python/subsystems')), matcher=AddressSpecsMatcher(tags=(), exclude_patterns=())), OptionsBootstrapper(env_tuples=(('PANTS_DEV', '1'),), bootstrap_args=('--no-v1', '--v2'), args=('/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/pants_loader.py', '--no-v1', '--v2', '--v2-ui', 'fmt', 'src/python/pants/backend/python/rules:', 'src/python/pants/backend/python/subsystems:'), config=_ChainedConfig(chained_configs=(_SingleFileConfig(config_path='/home/jsirois/.pants.rc', content_digest='a27c7d82405ec33f3f693569e578dffa1fb6d55a', values=_IniValues(parser=<configparser.ConfigParser object at 0x7f36038152e0>)), _SingleFileConfig(config_path='/home/jsirois/dev/pantsbuild/jsirois-pants/pants.toml', content_digest='ad65e29fa6a212c2756cb902e4d0a84adf30dcec', values=_TomlValues(values={'DEFAULT': {'buildroot': '/home/jsirois/dev/pantsbuild/jsirois-pants', 'homedir': '/home/jsirois', 'user': 'jsirois', 'pants_bootstrapdir': '/home/jsirois/.cache/pants', 'pants_configdir': '/home/jsirois/.config/pants', 'pants_workdir': '/home/jsirois/dev/pantsbuild/jsirois-pants/.pants.d', 'pants_supportdir': '/home/jsirois/dev/pantsbuild/jsirois-pants/build-support', 'pants_distdir': '/home/jsirois/dev/pantsbuild/jsirois-pants/dist', 'jvm_options': ['-Xmx1g'], 'local_artifact_cache': '%(pants_bootstrapdir)s/artifact_cache'}, 'GLOBAL': {'print_exception_stacktrace': True, 'v2': False, 'pythonpath': ['%(buildroot)s/contrib/avro/src/python', '%(buildroot)s/contrib/awslambda/python/src/python', '%(buildroot)s/contrib/buildrefactor/src/python', '%(buildroot)s/contrib/codeanalysis/src/python', '%(buildroot)s/contrib/cpp/src/python', '%(buildroot)s/contrib/confluence/src/python', '%(buildroot)s/contrib/errorprone/src/python', '%(buildroot)s/contrib/findbugs/src/python', '%(buildroot)s/contrib/go/src/python', '%(buildroot)s/contrib/googlejavaformat/src/python', '%(buildroot)s/contrib/jax_ws/src/python', '%(buildroot)s/contrib/mypy/src/python', '%(buildroot)s/contrib/node/src/python', '%(buildroot)s/contrib/scalajs/src/python', '%(buildroot)s/contrib/scrooge/src/python', '%(buildroot)s/contrib/thrifty/src/python', '%(buildroot)s/pants-plugins/src/python'], 'backend_packages': {'add': ['pants.backend.docgen', 'pants.contrib.avro', 'pants.contrib.awslambda.python', 'pants.contrib.buildrefactor', 'pants.contrib.codeanalysis', 'pants.contrib.cpp', 'pants.contrib.confluence', 'pants.contrib.errorprone', 'pants.contrib.findbugs', 'pants.contrib.go', 'pants.contrib.googlejavaformat', 'pants.contrib.jax_ws', 'pants.contrib.scalajs', 'pants.contrib.node', 'pants.contrib.scrooge', 'pants.contrib.thrifty', 'internal_backend.repositories', 'internal_backend.sitegen', 'internal_backend.utilities'], 'remove': ['pants.backend.python.lint.isort']}, 'backend_packages2': ['pants.backend.project_info', 'pants.backend.python', 'pants.backend.python.lint.black', 'pants.backend.python.lint.docformatter', 'pants.backend.python.lint.flake8', 'pants.backend.python.lint.isort', 'pants.backend.native', 'internal_backend.rules_for_testing'], 'pantsd_invalidation_globs': {'add': ['src/python/**/*.py']}, 'pants_ignore': {'add': ['/build-support/virtualenvs/', '/build-support/*.venv/', '/build-support/bin/native/src', '/src/rust/engine/target']}}, 'cache': {'read_from': [], 'write_to': [], 'bootstrap': {'read_from': ['%(local_artifact_cache)s'], 'write_to': ['%(local_artifact_cache)s']}, 'resolve': {'coursier': {'read_from': ['%(local_artifact_cache)s'], 'write_to': ['%(local_artifact_cache)s']}}}, 'ivy': {'ivy_settings': '%(pants_supportdir)s/ivy/ivysettings.xml', 'ivy_profile': '%(pants_supportdir)s/ivy/ivy.xml'}, 'gen': {'scrooge': {'service_deps': "{\n  'java': [\n    '3rdparty:slf4j-api',\n    '3rdparty:thrift',\n    '3rdparty/jvm/com/twitter:finagle-thrift',\n    '3rdparty/jvm/com/twitter:scrooge-core',\n  ],\n  'scala': [\n    '3rdparty:thrift',\n    '3rdparty/jvm/com/twitter:finagle-thrift',\n    '3rdparty/jvm/com/twitter:scrooge-core',\n  ],\n}\n", 'structs_deps': "{\n  'java': [\n    '3rdparty:thrift',\n    '3rdparty/jvm/com/twitter:scrooge-core',\n  ],\n  'scala': [\n    '3rdparty:thrift',\n    '3rdparty/jvm/com/twitter:scrooge-core',\n  ],\n}\n", 'service_exports': "{\n  'java': [\n    '3rdparty:thrift',\n  ],\n  'scala': [\n    '3rdparty:thrift',\n    '3rdparty/jvm/com/twitter:finagle-thrift',\n    '3rdparty/jvm/com/twitter:scrooge-core',\n  ],\n}\n", 'structs_exports': "{\n  'java': [\n    '3rdparty:thrift',\n  ],\n  'scala': [\n    '3rdparty:thrift',\n    '3rdparty/jvm/com/twitter:scrooge-core',\n  ],\n}\n"}, 'thrift-java': {'deps': ['3rdparty:thrift']}, 'thrift-py': {'deps': ['examples/3rdparty/python:thrift']}, 'thrifty': {'allow_dups': True}, 'antlr-py': {'antlr3_deps': ['3rdparty/python:antlr-3.1.3']}, 'go-protobuf': {'import_target': 'contrib/go/examples/src/protobuf/org/pantsbuild/example:protobuf-deps'}}, 'compile': {'errorprone': {'command_line_options': ['-Xep:CatchAndPrintStackTrace:OFF', '-Xep:StringSplitter:OFF'], 'exclude_patterns': ['contrib/errorprone/tests/java/org/pantsbuild/contrib/errorprone:error', 'testprojects/src/java/org/pantsbuild/testproject/.*']}, 'findbugs': {'max_rank': 4, 'fail_on_error': True, 'exclude_patterns': ['contrib/findbugs/tests/java/org/pantsbuild/contrib/findbugs:high', 'testprojects/src/java/org/pantsbuild/testproject/.*']}, 'rsc': {'jvm_options': ['-Xmx4g', '-XX:+UseConcMarkSweepGC', '-XX:ParallelGCThreads=4'], 'args': ['-C-encoding', '-CUTF-8', '-S-encoding', '-SUTF-8', '-S-g:vars', '-S-target:jvm-1.8'], 'warning_args': ['-S-deprecation', '-S-unchecked', '-S-feature'], 'no_warning_args': ['-S-nowarn']}, 'javac': {'args': ['-encoding', 'UTF-8', '-J-Xmx2g'], 'warning_args': ['-Xlint:deprecation', '-Xlint:empty', '-Xlint:finally', '-Xlint:overrides', '-Xlint:static', '-Xlint:unchecked', '-Xlint:try'], 'no_warning_args': ['-Xlint:none']}}, 'black': {'config': 'pyproject.toml'}, 'checkstyle': {'config': '%(pants_supportdir)s/checkstyle/coding_style.xml'}, 'docformatter': {'args': ['--wrap-summaries=100', '--wrap-descriptions=100']}, 'eslint': {'setupdir': '%(pants_supportdir)s/eslint', 'config': '%(pants_supportdir)s/eslint/.eslintrc', 'ignore': '%(pants_supportdir)s/eslint/.eslintignore'}, 'flake8': {'config': 'build-support/flake8/.flake8', 'extra_requirements': {'add': ['flake8-pantsbuild>=2.0,<3']}}, 'google-java-format': {'skip': True}, 'isort': {'config': ['.isort.cfg', 'contrib/.isort.cfg', 'examples/.isort.cfg']}, 'scalafmt': {'skip': True}, 'scalafix': {'skip': True}, 'scalastyle': {'config': '%(buildroot)s/build-support/scalastyle/scalastyle_config.xml'}, 'lint': {'scalastyle': {'excludes': '%(buildroot)s/build-support/scalastyle/excludes.txt'}}, 'protoc': {'gen': {'go-protobuf': {'version': '3.4.1'}}}, 'scala': {'version': 2.12}, 'java': {'strict_deps': True}, 'jvm': {'options': ['-Xmx1g'], 'bench': {'options': ['-Xmx1g']}, 'run': {'jvm': {'options': ['-Xmx1g']}}, 'test': {'junit': {'options': ['-Djava.awt.headless=true', '-Xmx1g']}}}, 'jvm-platform': {'default_platform': 'java8', 'platforms': "{\n  'java7': {'source': '7', 'target': '7', 'args': [] },\n  'java8': {'source': '8', 'target': '8', 'args': [] },\n  'java9': {'source': '9', 'target': '9', 'args': [] },\n}\n"}, 'pants-releases': {'branch_notes': "{\n  'master': 'src/python/pants/notes/master.rst',\n  '1.0.x': 'src/python/pants/notes/1.0.x.rst',\n  '1.1.x': 'src/python/pants/notes/1.1.x.rst',\n  '1.2.x': 'src/python/pants/notes/1.2.x.rst',\n  '1.3.x': 'src/python/pants/notes/1.3.x.rst',\n  '1.4.x': 'src/python/pants/notes/1.4.x.rst',\n  '1.5.x': 'src/python/pants/notes/1.5.x.rst',\n  '1.6.x': 'src/python/pants/notes/1.6.x.rst',\n  '1.7.x': 'src/python/pants/notes/1.7.x.rst',\n  '1.8.x': 'src/python/pants/notes/1.8.x.rst',\n  '1.9.x': 'src/python/pants/notes/1.9.x.rst',\n  '1.10.x': 'src/python/pants/notes/1.10.x.rst',\n  '1.11.x': 'src/python/pants/notes/1.11.x.rst',\n  '1.12.x': 'src/python/pants/notes/1.12.x.rst',\n  '1.13.x': 'src/python/pants/notes/1.13.x.rst',\n  '1.14.x': 'src/python/pants/notes/1.14.x.rst',\n  '1.15.x': 'src/python/pants/notes/1.15.x.rst',\n  '1.16.x': 'src/python/pants/notes/1.16.x.rst',\n  '1.17.x': 'src/python/pants/notes/1.17.x.rst',\n  '1.18.x': 'src/python/pants/notes/1.18.x.rst',\n  '1.19.x': 'src/python/pants/notes/1.19.x.rst',\n  '1.20.x': 'src/python/pants/notes/1.20.x.rst',\n  '1.21.x': 'src/python/pants/notes/1.21.x.rst',\n  '1.22.x': 'src/python/pants/notes/1.22.x.rst',\n  '1.23.x': 'src/python/pants/notes/1.23.x.rst',\n  '1.24.x': 'src/python/pants/notes/1.24.x.rst',\n  '1.25.x': 'src/python/pants/notes/1.25.x.rst',\n  '1.26.x': 'src/python/pants/notes/1.26.x.rst',\n}\n"}, 'publish': {'jar': {'ivy_settings': '%(pants_supportdir)s/ivy/publish.ivysettings.xml', 'push_postscript': '# Prevent Travis CI from running for this automated JAR publish commit:\n#  https://docs.travis-ci.com/user/customizing-the-build/#skipping-a-build\n[ci skip]\n', 'repos': "{\n  'public': {  # must match the name of the `Repository` object that you defined in your plugin.\n    'resolver': 'oss.sonatype.org', # must match hostname in ~/.netrc and the <url> parameter\n                                    # in your custom ivysettings.xml.\n    'auth': 'build-support/ivy:netrc',  # Pants spec to a 'credentials()' object.\n    'help': 'Configure your ~/.netrc for oss.sonatype.org access.'\n  }\n}\n", 'restrict_push_branches': ['master'], 'restrict_push_urls': ['git@github.com:pantsbuild/pants.git', 'https://github.com/pantsbuild/pants.git'], 'verify_commit': False}}, 'python-setup': {'interpreter_cache_dir': '%(pants_bootstrapdir)s/python_cache/interpreters', 'resolver_cache_dir': '%(pants_bootstrapdir)s/python_cache/requirements'}, 'pytest': {'pytest_plugins': {'add': ['setuptools==44.0.0', 'ipdb']}}, 'test': {'pytest': {'timeouts': True, 'timeout_default': 60}, 'junit': {'timeouts': True, 'timeout_default': 60}, 'go': {'chroot': False}}, 'buildgen': {'go': {'materialize': True, 'remote': True, 'fail_floating': True}}, 'reference': {'pants_reference_template': 'reference/pants_reference_body.html', 'build_dictionary_template': 'reference/build_dictionary_body.html'}, 'markdown': {'fragment': True}, 'sitegen': {'config_path': 'src/docs/docsite.json'}, 'native-build-step': {'cpp-compile-settings': {'default_compiler_option_sets': ['glibcxx_use_old_abi'], 'compiler_option_sets_enabled_args': "{'glibcxx_use_old_abi': ['-D_GLIBCXX_USE_CXX11_ABI=0']}", 'compiler_option_sets_disabled_args': "{'glibcxx_use_old_abi': ['-D_GLIBCXX_USE_CXX11_ABI=1']}"}}, 'libc': {'enable_libc_search': True}, 'sourcefile-validation': {'config': '@build-support/regexes/config.yaml'}, 'coursier': {'repos': ['https://maven-central.storage-download.googleapis.com/maven2', 'https://repo1.maven.org/maven2', 'https://raw.githubusercontent.com/toolchainlabs/binhost/master/', 'file://%(homedir)s/.m2/repository'], 'fetch_options': {'add': ['--retry-count', '3', '--no-default']}}}))))), Workspace(_scheduler=<pants.engine.scheduler.SchedulerSession object at 0x7d7dd759b730>)), Fmt)
  <None>



22:30:31 [ERROR] panic at 'state = Snapshot { is_running: true, is_notified: false, is_released: false, is_complete: false, is_canceled: false, is_join_interested: false, has_join_waker: false, is_final_ref: false }', /home/jsirois/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.13/src/task/state.rs:227
22:30:31 [ERROR] Please file a bug at https://github.com/pantsbuild/pants/issues.
```
Thanks, that panic is useful info, which I'm not getting on OSX. It would be helpful if you could provide a run with `RUST_BACKTRACE=full` also.

Resolving patch:
diff --git a/src/python/pants/engine/native.py b/src/python/pants/engine/native.py
--- a/src/python/pants/engine/native.py
+++ b/src/python/pants/engine/native.py
@@ -1002,6 +1002,7 @@ def ti(type_obj):
             execution_options.process_execution_use_local_cache,
             self.context.utf8_dict(execution_options.remote_execution_headers),
             execution_options.process_execution_local_enable_nailgun,
+            execution_options.experimental_fs_watcher,
         )
         if scheduler_result.is_throw:
             value = self.context.from_value(scheduler_result.throw_handle)
diff --git a/src/python/pants/option/global_options.py b/src/python/pants/option/global_options.py
--- a/src/python/pants/option/global_options.py
+++ b/src/python/pants/option/global_options.py
@@ -99,6 +99,7 @@ class ExecutionOptions:
     remote_execution_extra_platform_properties: Any
     remote_execution_headers: Any
     process_execution_local_enable_nailgun: bool
+    experimental_fs_watcher: bool
 
     @classmethod
     def from_bootstrap_options(cls, bootstrap_options):
@@ -124,6 +125,7 @@ def from_bootstrap_options(cls, bootstrap_options):
             remote_execution_extra_platform_properties=bootstrap_options.remote_execution_extra_platform_properties,
             remote_execution_headers=bootstrap_options.remote_execution_headers,
             process_execution_local_enable_nailgun=bootstrap_options.process_execution_local_enable_nailgun,
+            experimental_fs_watcher=bootstrap_options.experimental_fs_watcher,
         )
 
 
@@ -149,6 +151,7 @@ def from_bootstrap_options(cls, bootstrap_options):
     remote_execution_extra_platform_properties=[],
     remote_execution_headers={},
     process_execution_local_enable_nailgun=False,
+    experimental_fs_watcher=False,
 )
 
 
@@ -860,6 +863,14 @@ def register_bootstrap_options(cls, register):
             help="Whether or not to use nailgun to run the requests that are marked as nailgunnable.",
             advanced=True,
         )
+        register(
+            "--experimental-fs-watcher",
+            type=bool,
+            default=False,
+            advanced=True,
+            help="Whether to use the engine filesystem watcher which registers the workspace"
+            " for kernel file change events",
+        )
 
     @classmethod
     def register_options(cls, register):


Test patch:
diff --git a/src/rust/engine/src/watch_tests.rs b/src/rust/engine/src/watch_tests.rs
--- a/src/rust/engine/src/watch_tests.rs
+++ b/src/rust/engine/src/watch_tests.rs
@@ -57,8 +57,14 @@ fn setup_watch(
 ) -> InvalidationWatcher {
   let mut rt = tokio::runtime::Runtime::new().unwrap();
   let executor = Executor::new(rt.handle().clone());
-  let watcher = InvalidationWatcher::new(Arc::downgrade(&graph), executor, build_root, ignorer)
-    .expect("Couldn't create InvalidationWatcher");
+  let watcher = InvalidationWatcher::new(
+    Arc::downgrade(&graph),
+    executor,
+    build_root,
+    ignorer,
+    /*enabled*/ true,
+  )
+  .expect("Couldn't create InvalidationWatcher");
   rt.block_on(watcher.watch(file_path)).unwrap();
   watcher
 }
diff --git a/tests/python/pants_test/engine/scheduler_test_base.py b/tests/python/pants_test/engine/scheduler_test_base.py
--- a/tests/python/pants_test/engine/scheduler_test_base.py
+++ b/tests/python/pants_test/engine/scheduler_test_base.py
@@ -3,11 +3,12 @@
 
 import os
 import shutil
+from dataclasses import asdict
 
 from pants.base.file_system_project_tree import FileSystemProjectTree
 from pants.engine.nodes import Throw
 from pants.engine.scheduler import Scheduler
-from pants.option.global_options import DEFAULT_EXECUTION_OPTIONS
+from pants.option.global_options import DEFAULT_EXECUTION_OPTIONS, ExecutionOptions
 from pants.testutil.engine.util import init_native
 from pants.util.contextutil import temporary_file_path
 from pants.util.dirutil import safe_mkdtemp, safe_rmtree
@@ -48,12 +49,17 @@ def mk_scheduler(
         work_dir=None,
         include_trace_on_error=True,
         should_report_workunits=False,
+        execution_options=None,
     ):
         """Creates a SchedulerSession for a Scheduler with the given Rules installed."""
         rules = rules or []
         work_dir = work_dir or self._create_work_dir()
         project_tree = project_tree or self.mk_fs_tree(work_dir=work_dir)
         local_store_dir = os.path.realpath(safe_mkdtemp())
+        if execution_options is not None:
+            eo = asdict(DEFAULT_EXECUTION_OPTIONS)
+            eo.update(execution_options)
+            execution_options = ExecutionOptions(**eo)
         scheduler = Scheduler(
             native=self._native,
             ignore_patterns=project_tree.ignore_patterns,
@@ -61,7 +67,7 @@ def mk_scheduler(
             local_store_dir=local_store_dir,
             rules=rules,
             union_rules=union_rules,
-            execution_options=DEFAULT_EXECUTION_OPTIONS,
+            execution_options=execution_options or DEFAULT_EXECUTION_OPTIONS,
             include_trace_on_error=include_trace_on_error,
         )
         return scheduler.new_session(
diff --git a/tests/python/pants_test/engine/test_fs.py b/tests/python/pants_test/engine/test_fs.py
--- a/tests/python/pants_test/engine/test_fs.py
+++ b/tests/python/pants_test/engine/test_fs.py
@@ -722,7 +722,11 @@ def test_file_content_invalidated(self) -> None:
         on those files."""
 
         with self.mk_project_tree() as project_tree:
-            scheduler = self.mk_scheduler(rules=create_fs_rules(), project_tree=project_tree)
+            scheduler = self.mk_scheduler(
+                rules=create_fs_rules(),
+                project_tree=project_tree,
+                execution_options={"experimental_fs_watcher": True},
+            )
             fname = "4.txt"
             new_data = "rouf"
             # read the original file so we have a cached value.
@@ -748,7 +752,11 @@ def test_file_content_invalidated_after_parent_deletion(self) -> None:
         """Test that FileContent is invalidated after deleting parent directory."""
 
         with self.mk_project_tree() as project_tree:
-            scheduler = self.mk_scheduler(rules=create_fs_rules(), project_tree=project_tree)
+            scheduler = self.mk_scheduler(
+                rules=create_fs_rules(),
+                project_tree=project_tree,
+                execution_options={"experimental_fs_watcher": True},
+            )
             fname = "a/b/1.txt"
             # read the original file so we have nodes to invalidate.
             original_content = self.read_file_content(scheduler, [fname])
@@ -771,7 +779,11 @@ def assert_mutated_directory_digest(
         self, mutation_function: Callable[[FileSystemProjectTree, str], Exception]
     ):
         with self.mk_project_tree() as project_tree:
-            scheduler = self.mk_scheduler(rules=create_fs_rules(), project_tree=project_tree)
+            scheduler = self.mk_scheduler(
+                rules=create_fs_rules(),
+                project_tree=project_tree,
+                execution_options={"experimental_fs_watcher": True},
+            )
             dir_path = "a/"
             dir_glob = dir_path + "*"
             initial_snapshot = self.execute_expecting_one_result(