Repository: pantsbuild/pants

Issue:
Pants 2.15.0rc4 can hang under pantsd
I have suffered hangs over in scie-pants using Pants 2.15.0rc4, see:
+ Linux: https://github.com/pantsbuild/scie-pants/actions/runs/4128694155/jobs/7133454406
+ Mac: https://github.com/pantsbuild/scie-pants/actions/runs/4128694155/jobs/7133454550

The context is an integration test where Pants is launched (`--pantsd` mode) in a subprocess with stdout piped for test use in assertions. This code here: https://github.com/pantsbuild/scie-pants/blob/3d269eb21ab7d764054496cd33530aaa364c6252/package/src/main.rs#L863-L892 The hang happens in the `assert_pants_bin_name` calls. The links above on the 1st or 2nd call, the example below in the 3rd or 4th call (can't say for sure if the hang is on pantsd down or pantsd up, but I think its on the bring-down which would make these 1st and 3rd).

I used this ssh rig to debug: https://github.com/pantsbuild/scie-pants/pull/114

The commands run in the ssh session were:
```
$ sudo rm -rf target/
$ cargo run -p package -- test --tools-pex-mismatch-warn
<hang>
ctrl-Z
$ sudo apt install gdb
$ sudo gdb -p 16881
```

[log-slim.txt](https://github.com/pantsbuild/pants/files/10691455/log-slim.txt)
[log-slim2.txt](https://github.com/pantsbuild/pants/files/10691456/log-slim2.txt)



Discussion:
So my blundering reading leads here:
https://github.com/pantsbuild/pants/blob/ad53134b0e6d2efba94daab1fe5653f62c26b1fe/src/rust/engine/task_executor/src/lib.rs#L173-L187

That is implicated in a non-main thread in the backtrace capture attached in the OP.
If the comment is to be believed, I find `scheduler_execute` used by :
https://github.com/pantsbuild/pants/blob/ad53134b0e6d2efba94daab1fe5653f62c26b1fe/src/python/pants/engine/internals/scheduler.py#L454-L465
https://github.com/pantsbuild/pants/blob/ad53134b0e6d2efba94daab1fe5653f62c26b1fe/src/python/pants/engine/internals/scheduler.py#L502-L507
https://github.com/pantsbuild/pants/blob/ad53134b0e6d2efba94daab1fe5653f62c26b1fe/src/python/pants/engine/internals/scheduler.py#L543-L567


Which is used by:
```
$ git grep determine_bootstrap_environment
src/python/pants/bin/local_pants_runner.py:from pants.core.util_rules.environments import determine_bootstrap_environment
src/python/pants/bin/local_pants_runner.py:            determine_bootstrap_environment(self.graph_session.scheduler_session),
src/python/pants/bsp/protocol.py:from pants.core.util_rules.environments import determine_bootstrap_environment
src/python/pants/bsp/protocol.py:        self._env_name = determine_bootstrap_environment(self._scheduler_session)
src/python/pants/core/util_rules/environments.py:def determine_bootstrap_environment(session: SchedulerSession) -> EnvironmentName:
src/python/pants/engine/streaming_workunit_handler.py:from pants.core.util_rules.environments import determine_bootstrap_environment
src/python/pants/engine/streaming_workunit_handler.py:            determine_bootstrap_environment(self._scheduler),
src/python/pants/goal/explorer.py:from pants.core.util_rules.environments import determine_bootstrap_environment
src/python/pants/goal/explorer.py:        env_name = determine_bootstrap_environment(graph_session.scheduler_session)
src/python/pants/init/engine_initializer.py:from pants.core.util_rules.environments import determine_bootstrap_environment
src/python/pants/init/engine_initializer.py:        env_name = determine_bootstrap_environment(self.scheduler_session)
src/python/pants/init/plugin_resolver.py:from pants.core.util_rules.environments import determine_bootstrap_environment
src/python/pants/init/plugin_resolver.py:        params = Params(request, determine_bootstrap_environment(session))
src/python/pants/init/specs_calculator.py:from pants.core.util_rules.environments import determine_bootstrap_environment
src/python/pants/init/specs_calculator.py:    bootstrap_environment = determine_bootstrap_environment(session)
```
Thanks John.

The only thing that jumps out at me there is that there are 4 filesystem watching tasks (threads 11, 9, 7, 5), when I would expect 1. I don't see any lock interleavings/inversions in the stacktraces of any of the threads.

It's possible that the 4 filesystem watching tasks are using up enough blocking threads on the tokio runtime to cause a deadlock (depending on the computed value of [rule_threads_max](https://www.pantsbuild.org/v2.15/docs/reference-global#rule_threads_max)), but I'll need to investigate further.
> It's possible that the 4 filesystem watching tasks are using up enough blocking threads on the tokio runtime to cause a deadlock (depending on the computed value of [rule_threads_max](https://www.pantsbuild.org/v2.15/docs/reference-global#rule_threads_max)), but I'll need to investigate further.

This was a red herring: the filesystem watching threads are dedicated threads, running outside the runtime. They just confusingly don't have thread names assigned.

Will continue looking.
It looks like this is actually a `pantsd` client hang instead -- the stack is much more obviously weird: [pants-client.bt.txt](https://github.com/pantsbuild/pants/files/10700392/pants-client.bt.txt)

Patch forthcoming.
By obviously weird do you mean the non-main thread trying to get the GIL? That is not obviously weird to me. I think it might be good to record your analysis here since - unless I'm the only one who this isn't obvious to - you appear to be a man on the bus.

Resolving patch:
diff --git a/src/python/pants/bin/local_pants_runner.py b/src/python/pants/bin/local_pants_runner.py
--- a/src/python/pants/bin/local_pants_runner.py
+++ b/src/python/pants/bin/local_pants_runner.py
@@ -119,7 +119,7 @@ def create(
             bootstrap_options = options.bootstrap_option_values()
             assert bootstrap_options is not None
             scheduler = EngineInitializer.setup_graph(
-                bootstrap_options, build_config, dynamic_remote_options
+                bootstrap_options, build_config, dynamic_remote_options, executor
             )
         with options_initializer.handle_unknown_flags(options_bootstrapper, env, raise_=True):
             global_options = options.for_global_scope()
diff --git a/src/python/pants/bin/remote_pants_runner.py b/src/python/pants/bin/remote_pants_runner.py
--- a/src/python/pants/bin/remote_pants_runner.py
+++ b/src/python/pants/bin/remote_pants_runner.py
@@ -11,7 +11,11 @@
 from typing import List, Mapping
 
 from pants.base.exiter import ExitCode
-from pants.engine.internals.native_engine import PantsdConnectionException, PyNailgunClient
+from pants.engine.internals.native_engine import (
+    PantsdConnectionException,
+    PyExecutor,
+    PyNailgunClient,
+)
 from pants.option.global_options import GlobalOptions
 from pants.option.options_bootstrapper import OptionsBootstrapper
 from pants.pantsd.pants_daemon_client import PantsDaemonClient
@@ -114,15 +118,19 @@ def run(self, start_time: float) -> ExitCode:
         pantsd_handle = self._client.maybe_launch()
         logger.debug(f"Connecting to pantsd on port {pantsd_handle.port}")
 
-        return self._connect_and_execute(pantsd_handle, start_time)
+        executor = GlobalOptions.create_py_executor(self._bootstrap_options.for_global_scope())
+        try:
+            return self._connect_and_execute(pantsd_handle, executor, start_time)
+        finally:
+            executor.shutdown(3)
 
     def _connect_and_execute(
-        self, pantsd_handle: PantsDaemonClient.Handle, start_time: float
+        self,
+        pantsd_handle: PantsDaemonClient.Handle,
+        executor: PyExecutor,
+        start_time: float,
     ) -> ExitCode:
         global_options = self._bootstrap_options.for_global_scope()
-        # We do not explicitly shut this PyExecutor down, because the client should not run any long lived
-        # tasks which we would want to wait for (in particular: it runs no Python code). See #16105.
-        executor = GlobalOptions.create_py_executor(global_options)
 
         # Merge the nailgun TTY capability environment variables with the passed environment dict.
         ng_env = ttynames_to_env(sys.stdin, sys.stdout, sys.stderr)
diff --git a/src/python/pants/init/engine_initializer.py b/src/python/pants/init/engine_initializer.py
--- a/src/python/pants/init/engine_initializer.py
+++ b/src/python/pants/init/engine_initializer.py
@@ -188,7 +188,7 @@ def setup_graph(
         bootstrap_options: OptionValueContainer,
         build_configuration: BuildConfiguration,
         dynamic_remote_options: DynamicRemoteOptions,
-        executor: PyExecutor | None = None,
+        executor: PyExecutor,
         is_bootstrap: bool = False,
     ) -> GraphScheduler:
         build_root = get_buildroot()


Test patch: