Repository: pantsbuild/pants

Issue:
[pantsd] Nested pantsd runs create timeout.
After #7751, we have found the following situation in some of our pipelines:

If a `./pants` command, somewhere in its process calls `./pants` again (e.g. to build a pex), the inner run will wait for the outer run to finish, which will never happen. Therefore, the inner run will time out, ending the invocation with:

```
Another pants invocation is running. Will wait up to 60.0 seconds for it to finish before giving up.
Waiting for invocation to finish (waited for 1.0s so far)...
...
Waiting for invocation to finish (waited for 59.0s so far)...
21:45:13 [ERROR] Request <socket._socketobject object at 0x3344ac58d00> errored with <class 'pants.pantsd.pailgun_server.ExclusiveRequestTimeout'>(Timed out while waiting for another pants invocation to finish.)
```

**Proposed solution**
The most principled fix we have for now (until we implement fully parallel pantsd runs) is to disable pantsd in the inner run, which should be performance-neutral to not having pantsd.

I propose that we sistematically disable pantsd in any inner runs, by:
In `DaemonPantsRunner` or `PailgunHandler`, we could intentionally inject the `--no-enable-pantsd` flag into every (outer) run before we pass it to a `LocalPantsRunner`, and therefore inner runs would inherit the flag and not even try to connect to the daemon.


Discussion:
> The most principled fix we have for now (until we implement fully parallel pantsd runs) is to disable pantsd in the inner run, which should be performance-neutral to not having pantsd.

Yes, I am no pantsd expert but this sounds very reasonable.

An alternative solution, potentially more performant but also more complex, is if the outer run can temporarily yield control of the lock? I don't know if that's possible given how the code is set up, and if it requires major modifications it might not be worth it because we are planning on having fully parallel pantsd runs at some point.
> An alternative solution, potentially more performant but also more complex, is if the outer run can temporarily yield control of the lock?

I would really love to do that, but atm I'm not confident that two runs can exist in parallel regardless of any outer-inner relationship, because of global state. We can also not "pause" the outer run, unfortunately. Solving "we figured out that it's safe to have inner running at the same time as outer" would be the first thing to do when we want to enable parallel runs.

Resolving patch:
diff --git a/src/python/pants/pantsd/pailgun_server.py b/src/python/pants/pantsd/pailgun_server.py
--- a/src/python/pants/pantsd/pailgun_server.py
+++ b/src/python/pants/pantsd/pailgun_server.py
@@ -80,6 +80,11 @@ def handle(self):
     self.logger.info('handling pailgun request: `{}`'.format(' '.join(arguments)))
     self.logger.debug('pailgun request environment: %s', environment)
 
+    # We don't support parallel runs in pantsd, and therefore if this pailgun request
+    # triggers any pants command, we want it to not use pantsd at all.
+    # See https://github.com/pantsbuild/pants/issues/7881 for context.
+    environment["PANTS_CONCURRENT"] = "True"
+
     # Execute the requested command with optional daemon-side profiling.
     with maybe_profiled(environment.get('PANTSD_PROFILE')):
       self._run_pants(self.request, arguments, environment)
diff --git a/testprojects/src/python/nested_runs/run_pants_with_pantsd.py b/testprojects/src/python/nested_runs/run_pants_with_pantsd.py
new file mode 100644
--- /dev/null
+++ b/testprojects/src/python/nested_runs/run_pants_with_pantsd.py
@@ -0,0 +1,23 @@
+import pathlib
+import subprocess
+import sys
+
+
+def main():
+  workdir = sys.argv[1]
+  config = pathlib.Path(workdir) / 'pants.ini'
+
+  cmd = [
+    './pants',
+    '--no-pantsrc',
+    f'--pants-config-files={config}',
+    '--print-exception-stacktrace=True',
+    f'--pants-workdir={workdir}',
+    'goals'
+  ]
+  print(f'Running pants with command {cmd}')
+  subprocess.run(cmd, check=True)
+
+
+if __name__ == '__main__':
+  main()


Test patch:
diff --git a/tests/python/pants_test/pantsd/test_pantsd_integration.py b/tests/python/pants_test/pantsd/test_pantsd_integration.py
--- a/tests/python/pants_test/pantsd/test_pantsd_integration.py
+++ b/tests/python/pants_test/pantsd/test_pantsd_integration.py
@@ -771,3 +771,25 @@ def test_unhandled_exceptions_only_log_exceptions_once(self):
         'pants.bin.daemon_pants_runner._PantsRunFinishedWithFailureException: Terminated with 1',
         result.stderr_data,
       )
+
+  def test_inner_runs_dont_deadlock(self):
+    """
+    Create a pantsd run that calls testprojects/src/python/nested_runs with the appropriate
+    bootstrap options to avoid restarting pantsd.
+
+    Regression test for issue https://github.com/pantsbuild/pants/issues/7881
+    When a run under pantsd calls pants with pantsd inside it, the inner run will time out
+    waiting for the outer run to end.
+
+    NB: testprojects/src/python/nested_runs assumes that the pants.ini file is in ${workdir}/pants.ini
+    """
+    config = {
+      'GLOBAL': {
+        'pantsd_timeout_when_multiple_invocations': 1,
+      }
+    }
+    with self.pantsd_successful_run_context(extra_config=config) as (pantsd_run, checker, workdir, _):
+      result = pantsd_run(['run', 'testprojects/src/python/nested_runs', '--', workdir], expected_runs=2)
+      checker.assert_started()
+      self.assert_success(result)
+      self.assertNotIn("Another pants invocation is running", result.stderr_data)