Repository: pantsbuild/pants

Issue:
Target Hitting Recursion Limit During Pants Setup (with workaround)
# Description of Problem
We’re in the process of migrating from 1.25 to 2.1.0., and hit an issue trying to run a test on specific target. The target is large and results in a max recursion limit exceeded.

I tried hacking on `sys.setrecursionlimit` and found for our use case 1021 was the min that would allow the test to succeed.

We can try breaking that target up, but the app it is testing is kind of a monolith so i don’t know how successful that would be.

Can you make a runtime limit in pants to handle?

This error happens in the pants setup before our pytest is run.

# Workaround
In one of our plugin's `register.py` we added `sys.setrecursionlimit(1021)` and this resolved our problem.


Discussion:
Although the original issue was resolved, the fix from #11202 seems to be able to peg CPUs when running on a target with a sufficiently large number of files (44 files with 1500 deps each in a graph of more than 200k edges).

We have not yet been able to confirm that the issue is an infinite loop via a profile: profiles appear to show multiple threads individually making progress. And diving in to microbenchmark (of just the code ported in #11202), it appears that we have some super-linear behavior due to lock contention (both the GIL and possibly also our interning):
```
>>> took 0.40951013565063477 with 1 threads
>>> took 2.203711986541748 with 2 threads
>>> took 11.812678098678589 with 4 threads
>>> took 24.522939205169678 with 8 threads
```

This is a good microbenchmark to examine both GIL acquisition and interning, so I'd like to see how much we can improve this by reducing contention here (which should pay dividends elsewhere in the codebase). If reducing lock contention isn't sufficient, we can look into adjusting the `TransitiveTargets` code to do more structure sharing (as it did before #10409) to avoid redundant work.
Status update: I believe that I have a holistic fix for this, as well as for a few other `TransitiveTarget` performance concerns. I need to nail down one issue with the `@rule` graph first, but am hopeful that I can have it reviewable/cherry-pickable tomorrow.
Unfortunately the holistic solution (#11270) has a blocker (#11269). For now I'm going to revert the first fix for this ticket, and fix it via #11271.

Sorry for the big mess! Graph cycles are the bane of my existence =)

Resolving patch:
diff --git a/src/python/pants/bin/pants_loader.py b/src/python/pants/bin/pants_loader.py
--- a/src/python/pants/bin/pants_loader.py
+++ b/src/python/pants/bin/pants_loader.py
@@ -4,6 +4,7 @@
 import importlib
 import locale
 import os
+import sys
 import warnings
 from textwrap import dedent
 
@@ -14,6 +15,8 @@ class PantsLoader:
     ENTRYPOINT_ENV_VAR = "PANTS_ENTRYPOINT"
     DEFAULT_ENTRYPOINT = "pants.bin.pants_exe:main"
 
+    RECURSION_LIMIT_ENV_VAR = "PANTS_RECURSION_LIMIT"
+
     ENCODING_IGNORE_ENV_VAR = "PANTS_IGNORE_UNRECOGNIZED_ENCODING"
 
     class InvalidLocaleError(Exception):
@@ -67,6 +70,10 @@ def ensure_locale(cls):
                 )
             )
 
+    @classmethod
+    def set_recursion_limit(cls):
+        sys.setrecursionlimit(int(os.environ.get(cls.RECURSION_LIMIT_ENV_VAR, "10000")))
+
     @staticmethod
     def determine_entrypoint(env_var, default):
         return os.environ.pop(env_var, default)
@@ -86,6 +93,7 @@ def load_and_execute(entrypoint):
     def run(cls):
         cls.setup_warnings()
         cls.ensure_locale()
+        cls.set_recursion_limit()
         entrypoint = cls.determine_entrypoint(cls.ENTRYPOINT_ENV_VAR, cls.DEFAULT_ENTRYPOINT)
         cls.load_and_execute(entrypoint)
 


Test patch:
diff --git a/tests/python/pants_test/bin/loader_integration_test.py b/tests/python/pants_test/bin/loader_integration_test.py
--- a/tests/python/pants_test/bin/loader_integration_test.py
+++ b/tests/python/pants_test/bin/loader_integration_test.py
@@ -2,7 +2,7 @@
 # Licensed under the Apache License, Version 2.0 (see LICENSE).
 
 from pants.bin.pants_loader import PantsLoader
-from pants.testutil.pants_integration_test import run_pants
+from pants.testutil.pants_integration_test import PantsResult, run_pants
 
 
 def test_invalid_locale() -> None:
@@ -49,3 +49,17 @@ def test_alternate_entrypoint_scrubbing() -> None:
     )
     pants_run.assert_success()
     assert "PANTS_ENTRYPOINT=None" in pants_run.stdout
+
+
+def test_recursion_limit() -> None:
+    def run(limit: str) -> PantsResult:
+        return run_pants(command=["help"], extra_env={"PANTS_RECURSION_LIMIT": limit})
+
+    # Large value succeeds.
+    run("100000").assert_success()
+    # Very small value fails in an arbitrary spot.
+    small_run = run("1")
+    small_run.assert_failure()
+    assert "RecursionError" in small_run.stderr
+    # Non integer value fails.
+    run("this isn't an int").assert_failure()