Repository: pantsbuild/pants

Issue:
Target Hitting Recursion Limit During Pants Setup (with workaround)
# Description of Problem
We’re in the process of migrating from 1.25 to 2.1.0., and hit an issue trying to run a test on specific target. The target is large and results in a max recursion limit exceeded.

I tried hacking on `sys.setrecursionlimit` and found for our use case 1021 was the min that would allow the test to succeed.

We can try breaking that target up, but the app it is testing is kind of a monolith so i don’t know how successful that would be.

Can you make a runtime limit in pants to handle?

This error happens in the pants setup before our pytest is run.

# Workaround
In one of our plugin's `register.py` we added `sys.setrecursionlimit(1021)` and this resolved our problem.


Discussion:

Resolving patch:
diff --git a/src/python/pants/engine/internals/graph.py b/src/python/pants/engine/internals/graph.py
--- a/src/python/pants/engine/internals/graph.py
+++ b/src/python/pants/engine/internals/graph.py
@@ -7,7 +7,7 @@
 import os.path
 from dataclasses import dataclass
 from pathlib import PurePath
-from typing import Dict, Iterable, List, NamedTuple, Optional, Sequence, Set, Tuple, Type
+from typing import Dict, Iterable, List, NamedTuple, Optional, Sequence, Tuple, Type
 
 from pants.base.exceptions import ResolveError
 from pants.base.specs import (
@@ -37,6 +37,7 @@
     Snapshot,
     SourcesSnapshot,
 )
+from pants.engine.internals.native_engine import cyclic_paths
 from pants.engine.internals.target_adaptor import TargetAdaptor
 from pants.engine.rules import Get, MultiGet, collect_rules, rule
 from pants.engine.target import (
@@ -247,21 +248,18 @@ def __init__(self, subject: Address, path: Tuple[Address, ...]) -> None:
         self.path = path
 
 
-def _detect_cycles(
-    roots: Tuple[Address, ...], dependency_mapping: Dict[Address, Tuple[Address, ...]]
-) -> None:
-    path_stack: OrderedSet[Address] = OrderedSet()
-    visited: Set[Address] = set()
+def _detect_cycles(dependency_mapping: Dict[Address, Tuple[Address, ...]]) -> None:
+    for path in cyclic_paths(dependency_mapping):
+        address = path[-1]
 
-    def maybe_report_cycle(address: Address) -> None:
         # NB: File-level dependencies are cycle tolerant.
-        if not address.is_base_target or address not in path_stack:
+        if not address.is_base_target:
             return
 
         # The path of the cycle is shorter than the entire path to the cycle: if the suffix of
         # the path representing the cycle contains a file dep, it is ignored.
         in_cycle = False
-        for path_address in path_stack:
+        for path_address in path:
             if in_cycle and not path_address.is_base_target:
                 # There is a file address inside the cycle: do not report it.
                 return
@@ -273,27 +271,7 @@ def maybe_report_cycle(address: Address) -> None:
                 # the address in question.
                 in_cycle = path_address == address
         # If we did not break out early, it's because there were no file addresses in the cycle.
-        raise CycleException(address, (*path_stack, address))
-
-    def visit(address: Address):
-        if address in visited:
-            maybe_report_cycle(address)
-            return
-        path_stack.add(address)
-        visited.add(address)
-
-        for dep_address in dependency_mapping[address]:
-            visit(dep_address)
-
-        path_stack.remove(address)
-
-    for root in roots:
-        visit(root)
-        if path_stack:
-            raise AssertionError(
-                f"The stack of visited nodes should have been empty at the end of recursion, "
-                f"but it still contained: {path_stack}"
-            )
+        raise CycleException(address, path)
 
 
 @rule(desc="Resolve transitive targets")
@@ -332,10 +310,7 @@ async def transitive_targets(request: TransitiveTargetsRequest) -> TransitiveTar
         )
         visited.update(queued)
 
-    # NB: We use `roots_as_targets` to get the root addresses, rather than `request.roots`. This
-    # is because expanding from the `Addresses` -> `Targets` may have resulted in generated
-    # subtargets being used, so we need to use `roots_as_targets` to have this expansion.
-    _detect_cycles(tuple(t.address for t in roots_as_targets), dependency_mapping)
+    _detect_cycles(dependency_mapping)
 
     # Apply any transitive excludes (`!!` ignores).
     transitive_excludes: FrozenOrderedSet[Target] = FrozenOrderedSet()
@@ -392,7 +367,7 @@ async def transitive_targets_lite(request: TransitiveTargetsRequestLite) -> Tran
     # NB: We use `roots_as_targets` to get the root addresses, rather than `request.roots`. This
     # is because expanding from the `Addresses` -> `Targets` may have resulted in generated
     # subtargets being used, so we need to use `roots_as_targets` to have this expansion.
-    _detect_cycles(tuple(t.address for t in roots_as_targets), dependency_mapping)
+    _detect_cycles(dependency_mapping)
 
     # Apply any transitive excludes (`!!` ignores).
     wrapped_transitive_excludes = await MultiGet(


Test patch: