Repository: HypothesisWorks/hypothesis

Issue:
Improve handling of self-referential strategies
As mentioned in #2783, the following strategy is not explicitly forbidden but fails with an `AssertionError`:

```python
from hypothesis import strategies as st

SELF_REF = st.recursive(
    st.deferred(lambda: SELF_REF | st.booleans()),
    lambda s: st.lists(s, min_size=1)
)
```

There is an alternative strategy that produces the same data, but doesn't fail the same way:

```python
SELF_REF = st.recursive(
    st.booleans(),
    lambda s: st.lists(s, min_size=1)
)
```

I am not sure if all self-referential strategies can be rewritten like this, but probably we can either explicitly forbid such strategies or revisit our drawing approach for them.

Here is my reasoning from #2783 

> As far as I see, the cap is needed to prevent the drawing from this strategy & generating a certain maximum amount of leaves. However, assuming a single thread (more on the multi-threaded behavior in the next section) and such a self-referential strategy, I am not sure if capping is needed as it is - we can just apply it once on the first capped usage and make all subsequent calls no-op (e.g., just yield without modifying marked). Then we still have the marker set only once on the very first RecursiveStrategy.do_draw call, and it will be monotonically decreasing. Therefore, we'll have the max size properly maintained, and there will be no oversized subtrees because, at some point, LimitReached will occur.




Discussion:
As an API design principle, any strategy should be acceptable wherever a strategy is needed - so I'd much rather make this work than specifically disallow some strategies.  That wouldn't *technically* be a breaking change since it's never worked before, but IMO it *should have* worked.

Fortunately I think it should be reasonably easy to make the capping logic work for self-referential strategies; just remove the assertion and ensure we only un-set the cap if it was added by *this* draw (not an outer draw).  Plus some tests to make me confident that I'm right 😅 

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/strategies/_internal/recursive.py b/hypothesis-python/src/hypothesis/strategies/_internal/recursive.py
--- a/hypothesis-python/src/hypothesis/strategies/_internal/recursive.py
+++ b/hypothesis-python/src/hypothesis/strategies/_internal/recursive.py
@@ -66,13 +66,13 @@ def do_draw(self, data):
 
     @contextmanager
     def capped(self, max_templates):
-        assert not self.currently_capped
         try:
+            was_capped = self.currently_capped
             self.currently_capped = True
             self.marker = max_templates
             yield
         finally:
-            self.currently_capped = False
+            self.currently_capped = was_capped
 
 
 class RecursiveStrategy(SearchStrategy):


Test patch:
diff --git a/hypothesis-python/tests/nocover/test_recursive.py b/hypothesis-python/tests/nocover/test_recursive.py
--- a/hypothesis-python/tests/nocover/test_recursive.py
+++ b/hypothesis-python/tests/nocover/test_recursive.py
@@ -156,3 +156,15 @@ def test(data):
         thread.join()
 
     assert not errors
+
+
+SELF_REF = st.recursive(
+    st.deferred(lambda: st.booleans() | SELF_REF),
+    lambda s: st.lists(s, min_size=1),
+)
+
+
+@given(SELF_REF)
+def test_self_ref_regression(_):
+    # See https://github.com/HypothesisWorks/hypothesis/issues/2794
+    pass