Repository: HypothesisWorks/hypothesis

Issue:
KeyError thrown when running tests
Best I can tell, this exception is being thrown from `hypothesis` itself, and not from my code.

```bash
❯ time poetry run ./compare.py                                                                                                            ⏎
You can add @seed(292076761980949346109975285697619484774) to this test to reproduce this failure.
Traceback (most recent call last):
  File "/Users/reiddraper/Desktop/desktop/array-binary-encoding/./compare.py", line 72, in <module>
    test_compare()
  File "/Users/reiddraper/Desktop/desktop/array-binary-encoding/./compare.py", line 63, in test_compare
    @given(pair_of_version_list_with_same_types())
  File "/Users/reiddraper/Library/Caches/pypoetry/virtualenvs/array-binary-encoding-BxhZ887V-py3.9/lib/python3.9/site-packages/hypothesis/core.py", line 1195, in wrapped_test
    raise the_error_hypothesis_found
  File "/Users/reiddraper/Library/Caches/pypoetry/virtualenvs/array-binary-encoding-BxhZ887V-py3.9/lib/python3.9/site-packages/hypothesis/core.py", line 1164, in wrapped_test
    state.run_engine()
  File "/Users/reiddraper/Library/Caches/pypoetry/virtualenvs/array-binary-encoding-BxhZ887V-py3.9/lib/python3.9/site-packages/hypothesis/core.py", line 781, in run_engine
    runner.run()
  File "/Users/reiddraper/Library/Caches/pypoetry/virtualenvs/array-binary-encoding-BxhZ887V-py3.9/lib/python3.9/site-packages/hypothesis/internal/conjecture/engine.py", line 475, in run
    self._run()
  File "/Users/reiddraper/Library/Caches/pypoetry/virtualenvs/array-binary-encoding-BxhZ887V-py3.9/lib/python3.9/site-packages/hypothesis/internal/conjecture/engine.py", line 877, in _run
    self.generate_new_examples()
  File "/Users/reiddraper/Library/Caches/pypoetry/virtualenvs/array-binary-encoding-BxhZ887V-py3.9/lib/python3.9/site-packages/hypothesis/internal/conjecture/engine.py", line 611, in generate_new_examples
    self.__data_cache.pin(zero_data.buffer)
  File "/Users/reiddraper/Library/Caches/pypoetry/virtualenvs/array-binary-encoding-BxhZ887V-py3.9/lib/python3.9/site-packages/hypothesis/internal/cache.py", line 130, in pin
    i = self.keys_to_indices[key]
KeyError: b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
poetry run ./compare.py  272.05s user 11.90s system 99% cpu 4:45.48 total
(array-binary-encoding-BxhZ887V-py3.9)
```

## How to reproduce?

Strangely, I'm having trouble reproducing this with the seed(s) printed, but I am able to consistency reproduce without a specific seed provided. Unfortunately it takes 4-5 minutes on my laptop to hit the exception.

```bash
git clone git@github.com:reiddraper/array-binary-encoding.git
cd array-binary-encoding
git checkout 0ece9478c4e5def1cc4c8d448fe0ee49e014be64
poetry install
time poetry run ./compare.py # wait about 5 minutes
```

The code is in <https://github.com/reiddraper/array-binary-encoding>, if you'd like to examine before running.

---

- `hypothesis`: `6.29.0`
- `Python 3.9.7`


Discussion:
Thanks for reporting, and a beautiful repro!

This is definitely a bug in Hypothesis, though it looks like a hard one to track down and fix...
Unrelated to the bug (I think), you might find that your tests are faster if you can keep the correlated decisions "local", so that constraints apply to as small a sequence of decisions as possible:

```python
same_type_version_lists = st.lists(
    st.tuples(positive_integer, positive_integer) | st.tuples(text, text),
    min_size=1,
    max_size=8,
).map(lambda ls: ([a for a, _ in ls], [b for _, b in ls]))

@st.composite
def pair_of_version_list_with_same_types(draw):
    a, b = draw(same_type_version_lists)
    length_b = draw(st.integers(min_value=1, max_value=len(b)))
    return (a, b[:length_b])
```
> Unrelated to the bug (I think), you might find that your tests are faster if you can keep the correlated decisions "local", so that constraints apply to as small a sequence of decisions as possible:
> 
> ```python
> same_type_version_lists = st.lists(
>     st.tuples(positive_integer, positive_integer) | st.tuples(text, text),
>     min_size=1,
>     max_size=8,
> ).map(lambda ls: ([a for a, _ in ls], [b for _, b in ls]))
> 
> @st.composite
> def pair_of_version_list_with_same_types(draw):
>     a, b = draw(same_type_version_lists)
>     length_b = draw(st.integers(min_value=1, max_value=len(b)))
>     return (a, b[:length_b])
> ```

@Zac-HD brilliant, makes sense, and much appreciated 🙇 .
Hey @reiddraper - I couldn't get your repro to fail locally, but debugging from the other direction I'm pretty sure that https://github.com/HypothesisWorks/hypothesis/pull/3203 fixes it - it looks like the problem was that our least-recently-reused cache could drop an entry just before it got pinned, and the inherited pin method assumed that was impossible.

If you could confirm that the fix works for you (or not!), that would be awesome 🙏 

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/internal/cache.py b/hypothesis-python/src/hypothesis/internal/cache.py
--- a/hypothesis-python/src/hypothesis/internal/cache.py
+++ b/hypothesis-python/src/hypothesis/internal/cache.py
@@ -262,3 +262,16 @@ def on_access(self, key, value, score):
         score[0] = 2
         score[1] = self.tick()
         return score
+
+    def pin(self, key):
+        try:
+            super().pin(key)
+        except KeyError:
+            # The whole point of an LRU cache is that it might drop things for you
+            assert key not in self.keys_to_indices
+
+    def unpin(self, key):
+        try:
+            super().unpin(key)
+        except KeyError:
+            assert key not in self.keys_to_indices


Test patch:
diff --git a/hypothesis-python/tests/cover/test_cache_implementation.py b/hypothesis-python/tests/cover/test_cache_implementation.py
--- a/hypothesis-python/tests/cover/test_cache_implementation.py
+++ b/hypothesis-python/tests/cover/test_cache_implementation.py
@@ -316,3 +316,19 @@ def target():
         worker.join()
 
     assert not errors
+
+
+def test_pin_and_unpin_are_noops_if_dropped():
+    # See https://github.com/HypothesisWorks/hypothesis/issues/3169
+    cache = LRUReusedCache(max_size=10)
+    cache[30] = True
+    assert 30 in cache
+
+    for i in range(20):
+        cache[i] = False
+
+    assert 30 not in cache
+    cache.pin(30)
+    assert 30 not in cache
+    cache.unpin(30)
+    assert 30 not in cache