Repository: HypothesisWorks/hypothesis

Issue:
Strategy cache thread-safety
Hi!

I encountered a thread-safety issue when multiple threads create strategies. It happens in Schemathesis and the workflow is:
- multiple workers are spawned and consume incoming events
- each event contains data, that is used to create a strategy via `hypothesis_jsonschema.from_schema` 

With some probability, a worker thread can fail with an exception like this:

```python-traceback
File /.virtualenvs/schemathesis/lib/python3.8/site-packages/hypothesis/strategies/_internal/core.py", line 172, in cached_strategy
E             STRATEGY_CACHE[cache_key] = result
E           File "/.virtualenvs/schemathesis/lib/python3.8/site-packages/hypothesis/internal/cache.py", line 121, in __setitem__
E             assert evicted.score <= self.data[0].score
E         AssertionError: assert [1, 10062] <= [1, 6901]
```

However, this can be reproduced with this snippet:

```python
import threading
from hypothesis import strategies as st                                                                                                                                                                                                                                                                                       
from functools import partial                                                                                                                                                                                                                                                                                   

def target(): 
     for _ in range(1000): 
         st.builds(partial(str)) 
                                                                                                                                                                                                                                                                                                                 
for _ in range(4): 
     worker = threading.Thread(target=target) 
     worker.start()
```

Environment:
- Python 3.8.2
- Hypothesis 5.14.0

And the output is:

```python-traceback                                                                                                                                                                                                                                         
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
In [8]:     self._target(*self._args, **self._kwargs)                                                                                                                                                                                                                                                                   
  File "<ipython-input-4-5e40ee0ad813>", line 3, in target
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "<ipython-input-4-5e40ee0ad813>", line 3, in target
  File "/.virtualenvs/schemathesis/lib/python3.8/site-packages/hypothesis/strategies/_internal/core.py", line 1177, in builds
  File "/.virtualenvs/schemathesis/lib/python3.8/site-packages/hypothesis/strategies/_internal/core.py", line 1177, in builds
    @defines_strategy
  File "/.virtualenvs/schemathesis/lib/python3.8/site-packages/hypothesis/strategies/_internal/core.py", line 172, in cached_strategy
    STRATEGY_CACHE[cache_key] = result
  File "/.virtualenvs/schemathesis/lib/python3.8/site-packages/hypothesis/internal/cache.py", line 119, in __setitem__
    @defines_strategy
  File "/.virtualenvs/schemathesis/lib/python3.8/site-packages/hypothesis/strategies/_internal/core.py", line 172, in cached_strategy
    assert evicted.score <= self.data[0].score
AssertionError
    STRATEGY_CACHE[cache_key] = result
  File "/.virtualenvs/schemathesis/lib/python3.8/site-packages/hypothesis/internal/cache.py", line 119, in __setitem__
    assert evicted.score <= self.data[0].score
AssertionError
```

As far as I see it is caused by shared storage inside `hypothesis.strategies._internal.core.STRATEGY_CACHE` and access to it is not thread-safe.

I don't know if such usage workflow was expected, but a trick with `threading.local()` seems to work

```
class GenericCache:
    __slots__ = ("max_size", "__pinned_entry_count", "_storage")

    def __init__(self, max_size):
        self.max_size = max_size
        self._storage = threading.local()

    @property
    def data(self):
        try:
            return self._storage.data
        except AttributeError:
            self._storage.data = []
            return self._storage.data

    @property
    def keys_to_indices(self):
        ...
```
(a straightforward implementation, not sure if it will work in 100% cases)

I assume that alternative would be using locks either in Hypothesis or on the Schemathesis side, which doesn't seem nice to me.

What do you think about making this cache thread-safe?



Discussion:
Huh - thanks for reporting this.

I don't know if we think Hypothesis *should* be threadsafe (@DRMacIver?) - we're certainly not testing for that very well - but personally I'd be happy to take a small patch that improves your experience.

The fix should go in the parent `GenericCache` class, and tests - IMO your repro script would be fine, wrapped in `def test_cache_is_threadsafe_issue_2433_regression():` - in `tests/cover/test_cache_implementation.py`.

```diff
@@ -13,6 +13,8 @@
 #
 # END HEADER

+import threading
+
 import attr


@@ -56,7 +58,7 @@ class GenericCache:
     on_access and on_evict to implement a specific scoring strategy.
     """

-    __slots__ = ("keys_to_indices", "data", "max_size", "__pinned_entry_count")
+    __slots__ = ("keys_to_indices", "_storage", "max_size", "__pinned_entry_count")

     def __init__(self, max_size):
         self.max_size = max_size
@@ -67,9 +69,17 @@ class GenericCache:
         # heap. We keep these two in sync automatically - the heap is never
         # reordered without updating the index.
         self.keys_to_indices = {}
-        self.data = []
+        self._storage = threading.local()
         self.__pinned_entry_count = 0

+    @property
+    def data(self):
+        try:
+            return self._storage.data
+        except AttributeError:
+            self._storage.data = []
+            return self._storage.data
+
     def __len__(self):
         assert len(self.keys_to_indices) == len(self.data)
         return len(self.data)
```

FYI [GitHub supports syntax highlighting in code blocks](https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks) - `python` and `python-traceback` are particularly useful for reports like this.
> I don't know if we think Hypothesis should be threadsafe (@DRMacIver?) - we're certainly not testing for that very well - but personally I'd be happy to take a small patch that improves your experience.

I think it's reasonable that we should be threadsafe, yeah, at least in the sense that each thread should probably not interfere with tests running in other threads (there are more complications with threads within tests that I don't think we should support).

I think trying to make the cache itself thread safe is probably a mistake though - I'd probably suggest instead having `STRATEGY_CACHE` be a thread local pointing to a cache per thread.
Cool, we're on the same page then. The patch above does the cache-per-thread trick, hidden inside the class so call sites can continue to ignore threading. 
I don't think it does, because the cache has all sorts of other points of mutation. e.g. you don't put `__pinned_entry_count` field under a thread local.

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/strategies/_internal/core.py b/hypothesis-python/src/hypothesis/strategies/_internal/core.py
--- a/hypothesis-python/src/hypothesis/strategies/_internal/core.py
+++ b/hypothesis-python/src/hypothesis/strategies/_internal/core.py
@@ -19,6 +19,7 @@
 import random
 import string
 import sys
+import threading
 import typing
 from decimal import Context, Decimal, localcontext
 from fractions import Fraction
@@ -150,7 +151,20 @@ def convert_value(v):
     return (type(v), v)
 
 
-STRATEGY_CACHE = LRUReusedCache(1024)
+_CACHE = threading.local()
+
+
+def get_cache() -> LRUReusedCache:
+    try:
+        return _CACHE.STRATEGY_CACHE
+    except AttributeError:
+        _CACHE.STRATEGY_CACHE = LRUReusedCache(1024)
+        return _CACHE.STRATEGY_CACHE
+
+
+def clear_cache() -> None:
+    cache = get_cache()
+    cache.clear()
 
 
 def cacheable(fn: T) -> T:
@@ -161,18 +175,19 @@ def cached_strategy(*args, **kwargs):
         except TypeError:
             return fn(*args, **kwargs)
         cache_key = (fn, tuple(map(convert_value, args)), frozenset(kwargs_cache_key))
+        cache = get_cache()
         try:
-            if cache_key in STRATEGY_CACHE:
-                return STRATEGY_CACHE[cache_key]
+            if cache_key in cache:
+                return cache[cache_key]
         except TypeError:
             return fn(*args, **kwargs)
         else:
             result = fn(*args, **kwargs)
             if not isinstance(result, SearchStrategy) or result.is_cacheable:
-                STRATEGY_CACHE[cache_key] = result
+                cache[cache_key] = result
             return result
 
-    cached_strategy.__clear_cache = STRATEGY_CACHE.clear
+    cached_strategy.__clear_cache = clear_cache
     return cached_strategy
 
 


Test patch:
diff --git a/hypothesis-python/tests/cover/test_cache_implementation.py b/hypothesis-python/tests/cover/test_cache_implementation.py
--- a/hypothesis-python/tests/cover/test_cache_implementation.py
+++ b/hypothesis-python/tests/cover/test_cache_implementation.py
@@ -13,6 +13,8 @@
 #
 # END HEADER
 
+import threading
+from functools import partial
 from random import Random
 
 import pytest
@@ -293,3 +295,22 @@ def test_iterates_over_remaining_keys():
     for i in range(3):
         cache[i] = "hi"
     assert sorted(cache) == [1, 2]
+
+
+def test_cache_is_threadsafe_issue_2433_regression():
+    errors = []
+
+    def target():
+        for _ in range(1000):
+            try:
+                st.builds(partial(str))
+            except Exception as exc:
+                errors.append(exc)
+
+    workers = [threading.Thread(target=target) for _ in range(4)]
+    for worker in workers:
+        worker.start()
+    for worker in workers:
+        worker.join()
+
+    assert not errors