Repository: HypothesisWorks/hypothesis

Issue:
Document our de-facto thread-safety policy
As pointed out in #2717, we don't appear to have any documented policy on whether or not Hypothesis is thread-safe.

Actually making Hypothesis more thread-safe would be a bunch of work and out-of-scope for this issue, so here I'm mostly focused on being more transparent about the current situation.

I see three main scenarios worth documenting:

- Running in multiple processes (e.g. under `pytest -nauto`)
  - I don't know if we have explicit tests for this, but our own test suite routinely runs in multiple processes, so hopefully we should notice any problems pretty quickly.
  - This mostly involves making sure that our file-backed database can't tread on its own toes too badly.

- Running separate tests in multiple threads
  - Hypothesis isn't explicitly designed or tested for this, so it's not formally “supported“, but anecdotally it does mostly work and we would like it to keep working.
  - This mostly involves being careful about global state, shared caches, and cached strategies.
  - We accept bug reports and reasonable patches.

- Using multiple threads within a single test
  - Hypothesis assumes that tests are single-threaded, or do a sufficiently-good job of pretending to be single-threaded.
  - Tests that use helper threads internally should be OK, but the user must be careful to ensure that test outcomes are still deterministic.
    - Tests that change their dynamic draw behaviour based on helper-thread timing count as nondeterministic, so don't do that.
    - Deterministic stack traces might be a danger point here; I'm not sure.
  - Interacting with any Hypothesis APIs from helper threads might do weird/bad things, so don't do that.
    - This is because we rely on thread-local variables in a few places, and haven't explicitly tested/audited how they respond to cross-thread API calls.
    - Dynamic draws from `st.data()` are the most obvious danger, but other APIs might also be subtly affected.

Does this seem mostly accurate?


Discussion:
Yep, looks good to me - thanks for writing it up @Zalathar!

I'd note under "Running separate tests in multiple threads" that running a *single* test in multiple threads has all these problems but more so - it might work for simple examples, but I would expect this to be pretty fragile.

Resolving patch:
diff --git a/hypothesis-python/docs/conf.py b/hypothesis-python/docs/conf.py
--- a/hypothesis-python/docs/conf.py
+++ b/hypothesis-python/docs/conf.py
@@ -83,7 +83,7 @@ def setup(app):
     "attrs": ("https://www.attrs.org/en/stable/", None),
 }
 
-autodoc_mock_imports = ["pandas", "redis"]
+autodoc_mock_imports = ["numpy", "pandas", "redis"]
 
 # This config value must be a dictionary of external sites, mapping unique
 # short alias names to a base URL and a prefix.
diff --git a/tooling/src/hypothesistooling/__main__.py b/tooling/src/hypothesistooling/__main__.py
--- a/tooling/src/hypothesistooling/__main__.py
+++ b/tooling/src/hypothesistooling/__main__.py
@@ -358,17 +358,12 @@ def run_tox(task, version):
 # Via https://github.com/pyenv/pyenv/tree/master/plugins/python-build/share/python-build
 PY36 = "3.6.12"
 PY37 = "3.7.9"
-PY38 = "3.8.6"
-PY39 = "3.9.0"
+PY38 = PYMAIN = "3.8.7"  # Note: keep this in sync with build.sh
+PY39 = "3.9.1"
 PYPY36 = "pypy3.6-7.3.1"
 PYPY37 = "pypy3.7-7.3.2"
 
 
-@task()
-def install_core():
-    install.python_executable(PY36)
-
-
 # ALIASES are the executable names for each Python version
 ALIASES = {PYPY36: "pypy3", PYPY37: "pypy3"}
 
@@ -417,7 +412,7 @@ def check_pypy37():
 
 
 def standard_tox_task(name):
-    TASKS["check-" + name] = python_tests(lambda: run_tox(name, PY36))
+    TASKS["check-" + name] = python_tests(lambda: run_tox(name, PYMAIN))
 
 
 standard_tox_task("nose")
@@ -434,17 +429,12 @@ def standard_tox_task(name):
 
 @task()
 def check_quality():
-    run_tox("quality", PY36)
-
-
-examples_task = task(
-    if_changed=(hp.PYTHON_SRC, os.path.join(hp.HYPOTHESIS_PYTHON, "examples"))
-)
+    run_tox("quality", PYMAIN)
 
 
-@examples_task
+@task(if_changed=(hp.PYTHON_SRC, os.path.join(hp.HYPOTHESIS_PYTHON, "examples")))
 def check_examples3():
-    run_tox("examples3", PY36)
+    run_tox("examples3", PYMAIN)
 
 
 @task()


Test patch: