Repository: HypothesisWorks/hypothesis

Issue:
settings.derandomize is ineffective for some strategies
This was discovered while working on #962, and "fixed" by calling `random.seed(0)` before each doctest block.  Things that seem to be affected:

- strategies using `.map` and `.filter`
- `st.recursive` and `st.deferred`
- `find`

While `integers().example()` is unaffected, along with `tuple(...)`, `builds(...)`, etc.  I therefore suspect that something in the composition of strategies is making the derandomize machinery fail.

Note that this has only been observed in output from `s.example()` - it is unknown whether it also affects tests defined with `@given`.


Discussion:
I have an example code with @given in which derandomize does not work. I have tried to modify it into a minimal verifiable example (that is why it might seem slightly odd).
The attached zip file [hypothesis_derandomize_bug.zip](https://github.com/HypothesisWorks/hypothesis-python/files/1830032/hypothesis_derandomize_bug.zip) contains the file derandomize_bug.py. This is the code that observes the bug with both hypothesis 3.49.1 (from Arch Linux repository) and with the latest checkout of the git repository (tag 3.50.2).

If I change the code slightly, the bug sometimes disappears, which leads me to think that it only manifests with certain random seeds.

What is also interesting is that the code is not completely random, it switches between two different outcomes (tested on thousands of runs). The outcomes are also part of the zip file (derandomize_bug_result1.txt and ...result2.txt).
I have tried running git bisect with the following results:
- The code given above stops working between 3.44.20 and 3.44.21.
- A more convoluted code (which is not entirely mine and I do not have the permission to publish it here) has the same problem right after 3.30.1 (with the commit tagged 3.30.1 it works OK, with commit 341a9bc8... it exhibits the nondeterministic behaviour).
I don't know if this is of any help, though.
I encountered what I believe is the same issue and debugged it, and it boiled down to hypothesis leaning on the hash of strings being predictable. When hash randomization is active, derandomize results in a unique seed. This is always the case in python3, and in python2 can be activated with the interpreter `-R` option or setting `PYTHONHASHSEED=random` in the environment.

I did a hack to change derandomize to use hashlib.sha1, and it started working. I can (re-derive and) contribute this patch, if there's interest.
> I did a hack to change derandomize to use hashlib.sha1, and it started working. I can (re-derive and) contribute this patch, if there's interest.

That would be fantastic - especially if you can also write up a brief explanation of how the bug arose and why sha1 avoids it :smile: 
>  it boiled down to hypothesis leaning on the hash of strings being predictable.
> I did a hack to change derandomize to use hashlib.sha1, and it started working. I can (re-derive and) contribute this patch, if there's interest.

Hmm. Are you sure? I mean it makes sense that that's the problem somewhere, but I don't think it's in the derandomize code! [derandomize uses function_digest](https://github.com/HypothesisWorks/hypothesis-python/blob/master/src/hypothesis/core.py#L301), which is [already based on hashlib](https://github.com/HypothesisWorks/hypothesis-python/blob/master/src/hypothesis/internal/reflection.py#L63).

I'm sure you're right that the problem is hash randomization, I just think that it's probably got to be us depending on dict iteration order or something somewhere.
@DRMacIver `function_digest` says "Returns a string". There it is. It has to return an int for stable hashing.
@nikola-benes - can you test again to confirm that #1217 fixes the issue?  Note that this assumes you are running Python2 with hash randomisation enabled - if not I'll need to go trace everything again :disappointed: 

----------------

@bukzor - `random.seed` is stable given a string for any Python >= 3.2, so you're right that using an int fixes it.  We don't need to use sha1 though - the instability comes from the stdlib handling of the return type, not how we calculate the value!
> @DRMacIver function_digest says "Returns a string". There it is. It has to return an int for stable hashing.

Oh! Right. That makes sense. Sigh. Thanks.

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/core.py b/hypothesis-python/src/hypothesis/core.py
--- a/hypothesis-python/src/hypothesis/core.py
+++ b/hypothesis-python/src/hypothesis/core.py
@@ -54,7 +54,7 @@
 from hypothesis.statistics import note_engine_for_statistics
 from hypothesis.internal.compat import ceil, hbytes, qualname, \
     str_to_bytes, benchmark_time, get_type_hints, getfullargspec, \
-    encoded_filepath, bad_django_TestCase
+    int_from_bytes, encoded_filepath, bad_django_TestCase
 from hypothesis.internal.coverage import IN_COVERAGE_TESTS
 from hypothesis.utils.conventions import infer, not_set
 from hypothesis.internal.escalation import is_hypothesis_file, \
@@ -300,7 +300,7 @@ def get_random_for_wrapped_test(test, wrapped_test):
         return Random(
             wrapped_test._hypothesis_internal_use_seed)
     elif settings.derandomize:
-        return Random(function_digest(test))
+        return Random(int_from_bytes(function_digest(test)))
     elif global_force_seed is not None:
         return Random(global_force_seed)
     else:


Test patch: