Repository: HypothesisWorks/hypothesis

Issue:
Very slow unique sampling of np.int8
While looking into a very slow test (numpy/test_gen_data::test_efficiently_generates_all_unique_array), I came across this oddity: Sampling sets of `np.int8` is dramatically slower than sampling 8-bit capped sets of `int`.

```python
import numpy as np
import pytest

from hypothesis import given, settings, strategies as st

full_sets = lambda inner: st.sets(inner, min_size=256, max_size=256)
examples = 20

@given(full_sets(st.from_type(np.int8)))
@settings(max_examples=examples)
def test_int8(_):
    pass

@given(full_sets(st.integers(min_value=-128, max_value=127)))
@settings(max_examples=examples)
def test_integers(_):
    pass
```

The performance is 100x worse for `np.int8`, caused by retries and invalid examples. It looks like it is caused by the `.map(int8)` preventing the filter from being optimized into data generation?


Statistics:

```
================================================================================== Hypothesis Statistics ===================================================================================

hypothesis-python/test_gen_int8.py::test_int8:

  - during reuse phase (0.01 seconds):
    - Typical runtimes: ~ 12ms, of which ~ 12ms in data generation
    - 1 passing examples, 0 failing examples, 0 invalid examples

  - during generate phase (5.94 seconds):
    - Typical runtimes: ~ 0-33 ms, of which ~ 0-32 ms in data generation
    - 19 passing examples, 0 failing examples, 376 invalid examples
    - Events:
      * 87.85%, Retried draw from integers(min_value=-128, max_value=127).map(int8).filter(not_yet_in_unique_list) to satisfy filter
      * 3.54%, invalid because: Aborted test because unable to satisfy integers(min_value=-128, max_value=127).map(int8).filter(not_yet_in_unique_list)

  - Stopped because settings.max_examples=20


hypothesis-python/test_gen_int8.py::test_integers:

  - during reuse phase (0.00 seconds):
    - Typical runtimes: ~ 1ms, of which ~ 1ms in data generation
    - 1 passing examples, 0 failing examples, 0 invalid examples

  - during generate phase (0.05 seconds):
    - Typical runtimes: ~ 1-2 ms, of which ~ 1-2 ms in data generation
    - 19 passing examples, 0 failing examples, 0 invalid examples

  - Stopped because settings.max_examples=20


=================================================================================== 
```


Discussion:
(fyi: just reporting, not working on)
Unclear what we can do about this, beyond complicating our fast path with support for a broader range of strategies.  "Seeing through" a `MappedStrategy` is only valid in cases where the transform callable is fairly pure, which means we'd _also_ need an allowlist for that, and I'm just not convinced it's worth it.  At some point we have to tell power users to write `st.sets(st.sampled_from([your stuff here]))` rather than rewriting it for them.
Thanks, and I appreciate your point. I guess the test name (test_**efficiently**_generates_all_unique_array) got me wound up.

Still: it would be nice for that power user, or the np extra, to be able to derive custom types that are as powerful as the built-in types. A few ideas below, for posterity if this is revisited.

1. We don't really need to check purity of the mapping, just repeat the uniqueness check afterwards. A mapping can only narrow the underlying strategy.
2. We could even add purity hints for `MappedStrategy`. If we know the mapping is bijective, we don't need to repeat the uniqueness check. While possibly making users feel clever and empowered, I agree this is probably not worth the effort — unless we see other cases. OTOH, it would be simpler than (1) if we limited see-through to bijective mappings.
3. Simpler would be to add a (possibly private/undocumented) special form of `.map()`. Perhaps an `astype=` argument to the strategy. This would be applied immediately to the result of `.do_draw()` without deferring to another strategy, and also anywhere these draws are replaced of course.

Anyway, feel free to close.
Actually, don't close quite yet. I'll fix it quickly (I think) in the numpy extra.

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/extra/numpy.py b/hypothesis-python/src/hypothesis/extra/numpy.py
--- a/hypothesis-python/src/hypothesis/extra/numpy.py
+++ b/hypothesis-python/src/hypothesis/extra/numpy.py
@@ -46,8 +46,14 @@
 from hypothesis.internal.coverage import check_function
 from hypothesis.internal.reflection import proxies
 from hypothesis.internal.validation import check_type
+from hypothesis.strategies._internal.lazy import unwrap_strategies
 from hypothesis.strategies._internal.numbers import Real
-from hypothesis.strategies._internal.strategies import Ex, T, check_strategy
+from hypothesis.strategies._internal.strategies import (
+    Ex,
+    MappedSearchStrategy,
+    T,
+    check_strategy,
+)
 from hypothesis.strategies._internal.utils import defines_strategy
 
 
@@ -393,7 +399,6 @@ def do_draw(self, data):
         return result
 
 
-@check_function
 def fill_for(elements, unique, fill, name=""):
     if fill is None:
         if unique or not elements.has_reusable_values:
@@ -508,6 +513,11 @@ def arrays(
             )
         elements = from_dtype(dtype, **(elements or {}))
     check_strategy(elements, "elements")
+    # If there's a redundant cast to the requested dtype, remove it.  This unlocks
+    # optimizations such as fast unique sampled_from, and saves some time directly too.
+    unwrapped = unwrap_strategies(elements)
+    if isinstance(unwrapped, MappedSearchStrategy) and unwrapped.pack == dtype.type:
+        elements = unwrapped.mapped_strategy
     if isinstance(shape, int):
         shape = (shape,)
     shape = tuple(shape)


Test patch: