Repository: HypothesisWorks/hypothesis

Issue:
Consider adding a simple subsets / subsequences strategy?
I find myself implementing this every now and then, to draw subsets of some fixed set of elements:

```python
@st.composite
def subsets(draw, elements):
    return {e for e in elements if draw(st.booleans())}
```

(This is essentially equivalent to `st.sets(st.sampled_from(elements), max_size=len(elements))`, but simpler and presumably more efficient. (?))

Would this, or something like it, be useful enough to consider adding to the core Hypothesis strategies?


Discussion:
I'd definitely be fine with adding this to the API!

There are a couple things to consider though:

* Do we want it to have the usual collection size parameters of `min_size`, `average_size`, `max_size`? (I think yes probably, though they don't have to be in the initial PR if you'd rather do something minimal)
* What do we do about the ordering problem? Hypothesis kinda relies on having a stable order that things happen in across runs, and sets are in a potentially randomized order between runs. 😢 

One option for the latter would be to only support this for orderable types and to sort the values up front. Would that be acceptable for your use case?
Yup, those are good considerations!

The collection size parameters might be useful, yeah. If there's an elegant way to implement them without taking away from the simplicity of the base strategy, that would be cool (though I'd personally also be happy without them).

Regarding the ordering problem, my first instinct would be that the strategy should simply assume / require that the input is given in some reliable order, and warn or error if it isn't. This is what the `sampled_from` strategy does with its `check_sample` check: this strategy could probably just call the same `check_sample` and be fine?
I thought we were trying to get rid of `average_size` eventually?  Apart from that, collection size controls make sense.

Perhaps this could be implemented as an upgrade to `permutations()`?  Currently `permutations()` just casts to a list, which is nondeterministic if the input collection is unordered...  

*n*-choose-*k* is at least related, and equivalent to taking a subset once you discard the ordering.  It would make sense to me to use it as a general strategy, and just have a more efficient path for unordered operations internally.
> Regarding the ordering problem, my first instinct would be that the strategy should simply assume / require that the input is given in some reliable order, and warn or error if it isn't. This is what the sampled_from strategy does with its check_sample check: this strategy could probably just call the same check_sample and be fine?

I'm definitely fine with doing this is as a first pass! The problem is that it results in making it invalid to call `subsets` with a set argument, which I think is going to be a bit weird and trip people up a lot.
> I'm definitely fine with doing this is as a first pass! The problem is that it results in making it invalid to call `subsets` with a set argument, which I think is going to be a bit weird and trip people up a lot.

I don't think this is much different to `sampled_from`. When I started, I also intuitively called `sampled_from` with set arguments, but the warning sorted me out. (And the `check_sample` call makes sure that it works anyway, so I'd assume the same would hold for `subsets`.)
IIRC `sampled_from` only accepts unordered collections to preserve backwards compatibility - for a new function we should make a strict error (though of course with a helpful message).
Hmm, just thinking out loud, how's this for a strategy that also supports `min_size` / `max_size`?

```python
@st.composite
def subsets(draw, elements, *, min_size, max_size):
    def choices():
        always_size = min_size
        never_size = len(elements) - max_size
        maybe_size = len(elements) - always_size - never_size
        choices = (
            [True] * always_size +
            [draw(st.booleans()) for _ in range(maybe_size)] +
            [False] * never_size
        )
        assert len(elements) == len(choices)
        return draw(st.permutations(choices))

    return {element for (element, choose) in zip(elements, choices()) if choose}
```

If I understand it right, this variation should have the bonus feature of shrinking toward selecting earlier elements in the presence of `min_size` or `max_size`, due to how `permutations` works.
Thinking out loud some more, the above might actually be more generally useful as a list-based _n_-choose-_k_ strategy for subsequences, simply by changing the final set comprehension to a list comprehension. It could be named `subsequences` then instead?

In that case, getting actual subsets of a set would simply be a special case of doing `subsequences(elements).map(set)`, which I could certainly live with.
Simpler: add the min and max size arguments to `permutations`, along with the sorting/warning we discussed, then (psudeocode) `return draw(permutations(values))[:draw(integers(min_size, max_size))]`.  And yep, optionally `.map(set)`.
@Zac-HD: That wouldn't be the same thing as `subsequences`; it always permutes, rather than generating an ordered subsequence of the original.

It also  "wastes" the entropy spent on the elements that get discarded: I'm not sure how much effect that has on the effectiveness of shrinking, but my intuition tells me it might? (With the `subsequences` approach, all the entropy ends up in the output.)
True!  (assuming we make it not a set comprehension 😉)  I see that as a feature though; and it shrinks towards shortest prefix in input order.

The shrinker would be fine; worst case it lowers that value alone and notices no change in status.  The most efficient way is probably to loop over sampling an unused index, then including it in the output if drawing from an appropriately biased coin is True - this would allow the shrinker to delete pairs of draw calls.  However at this higher level I wouldn't worry about the shrinker at all!
Here's the updated straw implementation, just for easier reference:

```python
@st.composite
def subsequences(draw, elements, *, min_size, max_size):
    def choices():
        always_size = min_size
        never_size = len(elements) - max_size
        maybe_size = len(elements) - always_size - never_size
        choices = (
            [True] * always_size +
            [draw(st.booleans()) for _ in range(maybe_size)] +
            [False] * never_size
        )
        assert len(elements) == len(choices)
        return draw(st.permutations(choices))

    return [element for (element, choose) in zip(elements, choices()) if choose]
```
Hi, this looks like a good one to implement for the PyCon sprints; if no-one has any objections, I'll get started implementing this 😁 (see #1513)

Ok, I have a proof-of-concept implemented by https://github.com/tkcranny/hypothesis/commit/e49a0182ca92dcd0fe654cfc3189c6f413f01033

I'd love some feedback on how the argument validation should be done, and perhaps on more/better testing.

@tkcranny: Thanks for taking this to implementation! :)
Future work on this issue should build on #1533, with appropriate credit for that work.

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/internal/reflection.py b/hypothesis-python/src/hypothesis/internal/reflection.py
--- a/hypothesis-python/src/hypothesis/internal/reflection.py
+++ b/hypothesis-python/src/hypothesis/internal/reflection.py
@@ -427,8 +427,7 @@ def get_pretty_function_description(f):
         return repr(f)
     name = f.__name__
     if name == "<lambda>":
-        result = extract_lambda_source(f)
-        return result
+        return extract_lambda_source(f)
     elif isinstance(f, types.MethodType):
         self = f.__self__
         if not (self is None or inspect.isclass(self)):
diff --git a/hypothesis-python/src/hypothesis/searchstrategy/collections.py b/hypothesis-python/src/hypothesis/searchstrategy/collections.py
--- a/hypothesis-python/src/hypothesis/searchstrategy/collections.py
+++ b/hypothesis-python/src/hypothesis/searchstrategy/collections.py
@@ -21,7 +21,11 @@
 from hypothesis.errors import InvalidArgument
 from hypothesis.internal.compat import OrderedDict
 from hypothesis.internal.conjecture.utils import combine_labels
-from hypothesis.searchstrategy.strategies import MappedSearchStrategy, SearchStrategy
+from hypothesis.searchstrategy.strategies import (
+    MappedSearchStrategy,
+    SearchStrategy,
+    filter_not_satisfied,
+)
 
 
 class TupleStrategy(SearchStrategy):
@@ -143,13 +147,18 @@ def do_draw(self, data):
         seen = set()
         result = []
 
+        # We construct a filtered strategy here rather than using a check-and-reject
+        # approach because some strategies have special logic for generation under a
+        # filter, and FilteredStrategy can consolidate multiple filters.
+        filtered = self.element_strategy.filter(lambda val: self.key(val) not in seen)
         while elements.more():
-            value = data.draw(self.element_strategy)
-            k = self.key(value)
-            if k in seen:
+            value = filtered.filtered_strategy.do_filtered_draw(
+                data=data, filter_strategy=filtered
+            )
+            if value is filter_not_satisfied:
                 elements.reject()
             else:
-                seen.add(k)
+                seen.add(self.key(value))
                 result.append(value)
         assert self.max_size >= len(result) >= self.min_size
         return result
diff --git a/hypothesis-python/src/hypothesis/searchstrategy/misc.py b/hypothesis-python/src/hypothesis/searchstrategy/misc.py
--- a/hypothesis-python/src/hypothesis/searchstrategy/misc.py
+++ b/hypothesis-python/src/hypothesis/searchstrategy/misc.py
@@ -27,7 +27,7 @@ class BoolStrategy(SearchStrategy):
     distribution."""
 
     def __repr__(self):
-        return u"BoolStrategy()"
+        return "BoolStrategy()"
 
     def calc_has_reusable_values(self, recur):
         return True


Test patch:
diff --git a/hypothesis-python/tests/cover/test_slippage.py b/hypothesis-python/tests/cover/test_slippage.py
--- a/hypothesis-python/tests/cover/test_slippage.py
+++ b/hypothesis-python/tests/cover/test_slippage.py
@@ -20,7 +20,7 @@
 import pytest
 
 import hypothesis.strategies as st
-from hypothesis import Phase, given, settings
+from hypothesis import Phase, assume, given, settings
 from hypothesis.database import InMemoryExampleDatabase
 from hypothesis.errors import Flaky, MultipleFailures
 from hypothesis.internal.conjecture.engine import MIN_TEST_CALLS
@@ -36,6 +36,9 @@ def test(i):
         if abs(i) < 1000:
             return
         if target[0] is None:
+            # Ensure that we have some space to shrink into, so we can't
+            # trigger an minimal example and mask the other exception type.
+            assume(1003 < abs(i))
             target[0] = i
         exc_class = TypeError if target[0] == i else ValueError
         raise exc_class()
diff --git a/hypothesis-python/tests/nocover/test_sampled_from.py b/hypothesis-python/tests/nocover/test_sampled_from.py
--- a/hypothesis-python/tests/nocover/test_sampled_from.py
+++ b/hypothesis-python/tests/nocover/test_sampled_from.py
@@ -21,8 +21,9 @@
 
 import hypothesis.strategies as st
 from hypothesis import given
+from hypothesis.errors import FailedHealthCheck
 from hypothesis.internal.compat import hrange
-from tests.common.utils import counts_calls
+from tests.common.utils import counts_calls, fails_with
 
 
 @pytest.mark.parametrize("n", [100, 10 ** 5, 10 ** 6, 2 ** 25])
@@ -61,3 +62,14 @@ def forbid(s, forbidden):
 @given(rare_value_strategy(n=128, target=80))
 def test_chained_filters_find_rare_value(x):
     assert x == 80
+
+
+@fails_with(FailedHealthCheck)
+@given(st.sets(st.sampled_from(range(10)), min_size=11))
+def test_unsat_sets_of_samples(x):
+    assert False
+
+
+@given(st.sets(st.sampled_from(range(50)), min_size=50))
+def test_efficient_sets_of_samples(x):
+    assert x == set(range(50))