Repository: HypothesisWorks/hypothesis

Issue:
min_satisfying_examples doesn't consider uniqueness
If you set `min_satisfying_examples` to a number higher than the number of unique possible inputs, it still will pass

```
In [1]: import hypothesis as h

In [2]: import hypothesis.strategies as st

In [3]: @h.given(st.permutations([True, False]))
   ...: @h.settings(min_satisfying_examples=5)
   ...: def test_foo(bools):
   ...:     pass
   ...:

In [4]: test_foo()

In [5]:
```

There aren't 5 different permutations of a size-2 list, so I would expect this test to fail due to not meeting `min_satisfying_examples`. It defeats the purpose of setting a minimum if the same input gets counted multiple times.


Discussion:
As an aside, a `settings` shortcut for "exhaustively check all inputs" would be super handy.
This is a good spot, but it's quite hard to fix because Hypothesis doesn't have any high level notion of example equality - it can only tell when two things are equal at the level of their underlying representation, which is a superset of whether they're actually distinct (it currently also often generates duplicates at *that* level, which should be fixed soon).

Additionally, the smallest non-zero number of "distinct" examples a test can currently have is 256. I have some plans to improve that, and when I do your example will happen to start working, but that's only because it's a power of two and e.g. `permutations([True, True, False])` will appear to have an infinite (or at least large) number of distinct examples.

In general the Hypothesis settings around examples are a bit of an unfortunate trap for the unwary because of how tied they are to Hypothesis's internals.

An additional problem is that it's not really obvious how to do better than this, what with the flexibility of both Python and Hypothesis - what counts as duplicates?
I think it's completely fair to say that uniqueness checking works up to `__hash__` and `__eq__` equality. And add a flag to disable it for when people are using objects that don't behave nicely with value equality
What do you do about the fact that people can mutate examples? Do you compare equality before or after they've mutated them? What do you do about the fact that some of the most common types from the standard library raise TypeError if you try to hash them?

I'm not completely ruling it out but it is *very* unlikely that Hypothesis will ever start checking the values it generates for equality. Currently Hypothesis has the property that it never actually inspects anything non-trivial about the values  it generates, and that's a very useful simplifying assumption. It would have to be a very big win to justify removing it, and I'm not convinced the case of search spaces that are small enough to explicitly enumerate is valuable enough to justify it.
A more likely, though also not without its problems, scenario is that at some point Hypothesis might start treating tests which would have the same output on failure as duplicates.
That makes sense.
I too found it surprising how many tests `@given(booleans())` results in. The question here is whether Hypothesis is supposed to test for flakiness too. Only then does it make sense to repeat the same inputs.

Admittedly I used it as a convenient shortcut to having a parametrized test. That may not be the purpose of this tool, but combining different tools can be messy, especially when some of them require subclassing `TestCase` or even a different runner. Having to do that just to prevent repeated permutations would be unfortunate.

If flakiness is not a consideration, I think it could make sense to filter equal inputs as late in the process as feasible, to have the smallest possible impact on the design. I'm assuming that the bottleneck is in running the tests, rather than in generating the inputs. Alternatively there could be a strategy that will not sample but take each input exactly once and signals exhaustion.

Resolving patch:
diff --git a/src/hypothesis/_settings.py b/src/hypothesis/_settings.py
--- a/src/hypothesis/_settings.py
+++ b/src/hypothesis/_settings.py
@@ -409,8 +409,12 @@ class Setting(object):
     default=5,
     description="""
 Raise Unsatisfiable for any tests which do not produce at least this many
-values that pass all assume() calls and which have not exhaustively covered the
-search space.
+values that pass all :func:`hypothesis.assume` calls and which have not
+exhaustively covered the search space.
+
+Note that examples are compared at the level of the underlying byte-stream -
+for example, :func:`~hypothesis.strategies.booleans` contains 256 unique
+examples at this level because it always uses one byte.
 """
 )
 
diff --git a/src/hypothesis/core.py b/src/hypothesis/core.py
--- a/src/hypothesis/core.py
+++ b/src/hypothesis/core.py
@@ -1111,6 +1111,7 @@ def find(specifier, condition, settings=None, random=None, database_key=None):
     random = random or new_random()
     successful_examples = [0]
     last_data = [None]
+    last_repr = [None]
 
     def template_condition(data):
         with BuildContext(data):
@@ -1127,22 +1128,19 @@ def template_condition(data):
 
         if settings.verbosity == Verbosity.verbose:
             if not successful_examples[0]:
-                report(lambda: u'Trying example %s' % (
-                    nicerepr(result),
-                ))
+                report(
+                    u'Tried non-satisfying example %s' % (nicerepr(result),))
             elif success:
                 if successful_examples[0] == 1:
-                    report(lambda: u'Found satisfying example %s' % (
-                        nicerepr(result),
-                    ))
+                    last_repr[0] = nicerepr(result)
+                    report(u'Found satisfying example %s' % (last_repr[0],))
                     last_data[0] = data
                 elif (
                     sort_key(hbytes(data.buffer)) <
                     sort_key(last_data[0].buffer)
-                ):
-                    report(lambda: u'Shrunk example to %s' % (
-                        nicerepr(result),
-                    ))
+                ) and nicerepr(result) != last_repr[0]:
+                    last_repr[0] = nicerepr(result)
+                    report(u'Shrunk example to %s' % (last_repr[0],))
                     last_data[0] = data
         if success and not data.frozen:
             data.mark_interesting()


Test patch:
diff --git a/tests/cover/test_verbosity.py b/tests/cover/test_verbosity.py
--- a/tests/cover/test_verbosity.py
+++ b/tests/cover/test_verbosity.py
@@ -82,7 +82,7 @@ def not_first(x):
                 return x not in seen
             find(integers(), not_first)
 
-    assert u'Trying example' in o.getvalue()
+    assert u'Tried non-satisfying example' in o.getvalue()
 
 
 def test_includes_intermediate_results_in_verbose_mode():