Repository: numpy/numpy

Issue:
TST: Add the first test using hypothesis
This pull request adds the first test that uses hypothesis and hence brings in hypothesis as an additional test dependency.

@mattip Could you take a look at this please?



Discussion:
Thanks @kitchoi! 

This will need to be discussed on the mailing list, since it adds a new dependency.
@rgommers Thanks!  Yes I was wondering about that too, which is why I only added one test. The idea is that if the community agrees this is a good idea, this pull request will bring in the initial setup so that we can add more tests afterwards.  I am also happy to add more tests in this pull request, to make the inclusion of a new test dependency more worthwhile.
I think @rgommers is simply asking that you post a mail to numpy-discussion@python.org pointing to this PR, with a few words about hypothesis and why you think it is a good idea to start using it. The questions people might ask, fwiw, would be "does it find anything our tests miss" (the current test seems to pass), and "how much does it slow things down" (it would be nice to check that).
I see. Thanks @mattip 

When I started working on this in EuroScipy sprint, I was under the impression that there was already interest in the community to bring in hypothesis to numpy. I don't really have a strong opinion about bringing in hypothesis and therefore I am sorry to say that I would rather let someone else who does to start the discussion.

Shall we close this or does anyone of you want to be that advocate?
Let's leave this open and someone else will write to the list. Thanks for working out what needs to change for this to go in
> When I started working on this in EuroScipy sprint, I was under the impression that there was already interest in the community to bring in hypothesis to numpy. 

We discussed this at the SciPy'19 sprints as well. There was some interest there, also from one of the `hypothesis` maintainers (@Zac-HD), and the plan was to open a PR with some tests (like you did here) so we could have a discussion based on concrete examples and benefits rather than "hypothesis looks cool". Your PR just happens to be the first to be submitted.
> We discussed this at the SciPy'19 sprints as well. There was some interest there, also from one of the hypothesis maintainers (@Zac-HD), and the plan was to open a PR with some tests (like you did here) so we could have a discussion based on concrete examples and benefits rather than "hypothesis looks cool". Your PR just happens to be the first to be submitted.

Thank you for the clarification. This gives me a better understanding of where we are at.
If there is any part of the test suite you know could benefit from hypothesis, please let me know and I am happy to explore those areas. Perhaps a bit unconventional but given this requires some experimentation, if anyone wants to push directly to this branch, I am happy to add collaborators to my forked repo. Just let me know.
Hi all!  Hypothesis maintainer here and very happy to answer any questions you might have.  To get started:

## What does Hypothesis do that other tests don't?

Computer-generated test cases tend to expose edge cases that hand-written tests don't.  Usually they *could* have been exposed by hand-written tests, but it would take far more effort to do so without tools.

For Numpy itself, #10930, #13089, and #14239 have all been found by downstream users of Hypothesis!

## How slow are Hypothesis tests?

This mostly depends on how many test cases you run - the default is 100 cases but you can choose any other number on a global or per-test basis.  Typically generating data takes between 1% and 20% of the runtime for that test function, though it could be higher if we generate large arrays and apply a fast operation.

An important point is that we *do not* suggest using Hypothesis for everything!  Property-based tests tend to be more general but less numerous than traditional unit tests, for a given level of testing.  You can easily skip them from certain CI jobs (`pytest -m "not hypothesis"`) if that becomes an issue, or use a setting profile that runs fewer examples in CI.

Performance is rarely a problem in practice though, so I would ultimately suggest waiting to measure before trying to optimize!

## Reproducing test failures from CI, users, etc.

We have [docs about this](https://hypothesis.readthedocs.io/en/latest/reproducing.html) :smile:.  To summarize, the test may behave differently on different Python versions or operating systems (but that is true of non-Hypothesis tests too); short of that Hypothesis will print the input example and we can use the `print_blob=True` setting to be sure of reproducing the failure.  Changes between Hypothesis versions are uncommon but not unknown; which is why the `print_blob` setting dumps the exact Hypothesis version too.

I would *discourage* using the `@seed` decorator or `derandomize=True` setting, as the net effect is to reduce the number of bugs found.  We (Hypothesis developers) spend a lot of time ensuring that bugs can be reproduced once found, and I'd personally be happy to help if the Numpy project is having any trouble with Hypothesis.

## What parts of Numpy would Hypothesis be especially valuable for?

Generically, parts of your code which are particularly complex, where you have high-level invariants you can check, or both.  I would suggest paying particular attention to IO and serialisation round-trips, the dtype system, and broadcasting - the previous issues mentioned above also provide some examples.  @rsokl might also have some ideas!

Personally, I like to start simply by trying to call some tricky code using Hypothesis to generate inputs - without asserting anything at all about the result.  This often uncovers a more detailed definition of what inputs should be considered valid than has been documented!
Thanks @Zac-HD!

It seems to me that the best time to run hypothesis tests is (1) when writing new functionality, and (2) when touching existing code (in CI). Perhaps less so by end users when they run `np.test()` - it feels like we're then going to end up with lots of corner case bug reports (I could be wrong there).

Perhaps there's also a nice way to add generic tests for, for example, `nan`, `inf` and empty array input - that's the kind of thing that often gets forgotten about.
> It seems to me that the best time to run hypothesis tests is (1) when writing new functionality, and (2) when touching existing code (in CI). Perhaps less so by end users when they run `np.test()` - it feels like we're then going to end up with lots of corner case bug reports (I could be wrong there).

Empirically it's *rare* for additional runs in CI or on user machines to find new bugs after the tests have passed on a pull request and been merged, but not unknown - Hypothesis self-tests pretty heavily and we see maybe 1-3 such cases per year.  

On the other hand this is basically a perfect case for [settings profiles](https://hypothesis.readthedocs.io/en/latest/settings.html#settings-profiles): we can configure a profile for `np.test()` with `derandomize=True`.  That would allow users to run the same tests even on very exotic platforms, while still in deterministic mode.

> Perhaps there's also a nice way to add generic tests for, for example, `nan`, `inf` and empty array input - that's the kind of thing that often gets forgotten about.

Hypothesis may generate non-finite numbers wherever you ask for floats without specifying that they're disallowed.  For non-Hypothesis tests, I'd just use a checklist - they're both useful reminders for maintainers and a great way of documenting and sharing project best-practice with new contributors.

I'm not quite sure what you mean by empty arrays - those with a dimension of size zero?  Uninitialised values?  
I'm  +1 on using hypothesis for testing; I've already used it to evalutate NumPy & SciPy PRs in the past. Indeed, I maintain small property-based testing repos for [NumPy](https://github.com/tylerjereddy/numpy-property) and [SciPy](https://github.com/tylerjereddy/scipy-property) for this purpose.

On the NumPy side, this revealed some interesting details when reviewing Eric's [clip ufunc PR](https://github.com/numpy/numpy/pull/12519), that would have been difficult for a human to probe in a sane amount of time.
> On the other hand this is basically a perfect case for [settings profiles](https://hypothesis.readthedocs.io/en/latest/settings.html#settings-profiles): we can configure a profile for `np.test()` with `derandomize=True`. That would allow users to run the same tests even on very exotic platforms, while still in deterministic mode.

That makes perfect sense to me.

> I'm not quite sure what you mean by empty arrays - those with a dimension of size zero? Uninitialised values?

One of the dimensions has size zero, so `x.size == 0`. `x.shape` still matters, and can be n-D (e.g. `(5, 0, 3)` or `(0, 4)`. 

Empty arrays are quite easy to get in regular code, and often lead to unexpected behavior.

> For non-Hypothesis tests, I'd just use a checklist - they're both useful reminders for maintainers and a great way of documenting and sharing project best-practice with new contributors.

True. On the other hand, few people seem to read our existing docs on how to test/document in detail, so built-in testing is nice:)
> Perhaps there's also a nice way to add generic tests for, for example, nan, inf and empty array input - that's the kind of thing that often gets forgotten about.

@rgommers, the following hypothesis strategy:

```python
import hypothesis.extra.numpy as hnp
hnp.arrays(shape=hnp.array_shapes(min_side=0), dtype=hnp.floating_dtypes())
```

will take care of that straight away. E.g.

```python
>>> strat = hnp.arrays(
    shape=hnp.array_shapes(min_side=0, min_dims=0), dtype=hnp.floating_dtypes()
)
>>> [strat.example() for i in range(5)]
[array(nan, dtype=float16),
 array([], shape=(3, 0), dtype=float16),
array([[-inf, -inf, -inf, -inf, -inf],
        [-inf, -inf, -inf, -inf, -inf],
        [-inf, -inf, -inf, -inf, -inf],
        [-inf, -inf, -inf, -inf, -inf],
 array(0., dtype=float16),
 array([           inf, -2.0721057e+16,  9.9999997e-06,           -inf,
                  -inf], dtype=float32)]
```

You could also construct strategies that only explore the edge cases that you are concerned with (although that is a little bit antithetical to property-based testing), and create a simple decorator that runs functions through these cases.
Could we find a few more test cases for hypothesis: in concatenation with various shapes, or unicode round-tripping, or pickling/unpickling? 
> Could we find a few more test cases for hypothesis: in concatenation with various shapes, or unicode round-tripping, or pickling/unpickling?

@mattip I'd prefer to make these a follow-up PR just so we don't push everything on to new contributors, but there is some prior art:

- @tylerjereddy's [clip tests](https://github.com/tylerjereddy/numpy-property/blob/master/tests/test_clip.py) are great (though I'd translate the `integers(...)` strategies as `from_dtype(np.int64)`)
- @rsokl's [`einsum` tests](https://github.com/rsokl/MyGrad/blob/master/tests/linalg/test_einsum.py), which found #10559 (and other tests)
- Testing `np.dtype()` *ala* #14239 
- Testing IO that round-trips do in fact round-trip is always worthwhile; especially with a wide variety of shapes, dtypes, and IO methods.

I could see a case for copying over the first, but IMO the rest of them should be in separate PRs not least because they might find some bugs - and that always slows things down!

If it helps, I'd be happy to write a PR full of IO tests after this one is merged :smile: 
@mattip / @rgommers - is there anything I could do to help get this merged?
My hesitation is that the test showcased here is not a very good example of how I would like to see hypothesis used.
- There are not comments for people who are not familiar with hypothesis what is going on
- Character-based ndarrays are not really our bread-and-butter. Are there examples where hypothesis will help with numerical code, and not just with `__repr__`?
- Even better, can we use any of the issues/PRs/comments in the comment above to replace current tests, with comments to why hypothesis is better?

This PR should also extend the documentation somewhere with best practices around hypothesis. Has another project already done such documentation and we can reference it?

The [mailing list discussion](https://mail.python.org/pipermail/numpy-discussion/2019-September/080004.html) garnered one response which was "let's discuss it on the PR", I guess the general community has no real opinion.
> This PR should also extend the documentation somewhere with best practices around hypothesis. Has another project already done such documentation and we can reference it?

Agreed if there exists a battle-tested set of best practices, it would be good to reference it, but I don't know where it is. On top of that, may be there are sets of best practices that only apply to numpy?  For example, assuming we want reproducible builds (yes Zac discourages this), I am not sure whether we should use `seed` per test, or use [settings](https://hypothesis.readthedocs.io/en/latest/settings.html#settings-profiles) for the entire build.
> Character-based ndarrays are not really our bread-and-butter. Are there examples where hypothesis will help with numerical code, and not just with `__repr__`?

#10930 and #13089 are serious Numpy bugs which were detect using Hypothesis *in downstream projects*.  I've spoken to @rsokl and he's happy to contribute his `einsum` test - it's a complicated one but covers a far wider variety of signatures and inputs than would be possible with unit tests.

> Even better, can we use any of the issues/PRs/comments in the comment above to replace current tests, with comments to why hypothesis is better?

For example, this test would have failed for any array that did not contain a `1`:

https://github.com/numpy/numpy/blob/c01014507c6f128394ba1a66f91ef32fc94bbeba/numpy/core/tests/test_einsum.py#L490-L493

Generating a variety of shapes, as well as element values, is even more powerful.

> assuming we want reproducible builds (yes Zac discourages this), I am not sure whether we should use seed per test, or use settings for the entire build.

Use a settings profile with `derandomize=True`, this use-case is why the setting exists.  I do want to clarify that Hypothesis is designed so that failures are always reproducible, even from randomly generated inputs.

> This PR should also extend the documentation somewhere with best practices around hypothesis. Has another project already done such documentation and we can reference it?

I think this reduces to "when writing tests, use Hypothesis in the obvious way".  If that doesn't work, we'd consider that a bug in Hypothesis... the only tweaks I can think of is tuning the project-level settings profiles to run more or fewer examples for performance reasons.
> #10930 and #13089 are serious Numpy bugs which were detect using Hypothesis

Please add the hypothesis tests that detected those bugs into this PR, or open a new one with the tests, and mention the relevant issues. Can the tests affected by those issues be converted to hypothesis tests? That would be more convincing.

> when writing tests, use Hypothesis in the obvious way

That would be a great start to [this part](https://numpy.org/devdocs/dev/index.html#test-coverage) of the documentation, in a sentence about what kind of tests lend themselves to using hypothesis, with a link to what "obvious way" means for hypothesis.
@mattip - I'm aiming to open a new PR with those tests and some docs later this week :smile: 

And re-reading my comment, it's not helpful at all :sweat:... I meant something like "I would use Hypothesis for almost all (>90%) tests, and in almost all cases do so by describing the input data with builtin strategies with essentially the same test body I would usually write".  

People usually pick it up pretty quickly - the user experience is almost exactly that of `@pytest.mark.parametrize` where you haven't passed the expected output, but with a much more concise description of the inputs and wider variety of test cases.

Resolving patch:
diff --git a/numpy/conftest.py b/numpy/conftest.py
--- a/numpy/conftest.py
+++ b/numpy/conftest.py
@@ -3,6 +3,7 @@
 """
 import os
 
+import hypothesis
 import pytest
 import numpy
 
@@ -12,6 +13,12 @@
 _old_fpu_mode = None
 _collect_results = {}
 
+# See https://hypothesis.readthedocs.io/en/latest/settings.html
+hypothesis.settings.register_profile(
+    name="numpy-profile", deadline=None, print_blob=True,
+)
+hypothesis.settings.load_profile("numpy-profile")
+
 
 def pytest_configure(config):
     config.addinivalue_line("markers",


Test patch:
diff --git a/doc/TESTS.rst.txt b/doc/TESTS.rst.txt
--- a/doc/TESTS.rst.txt
+++ b/doc/TESTS.rst.txt
@@ -360,7 +360,17 @@ deterministic by setting the random number seed before generating it.  Use
 either Python's ``random.seed(some_number)`` or NumPy's
 ``numpy.random.seed(some_number)``, depending on the source of random numbers.
 
+Alternatively, you can use `Hypothesis`_ to generate arbitrary data.
+Hypothesis manages both Python's and Numpy's random seeds for you, and
+provides a very concise and powerful way to describe data (including
+``hypothesis.extra.numpy``, e.g. for a set of mutually-broadcastable shapes).
+
+The advantages over random generation include tools to replay and share
+failures without requiring a fixed seed, reporting *minimal* examples for
+each failure, and better-than-naive-random techniques for triggering bugs.
+
 
 .. _nose: https://nose.readthedocs.io/en/latest/
 .. _pytest: https://pytest.readthedocs.io
 .. _parameterization: https://docs.pytest.org/en/latest/parametrize.html
+.. _Hypothesis: https://hypothesis.readthedocs.io/en/latest/
diff --git a/numpy/core/tests/test_arrayprint.py b/numpy/core/tests/test_arrayprint.py
--- a/numpy/core/tests/test_arrayprint.py
+++ b/numpy/core/tests/test_arrayprint.py
@@ -1,6 +1,8 @@
 # -*- coding: utf-8 -*-
 import sys
 import gc
+from hypothesis import given
+from hypothesis.extra import numpy as hynp
 import pytest
 
 import numpy as np
@@ -393,6 +395,18 @@ def test_wide_element(self):
             "[ 'xxxxx']"
         )
 
+    @given(hynp.from_dtype(np.dtype("U")))
+    def test_any_text(self, text):
+        # This test checks that, given any value that can be represented in an
+        # array of dtype("U") (i.e. unicode string), ...
+        a = np.array([text, text, text])
+        # casting a list of them to an array does not e.g. truncate the value
+        assert_equal(a[0], text)
+        # and that np.array2string puts a newline in the expected location
+        expected_repr = "[{0!r} {0!r}\n {0!r}]".format(text)
+        result = np.array2string(a, max_line_width=len(repr(text)) * 2 + 3)
+        assert_equal(result, expected_repr)
+
     @pytest.mark.skipif(not HAS_REFCOUNT, reason="Python lacks refcounts")
     def test_refcount(self):
         # make sure we do not hold references to the array due to a recursive
diff --git a/numpy/core/tests/test_numeric.py b/numpy/core/tests/test_numeric.py
--- a/numpy/core/tests/test_numeric.py
+++ b/numpy/core/tests/test_numeric.py
@@ -14,6 +14,9 @@
     assert_warns, HAS_REFCOUNT
     )
 
+from hypothesis import assume, given, strategies as st
+from hypothesis.extra import numpy as hynp
+
 
 class TestResize:
     def test_copies(self):
@@ -2018,6 +2021,68 @@ def test_NaT_propagation(self, arr, amin, amax):
         actual = np.clip(arr, amin, amax)
         assert_equal(actual, expected)
 
+    @given(data=st.data(), shape=hynp.array_shapes())
+    def test_clip_property(self, data, shape):
+        """A property-based test using Hypothesis.
+
+        This aims for maximum generality: it could in principle generate *any*
+        valid inputs to np.clip, and in practice generates much more varied
+        inputs than human testers come up with.
+
+        Because many of the inputs have tricky dependencies - compatible dtypes
+        and mutually-broadcastable shapes - we use `st.data()` strategy draw
+        values *inside* the test function, from strategies we construct based
+        on previous values.  An alternative would be to define a custom strategy
+        with `@st.composite`, but until we have duplicated code inline is fine.
+
+        That accounts for most of the function; the actual test is just three
+        lines to calculate and compare actual vs expected results!
+        """
+        # Our base array and bounds should not need to be of the same type as
+        # long as they are all compatible - so we allow any int or float type.
+        dtype_strategy = hynp.integer_dtypes() | hynp.floating_dtypes()
+
+        # The following line is a total hack to disable the varied-dtypes
+        # component of this test, because result != expected if dtypes can vary.
+        dtype_strategy = st.just(data.draw(dtype_strategy))
+
+        # Generate an arbitrary array of the chosen shape and dtype
+        # This is the value that we clip.
+        arr = data.draw(hynp.arrays(dtype=dtype_strategy, shape=shape))
+
+        # Generate shapes for the bounds which can be broadcast with each other
+        # and with the base shape.  Below, we might decide to use scalar bounds,
+        # but it's clearer to generate these shapes unconditionally in advance.
+        in_shapes, result_shape = data.draw(
+            hynp.mutually_broadcastable_shapes(
+                num_shapes=2,
+                base_shape=shape,
+                # Commenting out the min_dims line allows zero-dimensional arrays,
+                # and zero-dimensional arrays containing NaN make the test fail.
+                min_dims=1  
+                            
+            )
+        )
+        amin = data.draw(
+            dtype_strategy.flatmap(hynp.from_dtype)
+            | hynp.arrays(dtype=dtype_strategy, shape=in_shapes[0])
+        )
+        amax = data.draw(
+            dtype_strategy.flatmap(hynp.from_dtype)
+            | hynp.arrays(dtype=dtype_strategy, shape=in_shapes[1])
+        )
+        # If we allow either bound to be a scalar `nan`, the test will fail -
+        # so we just "assume" that away (if it is, this raises a special
+        # exception and Hypothesis will try again with different inputs)
+        assume(not np.isscalar(amin) or not np.isnan(amin))
+        assume(not np.isscalar(amax) or not np.isnan(amax))
+
+        # Then calculate our result and expected result and check that they're
+        # equal!  See gh-12519 for discussion deciding on this property.
+        result = np.clip(arr, amin, amax)
+        expected = np.minimum(amax, np.maximum(arr, amin))
+        assert_array_equal(result, expected)
+
 
 class TestAllclose:
     rtol = 1e-5
diff --git a/test_requirements.txt b/test_requirements.txt
--- a/test_requirements.txt
+++ b/test_requirements.txt
@@ -1,4 +1,5 @@
 cython==0.29.14
+hypothesis==5.3.0
 pytest==5.3.5
 pytz==2019.3
 pytest-cov==2.8.1