TST: add hypothesis-based tests
State: closed

Addition of "hypothesis usage" in test cases of
tests/reshape/test_util.py as kind of POC.

- [x] closes #17978
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Hello @sushobhit27! Thanks for updating the PR.



 - In the file [`pandas/util/_hypothesis.py`](https://github.com/pandas-dev/pandas/blob/fa5bd75ba9306616030cf47f074fabb642d8f102/pandas/util/_hypothesis.py), following are the PEP8 issues :

> [Line 12:32](https://github.com/pandas-dev/pandas/blob/fa5bd75ba9306616030cf47f074fabb642d8f102/pandas/util/_hypothesis.py#L12): [E261](https://duckduckgo.com/?q=pep8%20E261) at least two spaces before inline comment



##### Comment last updated on May 22, 2018 at 04:09 Hours UTC
> Could you please check how it (hypothesis module) can be available, so that tests can be run.

You'll need to add hypothesis to one (or all) of the `ci/*.run` files. Also add it to `ci/environment-dev.yml`.
@TomAugspurger Now all test have passed. Can you please check what else is required at my end.
test are failing in this
To be clear, I wasn't suggesting a change. I haven't used hypothesis much.
It'd be good to establish best practices since this will be new to
pandas contributors.

On Tue, Apr 3, 2018 at 10:51 AM, sushobhit27 <notifications@github.com>
wrote:

> *@sushobhit27* commented on this pull request.
> ------------------------------
>
> In pandas/tests/reshape/test_util.py
> <https://github.com/pandas-dev/pandas/pull/20590#discussion_r178873106>:
>
> >      def test_datetimeindex(self):
>          # regression test for GitHub issue #6439
>          # make sure that the ordering on datetimeindex is consistent
> -        x = date_range('2000-01-01', periods=2)
> +        d = st.dates(min_value=date(1900, 1, 1), max_value=date(2100, 1, 1)).example()
>
> no specific reason but copy pasted after testing on console, as example()
> function can be easily used to check generated value :) and ya it can be
> moved to "given" decorator.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/20590#discussion_r178873106>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIgBoCofe-vdd4O7iDpVpbeUiDZ4yks5tk5pogaJpZM4TEidT>
> .
>

@TomAugspurger, 
Main purpose of using hypothesis or property based testing is to just write test cases based on the property/specification of the functionality being tested and let the framework generate all the different and random test cases for it. 
This will initially make the test cases fail but due to which we are able to find edge cases and then keep on refining boundary for input data of our test cases. e.g in below example initially test case failed when x was a list with empty string, thus in a way helped me to figure out that this is a case for test_empty function (due to which I adeed assume(len(x) != 0) statement). 

    def test_simple(self):
        x, y = list('ABC'), [1, 22]
        result1, result2 = cartesian_product([x, y])
        expected1 = np.array(['A', 'A', 'B', 'B', 'C', 'C'])
        expected2 = np.array([1, 22, 1, 22, 1, 22])
        tm.assert_numpy_array_equal(result1, expected1)
        tm.assert_numpy_array_equal(result2, expected2)
		

	@settings(max_examples=NO_OF_EXAMPLES_PER_TEST_CASE)
    @given(get_seq((str,), False, 1, 1),
           get_seq((int,), False, 1, 2))
    def test_simple(self, x, y):
        x = list(x[0])
        # non-empty test case is handled in test_empty, therefore ignore it here
        assume(len(x) != 0)
        result1, result2 = cartesian_product([x, y])
        expected1 = np.array([item1 for item1 in x for item2 in y])
        expected2 = np.array([item2 for item1 in x for item2 in y])

        tm.assert_numpy_array_equal(result1, expected1)
        tm.assert_numpy_array_equal(result2, expected2)
		

In my personal experience, I have found it to be most helpful in reducing huge parametized test cases where many examples are overlapping or can be tested by a single hypothesis statement.
		
In very layman terms, take another example of testing python's sum function using hypothesis by just a single statement.
from hypothesis import strategies as st
from hypothesis import given

@given(st.lists(st.integers()))
def test_sum(seq):
    total = 0
    for item in seq:
        total += item
    assert sum(seq) == total
    
While, IMO in case of example based testing at least 3-4 example would be required in terms of parametize statement.

Below are some links for property bases testing usage:
https://hypothesis.readthedocs.io/en/latest/quickstart.html
http://blog.jessitron.com/2013/04/property-based-testing-what-is-it.html
https://hypothesis.works/articles/what-is-property-based-testing/
Great, could you summarize appropriately for a new subsection in
contributing.rst? :)

On Tue, Apr 3, 2018 at 11:36 AM, sushobhit27 <notifications@github.com>
wrote:

> @TomAugspurger <https://github.com/TomAugspurger>,
> Main purpose of using hypothesis or property based testing is to just
> write test cases based on the property/specification of the functionality
> being tested and let the framework generate all the different and random
> test cases for it.
> This will initially make the test cases fail but due to which we are able
> to find edge cases and then keep on refining boundary for input data of our
> test cases. e.g in below example initially test case failed when x was a
> list with empty string, thus in a way helped me to figure out that this is
> a case for test_empty function (due to which I adeed assume(len(x) != 0)
> statement).
>
> def test_simple(self):
>     x, y = list('ABC'), [1, 22]
>     result1, result2 = cartesian_product([x, y])
>     expected1 = np.array(['A', 'A', 'B', 'B', 'C', 'C'])
>     expected2 = np.array([1, 22, 1, 22, 1, 22])
>     tm.assert_numpy_array_equal(result1, expected1)
>     tm.assert_numpy_array_equal(result2, expected2)
> 	
>
> @settings(max_examples=NO_OF_EXAMPLES_PER_TEST_CASE)
> @given(get_seq((str,), False, 1, 1),
>        get_seq((int,), False, 1, 2))
> def test_simple(self, x, y):
>     x = list(x[0])
>     # non-empty test case is handled in test_empty, therefore ignore it here
>     assume(len(x) != 0)
>     result1, result2 = cartesian_product([x, y])
>     expected1 = np.array([item1 for item1 in x for item2 in y])
>     expected2 = np.array([item2 for item1 in x for item2 in y])
>
>     tm.assert_numpy_array_equal(result1, expected1)
>     tm.assert_numpy_array_equal(result2, expected2)
>
> In my personal experience, I have found it to be most helpful in reducing
> huge parametized test cases where many examples are overlapping or can be
> tested by a single hypothesis statement.
>
> In very layman terms, take another example of testing python's sum
> function using hypothesis by just a single statement.
> from hypothesis import strategies as st
> from hypothesis import given
>
> @given <https://github.com/given>(st.lists(st.integers()))
> def test_sum(seq):
> total = 0
> for item in seq:
> total += item
> assert sum(seq) == total
>
> While, IMO in case of example based testing at least 3-4 example would be
> required in terms of parametize statement.
>
> Below are some links for property bases testing usage:
> https://hypothesis.readthedocs.io/en/latest/quickstart.html
> http://blog.jessitron.com/2013/04/property-based-testing-what-is-it.html
> https://hypothesis.works/articles/what-is-property-based-testing/
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/20590#issuecomment-378314248>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIo5e_FqkZQxJzfMGX5crRKRa50xeks5tk6UagaJpZM4TEidT>
> .
>

@TomAugspurger sure, I will do the same in a day or two in addition to incorporating other review comments.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/20590?src=pr&el=h1) Report
> Merging [#20590](https://codecov.io/gh/pandas-dev/pandas/pull/20590?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/81358e862022cedbac009985abac5135a873dde3?src=pr&el=desc) will **decrease** coverage by `0.04%`.
> The diff coverage is `17.24%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/20590/graphs/tree.svg?height=150&token=eZ4WkYLtcO&width=650&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/20590?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #20590      +/-   ##
==========================================
- Coverage   91.84%   91.79%   -0.05%     
==========================================
  Files         153      154       +1     
  Lines       49502    49534      +32     
==========================================
+ Hits        45463    45471       +8     
- Misses       4039     4063      +24
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.19% <17.24%> (-0.05%)` | :arrow_down: |
| #single | `41.86% <17.24%> (-0.02%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/20590?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/util/\_hypothesis.py](https://codecov.io/gh/pandas-dev/pandas/pull/20590/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvX2h5cG90aGVzaXMucHk=) | `17.24% <17.24%> (ø)` | |
| [pandas/core/indexes/multi.py](https://codecov.io/gh/pandas-dev/pandas/pull/20590/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9tdWx0aS5weQ==) | `94.99% <0%> (-0.09%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/20590/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.22% <0%> (-0.01%)` | :arrow_down: |
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/20590/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `94.12% <0%> (ø)` | :arrow_up: |
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/20590/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `96.68% <0%> (+0.04%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/20590?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/20590?src=pr&el=footer). Last update [81358e8...fa5bd75](https://codecov.io/gh/pandas-dev/pandas/pull/20590?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@TomAugspurger Now all test are passing and I have also refactored the code with review comments. Also added section of hypothesis in contributing.rst.
@jreback can you please elaborate what kind of changes you want.
@jreback , @TomAugspurger 
I know its going on for too long, but can you please have a look at latest changes.
Travis is failing due to other test case, TestClipboard.test_round_trip_valid_encodings to be precise.
@jreback I have incorporated all review comments except "removing hypothesis dependency from most of the files" as I could not really understand your below comment:
"I put them above

ci/install_travis_travis, ci/install_circle and appveyor.yaml"
Again travis test is failing due to some other test case.
@jreback, @TomAugspurger 
Was waiting for feedback, trying luck again :) 
 
@sushobhit27 I indicated where you need to update these files in ``pandas/ci/install_circle``, ``pandas/ci/appeveyor.yml`` and ``pandas/ci/install_travis.sh``
@jreback  I have tried to do changes as per your comments and now have removed hypothesis dependency from all *.run, except from the 3 files which you mentioned, although I am not sure if I did it correctly.
Let me know if it is still wrong.
Note: we made some changes to the CI that will cause some merge conflicts,
though hopefully the new setup is simpler.

You can add hypothesis next to `pytest` in the new
`ci/{appveyor,circle,travis}-*.yaml` files.

On Sun, Apr 22, 2018 at 10:30 AM, Sushobhit <notifications@github.com>
wrote:

> *@sushobhit27* commented on this pull request.
> ------------------------------
>
> In ci/requirements-2.7_COMPAT.run
> <https://github.com/pandas-dev/pandas/pull/20590#discussion_r183240738>:
>
> > @@ -11,4 +11,4 @@ psycopg2
>  pymysql=0.6.0
>  sqlalchemy=0.7.8
>  xlsxwriter=0.5.2
> -jinja2=2.8
>
> I will rebase my branch.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/20590#discussion_r183240738>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIgniHzQ_CiqdT7szj7EjyMLgwiopks5trKIOgaJpZM4TEidT>
> .
>

@TomAugspurger I totally forgot about this PR, now I have rebased branch. You can have a look at it now, as all tests are now passing.
closing in favor of #22280