Repository: pandas-dev/pandas

Issue:
Investigate using Hypothesis for some tests
Hypothesis recently added support for generating pandas objects: http://hypothesis.readthedocs.io/en/latest/numpy.html#pandas

This could be useful for testing some of the higher-level methods like `melt`, `merge`, etc.

Are there any objections to adding some hypothesis-based tests to our test suite? Is anyone familiar with hypothesis and interested in putting together examples of how we could use it?


Discussion:
As another example, I could see Hypothesis being *very* useful for something like https://github.com/pandas-dev/pandas/issues/13432 (buggy index set ops for duplicates, non monotonic, categorical)
This sounds like it's definitely worth trying! Is there a guarantee that the test cases it produces will always be the same? (Non-reproducible tests could be pretty painful.)
I believe that once a failing test case is found it's cached (somehow, not
sure on the details.)

On Wed, Oct 25, 2017 at 10:49 AM, Stephan Hoyer <notifications@github.com>
wrote:

> This sounds like it's definitely worth trying! Is there a guarantee that
> the test cases it produces will always be the same? (Non-reproducible tests
> could be pretty painful.)
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/issues/17978#issuecomment-339376521>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIvYHXq8I4yFc8irTLznyt0jUidU7ks5sv1h7gaJpZM4QF4u7>
> .
>

A test database is kept in a .hypothesis directory.  Not sure if that plays nicely with CI.
@TomAugspurger did this idea ever go anywhere?  I'm thinking this might be a useful for testing offsets.
Don't think so, go for it!
Hi, Can I work on this issue. I have recently implemented some unit test cases using hypothesis and therefore have lil bit of exp which I think can be used here.
Infact I have zeroed in some files, where it can be used easily like pandas/tests/tools/test_numeric.py, e.g

![image](https://user-images.githubusercontent.com/31987769/37984121-0dc4695e-3213-11e8-973b-6834195bffd2.png)

That seems reasonable. I think the main thing is ensuring this doesn't
increase our test times too much.

On Tue, Mar 27, 2018 at 12:33 PM, sushobhit27 <notifications@github.com>
wrote:

> Hi, Can I work on this issue. I have recently implemented some unit test
> cases using hypothesis and therefore have lil bit of exp which I think can
> be used here.
> Infact I have zeroed in some files, where it can be used easily like
> pandas/tests/tools/test_numeric.py, e.g
>
> [image: image]
> <https://user-images.githubusercontent.com/31987769/37984121-0dc4695e-3213-11e8-973b-6834195bffd2.png>
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/issues/17978#issuecomment-376609978>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIh1OvooYADMoaghVQl0p3DGKp1iqks5tinf9gaJpZM4QF4u7>
> .
>

Hypothesis test run time depends on mainly 3 things:
number of examples to run for a test case. (is configurable per test case)
range of the test data like in aforementioned example (bounded by min_size and max_size, therefore this is also configurable)
how internally hypothesis generates different datasets from the range (called shrinking mechanism AFAIK).
We can play with first two options to have some influence on test run time, however it will surely gonna take more time than current pytest test cases as here test cases are constant.
@TomAugspurger, I created a pull request https://github.com/pandas-dev/pandas/pull/20590 for this issue by adding hypothesis based test cases in just one file. However travis CI build failed with error "ImportError: No module named 'hypothesis'". Could you please check how it (hypothesis module) can be available, so that tests can be run.
Hi all - I'm a maintainer of Hypothesis and happy user of Pandas, so hopefully I can help with this :smile: 

* Hypothesis is *not* deterministic by default - though you can derandomize it, we don't recommend this.
* For obvious reasons though, we put a *lot* of effort into ensuring that [test failures are always reproducible](https://hypothesis.readthedocs.io/en/latest/reproducing.html).  Short version: we try to print a call you can copy-paste to get the error plus the seed value if necessary.  If it looks like that might fail, we have a decorator to force everything to the right internal state and print that code with the base64 data it needs to reproduce everything.
* Failing examples are also persisted to disk, so the test will fail immediately if you rerun the test suite before fixing the bug no matter how unlikely it was to find.
* Test runtime is a reasonable concern; there's not usually much overhead but running a test tens or hundreds of times is inherently slower than running it once!  The usual solution is some combination of "don't use it for everything", "turn down the `max_examples` setting in CI", and "just deal with it".  
* All hypothesis tests automatically apply a pytest marker to themselves too, so you can include or exclude them with `-m hypothesis` or `-m "not hypothesis"`.  So it would be easy to only run them with the slow tests, for example.

And my question to Pandas maintainers: what's the best way to add this as a dependency?  Hypothesis can be installed from PyPI or from conda-forge, so it's down to whatever would work best in your CI.
Thanks @Zac-HD , we have https://github.com/pandas-dev/pandas/pull/20590 that's seemingly ready to go. Need to find the time to take a look again.
Actually, @Zac-HD would you mind taking a look in https://github.com/pandas-dev/pandas/pull/20590? One of the issues we're facing is that none of the pandas maintainers have really used hypothesis before. We're trying to define some best practices in that PR, since others will mimic it. You may be the ideal person to set those standards for pandas :)
I have three high-level concerns about #20590:

1. Configuration can be centralised in `conftest.py`, which makes it easier to edit or just override with a command-line argument than using explicit decorators everywhere.

2. I would ditch the `util/_hypothesis.py` module, and just the Hypothesis api directly.  We've put a lot of thought into making it clear, flexible, and orthogonal - and it has much better performance and error reporting than anything I know of downstream.  If there's anything Pandas-specific that would be really helpful, we would be happy to add it to `hypothesis.extra.pandas` upstream so everyone can use it!

3. Pandas would probably get much more value out of Hypothesis by using it for integration-style tests, i.e. for more complex invariants where there are more interacting parts.  In Hypothesis' own test suite we still use traditional unit tests (often with `pytest.mark.parametrize`) to check input validation, because it's just not worth the runtime.
   #18761 has some interesting stuff for that, though it too could be much simpler with more idiomatic use of Hypothesis!

So rather than tread on toes by suggesting detailed edits to those PRs, I might open a *third* one this weekend incorporating some of both and some of my own notes.

Resolving patch:
diff --git a/ci/check_imports.py b/ci/check_imports.py
--- a/ci/check_imports.py
+++ b/ci/check_imports.py
@@ -9,6 +9,7 @@
     'html5lib',
     'ipython',
     'jinja2'
+    'hypothesis',
     'lxml',
     'numexpr',
     'openpyxl',
diff --git a/pandas/conftest.py b/pandas/conftest.py
--- a/pandas/conftest.py
+++ b/pandas/conftest.py
@@ -450,3 +450,37 @@ def mock():
         return importlib.import_module("unittest.mock")
     else:
         return pytest.importorskip("mock")
+
+
+# ----------------------------------------------------------------
+# Global setup for tests using Hypothesis
+
+from hypothesis import strategies as st
+
+# Registering these strategies makes them globally available via st.from_type,
+# which is use for offsets in tests/tseries/offsets/test_offsets_properties.py
+for name in 'MonthBegin MonthEnd BMonthBegin BMonthEnd'.split():
+    cls = getattr(pd.tseries.offsets, name)
+    st.register_type_strategy(cls, st.builds(
+        cls,
+        n=st.integers(-99, 99),
+        normalize=st.booleans(),
+    ))
+
+for name in 'YearBegin YearEnd BYearBegin BYearEnd'.split():
+    cls = getattr(pd.tseries.offsets, name)
+    st.register_type_strategy(cls, st.builds(
+        cls,
+        n=st.integers(-5, 5),
+        normalize=st.booleans(),
+        month=st.integers(min_value=1, max_value=12),
+    ))
+
+for name in 'QuarterBegin QuarterEnd BQuarterBegin BQuarterEnd'.split():
+    cls = getattr(pd.tseries.offsets, name)
+    st.register_type_strategy(cls, st.builds(
+        cls,
+        n=st.integers(-24, 24),
+        normalize=st.booleans(),
+        startingMonth=st.integers(min_value=1, max_value=12)
+    ))
diff --git a/pandas/util/_tester.py b/pandas/util/_tester.py
--- a/pandas/util/_tester.py
+++ b/pandas/util/_tester.py
@@ -12,6 +12,10 @@ def test(extra_args=None):
         import pytest
     except ImportError:
         raise ImportError("Need pytest>=3.0 to run tests")
+    try:
+        import hypothesis  # noqa
+    except ImportError:
+        raise ImportError("Need hypothesis>=3.58 to run tests")
     cmd = ['--skip-slow', '--skip-network']
     if extra_args:
         if not isinstance(extra_args, list):


Test patch:
diff --git a/pandas/tests/tseries/offsets/test_offsets_properties.py b/pandas/tests/tseries/offsets/test_offsets_properties.py
new file mode 100644
--- /dev/null
+++ b/pandas/tests/tseries/offsets/test_offsets_properties.py
@@ -0,0 +1,104 @@
+# -*- coding: utf-8 -*-
+"""
+Behavioral based tests for offsets and date_range.
+
+This file is adapted from https://github.com/pandas-dev/pandas/pull/18761 -
+which was more ambitious but less idiomatic in its use of Hypothesis.
+
+You may wish to consult the previous version for inspiration on further
+tests, or when trying to pin down the bugs exposed by the tests below.
+"""
+
+import pytest
+from hypothesis import given, assume, strategies as st
+from hypothesis.extra.pytz import timezones as pytz_timezones
+from hypothesis.extra.dateutil import timezones as dateutil_timezones
+
+import pandas as pd
+
+from pandas.tseries.offsets import (
+    MonthEnd, MonthBegin, BMonthEnd, BMonthBegin,
+    QuarterEnd, QuarterBegin, BQuarterEnd, BQuarterBegin,
+    YearEnd, YearBegin, BYearEnd, BYearBegin,
+)
+
+# ----------------------------------------------------------------
+# Helpers for generating random data
+
+gen_date_range = st.builds(
+    pd.date_range,
+    start=st.datetimes(
+        # TODO: Choose the min/max values more systematically
+        min_value=pd.Timestamp(1900, 1, 1).to_pydatetime(),
+        max_value=pd.Timestamp(2100, 1, 1).to_pydatetime()
+    ),
+    periods=st.integers(min_value=2, max_value=100),
+    freq=st.sampled_from('Y Q M D H T s ms us ns'.split()),
+    tz=st.one_of(st.none(), dateutil_timezones(), pytz_timezones()),
+)
+
+gen_random_datetime = st.datetimes(
+    min_value=pd.Timestamp.min.to_pydatetime(),
+    max_value=pd.Timestamp.max.to_pydatetime(),
+    timezones=st.one_of(st.none(), dateutil_timezones(), pytz_timezones())
+)
+
+# The strategy for each type is registered in conftest.py, as they don't carry
+# enough runtime information (e.g. type hints) to infer how to build them.
+gen_yqm_offset = st.one_of(*map(st.from_type, [
+    MonthBegin, MonthEnd, BMonthBegin, BMonthEnd,
+    QuarterBegin, QuarterEnd, BQuarterBegin, BQuarterEnd,
+    YearBegin, YearEnd, BYearBegin, BYearEnd
+]))
+
+
+# ----------------------------------------------------------------
+# Offset-specific behaviour tests
+
+
+# Based on CI runs: Always passes on OSX, fails on Linux, sometimes on Windows
+@pytest.mark.xfail(strict=False, reason='inconsistent between OSs, Pythons')
+@given(gen_random_datetime, gen_yqm_offset)
+def test_on_offset_implementations(dt, offset):
+    assume(not offset.normalize)
+    # check that the class-specific implementations of onOffset match
+    # the general case definition:
+    #   (dt + offset) - offset == dt
+    compare = (dt + offset) - offset
+    assert offset.onOffset(dt) == (compare == dt)
+
+
+@pytest.mark.xfail(strict=True)
+@given(gen_yqm_offset, gen_date_range)
+def test_apply_index_implementations(offset, rng):
+    # offset.apply_index(dti)[i] should match dti[i] + offset
+    assume(offset.n != 0)  # TODO: test for that case separately
+
+    # rng = pd.date_range(start='1/1/2000', periods=100000, freq='T')
+    ser = pd.Series(rng)
+
+    res = rng + offset
+    res_v2 = offset.apply_index(rng)
+    assert (res == res_v2).all()
+
+    assert res[0] == rng[0] + offset
+    assert res[-1] == rng[-1] + offset
+    res2 = ser + offset
+    # apply_index is only for indexes, not series, so no res2_v2
+    assert res2.iloc[0] == ser.iloc[0] + offset
+    assert res2.iloc[-1] == ser.iloc[-1] + offset
+    # TODO: Check randomly assorted entries, not just first/last
+
+
+@pytest.mark.xfail(strict=True)
+@given(gen_yqm_offset)
+def test_shift_across_dst(offset):
+    # GH#18319 check that 1) timezone is correctly normalized and
+    # 2) that hour is not incorrectly changed by this normalization
+    # Note that dti includes a transition across DST boundary
+    dti = pd.date_range(start='2017-10-30 12:00:00', end='2017-11-06',
+                        freq='D', tz='US/Eastern')
+    assert (dti.hour == 12).all()  # we haven't screwed up yet
+
+    res = dti + offset
+    assert (res.hour == 12).all()
diff --git a/pandas/tests/tseries/offsets/test_ticks.py b/pandas/tests/tseries/offsets/test_ticks.py
--- a/pandas/tests/tseries/offsets/test_ticks.py
+++ b/pandas/tests/tseries/offsets/test_ticks.py
@@ -6,6 +6,7 @@
 
 import pytest
 import numpy as np
+from hypothesis import given, assume, example, strategies as st
 
 from pandas import Timedelta, Timestamp
 from pandas.tseries import offsets
@@ -35,6 +36,45 @@ def test_delta_to_tick():
     assert (tick == offsets.Day(3))
 
 
+@pytest.mark.parametrize('cls', tick_classes)
+@example(n=2, m=3)
+@example(n=800, m=300)
+@example(n=1000, m=5)
+@given(n=st.integers(-999, 999), m=st.integers(-999, 999))
+def test_tick_add_sub(cls, n, m):
+    # For all Tick subclasses and all integers n, m, we should have
+    # tick(n) + tick(m) == tick(n+m)
+    # tick(n) - tick(m) == tick(n-m)
+    left = cls(n)
+    right = cls(m)
+    expected = cls(n + m)
+
+    assert left + right == expected
+    assert left.apply(right) == expected
+
+    expected = cls(n - m)
+    assert left - right == expected
+
+
+@pytest.mark.parametrize('cls', tick_classes)
+@example(n=2, m=3)
+@given(n=st.integers(-999, 999), m=st.integers(-999, 999))
+def test_tick_equality(cls, n, m):
+    assume(m != n)
+    # tick == tock iff tick.n == tock.n
+    left = cls(n)
+    right = cls(m)
+    assert left != right
+    assert not (left == right)
+
+    right = cls(n)
+    assert left == right
+    assert not (left != right)
+
+    if n != 0:
+        assert cls(n) != cls(-n)
+
+
 # ---------------------------------------------------------------------
 
 
@@ -200,21 +240,8 @@ def test_tick_zero(cls1, cls2):
 
 @pytest.mark.parametrize('cls', tick_classes)
 def test_tick_equalities(cls):
-    assert cls(3) == cls(3)
     assert cls() == cls(1)
 
-    # not equals
-    assert cls(3) != cls(2)
-    assert cls(3) != cls(-3)
-
-
-@pytest.mark.parametrize('cls', tick_classes)
-def test_tick_operators(cls):
-    assert cls(3) + cls(2) == cls(5)
-    assert cls(3) - cls(2) == cls(1)
-    assert cls(800) + cls(300) == cls(1100)
-    assert cls(1000) - cls(5) == cls(995)
-
 
 @pytest.mark.parametrize('cls', tick_classes)
 def test_tick_offset(cls):
@@ -226,11 +253,9 @@ def test_compare_ticks(cls):
     three = cls(3)
     four = cls(4)
 
-    # TODO: WTF?  What is this range(10) supposed to do?
-    for _ in range(10):
-        assert three < cls(4)
-        assert cls(3) < four
-        assert four > cls(3)
-        assert cls(4) > three
-        assert cls(3) == cls(3)
-        assert cls(3) != cls(4)
+    assert three < cls(4)
+    assert cls(3) < four
+    assert four > cls(3)
+    assert cls(4) > three
+    assert cls(3) == cls(3)
+    assert cls(3) != cls(4)