Repository: unionai-oss/pandera

Issue:
Make Hypothesis strategies more efficient with statistics resolver and reducing use of `.filter()`
Heya!  I'm a Hypothesis core dev, and *stoked* that you've found it useful enough to promote to your users 🥰  I also have some suggestions for how to make things more efficient - the short version is "avoid using `.filter()` wherever possible"; the long version is... well, longer.  And probably a lot of work for all of us 😅 

Filtering is very convenient, but can also cause some performance issues - because it works by rejecting and re-drawing whatever data did not pass the predicate.  This is OK (ish, usually) for scalars, but adds up *really* fast when you're rejecting whole columns or dataframes.  So "try to filter elements, not columns" is the first piece of advice, and one that makes sense to implement immediately.

More generally, in many cases it's possible to define a strategy which *always* passes the check, instead of filtering:

```python
pa.Column(float, checks=[pa.Check.gt(0), pa.Check.le(1)])     # the schema
st.floats().filter(lambda x: x >= 0).filter(lambda x: x < 1)  # the current strategy - it's slow!
st.floats(0, 1, exclude_max=True)                             # the ideal strategy - #nofilter 😉 
```

Now... it's basically fine to say this if you're writing it all by hand, but I agree that it's going to be painful to manage from library code.  That's why we have https://github.com/HypothesisWorks/hypothesis/issues/2701, a plan to automatically rewrite strategies when simple filter functions are applied - I'd propose that we work on that together rather than have everyone implement it separately, to split the work and share the benefits.


Discussion:
You also mention that 

https://github.com/pandera-dev/pandera/blob/637f2bf1745d9053eede2ac15655a03b29210178/pandera/strategies.py#L307-L308

...that's just because nobody has asked for them yet; and if you're working on categorical or object support we'd love to have that upstream rather than Pandera-specific if that would work for you 😄 
thanks @Zac-HD!

> This is OK (ish, usually) for scalars, but adds up really fast when you're rejecting whole columns or dataframes. So "try to filter elements, not columns" is the first piece of advice, and one that makes sense to implement immediately.

Yes, this is something I noticed when building out the strategy wrappers, so currently `pandera` mostly filters elements where it can and only falls back to filtering whole columns/dataframes for user-defined custom checks that haven't been registered with a user-provided strategy via the [extensions API](https://pandera.readthedocs.io/en/stable/extensions.html).

Building off of your suggestion, I think there are a couple of possible approaches on the `pandera` strategy-chaining implementation:

1. use `map` for strategy chaining:
```python
pa.Column(float, checks=[pa.Check.gt(0), pa.Check.le(1)])     # the schema

# currently, the first check is the "base strategy", and its statistics are used to build the first strategy
st.floats(min_value=0).filter(lambda x: x < 1)

# using map instead
st.floats(min_value=0).map(lambda x: x if x <= 1 else 1)
```

2. introduce concept of "statistics resolvers" in the backend to aggregate multiple checks into a single strategy. This would throw an error if the resolver finds incompatible statistics.
```python
pa.Column(float, checks=[pa.Check.gt(0), pa.Check.le(1)])     # the schema

# statistics resolver would collect all relevant constraints
agg_stats = {"min_value": 0, "max_value": 1, ...}

st.floats(agg_stats["min_value"], agg_stats["max_value"], exclude_max=True)

# this should raise an error in the 
pa.Column(float, checks=[pa.Check.gt(0), pa.Check.le(1)])
```

The pro about (1) is ease of implementation: the subsequent strategies in a chain don't necessarily have to know anything about the constraints of the prior strategies. Con would be potential issues around oversampling of values based on the first strategy that don't agree with the second strategy (leading to a lot of `1`s in the above example)

On the other hand, the pro about (2) is it elegantly handles multiple constraints and can catch incompatible sets of checks off the bat. Con would be potentially more complex logic for implementing the "statistics resolver".

> That's why we have HypothesisWorks/hypothesis#2701, a plan to automatically rewrite strategies when simple filter functions are applied - I'd propose that we work on that together rather than have everyone implement it separately, to split the work and share the benefits.

Agreed, this feature would be amazing! Let me know if there's anything we can do on our end to help (or even contribute to the hypothesis codebase 😀)
> 1. use `.map` for strategy chaining:

I wouldn't recommend this, as it skews the distribution *very* badly - you'll tend to end up testing the same forced-to-an-endpoint value essentially every time.

> introduce concept of "statistics resolvers" in the backend to aggregate multiple checks into a single strategy. This would throw an error if the resolver finds incompatible statistics.

This is the "right way to do it", though it can also get complicated - which is why I've suggested that we do it upstream 😉 

> Agreed, this feature [automatic filter rewriting] would be amazing! Let me know if there's anything we can do on our end to help (or even contribute to the hypothesis codebase 😀)

As it happens, there is!  I've just opened https://github.com/HypothesisWorks/hypothesis/pull/2853, and https://github.com/HypothesisWorks/hypothesis/issues/2701#issuecomment-774890624 describes the next steps - if I could "just" refactor the strategies and let you write the filter methods and tests for other integers and floats strategies, that would be so helpful!

After that we can look at string/regex strategies and handling lambdas in addition to `functools.partial(operator.foo, ...)`, or jump directly to ensuring that Pandera can take advantage of the new functionality for numeric filters 😀 
@cosmicBboy - we've just released our first filter rewriting, for `st.integers()`.

If you change the comparison and equality checks to use e.g. `functools.partial(operator.ge, min_value)` instead of a custom `less_than` function, it'll just work.  The downside is of course no keyword arguments or type annotations 😕 
thanks @Zac-HD! will update the minimum `hypothesis` version and make these improvements.

> The downside is of course no keyword arguments or type annotations

Thats okay, currently we're using lambda functions for these right now, so we're not type-annotating anyway
Looks like @Zac-HD followed up with another release! https://github.com/HypothesisWorks/hypothesis/releases/tag/hypothesis-python-6.12.0
Yep!  Support for floats is probably a month or two off, and then `text()`-filtering-by-regex sometime after that.

Unfortunately we'll never be able to "rewrite" length filters (😢), because it's possible that `len` would be redefined in the outer scope and then we'd give the wrong answer.  There are still some useful ideas to explore here, but they're all quite a lot of work and well in the future.

----------

On another note, you might want to [add the `Framework :: Hypothesis` trove classifier](https://hypothesis.readthedocs.io/en/latest/strategies.html#writing-an-extension)?

Resolving patch:
diff --git a/pandera/strategies/pandas_strategies.py b/pandera/strategies/pandas_strategies.py
--- a/pandera/strategies/pandas_strategies.py
+++ b/pandera/strategies/pandas_strategies.py
@@ -48,6 +48,7 @@
     import hypothesis
     import hypothesis.extra.numpy as npst
     import hypothesis.extra.pandas as pdst
+    from hypothesis.internal.filtering import max_len, min_len
     import hypothesis.strategies as st
     from hypothesis.strategies import SearchStrategy, composite
 except ImportError:  # pragma: no cover
@@ -463,7 +464,7 @@ def ne_strategy(
     """
     if strategy is None:
         strategy = pandas_dtype_strategy(pandera_dtype)
-    return strategy.filter(lambda x: x != value)
+    return strategy.filter(partial(operator.ne, value))
 
 
 def gt_strategy(
@@ -486,7 +487,7 @@ def gt_strategy(
             min_value=min_value,
             exclude_min=True if is_float(pandera_dtype) else None,
         )
-    return strategy.filter(lambda x: x > min_value)
+    return strategy.filter(partial(operator.lt, min_value))
 
 
 def ge_strategy(
@@ -509,7 +510,7 @@ def ge_strategy(
             min_value=min_value,
             exclude_min=False if is_float(pandera_dtype) else None,
         )
-    return strategy.filter(lambda x: x >= min_value)
+    return strategy.filter(partial(operator.le, min_value))
 
 
 def lt_strategy(
@@ -532,7 +533,7 @@ def lt_strategy(
             max_value=max_value,
             exclude_max=True if is_float(pandera_dtype) else None,
         )
-    return strategy.filter(lambda x: x < max_value)
+    return strategy.filter(partial(operator.gt, max_value))
 
 
 def le_strategy(
@@ -555,7 +556,7 @@ def le_strategy(
             max_value=max_value,
             exclude_max=False if is_float(pandera_dtype) else None,
         )
-    return strategy.filter(lambda x: x <= max_value)
+    return strategy.filter(partial(operator.ge, max_value))
 
 
 def in_range_strategy(
@@ -586,10 +587,10 @@ def in_range_strategy(
             exclude_min=not include_min,
             exclude_max=not include_max,
         )
-    min_op = operator.ge if include_min else operator.gt
-    max_op = operator.le if include_max else operator.lt
-    return strategy.filter(
-        lambda x: min_op(x, min_value) and max_op(x, max_value)
+    min_op = operator.le if include_min else operator.lt
+    max_op = operator.ge if include_max else operator.gt
+    return strategy.filter(partial(min_op, min_value)).filter(
+        partial(max_op, max_value)
     )
 
 
@@ -651,11 +652,7 @@ def str_matches_strategy(
         return st.from_regex(pattern, fullmatch=True).map(
             to_numpy_dtype(pandera_dtype).type
         )
-
-    def matches(x):
-        return re.match(pattern, x)
-
-    return strategy.filter(matches)
+    return strategy.filter(re.compile(pattern).fullmatch)
 
 
 def str_contains_strategy(
@@ -676,11 +673,7 @@ def str_contains_strategy(
         return st.from_regex(pattern, fullmatch=False).map(
             to_numpy_dtype(pandera_dtype).type
         )
-
-    def contains(x):
-        return re.search(pattern, x)
-
-    return strategy.filter(contains)
+    return strategy.filter(re.compile(pattern).search)
 
 
 def str_startswith_strategy(
@@ -697,12 +690,12 @@ def str_startswith_strategy(
     :param string: string pattern.
     :returns: ``hypothesis`` strategy
     """
+    pattern = rf"\A(?:{string})"
     if strategy is None:
-        return st.from_regex(f"\\A{string}", fullmatch=False).map(
+        return st.from_regex(pattern, fullmatch=False).map(
             to_numpy_dtype(pandera_dtype).type
         )
-
-    return strategy.filter(lambda x: x.startswith(string))
+    return strategy.filter(re.compile(pattern).search)
 
 
 def str_endswith_strategy(
@@ -719,12 +712,12 @@ def str_endswith_strategy(
     :param string: string pattern.
     :returns: ``hypothesis`` strategy
     """
+    pattern = rf"(?:{string})\Z"
     if strategy is None:
-        return st.from_regex(f"{string}\\Z", fullmatch=False).map(
+        return st.from_regex(pattern, fullmatch=False).map(
             to_numpy_dtype(pandera_dtype).type
         )
-
-    return strategy.filter(lambda x: x.endswith(string))
+    return strategy.filter(re.compile(pattern).search)
 
 
 def str_length_strategy(
@@ -747,8 +740,9 @@ def str_length_strategy(
         return st.text(min_size=min_value, max_size=max_value).map(
             to_numpy_dtype(pandera_dtype).type
         )
-
-    return strategy.filter(lambda x: min_value <= len(x) <= max_value)
+    return strategy.filter(partial(min_len, min_value)).filter(
+        partial(max_len, max_value)
+    )
 
 
 def _timestamp_to_datetime64_strategy(
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -8,7 +8,7 @@
     exec(fp.read(), version)
 
 _extras_require = {
-    "strategies": ["hypothesis >= 5.41.1"],
+    "strategies": ["hypothesis >= 6.92.7"],
     "hypotheses": ["scipy"],
     "io": ["pyyaml >= 5.1", "black", "frictionless <= 4.40.8"],
     "pyspark": ["pyspark >= 3.2.0"],


Test patch: