internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: unionai-oss/pandera

Issue:
MultiIndex with a str dtype schemas can produce invalid examples
Problem:

If you make a schema for a MultiIndex dataframe and some columns are `str`s but others are not, the example produced could have incorrect dtypes for the non-`str` indices and the resulting dataframe from `schema.example()` will not be valid according to the schema.

- [x] I have checked that this issue has not already been reported.
- [x] I have confirmed this bug exists on the latest version of pandera.
- [x] (optional) I have confirmed this bug exists on the master branch of pandera.

```python
import pandera as pa

# Make example schema
schema = pa.DataFrameSchema(
                   index=pa.MultiIndex([
                       pa.Index(str, name="index1"),
                       pa.Index(float, name="index2"),
                       pa.Index(float, name="index3"),
                   ]
                   ))
example = schema.example()
schema.validate(example)
```

#### Expected behavior
There should not be an error thrown, but instead the above code results in: `pandera.errors.SchemaError: expected series 'index3' to have type float64, got object`.



Discussion:
This can be fixed with a one line change to [lambda df: df.assign(**{name: df[name].map(str)})](https://github.com/unionai-oss/pandera/blob/5aa77957615720741dd817b668154e9a72cf6c73/pandera/strategies.py#L1195)

Change that line to:
`lambda df, name=name: df.assign(**{name: df[name].map(str)})`

Resolving patch:
diff --git a/pandera/strategies/pandas_strategies.py b/pandera/strategies/pandas_strategies.py
--- a/pandera/strategies/pandas_strategies.py
+++ b/pandera/strategies/pandas_strategies.py
@@ -1192,7 +1192,7 @@ def multiindex_strategy(
         if dtype in {"object", "str"} or dtype.startswith("string"):
             # pylint: disable=cell-var-from-loop,undefined-loop-variable
             strategy = strategy.map(
-                lambda df: df.assign(**{name: df[name].map(str)})
+                lambda df, name=name: df.assign(**{name: df[name].map(str)})
             )
 
     if any(nullable_index.values()):


Test patch:
Repository: unionai-oss/pandera

Issue:
Remove incorrect(?) warning for `register_check_method()` in docs
#### Location of the documentation

https://pandera.readthedocs.io/en/latest/reference/generated/pandera.extensions.html

#### Documentation problem

It's documented for `register_check_method()` that

> **Warning**
> This is the legacy method for registering check methods. Use the `register_check()` decorator instead.

I can't see any reference to `register_check()` in the docs or the repo, so I assume this is an out-dated warning and `register_check_method()` is infact the defacto function for this kind of stuff. 

Might I be missing something? Maybe this warning is supposed to refer to another function.

#### Suggested fix for documentation

Remove warning as it's seemingly not warranted (at least in this moment).


Discussion:

Resolving patch:
diff --git a/pandera/api/extensions.py b/pandera/api/extensions.py
--- a/pandera/api/extensions.py
+++ b/pandera/api/extensions.py
@@ -144,11 +144,6 @@ def register_check_method(
 
     See the :ref:`user guide<extensions>` for more details.
 
-    .. warning::
-
-        This is the legacy method for registering check methods. Use the
-        :py:func:`~pandera.api.extensions.register_check` decorator instead.
-
     :param check_fn: check function to register. The function should take one
         positional argument for the object to validate and additional
         keyword-only arguments for the check statistics.


Test patch:
Repository: unionai-oss/pandera

Issue:
Why python_requires <3.12?
In https://github.com/unionai-oss/pandera/commit/547aff1672fe455741f380c8bec1ed648074effc, `python_requires` was changed from `>=3.7` to `>=3.7,<=3.11`, and in a later commit, the upper bound was again changed to `<3.12`. This forces every downstream package or application to lower the upper bound from the typical default <4.0, which is unfortunate.

For example, with poetry, using the default `python = "^3.x"` version specification, pandera is now downgraded, or if one tries to force a newer version, version resolution fails:

```
> poetry update pandera

  • Updating pandera (0.15.1 -> 0.14.5)
```

```
> poetry add pandera@0.15.1

The current project's Python requirement (>=3.9,<4.0) is not compatible with some of the required packages Python requirement:
  - pandera requires Python >=3.7,<3.12, so it will not be satisfied for Python >=3.12,<4.0

Because my_package depends on pandera (0.15.1) which requires Python >=3.7,<3.12, version solving failed.
```

Is there a known issue with pandera on python 3.12? Otherwise, I recommend removing the constraint. While pandera might not be tested on 3.12 yet, it's common to assume the language will be backwards compatible as described in [PEP 387](https://peps.python.org/pep-0387/).


Discussion:
Hi @danhje, would you mind making a PR to remove the constraint? The motivation for adding an upper bound was to more clearly enforce what exact versions are tested. Wasn't aware of the knock-on effects of placing the upper bound.
> Hi @danhje, would you mind making a PR to remove the constraint? The motivation for adding an upper bound was to more clearly enforce what exact versions are tested. Wasn't aware of the knock-on effects of placing the upper bound.

I will. Hopefully tomorrow 😊

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -56,7 +56,7 @@
         "wrapt",
     ],
     extras_require=extras_require,
-    python_requires=">=3.7,<3.12",
+    python_requires=">=3.7",
     platforms="any",
     classifiers=[
         "Development Status :: 5 - Production/Stable",


Test patch:
Repository: unionai-oss/pandera

Issue:
Use of hypothesis type annotations forces downstream projects to install hypothesis
**Describe the bug**
A clear and concise description of what the bug is.

- [x] I have checked that this issue has not already been reported.
- [ ] I have confirmed this bug exists on the latest version of pandera.
- [x] (optional) I have confirmed this bug exists on the master branch of pandera.

**Note**: Please read [this guide](https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports) detailing how to provide the necessary information for us to reproduce your bug.

#### Code Sample, a copy-pastable example

In 1fc05dea a type annotation `SearchStrategy[pd.Series]` was added to `series_strategy`
(pandera/strategies/pandas_strategies.py). This forces downstream projects to install
hypothesis as the place holder for `SearchStrategy` defined in the same file does not
contain definition of `__getitem__`.

``` python
import pandera as pa
```

Importing pandera in a downstream project where hypothesis is not available will give the following error 

``` python-traceback
../pandera/pandera/__init__.py:5: in <module>
    from pandera.accessors import pandas_accessor
../pandera/pandera/accessors/pandas_accessor.py:7: in <module>
    from pandera.api.pandas.array import SeriesSchema
../pandera/pandera/api/pandas/array.py:10: in <module>
    from pandera import strategies as st
../pandera/pandera/strategies/__init__.py:3: in <module>
    from pandera.strategies.pandas_strategies import *
../pandera/pandera/strategies/pandas_strategies.py:830: in <module>
    ) -> SearchStrategy[pd.Series]:
E   TypeError: type 'SearchStrategy' is not subscriptable
```

If hypothesis is available this does not happen.

#### Expected behavior

It should be possible to use pandera without installing hypothesis.


Discussion:

Resolving patch:
diff --git a/pandera/strategies/pandas_strategies.py b/pandera/strategies/pandas_strategies.py
--- a/pandera/strategies/pandas_strategies.py
+++ b/pandera/strategies/pandas_strategies.py
@@ -20,6 +20,7 @@
     Any,
     Callable,
     Dict,
+    Generic,
     List,
     Optional,
     Sequence,
@@ -51,8 +52,10 @@
     from hypothesis.strategies import SearchStrategy, composite
 except ImportError:  # pragma: no cover
 
+    T = TypeVar("T")
+
     # pylint: disable=too-few-public-methods
-    class SearchStrategy:  # type: ignore
+    class SearchStrategy(Generic[T]):  # type: ignore
         """placeholder type."""
 
     def composite(fn):  # type: ignore


Test patch:
Repository: unionai-oss/pandera

Issue:
Fix to_script description
#### Location of the documentation

[DataFrameSchema.to_script](https://github.com/unionai-oss/pandera/blob/62bc4840508ff1ac0df595b57b2152737a1228a2/pandera/api/pandas/container.py#L1251)

#### Documentation problem

This method has the description for `from_yaml`.

#### Suggested fix for documentation

Something like "Write `DataFrameSchema` to script".


Discussion:
please feel free to create a PR fix for this!

Resolving patch:
diff --git a/pandera/api/pandas/container.py b/pandera/api/pandas/container.py
--- a/pandera/api/pandas/container.py
+++ b/pandera/api/pandas/container.py
@@ -1249,7 +1249,7 @@ def reset_index(
     #####################
 
     def to_script(self, fp: Union[str, Path] = None) -> "DataFrameSchema":
-        """Create DataFrameSchema from yaml file.
+        """Write DataFrameSchema to python script.
 
         :param path: str, Path to write script
         :returns: dataframe schema.


Test patch:
Repository: unionai-oss/pandera

Issue:
Add documentation for type coercion behavior
#### Question about pandera

Is there a place in the docs that explains the difference between Coerce at column level vs coerce at schema level?

```python
class Schema(pa.SchemaModel):

    year: Series[int] = pa.Field(gt=2000, coerce=True)
    month: Series[int] = pa.Field(ge=1, le=12, coerce=True)
    day: Series[int] = pa.Field(ge=0, le=365, coerce=True)

    class Config:
        name = "BaseSchema"
        strict = True
        coerce = True
        foo = "bar"  # Interpreted as dataframe check
```



Discussion:
hey @bfmcneill the current behavior is that `DataFrameSchema(coerce=True)` will override the coerce setting at the column/index level.

This is not currently explicitly documented anywhere. It would probably belong [here](https://pandera.readthedocs.io/en/stable/dataframe_schemas.html#coercing-types-on-columns) right after

> If you want to coerce all of the columns specified in the DataFrameSchema, you can specify the coerce argument with DataFrameSchema(..., coerce=True).

And also in the docstring [here](https://github.com/pandera-dev/pandera/blob/v0.6.4/pandera/schemas.py#L93-L95)

A contribution on that front would be much appreciated, let me know if you want to contribute!

Resolving patch:
diff --git a/pandera/api/pandas/container.py b/pandera/api/pandas/container.py
--- a/pandera/api/pandas/container.py
+++ b/pandera/api/pandas/container.py
@@ -65,8 +65,9 @@ def __init__(
             then assumes one of the valid pandas string values:
             http://pandas.pydata.org/pandas-docs/stable/basics.html#dtypes.
         :param coerce: whether or not to coerce all of the columns on
-            validation. This has no effect on columns where
-            ``dtype=None``
+            validation. This overrides any coerce setting at the column
+            or index level. This has no effect on columns where
+            ``dtype=None``.
         :param strict: ensure that all and only the columns defined in the
             schema are present in the dataframe. If set to 'filter',
             only the columns in the schema will be passed to the validated


Test patch:
Repository: unionai-oss/pandera

Issue:
Date type not exported
**Describe the bug**
In the `__all__` list [here](https://github.com/unionai-oss/pandera/blob/37c24d94ae719dcf4cdc36d1f204478539fce74a/pandera/__init__.py#L104-L106), the type `Date` is missing, causing complaints from mypy if you refer to the type as `pa.Date` -- you have to fully qualify it as `pa.typing.common.Date`.

- [x] I have checked that this issue has not already been reported.
- [x] I have confirmed this bug exists on the latest version of pandera.
- [x] (optional) I have confirmed this bug exists on the master branch of pandera.

**Note**: Please read [this guide](https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports) detailing how to provide the necessary information for us to reproduce your bug.

#### Code Sample, a copy-pastable example

```python
import pandera as pa

# Mypy errors [name-defined]
class ErrorSchema(pa.DataFrameModel):
    date_col: pa.Date

# Mypy is happy
class NoErrorSchema(pa.DataFrameModel):
    date_col: pa.typing.common.Date
```

#### Expected behavior
No errors from mypy in both cases

#### Desktop (please complete the following information):

 - OS: [Manjaro Linux kernel 6.1.60 - 1]
 - Browser [Firefox 119.0]
 - Version [pandera 0.17.2]


Discussion:

Resolving patch:
diff --git a/pandera/__init__.py b/pandera/__init__.py
--- a/pandera/__init__.py
+++ b/pandera/__init__.py
@@ -101,6 +101,7 @@
     "Complex64",
     "Complex128",
     "Complex256",
+    "Date",
     "DataType",
     "DateTime",
     "Float",


Test patch:
Repository: unionai-oss/pandera

Issue:
Pandas backend: `check_nullable` is inefficient when a column schema has `nullable=True`
`check_nullable` in the pandas backend seems to compute the null values mask by calling `isna()` even when not needed.

https://github.com/unionai-oss/pandera/blob/9c484a92cc6e63ba11652444e9e6df9e587d668e/pandera/backends/pandas/array.py#L196C9-L198

```
isna = check_obj.isna()
passed = schema.nullable or not isna.any()
```

We see that even if `schema.nullable=True`, `isna` is already computed. This can lead to a performance issue in dataframe with millions of rows.






Discussion:
@smarie feel free to make a PR!

Resolving patch:
diff --git a/pandera/backends/pandas/array.py b/pandera/backends/pandas/array.py
--- a/pandera/backends/pandas/array.py
+++ b/pandera/backends/pandas/array.py
@@ -194,8 +194,16 @@ def check_name(self, check_obj: pd.Series, schema) -> CoreCheckResult:
 
     @validate_scope(scope=ValidationScope.SCHEMA)
     def check_nullable(self, check_obj: pd.Series, schema) -> CoreCheckResult:
+        if schema.nullable:
+            # Avoid to compute anything for perf reasons. GH#1533
+            return CoreCheckResult(
+                passed=True,
+                check="not_nullable",
+            )
+
+        # Check actual column contents
         isna = check_obj.isna()
-        passed = schema.nullable or not isna.any()
+        passed = not isna.any()
         return CoreCheckResult(
             passed=cast(bool, passed),
             check="not_nullable",


Test patch:
Repository: unionai-oss/pandera

Issue:
Error Importing Pandera with Polars extra
**Describe the bug**
I get an error when importing pandera after installing the latest 0.19.0b2 version with the polars extra in a clean environment. I can import it successfully if I install without the polars extra.

- [x] I have checked that this issue has not already been reported.
- [x] I have confirmed this bug exists on the latest version of pandera.
- [ ] (optional) I have confirmed this bug exists on the main branch of pandera.

#### Code Sample, a copy-pastable example

I installed pandera 0.19.0b2 in a clean virtual environment using `pip install pandera[polars]==0.19.0b2` and attempted to import pandera:

```python
import pandera as pa
```

I got the following error message:
```
>>> import pandera as pa
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".venv/lib/python3.11/site-packages/pandera/__init__.py", line 6, in <module>
    from pandera import errors, external_config, typing
  File ".venv/lib/python3.11/site-packages/pandera/external_config.py", line 23, in <module>
    import pyspark.pandas
ModuleNotFoundError: No module named 'pyspark'
```

#### Versions:

 - Pandera: 0.19.0b2
 - Python: 3.11.7
 - Ubuntu: 22.04



Discussion:

Resolving patch:
diff --git a/pandera/external_config.py b/pandera/external_config.py
--- a/pandera/external_config.py
+++ b/pandera/external_config.py
@@ -21,6 +21,8 @@
         os.environ["PYARROW_IGNORE_TIMEZONE"] = "1"
 
     import pyspark.pandas
+except (ImportError, ModuleNotFoundError):
+    pass
 finally:
     if is_spark_local_ip_dirty:
         os.environ.pop("SPARK_LOCAL_IP")


Test patch:
Repository: castorini/pyserini

Issue:
Strange terms() behavior in IndexReaderUtils
I do this:

```
from pyserini.index import pyutils

index_utils = pyutils.IndexReaderUtils('lucene-index.robust04.pos+docvectors+rawdocs')

iter1 = index_utils.terms()
iter2 = index_utils.terms()

for term in iter1:
    print('{} (df={}, cf={})'.format(term.term, term.doc_freq, term.total_term_freq))
```

As expected, I iterate over all terms. Then:

```
for term in iter2:
    print('{} (df={}, cf={})'.format(term.term, term.doc_freq, term.total_term_freq))
```

Gives nothing... is `terms()` returning the same iterator every time?

Doesn't seem like the right behavior?


Discussion:

Resolving patch:
diff --git a/pyserini/index/pyutils.py b/pyserini/index/pyutils.py
--- a/pyserini/index/pyutils.py
+++ b/pyserini/index/pyutils.py
@@ -37,7 +37,6 @@ class IndexReaderUtils:
     def __init__(self, index_dir):
         self.object = JIndexReaderUtils()
         self.reader = self.object.getReader(JString(index_dir))
-        self.term_iterator = self.object.getTerms(self.reader)
 
     class DocumentVectorWeight:
         NONE = JDocumentVectorWeight.NONE
@@ -83,8 +82,9 @@ def terms(self):
         '''
         :return: generator over terms
         '''
-        while self.term_iterator.hasNext():
-            cur_term = self.term_iterator.next()
+        term_iterator = self.object.getTerms(self.reader)
+        while term_iterator.hasNext():
+            cur_term = term_iterator.next()
             yield self.IndexTerm(cur_term.getTerm(), cur_term.getDF(), cur_term.getTotalTF())
 
     def get_term_counts(self, term):
@@ -157,4 +157,4 @@ def dump_document_vectors(self, reqDocidsPath, weight):
         weight : DocumentVectorWeight
             the weight for dumped document vector(s)
         '''
-        self.object.dumpDocumentVectors(self.reader, reqDocidsPath, weight)
+        self.object.dumpDocumentVectors(self.reader, reqDocidsPath, weight)
\ No newline at end of file


Test patch:
Repository: castorini/pyserini

Issue:
Change WapoGenerator to WashingtonPostGenerator
This would pyserini more consistent with the naming of Anserini


Discussion:

Resolving patch:
diff --git a/pyserini/index/_base.py b/pyserini/index/_base.py
--- a/pyserini/index/_base.py
+++ b/pyserini/index/_base.py
@@ -55,7 +55,7 @@ class JGenerators(Enum):
     AclAnthologyGenerator = autoclass('io.anserini.index.generator.AclAnthologyGenerator')
     DefaultLuceneDocumentGenerator = autoclass('io.anserini.index.generator.DefaultLuceneDocumentGenerator')
     TweetGenerator = autoclass('io.anserini.index.generator.TweetGenerator')
-    WapoGenerator = autoclass('io.anserini.index.generator.WashingtonPostGenerator')
+    WashingtonPostGenerator = autoclass('io.anserini.index.generator.WashingtonPostGenerator')
 
 
 class Generator:


Test patch:
Repository: castorini/pyserini

Issue:
Add ability to select random question for interactive demo
hey @saileshnankani - how about we add a `/random` command to ask a random question from the dev set?


Discussion:
Sounds good. I can work on that next.

Resolving patch:
diff --git a/pyserini/demo/msmarco.py b/pyserini/demo/msmarco.py
--- a/pyserini/demo/msmarco.py
+++ b/pyserini/demo/msmarco.py
@@ -16,13 +16,18 @@
 
 import cmd
 import json
+import os
+import random
 
 from pyserini.search import SimpleSearcher
 from pyserini.dsearch import SimpleDenseSearcher, TctColBertQueryEncoder, AnceQueryEncoder
 from pyserini.hsearch import HybridSearcher
+from pyserini import search
 
 
 class MsMarcoDemo(cmd.Cmd):
+    dev_topics = list(search.get_topics('msmarco-passage-dev-subset').values())
+
     ssearcher = SimpleSearcher.from_prebuilt_index('msmarco-passage')
     dsearcher = None
     hsearcher = None
@@ -42,6 +47,7 @@ def do_help(self, arg):
         print(f'/k [NUM] : sets k (number of hits to return) to [NUM]')
         print(f'/model [MODEL] : sets encoder to use the model [MODEL] (one of tct, ance)')
         print(f'/mode [MODE] : sets retriver type to [MODE] (one of sparse, dense, hybrid)')
+        print(f'/random : returns results for a random question from dev subset')
 
     def do_k(self, arg):
         print(f'setting k = {int(arg)}')
@@ -85,6 +91,11 @@ def do_model(self, arg):
         self.hsearcher = HybridSearcher(self.dsearcher, self.ssearcher)
         print(f'setting model = {arg}')
 
+    def do_random(self, arg):
+        q = random.choice(self.dev_topics)['title']
+        print(f'question: {q}')
+        self.default(q)
+
     def do_EOF(self, line):
         return True
 


Test patch:
Repository: castorini/pyserini

Issue:
Switch to jnius_config.add_classpath
Currently, pyserini replaces any previously registered jars on the classpath in its setup code. Is there any reason to not use add_classpath() instead of set_classpath()?

Here is the pyjnius relevant code:
```python
def set_classpath(*path):
    """
    Sets the classpath for the JVM to use. Replaces any existing classpath, overriding the CLASSPATH environment variable.
    """
    check_vm_running()
    global classpath
    classpath = list(path)


def add_classpath(*path):
    """
    Appends items to the classpath for the JVM to use.
    Replaces any existing classpath, overriding the CLASSPATH environment variable.
    """
    check_vm_running()
    global classpath
    if classpath is None:
        classpath = list(path)
    else:
        classpath.extend(path)
```


Discussion:
Hi @cash good idea! Can you send PR please?

Resolving patch:
diff --git a/pyserini/setup.py b/pyserini/setup.py
--- a/pyserini/setup.py
+++ b/pyserini/setup.py
@@ -37,4 +37,4 @@ def configure_classpath(anserini_root="."):
         raise Exception('No matching jar file found in {}'.format(os.path.abspath(anserini_root)))
 
     latest = max(paths, key=os.path.getctime)
-    jnius_config.set_classpath(latest)
+    jnius_config.add_classpath(latest)


Test patch: