Repository: unionai-oss/pandera

Issue:
Missing failure case in SchemaError when coercing integer-like strings to datetime
**Describe the bug**
There is an issue when validating and coercing to datetime type for integer-like strings, for example `"123"`: the validation raises a `SchemaErrors` as expected, but the index of the errors are not present in the `SchemaErrors.failure_cases`.

This is due to the fact than when coercing the full Series of the column, the function `pandas.to_datetime` is used. If this function fails, then to check which index in particular is failing, all elements are tested using the function `numpy.datetime64`. And `pandas.to_datetime` and `numpy.datetime64` do not have the same behaviour: `numpy.datetime64("123")` will return  `numpy.datetime64('0123')`, whereas `pandas.to_datetime("123")` will raise a `ValueError`. Thus, the coercing of the whole Series fails using  `pandas.to_datetime`, but coercing each element with `numpy.datetime64` does not fail, hence no individual failure cases are found.

EDIT: this is related to the change to `pandera.engines.utils.numpy_pandas_coerce_failure_cases` in version 0.10.0, the example below is working fine for version 0.9.0.

I think that the same function should be used for coercing both the Series and each of its elements.

- [x] I have checked that this issue has not already been reported.
- [x] I have confirmed this bug exists on the latest version of pandera.
- [ ] (optional) I have confirmed this bug exists on the master branch of pandera.


#### Code Sample, a copy-pastable example

```python
import pandera
import pandas

schema = pandera.DataFrameSchema({"datetime_col": pandera.Column("datetime64[ns]", coerce=True)})
dataframe = pandas.DataFrame({"datetime_col": ["2022-01-01", "123"]})

try:
    schema.validate(dataframe, lazy=True)
except pandera.errors.SchemaErrors as schema_errors:
    print(schema_errors.failure_cases.failure_case)
```

#### Expected behavior
In the example above, the validation fails and there should be "123" printed as a failure case. However, even if the validation is failing, there's no failure case in the `pandera.errors.SchemaErrors`.
So either "123" should be coerced to a `datetime` object and the validation should pass, or the validation should fail and "123" should be listed as a failure case. I think the latter is the expected behaviour, since it is `pandas.to_datetime` behaviour.

#### Desktop (please complete the following information):

 - OS: Debian



Discussion:

Resolving patch:
diff --git a/pandera/engines/pandas_engine.py b/pandera/engines/pandas_engine.py
--- a/pandera/engines/pandas_engine.py
+++ b/pandera/engines/pandas_engine.py
@@ -11,7 +11,7 @@
 import datetime
 import inspect
 import warnings
-from typing import Any, Dict, Iterable, List, Optional, Type, Union
+from typing import Any, Callable, Dict, Iterable, List, Optional, Type, Union
 
 import numpy as np
 import pandas as pd
@@ -595,39 +595,46 @@ def __post_init__(self):
 
         object.__setattr__(self, "type", type_)
 
-    def coerce(self, data_container: PandasObject) -> PandasObject:
-        def _to_datetime(col: pd.Series) -> pd.Series:
-            # NOTE: this is a hack to support pyspark.pandas. This needs to be
-            # thoroughly tested, right now pyspark.pandas returns NA when a
-            # dtype value can't be coerced into the target dtype.
-            to_datetime_fn = pd.to_datetime
-            if type(col).__module__.startswith(
-                "pyspark.pandas"
-            ):  # pragma: no cover
-                # pylint: disable=import-outside-toplevel
-                import pyspark.pandas as ps
+    @staticmethod
+    def _get_to_datetime_fn(obj: Any) -> Callable:
 
-                to_datetime_fn = ps.to_datetime
-            if type(col).__module__.startswith("modin.pandas"):
-                # pylint: disable=import-outside-toplevel
-                import modin.pandas as mpd
+        # NOTE: this is a hack to support pyspark.pandas. This needs to be
+        # thoroughly tested, right now pyspark.pandas returns NA when a
+        # dtype value can't be coerced into the target dtype.
+        to_datetime_fn = pd.to_datetime
+        if type(obj).__module__.startswith(
+            "pyspark.pandas"
+        ):  # pragma: no cover
+            # pylint: disable=import-outside-toplevel
+            import pyspark.pandas as ps
+
+            to_datetime_fn = ps.to_datetime
+        if type(obj).__module__.startswith("modin.pandas"):
+            # pylint: disable=import-outside-toplevel
+            import modin.pandas as mpd
 
-                to_datetime_fn = mpd.to_datetime
+            to_datetime_fn = mpd.to_datetime
 
+        return to_datetime_fn
+
+    def coerce(self, data_container: PandasObject) -> PandasObject:
+        to_datetime_fn = self._get_to_datetime_fn(data_container)
+
+        def _to_datetime(col: PandasObject) -> PandasObject:
             col = to_datetime_fn(col, **self.to_datetime_kwargs)
             return col.astype(self.type)
 
         if isinstance(data_container, pd.DataFrame):
             # pd.to_datetime transforms a df input into a series.
             # We actually want to coerce every columns.
-            return data_container.transform(_to_datetime)
+            return data_container.transform(to_datetime_fn)
         return _to_datetime(data_container)
 
     def coerce_value(self, value: Any) -> Any:
         """Coerce an value to specified datatime type."""
-        if value is pd.NaT:
-            return value
-        return super().coerce_value(value)
+        return self._get_to_datetime_fn(value)(
+            value, **self.to_datetime_kwargs
+        )
 
     @classmethod
     def from_parametrized_dtype(cls, pd_dtype: pd.DatetimeTZDtype):
diff --git a/pandera/version.py b/pandera/version.py
--- a/pandera/version.py
+++ b/pandera/version.py
@@ -1,3 +1,3 @@
 """Version file"""
 
-__version__ = "0.10.0"
+__version__ = "0.10.1"


Test patch: