Repository: unionai-oss/pandera

Issue:
TypeError when using numpy.intc
Hello,

I've encountered a problem when trying to migrate from version 0.6.2 to 0.7.0. With versions up until 0.6.5 everything works as expected, but with >=0.7.0 I'm getting a TypeError exception when using numpy.intc in a pandas DataFrame. I'm not sure if this is a bug or the intended behaviour.

This is the full error message:

cls = <class 'pandera.engines.pandas_engine.Engine'>, data_type = <class 'numpy.intc'>

    def dtype(cls: _EngineType, data_type: Any) -> _DataType:
        """Convert input into a Pandera :class:`DataType` object."""
        if isinstance(data_type, cls._base_pandera_dtypes):
            return data_type
    
        if inspect.isclass(data_type) and issubclass(
            data_type, cls._base_pandera_dtypes
        ):
            try:
                return data_type()
            except (TypeError, AttributeError) as err:
                raise TypeError(
                    f"DataType '{data_type.__name__}' cannot be instantiated: "
                    f"{err}\n "
                    + "Usage Tip: Use an instance or a string representation."
                ) from err
    
        registry = cls._registry[cls]
    
        equivalent_data_type = registry.equivalents.get(data_type)
        if equivalent_data_type is not None:
            return equivalent_data_type
    
        try:
            return registry.dispatch(data_type)
        except (KeyError, ValueError):
            raise TypeError(
                f"Data type '{data_type}' not understood by {cls.__name__}.") from None

TypeError: Data type '<class 'numpy.intc'>' not understood by Engine.


Discussion:
Hi @bstefania. Thanks for reporting the error. That's indeed not intended.

Could you provide a minimal reproducible example? The error is not triggered when I run this simple code:

```python
import platform
import numpy as np
import pandas as pd
import pandera as pa
from pandera.engines import pandas_engine


schema = pa.DataFrameSchema({"col": pa.Column(np.intc)}, coerce=True)
df = pd.DataFrame({"col": [1]}, dtype=np.intc)

schema.validate(df).info()
#> <class 'pandas.core.frame.DataFrame'>
#> RangeIndex: 1 entries, 0 to 0
#> Data columns (total 1 columns):
#>  #   Column  Non-Null Count  Dtype
#> ---  ------  --------------  -----
#>  0   col     1 non-null      int32
#> dtypes: int32(1)
#> memory usage: 132.0 bytes

print(pandas_engine.Engine.dtype(np.intc)) # <-- Should trigger your error, called internally by DataFrameSchema
#> int32


print(f"Pandera {pa.__version__} @ {platform.system()} {platform.release()}")
#> Pandera 0.7.1 @ Linux 5.13.13-200.fc34.x86_64
```

^ On my Linux system, `np.intc` is an alias for `np.int32`. Could you also post the output you get from the above snippet? 

I've created a test_file.csv to be able to reproduce exactly what I want to do. 
It looks like this:
_number, type
1, int
2, int_

And here is the code:
```
import pandera as pa
import datatable as dt


column_names = ['number', 'type']
file = 'test_file.csv'

data = dt.fread(file, sep=",", strip_whitespace=True, dec=".", header=True, columns=column_names).to_pandas()

for cols in data.columns:
    print(type(data[cols].iloc[0]))

columns = {'number': pa.Column(pa.Int32, nullable=False)}
schema = pa.DataFrameSchema(strict=True, columns=columns)

schema.validate(data, lazy=True, inplace=True)
```

When I'm creating directly a data frame with dtype = np.intc for me too is converted to np.int32. 
I had to set `strict=False` in your example since the column `type` is not in the schema. Otherwise, the error described in your initial post is not triggered on my side. Does your example trigger the error on your machine?

Does the snippet I provided run without errors for you?
Both examples trigger the error on my machine. With your example it is triggered also in the
`schema = pa.DataFrameSchema({"col": pa.Column(np.intc)}, coerce=True)`
line, not just
`print(pandas_engine.Engine.dtype(np.intc))`

I'm on Windows 10 and using Python 3.7.11 in a conda environment. These are the package versions:
datatable 1.0.0
numpy 1.20.3
pandas 1.3.1
pandera 0.7.1
pyarrow 5.0.0
typing-extensions 3.10.0.2
typing-inspect 0.7.1
wrapt 1.12.1
@bstefania seems like this is related to https://github.com/numpy/numpy/issues/9799

what's the output of this?
```python
df = pd.DataFrame({"col": [1]}, dtype=np.intc)
print(df.info())
```
This is the output:

```
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1 entries, 0 to 0
Data columns (total 1 columns):
 #   Column  Non-Null Count  Dtype
---  ------  --------------  -----
 0   col     1 non-null      int32
dtypes: int32(1)
memory usage: 132.0 bytes
None

```

I don't think that would be a problem since the error appeared only after updating pandera, and if I set it back to 0.6.2 everything's fine. 
what I'm confused by is:

if `np.intc` resolves to `np.int32`, how is the error message reporting
```
TypeError: Data type '<class 'numpy.intc'>' not understood by Engine.
```
In the `Engine.dtype` call?

@bstefania what does `print(np.intc)` output?
Hi, sorry for the late reply. It is not always transformed to `np.int32`. When I run my example,  the printed type is `<class 'numpy.intc'>`. Maybe this is due to the datatable fread. 

The output for `print(np.intc)` is `<class 'numpy.intc'> `
@bstefania I confirm I was able to reproduce the error with your example on a windows machine. I'll work on a fix.

Resolving patch:
diff --git a/pandera/engines/numpy_engine.py b/pandera/engines/numpy_engine.py
--- a/pandera/engines/numpy_engine.py
+++ b/pandera/engines/numpy_engine.py
@@ -313,6 +313,17 @@ class Complex64(Complex128):
     bit_width: int = 64
 
 
+###############################################################################
+# bytes
+###############################################################################
+
+
+@Engine.register_dtype(equivalents=["bytes", bytes, np.bytes_])
+@immutable
+class Bytes(DataType):
+    type = np.dtype("bytes")
+
+
 ###############################################################################
 # string
 ###############################################################################
diff --git a/pandera/engines/pandas_engine.py b/pandera/engines/pandas_engine.py
--- a/pandera/engines/pandas_engine.py
+++ b/pandera/engines/pandas_engine.py
@@ -137,7 +137,10 @@ def dtype(cls, data_type: Any) -> "DataType":
                 # into a numpy or pandas dtype.
                 np_or_pd_dtype = pd.api.types.pandas_dtype(data_type)
                 if isinstance(np_or_pd_dtype, np.dtype):
-                    np_or_pd_dtype = np_or_pd_dtype.type
+                    # cast alias to platform-agnostic dtype
+                    # e.g.: np.intc -> np.int32
+                    common_np_dtype = np.dtype(np_or_pd_dtype.name)
+                    np_or_pd_dtype = common_np_dtype.type
 
             return engine.Engine.dtype(cls, np_or_pd_dtype)
 
@@ -454,13 +457,18 @@ def check(self, pandera_dtype: dtypes.DataType) -> bool:
     numpy_engine.Object,
     equivalents=[
         "object",
+        "object_",
+        "object0",
         "O",
         "bytes",
         "decimal",
         "mixed-integer",
         "mixed",
+        "bytes",
+        bytes,
         object,
         np.object_,
+        np.bytes_,
     ],
 )
 


Test patch:
diff --git a/tests/core/test_dtypes.py b/tests/core/test_dtypes.py
--- a/tests/core/test_dtypes.py
+++ b/tests/core/test_dtypes.py
@@ -397,6 +397,28 @@ def test_default_numeric_dtypes():
     )
 
 
+@pytest.mark.parametrize(
+    "alias, np_dtype",
+    [
+        (alias, np_dtype)
+        for alias, np_dtype in np.sctypeDict.items()
+        # int, uint have different bitwidth under pandas and numpy.
+        if np_dtype != np.void and alias not in ("int", "uint")
+    ],
+)
+def test_numpy_dtypes(alias, np_dtype):
+    """Test that all pandas-compatible numpy dtypes are understood."""
+    try:
+        np.dtype(alias)
+    except TypeError:
+        # not a valid alias
+        assert pandas_engine.Engine.dtype(np_dtype)
+    else:
+        assert pandas_engine.Engine.dtype(alias) == pandas_engine.Engine.dtype(
+            np_dtype
+        )
+
+
 @pytest.mark.parametrize(
     "examples",
     [