Repository: great-expectations/great_expectations

Issue:
Pandas ``IntXDtype`` types not supported
**Describe the bug**
When a dataframe uses one of Pandas' nullable ``IntXDtype()`` types (most commonly, ``Int32Dtype()``), there is no way to validate this type in ``expect_column_values_to_be_in_type_list``.

**To Reproduce**
1. Create a dataframe with a ``pd.Int32Dtype()`` column. It may need to contain some null values.
2. Save it to parquet (tested with pyarrow engine).
3. Create expectation suite that checks the type of that column. The validation can include all the usual int types. Also, for good measure, add ``Int32Dtype``, which, reading the ``_validate_pandas`` function, seems intended to work.
4. Run the expectation suite against the parquet file.

**Example**
```python
validator.expect_column_values_to_be_in_type_list(
    column="Week",
    type_list=[
        "int32",  # This seems like it should work based on error message
        "Int32Dtype",  # This should work, based on reading code
    ],
)
```
returns
```
{
  "exception_info": {
    "raised_exception": false,
    "exception_traceback": null,
    "exception_message": null
  },
  "meta": {},
  "success": false,
  "result": {
    "observed_value": "int32"
  }
}
```
even though
```python
validator.head().dtypes['Week']
```
returns
```
Int32Dtype()
```

This is because
```python
assert validator.head().dtypes['Week'] not in [pd.Int32Dtype]
assert validator.head().dtypes['Week'] in [pd.Int32Dtype()]
```

**Expected behavior**
- Including ``Int32Dtype`` ought to result in successful validation.
- Failure ought to indicate that this string should be added (instead, it prints that the column type is ``int32``, which is very confusing if ``int32`` is in the list of types to consider valid).
- The various ``IntXDtype`` types should be automatically included in the auto-generated list of integer types used by the default profiler.

**Environment (please complete the following information):**
 - Operating System: Windows 11
 - Great Expectations Version: 0.14.1

**Fix**
``ExpectColumnValuesToBeInTypeList._validate_pandas`` (line 345 and following) does provide functionality for looking up dtypes from pandas. However, it uses the types themselves, whereas these Pandas dtypes (subclasses of ``ExtensionDtype``) are intended to be instantiated like ``pd.Int32Dtype()`` to get the desired type.

Thus,
```python
                        pd_type = getattr(pd, type_)
                        if isinstance(pd_type, type):
                            comp_types.append(pd_type)
```
should be updated to something like
```python
                        pd_type = getattr(pd, type_)
                        if isinstance(pd_type, type):
                            comp_types.append(pd_type())
```
Applying this change makes my particular example succeed when ``Int32Dtype`` is included in the list.


Discussion:
I'd like to submit a PR for this.

Resolving patch:
diff --git a/great_expectations/expectations/core/expect_column_values_to_be_in_type_list.py b/great_expectations/expectations/core/expect_column_values_to_be_in_type_list.py
--- a/great_expectations/expectations/core/expect_column_values_to_be_in_type_list.py
+++ b/great_expectations/expectations/core/expect_column_values_to_be_in_type_list.py
@@ -4,6 +4,7 @@
 
 import numpy as np
 import pandas as pd
+from packaging import version
 
 from great_expectations.core import ExpectationConfiguration
 from great_expectations.exceptions import InvalidExpectationConfigurationError
@@ -341,11 +342,18 @@ def _validate_pandas(
                 except TypeError:
                     try:
                         pd_type = getattr(pd, type_)
-                        if isinstance(pd_type, type):
-                            comp_types.append(pd_type)
                     except AttributeError:
                         pass
-
+                    else:
+                        if isinstance(pd_type, type):
+                            comp_types.append(pd_type)
+                            try:
+                                if isinstance(
+                                    pd_type(), pd.core.dtypes.base.ExtensionDtype
+                                ):
+                                    comp_types.append(pd_type())
+                            except TypeError:
+                                pass
                     try:
                         pd_type = getattr(pd.core.dtypes.dtypes, type_)
                         if isinstance(pd_type, type):
@@ -357,6 +365,31 @@ def _validate_pandas(
                 if native_type is not None:
                     comp_types.extend(native_type)
 
+            # TODO: Remove when Numpy >=1.21 is pinned as a dependency
+            _pandas_supports_extension_dtypes = version.parse(
+                pd.__version__
+            ) >= version.parse("0.24")
+            _numpy_doesnt_support_extensions_properly = version.parse(
+                np.__version__
+            ) < version.parse("1.21")
+            if (
+                _numpy_doesnt_support_extensions_properly
+                and _pandas_supports_extension_dtypes
+            ):
+                # This works around a bug where Pandas nullable int types aren't compatible with Numpy dtypes
+                # Note: Can't do set difference, the whole bugfix is because numpy types can't be compared to
+                # ExtensionDtypes
+                actual_type_is_ext_dtype = isinstance(
+                    actual_column_type, pd.core.dtypes.base.ExtensionDtype
+                )
+                comp_types = {
+                    dtype
+                    for dtype in comp_types
+                    if isinstance(dtype, pd.core.dtypes.base.ExtensionDtype)
+                    == actual_type_is_ext_dtype
+                }
+            ###
+
             success = actual_column_type in comp_types
 
         return {
diff --git a/great_expectations/profile/base.py b/great_expectations/profile/base.py
--- a/great_expectations/profile/base.py
+++ b/great_expectations/profile/base.py
@@ -124,6 +124,14 @@ class ProfilerTypeMapping:
         "uint16",
         "uint32",
         "uint64",
+        "Int8Dtype",
+        "Int16Dtype",
+        "Int32Dtype",
+        "Int64Dtype",
+        "UInt8Dtype",
+        "UInt16Dtype",
+        "UInt32Dtype",
+        "UInt64Dtype",
         "INT",
         "INTEGER",
         "INT64",


Test patch:
diff --git a/tests/expectations/core/test_expect_column_values_to_be_in_type_list.py b/tests/expectations/core/test_expect_column_values_to_be_in_type_list.py
--- a/tests/expectations/core/test_expect_column_values_to_be_in_type_list.py
+++ b/tests/expectations/core/test_expect_column_values_to_be_in_type_list.py
@@ -1,9 +1,13 @@
 import pandas as pd
+import pytest
 
 from great_expectations.core.expectation_validation_result import (
     ExpectationValidationResult,
 )
-from great_expectations.self_check.util import build_sa_validator_with_data
+from great_expectations.self_check.util import (
+    build_pandas_validator_with_data,
+    build_sa_validator_with_data,
+)
 
 
 def test_expect_column_values_to_be_in_type_list_dialect_pyathena(sa):
@@ -45,3 +49,46 @@ def test_expect_column_values_to_be_in_type_list_dialect_pyathena(sa):
         },
         meta={},
     )
+
+
+def test_expect_column_values_to_be_in_type_list_nullable_int():
+    from packaging.version import parse
+
+    pandas_version = parse(pd.__version__)
+    if pandas_version < parse("0.24"):
+        # Prior to 0.24, Pandas did not have
+        pytest.skip("Prior to 0.24, Pandas did not have `Int32Dtype` or related.")
+
+    df = pd.DataFrame({"col": pd.Series([1, 2, None], dtype=pd.Int32Dtype())})
+    validator = build_pandas_validator_with_data(df)
+
+    result = validator.expect_column_values_to_be_in_type_list(
+        "col", type_list=["Int32Dtype"]
+    )
+    assert result == ExpectationValidationResult(
+        success=True,
+        expectation_config={
+            "expectation_type": "expect_column_values_to_be_in_type_list",
+            "kwargs": {
+                "column": "col",
+                "type_list": ["Int32Dtype"],
+            },
+            "meta": {},
+        },
+        result={
+            "element_count": 3,
+            "unexpected_count": 0,
+            "unexpected_percent": 0.0,
+            "partial_unexpected_list": [],
+            "missing_count": 0,
+            "missing_percent": 0.0,
+            "unexpected_percent_total": 0.0,
+            "unexpected_percent_nonmissing": 0.0,
+        },
+        exception_info={
+            "raised_exception": False,
+            "exception_traceback": None,
+            "exception_message": None,
+        },
+        meta={},
+    )
diff --git a/tests/profile/test_jsonschema_profiler.py b/tests/profile/test_jsonschema_profiler.py
--- a/tests/profile/test_jsonschema_profiler.py
+++ b/tests/profile/test_jsonschema_profiler.py
@@ -338,6 +338,14 @@ def test_profile_simple_schema(empty_data_context, simple_schema):
                     "uint16",
                     "uint32",
                     "uint64",
+                    "Int8Dtype",
+                    "Int16Dtype",
+                    "Int32Dtype",
+                    "Int64Dtype",
+                    "UInt8Dtype",
+                    "UInt16Dtype",
+                    "UInt32Dtype",
+                    "UInt64Dtype",
                     "INT",
                     "INTEGER",
                     "INT64",
@@ -757,6 +765,14 @@ def test_profile_integer_ranges_schema(empty_data_context, integer_ranges_schema
                     "uint16",
                     "uint32",
                     "uint64",
+                    "Int8Dtype",
+                    "Int16Dtype",
+                    "Int32Dtype",
+                    "Int64Dtype",
+                    "UInt8Dtype",
+                    "UInt16Dtype",
+                    "UInt32Dtype",
+                    "UInt64Dtype",
                     "INT",
                     "INTEGER",
                     "INT64",
@@ -797,6 +813,14 @@ def test_profile_integer_ranges_schema(empty_data_context, integer_ranges_schema
                     "uint16",
                     "uint32",
                     "uint64",
+                    "Int8Dtype",
+                    "Int16Dtype",
+                    "Int32Dtype",
+                    "Int64Dtype",
+                    "UInt8Dtype",
+                    "UInt16Dtype",
+                    "UInt32Dtype",
+                    "UInt64Dtype",
                     "INT",
                     "INTEGER",
                     "INT64",
@@ -842,6 +866,14 @@ def test_profile_integer_ranges_schema(empty_data_context, integer_ranges_schema
                     "uint16",
                     "uint32",
                     "uint64",
+                    "Int8Dtype",
+                    "Int16Dtype",
+                    "Int32Dtype",
+                    "Int64Dtype",
+                    "UInt8Dtype",
+                    "UInt16Dtype",
+                    "UInt32Dtype",
+                    "UInt64Dtype",
                     "INT",
                     "INTEGER",
                     "INT64",
@@ -887,6 +919,14 @@ def test_profile_integer_ranges_schema(empty_data_context, integer_ranges_schema
                     "uint16",
                     "uint32",
                     "uint64",
+                    "Int8Dtype",
+                    "Int16Dtype",
+                    "Int32Dtype",
+                    "Int64Dtype",
+                    "UInt8Dtype",
+                    "UInt16Dtype",
+                    "UInt32Dtype",
+                    "UInt64Dtype",
                     "INT",
                     "INTEGER",
                     "INT64",
@@ -932,6 +972,14 @@ def test_profile_integer_ranges_schema(empty_data_context, integer_ranges_schema
                     "uint16",
                     "uint32",
                     "uint64",
+                    "Int8Dtype",
+                    "Int16Dtype",
+                    "Int32Dtype",
+                    "Int64Dtype",
+                    "UInt8Dtype",
+                    "UInt16Dtype",
+                    "UInt32Dtype",
+                    "UInt64Dtype",
                     "INT",
                     "INTEGER",
                     "INT64",
@@ -977,6 +1025,14 @@ def test_profile_integer_ranges_schema(empty_data_context, integer_ranges_schema
                     "uint16",
                     "uint32",
                     "uint64",
+                    "Int8Dtype",
+                    "Int16Dtype",
+                    "Int32Dtype",
+                    "Int64Dtype",
+                    "UInt8Dtype",
+                    "UInt16Dtype",
+                    "UInt32Dtype",
+                    "UInt64Dtype",
                     "INT",
                     "INTEGER",
                     "INT64",
@@ -1026,6 +1082,14 @@ def test_profile_integer_ranges_schema(empty_data_context, integer_ranges_schema
                     "uint16",
                     "uint32",
                     "uint64",
+                    "Int8Dtype",
+                    "Int16Dtype",
+                    "Int32Dtype",
+                    "Int64Dtype",
+                    "UInt8Dtype",
+                    "UInt16Dtype",
+                    "UInt32Dtype",
+                    "UInt64Dtype",
                     "INT",
                     "INTEGER",
                     "INT64",
@@ -1075,6 +1139,14 @@ def test_profile_integer_ranges_schema(empty_data_context, integer_ranges_schema
                     "uint16",
                     "uint32",
                     "uint64",
+                    "Int8Dtype",
+                    "Int16Dtype",
+                    "Int32Dtype",
+                    "Int64Dtype",
+                    "UInt8Dtype",
+                    "UInt16Dtype",
+                    "UInt32Dtype",
+                    "UInt64Dtype",
                     "INT",
                     "INTEGER",
                     "INT64",
@@ -1126,6 +1198,14 @@ def test_profile_integer_ranges_schema(empty_data_context, integer_ranges_schema
                     "uint16",
                     "uint32",
                     "uint64",
+                    "Int8Dtype",
+                    "Int16Dtype",
+                    "Int32Dtype",
+                    "Int64Dtype",
+                    "UInt8Dtype",
+                    "UInt16Dtype",
+                    "UInt32Dtype",
+                    "UInt64Dtype",
                     "INT",
                     "INTEGER",
                     "INT64",
@@ -1635,6 +1715,14 @@ def test_has_profile_create_expectations_from_complex_schema(
                     "uint16",
                     "uint32",
                     "uint64",
+                    "Int8Dtype",
+                    "Int16Dtype",
+                    "Int32Dtype",
+                    "Int64Dtype",
+                    "UInt8Dtype",
+                    "UInt16Dtype",
+                    "UInt32Dtype",
+                    "UInt64Dtype",
                     "INT",
                     "INTEGER",
                     "INT64",
@@ -1846,6 +1934,14 @@ def test_null_fields_schema(empty_data_context, null_fields_schema):
                     "uint16",
                     "uint32",
                     "uint64",
+                    "Int8Dtype",
+                    "Int16Dtype",
+                    "Int32Dtype",
+                    "Int64Dtype",
+                    "UInt8Dtype",
+                    "UInt16Dtype",
+                    "UInt32Dtype",
+                    "UInt64Dtype",
                     "INT",
                     "INTEGER",
                     "INT64",