Repository: unionai-oss/pandera (Python)

Issue:
Pandera 16.0 can't be imported without installing pyspark
**Describe the bug**

- Create a fresh venv.
- `pip install pandera`
- `python -c 'import pandera'`

```
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/smackesey/stm/tmp/panderavenv/.venv/lib/python3.10/site-packages/pandera/__init__.py", line 4, in <module>
    import pandera.backends
  File "/Users/smackesey/stm/tmp/panderavenv/.venv/lib/python3.10/site-packages/pandera/backends/__init__.py", line 6, in <module>
    import pandera.backends.pandas
  File "/Users/smackesey/stm/tmp/panderavenv/.venv/lib/python3.10/site-packages/pandera/backends/pandas/__init__.py", line 5, in <module>
    import pandera.typing
  File "/Users/smackesey/stm/tmp/panderavenv/.venv/lib/python3.10/site-packages/pandera/typing/__init__.py", line 9, in <module>
    from pandera.typing import (
  File "/Users/smackesey/stm/tmp/panderavenv/.venv/lib/python3.10/site-packages/pandera/typing/dask.py", line 5, in <module>
    from pandera.typing.common import DataFrameBase, IndexBase, SeriesBase
  File "/Users/smackesey/stm/tmp/panderavenv/.venv/lib/python3.10/site-packages/pandera/typing/common.py", line 11, in <module>
    from pandera.engines import numpy_engine, pandas_engine, pyspark_engine
  File "/Users/smackesey/stm/tmp/panderavenv/.venv/lib/python3.10/site-packages/pandera/engines/pyspark_engine.py", line 17, in <module>
    import pyspark.sql.types as pst
ModuleNotFoundError: No module named 'pyspark'
```

- [x] I have checked that this issue has not already been reported.
- [x] I have confirmed this bug exists on the latest version of pandera.
- [ ] (optional) I have confirmed this bug exists on the master branch of pandera.

#### Desktop (please complete the following information):

 - OS: macOS




Discussion: To add to @smackesey - can also confirm this bug exists on the `main` branch, with a minimal Dockerfile example:

```Dockerfile
FROM python:3.10

ENV PIP_ROOT_USER_ACTION=ignore
RUN pip install --upgrade pip
RUN pip install git+https://github.com/unionai-oss/pandera@main

ENTRYPOINT python -c 'import pandera'
```

```
> docker build . --no-cache -t 'pandera-main' && docker run 'pandera-main'
[+] Building 19.2s (7/7) FINISHED
 => [internal] load build definition from Dockerfile                                                                                             0.0s
 => => transferring dockerfile: 228B                                                                                                             0.0s
 => [internal] load .dockerignore                                                                                                                0.0s
 => => transferring context: 2B                                                                                                                  0.0s
 => [internal] load metadata for docker.io/library/python:3.10                                                                                   1.3s
 => CACHED [1/3] FROM docker.io/library/python:3.10@sha256:1b2e0805e24189fbba4e55b9bee89e3c25533cbe4fb71ae151f3e7ae0c9b86c5                      0.0s
 => [2/3] RUN pip install --upgrade pip                                                                                                          3.8s
 => [3/3] RUN pip install git+https://github.com/unionai-oss/pandera@main                                                                       13.4s
 => exporting to image                                                                                                                           0.6s
 => => exporting layers                                                                                                                          0.6s
 => => writing image sha256:475ba4457bd99ca1fc9eb1cc7425bc96820a0e152278ae55d9c7983b228c047d                                                     0.0s
 => => naming to docker.io/library/pandera-main                                                                                                  0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.10/site-packages/pandera/__init__.py", line 4, in <module>
    import pandera.backends
  File "/usr/local/lib/python3.10/site-packages/pandera/backends/__init__.py", line 6, in <module>
    import pandera.backends.pandas
  File "/usr/local/lib/python3.10/site-packages/pandera/backends/pandas/__init__.py", line 5, in <module>
    import pandera.typing
  File "/usr/local/lib/python3.10/site-packages/pandera/typing/__init__.py", line 9, in <module>
    from pandera.typing import (
  File "/usr/local/lib/python3.10/site-packages/pandera/typing/dask.py", line 5, in <module>
    from pandera.typing.common import DataFrameBase, IndexBase, SeriesBase
  File "/usr/local/lib/python3.10/site-packages/pandera/typing/common.py", line 11, in <module>
    from pandera.engines import numpy_engine, pandas_engine, pyspark_engine
  File "/usr/local/lib/python3.10/site-packages/pandera/engines/pyspark_engine.py", line 17, in <module>
    import pyspark.sql.types as pst
ModuleNotFoundError: No module named 'pyspark'
```
I confirm I am also affected by that. For now, I am pinning to `0.15.2` in my use cases.

Buggy line of code: https://github.com/unionai-oss/pandera/blob/main/pandera/typing/common.py#L11
I also confirm this issue in that line, looks like the pyspark imports need to be wrapped in conditional logic similar to the geopandas ones.
https://github.com/unionai-oss/pandera/pull/1268 should fix this!

will cut a bugfix release later today
Just cut a `0.16.1` release. @smackesey can you try again?
It works now, thanks!



Fix (patch):
diff --git a/pandera/api/checks.py b/pandera/api/checks.py
index 51f7d11e6..1f2659c64 100644
--- a/pandera/api/checks.py
+++ b/pandera/api/checks.py
@@ -12,8 +12,6 @@
     Union,
 )
 
-import pandas as pd
-
 from pandera import errors
 from pandera.api.base.checks import BaseCheck, CheckResult
 from pandera.strategies import SearchStrategy
@@ -200,7 +198,7 @@ def __init__(
 
     def __call__(
         self,
-        check_obj: Union[pd.DataFrame, pd.Series],
+        check_obj: Any,
         column: Optional[str] = None,
     ) -> CheckResult:
         # pylint: disable=too-many-branches
diff --git a/pandera/api/extensions.py b/pandera/api/extensions.py
index 2e0f92350..c2a1b6c57 100644
--- a/pandera/api/extensions.py
+++ b/pandera/api/extensions.py
@@ -213,11 +213,7 @@ def register_check_method(  # pylint:disable=too-many-branches
         if supported_type not in ALLOWED_TYPES:
             raise TypeError(msg.format(supported_type))
 
-    if check_type is CheckType.ELEMENT_WISE and set(supported_types) != {
-        pd.DataFrame,
-        pd.Series,
-        ps.DataFrame,
-    }:  # type: ignore
+    if check_type is CheckType.ELEMENT_WISE and set(supported_types) != ALLOWED_TYPES:  # type: ignore
         raise ValueError(
             "Element-wise checks should support DataFrame and Series "
             "validation. Use the default setting for the 'supported_types' "
diff --git a/pandera/api/hypotheses.py b/pandera/api/hypotheses.py
index 2c1b46a5a..2ae53293b 100644
--- a/pandera/api/hypotheses.py
+++ b/pandera/api/hypotheses.py
@@ -1,6 +1,6 @@
 """Data validation checks for hypothesis testing."""
 
-from typing import Any, Callable, Dict, List, Optional, Union
+from typing import Any, Callable, Dict, List, Optional, TypeVar, Union
 
 from pandera import errors
 from pandera.api.checks import Check
@@ -9,6 +9,9 @@
 DEFAULT_ALPHA = 0.01
 
 
+T = TypeVar("T")
+
+
 class Hypothesis(Check):
     """Special type of :class:`Check` that defines hypothesis tests on data."""
 
diff --git a/pandera/backends/pandas/builtin_hypotheses.py b/pandera/backends/pandas/builtin_hypotheses.py
index e29fbc444..1dba2dca9 100644
--- a/pandera/backends/pandas/builtin_hypotheses.py
+++ b/pandera/backends/pandas/builtin_hypotheses.py
@@ -5,10 +5,6 @@
 
 from pandera.api.extensions import register_builtin_hypothesis
 from pandera.backends.pandas.builtin_checks import PandasData
-from pandera.backends.pandas.hypotheses import HAS_SCIPY
-
-if HAS_SCIPY:
-    from scipy import stats
 
 
 @register_builtin_hypothesis(
@@ -20,6 +16,8 @@ def two_sample_ttest(
     equal_var: bool = True,
     nan_policy: str = "propagate",
 ) -> Tuple[float, float]:
+    from scipy import stats  # pylint: disable=import-outside-toplevel
+
     assert (
         len(samples) == 2
     ), "Expected two sample ttest data to contain exactly two samples"
@@ -40,6 +38,8 @@ def one_sample_ttest(
     popmean: float,
     nan_policy: str = "propagate",
 ) -> Tuple[float, float]:
+    from scipy import stats  # pylint: disable=import-outside-toplevel
+
     assert (
         len(samples) == 1
     ), "Expected one sample ttest data to contain only one sample"
diff --git a/pandera/typing/__init__.py b/pandera/typing/__init__.py
index 08185c6d3..8de4e6ff4 100644
--- a/pandera/typing/__init__.py
+++ b/pandera/typing/__init__.py
@@ -20,16 +20,6 @@
     INT16,
     INT32,
     INT64,
-    PYSPARK_BINARY,
-    PYSPARK_BYTEINT,
-    PYSPARK_DATE,
-    PYSPARK_DECIMAL,
-    PYSPARK_FLOAT,
-    PYSPARK_INT,
-    PYSPARK_LONGINT,
-    PYSPARK_SHORTINT,
-    PYSPARK_STRING,
-    PYSPARK_TIMESTAMP,
     STRING,
     UINT8,
     UINT16,
diff --git a/pandera/typing/common.py b/pandera/typing/common.py
index 611588b7e..996fb64d5 100644
--- a/pandera/typing/common.py
+++ b/pandera/typing/common.py
@@ -8,7 +8,7 @@
 import typing_inspect
 
 from pandera import dtypes
-from pandera.engines import numpy_engine, pandas_engine, pyspark_engine
+from pandera.engines import numpy_engine, pandas_engine
 
 Bool = dtypes.Bool  #: ``"bool"`` numpy dtype
 Date = dtypes.Date  #: ``datetime.date`` object dtype
@@ -43,17 +43,7 @@
 #: fall back on the str-as-object-array representation.
 STRING = pandas_engine.STRING  #: ``"str"`` numpy dtype
 BOOL = pandas_engine.BOOL  #: ``"str"`` numpy dtype
-PYSPARK_STRING = pyspark_engine.String
-PYSPARK_INT = pyspark_engine.Int
-PYSPARK_LONGINT = pyspark_engine.BigInt
-PYSPARK_SHORTINT = pyspark_engine.ShortInt
-PYSPARK_BYTEINT = pyspark_engine.ByteInt
-PYSPARK_DOUBLE = pyspark_engine.Double
-PYSPARK_FLOAT = pyspark_engine.Float
-PYSPARK_DECIMAL = pyspark_engine.Decimal
-PYSPARK_DATE = pyspark_engine.Date
-PYSPARK_TIMESTAMP = pyspark_engine.Timestamp
-PYSPARK_BINARY = pyspark_engine.Binary
+
 
 try:
     Geometry = pandas_engine.Geometry  # : ``"geometry"`` geopandas dtype
@@ -101,16 +91,6 @@
             String,
             STRING,
             Geometry,
-            pyspark_engine.String,
-            pyspark_engine.Int,
-            pyspark_engine.BigInt,
-            pyspark_engine.ShortInt,
-            pyspark_engine.ByteInt,
-            pyspark_engine.Float,
-            pyspark_engine.Decimal,
-            pyspark_engine.Date,
-            pyspark_engine.Timestamp,
-            pyspark_engine.Binary,
         ],
     )
 else:
@@ -152,16 +132,6 @@
             Object,
             String,
             STRING,
-            pyspark_engine.String,
-            pyspark_engine.Int,
-            pyspark_engine.BigInt,
-            pyspark_engine.ShortInt,
-            pyspark_engine.ByteInt,
-            pyspark_engine.Float,
-            pyspark_engine.Decimal,
-            pyspark_engine.Date,
-            pyspark_engine.Timestamp,
-            pyspark_engine.Binary,
         ],
     )
 
@@ -236,20 +206,6 @@ def __get__(
         raise AttributeError("Indexes should resolve to pa.Index-s")
 
 
-class ColumnBase(Generic[GenericDtype]):
-    """Representation of pandas.Index, only used for type annotation.
-
-    *new in 0.5.0*
-    """
-
-    default_dtype: Optional[Type] = None
-
-    def __get__(
-        self, instance: object, owner: Type
-    ) -> str:  # pragma: no cover
-        raise AttributeError("column should resolve to pyspark.sql.Column-s")
-
-
 class AnnotationInfo:  # pylint:disable=too-few-public-methods
     """Captures extra information about an annotation.