Repository: pandas-dev/pandas
Pull request: Removed ABCs from pandas._typing
- [X] closes #27146
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

This reveals a few new typing issues so won't pass CI but submitting as draft

Here's one way I think that works to avoid import cycles. Basically any internal imports in pandas._typing need to be set in a TYPE_CHECKING block to prevent their runtime evaluation from causing import cycles.

To illustrate the effectiveness like this if you put something like this at the end of pandas._typing:

```python
def func(obj: FrameOrSeries) -> None:
    reveal_type(obj)
```

It would previously give you

```sh
pandas/_typing.py:43: note: Revealed type is 'Any'
```

Whereas now gives:

```sh
pandas/_typing.py:44: note: Revealed type is 'pandas.core.series.Series*'
pandas/_typing.py:44: note: Revealed type is 'pandas.core.frame.DataFrame*'
```

@topper-123 any way to check VSCode to see if this works with code completion?

In the long run this may not be needed with PEP 563:

https://www.python.org/dev/peps/pep-0563/

But I think this hack gets us there in the meantime

Problem statement:
Remove ABC Usage from pandas._typing
Right now we use the ABC classes in pandas._typing to work around circular imports, but I think this has the effect of making our type checking practically non-existent. The ABCs are factory generated by an unannotated function, so they get replaced with `Any`

To illustrate, run mypy on a module with this content

```python
from pandas.core.dtypes.generic import ABCDataFrame

some_obj = None  # type: ABCDataFrame
reveal_type(some_obj)
```

Mypy runs without issue yielding

```sh
test.py:4: note: Revealed type is 'Any'
```

By contrast, using the actual DataFrame class:

```python
import pandas as pd

some_obj = None  # type: pd.DataFrame
reveal_type(some_obj)
```

Yields an error:

```sh
(pandas-dev) mypy test.py
test.py:3: error: Incompatible types in assignment (expression has type "None", variable has type "DataFrame")
test.py:4: note: Revealed type is 'pandas.core.frame.DataFrame'
```

I'm not sure the best approach here. Direct imports into pandas._typing should cause circular imports but the ABCs won't provide anything with type checking as long as they are factory generated.

Hints from issue discussion:
what if we use the quoted imports eg ‘DataFrame’? doesn’t this achieve the same purpose?
In pandas._typing or in other modules?

This code:

```python
some_obj = None  # type: 'pd.DataFrame'
reveal_type(some_obj)
```

Yields:

```sh
test.py:1: error: Name 'pd' is not defined
test.py:2: note: Revealed type is 'Any'
```

You can avoid circular imports by guarding the imports in `if TYPE_CHECKING` blocks:
https://mypy.readthedocs.io/en/latest/common_issues.html#import-cycles

In xarray we wrote a little stub for `TYPE_CHECKING` so it even works in Python < 3.5.3:
https://github.com/pydata/xarray/blob/76aa4f059228befa788f07d751095452d8390426/xarray/core/pycompat.py#L16-LL17
does just ‘DataFrame’ or ‘pandas.core.frame.DataFrame’ work?

> You can avoid circular imports by guarding the imports in `if TYPE_CHECKING` blocks:

Right but IIUC we would then need to use that in all modules importing `pandas._typing` and then use literal references from there, and I'm not sure this fully resolves circular imports when the type checking is running (could be wrong will have to investigate more)

> does just ‘DataFrame’ or ‘pandas.core.frame.DataFrame’ work?

no those would yield `Name (pandas|DataFrame) is not defined`

Commit message: Removed failing annotation

Patch to review:
diff --git a/pandas/_typing.py b/pandas/_typing.py
index a1224a609579e..83ce82e968870 100644
--- a/pandas/_typing.py
+++ b/pandas/_typing.py
@@ -1,34 +1,37 @@
 from pathlib import Path
-from typing import IO, AnyStr, TypeVar, Union
+from typing import IO, TYPE_CHECKING, AnyStr, TypeVar, Union
 
 import numpy as np
 
-from pandas._libs import Timestamp
-from pandas._libs.tslibs.period import Period
-from pandas._libs.tslibs.timedeltas import Timedelta
+# To prevent import cycles place any internal imports in the branch below
+# and use a string literal forward reference to it in subsequent types
+# https://mypy.readthedocs.io/en/latest/common_issues.html#import-cycles
+if TYPE_CHECKING:
+    from pandas._libs import Period, Timedelta, Timestamp
+
+    from pandas.core.arrays.base import ExtensionArray
+    from pandas.core.dtypes.dtypes import ExtensionDtype
+    from pandas.core.dtypes.generic import (
+        ABCDataFrame,
+        ABCExtensionArray,
+        ABCIndexClass,
+        ABCSeries,
+        ABCSparseSeries,
+    )
+    from pandas.core.indexes.base import Index
+    from pandas.core.frame import DataFrame
+    from pandas.core.series import Series
+    from pandas.core.sparse.series import SparseSeries
 
-from pandas.core.dtypes.dtypes import ExtensionDtype
-from pandas.core.dtypes.generic import (
-    ABCDataFrame,
-    ABCExtensionArray,
-    ABCIndexClass,
-    ABCSeries,
-    ABCSparseSeries,
-)
 
 AnyArrayLike = TypeVar(
-    "AnyArrayLike",
-    ABCExtensionArray,
-    ABCIndexClass,
-    ABCSeries,
-    ABCSparseSeries,
-    np.ndarray,
+    "AnyArrayLike", "ExtensionArray", "Index", "Series", "SparseSeries", np.ndarray
 )
-ArrayLike = TypeVar("ArrayLike", ABCExtensionArray, np.ndarray)
-DatetimeLikeScalar = TypeVar("DatetimeLikeScalar", Period, Timestamp, Timedelta)
-Dtype = Union[str, np.dtype, ExtensionDtype]
+ArrayLike = TypeVar("ArrayLike", "ExtensionArray", np.ndarray)
+DatetimeLikeScalar = TypeVar("DatetimeLikeScalar", "Period", "Timestamp", "Timedelta")
+Dtype = Union[str, np.dtype, "ExtensionDtype"]
 FilePathOrBuffer = Union[str, Path, IO[AnyStr]]
 
-FrameOrSeries = TypeVar("FrameOrSeries", ABCSeries, ABCDataFrame)
+FrameOrSeries = TypeVar("FrameOrSeries", "Series", "DataFrame")
 Scalar = Union[str, int, float]
 Axis = Union[str, int]
diff --git a/pandas/core/dtypes/common.py b/pandas/core/dtypes/common.py
index f2571573bd1bc..054c97056a117 100644
--- a/pandas/core/dtypes/common.py
+++ b/pandas/core/dtypes/common.py
@@ -167,12 +167,13 @@ def ensure_int_or_float(arr: ArrayLike, copy=False) -> np.array:
     If the array is explicitly of type uint64 the type
     will remain unchanged.
     """
+    # TODO: GH27506 potential bug with ExtensionArrays
     try:
-        return arr.astype("int64", copy=copy, casting="safe")
+        return arr.astype("int64", copy=copy, casting="safe")  # type: ignore
     except TypeError:
         pass
     try:
-        return arr.astype("uint64", copy=copy, casting="safe")
+        return arr.astype("uint64", copy=copy, casting="safe")  # type: ignore
     except TypeError:
         return arr.astype("float64", copy=copy)
 
diff --git a/pandas/core/indexes/interval.py b/pandas/core/indexes/interval.py
index 7372dada3b48a..66290ae54e626 100644
--- a/pandas/core/indexes/interval.py
+++ b/pandas/core/indexes/interval.py
@@ -906,35 +906,35 @@ def get_indexer(
             )
             raise InvalidIndexError(msg)
 
-        target = ensure_index(target)
+        target_as_index = ensure_index(target)
 
-        if isinstance(target, IntervalIndex):
+        if isinstance(target_as_index, IntervalIndex):
             # equal indexes -> 1:1 positional match
-            if self.equals(target):
+            if self.equals(target_as_index):
                 return np.arange(len(self), dtype="intp")
 
             # different closed or incompatible subtype -> no matches
             common_subtype = find_common_type(
-                [self.dtype.subtype, target.dtype.subtype]
+                [self.dtype.subtype, target_as_index.dtype.subtype]
             )
-            if self.closed != target.closed or is_object_dtype(common_subtype):
-                return np.repeat(np.intp(-1), len(target))
+            if self.closed != target_as_index.closed or is_object_dtype(common_subtype):
+                return np.repeat(np.intp(-1), len(target_as_index))
 
-            # non-overlapping -> at most one match per interval in target
+            # non-overlapping -> at most one match per interval in target_as_index
             # want exact matches -> need both left/right to match, so defer to
             # left/right get_indexer, compare elementwise, equality -> match
-            left_indexer = self.left.get_indexer(target.left)
-            right_indexer = self.right.get_indexer(target.right)
+            left_indexer = self.left.get_indexer(target_as_index.left)
+            right_indexer = self.right.get_indexer(target_as_index.right)
             indexer = np.where(left_indexer == right_indexer, left_indexer, -1)
-        elif not is_object_dtype(target):
+        elif not is_object_dtype(target_as_index):
             # homogeneous scalar index: use IntervalTree
-            target = self._maybe_convert_i8(target)
-            indexer = self._engine.get_indexer(target.values)
+            target_as_index = self._maybe_convert_i8(target_as_index)
+            indexer = self._engine.get_indexer(target_as_index.values)
         else:
             # heterogeneous scalar index: defer elementwise to get_loc
             # (non-overlapping so get_loc guarantees scalar of KeyError)
             indexer = []
-            for key in target:
+            for key in target_as_index:
                 try:
                     loc = self.get_loc(key)
                 except KeyError:
@@ -947,21 +947,26 @@ def get_indexer(
     def get_indexer_non_unique(
         self, target: AnyArrayLike
     ) -> Tuple[np.ndarray, np.ndarray]:
-        target = ensure_index(target)
+        target_as_index = ensure_index(target)
 
-        # check that target IntervalIndex is compatible
-        if isinstance(target, IntervalIndex):
+        # check that target_as_index IntervalIndex is compatible
+        if isinstance(target_as_index, IntervalIndex):
             common_subtype = find_common_type(
-                [self.dtype.subtype, target.dtype.subtype]
+                [self.dtype.subtype, target_as_index.dtype.subtype]
             )
-            if self.closed != target.closed or is_object_dtype(common_subtype):
+            if self.closed != target_as_index.closed or is_object_dtype(common_subtype):
                 # different closed or incompatible subtype -> no matches
-                return np.repeat(-1, len(target)), np.arange(len(target))
+                return (
+                    np.repeat(-1, len(target_as_index)),
+                    np.arange(len(target_as_index)),
+                )
 
-        if is_object_dtype(target) or isinstance(target, IntervalIndex):
-            # target might contain intervals: defer elementwise to get_loc
+        if is_object_dtype(target_as_index) or isinstance(
+            target_as_index, IntervalIndex
+        ):
+            # target_as_index might contain intervals: defer elementwise to get_loc
             indexer, missing = [], []
-            for i, key in enumerate(target):
+            for i, key in enumerate(target_as_index):
                 try:
                     locs = self.get_loc(key)
                     if isinstance(locs, slice):
@@ -973,8 +978,10 @@ def get_indexer_non_unique(
                 indexer.append(locs)
             indexer = np.concatenate(indexer)
         else:
-            target = self._maybe_convert_i8(target)
-            indexer, missing = self._engine.get_indexer_non_unique(target.values)
+            target_as_index = self._maybe_convert_i8(target_as_index)
+            indexer, missing = self._engine.get_indexer_non_unique(
+                target_as_index.values
+            )
 
         return ensure_platform_int(indexer), ensure_platform_int(missing)
 
diff --git a/pandas/core/window.py b/pandas/core/window.py
index 20d5453cc43a2..184155293e684 100644
--- a/pandas/core/window.py
+++ b/pandas/core/window.py
@@ -240,7 +240,7 @@ def _prep_values(self, values: Optional[np.ndarray] = None) -> np.ndarray:
 
         return values
 
-    def _wrap_result(self, result, block=None, obj=None) -> FrameOrSeries:
+    def _wrap_result(self, result, block=None, obj=None):
         """
         Wrap a single result.
         """
diff --git a/setup.cfg b/setup.cfg
index 7f0062428c442..44b11a9db65e0 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -30,6 +30,7 @@ exclude =
     .eggs/*.py,
     versioneer.py,
     env  # exclude asv benchmark environments from linting
+    pandas/_typing.py
 
 [flake8-rst]
 bootstrap =
@@ -77,7 +78,9 @@ filterwarnings =
 
 [coverage:run]
 branch = False
-omit = */tests/*
+omit =
+     */tests/*
+     pandas/_typing.py
 plugins = Cython.Coverage
 
 [coverage:report]


Code review comments:
diff_hunk: @@ -1,34 +1,37 @@
 from pathlib import Path
-from typing import IO, AnyStr, TypeVar, Union
+from typing import IO, TYPE_CHECKING, AnyStr, TypeVar, Union
 
 import numpy as np
 
-from pandas._libs import Timestamp
-from pandas._libs.tslibs.period import Period
-from pandas._libs.tslibs.timedeltas import Timedelta
+# To prevent import cycles place any internal imports in the branch below
+# and use a string literal forward reference to it in subsequent types
+# https://mypy.readthedocs.io/en/latest/common_issues.html#import-cycles
+if TYPE_CHECKING:
+    from pandas._libs import Period, Timedelta, Timestamp
+
+    from pandas.core.arrays.base import ExtensionArray
+    from pandas.core.dtypes.dtypes import ExtensionDtype
+    from pandas.core.dtypes.generic import (
+        ABCDataFrame,
+        ABCExtensionArray,
+        ABCIndexClass,
+        ABCSeries,
+        ABCSparseSeries,
path: pandas/_typing.py
text: @user1:
can these abcs be safely removed now?

@author:
Just updated