Repository: pandas-dev/pandas

Issue:
Typing Cleanup - Remove Blacklisted Items
#25844 introduced a blacklist of modules in the mypy.ini config file which currently throw errors when analyzed with mypy. We would ideally be able to whittle these down (save exclusion of the test folder(s), which I think is fine).

I'm not sure there is one easy way to split it up as the degree of difficulty may vary wildly. PRs from the community to clean up one to a few of the blacklisted modules at a time would certainly be welcome!

Here's the list of modules for reference:

- [X] pandas._config.config (#25899)
- [X] pandas._version (#25949)
- [X] pandas.compat (#26251)
- [X] pandas.compat.numpy.function
- [X] pandas.compat.pickle_compat (#25991)
- [X] pandas.core.accessor (#26146)
- [x] pandas.core.api
- [X] pandas.core.apply
- [X] pandas.core.arrays.array_  (#26065)
- [X] pandas.core.arrays.datetimelike  (#26065)
- [X] pandas.core.arrays.integer  (#26065)
- [X] pandas.core.arrays.interval (#26065)
- [X] pandas.core.arrays.numpy_ (#26065)
- [X] pandas.core.arrays.period (#26065)
- [X] pandas.core.arrays.sparse (#26065)
- [X] pandas.core.arrays.timedeltas (#26065)
- [x] pandas.core.base
- [x] pandas.core.computation.expr
- [x] pandas.core.computation.ops
- [x] pandas.core.computation.pytables
- [x] pandas.core.config (#26048)
- [x] pandas.core.config_init
- [X] pandas.core.dtypes.dtypes (#26028)
- [X] pandas.core.dtypes.missing (#26028)
- [X] pandas.core.frame
- [X] pandas.core.generic
- [X] pandas.core.groupby.generic (#26089)
- [X] pandas.core.groupby.groupby (#26089)
- [X] pandas.core.groupby.ops (#26089)
- [x] pandas.core.indexes.base (#26315)
- [x] pandas.core.indexes.datetimelike
- [ ] pandas.core.indexes.datetimes (#26315)
- [x] pandas.core.indexes.period (#26315)
- [x] pandas.core.indexes.timedeltas (#26315)
- [x] pandas.core.indexing
- [x] pandas.core.internals.blocks
- [x] pandas.core.ops
- [X] pandas.core.panel (DEPRECATED - do not update)
- [x] pandas.core.resample
- [X] pandas.core.reshape.concat (#26019)
- [x] pandas.core.reshape.merge
- [x] pandas.core.reshape.reshape
- [X] pandas.core.reshape.tile (#26019)
- [X] pandas.core.series
- [X] pandas.core.sparse.frame
- [X] pandas.core.strings (#25902)
- [x] pandas.core.util.hashing
- [x] pandas.core.window
- [x] pandas.io.clipboards (#26048)
- [x] pandas.io.feather_format (#25939)
- [x] pandas.io.formats.css (#25900)
- [x] pandas.io.html (#25939)
- [x] pandas.io.json.json (#25939)
- [x] pandas.io.json.normalize (#25939)
- [x] pandas.io.json.table_schema (#25939)
- [x] pandas.io.packers (#25939)
- [x] pandas.io.parquet (#25939)
- [x] pandas.io.pytables
- [x] pandas.io.stata (#25940)
- [X] pandas.plotting._core (#26021)
- [X] pandas.tseries.frequencies (#26014)
- [X] pandas.tseries.holiday (#26014)
- [X] pandas.tseries.offsets (#26014)
- [x] pandas.util._doctools
- [x] pandas.util.testing


Discussion:
@WillAyd I've been studying a few of these files and related contributions today and was wondering if there are any catches I need to know about when fixing these mypy violations. 
As an example: in array_.py:231 adding dtype = cast(ExtensionDtype, dtype)" seems the least intrusive. However, I saw that typing was [added to this file partially already](https://github.com/pandas-dev/pandas/pull/25829/files) , so was there a specific reason to leave this violation in? Thanks a lot! 
Have you merged master?
@WillAyd Yes, I'm on the latest upstream/master. 
Sorry misread comment. We’ve had a few things on one of which was simply to compat syntax to Py3 annotation

For the module you mentioned we did that but haven’t focused on actually fixing the errors. If you still see some and want to submit a PR would certainly be welcome
@WillAyd All clear then! I'll keep working on it. 
@WillAyd I've run into a few cases where we should use an old style type hint for a global variable to maintain pre-3.6 compatibility, but the appropriate hint is just too long for the line. I haven't been able to find any documented way to break a variable's type hint across lines or put it on another line. 

Any ideas?

For example in pandas/compat/numpy/function.py:105 we want something like 
```py
ARGSORT_DEFAULTS = OrderedDict()  # type: OrderedDict[str, Union[int, str, None]]
```
but that line is too long. We could replace the `Union` with `Any`, but that reduces the usefulness of type checking. In this particular case, it probably doesn't matter because that ARGSORT_DEFAULTS  gets fed directly into a classconstructor and is never touched again. But there may be other places in the code where accuracy is more important and the line length is an issue.
@gwrome https://github.com/python/mypy/issues/4511 might be of use

tldr: type alias, or backslash at end of line
@WillAyd, I think these are ready for removal from the list above, because they're no longer in the blacklist. I've included the PR if I could find it:

- pandas.compat (#26251)
- pandas.compat.numpy.function
- pandas.core.accessor (#26146)
- pandas.core.apply
- pandas.core.arrays.* (#26065)
- pandas.core.dtypes.* (#26028)
- pandas.core.frame
- pandas.core.generic
- pandas.core.groupby.* (#26089)
- pandas.core.reshape.concat (#26019)
- pandas.core.reshape.tile (#26019)
- pandas.core.series
- pandas.core.sparse.frame
Great - I’ve been meaning to update this so this helps a lot. 

Thanks for all of the help on these!

Sent from my iPhone

> On May 2, 2019, at 5:50 AM, Gregory Rome <notifications@github.com> wrote:
> 
> @WillAyd, I think these are ready for removal from the list above, because they're no longer in the blacklist. I've included the PR if I could find it:
> 
> pandas.compat (#26251)
> pandas.compat.numpy.function
> pandas.core.accessor (#26146)
> pandas.core.apply
> pandas.core.arrays.* (#26065)
> pandas.core.dtypes.* (#26028)
> pandas.core.frame
> pandas.core.generic
> pandas.core.groupby.* (#26089)
> pandas.core.reshape.concat (#26019)
> pandas.core.reshape.tile (#26019)
> pandas.core.series
> pandas.core.sparse.frame
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.

You’re welcome. Hoping to knock a couple more out this weekend. 

Resolving patch:
diff --git a/pandas/core/algorithms.py b/pandas/core/algorithms.py
--- a/pandas/core/algorithms.py
+++ b/pandas/core/algorithms.py
@@ -3,6 +3,7 @@
 intended for public consumption
 """
 from textwrap import dedent
+from typing import Dict
 from warnings import catch_warnings, simplefilter, warn
 
 import numpy as np
@@ -27,7 +28,7 @@
 
 from pandas.core import common as com
 
-_shared_docs = {}
+_shared_docs = {}  # type: Dict[str, str]
 
 
 # --------------- #
diff --git a/pandas/core/arrays/base.py b/pandas/core/arrays/base.py
--- a/pandas/core/arrays/base.py
+++ b/pandas/core/arrays/base.py
@@ -6,7 +6,7 @@
    without warning.
 """
 import operator
-from typing import Any, Callable, Optional, Sequence, Tuple, Union
+from typing import Any, Callable, Dict, Optional, Sequence, Tuple, Union
 
 import numpy as np
 
@@ -26,7 +26,7 @@
 
 _not_implemented_message = "{} does not implement {}."
 
-_extension_array_shared_docs = dict()
+_extension_array_shared_docs = dict()  # type: Dict[str, str]
 
 
 class ExtensionArray:
diff --git a/pandas/core/indexes/datetimelike.py b/pandas/core/indexes/datetimelike.py
--- a/pandas/core/indexes/datetimelike.py
+++ b/pandas/core/indexes/datetimelike.py
@@ -2,6 +2,7 @@
 Base and utility classes for tseries type pandas objects.
 """
 import operator
+from typing import Set
 import warnings
 
 import numpy as np
@@ -62,14 +63,17 @@ class DatetimeIndexOpsMixin(ExtensionOpsMixin):
     # DatetimeLikeArrayMixin assumes subclasses are mutable, so these are
     # properties there.  They can be made into cache_readonly for Index
     # subclasses bc they are immutable
-    inferred_freq = cache_readonly(DatetimeLikeArrayMixin.inferred_freq.fget)
-    _isnan = cache_readonly(DatetimeLikeArrayMixin._isnan.fget)
-    hasnans = cache_readonly(DatetimeLikeArrayMixin._hasnans.fget)
+    inferred_freq = cache_readonly(
+        DatetimeLikeArrayMixin.inferred_freq.fget)  # type: ignore
+    _isnan = cache_readonly(DatetimeLikeArrayMixin._isnan.fget)  # type: ignore
+    hasnans = cache_readonly(
+        DatetimeLikeArrayMixin._hasnans.fget)  # type: ignore
     _hasnans = hasnans  # for index / array -agnostic code
-    _resolution = cache_readonly(DatetimeLikeArrayMixin._resolution.fget)
-    resolution = cache_readonly(DatetimeLikeArrayMixin.resolution.fget)
+    _resolution = cache_readonly(
+        DatetimeLikeArrayMixin._resolution.fget)  # type: ignore
+    resolution = cache_readonly(
+        DatetimeLikeArrayMixin.resolution.fget)  # type: ignore
 
-    _box_values = ea_passthrough(DatetimeLikeArrayMixin._box_values)
     _maybe_mask_results = ea_passthrough(
         DatetimeLikeArrayMixin._maybe_mask_results)
     __iter__ = ea_passthrough(DatetimeLikeArrayMixin.__iter__)
@@ -130,11 +134,11 @@ def _ndarray_values(self):
     # Abstract data attributes
 
     @property
-    def values(self) -> np.ndarray:
+    def values(self):
         # Note: PeriodArray overrides this to return an ndarray of objects.
         return self._data._data
 
-    @property
+    @property  # type: ignore # https://github.com/python/mypy/issues/1362
     @Appender(DatetimeLikeArrayMixin.asi8.__doc__)
     def asi8(self):
         return self._data.asi8
@@ -761,9 +765,9 @@ class DatetimelikeDelegateMixin(PandasDelegate):
         boxed in an index, after being returned from the array
     """
     # raw_methods : dispatch methods that shouldn't be boxed in an Index
-    _raw_methods = set()
+    _raw_methods = set()  # type: Set[str]
     # raw_properties : dispatch properties that shouldn't be boxed in an Index
-    _raw_properties = set()
+    _raw_properties = set()  # type: Set[str]
     name = None
     _data = None
 


Test patch: