Repository: pandas-dev/pandas

Issue:
[TYPING] pandas.io.formats.format._binify annotation is inconsistent with  DataFrameFormatter.__init__
#### Problem description

`pd.io.formats.format._binify` defines `line_width` as `Union[np.int32, int]`:

https://github.com/pandas-dev/pandas/blob/39602e7d7c5b663696f5a6eca9e66e65483bc868/pandas/io/formats/format.py#L1893

however it is inconsistent with `DataFrameFormatter.__init__` 

https://github.com/pandas-dev/pandas/blob/39602e7d7c5b663696f5a6eca9e66e65483bc868/pandas/io/formats/format.py#L518

and `DataFrameFormatter._join_multiline`

https://github.com/pandas-dev/pandas/blob/39602e7d7c5b663696f5a6eca9e66e65483bc868/pandas/io/formats/format.py#L871

which take and pass as-is `Optional[int]`.

This leads to Mypy error:

```
pandas/io/formats/format.py:871: error: Argument 2 to "_binify" has incompatible type "Optional[int]"; expected "Union[int32, int]"
```

Providing default in [`_join_multiline`](https://github.com/pandas-dev/pandas/blob/39602e7d7c5b663696f5a6eca9e66e65483bc868/pandas/io/formats/format.py#L860), i.e.

```python
lwidth = self.line_width or DEFAULT_LINE_WIDTH
```
could be reasonable fix. One could also ignore [the call](https://github.com/pandas-dev/pandas/blob/39602e7d7c5b663696f5a6eca9e66e65483bc868/pandas/io/formats/format.py#L871):

```python
col_bins = _binify(col_widths, lwidth)  # type: ignore
```

but that leaves possible correctness problem.



#### Expected Output

Passing Mypy type check

#### Output of ``pd.show_versions()``

```
INSTALLED VERSIONS
------------------
commit           : bee17d5f4c99e6d1e451cf9a7b6a1780aa25988d
python           : 3.7.3.final.0
...
```
Additionally
```
mypy==0.730
```

CC @WillAyd 


Discussion:
Cool thanks for taking a look! @simonjayhawkins as well

Does the function actually fail if line_width is None?
> Does the function actually fail if line_width is None?

if i recall these types were from using MonkeyType. so if line_width is typed as Union[np.int32, int], then we have no tests that pass None.
> Does the function actually fail if line_width is None?

Haven't tested it, but I think it is safe to safe that it does fail on `>`

https://github.com/pandas-dev/pandas/blob/39602e7d7c5b663696f5a6eca9e66e65483bc868/pandas/io/formats/format.py#L1902

and

https://github.com/pandas-dev/pandas/blob/39602e7d7c5b663696f5a6eca9e66e65483bc868/pandas/io/formats/format.py#L1904


Does `np.int32` actually resolve to anything? The following yield `Any` for me:

```python
import numpy as np
a_np_int = np.int32(32)
reveal_type(a_np_int)
```

Wonder if we shouldn't just defer dealing with numpy types until they are actually annotated. Could just simplify this signature to be `int` for now
 @WillAyd that's probably unrelated, but yes, all np.int to be replaced with python int is subsequent pass.
i'm using 0.730, but not getting that particular mypy error show up yet on my branch.

since there are many mypy errors, probably best to determine if it is an actual bug in the code, since we have mypy on the ci, and for the current options and version we have a passing build.

typing errors will be resolved as more types are added and more strictness flags are enabled.

> i'm using 0.730, but not getting that particular mypy error show up yet on my branch.

Interesting... I downgraded Mypy to 0.730 and still get this one. Do you use any specific flags?

> Do you use any specific flags?

see #28339
changing to `def _binify(cols: List[int], line_width: int) -> List[int]:`

and I get `pandas\io\formats\format.py:873:40: error: Argument 2 to "_binify" has incompatible type "Optional[int]"; expected "int"  [arg-type]`

so the Any is swallowing the None.

@zero323 do you have numpy stubs in your environment?
> @WillAyd that's probably unrelated, but yes, all np.int to be replaced with python int is subsequent pass.

i was wrong.
> @zero323 do you have numpy stubs in your environment?

Yes, I do. 

__Edit__:

And once removed this error disappears hidden by some missing import.


so I guess that adding stubs for __all__ the missing imports becomes more important with PEP 561 compatibility since other users could have stubs for the other packages.
From that point of view the problem with PEP 561 is [that](https://www.python.org/dev/peps/pep-0561/#id20):

> Stubs or Python source manually put in the beginning of the path. Type checkers SHOULD provide this to allow the user complete control of which stubs to use, and to patch broken stubs/inline types from packages. In mypy the $MYPYPATH environment variable can be used for this.

So effective outcome is rather hard to predict.

Not to mention that all signs show that mature stubs for scientific stack are nowhere near.

One possible workaround is to keep dynamic mock stubs that cover the part of the 3rd party API that you're interested in, and use these for internal testing. I had some luck with such approach in pyspark-stubs (mocking Pandas), but exposure there is relatively small.

One way or another ignoring missing imports and `type: ignore` are nuclear options and it  is really hard to evaluate their real impact.

Resolving patch:
diff --git a/pandas/io/formats/format.py b/pandas/io/formats/format.py
--- a/pandas/io/formats/format.py
+++ b/pandas/io/formats/format.py
@@ -868,6 +868,8 @@ def _join_multiline(self, *args) -> str:
             np.array([self.adj.len(x) for x in col]).max() if len(col) > 0 else 0
             for col in strcols
         ]
+
+        assert lwidth is not None
         col_bins = _binify(col_widths, lwidth)
         nbins = len(col_bins)
 
@@ -1890,7 +1892,7 @@ def set_eng_float_format(accuracy: int = 3, use_eng_prefix: bool = False) -> Non
     set_option("display.column_space", max(12, accuracy + 9))
 
 
-def _binify(cols: List[np.int32], line_width: Union[np.int32, int]) -> List[int]:
+def _binify(cols: List[int], line_width: int) -> List[int]:
     adjoin_width = 1
     bins = []
     curr_width = 0


Test patch: