CI: run pyright with mypy
State: closed

The github action running the pre-commit checks does not install all necessary python packages for type checking. This triggered a few unnecessary `reportOverlappingOverload` by pyright because numpy was not installed.

Keep pyright in the pre-commit but disable it on the CI (SKIP=pyright). Instead, run pyright together with mypy in code_checks.sh.


Labels:
name: CI
description: Continuous Integration
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
thanks @twoertwein 
@twoertwein 

I have merged master and done the following:

```
$ pip install -r requirement-dev.txt
$ python setup.py build_ext -j 4
$ pre-commit install
$ pre-commit autoupdate
$ pre-commit run --all-files pyright
```
I get the following error:

```
Failed
- hook id: pyright
- exit code: 1

No configuration file found.
pyproject.toml file found at /Users/Darbyshire/pandas.
Loading pyproject.toml file at /Users/Darbyshire/pandas/pyproject.toml
Assuming Python platform Darwin
stubPath /Users/Darbyshire/pandas/typings is not a valid directory.
Searching for source files
Found 308 source files
/Users/Darbyshire/pandas/pandas/core/base.py
  /Users/Darbyshire/pandas/pandas/core/base.py:1251:9 - error: Overload 2 for "searchsorted" will never be used because its parameters overlap overload 1 (reportOverlappingOverload)
```

Am I missing something.. not familiar with pyright.


Which numpy version do you have installed? I think I got a similar error when I used an older version of numpy.

I assume that  `npt.ArrayLike` evaluates to `Any` with your numpy, which then let's pyright (correctly) complain that the second overload will never be reached.

Do you mind adding a `reveal_type(npt.ArrayLike)` and then running pyright.
If you can confirm that upgrading `numpy>1.21.0` fixes this issue, we should bump the numpy version in requirement-dev.txt
When downgrading numpy to 1.20.0 pyright cannot determine the type anymore:
info: Type of "npt._ScalarLike_co" is "Unknown"

I will create a PR to bump the version of numpy
I re installed my env, but have the same error with numpy 1.20.3. Trying to upgrade now..
> I re installed my env, but have the same error with numpy 1.20.3. Trying to upgrade now..

Sorry about that, `numpy>1.21.0` or locally adding `reportOverlappingOverload = false` to pyproject.toml should be workaround.
I upgraded to numpy 1.21.2, and incidentally got the error: 

```
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
numba 0.54.0 requires numpy<1.21,>=1.17, but you have numpy 1.21.2 which is incompatible.
```

If running `pre-commit run --all-files pyright` then I get `PASSED`. 

If using git hooks and committing from my IDE (pycharm) I get the following error (and I did add `reportOverlappingOverload = false` to `pyproject.toml` so I don't know why its still reporting it: weird anyway late here so I take another look tomorrow) 

```
Failed - hook id: pyright - exit code: 1 No configuration file found. pyproject.toml file found at /Users/Darbyshire/pandas. 
Loading pyproject.toml file at /Users/Darbyshire/pandas/pyproject.toml Assuming Python platform Darwin stubPath 
/Users/Darbyshire/pandas/typings is not a valid directory. Searching for source files Found 308 source files 
/Users/Darbyshire/pandas/pandas/core/base.py /Users/Darbyshire/pandas/pandas/core/base.py:1251:9 - error: Overload 2 for
 "searchsorted" will never be used because its parameters overlap overload 1 (reportOverlappingOverload) 
/Users/Darbyshire/pandas/pandas/core/common.py /Users/Darbyshire/pandas/pandas/core/common.py:398:5 - error: 
Overload 2 for "random_state" will never be used because its parameters overlap overload 1 (reportOverlappingOverload) 
/Users/Darbyshire/pandas/pandas/core/frame.py /Users/Darbyshire/pandas/pandas/core/frame.py:1370:9 - error: Overload 1
 for "dot" overlaps overload 2 and returns an incompatible type (reportOverlappingOverload) 
/Users/Darbyshire/pandas/pandas/core/array_algos/take.py /Users/Darbyshire/pandas/pandas/core/array_algos/take.py:44:5 -
 error: Overload 2 for "take_nd" will never be used because its parameters overlap overload 1 (reportOverlappingOverload) 
/Users/Darbyshire/pandas/pandas/core/arrays/_mixins.py /Users/Darbyshire/pandas/pandas/core/arrays/_mixins.py:216:9 - 
error: Overload 2 for "__getitem__" will never be used because its parameters overlap overload 1 (reportOverlappingOverload) 
/Users/Darbyshire/pandas/pandas/core/arrays/base.py /Users/Darbyshire/pandas/pandas/core/arrays/base.py:309:9 - error:
 Overload 2 for "__getitem__" will never be used because its parameters overlap overload 1 (reportOverlappingOverload)
```

with 
```
Package                           Version
--------------------------------- -----------
aiobotocore                       1.4.1
aiohttp                           3.7.4.post0
aioitertools                      0.8.0
alabaster                         0.7.12
appdirs                           1.4.4
appnope                           0.1.2
argon2-cffi                       21.1.0
asv                               0.4.2
async-timeout                     3.0.1
attrs                             21.2.0
Babel                             2.9.1
backcall                          0.2.0
backports.entry-points-selectable 1.1.0
beautifulsoup4                    4.10.0
black                             21.5b2
bleach                            4.1.0
blosc                             1.10.4
boto3                             1.17.106
botocore                          1.20.106
Bottleneck                        1.3.2
cachetools                        4.2.3
certifi                           2021.5.30
cffi                              1.14.6
cfgv                              3.3.1
cftime                            1.5.1
chardet                           4.0.0
charset-normalizer                2.0.6
click                             8.0.1
cloudpickle                       2.0.0
coverage                          5.5
cpplint                           1.5.5
cramjam                           2.4.0
cryptography                      3.4.8
cycler                            0.10.0
Cython                            0.29.24
dask                              2021.9.1
debugpy                           1.4.3
decorator                         5.1.0
defusedxml                        0.7.1
distlib                           0.3.3
docutils                          0.17.1
entrypoints                       0.3
et-xmlfile                        1.1.0
execnet                           1.9.0
fastparquet                       0.7.1
feedparser                        6.0.8
filelock                          3.1.0
flake8                            3.9.2
flake8-bugbear                    21.3.2
flake8-comprehensions             3.1.0
Flask                             2.0.1
fsspec                            2021.5.0
gcsfs                             2021.5.0
gitdb                             4.0.7
GitPython                         3.1.24
google-auth                       2.2.1
google-auth-oauthlib              0.4.6
greenlet                          1.1.2
html5lib                          1.1
hypothesis                        6.23.1
identify                          2.2.15
idna                              3.2
imagesize                         1.2.0
iniconfig                         1.1.1
ipykernel                         6.4.1
ipython                           7.28.0
ipython-genutils                  0.2.0
ipywidgets                        7.6.5
isort                             5.9.3
itsdangerous                      2.0.1
jedi                              0.18.0
Jinja2                            3.0.1
jmespath                          0.10.0
jsonschema                        3.2.0
jupyter-client                    7.0.5
jupyter-core                      4.8.1
jupyterlab-pygments               0.1.2
jupyterlab-widgets                1.0.2
kiwisolver                        1.3.2
llvmlite                          0.37.0
locket                            0.2.1
lxml                              4.6.3
Markdown                          3.3.4
MarkupSafe                        2.0.1
matplotlib                        3.4.3
matplotlib-inline                 0.1.3
mccabe                            0.6.1
mistune                           0.8.4
more-itertools                    8.10.0
moto                              2.2.8
multidict                         5.1.0
mypy                              0.910
mypy-extensions                   0.4.3
natsort                           7.1.1
nbclient                          0.5.4
nbconvert                         6.2.0
nbformat                          5.1.3
nbsphinx                          0.8.7
nest-asyncio                      1.5.1
nodeenv                           1.6.0
notebook                          6.4.4
numba                             0.54.0
numexpr                           2.7.3
numpy                             1.21.2
numpydoc                          1.1.0
oauthlib                          3.1.1
odfpy                             1.4.1
openpyxl                          3.0.9
packaging                         21.0
pandas                            1.3.3
pandas-dev-flaker                 0.2.0
pandoc                            1.1.0
pandocfilters                     1.5.0
parso                             0.8.2
partd                             1.2.0
pathspec                          0.9.0
patsy                             0.5.2
pep517                            0.11.0
pexpect                           4.8.0
pickleshare                       0.7.5
Pillow                            8.3.2
pip                               21.2.4
pip-tools                         6.3.0
platformdirs                      2.4.0
pluggy                            1.0.0
plumbum                           1.7.0
ply                               3.11
pre-commit                        2.15.0
prometheus-client                 0.11.0
prompt-toolkit                    3.0.20
ptyprocess                        0.7.0
py                                1.10.0
pyarrow                           5.0.0
pyasn1                            0.4.8
pyasn1-modules                    0.2.8
pycodestyle                       2.7.0
pycparser                         2.20
pydata-sphinx-theme               0.6.3
pyflakes                          2.3.1
Pygments                          2.10.0
pyparsing                         2.4.7
pyreadstat                        1.1.2
pyrsistent                        0.18.0
pytest                            6.2.5
pytest-asyncio                    0.15.1
pytest-cov                        2.12.1
pytest-forked                     1.3.0
pytest-instafail                  0.4.2
pytest-xdist                      2.4.0
python-dateutil                   2.8.2
python-snappy                     0.6.0
pytz                              2021.1
pyupgrade                         2.29.0
PyYAML                            5.4.1
pyzmq                             22.3.0
regex                             2021.9.24
requests                          2.26.0
requests-oauthlib                 1.3.0
responses                         0.14.0
rsa                               4.7.2
s3fs                              2021.5.0
s3transfer                        0.4.2
scipy                             1.7.1
seaborn                           0.11.2
Send2Trash                        1.8.0
setuptools                        57.4.0
sgmllib3k                         1.0.0
six                               1.16.0
smmap                             4.0.0
snowballstemmer                   2.1.0
sortedcontainers                  2.4.0
soupsieve                         2.2.1
Sphinx                            4.2.0
sphinx-panels                     0.6.0
sphinxcontrib-applehelp           1.0.2
sphinxcontrib-devhelp             1.0.2
sphinxcontrib-htmlhelp            2.0.0
sphinxcontrib-jsmath              1.0.1
sphinxcontrib-qthelp              1.0.3
sphinxcontrib-serializinghtml     1.1.5
SQLAlchemy                        1.4.25
statsmodels                       0.12.2
tables                            3.6.1
tabulate                          0.8.9
terminado                         0.12.1
testpath                          0.5.0
thrift                            0.15.0
tokenize-rt                       4.1.0
toml                              0.10.2
tomli                             1.2.1
toolz                             0.11.1
tornado                           6.1
traitlets                         5.1.0
types-PyMySQL                     1.0.2
types-python-dateutil             2.8.0
types-pytz                        2021.1.2
types-setuptools                  57.4.0
typing-extensions                 3.10.0.2
urllib3                           1.26.7
virtualenv                        20.8.1
wcwidth                           0.2.5
webencodings                      0.5.1
Werkzeug                          2.0.1
wheel                             0.37.0
widgetsnbextension                3.5.1
wrapt                             1.12.1
xarray                            0.18.2
xlrd                              2.0.1
XlsxWriter                        3.0.1
xlwt                              1.3.0
xmltodict                         0.12.0
yarl                              1.6.3
```
I'm sorry for this frustrating experience.

Based on your output pyright reads the pyproejct.toml file so it should ignore reportOverlappingOverload errors if you have `reportOverlappingOverload = false` in the pyright section.

You could run pyright only during the manual stage by adding `stages: [manual]` to the pyright hook in .pre-commit-config.yaml
> I'm sorry for this frustrating experience.

Don't worry about it. I'm learning and these things happen. If I set stages to manual as described then I get the following issue (I think because I am not including the changes to the pre-commit file in the final commit) (but I see you have a PR that will do this anyway)

```
07:23	Commit failed with error
			0 file committed, 1 file failed to commit: add version added
			[ERROR] Your pre-commit configuration is unstaged.
			`git add .pre-commit-config.yaml` to fix this.
```

> numba 0.54.0 requires numpy<1.21,>=1.17, but you have numpy 1.21.2 which is incompatible.

this is strange since numba 0.53.1 is ok with numpy 1.21.2 (with conda)