CI: add pre-commit action, include pyupgrade
State: closed

xref this comment https://github.com/pandas-dev/pandas/issues/36426#issuecomment-694238629 by @TomAugspurger 

> I would recommend running the linting commands on CI through pre-commit. That way the versions in .pre-commit-config.yaml are used everywhere.

So, this PR does that.

It also adds pyupgrade as a pre-commit hook (xref #36450 ) and removes some unnecessary configurations from the `.pre-commit-config.yaml` file (all of these hooks already specify `language: python`, see e.g. https://github.com/psf/black/blob/master/.pre-commit-hooks.yaml )

TODO
----
update https://pandas.pydata.org/docs/development/contributing.html#code-standards (thanks Ali!)

Labels:
name: CI
description: Continuous Integration

Comments:
Might need to rework the code_check docs if we do this: https://pandas.pydata.org/docs/development/contributing.html#code-standards
Thanks. Happy to have these run in both code_checks and the pre-commit GH Action for a bit to make sure things are working properly.
@MarcoGorelli should pyupgrade be added to environment.yml?
> @MarcoGorelli should pyupgrade be added to environment.yml?

Not necessary, as pre-commit runs it a separate environment and that's the only place here pyupgrade is being used

https://github.com/pandas-dev/pandas/pull/36471/checks?check_run_id=1143999007 shows:
```
black....................................................................Passed
flake8...................................................................Passed
flake8-pyx...............................................................Passed
flake8-pxd...............................................................Passed
isort....................................................................Passed
pyupgrade................................................................Passed
```
pre-commit for the contributor is optional. (https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#pre-commit)

so as a reviewer, what's the process to recommend if a pyupgrade failure occurs. advise the contributor to install pyupgrade or does the output of the failed job tell you what needs to be done?
> pre-commit for the contributor is optional. (https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#pre-commit)
> 
> so as a reviewer, what's the process to recommend if a pyupgrade failure occurs. advise the contributor to install pyupgrade or does the output of the failed job tell you what needs to be done?

That's a really good point, and yes - it'll show you the diff, e.g. see here https://github.com/pymc-devs/pymc3/runs/1121338004 for an example from PyMC3, which shows:
```diff
nbqa-isort...............................................................Failed
- hook id: nbqa
- files were modified by this hook

Fixing /home/runner/work/pymc3/pymc3/docs/source/notebooks/GLM-logistic.ipynb

pre-commit hook(s) made changes.
If you are seeing this message in CI, reproduce locally with: `pre-commit run --all-files`.
To run `pre-commit` as part of git workflow, use `pre-commit install`.
All changes made by hooks:
diff --git a/docs/source/notebooks/GLM-logistic.ipynb b/docs/source/notebooks/GLM-logistic.ipynb
index 96c9d11..71d53e8 100644
--- a/docs/source/notebooks/GLM-logistic.ipynb
+++ b/docs/source/notebooks/GLM-logistic.ipynb
@@ -29,6 +29,9 @@
     }
    ],
    "source": [
+    "from collections import OrderedDict\n",
+    "from time import time\n",
+    "\n",
     "import matplotlib.pyplot as plt\n",
     "import numpy as np\n",
     "import pandas as pd\n",
@@ -37,10 +40,8 @@
     "import theano as thno\n",
     "import theano.tensor as T\n",
     "\n",
-    "from collections import OrderedDict\n",
     "from scipy import integrate\n",
     "from scipy.optimize import fmin_powell\n",
-    "from time import time\n",
     "\n",
     "print(\"Running on PyMC3 v{}\".format(pm.__version__))"
    ]
##[error]The process '/opt/hostedtoolcache/Python/3.8.5/x64/bin/pre-commit' failed with exit code 1
```
> @MarcoGorelli should pyupgrade be added to environment.yml?

Since all the other development tools (e.g., testing, typing and linting) are in environment.yml I think adding it makes sense, especially if it's also in CI (would probably minimize friction later when errors do occur). Thoughts @MarcoGorelli ?
Makes sense - have added that an pre-commit (since its the recommended way to run pyupgrade)
> Makes sense - have added that an pre-commit (since its the recommended way to run pyupgrade)

AFAIK pre-commit doesn't run out-of-the-box on Windows after using `pip install pre-commit`

you get `bash: pre-commit: command not found` in git bash and in anaconda prompt
```
'pre-commit' is not recognized as an internal or external command,
operable program or batch file.
```

so I think additional instructions/link to relevant docs could be included to lower the barrier to entry for new contributors.

:thinking: I have it working on my Windows laptop for work, pretty sure all I did was `pip install pre-commit` in the anaconda prompt. Is that not your experience? Could you try again with their latest version?

The docs already link to the pre-commit website https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#pre-commit , do you think more should be added (in terms of installation instructions)?
> 🤔 I have it working on my Windows laptop for work, pretty sure all I did was `pip install pre-commit` in the anaconda prompt. Is that not your experience? Could you try again with their latest version?

using `pip install -U pre-commit` updated from 2.6.0 -> 2.7.1 in the environment that I was using and solves `'pre-commit' is not recognized`

but creating a clean environment from environment.yml (from this PR)

```
(test) C:\Users\simon\pandas>conda list pre-commit
# packages in environment at C:\Users\simon\Anaconda3\envs\test:
#
# Name                    Version                   Build  Channel
pre-commit                2.7.1            py38h32f6830_0    conda-forge

(test) C:\Users\simon\pandas>pre-commit
[INFO] Initializing environment for https://github.com/PyCQA/isort.
[INFO] Initializing environment for https://github.com/asottile/pyupgrade.
[INFO] Installing environment for https://github.com/PyCQA/isort.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: command: ('C:\\Users\\simon\\Anaconda3\\envs\\test\\python.exe', '-mvirtualenv', 'C:\\Users\\simon\\.cache\\pre-commit\\repo6988w0ti\\py_env-default', '-p', 'C:\\Users\\simon\\Anaconda3\\envs\\test\\python.exe')
return code: 1
expected return code: 0
stdout:
    ModuleNotFoundError: No module named 'virtualenv.seed.via_app_data'

stderr: (none)
Check the log at C:\Users\simon\.cache\pre-commit\pre-commit.log
```

moving pre-commit to the pip section and creating a new environment from that seems to work out-the-box.

```
(test2) C:\Users\simon\pandas>conda list pre-commit
# packages in environment at C:\Users\simon\Anaconda3\envs\test2:
#
# Name                    Version                   Build  Channel
pre-commit                2.7.1                    pypi_0    pypi

(test2) C:\Users\simon\pandas>pre-commit
[WARNING] Unstaged files detected.
[INFO] Stashing unstaged files to C:\Users\simon\.cache\pre-commit\patch1600770739.
[INFO] Installing environment for https://github.com/PyCQA/isort.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/asottile/pyupgrade.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
black................................................(no files to check)Skipped
flake8...............................................(no files to check)Skipped
flake8-pyx...........................................(no files to check)Skipped
flake8-pxd...........................................(no files to check)Skipped
isort................................................(no files to check)Skipped
pyupgrade............................................(no files to check)Skipped
[INFO] Restored changes from C:\Users\simon\.cache\pre-commit\patch1600770739.
```

of course, this comparison is not very well controlled without caches etc being cleared and the first environment installing pre-commit using conda now works as well.

```
(test) C:\Users\simon\pandas>pre-commit
[WARNING] Unstaged files detected.
[INFO] Stashing unstaged files to C:\Users\simon\.cache\pre-commit\patch1600771578.
black................................................(no files to check)Skipped
flake8...............................................(no files to check)Skipped
flake8-pyx...........................................(no files to check)Skipped
flake8-pxd...........................................(no files to check)Skipped
isort................................................(no files to check)Skipped
pyupgrade............................................(no files to check)Skipped
[INFO] Restored changes from C:\Users\simon\.cache\pre-commit\patch1600771578.
```

and I'm can't remember what I did in the past to get pre-commit to work in conda environments that may have affected my settings/configuration.

so maybe just leave as is and see if any other contributors report issues. 
Cool, thanks for the detailed report
thanks @MarcoGorelli lgtm. @TomAugspurger @simonjayhawkins if ok pls merge.
Thanks!