CI: Upgrade 'pyupgrade' (v2.7.4 --> v2.9.0)
State: closed

- [x] closes #39523 
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry
------
This PR upgrades `pyupgrade` and also includes style/formatting changes required after upgrading `pyupgrade`, namely, removing unused imports (`flake8`), and running `black` and `isort`.


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
I converted it to draft because of failing typing validation. Few other tests are also failing, possibly because I used `autoflake` to remove unused imports and it also ended up removing `pass` statements. Although I am not sure about the exact cause (Could be because of PEP604 issue @MarcoGorelli [mentioned](https://github.com/pandas-dev/pandas/pull/39538#pullrequestreview-580664985)). 
Will investigate further and push the fix. 
> @simonjayhawkins @Dr-Irv are we OK with making these PEP604 changes? I just saw that an issue was raised here [asottile/pyupgrade#387 (comment)](https://github.com/asottile/pyupgrade/issues/387#issuecomment-770875261) , I don't know if that applies to pandas (I haven't yet looked into "runtime introspection of type hints" and don't know if it's a concern which applies)

I'm not 100% sure here, but it seems that we don't want all these typing changes to happen because it would then break doing `mypy` checks in a Python 3.7, 3.8 and even 3.9 environment.  E.g., the `Union[x,y]` to `x | y` is in Python 3.10, and that syntax wouldn't be supported in earlier versions.

So using `--keep-runtime-typing` on `pyupgrade` is probably a better way to go, IMHO.
I don't think runtime introspection is an issue. I'm generally happy with these changes. This sort of change where changes are made across the codebase can cause autobackports to fail and we don't want to backport style changes. But we will need to take that hit a some point, so no reason imo not to do it sooner rather than later.
> E.g., the Union[x,y] to x | y is in Python 3.10, and that syntax wouldn't be supported in earlier versions.

Isn't it supported as long as you have `from __future__ import annotations`?
> I converted it to draft because of failing typing validation. Few other tests are also failing, possibly because I used `autoflake` to remove unused imports and it also ended up removing `pass` statements. Although I am not sure about the exact cause (Could be because of PEP604 issue @MarcoGorelli [mentioned](https://github.com/pandas-dev/pandas/pull/39538#pullrequestreview-580664985)).
> Will investigate further and push the fix.

There's a number of CI error ATM so these may be unrelated, will come back to this when CI's fixed
> Isn't it supported as long as you have `from __future__ import annotations`?

Maybe.  I guess we would have to test it in different python versions to be sure.  My concern with this typing stuff is that if our library goes out with the latest PEP typing enhancements, and then someone is using python 3.7 and a version of mypy on their application, whether everything will work properly.  I think we should stick to "least common denominator" on typing things, but I could be convinced otherwise.
That's a good point - maybe this should wait until 1.4.0 then, so that 1.3.x would have a modern enough version of mypy?
How should I proceed? 
- Should I close the PR for the time being? or
- Update the PR to pass `--keep-runtime-typing` to `pyupgrade` ?

Thanks!  
@gunjan-solanki based on the above (which I hadn't considered when I opened the issue, sorry!), I think we can go with the latter, i.e.
```yaml
-   repo: https://github.com/asottile/pyupgrade
    rev: v2.9.0
    hooks:
    -   id: pyupgrade
        args: [--py37-plus, --keep-runtime-typing]
```
This argument can be removed when 1.3.0 is out and autobackports will be a bit easier

If you do
```
git checkout upstream/master -- .
```
and then make the above changes to `.pre-commit-config.yaml`, then I think `pre-commit run pyupgrade -a` should pass without it reformatting anything (so, the diff should only show `.pre-commit-config.yaml`)
> and then make the above changes to `.pre-commit-config.yaml`, then I think `pre-commit run pyupgrade -a` should pass without it reformatting anything (so, the diff should only show `.pre-commit-config.yaml`)

@MarcoGorelli  Your guess is spot on. I tested it yesterday and can confirm no reformatting is necessary.  

> @gunjan-solanki based on the above (which I hadn't considered when I opened the issue, sorry!), I think we can go with the latter, i.e.

Not an issue. Happy to help. I will go ahead and update the PR :)