[PERF] Vectorize select_dtypes
State: closed

- [x] closes #28317
- [x] tests added / passed (no new tests added, test_dtypes.py, passes locally) 
- [x] passes `black pandas` (passed locally)
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff` (passed locally)
- [x] whatsnew entry
- [x] ASV Benchmark
- [x] final refactoring

I was able to run this code for 10million columns in less than a second, which should solve the mentioned problem sufficiently well. 

<details><summary>ASV Benchmark:</summary>
```
[ 75.00%] ··· frame_methods.SelectDtypes.time_select_dtypes                                                                                                                                               ok
[ 75.00%] ··· ======== =============
                 n
              -------- -------------
                 10      1.53±0.2ms
                100     1.48±0.03ms
                1000    1.50±0.06ms
               10000    1.89±0.04ms
               100000    6.55±0.2ms
              ======== =============

[ 75.00%] · For pandas commit ad9fe5d7 <master^2> (round 2/2):
[ 75.00%] ·· Building for conda-py3.6-Cython-matplotlib-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt...
[ 75.00%] ·· Benchmarking conda-py3.6-Cython-matplotlib-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[100.00%] ··· frame_methods.SelectDtypes.time_select_dtypes                                                                                                                                               ok
[100.00%] ··· ======== ============
                 n
              -------- ------------
                 10     2.04±0.2ms
                100     10.0±0.3ms
                1000     91.2±2ms
               10000     898±50ms
               100000   8.93±0.08s
              ======== ============

       before           after         ratio
     [ad9fe5d7]       [ef1b9a49]
     <master^2>       <28317-select-dtypes-performance>
-      10.0±0.3ms      1.48±0.03ms     0.15  frame_methods.SelectDtypes.time_select_dtypes(100)
-        91.2±2ms      1.50±0.06ms     0.02  frame_methods.SelectDtypes.time_select_dtypes(1000)
-        898±50ms      1.89±0.04ms     0.00  frame_methods.SelectDtypes.time_select_dtypes(10000)
-      8.93±0.08s       6.55±0.2ms     0.00  frame_methods.SelectDtypes.time_select_dtypes(100000)
```
I hope this is sufficient
</details>

Labels:
name: Performance
description: Memory or execution speed performance
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
A review would be nice, as well as maybe a better name for `filter_unique_dtypes_on`
Just the ASV benchmark with the most recent code versions
<details><summary>ASV Benchmark</summary>
```
· Creating environments
· Discovering benchmarks
·· Uninstalling from conda-py3.6-Cython-matplotlib-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt.
·· Building 16554c61 <28317-select-dtypes-performance> for conda-py3.6-Cython-matplotlib-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt............................................
·· Installing 16554c61 <28317-select-dtypes-performance> into conda-py3.6-Cython-matplotlib-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt..
· Running 2 total benchmarks (2 commits * 1 environments * 1 benchmarks)
[  0.00%] · For pandas commit ad9fe5d7 <master^2> (round 1/2):
[  0.00%] ·· Building for conda-py3.6-Cython-matplotlib-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt...
[  0.00%] ·· Benchmarking conda-py3.6-Cython-matplotlib-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 25.00%] ··· Running (frame_methods.SelectDtypes.time_select_dtypes--).
[ 25.00%] · For pandas commit 16554c61 <28317-select-dtypes-performance> (round 1/2):
[ 25.00%] ·· Building for conda-py3.6-Cython-matplotlib-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt...
[ 25.00%] ·· Benchmarking conda-py3.6-Cython-matplotlib-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 50.00%] ··· Running (frame_methods.SelectDtypes.time_select_dtypes--).
[ 50.00%] · For pandas commit 16554c61 <28317-select-dtypes-performance> (round 2/2):
[ 50.00%] ·· Benchmarking conda-py3.6-Cython-matplotlib-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 75.00%] ··· frame_methods.SelectDtypes.time_select_dtypes                                                                                                                                               ok
[ 75.00%] ··· ======== =============
                 n
              -------- -------------
                 10      1.37±0.1ms
                100     1.41±0.07ms
                1000     1.56±0.3ms
               10000     1.86±0.3ms
               100000    6.43±0.3ms
              ======== =============

[ 75.00%] · For pandas commit ad9fe5d7 <master^2> (round 2/2):
[ 75.00%] ·· Building for conda-py3.6-Cython-matplotlib-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt...
[ 75.00%] ·· Benchmarking conda-py3.6-Cython-matplotlib-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[100.00%] ··· frame_methods.SelectDtypes.time_select_dtypes                                                                                                                                               ok
[100.00%] ··· ======== ============
                 n
              -------- ------------
                 10     2.17±0.3ms
                100     10.3±0.2ms
                1000     95.5±4ms
               10000     919±30ms
               100000   9.46±0.03s
              ======== ============

       before           after         ratio
     [ad9fe5d7]       [16554c61]
     <master^2>       <28317-select-dtypes-performance>
-      2.17±0.3ms       1.37±0.1ms     0.63  frame_methods.SelectDtypes.time_select_dtypes(10)
-      10.3±0.2ms      1.41±0.07ms     0.14  frame_methods.SelectDtypes.time_select_dtypes(100)
-        95.5±4ms       1.56±0.3ms     0.02  frame_methods.SelectDtypes.time_select_dtypes(1000)
-        919±30ms       1.86±0.3ms     0.00  frame_methods.SelectDtypes.time_select_dtypes(10000)
-      9.46±0.03s       6.43±0.3ms     0.00  frame_methods.SelectDtypes.time_select_dtypes(100000)

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```
</details>
I'm not totally sure about the type annotations, specifically which type I should use for type. I think I tackled the other issues as well. 
FYI, some CI checks are failing.

I think this is a generally promising approach. Will take a closer look later.
Sorry, I was at work. CI is only failing due to the type annotation. Will try to figure it put. The last commit has an unfortunate name. I just was fixing a wrong export from `pandas._typing`.
Hmm, sorry, I don't get the typing annotations here:

```
pandas/core/frame.py:3492: error: Argument 1 to "tuple" has incompatible type "FrozenSet[Union[str, Any, ExtensionDtype]]"; expected "Iterable[Union[type, Tuple[Any, ...]]]"
```

and tuple is called with
```
tuple(dtypes_set)
```
and from the type hints at the beginning:
```
dtypes_set: FrozenSet[Dtype]
```

Btw, I recharged to a list comprehension since I'm only having one loop now. But of course I would change back. Any hints on the typing related stuff?
Seems weird - I would expect frozen set to be a subset of Iterable in the typing world. Any chance you see a bug report about it?
To file a bug report for the typing issue, not sure that this is a bug though, I'll try to create an easier example that shows the issue.

Right now I'm having:
```
from typing import FrozenSet, Iterable, Any, Union

from pandas._typing import ExtensionDtype, Dtype
import numpy as np

dtypes : FrozenSet[Dtype] = frozenset([np.numbers, np.float64])

tuple(dtypes)
``` 
which does not throw the Mypy error.

Or shall we just remove, the annotations and keep going?

I think just add a `# type: ignore` to the line that is failing. Unfortunate this is failing but prefer to keep the annotations as they are at the very least good documentation
Okay, so I think I created and MVE:

```
from typing import List

l: List[int] = []
issubclass(object, tuple(l))
issubclass(object, l)
```

gives using Mypy
```
line 4: error: Argument 1 to "tuple" has incompatible type "List[int]"; expected "Iterable[Union[type, Tuple[Any, ...]]]"
line 5: error: Argument 2 to "issubclass" has incompatible type "List[int]"; expected "Union[type, Tuple[Union[type, Tuple[Any, ...]], ...]]"
```
The second line is only to show what is subclass expects. We have the analogous error message
```
pandas/core/frame.py:3492: error: Argument 1 to "tuple" has incompatible type "FrozenSet[Union[str, Any, ExtensionDtype]]"; expected "Iterable[Union[type, Tuple[Any, ...]]]"
```
 with Dtype. Can anybody explain this?  Is this a bug? Adding above a line
```
tuple(l)
```
is unnoticed by Mypy.

If not, I'll follow the suggestion from above, and will just use #type: ignore
@datajanko pretty confusing error message but aren't those invocations wrong? Should the second argument to `isinstance` be a type and not an instance thereof?
Thinking through this I think the definition of `DType` in pandas._typing might be wrong and maybe that's actually throwing the error message. Right now it's `Union[str, np.dtype, "ExtensionDtype"]`, effectively allowing an instance of any of those types. But it might need to be `Type[Union[str, np.dtype, "ExtensionDtype"]]` to require the *class* instead of the instance (or something to that effect).

Tried that locally on your branch but that caused some more failures. For now I'd suggest just `type: ignoring" the failing line and we can open up a follow up issue to address the above
cc @simonjayhawkins on previous comment
@WillAyd 

What works is the following:
```
        unique_dtypes = self.dtypes.unique()
        extracted_dtypes = lambda y:list(filter(lambda x: issubclass(x.type, tuple(y)), unique_dtypes))

        if include:
            keep_these &= self.dtypes.isin(extracted_dtypes(include))

        if exclude:
            keep_these &= ~self.dtypes.isin(extracted_dtypes(exclude))

        return self.iloc[:, keep_these.values]
```

note that extracted_dtypes is function. Do we prefer to `def` it? (We should call it extract_dtypes probably)
Nevermind then - I don't necessarily find that more readable
@jreback ping since green
thanks @datajanko nice patch!