API: Add pandas.api.typing
State: closed

- [x] closes #48577 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement
name: API Design
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Marking this as a draft until the discussion in the corresponding issue is resolved.
Can you merge main?
@phofl - done.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
This is missing the Styler, but that will take some refactoring in order to be able to import the Styler class without importing jina2. I plan to tackle in a (perhaps series of) followup(s).
> The following classes don't appear as a result in the documentation

They (DataFrameGroupBy / SeriesGroupBy) do appear in the API docs - I've updated the API docs to point at `pandas.api.typing`. Does that work?

> The following classes may need updates in the docs because we refer to types in a different package or without a package prefix

I think it's a great add to update the API docs to indicate the objects that are returned are in `pandas.api.typing` (i.e. what I did for DataFrameGroupBy / SeriesGroupBy in my last commit). But I don't think everywhere e.g. Resampler is mentioned we have to say `pandas.api.typing.Resampler`. For example, in timeseries.rst in the User Guide, we have:

```
Similar to the :ref:`aggregating API <basics.aggregate>`, :ref:`groupby API <groupby.aggregate>`, and the :ref:`window API <window.overview>`,
a ``Resampler`` can be selectively resampled.
```

I would think this instance doesn't need to be changed. Would you agree?


> > The following classes don't appear as a result in the documentation
> 
> They (DataFrameGroupBy / SeriesGroupBy) do appear in the API docs - I've updated the API docs to point at `pandas.api.typing`. Does that work?

Yes.  I missed that they were referenced, so adding the `pandas.typing` is what I was looking for. 

> 
> > The following classes may need updates in the docs because we refer to types in a different package or without a package prefix
> 
> I think it's a great add to update the API docs to indicate the objects that are returned are in `pandas.api.typing` (i.e. what I did for DataFrameGroupBy / SeriesGroupBy in my last commit). But I don't think everywhere e.g. Resampler is mentioned we have to say `pandas.api.typing.Resampler`. For example, in timeseries.rst in the User Guide, we have:
> 
> ```
> Similar to the :ref:`aggregating API <basics.aggregate>`, :ref:`groupby API <groupby.aggregate>`, and the :ref:`window API <window.overview>`,
> a ``Resampler`` can be selectively resampled.
> ```
> 
> I would think this instance doesn't need to be changed. Would you agree?

I think we have to look at them one by one.  Because when I read that text, I naturally ask "What is a `Resampler` ?"

Is this particular case, the docs never connect `df.resample()` with a `Resampler`.  So I think we can be better here, by saying something like:

```
The ``resample()`` method returns a ``pandas.typing.Resampler`` object.  Similar to the :ref:`aggregating API <basics.aggregate>`, :ref:`groupby API <groupby.aggregate>`, and the :ref:`window API <window.overview>`, a ``Resampler`` can be selectively resampled.
```
Thanks @Dr-Irv for the review - I think this is ready for another look.

> I think we have to look at them one by one. Because when I read that text, I naturally ask "What is a `Resampler` ?"
> 
> Is this particular case, the docs never connect `df.resample()` with a `Resampler`. So I think we can be better here, by saying something like:

I agree with this assessment, but isn't that out of the scope of this PR? In other words, this issue with the docs exists independently of wanting to put these objects in `pandas.api.typing`.

> 
> > I think we have to look at them one by one. Because when I read that text, I naturally ask "What is a `Resampler` ?"
> > Is this particular case, the docs never connect `df.resample()` with a `Resampler`. So I think we can be better here, by saying something like:
> 
> I agree with this assessment, but isn't that out of the scope of this PR? In other words, this issue with the docs exists independently of wanting to put these objects in `pandas.api.typing`.

Sure, but I'm guessing (hoping??) that the use of these classes is small in the user guides, which I think is the only place you have to worry about it.
LGTM.  Ping when green and I will approve/merge.
@simonjayhawkins - friendly ping
Thanks @simonjayhawkins for the review

> I think this could be coupled to https://github.com/pandas-dev/pandas/issues/27522, so as a follow-up or here if others think that the coupling is strong (e.g. need for api namespace)

The issue with the Styler is still a blocker here, and I plan to address it in a followup PR. Last I looked, it would require a bit of refactoring. Because of this and the amount of changes deprecating pandas.core would take, I'm quite resistant to including such work here.
> I'm quite resistant to including such work here.

sure. if we deprecate core it would enforce the usage of pandas[.api].typing but I guess changing the api docs for the public classes currently in core is a step in the right direction.
@simonjayhawkins - friendly ping
@Dr-Irv - friendly ping.
@rhshadrach I may be being a bit pedantic here, but now the docs don't correspond to what happens at runtime, and maybe you should add tests for this.

Consider this:
```python
>>> import pandas as pd
>>> df=pd.DataFrame({"x":[i for i in range(10)]}, index=pd.date_range("230101", periods=10))
>>> df
            x
2001-01-23  0
2001-01-24  1
2001-01-25  2
2001-01-26  3
2001-01-27  4
2001-01-28  5
2001-01-29  6
2001-01-30  7
2001-01-31  8
2001-02-01  9
>>> df.resample("2d")
<pandas.core.resample.DatetimeIndexResampler object at 0x000002425662CF40>
>>> type(df.rolling(2))
<class 'pandas.core.window.rolling.Rolling'>
```
So the docs are saying that `df.resample()` is returning a `pandas.api.typing.Resampler`, but that's not the user experience.  The docs say that `df.rolling()` returns a `pandas.api.typing.Rolling`, but that's not the user experience.  I think the same is true for the other types as well.

Now it is probably the case that the type `pandas.api.typing.Resampler` is the same as `pandas.core.resample.DatetimeIndex.Resampler` and that the type `pandas.api.typingRolling` is the same as `pandas.core.window.rolling.Rolling`, but shouldn't we have tests for each of these types you created to make sure that the types we document that we return do correspond to the actual types returned?

I'm probably suggesting this because when we change things in pandas-stubs to correspond to this change, we will end up creating those tests there (in a slightly different form)



Thanks @Dr-Irv - it's an interesting point. But is there something that can be done here? For example, within `DataFrame.groupby` replacing

    from pandas.core.groupby.generic import DataFrameGroupBy

with

    from pandas.api.typing import DataFrameGroupBy

I still see

```
print(type(pd.DataFrame({'a': [1, 1, 2], 'b': [3, 4, 5]}).groupby('a')))
# <class 'pandas.core.groupby.generic.DataFrameGroupBy'>
```
> it's an interesting point. But is there something that can be done here?

I don't think we want to through the effort of changing the returned classes, but I think you can create a test like this:
```python
gb = pd.DataFrame({'a': [1, 1, 2], 'b': [3, 4, 5]}).groupby('a')
assert isinstance(gb, pd.api.typing.DataFrameGroupBy)
```

That would at least test that you've correctly mapped the types in the `typing` module with the actual types.

> That would at least test that you've correctly mapped the types in the `typing` module with the actual types.

But there is no mapping here, yea? We import the object and then include said object in `__all__`. If there was a mapping, I would agree we should test it. As it stands, I think the only thing this would test is if there were two e.g. `DataFrameGroupBy`, that we've imported the right one. But if there were indeed two different `DataFrameGroupBy` objects, we'd have bigger problems I think :laughing:

I don't think such a test is warranted, and I'll also note we don't have tests like this for any existing `pandas.api` objects.
> But there is no mapping here, yea? We import the object and then include said object in `__all__`. If there was a mapping, I would agree we should test it. As it stands, I think the only thing this would test is if there were two e.g. `DataFrameGroupBy`, that we've imported the right one. But if there were indeed two different `DataFrameGroupBy` objects, we'd have bigger problems I think 😆
> 
> I don't think such a test is warranted, and I'll also note we don't have tests like this for any existing `pandas.api` objects.

Maybe "mapping" is the wrong word.

I'm thinking of the following.  Suppose that someone removed one of the types from `pandas/api/typing/__init__.py` or that you imported a class in that file, but forgot to include it in `__all__`.  The test would capture these kind of errors.

Just because we don't have tests for the other `pandas.api` objects, doesn't mean we can't start now....

> I'm thinking of the following. Suppose that someone removed one of the types from `pandas/api/typing/__init__.py` or that you imported a class in that file, but forgot to include it in `__all__`. The test would capture these kind of errors.

Ah, I was under the false impression that this was being done. Agreed we should be testing this! I've added the ones for api.typing and plan to do the others in a followup (I've opened #52688 to track).
Thanks @rhshadrach