TYP: Overload concat
State: closed

- [ ] Closes part of MarcoGorelli/pyladieslondon-sprints#29
- [ ] Adds typing.overload to: `concat` from https://github.com/pandas-dev/pandas/blob/master/pandas/core/reshape/concat.py

## `mypy` Setup and Sutput
```python
# t.py
import pandas as pd
from pandas._typing import Axis
from pandas.core.generic import NDFrame

axis: Axis

ndf1: NDFrame
ndf2: NDFrame

s1 = pd.Series
s2 = pd.Series

df1 = pd.DataFrame
df2 = pd.DataFrame

reveal_type(pd.concat([s1, s2]))
reveal_type(pd.concat([s1, s2], axis=0))
reveal_type(pd.concat([s1, s2], axis=1))
reveal_type(pd.concat([s1, s2], axis="index"))
reveal_type(pd.concat([s1, s2], axis="columns"))

reveal_type(pd.concat([df1, df2]))
reveal_type(pd.concat([df1, df2], axis=0))
reveal_type(pd.concat([df1, df2], axis=1))
reveal_type(pd.concat([df1, df2], axis="index"))
reveal_type(pd.concat([df1, df2], axis="columns"))

reveal_type(pd.concat([ndf1, ndf2]))
reveal_type(pd.concat([ndf1, ndf2], axis=0))
reveal_type(pd.concat([ndf1, ndf2], axis=1))
reveal_type(pd.concat([ndf1, ndf2], axis="index"))
reveal_type(pd.concat([ndf1, ndf2], axis="columns"))
```
which results in the following mypy output:
```bash
$ mypy t.py
  t.py:16: note: Revealed type is 'pandas.core.series.Series'
  t.py:17: note: Revealed type is 'pandas.core.series.Series'
  t.py:18: note: Revealed type is 'pandas.core.frame.DataFrame'
  t.py:19: note: Revealed type is 'pandas.core.series.Series'
  t.py:20: note: Revealed type is 'pandas.core.frame.DataFrame'
  t.py:22: note: Revealed type is 'pandas.core.frame.DataFrame'
  t.py:23: note: Revealed type is 'pandas.core.frame.DataFrame'
  t.py:24: note: Revealed type is 'pandas.core.frame.DataFrame'
  t.py:25: note: Revealed type is 'pandas.core.frame.DataFrame'
  t.py:26: note: Revealed type is 'pandas.core.frame.DataFrame'
  t.py:28: note: Revealed type is 'Union[pandas.core.frame.DataFrame, pandas.core.series.Series]'
  t.py:29: note: Revealed type is 'Union[pandas.core.frame.DataFrame, pandas.core.series.Series]'
  t.py:30: note: Revealed type is 'pandas.core.frame.DataFrame'
  t.py:31: note: Revealed type is 'Union[pandas.core.frame.DataFrame, pandas.core.series.Series]'
  t.py:32: note: Revealed type is 'pandas.core.frame.DataFrame'
```

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
The only issue left seems to be 
```bash
$ mypy t.py
...
  pandas/core/reshape/concat.py:58: error: Overloaded function signatures 1 and 3 overlap with incompatible return types  [misc]
...
```
but signature 1 has `objs: Iterable[DataFrame] | Mapping[Hashable, DataFrame],` while signature 3 has `objs: Iterable[Series] | Mapping[Hashable, Series],` so it should not overlap?

I played around with it but could not get the desired `mypy t.py` output without also encountering this error :(
Thanks for looking into this - I also wouldn't have expected them to overlap, so I'm not sure why this is happening, but I'll look into it later
Updated overloads as I just realized that a passed `objs` of type `NDFrame` should also return a `DataFrame` if `axis = Literal[1, "columns"]` and only else a `Union[pandas.core.frame.DataFrame, pandas.core.series.Series]`.
I can't make sense of the mypy error you're getting:
```python
from __future__ import annotations
from typing import overload, Iterable, Mapping, Hashable

@overload
def foo(myarg: Iterable[str] | Mapping[Hashable, str]) -> str: ...

@overload
def foo(myarg: Iterable[bool] | Mapping[Hashable, bool]) -> bool: ...

def foo(myarg) -> str | bool: ...
```
```python
$ mypy t.py
t.py:5: error: Overloaded function signatures 1 and 2 overlap with incompatible return types  [misc]
Found 1 error in 1 file (checked 1 source file)
```

I'll ask on StackOverflow, then open an issue in `mypy` if I don't get a response - will get back to you as soon as I have an answer!
OK, I see the issue: it's because `Mapping[TypeA, TypeB]` is an `Iterable[TypeA]`. E.g. the following type checks fine:
```python
from typing import Iterable


a: Iterable[str]
a = {'foo': 0}
```
So...gonna have to think about what to do about this

----

e.g.
```python
from typing import Hashable
from pandas import DataFrame


a: Hashable
a = DataFrame()
```
type checks fine.

So, if you pass a mapping in which the keys are `Series`, then it could match `Mapping[Hashable, DataFrame]`, but also `Iterable[Series]`
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
This is pending on https://github.com/pandas-dev/pandas/pull/41283 being merged first
> This is pending on #41283 being merged first

#41283 is now merged.
Hey @pckSF - if you fetch and merge upstream/master, then I'd expect the mypy issue to be fixed
> Hey @pckSF - if you fetch and merge upstream/master, then I'd expect the mypy issue to be fixed

Done, and the error is gone
nice - is this ready for review?
> nice - is this ready for review?

Thanks for the reminder, I always forget that I have them in preview...
Nice, looks good - the typing failure is unrelated, might need to merge again when that's fixed on master
> Nice, looks good - the typing failure is unrelated, might need to merge again when that's fixed on master

just merged #42614
```
Performing static analysis using mypy
pandas/core/arrays/categorical.py:2314: error: Incompatible types in assignment (expression has type "List[str]", variable has type "Index")  [assignment]
pandas/core/generic.py:5752: error: Incompatible return value type (got "DataFrame", expected "FrameOrSeries")  [return-value]
pandas/core/generic.py:6115: error: Incompatible return value type (got "DataFrame", expected "FrameOrSeries")  [return-value]
Found 3 errors in 2 files (checked 1301 source files)
Performing static analysis using mypy DONE
```

:thinking: maybe those overloads weren't redundant...I'm not sure why though, I'll look into it
> ```
> Performing static analysis using mypy
> pandas/core/arrays/categorical.py:2314: error: Incompatible types in assignment (expression has type "List[str]", variable has type "Index")  [assignment]
> pandas/core/generic.py:5752: error: Incompatible return value type (got "DataFrame", expected "FrameOrSeries")  [return-value]
> pandas/core/generic.py:6115: error: Incompatible return value type (got "DataFrame", expected "FrameOrSeries")  [return-value]
> Found 3 errors in 2 files (checked 1301 source files)
> Performing static analysis using mypy DONE
> ```
> 
> 🤔 maybe those overloads weren't redundant...I'm not sure why though, I'll look into it

Question regarding `pandas/core/generic.py:5752: error: Incompatible return value type (got "DataFrame", expected "FrameOrSeries")  [return-value]`:

```python
    def astype(
        self: FrameOrSeries, dtype, copy: bool_t = True, errors: str = "raise"
    ) -> FrameOrSeries:
    ...
    
    ...
    result = concat(results, axis=1, copy=False)
    result.columns = self.columns
    return result
```

given that it uses `concat` along `axis=1` shoudn't the returnd value be a `DataFrame`? 
(Similar case for `pandas/core/generic.py:6115: error: Incompatible return value type (got "DataFrame", expected "FrameOrSeries")  [return-value]`)
If I do
```diff
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
index 3b8458fac2..db6f213b3b 100644
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -5747,7 +5747,9 @@ class NDFrame(PandasObject, indexing.IndexingMixin):
             return self.copy()
 
         # GH 19920: retain column metadata after concat
+        reveal_type(results)
         result = concat(results, axis=1, copy=False)
+        reveal_type(result)
         result.columns = self.columns
         return result
```
then I get
```console
$ mypy pandas
pandas/core/arrays/categorical.py:2318: error: Incompatible types in assignment (expression has type "List[str]", variable has type "Index")  [assignment]
pandas/core/generic.py:5750: note: Revealed type is "builtins.list[Any]"
pandas/core/generic.py:5752: note: Revealed type is "pandas.core.frame.DataFrame"
pandas/core/generic.py:5754: error: Incompatible return value type (got "DataFrame", expected "FrameOrSeries")  [return-value]
pandas/core/generic.py:6117: error: Incompatible return value type (got "DataFrame", expected "FrameOrSeries")  [return-value]
Found 3 errors in 2 files (checked 1299 source files)
```
, so the revealed type at `pandas/core/generic.py:5752` looks correct. However, if I roll back to the previous commit, then that is revealed to be `Any`:
```console
$ mypy pandas
pandas/core/generic.py:5750: note: Revealed type is "builtins.list[Any]"
pandas/core/generic.py:5752: note: Revealed type is "Any"
```
So, it worked in the previous commit because the type checker was being disabled on the `result`.

Likewise, on `master`:
```console
$ mypy pandas
pandas/core/generic.py:5750: note: Revealed type is "builtins.list[Any]"
pandas/core/generic.py:5752: note: Revealed type is "Any"
```

So, what you have in the current commit seems correct, although I can't figure out why in your previous commit the type of `result` was not inferred correctly
OK, looks like we're running into https://github.com/python/mypy/issues/8354

So, I'd suggest you do something like
```diff
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
index 3b8458fac2..6205463aad 100644
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -5749,7 +5749,8 @@ class NDFrame(PandasObject, indexing.IndexingMixin):
         # GH 19920: retain column metadata after concat
         result = concat(results, axis=1, copy=False)
         result.columns = self.columns
-        return result
+        # https://github.com/python/mypy/issues/8354
+        return cast(FrameOrSeries, result)
 
     @final
     def copy(self: FrameOrSeries, deep: bool_t = True) -> FrameOrSeries:
@@ -6112,7 +6113,8 @@ class NDFrame(PandasObject, indexing.IndexingMixin):
                 for col_name, col in self.items()
             ]
             if len(results) > 0:
-                return concat(results, axis=1, copy=False)
+                # https://github.com/python/mypy/issues/8354
+                return cast(FrameOrSeries, concat(results, axis=1, copy=False))
             else:
                 return self.copy()
```

----

The other issue seems correct though, for that I'd suggest
```diff
diff --git a/pandas/core/arrays/categorical.py b/pandas/core/arrays/categorical.py
index 3fdb52a73d..57e7c0af52 100644
--- a/pandas/core/arrays/categorical.py
+++ b/pandas/core/arrays/categorical.py
@@ -2315,7 +2315,7 @@ class Categorical(NDArrayBackedExtensionArray, PandasObject, ObjectStringArrayMi
         from pandas.core.reshape.concat import concat
 
         result = concat([counts, freqs], axis=1)
-        result.columns = ["counts", "freqs"]
+        result.columns = Index(["counts", "freqs"])
         result.index.name = "categories"
 
         return result
```
thanks - now the CI is failing because of https://github.com/pandas-dev/pandas/issues/42716 , once that's fixed I'd like to think that this PR will be fine
> thanks - now the CI is failing because of #42716 , once that's fixed I'd like to think that this PR will be fine

should be good to fetch/merge upstream/master now
cc @simonjayhawkins in case you have comments
Thanks @pckSF !