TYP: Typ part of python_parser
State: closed

@simonjayhawkins I am wondering about mypy with overloads.

The overload for ``_do_date_conversion`` could be more specific, e.g.
```
    @overload
    def _do_date_conversions(
        self,
        names: list[Scalar | tuple],
        data: dict[Scalar | tuple, ArrayLike] | dict[Scalar | tuple, np.ndarray],
    ) -> tuple[
        list[Scalar | tuple],
        dict[Scalar | tuple, ArrayLike] | dict[Scalar | tuple, np.ndarray],
    ]:
        ...
```

could be transformed to 

```
    @overload
    def _do_date_conversions(
        self,
        names: list[Scalar | tuple],
        data: dict[Scalar | tuple, ArrayLike],
    ) -> tuple[
        list[Scalar | tuple],
        dict[Scalar | tuple, ArrayLike],
    ]:
        ...

    @overload
    def _do_date_conversions(
        self,
        names: list[Scalar | tuple],
        data: dict[Scalar | tuple, np.ndarray],
    ) -> tuple[
        list[Scalar | tuple],
        dict[Scalar | tuple, np.ndarray],
    ]:
        ...
```

But in this case mypy complains about:
`` error: Overloaded function signature 3 will never be matched: signature 2's parameter type(s) are the same or broader  [misc]``

On the other side, if typing this only with
```
    @overload
    def _do_date_conversions(
        self,
        names: list[Scalar | tuple],
        data: dict[Scalar | tuple, ArrayLike],
    ) -> tuple[
        list[Scalar | tuple],
        dict[Scalar | tuple, ArrayLike],
    ]:
        ...
```

and passing a ``dict[Scalar | tuple, np.ndarray]`` mypy complains with 
```
pandas/io/parsers/python_parser.py:283: error: Argument 2 to "_do_date_conversions" of "ParserBase" has incompatible type "Dict[Union[Union[Union[str, int, float, bool], Union[Period, Timestamp, Timedelta, Any]], Tuple[Any, ...]], ndarray[Any, Any]]"; expected "Dict[Union[Union[Union[str, int, float, bool], Union[Period, Timestamp, Timedelta, Any]], Tuple[Any, ...]], Union[ExtensionArray, ndarray[Any, Any]]]"  [arg-type]
pandas/io/parsers/python_parser.py:283: note: "Dict" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
pandas/io/parsers/python_parser.py:283: note: Consider using "Mapping" instead, which is covariant in the value type
```

This looks inconsistent. I think the overload should accept the distinction between ``ArrayLike`` and ``np.ndarray`` with dicts or lists?

Technically we could use ``Mapping`` probably, but we would loose some strictness in this case and the object is alwyas a ``dict`` and we know if passing only np.ndarray we will get them in return.


On another topic: Should we use an alias for ``Scalar | tuple``? We need this throughout the code to indicate column names

Labels:
name: IO CSV
description: read_csv, to_csv
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
> The overload for `_do_date_conversion` could be more specific, e.g.

`ArrayLike` is defiend as `Union["ExtensionArray", np.ndarray]`. I think you don't need an overload:
```
_ArrayLikeT = TypeVar("_ArrayLikeT", bound=ArrayLike)

def _do_date_conversions(
    self,
    names: list[Scalar | tuple],
    data: dict[Scalar | tuple, _ArrayLikeT],
) -> tuple[
    list[Scalar | tuple],
    dict[Scalar | tuple, _ArrayLikeT],
]:
    ...
```
Unfortunately mypy does not accept this, if I want to put ``dict[Scalar | tuple, np.ndarray]`` in, because a dict is invariant. Minimal example:

```
def test(x: dict[str, int | float]) -> None:
    print(x)


x: dict[str, int] = {"a": 1}
test(x)
```

This raises 
```
error: Argument 1 to "test" has incompatible type "Dict[str, int]"; expected "Dict[str, Union[int, float]]"  [arg-type]
note: "Dict" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
note: Consider using "Mapping" instead, which is covariant in the value type
```
@simonjayhawkins would you mind having a look?
> This looks inconsistent. I think the overload should accept the distinction between `ArrayLike` and `np.ndarray` with dicts or lists?

probably because ArrayLike includes np.ndarray

> Technically we could use `Mapping` probably, but we would loose some strictness in this case and the object is alwyas a `dict` and we know if passing only np.ndarray we will get them in return.

see my loosely related comments above, wide types for function parameters and narrow types for return types, even though this seems counter-intuitive if passing the return types from one function onto another.

> On another topic: Should we use an alias for `Scalar | tuple`? We need this throughout the code to indicate column names

commented above. why is this not just Hashable?

Thanks for the review. Regarding the ArrayLike thing, this comes basically down to this: https://github.com/pandas-dev/pandas/pull/44406#issuecomment-967738612

Since this is not accepted, overload should allow a redefinition 

@simonjayhawkins adressed comments
i think this now conflicts can you merge master and can get in
Yep, most of the relevant stuff should be in now
merged and resolved conflicts