Repository: python/mypy
Pull request: fix #16935 fix type of tuple[X,Y] expression
implement the mypy/checkexpr.py TODO: Specialize the callable for the type arguments
... so e.g. reveal_type(tuple[int, int]) gives expected
def (p0: tuple[builtins.int, builtins.int]) ->  tuple[builtins.int, builtins.int]

... rather than
def [_T_co] (typing.Iterable[_T_co`1] =) ->  builtins.tuple[_T_co`1, ...]

Fixes #16935


<!--
Checklist:
- Read the [Contributing Guidelines](https://github.com/python/mypy/blob/master/CONTRIBUTING.md)
- Add tests for all changed behaviour.
- If you can't add a test, please explain why and how you verified your changes work.
- Make sure CI passes.
- Please do not force push to the PR once it has been reviewed.
-->


Problem statement:
Tuple types don't work with higher level typing
mypy 1.8.0, python 3.10.10

So tuple types get matched and checked normally in simple situations like this:
```python
a: tuple[int, int] = (2, "a")
# Incompatible types in assignment ...
```

But imagine I want to define a function that takes a type and returns a value of that type (e.g. by deserializing something as the given type):
```python
from typing import Tuple, TypeVar, Type, List

_T = TypeVar("_T")

def structure(structure_into: Type[_T], obj: str) -> _T:
    raise NotImplementedError
```

This works with builtin types, or with generics like `List`:
```
reveal_type(structure(int, "a"))
# Revealed type is "builtins.int"
reveal_type(structure(List[int], "a"))
# Revealed type is "builtins.list[builtins.int]"
```

... but not with tuples:
```python
reveal_type(structure(Tuple[int, int], "a"))
# note: Revealed type is "Never"
# error: Argument 1 to "structure" has incompatible type "object"; expected "type[Never]"  [arg-type]
```

Trying to reveal the type of `Tuple` itself leads to strange results too (compared to `List[...]`):
```
reveal_type(Tuple[int, int])
# Revealed type is "builtins.object"
reveal_type(tuple[int, int])
# Revealed type is "def [_T_co] (typing.Iterable[_T_co`1] =) -> builtins.tuple[_T_co`1, ...]"
```

So the question is, why does `Tuple` get a special treatment, and is there a way to make it behave like other generics?

Hints from issue discussion:
Same issue with `UnionType` types (`T1 | T2 | ...`).
am having a go at this in https://github.com/urnest/mypy/tree/issue-16935

Commit message: fix #16935 fix type of tuple[X,Y] expression

... so e.g. reveal_type(tuple[int, int]) gives expected
def (p0: builtins.int, p1: builtins.int) ->
  tuple[builtins.int, builtins.int]

... rather than
def [_T_co] (typing.Iterable[_T_co`1] =) ->
  builtins.tuple[_T_co`1, ...]

Patch to review:
diff --git a/mypy/checkexpr.py b/mypy/checkexpr.py
index e8a2e501a452..11b00814d429 100644
--- a/mypy/checkexpr.py
+++ b/mypy/checkexpr.py
@@ -4836,8 +4836,21 @@ def apply_type_arguments_to_callable(
                 len(args) < min_arg_count or len(args) > len(tp.variables)
             ) and not has_type_var_tuple:
                 if tp.is_type_obj() and tp.type_object().fullname == "builtins.tuple":
-                    # TODO: Specialize the callable for the type arguments
-                    return tp
+                    # e.g. expression tuple[X, Y]
+                    # - want the type of the expression i.e. a function with that as its return type
+                    # - tp is type of tuple (note it won't have params as we are only called
+                    #   with generic callable type)
+                    # - tuple[X, Y]() takes a single arg that is a tuple containing an X and a Y
+                    return CallableType(
+                        [TupleType(list(args), self.chk.named_type("tuple"))],
+                        [ARG_POS],
+                        ["p1"],
+                        TupleType(list(args), self.chk.named_type("tuple")),
+                        self.chk.named_type("abc.ABCMeta"),
+                        name="tuple",
+                        definition=tp.definition,
+                        bound_args=tp.bound_args,
+                    )
                 self.msg.incompatible_type_application(
                     min_arg_count, len(tp.variables), len(args), ctx
                 )
diff --git a/test-data/unit/check-incremental.test b/test-data/unit/check-incremental.test
index a7f4fafc579e..0c5cd8737e65 100644
--- a/test-data/unit/check-incremental.test
+++ b/test-data/unit/check-incremental.test
@@ -3696,8 +3696,8 @@ cache_fine_grained = False
 [file mypy.ini.2]
 \[mypy]
 cache_fine_grained = True
-[rechecked _typeshed, a, builtins, typing]
-[stale _typeshed, a, builtins, typing]
+[rechecked _typeshed, a, abc, builtins, typing]
+[stale _typeshed, a, abc, builtins, typing]
 [builtins fixtures/tuple.pyi]
 
 [case testIncrementalPackageNameOverload]
@@ -3748,8 +3748,8 @@ Signature: 8a477f597d28d172789f06886806bc55
 [file b.py.2]
 # uh
 -- Every file should get reloaded, since the cache was invalidated
-[stale _typeshed, a, b, builtins, typing]
-[rechecked _typeshed, a, b, builtins, typing]
+[stale _typeshed, a, abc, b, builtins, typing]
+[rechecked _typeshed, a, abc, b, builtins, typing]
 [builtins fixtures/tuple.pyi]
 
 [case testIncrementalBustedFineGrainedCache2]
@@ -3761,8 +3761,8 @@ import b
 [file b.py.2]
 # uh
 -- Every file should get reloaded, since the settings changed
-[stale _typeshed, a, b, builtins, typing]
-[rechecked _typeshed, a, b, builtins, typing]
+[stale _typeshed, a, abc, b, builtins, typing]
+[rechecked _typeshed, a, abc, b, builtins, typing]
 [builtins fixtures/tuple.pyi]
 
 [case testIncrementalBustedFineGrainedCache3]
@@ -3777,8 +3777,8 @@ import b
 [file b.py.2]
 # uh
 -- Every file should get reloaded, since the cache was invalidated
-[stale _typeshed, a, b, builtins, typing]
-[rechecked _typeshed, a, b, builtins, typing]
+[stale _typeshed, a, abc, b, builtins, typing]
+[rechecked _typeshed, a, abc, b, builtins, typing]
 [builtins fixtures/tuple.pyi]
 
 [case testIncrementalWorkingFineGrainedCache]
diff --git a/test-data/unit/check-type-object-type-inference.test b/test-data/unit/check-type-object-type-inference.test
new file mode 100644
index 000000000000..baeca1e22ac7
--- /dev/null
+++ b/test-data/unit/check-type-object-type-inference.test
@@ -0,0 +1,41 @@
+[case testInferTupleType]
+# flags: --python-version 3.9
+from typing import TypeVar, Generic, Type
+from abc import abstractmethod
+
+T = TypeVar('T')
+class E(Generic[T]):
+    @abstractmethod
+    def e(self, t: T) -> str:
+        ...
+
+class F:
+    @abstractmethod
+    def f(self, tp: Type[T]) -> E[T]:
+        ...
+
+def g(f: F):
+    f.f(int).e(7)
+    f.f(tuple[int,str])
+    f.f(tuple[int,str]).e('x')  # E: Argument 1 to "e" of "E" has incompatible type "str"; expected "Tuple[int, str]"
+    f.f(tuple[int,str]).e( (7,8) )  # E: Argument 1 to "e" of "E" has incompatible type "Tuple[int, int]"; expected "Tuple[int, str]"
+    f.f(tuple[int,str]).e( (7,'x') )  # OK
+    reveal_type(f.f(tuple[int,str]).e)  # N: Revealed type is "def (t: Tuple[builtins.int, builtins.str]) -> builtins.str"
+
+def h(f: F):
+    f.f(int).e(7)
+    f.f(tuple)
+    f.f(tuple).e('y') # E: Argument 1 to "e" of "E" has incompatible type "str"; expected "Tuple[Any, ...]"
+    f.f(tuple).e( (8,'y') )  # OK
+    reveal_type(f.f(tuple).e)  # N: Revealed type is "def (t: builtins.tuple[Any, ...]) -> builtins.str"
+
+def i(f: F):
+    f.f(tuple[int,tuple[int,str]])
+    f.f(tuple[int,tuple[int,str]]).e('z')  # E: Argument 1 to "e" of "E" has incompatible type "str"; expected "Tuple[int, Tuple[int, str]]"
+    f.f(tuple[int,tuple[int,str]]).e( (8,9) )  # E: Argument 1 to "e" of "E" has incompatible type "Tuple[int, int]"; expected "Tuple[int, Tuple[int, str]]"
+    f.f(tuple[int,tuple[int,str]]).e( (17, (28, 29)) )  # E: Argument 1 to "e" of "E" has incompatible type "Tuple[int, Tuple[int, int]]"; expected "Tuple[int, Tuple[int, str]]"
+    f.f(tuple[int,tuple[int,str]]).e( (27,(28,'z')) )  # OK
+    reveal_type(f.f(tuple[int,tuple[int,str]]).e)  # N: Revealed type is "def (t: Tuple[builtins.int, Tuple[builtins.int, builtins.str]]) -> builtins.str"
+
+x = tuple[int,str][str]  # E: The type "Type[Tuple[Any, ...]]" is not generic and not indexable
+[builtins fixtures/tuple.pyi]
diff --git a/test-data/unit/fixtures/tuple.pyi b/test-data/unit/fixtures/tuple.pyi
index eb89de8c86ef..e00a1951d5f8 100644
--- a/test-data/unit/fixtures/tuple.pyi
+++ b/test-data/unit/fixtures/tuple.pyi
@@ -2,6 +2,7 @@
 
 import _typeshed
 from typing import Iterable, Iterator, TypeVar, Generic, Sequence, Optional, overload, Tuple, Type
+from abc import ABCMeta
 
 T = TypeVar("T")
 Tco = TypeVar('Tco', covariant=True)
diff --git a/test-data/unit/pythoneval.test b/test-data/unit/pythoneval.test
index b51a965c95da..f05901b13234 100644
--- a/test-data/unit/pythoneval.test
+++ b/test-data/unit/pythoneval.test
@@ -1828,7 +1828,6 @@ RHSAlias3: type = tuple[int, ...]
 WrongTypeElement = str | tuple[float, 1]  # Error
 WrongEllipsis = tuple[float, float, ...] | str  # Error
 
-# TODO: This should produce a fixed-length tuple
 reveal_type(tuple[int, str]((1, "x")))
 [out]
 _testTupleWithDifferentArgsPy310.py:15: note: Revealed type is "Union[builtins.str, Tuple[builtins.float, builtins.float, builtins.str]]"
@@ -1839,7 +1838,7 @@ _testTupleWithDifferentArgsPy310.py:19: note: Revealed type is "builtins.tuple[b
 _testTupleWithDifferentArgsPy310.py:20: note: Revealed type is "builtins.list[Tuple[builtins.int, builtins.str]]"
 _testTupleWithDifferentArgsPy310.py:26: error: Invalid type: try using Literal[1] instead?
 _testTupleWithDifferentArgsPy310.py:27: error: Unexpected "..."
-_testTupleWithDifferentArgsPy310.py:30: note: Revealed type is "builtins.tuple[builtins.object, ...]"
+_testTupleWithDifferentArgsPy310.py:29: note: Revealed type is "Tuple[builtins.int, builtins.str]"
 
 [case testEnumIterMetaInference]
 import socket


Code review comments:
diff_hunk: @@ -2,6 +2,7 @@
 
 import _typeshed
 from typing import Iterable, Iterator, TypeVar, Generic, Sequence, Optional, overload, Tuple, Type
+from abc import ABCMeta
path: test-data/unit/fixtures/tuple.pyi
text: @user1:
Is this from an earlier iteration of the PR?

It seems to be unused in the current code, and we probably don't want to add `abc` to all the tuple tests if we can help it.

@author:
This is the first iteration of the PR...

I'd happily remove it, just not sure how (and I'm no expert, so don't take that as "can't"!):

checkexpr.py line 4849 (that I added) needs it... without adding the tuple.pyi import I get unit test failures like:
data: /home/xju/mypy/test-data/unit/check-type-aliases.test:1016:
...
File "/home/xju/mypy/mypy/checkexpr.py", line 4849, in apply_type_arguments_to_callable
    self.chk.named_type("abc.ABCMeta"),
...
KeyError: 'abc'

... but maybe my checkexpr.py line 4849 is wrong or could be expressed differently? I chose that by looking at the result of CheckerPluginInterface.get_expression_type() for the expression "list[int]":
(Pdb) p expr_type
Overload(def () -> builtins.list[builtins.int], def (typing.Iterable[builtins.int]) -> builtins.list[builtins.int])
(Pdb) p expr_type.items[0]
def () -> builtins.list[builtins.int]
(Pdb) p expr_type.items[0].fallback
abc.ABCMeta
(Pdb) p type(expr_type.items[0].fallback)
<class 'mypy.types.Instance'>

... and somewhere I noticed that chk.named_type("abc.ABCMeta") appeared to produce exactly that:
(Pdb) self.chk.named_type("abc.ABCMeta")
abc.ABCMeta
(Pdb) type(self.chk.named_type("abc.ABCMeta"))
<class 'mypy.types.Instance'>

... thoughts/suggestions?

@user1:
Oh it's likely that I was mistaken here and it's actually needed, I'm also new to MyPy.

I suppose the import resolver determines which modules MyPy will parse, and that's why you hit an error without this line.

I didn't entirely understand why we need `ABCMeta` as a fallback either, but this is also related to me being new to MyPy.

@author:
mmm, I don't think ABCMeta is correct:
- in the list[int] case we want a function that returns a list[int], i.e. like def g() -> list[int]
- that is a specific instance of a function, so its fallback should be "function"
... I wonder what the fallback is for a plain function? Let's try:
def ff(): pass
... for that pdb mypy shows that CheckerPluginInterface.get_expression_type() returns:

(Pdb) p expr_type
def () -> Any
(Pdb) p type(expr_type)
<class 'mypy.types.CallableType'>
(Pdb) p expr_type.fallback
builtins.function

... that makes more sense, yes?

I will try that out... might be a day or two...

@author:
... ok after a bit more digging I believe ABCMeta is correct:
- ff's Callable above has fallback builtins.function, and that is correct for ff, which is a just a function
- but ff is not a mypy "type object" Callable, and the mypy type of expression "tuple" must be a type object Callable;  mypy uses Callable's fallback to distinguish (see mypy.types.Callable.is_type_obj() implementation)
- "int" gets fallback "builtins.type", I believe because its definition in builtins.pyi has no explicit base class
- "list" gets fallback "abc.ABCMeta", I believe because its definition in builtins.pyi has an abstract base class
- ... so "tuple" should get fallback "abc.ABCMeta" too because its definition in builtins.pyi has an abstract base class

... but I then realized that I can get the fallback from "tp" rather than looking it up, and doing that avoids needing to add abc.ABCMeta to test-data/unit/fixtures/tuple.pyi... so that is what I've done in commit [5bd1c4c]; I also removed the "p1" as param name, using None instead because that seems to be the convention for type object Callables