Repository: python/mypy

Issue:
Incorrect type inferred for **dict(iterable)
This was reported by @ahaven on our internal Slack; @ddfisher came up with the following minimal repro.

``` python
from typing import *

def f1(iterable: Iterable[Tuple[str, int]] = None) -> None:
    f2(**dict(iterable))  # <------- Error here

def f2(iterable: Iterable[Tuple[str, int]], **kw: Any) -> None:
    pass
```

We get the following mysterious error:

```
star.py: note: In function "f1":
star.py:4: error: Argument 1 to "dict" has incompatible type Iterable[Tuple[str, int]]; expected Iterable[...]
```

Some debugging shows that `Iterable[...]` in the error message stands for `Iterable[Tuple[Tuple[str, int], <something>]`, IOW it somehow takes `Dict[KT, VT]` and substitutes the `Tuple[str, int]` for just KT, even though the variant it chose from the overloaded dict constructor has `iterable: Iterable[Tuple[KT, VT]]` as its lone arg.

I think I've followed the trail into infer_type_arguments() at the very end of mypy/infer.py; it gets called with:

```
type_var_ids: [-1, -2]
template: builtins.dict[_KT`-1, _VT`-2]
actual: typing.Iterable[Tuple[builtins.str, builtins.int]]
```

After calling `infer_constraints(template, actual, SUBTYPE_OF)`, constraints is set to

```
[-1 <: Tuple[builtins.str, builtins.int], -1 :> Tuple[builtins.str, builtins.int]]
```

which seems to tell me that type var -1 (i.e. `_KT`) is both a subtype and a supertype of `Tuple[str, int]` while leaving type var -2 (i.e. `_VT`) unconstrained. The reported error follows from this I think.

I can also understand why these constraints are inferred -- it tries to match `Dict[KT, VT]` with `Iterable[Tuple[...]]` and of course `Dict[KT, VT]` inherits from `Iterable[KT]` so the tuple type ends up associated with KT -- but it appears confused because we should actually take the argument rather than the return type.

That's all I have for now.



Discussion:
Perhaps also interesting: if I comment out the call

``` python
                callee = self.infer_function_type_arguments_using_context(
                    callee, context)
```

from check_call() in mypy/checkexpr.py, I get a different error message:

```
star.py: note: In function "f1":
star.py:4: error: Argument 1 to "f2" has incompatible type Dict[str, int]; expected Iterable[Tuple[str, int]]
```

What's so interesting about this is that I get _exactly the same error_ (still with that infer_ call commented out) if I remove the `**` from the call to f2(), i.e. if I change line 4 to

``` python
    f2(dict(iterable))
```

That suggests to me that the original problem is really caused by incorrectly taking the type of the first argument to f2(), which also happens to be `Iterable[Tuple[str, int]]`, for the context of the expression `dict(iterable)`, despite the `**` in front of the latter.

Hm... Looks like either `infer_arg_types_in_context2()` is in need of skipping args of kind `ARG_STAR2`? Or `map_actuals_to_formals()` is doing a poor job if `kind == ARG_STAR2` (in the block asserting that).

I think `map_actuals_to_formals()` is correct -- it determines that in a call like `f2(**a)`, both of f2's arguments (`iterable` and `kw`) come from `a`, and for this it returns `[[0], [0]]`. But something that does something with the mapping is not properly taking the `**` in the call site into account.

Here's a simpler example showing that at least part of the problem seems to be the message formatting:

```
def f(a: int) -> None: pass
a = {'': 0}  # OK (!)
f(a)  # Error
f(**a)
b = {'': ''}
f(b)  # Error
f(**b)  # Error
```

All three lines with errors give the same error message, even though the reasons are actually different:

```
x.py:3: error: Argument 1 to "f" has incompatible type Dict[str, int]; expected "int"
x.py:6: error: Argument 1 to "f" has incompatible type Dict[str, str]; expected "int"
x.py:7: error: Argument 1 to "f" has incompatible type Dict[str, str]; expected "int"
```

But even if we were to fix this, we'd still have the problem that in a call like `f2(**f3(args))` the context for checking f3(args) is incorrectly taken to be the type of the first arg of `f2()` -- it should use `Dict[str, X]` where `X` is the type of the first arg of `f2()`. So there's still more than one bug here.


Resolving patch:
diff --git a/mypy/checkexpr.py b/mypy/checkexpr.py
--- a/mypy/checkexpr.py
+++ b/mypy/checkexpr.py
@@ -38,7 +38,7 @@
 
 # Type of callback user for checking individual function arguments. See
 # check_args() below for details.
-ArgChecker = Callable[[Type, Type, Type, int, int, CallableType, Context, MessageBuilder],
+ArgChecker = Callable[[Type, Type, int, Type, int, int, CallableType, Context, MessageBuilder],
                       None]
 
 
@@ -319,7 +319,7 @@ def infer_arg_types_in_context2(
         """Infer argument expression types using a callable type as context.
 
         For example, if callee argument 2 has type List[int], infer the
-        argument exprsession with List[int] type context.
+        argument expression with List[int] type context.
 
         Returns the inferred types of *actual arguments*.
         """
@@ -327,7 +327,7 @@ def infer_arg_types_in_context2(
 
         for i, actuals in enumerate(formal_to_actual):
             for ai in actuals:
-                if arg_kinds[ai] != nodes.ARG_STAR:
+                if arg_kinds[ai] not in (nodes.ARG_STAR, nodes.ARG_STAR2):
                     res[ai] = self.accept(args[ai], callee.arg_types[i])
 
         # Fill in the rest of the argument types.
@@ -604,7 +604,7 @@ def check_argument_types(self, arg_types: List[Type], arg_kinds: List[int],
                 # and **args this is the item type, not the collection type).
                 actual_type = get_actual_type(arg_type, arg_kinds[actual],
                                               tuple_counter)
-                check_arg(actual_type, arg_type,
+                check_arg(actual_type, arg_type, arg_kinds[actual],
                           callee.arg_types[i],
                           actual + 1, i + 1, callee, context, messages)
 
@@ -618,11 +618,12 @@ def check_argument_types(self, arg_types: List[Type], arg_kinds: List[int],
                         actual_type = get_actual_type(arg_type,
                                                       arg_kinds[actual],
                                                       tuple_counter)
-                        check_arg(actual_type, arg_type,
+                        check_arg(actual_type, arg_type, arg_kinds[actual],
                                   callee.arg_types[i],
                                   actual + 1, i + 1, callee, context, messages)
 
     def check_arg(self, caller_type: Type, original_caller_type: Type,
+                  caller_kind: int,
                   callee_type: Type, n: int, m: int, callee: CallableType,
                   context: Context, messages: MessageBuilder) -> None:
         """Check the type of a single argument in a call."""
@@ -632,7 +633,7 @@ def check_arg(self, caller_type: Type, original_caller_type: Type,
             messages.deleted_as_rvalue(caller_type, context)
         elif not is_subtype(caller_type, callee_type):
             messages.incompatible_argument(n, m, callee, original_caller_type,
-                                           context)
+                                           caller_kind, context)
 
     def overload_call_target(self, arg_types: List[Type], arg_kinds: List[int],
                              arg_names: List[str],
@@ -713,7 +714,7 @@ def erased_signature_similarity(self, arg_types: List[Type], arg_kinds: List[int
 
         similarity = 2
 
-        def check_arg(caller_type: Type, original_caller_type: Type,
+        def check_arg(caller_type: Type, original_caller_type: Type, caller_kind: int,
                       callee_type: Type, n: int, m: int, callee: CallableType,
                       context: Context, messages: MessageBuilder) -> None:
             nonlocal similarity
@@ -747,7 +748,7 @@ def match_signature_types(self, arg_types: List[Type], arg_kinds: List[int],
                                                   lambda i: arg_types[i])
         ok = True
 
-        def check_arg(caller_type: Type, original_caller_type: Type,
+        def check_arg(caller_type: Type, original_caller_type: Type, caller_kind: int,
                       callee_type: Type, n: int, m: int, callee: CallableType,
                       context: Context, messages: MessageBuilder) -> None:
             nonlocal ok
diff --git a/mypy/messages.py b/mypy/messages.py
--- a/mypy/messages.py
+++ b/mypy/messages.py
@@ -13,7 +13,10 @@
     Type, CallableType, Instance, TypeVarType, TupleType, UnionType, Void, NoneTyp, AnyType,
     Overloaded, FunctionLike, DeletedType
 )
-from mypy.nodes import TypeInfo, Context, MypyFile, op_methods, FuncDef, reverse_type_aliases
+from mypy.nodes import (
+    TypeInfo, Context, MypyFile, op_methods, FuncDef, reverse_type_aliases,
+    ARG_STAR, ARG_STAR2
+)
 
 
 # Constants that represent simple type checker error message, i.e. messages
@@ -400,7 +403,7 @@ def untyped_function_call(self, callee: CallableType, context: Context) -> Type:
         return AnyType()
 
     def incompatible_argument(self, n: int, m: int, callee: CallableType, arg_type: Type,
-                              context: Context) -> None:
+                              arg_kind: int, context: Context) -> None:
         """Report an error about an incompatible argument type.
 
         The argument type is arg_type, argument number is n and the
@@ -465,6 +468,10 @@ def incompatible_argument(self, n: int, m: int, callee: CallableType, arg_type:
             except IndexError:  # Varargs callees
                 expected_type = callee.arg_types[-1]
             arg_type_str, expected_type_str = self.format_distinctly(arg_type, expected_type)
+            if arg_kind == ARG_STAR:
+                arg_type_str = '*' + arg_type_str
+            elif arg_kind == ARG_STAR2:
+                arg_type_str = '**' + arg_type_str
             msg = 'Argument {} {}has incompatible type {}; expected {}'.format(
                 n, target, arg_type_str, expected_type_str)
         self.fail(msg, context)
diff --git a/mypy/myunit/__init__.py b/mypy/myunit/__init__.py
--- a/mypy/myunit/__init__.py
+++ b/mypy/myunit/__init__.py
@@ -228,7 +228,7 @@ def main(args: List[str] = None) -> None:
             patterns.append(a)
         else:
             sys.exit('Usage: python -m mypy.myunit [-v] [-q] [-u | -i]'
-                    + ' -m test.module [-m test.module ...] [filter ...]')
+                    + ' -m mypy.test.module [-m mypy.test.module ...] [filter ...]')
         i += 1
     if len(patterns) == 0:
         patterns.append('*')


Test patch:
diff --git a/mypy/test/data/check-inference-context.test b/mypy/test/data/check-inference-context.test
--- a/mypy/test/data/check-inference-context.test
+++ b/mypy/test/data/check-inference-context.test
@@ -721,3 +721,12 @@ def g(y: object) -> None: pass
 a = [1]
 g(f(a))
 [builtins fixtures/list.py]
+
+[case testStar2Context]
+from typing import Any, Dict, Tuple, Iterable
+def f1(iterable: Iterable[Tuple[str, Any]] = None) -> None:
+    f2(**dict(iterable))
+def f2(iterable: Iterable[Tuple[str, Any]], **kw: Any) -> None:
+    pass
+[builtins fixtures/dict.py]
+[out]
diff --git a/mypy/test/data/check-kwargs.test b/mypy/test/data/check-kwargs.test
--- a/mypy/test/data/check-kwargs.test
+++ b/mypy/test/data/check-kwargs.test
@@ -205,8 +205,8 @@ d = None # type: Dict[str, A]
 f(**d)
 f(x=A(), **d)
 d2 = None # type: Dict[str, B]
-f(**d2)        # E: Argument 1 to "f" has incompatible type Dict[str, B]; expected "A"
-f(x=A(), **d2) # E: Argument 2 to "f" has incompatible type Dict[str, B]; expected "A"
+f(**d2)        # E: Argument 1 to "f" has incompatible type **Dict[str, B]; expected "A"
+f(x=A(), **d2) # E: Argument 2 to "f" has incompatible type **Dict[str, B]; expected "A"
 class A: pass
 class B: pass
 [builtins fixtures/dict.py]
@@ -226,7 +226,7 @@ def f(a: 'A', b: 'B') -> None: pass
 d = None # type: Dict[str, Any]
 f(**d)
 d2 = None # type: Dict[str, A]
-f(**d2) # E: Argument 1 to "f" has incompatible type Dict[str, A]; expected "B"
+f(**d2) # E: Argument 1 to "f" has incompatible type **Dict[str, A]; expected "B"
 class A: pass
 class B: pass
 [builtins fixtures/dict.py]
@@ -286,3 +286,22 @@ f(1, 2) # E: Argument 2 to "f" has incompatible type "int"; expected "str"
 f(1, y=1) # E: Argument 2 to "f" has incompatible type "int"; expected "A"
 f(1, z=1) # E: Argument 2 to "f" has incompatible type "int"; expected "B"
 [builtins fixtures/dict.py]
+
+[case testCallsWithStars]
+def f(a: int) -> None:
+    pass
+
+s = ('',)
+f(*s) # E: Argument 1 to "f" has incompatible type *"Tuple[str]"; expected "int"
+
+a = {'': 0}
+f(a) # E: Argument 1 to "f" has incompatible type Dict[str, int]; expected "int"
+f(**a) # okay
+
+b = {'': ''}
+f(b) # E: Argument 1 to "f" has incompatible type Dict[str, str]; expected "int"
+f(**b) # E: Argument 1 to "f" has incompatible type **Dict[str, str]; expected "int"
+
+c = {0: 0}
+f(**c) # E: Keywords must be strings
+[builtins fixtures/dict.py]
diff --git a/mypy/test/data/check-varargs.test b/mypy/test/data/check-varargs.test
--- a/mypy/test/data/check-varargs.test
+++ b/mypy/test/data/check-varargs.test
@@ -109,7 +109,7 @@ it1 = None  # type: Iterable[int]
 it2 = None  # type: Iterable[str]
 def f(*x: int) -> None: pass
 f(*it1)
-f(*it2) # E: Argument 1 to "f" has incompatible type Iterable[str]; expected "int"
+f(*it2) # E: Argument 1 to "f" has incompatible type *Iterable[str]; expected "int"
 [builtins fixtures/for.py]
 
 
@@ -190,7 +190,7 @@ class A: pass
 class B: pass
 [builtins fixtures/list.py]
 [out]
-main:7: error: Argument 1 to "f" has incompatible type List[A]; expected "B"
+main:7: error: Argument 1 to "f" has incompatible type *List[A]; expected "B"
 
 [case testCallingWithTupleVarArgs]
 
@@ -199,9 +199,9 @@ b = None # type: B
 c = None # type: C
 cc = None # type: CC
 
-f(*(a, b, b)) # E: Argument 1 to "f" has incompatible type "Tuple[A, B, B]"; expected "C"
-f(*(b, b, c)) # E: Argument 1 to "f" has incompatible type "Tuple[B, B, C]"; expected "A"
-f(a, *(b, b)) # E: Argument 2 to "f" has incompatible type "Tuple[B, B]"; expected "C"
+f(*(a, b, b)) # E: Argument 1 to "f" has incompatible type *"Tuple[A, B, B]"; expected "C"
+f(*(b, b, c)) # E: Argument 1 to "f" has incompatible type *"Tuple[B, B, C]"; expected "A"
+f(a, *(b, b)) # E: Argument 2 to "f" has incompatible type *"Tuple[B, B]"; expected "C"
 f(b, *(b, c)) # E: Argument 1 to "f" has incompatible type "B"; expected "A"
 f(*(a, b))    # E: Too few arguments for "f"
 f(*(a, b, c, c)) # E: Too many arguments for "f"
@@ -259,26 +259,26 @@ class A: pass
 class B: pass
 [builtins fixtures/list.py]
 [out]
-main:3: error: Argument 1 to "f" has incompatible type List[A]; expected "B"
-main:4: error: Argument 2 to "f" has incompatible type List[A]; expected "B"
+main:3: error: Argument 1 to "f" has incompatible type *List[A]; expected "B"
+main:4: error: Argument 2 to "f" has incompatible type *List[A]; expected "B"
 main:5: error: Argument 1 to "f" has incompatible type "B"; expected "A"
 main:6: error: Argument 2 to "f" has incompatible type "A"; expected "B"
-main:7: error: Argument 3 to "f" has incompatible type List[A]; expected "B"
+main:7: error: Argument 3 to "f" has incompatible type *List[A]; expected "B"
 main:8: error: Argument 1 to "f" has incompatible type "B"; expected "A"
-main:9: error: Argument 1 to "g" has incompatible type List[B]; expected "A"
+main:9: error: Argument 1 to "g" has incompatible type *List[B]; expected "A"
 
 [case testCallingVarArgsFunctionWithTupleVarArgs]
 
 a, b, c, cc = None, None, None, None # type: (A, B, C, CC)
 
-f(*(b, b, b))   # E: Argument 1 to "f" has incompatible type "Tuple[B, B, B]"; expected "A"
-f(*(a, a, b))   # E: Argument 1 to "f" has incompatible type "Tuple[A, A, B]"; expected "B"
-f(*(a, b, a))   # E: Argument 1 to "f" has incompatible type "Tuple[A, B, A]"; expected "B"
-f(a, *(a, b))   # E: Argument 2 to "f" has incompatible type "Tuple[A, B]"; expected "B"
+f(*(b, b, b))   # E: Argument 1 to "f" has incompatible type *"Tuple[B, B, B]"; expected "A"
+f(*(a, a, b))   # E: Argument 1 to "f" has incompatible type *"Tuple[A, A, B]"; expected "B"
+f(*(a, b, a))   # E: Argument 1 to "f" has incompatible type *"Tuple[A, B, A]"; expected "B"
+f(a, *(a, b))   # E: Argument 2 to "f" has incompatible type *"Tuple[A, B]"; expected "B"
 f(b, *(b, b))   # E: Argument 1 to "f" has incompatible type "B"; expected "A"
 f(b, b, *(b,))  # E: Argument 1 to "f" has incompatible type "B"; expected "A"
 f(a, a, *(b,))  # E: Argument 2 to "f" has incompatible type "A"; expected "B"
-f(a, b, *(a,))  # E: Argument 3 to "f" has incompatible type "Tuple[A]"; expected "B"
+f(a, b, *(a,))  # E: Argument 3 to "f" has incompatible type *"Tuple[A]"; expected "B"
 f(*())          # E: Too few arguments for "f"
 f(*(a, b, b))
 f(a, *(b, b))
@@ -322,7 +322,7 @@ from typing import List
 aa = None # type: List[A]
 ab = None # type: List[B]
 
-g(*aa) # E: Argument 1 to "g" has incompatible type List[A]; expected "B"
+g(*aa) # E: Argument 1 to "g" has incompatible type *List[A]; expected "B"
 f(*aa)
 f(*ab)
 g(*ab)
@@ -359,9 +359,9 @@ class B: pass
 [builtins fixtures/list.py]
 [out]
 main:3: error: Too few arguments for "f"
-main:4: error: Argument 2 to "f" has incompatible type List[A]; expected "B"
-main:5: error: Argument 3 to "f" has incompatible type List[A]; expected "B"
-main:6: error: Argument 1 to "f" has incompatible type "Tuple[A, A, B]"; expected "B"
+main:4: error: Argument 2 to "f" has incompatible type *List[A]; expected "B"
+main:5: error: Argument 3 to "f" has incompatible type *List[A]; expected "B"
+main:6: error: Argument 1 to "f" has incompatible type *"Tuple[A, A, B]"; expected "B"
 
 [case testVarArgsAfterKeywordArgInCall1]
 def f(x: int, y: str) -> None: pass
@@ -369,7 +369,7 @@ f(x=1, *[2])
 [builtins fixtures/list.py]
 [out]
 main:2: error: "f" gets multiple values for keyword argument "x"
-main:2: error: Argument 2 to "f" has incompatible type List[int]; expected "str"
+main:2: error: Argument 2 to "f" has incompatible type *List[int]; expected "str"
 
 [case testVarArgsAfterKeywordArgInCall2]
 def f(x: int, y: str) -> None: pass
@@ -377,7 +377,7 @@ f(y='x', *[1])
 [builtins fixtures/list.py]
 [out]
 main:2: error: "f" gets multiple values for keyword argument "y"
-main:2: error: Argument 2 to "f" has incompatible type List[int]; expected "str"
+main:2: error: Argument 2 to "f" has incompatible type *List[int]; expected "str"
 
 [case testVarArgsAfterKeywordArgInCall3]
 def f(x: int, y: str) -> None: pass
@@ -469,11 +469,11 @@ class A: pass
 class B: pass
 [builtins fixtures/list.py]
 [out]
-main:6: error: Argument 1 to "f" has incompatible type List[A]; expected "B"
-main:7: error: Argument 1 to "f" has incompatible type List[A]; expected "B"
+main:6: error: Argument 1 to "f" has incompatible type *List[A]; expected "B"
+main:7: error: Argument 1 to "f" has incompatible type *List[A]; expected "B"
 main:8: error: Argument 1 to "f" has incompatible type "B"; expected "A"
-main:9: error: Argument 2 to "f" has incompatible type List[A]; expected "B"
-main:10: error: Argument 3 to "f" has incompatible type List[A]; expected "B"
+main:9: error: Argument 2 to "f" has incompatible type *List[A]; expected "B"
+main:10: error: Argument 3 to "f" has incompatible type *List[A]; expected "B"
 main:11: error: List or tuple expected as variable arguments
 main:12: error: List or tuple expected as variable arguments
 
@@ -483,9 +483,9 @@ S = TypeVar('S')
 T = TypeVar('T')
 a, b = None, None # type: (A, B)
 
-a, a = f(*(a, b))   # E: Argument 1 to "f" has incompatible type "Tuple[A, B]"; expected "A"
+a, a = f(*(a, b))   # E: Argument 1 to "f" has incompatible type *"Tuple[A, B]"; expected "A"
 b, b = f(a, *(b,))  # E: Argument 1 to "f" has incompatible type "A"; expected "B"
-a, a = f(*(a, b))   # E: Argument 1 to "f" has incompatible type "Tuple[A, B]"; expected "A"
+a, a = f(*(a, b))   # E: Argument 1 to "f" has incompatible type *"Tuple[A, B]"; expected "A"
 b, b = f(a, *(b,))  # E: Argument 1 to "f" has incompatible type "A"; expected "B"
 a, b = f(*(a, b, b)) # E: Too many arguments for "f"
 
@@ -525,7 +525,7 @@ class B: pass
 main:9: error: Incompatible types in assignment (expression has type List[A], variable has type "A")
 main:9: error: Incompatible types in assignment (expression has type List[None], variable has type List[A])
 main:10: error: Incompatible types in assignment (expression has type List[None], variable has type "A")
-main:11: error: Argument 1 to "f" of "G" has incompatible type List[A]; expected "B"
+main:11: error: Argument 1 to "f" of "G" has incompatible type *List[A]; expected "B"
 main:11: error: Incompatible types in assignment (expression has type List[None], variable has type List[A])
 
 
diff --git a/mypy/test/data/fixtures/dict.py b/mypy/test/data/fixtures/dict.py
--- a/mypy/test/data/fixtures/dict.py
+++ b/mypy/test/data/fixtures/dict.py
@@ -1,6 +1,6 @@
 # Builtins stub used in dictionary-related test cases.
 
-from typing import TypeVar, Generic, Iterable, Iterator, Any
+from typing import TypeVar, Generic, Iterable, Iterator, Tuple
 
 T = TypeVar('T')
 S = TypeVar('S')
@@ -10,9 +10,10 @@ def __init__(self) -> None: pass
 
 class type: pass
 
-class dict(Generic[T, S]):
-    def __init__(self, arg: Any = None) -> None: pass
+class dict(Iterable[T], Generic[T, S]):
+    def __init__(self, arg: Iterable[Tuple[T, S]] = None) -> None: pass
     def __setitem__(self, k: T, v: S) -> None: pass
+    def __iter__(self) -> Iterator[T]: pass
     def update(self, a: 'dict[T, S]') -> None: pass
 class int: pass # for convenience
 class str: pass # for keyword argument key type
diff --git a/mypy/test/testinfer.py b/mypy/test/testinfer.py
--- a/mypy/test/testinfer.py
+++ b/mypy/test/testinfer.py
@@ -158,6 +158,12 @@ def test_special_cases(self):
         self.assert_map([ARG_STAR, ARG_STAR2],
                         [ARG_STAR, ARG_STAR2],
                         [[0], [1]])
+        self.assert_map([ARG_STAR2],
+                        [(ARG_POS, 'x'), ARG_STAR2],
+                        [[0], [0]])
+        self.assert_map([ARG_STAR2],
+                        [ARG_STAR2],
+                        [[0]])
 
     def assert_map(self, caller_kinds, callee_kinds, expected):
         caller_kinds, caller_names = expand_caller_kinds(caller_kinds)