Repository: python/mypy

Issue:
NotImplementedError in map_actuals_to_formals
When I run mypy (0.4 or git master with Python 3.5) with `--check-untyped-defs` over [qutebrowser](https://github.com/The-Compiler/qutebrowser), I get:

```
Traceback (most recent call last):
  File "./.tox/mypy/bin/mypy", line 6, in <module>
    main(__file__)
  File ".../mypy/main.py", line 54, in main
    res = type_check_only(sources, bin_dir, options)
  File ".../mypy/main.py", line 102, in type_check_only
    python_path=options.python_path)
  File ".../mypy/build.py", line 209, in build
    dispatch(sources, manager)
  File ".../mypy/build.py", line 1321, in dispatch
    process_graph(graph, manager)
  File ".../mypy/build.py", line 1452, in process_graph
    process_stale_scc(graph, scc)
  File ".../mypy/build.py", line 1482, in process_stale_scc
    graph[id].type_check()
  File ".../mypy/build.py", line 1301, in type_check
    manager.type_checker.visit_file(self.tree, self.xpath)
  File ".../mypy/checker.py", line 419, in visit_file
    self.accept(d)
  File ".../mypy/checker.py", line 460, in accept
    typ = node.accept(self)
  File ".../mypy/nodes.py", line 650, in accept
    return visitor.visit_class_def(self)
  File ".../mypy/checker.py", line 1049, in visit_class_def
    self.accept(defn.defs)
  File ".../mypy/checker.py", line 460, in accept
    typ = node.accept(self)
  File ".../mypy/nodes.py", line 715, in accept
    return visitor.visit_block(self)
  File ".../mypy/checker.py", line 1144, in visit_block
    self.accept(s)
  File ".../mypy/checker.py", line 460, in accept
    typ = node.accept(self)
  File ".../mypy/nodes.py", line 531, in accept
    return visitor.visit_decorator(self)
  File ".../mypy/checker.py", line 1934, in visit_decorator
    e.func.accept(self)
  File ".../mypy/nodes.py", line 462, in accept
    return visitor.visit_func_def(self)
  File ".../mypy/checker.py", line 573, in visit_func_def
    self.check_func_item(defn, name=defn.name())
  File ".../mypy/checker.py", line 631, in check_func_item
    self.check_func_def(defn, typ, name)
  File ".../mypy/checker.py", line 739, in check_func_def
    self.accept_in_frame(item.body)
  File ".../mypy/checker.py", line 475, in accept_in_frame
    answer = self.accept(node, type_context)
  File ".../mypy/checker.py", line 460, in accept
    typ = node.accept(self)
  File ".../mypy/nodes.py", line 715, in accept
    return visitor.visit_block(self)
  File ".../mypy/checker.py", line 1144, in visit_block
    self.accept(s)
  File ".../mypy/checker.py", line 460, in accept
    typ = node.accept(self)
  File ".../mypy/nodes.py", line 753, in accept
    return visitor.visit_assignment_stmt(self)
  File ".../mypy/checker.py", line 1153, in visit_assignment_stmt
    self.check_assignment(s.lvalues[-1], s.rvalue, s.type is None)
  File ".../mypy/checker.py", line 1203, in check_assignment
    self.infer_variable_type(inferred, lvalue, self.accept(rvalue),
  File ".../mypy/checker.py", line 460, in accept
    typ = node.accept(self)
  File ".../mypy/nodes.py", line 1185, in accept
    return visitor.visit_call_expr(self)
  File ".../mypy/checker.py", line 1992, in visit_call_expr
    return self.expr_checker.visit_call_expr(e)
  File ".../mypy/checkexpr.py", line 141, in visit_call_expr
    return self.check_call_expr_with_callee_type(callee_type, e)
  File ".../mypy/checkexpr.py", line 192, in check_call_expr_with_callee_type
    e.arg_names, callable_node=e.callee)[0]
  File ".../mypy/checkexpr.py", line 226, in check_call
    lambda i: self.accept(args[i]))
  File ".../mypy/checkexpr.py", line 1570, in map_actuals_to_formals
    raise NotImplementedError()
NotImplementedError: 

*** INTERNAL ERROR ***

qutebrowser/browser/webpage.py:344: error: Internal error -- please report a bug at https://github.com/python/mypy/issues
```

I wasn't able to construct a minimal example which fails in the same way, but it fails [on this line](https://github.com/The-Compiler/qutebrowser/blob/37b5f49c8542c04e7aae6ded6e8ab0c6aa5f2010/qutebrowser/browser/webpage.py#L344):

``` python
    @pyqtSlot('QWebFrame*', 'QWebPage::Feature')
    def on_feature_permission_requested(self, frame, feature):
        """Ask the user for approval for geolocation/notifications."""
        options = {
            QWebPage.Notifications: ('content', 'notifications'),
            QWebPage.Geolocation: ('content', 'geolocation'),
        }
        config_val = config.get(*options[feature])  # <-----
        # [...]
```

Some interesting locals from a `--pdb` session:

```
>>> argt.items
[builtins.str, builtins.str]
>>> j
1
>>> ncallee
2
>>> callee_kinds
[2, 4]
```

Seems like the `4` for the current argument (`j=1`) is `nodes.UNBOUND_TVAR`.



Discussion:
Here's a smaller repro:

``` python
def xget(section, subsection):
    pass

def aget(*args, **kwds):
    return xget(*args, **kwds)

def foo(i):
    args = ('x', 'y')
    return aget(*args)  # <----- error
```

The explicit `raise NotImplementedError()` in the mypy code suggests that Jukka was aware that he left some complicated case unhandled. Probably not rocket science to fix it.

Has this been fixed already? Running Mypy on @gvanrossum's example now prints nothing.

@The-Compiler Can you confirm? Indeed my little example doesn't crash any more. Most likely @rwbarton fixed this as a side effect of one of his other fixes.

Unfortunately I can still reproduce this with the current master and my code - no time right now to investigate more though, sorry!

Thanks for the confirmation! We'll keep it open. Is the code that
reproduces this in your repo?

Indeed. You should be able to clone https://github.com/The-Compiler/qutebrowser, check out the `mypy` branch, replace `mypy-lang==0.4` by `git+https://github.com/python/mypy.git` in `tox.ini` and run `tox -e mypy`. When you do so, the issue will not happen because I worked around it in https://github.com/The-Compiler/qutebrowser/commit/996ebbdb142070d13765d7ac619fc09553670bf1. As soon as you do `git revert 996ebbdb142070d13765d7ac619fc09553670bf1` you should see it again.

I just found a smaller repro:

``` python
t = (1, 2)
'{} {}'.format(*t)
```

Here's a python2 repro we hit:

```
from typing import Any

def good_log(string, *n):
    # type: (str, *Any) -> None
    string %= n

def bad_log(string, *n, **kw):
    # type: (str, *Any, **Any) -> None
    string %= n

def testit():
    # type: () -> None
    good_log("Numbers: (%r %r, %r %r)", *(1, 2, 3, 4))

    bad_log("Numbers: (%r %r, %r %r)", *(1, 2, 3, 4))
```

Notice how the existence of the **kw is what causes this to happen.

Smallest test case yet:

``` python
def f(**kw): pass
f(*(1, 2))
```

I'm sure this has something to do with the case analysis for "arg_kind" and
the special-casing there for tuple unpacking.

Simply commenting out [these lines](https://github.com/python/mypy/blob/master/mypy/checkexpr.py#L1572-1573) seems to fix the issue and not introduce any worse bugs, though I can't be sure...

Well, something is clearly hitting the else branch there, so the question
is what the various variables are at that point. Could you come up with
some test?

Adding `print(map, i, j, argt)` right before the `raise` gives:

```
[[]] 0 0 Tuple[builtins.int, builtins.int]
```

Seems to be that mypy is trying to map the `*(1, 2)` to the `**kw`, which is causing the `callee_kinds[j] != nodes.ARG_STAR2` check to fail, ending up in the `NotImplementedError`.

I think we shouldn't be advancing through `*args` of the callee in one-to-one correspondence with the components of the Tuple argument. If we encounter `*args` in the callee, it should receive all the left-over fields of the Tuple. I believe that just means that the `*args` callee argument should be mapped to the `*args` argument of the caller, but I'm not sure.

Another case that mypy crashes on is when there's no `*args` in the callee and we reach a `**kwargs` argument. That should be a "too many arguments" error.

Actually, it's possible that the `*args` in the call site doesn't match the `*args` in the definition, like this:

```
def foo(a, *args):
    return dict(a=a, args=args)
args = (1, 2, 3)
foo(*args)  # {'a': 1, 'args': (2, 3)}
foo(1, 2, *args)  # {'a': 1, 'args': (2, 1, 2, 3)}
```

But apart from that Reid's right!

Hm, I was hoping that matching up the `*args` of the callee with the `*args` of the call would be okay even though not all of the fields of the `*args` of the call might actually get passed to `*args` in the callee (as in your first example above, where one of the fields goes to `a` instead). I thought it might be okay because argument compatibility checking for `*args` vs. `*args` only needs to consider the types of the elements of the sequence, not the length of the sequence. But this doesn't quite cover every case, because the actual `*args` might be a heterogeneous Tuple. For example we could adjust the example above to:

``` py
def foo(a: int, *args: str) -> None: ...
args = (1, 'b', 'c')  # type: Tuple[int, str, str]
foo(*args)  # Should be allowed
```

In order to support this we'd have to understand that callee argument 2 corresponds to a subsequence of caller argument 1. We could just not handle this case for now though (we would reject some valid programs like the above).

Would be nice if for all combinations it either worked or emitted a polite
error rather than crashing. I think we have enough information about tuples
though to be able to compute that subsequence. Do we do this for slices or
unpacking?

Moving the milestone closer as this keeps coming up.

Even smaller test case:

``` py
tail = ['b', 'c']
['a', *tail]
```

This bug is probably related to #704.

@jirutka: No, that's a repro for the issue mentioned in #704 but the traceback is different from what this issue is about. You can either add your repro to #704 or file a new issue to call it out (since #704 is really about introducing support for PEP 448 -- if mypy just gave an error saying "Syntax error" instead of a crash that would be a start).

Okay, I’ve opened new issue #1890.


Resolving patch:
diff --git a/mypy/checkexpr.py b/mypy/checkexpr.py
--- a/mypy/checkexpr.py
+++ b/mypy/checkexpr.py
@@ -1643,9 +1643,7 @@ def map_actuals_to_formals(caller_kinds: List[int],
     argument type with the given index.
     """
     ncallee = len(callee_kinds)
-    map = [None] * ncallee  # type: List[List[int]]
-    for i in range(ncallee):
-        map[i] = []
+    map = [[] for i in range(ncallee)]  # type: List[List[int]]
     j = 0
     for i, kind in enumerate(caller_kinds):
         if kind == nodes.ARG_POS:
@@ -1666,7 +1664,7 @@ def map_actuals_to_formals(caller_kinds: List[int],
                         if callee_kinds[j] != nodes.ARG_STAR2:
                             map[j].append(i)
                         else:
-                            raise NotImplementedError()
+                            break
                         j += 1
             else:
                 # Assume that it is an iterable (if it isn't, there will be


Test patch:
diff --git a/test-data/unit/check-kwargs.test b/test-data/unit/check-kwargs.test
--- a/test-data/unit/check-kwargs.test
+++ b/test-data/unit/check-kwargs.test
@@ -305,3 +305,8 @@ f(**b) # E: Argument 1 to "f" has incompatible type **Dict[str, str]; expected "
 c = {0: 0}
 f(**c) # E: Keywords must be strings
 [builtins fixtures/dict.py]
+
+[case testCallStar2WithStar]
+def f(**k): pass
+f(*(1, 2))  # E: Too many arguments for "f"
+[builtins fixtures/dict.py]