Repository: python/mypy

Issue:
internal error with kwargs
Encountered while running mypy on something using polib. Here's a reproducer:

```
import textwrap
def wrap(text, width=70, **kwargs):
    return textwrap.wrap(text, width=width, **kwargs)
```

and here's the traceback:

```
Traceback (most recent call last):
  File "/usr/bin/mypy", line 6, in <module>
    main(__file__)
  File "/usr/lib/python3.5/site-packages/mypy/main.py", line 50, in main
    type_check_only(sources, bin_dir, options)
  File "/usr/lib/python3.5/site-packages/mypy/main.py", line 93, in type_check_only
    python_path=options.python_path)
  File "/usr/lib/python3.5/site-packages/mypy/build.py", line 203, in build
    result = manager.process(initial_states)
  File "/usr/lib/python3.5/site-packages/mypy/build.py", line 385, in process
    next.process()
  File "/usr/lib/python3.5/site-packages/mypy/build.py", line 882, in process
    self.type_checker().visit_file(self.tree, self.tree.path)
  File "/usr/lib/python3.5/site-packages/mypy/checker.py", line 385, in visit_file
    self.accept(d)
  File "/usr/lib/python3.5/site-packages/mypy/checker.py", line 394, in accept
    typ = node.accept(self)
  File "/usr/lib/python3.5/site-packages/mypy/nodes.py", line 381, in accept
    return visitor.visit_func_def(self)
  File "/usr/lib/python3.5/site-packages/mypy/checker.py", line 449, in visit_func_def
    self.check_func_item(defn, name=defn.name())
  File "/usr/lib/python3.5/site-packages/mypy/checker.py", line 484, in check_func_item
    self.check_func_def(defn, typ, name)
  File "/usr/lib/python3.5/site-packages/mypy/checker.py", line 567, in check_func_def
    self.accept_in_frame(item.body)
  File "/usr/lib/python3.5/site-packages/mypy/checker.py", line 409, in accept_in_frame
    answer = self.accept(node, type_context)
  File "/usr/lib/python3.5/site-packages/mypy/checker.py", line 394, in accept
    typ = node.accept(self)
  File "/usr/lib/python3.5/site-packages/mypy/nodes.py", line 525, in accept
    return visitor.visit_block(self)
  File "/usr/lib/python3.5/site-packages/mypy/checker.py", line 984, in visit_block
    self.accept(s)
  File "/usr/lib/python3.5/site-packages/mypy/checker.py", line 394, in accept
    typ = node.accept(self)
  File "/usr/lib/python3.5/site-packages/mypy/nodes.py", line 622, in accept
    return visitor.visit_return_stmt(self)
  File "/usr/lib/python3.5/site-packages/mypy/checker.py", line 1386, in visit_return_stmt
    typ = self.accept(s.expr, self.return_types[-1])
  File "/usr/lib/python3.5/site-packages/mypy/checker.py", line 394, in accept
    typ = node.accept(self)
  File "/usr/lib/python3.5/site-packages/mypy/nodes.py", line 984, in accept
    return visitor.visit_call_expr(self)
  File "/usr/lib/python3.5/site-packages/mypy/checker.py", line 1841, in visit_call_expr
    return self.expr_checker.visit_call_expr(e)
  File "/usr/lib/python3.5/site-packages/mypy/checkexpr.py", line 126, in visit_call_expr
    return self.check_call_expr_with_callee_type(callee_type, e)
  File "/usr/lib/python3.5/site-packages/mypy/checkexpr.py", line 177, in check_call_expr_with_callee_type
    e.arg_names, callable_node=e.callee)[0]
  File "/usr/lib/python3.5/site-packages/mypy/checkexpr.py", line 224, in check_call
    arg_names, formal_to_actual, context)
  File "/usr/lib/python3.5/site-packages/mypy/checkexpr.py", line 535, in check_argument_count
    isinstance(actual_type, TupleType)):
UnboundLocalError: local variable 'actual_type' referenced before assignment

*** INTERNAL ERROR ***

tmp.py:3: error: Internal error -- please report a bug at https://github.com/JukkaL/mypy/issues
```

The problem seems to be in the conditional added in 06753e7c696b254d173a0abf4e61bc443cb3b6b5 but I'm not sure how actual_type should be set in this case or what should be used in its place.



Discussion:
I think this is what was intended:

```
 i < len(actual_types) and isinstance(actual_types[i], TupleType)):
```

With that, I don't get an internal error for your code snippet but I do get another error:

```
tmp.py:3: error: Too many positional arguments for "wrap"
```

but that seems due to a bug in the stub file (stdlib/3/textwrap.pyi) in the typeshed repo -- it's missing the `text` positional parameter.

Makes sense, though actual_types would have to be initialized to an empty list at some point.

Well, `actual_types` is a parameter of type `List[Type]`. Does that satisfy
you?

On Tue, Dec 22, 2015 at 12:47 PM, David Shea notifications@github.com
wrote:

> Makes sense, though actual_types would have to be initialized to an empty
> list at some point.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/JukkaL/mypy/issues/1095#issuecomment-166725903.

## 

--Guido van Rossum (python.org/~guido)

Oh, right. I was getting actual_types and actual_type mixed up. Thanks.

FWIW, I spent a bunch of time trying to understand what this code is intending to do, and failed. It does not look like the fix I suggested above is correct. I will get back to this but unfortunately I've got higher priorities (all related to mypy though :-).

Smallest test case:

This correctly errors with `x.py:3: error: "f" gets multiple values for keyword argument "a"`:

``` python
def f(a):
    return

f(1, a=1)
```

This crashes:

``` python
def f(a): pass

def g():
    f(1, a=1)
```

Thanks -- I had found a similar test case but failed to post it here. The
former gives an error message "f" gets multiple values for keyword argument
"a" right? The latter crashes.

Unfortunately there are a whole bunch of things I could substitute for
actual_type that make the crash go away, but when running unittests they
all seem to cause some test (usually in check-weak-typing.test) to fail.

FWIW if you add "->None" to g() in the latter example it doesn't crash (but
gives the same error message as the former).

On Tue, Jan 12, 2016 at 8:52 AM, Ryan Gonzalez notifications@github.com
wrote:

> Smallest test case:
> 
> This works:
> 
> def f(a):
>     return
> 
> f(1, a=1)
> 
> This fails:
> 
> def f(a): pass
> 
> def g():
>     f(1, a=1)
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/JukkaL/mypy/issues/1095#issuecomment-170972493.

## 

--Guido van Rossum (python.org/~guido)


Resolving patch:
diff --git a/misc/fix_annotate.py b/misc/fix_annotate.py
new file mode 100644
--- /dev/null
+++ b/misc/fix_annotate.py
@@ -0,0 +1,216 @@
+"""Fixer that inserts mypy annotations into all methods.
+
+This transforms e.g.
+
+  def foo(self, bar, baz=12):
+      return bar + baz
+
+into
+
+  def foo(self, bar, baz=12):
+      # type: (Any, int) -> Any
+      return bar + baz
+
+It does not do type inference but it recognizes some basic default
+argument values such as numbers and strings (and assumes their type
+implies the argument type).
+
+It also uses some basic heuristics to decide whether to ignore the
+first argument:
+
+  - always if it's named 'self'
+  - if there's a @classmethod decorator
+
+Finally, it knows that __init__() is supposed to return None.
+"""
+
+from __future__ import print_function
+
+import os
+import re
+
+from lib2to3.fixer_base import BaseFix
+from lib2to3.patcomp import compile_pattern
+from lib2to3.pytree import Leaf, Node
+from lib2to3.fixer_util import token, syms, touch_import
+
+
+class FixAnnotate(BaseFix):
+
+    # This fixer is compatible with the bottom matcher.
+    BM_compatible = True
+
+    # This fixer shouldn't run by default.
+    explicit = True
+
+    # The pattern to match.
+    PATTERN = """
+              funcdef< 'def' name=any parameters< '(' [args=any] ')' > ':' suite=any+ >
+              """
+
+    counter = None if not os.getenv('MAXFIXES') else int(os.getenv('MAXFIXES'))
+
+    def transform(self, node, results):
+        if FixAnnotate.counter is not None:
+            if FixAnnotate.counter <= 0:
+                return
+        suite = results['suite']
+        children = suite[0].children
+
+        # NOTE: I've reverse-engineered the structure of the parse tree.
+        # It's always a list of nodes, the first of which contains the
+        # entire suite.  Its children seem to be:
+        #
+        #   [0] NEWLINE
+        #   [1] INDENT
+        #   [2...n-2] statements (the first may be a docstring)
+        #   [n-1] DEDENT
+        #
+        # Comments before the suite are part of the INDENT's prefix.
+        #
+        # "Compact" functions (e.g. "def foo(x, y): return max(x, y)")
+        # have a different structure that isn't matched by PATTERN.
+
+        ## print('-'*60)
+        ## print(node)
+        ## for i, ch in enumerate(children):
+        ##     print(i, repr(ch.prefix), repr(ch))
+
+        # Check if there's already an annotation.
+        for ch in children:
+            if ch.prefix.lstrip().startswith('# type:'):
+                return  # There's already a # type: comment here; don't change anything.
+
+        # Compute the annotation
+        annot = self.make_annotation(node, results)
+
+        # Insert '# type: {annot}' comment.
+        # For reference, see lib2to3/fixes/fix_tuple_params.py in stdlib.
+        if len(children) >= 2 and children[1].type == token.INDENT:
+            children[1].prefix = '%s# type: %s\n%s' % (children[1].value, annot, children[1].prefix)
+            children[1].changed()
+            if FixAnnotate.counter is not None:
+                FixAnnotate.counter -= 1
+
+        # Also add 'from typing import Any' at the top.
+        if 'Any' in annot:
+            touch_import('typing', 'Any', node)
+
+    def make_annotation(self, node, results):
+        name = results['name']
+        assert isinstance(name, Leaf), repr(name)
+        assert name.type == token.NAME, repr(name)
+        decorators = self.get_decorators(node)
+        is_method = self.is_method(node)
+        if name.value == '__init__' or not self.has_return_exprs(node):
+            restype = 'None'
+        else:
+            restype = 'Any'
+        args = results.get('args')
+        argtypes = []
+        if isinstance(args, Node):
+            children = args.children
+        elif isinstance(args, Leaf):
+            children = [args]
+        else:
+            children = []
+        # Interpret children according to the following grammar:
+        # (('*'|'**')? NAME ['=' expr] ','?)*
+        stars = inferred_type = ''
+        in_default = False
+        at_start = True
+        for child in children:
+            if isinstance(child, Leaf):
+                if child.value in ('*', '**'):
+                    stars += child.value
+                elif child.type == token.NAME and not in_default:
+                    if not is_method or not at_start or 'staticmethod' in decorators:
+                        inferred_type = 'Any'
+                    else:
+                        # Always skip the first argument if it's named 'self'.
+                        # Always skip the first argument of a class method.
+                        if  child.value == 'self' or 'classmethod' in decorators:
+                            pass
+                        else:
+                            inferred_type = 'Any'
+                elif child.value == '=':
+                    in_default = True
+                elif in_default and child.value != ',':
+                    if child.type == token.NUMBER:
+                        if re.match(r'\d+[lL]?$', child.value):
+                            inferred_type = 'int'
+                        else:
+                            inferred_type = 'float'  # TODO: complex?
+                    elif child.type == token.STRING:
+                        if child.value.startswith(('u', 'U')):
+                            inferred_type = 'unicode'
+                        else:
+                            inferred_type = 'str'
+                    elif child.type == token.NAME and child.value in ('True', 'False'):
+                        inferred_type = 'bool'
+                elif child.value == ',':
+                    if inferred_type:
+                        argtypes.append(stars + inferred_type)
+                    # Reset
+                    stars = inferred_type = ''
+                    in_default = False
+                    at_start = False
+        if inferred_type:
+            argtypes.append(stars + inferred_type)
+        return '(' + ', '.join(argtypes) + ') -> ' + restype
+
+    # The parse tree has a different shape when there is a single
+    # decorator vs. when there are multiple decorators.
+    DECORATED = "decorated< (d=decorator | decorators< dd=decorator+ >) funcdef >"
+    decorated = compile_pattern(DECORATED)
+
+    def get_decorators(self, node):
+        """Return a list of decorators found on a function definition.
+
+        This is a list of strings; only simple decorators
+        (e.g. @staticmethod) are returned.
+
+        If the function is undecorated or only non-simple decorators
+        are found, return [].
+        """
+        if node.parent is None:
+            return []
+        results = {}
+        if not self.decorated.match(node.parent, results):
+            return []
+        decorators = results.get('dd') or [results['d']]
+        decs = []
+        for d in decorators:
+            for child in d.children:
+                if isinstance(child, Leaf) and child.type == token.NAME:
+                    decs.append(child.value)
+        return decs
+
+    def is_method(self, node):
+        """Return whether the node occurs (directly) inside a class."""
+        node = node.parent
+        while node is not None:
+            if node.type == syms.classdef:
+                return True
+            if node.type == syms.funcdef:
+                return False
+            node = node.parent
+        return False
+
+    RETURN_EXPR = "return_stmt< 'return' any >"
+    return_expr = compile_pattern(RETURN_EXPR)
+
+    def has_return_exprs(self, node):
+        """Traverse the tree below node looking for 'return expr'.
+
+        Return True if at least 'return expr' is found, False if not.
+        (If both 'return' and 'return expr' are found, return True.)
+        """
+        results = {}
+        if self.return_expr.match(node, results):
+            return True
+        for child in node.children:
+            if child.type not in (syms.funcdef, syms.classdef):
+                if self.has_return_exprs(child):
+                    return True
+        return False
diff --git a/mypy/checkexpr.py b/mypy/checkexpr.py
--- a/mypy/checkexpr.py
+++ b/mypy/checkexpr.py
@@ -542,7 +542,7 @@ def check_argument_count(self, callee: CallableType, actual_types: List[Type],
                           nodes.ARG_NAMED] and is_duplicate_mapping(
                     formal_to_actual[i], actual_kinds):
                 if (self.chk.typing_mode_full() or
-                        isinstance(actual_type, TupleType)):
+                        isinstance(actual_types[formal_to_actual[i][0]], TupleType)):
                     self.msg.duplicate_argument_value(callee, i, context)
             elif (kind == nodes.ARG_NAMED and formal_to_actual[i] and
                   actual_kinds[formal_to_actual[i][0]] != nodes.ARG_NAMED):


Test patch:
diff --git a/mypy/test/data/check-kwargs.test b/mypy/test/data/check-kwargs.test
--- a/mypy/test/data/check-kwargs.test
+++ b/mypy/test/data/check-kwargs.test
@@ -75,6 +75,12 @@ f(A(), a=A()) # E: "f" gets multiple values for keyword argument "a"
 class A: pass
 class B: pass
 
+[case testPositionalAndKeywordForSameArg]
+# This used to crash in check_argument_count(). See #1095.
+def f(a: int): pass
+def g(): f(0, a=1)
+[out]
+
 [case testInvalidKeywordArgument]
 import typing
 def f(a: 'A') -> None: pass # N: "f" defined here