Repository: python/mypy

Issue:
Fine grained crash when class is turned into a generic class
`a.py`:

```py
import b
def f() -> b.C: pass
```

Original `b.py`:

```py
import a
class C: pass
```

Updated `b.py`:

```py
from typing import TypeVar, Generic
import a

T = TypeVar('T')
class C(Generic[T]): pass

reveal_type(a.f)
c: C[int]
d = a.f()
d = c
c = d
```

Now if I run `mypy --quick a.py` first with the original files, then with the updated `b.py`, I get a crash:

```
$ mypy --quick --show-traceback a.py
b.py:7: error: Revealed type is 'def () -> b.C'
/Users/jukka/src/mypy/b.py:11: error: INTERNAL ERROR -- please report a bug at https://github.com/python/mypy/issues version: 0.510-dev-c856a53fe06c0ad9193d86ab35fb86693940f273-dirty
Traceback (most recent call last):
  File "/Users/jukka/src/mypy/scripts/mypy", line 6, in <module>
    main(__file__)
  File "/Users/jukka/src/mypy/mypy/main.py", line 46, in main
    res = type_check_only(sources, bin_dir, options)
  File "/Users/jukka/src/mypy/mypy/main.py", line 93, in type_check_only
    options=options)
  File "/Users/jukka/src/mypy/mypy/build.py", line 188, in build
    graph = dispatch(sources, manager)
  File "/Users/jukka/src/mypy/mypy/build.py", line 1570, in dispatch
    process_graph(graph, manager)
  File "/Users/jukka/src/mypy/mypy/build.py", line 1813, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/Users/jukka/src/mypy/mypy/build.py", line 1912, in process_stale_scc
    graph[id].type_check_first_pass()
  File "/Users/jukka/src/mypy/mypy/build.py", line 1485, in type_check_first_pass
    self.type_checker.check_first_pass()
  File "/Users/jukka/src/mypy/mypy/checker.py", line 177, in check_first_pass
    self.accept(d)
  File "/Users/jukka/src/mypy/mypy/checker.py", line 262, in accept
    stmt.accept(self)
  File "/Users/jukka/src/mypy/mypy/nodes.py", line 853, in accept
    return visitor.visit_assignment_stmt(self)
  File "/Users/jukka/src/mypy/mypy/checker.py", line 1177, in visit_assignment_stmt
    self.check_assignment(s.lvalues[-1], s.rvalue, s.type is None, s.new_syntax)
  File "/Users/jukka/src/mypy/mypy/checker.py", line 1239, in check_assignment
    rvalue_type = self.check_simple_assignment(lvalue_type, rvalue, lvalue)
  File "/Users/jukka/src/mypy/mypy/checker.py", line 1698, in check_simple_assignment
    rvalue_type = self.expr_checker.accept(rvalue, lvalue_type)
  File "/Users/jukka/src/mypy/mypy/checkexpr.py", line 2056, in accept
    typ = node.accept(self)
  File "/Users/jukka/src/mypy/mypy/nodes.py", line 1213, in accept
    return visitor.visit_name_expr(self)
  File "/Users/jukka/src/mypy/mypy/checkexpr.py", line 123, in visit_name_expr
    return self.narrow_type_from_binder(e, result)
  File "/Users/jukka/src/mypy/mypy/checkexpr.py", line 2271, in narrow_type_from_binder
    ans = narrow_declared_type(known_type, restriction)
  File "/Users/jukka/src/mypy/mypy/meet.py", line 46, in narrow_declared_type
    return meet_types(declared, narrowed)
  File "/Users/jukka/src/mypy/mypy/meet.py", line 25, in meet_types
    return t.accept(TypeMeetVisitor(s))
  File "/Users/jukka/src/mypy/mypy/types.py", line 381, in accept
    return visitor.visit_instance(self)
  File "/Users/jukka/src/mypy/mypy/meet.py", line 198, in visit_instance
    args.append(self.meet(t.args[i], si.args[i]))
IndexError: list index out of range
/Users/jukka/src/mypy/b.py:11: note: use --pdb to drop into pdb
```


Discussion:
I can also cause a crash if I switch a generic class into a non-generic one. However, I need to switch the order of assignments in the second version:

```py
...
reveal_type(a.f)
c: C[int]
d = a.f()
c = d   # this comes first now
d = c
```
I think this could be also a symptom of another problem: there are many places where e.g. ``named_type('builtins.tuple')`` is used instead of ``named_generic_type('builtins.tuple', [AnyType()])``. So that many types with wrong type argument count are created. This leads to crashes like #3117. I think that ideally we could "fix" all such code, this process is started in #3129 (also in protocols PR I do the same for ``typing.Awaitable``).
> I think this could be also a symptom of another problem: there are many places where e.g. named_type('builtins.tuple') is used instead of named_generic_type('builtins.tuple', [AnyType()]).

These are all wrong. We should only create `Instance` types with the correct number of type arguments, and crash hard if we try to create an bad type.
> These are all wrong. We should only create ``Instance`` types with the correct number of type arguments, and crash hard if we try to create an bad type.

OK, then a possible "robust" solution would be to remove ``named_type`` and only use ``named_generic_type`` that will check argument count on ``Instance`` creation. (We cannot make this check in ``Instance.__init__``, since type argument counts are not known until third pass.)
The quick mode was deprecated and is now removed, but now the same crash happens in the fine-grained mode. (Same files and the same traceback.)
(Removed the incremental label, since the crash doesn't happen in the normal incremental mode.)
Here is the full test case:
```
[case testFormerQuick]
import a
[file a.py]
import b
def f() -> b.C: pass
[file b.py]
import a
class C: pass
[file b.py.2]
from typing import TypeVar, Generic
import a

T = TypeVar('T')
class C(Generic[T]): pass

reveal_type(a.f)
c: C[int]
d = a.f()
d = c
c = d
[out]
==
tmp/b.py:7: error: Revealed type is 'def () -> b.C[Any]'
```

Resolving patch:
diff --git a/mypy/constraints.py b/mypy/constraints.py
--- a/mypy/constraints.py
+++ b/mypy/constraints.py
@@ -347,29 +347,33 @@ def visit_instance(self, template: Instance) -> List[Constraint]:
                     template.type.has_base(instance.type.fullname)):
                 mapped = map_instance_to_supertype(template, instance.type)
                 tvars = mapped.type.defn.type_vars
-                for i in range(len(instance.args)):
+                # N.B: We use zip instead of indexing because the lengths might have
+                # mismatches during daemon reprocessing.
+                for tvar, mapped_arg, instance_arg in zip(tvars, mapped.args, instance.args):
                     # The constraints for generic type parameters depend on variance.
                     # Include constraints from both directions if invariant.
-                    if tvars[i].variance != CONTRAVARIANT:
+                    if tvar.variance != CONTRAVARIANT:
                         res.extend(infer_constraints(
-                            mapped.args[i], instance.args[i], self.direction))
-                    if tvars[i].variance != COVARIANT:
+                            mapped_arg, instance_arg, self.direction))
+                    if tvar.variance != COVARIANT:
                         res.extend(infer_constraints(
-                            mapped.args[i], instance.args[i], neg_op(self.direction)))
+                            mapped_arg, instance_arg, neg_op(self.direction)))
                 return res
             elif (self.direction == SUPERTYPE_OF and
                     instance.type.has_base(template.type.fullname)):
                 mapped = map_instance_to_supertype(instance, template.type)
                 tvars = template.type.defn.type_vars
-                for j in range(len(template.args)):
+                # N.B: We use zip instead of indexing because the lengths might have
+                # mismatches during daemon reprocessing.
+                for tvar, mapped_arg, template_arg in zip(tvars, mapped.args, template.args):
                     # The constraints for generic type parameters depend on variance.
                     # Include constraints from both directions if invariant.
-                    if tvars[j].variance != CONTRAVARIANT:
+                    if tvar.variance != CONTRAVARIANT:
                         res.extend(infer_constraints(
-                            template.args[j], mapped.args[j], self.direction))
-                    if tvars[j].variance != COVARIANT:
+                            template_arg, mapped_arg, self.direction))
+                    if tvar.variance != COVARIANT:
                         res.extend(infer_constraints(
-                            template.args[j], mapped.args[j], neg_op(self.direction)))
+                            template_arg, mapped_arg, neg_op(self.direction)))
                 return res
             if (template.type.is_protocol and self.direction == SUPERTYPE_OF and
                     # We avoid infinite recursion for structural subtypes by checking
diff --git a/mypy/join.py b/mypy/join.py
--- a/mypy/join.py
+++ b/mypy/join.py
@@ -345,8 +345,10 @@ def join_instances(t: Instance, s: Instance) -> ProperType:
         if is_subtype(t, s) or is_subtype(s, t):
             # Compatible; combine type arguments.
             args = []  # type: List[Type]
-            for i in range(len(t.args)):
-                args.append(join_types(t.args[i], s.args[i]))
+            # N.B: We use zip instead of indexing because the lengths might have
+            # mismatches during daemon reprocessing.
+            for ta, sa in zip(t.args, s.args):
+                args.append(join_types(ta, sa))
             return Instance(t.type, args)
         else:
             # Incompatible; return trivial result object.
diff --git a/mypy/meet.py b/mypy/meet.py
--- a/mypy/meet.py
+++ b/mypy/meet.py
@@ -491,8 +491,10 @@ def visit_instance(self, t: Instance) -> ProperType:
                     # Combine type arguments. We could have used join below
                     # equivalently.
                     args = []  # type: List[Type]
-                    for i in range(len(t.args)):
-                        args.append(self.meet(t.args[i], si.args[i]))
+                    # N.B: We use zip instead of indexing because the lengths might have
+                    # mismatches during daemon reprocessing.
+                    for ta, sia in zip(t.args, si.args):
+                        args.append(self.meet(ta, sia))
                     return Instance(t.type, args)
                 else:
                     if state.strict_optional:


Test patch:
diff --git a/test-data/unit/fine-grained.test b/test-data/unit/fine-grained.test
--- a/test-data/unit/fine-grained.test
+++ b/test-data/unit/fine-grained.test
@@ -9288,3 +9288,92 @@ class B:
         self.x = 0
 [out]
 ==
+
+[case testGenericChange1]
+import a
+[file a.py]
+import b
+def f() -> b.C: pass
+[file b.py]
+import a
+class C: pass
+[file b.py.2]
+from typing import TypeVar, Generic, List
+import a
+
+T = TypeVar('T')
+class C(Generic[T]): pass
+
+reveal_type(a.f)
+c: C[int]
+l = a.f() if True else c
+d = a.f()
+d = c
+c = d
+
+x: List[C] = [a.f(), a.f()]
+
+[out]
+==
+b.py:7: note: Revealed type is 'def () -> b.C[Any]'
+[builtins fixtures/list.pyi]
+
+[case testGenericChange2]
+import a
+[file a.py]
+import b
+def f() -> b.C[int]: pass
+[file b.py]
+from typing import TypeVar, Generic
+import a
+T = TypeVar('T')
+class C(Generic[T]): pass
+[file b.py.2]
+from typing import List
+import a
+
+class C(): pass
+
+c: C
+l = a.f() if True else c
+d = a.f()
+d = c
+c = d
+
+x: List[C] = [a.f(), a.f()]
+
+[builtins fixtures/list.pyi]
+[out]
+==
+a.py:2: error: "C" expects no type arguments, but 1 given
+
+[case testGenericChange3]
+import a
+[file a.py]
+import b
+def f() -> b.C[int]: pass
+[file b.py]
+from typing import TypeVar, Generic
+import a
+T = TypeVar('T')
+class C(Generic[T]): pass
+[file b.py.2]
+from typing import TypeVar, Generic, List
+import a
+
+T = TypeVar('T')
+S = TypeVar('S')
+class C(Generic[S, T]): pass
+
+c: C[int, str]
+l = a.f() if True else c
+d = a.f()
+d = c
+c = d
+
+x: List[C] = [a.f(), a.f()]
+
+[out]
+==
+a.py:2: error: "C" expects 2 type arguments, but 1 given
+[builtins fixtures/list.pyi]