Repository: python/mypy

Issue:
mypy doesn't understand `six.with_metaclass`
Example:

``` python
import six


class TestBase(object):
    pass


class TestMeta(type):
    pass


class Test(six.with_metaclass(TestMeta, TestBase)):
    pass
```

In this case, mypy returns `error: Invalid base class`. However, in my larger project it actually caused `mypy` to crash while checking a type derived from the `Test` analogue. Unfortunately I wasn't able to reproduce this in a small example, but switching to `@six.add_metaclass` fixed the problem.



Discussion:
@gvanrossum Do you believe this could be solved with the hypothetical plugin system in https://github.com/python/mypy/issues/1240 or is `six` core enough it would make sense to bake understanding into `mypy`?

I think this should probably just be special-cased for six. Dynamically
created base classes aren't really mypy's core competency, but we should
handle at least some common cases.

This should probably be looked at again now that #2475 has been merged
More issues:
- six.add_metaclass causes a runtime error
```python
from typing import *
from typing import GenericMeta

import six

class _DestroyableMeta(type): pass

@six.add_metaclass(_DestroyableMeta)
class Destroyable:
    pass

T_co = TypeVar('T_co', bound='Destroyable', covariant=True)

class ArcMeta(GenericMeta, _DestroyableMeta):
    pass

@six.add_metaclass(ArcMeta)
class Arc(Generic[T_co], Destroyable):
    pass
```
This passes with `mypy -2` but running with python2.7 gives
```
Traceback (most recent call last):
  File "t.py", line 18, in <module>
    class Arc(Generic[T_co], Destroyable):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/typing.py", line 1008, in __new__
    self = super(GenericMeta, cls).__new__(cls, name, bases, namespace)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/typing.py", line 104, in __new__
    return super(TypingMeta, cls).__new__(cls, str(name), bases, namespace)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/abc.py", line 87, in __new__
    cls = super(ABCMeta, mcls).__new__(mcls, name, bases, namespace)
TypeError: Error when calling the metaclass bases
    metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases
```


- six.with_metaclass causes a mypy error (it gives both Invalid base class and also no longer recognizes the Generic base class). (Exact repro TBD, these are errors from our production code.)
```
    x.py:32: error: Invalid base class
    x.py:34: error: "Arc" expects no type arguments, but 1 given
    x.py:60: error: "Arc" expects no type arguments, but 1 given
    x.py:81: error: "Arc" expects no type arguments, but 1 given
```
> This passes with mypy -2 but running with python2.7 gives

Maybe this should be filed in the typing issue tracker?
> Maybe this should be filed in the typing issue tracker?

I think it is not something we can fix. ``six.add_metaclass`` applies to already created class. But the class can't actually be created because of an obvious metaclass conflict (this also fails the same way in 3.6).
Thinking about this more, Ivan is of course right that `six.add_metaclass` doesn't work for multiple inheritance. There are really several separate mypy bugs here:

- It completely ignores the `@six.add_metaclass(mcls)` decorator; it should interpret this as equivalent to `__metaclass__ = mcls` in the body for PY2, or the `metaclass=mcls` keyword argument to the class statement in PY3. (It's still useful for single inheritance.)
- It doesn't recognize `six.with_metaclass(mcls, base, ...)` in the "base classes" position of a class statement -- this is the only solution that works at runtime for PY2 and PY3 with multiple inheritance.

I intend to fix both by hard-coding checks for these two `six` functions in those positions.

There's also the issue that mypy accepted this, which doesn't work at runtime:
```
@six.add_metaclass(ArcMeta)
class Arc(Generic[T_co], Destroyable):
```
To address this we ought to duplicate Python's metaclass compatibility check. I think that's a lower priority though.
Related: mypy basically ignores class decorators (#3135). I agree that the six functions merit special treatment. (Or maybe Jukka's proposed plugin architecture from #3299 could be sufficient?)
I suspect the plugin architecture won't work here because metaclasses are computed before type checking begins.

Resolving patch:
diff --git a/mypy/semanal.py b/mypy/semanal.py
--- a/mypy/semanal.py
+++ b/mypy/semanal.py
@@ -758,8 +758,12 @@ def clean_up_bases_and_infer_type_variables(self, defn: ClassDef) -> None:
         Now we will remove Generic[T] from bases of Foo and infer that the
         type variable 'T' is a type argument of Foo.
 
+        We also process six.with_metaclass() here.
+
         Note that this is performed *before* semantic analysis.
         """
+        # First process six.with_metaclass if present and well-formed
+        defn.base_type_exprs, defn.metaclass = self.check_with_metaclass(defn)
         removed = []  # type: List[int]
         declared_tvars = []  # type: TypeVarList
         for i, base_expr in enumerate(defn.base_type_exprs):
@@ -938,6 +942,7 @@ def analyze_base_classes(self, defn: ClassDef) -> None:
 
         base_types = []  # type: List[Instance]
         info = defn.info
+
         for base_expr in defn.base_type_exprs:
             try:
                 base = self.expr_to_analyzed_type(base_expr)
@@ -989,6 +994,27 @@ def analyze_base_classes(self, defn: ClassDef) -> None:
         if defn.info.is_enum and defn.type_vars:
             self.fail("Enum class cannot be generic", defn)
 
+    def check_with_metaclass(self, defn: ClassDef) -> Tuple[List[Expression], Optional[str]]:
+        # Special-case six.with_metaclass(M, B1, B2, ...).
+        base_type_exprs, metaclass = defn.base_type_exprs, defn.metaclass
+        if metaclass is None and len(base_type_exprs) == 1:
+            base_expr = base_type_exprs[0]
+            if isinstance(base_expr, CallExpr) and isinstance(base_expr.callee, RefExpr):
+                base_expr.callee.accept(self)
+                if (base_expr.callee.fullname == 'six.with_metaclass'
+                        and len(base_expr.args) >= 1
+                        and all(kind == ARG_POS for kind in base_expr.arg_kinds)):
+                    metaclass_expr = base_expr.args[0]
+                    if isinstance(metaclass_expr, NameExpr):
+                        metaclass = metaclass_expr.name
+                    elif isinstance(metaclass_expr, MemberExpr):
+                        metaclass = get_member_expr_fullname(metaclass_expr)
+                    else:
+                        self.fail("Dynamic metaclass not supported for '%s'" % defn.name,
+                                  metaclass_expr)
+                    return (base_expr.args[1:], metaclass)
+        return (base_type_exprs, metaclass)
+
     def expr_to_analyzed_type(self, expr: Expression) -> Type:
         if isinstance(expr, CallExpr):
             expr.accept(self)


Test patch:
diff --git a/test-data/unit/check-classes.test b/test-data/unit/check-classes.test
--- a/test-data/unit/check-classes.test
+++ b/test-data/unit/check-classes.test
@@ -3137,3 +3137,94 @@ class M(type):
 class A(metaclass=M): pass
 reveal_type(type(A).x)  # E: Revealed type is 'builtins.int'
 
+[case testSixWithMetaclass]
+import six
+class M(type):
+    x = 5
+class A(six.with_metaclass(M)): pass
+reveal_type(type(A).x)  # E: Revealed type is 'builtins.int'
+
+[case testSixWithMetaclass_python2]
+import six
+class M(type):
+    x = 5
+class A(six.with_metaclass(M)): pass
+reveal_type(type(A).x)  # E: Revealed type is 'builtins.int'
+
+[case testFromSixWithMetaclass]
+from six import with_metaclass
+class M(type):
+    x = 5
+class A(with_metaclass(M)): pass
+reveal_type(type(A).x)  # E: Revealed type is 'builtins.int'
+
+[case testSixWithMetaclassImportFrom]
+import six
+from metadefs import M
+class A(six.with_metaclass(M)): pass
+reveal_type(type(A).x)  # E: Revealed type is 'builtins.int'
+[file metadefs.py]
+class M(type):
+    x = 5
+
+[case testSixWithMetaclassImport]
+import six
+import metadefs
+class A(six.with_metaclass(metadefs.M)): pass
+reveal_type(type(A).x)  # E: Revealed type is 'builtins.int'
+[file metadefs.py]
+class M(type):
+    x = 5
+
+[case testSixWithMetaclassAndBase]
+import six
+class M(type):
+    x = 5
+class A:
+    def foo(self): pass
+class B:
+    def bar(self): pass
+class C1(six.with_metaclass(M, A)): pass
+class C2(six.with_metaclass(M, A, B)): pass
+reveal_type(type(C1).x)  # E: Revealed type is 'builtins.int'
+reveal_type(type(C2).x)  # E: Revealed type is 'builtins.int'
+C1().foo()
+C1().bar()  # E: "C1" has no attribute "bar"
+C2().foo()
+C2().bar()
+C2().baz()  # E: "C2" has no attribute "baz"
+
+[case testSixWithMetaclassGenerics]
+from typing import Generic, GenericMeta, TypeVar
+import six
+class DestroyableMeta(type):
+    pass
+class Destroyable(six.with_metaclass(DestroyableMeta)):
+    pass
+T_co = TypeVar('T_co', bound='Destroyable', covariant=True)
+class ArcMeta(GenericMeta, DestroyableMeta):
+    pass
+class Arc(six.with_metaclass(ArcMeta, Generic[T_co], Destroyable)):
+    pass
+class MyDestr(Destroyable):
+    pass
+reveal_type(Arc[MyDestr]())  # E: Revealed type is '__main__.Arc[__main__.MyDestr*]'
+[builtins fixtures/bool.pyi]
+
+[case testSixWithMetaclassErrors]
+import six
+class M(type): pass
+class A(object): pass
+def f() -> type: return M
+class C1(six.with_metaclass(M), object): pass  # E: Invalid base class
+class C2(C1, six.with_metaclass(M)): pass  # E: Invalid base class
+class C3(six.with_metaclass(A)): pass  # E: Metaclasses not inheriting from 'type' are not supported
+class C4(six.with_metaclass(M), metaclass=M): pass  # E: Invalid base class
+class C5(six.with_metaclass(f())): pass  # E: Dynamic metaclass not supported for 'C5'
+
+[case testSixWithMetaclassErrors_python2-skip]
+# No error here yet
+import six
+class M(type): pass
+class C4(six.with_metaclass(M)):
+    __metaclass__ = M