Repository: python/mypy

Issue:
stubtest: catch type alias/variable mismatch
See python/typeshed#5271: stubtest didn't catch that a variable that was in reality a tuple of exceptions was defined as a type alias instead.

cc @hauntsaninja @Akuli 


Discussion:
Oh yeah, i just ignored type aliases since in most cases they're not meant to line up with anything in the runtime. But I guess if the runtime does have a matching object I should check compatibility.
So it looks like I have code to do this on a branch. It looks like it causes a fair number of new errors, but more because we now follow aliases around and recursively check than because type aliases are broken.  For a fun example, look at the xml.dom.minidom one:

<details>

```
_curses.window.attroff is inconsistent, stub argument "attr" should be positional-only (rename with a leading double underscore, i.e. "__attr")
_curses.window.attron is inconsistent, stub argument "attr" should be positional-only (rename with a leading double underscore, i.e. "__attr")
_curses.window.attrset is inconsistent, stub argument "attr" should be positional-only (rename with a leading double underscore, i.e. "__attr")
_curses.window.bkgd is inconsistent, stub argument "ch" should be positional-only (rename with a leading double underscore, i.e. "__ch")
_curses.window.bkgd is inconsistent, stub argument "attr" should be positional-only (rename with a leading double underscore, i.e. "__attr")
_curses.window.bkgdset is inconsistent, stub argument "ch" should be positional-only (rename with a leading double underscore, i.e. "__ch")
_curses.window.bkgdset is inconsistent, stub argument "attr" should be positional-only (rename with a leading double underscore, i.e. "__attr")
_curses.window.echochar is inconsistent, stub argument "ch" should be positional-only (rename with a leading double underscore, i.e. "__ch")
_curses.window.echochar is inconsistent, stub argument "attr" should be positional-only (rename with a leading double underscore, i.e. "__attr")
_curses.window.enclose is inconsistent, stub argument "y" should be positional-only (rename with a leading double underscore, i.e. "__y")
_curses.window.enclose is inconsistent, stub argument "x" should be positional-only (rename with a leading double underscore, i.e. "__x")
_curses.window.is_linetouched is inconsistent, stub argument "line" should be positional-only (rename with a leading double underscore, i.e. "__line")
_curses.window.putwin is inconsistent, stub argument "file" should be positional-only (rename with a leading double underscore, i.e. "__file")
_curses.window.redrawln is inconsistent, stub argument "beg" should be positional-only (rename with a leading double underscore, i.e. "__beg")
_curses.window.redrawln is inconsistent, stub argument "num" should be positional-only (rename with a leading double underscore, i.e. "__num")
_curses.window.setscrreg is inconsistent, stub argument "top" should be positional-only (rename with a leading double underscore, i.e. "__top")
_curses.window.setscrreg is inconsistent, stub argument "bottom" should be positional-only (rename with a leading double underscore, i.e. "__bottom")
_weakref.ref.__call__ is inconsistent, stub does not have *args argument "args"
builtins.EnvironmentError.characters_written is not present in stub
builtins.IOError.characters_written is not present in stub
contextlib.AbstractContextManager.__exit__ is inconsistent, stub argument "__exc_type" should be positional or keyword (remove leading double underscore)
contextlib.AbstractContextManager.__exit__ is inconsistent, stub argument "__exc_value" should be positional or keyword (remove leading double underscore)
contextlib.AbstractContextManager.__exit__ is inconsistent, stub argument "__traceback" should be positional or keyword (remove leading double underscore)
curses.window.attroff is inconsistent, stub argument "attr" should be positional-only (rename with a leading double underscore, i.e. "__attr")
curses.window.attron is inconsistent, stub argument "attr" should be positional-only (rename with a leading double underscore, i.e. "__attr")
curses.window.attrset is inconsistent, stub argument "attr" should be positional-only (rename with a leading double underscore, i.e. "__attr")
curses.window.bkgd is inconsistent, stub argument "ch" should be positional-only (rename with a leading double underscore, i.e. "__ch")
curses.window.bkgd is inconsistent, stub argument "attr" should be positional-only (rename with a leading double underscore, i.e. "__attr")
curses.window.bkgdset is inconsistent, stub argument "ch" should be positional-only (rename with a leading double underscore, i.e. "__ch")
curses.window.bkgdset is inconsistent, stub argument "attr" should be positional-only (rename with a leading double underscore, i.e. "__attr")
curses.window.echochar is inconsistent, stub argument "ch" should be positional-only (rename with a leading double underscore, i.e. "__ch")
curses.window.echochar is inconsistent, stub argument "attr" should be positional-only (rename with a leading double underscore, i.e. "__attr")
curses.window.enclose is inconsistent, stub argument "y" should be positional-only (rename with a leading double underscore, i.e. "__y")
curses.window.enclose is inconsistent, stub argument "x" should be positional-only (rename with a leading double underscore, i.e. "__x")
curses.window.is_linetouched is inconsistent, stub argument "line" should be positional-only (rename with a leading double underscore, i.e. "__line")
curses.window.putwin is inconsistent, stub argument "file" should be positional-only (rename with a leading double underscore, i.e. "__file")
curses.window.redrawln is inconsistent, stub argument "beg" should be positional-only (rename with a leading double underscore, i.e. "__beg")
curses.window.redrawln is inconsistent, stub argument "num" should be positional-only (rename with a leading double underscore, i.e. "__num")
curses.window.setscrreg is inconsistent, stub argument "top" should be positional-only (rename with a leading double underscore, i.e. "__top")
curses.window.setscrreg is inconsistent, stub argument "bottom" should be positional-only (rename with a leading double underscore, i.e. "__bottom")
dbm.dumb.error.characters_written is not present in stub
ftplib.all_errors is not a type
multiprocessing.dummy.Condition.acquire is not present at runtime
multiprocessing.dummy.Condition.release is not present at runtime
multiprocessing.dummy.Event.isSet is not present in stub
multiprocessing.dummy.Lock is not a type
multiprocessing.dummy.RLock is not a type
multiprocessing.pool.AsyncResult.__init__ is inconsistent, stub does not have argument "pool"
multiprocessing.pool.AsyncResult.__init__ is inconsistent, stub does not have argument "callback"
multiprocessing.pool.AsyncResult.__init__ is inconsistent, stub does not have argument "error_callback"
os.error.characters_written is not present in stub
resource.error.characters_written is not present in stub
select.error.characters_written is not present in stub
socket.error.characters_written is not present in stub
sqlite3.Binary.__contains__ is not present at runtime
sqlite3.Binary.__iter__ is not present at runtime
sqlite3.Binary.cast is inconsistent, stub argument "shape" is not keyword-only
sqlite3.dbapi2.Binary.__contains__ is not present at runtime
sqlite3.dbapi2.Binary.__iter__ is not present at runtime
sqlite3.dbapi2.Binary.cast is inconsistent, stub argument "shape" is not keyword-only
sre_parse.State.opengroup is inconsistent, runtime argument "name" has a default value of type None, which is incompatible with stub argument type builtins.str
threading.RLock is not a type
typing.ChainMap is not a type
typing.Counter is not a type
typing.DefaultDict is not a type
typing.Deque is not a type
typing.Dict is not a type
typing.FrozenSet is not a type
typing.List is not a type
typing.OrderedDict is not a type
typing.Set is not a type
typing_extensions.ChainMap is not a type
typing_extensions.Counter is not a type
typing_extensions.DefaultDict is not a type
typing_extensions.Deque is not a type
weakref.ref.__call__ is inconsistent, stub does not have *args argument "args"
xml.dom.minidom.Text.__add__ is not present at runtime
xml.dom.minidom.Text.__contains__ is not present at runtime
xml.dom.minidom.Text.__getitem__ is not present at runtime
xml.dom.minidom.Text.__getnewargs__ is not present at runtime
xml.dom.minidom.Text.__iter__ is not present at runtime
xml.dom.minidom.Text.__mod__ is not present at runtime
xml.dom.minidom.Text.__mul__ is not present at runtime
xml.dom.minidom.Text.__rmul__ is not present at runtime
xml.dom.minidom.Text.attributes is not present in stub
xml.dom.minidom.Text.capitalize is not present at runtime
xml.dom.minidom.Text.casefold is not present at runtime
xml.dom.minidom.Text.center is not present at runtime
xml.dom.minidom.Text.count is not present at runtime
xml.dom.minidom.Text.encode is not present at runtime
xml.dom.minidom.Text.endswith is not present at runtime
xml.dom.minidom.Text.expandtabs is not present at runtime
xml.dom.minidom.Text.find is not present at runtime
xml.dom.minidom.Text.format is not present at runtime
xml.dom.minidom.Text.format_map is not present at runtime
xml.dom.minidom.Text.index is not present at runtime
xml.dom.minidom.Text.isWhitespaceInElementContent is not present in stub
xml.dom.minidom.Text.isalnum is not present at runtime
xml.dom.minidom.Text.isalpha is not present at runtime
xml.dom.minidom.Text.isascii is not present at runtime
xml.dom.minidom.Text.isdecimal is not present at runtime
xml.dom.minidom.Text.isdigit is not present at runtime
xml.dom.minidom.Text.isidentifier is not present at runtime
xml.dom.minidom.Text.islower is not present at runtime
xml.dom.minidom.Text.isnumeric is not present at runtime
xml.dom.minidom.Text.isprintable is not present at runtime
xml.dom.minidom.Text.isspace is not present at runtime
xml.dom.minidom.Text.istitle is not present at runtime
xml.dom.minidom.Text.isupper is not present at runtime
xml.dom.minidom.Text.join is not present at runtime
xml.dom.minidom.Text.ljust is not present at runtime
xml.dom.minidom.Text.lower is not present at runtime
xml.dom.minidom.Text.lstrip is not present at runtime
xml.dom.minidom.Text.maketrans is not present at runtime
xml.dom.minidom.Text.nodeName is not present in stub
xml.dom.minidom.Text.nodeType is not present in stub
xml.dom.minidom.Text.partition is not present at runtime
xml.dom.minidom.Text.replace is not present at runtime
xml.dom.minidom.Text.replaceWholeText is not present in stub
xml.dom.minidom.Text.rfind is not present at runtime
xml.dom.minidom.Text.rindex is not present at runtime
xml.dom.minidom.Text.rjust is not present at runtime
xml.dom.minidom.Text.rpartition is not present at runtime
xml.dom.minidom.Text.rsplit is not present at runtime
xml.dom.minidom.Text.rstrip is not present at runtime
xml.dom.minidom.Text.split is not present at runtime
xml.dom.minidom.Text.splitText is not present in stub
xml.dom.minidom.Text.splitlines is not present at runtime
xml.dom.minidom.Text.startswith is not present at runtime
xml.dom.minidom.Text.strip is not present at runtime
xml.dom.minidom.Text.swapcase is not present at runtime
xml.dom.minidom.Text.title is not present at runtime
xml.dom.minidom.Text.translate is not present at runtime
xml.dom.minidom.Text.upper is not present at runtime
xml.dom.minidom.Text.wholeText is not present in stub
xml.dom.minidom.Text.writexml is not present in stub
xml.dom.minidom.Text.zfill is not present at runtime
```
</details>

Also as a fun bonus, here's an interesting Python bug:
```
import _curses
import inspect
inspect.signature(_curses.window.border)
```

This is the patch:
<details>

```
diff --git a/mypy/stubtest.py b/mypy/stubtest.py
index db8666c94..80c71e831 100644
--- a/mypy/stubtest.py
+++ b/mypy/stubtest.py
@@ -666,7 +666,7 @@ def verify_funcitem(
 
     try:
         signature = inspect.signature(runtime)
-    except (ValueError, RuntimeError):
+    except (ValueError, RuntimeError, AttributeError):
         # inspect.signature throws sometimes
         # catch RuntimeError because of https://bugs.python.org/issue39504
         return
@@ -869,8 +869,11 @@ def verify_decorator(
 def verify_typealias(
     stub: nodes.TypeAlias, runtime: MaybeMissing[Any], object_path: List[str]
 ) -> Iterator[Error]:
-    if False:
-        yield None
+    if isinstance(runtime, Missing):
+        # ignore type aliases that don't have a runtime counterpart
+        return
+    if isinstance(stub.target, mypy.types.Instance):
+        yield from verify(stub.target.type, runtime, object_path)
 
 
 SPECIAL_DUNDERS = ("__init__", "__new__", "__call__", "__init_subclass__", "__class_getitem__")
```
</details>

Resolving patch:
diff --git a/mypy/stubtest.py b/mypy/stubtest.py
--- a/mypy/stubtest.py
+++ b/mypy/stubtest.py
@@ -858,8 +858,31 @@ def verify_decorator(
 def verify_typealias(
     stub: nodes.TypeAlias, runtime: MaybeMissing[Any], object_path: List[str]
 ) -> Iterator[Error]:
-    if False:
-        yield None
+    if isinstance(runtime, Missing):
+        # ignore type aliases that don't have a runtime counterpart
+        return
+    stub_target = mypy.types.get_proper_type(stub.target)
+    if isinstance(stub_target, mypy.types.Instance):
+        yield from verify(stub_target.type, runtime, object_path)
+        return
+    if isinstance(stub_target, mypy.types.UnionType):
+        if not getattr(runtime, "__origin__", None) is Union:
+            yield Error(object_path, "is not a Union", stub, runtime, stub_desc=str(stub_target))
+        # could check Union contents here...
+        return
+    if isinstance(stub_target, mypy.types.TupleType):
+        if tuple not in getattr(runtime, "__mro__", ()):
+            yield Error(
+                object_path, "is not a subclass of tuple", stub, runtime,
+                stub_desc=str(stub_target)
+            )
+        # could check Tuple contents here...
+        return
+    if isinstance(stub_target, mypy.types.AnyType):
+        return
+    yield Error(
+        object_path, "is not a recognised type alias", stub, runtime, stub_desc=str(stub_target)
+    )
 
 
 SPECIAL_DUNDERS = ("__init__", "__new__", "__call__", "__init_subclass__", "__class_getitem__")
@@ -887,10 +910,11 @@ def is_probably_a_function(runtime: Any) -> bool:
 def safe_inspect_signature(runtime: Any) -> Optional[inspect.Signature]:
     try:
         return inspect.signature(runtime)
-    except (ValueError, RuntimeError, TypeError):
-        # inspect.signature throws sometimes
+    except Exception:
+        # inspect.signature throws ValueError all the time
         # catch RuntimeError because of https://bugs.python.org/issue39504
         # catch TypeError because of https://github.com/python/typeshed/pull/5762
+        # catch AttributeError because of inspect.signature(_curses.window.border)
         return None
 
 


Test patch:
diff --git a/mypy/test/teststubtest.py b/mypy/test/teststubtest.py
--- a/mypy/test/teststubtest.py
+++ b/mypy/test/teststubtest.py
@@ -571,6 +571,30 @@ def __init__(self):
             error=None,
         )
 
+    @collect_cases
+    def test_type_alias(self) -> Iterator[Case]:
+        yield Case(
+            stub="""
+            class X:
+                def f(self) -> None: ...
+            Y = X
+            """,
+            runtime="""
+            class X:
+                def f(self) -> None: ...
+            class Y: ...
+            """,
+            error="Y.f",
+        )
+        yield Case(
+            stub="""
+            from typing import Tuple
+            A = Tuple[int, str]
+            """,
+            runtime="A = (int, str)",
+            error="A",
+        )
+
     @collect_cases
     def test_enum(self) -> Iterator[Case]:
         yield Case(