Repository: python/mypy

Issue:
overload hides 'is not a function' error in stubtest
**Bug Report**

I've noticed that overloading a function can hide a mismatch between the runtime type and the stub type. I noticed this as I was going over typeshed stubtest exceptions, specifically for itertools.accumulate.
Tests for python 3.6 and 3.7 failed because the stubs declared accumulate as a function while it was actually a class. However for 3.8 it is defined as two functions with overload and the tests passed successfully.

**To Reproduce**

(Write your steps here:)

1. Run stubtest for typeshed in python 3.8.
2. Test passes.
3. Remove one of the overloads in itertools.pyi
4. run again and see it fails.

**Expected Behavior**

I t should fail on a type mismatch like it does for no overloads.

**Actual Behavior**

passed.

**Your Environment**

Ran from latest python 3.8 and latest mypy from github.



Discussion:
Thanks for reporting! This is easy to fix (just ensure `verify_overloadedfuncdef` makes the same check as `verify_funcitem`). That said, none of the dozen or so new errors on stdlib you get after fixing this are useful... arguably if you've gotten to the point of writing overloads, you're probably conscious of the fact that you're pretending the object is a function, so maybe this bug is a feature?
Even if the issues this currently finds are not too useful, it feels better for stubtest to consistently check everything it could check instead of just giving up. I can imagine some useful checks it could do with overloads, like checking that all overloads are at least a subset of possible useful calls.

Resolving patch:
diff --git a/mypy/stubtest.py b/mypy/stubtest.py
--- a/mypy/stubtest.py
+++ b/mypy/stubtest.py
@@ -265,7 +265,7 @@ def verify_typeinfo(
 
 
 def _verify_static_class_methods(
-    stub: nodes.FuncItem, runtime: types.FunctionType, object_path: List[str]
+    stub: nodes.FuncBase, runtime: Any, object_path: List[str]
 ) -> Iterator[str]:
     if stub.name in ("__new__", "__init_subclass__", "__class_getitem__"):
         # Special cased by Python, so don't bother checking
@@ -646,18 +646,20 @@ def _verify_signature(
 
 @verify.register(nodes.FuncItem)
 def verify_funcitem(
-    stub: nodes.FuncItem, runtime: MaybeMissing[types.FunctionType], object_path: List[str]
+    stub: nodes.FuncItem, runtime: MaybeMissing[Any], object_path: List[str]
 ) -> Iterator[Error]:
     if isinstance(runtime, Missing):
         yield Error(object_path, "is not present at runtime", stub, runtime)
         return
+
     if (
         not isinstance(runtime, (types.FunctionType, types.BuiltinFunctionType))
         and not isinstance(runtime, (types.MethodType, types.BuiltinMethodType))
         and not inspect.ismethoddescriptor(runtime)
     ):
         yield Error(object_path, "is not a function", stub, runtime)
-        return
+        if not callable(runtime):
+            return
 
     for message in _verify_static_class_methods(stub, runtime, object_path):
         yield Error(object_path, "is inconsistent, " + message, stub, runtime)
@@ -734,6 +736,18 @@ def verify_overloadedfuncdef(
         # We get here in cases of overloads from property.setter
         return
 
+    if (
+        not isinstance(runtime, (types.FunctionType, types.BuiltinFunctionType))
+        and not isinstance(runtime, (types.MethodType, types.BuiltinMethodType))
+        and not inspect.ismethoddescriptor(runtime)
+    ):
+        yield Error(object_path, "is not a function", stub, runtime)
+        if not callable(runtime):
+            return
+
+    for message in _verify_static_class_methods(stub, runtime, object_path):
+        yield Error(object_path, "is inconsistent, " + message, stub, runtime)
+
     try:
         signature = inspect.signature(runtime)
     except ValueError:


Test patch: