Repository: python/mypy

Issue:
stubtest crashes when testing django-types
Hello.

I want to veryfy [django-types](https://github.com/sbdchd/django-types) with stubtest. But it crashes with following message:

**Traceback**

```
Traceback (most recent call last):
  File "/home/serg/.cache/pypoetry/virtualenvs/django-types-BW7xHCiR-py3.10/bin/stubtest", line 8, in <module>
    sys.exit(main())
  File "/home/serg/.cache/pypoetry/virtualenvs/django-types-BW7xHCiR-py3.10/lib/python3.10/site-packages/mypy/stubtest.py", line 1260, in main
    return test_stubs(parse_options(sys.argv[1:]))
  File "/home/serg/.cache/pypoetry/virtualenvs/django-types-BW7xHCiR-py3.10/lib/python3.10/site-packages/mypy/stubtest.py", line 1153, in test_stubs
    for error in test_module(module):
  File "/home/serg/.cache/pypoetry/virtualenvs/django-types-BW7xHCiR-py3.10/lib/python3.10/site-packages/mypy/stubtest.py", line 180, in test_module
    yield from verify(stub, runtime, [module_name])
  File "/home/serg/.cache/pypoetry/virtualenvs/django-types-BW7xHCiR-py3.10/lib/python3.10/site-packages/mypy/stubtest.py", line 229, in verify_mypyfile
    yield from verify(
  File "/home/serg/.cache/pypoetry/virtualenvs/django-types-BW7xHCiR-py3.10/lib/python3.10/site-packages/mypy/stubtest.py", line 698, in verify_var
    if isinstance(runtime, Missing):
  File "/home/serg/.cache/pypoetry/virtualenvs/django-types-BW7xHCiR-py3.10/lib/python3.10/site-packages/django/utils/functional.py", line 240, in inner
    self._setup()
  File "/home/serg/.cache/pypoetry/virtualenvs/django-types-BW7xHCiR-py3.10/lib/python3.10/site-packages/django/contrib/admin/sites.py", line 540, in _setup
    AdminSiteClass = import_string(apps.get_app_config('admin').default_site)
  File "/home/serg/.cache/pypoetry/virtualenvs/django-types-BW7xHCiR-py3.10/lib/python3.10/site-packages/django/apps/registry.py", line 154, in get_app_config
    self.check_apps_ready()
  File "/home/serg/.cache/pypoetry/virtualenvs/django-types-BW7xHCiR-py3.10/lib/python3.10/site-packages/django/apps/registry.py", line 136, in check_apps_ready
    raise AppRegistryNotReady("Apps aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.
```

**To Reproduce**

Clone django-types, install django and mypy, run `stubtest django` from django-types directory.

**Problem**

Main problem in this case - is django refusing to work without explicit setup, and `django-stubs` fixes it with their `mypy_django_plugin`, but `django-types` doesn't use that plugin.

What do you think, how to fix this error?

I propose doing something like this:

```patch
From 513d6a3655634074c7894168efb92e689bf950f1 Mon Sep 17 00:00:00 2001
From: Serg Tereshchenko <serg.partizan@gmail.com>
Date: Thu, 2 Jun 2022 20:38:05 +0300
Subject: [PATCH] fix: Handle runtime errors during verification

---
 mypy/stubtest.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/mypy/stubtest.py b/mypy/stubtest.py
index a85e9335a..ad0a7e78d 100644
--- a/mypy/stubtest.py
+++ b/mypy/stubtest.py
@@ -187,7 +187,10 @@ def test_module(module_name: str) -> Iterator[Error]:

     with warnings.catch_warnings():
         warnings.simplefilter("ignore")
-        yield from verify(stub, runtime, [module_name])
+        try:
+            yield from verify(stub, runtime, [module_name])
+        except Exception as err:
+            yield Error([module_name], f"Runtime error: {err}", stub, runtime)


 @singledispatch
--
2.36.1
```

I could hide this behaviour behind a flag, if you don't think this should be default. But this is really helpful for testing django stubs, unless we can find other ideas.


Discussion:
Cc. @hauntsaninja

Resolving patch:
diff --git a/mypy/stubtest.py b/mypy/stubtest.py
--- a/mypy/stubtest.py
+++ b/mypy/stubtest.py
@@ -13,6 +13,7 @@
 import pkgutil
 import re
 import sys
+import traceback
 import types
 import typing
 import typing_extensions
@@ -201,7 +202,28 @@ def test_module(module_name: str) -> Iterator[Error]:
 
     with warnings.catch_warnings():
         warnings.simplefilter("ignore")
-        yield from verify(stub, runtime, [module_name])
+        try:
+            yield from verify(stub, runtime, [module_name])
+        except Exception as e:
+            bottom_frame = list(traceback.walk_tb(e.__traceback__))[-1][0]
+            bottom_module = bottom_frame.f_globals.get("__name__", "")
+            # Pass on any errors originating from stubtest or mypy
+            # These can occur expectedly, e.g. StubtestFailure
+            if bottom_module == "__main__" or bottom_module.split(".")[0] == "mypy":
+                raise
+            yield Error(
+                [module_name],
+                f"encountered unexpected error, {type(e).__name__}: {e}",
+                stub,
+                runtime,
+                stub_desc="N/A",
+                runtime_desc=(
+                    "This is most likely the fault of something very dynamic in your library. "
+                    "It's also possible this is a bug in stubtest.\nIf in doubt, please "
+                    "open an issue at https://github.com/python/mypy\n\n"
+                    + traceback.format_exc().strip()
+                ),
+            )
 
 
 @singledispatch


Test patch: