Repository: python/mypy

Issue:
"Invalid Type" error in v0.600 on uncached runs involving file with same name as parent package
When I run mypy on my project, I sometimes get an invalid type error. This error seems to go away over time (some kind of caching issue?), or if I avoid having a file in a package of the same name, or if I do an explicit direct import of the type from the file instead of importing it via the package.

This is kind of a complicated bug to reproduce, but here is my best attempt at making a succinct repro. We will need five files.

0. Create a fresh virtual environment and test project directory.

    ```bash
    mkvirtualenv repro --python=/usr/bin/python3
    pip install mypy
   mkdir /tmp/whatever
   cd /tmp/whatever
   mkdir project
   ```

1. Have a `project/__init__.py` file containing

    ```
    from project.study import CustomType
    ```

2. Have an empty `project/root.py` file.

3. Have a `project/neighbor/__init__.py` file containing

    ```python
    from project.study import CustomType
    def m(arg: CustomType) -> str:
        del arg
        return 'test'
    ```

4. Have a `project/study/__init__.py` file containing

    ```python
    from project.study.study import CustomType
    ```

5. Have a `project/study/study.py` file containing

    ```python
    from project import root
    CustomType = str
    ```

    Note: renaming this file to something that doesn't match its directory's name will cause the repro to fail.

6. (If you already ran mypy in this directory.) Clear the mypy cache.

    ```bash
    rm -rf .mypy_cache
    ```

7. Run mypy on *exactly these files in exactly this order*.

    ```bash
    mypy project/root.py project/study/study.py project/neighbor/__init__.py
   ```

8. And you will get this output:

    ```
    project/neighbor/__init__.py:4: error: Invalid type "project.study.study.CustomType"
    ```

    If you run mypy again without clearing its cache, then there will be no error.

    or, depending on factors I don't understand you may get this different error instead:

    ```
    project/neighbor/__init__.py:2: error: Internal error (node is None, kind=1)
    ```

An easy workaround for this issue is to avoid defining types in files that have the same name as their parent directory. Everyone likes avoiding cache invalidation issues by naming things, right?


Discussion:
Thanks for the detailed reproduction instructions!
Hm, I wonder what the cache contents are for those five files?
I was able to get this error to appear (including follow up internal errors) even if I renamed the submodule?
Bisecting shows that this is a regression from 0.590 introduced by #4910. I'll take a look.

Resolving patch:
diff --git a/mypy/build.py b/mypy/build.py
--- a/mypy/build.py
+++ b/mypy/build.py
@@ -713,16 +713,22 @@ def correct_rel_imp(imp: Union[ImportFrom, ImportAll]) -> str:
                 elif isinstance(imp, ImportFrom):
                     cur_id = correct_rel_imp(imp)
                     pos = len(res)
+                    all_are_submodules = True
                     # Also add any imported names that are submodules.
                     pri = import_priority(imp, PRI_MED)
                     for name, __ in imp.names:
                         sub_id = cur_id + '.' + name
                         if self.is_module(sub_id):
                             res.append((pri, sub_id, imp.line))
+                        else:
+                            all_are_submodules = False
                     # Add cur_id as a dependency, even if all of the
                     # imports are submodules. Processing import from will try
                     # to look through cur_id, so we should depend on it.
-                    pri = import_priority(imp, PRI_HIGH)
+                    # As a workaround for for some bugs in cycle handling (#4498),
+                    # if all of the imports are submodules, do the import at a lower
+                    # priority.
+                    pri = import_priority(imp, PRI_HIGH if not all_are_submodules else PRI_LOW)
                     res.insert(pos, ((pri, cur_id, imp.line)))
                 elif isinstance(imp, ImportAll):
                     pri = import_priority(imp, PRI_HIGH)


Test patch:
diff --git a/test-data/unit/check-modules.test b/test-data/unit/check-modules.test
--- a/test-data/unit/check-modules.test
+++ b/test-data/unit/check-modules.test
@@ -1001,6 +1001,48 @@ from foo import bar
 [file foo/bar.py]
 pass
 
+[case testImportReExportFromChildrenInCycle1]
+# cmd: mypy -m project.root project.study.a project.neighbor
+[file project/__init__.py]
+from project.study import CustomType
+x = 10
+[file project/root.py]
+[file project/study/__init__.py]
+from project.study.a import CustomType
+[file project/study/a.py]
+from project import root
+# TODO (#4498): This test is basically testing the `all_are_submodules` logic
+# in build, which skips generating a dependenecy to a module if
+# everything in it is a submodule. But that is still all just a
+# workaround for bugs in cycle handling. If we uncomment the next
+# line, we'll still break:
+# from project import x
+CustomType = str
+[file project/neighbor/__init__.py]
+from project.study import CustomType
+def m(arg: CustomType) -> str:
+    return 'test'
+
+[case testImportReExportFromChildrenInCycle2]
+# cmd: mypy -m project project.b project.ba project.c
+# See comments in above test about this being a workaround.
+[file foo.py]
+def get_foo() -> int: return 12
+
+[file project/ba.py]
+from . import b
+b.FOO
+
+[file project/b.py]
+import foo
+from . import c
+FOO = foo.get_foo()
+
+[file project/c.py]
+
+[file project/__init__.py]
+from . import ba
+
 [case testSuperclassInImportCycle]
 import a
 import d