Repository: python/mypy

Issue:
Uncertainty between submodule and local definition in new semantic analyzer
Consider this test case:
```
[case testNewAnalyzerImportFromTopLevelAlias]
import a.b
[file a/__init__.py]
from .b import B
from . import b as c
b = int
x: Forward
class Forward:
    ...

[file a/b.py]
class B: ...
[builtins fixtures/module.pyi]
```
It currently fails with `Name 'c' is already defined`, because on the first iteration mypy sees `a.b` as module, while on the second one it is a type alias. There is also a similar but different test case:
```
[case testNewAnalyzerImportFromTopLevelFunction]
import a.b
[file a/__init__.py]
from .b import B
from . import b as c
def b(): ...  # can be also b: int
x: Forward
class Forward:
    ...

[file a/b.py]
class B: ...
[builtins fixtures/module.pyi]
```
that fails with `Name 'b' already defined` for a similar reason. Note that we already added some special casing for a similar scenario, see https://github.com/python/mypy/pull/6630, to match old semantic analyzer (and also Python runtime). Setting priority to high since this causes problems with internal deployment of new analyzer. @JukkaL I assign this to you since this may be related to tracking the order of definitions.


Discussion:
It looks like the approach taken in #6630 isn't workable and needs to be replaced with something more general. The new semantic analyzer relies on only creating a non-placeholder symbol node once per name (#6300) and submodules shouldn't deviate from this, since any deviations increase complexity and likely cause bugs.

My current idea is that implicit submodule references shouldn't be added to symbol tables. Instead, we could look them up from a separate "module scope". If a name is defined in a module, we use that. Otherwise, we fall back to the module dictionary to check if the name refers to a module. 

To implement this logic, we'd need to keep track of where a definition is valid using the line number of the definition.

There are various edge cases that are still unclear to me and need more thought.
> My current idea is that implicit submodule references shouldn't be added to symbol tables. Instead, we could look them up from a separate "module scope". If a name is defined in a module, we use that. Otherwise, we fall back to the module dictionary to check if the name refers to a module.

Generally I like this idea. I was thinking about something similar, but even more global. There is another case where we want to have essentially two nodes in the same symbol table under the same name: the `-redefinition` thing. (Yet another will be conditional imports currently unsupported.)

We can make a symbol table essentially a mapping from names to a deque of nodes, but `__getitem__` will always return the leading one. There will be a method to access all of them. Plus two methods to push on either end, depending on whether we want to shadow existing name (like with named tuples or plugins), or not (like for redefinitions or for implicit imports).
Yet another related thing to consider is `--allow-redefinition`. If we are going to move towards something more SSA-like (e.g. allowing conditional redefinitions), then the current scheme with variable renaming with primes, `x`, `x'`, `x''`, etc., might not work well. We will need to be able to record that in
```python
if bool():
    x = int()
else:
    x = str()
foo(x)
```
the last `x` may refer to either `Var` node. With the deque logic this seems possible to address.

Resolving patch:
diff --git a/mypy/newsemanal/semanal.py b/mypy/newsemanal/semanal.py
--- a/mypy/newsemanal/semanal.py
+++ b/mypy/newsemanal/semanal.py
@@ -1628,10 +1628,20 @@ def visit_import_from(self, imp: ImportFrom) -> None:
         import_id = self.correct_relative_import(imp)
         module = self.modules.get(import_id)
         for id, as_id in imp.names:
-            node = module.names.get(id) if module else None
+            possible_module_id = import_id + '.' + id
+            if module is None:
+                node = None
+            elif import_id == self.cur_mod_id and possible_module_id in self.modules:
+                # Submodule takes precedence over definition in surround package, for
+                # compatibility with runtime semantics in typical use cases. This
+                # could more precisely model runtime semantics by taking into account
+                # the line number beyond which the local definition should take
+                # precedence, but doesn't seem to be important in most use cases.
+                node = SymbolTableNode(GDEF, self.modules[possible_module_id])
+            else:
+                node = module.names.get(id)
 
             missing = False
-            possible_module_id = import_id + '.' + id
             imported_id = as_id or id
 
             # If the module does not contain a symbol with the name 'id',


Test patch:
diff --git a/test-data/unit/check-newsemanal.test b/test-data/unit/check-newsemanal.test
--- a/test-data/unit/check-newsemanal.test
+++ b/test-data/unit/check-newsemanal.test
@@ -2516,3 +2516,72 @@ def get() -> int: ...
 import typing
 t = typing.typevar('t') # E: Module has no attribute "typevar"
 [builtins fixtures/module.pyi]
+
+[case testNewAnalyzerImportFromTopLevelFunction]
+import a.b  # This works at runtime
+reveal_type(a.b)  # N
+[file a/__init__.py]
+from .b import B
+from . import b as c
+def b() -> None: pass
+reveal_type(b)  # N
+reveal_type(c.B())  # N
+x: Forward
+class Forward:
+    ...
+
+[file a/b.py]
+class B: ...
+[builtins fixtures/module.pyi]
+
+[out]
+tmp/a/__init__.py:4: note: Revealed type is 'def ()'
+tmp/a/__init__.py:5: note: Revealed type is 'a.b.B'
+main:2: note: Revealed type is 'def ()'
+
+[case testNewAnalyzerImportFromTopLevelAlias]
+import a.b  # This works at runtime
+reveal_type(a.b)  # N
+[file a/__init__.py]
+from .b import B
+from . import b as c
+b = int
+y: b
+reveal_type(y)  # N
+reveal_type(c.B)  # N
+x: Forward
+class Forward:
+    ...
+
+[file a/b.py]
+class B: ...
+[builtins fixtures/module.pyi]
+
+[out]
+tmp/a/__init__.py:5: note: Revealed type is 'builtins.int'
+tmp/a/__init__.py:6: note: Revealed type is 'def () -> a.b.B'
+main:2: note: Revealed type is 'def () -> builtins.int'
+
+[case testNewAnalyzerImportAmbiguousWithTopLevelFunction]
+import a.b  # This works at runtime
+x: a.b.B  # E
+reveal_type(a.b)  # N
+[file a/__init__.py]
+import a.b
+import a.b as c
+def b() -> None: pass
+reveal_type(b)  # N
+reveal_type(c.B())  # N
+x: Forward
+class Forward:
+    ...
+
+[file a/b.py]
+class B: ...
+[builtins fixtures/module.pyi]
+
+[out]
+tmp/a/__init__.py:4: note: Revealed type is 'def ()'
+tmp/a/__init__.py:5: note: Revealed type is 'a.b.B'
+main:2: error: Name 'a.b.B' is not defined
+main:3: note: Revealed type is 'def ()'