Repository: pantsbuild/pants

Issue:
Scala partially qualified symbol not inferred
Code like:
```scala
import akka.http.javadsl.{ model => jm }

sealed trait ContentRange extends jm.ContentRange ... {
````
...will not infer a dependency on the symbol `akka.http.javadsl.model.ContentRange`.


Discussion:
This represented roughly 12% of the issues in `akka-http`.
It seems like completely solving this would amount to implementing symbol resolution such that we could differentiate between using a field of a local variable (`val lib = ...`) and using a relative import (`import x.lib`).

But since we do not need 100% accuracy here, it seems like an imports-only-aware bit of resolution would be feasible. To do that, I'm thinking of adding an `alias` field to [our import definition](https://github.com/pantsbuild/pants/blob/4a395dbd7f49acdd16e5691da13f9a84a6151519/src/python/pants/backend/scala/dependency_inference/ScalaParser.scala#L13) which would record the import's scope-specific name (to handle the case in the description), and then emitting potentially consumed symbols for any `consumed_symbols` which are relative to (i.e., start-with) any in-scope import aliases. Thoughts @tdyas , @chrisjrn ?

Also, while it's been useful to have more data emitted as JSON to Python, it occurs to me that we could be doing ~all of the [consumed symbols calculations](https://github.com/pantsbuild/pants/blob/4a395dbd7f49acdd16e5691da13f9a84a6151519/src/python/pants/backend/scala/dependency_inference/scala_parser.py#L140-L155) inline in the parser (or at least, before emitting JSON).
> Also, while it's been useful to have more data emitted as JSON to Python, it occurs to me that we could be doing ~all of the consumed symbols calculations inline in the parser (or at least, before emitting JSON).

That gets into partial type solving which is something that I had looked into a little but punted on as JavaParser’s existing code wants to completely type solve and it would’ve required a fair bit of work to try and do partial type solving. For Scala, I never got the chance to see what scalameta had for type solving. Definitely a path that we had considered earlier but could be a fair bit of work.

Resolving patch:
diff --git a/src/python/pants/backend/java/compile/javac.py b/src/python/pants/backend/java/compile/javac.py
--- a/src/python/pants/backend/java/compile/javac.py
+++ b/src/python/pants/backend/java/compile/javac.py
@@ -52,7 +52,7 @@ async def compile_java_source(
     union_membership: UnionMembership,
     request: CompileJavaSourceRequest,
 ) -> FallibleClasspathEntry:
-    # Request the component's direct dependency classpath, and additionally any preqrequisite.
+    # Request the component's direct dependency classpath, and additionally any prerequisite.
     classpath_entry_requests = [
         *((request.prerequisite,) if request.prerequisite else ()),
         *(
diff --git a/src/python/pants/backend/scala/dependency_inference/scala_parser.py b/src/python/pants/backend/scala/dependency_inference/scala_parser.py
--- a/src/python/pants/backend/scala/dependency_inference/scala_parser.py
+++ b/src/python/pants/backend/scala/dependency_inference/scala_parser.py
@@ -96,15 +96,17 @@
 @dataclass(frozen=True)
 class ScalaImport:
     name: str
+    alias: str | None
     is_wildcard: bool
 
     @classmethod
     def from_json_dict(cls, data: Mapping[str, Any]):
-        return cls(name=data["name"], is_wildcard=data["isWildcard"])
+        return cls(name=data["name"], alias=data.get("alias"), is_wildcard=data["isWildcard"])
 
     def to_debug_json_dict(self) -> dict[str, Any]:
         return {
             "name": self.name,
+            "alias": self.alias,
             "is_wildcard": self.is_wildcard,
         }
 
@@ -130,29 +132,40 @@ def fully_qualified_consumed_symbols(self) -> Iterator[str]:
         have been provided by any wildcard import in scope, as well as being declared in the current
         package.
         """
-        # Collect all wildcard imports.
-        wildcard_imports_by_scope = {}
-        for scope, imports in self.imports_by_scope.items():
-            wildcard_imports = tuple(imp for imp in imports if imp.is_wildcard)
-            if wildcard_imports:
-                wildcard_imports_by_scope[scope] = wildcard_imports
-
-        for scope, consumed_symbols in self.consumed_symbols_by_scope.items():
-            parent_scope_wildcards = {
-                wi
-                for s, wildcard_imports in wildcard_imports_by_scope.items()
-                for wi in wildcard_imports
-                if scope.startswith(s)
-            }
+
+        def scope_and_parents(scope: str) -> Iterator[str]:
+            while True:
+                yield scope
+                if scope == "":
+                    break
+                scope, _, _ = scope.rpartition(".")
+
+        for consumption_scope, consumed_symbols in self.consumed_symbols_by_scope.items():
+            parent_scopes = tuple(scope_and_parents(consumption_scope))
             for symbol in consumed_symbols:
-                for scope in self.scopes:
-                    yield f"{scope}.{symbol}"
-                if not self.scopes or "." in symbol:
+                symbol_rel_prefix, dot_in_symbol, symbol_rel_suffix = symbol.partition(".")
+                if not self.scopes or dot_in_symbol:
                     # TODO: Similar to #13545: we assume that a symbol containing a dot might already
                     # be fully qualified.
                     yield symbol
-                for wildcard_scope in parent_scope_wildcards:
-                    yield f"{wildcard_scope.name}.{symbol}"
+                for parent_scope in parent_scopes:
+                    if parent_scope in self.scopes:
+                        # A package declaration is a parent of this scope, and any of its symbols
+                        # could be in scope.
+                        yield f"{parent_scope}.{symbol}"
+
+                    for imp in self.imports_by_scope.get(parent_scope, ()):
+                        if imp.is_wildcard:
+                            # There is a wildcard import in a parent scope.
+                            yield f"{imp.name}.{symbol}"
+                        if dot_in_symbol:
+                            # If the parent scope has an import which defines the first token of the
+                            # symbol, then it might be a relative usage of an import.
+                            if imp.alias:
+                                if imp.alias == symbol_rel_prefix:
+                                    yield f"{imp.name}.{symbol_rel_suffix}"
+                            elif imp.name.endswith(f".{symbol_rel_prefix}"):
+                                yield f"{imp.name}.{symbol_rel_suffix}"
 
     @classmethod
     def from_json_dict(cls, d: dict) -> ScalaSourceDependencyAnalysis:


Test patch:
diff --git a/src/python/pants/backend/scala/dependency_inference/scala_parser_test.py b/src/python/pants/backend/scala/dependency_inference/scala_parser_test.py
--- a/src/python/pants/backend/scala/dependency_inference/scala_parser_test.py
+++ b/src/python/pants/backend/scala/dependency_inference/scala_parser_test.py
@@ -29,7 +29,6 @@
 @pytest.fixture
 def rule_runner() -> RuleRunner:
     rule_runner = RuleRunner(
-        preserve_tmpdirs=True,
         rules=[
             *scala_parser.rules(),
             *coursier_fetch_rules(),
@@ -48,19 +47,35 @@ def rule_runner() -> RuleRunner:
     return rule_runner
 
 
-def test_parser_simple(rule_runner: RuleRunner) -> None:
+def _analyze(rule_runner: RuleRunner, source: str) -> ScalaSourceDependencyAnalysis:
     rule_runner.write_files(
         {
-            "BUILD": textwrap.dedent(
-                """
-            scala_source(
-                name="simple-source",
-                source="SimpleSource.scala",
+            "BUILD": """scala_source(name="source", source="Source.scala")""",
+            "Source.scala": source,
+        }
+    )
+
+    target = rule_runner.get_target(address=Address("", target_name="source"))
+
+    source_files = rule_runner.request(
+        SourceFiles,
+        [
+            SourceFilesRequest(
+                (target.get(SourcesField),),
+                for_sources_types=(ScalaSourceField,),
+                enable_codegen=True,
             )
+        ],
+    )
+
+    return rule_runner.request(ScalaSourceDependencyAnalysis, [source_files])
+
+
+def test_parser_simple(rule_runner: RuleRunner) -> None:
+    analysis = _analyze(
+        rule_runner,
+        textwrap.dedent(
             """
-            ),
-            "SimpleSource.scala": textwrap.dedent(
-                """
             package org.pantsbuild
             package example
 
@@ -124,26 +139,7 @@ def this(bar: SomeTypeInSecondaryConstructor) {
                }
             }
             """
-            ),
-        }
-    )
-
-    target = rule_runner.get_target(address=Address("", target_name="simple-source"))
-
-    source_files = rule_runner.request(
-        SourceFiles,
-        [
-            SourceFilesRequest(
-                (target.get(SourcesField),),
-                for_sources_types=(ScalaSourceField,),
-                enable_codegen=True,
-            )
-        ],
-    )
-
-    analysis = rule_runner.request(
-        ScalaSourceDependencyAnalysis,
-        [source_files],
+        ),
     )
 
     assert sorted(list(analysis.provided_symbols)) == [
@@ -223,12 +219,18 @@ def this(bar: SomeTypeInSecondaryConstructor) {
     assert analysis.imports_by_scope == FrozenDict(
         {
             "org.pantsbuild.example.OuterClass": (
-                ScalaImport(name="foo.bar.SomeItem", is_wildcard=False),
+                ScalaImport(name="foo.bar.SomeItem", alias=None, is_wildcard=False),
             ),
             "org.pantsbuild.example": (
-                ScalaImport(name="scala.collection.mutable.ArrayBuffer", is_wildcard=False),
-                ScalaImport(name="scala.collection.mutable.HashMap", is_wildcard=False),
-                ScalaImport(name="java.io", is_wildcard=True),
+                ScalaImport(
+                    name="scala.collection.mutable.ArrayBuffer", alias=None, is_wildcard=False
+                ),
+                ScalaImport(
+                    name="scala.collection.mutable.HashMap",
+                    alias="RenamedHashMap",
+                    is_wildcard=False,
+                ),
+                ScalaImport(name="java.io", alias=None, is_wildcard=True),
             ),
         }
     )
@@ -325,42 +327,15 @@ def this(bar: SomeTypeInSecondaryConstructor) {
 
 
 def test_extract_package_scopes(rule_runner: RuleRunner) -> None:
-    rule_runner.write_files(
-        {
-            "BUILD": textwrap.dedent(
-                """
-                scala_source(
-                    name="source",
-                    source="Source.scala",
-                )
-                """
-            ),
-            "Source.scala": textwrap.dedent(
-                """
+    analysis = _analyze(
+        rule_runner,
+        textwrap.dedent(
+            """
                 package outer
                 package more.than.one.part.at.once
                 package inner
                 """
-            ),
-        }
-    )
-
-    target = rule_runner.get_target(address=Address("", target_name="source"))
-
-    source_files = rule_runner.request(
-        SourceFiles,
-        [
-            SourceFilesRequest(
-                (target.get(SourcesField),),
-                for_sources_types=(ScalaSourceField,),
-                enable_codegen=True,
-            )
-        ],
-    )
-
-    analysis = rule_runner.request(
-        ScalaSourceDependencyAnalysis,
-        [source_files],
+        ),
     )
 
     assert sorted(analysis.scopes) == [
@@ -368,3 +343,33 @@ def test_extract_package_scopes(rule_runner: RuleRunner) -> None:
         "outer.more.than.one.part.at.once",
         "outer.more.than.one.part.at.once.inner",
     ]
+
+
+def test_relative_import(rule_runner: RuleRunner) -> None:
+    analysis = _analyze(
+        rule_runner,
+        textwrap.dedent(
+            """
+            import java.io
+            import scala.{io => sio}
+            import nada.{io => _}
+
+            object OuterObject {
+                import org.pantsbuild.{io => pio}
+
+                val i = io.apply()
+                val s = sio.apply()
+                val p = pio.apply()
+            }
+            """
+        ),
+    )
+
+    assert set(analysis.fully_qualified_consumed_symbols()) == {
+        "io.apply",
+        "java.io.apply",
+        "org.pantsbuild.io.apply",
+        "pio.apply",
+        "scala.io.apply",
+        "sio.apply",
+    }