Repository: pantsbuild/pants

Issue:
Fix Scala using `add_dependencies_on_all_siblings=True`
After dependency inference was improved for Scala, `add_dependencies_on_all_siblings` was not removed:
https://github.com/pantsbuild/pants/blob/c2f6404c1ed5fd11a6a37eac8682a5d337bf22aa/src/python/pants/backend/scala/target_types.py#L233 This means that we are overly coarsening Scala (all BUILD targets end up compiled together).

We should fix this (or drive it via an option), but it will definitely impact compilation success rates, and might also impact performance. We should test out the impact in our testbed repositories.


Discussion:
I've experimented with removing this flag on our codebase, which required some more improvements to scala parser, after those improvements (PR is coming), the number of coarsened targets increased from 80 to around 800. It also allowed us to reduce `process_per_child_memory_usage` and increase the parallelism.

The best benefit for us would be less cache invalidation of test results.

I would like to contribute a PR to address this. Should I remove the flag or allow it to be configurable?
> I would like to contribute a PR to address this. Should I remove the flag or allow it to be configurable?

Thanks a ton! It will need to be configurable for sure, in order to allow folks to migrate from the existing default. But I expect that we can begin deprecating the existing value immediately.
Awesome! For other languages, we set it dynamically based on if dependency inference is enabled:

https://github.com/pantsbuild/pants/blob/c1b1eaabd8c5b12afac3b5aa9faa57557a4412b9/src/python/pants/backend/python/target_types_rules.py#L77-L82

https://github.com/pantsbuild/pants/blob/c1b1eaabd8c5b12afac3b5aa9faa57557a4412b9/src/python/pants/backend/python/target_types_rules.py#L556

We will want to preserve that. Without dependency inference, it is incredibly tedious to specify "sibling" dependencies and you have to do it via the `overrides` field.

So, I suspect we maybe also have a temporary option like `[scala-infer].force_sibling_dependencies` with a default to `true`, per our deprecation policy. We deprecate not setting the default, and then possibly deprecate the option entirely in the subsequent release. 

Resolving patch:
diff --git a/src/python/pants/backend/scala/subsystems/scala_infer.py b/src/python/pants/backend/scala/subsystems/scala_infer.py
--- a/src/python/pants/backend/scala/subsystems/scala_infer.py
+++ b/src/python/pants/backend/scala/subsystems/scala_infer.py
@@ -3,6 +3,7 @@
 
 from pants.option.option_types import BoolOption
 from pants.option.subsystem import Subsystem
+from pants.util.strutil import softwrap
 
 
 class ScalaInferSubsystem(Subsystem):
@@ -19,3 +20,18 @@ class ScalaInferSubsystem(Subsystem):
         default=True,
         help="Infer a target's dependencies by parsing consumed types from sources.",
     )
+    force_add_siblings_as_dependencies = BoolOption(
+        "--force-add-siblings-as-dependencies",
+        default=True,
+        help=softwrap(
+            """
+            If true, add a dependency on all scala_source targets generated by the same scala_sources target generator.
+            If false, the dependencies will only be added if [scala-infer].imports is also set to false.
+
+            Setting this option to true reduces the precision of dependency information.
+            That means that you may end up compiling more than is necessary for a particular task,
+            and that compilation will be invalidated more frequently than actually necessary.
+            However, setting to true may be helpful if compilation fails due to missing dependencies.
+            """
+        ),
+    )
diff --git a/src/python/pants/backend/scala/target_types.py b/src/python/pants/backend/scala/target_types.py
--- a/src/python/pants/backend/scala/target_types.py
+++ b/src/python/pants/backend/scala/target_types.py
@@ -5,6 +5,8 @@
 
 from dataclasses import dataclass
 
+from pants.backend.scala.subsystems.scala_infer import ScalaInferSubsystem
+from pants.base.deprecated import warn_or_error
 from pants.engine.rules import collect_rules, rule
 from pants.engine.target import (
     COMMON_TARGET_FIELDS,
@@ -36,9 +38,35 @@ class ScalaSettingsRequest(TargetFilesGeneratorSettingsRequest):
 
 
 @rule
-def scala_settings_request(_: ScalaSettingsRequest) -> TargetFilesGeneratorSettings:
-    # TODO: See https://github.com/pantsbuild/pants/issues/14382.
-    return TargetFilesGeneratorSettings(add_dependencies_on_all_siblings=True)
+def scala_settings_request(
+    scala_infer_subsystem: ScalaInferSubsystem, _: ScalaSettingsRequest
+) -> TargetFilesGeneratorSettings:
+    if scala_infer_subsystem.options.is_default("force_add_siblings_as_dependencies"):
+        warn_or_error(
+            removal_version="2.14.0.dev0",
+            entity="`force_add_siblings_as_dependencies` defaulting to True",
+            hint=softwrap(
+                """
+                Setting this option to true reduces the precision of dependency information.
+                That means that you may end up compiling more than is necessary for a particular task,
+                and that compilation will be invalidated more frequently than actually necessary.
+                However, setting to true may be helpful if compilation fails due to missing dependencies.
+
+                We have made several improvements to Pants's Scala dependency inference,
+                where we no longer think it's necessary to adding dependencies on sibling targets.
+                If you have compilation failures after disabling this option, please consider opening an issue at
+                https://github.com/pantsbuild/pants/issues/new so that we can continue to improve Pants's dependency inference.
+
+                To opt into the new default early, set `force_add_siblings_as_dependencies = false` in the `[scala_infer]`
+                section in `pants.toml`. Otherwise, set to `true` to silence this warning.
+                """
+            ),
+        )
+
+    return TargetFilesGeneratorSettings(
+        add_dependencies_on_all_siblings=scala_infer_subsystem.force_add_siblings_as_dependencies
+        or not scala_infer_subsystem.imports
+    )
 
 
 class ScalaSourceField(SingleSourceField):


Test patch: