Repository: bazelbuild/bazel
Issue: Implement "Locating runfiles with Bzlmod" proposal
Labels: type: feature request
P2
team-ExternalDeps
area-Bzlmod

The ["Locating runfiles with Bzlmod" proposal](https://github.com/bazelbuild/proposals/blob/main/designs/2022-07-21-locating-runfiles-with-bzlmod.md) consists of the following individual steps:

* [x] ~Add the `RunfilesLibraryInfo` provider (#16125)~
* [x] Collect repository names and mappings of the transitive closure of a target (#16278)
* [x] Emit a repository mapping manifest for every executable (#16321 and https://github.com/bazelbuild/bazel/pull/16652)
* [x] Revert `RunfilesLibraryInfo`
* [x] Expose the current repository name in rules
  * [x] Bash (#16693)
  * [x] C++ (#16216)
  * [x] Java (#16534)
  * [x] Python (#16341)
* [x] Parse and use the repository mapping manifest in runfiles libraries (https://github.com/bazelbuild/bazel/issues/15029)
  * [x] Bash (#16693)
  * [x] C++ (#16623 and https://github.com/bazelbuild/bazel/pull/16701)
  * [x] Java (#16549)
  * [X] ~Python~ has been updated in rules_python
* [x] Use the Java runfiles library to make Stardoc work with repository mappings (https://github.com/bazelbuild/bazel/issues/14140)

Status of runfiles libraries in third-party rulesets:

* [x] rules_go (https://github.com/bazelbuild/rules_go/pull/3347)
* [ ] [rules_rust](https://github.com/bazelbuild/rules_rust/blob/main/tools/runfiles/runfiles.rs) (???)
* [ ] [rules_haskell](https://github.com/tweag/rules_haskell/blob/master/tools/runfiles/README.md) (???)

Issue comments:
@Wyverald Could you assign me and add the appropriate labels?
~~@comius Do you expect `java_binary` to be starlarkified in time for Bazel 6? That would simplify the changes to the Java rules.~~ Most likely no longer necessary, as the changes will apply to `buildjar` itself.
@Wyverald Oops, I missed the use of `Closes ...` in the PR description - we should use `Work towards ...` instead. Could you reopen? For some I lack the permission to do so.
It's the "To fix #NNN" line... GitHub trying to be clever here
The most likely future seems to be that we'll standardize on Python imports _not_ containing the repository name, in which case bzlmod is a no-op.

It's not an optimal solution because it requires an entry in `$PYTHONPATH` for each repository in the transitive closure of the Python binary, but it's the status quo and I'd much prefer thinking up a solution to that (e.g. binary-specific symlink trees that do not look like runfiles trees, dunno) than to standardize on having the repository names in imports because the concept of "repository" is specific to Bazel.

Put differently: the only way for bzlmod to cause trouble for Python is if we decide to require repository names in imports, which seems to be unlikely and it looks like the decision on that will take a while and I prefer not blocking either Bazel 6.0 or the removal of bzlmod from the experimental domain to avoid an unlikely failure mode. And should that happen, we can always paper over it with some sort of import hook.

/cc @rickeylev 
@oquenchil I'm thinking about whether we should use a macro to pass `BAZEL_CURRENT_REPOSITORY` to the `Rlocation` function automatically (e.g. do `#define Rlocation(path) RlocationReal(path, BAZEL_CURRENT_REPOSITORY)`). Do you think that would be too intrusive/magical?
Would users have to write that line or is it part of a file that is automatically generated?

I don't mind adding whatever will make it easier but if it gets too magical it should maybe be wrapped so that the details aren't visible to the user unless they want to look.
> Would users have to write that line or is it part of a file that is automatically generated?
> 
> I don't mind adding whatever will make it easier but if it gets too magical it should maybe be wrapped so that the details aren't visible to the user unless they want to look.

@oquenchil I would want to make that magic part of the runfiles library header so that users don't have to write it themselves.

What I am worried about is having an `Rlocation` macro mess with people's existing codebase since there doesn't seem to be a way to put it in a namespace or restrict it to member functions only.
Aren't we in control of the name of the macro? Can we rename it to something that would be very unlikely to clash with anything?
> Aren't we in control of the name of the macro? Can we rename it to something that would be very unlikely to clash with anything?

We can call it something else, we just won't be able to have existing C++ codebases use the new repo mapping-aware lookup automatically if we do so. But of course having to replace all `runfiles::create()` calls with something else won't be too bad.
If the macro name is Rlocation and that may start clashing with people's code in C++ files then I think it would be prudent to rename it if we decide to go with the macro.

I guess the call to make here is how much of a breaking change this is compared to the benefit of passing BAZEL_CURRENT_REPOSITORY automatically for users. I would say that the benefit is not too great but as far as I understood the use of Bazel runfiles in C++ code wasn't too wide-spread either?

>  But of course having to replace all runfiles::create() calls with something else won't be too bad.

How do you know? Is it based on search results from GH? If a release note would be enough for it not to be too disruptive then let's rename it.
I have experimented a bit more with APIs while working on the corresponding change for Java (https://github.com/bazelbuild/bazel/pull/16549).

What I now like more than the macro approach is to have users (optionally) pass in a canonical repository name when creating the `Runfiles` instance and default to that repo in `Rlocation` calls, while providing ways to override the default with an overload of `Rlocation` that accepts a repository name as a second parameter. In this way, users only have to pass in `BAZEL_CURRENT_REPOSITORY` only once in the common case, which should render a macro unnecessary.
Update: All changes relevant for Bazel 6.0.0 are now in review.
This is mostly done now, with only the bugfix https://github.com/bazelbuild/bazel/pull/16755 and a number of cherry-picks to 6.0.0 missing.
@Wyverald You can remove this from the "6.0.0 nice to have" milestone as the only remaining issue is not tied to a Bazel release (Stardoc).
Downgrading because the most important points have been addressed; the only outstanding issue is Stardoc, which is undergoing a rewrite currently by @tetromino.

Pull request review comments:

Patch that resolved the issue:
diff --git a/src/main/java/com/google/devtools/build/lib/analysis/BUILD b/src/main/java/com/google/devtools/build/lib/analysis/BUILD
index b630ac7daeac91..4f5328526e0724 100644
--- a/src/main/java/com/google/devtools/build/lib/analysis/BUILD
+++ b/src/main/java/com/google/devtools/build/lib/analysis/BUILD
@@ -341,7 +341,6 @@ java_library(
         ":rule_configured_object_value",
         ":rule_definition_environment",
         ":run_environment_info",
-        ":runfiles_library_info",
         ":starlark/args",
         ":starlark/bazel_build_api_globals",
         ":starlark/function_transition_util",
@@ -1033,16 +1032,6 @@ java_library(
     ],
 )
 
-java_library(
-    name = "runfiles_library_info",
-    srcs = ["RunfilesLibraryInfo.java"],
-    deps = [
-        "//src/main/java/com/google/devtools/build/lib/concurrent",
-        "//src/main/java/com/google/devtools/build/lib/packages",
-        "//src/main/java/com/google/devtools/build/lib/starlarkbuildapi",
-    ],
-)
-
 java_library(
     name = "rule_definition_environment",
     srcs = ["RuleDefinitionEnvironment.java"],
diff --git a/src/main/java/com/google/devtools/build/lib/analysis/RunfilesLibraryInfo.java b/src/main/java/com/google/devtools/build/lib/analysis/RunfilesLibraryInfo.java
deleted file mode 100644
index 5e3107e2ea4f7e..00000000000000
--- a/src/main/java/com/google/devtools/build/lib/analysis/RunfilesLibraryInfo.java
+++ /dev/null
@@ -1,56 +0,0 @@
-// Copyright 2022 The Bazel Authors. All rights reserved.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//    http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package com.google.devtools.build.lib.analysis;
-
-import com.google.devtools.build.lib.concurrent.ThreadSafety.Immutable;
-import com.google.devtools.build.lib.packages.BuiltinProvider;
-import com.google.devtools.build.lib.packages.NativeInfo;
-import com.google.devtools.build.lib.starlarkbuildapi.RunfilesLibraryInfoApi;
-
-/**
- * Data-less provider that signals to direct dependents that this target is a runfiles library.
- *
- * <p>Rules that find this provider advertised by a dependency may use this as a signal to generate
- * code that helps with runfiles discovery, e.g., by providing a constant containing the name of the
- * repository that defines the rule.
- *
- * <p>Bazel itself uses the presence of this provider on a rule as a sign that it should include the
- * repository mapping of the repository containing the rule in the repository mapping manifest of
- * executable transitive dependents.
- */
-@Immutable
-public final class RunfilesLibraryInfo extends NativeInfo implements RunfilesLibraryInfoApi {
-
-  public static final RunfilesLibraryInfoProvider PROVIDER = new RunfilesLibraryInfoProvider();
-
-  @Override
-  public RunfilesLibraryInfoProvider getProvider() {
-    return PROVIDER;
-  }
-
-  /** Provider for {@link RunfilesLibraryInfo}. */
-  public static class RunfilesLibraryInfoProvider extends BuiltinProvider<RunfilesLibraryInfo>
-      implements RunfilesLibraryInfoApi.RunfilesLibraryInfoApiProvider {
-
-    private RunfilesLibraryInfoProvider() {
-      super("RunfilesLibraryInfo", RunfilesLibraryInfo.class);
-    }
-
-    @Override
-    public RunfilesLibraryInfoApi constructor() {
-      return new RunfilesLibraryInfo();
-    }
-  }
-}
diff --git a/src/main/java/com/google/devtools/build/lib/analysis/starlark/StarlarkModules.java b/src/main/java/com/google/devtools/build/lib/analysis/starlark/StarlarkModules.java
index a9007d783c404e..a77ff194bb82dd 100644
--- a/src/main/java/com/google/devtools/build/lib/analysis/starlark/StarlarkModules.java
+++ b/src/main/java/com/google/devtools/build/lib/analysis/starlark/StarlarkModules.java
@@ -19,7 +19,6 @@
 import com.google.devtools.build.lib.analysis.DefaultInfo;
 import com.google.devtools.build.lib.analysis.OutputGroupInfo;
 import com.google.devtools.build.lib.analysis.RunEnvironmentInfo;
-import com.google.devtools.build.lib.analysis.RunfilesLibraryInfo;
 import com.google.devtools.build.lib.packages.StarlarkLibrary;
 import com.google.devtools.build.lib.packages.StarlarkLibrary.SelectLibrary;
 import com.google.devtools.build.lib.packages.StructProvider;
@@ -49,6 +48,5 @@ public static void addPredeclared(ImmutableMap.Builder<String, Object> predeclar
     predeclared.put("Actions", ActionsProvider.INSTANCE);
     predeclared.put("DefaultInfo", DefaultInfo.PROVIDER);
     predeclared.put("RunEnvironmentInfo", RunEnvironmentInfo.PROVIDER);
-    predeclared.put("RunfilesLibraryInfo", RunfilesLibraryInfo.PROVIDER);
   }
 }
diff --git a/src/main/java/com/google/devtools/build/lib/starlarkbuildapi/RunfilesLibraryInfoApi.java b/src/main/java/com/google/devtools/build/lib/starlarkbuildapi/RunfilesLibraryInfoApi.java
deleted file mode 100644
index 191b539f57c63a..00000000000000
--- a/src/main/java/com/google/devtools/build/lib/starlarkbuildapi/RunfilesLibraryInfoApi.java
+++ /dev/null
@@ -1,47 +0,0 @@
-// Copyright 2022 The Bazel Authors. All rights reserved.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//    http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package com.google.devtools.build.lib.starlarkbuildapi;
-
-import com.google.devtools.build.docgen.annot.DocCategory;
-import com.google.devtools.build.docgen.annot.StarlarkConstructor;
-import com.google.devtools.build.lib.starlarkbuildapi.core.ProviderApi;
-import com.google.devtools.build.lib.starlarkbuildapi.core.StructApi;
-import net.starlark.java.annot.StarlarkBuiltin;
-import net.starlark.java.annot.StarlarkMethod;
-
-/** Interface for runfiles library info. */
-@StarlarkBuiltin(
-    name = "RunfilesLibraryInfo",
-    category = DocCategory.PROVIDER,
-    doc =
-        "Signals to direct dependents as well as Bazel itself that this target is a runfiles"
-            + " library. For example, rules for compiled languages may choose to generate and"
-            + " compile additional code that gives access to the name of the repository defining"
-            + " the current target if they find this provider advertised by a direct"
-            + " dependency.<p>This provider carries no data.")
-public interface RunfilesLibraryInfoApi extends StructApi {
-
-  /** Interface for runfiles library info provider. */
-  @StarlarkBuiltin(name = "Provider", category = DocCategory.PROVIDER, documented = false)
-  interface RunfilesLibraryInfoApiProvider extends ProviderApi {
-
-    @StarlarkMethod(
-        name = "RunfilesLibraryInfo",
-        doc = "The <code>RunfilesLibraryInfo</code> constructor.",
-        selfCall = true)
-    @StarlarkConstructor
-    RunfilesLibraryInfoApi constructor();
-  }
-}


Test patch:
diff --git a/src/test/java/com/google/devtools/build/lib/starlark/BUILD b/src/test/java/com/google/devtools/build/lib/starlark/BUILD
index c0e25bae180ae3..0ab20ce6319a18 100644
--- a/src/test/java/com/google/devtools/build/lib/starlark/BUILD
+++ b/src/test/java/com/google/devtools/build/lib/starlark/BUILD
@@ -46,7 +46,6 @@ java_test(
         "//src/main/java/com/google/devtools/build/lib/analysis:file_provider",
         "//src/main/java/com/google/devtools/build/lib/analysis:resolved_toolchain_context",
         "//src/main/java/com/google/devtools/build/lib/analysis:run_environment_info",
-        "//src/main/java/com/google/devtools/build/lib/analysis:runfiles_library_info",
         "//src/main/java/com/google/devtools/build/lib/analysis:starlark/args",
         "//src/main/java/com/google/devtools/build/lib/analysis:starlark/starlark_exec_group_collection",
         "//src/main/java/com/google/devtools/build/lib/analysis:test/analysis_test_result_info",
diff --git a/src/test/java/com/google/devtools/build/lib/starlark/StarlarkIntegrationTest.java b/src/test/java/com/google/devtools/build/lib/starlark/StarlarkIntegrationTest.java
index 21526932b03694..8ba84526fd64b9 100644
--- a/src/test/java/com/google/devtools/build/lib/starlark/StarlarkIntegrationTest.java
+++ b/src/test/java/com/google/devtools/build/lib/starlark/StarlarkIntegrationTest.java
@@ -30,7 +30,6 @@
 import com.google.devtools.build.lib.analysis.FilesToRunProvider;
 import com.google.devtools.build.lib.analysis.OutputGroupInfo;
 import com.google.devtools.build.lib.analysis.RunEnvironmentInfo;
-import com.google.devtools.build.lib.analysis.RunfilesLibraryInfo;
 import com.google.devtools.build.lib.analysis.RunfilesProvider;
 import com.google.devtools.build.lib.analysis.config.StarlarkDefinedConfigTransition;
 import com.google.devtools.build.lib.analysis.configuredtargets.FileConfiguredTarget;
@@ -3653,34 +3652,4 @@ public void nonExecutableStarlarkRuleReturningTestEnvironmentProducesAWarning()
             + " non-test target has no effect",
         ImmutableSet.of(EventKind.WARNING));
   }
-
-  @Test
-  public void runfilesLibraryInfoCanBeReturnedAndQueried() throws Exception {
-    scratch.file(
-        "examples/rules.bzl",
-        "def my_runfiles_library_impl(ctx):",
-        "  return [RunfilesLibraryInfo()]",
-        "my_runfiles_library = rule(implementation = my_runfiles_library_impl)",
-        "def language_rule_impl(ctx):",
-        "  if RunfilesLibraryInfo not in ctx.attr.dep:",
-        "    fail('dep does not advertise RunfilesLibraryInfo')",
-        "language_rule = rule(",
-        "  implementation = language_rule_impl,",
-        "  attrs = {'dep': attr.label()},",
-        ")");
-    scratch.file(
-        "examples/BUILD",
-        "load(':rules.bzl', 'language_rule', 'my_runfiles_library')",
-        "my_runfiles_library(name = 'runfiles_library')",
-        "language_rule(",
-        "  name = 'target',",
-        "  dep = ':runfiles_library',",
-        ")");
-
-    ConfiguredTarget runfilesLibrary = getConfiguredTarget("//examples:runfiles_library");
-    assertThat(runfilesLibrary.get(RunfilesLibraryInfo.PROVIDER)).isNotNull();
-
-    // Succeeds only if targets can be queried for the presence of RunfilesLibraryInfo in Starlark.
-    getConfiguredTarget("//examples:target");
-  }
 }