Repository: bazelbuild/bazel
Issue: unflip --repository_cache_urls_as_default_canonical_id
Labels: type: bug
P2
team-ExternalDeps

### Description of the bug:


Looks like https://github.com/bazelbuild/bazel/pull/19549 flipped `--repository_cache_urls_as_default_canonical_id` flag on by default which hurts `rctx.download` performance and breaks rules_oci's layer deduping mechanism completely. 


Cross-ref: https://github.com/bazel-contrib/rules_oci/issues/387

### Which category does this issue belong to?


External Dependency

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.


- clone https://github.com/bazel-contrib/rules_oci
- run `bazel build //oci/tests/...  --repository_cache_urls_as_default_canonical_id`

### Which operating system are you running Bazel on?


darwin/arm64

### What is the output of `bazel info release`?


release 6.3.2

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.


_No response_

### What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?


_No response_

### Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.


_No response_

### Have you found anything relevant by searching the web?

_No response_

### Any other information, logs, or outputs that you want to share?

_No response_

Issue comments:
I think of this a bit different than @fmeum (in the closed PR).

Imo for WORKSPACE downloads, the download cache deduplication should only matter “much” across source code revisions(git commits) and not within the same revision. In practice, you would want to group most duplication within a single http_archive with multiple URLs, and there is good value into having a URL invalidation scheme.

However I could see a potential fix to this:
- Create a CAS store for download content, potentially the same CAS as disk_cache.
- Make repository_cache (the download cache, not the new proposed cache) to track pointer record to CAS.

This means upon download, the archives should be written to CAS, then Bazel should create a pointer to the CAS in repository_cache dir. We could name this repository_download_cache or http_download_cache for good. This means that different http_archive rules downloading the same blob would create different file pointer records pointing to the same object in CAS.
@sluongng This additional indirection is a good idea to avoid unnecessary duplication (in both downloads and storage). 

I think that we should figure out these details before we enable the invalidation in a release. I mostly see two problems with the behavior of `--repository_cache_urls_as_default_canonical_id`:

1. The invalidation behavior can only realistically catch mistakes if both the list of URLs and the hash are specified by the end user. However, the flag applies on the level of `repository_ctx.download` and thus affects all repository rules, with the vast majority not reaping the benefits as they don't satisfy this condition. Restricting this behavior to `http_*` rules would avoid these kinds of silent performance degradations without benefits.
2. The behavior of the flag is trivial to implement in Starlark: `canonical_id = " ".join(urls)` is really all it does. Repository rules that satisfy the condition in 1. could just implement this logic themselves.
3. The invalidation scheme makes it so that adding a mirror to an existing repository rule invalidates the cache, which is pretty counter-intuitive. @sluongng's suggestion would help here.

Even if we think that 3. is acceptable in order to catch a common mistake (I can get behind this), we would probably still be better off by implementing this for `http_*` and in Starlark only.

(I will flag this for 7.0.0 so that we end up making a conscious decision rather than running out of time)
@bazel-io flag
@bazel-io fork 7.0.0
1 and 2 are good suggestions. 3 is rare enough that I think we could just slow roll it and target for 7.1, 7.2.

In the long run, i think it would be beneficial if we could provide a bit more information of the repository cache (real and download) via repository_ctx api. It would help us implement custom downloaders(i.e. go_repository fetcher) while still leveraging the cache. 
Okay, I can implement the Starlark feature in `http_*`.

@meteorcloudy Could you roll back https://github.com/bazelbuild/bazel/commit/fc3460e92597091997fa4154173e2d4e973ede31 as a first step? I can then replace the flag with an env var and Starlark logic.
@fmeum Is it sufficient to implement this in starlark I wonder.
The problem is not just about cache invalidation, the problem is in the Java layer, the downloader will bias toward cache entry with SHA256 declared in the `http_archive`. So even if the starlark invalidate and re-trigger the downloader, the downloader would still grab the same binary out of the cache.

I think the only concrete way to guard against this issue, where user updates the URLs but forgot to update SHA256, is to have the flag enabled. It does come with the reduction in repository_cache deduplication… however introducing a re-direction layer, such as a pointer system referencing a CAS, should solve the deduplication issue.
> So even if the starlark invalidate and re-trigger the downloader, the downloader would still grab the same binary out of the cache.

it would not use the cache if the canonical id is different, even if the digest is the same 
Ah true. For some reason i thought we are still passing sha256 down to java. I has been a long day 🥲
@fmeum Thanks for the analyzing! I'll take care of the rollback.
I submitted a Starlark implementation as https://github.com/bazelbuild/bazel/pull/20047.
A fix for this issue has been included in [Bazel 7.0.0 RC5](https://releases.bazel.build/7.0.0/rc5/index.html). Please test out the release candidate and report any issues as soon as possible. Thanks!

Pull request review comments:

Patch that resolved the issue:
diff --git a/MODULE.bazel.lock b/MODULE.bazel.lock
index d9e52f66ae9f1a..766eaed0864425 100644
--- a/MODULE.bazel.lock
+++ b/MODULE.bazel.lock
@@ -2158,7 +2158,7 @@
   "moduleExtensions": {
     "//:extensions.bzl%bazel_android_deps": {
       "general": {
-        "bzlTransitiveDigest": "U0PdF0Y6pi1ibjV47I9znaKyMppoEOhJX/RSf+0SYm8=",
+        "bzlTransitiveDigest": "JIUppb9cYpap/pfU76BpY/F/Qd6Qld7XVEsAa78X0T4=",
         "accumulatedFileDigests": {},
         "envVariables": {},
         "generatedRepoSpecs": {
@@ -2177,9 +2177,9 @@
     },
     "//:extensions.bzl%bazel_build_deps": {
       "general": {
-        "bzlTransitiveDigest": "U0PdF0Y6pi1ibjV47I9znaKyMppoEOhJX/RSf+0SYm8=",
+        "bzlTransitiveDigest": "JIUppb9cYpap/pfU76BpY/F/Qd6Qld7XVEsAa78X0T4=",
         "accumulatedFileDigests": {
-          "@@//src/test/tools/bzlmod:MODULE.bazel.lock": "57004ac9cc0c88944c08f5156a299addbe3ff1caa9209ec0809edda9a4315b31",
+          "@@//src/test/tools/bzlmod:MODULE.bazel.lock": "8d144d3e597e69c9cb6f572e9cae8333cc991ba75d61fff1c4d9bd2ad4e2d429",
           "@@//:MODULE.bazel": "016812bd09c1de57c7cbfd60192cc76e75a4f564caafc4af0a018cd66ffb0cae"
         },
         "envVariables": {},
@@ -2428,7 +2428,7 @@
     },
     "//:extensions.bzl%bazel_test_deps": {
       "general": {
-        "bzlTransitiveDigest": "U0PdF0Y6pi1ibjV47I9znaKyMppoEOhJX/RSf+0SYm8=",
+        "bzlTransitiveDigest": "JIUppb9cYpap/pfU76BpY/F/Qd6Qld7XVEsAa78X0T4=",
         "accumulatedFileDigests": {},
         "envVariables": {},
         "generatedRepoSpecs": {
@@ -2478,7 +2478,7 @@
     },
     "//tools/android:android_extensions.bzl%remote_android_tools_extensions": {
       "general": {
-        "bzlTransitiveDigest": "EFYd5Zc37KUKoseMe8brwJ5A2j/kUvPs5pIvTfGf3ok=",
+        "bzlTransitiveDigest": "hL6dUUghnqmrcFrs2bvFCvxPHBnPOY3ymycjIbgRQac=",
         "accumulatedFileDigests": {},
         "envVariables": {},
         "generatedRepoSpecs": {
@@ -2505,7 +2505,7 @@
     },
     "//tools/test:extensions.bzl%remote_coverage_tools_extension": {
       "general": {
-        "bzlTransitiveDigest": "wd0+Kn35gYWv/xzdEzWg7vRQz6FZcfpne4WcwCs9d+o=",
+        "bzlTransitiveDigest": "DEE4EmNqW6/brpzbLQMxPvdWKALzpq/MeZyydMFsjLI=",
         "accumulatedFileDigests": {},
         "envVariables": {},
         "generatedRepoSpecs": {
diff --git a/scripts/bootstrap/bootstrap.sh b/scripts/bootstrap/bootstrap.sh
index 87b9df5579cdbb..9c7cdbee60b2d5 100755
--- a/scripts/bootstrap/bootstrap.sh
+++ b/scripts/bootstrap/bootstrap.sh
@@ -31,11 +31,14 @@ fi
 
 : ${JAVA_VERSION:="11"}
 
+# TODO: remove `--repo_env=BAZEL_HTTP_RULES_URLS_AS_DEFAULT_CANONICAL_ID=0` once all dependencies are
+#  mirrored. See https://github.com/bazelbuild/bazel/pull/19549 for more context.
 _BAZEL_ARGS="--spawn_strategy=standalone \
       --nojava_header_compilation \
       --strategy=Javac=worker --worker_quit_after_build --ignore_unsupported_sandboxing \
       --compilation_mode=opt \
       --repository_cache=derived/repository_cache \
+      --repo_env=BAZEL_HTTP_RULES_URLS_AS_DEFAULT_CANONICAL_ID=0 \
       --extra_toolchains=//scripts/bootstrap:all \
       --extra_toolchains=@bazel_tools//tools/python:autodetecting_toolchain \
       --enable_bzlmod \
diff --git a/src/main/java/com/google/devtools/build/lib/bazel/BazelRepositoryModule.java b/src/main/java/com/google/devtools/build/lib/bazel/BazelRepositoryModule.java
index c4e466422db0a4..0a2661fde95dc7 100644
--- a/src/main/java/com/google/devtools/build/lib/bazel/BazelRepositoryModule.java
+++ b/src/main/java/com/google/devtools/build/lib/bazel/BazelRepositoryModule.java
@@ -349,7 +349,6 @@ public void beforeCommand(CommandEnvironment env) throws AbruptExitException {
       if (repoOptions.repositoryDownloaderRetries >= 0) {
         downloadManager.setRetries(repoOptions.repositoryDownloaderRetries);
       }
-      downloadManager.setUrlsAsDefaultCanonicalId(repoOptions.urlsAsDefaultCanonicalId);
 
       repositoryCache.setHardlink(repoOptions.useHardlinks);
       if (repoOptions.experimentalScaleTimeouts > 0.0) {
diff --git a/src/main/java/com/google/devtools/build/lib/bazel/repository/RepositoryOptions.java b/src/main/java/com/google/devtools/build/lib/bazel/repository/RepositoryOptions.java
index 0e6a234a8c56f8..e984bad9e35c94 100644
--- a/src/main/java/com/google/devtools/build/lib/bazel/repository/RepositoryOptions.java
+++ b/src/main/java/com/google/devtools/build/lib/bazel/repository/RepositoryOptions.java
@@ -264,15 +264,15 @@ public Converter() {
 
   @Option(
       name = "experimental_repository_cache_urls_as_default_canonical_id",
-      defaultValue = "false",
-      documentationCategory = OptionDocumentationCategory.BAZEL_CLIENT_OPTIONS,
-      effectTags = {OptionEffectTag.LOADING_AND_ANALYSIS},
-      metadataTags = {OptionMetadataTag.EXPERIMENTAL},
-      help =
-          "If true, use a string derived from the URLs of repository downloads as the canonical_id "
-              + "if not specified. This causes a change in the URLs to result in a redownload even "
-              + "if the cache contains a download with the same hash. This can be used to verify "
-              + "that URL changes don't result in broken repositories being masked by the cache.")
+      defaultValue = "true",
+      documentationCategory = OptionDocumentationCategory.UNDOCUMENTED,
+      metadataTags = OptionMetadataTag.DEPRECATED,
+      effectTags = {OptionEffectTag.NO_OP},
+      deprecationWarning =
+          "This behavior is enabled by default for http_* and jvm_* rules and no "
+              + "longer controlled by this flag. Use "
+              + "--repo_env=BAZEL_HTTP_RULES_URLS_AS_DEFAULT_CANONICAL_ID=0 to disable it instead.",
+      help = "No-op.")
   public boolean urlsAsDefaultCanonicalId;
 
   @Option(
diff --git a/src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/DownloadManager.java b/src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/DownloadManager.java
index ae9fe3543a2806..02656d8221f3e8 100644
--- a/src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/DownloadManager.java
+++ b/src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/DownloadManager.java
@@ -42,7 +42,6 @@
 import java.util.Map;
 import java.util.Map.Entry;
 import java.util.Optional;
-import java.util.stream.Collectors;
 import javax.annotation.Nullable;
 
 /**
@@ -59,7 +58,6 @@ public class DownloadManager {
   private final Downloader downloader;
   private boolean disableDownload = false;
   private int retries = 0;
-  private boolean urlsAsDefaultCanonicalId;
   @Nullable private Credentials netrcCreds;
   private CredentialFactory credentialFactory = StaticCredentials::new;
 
@@ -90,10 +88,6 @@ public void setRetries(int retries) {
     this.retries = retries;
   }
 
-  public void setUrlsAsDefaultCanonicalId(boolean urlsAsDefaultCanonicalId) {
-    this.urlsAsDefaultCanonicalId = urlsAsDefaultCanonicalId;
-  }
-
   public void setNetrcCreds(Credentials netrcCreds) {
     this.netrcCreds = netrcCreds;
   }
@@ -134,9 +128,6 @@ public Path download(
     if (Thread.interrupted()) {
       throw new InterruptedException();
     }
-    if (Strings.isNullOrEmpty(canonicalId) && urlsAsDefaultCanonicalId) {
-      canonicalId = originalUrls.stream().map(URL::toExternalForm).collect(Collectors.joining(" "));
-    }
 
     // TODO(andreisolo): This code path is inconsistent as the authHeaders are fetched from a
     //  .netrc only if it comes from a http_{archive,file,jar} - and it is handled directly
diff --git a/tools/build_defs/repo/BUILD b/tools/build_defs/repo/BUILD
index 09dfa068a9f108..b545c12ad475bd 100644
--- a/tools/build_defs/repo/BUILD
+++ b/tools/build_defs/repo/BUILD
@@ -15,6 +15,7 @@ filegroup(
 filegroup(
     name = "http_src",
     srcs = [
+        "cache.bzl",
         "http.bzl",
         "utils.bzl",
     ],
diff --git a/tools/build_defs/repo/cache.bzl b/tools/build_defs/repo/cache.bzl
new file mode 100644
index 00000000000000..8381446da242ff
--- /dev/null
+++ b/tools/build_defs/repo/cache.bzl
@@ -0,0 +1,50 @@
+# Copyright 2023 The Bazel Authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# WARNING:
+# https://github.com/bazelbuild/bazel/issues/17713
+# .bzl files in this package (tools/build_defs/repo) are evaluated
+# in a Starlark environment without "@_builtins" injection, and must not refer
+# to symbols associated with build/workspace .bzl files
+
+"""Returns the default canonical id to use for downloads."""
+
+visibility("private")
+
+DEFAULT_CANONICAL_ID_ENV = "BAZEL_HTTP_RULES_URLS_AS_DEFAULT_CANONICAL_ID"
+
+CANONICAL_ID_DOC = """A canonical ID of the file downloaded.
+
+If specified and non-empty, Bazel will not take the file from cache, unless it
+was added to the cache by a request with the same canonical ID.
+
+If unspecified or empty, Bazel by default uses the URLs of the file as the
+canonical ID. This helps catch the common mistake of updating the URLs without
+also updating the hash, resulting in builds that succeed locally but fail on
+machines without the file in the cache. This behavior can be disabled with
+--repo_env={env}=0.
+""".format(env = DEFAULT_CANONICAL_ID_ENV)
+
+def get_default_canonical_id(repository_ctx, urls):
+    """Returns the default canonical id to use for downloads."""
+    if repository_ctx.os.environ.get(DEFAULT_CANONICAL_ID_ENV) == "0":
+        return ""
+
+    # Do not sort URLs to prevent the following scenario:
+    # 1. http_archive with urls = [B, A] created.
+    # 2. Successful fetch from B results in canonical ID "A B".
+    # 3. Order of urls is flipped to [A, B].
+    # 4. Fetch would reuse cache entry for "A B", even though A may be broken (it has never been
+    #    fetched before).
+    return " ".join(urls)
diff --git a/tools/build_defs/repo/http.bzl b/tools/build_defs/repo/http.bzl
index f1e79da78cc105..b61c4046197ead 100644
--- a/tools/build_defs/repo/http.bzl
+++ b/tools/build_defs/repo/http.bzl
@@ -37,6 +37,12 @@ These rules are improved versions of the native http rules and will eventually
 replace the native rules.
 """
 
+load(
+    ":cache.bzl",
+    "CANONICAL_ID_DOC",
+    "DEFAULT_CANONICAL_ID_ENV",
+    "get_default_canonical_id",
+)
 load(
     ":utils.bzl",
     "patch",
@@ -142,7 +148,7 @@ def _http_archive_impl(ctx):
         ctx.attr.sha256,
         ctx.attr.type,
         ctx.attr.strip_prefix,
-        canonical_id = ctx.attr.canonical_id,
+        canonical_id = ctx.attr.canonical_id or get_default_canonical_id(ctx, all_urls),
         auth = auth,
         integrity = ctx.attr.integrity,
     )
@@ -182,7 +188,7 @@ def _http_file_impl(ctx):
         "file/" + downloaded_file_path,
         ctx.attr.sha256,
         ctx.attr.executable,
-        canonical_id = ctx.attr.canonical_id,
+        canonical_id = ctx.attr.canonical_id or get_default_canonical_id(ctx, all_urls),
         auth = auth,
         integrity = ctx.attr.integrity,
     )
@@ -219,7 +225,7 @@ def _http_jar_impl(ctx):
         all_urls,
         "jar/" + downloaded_file_name,
         ctx.attr.sha256,
-        canonical_id = ctx.attr.canonical_id,
+        canonical_id = ctx.attr.canonical_id or get_default_canonical_id(ctx, all_urls),
         auth = auth,
         integrity = ctx.attr.integrity,
     )
@@ -257,11 +263,7 @@ easier but either this attribute or `sha256` should be set before shipping.""",
         doc = _AUTH_PATTERN_DOC,
     ),
     "canonical_id": attr.string(
-        doc = """A canonical id of the archive downloaded.
-
-If specified and non-empty, bazel will not take the archive from cache,
-unless it was added to the cache by a request with the same canonical id.
-""",
+        doc = CANONICAL_ID_DOC,
     ),
     "strip_prefix": attr.string(
         doc = """A directory prefix to strip from the extracted files.
@@ -382,6 +384,7 @@ following: `"zip"`, `"jar"`, `"war"`, `"aar"`, `"tar"`, `"tar.gz"`, `"tgz"`,
 http_archive = repository_rule(
     implementation = _http_archive_impl,
     attrs = _http_archive_attrs,
+    environ = [DEFAULT_CANONICAL_ID_ENV],
     doc =
         """Downloads a Bazel repository as a compressed archive file, decompresses it,
 and makes its targets available for binding.
@@ -457,11 +460,7 @@ field will make your build non-hermetic. It is optional to make development
 easier but either this attribute or `sha256` should be set before shipping.""",
     ),
     "canonical_id": attr.string(
-        doc = """A canonical id of the archive downloaded.
-
-If specified and non-empty, bazel will not take the archive from cache,
-unless it was added to the cache by a request with the same canonical id.
-""",
+        doc = CANONICAL_ID_DOC,
     ),
     "url": attr.string(doc = _URL_DOC),
     "urls": attr.string_list(doc = _URLS_DOC),
@@ -476,6 +475,7 @@ unless it was added to the cache by a request with the same canonical id.
 http_file = repository_rule(
     implementation = _http_file_impl,
     attrs = _http_file_attrs,
+    environ = [DEFAULT_CANONICAL_ID_ENV],
     doc =
         """Downloads a file from a URL and makes it available to be used as a file
 group.
@@ -517,11 +517,7 @@ field will make your build non-hermetic. It is optional to make development
 easier but either this attribute or `sha256` should be set before shipping.""",
     ),
     "canonical_id": attr.string(
-        doc = """A canonical id of the archive downloaded.
-
-If specified and non-empty, bazel will not take the archive from cache,
-unless it was added to the cache by a request with the same canonical id.
-""",
+        doc = CANONICAL_ID_DOC,
     ),
     "url": attr.string(doc = _URL_DOC + "\n\nThe URL must end in `.jar`."),
     "urls": attr.string_list(doc = _URLS_DOC + "\n\nAll URLs must end in `.jar`."),
@@ -540,6 +536,7 @@ unless it was added to the cache by a request with the same canonical id.
 http_jar = repository_rule(
     implementation = _http_jar_impl,
     attrs = _http_jar_attrs,
+    environ = [DEFAULT_CANONICAL_ID_ENV],
     doc =
         """Downloads a jar from a URL and makes it available as java_import
 
diff --git a/tools/build_defs/repo/jvm.bzl b/tools/build_defs/repo/jvm.bzl
index 98ce90133e1529..c7e87d2f4a97bd 100644
--- a/tools/build_defs/repo/jvm.bzl
+++ b/tools/build_defs/repo/jvm.bzl
@@ -38,6 +38,13 @@ the following macros are defined below that utilize jvm_import_external:
 - java_import_external - uses `java_import` as the underlying build rule
 """
 
+load(
+    ":cache.bzl",
+    "CANONICAL_ID_DOC",
+    "DEFAULT_CANONICAL_ID_ENV",
+    "get_default_canonical_id",
+)
+
 _HEADER = "# DO NOT EDIT: generated by jvm_import_external()"
 
 _PASS_PROPS = (
@@ -116,7 +123,7 @@ def _jvm_import_external(repository_ctx):
         urls,
         path,
         sha,
-        canonical_id = repository_ctx.attr.canonical_id,
+        canonical_id = repository_ctx.attr.canonical_id or get_default_canonical_id(repository_ctx, urls),
     )
     if srcurls and _should_fetch_sources_in_current_env(repository_ctx):
         repository_ctx.download(
@@ -239,7 +246,9 @@ jvm_import_external = repository_rule(
         "additional_rule_attrs": attr.string_dict(),
         "srcjar_urls": attr.string_list(),
         "srcjar_sha256": attr.string(),
-        "canonical_id": attr.string(),
+        "canonical_id": attr.string(
+            doc = CANONICAL_ID_DOC,
+        ),
         "deps": attr.string_list(),
         "runtime_deps": attr.string_list(),
         "testonly_": attr.bool(),
@@ -250,7 +259,10 @@ jvm_import_external = repository_rule(
         "default_visibility": attr.string_list(default = ["//visibility:public"]),
         "extra_build_file_content": attr.string(),
     },
-    environ = [_FETCH_SOURCES_ENV_VAR],
+    environ = [
+        _FETCH_SOURCES_ENV_VAR,
+        DEFAULT_CANONICAL_ID_ENV,
+    ],
     implementation = _jvm_import_external,
 )
 


Test patch:
diff --git a/src/test/java/com/google/devtools/build/lib/blackbox/bazel/DefaultToolsSetup.java b/src/test/java/com/google/devtools/build/lib/blackbox/bazel/DefaultToolsSetup.java
index 916175736adce7..48394f478222fa 100644
--- a/src/test/java/com/google/devtools/build/lib/blackbox/bazel/DefaultToolsSetup.java
+++ b/src/test/java/com/google/devtools/build/lib/blackbox/bazel/DefaultToolsSetup.java
@@ -78,6 +78,9 @@ public void setup(BlackBoxTestContext context) throws IOException {
     String sharedRepoCache = System.getenv("REPOSITORY_CACHE");
     if (sharedRepoCache != null) {
       lines.add("common --repository_cache=" + sharedRepoCache);
+      // TODO: Remove this flag once all dependencies are mirrored.
+      // See https://github.com/bazelbuild/bazel/pull/19549 for more context.
+      lines.add("common --repo_env=BAZEL_HTTP_RULES_URLS_AS_DEFAULT_CANONICAL_ID=0");
       if (OS.getCurrent() == OS.DARWIN) {
         // For reducing SSD usage on our physical Mac machines.
         lines.add("common --experimental_repository_cache_hardlinks");
diff --git a/src/test/py/bazel/bzlmod/bazel_fetch_test.py b/src/test/py/bazel/bzlmod/bazel_fetch_test.py
index 8799a8e9f3ea84..d7867d6d4e99dc 100644
--- a/src/test/py/bazel/bzlmod/bazel_fetch_test.py
+++ b/src/test/py/bazel/bzlmod/bazel_fetch_test.py
@@ -61,6 +61,10 @@ def generatBuiltinModules(self):
     self.ScratchFile('tools_mock/WORKSPACE')
     self.ScratchFile('tools_mock/MODULE.bazel', ['module(name="bazel_tools")'])
     self.ScratchFile('tools_mock/tools/build_defs/repo/BUILD')
+    self.CopyFile(
+        self.Rlocation('io_bazel/tools/build_defs/repo/cache.bzl'),
+        'tools_mock/tools/build_defs/repo/cache.bzl',
+    )
     self.CopyFile(
         self.Rlocation('io_bazel/tools/build_defs/repo/http.bzl'),
         'tools_mock/tools/build_defs/repo/http.bzl',
diff --git a/src/test/py/bazel/test_base.py b/src/test/py/bazel/test_base.py
index 9366f97cf853cc..a9362e9f73b5b3 100644
--- a/src/test/py/bazel/test_base.py
+++ b/src/test/py/bazel/test_base.py
@@ -127,6 +127,12 @@ def setUp(self):
       shared_repo_cache = os.environ.get('REPOSITORY_CACHE')
       if shared_repo_cache:
         f.write('common --repository_cache={}\n'.format(shared_repo_cache))
+        # TODO(pcloudy): Remove this flag once all dependencies are mirrored.
+        # See https://github.com/bazelbuild/bazel/pull/19549 for more context.
+        f.write(
+            'common'
+            ' --repo_env=BAZEL_HTTP_RULES_URLS_AS_DEFAULT_CANONICAL_ID=0\n'
+        )
         if TestBase.IsDarwin():
           # For reducing SSD usage on our physical Mac machines.
           f.write('common --experimental_repository_cache_hardlinks\n')
diff --git a/src/test/shell/bazel/bazel_repository_cache_test.sh b/src/test/shell/bazel/bazel_repository_cache_test.sh
index 28235a0557c272..4836db5b09ebef 100755
--- a/src/test/shell/bazel/bazel_repository_cache_test.sh
+++ b/src/test/shell/bazel/bazel_repository_cache_test.sh
@@ -221,6 +221,9 @@ function test_fetch_value_with_existing_cache_and_no_network() {
   cache_entry="$repo_cache_dir/content_addressable/sha256/$sha256"
   mkdir -p "$cache_entry"
   cp "$repo2_zip" "$cache_entry/file" # Artifacts are named uniformly as "file" in the cache
+  http_archive_url="http://localhost:$nc_port/bleh"
+  canonical_id_hash=$(printf "$http_archive_url" | sha256sum | cut -f 1 -d ' ')
+  touch "$cache_entry/id-$canonical_id_hash"
 
   # Fetch without a server
   shutdown_server
@@ -271,6 +274,7 @@ EOF
   # to do without checksum. But we can safely do so, as the loopback device
   # is reasonably safe against man-in-the-middle attacks.
   bazel fetch --repository_cache="$repo_cache_dir" \
+        --repo_env=BAZEL_HTTP_RULES_URLS_AS_DEFAULT_CANONICAL_ID=0 \
         //zoo:breeding-program >& $TEST_log \
     || fail "expected fetch to succeed"
 
@@ -283,6 +287,7 @@ EOF
 
   # As we don't have a predicted cache, we expect fetching to fail now.
   bazel fetch --repository_cache="$repo_cache_dir" //zoo:breeding-program >& $TEST_log \
+        --repo_env=BAZEL_HTTP_RULES_URLS_AS_DEFAULT_CANONICAL_ID=0 \
     && fail "expected failure" || :
 
   # However, if we add the hash, the value is taken from cache
@@ -298,6 +303,7 @@ http_archive(
 )
 EOF
   bazel fetch --repository_cache="$repo_cache_dir" //zoo:breeding-program >& $TEST_log \
+        --repo_env=BAZEL_HTTP_RULES_URLS_AS_DEFAULT_CANONICAL_ID=0 \
     || fail "expected fetch to succeed"
 }
 
@@ -465,15 +471,19 @@ EOF
   expect_log "Error downloading"
 }
 
-function test_break_url() {
+function test_http_archive_no_default_canonical_id() {
   setup_repository
 
-  bazel fetch --repository_cache="$repo_cache_dir" //zoo:breeding-program >& $TEST_log \
+  bazel fetch --repository_cache="$repo_cache_dir" \
+    --repo_env=BAZEL_HTTP_RULES_URLS_AS_DEFAULT_CANONICAL_ID=0 \
+    //zoo:breeding-program >& $TEST_log \
     || echo "Expected fetch to succeed"
 
   shutdown_server
 
-  bazel fetch --repository_cache="$repo_cache_dir" //zoo:breeding-program >& $TEST_log \
+  bazel fetch --repository_cache="$repo_cache_dir" \
+    --repo_env=BAZEL_HTTP_RULES_URLS_AS_DEFAULT_CANONICAL_ID=0 \
+    //zoo:breeding-program >& $TEST_log \
     || echo "Expected fetch to succeed"
 
   # Break url in WORKSPACE
@@ -489,24 +499,27 @@ http_archive(
 )
 EOF
 
-  # By default, cache entry will still match by sha256, even if url is changed.
-  bazel fetch --repository_cache="$repo_cache_dir" //zoo:breeding-program >& $TEST_log \
+  # Without the default canonical id, cache entry will still match by sha256, even if url is
+  # changed.
+  bazel fetch --repository_cache="$repo_cache_dir" \
+    --repo_env=BAZEL_HTTP_RULES_URLS_AS_DEFAULT_CANONICAL_ID=0 \
+    //zoo:breeding-program >& $TEST_log \
     || echo "Expected fetch to succeed"
 }
 
-function test_experimental_repository_cache_urls_as_default_canonical_id() {
+
+function test_http_archive_urls_as_default_canonical_id() {
+  # TODO: Remove when the integration test setup itself no longer relies on this.
+  add_to_bazelrc "common --repo_env=BAZEL_HTTP_RULES_URLS_AS_DEFAULT_CANONICAL_ID=1"
+
   setup_repository
 
-  bazel fetch --repository_cache="$repo_cache_dir" \
-        --experimental_repository_cache_urls_as_default_canonical_id \
-        //zoo:breeding-program >& $TEST_log \
+  bazel fetch --repository_cache="$repo_cache_dir" //zoo:breeding-program >& $TEST_log \
     || echo "Expected fetch to succeed"
 
   shutdown_server
 
-  bazel fetch --repository_cache="$repo_cache_dir" \
-        --experimental_repository_cache_urls_as_default_canonical_id \
-        //zoo:breeding-program >& $TEST_log \
+  bazel fetch --repository_cache="$repo_cache_dir" //zoo:breeding-program >& $TEST_log \
     || echo "Expected fetch to succeed"
 
   # Break url in WORKSPACE
@@ -523,9 +536,7 @@ http_archive(
 EOF
 
   # As repository cache key should depend on urls, we expect fetching to fail now.
-  bazel fetch --repository_cache="$repo_cache_dir" \
-        --experimental_repository_cache_urls_as_default_canonical_id \
-        //zoo:breeding-program >& $TEST_log \
+  bazel fetch --repository_cache="$repo_cache_dir" //zoo:breeding-program >& $TEST_log \
     && fail "expected failure" || :
 }
 
diff --git a/src/test/shell/testenv.sh.tmpl b/src/test/shell/testenv.sh.tmpl
index e7f53f1c8820c9..80d15b6df85e54 100755
--- a/src/test/shell/testenv.sh.tmpl
+++ b/src/test/shell/testenv.sh.tmpl
@@ -327,6 +327,9 @@ EOF
   if [[ -n ${REPOSITORY_CACHE:-} ]]; then
     echo "testenv.sh: Using repository cache at $REPOSITORY_CACHE."
     echo "common --repository_cache=$REPOSITORY_CACHE" >> $TEST_TMPDIR/bazelrc
+    # TODO: Remove this flag once all dependencies are mirrored.
+    # See https://github.com/bazelbuild/bazel/pull/19549 for more context.
+    echo "common --repo_env=BAZEL_HTTP_RULES_URLS_AS_DEFAULT_CANONICAL_ID=0" >> $TEST_TMPDIR/bazelrc
     if is_darwin; then
       # For reducing SSD usage on our physical Mac machines.
       echo "testenv.sh: Enabling --experimental_repository_cache_hardlinks"
diff --git a/src/test/tools/bzlmod/MODULE.bazel.lock b/src/test/tools/bzlmod/MODULE.bazel.lock
index 98461cff5ac2b6..ef16ccfb6da976 100644
--- a/src/test/tools/bzlmod/MODULE.bazel.lock
+++ b/src/test/tools/bzlmod/MODULE.bazel.lock
@@ -646,7 +646,7 @@
     },
     "@bazel_tools//tools/android:android_extensions.bzl%remote_android_tools_extensions": {
       "general": {
-        "bzlTransitiveDigest": "EFYd5Zc37KUKoseMe8brwJ5A2j/kUvPs5pIvTfGf3ok=",
+        "bzlTransitiveDigest": "hL6dUUghnqmrcFrs2bvFCvxPHBnPOY3ymycjIbgRQac=",
         "accumulatedFileDigests": {},
         "envVariables": {},
         "generatedRepoSpecs": {
@@ -730,7 +730,7 @@
     },
     "@bazel_tools//tools/test:extensions.bzl%remote_coverage_tools_extension": {
       "general": {
-        "bzlTransitiveDigest": "wd0+Kn35gYWv/xzdEzWg7vRQz6FZcfpne4WcwCs9d+o=",
+        "bzlTransitiveDigest": "DEE4EmNqW6/brpzbLQMxPvdWKALzpq/MeZyydMFsjLI=",
         "accumulatedFileDigests": {},
         "envVariables": {},
         "generatedRepoSpecs": {
@@ -750,7 +750,7 @@
     },
     "@rules_java~7.1.0//java:extensions.bzl%toolchains": {
       "general": {
-        "bzlTransitiveDigest": "EXsxSX2vQjCcI8jYez/O+Yb9H5reAMhLL3WXGPD6Scw=",
+        "bzlTransitiveDigest": "QHrMRjNPjXXgiGx8x2I1hGry4XALDWB19awlz4iYfwg=",
         "accumulatedFileDigests": {},
         "envVariables": {},
         "generatedRepoSpecs": {
@@ -1290,7 +1290,7 @@
     },
     "@rules_python~0.4.0//bzlmod:extensions.bzl%pip_install": {
       "general": {
-        "bzlTransitiveDigest": "fWWk0VJDA4P65oiSwJr5IKwxMWlFzootX8ZiYu5ETv8=",
+        "bzlTransitiveDigest": "jfaPoItGn7QX/GUEdWPqrBf5a380ATtgwvSoPQF/t/Q=",
         "accumulatedFileDigests": {},
         "envVariables": {},
         "generatedRepoSpecs": {