Repository: bazelbuild/bazel
Issue: Allow arbitrary headers to sent with Bazel Downloader
Labels: type: feature request
P3
team-ExternalDeps

### Description of the feature request:


Currently, bazel sets some headers when sending an HTTP request to fetch artifacts over the internet. While this works well for most cases, it doesn't work at all when the remote server requires some HTTP headers to be set to special values. 

For instance, currently bazel sets all the `Accept` header to `text/html, image/gif, image/jpeg, */*` in outgoing HTTP requests. In my case, its `Accept` header that needs to be set to `application/vnd.oci.image.manifest.v1+json` so that docker registry sees no problem responding when the image has oci manifest types.

See: https://github.com/bazelbuild/bazel/issues/16659#issuecomment-1470347164 

Related to #16682 

### What underlying problem are you trying to solve with this feature?


_No response_

### Which operating system are you running Bazel on?


darwin/arm64

### What is the output of `bazel info release`?


_No response_

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.


_No response_

### What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?


_No response_

### Have you found anything relevant by searching the web?

_No response_

### Any other information, logs, or outputs that you want to share?

_No response_

Issue comments:
cc @Wyverald 
Note, we'll need to add a workaround in rules_oci for now, however it uses a requires using another tool during the repository rule, making it require a fake toolchain resolution and also the download it makes is not cached anywhere.
@meteorcloudy Do you see any potential issues with allowing `rctx.download[_and_extract]` to set arbitrary headers? 

@thesayyn Would you be interested in contributing this functionality if you get the okay from maintainers? I could help with pre-reviews and references if needed.
> Do you see any potential issues with allowing rctx.download[_and_extract] to set arbitrary headers?

Not really. @tjgq integrated credential helper for repository download, do you have any concern here?
We already [merge](https://cs.opensource.google/bazel/bazel/+/master:src/test/shell/bazel/starlark_repository_test.sh?q=test_auth_from_credential_helper_overrides_starlark) authentication headers from the credential helper and the argument to `rctx.download*` so adding additional headers from the Starlark side should work.
> @thesayyn Would you be interested in contributing this functionality if you get the okay from maintainers? I could help with pre-reviews and references if needed.

Yes, I'd be willing to do that. 
@fmeum I'd appreciate your points on https://github.com/bazelbuild/bazel/pull/19501
I know this may not be directly related to bazel, but this is a message for others who are directed here by the rules_oci error log.

An checksum failed was throwed, and the error log looks like below:
```
WARNING: Could not fetch the manifest. Either there was an authentication issue or trying to pull an image with OCI image media types.
Falling back to using `curl`. See https://github.com/bazelbuild/bazel/issues/17829 for the context.
Analyzing: target ... (89 packages loaded, 6572 targets configured)
INFO: Repository rules_oci~1.4.0~oci~distroless_static_linux_amd64 instantiated at:
callstack not available
Repository rule oci_pull defined at:
/home/admin/.cache/bazel/_bazel_admin/23e125c776c0411833d0c469fe9d82fd/external/rules_oci~1.4.0/oci/private/pull.bzl:427:27: in <toplevel>
ERROR: An error occurred during the fetch of repository 'rules_oci~1.4.0~oci~distroless_static_linux_amd64':
Traceback (most recent call last):
File "/home/admin/.cache/bazel/_bazel_admin/23e125c776c0411833d0c469fe9d82fd/external/rules_oci~1.4.0/oci/private/pull.bzl", line 378, column 57, in _oci_pull_impl
mf, mf_len, mf_digest = downloader.download_manifest(rctx.attr.identifier, "manifest.json")
File "/home/admin/.cache/bazel/_bazel_admin/23e125c776c0411833d0c469fe9d82fd/external/rules_oci~1.4.0/oci/private/pull.bzl", line 319, column 74, in lambda
download_manifest = lambda identifier, output: _download_manifest(rctx, state, identifier, output),
File "/home/admin/.cache/bazel/_bazel_admin/23e125c776c0411833d0c469fe9d82fd/external/rules_oci~1.4.0/oci/private/pull.bzl", line 281, column 18, in _download_manifest
_download(
File "/home/admin/.cache/bazel/_bazel_admin/23e125c776c0411833d0c469fe9d82fd/external/rules_oci~1.4.0/oci/private/pull.bzl", line 247, column 23, in _download
return download_fn(
File "/home/admin/.cache/bazel/_bazel_admin/23e125c776c0411833d0c469fe9d82fd/external/rules_oci~1.4.0/oci/private/download.bzl", line 129, column 29, in _download
cache_it = rctx.download(
Error in download: com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException: Checksum was e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 but wanted 6706c73aae2afaa8201d63cc3dda48753c09bcd6c300762251065c0f7e602b25
ERROR: <builtin>: fetching oci_pull rule //:rules_oci~1.4.0~oci~distroless_static_linux_amd64: Traceback (most recent call last):
File "/home/admin/.cache/bazel/_bazel_admin/23e125c776c0411833d0c469fe9d82fd/external/rules_oci~1.4.0/oci/private/pull.bzl", line 378, column 57, in _oci_pull_impl
mf, mf_len, mf_digest = downloader.download_manifest(rctx.attr.identifier, "manifest.json")
File "/home/admin/.cache/bazel/_bazel_admin/23e125c776c0411833d0c469fe9d82fd/external/rules_oci~1.4.0/oci/private/pull.bzl", line 319, column 74, in lambda
download_manifest = lambda identifier, output: _download_manifest(rctx, state, identifier, output),
File "/home/admin/.cache/bazel/_bazel_admin/23e125c776c0411833d0c469fe9d82fd/external/rules_oci~1.4.0/oci/private/pull.bzl", line 281, column 18, in _download_manifest
_download(
File "/home/admin/.cache/bazel/_bazel_admin/23e125c776c0411833d0c469fe9d82fd/external/rules_oci~1.4.0/oci/private/pull.bzl", line 247, column 23, in _download
return download_fn(
File "/home/admin/.cache/bazel/_bazel_admin/23e125c776c0411833d0c469fe9d82fd/external/rules_oci~1.4.0/oci/private/download.bzl", line 129, column 29, in _download
cache_it = rctx.download(
Error in download: com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException: Checksum was e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 but wanted 6706c73aae2afaa8201d63cc3dda48753c09bcd6c300762251065c0f7e602b25
ERROR: /home/admin/.cache/bazel/_bazel_admin/23e125c776c0411833d0c469fe9d82fd/external/rules_oci~1.4.0~oci~distroless_static/BUILD.bazel:1:6: @rules_oci~1.4.0~oci~distroless_static//:distroless_static depends on @rules_oci~1.4.0~oci~distroless_static_linux_amd64//:distroless_static_linux_amd64 in repository @rules_oci~1.4.0~oci~distroless_static_linux_amd64 which failed to fetch. no such package '@rules_oci~1.4.0~oci~distroless_static_linux_amd64//': com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException: Checksum was e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 but wanted 6706c73aae2afaa8201d63cc3dda48753c09bcd6c300762251065c0f7e602b25
```

I believe the reason that causes this problem is [here](https://github.com/bazel-contrib/rules_oci/blob/v1.4.0/oci/private/download.bzl#L130). To cache the file downloaded from `curl`, here using `ctx.download` downloads file from local which is impossible to fetch from remote downloader machine.

To workaround this, flag `--experimental_remote_downloader_local_fallback=true` could be helpful.


releate: 
- https://github.com/bazelbuild/bazel/issues/17771
- https://github.com/bazel-contrib/rules_oci/issues/275
Is this something that will require changes to `FetchBlob` RPC? That is, https://github.com/bazelbuild/remote-apis/blob/7e33c12ee96192f12b7fbb089e6fe1af6a29ef14/build/bazel/remote/asset/v1/remote_asset.proto#L154 doesn't include a mechanism for the client to specify headers.
I don't there's a mechanism for it for remote downloaders. 
I agree - it sounds like it'll require an addition to the protocol.
A fix for this issue has been included in [Bazel 7.1.0 RC1](https://releases.bazel.build/7.1.0/rc1/index.html). Please test out the release candidate and report any issues as soon as possible. Thanks!
I've opened https://github.com/bazelbuild/bazel/pull/21490 which is necessary so that rulesets which adopt this feature don't break users of the remote downloader service. If it looks good, I'd appreciate it if we could pull into either 7.1.0 or 7.1.1.
> I've opened #21490 which is necessary so that rulesets which adopt this feature don't break users of the remote downloader service. If it looks good, I'd appreciate it if we could pull into either 7.1.0 or 7.1.1.

cc: @Wyverald @meteorcloudy
> We already [merge](https://cs.opensource.google/bazel/bazel/+/master:src/test/shell/bazel/starlark_repository_test.sh?q=test_auth_from_credential_helper_overrides_starlark) authentication headers from the credential helper and the argument to `rctx.download*` so adding additional headers from the Starlark side should work.

@tjgq Just want to flag that `credential_helper` do not override the headers, it just adds another header line. The headers that Bazel sends when `credential_helper` returns `Accept` are something like this:
```
    Accept: text/html, image/gif, image/jpeg, */*\r\n
    User-Agent: Bazel/release 7.1.0\r\n
    Authorization: Bearer XXXXXX\r\n
    Accept-Encoding: gzip\r\n
    Accept: application/octet-stream\r\n
```

The tests in [`starlark_repository_test.sh`](https://github.com/bazelbuild/bazel/blob/master/src/test/shell/bazel/starlark_repository_test.sh) pass only since [`testing_server.py`](https://github.com/bazelbuild/bazel/blob/master/src/test/shell/bazel/testing_server.py) does not correctly merge headers despite [RFC9110](https://www.rfc-editor.org/rfc/rfc9110.html#name-field-order).

If you

```bash
python src/test/shell/bazel/testing_server.py --dump_headers headers.txt always /dev/null
```

and then 

```bash
curl -L http://localhost:43726/ -H "Accept: this" -H "Accept: that" 
cat header.txt
{"Accept": "that"}
```

Only the last header is captured. 

Pull request review comments:

Patch that resolved the issue:
diff --git a/src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/DelegatingDownloader.java b/src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/DelegatingDownloader.java
index 249cc22c7c0d19..3b06fec38ce2ee 100644
--- a/src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/DelegatingDownloader.java
+++ b/src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/DelegatingDownloader.java
@@ -47,6 +47,7 @@ public void setDelegate(@Nullable Downloader delegate) {
   @Override
   public void download(
       List<URL> urls,
+      Map<String, List<String>> headers,
       Credentials credentials,
       Optional<Checksum> checksum,
       String canonicalId,
@@ -60,6 +61,14 @@ public void download(
       downloader = delegate;
     }
     downloader.download(
-        urls, credentials, checksum, canonicalId, destination, eventHandler, clientEnv, type);
+        urls,
+        headers,
+        credentials,
+        checksum,
+        canonicalId,
+        destination,
+        eventHandler,
+        clientEnv,
+        type);
   }
 }
diff --git a/src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/DownloadManager.java b/src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/DownloadManager.java
index 43853fdfdd6701..a56ce634c24307 100644
--- a/src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/DownloadManager.java
+++ b/src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/DownloadManager.java
@@ -116,6 +116,7 @@ public void setCredentialFactory(CredentialFactory credentialFactory) {
 
   public Future<Path> startDownload(
       List<URL> originalUrls,
+      Map<String, List<String>> headers,
       Map<URI, Map<String, List<String>>> authHeaders,
       Optional<Checksum> checksum,
       String canonicalId,
@@ -129,6 +130,7 @@ public Future<Path> startDownload(
           try (SilentCloseable c = Profiler.instance().profile("fetching: " + context)) {
             return downloadInExecutor(
                 originalUrls,
+                headers,
                 authHeaders,
                 checksum,
                 canonicalId,
@@ -154,6 +156,7 @@ public Path finalizeDownload(Future<Path> download) throws IOException, Interrup
 
   public Path download(
       List<URL> originalUrls,
+      Map<String, List<String>> headers,
       Map<URI, Map<String, List<String>>> authHeaders,
       Optional<Checksum> checksum,
       String canonicalId,
@@ -166,6 +169,7 @@ public Path download(
     Future<Path> future =
         startDownload(
             originalUrls,
+            headers,
             authHeaders,
             checksum,
             canonicalId,
@@ -197,6 +201,7 @@ public Path download(
    */
   private Path downloadInExecutor(
       List<URL> originalUrls,
+      Map<String, List<String>> headers,
       Map<URI, Map<String, List<String>>> authHeaders,
       Optional<Checksum> checksum,
       String canonicalId,
@@ -339,6 +344,7 @@ private Path downloadInExecutor(
       try {
         downloader.download(
             rewrittenUrls,
+            headers,
             credentialFactory.create(rewrittenAuthHeaders),
             checksum,
             canonicalId,
diff --git a/src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/Downloader.java b/src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/Downloader.java
index 1e8fc932b43b08..79a0076a3f56e1 100644
--- a/src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/Downloader.java
+++ b/src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/Downloader.java
@@ -42,6 +42,7 @@ public interface Downloader {
    */
   void download(
       List<URL> urls,
+      Map<String, List<String>> headers,
       Credentials credentials,
       Optional<Checksum> checksum,
       String canonicalId,
diff --git a/src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/HttpConnectorMultiplexer.java b/src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/HttpConnectorMultiplexer.java
index 1d5ba9b08a435c..b82587b5a91a0c 100644
--- a/src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/HttpConnectorMultiplexer.java
+++ b/src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/HttpConnectorMultiplexer.java
@@ -75,7 +75,7 @@ final class HttpConnectorMultiplexer {
   }
 
   public HttpStream connect(URL url, Optional<Checksum> checksum) throws IOException {
-    return connect(url, checksum, StaticCredentials.EMPTY, Optional.empty());
+    return connect(url, checksum, ImmutableMap.of(), StaticCredentials.EMPTY, Optional.empty());
   }
 
   /**
@@ -96,14 +96,23 @@ public HttpStream connect(URL url, Optional<Checksum> checksum) throws IOExcepti
    * @throws IllegalArgumentException if {@code urls} is empty or has an unsupported protocol
    */
   public HttpStream connect(
-      URL url, Optional<Checksum> checksum, Credentials credentials, Optional<String> type)
+      URL url,
+      Optional<Checksum> checksum,
+      Map<String, List<String>> headers,
+      Credentials credentials,
+      Optional<String> type)
       throws IOException {
     Preconditions.checkArgument(HttpUtils.isUrlSupportedByDownloader(url));
     if (Thread.interrupted()) {
       throw new InterruptedIOException();
     }
+    ImmutableMap.Builder<String, List<String>> baseHeaders = new ImmutableMap.Builder<>();
+    baseHeaders.putAll(headers);
+    // REQUEST_HEADERS should not be overridable by user provided headers
+    baseHeaders.putAll(REQUEST_HEADERS);
+
     Function<URL, ImmutableMap<String, List<String>>> headerFunction =
-        getHeaderFunction(REQUEST_HEADERS, credentials);
+        getHeaderFunction(baseHeaders.buildKeepingLast(), credentials);
     URLConnection connection = connector.connect(url, headerFunction);
     return httpStreamFactory.create(
         connection,
diff --git a/src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/HttpDownloader.java b/src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/HttpDownloader.java
index 3e9e0f150a6ac8..35e0ea2ebb8577 100644
--- a/src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/HttpDownloader.java
+++ b/src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/HttpDownloader.java
@@ -16,6 +16,7 @@
 
 import com.google.auth.Credentials;
 import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
 import com.google.common.collect.Iterables;
 import com.google.common.io.ByteStreams;
 import com.google.devtools.build.lib.buildeventstream.FetchEvent;
@@ -75,6 +76,7 @@ public void setMaxRetryTimeout(Duration maxRetryTimeout) {
   @Override
   public void download(
       List<URL> urls,
+      Map<String, List<String>> headers,
       Credentials credentials,
       Optional<Checksum> checksum,
       String canonicalId,
@@ -94,7 +96,7 @@ public void download(
     for (URL url : urls) {
       SEMAPHORE.acquire();
 
-      try (HttpStream payload = multiplexer.connect(url, checksum, credentials, type);
+      try (HttpStream payload = multiplexer.connect(url, checksum, headers, credentials, type);
           OutputStream out = destination.getOutputStream()) {
         try {
           ByteStreams.copy(payload, out);
@@ -153,7 +155,8 @@ public byte[] downloadAndReadOneUrl(
     ByteArrayOutputStream out = new ByteArrayOutputStream();
     SEMAPHORE.acquire();
     try (HttpStream payload =
-        multiplexer.connect(url, Optional.empty(), credentials, Optional.empty())) {
+        multiplexer.connect(
+            url, Optional.empty(), ImmutableMap.of(), credentials, Optional.empty())) {
       ByteStreams.copy(payload, out);
     } catch (SocketTimeoutException e) {
       // SocketTimeoutExceptions are InterruptedIOExceptions; however they do not signify
diff --git a/src/main/java/com/google/devtools/build/lib/bazel/repository/starlark/StarlarkBaseExternalContext.java b/src/main/java/com/google/devtools/build/lib/bazel/repository/starlark/StarlarkBaseExternalContext.java
index 50c5554c772e6f..30e9f4f34d411c 100644
--- a/src/main/java/com/google/devtools/build/lib/bazel/repository/starlark/StarlarkBaseExternalContext.java
+++ b/src/main/java/com/google/devtools/build/lib/bazel/repository/starlark/StarlarkBaseExternalContext.java
@@ -291,6 +291,32 @@ private static ImmutableMap<URI, Map<String, List<String>>> getAuthHeaders(
     return res;
   }
 
+  private static ImmutableMap<String, List<String>> getHeaderContents(Dict<?, ?> x, String what)
+      throws EvalException {
+    Dict<String, Object> headersUnchecked =
+        (Dict<String, Object>) Dict.cast(x, String.class, Object.class, what);
+    ImmutableMap.Builder<String, List<String>> headers = new ImmutableMap.Builder<>();
+
+    for (Map.Entry<String, Object> entry : headersUnchecked.entrySet()) {
+      ImmutableList<String> headerValue;
+      Object valueUnchecked = entry.getValue();
+      if (valueUnchecked instanceof Sequence) {
+        headerValue =
+            Sequence.cast(valueUnchecked, String.class, "header values").getImmutableList();
+      } else if (valueUnchecked instanceof String) {
+        headerValue = ImmutableList.of(valueUnchecked.toString());
+      } else {
+        throw new EvalException(
+            String.format(
+                "%s argument must be a dict whose keys are string and whose values are either"
+                    + " string or sequence of string",
+                what));
+      }
+      headers.put(entry.getKey(), headerValue);
+    }
+    return headers.buildOrThrow();
+  }
+
   private static ImmutableList<String> checkAllUrls(Iterable<?> urlList) throws EvalException {
     ImmutableList.Builder<String> result = ImmutableList.builder();
 
@@ -587,6 +613,11 @@ private StructImpl completeDownload(PendingDownload pendingDownload)
             defaultValue = "{}",
             named = true,
             doc = "An optional dict specifying authentication information for some of the URLs."),
+        @Param(
+            name = "headers",
+            defaultValue = "{}",
+            named = true,
+            doc = "An optional dict specifying http headers for all URLs."),
         @Param(
             name = "integrity",
             defaultValue = "''",
@@ -617,6 +648,7 @@ public Object download(
       Boolean allowFail,
       String canonicalId,
       Dict<?, ?> authUnchecked, // <String, Dict> expected
+      Dict<?, ?> headersUnchecked, // <String, List<String> | String> expected
       String integrity,
       Boolean block,
       StarlarkThread thread)
@@ -625,6 +657,8 @@ public Object download(
     ImmutableMap<URI, Map<String, List<String>>> authHeaders =
         getAuthHeaders(getAuthContents(authUnchecked, "auth"));
 
+    ImmutableMap<String, List<String>> headers = getHeaderContents(headersUnchecked, "headers");
+
     ImmutableList<URL> urls =
         getUrls(
             url,
@@ -670,6 +704,7 @@ public Object download(
       Future<Path> downloadFuture =
           downloadManager.startDownload(
               urls,
+              headers,
               authHeaders,
               checksum,
               canonicalId,
@@ -778,6 +813,11 @@ public Object download(
             defaultValue = "{}",
             named = true,
             doc = "An optional dict specifying authentication information for some of the URLs."),
+        @Param(
+            name = "headers",
+            defaultValue = "{}",
+            named = true,
+            doc = "An optional dict specifying http headers for all URLs."),
         @Param(
             name = "integrity",
             defaultValue = "''",
@@ -809,13 +849,16 @@ public StructImpl downloadAndExtract(
       String stripPrefix,
       Boolean allowFail,
       String canonicalId,
-      Dict<?, ?> auth, // <String, Dict> expected
+      Dict<?, ?> authUnchecked, // <String, Dict> expected
+      Dict<?, ?> headersUnchecked, // <String, List<String> | String> expected
       String integrity,
       Dict<?, ?> renameFiles, // <String, String> expected
       StarlarkThread thread)
       throws RepositoryFunctionException, InterruptedException, EvalException {
     ImmutableMap<URI, Map<String, List<String>>> authHeaders =
-        getAuthHeaders(getAuthContents(auth, "auth"));
+        getAuthHeaders(getAuthContents(authUnchecked, "auth"));
+
+    ImmutableMap<String, List<String>> headers = getHeaderContents(headersUnchecked, "headers");
 
     ImmutableList<URL> urls =
         getUrls(
@@ -862,6 +905,7 @@ public StructImpl downloadAndExtract(
       Future<Path> pendingDownload =
           downloadManager.startDownload(
               urls,
+              headers,
               authHeaders,
               checksum,
               canonicalId,
diff --git a/src/main/java/com/google/devtools/build/lib/remote/downloader/GrpcRemoteDownloader.java b/src/main/java/com/google/devtools/build/lib/remote/downloader/GrpcRemoteDownloader.java
index 29466a5e9d9407..ddc011ef99432d 100644
--- a/src/main/java/com/google/devtools/build/lib/remote/downloader/GrpcRemoteDownloader.java
+++ b/src/main/java/com/google/devtools/build/lib/remote/downloader/GrpcRemoteDownloader.java
@@ -110,6 +110,7 @@ public void close() {
   @Override
   public void download(
       List<URL> urls,
+      Map<String, List<String>> headers,
       Credentials credentials,
       Optional<Checksum> checksum,
       String canonicalId,
@@ -154,7 +155,15 @@ public void download(
       eventHandler.handle(
           Event.warn("Remote Cache: " + Utils.grpcAwareErrorMessage(e, verboseFailures)));
       fallbackDownloader.download(
-          urls, credentials, checksum, canonicalId, destination, eventHandler, clientEnv, type);
+          urls,
+          headers,
+          credentials,
+          checksum,
+          canonicalId,
+          destination,
+          eventHandler,
+          clientEnv,
+          type);
     }
   }
 


Test patch:
diff --git a/src/test/java/com/google/devtools/build/lib/bazel/repository/downloader/HttpDownloaderTest.java b/src/test/java/com/google/devtools/build/lib/bazel/repository/downloader/HttpDownloaderTest.java
index 2451af6027aabf..0e59d412abb975 100644
--- a/src/test/java/com/google/devtools/build/lib/bazel/repository/downloader/HttpDownloaderTest.java
+++ b/src/test/java/com/google/devtools/build/lib/bazel/repository/downloader/HttpDownloaderTest.java
@@ -117,6 +117,7 @@ public void downloadFrom1UrlOk() throws IOException, InterruptedException {
               Collections.singletonList(
                   new URL(String.format("http://localhost:%d/foo", server.getLocalPort()))),
               Collections.emptyMap(),
+              Collections.emptyMap(),
               Optional.empty(),
               "testCanonicalId",
               Optional.empty(),
@@ -181,6 +182,7 @@ public void downloadFrom2UrlsFirstOk() throws IOException, InterruptedException
           downloadManager.download(
               urls,
               Collections.emptyMap(),
+              Collections.emptyMap(),
               Optional.empty(),
               "testCanonicalId",
               Optional.empty(),
@@ -248,6 +250,7 @@ public void downloadFrom2UrlsFirstSocketTimeoutOnBodyReadSecondOk()
           downloadManager.download(
               urls,
               Collections.emptyMap(),
+              Collections.emptyMap(),
               Optional.empty(),
               "testCanonicalId",
               Optional.empty(),
@@ -317,6 +320,7 @@ public void downloadFrom2UrlsBothSocketTimeoutDuringBodyRead()
         downloadManager.download(
             urls,
             Collections.emptyMap(),
+            Collections.emptyMap(),
             Optional.empty(),
             "testCanonicalId",
             Optional.empty(),
@@ -371,6 +375,7 @@ public void downloadOneUrl_ok() throws IOException, InterruptedException {
       httpDownloader.download(
           Collections.singletonList(
               new URL(String.format("http://localhost:%d/foo", server.getLocalPort()))),
+          Collections.emptyMap(),
           StaticCredentials.EMPTY,
           Optional.empty(),
           "testCanonicalId",
@@ -410,6 +415,7 @@ public void downloadOneUrl_notFound() throws IOException, InterruptedException {
               httpDownloader.download(
                   Collections.singletonList(
                       new URL(String.format("http://localhost:%d/foo", server.getLocalPort()))),
+                  Collections.emptyMap(),
                   StaticCredentials.EMPTY,
                   Optional.empty(),
                   "testCanonicalId",
@@ -470,6 +476,7 @@ public void downloadTwoUrls_firstNotFoundAndSecondOk() throws IOException, Inter
       Path destination = fs.getPath(workingDir.newFile().getAbsolutePath());
       httpDownloader.download(
           urls,
+          Collections.emptyMap(),
           StaticCredentials.EMPTY,
           Optional.empty(),
           "testCanonicalId",
@@ -564,13 +571,14 @@ public void download_contentLengthMismatch_propagateErrorIfNotRetry() throws Exc
                   throw new ContentLengthMismatchException(0, data.length);
                 })
         .when(downloader)
-        .download(any(), any(), any(), any(), any(), any(), any(), any());
+        .download(any(), any(), any(), any(), any(), any(), any(), any(), any());
 
     assertThrows(
         ContentLengthMismatchException.class,
         () ->
             downloadManager.download(
                 ImmutableList.of(new URL("http://localhost")),
+                Collections.emptyMap(),
                 ImmutableMap.of(),
                 Optional.empty(),
                 "testCanonicalId",
@@ -597,7 +605,7 @@ public void download_contentLengthMismatch_retries() throws Exception {
                   if (times.getAndIncrement() < 3) {
                     throw new ContentLengthMismatchException(0, data.length);
                   }
-                  Path output = invocationOnMock.getArgument(4, Path.class);
+                  Path output = invocationOnMock.getArgument(5, Path.class);
                   try (OutputStream outputStream = output.getOutputStream()) {
                     ByteStreams.copy(new ByteArrayInputStream(data), outputStream);
                   }
@@ -605,12 +613,13 @@ public void download_contentLengthMismatch_retries() throws Exception {
                   return null;
                 })
         .when(downloader)
-        .download(any(), any(), any(), any(), any(), any(), any(), any());
+        .download(any(), any(), any(), any(), any(), any(), any(), any(), any());
 
     Path result =
         downloadManager.download(
             ImmutableList.of(new URL("http://localhost")),
             ImmutableMap.of(),
+            ImmutableMap.of(),
             Optional.empty(),
             "testCanonicalId",
             Optional.empty(),
diff --git a/src/test/java/com/google/devtools/build/lib/remote/downloader/GrpcRemoteDownloaderTest.java b/src/test/java/com/google/devtools/build/lib/remote/downloader/GrpcRemoteDownloaderTest.java
index ced0fe561c830a..f881d862800e83 100644
--- a/src/test/java/com/google/devtools/build/lib/remote/downloader/GrpcRemoteDownloaderTest.java
+++ b/src/test/java/com/google/devtools/build/lib/remote/downloader/GrpcRemoteDownloaderTest.java
@@ -178,6 +178,7 @@ private static byte[] downloadBlob(
     final Path destination = scratch.resolve("output file path");
     downloader.download(
         urls,
+        ImmutableMap.of(),
         StaticCredentials.EMPTY,
         checksum,
         canonicalId,
@@ -240,13 +241,13 @@ public void fetchBlob(
             invocation -> {
               List<URL> urls = invocation.getArgument(0);
               if (urls.equals(ImmutableList.of(new URL("http://example.com/content.txt")))) {
-                Path output = invocation.getArgument(4);
+                Path output = invocation.getArgument(5);
                 FileSystemUtils.writeContent(output, content);
               }
               return null;
             })
         .when(fallbackDownloader)
-        .download(any(), any(), any(), any(), any(), any(), any(), any());
+        .download(any(), any(), any(), any(), any(), any(), any(), any(), any());
     final GrpcRemoteDownloader downloader = newDownloader(cacheClient, fallbackDownloader);
 
     final byte[] downloaded =
diff --git a/src/test/shell/bazel/remote_helpers.sh b/src/test/shell/bazel/remote_helpers.sh
index 6a66398d3eabd9..e8181d210ff7db 100755
--- a/src/test/shell/bazel/remote_helpers.sh
+++ b/src/test/shell/bazel/remote_helpers.sh
@@ -179,6 +179,28 @@ function serve_timeout() {
   cd -
 }
 
+# Serves a HTTP 200 Ok response with headers dumped into the file
+# Args:
+#  $1: required; path to the file
+#  $2: optional; path to the file where headers will be written to.
+function serve_file_header_dump() {
+  file_name=served_file.$$
+  cat $1 > "${TEST_TMPDIR}/$file_name"
+  nc_log="${TEST_TMPDIR}/nc.log"
+  rm -f $nc_log
+  touch $nc_log
+  cd "${TEST_TMPDIR}"
+  port_file=server-port.$$
+  rm -f $port_file
+  python3 $python_server always $file_name --dump_headers ${2:-"headers.json"} > $port_file &
+  nc_pid=$!
+  while ! grep started $port_file; do sleep 1; done
+  nc_port=$(head -n 1 $port_file)
+  fileserver_port=$nc_port
+  wait_for_server_startup
+  cd -
+}
+
 # Waits for the SimpleHTTPServer to actually start up before the test is run.
 # Otherwise the entire test can run before the server starts listening for
 # connections, which causes flakes.
diff --git a/src/test/shell/bazel/starlark_repository_test.sh b/src/test/shell/bazel/starlark_repository_test.sh
index dc45b9e2be0f54..d5008c47cd5dd4 100755
--- a/src/test/shell/bazel/starlark_repository_test.sh
+++ b/src/test/shell/bazel/starlark_repository_test.sh
@@ -2354,8 +2354,8 @@ genrule(
   cmd = "cp $< $@",
 )
 EOF
-
-  bazel build --repository_disable_download //:it || fail "Failed to build"
+  # for some reason --repository_disable_download fails with bzlmod trying to download @platforms.
+  bazel build --repository_disable_download --noenable_bzlmod //:it || fail "Failed to build"
 }
 
 function test_no_restarts_fetching_with_worker_thread() {
@@ -2408,6 +2408,181 @@ EOF
     || fail "Expected build to succeed"
 }
 
+
+function test_cred_helper_overrides_starlark_headers() {
+  if "$is_windows"; then
+    # Skip on Windows: credential helper is a Python script.
+    return
+  fi
+
+  setup_credential_helper
+
+  filename="cred_helper_starlark.txt"
+  echo $filename > $filename
+  sha256="$(sha256sum $filename | head -c 64)"
+  serve_file_header_dump $filename credhelper_headers.json
+
+  setup_starlark_repository
+
+  cat > test.bzl <<EOF
+def _impl(repository_ctx):
+  repository_ctx.file("BUILD")
+  repository_ctx.download(
+    url = "http://127.0.0.1:$nc_port/$filename",
+    output = "test.txt",
+    sha256 = "$sha256",
+    headers = {
+      "Authorization": ["should be overidden"],
+      "X-Custom-Token": ["foo"]
+    }
+  )
+
+repo = repository_rule(implementation=_impl)
+EOF
+
+
+  bazel build --credential_helper="${TEST_TMPDIR}/credhelper" @foo//:all || fail "expected bazel to succeed"
+
+  headers="${TEST_TMPDIR}/credhelper_headers.json"
+  assert_contains '"Authorization": "Bearer TOKEN"' "$headers"
+  assert_contains '"X-Custom-Token": "foo"' "$headers"
+  assert_not_contains "should be overidden" "$headers"
+}
+
+function test_netrc_overrides_starlark_headers() {
+  filename="netrc_headers.txt"
+  echo $filename > $filename
+  sha256="$(sha256sum $filename | head -c 64)"
+  serve_file_header_dump $filename netrc_headers.json
+
+  setup_starlark_repository
+
+  cat > .netrc <<EOF
+machine 127.0.0.1
+login foo
+password bar
+EOF
+
+  cat > test.bzl <<EOF
+load("@bazel_tools//tools/build_defs/repo:utils.bzl", "read_netrc", "use_netrc")
+
+def _impl(repository_ctx):
+  url = "http://127.0.0.1:$nc_port/$filename"
+  netrc = read_netrc(repository_ctx, repository_ctx.attr.netrc)
+  auth = use_netrc(netrc, [url], {})
+  repository_ctx.file("BUILD")
+  repository_ctx.download(
+    url = url,
+    output = "$filename",
+    sha256 = "$sha256",
+    headers = {
+      "Authorization": ["should be overidden"],
+      "X-Custom-Token": ["foo"]
+    },
+    auth = auth
+  )
+
+repo = repository_rule(implementation=_impl, attrs = {"netrc": attr.label(default = ":.netrc")})
+EOF
+
+
+  bazel build @foo//:all || fail "expected bazel to succeed"
+
+  headers="${TEST_TMPDIR}/netrc_headers.json"
+  assert_contains '"Authorization": "Basic Zm9vOmJhcg=="' "$headers"
+  assert_contains '"X-Custom-Token": "foo"' "$headers"
+  assert_not_contains "should be overidden" "$headers"
+}
+
+
+function test_starlark_headers_override_default_headers() {
+
+  filename="default_headers.txt"
+  echo $filename > $filename
+  sha256="$(sha256sum $filename | head -c 64)"
+  serve_file_header_dump $filename default_headers.json
+
+  setup_starlark_repository
+
+  cat > test.bzl <<EOF
+def _impl(repository_ctx):
+  repository_ctx.file("BUILD")
+  repository_ctx.download(
+    url = "http://127.0.0.1:$nc_port/$filename",
+    output = "$filename",
+    sha256 = "$sha256",
+    headers = {
+      "Accept": ["application/vnd.oci.image.index.v1+json, application/vnd.oci.image.manifest.v1+json"],
+    }
+  )
+
+repo = repository_rule(implementation=_impl)
+EOF
+
+  bazel build @foo//:all || fail "expected bazel to succeed"
+
+  headers="${TEST_TMPDIR}/default_headers.json"
+  assert_contains '"Accept": "application/vnd.oci.image.index.v1+json, application/vnd.oci.image.manifest.v1+json"' "$headers"
+  assert_not_contains '"Accept": "text/html, image/gif, image/jpeg, */*"' "$headers"
+}
+
+function test_invalid_starlark_headers() {
+
+  filename="invalid_headers.txt"
+  echo $filename > $filename
+  sha256="$(sha256sum $filename | head -c 64)"
+  serve_file_header_dump $filename invalid_headers.json
+
+  setup_starlark_repository
+
+  cat > test.bzl <<EOF
+def _impl(repository_ctx):
+  repository_ctx.file("BUILD")
+  repository_ctx.download(
+    url = "http://127.0.0.1:$nc_port/$filename",
+    output = "$filename",
+    sha256 = "$sha256",
+    headers = {
+      "Accept": 1,
+    }
+  )
+
+repo = repository_rule(implementation=_impl)
+EOF
+
+  bazel build @foo//:all >& $TEST_log && fail "expected bazel to fail" || :
+  expect_log "headers argument must be a dict whose keys are string and whose values are either string or sequence of string"
+}
+
+function test_string_starlark_headers() {
+
+  filename="string_headers.txt"
+  echo $filename > $filename
+  sha256="$(sha256sum $filename | head -c 64)"
+  serve_file_header_dump $filename string_headers.json
+
+  setup_starlark_repository
+
+  cat > test.bzl <<EOF
+def _impl(repository_ctx):
+  repository_ctx.file("BUILD")
+  repository_ctx.download(
+    url = "http://127.0.0.1:$nc_port/$filename",
+    output = "$filename",
+    sha256 = "$sha256",
+    headers = {
+      "Accept": "application/text",
+    }
+  )
+
+repo = repository_rule(implementation=_impl)
+EOF
+
+  bazel build @foo//:all || fail "expected bazel to succeed"
+  headers="${TEST_TMPDIR}/string_headers.json"
+  assert_contains '"Accept": "application/text"' "$headers"
+}
+
 function test_repo_boundary_files() {
   create_new_workspace
   cat > MODULE.bazel <<EOF
diff --git a/src/test/shell/bazel/testing_server.py b/src/test/shell/bazel/testing_server.py
index 00c714792d2d95..e4255b46687394 100644
--- a/src/test/shell/bazel/testing_server.py
+++ b/src/test/shell/bazel/testing_server.py
@@ -17,6 +17,8 @@
 # pylint: disable=g-import-not-at-top,g-importing-member
 import argparse
 import base64
+import json
+
 try:
   from http.server import BaseHTTPRequestHandler
 except ImportError:
@@ -44,11 +46,13 @@ class Handler(BaseHTTPRequestHandler):
   auth = False
   not_found = False
   simulate_timeout = False
+  dump_headers = None
   filename = None
   redirect = None
   valid_headers = [
       b'Basic ' + base64.b64encode('foo:bar'.encode('ascii')), b'Bearer TOKEN'
   ]
+  unstable_headers = ['Host', 'User-Agent']
 
   def do_HEAD(self):  # pylint: disable=invalid-name
     self.send_response(200)
@@ -67,6 +71,13 @@ def do_GET(self):  # pylint: disable=invalid-name
       # fail without this being set.
       self.client_address = 'localhost'
 
+    if self.dump_headers:
+      headers = filter(
+          lambda hv: hv[0] not in self.unstable_headers, self.headers.items()
+      )
+      with open(self.dump_headers, 'w') as f:
+        f.write(json.dumps(dict(headers)))
+
     if self.simulate_timeout:
       while True:
         time.sleep(1)
@@ -110,6 +121,8 @@ def serve_file(self):
 def main(argv):
   parser = argparse.ArgumentParser()
   parser.add_argument('--unix_socket', action='store')
+  parser.add_argument('--dump_headers', action='store')
+
   parser.add_argument('mode', type=str, nargs='?')
   parser.add_argument('target', type=str, nargs='?')
   args = parser.parse_args(argv)
@@ -128,6 +141,8 @@ def main(argv):
       if args.target:
         Handler.filename = args.target
 
+  Handler.dump_headers = args.dump_headers
+
   httpd = None
   if args.unix_socket:
     httpd = UnixStreamServer(args.unix_socket, Handler)