Repository: bazelbuild/bazel
Issue: --disk_cache overrides --noremote_upload_local_results
Labels: type: bug
team-Remote-Exec

### Description of the feature request:

I would like to be able to use the `--disk_cache` as a read/write cache, but the remote cache as a read-only cache. I thought `--noremote_upload_local_results` would do this, but it seems that flag is just overridden by the `--disk_cache` option. It seems this is the expected behavior, since in #7512, I see it says:
>On put, Bazel will store items in both the disk and the HTTP cache.

Could this be changed to support this mixed use case?

### What underlying problem are you trying to solve with this feature?

We are still in the process of moving our build to bazel, and not all builds on dev machines are safe to be pushed to the shared remote cache, so we only populate the remote cache from CI builds. Being able to have a local cache is very helpful when switching between dev branches for super fast builds. Our current workaround is to run a remote cache on every machine that acts as a proxy to the real shared cache, but we would like to move away from all of that overhead.

Issue comments:
I agree. disk cache should always be read/write and the `remote_upload_local_results` flag should only apply to remote caching.
Hey @ishikhman! Have you had a chance to look any further into this?
@SrodriguezO we love contributions :)
We're happy to help! Just wanted to avoid duplicate efforts, since the issue's assigned. We can start looking into it next week.
We'll still hold off to hear back from @ishikhman though (to avoid duplicate efforts)
> We'll still hold off to hear back from @ishikhman though (to avoid duplicate efforts)

unfortunately, not yet. That would be really great if you could look into it! Removed myself from the assignment ;) 
Silly me.. I just realized this won't actually solve our problem :( 

We're exploring a move to RBE and are hoping to utilize its remote cache, but it isn't compatible with local disk caching in the first place :/
@SrodriguezO it's just not been implemented yet. It should be a straightforward PR. Would also love your contribution :)
@buchgr I dug around a bit and it seems a bit involved. I opened a draft PR to test things out as well as a feature request to formalize the process and to get some input. I'd love your feedback. https://github.com/bazelbuild/bazel/issues/8690

@buchgr any chance we could mark this as a blocker for 1.0?
I am hopeful that we'll get this fixed before 1.0 (September 2nd). However, I don't think that it qualifies as a blocker. Why do you think so?
It just seems like this feature is dead in the water until this is fixed
@meteorcloudy it seems like we're missing the `incompatible-change` tracking issue for flipping the default value of --incompatible_remote_results_ignore_disk to true? Can you create it?
@coeuvre Can you help create the tracking issue for this remote execution flag? https://bazel.build/contribute/breaking-changes
Created #15148.

Pull request review comments:
Suggested rephrasing:

"If set to true, --noremote_upload_local_results will not apply to the disk cache. If a combined cache is used, --noremote_upload_local_results will cause results to be written to the disk cache, but not uploaded to the remote cache. See #8216 for details."
Thanks.
I find the following version a bit more readable, but it's up to you: to change it or not :)
 `if (options.incompatibleUploadLocalResultsDiskOnly && !options.remoteUploadLocalResults)`
You'd need to negate the entire clause for those to be equivalent: 
`!(options.incompatibleUploadLocalResultsDiskOnly && !options.remoteUploadLocalResults)`

I think either way works, but I prefer the original : )
I agree with @SrodriguezO, also prefer the original. More suggestions are welcome.
How come you removed this?
hmm I see that you moved the progress status down to the other block, but will the logic between here and there cause a delay?
Can we move the `options.diskCache != null && !options.diskCache.isEmpty())` part to a helper `usesDiskCache` method? 
Can we rename to `incompatible_remote_results_flags_ignore_disk`? The current name makes me think this flag makes remote results flags apply to the disk cache only (which is the opposite of what they do)
It passes the tests. I don't see a strong reason to keep it separate. I might be wrong. @ishikhman do you have more context on it?
```
+ "If a combined cache is used: "
+ "\n\t--noremote_upload_local_results will cause results to be written to the disk cache, but not uploaded to the remote cache"
+ "\n\t--noremote_accept_cached will result in Bazel checking for results in the disk cache, but not in the remote cache."
+ "See #8216 for details."
```
Good idea.
Sounds good. I prefer just `incompatible_remote_results_ignore_disk` to make it a bit shorter.
Done.
It wouldn't cause tests to fail, the message might just not show up right away
I don't see a bit difference to be honest. Should be fine both ways.

Patch that resolved the issue:
diff --git a/src/main/java/com/google/devtools/build/lib/remote/RemoteCacheClientFactory.java b/src/main/java/com/google/devtools/build/lib/remote/RemoteCacheClientFactory.java
index a5944544bf79bc..2a8d9ddd98f7a4 100644
--- a/src/main/java/com/google/devtools/build/lib/remote/RemoteCacheClientFactory.java
+++ b/src/main/java/com/google/devtools/build/lib/remote/RemoteCacheClientFactory.java
@@ -47,11 +47,12 @@ public static RemoteCacheClient createDiskAndRemoteClient(
       PathFragment diskCachePath,
       boolean remoteVerifyDownloads,
       DigestUtil digestUtil,
-      RemoteCacheClient remoteCacheClient)
+      RemoteCacheClient remoteCacheClient,
+      RemoteOptions options)
       throws IOException {
     DiskCacheClient diskCacheClient =
         createDiskCache(workingDirectory, diskCachePath, remoteVerifyDownloads, digestUtil);
-    return new DiskAndRemoteCacheClient(diskCacheClient, remoteCacheClient);
+    return new DiskAndRemoteCacheClient(diskCacheClient, remoteCacheClient, options);
   }
 
   public static ReferenceCountedChannel createGrpcChannel(
@@ -159,7 +160,7 @@ private static RemoteCacheClient createDiskAndHttpCache(
 
     RemoteCacheClient httpCache = createHttp(options, cred, digestUtil);
     return createDiskAndRemoteClient(
-        workingDirectory, diskCachePath, options.remoteVerifyDownloads, digestUtil, httpCache);
+        workingDirectory, diskCachePath, options.remoteVerifyDownloads, digestUtil, httpCache, options);
   }
 
   public static boolean isDiskCache(RemoteOptions options) {
diff --git a/src/main/java/com/google/devtools/build/lib/remote/RemoteModule.java b/src/main/java/com/google/devtools/build/lib/remote/RemoteModule.java
index 9f4a11ba6ddcda..c589b37446e2d9 100644
--- a/src/main/java/com/google/devtools/build/lib/remote/RemoteModule.java
+++ b/src/main/java/com/google/devtools/build/lib/remote/RemoteModule.java
@@ -298,7 +298,8 @@ public void beforeCommand(CommandEnvironment env) throws AbruptExitException {
                   remoteOptions.diskCache,
                   remoteOptions.remoteVerifyDownloads,
                   digestUtil,
-                  cacheClient);
+                  cacheClient,
+                  remoteOptions);
         }
 
         RemoteCache remoteCache = new RemoteCache(cacheClient, remoteOptions, digestUtil);
diff --git a/src/main/java/com/google/devtools/build/lib/remote/RemoteSpawnCache.java b/src/main/java/com/google/devtools/build/lib/remote/RemoteSpawnCache.java
index 5c25075f592497..b1fa2a977dd09a 100644
--- a/src/main/java/com/google/devtools/build/lib/remote/RemoteSpawnCache.java
+++ b/src/main/java/com/google/devtools/build/lib/remote/RemoteSpawnCache.java
@@ -114,16 +114,11 @@ final class RemoteSpawnCache implements SpawnCache {
   public CacheHandle lookup(Spawn spawn, SpawnExecutionContext context)
       throws InterruptedException, IOException, ExecException {
     if (!Spawns.mayBeCached(spawn)
-        || (!Spawns.mayBeCachedRemotely(spawn) && isRemoteCache(options))) {
+        || (!Spawns.mayBeCachedRemotely(spawn) && useRemoteCache(options))) {
       // returning SpawnCache.NO_RESULT_NO_STORE in case the caching is disabled or in case
       // the remote caching is disabled and the only configured cache is remote.
       return SpawnCache.NO_RESULT_NO_STORE;
     }
-    boolean checkCache = options.remoteAcceptCached;
-
-    if (checkCache) {
-      context.report(ProgressStatus.CHECKING_CACHE, "remote-cache");
-    }
 
     SortedMap<PathFragment, ActionInput> inputMap = context.getInputMapping(true);
     MerkleTree merkleTree =
@@ -150,7 +145,9 @@ public CacheHandle lookup(Spawn spawn, SpawnExecutionContext context)
         TracingMetadataUtils.contextWithMetadata(buildRequestId, commandId, actionKey);
 
     Profiler prof = Profiler.instance();
-    if (checkCache) {
+    if (options.remoteAcceptCached ||
+        (options.incompatibleRemoteResultsIgnoreDisk && useDiskCache(options))) {
+      context.report(ProgressStatus.CHECKING_CACHE, "remote-cache");
       // Metadata will be available in context.current() until we detach.
       // This is done via a thread-local variable.
       Context previous = withMetadata.attach();
@@ -211,7 +208,8 @@ public CacheHandle lookup(Spawn spawn, SpawnExecutionContext context)
 
     context.prefetchInputs();
 
-    if (options.remoteUploadLocalResults) {
+    if (options.remoteUploadLocalResults ||
+        (options.incompatibleRemoteResultsIgnoreDisk && useDiskCache(options))) {
       return new CacheHandle() {
         @Override
         public boolean hasResult() {
@@ -297,7 +295,11 @@ private void report(Event evt) {
     }
   }
 
-  private static boolean isRemoteCache(RemoteOptions options) {
+  private static boolean useRemoteCache(RemoteOptions options) {
     return !isNullOrEmpty(options.remoteCache);
   }
+
+  private static boolean useDiskCache(RemoteOptions options) {
+    return options.diskCache != null && !options.diskCache.isEmpty();
+  }
 }
diff --git a/src/main/java/com/google/devtools/build/lib/remote/disk/BUILD b/src/main/java/com/google/devtools/build/lib/remote/disk/BUILD
index 434efb75402db1..97a78cfa7ad1d5 100644
--- a/src/main/java/com/google/devtools/build/lib/remote/disk/BUILD
+++ b/src/main/java/com/google/devtools/build/lib/remote/disk/BUILD
@@ -14,6 +14,7 @@ java_library(
     tags = ["bazel"],
     deps = [
         "//src/main/java/com/google/devtools/build/lib/remote/common",
+        "//src/main/java/com/google/devtools/build/lib/remote/options",
         "//src/main/java/com/google/devtools/build/lib/remote/util",
         "//src/main/java/com/google/devtools/build/lib/vfs",
         "//src/main/java/com/google/devtools/common/options",
diff --git a/src/main/java/com/google/devtools/build/lib/remote/disk/DiskAndRemoteCacheClient.java b/src/main/java/com/google/devtools/build/lib/remote/disk/DiskAndRemoteCacheClient.java
index 8307f8de26af11..8380d165005d83 100644
--- a/src/main/java/com/google/devtools/build/lib/remote/disk/DiskAndRemoteCacheClient.java
+++ b/src/main/java/com/google/devtools/build/lib/remote/disk/DiskAndRemoteCacheClient.java
@@ -21,6 +21,7 @@
 import com.google.common.util.concurrent.ListenableFuture;
 import com.google.common.util.concurrent.MoreExecutors;
 import com.google.devtools.build.lib.remote.common.RemoteCacheClient;
+import com.google.devtools.build.lib.remote.options.RemoteOptions;
 import com.google.devtools.build.lib.vfs.Path;
 import com.google.protobuf.ByteString;
 import java.io.IOException;
@@ -37,17 +38,21 @@ public final class DiskAndRemoteCacheClient implements RemoteCacheClient {
 
   private final RemoteCacheClient remoteCache;
   private final DiskCacheClient diskCache;
+  private final RemoteOptions options;
 
-  public DiskAndRemoteCacheClient(DiskCacheClient diskCache, RemoteCacheClient remoteCache) {
+  public DiskAndRemoteCacheClient(DiskCacheClient diskCache, RemoteCacheClient remoteCache, RemoteOptions options) {
     this.diskCache = Preconditions.checkNotNull(diskCache);
     this.remoteCache = Preconditions.checkNotNull(remoteCache);
+    this.options = options;
   }
 
   @Override
   public void uploadActionResult(ActionKey actionKey, ActionResult actionResult)
       throws IOException, InterruptedException {
     diskCache.uploadActionResult(actionKey, actionResult);
-    remoteCache.uploadActionResult(actionKey, actionResult);
+    if (!options.incompatibleRemoteResultsIgnoreDisk || options.remoteUploadLocalResults) {
+      remoteCache.uploadActionResult(actionKey, actionResult);
+    }
   }
 
   @Override
@@ -60,7 +65,9 @@ public void close() {
   public ListenableFuture<Void> uploadFile(Digest digest, Path file) {
     try {
       diskCache.uploadFile(digest, file).get();
-      remoteCache.uploadFile(digest, file).get();
+      if (!options.incompatibleRemoteResultsIgnoreDisk || options.remoteUploadLocalResults) {
+        remoteCache.uploadFile(digest, file).get();
+      }
     } catch (ExecutionException e) {
       return Futures.immediateFailedFuture(e.getCause());
     } catch (InterruptedException e) {
@@ -73,7 +80,9 @@ public ListenableFuture<Void> uploadFile(Digest digest, Path file) {
   public ListenableFuture<Void> uploadBlob(Digest digest, ByteString data) {
     try {
       diskCache.uploadBlob(digest, data).get();
-      remoteCache.uploadBlob(digest, data).get();
+      if (!options.incompatibleRemoteResultsIgnoreDisk || options.remoteUploadLocalResults) {
+        remoteCache.uploadBlob(digest, data).get();
+      }
     } catch (ExecutionException e) {
       return Futures.immediateFailedFuture(e.getCause());
     } catch (InterruptedException e) {
@@ -130,22 +139,26 @@ public ListenableFuture<Void> downloadBlob(Digest digest, OutputStream out) {
       return Futures.immediateFailedFuture(e);
     }
 
-    ListenableFuture<Void> download =
-        closeStreamOnError(remoteCache.downloadBlob(digest, tempOut), tempOut);
-    ListenableFuture<Void> saveToDiskAndTarget =
-        Futures.transformAsync(
-            download,
-            (unused) -> {
-              try {
-                tempOut.close();
-                diskCache.captureFile(tempPath, digest, /* isActionCache= */ false);
-              } catch (IOException e) {
-                return Futures.immediateFailedFuture(e);
-              }
-              return diskCache.downloadBlob(digest, out);
-            },
-            MoreExecutors.directExecutor());
-    return saveToDiskAndTarget;
+    if (!options.incompatibleRemoteResultsIgnoreDisk || options.remoteAcceptCached) {
+      ListenableFuture<Void> download =
+          closeStreamOnError(remoteCache.downloadBlob(digest, tempOut), tempOut);
+      ListenableFuture<Void> saveToDiskAndTarget =
+          Futures.transformAsync(
+              download,
+              (unused) -> {
+                try {
+                  tempOut.close();
+                  diskCache.captureFile(tempPath, digest, /* isActionCache= */ false);
+                } catch (IOException e) {
+                  return Futures.immediateFailedFuture(e);
+                }
+                return diskCache.downloadBlob(digest, out);
+              },
+              MoreExecutors.directExecutor());
+      return saveToDiskAndTarget;
+    } else {
+      return Futures.immediateFuture(null);
+    }
   }
 
   @Override
@@ -154,16 +167,20 @@ public ListenableFuture<ActionResult> downloadActionResult(ActionKey actionKey)
       return diskCache.downloadActionResult(actionKey);
     }
 
-    return Futures.transformAsync(
-        remoteCache.downloadActionResult(actionKey),
-        (actionResult) -> {
-          if (actionResult == null) {
-            return Futures.immediateFuture(null);
-          } else {
-            diskCache.uploadActionResult(actionKey, actionResult);
-            return Futures.immediateFuture(actionResult);
-          }
-        },
-        MoreExecutors.directExecutor());
+    if (!options.incompatibleRemoteResultsIgnoreDisk || options.remoteAcceptCached) {
+      return Futures.transformAsync(
+          remoteCache.downloadActionResult(actionKey),
+          (actionResult) -> {
+            if (actionResult == null) {
+              return Futures.immediateFuture(null);
+            } else {
+              diskCache.uploadActionResult(actionKey, actionResult);
+              return Futures.immediateFuture(actionResult);
+            }
+          },
+          MoreExecutors.directExecutor());
+    } else {
+      return Futures.immediateFuture(null);
+    }
   }
 }
diff --git a/src/main/java/com/google/devtools/build/lib/remote/options/RemoteOptions.java b/src/main/java/com/google/devtools/build/lib/remote/options/RemoteOptions.java
index 88571221363f3a..27c9fc925b91d9 100644
--- a/src/main/java/com/google/devtools/build/lib/remote/options/RemoteOptions.java
+++ b/src/main/java/com/google/devtools/build/lib/remote/options/RemoteOptions.java
@@ -143,6 +143,25 @@ public final class RemoteOptions extends OptionsBase {
       help = "Whether to upload locally executed action results to the remote cache.")
   public boolean remoteUploadLocalResults;
 
+  @Option(
+      name = "incompatible_remote_results_ignore_disk",
+      defaultValue = "false",
+      category = "remote",
+      documentationCategory = OptionDocumentationCategory.REMOTE,
+      effectTags = {OptionEffectTag.UNKNOWN},
+      metadataTags = {
+        OptionMetadataTag.INCOMPATIBLE_CHANGE,
+        OptionMetadataTag.TRIGGERED_BY_ALL_INCOMPATIBLE_CHANGES
+      },
+      help =
+          "If set to true, --noremote_upload_local_results and --noremote_accept_cached "
+              + "will not apply to the disk cache. "
+              + "If a combined cache is used:"
+              + "\n\t--noremote_upload_local_results will cause results to be written to the disk cache, but not uploaded to the remote cache."
+              + "\n\t--noremote_accept_cached will result in Bazel checking for results in the disk cache, but not in the remote cache."
+              + "\nSee #8216 for details.")
+  public boolean incompatibleRemoteResultsIgnoreDisk;
+
   @Option(
       name = "remote_instance_name",
       defaultValue = "",


Test patch:
diff --git a/src/test/shell/bazel/remote/remote_execution_http_test.sh b/src/test/shell/bazel/remote/remote_execution_http_test.sh
index 2925a1a12a55d2..0bfafd14fc9644 100755
--- a/src/test/shell/bazel/remote/remote_execution_http_test.sh
+++ b/src/test/shell/bazel/remote/remote_execution_http_test.sh
@@ -364,6 +364,10 @@ EOF
 function test_genrule_combined_disk_http_cache() {
   # Test for the combined disk and http cache.
   # Built items should be pushed to both the disk and http cache.
+  # If --noremote_upload_local_results flag is set,
+  # built items should only be pushed to the disk cache.
+  # If --noremote_accept_cached flag is set,
+  # built items should only be checked from the disk cache.
   # If an item is missing on disk cache, but present on http cache,
   # then bazel should copy it from http cache to disk cache on fetch.
 
@@ -383,11 +387,67 @@ EOF
   rm -rf $cache
   mkdir $cache
 
-  # Build and push to disk and http cache
-  bazel build $disk_flags $http_flags //a:test \
+  # Build and push to disk cache but not http cache
+  bazel build $disk_flags $http_flags --incompatible_remote_results_ignore_disk=true --noremote_upload_local_results //a:test \
     || fail "Failed to build //a:test with combined disk http cache"
   cp -f bazel-genfiles/a/test.txt ${TEST_TMPDIR}/test_expected
 
+  # Fetch from disk cache
+  bazel clean
+  bazel build $disk_flags //a:test --incompatible_remote_results_ignore_disk=true --noremote_upload_local_results &> $TEST_log \
+    || fail "Failed to fetch //a:test from disk cache"
+  expect_log "1 remote cache hit" "Fetch from disk cache failed"
+  diff bazel-genfiles/a/test.txt ${TEST_TMPDIR}/test_expected \
+    || fail "Disk cache generated different result"
+
+  # No cache result from http cache, rebuild target
+  bazel clean
+  bazel build $http_flags //a:test --incompatible_remote_results_ignore_disk=true --noremote_upload_local_results &> $TEST_log \
+    || fail "Failed to build //a:test"
+  expect_not_log "1 remote cache hit" "Should not get cache hit from http cache"
+  expect_log "1 linux-sandbox" "Rebuild target failed"
+  diff bazel-genfiles/a/test.txt ${TEST_TMPDIR}/test_expected \
+    || fail "Rebuilt target generated different result"
+
+  rm -rf $cache
+  mkdir $cache
+
+  # No cache result from http cache, rebuild target, and upload result to http cache
+  bazel clean
+  bazel build $http_flags //a:test --incompatible_remote_results_ignore_disk=true --noremote_accept_cached &> $TEST_log \
+    || fail "Failed to build //a:test"
+  expect_not_log "1 remote cache hit" "Should not get cache hit from http cache"
+  expect_log "1 linux-sandbox" "Rebuild target failed"
+  diff bazel-genfiles/a/test.txt ${TEST_TMPDIR}/test_expected \
+    || fail "Rebuilt target generated different result"
+
+  # No cache result from http cache, rebuild target, and upload result to disk cache
+  bazel clean
+  bazel build $disk_flags $http_flags //a:test --incompatible_remote_results_ignore_disk=true --noremote_accept_cached &> $TEST_log \
+    || fail "Failed to build //a:test"
+  expect_not_log "1 remote cache hit" "Should not get cache hit from http cache"
+  expect_log "1 linux-sandbox" "Rebuild target failed"
+  diff bazel-genfiles/a/test.txt ${TEST_TMPDIR}/test_expected \
+    || fail "Rebuilt target generated different result"
+
+  # Fetch from disk cache
+  bazel clean
+  bazel build $disk_flags $http_flags //a:test --incompatible_remote_results_ignore_disk=true --noremote_accept_cached &> $TEST_log \
+    || fail "Failed to build //a:test"
+  expect_log "1 remote cache hit" "Fetch from disk cache failed"
+  diff bazel-genfiles/a/test.txt ${TEST_TMPDIR}/test_expected \
+    || fail "Disk cache generated different result"
+
+  rm -rf $cache
+  mkdir $cache
+
+  # Build and push to disk cache and http cache
+  bazel clean
+  bazel build $disk_flags $http_flags //a:test \
+    || fail "Failed to build //a:test with combined disk http cache"
+  diff bazel-genfiles/a/test.txt ${TEST_TMPDIR}/test_expected \
+    || fail "Built target generated different result"
+
   # Fetch from disk cache
   bazel clean
   bazel build $disk_flags //a:test &> $TEST_log \
diff --git a/src/test/shell/bazel/remote/remote_execution_test.sh b/src/test/shell/bazel/remote/remote_execution_test.sh
index 1ac08c6c6a18e3..90ac4d623182cb 100755
--- a/src/test/shell/bazel/remote/remote_execution_test.sh
+++ b/src/test/shell/bazel/remote/remote_execution_test.sh
@@ -1514,6 +1514,10 @@ EOF
 function test_genrule_combined_disk_grpc_cache() {
   # Test for the combined disk and grpc cache.
   # Built items should be pushed to both the disk and grpc cache.
+  # If --noremote_upload_local_results flag is set,
+  # built items should only be pushed to the disk cache.
+  # If --noremote_accept_cached flag is set,
+  # built items should only be checked from the disk cache.
   # If an item is missing on disk cache, but present on grpc cache,
   # then bazel should copy it from grpc cache to disk cache on fetch.
 
@@ -1533,11 +1537,67 @@ EOF
   rm -rf $cache
   mkdir $cache
 
-  # Build and push to disk and grpc cache
-  bazel build $disk_flags $grpc_flags //a:test \
+  # Build and push to disk cache but not grpc cache
+  bazel build $disk_flags $grpc_flags --incompatible_remote_results_ignore_disk=true --noremote_upload_local_results //a:test \
     || fail "Failed to build //a:test with combined disk grpc cache"
   cp -f bazel-genfiles/a/test.txt ${TEST_TMPDIR}/test_expected
 
+  # Fetch from disk cache
+  bazel clean
+  bazel build $disk_flags //a:test --incompatible_remote_results_ignore_disk=true --noremote_upload_local_results &> $TEST_log \
+    || fail "Failed to fetch //a:test from disk cache"
+  expect_log "1 remote cache hit" "Fetch from disk cache failed"
+  diff bazel-genfiles/a/test.txt ${TEST_TMPDIR}/test_expected \
+    || fail "Disk cache generated different result"
+
+  # No cache result from grpc cache, rebuild target
+  bazel clean
+  bazel build $grpc_flags //a:test --incompatible_remote_results_ignore_disk=true --noremote_upload_local_results &> $TEST_log \
+    || fail "Failed to build //a:test"
+  expect_not_log "1 remote cache hit" "Should not get cache hit from grpc cache"
+  expect_log "1 linux-sandbox" "Rebuild target failed"
+  diff bazel-genfiles/a/test.txt ${TEST_TMPDIR}/test_expected \
+    || fail "Rebuilt target generated different result"
+
+  rm -rf $cache
+  mkdir $cache
+
+  # No cache result from grpc cache, rebuild target, and upload result to grpc cache
+  bazel clean
+  bazel build $grpc_flags //a:test --incompatible_remote_results_ignore_disk=true --noremote_accept_cached &> $TEST_log \
+    || fail "Failed to build //a:test"
+  expect_not_log "1 remote cache hit" "Should not get cache hit from grpc cache"
+  expect_log "1 linux-sandbox" "Rebuild target failed"
+  diff bazel-genfiles/a/test.txt ${TEST_TMPDIR}/test_expected \
+    || fail "Rebuilt target generated different result"
+
+  # No cache result from grpc cache, rebuild target, and upload result to disk cache
+  bazel clean
+  bazel build $disk_flags $grpc_flags //a:test --incompatible_remote_results_ignore_disk=true --noremote_accept_cached &> $TEST_log \
+    || fail "Failed to build //a:test"
+  expect_not_log "1 remote cache hit" "Should not get cache hit from grpc cache"
+  expect_log "1 linux-sandbox" "Rebuild target failed"
+  diff bazel-genfiles/a/test.txt ${TEST_TMPDIR}/test_expected \
+    || fail "Rebuilt target generated different result"
+
+  # Fetch from disk cache
+  bazel clean
+  bazel build $disk_flags $grpc_flags //a:test --incompatible_remote_results_ignore_disk=true --noremote_accept_cached &> $TEST_log \
+    || fail "Failed to build //a:test"
+  expect_log "1 remote cache hit" "Fetch from disk cache failed"
+  diff bazel-genfiles/a/test.txt ${TEST_TMPDIR}/test_expected \
+    || fail "Disk cache generated different result"
+
+  rm -rf $cache
+  mkdir $cache
+
+  # Build and push to disk cache and grpc cache
+  bazel clean
+  bazel build $disk_flags $grpc_flags //a:test \
+    || fail "Failed to build //a:test with combined disk grpc cache"
+  diff bazel-genfiles/a/test.txt ${TEST_TMPDIR}/test_expected \
+    || fail "Built target generated different result"
+
   # Fetch from disk cache
   bazel clean
   bazel build $disk_flags //a:test &> $TEST_log \