Repository: iterative/dvc

Issue:
Latest release of DVC breaks object versioned configs
# Bug Report

<!--
## Issue name

Issue names must follow the pattern `command: description` where the command is the dvc command that you are trying to run. The description should describe the consequence of the bug.

Example: `repro: doesn't detect input changes`
-->

## Description

The latest release of DVC v3.51.0 breaks workflows where data is stored with `version_aware = true` on a remote bucket.
It seems to be due to this PR which is included in the latest release:
https://github.com/iterative/dvc/pull/10433
<!--
A clear and concise description of what the bug is.
-->
Here is a CI run from a week ago with version v3.50.3 of dvc, which succeeds: 
https://github.com/SkafteNicki/example_mlops/actions/runs/9112178019/job/25050892637

Here is a CI run from today with version v3.51.0 of dvc, which fails with error:
```
ERROR: failed to pull data from the cloud - config file error: 'fetch --run-cache' is unsupported for cloud versioned remotes: config file error: 'fetch --run-cache' is unsupported for cloud versioned remotes
```
https://github.com/SkafteNicki/example_mlops/actions/runs/9222033174/job/25372344887

Nothing has changed regarding the data, config of dvc etc. only the version being used by the CI.

By changing `dvc pull` to `dvc pull --no-run-cache` fixes the issue:
https://github.com/SkafteNicki/example_mlops/actions/runs/9222304205/job/25373194648

### Reproduce

<!--
Step list of how to reproduce the bug
-->

<!--
Example:

1. dvc init
2. Copy dataset.zip to the directory
3. dvc add dataset.zip
4. dvc run -d dataset.zip -o model ./train.sh
5. modify dataset.zip
6. dvc repro
-->

### Expected

<!--
A clear and concise description of what you expect to happen.
-->

I have already found the solution for this problem, however I would not have expected such a breaking change to happen in a minor release of DVC. I would recommend the maintainers to add to the documentation that `--no-run-cache` argument needs to be added when `version_aware=true` in dvc config (alternatively, this could maybe be auto-detected from the config and automatically set?)

On a sidenote: it seems all reference in the documentation to setting a remote storage to version aware is gone?
The relevant page for this information: https://dvc.org/doc/user-guide/data-management/cloud-versioning#cloud-versioning does not really contain how to do it:
```bash
dvc remote modify remote_storage version_aware true
```

### Environment information

<!--
This is required to ensure that we can reproduce the bug.
-->

**Output of `dvc doctor`:**

```console
$ dvc doctor
```

**Additional Information (if any):**

<!--
Please check https://github.com/iterative/dvc/wiki/Debugging-DVC on ways to gather more information regarding the issue.

If applicable, please also provide a `--verbose` output of the command, eg: `dvc add --verbose`.
If the issue is regarding the performance, please attach the profiling information and the benchmark comparisons.
-->



Discussion:
Seems DVC builds are also broken here https://github.com/iterative/dvc-s3-repo 

Resolving patch:
diff --git a/dvc/repo/fetch.py b/dvc/repo/fetch.py
--- a/dvc/repo/fetch.py
+++ b/dvc/repo/fetch.py
@@ -2,6 +2,7 @@
 
 from dvc.exceptions import DownloadError
 from dvc.log import logger
+from dvc.stage.cache import RunCacheNotSupported
 from dvc.ui import ui
 from dvc_data.index import DataIndex, FileStorage
 
@@ -136,6 +137,8 @@ def fetch(  # noqa: PLR0913
     try:
         if run_cache:
             self.stage_cache.pull(remote)
+    except RunCacheNotSupported as e:
+        logger.debug("failed to pull run cache: %s", e)
     except DownloadError as exc:
         failed_count += exc.amount
 
diff --git a/dvc/repo/push.py b/dvc/repo/push.py
--- a/dvc/repo/push.py
+++ b/dvc/repo/push.py
@@ -1,10 +1,14 @@
 from contextlib import suppress
 
 from dvc.exceptions import InvalidArgumentError, UploadError
+from dvc.log import logger
+from dvc.stage.cache import RunCacheNotSupported
 from dvc.ui import ui
 
 from . import locked
 
+logger = logger.getChild(__name__)
+
 
 def _rebuild(idx, path, fs, cb):
     from dvc_data.index import DataIndex, DataIndexEntry, Meta
@@ -95,8 +99,11 @@ def push(  # noqa: PLR0913
                 "Multiple rev push is unsupported for cloud versioned remotes"
             )
 
-    used_run_cache = self.stage_cache.push(remote) if run_cache else []
-    transferred_count += len(used_run_cache)
+    try:
+        used_run_cache = self.stage_cache.push(remote) if run_cache else []
+        transferred_count += len(used_run_cache)
+    except RunCacheNotSupported as e:
+        logger.debug("failed to push run cache: %s", e)
 
     if isinstance(targets, str):
         targets = [targets]


Test patch:
diff --git a/tests/func/test_run_cache.py b/tests/func/test_run_cache.py
--- a/tests/func/test_run_cache.py
+++ b/tests/func/test_run_cache.py
@@ -1,3 +1,4 @@
+import logging
 import os
 
 import pytest
@@ -231,13 +232,15 @@ def test_restore_pull(tmp_dir, dvc, run_copy, mocker, local_remote):
     assert (tmp_dir / LOCK_FILE).exists()
 
 
-def test_push_pull_unsupported(tmp_dir, dvc, mocker, run_copy):
+def test_push_pull_unsupported(tmp_dir, dvc, mocker, run_copy, local_remote, caplog):
     tmp_dir.gen("foo", "foo")
     run_copy("foo", "bar", name="copy-foo-bar")
     mocker.patch.object(
         dvc.cloud, "get_remote_odb", side_effect=RunCacheNotSupported("foo")
     )
-    with pytest.raises(RunCacheNotSupported):
+    with caplog.at_level(logging.DEBUG):
         dvc.push(run_cache=True)
-    with pytest.raises(RunCacheNotSupported):
+        assert "failed to push run cache" in caplog.text
+    with caplog.at_level(logging.DEBUG):
         dvc.pull(run_cache=True)
+        assert "failed to pull run cache" in caplog.text