Repository: iterative/dvc

Issue:
Latest release of DVC breaks object versioned configs
# Bug Report

<!--
## Issue name

Issue names must follow the pattern `command: description` where the command is the dvc command that you are trying to run. The description should describe the consequence of the bug.

Example: `repro: doesn't detect input changes`
-->

## Description

The latest release of DVC v3.51.0 breaks workflows where data is stored with `version_aware = true` on a remote bucket.
It seems to be due to this PR which is included in the latest release:
https://github.com/iterative/dvc/pull/10433
<!--
A clear and concise description of what the bug is.
-->
Here is a CI run from a week ago with version v3.50.3 of dvc, which succeeds: 
https://github.com/SkafteNicki/example_mlops/actions/runs/9112178019/job/25050892637

Here is a CI run from today with version v3.51.0 of dvc, which fails with error:
```
ERROR: failed to pull data from the cloud - config file error: 'fetch --run-cache' is unsupported for cloud versioned remotes: config file error: 'fetch --run-cache' is unsupported for cloud versioned remotes
```
https://github.com/SkafteNicki/example_mlops/actions/runs/9222033174/job/25372344887

Nothing has changed regarding the data, config of dvc etc. only the version being used by the CI.

By changing `dvc pull` to `dvc pull --no-run-cache` fixes the issue:
https://github.com/SkafteNicki/example_mlops/actions/runs/9222304205/job/25373194648

### Reproduce

<!--
Step list of how to reproduce the bug
-->

<!--
Example:

1. dvc init
2. Copy dataset.zip to the directory
3. dvc add dataset.zip
4. dvc run -d dataset.zip -o model ./train.sh
5. modify dataset.zip
6. dvc repro
-->

### Expected

<!--
A clear and concise description of what you expect to happen.
-->

I have already found the solution for this problem, however I would not have expected such a breaking change to happen in a minor release of DVC. I would recommend the maintainers to add to the documentation that `--no-run-cache` argument needs to be added when `version_aware=true` in dvc config (alternatively, this could maybe be auto-detected from the config and automatically set?)

On a sidenote: it seems all reference in the documentation to setting a remote storage to version aware is gone?
The relevant page for this information: https://dvc.org/doc/user-guide/data-management/cloud-versioning#cloud-versioning does not really contain how to do it:
```bash
dvc remote modify remote_storage version_aware true
```

### Environment information

<!--
This is required to ensure that we can reproduce the bug.
-->

**Output of `dvc doctor`:**

```console
$ dvc doctor
```

**Additional Information (if any):**

<!--
Please check https://github.com/iterative/dvc/wiki/Debugging-DVC on ways to gather more information regarding the issue.

If applicable, please also provide a `--verbose` output of the command, eg: `dvc add --verbose`.
If the issue is regarding the performance, please attach the profiling information and the benchmark comparisons.
-->


Discussion:
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/iterative/dvc/pull/10441?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iterative) Report
Attention: Patch coverage is `94.11765%` with `1 lines` in your changes are missing coverage. Please review.
> Project coverage is 90.31%. Comparing base [(`83ec952`)](https://app.codecov.io/gh/iterative/dvc/commit/83ec952da139853cdc8532ce6b7758d6e990e531?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iterative) to head [(`5e5c5c8`)](https://app.codecov.io/gh/iterative/dvc/commit/5e5c5c8583e2f411910c08289379c8fa5c7d06b1?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iterative).

| [Files](https://app.codecov.io/gh/iterative/dvc/pull/10441?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iterative) | Patch % | Lines |
|---|---|---|
| [dvc/repo/fetch.py](https://app.codecov.io/gh/iterative/dvc/pull/10441?src=pr&el=tree&filepath=dvc%2Frepo%2Ffetch.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iterative#diff-ZHZjL3JlcG8vZmV0Y2gucHk=) | 66.66% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/iterative/dvc/pull/10441?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iterative) |

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main   #10441      +/-   ##
==========================================
- Coverage   90.85%   90.31%   -0.54%     
==========================================
  Files         503      481      -22     
  Lines       39069    35981    -3088     
  Branches     5638     4764     -874     
==========================================
- Hits        35498    32498    -3000     
+ Misses       2920     2886      -34     
+ Partials      651      597      -54     
```



</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/iterative/dvc/pull/10441?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iterative).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iterative).


Patch:
diff --git a/dvc/repo/fetch.py b/dvc/repo/fetch.py
index 05f92921d..047a45f9c 100644
--- a/dvc/repo/fetch.py
+++ b/dvc/repo/fetch.py
@@ -2,6 +2,7 @@ from typing import TYPE_CHECKING
 
 from dvc.exceptions import DownloadError
 from dvc.log import logger
+from dvc.stage.cache import RunCacheNotSupported
 from dvc.ui import ui
 from dvc_data.index import DataIndex, FileStorage
 
@@ -136,6 +137,8 @@ def fetch(  # noqa: PLR0913
     try:
         if run_cache:
             self.stage_cache.pull(remote)
+    except RunCacheNotSupported as e:
+        logger.debug("failed to pull run cache: %s", e)
     except DownloadError as exc:
         failed_count += exc.amount
 
diff --git a/dvc/repo/push.py b/dvc/repo/push.py
index 202827c83..acf31573c 100644
--- a/dvc/repo/push.py
+++ b/dvc/repo/push.py
@@ -1,10 +1,14 @@
 from contextlib import suppress
 
 from dvc.exceptions import InvalidArgumentError, UploadError
+from dvc.log import logger
+from dvc.stage.cache import RunCacheNotSupported
 from dvc.ui import ui
 
 from . import locked
 
+logger = logger.getChild(__name__)
+
 
 def _rebuild(idx, path, fs, cb):
     from dvc_data.index import DataIndex, DataIndexEntry, Meta
@@ -95,8 +99,11 @@ def push(  # noqa: PLR0913
                 "Multiple rev push is unsupported for cloud versioned remotes"
             )
 
-    used_run_cache = self.stage_cache.push(remote) if run_cache else []
-    transferred_count += len(used_run_cache)
+    try:
+        used_run_cache = self.stage_cache.push(remote) if run_cache else []
+        transferred_count += len(used_run_cache)
+    except RunCacheNotSupported as e:
+        logger.debug("failed to push run cache: %s", e)
 
     if isinstance(targets, str):
         targets = [targets]


Test patch:
diff --git a/tests/func/test_run_cache.py b/tests/func/test_run_cache.py
index a8669e88b..495e64c83 100644
--- a/tests/func/test_run_cache.py
+++ b/tests/func/test_run_cache.py
@@ -1,3 +1,4 @@
+import logging
 import os
 
 import pytest
@@ -231,13 +232,15 @@ def test_restore_pull(tmp_dir, dvc, run_copy, mocker, local_remote):
     assert (tmp_dir / LOCK_FILE).exists()
 
 
-def test_push_pull_unsupported(tmp_dir, dvc, mocker, run_copy):
+def test_push_pull_unsupported(tmp_dir, dvc, mocker, run_copy, local_remote, caplog):
     tmp_dir.gen("foo", "foo")
     run_copy("foo", "bar", name="copy-foo-bar")
     mocker.patch.object(
         dvc.cloud, "get_remote_odb", side_effect=RunCacheNotSupported("foo")
     )
-    with pytest.raises(RunCacheNotSupported):
+    with caplog.at_level(logging.DEBUG):
         dvc.push(run_cache=True)
-    with pytest.raises(RunCacheNotSupported):
+        assert "failed to push run cache" in caplog.text
+    with caplog.at_level(logging.DEBUG):
         dvc.pull(run_cache=True)
+        assert "failed to pull run cache" in caplog.text