Repository: iterative/dvc

Issue:
`dvc repro --pull`: invalidates cache
# Bug Report

## `repro`: `dvc repro --pull` invalidates cache. doing `dvc pull` and then `dvc repro` works.
<!--
## Issue name

Issue names must follow the pattern `command: description` where the command is the dvc command that you are trying to run. The description should describe the consequence of the bug.

Example: `repro: doesn't detect input changes`
-->

## Description

When I use `dvc repro --pull` on my dvc pipelines, it invalidates cache for certain stages and runs them again.
However, if we do `dvc pull` and then do `dvc repro` it respects the cache and skips the stages. This is the behaviour I expect for `dvc repro --pull` as well.

Also for some reason the hash key of the stage outputs keeps changing for every run as well.

### Reproduce

<!--
Step list of how to reproduce the bug
-->

<!--
Example:

1. dvc init
2. Copy dataset.zip to the directory
3. dvc add dataset.zip
4. dvc run -d dataset.zip -o model ./train.sh
5. modify dataset.zip
6. dvc repro
-->

1. Delete all dvc files in directory.
2. `dvc repro --pull`

### Expected

<!--
A clear and concise description of what you expect to happen.
-->

`dvc repro --pull` must behave the same way as if we do `dvc pull` and then `dvc repro`.

### Environment information

<!--
This is required to ensure that we can reproduce the bug.
-->

**Output of `dvc doctor`:**

```console
$ dvc doctor

DVC version: 3.49.0 (pip)
-------------------------
Platform: Python 3.11.3 on Linux-6.5.0-1017-azure-x86_64-with-glibc2.35
Subprojects:
        dvc_data = 3.15.1
        dvc_objects = 5.1.0
        dvc_render = 1.0.2
        dvc_task = 0.4.0
        scmrepo = 3.3.1
Supports:
        azure (adlfs = 2024.4.0, knack = 0.11.0, azure-identity = 1.12.0),
        http (aiohttp = 3.9.4, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.9.4, aiohttp-retry = 2.8.3)
Config:
        Global: /home/user/.config/dvc
        System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/mapper/data_vg-lv_data
Caches: local
Remotes: azure
Workspace directory: ext4 on /dev/mapper/data_vg-lv_data
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/cc226011e8332423634ca4bb8822ef4b
```

**Additional Information (if any):**

<!--
Please check https://github.com/iterative/dvc/wiki/Debugging-DVC on ways to gather more information regarding the issue.

If applicable, please also provide a `--verbose` output of the command, eg: `dvc add --verbose`.
If the issue is regarding the performance, please attach the profiling information and the benchmark comparisons.
-->

`dvc repro --pull data/dvc.yaml -v`: [repro_pull_verbose.log](https://github.com/iterative/dvc/files/15192664/repro_pull_verbose.log)



Discussion:
Are you able to provide a reproducible example? I'm not able to reproduce with https://github.com/iterative/example-get-started:

```bash
$ dvc repro --pull
WARNING: Failed to pull run cache: run-cache is not supported for http filesystem: https://remote.dvc.org/get-started
Importing 'get-started/data.xml (https://github.com/iterative/dataset-registry)' -> 'data/data.xml'

Stage 'prepare' is cached - skipping run, checking out outputs

Stage 'featurize' is cached - skipping run, checking out outputs

Stage 'train' is cached - skipping run, checking out outputs

Stage 'evaluate' is cached - skipping run, checking out outputs
Use `dvc push` to send your updates to remote storage.
```
I forgot to upload the `dvc.yaml` file used

```yaml
vars:
  - BLOB_URL: "<URL>"
  - data-version.toml
stages:
  download_full_parquet:
    desc: Download the exported parquet files
    params:
      - data/data-version.toml:
          - DATA_DIR
    cmd: >2
      <script to download from $BLOB_URL>
    wdir: ..
    outs:
      - data/full_output:
          persist: true
          cache: true
  download_dbg_parquet:
    desc: Download the exported dbg parquet files
    params:
      - data/data-version.toml:
          - DEBUG_DATA_DIR
    cmd: >2
      <script to download from $BLOB_URL>
    wdir: ..
    outs:
      - data/dbg_output:
          persist: true
          cache: true
  generate_full_json:
    desc: Generate the json intermediaries for the full dataset.
    cmd: python <script to generate json data> data/full_output data/full_json
    wdir: ../
    deps:
      - data/full_output
      - data/cached
      - .hashes/code.hash
      - poetry.lock
    outs:
      - data/full_json:
          cache: true
          persist: false
  generate_dbg_json:
    desc: Generate the json intermediaries for the debug dataset.
    cmd: python <script to generate json data> data/dbg_output data/dbg_json
    wdir: ../
    deps:
      - data/dbg_output
      - data/cached
      - .hashes/code.hash
      - poetry.lock
    outs:
      - data/dbg_json:
          cache: true
          persist: false
  generate_full_hdf5:
    desc: Transform the raw json files into hdf5 files the model can understand.
    cmd: python <script to transform json to hdf5> data/full_json data/full_wholesalers
    wdir: ../
    deps:
      - poetry.lock
      - .hashes/code.hash
      - data/full_json
      - data/fred/state_economics.json
      - data/fred/fred_signals_included.json
      - data/fred/national_economics.json
      - data/gas_price/output/state_gas_price.json
      - data/industry_trend/processed_trend.csv
      - data/state_region_mapping/state_region_mapping.json
    outs:
      - data/full_wholesalers:
          cache: true
          persist: true
  generate_dbg_hdf5:
    desc: Transform the raw debug json files into hdf5 files the model can understand.
    cmd: python <script to transform json to hdf5> data/dbg_json data/dbg_wholesalers
    wdir: ../
    deps:
      - poetry.lock
      - .hashes/code.hash
      - data/dbg_json
      - data/fred/state_economics.json
      - data/fred/fred_signals_included.json
      - data/fred/national_economics.json
      - data/gas_price/output/state_gas_price.json
      - data/industry_trend/processed_trend.csv
      - data/state_region_mapping/state_region_mapping.json
    outs:
      - data/dbg_wholesalers:
          cache: true
          persist: true

```
I think the issue is perhaps due to persist=true?

I assume on pull, persisted output stages are downloaded. I have seen that stages which are set to persist=true requires that we pull the data first. if the output path content has changed and persist was set to true, dvc doesn't skip the stage. like in the case when I forget to checkout dvc while switching branches.

With `--pull` does it pull the stages first especially for persisted stages?
Ah, you figured it out. `persist=True` is the issue. When `persist=True`, it creates a circular dependency, where the output depends not only on the listed dependencies, but also on the existing output. DVC has no way of tracking the contents of the previous iteration of that output, so it does not save that stage to the run-cache. If anything, I think the bug may be that DVC ever skips a stage with `persist=True`.
so is this by design? I assume that for `dvc repro --pull` we need to pull the stages that the workflow depends on? if persist is True, it should checkout the cache that is currently in `dvc.lock`. This issue doesn't pop up if I did a normal `dvc pull` and then `dvc repro`.
> so is this by design?

I think so, although I agree it's confusing. There are two ways that dvc can skip a stage:
1. If the workspace hasn't changed from what's in `dvc.lock`.
2. If the stage can be restored from the run-cache (it has been run before and the data is in the cache).

In `dvc repro --pull`, 1 fails for _all_ your stages because there is nothing in your workspace. This is the difference from doing `dvc pull` first. At the time dvc checks whether the stage has changed, there is nothing in the workspace yet, so it looks like the data has been deleted.

However, 2 succeeds for all your stages except for the persist one since that one was never saved to the run-cache for the reasons mentioned above.

I think what you want is `dvc repro --pull --allow-missing`. That will make 1 succeed when the data is missing. Maybe it makes sense for `--pull` to imply `--allow-missing`.
isn't `--allow-missing` for ignoring data doesn't exist in remote cache error?

When it comes to the 2nd case, if the workspace is empty then pull the data for the given hash key. If the workspace has different data in the workspace, either rerun the stage or give an error saying the data is different and we could provide a cmd flag or prompt input to force pull the persisted stages. Similar to dvc pull when we have data which isn't tracked by dvc (like the times when we kill the process in the middle you have dvc changes that aren't tracked).
> isn't `--allow-missing` for ignoring data doesn't exist in remote cache error?

Unfortunately, it is used to mean different things under different commands today. This is not the meaning for `repro`.

> if the workspace is empty then pull the data for the given hash key

This is what `--allow-missing` does except that it skips the stage entirely and doesn't waste time pulling the data. 

Resolving patch:
diff --git a/dvc/commands/repro.py b/dvc/commands/repro.py
--- a/dvc/commands/repro.py
+++ b/dvc/commands/repro.py
@@ -117,10 +117,7 @@ def add_arguments(repro_parser):
         "--pull",
         action="store_true",
         default=False,
-        help=(
-            "Try automatically pulling missing cache for outputs restored "
-            "from the run-cache."
-        ),
+        help="Try automatically pulling missing data.",
     )
     repro_parser.add_argument(
         "--allow-missing",
diff --git a/dvc/stage/__init__.py b/dvc/stage/__init__.py
--- a/dvc/stage/__init__.py
+++ b/dvc/stage/__init__.py
@@ -339,7 +339,7 @@ def changed_outs(self, allow_missing: bool = False) -> bool:
         for out in self.outs:
             status = out.status()
             if status:
-                if allow_missing and status[str(out)] == "not in cache":
+                if allow_missing and status[str(out)] in ["not in cache", "deleted"]:
                     continue
                 logger.debug(
                     "Output '%s' of %s changed because it is '%s'.",
@@ -418,17 +418,28 @@ def transfer(
 
     @rwlocked(read=["deps"], write=["outs"])
     def reproduce(self, interactive=False, **kwargs) -> Optional["Stage"]:
-        if not (
-            kwargs.get("force", False)
-            or self.changed(
-                kwargs.get("allow_missing", False), kwargs.pop("upstream", None)
-            )
-        ):
+        force = kwargs.get("force", False)
+        allow_missing = kwargs.get("allow_missing", False)
+        pull = kwargs.get("pull", False)
+        upstream = kwargs.pop("upstream", None)
+        if force:
+            pass
+        # Skip stages with missing data if otherwise unchanged
+        elif not self.changed(allow_missing, upstream):
             if not isinstance(self, PipelineStage) and self.is_data_source:
                 logger.info("'%s' didn't change, skipping", self.addressing)
             else:
                 logger.info("Stage '%s' didn't change, skipping", self.addressing)
             return None
+        # Pull stages with missing data if otherwise unchanged
+        elif not self.changed(True, upstream) and pull:
+            try:
+                logger.info("Pulling data for %s", self)
+                self.repo.pull(self.addressing, jobs=kwargs.get("jobs", None))
+                self.checkout()
+                return None
+            except CheckoutError:
+                logger.info("Unable to pull data for %s", self)
 
         msg = f"Going to reproduce {self}. Are you sure you want to continue?"
         if interactive and not prompt.confirm(msg):
@@ -597,16 +608,10 @@ def run(
         if (self.cmd or self.is_import) and not self.frozen and not dry:
             self.remove_outs(ignore_remove=False, force=False)
 
-        if (
-            self.is_import and (not self.frozen or kwargs.get("pull"))
-        ) or self.is_partial_import:
+        if (self.is_import and not self.frozen) or self.is_partial_import:
             self._sync_import(dry, force, kwargs.get("jobs", None), no_download)
         elif not self.frozen and self.cmd:
             self._run_stage(dry, force, **kwargs)
-        elif kwargs.get("pull"):
-            logger.info("Pulling data for %s", self)
-            self.repo.pull(self.addressing, jobs=kwargs.get("jobs", None))
-            self.checkout()
         elif not dry:
             args = ("outputs", "frozen ") if self.frozen else ("data sources", "")
             logger.info("Verifying %s in %s%s", *args, self)


Test patch:
diff --git a/tests/func/repro/test_repro_allow_missing.py b/tests/func/repro/test_repro_allow_missing.py
--- a/tests/func/repro/test_repro_allow_missing.py
+++ b/tests/func/repro/test_repro_allow_missing.py
@@ -55,3 +55,16 @@ def test_repro_allow_missing_upstream_stage_modified(
     ret = dvc.reproduce(pull=True, allow_missing=True)
     # create-foo is skipped ; copy-foo pulls modified dep
     assert len(ret) == 1
+
+
+def test_repro_allow_missing_cached(tmp_dir, dvc):
+    tmp_dir.gen("fixed", "fixed")
+    dvc.stage.add(name="create-foo", cmd="echo foo > foo", deps=["fixed"], outs=["foo"])
+    dvc.stage.add(name="copy-foo", cmd="cp foo bar", deps=["foo"], outs=["bar"])
+    dvc.reproduce()
+
+    remove("foo")
+
+    ret = dvc.reproduce(allow_missing=True)
+    # both stages are skipped
+    assert not ret
diff --git a/tests/func/repro/test_repro_pull.py b/tests/func/repro/test_repro_pull.py
--- a/tests/func/repro/test_repro_pull.py
+++ b/tests/func/repro/test_repro_pull.py
@@ -1,5 +1,7 @@
 import os
 
+import pytest
+
 from dvc.stage.cache import RunCacheNotSupported
 from dvc.utils.fs import remove
 
@@ -49,11 +51,14 @@ def test_repro_skip_pull_if_no_run_cache_is_passed(tmp_dir, dvc, mocker, local_r
     (foo,) = tmp_dir.dvc_gen("foo", "foo")
 
     dvc.push()
-    spy_pull = mocker.spy(dvc.stage_cache, "pull")
     dvc.stage.add(name="copy-foo", cmd="cp foo bar", deps=["foo"], outs=["bar"])
+
+    assert dvc.reproduce(pull=True)
+
     remove("foo")
     remove(foo.outs[0].cache_path)
-
+    remove("dvc.lock")
+    spy_pull = mocker.spy(dvc.stage_cache, "pull")
     assert dvc.reproduce(pull=True, run_cache=False)
     assert not spy_pull.called
 
@@ -68,3 +73,46 @@ def test_repro_skip_pull_if_single_item_is_passed(tmp_dir, dvc, mocker, local_re
     remove(foo.outs[0].cache_path)
 
     assert dvc.reproduce(pull=True, single_item=True)
+
+
+def test_repro_pulls_persisted_output(tmp_dir, dvc, mocker, local_remote):
+    tmp_dir.dvc_gen("foo", "foo")
+
+    dvc.push()
+
+    dvc.stage.add(name="copy-foo", cmd="cp foo bar", deps=["foo"], outs_persist=["bar"])
+    dvc.reproduce()
+    remove("bar")
+
+    # stage is skipped
+    assert not dvc.reproduce(pull=True)
+
+
+@pytest.mark.parametrize("allow_missing", [True, False])
+def test_repro_pulls_allow_missing(tmp_dir, dvc, mocker, local_remote, allow_missing):
+    tmp_dir.dvc_gen("foo", "foo")
+
+    dvc.push()
+
+    dvc.stage.add(name="copy-foo", cmd="cp foo bar", deps=["foo"], outs=["bar"])
+    dvc.reproduce()
+    remove("foo")
+
+    # stage is skipped
+    assert not dvc.reproduce(pull=True, allow_missing=allow_missing)
+    # data only pulled if allow_missing is false
+    assert (tmp_dir / "foo").exists() != allow_missing
+
+
+def test_repro_pull_fails(tmp_dir, dvc, mocker, local_remote):
+    tmp_dir.dvc_gen("foo", "foo")
+
+    dvc.stage.add(
+        name="concat-foo", cmd="cat foo foo > bar", deps=["foo"], outs=["bar"]
+    )
+    stages = dvc.reproduce()
+    remove("bar")
+    remove(stages[0].outs[0].cache_path)
+
+    # stage is run
+    assert dvc.reproduce(pull=True)