Repository: iterative/dvc

Issue:
`exp run`: git ignored files committed to experiments
# Bug Report

<!--
## Issue name

Issue names must follow the pattern `command: description` where the command is the dvc command that you are trying to run. The description should describe the consequence of the bug. 

Example: `repro: doesn't detect input changes`
-->

## Description

<!--
A clear and concise description of what the bug is.
-->

I was working on a new command in the extension where you can "share an experiment". I got some surprising results in that the experiment's input data that is tracked by DVC and listed in .gitignore files was committed into Git. I discovered this when I sent the branch to the remote and opened a PR.

### Reproduce

<!--
Step list of how to reproduce the bug
-->

<!--
Example:

1. dvc init
2. Copy dataset.zip to the directory
3. dvc add dataset.zip
4. dvc run -d dataset.zip -o model ./train.sh
5. modify dataset.zip
6. dvc repro
-->

1. run several checkpoint experiments in the workspace.
2. pick one of the experiments.
3. dvc exp branch [exp-name] [branch-name]
4. dvc exp apply [exp-name]
5. dvc push
6. git push origin [branch-name]
7. create a PR for the branch pushed to origin. Tracked data will be shown in the PR. e.g https://github.com/iterative/vscode-dvc/pull/2156


### Expected

<!--
A clear and concise description of what you expect to happen.
-->

`exp branch` does not commit DVC tracked/git ignored data.

### Environment information

<!--
This is required to ensure that we can reproduce the bug.
-->

**Output of `dvc doctor`:**

```console
$ dvc doctor
DVC version: 2.16.0 (pip)
---------------------------------
Platform: Python 3.8.9 on macOS-12.5-arm64-arm-64bit
Supports:
        webhdfs (fsspec = 2022.5.0),
        http (aiohttp = 3.8.1, aiohttp-retry = 2.5.1),
        https (aiohttp = 3.8.1, aiohttp-retry = 2.5.1),
        s3 (s3fs = 2022.5.0, boto3 = 1.21.21)
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk3s1s1
Caches: local
Remotes: https
Workspace directory: apfs on /dev/disk3s1s1
Repo: dvc (subdir), git
```

**Additional Information (if any):**

<!--
Please check https://github.com/iterative/dvc/wiki/Debugging-DVC on ways to gather more information regarding the issue.

If applicable, please also provide a `--verbose` output of the command, eg: `dvc add --verbose`.
If the issue is regarding the performance, please attach the profiling information and the benchmark comparisons.
-->

Could definitely be related to the way a branch is created for checkpoint experiments. I have not tested to see if the same problem occurs with non-checkpoint experiments.



Comments:
To clarify, I don't think the issue here is with `exp branch`. The problem here is that DVC-tracked/git-ignored files are being included in the commits generated by `exp run`. `exp branch` does not generate git commits, it just creates a new git branch ref (`refs/heads/...` that points to the existing experiment commit)

This bug just became more apparent due to using `exp branch` and pushing to github (and the unexpected files showing up in the github PR UI)
This issue was also noted by a discord user, but it was unclear on how to reproduce it and not followed up into a separate bug report

https://discord.com/channels/485586884165107732/485596304961962003/1003959217826300025
Thanks @pmrowla
still bisecting, but it looks like this bug was introduced in 2.11.0

Using `git checkout <exp-commit>` from example-get-started runs:

2.10.2
```
example-get-started git:e5e08a2  py:example-get-started ❯ git ls-files
.dvc/.gitignore
.dvc/config
.dvc/plots/confusion.json
.dvc/plots/confusion_normalized.json
.dvc/plots/default.json
.dvc/plots/linear.json
.dvc/plots/scatter.json
.dvc/plots/smooth.json
.dvcignore
.github/workflows/cml.yaml
.gitignore
README.md
data/.gitignore
data/data.xml.dvc
dvc.lock
dvc.yaml
params.yaml
prc.json
roc.json
scores.json
src/evaluate.py
src/featurization.py
src/prepare.py
src/requirements.txt
src/train.py
```

2.11.0 (`data/data.xml`, `data/features/`, `data/prepared/`) have all been committed in git
```
example-get-started git:48306b2  py:example-get-started ❯ git ls-files
.dvc/.gitignore
.dvc/config
.dvc/plots/confusion.json
.dvc/plots/confusion_normalized.json
.dvc/plots/default.json
.dvc/plots/linear.json
.dvc/plots/scatter.json
.dvc/plots/smooth.json
.dvcignore
.github/workflows/cml.yaml
.gitignore
README.md
data/.gitignore
data/data.xml
data/data.xml.dvc
data/features/test.pkl
data/features/train.pkl
data/prepared/test.tsv
data/prepared/train.tsv
dvc.lock
dvc.yaml
model.pkl
params.yaml
prc.json
roc.json
scores.json
src/evaluate.py
src/featurization.py
src/prepare.py
src/requirements.txt
src/train.py
```
Bug was introduced with https://github.com/iterative/dvc/pull/7712

When we reproduce the exp pipeline, we ensure that all deps and outputs are tracked in either DVC or git. So the end result is that all non-DVC-tracked deps are staged in git.

The check to determine whether or not a dep is already tracked by DVC uses `dep.fs_path`, which is no longer valid after the repo root change. The bug happens because the fs path will not be identified as a DVC file, so we end up force staging all pipeline deps in git.

https://github.com/iterative/dvc/blob/c134d4ab8e9fb2a56ffefeabbc5b636fb91a7b51/dvc/repo/reproduce.py#L61-L69

After the root fs change, we need to checking the dep path relative to repo root and not the local filesystem path 

Patch:
diff --git a/dvc/repo/reproduce.py b/dvc/repo/reproduce.py
--- a/dvc/repo/reproduce.py
+++ b/dvc/repo/reproduce.py
@@ -64,7 +64,9 @@ def _get_stage_files(stage: "Stage") -> typing.Iterator[str]:
         if (
             not dep.use_scm_ignore
             and dep.is_in_repo
-            and not stage.repo.dvcfs.isdvc(dep.fs_path)
+            and not stage.repo.dvcfs.isdvc(
+                stage.repo.dvcfs.from_os_path(str(dep))
+            )
         ):
             yield dep.fs_path
     for out in stage.outs:


Test patch:
diff --git a/tests/unit/repo/test_reproduce.py b/tests/unit/repo/test_reproduce.py
--- a/tests/unit/repo/test_reproduce.py
+++ b/tests/unit/repo/test_reproduce.py
@@ -1,3 +1,8 @@
+import os
+
+from dvc.repo.reproduce import _get_stage_files
+
+
 def test_number_reproduces(tmp_dir, dvc, mocker):
     reproduce_stage_mock = mocker.patch(
         "dvc.repo.reproduce._reproduce_stage", returns=[]
@@ -22,3 +27,42 @@ def test_number_reproduces(tmp_dir, dvc, mocker):
     dvc.reproduce(all_pipelines=True)
 
     assert reproduce_stage_mock.call_count == 5
+
+
+def test_get_stage_files(tmp_dir, dvc):
+    tmp_dir.dvc_gen("dvc-dep", "dvc-dep")
+    tmp_dir.gen("other-dep", "other-dep")
+
+    stage = dvc.stage.add(
+        name="stage",
+        cmd="foo",
+        deps=["dvc-dep", "other-dep"],
+        outs=["dvc-out"],
+        outs_no_cache=["other-out"],
+    )
+    result = set(_get_stage_files(stage))
+    assert result == {
+        stage.dvcfile.relpath,
+        str(tmp_dir / "other-dep"),
+        str(tmp_dir / "other-out"),
+    }
+
+
+def test_get_stage_files_wdir(tmp_dir, dvc):
+    tmp_dir.gen({"dir": {"dvc-dep": "dvc-dep", "other-dep": "other-dep"}})
+    dvc.add(os.path.join("dir", "dvc-dep"))
+
+    stage = dvc.stage.add(
+        name="stage",
+        cmd="foo",
+        wdir="dir",
+        deps=["dvc-dep", "other-dep"],
+        outs=["dvc-out"],
+        outs_no_cache=["other-out"],
+    )
+    result = set(_get_stage_files(stage))
+    assert result == {
+        stage.dvcfile.relpath,
+        str(tmp_dir / "dir" / "other-dep"),
+        str(tmp_dir / "dir" / "other-out"),
+    }