Repository: iterative/dvc

Issue:
dvc exp run unavailable in CML
Is there any way to run DVC exp run in CML?

My use case includes training several experiments in a Cloud or self hosted Server and publish in the PR the results of my work. 

For now when trying to do this using the setup action I get an unexpected error.

I understand that commit hides experiment results and I read that CML is not intended for experimentation but that is not quite clear in official documentation.


Discussion:
I think this is an upstream bug - https://github.com/iterative/dvc/issues/7531.

What error are you getting? Are you using `setup-cml`? In which case you should also use `setup-dvc`.

Or are you using the CML Docker images?
> commit hides experiment results and I read that CML is not intended for experimentation

Sorry I don't quite follow. How do commits hide results? CML is intended for provisioning & reporting experiments.
> > commit hides experiment results and I read that CML is not intended for experimentation
> 
> Sorry I don't quite follow. How do commits hide results? CML is intended for provisioning & reporting experiments.

When I say experiments I refer to using `dvc exp`. Once you commit your changes `dvc exp show` no longer shows experiments table.

In my specific case I wanted to try the following:

```yaml
name: experiments
on: [push]
jobs:
  train-model:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: iterative/setup-cml@v1
      - uses: actions/setup-python@v2
        with:
          python-version: '3.x'
      - name: Train model
        env:
          repo_token: ${{ secrets.GITHUB_TOKEN }}
        run: |
          pip3 install -r requirements.txt
          #bash exp_file.sh
          dvc exp run -S train.C=0.005
          dvc exp run -S train.C=100

          echo "## Resultados del Experimento" >> report.md
          dvc exp show --only-changed --drop 'assets|src' --no-pager --md >> report.md
          
          echo "## Parallel Plot\n" >> report.md
          dvc exp show --only-changed --drop 'assets|src' --pcp --sort-by test_recall >> report.md

          cml send-comment report.md
```

In this particular case I wanted to report the results of 2 experiments with `dvc exp run`. I've tried also using a bash file (that you can see commented out) with the following: 

```bash
dvc exp run --queue -S train.C=5
dvc exp run --queue -S train.C=30
dvc exp run --queue -S train.C=60
dvc exp run --queue -S train.C=120

dvc exp run --run-all
```

All of the experiments resulted in something like:

![image](https://user-images.githubusercontent.com/48638337/161554731-2dce5c82-c33a-46b6-a80a-8f153fc35d4e.png)

So my question would be: Is it allowed to run something like what I'm proposing?
Can I run my experiments and report it in the PR?

I can't remember when I read that Experimentation was meant to be done locally and only `dvc repro` on the CI Pipeline. Can you clarify that please?

Thanks in advance,

Alfonso
My brain has stopped working for the day already, but I don't think there isn't any reason you shouldn't be able to do this. Without having tested anything like this I would blindly expand your checkout step to see if that fixes it:
```yml
- uses: actions/checkout@v3
  with:
     fetch-depth: 0
```
https://github.com/actions/checkout#fetch-all-history-for-all-tags-and-branches
(Transferred to DVC because it looks like a config/setup issue unrelated to CML)
@dacbd's suggestion should be correct, github's checkout action only fetches a single shallow commit by default, this isn't enough of the Git repo for `exp run` to work properly.
@pmrowla Why does `exp run` require more than the baseline commit to be fetched?
> @pmrowla Why does `exp run` require more than the baseline commit to be fetched?

I understand you say `dvc exp run` depends on the rest of git history, probably because `dvc exp show` displays info of master metrics too. But if this is my first run of experiments `dvc exp run` should not depend on data of a different commit to work...

Anyways, I'll be testing this small change during the day and I'll get back to you... 
Thanks,

Alfonso
> @pmrowla Why does `exp run` require more than the baseline commit to be fetched?

The issue is that we use `git merge` in experiments and doing a `merge-base` operation in git requires fetch depth > 1 (the specific required depth depends on the commits being merged).

Basically, doing the merge-base requires finding a common ancestor for the commits being merged (that is not any of the merge commits themselves). For our purposes w/DVC experiments, this means that a fetch depth of 2 will usually be sufficient (so baseline commit + 1 parent), but that won't be the case 100% of the time. So if users are trying to use `exp run` inside github actions, the general solution is to use `fetch-depth: 0` (and not use a shallow clone at all).

@dberenbaum 
Thanks @pmrowla. I guess even if we tried to minimize operations where merge is needed, it will still be needed sometimes, and it's probably better to document the proposed workaround above. Opened https://github.com/iterative/dvc.org/issues/3416.
Ah right so the CML solution would be [`cml ci --unshallow`](https://cml.dev/doc/ref/ci) (a bit better than `fetch-depth: 0`)
If we go with the suggestion in https://github.com/iterative/dvc.org/issues/3416#issuecomment-1089866190, we should keep this open to update the message in DVC to be more informative and helpful. Even if DVC can't tell whether it's in a CI context, it could at least suggest to fetch missing commits and maybe link to docs that could go into more depth about specific use cases like cml.
On the UI side, we could detect the presence of `CI`/`GITHUB_ACTIONS`/`GITLAB_CI` environment variables. 
If we get the invalid commit exception we can just do the extra check to see whether or not we are in a shallow repo (you just have to check whether or not `.git/shallow` exists). This problem isn't specific to CI environments, it's just more likely to show up there since github uses shallow clones in CI by default.
Right, I guess the question is whether the message should be general like "don't use a shallow clone" or more CI-specific like "try fetch-depth: 0"? I think probably the former is better even if we can detect CI, since we don't want to have to cover the syntax of different git server behaviors, although we could link to docs where that level of detail could be provided.
+1 for a general message w/  link to docs for more specific details/solutions.
> Right, I guess the question is whether the message should be general like "don't use a shallow clone" or more CI-specific like "try fetch-depth: 0"? I think probably the former is better even if we can detect CI, since we don't want to have to cover the syntax of different git server behaviors, although we could link to docs where that level of detail could be provided.

I'd prefer general message with a CI specific hint and link to the docs wherever possible. I'd prefer less redirection as much as possible.

Example:
```
dvc exp run
ERROR: Cannot run in a shallow-cloned repository

If you are using `actions/checkout`, please set `fetch-depth: 0` so that the repo is not shallow-cloned.
See https://dvc.org/doc/user-guide/troubleshooting#shallow-clone.
```

Dear all, 
Thanks for all of your answers. I think the spirit of good open source is havomng everyone really involved, and I really appreciate your help on trying to solve this issue. 

I implemented the steps mentioned above and it worked partially. Once I run the experiments it turns out git is not recognized in the Github Machine:

![image](https://user-images.githubusercontent.com/48638337/162589259-e6fb8ac9-0006-4fa0-9a13-a482c102c93d.png)

In order to solve this issue I found a really nice action here: `fregante/setup-git-user@v1`. This helps the remote machine make the connection with git in order to make `dvc run exp` to work.

The final workflow will be like this:

```yaml
name: experiments
on: [push]
jobs:
  train-model:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with: 
          fetch-depth: 0
      - uses: fregante/setup-git-user@v1
      - uses: iterative/setup-cml@v1
      - uses: iterative/setup-dvc@v1
      - uses: actions/setup-python@v2
        with:
          python-version: '3.x'
      - name: Experiment
        env:
          repo_token: ${{ secrets.GITHUB_TOKEN }}
        run: |
          pip3 install -r requirements.txt
          #bash exp_file.sh
          dvc exp run -S train.C=0.005
          dvc exp run -S train.C=100

          echo "## Resultados del Experimento" >> report.md
          dvc exp show --only-changed --drop 'assets|src' --no-pager --md >> report.md

          cml send-comment report.md
```

I think this could be mentioned as a user-case in the documentation because is a very common experimentation workflow, and if we can externalize experimentation to a Github machine or a Cloud Runner would be very good.

On another note, I noticed `iterative/setup-dvc@v1` doesn't include yaml package which is a dependency on DVC. So you have to make sure to add it to the requirements.txt otherwise DVC will complain. Of course, unless you think it's a good idea to add it in the action.

Thanks and hope this helps DVC,

Alfonso
@datacubeR, thanks for the feedback; since you are already using `cml`, you can take a [look at running](https://cml.dev/doc/ref/ci) `cml ci`, which would solve the git user problem. 

>  and if we can externalize experimentation to a Github machine or a Cloud Runner would be very good.

This is part of what [`cml runner`](https://cml.dev/doc/ref/runner) command can help you with, running training workflows on cloud-hosted machines like ec2. If you decide to give it a try and run into any issues or have any other feedback be sure to let us know.
Bumping the priority of this since it is necessary for integration with cml.

> ```
> If you are using `actions/checkout`, please set `fetch-depth: 0` so that the repo is not shallow-cloned.
> ```

Is this sufficiently general? It's only applicable to GH actions AFAIK. Each git server has its own depth settings and variables, not to mention people who manually do shallow clones.

> Ah right so the CML solution would be [`cml ci`](https://cml.dev/doc/ref/ci) (a bit better than `fetch-depth: 0`)

Also, this is probably a better suggestion, but only if cml is installed.

So maybe we can generalize the message and provide specific examples like `fetch-depth: 0` in the docs. I agree with the general point of @skshetry, but I don't see a way to specifically address every possible scenario in the error message.
If you allow me to chime in, from the user perspective I think use cases are way clearer than just random notes.

For instance, I understand these fixes are only applicable to GitHub Actions, but in my opinion there is no clear documentation on CML CI which I think won't be normally installed since docker containers or virtual machines are normally the most common way to deal with this part. 



Should note that `cml ci` does not actually unshallow the repo (but will after https://github.com/iterative/cml/pull/957). The correct fix for running `dvc exp run` inside github actions is to set `fetch-depth: 0` in the checkout action.
Keeping this open on the DVC side until we address clarifying the shallow repo error in DVC UI and/or docs
Yup probably just adding "`did you forget to unshallow? https://error.dvc.org/unknown-object`" or similar to DVC's error message would be gewd.
`dvc exp run` succeeded in the workspace (but not in a temp dir) on a shallow clone until https://github.com/iterative/dvc/commit/fa819b0b70ba5f0f7ed7010280dbe840a813c1d3:

```
$ dvc -V
2.9.3.dev8+g37f8010b4
$ git clone --depth=1 git@github.com:iterative/example-get-started.git
Cloning into 'example-get-started'...
remote: Enumerating objects: 35, done.
remote: Counting objects: 100% (35/35), done.
remote: Compressing objects: 100% (25/25), done.
remote: Total 35 (delta 6), reused 24 (delta 4), pack-reused 0
Receiving objects: 100% (35/35), 35.62 KiB | 17.81 MiB/s, done.
Resolving deltas: 100% (6/6), done.
$ cd example-get-started
$ dvc pull
A       data/prepared/
A       data/data.xml
A       data/features/
A       model.pkl
4 files added and 6 files fetched
$ dvc exp run
'data/data.xml.dvc' didn't change, skipping
Stage 'prepare' didn't change, skipping
Stage 'featurize' didn't change, skipping
Stage 'train' didn't change, skipping
Stage 'evaluate' didn't change, skipping
$ dvc -V # Upgrade DVC version
2.9.3.dev9+gfa819b0b7
(dvc) dave@davids-air:/tmp/example-get-started 13:15:15
$ dvc exp run
ERROR: Merge failed
```

@pmrowla It makes sense as part of making the workspace and temp implementations more consistent, but is there any way to make at least workspace runs succeed in this scenario? It breaks a previously supported workflow for anyone using CML or other CI actions with `exp run`.

> The issue is that we use `git merge` in experiments and doing a `merge-base` operation in git requires fetch depth > 1 (the specific required depth depends on the commits being merged).
> 
> Basically, doing the merge-base requires finding a common ancestor for the commits being merged (that is not any of the merge commits themselves). For our purposes w/DVC experiments, this means that a fetch depth of 2 will usually be sufficient (so baseline commit + 1 parent), but that won't be the case 100% of the time. So if users are trying to use `exp run` inside github actions, the general solution is to use `fetch-depth: 0` (and not use a shallow clone at all).

Is a merge necessary to run an experiment? I thought behavior for running experiments was similar to git stash apply, which works fine in this scenario.
> It makes sense as part of making the workspace and temp implementations more consistent, but is there any way to make at least workspace runs succeed in this scenario?

Prior to that change, workspace and temp implementations were essentially two completely different things, which made testing and maintaining both difficult, and led to them having different behavior in a lot of scenarios. I don't think it is worth the development effort required to maintain two separate `exp run` implementations vs requiring that users unshallow repos in CI for both the workspace and tempdir cases.

> Is a merge necessary to run an experiment? I thought behavior for running experiments was similar to git stash apply, which works fine in this scenario.

It works "similar to" stash apply, not exactly like `git stash apply`. We have to do a merge because libgit2/pygit2 and dulwich do not have implementations for `stash apply`-ing arbitrary merge commits. In order for us to get the similar behavior, we have to just use regular `merge` to apply the merge commit.

Resolving patch:
diff --git a/dvc/repo/experiments/apply.py b/dvc/repo/experiments/apply.py
--- a/dvc/repo/experiments/apply.py
+++ b/dvc/repo/experiments/apply.py
@@ -22,7 +22,7 @@ def apply(repo, rev, force=True, **kwargs):
     from scmrepo.exceptions import MergeConflictError
 
     from dvc.repo.checkout import checkout as dvc_checkout
-    from dvc.scm import RevError, SCMError, resolve_rev
+    from dvc.scm import GitMergeError, RevError, resolve_rev
 
     exps = repo.experiments
 
@@ -51,7 +51,7 @@ def apply(repo, rev, force=True, **kwargs):
     try:
         repo.scm.merge(exp_rev, commit=False, squash=True)
     except _SCMError as exc:
-        raise SCMError(str(exc))
+        raise GitMergeError(str(exc), scm=repo.scm)
 
     if workspace:
         try:
diff --git a/dvc/repo/experiments/executor/local.py b/dvc/repo/experiments/executor/local.py
--- a/dvc/repo/experiments/executor/local.py
+++ b/dvc/repo/experiments/executor/local.py
@@ -5,8 +5,9 @@
 from typing import TYPE_CHECKING, Optional
 
 from funcy import cached_property
+from scmrepo.exceptions import SCMError as _SCMError
 
-from dvc.scm import SCM
+from dvc.scm import SCM, GitMergeError
 from dvc.utils.fs import remove
 
 from ..base import (
@@ -88,7 +89,10 @@ def init_git(self, scm: "Git", branch: Optional[str] = None):
         head = EXEC_BRANCH if branch else EXEC_HEAD
         self.scm.checkout(head, detach=True)
         merge_rev = self.scm.get_ref(EXEC_MERGE)
-        self.scm.merge(merge_rev, squash=True, commit=False)
+        try:
+            self.scm.merge(merge_rev, squash=True, commit=False)
+        except _SCMError as exc:
+            raise GitMergeError(str(exc), scm=self.scm)
 
     def _config(self, cache_dir):
         local_config = os.path.join(
@@ -155,7 +159,10 @@ def init_git(self, scm: "Git", branch: Optional[str] = None):
             )
         )
         merge_rev = self.scm.get_ref(EXEC_MERGE)
-        self.scm.merge(merge_rev, squash=True, commit=False)
+        try:
+            self.scm.merge(merge_rev, squash=True, commit=False)
+        except _SCMError as exc:
+            raise GitMergeError(str(exc), scm=self.scm)
         if branch:
             self.scm.set_ref(EXEC_BRANCH, branch, symbolic=True)
         elif scm.get_ref(EXEC_BRANCH):
diff --git a/dvc/scm.py b/dvc/scm.py
--- a/dvc/scm.py
+++ b/dvc/scm.py
@@ -1,4 +1,5 @@
 """Manages source control systems (e.g. Git)."""
+import os
 from contextlib import contextmanager
 from functools import partial
 from typing import TYPE_CHECKING, Iterator, List, Mapping, Optional
@@ -10,6 +11,7 @@
 
 from dvc.exceptions import DvcException
 from dvc.progress import Tqdm
+from dvc.utils import format_link
 
 if TYPE_CHECKING:
     from scmrepo.progress import GitProgressEvent
@@ -47,6 +49,25 @@ def __init__(self, reason: str) -> None:
         super().__init__(f"{reason}\n{doc}")
 
 
+class GitMergeError(SCMError):
+    def __init__(self, msg: str, scm: Optional["Git"] = None) -> None:
+        if scm and self._is_shallow(scm):
+            url = format_link(
+                "https://dvc.org/doc/user-guide/troubleshooting#git-shallow"
+            )
+            msg = (
+                f"{msg}: `dvc exp` does not work in shallow Git repos. "
+                f"See {url} for more information."
+            )
+        super().__init__(msg)
+
+    @staticmethod
+    def _is_shallow(scm: "Git") -> bool:
+        if os.path.exists(os.path.join(scm.root_dir, Git.GIT_DIR, "shallow")):
+            return True
+        return False
+
+
 @contextmanager
 def map_scm_exception(with_cause: bool = False) -> Iterator[None]:
     from scmrepo.exceptions import SCMError as InternalSCMError
@@ -115,8 +136,6 @@ def update_git(self, event: "GitProgressEvent") -> None:
 
 
 def clone(url: str, to_path: str, **kwargs):
-    import os
-
     from scmrepo.exceptions import CloneError as InternalCloneError
 
     from dvc.repo.experiments.utils import fetch_all_exps


Test patch:
diff --git a/tests/unit/scm/test_scm.py b/tests/unit/scm/test_scm.py
--- a/tests/unit/scm/test_scm.py
+++ b/tests/unit/scm/test_scm.py
@@ -1,5 +1,5 @@
 from dvc.repo.experiments import ExpRefInfo
-from dvc.scm import iter_revs
+from dvc.scm import GitMergeError, iter_revs
 
 
 def test_iter_revs(
@@ -61,3 +61,11 @@ def test_iter_revs(
         rev_old: [rev_old],
         rev_root: [rev_root],
     }
+
+
+def test_merge_error(tmp_dir, scm):
+    exc = GitMergeError("Merge failed")
+    assert "shallow" not in str(exc)
+    tmp_dir.gen({".git": {"shallow": ""}})
+    exc = GitMergeError("Merge failed", scm=scm)
+    assert "shallow" in str(exc)