Repository: iterative/dvc

Issue:
dvc exp run: revision for not queue/temp scenario
DVC start each checkpoint from scratch when use `--rev` not from queue/temp:
```
$ dvc exp run --rev 452b20a --set-param lr=0.01
...
```
Accuracy is improving from 0.0 to ~0.16 in each checkpoint and starts from 0.0 again while it should start from 0.735 and got up with each of the checkpoint:

```
$ dvc exp show --no-pager
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━┳━━━━━━━━┳━━━┓
┃ Experiment              ┃ Created      ┃ step ┃   loss ┃ accuracy ┃ … ┃ epochs ┃ … ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━╇━━━━━━━━╇━━━┩
│ workspace               │ -            │   41 │ 2.1555 │  0.16285 │ … │ 15     │ … │
│ master                  │ Mar 30, 2021 │    - │      - │        - │ - │ 15     │ … │
│ │ ╓ exp-09c97           │ 01:54 AM     │   41 │ 2.1555 │  0.16285 │ … │ 15     │ … │
│ │ ╟ 093d983             │ 01:54 AM     │   40 │ 2.1634 │   0.1629 │ … │ 15     │ … │
│ │ ╟ a8bb8f5             │ 01:54 AM     │   39 │ 2.1638 │  0.16078 │ … │ 15     │ … │
│ │ ╟ 85a61a9 (452b20a)   │ 01:54 AM     │   38 │ 2.2047 │  0.16487 │ … │ 15     │ … │
...

│ │ ╟ 452b20a             │ Apr 12, 2021 │   18 │ 1.4032 │    0.735 │ … │ 15     │ … │
```

PS: `--temp` option name is a bit confusing. I'd prefer something like `--background`.


Discussion:
So the issue here is that the intended workflow for workspace runs is that the user should do:

```
dvc exp apply 452b20a
dvc exp run --set-param lr=0.01
```

Right now what happens internally when with `dvc exp run --rev 452b20a --set-param lr=0.01` is that an experiment is started based on `452b20a`, but the current workspace state is also used on top of `452b20a`. So the end result is that your current workspace's version of your model file is actually used in your new run, which is why you are seeing the wrong accuracy values in your new run.

`--rev` is really only meant to be used explicitly for `--temp`/`--queue` runs to account for the scenario where the user never applies anything to their workspace in the first place.

I think we should disallow `--rev` for workspace runs and enforce that the user do `exp apply` followed by `exp run` here. (enforcing this properly will also require the fix for https://github.com/iterative/dvc/issues/5737, since right now `exp apply` may fail due to the ambiguous commit issue)

@dmpetrov @dberenbaum thoughts?
@pmrowla disabling should work. If it is a queue specific, does it make sense to extract it to `dvc exp queue` umbrella command? Might be related to #5615.

I see `dvc exp run --rev 452b20a --set-param lr=0.01` as a really useful shortcut for `apply+run` for regular workflow (not queue). It would be great to have that feature. Does anything prevent us from generalization this feature for both of the scenarios?

CC @jorgeorpinel 
> --rev is really only meant to be used explicitly for --temp/--queue runs

We currently say that "You can add a `--rev` to continue from a previous checkpoint" (workspace run context), and separately that `--rev` needed to resume `--queued/temp` runs. So are we saying that's incorrect and `--rev` is actually meant exclusively for out-of-workspace runs? Thanks

> PS: --temp option name is a bit confusing. I'd prefer something like --background.

Extracted to #5826
> @pmrowla disabling should work. If it is a queue specific, does it make sense to extract it to `dvc exp queue` umbrella command? Might be related to #5615.

@dmpetrov having the separate queue command is something that we will definitely need at some point, and it would make sense to move `--rev` into that set of commands at that time.

> I see `dvc exp run --rev 452b20a --set-param lr=0.01` as a really useful shortcut for `apply+run` for regular workflow (not queue). It would be great to have that feature. Does anything prevent us from generalization this feature for both of the scenarios?

We could implement it this way, but it's not clear what we should be doing with changes in the users current workspace. As a shortcut for a pure `apply`+`run`, we would remove any modifications that the user has in their workspace, which I'm not sure would be expected here (basically this would end up having the same effect as `git reset --hard 452b20a` at the start of the run)

> We currently say that "You can add a --rev to continue from a previous checkpoint" (workspace run context), and separately that --rev needed to resume --queued/temp runs. So are we saying that's incorrect and --rev is actually meant exclusively for out-of-workspace runs? Thanks

@jorgeorpinel the current implementation does allow using `--rev` from in the workspace, so the current docs are technically correct. We're saying that we should change this and make it so `--rev` is only applicable for out of workspace runs.
> make it so --rev is only applicable for out of workspace runs

I see. So `exp apply` would be the way to "resume" from a certain checkpoint?

*I guess that's what you said in https://github.com/iterative/dvc/issues/5814#issuecomment-820270004 (sry gets confusing with all the types of exps and runs).*

Resolving patch:
diff --git a/dvc/command/experiments.py b/dvc/command/experiments.py
--- a/dvc/command/experiments.py
+++ b/dvc/command/experiments.py
@@ -509,10 +509,16 @@ class CmdExperimentsRun(CmdRepro):
     def run(self):
         from dvc.command.metrics import _show_metrics
 
-        if self.args.reset and self.args.checkpoint_resume:
-            raise InvalidArgumentError(
-                "--reset and --rev are mutually exclusive."
-            )
+        if self.args.checkpoint_resume:
+            if self.args.reset:
+                raise InvalidArgumentError(
+                    "--reset and --rev are mutually exclusive."
+                )
+            if not (self.args.queue or self.args.tmp_dir):
+                raise InvalidArgumentError(
+                    "--rev can only be used in conjunction with "
+                    "--queue or --temp."
+                )
 
         if self.args.reset:
             logger.info("Any existing checkpoints will be reset and re-run.")
@@ -932,9 +938,8 @@ def add_parser(subparsers, parent_parser):
         type=str,
         dest="checkpoint_resume",
         help=(
-            "Continue the specified checkpoint experiment. "
-            "(Only required for explicitly resuming checkpoints in queued "
-            "or temp dir runs.)"
+            "Continue the specified checkpoint experiment. Can only be used "
+            "in conjunction with --queue or --temp."
         ),
         metavar="<experiment_rev>",
     ).complete = completion.EXPERIMENT


Test patch: