Repository: iterative/dvc

Issue:
exp run: Remove unsupported `dvc repro` options from the help text
# Bug Report

UPDATE: Jump to https://github.com/iterative/dvc/issues/6791#issuecomment-1043363367

## Description

`dvc exp run --help` lists the following options:

```
  -s, --single-item     Reproduce only single data item without recursive dependencies check.
  -m, --metrics         Show metrics after reproduction.
  --dry                 Only print the commands that would be executed without actually executing.
  -i, --interactive     Ask for confirmation before reproducing each stage.
  -p, --pipeline        Reproduce the whole pipeline that the specified targets belong to.
  -P, --all-pipelines   Reproduce all pipelines in the repo.
  -R, --recursive       Reproduce all stages in the specified directory.
  --no-run-cache        Execute stage commands even if they have already been run with the same command/dependencies/outputs/etc
                        before.
  --force-downstream    Reproduce all descendants of a changed stage even if their direct dependencies didn't change.
  --no-commit           Don't put files/directories into cache.
  --downstream          Start from the specified stages when reproducing pipelines.
  --pull                Try automatically pulling missing cache for outputs restored from the run-cache.
  --glob                Allows targets containing shell-style wildcards.
```

Some of them are not supported. I tested a few of them and testing the rest. Could you remove those from the help text? We're going to update the command reference according to this. 

- [x] `--force`: Seems to run
- [x] `--single-item`: I didn't get what does this do in `dvc exp run` context
- [x] `--metrics` : ✅
- [x] `--interactive`: ✅
- [x] `--pipeline`: ✅
- [x] `--all-pipelines`: seems to run ✅
- [x] `--recursive`: ✅
- [x] `--no-run-cache`: seems to have no effect ❌
- [x] `--force-downstream`: 
- [x] `--no-commit`: seems to have no effect ❌
- [x] `--downstream`:
- [x] `--pull`:
- [x] `--glob`: ❌



Discussion:
Thank you @iesahin, this is great!

>     * [x]  `--no-run-cache`: seems to have no effect ❌

>     * [x]  `--no-commit`: seems to have no effect ❌

It makes sense to me that these wouldn't work since experiments need to be able to cache data to retrieve it. I'm fine with removing these unless anyone else objects.


>  `--glob`: ❌

This probably still needs to be fixed in #6457. We may need to bump the priority up unless anyone has a reason not to, in which case we should remove the option for now.

>     * [x]  `--interactive`: ✅

Hm, does this make sense for experiments? How does it work with checkpoints or experiments outside of the working directory?
> It makes sense to me that these wouldn't work since experiments need to be able to cache data to retrieve it. I'm fine with removing these unless anyone else objects.

I think we should remove these even if someone objects. 😄 We can ask for PRs from them. 



> Hm, does this make sense for experiments? How does it work with checkpoints or experiments outside of the working directory?

Interactive reproduction may lead to some unforeseen results about the experiments and checkpoints. It's identical with the basic `dvc exp run` if you say `y` to all stages, but what happens if you skip some of these stages, I don't know. (e.g., do we generate `exp-id` from the whole pipeline or only the stages that run?)

I believe, in general, experiments should not be granularized by stages, we already have checkpoints for this. Interaction between these two features may lead to some inconsistent behavior. 
> I think we should remove these even if someone objects. 😄 We can ask for PRs from them.

😄 I meant someone from @iterative/dvc 



> Interactive reproduction may lead to some unforeseen results about the experiments and checkpoints. It's identical with the basic `dvc exp run` if you say `y` to all stages, but what happens if you skip some of these stages, I don't know. (e.g., do we generate `exp-id` from the whole pipeline or only the stages that run?)
> 
> I believe, in general, experiments should not be granularized by stages, we already have checkpoints for this. Interaction between these two features may lead to some inconsistent behavior.

Agreed. I think we should drop this option.
I believe if some option from `dvc repro` seems not relevant clearly, we can better drop it from `dvc exp run`. If we decide on a clear behavior later, we can revive them. :)

If _stage selection_ is not particularly clear, we can drop `--glob` as well. It's about stage selection. 
> If _stage selection_ is not particularly clear, we can drop `--glob` as well. It's about stage selection.

I'm not sure how often `--glob` will be needed, but `exp run` takes a stage name as a target, so stage selection is included. Most of these options are about stage selection. Looking at #4912, I can see this being just as useful in `exp run` as in `repro`.
I think, for the time being, it's better to remove these options from the help text and the documentation. If we get some feedback about the lack of these, we can create new tickets and move forward.
Hey everyone! I'm having an issue with `dvc repro --glob` where the pattern I'm passing is being ignored. 
 (i.e. all target stages match instead of just those specified by my simple pattern).

Is it possible that `--glob` is currently being ignored by `dvc repro` :question: 
Not sure if it's related, but the `glob_targets` method only appears on the `pull`, `push` and `add` commands (in [this search](https://github.com/iterative/dvc/search?q=glob_targets)) which, along with this issue, led me to think that `--glob` might not be supported by `dvc repro` yet.

Since this issue looked _active_ and similar to my problem, I thought it'd be easier to ask here instead of opening a possibly duplicate issue. Apologies if I should have proceeded differently.

Thank you so much!

PD: I'm running version `2.7.2` (I skimmed changes up to the latest in the releases page but didn't see anything related to `glob` nor `repro`, so I didn't bother upgrading)
@erovira `dvc repro --glob` works as expected for me. Could you open a separate issue if you are seeing an issue with it?
Hey @dberenbaum , sure! I'll see if I can come up with a minimal reproducible example and create an issue. Thanks for answering!

**EDIT:** No need to open a new issue. I found out what my mistake was.
Suppose I have three stages with the following DAG: `A` -> `prefix-B` -> `prefix-C`.
I was trying to do a repro of B and C just by doing: `dvc repro --glob prefix-*`, but I was seeing `A` being run and thought `--glob` wasn't working. As I now (think I) understand, A is run because of the default recursive search for changed dependencies.

Combining `--single-item` and `--glob` like so `dvc repro --single-item --glob prefix-*` did the trick for me.

Thanks again!
>  --no-commit: seems to have no effect ❌

I unchecked this since the option is still shown and accepted (2.8.1).

Didn't double check any other ones.
Those checkmarks show that I've checked them by running the command. :) @jorgeorpinel 
OK. A bit confused by the check on one side and the x on the other.

![image](https://user-images.githubusercontent.com/1477535/146501297-2e9c89c1-fa03-4cc6-bb9d-eab46d3eabee.png)
All of these commands have been tested here or in https://github.com/iterative/dvc.org/issues/2861. We need to remove `--no-run-cache`, `--no-commit`, and `--glob` (unless someone prefers to address #6457).

cc @pmrowla @karajan1001 

Resolving patch:
diff --git a/dvc/commands/experiments/run.py b/dvc/commands/experiments/run.py
--- a/dvc/commands/experiments/run.py
+++ b/dvc/commands/experiments/run.py
@@ -39,7 +39,7 @@ def run(self):
             reset=self.args.reset,
             tmp_dir=self.args.tmp_dir,
             machine=self.args.machine,
-            **self._repro_kwargs,
+            **self._common_kwargs,
         )
 
         if self.args.metrics and results:
diff --git a/dvc/commands/repro.py b/dvc/commands/repro.py
--- a/dvc/commands/repro.py
+++ b/dvc/commands/repro.py
@@ -10,7 +10,9 @@ class CmdRepro(CmdBase):
     def run(self):
         from dvc.ui import ui
 
-        stages = self.repo.reproduce(**self._repro_kwargs)
+        stages = self.repo.reproduce(
+            **self._common_kwargs, **self._repro_kwargs
+        )
         if len(stages) == 0:
             ui.write(CmdDataStatus.UP_TO_DATE_MSG)
         else:
@@ -27,7 +29,7 @@ def run(self):
         return 0
 
     @property
-    def _repro_kwargs(self):
+    def _common_kwargs(self):
         return {
             "targets": self.args.targets,
             "single_item": self.args.single_item,
@@ -36,12 +38,17 @@ def _repro_kwargs(self):
             "interactive": self.args.interactive,
             "pipeline": self.args.pipeline,
             "all_pipelines": self.args.all_pipelines,
-            "run_cache": not self.args.no_run_cache,
-            "no_commit": self.args.no_commit,
             "downstream": self.args.downstream,
             "recursive": self.args.recursive,
             "force_downstream": self.args.force_downstream,
             "pull": self.args.pull,
+        }
+
+    @property
+    def _repro_kwargs(self):
+        return {
+            "run_cache": not self.args.no_run_cache,
+            "no_commit": self.args.no_commit,
             "glob": self.args.glob,
         }
 
@@ -110,15 +117,6 @@ def add_arguments(repro_parser):
         default=False,
         help="Reproduce all stages in the specified directory.",
     )
-    repro_parser.add_argument(
-        "--no-run-cache",
-        action="store_true",
-        default=False,
-        help=(
-            "Execute stage commands even if they have already been run with "
-            "the same command/dependencies/outputs/etc before."
-        ),
-    )
     repro_parser.add_argument(
         "--force-downstream",
         action="store_true",
@@ -126,12 +124,6 @@ def add_arguments(repro_parser):
         help="Reproduce all descendants of a changed stage even if their "
         "direct dependencies didn't change.",
     )
-    repro_parser.add_argument(
-        "--no-commit",
-        action="store_true",
-        default=False,
-        help="Don't put files/directories into cache.",
-    )
     repro_parser.add_argument(
         "--downstream",
         action="store_true",
@@ -147,12 +139,6 @@ def add_arguments(repro_parser):
             "from the run-cache."
         ),
     )
-    repro_parser.add_argument(
-        "--glob",
-        action="store_true",
-        default=False,
-        help="Allows targets containing shell-style wildcards.",
-    )
 
 
 def add_parser(subparsers, parent_parser):
@@ -166,5 +152,28 @@ def add_parser(subparsers, parent_parser):
         help=REPRO_HELP,
         formatter_class=argparse.RawDescriptionHelpFormatter,
     )
+    # repro/exp run shared args
     add_arguments(repro_parser)
+    # repro only args
+    repro_parser.add_argument(
+        "--no-run-cache",
+        action="store_true",
+        default=False,
+        help=(
+            "Execute stage commands even if they have already been run with "
+            "the same command/dependencies/outputs/etc before."
+        ),
+    )
+    repro_parser.add_argument(
+        "--no-commit",
+        action="store_true",
+        default=False,
+        help="Don't put files/directories into cache.",
+    )
+    repro_parser.add_argument(
+        "--glob",
+        action="store_true",
+        default=False,
+        help="Allows targets containing shell-style wildcards.",
+    )
     repro_parser.set_defaults(func=CmdRepro)


Test patch:
diff --git a/tests/unit/command/test_experiments.py b/tests/unit/command/test_experiments.py
--- a/tests/unit/command/test_experiments.py
+++ b/tests/unit/command/test_experiments.py
@@ -22,7 +22,7 @@
 from tests.utils import ANY
 from tests.utils.asserts import called_once_with_subset
 
-from .test_repro import default_arguments as repro_arguments
+from .test_repro import common_arguments as repro_arguments
 
 
 def test_experiments_apply(dvc, scm, mocker):
diff --git a/tests/unit/command/test_repro.py b/tests/unit/command/test_repro.py
--- a/tests/unit/command/test_repro.py
+++ b/tests/unit/command/test_repro.py
@@ -1,22 +1,24 @@
 from dvc.cli import parse_args
 from dvc.commands.repro import CmdRepro
 
-default_arguments = {
+common_arguments = {
     "all_pipelines": False,
     "downstream": False,
     "dry": False,
     "force": False,
-    "run_cache": True,
     "interactive": False,
-    "no_commit": False,
     "pipeline": False,
     "single_item": False,
     "recursive": False,
     "force_downstream": False,
     "pull": False,
-    "glob": False,
     "targets": [],
 }
+repro_arguments = {
+    "run_cache": True,
+    "no_commit": False,
+    "glob": False,
+}
 
 
 def test_default_arguments(dvc, mocker):
@@ -24,14 +26,17 @@ def test_default_arguments(dvc, mocker):
     mocker.patch.object(cmd.repo, "reproduce")
     cmd.run()
     # pylint: disable=no-member
-    cmd.repo.reproduce.assert_called_with(**default_arguments)
+    cmd.repo.reproduce.assert_called_with(
+        **common_arguments, **repro_arguments
+    )
 
 
 def test_downstream(dvc, mocker):
     cmd = CmdRepro(parse_args(["repro", "--downstream"]))
     mocker.patch.object(cmd.repo, "reproduce")
     cmd.run()
-    arguments = default_arguments.copy()
+    arguments = common_arguments.copy()
+    arguments.update(repro_arguments)
     arguments.update({"downstream": True})
     # pylint: disable=no-member
     cmd.repo.reproduce.assert_called_with(**arguments)