Repository: iterative/dvc

Issue:
Easy way to `repro` all the outdated `.dvc` files in a repository tree
Currently, the way I integrate DVC into my workflow is:

Consider the sample repo:
```
- [repo root]
  - src
    - script1.py
    - script2.py
    - script3.py
  - data
    - file1.csv
    - file1.csv.dvc
    - file2.csv
    - file2.csv.dvc
    - file2_1.csv
    - file2_1.csv.dvc
    - file2_2.csv
    - file2_2.csv.dvc
```

Where:
- `file2.csv` is generated by `script1.py` and depends on `file1.csv`.
- `file2_1.csv` is generated by `script2.py` and depends on `file2.csv`.
- `file2_2.csv` is generated by `script3.py` and depends on `file2.csv`.


Imagine that I change `script1.py`, that changes `file2.csv`, which is a dependency for `file2_1.csv` and `file2_2.csv`.


I believe that, at the moment, the only way to update all files that became outdated with such change, is to run `dvc status`, check everything that changed (imagine I have dozens of files in a real project) and individually run `dvc repro` on each outdated file.
I know that if there was a `file3.csv` that depends on both `file2_1.csv` and `file2_2.csv` I could just run one `dvc repro file3.csv.dvc`. Still that is not always the case and furthermore requires the user to actively be aware of these cases.


Is there currently an easy way to `repro` all the outdated `.dvc` files in a repository tree?


If not, I think this would be a very nice addition. I would still first show the user what will be executed for him/her to be aware. For confirmation purposes.




Discussion:
Hi @andrethrill !

As a workaround you could use the following trick to tie down that nicely with a dummy Dvcfile placed in a project root(as an example) that will have all your file*.cvs as dependencies and will not have a command nor will it have outputs. To generate that file run:

```
$ dvc run -d data/file2.csv -d data/file2_1.csv ... -f Dvcfile
```
and then you will be able to just run
```
$ dvc repro
```
to make dvc handle everything for you.

This will essentially tie down all the hanging ends(outputs) of your pipeline into a single final stage that is there just for convenience, so you don't even have to wonder which pipeline branch you need to call `repro` on when you come back to the project in a while.

That being said, I think we could definitely introduce some option like `--pipeline`(since `dvc pipeline show --ascii stage.dvc` already outputs the whole pipeline for the specified stage and not only its dependencies) or something for the `dvc repro` that will find the end stages for the pipeline that the stage belongs to and will traverse it from there. Would that be suitable for you?

Thanks,
Ruslan
Hi @efiop! :)

Thank you very much for the quick reply as usual.

If I understand well your proposed workaround, that would require the user to actively remember to go and update such dummy dvc file everytime a new .dvc file is created or removed right? It seems a bit cumbersome from the user perspective.

The solution you propose at the end seems much more appealing (for me at least), even more given that you seem to have already the logic implemented to walk down the repository tree looking for .dvc files and representing the entire pipeline.

At first thought, it seems to me more logical to implement such feature as an option to the `dvc repro` command. (Although I’m not sure how that will change in version 1.0 when dvc commands look more “git like”)

Thanks,
André 
> If I understand well your proposed workaround, that would require the user to actively remember to go and update such dummy dvc file everytime a new .dvc file is created or removed right? It seems a bit cumbersome from the user perspective.

Agreed, that workaround is not ideal for this scenario.

> The solution you propose at the end seems much more appealing (for me at least), even more given that you seem to have already the logic implemented to walk down the repository tree looking for .dvc files and representing the entire pipeline.
> At first thought, it seems to me more logical to implement such feature as an option to the dvc repro command. (Although I’m not sure how that will change in version 1.0 when dvc commands look more “git like)

Yes, I meant `dvc repro -p|--pipeline`. I'll send a patch for it soon, should be released with 0.19 until the end of the week. Regarding v1.0, I'm sure --pipeline will fit nicely there as well 🙂 

Thank you for the feedback!

-Ruslan

Resolving patch:
diff --git a/dvc/cli.py b/dvc/cli.py
--- a/dvc/cli.py
+++ b/dvc/cli.py
@@ -396,6 +396,13 @@ def parse_args(argv=None):
                         default=False,
                         help='Ask for confirmation before reproducing each '
                              'stage.')
+    repro_parser.add_argument(
+                        '-p',
+                        '--pipeline',
+                        action='store_true',
+                        default=False,
+                        help='Reproduce the whole pipeline that the '
+                             'specified stage file belongs to.')
     repro_parser.set_defaults(func=CmdRepro)
 
     # Remove
diff --git a/dvc/command/repro.py b/dvc/command/repro.py
--- a/dvc/command/repro.py
+++ b/dvc/command/repro.py
@@ -20,7 +20,8 @@ def run(self):
                                        recursive=recursive,
                                        force=self.args.force,
                                        dry=self.args.dry,
-                                       interactive=self.args.interactive)
+                                       interactive=self.args.interactive,
+                                       pipeline=self.args.pipeline)
 
                 if len(stages) == 0:
                     self.project.logger.info(CmdDataStatus.UP_TO_DATE_MSG)
diff --git a/dvc/project.py b/dvc/project.py
--- a/dvc/project.py
+++ b/dvc/project.py
@@ -311,39 +311,73 @@ def reproduce(self,
                   recursive=True,
                   force=False,
                   dry=False,
-                  interactive=False):
-        import networkx as nx
-
-        stage = Stage.load(self, target)
-        G = self.graph()[1]
-        stages = nx.get_node_attributes(G, 'stage')
-        node = os.path.relpath(stage.path, self.root_dir)
+                  interactive=False,
+                  pipeline=False):
 
         if not interactive:
             config = self.config
             core = config._config[config.SECTION_CORE]
             interactive = core.get(config.SECTION_CORE_INTERACTIVE, False)
 
+        targets = []
+        if pipeline:
+            stage = Stage.load(self, target)
+            node = os.path.relpath(stage.path, self.root_dir)
+            pipelines = list(filter(lambda g: node in g.nodes(),
+                                    self.pipelines()))
+            assert len(pipelines) == 1
+            G = pipelines[0]
+            for node in G.nodes():
+                if G.in_degree(node) == 0:
+                    targets.append(os.path.join(self.root_dir, node))
+        else:
+            targets.append(target)
+
         self._files_to_git_add = []
+
+        ret = []
         with self.state:
-            if recursive:
-                ret = self._reproduce_stages(G,
-                                             stages,
-                                             node,
-                                             force,
-                                             dry,
-                                             interactive)
-            else:
-                ret = self._reproduce_stage(stages,
-                                            node,
-                                            force,
-                                            dry,
-                                            interactive)
+            for target in targets:
+                stages = self._reproduce(target,
+                                         recursive=recursive,
+                                         force=force,
+                                         dry=dry,
+                                         interactive=interactive)
+                ret.extend(stages)
 
         self._remind_to_git_add()
 
         return ret
 
+    def _reproduce(self,
+                   target,
+                   recursive=True,
+                   force=False,
+                   dry=False,
+                   interactive=False):
+        import networkx as nx
+
+        stage = Stage.load(self, target)
+        G = self.graph()[1]
+        stages = nx.get_node_attributes(G, 'stage')
+        node = os.path.relpath(stage.path, self.root_dir)
+
+        if recursive:
+            ret = self._reproduce_stages(G,
+                                         stages,
+                                         node,
+                                         force,
+                                         dry,
+                                         interactive)
+        else:
+            ret = self._reproduce_stage(stages,
+                                        node,
+                                        force,
+                                        dry,
+                                        interactive)
+
+        return ret
+
     def _reproduce_stages(self, G, stages, node, force, dry, interactive):
         import networkx as nx
 


Test patch:
diff --git a/tests/test_repro.py b/tests/test_repro.py
--- a/tests/test_repro.py
+++ b/tests/test_repro.py
@@ -205,6 +205,18 @@ def test(self):
         self.assertEqual(len(stages), 3)
 
 
+class TestReproPipeline(TestReproChangedDeepData):
+    def test(self):
+        stages = self.dvc.reproduce(self.file1_stage,
+                                    force=True,
+                                    pipeline=True)
+        self.assertEqual(len(stages), 3)
+
+    def test_cli(self):
+        ret = main(['repro', '--pipeline', '-f', self.file1_stage])
+        self.assertEqual(ret, 0)
+
+
 class TestReproLocked(TestReproChangedData):
     def test(self):
         file2 = 'file2'