Repository: iterative/dvc

Issue:
Support multiline command?
Something like done in Github actions with run and script?
```yaml
stages:
  stage1:
    cmd: |
      cmd1
      cmd2
```

Or, perhaps:
```yaml
stages:
  stage1:
    cmd:
      - cmd1
      - cmd2
```

Or, maybe both?

Need to double check if we will be able to keep the dump in this shape or form.

#### Why?
Better expression


#### Current Workaround:
```yaml
stages:
  stage1:
    cmd: >-
      cmd1
      && cmd2
      && cmd3
```






Discussion:
For the record:
```
stages:
  stage1:
    cmd: |
      cmd1
      cmd2
```

this is already supported and will be passed as is to the shell (we use shell=True in Popen). I would keep it as is.

```
stages:
  stage1:
    cmd:
      - cmd1
      - cmd2
```

This looks great and I think it should behave like cmd1 && cmd2 && cmd3, meaning that if one command fails then the rest is not executed and results in a stage failure.
I have started to look at the code. @efiop is it ok to add a `self.cmd = " && ".join(cmd) if isintance(cmd, list) else cmd` in `Stage.__init__` or should the `Stage` be able to truly handle a list of commands?
Hi @ClementWalter !

Thanks for looking into it! What advantages/disadvantages do you see for doing a `join` like that?
Keeping open for the question about multiline cmd handling raised by @skshetry 
Hi. I'm curious what's the use case for this. Is there a good example of multiple commands that share deps and outs? (to improve the docs)
> Hi. I'm curious what's the use case for this.

This is more of a convenience/ergonomical feature. So, anything you can write in a single line can be written in multiple lines or vice versa but might be easier than just cramming it in one with `&&`.


> Is there a good example of multiple commands that share deps and outs? (to improve the docs)

You can use Github search for this (about usage of `&&`). But, why does it have to share `deps`/`outs` at all?

Here's a list of what I could find:

- building a model and testing it with a different script
- building each model in a separate virtual environment and evaluating them
- building each model in a separate directory
- `cd ../directory && python script.py` - looks like they are keeping dvc repo and code separately.
- fetching a pre-trained model and training them on local datasets (`wget`, etc).
- cleaning up randomly generated files after building/training/evaluating.
- setting some env vars, PythonPath, etc.
- `cd directory && python test.py` - maybe `cd` is more convenient than `wdir` as `cd` only affects during runtime. 
Thanks! I'm aware of POSIX `&&`. My question is about a real user need that this issue initially tried to cover. I'm getting the idea that this is basically for nice formatting (alt. to `&&` basically).

Also thanks for the specific example ideas! I think that most of them may be better served by other forms thouogh:

<details>

> - building a model and testing it with a different script
> - fetching a pre-trained model and training them on local datasets (wget, etc).
> - cleaning up randomly generated files after building/training/evaluating.

Multiple stages

> - building each model in a separate virtual environment and evaluating them
> - building each model in a separate directory

Stage groups (`foreach` syntax in DVC 2.0)

> - cd ../directory && python script.py
> - cd directory && python test.py

`wdir` (or just keeping `&&`)

</details>

**setting several env vars** seems like the only instance where this is really convenient, to avoid an ugly string full of `&&`.

> why does it have to share deps/outs at all?

Initially I was assuming that would be the point of having several commands in a stage, since having each command have separate deps/outs and listing those in a single stage doesn't sound very clean (again, better to use 2 stages instead?) Although I suppose **if the stages are parallel**, it could be a nice use of this feature, along with a `desc` or `# comment` about that perhaps.

Anyway, my Q is answered. Will check with Ivan whether it's worth improving the docs for this. Thanks again!
> Multiple stages

You are thinking of an ideal situation. I won't be creating a separate stage just to clean the logfile the script creates. It's a convenience feature. 

> 
> > * building each model in a separate virtual environment and evaluating them
> > * building each model in a separate directory
> 
> Stage groups (`foreach` syntax in DVC 2.0)

How do you create a virtual environment and run the script with `foreach`? It's a different thing.

> 
> > * cd ../directory && python script.py
> > * cd directory && python test.py
> 
> `wdir` (or just keeping `&&`)

Like I said, `cd` definitely has its place. `wdir` applies to all `outs`/`deps` whereas `cd` is a runtime component.



It all seems very hypothetical to me but yes, you may be right. It's nice to have this option anyway.
> It all seems very hypothetical to me but yes, you may be right.

All of the examples are from Github's search. How could that be hypothetical?
Oh I meant not from DVC users.
@jorgeorpinel 

> Oh I meant not from DVC users.

I guess, they are from DVC users since they come from Github repos that use DVC.
Ah, I didn't realize those GitHub repos use DVC! I thought it was from generic repos. Will try to find some some of them...
Too bad that

> You can't use the following wildcard characters as part of your search query: . , : ; / \ ` ' " = * ! ? # $ & + ^ | ~ < > ( ) { } [ ]. The search will simply ignore these symbols.

i.e. I can't actually search for `&&` \- according to [GH search docs](https://docs.github.com/en/free-pro-team@latest/github/searching-for-information-on-github/searching-code#considerations-for-code-search).
 
So far I've been able to find one interesting example (`python -m ... && fdupes ...`) in [my search](https://github.com/search?q=filename%3Advc.yaml+%26%26&type=Code) but @skshetry if you have a better search can you share it please? I'll keep looking for now.
> building each model in a separate virtual environment and evaluating them

Here's the Julia example:

- each stage has a separate virtual environment: `--project=prepare`. Package dependencies are provided by `prepare/Project.toml` and tracked by `git`.
- `Pkg.instantiate()` sets up the project environment (like `pip install -r requirements.txt`)

```yaml
prepare:
  foreach: ${models}
  do:
    cmd: >-
      julia --project=prepare -e "using Pkg; Pkg.instantiate()" &&
      julia --project=prepare \
        prepare/main.jl data/train/rawdata/ data/train/prepared/${item.name}.h5 \
        --patch-stride ${item.data.patch_stride} \
        --patch-size ${item.data.patch_size} \
        --augment-times ${item.data.augment_times}
    deps:
    - data/train/rawdata
    - prepare
    outs:
    - data/train/prepared/${item.name}.h5
```

I'm thinking of `dvc` as a build system like `make`, so setting up a virtual environment has real merits for reproducibility. Version compatibility is the root of many hidden problems.
@johnnychen94, we do support list now if you are using from the `master`. 

So, you could write that as:
```yaml
cmd:
  - julia --project=prepare -e "using Pkg; Pkg.instantiate()"
  - julia --project=prepare \
        prepare/main.jl data/train/rawdata/ data/train/prepared/${item.name}.h5 \
        --patch-stride ${item.data.patch_stride} \
        --patch-size ${item.data.patch_size} \
        --augment-times ${item.data.augment_times}
```

I see that you use `\` at the end which should not be required as you are using folded style string (with `>-`), which will make it even cleaner.

```yaml
cmd:
  - julia --project=prepare -e "using Pkg; Pkg.instantiate()"
  - >-
    julia --project=prepare
        prepare/main.jl data/train/rawdata/ data/train/prepared/${item.name}.h5
        --patch-stride ${item.data.patch_stride}
        --patch-size ${item.data.patch_size}
        --augment-times ${item.data.augment_times}
```

> i.e. I can't actually search for && - according to GH search docs.

@jorgeorpinel, I use a [simple script](https://gist.github.com/skshetry/94d04f5b505d84ea614834617d647cde) to fetch all the files matching a query using Github's token and search locally. There aren't that many, so should be quick. Try the script in `wsl`.
Thanks for sharing your real like example @johnnychen94 ! I'm not familiar with Julia so forgive me if this is obvious but why do you run `using Pkg; Pkg.instantiate()` separately instead of having it in `prepare/main.jl`?

I'll try your script and lyk @skshetry! Thanks
> I'm not familiar with Julia so forgive me if this is obvious but why do you run using Pkg; Pkg.instantiate() separately instead of having it in prepare/main.jl?

There are no standard rules about it. It's doable but I deliberately kept them apart because I think the environment initialization part has nothing to do with the real code. We normally run the real code multiple times (for developing or debugging purposes) while only initialize the environment once.

This is also something that the whole Julia community tries to commit when dealing with CI. For example: https://github.com/JuliaImages/juliaimages.github.io/blob/cbd83915f5dfb0259fedaa3d1d14c67f34d983eb/.travis.yml#L14-L16

Resolving patch:
diff --git a/dvc/stage/run.py b/dvc/stage/run.py
--- a/dvc/stage/run.py
+++ b/dvc/stage/run.py
@@ -47,7 +47,7 @@ def warn_if_fish(executable):
 
 def _enforce_cmd_list(cmd):
     assert cmd
-    return cmd if isinstance(cmd, list) else [cmd]
+    return cmd if isinstance(cmd, list) else cmd.splitlines()
 
 
 def prepare_kwargs(stage, checkpoint_func=None):


Test patch:
diff --git a/tests/func/test_repro_multistage.py b/tests/func/test_repro_multistage.py
--- a/tests/func/test_repro_multistage.py
+++ b/tests/func/test_repro_multistage.py
@@ -529,7 +529,14 @@ def test_repro_multiple_params(tmp_dir, dvc):
     assert dvc.reproduce(stage.addressing) == [stage]
 
 
-def test_repro_list_of_commands_in_order(tmp_dir, dvc):
+@pytest.mark.parametrize("multiline", [True, False])
+def test_repro_list_of_commands_in_order(tmp_dir, dvc, multiline):
+    cmd = ["echo foo>foo", "echo bar>bar"]
+    if multiline:
+        cmd = "\n".join(cmd)
+
+    dump_yaml("dvc.yaml", {"stages": {"multi": {"cmd": cmd}}})
+
     (tmp_dir / "dvc.yaml").write_text(
         dedent(
             """\
@@ -546,19 +553,16 @@ def test_repro_list_of_commands_in_order(tmp_dir, dvc):
     assert (tmp_dir / "bar").read_text() == "bar\n"
 
 
-def test_repro_list_of_commands_raise_and_stops_after_failure(tmp_dir, dvc):
-    (tmp_dir / "dvc.yaml").write_text(
-        dedent(
-            """\
-            stages:
-              multi:
-                cmd:
-                - echo foo>foo
-                - failed_command
-                - echo baz>bar
-        """
-        )
-    )
+@pytest.mark.parametrize("multiline", [True, False])
+def test_repro_list_of_commands_raise_and_stops_after_failure(
+    tmp_dir, dvc, multiline
+):
+    cmd = ["echo foo>foo", "failed_command", "echo baz>bar"]
+    if multiline:
+        cmd = "\n".join(cmd)
+
+    dump_yaml("dvc.yaml", {"stages": {"multi": {"cmd": cmd}}})
+
     with pytest.raises(ReproductionError):
         dvc.reproduce(targets=["multi"])
     assert (tmp_dir / "foo").read_text() == "foo\n"