Repository: iterative/dvc

Issue:
More flexible dvc.yaml parameterisation
## Context

One of DVC usage scenarios we should consider is the following:
1. User has a big and complex real-life repo and workflows around it (one example is CI/CD)
1. User wants to apply DVC without modifications to his current scripts, because that will trigger code changes in subsequent workflows, which the user don’t want to do right now, but rather later, when DVC will be integrated and will prove it is useful for the user.

One example for this is the [YOLOv5 repo](https://github.com/ultralytics/yolov5). The github repo has only [testing in CI](https://github.com/ultralytics/yolov5/blob/develop/.github/workflows), but for sure we can have a private fork and multiple CI/CD workflows which train, test, export and deploy models.

## Problem

In this case user could have scripts [which run like this](https://github.com/ultralytics/yolov5/blob/develop/.github/workflows/ci-testing.yml#L67):
```
python train.py --img 128 --batch 16 --weights weights/${{ matrix.model }}.pt --cfg models/${{ matrix.model }}.yaml --epochs 1 --device $di
```

The scripts could have a lot of args (this `train.py` has about 20 of them, all handled by Python argparse module). If I don’t want to modify my scripts yet, I’d need to duplicate all arguments in two places: in params.yaml and in dvc.yaml, thus having three slightly different copies of the arguments.

Indeed, this is a bad practice that creates a technical debt of maintaining three copies of the same and shouldn’t be used for too long, but as a user I would prefer to migrate to DVC step by step, not breaking anything in current workflows in the process. And if I’m not sure I want to migrate and just want to give DVC a try I would want to keep all my current scripts unchanged to remove DVC later if I won’t be planning on using it.

## Suggestion
Expand args dictionary and lists from params in dvc.yaml to allow user to avoid duplicating every param in dvc.yaml:
```
# params.yaml
train-params:
  param1: 1
  param2: 2
  param3: [value1, value2]


# dvc.yaml
stages:
  train:
    cmd: python train.py ${train-params}
```

which gets translated to
```
python train.py --param1 1 --param2 2 --param3 value1 value2
```

## Few questions we should also consider:

<details>
<summary><b>1. Is there a use case when a user wants to expand a nested dict?</b></summary>

Example:
```
# params.yaml
train-params:
  param1:
    nested-param: value

# dvc.yaml
stages:
  train:
    cmd: python train.py ${train-params}
```
</details>

<details>
<summary><b>2. There are optional args like `--resume` which could be supplied or not. We could support this like following:</b></summary>

```
# dvc.yaml
stages:
  train:
    deps:
      - train.py
    cmd: python train.py --resume ${resume}
```

In case we want to run just “python train.py”
```
$ dvc exp run -S resume= 
> python train.py

# params.yaml
resume:
```
In case we want to run “python train.py --resume path-to-weights.pt”:
```
$ dvc exp run -S resume=path-to-weights.pt
> python train.py --resume path-to-weights.pt

# params.yaml
resume: path-to-weights.pt
```
Right now the first use case is not supported at all:
```
$ dvc exp run -S param=  
ERROR: Must provide a value for parameter 'param'
```
Though params.yaml could be like
```
# params.yaml
param:
```
And experiment could be run, but “None” value will be used
```
$ dvc exp run
> python train.py None
```
That said, the last modification could break backwards compatibility if anyone uses the later approach with None substitution, but it looks more like a bug, then a solution, and we can replace this behaviour in the future.
</details>

<details>
<summary><b>3. Are there other examples of CLI parameters that scripts can take, but dvc.yaml / `dvc exp run` doesn't support now?</b></summary>

Would be great to gather them is well.
</details>

My perspective here is limited to python and bash scripts, so I can miss something important. I'm going to gather more examples and possible issues and add more information about them. Meanwhile, it would be great to hear what others think.


Comments:
@skshetry, I tag you because AFAIK you work with yaml in DVC a lot :)
1. We have discussed this before. It's very opinionated to assume it to be `--flag=value`. What if some user wants it in `json` to pass to `jq` for example? So, it's hard to make this decision for the users.

2. Could you please create a separate issue for this? It's an `experiments` related issue, stemming from the checks here:
https://github.com/iterative/dvc/blob/eb46c03c6c02e837c09d76cc3c96c889a84bafe8/dvc/utils/cli_parse.py#L36-L38

Note that the set-params are not the same as parametrized variables. They just happen to use `params.yaml` as a default.

> And experiment could be run, but “None” value will be used

What should the correct behavior be? What you are specifying there is a null value as per yaml/json terms. There can be questions about how to represent it though.
The unambiguous thing to do is to set empty-strings  `''`. 

3. There were a few discussions on supporting lists on `cmd`. But we did not implement it because there might be different representations of the list that the user might need (comma-separated, space-separated, python's list repr, etc), and I did not want to make `cmd` be a special case (you can use parametrization anywhere). 🙂 



@aguschin Have you tried to search through past issues and discussions for context?

@skshetry Are there past issues or discussions that you could provide for context? I can try to find some, but I thought you might be more familiar.

For example, AFAIK, the current templating was an intentional choice to add some limited flexibility without the complexity of a full templating language or python API. Similarly, I think there are past discussions about how to automatically detect dependencies (like parameters) and avoid duplication.
Interesting point about passing params to a script. 

> We have discussed this before. It's very opinionated to assume it to be --flag=value. What if some user wants it in json to pass to jq for example? So, it's hard to make this decision for the users.

I think we still should till try to analyze how common for DVC users it would be (vs `json to jq` which is quite artificial compared to YOLO5 repo). It might turn out that but giving this option (not making it default) we can simplify a lot of ML scenarios.
I might be missing some context (although I have been reading other related issues) but I think that maybe this specific issue could be addressed within `dvclive` instead of (or in addition to) trying to find a solution as part of `dvc params`. 

To elaborate a bit:

The problem seems to be a good candidate to be solved with a python API and `dvclive` is a separate library focused on providing Python APIs so it doesn't have the "restrictions" of being language agnostic, like it is the case for `dvc`. 

In addition, it looks like the issue could be quite common in the training stages of a *ML Scenario* while also being, at the low level, different depending on the *ML repository/framework* used. 

In this case `Yolov5` uses `argparse` but other ML repositories/frameworks could use other options or even custom configuration files like it is the case for [detectron2](https://github.com/facebookresearch/detectron2) or [mmdetection](https://github.com/open-mmlab/mmdetection), just to give examples of frameworks focused on object detection models.

So, given that `dvclive` is intended to eventually [further expand it's integrations](https://github.com/iterative/dvclive#call-to-collaboration) with *ML frameworks/repositories*, maybe this `dvc params`<>`training args` link could be solved as part of each individual `dvclive<>ML framework/repository` integration (i.e. in this case a `dvclive<>yolov5`)? Does this make sense?

Thread on discord related (to some extent) :

https://discordapp.com/channels/485586884165107732/563406153334128681/865257349555027969
The full command (including command line arguments) are already part of the stage dependencies. Maybe we could look into parsing and showing the arguments (or at least the full command) as part of `dvc exp show`? These could potentially be seen as either implicit parameters or another type of dependency.

Edit: To further explain, if we had better support for showing the command line arguments as dependencies, it wouldn't be necessary to also list these same values in the params file.
Another idea: any templated values in the command (anything using `${}` syntax) could be automatically included as parameters. This at least avoids having to repeat values in `cmd` and `params` in `dvc.yaml`.
@dberenbaum, templated values are automatically included in `params show/diff`/`exp show` results.
```
# custom-params.yaml
epochs: 100
seed: 20170428

# dvc.yaml
vars:
- custom-params.yaml
stages:
  stage1:
    cmd: ./script.py --epochs ${epochs}
```

```console
$ dvc exp show --md --no-pager
| Experiment   | Created   | epochs   |
|--------------|-----------|----------|
| workspace    | -         | 100      |
| master       | 11:00 AM  | 100      |

$ python -c 'from dvc.repo import Repo; print(Repo().params.show())'
{'': {'data': defaultdict(<class 'dict'>, {'custom-params.yaml': {'data': {'epochs': 100}}})}}
If I can proffer another potential solution, if DVC could specify a format for not only the dvc.yaml but also yaml files for the `-d` and `-p ` arguments for `dvc run`. That way, quite complex dependencies and parameters could be created with utility functions by the helper classes/scripts, and re-used for individual stages, without having to create extremely verbose dvc.yaml files.
I think it also should support a simple array like this:
(Similar to `func(*[a,b,c,d])` in python or `[...array, new_time, new_item2]` in js.

> When we run `foreach .. do ...` for items in an array inside `dvc.yaml` , we might want to join results of each `do` into one. It would be nice to be able to unroll the array instead of adding each item separately.
> 
> Now I do this:
> 
> ```
> vars:
>   - datasets:
>      - one
>      - two
>      - three
> 
> stages:
>   prep:
>     foreach: ${datasets}
>     do:
>       cmd: python main.py prep ${item}
>       outs:
>        - data/processed/{item}
>   join:
>     cmd: python main.py merge ${datasets[0]} ${datasets[1]} ${datasets[2]}
>     deps:
>       - data/processed/${datasets[0]}
>       - data/processed/${datasets[1]}
>       - data/processed/${datasets[2]}
> ```
> 
> Obviously this leads to errors when we forget to update `join` stage after updating `datasets` variable.
> 
> We can have an option to unroll an array like this:
> 
> ```
> join:
>     cmd: python main.py merge ${...datasets}
>     deps:
>       -  data/processed/
> ```


Thanks, @Rusteam! Why do you change `deps` between your two examples? Is your data dependency the entire `data/processed/` directory or only the specified dataset paths? Your request may also be related to #7151 if you need to specify each dataset dependency independently.
well, it should only be the specified paths but I don't see how can I do it without listing each of them separately (unless i create another dir `data/processed/dataset`) and put all three there.

Patch:
diff --git a/dvc/config_schema.py b/dvc/config_schema.py
--- a/dvc/config_schema.py
+++ b/dvc/config_schema.py
@@ -280,4 +280,8 @@ class RelPath(str):
         "plots": str,
         "live": str,
     },
+    "parsing": {
+        "bool": All(Lower, Choices("store_true", "boolean_optional")),
+        "list": All(Lower, Choices("nargs", "append")),
+    },
 }
diff --git a/dvc/parsing/__init__.py b/dvc/parsing/__init__.py
--- a/dvc/parsing/__init__.py
+++ b/dvc/parsing/__init__.py
@@ -303,7 +303,7 @@ def _resolve(
     ) -> DictStr:
         try:
             return context.resolve(
-                value, skip_interpolation_checks=skip_checks
+                value, skip_interpolation_checks=skip_checks, key=key
             )
         except (ParseError, KeyNotInContext) as exc:
             format_and_raise(
diff --git a/dvc/parsing/context.py b/dvc/parsing/context.py
--- a/dvc/parsing/context.py
+++ b/dvc/parsing/context.py
@@ -18,6 +18,7 @@
     normalize_key,
     recurse,
     str_interpolate,
+    validate_value,
 )
 
 logger = logging.getLogger(__name__)
@@ -506,7 +507,7 @@ def set_temporarily(self, to_set: DictStr, reserve: bool = False):
                 self.data.pop(key, None)
 
     def resolve(
-        self, src, unwrap=True, skip_interpolation_checks=False
+        self, src, unwrap=True, skip_interpolation_checks=False, key=None
     ) -> Any:
         """Recursively resolves interpolation and returns resolved data.
 
@@ -522,10 +523,10 @@ def resolve(
         {'lst': [1, 2, 3]}
         """
         func = recurse(self.resolve_str)
-        return func(src, unwrap, skip_interpolation_checks)
+        return func(src, unwrap, skip_interpolation_checks, key)
 
     def resolve_str(
-        self, src: str, unwrap=True, skip_interpolation_checks=False
+        self, src: str, unwrap=True, skip_interpolation_checks=False, key=None
     ) -> str:
         """Resolves interpolated string to it's original value,
         or in case of multiple interpolations, a combined string.
@@ -543,10 +544,12 @@ def resolve_str(
             expr = get_expression(
                 matches[0], skip_checks=skip_interpolation_checks
             )
-            return self.select(expr, unwrap=unwrap)
+            value = self.select(expr, unwrap=unwrap)
+            validate_value(value, key)
+            return value
         # but not "${num} days"
         return str_interpolate(
-            src, matches, self, skip_checks=skip_interpolation_checks
+            src, matches, self, skip_checks=skip_interpolation_checks, key=key
         )
 
 
diff --git a/dvc/parsing/interpolate.py b/dvc/parsing/interpolate.py
--- a/dvc/parsing/interpolate.py
+++ b/dvc/parsing/interpolate.py
@@ -1,11 +1,12 @@
 import re
 import typing
-from collections.abc import Mapping
+from collections.abc import Iterable, Mapping
 from functools import singledispatch
 
 from funcy import memoize, rpartial
 
 from dvc.exceptions import DvcException
+from dvc.utils.flatten import flatten
 
 if typing.TYPE_CHECKING:
     from typing import List, Match
@@ -80,6 +81,45 @@ def _(obj: bool):
     return "true" if obj else "false"
 
 
+@to_str.register(dict)
+def _(obj: dict):
+    from dvc.config import Config
+
+    config = Config().get("parsing", {})
+
+    result = ""
+    for k, v in flatten(obj).items():
+
+        if isinstance(v, bool):
+            if v:
+                result += f"--{k} "
+            else:
+                if config.get("bool", "store_true") == "boolean_optional":
+                    result += f"--no-{k} "
+
+        elif isinstance(v, str):
+            result += f"--{k} '{v}' "
+
+        elif isinstance(v, Iterable):
+            for n, i in enumerate(v):
+                if isinstance(i, str):
+                    i = f"'{i}'"
+                elif isinstance(i, Iterable):
+                    raise ParseError(
+                        f"Cannot interpolate nested iterable in '{k}'"
+                    )
+
+                if config.get("list", "nargs") == "append":
+                    result += f"--{k} {i} "
+                else:
+                    result += f"{i} " if n > 0 else f"--{k} {i} "
+
+        else:
+            result += f"--{k} {v} "
+
+    return result.rstrip()
+
+
 def _format_exc_msg(exc: "ParseException"):
     from pyparsing import ParseException
 
@@ -148,23 +188,33 @@ def get_expression(match: "Match", skip_checks: bool = False):
     return inner if skip_checks else parse_expr(inner)
 
 
+def validate_value(value, key):
+    from .context import PRIMITIVES
+
+    not_primitive = value is not None and not isinstance(value, PRIMITIVES)
+    not_foreach = key is not None and "foreach" not in key
+    if not_primitive and not_foreach:
+        if isinstance(value, dict):
+            if key == "cmd":
+                return True
+        raise ParseError(
+            f"Cannot interpolate data of type '{type(value).__name__}'"
+        )
+
+
 def str_interpolate(
     template: str,
     matches: "List[Match]",
     context: "Context",
     skip_checks: bool = False,
+    key=None,
 ):
-    from .context import PRIMITIVES
-
     index, buf = 0, ""
     for match in matches:
         start, end = match.span(0)
         expr = get_expression(match, skip_checks=skip_checks)
         value = context.select(expr, unwrap=True)
-        if value is not None and not isinstance(value, PRIMITIVES):
-            raise ParseError(
-                f"Cannot interpolate data of type '{type(value).__name__}'"
-            )
+        validate_value(value, key)
         buf += template[index:start] + to_str(value)
         index = end
     buf += template[index:]


Test patch:
diff --git a/tests/func/parsing/test_errors.py b/tests/func/parsing/test_errors.py
--- a/tests/func/parsing/test_errors.py
+++ b/tests/func/parsing/test_errors.py
@@ -119,18 +119,34 @@ def test_wdir_failed_to_interpolate(tmp_dir, dvc, wdir, expected_msg):
 
 def test_interpolate_non_string(tmp_dir, dvc):
     definition = make_entry_definition(
-        tmp_dir, "build", {"cmd": "echo ${models}"}, Context(models={})
+        tmp_dir, "build", {"outs": "${models}"}, Context(models={})
     )
     with pytest.raises(ResolveError) as exc_info:
         definition.resolve()
 
     assert str(exc_info.value) == (
-        "failed to parse 'stages.build.cmd' in 'dvc.yaml':\n"
+        "failed to parse 'stages.build.outs' in 'dvc.yaml':\n"
         "Cannot interpolate data of type 'dict'"
     )
     assert definition.context == {"models": {}}
 
 
+def test_interpolate_nested_iterable(tmp_dir, dvc):
+    definition = make_entry_definition(
+        tmp_dir,
+        "build",
+        {"cmd": "echo ${models}"},
+        Context(models={"list": [1, [2, 3]]}),
+    )
+    with pytest.raises(ResolveError) as exc_info:
+        definition.resolve()
+
+    assert str(exc_info.value) == (
+        "failed to parse 'stages.build.cmd' in 'dvc.yaml':\n"
+        "Cannot interpolate nested iterable in 'list'"
+    )
+
+
 def test_partial_vars_doesnot_exist(tmp_dir, dvc):
     (tmp_dir / "test_params.yaml").dump({"sub1": "sub1", "sub2": "sub2"})
 
diff --git a/tests/func/parsing/test_interpolated_entry.py b/tests/func/parsing/test_interpolated_entry.py
--- a/tests/func/parsing/test_interpolated_entry.py
+++ b/tests/func/parsing/test_interpolated_entry.py
@@ -259,3 +259,59 @@ def test_vars_load_partial(tmp_dir, dvc, local, vars_):
         d["vars"] = vars_
     resolver = DataResolver(dvc, tmp_dir.fs_path, d)
     resolver.resolve()
+
+
+@pytest.mark.parametrize(
+    "bool_config, list_config",
+    [(None, None), ("store_true", "nargs"), ("boolean_optional", "append")],
+)
+def test_cmd_dict(tmp_dir, dvc, bool_config, list_config):
+    with dvc.config.edit() as conf:
+        if bool_config:
+            conf["parsing"]["bool"] = bool_config
+        if list_config:
+            conf["parsing"]["list"] = list_config
+
+    data = {
+        "dict": {
+            "foo": "foo",
+            "bar": 2,
+            "string": "spaced string",
+            "bool": True,
+            "bool-false": False,
+            "list": [1, 2, "foo"],
+            "nested": {"foo": "foo"},
+        }
+    }
+    (tmp_dir / DEFAULT_PARAMS_FILE).dump(data)
+    resolver = DataResolver(
+        dvc,
+        tmp_dir.fs_path,
+        {"stages": {"stage1": {"cmd": "python script.py ${dict}"}}},
+    )
+
+    if bool_config is None or bool_config == "store_true":
+        bool_resolved = " --bool"
+    else:
+        bool_resolved = " --bool --no-bool-false"
+
+    if list_config is None or list_config == "nargs":
+        list_resolved = " --list 1 2 'foo'"
+    else:
+        list_resolved = " --list 1 --list 2 --list 'foo'"
+
+    assert_stage_equal(
+        resolver.resolve(),
+        {
+            "stages": {
+                "stage1": {
+                    "cmd": "python script.py"
+                    " --foo 'foo' --bar 2"
+                    " --string 'spaced string'"
+                    f"{bool_resolved}"
+                    f"{list_resolved}"
+                    " --nested.foo 'foo'"
+                }
+            }
+        },
+    )