Repository: iterative/dvc

Issue:
Nested Interpolation / Expansions
It would be really nice to have the possibility to nest string interpolations / variables expansions, ex. interpolating in `dvc.yaml` a field of `params.yaml` that is itself an interpolation of another field...

For example: 

In `params.yaml`

```yaml
paths: 
  data: 
    root: path/to/data/dir
    raw: ${paths.data.root}/raw/
  scripts: 
    root: path/to/stages/dir
    load: ${paths.scripts.root}
 ```

and then `dvc.yaml`
 
```yaml
stages:
  load:
    cmd: python ${paths.scripts.load}
    outs: 
      - ${paths.raw}/filename.extension
```

This would really help in avoiding cumbersome & "error prone" hardcoding. 

Many thanks in advance 
Regards 
M


Discussion:
+1
@m-gris You are using the hydra integration, right? Does it not solve your use case, or you just want to simplify by not needing hydra?
@dberenbaum  
Yes I do use the hydra integration indeed, but, afaik, it does not allow to do what I'm trying to do, i.e reduce hardcoding to the bare minimum by composing expanded variables as much as possible.  

Here is a silly toy example that should suffice in illustrating what I have in mind: 

Let's say that in `config.yaml` I have the following: 

```yaml 
hydra_test:
    dir: 'raw'
    file: 'items.csv'
    path: ${hydra-test.dir}/${hydra_test.file}
```

The `path` key will indeed be properly parsed & evaluated if using OmegaConf in a python script. 

But if, in `dvc.yaml`, I try to do the following: 

```yaml 
stages:
    hydra_test:
        cmd: echo ${hydra_test.path}
```

It results in the following error: 

`/bin/bash: line 1: ${hydra_test.repo}/${hydra_test.file}: bad substitution`

It would actually be really nice if dvc could interpret / evaluate those. :) 


> The path key will indeed be properly parsed & evaluated if using OmegaConf in a python script.

This should be also working and the params.yaml should contain the interpolated value.
I will take a look to find out why it is not working

Resolving patch:
diff --git a/dvc/utils/hydra.py b/dvc/utils/hydra.py
--- a/dvc/utils/hydra.py
+++ b/dvc/utils/hydra.py
@@ -36,6 +36,8 @@ def compose_and_dump(
     with initialize_config_dir(config_dir, version_base=None):
         cfg = compose(config_name=config_name, overrides=overrides)
 
+    OmegaConf.resolve(cfg)
+
     suffix = Path(output_file).suffix.lower()
     if suffix not in [".yml", ".yaml"]:
         dumper = DUMPERS[suffix]


Test patch:
diff --git a/tests/func/utils/test_hydra.py b/tests/func/utils/test_hydra.py
--- a/tests/func/utils/test_hydra.py
+++ b/tests/func/utils/test_hydra.py
@@ -178,7 +178,7 @@ def test_compose_and_dump(tmp_dir, suffix, overrides, expected):
 
 
 def test_compose_and_dump_yaml_handles_string(tmp_dir):
-    """Regression test for 8583"""
+    """Regression test for https://github.com/iterative/dvc/issues/8583"""
     from dvc.utils.hydra import compose_and_dump
 
     config = tmp_dir / "conf" / "config.yaml"
@@ -189,6 +189,20 @@ def test_compose_and_dump_yaml_handles_string(tmp_dir):
     assert output_file.read_text() == "foo: 'no'\n"
 
 
+def test_compose_and_dump_resolves_interpolation(tmp_dir):
+    """Regression test for https://github.com/iterative/dvc/issues/9196"""
+    from dvc.utils.hydra import compose_and_dump
+
+    config = tmp_dir / "conf" / "config.yaml"
+    config.parent.mkdir()
+    config.dump({"data": {"root": "path/to/root", "raw": "${.root}/raw"}})
+    output_file = tmp_dir / "params.yaml"
+    compose_and_dump(output_file, str(config.parent), "config", [])
+    assert output_file.parse() == {
+        "data": {"root": "path/to/root", "raw": "path/to/root/raw"}
+    }
+
+
 @pytest.mark.parametrize(
     "overrides, expected",
     [