Repository: facebookresearch/hydra

Issue:
dir interpolation from hydra into job config is not working
I can set the `second_dir`  from `base_dir`, but `output_subdir` can not in this case.
```
experiment:
  base_dir: ???
  second_dir: ${experiment.base_dir}
hydra:
  output_subdir: ${experiment.base_dir}/.hydra
```
`KeyError: "str interpolation key 'experiment.base_dir' not found"`

Is there any way I can change the hydra config based on current .yaml or some alternately way?  
 Thanks!


Discussion:
I need to take a look at why this is failing, but a more basic question is what are you trying to achieve?
what would you like your output directory structure to look like?
Since I don't want to change the current working directory, I override the `hydra.run.dir` to `.`. 
Then I want to keep output .hydra and try to place it to the experiment directory (i.e. base_dir above).
The motivation is simple, just want to keep the configuration for each experiment. 
The result I expected: `python myapp.py experiment.base_dir=experiment/exp1`
```
+---main
|   +---config
|   |   +---hydra
|   |   \---other
|   \---experiment
|       +---exp1
|       |   +---.hydra
|       |   \---result
|       \---exp2
|           +---.hydra
|           \---result
```
I know changing the working directory is unusual, if what you are describing is pretty easy to do if you did change it and just write to the current working directory.
Your use case sounds pretty typical for research. What doesn't work for well for you if allow Hydra to change the current working directory?
My program is sensitive to the original directory structure. But it would not be a problem to allow Hydra to change the cwd. In this case, I can switch the `hydra.run.dir` to `experiment/exp1` and 
run: `python myapp.py experiment.base_dir=.` to achieve the expected result.
Just wondering if it can set by the interpolation way. 😃 
I see.
Looking at this, this is a bug.
You should be able to do it after I fix it.
Before your app launches, the config for both it and hydra are int he came config object.
at that  point it's possible to have interpolation between your own config and hydra and the other way around.
Once your app launches, it only sees it's own config and at that point it is not (currently) possible to interpolate into Hydra's config.

I will fix this soon.

Resolving patch:
diff --git a/hydra/plugins/common/utils.py b/hydra/plugins/common/utils.py
--- a/hydra/plugins/common/utils.py
+++ b/hydra/plugins/common/utils.py
@@ -73,8 +73,6 @@ def run_job(config, task_function, job_dir_key, job_subdir_key):
         ret = JobReturn()
         ret.working_dir = working_dir
         task_cfg = copy.deepcopy(config)
-        # TODO: update this after https://github.com/omry/omegaconf/issues/42 is resolved
-        hydra_cfg = OmegaConf.create({"hydra": task_cfg["hydra"]})
         del task_cfg["hydra"]
         ret.cfg = task_cfg
         ret.hydra_cfg = copy.deepcopy(HydraConfig())
@@ -82,9 +80,13 @@ def run_job(config, task_function, job_dir_key, job_subdir_key):
         # handle output directories here
         Path(str(working_dir)).mkdir(parents=True, exist_ok=True)
         os.chdir(working_dir)
-        hydra_output = Path(hydra_cfg.hydra.output_subdir)
+        hydra_output = Path(config.hydra.output_subdir)
+
+        configure_log(config.hydra.job_logging, config.hydra.verbose)
+
+        # TODO: update this after https://github.com/omry/omegaconf/issues/42 is resolved
+        hydra_cfg = OmegaConf.create({"hydra": config["hydra"]})
 
-        configure_log(hydra_cfg.hydra.job_logging, hydra_cfg.hydra.verbose)
         _save_config(task_cfg, "config.yaml", hydra_output)
         _save_config(hydra_cfg, "hydra.yaml", hydra_output)
         _save_config(config.hydra.overrides.task, "overrides.yaml", hydra_output)


Test patch:
diff --git a/tests/test_apps/interpolating_dir_hydra_to_app/__init__.py b/tests/test_apps/interpolating_dir_hydra_to_app/__init__.py
new file mode 100644
--- /dev/null
+++ b/tests/test_apps/interpolating_dir_hydra_to_app/__init__.py
@@ -0,0 +1 @@
+# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved
diff --git a/tests/test_apps/interpolating_dir_hydra_to_app/config.yaml b/tests/test_apps/interpolating_dir_hydra_to_app/config.yaml
new file mode 100644
--- /dev/null
+++ b/tests/test_apps/interpolating_dir_hydra_to_app/config.yaml
@@ -0,0 +1,5 @@
+experiment:
+  base_dir: ???
+  second_dir: ${experiment.base_dir}
+hydra:
+  output_subdir: ${experiment.base_dir}/.hydra
\ No newline at end of file
diff --git a/tests/test_apps/interpolating_dir_hydra_to_app/my_app.py b/tests/test_apps/interpolating_dir_hydra_to_app/my_app.py
new file mode 100644
--- /dev/null
+++ b/tests/test_apps/interpolating_dir_hydra_to_app/my_app.py
@@ -0,0 +1,11 @@
+# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved
+import hydra
+
+
+@hydra.main(config_path="config.yaml")
+def my_app(_):
+    pass
+
+
+if __name__ == "__main__":
+    my_app()
diff --git a/tests/test_hydra.py b/tests/test_hydra.py
--- a/tests/test_hydra.py
+++ b/tests/test_hydra.py
@@ -1,6 +1,7 @@
 # Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved
 import os
 import sys
+from hydra._internal.pathlib import Path
 
 import pytest
 import subprocess
@@ -318,3 +319,24 @@ def test_help(tmpdir, script, overrides, expected):
     # normalize newlines on Windows to make testing easier
     result = result.replace("\r\n", "\n")
     assert result == expected
+
+
+@pytest.mark.parametrize(
+    "calling_file, calling_module",
+    [
+        ("tests/test_apps/interpolating_dir_hydra_to_app/my_app.py", None),
+        (None, "tests.test_apps.interpolating_dir_hydra_to_app.my_app"),
+    ],
+)
+def test_interpolating_dir_hydra_to_app(
+    task_runner, calling_file, calling_module  # noqa: F811
+):
+    basedir = "foo"
+    with task_runner(
+        calling_file=calling_file,
+        calling_module=calling_module,
+        config_path="config.yaml",
+        overrides=["experiment.base_dir=" + basedir],
+    ) as task:
+        path = Path(task.temp_dir) / basedir
+        assert path.exists()