Repository: facebookresearch/hydra

Issue:
[Feature Request] Override loaded config file through a command line flag
# 🚀 Feature Request

<!-- A clear and concise description of the feature you are requesting -->

I want to pass yaml file instead of parameters like `foo.bar=1 ...` to override default configuration:

```bash
$ python app.py --yaml foo.yaml
```



Discussion:
Can this file be inside the search path (in the same directory as the config file you specify in @hydra.main() )?

If so, try:

```
conf/
  config.yaml
  experiment/
      exp1.yaml
```

and from the command line:
```
$ python foo.py experiment=exp1
```
This will override your config file with exp1.
> Can this file be inside the search path (in the same directory as the config file you specify in @hydra.main() )?

No.
I'm creating a cli with hydra, and this use case is for the user, so the config file cannot be placed inside `conf` directory.
The pattern of the modification  is so enormous that cannot be defined in advance.
The reason why I use hydra is that the configuration of my cli is a little bit complexed and hydra is suitable for my cli.
This app is closed source now, but will be opened before long. (After opened, I'll tell you that)
Yes it would be definitly helpful to be able to change the config_name from the CLI interface. Hydra sounds super helpful to simplify the architecture of some of my modules but the user should be able to redefine the config from scratch...
This is more involved than it sounds.
config_path is currently describing both the config search path **relative to the Python file with hydra.main**, and optionally a config file to load from the search path.
There are a few related issues open for allowing some control over the search path from the command line as well this one.
When loading a config file using some command line override, what does it do (if anything) to the search path?

This is also a user facing interface. which means it's not very forgiving to many changes. I want to get this right from the first time.

There are other higher priority issues I am working on. I will get to it.
This is a clean workaround that addresses most of the use cases:

In conf, create a directory called experiment:
```
conf\
  experiment\
    exp1.yaml
    exp2.yaml
```
Each one can specialize the generated config, for example exp1.yaml can be:

exp1.yaml:
```yaml
learning_rate: 0.1
batch_size: 32
```

Now from the command line:
`$ python foo.py experiment=exp1`

There are some use cases where an external config file can be useful, but this should cover a large number of the use cases.
Hi there,

I just wanted to +1 for this feature request. My use case is as follows. I have a complex configuration file with lots of details the user will not need to interact with, e.g. optimizers. There are elements, particularly regarding the data and augmentations, that all users will have to customize. 

Default `conf/config.yaml`: 
```
train
  lr: 0.0001
  optimizer: adam
dataset
  name: imagenet
  classes: 
    - tench
    - goldfish
    - great white shark
    - ... etc
augs
  resize: 224
  flip_lr: true
  flip_ud: false
```

`user_config.yaml`: 
```
dataset
  name: not hotdog
  classes:
    - hotdog
    - not hotdog
augs
  resize: 64
  flip_lr: true
  flip_ud: true
```

What I want is to be able to use the following syntax and have Hydra load the default configuration file first, and optionally override with the user's supplied config file:
` python train.py --config user_config.yaml`

Ideally, the user could also supply command line arguments which would take precedence over all. I'm not sure if this is too unique to my use case. I want the priority to be `command line args` > `user defaults from user_config.yaml` > `conf/config.yaml`.

Is there a way to do this currently with hydra? 

Please reach out on the chat.
I'm currently using a workaround for a similar case. 

Default **conf/config.yaml**:
```yaml
user_config: # path to config file to update.
train:
  lr: 0.0001
  optimizer: adam
dataset:
  name: imagenet
  classes: 
    - tench
    ...
```
Then, in the `train.py` you can override hydra config object with [`OmegaConf.merge`](https://omegaconf.readthedocs.io/en/1.4_branch/usage.html#merging-configurations) function when `user_config` is given.
```python
if config.user_config is not None:
    user_config = OmegaConf.load(config.user_config)
    config = OmegaConf.merge(config, user_config)
```

Now, you can use `python train.py user_config=user_config.yaml`
This comment goes in the direction of @jbohnslav, I think @tamuhey 's question can, but does not have to be connected with it because in this case I assume you can have a userconfig inside the search path. 

```
├── conf
│   ├── config_default.yaml
│   ├── cfgs
│   │   ├── base.yaml
│   │   └── userconfig.yaml
└── my_app.py
```

`config_default.yaml`:
```
defaults:
  - cfgs: base
  - cfgs: userconfig
```
`base.yaml`:
```
db:
  first_param: 1
```
`userconfig.yaml`:
```
db:
  userdefined_param: 2
```

`my_app.py`
```
import hydra
from omegaconf import DictConfig


@hydra.main(config_path="conf/config_default.yaml")
def my_app(cfg: DictConfig) -> None:
    print(cfg.pretty())

if __name__ == "__main__":
    my_app()
```


And the output is:

> db:
  first_param: 1
  userdefined_param: 2

For `multirun`, I added `userconfig2.yaml` inside cfgs:
```
db:
  anotherParam: 2
```
Then start via: 
`python my_app.py cfgs=userconfig,userconfig2 -m`
>[2020-05-16 12:04:38,165][HYDRA] Sweep output dir : multirun/2020-05-16/12-04-38
[2020-05-16 12:04:38,166][HYDRA] Launching 2 jobs locally
[2020-05-16 12:04:38,166][HYDRA] 	#0 : cfgs=userconfig
db:
  first_param: 1
  userdefined_param: 2

>[2020-05-16 12:04:38,246][HYDRA] 	#1 : cfgs=userconfig2
db:
  anotherParam: 2
  first_param: 1


I do not know what happens here exactly but as long as it works.. 
I just found that it is sufficient to have the following main config file `config_default.yaml`: 

```
defaults:
  - cfgs: base
  - cfgs: base
```

Then generally, its base and you can overwrite it on the commandline. 

Still, I do not know if there are unwanted side effects.
@gunthergl, I mentioned this approach in multiple comments above.
This is also documented in the tutorial to some extent [here](https://hydra.cc/docs/tutorial/config_groups).
> #  Feature Request
> I want to pass yaml file instead of parameters like `foo.bar=1 ...` to override default configuration:
> 
> ```shell
> $ python app.py --yaml foo.yaml
> ```

Hi,
I believe the requested feature is a must-have.

By the way, I suggest this workaround:
```
def main(cfg: DictConfig) -> None:
    print(cfg.pretty())

if __name__ == "__main__":
    config_path = "./conf/default.yaml"

    if len(sys.argv) > 1 and sys.argv[1].startswith("config="):
        config_path = sys.argv[1].split("=")[-1]
        sys.argv.pop(1)

    main_wrapper = hydra.main(config_path, strict=True)
    main_wrapper(main)()
```

This way you can specify your config file by adding `config=somefile.yaml` at the beginning of your command.
This requested feature is also a will-have.
Patience.

Resolving patch:
diff --git a/hydra/_internal/utils.py b/hydra/_internal/utils.py
--- a/hydra/_internal/utils.py
+++ b/hydra/_internal/utils.py
@@ -209,6 +209,13 @@ def run_hydra(
 
     from .hydra import Hydra
 
+    args = args_parser.parse_args()
+    if args.config_name is not None:
+        config_name = args.config_name
+
+    if args.config_path is not None:
+        config_path = args.config_path
+
     calling_file, calling_module = detect_calling_file_or_module_from_task_function(
         task_function
     )
@@ -224,7 +231,6 @@ def run_hydra(
         task_name=task_name, config_search_path=search_path, strict=strict
     )
     try:
-        args = args_parser.parse_args()
         if args.help:
             hydra.app_help(config_name=config_name, args_parser=args_parser, args=args)
             sys.exit(0)
@@ -341,6 +347,18 @@ def get_args_parser() -> argparse.ArgumentParser:
         help=f"Install or Uninstall shell completion:\n{_get_completion_help()}",
     )
 
+    parser.add_argument(
+        "--config_path",
+        "-cp",
+        help="""Overrides the config_path specified in hydra.main().
+                    The config_path is relative to the Python file declaring @hydra.main()""",
+    )
+
+    parser.add_argument(
+        "--config_name",
+        "-cn",
+        help="Overrides the config_name specified in hydra.main()",
+    )
     return parser
 
 


Test patch:
diff --git a/tests/test_apps/app_with_multiple_config_dirs/__init__.py b/tests/test_apps/app_with_multiple_config_dirs/__init__.py
new file mode 100644
--- /dev/null
+++ b/tests/test_apps/app_with_multiple_config_dirs/__init__.py
@@ -0,0 +1 @@
+# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved
diff --git a/tests/test_apps/app_with_multiple_config_dirs/dir1/__init__.py b/tests/test_apps/app_with_multiple_config_dirs/dir1/__init__.py
new file mode 100644
--- /dev/null
+++ b/tests/test_apps/app_with_multiple_config_dirs/dir1/__init__.py
@@ -0,0 +1 @@
+# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved
diff --git a/tests/test_apps/app_with_multiple_config_dirs/dir1/cfg1.yaml b/tests/test_apps/app_with_multiple_config_dirs/dir1/cfg1.yaml
new file mode 100644
--- /dev/null
+++ b/tests/test_apps/app_with_multiple_config_dirs/dir1/cfg1.yaml
@@ -0,0 +1 @@
+dir1_cfg1: true
\ No newline at end of file
diff --git a/tests/test_apps/app_with_multiple_config_dirs/dir1/cfg2.yaml b/tests/test_apps/app_with_multiple_config_dirs/dir1/cfg2.yaml
new file mode 100644
--- /dev/null
+++ b/tests/test_apps/app_with_multiple_config_dirs/dir1/cfg2.yaml
@@ -0,0 +1 @@
+dir1_cfg2: true
\ No newline at end of file
diff --git a/tests/test_apps/app_with_multiple_config_dirs/dir2/__init__.py b/tests/test_apps/app_with_multiple_config_dirs/dir2/__init__.py
new file mode 100644
--- /dev/null
+++ b/tests/test_apps/app_with_multiple_config_dirs/dir2/__init__.py
@@ -0,0 +1 @@
+# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved
diff --git a/tests/test_apps/app_with_multiple_config_dirs/dir2/cfg1.yaml b/tests/test_apps/app_with_multiple_config_dirs/dir2/cfg1.yaml
new file mode 100644
--- /dev/null
+++ b/tests/test_apps/app_with_multiple_config_dirs/dir2/cfg1.yaml
@@ -0,0 +1 @@
+dir2_cfg1: true
\ No newline at end of file
diff --git a/tests/test_apps/app_with_multiple_config_dirs/dir2/cfg2.yaml b/tests/test_apps/app_with_multiple_config_dirs/dir2/cfg2.yaml
new file mode 100644
--- /dev/null
+++ b/tests/test_apps/app_with_multiple_config_dirs/dir2/cfg2.yaml
@@ -0,0 +1 @@
+dir2_cfg2: true
\ No newline at end of file
diff --git a/tests/test_apps/app_with_multiple_config_dirs/my_app.py b/tests/test_apps/app_with_multiple_config_dirs/my_app.py
new file mode 100644
--- /dev/null
+++ b/tests/test_apps/app_with_multiple_config_dirs/my_app.py
@@ -0,0 +1,13 @@
+# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved
+from omegaconf import DictConfig
+
+import hydra
+
+
+@hydra.main()
+def my_app(cfg: DictConfig) -> None:
+    print(cfg.pretty())
+
+
+if __name__ == "__main__":
+    my_app()
diff --git a/tests/test_hydra.py b/tests/test_hydra.py
--- a/tests/test_hydra.py
+++ b/tests/test_hydra.py
@@ -553,6 +553,9 @@ def test_sweep_complex_defaults(
     Fish - Uninstall:
     python my_app.py -sc uninstall=fish | source
 
+--config_path,-cp : Overrides the config_path specified in hydra.main().
+                    The config_path is relative to the Python file declaring @hydra.main()
+--config_name,-cn : Overrides the config_name specified in hydra.main()
 Overrides : Any key=value arguments to override config values (use dots for.nested=overrides)
 """,
             id="overriding_help_template:$FLAGS_HELP",
@@ -597,6 +600,9 @@ def test_sweep_complex_defaults(
     Fish - Uninstall:
     python my_app.py -sc uninstall=fish | source
 
+--config_path,-cp : Overrides the config_path specified in hydra.main().
+                    The config_path is relative to the Python file declaring @hydra.main()
+--config_name,-cn : Overrides the config_name specified in hydra.main()
 Overrides : Any key=value arguments to override config values (use dots for.nested=overrides)
 """,
             id="overriding_hydra_help_template:$FLAGS_HELP",
@@ -731,3 +737,22 @@ def test_override_with_invalid_group_choice(
             overrides=[f"db={override}"],
         ):
             ...
+
+
+@pytest.mark.parametrize("config_path", ["dir1", "dir2"])  # type: ignore
+@pytest.mark.parametrize("config_name", ["cfg1", "cfg2"])  # type: ignore
+def test_config_name_and_path_overrides(
+    tmpdir: Path, config_path: str, config_name: str
+) -> None:
+    cmd = [
+        sys.executable,
+        "tests/test_apps/app_with_multiple_config_dirs/my_app.py",
+        "hydra.run.dir=" + str(tmpdir),
+        f"--config_name={config_name}",
+        f"--config_path={config_path}",
+    ]
+    print(" ".join(cmd))
+    result = str(subprocess.check_output(cmd).decode("utf-8")).strip()
+    # normalize newlines on Windows to make testing easier
+    result = result.replace("\r\n", "\n")
+    assert result == f"{config_path}_{config_name}: true"