Repository: facebookresearch/hydra

Issue:
[Bug] Cannot perform a multirun when the strict option is enforced and the default yaml does not contain the config over which multirun performs a sweep
# 🐛 Bug

Cannot perform a multirun on the minimal example from hydra tutorial when the strict option is enforced and the default yaml does not contain the configuration over which multirun performs a sweep

## To reproduce

** Minimal Code/Config snippet to reproduce **

[hydra.zip](https://github.com/facebookresearch/hydra/files/3673714/hydra.zip)
unzip attached folder
$ cd hydra
$ python my_app.py -m exp=1,2

** Stack trace/error message **
```
(tutorial_env) dasneha@devfair0254:~/tutorials/hydra$ python my_app.py -m exp=1,2
Traceback (most recent call last):
  File "my_app.py", line 7, in <module>
    my_app()
  File "/private/home/dasneha/.conda/envs/tutorial_env/lib/python3.7/site-packages/hydra/main.py", line 25, in decorated_main
    strict,
  File "/private/home/dasneha/.conda/envs/tutorial_env/lib/python3.7/site-packages/hydra/_internal/utils.py", line 52, in run_hydra
    hydra.multirun(overrides=args.overrides)
  File "/private/home/dasneha/.conda/envs/tutorial_env/lib/python3.7/site-packages/hydra/_internal/hydra.py", line 113, in multirun
    cfg = self._load_config(overrides)
  File "/private/home/dasneha/.conda/envs/tutorial_env/lib/python3.7/site-packages/hydra/_internal/hydra.py", line 259, in _load_config
    cfg = self.config_loader.load_configuration(overrides)
  File "/private/home/dasneha/.conda/envs/tutorial_env/lib/python3.7/site-packages/hydra/_internal/config_loader.py", line 66, in load_configuration
    cfg.merge_with_dotlist(remaining_overrides)
  File "/private/home/dasneha/.conda/envs/tutorial_env/lib/python3.7/site-packages/omegaconf/config.py", line 240, in merge_with_dotlist
    self.update(key, value)
  File "/private/home/dasneha/.conda/envs/tutorial_env/lib/python3.7/site-packages/omegaconf/config.py", line 260, in update
    root[last] = value
  File "/private/home/dasneha/.conda/envs/tutorial_env/lib/python3.7/site-packages/omegaconf/dictconfig.py", line 28, in __setitem__
    raise KeyError("Accessing unknown key in a struct : {}".format(self.get_full_key(key)))
KeyError: 'Accessing unknown key in a struct : exp'
```
## Expected Behavior
multirun should work

## System information
- hydra-core==0.1.5
- Ubuntu 18.04.1 LTS




[Bug] Cannot perform a multirun when the strict option is enforced and the default yaml does not contain the config over which multirun performs a sweep
# 🐛 Bug

Cannot perform a multirun on the minimal example from hydra tutorial when the strict option is enforced and the default yaml does not contain the configuration over which multirun performs a sweep

## To reproduce

** Minimal Code/Config snippet to reproduce **

[hydra.zip](https://github.com/facebookresearch/hydra/files/3673714/hydra.zip)
unzip attached folder
$ cd hydra
$ python my_app.py -m exp=1,2

** Stack trace/error message **
```
(tutorial_env) dasneha@devfair0254:~/tutorials/hydra$ python my_app.py -m exp=1,2
Traceback (most recent call last):
  File "my_app.py", line 7, in <module>
    my_app()
  File "/private/home/dasneha/.conda/envs/tutorial_env/lib/python3.7/site-packages/hydra/main.py", line 25, in decorated_main
    strict,
  File "/private/home/dasneha/.conda/envs/tutorial_env/lib/python3.7/site-packages/hydra/_internal/utils.py", line 52, in run_hydra
    hydra.multirun(overrides=args.overrides)
  File "/private/home/dasneha/.conda/envs/tutorial_env/lib/python3.7/site-packages/hydra/_internal/hydra.py", line 113, in multirun
    cfg = self._load_config(overrides)
  File "/private/home/dasneha/.conda/envs/tutorial_env/lib/python3.7/site-packages/hydra/_internal/hydra.py", line 259, in _load_config
    cfg = self.config_loader.load_configuration(overrides)
  File "/private/home/dasneha/.conda/envs/tutorial_env/lib/python3.7/site-packages/hydra/_internal/config_loader.py", line 66, in load_configuration
    cfg.merge_with_dotlist(remaining_overrides)
  File "/private/home/dasneha/.conda/envs/tutorial_env/lib/python3.7/site-packages/omegaconf/config.py", line 240, in merge_with_dotlist
    self.update(key, value)
  File "/private/home/dasneha/.conda/envs/tutorial_env/lib/python3.7/site-packages/omegaconf/config.py", line 260, in update
    root[last] = value
  File "/private/home/dasneha/.conda/envs/tutorial_env/lib/python3.7/site-packages/omegaconf/dictconfig.py", line 28, in __setitem__
    raise KeyError("Accessing unknown key in a struct : {}".format(self.get_full_key(key)))
KeyError: 'Accessing unknown key in a struct : exp'
```
## Expected Behavior
multirun should work

## System information
- hydra-core==0.1.5
- Ubuntu 18.04.1 LTS






Discussion:
A Related issue: I get a similar error when I run eval "$(python my_app.py -sc install=bash)". This goes away if I change the mode to Non-Strict.


Traceback (most recent call last):
  File "my_app.py", line 7, in <module>
    my_app()
  File "/private/home/dasneha/.conda/envs/tutorial_env/lib/python3.7/site-packages/hydra/main.py", line 25, in decorated_main
    strict,
  File "/private/home/dasneha/.conda/envs/tutorial_env/lib/python3.7/site-packages/hydra/_internal/utils.py", line 56, in run_hydra
    hydra.shell_completion(overrides=args.overrides)
  File "/private/home/dasneha/.conda/envs/tutorial_env/lib/python3.7/site-packages/hydra/_internal/hydra.py", line 146, in shell_completion
    config = self._load_config(overrides)
  File "/private/home/dasneha/.conda/envs/tutorial_env/lib/python3.7/site-packages/hydra/_internal/hydra.py", line 259, in _load_config
    cfg = self.config_loader.load_configuration(overrides)
  File "/private/home/dasneha/.conda/envs/tutorial_env/lib/python3.7/site-packages/hydra/_internal/config_loader.py", line 66, in load_configuration
    cfg.merge_with_dotlist(remaining_overrides)
  File "/private/home/dasneha/.conda/envs/tutorial_env/lib/python3.7/site-packages/omegaconf/config.py", line 240, in merge_with_dotlist
    self.update(key, value)
  File "/private/home/dasneha/.conda/envs/tutorial_env/lib/python3.7/site-packages/omegaconf/config.py", line 260, in update
    root[last] = value
  File "/private/home/dasneha/.conda/envs/tutorial_env/lib/python3.7/site-packages/omegaconf/dictconfig.py", line 28, in __setitem__
    raise KeyError("Accessing unknown key in a struct : {}".format(self.get_full_key(key)))
KeyError: 'Accessing unknown key in a struct : install'
A Related issue: I get a similar error when I run eval "$(python my_app.py -sc install=bash)". This goes away if I change the mode to Non-Strict.


Traceback (most recent call last):
  File "my_app.py", line 7, in <module>
    my_app()
  File "/private/home/dasneha/.conda/envs/tutorial_env/lib/python3.7/site-packages/hydra/main.py", line 25, in decorated_main
    strict,
  File "/private/home/dasneha/.conda/envs/tutorial_env/lib/python3.7/site-packages/hydra/_internal/utils.py", line 56, in run_hydra
    hydra.shell_completion(overrides=args.overrides)
  File "/private/home/dasneha/.conda/envs/tutorial_env/lib/python3.7/site-packages/hydra/_internal/hydra.py", line 146, in shell_completion
    config = self._load_config(overrides)
  File "/private/home/dasneha/.conda/envs/tutorial_env/lib/python3.7/site-packages/hydra/_internal/hydra.py", line 259, in _load_config
    cfg = self.config_loader.load_configuration(overrides)
  File "/private/home/dasneha/.conda/envs/tutorial_env/lib/python3.7/site-packages/hydra/_internal/config_loader.py", line 66, in load_configuration
    cfg.merge_with_dotlist(remaining_overrides)
  File "/private/home/dasneha/.conda/envs/tutorial_env/lib/python3.7/site-packages/omegaconf/config.py", line 240, in merge_with_dotlist
    self.update(key, value)
  File "/private/home/dasneha/.conda/envs/tutorial_env/lib/python3.7/site-packages/omegaconf/config.py", line 260, in update
    root[last] = value
  File "/private/home/dasneha/.conda/envs/tutorial_env/lib/python3.7/site-packages/omegaconf/dictconfig.py", line 28, in __setitem__
    raise KeyError("Accessing unknown key in a struct : {}".format(self.get_full_key(key)))
KeyError: 'Accessing unknown key in a struct : install'

Resolving patch:
diff --git a/hydra/_internal/config_loader.py b/hydra/_internal/config_loader.py
--- a/hydra/_internal/config_loader.py
+++ b/hydra/_internal/config_loader.py
@@ -133,15 +133,15 @@ def _apply_defaults_overrides(overrides, defaults):
         for override in copy.deepcopy(overrides):
             key, value = override.split("=")
             if key in key_to_idx:
-                # Do not add sweep configs into defaults, those will be added to the sweep config
+                # Do not add multirun configs into defaults, those will be added to the multirun config
                 # after the list is broken into items
                 if "," not in value:
                     if value == "null":
                         del defaults[key_to_idx[key]]
                     else:
                         defaults[key_to_idx[key]][key] = value
-                    overrides.remove(override)
-                    consumed.append(override)
+                overrides.remove(override)
+                consumed.append(override)
         return consumed
 
     def _apply_free_defaults(self, defaults, overrides):
@@ -149,11 +149,12 @@ def _apply_free_defaults(self, defaults, overrides):
         for override in overrides:
             key, value = override.split("=")
             if self.exists_in_search_path(key):
-                # Do not add sweep configs into defaults, those will be added to the defaults
-                # during sweep when after list is broken into items
+                # Do not add multirun configs into defaults, those will be added to the defaults
+                # during the runs after list is broken into items
                 if "," not in value:
                     defaults.append({key: value})
-                    consumed.append(override)
+                overrides.remove(override)
+                consumed.append(override)
 
         return consumed
 
diff --git a/hydra/_internal/core_plugins/basic_launcher.py b/hydra/_internal/core_plugins/basic_launcher.py
--- a/hydra/_internal/core_plugins/basic_launcher.py
+++ b/hydra/_internal/core_plugins/basic_launcher.py
@@ -49,14 +49,12 @@ def launch(self, job_overrides):
                     sweep_config.hydra.overrides.task
                 )
             HydraConfig().set_config(sweep_config)
-
-            runs.append(
-                run_job(
-                    config=sweep_config,
-                    task_function=self.task_function,
-                    job_dir_key="hydra.sweep.dir",
-                    job_subdir_key="hydra.sweep.subdir",
-                )
+            ret = run_job(
+                config=sweep_config,
+                task_function=self.task_function,
+                job_dir_key="hydra.sweep.dir",
+                job_subdir_key="hydra.sweep.subdir",
             )
+            runs.append(ret)
             configure_log(self.config.hydra.hydra_logging, self.config.hydra.verbose)
         return runs
diff --git a/hydra/plugins/step_sweeper.py b/hydra/plugins/step_sweeper.py
--- a/hydra/plugins/step_sweeper.py
+++ b/hydra/plugins/step_sweeper.py
@@ -58,6 +58,7 @@ def sweep(self, arguments):
         while not self.is_done():
             batch = self.get_job_batch()
             results = self.launcher.launch(batch)
+            # TODO: use extend
             returns.append(results)
             self.update_results(results)
         return returns


Test patch:
diff --git a/tests/test_app/app_with_free_group.py b/tests/test_app/app_with_free_group.py
new file mode 100644
--- /dev/null
+++ b/tests/test_app/app_with_free_group.py
@@ -0,0 +1,11 @@
+# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved
+import hydra
+
+
+@hydra.main(config_path="config_with_free_group/config.yaml")
+def my_app(_cfg):
+    print(_cfg.pretty())
+
+
+if __name__ == "__main__":
+    my_app()
diff --git a/tests/test_app/config_with_free_group/config.yaml b/tests/test_app/config_with_free_group/config.yaml
new file mode 100644
--- /dev/null
+++ b/tests/test_app/config_with_free_group/config.yaml
@@ -0,0 +1,2 @@
+defaults:
+  - group: opt1
diff --git a/tests/test_app/config_with_free_group/free_group/opt1.yaml b/tests/test_app/config_with_free_group/free_group/opt1.yaml
new file mode 100644
--- /dev/null
+++ b/tests/test_app/config_with_free_group/free_group/opt1.yaml
@@ -0,0 +1 @@
+free_group_opt1: true
\ No newline at end of file
diff --git a/tests/test_app/config_with_free_group/free_group/opt2.yaml b/tests/test_app/config_with_free_group/free_group/opt2.yaml
new file mode 100644
--- /dev/null
+++ b/tests/test_app/config_with_free_group/free_group/opt2.yaml
@@ -0,0 +1 @@
+free_group_opt2: true
\ No newline at end of file
diff --git a/tests/test_app/config_with_free_group/group/opt1.yaml b/tests/test_app/config_with_free_group/group/opt1.yaml
new file mode 100644
--- /dev/null
+++ b/tests/test_app/config_with_free_group/group/opt1.yaml
@@ -0,0 +1 @@
+group_opt1: true
\ No newline at end of file
diff --git a/tests/test_app/config_with_free_group/group/opt2.yaml b/tests/test_app/config_with_free_group/group/opt2.yaml
new file mode 100644
--- /dev/null
+++ b/tests/test_app/config_with_free_group/group/opt2.yaml
@@ -0,0 +1 @@
+group_opt2: true
\ No newline at end of file
diff --git a/tests/test_app/sweep_cfg/__init__.py b/tests/test_app/sweep_cfg/__init__.py
deleted file mode 100644
--- a/tests/test_app/sweep_cfg/__init__.py
+++ /dev/null
@@ -1 +0,0 @@
-# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved
diff --git a/tests/test_app/sweep_cfg/hydra/launcher/__init__.py b/tests/test_app/sweep_cfg/hydra/launcher/__init__.py
deleted file mode 100644
--- a/tests/test_app/sweep_cfg/hydra/launcher/__init__.py
+++ /dev/null
@@ -1 +0,0 @@
-# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved
diff --git a/tests/test_hydra.py b/tests/test_hydra.py
--- a/tests/test_hydra.py
+++ b/tests/test_hydra.py
@@ -192,3 +192,26 @@ def test_short_module_name(tmpdir):
         }
     finally:
         chdir_hydra_root()
+
+
+@pytest.mark.parametrize(
+    "calling_file, calling_module",
+    [("tests/test_app/app_with_cfg.py", None), (None, "tests.test_app.app_with_cfg")],
+)
+@pytest.mark.parametrize("overrides", [["free_group=opt1,opt2"]])
+def test_multirun_with_free_override(
+    sweep_runner, calling_file, calling_module, overrides  # noqa: F811
+):
+    sweep = sweep_runner(
+        calling_file=calling_file,
+        calling_module=calling_module,
+        config_path="config_with_free_group/config.yaml",
+        overrides=overrides,
+        strict=True,
+    )
+    with sweep:
+        assert len(sweep.returns[0]) == 2
+        assert sweep.returns[0][0].overrides == ["free_group=opt1"]
+        assert sweep.returns[0][0].cfg == {"group_opt1": True, "free_group_opt1": True}
+        assert sweep.returns[0][1].overrides == ["free_group=opt2"]
+        assert sweep.returns[0][1].cfg == {"group_opt1": True, "free_group_opt2": True}