Repository: facebookresearch/hydra

Issue:
Allow configuring the search path via the config file
Design doc outlining consideration and possible implementations:

https://docs.google.com/document/d/1Dx77SThg1ugnGHvZ8Dq1FQusXDvGn0tyyWBeJ3nXvro/edit?usp=sharing


Discussion:
Not sure if this is the feature I'm waiting for, but I want my users to install my app and easily point to a new config directory that overrides my defaults. My app is based on the hydra_app_example. Please advise.
Ok, I think I figured it out. First add a default search path to config.yaml:

```yaml
search_path: config.yaml
```

Then merge it using OmegaConf

```python
import hydra
from omegaconf import OmegaConf

@hydra.main(config_path='conf/config.yaml', strict = False)
def main(cfg):
    if cfg.search_path is not None:
        override_path = hydra.utils.to_absolute_path(cfg.search_path)
        override_conf = OmegaConf.load(override_path)
        cfg = OmegaConf.merge(cfg, override_conf)
```
Now the user can run your app from any directory containing a config.yaml, or specify their own:
```console
myapp search_path=path/to/user/config.yaml
```
That's a pretty good workaround. I would call this something like config_override and not search_path, as search path is an internal concept in Hydra that will eventually be exposed.

Asher, can you share a bit about your project? I am curious about how people are using Hydra and for what.
@omry It's for a project called Kamodo, which is a functional, publication-focused API for space weather models and data (though it can be used for much more!). Kamodo handles function composition, unit conversion, and automatic plot generation via function inspection, which should minimize the effort needed to create analysis pipelines and perform data-model comparisons. Kamodo is built with sympy and plotly.

Most of our science users are comfortable with command-line but are inexperienced with modern programing languages. The CLI for kamodo will allow them to configure their data sources and any equations they want to use for post processing and quickly generate plots. Our users will often test different models or data sources for science relevance and will want to easily reproduce those results, so Hydra's ability to compose several overlapping configurations and have timestamped output directories is perfect for our use case. 

Github - https://github.com/nasa/Kamodo
Official site - https://ccmc.gsfc.nasa.gov/Kamodo/

I'm also considering Hydra for my other project Hourly, which uses git "clock in" and "clock out" messages to track hours worked, generate time sheets, and produce work charts. Since I started freelancing, I've logged at least 1000 sessions across 5 different projects using Hourly.

Currently, Hourly's cli is based on click, but I need to provide default functionality specific to a particular repo (to ignore errant clock-in/outs, etc). I'd also like to generate time sheets from related projects, so config composition would come in handy. Finally, I'm usually the only one who uses hourly, but eventually I'll need to iterate over all contributors to a project (assuming I get more users!).

Project page - https://asherp.github.io/hourly/index.html
Github - https://github.com/asherp/hourly
Kamodo looks awesome, and I am really happy to start seeing serious projects that are making use of Hydra.

A few notes:
* Can you add Hydra as a dependency of Kamodo (in setup.py/cfg or requirements.txt). this will allow GitHub to detect it as a project using Hydra.
* Did you see the new [compose API](https://hydra.cc/docs/advanced/compose_api/)? it lets you compose configs with Hydra anywhere, not just in the main. This is specifically useful for Jupyter notebooks. [here is an example](https://github.com/facebookresearch/hydra/tree/master/examples/notebook) (you can launch the binder to play with it online).
* Take a look at [Pytorch](https://pytorch.org/), it might be very useful for Kamodo as an alternative to numpy. It allows highly optimized vector/matrix operations for both CPU and GPU, as well as automatic differentiation.
* Hourly looks like a nice and useful project for freelancers, it would be great if you use Hydra there as well, and I would love to know about your experience porting from Click to Hydra. Click is super popular but I think Hydra is much more powerful and it can probably be used as a replacement for Click for most use cases.
* I bet NASA got some serious compute clusters. While there are no official public plugins for it yet, we are using Hydra at FAIR to launch to our internal cluster. Hydra has a plugin mechanism that can support any cluster (including private clusters with non public APIs).
I would be happy to guide you in writing a plugin that will allow you to use Hydra to launch to your internal cluster from the command line.

I am considering creating a "Who is using Hydra" section in the docs, would you agree to write a short testimonial saying why you find it useful?


One more thing:
As a framework, Kamodo might want to provide some configuration to its users.
For example, some default configurations to instantiate commonly used models etc.
The best way to do it with Hydra is to add those configs to the search path.

You can do it programmatically and automatically via the search plugin API.
I don't yet have a super clean example for this in [hydra/plugins/examples](https://github.com/facebookresearch/hydra/tree/master/plugins/examples), but the colorlog plugin comes very close.
Essentially this is all its doing. check it out [here](https://github.com/facebookresearch/hydra/tree/master/plugins/hydra_colorlog). 
I have the similar use cases as @asherp . 

- The major interface of my program is in binary form, most users don't write any python codes.
- The default value is defined in `structured config` rather than a default config file
- Users write their  own config file in their work dir, and runs through `python -m xxx.main`

@omry 
I think the current version *1.0rc* supports specify `config_path`, but the path is **resolved relative** to the `main.py` file, so I have to specify absolute path to make it work.

Can we resolve the config_path relative to the $pwd, which makes more sense in CLI cases
You can probably achieve that now via a config searchpath plugin.
[example](https://github.com/facebookresearch/hydra/tree/master/plugins/examples/example_searchpath_plugin).
by the way, take a look at the [Application Packaging](https://hydra.cc/docs/next/advanced/app_packaging) page.
1. You CAN use config files with your application even if it's installed.
2. You can get a native command to execute it instead of the clunky python -m module.
I have a related use case I'm not sure can be supported easily right now:

I would like to have an application coming with its own config file (packaged with the app, not to be edited), that would also let the user override some parameters through their own config file (stored in the directory where they use the app).

I haven't looked closely yet at the config search path plugin, but reading the doc it sounds like it's going to stop at the first matching file (instead of merging configurations from multiple files).
It looks like I could use the compose API but that would require some extra manual work.

Just curious if there's an option I'm not seeing, or something I misunderstood?
> I have a related use case I'm not sure can be supported easily right now:
I would like to have an application coming with its own config file (packaged with the app, not to be edited), that would also let the user override some parameters through their own config file (stored in the directory where they use the app).
I haven't looked closely yet at the config search path plugin, but reading the doc it sounds like it's going to stop at the first matching file (instead of merging configurations from multiple files).
It looks like I could use the compose API but that would require some extra manual work.
Just curious if there's an option I'm not seeing, or something I misunderstood?



It seems like something Hydra may already support? As an example, all of Hydra's plugin gets their config added by SearchPathPlugin, example here: https://github.com/facebookresearch/hydra/blob/master/plugins/hydra_colorlog/hydra_plugins/hydra_colorlog/colorlog.py#L6-L9
I imagine in your case you can create multiple SearchPathPlugins pointing all the configs you are interested in and have Hydra merge them all.
It would be helpful if you can provide a minimal example of what the config structure would look like, that way I may better help you :) 

> I imagine in your case you can create multiple SearchPathPlugins pointing all the configs you are interested in and have Hydra merge them all.

Maybe -- I haven't looked closely at it, the reason why I thought it wouldn't work is because the doc says "When a config is requested, The first matching config in the search path is used", suggesting that there is no merging (just use whichever is found first). But I may not be understanding properly.

As an example I may want to have a directory structure like this:

```
├── my_app
│   └── config
│       └── config.yaml
└── user_code
    └── .my_app
        └── config.yaml
```
And I would like both config files to be merged (in my case the config found under "user_code" would only modify keys already existing in the "my_app" config, not add new keys).

I managed to make it work with the Compose API but (1) I had to drop `@hydra.main()`, and (2) it's a bit cumbersome because I need to declare my defaults list in the "user_code" config even if I would like to reuse the one from "my_app". So overall not so convenient...
Configuring the defaults list (the topic of this issue) has little to do with your question.

First of all, if you have two configs in the same name and the same group, only one will be visible.
This is not changing (and in fact enabling replacing an existing be prepending a searchpath containing a config with the same name.

In 1.0, you only get one defaults list - in the primary config.
That primary config can refer to a file from your app:

```
library/conf
  app_config.yaml
  ...

user_app/conf
  user_overrides.yaml # contains things the user want to override in app_config.
  user_conf.yaml 
    defaults:
      - app_config
      - user_overrides  # this is needed by default the composition order is placing this file before things in the defaults list.
```

The chunkiness in this solution will be addressed once recursive default supports coming in 1.1.
Refer to [this](https://docs.google.com/document/d/15z3IPmTOS4E6Xxv8RUFExuXi1WXjQs0DqF-39dnN_kc/edit?ts=5fa0df75#) design doc to see what's coming. Feedback is welcome.


> Configuring the defaults list (the topic of this issue) has little to do with your question.

It may not have been what you had in mind when you created this issue, but https://github.com/facebookresearch/hydra/issues/274#issuecomment-568036695 is pretty close to what I want to achieve.

> The chunkiness in this solution will be addressed once recursive default supports coming in 1.1.
> Refer to [this](https://docs.google.com/document/d/15z3IPmTOS4E6Xxv8RUFExuXi1WXjQs0DqF-39dnN_kc/edit?ts=5fa0df75#) design doc to see what's coming. Feedback is welcome.

I had a look, I may not understand it all, but I'm not sure it would work for my use case.

To make things more concrete, let's take this example from Hydra docs: https://hydra.cc/docs/next/tutorials/structured_config/static_schema
Let's say I have a packaged app with config files as shown in this example. I would like to let a user of my app create their own config file(s) to override only some of these settings. For instance they could create a file `.my_app/config/db/staging.yaml` that would contain:

```yaml
host: mysql007.staging
```

And then by running `python /path/to/my_app.py --config-dir .my_app/config`, the app would run with the user-specified host when using the staging db, instead of the default one.

The reason why I'm doubtful about recursive defaults enabling this is because of this example in the doc you shared:

```yaml
defaults:
 - d2/model: resnet50  # model from an installed framework
 - _self_
 - db: mysql    # This will override the value provided by the framework

d2:
  model:
    num_layers: 100
```

Here, `num_layers` will be set to 100 regardless of which model is used, but what if I want to set it to 100 for resnet50 and to 200 for resnet101?
Just an additional thought: if there's a way to do it via config files it'd make sense to also be able to do it via a command line override. Something like:
```
python /path/to/my_app.py db/staging.host=mysql007.staging
```
> The reason why I'm doubtful about recursive defaults enabling this is because of this example in the doc you shared:
That example is just one of many ways to use recursive defaults.

Here is another example from the docs of the next version (both the logic this is documenting and the docs themselves are work in progress):
https://hydra.cc/docs/next/advanced/defaults_list#config-inheritance-via-composition

Please move the discussion to a dedicated issue.
It has nothing at all to do with this issue.
> Please move the discussion to a dedicated issue.
> It has nothing at all to do with this issue.

Sorry for the hijack :) Thanks for the pointers, I think I have a better understanding now of how things currently are and how they will be in the future. I believe I can solve my problem with a slightly different approach which makes more sense in my situation, so I won't open another issue.
@omry After reading the [design doc](https://docs.google.com/document/d/15z3IPmTOS4E6Xxv8RUFExuXi1WXjQs0DqF-39dnN_kc/edit?ts=5fa0df75), especially the new feature `__self__` which changes the default semantics of composition order, I think it's more user-friendly. For example, dl researchers can easily configure their experiments using [this pattern](https://github.com/facebookresearch/hydra/blob/master/website/docs/patterns/configuring_experiments.md)

My question related to this proposal is, how can we combine that with allowing users to provide another config file/group when hydra-based framework is installed.

For example,  we have some configuration files installed with our hydra app, and the default configuration is:
```yaml title="config.yaml"
defaults:
  - dataset: mnist
  - model: vanilla
  - trainer: basic_cv_trainer
```

When the user has installed our framework, he/she can follow the [doc](https://github.com/facebookresearch/hydra/blob/master/website/docs/advanced/packaging.md) to run the framework with pre-defined configurations or providing override command line arguments.

Now the user wants to arrange his/her experiments in his own repo using overriding configs and the structure is like this:
```
.
├── cv
│   ├── alexnet
│   │   └── configs
│   │       ├── big_network.yaml
│   │       ├── medium_network.yaml
│   │       └── small_network.yaml
│   └── vgg
│       ├── big_network.yaml
│       ├── medium_network.yaml
│       └── small_network.yaml
└── nlp
    └── bert
        ├── version_1.yaml
        ├── version_2.yaml
        └── version_3.yaml
```
Here the single yaml file can be a folder to override the default settings.

According to what I learned [here](https://github.com/facebookresearch/hydra/blob/master/website/docs/patterns/configuring_experiments.md), the overriding yaml file (nlp/bert/version_1.yaml) should be like this:
```yaml 
# @package _global_
default:
  - overrider /dataset: swag_dataset
  - override /model: bert
  - override /trainer: bert_trainer
 
model:
  bert:
      d_model: 384
```

How can I select different overriding configs when running different tasks? Maybe add that config to the search path?
Hi @npuichigo!

> @omry After reading the [design doc](https://docs.google.com/document/d/15z3IPmTOS4E6Xxv8RUFExuXi1WXjQs0DqF-39dnN_kc/edit?ts=5fa0df75), especially the new feature `__self__` which changes the default semantics of composition order, I think it's more user-friendly. For example, dl researchers can easily configure their experiments using [this pattern](https://github.com/facebookresearch/hydra/blob/master/website/docs/patterns/configuring_experiments.md)

That design doc is not the latest one, [this](https://docs.google.com/document/d/1k3dgaWkH_7wLx6gI84GSkoi8rocJ6NRhaVok7Zr_62k/edit#) is. :).
Specifically, the design if `_self_` did not change between the docs.
There is no need to refer to the design doc though, the new defaults list is documented [here](https://hydra.cc/docs/next/advanced/defaults_list).


> My question related to this proposal is, how can we combine that with allowing users to provide another config file/group when hydra-based framework is installed.

> How can I select different overriding configs when running different tasks? Maybe add that config to the search path?

The Config search path is similar to the PYTHONPATH or the Java CLASSPATH.
There is some minimal (and insufficient) documentation of it [here](https://hydra.cc/docs/next/advanced/search_path).

Manipulation of the config search path should be done in order to add new roots to the search path.

It seems like your question is more about how to use configs that are in arbitrary config groups.

Please go ahead and read the following pages:
- [The Defaults List](https://hydra.cc/docs/next/advanced/overriding_packages).
- [Packages](https://hydra.cc/docs/next/advanced/overriding_packages)
- [Extending Configs](https://hydra.cc/docs/next/patterns/extending_configs) can also be useful.

I think between those pages you should have enough information to do what you want.

While you can use the config search path add different config roots and thus slightly change the config structure, I suggest that you try to leverage the default package when organizing your config (the default package marches the config group).
Thanks for ur reply @omry. 

> While you can use the config search path add different config roots and thus slightly change the config structure, I suggest that you try to leverage the default package when organizing your config (the default package marches the config group).

Here I already make the installed configs of my framework march the config group, for example:
```
framework
├── config.yaml
├── dataset
│   ├── mnist.yaml
│   └── swag.yaml
├── model
│   ├── bert.yaml
│   └── vanilla.yaml
└── trainer
    ├── basic_cv_trainer.yaml
    └── bert_trainer.yaml
```
These part are aligned with the framework codes, so users have no direct access to the configs. In order to override the pre-installed config, one way is to provide command line args:
```
python hydra-app.py model=bert model.d_model=384 trainer=bert_trainer trainer.max_steps=1000
```
It's a little strange to record different command line args for different experiments. So they may provide another config file to override them.
```
user_experiments
├── cv
   ├── alexnet
    │   └── configs
    │       ├── version_1.yaml
    │       ├── version_2.yaml
```
Here the user configs play a role as the command line arguments.
```
python hydra_app.py --user_config=cv/alexnet/configs/version_1.yaml

# when version_1.yaml contains the following content, the cmd is the same as
# python hydra-app.py model=bert model.d_model=384 trainer=bert_trainer trainer.max_steps=1000

# @package _global_
default:
  - override /model: bert
  - override /trainer: bert_trainer
 
model.d_model: 384
trainer.max_steps: 1000
```

So I still need to deal with case of two config sources (pre-installed configs / user overrides). 
1. --user_config is not a Hydra flag.
2. As far as I understand, your problem is not related to this issue.
Using existing issues for support is bad practice, it sends notifications and emails to all the subscribers.
If you have further questions please open a dedicated issue.
Sorry. Just find several guys above are talking about the same thing in this thread.
No worries.
Feel free to continue the discussion in a different issue if need help.
I need to ask clarifying questions to continue helping and I don't want to do it on this issue for the reasons I mentioned above.

Patch:
diff --git a/hydra/_internal/config_loader_impl.py b/hydra/_internal/config_loader_impl.py
--- a/hydra/_internal/config_loader_impl.py
+++ b/hydra/_internal/config_loader_impl.py
@@ -8,9 +8,9 @@
 from collections import defaultdict
 from dataclasses import dataclass
 from textwrap import dedent
-from typing import Any, Dict, List, Optional
+from typing import Any, Dict, List, MutableSequence, Optional
 
-from omegaconf import DictConfig, OmegaConf, flag_override, open_dict
+from omegaconf import Container, DictConfig, OmegaConf, flag_override, open_dict
 from omegaconf.errors import (
     ConfigAttributeError,
     ConfigKeyError,
@@ -152,6 +152,62 @@ def load_configuration(
         except OmegaConfBaseException as e:
             raise ConfigCompositionException().with_traceback(sys.exc_info()[2]) from e
 
+    def _process_config_searchpath(
+        self,
+        config_name: Optional[str],
+        parsed_overrides: List[Override],
+        repo: CachingConfigRepository,
+    ) -> None:
+        if config_name is not None:
+            loaded = repo.load_config(config_path=config_name)
+            primary_config: Container
+            if loaded is None:
+                primary_config = OmegaConf.create()
+            else:
+                primary_config = loaded.config
+        else:
+            primary_config = OmegaConf.create()
+
+        def is_searchpath_override(v: Override) -> bool:
+            return v.get_key_element() == "hydra.searchpath"
+
+        override = None
+        for v in parsed_overrides:
+            if is_searchpath_override(v):
+                override = v.value()
+                break
+
+        searchpath = OmegaConf.select(primary_config, "hydra.searchpath")
+        if override is not None:
+            provider = "hydra.searchpath in command-line"
+            searchpath = override
+        else:
+            provider = "hydra.searchpath in main"
+
+        def _err() -> None:
+            raise ConfigCompositionException(
+                f"hydra.searchpath must be a list of strings. Got: {searchpath}"
+            )
+
+        if searchpath is None:
+            return
+
+        # validate hydra.searchpath.
+        # Note that we cannot rely on OmegaConf validation here because we did not yet merge with the Hydra schema node
+        if not isinstance(searchpath, MutableSequence):
+            _err()
+        for v in searchpath:
+            if not isinstance(v, str):
+                _err()
+
+        new_csp = copy.deepcopy(self.config_search_path)
+        schema = new_csp.get_path().pop(-1)
+        assert schema.provider == "schema"
+        for sp in searchpath:
+            new_csp.append(provider=provider, path=sp)
+        new_csp.append("schema", "structured://")
+        repo.initialize_sources(new_csp)
+
     def _load_configuration_impl(
         self,
         config_name: Optional[str],
@@ -165,6 +221,8 @@ def _load_configuration_impl(
         parser = OverridesParser.create()
         parsed_overrides = parser.parse_overrides(overrides=overrides)
 
+        self._process_config_searchpath(config_name, parsed_overrides, caching_repo)
+
         self.validate_sweep_overrides_legal(
             overrides=parsed_overrides, run_mode=run_mode, from_shell=from_shell
         )
@@ -401,7 +459,17 @@ def _load_single_config(
 
                 assert isinstance(merged, DictConfig)
 
-        return self._embed_result_config(ret, default.package)
+        res = self._embed_result_config(ret, default.package)
+        if (
+            not default.primary
+            and config_path != "hydra/config"
+            and OmegaConf.select(res.config, "hydra.searchpath") is not None
+        ):
+            raise ConfigCompositionException(
+                f"In '{config_path}': Overriding hydra.searchpath is only supported from the primary config"
+            )
+
+        return res
 
     @staticmethod
     def _embed_result_config(
diff --git a/hydra/_internal/config_repository.py b/hydra/_internal/config_repository.py
--- a/hydra/_internal/config_repository.py
+++ b/hydra/_internal/config_repository.py
@@ -50,6 +50,10 @@ def get_group_options(
     def get_sources(self) -> List[ConfigSource]:
         ...
 
+    @abstractmethod
+    def initialize_sources(self, config_search_path: ConfigSearchPath) -> None:
+        ...
+
 
 class ConfigRepository(IConfigRepository):
 
@@ -57,6 +61,9 @@ class ConfigRepository(IConfigRepository):
     sources: List[ConfigSource]
 
     def __init__(self, config_search_path: ConfigSearchPath) -> None:
+        self.initialize_sources(config_search_path)
+
+    def initialize_sources(self, config_search_path: ConfigSearchPath) -> None:
         self.sources = []
         for search_path in config_search_path.get_path():
             assert search_path.path is not None
@@ -71,7 +78,7 @@ def get_schema_source(self) -> ConfigSource:
         assert (
             source.__class__.__name__ == "StructuredConfigSource"
             and source.provider == "schema"
-        )
+        ), "schema config source must be last"
         return source
 
     def load_config(self, config_path: str) -> Optional[ConfigResult]:
@@ -313,6 +320,12 @@ def __init__(self, delegate: IConfigRepository):
     def get_schema_source(self) -> ConfigSource:
         return self.delegate.get_schema_source()
 
+    def initialize_sources(self, config_search_path: ConfigSearchPath) -> None:
+        self.delegate.initialize_sources(config_search_path)
+        # not clearing the cache.
+        # For the use case this is used, the only thing in the cache is the primary config
+        # and we want to keep it even though we re-initialized the sources.
+
     def load_config(self, config_path: str) -> Optional[ConfigResult]:
         cache_key = f"config_path={config_path}"
         if cache_key in self.cache:
diff --git a/hydra/_internal/config_search_path_impl.py b/hydra/_internal/config_search_path_impl.py
--- a/hydra/_internal/config_search_path_impl.py
+++ b/hydra/_internal/config_search_path_impl.py
@@ -1,5 +1,5 @@
 # Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved
-from typing import List, Optional, Sequence
+from typing import List, MutableSequence, Optional
 
 from hydra.core.config_search_path import (
     ConfigSearchPath,
@@ -14,7 +14,7 @@ class ConfigSearchPathImpl(ConfigSearchPath):
     def __init__(self) -> None:
         self.config_search_path = []
 
-    def get_path(self) -> Sequence[SearchPathElement]:
+    def get_path(self) -> MutableSequence[SearchPathElement]:
         return self.config_search_path
 
     def find_last_match(self, reference: SearchPathQuery) -> int:
diff --git a/hydra/conf/__init__.py b/hydra/conf/__init__.py
--- a/hydra/conf/__init__.py
+++ b/hydra/conf/__init__.py
@@ -101,6 +101,10 @@ class HydraConf:
         ]
     )
 
+    # Elements to append to the config search path.
+    # Note: This can only be configured in the primary config.
+    searchpath: List[str] = field(default_factory=list)
+
     # Normal run output configuration
     run: RunDir = RunDir()
     # Multi-run output configuration
diff --git a/hydra/core/config_search_path.py b/hydra/core/config_search_path.py
--- a/hydra/core/config_search_path.py
+++ b/hydra/core/config_search_path.py
@@ -1,7 +1,7 @@
 # Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved
 from abc import ABC, abstractmethod
 from dataclasses import dataclass
-from typing import Optional, Sequence
+from typing import MutableSequence, Optional
 
 
 class SearchPathElement:
@@ -28,7 +28,7 @@ class SearchPathQuery:
 
 class ConfigSearchPath(ABC):
     @abstractmethod
-    def get_path(self) -> Sequence[SearchPathElement]:
+    def get_path(self) -> MutableSequence[SearchPathElement]:
         ...
 
     @abstractmethod


Test patch:
diff --git a/tests/test_compose.py b/tests/test_compose.py
--- a/tests/test_compose.py
+++ b/tests/test_compose.py
@@ -7,7 +7,7 @@
 from typing import Any, Dict, List, Optional
 
 from omegaconf import OmegaConf
-from pytest import mark, param, raises
+from pytest import fixture, mark, param, raises
 
 from hydra._internal.config_search_path_impl import ConfigSearchPathImpl
 from hydra.core.config_search_path import SearchPathQuery
@@ -25,6 +25,26 @@
 chdir_hydra_root()
 
 
+@fixture
+def initialize_hydra(config_path: Optional[str]) -> Any:
+    try:
+        init = initialize(config_path=config_path)
+        init.__enter__()
+        yield
+    finally:
+        init.__exit__(*sys.exc_info())
+
+
+@fixture
+def initialize_hydra_no_path() -> Any:
+    try:
+        init = initialize()
+        init.__enter__()
+        yield
+    finally:
+        init.__exit__(*sys.exc_info())
+
+
 def test_initialize(hydra_restore_singletons: Any) -> None:
     assert not GlobalHydra().is_initialized()
     initialize(config_path=None)
@@ -46,7 +66,7 @@ def test_initialize_with_config_path(hydra_restore_singletons: Any) -> None:
     assert idx != -1
 
 
-@mark.usefixtures("hydra_restore_singletons")
+@mark.usefixtures("initialize_hydra")
 @mark.parametrize("config_path", ["../hydra/test_utils/configs"])
 @mark.parametrize(
     "config_file, overrides, expected",
@@ -59,20 +79,21 @@ def test_initialize_with_config_path(hydra_restore_singletons: Any) -> None:
 )
 class TestCompose:
     def test_compose_config(
-        self, config_path: str, config_file: str, overrides: List[str], expected: Any
+        self,
+        config_file: str,
+        overrides: List[str],
+        expected: Any,
     ) -> None:
-        with initialize(config_path=config_path):
-            cfg = compose(config_file, overrides)
-            assert cfg == expected
+        cfg = compose(config_file, overrides)
+        assert cfg == expected
 
     def test_strict_failure_global_strict(
-        self, config_path: str, config_file: str, overrides: List[str], expected: Any
+        self, config_file: str, overrides: List[str], expected: Any
     ) -> None:
-        with initialize(config_path=config_path):
-            # default strict True, call is unspecified
-            overrides.append("fooooooooo=bar")
-            with raises(HydraException):
-                compose(config_file, overrides)
+        # default strict True, call is unspecified
+        overrides.append("fooooooooo=bar")
+        with raises(HydraException):
+            compose(config_file, overrides)
 
 
 @mark.usefixtures("hydra_restore_singletons")
@@ -297,6 +318,7 @@ def test_initialization_root_module(monkeypatch: Any) -> None:
     subprocess.check_call([sys.executable, "-m", "main"])
 
 
+@mark.usefixtures("initialize_hydra_no_path")
 @mark.parametrize(
     ("overrides", "expected"),
     [
@@ -314,50 +336,175 @@ class Config:
 
     ConfigStore.instance().store(name="config", node=Config)
 
-    with initialize():
-        cfg = compose(config_name="config", overrides=overrides)
-        assert cfg == expected
+    cfg = compose(config_name="config", overrides=overrides)
+    assert cfg == expected
 
 
+@mark.usefixtures("hydra_restore_singletons")
+@mark.usefixtures("initialize_hydra_no_path")
 class TestAdd:
-    def test_add(self, hydra_restore_singletons: Any) -> None:
+    def test_add(self) -> None:
         ConfigStore.instance().store(name="config", node={"key": 0})
-        with initialize():
-            with raises(
-                ConfigCompositionException,
-                match="Could not append to config. An item is already at 'key'",
-            ):
-                compose(config_name="config", overrides=["+key=value"])
+        with raises(
+            ConfigCompositionException,
+            match="Could not append to config. An item is already at 'key'",
+        ):
+            compose(config_name="config", overrides=["+key=value"])
 
-            cfg = compose(config_name="config", overrides=["key=1"])
-            assert cfg == {"key": 1}
+        cfg = compose(config_name="config", overrides=["key=1"])
+        assert cfg == {"key": 1}
 
-    def test_force_add(self, hydra_restore_singletons: Any) -> None:
+    def test_force_add(self) -> None:
         ConfigStore.instance().store(name="config", node={"key": 0})
-        with initialize():
-            cfg = compose(config_name="config", overrides=["++key=1"])
-            assert cfg == {"key": 1}
+        cfg = compose(config_name="config", overrides=["++key=1"])
+        assert cfg == {"key": 1}
 
-            cfg = compose(config_name="config", overrides=["++key2=1"])
-            assert cfg == {"key": 0, "key2": 1}
+        cfg = compose(config_name="config", overrides=["++key2=1"])
+        assert cfg == {"key": 0, "key2": 1}
 
-    def test_add_config_group(self, hydra_restore_singletons: Any) -> None:
+    def test_add_config_group(self) -> None:
         ConfigStore.instance().store(group="group", name="a0", node={"key": 0})
         ConfigStore.instance().store(group="group", name="a1", node={"key": 1})
-        with initialize():
-            # overriding non existing group throws
-            with raises(ConfigCompositionException):
-                compose(overrides=["group=a0"])
-
-            # appending a new group
-            cfg = compose(overrides=["+group=a0"])
-            assert cfg == {"group": {"key": 0}}
-
-            # force adding is not supported for config groups.
-            with raises(
-                ConfigCompositionException,
-                match=re.escape(
-                    "force-add of config groups is not supported: '++group=a1'"
+        # overriding non existing group throws
+        with raises(ConfigCompositionException):
+            compose(overrides=["group=a0"])
+
+        # appending a new group
+        cfg = compose(overrides=["+group=a0"])
+        assert cfg == {"group": {"key": 0}}
+
+        # force adding is not supported for config groups.
+        with raises(
+            ConfigCompositionException,
+            match=re.escape(
+                "force-add of config groups is not supported: '++group=a1'"
+            ),
+        ):
+            compose(overrides=["++group=a1"])
+
+
+@mark.usefixtures("hydra_restore_singletons")
+@mark.usefixtures("initialize_hydra_no_path")
+class TestConfigSearchPathOverride:
+    @fixture
+    def init_configs(self) -> Any:
+        cs = ConfigStore.instance()
+        cs.store(
+            name="with_sp",
+            node={"hydra": {"searchpath": ["pkg://hydra.test_utils.configs"]}},
+        )
+        cs.store(name="without_sp", node={})
+
+        cs.store(name="bad1", node={"hydra": {"searchpath": 42}})
+        cs.store(name="bad2", node={"hydra": {"searchpath": [42]}})
+
+        # Using this triggers an error. Only primary configs are allowed to override hydra.searchpath
+        cs.store(
+            group="group2",
+            name="overriding_sp",
+            node={"hydra": {"searchpath": ["abc"]}},
+            package="_global_",
+        )
+        yield
+
+    @mark.parametrize(
+        ("config_name", "overrides", "expected"),
+        [
+            # config group is interpreted as simple config value addition.
+            param("without_sp", ["+group1=file1"], {"group1": "file1"}, id="without"),
+            param("with_sp", ["+group1=file1"], {"foo": 10}, id="with"),
+            # Overriding hydra.searchpath
+            param(
+                "without_sp",
+                ["hydra.searchpath=[pkg://hydra.test_utils.configs]", "+group1=file1"],
+                {"foo": 10},
+                id="sp_added_by_override",
+            ),
+            param(
+                "with_sp",
+                ["hydra.searchpath=[]", "+group1=file1"],
+                {"group1": "file1"},
+                id="sp_removed_by_override",
+            ),
+        ],
+    )
+    def test_searchpath_in_primary_config(
+        self,
+        init_configs: Any,
+        config_name: str,
+        overrides: List[str],
+        expected: Any,
+    ) -> None:
+        cfg = compose(config_name=config_name, overrides=overrides)
+        assert cfg == expected
+
+    @mark.parametrize(
+        ("config_name", "overrides", "expected"),
+        [
+            param(
+                "bad1",
+                [],
+                raises(
+                    ConfigCompositionException,
+                    match=re.escape(
+                        "hydra.searchpath must be a list of strings. Got: 42"
+                    ),
                 ),
-            ):
-                compose(overrides=["++group=a1"])
+                id="bad_cp_in_config",
+            ),
+            param(
+                "bad2",
+                [],
+                raises(
+                    ConfigCompositionException,
+                    match=re.escape(
+                        "hydra.searchpath must be a list of strings. Got: [42]"
+                    ),
+                ),
+                id="bad_cp_element_in_config",
+            ),
+            param(
+                "without_sp",
+                ["hydra.searchpath=42"],
+                raises(
+                    ConfigCompositionException,
+                    match=re.escape(
+                        "hydra.searchpath must be a list of strings. Got: 42"
+                    ),
+                ),
+                id="bad_override1",
+            ),
+            param(
+                "without_sp",
+                ["hydra.searchpath=[42]"],
+                raises(
+                    ConfigCompositionException,
+                    match=re.escape(
+                        "hydra.searchpath must be a list of strings. Got: [42]"
+                    ),
+                ),
+                id="bad_override2",
+            ),
+            param(
+                "without_sp",
+                ["+group2=overriding_sp"],
+                raises(
+                    ConfigCompositionException,
+                    match=re.escape(
+                        "In 'group2/overriding_sp': Overriding hydra.searchpath "
+                        "is only supported from the primary config"
+                    ),
+                ),
+                id="overriding_sp_from_non_primary_config",
+            ),
+        ],
+    )
+    def test_searchpath_config_errors(
+        self,
+        init_configs: Any,
+        config_name: str,
+        overrides: List[str],
+        expected: Any,
+    ) -> None:
+        with expected:
+            compose(config_name=config_name, overrides=overrides)