Repository: facebookresearch/hydra

Issue:
[Bug] adding ColorLog override will disable other overrides
Hi
here i follow the basic tutorial and experiment section :
main.py:
```
from omegaconf import DictConfig, OmegaConf
import hydra

@hydra.main(config_path="conf", config_name="config")
def my_app(cfg: DictConfig) -> None:
    print(OmegaConf.to_yaml(cfg))

if __name__ == "__main__":
    my_app()
```

config.yaml:

```
defaults:
  - segment : color
  #- override hydra/job_logging: colorlog
  #- override hydra/hydra_logging: colorlog
volume: 1
```

segment/color.yaml :
`color_num=1`
segment/color2.yaml :
`color_num=10`


experiment/test.yaml :
```
# @package _global_
defaults:
  - override /segment: color2
volume: 9999
```

running the
 `:\python main.py +experiment=test`
will return `volume:9999`
BUT while un-commenting colorlog override in config.yaml the volume returned will be `volume:1`
why adding colorlog impact the other config ?
btw the colorlog do his job and the colorizing the output

thanks


Discussion:
Hi @NinjaAtWork 

I created this [repro](https://github.com/facebookresearch/hydra/files/6786410/5_defaults.zip) based on the info provided.

This does look like a bug - I could reproduce your example locally. If we run `--info defaults-tree` on the example app, we can see that the defaults tree are generated differently in two cases.

### without `override hydra/job_logging` in primary config
```
Defaults Tree
*************
<root>:
  hydra/config:
    hydra/output: default
    hydra/launcher: basic
    hydra/sweeper: basic
    hydra/help: default
    hydra/hydra_help: default
    hydra/hydra_logging: default
    hydra/job_logging: default
    hydra/callbacks: null
    hydra/env: default
    _self_
  config:
    segment: color2
    _self_
    experiment: test
```

### with `override hydra/job_logging` in primary config

```
Defaults Tree
*************
<root>:
  hydra/config:
    hydra/output: default
    hydra/launcher: basic
    hydra/sweeper: basic
    hydra/help: default
    hydra/hydra_help: default
    hydra/hydra_logging: colorlog
    hydra/job_logging: colorlog
    hydra/callbacks: null
    hydra/env: default
    _self_
  config:
    segment: color2
    experiment: test
    _self_
```

For now, to get around this, you can try adding the `_self_` keyword to your config.yaml file. [link to doc](https://hydra.cc/docs/next/advanced/defaults_list/#composition-order)
```
defaults:
  - _self_
  - /segment: color
  - override /hydra/job_logging: colorlog
  - override /hydra/hydra_logging: colorlog
```

Please give this a try and let us know if it helps. We will also take a look at this issue and apply fix if needed. 


Hi  @jieru-hu 

adding `_self_ ` to start of the `defaults ` worked and thanks for the solution and your time for this. 💯 
just for fun: i read all the docs , as someone who like architecture more than programing , my brain will prefer skip when processing words which start with underline like _ self _ or __ main __ or same others ,you know 😄 
please close this issue after a few days which i will be able to report if something somewhere goes tricky or need help.
kindly , by far ,when i use colorlog , then i am unable to use custom job_logging which is described in docs. its great to be able to switch between save to files and save to prompt in colorlog mode .let me know its solution too .



Hi @NinjaAtWork 

Do you mean you want to have both colorlog and somehow also follow [this example](https://hydra.cc/docs/configure_hydra/logging)? 

If so, you can certainly create your own logging configuration. You can extend the `colorlog`'s config and override the any field as needed (all the Hydra loggings configurations are passed in to [logging.config](https://github.com/facebookresearch/hydra/blob/91a954df2ae38b26c621adeee31ac7fcb4087230/hydra/core/utils.py#L38-L71))

Here is the documentation on how to extend a plugin config https://hydra.cc/docs/patterns/configuring_plugins#extending-plugin-default-config
> This does look like a bug - I could reproduce your example locally. If we run `--info defaults-tree` on the example app, we can see that the defaults tree are generated differently in two cases.
> ### without `override hydra/job_logging` in primary config
> ...
> ### with `override hydra/job_logging` in primary config
> ...

@jieru-hu The desired behavior is that both cases should look like the first defaults tree, right?

I'm going to start working on a fix for this soon (unless you've already got work in progress :) ).
Yes, I think this is what I would expect in this situation for the user's config section:
```
  config:
    segment: color2
    _self_
    experiment: test
```
@jieru-hu

my custom job logging as you mentioned but this will disable the color effect :

```
defaults:
  - colorlog
version: 1
formatters:
  simple:
    format: '[%(levelname)s] - %(message)s'
handlers:
  console:
    class: logging.StreamHandler
    formatter: simple
    stream: ext://sys.stdout
root:
  handlers: [console]

disable_existing_loggers: false
```

i think this will be linked with that minor bug of override. ill wait for new update and try this again. :)
@NinjaAtWork I'm not sure if this is related to this particular bug. You overrides actually disable the colorlog, in particular this line
```
  console:
    class: logging.StreamHandler
    formatter: simple
    stream: ext://sys.stdout
```

I've verified this by copy pasting your config and update `formatter: colorlog`.
@jieru-hu 
replacing formatter from `simple `to `colorlog ` will color the console but also change format from `'[%(levelname)s] - %(message)s'` to colorlog format which include date and time. 
i think when user able to define colorlog as `defaults` in first line then could change `format `to something desired 
all i need is to do something like this :
```
formatters:
  colorlog:
    format: '[%(levelname)s] - %(message)s'
```


Resolving patch:
diff --git a/hydra/_internal/defaults_list.py b/hydra/_internal/defaults_list.py
--- a/hydra/_internal/defaults_list.py
+++ b/hydra/_internal/defaults_list.py
@@ -98,6 +98,7 @@ def __init__(self, repo: IConfigRepository, overrides_list: List[Override]) -> N
                         group=override.key_or_group,
                         package=override.package,
                         value=value,
+                        external_append=True,
                     )
                 )
             else:
@@ -364,7 +365,9 @@ def _update_overrides(
             assert d.group is not None
             legacy_hydra_override = not d.is_override() and d.group.startswith("hydra/")
 
-        if seen_override and not (d.is_override() or legacy_hydra_override):
+        if seen_override and not (
+            d.is_override() or d.is_external_append() or legacy_hydra_override
+        ):
             assert isinstance(last_override_seen, GroupDefault)
             pcp = parent.get_config_path()
             okey = last_override_seen.get_override_key()
@@ -467,14 +470,7 @@ def _create_defaults_tree_impl(
         self_added = _validate_self(containing_node=parent, defaults=defaults_list)
 
     if is_root_config:
-        # To ensure config overrides are last, insert the external overrides before the first config override.
-        insert_idx = len(defaults_list)
-        for idx, default in enumerate(defaults_list):
-            if default.is_override():
-                insert_idx = idx
-                break
-
-        defaults_list[insert_idx:insert_idx] = overrides.append_group_defaults
+        defaults_list.extend(overrides.append_group_defaults)
 
     _update_overrides(defaults_list, overrides, parent, interpolated_subtree)
 
diff --git a/hydra/core/default_element.py b/hydra/core/default_element.py
--- a/hydra/core/default_element.py
+++ b/hydra/core/default_element.py
@@ -261,6 +261,9 @@ def get_relative_override_key(self) -> str:
     def is_override(self) -> bool:
         raise NotImplementedError()
 
+    def is_external_append(self) -> bool:
+        raise NotImplementedError()
+
 
 @dataclass
 class VirtualRoot(InputDefault):
@@ -306,6 +309,9 @@ def resolve_interpolation(self, known_choices: DictConfig) -> None:
     def is_override(self) -> bool:
         return False
 
+    def is_external_append(self) -> bool:
+        return False
+
 
 @dataclass(repr=False)
 class ConfigDefault(InputDefault):
@@ -420,6 +426,9 @@ def is_missing(self) -> bool:
     def is_override(self) -> bool:
         return False
 
+    def is_external_append(self) -> bool:
+        return False
+
 
 _legacy_interpolation_pattern: Pattern[str] = re.compile(r"\${defaults\.\d\.")
 
@@ -436,6 +445,8 @@ class GroupDefault(InputDefault):
     deleted: Optional[bool] = None
 
     config_name_overridden: bool = field(default=False, compare=False, repr=False)
+    # True if this item was added using +foo=bar from the external overrides
+    external_append: bool = field(default=False, compare=False, repr=False)
 
     def __post_init__(self) -> None:
         assert self.group is not None and self.group != ""
@@ -553,6 +564,9 @@ def get_relative_override_key(self) -> str:
             key = f"{key}@{self.package}"
         return key
 
+    def is_external_append(self) -> bool:
+        return self.external_append
+
 
 @dataclass
 class DefaultsTreeNode:


Test patch:
diff --git a/tests/defaults_list/data/override_hydra.yaml b/tests/defaults_list/data/override_hydra.yaml
new file mode 100644
--- /dev/null
+++ b/tests/defaults_list/data/override_hydra.yaml
@@ -0,0 +1,3 @@
+defaults:
+  - group1: file1
+  - override hydra/output: disabled
diff --git a/tests/defaults_list/test_defaults_tree.py b/tests/defaults_list/test_defaults_tree.py
--- a/tests/defaults_list/test_defaults_tree.py
+++ b/tests/defaults_list/test_defaults_tree.py
@@ -917,6 +917,52 @@ def test_group_default_with_appended_experiment(
     )
 
 
+@mark.parametrize(
+    "config_name,overrides,expected",
+    [
+        param(
+            "override_hydra",
+            ["+experiment=override_config_group"],
+            DefaultsTreeNode(
+                node=VirtualRoot(),
+                children=[
+                    DefaultsTreeNode(
+                        node=ConfigDefault(path="hydra/config"),
+                        children=[
+                            GroupDefault(group="help", value="default"),
+                            GroupDefault(group="output", value="disabled"),
+                            ConfigDefault(path="_self_"),
+                        ],
+                    ),
+                    DefaultsTreeNode(
+                        node=ConfigDefault(path="override_hydra"),
+                        children=[
+                            GroupDefault(group="group1", value="file2"),
+                            ConfigDefault(path="_self_"),
+                            GroupDefault(
+                                group="experiment", value="override_config_group"
+                            ),
+                        ],
+                    ),
+                ],
+            ),
+            id="override_hydra_with_experiment",
+        ),
+    ],
+)
+def test_experiment_where_primary_config_has_override(
+    config_name: str,
+    overrides: List[str],
+    expected: DefaultsTreeNode,
+) -> None:
+    _test_defaults_tree_impl(
+        config_name=config_name,
+        input_overrides=overrides,
+        expected=expected,
+        prepend_hydra=True,
+    )
+
+
 @mark.parametrize(
     "config_name,overrides,expected",
     [