Repository: spack/spack

Issue:
`spack.config.override()` prepends to lists rather than replacing
Given the case where a list configuration item is specified in YAML (`config.yaml`, say):
```YAML
...
extensions:
  - /my/path/to/extension
```
and a test overrides that item, _viz_:
```Python
    with spack.config.override('config:extensions', ['/my/path/to/extension-1', '/my/path/to/extension-2']):
        ...
```
Then the actual value of `config:extensions` inside the overridden context as returned by `spack.config.get()` is:
```Python
['/my/path/to/extension-1', '/my/path/to/extension-2', '/my/path/to/extension']
```
rather than the expected:
```Python
['/my/path/to/extension-1', '/my/path/to/extension-2']
```

I would expect higher priority scopes to replace lists rather than prepend to them, at least as a default behavior. Is this intended, and what is the use case?


Discussion:
Created following a conversation on [Slack#General](https://spackpm.slack.com/archives/C5W7NKZJT/p1578346978172300) with @tgamblin and @tldahlgren .
@tldahlgren Per your [question](https://spackpm.slack.com/archives/C5W7NKZJT/p1578356429193100?thread_ts=1578346978.172300&cid=C5W7NKZJT):
```console
==> Warning: [1] config.scopes.values() = odict_values([<InternalConfigScope: _builtin>, <ConfigScope: defaults: /Users/greenc/work/cet-is/sources/spack-test/etc/spack/defaults>, <ConfigScope: defaults/test: /Users/greenc/work/cet-is/sources/spack-test/etc/spack/defaults/test>, <ConfigScope: system: /etc/spack>, <ConfigScope: system/test: /etc/spack/test>, <ConfigScope: site: /Users/greenc/work/cet-is/sources/spack-test/etc/spack>, <ConfigScope: site/test: /Users/greenc/work/cet-is/sources/spack-test/etc/spack/test>, <ConfigScope: user: /Users/greenc/.spack>, <ConfigScope: user/test: /Users/greenc/.spack/test>, <InternalConfigScope: overrides-0>, <InternalConfigScope: command_line>])
```
Note that the `command_line` scope is always highest priority. `overrides-0` is directly below that, as one would expect.

I misspoke.  The extra logic in the function that ensures `command_line` is always highest priority ensures that is the case.  

Thanks.
what happens if you do with `spack.config.override('config:extensions:')` (note the colon)?  Does that make the top-level config completely replace lower-level ones?  We have the (heretofore somewhat inconsistently supported) `::` syntax for this -- it'd be cool if we could use it here.
@tgamblin I receive a `KeyError` in `spack.config.set()`
> what happens if you do with `spack.config.override('config:extensions:')` (note the colon)? Does that make the top-level config completely replace lower-level ones? We have the (heretofore somewhat inconsistently supported) `::` syntax for this -- it'd be cool if we could use it here.

@tgamblin Did you mean to ask @chissg to try `spack.config.override('config:extensions::')` (i.e., end in a double-colon)?
@tldahlgren 's suggestion results in the same `KeyError`. Adding a `:` or `::` to a `spack.config.get()` call results in the error:
```console
E           AttributeError: 'CommentedSeq' object has no attribute 'get'
```
in both cases.
@chissg  Interesting.  Thanks for checking.

Note: A `Configuration` object's `get` method uses its `get_config` method, which does use the  `_merge_yaml` function for `dict` entries per comments:

```Config file authors can optionally end any attribute in a dict with `::` instead of `:`, and the key will override that of the parent instead of merging.```

IIUC this is the preferred approach to resolve this issue for this list case.
@tldahlgren  One would have to add this functionality for lists in `_merge_yaml()`, but it would seem also that `config.set()` would need to understand the `::` notation (the current use of `split()` is insufficient) and create the relevant keys as `syaml.syaml_str` entities with the `override` attribute set (rather than simple strings) in order to allow the override functionality to be triggered. Is there anything else I'm missing?

Resolving patch:
diff --git a/lib/spack/spack/config.py b/lib/spack/spack/config.py
--- a/lib/spack/spack/config.py
+++ b/lib/spack/spack/config.py
@@ -280,6 +280,7 @@ def __init__(self, name, data=None):
         self.sections = syaml.syaml_dict()
 
         if data:
+            data = InternalConfigScope._process_dict_keyname_overrides(data)
             for section in data:
                 dsec = data[section]
                 validate({section: dsec}, section_schemas[section])
@@ -306,6 +307,25 @@ def write_section(self, section):
     def __repr__(self):
         return '<InternalConfigScope: %s>' % self.name
 
+    @staticmethod
+    def _process_dict_keyname_overrides(data):
+        """Turn a trailing `:' in a key name into an override attribute."""
+        result = {}
+        for sk, sv in iteritems(data):
+            if sk.endswith(':'):
+                key = syaml.syaml_str(sk[:-1])
+                key.override = True
+            else:
+                key = sk
+
+            if isinstance(sv, dict):
+                result[key]\
+                    = InternalConfigScope._process_dict_keyname_overrides(sv)
+            else:
+                result[key] = copy.copy(sv)
+
+        return result
+
 
 class Configuration(object):
     """A full Spack configuration, from a hierarchy of config files.
@@ -505,14 +525,14 @@ def set(self, path, value, scope=None):
 
         Accepts the path syntax described in ``get()``.
         """
-        section, _, rest = path.partition(':')
+        parts = _process_config_path(path)
+        section = parts.pop(0)
 
-        if not rest:
+        if not parts:
             self.update_config(section, value, scope=scope)
         else:
             section_data = self.get_config(section, scope=scope)
 
-            parts = rest.split(':')
             data = section_data
             while len(parts) > 1:
                 key = parts.pop(0)
@@ -612,7 +632,7 @@ def _config():
     """Singleton Configuration instance.
 
     This constructs one instance associated with this module and returns
-    it. It is bundled inside a function so that configuratoin can be
+    it. It is bundled inside a function so that configuration can be
     initialized lazily.
 
     Return:
@@ -763,17 +783,12 @@ def _merge_yaml(dest, source):
     Config file authors can optionally end any attribute in a dict
     with `::` instead of `:`, and the key will override that of the
     parent instead of merging.
-
     """
     def they_are(t):
         return isinstance(dest, t) and isinstance(source, t)
 
-    # If both are None, handle specially and return None.
-    if source is None and dest is None:
-        return None
-
     # If source is None, overwrite with source.
-    elif source is None:
+    if source is None:
         return None
 
     # Source list is prepended (for precedence)
@@ -799,8 +814,9 @@ def they_are(t):
             # to copy mark information on source keys to dest.
             key_marks[sk] = sk
 
-        # ensure that keys are marked in the destination.  the key_marks dict
-        # ensures we can get the actual source key objects from dest keys
+        # ensure that keys are marked in the destination. The
+        # key_marks dict ensures we can get the actual source key
+        # objects from dest keys
         for dk in list(dest.keys()):
             if dk in key_marks and syaml.markable(dk):
                 syaml.mark(dk, key_marks[dk])
@@ -812,9 +828,34 @@ def they_are(t):
 
         return dest
 
-    # In any other case, overwrite with a copy of the source value.
-    else:
-        return copy.copy(source)
+    # If we reach here source and dest are either different types or are
+    # not both lists or dicts: replace with source.
+    return copy.copy(source)
+
+
+#
+# Process a path argument to config.set() that may contain overrides ('::' or
+# trailing ':')
+#
+def _process_config_path(path):
+    result = []
+    if path.startswith(':'):
+        raise syaml.SpackYAMLError("Illegal leading `:' in path `{0}'".
+                                   format(path), '')
+    seen_override_in_path = False
+    while path:
+        front, sep, path = path.partition(':')
+        if (sep and not path) or path.startswith(':'):
+            if seen_override_in_path:
+                raise syaml.SpackYAMLError("Meaningless second override"
+                                           " indicator `::' in path `{0}'".
+                                           format(path), '')
+            path = path.lstrip(':')
+            front = syaml.syaml_str(front)
+            front.override = True
+            seen_override_in_path = True
+        result.append(front)
+    return result
 
 
 #


Test patch:
diff --git a/lib/spack/spack/test/config.py b/lib/spack/spack/test/config.py
--- a/lib/spack/spack/test/config.py
+++ b/lib/spack/spack/test/config.py
@@ -46,7 +46,19 @@
 
 config_override_list = {
     'config': {
-        'build_stage:': ['patha', 'pathb']}}
+        'build_stage:': ['pathd', 'pathe']}}
+
+config_merge_dict = {
+    'config': {
+        'info': {
+            'a': 3,
+            'b': 4}}}
+
+config_override_dict = {
+    'config': {
+        'info:': {
+            'a': 7,
+            'c': 9}}}
 
 
 @pytest.fixture()
@@ -382,7 +394,7 @@ def test_read_config_override_list(mock_low_high_config, write_config_file):
     write_config_file('config', config_override_list, 'high')
     assert spack.config.get('config') == {
         'install_tree': 'install_tree_path',
-        'build_stage': ['patha', 'pathb']
+        'build_stage': config_override_list['config']['build_stage:']
     }
 
 
@@ -857,3 +869,74 @@ def test_dotkit_in_config_does_not_raise(
     # we throw a a deprecation warning without raising
     assert '_sp_sys_type' in captured[0]  # stdout
     assert 'Warning' in captured[1]  # stderr
+
+
+def test_internal_config_section_override(mock_low_high_config,
+                                          write_config_file):
+    write_config_file('config', config_merge_list, 'low')
+    wanted_list = config_override_list['config']['build_stage:']
+    mock_low_high_config.push_scope(spack.config.InternalConfigScope
+                                    ('high', {
+                                        'config:': {
+                                            'build_stage': wanted_list
+                                        }
+                                    }))
+    assert mock_low_high_config.get('config:build_stage') == wanted_list
+
+
+def test_internal_config_dict_override(mock_low_high_config,
+                                       write_config_file):
+    write_config_file('config', config_merge_dict, 'low')
+    wanted_dict = config_override_dict['config']['info:']
+    mock_low_high_config.push_scope(spack.config.InternalConfigScope
+                                    ('high', config_override_dict))
+    assert mock_low_high_config.get('config:info') == wanted_dict
+
+
+def test_internal_config_list_override(mock_low_high_config,
+                                       write_config_file):
+    write_config_file('config', config_merge_list, 'low')
+    wanted_list = config_override_list['config']['build_stage:']
+    mock_low_high_config.push_scope(spack.config.InternalConfigScope
+                                    ('high', config_override_list))
+    assert mock_low_high_config.get('config:build_stage') == wanted_list
+
+
+def test_set_section_override(mock_low_high_config, write_config_file):
+    write_config_file('config', config_merge_list, 'low')
+    wanted_list = config_override_list['config']['build_stage:']
+    with spack.config.override('config::build_stage', wanted_list):
+        assert mock_low_high_config.get('config:build_stage') == wanted_list
+    assert config_merge_list['config']['build_stage'] == \
+        mock_low_high_config.get('config:build_stage')
+
+
+def test_set_list_override(mock_low_high_config, write_config_file):
+    write_config_file('config', config_merge_list, 'low')
+    wanted_list = config_override_list['config']['build_stage:']
+    with spack.config.override('config:build_stage:', wanted_list):
+        assert wanted_list == mock_low_high_config.get('config:build_stage')
+    assert config_merge_list['config']['build_stage'] == \
+        mock_low_high_config.get('config:build_stage')
+
+
+def test_set_dict_override(mock_low_high_config, write_config_file):
+    write_config_file('config', config_merge_dict, 'low')
+    wanted_dict = config_override_dict['config']['info:']
+    with spack.config.override('config:info:', wanted_dict):
+        assert wanted_dict == mock_low_high_config.get('config:info')
+    assert config_merge_dict['config']['info'] == \
+        mock_low_high_config.get('config:info')
+
+
+def test_set_bad_path(config):
+    with pytest.raises(syaml.SpackYAMLError, match='Illegal leading'):
+        with spack.config.override(':bad:path', ''):
+            pass
+
+
+def test_bad_path_double_override(config):
+    with pytest.raises(syaml.SpackYAMLError,
+                       match='Meaningless second override'):
+        with spack.config.override('bad::double:override::directive', ''):
+            pass