Repository: spack/spack

Issue:
spack config update doesn't handle comments in yaml file
The `spack config update` command doesn't handle comments (#).

AFAIK, comments are allowed and I've been using them for years.  For
example, here is a sample `packages.yaml` file, and what happens when
I try to update it.

```
packages:
  # system cmake in /usr
  cmake:
    paths:
      cmake@3.11.0:  /usr
    buildable: False

$ spack config update packages
...
==> Do you want to proceed? [y/N] y
==> Error: 'dict' object has no attribute '_yaml_comment'
```

And the file isn't updated.

----------

Btw, there are two formats to specify lists in yaml, either with
`[...]` on one line, or with `-` on multiple lines.  I think this is
standard yaml and AFAIK, they're equivalent.

But spack update is inconsistent in how it updates them.  If I update
this, including cmake which needs updating:

```
packages:
  cmake:
    paths:
      cmake@3.11.0:  /usr
    buildable: False

  binutils:
    version:  [2.34]
    variants: +libiberty ~nls
```

Then binutils changes to:

```
  binutils:
    version:
    - 2.34
    variants: +libiberty ~nls
```

But if I leave out cmake, then spack update reports nothing to do and
leaves the file unchanged.

```
packages:
  binutils:
    version:  [2.34]
    variants: +libiberty ~nls

$ spack config update packages
==> No updates needed for "packages" section.
```

I don't know if I really care one way or the other, but it's odd that
it's inconsistent like this.

----------

Also, spack update deletes all the blank lines in the file, which I
don't like.




Discussion:
Bug confirmed:
```console
$ cat ~/.spack/packages.yaml
packages:
  # system cmake in /usr
  cmake:
    paths:
      cmake@3.11.0:  /usr
    buildable: False

$ spack -d config update packages -y
==> [2020-08-13-08:29:36.695023] Imported config from built-in commands
==> [2020-08-13-08:29:36.700339] Imported config from built-in commands
==> [2020-08-13-08:29:36.701393] Reading config file /home/culpo/PycharmProjects/spack/etc/spack/defaults/packages.yaml
==> [2020-08-13-08:29:36.724921] Reading config file /home/culpo/.spack/packages.yaml
==> [2020-08-13-08:29:36.728661] Warning: the attribute "paths" in the "packages" section of the configuration has been deprecated [entry=CommentedMap([('cmake@3.11.0', '/usr')])]
==> [2020-08-13-08:29:36.728953] OUTDATED CONFIGURATION FILE [section=packages, scope=user, dir=/home/culpo/.spack]
==> [2020-08-13-08:29:36.730702] Reading config file /home/culpo/PycharmProjects/spack/etc/spack/defaults/config.yaml
Traceback (most recent call last):
  File "/home/culpo/PycharmProjects/spack/bin/spack", line 64, in <module>
    sys.exit(spack.main.main())
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/main.py", line 762, in main
    return _invoke_command(command, parser, args, unknown)
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/main.py", line 490, in _invoke_command
    return_val = command(parser, args)
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/cmd/config.py", line 455, in config
    action[args.config_command](args)
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/cmd/config.py", line 374, in config_update
    args.section, data, scope=scope.name, force=True
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/config.py", line 490, in update_config
    comments)
AttributeError: 'dict' object has no attribute '_yaml_comment'
```
Currently using :

* **Spack:** 0.15.3-508-5512340a5
* **Python:** 3.6.9
* **Platform:** linux-ubuntu18.04-broadwell


Resolving patch:
diff --git a/lib/spack/spack/cmd/config.py b/lib/spack/spack/cmd/config.py
--- a/lib/spack/spack/cmd/config.py
+++ b/lib/spack/spack/cmd/config.py
@@ -363,7 +363,7 @@ def config_update(args):
             scope.name, args.section
         )
         with open(cfg_file) as f:
-            data = syaml.load(f) or {}
+            data = syaml.load_config(f) or {}
             data = data.pop(args.section, {})
         update_fn(data)
 


Test patch:
diff --git a/lib/spack/spack/test/cmd/config.py b/lib/spack/spack/test/cmd/config.py
--- a/lib/spack/spack/test/cmd/config.py
+++ b/lib/spack/spack/test/cmd/config.py
@@ -515,6 +515,42 @@ def test_updating_multiple_scopes_at_once(packages_yaml_v015):
         check_update(data)
 
 
+@pytest.mark.regression('18031')
+def test_config_update_can_handle_comments(mutable_config):
+    # Create an outdated config file with comments
+    scope = spack.config.default_modify_scope()
+    cfg_file = spack.config.config.get_config_filename(scope, 'packages')
+    with open(cfg_file, mode='w') as f:
+        f.write("""
+packages:
+  # system cmake in /usr
+  cmake:
+    paths:
+      cmake@3.14.0:  /usr
+    # Another comment after the outdated section
+    buildable: False
+""")
+
+    # Try to update it, it should not raise errors
+    config('update', '-y', 'packages')
+
+    # Check data
+    data = spack.config.get('packages', scope=scope)
+    assert 'paths' not in data['cmake']
+    assert 'externals' in data['cmake']
+    externals = data['cmake']['externals']
+    assert len(externals) == 1
+    assert externals[0]['spec'] == 'cmake@3.14.0'
+    assert externals[0]['prefix'] == '/usr'
+
+    # Check the comment is there
+    with open(cfg_file) as f:
+        text = ''.join(f.readlines())
+
+    assert '# system cmake in /usr' in text
+    assert '# Another comment after the outdated section' in text
+
+
 def check_update(data):
     """Check that the data from the packages_yaml_v015
     has been updated.