Repository: spack/spack

Issue:
spack concretize reorganizes spack.yaml
The way Spack Environments were explained to me was that if you maintained a project on GitHub, you could add a `spack.yaml` file to your repo that lists its dependencies. Then you could clone the project, cd into the directory, and run `spack install` or `spack concretize; spack install` to install all of its dependencies. However, `spack concretize` appears to modify `spack.yaml`, a file which is likely to be under version control. No other Spack command that I know of reorganizes your configuration files, and `spack concretize` certainly shouldn't either.

### Steps to reproduce the issue

```console
$ cat spack.yaml
spack:
  specs:
  - 'gdal+python'
  - 'python@3:+sqlite3'
  - 'py-earthengine-api'
  - 'py-flake8'
  - 'py-matplotlib'
  - 'py-numpy'
  - 'py-pandas'
  - 'py-pytest@3.9:'
  - 'py-scikit-learn'
  - 'py-scipy@0.14:'
  - 'py-tables'
  - 'py-torch@0.4:'
  - 'py-torchvision'
$ spack concretize
...
$ cat spack.yaml
spack:
  specs:
  - gdal+python
  - python@3:+sqlite3
  - py-earthengine-api
  - py-flake8
  - py-matplotlib
  - py-numpy
  - py-pandas
  - 'py-pytest@3.9:'
  - py-scikit-learn
  - 'py-scipy@0.14:'
  - py-tables
  - 'py-torch@0.4:'
  - py-torchvision
  config: {}
  mirrors: {}
  modules:
    enable: []
  packages: {}
  repos: []
  upstreams: {}
  definitions: []
  view:
    default:
      root: /Users/Adam/Documents/UIUC/Research/Crops/crop_classification/.spack-env/view
      projections: {}
```
Removing quotes on specs containing `:` is one thing, but adding a root that is computer-specific makes it possible to maintain this file under version control.

@tgamblin @citibeth 


Discussion:
I agree, this is not nice.  Actually, I maintain a Spack fork on GitHub,
and the environments I use are maintained as `spack.yaml` files within it.

On Wed, Aug 7, 2019 at 4:28 PM Adam J. Stewart <notifications@github.com>
wrote:

> The way Spack Environments were explained to me was that if you maintained
> a project on GitHub, you could add a spack.yaml file to your repo that
> lists its dependencies. Then you could clone the project, cd into the
> directory, and run spack install or spack concretize; spack install to
> install all of its dependencies. However, spack concretize appears to
> modify spack.yaml, a file which is likely to be under version control. No
> other Spack command that I know of reorganizes your configuration files,
> and spack concretize certainly shouldn't either.
> Steps to reproduce the issue
>
> $ cat spack.yamlspack:  specs:  - 'gdal+python'  - 'python@3:+sqlite3'  - 'py-earthengine-api'  - 'py-flake8'  - 'py-matplotlib'  - 'py-numpy'  - 'py-pandas'  - 'py-pytest@3.9:'  - 'py-scikit-learn'  - 'py-scipy@0.14:'  - 'py-tables'  - 'py-torch@0.4:'  - 'py-torchvision'
> $ spack concretize...
> $ cat spack.yamlspack:  specs:  - gdal+python  - python@3:+sqlite3  - py-earthengine-api  - py-flake8  - py-matplotlib  - py-numpy  - py-pandas  - 'py-pytest@3.9:'  - py-scikit-learn  - 'py-scipy@0.14:'  - py-tables  - 'py-torch@0.4:'  - py-torchvision  config: {}  mirrors: {}  modules:    enable: []  packages: {}  repos: []  upstreams: {}  definitions: []  view:    default:      root: /Users/Adam/Documents/UIUC/Research/Crops/crop_classification/.spack-env/view      projections: {}
>
> Removing quotes on specs containing : is one thing, but adding a root
> that is computer-specific makes it possible to maintain this file under
> version control.
>
> @tgamblin <https://github.com/tgamblin> @citibeth
> <https://github.com/citibeth>
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/spack/spack/issues/12313?email_source=notifications&email_token=AAOVY55CMGE7FYTJYWLBPELQDMV6DA5CNFSM4IKD7YIKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HD7TFHA>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAOVY53CKEEGTNPQ7N64DV3QDMV6DANCNFSM4IKD7YIA>
> .
>

I stumbled on that too, and was about to open an issue. I don't think Spack should modify `spack.yaml` when producing `spack.lock`. It's a very weird use case to:
- Write manually your `spack.yaml`
- Concretize and produce `spack.lock`
- Find out that `spack.yaml` has been rewritten and new attributes have been added to it
Ultimately we decided `spack.yaml` could be modified by Spack. For example that `spack add` can add specs. Would it be annoying to maintain a separate copy (e.g. if you want to regenerate the environment from scratch)? We could also add options to copy the `spack.yaml` file and manage it separately.

Overall I can generally agree that this may not be desirable, but if @citibeth @adamjstewart @alalazo you could mention the use cases you have which this interferes with I think that would help us figure out how to address it.

> No other Spack command that I know of reorganizes your configuration files, and spack concretize certainly shouldn't either.

Most commands can modify `compilers.yaml`, and generally Spack isn't shy about reaching into any configuration file as long as it lives in a scope that it is aware of, but your point stands that this isn't necessarily desirable behavior for `spack.yaml`.
> if @citibeth @adamjstewart @alalazo you could mention the use cases you have which this interferes with I think that would help us figure out how to address it.

My use case is as follows. I have a GitHub repo for a research project I'm working on. This project depends on several Python and C libraries that I would like to install with Spack. I check a `spack.yaml` file into my Git repo and then I can clone and `spack install` and go. The issue with Spack modifying `spack.yaml` is that it is often under version control. 

Yes, commands like `spack compiler add` modify `compilers.yaml`, but this is expected. I would not expect `spack install` to modify `spack.yaml`. Honestly, I would be fine with Spack modifying `spack.yaml` as long as it didn't specify a computer-specific `root` inside. Because of that, the file is impossible to version control and share across multiple computers.
@scheibelp Commands like `spack add` are meant to modify `spack.yaml`, so it's absolutely fine to do it. What is unexpected is that `spack concretize` and `spack install`, that are meant to take `spack.yaml` as input, end up overwriting it. 
My use case is similar to @adamjstewart's.  In my case, I maintain a fork of Spack, and I put my environments in that fork as `spack.yaml` files:

https://github.com/citibeth/spack/tree/efischer/giss2/var/spack/environments

When I want to build the environment (from scratch), I do something like:
```
git pull
spack -e myenv concretize -f
spack -e myenv install
spack -e myenv env loads
```

Obviously, automatically overwriting files under version control is a bad idea.  When combined with Docker, it created some additional annoyances.  I ran something like the above in one Docker script to install the environment.  That prevented any future `git pull` commands on my Spack fork because `spack.yaml` had been changed.  In order to `git pull`, I first had to `git stash`.  But that had its own problems because the in-Docker-container git did not have proper usernames set up for that kind of stuff.  All in all, a big headache.



@scheibelp:
> Ultimately we decided spack.yaml could be modified by Spack. For example that spack add can add specs. Would it be annoying to maintain a separate copy (e.g. if you want to regenerate the environment from scratch)? We could also add options to copy the spack.yaml file and manage it separately.

This misses the point.  We want spack environments to be usable via the command line, yes, and there are commands like `spack add` and `spack remove` that need to add things to lists in the `spack.yaml` file.  I think their changes should be minimal as well (which is why we used ruamel to modify the file instead of `pyyaml`), but that's a separate issue.

This issue is about `spack concretize` and `spack install` which shouldn't have to touch `spack.yaml` at all.  They should just do their job and write `spack.lock`.
Oh looks like @alalazo already said the same thing 😄 
To put in in a wider context, `spack edit` edits `package.py` files.  But
that doesn't mean that Spack can auto-edit those files whenever it feels
like it.

>

I appreciate the responses and I think they clarify a few things:

* https://github.com/spack/spack/issues/12313#issuecomment-519651718 is ok with *some* automatic edits to `spack.yaml` but not hardcoded paths etc. which are totally wrong for version control
* https://github.com/spack/spack/issues/12313#issuecomment-519996564 appears to be totally against all modifications to `spack.yaml` that are not from commands like `spack add` (the analog to explicit configuration editing commands like `spack repo add`)

The first is absolutely avoidable and I 100% agree with it (and is probably simple to fix). The second may require more significant editing (but would be necessary if we want to avoid the concerns brought up in the second point). In other words perhaps this can be distilled into a set of requirements and those requirements addressed with separate PRs.
@scheibelp: I think we really want both of these.  I think it's worth making issues for them.

Resolving patch:
diff --git a/lib/spack/spack/cmd/release_jobs.py b/lib/spack/spack/cmd/release_jobs.py
--- a/lib/spack/spack/cmd/release_jobs.py
+++ b/lib/spack/spack/cmd/release_jobs.py
@@ -409,9 +409,7 @@ def find_matching_config(spec, ci_mappings):
 def release_jobs(parser, args):
     env = ev.get_env(args, 'release-jobs', required=True)
 
-    # FIXME: What's the difference between one that opens with 'spack'
-    # and one that opens with 'env'?  This will only handle the former.
-    yaml_root = env.yaml['spack']
+    yaml_root = ev.config_dict(env.yaml)
 
     if 'gitlab-ci' not in yaml_root:
         tty.die('Environment yaml does not have "gitlab-ci" section')
diff --git a/lib/spack/spack/environment.py b/lib/spack/spack/environment.py
--- a/lib/spack/spack/environment.py
+++ b/lib/spack/spack/environment.py
@@ -70,8 +70,7 @@
 # configuration settings.
 spack:
   # add package specs to the `specs` list
-  specs:
-  -
+  specs: []
   view: true
 """
 #: regex for validating enviroment names
@@ -391,20 +390,31 @@ def all_environments():
 
 
 def validate(data, filename=None):
+    # validating changes data by adding defaults. Return validated data
+    validate_data = copy.deepcopy(data)
+    # HACK to fully copy ruamel CommentedMap that doesn't provide copy method
+    import ruamel.yaml as yaml
+    setattr(
+        validate_data,
+        yaml.comments.Comment.attrib,
+        getattr(data, yaml.comments.Comment.attrib, yaml.comments.Comment())
+    )
+
     import jsonschema
     try:
-        spack.schema.Validator(spack.schema.env.schema).validate(data)
+        spack.schema.Validator(spack.schema.env.schema).validate(validate_data)
     except jsonschema.ValidationError as e:
         raise spack.config.ConfigFormatError(
             e, data, filename, e.instance.lc.line + 1)
+    return validate_data
 
 
 def _read_yaml(str_or_file):
     """Read YAML from a file for round-trip parsing."""
     data = syaml.load_config(str_or_file)
     filename = getattr(str_or_file, 'name', None)
-    validate(data, filename)
-    return data
+    default_data = validate(data, filename)
+    return (data, default_data)
 
 
 def _write_yaml(data, str_or_file):
@@ -444,6 +454,13 @@ def __init__(self, root, projections={}, select=[], exclude=[],
                                             for e in self.exclude)
         self.link = link
 
+    def __eq__(self, other):
+        return all([self.root == other.root,
+                    self.projections == other.projections,
+                    self.select == other.select,
+                    self.exclude == other.exclude,
+                    self.link == other.link])
+
     def to_dict(self):
         ret = {'root': self.root}
         if self.projections:
@@ -540,7 +557,7 @@ def __init__(self, path, init_file=None, with_view=None):
                     self._read_lockfile(f)
                     self._set_user_specs_from_lockfile()
                 else:
-                    self._read_manifest(f)
+                    self._read_manifest(f, raw_yaml=default_manifest_yaml)
         else:
             default_manifest = not os.path.exists(self.manifest_path)
             if default_manifest:
@@ -573,9 +590,13 @@ def __init__(self, path, init_file=None, with_view=None):
         # If with_view is None, then defer to the view settings determined by
         # the manifest file
 
-    def _read_manifest(self, f):
+    def _read_manifest(self, f, raw_yaml=None):
         """Read manifest file and set up user specs."""
-        self.yaml = _read_yaml(f)
+        if raw_yaml:
+            _, self.yaml = _read_yaml(f)
+            self.raw_yaml, _ = _read_yaml(raw_yaml)
+        else:
+            self.raw_yaml, self.yaml = _read_yaml(f)
 
         self.spec_lists = OrderedDict()
 
@@ -1333,6 +1354,9 @@ def write(self):
         # ensure path in var/spack/environments
         fs.mkdirp(self.path)
 
+        yaml_dict = config_dict(self.yaml)
+        raw_yaml_dict = config_dict(self.raw_yaml)
+
         if self.specs_by_hash:
             # ensure the prefix/.env directory exists
             fs.mkdirp(self.env_subdir_path)
@@ -1367,8 +1391,7 @@ def write(self):
                 # The primary list is handled differently
                 continue
 
-            conf = config_dict(self.yaml)
-            active_yaml_lists = [l for l in conf.get('definitions', [])
+            active_yaml_lists = [l for l in yaml_dict.get('definitions', [])
                                  if name in l and
                                  _eval_conditional(l.get('when', 'True'))]
 
@@ -1392,10 +1415,11 @@ def write(self):
         # put the new user specs in the YAML.
         # This can be done directly because there can't be multiple definitions
         # nor when clauses for `specs` list.
-        yaml_spec_list = config_dict(self.yaml).setdefault(user_speclist_name,
-                                                           [])
+        yaml_spec_list = yaml_dict.setdefault(user_speclist_name,
+                                              [])
         yaml_spec_list[:] = self.user_specs.yaml_list
 
+        # Construct YAML representation of view
         default_name = default_view_name
         if self.views and len(self.views) == 1 and default_name in self.views:
             path = self.default_view.root
@@ -1412,18 +1436,26 @@ def write(self):
         else:
             view = False
 
-        yaml_dict = config_dict(self.yaml)
-        if view is not True:
-            # The default case is to keep an active view inside of the
-            # Spack environment directory. To avoid cluttering the config,
-            # we omit the setting in this case.
-            yaml_dict['view'] = view
-        elif 'view' in yaml_dict:
-            del yaml_dict['view']
+        yaml_dict['view'] = view
+
+        # Remove yaml sections that are shadowing defaults
+        # construct garbage path to ensure we don't find a manifest by accident
+        bare_env = Environment(os.path.join(self.manifest_path, 'garbage'),
+                               with_view=self.view_path_default)
+        keys_present = list(yaml_dict.keys())
+        for key in keys_present:
+            if yaml_dict[key] == config_dict(bare_env.yaml).get(key, None):
+                if key not in raw_yaml_dict:
+                    del yaml_dict[key]
 
         # if all that worked, write out the manifest file at the top level
-        with fs.write_tmp_and_move(self.manifest_path) as f:
-            _write_yaml(self.yaml, f)
+        # Only actually write if it has changed or was never written
+        changed = self.yaml != self.raw_yaml
+        written = os.path.exists(self.manifest_path)
+        if changed or not written:
+            self.raw_yaml = copy.deepcopy(self.yaml)
+            with fs.write_tmp_and_move(self.manifest_path) as f:
+                _write_yaml(self.yaml, f)
 
         # TODO: for operations that just add to the env (install etc.) this
         # could just call update_view
diff --git a/lib/spack/spack/schema/env.py b/lib/spack/spack/schema/env.py
--- a/lib/spack/spack/schema/env.py
+++ b/lib/spack/spack/schema/env.py
@@ -63,6 +63,7 @@
                 {
                     'include': {
                         'type': 'array',
+                        'default': [],
                         'items': {
                             'type': 'string'
                         },


Test patch:
diff --git a/lib/spack/spack/test/cmd/env.py b/lib/spack/spack/test/cmd/env.py
--- a/lib/spack/spack/test/cmd/env.py
+++ b/lib/spack/spack/test/cmd/env.py
@@ -1763,3 +1763,13 @@ def test_duplicate_packages_raise_when_concretizing_together():
 
     with pytest.raises(ev.SpackEnvironmentError, match=r'cannot contain more'):
         e.concretize()
+
+
+def test_env_write_only_non_default():
+    print(env('create', 'test'))
+
+    e = ev.read('test')
+    with open(e.manifest_path, 'r') as f:
+        yaml = f.read()
+
+    assert yaml == ev.default_manifest_yaml