Repository: spack/spack

Issue:
Invalid character error when referencing definitions in an environment's spec
When leveraging the `definitions` found int he Spack Stacks enhancement I encounter a `Invalid character` error after successful concretization. This only occurs in an environment with a definition reference in the spec (e.g. `$first` from the https://spack.readthedocs.io/en/latest/environments.html#spec-list-references).

This occurs with both Python 2.7.5 and 3.7.3

### Steps to reproduce the issue

```console
$ cat spack.yaml
spack:
  definitions:
    - first: [libelf, libdwarf]
    - compilers: ['%gcc@4.8.5']
    - second:
      - $first
      - matrix:
        - [zlib]
        - [$compilers]
  specs:
    - $second
    - cmake

$ spack concretize
==> Error: Invalid character
```

### Error Message
```console
$ spack --debug concretize
...
==> [2019-07-23-09:09:34.269444] LexError: Invalid character
==> [2019-07-23-09:09:34.269503] Error: Invalid character
Traceback (most recent call last):
  File "/Users/pbryant/Development/40-ECP/spack/lib/spack/spack/main.py", line 697, in main
    return _invoke_command(command, parser, args, unknown)
  File "/Users/pbryant/Development/40-ECP/spack/lib/spack/spack/main.py", line 447, in _invoke_command
    return_val = command(parser, args)
  File "/Users/pbryant/Development/40-ECP/spack/lib/spack/spack/cmd/concretize.py", line 22, in concretize
    env.write()
  File "/Users/pbryant/Development/40-ECP/spack/lib/spack/spack/environment.py", line 1242, in write
    Spec(s) in speclist.specs]
  File "/Users/pbryant/Development/40-ECP/spack/lib/spack/spack/spec.py", line 949, in __init__
    spec_list = SpecParser(self).parse(spec_like)
  File "/Users/pbryant/Development/40-ECP/spack/lib/spack/spack/parse.py", line 151, in parse
    self.setup(text)
  File "/Users/pbryant/Development/40-ECP/spack/lib/spack/spack/parse.py", line 148, in setup
    self.push_tokens(self.lexer.lex(text))
  File "/Users/pbryant/Development/40-ECP/spack/lib/spack/spack/parse.py", line 86, in lex
    tokens = self.lex_word(word)
  File "/Users/pbryant/Development/40-ECP/spack/lib/spack/spack/parse.py", line 79, in lex_word
    raise LexError("Invalid character", word, word.index(remainder))
spack.parse.LexError: Invalid character
```


Discussion:

Resolving patch:
diff --git a/lib/spack/spack/environment.py b/lib/spack/spack/environment.py
--- a/lib/spack/spack/environment.py
+++ b/lib/spack/spack/environment.py
@@ -1237,9 +1237,11 @@ def write(self):
             # Remove any specs in yaml that are not in internal representation
             for ayl in active_yaml_lists:
                 # If it's not a string, it's a matrix. Those can't have changed
+                # If it is a string that starts with '$', it's a reference.
+                # Those also can't have changed.
                 ayl[name][:] = [s for s in ayl.setdefault(name, [])
-                                if not isinstance(s, six.string_types) or
-                                Spec(s) in speclist.specs]
+                                if (not isinstance(s, six.string_types)) or
+                                s.startswith('$') or Spec(s) in speclist.specs]
 
             # Put the new specs into the first active list from the yaml
             new_specs = [entry for entry in speclist.yaml_list


Test patch:
diff --git a/lib/spack/spack/test/cmd/env.py b/lib/spack/spack/test/cmd/env.py
--- a/lib/spack/spack/test/cmd/env.py
+++ b/lib/spack/spack/test/cmd/env.py
@@ -965,6 +965,29 @@ def test_stack_yaml_definitions(tmpdir):
         assert Spec('callpath') in test.user_specs
 
 
+@pytest.mark.regression('12095')
+def test_stack_yaml_definitions_write_reference(tmpdir):
+    filename = str(tmpdir.join('spack.yaml'))
+    with open(filename, 'w') as f:
+        f.write("""\
+env:
+  definitions:
+    - packages: [mpileaks, callpath]
+    - indirect: [$packages]
+  specs:
+    - $packages
+""")
+    with tmpdir.as_cwd():
+        env('create', 'test', './spack.yaml')
+
+        with ev.read('test'):
+            concretize()
+        test = ev.read('test')
+
+        assert Spec('mpileaks') in test.user_specs
+        assert Spec('callpath') in test.user_specs
+
+
 def test_stack_yaml_add_to_list(tmpdir):
     filename = str(tmpdir.join('spack.yaml'))
     with open(filename, 'w') as f: