Repository: spack/spack

Issue:
Hash error when trying to install or activate an environment
<!-- Explain, in a clear and concise way, the command you ran and the result you were trying to achieve.
Example: "I ran `spack find` to list all the installed packages and ..." -->

Updated to spack to v0.15. Rewrote the environment to comply to the new format.  Tried to reinstall the environment and it fails part way through the installation of the environment.  At that point it also fails to activate the environment. 

### Steps to reproduce the issue

```console
./scripts/install_lbann.sh -d
spack env create lbann-dev-power9le /tmp/tmp.sNWlgHKVtI
==> Updating view at /autofs/nccs-svm1_home1/vanessen/spack.git/var/spack/environments/lbann-dev-power9le/.spack-env/view
==> Created environment 'lbann-dev-power9le' in /autofs/nccs-svm1_home1/vanessen/spack.git/var/spack/environments/lbann-dev-power9le
==> You can activate this environment with:
==>   spack env activate lbann-dev-power9le
spack env activate -p lbann-dev-power9le
spack install
<some packages install>
...
==> Installing py-pytest
==> No binary for py-pytest found: installing from source
==> Fetching https://spack-llnl-mirror.s3-us-west-2.amazonaws.com/_source-cache/archive/1d/1d122e8be54d1a709e56f82e2d85dcba3018313d64647f38a91aec88c239b600.tar.gz
######################################################################## 100.0%
==> py-pytest: Executing phase: 'build'
==> py-pytest: Executing phase: 'install'
[+] /ccs/home/vanessen/spack_installed_packages/opt/spack/linux-rhel7-power9le/gcc-8.1.1/py-pytest-5.3.4-ln2l2mdp4jvtfctmfalqlzcjkomctj62
==> gcc@8.1.1 : has external module in ['gcc/8.1.1']
==> Error: '3easygztiqbtrszh4rxogt6wb5r7nvra'...
```

### Error Message

<!-- If Spack reported an error, provide the error message. If it did not report an error but the output appears incorrect, provide the incorrect output. If there was no error message and no output but the result is incorrect, describe how it does not match what you expect. -->
```console
vanessen@login4 [lbann.git] %> spack env activate lbann-dev-power9le
==> Error: '3easygztiqbtrszh4rxogt6wb5r7nvra'
vanessen@login4 [lbann.git] %> spack -d env activate lbann-dev-power9le
==> [2020-08-27-18:17:23.300408] Imported env from built-in commands
==> [2020-08-27-18:17:23.304057] Imported env from built-in commands
==> [2020-08-27-18:17:23.305158] Reading config file /autofs/nccs-svm1_home1/vanessen/spack.git/etc/spack/defaults/config.yaml
==> [2020-08-27-18:17:23.338516] Reading config file /ccs/home/vanessen/.spack/config.yaml
==> [2020-08-27-18:17:23.840768] Using environmennt 'lbann-dev-power9le'
==> [2020-08-27-18:17:23.841232] Reading config file /autofs/nccs-svm1_home1/vanessen/spack.git/var/spack/environments/lbann-dev-power9le/spack.yaml
==> [2020-08-27-18:17:24.030551] DATABASE LOCK TIMEOUT: 3s
==> [2020-08-27-18:17:24.030634] PACKAGE LOCK TIMEOUT: No timeout
Traceback (most recent call last):
  File "/ccs/home/vanessen/spack.git/bin/spack", line 64, in <module>
    sys.exit(spack.main.main())
  File "/autofs/nccs-svm1_home1/vanessen/spack.git/lib/spack/spack/main.py", line 762, in main
    return _invoke_command(command, parser, args, unknown)
  File "/autofs/nccs-svm1_home1/vanessen/spack.git/lib/spack/spack/main.py", line 490, in _invoke_command
    return_val = command(parser, args)
  File "/autofs/nccs-svm1_home1/vanessen/spack.git/lib/spack/spack/cmd/env.py", line 509, in env
    action(args)
  File "/autofs/nccs-svm1_home1/vanessen/spack.git/lib/spack/spack/cmd/env.py", line 120, in env_activate
    prompt=env_prompt if args.prompt else None
  File "/autofs/nccs-svm1_home1/vanessen/spack.git/lib/spack/spack/environment.py", line 181, in activate
    cmds += env.add_default_view_to_shell(shell)
  File "/autofs/nccs-svm1_home1/vanessen/spack.git/lib/spack/spack/environment.py", line 1198, in add_default_view_to_shell
    mods, errors = self._env_modifications_for_default_view()
  File "/autofs/nccs-svm1_home1/vanessen/spack.git/lib/spack/spack/environment.py", line 1173, in _env_modifications_for_default_view
    for _, spec in self.concretized_specs():
  File "/autofs/nccs-svm1_home1/vanessen/spack.git/lib/spack/spack/environment.py", line 1351, in concretized_specs
    yield (s, self.specs_by_hash[h])
KeyError: '3easygztiqbtrszh4rxogt6wb5r7nvra'
```

### Information on your system

<!-- Please include the output of `spack debug report` -->

<!-- If you have any relevant configuration detail (custom `packages.yaml` or `modules.yaml`, etc.) you can add that here as well. -->

### Additional information

<!-- These boxes can be checked by replacing [ ] with [x] or by clicking them after submitting the issue. -->
- [ ] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [ ] I have searched the issues of this repo and believe this is not a duplicate
- [ ] I have run the failing commands in debug mode and reported the output

<!-- We encourage you to try, as much as possible, to reduce your problem to the minimal example that still reproduces the issue. That would help us a lot in fixing it quickly and effectively!

If you want to ask a question about the tool (how to use it, what it can currently do, etc.), try the `#general` channel on our Slack first. We have a welcoming community and chances are you'll get your reply faster and without opening an issue.

Other than that, thanks for taking the time to contribute to Spack! -->

Investigate environment activation error 18338
See: https://github.com/spack/spack/issues/18338

@bvanessen using this PR, can you try activating you environment with `-d` like `spack -d env activate <env-name>` and show me the output?

This will likely require some iteration to understand the source of the problem. While I have trouble replicating it myself, I think this is the only way forward.


Discussion:
I am experiencing similar problems. However, in my case the environment has been created with an earlier version of spack (before the update to the new config file format). I am not sure if this information is of any help, but by chance I had an environment activated before updating spack and after updating I can't deactivate that environment any longer (via `spack env deactivate`).

I get a similarly cryptic error message, only the printed hash is different.
I believe that part of my problem is related to the line that says 

view: true

That is at the end of my spack.yaml file. It is added during the creation of the environment and if I delete it and reconcretize the environment it seems to work. 

Brian C. Van Essen
vanessen1@llnl.gov
(w) 925-422-9300
(c) 925-290-5470
---
Sent from my iPhone

> On Aug 28, 2020, at 1:50 AM, Thomas Madlener <notifications@github.com> wrote:
> 
> ﻿
> I am experiencing similar problems. However, in my case the environment has been created with an earlier version of spack (before the update to the new config file format). I am not sure if this information is of any help, but by chance I had an environment activated before updating spack and after updating I can't deactivate that environment any longer (via spack env deactivate).
> 
> I get a similarly cryptic error message, only the printed hash is different.
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.

It looks like creating a new environment with explicitly disabling the creation of a view also allows me to create, activate and deactivate environments. However, setting `view: false` in the `spack.yaml` file after an environment has been created, concretized and all packages installed with an earlier version of spack has no effect.

The hash that is printed in the error message seems to be the one that belongs to the first package in `spack.lock` that has been installed via spack (i.e. no system packages, even if they appear earlier in the list).

I am not entirely sure on how to best proceed from here, or what information would be the most helpful for the developers. If there is anything that I can do from my side, let me know.
I did find the problematic has in the lock file as well it is the first one:
```
 "roots": [
  {
   "hash": "3easygztiqbtrszh4rxogt6wb5r7nvra",
   "spec": "aluminum"
  },
```
It looks like all of the hash values in the lock file do not correspond to the hash values of the installed packages.
@tmadlener one thing I wanted to check about https://github.com/spack/spack/issues/18338#issuecomment-682408670

@bvanessen reported that this issue comes up if he tries to make an environment from scratch:

```
spack env create <env-name> path/to/spack.yaml
spack env activate <env-name>
spack install
```

Does this also occur for you? Or do activation errors only occur when activating preexisting environments?
Also @tmadlener is your stack trace the same as https://github.com/spack/spack/issues/18338#issue-687624016, or different?
Hi @scheibelp, I get the same stack trace as above.
> Updated to spack to v0.15. Rewrote the environment to comply to the new format.

Am I correct that Spack v0.15.X should not be affected by this issue, since the change in `packages.yaml` is only in `develop` at the moment? 
> Am I correct that Spack v0.15.X should not be affected by this issue, since the change in packages.yaml is only in develop at the moment?

Yes, at least for me checking out `0.15.4` (and reverting the changes in the `packages.yaml` config file) allows me to activate and deactivate environments again. Also concretizing environments seems to work again.
I guess to be precise, I was using head of develop, not the explicit 0.15.4 release.

Brian C. Van Essen
vanessen1@llnl.gov
(w) 925-422-9300
(c) 925-290-5470





> On Aug 31, 2020, at 7:06 AM, Thomas Madlener <notifications@github.com> wrote:
> 
> 
> Am I correct that Spack v0.15.X should not be affected by this issue, since the change in packages.yaml is only in develop at the moment?
> 
> Yes, at least for me checking out 0.15.4 (and reverting the changes in the packages.yaml config file) allows me to activate and deactivate environments again. Also concretizing environments seems to work again.
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.
> 


@tmadlener does removing spaces after module file names and/or enclosing config file values in quotes resolve the issue as mentioned at https://github.com/spack/spack/pull/18368#issuecomment-683580919?

If not, would you be able to post your config file?

@alalazo mentioned being able to fix this issue in the next day so I am assigning him.
FYI, removing the space after the module file names solved my issues.

Brian C. Van Essen
vanessen1@llnl.gov
(w) 925-422-9300
(c) 925-290-5470





> On Aug 31, 2020, at 7:31 PM, Peter Scheibel <notifications@github.com> wrote:
> 
> 
> @tmadlener does removing spaces after module file names and/or enclosing config file values in quotes resolve the issue as mentioned at #18368 (comment)?
> 
> If not, would you be able to post your config file?
> 
> @alalazo mentioned being able to fix this issue in the next day so I am assigning him.
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.
> 


Trying to debug this simple environment:
```yaml
spack:
  specs:
  - openjpeg
  packages:
    cmake:
      externals:
      - spec: cmake@3.18.2
        modules:
        - cmake/1.18.2

  concretization: together
```
which exhibits the same behavior as reported in the description. It seems that doing:
```console
$ spack --env . concretize
```
we read the environment 4 times and we rewrite the `spack.yaml` (so  I think there's a regression on #13546). 
> @bvanessen reported that this issue comes up if he tries to make an environment from scratch:
>```
>spack env create <env-name> path/to/spack.yaml
>spack env activate <env-name>
>spack install
>```
>Does this also occur for you? Or do activation errors only occur when activating preexisting environments?

For me activation errors only occur when I am trying to activate a preexisting environment. Creating one from scratch using the exact same yaml file I used for the original one seems to work just fine.

However, interestingly enough, concretizing the newly created environment yields different hashes for some of the packages. I would have naively have expected that these are stable between spack versions. But maybe that assumption is wrong?
There are further issues reproducible with the environment posted above:
```console
$ spack --env . concretize 
==> Concretized openjpeg
 -   526rapi  openjpeg@2.3.1%gcc@10.1.0 build_type=RelWithDebInfo arch=linux-ubuntu18.04-broadwell
 -   qevuoda      ^cmake@3.18.2%gcc@10.1.0~doc+ncurses+openssl+ownlibs~qt arch=linux-ubuntu18.04-broadwell
 -   kepdmuv      ^zlib@1.2.11%gcc@10.1.0+optimize+pic+shared arch=linux-ubuntu18.04-broadwell

==> Updating view at /home/culpo/tmp/spack-issues/18338/.spack-env/view

$ ls spack.{yaml,lock}
spack.lock  spack.yaml

$ spack --env . concretize 
==> Error: '546hepoobjp5st4xioqkllziy2tbceoz'

$ ls spack.{yaml,lock}
ls: cannot access 'spack.lock': No such file or directory
spack.yaml
```
On a failed command the previously existing lockfile is deleted from disk.
I think I have a full diagnosis. The issue is particularly difficult to trace, since in memory there's no sign of newlines for the `modules` attribute. In fact one can stop and debug and see the following:
```console
>>> self['cmake'].external_modules
['cmake/1.18.2']
```
for both the case where we have a lockfile and where we don't. Anyhow, when we compute the `build_hash` of a spec we pass through the YAML representation of the spec, exactly at this line:

https://github.com/spack/spack/blob/3a5746c6c75d4187e0e6dfc3665a2e34c30de37a/lib/spack/spack/spec.py#L1386-L1387

When we don't have a lockfile present the node dictionary passed to the `syaml.dump` function has a newline in it, which comes from a YAML "comment" in a `CommentedSeq` object (comments are reflected in the dump). When we read from a lockfile the type of the same attribute changes and it's just a Python list, with no attached comments. The new line is thus not present in the dump we use to compute the hash and so the error.
The output for the debugging of activation is in the original bug report.  Here is what happens if I am in the environment and try to deactivate it (after toggling the state of the view field)

```
[lbann-dev-power9le] vanessen@login4 [lbann.git] %> spack -d env deactivate
==> [2020-08-28-17:14:38.413048] Imported env from built-in commands
==> [2020-08-28-17:14:38.416703] Imported env from built-in commands
==> [2020-08-28-17:14:38.420951] DATABASE LOCK TIMEOUT: 3s
==> [2020-08-28-17:14:38.421013] PACKAGE LOCK TIMEOUT: No timeout
Traceback (most recent call last):
  File "/ccs/home/vanessen/spack.git/bin/spack", line 64, in <module>
    sys.exit(spack.main.main())
  File "/autofs/nccs-svm1_home1/vanessen/spack.git/lib/spack/spack/main.py", line 762, in main
    return _invoke_command(command, parser, args, unknown)
  File "/autofs/nccs-svm1_home1/vanessen/spack.git/lib/spack/spack/main.py", line 490, in _invoke_command
    return_val = command(parser, args)
  File "/autofs/nccs-svm1_home1/vanessen/spack.git/lib/spack/spack/cmd/env.py", line 509, in env
    action(args)
  File "/autofs/nccs-svm1_home1/vanessen/spack.git/lib/spack/spack/cmd/env.py", line 160, in env_deactivate
    cmds = ev.deactivate(shell=args.shell)
  File "/autofs/nccs-svm1_home1/vanessen/spack.git/lib/spack/spack/environment.py", line 247, in deactivate
    cmds += _active_environment.rm_default_view_from_shell(shell)
  File "/autofs/nccs-svm1_home1/vanessen/spack.git/lib/spack/spack/environment.py", line 1220, in rm_default_view_from_shell
    mods, _ = self._env_modifications_for_default_view(reverse=True)
  File "/autofs/nccs-svm1_home1/vanessen/spack.git/lib/spack/spack/environment.py", line 1173, in _env_modifications_for_default_view
    for _, spec in self.concretized_specs():
  File "/autofs/nccs-svm1_home1/vanessen/spack.git/lib/spack/spack/environment.py", line 1351, in concretized_specs
    yield (s, self.specs_by_hash[h])
KeyError: '3easygztiqbtrszh4rxogt6wb5r7nvra'
```
Oh sorry, I miss understood.  Let me pull the PR.

Here is the updated output:

```
vanessen@login4 [lbann.git] %> spack -d env activate -p lbann-dev-power9le
==> [2020-08-28-17:20:02.815264] Imported env from built-in commands
==> [2020-08-28-17:20:02.818880] Imported env from built-in commands
==> [2020-08-28-17:20:02.820220] Reading config file /autofs/nccs-svm1_home1/vanessen/spack.git/etc/spack/defaults/config.yaml
==> [2020-08-28-17:20:02.853968] Reading config file /ccs/home/vanessen/.spack/config.yaml
==> [2020-08-28-17:20:03.358944] Using environmennt 'lbann-dev-power9le'
==> [2020-08-28-17:20:03.359442] Reading config file /autofs/nccs-svm1_home1/vanessen/spack.git/var/spack/environments/lbann-dev-power9le/spack.yaml
==> [2020-08-28-17:20:03.551411] DATABASE LOCK TIMEOUT: 3s
==> [2020-08-28-17:20:03.551505] PACKAGE LOCK TIMEOUT: No timeout
==> [2020-08-28-17:20:03.665376] [18338] concretized specs:
        spec: aluminum
                hash: 3easygztiqbtrszh4rxogt6wb5r7nvra
                MISSING
        spec: hydrogen
                hash: os7an62k754vsti44qiyt2x6miyipvno
                MISSING
        spec: cereal
                hash: jkhurievuy4ast3uttjv5ugcllcfzp4b
        spec: clara
                hash: pjnrr5h2wwrpharvwfxvpanhtnbzrqqx
        spec: cnpy
                hash: 5ivz3nf3ystrzgxzuhhomeql5itn7cfy
        spec: conduit
                hash: pe6qrrowe3dvlz55bh3wovs544aqnnwn
        spec: hwloc
                hash: znq665g7jo5bkj62aa7vhkajdrrc5qqc
        spec: opencv
                hash: k3thbvvkyb6krvfyh73rfwzxgkuv4vd6
        spec: zlib
                hash: oahdokybko632taim7jrn65uaktx3caf
        spec: cudnn
                hash: 6syesbuj34lqheadekgw6boxtko5xmru
        spec: cuda
                hash: wy2kjyohhwdtbgaeidopkgsw5zz6l6pe
        spec: nccl
                hash: ijyobdkhlwscsocsae3fejrp5u3dle44
        spec: py-numpy
                hash: qrewhretfzn3ayragrz6zstymevsrx2r
        spec: py-protobuf
                hash: jyg2i32q2s7jrf5h67ajrky2fxeqqh6p
        spec: py-setuptools
                hash: hubb7w6qb53avo4k4trr2y6voubvzi62
        spec: mpi
                hash: jt2s5qmwaxpnzhnkvmgxn3cch7lyka32
        spec: catch2
                hash: crhfuzxrliz2yf6y2doslllusyl7gjuk
        spec: cmake
                hash: 6lezyty5xseynkep3ecllvuo2x5psuth
        spec: ninja
                hash: lmwqhz26l2qcg25i2dr3kpaxg4nz23az
        spec: python
                hash: red7gcie24nsoez25t2pv4bgchmfdnl5
        spec: py-pytest
                hash: i5lcdudmiweelzmlc6utgn5l2ii6zgq6
        spec: gcc
                hash: z4spojzeqvdddya3jutcxo7b4yb7lj34

Traceback (most recent call last):
  File "/ccs/home/vanessen/spack.git/bin/spack", line 64, in <module>
    sys.exit(spack.main.main())
  File "/autofs/nccs-svm1_home1/vanessen/spack.git/lib/spack/spack/main.py", line 762, in main
    return _invoke_command(command, parser, args, unknown)
  File "/autofs/nccs-svm1_home1/vanessen/spack.git/lib/spack/spack/main.py", line 490, in _invoke_command
    return_val = command(parser, args)
  File "/autofs/nccs-svm1_home1/vanessen/spack.git/lib/spack/spack/cmd/env.py", line 509, in env
    action(args)
  File "/autofs/nccs-svm1_home1/vanessen/spack.git/lib/spack/spack/cmd/env.py", line 120, in env_activate
    prompt=env_prompt if args.prompt else None
  File "/autofs/nccs-svm1_home1/vanessen/spack.git/lib/spack/spack/environment.py", line 181, in activate
    cmds += env.add_default_view_to_shell(shell)
  File "/autofs/nccs-svm1_home1/vanessen/spack.git/lib/spack/spack/environment.py", line 1198, in add_default_view_to_shell
    mods, errors = self._env_modifications_for_default_view()
  File "/autofs/nccs-svm1_home1/vanessen/spack.git/lib/spack/spack/environment.py", line 1173, in _env_modifications_for_default_view
    for _, spec in self.concretized_specs():
  File "/autofs/nccs-svm1_home1/vanessen/spack.git/lib/spack/spack/environment.py", line 1359, in concretized_specs
    yield (s, self.specs_by_hash[h])
KeyError: '3easygztiqbtrszh4rxogt6wb5r7nvra'
```
Here is an updated output

```
vanessen@login4 [lbann.git] %> spack -d env activate -p lbann-dev-power9le
==> [2020-08-28-18:37:55.537456] Imported env from built-in commands
==> [2020-08-28-18:37:55.541079] Imported env from built-in commands
==> [2020-08-28-18:37:55.543748] Reading config file /autofs/nccs-svm1_home1/vanessen/spack.git/etc/spack/defaults/config.yaml
==> [2020-08-28-18:37:55.577837] Reading config file /ccs/home/vanessen/.spack/config.yaml
==> [2020-08-28-18:37:55.796006] [18338] Found 3easygz: aluminum@0.4.0%gcc@8.1.1+cuda+cuda_rma+ht+nccl build_type=RelWithDebInfo cuda_arch=none arch=linux-rhel7-power9le
==> [2020-08-28-18:37:55.796075] [18338] Read aluminum
        3easygz
==> [2020-08-28-18:37:55.796388] [18338] Read cmake
        6lezyty
==> [2020-08-28-18:37:55.796635] [18338] Read cuda
        2nkrdnx
==> [2020-08-28-18:37:55.796949] [18338] Read hwloc
        znq665g
==> [2020-08-28-18:37:55.797208] [18338] Read nccl
        6umgh6e
==> [2020-08-28-18:37:55.797478] [18338] Read rdma-core
        igaqncs
==> [2020-08-28-18:37:55.797721] [18338] Read ninja
        lmwqhz2
==> [2020-08-28-18:37:55.798126] [18338] Read python
        red7gci
==> [2020-08-28-18:37:55.798364] [18338] Read gdbm
        r6ub4zx
==> [2020-08-28-18:37:55.798611] [18338] Read readline
        up3waz7
==> [2020-08-28-18:37:55.798871] [18338] Read ncurses
        neawwjf
==> [2020-08-28-18:37:55.799115] [18338] Read pkgconf
        mw6meid
==> [2020-08-28-18:37:55.799409] [18338] Read gettext
        wyofqaz
==> [2020-08-28-18:37:55.799709] [18338] Read bzip2
        wtadsde
==> [2020-08-28-18:37:55.799958] [18338] Read diffutils
        htmbww4
==> [2020-08-28-18:37:55.800204] [18338] Read libiconv
        66s2tyr
==> [2020-08-28-18:37:55.800457] [18338] Read libxml2
        qljmos5
==> [2020-08-28-18:37:55.800695] [18338] Read xz
        yxoi6hf
==> [2020-08-28-18:37:55.800953] [18338] Read zlib
        oahdoky
==> [2020-08-28-18:37:55.801186] [18338] Read tar
        e5xoed7
==> [2020-08-28-18:37:55.801445] [18338] Read libffi
        xfldrg6
==> [2020-08-28-18:37:55.801713] [18338] Read openssl
        mpj5mfw
==> [2020-08-28-18:37:55.801973] [18338] Read perl
        bkhloma
==> [2020-08-28-18:37:55.802229] [18338] Read berkeley-db
        26zoyqb
==> [2020-08-28-18:37:55.802501] [18338] Read sqlite
        bmd7542
==> [2020-08-28-18:37:55.802763] [18338] Read spectrum-mpi
        jt2s5qm
==> [2020-08-28-18:37:55.803590] [18338] Found os7an62: hydrogen@1.4.0%gcc@8.1.1+al+cuda~half+hybrid+int64~int64_blas~mpfr~omp_taskloops+openmp_blas~quad~scalapack+shared~test blas=openblas build_type=Release cuda_arch=none arch=linux-rhel7-power9le
==> [2020-08-28-18:37:55.803651] [18338] Read hydrogen
        os7an62
==> [2020-08-28-18:37:55.803889] [18338] Read cub
        u3763vd
==> [2020-08-28-18:37:55.804186] [18338] Read openblas
        hpzkkjt
==> [2020-08-28-18:37:55.804466] [18338] Read cereal
        jkhurie
==> [2020-08-28-18:37:55.804730] [18338] Read clara
        pjnrr5h
==> [2020-08-28-18:37:55.805037] [18338] Read cnpy
        5ivz3nf
==> [2020-08-28-18:37:55.805368] [18338] Read conduit
        pe6qrro
==> [2020-08-28-18:37:55.805695] [18338] Read hdf5
        cl3pxxf
==> [2020-08-28-18:37:55.805944] [18338] Read py-mpi4py
        4db3ftj
==> [2020-08-28-18:37:55.806202] [18338] Read py-setuptools
        hubb7w6
==> [2020-08-28-18:37:55.806484] [18338] Read py-numpy
        qrewhre
==> [2020-08-28-18:37:55.807041] [18338] Read opencv
        k3thbvv
==> [2020-08-28-18:37:55.807305] [18338] Read libjpeg-turbo
        pny6rlj
==> [2020-08-28-18:37:55.807542] [18338] Read nasm
        bsdxh3e
==> [2020-08-28-18:37:55.807784] [18338] Read libpng
        wjno6mr
==> [2020-08-28-18:37:55.808028] [18338] Read libtiff
        ollqlqd
==> [2020-08-28-18:37:55.808291] [18338] Read cudnn
        6syesbu
==> [2020-08-28-18:37:55.808554] [18338] Read py-protobuf
        jyg2i32
==> [2020-08-28-18:37:55.808827] [18338] Read protobuf
        56gxfom
==> [2020-08-28-18:37:55.809068] [18338] Read py-six
        rkx2vxw
==> [2020-08-28-18:37:55.809328] [18338] Read catch2
        crhfuzx
==> [2020-08-28-18:37:55.809625] [18338] Read py-pytest
        i5lcdud
==> [2020-08-28-18:37:55.809875] [18338] Read py-attrs
        dzgu4iu
==> [2020-08-28-18:37:55.810153] [18338] Read py-importlib-metadata
        22kr35d
==> [2020-08-28-18:37:55.810430] [18338] Read py-setuptools-scm
        4zns3fr
==> [2020-08-28-18:37:55.810674] [18338] Read py-zipp
        px4ihty
==> [2020-08-28-18:37:55.810943] [18338] Read py-more-itertools
        buovk6v
==> [2020-08-28-18:37:55.811198] [18338] Read py-packaging
        5dj6eyy
==> [2020-08-28-18:37:55.811455] [18338] Read py-pyparsing
        z4h5znb
==> [2020-08-28-18:37:55.811710] [18338] Read py-pluggy
        c4cyxwr
==> [2020-08-28-18:37:55.811948] [18338] Read py-py
        fxmhplr
==> [2020-08-28-18:37:55.812204] [18338] Read py-wcwidth
        jwmqllx
==> [2020-08-28-18:37:55.812508] [18338] Read gcc
        rbskn7z
==> [2020-08-28-18:37:55.928996] aluminum
        Calculated hash: j7ahykg2ce3c535fsajrpog5vb7pimol
        Hash from file: 3easygztiqbtrszh4rxogt6wb5r7nvra
        HASH DIFFERENCE
==> [2020-08-28-18:37:55.929071] cuda
        Calculated hash: wy2kjyohhwdtbgaeidopkgsw5zz6l6pe
        Hash from file: 2nkrdnx7d7og5su6sjbbjqvg6vrh7jgv
        HASH DIFFERENCE
==> [2020-08-28-18:37:55.929138] nccl
        Calculated hash: ijyobdkhlwscsocsae3fejrp5u3dle44
        Hash from file: 6umgh6ek7lhdxf3dgpsjnua6r7abnpbw
        HASH DIFFERENCE
==> [2020-08-28-18:37:55.945099] hydrogen
        Calculated hash: ln3pn3yklerydnpx6yk6uupbdubeywsb
        Hash from file: os7an62k754vsti44qiyt2x6miyipvno
        HASH DIFFERENCE
==> [2020-08-28-18:37:56.097072] gcc
        Calculated hash: z4spojzeqvdddya3jutcxo7b4yb7lj34
        Hash from file: rbskn7zrgjxd3ztcldlumuvu35xfd7xj
        HASH DIFFERENCE
==> [2020-08-28-18:37:56.097634] Using environmennt 'lbann-dev-power9le'
==> [2020-08-28-18:37:56.098112] Reading config file /autofs/nccs-svm1_home1/vanessen/spack.git/var/spack/environments/lbann-dev-power9le/spack.yaml
==> [2020-08-28-18:37:56.287982] DATABASE LOCK TIMEOUT: 3s
==> [2020-08-28-18:37:56.288059] PACKAGE LOCK TIMEOUT: No timeout
==> [2020-08-28-18:37:56.397504] [18338] concretized specs:
        spec: aluminum
                hash: 3easygztiqbtrszh4rxogt6wb5r7nvra
                MISSING
        spec: hydrogen
                hash: os7an62k754vsti44qiyt2x6miyipvno
                MISSING
        spec: cereal
                hash: jkhurievuy4ast3uttjv5ugcllcfzp4b
        spec: clara
                hash: pjnrr5h2wwrpharvwfxvpanhtnbzrqqx
        spec: cnpy
                hash: 5ivz3nf3ystrzgxzuhhomeql5itn7cfy
        spec: conduit
                hash: pe6qrrowe3dvlz55bh3wovs544aqnnwn
        spec: hwloc
                hash: znq665g7jo5bkj62aa7vhkajdrrc5qqc
        spec: opencv
                hash: k3thbvvkyb6krvfyh73rfwzxgkuv4vd6
        spec: zlib
                hash: oahdokybko632taim7jrn65uaktx3caf
        spec: cudnn
                hash: 6syesbuj34lqheadekgw6boxtko5xmru
        spec: cuda
                hash: wy2kjyohhwdtbgaeidopkgsw5zz6l6pe
        spec: nccl
                hash: ijyobdkhlwscsocsae3fejrp5u3dle44
        spec: py-numpy
                hash: qrewhretfzn3ayragrz6zstymevsrx2r
        spec: py-protobuf
                hash: jyg2i32q2s7jrf5h67ajrky2fxeqqh6p
        spec: py-setuptools
                hash: hubb7w6qb53avo4k4trr2y6voubvzi62
        spec: mpi
                hash: jt2s5qmwaxpnzhnkvmgxn3cch7lyka32
        spec: catch2
                hash: crhfuzxrliz2yf6y2doslllusyl7gjuk
        spec: cmake
                hash: 6lezyty5xseynkep3ecllvuo2x5psuth
        spec: ninja
                hash: lmwqhz26l2qcg25i2dr3kpaxg4nz23az
        spec: python
                hash: red7gcie24nsoez25t2pv4bgchmfdnl5
        spec: py-pytest
                hash: i5lcdudmiweelzmlc6utgn5l2ii6zgq6
        spec: gcc
                hash: z4spojzeqvdddya3jutcxo7b4yb7lj34

Traceback (most recent call last):
  File "/ccs/home/vanessen/spack.git/bin/spack", line 64, in <module>
    sys.exit(spack.main.main())
  File "/autofs/nccs-svm1_home1/vanessen/spack.git/lib/spack/spack/main.py", line 762, in main
    return _invoke_command(command, parser, args, unknown)
  File "/autofs/nccs-svm1_home1/vanessen/spack.git/lib/spack/spack/main.py", line 490, in _invoke_command
    return_val = command(parser, args)
  File "/autofs/nccs-svm1_home1/vanessen/spack.git/lib/spack/spack/cmd/env.py", line 509, in env
    action(args)
  File "/autofs/nccs-svm1_home1/vanessen/spack.git/lib/spack/spack/cmd/env.py", line 120, in env_activate
    prompt=env_prompt if args.prompt else None
  File "/autofs/nccs-svm1_home1/vanessen/spack.git/lib/spack/spack/environment.py", line 181, in activate
    cmds += env.add_default_view_to_shell(shell)
  File "/autofs/nccs-svm1_home1/vanessen/spack.git/lib/spack/spack/environment.py", line 1198, in add_default_view_to_shell
    mods, errors = self._env_modifications_for_default_view()
  File "/autofs/nccs-svm1_home1/vanessen/spack.git/lib/spack/spack/environment.py", line 1173, in _env_modifications_for_default_view
    for _, spec in self.concretized_specs():
  File "/autofs/nccs-svm1_home1/vanessen/spack.git/lib/spack/spack/environment.py", line 1359, in concretized_specs
    yield (s, self.specs_by_hash[h])
KeyError: '3easygztiqbtrszh4rxogt6wb5r7nvra'
```
`cuda` is the simplest package with a hash difference from https://github.com/spack/spack/pull/18368#issuecomment-683176061 so I have pushed an update to analyze differences for that package specifically.

Could you

* (optional, but possibly speeding things up) Create a copy of your env with all packages except CUDA removed
* Instantiate a new environment and `spack -d install`
* Add the output here

Thanks!
In @bvanessen's case I think the issue is configurations where module names are not enclosed in quotes and include extra newlines:

```
  specs:
  - cuda
  packages:
    'cuda:':
      buildable: false
      version:
      - 11.0.2
      externals:
      - spec: cuda@11.0.2 %gcc@8.1.1 arch=linux-rhel7-power9le
        modules:
          - cuda/11.0.2
                                                         <---- this newline is being included as part of the module name, and therefore the YAML dump and hash
    'bzip2:':
      buildable: true
      version:
      ...
```

Resolving patch:
diff --git a/lib/spack/spack/spec.py b/lib/spack/spack/spec.py
--- a/lib/spack/spack/spec.py
+++ b/lib/spack/spack/spec.py
@@ -1008,7 +1008,7 @@ def __init__(self, spec_like=None,
         self._normal = normal
         self._concrete = concrete
         self.external_path = external_path
-        self.external_modules = external_modules
+        self.external_modules = Spec._format_module_list(external_modules)
         self._full_hash = full_hash
 
         # This attribute is used to store custom information for
@@ -1025,6 +1025,24 @@ def __init__(self, spec_like=None,
         elif spec_like is not None:
             raise TypeError("Can't make spec out of %s" % type(spec_like))
 
+    @staticmethod
+    def _format_module_list(modules):
+        """Return a module list that is suitable for YAML serialization
+        and hash computation.
+
+        Given a module list, possibly read from a configuration file,
+        return an object that serializes to a consistent YAML string
+        before/after round-trip serialization to/from a Spec dictionary
+        (stored in JSON format): when read in, the module list may
+        contain YAML formatting that is discarded (non-essential)
+        when stored as a Spec dictionary; we take care in this function
+        to discard such formatting such that the Spec hash does not
+        change before/after storage in JSON.
+        """
+        if modules:
+            modules = list(modules)
+        return modules
+
     @property
     def external(self):
         return bool(self.external_path) or bool(self.external_modules)
@@ -1383,8 +1401,8 @@ def _spec_hash(self, hash):
         """
         # TODO: curently we strip build dependencies by default.  Rethink
         # this when we move to using package hashing on all specs.
-        yaml_text = syaml.dump(
-            self.to_node_dict(hash=hash), default_flow_style=True)
+        node_dict = self.to_node_dict(hash=hash)
+        yaml_text = syaml.dump(node_dict, default_flow_style=True)
         sha = hashlib.sha1(yaml_text.encode('utf-8'))
         b32_hash = base64.b32encode(sha.digest()).lower()
 


Test patch:
diff --git a/lib/spack/spack/test/cmd/env.py b/lib/spack/spack/test/cmd/env.py
--- a/lib/spack/spack/test/cmd/env.py
+++ b/lib/spack/spack/test/cmd/env.py
@@ -2132,3 +2132,38 @@ def test_can_update_attributes_with_override(tmpdir):
 
     # Check that an update does not raise
     env('update', '-y', str(abspath.dirname))
+
+
+@pytest.mark.regression('18338')
+def test_newline_in_commented_sequence_is_not_an_issue(tmpdir):
+    spack_yaml = """
+spack:
+  specs:
+  - dyninst
+  packages:
+    libelf:
+      externals:
+      - spec: libelf@0.8.13
+        modules:
+        - libelf/3.18.1
+
+  concretization: together
+"""
+    abspath = tmpdir.join('spack.yaml')
+    abspath.write(spack_yaml)
+
+    def extract_build_hash(environment):
+        _, dyninst = next(iter(environment.specs_by_hash.items()))
+        return dyninst['libelf'].build_hash()
+
+    # Concretize a first time and create a lockfile
+    with ev.Environment(str(tmpdir)) as e:
+        concretize()
+        libelf_first_hash = extract_build_hash(e)
+
+    # Check that a second run won't error
+    with ev.Environment(str(tmpdir)) as e:
+        concretize()
+        libelf_second_hash = extract_build_hash(e)
+
+    assert libelf_first_hash == libelf_second_hash