Repository: spack/spack

Issue:
Packages that have the same SHA-1 prefix
This is a continuation of the discussion in #5564 since it seems to be a different issue.

It might be a duplicate of #5578.

As mentioned in #5564, patching my tree to add configure_args() to the mpfr package and building mpfr generated this warning:

```
[+] /isilon/scratch/hartzelg/spack-mpfr-debug/opt/spack/linux-centos7-x86_64/gcc-4.8.3/mpfr-3.1.5-ejbuuahwm3gwrqf3bfj7f7xvu3jgfki7
==> Warning: Dependency missing due to corrupt install directory:
  /isilon/scratch/hartzelg/spack-mpfr-debug/opt/spack/linux-centos7-x86_64/gcc-4.8.3/mpfr-3.1.5-ejbuuahwm3gwrqf3bfj7f7xvu3jgfki7
  Specs mpfr@3.1.5%gcc@4.8.3 patches=dfd62a42a239c427bb44087b4a520edbcd65bae6ad1fbe07303c96ee8696afbd,88dfefa6d39c9fd5a26a40d9bbc73df8da93f946c65c06038026261d78d919f5 arch=linux-centos7-x86_64 ^autoconf@2.69%gcc@4.8.3 arch=linux-centos7-x86_64 ^automake@1.15.1%gcc@4.8.3 arch=linux-centos7-x86_64 ^gdbm@1.13%gcc@4.8.3 arch=linux-centos7-x86_64 ^gmp@6.1.2%gcc@4.8.3 arch=linux-centos7-x86_64 ^libsigsegv@2.11%gcc@4.8.3 arch=linux-centos7-x86_64 ^libtool@2.4.6%gcc@4.8.3 arch=linux-centos7-x86_64 ^m4@1.4.18%gcc@4.8.3 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00 +sigsegv arch=linux-centos7-x86_64 ^ncurses@6.0%gcc@4.8.3 patches=f84b2708a42777aadcc7f502a261afe10ca5646a51c1ef8b5e60d2070d926b57 ~symlinks arch=linux-centos7-x86_64 ^perl@5.24.1%gcc@4.8.3+cpanm+shared arch=linux-centos7-x86_64 ^pkg-config@0.29.2%gcc@4.8.3+internal_glib arch=linux-centos7-x86_64 ^readline@7.0%gcc@4.8.3 arch=linux-centos7-x86_64 and mpfr@3.1.5%gcc@4.8.3 patches=dfd62a42a239c427bb44087b4a520edbcd65bae6ad1fbe07303c96ee8696afbd,88dfefa6d39c9fd5a26a40d9bbc73df8da93f946c65c06038026261d78d919f5 arch=linux-centos7-x86_64 ^gmp@6.1.2%gcc@4.8.3 arch=linux-centos7-x86_64 have the same SHA-1 prefix!
```

In a fit of optimism, I tried to build `gcc@5.4.0` but ended up with an error:

```
hartzelg@rpbuchop001:spack-mpfr-debug (bug/fix-mpfr-build)$ spack build gcc@5.4.0
==> Checking dependencies for ['gcc@5.4.0']
==> gmp is already installed in /isilon/scratch/hartzelg/spack-mpfr-debug/opt/spack/linux-centos7-x86_64/gcc-4.8.3/gmp-6.1.2-yttyaes254rx44dkpsfco2pgx3togxqp
==> Installing mpfr
==> Using cached archive: /isilon/scratch/hartzelg/spack-mpfr-debug/var/spack/cache/mpfr/mpfr-3.1.5.tar.bz2
==> Staging archive: /isilon/scratch/hartzelg/spack-mpfr-debug/var/spack/stage/mpfr-3.1.5-ejbuuahwm3gwrqf3bfj7f7xvu3jgfki7/mpfr-3.1.5.tar.bz2
==> Created stage in /isilon/scratch/hartzelg/spack-mpfr-debug/var/spack/stage/mpfr-3.1.5-ejbuuahwm3gwrqf3bfj7f7xvu3jgfki7
==> Applied patch vasprintf.patch
==> Applied patch strtofr.patch
==> Building mpfr [AutotoolsPackage]
==> Executing phase: 'autoreconf'
==> Executing phase: 'configure'
==> Executing phase: 'build'
==> Executing phase: 'install'
==> Warning: Module file already exists : skipping creation
file : /isilon/scratch/hartzelg/spack-mpfr-debug/share/spack/modules/linux-centos7-x86_64/mpfr-3.1.5-gcc-4.8.3-ejbuuah
spec : mpfr@3.1.5%gcc@4.8.3 patches=dfd62a42a239c427bb44087b4a520edbcd65bae6ad1fbe07303c96ee8696afbd,88dfefa6d39c9fd5a26a40d9bbc73df8da93f946c65c06038026261d78d919f5 arch=linux-centos7-x86_64 ^autoconf@2.69%gcc@4.8.3 arch=linux-centos7-x86_64 ^automake@1.15.1%gcc@4.8.3 arch=linux-centos7-x86_64 ^gdbm@1.13%gcc@4.8.3 arch=linux-centos7-x86_64 ^gmp@6.1.2%gcc@4.8.3 arch=linux-centos7-x86_64 ^libsigsegv@2.11%gcc@4.8.3 arch=linux-centos7-x86_64 ^libtool@2.4.6%gcc@4.8.3 arch=linux-centos7-x86_64 ^m4@1.4.18%gcc@4.8.3 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00 +sigsegv arch=linux-centos7-x86_64 ^ncurses@6.0%gcc@4.8.3 patches=f84b2708a42777aadcc7f502a261afe10ca5646a51c1ef8b5e60d2070d926b57 ~symlinks arch=linux-centos7-x86_64 ^perl@5.24.1%gcc@4.8.3+cpanm+shared arch=linux-centos7-x86_64 ^pkg-config@0.29.2%gcc@4.8.3+internal_glib arch=linux-centos7-x86_64 ^readline@7.0%gcc@4.8.3 arch=linux-centos7-x86_64
==> Warning: Module file already exists : skipping creation
file : /isilon/scratch/hartzelg/spack-mpfr-debug/share/spack/dotkit/linux-centos7-x86_64/mpfr-3.1.5-gcc-4.8.3-ejbuuah.dk
spec : mpfr@3.1.5%gcc@4.8.3 patches=dfd62a42a239c427bb44087b4a520edbcd65bae6ad1fbe07303c96ee8696afbd,88dfefa6d39c9fd5a26a40d9bbc73df8da93f946c65c06038026261d78d919f5 arch=linux-centos7-x86_64 ^autoconf@2.69%gcc@4.8.3 arch=linux-centos7-x86_64 ^automake@1.15.1%gcc@4.8.3 arch=linux-centos7-x86_64 ^gdbm@1.13%gcc@4.8.3 arch=linux-centos7-x86_64 ^gmp@6.1.2%gcc@4.8.3 arch=linux-centos7-x86_64 ^libsigsegv@2.11%gcc@4.8.3 arch=linux-centos7-x86_64 ^libtool@2.4.6%gcc@4.8.3 arch=linux-centos7-x86_64 ^m4@1.4.18%gcc@4.8.3 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00 +sigsegv arch=linux-centos7-x86_64 ^ncurses@6.0%gcc@4.8.3 patches=f84b2708a42777aadcc7f502a261afe10ca5646a51c1ef8b5e60d2070d926b57 ~symlinks arch=linux-centos7-x86_64 ^perl@5.24.1%gcc@4.8.3+cpanm+shared arch=linux-centos7-x86_64 ^pkg-config@0.29.2%gcc@4.8.3+internal_glib arch=linux-centos7-x86_64 ^readline@7.0%gcc@4.8.3 arch=linux-centos7-x86_64
==> Successfully installed mpfr
  Fetch: 0.01s.  Build: 15.15s.  Total: 15.16s.
[+] /isilon/scratch/hartzelg/spack-mpfr-debug/opt/spack/linux-centos7-x86_64/gcc-4.8.3/mpfr-3.1.5-ejbuuahwm3gwrqf3bfj7f7xvu3jgfki7
==> gmp is already installed in /isilon/scratch/hartzelg/spack-mpfr-debug/opt/spack/linux-centos7-x86_64/gcc-4.8.3/gmp-6.1.2-yttyaes254rx44dkpsfco2pgx3togxqp
==> gmp is already installed in /isilon/scratch/hartzelg/spack-mpfr-debug/opt/spack/linux-centos7-x86_64/gcc-4.8.3/gmp-6.1.2-yttyaes254rx44dkpsfco2pgx3togxqp
==> Error: Specs mpfr@3.1.5%gcc@4.8.3 patches=dfd62a42a239c427bb44087b4a520edbcd65bae6ad1fbe07303c96ee8696afbd,88dfefa6d39c9fd5a26a40d9bbc73df8da93f946c65c06038026261d78d919f5 arch=linux-centos7-x86_64 ^autoconf@2.69%gcc@4.8.3 arch=linux-centos7-x86_64 ^automake@1.15.1%gcc@4.8.3 arch=linux-centos7-x86_64 ^gdbm@1.13%gcc@4.8.3 arch=linux-centos7-x86_64 ^gmp@6.1.2%gcc@4.8.3 arch=linux-centos7-x86_64 ^libsigsegv@2.11%gcc@4.8.3 arch=linux-centos7-x86_64 ^libtool@2.4.6%gcc@4.8.3 arch=linux-centos7-x86_64 ^m4@1.4.18%gcc@4.8.3 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00 +sigsegv arch=linux-centos7-x86_64 ^ncurses@6.0%gcc@4.8.3 patches=f84b2708a42777aadcc7f502a261afe10ca5646a51c1ef8b5e60d2070d926b57 ~symlinks arch=linux-centos7-x86_64 ^perl@5.24.1%gcc@4.8.3+cpanm+shared arch=linux-centos7-x86_64 ^pkg-config@0.29.2%gcc@4.8.3+internal_glib arch=linux-centos7-x86_64 ^readline@7.0%gcc@4.8.3 arch=linux-centos7-x86_64 and mpfr@3.1.5%gcc@4.8.3 patches=dfd62a42a239c427bb44087b4a520edbcd65bae6ad1fbe07303c96ee8696afbd,88dfefa6d39c9fd5a26a40d9bbc73df8da93f946c65c06038026261d78d919f5 arch=linux-centos7-x86_64 ^gmp@6.1.2%gcc@4.8.3 arch=linux-centos7-x86_64 have the same SHA-1 prefix!
```

It's a little funny that it build `mpfr` again, since I'd just built it by hand (though perhaps that warning resulted in nothing being installed.


Discussion:
I grabbed the changes in #5580 as a patch (https://patch-diff.githubusercontent.com/raw/LLNL/spack/pull/5580.patch) and dropped them onto the tip of develop, hoping that it would fix my SHA-1 collisions, but it didn't.

```
patch -p1 < foo.patch
spack uninstall mpfr
spack install mpfr
```

Gave me the same warning.

g.

I'm looking into this. To be clear, when you say

> Gave me the same warning.

Are you saying the gcc install failed again with the spec hash collision error when applying #5580, or that you still got the message

`==> Warning: Dependency missing due to corrupt install directory:`

when installing `mpfr`?
@tgamblin is there a reason that `Spec._cmp_key` uses `hash` vs. `dag_hash` for dependencies when constructing a key tuple? e.g. 

```
        dep_tuple = tuple(
            (d.spec.name, hash(d.spec), tuple(sorted(d.deptypes)))
            for name, d in sorted(self._dependencies.items()))
```

`dag_hash` omits build dependencies while `hash` itself uses `_cmp_key` and ~~so does not~~ therefore includes build dependencies (EDIT). I don't know why this would surface now but that does seem like strange behavior to me and in my case is what is producing the warning ~~here~~ in this issue.
Really quickly, yes -- speed.  Dag hash is really slow, and hash is fast (and uses cmp_key recursively).
I only took it as far as the warning when building `mpfr`.  Would it be useful to try gcc (I assumed it would fail, but you know what they say...).

FWIW, `spack uninstall -a -y; spack install gcc` is running with the patch from #5580 installed.  I'll summarize when it does something informative.
> I only took it as far as the warning when building mpfr. Would it be useful to try gcc (I assumed it would fail, but you know what they say...).

If you reindex after the mpfr installation, Spack appears to pick up the installed mpfr. At that point the gcc installation should succeed. 

(note you shouldn't have to do that but as a temporary hack around the bug reported in this issue that should work)

I'll be looking at this again today but need to step away for a few hours.
> FWIW, `spack uninstall -a -y; spack install gcc` is running with the patch from #5580 installed

I should say I think you will need https://github.com/LLNL/spack/pull/5586 unless you are using your `--with-gmp` fix. You may still need #5586, the bug it fixes basically removes all `-L` lines that Spack was adding.
Ok, I built gcc@7.2.0 by mistake (actually, just `gcc`).  Will try @5.4.0, which is what I need, next.

#5580 + #5564 results in a couple of warnings related to this discussion (search through the clipping below but ignore the sbang warnings) but made it into the gcc build step before I realized I was building the wrong version.

```
==> Installing mpfr
==> Using cached archive: /isilon/scratch/hartzelg/spack-mpfr-debug/var/spack/cache/mpfr/mpfr-3.1.5.tar.bz2
==> Staging archive: /isilon/scratch/hartzelg/spack-mpfr-debug/var/spack/stage/mpfr-3.1.5-ejbuuahwm3gwrqf3bfj7f7xvu3jgfki7/mpfr-3.1.5.tar.bz2
==> Created stage in /isilon/scratch/hartzelg/spack-mpfr-debug/var/spack/stage/mpfr-3.1.5-ejbuuahwm3gwrqf3bfj7f7xvu3jgfki7
==> Applied patch vasprintf.patch
==> Applied patch strtofr.patch
==> Building mpfr [AutotoolsPackage]
==> Executing phase: 'autoreconf'
==> Executing phase: 'configure'
==> Executing phase: 'build'
==> Executing phase: 'install'
==> Successfully installed mpfr
  Fetch: 0.01s.  Build: 16.24s.  Total: 16.25s.
[+] /isilon/scratch/hartzelg/spack-mpfr-debug/opt/spack/linux-centos7-x86_64/gcc-4.8.3/mpfr-3.1.5-ejbuuahwm3gwrqf3bfj7f7xvu3jgfki7
==> Warning: Dependency missing due to corrupt install directory:
  /isilon/scratch/hartzelg/spack-mpfr-debug/opt/spack/linux-centos7-x86_64/gcc-4.8.3/mpfr-3.1.5-ejbuuahwm3gwrqf3bfj7f7xvu3jgfki7
  Specs mpfr@3.1.5%gcc@4.8.3 patches=dfd62a42a239c427bb44087b4a520edbcd65bae6ad1fbe07303c96ee8696afbd,88dfefa6d39c9fd5a26a40d9bbc73df8da93f946c65c06038026261d78d919f5 arch=linux-centos7-x86_64 ^autoconf@2.69%gcc@4.8.3 arch=linux-centos7-x86_64 ^automake@1.15.1%gcc@4.8.3 arch=linux-centos7-x86_64 ^gdbm@1.13%gcc@4.8.3 arch=linux-centos7-x86_64 ^gmp@6.1.2%gcc@4.8.3 arch=linux-centos7-x86_64 ^libsigsegv@2.11%gcc@4.8.3 arch=linux-centos7-x86_64 ^libtool@2.4.6%gcc@4.8.3 arch=linux-centos7-x86_64 ^m4@1.4.18%gcc@4.8.3 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00 +sigsegv arch=linux-centos7-x86_64 ^ncurses@6.0%gcc@4.8.3 patches=f84b2708a42777aadcc7f502a261afe10ca5646a51c1ef8b5e60d2070d926b57 ~symlinks arch=linux-centos7-x86_64 ^perl@5.24.1%gcc@4.8.3+cpanm+shared arch=linux-centos7-x86_64 ^pkg-config@0.29.2%gcc@4.8.3+internal_glib arch=linux-centos7-x86_64 ^readline@7.0%gcc@4.8.3 arch=linux-centos7-x86_64 and mpfr@3.1.5%gcc@4.8.3 patches=dfd62a42a239c427bb44087b4a520edbcd65bae6ad1fbe07303c96ee8696afbd,88dfefa6d39c9fd5a26a40d9bbc73df8da93f946c65c06038026261d78d919f5 arch=linux-centos7-x86_64 ^gmp@6.1.2%gcc@4.8.3 arch=linux-centos7-x86_64 have the same SHA-1 prefix!
==> Installing zlib
==> Using cached archive: /isilon/scratch/hartzelg/spack-mpfr-debug/var/spack/cache/zlib/zlib-1.2.11.tar.gz
==> Staging archive: /isilon/scratch/hartzelg/spack-mpfr-debug/var/spack/stage/zlib-1.2.11-nnvax47zhgexamsrsugv576ygxthgmhv/zlib-1.2.11.tar.gz
==> Created stage in /isilon/scratch/hartzelg/spack-mpfr-debug/var/spack/stage/zlib-1.2.11-nnvax47zhgexamsrsugv576ygxthgmhv
==> No patches needed for zlib
==> Building zlib [Package]
==> Executing phase: 'install'
==> Successfully installed zlib
  Fetch: 0.03s.  Build: 1.94s.  Total: 1.97s.
[+] /isilon/scratch/hartzelg/spack-mpfr-debug/opt/spack/linux-centos7-x86_64/gcc-4.8.3/zlib-1.2.11-nnvax47zhgexamsrsugv576ygxthgmhv
==> gmp is already installed in /isilon/scratch/hartzelg/spack-mpfr-debug/opt/spack/linux-centos7-x86_64/gcc-4.8.3/gmp-6.1.2-yttyaes254rx44dkpsfco2pgx3togxqp
==> gmp is already installed in /isilon/scratch/hartzelg/spack-mpfr-debug/opt/spack/linux-centos7-x86_64/gcc-4.8.3/gmp-6.1.2-yttyaes254rx44dkpsfco2pgx3togxqp
==> Installing mpfr
==> Using cached archive: /isilon/scratch/hartzelg/spack-mpfr-debug/var/spack/cache/mpfr/mpfr-3.1.5.tar.bz2
==> Staging archive: /isilon/scratch/hartzelg/spack-mpfr-debug/var/spack/stage/mpfr-3.1.5-ejbuuahwm3gwrqf3bfj7f7xvu3jgfki7/mpfr-3.1.5.tar.bz2
==> Created stage in /isilon/scratch/hartzelg/spack-mpfr-debug/var/spack/stage/mpfr-3.1.5-ejbuuahwm3gwrqf3bfj7f7xvu3jgfki7
==> Applied patch vasprintf.patch
==> Applied patch strtofr.patch
==> Building mpfr [AutotoolsPackage]
==> Executing phase: 'autoreconf'
==> Executing phase: 'configure'
==> Executing phase: 'build'
==> Executing phase: 'install'
==> Warning: Module file already exists : skipping creation
file : /isilon/scratch/hartzelg/spack-mpfr-debug/share/spack/modules/linux-centos7-x86_64/mpfr-3.1.5-gcc-4.8.3-ejbuuah
spec : mpfr@3.1.5%gcc@4.8.3 patches=dfd62a42a239c427bb44087b4a520edbcd65bae6ad1fbe07303c96ee8696afbd,88dfefa6d39c9fd5a26a40d9bbc73df8da93f946c65c06038026261d78d919f5 arch=linux-centos7-x86_64 ^autoconf@2.69%gcc@4.8.3 arch=linux-centos7-x86_64 ^automake@1.15.1%gcc@4.8.3 arch=linux-centos7-x86_64 ^gdbm@1.13%gcc@4.8.3 arch=linux-centos7-x86_64 ^gmp@6.1.2%gcc@4.8.3 arch=linux-centos7-x86_64 ^libsigsegv@2.11%gcc@4.8.3 arch=linux-centos7-x86_64 ^libtool@2.4.6%gcc@4.8.3 arch=linux-centos7-x86_64 ^m4@1.4.18%gcc@4.8.3 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00 +sigsegv arch=linux-centos7-x86_64 ^ncurses@6.0%gcc@4.8.3 patches=f84b2708a42777aadcc7f502a261afe10ca5646a51c1ef8b5e60d2070d926b57 ~symlinks arch=linux-centos7-x86_64 ^perl@5.24.1%gcc@4.8.3+cpanm+shared arch=linux-centos7-x86_64 ^pkg-config@0.29.2%gcc@4.8.3+internal_glib arch=linux-centos7-x86_64 ^readline@7.0%gcc@4.8.3 arch=linux-centos7-x86_64
==> Warning: Module file already exists : skipping creation
file : /isilon/scratch/hartzelg/spack-mpfr-debug/share/spack/dotkit/linux-centos7-x86_64/mpfr-3.1.5-gcc-4.8.3-ejbuuah.dk
spec : mpfr@3.1.5%gcc@4.8.3 patches=dfd62a42a239c427bb44087b4a520edbcd65bae6ad1fbe07303c96ee8696afbd,88dfefa6d39c9fd5a26a40d9bbc73df8da93f946c65c06038026261d78d919f5 arch=linux-centos7-x86_64 ^autoconf@2.69%gcc@4.8.3 arch=linux-centos7-x86_64 ^automake@1.15.1%gcc@4.8.3 arch=linux-centos7-x86_64 ^gdbm@1.13%gcc@4.8.3 arch=linux-centos7-x86_64 ^gmp@6.1.2%gcc@4.8.3 arch=linux-centos7-x86_64 ^libsigsegv@2.11%gcc@4.8.3 arch=linux-centos7-x86_64 ^libtool@2.4.6%gcc@4.8.3 arch=linux-centos7-x86_64 ^m4@1.4.18%gcc@4.8.3 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00 +sigsegv arch=linux-centos7-x86_64 ^ncurses@6.0%gcc@4.8.3 patches=f84b2708a42777aadcc7f502a261afe10ca5646a51c1ef8b5e60d2070d926b57 ~symlinks arch=linux-centos7-x86_64 ^perl@5.24.1%gcc@4.8.3+cpanm+shared arch=linux-centos7-x86_64 ^pkg-config@0.29.2%gcc@4.8.3+internal_glib arch=linux-centos7-x86_64 ^readline@7.0%gcc@4.8.3 arch=linux-centos7-x86_64
==> Successfully installed mpfr
  Fetch: 0.01s.  Build: 15.80s.  Total: 15.81s.
[+] /isilon/scratch/hartzelg/spack-mpfr-debug/opt/spack/linux-centos7-x86_64/gcc-4.8.3/mpfr-3.1.5-ejbuuahwm3gwrqf3bfj7f7xvu3jgfki7
==> Installing mpc
==> Fetching https://ftp.gnu.org/gnu/mpc/mpc-1.0.3.tar.gz
######################################################################## 100.0%
==> Staging archive: /isilon/scratch/hartzelg/spack-mpfr-debug/var/spack/stage/mpc-1.0.3-iymd7n4omjameziq34gvb4wbg5aog5sl/mpc-1.0.3.tar.gz
==> Created stage in /isilon/scratch/hartzelg/spack-mpfr-debug/var/spack/stage/mpc-1.0.3-iymd7n4omjameziq34gvb4wbg5aog5sl
==> No patches needed for mpc
==> Building mpc [AutotoolsPackage]
==> Executing phase: 'autoreconf'
==> Executing phase: 'configure'
==> Executing phase: 'build'
==> Executing phase: 'install'
==> Successfully installed mpc
  Fetch: 6.52s.  Build: 9.86s.  Total: 16.38s.
[+] /isilon/scratch/hartzelg/spack-mpfr-debug/opt/spack/linux-centos7-x86_64/gcc-4.8.3/mpc-1.0.3-iymd7n4omjameziq34gvb4wbg5aog5sl
==> Warning: Dependency missing due to corrupt install directory:
  /isilon/scratch/hartzelg/spack-mpfr-debug/opt/spack/linux-centos7-x86_64/gcc-4.8.3/mpc-1.0.3-iymd7n4omjameziq34gvb4wbg5aog5sl
  Specs mpc@1.0.3%gcc@4.8.3 arch=linux-centos7-x86_64 ^autoconf@2.69%gcc@4.8.3 arch=linux-centos7-x86_64 ^automake@1.15.1%gcc@4.8.3 arch=linux-centos7-x86_64 ^gdbm@1.13%gcc@4.8.3 arch=linux-centos7-x86_64 ^gmp@6.1.2%gcc@4.8.3 arch=linux-centos7-x86_64 ^libsigsegv@2.11%gcc@4.8.3 arch=linux-centos7-x86_64 ^libtool@2.4.6%gcc@4.8.3 arch=linux-centos7-x86_64 ^m4@1.4.18%gcc@4.8.3 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00 +sigsegv arch=linux-centos7-x86_64 ^mpfr@3.1.5%gcc@4.8.3 patches=88dfefa6d39c9fd5a26a40d9bbc73df8da93f946c65c06038026261d78d919f5,dfd62a42a239c427bb44087b4a520edbcd65bae6ad1fbe07303c96ee8696afbd arch=linux-centos7-x86_64 ^ncurses@6.0%gcc@4.8.3 patches=f84b2708a42777aadcc7f502a261afe10ca5646a51c1ef8b5e60d2070d926b57 ~symlinks arch=linux-centos7-x86_64 ^perl@5.24.1%gcc@4.8.3+cpanm+shared arch=linux-centos7-x86_64 ^pkg-config@0.29.2%gcc@4.8.3+internal_glib arch=linux-centos7-x86_64 ^readline@7.0%gcc@4.8.3 arch=linux-centos7-x86_64 and mpc@1.0.3%gcc@4.8.3 arch=linux-centos7-x86_64 ^gmp@6.1.2%gcc@4.8.3 arch=linux-centos7-x86_64 ^mpfr@3.1.5%gcc@4.8.3 patches=88dfefa6d39c9fd5a26a40d9bbc73df8da93f946c65c06038026261d78d919f5,dfd62a42a239c427bb44087b4a520edbcd65bae6ad1fbe07303c96ee8696afbd arch=linux-centos7-x86_64 have the same SHA-1 prefix!
==> gmp is already installed in /isilon/scratch/hartzelg/spack-mpfr-debug/opt/spack/linux-centos7-x86_64/gcc-4.8.3/gmp-6.1.2-yttyaes254rx44dkpsfco2pgx3togxqp
```
```
spack uninstall -a -y
spack install gcc@5.4.0
```

results in a failure:

```
[...]
==> Executing phase: 'install'
==> Successfully installed mpfr
  Fetch: 0.01s.  Build: 15.70s.  Total: 15.71s.
[+] /isilon/scratch/hartzelg/spack-mpfr-debug/opt/spack/linux-centos7-x86_64/gcc-4.8.3/mpfr-3.1.5-ejbuuahwm3gwrqf3bfj7f7xvu3jgfki7
==> gmp is already installed in /isilon/scratch/hartzelg/spack-mpfr-debug/opt/spack/linux-centos7-x86_64/gcc-4.8.3/gmp-6.1.2-yttyaes254rx44dkpsfco2pgx3togxqp
==> gmp is already installed in /isilon/scratch/hartzelg/spack-mpfr-debug/opt/spack/linux-centos7-x86_64/gcc-4.8.3/gmp-6.1.2-yttyaes254rx44dkpsfco2pgx3togxqp
==> Error: Specs mpfr@3.1.5%gcc@4.8.3 patches=dfd62a42a239c427bb44087b4a520edbcd65bae6ad1fbe07303c96ee8696afbd,88dfefa6d39c9fd5a26a40d9bbc73df8da93f946c65c06038026261d78d919f5 arch=linux-centos7-x86_64 ^autoconf@2.69%gcc@4.8.3 arch=linux-centos7-x86_64 ^automake@1.15.1%gcc@4.8.3 arch=linux-centos7-x86_64 ^gdbm@1.13%gcc@4.8.3 arch=linux-centos7-x86_64 ^gmp@6.1.2%gcc@4.8.3 arch=linux-centos7-x86_64 ^libsigsegv@2.11%gcc@4.8.3 arch=linux-centos7-x86_64 ^libtool@2.4.6%gcc@4.8.3 arch=linux-centos7-x86_64 ^m4@1.4.18%gcc@4.8.3 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00 +sigsegv arch=linux-centos7-x86_64 ^ncurses@6.0%gcc@4.8.3 patches=f84b2708a42777aadcc7f502a261afe10ca5646a51c1ef8b5e60d2070d926b57 ~symlinks arch=linux-centos7-x86_64 ^perl@5.24.1%gcc@4.8.3+cpanm+shared arch=linux-centos7-x86_64 ^pkg-config@0.29.2%gcc@4.8.3+internal_glib arch=linux-centos7-x86_64 ^readline@7.0%gcc@4.8.3 arch=linux-centos7-x86_64 and mpfr@3.1.5%gcc@4.8.3 patches=dfd62a42a239c427bb44087b4a520edbcd65bae6ad1fbe07303c96ee8696afbd,88dfefa6d39c9fd5a26a40d9bbc73df8da93f946c65c06038026261d78d919f5 arch=linux-centos7-x86_64 ^gmp@6.1.2%gcc@4.8.3 arch=linux-centos7-x86_64 have the same SHA-1 prefix!
hartzelg@blah:spack-mpfr-debug (develop *)$
```
Apologies for the scattershot testing....

Another clean try, this time with develop+ #5586 + #5580 + #5585 (git reset --hard and apply patches),  then `spack uninstall -a -y; spack install gcc@5.4.0`

```
==> Successfully installed mpfr
  Fetch: 0.08s.  Build: 14.80s.  Total: 14.88s.
[+] /isilon/scratch/hartzelg/spack-mpfr-debug/opt/spack/linux-centos7-x86_64/gcc-4.8.3/mpfr-3.1.5-ejbuuahwm3gwrqf3bfj7f7xvu3jgfki7
==> gmp is already installed in /isilon/scratch/hartzelg/spack-mpfr-debug/opt/spack/linux-centos7-x86_64/gcc-4.8.3/gmp-6.1.2-yttyaes254rx44dkpsfco2pgx3togxqp
==> gmp is already installed in /isilon/scratch/hartzelg/spack-mpfr-debug/opt/spack/linux-centos7-x86_64/gcc-4.8.3/gmp-6.1.2-yttyaes254rx44dkpsfco2pgx3togxqp
==> Error: Specs mpfr@3.1.5%gcc@4.8.3 patches=dfd62a42a239c427bb44087b4a520edbcd65bae6ad1fbe07303c96ee8696afbd,88dfefa6d39c9fd5a26a40d9bbc73df8da93f946c65c06038026261d78d919f5 arch=linux-centos7-x86_64 ^autoconf@2.69%gcc@4.8.3 arch=linux-centos7-x86_64 ^automake@1.15.1%gcc@4.8.3 arch=linux-centos7-x86_64 ^gdbm@1.13%gcc@4.8.3 arch=linux-centos7-x86_64 ^gmp@6.1.2%gcc@4.8.3 arch=linux-centos7-x86_64 ^libsigsegv@2.11%gcc@4.8.3 arch=linux-centos7-x86_64 ^libtool@2.4.6%gcc@4.8.3 arch=linux-centos7-x86_64 ^m4@1.4.18%gcc@4.8.3 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00 +sigsegv arch=linux-centos7-x86_64 ^ncurses@6.0%gcc@4.8.3 patches=f84b2708a42777aadcc7f502a261afe10ca5646a51c1ef8b5e60d2070d926b57 ~symlinks arch=linux-centos7-x86_64 ^perl@5.24.1%gcc@4.8.3+cpanm+shared arch=linux-centos7-x86_64 ^pkg-config@0.29.2%gcc@4.8.3+internal_glib arch=linux-centos7-x86_64 ^readline@7.0%gcc@4.8.3 arch=linux-centos7-x86_64 and mpfr@3.1.5%gcc@4.8.3 patches=dfd62a42a239c427bb44087b4a520edbcd65bae6ad1fbe07303c96ee8696afbd,88dfefa6d39c9fd5a26a40d9bbc73df8da93f946c65c06038026261d78d919f5 arch=linux-centos7-x86_64 ^gmp@6.1.2%gcc@4.8.3 arch=linux-centos7-x86_64 have the same SHA-1 prefix!
```
Just noticed your reindex suggestion.  I'll give it a try.

**Thanks for digging into this!**
> If you reindex after the mpfr installation, Spack appears to pick up the installed mpfr. At that point the gcc installation should succeed.

I ran:

```
spack uninstall -a -y
spack install mpfr;
```

and then I tried to reindex it (*Did I do it correctly?*) and still had a problem:

```
[...]
==> Executing phase: 'install'
==> Successfully installed mpfr
  Fetch: 0.01s.  Build: 15.54s.  Total: 15.54s.
[+] /isilon/scratch/hartzelg/spack-mpfr-debug/opt/spack/linux-centos7-x86_64/gcc-4.8.3/mpfr-3.1.5-ejbuuahwm3gwrqf3bfj7f7xvu3jgfki7
hartzelg@rpbuchop001:spack-mpfr-debug (develop *)$ spack reindex
hartzelg@rpbuchop001:spack-mpfr-debug (develop *)$ spack reindex
hartzelg@rpbuchop001:spack-mpfr-debug (develop *)$ spack install gcc@5.4.0
==> Error: Specs mpfr@3.1.5%gcc@4.8.3 patches=dfd62a42a239c427bb44087b4a520edbcd65bae6ad1fbe07303c96ee8696afbd,88dfefa6d39c9fd5a26a40d9bbc73df8da93f946c65c06038026261d78d919f5 arch=linux-centos7-x86_64 ^autoconf@2.69%gcc@4.8.3 arch=linux-centos7-x86_64 ^automake@1.15.1%gcc@4.8.3 arch=linux-centos7-x86_64 ^gdbm@1.13%gcc@4.8.3 arch=linux-centos7-x86_64 ^gmp@6.1.2%gcc@4.8.3 arch=linux-centos7-x86_64 ^libsigsegv@2.11%gcc@4.8.3 arch=linux-centos7-x86_64 ^libtool@2.4.6%gcc@4.8.3 arch=linux-centos7-x86_64 ^m4@1.4.18%gcc@4.8.3 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00 +sigsegv arch=linux-centos7-x86_64 ^ncurses@6.0%gcc@4.8.3 patches=f84b2708a42777aadcc7f502a261afe10ca5646a51c1ef8b5e60d2070d926b57 ~symlinks arch=linux-centos7-x86_64 ^perl@5.24.1%gcc@4.8.3+cpanm+shared arch=linux-centos7-x86_64 ^pkg-config@0.29.2%gcc@4.8.3+internal_glib arch=linux-centos7-x86_64 ^readline@7.0%gcc@4.8.3 arch=linux-centos7-x86_64 and mpfr@3.1.5%gcc@4.8.3 patches=dfd62a42a239c427bb44087b4a520edbcd65bae6ad1fbe07303c96ee8696afbd,88dfefa6d39c9fd5a26a40d9bbc73df8da93f946c65c06038026261d78d919f5 arch=linux-centos7-x86_64 ^gmp@6.1.2%gcc@4.8.3 arch=linux-centos7-x86_64 have the same SHA-1 prefix!
```

This is with develop+ #5586 + #5580 + #5585.  Diff, just in case:

```diff
diff --git a/lib/spack/spack/build_environment.py b/lib/spack/spack/build_environment.py
index 849ee0b15..af3329ed5 100644
--- a/lib/spack/spack/build_environment.py
+++ b/lib/spack/spack/build_environment.py
@@ -191,9 +191,9 @@ def set_build_environment_variables(pkg, env, dirty):
         dirty (bool): Skip unsetting the user's environment settings
     """
     # Gather information about various types of dependencies
-    build_deps      = pkg.spec.dependencies(deptype=('build', 'test'))
-    link_deps       = pkg.spec.traverse(root=False, deptype=('link'))
-    build_link_deps = list(build_deps) + list(link_deps)
+    build_deps      = set(pkg.spec.dependencies(deptype=('build', 'test')))
+    link_deps       = set(pkg.spec.traverse(root=False, deptype=('link')))
+    build_link_deps = build_deps | link_deps
     rpath_deps      = get_rpath_deps(pkg)

     build_prefixes      = [dep.prefix for dep in build_deps]
diff --git a/lib/spack/spack/spec.py b/lib/spack/spack/spec.py
index d919aeecb..80592ffc0 100644
--- a/lib/spack/spack/spec.py
+++ b/lib/spack/spack/spec.py
@@ -1612,6 +1612,10 @@ class Spec(object):
         if self.name in visited:
             return False

+        if self.concrete:
+            visited.add(self.name)
+            return False
+
         changed = False

         # Concretize deps first -- this is a bottom-up process.
@@ -1805,6 +1809,9 @@ class Spec(object):
             if s.namespace is None:
                 s.namespace = spack.repo.repo_for_pkg(s.name).namespace

+            if s.concrete:
+                continue
+
             # Add any patches from the package to the spec.
             patches = []
             for cond, patch_list in s.package_class.patches.items():
@@ -1824,6 +1831,9 @@ class Spec(object):
             if dspec.spec.name not in pkg_deps:
                 continue

+            if dspec.spec.concrete:
+                continue
+
             patches = []
             for cond, dependency in pkg_deps[dspec.spec.name].items():
                 if dspec.parent.satisfies(cond):
@@ -1878,6 +1888,8 @@ class Spec(object):
         unless there is a need to force a spec to be concrete.
         """
         for s in self.traverse(deptype_query=all):
+            if s.concrete and s.package.installed:
+                continue
             s._normal = value
             s._concrete = value

@@ -2067,6 +2079,9 @@ class Spec(object):
         # caller, it's a copy from _evaluate_dependency_conditions. If it
         # comes from a vdep, it's a defensive copy from _find_provider.
         if dep.name not in spec_deps:
+            if self.concrete:
+                return False
+
             spec_deps[dep.name] = dep
             changed = True
         else:
diff --git a/var/spack/repos/builtin/packages/mpfr/package.py b/var/spack/repos/builtin/packages/mpfr/package.py
index 3e531ef24..4bc91e05f 100644
--- a/var/spack/repos/builtin/packages/mpfr/package.py
+++ b/var/spack/repos/builtin/packages/mpfr/package.py
@@ -42,3 +42,9 @@ class Mpfr(AutotoolsPackage):

     patch('vasprintf.patch', when='@3.1.5')
     patch('strtofr.patch',   when='@3.1.5')
+
+    def configure_args(self):
+        args = [
+            '--with-gmp=' + self.spec['gmp'].prefix,
+        ]
+        return args
```

> (Did I do it correctly?)

That was what I wanted to know. I really appreciate you trying that out!

> and then I tried to reindex it and still had a problem:

OK let me try to check in on this. I should be able to reproduce it.
I was able to replicate the issue here and also have a proposed fix at #5593 (it's a WIP at the moment now because my implementation is off)

@tgamblin: my complaint about calling `hash()` in `Spec._cmp_key` at https://github.com/LLNL/spack/issues/5587#issuecomment-333948223 was a red herring, but it looks like multi-valued variants do not properly implement `_cmp_key()` themselves.
Hi @hartzell,

#5586 is now merged. If I also merge in #5580 and #5593 then this is resolved for me.

>Thanks for digging into this!

Thanks for all the testing here! This is quite thorough and makes it easier to debug.
I am still having a problem with colliding hashes after starting with a clean repository at the head of develop.  When trying to install py-keras I see a failure with ncurses.  If I then uninstall ncurses and try again the error persists.

spack install py-keras ^py-theano +gpu
==> bzip2 is already installed in /usr/workspace/wsa/vanessen/spack_candle.git/opt/spack/linux-rhel7-x86_64/gcc-7.1.0/bzip2-1.0.6-7ymtt2jjwlv5lioxena3unlffrzlwcn6
==> pkg-config is already installed in /usr/workspace/wsa/vanessen/spack_candle.git/opt/spack/linux-rhel7-x86_64/gcc-7.1.0/pkg-config-0.29.2-qwzakozc657b22swrkouj6nauay5ugdi
==> Installing ncurses
==> Using cached archive: /usr/workspace/wsa/vanessen/spack_candle.git/var/spack/cache/ncurses/ncurses-6.0.tar.gz
==> Staging archive: /usr/workspace/wsa/vanessen/spack_candle.git/var/spack/stage/ncurses-6.0-vdec55phgae5f4tideiodjtumcl5bjeo/ncurses-6.0.tar.gz
==> Created stage in /usr/workspace/wsa/vanessen/spack_candle.git/var/spack/stage/ncurses-6.0-vdec55phgae5f4tideiodjtumcl5bjeo
==> Applied patch sed_pgi.patch
==> Applied patch patch_gcc_5.txt
==> Building ncurses [AutotoolsPackage]
==> Executing phase: 'autoreconf'
==> Executing phase: 'configure'
==> Executing phase: 'build'
==> Executing phase: 'install'
==> Successfully installed ncurses
  Fetch: 0.01s.  Build: 1m 47.45s.  Total: 1m 47.46s.
[+] /usr/workspace/wsa/vanessen/spack_candle.git/opt/spack/linux-rhel7-x86_64/gcc-7.1.0/ncurses-6.0-vdec55phgae5f4tideiodjtumcl5bjeo
==> zlib is already installed in /usr/workspace/wsa/vanessen/spack_candle.git/opt/spack/linux-rhel7-x86_64/gcc-7.1.0/zlib-1.2.11-zqrm3wqa4icvkftszlqhyte5ndwxc2r3
==> openssl is already installed in /usr/workspace/wsa/vanessen/spack_candle.git/opt/spack/linux-rhel7-x86_64/gcc-7.1.0/openssl-1.0.2k-mffljguzpuuiaqxrdqjmwr22kq7543ea
==> Error: Specs ncurses@6.0%gcc@7.1.0 patches=f84b2708a42777aadcc7f502a261afe10ca5646a51c1ef8b5e60d2070d926b57,4110a40613b800da2b2888c352b64c75a82809d48341061e4de5861e8b28423f ~symlinks arch=linux-rhel7-x86_64 ^pkg-config@0.29.2%gcc@7.1.0+internal_glib arch=linux-rhel7-x86_64 and ncurses@6.0%gcc@7.1.0 patches=f84b2708a42777aadcc7f502a261afe10ca5646a51c1ef8b5e60d2070d926b57,4110a40613b800da2b2888c352b64c75a82809d48341061e4de5861e8b28423f ~symlinks arch=linux-rhel7-x86_64 have the same SHA-1 prefix!
@bvanessen The pull requests mentioned in https://github.com/LLNL/spack/issues/5587#issuecomment-334033073 haven't been merged into develop yet, so that's to be expected :) Plus for the second one, it looks like the fix might be something else than what's currently in the pull request (see https://github.com/LLNL/spack/pull/5593#issuecomment-334061356a).

Resolving patch:
diff --git a/lib/spack/spack/spec.py b/lib/spack/spack/spec.py
--- a/lib/spack/spack/spec.py
+++ b/lib/spack/spack/spec.py
@@ -1812,10 +1812,12 @@ def concretize(self):
                     for patch in patch_list:
                         patches.append(patch.sha256)
             if patches:
-                # Special-case: keeps variant values unique but ordered.
-                s.variants['patches'] = MultiValuedVariant('patches', ())
-                mvar = s.variants['patches']
-                mvar._value = mvar._original_value = tuple(dedupe(patches))
+                mvar = s.variants.setdefault(
+                    'patches', MultiValuedVariant('patches', ())
+                )
+                mvar.value = patches
+                # FIXME: Monkey patches mvar to store patches order
+                mvar._patches_in_order_of_appearance = patches
 
         # Apply patches required on dependencies by depends_on(..., patch=...)
         for dspec in self.traverse_edges(deptype=all,
@@ -1832,15 +1834,13 @@ def concretize(self):
                             for patch in patch_list:
                                 patches.append(patch.sha256)
             if patches:
-                # note that we use a special multi-valued variant and
-                # keep the patches ordered.
-                if 'patches' not in dspec.spec.variants:
-                    mvar = MultiValuedVariant('patches', ())
-                    dspec.spec.variants['patches'] = mvar
-                else:
-                    mvar = dspec.spec.variants['patches']
-                mvar._value = mvar._original_value = tuple(
-                    dedupe(list(mvar._value) + patches))
+                mvar = dspec.spec.variants.setdefault(
+                    'patches', MultiValuedVariant('patches', ())
+                )
+                mvar.value = mvar.value + tuple(patches)
+                # FIXME: Monkey patches mvar to store patches order
+                l = getattr(mvar, '_patches_in_order_of_appearance', [])
+                mvar._patches_in_order_of_appearance = dedupe(l + patches)
 
         for s in self.traverse():
             if s.external_module:
@@ -2523,7 +2523,11 @@ def patches(self):
             return []
 
         patches = []
-        for sha256 in self.variants['patches'].value:
+
+        # FIXME: The private attribute below is attached after
+        # FIXME: concretization to store the order of patches somewhere.
+        # FIXME: Needs to be refactored in a cleaner way.
+        for sha256 in self.variants['patches']._patches_in_order_of_appearance:
             patch = self.package.lookup_patch(sha256)
             if patch:
                 patches.append(patch)


Test patch:
diff --git a/lib/spack/spack/test/patch.py b/lib/spack/spack/test/patch.py
--- a/lib/spack/spack/test/patch.py
+++ b/lib/spack/spack/test/patch.py
@@ -99,9 +99,11 @@ def test_patch_in_spec(builtin_mock, config):
     spec.concretize()
     assert 'patches' in list(spec.variants.keys())
 
-    # foo, bar, baz
-    assert (('b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c',
-             '7d865e959b2466918c9863afca942d0fb89d7c9ac0c99bafc3749504ded97730',
+    # Here the order is bar, foo, baz. Note that MV variants order
+    # lexicographically based on the hash, not on the position of the
+    # patch directive.
+    assert (('7d865e959b2466918c9863afca942d0fb89d7c9ac0c99bafc3749504ded97730',
+             'b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c',
              'bf07a7fbb825fc0aae7bf4a1177b2b31fcf8a3feeaf7092761e18c859ee52a9c') ==
             spec.variants['patches'].value)
 
@@ -131,8 +133,8 @@ def test_multiple_patched_dependencies(builtin_mock, config):
     # URL patches
     assert 'patches' in list(spec['fake'].variants.keys())
     # urlpatch.patch, urlpatch.patch.gz
-    assert (('abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234',
-             '1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd') ==
+    assert (('1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd',
+             'abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234') ==
             spec['fake'].variants['patches'].value)
 
 
@@ -159,8 +161,8 @@ def test_conditional_patched_dependencies(builtin_mock, config):
     # URL patches
     assert 'patches' in list(spec['fake'].variants.keys())
     # urlpatch.patch, urlpatch.patch.gz
-    assert (('abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234',
-             '1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd') ==
+    assert (('1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd',
+             'abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234') ==
             spec['fake'].variants['patches'].value)
 
 
@@ -187,6 +189,6 @@ def test_conditional_patched_deps_with_conditions(builtin_mock, config):
     # URL patches
     assert 'patches' in list(spec['fake'].variants.keys())
     # urlpatch.patch, urlpatch.patch.gz
-    assert (('abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234',
-             '1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd') ==
+    assert (('1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd',
+             'abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234') ==
             spec['fake'].variants['patches'].value)