Repository: spack/spack

Issue:
Virtual dependencies incorrectly affecting the hash
@tgamblin @scheibelp @alalazo @davydden This is a really weird one that I've never seen before. 

From a fresh clone of Spack, with no `packages.yaml`, I'm seeing the following behavior. If I don't specify an MPI provider and let it pick up the default (`openmpi`), I see:
```
$ spack spec -L hdf5+mpi
...
tgprd6nwfgifpaff4a7dvcv3tynuaowd  hdf5@1.10.1%gcc@6.3.1+cxx~debug+fortran+mpi+pic+shared~szip~threadsafe arch=linux-fedora25-x86_64 
hj3i67mhesgu3x5ymwpj67m4nbq6d3c2      ^openmpi@2.1.1%gcc@6.3.1~cuda fabrics= ~java schedulers= ~sqlite3~thread_multiple+vt arch=linux-fedora25-x86_64 
eva5cqpkbjggxvpp23ycr62yiocie64g          ^hwloc@1.11.7%gcc@6.3.1~cuda+libxml2+pci arch=linux-fedora25-x86_64 
reafmcemqbmvzqzyiitmyfcdc4ia7tzl              ^libpciaccess@0.13.5%gcc@6.3.1 arch=linux-fedora25-x86_64 
4w4hvmepes4p75z37og3dn3gbzjpnwho                  ^libtool@2.4.6%gcc@6.3.1 arch=linux-fedora25-x86_64 
zcwwrrt7od3rrhxb5du2iybu65jlrlbw                      ^m4@1.4.18%gcc@6.3.1+sigsegv arch=linux-fedora25-x86_64 
625n6uiwuqes737tignpxu3snoibyc43                          ^libsigsegv@2.11%gcc@6.3.1 arch=linux-fedora25-x86_64 
ywniw4sq7xd3fzz6c4c7jrhqj5htwobl                  ^pkg-config@0.29.2%gcc@6.3.1+internal_glib arch=linux-fedora25-x86_64 
lnv6kl5fucdneeh7h7ickta4vdis7u4v                  ^util-macros@1.19.1%gcc@6.3.1 arch=linux-fedora25-x86_64 
ggtx25lpnfriqn4n35fsgquiug6uju7t              ^libxml2@2.9.4%gcc@6.3.1~python arch=linux-fedora25-x86_64 
yw6ju7g3kyc27eonzrrvu73cdpiy5hzq                  ^xz@5.2.3%gcc@6.3.1 arch=linux-fedora25-x86_64 
jcukpuaeyykzu647duoufe2bdrgru5ko                  ^zlib@1.2.11%gcc@6.3.1+pic+shared arch=linux-fedora25-x86_64 
```
If I do specify the MPI provider and explicitly tell it to use `openmpi`, I see:
```
$ spack spec -L hdf5+mpi^openmpi
...
ohnyu6ch2ewnkf2dnd32qrb6t2hlopgj  hdf5@1.10.1%gcc@6.3.1+cxx~debug+fortran+mpi+pic+shared~szip~threadsafe arch=linux-fedora25-x86_64 
yyiri5gxf7jtu2c3vrhj6on4wz3jf2jh      ^openmpi@2.1.1%gcc@6.3.1~cuda fabrics= ~java schedulers= ~sqlite3~thread_multiple+vt arch=linux-fedora25-x86_64 
eva5cqpkbjggxvpp23ycr62yiocie64g          ^hwloc@1.11.7%gcc@6.3.1~cuda+libxml2+pci arch=linux-fedora25-x86_64 
reafmcemqbmvzqzyiitmyfcdc4ia7tzl              ^libpciaccess@0.13.5%gcc@6.3.1 arch=linux-fedora25-x86_64 
4w4hvmepes4p75z37og3dn3gbzjpnwho                  ^libtool@2.4.6%gcc@6.3.1 arch=linux-fedora25-x86_64 
zcwwrrt7od3rrhxb5du2iybu65jlrlbw                      ^m4@1.4.18%gcc@6.3.1+sigsegv arch=linux-fedora25-x86_64 
625n6uiwuqes737tignpxu3snoibyc43                          ^libsigsegv@2.11%gcc@6.3.1 arch=linux-fedora25-x86_64 
ywniw4sq7xd3fzz6c4c7jrhqj5htwobl                  ^pkg-config@0.29.2%gcc@6.3.1+internal_glib arch=linux-fedora25-x86_64 
lnv6kl5fucdneeh7h7ickta4vdis7u4v                  ^util-macros@1.19.1%gcc@6.3.1 arch=linux-fedora25-x86_64 
ggtx25lpnfriqn4n35fsgquiug6uju7t              ^libxml2@2.9.4%gcc@6.3.1~python arch=linux-fedora25-x86_64 
yw6ju7g3kyc27eonzrrvu73cdpiy5hzq                  ^xz@5.2.3%gcc@6.3.1 arch=linux-fedora25-x86_64 
jcukpuaeyykzu647duoufe2bdrgru5ko                  ^zlib@1.2.11%gcc@6.3.1+pic+shared arch=linux-fedora25-x86_64 
```
As you can see, the packages to be installed are identical. However, the hash for `openmpi` (and consequently `hdf5`) changed!
Virtual dependencies incorrectly affecting the hash
@tgamblin @scheibelp @alalazo @davydden This is a really weird one that I've never seen before. 

From a fresh clone of Spack, with no `packages.yaml`, I'm seeing the following behavior. If I don't specify an MPI provider and let it pick up the default (`openmpi`), I see:
```
$ spack spec -L hdf5+mpi
...
tgprd6nwfgifpaff4a7dvcv3tynuaowd  hdf5@1.10.1%gcc@6.3.1+cxx~debug+fortran+mpi+pic+shared~szip~threadsafe arch=linux-fedora25-x86_64 
hj3i67mhesgu3x5ymwpj67m4nbq6d3c2      ^openmpi@2.1.1%gcc@6.3.1~cuda fabrics= ~java schedulers= ~sqlite3~thread_multiple+vt arch=linux-fedora25-x86_64 
eva5cqpkbjggxvpp23ycr62yiocie64g          ^hwloc@1.11.7%gcc@6.3.1~cuda+libxml2+pci arch=linux-fedora25-x86_64 
reafmcemqbmvzqzyiitmyfcdc4ia7tzl              ^libpciaccess@0.13.5%gcc@6.3.1 arch=linux-fedora25-x86_64 
4w4hvmepes4p75z37og3dn3gbzjpnwho                  ^libtool@2.4.6%gcc@6.3.1 arch=linux-fedora25-x86_64 
zcwwrrt7od3rrhxb5du2iybu65jlrlbw                      ^m4@1.4.18%gcc@6.3.1+sigsegv arch=linux-fedora25-x86_64 
625n6uiwuqes737tignpxu3snoibyc43                          ^libsigsegv@2.11%gcc@6.3.1 arch=linux-fedora25-x86_64 
ywniw4sq7xd3fzz6c4c7jrhqj5htwobl                  ^pkg-config@0.29.2%gcc@6.3.1+internal_glib arch=linux-fedora25-x86_64 
lnv6kl5fucdneeh7h7ickta4vdis7u4v                  ^util-macros@1.19.1%gcc@6.3.1 arch=linux-fedora25-x86_64 
ggtx25lpnfriqn4n35fsgquiug6uju7t              ^libxml2@2.9.4%gcc@6.3.1~python arch=linux-fedora25-x86_64 
yw6ju7g3kyc27eonzrrvu73cdpiy5hzq                  ^xz@5.2.3%gcc@6.3.1 arch=linux-fedora25-x86_64 
jcukpuaeyykzu647duoufe2bdrgru5ko                  ^zlib@1.2.11%gcc@6.3.1+pic+shared arch=linux-fedora25-x86_64 
```
If I do specify the MPI provider and explicitly tell it to use `openmpi`, I see:
```
$ spack spec -L hdf5+mpi^openmpi
...
ohnyu6ch2ewnkf2dnd32qrb6t2hlopgj  hdf5@1.10.1%gcc@6.3.1+cxx~debug+fortran+mpi+pic+shared~szip~threadsafe arch=linux-fedora25-x86_64 
yyiri5gxf7jtu2c3vrhj6on4wz3jf2jh      ^openmpi@2.1.1%gcc@6.3.1~cuda fabrics= ~java schedulers= ~sqlite3~thread_multiple+vt arch=linux-fedora25-x86_64 
eva5cqpkbjggxvpp23ycr62yiocie64g          ^hwloc@1.11.7%gcc@6.3.1~cuda+libxml2+pci arch=linux-fedora25-x86_64 
reafmcemqbmvzqzyiitmyfcdc4ia7tzl              ^libpciaccess@0.13.5%gcc@6.3.1 arch=linux-fedora25-x86_64 
4w4hvmepes4p75z37og3dn3gbzjpnwho                  ^libtool@2.4.6%gcc@6.3.1 arch=linux-fedora25-x86_64 
zcwwrrt7od3rrhxb5du2iybu65jlrlbw                      ^m4@1.4.18%gcc@6.3.1+sigsegv arch=linux-fedora25-x86_64 
625n6uiwuqes737tignpxu3snoibyc43                          ^libsigsegv@2.11%gcc@6.3.1 arch=linux-fedora25-x86_64 
ywniw4sq7xd3fzz6c4c7jrhqj5htwobl                  ^pkg-config@0.29.2%gcc@6.3.1+internal_glib arch=linux-fedora25-x86_64 
lnv6kl5fucdneeh7h7ickta4vdis7u4v                  ^util-macros@1.19.1%gcc@6.3.1 arch=linux-fedora25-x86_64 
ggtx25lpnfriqn4n35fsgquiug6uju7t              ^libxml2@2.9.4%gcc@6.3.1~python arch=linux-fedora25-x86_64 
yw6ju7g3kyc27eonzrrvu73cdpiy5hzq                  ^xz@5.2.3%gcc@6.3.1 arch=linux-fedora25-x86_64 
jcukpuaeyykzu647duoufe2bdrgru5ko                  ^zlib@1.2.11%gcc@6.3.1+pic+shared arch=linux-fedora25-x86_64 
```
As you can see, the packages to be installed are identical. However, the hash for `openmpi` (and consequently `hdf5`) changed!


Discussion:
And the plot thickens. `spack install` correctly detects that the package is already installed even though `spack spec -I` reports that it is not:
```
$ spack spec -IL hdf5
...
     pbtdxy2hzuyrqgmsid7sprzvrmz6ku76  hdf5@1.10.1%gcc@7.1.0+cxx~debug+fortran+mpi+pic+shared~szip~threadsafe arch=linux-centos7-x86_64 
     gwc3dc5lies7ohtmagqqcj5xllp7vqpp      ^openmpi@2.1.1%gcc@7.1.0~cuda fabrics=verbs ~java schedulers= ~sqlite3~thread_multiple+vt arch=linux-centos7-x86_64 
[+]  xrj7vaf7w6ov3wlhzn2rlandrilp5ust          ^hwloc@1.11.7%gcc@7.1.0~cuda+libxml2+pci arch=linux-centos7-x86_64 
[+]  vrxqc7yqia4ok2yv7dd6jictfwhbgyqs              ^libpciaccess@0.13.5%gcc@7.1.0 arch=linux-centos7-x86_64 
[+]  wglwwndwwxupermiggpmk53q7nhrqnn7                  ^libtool@2.4.6%gcc@7.1.0 arch=linux-centos7-x86_64 
[+]  ugftxz5tjb2t3x5t46fnnzigbeki7tsh                      ^m4@1.4.18%gcc@7.1.0+sigsegv arch=linux-centos7-x86_64 
[+]  heh7sjdl245ey3qwfthctjuetkfgn2z7                          ^libsigsegv@2.11%gcc@7.1.0 arch=linux-centos7-x86_64 
[+]  uhtft2o4sjjclpwl7zm7ossd7vw65gtr                  ^pkg-config@0.29.2%gcc@7.1.0+internal_glib arch=linux-centos7-x86_64 
[+]  byuclfvoy3o7m6zamydu2llnlxwkqvum                  ^util-macros@1.19.1%gcc@7.1.0 arch=linux-centos7-x86_64 
[+]  h5m3a4nxpubsyfxd77tjz4urmbqojppa              ^libxml2@2.9.4%gcc@7.1.0~python arch=linux-centos7-x86_64 
[+]  nghiqunsmdbs4a264hjy3bpglnq6dyqu                  ^xz@5.2.3%gcc@7.1.0 arch=linux-centos7-x86_64 
[+]  aoqw6ohtji4y7n6mnogamqodr7hzcvq7                  ^zlib@1.2.11%gcc@7.1.0+pic+shared arch=linux-centos7-x86_64 
$ spack install hdf5
==> hdf5 is already installed in /blues/gpfs/home/ajstewart/spack/opt/spack/linux-centos7-x86_64/gcc-7.1.0/hdf5-1.10.1-ny5trzgmjevfransvftyfelw5xgakty2
```
P.S. This in on a different cluster so the hash is different from my previous post.
Ouch, that hurts: `git bisect` tells me it's coming from #2386 
could be ultimately the same bug somewhere in conretization as in https://github.com/LLNL/spack/issues/4107 but manifests itself differently.
p.s. I also noticed that `spack spec -I` sometimes incorrectly reports a package as not being installed.
@tgamblin @adamjstewart I'll try to debug where this is coming from. Will post here anything I find.
Debugging session running `spack spec -L hdf5+mpi` at line 92 of the command (after `spec.concretize()`):
```python
In[1]: openmpi = spec['openmpi']

In[2]: openmpi.dag_hash()
Out[3]: 'gw3622ptsqq5tuvxrd5z3xynjysldufi'

In[3]: openmpi._hash = None

In[4]: openmpi.dag_hash()
Out[5]: 'his36lswhij5wjcfll7rgprw4lt6sfvb'
```
In a session running `spack spec -L hdf5+mpi ^openmpi`:
```python
In[1]: openmpi = spec['openmpi']

In[2]: openmpi.dag_hash()
Out[3]: 'his36lswhij5wjcfll7rgprw4lt6sfvb'

In[3]: openmpi._hash = None

In[4]: openmpi.dag_hash()
Out[5]: 'his36lswhij5wjcfll7rgprw4lt6sfvb'
```
In the first case it seems we are caching the result of hashing too early, or wrongly in some way.

EDIT: This is partially reassuring, because it means that the two specs are effectively the same and there is no misconstruction in terms of variants.
Not sure why, but in the first case (`spack spec -L hdf5+mpi`) `openmpi` is marked concrete when dependencies are not yet added to it. We are taking the hash of the following:
```yaml
{openmpi: {version: 2.1.1, arch: {platform: linux, platform_os: ubuntu14, target: x86_64},
    compiler: {name: gcc, version: '4.8'}, namespace: builtin, parameters: {cuda: false,
      fabrics: [''], java: false, schedulers: [''], sqlite3: false, thread_multiple: false,
      vt: true, cflags: [], cppflags: [], cxxflags: [], fflags: [], ldflags: [], ldlibs: []}}}
```
In the second case we are hashing instead the correct yaml dump, which is:
```yaml
{openmpi: {version: 2.1.1, arch: {platform: linux, platform_os: ubuntu14, target: x86_64},
    compiler: {name: gcc, version: '4.8'}, namespace: builtin, parameters: {cuda: false,
      fabrics: [''], java: false, schedulers: [''], sqlite3: false, thread_multiple: false,
      vt: true, cflags: [], cppflags: [], cxxflags: [], fflags: [], ldflags: [], ldlibs: []},
    dependencies: {hwloc: {hash: e7pwcoccg3svl7lzzu5qqd65w25zhbvt, type: [build, link]}}}}
```
**TLDR**: possibly found the bug. It's in:
```python
class DependencyMap(HashableMap):
    """Each spec has a DependencyMap containing specs for its dependencies.
       The DependencyMap is keyed by name. """

    @property
    def concrete(self):
        return all((d.spec.concrete and d.deptypes)
                   for d in self.values())
...
```
and it is due to the fact that `concrete` will return `True` for an empty set of values.

---

Longer explanation. The first command (`spack spec -L hdf5+mpi`) takes the wrong path at `spec.py:2290`, when evaluating:
```python
# If self is a concrete spec, and other is not virtual, then we need
# to substitute every multi-valued variant that needs it with a
# single-valued variant.
if self.concrete: # < -- Evaluating this line
    try:
        # When parsing a spec every variant of the form
        # 'foo=value' will be interpreted by default as a
        # multi-valued variant. During validation of the
        # variants we use the information in the package
        # to turn any variant that needs it to a single-valued
        # variant.
        substitute_single_valued_variants(other)
    except (SpecError, KeyError):
        # Catch the two things that could go wrong above:
        # 1. name is not a valid variant (KeyError)
        # 2. the variant is not validated (SpecError)
        return False
```
which was introduced as part of #2386 (and explains the bisect), because there's a possible bug in the way `self.concrete` is computed:
```python
@property
def concrete(self):
    """A spec is concrete if it can describe only ONE build of a package.
       If any of the name, version, architecture, compiler,
       variants, or depdenencies are ambiguous,then it is not concrete.
    """
    if self._concrete:
        return True

    self._concrete = bool(not self.virtual and
                          self.namespace is not None and
                          self.versions.concrete and
                          self.variants.concrete and
                          self.architecture and
                          self.architecture.concrete and
                          self.compiler and self.compiler.concrete and
                          self.compiler_flags.concrete and
                          self._dependencies.concrete) # <-- The problem is at this line
    return self._concrete
```

When we are evaluating `self.concrete` in this context we are in the middle of concretization, and `openmpi` has no dependencies attached but is otherwise concrete. The method to check if the dependencies are concrete reads:
```python
@property
def concrete(self):
    return all((d.spec.concrete and d.deptypes)
               for d in self.values())
```
which is the same as:
```python
In[3]: all(tuple())
Out[4]: True
```
and gives the wrong result as `openmpi` has no dependencies attached yet.
@alalazo I guess one can not simply check in `DependencyMap` if `self.values()` is empty and return `False` (not-conrete) as this may be a package without any dependencies. Probably one has to add additional flag somewhere to say if a package has dependencies. Or maybe modify the logic in `self._concrete` of Spec.
@davydden I was thinking of adding a constructor to `DependencyMap` similar to the one in `FlagMap`:
```python
def __init__(self, spec):
    super(FlagMap, self).__init__()
    self.spec = spec
```
Once we have the reference to the `spec` we may query the corresponding package to see:

1. If we satisfy every dependency in the directives that are "active" (meaning the spec satisfies their condition)
2. If the dependencies are concrete and of the correct type

The latter point is already coded in the `DependencyMap.concrete` property. The former is what is missing. I'll try to put together a PR tomorrow. Let me know if you have any suggestion.
And the plot thickens. `spack install` correctly detects that the package is already installed even though `spack spec -I` reports that it is not:
```
$ spack spec -IL hdf5
...
     pbtdxy2hzuyrqgmsid7sprzvrmz6ku76  hdf5@1.10.1%gcc@7.1.0+cxx~debug+fortran+mpi+pic+shared~szip~threadsafe arch=linux-centos7-x86_64 
     gwc3dc5lies7ohtmagqqcj5xllp7vqpp      ^openmpi@2.1.1%gcc@7.1.0~cuda fabrics=verbs ~java schedulers= ~sqlite3~thread_multiple+vt arch=linux-centos7-x86_64 
[+]  xrj7vaf7w6ov3wlhzn2rlandrilp5ust          ^hwloc@1.11.7%gcc@7.1.0~cuda+libxml2+pci arch=linux-centos7-x86_64 
[+]  vrxqc7yqia4ok2yv7dd6jictfwhbgyqs              ^libpciaccess@0.13.5%gcc@7.1.0 arch=linux-centos7-x86_64 
[+]  wglwwndwwxupermiggpmk53q7nhrqnn7                  ^libtool@2.4.6%gcc@7.1.0 arch=linux-centos7-x86_64 
[+]  ugftxz5tjb2t3x5t46fnnzigbeki7tsh                      ^m4@1.4.18%gcc@7.1.0+sigsegv arch=linux-centos7-x86_64 
[+]  heh7sjdl245ey3qwfthctjuetkfgn2z7                          ^libsigsegv@2.11%gcc@7.1.0 arch=linux-centos7-x86_64 
[+]  uhtft2o4sjjclpwl7zm7ossd7vw65gtr                  ^pkg-config@0.29.2%gcc@7.1.0+internal_glib arch=linux-centos7-x86_64 
[+]  byuclfvoy3o7m6zamydu2llnlxwkqvum                  ^util-macros@1.19.1%gcc@7.1.0 arch=linux-centos7-x86_64 
[+]  h5m3a4nxpubsyfxd77tjz4urmbqojppa              ^libxml2@2.9.4%gcc@7.1.0~python arch=linux-centos7-x86_64 
[+]  nghiqunsmdbs4a264hjy3bpglnq6dyqu                  ^xz@5.2.3%gcc@7.1.0 arch=linux-centos7-x86_64 
[+]  aoqw6ohtji4y7n6mnogamqodr7hzcvq7                  ^zlib@1.2.11%gcc@7.1.0+pic+shared arch=linux-centos7-x86_64 
$ spack install hdf5
==> hdf5 is already installed in /blues/gpfs/home/ajstewart/spack/opt/spack/linux-centos7-x86_64/gcc-7.1.0/hdf5-1.10.1-ny5trzgmjevfransvftyfelw5xgakty2
```
P.S. This in on a different cluster so the hash is different from my previous post.
Ouch, that hurts: `git bisect` tells me it's coming from #2386 
could be ultimately the same bug somewhere in conretization as in https://github.com/LLNL/spack/issues/4107 but manifests itself differently.
p.s. I also noticed that `spack spec -I` sometimes incorrectly reports a package as not being installed.
@tgamblin @adamjstewart I'll try to debug where this is coming from. Will post here anything I find.
Debugging session running `spack spec -L hdf5+mpi` at line 92 of the command (after `spec.concretize()`):
```python
In[1]: openmpi = spec['openmpi']

In[2]: openmpi.dag_hash()
Out[3]: 'gw3622ptsqq5tuvxrd5z3xynjysldufi'

In[3]: openmpi._hash = None

In[4]: openmpi.dag_hash()
Out[5]: 'his36lswhij5wjcfll7rgprw4lt6sfvb'
```
In a session running `spack spec -L hdf5+mpi ^openmpi`:
```python
In[1]: openmpi = spec['openmpi']

In[2]: openmpi.dag_hash()
Out[3]: 'his36lswhij5wjcfll7rgprw4lt6sfvb'

In[3]: openmpi._hash = None

In[4]: openmpi.dag_hash()
Out[5]: 'his36lswhij5wjcfll7rgprw4lt6sfvb'
```
In the first case it seems we are caching the result of hashing too early, or wrongly in some way.

EDIT: This is partially reassuring, because it means that the two specs are effectively the same and there is no misconstruction in terms of variants.
Not sure why, but in the first case (`spack spec -L hdf5+mpi`) `openmpi` is marked concrete when dependencies are not yet added to it. We are taking the hash of the following:
```yaml
{openmpi: {version: 2.1.1, arch: {platform: linux, platform_os: ubuntu14, target: x86_64},
    compiler: {name: gcc, version: '4.8'}, namespace: builtin, parameters: {cuda: false,
      fabrics: [''], java: false, schedulers: [''], sqlite3: false, thread_multiple: false,
      vt: true, cflags: [], cppflags: [], cxxflags: [], fflags: [], ldflags: [], ldlibs: []}}}
```
In the second case we are hashing instead the correct yaml dump, which is:
```yaml
{openmpi: {version: 2.1.1, arch: {platform: linux, platform_os: ubuntu14, target: x86_64},
    compiler: {name: gcc, version: '4.8'}, namespace: builtin, parameters: {cuda: false,
      fabrics: [''], java: false, schedulers: [''], sqlite3: false, thread_multiple: false,
      vt: true, cflags: [], cppflags: [], cxxflags: [], fflags: [], ldflags: [], ldlibs: []},
    dependencies: {hwloc: {hash: e7pwcoccg3svl7lzzu5qqd65w25zhbvt, type: [build, link]}}}}
```
**TLDR**: possibly found the bug. It's in:
```python
class DependencyMap(HashableMap):
    """Each spec has a DependencyMap containing specs for its dependencies.
       The DependencyMap is keyed by name. """

    @property
    def concrete(self):
        return all((d.spec.concrete and d.deptypes)
                   for d in self.values())
...
```
and it is due to the fact that `concrete` will return `True` for an empty set of values.

---

Longer explanation. The first command (`spack spec -L hdf5+mpi`) takes the wrong path at `spec.py:2290`, when evaluating:
```python
# If self is a concrete spec, and other is not virtual, then we need
# to substitute every multi-valued variant that needs it with a
# single-valued variant.
if self.concrete: # < -- Evaluating this line
    try:
        # When parsing a spec every variant of the form
        # 'foo=value' will be interpreted by default as a
        # multi-valued variant. During validation of the
        # variants we use the information in the package
        # to turn any variant that needs it to a single-valued
        # variant.
        substitute_single_valued_variants(other)
    except (SpecError, KeyError):
        # Catch the two things that could go wrong above:
        # 1. name is not a valid variant (KeyError)
        # 2. the variant is not validated (SpecError)
        return False
```
which was introduced as part of #2386 (and explains the bisect), because there's a possible bug in the way `self.concrete` is computed:
```python
@property
def concrete(self):
    """A spec is concrete if it can describe only ONE build of a package.
       If any of the name, version, architecture, compiler,
       variants, or depdenencies are ambiguous,then it is not concrete.
    """
    if self._concrete:
        return True

    self._concrete = bool(not self.virtual and
                          self.namespace is not None and
                          self.versions.concrete and
                          self.variants.concrete and
                          self.architecture and
                          self.architecture.concrete and
                          self.compiler and self.compiler.concrete and
                          self.compiler_flags.concrete and
                          self._dependencies.concrete) # <-- The problem is at this line
    return self._concrete
```

When we are evaluating `self.concrete` in this context we are in the middle of concretization, and `openmpi` has no dependencies attached but is otherwise concrete. The method to check if the dependencies are concrete reads:
```python
@property
def concrete(self):
    return all((d.spec.concrete and d.deptypes)
               for d in self.values())
```
which is the same as:
```python
In[3]: all(tuple())
Out[4]: True
```
and gives the wrong result as `openmpi` has no dependencies attached yet.
@alalazo I guess one can not simply check in `DependencyMap` if `self.values()` is empty and return `False` (not-conrete) as this may be a package without any dependencies. Probably one has to add additional flag somewhere to say if a package has dependencies. Or maybe modify the logic in `self._concrete` of Spec.
@davydden I was thinking of adding a constructor to `DependencyMap` similar to the one in `FlagMap`:
```python
def __init__(self, spec):
    super(FlagMap, self).__init__()
    self.spec = spec
```
Once we have the reference to the `spec` we may query the corresponding package to see:

1. If we satisfy every dependency in the directives that are "active" (meaning the spec satisfies their condition)
2. If the dependencies are concrete and of the correct type

The latter point is already coded in the `DependencyMap.concrete` property. The former is what is missing. I'll try to put together a PR tomorrow. Let me know if you have any suggestion.

Resolving patch:
diff --git a/lib/spack/spack/spec.py b/lib/spack/spack/spec.py
--- a/lib/spack/spack/spec.py
+++ b/lib/spack/spack/spec.py
@@ -678,10 +678,72 @@ class DependencyMap(HashableMap):
     """Each spec has a DependencyMap containing specs for its dependencies.
        The DependencyMap is keyed by name. """
 
+    def __init__(self, owner):
+        super(DependencyMap, self).__init__()
+
+        # Owner Spec for the current map
+        self.owner = owner
+
     @property
     def concrete(self):
-        return all((d.spec.concrete and d.deptypes)
-                   for d in self.values())
+
+        # Check if the dependencies are concrete and of the correct type
+        dependencies_are_concrete = all(
+            (d.spec.concrete and d.deptypes) for d in self.values()
+        )
+
+        if not dependencies_are_concrete:
+            return False
+
+        # If we are dealing with an external spec, it clearly has no
+        # dependencies managed by spack
+        if self.owner.external:
+            return True
+
+        # Now check we have every dependency we need
+        pkg = self.owner.package
+        for dependency in pkg.dependencies.values():
+            # Check that for every condition we satisfy we satisfy also
+            # the associated constraint
+            for condition, constraint in dependency.items():
+                # WARNING: This part relies on the fact that dependencies are
+                # the last item to be checked when computing if a spec is
+                # concrete. This means that we are ensured that all variants,
+                # versions, compilers, etc. are there with their final value
+                # when we call 'Spec.satisfies(..., strict=True)'
+
+                # FIXME: at the moment check only for non conditional
+                # FIXME: dependencies, to avoid infinite recursion
+
+                # satisfy_condition = self.owner.satisfies(
+                #     condition, strict=True
+                # )
+
+                satisfy_condition = str(condition) == self.owner.name
+
+                if not satisfy_condition:
+                    continue
+
+                dependency_name = constraint.name
+
+                # We don't want to check build dependencies
+                if pkg.dependency_types[dependency_name] == set(['build']):
+                    continue
+
+                try:
+                    dependency = self.owner[dependency_name]
+                    satisfy_constraint = dependency.satisfies(
+                        constraint, strict=True
+                    )
+                except KeyError:
+                    # If the dependency is not there we don't
+                    # satisfy the constraint
+                    satisfy_constraint = False
+
+                if satisfy_condition and not satisfy_constraint:
+                    return False
+
+        return True
 
     def __str__(self):
         return "{deps: %s}" % ', '.join(str(d) for d in sorted(self.values()))
@@ -1144,9 +1206,13 @@ def concrete(self):
            If any of the name, version, architecture, compiler,
            variants, or depdenencies are ambiguous,then it is not concrete.
         """
+        # If we have cached that the spec is concrete, then it's concrete
         if self._concrete:
             return True
 
+        # Otherwise check if the various parts of the spec are concrete.
+        # It's crucial to have the check on dependencies last, as it relies
+        # on all the other parts to be already checked for concreteness.
         self._concrete = bool(not self.virtual and
                               self.namespace is not None and
                               self.versions.concrete and
@@ -1647,8 +1713,8 @@ def _expand_virtual_packages(self):
                 if replacement.external:
                     if (spec._dependencies):
                         changed = True
-                        spec._dependencies = DependencyMap()
-                    replacement._dependencies = DependencyMap()
+                        spec._dependencies = DependencyMap(spec)
+                    replacement._dependencies = DependencyMap(replacement)
                     replacement.architecture = self.architecture
 
                 # TODO: could this and the stuff in _dup be cleaned up?
@@ -1676,6 +1742,7 @@ def feq(cfield, sfield):
                 if spec._dup(replacement, deps=False, cleardeps=False):
                     changed = True
 
+                spec._dependencies.owner = spec
                 self_index.update(spec)
                 done = False
                 break
@@ -1807,7 +1874,7 @@ def flat_dependencies(self, **kwargs):
     def index(self, deptype=None):
         """Return DependencyMap that points to all the dependencies in this
            spec."""
-        dm = DependencyMap()
+        dm = DependencyMap(None)
         for spec in self.traverse(deptype=deptype):
             dm[spec.name] = spec
         return dm
@@ -2416,8 +2483,8 @@ def _dup(self, other, deps=True, cleardeps=True):
             else None
         self.compiler = other.compiler.copy() if other.compiler else None
         if cleardeps:
-            self._dependents = DependencyMap()
-            self._dependencies = DependencyMap()
+            self._dependents = DependencyMap(self)
+            self._dependencies = DependencyMap(self)
         self.compiler_flags = other.compiler_flags.copy()
         self.variants = other.variants.copy()
         self.variants.spec = self
@@ -3078,8 +3145,8 @@ def spec(self, name):
         spec.external_path = None
         spec.external_module = None
         spec.compiler_flags = FlagMap(spec)
-        spec._dependents = DependencyMap()
-        spec._dependencies = DependencyMap()
+        spec._dependents = DependencyMap(spec)
+        spec._dependencies = DependencyMap(spec)
         spec.namespace = spec_namespace
         spec._hash = None
         spec._cmp_key_cache = None


Test patch:
diff --git a/lib/spack/spack/test/concretize.py b/lib/spack/spack/test/concretize.py
--- a/lib/spack/spack/test/concretize.py
+++ b/lib/spack/spack/test/concretize.py
@@ -26,7 +26,8 @@
 import spack
 import spack.architecture
 from spack.concretize import find_spec
-from spack.spec import Spec, CompilerSpec, ConflictsInSpecError, SpecError
+from spack.spec import Spec, CompilerSpec
+from spack.spec import ConflictsInSpecError, SpecError
 from spack.version import ver
 
 
@@ -397,3 +398,21 @@ def test_conflicts_in_spec(self, conflict_spec):
             s = Spec(conflict_spec)
             with pytest.raises(exc_type):
                 s.concretize()
+
+    def test_regression_issue_4492(self):
+        # Constructing a spec which has no dependencies, but is otherwise
+        # concrete is kind of difficult. What we will do is to concretize
+        # a spec, and then modify it to have no dependency and reset the
+        # cache values.
+
+        s = Spec('mpileaks')
+        s.concretize()
+
+        # Check that now the Spec is concrete, store the hash
+        assert s.concrete
+
+        # Remove the dependencies and reset caches
+        s._dependencies.clear()
+        s._concrete = False
+
+        assert not s.concrete