Repository: spack/spack

Issue:
Error in Environment.matching_specs() method
Today I hit an error in this [line](https://codecov.io/gh/spack/spack/src/develop/lib/spack/spack/environment.py#L1577).  The problem there is that `root_matches` is a dictionary mapping concrete specs to abstract specs, but it is indexed with an integer.

I hit this while attempting to use `matching_spec()` in #22657, but you can also make it happen using the `spack location` command (or probably any command that uses `spack.cmd.matching_spec_from_env(spec)`).

### Steps to reproduce the issue

It seems you need to have an environment where a particular package appears both as a root and a dependency (so concretized separately), but that might not be the complete necessary/sufficient set of conditions. To create an example problem environment:

```console
$ spack env create foo
$ spack env activate foo
$ spack add readline
$ spack add ncurses ~termlib
$ spack concretize
```

Now the `foo` environment contains two `ncurses` specs.

### Error Message

To cause the error:

```console
$ spack -d location -s ncurses
==> [2021-04-28-15:41:16.385703] Imported location from built-in commands
==> [2021-04-28-15:41:16.387569] Imported location from built-in commands
==> [2021-04-28-15:41:16.388594] Reading config file /projects/spack/etc/spack/defaults/repos.yaml
==> [2021-04-28-15:41:16.392801] Reading config file /projects/spack/var/spack/environments/blah/spack.yaml
Traceback (most recent call last):
  File "/projects/spack/bin/spack", line 76, in <module>
    sys.exit(spack.main.main())
  File "/projects/spack/lib/spack/spack/main.py", line 772, in main
    return _invoke_command(command, parser, args, unknown)
  File "/projects/spack/lib/spack/spack/main.py", line 496, in _invoke_command
    return_val = command(parser, args)
  File "/projects/spack/lib/spack/spack/cmd/location.py", line 112, in location
    spec = spack.cmd.matching_spec_from_env(spec)
  File "/projects/spack/lib/spack/spack/cmd/__init__.py", line 192, in matching_spec_from_env
    return env.matching_spec(spec) or spec.concretized()
  File "/projects/spack/lib/spack/spack/environment.py", line 1588, in matching_spec
    return root_matches[0][1]
KeyError: 0
```

### Information on your system

```
$ spack debug report
* **Spack:** 0.16.1-2415-ceacc6706f
* **Python:** 3.6.9
* **Platform:** linux-ubuntu18.04-haswell
* **Concretizer:** original
```

### Additional information

<!-- These boxes can be checked by replacing [ ] with [x] or by clicking them after submitting the issue. -->
- [x] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [x] I have searched the issues of this repo and believe this is not a duplicate
- [x] I have run the failing commands in debug mode and reported the output



Discussion:
@scheibelp You have the honor of having touched that line last :stuck_out_tongue_winking_eye: 

Apart from just alerting you to this issue, which looks simple to fix, I also wonder if you can help me understand whether `matching_spec()` is right for my use case in #22657.  I'm wondering if I'm the only one hitting this because I'm doing something unexpected that should really be done another way.  If you're not the right person to share the thinking behind that method, maybe you know who is.
The problem is 

```
        if len(root_matches) == 1:
            return root_matches[0][1]
```

`root_matches` is a dictionary, and this code (that I wrote) is treating like a list and indexing the first element (but python is complaining because the dictionary does not have a `0` key). I can fix this tomorrow.

> Apart from just alerting you to this issue, which looks simple to fix, I also wonder if you can help me understand whether matching_spec() is right for my use case in #22657. 

Note that `Environment.matching_spec` will return a root spec or a dependency of a root spec. Also if the user provides a spec like

`x`

which would normally concretize to

`x+foo`

but the environment has

`x~foo`

then `matching_spec` would return `x~foo` (since that satisfies `x`). If those semantics work for you, then I think `matching_spec` would be a good fit. You would want to make sure to install the spec returned by `matching_spec`, rather than concretizing it separately (I would recommend this even if you concretize before calling `matching_spec`).
Thanks for that explanation @scheibelp.  For my use case (CI pipelines) I'm mainly going to want to install concrete specs (either via cli hash or concrete `spec.yaml` on disk).  In that case there should be no ambiguity for `matching_spec` to deal with, so probably nothing for me to worry about.  

Question in general though: In the environment I described above, there is a single root `ncurses` and a single distinct dep `ncurses`.  In that case `matching_spec('ncurses')` (providing a spec abstract enough that both satisfy it) returns the root one instead of raising an error indicating multiple matches in the environment.  Do you think that's what a user, typing `spack install ncurses`, would expect and thus the correct behavior?

Resolving patch:
diff --git a/lib/spack/spack/cmd/install.py b/lib/spack/spack/cmd/install.py
--- a/lib/spack/spack/cmd/install.py
+++ b/lib/spack/spack/cmd/install.py
@@ -139,6 +139,10 @@ def setup_parser(subparser):
     subparser.add_argument(
         '--only-concrete', action='store_true', default=False,
         help='(with environment) only install already concretized specs')
+    subparser.add_argument(
+        '--no-add', action='store_true', default=False,
+        help="""(with environment) only install specs provided as argument
+if they are already in the concretized environment""")
     subparser.add_argument(
         '-f', '--file', action='append', default=[],
         dest='specfiles', metavar='SPEC_YAML_FILE',
@@ -205,11 +209,66 @@ def install_specs(cli_args, kwargs, specs):
 
     try:
         if env:
+            specs_to_install = []
+            specs_to_add = []
             for abstract, concrete in specs:
-                with env.write_transaction():
-                    concrete = env.concretize_and_add(abstract, concrete)
-                    env.write(regenerate_views=False)
-            env.install_all(cli_args, **kwargs)
+                # This won't find specs added to the env since last
+                # concretize, therefore should we consider enforcing
+                # concretization of the env before allowing to install
+                # specs?
+                m_spec = env.matching_spec(abstract)
+
+                # If there is any ambiguity in the above call to matching_spec
+                # (i.e. if more than one spec in the environment matches), then
+                # SpackEnvironmentError is rasied, with a message listing the
+                # the matches.  Getting to this point means there were either
+                # no matches or exactly one match.
+
+                if not m_spec:
+                    tty.debug('{0} matched nothing in the env'.format(
+                        abstract.name))
+                    # no matches in the env
+                    if cli_args.no_add:
+                        msg = ('You asked to install {0} without adding it ' +
+                               '(--no-add), but no such spec exists in ' +
+                               'environment').format(abstract.name)
+                        tty.die(msg)
+                    else:
+                        tty.debug('adding {0} as a root'.format(abstract.name))
+                        specs_to_add.append((abstract, concrete))
+
+                    continue
+
+                tty.debug('exactly one match for {0} in env -> {1}'.format(
+                    m_spec.name, m_spec.dag_hash()))
+
+                if m_spec in env.roots() or cli_args.no_add:
+                    # either the single match is a root spec (and --no-add is
+                    # the default for roots) or --no-add was stated explictly
+                    tty.debug('just install {0}'.format(m_spec.name))
+                    specs_to_install.append(m_spec)
+                else:
+                    # the single match is not a root (i.e. it's a dependency),
+                    # and --no-add was not specified, so we'll add it as a
+                    # root before installing
+                    tty.debug('add {0} then install it'.format(m_spec.name))
+                    specs_to_add.append((abstract, concrete))
+
+            if specs_to_add:
+                tty.debug('Adding the following specs as roots:')
+                for abstract, concrete in specs_to_add:
+                    tty.debug('  {0}'.format(abstract.name))
+                    with env.write_transaction():
+                        specs_to_install.append(
+                            env.concretize_and_add(abstract, concrete))
+                        env.write(regenerate_views=False)
+
+            # Install the validated list of cli specs
+            if specs_to_install:
+                tty.debug('Installing the following cli specs:')
+                for s in specs_to_install:
+                    tty.debug('  {0}'.format(s.name))
+                env.install_specs(specs_to_install, args=cli_args, **kwargs)
         else:
             installs = [(concrete.package, kwargs) for _, concrete in specs]
             builder = PackageInstaller(installs)
diff --git a/lib/spack/spack/environment.py b/lib/spack/spack/environment.py
--- a/lib/spack/spack/environment.py
+++ b/lib/spack/spack/environment.py
@@ -1440,21 +1440,31 @@ def install_all(self, args=None, **install_args):
             args (Namespace): argparse namespace with command arguments
             install_args (dict): keyword install arguments
         """
+        self.install_specs(None, args=args, **install_args)
+
+    def install_specs(self, specs=None, args=None, **install_args):
         from spack.installer import PackageInstaller
+
         tty.debug('Assessing installation status of environment packages')
         # If "spack install" is invoked repeatedly for a large environment
         # where all specs are already installed, the operation can take
         # a large amount of time due to repeatedly acquiring and releasing
         # locks, this does an initial check across all specs within a single
-        # DB read transaction to reduce time spent in this case.
-        specs_to_install = self.uninstalled_specs()
+        # DB read transaction to reduce time spent in this case. In the next
+        # three lines we remove any already-installed root specs from the list
+        # to install.  However, uninstalled_specs() only considers root specs,
+        # so this will allow dep specs to be unnecessarily re-installed.
+        uninstalled_roots = self.uninstalled_specs()
+        specs_to_install = specs or uninstalled_roots
+        specs_to_install = [s for s in specs_to_install
+                            if s not in self.roots() or s in uninstalled_roots]
 
         if not specs_to_install:
             tty.msg('All of the packages are already installed')
             return
 
-        tty.debug('Processing {0} uninstalled specs'
-                  .format(len(specs_to_install)))
+        tty.debug('Processing {0} uninstalled specs'.format(
+            len(specs_to_install)))
 
         install_args['overwrite'] = install_args.get(
             'overwrite', []) + self._get_overwrite_specs()
@@ -1574,7 +1584,7 @@ def matching_spec(self, spec):
                             if abstract)
 
         if len(root_matches) == 1:
-            return root_matches[0][1]
+            return list(root_matches.items())[0][0]
 
         # More than one spec matched, and either multiple roots matched or
         # none of the matches were roots
diff --git a/var/spack/repos/builtin.mock/packages/b/package.py b/var/spack/repos/builtin.mock/packages/b/package.py
--- a/var/spack/repos/builtin.mock/packages/b/package.py
+++ b/var/spack/repos/builtin.mock/packages/b/package.py
@@ -13,5 +13,6 @@ class B(Package):
     url      = "http://www.example.com/b-1.0.tar.gz"
 
     version('1.0', '0123456789abcdef0123456789abcdef')
+    version('0.9', 'abcd456789abcdef0123456789abcdef')
 
     depends_on('test-dependency', type='test')


Test patch:
diff --git a/lib/spack/spack/test/cmd/common/arguments.py b/lib/spack/spack/test/cmd/common/arguments.py
--- a/lib/spack/spack/test/cmd/common/arguments.py
+++ b/lib/spack/spack/test/cmd/common/arguments.py
@@ -96,3 +96,21 @@ def test_multiple_env_match_raises_error(mock_packages, mutable_mock_env_path):
             spack.cmd.matching_spec_from_env(spack.cmd.parse_specs(['a'])[0])
 
     assert 'matches multiple specs' in exc_info.value.message
+
+
+@pytest.mark.usefixtures('config')
+def test_root_and_dep_match_returns_root(mock_packages, mutable_mock_env_path):
+    e = ev.create('test')
+    e.add('b@0.9')
+    e.add('a foobar=bar')  # Depends on b, should choose b@1.0
+    e.concretize()
+    with e:
+        # This query matches the root b and b as a dependency of a. In that
+        # case the root instance should be preferred.
+        env_spec1 = spack.cmd.matching_spec_from_env(
+            spack.cmd.parse_specs(['b'])[0])
+        assert env_spec1.satisfies('@0.9')
+
+        env_spec2 = spack.cmd.matching_spec_from_env(
+            spack.cmd.parse_specs(['b@1.0'])[0])
+        assert env_spec2
diff --git a/lib/spack/spack/test/cmd/install.py b/lib/spack/spack/test/cmd/install.py
--- a/lib/spack/spack/test/cmd/install.py
+++ b/lib/spack/spack/test/cmd/install.py
@@ -32,6 +32,7 @@
 mirror = SpackCommand('mirror')
 uninstall = SpackCommand('uninstall')
 buildcache = SpackCommand('buildcache')
+find = SpackCommand('find')
 
 
 @pytest.fixture()
@@ -712,6 +713,112 @@ def test_install_only_dependencies_of_all_in_env(
                 assert os.path.exists(dep.prefix)
 
 
+def test_install_no_add_in_env(tmpdir, mock_fetch, install_mockery,
+                               mutable_mock_env_path):
+    # To test behavior of --no-add option, we create the following environment:
+    #
+    #     mpileaks
+    #         ^callpath
+    #             ^dyninst
+    #                 ^libelf@0.8.13     # or latest, really
+    #                 ^libdwarf
+    #         ^mpich
+    #     libelf@0.8.10
+    #     a~bvv
+    #         ^b
+    #     a
+    #         ^b
+    e = ev.create('test')
+    e.add('mpileaks')
+    e.add('libelf@0.8.10')  # so env has both root and dep libelf specs
+    e.add('a')
+    e.add('a ~bvv')
+    e.concretize()
+    env_specs = e.all_specs()
+
+    a_spec = None
+    b_spec = None
+    mpi_spec = None
+
+    # First find and remember some target concrete specs in the environment
+    for e_spec in env_specs:
+        if e_spec.satisfies(Spec('a ~bvv')):
+            a_spec = e_spec
+        elif e_spec.name == 'b':
+            b_spec = e_spec
+        elif e_spec.satisfies(Spec('mpi')):
+            mpi_spec = e_spec
+
+    assert(a_spec)
+    assert(a_spec.concrete)
+
+    assert(b_spec)
+    assert(b_spec.concrete)
+    assert(b_spec not in e.roots())
+
+    assert(mpi_spec)
+    assert(mpi_spec.concrete)
+
+    # Activate the environment
+    with e:
+        # Assert using --no-add with a spec not in the env fails
+        inst_out = install(
+            '--no-add', 'boost', fail_on_error=False, output=str)
+
+        assert('no such spec exists in environment' in inst_out)
+
+        # Ensure using --no-add with an ambiguous spec fails
+        with pytest.raises(ev.SpackEnvironmentError) as err:
+            inst_out = install(
+                '--no-add', 'a', output=str)
+
+        assert('a matches multiple specs in the env' in str(err))
+
+        # With "--no-add", install an unambiguous dependency spec (that already
+        # exists as a dep in the environment) using --no-add and make sure it
+        # gets installed (w/ deps), but is not added to the environment.
+        install('--no-add', 'dyninst')
+
+        find_output = find('-l', output=str)
+        assert('dyninst' in find_output)
+        assert('libdwarf' in find_output)
+        assert('libelf' in find_output)
+        assert('callpath' not in find_output)
+
+        post_install_specs = e.all_specs()
+        assert all([s in env_specs for s in post_install_specs])
+
+        # Make sure we can install a concrete dependency spec from a spec.yaml
+        # file on disk, using the ``--no-add` option, and the spec is installed
+        # but not added as a root
+        mpi_spec_yaml_path = tmpdir.join('{0}.yaml'.format(mpi_spec.name))
+        with open(mpi_spec_yaml_path.strpath, 'w') as fd:
+            fd.write(mpi_spec.to_yaml(hash=ht.full_hash))
+
+        install('--no-add', '-f', mpi_spec_yaml_path.strpath)
+        assert(mpi_spec not in e.roots())
+
+        find_output = find('-l', output=str)
+        assert(mpi_spec.name in find_output)
+
+        # Without "--no-add", install an unambiguous depependency spec (that
+        # already exists as a dep in the environment) without --no-add and make
+        # sure it is added as a root of the environment as well as installed.
+        assert(b_spec not in e.roots())
+
+        install('b')
+
+        assert(b_spec in e.roots())
+        assert(b_spec not in e.uninstalled_specs())
+
+        # Without "--no-add", install a novel spec and make sure it is added
+        # as a root and installed.
+        install('bowtie')
+
+        assert(any([s.name == 'bowtie' for s in e.roots()]))
+        assert(not any([s.name == 'bowtie' for s in e.uninstalled_specs()]))
+
+
 def test_install_help_does_not_show_cdash_options(capsys):
     """
     Make sure `spack install --help` does not describe CDash arguments