Repository: spack/spack

Issue:
ASP solver does not handle missing packages during `--reuse`
### Steps to reproduce

The clingo concretizer fails when attempting to `--reuse` existing builds when a package that at exists in the builtin package repo was previously installed using a custom fork of the package (with site-specific modifications) located in a higher-priority local package repo has since been removed from the local repo so that a newer builtin-repo version of the package is preferred for future builds.

```console
# add modified package with same name as a builtin package to a local repo
spack install <package>
# remove modified package from local repo so builtin version is used
spack -vd --stacktrace concretize --reuse -f
```

Ideally, a `spack.repo.UnknownPackageError` would mark the previous build as ineligible for re-use and concertize specs using only packages still existing in available repos, even under `--reuse`.

While it's certainly possible to restore the missing package to the local repo as a direct clone of the upstream package, my hope is that when upstream packages are updated to handle issues that previously required site-specific hacks applied through a fork of the package existing in a private repo, I would be able to drop my private fork of the package.

### Error message

```console
spack -vd --stacktrace concretize --reuse -f
```

```console
[PID=3602251] Traceback (most recent call last):                                                                                                                                                                                               
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/util/parallel.py", line 53, in __call__                                                                                                      
    value = self.func(*args, **kwargs)                                                                                                                                                                                                         
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/environment/environment.py", line 2057, in _concretize_task                                                                                  
    return _concretize_from_constraints(spec_constraints, tests, reuse)                                                                                                                                                                        
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/environment/environment.py", line 2035, in _concretize_from_constraints                                                                      
    return s.concretized(tests=tests, reuse=reuse)                                                                                                                                                                                             
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/spec.py", line 2692, in concretized                                                                                                          
    clone.concretize(tests=tests, reuse=reuse)                                                                                                                                                                                                 
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/spec.py", line 2648, in concretize                                                                                                           
    self._new_concretize(tests, reuse=reuse)                                                                                                                                                                                                   
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/spec.py", line 2619, in _new_concretize                                                                                                      
    result = spack.solver.asp.solve([self], tests=tests, reuse=reuse)                                                                                                                                                                          
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/solver/asp.py", line 2054, in solve                                                                                                          
    setup, specs, dump, models, timers, stats, tests, reuse                                                                                                                                                                                    
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/solver/asp.py", line 621, in solve                                                                                                           
    answers = builder.build_specs(tuples)                                                                                                                                                                                                      
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/solver/asp.py", line 1995, in build_specs                                                                                                    
    spack.spec.Spec.inject_patches_variant(root)                                                                                                                                                                                               
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/spec.py", line 2534, in inject_patches_variant                                                                                               
    pkg_deps = dspec.parent.package_class.dependencies                                                                                                                                                                                         
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/spec.py", line 1305, in package_class                                                                                                        
    return spack.repo.path.get_pkg_class(self.fullname)                                                                                                                                                                                        
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/repo.py", line 647, in get_pkg_class                                                                                                         
    return self.repo_for_pkg(pkg_name).get_pkg_class(pkg_name)                                                                                                                                                                                 
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/repo.py", line 1107, in get_pkg_class                                                                                                        
    module = self._get_pkg_module(pkg_name)                                                                                                                                                                                                    
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/repo.py", line 1067, in _get_pkg_module                                                                                                      
    raise UnknownPackageError(pkg_name, self)                                                                                                                                                                                                  
spack.repo.UnknownPackageError: Package 'ncl' not found in repository '/ccs/home/belhorn/development/olcf-spack-environments/share/spack/repos/olcf'
```

### Information on your system

$ spack debug report
* **Spack:** 0.17.1
* **Python:** 3.6.8
* **Platform:** linux-rhel8-zen2
* **Concretizer:** clingo

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output
Demote UnknownNamespaceError from --reuse with upstreams into a warning
closes #30092 

When concretizing with the --reuse flag, spack goes through all
installations and errors if a package comes from an unknown namespace.
This makes certain upstreams really hard to use. It's a likely scenario that an upstream contains installations from several spack repos,
and it is an unnecessary burden to setup all those repos that may not be
of interest to the user.


Discussion:
Reproduced by creating a repo with just `zlib`:
```console
myrepo$ tree
.
├── packages
│   └── zlib
│       ├── configure-cc.patch
│       ├── package.py
│       └── w_patch.patch
└── repo.yaml

2 directories, 4 files
```
then:
```console
$ spack repo add .
==> Added repo with namespace 'myrepo'.
$ spack install zlib
[ ... ]
$ spack repo remove .
$ spack solve -I --reuse zlib
==> [2022-04-15-15:49:22.931162] Imported solve from built-in commands
==> [2022-04-15-15:49:22.932430] Imported solve from built-in commands
==> [2022-04-15-15:49:22.933327] Reading config file /home/culpo/PycharmProjects/spack/etc/spack/defaults/bootstrap.yaml
==> [2022-04-15-15:49:22.942795] Reading config file /home/culpo/PycharmProjects/spack/etc/spack/defaults/config.yaml
==> [2022-04-15-15:49:22.960173] Reading config file /home/culpo/PycharmProjects/spack/etc/spack/defaults/config.yaml
==> [2022-04-15-15:49:22.973288] DATABASE LOCK TIMEOUT: 3s
==> [2022-04-15-15:49:22.973331] PACKAGE LOCK TIMEOUT: No timeout
==> [2022-04-15-15:49:22.973628] Reading config file /home/culpo/PycharmProjects/spack/etc/spack/defaults/bootstrap.yaml
==> [2022-04-15-15:49:22.981017] DATABASE LOCK TIMEOUT: 3s
==> [2022-04-15-15:49:22.981057] PACKAGE LOCK TIMEOUT: No timeout
==> [2022-04-15-15:49:22.981105] [BOOTSTRAP CONFIG SCOPE] name=_builtin
==> [2022-04-15-15:49:22.981383] [BOOTSTRAP CONFIG SCOPE] name=defaults, path=/home/culpo/PycharmProjects/spack/etc/spack/defaults
==> [2022-04-15-15:49:22.981413] [BOOTSTRAP CONFIG SCOPE] name=defaults/linux, path=/home/culpo/PycharmProjects/spack/etc/spack/defaults/linux
==> [2022-04-15-15:49:22.981436] [BOOTSTRAP CONFIG SCOPE] name=bootstrap, path=/home/culpo/.spack/bootstrap/config
==> [2022-04-15-15:49:22.981457] [BOOTSTRAP CONFIG SCOPE] name=bootstrap/linux, path=/home/culpo/.spack/bootstrap/config/linux
==> [2022-04-15-15:49:22.981662] Reading config file /home/culpo/.spack/bootstrap/config/linux/compilers.yaml
==> [2022-04-15-15:49:23.002132] [BOOTSTRAP ROOT SPEC] clingo-bootstrap@spack+python %gcc target=x86_64
==> [2022-04-15-15:49:23.002191] [BOOTSTRAP MODULE clingo] Try importing from Python
==> [2022-04-15-15:49:23.002360] Reading config file /home/culpo/PycharmProjects/spack/etc/spack/defaults/bootstrap.yaml
==> [2022-04-15-15:49:23.114100] Reading config file /home/culpo/PycharmProjects/spack/etc/spack/defaults/config.yaml
==> [2022-04-15-15:49:23.195629] [BOOTSTRAP MODULE clingo] The installed spec "clingo-bootstrap@spack+python %gcc target=x86_64 ^python@3.8 /shqedxgvjnhiwdcdrvjhbd73jaevv7wt" provides the "clingo" Python module
==> [2022-04-15-15:49:23.195884] Reading config file /home/culpo/PycharmProjects/spack/etc/spack/defaults/concretizer.yaml
==> [2022-04-15-15:49:23.197623] Reading config file /home/culpo/PycharmProjects/spack/etc/spack/defaults/repos.yaml
==> [2022-04-15-15:49:23.198864] Reading config file /home/culpo/.spack/repos.yaml
==> [2022-04-15-15:49:23.228975] Reading config file /home/culpo/.spack/linux/compilers.yaml
==> [2022-04-15-15:49:23.236692] Reading config file /home/culpo/PycharmProjects/spack/etc/spack/defaults/config.yaml
==> [2022-04-15-15:49:23.249716] Reading config file /home/culpo/PycharmProjects/spack/etc/spack/defaults/packages.yaml
==> [2022-04-15-15:49:23.271535] Reading config file /home/culpo/PycharmProjects/spack/etc/spack/defaults/mirrors.yaml
==> [2022-04-15-15:49:24.034166] Fetching index from https://mirror.spack.io/build_cache/index.json
==> [2022-04-15-15:49:24.913261] UnknownNamespaceError: Unknown namespace: myrepo
==> [2022-04-15-15:49:24.913312] Error: Unknown namespace: myrepo
Traceback (most recent call last):
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/main.py", line 915, in main
    return _main(argv)
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/main.py", line 870, in _main
    return finish_parse_and_run(parser, cmd_name, env_format_error)
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/main.py", line 898, in finish_parse_and_run
    return _invoke_command(command, parser, args, unknown)
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/main.py", line 550, in _invoke_command
    return_val = command(parser, args)
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/cmd/solve.py", line 111, in solve
    result = solver.solve(
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/solver/asp.py", line 2141, in solve
    return self.driver.solve(
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/solver/asp.py", line 641, in solve
    answers = builder.build_specs(tuples)
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/solver/asp.py", line 2038, in build_specs
    repo = spack.repo.path.repo_for_pkg(spec)
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/repo.py", line 933, in repo_for_pkg
    raise UnknownNamespaceError(namespace)
spack.repo.UnknownNamespaceError: Unknown namespace: myrepo

```


Resolving patch:
diff --git a/lib/spack/spack/solver/asp.py b/lib/spack/spack/solver/asp.py
--- a/lib/spack/spack/solver/asp.py
+++ b/lib/spack/spack/solver/asp.py
@@ -1659,6 +1659,12 @@ def _facts_from_concrete_spec(self, spec, possible):
         # be dependencies (don't tell it about the others)
         h = spec.dag_hash()
         if spec.name in possible and h not in self.seen_hashes:
+            try:
+                # Only consider installed packages for repo we know
+                spack.repo.path.repo_for_pkg(spec)
+            except spack.repo.UnknownNamespaceError:
+                return
+
             # this indicates that there is a spec like this installed
             self.gen.fact(fn.installed_hash(spec.name, h))
 


Test patch:
diff --git a/lib/spack/spack/test/concretize.py b/lib/spack/spack/test/concretize.py
--- a/lib/spack/spack/test/concretize.py
+++ b/lib/spack/spack/test/concretize.py
@@ -204,6 +204,28 @@ def change(self, changes=None):
     return _changing_pkg
 
 
+@pytest.fixture()
+def additional_repo_with_c(tmpdir_factory, mutable_mock_repo):
+    """Add a repository with a simple package"""
+    repo_dir = tmpdir_factory.mktemp('myrepo')
+    repo_dir.join('repo.yaml').write("""
+repo:
+  namespace: myrepo
+""", ensure=True)
+    packages_dir = repo_dir.ensure('packages', dir=True)
+    package_py = """
+class C(Package):
+    homepage = "http://www.example.com"
+    url      = "http://www.example.com/root-1.0.tar.gz"
+
+    version(1.0, sha256='abcde')
+"""
+    packages_dir.join('c', 'package.py').write(package_py, ensure=True)
+    repo = spack.repo.Repo(str(repo_dir))
+    mutable_mock_repo.put_first(repo)
+    return repo
+
+
 # This must use the mutable_config fixture because the test
 # adjusting_default_target_based_on_compiler uses the current_host fixture,
 # which changes the config.
@@ -1607,3 +1629,19 @@ def test_installed_version_is_selected_only_for_reuse(
             new_root = Spec('root').concretized()
 
         assert not new_root['changing'].satisfies('@1.0')
+
+    @pytest.mark.regression('28259')
+    def test_reuse_with_unknown_namespace_dont_raise(
+            self, additional_repo_with_c, mutable_mock_repo
+    ):
+        s = Spec('c').concretized()
+        assert s.namespace == 'myrepo'
+        s.package.do_install(fake=True, explicit=True)
+
+        # TODO: To mock repo removal we need to recreate the RepoPath
+        mutable_mock_repo.remove(additional_repo_with_c)
+        spack.repo.path = spack.repo.RepoPath(*spack.repo.path.repos)
+
+        with spack.config.override("concretizer:reuse", True):
+            s = Spec('c').concretized()
+        assert s.namespace == 'builtin.mock'