Repository: spack/spack

Issue:
Spack mirror --all fails for packages conflicting with the current platform
### Steps to reproduce

Run `spack mirror --all -n all`

The `spec.concretize()` @alalazo added needs to be in a try/except of some sort.

### Error message

```
lib/spack/spack/error.py:54 ==> [2022-07-25-15:24:39.514303] Error: Package 'armpl' not found.
Traceback (most recent call last):
  File "/storage/spack/spack/lib/spack/spack/repo.py", line 1291, in get_pkg_class
    module = importlib.import_module(fullname)
  File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'spack.pkg.builtin.armpl'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/storage/spack/spack/lib/spack/spack/main.py", line 893, in main
    return _main(argv)
  File "/storage/spack/spack/lib/spack/spack/main.py", line 848, in _main
    return finish_parse_and_run(parser, cmd_name, env_format_error)
  File "/storage/spack/spack/lib/spack/spack/main.py", line 876, in finish_parse_and_run
    return _invoke_command(command, parser, args, unknown)
  File "/storage/spack/spack/lib/spack/spack/main.py", line 533, in _invoke_command
    return_val = command(parser, args)
  File "/storage/spack/spack/lib/spack/spack/cmd/mirror.py", line 380, in mirror
    action[args.mirror_command](args)
  File "/storage/spack/spack/lib/spack/spack/cmd/mirror.py", line 336, in mirror_create
    directory, mirror_specs, args.skip_unstable_versions)
  File "/storage/spack/spack/lib/spack/spack/mirror.py", line 537, in create
    _add_single_spec(spec, mirror_cache, mirror_stats)
  File "/storage/spack/spack/lib/spack/spack/mirror.py", line 642, in _add_single_spec
    spec = spec.concretized()
  File "/storage/spack/spack/lib/spack/spack/spec.py", line 3024, in concretized
    clone.concretize(tests=tests)
  File "/storage/spack/spack/lib/spack/spack/spec.py", line 2940, in concretize
    self._new_concretize(tests)
  File "/storage/spack/spack/lib/spack/spack/spec.py", line 2914, in _new_concretize
    result = solver.solve([self], tests=tests)
  File "/storage/spack/spack/lib/spack/spack/solver/asp.py", line 2309, in solve
    setup_only=setup_only,
  File "/storage/spack/spack/lib/spack/spack/solver/asp.py", line 605, in solve
    setup.setup(self, specs, reuse=reuse)
  File "/storage/spack/spack/lib/spack/spack/solver/asp.py", line 1854, in setup
    self.build_version_dict(possible, specs)
  File "/storage/spack/spack/lib/spack/spack/solver/asp.py", line 1397, in build_version_dict
    pkg_cls = spack.repo.path.get_pkg_class(pkg_name)
  File "/storage/spack/spack/lib/spack/spack/repo.py", line 954, in get_pkg_class
    return self.repo_for_pkg(pkg_name).get_pkg_class(pkg_name)
  File "/storage/spack/spack/lib/spack/spack/repo.py", line 1293, in get_pkg_class
    raise UnknownPackageError(pkg_name)
spack.repo.UnknownPackageError: Package 'armpl' not found. 
```

### Information on your system

 spack debug report
* **Spack:** 0.19.0.dev0
* **Python:** 3.6.8
* **Platform:** linux-rhel7-broadwell
* **Concretizer:** clingo


### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:
This is caused by an issue with ~/.spack directories. Removing that directory and re-bootstrapping fixes it.
@Paul-Ferrell Sorry I didn't see this issue before. This is a known problem that happens when:
1. A package is renamed, or deleted etc.
2. A user is pulling from a version before the change at 1. to a version after the change at 1

In that case the cache for the builtin repository is not updated and you'll see weird error messages like the one you posted. Usually:
```console
$ spack clean -a
```
solves the issue.
@alalazo I just reopened this. The problem extends beyond just what you referred to. Any spec that can't concretize on the given system also throws an uncaught exception, which breaks the mirroring process. This happens pretty quickly, as 7zip conflicts with `platform=linux`. 

I think actually concretizing when using `--all` is a bad idea in general because of this. When we say `--all`, we mean *ALL* even if the package isn't supported on the current platform.
> I think actually concretizing when using --all is a bad idea in general because of this. When we say --all, we mean ALL even if the package isn't supported on the current platform.

I think we can't implement this idea yet, though in principle I concur with you, at least until we allow to have multiple URLs associated with the same version in a package. See for instance what the `arm` or `cuda` package do to declare their versions. 

In any case I'll check what I can do to mitigate this issue.

Resolving patch:
diff --git a/lib/spack/spack/mirror.py b/lib/spack/spack/mirror.py
--- a/lib/spack/spack/mirror.py
+++ b/lib/spack/spack/mirror.py
@@ -632,9 +632,26 @@ def error(self):
 
 
 def _add_single_spec(spec, mirror, mirror_stats):
+    """Add a single spec to a mirror.
+
+    Args:
+        spec (spack.spec.Spec): spec to be added. If not concrete it will
+            be concretized.
+        mirror (spack.mirror.Mirror): mirror where to add the spec.
+        mirror_stats (spack.mirror.MirrorStats): statistics on the current mirror
+
+    Return:
+        True if the spec was added successfully, False otherwise
+    """
     # Ensure that the spec is concrete, since we'll stage it later
-    if not spec.concrete:
-        spec = spec.concretized()
+    try:
+        if not spec.concrete:
+            spec = spec.concretized()
+    except Exception as e:
+        msg = "Skipping '{0}', as it fails to concretize [{1}]".format(spec, str(e))
+        tty.debug(msg)
+        mirror_stats.error()
+        return False
 
     tty.msg("Adding package {pkg} to mirror".format(pkg=spec.format("{name}{@version}")))
     num_retries = 3
@@ -662,6 +679,9 @@ def _add_single_spec(spec, mirror, mirror_stats):
                 getattr(exception, "message", exception),
             )
         mirror_stats.error()
+        return False
+
+    return True
 
 
 def push_url_from_directory(output_directory):


Test patch:
diff --git a/lib/spack/spack/test/mirror.py b/lib/spack/spack/test/mirror.py
--- a/lib/spack/spack/test/mirror.py
+++ b/lib/spack/spack/test/mirror.py
@@ -311,3 +311,19 @@ def test_get_all_versions(specs, expected_specs):
     output_list = [str(x) for x in output_list]
     # Compare sets since order is not important
     assert set(output_list) == set(expected_specs)
+
+
+@pytest.mark.regression("31736")
+def test_non_concretizable_spec_does_not_raise():
+    s = Spec("doesnotexist")
+
+    class Stats(object):
+        called = False
+
+        def error(self):
+            self.called = True
+
+    mirror_stats = Stats()
+    result = spack.mirror._add_single_spec(s, mirror=None, mirror_stats=mirror_stats)
+    assert result is False
+    assert mirror_stats.called is True