Repository: spack/spack

Issue:
Clingo should try harder to avoid deprecated packages
### Steps to reproduce the issue

```console
$ spack spec slurm | grep glib
    ^glib@2.53.1
```

but this version of glib is marked as `deprecated`.

I can nudge clingo into taking a newer version of glib:

```console
$ spack spec slurm ^glib@2.60: | grep glib
    ^glib@2.66.7
```
and it concretizes fine to the latest glib.

The problem is that `glib@2.66.7` depends on `meson` whereas `glib@2.53` does not, and this ultimately pulls in a virtual dependency on `uuid` only for the former.

Apparently clingo prefers fewer virtual dependencies over newer versions, so it ends up preferring `glib@2.53` even though it is deprecated.

When setting `variant('uuid',     default=False,  description='Build uuid module')` by default in Python, `spack spec slurm` concretizes to using `glibc@2.66.7` indeed.

So I think there's 2 issues:

- Clingo should try to prefer new versions over fewer virtual deps
- Clingo should avoid concretizing to deprecated packages when there is an alternative

### Information on your system

* **Spack:** 0.16.0-1812-8d2944bf77
* **Python:** 3.8.7
* **Platform:** linux-ubuntu20.04-zen2
* **Concretizer:** clingo

### Additional information

- [x] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [x] I have searched the issues of this repo and believe this is not a duplicate




Discussion:
Clingo does not currently take "deprecated" into account, as it was added after the new concretizer was introduced. We should add support for this, just like we support "preferred".
@adamjstewart You're correct and I agree that accounting for "deprecated" should be added, but the problem here is inherently different (as I saw debugging this spec). 

We have rules to minimize weights on providers and the weights used are strictly positive at the moment (there were reasons for that, need to check if they still hold). This introduces a bias where a new vdep makes the minimization score higher. 

The optimization to select the latest version of a package kicks in at lower priority, and that's for good reasons. If e.g. OpenMPI is your preferred provider, latest version v4.0.1, and MPICH is the second best, latest version v3.2.0 we don't want that a spec that conflicts with OpenMPI v4.0.1 specifically makes the solver decide to switch to MPICH v3.2.0. We'd rather want to use OpenMPI v4.0.0.

Part of the problem is that there is no easy way (or at least I haven't found it) to reason about average quantities rather than sums over the entire DAG. I don't think that would solve all cases but it would for sure mitigate the occurrences of these issues. Another mitigation, if possible, would be to check if we can score the best possible solution at `0` which wouldn't affect the minimisation score.
Can the dependency count minimization aspect of the new concretizer be disabled via a setting?
> The optimization to select the latest version of a package kicks in at lower priority, and that's for good reasons. If e.g. OpenMPI is your preferred provider, latest version v4.0.1, and MPICH is the second best, latest version v3.2.0 we don't want that a spec that conflicts with OpenMPI v4.0.1 specifically makes the solver decide to switch to MPICH v3.2.0. We'd rather want to use OpenMPI v4.0.0.

This is more acceptable than selecting a very old version because the newer versions pull in more virtual deps... Should we just default to concretizing to new versions? The user can always force `pkg ^openmpi` if they're unhappy
> This is more acceptable than selecting a very old version because the newer versions pull in more virtual deps... Should we just default to concretizing to new versions?

Not as simple as that I fear. Let me have a look at this issue again to see if I can find an easy way out.
@eugeneswalker 

> Can the dependency count minimization aspect of the new concretizer be disabled via a setting?

There's no direct dependency count minimization... What is a spec that reproduces your issue? Would:
```console
$ spack solve geopm
```
be sufficient? Asking since I want to determine if it's the same issue or two different but similar ones.
I am having a look at this issue. I'll solve it as suggested in https://github.com/spack/spack/issues/22351#issuecomment-802507256 and give the highest priority to the minimization of deprecated versions. This would be, in my opinion, a sensible behavior since usually versions are deprecated for good reasons (security, serious bugs, etc.).

Nonetheless, I think our ASP-based solver should select in both of the cases reported here the newest version of `glib` and `openssl`  regardless of whether the old versions are deprecated or not. More details on what makes the current logic program select old versions are in https://github.com/spack/spack/issues/22351#issuecomment-802642995. I will not address this issue in the PR that I am about to submit.

Resolving patch:
diff --git a/lib/spack/spack/solver/asp.py b/lib/spack/spack/solver/asp.py
--- a/lib/spack/spack/solver/asp.py
+++ b/lib/spack/spack/solver/asp.py
@@ -411,6 +411,7 @@ def __init__(self):
         self.gen = None  # set by setup()
         self.possible_versions = {}
         self.versions_in_package_py = {}
+        self.deprecated_versions = {}
         self.versions_from_externals = {}
         self.possible_virtuals = None
         self.possible_compilers = []
@@ -481,6 +482,11 @@ def pkg_version_rules(self, pkg):
         for i, v in enumerate(most_to_least_preferred):
             self.gen.fact(fn.version_declared(pkg.name, v, i))
 
+        # Declare deprecated versions for this package, if any
+        deprecated = self.deprecated_versions[pkg.name]
+        for v in sorted(deprecated):
+            self.gen.fact(fn.deprecated_version(pkg.name, v))
+
     def spec_versions(self, spec):
         """Return list of clauses expressing spec's version constraints."""
         spec = specify(spec)
@@ -1020,12 +1026,16 @@ def build_version_dict(self, possible_pkgs, specs):
         self.possible_versions = collections.defaultdict(set)
         self.versions_in_package_py = collections.defaultdict(set)
         self.versions_from_externals = collections.defaultdict(set)
+        self.deprecated_versions = collections.defaultdict(set)
 
         for pkg_name in possible_pkgs:
             pkg = spack.repo.get(pkg_name)
-            for v in pkg.versions:
+            for v, version_info in pkg.versions.items():
                 self.versions_in_package_py[pkg_name].add(v)
                 self.possible_versions[pkg_name].add(v)
+                deprecated = version_info.get('deprecated', False)
+                if deprecated:
+                    self.deprecated_versions[pkg_name].add(v)
 
         for spec in specs:
             for dep in spec.traverse():
@@ -1553,6 +1563,10 @@ def reorder_flags(self):
             check_same_flags(spec.compiler_flags, flags)
             spec.compiler_flags.update(flags)
 
+    def deprecated(self, pkg, version):
+        msg = 'using "{0}@{1}" which is a deprecated version'
+        tty.warn(msg.format(pkg, version))
+
     def build_specs(self, function_tuples):
         # Functions don't seem to be in particular order in output.  Sort
         # them here so that directives that build objects (like node and
diff --git a/var/spack/repos/builtin.mock/packages/deprecated-versions/package.py b/var/spack/repos/builtin.mock/packages/deprecated-versions/package.py
new file mode 100644
--- /dev/null
+++ b/var/spack/repos/builtin.mock/packages/deprecated-versions/package.py
@@ -0,0 +1,12 @@
+# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+class DeprecatedVersions(Package):
+    """Package with the most recent version deprecated"""
+
+    homepage = "http://www.example.com"
+    url = "http://www.example.com/c-1.0.tar.gz"
+
+    version('1.1.0', '0123456789abcdef0123456789abcdef', deprecated=True)
+    version('1.0.0', '0123456789abcdef0123456789abcdef')


Test patch:
diff --git a/lib/spack/spack/test/concretize.py b/lib/spack/spack/test/concretize.py
--- a/lib/spack/spack/test/concretize.py
+++ b/lib/spack/spack/test/concretize.py
@@ -1228,3 +1228,19 @@ def test_multivalued_variants_from_cli(self, spec_str, expected_dict):
 
         for constraint, value in expected_dict.items():
             assert s.satisfies(constraint) == value
+
+    @pytest.mark.regression('22351')
+    @pytest.mark.parametrize('spec_str,expected', [
+        # Version 1.1.0 is deprecated and should not be selected, unless we
+        # explicitly asked for that
+        ('deprecated-versions', ['deprecated-versions@1.0.0']),
+        ('deprecated-versions@1.1.0', ['deprecated-versions@1.1.0']),
+    ])
+    def test_deprecated_versions_not_selected(self, spec_str, expected):
+        if spack.config.get('config:concretizer') == 'original':
+            pytest.xfail('Known failure of the original concretizer')
+
+        s = Spec(spec_str).concretized()
+
+        for abstract_spec in expected:
+            assert abstract_spec in s