Repository: spack/spack

Issue:
Issues with the new clingo concretizer
<!-- Explain, in a clear and concise way, the command you ran and the result you were trying to achieve.
Example: "I ran `spack find` to list all the installed packages and ..." -->
I am trying the new concretizer and ran into two issues. (I also posted these in a thread on Slack.)

### Steps to reproduce the issue
The first issue is that the clingo concretizer seems to ignore some settings from `packages.yaml` like these:
```yaml
packages:
  netlib-lapack:
    variants: +external-blas~lapacke
```
Basically, when I run
```console
spack spec -I netlib-lapack
```
I get
```console
Input spec
--------------------------------
 -   netlib-lapack

Concretized
--------------------------------
 -   netlib-lapack@3.8.0%xl_r@16.1~external-blas~ipo~lapacke+shared~xblas build_type=RelWithDebInfo patches=0046d24487c0b7cbc84527fc36300d4e970cb4191b44e8b1619523fae15137f1,5c79286f3d08a0b0f1f3acba2a92ee698647716ba8c6c0ae20c9cc2713e6f139,ad3d41fe9ff94b7661e09fceaf2b2e4b8c83510c1465c016e161541b4429b5ee arch=linux-rhel7-power9le
 -       ^cmake@3.14.5%xl_r@16.1~doc+ncurses+openssl+ownlibs~qt patches=1c540040c7e203dd8e27aa20345ecb07fe06570d56410a24a266ae570b1c4c39 arch=linux-rhel7-power9le
```
As you can see the spec has `~external-blas` instead of `+external-blas` as specified in `packages.yaml`.

The second issue I noticed is that sometimes variants which are off by default are enabled (without explicitly being enabled anywhere). The specific case I ran into is this:
```console
spack spec -I mfem
```
which gives:
```console
Input spec
--------------------------------
 -   mfem

Concretized
--------------------------------
 -   mfem@4.2.0%xl_r@16.1~amgx~conduit~cuda~debug~examples~gnutls~gslib~lapack~libceed~libunwind+metis~miniapps~mpfr+mpi~netcdf~occa~openmp~petsc~pumi~raja~shared+static~strumpack~suite-sparse~sundials~superlu-dist~threadsafe~umpire+zlib cuda_arch=sm_60 timer=auto arch=linux-rhel7-power9le
 -       ^hypre@2.20.0%xl_r@16.1~complex~debug~int64~internal-superlu~mixedint+mpi~openmp+shared+superlu-dist patches=6e3336b1d62155f6350dfe42b0f9ea25d4fa0af60c7e540959139deb93a26059 arch=linux-rhel7-power9le
 -           ^netlib-lapack@3.8.0%xl_r@16.1~external-blas~ipo~lapacke+shared~xblas build_type=RelWithDebInfo patches=0046d24487c0b7cbc84527fc36300d4e970cb4191b44e8b1619523fae15137f1,5c79286f3d08a0b0f1f3acba2a92ee698647716ba8c6c0ae20c9cc2713e6f139,ad3d41fe9ff94b7661e09fceaf2b2e4b8c83510c1465c016e161541b4429b5ee arch=linux-rhel7-power9le
 -               ^cmake@3.14.5%xl_r@16.1~doc+ncurses+openssl+ownlibs~qt patches=1c540040c7e203dd8e27aa20345ecb07fe06570d56410a24a266ae570b1c4c39 arch=linux-rhel7-power9le
[+]          ^spectrum-mpi@rolling%xl_r@16.1 arch=linux-rhel7-power9le
 -           ^superlu-dist@6.4.0%xl_r@16.1+cuda~int64~ipo~openmp+shared build_type=RelWithDebInfo cuda_arch=none arch=linux-rhel7-power9le
[+]              ^cuda@10.1.243%xl_r@16.1 arch=linux-rhel7-power9le
[+]              ^metis@5.1.0%xl_r@16.1~gdb~int64~real64+shared build_type=Release patches=4991da938c1d3a1d3dea78e49bbebecba00273f98df2a656e38b83d55b281da1 arch=linux-rhel7-power9le
[+]              ^parmetis@4.0.3%xl_r@16.1~gdb~int64~ipo+shared build_type=RelWithDebInfo patches=4f892531eb0a807eb1b82e683a416d3e35154a455274cf9b162fb02054d11a5b,50ed2081bc939269689789942067c58b3e522c269269a430d5d34c00edbc5870,704b84f7c7444d4372cb59cca6e1209df4ef3b033bc4ee3cf50f369bce972a9d arch=linux-rhel7-power9le
[+]      ^zlib@1.2.7%xl_r@16.1+optimize+pic+shared arch=linux-rhel7-power9le
```
You can see that the variant `superlu-dist` of `hypre` is enabled while its default is off (and I do not have anything related to `mfem`, `hypre`, or `superlu-dist` in my `packages.yaml`).

Note that if I run
```console
spack spec -I hypre
```
I get a spec with `superlu-dist` disabled, as expected:
```console
Input spec
--------------------------------
 -   hypre

Concretized
--------------------------------
 -   hypre@2.20.0%xl_r@16.1~complex~debug~int64~internal-superlu~mixedint+mpi~openmp+shared~superlu-dist patches=6e3336b1d62155f6350dfe42b0f9ea25d4fa0af60c7e540959139deb93a26059 arch=linux-rhel7-power9le
 -       ^netlib-lapack@3.8.0%xl_r@16.1~external-blas~ipo~lapacke+shared~xblas build_type=RelWithDebInfo patches=0046d24487c0b7cbc84527fc36300d4e970cb4191b44e8b1619523fae15137f1,5c79286f3d08a0b0f1f3acba2a92ee698647716ba8c6c0ae20c9cc2713e6f139,ad3d41fe9ff94b7661e09fceaf2b2e4b8c83510c1465c016e161541b4429b5ee arch=linux-rhel7-power9le
 -           ^cmake@3.14.5%xl_r@16.1~doc+ncurses+openssl+ownlibs~qt patches=1c540040c7e203dd8e27aa20345ecb07fe06570d56410a24a266ae570b1c4c39 arch=linux-rhel7-power9le
[+]      ^spectrum-mpi@rolling%xl_r@16.1 arch=linux-rhel7-power9le
```

### Error Message

<!-- If Spack reported an error, provide the error message. If it did not report an error but the output appears incorrect, provide the incorrect output. If there was no error message and no output but the result is incorrect, describe how it does not match what you expect. -->
N/A

### Information on your system

<!-- Please include the output of `spack debug report` -->
The output from `spack debug report` is:
```console
* **Spack:** 0.16.0-34-8d5a93358
* **Python:** 3.8.2
* **Platform:** linux-rhel7-power9le
* **Concretizer:** clingo
```
These observations are from LLNL's Lassen machine with custom `packages.yaml`, `compilers.yaml`, and `config.yaml` (to enable clingo) which are all placed in `<spack-root>/etc/spack/`. I've only included above the portion of `packages.yaml` that is relevant. I can post the whole files too, if necessary.

<!-- If you have any relevant configuration detail (custom `packages.yaml` or `modules.yaml`, etc.) you can add that here as well. -->

### Additional information

<!-- These boxes can be checked by replacing [ ] with [x] or by clicking them after submitting the issue. -->
- [x] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [x] I have searched the issues of this repo and believe this is not a duplicate
- [ ] I have run the failing commands in debug mode and reported the output

<!-- We encourage you to try, as much as possible, to reduce your problem to the minimal example that still reproduces the issue. That would help us a lot in fixing it quickly and effectively!

If you want to ask a question about the tool (how to use it, what it can currently do, etc.), try the `#general` channel on our Slack first. We have a welcoming community and chances are you'll get your reply faster and without opening an issue.

Other than that, thanks for taking the time to contribute to Spack! -->



Discussion:
> As you can see the spec has ~external-blas instead of +external-blas as specified in packages.yaml

@vdobrev I get the same on my system, but only for `~external-blas`. For instance, if you add `~shared` to the same preference it is picked by the clingo concretizer. As far as I can tell in my case it's due to the concretizer not being able to find a proper standalone BLAS provider to use, since activating that variant implies:
```python
depends_on('blas', when='+external-blas')
```
Given that the configuration in `packages.yaml` is to be regarded as a preference, the concretizer drops that preference instead of erroring. If I am explicit about using that variant I get the following unsat core:

<details>
<summary>$ spack solve netlib-lapack+external-blas</summary>

```console
$ spack solve netlib-lapack+external-blas
==> The following constraints are unsatisfiable:
  node("amdblis") :- possible_provider("blas","amdblis",0).
  node("amdblis") :- possible_provider("blas","amdblis",1).
  node("atlas") :- possible_provider("blas","atlas",2).
  node("blis") :- possible_provider("blas","blis",3).
  node("blis") :- possible_provider("blas","blis",4).
  node("cray-libsci") :- possible_provider("blas","cray-libsci",5).
  node("essl") :- possible_provider("blas","essl",6).
  node("flexiblas") :- possible_provider("blas","flexiblas",7).
  node("fujitsu-ssl2") :- possible_provider("blas","fujitsu-ssl2",8).
  node("intel-mkl") :- possible_provider("blas","intel-mkl",9).
  node("intel-parallel-studio") :- possible_provider("blas","intel-parallel-studio",10).
  node("netlib-xblas") :- possible_provider("blas","netlib-xblas",12).
  node("nvhpc") :- possible_provider("blas","nvhpc",13).
  node("openblas") :- possible_provider("blas","openblas",14).
  node("veclibfort") :- possible_provider("blas","veclibfort",15).
  provides_virtual("amdblis","blas")
  provides_virtual("atlas","blas")
  provides_virtual("blis","blas")
  provides_virtual("cray-libsci","blas")
  provides_virtual("flexiblas","blas")
  provides_virtual("intel-mkl","blas")
  provides_virtual("intel-parallel-studio","blas")
  provides_virtual("netlib-lapack","blas")
  provides_virtual("netlib-xblas","blas")
  provides_virtual("nvhpc","blas")
  provides_virtual("openblas","blas")
  provides_virtual("veclibfort","blas")
  root("netlib-lapack")
  single_provider_for("blas",":") :- real_node("netlib-lapack"),node("netlib-lapack"),variant_value("netlib-lapack","external-blas","True").
  variant("netlib-lapack","external-blas")
  variant_possible_value("netlib-lapack","external-blas","False")
  variant_possible_value("netlib-lapack","external-blas","True")
  variant_set("netlib-lapack","external-blas","True")
  variant_single_value("netlib-lapack","external-blas")
  variant_value("netlib-lapack","external-blas","False") :- possible_provider("blas","netlib-lapack",11).
  virtual("blas")
```
</details>
I'm trying to understand if this unsat core is legitimate, or not. What is the configuration that you would expect your machine?


With the original concretizer I get:
```console
$ spack spec netlib-lapack
Input spec
--------------------------------
netlib-lapack

Concretized
--------------------------------
netlib-lapack@3.8.0%gcc@10.1.0+external-blas~ipo~lapacke+shared~xblas build_type=RelWithDebInfo patches=5c79286f3d08a0b0f1f3acba2a92ee698647716ba8c6c0ae20c9cc2713e6f139,ad3d41fe9ff94b7661e09fceaf2b2e4b8c83510c1465c016e161541b4429b5ee arch=linux-ubuntu18.04-broadwell
    ^cmake@3.19.0%gcc@10.1.0~doc+ncurses+openssl+ownlibs~qt arch=linux-ubuntu18.04-broadwell
        ^ncurses@6.2%gcc@10.1.0~symlinks+termlib arch=linux-ubuntu18.04-broadwell
            ^pkgconf@1.7.3%gcc@10.1.0 arch=linux-ubuntu18.04-broadwell
        ^openssl@1.1.1h%gcc@10.1.0+systemcerts arch=linux-ubuntu18.04-broadwell
            ^perl@5.32.0%gcc@10.1.0+cpanm+shared+threads arch=linux-ubuntu18.04-broadwell
                ^berkeley-db@18.1.40%gcc@10.1.0 arch=linux-ubuntu18.04-broadwell
                ^gdbm@1.18.1%gcc@10.1.0 patches=7f569b9287e9e5a4236636eef32ff3428f0462ec17aea359e0ebee07649f1f16 arch=linux-ubuntu18.04-broadwell
                    ^readline@8.0%gcc@10.1.0 arch=linux-ubuntu18.04-broadwell
            ^zlib@1.2.11%gcc@10.1.0+optimize+pic+shared arch=linux-ubuntu18.04-broadwell
    ^openblas@0.3.12%gcc@10.1.0~consistent_fpcsr~ilp64+pic+shared threads=none arch=linux-ubuntu18.04-broadwell
```
which I assume is wrong, since both `openblas` and `netlib-lapack` provide `lapack` in this DAG.
I confirm the first issue. In my case there are these clauses generated for `netlib-lapack`:
```
provides_virtual("netlib-lapack","blas").
node("netlib-lapack") :- possible_provider("blas","netlib-lapack",11).
variant_value("netlib-lapack","external-blas","False") :- possible_provider("blas","netlib-lapack",11).
```
which together with the following rules in `concretize.lp`:
```
provider(Package, Virtual)
  :- node(Package), provides_virtual(Package, Virtual).
```
effectively make `netlib-lapack` an unconditional provider for BLAS, which is wrong.
@v-dobrev #20086 Should solve the issue with `netlib-lapack`.
@v-dobrev I am unable to reproduce your second issue on my dev machine. That likely means that I need a closer match to your situation. Can you share your `packages.yaml` and `compilers.yaml` to start with? If so I'll try them on Marconi (CINECA), which should be similar enough to Lassen.
I've been hitting the second issue trying to update the `ecp-viz-sdk` to add `ecp-io-sdk` as a dependency.  To reproduce it:

```
$ spack spec ecp-viz-sdk
Input spec
--------------------------------
ecp-viz-sdk

Concretized
--------------------------------
ecp-viz-sdk@1.0%gcc@8.3.0~ascent~paraview~sz~vtkm~zfp arch=linux-centos7-haswell
```
This is expected since for the `ecp-viz-sdk` package the variants are all off by default.

Similary for the `ecp-io-sdk`:
```
$ spack spec ecp-io-sdk
Input spec
--------------------------------
ecp-io-sdk

Concretized
--------------------------------
ecp-io-sdk@1.0%gcc@8.3.0+adios2+darshan~faodel+hdf5+pnetcdf+unifyfs+veloc arch=linux-centos7-haswell
  <lots of output>
```
This is also expected because for the `ecp-io-sdk` the variants are all on by default (except `~faodel`)

The problem occurs when I add `ecp-io-sdk` as an unconditional dependency to `ecp-viz-sdk`:
```diff
diff --git a/var/spack/repos/builtin/packages/ecp-viz-sdk/package.py b/var/spack/repos/builtin/packages/ecp-viz-sdk/package.py
index 337ca17237..eb56240e21 100644
--- a/var/spack/repos/builtin/packages/ecp-viz-sdk/package.py
+++ b/var/spack/repos/builtin/packages/ecp-viz-sdk/package.py
@@ -30,6 +30,8 @@ class EcpVizSdk(BundlePackage):
     # variant('cinema', default=False, description="Enable Cinema")
     # variant('rover', default=False, description="Enable ROVER")
 
+    depends_on('ecp-io-sdk')
+
     depends_on('ascent+shared+mpi+fortran+openmp+python+vtkh+dray', when='+ascent')
     depends_on('catalyst', when='+catalyst')
     depends_on('paraview+shared+mpi+python3+hdf5+kits', when='+paraview')
```

Now when concretizing, all the unspecified variants of the `ecp-io-sdk` dependency are off instead of their default on, even though there is clearly no conflict since it's the only dependency:
```
$ spack spec ecp-viz-sdk
Input spec
--------------------------------
ecp-viz-sdk

Concretized
--------------------------------
ecp-viz-sdk@1.0%gcc@8.3.0~ascent~paraview~sz~vtkm~zfp arch=linux-centos7-haswell
    ^ecp-io-sdk@1.0%gcc@8.3.0~adios2~darshan~faodel~hdf5~pnetcdf~unifyfs~veloc arch=linux-centos7-haswell
```
> @v-dobrev #20086 Should solve the issue with `netlib-lapack`.

Yes, I can confirm that this issue is now resolved in the latest `develop` (983fb11de). Thanks!
> @v-dobrev I am unable to reproduce your second issue on my dev machine. That likely means that I need a closer match to your situation. Can you share your `packages.yaml` and `compilers.yaml` to start with? If so I'll try them on Marconi (CINECA), which should be similar enough to Lassen.

@alalazo, here are the `.yaml` config files I use on Lassen:
* [compilers.yaml.txt](https://github.com/spack/spack/files/5599940/compilers.yaml.txt)
* [packages.yaml.txt](https://github.com/spack/spack/files/5599942/packages.yaml.txt)
* [config.yaml.txt](https://github.com/spack/spack/files/5599944/config.yaml.txt)


Another one for the list:

```
spack spec sirius@develop arch=linux-centos7-zen amdgpu_target=gfx906 +rocm ^spfft +rocm amdgpu_target=gfx906 ^openmpi +pmi schedulers=slurm ^rocfft@3.9.0 amdgpu_target=gfx906 ^openblas threads=openmp ^rocblas@3.9.0 amdgpu_target=gfx906
```

turns `opengl` off in mesa18, causing the build to fail.

Curiously when adding `conflicts('~opengl ~opengles`) it turns off `~opengl` and turns on `+opengles`, basically undoing the default.
@chuckatkins I checked your spec, can reproduce and have a diagnosis. The issue comes from having this minimization:
```
#minimize{
    Weight@8,Provider: provider_weight(Provider, Weight)
}.
```
at a higher priority than selecting default variants. What happens is that for some reason in your spec the default provider cannot be selected (for me it's `openmpi@3.1.6` that can't be used) which would give a value of 0 to the minimization above. Other mpi providers have increasing positive weights. Now: not having an mpi provider at all would also count as a 0 for the target above and since this is possible by flipping all the variants the solver is happily doing that as a second option.
@chuckatkins I think #20182 solves your issue. Let me know if anything remains to be fixed.

@v-dobrev Can you check if #20182 solves the problem you have with `mfem`? If not I'll have a look at your specific case tomorrow on Marconi. Thanks!
Unfortunately, #20182 does not fix the issue with `mfem`.

Resolving patch:
diff --git a/var/spack/repos/builtin.mock/packages/vdefault-or-external-root/package.py b/var/spack/repos/builtin.mock/packages/vdefault-or-external-root/package.py
new file mode 100644
--- /dev/null
+++ b/var/spack/repos/builtin.mock/packages/vdefault-or-external-root/package.py
@@ -0,0 +1,15 @@
+# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+class VdefaultOrExternalRoot(Package):
+    """Test that we don't prefer adding an external to using
+    a default variant value.
+    """
+
+    homepage = 'https://www.example.org'
+    url = 'https://example.org/files/v3.4/cmake-3.4.3.tar.gz'
+
+    version('1.0', '4cb3ff35b2472aae70f542116d616e63')
+
+    depends_on('vdefault-or-external')
diff --git a/var/spack/repos/builtin.mock/packages/vdefault-or-external/package.py b/var/spack/repos/builtin.mock/packages/vdefault-or-external/package.py
new file mode 100644
--- /dev/null
+++ b/var/spack/repos/builtin.mock/packages/vdefault-or-external/package.py
@@ -0,0 +1,17 @@
+# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+class VdefaultOrExternal(Package):
+    """Test that we don't prefer adding an external to using
+    a default variant value.
+    """
+
+    homepage = 'https://www.example.org'
+    url = 'https://example.org/files/v3.4/cmake-3.4.3.tar.gz'
+
+    version('1.0', '4cb3ff35b2472aae70f542116d616e63')
+
+    variant('external', default=False, description='nope')
+
+    depends_on('externaltool', when='+external')


Test patch:
diff --git a/lib/spack/spack/test/concretize_preferences.py b/lib/spack/spack/test/concretize_preferences.py
--- a/lib/spack/spack/test/concretize_preferences.py
+++ b/lib/spack/spack/test/concretize_preferences.py
@@ -371,3 +371,13 @@ def test_config_perms_fail_write_gt_read(self, configure_permissions):
         spec = Spec('callpath')
         with pytest.raises(ConfigError):
             spack.package_prefs.get_package_permissions(spec)
+
+    @pytest.mark.regression('20040')
+    def test_variant_not_flipped_to_pull_externals(self):
+        """Test that a package doesn't prefer pulling in an
+        external to using the default value of a variant.
+        """
+        s = Spec('vdefault-or-external-root').concretized()
+
+        assert '~external' in s['vdefault-or-external']
+        assert 'externaltool' not in s