Repository: spack/spack

Issue:
Using unify:false causes variants to not propagate
### Steps to reproduce

```
> spack env create test
> spack env activate -p test
> cat spack/var/spack/environments/test/spack.yaml 
spack:
  specs:
    - nco ^netcdf-c ~~mpi
> spack spec
```
These above commands should concretize the nco spec with ~mpi in netcdf and propagate ~mpi to all its dependencies, such as HDF5.  But using ~~ does not propagate the variants to the dependencies when concretizing the spec via spack.yaml file.

Here is the output of the command above, 

```
> spack spec -INlt
==> Starting concretization
==> Environment concretized in 18.16 seconds.
Input spec
--------------------------------
 -   [    ]  .nco
 -   [    ]      ^.netcdf-c~~mpi

Concretized
--------------------------------
 -   7kxljfv  [    ]  builtin.nco@5.0.6%gcc@12.2.0~doc build_system=autotools arch=linux-sles15-haswell
 -   kdcb7av  [bl  ]      ^builtin.antlr@2.7.7%gcc@12.2.0+cxx~java~pic~python build_system=autotools patches=33897ad arch=linux-sles15-haswell
 -   hk3brcx  [b   ]      ^builtin.bison@3.8.2%gcc@12.2.0 build_system=autotools arch=linux-sles15-haswell
[+]  5ual2mi  [b   ]          ^builtin.diffutils@3.8%gcc@12.2.0 build_system=autotools arch=linux-sles15-haswell
[+]  ioqjjqf  [bl  ]              ^builtin.libiconv@1.17%gcc@12.2.0 build_system=autotools libs=shared,static arch=linux-sles15-haswell
[+]  kejdamc  [b r ]          ^builtin.m4@1.4.19%gcc@12.2.0+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7 arch=linux-sles15-haswell
[+]  e5jaqts  [bl  ]              ^builtin.libsigsegv@2.13%gcc@12.2.0 build_system=autotools arch=linux-sles15-haswell
[+]  23ubnxa  [b   ]          ^builtin.perl@5.36.0%gcc@12.2.0+cpanm+open+shared+threads build_system=generic arch=linux-sles15-haswell
[+]  dxtnxns  [bl  ]              ^builtin.berkeley-db@18.1.40%gcc@12.2.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-sles15-haswell
[+]  y3j4wvq  [bl  ]              ^builtin.bzip2@1.0.8%gcc@12.2.0~debug~pic+shared build_system=generic arch=linux-sles15-haswell
[+]  q5lz6oc  [bl  ]              ^builtin.gdbm@1.23%gcc@12.2.0 build_system=autotools arch=linux-sles15-haswell
[+]  2la6hq6  [bl  ]                  ^builtin.readline@8.2%gcc@12.2.0 build_system=autotools patches=bbf97f1 arch=linux-sles15-haswell
 -   ulkawal  [b   ]      ^builtin.flex@2.6.3%gcc@12.2.0+lex~nls build_system=autotools arch=linux-sles15-haswell
 -   f4mhkf2  [b   ]          ^builtin.findutils@4.9.0%gcc@12.2.0 build_system=autotools patches=440b954 arch=linux-sles15-haswell
 -   rsnhmct  [bl  ]      ^builtin.gsl@2.7.1%gcc@12.2.0~external-cblas build_system=autotools arch=linux-sles15-haswell
 -   w4stxdo  [bl  ]      ^builtin.netcdf-c@4.9.0%gcc@12.2.0~dap~fsync~hdf4~jna~mpi+optimize~parallel-netcdf+pic+shared+zstd build_system=autotools arch=linux-sles15-haswell
 -   apq47ms  [bl  ]          ^builtin.hdf5@1.12.2%gcc@12.2.0+cxx+fortran+hl~ipo~java+mpi+shared+szip+threadsafe+tools api=default build_system=cmake build_type=RelWithDebInfo arch=linu
```

Whereas, concretizing from the command line, works just fine. 

```
spack spec -INlt nco ^netcdf-c ~~mpi
Input spec
--------------------------------
 -   [    ]  .nco
 -   [    ]      ^.netcdf-c~~mpi

Concretized
--------------------------------
 -   c5wp4x6  [    ]  builtin.nco@5.0.6%gcc@12.2.0~doc build_system=autotools arch=linux-sles15-haswell
 -   kdcb7av  [bl  ]      ^builtin.antlr@2.7.7%gcc@12.2.0+cxx~java~pic~python build_system=autotools patches=33897ad arch=linux-sles15-haswell
 -   hk3brcx  [b   ]      ^builtin.bison@3.8.2%gcc@12.2.0 build_system=autotools arch=linux-sles15-haswell
[+]  5ual2mi  [b   ]          ^builtin.diffutils@3.8%gcc@12.2.0 build_system=autotools arch=linux-sles15-haswell
[+]  ioqjjqf  [bl  ]              ^builtin.libiconv@1.17%gcc@12.2.0 build_system=autotools libs=shared,static arch=linux-sles15-haswell
[+]  kejdamc  [b r ]          ^builtin.m4@1.4.19%gcc@12.2.0+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7 arch=linux-sles15-haswell
[+]  e5jaqts  [bl  ]              ^builtin.libsigsegv@2.13%gcc@12.2.0 build_system=autotools arch=linux-sles15-haswell
[+]  23ubnxa  [b   ]          ^builtin.perl@5.36.0%gcc@12.2.0+cpanm+open+shared+threads build_system=generic arch=linux-sles15-haswell
[+]  dxtnxns  [bl  ]              ^builtin.berkeley-db@18.1.40%gcc@12.2.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-sles15-haswell
[+]  y3j4wvq  [bl  ]              ^builtin.bzip2@1.0.8%gcc@12.2.0~debug~pic+shared build_system=generic arch=linux-sles15-haswell
[+]  q5lz6oc  [bl  ]              ^builtin.gdbm@1.23%gcc@12.2.0 build_system=autotools arch=linux-sles15-haswell
[+]  2la6hq6  [bl  ]                  ^builtin.readline@8.2%gcc@12.2.0 build_system=autotools patches=bbf97f1 arch=linux-sles15-haswell
 -   ulkawal  [b   ]      ^builtin.flex@2.6.3%gcc@12.2.0+lex~nls build_system=autotools arch=linux-sles15-haswell
 -   f4mhkf2  [b   ]          ^builtin.findutils@4.9.0%gcc@12.2.0 build_system=autotools patches=440b954 arch=linux-sles15-haswell
 -   rsnhmct  [bl  ]      ^builtin.gsl@2.7.1%gcc@12.2.0~external-cblas build_system=autotools arch=linux-sles15-haswell
 -   kwavdbu  [bl  ]      ^builtin.netcdf-c@4.9.0%gcc@12.2.0~dap~fsync~hdf4~jna~mpi+optimize~parallel-netcdf+pic+shared+zstd build_system=autotools arch=linux-sles15-haswell
 -   vntpwpy  [bl  ]          ^builtin.hdf5@1.12.2%gcc@12.2.0+cxx+fortran+hl~ipo~java~mpi+shared+szip+threadsafe+tools api=default build_system=cmake build_type=RelWithDebInfo arch=linux-sles15-haswell
 ```

Why this could be? Is it a bug? 

### Error message

_No response_

### Information on your system

* **Spack:** 0.20.0.dev0 (cd2d6a6397a34244ee9af3f102be801cd0f14d84)
* **Python:** 3.6.12
* **Platform:** linux-sles15-haswell
* **Concretizer:** clingo


### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:
I see something similar, specifically, when the variant propagation starts one or more levels down, eg:

```
> Concretized alquimia%gcc ^petsc~~cuda
 -   wksd3pe  alquimia@1.0.10%gcc@11.2.0~ipo+shared build_system=cmake build_type=RelWithDebInfo arch=linux-sles15-zen3
[snip]
 -   ksdgl63      ^pflotran@4.0.1%gcc@11.2.0 build_system=autotools arch=linux-sles15-zen3
 -   3cazor4          ^petsc@3.18.1%gcc@11.2.0~X~batch~cgns~complex~cuda~debug+double~exodusii~fftw+fortran~giflib+hdf5~hpddm~hwloc+hypre~int64~jpeg~knl~kokkos~libpng~libyaml~memkind+metis~mkl-pardiso~mmg~moab~mpfr+mpi~mumps~openmp~p4est~parmmg~ptscotch~random123~rocm~saws~scalapack+shared~strumpack~suite-sparse+superlu-dist~tetgen~trilinos~valgrind build_system=generic clanguage=C arch=linux-sles15-zen3
[snip]
[+]  ggmj3t7              ^hypre@2.26.0%gcc@11.2.0~complex+cuda~debug+fortran~gptune~int64~internal-superlu~mixedint+mpi~openmp~rocm+shared~superlu-dist~umpire~unified-memory build_system=autotools cuda_arch=80 arch=linux-sles15-zen3
[^]  xgfnclg                  ^cuda@11.7%gcc@11.2.0~allow-unsupported-compilers~dev build_system=generic arch=linux-sles15-zen3
[snip]
```

(I've also seen it with netcdf family propagating mpi, and was able to mitigate it by adding an `mpi` variant at the top level .. but that is a bit inelegant and labor-intensive).

In this case, I think `unify:when_possible` is a factor, because I can make an environment with **only** `alquimia%gcc ^petsc~~cuda` and the `~cuda` propagates to hypre, but in my larger environment it tries to build on `hypre+cuda` (even though there is a `hypre~cuda` already installed there)


@becker33 @RikkiButler20 fyi
FWIW. Those of you who have it probably have `config:concretizer:unify` set to `False` somewhere _other than_ your `spack.yaml` file (e.g., `concretizer.yaml` perhaps).

```
$ spack config blame concretizer
```
This actually uncovers another bug; namely that the environment is not using the `concretizer:unify` option when set in the `spack.yaml` file.

Resolving patch:
diff --git a/lib/spack/spack/environment/environment.py b/lib/spack/spack/environment/environment.py
--- a/lib/spack/spack/environment/environment.py
+++ b/lib/spack/spack/environment/environment.py
@@ -1484,7 +1484,7 @@ def _concretize_separately(self, tests=False):
         for uspec, uspec_constraints in zip(self.user_specs, self.user_specs.specs_as_constraints):
             if uspec not in old_concretized_user_specs:
                 root_specs.append(uspec)
-                args.append((i, uspec_constraints, tests))
+                args.append((i, [str(x) for x in uspec_constraints], tests))
                 i += 1
 
         # Ensure we don't try to bootstrap clingo in parallel
@@ -2400,6 +2400,7 @@ def _concretize_from_constraints(spec_constraints, tests=False):
 
 def _concretize_task(packed_arguments) -> Tuple[int, Spec, float]:
     index, spec_constraints, tests = packed_arguments
+    spec_constraints = [Spec(x) for x in spec_constraints]
     with tty.SuppressOutput(msg_enabled=False):
         start = time.time()
         spec = _concretize_from_constraints(spec_constraints, tests)
diff --git a/var/spack/repos/builtin.mock/packages/client-not-foo/package.py b/var/spack/repos/builtin.mock/packages/client-not-foo/package.py
new file mode 100644
--- /dev/null
+++ b/var/spack/repos/builtin.mock/packages/client-not-foo/package.py
@@ -0,0 +1,17 @@
+# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+from spack.package import *
+
+
+class ClientNotFoo(Package):
+    """This package has a variant "foo", which is False by default."""
+
+    homepage = "http://www.example.com"
+    url = "http://www.example.com/c-1.0.tar.gz"
+
+    version("1.0", md5="0123456789abcdef0123456789abcdef")
+
+    variant("foo", default=False, description="")
diff --git a/var/spack/repos/builtin.mock/packages/parent-foo/package.py b/var/spack/repos/builtin.mock/packages/parent-foo/package.py
new file mode 100644
--- /dev/null
+++ b/var/spack/repos/builtin.mock/packages/parent-foo/package.py
@@ -0,0 +1,21 @@
+# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+from spack.package import *
+
+
+class ParentFoo(Package):
+    """This package has a variant "foo", which is True by default, and depends on another
+    package which has the same variant defaulting to False.
+    """
+
+    homepage = "http://www.example.com"
+    url = "http://www.example.com/c-1.0.tar.gz"
+
+    version("1.0", md5="0123456789abcdef0123456789abcdef")
+
+    variant("foo", default=True, description="")
+
+    depends_on("client-not-foo")


Test patch:
diff --git a/lib/spack/spack/test/env.py b/lib/spack/spack/test/env.py
--- a/lib/spack/spack/test/env.py
+++ b/lib/spack/spack/test/env.py
@@ -690,3 +690,29 @@ def test_removing_spec_from_manifest_with_exact_duplicates(
     assert "zlib" in manifest.read_text()
     with ev.Environment(tmp_path) as env:
         assert len(env.user_specs) == 1
+
+
+@pytest.mark.regression("35298")
+@pytest.mark.only_clingo("Propagation not supported in the original concretizer")
+def test_variant_propagation_with_unify_false(tmp_path, mock_packages):
+    """Spack distributes concretizations to different processes, when unify:false is selected and
+    the number of roots is 2 or more. When that happens, the specs to be concretized need to be
+    properly reconstructed on the worker process, if variant propagation was requested.
+    """
+    manifest = tmp_path / "spack.yaml"
+    manifest.write_text(
+        """
+    spack:
+      specs:
+      - parent-foo ++foo
+      - c
+      concretizer:
+        unify: false
+    """
+    )
+    with ev.Environment(tmp_path) as env:
+        env.concretize()
+
+    root = env.matching_spec("parent-foo")
+    for node in root.traverse():
+        assert node.satisfies("+foo")
diff --git a/lib/spack/spack/test/spec_syntax.py b/lib/spack/spack/test/spec_syntax.py
--- a/lib/spack/spack/test/spec_syntax.py
+++ b/lib/spack/spack/test/spec_syntax.py
@@ -525,6 +525,31 @@ def _specfile_for(spec_str, filename):
             ],
             "zlib@git.foo/bar",
         ),
+        # Variant propagation
+        (
+            "zlib ++foo",
+            [
+                Token(TokenType.UNQUALIFIED_PACKAGE_NAME, "zlib"),
+                Token(TokenType.PROPAGATED_BOOL_VARIANT, "++foo"),
+            ],
+            "zlib++foo",
+        ),
+        (
+            "zlib ~~foo",
+            [
+                Token(TokenType.UNQUALIFIED_PACKAGE_NAME, "zlib"),
+                Token(TokenType.PROPAGATED_BOOL_VARIANT, "~~foo"),
+            ],
+            "zlib~~foo",
+        ),
+        (
+            "zlib foo==bar",
+            [
+                Token(TokenType.UNQUALIFIED_PACKAGE_NAME, "zlib"),
+                Token(TokenType.PROPAGATED_KEY_VALUE_PAIR, "foo==bar"),
+            ],
+            "zlib foo==bar",
+        ),
     ],
 )
 def test_parse_single_spec(spec_str, tokens, expected_roundtrip):