Repository: spack/spack

Issue:
Installation issue: MFEM post-install fails with XL + gcc-toolchain flag
### Steps to reproduce the issue

```console
$ spack/bin/spack spec -I mfem@4.4.0%xl@16.1.1.1
Input spec
--------------------------------
mfem@4.4.0%xl@16.1.1.1

Concretized
--------------------------------
elnm3d7  mfem@4.4.0%xl@16.1.1.1 cflags="--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1" cxxflags="--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1" fflags="--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1" ~amgx~conduit~cuda~debug~examples~exceptions~fms~ginkgo~gnutls~gslib~hiop~lapack~libceed~libunwind+metis~miniapps~mpfr+mpi~netcdf~occa~openmp~petsc~pumi~raja~rocm~shared~slepc+static~strumpack~suite-sparse~sundials~superlu-dist~threadsafe~umpire+zlib build_system=generic timer=auto arch=linux-rhel7-power9le
tonjvvm      ^hypre@2.24.0%xl@16.1.1.1 cflags="--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1" cxxflags="--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1" fflags="--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1" ~complex~cuda~debug+fortran~gptune~int64~internal-superlu~mixedint+mpi~openmp~rocm+shared~superlu-dist~umpire~unified-memory build_system=autotools arch=linux-rhel7-power9le
gmuh4my          ^gnuconfig@2022-09-17%xl@16.1.1.1 cflags="--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1" cxxflags="--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1" fflags="--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1"  build_system=generic arch=linux-rhel7-power9le
vtcdnfk          ^netlib-lapack@3.9.0%xl@16.1.1.1 cflags="--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1" cxxflags="--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1" fflags="--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1" ~external-blas~ipo+lapacke+shared~xblas build_system=cmake build_type=RelWithDebInfo arch=linux-rhel7-power9le
lmrjgod      ^metis@5.1.0%xl@16.1.1.1 cflags="--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1" cxxflags="--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1" fflags="--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1" ~gdb~int64~ipo~real64+shared build_system=cmake build_type=RelWithDebInfo patches=4991da9,93a7903 arch=linux-rhel7-power9le
gzlg3cw          ^cmake@3.14.5%xl@16.1.1.1 cflags="--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1" cxxflags="--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1" fflags="--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1" ~doc+ncurses+ownlibs~qt build_system=generic build_type=Release patches=1c54004 arch=linux-rhel7-power9le
o47ypcv      ^spectrum-mpi@release%xl@16.1.1.1 cflags="--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1" cxxflags="--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1" fflags="--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1"  build_system=bundle arch=linux-rhel7-power9le
rrmkl3x      ^zlib@1.2.13%xl@16.1.1.1 cflags="--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1" cxxflags="--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1" fflags="--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1" +optimize+pic+shared build_system=makefile arch=linux-rhel7-power9le

$ spack/bin/spack -d -v install mfem@4.4.0%xl@16.1.1.1
...
==> [2022-11-23-12:31:47.734199] mfem: Building mfem-4.4.0-elnm3d7dkxjiugbbbnec3odrt4gngb7s [Package]
...
==> [2022-11-23-12:31:47.761096] mfem: Executing phase: 'configure'
==> [2022-11-23-12:31:48.223300] mfem: Executing phase: 'build'
==> [2022-11-23-12:34:32.877899] mfem: Executing phase: 'install'
```


### Error message

<details><summary>Error message</summary><pre>
==> [2022-11-23-12:34:34.629302] Error: AttributeError: 'str' object has no attribute 'propagate'

The 'mfem' package cannot find an attribute while trying to build from sources. This might be due to a change in Spack's package format to support multiple build-systems for a single package. You can fix this by updating the build recipe, and you can also report the issue as a bug. More information at https://spack.readthedocs.io/en/latest/packaging_guide.html#installation-procedure
See build log for details:
  /var/tmp/yang39/spack-stage/spack-stage-mfem-4.4.0-elnm3d7dkxjiugbbbnec3odrt4gngb7s/spack-build-out.txt

Traceback (most recent call last):
  File "/usr/WS2/yang39/spack/lib/spack/spack/build_environment.py", line 1068, in _setup_pkg_and_run
    return_value = function(pkg, kwargs)
  File "/usr/WS2/yang39/spack/lib/spack/spack/installer.py", line 2081, in build_process
    return installer.run()
  File "/usr/WS2/yang39/spack/lib/spack/spack/installer.py", line 1953, in run
    spack.hooks.post_install(self.pkg.spec)
  File "/usr/WS2/yang39/spack/lib/spack/spack/hooks/__init__.py", line 74, in __call__
    hook(*args, **kwargs)
  File "/usr/WS2/yang39/spack/lib/spack/spack/hooks/module_file_generation.py", line 48, in post_install
    _for_each_enabled(spec, "write")
  File "/usr/WS2/yang39/spack/lib/spack/spack/hooks/module_file_generation.py", line 32, in _for_each_enabled
    getattr(generator, method_name)()
  File "/usr/WS2/yang39/spack/lib/spack/spack/modules/common.py", line 847, in write
    if self.conf.excluded:
  File "/usr/WS2/yang39/spack/lib/spack/spack/modules/common.py", line 519, in excluded
    installed_implicitly = not spec._installed_explicitly()
  File "/usr/WS2/yang39/spack/lib/spack/spack/spec.py", line 4615, in _installed_explicitly
    record = spack.store.db.get_record(self)
  File "/usr/WS2/yang39/spack/lib/spack/spack/database.py", line 122, in converter
    return function(self, spec_like, *args, **kwargs)
  File "/usr/WS2/yang39/spack/lib/spack/spack/database.py", line 1212, in get_record
    key = self._get_matching_spec_key(spec, **kwargs)
  File "/usr/WS2/yang39/spack/lib/spack/spack/database.py", line 1207, in _get_matching_spec_key
    raise KeyError("No such spec in database! %s" % spec)
  File "/usr/WS2/yang39/spack/lib/spack/spack/spec.py", line 4597, in __str__
    spec_str = " ^".join(d.format() for d in sorted_nodes)
  File "/usr/WS2/yang39/spack/lib/spack/spack/spec.py", line 4597, in <genexpr>
    spec_str = " ^".join(d.format() for d in sorted_nodes)
  File "/usr/WS2/yang39/spack/lib/spack/spack/spec.py", line 4328, in format
    write_attribute(self, attribute, color)
  File "/usr/WS2/yang39/spack/lib/spack/spack/spec.py", line 4314, in write_attribute
    write(sig + morph(spec, str(current)), col)
  File "/usr/WS2/yang39/spack/lib/spack/spack/spec.py", line 882, in __str__
    + cond_symbol
  File "/usr/WS2/yang39/spack/lib/spack/spack/spec.py", line 880, in <genexpr>
    for key in sorted_keys
  File "/usr/WS2/yang39/spack/lib/spack/spack/spec.py", line 877, in <listcomp>
    + ('=="' if True in [f.propagate for f in self[key]] else '="')
AttributeError: 'str' object has no attribute 'propagate'
</pre></details>


### Information on your system

* **Spack:** 0.20.0.dev0 (bf1b846f26ec249c947f0cd258afd8f248f64700)
* **Python:** 3.7.2
* **Platform:** linux-rhel7-power9le
* **Concretizer:** clingo

`compilers.yaml`:
```yaml
- compiler:
     environment: {}
     extra_rpaths: []
     flags:
       cflags: --gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1
       cxxflags: --gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1
       fflags: --gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1
     modules: []
     operating_system: rhel7
     paths:
       cc: /usr/tce/packages/xl/xl-2022.08.19/bin/xlc
       cxx: /usr/tce/packages/xl/xl-2022.08.19/bin/xlC
       fc: /usr/tce/packages/xl/xl-2022.08.19/bin/xlf2003
       f77: /usr/tce/packages/xl/xl-2022.08.19/bin/xlf
     spec: xl@16.1.1.1
     target: ppc64le
```

### Additional information

[spack-build-env.txt](https://github.com/spack/spack/files/10079112/spack-build-env.txt)
[spack-build-out.txt](https://github.com/spack/spack/files/10079113/spack-build-out.txt)

CC: @acfisher @goxberry @markcmiller86 @tzanio @v-dobrev 

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have run `spack maintainers <name-of-the-package>` and **@mentioned** any maintainers
- [X] I have uploaded the build log and environment files
- [X] I have searched the issues of this repo and believe this is not a duplicate


Discussion:
Judging by the `spack-build-out.txt` log file, the installation of MFEM is completed without any issues.

The error must be happening after that, maybe during the module file generation -- I see `module_file_generation.py` being executed in the python stack trace. On the other hand the `.../mfem/package.py` is not visible in the stack trace, so this looks like a generic Spack issue.

I see the message: "This might be due to a change in Spack's package format to support multiple build-systems for a single package." but I have no idea if recent Spack changes have rendered `.../mfem/package.py` incorrect. I'm not sure what exactly is no longer backward compatible.

Maybe @tgamblin, or @becker33 can help figure out what exactly the issue is.
@white238 pointed me to @publixsubfan's one-line workaround for this:  
https://github.com/LLNL/axom/commit/a7396abe6e1c4180c4806fc6dbb3add228f2e5c5

The latter references this issue, but hopefully this comment will make the fix more obvious.
For reference, the reason that fix is relevant is because later on the mfem package assigns to the `cxxflags` variable, and assigning strings into the flag map will no longer work. The flags are `CompilerFlag` objects, which inherit from `str` but carry additional information about compiler propagation.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/mfem/package.py b/var/spack/repos/builtin/packages/mfem/package.py
--- a/var/spack/repos/builtin/packages/mfem/package.py
+++ b/var/spack/repos/builtin/packages/mfem/package.py
@@ -601,7 +601,7 @@ def find_optional_library(name, prefix):
             else:
                 cxxstd_flag = getattr(self.compiler, "cxx" + cxxstd + "_flag")
 
-        cxxflags = spec.compiler_flags["cxxflags"]
+        cxxflags = spec.compiler_flags["cxxflags"].copy()
 
         if cxxflags:
             # Add opt/debug flags if they are not present in global cxx flags
@@ -1116,7 +1116,7 @@ def patch(self):
             "miniapps/gslib/findpts.cpp",
             "miniapps/gslib/pfindpts.cpp",
         ]
-        bom = "\xef\xbb\xbf" if sys.version_info < (3,) else u"\ufeff"
+        bom = "\xef\xbb\xbf" if sys.version_info < (3,) else "\ufeff"
         for f in files_with_bom:
             filter_file(bom, "", f)
 


Test patch: