Repository: spack/spack

Issue:
Installation issue: intel-oneapi-mkl on macOS is unable to provide BLAS/LAPACK libraries
### Steps to reproduce the issue

```console
$ spack spec -I netlib-scalapack
Input spec
--------------------------------
 -   netlib-scalapack

Concretized
--------------------------------
[+]  netlib-scalapack@2.2.0%gcc@12.2.0~ipo~pic+shared build_system=cmake build_type=Release patches=072b006,1c9ce5f,244a9aa arch=darwin-ventura-skylake
[+]      ^cmake@3.25.2%gcc@12.2.0~doc+ncurses+ownlibs~qt build_system=generic build_type=Release arch=darwin-ventura-skylake
[+]      ^intel-oneapi-mkl@2023.0.0%gcc@12.2.0~cluster+envmods~ilp64+shared build_system=generic arch=darwin-ventura-skylake
[+]      ^openmpi@4.1.4%gcc@12.2.0~atomics~cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~java~legacylaunchers~lustre~memchecker~orterunprefix+pmi+romio+rsh~singularity~static+vt~wrapper-rpath build_system=autotools fabrics=none schedulers=slurm arch=darwin-ventura-skylake

```

with the following in `packages.yaml`:

```
  intel-oneapi-mkl:
    externals:
    - spec: intel-oneapi-mkl@2023.0.0
      prefix: /opt/intel/oneapi
  intel-oneapi-tbb:
    externals:
    - spec: intel-oneapi-tbb@2021.8.0
      prefix: /opt/intel/oneapi
```

and intel-oneapi-mkl installed with the OneAPI base package installer.

### Error message

<pre>
==> netlib-scalapack: Executing phase: 'cmake'
==> [2023-02-18-16:27:26.677957] '/usr/local/bin/cmake' '-G' 'Unix Makefiles' '-DCMAKE_INSTALL_PREFIX:STRING=/Users/tiziano/Projects/spack/opt/spack/darwin-ventura-skylake/gcc-12.2.0/netlib-scalapack-2.2.0-duvvf2vs6c4vt4xu3ho4wl4dulmca7o2' '-DCMAKE_BUILD_TYPE:STRING=Release' '-DBUILD_TESTING
:BOOL=OFF' '-DCMAKE_INTERPROCEDURAL_OPTIMIZATION:BOOL=OFF' '-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON' '-DCMAKE_FIND_FRAMEWORK:STRING=LAST' '-DCMAKE_FIND_APPBUNDLE:STRING=LAST' '-DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON' '-DCMAKE_INSTALL_RPATH:STRING=/Users/tiziano/Projects/spack/opt/spack/darwin-v
entura-skylake/gcc-12.2.0/netlib-scalapack-2.2.0-duvvf2vs6c4vt4xu3ho4wl4dulmca7o2/lib;/Users/tiziano/Projects/spack/opt/spack/darwin-ventura-skylake/gcc-12.2.0/netlib-scalapack-2.2.0-duvvf2vs6c4vt4xu3ho4wl4dulmca7o2/lib64' '-DCMAKE_PREFIX_PATH:STRING=/opt/intel/oneapi/mkl' '-DBUILD_SHARED_LI
BS:BOOL=ON' '-DBUILD_STATIC_LIBS:BOOL=OFF' '-DLAPACK_FOUND=true' '-DLAPACK_INCLUDE_DIRS=/opt/intel/oneapi/mkl/include' '-DLAPACK_LIBRARIES=' '-DBLAS_LIBRARIES=' '-DCMAKE_C_FLAGS:STRING=' '/var/folders/q_/hmjtsjld2d7csxq_lt8ttrtc0000gn/T/tiziano/spack-stage/spack-stage-netlib-scalapack-2.2.0-
duvvf2vs6c4vt4xu3ho4wl4dulmca7o2/spack-src
[...]
Undefined symbols for architecture x86_64:
  "_caxpy_", referenced from:
      _claghe_ in claghe.f.o
      _clagsy_ in clagsy.f.o
      _cmmadd_ in cmmadd.f.o
      _cmmtadd_ in cmmtadd.f.o
      _cmmdda_ in cmmdda.f.o
      _cmmddat_ in cmmddat.f.o
      _PB_Cctypeset in PB_Cctypeset.c.o
      ...
[...]
</pre>

### Information on your system

* **Spack:** 0.20.0.dev0 (c42a4ec1ec3ea8b339752c9257d91f0e07c254b1)
* **Python:** 3.11.2
* **Platform:** darwin-ventura-skylake
* **Concretizer:** clingo

### Additional information

@rscohn2 the problem I believe is that on macOS there is no subdir `intel64`:

```
❯ ls /opt/intel/oneapi/mkl/2023.0.0
 benchmarks   bin   env   examples   include   interfaces   lib   licensing   tools
```

and this change fixed it:

```
diff --git a/var/spack/repos/builtin/packages/intel-oneapi-mkl/package.py b/var/spack/repos/builtin/packages/intel-oneapi-mkl/package.py
index 8be451f656..bc149a5e54 100644
--- a/var/spack/repos/builtin/packages/intel-oneapi-mkl/package.py
+++ b/var/spack/repos/builtin/packages/intel-oneapi-mkl/package.py
@@ -153,7 +153,7 @@ def _find_mkl_libs(self, shared):
         if "+cluster" in self.spec:
             libs.append(self._xlp64_lib("libmkl_blacs_intelmpi"))

-        return find_libraries(libs, self.component_prefix.lib.intel64, shared=shared)
+        return find_libraries(libs, self.component_prefix.lib, shared=shared)

     def _xlp64_lib(self, lib):
         return lib + ("_ilp64" if "+ilp64" in self.spec else "_lp64")
```

[spack-build-env.txt](https://github.com/spack/spack/files/10777070/spack-build-env.txt)
[spack-build-out.txt](https://github.com/spack/spack/files/10777071/spack-build-out.txt)


### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have run `spack maintainers <name-of-the-package>` and **@mentioned** any maintainers
- [X] I have uploaded the build log and environment files
- [X] I have searched the issues of this repo and believe this is not a duplicate


Discussion:
Hello,
I have the same problem.
Will you create a PR for this issue?
@William-Mou not with the exact change outlined above since this might break on other systems. While `find_libraries` is recursive and should find libraries in cases where they are inside `lib/intel64/`, it may discover other libraries possibly available in other subfolders of `lib/`.
I hope @rscohn2 can comment on whether we have to guard against this or whether `lib/intel64/` is anyway the only folder where libs are located on all platforms other than macOS.
Prior to 2023.0, there were ia32 and intel64 directories for linux. You need: `if sys.platform == "darwin"` There is no support for installing packages on windows or macOS. I did not realize you could use an externally installed mkl on macos.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/cp2k/package.py b/var/spack/repos/builtin/packages/cp2k/package.py
--- a/var/spack/repos/builtin/packages/cp2k/package.py
+++ b/var/spack/repos/builtin/packages/cp2k/package.py
@@ -274,31 +274,14 @@ def archive_files(self):
     def edit(self, spec, prefix):
         pkgconf = which("pkg-config")
 
-        if "^fftw" in spec:
-            fftw = spec["fftw:openmp" if "+openmp" in spec else "fftw"]
-            fftw_header_dir = fftw.headers.directories[0]
-        elif "^amdfftw" in spec:
-            fftw = spec["amdfftw:openmp" if "+openmp" in spec else "amdfftw"]
-            fftw_header_dir = fftw.headers.directories[0]
-        elif "^armpl-gcc" in spec:
-            fftw = spec["armpl-gcc:openmp" if "+openmp" in spec else "armpl-gcc"]
-            fftw_header_dir = fftw.headers.directories[0]
-        elif "^intel-mkl" in spec:
-            fftw = spec["intel-mkl"]
-            fftw_header_dir = fftw.headers.directories[0] + "/fftw"
-        elif "^intel-oneapi-mkl" in spec:
-            fftw = spec["intel-oneapi-mkl"]
-            fftw_header_dir = fftw.headers.directories[0] + "/fftw"
-        elif "^intel-parallel-studio+mkl" in spec:
-            fftw = spec["intel-parallel-studio"]
-            fftw_header_dir = "<NOTFOUND>"
-            for incdir in [join_path(f, "fftw") for f in fftw.headers.directories]:
-                if os.path.exists(incdir):
-                    fftw_header_dir = incdir
-                    break
-        elif "^cray-fftw" in spec:
-            fftw = spec["cray-fftw"]
-            fftw_header_dir = fftw.headers.directories[0]
+        fftw = spec["fftw-api"]
+        fftw_header_dir = fftw.headers.directories[0]
+
+        # some providers (mainly Intel) keep the fftw headers in a subdirectory, find it
+        for incdir in [join_path(f, "fftw") for f in fftw.headers.directories]:
+            if os.path.exists(incdir):
+                fftw_header_dir = incdir
+                break
 
         optimization_flags = {
             "gcc": ["-O2", "-funroll-loops", "-ftree-vectorize"],
@@ -401,11 +384,12 @@ def edit(self, spec, prefix):
         ldflags.append((lapack + blas).search_flags)
         libs.extend([str(x) for x in (fftw.libs, lapack, blas)])
 
-        if any(
-            p in spec for p in ("^intel-mkl", "^intel-parallel-studio+mkl", "^intel-oneapi-mkl")
-        ):
+        if self.spec.satisfies("platform=darwin"):
+            cppflags.extend(["-D__NO_STATM_ACCESS"])
+
+        if spec["blas"].name in ("intel-mkl", "intel-parallel-studio", "intel-oneapi-mkl"):
             cppflags += ["-D__MKL"]
-        elif "^accelerate" in spec:
+        elif spec["blas"].name == "accelerate":
             cppflags += ["-D__ACCELERATE"]
 
         if "+cosma" in spec:
@@ -418,15 +402,15 @@ def edit(self, spec, prefix):
         if "+mpi" in spec:
             cppflags.extend(["-D__parallel", "-D__SCALAPACK"])
 
-            if "^intel-oneapi-mpi" in spec:
+            if spec["mpi"].name == "intel-oneapi-mpi":
                 mpi = [join_path(spec["intel-oneapi-mpi"].libs.directories[0], "libmpi.so")]
             else:
                 mpi = spec["mpi:cxx"].libs
 
             # while intel-mkl has a mpi variant and adds the scalapack
             # libs to its libs, intel-oneapi-mkl does not.
-            if "^intel-oneapi-mkl" in spec:
-                mpi_impl = "openmpi" if "^openmpi" in spec else "intelmpi"
+            if spec["scalapack"].name == "intel-oneapi-mkl":
+                mpi_impl = "openmpi" if spec["mpi"] == "openmpi" else "intelmpi"
                 scalapack = [
                     join_path(
                         spec["intel-oneapi-mkl"].libs.directories[0], "libmkl_scalapack_lp64.so"
@@ -662,10 +646,10 @@ def edit(self, spec, prefix):
                 #
                 # and use `-fpp` instead
                 mkf.write("CPP = # {0} -P\n".format(spack_cc))
-                mkf.write("AR  = {0}/xiar -r\n".format(intel_bin_dir))
+                mkf.write("AR  = {0}/xiar -qs\n".format(intel_bin_dir))
             else:
                 mkf.write("CPP = # {0} -E\n".format(spack_cc))
-                mkf.write("AR  = ar -r\n")
+                mkf.write("AR  = ar -qs\n")  # r = qs is a GNU extension
 
             if "+cuda" in spec:
                 mkf.write(
@@ -762,7 +746,7 @@ def fix_package_config(self):
                 content += " " + self.spec["lapack"].libs.ld_flags
                 content += " " + self.spec["fftw-api"].libs.ld_flags
 
-                if "^fftw+openmp" in self.spec:
+                if (self.spec["fftw-api"].name == "fftw") and ("+openmp" in self.spec["fftw"]):
                     content += " -lfftw3_omp"
 
                 content += "\n"
diff --git a/var/spack/repos/builtin/packages/intel-oneapi-mkl/package.py b/var/spack/repos/builtin/packages/intel-oneapi-mkl/package.py
--- a/var/spack/repos/builtin/packages/intel-oneapi-mkl/package.py
+++ b/var/spack/repos/builtin/packages/intel-oneapi-mkl/package.py
@@ -3,6 +3,7 @@
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
+from os.path import dirname, isdir
 
 from spack.package import *
 
@@ -153,7 +154,10 @@ def _find_mkl_libs(self, shared):
         if "+cluster" in self.spec:
             libs.append(self._xlp64_lib("libmkl_blacs_intelmpi"))
 
-        return find_libraries(libs, self.component_prefix.lib.intel64, shared=shared)
+        lib_path = self.component_prefix.lib.intel64
+        lib_path = lib_path if isdir(lib_path) else dirname(lib_path)
+
+        return find_libraries(libs, lib_path, shared=shared)
 
     def _xlp64_lib(self, lib):
         return lib + ("_ilp64" if "+ilp64" in self.spec else "_lp64")
diff --git a/var/spack/repos/builtin/packages/netlib-scalapack/package.py b/var/spack/repos/builtin/packages/netlib-scalapack/package.py
--- a/var/spack/repos/builtin/packages/netlib-scalapack/package.py
+++ b/var/spack/repos/builtin/packages/netlib-scalapack/package.py
@@ -44,6 +44,8 @@ class ScalapackBase(CMakePackage):
         sha256="072b006e485f0ca4cba56096912a986e4d3da73aae51c2205928aa5eb842cefd",
         when="@2.2.0",
     )
+    # From Homebrew, integrated @upstream in different form over multiple commits
+    patch("fix-build-macos.patch", when="@2.2.0")
 
     def flag_handler(self, name, flags):
         iflags = []


Test patch: