Repository: spack/spack

Issue:
Installation issue: hpl with intel-oneapi-mkl
### Steps to reproduce the issue
Modify the hpl `package.py` adding:
```
        if (self.spec.satisfies('^intel-mkl') or
+           self.spec.satisfies('^intel-oneapi-mkl') or
            self.spec.satisfies('^intel-parallel-studio+mkl')):
            config.append('LDFLAGS={0}'.format(
                self.spec['blas'].libs.ld_flags))
```

```console
$ spack install hpl +openmp arch=x86_64 ^intel-oneapi-mpi ^intel-oneapi-mkl
[+] /opt/intel/oneapi (external intel-oneapi-mkl-2021.1.1-qyjzf2zkff4rhbw64iduevpqpb35dkmg)
[+] /opt/intel/oneapi (external intel-oneapi-mpi-2021.1.1-um6h5no2hqudztstzosrnxz676vmoct2)
==> Installing hpl-2.3-4gwoqbhrvbhpjbu5wkgefawvadvxrhtq
==> No binary for hpl-2.3-4gwoqbhrvbhpjbu5wkgefawvadvxrhtq found: installing from source
==> Using cached archive: /home/vasco/spack/var/spack/cache/_source-cache/archive/32/32c5c17d22330e6f2337b681aded51637fb6008d3f0eb7c277b163fadd612830.tar.gz
==> No patches needed for hpl
==> hpl: Executing phase: 'autoreconf'
==> hpl: Executing phase: 'configure'
==> Error: ProcessError: Command exited with status 77:
    '/dev/shm/vasco/spack-stage/spack-stage-hpl-2.3-4gwoqbhrvbhpjbu5wkgefawvadvxrhtq/spack-src/configure' '--prefix=/home/vasco/spack/opt/spack/linux-centos7-x86_64/gcc-9.2.0/hpl-2.3-4gwoqbhrvbhpjbu5wkgefawvadvxrhtq' 'CFLAGS=-O3 -fopenmp' 'LDFLAGS=-L/opt/intel/oneapi/mkl/latest/lib/intel64 -lmkl_blacs_sgimpt_lp64 -lmkl_cdft_core -lmkl_core -lmkl_gf_ilp64 -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_intel_ilp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_pgi_thread -lmkl_rt -lmkl_scalapack_ilp64 -lmkl_scalapack_lp64 -lmkl_sequential -lmkl_sycl -lmkl_tbb_thread -lmkl_blacs_intelmpi_ilp64 -lmkl_blacs_intelmpi_lp64 -lmkl_blacs_openmpi_ilp64 -lmkl_blacs_openmpi_lp64 -lmkl_blacs_sgimpt_ilp64'

2 errors found in build log:
     1    ==> hpl: Executing phase: 'autoreconf'
     2    ==> hpl: Executing phase: 'configure'
     3    ==> [2021-02-17-10:53:34.682876] FILTER FILE: configure [replacing "^libs10=.*"]
     4    ==> [2021-02-17-10:53:34.698402] '/dev/shm/vasco/spack-stage/spack-stage-hpl-2.3-4gwoqbhrvbhpjbu5wkgefawvadvxrhtq/spack-src/configure' '--prefix
          =/home/vasco/spack/opt/spack/linux-centos7-x86_64/gcc-9.2.0/hpl-2.3-4gwoqbhrvbhpjbu5wkgefawvadvxrhtq' 'CFLAGS=-O3 -fopenmp' 'LDFLAGS=-L/opt/inte
          l/oneapi/mkl/latest/lib/intel64 -lmkl_blacs_sgimpt_lp64 -lmkl_cdft_core -lmkl_core -lmkl_gf_ilp64 -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_intel_ilp
          64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_pgi_thread -lmkl_rt -lmkl_scalapack_ilp64 -lmkl_scalapack_lp64 -lmkl_sequential -lmkl_sycl -lmkl_tb
          b_thread -lmkl_blacs_intelmpi_ilp64 -lmkl_blacs_intelmpi_lp64 -lmkl_blacs_openmpi_ilp64 -lmkl_blacs_openmpi_lp64 -lmkl_blacs_sgimpt_ilp64'
     5    checking for gcc... /home/vasco/spack/lib/spack/env/gcc/gcc
     6    checking whether the C compiler works... no
  >> 7    configure: error: in `/dev/shm/vasco/spack-stage/spack-stage-hpl-2.3-4gwoqbhrvbhpjbu5wkgefawvadvxrhtq/spack-src':
  >> 8    configure: error: C compiler cannot create executables
     9    See `config.log' for more details

See build log for details:
  /dev/shm/vasco/spack-stage/spack-stage-hpl-2.3-4gwoqbhrvbhpjbu5wkgefawvadvxrhtq/spack-build-out.txt
```

### Information on your system
```
$ spack debug report
* **Spack:** 0.16.0-1310-ee1b0b9
* **Python:** 3.6.8
* **Platform:** linux-centos7-skylake
* **Concretizer:** original
```

### Additional information
@rscohn2
[spack-cc-hpl-4gwoqbh.out.log](https://github.com/spack/spack/files/5995062/spack-cc-hpl-4gwoqbh.out.log)
[spack-cc-hpl-4gwoqbh.in.log](https://github.com/spack/spack/files/5995063/spack-cc-hpl-4gwoqbh.in.log)

### General information
- [x] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [x] I have run `spack maintainers <name-of-the-package>` and @mentioned any maintainers
- [x] I have uploaded the build log and environment files
- [x] I have searched the issues of this repo and believe this is not a duplicate


Discussion:
In the config.log file:
```
Using built-in specs.
COLLECT_GCC=/opt/gcc-9.2.0/bin/gcc
COLLECT_LTO_WRAPPER=/opt/gcc-9.2.0/libexec/gcc/x86_64-pc-linux-gnu/9.2.0/lto-wrapper
Target: x86_64-pc-linux-gnu
Configured with: ./configure --disable-multilib --prefix=/opt/gcc-9.2.0
Thread model: posix
gcc version 9.2.0 (GCC)
configure:2617: $? = 0
configure:2606: /home/vasco/spack/lib/spack/env/gcc/gcc -V >&5
gcc: error: unrecognized command line option '-V'
gcc: fatal error: no input files
compilation terminated.
configure:2617: $? = 1
configure:2606: /home/vasco/spack/lib/spack/env/gcc/gcc -qversion >&5
gcc: error: unrecognized command line option '-qversion'; did you mean '--version'?
configure:2617: $? = 1
configure:2637: checking whether the C compiler works
configure:2659: /home/vasco/spack/lib/spack/env/gcc/gcc -O3 -fopenmp  -L/opt/intel/oneapi/mkl/latest/lib/intel64 -lmkl_blacs_sgimpt_lp64 -lmkl_cdft_core -lmkl_core -lmkl_gf_ilp64 -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_intel_ilp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_pgi_thread -lmkl_rt -lmkl_scalapack_ilp64 -lmkl_scalapack_lp64 -lmkl_sequential -lmkl_sycl -lmkl_tbb_thread -lmkl_blacs_intelmpi_ilp64 -lmkl_blacs_intelmpi_lp64 -lmkl_blacs_openmpi_ilp64 -lmkl_blacs_openmpi_lp64 -lmkl_blacs_sgimpt_ilp64 conftest.c  >&5
/usr/bin/ld: warning: libsycl.so.5, needed by /opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_sycl.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libtbb.so.12, needed by /opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_tbb_thread.so, not found (try using -rpath or -rpath-link)
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_sycl.so: undefined reference to `cl::sycl::device::is_gpu() const'
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_sycl.so: undefined reference to `vtable for cl::sycl::detail::buffer_impl'
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_pgi_thread.so: undefined reference to `_mp_scheds_dyn_init8'
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_sycl.so: undefined reference to `cl::sycl::detail::OSUtil::getOSModuleHandle(void const*)'
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_sycl.so: undefined reference to `cl::sycl::malloc_shared(unsigned long, cl::sycl::queue const&)'
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_pgi_thread.so: undefined reference to `__c_mcopy4_bwd'
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_sycl.so: undefined reference to `cl::__host_std::fma(float, float, float)'
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_pgi_thread.so: undefined reference to `pgf90_compiled'
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_sycl.so: undefined reference to `clReleaseKernel'
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_sycl.so: undefined reference to `cl::sycl::queue::queue(_cl_command_queue*, cl::sycl::context const&, std::function<void (cl::sycl::exception_list)> const&)'
```
Thanks for the report. I will investigate.
With the merge of #22202 I have tried to build hpl:
```
# Add externally installed intel-oneapi-mpi and intel-oneapi-mkl
cat >>spack/etc/spack/defaults/packages.yaml <<EOF

  intel-oneapi-mpi:
    externals:
    - spec: intel-oneapi-mpi @2021.1.1
      prefix: /opt/intel/oneapi/

  intel-oneapi-mkl:
    externals:
    - spec: intel-oneapi-mkl @2021.11.
      prefix: /opt/intel/oneapi/
EOF

# Add the following to the hpl package:
    if (self.spec.satisfies('^intel-mkl') or
            self.spec.satisfies('^intel-oneapi-mkl') or
            self.spec.satisfies('^intel-parallel-studio+mkl')):
            config.append('LDFLAGS={0}'.format(
                self.spec['blas'].libs.ld_flags))

$ spack install hpl +openmp arch=x86_64 ^intel-oneapi-mpi ^intel-oneapi-mkl
```

The `config.log` contains:
```
configure:2659: /home/vasco/spack/lib/spack/env/gcc/gcc -O3 -fopenmp  -L/opt/intel/oneapi/mkl/latest/lib/intel64 -lmkl_blacs_sgimpt_lp64 -lmkl_cdft_core -lmkl_core -lmkl_gf_ilp64 -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_intel_ilp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_pgi_thread -lmkl_rt -lmkl_scalapack_ilp64 -lmkl_scalapack_lp64 -lmkl_sequential -lmkl_sycl -lmkl_tbb_thread -lmkl_blacs_intelmpi_ilp64 -lmkl_blacs_intelmpi_lp64 -lmkl_blacs_openmpi_ilp64 -lmkl_blacs_openmpi_lp64 -lmkl_blacs_sgimpt_ilp64 conftest.c  >&5
/usr/bin/ld: warning: libsycl.so.5, needed by /opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_sycl.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libtbb.so.12, needed by /opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_tbb_thread.so, not found (try using -rpath or -rpath-link)
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_sycl.so: undefined reference to `cl::sycl::device::is_gpu() const'
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_sycl.so: undefined reference to `vtable for cl::sycl::detail::buffer_impl'
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_pgi_thread.so: undefined reference to `_mp_scheds_dyn_init8'
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_sycl.so: undefined reference to `cl::sycl::detail::OSUtil::getOSModuleHandle(void const*)'
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_sycl.so: undefined reference to `cl::sycl::malloc_shared(unsigned long, cl::sycl::queue const&)'
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_pgi_thread.so: undefined reference to `__c_mcopy4_bwd'
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_sycl.so: undefined reference to `cl::__host_std::fma(float, float, float)'
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_pgi_thread.so: undefined reference to `pgf90_compiled'
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_sycl.so: undefined reference to `clReleaseKernel'
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_sycl.so: undefined reference to `cl::sycl::queue::queue(_cl_command_queue*, cl::sycl::context const&, std::function<void (cl::sycl::exception_list)> const&)'
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_sycl.so: undefined reference to `cl::__host_std::sin(double)'
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_sycl.so: undefined reference to `cl::__host_std::sincos(float, float*)'
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_sycl.so: undefined reference to `cl::sycl::detail::SYCLMemObjT::updateHostMemory()'
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_sycl.so: undefined reference to `cl::__host_std::copysign(double, double)'
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_sycl.so: undefined reference to `cl::sycl::detail::AccessorImplHost::~AccessorImplHost()'
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_sycl.so: undefined reference to `cl::__host_std::fmin(double, double)'
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_sycl.so: undefined reference to `cl::sycl::event::wait_and_throw()'
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_blacs_openmpi_ilp64.so: undefined reference to `ompi_mpi_dblcplex'
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_sycl.so: undefined reference to `cl::sycl::opencl::make_device(unsigned long)'
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_sycl.so: undefined reference to `cl::__host_std::tan(double)'
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_sycl.so: undefined reference to `cl::sycl::event::event()'
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_sycl.so: undefined reference to `cl::sycl::platform::platform()'
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_blacs_sgimpt_lp64.so: undefined reference to `MPI_Alltoallv'
/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_tbb_thread.so: undefined reference to `tbb::detail::r1::wait_bounded_queue_monitor(tbb::detail::r1::concurrent_monitor*, unsigned long, long, tbb::detail::d1::delegate_base&)'
```

I am getting the same output by letting spack install `intel-oneapi-mpi` and `intel-oneapi-mkl`.
MKL, DAL, etc all include optional libraries for SYCL. The SYCL library needs libsycl, which is part of the compiler runtime. I assumed package owners would only add the -lmkl_sycl when they wanted to use SYCL and would add a dependence on the compiler package.  It looks like this package is adding -l for every possible mkl library. 

I am not sure how this should be fixed. Ideally, there would be a separate SYCL runtime package and MKL would depend on it. Package owners like hpl would not have to understand the dependencies of mkl.

As a test, I tried adding a dependence on the compiler package, but that did not help. I need to check that.

@scheibelp @frankwillmore @glennpj @DanVev 
Note if all the `-l` flags are coming from `self.spec['blas'].libs.ld_flags`, then `intel-oneapi-mkl` can define a custom `libs` property which excludes the SYCL-based libraries.

> MKL, DAL, etc all include optional libraries for SYCL

Are these additional libraries needed for SYCL support? Or are they alternatives libraries built with SYCL support (in which case for example you wouldn't link `-lmkl_sequential` and `-lmkl_sycl`)?
The pre-oneapi intel-mkl package and has a much smaller set of libraries in libs so I will try to reproduce that.

They are additional libraries. I will make a test for mkl sycl interfaces with/without icx and hande that separately.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/intel-oneapi-mkl/package.py b/var/spack/repos/builtin/packages/intel-oneapi-mkl/package.py
--- a/var/spack/repos/builtin/packages/intel-oneapi-mkl/package.py
+++ b/var/spack/repos/builtin/packages/intel-oneapi-mkl/package.py
@@ -33,3 +33,9 @@ class IntelOneapiMkl(IntelOneApiLibraryPackage):
     @property
     def component_dir(self):
         return 'mkl'
+
+    @property
+    def libs(self):
+        lib_path = '{0}/{1}/latest/lib/intel64'.format(self.prefix, self.component_dir)
+        mkl_libs = ['libmkl_intel_ilp64', 'libmkl_sequential', 'libmkl_core']
+        return find_libraries(mkl_libs, root=lib_path, shared=True, recursive=False)


Test patch: