Repository: spack/spack

Issue:
Installation issue: libnvcpumath.so not found when compiling with nvhpc
<!-- Thanks for taking the time to report this build failure. To proceed with the report please:

1. Title the issue "Installation issue: <name-of-the-package>".
2. Provide the information required below.

We encourage you to try, as much as possible, to reduce your problem to the minimal example that still reproduces the issue. That would help us a lot in fixing it quickly and effectively! -->

### Steps to reproduce the issue

<!-- Fill in the exact spec you are trying to build and the relevant part of the error message -->
```console
[sajid@xrmlite ~]$ spack install hdf5@1.12.0+mpi+hl+szip%nvhpc ^nvhpc+mpi+blas+lapack
[+] /home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/nvhpc-21.2/libszip-2.1.1-rsup3nfecownmtipknab5wf5aql22lgj
[+] /home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/nvhpc-21.2/nvhpc-21.2-kmv5w5kzoxo22cdpali5ljffmozqtr5n
[+] /home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/nvhpc-21.2/zlib-1.2.11-5uzatagcbg42ods5y2fspk2z5tv576hn
==> Installing hdf5-1.12.0-mokjam2du3wri26dflntfpl2647cequz
==> No binary for hdf5-1.12.0-mokjam2du3wri26dflntfpl2647cequz found: installing from source
==> Fetching https://spack-llnl-mirror.s3-us-west-2.amazonaws.com/_source-cache/archive/a6/a62dcb276658cb78e6795dd29bf926ed7a9bc4edf6e77025cd2c689a8f97c17a.tar.gz
###################################################################################################################################################################################################################################### 100.0%
==> Ran patch() for hdf5
==> hdf5: Executing phase: 'autoreconf'
==> hdf5: Executing phase: 'configure'
==> Error: ProcessError: Command exited with status 77:
    '/tmp/sajid/spack-stage/spack-stage-hdf5-1.12.0-mokjam2du3wri26dflntfpl2647cequz/spack-src/configure' '--prefix=/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/nvhpc-21.2/hdf5-1.12.0-mokjam2du3wri26dflntfpl2647cequz' '--enable-unsupported' '--enable-symbols=yes' '--with-zlib' '--disable-threadsafe' '--disable-cxx' '--enable-hl' '--disable-fortran' '--disable-java' '--with-szlib' '--enable-build-mode=production' '--enable-shared' 'CFLAGS=-fpic' 'CXXFLAGS=-fpic' 'FCFLAGS=-fpic' '--enable-parallel' 'CC=/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/nvhpc-21.2/nvhpc-21.2-kmv5w5kzoxo22cdpali5ljffmozqtr5n/Linux_x86_64/21.2/comm_libs/mpi/bin/mpicc'

2 errors found in build log:
     27    No match to get fc_version_info for /home/sajid/packages/spack/lib/spack/env/nvhpc/nvfortran
     28    No match to get cxx_version_info for /home/sajid/packages/spack/lib/spack/env/nvhpc/nvc++
     29    checking for config ./config/site-specific/host-xrmlite... no
     30    checking build mode... production
     31    checking for gcc... /home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/nvhpc-21.2/nvhpc-21.2-kmv5w5kzoxo22cdpali5ljffmozqtr5n/Linux_x86_64/21.2/comm_libs/mpi/bin/mpicc
     32    checking whether the C compiler works... no
  >> 33    configure: error: in `/tmp/sajid/spack-stage/spack-stage-hdf5-1.12.0-mokjam2du3wri26dflntfpl2647cequz/spack-src':
  >> 34    configure: error: C compiler cannot create executables
     35    See `config.log' for more details

See build log for details:
  /tmp/sajid/spack-stage/spack-stage-hdf5-1.12.0-mokjam2du3wri26dflntfpl2647cequz/spack-build-out.txt
...
```

Specific error :
```
configure:4575: checking whether the C compiler works
configure:4597: /home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/nvhpc-21.2/nvhpc-21.2-kmv5w5kzoxo22cdpali5ljffmozqtr5n/Linux_x86_64/21.2/comm_libs/mpi/bin/mpicc  -fpic     conftest.c  >&5
/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/nvhpc-21.2/nvhpc-21.2-kmv5w5kzoxo22cdpali5ljffmozqtr5n/Linux_x86_64/21.2/comm_libs/openmpi/openmpi-3.1.5/bin/.bin/mpicc: error while loading shared libraries: libnvcpumath.so: cannot open shared object file: No such file or directory
```

### Information on your system

<!-- Please include the output of `spack debug report` -->
```
[sajid@xrmlite ~]$ spack debug report
* **Spack:** 0.15.4-4061-1803070d3d
* **Python:** 3.6.8
* **Platform:** linux-centos8-broadwell
* **Concretizer:** original
[sajid@xrmlite ~]$
```
<!-- If you have any relevant configuration detail (custom `packages.yaml` or `modules.yaml`, etc.) you can add that here as well. -->

### Additional information

<!-- Please upload the following files. They should be present in the stage directory of the failing build. Also upload any config.log or similar file if one exists. -->
* [spack-build-out.txt](https://github.com/spack/spack/files/6199559/spack-build-out.txt)
* [spack-build-env.txt](https://github.com/spack/spack/files/6199560/spack-build-env.txt)
* [config.log](https://github.com/spack/spack/files/6199573/config.log)

PS: I added `extra_rpaths` in the compiler config, but that didn't help.

```
[sajid@xrmlite ~]$ cat .spack/linux/compilers.yaml
...
...
...
- compiler:
    spec: nvhpc@21.2
    paths:
      cc: /home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-8.3.1/nvhpc-21.2-nylqnmndrwuppcz4mrw22n6orbo5ozze/Linux_x86_64/21.2/compilers/bin/nvc
      cxx: /home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-8.3.1/nvhpc-21.2-nylqnmndrwuppcz4mrw22n6orbo5ozze/Linux_x86_64/21.2/compilers/bin/nvc++
      f77: /home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-8.3.1/nvhpc-21.2-nylqnmndrwuppcz4mrw22n6orbo5ozze/Linux_x86_64/21.2/compilers/bin/nvfortran
      fc: /home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-8.3.1/nvhpc-21.2-nylqnmndrwuppcz4mrw22n6orbo5ozze/Linux_x86_64/21.2/compilers/bin/nvfortran
    flags: {}
    operating_system: centos8
    target: x86_64
    modules: []
    environment: {}
    extra_rpaths:
    - /home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/nvhpc-21.2/nvhpc-21.2-kmv5w5kzoxo22cdpali5ljffmozqtr5n/Linux_x86_64/21.2/compilers/lib
    - /home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/nvhpc-21.2/nvhpc-21.2-kmv5w5kzoxo22cdpali5ljffmozqtr5n/Linux_x86_64/21.2/math_libs/lib64
    - /home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/nvhpc-21.2/nvhpc-21.2-kmv5w5kzoxo22cdpali5ljffmozqtr5n/Linux_x86_64/21.2/comm_libs/mpi/lib
[sajid@xrmlite ~]$
```


PPS: I ended up installing `nvhpc` twice as spack doesn't seem to be able to use the blas/lapack/mpi from the compiler :
```
[sajid@xrmlite ~]$ spack spec -I hdf5@1.12.0+mpi+hl+szip %nvhpc
Input spec
--------------------------------
 -   hdf5@1.12.0%nvhpc+hl+mpi+szip

Concretized
--------------------------------
 -   hdf5@1.12.0%nvhpc@21.2~cxx~debug~fortran+hl~java+mpi+pic+shared+szip~threadsafe api=none arch=linux-centos8-broadwell
[+]      ^libszip@2.1.1%nvhpc@21.2 arch=linux-centos8-broadwell
 -       ^openmpi@4.0.5%nvhpc@21.2~atomics~cuda~cxx~cxx_exceptions+gpfs~internal-hwloc~java~legacylaunchers~lustre~memchecker~pmi~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none patches=60ce20bc14d98c572ef7883b9fcd254c3f232c2f3a13377480f96466169ac4c8,fba0d3a784a9723338722b48024a22bb32f6a951db841a4e9f08930a93f41d7a schedulers=none arch=linux-centos8-broadwell
 -           ^hwloc@2.4.1%nvhpc@21.2~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-centos8-broadwell
 -               ^libpciaccess@0.16%nvhpc@21.2 patches=6e08dc445ece06e9e8b1344397f2d3f169005703ddc0f2ae24f366cde78c7377 arch=linux-centos8-broadwell
[+]                  ^libtool@2.4.6%nvhpc@21.2 arch=linux-centos8-broadwell
[+]                      ^m4@1.4.18%nvhpc@21.2+sigsegv patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,5746cf51f45b405661c3edae7a78c33d41e54d83f635d16e2bf1f956dbfbf635,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 arch=linux-centos8-broadwell
[+]                          ^libsigsegv@2.12%nvhpc@21.2 arch=linux-centos8-broadwell
[+]                  ^pkgconf@1.7.3%nvhpc@21.2 patches=6e94c68d45d4cb8148c8c53699921d10b13784f758f0563ce64c838e0ca48ceb arch=linux-centos8-broadwell
 -                   ^util-macros@1.19.1%nvhpc@21.2 arch=linux-centos8-broadwell
[+]              ^libxml2@2.9.10%nvhpc@21.2~python patches=05ff238cf435825ef835c7ae39376b52dc83d8caf19e962f0766c841386a305a,10a88ad47f9797cf7cf2d7d07241f665a3b6d1f31fa026728c8c2ae93e1664e9 arch=linux-centos8-broadwell
[+]                  ^libiconv@1.16%nvhpc@21.2 arch=linux-centos8-broadwell
[+]                  ^xz@5.2.5%nvhpc@21.2~pic arch=linux-centos8-broadwell
[+]                  ^zlib@1.2.11%nvhpc@21.2+optimize+pic+shared arch=linux-centos8-broadwell
[+]              ^ncurses@6.2%nvhpc@21.2~symlinks+termlib arch=linux-centos8-broadwell
 -           ^libevent@2.1.12%nvhpc@21.2+openssl arch=linux-centos8-broadwell
 -               ^openssl@1.1.1j%nvhpc@21.2~docs+systemcerts arch=linux-centos8-broadwell
[+]                  ^perl@5.26.3%nvhpc@21.2~cpanm+shared+threads arch=linux-centos8-broadwell
 -           ^numactl@2.0.14%nvhpc@21.2 patches=4e1d78cbbb85de625bad28705e748856033eaafab92a66dffd383a3d7e00cc94,62fc8a8bf7665a60e8f4c93ebbd535647cebf74198f7afafec4c085a8825c006 arch=linux-centos8-broadwell
 -               ^autoconf@2.69%nvhpc@21.2 arch=linux-centos8-broadwell
[+]                  ^help2man@1.47.16%nvhpc@21.2 arch=linux-centos8-broadwell
[+]                      ^gettext@0.21%nvhpc@21.2+bzip2+curses+git~libunistring+libxml2+tar+xz patches=6e530daaae14725d578d6cafbf7d523accc9ed29fd817bd421cf98a5f51e9e1b,bbe9f0539aa504966ac104224d25a9260faa1015ed3adda936467be9c7de4eae,fb27a3fb5e414bdc50ffebbbe2da986473df70a493caa4396226f51a67c55424 arch=linux-centos8-broadwell
[+]                          ^bzip2@1.0.8%nvhpc@21.2+shared arch=linux-centos8-broadwell
[+]                              ^diffutils@3.7%nvhpc@21.2 patches=6e42dc243f17aab29fd167f060f5bc1f08813e03368eb301b43c95d4b1386681 arch=linux-centos8-broadwell
[+]                          ^tar@1.32%nvhpc@21.2 patches=4e8cdf69294c376c45c39d059f11769c5c996c2d84cdfb56eb69890e0aa4e991 arch=linux-centos8-broadwell
[+]              ^automake@1.16.3%nvhpc@21.2 arch=linux-centos8-broadwell
 -           ^openssh@8.5p1%nvhpc@21.2 arch=linux-centos8-broadwell
 -               ^libedit@3.1-20210216%nvhpc@21.2 arch=linux-centos8-broadwell

[sajid@xrmlite ~]$

[sajid@xrmlite ~]$ spack find -ldv nvhpc
==> 2 installed packages
-- linux-centos8-broadwell / gcc@8.3.1 --------------------------
nylqnmn nvhpc@21.2+blas+lapack+mpi install_type=single


-- linux-centos8-broadwell / nvhpc@21.2 -------------------------
kmv5w5k nvhpc@21.2+blas+lapack+mpi install_type=single

[sajid@xrmlite ~]$
```

<!-- Some packages have maintainers who have volunteered to debug build failures. Run `spack maintainers <name-of-the-package>` and @mention them here if they exist. -->

### General information

<!-- These boxes can be checked by replacing [ ] with [x] or by clicking them after submitting the issue. -->
- [ x] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [ x] I have run `spack maintainers <name-of-the-package>` and @mentioned any maintainers
- [ x] I have uploaded the build log and environment files
- [ ?] I have searched the issues of this repo and believe this is not a duplicate

@samcmill 


Discussion:
I can reproduce the issue and am looking into a fix.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/nvhpc/package.py b/var/spack/repos/builtin/packages/nvhpc/package.py
--- a/var/spack/repos/builtin/packages/nvhpc/package.py
+++ b/var/spack/repos/builtin/packages/nvhpc/package.py
@@ -123,6 +123,23 @@ def setup_run_environment(self, env):
             env.prepend_path('CPATH', mpi_prefix.include)
             env.prepend_path('LD_LIBRARY_PATH', mpi_prefix.lib)
 
+    def setup_dependent_build_environment(self, env, dependent_spec):
+        prefix = Prefix(join_path(self.prefix,
+                                  'Linux_%s' % self.spec.target.family,
+                                  self.version, 'compilers'))
+
+        env.prepend_path('CPATH',           prefix.include)
+        env.prepend_path('LIBRARY_PATH',    prefix.lib)
+        env.prepend_path('LD_LIBRARY_PATH', prefix.lib)
+
+        if '+mpi' in self.spec:
+            mpi_prefix = Prefix(join_path(self.prefix,
+                                          'Linux_%s' % self.spec.target.family,
+                                          self.version, 'comm_libs', 'mpi'))
+
+            env.prepend_path('CPATH', mpi_prefix.include)
+            env.prepend_path('LD_LIBRARY_PATH', mpi_prefix.lib)
+
     def setup_dependent_package(self, module, dependent_spec):
         if '+mpi' in self.spec or self.provides('mpi'):
             mpi_prefix = Prefix(join_path(self.prefix,


Test patch: