Repository: spack/spack

Issue:
Installation issue: cuda
### Description

Installing cuda 10.1.243 fails on a POWER machine.  On the other hand, cuda 10.2.89 installation works (but I specifically need 10.1.X).

### Steps to reproduce the issue

```console
$ spack install cuda@10.1.243
==> Installing cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u
==> No binary for cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u found: installing from source
==> Using cached archive: /grid5000/spack/var/spack/cache/_source-cache/archive/b1/b198002eef010bab9e745ae98e47567c955d00cf34cc8f8d2f0a6feb810523bf                                                                                          
==> cuda: Executing phase: 'install'
==> Error: ProcessError: Command exited with status 1:
    '/usr/bin/sh' '/var/spack/stage/root/spack-stage-cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/spack-src/cuda_10.1.243_418.87.00_linux_ppc64le.run' '--silent' '--override' '--toolkit' '--installpath=/grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u'
See build log for details:
  /var/spack/stage/root/spack-stage-cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/spack-build-out.txt
```

### Information on your system

<!-- Please include the output of `spack debug report` -->
* **Spack:** 0.16.0-873-c63f680d2a
* **Python:** 3.7.3
* **Platform:** linux-debian10-power8le
* **Concretizer:** original
<!-- If you have any relevant configuration detail (custom `packages.yaml` or `modules.yaml`, etc.) you can add that here as well. -->

### Additional information

<!-- Please upload the following files. They should be present in the stage directory of the failing build. Also upload any config.log or similar file if one exists. -->
* [spack-build-out.txt](https://github.com/spack/spack/files/5843878/spack-build-out.txt)
* [spack-build-env.txt](https://github.com/spack/spack/files/5843880/spack-build-env.txt)
* [/var/log/cuda-installer.log](https://github.com/spack/spack/files/5843882/cuda-installer.log)

It looks like this part of the log is about the failure:
```
/grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/include/cublas.h can't be opened
/grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/include/cublasLt.h can't be opened
/grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/include/cublasXt.h can't be opened
/grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/include/cublas_api.h can't be opened
/grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/include/cublas_v2.h can't be opened
/grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/include/nvblas.h can't be opened
/grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/src/fortran.c can't be opened
/grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/src/fortran.h can't be opened
/grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/src/fortran_common.h can't be opened
/grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/src/fortran_thunking.c can't be opened
/grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/src/fortran_thunking.h can't be opened
```

Corresponding output in cuda-installer.log:
```
[ERROR]: boost::filesystem::copy_file: No such file or directory: "./builds/cublas/include/cublas.h", "/grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/include/cublas.h"
[INFO]: Installed: /grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/include/cublas.h
[ERROR]: boost::filesystem::copy_file: No such file or directory: "./builds/cublas/include/cublasLt.h", "/grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/include/cublasLt.h"
[INFO]: Installed: /grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/include/cublasLt.h
[ERROR]: boost::filesystem::copy_file: No such file or directory: "./builds/cublas/include/cublasXt.h", "/grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/include/cublasXt.h"
[INFO]: Installed: /grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/include/cublasXt.h
[ERROR]: boost::filesystem::copy_file: No such file or directory: "./builds/cublas/include/cublas_api.h", "/grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/include/cublas_api.h"
[INFO]: Installed: /grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/include/cublas_api.h
[ERROR]: boost::filesystem::copy_file: No such file or directory: "./builds/cublas/include/cublas_v2.h", "/grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/include/cublas_v2.h"
[INFO]: Installed: /grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/include/cublas_v2.h
[ERROR]: boost::filesystem::copy_file: No such file or directory: "./builds/cublas/include/nvblas.h", "/grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/include/nvblas.h"
[INFO]: Installed: /grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/include/nvblas.h
[ERROR]: boost::filesystem::copy_file: No such file or directory: "./builds/cublas/src/fortran.c", "/grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/src/fortran.c"
[INFO]: Installed: /grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/src/fortran.c
[ERROR]: boost::filesystem::copy_file: No such file or directory: "./builds/cublas/src/fortran.h", "/grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/src/fortran.h"
[INFO]: Installed: /grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/src/fortran.h
[ERROR]: boost::filesystem::copy_file: No such file or directory: "./builds/cublas/src/fortran_common.h", "/grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/src/fortran_common.h"
[INFO]: Installed: /grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/src/fortran_common.h
[ERROR]: boost::filesystem::copy_file: No such file or directory: "./builds/cublas/src/fortran_thunking.c", "/grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/src/fortran_thunking.c"
[INFO]: Installed: /grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/src/fortran_thunking.c
[ERROR]: boost::filesystem::copy_file: No such file or directory: "./builds/cublas/src/fortran_thunking.h", "/grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/src/fortran_thunking.h"
[INFO]: Installed: /grid5000/spack/opt/spack/linux-debian10-ppc64le/gcc-8.3.0/cuda-10.1.243-rwagj62sy2hg3rx6n5tpcpcvfqbxou6u/src/fortran_thunking.
```

<!-- Some packages have maintainers who have volunteered to debug build failures. Run `spack maintainers <name-of-the-package>` and @mention them here if they exist. -->
@Rombur  @ax3l

### General information

<!-- These boxes can be checked by replacing [ ] with [x] or by clicking them after submitting the issue. -->
- [x] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [x] I have run `spack maintainers <name-of-the-package>` and @mentioned any maintainers
- [x] I have uploaded the build log and environment files
- [x] I have searched the issues of this repo and believe this is not a duplicate



Discussion:
It works fine when running the cuda installer without `--installpath`, which defaults to `/usr/local/cuda-10.1/` (but this is useless for Spack of course)

There is a discussion about this issue here: https://forums.developer.nvidia.com/t/cuda-10-1-243-10-1-update-2-ppc64le-run-file-installation-issue/82433/8

Here is the workaround from the above link:
```
$ mkdir -p $INSTALLDIR/targets/x86_64-linux/include $INSTALLDIR/src
$ ln -s targets/x86_64-linux/include/ $INSTALLDIR/include
$ sh cuda_10.1.243_418.87.00_linux_ppc64le.run --silent --override --toolkit --installpath=$INSTALLDIR
```

I tested it and it works. The `x86_64-linux` directory is of course strange (this is for ppc64le), but since applications use the `include` symlink, it shouldn't really matter.

I will try to integrate that workaround in the Spack package.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/cuda/package.py b/var/spack/repos/builtin/packages/cuda/package.py
--- a/var/spack/repos/builtin/packages/cuda/package.py
+++ b/var/spack/repos/builtin/packages/cuda/package.py
@@ -134,6 +134,15 @@ def install(self, spec, prefix):
         # https://gist.github.com/ax3l/9489132
         # for details.
 
+        # CUDA 10.1 on ppc64le fails to copy some files, the workaround is adapted from
+        # https://forums.developer.nvidia.com/t/cuda-10-1-243-10-1-update-2-ppc64le-run-file-installation-issue/82433
+        # See also #21170
+        if spec.satisfies('@10.1.243') and platform.machine() == 'ppc64le':
+            includedir = "targets/ppc64le-linux/include"
+            os.makedirs(os.path.join(prefix, includedir))
+            os.makedirs(os.path.join(prefix, "src"))
+            os.symlink(includedir, os.path.join(prefix, "include"))
+
         # CUDA 10.1+ has different cmdline options for the installer
         arguments = [
             runfile,            # the install script


Test patch: