Repository: spack/spack

Issue:
py-torch@1.6.0 produces a GLIBC error when built with CUDA-11
When trying to use py-torch -1.6.0 built with CUDA 11 I see the following error:
```console
python -c 'import torch'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/packages/gpjohnsn/spack/opt/spack/linux-centos7-skylake_avx512/gcc-9.3.0/py-torch-1.6.0-vw3qfqc6okp7suoc4ifgkdgqab5pvh56/lib/python3.8/site-packages/torch/__init__.py", line 189, in <module>
    from torch._C import *
ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.22' not found (required by /opt/packages/gpjohnsn/spack/opt/spack/linux-centos7-skylake_avx512/gcc-9.3.0/py-torch-1.6.0-vw3qfqc6okp7suoc4ifgkdgqab5pvh56/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
```
This works fine when using `CUDA-10.2.89` and `7.6.5.32-10.2-linux-x64`

### Steps to reproduce the issue

```console
$ spack install py-torch cuda_arch=35,75 ^cudnn@8.0.2.39-11.0-linux-x64
...
```

### Information on your system

```console
$ spack debug report
* **Spack:** 0.11.2-8723-01df552
* **Python:** 3.8.5
* **Platform:** linux-centos7-skylake_avx512
```
```console
$ spack find -vd py-torch
==> 1 installed package
-- linux-centos7-skylake_avx512 / gcc@9.3.0 ---------------------
py-torch@1.6.0~binary~caffe2+cuda+cudnn~distributed~fbgemm~ffmpeg~gloo~leveldb~lmdb~magma~miopen+mkldnn+nccl~nnpack~opencv+openmp~qnnpack~redis~rocm~tbb~test~xnnpack~zstd cuda_arch=35,75
    cuda@11.0.2
        libxml2@2.9.10~python
            libiconv@1.16
            xz@5.2.5~pic
            zlib@1.2.11+optimize+pic+shared
    cudnn@8.0.2.39-11.0-linux-x64
    eigen@3.3.7 build_type=RelWithDebInfo
    nccl@2.7.8-1+cuda cuda_arch=none
        rdma-core@20 build_type=RelWithDebInfo
            libnl@3.3.0
    openblas@0.3.10~consistent_fpcsr~ilp64+pic+shared threads=none
    protobuf@3.12.2+shared build_type=Release
    py-numpy@1.19.2+blas+lapack
        py-setuptools@50.1.0
            python@3.8.5+bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87
                bzip2@1.0.8+shared
                expat@2.2.9+libbsd
                    libbsd@0.10.0
                gdbm@1.18.1
                    readline@8.0
                        ncurses@6.2~symlinks+termlib
                gettext@0.21+bzip2+curses+git~libunistring+libxml2+tar+xz
                    tar@1.32
                libffi@3.3 patches=26f26c6f29a7ce9bf370ad3ab2610f99365b4bdd7b82e7c31df41a3370d685c0
                libuuid@1.0.3
                openssl@1.1.1g+systemcerts
                sqlite@3.33.0+column_metadata+fts~functions~rtree
    py-pybind11@2.5.0 build_type=RelWithDebInfo
    py-pyyaml@5.3.1+libyaml
        libyaml@0.2.4
```
<!-- If you have any relevant configuration detail (custom `packages.yaml` or `modules.yaml`, etc.) you can add that here as well. -->

### Additional information

<!-- These boxes can be checked by replacing [ ] with [x] or by clicking them after submitting the issue. -->
- [x] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [x] I have searched the issues of this repo and believe this is not a duplicate
- [ ] I have run the failing commands in debug mode and reported the output

<!-- We encourage you to try, as much as possible, to reduce your problem to the minimal example that still reproduces the issue. That would help us a lot in fixing it quickly and effectively!

If you want to ask a question about the tool (how to use it, what it can currently do, etc.), try the `#general` channel on our Slack first. We have a welcoming community and chances are you'll get your reply faster and without opening an issue.

Other than that, thanks for taking the time to contribute to Spack! -->



Discussion:
How are you running PyTorch? Python libs don't always support RPATH, so sometimes you need to use `LD_LIBRARY_PATH` for the dependencies.
I autoload the modules.
```console
$ module load py-torch/1.6.0_gcc-9.3.0
$ module li

Currently Loaded Modules:
  1) libiconv/1.16_gcc-9.3.0                  12) protobuf/3.12.2_gcc-9.3.0  23) readline/8.0_gcc-9.3.0
  2) xz/5.2.5_gcc-9.3.0                       13) bzip2/1.0.8_gcc-9.3.0      24) sqlite/3.33.0_gcc-9.3.0
  3) zlib/1.2.11_gcc-9.3.0                    14) libbsd/0.10.0_gcc-9.3.0    25) py-numpy/1.19.2_gcc-9.3.0
  4) libxml2/2.9.10_gcc-9.3.0                 15) expat/2.2.9_gcc-9.3.0      26) py-pybind11/2.5.0_gcc-9.3.0
  5) cuda/11.0.2_gcc-9.3.0                    16) gdbm/1.18.1_gcc-9.3.0      27) libyaml/0.2.4_gcc-9.3.0
  6) cudnn/8.0.2.39-11.0-linux-x64_gcc-9.3.0  17) tar/1.32_gcc-9.3.0         28) py-pyyaml/5.3.1_gcc-9.3.0
  7) eigen/3.3.7_gcc-9.3.0                    18) gettext/0.21_gcc-9.3.0     29) py-setuptools/50.1.0_gcc-9.3.0
  8) libnl/3.3.0_gcc-9.3.0                    19) libffi/3.3_gcc-9.3.0       30) python/3.8.5_gcc-9.3.0
  9) rdma-core/20_gcc-9.3.0                   20) libuuid/1.0.3_gcc-9.3.0    31) py-torch/1.6.0_gcc-9.3.0
 10) nccl/2.7.8-1_gcc-9.3.0                   21) ncurses/6.2_gcc-9.3.0
 11) openblas/0.3.10_gcc-9.3.0                22) openssl/1.1.1g_gcc-9.3.0
```
What compiler did you use to build PyTorch? Is that compiler loaded as well?
The compiler is gcc-9.3.0. I have never had to load a compiler environment module before but the compiler paths are available. This does bring up a point though. I verified that this works with CUDA-10.2 but that used gcc-8.4.0. Do you think it might be a compiler issue?
I just did the build with CUDA-11 using gcc-8.3.0 and see the same result.
```console
$ python -c 'import torch'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/packages/gpjohnsn/spack/opt/spack/linux-centos7-skylake_avx512/gcc-8.3.0/py-torch-1.6.0-2vec3dbsmp2j6woeduxtgr6rbiq3f5i3/lib/python3.8/site-packages/torch/__init__.py", line 189, in <module>
    from torch._C import *
ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.22' not found (required by /opt/packages/gpjohnsn/spack/opt/spack/linux-centos7-skylake_avx512/gcc-8.3.0/py-torch-1.6.0-2vec3dbsmp2j6woeduxtgr6rbiq3f5i3/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
```
I don't think it's a compiler issue. I think the problem is that Python libs like PyTorch don't RPATH dependencies, including compiler libraries like GLIBCXX. So when you use that executable, you always need to load the module for the compiler it was built with.
I understand what you are saying, and things do work when I load the compiler module, but that does not seem right. I have never had to load gcc compiler modules and that is not necessary when using pytorch-1.6.0 with cuda-10.2. I agree that this is an RPATH problem, but one that seems to be specific to CUDA-11.
@glennpj, if you rebuild with this, does the problem go away?

```diff
diff --git a/var/spack/repos/builtin/packages/py-torch/package.py b/var/spack/repos/builtin/packages/py-torch/package.py
index 971db1c..17a32cb 100644
--- a/var/spack/repos/builtin/packages/py-torch/package.py
+++ b/var/spack/repos/builtin/packages/py-torch/package.py
@@ -207,6 +207,12 @@ def libs(self):
             'torch', 'lib')
         return find_libraries('libtorch', root)
 
+    @when('@1.5.0:')
+    def patch(self):
+        filter_file('torch_global_deps PROPERTIES LINKER_LANGUAGE C',
+                    'torch_global_deps PROPERTIES LINKER_LANGUAGE CXX',
+                    'caffe2/CMakeLists.txt')
+
     @property
     def headers(self):
         root = join_path(

```
Yes! This patch has worked for us. Thanks
@aweits we should add this to the package, although I would first like to open a PR with the PyTorch devs to comment on this.
> @aweits we should add this to the package, although I would first like to open a PR with the PyTorch devs to comment on this.

Will do, just getting back to all things spack-ish...

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/py-torch/package.py b/var/spack/repos/builtin/packages/py-torch/package.py
--- a/var/spack/repos/builtin/packages/py-torch/package.py
+++ b/var/spack/repos/builtin/packages/py-torch/package.py
@@ -186,6 +186,13 @@ def headers(self):
         headers.directories = [root]
         return headers
 
+    @when('@1.5.0:')
+    def patch(self):
+        # https://github.com/pytorch/pytorch/issues/52208
+        filter_file('torch_global_deps PROPERTIES LINKER_LANGUAGE C',
+                    'torch_global_deps PROPERTIES LINKER_LANGUAGE CXX',
+                    'caffe2/CMakeLists.txt')
+
     def setup_build_environment(self, env):
         def enable_or_disable(variant, keyword='USE', var=None, newer=False):
             """Set environment variable to enable or disable support for a


Test patch: