Repository: spack/spack

Issue:
Installation issue: py-torchvision: PyTorch has CUDA Version=11.7 and torchvision has CUDA Version=11.8
On a system with P100 GPUs, with the following Spack environment (where I explicitely set `cuda_arch=60` for some packages):

```yaml
spack:
  config:
    install_tree: /my-spack/spack
    build_stage: /tmp/spack-stage
  repos:
  - /home/tbouvier/spack-lille/mochi-spack-packages
  packages:
    all:
      providers:
        mpi: [openmpi]
      compiler:
      - gcc@10.4.0
    magma:
      variants: [+cuda cuda_arch=60]
    nccl:
      variants: [+cuda cuda_arch=60]
    nccl-tests:
      variants: [+cuda cuda_arch=60]
    py-horovod:
      variants: [+cuda cuda_arch=60 tensor_ops=nccl]
    py-torch:
      variants: [+cuda cuda_arch=60 +nccl +cudnn +magma +distributed]
  specs:
  # Precedence is important. These packages will be picked up in the mochi repo.
  - mochi-thallium ^argobots ^mercury +boostsys +ofi +debug ^libfabric +cuda +debug fabrics=tcp,rxm,verbs
  - cuda
  - cudnn@8.4
  - mpi
  - nccl
  - nccl-tests
  - python@3.10
  - nvtx +python
  - py-pip
  - py-pybind11
  - py-ruamel-yaml
  - py-filelock
  - py-wandb
  - py-continuum
  - py-timm
  - py-nvidia-dali@1.27.0-cuda110
  - py-torch@2.0.1 ~gloo
  - py-torchsummary
  - py-bokeh
  - py-matplotlib
  - py-horovod@0.28 controllers=mpi frameworks=pytorch
  - nano
  concretizer:
    unify: true
  view: /my-spack/view/chifflot-p100
```

From the error I get at runtime, it seems that there is a discrepancy between CUDA versions used to compile PyTorch and torchvision. I don't understand how this can happen, as I'm using `unify: true` in my environment. Could this be caused by [this](https://github.com/spack/spack/blob/da10487219ccd3b738b910000fe307de3fc4fdbd/var/spack/repos/builtin/packages/py-torchvision/package.py#L151) not picking up the correct `cuda_arch` for torchvision?

<pre>
[1,0]<stderr>:Traceback (most recent call last):
[1,0]<stderr>:  File "/root/distributed-continual-learning/main.py", line 21, in <module>
[1,0]<stderr>:    import backbone
[1,0]<stderr>:  File "/root/distributed-continual-learning/backbone/__init__.py", line 2, in <module>
[1,0]<stderr>:    from .ghostnet import *
[1,0]<stderr>:  File "/root/distributed-continual-learning/backbone/ghostnet.py", line 4, in <module>
[1,0]<stderr>:    import timm
[1,0]<stderr>:  File "/my-spack/view/chifflot-p100/lib/python3.10/site-packages/timm/__init__.py", line 2, in <module>
[1,0]<stderr>:    from .layers import is_scriptable, is_exportable, set_scriptable, set_exportable
[1,0]<stderr>:  File "/my-spack/view/chifflot-p100/lib/python3.10/site-packages/timm/layers/__init__.py", line 11, in <module>
[1,0]<stderr>:    from .conv_bn_act import ConvNormAct, ConvNormActAa, ConvBnAct
[1,0]<stderr>:  File "/my-spack/view/chifflot-p100/lib/python3.10/site-packages/timm/layers/conv_bn_act.py", line 9, in <module>
[1,0]<stderr>:    from .create_norm_act import get_norm_act_layer
[1,0]<stderr>:  File "/my-spack/view/chifflot-p100/lib/python3.10/site-packages/timm/layers/create_norm_act.py", line 14, in <module>
[1,0]<stderr>:    from .norm_act import BatchNormAct2d, GroupNormAct, LayerNormAct, LayerNormAct2d
[1,0]<stderr>:  File "/my-spack/view/chifflot-p100/lib/python3.10/site-packages/timm/layers/norm_act.py", line 20, in <module>
[1,0]<stderr>:    from torchvision.ops.misc import FrozenBatchNorm2d
[1,0]<stderr>:  File "/my-spack/view/chifflot-p100/lib/python3.10/site-packages/torchvision/__init__.py", line 6, in <module>
[1,0]<stderr>:    from torchvision import datasets, io, models, ops, transforms, utils
[1,0]<stderr>:  File "/my-spack/view/chifflot-p100/lib/python3.10/site-packages/torchvision/datasets/__init__.py", line 1, in <module>
[1,0]<stderr>:    from ._optical_flow import FlyingChairs, FlyingThings3D, HD1K, KittiFlow, Sintel
[1,0]<stderr>:  File "/my-spack/view/chifflot-p100/lib/python3.10/site-packages/torchvision/datasets/_optical_flow.py", line 12, in <module>
[1,0]<stderr>:    from ..io.image import _read_png_16
[1,0]<stderr>:  File "/my-spack/view/chifflot-p100/lib/python3.10/site-packages/torchvision/io/__init__.py", line 8, in <module>
[1,0]<stderr>:    from ._load_gpu_decoder import _HAS_GPU_VIDEO_DECODER
[1,0]<stderr>:  File "/my-spack/view/chifflot-p100/lib/python3.10/site-packages/torchvision/io/_load_gpu_decoder.py", line 1, in <module>
[1,0]<stderr>:    from ..extension import _load_library
[1,0]<stderr>:  File "/my-spack/view/chifflot-p100/lib/python3.10/site-packages/torchvision/extension.py", line 107, in <module>
[1,0]<stderr>:    _check_cuda_version()
[1,0]<stderr>:  File "/my-spack/view/chifflot-p100/lib/python3.10/site-packages/torchvision/extension.py", line 80, in _check_cuda_version
[1,0]<stderr>:    raise RuntimeError(
[1,0]<stderr>:RuntimeError: Detected that PyTorch and torchvision were compiled with different CUDA versions. PyTorch has CUDA Version=11.7 and torchvision has CUDA Version=11.8. Please reinstall the torchvision that matches your PyTorch install.
</pre>

Some information on my system:

* **Spack:** 0.21.0.dev0
* **Python:** 3.10.13
* **Platform:** linux-debian11-skylake_avx512
* **Concretizer:** clingo



Discussion:
Can you run `spack find -v` in your environment? Specifically interested in how it claims to have built PyTorch and torchvision.
Here is the output of `spack find -v`:

Interesting lines:

```
py-torch@2.0.1~caffe2+cuda+cudnn~debug+distributed+fbgemm~gloo+kineto+magma~metal+mkldnn+mpi+nccl+nnpack+numa+numpy+onnx_ml+openmp+qnnpack~rocm+tensorpipe~test+valgrind+xnnpack build_system=python_pip cuda_arch=60 patches=65060b5,6730561
py-torchvision@0.15.2~ffmpeg~jpeg~nvjpeg~png~video_codec build_system=python_pip
```

I don't know what to conclude from these.

```
==> In environment /home/tbouvier/spack-lille/spack-envs/chifflot/p100
==> Root specs
cuda
cudnn@8.4
mochi-thallium
mpi
nano
nccl
nccl-tests
nvtx +python
py-bokeh
py-continuum
py-filelock
py-horovod@0.28  controllers=mpi frameworks=pytorch
py-matplotlib
py-nvidia-dali@1.27.0-cuda110
py-pip
py-pybind11
py-ruamel-yaml
py-timm
py-torch@2.0.1 ~gloo
py-torchsummary
py-wandb
python@3.10

==> Installed packages
-- linux-debian11-skylake_avx512 / gcc@10.4.0 -------------------
alsa-lib@1.2.3.2~python build_system=autotools
argobots@1.1~affinity~debug~lazy_stack_alloc+perf~stackunwind~tool~valgrind build_system=autotools stackguard=none
arrow@10.0.1~brotli~compute~cuda~gandiva~glog~hdfs+ipc~ipo~jemalloc~lz4~orc+parquet+python+shared~snappy~tensorflow~zlib~zstd build_system=cmake build_type=Release generator=make
autoconf@2.69 build_system=autotools patches=35c4492,7793209,a49dd5b
automake@1.16.5 build_system=autotools
berkeley-db@18.1.40+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc
bison@3.8.2 build_system=autotools
bison@3.8.2 build_system=autotools
boost@1.82.0+atomic+chrono~clanglibcpp~container~context~contract~coroutine+date_time~debug+exception~fiber+filesystem+graph~graph_parallel~icu+iostreams~json+locale+log+math~mpi+multithreaded~nowide~numpy~pic+program_options~python+random+regex+serialization+shared+signals~singlethreaded~stacktrace+system~taggedlayout+test+thread+timer~type_erasure~versionedlayout+wave build_system=generic cxxstd=98 patches=a440f96,a7c807f visibility=hidden
bzip2@1.0.8~debug~pic+shared build_system=generic
c-ares@1.15.0~ipo build_system=cmake build_type=Release generator=make
ca-certificates-mozilla@2023-05-30 build_system=generic
cereal@1.3.2~ipo build_system=cmake build_type=Release generator=make patches=2dfa0bf
cmake@3.26.3~doc+ncurses+ownlibs~qt build_system=generic build_type=Release
cmake@3.27.4~doc+ncurses+ownlibs build_system=generic build_type=Release
coreutils@9.3~gprefix build_system=autotools
cuda@11.8.0~allow-unsupported-compilers~dev build_system=generic
cudnn@8.4.0.27-11.6 build_system=generic
curl@8.1.2~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl
curl@8.1.2~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl
diffutils@3.9 build_system=autotools
eigen@3.4.0~ipo build_system=cmake build_type=RelWithDebInfo generator=make
expat@2.5.0+libbsd build_system=autotools
ffmpeg@5.1.2~X+bzlib~drawtext+gpl~libaom~libmp3lame~libopenjpeg~libopus~libsnappy~libspeex~libssh~libvorbis~libvpx~libwebp~libx264~libxml2~libzmq~lzma~nonfree~openssl~sdl2+shared+version3 build_system=autotools patches=967d25a
findutils@4.9.0 build_system=autotools patches=440b954
flatbuffers@2.0.6~ipo~python+shared build_system=cmake build_type=Release generator=make
flex@2.6.3+lex~nls build_system=autotools
flex@2.6.3+lex~nls build_system=autotools
freetype@2.11.1+pic+shared build_system=autotools
fxdiv@2020-04-17~ipo build_system=cmake build_type=Release generator=ninja
gdbm@1.23 build_system=autotools
gettext@0.21.1+bzip2+curses+git~libunistring+libxml2+tar+xz build_system=autotools
gettext@0.21.1+bzip2+curses+git~libunistring+libxml2+tar+xz build_system=autotools
git@2.41.0+man+nls+perl+subtree~svn~tcltk build_system=autotools
gmake@4.4.1~guile build_system=autotools
gmake@4.4.1~guile build_system=autotools
hdf5@1.14.2~cxx~fortran+hl~ipo~java~map+mpi+shared~szip~threadsafe+tools api=default build_system=cmake build_type=Release generator=make
hwloc@2.9.1~cairo~cuda~gl~libudev+libxml2~netloc~nvml~oneapi-level-zero~opencl+pci~rocm build_system=autotools libs=shared,static
intel-oneapi-mkl@2023.2.0~cluster+envmods~ilp64+shared build_system=generic threads=none
intel-tbb@2021.9.0~ipo+shared+tm build_system=cmake build_type=Release cxxstd=default generator=make patches=12a9c3e
json-c@0.16~ipo build_system=cmake build_type=Release generator=make
keyutils@1.6.1 build_system=makefile
krb5@1.20.1+shared build_system=autotools
libbsd@0.11.7 build_system=autotools
libedit@3.1-20210216 build_system=autotools
libevent@2.1.12+openssl build_system=autotools
libfabric@1.19.0+cuda+debug~disable-spinlocks~kdreg build_system=autotools fabrics=rxm,tcp,verbs
libffi@3.4.4 build_system=autotools
libgit2@1.6.4~curl~ipo+mmap+ssh build_system=cmake build_type=Release generator=make https=system
libiconv@1.17 build_system=autotools libs=shared,static
libidn2@2.3.4 build_system=autotools
libjpeg-turbo@3.0.0~ipo~jpeg8+shared+static build_system=cmake build_type=Release generator=make
libmd@1.0.4 build_system=autotools
libnl@3.3.0 build_system=autotools
libpciaccess@0.17 build_system=autotools
libpng@1.6.39~ipo build_system=cmake build_type=Release generator=make libs=shared,static
libsigsegv@2.14 build_system=autotools
libssh2@1.11.0+shared build_system=autotools crypto=openssl patches=011d926
libtool@2.4.7 build_system=autotools
libunistring@1.1 build_system=autotools
libxcrypt@4.4.35~obsolete_api build_system=autotools patches=4885da3
libxml2@2.10.3~python build_system=autotools
libxml2@2.10.3~python build_system=autotools
libyaml@0.2.5 build_system=autotools
m4@1.4.19+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7
magma@2.7.2+cuda+fortran~ipo~rocm+shared build_system=cmake build_type=Release cuda_arch=60 generator=make
mercury@2.3.0~bmi+boostsys+checksum+debug~hwloc~ipo~mpi+ofi~psm~psm2+shared+sm~ucx~udreg build_system=cmake build_type=Release generator=make
meson@1.2.2 build_system=python_pip patches=0f0b1bd,ae59765
mochi-margo@0.14.1~pvar build_system=autotools
mochi-thallium@0.11.1+cereal~ipo build_system=cmake build_type=Release generator=make
nano@7.2 build_system=autotools
nasm@2.15.05 build_system=autotools
nccl@2.18.5-1+cuda build_system=makefile cuda_arch=60
nccl-tests@2.13.6+cuda+mpi build_system=makefile cuda_arch=60
ncurses@6.4~symlinks+termlib abi=none build_system=autotools
nghttp2@1.52.0 build_system=autotools
ninja@1.11.1+re2c build_system=generic
numactl@2.0.14 build_system=autotools patches=4e1d78c,62fc8a8,ff37630
nvtx@3.1.0+python build_system=generic patches=8f82f00
openmpi@4.1.5~atomics~cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~internal-pmix~java~legacylaunchers~lustre~memchecker~openshmem~orterunprefix+romio+rsh~singularity+static+vt+wrapper-rpath build_system=autotools fabrics=none schedulers=none
openssh@9.3p1+gssapi build_system=autotools
openssl@3.1.2~docs~shared build_system=generic certs=mozilla
openssl@3.1.2~docs+shared build_system=generic certs=mozilla
pcre@8.45~jit+multibyte+utf build_system=autotools
pcre2@10.42~jit+multibyte build_system=autotools
perl@5.38.0+cpanm+opcode+open+shared+threads build_system=generic patches=714e4d1
perl@5.38.0+cpanm+opcode+open+shared+threads build_system=generic
pigz@2.7 build_system=makefile
pigz@2.7 build_system=makefile
pkgconf@1.9.5 build_system=autotools
pmix@4.2.4~docs+pmi_backwards_compatibility~python~restful build_system=autotools
protobuf@3.20.3~ipo+shared build_system=cmake build_type=Release generator=make
psimd@2020-05-17~ipo build_system=cmake build_type=Release generator=ninja
pthreadpool@2021-04-13~ipo build_system=cmake build_type=Release generator=ninja
py-absl-py@1.4.0 build_system=python_pip
py-aiohttp@3.8.4 build_system=python_pip
py-aiosignal@1.2.0 build_system=python_pip
py-appdirs@1.4.4 build_system=python_pip patches=006d203
py-astunparse@1.6.3 build_system=python_pip
py-async-timeout@4.0.2 build_system=python_pip
py-attrs@23.1.0 build_system=python_pip
py-beniget@0.4.1 build_system=python_pip
py-bokeh@2.4.3 build_system=python_pip
py-bottleneck@1.3.7 build_system=python_pip
py-build@1.0.3~virtualenv build_system=python_pip
py-cachetools@5.2.0 build_system=python_pip
py-calver@2022.6.26 build_system=python_pip
py-certifi@2023.5.7 build_system=python_pip
py-cffi@1.15.1 build_system=python_pip
py-charset-normalizer@3.1.0 build_system=python_pip
py-click@8.1.3 build_system=python_pip
py-cloudpickle@2.2.0 build_system=python_pip
py-continuum@1.2.7 build_system=python_pip patches=3a6b6fc
py-contourpy@1.0.7 build_system=python_pip
py-cppy@1.2.1 build_system=python_pip
py-cycler@0.11.0 build_system=python_pip
py-cython@0.29.36 build_system=python_pip patches=c4369ad
py-datasets@2.8.0 build_system=python_pip
py-dill@0.3.6 build_system=python_pip patches=daf79b1
py-dm-tree@0.1.8 build_system=python_pip
py-dockerpy-creds@0.4.0 build_system=python_pip
py-docutils@0.20.1 build_system=python_pip
py-editables@0.3 build_system=python_pip
py-exceptiongroup@1.1.1 build_system=python_pip
py-filelock@3.12.4 build_system=python_pip
py-flit-core@3.9.0 build_system=python_pip
py-flit-scm@1.7.0 build_system=python_pip
py-fonttools@4.39.4 build_system=python_pip
py-frozenlist@1.3.1 build_system=python_pip
py-fsspec@2023.1.0+http build_system=python_pip
py-future@0.18.3 build_system=python_pip
py-gast@0.5.3 build_system=python_pip
py-gitdb@4.0.9 build_system=python_pip
py-gitpython@3.1.27 build_system=python_pip
py-google-auth@2.20.0~aiohttp build_system=python_pip
py-google-auth-oauthlib@0.5.2 build_system=python_pip
py-grpcio@1.52.0 build_system=python_pip
py-h5py@3.8.0+mpi build_system=python_pip
py-hatch-fancy-pypi-readme@23.1.0 build_system=python_pip
py-hatch-vcs@0.3.0 build_system=python_pip
py-hatchling@1.18.0 build_system=python_pip
py-horovod@0.28.1+cuda~rocm build_system=python_pip controllers=mpi cuda_arch=60 frameworks=pytorch tensor_ops=nccl
py-huggingface-hub@0.14.1~cli build_system=python_pip
py-idna@3.4 build_system=python_pip
py-imagehash@4.3.1 build_system=python_pip
py-imageio@2.30.0 build_system=python_pip
py-iniconfig@2.0.0 build_system=python_pip
py-jinja2@3.1.2~i18n build_system=python_pip
py-joblib@1.2.0 build_system=python_pip
py-kerberos@1.3.0 build_system=python_pip
py-kiwisolver@1.4.4 build_system=python_pip
py-lazy-loader@0.1 build_system=python_pip
py-lightning-utilities@0.8.0 build_system=python_pip
py-markdown@3.4.1 build_system=python_pip
py-markupsafe@2.1.3 build_system=python_pip
py-matplotlib@3.8.0~animation~fonts~latex~movies backend=agg build_system=python_pip
py-meson-python@0.13.1 build_system=python_pip
py-mpi4py@3.1.4 build_system=python_pip
py-mpmath@1.2.1 build_system=python_pip
py-multidict@6.0.4 build_system=python_pip
py-multiprocess@0.70.12.2 build_system=python_pip
py-networkx@3.1+default~extra build_system=python_pip
py-numexpr@2.8.4 build_system=python_pip
py-numpy@1.25.2+blas+lapack build_system=python_pip patches=873745d
py-nvidia-dali@1.27.0-cuda110 build_system=python_pip
py-oauthlib@3.2.1~rsa~signals~signedtoken build_system=python_pip
py-packaging@23.1 build_system=python_pip
py-pandas@1.5.3 build_system=python_pip
py-pathspec@0.11.1 build_system=python_pip
py-pathtools@0.1.2 build_system=python_pip
py-pillow@10.0.0~freetype~imagequant+jpeg~jpeg2000~lcms~raqm~tiff~webp~xcb+zlib build_system=python_pip
py-pip@23.0 build_system=generic
py-pkgconfig@1.5.5 build_system=python_pip
py-pluggy@1.0.0 build_system=python_pip
py-ply@3.11 build_system=python_pip
py-poetry-core@1.6.1 build_system=python_pip
py-protobuf@3.20.3~cpp build_system=python_pip
py-psutil@5.9.5 build_system=python_pip
py-pure-sasl@0.6.2+gssapi build_system=python_pip
py-pyarrow@10.0.1~cuda~dataset~orc+parquet build_system=python_pip
py-pyasn1@0.4.8 build_system=python_pip
py-pyasn1-modules@0.2.8 build_system=python_pip
py-pybind11@2.10.1~ipo build_system=cmake build_type=Release generator=ninja
py-pycparser@2.21 build_system=python_pip
py-pydeprecate@0.3.1 build_system=python_pip
py-pyparsing@3.0.9 build_system=python_pip
py-pyproject-hooks@1.0.0 build_system=python_pip
py-pyproject-metadata@0.7.1 build_system=python_pip
py-pytest@7.3.2 build_system=python_pip
py-pytest-mock@3.10.0 build_system=python_pip
py-python-dateutil@2.8.2 build_system=python_pip
py-pythran@0.12.2 build_system=python_pip
py-pytorch-lightning@1.5.3 build_system=python_pip
py-pytz@2023.3 build_system=python_pip
py-pywavelets@1.4.1 build_system=python_pip
py-pyyaml@6.0+libyaml build_system=python_pip
py-requests@2.31.0~socks build_system=python_pip
py-requests-oauthlib@1.3.1 build_system=python_pip
py-responses@0.13.3 build_system=python_pip
py-rsa@4.9 build_system=python_pip
py-ruamel-yaml@0.17.32 build_system=python_pip
py-ruamel-yaml-clib@0.2.7 build_system=python_pip
py-safetensors@0.3.1 build_system=python_pip
py-scikit-image@0.20.0 build_system=python_pip
py-scikit-learn@1.3.1+openmp build_system=python_pip
py-scipy@1.9.3 build_system=python_pip
py-semantic-version@2.10.0 build_system=python_pip
py-sentry-sdk@1.5.5~aiohttp~beam~bottle~celery~chalice~django~falcon~flask~httpx~pure_eval~pyspark~quart~rq~sanic~sqlalchemy~tornado build_system=python_pip
py-setproctitle@1.1.10 build_system=python_pip
py-setuptools@63.4.3 build_system=python_pip
py-setuptools@68.0.0 build_system=python_pip
py-setuptools-rust@1.6.0 build_system=python_pip
py-setuptools-scm@7.1.0+toml build_system=python_pip
py-six@1.16.0 build_system=python_pip
py-smmap@5.0.0 build_system=python_pip
py-sympy@1.11.1 build_system=python_pip
py-tensorboard@2.13.0 build_system=python_pip
py-tensorboard-data-server@0.7.0 build_system=python_pip
py-tensorboard-plugin-wit@1.8.1 build_system=python_pip
py-threadpoolctl@3.1.0 build_system=python_pip
py-tifffile@2023.8.30 build_system=python_pip
py-timm@0.9.5 build_system=python_pip
py-tomli@2.0.1 build_system=python_pip
py-torch@2.0.1~caffe2+cuda+cudnn~debug+distributed+fbgemm~gloo+kineto+magma~metal+mkldnn+mpi+nccl+nnpack+numa+numpy+onnx_ml+openmp+qnnpack~rocm+tensorpipe~test+valgrind+xnnpack build_system=python_pip cuda_arch=60 patches=65060b5,6730561
py-torchmetrics@1.2.0 build_system=python_pip
py-torchsummary@1.5.1 build_system=python_pip
py-torchvision@0.15.2~ffmpeg~jpeg~nvjpeg~png~video_codec build_system=python_pip
py-tornado@6.3.3 build_system=python_pip
py-tqdm@4.66.1~notebook~telegram build_system=python_pip
py-trove-classifiers@2023.8.7 build_system=python_pip
py-twisted@15.4.0 build_system=python_pip
py-typing-extensions@4.8.0 build_system=python_pip
py-urllib3@1.26.12~brotli~secure~socks build_system=python_pip
py-versioneer@0.29+toml build_system=python_pip
py-wandb@0.13.9 build_system=python_pip
py-werkzeug@3.0.0 build_system=python_pip
py-wheel@0.37.1 build_system=generic
py-xxhash@3.2.0 build_system=python_pip
py-yarl@1.9.2 build_system=python_pip
py-zope-interface@5.4.0 build_system=python_pip
python@3.10.13+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=0d98e93,7d40923,ebdca64,f2fd060
qhull@2020.2~ipo build_system=cmake build_type=Release generator=make
rapidjson@1.2.0-2022-03-09~doc~ipo build_system=cmake build_type=Release generator=make patches=a4316c5
rdma-core@41.0~ipo+static build_system=cmake build_type=Release generator=make
re2@2021-06-01~ipo+pic+shared build_system=cmake build_type=Release generator=make
re2c@2.2 build_system=generic
readline@8.2 build_system=autotools patches=bbf97f1
rust@1.70.0~analysis+clippy~docs+rustfmt+src build_system=generic
rust-bootstrap@1.70.0 build_system=generic
scons@4.5.2 build_system=python_pip
snappy@1.1.10~ipo+pic~shared build_system=cmake build_type=Release generator=make
sqlite@3.42.0+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools
tar@1.34 build_system=autotools zip=pigz
tar@1.34 build_system=autotools zip=pigz
thrift@0.18.1+c~java+pic+python build_system=generic
utf8proc@2.8.0~ipo+shared build_system=cmake build_type=Release generator=make
util-linux-uuid@2.38.1 build_system=autotools
util-macros@1.19.3 build_system=autotools
valgrind@3.20.0+boost+mpi+only64bit~ubsan build_system=autotools libs=shared,static
xsimd@8.1.0~ipo build_system=cmake build_type=Release generator=make
xxhash@0.8.1 build_system=makefile
xz@5.4.1~pic build_system=autotools libs=shared,static
yasm@1.3.0 build_system=autotools
zlib@1.2.13+optimize+pic+shared build_system=makefile
zlib-ng@2.1.3+compat+opt build_system=autotools patches=299b958,ae9077a,b692621
zstd@1.5.5+programs build_system=makefile compression=none libs=shared,static
```

A side question: would you know why some are entries duplicated in my output?
> would you know why some are entries duplicated in my output?

Build-only deps are now handled separately by the concretizer to make things easier for complex environments.

Can you try the following code?
```python
import torch

print(torch.version.cuda)
print(torch.ops.torchvision._cuda_version())
```
This should confirm that the versions are different. Then we just have to figure out why...

Can you also upload the `spack-build-out.txt` and `spack-build-env-mods.txt` files found in `<prefix>/.spack` for the PyTorch and torchvision installation directories? Use `spack find -p py-torch` to find them.
I tried the code and got a weird result:

```
11.7
Traceback (most recent call last):
  File "/my-spack/view/chifflot-p100/lib/python3.10/site-packages/torch/_ops.py", line 565, in __getattr__
    op, overload_names = torch._C._jit_get_operation(qualified_op_name)
RuntimeError: No such operator torchvision::_cuda_version

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/distributed-continual-learning/test.py", line 4, in <module>
    print(torch.ops.torchvision._cuda_version())
  File "/my-spack/view/chifflot-p100/lib/python3.10/site-packages/torch/_ops.py", line 569, in __getattr__
    raise AttributeError(
AttributeError: '_OpNamespace' 'torchvision' object has no attribute '_cuda_version'
```

My installation seems to be broken for some reason.

torch logs
[spack-build-out.txt](https://github.com/spack/spack/files/13073041/spack-build-out.txt)
[spack-build-env.txt](https://github.com/spack/spack/files/13073044/spack-build-env.txt)

torch says:
```
  -- Found CUDA: /grid5000/spack/v1/opt/spack/linux-debian11-x86_64_v2/gcc-10.4.0/cuda-11.7.1-mniz3r7mpcnkeb2ekh7kcscflfid42om (found version "11.7")
```

Which is wrong. This is referring to an external CUDA with version 11.7 instead of what's in my env. So this is probably the issue. I used `spack concretize --force` and `spack install --fresh`, aren't these commands enough to make sure that the env will be installed as defined in the yaml file? I thought so because I can see `cuda-11.8.0` in both outputs. There is probably something cached somewhere that I am not aware of.

torchvision logs
[spack-build-out.txt](https://github.com/spack/spack/files/13073036/spack-build-out.txt)
[spack-build-env.txt](https://github.com/spack/spack/files/13073034/spack-build-env.txt)

torchvision says:
```
  /my-spack/spack/linux-debian11-skylake_avx512/gcc-10.4.0/py-torch-2.0.1-vl52lzsvezacxv5244yohoarwrlia7as/lib/python3.10/site-packages/torch/utils/cpp_extension.py:388: UserWarning: The detected CUDA version (11.8) has a minor version mismatch with the version that was used to compile PyTorch (11.7). Most likely this shouldn't be a problem.
```

I am confident that forcing `cuda@11.8` in my env will solve my issue, however I am interested in understanding where my external cuda could be cached.
Odd, maybe it's seeing `/grid5000/spack/v1/opt/spack/linux-debian11-x86_64_v2/gcc-10.4.0/cuda-11.7.1-mniz3r7mpcnkeb2ekh7kcscflfid42om` in your `PATH`? But it comes after 11.8.0 and it isn't in `CMAKE_PREFIX_PATH`. But it somehow gets added to `-DCMAKE_PREFIX_PATH` somewhere... I don't know anything about CMake, but it's something to do with how CMake detects CUDA.

Some ideas worth trying:

1. Remove the `cmake_prefix_paths` property from `py-torch`
2. Try setting one of the following env vars: `CUDA_HOME` (Linux/macOS), `CUDA_PATH` (Windows), `CUDA_ROOT`, `CUDA_TOOLKIT_ROOT_DIR`, `CUDA_NVCC_EXECUTABLE`
Removing `cmake_prefix_paths` didn't help.

I tried with PyTorch version 2.1: it doesn't even compile anymore, but it catches my error early:

```
  CMake Error at cmake/public/cuda.cmake:65 (message):
    Found two conflicting CUDA installs:                                                                                                          
    V11.8.89 in
    '/grid5000/spack/v1/opt/spack/linux-debian11-x86_64_v2/gcc-10.4.0/cuda-11.7.1-mniz3r7mpcnkeb2ekh7kcscflfid42om/include'
    and
                                                                                                                                           
    V11.8.89 in
    '/my-spack/spack/chifflot-p100/linux-debian11-skylake_avx512/gcc-10.4.0/cuda-11.8.0-my6q3bskrb5y6cjgvq554cqok3vrwxyp/include'
  Call Stack (most recent call first):
    cmake/Dependencies.cmake:44 (include)
    CMakeLists.txt:722 (include)

```

[spack-build-out.txt](https://github.com/spack/spack/files/13244343/spack-build-out.txt)
[spack-build-env-mods.txt](https://github.com/spack/spack/files/13244615/spack-build-env-mods.txt)

For some reason CMake is picking up the CUDA install in `'/grid5000/spack/v1/opt/spack/linux-debian11-x86_64_v2/gcc-10.4.0/`. 

I'd like to try setting `CUDA_HOME`. How should I do that?
> I'd like to try setting CUDA_HOME. How should I do that?

Run `spack edit py-torch` to open the package recipe, and then in the `setup_build_environment` function add a line like:
```python
env.set("CUDA_HOME", self.spec["cuda"].prefix)
```
Same for the other env vars if that one doesn't work.
Setting the variable `CUDA_HOME` fixed my issue. To my understanding, this issue is related to PyTorch (to its CMake recipe specifically), not Spack?
Great! This is something I can fix in Spack's build recipe. I'll submit a PR.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/py-torch/package.py b/var/spack/repos/builtin/packages/py-torch/package.py
--- a/var/spack/repos/builtin/packages/py-torch/package.py
+++ b/var/spack/repos/builtin/packages/py-torch/package.py
@@ -498,9 +498,8 @@ def enable_or_disable(variant, keyword="USE", var=None, newer=False):
 
         enable_or_disable("cuda")
         if "+cuda" in self.spec:
-            # cmake/public/cuda.cmake
-            # cmake/Modules_CUDA_fix/upstream/FindCUDA.cmake
-            env.unset("CUDA_ROOT")
+            env.set("CUDA_HOME", self.spec["cuda"].prefix)  # Linux/macOS
+            env.set("CUDA_PATH", self.spec["cuda"].prefix)  # Windows
             torch_cuda_arch = ";".join(
                 "{0:.1f}".format(float(i) / 10.0) for i in self.spec.variants["cuda_arch"].value
             )


Test patch: