Nccl_external fails while trying to compile pytroch from source

chanaka: <p>Hello, I’m trying to compile <code>pytorch</code> from source and encountering the following build error.</p>
<pre><code class="lang-auto">$ CC=gcc-10 CXX=g++-10 python setup.py develop
...
[5995/6841] Linking CXX executable bin/HashStoreTest
Warning: Unused direct dependencies:                                                                                                                                                                                                                                                  
/home/netfpga/research/collective/pytorch/build/lib/libc10.so                                                                                                                                                                                                                 
/home/netfpga/anaconda3/envs/pytorch_base/lib/libmkl_intel_lp64.so.1                                                                                                                                                                                                          
/home/netfpga/anaconda3/envs/pytorch_base/lib/libmkl_gnu_thread.so.1                                                                                                                                                                                                          
/home/netfpga/anaconda3/envs/pytorch_base/lib/libmkl_core.so.1                                                                                                                                                                                                                
/lib/x86_64-linux-gnu/libdl.so.2                                                                                                                                                                                                                                              
/home/netfpga/anaconda3/envs/pytorch_base/lib/libgomp.so.1                                                                                                                                                                                                            
[5996/6841] Performing build step for 'nccl_external'                                                                                                                                                                                                                         
FAILED: nccl_external-prefix/src/nccl_external-stamp/nccl_external-build nccl/lib/libnccl_static.a /home/netfpga/research/collective/pytorch/build/nccl_external-prefix/src/nccl_external-stamp/nccl_external-build /home/netfpga/research/collective/pytorch/build/nccl/lib/libnccl_static.a                                                                                                                                                                                                                                                               
cd /home/netfpga/research/collective/pytorch/third_party/nccl/nccl &amp;&amp; make -j24 -l24 CXX=/usr/bin/g++-10 CUDA_HOME=/home/netfpga/anaconda3/envs/pytorch_base NVCC=/home/netfpga/anaconda3/envs/pytorch_base/bin/nvcc "NVCC_GENCODE=-gencode=arch=compute_35,code=sm_35 -gencode=arch=compute_50,code=sm_50 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=sm_86 -gencode=arch=compute_86,code=compute_86" 
BUILDDIR=/home/netfpga/research/collective/pytorch/build/nccl VERBOSE=0 DEBUG=0 &amp;&amp; /home/netfpga/.local/lib/python3.8/site-packages/cmake/data/bin/cmake -E touch /home/netfpga/research/collective/pytorch/build/nccl_external-prefix/src/nccl_external-
stamp/nccl_external-build                                                                                                                           
make -C src build BUILDDIR=/home/netfpga/research/collective/pytorch/build/nccl
make[1]: Entering directory '/home/netfpga/research/collective/pytorch/third_party/nccl/nccl/src'                                                                                                                                                                             
NVCC_GENCODE is -gencode=arch=compute_35,code=sm_35 -gencode=arch=compute_50,code=sm_50 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=sm_86 -gencode=arch=compute_86,code=compute_86
...
...
Compiling /home/netfpga/research/collective/pytorch/build/nccl/obj/device/gensrc/reduce_prod_f32.cu
[264/1934]nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
Compiling       /home/netfpga/research/collective/pytorch/build/nccl/obj/device/gensrc/reduce_prod_f64.cu                                                                                                                                                                     nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).                                                         
Compiling       /home/netfpga/research/collective/pytorch/build/nccl/obj/device/gensrc/reduce_prod_u32.cu
nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).                                                        
Compiling       /home/netfpga/research/collective/pytorch/build/nccl/obj/device/gensrc/reduce_prod_u64.cu                                                                                                                                                                     nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).                                                         
Compiling       /home/netfpga/research/collective/pytorch/build/nccl/obj/device/gensrc/reduce_prod_u8.cu                                                                                                                                                                      nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).                                                         
nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).                                                         
make[2]: Leaving directory '/home/netfpga/research/collective/pytorch/third_party/nccl/nccl/src/device'                                                                                                                                                                       

Linking    libnccl.so.2.19.3  &gt; /home/netfpga/research/collective/pytorch/build/nccl/lib/libnccl.so.2.19.3
Archiving  libnccl_static.a                    &gt; 
/home/netfpga/research/collective/pytorch/build/nccl/lib/libnccl_static.a                                                                                                                                                    
/usr/bin/ld: cannot find -lcudart_static                                                                                                                                                                                                                                      
collect2: error: ld returned 1 exit status                                                                                                                                                                                                                                    
make[1]: *** [Makefile:79: /home/netfpga/research/collective/pytorch/build/nccl/lib/libnccl.so.2.19.3] Error 1                                                                                                                                                                
make[1]: *** Waiting for unfinished jobs....                                                                                                                                                                                                                                  
make[1]: Leaving directory '/home/netfpga/research/collective/pytorch/third_party/nccl/nccl/src'                                                                                                                                                                              
make: *** [Makefile:25: src.build] Error 2                                                                                                                                                                                                                                    
ninja: build stopped: subcommand failed.
</code></pre>
<p><a href="https://drive.google.com/file/d/1wXD9GOhHIolUoM7hdeDbR6urnlYC6cx9/view?usp=sharing" rel="noopener nofollow ugc">Access full log here</a>. In case it’s helpful.</p>
<p>The main error is:</p>
<pre><code class="lang-auto">[5996/6841] Performing build step for 'nccl_external'                                                                                                                                                                                                                         
FAILED: nccl_external-prefix/src/nccl_external-stamp/nccl_external-build nccl/lib/libnccl_static.a /home/netfpga/research/collective/pytorch/build/nccl_external-prefix/src/nccl_external-stamp/nccl_external-build /home/netfpga/research/collective/pytorch/build/nccl/lib/libnccl_static.a 
</code></pre>
<p>I’m on a tesla K80, so:</p>
<pre><code class="lang-auto">$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Wed_Jun__9_20:32:38_PDT_2021
Cuda compilation tools, release 11.3, V11.3.122
Build cuda_11.3.r11.3/compiler.30059648_0
</code></pre>
<pre><code class="lang-auto">$ nvidia-smi
Fri Nov 17 00:48:09 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.223.02   Driver Version: 470.223.02   CUDA Version: 12.3     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 00000000:04:00.0 Off |                    0 |
| N/A   38C    P8    26W / 149W |      4MiB / 11441MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K80           Off  | 00000000:05:00.0 Off |                    0 |
| N/A   32C    P8    30W / 149W |      4MiB / 11441MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  Tesla K80           Off  | 00000000:08:00.0 Off |                    0 |
| N/A   32C    P8    26W / 149W |      4MiB / 11441MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  Tesla K80           Off  | 00000000:09:00.0 Off |                    0 |
| N/A   25C    P8    30W / 149W |      4MiB / 11441MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   4  Tesla K80           Off  | 00000000:84:00.0 Off |                    0 |
| N/A   31C    P8    26W / 149W |      4MiB / 11441MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   5  Tesla K80           Off  | 00000000:85:00.0 Off |                    0 |
| N/A   24C    P8    31W / 149W |      4MiB / 11441MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   6  Tesla K80           Off  | 00000000:88:00.0 Off |                    0 |
| N/A   31C    P8    26W / 149W |      4MiB / 11441MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   7  Tesla K80           Off  | 00000000:89:00.0 Off |                    0 |
| N/A   25C    P8    29W / 149W |      4MiB / 11441MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      2079      G   /usr/lib/xorg/Xorg                  3MiB |
|    1   N/A  N/A      2079      G   /usr/lib/xorg/Xorg                  3MiB |
|    2   N/A  N/A      2079      G   /usr/lib/xorg/Xorg                  3MiB |
|    3   N/A  N/A      2079      G   /usr/lib/xorg/Xorg                  3MiB |
|    4   N/A  N/A      2079      G   /usr/lib/xorg/Xorg                  3MiB |
|    5   N/A  N/A      2079      G   /usr/lib/xorg/Xorg                  3MiB |
|    6   N/A  N/A      2079      G   /usr/lib/xorg/Xorg                  3MiB |
|    7   N/A  N/A      2079      G   /usr/lib/xorg/Xorg                  3MiB |
+-----------------------------------------------------------------------------+
</code></pre>
<p>Though <code>nvidia-smi</code> shows CUDA as 12.3, I’m on 11.3.1, which is the supported version for K80. The conda environment is:</p>
<pre><code class="lang-auto">$ conda list
# packages in environment at /home/netfpga/anaconda3/envs/pytorch_base:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main
_openmp_mutex             5.1                       1_gnu
anaconda-client           1.11.2          py310h06a4308_0
anaconda-navigator        2.4.1           py310h06a4308_0
anaconda-project          0.11.1          py310h06a4308_0
asttokens                 2.2.1                    pypi_0    pypi
astunparse                1.6.3                    pypi_0    pypi
attrs                     22.1.0          py310h06a4308_0
backcall                  0.2.0                    pypi_0    pypi
backports                 1.1                pyhd3eb1b0_0
backports.functools_lru_cache 1.6.4              pyhd3eb1b0_0
backports.tempfile        1.0                pyhd3eb1b0_1
backports.weakref         1.0.post1                  py_1
beautifulsoup4            4.12.2          py310h06a4308_0
boltons                   23.0.0          py310h06a4308_0
brotli-python             1.0.9           py310h6a678d5_7
brotlipy                  0.7.0           py310h7f8727e_1002
bzip2                     1.0.8                h7b6447c_0
c-ares                    1.19.0               h5eee18b_0
ca-certificates           2023.08.22           h06a4308_0
certifi                   2022.12.7                pypi_0    pypi
cffi                      1.15.1          py310h5eee18b_3
chardet                   4.0.0           py310h06a4308_1003
charset-normalizer        3.1.0                    pypi_0    pypi
click                     8.0.4           py310h06a4308_0
clyent                    1.2.2           py310h06a4308_1
cmake                     3.26.3                   pypi_0    pypi
conda                     23.5.0          py310h06a4308_0
conda-build               3.24.0          py310h06a4308_0
conda-content-trust       0.1.3           py310h06a4308_0
conda-pack                0.6.0              pyhd3eb1b0_0
conda-package-handling    2.0.2           py310h06a4308_0
conda-package-streaming   0.7.0           py310h06a4308_0
conda-repo-cli            1.0.41          py310h06a4308_0
conda-token               0.4.0              pyhd3eb1b0_0
conda-verify              3.4.2                      py_1
cryptography              39.0.1          py310h9ce1e76_0
cuda-command-line-tools   11.3.1               h712c49d_0    nvidia/label/cuda-11.3.1
cuda-compiler             11.3.1               h712c49d_0    nvidia/label/cuda-11.3.1
cuda-cudart               11.3.109             hfb95d0c_0    nvidia/label/cuda-11.3.1
cuda-cuobjdump            11.3.122             hbf6ec6b_0    nvidia/label/cuda-11.3.1
cuda-cupti                11.3.111             h12ad217_0    nvidia/label/cuda-11.3.1
cuda-cuxxfilt             11.3.122             h4dc11a3_0    nvidia/label/cuda-11.3.1
cuda-gdb                  11.3.109             h33b7820_0    nvidia/label/cuda-11.3.1
cuda-libraries            11.3.1               h712c49d_0    nvidia/label/cuda-11.3.1
cuda-libraries-dev        11.3.1               h712c49d_0    nvidia/label/cuda-11.3.1
cuda-memcheck             11.3.109             hf5cb439_0    nvidia/label/cuda-11.3.1
cuda-nvcc                 11.3.122             h4814707_0    nvidia/label/cuda-11.3.1
cuda-nvdisasm             11.3.122             ha26faa6_0    nvidia/label/cuda-11.3.1
cuda-nvml-dev             11.3.58              hc25e488_0    nvidia/label/cuda-11.3.1
cuda-nvprof               11.3.111             h95a27d4_0    nvidia/label/cuda-11.3.1
cuda-nvprune              11.3.122             hb3346b8_0    nvidia/label/cuda-11.3.1
cuda-nvrtc                11.3.122             h1aa17d8_0    nvidia/label/cuda-11.3.1
cuda-nvtx                 11.3.109             h4ec7630_0    nvidia/label/cuda-11.3.1
cuda-nvvp                 11.3.111             h4c4416a_0    nvidia/label/cuda-11.3.1
cuda-samples              11.3.58              h6d5b628_0    nvidia/label/cuda-11.3.1
cuda-sanitizer-api        11.3.111             h2446cfc_0    nvidia/label/cuda-11.3.1
cuda-thrust               11.3.109             he8b717c_0    nvidia/label/cuda-11.3.1
cuda-toolkit              11.3.1               h712c49d_0    nvidia/label/cuda-11.3.1
cuda-tools                11.3.1               h712c49d_0    nvidia/label/cuda-11.3.1
cuda-visual-tools         11.3.1               h712c49d_0    nvidia/label/cuda-11.3.1
cudatoolkit               11.3.1               h2bc3f7f_2
cudnn                     8.2.1                cuda11.3_0
dbus                      1.13.18              hb2f20db_0
decorator                 5.1.1                    pypi_0    pypi
defusedxml                0.7.1              pyhd3eb1b0_0
exceptiongroup            1.1.1                    pypi_0    pypi
executing                 1.2.0                    pypi_0    pypi
expat                     2.4.9                h6a678d5_0
expecttest                0.1.4                    pypi_0    pypi
filelock                  3.12.0                   pypi_0    pypi
fontconfig                2.14.1               h4c34cd2_2
freetype                  2.12.1               h4a9f257_0
fsspec                    2023.6.0                 pypi_0    pypi
future                    0.18.3          py310h06a4308_0
gdb                       11.2            py310he3c4487_2
giflib                    5.2.1                h5eee18b_3
glib                      2.69.1               he621ea3_2
glob2                     0.7                pyhd3eb1b0_0
gst-plugins-base          1.14.1               h6a678d5_1
gstreamer                 1.14.1               h5eee18b_1
hypothesis                6.75.1                   pypi_0    pypi
icu                       58.2                 he6710b0_3
idna                      3.4             py310h06a4308_0
intel-openmp              2021.4.0          h06a4308_3561
ipython                   8.14.0                   pypi_0    pypi
jedi                      0.19.0                   pypi_0    pypi
jinja2                    3.1.2           py310h06a4308_0
jpeg                      9e                   h5eee18b_1
jsonpatch                 1.32               pyhd3eb1b0_0
jsonpointer               2.1                pyhd3eb1b0_0
jsonschema                4.17.3          py310h06a4308_0
jupyter_core              5.3.0           py310h06a4308_0
krb5                      1.19.4               h568e23c_0
lcms2                     2.12                 h3be6417_0
ld_impl_linux-64          2.38                 h1181459_1
lerc                      3.0                  h295c915_0
libarchive                3.6.2                hb4bd9a0_1
libclang                  14.0.6          default_hc6dbbc7_1
libclang13                14.0.6          default_he11475f_1
libcublas                 11.5.1.109           h0fd73e7_0    nvidia/label/cuda-11.3.1
libcufft                  10.4.2.109           h2344711_0    nvidia/label/cuda-11.3.1
libcurand                 10.2.4.109           h0189693_0    nvidia/label/cuda-11.3.1
libcurl                   7.88.1               h91b91d3_0
libcusolver               11.1.2.109           h1e009e5_0    nvidia/label/cuda-11.3.1
libcusparse               11.6.0.109           hf5bfba9_0    nvidia/label/cuda-11.3.1
libdeflate                1.17                 h5eee18b_0
libedit                   3.1.20221030         h5eee18b_0
libev                     4.33                 h7f8727e_1
libevent                  2.1.12               h8f2d780_0
libffi                    3.4.4                h6a678d5_0
libgcc-ng                 11.2.0               h1234567_1
libgomp                   11.2.0               h1234567_1
liblief                   0.12.3               h6a678d5_0
libllvm14                 14.0.6               hdb19cb5_3
libnghttp2                1.52.0               ha637b67_1
libnpp                    11.3.3.95            h122bb27_0    nvidia/label/cuda-11.3.1
libnvjpeg                 11.5.0.109           h159916b_0    nvidia/label/cuda-11.3.1
libpng                    1.6.39               h5eee18b_0
libpq                     12.9                 h16c4e8d_3
libssh2                   1.10.0               h37d81fd_2
libstdcxx-ng              11.2.0               h1234567_1
libtiff                   4.5.0                h6a678d5_2
libuuid                   1.41.5               h5eee18b_0
libuv                     1.44.2               h5eee18b_0
libwebp                   1.2.4                h11a3e52_1
libwebp-base              1.2.4                h5eee18b_1
libxcb                    1.15                 h7f8727e_0
libxkbcommon              1.0.1                h5eee18b_1
libxml2                   2.10.3               hcbfbd50_0
libxslt                   1.1.37               h2085143_0
lit                       16.0.1                   pypi_0    pypi
lz4-c                     1.9.4                h6a678d5_0
magma-cuda113             2.5.2                         1    pytorch
markupsafe                2.1.2                    pypi_0    pypi
matplotlib-inline         0.1.6                    pypi_0    pypi
mkl                       2021.4.0           h06a4308_640
mkl-include               2023.1.0         h06a4308_46344
mpmath                    1.3.0                    pypi_0    pypi
navigator-updater         0.4.0           py310h06a4308_0
nbformat                  5.7.0           py310h06a4308_0
ncurses                   6.4                  h6a678d5_0
networkx                  3.1                      pypi_0    pypi
ninja                     1.10.2               h06a4308_5
ninja-base                1.10.2               hd09550d_5
nspr                      4.35                 h6a678d5_0
nss                       3.89.1               h6a678d5_0
numpy                     1.24.2                   pypi_0    pypi
nvidia-cublas-cu11        11.10.3.66               pypi_0    pypi
nvidia-cuda-cupti-cu11    11.7.101                 pypi_0    pypi
nvidia-cuda-nvrtc-cu11    11.7.99                  pypi_0    pypi
nvidia-cuda-runtime-cu11  11.7.99                  pypi_0    pypi
nvidia-cuda-runtime-cu113 11.3.109                 pypi_0    pypi
nvidia-cudnn-cu11         8.5.0.96                 pypi_0    pypi
nvidia-cufft-cu11         10.9.0.58                pypi_0    pypi
nvidia-curand-cu11        10.2.10.91               pypi_0    pypi
nvidia-cusolver-cu11      11.4.0.1                 pypi_0    pypi
nvidia-cusparse-cu11      11.7.4.91                pypi_0    pypi
nvidia-ml-py3             7.352.0                  pypi_0    pypi
nvidia-nccl-cu11          2.14.3                   pypi_0    pypi
nvidia-nvtx-cu11          11.7.91                  pypi_0    pypi
nvidia-pyindex            1.0.9                    pypi_0    pypi
openssl                   1.1.1w               h7f8727e_0
optree                    0.10.0                   pypi_0    pypi
packaging                 23.0            py310h06a4308_0
parso                     0.8.3                    pypi_0    pypi
patch                     2.7.6             h7b6447c_1001
patchelf                  0.17.2               h6a678d5_0
pathlib                   1.0.1              pyhd3eb1b0_1
pcre                      8.45                 h295c915_0
pexpect                   4.8.0                    pypi_0    pypi
pickleshare               0.7.5                    pypi_0    pypi
pillow                    9.5.0                    pypi_0    pypi
pip                       22.3.1          py310h06a4308_0
pkginfo                   1.9.6           py310h06a4308_0
platformdirs              2.5.2           py310h06a4308_0
pluggy                    1.0.0           py310h06a4308_1
ply                       3.11            py310h06a4308_0
prompt-toolkit            3.0.39                   pypi_0    pypi
psutil                    5.9.0           py310h5eee18b_0
ptyprocess                0.7.0                    pypi_0    pypi
pure-eval                 0.2.2                    pypi_0    pypi
py-lief                   0.12.3          py310h6a678d5_0
pycosat                   0.6.4           py310h5eee18b_0
pycparser                 2.21               pyhd3eb1b0_0
pygments                  2.15.1          py310h06a4308_1
pyjwt                     2.4.0           py310h06a4308_0
pyopenssl                 23.0.0          py310h06a4308_0
pyqt                      5.15.7          py310h6a678d5_1
pyqt5-sip                 12.11.0                  pypi_0    pypi
pyrsistent                0.18.0          py310h7f8727e_0
pysocks                   1.7.1           py310h06a4308_0
python                    3.10.9               h7a1cb2a_1
python-dateutil           2.8.2              pyhd3eb1b0_0
python-fastjsonschema     2.16.2          py310h06a4308_0
python-libarchive-c       2.9                pyhd3eb1b0_1
pytorch-triton            2.1.0+6e4932cda8          pypi_0    pypi
pytz                      2022.7          py310h06a4308_0
pyyaml                    6.0             py310h5eee18b_1
qt-main                   5.15.2               h8373d8f_8
qt-webengine              5.15.9               hbbf29b9_6
qtpy                      2.2.0           py310h06a4308_0
qtwebkit                  5.212                h3fafdc1_5
readline                  8.2                  h5eee18b_0
requests                  2.28.2                   pypi_0    pypi
requests-toolbelt         0.9.1              pyhd3eb1b0_0
rhash                     1.4.1                h3c74f83_1
ripgrep                   13.0.0               hbdeaff8_0
ruamel.yaml               0.17.21         py310h5eee18b_0
ruamel.yaml.clib          0.2.6           py310h5eee18b_1
ruamel_yaml               0.17.21         py310h5eee18b_0
scapy                     2.5.0                    pypi_0    pypi
setuptools                65.6.3          py310h06a4308_0
sip                       6.6.2           py310h6a678d5_0
six                       1.16.0             pyhd3eb1b0_1
sortedcontainers          2.4.0                    pypi_0    pypi
soupsieve                 2.4             py310h06a4308_0
sqlite                    3.41.2               h5eee18b_0
stack-data                0.6.2                    pypi_0    pypi
sympy                     1.11.1                   pypi_0    pypi
tk                        8.6.12               h1ccaba5_0
toml                      0.10.2             pyhd3eb1b0_0
tomli                     2.0.1           py310h06a4308_0
toolz                     0.12.0          py310h06a4308_0
torch                     1.13.1                   pypi_0    pypi
torchaudio                0.13.1                   pypi_0    pypi
torchtext                 0.14.1                   pypi_0    pypi
torchvision               0.14.1                   pypi_0    pypi
tornado                   6.2             py310h5eee18b_0
tqdm                      4.65.0          py310h2f386ee_0
traitlets                 5.7.1           py310h06a4308_0
types-dataclasses         0.6.6                    pypi_0    pypi
typing-extensions         4.5.0                    pypi_0    pypi
tzdata                    2023c                h04d1e81_0
ujson                     5.4.0           py310h6a678d5_0
urllib3                   1.26.15                  pypi_0    pypi
wcwidth                   0.2.6                    pypi_0    pypi
wheel                     0.38.4          py310h06a4308_0
xz                        5.4.2                h5eee18b_0
yaml                      0.2.5                h7b6447c_0
zlib                      1.2.13               h5eee18b_0
zstandard                 0.19.0          py310h5eee18b_0
zstd                      1.5.5                hc292b87_0
</code></pre>
<p>I tried so many solutions on the forum including purging and reinstalling drivers to using <code>gcc 10</code> and specifying <code>CUDA</code> home to be the <code>Conda</code> enviornment and many other countless solutions. But I still get this error. Any help in this regard is most welcome. I assume <code>/usr/bin/ld: cannot find -lcudart_static</code> has something to do with the error but can’t seems to solve it.</p>
<p>Also, please provide me full details for the solutions as i’m too new to this.</p>
<p>Thank you in advance.</p>

ptrblck: <p>If you are using your conda binaries to compile PyTorch you could try to uninstall these and instead install a full CUDA toolkit, including the compiler, locally from <a href="https://developer.nvidia.com/cuda-downloads">here</a>.</p>

chanaka: <p>Sorted this out by installing Cuda Toolkit from <code>run</code> file, not the Application Manger [<a href="https://developer.nvidia.com/cuda-11-3-1-download-archive?target_os=Linux&amp;target_arch=x86_64&amp;Distribution=Ubuntu&amp;target_version=20.04&amp;target_type=runfile_local" rel="noopener nofollow ugc">Link to 11.3.1. here</a>] and then sourcing the <code>CUDA_HOME</code> to be <code>/usr/local/cuda/</code> using <code>export CUDA_HOME=/usr/local/cuda/</code>. Thanks for the tip <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>