Compile the PyTorch from source code by using Dockerfile, I got a fishy error. It seems to be related to the ld linker

shysuen001: <p>When I tried to compile the PyTorch from source code using a Dockerfile, I got a fishy error. The error seems to be related to the ld linker.</p>
<h3><a name="p-459243-error-logs-1" class="anchor" href="#p-459243-error-logs-1"></a>Error logs</h3>
<p><span class="hashtag-raw">#0</span> 3395.3 [5555/5977] Linking CXX executable bin/cuda_apply_test<br>
<span class="hashtag-raw">#0</span> 3395.3 FAILED: bin/cuda_apply_test<br>
<span class="hashtag-raw">#0</span> 3395.3 : &amp;&amp; /usr/bin/c++ -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow -DHAVE_AVX_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -rdynamic caffe2/CMakeFiles/cuda_apply_test.dir/__/aten/src/ATen/test/cuda_apply_test.cpp.o -o bin/cuda_apply_test -Wl,-rpath,/usr/local/cuda/lib64:/opt/conda/envs/pytorch/lib:/pytorch/build/lib: /usr/local/cuda/lib64/libcudart.so lib/libgtest_main.a -Wl,–no-as-needed,“/pytorch/build/lib/libtorch.so” -Wl,–as-needed -Wl,–no-as-needed,“/pytorch/build/lib/libtorch_cpu.so” -Wl,–as-needed lib/libprotobuf.a /opt/conda/envs/pytorch/lib/libmkl_intel_lp64.so /opt/conda/envs/pytorch/lib/libmkl_gnu_thread.so /opt/conda/envs/pytorch/lib/libmkl_core.so -fopenmp -lpthread -lm /usr/lib/x86_64-linux-gnu/libdl.so lib/libdnnl.a -Wl,–no-as-needed,“/pytorch/build/lib/libtorch_cuda.so” -Wl,–as-needed lib/libc10_cuda.so lib/libc10.so /usr/local/cuda/lib64/libnvToolsExt.so /usr/local/cuda/lib64/libcufft_static_nocallback.a /usr/local/cuda/lib64/libcurand_static.a /usr/local/cuda/lib64/libculibos.a /usr/local/cuda/lib64/libcublas_static.a /usr/local/cuda/lib64/libcublasLt_static.a /usr/local/cuda/lib64/libcudart_static.a -lrt -ldl lib/libgtest.a -pthread &amp;&amp; :<br>
<span class="hashtag-raw">#0</span> 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to <code>cublasHgemmBatched' #0 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to </code>cublasCgetriBatched’<br>
<span class="hashtag-raw">#0</span> 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to <code>cublasCgemm_v2' #0 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to </code>cublasCgetrfBatched’<br>
<span class="hashtag-raw">#0</span> 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to <code>cublasDdot_v2' #0 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to </code>cublasDgetrfBatched’<br>
<span class="hashtag-raw">#0</span> 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to <code>cublasGetPointerMode_v2' #0 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to </code>cublasDgemmStridedBatched’<br>
<span class="hashtag-raw">#0</span> 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to <code>cublasDscal_v2' #0 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to </code>cublasDgetriBatched’<br>
<span class="hashtag-raw">#0</span> 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to <code>cublasZgemm_v2' #0 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to </code>cublasGemmEx’<br>
<span class="hashtag-raw">#0</span> 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to <code>cublasDestroy_v2' #0 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to </code>cublasCgemv_v2’<br>
<span class="hashtag-raw">#0</span> 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to <code>cublasCdotu_v2' #0 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to </code>cublasSaxpy_v2’<br>
<span class="hashtag-raw">#0</span> 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to <code>cublasGemmBatchedEx' #0 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to </code>cublasGemmStridedBatchedEx’<br>
<span class="hashtag-raw">#0</span> 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to <code>cublasDgemm_v2' #0 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to </code>cublasSgeam’<br>
<span class="hashtag-raw">#0</span> 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to <code>cublasHgemm' #0 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to </code>cublasCreate_v2’<br>
<span class="hashtag-raw">#0</span> 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to <code>cublasDgeam' #0 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to </code>cublasSetStream_v2’<br>
<span class="hashtag-raw">#0</span> 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to <code>cublasSetMathMode' #0 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to </code>cublasSetPointerMode_v2’<br>
<span class="hashtag-raw">#0</span> 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to <code>cublasSgemmStridedBatched' #0 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to </code>cublasDotEx’<br>
<span class="hashtag-raw">#0</span> 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to <code>cublasZgemv_v2' #0 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to </code>cublasScalEx’<br>
<span class="hashtag-raw">#0</span> 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to <code>cublasZgetrfBatched' #0 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to </code>cublasCdotc_v2’<br>
<span class="hashtag-raw">#0</span> 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to <code>cublasZdotc_v2' #0 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to </code>cublasZgetriBatched’<br>
<span class="hashtag-raw">#0</span> 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to <code>cublasSgemv_v2' #0 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to </code>cublasSgetrfBatched’<br>
<span class="hashtag-raw">#0</span> 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to <code>cublasSdot_v2' #0 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to </code>cublasSgetriBatched’<br>
<span class="hashtag-raw">#0</span> 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to <code>cublasSgemmEx' #0 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to </code>cublasSgemm_v2’<br>
<span class="hashtag-raw">#0</span> 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to <code>cublasDgemv_v2' #0 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to </code>cublasHgemmStridedBatched’<br>
<span class="hashtag-raw">#0</span> 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to <code>cublasCgemmStridedBatched' #0 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to </code>cublasSscal_v2’<br>
<span class="hashtag-raw">#0</span> 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to <code>cublasSgemmBatched' #0 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to </code>cublasAxpyEx’<br>
<span class="hashtag-raw">#0</span> 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to <code>cublasZdotu_v2' #0 3395.3 /pytorch/build/lib/libtorch_cuda.so: undefined reference to </code>cublasZgemmStridedBatched’<br>
<span class="hashtag-raw">#0</span> 3395.3 collect2: error: ld returned 1 exit status<br>
<span class="hashtag-raw">#0</span> 3395.3 [5556/5977] Linking CXX executable bin/cuda_stream_test<br>
<span class="hashtag-raw">#0</span> 3395.3 FAILED: bin/cuda_stream_test</p>
<h3><a name="p-459243-minified-repro-2" class="anchor" href="#p-459243-minified-repro-2"></a>Minified repro</h3>
<p>This is my dockerfile:</p>
<p>FROM <a href="http://nvcr.io/nvidia/cuda:10.2-cudnn8-devel-ubuntu18.04" rel="noopener nofollow ugc">nvcr.io/nvidia/cuda:10.2-cudnn8-devel-ubuntu18.04</a></p>
<p>ENV LANG=C.UTF-8 LC_ALL=C.UTF-8<br>
ENV PATH /opt/conda/bin:$PATH<br>
ENV PYTORCH_BUILD_VERSION=1.8.1+cu102<br>
ENV PYTORCH_BUILD_NUMBER=1</p>
<p>RUN apt-get update &amp;&amp; apt-get install -y --no-install-recommends<br>
build-essential<br>
cmake<br>
git<br>
curl<br>
ca-certificates<br>
libjpeg-dev<br>
libpng-dev<br>
&amp;&amp; rm -rf /var/lib/apt/lists/*</p>
<p>RUN curl -o ~/miniconda.sh -O <a href="https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh" rel="noopener nofollow ugc">https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh</a> &amp;&amp;<br>
chmod +x ~/miniconda.sh &amp;&amp;<br>
~/miniconda.sh -b -p /opt/conda &amp;&amp;<br>
rm ~/miniconda.sh &amp;&amp;<br>
/opt/conda/bin/conda clean -ya</p>
<p>RUN conda create -n pytorch python=3.9.20 &amp;&amp; conda clean -ya<br>
ENV CONDA_DEFAULT_ENV=pytorch<br>
ENV PATH /opt/conda/envs/pytorch/bin:$PATH</p>
<p>RUN echo “source activate pytorch” &gt; ~/.bashrc<br>
ENV BASH_ENV ~/.bashrc<br>
ENV ENV ~/.bashrc</p>
<p>RUN conda install -y numpy=1.23 ninja pyyaml mkl mkl-include setuptools cmake cffi typing_extensions future six requests dataclasses</p>
<p>COPY pytorch-v1.8.1.tar.gz /pytorch-v1.8.1.tar.gz<br>
WORKDIR /<br>
RUN tar xvzf pytorch-v1.8.1.tar.gz</p>
<p>WORKDIR /pytorch<br>
RUN git submodule sync &amp;&amp; git submodule update --init --recursive</p>
<p>ENV TORCH_CUDA_ARCH_LIST=“6.0;6.1;7.0;7.5”<br>
ENV USE_CUDA=1<br>
ENV USE_CUDNN=1<br>
ENV CMAKE_PREFIX_PATH=“$(dirname $(which conda))/…/”<br>
ENV USE_STATIC_CUDA=ON</p>
<p>RUN ln -s /usr/lib/x86_64-linux-gnu/libcublasLt_static.a /usr/local/cuda/lib64/libcublasLt_static.a &amp;&amp; ln -s /usr/lib/x86_64-linux-gnu/libcublasLt_static.a /usr/local/cuda-10.2/lib64/libcublas_static.a<br>
RUN python setup.py bdist_wheel</p>
<h3><a name="p-459243-versions-3" class="anchor" href="#p-459243-versions-3"></a>Versions</h3>
<p>this is the host ENV:</p>
<p>OS: CentOS Linux release 7.7.1908 (Core) (x86_64)<br>
GCC version: (Anaconda gcc) 11.2.0<br>
Clang version: Could not collect<br>
CMake version: Could not collect<br>
Libc version: glibc-2.17</p>
<p>Python version: 3.9.20 (main, Oct 3 2024, 07:27:41) [GCC 11.2.0] (64-bit runtime)<br>
Python platform: Linux-3.10.0-1062.12.1.el7.x86_64-x86_64-with-glibc2.17<br>
Is CUDA available: N/A<br>
CUDA runtime version: 10.2.89<br>
CUDA_MODULE_LOADING set to: N/A<br>
GPU models and configuration:<br>
GPU 0: Tesla V100S-PCIE-32GB<br>
GPU 1: Tesla V100S-PCIE-32GB</p>

ptrblck: <p>It seems your attempt to statically link cuBLAS failed. What’s the reason you want to use static linking instead of dynamic?</p>

shysuen001: <p>Thank you for your response. In fact, I am a newcomer to compiling PyTorch from source (only able to follow PyTorch’s GitHub README). I am not very clear on how and why to use static linking and dynamic linking. Could you please tell me how to use dynamic linking?</p>

sgomber: <p>I tried dynamic linking but what I see is that it is spending some time (around 0.4s) just in dynamic linking. I was of the understanding that this time can be saved if we use static linking?</p>

shysuen001: <p>I have found the solution to this problem!</p>
<p>The reason this issue occurs is that the NVIDIA image does not contain <code>libcublasLt_static.a</code> and <code>libcublas_static.a</code>.</p>
<p>The solution is to mount the corresponding files from the host to the <code>/usr/local/cuda/include/</code> path within the Docker container.</p>