Libtorch_cuda.so is too large (>2GB)

Trevor_Hickey: <p>We use <a href="https://bazel.build/" rel="noopener nofollow ugc">bazel</a> as part of large monorepo to integrate with torch.  In order to support large amounts of concurrent builds, we must execute our build actions remotely and this entails serializing the files needed for the build action.  However, protobuf serialization has a <a href="https://github.com/bazelbuild/bazel/issues/12487" rel="noopener nofollow ugc">hard limit of 2gb</a> which makes the <code>libtorch_cuda.so</code> unable to be used remotely, which in turn makes many actions in our repository unable to execute remotely.  Running these actions locally would be very bad for build performance, so I was looking for ways to shrink <code>libtorch_cuda.so</code>.</p>
<p>I tried stripping out debug symbols:<br>
strip --strip-debug libtorch_cuda.so<br>
strip --strip-unneeded libtorch_cuda.so<br>
no effect.</p>
<p>Do you have any suggestions?</p>

Trevor_Hickey: <p>we got the wheels from:<br>
<a href="https://download.pytorch.org/whl/torch_stable.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://download.pytorch.org/whl/torch_stable.html</a></p>
<p>we also build from source, but that ended up being 2.4GB.  Not sure if there are better compiler flags / env variables to make it smaller.</p>

albanD: <p>Hi,</p>
<p>It should be possible to shrink it as we do meet the pypi max package size. You can see in the release that our cuda binaries are around 700MB.</p>
<p>You might want to make sure that the cuda architecture detection works fine and only compile for the architecture that is relevant for you. You can use <code>TORCH_CUDA_ARCH_LIST</code> to force specific architectures. If it compiles for all of them, that will make the binary very large.<br>
Also building with <code>DEBUG=0</code> will avoid all debug symbols.<br>
Do these help?</p>

Trevor_Hickey: <p>good tips!  I’ll do a rebuild with these env variables and see if we get some smaller artifacts.</p>

Trevor_Hickey: <p><code>DEBUG=0</code> did not make a difference for our build.  perhaps it was already off by default.<br>
our <code>TORCH_CUDA_ARCH_LIST is "5.2;6.1;7.0;7.5+PTX"</code></p>
<p>as an experiment, I removed <code>5.2</code> and the size went from <code>2.5GB</code> to <code>2.4GB</code>.<br>
then removed <code>7.0</code> to go to <code>2.3GB</code></p>
<p>I did notice that cuda libraries got much larger between cuda 10.2 to 11 which is what revealed the size issues.  But I’ll need to see if nividia has anything to say about that.</p>

ptrblck: <p>Our source build with <code>TORCH_CUDA_ARCH_LIST="5.2 6.0 6.1 7.0 7.5 8.0 8.6+PTX"</code> creates a <code>libtorch_cuda.so</code> of <code>1269MB</code> with CUDA11.1, so I’m unsure why your build creates a bigger lib with less compute capabilities.</p>
<p>We are currently working internally to provide CUDA and cudnn conda and pip wheels, which would allow to dynamically link these runtimes for the binaries. However, this would not solve your local build issue.</p>

Trevor_Hickey: <p>Thank you for the extra data point.</p>
<p>looking at the already published wheels:  <a href="https://download.pytorch.org/whl/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/torch_stable.html</a></p>
<p>I see the following:<br>
cu102/torch-1.7.0-cp37-cp37m-linux_x86_64.whl -&gt; torch/lib/libtorch_cuda.so: <strong>1.1GB</strong><br>
cu110/torch-1.7.0%2Bcu110-cp37-cp37m-linux_x86_64.whl -&gt; torch/lib/libtorch_cuda.so: <strong>2.2GB</strong></p>
<p>So something caused this artifact to double in size between cuda 10.2 and 11.<br>
And from what I can tell between <a href="http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/" rel="noopener nofollow ugc">cuda releases</a>, it may just be cuda itself.</p>
<p>I also tried using <code>CXXFLAGS</code>/<code>CFLAGS</code> with <code>"-Oz"</code> to optimize size. but this seems to have had no effect.</p>
<p>Here are some of the other options we use during local build:</p>
<pre><code class="lang-auto">export EXTRA_CAFFE2_CMAKE_FLAGS=("-DATEN_NO_TEST=ON")
export TORCH_NVCC_FLAGS="-Xfatbin -compress-all"
export TORCH_CUDA_ARCH_LIST="6.1;7.5+PTX"
export TH_BINARY_BUILD=1
export CMAKE_LIBRARY_PATH="/opt/intel/lib:/lib"
export CMAKE_INCLUDE_PATH="/opt/intel/include"
export DEBUG=0
export CXXFLAGS="-Oz"
export CFLAGS="-Oz"
export USE_CUDA=1
export USE_NNPACK=0
export USE_QNNPACK=0
export USE_FBGEMM=1
export USE_OPENMP=0
export USE_TBB=1
export USE_GFLAGS=0
export USE_GLOG=0
export BLAS=MKL
export ATEN_THREADING=TBB
export MKL_THREADING=TBB
export MKLDNN_CPU_RUNTIME=TBB
export PARALLEL_BACKEND=NATIVE_TBB
export CC=/clang_9.0.0/bin/clang
export CXX=/clang_9.0.0/bin/clang++
export PATCHELF_BIN=/usr/local/bin/patchelf
export VERBOSE=1
</code></pre>
<p>Maybe “fatbin” is a problem.  Going to try that next.</p>

Trevor_Hickey: <p>nvidia suggested we use <a href="https://docs.nvidia.com/cuda/cuda-binary-utilities/index.html#nvprune" rel="noopener nofollow ugc">nvprune</a>, but that doesn’t work if the object is not relocatable.</p>
<pre><code class="lang-auto">nvprune --arch sm_35 torch/lib/libtorch_cuda.so -o ~/Desktop/prune_test.so 
nvprune fatal   : Input file 'torch/lib/libtorch_cuda.so' not relocatable
</code></pre>

Trevor_Hickey: <p>Our specific goal would be to prune pytorch’s distributed whl to get it &lt; 2GB.<br>
Here is what that looks like:</p>
<pre><code class="lang-auto"># get
wget https://download.pytorch.org/whl/cu110/torch-1.7.0%2Bcu110-cp37-cp37m-linux_x86_64.whl
unzip https://download.pytorch.org/whl/cu110/torch-1.7.0%2Bcu110-cp37-cp37m-linux_x86_64.whl

# see info
du -h torch/lib/libtorch_cuda.so
2.1G torch/lib/libtorch_cuda.so

file torch/lib/libtorch_cuda.so
torch/lib/libtorch_cuda.so: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, BuildID[sha1]=e88b68dc083cd4dc547483c896d95ef6bedd64fc, with debug_info, not stripped

# see info after stripping
strip torch/lib/libtorch_cuda.so
file torch/lib/libtorch_cuda.so
torch/lib/libtorch_cuda.so: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, BuildID[sha1]=e88b68dc083cd4dc547483c896d95ef6bedd64fc, stripped

du -h torch/lib/libtorch_cuda.so
2.0G torch/lib/libtorch_cuda.so

# try prune
nvprune torch/lib/libtorch_cuda.so -arch sm_35 -o shrink_me.so
nvprune fatal : Input file 'torch/lib/libtorch_cuda.so' not relocatable
</code></pre>

Trevor_Hickey: <p>Can you share the script or the general logic of how you are building this.  I can compare it to how we are building and hopefully spot the differences.</p>

ptrblck: <p>I’m building PyTorch via:</p>
<pre><code class="lang-python">TORCH_CUDA_ARCH_LIST="5.2 6.0 6.1 7.0 7.5 8.0 8.6+PTX" \
CUDA_HOME="/usr/local/cuda" \
CMAKE_PREFIX_PATH="$(dirname $(which conda))/../" \
NCCL_INCLUDE_DIR="/usr/include/" \
NCCL_LIB_DIR="/usr/lib/" \
USE_SYSTEM_NCCL=1 \
python setup.py develop
</code></pre>

Trevor_Hickey: <p>It seems like <code>TORCH_CUDA_ARCH_LIST</code> is being ignored for me when I create the wheel:</p>
<p>I run:</p>
<pre><code class="lang-auto">  python3 setup.py clean --all
  time USE_STATIC_CUDNN=1 USE_STATIC_NCCL=1 ATEN_STATIC_CUDA=1 USE_CUDA_STATIC_LINK=1 \
       TORCH_CUDA_ARCH_LIST="5.2;6.1;7.0;7.5+PTX" \
       CXXFLAGS="-stdlib=libstdc++" \
       python3 setup.py bdist_wheel -d "${BUILD_DIR}"
</code></pre>
<p>but the results show:</p>
<pre><code class="lang-auto">./cuobjdump libtorch_cuda.so -lelf | awk -F. '{print $3}' | sort -u
cubin
sm_35
sm_37
sm_50
sm_52
sm_60
sm_61
sm_70
sm_75
sm_80
</code></pre>
<p>I also got an interesting warning.  Is this relevant?</p>
<blockquote>
<p>CMake Warning at cmake/External/nccl.cmake:65 (message):<br>
Objcopy version is too old to support NCCL library slimming</p>
</blockquote>

Trevor_Hickey: <p>maybe these architectures show up because they already exist in the cuda objects?</p>

Trevor_Hickey: <p><a class="mention" href="/u/alband">@albanD</a><br>
are all of these <code>sm_*</code>s normal given what I passed <code>TORCH_CUDA_ARCH_LIST</code>?</p>

albanD: <p>At the beginning of the build process, during the cmake setup, you should have a line that lists the cuda architecture it will be using and why. Could you check that? You should be able to grep for CUDA in the build logs.</p>

Trevor_Hickey: <p>I see the CUDA NVCC flags chosen in the cmake log.  They look correct according to using:<br>
<code>TORCH_CUDA_ARCH_LIST="5.2;6.1;7.0;7.5+PTX"</code></p>
<pre><code class="lang-auto">-- Found CUDA: /usr/local/cuda (found version "11.0") 
-- Caffe2: CUDA detected: 11.0
-- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda
-- Caffe2: Header version is: 11.0
-- Found CUDNN: /usr/local/cuda/lib64/libcudnn_static.a  
-- Found cuDNN: v8.0.4  (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libcudnn_static.a)
CMake Warning at cmake/public/utils.cmake:196 (message):
  In the future we will require one to explicitly pass TORCH_CUDA_ARCH_LIST
  to cmake instead of implicitly setting it as an env variable.  This will
  become a FATAL_ERROR in future version of pytorch.
Call Stack (most recent call first):
  cmake/public/cuda.cmake:452 (torch_cuda_get_nvcc_gencode_flag)
  cmake/Dependencies.cmake:1097 (include)
  CMakeLists.txt:469 (include)


-- Added CUDA NVCC flags for: -gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_75,code=compute_75
CMake Warning at cmake/External/nccl.cmake:65 (message):
  Objcopy version is too old to support NCCL library slimming
Call Stack (most recent call first):
  cmake/Dependencies.cmake:1219 (include)
  CMakeLists.txt:469 (include)
</code></pre>

albanD: <p>Ok so it does look like it picks up these arguments just fine. Not sure what the issue is to be honest…</p>

Trevor_Hickey: <p>If I figure out what I’m doing wrong, would you expect<br>
<code>./cuobjdump libtorch_cuda.so -lelf | awk -F. '{print $3}' | sort -u</code><br>
to show less results?</p>
<p>I don’t have a point of reference other than the public wheel which was specifically built for all architectures.  But I’m working on trying to get those symbols out of the shared object.  Does that goal make sense, or are they going to be there regardless?</p>

Trevor_Hickey: <p>Do you think there’s a script I can use in <a href="https://github.com/pytorch/builder" rel="noopener nofollow ugc">https://github.com/pytorch/builder</a> that will produce exactly the same public wheel but only for the architectures we care about?  Maybe it would be easier to use something closer to your CI.</p>
<p>The other mystery still remains.  What made the public wheel double in size?<br>
cu102/torch-1.7.0-cp37-cp37m-linux_x86_64.whl -&gt; torch/lib/libtorch_cuda.so: <strong>1.1GB</strong><br>
cu110/torch-1.7.0%2Bcu110-cp37-cp37m-linux_x86_64.whl -&gt; torch/lib/libtorch_cuda.so: <strong>2.2GB</strong></p>

albanD: <p>I think the new archs were responsible for 300MB. But I don’t know for the other 700MB <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"><br>
cc <a class="mention" href="/u/seemethere">@seemethere</a> who know the builder repo quite well. He will be able to help you with that!</p>