Why is torch-1.10.0+cu113 much larger than torch-1.10.0

ralphmao: <p>I am deploying our ML models in a docker container, and we try to reduce the size of this docker image.  I notice that <code>torch-1.10.0+cu113</code> is more than 1GB larger than <code>torch-1.10.0</code>. The main difference is caused by <code>torch/lib/libtorch_cuda_cpp.so</code>, which only exists in <code>torch-1.10.0+cu113</code>. What’s the use of this file and can we not have it but still make it works with CUDA 11.3?</p>

eqy: <p>The GPU kernels are separate in PyTorch precisely for the reason you’re describing: they add a substantial amount of footprint to the library size. This file contains all the CUDA kernels for GPU execution; without them there wouldn’t be any native CUDA support.</p>

ralphmao: <p>Thanks for your reply! But torch-1.10.0 also has GPU support (for cuda 10.2)</p>

ralphmao: <p>In torch-1.10.0-cu113, there are:</p>
<pre><code class="lang-auto">libtorch_cuda.so 
libtorch_cuda_cpp.so
libtorch_cpu.so
</code></pre>
<p>In torch-1.10.0 (which works for cu102), there are:</p>
<pre><code class="lang-auto">libtorch_cuda.so 
libtorch_cpu.so
</code></pre>
<p>So CUDA kernels are contained in <code>libtorch_cuda.so</code>. So what is the difference between <code>libtorch_cuda.so</code> and <code>libtorch_cuda_cpp.so</code>?</p>

eqy: <p>Ah, sorry, I misread your question. I think the difference might be that CUDA 11 will support more GPU architectures; there are corresponding kernels for the newer architectures with the newer CUDA version e.g.,</p><aside class="quote quote-modified" data-post="6" data-topic="120235">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/need-help-trouble-with-cuda-capability-sm-86/120235/6">[NEED HELP] Trouble with CUDA capability sm_86</a> 
  </div>
  <blockquote>
    NIT: TORCH_CUDA_ARCH_LIST will be used while building from source and won’t change the shipped compute capabilities in the binaries, which you can get via print(torch.cuda.get_arch_list()). 

That’s expected, since Ampere GPUs need CUDA&gt;=11.0 

No, the 3070 uses sm_86, which is natively supported in CUDA&gt;=11.1 and is binary compatible to sm_80, so would already work in CUDA=11.0. 
In any case, good to hear it’s working now <img width="20" height="20" src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title="slight_smile" alt="slight_smile" class="emoji">
  </blockquote>
</aside>

<p>As far as the file split, I think that might be just an artifact of a tweak to the build process, but I’m not very knowledgeable on the details here.</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/9ae3f3945b519c04ef0aa4f9e441be6401292f86/CMakeLists.txt#L181">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/9ae3f3945b519c04ef0aa4f9e441be6401292f86/CMakeLists.txt#L181" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/9ae3f3945b519c04ef0aa4f9e441be6401292f86/CMakeLists.txt#L181" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/9ae3f3945b519c04ef0aa4f9e441be6401292f86/CMakeLists.txt#L181</a></h4>



    <pre class="onebox"><code class="lang-txt">
      <ol class="start lines" start="171" style="counter-reset: li-counter 170 ;">
          <li>option(BUILD_JNI "Build JNI bindings" OFF)</li>
          <li>option(BUILD_MOBILE_AUTOGRAD "Build autograd function in mobile build (in development)" OFF)</li>
          <li>cmake_dependent_option(</li>
          <li>    INSTALL_TEST "Install test binaries if BUILD_TEST is on" ON</li>
          <li>    "BUILD_TEST" OFF)</li>
          <li>option(USE_CPP_CODE_COVERAGE "Compile C/C++ with code coverage flags" OFF)</li>
          <li>option(COLORIZE_OUTPUT "Colorize output during compilation" ON)</li>
          <li>option(USE_ASAN "Use Address Sanitizer" OFF)</li>
          <li>option(USE_TSAN "Use Thread Sanitizer" OFF)</li>
          <li>option(USE_CUDA "Use CUDA" ON)</li>
          <li class="selected"># BUILD_SPLIT_CUDA must also be exported as an environment variable before building, with</li>
          <li># `export BUILD_SPLIT_CUDA=1` because cpp_extension.py can only work properly if this variable</li>
          <li># also exists in the environment.</li>
          <li># This option is incompatible with CUDA_SEPARABLE_COMPILATION.</li>
          <li>cmake_dependent_option(</li>
          <li>    BUILD_SPLIT_CUDA "Split torch_cuda library into torch_cuda_cu and torch_cuda_cpp" OFF</li>
          <li>    "USE_CUDA AND NOT CUDA_SEPARABLE_COMPILATION" OFF)</li>
          <li>option(USE_FAST_NVCC "Use parallel NVCC build" OFF)</li>
          <li>option(USE_ROCM "Use ROCm" ON)</li>
          <li>option(CAFFE2_STATIC_LINK_CUDA "Statically link CUDA libraries" OFF)</li>
          <li>cmake_dependent_option(</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


ralphmao: <p>Thank you for your reply! <code>libtorch_cuda_cpp.so</code> is taking more than 2.5GB of space. I tried deleting it and so far the code runs without any problem. Very interesting.</p>

eqy: <p>(see more potential details here):</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/4262c8913c2bddb8d91565888b4871790301faba/caffe2/CMakeLists.txt#L177">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/4262c8913c2bddb8d91565888b4871790301faba/caffe2/CMakeLists.txt#L177" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/4262c8913c2bddb8d91565888b4871790301faba/caffe2/CMakeLists.txt#L177" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/4262c8913c2bddb8d91565888b4871790301faba/caffe2/CMakeLists.txt#L177</a></h4>



  <pre class="onebox">    <code class="lang-txt">
      <ol class="start lines" start="167" style="counter-reset: li-counter 166 ;">
          <li>endif()</li>
          <li>
          </li>
<li># Advanced: if we have allow list specified, we will do intersections for all</li>
          <li># main lib srcs.</li>
          <li>if(CAFFE2_ALLOWLISTED_FILES)</li>
          <li>  caffe2_do_allowlist(Caffe2_CPU_SRCS CAFFE2_ALLOWLISTED_FILES)</li>
          <li>  caffe2_do_allowlist(Caffe2_GPU_SRCS CAFFE2_ALLOWLISTED_FILES)</li>
          <li>  caffe2_do_allowlist(Caffe2_HIP_SRCS CAFFE2_ALLOWLISTED_FILES)</li>
          <li>endif()</li>
          <li>
          </li>
<li class="selected">if(BUILD_SPLIT_CUDA)</li>
          <li>  # Splitting the source files that'll be in torch_cuda between torch_cuda_cu and torch_cuda_cpp</li>
          <li>  foreach(tmp ${Caffe2_GPU_SRCS})</li>
          <li>    if("${tmp}" MATCHES "(.*aten.*\\.cu|.*(b|B)las.*|.*((s|S)olver|Register.*CUDA|Legacy|THC|TensorShapeCUDA|BatchLinearAlgebra|ReduceOps|Equal|Activation|ScanKernels|Sort|TensorTopK).*\\.cpp)" AND NOT "${tmp}" MATCHES ".*(THC((CachingHost)?Allocator|General)).*")</li>
          <li>      # Currently, torch_cuda_cu will have all the .cu files in aten, as well as some others that depend on those files</li>
          <li>      list(APPEND Caffe2_GPU_SRCS_CU ${tmp})</li>
          <li>    else()</li>
          <li>      list(APPEND Caffe2_GPU_SRCS_CPP ${tmp})</li>
          <li>    endif()</li>
          <li>  endforeach()</li>
          <li>
      </li>
</ol>
    </code>
  </pre>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>It might be that you’re not hitting any of the functions defined there…</p>

ralphmao: <p>Since torch-1.10.0 with cuda10.2 doesn’t have <code>libtorch_cuda_cpp.so</code> and still works fine on GPU, I suppose <code>libtorch_cuda_cpp.so</code> is only needed by caffe2.</p>

eqy: <p>That doesn’t sound quite right… if you take a look at the file it’s going over sources file in <code>aten</code> so that warrants some caution</p>

ptrblck: <p>No, the caffe2 bits are built in <code>libcaffe2_...so</code>. The library splitting was introduced to avoid the <a href="https://github.com/pytorch/pytorch/issues/39968">relocation issue</a> introduced by the large library size during linking due to the size increase in CUDA, cuDNN etc.<br>
The <code>libtorch_cuda_cpp.so</code> contains symbols for cuDNN, NCCL etc. Also, if I delete it, I’m unable to <code>import torch</code> and get:</p>
<pre><code class="lang-python">ImportError: libtorch_cuda_cpp.so: cannot open shared object file: No such file or directory
</code></pre>
<p>so unsure what your workflow is.<br>
TL;DR: don’t delete it, as this lib is needed.</p>

lorenzznerol: <p>Not sure whether this adds any value, but it might help looking at how pytorch gets installed from source when it comes to questions about which uses which in pytorch. See <a href="https://discuss.pytorch.org/t/i-cannot-use-the-pytorch-that-was-built-successfully-from-source-dll-initialization-routine-failed-error-loading-caffe2-detectron-ops-gpu-dll/93243/1">I cannot use the pytorch that was built successfully from source: (DLL) initialization routine failed. Error loading caffe2_detectron_ops_gpu.dll</a>. At least, there, the same players take part. Just a side note, I guess that this is not useful in the end.</p>