Large CUDA dependencies in PyTorch 1.13 (conda installation)

gau-nernst: <p>Hello,</p>
<p>I have tried searching on the internet but no one seems to report this issue. When installing PyTorch 1.13, there are a lot of CUDA dependencies (apart from <code>cudatoolkit</code>) which are quite large, making the conda environment huge. I’m not sure if all of those dependencies are necessary, as it seems previous versions of PyTorch don’t need them?</p>
<p>Following the official installation instruction</p>
<pre><code class="lang-auto">conda install pytorch torchvision pytorch-cuda=11.7 -c pytorch -c nvidia
</code></pre>
<p>Output from conda</p>
<pre><code class="lang-auto">The following NEW packages will be INSTALLED:

  blas               pkgs/main/linux-64::blas-1.0-mkl
  cuda               nvidia/linux-64::cuda-11.7.1-0
  cuda-cccl          nvidia/linux-64::cuda-cccl-11.7.91-0
  cuda-command-line~ nvidia/linux-64::cuda-command-line-tools-11.7.1-0
  cuda-compiler      nvidia/linux-64::cuda-compiler-11.7.1-0
  cuda-cudart        nvidia/linux-64::cuda-cudart-11.7.99-0
  cuda-cudart-dev    nvidia/linux-64::cuda-cudart-dev-11.7.99-0
  cuda-cuobjdump     nvidia/linux-64::cuda-cuobjdump-11.7.91-0
  cuda-cupti         nvidia/linux-64::cuda-cupti-11.7.101-0
  cuda-cuxxfilt      nvidia/linux-64::cuda-cuxxfilt-11.7.91-0
  cuda-demo-suite    nvidia/linux-64::cuda-demo-suite-11.8.86-0
  cuda-documentation nvidia/linux-64::cuda-documentation-11.8.86-0
  cuda-driver-dev    nvidia/linux-64::cuda-driver-dev-11.7.99-0
  cuda-gdb           nvidia/linux-64::cuda-gdb-11.8.86-0
  cuda-libraries     nvidia/linux-64::cuda-libraries-11.7.1-0
  cuda-libraries-dev nvidia/linux-64::cuda-libraries-dev-11.7.1-0
  cuda-memcheck      nvidia/linux-64::cuda-memcheck-11.8.86-0
  cuda-nsight        nvidia/linux-64::cuda-nsight-11.8.86-0
  cuda-nsight-compu~ nvidia/linux-64::cuda-nsight-compute-11.8.0-0
  cuda-nvcc          nvidia/linux-64::cuda-nvcc-11.7.99-0
  cuda-nvdisasm      nvidia/linux-64::cuda-nvdisasm-11.8.86-0
  cuda-nvml-dev      nvidia/linux-64::cuda-nvml-dev-11.7.91-0
  cuda-nvprof        nvidia/linux-64::cuda-nvprof-11.8.87-0
  cuda-nvprune       nvidia/linux-64::cuda-nvprune-11.7.91-0
  cuda-nvrtc         nvidia/linux-64::cuda-nvrtc-11.7.99-0
  cuda-nvrtc-dev     nvidia/linux-64::cuda-nvrtc-dev-11.7.99-0
  cuda-nvtx          nvidia/linux-64::cuda-nvtx-11.7.91-0
  cuda-nvvp          nvidia/linux-64::cuda-nvvp-11.8.87-0
  cuda-runtime       nvidia/linux-64::cuda-runtime-11.7.1-0
  cuda-sanitizer-api nvidia/linux-64::cuda-sanitizer-api-11.8.86-0
  cuda-toolkit       nvidia/linux-64::cuda-toolkit-11.7.1-0
  cuda-tools         nvidia/linux-64::cuda-tools-11.7.1-0
  cuda-visual-tools  nvidia/linux-64::cuda-visual-tools-11.7.1-0
  gds-tools          nvidia/linux-64::gds-tools-1.4.0.31-0
  giflib             pkgs/main/linux-64::giflib-5.2.1-h7b6447c_0
  intel-openmp       pkgs/main/linux-64::intel-openmp-2021.4.0-h06a4308_3561
  jpeg               pkgs/main/linux-64::jpeg-9e-h7f8727e_0
  lcms2              pkgs/main/linux-64::lcms2-2.12-h3be6417_0
  lerc               pkgs/main/linux-64::lerc-3.0-h295c915_0
  libcublas          nvidia/linux-64::libcublas-11.11.3.6-0
  libcublas-dev      nvidia/linux-64::libcublas-dev-11.11.3.6-0
  libcufft           nvidia/linux-64::libcufft-10.9.0.58-0
  libcufft-dev       nvidia/linux-64::libcufft-dev-10.9.0.58-0
  libcufile          nvidia/linux-64::libcufile-1.4.0.31-0
  libcufile-dev      nvidia/linux-64::libcufile-dev-1.4.0.31-0
  libcurand          nvidia/linux-64::libcurand-10.3.0.86-0
  libcurand-dev      nvidia/linux-64::libcurand-dev-10.3.0.86-0
  libcusolver        nvidia/linux-64::libcusolver-11.4.1.48-0
  libcusolver-dev    nvidia/linux-64::libcusolver-dev-11.4.1.48-0
  libcusparse        nvidia/linux-64::libcusparse-11.7.5.86-0
  libcusparse-dev    nvidia/linux-64::libcusparse-dev-11.7.5.86-0
  libdeflate         pkgs/main/linux-64::libdeflate-1.8-h7f8727e_5
  libnpp             nvidia/linux-64::libnpp-11.8.0.86-0
  libnpp-dev         nvidia/linux-64::libnpp-dev-11.8.0.86-0
  libnvjpeg          nvidia/linux-64::libnvjpeg-11.9.0.86-0
  libnvjpeg-dev      nvidia/linux-64::libnvjpeg-dev-11.9.0.86-0
  libtiff            pkgs/main/linux-64::libtiff-4.4.0-hecacb30_0
  libwebp            pkgs/main/linux-64::libwebp-1.2.4-h11a3e52_0
  libwebp-base       pkgs/main/linux-64::libwebp-base-1.2.4-h5eee18b_0
  mkl                pkgs/main/linux-64::mkl-2021.4.0-h06a4308_640
  mkl-service        pkgs/main/linux-64::mkl-service-2.4.0-py38h7f8727e_0
  mkl_fft            pkgs/main/linux-64::mkl_fft-1.3.1-py38hd3c417c_0
  mkl_random         pkgs/main/linux-64::mkl_random-1.2.2-py38h51133e4_0
  nsight-compute     nvidia/linux-64::nsight-compute-2022.3.0.22-0
  numpy              pkgs/main/linux-64::numpy-1.23.3-py38h14f4228_1
  numpy-base         pkgs/main/linux-64::numpy-base-1.23.3-py38h31eccc5_1
  pillow             pkgs/main/linux-64::pillow-9.2.0-py38hace64e9_1
  pytorch            pytorch/linux-64::pytorch-1.13.0-py3.8_cuda11.7_cudnn8.5.0_0
  pytorch-cuda       pytorch/noarch::pytorch-cuda-11.7-h67b0de4_0
  pytorch-mutex      pytorch/noarch::pytorch-mutex-1.0-cuda
  torchvision        pytorch/linux-64::torchvision-0.14.0-py38_cu117
  typing_extensions  pkgs/main/linux-64::typing_extensions-4.3.0-py38h06a4308_0
</code></pre>
<p>Listing libraries size with <code>du -h -s $(conda info --base)/envs/diffusers/lib/* | sort -hr</code> (<code>diffusers</code> is my environment name)</p>
<pre><code class="lang-auto">2.9G    /home/students/acct3001_02/miniconda3/envs/diffusers/lib/python3.10
916M    /home/students/acct3001_02/miniconda3/envs/diffusers/lib/libcublasLt_static.a
548M    /home/students/acct3001_02/miniconda3/envs/diffusers/lib/libcublasLt.so.11.11.3.6
308M    /home/students/acct3001_02/miniconda3/envs/diffusers/lib/libcusparse_static.a
300M    /home/students/acct3001_02/miniconda3/envs/diffusers/lib/libcusolver_static.a
294M    /home/students/acct3001_02/miniconda3/envs/diffusers/lib/libcufft_static_nocallback.a
286M    /home/students/acct3001_02/miniconda3/envs/diffusers/lib/libcusolver.so.11.4.1.48
281M    /home/students/acct3001_02/miniconda3/envs/diffusers/lib/libcufft_static.a
267M    /home/students/acct3001_02/miniconda3/envs/diffusers/lib/libcusparse.so.11.7.5.86
267M    /home/students/acct3001_02/miniconda3/envs/diffusers/lib/libcufft.so.10.9.0.58
175M    /home/students/acct3001_02/miniconda3/envs/diffusers/lib/libcusolverMg.so.11.4.1.48
120M    /home/students/acct3001_02/miniconda3/envs/diffusers/lib/libcublas_static.a
102M    /home/students/acct3001_02/miniconda3/envs/diffusers/lib/libnppif_static.a
99M     /home/students/acct3001_02/miniconda3/envs/diffusers/lib/libnppif.so.11.8.0.86
97M     /home/students/acct3001_02/miniconda3/envs/diffusers/lib/libcurand_static.a
97M     /home/students/acct3001_02/miniconda3/envs/diffusers/lib/libcurand.so.10.3.0.86
91M     /home/students/acct3001_02/miniconda3/envs/diffusers/lib/libcublas.so.11.11.3.6
72M     /home/students/acct3001_02/miniconda3/envs/diffusers/lib/libmkl_core.so.1
68M     /home/students/acct3001_02/miniconda3/envs/diffusers/lib/libnvrtc_static.a
</code></pre>
<p>There are also <code>nsight-compute</code> (at <code>/home/students/acct3001_02/miniconda3/envs/diffusers/nsight-compute</code>) taking another 1GB. I probably haven’t counted all the new CUDA libraries. The total size of CUDA libraries easily add up to 5GB+. This is much larger than before.</p>
<p>My questions would be:</p>
<ol>
<li>Is this a constraint by newer version of CUDA i.e. CUDA 11.7 ?</li>
<li>If it is not, it is possible to exclude unnecessary CUDA libraries for the default installation?</li>
</ol>
<p>Thank you!</p>

ptrblck: <ol>
<li>
<p>No, these libs are not specific to CUDA 11.7, but the build process was switched from the unsupported <code>cudatoolkit</code> to the <code>cuda</code> conda binary.</p>
</li>
<li>
<p>Yes, I believe the PyTorch metapackage could filter our more unnecessary libraries before installing <code>cuda</code>.</p>
</li>
</ol>

gau-nernst: <p>Thank you for your quick reply!</p>
<ol>
<li>What do you mean by “unsupported <code>cudatoolkit</code>”? <code>cudatoolkit</code> is no longer supported?</li>
<li>I suppose this can only be done by the PyTorch team, the users cannot directly do this? Then will the PyTorch team plan to filter out unnecessary libraries in future releases?</li>
</ol>
<p>I tried digging into package dependencies of the conda packages and found the following</p>
<ul>
<li>
<code>pytorch-cuda-11.7</code> only depends on <code>cuda-11.7</code>, while other unnecessary libraries are runtime constrains</li>
<li>However, <code>cuda-11.7</code> depends on (<code>cuda-demo-suite</code>, <code>cuda-runtime</code>, and <code>cuda-toolkit</code>). Among them, <code>cuda-runtime</code> and <code>cuda-toolkit</code> depend on <code>cuda-libraries</code>, which are all of the unnecessary libraries.</li>
</ul>
<p>I guess in that case, the limitation is imposed by how NVIDIA package their CUDA libraries.</p>

truatpasteurdotfr: <p>Request for pytorch packagers: could <code>pytorch-cuda-11.7</code> only requires <code>cuda-toolkit</code> or <code>cuda-runtime</code> or whatever is called the  cuda runtime is named by NVidia? in order du decrease the disk space utilisation?</p>

ptrblck: <p>The nightly binaries should have already reduced the dependencies. Did you check these or could you do it, please?</p>

truatpasteurdotfr: <p>I have been checking the stable pytorch 1.13.1 version and indeed the nightly has greatly improved the disk usage!</p>
<p><code>conda install -y pytorch pytorch-cuda=11.7 -c pytorch -c nvidia</code> from <code>continuumio/miniconda3</code> docker image:</p>
<pre><code class="lang-auto">## Package Plan ##

  environment location: /opt/conda/envs/py310-pytorch-1.13.1

  added / updated specs:
    - pytorch
    - pytorch-cuda=11.7


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    _libgcc_mutex-0.1          |             main           3 KB
    _openmp_mutex-5.1          |            1_gnu          21 KB
    blas-1.0                   |              mkl           6 KB
    bzip2-1.0.8                |       h7b6447c_0          78 KB
    certifi-2022.12.7          |  py310h06a4308_0         150 KB
    cuda-11.7.1                |                0           1 KB  nvidia
    cuda-cccl-11.7.91          |                0         1.2 MB  nvidia
    cuda-command-line-tools-11.7.1|                0           1 KB  nvidia
    cuda-compiler-11.7.1       |                0           1 KB  nvidia
    cuda-cudart-11.7.99        |                0         194 KB  nvidia
    cuda-cudart-dev-11.7.99    |                0         1.1 MB  nvidia
    cuda-cuobjdump-11.7.91     |                0         158 KB  nvidia
    cuda-cupti-11.7.101        |                0        22.9 MB  nvidia
    cuda-cuxxfilt-11.7.91      |                0         293 KB  nvidia
    cuda-demo-suite-12.1.55    |                0         5.0 MB  nvidia
    cuda-documentation-12.1.55 |                0          89 KB  nvidia
    cuda-driver-dev-11.7.99    |                0          16 KB  nvidia
    cuda-gdb-12.1.55           |                0         5.3 MB  nvidia
    cuda-libraries-11.7.1      |                0           1 KB  nvidia
    cuda-libraries-dev-11.7.1  |                0           2 KB  nvidia
    cuda-memcheck-11.8.86      |                0         168 KB  nvidia
    cuda-nsight-12.1.55        |                0       113.6 MB  nvidia
    cuda-nsight-compute-12.1.0 |                0           1 KB  nvidia
    cuda-nvcc-11.7.99          |                0        42.7 MB  nvidia
    cuda-nvdisasm-12.1.55      |                0        47.9 MB  nvidia
    cuda-nvml-dev-11.7.91      |                0          80 KB  nvidia
    cuda-nvprof-12.1.55        |                0         4.8 MB  nvidia
    cuda-nvprune-11.7.91       |                0          64 KB  nvidia
    cuda-nvrtc-11.7.99         |                0        17.3 MB  nvidia
    cuda-nvrtc-dev-11.7.99     |                0        16.9 MB  nvidia
    cuda-nvtx-11.7.91          |                0          57 KB  nvidia
    cuda-nvvp-12.1.55          |                0       114.5 MB  nvidia
    cuda-runtime-11.7.1        |                0           1 KB  nvidia
    cuda-sanitizer-api-12.1.55 |                0        16.7 MB  nvidia
    cuda-toolkit-11.7.1        |                0           1 KB  nvidia
    cuda-tools-11.7.1          |                0           1 KB  nvidia
    cuda-visual-tools-11.7.1   |                0           1 KB  nvidia
    flit-core-3.6.0            |     pyhd3eb1b0_0          42 KB
    gds-tools-1.6.0.25         |                0        40.9 MB  nvidia
    intel-openmp-2023.0.0      |   h9e868ea_25371        15.1 MB
    ld_impl_linux-64-2.38      |       h1181459_1         654 KB
    libcublas-11.10.3.66       |                0       286.1 MB  nvidia
    libcublas-dev-11.10.3.66   |                0       296.4 MB  nvidia
    libcufft-10.7.2.124        |       h4fbf590_0        93.6 MB  nvidia
    libcufft-dev-10.7.2.124    |       h98a8f43_0       197.3 MB  nvidia
    libcufile-1.6.0.25         |                0         763 KB  nvidia
    libcufile-dev-1.6.0.25     |                0          13 KB  nvidia
    libcurand-10.3.2.56        |                0        51.7 MB  nvidia
    libcurand-dev-10.3.2.56    |                0         449 KB  nvidia
    libcusolver-11.4.0.1       |                0        78.7 MB  nvidia
    libcusolver-dev-11.4.0.1   |                0        55.9 MB  nvidia
    libcusparse-11.7.4.91      |                0       151.1 MB  nvidia
    libcusparse-dev-11.7.4.91  |                0       309.5 MB  nvidia
    libffi-3.4.2               |       h6a678d5_6         136 KB
    libgcc-ng-11.2.0           |       h1234567_1         5.3 MB
    libgomp-11.2.0             |       h1234567_1         474 KB
    libnpp-11.7.4.75           |                0       129.3 MB  nvidia
    libnpp-dev-11.7.4.75       |                0       126.6 MB  nvidia
    libnvjpeg-11.8.0.2         |                0         2.2 MB  nvidia
    libnvjpeg-dev-11.8.0.2     |                0         1.9 MB  nvidia
    libstdcxx-ng-11.2.0        |       h1234567_1         4.7 MB
    libuuid-1.41.5             |       h5eee18b_0          27 KB
    mkl-2023.0.0               |   h6d00ec8_25399       171.4 MB
    nsight-compute-2023.1.0.15 |                0       770.2 MB  nvidia
    pip-22.3.1                 |  py310h06a4308_0         2.8 MB
    pytorch-1.13.1             |py3.10_cuda11.7_cudnn8.5.0_0        1.14 GB  pytorch
    pytorch-cuda-11.7          |       h67b0de4_1           3 KB  pytorch
    pytorch-mutex-1.0          |             cuda           3 KB  pytorch
    readline-8.2               |       h5eee18b_0         357 KB
    tbb-2021.7.0               |       hdb19cb5_0         1.6 MB
    tk-8.6.12                  |       h1ccaba5_0         3.0 MB
    typing_extensions-4.4.0    |  py310h06a4308_0          46 KB
    tzdata-2022g               |       h04d1e81_0         114 KB
    zlib-1.2.13                |       h5eee18b_0         103 KB
    ------------------------------------------------------------
                                           Total:        4.28 GB
</code></pre>
<p>compared to  <code>conda install -y pytorch pytorch-cuda=11.7 -c pytorch-nightly -c nvidia</code></p>
<pre><code class="lang-auto">## Package Plan ##

  environment location: /opt/conda/envs/py310-pytorch-nightly

  added / updated specs:
    - pytorch
    - pytorch-cuda=11.7


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    _libgcc_mutex-0.1          |             main           3 KB
    _openmp_mutex-5.1          |            1_gnu          21 KB
    blas-1.0                   |              mkl           6 KB
    bzip2-1.0.8                |       h7b6447c_0          78 KB
    certifi-2022.9.24          |  py311h06a4308_0         155 KB
    cuda-cudart-11.7.99        |                0         194 KB  nvidia
    cuda-cupti-11.7.101        |                0        22.9 MB  nvidia
    cuda-libraries-11.7.1      |                0           1 KB  nvidia
    cuda-nvrtc-11.7.99         |                0        17.3 MB  nvidia
    cuda-nvtx-11.7.91          |                0          57 KB  nvidia
    cuda-runtime-11.7.1        |                0           1 KB  nvidia
    filelock-3.9.0             |          py311_0          20 KB  pytorch-nightly
    intel-openmp-2023.0.0      |   h9e868ea_25371        15.1 MB
    ld_impl_linux-64-2.38      |       h1181459_1         654 KB
    libcublas-11.10.3.66       |                0       286.1 MB  nvidia
    libcufft-10.7.2.124        |       h4fbf590_0        93.6 MB  nvidia
    libcufile-1.6.0.25         |                0         763 KB  nvidia
    libcurand-10.3.2.56        |                0        51.7 MB  nvidia
    libcusolver-11.4.0.1       |                0        78.7 MB  nvidia
    libcusparse-11.7.4.91      |                0       151.1 MB  nvidia
    libffi-3.4.2               |       h6a678d5_6         136 KB
    libgcc-ng-11.2.0           |       h1234567_1         5.3 MB
    libgomp-11.2.0             |       h1234567_1         474 KB
    libnpp-11.7.4.75           |                0       129.3 MB  nvidia
    libnvjpeg-11.8.0.2         |                0         2.2 MB  nvidia
    libstdcxx-ng-11.2.0        |       h1234567_1         4.7 MB
    libuuid-1.41.5             |       h5eee18b_0          27 KB
    mkl-2023.0.0               |   h6d00ec8_25399       171.4 MB
    mpmath-1.2.1               |          py311_0         1.2 MB  pytorch-nightly
    pip-22.2.2                 |  py311h06a4308_0         2.9 MB
    python-3.11.0              |       h7a1cb2a_2        32.6 MB
    pytorch-2.0.0.dev20230301  |py3.11_cuda11.7_cudnn8.5.0_0        1.20 GB  pytorch-nightly
    pytorch-cuda-11.7          |       h778d358_3           7 KB  pytorch-nightly
    pytorch-mutex-1.0          |             cuda           3 KB  pytorch-nightly
    readline-8.2               |       h5eee18b_0         357 KB
    setuptools-65.5.0          |  py311h06a4308_0         1.4 MB
    sympy-1.11.1               |          py311_0        18.8 MB  pytorch-nightly
    tbb-2021.7.0               |       hdb19cb5_0         1.6 MB
    tk-8.6.12                  |       h1ccaba5_0         3.0 MB
    torchtriton-2.0.0+b8b470bc59|            py311        62.7 MB  pytorch-nightly
    typing_extensions-4.1.1    |     pyh06a4308_0          28 KB
    tzdata-2022g               |       h04d1e81_0         114 KB
    wheel-0.37.1               |     pyhd3eb1b0_0          33 KB
    zlib-1.2.13                |       h5eee18b_0         103 KB
    ------------------------------------------------------------
                                           Total:        2.33 GB
</code></pre>
<p>Do you have some ETA fot the diet version to appear? 4.28 GB down to 2.33 GB is quite significant <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
<p>Best regards</p>
<p>Tru</p>

ptrblck: <p>The change landed in the nightly release a few weeks ago and the next stable 2.0 release is now at RC2 while still being finalized with critical bug fixes. I would guess it might take ~2 more weeks of testing and fixing important issues until the release, but that’s just my guess. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>