Performance degration when building from source

yzs: <p>Hi, I’m experiencing performance degradation when trying to compile from source, and I’m wondering if there are some details I’m missing.</p>
<p>Using pytorch v2.0.0 as an example, I used both cuda 11.7 and cudnn 8.5 (consistent with <a href="https://github.com/pytorch/builder/blob/main/common/install_cuda.sh" class="inline-onebox" rel="noopener nofollow ugc">builder/install_cuda.sh at main · pytorch/builder · GitHub</a> in pytorch/builder ), and read carefully about the log of nightly build of the v2.0.0 commit in github actions (<a href="https://github.com/pytorch/pytorch/actions/runs/4379466984/jobs/7667264726" class="inline-onebox" rel="noopener nofollow ugc">[inductor] use triu ref instead of lowering (#96040) (#96462) · pytorch/pytorch@c263bd4 · GitHub</a>).</p>
<p>The final compiling command of my own is</p>
<pre><code class="lang-bash">export USE_NINJA=OFF
export TORCH_NVCC_FLAGS="-Xfatbin -compress-all"
export USE_STATIC_NCCL=1
export TORCH_CUDA_ARCH_LIST='8.0'
export INSTALL_TEST=0
export USE_FBGEMM=1

export CUDA_HOME=/usr/local/cuda-11.7/
export CUDACXX=/usr/local/cuda-11.7/bin/nvcc
export LD_LIBRARY_PATH=/usr/local/cuda-11.7/lib64/:$LD_LIBRARY_PATH
export CUDNN_LIBRARY_PATH=/usr/local/cuda-11.7/lib64/

CMAKE_PREFIX_PATH="$(dirname $(which conda))/../"  python setup.py install
</code></pre>
<p>I found that the pytorch I compiled myself has a performance difference with the pytorch obtained by conda install: using the mnist example on A100, the forward time increased by 5%.</p>

eqy: <p>What commit did you build from source with and were you able to verify that e.g., all library versions (cuDNN, cuBLAS, etc.) were the same between your source build and the upstream wheels?</p>
<p>Would you also be able to share how you are benchmarking MNIST?</p>

yzs: <p>Thanks for the quick reply!</p>
<p>The specific commit is <a href="https://github.com/pytorch/pytorch/commit/c263bd43e8e8502d4726643bc6fd046f0130ac0e" rel="noopener nofollow ugc">c263bd4</a> (the v2.0.0 tag). The <code>torch.__version__</code> returns <code>2.0.0a0+gitc263bd4</code>.</p>
<p>I could confirm that the cuDNN is the same (cuDNN v8.5.0, installed according to the script in pytorch/builder, both returns <code>8500</code> for <code>torch.backends.cudnn.version()</code>). The other CUDA libraries (e.g., cuBlas) should be shipped with CUDA 11.7.0. NCCL is installed automatically by <code>setup.py</code> when setting <code>USE_STATIC_NCCL=1</code> in my case while it may be not relevant.</p>
<p>The mnist example I used is <a href="https://pastebin.com/iZPYnbzp" class="inline-onebox" rel="noopener nofollow ugc">import torchimport argparseimport torch.nn as nnimport torch.nn.functional - Pastebin.com</a>. Use random data as input and add a CUDA event-based timer to record time. It will print average step forward time after each epoch</p>

eqy: <p>Interesting, does profiling with e.g., <code>nsys nvprof</code> show the same kernels for both the source build vs. the prebuilt binaries? If so then we can basically rule out differences due to math libraries.</p>

yzs: <p>Thanks for the hint.</p>
<p>After investigating the kernel launched from nsys, I found that they are launching exactly the same kernel, attached below.</p>
<p>Compiling from source:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/0/d03b6e3a0e72a60a5549d36d8a5b6c6b40901b71.png" data-download-href="https://discuss.pytorch.org/uploads/default/d03b6e3a0e72a60a5549d36d8a5b6c6b40901b71" title="dev"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/0/d03b6e3a0e72a60a5549d36d8a5b6c6b40901b71_2_690x476.png" alt="dev" data-base62-sha1="tI6DSJ9WvLFG0sXAkSF6qzgPyP7" width="690" height="476" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/0/d03b6e3a0e72a60a5549d36d8a5b6c6b40901b71_2_690x476.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/0/d03b6e3a0e72a60a5549d36d8a5b6c6b40901b71_2_1035x714.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/0/d03b6e3a0e72a60a5549d36d8a5b6c6b40901b71_2_1380x952.png 2x" data-dominant-color="EDEDED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">dev</span><span class="informations">1436×991 106 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Installed from conda:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/4/a4f63dbde96507aa0274ccb5e6e4a59a5bce05ce.png" data-download-href="https://discuss.pytorch.org/uploads/default/a4f63dbde96507aa0274ccb5e6e4a59a5bce05ce" title="conda"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/4/a4f63dbde96507aa0274ccb5e6e4a59a5bce05ce_2_690x470.png" alt="conda" data-base62-sha1="nxjSdn819jaDQoWoBi4XLMEQk1w" width="690" height="470" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/4/a4f63dbde96507aa0274ccb5e6e4a59a5bce05ce_2_690x470.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/4/a4f63dbde96507aa0274ccb5e6e4a59a5bce05ce_2_1035x705.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/4/a4f63dbde96507aa0274ccb5e6e4a59a5bce05ce_2_1380x940.png 2x" data-dominant-color="EDEDED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">conda</span><span class="informations">1449×989 108 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Another interesting thing is the cold start overhead between the compiled and the installed differs from each other greatly and could be stably reproduced. Therefore, I believe there should be some differences between them.</p>

eqy: <p>Are the differences present only on cold-start and with just a single layer? I’m curious if it could be caused by e.g., lazy module loading: <a href="https://docs.nvidia.com/cuda/cuda-c-programming-guide/lazy-loading.html?highlight=cuda_module_loading" class="inline-onebox" rel="noopener nofollow ugc">CUDA C++ Programming Guide</a> which was introduced in 11.7.</p>

yzs: <p>Thanks for the explanation. I suppose I am not using lazy loading since I only upgrade CUDA to 11.7, without a R515+ driver.</p>
<p>Leaving the cold start difference back, my primary concern is that are there any differences (e.g., compiling args) between building PyTorch from source with wheels (e.g., from conda)? I believe that if I understand the scripts in pytorch/builder right, the scripts and envs above should exactly match the building procedure, thus producing the same artifact (not including version). Or if there is something I have already missed?</p>

Kehuan_Feng: <p><a class="mention" href="/u/yzs">@yzs</a> I met similar performance issue when building from source. Have you figured out the root cause?</p>

yzs: <p>Not yet. Cannot achieve the same performance.</p>