From source installation of QUADRO RTX6000 slower then conda installation

username: thomasB
cooked: <p>Hi all,</p>
<p>We recently got new server nodes with QUADRO RTX6000 GPUs and I have some problems of getting pytorch to it’s top speed on them. Weirdly the from source installation is partially slower then the conda installations. I benchmarked one forward + backward pass of a few stacks of 2d and 3d convolutions. For the 2d convolutions I could see that for both with and without the use of amp the performance was twice as slow using from source installation (CUDA 11.1, cudnn 8.0.4, pytorch 1.8) compared to the conda installation (cudatoolkit=11.0, pytorch=1.7).</p>
<p>I could compare the running times against my workstation that has a GEFORCE RTX 2080 ti. The 2d convolutions of the conda installation of the quadro card are a bit faster then the from source installation using the geforce card. This is to be expected as they that the same architecture and the quardo has slightly more cuda kernels.</p>
<p>However I mostly use 3d convolutions and amp in my research. Here the from source installation was the fastest on both cards, but the quadro is 2.5 times slower then the geforce card.</p>
<p>When I compare conda installations on both cards I can see that the quadro is constantly a bit faster then the geforce card, just the from source installation on the quadro is why to slow.</p>
<p>Does anyone have an idea what the reason behind this is? Is it possible that something is wrong with the cuda compiler? We had problems with compiling cuda code on an other end. As this is a departmental machine we’re having a range of different cuda modules.</p>
<p>Thanks for the help!</p>

username: ptrblck
cooked: <p>The <code>1.7.1</code> binaries use cudnn8.0.5.39, which I assume is the reason for the main difference between your setup and the binaries.<br>
Try to install this particular version for a proper comparison or the latest one (cudnn8.1).</p>

username: thomasB
cooked: <p>Thanks! In the meanwhile I figured out that it is not the version of cudnn, but that my from source installation didn’t recognize cudnn. torch.backends.cudnn.is_available() is False for all my attempts of in installing torch from source and True for the conda installations.<br>
I installed pytorch from source by creating a new conda environment with python3.8, loading a cuda-11.0.2 module following the steps in the installation instructions I exported CUDA_HOME, CUDNN_LIB_DIR, CUDNN_LIBRARY and CUDNN_INCLUDE before executing “python setup.py install”. The installation works and cuda is available, but cudnn not. Any ideas how to fix it?</p>

username: ptrblck
cooked: <p>Did you follow the <a>cudnn install guide</a><a href="https://docs.nvidia.com/deeplearning/cudnn/install-guide/index.html" class="inline-onebox">Installation Guide :: NVIDIA Deep Learning cuDNN Documentation</a>? If so, could you download the cudnn samples and try to build and execute them?<br>
If PyTorch isn’t able to find cudnn, I guess the installation wasn’t successful.</p>

username: thomasB
cooked: <p>I couldn’t go through the full installation from your link as it requires root access. Is there a way to tell torch where the cudnn library is after import?<br>
I think the solution you suggest requires a local installation of cuda directly on that machine. The system administrators would prefer us working with cuda/cudnn modules instead.<br>
EDIT: if it is not possible to install pytorch from source using cuda cudnn modules and having it recognise cudnn, which are the versions of CUDA and cuDNN you would suggest using for an up to date from source installation? The newest that are compatible with torch?<br>
Thanks for the great help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ptrblck
cooked: <p>I’m unsure what you mean by CUDA and cudnn “modules”. A source build would require a local CUDA toolkit installation including the compiler (<code>nvcc</code>).<br>
Yes, the latest CUDA and cudnn versions are compatible with the current PyTorch master branch, but note some issues we’ve found, such as:</p><aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/51992" target="_blank" rel="noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/51992" target="_blank" rel="noopener">test_optim.py failures for CUDA 11.2 on CI</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2021-02-09" data-time="20:31:32" data-timezone="UTC">08:31PM - 09 Feb 21 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/janeyx99" target="_blank" rel="noopener">
          <img alt="janeyx99" src="https://avatars.githubusercontent.com/u/31798555?v=4" class="onebox-avatar-inline" width="20" height="20">
          janeyx99
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🐛 Bug
The following tests in test_optim.py fail in CI on CUDA 11.2 in linux and windows.
test_adadelta
test_adam
test_adamw
test_multi_tensor_optimizers
test_rmsprop
To Reproduce
Steps to reproduce the behavior:
Run...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">high priority</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: ci</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: cuda</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triage review</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triaged</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>While it seems to be a CUDA11.2 compiler issue, we’ve reproduced an illegal memory access internally caused in the <code>multi_tensor_apply</code> implementation, which is now visible by using more aggressive optimizations in the compiler.</p>

username: thomasB
cooked: <p>Sorry for the delay in getting back. With modules I mean environment modules:<br>
<a href="http://modules.sourceforge.net/" class="onebox" target="_blank" rel="noopener nofollow ugc">http://modules.sourceforge.net/</a><br>
I could find out that the reason the from source installation with slower was indeed that it didn’t use cuDNN. I could download cuDNN and set the respective pathes (CUDNN_LIBRARY and CUDNN_INCLUDE_DIR) before executing setup.py. However when loading an environment module and exporting the CUDA_HOME pytorch was not finding a cudnn file after installation even with the cudnn pathes set. When using a local installation of cuda (11.0 in our case) we didn’t have this problem.<br>
Anyways we now have the computational speed we want, just the from source installation with cuda modules doesn’t seem to work.<br>
Thanks for the help!</p>

username: thomasB
cooked: <p>EDIT: I managed to install pytorch from source using only environment modules. One obviously needs to load the same environment modules again before using torch, but also during installation one has to set CUDNN_ROOT to point to the cudnn folder containing “include” and “lib64”. Maybe this variable should be added to the setup.py?</p>

username: ptrblck
cooked: <p>By default the cudnn installation would be located <code>/usr/local/cuda/include/</code> and <code>/usr/local/cuda/lib64/</code>, which are the standard CUDA paths (or would be set via <code>CUDA_HOME</code>). In case your cudnn installation uses a different path, you would most likely need to set the cudnn path manually. In the former case, the manual setting of <code>CUDNN_ROOT</code> is not needed.</p>