Feedback about PyTorch profiling - too many cudaGetDevice() calls

Konstantin_Burlachen: <p>During profiling PyTorch 1.9.0 we realize that there is a lot of the C++ code and 3rd party libraries that it used. At the moment of 2021, consists of $5.6M$ lines of code and it’s pretty big.</p>
<p>One set that we have used was a single GPU, and in the case of (near) standard training ResNet type like model on CIFAR-10 during using NVVP we have observed that there are too much <code>cudaGetDevice</code> calls. During training number of calls of “cudaGetDevice()” was totally 6 115 338 calls. And it eats <strong>5 seconds</strong> from all <em>110 seconds</em> of training, which 5% of the time.</p>
<p>There are places within a PyTorch where this is called. Instead of calling that it’s better to have just control over the variable which states which is a current GPU.</p>
<p>We realize that it’s in complicated software such as pooling may have a place to be. Of course, if you have control it’s better not to poll it.</p>
<p>But in any case - in the case of using a single GPU all cudaGetDevice() calls are waste of time and they are not needed, because all libraries contexts and all memory and all stream should be associated with a single device.</p>

tom: <p>Hi!</p>
<p>With the help of the one and only <a class="mention" href="/u/ptrblck">@ptrblck</a> : this has come up before and it seems that it is mostly a profiling artefact:</p>
<aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/18048">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/18048" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/18048" target="_blank" rel="noopener nofollow ugc">Excessive call to cudaGetDevice and  cudaSetDevice</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-03-15" data-time="00:27:00" data-timezone="UTC">12:27AM - 15 Mar 19 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2019-03-18" data-time="19:46:20" data-timezone="UTC">07:46PM - 18 Mar 19 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/omry" target="_blank" rel="noopener nofollow ugc">
          <img alt="omry" src="https://avatars.githubusercontent.com/u/376455?v=4" class="onebox-avatar-inline" width="20" height="20">
          omry
        </a>
      </div>
    </div>

    <div class="labels">
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">## 🐛 Bug

While profiling my code, I noticed millions of calls to cudaGetDevic<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">e which were slowing down my 
code significantly (it's an RL use case and the model I am using is small).
I have a minimal repro that demonstrate that a single forward call on nn.Linear() is triggering 11 cudaGetDevice() calls and 3 cudaSetDevice() calls.

## To Reproduce
See attached code snippet.

Steps to reproduce the behavior:
/usr/local/cuda/bin/nvprof --profile-from-start off python  cudagetdevice.py
Output:

```
$ /usr/local/cuda/bin/nvprof --profile-from-start off python  cudagetdevice.py 

==112656== NVPROF is profiling process 112656, command: python cudagetdevice.py
==112656== Warning: Profiling results might be incorrect with current version of nvcc compiler used to compile cuda app. Compile with nvcc compiler 9.0 or later version to get correct profiling results. Ignore this warning if code is already compiled with the recommended nvcc version 
==112656== Profiling application: python cudagetdevice.py
==112656== Profiling result:
            Type  Time(%)      Time     Calls       Avg       Min       Max  Name
 GPU activities:   73.95%  5.0880us         1  5.0880us  5.0880us  5.0880us  sgemm_32x32x32_NT_vec
                   26.05%  1.7920us         1  1.7920us  1.7920us  1.7920us  _ZN2at4cuda75_GLOBAL__N__51_tmpxft_00005244_00000000_11_Copy_compute_75_cpp1_ii_dd3fb9a321kernelPointwiseApply2IZN75_GLOBAL__N__51_tmpxft_00005244_00000000_11_Copy_compute_75_cpp1_ii_dd3fb9a36CopyOpIffE5applyERNS_6TensorERKS6_EUlRfRKfE_ffjLi1ELi2ELi1EEEvNS0_6detail10TensorInfoIT0_T2_EENSF_IT1_SH_EESH_T_
      API calls:   99.99%  1.03481s         2  517.41ms  12.000us  1.03480s  cudaLaunchKernel
                    0.01%  67.541us        11  6.1400us  4.5000us  13.500us  cudaGetDevice
                    0.00%  14.583us         3  4.8610us  4.5000us  5.5830us  cudaSetDevice
                    0.00%  9.0410us         2  4.5200us  4.5000us  4.5410us  cudaGetLastError
```

## Expected behavior
Number of calls to cudaGetDevice should be minimal, I would argue that it should not be called at most once, if that.

## Environment

PyTorch version: 1.0.1.post2
Is debug build: No
CUDA used to build PyTorch: 10.0.130

OS: Ubuntu 16.04.6 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609
CMake version: version 3.5.1

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration: 
GPU 0: Quadro P5000
GPU 1: Quadro P5000

Nvidia driver version: 418.39
cuDNN version: Could not collect

Versions of relevant libraries:
[pip] numpy==1.15.4
[pip] torch==1.0.1.post2
[pip] torchfile==0.1.0
[pip] torchvision==0.2.2
[conda] blas                      1.0                         mkl  
[conda] mkl                       2019.1                      144  
[conda] mkl_fft                   1.0.6            py36hd81dba3_0  
[conda] mkl_random                1.0.2            py36hd81dba3_0  
[conda] pytorch                   1.0.1           py3.6_cuda10.0.130_cudnn7.4.2_2    pytorch
[conda] torchfile                 0.1.0                     &lt;pip&gt;
[conda] torchvision               0.2.2                      py_3    pytorch


 - PyTorch Version (e.g., 1.0):
1.0.1
 - OS (e.g., Linux): Ubuntu 16.04
 - How you installed PyTorch (`conda`, `pip`, source):
pip inside a conda env
 - Build command you used (if compiling from source):
 - Python version:
3.6
 - CUDA/cuDNN version:
10.0
 - GPU models and configuration:
Quadro P5000</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Of course, you could re-run the experiment of replacing all calls to cudaGetDevice/cudaSetDevice with a noop and see if things have changed.</p>
<p>Best regards</p>
<p>Thomas</p>