TypeError on torch.cuda.list_gpu_processes()

JiWeon_Park: <pre data-code-wrap="python"><code class="lang-python">print(torch.version.cuda)
print(torch.backends.cudnn.version())
print(torch.cuda.is_available())
print(torch.cuda.get_device_name(torch.cuda.current_device()))
print(torch.cuda.list_gpu_processes())
</code></pre>
<pre><code class="lang-auto">12.4
90100
True
NVIDIA GeForce RTX 3060
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[10], line 5
      3 print(torch.cuda.is_available())
      4 print(torch.cuda.get_device_name(torch.cuda.current_device()))
----&gt; 5 print(torch.cuda.list_gpu_processes())

File ~\anaconda3\envs\kaggle-torch\lib\site-packages\torch\cuda\memory.py:655, in list_gpu_processes(device)
    653 for p in procs:
    654     if not torch.version.hip:
--&gt; 655         mem = p.usedGpuMemory / (1024 * 1024)
    656         pid = p.pid
    657     else:

TypeError: unsupported operand type(s) for /: 'NoneType' and 'int'
</code></pre>
<p>I don’t understand why this error is occurring. Could you please help me resolve it?</p>

Tony-Y: <p>I found some information about this issue:</p>
<aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/64491">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/64491" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue" data-github-private-repo="false">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/64491" target="_blank" rel="noopener nofollow ugc">TypeError when using torch.cuda.list_gpu_processes() on Windows with the WDDM driver</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2021-09-03" data-time="12:45:51" data-timezone="UTC">12:45PM - 03 Sep 21 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/sluki" target="_blank" rel="noopener nofollow ugc">
          <img alt="sluki" src="https://avatars.githubusercontent.com/u/10222653?v=4" class="onebox-avatar-inline" width="20" height="20">
          sluki
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: cuda
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          triaged
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">## Issue description
torch.cuda.list_gpu_processes() can raise TypeError becaus<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">e p.usedGpuMemory can be None.

TypeError: unsupported operand type(s) for /: 'NoneType' and 'int'

Quoting from pynvml library (nvml.py:948):
"On Windows with the WDDM driver, usedGpuMemory is reported as None
Code that processes this structure should check for None, I.E."

## Code example
import torch
torch.cuda.list_gpu_processes()  # execute on Windows with the WDDM driver

## System Info
PyTorch version: 1.9.0+cu111
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A
OS: Microsoft Windows 10 Enterprise
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A
Python version: 3.9.6 (tags/v3.9.6:db3ff76, Jun 28 2021, 15:26:21) [MSC v.1929 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.18363-SP0
Is CUDA available: True
CUDA runtime version: 10.0.130
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 2070
Nvidia driver version: 471.41
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Versions of relevant libraries:
[pip3] efficientnet-pytorch==0.6.3
[pip3] numpy==1.21.1
[pip3] pytorch-lightning==1.4.0
[pip3] segmentation-models-pytorch==0.2.0
[pip3] torch==1.9.0+cu111
[pip3] torch-inspect==0.0.3
[pip3] torchaudio==0.9.0
[pip3] torchmetrics==0.4.1
[pip3] torchvision==0.10.0+cu111
[conda] Could not collect


cc @ngimel</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<aside class="onebox githubblob" data-onebox-src="https://github.com/nicolargo/nvidia-ml-py3/blob/11b14768bbd0076c61da077490d79327a1bb2589/pynvml.py#L479-L480">
  <header class="source">

      <a href="https://github.com/nicolargo/nvidia-ml-py3/blob/11b14768bbd0076c61da077490d79327a1bb2589/pynvml.py#L479-L480" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/nicolargo/nvidia-ml-py3/blob/11b14768bbd0076c61da077490d79327a1bb2589/pynvml.py#L479-L480" target="_blank" rel="noopener nofollow ugc">nicolargo/nvidia-ml-py3/blob/11b14768bbd0076c61da077490d79327a1bb2589/pynvml.py#L479-L480</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="479" style="counter-reset: li-counter 478 ;">
          <li># On Windows with the WDDM driver, usedGpuMemory is reported as None</li>
          <li># Code that processes this structure should check for None, I.E.</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>