Creating Pytorch variables directly on GPU somehow uses up lots of CPU during creation

Santosh-Gupta: <p>I need to create variables directly on the GPU because I am very limited in my CPU ram.</p>
<p>I found the method to do this here</p>
<aside class="quote" data-post="3" data-topic="2128">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mderakhshani/48/1735_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-create-a-tensor-on-gpu-as-default/2128/3">How to create a tensor on GPU as default</a> 
  </div>
  <blockquote>
    Hi, here is my code: 

import torch 
torch.set_default_tensor_type(‘torch.cuda.FloatTensor’) 
t = torch.rand(1,3,24,24) 

and it caused bellow error: 

TypeError                                 Traceback (most recent call last) 
 in () 
----&gt; 1 t = torch.rand(1,3,24,24) 
TypeError: Type torch.cuda.FloatTensor doesn’t implement stateless methods 

Any idea for this type of error?
  </blockquote>
</aside>

<p>Which mentions using</p>
<p><code>torch.set_default_tensor_type(‘torch.cuda.FloatTensor’)</code></p>
<p>However, when I tried</p>
<pre><code class="lang-auto">torch.set_default_tensor_type('torch.cuda.FloatTensor')
pytorchGPUDirectCreate = torch.FloatTensor(20000000, 128).uniform_(-1, 1).cuda()
</code></pre>
<p>It still seemed to take up mostly CPU RAM, before being transferred to GPU ram.</p>
<p>I am using Google Colab. To view RAM usage during the variable creation process, after running the cell, go to Runtime → Manage Sessions</p>
<p>With and without using <code>torch.set_default_tensor_type('torch.cuda.FloatTensor')</code> , the CPU RAM bumps up to 11.34 GB while GPU ram stays low, and then GPU RAM goes to 9.85 and CPU ram goes back down.</p>
<p>It seems that <code>torch.set_default_tensor_type(‘torch.cuda.FloatTensor’)</code> didn’t make a difference</p>
<p>For convenience here’s a direct link to a notebook anyone can directly run</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1LxPMHl8yFAATH0i0PBYRURo5tqj0_An7">
  <header class="source">
      <img src="https://ssl.gstatic.com/colaboratory-static/common/2580b6a0a5b3a23f1efbba8a2eebb5a3/img/favicon.ico" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1LxPMHl8yFAATH0i0PBYRURo5tqj0_An7" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img width="260" height="260" src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar">

<h3><a href="https://colab.research.google.com/drive/1LxPMHl8yFAATH0i0PBYRURo5tqj0_An7" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


juanmed: <p>Hi,</p>
<p>You can directly create a tensor on a GPU by using the <code>device</code> argument:</p>
<pre><code class="lang-python">device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
pytorchGPUDirectCreate = torch.rand(20000000, 128, device = device).uniform_(-1, 1).cuda()
</code></pre>
<p>I just tried this in your notebook and got RAM 1.76GB used and GPU 9.86GB. Still, a lot of RAM is used but that ~10GB less than originally.</p>
<p>Hope it helps!</p>

Santosh-Gupta: <p>Thanks! I am wondering if the <code>.cuda()</code> is still necessary since you’re already specifying the device on initialization .</p>