Pin_memory() on variables?

username: ulgus
cooked: <p>Hi there,</p>
<p>this is a newbie question (sorry…). I’m trying to speed-up my GPU training. The <a href="http://pytorch.org/docs/master/notes/cuda.html" rel="noopener nofollow ugc">docs</a> recommend pin_memory(), which works with tensors, but not with variables (torch version 0.3.0):</p>
<blockquote>
<p>import torch<br>
from torch.autograd import Variable<br>
dtype = torch.FloatTensor</p>
<p>my_tensor = torch.randn(10,10).type(dtype)<br>
my_variable = Variable(torch.randn(10,10).type(dtype), requires_grad=False)</p>
<p>my_tensor.cuda() # → works<br>
my_tensor.pin_memory() # → works</p>
<p>my_variable.cuda() # → works<br>
my_variable.pin_memory() # → does not work</p>
</blockquote>
<p>gives me the error</p>
<blockquote>
<pre><code>Traceback (most recent call last):
  File "check.py", line 14, in &lt;module&gt;
    my_variable.pin_memory() # -&gt; does not work
  File "/usr/local/lib/python2.7/dist-packages/torch/autograd/variable.py", line 67, in __getattr__
    return object.__getattribute__(self, name)
AttributeError: 'Variable' object has no attribute 'pin_memory'
</code></pre>
</blockquote>
<p>I wonder why this is… As my_variable.cuda() allows me to send Variables to GPU (just like tensors), I would expect pin_memory() to work for Variables too.</p>
<ol>
<li>Is my understanding fundamentally wrong here?</li>
<li>What’s the <em>correct</em> way of pinning memory (a) for all of a model’s variables/tensors, (b) when only pinning some of the variables? (A web search gives me only examples in combination with DataLoader(), which does not apply in my case).</li>
</ol>
<p>Thanks a lot!</p>

username: smth
cooked: <p>you can do <code>my_variable.data.pin_memory()</code> for now. But do make sure you understand why you are pinning memory, easy to fall into a trap there. Pinning memory is only useful for CPU Tensors that have to be moved to the GPU. So, pinning all of a model’s variables/tensors doesn’t make sense at all.</p>

username: ulgus
cooked: <p>Hi there,</p>
<p>thanks for the quick reply -</p>
<blockquote>
<p>Pinning memory is only useful for CPU Tensors that have to be moved to the GPU.</p>
</blockquote>
<p>OK, I understand now that this is about moving input data to the GPU.</p>
<p>But what about the model? Am I correct that the model (and its parameters + operations) reside / are carried out on the GPU anyway?</p>
<p>And if not: How can I taylor which operations to carry out on the GPU? (sorry if this is a stupid question, but I haven’t been able to dig up proper documentation for that…).</p>

username: smth
cooked: <aside class="quote no-group" data-username="ulgus" data-post="3" data-topic="11874">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/u/c37758/48.png" class="avatar"> ulgus:</div>
<blockquote>
<p>Am I correct that the model (and its parameters + operations) reside / are carried out on the GPU anyway?</p>
</blockquote>
</aside>
<p>Yes, as long as you have <code>model.cuda()</code> somewhere.</p>

username: ulgus
cooked: <p>Cool - thanks for the quick reply!</p>

username: aifartist
cooked: <blockquote>
<p>So, pinning all of a model’s variables/tensors doesn’t make sense at all.</p>
</blockquote>
<p>Yes, it does.  I have a complex application which has a number of Tensors which I have no idea which ones and how often there are copied back and forth from the GPU.  Before I go through 10’s of thousands of lines of code I didn’t write to see if any improvements can be make I’d like to just test and see what happens.  Do I realize the risks of pinning a lot of memory.  Yes, I am a perf expert and can judge the footprint of the process relative to the free memory I have and at most risk crashing my system.</p>
<p>If I have to I’ll binary patch the assembly code using “gdb -w” on the function at::_ops::is_pinned::call and see what happens.  Is this a bad thing if I suppress the pin/extra copy if the memory page gets swapped out?  Yes, it is bad and will probably segv.  99% chance it won’t and I’ll have some idea whether ANY speed up is possible without spending 3 days wading through large amounts of app code to only find it doesn’t help.</p>