What is happening

Diego_Lovison: <p>I have the following code</p>
<pre><code class="lang-auto">device = "cuda:0"
model.to(device)
for epoch in range(20000):
    for data in dataloader:
      inputs = data.to(device)
      labels = data.to(device)
  
      optimizer.zero_grad()
  
      outputs = model(inputs)
      loss = criterion(outputs, labels)
      loss.backward()
  
      optimizer.step()
</code></pre>
<p>I am doing <code>.to(device)</code> every iteration…<br>
I would like to understand internally what Pytorch is doing.</p>
<ul>
<li>In the next epoch, what is going to happen with X and y? will Pytorch send again to gpu or will it ignore?</li>
<li>If I have data that is greater then my GPU memory. How Pytorch is handling that?</li>
<li>If my GPU memory is greater than my data. How can I send the data to gpu once?</li>
</ul>
<p>I tried to search in the forum but wasn’t able to find answers.<br>
Thanks.</p>

ptrblck: <aside class="quote no-group" data-username="Diego_Lovison" data-post="1" data-topic="198576">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/diego_lovison/48/62231_2.png" class="avatar"> Diego_Lovison:</div>
<blockquote>
<p>In the next epoch, what is going to happen with X and y? will Pytorch send again to gpu or will it ignore?</p>
</blockquote>
</aside>
<p><code>X</code> and <code>y</code> are undefined so unclear what these variables contain.<br>
<code>inputs</code> and <code>labels</code> will be replaced with the new <code>data</code> (in fact with the same data) and will reuse the same memory if possible.</p>
<aside class="quote no-group" data-username="Diego_Lovison" data-post="1" data-topic="198576">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/diego_lovison/48/62231_2.png" class="avatar"> Diego_Lovison:</div>
<blockquote>
<p>If I have data that is greater then my GPU memory. How Pytorch is handling that?</p>
</blockquote>
</aside>
<p>It will raise an Out of memory error.</p>
<aside class="quote no-group" data-username="Diego_Lovison" data-post="1" data-topic="198576">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/diego_lovison/48/62231_2.png" class="avatar"> Diego_Lovison:</div>
<blockquote>
<p>If my GPU memory is greater than my data. How can I send the data to gpu once?</p>
</blockquote>
</aside>
<p>I don’t understand this question, since you are handling batches of data. If you want to move the entire dataset to the GPU, you can use <code>.to(device)</code> in the same way as is already done for batches.</p>

Diego_Lovison: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="198576">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>inputs</code> and <code>labels</code> will be replaced with the new <code>data</code> (in fact with the same data) and will reuse the same memory if possible.</p>
</blockquote>
</aside>
<p>Given the following scenario,</p>
<pre><code class="lang-auto">device = 'cuda'
for epoch in range(20000):
    for i, data in enumerate(dataloader):
      print(epoch)
      print(i)
      inputs, labels = data
      inputs = inputs.to(device)
      labels = labels.to(device)
</code></pre>
<p>When <code>epoch</code> equals 0 and <code>i</code> equals 0, <code>inputs</code> and <code>labels</code> will be copied from cpu to gpu.<br>
When <code>epoch</code> equals 0 and <code>i</code> equals 1, <code>inputs</code> and <code>labels</code> will be copied from cpu to gpu.</p>
<p>When the data that I sent to gpu in <code>epoch</code> equals 0 and <code>i</code> equals 0 will be removed from the gpu?</p>

ptrblck: <p>I’m still unsure if I’m misunderstanding the question, but the variables will be replaced and the previously used memory will be reused via the internal cache if possible:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch

data = torch.randn(1024, 1024, device="cpu")
print("allocated {:.3f}, in cache: {:.3f}".format(
    torch.cuda.memory_allocated() / 1024**2, torch.cuda.memory_reserved() / 1024**2))
# allocated 0.000, in cache: 0.000

for i in range(3):
    x = data.to("cuda")

    print("i: {}, allocated {:.3f}, in cache: {:.3f}".format(
        i, torch.cuda.memory_allocated() / 1024**2, torch.cuda.memory_reserved() / 1024**2))

# i: 0, allocated 4.000, in cache: 20.000
# i: 1, allocated 4.000, in cache: 20.000
# i: 2, allocated 4.000, in cache: 20.000
</code></pre>
<p>You are not increasing the allocated memory since the previously used tensors are not referenced anymore (unless you explicitly create a reference to them of course).</p>

Diego_Lovison: <p>Exactly the answer that I was looking for.<br>
Thanks for the example as well.<br>
I appreciate your patience.</p>