GPU consumption

Mohamed_Farag: <p>Hi,</p>
<p>I have a question I loaded the data in two different ways from numpy array, however there is a one which consumes more GPU memory compared to the another one</p>
<p>First method:</p>
<pre><code class="lang-auto">X_train = torch.from_numpy(X_train).cuda() # transform to torch tensor
y_train = torch.from_numpy(y_train).cuda()

X_val = torch.from_numpy(X_val).cuda()
y_val = torch.from_numpy(y_val).cuda()

X_train = X_train.view(-1, 3, 480, 480).float()
X_val = X_val.view(-1, 3, 480, 480).float()
</code></pre>
<p><strong>Consumption = 12.4 / 15.0 GB</strong></p>
<p>Second one:</p>
<pre><code class="lang-auto">X_train = torch.FloatTensor(X_train).cuda()
X_val = torch.FloatTensor(X_val).cuda()
y_train = torch.LongTensor(y_train).cuda()
y_val = torch.LongTensor(y_val).cuda()

X_train = X_train.view(-1, 3, 480, 480)
X_val = X_val.view(-1, 3, 480, 480)
</code></pre>
<p><strong>Consumption = 9.5 / 15.0 GB</strong></p>
<p>May anyone help <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"> ?</p>
<p>Also after seeing this I am too interested to understand more about the hardware, so may anyone please recommend books\courses regarding the hardware architecture?</p>
<p>Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/heart.png?v=12" title=":heart:" class="emoji" alt=":heart:" loading="lazy" width="20" height="20"></p>

Mohamed_Farag: <p>Also less RAM <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"> , I don’t know why</p>

ptrblck: <p>The allocated and used memory will be the same, but the cache might be different (and thus also the peak memory usage) as you are potentially moving the <code>float64</code> tensors to the GPU first and are transforming them to <code>float32</code> afterwards in the first example while you are directly using the (deprecated) <code>FloatTensor</code> constructor in the second example and are thus transforming the data on the CPU.<br>
Here is a small example:</p>
<pre><code class="lang-python">import torch
import numpy as np

print(torch.cuda.memory_allocated()/1024**2)
# 0.0
print(torch.cuda.memory_reserved()/1024**2)
# 0.0

X_train = np.random.randn(10, 3, 480, 480)
y_train = np.random.randn(10, 3, 480, 480)
X_val = np.random.randn(10, 3, 480, 480)
y_val = np.random.randn(10, 3, 480, 480)

X_train = torch.from_numpy(X_train).cuda() # transform to torch tensor
y_train = torch.from_numpy(y_train).cuda()

X_val = torch.from_numpy(X_val).cuda()
y_val = torch.from_numpy(y_val).cuda()

X_train = X_train.view(-1, 3, 480, 480).float()
X_val = X_val.view(-1, 3, 480, 480).float()

print(torch.cuda.memory_allocated()/1024**2)
# 158.203125
print(torch.cuda.memory_reserved()/1024**2)
# 244.0


## 

import torch
import numpy as np

print(torch.cuda.memory_allocated()/1024**2)
# 0.0
print(torch.cuda.memory_reserved()/1024**2)

X_train = np.random.randn(10, 3, 480, 480)
y_train = np.random.randn(10, 3, 480, 480)
X_val = np.random.randn(10, 3, 480, 480)
y_val = np.random.randn(10, 3, 480, 480)


X_train = torch.FloatTensor(X_train).cuda()
X_val = torch.FloatTensor(X_val).cuda()
y_train = torch.LongTensor(y_train).cuda()
y_val = torch.LongTensor(y_val).cuda()

X_train = X_train.view(-1, 3, 480, 480)
X_val = X_val.view(-1, 3, 480, 480)

print(torch.cuda.memory_allocated()/1024**2)
# 158.203125
print(torch.cuda.memory_reserved()/1024**2)
# 164.0
</code></pre>