Move data to gpu, Am I doing it correctly?

witl: <p>In the code below I’m moving all the data to the GPU.</p>
<pre><code class="lang-auto"># Data Loader for easy mini-batch return in training
train_loader = Data.DataLoader(dataset=train_data, batch_size=BATCH_SIZE, shuffle=True, pin_memory=True)

# Move data to GPU
batch_x = []
batch_y = []
for x, y in train_loader:
    batch_x.append(Variable(x.view(-1, 28 * 28)).to(device))
    batch_y.append(Variable(x.view(-1, 28 * 28)).to(device))

gpu_data = zip(batch_x, batch_y)
</code></pre>
<p>Probably there is a way I can accomplish it without looping through the batches. Can someone point it out?</p>

shivammehta007: <p>Just output from</p>
<pre><code class="lang-auto">for x, y in train_loader:
    x  = x.to(device)
    y = y.to(device)
    # Rest of training loop
</code></pre>
<p>Is enough to move the data to GPU and train with it, you don’t need to put it into a list and then later use it it is not a right move as all data will be in your memory and the whole point of batches will be ruined!</p>

witl: <p>You are not correct when you say that “the whole point of batches will be ruined”. That’s not the case at all. There are many other situations/advantages of using batches.</p>
<p>In terms of performance, you only would iteratively move your batches to your GPU if and only if your dataset does not fit into your GPU’s memory. If it fits, you can just move everything before training so you dont lose time waiting for the data to be moved at every batch iteration.</p>
<p>Moving data before training:</p>
<blockquote>
<p>Autoencoder 0, mean loss: 0.049266365027106775, time: 5.8063098430633545<br>
Classifier 0, mean loss: 0.6276001909743748, time: 5.8063098430633545<br>
Autoencoder 1, mean loss: 0.02765635310459747, time: 5.40066180229187<br>
Classifier 1, mean loss: 0.30130706248538836, time: 5.40066180229187<br>
Autoencoder 2, mean loss: 0.025990864579706813, time: 5.43626651763916<br>
Classifier 2, mean loss: 0.24578058288327412, time: 5.43626651763916</p>
</blockquote>
<p>Moving one batch at a time</p>
<blockquote>
<p>Autoencoder 0, mean loss: 0.05069048443773408, time: 7.177410364151001<br>
Classifier 0, mean loss: 0.5955254633638905, time: 7.177410364151001<br>
Autoencoder 1, mean loss: 0.02769256935619723, time: 6.592367887496948<br>
Classifier 1, mean loss: 0.2981912835336316, time: 6.592367887496948<br>
Autoencoder 2, mean loss: 0.025816337348444504, time: 6.52787971496582<br>
Classifier 2, mean loss: 0.2456918565480948, time: 6.52787971496582</p>
</blockquote>

shivammehta007: <aside class="quote no-group" data-username="witl" data-post="3" data-topic="113806">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/258eb7/48.png" class="avatar"> witl:</div>
<blockquote>
<p>You are not correct when you say that “the whole point of batches will be ruined”. That’s not the case at all. There are many other situations/advantages of using batches.</p>
</blockquote>
</aside>
<p>As per my understanding, the point of batching is when you cannot move your whole population (dataset in this case), so you take a population sample (a batch) and estimate parameters based on this iteratively as the expectation of the sample parameter will give you population parameters.</p>
<p>So you are perfectly right, if you can move all the dataset onto GPU will be faster and even better at estimating parameters, that is the best case but not a practical one <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Also, when I say the whole point of batching is ruined maybe I should have reformulated it as, ideally, you only need batches when you cannot fit your dataset onto GPU at one go. If you can, that is the best therefore the whole point of a batch is useless.</p>

ptrblck: <p>While the performance might increase in trade of more memory usage on the GPU, I wouldn’t argue that batching is completely “useless”, as <a class="mention" href="/u/witl">@witl</a> would lose all random transformations.<br>
Iteration the dataset once, will apply the transformations (if used) once for each sample and store move it to the GPU.<br>
Of course it depends on the use case, if a performance (accuracy) drop would be seen, but this should be kept in mind when using this “static” approach.</p>

shivammehta007: <p>Aha! That that a good catch, but I have a flowing question If we have the capability to process the data at once. Can’t we first do all the transformations and then put the final data to GPU at once. Will it be any different than the batch approach?</p>

ptrblck: <p>You could also perform all transformations once and push the data afterwards to the GPU.<br>
Note however, that also in your approach each sample would be “randomly” transformed only once, since the data on the GPU wouldn’t be transformed anymore.</p>