Fully utilize GPU memory for trainng

Flock1: <p>Hi,</p>
<p>I am training a transformer and I see that the training is only using 1GB memory out of 8, which is obviously training very slowly. I want to know if there’s a way I can parallelize the training on the same GPU. I read posts like <a href="https://towardsdatascience.com/distributed-neural-network-training-in-pytorch-5e766e2a9e62" rel="noopener nofollow ugc">these</a> but they seem to talk about multiple GPUs. I also thought of increasing the batch size but that might lead to an efficient model.</p>
<p>So is there a way I can increase the speed by parallelising?</p>

ptrblck: <aside class="quote no-group" data-username="Flock1" data-post="1" data-topic="125603">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/flock1/48/12220_2.png" class="avatar"> Flock1:</div>
<blockquote>
<p>I also thought of increasing the batch size but that might lead to an efficient model.</p>
</blockquote>
</aside>
<p>I’m not sure I understand the concern of creating an “efficient model”. Could you explain, why this wouldn’t work or what your concern is?</p>
<aside class="quote no-group" data-username="Flock1" data-post="1" data-topic="125603">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/flock1/48/12220_2.png" class="avatar"> Flock1:</div>
<blockquote>
<p>So is there a way I can increase the speed by parallelising?</p>
</blockquote>
</aside>
<p>The used GPU memory is independent from the hardware compute resources used to parallelize the computation (same as you don’t need to fill your host RAM to be able to use all CPU cores).<br>
Increasing the batch size would also increase the compute intensity and could speed up your workload. However, also your current script using 1GB only could already use all hardware resources and you won’t be able to run any other workload in parallel. To check it, you could either run another script on the same device and check the end2end iteration times or use streams and check, if you are seeing overlapping kernels.</p>

Flock1: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="125603">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>I’m not sure I understand the concern of creating an “efficient model”. Could you explain, why this wouldn’t work or what your concern is?</p>
</blockquote>
</aside>
<p>Sorry for the confusion. I meant an accurate model. I don’t know why I wrote efficient.</p>
<p>Regarding the 2nd one, I ran some matrix multiplication using the GPU on my terminal and it gve an answer. I did this:</p>
<pre><code class="lang-auto">x = torch.rand(100).cuda()
y = torch.rand(100).cuda()
x*y
</code></pre>
<p>It even showed up in <code>nvidia-smi</code>. So did I do it correctly? Or were you referring to something else?</p>