Torch cuda erorr

Aylore: <p>hello all<br>
I am training a CNN - LSTM model and every time I try to run the training I always get the following cuda error , I am working on hpc server which have two GPU and using nn.DataParrallel(model) , doesn’t seem to change anything , so I need some help please</p>
<p>PS<br>
image size : 224 ,batch_size= 4<br>
image sequence = 64</p>
<p>thanks<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8c65152571ddb9ce23693f640165eac61228c56.png" data-download-href="https://discuss.pytorch.org/uploads/default/a8c65152571ddb9ce23693f640165eac61228c56" title="Screenshot from 2022-04-24 01-52-33"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a8c65152571ddb9ce23693f640165eac61228c56_2_690x388.png" alt="Screenshot from 2022-04-24 01-52-33" data-base62-sha1="o536JIEtZMh7gZQ8gOXEs8v0toy" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a8c65152571ddb9ce23693f640165eac61228c56_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a8c65152571ddb9ce23693f640165eac61228c56_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a8c65152571ddb9ce23693f640165eac61228c56_2_1380x776.png 2x" data-dominant-color="E0E0E0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2022-04-24 01-52-33</span><span class="informations">1920×1080 301 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>You are running out of memory, so you would either need to reduce the memory requirement e.g. by lowering the batch size, or you could trade compute for memory via <code>torch.utils.checkpoint</code>.</p>

Aylore: <p>thanks for your help<br>
but doesn’t the allocated memory and the reserved by pytorch are too large for what they supposed to be ?</p>

ptrblck: <p>The memory requirement depend on the used model as well as the input shape. Dont forget that during training the intermediate forward activations, the gradients, and the optimizer’s states (if available) will use additional memory.</p>

Aylore: <p>I am aware of the the gradients , but this error happens even if i try to just pass the model the input data no training or anything</p>

InnovArul: <p>If i understand your question correctly,  the input size is <code>4x3x64×224×224</code>?<br>
That’s essentially 256 RGB images of size <code>224x224</code>.  I do not know the type of model you are using. But i would imagine,  for a model like ResNet, this input size is already huge for forward pass in a ~11GB GPU. Try reducing the number of images in the sequence or batch size or spatial size of the image.</p>

Aylore: <p>it is grey scale not rgb  + I changed the spatial size to 64 and the batch to 2 and still doesn’t fit , I use efficient net as an encoder before the LSTM</p>

InnovArul: <p>I haven’t used EfficientNet a lot. I’m not sure about it’s memory requirements.</p>
<p>I would try to reduce the number of images in the sequence as far as possible to get a lowest baseline. Then, start from the lowest possible baseline and try to increase the dimensions (batch size,  spatial size or number of images in a sequence) one by one to see what’s the maximum limit that can be used in your GPU.</p>

Aylore: <p>ok thanks I will try</p>