RuntimeError:Cuda out of memory[Unable to use]

asha97: <p>Traceback (most recent call last):<br>
File “check1.py”, line 317, in <br>
main()<br>
File “check1.py”, line 172, in main<br>
iteration = train(epoch,iteration)<br>
File “check1.py”, line 128, in train<br>
pred = model(img)<br>
File “/DATA/rani.1/.local/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 550, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “./model/model.py”, line 155, in forward<br>
x = self.backend(x)<br>
File “/DATA/rani.1/.local/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 550, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/DATA/rani.1/.local/lib/python3.6/site-packages/torch/nn/modules/container.py”, line 100, in forward<br>
input = module(input)<br>
File “/DATA/rani.1/.local/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 550, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/DATA/rani.1/.local/lib/python3.6/site-packages/torch/nn/modules/container.py”, line 100, in forward<br>
input = module(input)<br>
File “/DATA/rani.1/.local/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 550, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/DATA/rani.1/.local/lib/python3.6/site-packages/torchvision/models/resnet.py”, line 113, in forward<br>
out = self.bn3(out)<br>
File “/DATA/rani.1/.local/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 550, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/DATA/rani.1/.local/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py”, line 106, in forward<br>
exponential_average_factor, self.eps)<br>
File “/DATA/rani.1/.local/lib/python3.6/site-packages/torch/nn/functional.py”, line 1923, in batch_norm<br>
training, momentum, eps, torch.backends.cudnn.enabled<br>
RuntimeError: CUDA out of memory. Tried to allocate 1.10 GiB (GPU 0; 10.92 GiB total capacity; 9.94 GiB already allocated; 413.50 MiB free; 9.96 GiB reserved in total by PyTorch)</p>
<p>I am getting the above error whenever passing model.cuda() but when  I remove it runs fine but on cpu.Unable to use gpu</p>

ptrblck: <p>Your model might be too big for your device. It’s a bit weird that you are seeing this error after calling <code>model.cuda()</code>, as this would mean that your model parameters are using more than 11GB.<br>
Could this be the case?<br>
If not, make sure that your GPU memory is free before running the script (check it via <code>nvidia-smi</code>).</p>

asha97: <p>I checked using nvidia-smi ,all the 8 gpus are empty and even alloting 2 to3 gives same error ,it goes to cuda:0 again n again</p>

asha97: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Also I am using "import torch.backends.cudnn as cudnn  “”, when I search for codes of pytorch using gpu, everywhere pycuda is refered</p>
<p>Is some problem here , I m using cuda 10.1</p>

ptrblck: <aside class="quote no-group" data-username="asha97" data-post="3" data-topic="85182" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/asha97/48/25662_2.png" class="avatar"> asha97:</div>
<blockquote>
<p>I checked using nvidia-smi ,all the 8 gpus are empty and even alloting 2 to3 gives same error ,it goes to cuda:0 again n again</p>
</blockquote>
</aside>
<p>This would mean that your single model is too large. You could count the number of parameters and buffers and calculate the theoretical size it would need on the GPU.</p>
<aside class="quote no-group" data-username="asha97" data-post="4" data-topic="85182">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/asha97/48/25662_2.png" class="avatar"> asha97:</div>
<blockquote>
<p>when I search for codes of pytorch using gpu, everywhere pycuda is refered</p>
</blockquote>
</aside>
<p>Could you post a link to this, please?</p>

asha97: <p>you mean all the parameters or the trainable parameters</p>

ptrblck: <p>All parameters and buffers, as they will all be pushed to the GPU.</p>

asha97: <p>total parameters are : 29804609</p>

ptrblck: <p>29804609 parameters in <code>float32</code> (4 Bytes per element) would only use ~113MB, so something else would use the memory.</p>
<p>Feel free to post the model definition, so that we could have a look.</p>

asha97: <p>Mem buffers= 212904<br>
Model Definition</p>
<pre><code class="lang-auto">class Sequential(nn.Module):
	def __init__(self,backend,seq_len,embed_size=512):
		super(Sequential,self).__init__()
		self.seq_len = seq_len
		self.bk_name = backend
		# defining backend 
		if self.bk_name == 'resnet':
			resnet = models.resnet50(pretrained=True)
			# self.dilate_resnet(resnet)
			self.init_resnet(resnet)
			self.encoder = nn.Linear(8192,embed_size,bias=True)
		elif self.bk_name == 'vgg':
			vgg = models.vgg19(pretrained=True) 
			self.init_vgg(vgg)
		else:
			assert 0, 'Backend not implemented'

		self.rnn = LSTM(embed_size)
		#print(self.rnn)
		#print("feature extracted")
		# self.rnn = G_LSTM(embed_size)
		# self.rnn = GRU(embed_size)
		self.decoder = nn.Linear(embed_size,1,bias=True)
		# self.decoder_extra = nn.Linear(embed_size,1024)
		# self.decoder = nn.Linear(1024,1,bias=True)

		#fixting saliency module
		for para in self.backend.parameters():
			para.requires_grad = True # fixing pretrained modules or not

	def init_resnet(self,resnet):
		self.backend = nn.Sequential(*list(resnet.children())[:-2])

	def init_vgg(self,vgg):
		# self.backend = vgg.features
		self.backend = nn.Sequential(*list(vgg.features.children())[:-1]) # omitting the last Max Pooling 

	def dilate_resnet(self, resnet): #modifying resnet as in SAM paper
		resnet.layer3[0].conv1.stride = 1
		resnet.layer3[0].downsample[0].stride = 1
		resnet.layer4[0].conv1.stride = 1
		resnet.layer4[0].downsample[0].stride = 1
		
		for block in resnet.layer3:
			block.conv2.dilation = 2
			block.conv2.padding = 2

		for block in resnet.layer4:
			block.conv2.dilation = 4
			block.conv2.padding = 4 

	def init_hidden(self,x): #initializing hidden state as all zero
		h = x.data.new().resize_as_(x.data).fill_(0)
		c = x.data.new().resize_as_(x.data).fill_(0)
		return (Variable(h),Variable(c))
		# return Variable(h) # return only h when using GRU

	def forward(self,x):
		batch, seq, c, h, w = x.size()
		x = x.view(batch*seq,c,h,w)
		# backend forward pass
		if self.bk_name == 'resnet':
			x = self.backend(x)
			x = F.avg_pool2d(x,(7,7)) # Global Average Pooling
			# x = F.max_pool2d(x,(7,7)) # Global Max Pooling
			x = x.view(batch*seq,-1)
			x = F.relu(self.encoder(F.dropout(x,p=0.4)))
		else:
			x = self.backend(x)
			x = F.avg_pool2d(x,(14,14)) # Global Average Pooling for vgg19 without final maxpool
		x = x.view(batch,seq,-1) # unroll features

		# recurrent loop
		state = self.init_hidden(x[:,0,:]) # initialize hidden state
		for i in range(self.seq_len):
			cur_x = x[:,i,:].contiguous()
			state = self.rnn(cur_x,state)

			
		h, c = state # LSTM
		x = torch.sigmoid(self.decoder(F.dropout(h,p=0.5)))
		
		return x


</code></pre>

ptrblck: <p>Your <code>LSTM</code> class is undefined, so I just used <code>nn.LSTM</code> with a hidden size of 1.<br>
As expected, the model just uses approx. 100 MB on the device:</p>
<pre><code class="lang-python">model = Sequential('resnet', 100)
print(torch.cuda.memory_allocated()/1024**2)
&gt; 0
model.cuda()
print(torch.cuda.memory_allocated()/1024**2)
&gt; 105.92431640625
</code></pre>

asha97: <p>My LSTM Class definition:</p>
<pre><code class="lang-auto">class LSTM(nn.Module):
	"""
	Original LSTM model
	"""
	def __init__(self,embed_size=512):
		super(LSTM,self).__init__()
		self.input_x = nn.Linear(embed_size,embed_size,bias=True)
		self.forget_x = nn.Linear(embed_size,embed_size,bias=True)
		self.output_x = nn.Linear(embed_size,embed_size,bias=True)
		self.memory_x = nn.Linear(embed_size,embed_size,bias=True)
		self.input_h = nn.Linear(embed_size,embed_size,bias=True)
		self.forget_h = nn.Linear(embed_size,embed_size,bias=True)
		self.output_h = nn.Linear(embed_size,embed_size,bias=True)
		self.memory_h = nn.Linear(embed_size,embed_size,bias=True)

	def forward(self,x,state):
		h, c = state
		i = F.sigmoid(self.input_x(x) + self.input_h(h))
		f = F.sigmoid(self.forget_x(x) + self.forget_h(h))
		o = F.sigmoid(self.output_x(x) + self.output_h(h))

		g = F.tanh(self.memory_x(x) + self.memory_h(h))

		next_c = torch.mul(f,c) + torch.mul(i,g)
		h = torch.mul(o,next_c)
		state = (h,next_c)

		return state

</code></pre>
<p>Do CUDA version effect the execution of code?</p>

ptrblck: <p>With this <code>LSTM</code> definition, the memory usage is now ~114MB.</p>
<aside class="quote no-group" data-username="asha97" data-post="12" data-topic="85182">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/asha97/48/25662_2.png" class="avatar"> asha97:</div>
<blockquote>
<p>Do CUDA version effect the execution of code?</p>
</blockquote>
</aside>
<p>The memory usage for the CUDA context might differ from different CUDA versions.<br>
The model itself should not use more or less memory.</p>

asha97: <p>For me this code is only running when batch size=1 else  I m getting the same error</p>

ptrblck: <p>Are you seeing the OOM during the training or after calling <code>model.cuda()</code>?<br>
From the first answers I understood that a simple <code>model.cuda()</code> is causing the OOM.</p>

asha97: <p>yes , many message of MEMORY ERROR was there</p>

asha97: <p>main()<br>
File “t1.py”, line 179, in main<br>
iteration = train(epoch,iteration)<br>
File “t1.py”, line 135, in train<br>
pred = model(img)<br>
File “/DATA/rani.1/.local/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 550, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “./model/model.py”, line 155, in forward<br>
x = self.backend(x)<br>
File “/DATA/rani.1/.local/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 550, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/DATA/rani.1/.local/lib/python3.6/site-packages/torch/nn/modules/container.py”, line 100, in forward<br>
input = module(input)<br>
File “/DATA/rani.1/.local/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 550, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/DATA/rani.1/.local/lib/python3.6/site-packages/torch/nn/modules/pooling.py”, line 141, in forward<br>
self.return_indices)<br>
File “/DATA/rani.1/.local/lib/python3.6/site-packages/torch/_jit_internal.py”, line 209, in fn<br>
return if_false(*args, **kwargs)<br>
File “/DATA/rani.1/.local/lib/python3.6/site-packages/torch/nn/functional.py”, line 539, in _max_pool2d<br>
input, kernel_size, stride, padding, dilation, ceil_mode)<br>
RuntimeError: CUDA out of memory. Tried to allocate 1.10 GiB (GPU 1; 10.92 GiB total capacity; 9.72 GiB already allocated; 637.50 MiB free; 9.74 GiB reserved in total by PyTorch)</p>
<p>I am getting this error even now after having batch size 1, any solution?<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <p>This error is raised during a forward pass, so the <code>model.cuda()</code> call will not cause the OOM as previously mentioned?<br>
If so, please post a minimal, executable code snippet using random input tensors, so that we could debug it.</p>

asha97: <p>One more thing, it was running fine when I was sending about 12 images, together after concatenation, but when doing it for about 240 images. I m getting error again<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <p>You would have to use a batch size for the model and training, which fits into the memory of your device.<br>
Too large batch sizes will try to use too much memory and will thus yield the “out of memory” issue.</p>