RuntimeError: cuda runtime error (2) : out of memory at ……THCStorage.cu:58

username: Hxb110
cooked: <p><strong>Hello Everyone,</strong></p>
<blockquote>
<p>RuntimeError: cuda runtime error (2) : out of memory at c:\users\administrator\downloads\new-builder\win-wheel\pytorch\aten\src\thc\generic/THCStorage.cu:58</p>
</blockquote>
<p><strong>When I trained my network, i got this error. My network is modified from RCAN. After 1 epoch over, means my code</strong></p>
<blockquote>
<p>t.train()</p>
</blockquote>
<p><strong>is over.And then going to</strong></p>
<blockquote>
<p>t.test()</p>
</blockquote>
<p><strong>The entire main code is as follows</strong></p>
<blockquote>
<p>import torch<br>
import utility<br>
import data<br>
import model<br>
import loss<br>
from option import args<br>
from trainer import Trainer</p>
</blockquote>
<pre><code>if __ name __ == '__ main __':
torch.backends.cudnn.enabled = False
torch.manual_seed(args.seed)
checkpoint = utility.checkpoint(args)

if checkpoint.ok:
    loader = data.Data(args)
    model = model.Model(args, checkpoint)
    loss = loss.Loss(args, checkpoint) if not args.test_only else None
    t = Trainer(args, loader, model, loss, checkpoint)
    while not t.terminate():
        t.train()
        t.test()

    checkpoint.done()
</code></pre>
<p><strong>The following is the result:</strong></p>
<blockquote>
<p>C:\Users\HP\Anaconda3\envs\pytorch\python.exe D:/hxb/RCAN-master/RCAN_TrainCode/code/main.py<br>
Preparing seperated binary files<br>
Preparing seperated binary files<br>
Making model…<br>
Preparing loss function:<br>
1.000 * L1<br>
[Epoch 1]	Learning rate: 1.00e-4<br>
[1600/16000]	[L1: 17.5433]	88.1+5.9s<br>
[3200/16000]	[L1: 14.3299]	87.8+0.4s<br>
[4800/16000]	[L1: 12.7960]	88.0+0.4s<br>
[6400/16000]	[L1: 11.8648]	88.0+0.3s<br>
[8000/16000]	[L1: 11.2393]	88.0+0.3s<br>
[9600/16000]	[L1: 10.8215]	88.0+0.4s<br>
[11200/16000]	[L1: 10.4444]	88.0+0.3s<br>
[12800/16000]	[L1: 10.1431]	87.8+0.4s<br>
[14400/16000]	[L1: 9.9283]	87.8+0.4s<br>
[16000/16000]	[L1: 9.6954]	87.9+0.4s<br>
Evaluation:<br>
0%|                                                     | 0/5 [00:00&lt;?, ?it/s]THCudaCheck FAIL file=c:\users\administrator\downloads\new-builder\win-wheel\pytorch\aten\src\thc\generic/THCStorage.cu line=58 error=2 : out of memory<br>
Traceback (most recent call last):<br>
File “D:/hxb/RCAN-master/RCAN_TrainCode/code/main.py”, line 23, in <br>
t.test()<br>
File “D:\hxb\RCAN-master\RCAN_TrainCode\code\trainer.py”, line 93, in test<br>
sr = self.model(lr, idx_scale)<br>
File “C:\Users\HP\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py”, line 491, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “D:\hxb\RCAN-master\RCAN_TrainCode\code\model_<em>init</em>_.py”, line 54, in forward<br>
return self.model(x)<br>
File “C:\Users\HP\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py”, line 491, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “D:\hxb\RCAN-master\RCAN_TrainCode\code\model\mynetwork.py”, line 149, in forward<br>
x = self.tail(res)<br>
File “C:\Users\HP\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py”, line 491, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “C:\Users\HP\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\container.py”, line 91, in forward<br>
input = module(input)<br>
File “C:\Users\HP\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py”, line 491, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “C:\Users\HP\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\conv.py”, line 301, in forward<br>
self.padding, self.dilation, self.groups)<br>
RuntimeError: cuda runtime error (2) : out of memory at c:\users\administrator\downloads\new-builder\win-wheel\pytorch\aten\src\thc\generic/THCStorage.cu:58</p>
</blockquote>
<p><strong>And we can see that there are some problem in</strong></p>
<blockquote>
<p>t.test()</p>
</blockquote>
<p><strong>but I dont know why.</strong> <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
<p>Is there anyone can help me. The question is <strong>cu:58</strong> .Thanks for reading my amateur english.<img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: avalon1511
cooked: <p>This just means that your GPU ran out of memory and you might need to assign more than one GPU to execute this program</p>

username: Hxb110
cooked: <p>This should be a solution, but I have no conditions to verify it. <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"></p>