Cuda out of memory occurs while I have enough cuda memory

Wei_Lighter: <p>I am training my models(pretrained resnet and densenet) in rtx 2080ti, it works well. When I move the models to rtx a6000(i need lager batch size)the bug occurs, about 4.5GB is allocated and nearly 40GB is free! I have no idea about this bug, anything could help thank you very much.</p>

ptrblck: <p>Could you post an executable code snippet to reproduce this issue as well as the output of <code>python -m torch.utils.collect_env</code>, please?</p>

Wei_Lighter: <p>I am sorry to answer you so late, I wasbusy with the project of my tutor recent days.</p>
<p>Here is the output of <code>python -m torch.utils.collect_env</code><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/0/409ea9e4d1cc1183f20d9346cd579a9e21004211.png" data-download-href="https://discuss.pytorch.org/uploads/default/409ea9e4d1cc1183f20d9346cd579a9e21004211" title="bf16e28fa8edecc349be2fafd90085b"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/0/409ea9e4d1cc1183f20d9346cd579a9e21004211.png" alt="bf16e28fa8edecc349be2fafd90085b" data-base62-sha1="9dEuAExeX23DZFd0tuacMSCJRkJ" width="505" height="500" data-dominant-color="292929"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">bf16e28fa8edecc349be2fafd90085b</span><span class="informations">850×840 25.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>here is the output when the bug occurs<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/3/833f7a939af9c514701360b00bd9cd20fcddf318.png" data-download-href="https://discuss.pytorch.org/uploads/default/833f7a939af9c514701360b00bd9cd20fcddf318" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/3/833f7a939af9c514701360b00bd9cd20fcddf318_2_690x13.png" alt="image" data-base62-sha1="iJ4ybr3TWN3DycVdXh9NfG7mRiE" width="690" height="13" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/3/833f7a939af9c514701360b00bd9cd20fcddf318_2_690x13.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/3/833f7a939af9c514701360b00bd9cd20fcddf318_2_1035x19.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/3/833f7a939af9c514701360b00bd9cd20fcddf318_2_1380x26.png 2x" data-dominant-color="373737"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1800×35 8.11 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>here is the training part of my code and the criterion_T is a self-defined loss function in this paper <a href="https://proceedings.neurips.cc/paper/2018/file/f2925f97bc13ad2852a7a551802feea0-Paper.pdf" rel="noopener nofollow ugc">Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels</a><br>
and here is the code of the paper <a href="https://github.com/AlanChou/Truncated-Loss" rel="noopener nofollow ugc">code</a>, my criterion_T’s loss is the ‘Truncated-Loss.py’ in that code<br>
the bug occur in the line <img src="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f009cb4fbc39bf8547e0e52dd3a6c598eabad6aa.png" alt="image" data-base62-sha1="yftzLqhm9ZZtPft4kmU6LsyG3Ds" width="292" height="35"></p>
<pre><code class="lang-auto">        for image, target, index in tqdm(loader_train, total=int(data_train.__len__()/opt.batch_size)):
        #for image, target, index in loader_train:
            target = target.cuda(opt.cuda_device)
            # Target = F.one_hot(target[2].cuda(opt.cuda_device), opt.n_classes)
            image = image.cuda(opt.cuda_device)

            ########## train resnet1
            resnet.net.zero_grad()
            out1 = resnet.net.forward(image)

            with torch.no_grad():
                cee1 = F.cross_entropy(out1, target, reduce=False, reduction='none')
                L1 = cee1.cpu().numpy()
                ind1 = np.argsort(L1)
                update_num1 = get_update_num(L1[ind1])
                # dataset_all.static_right_label(index[ind[:clean_num]].tolist())
                update_all_num1 += update_num1

            ########## train resnet1
            densenet.net.zero_grad()
            out2 = densenet.net.forward(image)

            with torch.no_grad():
                cee2 = F.cross_entropy(out2, target, reduce=False, reduction='none')
                L2 = cee2.cpu().numpy()
                ind2 = np.argsort(L2)
                update_num2 = get_update_num(L2[ind2])
                # dataset_all.static_right_label(index[ind[:clean_num]].tolist())
                update_all_num2 += update_num2

            loss1 = criterion_T(out1, Target, index)
            loss2 = criterion_T(out2, Target, index)

            resnet.optimizer.step()

            loss2.backward()
            densenet.optimizer.step()```

Finally thank you very much for answer my question, good luck to you.</code></pre>

ptrblck: <p>Thanks for the update.<br>
The screenshot shows that your GPU has a total capacity of 10.76GiB and cannot allocate the needed 98MiB anymore, so you are not using the A6000 as described in the previous post.<br>
I guess you might have specified the wrong device while executing your script (either inside the script via <code>to('cuda:wriong_id')</code> or via <code>CUDA_VISIBLE_DEVICES=wrong_id</code>).</p>

Wei_Lighter: <p>Sorry, I forget shifting modules and values to the a6000 here is the screenshot while using that gpu.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/6/667fa813f0f9d18c04bfbf68fe8acca3c9856768.png" data-download-href="https://discuss.pytorch.org/uploads/default/667fa813f0f9d18c04bfbf68fe8acca3c9856768" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/667fa813f0f9d18c04bfbf68fe8acca3c9856768_2_690x14.png" alt="image" data-base62-sha1="eCKcHBuxZn7906IgRQ5V54trxzO" width="690" height="14" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/667fa813f0f9d18c04bfbf68fe8acca3c9856768_2_690x14.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/667fa813f0f9d18c04bfbf68fe8acca3c9856768_2_1035x21.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/667fa813f0f9d18c04bfbf68fe8acca3c9856768_2_1380x28.png 2x" data-dominant-color="373737"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1786×37 7.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Could you check via <code>nvidia-smi</code> how much memory is reported to be free there when the script runs OOM?<br>
Also, could you update the code snippet to be executable so that we could try to reproduce it on an A6000?</p>

Wei_Lighter: <p>Here is the output after executing nvidia-smi , the PID 29621. As for my code, it is too long to upload to here, so could you please give me your email that i can send my python file to you directly.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/f/6f68eacdae846061a30d3d6e748f84dbdfa6e1ac.png" data-download-href="https://discuss.pytorch.org/uploads/default/6f68eacdae846061a30d3d6e748f84dbdfa6e1ac" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/f/6f68eacdae846061a30d3d6e748f84dbdfa6e1ac.png" alt="image" data-base62-sha1="fTzMyX20qCapOS1kkA0w5Il4E0s" width="634" height="500" data-dominant-color="2D2D2D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">859×677 22.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Try to reduce the code as much as possible by removing unnecessary code parts and please post it here or create e.g. a gist on GitHub.</p>

Wei_Lighter: <p>Thank you for answer my question so patient, here is link to my <a href="https://github.com/which-light/OOM-occurs-while-I-have-enough-cuda-memory" rel="noopener nofollow ugc">code</a></p>

ptrblck: <p>Thanks for the code. It seems the data might not have been created as I’m running into:</p>
<pre><code class="lang-python">FileNotFoundError: [Errno 2] No such file or directory: '/data/guest1/clothing1m/noised_label.txt'
</code></pre>
<p>while executing <code>run.py</code>.</p>

Wei_Lighter: <p>I am sorry for that, it is so busy recent days, I have already up load the txt files to the code link, please unzip noised_label.zip before using. and you can get the data through <a href="https://drive.google.com/drive/folders/0B67_d0rLRTQYU2E4aHNHaE1uMTg?resourcekey=0-_FShcGYZwIyESjnz6S6aLQ" rel="noopener nofollow ugc">clothing-1m data set</a>.<br>
and here is the location in my computer<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/b/8b5f2441b2fb6b0a2796eda6d9972b1a31369eb8.png" alt="image" data-base62-sha1="jSWcV2Z3ge9LjEGMt7KZwwgGr8I" width="416" height="361"><br>
After downloading, please unzip all the tar files into ‘images’ folder directly. Thank you very much.</p>

ptrblck: <p>Could you update the code to work with random input tensors in the needed shape, please?<br>
I assume the error would also show without downloading the real dataset (where each <code>.tar</code> file seems to be ~2.3GB large?).</p>

Wei_Lighter: <p>I have already uploaded two new files(run2.py and fake_img_set.py) to that link, please put the fake_img_set.py into the folder named loader, and start the project through run2.py. Just forget run.py, here is the oom bug in my computer, in the line 60( out1 = resnet.net.forward(image))<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/f/5f67d9c2b9f9164990dc675fe1f61beee4598afb.png" data-download-href="https://discuss.pytorch.org/uploads/default/5f67d9c2b9f9164990dc675fe1f61beee4598afb" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5f67d9c2b9f9164990dc675fe1f61beee4598afb_2_690x14.png" alt="image" data-base62-sha1="dBZReB0bGZFm78xYtAVgrDMee5t" width="690" height="14" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5f67d9c2b9f9164990dc675fe1f61beee4598afb_2_690x14.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5f67d9c2b9f9164990dc675fe1f61beee4598afb_2_1035x21.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5f67d9c2b9f9164990dc675fe1f61beee4598afb_2_1380x28.png 2x" data-dominant-color="373737"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1783×37 7.91 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Thanks for the update to the code.<br>
I cannot reproduce the issue on an A6000 and the script runs fine for a few epochs:</p>
<pre><code class="lang-python">root@dc9def623dde:/workspace/src/OOM-occurs-while-I-have-enough-cuda-memory# python run2.py 
Namespace(base_path='/data/guest1/clothing1m/', batch_size=64, cuda_device=0, img_size=224, lr=0.01, n_classes=14, n_epochs=500)
Downloading: "https://download.pytorch.org/models/resnet101-63fe2227.pth" to /root/.cache/torch/hub/checkpoints/resnet101-63fe2227.pth
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 171M/171M [00:04&lt;00:00, 38.8MB/s]
Downloading: "https://download.pytorch.org/models/densenet121-a639ec97.pth" to /root/.cache/torch/hub/checkpoints/densenet121-a639ec97.pth
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30.8M/30.8M [00:00&lt;00:00, 109MB/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 156/156 [01:25&lt;00:00,  1.82it/s]
the 1 epoch right ratio:  0.0  0.0 ---accuracy1: 0.096154 accuracy2: 0.096154 learning rate:  0.01 0.01
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 156/156 [01:20&lt;00:00,  1.94it/s]
the 2 epoch right ratio:  0.0  0.0 ---accuracy1: 0.057692 accuracy2: 0.057692 learning rate:  0.01 0.01
</code></pre>
<p><code>nvidia-smi</code> also reports a max. memory usage of ~20GB:</p>
<pre><code class="lang-python">|===============================+======================+======================|
|   0  NVIDIA RTX A6000    On   | 00000000:01:00.0 Off |                  Off |
| 56%   83C    P2   247W / 300W |  19589MiB / 48684MiB |     99%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
</code></pre>
<p>Did you enable MIG on this device or limit the max. memory usage via e.g. PyTorch?</p>

Wei_Lighter: <p>Thanks a lot, there is a limitation for my account, my team mate did that a few days ago without telling me. good luck to you!</p>