RuntimeError: CUDA error: out of memory | return input.log_softmax(dim)

username: ntelo007
cooked: <p>Hi,</p>
<p>I started training the following model, using the train_mtl.py:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/anilbatra2185/road_connectivity" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/7984207?s=400&amp;v=4" class="thumbnail onebox-avatar" width="420" height="420">

<h3><a href="https://github.com/anilbatra2185/road_connectivity" target="_blank" rel="nofollow noopener">anilbatra2185/road_connectivity</a></h3>

<p>Improved Road Connectivity by Joint Learning of Orientation and Segmentation (CVPR2019) - anilbatra2185/road_connectivity</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>After I entered training epoch 45 (out of 120), I received the following error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train_mtl.py", line 472, in &lt;module&gt;
    main()
  File "train_mtl.py", line 466, in main
    val_loss = test(epoch)
  File "train_mtl.py", line 367, in test
    loss2 = angle_loss(pred_vecmaps, util.to_variable(labels[0], True))
  File "C:\Users\...\road_connectivity-master\venv\lib\site-packages\torch\nn\modules\module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\...\road_connectivity-master\utils\loss.py", line 15, in forward
    log_p = F.log_softmax(inputs, dim=1)
  File "C:\Users\...\road_connectivity-master\venv\lib\site-packages\torch\nn\functional.py", line 975, in log_softmax
    return input.log_softmax(dim)
RuntimeError: CUDA error: out of memory
</code></pre>
<p>I am running PyTorch 0.4.1 on Windows 10 with NVIDIA Quadro P3200 with Max-Q Design.</p>
<p>Any ideas on how to solve this?</p>

username: anil_batra
cooked: <p>Hi <a class="mention" href="/u/ntelo007">@ntelo007</a>,</p>
<p>I think GPU (NVIDIA Quadro P3200) has memory of 6GB and in my training code I used GPU with 12 GB of RAM. So, I suggest you to please lower the batch size (probably half it) to train the code.</p>
<p>Thanks</p>

username: ntelo007
cooked: <aside class="quote no-group" data-username="anil_batra" data-post="2" data-topic="64302">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anil_batra/48/11629_2.png" class="avatar"> anil_batra:</div>
<blockquote>
<p>I suggest you to please lower the batch size (probably half it) to train the code</p>
</blockquote>
</aside>
<p>That means that I should change the batch size to 8 (initial 16) and start training from the beginning right? Is it possible to continue training with the best so far model, using a different batch size?</p>

username: anil_batra
cooked: <p>Hi <a class="mention" href="/u/ntelo007">@ntelo007</a>,</p>
<p>Yes, it is possible to resume the training from the best saved model.</p>
<p>I am confused now, if you were able to train and save the model then I am wondering where OOM issue occurred. Can you post the full logs? Is it happening at testing time?</p>
<p>Thanks</p>

username: ntelo007
cooked: <aside class="quote no-group" data-username="anil_batra" data-post="4" data-topic="64302">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anil_batra/48/11629_2.png" class="avatar"> anil_batra:</div>
<blockquote>
<p>Is it happening at testing time?</p>
</blockquote>
</aside>
<p>Yes</p>
<aside class="quote no-group" data-username="anil_batra" data-post="4" data-topic="64302">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anil_batra/48/11629_2.png" class="avatar"> anil_batra:</div>
<blockquote>
<p>Can you post the full logs?</p>
</blockquote>
</aside>
<p>Which files are you requesting exactly?</p>

username: anil_batra
cooked: <p>Now, it make sense. There was a small issue which was causing to increase the memory. I have updated my code (train_mtl.py and util.py). Request you to take latest and run it again from scratch.</p>
<p>Thanks</p>

username: ntelo007
cooked: <p>I replaced the files but now I am receiving many errors and I cannot train your model. What did you change exactly? It seems that you even removed the main function.</p>