Error in `python': free(): invalid pointer: When using model.cuda() on AWS Instance

sujithv28: <p>For some reason when I set model.cuda() along with the other examples I get the following error:<br>
*** Error in `python’: free(): invalid pointer: 0x00007f8af6c2bae0 ***</p>
<p>However when I no longer set model.cuda() I get no pointer free errors and the model trains fine. Do I have to set .cuda() on every single variable including the criterion?</p>
<p>I am using python2 and using the Udacity Tensorflow g2.2xlarge instance on Amazon AWS.</p>
<p>Here is a link to my code:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/sujithv28/Deep-LeafSnap/blob/master/model.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/sujithv28/Deep-LeafSnap/blob/master/model.py" target="_blank" rel="nofollow noopener">sujithv28/Deep-LeafSnap/blob/master/model.py</a></h4>
<pre><code class="lang-py">import argparse
import cv2
import json
import numpy as np
import os
import pandas as pd
import scipy.misc
import shutil
import time
import torch
import torch.backends.cudnn as cudnn
import torch.nn as nn
import torch.nn.functional as F
import torch.nn.parallel
import torch.optim as optim
import torchvision
import torchvision.models as models
import utils

from PIL import Image
</code></pre>

  This file has been truncated. <a href="https://github.com/sujithv28/Deep-LeafSnap/blob/master/model.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>Thank you!</p>

albanD: <p>Hi,</p>
<p>The <code>.cuda()</code> operation is not inplace for tensors, your should do <code>input = input.cuda()</code>.<br>
That being said it should just raise an error, not fail like that.</p>

sujithv28: <p>I changed everything to <code>.cuda()</code> now but this is the error I get instead:</p>
<pre><code class="lang-auto">THCudaCheck FAIL file=/py/conda-bld/pytorch_1490983232023/work/torch/lib/THC/generic/THCStorage.cu line=66 error=2 : out of memory
Traceback (most recent call last):
  File "model.py", line 244, in &lt;module&gt;
    train(train_loader, model, criterion, optimizer, epoch)
  File "model.py", line 118, in train
    loss.backward()
  File "/home/ubuntu/anaconda3/lib/python3.5/site-packages/torch/autograd/variable.py", line 146, in backward
    self._execution_engine.run_backward((self,), (gradient,), retain_variables)
RuntimeError: cuda runtime error (2) : out of memory at /py/conda-bld/pytorch_1490983232023/work/torch/lib/THC/generic/THCStorage.cu:66
</code></pre>

albanD: <p>You don’t have enough memory on the GPU, you may want to reduce the batch size.</p>

iammohitchhabra: <p>I get the same error on running the cartpole example with cuda. Hower as mentioned above without cuda it runs fine. The error persists even on reducing the batch_size to 2. Any solutions?</p>

iammohitchhabra: <p>sudo apt-get install libtcmalloc-minimal4<br>
export LD_PRELOAD="/usr/lib/libtcmalloc_minimal.so.4"</p>
<p>Fixes the error.</p>

mbednarski: <p>It works, thanks. But do you know why?</p>

jupiter: <p>Indeed it solves the “invalid pointer error”! Can anyone explain why?</p>

Jared_77: <p>Is there any solution to this for someone on an academic institution cluster without sudo privileges?</p>

MJ-Osis: <p>This solved the problem, but after a few more epochs it crashed again. Any more suggestions?</p>