Running out of memory while running a model

username: Varun_Kumar_VIjay
cooked: <p>I am trying to run a small neural network on the CPU and am finding that the memory used by my script increases without limit. Since my script does not do much besides call the network, the problem appears to be a memory leak within pytorch.</p>
<p>The problem does not occur if I run the model on the gpu.</p>
<p>I’ve also posted this to the pytorch github, but I was hoping someone on here might be able to help.</p>
<p>Here’s the script I’m using. The output of the program increases until the machine runs out of memory, at which point it crashes.</p>
<p>import torch<br>
import torch.nn.functional as F<br>
import torch.nn as nn</p>
<p>import resource<br>
import gc</p>
<p>class Net(nn.Module):<br>
def <strong>init</strong>(self, in_size, h_size, out_size):<br>
super(Net, self).<strong>init</strong>()</p>
<pre><code>    self.lin1 = nn.Linear(in_size, h_size)
    self.lin2 = nn.Linear(h_size, out_size)
    self.nonlin = F.relu

def forward(self, x):
    h = self.nonlin(self.lin1(x))
    h = self.nonlin(self.lin2(h))
    return h
</code></pre>
<p>def main():<br>
in_size = 100<br>
h_size = 256<br>
out_size = 512</p>
<pre><code>net = Net(in_size, h_size, out_size)

while True:
    gc.collect()
    print('maxrss = {}'.format(
        resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1e6))
    for i in range(1000):
        net(torch.randn(in_size))
</code></pre>
<p>if <strong>name</strong> == ‘<strong>main</strong>’:<br>
main()</p>

username: Varun_Kumar_VIjay
cooked: <p>This is a dockerfile for building my environment:</p>
<p>FROM nvidia/cuda:10.0-cudnn7-devel-ubuntu18.04</p>
<p>RUN apt-get update -y &amp;&amp; <br>
apt-get upgrade -y</p>
<p>RUN apt-get clean autoclean -y &amp;&amp; <br>
apt-get autoremove -y</p>
<p>ENV DEBIAN_FRONTEND=noninteractive<br>
RUN apt-get install -y libsm6 libxext6 wget curl unzip <br>
apt-utils git python-virtualenv python3-dev python3-pip software-properties-common python3-opencv</p>
<p>RUN apt-get clean autoclean -y &amp;&amp; <br>
apt-get autoremove -y</p>
<p>RUN apt-get install python3-pip cmake zlib1g-dev python3-tk python-opencv -y<br>
RUN apt-get install libboost-all-dev -y<br>
RUN apt-get install libblas-dev liblapack-dev libatlas-base-dev gfortran libosmesa6-dev -y<br>
RUN apt-get install libsdl-dev libsdl-image1.2-dev libsdl-mixer1.2-dev libsdl-ttf2.0-dev libsmpeg-dev libportmidi-dev libavformat-dev libswscale-dev -y<br>
RUN apt-get install dpkg-dev build-essential python3-dev libjpeg-dev libtiff-dev libsdl1.2-dev libnotify-dev freeglut3 freeglut3-dev libsm-dev libgtk2.0-dev libgtk-3-dev libwebkitgtk-dev libgtk-3-dev libwebkitgtk-3.0-dev libgstreamer-plugins-base1.0-dev -y<br>
RUN apt-get install libsdl2-dev swig cmake -y<br>
RUN apt-get install iputils-ping mongodb-clients vim -y<br>
RUN apt-get install -y libsm6 libxext6 libxrender-dev vim python-opencv ffmpeg tmux<br>
RUN apt-get install -y libosmesa6-dev libgl1-mesa-glx libglfw3 wget<br>
RUN apt-get install -y cmake libopenmpi-dev python3-dev zlib1g-dev patchelf ssh rsync<br>
RUN apt-get install -y git</p>
<h1>Apt cleaning</h1>
<p>RUN apt-get clean autoclean &amp;&amp; <br>
apt-get autoremove -y</p>
<h1>set environment variables</h1>
<p>ENV PATH $PATH:/usr/local/cuda/bin<br>
ENV LD_LIBRARY_PATH /usr/local/cuda/lib64:/usr/local/cuda/lib:/usr/local/lib:$LD_LIBRARY_PATH</p>
<p>RUN apt-get update -q <br>
&amp;&amp; DEBIAN_FRONTEND=noninteractive apt-get install -y <br>
curl <br>
git <br>
libgl1-mesa-dev <br>
libgl1-mesa-glx <br>
libglew-dev <br>
libosmesa6-dev <br>
software-properties-common <br>
net-tools <br>
unzip <br>
vim <br>
virtualenv <br>
wget <br>
xpra <br>
xserver-xorg-dev <br>
sudo <br>
&amp;&amp; apt-get clean <br>
&amp;&amp; rm -rf /var/lib/apt/lists/*</p>
<p>RUN apt-get update -q<br>
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y swig</p>
<p>RUN DEBIAN_FRONTEND=noninteractive add-apt-repository --yes ppa:deadsnakes/ppa &amp;&amp; apt-get update<br>
RUN DEBIAN_FRONTEND=noninteractive apt-get install --yes python3.5-dev python3.5 python3-pip<br>
RUN virtualenv --python=python3.5 env</p>
<p>RUN rm /usr/bin/python<br>
RUN ln -s /env/bin/python3.5 /usr/bin/python<br>
RUN ln -s /env/bin/pip3.5 /usr/bin/pip<br>
RUN ln -s /env/bin/pytest /usr/bin/pytest</p>
<p>RUN curl -o /usr/local/bin/patchelf <a href="https://s3-us-west-2.amazonaws.com/openai-sci-artifacts/manual-builds/patchelf_0.9_amd64.elf" rel="nofollow noopener">https://s3-us-west-2.amazonaws.com/openai-sci-artifacts/manual-builds/patchelf_0.9_amd64.elf</a> <br>
&amp;&amp; chmod +x /usr/local/bin/patchelf</p>
<p>ENV LANG C.UTF-8</p>
<p>ARG GID<br>
ARG UID<br>
ARG UNAME<br>
RUN groupadd -g ${GID} -o ${UNAME}<br>
RUN useradd -m -u ${UID} -g $GID -o -s /bin/bash ${UNAME}</p>
<p>COPY docker/vendor/Xdummy /usr/local/bin/Xdummy<br>
RUN chmod +x /usr/local/bin/Xdummy</p>
<h1>Workaround for <a href="https://bugs.launchpad.net/ubuntu/+source/nvidia-graphics-drivers-375/+bug/1674677" rel="nofollow noopener">https://bugs.launchpad.net/ubuntu/+source/nvidia-graphics-drivers-375/+bug/1674677</a>
</h1>
<p>COPY docker/vendor/10_nvidia.json /usr/share/glvnd/egl_vendor.d/10_nvidia.json</p>
<p>RUN pip install cloudpickle==0.5.2<br>
RUN pip install cached-property==1.3.1</p>
<h1>For atari-py</h1>
<p>RUN apt-get update -y<br>
RUN apt-get install -y zlib1g-dev swig cmake<br>
RUN pip install --upgrade pip<br>
RUN pip install gym[all]==0.10.5<br>
RUN pip install gitpython==2.1.7<br>
RUN pip install gtimer==1.0.0b5<br>
RUN pip install pygame<br>
RUN pip install awscli==1.11.179<br>
RUN pip install boto3==1.4.8<br>
RUN pip install dominate==2.3.1<br>
RUN pip install ray==0.2.2<br>
RUN pip install path.py==10.3.1</p>
<p>RUN pip install numpy ninja pyyaml mkl mkl-include setuptools cmake cffi<br>
RUN pip install torch==1.3.1 torchvision<br>
RUN pip install joblib==0.9.4</p>
<p>RUN apt-get update &amp;&amp; apt-get install -y ffmpeg tmux</p>
<h1>RUN apt-get install -y nvidia-390</h1>
<p>RUN pip install opencv-python==3.4.0.12<br>
RUN pip install sk-video==1.1.10<br>
RUN pip install ipdb pyarrow gitpython objgraph pympler</p>
<p>USER $UNAME</p>

username: mailcorahul
cooked: <p>Can you try running your model inside <code>torch.no_grad()</code>? By default, autograd engine is on, therefore gradients will be computed. Which is why your RAM gets accumulated and goes out of memory after consecutive forward passes.<br>
Something like,</p>
<pre><code class="lang-auto">for i in range(1000):
    with torch.no_grad():
        net(torch.randn(in_size))
</code></pre>

username: Varun_Kumar_VIjay
cooked: <p>I don’t believe gradient accumulation can cause the problem. It does take some extra memory to maintain the gradients and intermediate values, but this memory should be reused in the next iteration of the model.</p>
<p>Just to make sure, I repeated the test with your change and see that the memory usage still increases.</p>

username: mailcorahul
cooked: <p>Right. I have faced something similar to this, but while doing inference on GPU.<br>
Link to my post: <a href="https://discuss.pytorch.org/t/gpu-memory-gets-accumulated-during-consecutive-forward-passes/62403" class="inline-onebox">Gpu memory gets accumulated during consecutive forward passes</a><br>
Haven’t found any solution yet.</p>

username: Varun_Kumar_VIjay
cooked: <p>Your issue looks similar. The only element that could cause a memory leak is the outputs variable, if you are holding on to it. In my test, I avoid storing the result of net.<strong>call</strong> so that it can be freed immediately.</p>
<p>Also, I don’t encounter a memory problem when I use cuda.</p>