Why would nn.ReLU cause my model to go over memory?

Alexander_Soare: <p>I have a model which looks like this:</p>
<pre><code class="lang-auto">class PFNet(nn.Module):
    def __init__(self):
        super(PFNet, self).__init__()
        self.conv1 = nn.Conv3d(1, 16, 3)
        self.conv2 = nn.Conv3d(16, 32, 3)
        self.conv3 = nn.Conv3d(32, 96, 2)
        self.conv4 = nn.Conv3d(96, 1, 1)
        self.pool1 = nn.MaxPool3d(kernel_size=2, stride=2)
        self.pool2 = nn.MaxPool3d(kernel_size=3, stride=3)
        self.pool3 = nn.MaxPool3d(kernel_size=2, stride=2)
        self.pool4 = nn.MaxPool3d(kernel_size=2, stride=2)
        self.relu = nn.ReLU()
        self.fc1 = nn.Linear(400, 1)

    def forward(self, x):
        x = self.conv1(x)
        x = self.relu(x)
        x = self.pool1(x)

        x = self.conv2(x)
        x = self.relu(x)
        x = self.pool2(x)

        x = self.conv3(x)
        x = self.relu(x)
        x = self.pool3(x)

        x = self.conv4(x)
        x = self.relu(x)
        x = self.pool4(x)
        x = x.view(-1, 400)
        x = self.fc1(x)

        return x
</code></pre>
<p>Before I added in the ReLUs it was working fine on a batch size of 8. Then with the ReLUs I get</p>
<blockquote>
<p>CUDA out of memory. Tried to allocate 1.04 GiB (GPU 0; 15.75 GiB total capacity; 13.68 GiB already allocated; 848.88 MiB free; 13.76 GiB reserved in total by PyTorch)</p>
</blockquote>
<p>ReLU being literally something like <code>np.max(tensor, 0)</code>, I’d think that this shouldn’t have any impact on the memory.</p>

Nikronic: <p>Hi,</p>
<p><code>Relu</code> by default allocate new memory for output. You can modify input directly by setting <code>inplace=True</code> flag. Although, I am not sure this is the only reason or not but for sure <code>Relu</code> will consume memory.</p>
<aside class="quote" data-post="2" data-topic="948">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/apaszke/48/21_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/whats-the-difference-between-nn-relu-and-nn-relu-inplace-true/948/2">What's the difference between nn.ReLU() and nn.ReLU(inplace=True)?</a> 
  </div>
  <blockquote>
    inplace=True means that it will modify the input directly, without allocating any additional output. It can sometimes slightly decrease the memory usage, but may not always be a valid operation (because the original input is destroyed). However, if you don’t see an error, it means that your use case is valid.
  </blockquote>
</aside>

<p>Bests</p>

Alexander_Soare: <p>Thanks. Good answer but unfortunately, as you warned, no cigar. Still running into the memory issue.</p>
<p>… although, I was able to up my batch size from 4 to 6 with this change. But still not the 8 I had originally.</p>

Nikronic: <p>If the only change was introducing <code>Relu</code>, I cannot really figure it out at the moment.<br>
One proper way to debug such issues is that to use profiler. It helps you to find bottlenecks even when no error/warning is happening so you can optimize your code much more.<br>
Sorry for the lack of knowledge.<br>
<a href="https://pytorch.org/tutorials/recipes/recipes/profiler.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/recipes/recipes/profiler.html</a></p>

singleroc: <p>Hi,</p>
<p>Based on my own understanding of graphic mode, the ReLU will introduce extra memory when training even the inplace flag is set.</p>
<p>The inplace flag only indicates the output resue the input memory in the forward round. However, it need to save extra information for the latter backward propogation.  It either saves the input tensor X or positions where X&gt;=0.  I do not know where it happens, but I believe the saving for backward is indeed required.</p>
<p>If only in inference mode, I think ReLU(True) will not introduce extra memory.</p>

Rui_Wang: <p>Hi,</p>
<p>This maybe 2 years too late, but you could switch the order of the maxpooling and relu and set relu to inplace to make it work I believe.</p>