Number of allocated tensors starts to grow if 3D grid size is above (110, 110, 110)

weders: <p>Hi,</p>
<p>I am currently working 3D deep learning with 3D convolutions on grids. Now, I encountered the problem that if the 3D grid size is above (110, 110, 110) the number of allocated tensors on the GPU starts to grow when calling the loss.backward() causing a GPU memory leak.</p>
<p>Why is that the case?</p>
<p>An example:</p>
<pre><code>    x1 = self.padding(x)
    x1 = self.kernel_1(x1)
    x1 = self.tanh(x1)

    x1 = torch.cat((x, x1), dim=1)
    
    x1 = self.kernel_2(x1)
    x1 = self.tanh(x1)

    x1 = self.kernel_3(x1)
    x1 = self.tanh(x1)

    x1 = self.kernel_4(x1)
    x1 = self.tanh(x1)

    x1 = self.kernel5(x1)
    x1 = self.tanh(x1)

    return x1
</code></pre>
<p>If I return x1 after kernel_3, then the problem does not happen. If I return x1 after kernel_4 or later the problem happens. All kernels are 3D convolutions with 1x1x1 kernels.</p>
<p>Thanks in advance!</p>

ptrblck: <p>Are you passing an input of <code>[batch_size, channels, 110, 110, 110]</code> to <code>nn.Conv3d</code> layers or do I misunderstand your question regarding the grid?<br>
Could you post the model definition so that we can reproduce this issue, please?</p>

weders: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>Thanks for the fast reply!</p>
<p>Yes, exactly. So the problem of new tensors being allocated starts when increasing the input size from [1, 16, 110, 110, 110] to e.g. [1, 16, 115, 115, 115].</p>
<p>The model definition is:</p>
<pre><code>    self.kernel_1 = torch.nn.Conv3d(16, 16,
                                    kernel_size=3,
                                    stride=1,
                                    padding=0)
    
    self.kernel_2 = torch.nn.Conv3d(32, 16,
                                   kernel_size=1,
                                   stride=1,
                                   padding=0)

    self.kernel_3 = torch.nn.Conv3d(16, 8,
                                   kernel_size=1,
                                   stride=1,
                                   padding=0)

    self.kernel_4 = torch.nn.Conv3d(8, 4,
                                   kernel_size=1,
                                   stride=1,
                                   padding=0)

    self.kernel_5 = torch.nn.Conv3d(4, 1,
                                   kernel_size=1,
                                   stride=1,
                                   padding=0)


    self.padding = torch.nn.ReplicationPad3d(1)

    self.relu = torch.nn.LeakyReLU()
    self.tanh = torch.nn.Tanh()
</code></pre>
<p>Thanks!</p>

ptrblck: <p>Thanks for the code!<br>
I cannot reproduce a memory leak with a master build + CUDA10.2 or with the <code>1.4.0</code> binaries + CUDA10.1.</p>
<p>I’m using your model and this code:</p>
<pre><code class="lang-python">model = MyModel().cuda()
x = torch.randn(1, 16, 115, 115, 115).cuda()
torch.cuda.synchronize()
print('mem allocated ', torch.cuda.memory_allocated()/1024**3)

for idx in range(100):
    output  = model(x)
    print('iter {} mem allocated {}'.format(
        idx, torch.cuda.memory_allocated()/1024**3))
</code></pre>
<p>Do you see a memory leak using my test code?</p>
<p>PS: I tested the volumetric shapes of 110, 115, and 125.</p>