Single Conv2D layer has larger footprint than resnet50

AljoSt: <p>While investigating another issue, I stumbled over some weird behaviour, where using a single Con2D layer for inference blows up my memory usage more than using a resnet50. This is the code that I’m using:</p>
<pre><code class="lang-auto">import os

from tqdm import tqdm
from torchvision.transforms import ToTensor
from torchvision.models import resnet50, resnet18
from torch.utils.data import DataLoader
from torch.utils.data import Dataset
import torch
import torch.nn as nn
import cv2

n = 1000000

def count_parameters(model):
    return sum(p.numel() for p in model.parameters() if p.requires_grad)

class RandomDs(Dataset):
    def __init__(self, ):
        pass

    def __len__(self):
        return n

    def __getitem__(self, index):
        return torch.rand(3, 256, 256)


if __name__ == '__main__':

    dataset = RandomDs()
    data_loader = DataLoader(dataset, batch_size=128, shuffle=False, num_workers=4, pin_memory=False)


    model = resnet50()
    # model = nn.Conv2d(3, 128, 3)

    os.environ["CUDA_VISIBLE_DEVICES"] = "0"
    device = torch.device('cuda:0')
    # model = nn.DataParallel(model)
    
    model.to(device)

    s = f"{count_parameters(model):.3e} trainable parameters"
    print(s)

    with torch.no_grad():
        for batch in tqdm(data_loader):
            batch = batch.to(device)
            model(batch)

</code></pre>
<p>I ran this code with 4 configurations (<code>model</code> is either <code>resnet50</code> or <code>nn.Conv2d(3, 128, 3)</code>; and <code>DataParallel</code> is turned on or off (off means I just use a single GPU).</p>
<p>For <code>Conv2D</code>, single GPU I get<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/6/c6a9ddc212a40a076cba8a819674b9b4775eeeea.png" data-download-href="https://discuss.pytorch.org/uploads/default/c6a9ddc212a40a076cba8a819674b9b4775eeeea" title="Screenshot from 2020-04-20 12-36-05"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/6/c6a9ddc212a40a076cba8a819674b9b4775eeeea.png" alt="Screenshot from 2020-04-20 12-36-05" data-base62-sha1="slstvl8FvPu7uvIb8SAbieXxMKu" width="690" height="384" data-dominant-color="0B0B0B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-04-20 12-36-05</span><span class="informations">727×405 40.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>For <code>Conv2D</code>, DataParallel I get<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/6/d6c9fe5d363f6e3dd59e2e999bd8302d9c1833eb.png" data-download-href="https://discuss.pytorch.org/uploads/default/d6c9fe5d363f6e3dd59e2e999bd8302d9c1833eb" title="Screenshot from 2020-04-20 12-37-15"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/6/d6c9fe5d363f6e3dd59e2e999bd8302d9c1833eb.png" alt="Screenshot from 2020-04-20 12-37-15" data-base62-sha1="uE6WM4mIEbqSMMswBBSvCd8pIxB" width="690" height="406" data-dominant-color="080808"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-04-20 12-37-15</span><span class="informations">731×431 40.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>For <code>resnet50</code>, single GPU I get<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/b/6b2589f1ae66fe2234ad4ea9c53b4313aec32d67.png" data-download-href="https://discuss.pytorch.org/uploads/default/6b2589f1ae66fe2234ad4ea9c53b4313aec32d67" title="Screenshot from 2020-04-20 12-40-17"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/b/6b2589f1ae66fe2234ad4ea9c53b4313aec32d67.png" alt="Screenshot from 2020-04-20 12-40-17" data-base62-sha1="fhRvZh33yvgrUjGLsPrZPaIfOT5" width="690" height="391" data-dominant-color="090909"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-04-20 12-40-17</span><span class="informations">720×408 38.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>For <code> resnet50</code>, DataParallel I get<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/e/6ef0bfb6ce49b275a5f99279556aff6d7e5d498e.png" data-download-href="https://discuss.pytorch.org/uploads/default/6ef0bfb6ce49b275a5f99279556aff6d7e5d498e" title="Screenshot from 2020-04-20 12-41-34"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/e/6ef0bfb6ce49b275a5f99279556aff6d7e5d498e.png" alt="Screenshot from 2020-04-20 12-41-34" data-base62-sha1="fPqk4Ta1HMzeYsXu2kshpYuzDga" width="690" height="400" data-dominant-color="080808"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-04-20 12-41-34</span><span class="informations">739×429 41 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Why is that? <code>Conv2D</code> uses way more memory than <code>resnet50</code> and memory increases for <code>Conv2D</code> if I use <code>Dataparallel</code>.</p>
<p>I am using Cuda 10.2 and pytorch  1.2.0</p>

albanD: <p>Hi,</p>
<p>Keep in mind that pytorch won’t release the memory after using it to make follow up allocation faster.<br>
So this does not necessarily shows the overall needed memory but the maximum allocated memory.</p>
<p>Also you might want to check using torch.backends.cudnn.benchmark to allow it to find good kernels.<br>
I am sure cudnn is heavily optimized for resnet’s convolutions <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

AljoSt: <p>hmm ok, but we are talking here about a 3x3 convolution with 3 input and 128 output channels. So less than 4000 parameters, compared to more than 10,000,000 of resnet50. For a similar operation it’s allocating twice the amount of memory, while the model only has 0.04% of the parameters?</p>

albanD: <p>Depending on the algorithm you use to perform the convolution, the intermediary results can be very (very) big. In particular, if you do it as a matrix multiplication, the unfolded Tensors will be huge here.</p>
<p>The reason I recommend benchmark is that it will adapt the algorithm to the inputs instead of simply using the default one. This should help you get lower memory (and better speed after the first call ran the benchmark).</p>