Is this memory use normal (ResUnet) : 7GO for one image

Lelouch1: <p>I am currently working with a R2U-Net from this paper <a href="https://arxiv.org/abs/1802.06955" class="inline-onebox" rel="noopener nofollow ugc">[1802.06955] Recurrent Residual Convolutional Neural Network based on U-Net (R2U-Net) for Medical Image Segmentation</a></p>
<p>The model contains ~40 millions parameters with a 1-&gt;64-&gt;128-&gt;256-&gt;512-&gt;1024-&gt;512-&gt;256-&gt;128-&gt;64-&gt;1 architecture, which is similar to a ResNet 101 in terms of number of paramater.</p>
<p>However I find that when I am using this model for training with  448x448 images, it takes almost 7GO of memory on my GPU which seems pretty big. I can only use a batch size of 1.</p>
<p>Do you this this is normal or would it rather comes from an unoptimized code ? I lack knowledge about plausible memory use for such architectures.</p>

ptrblck: <p>It might be expected and you could check the number of trainable parameters, buffers, and the shape of all intermediate activations to verify it. <a href="https://discuss.pytorch.org/t/resnet-50-takes-10-13gb-to-run-with-batch-size-of-96/117402/2">This post</a> gives you an example how to do so.<br>
Alternatively, you could also add debug print statements into the <code>forward</code> method and check how much memory is used where via <code>print(torch.cuda.memory_summary())</code>.</p>

Lelouch1: <p>I used your code and it tells me that it should take 2.7GB which is far from 6-7GB. However I have reccurent layers that will loop over the same conv module several times, so I suppose I have to change what you did to have a proper estimation because it’s underestimating the number of activations.</p>
<p>I also used print(torch.cuda.memory_summary()) at the end of a forward pass and it tells my that the current usage is 4.1GB and the peak usage is 8.4GB. The peaks seems to be a bit above what I observe but I suppose this is because I used a DataParallel and therefore the first gpu used a bit more  memory.</p>

ptrblck: <aside class="quote no-group" data-username="Lelouch1" data-post="3" data-topic="159861">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lelouch1/48/38438_2.png" class="avatar"> Lelouch1:</div>
<blockquote>
<p>I used your code and it tells me that it should take 2.7GB which is far from 6-7GB.</p>
</blockquote>
</aside>
<p>Did you only calculate the parameters/buffers or also the intermediate activations?</p>
<aside class="quote no-group" data-username="Lelouch1" data-post="3" data-topic="159861">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lelouch1/48/38438_2.png" class="avatar"> Lelouch1:</div>
<blockquote>
<p>However I have reccurent layers that will loop over the same conv module several times, so I suppose I have to change what you did to have a proper estimation because it’s underestimating the number of activations.</p>
</blockquote>
</aside>
<p>Yes, this sounds reasonable and my simple code snippet might not be able to capture the recurrent usage of modules. In any case, it is also not a perfect calculation, but can be used to estimate which part of the model uses the majority of memory.</p>