How loss.backward() reduces memory usage than expected?

Jang_BongWon: <p>According to these links, I could understand that non-leaf variables’ gradients are not retained to save memory usage during backpropagation.<br>
<a href="https://discuss.pytorch.org/t/calling-loss-backward-reduce-memory-usage/2735">link 1</a><br>
<a href="https://discuss.pytorch.org/t/why-cant-i-see-grad-of-an-intermediate-variable/94">link 2</a></p>
<p>However, I still wonder <strong>How the memory saving method works</strong>.<br>
Before asking the question precisely, please let me tell you my situation.</p>
<p>Currently, I am programming a simple deep learning framework for my project using CUDA/C++.<br>
The model training structure of my framework is as follows:<br>
<a href="https://imgur.com/a/wakUxJv" rel="noopener nofollow ugc">Image1 Link</a><br>
Which is really simple and directly implemented structure. I hope you understand this image easily.</p>
<p>Then, I started to compare the memory usage between my framework and Pytorch.<br>
I tested ResNet50 model with ImageNet dataset.<br>
(I guess I almost successfully implemented the structure of torchvision resnet50 model architecture in my framework.)<br>
Relu inplace operation and nesterov momentum method is applied.<br>
And this table is the comparison results:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/1/212b5d2a0c615db500fababd2caddac97965b9d1.png" alt="image" data-base62-sha1="4JqF7Jdjf2rGDZnAUtEEt7AMTbH" width="339" height="200"></p>
<p>As you can see, the increasing memory size of My framework is almost twice bigger than Pytorch.<br>
I tried to find out what makes such difference. Then, I found the fact that Pytorch reduces memory usage at the backpropagation by make intermediate buffer free as the links mentioned.</p>
<p>However, I still could not find <strong>How such memory saving method works in Pytorch</strong>.<br>
Below image shows my guess on the memory saving technique.<br>
At backpropagation, since the old gradient value is not necessary anymore, it can be freed.<br>
<strong>To do so, making 2 shared buffer for input grad and output grad</strong>.<br>
Then such buffer is used for temporarily saving and computing gradient values.<br>
(I don’t think allocating and deleting GPU memory space at every backpropagation step is proper approach. Since frequently managing GPU memory can reduce hardware efficiency.)<br>
<a href="https://imgur.com/a/Pbe8rAq" rel="noopener nofollow ugc">Image2 link</a></p>
<p>I’m not sure if I guess the backpropagation workflow correctly.<br>
I would be really appreciate if you teach me the magic of Pytorch backpropagation…</p>

soulitzer: <aside class="quote no-group" data-username="Jang_BongWon" data-post="1" data-topic="177264">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jang_bongwon/48/58713_2.png" class="avatar"> Jang_BongWon:</div>
<blockquote>
<p>Since frequently managing GPU memory can reduce hardware efficiency.</p>
</blockquote>
</aside>
<p>We don’t actually call cuda{free,malloc} everytime we allocate or free tensors. GPU memory is actually managed by a cuda caching allocator. <a href="https://zdevito.github.io/2022/08/04/cuda-caching-allocator.html" class="inline-onebox" rel="noopener nofollow ugc">A guide to PyTorch’s CUDA Caching Allocator | Zach’s Blog</a></p>
<p>Does that answer your question?</p>

Jang_BongWon: <p>Thank you for your answer!</p>
<p>Well, then can I think such memory block (a cache of allocations) method is used during backpropagation to save memory usage?<br>
If so, all questions will be solved.</p>