The tensor storage address do no change during different iterations

username: Cru
cooked: <p>Hi,<br>
I have print the tensor storage information during forward and backward, as this code snippet. I have inserted in pytorch c++ backend source code.</p>
<pre><code class="lang-auto">    void Record_tensors(std::string func_name, std::vector&lt;std::pair&lt;std::string, std::reference_wrapper&lt;at::Tensor&gt;&gt;&gt; datas){
        auto now = Clock::now();
        Duration duration = std::chrono::duration_cast&lt;FinalTime&gt;(now - startTime);
        out &lt;&lt; func_name &lt;&lt; " " &lt;&lt; duration.count() &lt;&lt; "ns ";
        for (auto data: datas){
            if(data.second.get().has_storage()){
            void* storage_ptr = data.second.get().data_ptr();
            out &lt;&lt; data.first &lt;&lt;" : "&lt;&lt;storage_ptr&lt;&lt;" ";
            }
        }
        out&lt;&lt;std::endl;
    }
</code></pre>
<p>however, the result really confused me.<br>
The resnet-50 network is used in my test, for the 1st iteration during training, the storage address of the first conv is 0x7fd25a900000.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6a3e31c7bf77c93f43d8780251426875f00d2446.png" data-download-href="https://discuss.pytorch.org/uploads/default/6a3e31c7bf77c93f43d8780251426875f00d2446" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6a3e31c7bf77c93f43d8780251426875f00d2446.png" alt="image" data-base62-sha1="f9RRuSlcq545UBnEUMmhweMx7H8" width="690" height="190" data-dominant-color="2A2B2C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">790×218 15.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
And, it will still be used in backward process, as follow,<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/4/44827e660b85d674ae2b4a794b6a715721f4d4b7.png" data-download-href="https://discuss.pytorch.org/uploads/default/44827e660b85d674ae2b4a794b6a715721f4d4b7" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/44827e660b85d674ae2b4a794b6a715721f4d4b7_2_690x149.png" alt="image" data-base62-sha1="9M434sT203QIYKBlMgba8KudpaL" width="690" height="149" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/44827e660b85d674ae2b4a794b6a715721f4d4b7_2_690x149.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/44827e660b85d674ae2b4a794b6a715721f4d4b7_2_1035x223.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/4/44827e660b85d674ae2b4a794b6a715721f4d4b7.png 2x" data-dominant-color="292B2D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1355×293 40.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div>.<br>
This seems normal, and then the following iterations process is weird. <img src="https://discuss.pytorch.org/images/emoji/apple/melting_face.png?v=12" title=":melting_face:" class="emoji" alt=":melting_face:" loading="lazy" width="20" height="20"><br>
This is the storage address of the first conv in 2nd iteration:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/9/591d640c828c992c805b587e7d271ad1d8e0e1fe.png" data-download-href="https://discuss.pytorch.org/uploads/default/591d640c828c992c805b587e7d271ad1d8e0e1fe" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/9/591d640c828c992c805b587e7d271ad1d8e0e1fe.png" alt="image" data-base62-sha1="cIlsNa8IIzUHnegzulAVYAF8yBM" width="690" height="222" data-dominant-color="2B2D2F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">832×268 15.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
and it is same as the 3rd iteration and it’s the same as the iterations after that:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/7/470a2fcce1c637d5727c062d2e2cc0562d6d37c1.png" data-download-href="https://discuss.pytorch.org/uploads/default/470a2fcce1c637d5727c062d2e2cc0562d6d37c1" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/7/470a2fcce1c637d5727c062d2e2cc0562d6d37c1.png" alt="image" data-base62-sha1="a8rIYeVZyng0mAVjOD6saWhvoRP" width="690" height="188" data-dominant-color="28292A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1345×367 39.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/8/48096d7dccd80b959b4a1ce0057cdeb609c99dae.png" data-download-href="https://discuss.pytorch.org/uploads/default/48096d7dccd80b959b4a1ce0057cdeb609c99dae" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/8/48096d7dccd80b959b4a1ce0057cdeb609c99dae.png" alt="image" data-base62-sha1="ahgzKyDpirlEq3nkJZRUpVG9WdE" width="690" height="132" data-dominant-color="28292A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1312×252 27.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
the different line number can reperesent for the different iterations. This is the same phenomenon for other intermediate results.<br>
Can someone explain why this happens, because the address of the first iteration and the second iteration are different, I don’t think it is a problem of the code I wrote. Is there a sharing memory mechanism for the Pytorch?</p>

username: ptrblck
cooked: <p>Memory can be reused so why should the object not be able to allocate the same memory location?</p>

username: Cru
cooked: <p>Thank you for your reply, which was very helpful and cleared up my doubts. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>