Does torch.cholesky have a memory leak?

username: Raphikowski
cooked: <p>Hi all,</p>
<p>I recognized that my code consumes more and more memory with each epoch that I am training my model. I have tracked the memory leak down by commenting out parts of my code and its seems that torch.choleksy is the culprit (If i only comment out the line with torch.cholesky my code has no increase in memory usage).<br>
I am training on CPU, so this is not an issure of the GPU memory but of the actual main memory of the pc. I have seen that in the past there has already been problems with memory leakage and torch.choleksy, however, with respect to gpu ram:<br>
</p><aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/15539" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/15539" target="_blank" rel="nofollow noopener">Memory leak in torch.cholesky</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2018-12-26" data-time="13:09:37" data-timezone="UTC">01:09PM - 26 Dec 18 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2019-02-07" data-time="06:00:28" data-timezone="UTC">06:00AM - 07 Feb 19 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/ozzzp" target="_blank" rel="nofollow noopener">
          <img alt="ozzzp" src="https://avatars3.githubusercontent.com/u/2487585?v=4" class="onebox-avatar-inline" width="20" height="20">
          ozzzp
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🐛 Bug
There exists memory leak of GPU memory in torch.cholesky
Code
import torch
def main():
 def test_cholesky():
 A = torch.rand([4, 1280, 1280]).cuda()
 H =...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Unfortunately, my model is far too large to provide the code here. So I just wanted to ask if someone has had similar issues?</p>
<p>I check the memory consumption with:<br>
77 def memory_usage_psutil():<br>
78     # return the memory usage in MB<br>
79     process = psutil.Process(os.getpid())<br>
80     mem = process.memory_info().rss / float(10 ** 6) # /float(2 ** 20)<br>
81     return mem</p>

username: albanD
cooked: <p>Hi,</p>
<p>The issue you linked does mention that old version of magma have memory leaks. Make sure that you use a recent enough version of magma (and if you use pytorch binaries, the latest stable version or nightly).</p>

username: Raphikowski
cooked: <p>Hi,</p>
<p>thank you for your reply. Do think that this issue might also affect the main memory? (in that link they only talk about gpu memory)<br>
Have you ever heard of someone having a similar issue like me with this function?<br>
I am just wondering because this forum thread I have linked is like 2 years old and I have installed my current pytorch about half a year ago so I would assume that I have a MAGMA version that does not have this bug anymore</p>

username: albanD
cooked: <p>No I don’t recall any other issue like that.<br>
Are you sure that you don’t hold on to some memory across iterations?</p>
<p>Does passing the exact same inputs to cholesky outside of your model gives the same behavior?</p>

username: Raphikowski
cooked: <p>Unfortunately, it is pretty difficult to do it outside of my model because there are many computations performed with the data before actually handing them to torch.cholesky, but as I said, if I comment out the line with the torch.cholesky call, I dont have these memory issues…</p>
<p>But still, maybe you re right and these issues stem from something else but only get revealed somehow through the usage of torch.cholesky.<br>
When you say “Are you sure that you don’t hold on to some memory across iterations?” what do you mean by that? What kind of functions or coding could lead to holding onto memory?</p>

username: albanD
cooked: <blockquote>
<p>Unfortunately, it is pretty difficult to do it outside of my model because there are many computations performed with the data before actually handing them to torch.cholesky</p>
</blockquote>
<p>What about just logging the size of the matrix and try with a matrix with random values?</p>
<blockquote>
<p>When you say “Are you sure that you don’t hold on to some memory across iterations?” what do you mean by that? What kind of functions or coding could lead to holding onto memory?</p>
</blockquote>
<p>This can happen mostly if store data in a data structure at each iteration, always appending. Accumulate across iteration (like a epoch_loss) Tensors that still require gradients and hold onto their history</p>

username: Raphikowski
cooked: <p>So I tried what you suggested and just always created a random tensor (in each iteration a new random tensor is created) with torch.randn of the same shape as the data. Then making this random tensor positive definite by matrix multiplying it with its transpose and then computing torch.cholesky of that random matrix instead of calculating torch cholesky of the data. And the result in terms of memory usage is the same: I.e., the memory usage still increased every epoch.<br>
And in case that I comment out that torch.choleksy call, the memory usage does still increase but only marginally and only every 20 epochs or so and orders of magnitude smaller. So to me it seems like torch.cholesky was the culprit.</p>
<p>Or could you imagine that actually another error is the root of all evil and is only enhanced by the torch cholesky call? I.e., could you imagine that still one of the other possible issues you mentioned could be at the heart of the problem?</p>

username: albanD
cooked: <aside class="quote no-group" data-username="Raphikowski" data-post="7" data-topic="88350">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/ec9cab/48.png" class="avatar"> Raphikowski:</div>
<blockquote>
<p>Or could you imagine that actually another error is the root of all evil and is only enhanced by the torch cholesky call?</p>
</blockquote>
</aside>
<p>Given that you still see issues without it, it is possible yes.</p>
<ul>
<li>Does calling <code>gc.collect()</code> at every iteration solves the issues (both with and without the cholesky)?</li>
<li>What do you use when you remove the cholesky, do you link the output with the input with a differentiable manner? That could explain the reduction in memory when you remove it.</li>
<li>Do you use <code>.backward(create_graph=True)</code> ?</li>
</ul>