Large disk usage for some torch tensors (200MB vs 4MB) with same shape and dtype

SammyCui: <p>Hello,</p>
<p>I’m saving my model outputs as tensors by torch.save() under the no_grad() context manager. However, I noticed that from one job, tensors with shape (524, 720) and float32 consume ~200 MB disk usage, whereas from another set of tensors I created from a separate job with the same shape and dtype, they only take ~4MB. Also, when I take the clone() of the 200MB tensors, the cloned version takes a regular 4MB disk usage. I was wondering if it was due to gradient tracking that occupies the additional disk space, but both of them have requires_grad=False and have no grad function.  Interestingly, when I rerun the same code I got the regular 4MB tensors and cannot reproduce the 200MB ones. I’m using torch 2.0.1 on a linux server, and jobs are submitted using slurm.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/8/f8f4cbab610c772a6eaa068d16e041dd89300f4a.png" data-download-href="https://discuss.pytorch.org/uploads/default/f8f4cbab610c772a6eaa068d16e041dd89300f4a" title="Screenshot 2024-01-22 at 10.24.06 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/8/f8f4cbab610c772a6eaa068d16e041dd89300f4a_2_510x500.png" alt="Screenshot 2024-01-22 at 10.24.06 AM" data-base62-sha1="zwmSR8q0txQTvVHe9u03rv1fZ5E" width="510" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/8/f8f4cbab610c772a6eaa068d16e041dd89300f4a_2_510x500.png, https://discuss.pytorch.org/uploads/default/original/3X/f/8/f8f4cbab610c772a6eaa068d16e041dd89300f4a.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/8/f8f4cbab610c772a6eaa068d16e041dd89300f4a.png 2x" data-dominant-color="242424"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2024-01-22 at 10.24.06 AM</span><span class="informations">753×738 53.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What could be the reason behind the discrepancy in their disk usage?</p>
<p>Below is the link to the 200MB tensor uploaded to my google drive, if anyone would like to take a look <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
<aside class="onebox googledrive" data-onebox-src="https://drive.google.com/file/d/1jiDiDEUG0Tkxd8_Jov2RQ6RjhJU8vNnk/view?usp=sharing">
  <header class="source">

      <a href="https://drive.google.com/file/d/1jiDiDEUG0Tkxd8_Jov2RQ6RjhJU8vNnk/view?usp=sharing" target="_blank" rel="noopener nofollow ugc">drive.google.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://drive.google.com/file/d/1jiDiDEUG0Tkxd8_Jov2RQ6RjhJU8vNnk/view?usp=sharing" target="_blank" rel="noopener nofollow ugc"><span class="googledocs-onebox-logo g-drive-logo"></span></a>



<h3><a href="https://drive.google.com/file/d/1jiDiDEUG0Tkxd8_Jov2RQ6RjhJU8vNnk/view?usp=sharing" target="_blank" rel="noopener nofollow ugc">big.pth</a></h3>

<p>Google Drive file.</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Thank you so much for the help in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> And sorry if this is trivial…</p>

ptrblck: <p>This is a known issue observable when tensors are e.g. sliced:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(524, 720)
torch.save(x, "1.pt")
stats = os.stat("1.pt")
print("{}MB".format(stats.st_size / 1024**2))
# 1.4401836395263672MB

x = torch.randn(10000, 10000)
y = x[:524, :720]
torch.save(y, "2.pt")
stats = os.stat("2.pt")
print("{}MB".format(stats.st_size / 1024**2))
# 381.47070121765137MB

y = y.contiguous()
torch.save(y, "2.pt")
stats = os.stat("2.pt")
print("{}MB".format(stats.st_size / 1024**2))
# 1.4401836395263672MB
</code></pre>
<p>as the underlying tensor will be stored.</p>

SammyCui: <p>Thank you so much for the answer! This explains my problem precisely! There is indeed a slice in the code which I wasn’t aware of…</p>
<p>Many thanks!</p>