Two equivalent groups of parameters have very different file sizes when stored on .pth

alet: <p>Hi,</p>
<p>I have two groups of parameters (a test dataset and an extrapolation dataset, with the same number of elements) saved on two different .pth files which supposedly should have the same size. I’ve used the (<a href="https://discuss.pytorch.org/t/why-does-a-pth-file-take-so-much-larger-space-i-am-missing-something/49579/5">this comment</a> to compute the memory and they both return 73,400,648. My training instead returns 384,420,168. In particular I run:</p>
<pre><code class="lang-auto">sum ( p.nelement() * p.element_size() for p in extrapolation.parameters())
</code></pre>
<p>However the size files are very different (extrapolation 71M and test 417M), the latter being almost the same size as train (433M).<br>
Train and test come from the same data distribution, but not that different from extrapolation; also not sure if data distribution (not tensor type and size) should be the primary cause of .pth file size.<br>
Running <code>for k in extrapolation: print(extrapolation[k].dtype)</code> I’ve made sure both test and extrapolation only have torch.float32 tensors and when comparing tensors I get identical results for test and validation, with train differing in total number of points:</p>
<pre><code class="lang-auto">for k in test: print(extrapolation[k].shape, test[k].shape, train[k].shape)

torch.Size([2048, 100, 20]) torch.Size([2048, 100, 20]) torch.Size([10726, 100, 20])
torch.Size([2048, 99, 20]) torch.Size([2048, 99, 20]) torch.Size([10726, 99, 20])
torch.Size([2048, 100, 5]) torch.Size([2048, 100, 5]) torch.Size([10726, 100, 5])
torch.Size([1, 20]) torch.Size([1, 20]) torch.Size([1, 20])
torch.Size([1, 20]) torch.Size([1, 20]) torch.Size([1, 20])
torch.Size([1]) torch.Size([1]) torch.Size([1])
torch.Size([1, 20]) torch.Size([1, 20]) torch.Size([1, 20])
torch.Size([1, 20]) torch.Size([1, 20]) torch.Size([1, 20])
torch.Size([1]) torch.Size([1]) torch.Size([1])
torch.Size([2048, 100, 20]) torch.Size([2048, 100, 20]) torch.Size([10726, 100, 20])
torch.Size([2048, 99, 20]) torch.Size([2048, 99, 20]) torch.Size([10726, 99, 20])
torch.Size([2048, 100, 5]) torch.Size([2048, 100, 5]) torch.Size([10726, 100, 5])
</code></pre>
<p>What could be going on?</p>
<p>Thanks!</p>

ptrblck: <p>I’m not sure I understand the datasets correctly, so please feel free to correct me. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>It seems you are dealing with “groups of parameters”, i.e. are these different <code>state_dicts</code> of different models?<br>
Both return 73 million parameters or MB?<br>
What does “My training instead returns 384,420,168.” mean? Does it mean that the number changes if you run it during training? How was it run before?</p>
<p>The last loop loops as if you are printing shapes of data samples from different datasets?</p>

alet: <p>Yes, sorry I’m dealing with groups of parameters <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I’m trying to store train, test, test2 (extrapolation), so they’re not really models, more like groups of Tensors for each dataset (mostly X,y, some variations of X,y and statistics about them). They should have the same size and types except that train has 10,726 elements and test&amp;extrapolation have 2,048 elements each.</p>
<p>Both test&amp;extrapolation return 73 million parameters,  but extrapolation file is 71MB and test is 417MB (sorry, I now see I put ‘M’ not ‘MB’ on the original post). Train has 384 million parameters and its file is 433MB.</p>
<p>The last loops are indeed printing the shape of each parameter in the data, since they all have parameters named the same it shows that test&amp;extrapolation have the same size for all parameters and train has some parameters bigger on the first dimension (which makes sense).</p>
<p>Basically, the only thing that doesn’t make sense to me is the comparative size of the test file, as I would expect it to be the same as the extrapolation given that all the internal types and sizes are the same.</p>
<p>Thanks for your help,</p>

ptrblck: <p>Thanks for the information.<br>
I cannot reproduce this issue using your shapes:</p>
<pre><code class="lang-python">s1 = [torch.Size([2048, 100, 20]),
      torch.Size([2048, 99, 20]),
      torch.Size([2048, 100, 5]),
      torch.Size([1, 20]),
      torch.Size([1, 20]),
      torch.Size([1]),
      torch.Size([1, 20]),
      torch.Size([1, 20]),
      torch.Size([1]),
      torch.Size([2048, 100, 20]), 
      torch.Size([2048, 99, 20]),
      torch.Size([2048, 100, 5])]

x1 = [torch.randn(s) for s in s1]


s2 = [torch.Size([2048, 100, 20]),
      torch.Size([2048, 99, 20]),
      torch.Size([2048, 100, 5]),
      torch.Size([1, 20]),
      torch.Size([1, 20]),
      torch.Size([1]),
      torch.Size([1, 20]),
      torch.Size([1, 20]),
      torch.Size([1]),
      torch.Size([2048, 100, 20]),
      torch.Size([2048, 99, 20]),
      torch.Size([2048, 100, 5])]

x2 = [torch.randn(s) for s in s2]


s3 = [torch.Size([10726, 100, 20]),
      torch.Size([10726, 99, 20]),
      torch.Size([10726, 100, 5]),
      torch.Size([1, 20]),
      torch.Size([1, 20]),
      torch.Size([1]),
      torch.Size([1, 20]),
      torch.Size([1, 20]),
      torch.Size([1]),
      torch.Size([10726, 100, 20]),
      torch.Size([10726, 99, 20]),
      torch.Size([10726, 100, 5])]
    
x3 = [torch.randn(s) for s in s3]
    
torch.save(x1, 'x1.pt')
torch.save(x2, 'x2.pt')
torch.save(x3, 'x3.pt')

import os
print(os.path.getsize('x1.pt'))
&gt; 73402073
print(os.path.getsize('x2.pt'))
&gt; 73402073
print(os.path.getsize('x3.pt'))
&gt; 384421593
</code></pre>
<p>Could you check my code for differences?</p>

Rolihlahla: <p>Rekindling this post.  My colleague and I are having the same problem.   I will try to do two things (1) Give you some more specific information and (2) come up with the simplest version I can to reproduce the behavior.</p>
<p>First the extra information for my case. If I understand the poster correctly, he has to lists of tensors of the same types and sizes.  When he saves the list of tensors to a file, he gets very different file sizes.  One would expect the file sizes to be the exact same if the lists contained the same size and dtype of tensors.</p>
<p>Our problem is the same except the tensors are identical!  However, the way we are generating the tensors are not identical.  We are working with wav files and our features are time slices of a STFT that have large overlapping portions.  Our (X,y) pairs are both dtype float32 of sizes  (1,41,128) and (1) respectively.  Let’s ignore the y elements for a moment (I don’t think this is our issue).  When we generate the data, we are taking overlapping portions of the STFT,  So if the entire STFT of the wav file is a tensor D of size (1,1000,128)  and XD is our list of X data elements, then  XD[0] will be a slice of the first 41 out of 1000 columns of D starting at 0,  so XD[0] = W[:,0:41,:]  and the next element in the list  XD[1] = W[:,1:42,:], etc… So in this example I have a python list of length 959=1000-41 long with tensors of size (1,41,128).   If I were to clone each element of XD to a different list XD2 and save this, I get a file size that is very different.  Sample code below:</p>
<p>import torch<br>
W = torch.randn(torch.Size([1,10000,128])<br>
XD = [W[:,idx:idx+41,:] for idx in range(10000-41)]<br>
XD2 = [torch.clone(X) for X in XD]<br>
torch.save(XD,‘xd.pt’)<br>
torch.save(XD2,‘xd2.pt’)</p>
<p>import os<br>
print(os.path.getsize(‘xd.pt’))</p>
<blockquote>
<p>5895723</p>
</blockquote>
<p>print(os.path.getsize(‘xd2.pt’))</p>
<blockquote>
<p>211804978</p>
</blockquote>
<p>Both of the python lists contain the same elements, so we are guessing it has something to do with the slicing, indexing or ordering of the elements and something that is going on under the covers.  We thought order mattered in the list so we then did:</p>
<p>import random<br>
random.shuffle(XD)</p>
<p>and</p>
<p>torch.save(XD,‘xdshuf.pt’)<br>
print(os.path.getsize(‘xdshuf.pt’))</p>
<blockquote>
<p>5895723</p>
</blockquote>
<p>And now we are really perplexed!  We came across this when trying to thread the data generation and had to send the data over a pipe thread.  We have since managed to get the file with the smaller size by using torch’s byte_io method, but we are wondering what exactly is going on here.  Any info much appreciated - and we love pytorch by the way  <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>Your issue sounds a bit similar to <a href="https://discuss.pytorch.org/t/save-file-size-doubled-after-loading-scheduler-checkpoint/145435/4">this post</a>. In any case, if you are still seeing this unexpected behavior in the latest nightly binary or master build, could you create an issue on GitHub with a minimal, executable code snippet, please?</p>

Rolihlahla: <p>Thanks for the quick reply.  I read the referenced post and I don’t believe it is related to that.  Also, I may not have been clear in what I explained,  but it is not necessarily a bug,  more just something that pytorch is doing under the covers that we don’t understand.  To reproduce what we are seeing,  here is the code snippett:</p>
<p>import torch<br>
W = torch.randn(torch.Size([1,10000,128])<br>
XD = [W[:,idx:idx+41,:] for idx in range(10000-41)]<br>
XD2 = [torch.clone(X) for X in XD]<br>
torch.save(XD,‘xd.pt’)<br>
torch.save(XD2,‘xd2.pt’)</p>
<p>when you compare the lists XD and XD2 they are exactly the same,  but when you compare the .pt files that were saved off, xd2.pt is 35 times bigger than xd.pt.   211804978 vs.  5895723.</p>

ptrblck: <p>Ah I see. Based on this code snippet <code>XD</code> is storing the views of the slicing operation while you are explicitly cloning the data in <code>XD2</code> and thus trigger a copy.</p>
<p>Have a look at:</p>
<pre><code class="lang-python"># inplace op on view
x = torch.zeros(5)
xd = x[1:2]
xd.fill_(1.)

print(x)
# tensor([0., 1., 0., 0., 0.])
print(xd)
# tensor([1.])

# clone
x = torch.zeros(5)
xd = torch.clone(x[1:2])
xd.fill_(1.)

print(x)
# tensor([0., 0., 0., 0., 0.])
print(xd)
# tensor([1.])
</code></pre>
<p>Here you can also see that the first approach is able to manipulate the original <code>x</code> tensor, while the second works on a copy, so I think the size increase might be expected in your posted code snippet.</p>