Getting RuntimeError While Loading Model Weights

Harsh_Mittal: <p>I trained an LLM over an EC2 instance &amp; saved the model weights using torch.save().</p>
<p>When I’m trying to load the weights locally, I’m getting this error.</p>
<pre><code class="lang-auto">  File "/Users/harshmittal/Documents/GitHub/LLM Hustle/241M_Model_1.py", line 46, in &lt;module&gt;
    state_dict = torch.load("models/241M_Mode_1.pth")
  File "/Users/harshmittal/Library/Python/3.9/lib/python/site-packages/torch/serialization.py", line 1005, in load
    with _open_zipfile_reader(opened_file) as opened_zipfile:
  File "/Users/harshmittal/Library/Python/3.9/lib/python/site-packages/torch/serialization.py", line 457, in __init__
    super().__init__(torch._C.PyTorchFileReader(name_or_buffer))
RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory
</code></pre>
<p>While things are working perfectly fine, when I train, save the weights locally &amp; re-load them. I’m not sure why the weight file is getting corrupted on EC2.</p>
<p>Looking forward to a solution from the community <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
Cheers!</p>

ptrblck: <p>Are you able to open the checkpoint manually as it’s indeed an archive? If not, the file might be corrupt indeed.</p>

Harsh_Mittal: <p>Thanks for a reply <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>
<p>I’m new to this &amp; not sure about opening the checkpoint manually.</p>
<p>I saved the model weights using “torch.save(model_instance.state_dict(), file_path)” on an EC2 instance.</p>
<p>Then I simply downloaded the .pth file to my system.</p>
<p>Can you suggest me any robust practice to save model weights? I’d appreciate it <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=12" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:" loading="lazy" width="20" height="20"></p>
<p>Cheers!</p>

marksaroufim: <p>So when you <code>torch.load()</code> the file on the ec2 machine directly it works fine? In which case this almost certainly sounds like a data corruption when you’re doing the local download. How are you doing the local download?</p>

Harsh_Mittal: <p>Hi <a class="mention" href="/u/marksaroufim">@marksaroufim</a>,</p>
<p>&lt;&lt;&lt; So when you <code>torch.load()</code> the file on the ec2 machine directly it works fine? &gt;&gt;&gt;<br>
Yes! everything worked fine.</p>
<p>&lt;&lt;&lt;How are you doing the local download?&gt;&gt;&gt;<br>
I connected to the EC2 via VS Code SSH. I selected the file &amp; saw an option to download (within VS code).</p>
<p>Would appreciate your guidance on robust approach to save &amp; download model weights.</p>
<p>Cheers!</p>

marksaroufim: <p>Yeah VS Code’s download from file option can be very flaky and I’ve run into this issue many times</p>
<p>So I’d suggest either using scp which has a format like this</p>
<p><code>scp -i /path/to/privatekey.pem ec2-user@ec2-instance-public-dns:path/to/remote/file /local/destination</code></p>
<p>Or you can copy your artifacts to an S3 bucket and then download them from there</p>

Harsh_Mittal: <p>Oh wow! scp sounds like a robust solution to me.</p>
<p>Thanks <a class="mention" href="/u/marksaroufim">@marksaroufim</a> for sharing it. It’ll save me from lot of trouble in the future <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Btw - I really admire the engineering side of the Pytorch &amp; what an engineering marvel it is. It does so many things under the hood &amp; as a fresher, now I’m started getting the real meaning of a ‘Framework’.</p>
<p>Cheers!</p>