Runtime Error when loading pytorch model from .pkl

nabsabs: <p>Hi all,</p>
<p>I am using a cloud service that requires my model to be serialized with pickle. I’ve trained a unet model and saved the full model in .pth extension and .pkl but when I try to load the model from the .pkl format I get the following RuntimeError:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-81-14dbf4cf4d26&gt; in &lt;module&gt;()
----&gt; 1 unet = torch.load('unet_bus.pkl')
      2 unet.eval()

~/anaconda3/envs/pytorch/lib/python3.5/site-packages/torch/serialization.py in load(f, map_location, pickle_module)
    356         f = open(f, 'rb')
    357     try:
--&gt; 358         return _load(f, map_location, pickle_module)
    359     finally:
    360         if new_fd:

~/anaconda3/envs/pytorch/lib/python3.5/site-packages/torch/serialization.py in _load(f, map_location, pickle_module)
    532     magic_number = pickle_module.load(f)
    533     if magic_number != MAGIC_NUMBER:
--&gt; 534         raise RuntimeError("Invalid magic number; corrupt file?")
    535     protocol_version = pickle_module.load(f)
    536     if protocol_version != PROTOCOL_VERSION:

RuntimeError: Invalid magic number; corrupt file?
</code></pre>
<p>This is how I’m saving and loading the pickle model:</p>
<pre><code class="lang-auto">import pickle

pkl_filename = "unet_bus.pkl"  
with open(pkl_filename, 'wb') as file:  
    pickle.dump(model, file)

unet = torch.load('unet_bus.pkl')
unet.eval()
</code></pre>
<p>Any help appreciated!</p>

ptrblck: <p>Could you try to use <code>torch.save</code> instead of <code>pickle.dump</code>?</p>

nabsabs: <p>Nice thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> that worked! Do you know why writing it the former way is an issue?</p>

ptrblck: <p>It seems PyTorch uses a magic number to identify the file format or protocol as seen <a href="https://pytorch.org/docs/stable/_modules/torch/serialization.html" rel="nofollow noopener">here</a>. If you save it with another library and try to load it using PyTorch, you’ll encounter this error.</p>

nabsabs: <p>Ahh, cool! Didn’t know that. Thanks for your promptness <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Wwinwd: <p>How to solve this bug？<br>
Restoring states from the checkpoint path at /home/ww/Coding/AeDet/data/nuscenes/nuscenes_12hz_infos_train.pkl<br>
Traceback (most recent call last):<br>
File “/home/ww/Coding/AeDet/exps/aedet/aedet_lss_r50_256x704_128x128_24e_2key.py”, line 109, in <br>
run_cli()<br>
File “/home/ww/Coding/AeDet/exps/aedet/aedet_lss_r50_256x704_128x128_24e_2key.py”, line 105, in run_cli<br>
main(args)<br>
File “/home/ww/Coding/AeDet/exps/aedet/aedet_lss_r50_256x704_128x128_24e_2key.py”, line 75, in main<br>
trainer.fit(model, ckpt_path=args.ckpt_path)<br>
File “/home/ww/.conda/envs/aedet2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py”, line 771, in fit<br>
self._call_and_handle_interrupt(<br>
File “/home/ww/.conda/envs/aedet2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py”, line 722, in _call_and_handle_interrupt<br>
return self.strategy.launcher.launch(trainer_fn, *args, trainer=self, **kwargs)<br>
File “/home/ww/.conda/envs/aedet2/lib/python3.8/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py”, line 93, in launch<br>
return function(*args, **kwargs)<br>
File “/home/ww/.conda/envs/aedet2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py”, line 812, in _fit_impl<br>
results = self._run(model, ckpt_path=self.ckpt_path)<br>
File “/home/ww/.conda/envs/aedet2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py”, line 1180, in _run<br>
self._restore_modules_and_callbacks(ckpt_path)<br>
File “/home/ww/.conda/envs/aedet2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py”, line 1140, in _restore_modules_and_callbacks<br>
self._checkpoint_connector.resume_start(checkpoint_path)<br>
File “/home/ww/.conda/envs/aedet2/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/checkpoint_connector.py”, line 84, in resume_start<br>
self._loaded_checkpoint = self._load_and_validate_checkpoint(checkpoint_path)<br>
File “/home/ww/.conda/envs/aedet2/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/checkpoint_connector.py”, line 88, in _load_and_validate_checkpoint<br>
loaded_checkpoint = self.trainer.strategy.load_checkpoint(checkpoint_path)<br>
File “/home/ww/.conda/envs/aedet2/lib/python3.8/site-packages/pytorch_lightning/strategies/strategy.py”, line 316, in load_checkpoint<br>
return self.checkpoint_io.load_checkpoint(checkpoint_path)<br>
File “/home/ww/.conda/envs/aedet2/lib/python3.8/site-packages/pytorch_lightning/plugins/io/torch_plugin.py”, line 85, in load_checkpoint<br>
return pl_load(path, map_location=map_location)<br>
File “/home/ww/.conda/envs/aedet2/lib/python3.8/site-packages/pytorch_lightning/utilities/cloud_io.py”, line 47, in load<br>
return torch.load(f, map_location=map_location)<br>
File “/home/ww/.conda/envs/aedet2/lib/python3.8/site-packages/torch/serialization.py”, line 608, in load<br>
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)<br>
File “/home/ww/.conda/envs/aedet2/lib/python3.8/site-packages/torch/serialization.py”, line 780, in _legacy_load<br>
raise RuntimeError(“Invalid magic number; corrupt file?”)<br>
RuntimeError: Invalid magic number; corrupt file?<br>
Any help appreciated!</p>