RuntimeError: storage has wrong size

username: kaimss
cooked: <p>I use single machine with 4 gpus to train my models, and then I save paramaters of the model with <code>torch.save(aemodel.state_dict(), file)</code>. And when I use <code>aemodel.load_state_dict(torch.load(output_path + 'ae.pkl', map_location='cpu'))</code> to load its paramaters, error occurs:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "Run.py", line 714, in &lt;module&gt;
    ready_train(args)
  File "Run.py", line 384, in ready_train
    aemodel.load_state_dict(torch.load(output_path + 'ae.pkl', map_location='cpu'))
  File "/home_ex/tianhongtao/SW/anaconda3/envs/test/lib/python3.7/site-packages/torch/serialization.py", line 593, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "/home_ex/tianhongtao/SW/anaconda3/envs/test/lib/python3.7/site-packages/torch/serialization.py", line 780, in _legacy_load
    deserialized_objects[key]._set_from_file(f, offset, f_should_read_directly)
RuntimeError: storage has wrong size: expected 262144 got 1536

</code></pre>
<p>Could anyone tell me what happened? And how do I save and load model with distributed training? Thank you very much.</p>

username: ptrblck
cooked: <p>This error might be raised, if you were using multiple processes (e.g. via <code>DistributedDataParallel</code>) and didn’t guard the storing of the checkpoint against multiple writers.<br>
You could use the <a href="https://github.com/pytorch/examples/blob/e7870c1fd4706174f52a796521382c9342d4373f/imagenet/main.py#L254-L255">ImageNet example</a> to only use the first process to store the checkpoint.</p>

username: kaimss
cooked: <p>Thank you very much~I understand~</p>

username: Brando_Miranda
cooked: <p>this was fixed for me once I uploaded the checkpoint to the cluster again.</p>
<aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/71302308/issues-loading-pytorch-checkpoint-storage-error-wrong-size-how-to-fix/71302309#71302309">
  <header class="source">

      <a href="https://stackoverflow.com/questions/71302308/issues-loading-pytorch-checkpoint-storage-error-wrong-size-how-to-fix/71302309#71302309" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/1601580/charlie-parker" target="_blank" rel="noopener nofollow ugc">
    <img alt="Charlie Parker" src="https://i.stack.imgur.com/2B6rV.png?s=256&amp;g=1" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/71302308/issues-loading-pytorch-checkpoint-storage-error-wrong-size-how-to-fix/71302309#71302309" target="_blank" rel="noopener nofollow ugc">Issues loading pytorch checkpoint -- storage error wrong size -- how to fix?</a>
</h4>

<div class="tags">
  <strong>pytorch</strong>
</div>

<div class="date">
  
  answered by
  <a href="https://stackoverflow.com/users/1601580/charlie-parker" target="_blank" rel="noopener nofollow ugc">
    Charlie Parker
  </a>
  on <a href="https://stackoverflow.com/questions/71302308/issues-loading-pytorch-checkpoint-storage-error-wrong-size-how-to-fix/71302309#71302309" target="_blank" rel="noopener nofollow ugc">11:50PM - 28 Feb 22 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>