Segmentation fault when loading weight

david-leon: <p>When loading weight from file with<br>
<code>model.load_state_dict(torch.load(model_file))</code><br>
exception raised:<br>
<code>THCudaCheck FAIL file=/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/generic/THCStorage.c line=79 error=2 : out of memory Segmentation fault (core dumped)</code></p>
<p>Previously this runs with no problem, actually two training processes are still running (on another two GPUs), however this breaks when I want to start an additional training process.</p>

smth: <p>it seems that you are running out of memory on the GPUs</p>

david-leon: <aside class="quote no-group" data-username="david-leon" data-post="1" data-topic="1381">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/david-leon/48/231_2.png" class="avatar"> david-leon:</div>
<blockquote>
<p>torch.load(model_fil</p>
</blockquote>
</aside>
<p>No, the target GPU is idle, and there’re still 22GB memory available on this GPU.</p>

david-leon: <p>OK, I think I’ve got where the problem rises: the model weight saved with<br>
<code>torch.save(model.state_dict(), file)</code><br>
contains device info and<br>
<code>torch.load(model_file)</code> will load the weight directly into the device according to the saved device info rather than load into CPU. So, if the previously used device is short of memory, this loading process will crash.</p>

david-leon: <p>I’ve tried this:</p>
<pre><code class="lang-python">    def map_loc(storage, loc):
        if loc.startswith('cuda'):
            return storage.cuda(device)
        else:
            return storage
    print('model weights loading...')
    model.load_state_dict(torch.load(model_file,map_location=map_loc))
    print('model weights loaded')
</code></pre>
<p>And still, exception raised:</p>
<pre><code class="lang-auto">model weights loading...
THCudaCheck FAIL file=/data/users/soumith/builder/wheel/pytorch-src/torch/csrc/generic/serialization.cpp line=145 error=2 : out of memory
Traceback (most recent call last):
  File "PTR_evaluation_pytorch.py", line 197, in &lt;module&gt;
    model.load_state_dict(torch.load(model_file,map_location=map_loc))
  File "/home/David/App/anaconda3/lib/python3.5/site-packages/torch/serialization.py", line 222, in load
    return _load(f, map_location, pickle_module)
  File "/home/David/App/anaconda3/lib/python3.5/site-packages/torch/serialization.py", line 377, in _load
    deserialized_objects[key]._set_from_file(f, offset)
RuntimeError: cuda runtime error (2) : out of memory at /data/users/soumith/builder/wheel/pytorch-src/torch/csrc/generic/serialization.cpp:145
</code></pre>
<p>The target device is idle with over 20GB memory free.</p>

smth: <p>there was a bug in the serialization where remapping devices still used the device memory. this is fixed in master. i am working on binaries of version 0.1.11 and that will have this fix.</p>

david-leon: <p>Good to know that, thanks <a class="mention" href="/u/smth">@smth</a>.</p>

Zhang_Wen: <p><a class="mention" href="/u/david-leon">@david-leon</a>  hello, have you solved your problem ? i meet the same one, i want to load model in different device, and there are following errors :</p>
<p>THCudaCheck FAIL file=/py/conda-bld/pytorch_1493676237139/work/torch/lib/THC/generic/THCStorage.c line=79 error=2 : out of memory<br>
[1]    2276 segmentation fault (core dumped)  python guitar_rnnsearch_ia.py</p>

will_soon: <p>Is because the version of pytorch?</p>

todpole3: <p>Met the same problem today.</p>
<p>torch.<strong>version</strong> = ‘0.4.0a0+0876bab’</p>

Jonson: <p>I got the same problem, do you figure out how to solve it yet?</p>

ptrblck: <p>Did you specify the <code>map_location</code> while loading?<br>
What kind of error message did you get?</p>

Jonson: <p>Thanks! I have solved this problem by specifying <code>map_location</code> and using <code>torch.cuda.set_device()</code> to set target GPU manually.</p>

Debatrix: <p>Thanks! It’s work.<br>
<a href="https://pytorch.org/docs/stable/torch.html?highlight=load#torch.load" rel="noopener nofollow ugc">map_location </a></p>
<blockquote>
<p><strong>map_location</strong> – a function, torch.device, string or a dict specifying how to remap storage locations</p>
</blockquote>
<pre><code class="lang-auto">if torch.cuda.is_available() and cfg.use_gpu is not None:
    device = torch.device(use_gpu)
else:
    device = torch.device("cpu")
checkpoint_data = torch.load(checkpoint_path, map_location=device)
model.load_state_dict(checkpoint_data['model'])
optimizer.load_state_dict(checkpoint_data['optimizer'])
</code></pre>
<pre><code class="lang-auto">checkpoint_data = dict(
                    optimizer=optimizer.state_dict(),
                    model=model.state_dict(),
                )
</code></pre>

Bridget_Murphy: <p>Good luck on your project.</p>