ERROR - Exception caught while uploading models: Error(s) in loading state_dict for Ensemble: Unexpected key(s) in state_dict

username: Abhijit_Manepatil
cooked: <p>was trying to load yolov5s.pt on CPU first then want to transfer to GPU with the following way:</p>
<pre><code class="lang-auto">class Ensemble(nn.ModuleList):
    # Ensemble of models
    def __init__(self):
        super(Ensemble, self).__init__()

    def forward(self, x, augment=False):
        y = []
        for module in self:
            y.append(module(x, augment)[0])
        # y = torch.stack(y).max(0)[0]  # max ensemble
        # y = torch.stack(y).mean(0)  # mean ensemble
        y = torch.cat(y, 1)  # nms ensemble
        return y, None  # inference, train output

model = Ensemble()
params = torch.load('yolov5s.pt' map_location='cpu')
model.load_state_dict(params)
ckpt = model.to('cuda:0')
</code></pre>
<p>reference from <a href="https://github.com/pytorch/pytorch/issues/7415#issuecomment-693424574" rel="noopener nofollow ugc">link</a><br>
but gives me the following error:</p>
<p><em>ERROR - Exception caught while uploading models: Error(s) in loading state_dict for Ensemble:<br>
Unexpected key(s) in state_dict: “epoch”, “best_fitness”, “training_results”, “model”, “optimizer”, “wandb_id”.</em></p>
<p>Help needed, Thanks</p>

username: ptrblck
cooked: <p><code>params</code> seems to be a <code>dict</code> object containing the model’s <code>state_dict</code> in its <code>['model']</code> key as well as other objects such as the optimizer’s <code>state_dict</code> in <code>['optimizer']</code>, the training results etc. as given in the error message.<br>
Index the <code>'model'</code> key and it might work,</p>

username: Abhijit_Manepatil
cooked: <p>Hi, Thanks for your response,<br>
I tried the above solution but not worked.</p>

username: ptrblck
cooked: <p>Could you describe a bit more what exactly did not work?</p>

username: Abhijit_Manepatil
cooked: <p>Yes, while checking with above solution, I got the following error:</p>
<p>ERROR - Exception caught while uploading models: ‘Model’ object has no attribute ‘copy’</p>
<pre><code class="lang-auto">File "/home/user/anaconda3/envs/dali/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1455, in load_state_dict
    state_dict = state_dict.copy()
  File "/home/user/anaconda3/envs/dali/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1177, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'Model' object has no attribute 'copy'
</code></pre>
<p>can you please help and check?</p>

username: ptrblck
cooked: <p>It seems you’ve saved the model object directly instead if its <code>state_dict</code>, so you could try to assign the model directly via <code>model = params['model']</code>.</p>

username: Abhijit_Manepatil
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/8/5816b58c774dcf45e3d819132ba504c88036cc33.png" data-download-href="https://discuss.pytorch.org/uploads/default/5816b58c774dcf45e3d819132ba504c88036cc33" title="Screenshot from 2022-02-14 10-24-41"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/8/5816b58c774dcf45e3d819132ba504c88036cc33_2_690x470.png" alt="Screenshot from 2022-02-14 10-24-41" data-base62-sha1="czgFDyIG3HTwU64sbP3YCfrJA79" width="690" height="470" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/8/5816b58c774dcf45e3d819132ba504c88036cc33_2_690x470.png, https://discuss.pytorch.org/uploads/default/original/3X/5/8/5816b58c774dcf45e3d819132ba504c88036cc33.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/8/5816b58c774dcf45e3d819132ba504c88036cc33.png 2x" data-dominant-color="3A152E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2022-02-14 10-24-41</span><span class="informations">924×630 85.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/0/20907b04453773f84e869345db7163b8a1088284.png" data-download-href="https://discuss.pytorch.org/uploads/default/20907b04453773f84e869345db7163b8a1088284" title="Screenshot from 2022-02-14 10-26-00"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/0/20907b04453773f84e869345db7163b8a1088284_2_690x398.png" alt="Screenshot from 2022-02-14 10-26-00" data-base62-sha1="4E4PhDVuQtOsyr5Gh17tcwPb0So" width="690" height="398" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/0/20907b04453773f84e869345db7163b8a1088284_2_690x398.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/0/20907b04453773f84e869345db7163b8a1088284_2_1035x597.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/0/20907b04453773f84e869345db7163b8a1088284.png 2x" data-dominant-color="37122B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2022-02-14 10-26-00</span><span class="informations">1117×645 75.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I have got the above output as a param from torch.load(‘yolov5s.pt’ map_location=‘cpu’).<br>
I have used <code>model = params['model']</code> .<br>
got the following issue:</p>
<pre><code class="lang-auto"> File "models/load.py", line 73, in attempt_load
    model.append(ckpt['ema' if ckpt.get('ema') else 'model'].float().fuse().eval()) 
  File "/home/user/anaconda3/envs/dali/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1177, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'Model' object has no attribute 'append'
</code></pre>

username: ptrblck
cooked: <p>I assume <code>models.load.py</code> and <code>attempt_load</code> are custom methods, so you would need to check your code to see how the model loading is expected.<br>
Based on the new error it seems as if the used <code>model</code> object is supposed to be a <code>list</code>.</p>

username: Abhijit_Manepatil
cooked: <p>while loading model using CPU, it only usages 17.5MB size of memory<br>
see the following profiling result (check line number 35)</p>
<p>Line    Mem usage    Increment  Occurrences   Line Contents</p>
<pre><code>30    386.0 MiB    386.0 MiB           1   @profile
31                                         def attempt_load(weights, map_location=None):
32                                             # Loads an ensemble of models weights=[a,b,c] or a single model weights=[a] or weights=a
33    386.0 MiB      0.0 MiB           1       model = Ensemble()
34    422.7 MiB      0.0 MiB           2       for w in weights if isinstance(weights, list) else [weights]:
35    403.6 MiB     17.5 MiB           1           ckpt = torch.load(w, map_location='cpu')
</code></pre>
<p>Now, I tried the load same model on GPU cuda:0 but memory behavior is drastically changed,<br>
memory cpu ram surges to 2713.8 MiB, see the profiling result below(line no. 35):</p>
<p>Line    Mem usage    Increment  Occurrences   Line Contents</p>
<pre><code>30    385.5 MiB    385.5 MiB           1   @profile
31                                         def attempt_load(weights, map_location=None):
32                                             # Loads an ensemble of models weights=[a,b,c] or a single model weights=[a] or weights=a
33    385.5 MiB      0.0 MiB           1       model = Ensemble()
34   3099.6 MiB      0.0 MiB           2       for w in weights if isinstance(weights, list) else [weights]:
35  3099.3 MiB   2713.8 MiB            1           ckpt = torch.load(w, map_location='cuda:0')
</code></pre>
<p>that’s why I was looking for some solution referred from (<a href="https://github.com/pytorch/pytorch/issues/7415#issuecomment-693424574" rel="noopener nofollow ugc">link</a>) to load model first on cpu then transfer to gpu, with the above discussion I was trying to find a solution using state_dict but not resolved that yet, Want to understand why this memory surges even I changed the load device?</p>

username: ptrblck
cooked: <p>Assuming you are profiling the host RAM, the memory increase would be caused by loading all CUDA libs and is expected.<br>
The linked issue targets an OOM on the device, which I don’t know if you are running into.</p>

username: Abhijit_Manepatil
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>  Thanks for the response,<br>
As discussed and suggested I tried with following way to get <code>state_dict</code></p>
<p><code>model.load_state_dict(params['model'].state_dict(),strict=False)</code></p>
<p>with the above way, I was able to load <code>state_dict</code> which contains weights and biases, using <code>load_state_dict</code> to the existing model. which resolve my above-mentioned state_dict issue.</p>
<p>Now, I want to load the same model which I already loaded in CPU to GPU but the model is not able to migrate from cpu to gpu, even not getting any error but model stayed only on cpu even I tried to migrate it to GPU with following way:</p>
<pre><code class="lang-auto">params = torch.load('yolov5s.pt' map_location='cpu')
device = torch.device("cuda")
model.load_state_dict(params['model'].state_dict(),strict=False)  
model.to(device)
</code></pre>
<p>can you suggest, how should I do this from cpu loaded model to gpu?</p>

username: ptrblck
cooked: <p><code>model.to(device)</code> will move all registered parameters, buffers, and submodules to the specified <code>device</code> as seen here:</p>
<pre><code class="lang-python">model = models.resnet18()

for name, param in model.named_parameters():
    print(name, param.device)
# &gt; conv1.weight cpu
#   bn1.weight cpu
#   bn1.bias cpu
#   ...

for name, buf in model.named_buffers():
    print(name, buf.device)
# &gt; bn1.running_mean cpu
#   bn1.running_var cpu
#   bn1.num_batches_tracked cpu
#   ...

model.to('cuda')

for name, param in model.named_parameters():
    print(name, param.device)
# &gt; conv1.weight cuda:0
#   bn1.weight cuda:0
#   bn1.bias cuda:0
#   ...

for name, buf in model.named_buffers():
    print(name, buf.device)
# &gt; bn1.running_mean cuda:0
#   bn1.running_var cuda:0
#   bn1.num_batches_tracked cuda:0
#   ...
</code></pre>
<p>If this mechanism is not working in your model, I guess you have somehow “unregistered” the parameters, buffers, etc.</p>

username: Abhijit_Manepatil
cooked: <p>I have tested model in above-mentioned way,<br>
observation is : <code>model.named_parameters()</code> and <code>model.named_buffers()</code> shows the blank result, inside that no weights and bias. looks like as you mentioned it is “unregistered”.<br>
So how can I resolve this to do register those?</p>

username: ptrblck
cooked: <p>Did you manipulate the internal parameters, buffers, or submodules somehow by e.g. using <code>setattr</code> etc.?<br>
Parameters, buffers, and submodules (containing both) are directly registered once they are assigned to the parent module via:</p>
<pre><code class="lang-python">self.param_name = nn.Parameter(torch.randn(shape))
self.register_buffer('name', torch.randn(shape))
self.submodule = nn.Linear(1, 1)
</code></pre>