Delete state_dict keys

Kirtikumar_Pandya: <p>How can I remove model state_dict keys?</p>
<pre><code class="lang-auto">model_checkpoint_path = 'xyz.ckpt'
model_checkpoint_load = torch.load(model_checkpoint_path, map_location='cpu')
model_state_dict = model_checkpoint_load['state_dict']
model_state_dict = model_state_dict.copy()

for key in model_state_dict.keys():
    
    if key.startswith('loss'):
        model_state_dict.pop(key)
        # print(key) # ==&gt; This would not throw an error
</code></pre>
<p>Error:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[89], line 1
----&gt; 1 for key in model_state_dict.keys():
      3     if key.startswith('loss'):
      4         model_state_dict.pop(key)

RuntimeError: OrderedDict mutated during iteration
</code></pre>
<p>Thanking you.</p>

ptrblck: <p>You could collect the keys you want to remove in the loop and delete it afterwards:</p>
<pre><code class="lang-python">model = models.resnet18()

sd = model.state_dict()

layers_to_remove = []
for key in sd:
    if "conv" in key:
        layers_to_remove.append(key)
print(layers_to_remove)
# ['conv1.weight', 'layer1.0.conv1.weight', 'layer1.0.conv2.weight', 'layer1.1.conv1.weight', 'layer1.1.conv2.weight', 'layer2.0.conv1.weight', 'layer2.0.conv2.weight', 'layer2.1.conv1.weight', 'layer2.1.conv2.weight', 'layer3.0.conv1.weight', 'layer3.0.conv2.weight', 'layer3.1.conv1.weight', 'layer3.1.conv2.weight', 'layer4.0.conv1.weight', 'layer4.0.conv2.weight', 'layer4.1.conv1.weight', 'layer4.1.conv2.weight']

print(len(sd))
# 122
print(len(layers_to_remove))
# 17

for key in layers_to_remove:
    del sd[key]
print(len(sd))
# 105
</code></pre>

Kirtikumar_Pandya: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

dunlopj: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, I know this post has been marked as solved, but I am having a similar issue. Now I have a model that I’m quantizing. Before quantization, it goes through some calibration. Problem is, when I try to load the model from a checkpoint to resume training after server timeout, I get a missing keys error. The calibration features are not saved. I removed those from the state dict and saved in sd as you suggested, but how do I load the new sd into the model so that I can correctly load the saved model?</p>

ptrblck: <p>Could you post a code snippet showing how you are quantizing, storing, and reloading the model?</p>

dunlopj: <p>I quantize using IBM’s foundation model stack: <code>github.com/foundation-model-stack/fms-model-optimizer</code>. The call to quantize is:</p>
<p><code>self.model, optimizer = qmodel_prep(self.model, data_mb, self.qcfg,</code></p>
<pre><code>                            optimizer=optimizer,
                            save_fname=f"{base_name}_graph.pt",
                            qlast=self.args.qlast)`
</code></pre>
<p>Saving a model:</p>
<p><code>def save(self, model, path):</code></p>
<pre><code> state = {}
 self.save_to_state(model, state)

 path.parent.mkdir(parents=True, exist_ok=True)
 torch.save(state, path)`
</code></pre>
<p><code>def save_to_state(self, model, state):</code></p>
<pre><code>    state["model_hash"]  = self.model_hash
    state["model_state"] = model.state_dict()`
</code></pre>
<p>Loading saved checkpoint:</p>
<p><code>def _default_dict_loader(model, model_state):</code></p>
<pre><code>    model.load_state_dict(model_state)
    # if I set strict = False, it loads from the checkpoint, but testing returns all nans
</code></pre>
<p><code>def load(self, model, path, device, checkHash=True):</code></p>
<pre><code>    state = torch.load(path, map_location=device)
    self.load_from_state(model, state, checkHash)
</code></pre>
<p><code>def load_from_state(self, model, state, checkHash=True):</code></p>
<pre><code>    if checkHash and self.model_hash != state["model_hash"]:
        msg = "ERROR: saved model does not match current model params"
        print(msg) # log if we are logging
        print(msg, file=sys.stderr)
        exit(1)

    self.dict_loader(model, state["model_state"])
</code></pre>
<p>I used your example to modify the state_dict after the qmodel prep call, then attempt to load the new state_dict into the model using:</p>
<p><code>self.model.load_state_dict(state_dict, strict=False)</code><br>
<code># if strict=True, this fails.</code></p>

ptrblck: <p>Unfortunately, I’m not familiar with this repository and also don’t know how the <code>model_hash</code> is created, which seems to fail in your code?<br>
Do you get any proper error message while trying to load the <code>state_dict</code>?</p>

dunlopj: <p>Below is a snippet of the error messages. There are many more keys that are missing. There are quantized features in the saved model, however, those specific calibration features were not saved.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/4/4460a689cc5134d3e27217c13db720fe1b04191e.png" alt="Screenshot 2025-04-17 at 7.05.49 AM" data-base62-sha1="9KTxwU0VDLpNR6tUQr8jYM3cdzw" width="467" height="38"><br>
<img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/85b65ebcc3a6b92275afe719438ec534c428f9d5_2_690x15.png" alt="Screenshot 2025-04-17 at 7.05.12 AM" data-base62-sha1="j4SeeJal7yFe1k4j369yrkGCM5L" width="690" height="15" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/85b65ebcc3a6b92275afe719438ec534c428f9d5_2_690x15.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/85b65ebcc3a6b92275afe719438ec534c428f9d5_2_1035x22.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/5/85b65ebcc3a6b92275afe719438ec534c428f9d5.png 2x" data-dominant-color="2C2C2C"></p>

dunlopj: <p>I got the checkpoint to load correctly. I disabled both calibration functions and then set <code>strict = False</code> when loading the model from the checkpoint. The model hash was created correctly. The issue was calibration features were not saved after the first round of training. Two sets of calibration are automatically performed if their arguments are &gt; 0, but they aren’t necessary when resuming QAT from a checkpoint, for obvious reasons. The model is required to be prepared for quantization once resuming from a checkpoint. Some calibration features are still loaded even though calibration is turned off, but these can be ignored during load, since they are discarded during training.</p>