Correct way to upload saved model weights faster r-cnn

username: SU801T
cooked: <p>I use a pretrained model to train a faster r-cnn, where I set pretrained to true including the backbone:</p>
<pre><code class="lang-auto"># set up model 
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True, pretrained_backbone=True)
num_classes = 2  # 1 class (object) + background

# get number of input features for the classifier
in_features = model.roi_heads.box_predictor.cls_score.in_features

# replace the pre-trained head with a new one
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
</code></pre>
<p>At inference, however, I’m unsure of the correct way of uploading my saved model weights.</p>
<p>There is a slight difference when I initialise my model like this where I set pretrained to false:</p>
<pre><code class="lang-auto"># set up model 
model_test = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=False, pretrained_backbone=False)
num_classes = num_classes  # 1 class (object) + background 

# get number of input features for the classifier
in_features = model_test.roi_heads.box_predictor.cls_score.in_features
# replace the pre-trained head with a new one
model_test.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
model_test.to(device)
cpu_device = torch.device("cpu")

bestmodel = torch.load(bestmodel)
model_test.load_state_dict(bestmodel['state_dict'])

</code></pre>
<p>And like this when I set pretrained to true:</p>
<pre><code class="lang-auto"># set up model 
model_test = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True, pretrained_backbone= True)
</code></pre>
<p>I’m talking a few more object proposals but a difference nonetheless. Any reason why this might be happening?</p>

username: ptrblck
cooked: <p>That’s an interesting observation as I would assume your <code>load_state_dict</code> operation would replace <em>all</em> parameters and buffers, which apparently doesn’t seem to be the case.<br>
Could you write a quick check in comparing all parameters and buffers using both models and check where a mismatch is happening?</p>

username: SU801T
cooked: <p>Thanks for the reply. Yes, I assumed that loading the state_dict would override everything.</p>
<p>Would this be sufficient to check the parameters?</p>
<pre><code class="lang-auto">pytorch_total_parameters = sum(p.numel() for p in model.parameters())

</code></pre>

username: ptrblck
cooked: <p>No, you would have to use something like:</p>
<pre><code class="lang-python">modelA = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True, pretrained_backbone=True)
num_classes = 2  # 1 class (object) + background

# get number of input features for the classifier
in_features = modelA.roi_heads.box_predictor.cls_score.in_features

# replace the pre-trained head with a new one
modelA.roi_heads.box_predictor = torchvision.models.detection.faster_rcnn.FastRCNNPredictor(in_features, num_classes)


model_test = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=False, pretrained_backbone=False)
num_classes = num_classes  # 1 class (object) + background 

# get number of input features for the classifier
in_features = model_test.roi_heads.box_predictor.cls_score.in_features
# replace the pre-trained head with a new one
model_test.roi_heads.box_predictor = torchvision.models.detection.faster_rcnn.FastRCNNPredictor(in_features, num_classes)

model_test.load_state_dict(modelA.state_dict())
# &lt;All keys matched successfully&gt;


for name, param in modelA.named_parameters():
    param_test = model_test.state_dict()[name]
    if not (param==param_test).all():
        print("Mismatch in {}".format(name))

for name, param in modelA.named_buffers():
    param_test = model_test.state_dict()[name]
    if not (param==param_test).all():
        print("Mismatch in {}".format(name))
</code></pre>

username: SU801T
cooked: <p>So I’ve just got back round to this and tested the code you provided. There are no mismatches apparently, as nothing is caught.</p>
<p>So I’m not really sure what is going on here. The only thing that changes with this code is the loading of the model as you have seen.</p>
<p>I have also done inference with simply a pretrained model (no weights uploaded) and a model that is not trained on anything whatsoever. Naturally the results are poor, but nonetheless shows that uploading a saved model produces desirable results.</p>
<p>Could it be that uploading the state dicts does some kind of operation on the previously loaded parameters?</p>

username: ptrblck
cooked: <p>Just to make sure: you did test your actual models using my code snippet and you weren’t able to find any mismatches?</p>
<aside class="quote no-group" data-username="SU801T" data-post="5" data-topic="157169">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/94ad74/48.png" class="avatar"> SU801T:</div>
<blockquote>
<p>Could it be that uploading the state dicts does some kind of operation on the previously loaded parameters?</p>
</blockquote>
</aside>
<p>I’m not sure what “uploading” the <code>state_dict</code> means in this context, but accessing it via <code>model.state_dict()</code> and storing it will not change anything in the model.</p>
<p>If you are still seeing different results I would guess that parts of the model are not initialized as parameters or buffers and thus not shown in the <code>state_dict</code>.<br>
Could you check which layer creates the different outputs in both approaches?</p>

username: SU801T
cooked: <p>So I implemented it like this:</p>
<pre><code class="lang-auto">
    modelA = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True, pretrained_backbone=True)
    num_classes = 2  # 1 class (object) + background

    # get number of input features for the classifier
    in_features = modelA.roi_heads.box_predictor.cls_score.in_features

    # replace the pre-trained head with a new one
    modelA.roi_heads.box_predictor = torchvision.models.detection.faster_rcnn.FastRCNNPredictor(in_features, num_classes)

    modelA.load_state_dict(bestmodel['state_dict'])  #LOAD STATE DICT


    model_test = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=False, pretrained_backbone=False)
    num_classes = num_classes  # 1 class (object) + background 

    # get number of input features for the classifier
    in_features = model_test.roi_heads.box_predictor.cls_score.in_features
    # replace the pre-trained head with a new one
    model_test.roi_heads.box_predictor = torchvision.models.detection.faster_rcnn.FastRCNNPredictor(in_features, num_classes)

    # model_test.load_state_dict(bestmodel['state_dict'])   #LOAD STATE DICT and uncomment for comparison 

    model_test.load_state_dict(modelA.state_dict())
    # &lt;All keys matched successfully&gt;


    for name, param in modelA.named_parameters():
        param_test = model_test.state_dict()[name]
        if not (param==param_test).all():
            print("Mismatch in {}".format(name))

    for name, param in modelA.named_buffers():
        param_test = model_test.state_dict()[name]
        if not (param==param_test).all():
            print("Mismatch in {}".format(name))

</code></pre>
<p>I tried different variations of loading models in above. I get no differences.</p>
<p>I’m certain I save the models correctly and have demonstrated in the first comment show I initialise my models.</p>
<p>How would I check which layers create the different outputs in both approaches?</p>

username: ptrblck
cooked: <p>You could use forward hooks as described e.g. <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a>, store the intermediate results in a <code>dict</code> or <code>list</code> for both models, and compare them using the same model input (and calling <code>model.eval()</code> to avoid random operations from e.g. dropout if these layers are used).</p>