Validation and test results not the same for same data

username: rdmpage
cooked: <p>I’m new to the joys of PyTorch and am using someone else’s code, so forgive what feels like a naive question. I am training an image recognition model (based on inceptionv3) based on code in <a href="https://github.com/macaodha/inat_comp_2018" class="inline-onebox" rel="noopener nofollow ugc">GitHub - macaodha/inat_comp_2018: CNN training code for iNaturalist 2018 image classification competition</a>. I’ve only run a few epochs while I get a sense of how the code works.</p>
<p>As the code trains the model it checks it against a validation set of images.Once trailing is complete you can run a test set of images against the model to get predictions.</p>
<p>My expectation was that if I used the validation data set as “test” data I would get the same set of predictions as the last round of training generates (because the “test” and validation data are the same). However the test results generate very poor scores, and suggest values for the identity of the images that do not exist in the training data.</p>
<p>I suspect that the code is not correctly loading the saved model when it does the testing. But can anyone confirm that my assumption (validation results for last training round = test results if test data = validation data) are correct?</p>
<p>To add some detail, this is the code to load the model:</p>
<pre><code class="lang-auto">def build_model_and_optim():
    global device, args, resume
    # load pretrained model
    print("Using pre-trained inception_v3")
    # use this line if instead if you want to train another model
    #model = models.__dict__[args.arch](pretrained=True)
    model = inception_v3(pretrained=True)
    model.fc = nn.Linear(2048, args.num_classes)
    model.aux_logits = False
    model = model.to(device)

    optimizer = SGD(model.parameters(), args.lr,
                    momentum=args.momentum,
                    weight_decay=args.weight_decay)
    # optionally resume from a checkpoint
    if args.resume:
        if os.path.isfile(args.resume):
            print("=&gt; loading checkpoint '{}' for inaturalist-inception".format(
                args.resume))
            checkpoint = torch.load(args.resume)
            args.start_epoch = checkpoint['epoch']
            best_prec3 = checkpoint['best_prec3']
            model.load_state_dict(checkpoint['state_dict'], strict=False) # https://stackoverflow.com/questions/63057468/how-to-ignore-and-initialize-missing-keys-in-state-dict
            optimizer.load_state_dict(checkpoint['optimizer'])
            print("=&gt; loaded checkpoint '{}' (epoch {})".format(
                args.resume, checkpoint['epoch']))
        else:
            print("=&gt; no checkpoint found at '{}'".format(args.resume))

    return model, optimizer
</code></pre>
<p>If training, the following loop is invoked:</p>
<pre><code class="lang-auto">    for epoch in range(args.start_epoch, args.epochs):
        adjust_learning_rate(optimizer, epoch)

        # train for one epoch
        train(train_loader, model, criterion, optimizer, epoch)

        # evaluate on validation set
        if 1:
            prec3, preds, im_ids = validate(val_loader, model, criterion, True)
            with open('predictions-epoch-' + epoch + '.csv', 'w') as opfile:
                opfile.write('id,predicted\n')
                for ii in range(len(im_ids)):
                    opfile.write(str(im_ids[ii]) + ',' + ' '.join(str(x) for x in preds[ii,:])+'\n')            
        else:
            prec3 = validate(val_loader, model, criterion, False)

        # remember best prec@1 and save checkpoint
        is_best = prec3 &gt; best_prec3
        best_prec3 = max(prec3, best_prec3)
        save_checkpoint({
             'epoch': epoch + 1,
             #'arch': args.arch,
             'state_dict': model.state_dict(),
             'best_prec3': best_prec3,
             'optimizer' : optimizer.state_dict(),
        }, is_best)
</code></pre>
<p>This seems to work, the model gets better over time, and the checkpoints are saved. But testing fails (even though validation works).</p>

username: ptrblck
cooked: <p>Using <code>strict=False</code> in:</p>
<pre><code class="lang-python">model.load_state_dict(checkpoint['state_dict'], strict=False)
</code></pre>
<p>is generally not recommended unless you know why it’s used and why mismatches are expected.<br>
Did you check that the returned mismatched keys are expected?</p>
<p>If so, you could check the output of the model before and after storing using a static input (e.g. <code>torch.ones</code>) and see if the results are equal (up to the expected floating point precision noise).</p>
<p>If this test passes you could then check the data loading pipelines between the validation and test runs to see where the difference in the data comes from.</p>

username: rdmpage
cooked: <p>Yes, this was a case of me trying anything to get the code to “work” (having also dealt with out of date libraries and getting tings to work with Apple’s M1. I followed the suggestion you made elsewhere <a href="https://discuss.pytorch.org/t/runtimeerror-error-s-in-loading-state-dict-for-inception3/90273/2" class="inline-onebox">RuntimeError: Error(s) in loading state_dict for Inception3: - #2 by ptrblck</a> at that seems to help with a trivial test case. I’m now running a larger dataset to see what happens. Thank you for your help.</p>

username: rdmpage
cooked: <p>If anyone has the same issue, the two things I did to fix the problem were:</p>
<ol>
<li>
<p>Restore original code <code>model.load_state_dict(checkpoint['state_dict'])</code> as suggested by <a class="mention" href="/u/ptrblck">@ptrblck</a> above.</p>
</li>
<li>
<p>In the epoch loop just before calling <code>save_checkpoint</code> I added:</p>
</li>
</ol>
<pre><code class="lang-auto">try:
    model_state_dict = model.module.state_dict()
except AttributeError:
    model_state_dict = model.state_dict()
</code></pre>
<p>Based on suggestion by <a class="mention" href="/u/alex.veuthey">@alex.veuthey</a> <a href="https://discuss.pytorch.org/t/chexnet-code-error-runtimeerror-error-s-in-loading-state-dict-for-dataparallel-missing-key-s-in-state-dict-module-densenet121-features-conv0-weight/43790/3">here</a></p>