Checkpointing model not working at evaluation time

Nitesh: <p>I’m using the technique of saving and loading the models as suggested in the Pytorch docs. I checkpoint the model whenever the current loss is lesser than the previous one. The problem here is that when I evaluate my validation data set, I’m getting the same accuracy with or without using checkpoint to load the model. Ideally the accuracy should vary right?</p>
<p>Here are my code snippets</p>
<pre><code class="lang-auto">def evaluate(dataloader, model, checkpoint=None):
  correct = 0
  tot_box = 0
  if checkpoint:
    model.load_state_dict(checkpoint['state'])
  model.eval()
  with torch.no_grad():
    for X, Y in dataloader:
      X, Y = X.to(device), Y.to(device)
      outputs = model(X)
      for batch in range(Y.shape[0]):
        boxes = decode_to_boxes(Y[batch], outputs[batch], grid_size, input_img_size[0])
        tot_box += boxes.shape[0]
        for box in boxes:
          iou_score = iou(box[0], box[1])
          if iou_score &gt;=0.8:
            correct += 1
  
  accuracy = correct*100/tot_box
  return accuracy
</code></pre>
<pre><code class="lang-auto">checkpoint = {}
def fit(epochs=10, lr=0.0001):
  opt = optim.Adam(model.parameters(), lr)
  min_loss = 10
  for epoch in tqdm(range(epochs), total=epochs, unit="epoch"):
    for X_train, Y_train in train_loader:
      X_train, Y_train = X_train.to(device), Y_train.to(device)
      outputs = model(X_train)
      loss = loss_fn(outputs, Y_train)

      opt.zero_grad()
      loss.backward()
      opt.step()

      if loss.item() &lt; min_loss:
        min_loss = loss.item()
        checkpoint['loss'] = (min_loss, epoch)
        checkpoint['state'] = model.state_dict()

    print('Epoch {}/{} loss {} min_loss {}'.format(epoch+1, epochs, round(loss.item(), 2), round(min_loss, 2)))

model = Network().to(device)
fit(40)
</code></pre>
<pre><code class="lang-auto">print(checkpoint['loss'])
-&gt; (0.16597580909729004, 33)
</code></pre>
<pre><code class="lang-auto">evaluate(val_loader, model)
-&gt; 45.06578947368421
</code></pre>
<pre><code class="lang-auto">evaluate(val_loader, model, checkpoint)
-&gt; 45.06578947368421
</code></pre>
<p>I’m not able to understand why the accuracy is coming same. I hope I’m able to convey my question clearly.</p>

albanD: <p>Hi,</p>
<p>The problem is that when you get the state dict, it does not clone all the Tensors but gives you reference to them.<br>
So when further training updates your model, it also updates the values stored in your checkpoint.<br>
You can do: <code>checkpoint['state'] = {name: t.clone() for name, t in model.state_dict()}</code></p>

Nitesh: <p>Hey thanks, it worked.<br>
By the way, <code>checkpoint['state'] = model.state_dict()</code> this code used to work on my previous projects. In fact Pytorch also suggest the same code in its docs for checkpointing. But this code is not working anymore.<br>
Is this <code>checkpoint['state'] = {name: t.clone() for name, t in model.state_dict()}</code> the new way of checkpointing?</p>

ptrblck: <p>It’s a bit hard to tell, as the plain assignment would work, if you store the checkpoint directly afterwards.<br>
However, since you are temporarily storing the <code>state_dict</code> and are continuing the training, the checkpoint will also be updated, as explained by <a class="mention" href="/u/alband">@albanD</a>.</p>
<p>I <em>think</em> this behavior didn’t change, but in which version was your approach working?</p>