Problem with my best parameter save algorithm

Oguzhan: <p>Hi, I have a structure below:</p>
<pre><code class="lang-auto">if valid_loss &lt;= valid_loss_min:
                a,b,c,d = self.test(testloader)
                print(a,b,c,d)
                self.best = copy.deepcopy(self.state_dict())

                valid_loss_min = valid_loss
</code></pre>
<p>this structure is in my training loop. And I have a test function. When I call self.test in the code above, I expect the highest a,b,c,d (presicion recall f1 accuracy) values I get to be the same as the outputs of the test function I called after the training loop is over. But the values are different. Where is my mistake?</p>
<p>(I do self.load_state_dict(self.best) after my training loop is finished.)</p>

ptrblck: <p>Are you calling <code>model.eval()</code> while computing the test stats?<br>
Also, are you able to get the same values outside of this <code>valid_loss</code> condition just by repeatedly calling into the <code>self.test</code> method?</p>

Oguzhan: <p>Yes, I call model.eval() while computing test. Also I am not able to get the same values outside of this <code>valid_loss</code> condition just by repeatedly calling into the <code>self.test</code> method. When I turn off shuffle for dataloader I get same result.</p>
<p>Here is the computation of a,b,c,d → </p>
<pre><code class="lang-auto">def test(self,testloader):
        test_score_f1 = 0
        test_score_accuracy = 0
        test_score_recall = 0
        test_score_presicion = 0
        self.eval()
        for data, target in testloader:
            with torch.no_grad():
                output = self(data)
            _,output = torch.max(output,1)
            test_score_f1 = test_score_f1 + self.f1(output,target)
            test_score_accuracy = test_score_accuracy + self.accuracy(output,target)
            test_score_recall = test_score_recall + self.recall(output,target)
            test_score_presicion = test_score_presicion + self.presicion(output,target)
        
        test_score_f1 = data.shape[0]*test_score_f1/len(test_dataset)
        test_score_accuracy = data.shape[0]*test_score_accuracy/len(test_dataset)
        test_score_recall = data.shape[0]*test_score_recall/len(test_dataset)
        test_score_presicion = data.shape[0]*test_score_presicion/len(test_dataset)
        return test_score_f1,test_score_accuracy,test_score_recall,test_score_presicion
</code></pre>

ptrblck: <aside class="quote no-group" data-username="Oguzhan" data-post="3" data-topic="167802">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oguzhan/48/34324_2.png" class="avatar"> Oguzhan:</div>
<blockquote>
<p>When I turn off shuffle for dataloader I get same result.</p>
</blockquote>
</aside>
<p>This would indicate a dependency on the actual order or batches, which shouldn’t be the case if <code>model.eval()</code> is working properly. I would recommend to try to narrow down which operation is creating different outputs depending on the order of samples.</p>

Oguzhan: <p>Thanks, I could not understand why model.eval() is not working properly. I was thinking that Norm layer causes the point you mention beacuse there is no other “thing” that could effect but when I turn of norm layer I cross with same problem.</p>