When resuming, model.train accuracy is virtually 0 but model.eval accuracy is consistent (same for losses)

username: ragnaarok
cooked: <p>Hi,</p>
<p>I have a problem where I have 0 accuracy (and big loss) in training after resuming.<br>
I managed to understand where it’s coming from.<br>
When I resume to evaluate, and do “model.eval()” before processing data, I have a nice accuracy.<br>
But when I resume before training, and do “model.train()”, my accuracy drops to 0.<br>
I have a normal batch size, same between train/eval, I’m processing the same data, the same way, the only difference is the “model.train()” =&gt; really high loss, but with “model.eval()” =&gt; low loss.</p>
<p>Do you know how I could solve it?<br>
Thanks for your answers</p>

username: RaLo4
cooked: <p>could it be that you save / load only the <code>model</code> or <code>model.state_dict</code>?<br>
Because for resuming training you should save and load a <code>Checkpoint</code></p>
<p>If so please refer to <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-a-general-checkpoint-for-inference-and-or-resuming-training" rel="nofollow noopener">this</a> and/or <a href="https://pytorch.org/tutorials/recipes/recipes/saving_and_loading_a_general_checkpoint.html#saving-and-loading-a-general-checkpoint-in-pytorch" rel="nofollow noopener">this</a> tutorial</p>

username: ragnaarok
cooked: <p>Maybe I wasn’t precise enough. It’s not that my accuracy slowly drops to 0, or that I’m wrongly training.<br>
Because in that case, it could be a problem with the optimizer state dict etc. (I checked that possibility)</p>
<p>The problem is that, even without training, without doing any loss.backward() or optimizer.step(), I already have a loss that indicates that my model is garbage when it’s configured with model.train(). But everything is ok when I use model.eval(). For the exact same code, dataloader etc…<br>
It’s like if the train() method was making my model completely useless.</p>
<p>It’s not the training procedure that I don’t even have to execute to already see the big difference in loss between train() and eval(). I’m doing fp32 training. I know that with misconfigured fp16 training you could have bad eval() and good train(), but here it’s the opposite.<br>
Is there a way to reconfigure the model.train() the same way it’s configured when I do model.eval()? like manually re-modifying the batchnorm layers etc… ?</p>

username: RaLo4
cooked: <p>I am sorry I misunderstood your question at first.</p>
<p>But with the information given I can’t say anything more.<br>
I also tried to recreate your problem but couldn’t. Everything works fine on my end.<br>
There is no noticeably large difference in loss between using model.eval() or model.train() after loading.</p>

username: ragnaarok
cooked: <p>That’s also my results for many models.<br>
But I don’t know why this one has this problem, if someone has had this problem, maybe it could help.<br>
It would be hard for me to give a minimal reproducible example, I’m using other libs, maybe it’s because I’m using <a href="https://github.com/CharlesShang/DCNv2" rel="nofollow noopener">deformable conv</a> but no one reported this issue.</p>
<p>I’ll keep looking into it anyway</p>

username: ragnaarok
cooked: <p>Don’t know if that could help in helping me, but if I do the standard training procedure but I use model.eval() instead of model.train(), everything goes as expected. No accuracy drop whatsoever</p>

username: ragnaarok
cooked: <p>One more info, despite the training starting back from 0, it gets close to its saved score really fast. So I guess the convolution layers are fine but It’s just like if I had to retrain the batchnorm from start maybe</p>

username: yairkit
cooked: <p>Hi,<br>
I’m having the exactly same issue.<br>
did you find the reason for this?</p>

username: ragnaarok
cooked: <p>No. You can restart a training with the old weights, some of them are good so the model will progress quickly and you can see then if it happens again, it probably won’t.<br>
I had this problem but I don’t have it anymore</p>