TrainerEvaluationLoopMixin activates model.train() at the end
🐛 Bug
According to the example on fine-tuning, it is important to set the frozen sub-modules to eval mode. This is sensitive because when in training mode, BatchNorm and Dropout change state.
However, at the end of TrainerEvaluationLoopMixin._evaluate there is following code:
# enable train mode again
model.train()
So after the first validation run, the model is again completely in training mode and the freezing is partially undone (for layers like BatchNorm and Dropout).