Different values between validation/testing when sets are the same
🐛 Bug
Hi. I've come across a weird bug (?). To test out my workflow, I trained, validated and tested a video classifier on the same set. First bug, I get different accuracy and loss values between my validation and test sets (even though they are the same). Second bug, when loading the model from a check point, the test accuracy is again different. Third issue (maybe not a bug), I notice training/evaluation is slower when running from a checkpoint.
To Reproduce
Here are some code snippets from my project.
Main:
    checkpoint_callback = ModelCheckpoint(monitor='val_acc', save_top_k=1, mode='max')

    # Create PTL trainer
    trainer = Trainer.from_argparse_args(Namespace(**dict(train_config)),
                                         default_root_dir=os.getcwd()
                                                           + '/'
                                                           + config["project"]["models"]
                                                           + experiment_name,
                                         checkpoint_callback=checkpoint_callback)

    # Create model from checkpoint
    if model_config["checkpoint"]["load"]:
        model = globals()[hparams["model"]["name"]].load_from_checkpoint(model_config["checkpoint"]["path"])
        trainer = Trainer(resume_from_checkpoint=model_config["checkpoint"]["path"])
        
        # Evaluate test set from loaded model
        if model_config["test_only"]:
            trainer.test(model)
            exit()

    # Train from scratch
    else:
        model = globals()[hparams["model"]["name"]](Namespace(**dict(hparams)))

    # Train and test with best model
    trainer.fit(model)
    trainer.test(ckpt_path='best')
Validation and test steps from the model:
    def validation_step(self, batch, batch_idx):
        batch, y = batch
        y_hat = self.forward(batch)

        loss = F.cross_entropy(y_hat, y.long())
        labels_hat = torch.argmax(y_hat, dim=1)
        n_correct_pred = torch.sum(y == labels_hat).item()

        return {'val_loss': loss, "n_correct_pred": n_correct_pred, "n_pred": len(y)}

    def validation_epoch_end(self, outputs):
        avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        val_acc = torch.tensor(sum([x['n_correct_pred'] for x in outputs]) / sum(x['n_pred'] for x in outputs))
        tensorboard_logs = {'val_loss': avg_loss, 'val_acc': val_acc, 'step': self.current_epoch}

        return {'log': tensorboard_logs}

    def test_step(self, batch, batch_idx):
        batch, y = batch
        y_hat = self.forward(batch)

        loss = F.cross_entropy(y_hat, y.long())
        labels_hat = torch.argmax(y_hat, dim=1)
        n_correct_pred = torch.sum(y == labels_hat).item()

        return {'test_loss': loss, "n_correct_pred": n_correct_pred, "n_pred": len(y)}

    def test_epoch_end(self, outputs):
        avg_loss = torch.stack([x['test_loss'] for x in outputs]).mean()
        test_acc = sum([x['n_correct_pred'] for x in outputs]) / sum(x['n_pred'] for x in outputs)
        tensorboard_logs = {'test_loss': avg_loss, 'test_acc': test_acc, 'step': self.current_epoch}

        return {'log': tensorboard_logs}

Environment

PyTorch Version: 1.5.1
PyTorch Lightning version: 0.8.5
OS: Linux
How you installed PyTorch : pip
Python version: 3.6.9
CUDA/cuDNN version: 10.2
GPU models and configuration: GeForce GTX 860M

Additional context
I am using 16 bits precision but the same bugs appear when using 32 bits.
Thanks'!