Difference between results during training and testing
🐛 Bug
I noticed that the results achieved by my best checkpoint during training dont match the results of calling the trainer.test() function for the same dataset. Note that my LightningModule redirects the test_step and test_end methods to the validation_step and test_end methods, respectively.
    def test_step(self, batch: list, batch_nb: int, *args, **kwargs) -> dict:
        """ Redirects to validation step. """
        return self.validation_step(batch, batch_nb, *args, **kwargs)

    def test_end(self, outputs: list) -> dict:
        """ Redirects to validation end. """
        return self.validation_end(outputs)
Also, if I run one sample of the dev set at each time without using lightning, I get different results.
To Reproduce
To reproduce the error I decided to write a script that trains the model for 1 epoch and then tests the model using trainer.test() and running the test without Lightining.
Code sample
import torch
from comet.data_loader import add_data_args
from comet.models.estimators import FeedForwardEstimator
from pytorch_lightning import Trainer
from test_tube import HyperOptArgumentParser
from comet.optimizers import add_optimizer_args
from comet.schedulers import add_scheduler_args
from tqdm import tqdm
import numpy as np

parser = HyperOptArgumentParser()
parser.add_argument("--loss", default="mse")
parser.add_argument("--disable_progress_bar", default=False)
parser.add_argument("--nr_frozen_epochs", default=0)
parser.add_argument("--pretrained_model", default="")
parser.add_argument("--encoder_learning_rate",default=1e-05)
parser.add_argument("--dropout",default=0.1)
parser.add_argument("--hidden_sizes", default="3072,1536,768")
parser.add_argument("--activations", default="Tanh")
parser.add_argument("--encoder_model", default="LASER")
parser.add_argument("--pool", default="default",)
parser.add_argument("--layer", default="-1")
parser.add_argument("--scalar_mix_dropout", default=0.1)
parser.add_argument("--scalar_mix_layer_norm", default=True)
parser.add_argument("--monitor", default="pearson")
parser.add_argument("--optimizer", default="Adam")
parser.add_argument("--scheduler", default="constant")
parser.add_argument("--batch_size", default=8)
parser.add_argument("--train_percent_check", default=1.0)
parser = add_optimizer_args(parser, 'Adam')
parser = add_scheduler_args(parser, 'constant')
parser = add_data_args(parser)

hparams = parser.parse_args()
model = FeedForwardEstimator(hparams)

trainer = Trainer(
    max_nb_epochs=1,
    min_nb_epochs=1,
    logger=False, 
    gpus=None,
    show_progress_bar=False
)

trainer.fit(model)
print ("training end.")
print ("Starting testing trial 1:")
trainer.test(model)
print ("Starting testing trial 3:")
trainer.test()

model.eval()
print ("Starting manual testing:")
testset = model._retrieve_dataset(hparams, train=False, val=True, test=False)[0]
model_outputs = []
for sample in tqdm(testset):
    model_input, _ = model.prepare_sample([sample])
    prediction = model.forward(**model_input)
    sample["prediction"] = prediction["score"].data.tolist()[0][0]
    model_outputs.append(sample)

predictions = [sample["prediction"] for sample in model_outputs]
targets = [sample["score"] for sample in model_outputs]
metrics = model._compute_metrics(np.array(predictions), np.array(targets))
print (metrics)
Expected behavior
Results:
"pearson": -0.03711935458763926    -> result produced during training in the end of the epoch.
"pearson": -0.03711942892090772     -> result of calling trainer.test()
"pearson": -0.03711942892090772     -> result of calling trainer.test(model)
"pearson": -0.037119481426416974   -> metrics print in the end of the script
Additional context
When using big transformer models these differences are even bigger.
Questions

Has anyone encountered/had the same issue?
Is there any difference between the way the validation loop runs inside the training loop and outside it?