Enable outputs in callbacks
❓ Questions and Help

What is your question?
( Possibly related to #3592 )
I'm in the process to write a custom serializer callback to handle writing results from testing to disk. I understand there are self.write and self.write_dict methods available, but I need to handle some custom logic for serialization and a callback would be the right encapsulation. However, when I override on_test_epoch_end or on_test_end, the 'trainer.logger_connector.callback_metricscontains only the results from the last batch, rather than the list of results from eachtest_step`.

Not sure if this is a bug or if I'm doing something wrong. Could someone please confirm if this is the expected way to access evaluation loop results in callbacks?
I see that each batch result is automatically logged to console with the prefix "DATALOADER :  TEST_RESULTS:". I don't think this is from my user code. Where is this coming from? Also, why is each batch treated as a separate dataloader?

Code
   
I'm loading the model from checkpoint, initializing a trainer with the serializer callback and passing in a test loader.
# Create the serializer to write test results in args.result_dir within the
# parent of the checkpoints dir
out_dir = ckpt_path.parents[1] / args.results_dir
serializer = TestSerializer(out_dir)

# Test the model on the test data and serialize
trainer = Trainer.from_argparse_args(
    args, logger=False, checkpoint_callback=False,
    early_stop_callback=False, callbacks=[serializer]
)

trainer.test(model, test_dataloaders=test_loader)

Here is the test_step in my lightning module, actual forward pass and custom types are irrelevant here:
 def test_step(self, batch: types.DataSplit, _):
        """ Perform a test step """
        preds = self.forward(batch)
        return {"inputs": batch.target, "preds": preds}

And here is my custom serializer so far:
class TestSerializer(Callback):

    """ Serialize test results from the model.

    Results are pickled as well as writted to csv for visualization.

    """
    def __init__(self, out_dir: Path):
        """ Initialize the callback object """
        super().__init__()
        self.__init_output_dir(out_dir)

    def __init_output_dir(self, out_dir: Path):
        """ Check if directory already exists, and create one """
        if out_dir.is_file():
            raise ValueError("Serializer expected directory path, got filepath"
                             " instead: {}".format(str(out_dir)))
        if out_dir.is_dir():
            warnings.warn("Serialization output directory already exists, "
                          "its contents will be overwritten!")
        out_dir.mkdir(parents=True, exist_ok=True)
        self.out_dir = out_dir

    def on_test_end(self, trainer, pl_module):
        """ Serialize test results to self.out_dir """
        print("CALLBACK")
        import pdb; pdb.set_trace()


What have you tried?
Running a test loader with multiple batches and logging the outputs in the debugger reveals that trainer.logger_connector.callback_metrics only has results from the last batch.
What's your environment?
CUDA:

GPU:
Quadro P4000
available: True
version: 10.2

Packages:

numpy: 1.19.1
pyTorch_debug: False
pyTorch_version: 1.6.0
pytorch-lightning: 0.10.0rc1
tqdm: 4.48.2

System:

OS: Linux
architecture:
64bit
ELF
processor: x86_64
python: 3.8.5
version: #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020