Most efficient way to log data to file within training/validation_step
Q1: Given the data sample paths of a batch, their losses, the batch number, and the epoch number - I would like to log everything into a file during training_step and validation_step. My goal is to identify hard samples using this file.
So I know how to get the data that's given - I would like to know what the most effective way if to log this data in a file while using ddp over multiple gpus (in slurm).
Q2: how can I write whether the current epoch was saved following callbacks.model_checkpoint.ModelCheckpoint to that same file (in validation_epoch_end)? My goal is to identify which epochs/batches were the best performing.
Thank you!