Checkpoint loading at the test time
🐛 Bug
Hi,
I am having problems with using checkpointing and testing with the use of DDP in multi-GPU setup. I guess that model in each process tries to save its model during the training and then load it at the test time. However, there is just one such model saved and therefore the other processes cannot find their saved model and I get the following error:
FileNotFoundError: [Errno 2] No such file or directory: '/home/vobecant/PhD/system_validation/results/debug_cifar10_multigpu_gradclip/checkpoints/ckpt_epoch=07-val_loss=0.80.ckpt'
Please reproduce using the BoringModel and post here
I don't know how to test multi-GPU behavior with DDP in colab. I post the snippet here:
model = BoringModel()

parser = argparse.ArgumentParser()
parser.add_argument('--gpus_per_node', default=1, type=int)
parser.add_argument('--num_nodes', default=1, type=int)
parser.add_argument('--distributed_backend', default='ddp')
trainer_args = parser.parse_args()

callbacks = []

early_stopping = EarlyStopping('val_loss', patience=args.patience)
setattr(trainer_args, 'early_stopping_callback', early_stopping)
filepath = os.path.join('./tmp', 'ckpt_{epoch:02d}-{val_loss:.2f}')
checkpoint_callback = pl.callbacks.ModelCheckpoint(monitor=args.monitor, filepath=filepath)
setattr(trainer_args, 'checkpoint_callback', checkpoint_callback)
callbacks.append(checkpoint_callback)

lr_monitor = LearningRateMonitor(logging_interval='epoch')
callbacks.append(lr_monitor)
setattr(trainer_args, 'callback', callbacks)
tb_logger = pl_loggers.TensorBoardLogger(args.logs_dir)
setattr(trainer_args, 'logger', tb_logger)
print('Used callbacks: {}'.format(callbacks))
trainer = pl.Trainer.from_argparse_args(trainer_args, callbacks=callbacks,
                                        checkpoint_callback=checkpoint_callback, max_epoch=1)

trainer.fit(model, datamodule=dm)

result = trainer.test() if (args.monitor and args.val_size > 0) else trainer.test(model)
print(result)

trainer.test()

Expected behavior
Test the model with the best model saved during the training.
Environment

CUDA:

GPU:

GeForce GTX 1080 Ti


available:         True
version:           10.1


Packages:

numpy:             1.19.1
pyTorch_debug:     False
pyTorch_version:   1.6.0
pytorch-lightning: 1.0.2
tqdm:              4.50.2


System:

OS:                Linux
architecture:

64bit
ELF


processor:         x86_64
python:            3.8.5
version:           #1 SMP Tue Aug 25 17:23:54 UTC 2020