multi-node multi-gpu with time constraint and model checkpointing
Hi,
I am using a multi-node multi-gpu cluster which has a scheduled time limit of 6 hours for the SLURM jobs.
Is there any quick sample code to understand how to use multi-node and multi-gpu and checkpoint and resume after 6 hours automatically?
Any suggestions would be helpful.
Thanks!