Expected behavior when 1 rank crashes in distributed training?
What's the expected error handling in distributed training when one rank raises an exception but others run fine? Currently we wrap the train loop in a try/catch and checkpoint if anything fails. This means is if one rank raises an exception but not all, that one rank goes through the checkpointing flow and hangs when we try to broadcast the filename in the model checkpoint callback. This hang obscures the primary error message for why that rank failed
This is also an issue if one rank skips a batch during training but others don't. The implicit synchronization is broken as a result