Pytorch lightning spawns processes after each epoch of training, causing training script to crash unexpectedly
🐛 Bug
When running my training script with PTL, I noticed that after the script is running process spawning is happening at the end of every epoch, which reloads the script. This unintended behavior often causes the training to crash if I'm in the middle of changing the code, and essentially locks the script running from any edits to it.
A possible fix for this is to create the Data Loader object only once here.
To Reproduce
Train with PTL using DDP and num_workers > 1, and place a breakpoint at the top of the main file of your project.