Significant Amount of Idle Time Not Part of Training or Validation Steps
🐛 Bug
When running in AWS using multi-GPU machines, we begin to notice that time spent between epochs becomes very significant. For our case, we spend 17 seconds without GPU processing between epochs that isn't part of either train or validation time. We don't have other custom functions running, so it isn't anything else beyond PyTorch lightning's other functions. When we go to 8 GPUs, this 17 seconds starts to become very significant (25% of the total time). With 1 GPU, it's not much of a factor. We considered it was model saving, but it also happens if we skip validation on some epochs.
We are using PyTorch Lightning 0.8.5 with ddp and PyTorch 1.5.0. In case it matters, our dataloaders load images each mini-batch from Lustre and aren't limited by bandwidth.