GPU usage decreasing with epochs
I'm having a similar issue to #373 with gradual slowing down while training.
The model works fine but gradually with each epoch GPU utilization is decreasing, and the time needed to complete it increases.

GPU memory usage is also slowly decreasing at the same rate, so I don't think there's a memory leak, and i/o usage is roughly constant at 1200 K/s, well below the sustained speed my HD will support. CPU usage is 20-40%, total CPU memory is <30%, and GPU usage, power and temperature are all in regime where there should be no throttling, so I don't think it's a hardware issue.
Stopping the run, letting things cool down, and restarting with loaded weights doesn't revert speed, and continues with the same slower speed.
I don't have these issues if I replace lightning with a basic training loop.
Has there been any other indication as to what might be causing this?