Multi-GPU Training GPU Usage
❓ Multi-GPU Training GPU Usage
Before asking:

search the issues.
search the docs.


Hi, I'm using lightning and ddp as backend to do multi-gpu training, with Apex amp (amp_level = 'O1'). The gpu number is 8. I noticed that during training, most of time GPU0's utilization is 0%, while others are almost 100%. But their memory usage are the same. Is this normal? I use OpenPAI and have attached their utilization and memeory usage below. Thanks.

Code
   
What have you tried?
What's your environment?

OS: [e.g. iOS, Linux, Win]
Packaging [e.g. pip, conda]
Version [e.g. 0.5.2.1]
Your cpu usage seems high. It could be the cpu is the bottleneck here. Try fewer gpus and observe then observe the gpu utilization.