question: Multi-GPU Training GPU Usage
context: ❓ Multi-GPU Training GPU Usage
  Before asking:
  
  search the issues.
  search the docs.
  
  
  Hi, I'm using lightning and ddp as backend to do multi-gpu training, with Apex amp (amp_level = 'O1'). The gpu number is 8. I noticed that during training, most of time GPU0's utilization is 0%, while others are almost 100%. But their memory usage are the same. Is this normal? I use OpenPAI and have attached their utilization and memeory usage below. Thanks.
  
  Code
     
  What have you tried?
  What's your environment?
  
  OS: [e.g. iOS, Linux, Win]
  Packaging [e.g. pip, conda]
  Version [e.g. 0.5.2.1]
answer: Your cpu usage seems high. It could be the cpu is the bottleneck here. Try fewer gpus and observe then observe the gpu utilization.