question: Emulating multiple devices with a single GPU
context: Hello,
  I have a single GPU, but I would like to spawn multiple replicas on that single GPU and train a model with DDP. Of course, each replica would have to use a smaller batch size in order to fit in memory. (For my use case, I am not interested in having a single replica with a large batch size).
  I tried to pass --gpus "0,0" to the Lightning Trainer, and it managed to spawn two processes on the same GPU:
  LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
  initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/2
  initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/2
  
  But in the end it crashed with RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:911, invalid usage.
  Please, is there any way to split a single GPU into multiple replicas with Lightning?
  Thanks!
  P.S.: Ray has a really nice support for fractional GPUs: https://docs.ray.io/en/master/using-ray-with-gpus.html#fractional-gpus. I've never used them with Lightning, but maybe it could be a workaround?
answer: For reference: it seems to be possible when the backend is gloo instead of nccl. See discussion here: #8630 (reply in thread).