Emulating multiple devices with a single GPU
Hello,
I have a single GPU, but I would like to spawn multiple replicas on that single GPU and train a model with DDP. Of course, each replica would have to use a smaller batch size in order to fit in memory. (For my use case, I am not interested in having a single replica with a large batch size).
I tried to pass --gpus "0,0" to the Lightning Trainer, and it managed to spawn two processes on the same GPU:
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/2
initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/2

But in the end it crashed with RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:911, invalid usage.
Please, is there any way to split a single GPU into multiple replicas with Lightning?
Thanks!
P.S.: Ray has a really nice support for fractional GPUs: https://docs.ray.io/en/master/using-ray-with-gpus.html#fractional-gpus. I've never used them with Lightning, but maybe it could be a workaround?
For reference: it seems to be possible when the backend is gloo instead of nccl. See discussion here: #8630 (reply in thread).