I cannot assign the GPU index by ddp or dp backends..
It's weired. I have a single machine with 8 GPUSs and now 0~4 are full load.
I would like to use parallel training on 5 6 7 GPUs but cannot assign the task to them.
Some of my codes:
`    parser = argparse.ArgumentParser(description='Solver')
parser.add_argument('--config', required=True, type=str)
fargs = parser.parse_args()
args = parse_config(fargs.config)
data = DDPMData(args.Data)
data.train_dataloader()
data.test_dataloader()

num_cls = data.n_classes
shape = data.data_shapes
args.Model.Unet.kwargs.num_classes = num_cls
args.Model.DiscreteDiffusion.kwargs.num_class = num_cls
args.Model.DiscreteDiffusion.kwargs.shape = shape
model = DDDPM(args)

wandb_logger = WandbLogger()

callbacks = []
callbacks.append(ModelCheckpoint(save_last=True,every_n_train_steps=600))
callbacks.append(LearningRateMonitor(logging_interval='step'))

trainer = pl.Trainer(callbacks=callbacks,max_steps=args.max_steps, 
                accelerator='dp', gpus=[5,6,7],
                logger = wandb_logger, check_val_every_n_epoch=120,
                num_sanity_val_steps=0)
trainer.fit(model, data)

`
And now nvidia-smi shows:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.42.01    Driver Version: 470.42.01    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:04:00.0 Off |                  N/A |
| 63%   72C    P2   151W / 200W |   7997MiB /  8119MiB |     85%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  On   | 00000000:06:00.0 Off |                  N/A |
| 71%   77C    P2   134W / 200W |   8109MiB /  8119MiB |     98%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  NVIDIA GeForce ...  On   | 00000000:07:00.0 Off |                  N/A |
| 82%   83C    P2   143W / 200W |   7405MiB /  8119MiB |     89%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  NVIDIA GeForce ...  On   | 00000000:08:00.0 Off |                  N/A |
| 80%   83C    P2   158W / 200W |   7263MiB /  8119MiB |     59%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   4  NVIDIA GeForce ...  On   | 00000000:0C:00.0 Off |                  N/A |
| 78%   82C    P2   148W / 200W |   5719MiB /  8119MiB |    100%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   5  NVIDIA GeForce ...  On   | 00000000:0D:00.0 Off |                  N/A |
|  0%   24C    P8     7W / 200W |      4MiB /  8119MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   6  NVIDIA GeForce ...  On   | 00000000:0E:00.0 Off |                  N/A |
|  0%   34C    P8     8W / 200W |      4MiB /  8119MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   7  NVIDIA GeForce ...  On   | 00000000:0F:00.0 Off |                  N/A |
|  0%   25C    P8     7W / 200W |      4MiB /  8119MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      4082      C   python                           7993MiB |
|    1   N/A  N/A     13619      C   python                           8105MiB |
|    2   N/A  N/A      4082      C   python                           7401MiB |
|    3   N/A  N/A      4082      C   python                           7259MiB |
|    4   N/A  N/A     24206      C   python                           5715MiB |
+-----------------------------------------------------------------------------+
and when I run the codes, it will assign task to GPU:2 , I don't know why..
 RuntimeError: CUDA out of memory. Tried to allocate 60.00 MiB (GPU 2; 7.93 GiB total capacity; 117.19 MiB already allocated; 45.50 MiB free; 120.00 MiB reserved in total by PyTorch)
Any idea..?
##############################
torch = 1.7.1
pl = 1.3.8
Problemed solved.
I forgot to map device when I load a ckpt file as data...