question: I cannot assign the GPU index by ddp or dp backends..
context: It's weired. I have a single machine with 8 GPUSs and now 0~4 are full load.
  I would like to use parallel training on 5 6 7 GPUs but cannot assign the task to them.
  Some of my codes:
  `    parser = argparse.ArgumentParser(description='Solver')
  parser.add_argument('--config', required=True, type=str)
  fargs = parser.parse_args()
  args = parse_config(fargs.config)
  data = DDPMData(args.Data)
  data.train_dataloader()
  data.test_dataloader()
  
  num_cls = data.n_classes
  shape = data.data_shapes
  args.Model.Unet.kwargs.num_classes = num_cls
  args.Model.DiscreteDiffusion.kwargs.num_class = num_cls
  args.Model.DiscreteDiffusion.kwargs.shape = shape
  model = DDDPM(args)
  
  wandb_logger = WandbLogger()
  
  callbacks = []
  callbacks.append(ModelCheckpoint(save_last=True,every_n_train_steps=600))
  callbacks.append(LearningRateMonitor(logging_interval='step'))
  
  trainer = pl.Trainer(callbacks=callbacks,max_steps=args.max_steps, 
                  accelerator='dp', gpus=[5,6,7],
                  logger = wandb_logger, check_val_every_n_epoch=120,
                  num_sanity_val_steps=0)
  trainer.fit(model, data)
  
  `
  And now nvidia-smi shows:
  +-----------------------------------------------------------------------------+
  | NVIDIA-SMI 470.42.01    Driver Version: 470.42.01    CUDA Version: 11.4     |
  |-------------------------------+----------------------+----------------------+
  | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
  | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
  |                               |                      |               MIG M. |
  |===============================+======================+======================|
  |   0  NVIDIA GeForce ...  On   | 00000000:04:00.0 Off |                  N/A |
  | 63%   72C    P2   151W / 200W |   7997MiB /  8119MiB |     85%      Default |
  |                               |                      |                  N/A |
  +-------------------------------+----------------------+----------------------+
  |   1  NVIDIA GeForce ...  On   | 00000000:06:00.0 Off |                  N/A |
  | 71%   77C    P2   134W / 200W |   8109MiB /  8119MiB |     98%      Default |
  |                               |                      |                  N/A |
  +-------------------------------+----------------------+----------------------+
  |   2  NVIDIA GeForce ...  On   | 00000000:07:00.0 Off |                  N/A |
  | 82%   83C    P2   143W / 200W |   7405MiB /  8119MiB |     89%      Default |
  |                               |                      |                  N/A |
  +-------------------------------+----------------------+----------------------+
  |   3  NVIDIA GeForce ...  On   | 00000000:08:00.0 Off |                  N/A |
  | 80%   83C    P2   158W / 200W |   7263MiB /  8119MiB |     59%      Default |
  |                               |                      |                  N/A |
  +-------------------------------+----------------------+----------------------+
  |   4  NVIDIA GeForce ...  On   | 00000000:0C:00.0 Off |                  N/A |
  | 78%   82C    P2   148W / 200W |   5719MiB /  8119MiB |    100%      Default |
  |                               |                      |                  N/A |
  +-------------------------------+----------------------+----------------------+
  |   5  NVIDIA GeForce ...  On   | 00000000:0D:00.0 Off |                  N/A |
  |  0%   24C    P8     7W / 200W |      4MiB /  8119MiB |      0%      Default |
  |                               |                      |                  N/A |
  +-------------------------------+----------------------+----------------------+
  |   6  NVIDIA GeForce ...  On   | 00000000:0E:00.0 Off |                  N/A |
  |  0%   34C    P8     8W / 200W |      4MiB /  8119MiB |      0%      Default |
  |                               |                      |                  N/A |
  +-------------------------------+----------------------+----------------------+
  |   7  NVIDIA GeForce ...  On   | 00000000:0F:00.0 Off |                  N/A |
  |  0%   25C    P8     7W / 200W |      4MiB /  8119MiB |      0%      Default |
  |                               |                      |                  N/A |
  +-------------------------------+----------------------+----------------------+
  +-----------------------------------------------------------------------------+
  | Processes:                                                                  |
  |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
  |        ID   ID                                                   Usage      |
  |=============================================================================|
  |    0   N/A  N/A      4082      C   python                           7993MiB |
  |    1   N/A  N/A     13619      C   python                           8105MiB |
  |    2   N/A  N/A      4082      C   python                           7401MiB |
  |    3   N/A  N/A      4082      C   python                           7259MiB |
  |    4   N/A  N/A     24206      C   python                           5715MiB |
  +-----------------------------------------------------------------------------+
  and when I run the codes, it will assign task to GPU:2 , I don't know why..
   RuntimeError: CUDA out of memory. Tried to allocate 60.00 MiB (GPU 2; 7.93 GiB total capacity; 117.19 MiB already allocated; 45.50 MiB free; 120.00 MiB reserved in total by PyTorch)
  Any idea..?
  ##############################
  torch = 1.7.1
  pl = 1.3.8
answer: Problemed solved.
  I forgot to map device when I load a ckpt file as data...