Repository: Lightning-AI/pytorch-lightning

Issue:
cuda runtime error (101) : invalid device ordinal
CUDA_VISIBLE_DEVICES and ddp are not compatible.

https://github.com/PyTorchLightning/pytorch-lightning/blob/25ee51bc570503f331dceecc610d0eb355e22327/pytorch_lightning/trainer/distrib_data_parallel.py#L504

the pytorch respects the CUDA_VISIBLE_DEVICES.
gpu_idx should be the same as local_rank.

If I set CUDA_VISIBLE_DEVICES to be 1. It will break.


Discussion:
BTW simply set https://github.com/PyTorchLightning/pytorch-lightning/blob/25ee51bc570503f331dceecc610d0eb355e22327/pytorch_lightning/trainer/distrib_data_parallel.py#L509

to be
`gpu_idx = self.local_rank`
fixes my issue now.

And I think #2420 is related too.
I am experiencing this issue, also.
The fix described above seems to fix not only this particular issue for me, but a secondary problem whereby during the "Validation Check" GPU memory would be steadily climb to be significantly higher on the first visible device than any of the others. 
Here's a fix that doesn't require editing Lightning code:

```python
num_gpus = os.environ['CUDA_VISIBLE_DEVICES'].split(',').__len__()
os.environ['CUDA_VISIBLE_DEVICES'] = ','.join(f'{i}' for i in range(num_gpus))
```

If you run this before setting your models it should work.
when this will be fixed in master?
Stuck on the same issue. 

I recently started using PyTorch lightning on the HPC cluster. The node which I am trying has 4 Nvidia V100 GPUs. An imagenet classification code is running on the first 2 GPUs [0,1]. I am trying to run a segmentation code on the next 2 GPUs [2,3]. While trying to run with the configuration given below causes the error.

`    trainer = pl.Trainer(
        default_root_dir=hparams.save_dir,
        auto_select_gpus=True,
        gpus=hparams.gpu,
        logger=[testTube_logger],
        max_epochs=hparams.epochs,
        distributed_backend=hparams.distributed_backend,
        use_amp=False,
        checkpoint_callback=checkpoint_callback,
        fast_dev_run=hparams.debug,
        row_log_interval=hparams.log_iter,
        progress_bar_refresh_rate=0)`

Error Message:

```> THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1587428266983/work/torch/csrc/cuda/Module.cpp line=59 error=101 : invalid device ordinal
Traceback (most recent call last):
  File "./../../lightning_scipts/segmentation/distributed_main.py", line 340, in <module>
    main(args)
  File "./../../lightning_scipts/segmentation/distributed_main.py", line 335, in main
    trainer.fit(model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1095, in fit
    results = self.spawn_ddp_children(model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 482, in spawn_ddp_children
    results = self.ddp_train(local_rank, mp_queue=None, model=model, is_master=True)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 554, in ddp_train
    torch.cuda.set_device(self.root_gpu)
  File "/opt/conda/lib/python3.7/site-packages/torch/cuda/__init__.py", line 245, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: cuda runtime error (101) : invalid device ordinal at /opt/conda/conda-bld/pytorch_1587428266983/work/torch/csrc/cuda/Module.cpp:59
Traceback (most recent call last):
  File "/cluster/kappa/90-days-archive/panettalab/skamat04/Nutrition/code_/lightning_scipts/segmentation/distributed_main.py", line 340, in <module>
    main(args)
  File "/cluster/kappa/90-days-archive/panettalab/skamat04/Nutrition/code_/lightning_scipts/segmentation/distributed_main.py", line 335, in main
    trainer.fit(model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1079, in fit
    self.ddp_train(process_idx=task, mp_queue=None, model=model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 577, in ddp_train
    model = model.configure_ddp(model, device_ids)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/core/lightning.py", line 887, in configure_ddp
    model = LightningDistributedDataParallel(model, device_ids=device_ids, find_unused_parameters=True)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 285, in __init__
    self.broadcast_bucket_size)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 483, in _distributed_broadcast_coalesced
    dist._broadcast_coalesced(self.process_group, tensors, buffer_size)
RuntimeError: Broken pipe

```


Please help!
Thank you!
Hi,

I just updated the pytorch lightning module on the cluster to the latest master. But unfortunately, the error still exists.

```
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1587428266983/work/torch/csrc/cuda/Module.cpp line=59 error=101 : invalid device ordinal
Traceback (most recent call last):
  File "./../../lightning_scipts/segmentation/distributed_main.py", line 340, in <module>
    main(args)
  File "./../../lightning_scipts/segmentation/distributed_main.py", line 335, in main
    trainer.fit(model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1020, in fit
    results = self.accelerator_backend.spawn_ddp_children(model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/accelerator_backends/ddp_backend.py", line 118, in spawn_ddp_children
    results = self.ddp_train(local_rank, mp_queue=None, model=model, is_master=True)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/accelerator_backends/ddp_backend.py", line 193, in ddp_train
    torch.cuda.set_device(self.trainer.root_gpu)
  File "/opt/conda/lib/python3.7/site-packages/torch/cuda/__init__.py", line 245, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: cuda runtime error (101) : invalid device ordinal at /opt/conda/conda-bld/pytorch_1587428266983/work/torch/csrc/cuda/Module.cpp:59
Traceback (most recent call last):
  File "/cluster/kappa/90-days-archive/panettalab/skamat04/Nutrition/code_/lightning_scipts/segmentation/distributed_main.py", line 340, in <module>
    main(args)
  File "/cluster/kappa/90-days-archive/panettalab/skamat04/Nutrition/code_/lightning_scipts/segmentation/distributed_main.py", line 335, in main
    trainer.fit(model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1007, in fit
    self.accelerator_backend.train(model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/accelerator_backends/ddp_backend.py", line 56, in train
    self.ddp_train(process_idx=self.task_idx, mp_queue=None, model=model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/accelerator_backends/ddp_backend.py", line 214, in ddp_train
    model = model.configure_ddp(model, device_ids)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/core/lightning.py", line 888, in configure_ddp
    model = LightningDistributedDataParallel(model, device_ids=device_ids, find_unused_parameters=True)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 285, in __init__
    self.broadcast_bucket_size)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 483, in _distributed_broadcast_coalesced
    dist._broadcast_coalesced(self.process_group, tensors, buffer_size)
RuntimeError: Broken pipe
```

My sbatch file:

```
#!/bin/sh
#SBATCH -N 1
#SBATCH --mem=48G  # memory in Mb
#SBATCH --gres=gpu:v100:2
#SBATCH -t 72:00:00
#SBATCH --job-name="distributedqopramidal_50_imagenet"
#SBATCH -e error_%j_%x.dat   # send stderr to errfile
#SBATCH -o output_%j_%x.dat # send stdout to outfile


module purge
module load singularity/3.1.0
module load cuda/10.2
module load cudnn/7.1
export CUDA_VISIBLE_DEVICES=2,3


MODEL=qicnet
BACKBONE=qoresnet50_v1s
DATASET=unimib
RUNDIR=./../../../Trained_Model_Segmentation/Lightning/"$MODEL"_"$DATASET"_"$BACKBONE"_test/
mkdir -p "$RUNDIR"
cp test_lightning.sh "$RUNDIR"


singularity exec --nv  /cluster/kappa/90-days-archive/panettalab/singularity/avocado_pytorch_lightning_v_1_5_0.sif \
python  ./../../lightning_scipts/segmentation/distributed_main.py \
--model "$MODEL" \
--pretrained-base True \
--pretrained False \
--backbone "$BACKBONE" \
--dataset "$DATASET" \
--gpus 2 \
--distributed-backend 'ddp' \
--save_dir "$RUNDIR"
```

My trainer code:
```

    checkpoint_callback = ModelCheckpoint(filepath=os.path.join(ckpt_dir, '{epoch}-{val_mIOU:.2f}'),
                                                save_top_k=1,
                                                verbose=True,
                                                monitor='val_mIOU',
                                                mode='max',
                                                save_last = True,
                                                prefix='')

    trainer = pl.Trainer(
        default_root_dir=hparams.save_dir,
        auto_select_gpus=True,
        gpus=hparams.gpus,
        logger=[testTube_logger],
        max_epochs=hparams.epochs,
        distributed_backend=hparams.distributed_backend,
        callbacks=[Steps_Per_LR_Scheduler(), # This callback has to come first
                   pl.callbacks.lr_logger.LearningRateLogger()],
        checkpoint_callback=checkpoint_callback,
        fast_dev_run=hparams.debug,
        progress_bar_refresh_rate=0,
        resume_from_checkpoint = hparams.resume,
        amp_level = 'O0' )

```

what gpus flag are you passing to the model? since you set visible devices 2,3 your gpus flag should just be —gpus 2
Sorry for the incomplete comment. I have updated the previous comment to reflect the correct arguments sent. I also tried the specification ```--ntasks-per-node=1 ``` mentioned in https://github.com/horovod/horovod/issues/1586#issuecomment-566075641. However, the error doesn't seem to change.  Is there any particular flag I need to set in the SBATCH file?
There is one more mistake I think, the GPUs in the HPC look like below:

![Screenshot from 2020-08-02 20-31-27](https://user-images.githubusercontent.com/16343884/89136013-71d8bd00-d4ff-11ea-9675-15e527e4bbdc.png)

The PCI_BUS_ID  is wrong for some reason. So when I use use ``` export CUDA_VISIBLE_DEVICES=0,1``` The V100s get initialized [GPU 2,3], and when I use ``` export CUDA_VISIBLE_DEVICES=2,3``` [the first 2 P100 GPU ID 0,1] get initialized.

Interestingly, when I change the above code to 
```
module purge
module load singularity/3.1.0
module load cuda/10.2
module load cudnn/7.1
export SINGULARITY_BINDPATH="/cluster/kappa/90-days-archive/panettalab/skamat04/Nutrition/:/home --pwd home/code/bash/classification/"
export CUDA_VISIBLE_DEVICES=0,1
```
The V100s get initialized, Along with that, the P100s are grabbed as well. So finally, 4 GPUs are considered i.e. GPU 0,1,2,3 . Is this expected? 
 
![Screenshot from 2020-08-02 20-39-16](https://user-images.githubusercontent.com/16343884/89136167-4d311500-d500-11ea-910a-2b18a387f720.png)

Thank you,
Best,
Shreyas


 
Hi,

Any updates on this?

Thank you,
Best,
Shreyas
try master?
> try master?

tried, still not fixed. using CUDA_VISIBLE_DEVICES with ddp if CUDA_VISIBLE_DEVICES not start with 0 fails with
`RuntimeError: cuda runtime error (101) : invalid device ordinal at /opt/conda/conda-bld/pytorch_1595629427478/work/torch/csrc/cuda/Module.cpp:59`
sorry. you should use --gpus=2,3 instead of CUDA_VISIBLE_DEVICES=2,3 with --gpus 2

not setting `CUDA_VISIBLE_DEVICES` and setting only `--gpus` works.
So we should mention the GPU number in the --gpus flag?
> So we should mention the GPU number in the --gpus flag?

yes, exactly. @williamFalcon right?
Hi, 

I tried your suggestion after making the following changes.
Note: I haven't set os.environ['CUDA_VISIBLE_DEVICES'] in the sbatch file as gres allocates specific GPUs for the jobs. So I am using ```os.environ['CUDA_VISIBLE_DEVICES']``` to figure out which GPUs are allocated.

```
        num_gpus = os.environ['CUDA_VISIBLE_DEVICES'].split(',').__len__()
        logger.info("Number of GPUS available {}".format(num_gpus))
        gpus = [int(x) for x in os.environ['CUDA_VISIBLE_DEVICES'].split(',')]

```

The trainer is initialized as 
```
    trainer = pl.Trainer(
        default_root_dir=hparams.save_dir,
        gpus=gpus,
        logger=[testTube_logger, wandb_logger],
        max_epochs=hparams.epochs,
        distributed_backend=hparams.distributed_backend,
        callbacks=[Steps_Per_LR_Scheduler(),  # This callback has to come first
                   pl.callbacks.lr_logger.LearningRateLogger(),
                   Custom_Logging_Callback_with_Metrics(logger, meters)],
        checkpoint_callback=checkpoint_callback,
        fast_dev_run=hparams.debug,
        progress_bar_refresh_rate=0,
        resume_from_checkpoint=hparams.resume,
        amp_level='O0')
```
After doing this I get the following error:

```
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1595629427478/work/torch/csrc/cuda/Module.cpp line=59 error=101 : invalid device ordinal

wandb: Waiting for W&B process to finish, PID 21991
Traceback (most recent call last):
  File "./../../lightning_scipts/encoder/distributed_imagenet.py", line 288, in <module>
    main(args)
  File "./../../lightning_scipts/encoder/distributed_imagenet.py", line 283, in main
    trainer.fit(model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/states.py", line 34, in wrapped_fn
    result = fn(self, *args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1039, in fit
    results = self.accelerator_backend.spawn_ddp_children(model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/accelerators/ddp_backend.py", line 118, in spawn_ddp_children
    results = self.ddp_train(local_rank, mp_queue=None, model=model, is_master=True)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/accelerators/ddp_backend.py", line 197, in ddp_train
    torch.cuda.set_device(self.trainer.root_gpu)
  File "/opt/conda/lib/python3.7/site-packages/torch/cuda/__init__.py", line 281, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: cuda runtime error (101) : invalid device ordinal at /opt/conda/conda-bld/pytorch_1595629427478/work/torch/csrc/cuda/Module.cpp:59
wandb: Program failed with code 1. Press ctrl-c to abort syncing.
wandb: Process crashed early, not syncing files
```

Am I doing something wrong? Been trying for weeks now, but no luck! Any help is appreciated.
I have not fully isolated the issue, but it seems to start with the call to get_all_available_gpus:

https://github.com/PyTorchLightning/pytorch-lightning/blob/4354690e55704f0c1742e5a95411b63c6055ddc7/pytorch_lightning/trainer/distrib_parts.py#L259

in sanitize_gpu_ids here:

https://github.com/PyTorchLightning/pytorch-lightning/blob/4354690e55704f0c1742e5a95411b63c6055ddc7/pytorch_lightning/trainer/distrib_parts.py#L304

What happens is the initial procedure gets the right counts with the correct gpus, etc, but then the spawned procedures get told their gpu to use along with an incorrect list of available gpus. For example, if I say `--gpus 3,4,5`, then the first spawned procedure will get gpus as [3] but available gpus as [0,1,2] in the sanitize procedure. 

If you change get_all_available_gpus to first check for `os.environ.get('CUDA_VISIBLE_DEVICES')`, then there is an issue in ddp_backend:

https://github.com/PyTorchLightning/pytorch-lightning/blob/4354690e55704f0c1742e5a95411b63c6055ddc7/pytorch_lightning/accelerators/ddp_backend.py#L179

where the wrong gpu_idx is set in non-master processes. You could try ameliorating that just below with the `is_master` check:

https://github.com/PyTorchLightning/pytorch-lightning/blob/4354690e55704f0c1742e5a95411b63c6055ddc7/pytorch_lightning/accelerators/ddp_backend.py#L185 

However, at that point, I do not trust that I know enough about what is going on in this system to make this change. It could very well break other dependencies.

That being said, it would be really great if this could get addressed. This works np when using SLURM, but breaks when not.

(@williamFalcon)
try master? we pushed fixes this week
which commit fixes this? i was testing on 0.9.0rc12.
ok, can you share a simple example to reproduce? i can test this now. this AM we merged a ddp fix but it might only affect ddp_cpu. 

I can run this on a 2-8 gpu machine

i dont have time in the next few hrs to make a quick example, but i suspect most anything will fail if you use ddp backend with something like `python ... --gpus 3,4,5`. i'm doing that on dgx and it's saying nah.

Resolving patch:
diff --git a/pytorch_lightning/accelerators/ddp_backend.py b/pytorch_lightning/accelerators/ddp_backend.py
--- a/pytorch_lightning/accelerators/ddp_backend.py
+++ b/pytorch_lightning/accelerators/ddp_backend.py
@@ -86,12 +86,18 @@ def spawn_ddp_children(self, model):
         # use the same python interpreter and actually running
         command = [sys.executable] + command
 
-        # since this script sets the visible devices we replace the gpus flag with a number
-        num_gpus = torch.cuda.device_count()
-
-        if '--gpus' in command:
-            gpu_flag_idx = command.index('--gpus')
-            command[gpu_flag_idx + 1] = f'{num_gpus}'
+        # the visible devices tell us how many GPUs we want to use.
+        # when the trainer script was called the device has already been scoped by the time
+        # code reaches this point. so, to call the scripts, we need to leave cuda visible devices alone
+        # but forward the GPUs selected via environment variables
+        gpu_ids = os.environ.get('CUDA_VISIBLE_DEVICES', '')
+        if len(gpu_ids) == 1:
+            gpu_ids = f'{gpu_ids},'
+
+        num_gpus = max(1, len(gpu_ids.split(',')))
+
+        # set the flag for ddp scripts
+        os.environ['PL_TRAINER_GPUS'] = gpu_ids
 
         os.environ['WORLD_SIZE'] = f'{num_gpus * self.trainer.num_nodes}'
 
diff --git a/pytorch_lightning/trainer/trainer.py b/pytorch_lightning/trainer/trainer.py
--- a/pytorch_lightning/trainer/trainer.py
+++ b/pytorch_lightning/trainer/trainer.py
@@ -498,6 +498,9 @@ def __init__(
         self.accumulate_grad_batches = accumulate_grad_batches
         self.configure_accumulated_gradients(accumulate_grad_batches)
 
+        # override with environment flag
+        gpus = os.environ.get('PL_TRAINER_GPUS', gpus)
+
         # for gpus allow int, string and gpu list
         if auto_select_gpus and isinstance(gpus, int):
             self.gpus = pick_multiple_gpus(gpus)


Test patch: