RuntimeError: All input tensors must be on the same device. Received cuda:2 and cuda:0
🐛 Bug

RuntimeError: All input tensors must be on the same device. Received cuda:2 and cuda:0
When I try to train with 4 GPUs (it happens if the number of GPUS > 1), there is an error raised. When the validation epoch completed and before train continuously, this error happened.
Besides, this error wouldn't happen when I only use 1 GPU to train. I think this is a bug of pl?
Environment

PyTorch Version : 1.7.0
PyTorch-Lightning version: 1.0.7
OS (e.g., Linux): Ubuntu 18.04
How you installed PyTorch (conda, pip, source): pip
Python version: py3.6.2
CUDA/cuDNN version: installed cudatoolkit with version: 10.0.130
GPU models and configuration: 2080Ti 10GB
Any other relevant information:

File "DCK3.py", line 584, in 
trainer.fit(dck, datamodule=dm)
File "/home/nudt302/anaconda3/envs/pltorch17tf14/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 446, in fit
results = self.accelerator_backend.train()
File "/home/nudt302/anaconda3/envs/pltorch17tf14/lib/python3.6/site-packages/pytorch_lightning/accelerators/dp_accelerator.py", line 106, in train
results = self.train_or_test()
File "/home/nudt302/anaconda3/envs/pltorch17tf14/lib/python3.6/site-packages/pytorch_lightning/accelerators/accelerator.py", line 66, in train_or_test
results = self.trainer.train()
File "/home/nudt302/anaconda3/envs/pltorch17tf14/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 495, in train
self.train_loop.run_training_epoch()
File "/home/nudt302/anaconda3/envs/pltorch17tf14/lib/python3.6/site-packages/pytorch_lightning/trainer/training_loop.py", line 589, in run_training_epoch
self.trainer.run_evaluation(test_mode=False)
File "/home/nudt302/anaconda3/envs/pltorch17tf14/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 611, in run_evaluation
eval_loop_results = self.evaluation_loop.log_epoch_metrics(deprecated_eval_results, epoch_logs, test_mode)
File "/home/nudt302/anaconda3/envs/pltorch17tf14/lib/python3.6/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 214, in log_epoch_metrics
test_mode
File "/home/nudt302/anaconda3/envs/pltorch17tf14/lib/python3.6/site-packages/pytorch_lightning/trainer/connectors/logger_connector.py", line 127, in on_evaluation_epoch_end
self._log_on_evaluation_epoch_end_metrics(epoch_logs)
File "/home/nudt302/anaconda3/envs/pltorch17tf14/lib/python3.6/site-packages/pytorch_lightning/trainer/connectors/logger_connector.py", line 195, in _log_on_evaluation_epoch_end_metrics
reduced_epoch_metrics = dl_metrics[0].class.reduce_on_epoch_end(dl_metrics)
File "/home/nudt302/anaconda3/envs/pltorch17tf14/lib/python3.6/site-packages/pytorch_lightning/core/step_result.py", line 469, in reduce_on_epoch_end
recursive_stack(result)
File "/home/nudt302/anaconda3/envs/pltorch17tf14/lib/python3.6/site-packages/pytorch_lightning/core/step_result.py", line 608, in recursive_stack
result[k] = collate_tensors(v)
File "/home/nudt302/anaconda3/envs/pltorch17tf14/lib/python3.6/site-packages/pytorch_lightning/core/step_result.py", line 630, in collate_tensors
return torch.stack(items)
RuntimeError: All input tensors must be on the same device. Received cuda:2 and cuda:3