Error occur when using DDP and include_inputs_for_metrics
Author: Vicky-Meng
Pull request: no
Labels: name: bug

### System Info

when in DDP settings, with `args.include_inputs_for_metrics = True`, an error will occur in evaluation step.

Seems that the `inputs_decode` haven't been placed to device before calling `_pad_across_processes`.

https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L2963

I've tried that calling something like `inputs_decode.to(device)` would help.


> Traceback (most recent call last):
File "run_trainer.py", line 152, in <module>
Traceback (most recent call last):
  File "run_trainer.py", line 152, in <module>
    main()
  File "run_trainer.py", line 101, in main
    main()
  File "run_trainer.py", line 101, in main
    train_result = task.train(resume_from_checkpoint=checkpoint)
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1413, in train
    train_result = task.train(resume_from_checkpoint=checkpoint)
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1413, in train
    ignore_keys_for_eval=ignore_keys_for_eval,
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1728, in _inner_training_loop
    ignore_keys_for_eval=ignore_keys_for_eval,
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1728, in _inner_training_loop
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1912, in _maybe_log_save_evaluate
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1912, in _maybe_log_save_evaluate
    metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 2628, in evaluate
    metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 2628, in evaluate
    metric_key_prefix=metric_key_prefix,
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 2826, in evaluation_loop
    metric_key_prefix=metric_key_prefix,
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 2826, in evaluation_loop
    inputs_decode = self._pad_across_processes(inputs_decode)
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 2969, in _pad_across_processes
    inputs_decode = self._pad_across_processes(inputs_decode)
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 2969, in _pad_across_processes
    sizes = self._nested_gather(size).cpu()
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 2947, in _nested_gather
    sizes = self._nested_gather(size).cpu()
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 2947, in _nested_gather
    tensors = distributed_concat(tensors)
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer_pt_utils.py", line 181, in distributed_concat
    tensors = distributed_concat(tensors)
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer_pt_utils.py", line 181, in distributed_concat
    dist.all_gather(output_tensors, tensor)
  File "/usr/local/lib/python3.7/dist-packages/torch/distributed/distributed_c10d.py", line 2003, in all_gather
    dist.all_gather(output_tensors, tensor)
  File "/usr/local/lib/python3.7/dist-packages/torch/distributed/distributed_c10d.py", line 2003, in all_gather
    work = default_pg.allgather([tensor_list], [tensor])
RuntimeError: Tensors must be CUDA and dense
    work = default_pg.allgather([tensor_list], [tensor])
RuntimeError: Tensors must be CUDA and dense
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 127) of binary: /usr/bin/python3
Traceback (most recent call last):
  File "/usr/local/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/distributed/run.py", line 719, in main
    run(args)
  File "/usr/local/lib/python3.7/dist-packages/torch/distributed/run.py", line 713, in run
    )(*cmd_args)
  File "/usr/local/lib/python3.7/dist-packages/torch/distributed/launcher/api.py", line 131, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/usr/local/lib/python3.7/dist-packages/torch/distributed/launcher/api.py", line 261, in launch_agent
    failures=result.failures,
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
run_trainer.py FAILED
------------------------------------------------------------
>


### Who can help?

@sgugger 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

just run script on a DDP settings and make sure `include_inputs_for_metrics = True`

### Expected behavior

RuntimeError: Tensors must be CUDA and dense

Comments:
Thanks for flagging! This should be fixed by the PR mentioned above.