nested_detach of trainer fails in evaluation_loop for labels formatted for YOLOs model through feature extractor.
Author: gauravrajguru
Pull request: no

### System Info

@NielsRogge, @sgugger

nested_detach of trainer fails in evaluation_loop for labels formatted for YOLOs model through feature extractor.

OD label has dict format like 
'labels': {
        'boxes': tensor([[0.4575, 0.5120, 0.6450, 0.2726]]),
        'class_labels': tensor([0]),
        'image_id': tensor([8081]),
        'area': tensor([172346.9688]),
        'iscrowd': tensor([0]),
        'orig_size': tensor([1086,  600]),
        'size': tensor([1332,  736])
    }

nested_detach fails with error dict don't have detach method while taking back up of labels.


### Who can help?

@NielsRogge, @sgugger

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Defined above.

### Expected behavior

Evalution loop should run properly.

Comments:
Thanks for reporting. There is nothing we can do without a code reproducer however. The `nested_detach` function is build to handle dictionaries of tensors, so if I pass it the example you give me above, I don't get an error.

Thanks for reply. I can see latest PR managed this scenario and it's working with new version.

@sgugger @vanpelt 

Facing anotjer issue while using COCO formatted data (please check actual sample above) with transformer trainer in eval_loop.
It is occurring while running the training job on multi gpu (DDP) set up.
However, job succeeded on single gpu (without DDP) set up. Attaching call stack:

File "/opt/conda/envs/ptca/lib/python3.8/site-packages/transformers/trainer.py", line 1501, in train
return inner_training_loop(
File "/opt/conda/envs/ptca/lib/python3.8/site-packages/transformers/trainer.py", line 1841, in _inner_training_loop
self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
File "/opt/conda/envs/ptca/lib/python3.8/site-packages/transformers/trainer.py", line 2089, in _maybe_log_save_evaluate
metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
File "/opt/conda/envs/ptca/lib/python3.8/site-packages/transformers/trainer.py", line 2796, in evaluate
output = eval_loop(
File "/opt/conda/envs/ptca/lib/python3.8/site-packages/transformers/trainer.py", line 2986, in evaluation_loop
labels = self._nested_gather(labels)
File "/opt/conda/envs/ptca/lib/python3.8/site-packages/transformers/trainer.py", line 3112, in _nested_gather
tensors = distributed_concat(tensors)
File "/opt/conda/envs/ptca/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 191, in distributed_concat
return type(tensor)(distributed_concat(t, num_total_examples) for t in tensor)
File "/opt/conda/envs/ptca/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 191, in
return type(tensor)(distributed_concat(t, num_total_examples) for t in tensor)
File "/opt/conda/envs/ptca/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 193, in distributed_concat
output_tensors = [tensor.clone() for _ in range(dist.get_world_size())]
File "/opt/conda/envs/ptca/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 193, in
output_tensors = [tensor.clone() for _ in range(dist.get_world_size())]
AttributeError: 'numpy.ndarray' object has no attribute 'clone'

There is nothing we can do without a small reproducible example.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.