BatchEncoding breaks duck-typing, either document or auto-cast to dict
Author: atyshka
Pull request: no

### Feature request

I would like to propose that somewhere in the typical model pipeline, such as the data collators or the Trainer loop, instances of BatchEncoding should be converted to dicts to avoid breaking DataParallel models. Alternatively, I could add to the documentation to state that BatchEncodings should not be passed to PyTorch models. Maintainers, please let me know which solution you would prefer.

### Motivation

I am currently trying to integrate the sentence-transformers library with the nice Trainer API. One of the peculiarities of sentence-transformers is that rather than unpacking the model parameters like so `outputs = model(**inputs)`, the model expects a dict as input, which is internally unpacked. No big deal, I just override the `prediction_step` and remove the unpacking. However, this strangely failed when using a DataParallel setup. I realized this is because the following code in DataParallel could not handle BatchEncodings:

```
def scatter(inputs, target_gpus, dim=0):
    r"""
    Slices tensors into approximately equal chunks and
    distributes them across given GPUs. Duplicates
    references to objects that are not tensors.
    """
    def scatter_map(obj):
        if isinstance(obj, torch.Tensor):
            return Scatter.apply(target_gpus, None, dim, obj)
        if _is_namedtuple(obj):
            return [type(obj)(*args) for args in zip(*map(scatter_map, obj))]
        if isinstance(obj, tuple) and len(obj) > 0:
            return list(zip(*map(scatter_map, obj)))
        if isinstance(obj, list) and len(obj) > 0:
            return [list(i) for i in zip(*map(scatter_map, obj))]
        if isinstance(obj, dict) and len(obj) > 0:
            return [type(obj)(i) for i in zip(*map(scatter_map, obj.items()))]
        return [obj for targets in target_gpus]
```
DataParallel walks like a dict, talks like a dict, prints out exactly like a dict, but is not a subclass of dict, and therefore breaks this code. The tensors are not distributed across GPUs, and errors result. In my opinion, this violates the idea of duck typing, but I realize that technically it's PyTorch who's breaking duck typing here.

Normally, this isn't a problem since Trainer unpacks the model args (BatchEncoding) in its predict_step, but I can't imagine sentence-transformers is the only library that does not adopt this convention.

### Your contribution

I would like to do one of three things:

1. Change DataCollators to output dicts or have Trainer check for BatchEncodings and convert them to dicts
2. Document that BatchEncodings should not be passed to models
3. Submit a PR to PyTorch that modifies this function to check for UserDicts like BatchEncodings

Comments:
This was additionally made harder to debug because the typing for prediction_step indicates a Dict when it is actually a BatchEncoding. Since they print out the same, the only way I identified the cause of the bug was by stepping through and inspecting the types.

```
    def prediction_step(
        self,
        model: nn.Module,
        inputs: Dict[str, Union[torch.Tensor, Any]],
        prediction_loss_only: bool,
        ignore_keys: Optional[List[str]] = None,
    ) -> Tuple[Optional[torch.Tensor], Optional[torch.Tensor], Optional[torch.Tensor]]:
```

1 and 2 are not viable options on our side. We do rely on properties of `BatchEncoding` in our inputs. You can try 3 for PyTorch, the check that is needed is
```
from collections.abc import Mapping

if isinstance(obj, Mapping) and len(obj) > 0:
    return [type(obj)(i) for i in zip(*map(scatter_map, obj.items()))]
```
(which is the one we use in all our tooling), but I don't guarantee they will accept it.

Or you can do the conversion from `BatchEncoding` to `dict` in your own subclass of the Trainer.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.