Option to set 'non_blocking' for to(device) in BatchEncoding and BatchFeature
Author: daniel-bogdoll
Pull request: yes

Option to set 'non_blocking' for to(device) operation in BatchEncoding for performance improvements. Defaults to 'false', thus no behavioral changes.

# What does this PR do?

This minor PR adds the non_blocking option to the to() function.

Previous: def to(self, device: Union[str, "torch.device"]) -> "BatchEncoding":
New: def to(self, device: Union[str, "torch.device"], non_blocking: bool = False) -> "BatchEncoding":

Since non_blocking defaults to 'False', this PR does not introduce behavioral changes. 

I realized, when utilizing Zero Shot Object Detection models, that it was not possible to set this option, leading to sub-optimal performance during inference.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

- vision models: @amyeroberts, @qubvel


Comments:
Thanks @qubvel, sure thing! Which tests would I need to run to make sure modifications in the to() function of BatchFeature get tested?

Just to make sure, I assume you refer to https://github.com/huggingface/transformers/blob/54be2d7ae87e873482b984cc956e165ca4dc0ba3/src/transformers/feature_extraction_utils.py#L206 ?

Yes, I refer to this one, but not sure it's properly tested anywhere, I was able to find only `SequenceFeatureExtractionTestMixin`

Maybe we can do it as simple as 

```python
non_blocking = kwargs.get("non_blocking", False)
...
elif isinstance(v, torch.Tensor) and device is not None:
      new_data[k] = v.to(device=device, non_blocking=non_blocking)
...
```

That's how I would have tried it as well. But what about this block?

```
# Check if the args are a device or a dtype
        if device is None and len(args) > 0:
            # device should be always the first argument
            arg = args[0]
            if is_torch_dtype(arg):
                # The first argument is a dtype
                pass
            elif isinstance(arg, str) or is_torch_device(arg) or isinstance(arg, int):
                device = arg
            else:
                # it's something else
                raise ValueError(f"Attempting to cast a BatchFeature to type {str(arg)}. This is not supported.")
```

Here device is derived from `args` rather than `kwargs`. Should this be extended in some way to also consider deriving non_blocking? Not sure where or how this is used.

>Here device is derived from args rather than kwargs. Should this be extended in some way to also consider deriving non_blocking? Not sure where or how this is used.

I don't think so, maybe at some moment, it is worth refactoring this method for more explicit args and kwargs. For now, we can add a note in docstring that `non_blocking` should be passed as a keyword argument.

@qubvel Done! Thanks for the super-fast replies, was a pleasure! Tests fail now, though:

For the first one, as you stated here (https://github.com/huggingface/transformers/pull/34826#issuecomment-2491562222), it does not seem to be related.

https://app.circleci.com/pipelines/github/huggingface/transformers/111324/workflows/3351b194-4b9e-4a17-876b-85360fc7ff01/jobs/1482124?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-checks-link&utm_content=summary

```
FAILED
tests/models/xlm_roberta_xl/test_modeling_xlm_roberta_xl.py::XLMRobertaXLModelTest::test_assisted_decoding_matches_greedy_search_1_same 
- AssertionError: False is not true
```


As the second one is a timeout issue, it also seems unrelated:

https://app.circleci.com/pipelines/github/huggingface/transformers/111324/workflows/3351b194-4b9e-4a17-876b-85360fc7ff01/jobs/1482127?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-checks-link&utm_content=summary

```
FAILED
tests/models/convbert/test_modeling_convbert.py::ConvBertModelTest::test_pipeline_fill_mask -
requests.exceptions.ReadTimeout: (ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443):
Read timed out. (read timeout=10)"), '(Request ID: 04e3d1b8-11fc-4791-ba74-3d7d67a5f3f2)')
```

@ArthurZucker or @LysandreJik please review when you have bandwidth