Repository: huggingface/transformers
Pull request: Fix BatchEncoding.to() for nested elements

# What does this PR do?

Extend `BatchEncoding.to()` to also work for nested elements.

When using voice presets in Bark, the processor returns a `BatchEncoding` of 
- `{ "input_ids": torch.Tensor, "attention_mask": torch.Tensor, "history_prompt": BatchFeature}`

Currently, only tensor elements are moved, so running on `cuda` the following code fails with `RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA__index_select)`:

```python
import scipy
import torch

from transformers import AutoProcessor
from transformers import BarkModel

model = BarkModel.from_pretrained("suno/bark-small")
device = "cuda:0" if torch.cuda.is_available() else "cpu"
model = model.to(device)

sampling_rate = model.generation_config.sample_rate
processor = AutoProcessor.from_pretrained("suno/bark-small")
voice_preset = "v2/en_speaker_6"

# prepare the inputs
text_prompt = "Let's try generating speech, with Bark, a text-to-speech model"
inputs = processor(text_prompt, voice_preset=voice_preset)

# generate speech
speech_output = model.generate(**inputs.to(device))
scipy.io.wavfile.write("bark_out.wav", rate=sampling_rate, data=speech_output[0].cpu().numpy())
```
A workaround was to manually do `inputs["history_prompt"].to(device)`. This PR fixes this by moving all nested elements with a callable `to()`.

Fixes #34634


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case: https://github.com/huggingface/transformers/issues/34634#issuecomment-2996386321
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@Rocketknight1

Changed files:
filename: src/transformers/tokenization_utils_base.py
patch: @@ -801,14 +801,13 @@ def to(self, device: Union[str, "torch.device"], *, non_blocking: bool = False)
             [`BatchEncoding`]: The same instance after modification.
         """
         requires_backends(self, ["torch"])
-        import torch
 
         # This check catches things like APEX blindly calling "to" on all inputs to a module
         # Otherwise it passes the casts down and casts the LongTensor containing the token idxs
         # into a HalfTensor
         if isinstance(device, str) or is_torch_device(device) or isinstance(device, int):
             self.data = {
-                k: v.to(device=device, non_blocking=non_blocking) if isinstance(v, torch.Tensor) else v
+                k: v.to(device=device, non_blocking=non_blocking) if hasattr(v, "to") and callable(v.to) else v
                 for k, v in self.data.items()
             }
         else: