bug with AutoVideoProcessor for VJEPA 2
Author: FrancoisPorcher
Pull request: no
Labels: name: bug

### System Info

### Bug Report

Hi,

The `AutoVideoProcessor` for **VJEPA 2** was working fine, but after upgrading `transformers` to **4.56.1** it stopped working.  

I’m not sure if the issue comes from `accelerate` or from the `AutoVideoProcessor` itself.

---

### Code Snippet

```python
video = self.preprocessor(inputs["video"], return_tensors="pt")["pixel_values_videos"]


Traceback (most recent call last):
  File "/private/home/francoisporcher/.conda/envs/future_latents/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/private/home/francoisporcher/.conda/envs/future_latents/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/private/home/francoisporcher/FutureLatents/src/main.py", line 175, in <module>
    main()
  File "/private/home/francoisporcher/FutureLatents/src/main.py", line 161, in main
    trainer.fit(
  File "/private/home/francoisporcher/FutureLatents/training/trainer.py", line 337, in fit
    train_loss = self.train_epoch(train_loader)
  File "/private/home/francoisporcher/FutureLatents/training/trainer.py", line 200, in train_epoch
    total_loss += self.train_step(batch)
  File "/private/home/francoisporcher/FutureLatents/training/trainer.py", line 123, in train_step
    outputs = self.model(batch, return_norms=self.debug)
  File "/private/home/francoisporcher/.conda/envs/future_latents/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/private/home/francoisporcher/.conda/envs/future_latents/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
  File "/private/home/francoisporcher/.conda/envs/future_latents/lib/python3.10/site-packages/accelerate/utils/operations.py", line 818, in forward
    return model_forward(*args, **kwargs)
  File "/private/home/francoisporcher/.conda/envs/future_latents/lib/python3.10/site-packages/accelerate/utils/operations.py", line 806, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
  File "/private/home/francoisporcher/.conda/envs/future_latents/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 44, in decorate_autocast
    return func(*args, **kwargs)
  File "/private/home/francoisporcher/FutureLatents/models/models.py", line 148, in forward
    latents = self.encode_inputs(batch)  # [B, D, T, H, W]
  File "/private/home/francoisporcher/FutureLatents/models/models.py", line 79, in encode_inputs
    video = self.preprocessor(inputs["video"], return_tensors="pt")["pixel_values_videos"]
  File "/private/home/francoisporcher/.conda/envs/future_latents/lib/python3.10/site-packages/transformers/video_processing_utils.py", line 212, in __call__
    return self.preprocess(videos, **kwargs)
  File "/private/home/francoisporcher/.conda/envs/future_latents/lib/python3.10/site-packages/transformers/video_processing_utils.py", line 378, in preprocess
    videos, video_metadata = self._decode_and_sample_videos(
  File "/private/home/francoisporcher/.conda/envs/future_latents/lib/python3.10/site-packages/transformers/video_processing_utils.py", line 302, in _decode_and_sample_videos
    videos = make_batched_videos(videos)
  File "/private/home/francoisporcher/.conda/envs/future_latents/lib/python3.10/site-packages/transformers/video_utils.py", line 199, in make_batched_videos
    return [np.array(videos)[None, ...]]
  File "/private/home/francoisporcher/.conda/envs/future_latents/lib/python3.10/site-packages/torch/_tensor.py", line 1225, in __array__
    return self.numpy()
TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
video = self.preprocessor(inputs["video"], return_tensors="pt")["pixel_values_videos"]

### Expected behavior

not crashing

Comments:
The issue is due to the newly updated AutoVideoProcessor, specifically the video_utils.py file, not accelerate. Notice this function call:

videos = make_batched_videos(videos) return [np.array(videos)[None, ...]]

When you return np.array(videos), it will try to list the video tensors into an NP array. Now notice the error:

TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first

The problem here is that NP arrays exist in CPU RAM. It cannot handle a PyTorch tensor that is located on the GPU. Since you are using cuda, this will cause the error, because the new AutoVideoProcessor assumes that PyTorch tensors are on the CPU, not on the GPU.

Dont worry though. The code itself is mostly fine, but since you directly use PyTorch tensors on the GPU, it might not work for the new transformers version. This is indeed a bug. 

If you want some help, then just reply. 

cc @zucchini-nlp @qubvel

What is the content of `inputs["video"]` and its shape? The numpy call is nit supposed to reached unless the input is not in correct video format, Then we assume that is might be an image and try to unsqueeze a frame dimension

Only case I can think of is that the input is a single frame (3D torch tensor) for whcih I opened a fix in https://github.com/huggingface/transformers/pull/41218