[smolvlm] fix video inference
Author: zucchini-nlp
Pull request: yes
Labels: name: for patch

# What does this PR do?

Fixes https://github.com/huggingface/transformers/pull/39006. The model actually had default values for sampling and thus the flag has to be set to `True` for BC

Added a small tests, we had no video tests and thus the bug went unnoticed

Comments:
Nice! Generation works now, although greedy generation does not match the results from `v4.51.3`. I assume this may be because we are now defaulting to ~~bilinear~~ bicubic in the fast processor, whereas we were using Lanczos resampling before?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39147). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

We still read **all** the video frames [here](https://github.com/huggingface/transformers/blob/f8b88866f552e0eeb21059c90e2c30dba058c8e9/src/transformers/video_utils.py#L402) (shape using the repro above: `(559, 730, 1920, 3)`). I think that was not the case in previous versions.

Shape in previous (v4.51.3)  `read_video_pyav` (was in `image_utils.py`): `(9, 730, 1920, 3)`.

> We still read all the video frames [here](https://github.com/huggingface/transformers/blob/f8b88866f552e0eeb21059c90e2c30dba058c8e9/src/transformers/video_utils.py#L402) (shape using the repro above: (559, 730, 1920, 3)). I think that was not the case in previous versions.

Yes, this is expected though I didn't think about extra RAM usage when we decode the whole video. I will open a subsequent PR, allowing video processors to directly accept url/paths at input and decode/sample videos before preprocessing

Also unrelated to this PR: even when we use `use_fast=False`, we get the following warning from [here](https://github.com/huggingface/transformers/blob/128387757105c7c0b57b519ac2aaff217a20e3f0/src/transformers/models/smolvlm/video_processing_smolvlm.py#L177):

```
            logger.warning_once(
                "You have used fast image processor with LANCZOS resample which not yet supported for torch.Tensor. "
                "BICUBIC resample will be used as an alternative. Please fall back to image processor if you "
                "want full consistency with the original model."
            )
```

I'm not sure if this is actionable, how can the user fall back to using the image processor instead?

> this is expected

Noting that this broke in v4.52. Previously, we [passed the sampling function here](https://github.com/huggingface/transformers/blob/5f4ecf2d9f867a1255131d2461d75793c0cf1db2/src/transformers/processing_utils.py#L1399), and then frames were filtered.

> logger.warning_once(
                "You have used fast image processor with LANCZOS resample which not yet supported for torch.Tensor. "
                "BICUBIC resample will be used as an alternative. Please fall back to image processor if you "
                "want full consistency with the original model."
            )

Does it mean we should use slow `ImageProcessor` instead of fast `VideoProcessor`? Probably a code snippet in the warning should clarify it. 
`processor = *VideoProcessor.from_pretrained` -> `processor = *ImageProcessor.from_pretrained(..., use_fast=False)`

> I'm not sure if this is actionable, how can the user fall back to using the image processor instead?

Yeah, this is one thing that is not really actionable since video have no slow processing option, and we'll be defaulting to using only fast processors slowly. Maybe we can delete that, I remember @yonigozlan wanted to ask smolVLM team about defaulting to bilinear interpolation for models

> Does it mean we should use slow `ImageProcessor` instead of fast `VideoProcessor`? Probably a code snippet in the warning should clarify it.

How? The only solution would be to instantiate the image processor and tokenizer manually, and then build `SmolVLMProcessor` with them. Is there another way? This is probably far from what the user was doing (`AutoProcessor.from_pretrained`, in my case).

> The only solution would be to instantiate the image processor and tokenizer manually, and then build SmolVLMProcessor with them. Is there another way?

+1, i think we either just remove it (including in image processor) or create a non-very-efficient resizing by forcing PIL based `lanczos`. Personally, I prefer the first option as we have only 3 such models (SmolVLM, Flava, Chameleon), though that means model's will not match to their original implementations fully

cc @andimarafioti for info on lanczos vs bicubic.

(This doesn't have to be decided for this PR in my opinion)

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: smolvlm

run-slow: smolvlm

This comment contains run-slow, running the specified jobs: 

models: ['models/smolvlm']
quantizations: [] ...