[vlm] adjust max length for special tokens
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/37222

Instead of raising errors, we adjust the max length manually and log the new length. This way is simpler, than the first idea I had because tokenizer's "overflow_tokens" do not work consistently for slow vs fast tokenizers

TODO: forgot about PG and mistral

For ex if a users sets max length to a small value and sets `truncation=True`, we will crop the input text including image tokens. It doesn't raise any error or warning, up until when inputs are forwarded to model and a mismatch occurs between `placeholder_in_text` and `image_hidden_state.shape`.
The below ex from now on will return all image tokens, even though max length is 20. Also we'll raise warning, so users don't get confused as to why `20` is not the length they got 

```python
       processor = LlavaProcessor.from_pretrained("llava-hf/llava-1.5-7b-hf")

        input_str = self.prepare_text_inputs(batch_size=2)
        image_input = self.prepare_image_inputs(batch_size=2)
        input_str = [f"<image>{sample}" for sample in input_str]
        inputs_truncated = processor(
            text=input_str,
            images=image_input,
            return_tensors="pt",
            truncation=True,
            padding=True,
            max_length=20,
        )


```

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37342). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@qubvel done! Basically we should not be removing placeholder tokens from text, even if it exceeds `max-length`. Otherwise, model forward will fail without clear message about what went wrong

We have it already told to users, but with more VLMs being released and more beginner users trying it the issue is coming back over and over. So let's just adjust it manually in processor code

Thanks, I suppose I got it, the only thing that bothers me - we do not respect `max_length` anymore, and it's becoming a variable. It would also be a breaking change, no? It will increase the max_length for existing pipelines, so maybe better to raise an error

@qubvel I don't think it's breaking. The length is only adjusted when `truncation=True` with a `max_length` set. Before this PR, users either 1) hit an error during forward pass, or 2) had a high enough `max_length` already. In case (1), we now handle it by adjusting length; in case (2), nothing changes since truncation only affects overly long inputs.

The only potentially breaking case is when `truncation=True, max_length=LEN, padding="max_length"`. We’ll pad to a different `max_length` from now on.

However we can't just raise an error on user-defined `max_length`. Ideally, we'd check if the expanded input exceeds `max_length` using `overflowing_tokens`, but that flag is unreliable and calling the tokenizer twice is inefficient. So I opted for logging a warning instead. A few users padding to `max_length` may need to adjust based on warning message. LMK if you have another way without increasing processing time, I would also love to just raise error and let users handle it themselves. Less hidden code, the better for us tbh 


Can we count the number of image tokens IDs for each sample and compare with the expected? smt like `(input_ids.isin(image_token_ids).sum(0) < expected).any()` should be pretty fast I suppose

Oke, let's merge this after https://github.com/huggingface/transformers/pull/37362, otherwise many tests will be failing. Updated code to raise error

@qubvel the blocker PR is merged so this can be reviewed now

EDIT: hmm most tests failing now, probably because the dummy tests didn't have matching number of image tokens before. And now we raise error for all those cases