Llava processor changing padding side of tokenizer has no effect.
Author: laurentd-lunit
Pull request: no
Labels: name: WIP

name: bug

### System Info

When manually setting `processor.tokenizer.padding_side` to a different value than its init value for `LlavaProcessor` (also happens with `LlavaNextProcessor`, `LlavaOnevisionProcessor` and most likely all processor from Llava family), the padding side when using `processor(texts=...)` remains unchanged.

This is due to the following being used in the `__call__` method:
```python
output_kwargs = self._merge_kwargs(
            LlavaProcessorKwargs,
            tokenizer_init_kwargs=self.tokenizer.init_kwargs,
            **kwargs,
        )
[...]
text_inputs = self.tokenizer(prompt_strings, **output_kwargs["text_kwargs"])
```
Since the original padding side is contained in `self.tokenizer.init_kwargs`, it will then be passed to the tokenizer call through `**output_kwargs["text_kwargs"]`. One can go around the issue by forcing calling `processor(texts=texts, padding_side="right")` but I think it would be better that the processor behaves as expected after `processor.tokenizer.padding_side` is changed from its init value.


- `transformers` version: 4.45.0
- Platform: Linux-5.15.0-78-generic-x86_64-with-glibc2.35
- Python version: 3.11.9
- Huggingface_hub version: 0.25.1
- Safetensors version: 0.4.5
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.4.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: No
- GPU type: NVIDIA H100 80GB HBM3

### Who can help?

@zucchini-nlp @LysandreJik 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoProcessor

processor = AutoProcessor.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf")
texts = ["hi there", "hi there longer"]
outputs = processor(text=texts, padding=True)
print(outputs.attention_mask)
processor.tokenizer.padding_side = "right"
outputs = processor(text=texts, padding=True)
print(outputs.attention_mask)
```

### Expected behavior

With the current code we get:
[[0, 1, 1, 1], [1, 1, 1, 1]]
[[0, 1, 1, 1], [1, 1, 1, 1]]

(No changes even after changing padding side of tokenizer).

Should print the following:
[[0, 1, 1, 1], [1, 1, 1, 1]]
[[1, 1, 1, 0], [1, 1, 1, 1]]

After changing padding side to right, we expect to see the 0 corresponding to pad token in attention mask to the right.

Comments:
@laurentd-lunit right! I recently added a feature to accept `padding_side` as a call-time kwarg for tokenizers, so that users can set it at `tokenizer.paddingg_side` and also at `tokenizer(text, padding_side={{}})`

Haven't noticed that it would be breaking in such cases as Llava and other VLMs that are now being standardized, since they always had a dummy default value for `padding_side`. I will make a PR to fix this, thanks for letting know :)