How can I disable legacy processing in llava-next
Author: foreverpiano
Pull request: no
Labels: name: bug

### System Info

4.47.1

### Who can help?

vision models: @amyeroberts, @qubvel

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

https://github.com/huggingface/transformers/blob/5cabc75b4bdb2e67935f7195f901afd150746eb3/src/transformers/models/llava_next/modeling_llava_next.py#L866-L916

### Sample script

```
def main():
    args = parse_args()
    
    processor = LlavaNextProcessor.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf")
    model = LlavaNextForConditionalGeneration.from_pretrained(
        "llava-hf/llava-v1.6-mistral-7b-hf",
        torch_dtype=torch.float16,
        low_cpu_mem_usage=True,
        attn_implementation="flash_attention_2",
    ).to("cuda:0")
    
    setup_model_with_compression(model, args)
    
    url = "https://github.com/haotian-liu/LLaVA/blob/1a91fc274d7c35a9b50b3cb29c4247ae5837ce39/images/llava_v1_5_radar.jpg?raw=true"
    image = Image.open(requests.get(url, stream=True).raw)
    
    conversation = [
        {
            "role": "user",
            "content": [
                {"type": "image"},
                {"type": "text", "text": "What is shown in this image?"},
            ],
        },
    ]
    
    prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
    inputs = processor(image, prompt, return_tensors="pt").to("cuda:0")
    
    output = model.generate(**inputs, max_new_tokens=100)
    print(processor.decode(output[0], skip_special_tokens=True))

```

### Expected behavior

how does the legacy processing work? can I disable it ? 

Comments:
cc @zucchini-nlp as well

@foreverpiano the legacy processing is disabled at this moment with the official checkpoints. There is one false warning emitted though which will be removed in the next release

If you want to update your own llava checkpoint for the new processing logic, you can do so with
```python
processor.vision_feature_select_strategy = model.config.vision_feature_select_strategy
processor.patch_size = model.config.vision_config.patch_size
model.config.image_seq_length = {{MIN_SEQ_LENGTH_ONE_IMAGE_TAKES_FOR_YOUR_ARCHITECTURE}}
```


@zucchini-nlp  Thanks for your reply. But when I use the above script, it actually runs the legacy processing. Is it about the transformer version?

pip info: 

```
torch==2.4.0
torchvision==0.19.0
numpy==1.26.0
transformers==4.47.1
datasets==2.15.0
accelerate==1.2.1
urllib3==2.1.0
bitsandbytes==0.41.2
chardet==5.2.0
scipy==1.11.4
deepdiff==6.7.1
diffusers==0.8.0
sentencepiece==0.1.99
protobuf==4.25.1

jieba==0.42.1
fuzzywuzzy==0.18.0
rouge==1.0.1
IPython==8.18.1

# tools for installing flash-attn
packaging==24.1
ninja
triton==3.0.0
pytest==8.2.1
flash_attn

matplotlib
```

@foreverpiano yes, there is once a warning emitted and that will be fixed when the legacy processing is removed in https://github.com/huggingface/transformers/pull/34502. But it should not mess up much with the generation quality



```python
def main():
    args = parse_args()
    
    processor = LlavaNextProcessor.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf")
    model = LlavaNextForConditionalGeneration.from_pretrained(
        "llava-hf/llava-v1.6-mistral-7b-hf",
        torch_dtype=torch.float16,
        low_cpu_mem_usage=True,
        attn_implementation="flash_attention_2",
    ).to("cuda:0")
    

    processor.vision_feature_select_strategy = model.config.vision_feature_select_strategy # 
    processor.patch_size = model.config.vision_config.patch_size # 

    setup_model_with_compression(model, args)
    
    url = "https://github.com/haotian-liu/LLaVA/blob/1a91fc274d7c35a9b50b3cb29c4247ae5837ce39/images/llava_v1_5_radar.jpg?raw=true"
    image = Image.open(requests.get(url, stream=True).raw)
    
    conversation = [
        {
            "role": "user",
            "content": [
                {"type": "image"},
                {"type": "text", "text": "What is shown in this image?"},
            ],
        },
    ]
    
    prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
    inputs = processor(image, prompt, return_tensors="pt").to("cuda:0")
    
    output = model.generate(**inputs, max_new_tokens=100)
    print(processor.decode(output[0], skip_special_tokens=True))

```

Hi, I add this patch_size in my script, but it still goes into legacy processing. I don't know why we need legacy processing.

@zucchini-nlp  thanks

Here is the log. It went into legacy processing many times.
![图片](https://github.com/user-attachments/assets/b1c2afad-5b41-4165-95cd-0153ec74e8be)


We just released transformers v4.48 last week, and the new release has all the legacy code removed. Can you try updating with `pip install transformers==4.48` ?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Closing since I think this is no longer relevant as of `4.48`