Text Only input using LlaVa Next
Author: sinngam-khaidem
Pull request: no
Labels: name: bug

name: VLM

### System Info

- `transformers` version: 4.47.1
- Platform: Linux-5.4.0-113-generic-x86_64-with-glibc2.39
- Python version: 3.11.11
- Huggingface_hub version: 0.27.0
- Safetensors version: 0.4.5
- Accelerate version: 1.2.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.5.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: Yes
- GPU type: NVIDIA A100-SXM4-80GB

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Code Snippet
```python
from transformers import LlavaNextProcessor, LlavaNextForConditionalGeneration
import torch

processor = LlavaNextProcessor.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf")

model = LlavaNextForConditionalGeneration.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf", torch_dtype=torch.float16)
model.to("cuda:0")  # Use GPU if available

conversation = [
    {"role": "user", "content": [{"type": "text", "text": "What is the capital of France?"}]},
    {"role": "assistant", "content": [{"type": "text", "text": "The capital of France is Paris."}]},
    {"role": "user", "content": [{"type": "text", "text": "What is the capital of India?"}]},
]

text_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)

prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
inputs = processor(text = prompt, images=None,  return_tensors="pt").to("cuda:0")

output = model.generate(**inputs, max_new_tokens=100)

generated_text = processor.decode(output[0], skip_special_tokens=True)
print(generated_text)
```

2. Error Message
```
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00,  6.30it/s]
Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.
Expanding inputs for image tokens in LLaVa-NeXT should be done in processing. Please add `patch_size` and `vision_feature_select_strategy` to the model's processing config or set directly with `processor.patch_size = {{patch_size}}` and processor.vision_feature_select_strategy = {{vision_feature_select_strategy}}`. Using processors without these attributes in the config is deprecated and will throw an error in v4.50.
Traceback (most recent call last):
  File "/home/gpuuser3/reddit_data_annotation/Llava_play.py", line 20, in <module>
    output = model.generate(**inputs, max_new_tokens=100)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gpuuser3/miniconda3/envs/mmsd_annotation/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/gpuuser3/miniconda3/envs/mmsd_annotation/lib/python3.11/site-packages/transformers/generation/utils.py", line 2252, in generate
    result = self._sample(
             ^^^^^^^^^^^^^
  File "/home/gpuuser3/miniconda3/envs/mmsd_annotation/lib/python3.11/site-packages/transformers/generation/utils.py", line 3251, in _sample
    outputs = self(**model_inputs, return_dict=True)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gpuuser3/miniconda3/envs/mmsd_annotation/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gpuuser3/miniconda3/envs/mmsd_annotation/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gpuuser3/miniconda3/envs/mmsd_annotation/lib/python3.11/site-packages/transformers/models/llava_next/modeling_llava_next.py", line 874, in forward
    inputs_embeds = inputs_embeds.to(image_features.dtype)
                                     ^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'dtype'
```

### Expected behavior

I was trying to generate a response using pure-text input. I was assuming that simply setting images=None in  processor() function call would achieve this. But it gives the mentioned error. Is there something I'm missing here?

Comments:
Will close as it was resolved in v4.48 by https://github.com/huggingface/transformers/pull/34502 🤗