Loading LLaVA-NeXT from AutoProcessor recently started causing errors in the forward function
Author: 7AtAri
Pull request: no
Labels: name: bug

### System Info

container built on 24th of october:
including
  pip install tqdm
  pip install torch 
  pip install torchvision
  pip install transformers
  pip install deepspeed==0.15.2
  pip install accelerate
  pip install wandb
  pip install lightning
  pip install optuna
  pip install ray[tune]
  pip install pyarrow
  pip install nltk

  pip install pandas
  pip install numpy
  pip install matplotlib

  pip install scipy
  pip install scikit-learn

  pip install bitsandbytes
  pip install peft
  pip install pillow
  pip install flash-attn --no-build-isolation

### Who can help?

@zucchini-nlp 
@arthur

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

using a batch size of 8 in a train_collate function with the AutoProcessor as well as LLaVA-Next Processor:

    processor = AutoProcessor.from_pretrained(cfg.MODEL_ID_LLAVA_NEXT) # this used to work 
    # processor = LlavaNextProcessor.from_pretrained(cfg.MODEL_ID_LLAVA_NEXT)  # same result if I try this instead
     
    
    def train_collate_fn(examples, processor):
      processor.tokenizer.padding_side = "right"  # during training pad on the right side
      # collect images, texts and true coordinates of the examples batch
      images = [example[0] for example in examples]
      #print("image", images[0])
      texts = [example[1][0]["value"] for example in examples]
  
      batch = processor(images=images,                # images to be processed
                        text=texts,                   # text to be tokenized
                        padding=True,                 # pad the sequences if not long enough
                        truncation=True,              # truncate sequences longer than max_length
                        max_length=cfg.MAX_LENGTH,    # set the maximum output token length
                        return_tensors="pt")          # return a pytorch tensor
      return batch["input_ids"], batch["attention_mask"], batch['pixel_values'], batch["image_sizes"]


in the forward pass:
        # the forward pass
        outputs = self.base_model(
            input_ids=input_ids, 
            attention_mask=attention_mask, 
            pixel_values=pixel_values,
            image_sizes = image_sizes,
            #vision_feature_layer =-1,
            #vision_feature_select_strategy = "full", 
            output_hidden_states = True,  # all hidden states should be output
            return_dict = True  # to get a dict output that can be accessed with the . operator
        )
        
        
        all of this worked two weeks ago. But now I get this error:
        
return self.model.forward(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/accelerate/hooks.py", line 170, in new_forward
output = module._old_forward(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/transformers/models/llava_next/modeling_llava_next.py", line 873, in forward
inputs_embeds, attention_mask, position_ids, labels, _ = self._merge_input_ids_with_image_features(
File "/opt/conda/lib/python3.10/site-packages/transformers/models/llava_next/modeling_llava_next.py", line 551, in _merge_input_ids_with_image_features
raise ValueError(
ValueError: Number of image tokens in input_ids (2040) different from num_images (8).

### Expected behavior

did not throw this error before

Comments:
@7AtAri hey which version of transformers you are using? Can you try to update to the latest v4.46 version with `pip install transformers==4.46` as it works for me in the latest version

with transformers==4.46 the same error persists

@7AtAri it works for me with transformer 4.46 with the code from demo in the hub. If you are using jupyter notebook, make sure you restart the kernel and that the transformers version being imported is indeed `v4.46`. If the error persists, share your env with `transformers-cli env` please

The code should never go in the path with `_merge_input_ids_with_image_features` tbh

```python
from transformers import LlavaNextProcessor, LlavaNextForConditionalGeneration
import torch
from PIL import Image
import requests

processor = LlavaNextProcessor.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf")

model = LlavaNextForConditionalGeneration.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf", torch_dtype=torch.float16, low_cpu_mem_usage=True, device_map="cuda:0") 

# prepare image and text prompt, using the appropriate prompt template
url = "https://www.ilankelman.org/stopsigns/australia.jpg"
image = Image.open(requests.get(url, stream=True).raw)

# Define a chat history and use `apply_chat_template` to get correctly formatted prompt
# Each value in "content" has to be a list of dicts with types ("text", "image") 
conversation = [
    {

      "role": "user",
      "content": [
          {"type": "text", "text": "What is shown in this image?"},
          {"type": "image"},
        ],
    },
]
prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)

inputs = processor(images=image, text=prompt, return_tensors="pt").to("cuda:0")

# autoregressively complete prompt
output = model.generate(**inputs, max_new_tokens=100)

print(processor.decode(output[0], skip_special_tokens=True))

```

GitPython==3.1.43
Jinja2==3.1.2
Mako==1.3.6
MarkupSafe==2.1.1
Pillow==10.0.1
PySocks==1.7.1
PyYAML==6.0
Pygments==2.15.1
SQLAlchemy==2.0.36
accelerate==1.1.1
aiohappyeyeballs==2.4.3
aiohttp==3.11.7
aiosignal==1.3.1
alembic==1.14.0
annotated-types==0.7.0
asttokens==2.0.5
astunparse==1.6.3
async-timeout==5.0.1
attrs==23.1.0
backcall==0.2.0
beautifulsoup4==4.12.2
bitsandbytes==0.44.1
boltons==23.0.0
brotlipy==0.7.0
certifi==2023.7.22
cffi==1.15.1
chardet==4.0.0
charset-normalizer==2.0.4
click==8.0.4
colorlog==6.9.0
conda==23.9.0
conda-build==3.27.0
conda-content-trust==0.2.0
conda_index==0.3.0
conda-libmamba-solver==23.7.0
conda-package-handling==2.2.0
conda_package_streaming==0.9.0
contourpy==1.3.1
cryptography==41.0.3
cycler==0.12.1
decorator==5.1.1
deepspeed==0.15.2
dnspython==2.4.2
docker-pycreds==0.4.0
einops==0.8.0
exceptiongroup==1.0.4
executing==0.8.3
expecttest==0.1.6
filelock==3.9.0
flash-attn==2.7.0.post2
fonttools==4.55.0
frozenlist==1.5.0
fsspec==2023.9.2
gitdb==4.0.11
gmpy2==2.1.2
greenlet==3.1.1
hjson==3.1.0
huggingface-hub==0.26.2
hypothesis==6.87.2
idna==3.4
ipython==8.15.0
jedi==0.18.1
joblib==1.4.2
jsonpatch==1.32
jsonpointer==2.1
jsonschema==4.23.0
jsonschema-specifications==2024.10.1
kiwisolver==1.4.7
libarchive-c==2.9
libmambapy==1.4.1
lightning==2.4.0
lightning-utilities==0.11.9
matplotlib==3.9.2
matplotlib-inline==0.1.6
mkl-fft==1.3.8
mkl-random==1.2.4
mkl-service==2.4.0
more-itertools==8.12.0
mpmath==1.3.0
msgpack==1.1.0
multidict==6.1.0
networkx==3.1
ninja==1.11.1.2
nltk==3.9.1
numpy==1.26.0
nvidia-ml-py==12.560.30
optuna==4.1.0
packaging==23.1
pandas==2.2.3
parso==0.8.3
peft==0.13.2
pexpect==4.8.0
pickleshare==0.7.5
pip==23.2.1
pkginfo==1.9.6
platformdirs==4.3.6
pluggy==1.0.0
prompt-toolkit==3.0.36
propcache==0.2.0
protobuf==5.28.3
psutil==5.9.0
ptyprocess==0.7.0
pure-eval==0.2.2
pyOpenSSL==23.2.0
py-cpuinfo==9.0.0
pyarrow==18.0.0
pycosat==0.6.6
pycparser==2.21
pydantic==2.10.1
pydantic_core==2.27.1
pyparsing==3.2.0
python-dateutil==2.9.0.post0
python-etcd==0.4.5
pytorch-lightning==2.4.0
pytz==2023.3.post1
ray==2.39.0
referencing==0.35.1
regex==2024.11.6
requests==2.31.0
rpds-py==0.21.0
ruamel.yaml==0.17.21
ruamel.yaml.clib==0.2.6
safetensors==0.4.5
scikit-learn==1.5.2
scipy==1.14.1
sentry-sdk==2.19.0
setproctitle==1.3.4
setuptools==68.0.0
six==1.16.0
smmap==5.0.1
sortedcontainers==2.4.0
soupsieve==2.5
stack-data==0.2.0
sympy==1.11.1
sympy==1.12
tensorboardX==2.6.2.2
threadpoolctl==3.5.0
tokenizers==0.20.3
tomli==2.0.1
toolz==0.12.0
torch==2.1.0
torchaudio==2.1.0
torchelastic==0.2.2
torchmetrics==1.6.0
torchvision==0.16.0
tqdm==4.65.0
traitlets==5.7.1
transformers==4.46.0
triton==2.1.0
truststore==0.8.0
types-dataclasses==0.6.6
typing_extensions==4.12.2
typing_extensions==4.8.0
tzdata==2024.2
urllib3==1.26.16
wandb==0.18.7
wcwidth==0.2.5
wheel==0.41.2
yarl==1.18.0
zstandard==0.19.0

is the requirements.txt of my environment saved automatically from wandb 
python version is 3.10.13

is this sufficient env information?

I adapted 
https://github.com/NielsRogge/Transformers-Tutorials/blob/master/LLaVa-NeXT/Fine_tune_LLaVaNeXT_on_a_custom_dataset_(with_PyTorch_Lightning).ipynb

to my usecase. And in my lightning function I do it like this (similar to the tutorial):
        outputs = self.model(input_ids=input_ids,
                            attention_mask=attention_mask,
                            pixel_values=pixel_values,
                            image_sizes=image_sizes,
                          )
                          
The error is then thrown in forward...

                    

@7AtAri hmm, I also tried to reproduce the error on colab but I couldn't. The demo notebook is [here](https://colab.research.google.com/drive/1-iyM376q-GSYuC3KJrKEVvkewZy_0awG?usp=sharing)

Therefore I think there is something wrong in your environment setup. Btw, are you using 'cuda' or 'mps'? There were some cases when 'mps' would give weird slicing errors which can't be reproduced with cuda. Apart from that I can suggest to try and create a new venv/conda and run the script again, if it continues to fail 

no its cuda. And there can not be anything wrong on my side. it is containerized code that was running before...

@7AtAri i see but the issue is that it is not reproducible for me currently. I see it is `forward` run for you, but `generate` calls forward internally and it still works if you modify the colab demo to forward. Can you try to print the `model.config` after it is loaded and for all inputs that go to `model.generate()` print `(input.input_ids == model.config.image_token_index).sum()`?

Seems like the model is not able to induce that text is already expanded and can be that the model config is not updated (for some reason) or something is wrong with particular input images you are using 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.