InstructBlip generate function
Author: Zhiyuan-Fan
Pull request: no
Labels: name: Good Second Issue

### System Info

I encountered the following error when using the generate function of the instructblip model to generate multiple samples in parallel:
```
 /python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 88, in forward       
    hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)                                                                              
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.                                        
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [24,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed.        
../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [24,0,0], thread: [33,0,0] Assertion `srcIndex < srcSelectDimSize` failed.        
../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [24,0,0], thread: [34,0,0] Assertion `srcIndex < srcSelectDimSize` failed. 
```

code is:
```py
from transformers import InstructBlipProcessor, InstructBlipForConditionalGeneration
import torch
from PIL import Image

processor = InstructBlipProcessor.from_pretrained("Salesforce/instructblip-vicuna-7b")

device = "cuda" if torch.cuda.is_available() else "cpu"

image = Image.open('./Confusing-Pictures.jpg').convert("RGB")
prompt = "What is unusual about this image?"
inputs = processor(images=image, text=prompt,return_tensors="pt").to(device)
print(inputs)

model = InstructBlipForConditionalGeneration.from_pretrained("Salesforce/instructblip-vicuna-7b")
model.to(device)
inputs = processor(images=[image,image], text=[prompt,'another prompt'],padding=True,return_tensors="pt").to(device)
outputs = model.generate(
        **inputs,
        do_sample=False,
        num_beams=5,
        max_length=256,
        min_length=1,
        top_p=0.9,
        repetition_penalty=1.5,
        length_penalty=1.0,
        temperature=1,
)

print(outputs)

generated_text = processor.batch_decode(outputs, skip_special_tokens=False)
print(generated_text)

```

### Who can help?

@ArthurZucker @amyeroberts @gante

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```py
from transformers import InstructBlipProcessor, InstructBlipForConditionalGeneration
import torch
from PIL import Image

processor = InstructBlipProcessor.from_pretrained("Salesforce/instructblip-vicuna-7b")

device = "cuda" if torch.cuda.is_available() else "cpu"

image = Image.open('./Confusing-Pictures.jpg').convert("RGB")
prompt = "What is unusual about this image?"
inputs = processor(images=image, text=prompt,return_tensors="pt").to(device)
print(inputs)

model = InstructBlipForConditionalGeneration.from_pretrained("Salesforce/instructblip-vicuna-7b")
model.to(device)
inputs = processor(images=[image,image], text=[prompt,'another prompt'],padding=True,return_tensors="pt").to(device)
outputs = model.generate(
        **inputs,
        do_sample=False,
        num_beams=5,
        max_length=256,
        min_length=1,
        top_p=0.9,
        repetition_penalty=1.5,
        length_penalty=1.0,
        temperature=1,
)

print(outputs)

generated_text = processor.batch_decode(outputs, skip_special_tokens=False)
print(generated_text)
```

### Expected behavior

no error.

Comments:
Hey, the error you encountered is usually from trying to access an embedding with a value that is outside of range. You can move everything to CPU to make sure that the error is from there. I suspect that the `padding = True` is padding with the `[PAD]` token that has index `32000`, but the model.config has `pad_token_id=-1` in the text config and `0` in the `qformer_config`. 
Make sure they match! 

thank you very much ~

@Elvisambition I'm getting the same error right now. Which number did you end up setting all three pad_token_ids to in order to get this to work?

@ArthurZucker I dug into this and your hunch was correct. The default config for InstructBLIP ("Salesforce/instructblip-vicuna-7b") is incorrect. The model.config.text_config.pad_token_id has to be 32000, not 0. Would it be possible to get this fixed?

This error occurs when calling instruct-blip like so over a batch of images (this error doesn't happen when calling instruct-blip over one image):
```python
instruct_blip_processor = InstructBlipProcessor.from_pretrained("Salesforce/instructblip-vicuna-7b")
        instruct_blip_model = InstructBlipForConditionalGeneration.from_pretrained(
           "Salesforce/instructblip-vicuna-7b", torch_dtype=torch.float16).to(device="cuda")
        
with torch.no_grad():
  for batch in tqdm(dataloader):
      prompts = ['What is going on in this image?' for _ in range (batch['img'].shape[0])]
      inputs = instruct_blip_processor(images=batch['img'],
                                       text=prompts,
                                       return_tensors="pt").to(device="cuda",
                                                               dtype=torch.float16)

      outputs = instruct_blip_model.generate(
          **inputs,
          num_beams=5,
          max_new_tokens=256,
          min_length=1,
          top_p=0.9,
          repetition_penalty=1.5,
          length_penalty=1.0,
          temperature=1,
      )
```

Specifically, the error occurs during beam search that's triggered by this call in transformers.generation.utils.py starting on line 1627. This call is triggered from by the self.language_model.generate call in modeling_instructblip.py :
```python
return self.beam_search(
                input_ids,
                beam_scorer,
                logits_processor=logits_processor,
                stopping_criteria=stopping_criteria,
                pad_token_id=generation_config.pad_token_id,
                eos_token_id=generation_config.eos_token_id,
                output_scores=generation_config.output_scores,
                return_dict_in_generate=generation_config.return_dict_in_generate,
                synced_gpus=synced_gpus,
                **model_kwargs,
            )
```
Here, generation_config.pad_token_id has to be 32000, not -1 which it is by default. It seems to inherit the pad_token_id from the model.config.text_config. When this is run over a batch of inputs, some outputs will have 256 tokens, but some responses will have fewer tokens. The [PAD] token is used to pad each output in the batch to 256 tokens, so if the pad token is incorrect, it will error out.



Feel free to open a PR here: https://huggingface.co/Salesforce/instructblip-vicuna-7b/blob/main/config.json. I am not admin but you should ping Niels there! 

Hey,

batched generation hasn't been tested yet with InstructBLIP. Would be great to add a corresponding integration test [here](https://github.com/huggingface/transformers/blob/4e1dee0e8e06c1146d023c43812b88bfe2763329/tests/models/instructblip/test_modeling_instructblip.py#L523) for that. @ArthurZucker could you reopen this issue and perhaps label with "good second issue"?

For me it works fine if you set the padding token IDs properly of the language model (Vicuna-7b in this case):
```
model.config.text_config.pad_token_id = 0
```
however we need to check whether the integration tests are still passing with this update.

Noting here that I was getting:
```OverflowError: out of range integral type conversion attempted```
when using the `generate` and then `batch_decode` of InstructBlip.

On inspection, this was because the model was outputting `-1` tokens (which was what `model.config.text_config.pad_token_id` was set to).

I fixed it with:
`model.config.text_config.pad_token_id = preproc.tokenizer.pad_token_id
`

@ArthurZucker @NielsRogge @danielamassiceti Can I work on this if it's not fixed yet??

Yes please. Batched generation of InstructBLIP still needs to be addressed

This is a problem only with vicuna model, and specifically specific indexing errors with batched inference in modelling_llama.py.

I have tried a bit to diagnose this. Even with making the `model.config.text_config.pad_token_id = 0` or `model.config.text_config.pad_token_id = processor.tokenizer.pad_token_id`, it doesn't work.

For me, the errors are appearing at this line:
```
if inputs_embeds is None:
     inputs_embeds = self.embed_tokens(input_ids)
```
The issue is when a "-1" exists in input_ids. If we revert it to 0, I don't think generation results are correct. So
Sample of input_ids:
```
torch.Size([32, 1])
tensor([[26205],
        [  565],
        [  366],
        [ 1554],
        [ 6635],
        **[   -1]**,
        [ 2318],
        [  902],
        [13736],
        [  683],
        [  393],
        [  411],
        [  472],
        [  322],
        [ 1432],
        [  683],
        [ 2172],
        [  278],
        [  292],
        [  322],
        [    2],
        [    2],
        [  683],
        [  322],
        **[   -1]**,
        [11203],
        [  278],
        [  278],
        [  411],
        [  263],
        [ 4799],
        [ 1361]], device='cuda:0')
```

This -1 in input_ids causes indexing errors here:
`return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)`

Any help is appreciated in looking into this further to get to the root cause.

you should set `model.config.pad_token_id = -1` if you want -1 to be the pad token used by the embedding, but not by the tokenizer. Overall, negative indexing is never a good solution. Would recommend adding tokens and resizing the vocab. If padding_idx is set to -1 there should not be errors

There is no attribute padding_token in the model config.
```
print(model.config.padding_token)
AttributeError: 'InstructBlipConfig' object has no attribute 'padding_token'
```

I can explicitly set it as you said, but it doesn't fix the problem.

Just to show the difference in token ids of model and the tokenizer, it looks like below:
```
print(model.config.text_config.pad_token_id)  --> -1
print(model.config.text_config.eos_token_id)  --> 1
print(processor.tokenizer.pad_token_id)  --> 32000
print(processor.tokenizer.eos_token_id)  --> 2
```

**Would recommend adding tokens and resizing the vocab.** 
-> I can try this, but I don't think, this will solve the problem I just pointed out.

Sorry it's the pad token id. 
If it's vicuna specific I would recommend this. However I don't really understand if the issue is with the results you obtain or with the inputs? 

> Noting here that I was getting: `OverflowError: out of range integral type conversion attempted` when using the `generate` and then `batch_decode` of InstructBlip.
> 
> On inspection, this was because the model was outputting `-1` tokens (which was what `model.config.text_config.pad_token_id` was set to).
> 
> I fixed it with: `model.config.text_config.pad_token_id = preproc.tokenizer.pad_token_id `

This is to me the solution

This line, where the error occurs, is within modeling_llama.py
```
if inputs_embeds is None:
     inputs_embeds = self.embed_tokens(input_ids)
```

So, error occurs during generation. The inputs to the model.generate(**inputs) are fine.

I have tried both solutions:
making the model pad_token_id as 0 or same as tokenizer's pad_token_id. The problem still persists internally during generation. The -1 values that I showed in the first comment are from modeling_llama.py during generation.

I am not getting an overflow error, but similar error to the original issue of this thread, that is an indexing error which looks like below:
```
/opt/conda/conda-bld/pytorch_1699449181202/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [188,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181202/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [188,0,0], thread: [33,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181202/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [188,0,0], thread: [34,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
...
```

I am not sure I understand how a model can generate a negative token? The output of a model is determined by an argmax on the logits, which is always positive. The -1 cannot possibly be generated no?

The pad_token_id is set to -1 in config.yaml. I found the command `model.config.text_config.pad_token_id=0 or model.config.text_config.pad_token_id = preproc.tokenizer.pad_token_id ` don't modify output's padding value(-1). I fixed it with `outputs = torch.where(outputs != -1, outputs, processor.tokenizer.pad_token_id)`

Hi, I've just thrown in vicuna7b:
`model.config.text_config.pad_token_id = 0`
before batch generating the output. It works fine....Don't know whether it affects the correctness of the output or not. I just see the warning of python saying changing model params might lead to significant perf drop 

Update for v4.38:
The above fixes with changing the config no longer seem to work but `model.generate(**batch, pad_token_id=processor.tokenizer.pad_token_id)` works just fine.

This is still an issue in batch processing with vicuna models. None of the above solutions work for me. 
The tokenizer config and model config values are still inconsistent and not same as with flan models.

For vicuna models, these are still:
```
print(processor.tokenizer.pad_token_id)   --> 32000
print(model.config.text_config.pad_token_id)   --> -1

print(processor.tokenizer.eos_token_id)   --> 2   
print(model.config.text_config.eos_token_id)   --> 1
```

As a workaround, I have had to process one image-one prompt with vicuna models !!

cc @zucchini-nlp would you be able to take a look here?

From the discussion it seems like all I can do is add a warning if the generation config has incorrect token ids (linked PR). Also I opened PRs on the hub to change those ids from -1 to 32000 in the config, however not sure if the owners actively monitor the hub...

@gullalc can you share the code you are using and transformers version? Passing pad_token_id into the `generate` should have worked.

This issue now has been fixed thanks to @zucchini-nlp! Corresponding PRs:
* https://huggingface.co/Salesforce/instructblip-vicuna-7b/commit/52ba0cb2c44d96b2fcceed4e84141dc40d2b6a92
* https://huggingface.co/Salesforce/instructblip-vicuna-13b/commit/164da4f70fe98b84235f25744c2dad479b90795b