Integration of InstructBLIP with Pipeline in Transformers Library
Author: bxrjmfh
Pull request: no

### System Info

I am curious about the possibility of integrating the InstructBLIP model with the pipeline functionality provided by the Transformers library. After reviewing the documentation related to pipelines, I discovered that the VisualQuestionAnsweringPipeline might be a good fit for this model.

I tried specifying the InstructBLIP model when creating an instance of the VisualQuestionAnsweringPipeline, but I encountered errors both when loading the model from a local directory and when attempting to load it from the Hugging Face Model Hub.

Here are my questions:

Is my approach for loading the InstructBLIP model into the pipeline correct, or am I missing any specific steps?
Is the InstructBLIP model currently compatible with the VisualQuestionAnsweringPipeline, or are there any known limitations or incompatibilities?
I would greatly appreciate any guidance or insights you can provide regarding the integration of InstructBLIP with the pipeline. If there are specific code examples, configuration settings, or additional dependencies required, please let me know.

Thank you in advance for your assistance in clarifying this matter! 

### Who can help?

@Narsil 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I want to load instructBLIP to pipeline, using the following code
```python
import torch
from transformers import pipeline 
from transformers import InstructBlipProcessor, InstructBlipForConditionalGeneration
oracle = pipeline(model='Salesforce/instructblip-vicuna-7b')
image_url = "https://huggingface.co/datasets/Narsil/image_dummy/raw/main/lena.png"
oracle(question="What is she wearing ?", image=image_url)
```
![image](https://github.com/huggingface/transformers/assets/79852164/12a34ba7-d3b1-4f4d-9117-664420f06c9a)

It's showing an internet error, I tried to load from local file.

```python
import torch
from transformers import pipeline 
from transformers import InstructBlipProcessor, InstructBlipForConditionalGeneration

model = InstructBlipForConditionalGeneration.from_pretrained("/root/VLN_2023/temp/instructblip-vicuna-7b",torch_dtype=torch.bfloat16)
processor = InstructBlipProcessor.from_pretrained("/root/VLN_2023/temp/instructblip-vicuna-7b")
device = "cuda:3" if torch.cuda.is_available() else "cpu"
model.to(device)

oracle = pipeline(model=model)
image_url = "https://huggingface.co/datasets/Narsil/image_dummy/raw/main/lena.png"
oracle(question="What is she wearing ?", image=image_url)
```
![image](https://github.com/huggingface/transformers/assets/79852164/d3d359d5-7b3c-4c19-9148-ae191473c900)

It seems that the model architecture is parsed into a string. I still cannot load it.


### Expected behavior

Create a pipeline using the instructBLIP.

Comments:
I found a way to fix this. The trick was to load the image using the [Pillow](https://pypi.org/project/pillow/) (PIL) library.

Here is the full code:

```
from transformers import InstructBlipProcessor, InstructBlipForConditionalGeneration
import torch
from PIL import Image
import requests

model = InstructBlipForConditionalGeneration.from_pretrained("Salesforce/instructblip-vicuna-7b")
processor = InstructBlipProcessor.from_pretrained("Salesforce/instructblip-vicuna-7b")

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)

url = "https://huggingface.co/datasets/Narsil/image_dummy/raw/main/lena.png"
image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
prompt = "What is she wearing?"
inputs = processor(images=image, text=prompt, return_tensors="pt").to(device)

outputs = model.generate(
        **inputs,
        do_sample=False,
        num_beams=5,
        max_length=256,
        min_length=1,
        top_p=0.9,
        repetition_penalty=1.5,
        length_penalty=1.0,
        temperature=1,
)
generated_text = processor.batch_decode(outputs, skip_special_tokens=True)[0].strip()
print(generated_text)
```

The output looks like this:
```
% python3 test.py
Loading checkpoint shards: 100%|█████████████████████████████████████████████| 4/4 [00:43<00:00, 10.82s/it]
qformer_tokenizer/tokenizer_config.json: 100%|████████████████████████████| 343/343 [00:00<00:00, 1.38MB/s]
qformer_tokenizer/vocab.txt: 100%|██████████████████████████████████████| 232k/232k [00:00<00:00, 3.57MB/s]
qformer_tokenizer/tokenizer.json: 100%|█████████████████████████████████| 712k/712k [00:00<00:00, 5.49MB/s]
qformer_tokenizer/added_tokens.json: 100%|██████████████████████████████| 21.0/21.0 [00:00<00:00, 80.7kB/s]
(…)former_tokenizer/special_tokens_map.json: 100%|█████████████████████████| 149/149 [00:00<00:00, 393kB/s]
<warning redacted>
she is wearing a hat
```

Hope this works for you too!

> I found a way to fix this. The trick was to load the image using the [Pillow](https://pypi.org/project/pillow/) (PIL) library.
> 
> Here is the full code:
> 
> ```
> from transformers import InstructBlipProcessor, InstructBlipForConditionalGeneration
> import torch
> from PIL import Image
> import requests
> 
> model = InstructBlipForConditionalGeneration.from_pretrained("Salesforce/instructblip-vicuna-7b")
> processor = InstructBlipProcessor.from_pretrained("Salesforce/instructblip-vicuna-7b")
> 
> device = "cuda" if torch.cuda.is_available() else "cpu"
> model.to(device)
> 
> url = "https://huggingface.co/datasets/Narsil/image_dummy/raw/main/lena.png"
> image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
> prompt = "What is she wearing?"
> inputs = processor(images=image, text=prompt, return_tensors="pt").to(device)
> 
> outputs = model.generate(
>         **inputs,
>         do_sample=False,
>         num_beams=5,
>         max_length=256,
>         min_length=1,
>         top_p=0.9,
>         repetition_penalty=1.5,
>         length_penalty=1.0,
>         temperature=1,
> )
> generated_text = processor.batch_decode(outputs, skip_special_tokens=True)[0].strip()
> print(generated_text)
> ```
> 
> The output looks like this:
> 
> ```
> % python3 test.py
> Loading checkpoint shards: 100%|█████████████████████████████████████████████| 4/4 [00:43<00:00, 10.82s/it]
> qformer_tokenizer/tokenizer_config.json: 100%|████████████████████████████| 343/343 [00:00<00:00, 1.38MB/s]
> qformer_tokenizer/vocab.txt: 100%|██████████████████████████████████████| 232k/232k [00:00<00:00, 3.57MB/s]
> qformer_tokenizer/tokenizer.json: 100%|█████████████████████████████████| 712k/712k [00:00<00:00, 5.49MB/s]
> qformer_tokenizer/added_tokens.json: 100%|██████████████████████████████| 21.0/21.0 [00:00<00:00, 80.7kB/s]
> (…)former_tokenizer/special_tokens_map.json: 100%|█████████████████████████| 149/149 [00:00<00:00, 393kB/s]
> <warning redacted>
> she is wearing a hat
> ```
> 
> Hope this works for you too!

Yes, it is helpful to load a web image. But do you know how to load the local model weights into the pipeline?

I'm addressing this in #29572.

> Yes, it is helpful to load a web image. But do you know how to load the local model weights into the pipeline?

You load the weights like this.

```
model_path = "/Users/jacky/.cache/huggingface/hub/models--Salesforce--instructblip-vicuna-7b/snapshots/ef9d8b3bcb7a0422d7b33a8917e867944312ef22"
model = InstructBlipForConditionalGeneration.from_pretrained(model_path)
processor = InstructBlipProcessor.from_pretrained(model_path)
```

In your case, you might need to add a "." at the beginning of the path to indicate that it's a relative path.

```
"./root/VLN_2023/temp/instructblip-vicuna-7b"
```

> > Yes, it is helpful to load a web image. But do you know how to load the local model weights into the pipeline?
> 
> You load the weights like this.
> 
> ```
> model_path = "/Users/jacky/.cache/huggingface/hub/models--Salesforce--instructblip-vicuna-7b/snapshots/ef9d8b3bcb7a0422d7b33a8917e867944312ef22"
> model = InstructBlipForConditionalGeneration.from_pretrained(model_path)
> processor = InstructBlipProcessor.from_pretrained(model_path)
> ```
> 
> In your case, you might need to add a "." at the beginning of the path to indicate that it's a relative path.
> 
> ```
> "./root/VLN_2023/temp/instructblip-vicuna-7b"
> ```

But, I want to load in pipeline locally. Instead of call the model directly.

> But, I want to load in pipeline locally. Instead of call the model directly.

We can do that in @NielsRogge's branch.

```
git clone git@github.com:NielsRogge/transformers.git
cd transformers
git checkout feature/use_processor
```

Then add this file in `src/` with your modifications.

```
from transformers import pipeline

model_path = "/home/jacky/.cache/huggingface/hub/models--Salesforce--instructblip-vicuna-7b/snapshots/ef9d8b3bcb7a0422d7b33a8917e867944312ef22/"

pipe = pipeline(task="image-text-to-text", model=model_path)

outputs = pipe(
    images="https://huggingface.co/datasets/Narsil/image_dummy/raw/main/lena.png",
    text="What is she wearing?",
    max_new_tokens=30,
)

print(outputs)
```

Of course, when the PR is merged, we don't have to use this workaround anymore.

Hope it works for you :) 

> > But, I want to load in pipeline locally. Instead of call the model directly.
> 
> We can do that in @NielsRogge's branch.
> 
> ```
> git clone git@github.com:NielsRogge/transformers.git
> cd transformers
> git checkout feature/use_processor
> ```
> 
> Then add this file in `src/` with your modifications.
> 
> ```
> from transformers import pipeline
> 
> model_path = "/home/jacky/.cache/huggingface/hub/models--Salesforce--instructblip-vicuna-7b/snapshots/ef9d8b3bcb7a0422d7b33a8917e867944312ef22/"
> 
> pipe = pipeline(task="image-text-to-text", model=model_path)
> 
> outputs = pipe(
>     images="https://huggingface.co/datasets/Narsil/image_dummy/raw/main/lena.png",
>     text="What is she wearing?",
>     max_new_tokens=30,
> )
> 
> print(outputs)
> ```
> 
> Of course, when the PR is merged, we don't have to use this workaround anymore.
> 
> Hope it works for you :)

Thanks for your help!