Add InstructBLIP
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds [InstructBLIP](https://github.com/salesforce/LAVIS/tree/main/projects/instructblip), a visual instruction tuned version of [BLIP-2](https://huggingface.co/docs/transformers/main/model_doc/blip-2).

It's a bit like an open-source multimodal GPT-4, leveraging Flan-T5 and Vicuna pre-trained checkpoints.

Basic usage is as follows:
```
from transformers import InstructBlipProcessor, InstructBlipForConditionalGeneration
import torch
from PIL import Image
import requests

model = InstructBlipForConditionalGeneration.from_pretrained("...")
processor = InstructBlipProcessor.from_pretrained("...")

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)

url = "https://raw.githubusercontent.com/salesforce/LAVIS/main/docs/_static/Confusing-Pictures.jpg"
image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
prompt = "What is unusual about this image?"
inputs = processor(images=image, text=prompt, return_tensors="pt")

outputs = model.generate(
        **inputs,
        do_sample=False,
        num_beams=1,
        max_length=256,
        min_length=1,
        top_p=0.9,
        repetition_penalty=1.5,
        length_penalty=1.0,
        temperature=1,
)
generated_text = processor.batch_decode(outputs, skip_special_tokens=True)[0].strip()
print(generated_text)
```

To do:

- [x] discuss whether to integrate the `QFormerTokenizer` into the processor
- [x] integration tests
- [x] figure out the the best way to handle the various dtypes of the vision encoder and language model 

Nice to haves:
- [ ] doc tests
- [ ] int8 support (cc @younesbelkada)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thank you for your contribution! I noticed a potential problem with this open PR. It seems that the InstructBLIP processor is missing the QformerTokenizer compared to the BLIP2Processor. 

Thanks for your review. Updates:

- all `autocast` logic was removed, turns out the implementation returns the same exact logits as the original implementation when also using `float32` for the original implementation. However, we may need to think about supporting various dtypes of building blocks of a model, cause if you'd do `from_pretrained("...", dtype=torch.float16")`, that would break for the Flan-T5 checkpoints, which require `bfloat16`. It would be nice to provide the possibility to load the vision encoder in `float16` and the language model in `bfloat16`.
- The `InstructBlipProcessor` is a bit different than other processors in the sense that it consists of 1 image processor and 2 tokenizers (one for the language model, one for the Q-Former). I've included logic to save the Q-Former tokenizer files in a separate folder on the hub as can be seen [here](https://huggingface.co/nielsr/instructblip-vicuna-7b/tree/main), and had to overwrite the `from_pretrained` and `save_pretrained` methods to make this work. I know that this logic may need to be addressed in a separate PR.

Will the converted weights be hosted on the model hub like blip-2?

All checkpoints are transferred: https://huggingface.co/models?other=instructblip.

Feel free to merge the PR.

The only thing left is uploading fast tokenizer files for the Vicuna-based checkpoints, but that can only be done once https://github.com/huggingface/transformers/issues/23889 is fixed. Currently the fast tokenizer is created on-the-fly based on the slow tokenizer files when loading from the hub.

Update: that's now also done, so it's entirely ready

@amyeroberts Could you have a final look and merge if you are happy?

> There's InstructBlipTextModelTester but no InstructBlipTextModelTest 

In general, I would say yes to have 1-1 correspondence. But I don't want to make it strict if it doesn't really bring anything valuable.

The pipeline testing script would be easier if we have such correspondence, but since I was able to manage BLIP2 already, and this test file here is similar to BLIP2, I think it's fine.

> and some tests for InstructBlipModel are skipped because they're run in individual model tests.

It's same as CLIP test file, so it's OK :-)



@ydshieh Thanks for reviewing & info about the tests! 

> >and some tests for InstructBlipModel are skipped because they're run in individual model tests.
> It's same as CLIP test file, so it's OK :-)

Ah, sorry, I wasn't clear. What I meant was: if tests are skipped with the reason of being already tested in individual model tests, don't we need the modular tests classes implemented i.e. `InstructBlipTextModelTest`?  

> Ah, sorry, I wasn't clear. What I meant was: if tests are skipped with the reason of being already tested in individual model tests, don't we need the modular tests classes implemented i.e. InstructBlipTextModelTest?

I agree (was thinking the same but my mind is lost in my reply).

@NielsRogge I will let you to express why there is no text model test class :-), which is same as in BLIP2.

Well, after looking a bit, the text part is not a fixed model class
```
        if config.use_decoder_only_language_model:
            language_model = AutoModelForCausalLM.from_config(config.text_config)
        else:
            language_model = AutoModelForSeq2SeqLM.from_config(config.text_config)
```
I think that's the main reason why we don't have the test for that part.


Hi, will this land soon? I would love to try out this model. Thanks!

Thanks @amyeroberts for your review, there was a bug with `LlamaTokenizerFast` that has now been fixed, now the absolute tolerance is much lower (1e-4 and 1e-5).

I've removed `InstructBlipModel` from this PR as that was copied from `Blip2Model` using the CookieCutter template. The latter was added in this PR: #21817. However I'm not sure why the latter got approved, cause it's not really in lign with the design of the library, meaning that `xxxModel` are models not including any head on top and not accepting a `labels` argument. However `Blip2Model` seems like an entire copy of `Blip2ForConditionalGeneration`, which seems odd to me.

Do the prompt need further packaging when inference? For example, BLIP2 use "Question: {prompt}? Answer: " as prompt. And which type of prompt be used in InstructBLIP? Or we only use question to ask the model?

@NielsRogge It appears in the current diff that there a some changes unrelated to this PR? Could you rebase to sync up with `main`? Could you also respond to the questions in the PR review instead of just marking as resolved? 

Well 💚 

Merge it now as 🟢 is approved.

Hi @zdxff there's no specific prompt being used for InstructBLIP. You can just ask it questions like "What is unusual about this image?"

Will work on the 8bit / 4bit integration ASAP !

EDIT: here you go https://github.com/huggingface/transformers/pull/24488