is there any plan to add falcon to instructblip?
Author: Ashwath-Shetty
Pull request: no

### Model description

instructblip seems to be really cool, is there any possibility to add falcon also to the pipeline in the future?. currently we have option for flant5 & vicuna. problem is you cannot use vicuna for commercial use & flant5 performance is poor, also vicuna is not that great too, adding falcon to the pipeline will massively boost the performance of instructblip.
incase if i want to add it myself how can i do that? code base seems to be really heavy.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

https://huggingface.co/tiiuae
@NielsRogge @DanielHesslow @guipenedo @slippylolo @FalconLLM @mickbo32 @karnakar 

Comments:
cc @NielsRogge 

However, falcon port is still in WIP


https://github.com/huggingface/transformers/pull/24523

thanks for replying @ydshieh , any idea when it'll come to the instructblip pipeline?,  any tentative timeline for the same? 
also, is there a way to add the falcon locally by our side to the pipeline in place of vicuna/flant-5?
plan is to plugin this falcon (https://huggingface.co/tiiuae/falcon-40b) to the instructblip pipeline somehow & then we can choose modeltype="falcon"
model, vis_processors, _ = load_model_and_preprocess(name="blip2_vicuna_instruct", model_type="vicuna7b", is_eval=True, device=device)



@ydshieh Falcon is merged, but I'm not sure exactly how to add it to the instructblip pipeline, I don't find any weights for the same and we do have this #25789 as an alternative. 

Hi, I am not sure what `add falcon to the instructblip pipeline`. Could you elaborate this in more details?

Okay so I wasn't very sure how to make this work, but InstructBlip does take in the language model like this, right now it takes in LLaMA by default - 
https://github.com/shauray8/transformers/blob/eb8489971ac1415f67b0abdd1584fde8b659ced9/src/transformers/models/instructblip/modeling_instructblip.py#L1272-L1275

OK, I understand better now. Will check

Hi @Ashwath-Shetty @shauray8 Sorry for being so long to this issue.

I believe you want to load the pretrained model (`vision_model`, `qformer` for example) from the one existing InstuctBlip checkpoint, but with `language_model` being from a `falcon` checkpoint.

There is not direct support for this however. What we can do is to
- load an existing ` InstuctBlip` checkpoint , say `instruct_model = ...`
- load an existing `Falcon` checkpoint, say, `falcon_model = ...`
- then set `instruct_model.language_model = falcon_model`
- we have to set also `instruct_model.config.text_config = falcon_model.config`

But if we do so, we have to make sure the `hidden_size` of the loaded falcon model is the same as `instruct_model .config.text_config.hidden_size`, otherwise we have to replace that component too.

Don't hesitate if you need further guides.

cc @NielsRogge to see if he has any further comment.




Yes technically you can instantiate an InstructBLIP model with any `AutoModelForCausalLM`, like so:
```
from transformers import FalconConfig, InstructBlipConfig, InstructBlipForConditionalGeneration

text_config = FalconConfig.from_pretrained("tiiuae/falcon-7b")
config = InstructBlipConfig(text_config=text_config)

model = InstructBlipForConditionalGeneration(config)
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.