Replacing agent image_qa tool with InstructBLIP
Author: austinmw
Pull request: no

### System Info

Hi, I'm attempting to use InstructBLIP for image QA:

```python
from transformers import HfAgent, load_tool
from diffusers.utils import load_image

agent = HfAgent("https://api-inference.huggingface.co/models/bigcode/starcoder")
image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/rivers_and_lakes.png")
agent.toolbox["image_qa"] = load_tool(task_or_repo_id="image-question-answering", model_repo_id="Salesforce/instructblip-vicuna-13b")

agent.run("what colors are in this image?", image=image)
```

However this gives me the error:

> ValueError: Unrecognized configuration class  for this kind of AutoModel: AutoModelForVisualQuestionAnswering.
> Model type should be one of ViltConfig.

I'm not sure if I'm doing this incorrectly, it's unsupported, or there's a bug.


### Who can help?

@amyeroberts @Narsil 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Code above

### Expected behavior

Was hoping that InstructBLIP would answer the question instead of VilT

Comments:
Hi @austinmw, thanks for raising this issue! 

This is because the model is being loaded [using the AutoModelForVisualQuestionAnswering](https://github.com/huggingface/transformers/blob/91d7df58b6537d385e90578dac40204cb550f706/src/transformers/tools/image_question_answering.py#L39C2-L39C2) class, [which only has Vilt listed as a compatible model](https://github.com/huggingface/transformers/blob/91d7df58b6537d385e90578dac40204cb550f706/src/transformers/models/auto/modeling_auto.py#L818).

BLIP can be loaded using `AutoModelForVision2Seq` - listed here with [other models like Pix2Struct](https://github.com/huggingface/transformers/blob/91d7df58b6537d385e90578dac40204cb550f706/src/transformers/models/auto/modeling_auto.py#L543C1-L543C37). 

The reason for this distinction is that the 'answers' from these models are obtained in two different ways. ViltForQuestionAnswering is really a classifier, and predicts the class most likely to be the answer to the question. You can see an example of the [categories here](https://huggingface.co/dandelin/vilt-b32-finetuned-vqa/blob/d0a1f6ab88522427a7ae76ceb6e1e1e7b68a1d08/config.json#L9). Whereas BLIP generates an answer using causal language modeling.

If you wish to use BLIP, you can easily define your own `ImageQuestionAnsweringTool` which you can modify to suit the behaviour (and generation strategy) you desire e.g.:

```python
import requests

from PIL import Image

from transformers import AutoModelForVision2Seq, AutoProcessor
from transformers.tools import PipelineTool
from transformers.utils import requires_backends

class ImageQuestionAnsweringTool(PipelineTool):
    default_checkpoint = "Salesforce/blip2-opt-2.7b"
    description = (
        "This is a tool that answers a question about an image. It takes an input named `image` which should be the "
        "image containing the information, as well as a `question` which should be the question in English. It "
        "returns a text that is the answer to the question."
    )
    name = "image_qa"
    pre_processor_class = AutoProcessor
    model_class = AutoModelForVision2Seq

    inputs = ["image", "text"]
    outputs = ["text"]

    def __init__(self, *args, **kwargs):
        requires_backends(self, ["vision"])
        super().__init__(*args, **kwargs)

    def encode(self, image, question: str):
        return self.pre_processor(image, question, return_tensors="pt")

    def forward(self, inputs):
        outputs = self.model.generate(**inputs, max_new_tokens=50)
        return outputs

    def decode(self, outputs):
        return self.pre_processor.batch_decode(outputs, skip_special_tokens=True)[0]


url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
question = "Question: what is the colour of the sofa? Answer: "

tool = ImageQuestionAnsweringTool()
response = tool(image=image, question=question)
print(response)
```


cc @LysandreJik

Ah I see, thanks, and I really appreciate the example you provided!! 🙏🏻

One follow-up question though, how can I set `load_in_4bit=True, torch_dtype=torch.float16` for the model_class?

@austinmw You can pass in an [instantiated model](https://huggingface.co/docs/transformers/v4.31.0/en/main_classes/agent#transformers.PipelineTool.model) when instantiating the tool.