Error while running agent for Image Question Answering
Author: pratikkotian04
Pull request: no

### System Info

Code :
import requests
from PIL import Image
import torch
image_url = "https://raw.githubusercontent.com/vis-nlp/ChartQA/main/ChartQA%20Dataset/val/png/20294671002019.png"
image = Image.open(requests.get(image_url, stream=True).raw)
#document = '/content/child_death.png'

agent.run(
    question = "Which Country has the highest number of child deaths ?",
    image=image,
    task = 'image_qa'
)

Error:
==Explanation from the agent==
I will use the following  tool: `image_qa` to answer a question about an image.


==Code generated by the agent==
answer = image_qa(image=image, question=question)
print(f"The answer is {answer}.")


==Result==
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ in <cell line: 8>:8                                                                              │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/tools/agents.py:323 in run                  │
│                                                                                                  │
│   320 │   │   if not return_code:                                                                │
│   321 │   │   │   print("\n\n==Result==")                                                        │
│   322 │   │   │   self.cached_tools = resolve_tools(code, self.toolbox, remote=remote, cached_   │
│ ❱ 323 │   │   │   return evaluate(code, self.cached_tools, state=kwargs.copy())                  │
│   324 │   │   else:                                                                              │
│   325 │   │   │   tool_code = get_tool_creation_code(code, self.toolbox, remote=remote)          │
│   326 │   │   │   return f"{tool_code}\n{code}"                                                  │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/tools/python_interpreter.py:61 in evaluate  │
│                                                                                                  │
│    58 │   result = None                                                                          │
│    59 │   for idx, node in enumerate(expression.body):                                           │
│    60 │   │   try:                                                                               │
│ ❱  61 │   │   │   line_result = evaluate_ast(node, state, tools)                                 │
│    62 │   │   except InterpretorError as e:                                                      │
│    63 │   │   │   msg = f"Evaluation of the code stopped at line {idx} before the end because    │
│    64 │   │   │   if chat_mode:                                                                  │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/tools/python_interpreter.py:98 in           │
│ evaluate_ast                                                                                     │
│                                                                                                  │
│    95 │   if isinstance(expression, ast.Assign):                                                 │
│    96 │   │   # Assignement -> we evaluate the assignement which should update the state         │
│    97 │   │   # We return the variable assigned as it may be used to determine the final resul   │
│ ❱  98 │   │   return evaluate_assign(expression, state, tools)                                   │
│    99 │   elif isinstance(expression, ast.Call):                                                 │
│   100 │   │   # Function call -> we return the value of the function call                        │
│   101 │   │   return evaluate_call(expression, state, tools)                                     │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/tools/python_interpreter.py:139 in          │
│ evaluate_assign                                                                                  │
│                                                                                                  │
│   136                                                                                            │
│   137 def evaluate_assign(assign, state, tools):                                                 │
│   138 │   var_names = assign.targets                                                             │
│ ❱ 139 │   result = evaluate_ast(assign.value, state, tools)                                      │
│   140 │                                                                                          │
│   141 │   if len(var_names) == 1:                                                                │
│   142 │   │   state[var_names[0].id] = result                                                    │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/tools/python_interpreter.py:101 in          │
│ evaluate_ast                                                                                     │
│                                                                                                  │
│    98 │   │   return evaluate_assign(expression, state, tools)                                   │
│    99 │   elif isinstance(expression, ast.Call):                                                 │
│   100 │   │   # Function call -> we return the value of the function call                        │
│ ❱ 101 │   │   return evaluate_call(expression, state, tools)                                     │
│   102 │   elif isinstance(expression, ast.Constant):                                             │
│   103 │   │   # Constant -> just return the value                                                │
│   104 │   │   return expression.value                                                            │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/tools/python_interpreter.py:167 in          │
│ evaluate_call                                                                                    │
│                                                                                                  │
│   164 │   # Todo deal with args                                                                  │
│   165 │   args = [evaluate_ast(arg, state, tools) for arg in call.args]                          │
│   166 │   kwargs = {keyword.arg: evaluate_ast(keyword.value, state, tools) for keyword in call   │
│ ❱ 167 │   return func(*args, **kwargs)                                                           │
│   168                                                                                            │
│   169                                                                                            │
│   170 def evaluate_subscript(subscript, state, tools):                                           │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/tools/base.py:534 in __call__               │
│                                                                                                  │
│   531 │   │   if not self.is_initialized:                                                        │
│   532 │   │   │   self.setup()                                                                   │
│   533 │   │                                                                                      │
│ ❱ 534 │   │   encoded_inputs = self.encode(*args, **kwargs)                                      │
│   535 │   │   encoded_inputs = send_to_device(encoded_inputs, self.device)                       │
│   536 │   │   outputs = self.forward(encoded_inputs)                                             │
│   537 │   │   outputs = send_to_device(outputs, "cpu")                                           │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/tools/image_question_answering.py:49 in     │
│ encode                                                                                           │
│                                                                                                  │
│   46 │   │   super().__init__(*args, **kwargs)                                                   │
│   47 │                                                                                           │
│   48 │   def encode(self, image: "Image", question: str):                                        │
│ ❱ 49 │   │   return self.pre_processor(image, question, return_tensors="pt")                     │
│   50 │                                                                                           │
│   51 │   def forward(self, inputs):                                                              │
│   52 │   │   with torch.no_grad():                                                               │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/models/vilt/processing_vilt.py:107 in       │
│ __call__                                                                                         │
│                                                                                                  │
│   104 │   │   │   **kwargs,                                                                      │
│   105 │   │   )                                                                                  │
│   106 │   │   # add pixel_values + pixel_mask                                                    │
│ ❱ 107 │   │   encoding_image_processor = self.image_processor(images, return_tensors=return_te   │
│   108 │   │   encoding.update(encoding_image_processor)                                          │
│   109 │   │                                                                                      │
│   110 │   │   return encoding                                                                    │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/image_processing_utils.py:464 in __call__   │
│                                                                                                  │
│   461 │                                                                                          │
│   462 │   def __call__(self, images, **kwargs) -> BatchFeature:                                  │
│   463 │   │   """Preprocess an image or a batch of images."""                                    │
│ ❱ 464 │   │   return self.preprocess(images, **kwargs)                                           │
│   465 │                                                                                          │
│   466 │   def preprocess(self, images, **kwargs) -> BatchFeature:                                │
│   467 │   │   raise NotImplementedError("Each image processor must implement its own preproces   │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/models/vilt/image_processing_vilt.py:462 in │
│ preprocess                                                                                       │
│                                                                                                  │
│   459 │   │   images = [to_numpy_array(image) for image in images]                               │
│   460 │   │                                                                                      │
│   461 │   │   if do_resize:                                                                      │
│ ❱ 462 │   │   │   images = [                                                                     │
│   463 │   │   │   │   self.resize(image=image, size=size, size_divisor=size_divisor, resample=   │
│   464 │   │   │   ]                                                                              │
│   465                                                                                            │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/models/vilt/image_processing_vilt.py:463 in │
│ <listcomp>                                                                                       │
│                                                                                                  │
│   460 │   │                                                                                      │
│   461 │   │   if do_resize:                                                                      │
│   462 │   │   │   images = [                                                                     │
│ ❱ 463 │   │   │   │   self.resize(image=image, size=size, size_divisor=size_divisor, resample=   │
│   464 │   │   │   ]                                                                              │
│   465 │   │                                                                                      │
│   466 │   │   if do_rescale:                                                                     │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/models/vilt/image_processing_vilt.py:230 in │
│ resize                                                                                           │
│                                                                                                  │
│   227 │   │   │   raise ValueError(f"The `size` dictionary must contain the key `shortest_edge   │
│   228 │   │   shorter = size["shortest_edge"]                                                    │
│   229 │   │   longer = int(1333 / 800 * shorter)                                                 │
│ ❱ 230 │   │   output_size = get_resize_output_image_size(image, shorter=shorter, longer=longer   │
│   231 │   │   return resize(image, size=output_size, resample=resample, data_format=data_forma   │
│   232 │                                                                                          │
│   233 │   def rescale(                                                                           │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/models/vilt/image_processing_vilt.py:87 in  │
│ get_resize_output_image_size                                                                     │
│                                                                                                  │
│    84 def get_resize_output_image_size(                                                          │
│    85 │   input_image: np.ndarray, shorter: int = 800, longer: int = 1333, size_divisor: int =   │
│    86 ) -> Tuple[int, int]:                                                                      │
│ ❱  87 │   input_height, input_width = get_image_size(input_image)                                │
│    88 │   min_size, max_size = shorter, longer                                                   │
│    89 │                                                                                          │
│    90 │   scale = min_size / min(input_height, input_width)                                      │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/image_utils.py:205 in get_image_size        │
│                                                                                                  │
│   202 │   │   A tuple of the image's height and width.                                           │
│   203 │   """                                                                                    │
│   204 │   if channel_dim is None:                                                                │
│ ❱ 205 │   │   channel_dim = infer_channel_dimension_format(image)                                │
│   206 │                                                                                          │
│   207 │   if channel_dim == ChannelDimension.FIRST:                                              │
│   208 │   │   return image.shape[-2], image.shape[-1]                                            │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/image_utils.py:169 in                       │
│ infer_channel_dimension_format                                                                   │
│                                                                                                  │
│   166 │   │   return ChannelDimension.FIRST                                                      │
│   167 │   elif image.shape[last_dim] in (1, 3):                                                  │
│   168 │   │   return ChannelDimension.LAST                                                       │
│ ❱ 169 │   raise ValueError("Unable to infer channel dimension format")                           │
│   170                                                                                            │
│   171                                                                                            │
│   172 def get_channel_dimension_axis(image: np.ndarray) -> int:                                  │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: Unable to infer channel dimension format



### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Code :
import requests
from PIL import Image
import torch
image_url = "https://raw.githubusercontent.com/vis-nlp/ChartQA/main/ChartQA%20Dataset/val/png/20294671002019.png"
image = Image.open(requests.get(image_url, stream=True).raw)
#document = '/content/child_death.png'

agent.run(
    question = "Which Country has the highest number of child deaths ?",
    image=image,
    task = 'image_qa'
)

### Expected behavior

Answer from the graph

Comments:
Hi @pratikkotian04, thanks for raising this issue and for providing so much detail. 

I'm able to reproduce the issue. It seems the error is coming from our image processing library, I'll dig into it.

@pratikkotian04 The error is arising because the image has a depth channel i.e. is in RGBA format, and the image processors expect images to have 1 or 3 image channels. This is a know brittleness with the image processing library and a priority for me to address. 

A robust solution is a bit involved so won't be merge in in the next day or two. A quick solution is to convert the image to RGB format before passing to the agent: 

```python
import requests
from PIL import Image
from transformers import HfAgent

image_url = "https://raw.githubusercontent.com/vis-nlp/ChartQA/main/ChartQA%20Dataset/val/png/20294671002019.png"
image = Image.open(requests.get(image_url, stream=True).raw)
image = image.convert("RGB")
agent = HfAgent("https://api-inference.huggingface.co/models/bigcode/starcoder")
agent.run(
    question = "Which Country has the highest number of child deaths ?",
    image=image,
    task = 'image_qa'
)
```

I was able to run the above snippet (although the response was: `The answer is england.` 👀 )



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.