Cannot decode image from remote image segmentation tool
Author: freddyaboulton
Pull request: no

### System Info

- `transformers` version: 4.29.0
- Platform: macOS-12.2.1-arm64-arm-64bit
- Python version: 3.9.16
- Huggingface_hub version: 0.14.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


### Who can help?

@sgugger @LysandreJik 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run the following script. The problem goes away when running the local tool.

```python
from transformers import HfAgent
from diffusers.utils import load_image

bunny_img = load_image("https://gradio-builds.s3.amazonaws.com/sample-images/SpaceBunny.png")

agent = HfAgent("https://api-inference.huggingface.co/models/bigcode/starcoder")

segmented_img = agent.run("Please locate the bunny in the image", image=bunny_img, remote=True)
```

```
in <module>:8                                                                                    │
│                                                                                                  │
│   5                                                                                              │
│   6 agent = HfAgent("https://api-inference.huggingface.co/models/bigcode/starcoder")             │
│   7                                                                                              │
│ ❱ 8 segmented_img = agent.run("Please locate the bunny in the image", image=bunny_img, remot     │
│   9                                                                                              │
│                                                                                                  │
│ /Users/freddy/miniconda3/envs/gradio-tools/lib/python3.9/site-packages/transformers/tools/agents │
│ .py:323 in run                                                                                   │
│                                                                                                  │
│   320 │   │   if not return_code:                                                                │
│   321 │   │   │   print("\n\n==Result==")                                                        │
│   322 │   │   │   self.cached_tools = resolve_tools(code, self.toolbox, remote=remote, cached_   │
│ ❱ 323 │   │   │   return evaluate(code, self.cached_tools, state=kwargs.copy())                  │
│   324 │   │   else:                                                                              │
│   325 │   │   │   tool_code = get_tool_creation_code(code, self.toolbox, remote=remote)          │
│   326 │   │   │   return f"{tool_code}\n{code}"                                                  │
│                                                                                                  │
│ /Users/freddy/miniconda3/envs/gradio-tools/lib/python3.9/site-packages/transformers/tools/python │
│ _interpreter.py:61 in evaluate                                                                   │
│                                                                                                  │
│    58 │   result = None                                                                          │
│    59 │   for idx, node in enumerate(expression.body):                                           │
│    60 │   │   try:                                                                               │
│ ❱  61 │   │   │   line_result = evaluate_ast(node, state, tools)                                 │
│    62 │   │   except InterpretorError as e:                                                      │
│    63 │   │   │   msg = f"Evaluation of the code stopped at line {idx} before the end because    │
│    64 │   │   │   if chat_mode:                                                                  │
│                                                                                                  │
│ /Users/freddy/miniconda3/envs/gradio-tools/lib/python3.9/site-packages/transformers/tools/python │
│ _interpreter.py:98 in evaluate_ast                                                               │
│                                                                                                  │
│    95 │   if isinstance(expression, ast.Assign):                                                 │
│    96 │   │   # Assignement -> we evaluate the assignement which should update the state         │
│    97 │   │   # We return the variable assigned as it may be used to determine the final resul   │
│ ❱  98 │   │   return evaluate_assign(expression, state, tools)                                   │
│    99 │   elif isinstance(expression, ast.Call):                                                 │
│   100 │   │   # Function call -> we return the value of the function call                        │
│   101 │   │   return evaluate_call(expression, state, tools)                                     │
│                                                                                                  │
│ /Users/freddy/miniconda3/envs/gradio-tools/lib/python3.9/site-packages/transformers/tools/python │
│ _interpreter.py:139 in evaluate_assign                                                           │
│                                                                                                  │
│   136                                                                                            │
│   137 def evaluate_assign(assign, state, tools):                                                 │
│   138 │   var_names = assign.targets                                                             │
│ ❱ 139 │   result = evaluate_ast(assign.value, state, tools)                                      │
│   140 │                                                                                          │
│   141 │   if len(var_names) == 1:                                                                │
│   142 │   │   state[var_names[0].id] = result                                                    │
│                                                                                                  │
│ /Users/freddy/miniconda3/envs/gradio-tools/lib/python3.9/site-packages/transformers/tools/python │
│ _interpreter.py:101 in evaluate_ast                                                              │
│                                                                                                  │
│    98 │   │   return evaluate_assign(expression, state, tools)                                   │
│    99 │   elif isinstance(expression, ast.Call):                                                 │
│   100 │   │   # Function call -> we return the value of the function call                        │
│ ❱ 101 │   │   return evaluate_call(expression, state, tools)                                     │
│   102 │   elif isinstance(expression, ast.Constant):                                             │
│   103 │   │   # Constant -> just return the value                                                │
│   104 │   │   return expression.value                                                            │
│                                                                                                  │
│ /Users/freddy/miniconda3/envs/gradio-tools/lib/python3.9/site-packages/transformers/tools/python │
│ _interpreter.py:167 in evaluate_call                                                             │
│                                                                                                  │
│   164 │   # Todo deal with args                                                                  │
│   165 │   args = [evaluate_ast(arg, state, tools) for arg in call.args]                          │
│   166 │   kwargs = {keyword.arg: evaluate_ast(keyword.value, state, tools) for keyword in call   │
│ ❱ 167 │   return func(*args, **kwargs)                                                           │
│   168                                                                                            │
│   169                                                                                            │
│   170 def evaluate_subscript(subscript, state, tools):                                           │
│                                                                                                  │
│ /Users/freddy/miniconda3/envs/gradio-tools/lib/python3.9/site-packages/transformers/tools/base.p │
│ y:399 in __call__                                                                                │
│                                                                                                  │
│   396 │   │   output_image = self.tool_class is not None and self.tool_class.outputs == ["imag   │
│   397 │   │   inputs = self.prepare_inputs(*args, **kwargs)                                      │
│   398 │   │   if isinstance(inputs, dict):                                                       │
│ ❱ 399 │   │   │   outputs = self.client(**inputs, output_image=output_image)                     │
│   400 │   │   else:                                                                              │
│   401 │   │   │   outputs = self.client(inputs, output_image=output_image)                       │
│   402 │   │   if isinstance(outputs, list) and len(outputs) == 1 and isinstance(outputs[0], li   │
│                                                                                                  │
│ /Users/freddy/miniconda3/envs/gradio-tools/lib/python3.9/site-packages/transformers/tools/base.p │
│ y:721 in __call__                                                                                │
│                                                                                                  │
│   718 │   │                                                                                      │
│   719 │   │   # By default, parse the response for the user.                                     │
│   720 │   │   if output_image:                                                                   │
│ ❱ 721 │   │   │   return self.decode_image(response.content)                                     │
│   722 │   │   else:                                                                              │
│   723 │   │   │   return response.json()                                                         │
│   724                                                                                            │
│                                                                                                  │
│ /Users/freddy/miniconda3/envs/gradio-tools/lib/python3.9/site-packages/transformers/tools/base.p │
│ y:698 in decode_image                                                                            │
│                                                                                                  │
│   695 │   │                                                                                      │
│   696 │   │   from PIL import Image                                                              │
│   697 │   │                                                                                      │
│ ❱ 698 │   │   b64 = base64.b64decode(raw_image)                                                  │
│   699 │   │   _bytes = io.BytesIO(b64)                                                           │
│   700 │   │   return Image.open(_bytes)                                                          │
│   701                                                                                            │
│                                                                                                  │
│ /Users/freddy/miniconda3/envs/gradio-tools/lib/python3.9/base64.py:87 in b64decode               │
│                                                                                                  │
│    84 │   │   s = s.translate(bytes.maketrans(altchars, b'+/'))                                  │
│    85 │   if validate and not re.fullmatch(b'[A-Za-z0-9+/]*={0,2}', s):                          │
│    86 │   │   raise binascii.Error('Non-base64 digit found')                                     │
│ ❱  87 │   return binascii.a2b_base64(s)                                                          │
│    88                                                                                            │
│    89                                                                                            │
│    90 def standard_b64encode(s):                                                                 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
Error: Incorrect padding
```

### Expected behavior

The image is able to be correctly decoded

Comments:
The endpoint seems to have trouble indeed. It might need a restart @LysandreJik. Form what I see it complains about the inputs being named `image` and `label` and expects a `prompt`.

Thank you, nice catch @freddyaboulton! 

It should be working now.

Thank you for the very speedy fix @LysandreJik !