agent fail
Author: ltm920716
Pull request: no

### System Info

transformers==4.30.0.dev0

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

![image](https://github.com/huggingface/transformers/assets/12007227/ca5005b7-9704-436a-86d5-64c4e9d92930)

from transformers import HfAgent
agent = HfAgent("https://api-inference.huggingface.co/models/bigcode/starcoder")

from PIL import Image
x = Image.open('test.jpg')

agent.run("here is a image named `image`, what does it belong to?", image=x, remote=True)
==Explanation from the agent==
I will use the following  tool: `image_classifier` to classify the image.


==Code generated by the agent==
label = image_classifier(image)
print(f"The label is {label}.")


==Result==
Evaluation of the code stopped at line 0 before the end because of the following error:
It is not permitted to evaluate other functions than the provided tools (tried to execute image_classifier).

agent.run("here is a image named `image`, what does it belong to?", image='test.jpg', remote=True)
==Explanation from the agent==
I will use the following  tool: `image_classifier` to classify the image.


==Code generated by the agent==
label = image_classifier(image)
print(f"The label is {label}.")


==Result==
Evaluation of the code stopped at line 0 before the end because of the following error:
It is not permitted to evaluate other functions than the provided tools (tried to execute image_classifier).



### Expected behavior

run normally

Comments:
cc @sgugger 

Run normally meaning? I'm not even completely sure what you want the agent to run so I don't see how the LLM could find out too ;-)
Make sure to:
1. Use openAI, sadly it's better than the opensource alternatives
2. refine your prompt input, we have a great guide for that [here](https://huggingface.co/docs/transformers/custom_tools#writing-good-user-inputs)

hi @sgugger ,
I am sorry for my bad description.

There are two problems when I run the command `agent.run("here is a image named image, what does it belong to?", image=x, remote=True)`

1、The agent return tool `image_classifier ` which not in toolbox. According to the base run-prompt，it should only return tool in toolbox. So this is only because the llm's capacity？

2、for tool like `image_classifier` or `image_caption`， it‘s input is `image`，what is the type  of `image`? PIL or Numpy or str(local path)? 

thanks!


it seems that the param `remote=True` does not work, what tools can be loaded remotely?

The agent can return whatever the hell it wants. If it decides to use tools that don't exist, there is nothing we can do (again use openAI to get better results).
There is no image classifier tool. For tools working on images, the input type required is a standard PIL Image.

As for your last comment `remote=True` on all tools.

@sgugger 
thanks

![image](https://github.com/huggingface/transformers/assets/12007227/8517c922-b0cc-4402-ae4b-9f93d0720211)

for param `remote=True`， it seems does not work. What tools support remote and where I can search?


@ltm920716 there is a dataset that lists all the standard tools that use an endpoint for demonstration purpose, and then that you can use remotely : https://huggingface.co/datasets/huggingface-tools/default-endpoints