HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
Authors: Yongliang Shen
Kaitao Song
Xu Tan
Dongsheng Li
Weiming Lu
Yueting Zhuang
Venue: NeurIPS 2023 poster (2023)
Keywords: LLM
ChatGPT
Hugging Face
Autonomous LLM

Abstract: Solving complicated AI tasks with different domains and modalities is a key step toward artificial general intelligence. While there are numerous AI models available for various domains and modalities, they cannot handle complicated AI tasks autonomously. Considering large language models (LLMs) have exhibited exceptional abilities in language understanding, generation, interaction, and reasoning, we advocate that LLMs could act as a controller to manage existing AI models to solve complicated AI tasks, with language serving as a generic interface to empower this. Based on this philosophy, we present HuggingGPT, an LLM-powered agent that leverages LLMs (e.g., ChatGPT) to connect various AI models in machine learning communities (e.g., Hugging Face) to solve AI tasks. Specifically, we use ChatGPT to conduct task planning when receiving a user request, select models according to their function descriptions available in Hugging Face, execute each subtask with the selected AI model, and summarize the response according to the execution results. By leveraging the strong language capability of ChatGPT and abundant AI models in Hugging Face, HuggingGPT can tackle a wide range of sophisticated AI tasks spanning different modalities and domains and achieve impressive results in language, vision, speech, and other challenging tasks, which paves a new way towards the realization of artificial general intelligence.

HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
Yongliang Shen Kaitao Song2.*t,, Xu Tan",
Dongsheng Li2, Weiming Lu'st, Yueting Zhuang
Zhejiang University' Microsoft Mffrrooff Research Asia
[syl, luwm, yzhuang)@2 edu. en, (kaitaosong, xuta, dongsli)@microsonl com
Abstract
Solving complicated AI tasks with different domains and modalities is key step toward artificial general intelligence. While there are numerous AI models avail- able for various domains and modalities, they cannot handle complicated Al tasks autonomously. Considering large language models (LLMs) have exhibited excep- tional abilities language understanding, generation, interaction, and reasoning, we advocate that LLMs could act controller to manage existing AI models to solve complicated Al tasks, with language serving as a generic interface to em- power this. Based this philosophy, we present HuggingGPT,, agent that leverages LLMs (e.g.. ChatGPT) to connect various AI models in ma- chine learning communities (e.g.. Hugging Face) solve AI tasks. Specifically, we use ChatGPT to conduct task planning when receiving a user request, select models according their function descriptions available in Hugging Face, execute each subtask with the selected AI model, and summarize the response according the execution results. By leveraging the strong language capability of ChatGPT and abundant AI models in Hugging Face, HuggingGPT can tackle a wide range of sophisticated AI tasks spanning different modalities and domains and achieve impressive results in language, vision, speech, and other challenging tasks, which paves new way towards the realization of artificial general intelligence.
LLM-ppwered
Introduction
Large language models (LLMs) [IL 2. 0L 9. 5. B),, such as ChatGPT, have attracted substantial at- tention from both academia and industry, due their remarkable performance various natural language processing (NIP) tasks. Based on large-scale pre-training on massive text corpora and re- inforcement learning from human feedback [2), LLMs can exhibit superior capabilities language understanding, generation, and reasoning. The powerful capability of LLMs also drives many emer- gent research topics (e.g., in-context learning (II, ZZ. N], instruction learning [9] III, II2, II3, [3]], and chain-of-Ne of-tthutgtt prompting [0.55 DTG. 17. 18)) to further investigate the huge potential of LLMs, and brings unlimited possibilities for us for advancing artificial general intelligence.
Despite these great successes, current LLM technologies are still imperfect and confront some urgent challenges the way building advanced AI system. We discuss them from these aspects: 1) Limited to the input and output forms of text generation, current LLMs lack the ability to process complex information such vision and speech, regardless of their significant achievements in NLP
The first two authors have equal contributions. This work was done when the first author was an intern at Microsoft Research Asia. Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

HuggingGPT
LLM Controller
HuggingFace
Can you describe this picture count how many objects in the picture?
I Task Planing
Prediction
facebook/ detr-resnet 101
Model Selection
A text can describe the given image: a herd of giraffes and zebras grazing a fields. In addition, giratte with score 99,9%, zebra score 99.7%, zebra with 99,5 9%, giraffe with score 97.1% and zebra with score 8%. The bounding baxes are shown in the ciassilication, object detection and image caption on this image. Combining g the predictions of (6 nipconnet/vit will apt2-image- captioning, facebook/dett-- -101 and google/vit models, get results. you.
III Task Execution
detected
nipconnet/ vtt-pit--magge-aption
Prediction
performed
IV Response Generation
interface for LLMs (e.g.. CharGPT) to connect numerous Al models (e.g., those in Hugging Face) for solving complicated Al tasks. In this concept, an LLM acts as a controller, managing and organizing the cooperation of expert models. The LLM first plans list of tasks based on the user request and then assigns expert models to each task. After the experts execute the tasks, the LLM collects the results and responds the user.
Figure 1: Language serves
tasks; 2) In real-world scenarios, some complex tasks are usually composed of multiple sub-tasks, and thus require the scheduling and cooperation of multiple models, which are also beyond the capability of language models; 3) For some challenging tasks, LLMs demonstrate excellent results in zero-shot or few-shot settings, but they are still weaker than some experts (e.g., fine-tuned models). How address these issues could be the critical step for LLMs toward artificial general intelligence.
In this paper, we point out that in order to handle complicated AI tasks, LLMs should be able to co- ordinate with external models to harness their powers. Hence, the pivotal question is how choose suitable middleware to bridge the connections between LLMs and Al models. To tackle this issue, we notice that each AI model can be described in the form of language by summarizing its function. Therefore, we introduce a concept: "Language as a generic interface for LLMs to collaborate with Al models". In other words, by incorporating these model descriptions into prompts, LLMs can be considered as the brain to manage AI models such as planning, scheduling, and cooperation. As a re- sult, this strategy empowers LLMs invoke external models for solving Al tasks. However, when comes to integrating multiple AI models into LLMs, another challenge emerges: solving numerous Al tasks needs collecting large number of high-quality model descriptions, which in turn requires heavy prompt engineering. Coincidentally, we notice that some public MI. communities usually offer wide range of applicable models with well-defined model descriptions for solving specific AI tasks such as language, vision, and speech. These observations bring us some inspiration: Can we link LLMs (e.g., ChatGPT) with public ML communities (e.g., GitHub, Hugging Face D. etc) for solving complex AI tasks via language-based interface?
In this paper, we propose an LLM- agent named HuggingGPT to autonomously tackle wide range of complex AI tasks, which connects LLMs (i.e., ChatGPT) and the ML community (i.c., Hugging Face) and can process inputs from different modalities. More specifically, the LLM acts as brain: on one hand, it disassembles tasks based user requests, and on the other hand, assigns suitable models to the tasks according to the model description. By executing models and integrating results in the planned tasks, HuggingGPT can autonomously fulfill complex user requests. The whole process of HuggingGPT, illustrated in Figure II can be divided into four stages:
Task Planning: Using ChatGPT to analyze the requests of users understand their intention, and disassemble them into possible solvable tasks.
Model Selection: To solve the planned tasks, ChatGPT selects expert models that are hosted on Hugging Face based on model descriptions.
Task Execution: Invoke and execute each selected model, and return the results ChatGPT.
hattps://uugiiggaaat co/noell

---

Please generate an image where girl is reading book, and her pose is the same as the boy in the image example.jpg. then please describe the new image with your voice.
Request
Stage #1: Task Planning
Stage #2: Model Selection
task: pose-det.
lack:lmage-coo
task: pose-det.
context task assignment:
args:
arga:
angs:
image: example.jpp jpg
image: cresource--2
imager cresource-2:
Query
facebook/cet resnet 101
lask deppeeeddtyttyy
object
TahaDougi/ www----bl--deeeeeeeeeee
task: pose imaaaa
[T4]
angs:
Meri Card
arge:
taskc object-det. argst
speech
imager cresourcee-
text: reading girl
args:
image: cresource 1>
text: cressurce--
tosk: Image-class
Stage #3: Task Execution
Stage #4: Response Generation
[[T]
pose-det.
HT4] object-det.
Hybrid Endpoints
openpose
facebook/detr resnet
task, model
[image-2]
[bounding box]
HuggingFace Endpoint [racsbook/Setr 101)
wiith probabilities
[T2]
pose-to-image
[TS] image-to-text
obj-det. cresource-2>
lym/sd-ctl-pose
nipconnect/vil /vit gpt2
Predictions
[image-3]
girl sitting
Local Endpoint (Secaioock
image-class.
text-to-speed
task, args,
google/vit
facebook/fastseech
[classes with prob]
[audio-1]
Response
girl sitting
reading book
[Text-1] <resource-5>
[Image-1] = example.jpg [Image-2] 2]= <resource-1> [Image-3] <resaurce-2>
[Image-4]
[Audio-1]
The image you gave me of "boy". The first thing was use ( OpenC'Vss opennposss model to analyze the pase the boy the image. fed that specifically located [Image-1], the model to get /tt////ooooonett--ee
the pase, located (Image-2), Then take that
pose and generate new image, located at (Image-3), using & lllyaviiel/ foll/dd-con petooonntt-ppppe opoeppos and the instruction "a sirl reading". Next, image generated
previous step, get predictions the objects present location. These predictions were run through the ( facebook/ /e///de---eeeee resnet-101 model. The box was drawn on the image, which located predictions image classification model soogle/vit-base patch16-222 and image caption model enipconnect/vee /iit- vit--gttttinn gotg-lmaggeee captionin model to generate the caption for newly generated image. It generated text: "a girl sitting bed wewsssseeeeeee--lpleeee een--sjseeecch model to convert the generated text into audio, located at [Audio-1].
object
image, including box
[Image-4] 4]. Then,
reading book", Lastly, ran the
Figure 2: Overview of HPggingGPT. With an LLM (e.g., ChatGPT) as the core controller and the expert models the executors, the workflow of HuggingGPT consists of four stages: 1) Task planning: LLM parses the user request into a task list and determines the execution order and resource dependencies among tasks; Model selection: LLM assigns appropriate models to tasks based on the description of expert models on Hugging Face; 3) Task execution: Expert models on hybrid endpoints execute the assigned tasks; 4) Response generation: LLM integrates the inference results of experts and generates a summary of workflow logs to respond to the user.
Response Generation: Finally, ChatGPT utilized to integrate the predictions from models and generate responses for users.
Benefiting from such a design, HuggingGPT can automatically generate plans from user requests and use externa models, enabling integrate multimodal perceptual capabilities and tackle var- ious complex AI tasks. More notably, this pipeline allows HuggingGPT to continually absorb the powers from task-specific experts, facilitating the growth and scalability of AI capabilities.
Overall, our contributions can be summarized as follows:
To complement the advantages large language models and expert models, we propose Hug- gingGPT with an inter-model cooperation protocol. HuggingGPT applies LLMs as the brain for planning and decision, and automatically invokes and executes expert models for each specific task, providing way for designing general AI solutions.

---

2. By integrating the Hugging Face hub with numerous task-specifie models around ChatGPT, HuggingGPT is able to tackle generalized Al tasks covering multiple modalities and domains. Through the open collaboration of models, HuggingGPP can provide users with multimodal and reliable conversation services.
3. We point out the importance of task planning and model selection in HuggingGPT (and au- tonomous agents), and formulate some experimental evaluations for measuring the capability of LLMs in planning and model selection.
4. Extensive experiments on multiple challenging Al tasks across language, vision, speech, and cross-modality demonstrate the capability and huge potential of HuggingGPT in understanding and solving complex tasks from multiple modalities and domains.
Related Works
In recent years, the field of natural language processing (NIP) has been revolutionized by the emer- gence of large language models (LLMs) [II. 2. R, gL
R Io. [i, exemplified by models such as GPT-
[_]] GPT-4 [20]] PaLM [3], and LLaMa [6]] LLMs have demonstrated impressive capabilities in zero-shot and few-shot tasks, as well as more complex tasks such as mathematical problems and commonsense reasoning, due their massive corpus and intensive training computation. To ex- tend the scope of large language models (LLMs) beyond text generation, contemporary research can be divided into two branches: 1) Some works have devised unified multimodal language models for solving various Al tasks [2T] 22 23). For example, Flamingo [2]]] combines frozen pre-trained vision and language models for perception and reasoning. BLIP-2
[22]] utilizes a Q-former to harmo-
nize linguistic and visual semanties, and Kosmos-1 [23] incorporates visual input into text sequences to amalgamate linguistic and visual inputs. 2) Recently, some researchers started to investigate the
integration of using tools models in LLMs [M4] 25. M.. 27 28]. Toolformer [[2]] is the pioneer-
ing work to introduce external API tags within text sequences, facilitating the ability of LLMs to access external tools. Consequently, numerous works have expanded LLMs to encompass the visual modality. Visual ChatGPT [26]] fuses visual foundation models, such as BLIP
[29]] and Control-
Net [BO], with LLMs. Visual Programming [MI]] and ViperGPT [25] apply LLMs to visual objects by employing programming languages, parsing visual queries into interpretable steps expressed as Python code. More diseussions about related works included in Appendix
Distinet from these approaches, HuggingGPT advances towards more general Al capabilities in the following aspects: 1) HuggingGPT uses the I.LM as the controller to route user requests to expert models, effectively combining the language comprehension capabilities the LLM with the expertise of other expert models; 2) The mechanism of HuggingGPT allows it address tasks in any modality or any domain by organizing cooperation among models through the LLM. Benefiting from the design of task planning in HuggingGPT, our system automatically and effectively generate task procedures and solve more complex problems; 3) HuggingGPT offers a more flexible approach model selection, which assigns and orchestrates tasks based on model descriptions. By providing only the model descriptions, HuggingGPTT can continuously and conveniently integrate diverse expert models from AI communities, without altering any structure or prompt settings. This open and continuous manner brings us one step closer to realizing artificial general intelligence.
HuggingGPT
HuggingGPT is collaborative system for solving AI tasks, composed of large language model (LLM) and numerous expert models from ML communities. Its workflow includes four stages: task planning, model selection, task execution, and response generation, shown in Figure Given user request, our HuggingGPT, which adopts an LLM as the controller, will automatically deploy the whole workflow, thereby coordinating and executing the expert models to fulfill the target. Ihe be presents presents the detailed prompt design in our HuggingGPT. In the following subsections, we will introduce the design of each stage.
3.1 Task Planning
Generally, in real-world scenarios, user requests usually encompass some intricate intentions and thus need to orchestrate multiple sub-tasks to fulfill the target. Therefore, we formulate task plan-

---

Prompt
input, generating list of tasks with the following format: [[ I["task:: task, "id", task id, "dep": dependency_Lass ids, "args": ( ("text": text, "image": URI., "audio": URI. "video": URI.] ]]. The "dep" field denotes the id of the previous task which generates new resource upon which the current task relies. The tag <resource>>task id" represents the generated text, image, audio, or video from the dependency task with the corresponding task id. The task must be selected from the following options: If Available Task List }l. Please note that there exists a logical connections and order between the tasks. In case the user input cannot be parsed, an empty JSON response should be provided. Here are several cases for your reference: {{ Demonstrations }). To assist with task planning, the chat history is available as {{ Chot Logs }), where you trace the user- men-mentioned resources and incorporate them into the task planning stage.
#1 Task Planning Stage The Al assistant performs task parsing
Task Planning
Demonstrations
Can you tell me how many objects in el.jpg?
[[ ("task": "ohject-detection" "id": 0, "dep": : [-1], "args": ("im age : el.jpg"]]] []]
[[taas";; timage-to-txxt "id": 0, [dl)":[-1]. "args": ("im
age": "22.jng H. [ ("task": : image-cls', "id": ''dep"
11. "arge": ("image": "e2.jng" )) ("task"." "object--eecce
In e2.jpg, what's the animal and what's doing?
tion". "id": 2. "dep": [-1], "args": ("image": "e2.ipg"
II ("task": qusuallquesrion-answwwee
"id": 3. "dep": "args": ["text" "what's the animal doing?", "image": "c2.jpg" 111
First generate HED image of e3.jpg, then based on the HED image and a text "a girl reading book", create image response.
"taak"" "pose- proseddetection", "id": O, "dep": [-1]. "args": ("im age": "e3.ig" }}. "ttask: pposetext-to-mmage "id": 1, "dep": [0], "args": ("text": girl reading a book", "image": "<re-
source>-0" ]]]
Prompt
#2 Model Selection Stage Given the user request and the call command, the Al assistant helps the user to select a suitable model from list of models to process the user request. The Al assistant merely outputs the model id of the most appropriate model. The output must be in strict JSON format: ("id": "id", "reason": "your detail reason for the choice"), We have list of models for you choose from If Candidate Models )1. Please select one model from the list.
Model Selection
Candidate Models
["model_ie id": model id #1, "metadata": meta-info #1, "description". description of model #1) ("model_id", model id #2, "metadata": meta-info #2, "deseription": description of model #2)
("model_id" id": model id #K, "metadata": meta-info #K, "description": description of model #K)
Response Generation
Prompt
#4 Response Generation Stage With the input and the inference results, the AI assistant needs describe the process and results. The previous stages can be formed as User Input: {{ User Imput H), Task Planning: [[ Tasks }), Model Selection: [{ Model Assignment }}, Task Execution: (I Predictions }). You must first answer the user's request in a straightforward manner. Then describe the task process and show your analysis and model inference results the user in the first person. If inference results contain file path, must tell the user the complete file path. there nothing in the results, please tell me you can't make
Table 1: The details of the prompt design in HuggingGPT. In the prompts, we some injectable slots such as {{ Demonstrations }} and [( Candidate Models }). These slots are uniformly replaced with the corresponding text before being fed into the LLM.
ning the first stage of HuggingGPT, which aims use LLM to analyze the user request and then decompose it into a collection of structured tasks. Moreover, we require the L.M to determine de- pendencies and execution orders for these decomposed tasks, to build their connections. To enhance the efficacy of task planning in LLMs, HuggingGPT employs a prompt design, which consists of specification-base instruction and demonstration-basee parsing. We introduce these details in the following paragraphs.
Specification-bast Instruction To better represent the expected tasks of user requests and use them in the subsequent stages, expect the LLM to parse tasks by adhering to specific specifica-
5

---

tions (e.g., JSON format). Therefore, we design a standardized template for tasks and instruct the LLM conduct task parsing through slot filing. As shown in Table Il the task parsing template comprises four slots ("task", "id", "dep", and "args") to represent the task name, unique iden- tifier, dependencies and arguments. Additional details for each slot can found in the template description (see the Appendix A.I.I). By adhering to these task specifications, HuggingGPP can automatically employ the LLM to analyze user requests and parse tasks accordingly.
Demonstration-basse Parsing To better understand the intention and criteria for task planning. HuggingGPT incorporates multiple demonstrations in the prompt. Each demonstration consists of a user request and its corresponding output, which represents the expected sequence of parsed tasks. By incorporating dependencies among tasks, these demonstrations aid HuggingGPT understand- ing the logical connections between tasks, facilitating accurate determination of execution order and identification of resource dependencies. The details of our demonstrations is presented in Table II
Furthermore, to support more complex scenarios (c.g., multi-turn dialogues), we include chat logs in the prompt by appending the following instruction: "To assist with task planning, the chat history is available as |l Chat Logs JJ. where you can trace the user-mentioned resources and incorporate them into the task planning.". " Here /f Chat Logs J/ represents the previous chat logs. This design allows HuggingGPT to better manage context and respond to user requests in multi-tune turtiittummm dialogues.
3.2 Model Selection
Following task planning, HuggingGPT proceeds to the task of matching tasks with models, i.c., selecting the most appropriate model for each task in the parsed task list. To this end, we use model descriptions as the language interface to connect each model. More specifically, we first gather the descriptions of expert models from the ML community (e.g., Hugging Face) and then employ a dynamic in-context task-model assignment mechanism to choose models for the tasks. This strategy enables incremental model access (simply providing the description of the expert models) and can be more open and flexible to use MI communities. More details are introduced in the next paragraph.
In-context Task-model Assignment We formulate the task-model assignment as a single-choice problem, where available models are presented options within a given context. Generally, based on the provided user instruction and task information in the prompt. HuggingGPT is able to select the most appropriate model for each parsed task. However, due to the limits of maximum context length, it is not feasible to encompass the information of all relevant models within one prompt. To mitigate this issue, we first filter out models based their task type to select the ones that match the current task. Among these selected models, we rank them based on the number of downloads on Hugging Face and then select the top-K models as the candidates. This strategy can substantially reduce the token usage in the prompt and effectively select the appropriate models for each task.
3.3 Task Execution
Once a specific model is assigned to a parsed task, the next step is to execute the task (i.e., perform model inference). In this stage, HuggingGPT will automatically feed these task arguments into the models, execute these models to obtain the inference results, and then send them back to the LLM. It is necessary to emphasize the issue of resource dependencies at this stage. Since the outputs of the prerequisite tasks are dynamically produced, HuggingGPT also needs to dynamically specify the dependent resources for the task before launching it. Therefore, it is challenging to build the connections between tasks with resource dependencies at this stage.
Resource Dependency To address this issue, we use a unique symbol, "Kresource>", to main- tain resource dependencies. Specifically, HuggingGPT identifies the resources generated by the prerequisite task as cresource>-tasse id, where task_ id is the id of the prerequisite task. Dur- ing the task planning stage, if some tasks are dependent the outputs of previously executed tasks (c.g., task fask_id), HuggingGPT sets this symbol (i.c., cresource>-task __dd to the corresponding resource subfield in the arguments. Then in the task execution stage, HuggingGPT dynamically replaces this symbol with the resource generated by the prerequisite task. As a result, this strategy empowers HUg to efficiently handle resource dependencies during task execution. "To some extent, think the downloads can reflect the popularity and quality of the model.

---

Task Type
Diagram
Example
Metrics
Show funny image of a cat
Precision, Recall, Accuracy
Single Task
Task 1
Replace the cat with dog in example.jpg
Precision, Recall, Edit Distance
Sequential Task
Task 1
Task 2
Task 3
Given collection of image A: a.jpg. B: b.jpg, C: c.jpg, please tell which image more like image B terms of semantic, or C?
Task 1
Task 4
Precision, Recall, F1 Sco-e
Graph Task
Task
Task
Task
Table 2: Evaluation for task planning in different task types.
Furthermore, for the remaining tasks without any resource dependencies, we will execute these tasks directly in parallel further improve inference efficiency. This means that multiple tasks can be executed simultaneously if they meet the prerequisite dependencies. Additionally, we offer hybrid inference endpoint deploy these models for speedup and computational stability. For more details, please refer to Appendix A.I.3.
3.4 Response Generation
After all task executions are completed, HuggingGPT needs generate the final responses. As shown in Table_ I, HuggingGPT integrates all the information from the previous three stages (task planning, model selection, and task execution) into concise summary in this stage, including the list of planned tasks, the selected models for the tasks, and the inference results of the models.
Most important among them are the inference results, which the key points for HuggingGPT to make the final decisions. These inference results are presented bounding boxes with detection probabilities in the object detection model, answer distributions in the question-answent answering model, etc. HuggingGPT allows LLM to receive these structured inference results as input and generate responses in the form of friendly human language. Moreover, instead of simply aggregating the results, LLM generates responses that actively respond to user requests, providing reliable decision with confidence level.
structured format, such as
4 Experiments
4.1
Settings
In our experiments, we employed the 5-ttrb, text text-davinci-003 and gpt-4 variants of the GPT models as the main LLMs, which are publicly accessible through the OpenAl API To enable more stable outputs of LLM, we set the decoding temperature to 0. addition, to regulate the LLM output to satisfy the expected format (e.g.. JSON format), we set the logit_ bias to 0.2 on the format constraints (e.g., "(" and ")"). We provide detailed prompts designed for the task planning, model selection, and response generation stages in [able II where ((variable)) indicates the slot which needs be populated with the corresponding text before being fed into the LLM.
4.2 Qualitative Results
Figure ] and Figure 2 have shown two demonstrations of HuggingGPT. In Figure IL the user re- quest consists of two sub-tasks: describing the image and object counting. In response to the re- quest, HuggingGPT planned three tasks: image classification, image captioning, and object detec- tion, and launched the google/vis /viit [32]]
[B3], and [62]] models, respectively. Finally, HuggingGPT integrated the re- sults of the model inference and generated responses (describing the image and providing the count of contained objects) to the user.
f facebook/ detr-resnet-100
httts::/pllafon www.opi..c......eo

---

A more detailed example is shown in Figure 2. In this case, the user's request included three tasks: detecting the pose of person in an example image, generating a new image based on that pose and specified text, and creating speech describing the image. HuggingGPT parsed these into six tasks, including pose detection, text-to-image conditional on pose, object detection, image classification, image captioning, and text-to-speech. We observed that HuggingGPPT can correctly orchestrate the execution order and resource dependencies among tasks. For instance, the pose conditional text-to- image task had to follow pose detection and use its output as input. After this, HuggingGPT selected the appropriate model for each task and synthesized the results of the model execution into a final response. For more demonstrations, please refer to the Appendix A.3
4.3 Quantitative Evaluation
In HuggingGPT, task planning plays a pivotal role in the whole workflow, since determines which tasks will be executed in the subsequent pipeline. Therefore, we deem that the quality of task plan- ning can be utilized to measure the capability of LLMs as controller in HuggingGPT. For this purpose, we conduct quantitative evaluations to measure the capability of LLMs. Here we simpli- fied the evaluation by only considering the task type, without its associated arguments. To better conduct evaluations on task planning, we group tasks into three distinct categories (see (able and formulate different metrics for them:
LLM
Ace f
Pre f
Recall f
F1 f
Alpaca-7b
6,48
35.60
6.64
4.88
Vicuna-7b GPT-3.5
23.86
45.51
26.51
29.44
52.62
62.12
52.62
54.45
Table 3: Evaluation for the single task. "Acc" and "Pre" represents Accuracy and Precision.
Single Task refers to request that involves only one task. We consider the planning be correct if and only if the task name (i.e., "task") and the predicted label are identically equal. In this context, we utilize F1 and accuracy as the evaluation metrics.
Sequential Task indicates that the user's request can be decomposed into a sequence of multiple sub-tasks. In this case, we employ Fl and normalized Edit Distance [[5] as the evaluation metrics.
Graph Task indicates that user requests can be decomposed into directed acyclic graphs. Con- sidering the possibility of multiple planning topologies within graph tasks, relying solely on the F1-score is not enough to reflect the LLM capability in planning. To address this, following Vicuna [[6]], we employed GPT-4 as critic to evaluate the correctness of the planning. The ac- curacy is obtained by evaluating the judgment of GPT-4, referred to as the GPT-4 Score. Detailed information about the GPT-4 Score can be found in Appendix A.I.5.
Dataset To conduct our evaluation, invite some annotators to submit some requests. We col- lect these data as the evaluation dataset. We use GPT-4 to generate task planning as the pseudo la- bels, which cover single, sequential, and graph tasks. Furthermore, we invite some expert an- notators to label task planning for some complex requests (46 examples) as a high-quality human- annotated dataset. We also plan to improve the quality and quantity of this dataset to further assist in evaluating the LLM's planning capabilities, which remains future work. More details about this dataset are in Appendiix A.2. Using this dataset, we conduct experimental evaluations on various LLMs, including Alpaca-7b [7]. Vicuna-7b [36],
LLM
ED J.
Pre
Recall t
F1 1
Alpaca-7b
0.83
22.27
23.35
22.80
Vicuna-7b
0.80
19.15
28.45
22.89
GPT-3.5
0.54
61.09
45.15
51.92
Evaluation for the sequential task. "ED" means Edit Distance.
Table 4:
and GPT models, for task planning.
Performance Tables and [ show the planning capabilities of Hugging- GPT on the three categories of GPT- 4 annotated datasets, respectively. We observed that GPT-3.5 exhibits more prominent planning capabilities, outper- forming the open-source LLMS Alpaca- 7 and Vicuna-7b in terms of all types
LLM
GPT-4 Score +
Pre t
Recall t
F1 1
Alpaca-7b
13.14
16.18
28.33
20.59
Vicuna-7b
19.17
13.97
28.08
18.66
GPT-3.5
50.48
54.90
49.23
51.91
Table 5: Evaluation for the graph task.
8

---

of user requests. Specifically, in more complex tasks (e.g., sequential and graph tasks), GPT-3.5 has shown absolute predominance over other LLMs. These results also demonstrate the evaluation of task planning can reflect the capability of LLMS as controller. Therefore, we believe that develop- ing technologies to improve the ability of L.MMs in task planning very important, and we leave as future research direction.
Furthermore, we conduct experiments on the high-quality human-annotated dataset to obtain more precise evaluation. Iae ble reports the comparisons on the human- annotated dataset. These results align with the aforementioned conclusion. highlighting that more powerful L.LS demonstrate better performance in task planning. Moreover, we compare the results between human annota- tions and GPT-4 annotations. We find that even though GPT-4 outperforms other LLMs, there still remains substantial gap when compared with human annotations. These observations further underscore the importance of enhancing the planning capabilities of LLMs.
Sequential Task
Graph Task
LLM
Acc
ED |
Acc f
F1
Alpaca-7b
0
0.96
4.17
4.17
Vicuna-7b
7.45
0.89
10.12
7.84
GPT-3.5
18.18
0.76
20.83
16.45
GPT-4
41.36
0.61
58.33
49.28
Table 6: Evaluation on the numan-annotated dataset.
4.4 Ablation Study
Demo Variety
Single Task
Sequencial Task
Graph Task
LLM
(ff task types)
Ace f
ED (%) 4
F1 f
GPT-3.5
43.31
48.29
71.27
32.15
43.42
2
GPT-4
65.59
67.08
47.17
55.13
53.96
GPT-3.5
51.31
51.81
60.81
43.19
58.51
GPT-4
66,83
68.
42.20
58.18
64.34
GPT-3.5
52.83
53.70
56.52
47.03
64.24
GPT-4
67.52
71.05
39.32
60.80
66.90
Table 7: Evaluation of task planning in terms of the variety of demonstrations. We refer to the variety of demonstrations as the number of different task types involved in the demonstrations.
Single
Soquential
Graph
70-
Perfemance 196)
60-
50-
Melx
50-
Meric
40-
UM
ULV
Gma3
Gmra3
Musi:
Number of Shots
Number of Shots
Number of Shots
Figure Evaluation of task planning with different numbers demonstrations.
As previously mentioned in our default setting. we apply few-shot demonstrations enhance the capability of LLMs in understanding user intent and parsing task sequences. To better investigate the effect of demonstrations on our framework, we conducted a series of ablation studies from two perspectives: the number of demonstrations and the variety of demonstrations. Table M reports the planning results under the different variety of demonstrations, We observe that increasing the vari- ety among demonstrations can moderately improve the performance of LLMs in conduct planning. Moreover, Figure illustrates the results task planning with different number of demonstrations. We can find that adding some demonstrations can slightly improve model performance but this im- provement will be limited when the number is over 4 demonstrations. In the future, we will continue to explore more elements that can improve the capability of LLMs at different stages.

---

Task Planning
Model Selection
Response
LLM
Passing Rate f
Rationality 1
Passing Rate
Rationality
Success Rate
Alpaca-13b
51.04
32.17
6.92
Vicuna-13k
79.41
58.41
15.64
GPT-3.5
91.22
78.47
93.89
84.29
63.08
Table 8: Human Evaluation on different LLMs. We report two metrics, passing rate (%) and ratio-
nality (%), in the task planning and model selection stages and report a straightforward success rate (%) to evaluate whether the request raised by the user is finally resolved.
4.5 Human Evaluation
In addition objective evaluations, we also invite human experts to conduct a subjective evaluation in our experiments. We collected 130 diverse requests to evaluate the performance of HuggingGPP at various stages, including task planning, model selection, and final response generation. We de- signed three evaluation metrics, namely passing rate, rationality, and success rate. The definitions of each metric can be found in Appendix A. The results are reported in Table N. From Table N we can observe similar conclusions that GPT-3.5 can significantly outperform open-source LLMs like Alpaca-13b and Vicuna-13b by a large margin across different stages, from task planning to response generation stages. These results indicate that objective evaluations are aligned with human evaluation and further demonstrate the necessity of a powerful LLM as a controller in the framework of autonomous agents.
5 Limitations
HuggingGPT has presented a new paradigm for designing AI solutions, but we want to highlight that there still remain some limitations or improvement spaces: 1) Planning in HuggingGPT heav- ily relies on the capability of LLM. Consequently, we cannot ensure that the generated plan will always be feasible and optimal. Therefore, is crucial to explore ways to optimize the LLM in order to enhance its planning abilities; 2) Efficiency poses common challenge in our framework. To build such a collaborative system (i.e., HuggingGPPT) with task automation, it heavily relies on powerful controller (e.g.. ChatGPT). However, HuggingGPT requires multiple interactions with LLMs throughout the whole workflow and thus brings increasing time costs for generating the re- sponse; 3) Token Lengths is another common problem when using LLM, since the maximum token length always limited. Although some works have extended the maximum length to 32K, is still insatiable for us if we want to connect numerous models. Therefore, how to briefly and effec- tively summarize model descriptions is also worthy of exploration; 4) Instability is mainly caused because LLMs usually uncontrollable. Although LLM is skilled in generation, it still possibly fails to conform to instructions or give incorrect answers during the prediction, leading to exceptions in the program workflow. How to reduce these uncertainties during inference should be considered in designing systems.
Conclusion
In this paper, we propose a system named HuggingGPT solve Al tasks, with language as the interface to connect LLMs with AI models. The principle of our system is that an LLM can be viewed as controller to manage AI models, and can utilize models from ML communities like Hugging Face to automatically solve different requests of users. By exploiting the advantages of LLMs understanding and reasoning, HuggingGPT can dissect the intent of users and decompose into multiple sub-tasks. And then, based on expert model descriptions. HuggingGPTT is able to assign the most suitable models for each task and integrate results from different models to generate final response. By utilizing the ability of numerous AI models from machine learning communities, HuggingGPT demonstrates immense potential solving challenging Al tasks, thereby paving a new pathway towards achieving artificial general intelligence.
10

---

Acknowledgement
We appreciate the support of the Hugging Face team help us improving our GitHub project and web demo. Besides, we also appreciate the contributions of Bei Li, Kai Shen, Meiqi Chen, Qingyao Guo, Yichong Leng, Yuancheng Wang, Dingyao Yu for the data labeling and Wenqi Zhang, Wen Wang. Zeqi Tan for paper revision.
This work is partly supported by the Fundamental Research Funds for the Central Universi- ties (No. 226-2023-00060), Key Research and Development Program of Zhejiang Province (No. 8023C01152), National Key Research and Development Project of China (No. 2018AAA0101900, and MOE Engineering Research Center of Digital Library.
References
[1] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Leamers. In NeurlPS, 2020.
[2] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Chris- tiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. CoRR, abs/2203.02155. 2022.
[3] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, and others. Palm:
Scaling language modeling with pathways. ArXiv, abs/2204.02311 2022.
[4] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig. Punit Singh Koura, Anjali Sridhar, Tianlu Wang. and Luke Zettlemoyer. Opt: Open Pre-trained Transformer Language Models. ArXiv, abs/2205.01063 2022.
[5] Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. Glm-130b: An Open Bilingual Pre-trained Model. ICLR 2023 poster, 2023.
[6] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, eur'elien Ro- driguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and Efficient Foundation Language Models. ArXiv, abs/2302. 2023.
[7] Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An Explanation of In- context Learning as Implicit Bayesian Inference. ICLR 2022 Poster, 2022.
[8] Sewon Min, Xinxi Lyu. Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannanch Hajishirzi, and Luke Zettlemoyer. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2022.
[9] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022.
[10] Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasckaran, Anjana Arunkumar, David Stap,

---

Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Virendrabhai Purohit, Is- hani Mondal, Jacob William Anderson, Kirby C. Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Ro- hitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, rushang karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, Xudong Shen, Chitta Baral, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi, and Daniel Khashabi. Super- NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2022.
[11] S. Iyer, Xiaojuan Lin, Ramakanth Pasumuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, Xian Li, Brian O'Horo, Gabriel Pereyra, Jeff Wang. Christopher Dewan, Asli Celikyilmaz, Luke Zettlemoyer, and Veselin Stoyanov. Opt-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generaliza- tion. ArXiv, abs/2212. 12017, 2022.
[12] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction finettund language models. CoRR, abs/2210.1144 11416, 2022.
[13] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instruc- tions, 2022.
[14] Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe Le, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning. CoRR, abs/2301.13688 2023.
[15] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain of Thought Prompting Elicits Reasoning in Large Lan- guage Models. In Conference on Neural Information Processing Systems (NeurIPS), 2022.
[16] Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large Language Models are Zero-Shot Reasoners. In Conference on Neural Information Pro- cessing Systems (NeurIPS), 2022.
[17] Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang. Jamie Callan, and Graham Neubig. Pal: coogram-aider Language Models. ArXiv, abs/2211. 1.111..111 1.1445,, 2022.
[18] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-Consistency Improves Chain of Thought Rea- soning in Language Models. ICLR 2023 poster, abs/2203. 11111, 2023.
[19] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. CoRR, abs/2206.07682, 2022.
[20] OpenAl. Gpt-4 technical report, 2023.
[21]
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Ruther- ford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Miko- laj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning, 2022.
[22] Junnan Li, Dongxu Li, S. Savarese, and Steven Hoi. Blip-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models, ArXiv, abs/2301. 12597, 2023.
12

---

[23] Shaohan Huang, Li Dong, Wenhui Wang, Y. Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, O. Mohammed, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, and Furu Wei. Language Is Not All You Need: Aligning Perception with Language Models. ArXiv, abs/2302. 14444502..4445, 2023.
[24] Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, M. Lomeli, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language Models Can Teach Themselves to Use Tools. ArXiv, abs/2302.04761, 2023.
[25] Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execu- tion for reasoning, 2023.
[26] Chenfei Wu, Sheng-Kai Kai Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models. arXiv, 2023.
[27] Yaobo Liang, Chenfei Wu, Ting Song, Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuai Lu, Lei Ji, Shaoguang Mao, Yun Wang. Linjun Shou, Ming Gong, and Nan Duan. Taskmatrix.ar Completing tasks by connecting foundation models with millions of apis, 2023.
[28] Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, Yi Ren Fung, Yusheng Su, Huadong Wang. Cheng Qian, Runchu Tian, Kunlun Zhu, Shihao Liang, Xingyu Shen, Bokai Xu, Zhen Zhang, Yining Ye, Bowen Li, Ziwei Tang, Jing Yi, Yuzhang Zhu, Zhenning Dai, Lan Yan, Xin Cong, Yaxi Lu, Weilin Zhao, Yuxiang Huang, Junxi Yan, Xu Han, Xian Sun, Dahai Li, Jason Phang, Cheng Yang, Tongshuang Wu, Heng Ji, Zhiyuan Liu, and Maosong Sun. Tool learning with foundation models, 2023.
[29] Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. Blip: Bootstrapping Language- Image Pre-training for Unified Vision-Language Understanding and Generation. In Interna- tional Conference on Machine Learning (ICML), pages 12888- 12980, 2022.
[30] Lvmin Zhang and Maneesh Agrawala. Adding Conditional Control to Text-to-Image Diffusion Models. ArXiv, abs/2302.05543, 2023.
[31] Tanmay Gupta and Aniruddha Kembhavi. Visual Programming: Compositional visual reason- ing without training. arXiv, abs/2211..115555 2022.
[32] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. 2021.
[33] Ankur Kumar. The illustrated image captioning using transformers. ankur3107 10itttuub..t 2022.
[34] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers, 2020.
[35] A. Marzal and E. Vidal. Computation of normalized edit distance and applications. IEEE Transactions Pattern Analysis and Machine Intelligence, 15(9):926-99) 9322, 1993.
[36] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang. Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023.
[37] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-followe llama model. https://g thub. http:/tatsullabsstaaaaaa ab/saofforddd alpaca, 2023.
13

---

A
Appendix
A.1
More details
In this section, we will present more details about some designs of cach stage in HPg
A.1.1 Template for Task Planning
To format the parsed lask, we define the template [("task": task, "id", task id. "dep": depen- dency task_ "args": ("text": text, "image": URL, "audio": URL, "video": URL with four slots: "task", "id", "dep", and "args". [able presents the definitions of each slot.
Name
Definitions
"task" "
represents the type parsed task. covers different tasks language. visual, video, audio, etc. The currently supported task list of HuggingGPT is shown in lable
"id"
The unique identifier for task planning, which is used for references to dependent tasks and their generated resources.
"dep"
It defines the pre-requisite tasks required for execution. The task will be launched only when all the pre-requisite dependent tasks finished.
"args" "
It contains the list of required arguments for task exccution. It contains three subficlds popu- lated with text, image, and audio resources according to the task type. They are resolved from either user's request or the generated resources of the dependent tasks. The corresponding argument types for different task types are shown in Dable 13
Table 9; Definitions for each slot for parsed tasks in the task planning.
A.1.2 Model Descriptions
In general, the Hugging Face Hub hosts expert models that come with detailed model descriptions, typically provided by the developers. These descriptions encompass various aspects of the model, such as its function, architecture, supported languages and domains, licensing, and other relevant details. These comprehensive model descriptions play crucial role in aiding the decision of Hug- gingGPT. By assessing the user's requests and comparing them with the model descriptions, Hug- gingGPT can effectively determine the most suitable model for the given task.
A.1.3 Hybrid Endpoint in System Deployment
An ideal scenario is that we only use inference endpoints on cloud service (e.g., Hugging Face). However, in some cases, we have to deploy local inference endpoints, such as when inference end- points for certain models do not exist, the inference is cime-consuming, or network access is limited. To keep the stability and efficiency of the system, HuggingGPT allows us pull and run some common or time-consuming models locally. The local inference endpoints are fast but cover fewer models, while the inference endpoints in the cloud service (e.g., Hugging Face) are the opposite. Therefore, local endpoints have higher priority than cloud inference endpoints. Only if the matched model is not deployed locally, HuggingGPT will run the model on the cloud endpoint like Hugging Face. Overall, we think that how to design and deploy systems with better stability for HuggingGPT or other autonomous agents will be very important the future.
A.1.4
Task List
Up now, HuggingGPT has supported 24 Al tasks, which cover language, vision, speech and etc. Dable I3 presents the detailed information of the supported task list in HuggingGPT.
A.1.5 GPT-4 Score
Following the evaluation method used by Vicuna [36], we employed GPT-4 as evaluator to assess the planning capabilities of L.LMs. In more detail, we include the user request and the task list planned by LLM in the prompt, and then let GPT-4 judge whether the list of tasks is accurate and
14

---

also provide rationale. To guide GPT-4 to make the correct judgments, we designed some task guidelines: 1) the tasks are the supported task list (see lable 13); 2) the planned task list can reach the solution to the user request; 3) the logical relationship and order among the tasks are reasonable. In the prompt, we also supplement several positive and negative demonstrations of task planning to provide reference for GPT-4. The prompt for GPT-4 score is shown in Dable II We further want to emphasize that GPT-4 score is not always correct although it has shown a high correlation. Therefore, we also expect to explore more confident metrics to evaluate the ability of LLMs in planning.
As critic, your lask is to assess whether the Al assistant has properly planned the task based on the user's request. To do so, carefully examine both the user's request and the assistant's output, and then provide decision using either "Yes" or "No" ("Yes" indicates accurate planning and "No" indicates inaccurate planning). Additionally, provide a rationale for your choice using the following structure: { ("choice" "yes"/"no". "reason": "Your reason for your choice"). Please adhere to the following guidelines: 1. The task must be selected from the following options: I1 Available Task List }1. 2. Please note that there exists logical relationship and order between the tasks. 3. Simply focus the correctness of the task planning without considering the task arguments. Positive examples: ((Positive Demos) Negative examples: [Neggatvv Demos)} Current user request: ((/put)) Al assistant's output: ([Ouputt) Your judgement:
Table 10: The prompt design for GPT-4 Score.
A.1.6 Human Evaluation
To better align human preferences, we invited three human experts to evaluate the different stages of HuggingGPT. First, we selected 3-5 tasks from the task list of Hugging Face and then manually created user requests based on the selected tasks. We will discard samples that cannot generate new requests from the selected tasks. Totally, we conduct random sampling by using different seeds, resulting in a collection of 130 diverse user requests. Based on the produced samples, we evaluate the performance of LLMs at different stages (e.g.. task planning, model selection, and response generation). Here, we designed three evaluation metrics:
Passing Rate: to determine whether the planned task graph or selected model can be successfully executed;
Rationality: to assess whether the generated task sequence or selected tools align with user re- quests in a rational manner;
Success Rate: to verify the final results satisfy the user's request.
Three human experts were asked to annotate the provided data according to our well-designed met- rics and then calculated the average values to obtain the final scores.
A.2 Datasets for Task Planning Evaluation
As aforementioned, we create two datasets for evaluating task planning. Here we provide more de- tails about these datasets. In total, we gathered a diverse set of 3,497 user requests. Since labeling this dataset to obtain the task planning for each request is heavy, GPT-4 to annotate them. Finally, these auto-labeled requests can be categorized into three types: sin- gle task (1,450 requests), sequence task (1,917 requests), and graph task (130 requests). For more reliable evaluation, we also construct a human--nnotated dataset. We invite some expert annotators to label some complex requests, which include 46 examples. Currently, the includes 24 sequential tasks and 22 graph tasks. Detailed statistics about the GPT-4-annotated and datasets shown in Table [..
employed the capabilities of
dataset
A.3 Case Study
A.3.1 Case Study on Various Tasks
Through task planning and model selection, HuggingGPT, multi-model collaborative system, em- powers LLMS with an extended range of capabilities. Here, we extensively evaluate HuggingGPT
15

---

Number of Requests by Type
Request Length
Number of Tasks
Datasets
Single
Sequential
Graph
Max
Average
Max
Average
GPT-4-amootate
1,450
1,917
130
52
13.26
13
1.82
Human-annotated
22
95
10.20
12
2.00
Table 11: Statistics on datasets for task planning evaluation.
across diverse multimodal tasks, and some selected cases are shown in Figures 4 and EL With the cooperation of powerful LLM and numerous expert models, HuggingGPT effectively tackles tasks spanning various modalities, including language, image, audio, and video. Its proficiency encom- passes diverse task forms, such as detection, generation, classification, and question answering.
A.3.2 Case Study Complex Tasks
Sometimes, user requests may contain multiple implicit tasks require multi-faceted information, in which case we cannot rely on a single expert model to solve them. To overcome this challenge, Hug organizes the collaboration of multiple models through task planning. As shown in Figures & D and & we conducted experiments to evaluate the effectiveness of HuggingGPT in the case of complex tasks:
Figure demonstrates the ability of HuggingGPT to cope with complex tasks in multi-round conversation scenario. The user splits a complex request into several steps and reaches the final goal through multiple rounds of interaction. We find that HuggingGPT can track the contextual state of user requests through the dialogue context management in the task planning stage. More- over, HuggingGPT demonstrates the ability access user-referenced resources and proficiently resolve dependencies between tasks in the dialogue scenario.
Figure 7 shows that for a simple request like "describe the image in as much detail as possible", HuggingGPT decompose it into five related tasks, namely image captioning, image classifica- tion, object detection, segmentation, and visual question answering tasks. HuggingGPT assigns expert models to handle each task gather information about the image from various perspec- tives. Finally, the LLM integrates this diverse information to deliver a comprehensive and detailed description the user.
Figure X shows two GPT first performs all the tasks requested by the user by orchestrating the work of multiple expert models, and then let the LLM aggregate the model inference results respond the user.
where request can contain several tasks. In these cases, Hugging-
In summary, Hug establishes the collaboration of LLM with external expert models and shows promising performance on various forms of complex tasks.
A.3.3 Case Study on More Scenarios
We show more cases here to illustrate HuggingGPT's ability to handle realistic scenarios with task resource dependencies, multimodality, multiple resources, etc. To make clear the workflow of Hug- gingGPT, we also provide the results of the task planning and task execution stages.
Figure illustrates the operational process of HuggingGPT in the presence of resource depen- dencies among tasks. In this case, HuggingGPT can parse out concrete tasks based on abstract requests from the user, including pose detection, image captioning. and pose conditional image generation tasks. Furthermore, HuggingGPT effectively recognizes the dependencies between task #3 and tasks #1, #2, and injected the inferred results of tasks #1 and #2 into the input argu- ments of task #3 after the dependency tasks were completed.
Figure 10 demonstrates the conversational ability of HuggingGPT on audio and video modalities. the two cases, shows HuggingGPT completes the user-requested text-to-audio and text-to- video tasks via the expert models, respectively. In the top one, the two models are executed parallel (generating audio and generating video concurrently). and in the bottom one, the two models are executed serially (generating text from the image first, and then generating audio based
16

---

on the text). This further validates that HuggingGPT can organize the cooperation between models and the resource dependencies between tasks.
Figure T shows HuggingGPT integrating multiple user-input resources to perform simple reason- ing. We can find that HuggingGPT can break up the main task into multiple basic tasks even with multiple resources, and finally integrate the results of multiple inferences from multiple models to get the correct answer.
B More Discussion about Related Works
The emergence of ChatGPT and its subsequent variant GPT-4, has created a revolutionary tech- nology wave in LLM and AI area. Especially in the past several weeks, also have witnessed some experimental but also very interesting LLM applications, such as AutoGPT B AgentGPT B BabyAGI and etc. Therefore, also give some discussions about these works and provide some comparisons from multiple dimensions, including scenarios, planning, tools, as shown in Dable 12
Scenarios Currently, these experimental agents (e.g., AutoGPT, AgentGPT and BabyAGI) mainly used to solve daily requests. While for HuggingGPT. it focuses on solving tasks in the AI area (c.g., vision, language, speech, etc), by utilizing the powers of Hugging Face. Therefore, HuggingGPT can be considered as more professional agent. Generally speaking, users can choose the most suitable agent based on their requirements (e.g., daily requests or professional areas) or customize their own agent by defining knowledge, planning strategy and toolkits.
Name
Scenarios
Planning
Tools
BabyAGI
AgentGPT
Daily
Iterative Planning
AutoGPT
Web Search, Code Executor,
HuggingGPT Al area
Global Planning
Models in Hugging Face
Table 12: Comparision between HuggingGPT and other autonomous agents.
Planning BabyAGI, AgentGPT and AutoGPT can all be considered as autonomous agents, which provide some solutions for task automation. For these agents, all of them adopt step-by-step thinking, which iteratively generates the next task by using LLMs. Besides, AutoGPT employs an addition reflexion module for each task generation, which is used check whether the current predicted task is appropriate or not. Compared with these applications, HuggingGPT adopts a global planning strategy to obtain the entire task queue within one query. is difficult to judge which one is better, since each one has its deficiencies and both of them heavily rely on the ability of LLMs, even though existing LLMs are not specifically designed for task planning. For example, iterative planning com- bined with reflexion requires huge amount of LLM queries, and if one step generates error prediction, the entire workflow would possibly enter an endless loop. While for global planning, although it can always produce a solution for each user request within one query, it still cannot guar- antee the correctness of each step or the optimality of the entire plan. Therefore, both iterative and global planning have their own merits and can borrow from each other to alleviate their shortcoming. Additionally, one notable point is that the difficulty of task planning is also lincarly correlated to the task range. As the scope of tasks increases, it becomes more challenging for the controller to predict precise plans. Consequently, optimizing the controller (i.e., LLM) for task planning will be crucial in building autonomous agents.
Among these agents, AutoGPT is the main one to involve other tools for usage. More specif- ically, AutoGPT primarily uses some common tools (e.g., web search, code executor), while Hug- gingGPT utilizes the expert models of MI. communities (e.g., Hugging Face). Therefore, AutoGPT
Tools
17

---

has a broader task range but is not suitable for more professional problems, whereas HuggingGPT is more specialized and focuses on solving more complex AI tasks. Therefore, the range of tools used in LLMs will be trade-off between task depth and task range. In addition, we also note some indus- try products for I.LM applications (e.g., ChatGPT plugins 9) and developer tools (e.g., LangChain HuggingFace Transformer Agent 9, Semantic Kernels ) for LLM applications. We believe these rapid developments will also facilitate the community to explore how to better integrate LLMs with external tools.
Overall, HuggingGPT can also be considered as an autonomous agent. Compared with these exper- imental agents, which mainly use GPT models to generate solutions for user requests, HuggingGPT systematically presents a clear pipeline with four stages: task planning, model selection, task exe- cution and response generation. Such pipeline can effectively improve the success rate of solving user requests. Besides, HuggingGPT also introduces global planning strategy decompose user requests and thus accomplish task automation. Furthermore, HuggingGPT is a collaborative system, which fully utilizes the power of expert models from MI. communities to solve Al tasks and present the huge potential of using external tools. Compared with these agents, HuggingGPT allows to better solve tasks more effectively in professional areas and can be easily extended to any vertical domain. In the future, we will continue to enhance HuggingGPT with even more powerful abilities to develop a versatile autonomous agent with unlimited possibilities.
httess//gggggiiee.cccececclaa/ss a_gete
18

---

NLP > Named Entity Recognition
Visual Question Answering
Query: please recognize named entities sentence [" that is to end the state hostility, TThrrdayy's overseas edition of the People's quoted saying.
Query: Look at jsff.jpg, what the red topping?
entities sentence 'that end the state of hostility. Thursday's overseas edition of the People's Daily quoted Tang as saying." The entities recognized are PPoole's Dally' and 'Tang'. Both are organizations used dslim/bert-ban beer-baae--EE which a fine-tuned BERT model that ready to recognize named entities in the sentence.
Response:
recognized
people respectively.
model
results very accurate, score of 0.9992 for 'People's Daily' and 0.998 for 'Tang'.
NLP Text Generation
Query: please complete a sentence | COVID-19 transmitted
via droplets, air-borne,
Response: Sure, the red topping on the pizza in the Image is tomato. used visual question answering model @ "What is the red topping?" based on the image. The answer with the highest score "tomato" with score of 0.9698325955884105
have completed the sentence, the result is COVVD-19 is transmitted via droplets, air-borne, and contact @ microsoft /blogpt madel, domain- specific generative Transformer language model pre-trained large-scale biomedical literature.
Response:
dandelin/vilte (332-finttueed-vg
Depth Estimate
CV Object Detection
Query: please estimate depth ks4j.pp
Query: please detect
examples.jpo
Response:
have detected all objects the image located at
examples.jpg used the ( facebook/ /ee////eeeeee--ll-- so1eet101 model for object detection. It DETR [End-to-End Object Detection) model ResNet-1101 backbone, which known for its high image with predicted boxes and labels for each object. The complete path of the generated image is examples/d706 The objects detected in the image chair, dining table, clock, tv, book, and vase. The predicted boxes and labels for each object are also provided the inference results.
object detection tasks. generated
Response: estimated depth image
examples/ksl /js using @ Intel/dpt-large model. This
model specifica trained 1.4 million images and is
Dense Prediction Transformer (DPT) model. The generated
depth image can at J45s.jpg'
CV > Text-to-lmage
Query: give
"wide angle of castle,
blue sky background"
Sure, I can do that. For your request, used the model runwayml/stal wtwaayym/ttale-iiiiii htts://blldfffuulono.... diffussooo---ii- for text-to text-to-image task, because this model the most sultable the task as latent text-to-image diffusion model capable of generating high-quality images from text descriptions. For the text-to- image task, I used the ( wwwaaamm//sabbediiiiii diffusion-v1-5 model to generate an image from the text description provided. The generated Image located at Images/asd4.jpg
Response:
Figure 4: Case study on various tasks (a).
19

---

CV Document Question Answering
Query: look the document doc.jpg, what is the purchase amount?
Response: Sure. have used ( mmpire//ayoutmm-dommmeee
model analyze the purchase
the analysis, amount is $1,000,000,000 As for the workflow, have used deep learning model to analyze the document and extract the purchese amount. The modell used is a convolutional neural network, which trained on large dataset documents, inference results information, please let me know.
doc.jpg you provided.
CAT
$1,000,000,000.
CV Image to Image
Query: give similar image with example.jjg
Sure. For your request, used the madel @ lambdalabs/ /sddad////ddaa image-variations wariatioos--iffuser
image you provided. generated image located Timages/$373.jpg there is nothing the results, will tell that can't make there anything else I can help you with?
generate an image
Video Generation
Query: please generate a video based on "Spiderman is surfing"
Response:
that. have generated video based your prompt "Spiderman surfing". The
video is saved at the following path: "/videos/407 /40m//444"""" Task Planning: The task planning stage involved analyzing your request and identifying the appropriate task to execute. In this case, the task was text-to-vided 2. Model Selection: The model scheduling stage involved selecting the appropriate model to use the task, this case, only available model was damo- vilab/ten htt///etttttttdeeeeeee
Execution: The execution stage involved feeding "Spiderman is surfing" and generating the video. The inference result was generated video saved at the path 4076.mp4". hope this response meets your requirements.
with the input
/videos//
stock sh stock
pterstock
stckk
teersink
Audio Generation
Query: please sentence there, thousand models for dozens tasks. For more fun and creativity, have invited Diffusers family team. Feel free experience it!" for me.
empowered Huggingface family! Yes, more
Sure, can do that. For request, used the file from the text you provided. The generated audio file is located at /audios/dSf3.wav ais !|| was chosen because It the best choice the task text-to-speech as it FastSpeech model which specifically designed model
Response:
model generate audio alt
a|I Il|
The model
and has potential this answers your question. there anything else can help you with?
text-to-speech
hope
Figure 5: Case study on various tasks (b).
20

---

Task
Args
Candidate Models
Descriptions
NLP Tasks
[cardffilphhiitee--bbete bose-sentiment,..]
["This is a RoBERTa-base model trained 58M tweets ...'...]
Text-CLS
text
Seeer-bbse-NERR is a fine-tumed BERT model that is ready to... ....
Token-CLS
text
(dslim/bert-base-EER ...]
I'I you already know T3, FLAN-I5 is just better everything...
Texe2eext-Generation
text
Igoogle/flan-15xkl
"BART model pre-trained on English language, and fine-tuned..." ...]
Summarization
text
[bart-large-cm,...
[ With T5, we propose reframing all
Translation
text
[15-buse, ...]
NLP tasks into a unified..." ...l
[deepsetroberto-buse
["This the roberta- model, fine-tuned using the SQuAD2.00 0..."
Question-Answering
text
squad2, ...]
[PygmalioAA/pygmaaion
Pmmyliooon 6B proof-of-concep dialogue model based on..."....
Conversation
text
6b. ...] [gpt2.
Text-Generation
text
P"Petrriined model on English ... ...] l'Autmmatic detection of blast cells in ALL data using transformers.... ....
Tabular-CL
text
[matth/lowsormmr... ...]
CV Tasks
[ndpconnect/vit-gee http:/ittttt2-memaee captioning. ...]
["Th an image captioning model trained by @ @ddshieh in flax..." ...]
Image-to-Text
image
Irunwaym/staall-ddfffio
I"Stable Diffusion is a latent text-to-image diffusion model..." ...]
Text-to-Image
image
v1-5,
[dandelin/vitt-:33
[" "Vision-and and--anguaLe Transformer
VQA
text + image
finetuned-vag -qaa ...]
(ViLT) model fine-tuned on... ...l ["DEtection TRansformer (DETR) model trained end- end-to-end on ...".
[facebook/detr-resttt resner--eeet panoptic. ...]
Segmentation
image
limpira/aayuulm
["This is fine-tuned version of the multi-modal Layoutl M model ...".... [ ResNet model pre- pre-trained on...".
DQA
text + image
document-qu, ...]
Image-CLS
image
[microsoft/resnet-S50 ...]
hettee///stabtdiffffsieee vI o1-5-img2img. ...] (fucebookdeerr-ee resnet-500 ...]
("Stable Diffusion is a latent text-to-image diffusion model..." ....
Image-to-image
image
["DEtection TRansformer (DETR) model trained end-to-end on ......
Object-Detectie
image
delvasvielsd-conntrlle canny, ...]
("ControlNet is neural network structure control diffusion..."
ControlNet-SD
image
Audio Tasks
[espnet/kan-
["his model trained by kan-bayashi using ljspeech/ttsl recipe in..." ...]
Text-to-Speech
text
bayashi ljspeech_ vits, ...]
["This spoken language recognition model trained on the.."" ...]
Audio-CLS
audio
epaca-tdum, ...]
(jonatasgrommanvaaveeeers
""Fie-ttune XLSR 53 large model for speech recognition in English ...", ...]
ASR
audio
Large-xlsr-533 -ewer-------nee 53-english, ...]
Audio-to-Audio
Isprechbruin/metrieqqmm
[ ["MetricGAN GAN-trained model for Enhancement... ...]
audio
plas-voicebunk. ...]
Video Tasks
I'is model is hased on multi-stage text-to-video generation.."" "..iatttoo.................. ...]
Text-to-Viee Video
text
ms-1.75 .7.. ...]
(MCG-NJU/vieoomal base, ...]
["VideoMAE model pre-trained on Kinetics-400 for 1600 epochs..." ...]
Video-CLS
video
Table 13: The task list used in HuggingGPT. The first and second columns are the name and ar- guments of the corresponding task. The third and fourth columns provide some examples of the candidate models and their model descriptions.
21

---

Query #1:Please generate canny image based on example.jpe
Response #1: have generated canny image for you based on the image located at example.jpg. The task process involved selecting the @ canny-control model for image processing, as it covers common image processing functions. The inference results include generated image, which be found images/df8t.jps
Task Planning #1:
("task": "image-to-text", "Id": dep: [-1], "args": ("image": "example.jpe")
2: ("task": 'controlccany",, "Id": o, dep: [-1], "args": ("Image"; "example.jpg")
Execution Results #1:
1. ("generated text": "a herd of giraffes and zebras grazing in field")
("generated image": "images/dfät.jpg""
Query #2: then based on the above canny image and a prompt
"a photo of zoo", generate a new image.
Response #2: have received your request to generate canny image of photo of zoo. have selected the model dyyasvie/sd-controlnet.com
the best model for this task. have processed the image and generated new image located at images/ja2f.jpg, The generated image highlights the edges of the objects in the photo, making it easier see the details the animals and their surroundings. Overall, the task
was to convert a text prompt's ' photo of zoo' into canny image using the selected model.
Task Planning #2:
1: ("task": "canny-text-tor tett-----x-t--iee"" "id": 0, dep: [-1], "args": ("text":"a photo of zoo", "image": "example.jpg")
Execution Results #2:
("generated image": "images/ja2f.jpg")
Task Planning
Query #3: please detect objects in the above image
1: ('task:: 'object- depectddeeeciinn, 'dd:: 0, 'dep':
[-1], 'args': ('image:: 'images/ja2f.png")
Response #3:
Sure, here result of your command. successfully detected objects in the image you provided. The detected objects are giraffes and zebras. used the @ facebook/dett resnet-101 model for object detection, which is specifically designed for object detection tasks and has ResNet-101 backbone, which is a powerful feature extractor. Additionally, the DETR architecture used in this model is end-to-end object detection model that has shown promising results in various benchmarks. have generated image with predicted boxes around the detected objects. You can find the generated image at the following path: images/dif2.jppp The predicted objects and their corresponding
Execution Results #3:
1. f'generated image with predicted box': /lmmaess/dff2jjpe/ 'predicted': [["box:: ('xmax:: 760, 'xmin': 515, 'ymax': 437, 'ymin': 136), 'label': 'giraffe', 'score': 0.9992127418518066)) ('box': ('xmax':
561, 'xmin': 510, 'ymax': 447,
'ymin': 357),
'label': 'zebra', 'score':
0.99839043617248544 ('box': ('xmax:: 324, 'xminn 178, 'ymax': 446,
'ymin': 336),
'label': 'zebra', 'score':
follows: Giraffe with score 0.999- Zebra with score 0.998- Zebra with score of 0.998- Giraffe with score of 0.969- Zebra with score of 0.997. hope this information helpful to you.
0.99660280721822011 ('box': ('xmxx' 297, 'xmin' 4, 'ymax': 372, 'ymin': 213),
'label': giraffe' 'score::
0.66947240829467771|
Figure 6: Case study complex tasks (a).
22

---

Describe image example-jpg much detail as possible.
Query:
The image provided appears to show family four dogs playing the grass. this task, ran multiple tasks, including image-to- text, image-clasiffiattion object-detection, visual- question-answering and document-question-
Response:
answering. chose specific models for each task that were best suited addressing your query, based on their training and performance characteristics.
For image-to-text, chose the nipconnect/vit gpt2-image-captoonn model. This model generates image caption with the given image. applied this model to the input image and the developed output grass
Task Planning:
1. ("id": "0", "dep": [-1], "task"; "image-to-text",
"args": ("Image": "example.jog" }},
2. ("id": "1", "dep": [-11, "task": "Image-clasiffiattoon"
"args": ("image": "exampll
was: family four dogs are playing For maggeclassification, selected the @ google/vit-
3. ("Id": "2", "dep": [-1], "task": "object-detectioon,,
mase-patchh6-222 model. This model is trained natural images dataset and
"args": ("Image": "example.jpg
predict the label of the image output. applied this model to the image and get the results showing the risk of each label. It shows the highest risk at "Rhodesian ridgeback" with score of 93.8% For object-detection, used facebook/detr- resnet-101 model. This model is trained on COCO panoptic segmentation for object detection. I used this model generate image file with the predicted box. The generated image with predicted box is at:
4. ("id": "3", "dep": [-1], "task": "image-
segmentation", "args": ("image": "example.jpg" }),
5. ("id": "4", "dep": [-1], "task": "visual-question-
answering", ("textt""" ("text": "describe this image detail", "image": "eexamplejpgg" }),
Execution Results:
("generated ttxt": family of four dogs playing in the grass ")
2. [('score: 0.938717240055603, 'label': 'Rhodesian
ridgeback'), ('score': 0.016522558055773716 'label':
mamaage/sdd4.jjpp
'boxer'), ('score': 0.0063816699547408881,, 'label':
Regarding your request for image segmentation, have completed task using a pre-trained model -itt-ressnet----popptiii The
'Great Dane'), ('score' 0.00624415311166664
'label': 'vizsla, Hungarian pointer'), ('score':
called @ segmentation mask shows the different regions of the image that belong different classes, this case, the model has identified two main classes in the image, which are "dog", "grass" and "tree". The generated image with the segmentation mask can be found /images/t4s1.jpg
0.009944834990669291, 'label': 'bull mastiff))
[["generated image with predicted box": "/images/
sd48.jpg", predicted": (""ox" :"xmax": "xmin":
373, "ymax": 267, "ymin": 199), "label": "bus", "score": 0.9981155395507882),, ("bax": ("xmax": 292,
"xmin": 124, "ymax": 224, "ymin": 175), "label":
"airplane", "score": 0. 0.9983693188332215 ("box":
For visual question answering, used dandelin/
("xmax": 554, "xmin": 487, "ymax": 277, "ymin": 226), "label": "person", "score": 0.99108362197875981]
wit.-----iieuuuddaaaaa model. This model is a Vision-
and-Language Transformer (ViLT) model, which is fine-tuned VQAv2. suitable for visual- question-answering tasks, and is likely to provide accurate results for the given image. For the question
4. (" ("generated image with segmentation': "/images/ t4s1.jpg", "predicted":: [("score": 0.989, "label": "grass"), ("score": 0.999, "label": "dog"), ("score":
0.999, "label": "tree") 0.999, "label": "dog"))
what the image?" It reply with prediction "dogs" the confidence 84.9%.
5. [('answer': 'doogs 'score' 0.4440098888952443399955 ('answer': 'dog', 'score': (ccore00000156111550000000
Figure 7: Case study complex tasks (b).
23

---

Query: Can you tell me how many objects the picture example1.jpg"
There are objects in the picture. used the model bcebbokk/ett-resnet-bD for this task which is trained popular object detection dataset called COCO, and has the highest number of downloads. generated an image with predicted boxes that available under the path:
Response:
As for image captioning, selected which is more frequently used works better in this case. generated from the image is a large jetliner sitting on top of an airport tarmac.'
the model @ poaonneet/it-gpt--iaa---ee
Task Planning:
("task": object-detection", "id": 0, dep: [-1], "args": ("image": "example1.jpg") IB,
2: ("task"; "image-to-text", "id": 1, dep: [-1], "args": ("image":
"example1.jpg"))
Execution Results:
1. [["generated image with predicted box": "/images/79f2.jp"" "predicted": [("box": ("xmax": 463, "xmin":
373, "ymax": 267, "ymin": 199;, "label": "bus", "score": 0.99811553955008122,, ("box": ("xmax": 292, "xmin";
124, "ymax": 224, "ymin": 175), "label": "airplane", "score": 0.99836093118333115), ("box"; ("xmax": 554,
"xmin": 487, "ymax": 277, "ymin": 226), "label": "person", "score"; (911:999999111211777998881111111 ("box": ("xmax":
230, "xmin": 219, "ymax": 264, "ymin": 227). "label": "person", "score": 0.9985797405222922, ('box":
("xmax": 361, "xmin": 330, "ymax": 231, "ymin": 220), "label": "truck", "score": 03060088000088888333335)
("box": ("xmax": : 639, "xmin": 460, "ymax": 420, "ymin": 142), "label":"bus"" "score": 0.9987366795338856
("box": ("xmax": 363, "xmin": 279, "ymax": 222, "ymin": 192), "label": "airplane", "score":
0.9976897236885559, ("box": ("xmax": 317, "xmin": 287, "ymax": 230, "ymin": 219), "score": 033391888811180032888)81
"label": "truck",
["generated textt: "a large jetliner sitting on top an airport tarmac "]
Query: In example2.jpg, what's the animal and what's it doing?
Response: Sure, the animal in the image tiger and it's laying down. performed 3 tasks including image-to-text texxt, image-classfficttion, and on the image you provided. To accomplish this,
visual-question sunuullgsstiionnnseeett selected models @ poonnnct/vitgggt-imaagee
@ google/
vit-base-patchh6-22 and dandelin/vilt-ale /vilt-ba2-finetuned--o because
they are the perfect models this use case respectively. According to the results, have 88% confidence that the animal is a tiger and also have 77.8% confidence that it's laying down. The generated image with predicted box stored the path: /lmages/d59a.jpg.
Task Planning:
1: ("task": " "Id": 0, dep: [-1], "args": : ("image": "example2.jpg" }), 2: ("task"; lussual-uusstion-nnssteae
"id": 1, dep: [-1], "args": ("text"; "What is the animal doing?",
"image": "xxample2.jpg" }}
("task":"imape to-text" "id": 2, dep: [-1], "args": ("image": "example2.jpg"
Execution Results:
[('score': 0.8882273999330139 'label': 'tiger, Panthera tigris'), ('score': 0.109405115049099555 'label':
'tiger cat'), ('score': 0. 0.0062216465591022588,,, 'label': 'jaguar, panther, Panthera onca, Felis onca'), ('score':
.0000262701258994699, 'label': 'dhole, Cuon alpinus'), ('score': 0.00030842573331957787 of beasts, Panthera leo')]
'label': 'lion, king
['answer': 'sitting 'score': 0.8294351696968079, ('answer': 'laying down', 'score': 0. 0.78879253867788833,
('answer': 'resting', 'score': 0.22247001252554)
('answer': 'laying', 'score': 0.61143316811118555), f'answer':
'relaxing', 'score': 0444700357950332644)
3. ('generated_tewe text': 'a brown and white striped zebra laying rock')
Figure 8: Case study on complex tasks (c).

---

Query: based the image of example1.jpe and the description of example2.jpg please generate new one.
Response: Based the pose image of example1.jpg and the description of example2.jpg, have generated a new image s5f8.jpg for you. The process generating image involved using the openpose-contes conpooo model to extract the pase information from the basketball image, and then using the nipconnect/ wwt.-ppt--mmge-captteee the Illyasviel/ htt.:/ootornneetoopppooooo model combine the pose information the description generate the final image. The complete path the generated image s5f8.jpg. you have any other questions, please let me know.
to generate description of the boy in the image. Finally, used
Task Planning:
1: ('task:: 'pose-detection", 'id': 'dep': [-1], 'args': ('image":
'example1.jpg")
2. ('task': 'image-to-text', 'id': 1, 'dep': [-1], 'args':
('image': 'example2.jpp'')
("task": "pose-text-to-imagee" "id": 2, "dep": [0, 11, "args": ("text":" "<resoure>>>" "image": "<resource>-0"))
Execution Results:
<resource>-0
Illyasviel/ sd-controlnet- openpese
young boy is riding bike with basket
<resource>-1
Figure Qualitative analysis of multi-mode cooperation with resource dependency.
25

---

Query: generate a video with the title "An astronaut is walking in space" and dub it.
Sure. have generated video with the title 'An astronaut walking in space' and dubbed it. The workflow used is follows: First,
Response:
the text-to-video model' damoo-viabbtextttovvieee
1.7b' to generate video from the text. The inference result is the generated video /videos/5696.mpp mpp4 Second, used the text-to-speech model' facebook/tts transformer-es-esste
generate audio from the text. The inference result the generated audio muuios/3ff9..ff Finally, combined the video and audio to generate the dubbed video. hope this answers your question. Is there anything else can help you with?
1/1
11
1ll
oft
All all olt Alp 1ll 1lp olt App 1lt
alp
Alt
dlp ale
4lp
1/1
110
110
a/t
alt
Task Planning:
1: ("task": "text-to-video", "id": "dep": [-1], "args": ("text": : "An astronaut is walking space" })
2: ("task"; "text-to-speech" "id": "dep": [-1), "args": ("text": "An astronaut is walking space" }}
Execution Results:
("generated video": "/videos/5696.mpp""
("generated audio": "/audios/319e.flaa")
Query: read the image example.jpg for me.
INVOICE NO : 18301/102/T0305
Response: Sure. have
image example-jpg you. The inference result 'INVOICE NO
18301/102/ 15301/102/T0305'. The model used for this task is microsoft/ corossofttttorr-rooo (trocr-base-peee base--rriiteedd which is best suited for the task of image-to-text it fine-tuned the SROIE dataset which is specifically designed for this task.
Then have converted text audio, the generated audio
laudios/dass.we c( all alt the model used this task @ facebook/ htt///fssseee--eeemmee -n-sseeeehh,, which FastSpeech text-to- speech model which is suitable for the task of text-to-speech. Is there anything else can help you with?
Task Planning:
("task": "image-to-texx" "Id": 0, "dep": : [-1], "args": ("Image": "example.jpe"""""
2: ("task"; "text-to to-tto--ppee to-speech", "Id": 1, "dep": [0), "args": ("text"; "<resource-0>")
Execution Results:
("generated text": "INVOICE NO 18301/102/100055 ("generated audio": "/adiodd//audossddsss.aall
Figure 10: Qualitative analysis of multi-model cooperation on video and audio modalities.
26

---

Query: Given collection images /examples/a.jpg B: /examples/b..jp C: /examples/cc.jpg please tell me how many zebras in these pictures?
C, are of zebras. determine this, I first used an image-to-text model to generate captions for each image. This model, called @ captions, was specifically designed for image captioning and trained by determine that image contained cat and image contained two zebras. Next, used an object detection model called 3B identify and locate any zebras in the images. This model has BesNet-101 backbone which is known for high accuracy in object detection. The model generated images with predicted boxes for each zebra it identified, and these images can be found bel1.jpg. Finally, used text classification model called @ cardiffn /witter-xlm-robertaa
Response: the collection of images A, @
@ydshleh.
to
/images/ following locations: w //mages/9831.jp,
sentiment to analyze the generated captions and predicted boxes to confirm the presence of zebras in the images. This model multilingual XLM-roBERTa-bas model that trained on sentiment analysis. The model confirmed that there were indeed zebras present the collection of images A, B, and
Figure 11: Qualitative analysis of multi-mode cooperation with multiple sources.
27


Reviews:
[{"review_id": "7J2G1Ip6Rf", "reviewer": "Reviewer_hS4Z", "summary": "This paper presents a framework that uses LLM as controller over modularized and specialized task models to plan and execute a complex task. The approach is to prompt LLM to decompose a given task command into a execution DAG, and for each step, parse model specifications (as metadata expressed in HuggingFace model cards) and select according modularized models for execution, and finally summarize a response to give it to the user. The idea is on the line with recent works on LLM-based planning and tool using.", "questions": "Please see my general comments.", "limitations": "Please see my general comments.\n\n--------------\n**This section summarizes my concern based on rebuttal.**\n\nThere is a very simple baseline this paper needs to compare with. That is a simple one-pass reasoning by LLM without any implicit or explicit planning or model selection. The author so far refuses to do such ablation, therefore I am not convinced that the approach is effective. The proposed framework is indeed interesting, but an interesting thought remains a thought without proven effectiveness.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "I think the idea is rather novel. It aligns with recent works on using LLM as a central component to query more specialized models to complete a complex task. This paper proposes to exploit the vastly available models hosted on Hugging Face in a combinatorial way. If things work out well, it could have a substantial application impact.", "weaknesses": "The biggest selling point of this paper, as is repetitively mentioned in the paper itself, is the planning. But compared to recent works, the planning strategy in this paper is actually rather simple. Existing works on planning often involves an iterative process where LLM plans, executes, and observes, and improves (for instance, Reflexion, AdaPlanner, Self-Refine, etc). In this paper, it's just plan and execute. So, on the planning part, I do not see any contribution. Maybe, compared to the planning, a bigger contribution of this paper is the task decomposition.\n\nExperiments are very limited.\n- Data scale is quite small (46 trace annotations).\n- No planning baseline, no ablation study, and no insight about model interplay.\n- Comparisons are all on different LLMs; this has little to do with the claimed contribution on planning.\n\nA simple baseline can be directly using specialized models. I think even if in some cases the proposed HuggingGPT does not outperform, it still gives reader a good picture about the pros and cons. An immediate ablation study I can think of is why not merge the model selection and task planning into one step, or what is the impact of model selection, especially given the large number of models on Hugging Face. I see no clue in this paper.\n\nIt is not clear how the proposed approach rely on the few-shot demonstration. Prior works on planning mostly rely on few-shot in-context prompting. It seems this paper is also on this technical line. But such dependency is also a limitation in the general use case. It's also no clear about the variance of the few-shot prompt used, e.g., whether the few-shot examples are fixed, and what about their diversity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. Below are our responses to your concerns:\n\n***Q1: The biggest selling point of this paper is planning but the planning strategy in this paper is actually rather simple.***\n\nThanks for your question. Below are our answers: \n\n- First, we want to highlight that while planning is a crucial stage in our framework, the biggest contribution of our paper is to present how to utilize LLMs to coordinate multiple AI models from ML communities (e.g., Hugging Face) for solving AI tasks. And to the best of our knowledge, this is the first work to build connections between LLMs and AI models from ML communities.\n- Second, existing works (e.g., Reflexion, AdaPlanner, Self-Refine) for planning are mainly designed for solving language tasks (e.g., reasoning, coding, and etc). But the planning in our framework is to use LLMs for analyzing user requests and thus connecting external models for solving AI tasks. To fulfill this target, we introduce task decomposition with well-designed prompts plus demonstrations to instruct LLMs to conduct task planning. Therefore, essentially, our planning strategy is well designed and not simple, and the goal of our planning is quite different from existing planning works. \n\n***Q2: Experiments are limited.***\n\nThanks for your question. Below are our answers:\n\n- **Data scale is quite small**: Our dataset is composed of GPT-annotated (3,497 cases) and human-annotated (46 cases) data (please see Appendix A.2). As aforementioned, our work is the first attempt to connect LLMs with the ML community. As a result, there are no existing datasets for reference and we need to construct datasets by ourselves, which is very difficulty. Until now, only a few works (e.g., Gollria [1], ToolLLM [2]) released some datasets, but all of them are constructed by LLMs. However, we will try our best to build more high-quality data to further facilitate the investigation of this direction.\n  \n- **No Planning baseline and ablation study**: As aforementioned, the motivation of our paper is to present how to connect various AI models by using LLMs, not just for designing a planning strategy. And most of the existing planning works are for solving language tasks, while ours is to use LLMs to conduct task planning and thus invoke other models for solving tasks. We also give a discussion about planning in Appendix B. Besides, we have added more ablation studies (Please see Rebuttal PDF).\n  \n- **Comparisons are on different LLMs, not for planning**: The experiments on different LLMs are used to highlight the importance of planning in our framework, and planning could also be a critical metric to reflect the capability of LLMs. Besides, we also conduct a discussion (Appendix B) with other works (e.g., AutoGPT) in different dimensions, including planning. \n\nBesides, we also add more experiments, ablation studies, and human evaluations about other modules (Please see Rebuttal PDF).\n\n[1] Gorilla: large language model connected with massive apis\n\n[2] ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs\n\n\n***Q3: Why not merge the model selection and task planning into one step?***\n\nThanks for your question. We think it is necessary to keep the model selection as an individual step and give these reasons:\n\n1. **Motivation**: First, task planning helps us to analyze user request and decompose it into a task sequence, while model selection is to answer which model is suitable for this task. As a result, the target of each stage is different and both of them play very important roles in our framework. \n\n2. **Efficiency**: Sometimes, the number of models from ML communities could be massive. Since we need to use model descriptions to call the most suitable model, selecting models will bring more token consumption in prompts. As a result, combining task planning and model selection will cost more tokens within one query, and separating these two steps will be more friendly for LLM computations.\n\n3. **Flexibility and Extensibility**: Furthermore, keeping task planning and model selection as two parts can provide better flexibility and extensibility: 1) the model library is usually dynamic since developers will continually add the latest models into the community. Therefore, we expect that the model selection is independent of task planning so that it is easy for us to update the model library; 2) some models could include more detailed granularity (e.g., specific domains). These settings necessitate the development of more flexible selection strategies.\n\nTherefore, we prefer to keep model selection and task planning individually to help us to develop each stage. We will also add these descriptions to our final version to better explain the design of our framework.\n\n***Q4: How does the proposed method rely on the few-shot demonstration? And the variance of the few-shot prompt used?***\n\nThanks for your question. We utilize the few-shot demonstration to instruct LLMs to understand user intent, task dependency, and how to parse task sequence. By introducing few-shot demonstrations, LLMs is able to produce better results by following our instructions. To further investigate the effect, we respectively conduct experiments to investigate the effect of the number of demonstration examples (0-5). The results are reported in Figure 1 (Please see Rebuttal PDF). We can find that the planning capabilities of LLMs can obtain significant improvements by introducing few-shot demonstrations.\n\nBesides, we also conduct experiments to investigate the variance of demonstration examples. Here, we test different numbers of task types involved in the demonstrations to measure the variance. The results can be found in Table 1 (please see Rebuttal PDF). We observe that increasing the variance of demonstration could also improve the planning capability of LLMs."}, {"review_id": "GwpDHEpHA5", "reviewer": "Reviewer_FNMr", "summary": "This paper considers large language models (LLMs) like ChatGPT as a controller and presents a new framework called HuggingGPT, which connects various AI models in the existing ML community (i.e., HuggingFace). Specifically, HuggingGPT consists of four steps including task planning, model selection, task execution, and response generation. By leveraging the strong capability of LLMs and numerous AI models in different modalities, HuggingGPT can solve sophisticated AI tasks and achieve promising results.", "questions": "I have included my questions in the weaknesses part.", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1) The idea to connect LLMs with rapidly developing ML communities like HuggingFace is novel. It largely extends the applicability of LLMs to solve multi-modality AI tasks by fully utilizing the existing powerful models in HuggingFace.\n2) Each step of HuggingGPT is well designed from Figure 2. The whole paradigm of HuggingGPT is neat and effective.\n3) This paper is well written and easy to follow.\n", "weaknesses": "1. In Section 3.1, the demonstration examples may have an important impact on the parsing performance. The authors should provide more details such as the number of demonstration examples and the method to select these examples. The demonstration case provided in Table 1 is somewhat confusing for me because it only involves the tasks about images and texts. Can these demonstration examples benefit the parsing of tasks in other modalities like audio? I also wonder whether the demonstration examples for each user request are the same.\n\n2. In Section 3.2, the authors propose a model selection strategy based on in-context alignments and the number of downloads. But in my view, the contribution of this module is questionable. Since the performance gap between different models for each task may be significantly large due to model scales (e.g., GPT-2 and LLaMA for text generation), it’s nearly impossible to select weaker models to dealing with the corresponding task. Thus, I’m curious about the performance if we directly use the best model for each task. There are also no empirical results to show the necessity of model selection.\n\n3. The experimental result is somewhat weak in terms of the following points:\n\n(1) The authors only conduct the empirical analysis on task planning. Other modules should be also tested individually. \n\n(2) The authors only use automatic evaluation metrics to measure the model performance. However, even GPT-4 score may have potential biases in the evaluation of generated texts. It’s better to involve human evaluation to make experimental results stronger.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. Below are our responses to your concerns:\n\n***Q1: More details about demonstration examples.***\n\nThe number of demonstration examples is set as 3 in our default settings. Here, we select demonstration examples that contain more tasks and complex task dependencies, and simultaneously use fewer tokens in prompts. The target of using these demonstration examples is to expect LLMs to understand how to parse the task queue. Therefore, even if these examples only involve text and image tasks, our framework will still make correct predictions on audio tasks. And currently, the demonstration examples are set as the same for the user.\n\nFurthermore, we also conduct different experiments to investigate the effects of demonstration examples, from the perspectives of variance and the number of demonstrations. Here, we deploy different numbers of task types involved in the demonstrations to measure the variance in demonstration examples. The results can be found in Figure 1 (please see Rebuttal PDF) and Table 1. We found that increasing variance or the number of demonstrations can both improve the planning capability of LLMs. In the future, we will continue to investigate more elements to explore their effects on our framework.\n\n| **Demo Variance** | **LLM** | **Single Task**    |  | **Sequential Task**   |  | **Graph Task** |  |\n|:------|:----|:-----|:---------|:-----|:--------:|----|-----|\n|                                       |         | **Acc** ↑ | **F1** ↑  | **ED (%)**  ↓  | **F1** ↑  | **F1**    ↑    |\n| 2                                     | GPT-3.5  | 43.31      | 48.29     | 71.27        | 32.15     | 43.42         |\n| 2                                     | GPT-4    | 65.59      | 67.08     | 47.17        | 55.13     | 53.96         |\n| 6                                     | GPT-3.5  | 51.31      | 51.81     | 60.81        | 43.19     | 58.51         |\n| 6                                     | GPT-4    | 66.83      | 68.14     | 42.20        | 58.18     | 64.34         |\n| 10                                    | GPT-3.5  | 52.83      | 53.70     | 56.52        | 47.03     | 64.24         |\n| 10                                    | GPT-4    | 67.52      | 71.05     | 39.32        | 60.80     | 66.90         |\n\n*Table 1. Evaluation of task planning in terms of the diversity of demonstrations. (Same as Table 1 in Rebuttal PDF.)*\n\n***Q2: Can we directly use the best model for each task? The necessity of model selection?***\n\nYes. The motivation of model selection is to answer which model is suitable for our task, and expect us to determine a selection strategy. Here, directly selecting the best model can be considered as an available strategy. However, \"the best model\" is difficult to define, since it may involve too many elements. For example, which datasets, metrics and domains to measure the best model. And when involving some domain-specific requests (as shown in Table 3 of Rebuttal PDF), e.g., creating anime-style images, generating protein sequences, etc., it is difficult to cope with such scenarios by directly selecting the model with the best performance.\n\nTherefore, we expect to fully utilize the provided information (e.g., model structure, training data, applicable domain, performance, download, and etc) in the model card to more comprehensive support for model selection. Besides, as mentioned in our comments (Q1) to all reviewers, keeping model selection individually can also enable better efficiency, flexibility, and extensibility. In the future, we will continue to explore better strategies for model selection.\n\n***Q3: Other modules should also be tested individually.***\n\nThank you for your suggestions. In the initial version, we mainly evaluate our method on task planning since this module is very important to determine the subsequent results. Following your suggestions, we have added more experiments to evaluate demonstrations (please see Q1), and human evaluation on model selection and response generation (please see Q4). We found that the capability of LLMs could also affect the results of model selection, so how to optimize model selection is also very important. In the future, we will conduct more experiments to investigate more elements in our framework.\n\n\n***Q4: Involve human evaluation.***\n\nThanks for your suggestions. We further collected 130 diversity requests and have conducted human evaluations to better measure the planning capability and final generation of LLMs in our framework. Here, we design passing rate, rationality and success rate to evaluate different stages:\n\n- Passing rate: Check LLMs whether can understand instructions and execute in the current stage. \n- Rationality: Check whether the generated task sequence / selected tools are rational when compared with user requests.\n- Success rate: Check the final results whether satisfy the user request.\n\nWe invite three human experts to annotate the provided data based on our designed metrics, and then calculate the average value as the final score. The results of human evaluation are reported in Table 2 (please see Rebuttal PDF). These results can further validate the effectiveness of our proposed framework. We will continue to design more reasonable metrics for evaluation and add these results into our final version. \n\n\n| **LLM** | **Task Planning** | | **Model Selection** | | **Response Generation**  |\n|-|-|-|-|-|-|\n|              | **Passing Rate** ↑  | **Rationality** ↑  | **Passing Rate** ↑  | **Rationality** ↑  | **Success Rate**↑     |\n| Alpaca-13b   | 51.04      | 40.26    | -   | -     | 6.92    |\n| Vicuna-13b   | 79.41      | 66.15             | -  | -                | 15.64    |\n| GPT-3.5      | 99.42      | 91.03             | 93.89       | 84.29    | 63.08     |\n\n*Table 2. Human Evaluation. We design passing rate, rationality and success rate as the subjective metrics and invite human experts to evaluate them. (Same as Table 2 in Rebuttal PDF.)*"}, {"review_id": "dR77G6n0kk", "reviewer": "Reviewer_sGd8", "summary": "The authors propose HuggingGPT , a collaborative system for solving AI tasks, which is composed of a large language model (LLM) and numerous expert models from ML communities. They provide methods for each of the four stages involved in HuggingGPT's workflow: task planning, model selection, task execution, and response generation.", "questions": "No questions.", "limitations": "Yes, the authors list the limitations of their work as a separate section.", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Impactful and well written paper\n- simple strategy for handing resource dependencies for executing tasks\n- compelling idea of chaining of expert models to provide a tool to decompose a task into sub-tasks and identify the appropriate expert models to solve these sub-tasks\n- reasonable format for inputting task request along with examples from the user\n- robust evaluation  - human evaluation along with automated evaluation", "weaknesses": "I don't see any weaknesses in the experiments, evaluation or novelty of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your positive comments. We will continue to refine our paper and devote more effort to the subsequent works to facilitate the community to better understand and explore this new research direction."}, {"review_id": "7LEmo29Kik", "reviewer": "Reviewer_WQcy", "summary": "This paper presents a pipeline to manipulate many autonomous agents (mainly open-sourced models in  Hugging Face) . Tother with these models could solve NLP, CV, audio and Video tasks,  the resulted HuugingGPT could could complicated multi-modal tasks that might be decomposed a sequence  of atomic  tasks or a graph. \n", "questions": "- It seesm the authors manually filter top-K models based on the number of downloads. What is the exact number of models in each task? and how do we select the best model among the top-K models? based on which criterion?\n\n\n- It seems that it uses different evaluation metrics in table 3,4, and 5? Is there any explaination?  Can we use some straightforward metrics like  task success rate in Table 4 and 5? \n\n- We do not know quality of the GPT-4 anotated data during evaluation. Is it possiable to invite experts to manually check some of them.  Don't you think reporting results in human-annotated and GPT-4 anotated data separately could be a better idea.\n\n\n- Will these evaluation examples  and evaluation scripts be open-sourced? ", "limitations": "- Evalution is weak", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The phylosophy is interesting and insightful. I like the idea very much.\n- This seems be a promising direction to solve multi-modal tasks using HuggingGPT.\n", "weaknesses": "- The whole pipeline seems to provide a series of prompts to solve some combined tasks. The method is not scientific from a traditional point of view.\n- The evalaution protocol seems not mature.  For example, there is not evidence to check whether the evalution makes sense or not.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. Below are our responses to your concerns:\n\n***Q1: The method is not scientific from a traditional point of view.***\n\nThanks for your question. The success of our framework benefits from the advent of powerful LLMs (i.e., ChatGPT or GPT-4). Therefore, we adopt a series of prompts as a workflow to implement each step and establish the connection between each stage. By introducing these well-designed prompts,  our framework exhibits powerful capabilities in autonomously analyzing user requests and selecting suitable models for solving tasks. This method also verifies the feasibility of using LLMs as the controller to manage AI models. With the advent of more powerful open-source LLMs (e.g., LLaMA-2), we will introduce more advanced learning strategies into our framework.\n\n***Q2: What is the exact number of models in each task? How do we select the best model? Based on which criteria?***\n\nThanks for your question. Here, we filter out the Top-K models based on the number of downloads (lines 167-168). Due to the limitation of prompt length, we set the number of candidate models for each task as 20 (i.e., the value of K). And for each task, its original number of models before filtering is usually varying (the average number is nearly 60 but some tasks only have few models). After that, we do not set up any fixed criteria but let LLMs automatically select the best model from the Top-k models by providing these model descriptions, which include its downloads, domain, task type and etc. We think that LLMs could select the best suitable models after comprehensively considering these information. We will add these details to our final version.\n\n***Q3: Explanations about different metrics in Table 3, 4, and 5? Can we use some straightforward metrics like task success rate in Table 4 and 5?***\n\nGenerally, we think any user request can be converted as a directed acyclic graph (DAG). However, directly evaluating DAG is quite difficult. Therefore, based on the complexity, we divide user requests into three types (i.e., single task, sequential task, and graph task). We first use Recall, Precision, and F1-score as the uniform metrics. But for complex tasks (sequential or graph tasks), only these metrics cannot reflect the planning capability in task dependency. Hence, we additionally use [edit distance](https://en.wikipedia.org/wiki/Edit_distance) to estimate the matching degree in sequential tasks. And for the graph task, since it is very difficult to measure the matching degree of graph data, we currently adopt GPT-4 as a subjective metric to measure its performance.\n\nAnd thanks for your suggestions. We have conducted human evaluations to better measure the planning capability and final generation of LLMs in our framework. Here, we design passing rate, rationality and success rate as the subjective metrics to evaluate different stages:\n\n- Passing rate: Check LLMs whether can understand instructions and execute in the current stage. \n- Rationality: Check whether the generated task sequence / selected tools are rational when compared with user requests.\n- Success rate: Check the final results whether satisfy the user request.\n\nWe invite three human experts to annotate the provided data based on our designed metrics, and then calculate the average value as the final score. The results of human evaluation are reported in Table 2. These results can further validate the effectiveness of our proposed framework. We will continue to design more reasonable metrics for evaluation and add these results into our final version. \n\n| **LLM** | **Task Planning** | | **Model Selection** | | **Response Generation**  |\n|-|-|-|-|-|-|\n|  | **Passing Rate** ↑  | **Rationality** ↑  | **Passing Rate** ↑  | **Rationality** ↑  | **Success Rate**↑     |\n| Alpaca-13b   | 51.04      | 40.26    | -   | -     | 6.92    |\n| Vicuna-13b   | 79.41      | 66.15  | -  | -  | 15.64    |\n| GPT-3.5      | 99.42      | 91.03 | 93.89   | 84.29    | 63.08     |\n\n*Table 2. Human Evaluation. We design passing rate, rationality and success rate as the subjective metrics and invite human experts to evaluate them. (Same as Table 2 in Rebuttal PDF.)*\n\n***Q4: Reporting results in human-annotated and GPT-4 annotated data separately.***\n\nThanks for your suggestion. Actually, we have separated results into human-annotated and GPT-4 annotated data.  Table 3, 4, and 5 reported results on GPT-annotated data (in single, sequential, and graph tasks). We will make a more clear description of this part in the final version. Furthermore, we also invite human experts to conduct human evaluations for our experiments to further verify our results. The results are shown in Table 2 (please see Rebuttal PDF).\n\n***Q5: Will these evaluation examples and evaluation scripts be open-sourced?***\n\nYes. Currently, we are inviting the labeling team to conduct human evaluation and help us to iteratively clean up the whole dataset. We will release all evaluation examples and the evaluation scripts in the final version. "}, {"review_id": "BEPr6YnExY", "reviewer": "Reviewer_FvPX", "summary": "The paper studies an interesting and important question, i.e., how to automate LLMs to call existing models for solving specific tasks. The authors propose a novel framework that contains the following steps: (1) task planning, (2) model selection, (3) task execution, and (4) response generation. The experimental results well support the claim. By leveraging ChatGPT and abundant AI models Hugging Face, the proposed method is able to cover numerous sophisticated AI tasks in different modalities and domains and achieve impressive results in many tasks.", "questions": "For the model selection part, what if the number of models hosted on Huggingface becomes extremely large? How to ensure that the model selects the most appropriate one from them?", "limitations": "NA", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The paper is well-written and easy to follow.\n\n+ A novel idea proposed and well-supported experiments conducted.", "weaknesses": "- The method heavily relies on the existence of SOTA LLMs (e.g., ChatGPT), which may involve some practical issues (e.g., unaffordable API costs). It remains under-explored whether open-source LLMs (e.g., vicuna) can be leveraged for the framework (or how to adapt vicuna for better task planning).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. Below are our responses to your concerns:\n\n***Q1: Whether open-source LLMs can be leveraged for the framework?***\n\nYes. The open-source LLMs are also suitable for our framework. In our experiments (please see Table 3, 4, 5), we also deploy open-source LLMs for evaluation. Currently, although open-source LLMs are still weaker than ChatGPT and GPT-4, we believe these gaps can be reduced with the development of open-source LLMs, for example, LLaMA-2 [1]. Besides, by collecting high-quality user instruction data, we can fine-tune these open-source LLMs to better utilize these AI models from ML communities for solving AI tasks. Some recent works [2] also indicate the availability of this strategy. \n\n***Q2: What if the number of models hosted on Huggingface becomes extremely large? How to ensure that the model selects the most appropriate one from them?***\n\nThanks for your question. Generally, it could possibly affect the accuracy of the selected model and bring additional costs due to longer contexts (e.g., model descriptions), when the number of models hosted on Hugging Face becomes extremely large. To handle this problem, in our framework, we currently design a filtering strategy (please see section 3.2) to guarantee the stability of the system: 1) for the current task, we first retain the models whose task type can match it; 2) then we filter out the top-K candidate models based on the number of their downloads; 3) finally, we enable LLMs to select the most suitable model from these candidate models by considering the user request. We will make a more clear description of this part in the final version and continue to investigate this problem in the future.\n\nBesides, our strategy is just one of these solutions and we believe there should have more optimal strategies for this problem, which can be studied in the future. For example: 1) we could construct a hierarchy graph of all models based on the model cards on Hugging Face, to enable better selection; 2) by collecting many user requests which include model(tool)-use, we could optimize public LLMs (e.g., LLaMA-2) to enable it to make a better strategy for model selection. We will also continue to investigate this direction to search for better strategies.\n\n[1] LLaMA 2: Open Foundation and Fine-Tuned Chat Models\n\n[2] ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs."}]

Author rebuttal:
# To All Reviewers


We sincerely thank each reviewer for providing constructive comments for our paper, which are very helpful to improve our paper. Below are our responses to some general issues:

***Q1: Model Selection***

Thanks for the comments of each reviewer. Here, we will provide more details and descriptions to better explain the design of model selection and its correlation with task planning. We think it is necessary to keep the model selection as an individual step and give these reasons:

1. **Motivation**: First, task planning helps us to analyze user request and decompose it into a task sequence, while model selection is to answer which model is suitable for this task. Therefore, the target of each stage is different, and we prefer to keep two steps respectively and thus encourage to design different strategies for each.

2. **Efficiency**: Sometimes, the number of models from ML communities could be massive. Since we need to use model descriptions to call the most suitable model, selecting models will bring more token consumption in prompts. As a result, combining task planning and model selection will cost more tokens within one query, and separating these two steps will be more friendly for LLM computations.

3. **Flexibility and Extensibility**: Furthermore, keeping task planning and model selection as two parts can provide better flexibility and extensibility: 1) the model library is usually dynamic since developers will continually add the latest models into the community. Therefore, we expect that the model selection is independent of task planning so that it is easy for us to update the model library; 2) some models could include more detailed granularity (e.g., specific domains). These settings necessitate the development of more flexible selection strategies.

We will add these descriptions to the final version to better explain the design of our framework.

***Q2: Experiments & Evaluation***

Thanks for the suggestions provided by each reviewer. We have added these results to better explain the settings in our framework. 

+ Table 1 and Figure 1 in Rebuttal PDF show the effects of variance and number of demonstrations. 
+ Table 2 in Rebuttal PDF demonstrates the evaluation of our framework by human experts. 
+ Table 3 in Rebuttal PDF presents some cases illustrating the model selection. 

Specifically, for human evaluation, we collected 130 diversity requests to measure the planning capability and final generation of LLMs in our framework. Here, we design passing rate, rationality and success rate to evaluate different stages:

- Passing rate: Check LLMs whether can understand instructions and execute in the current stage.
- Rationality: Check whether the generated task sequence / selected tools are rational when compared with user requests.
- Success rate: Check the final results whether satisfy the user request.

We invite three human experts to annotate the provided data based on our designed metrics, and then calculate the average value as the final score.

Please refer to the attached Rebuttal PDF for the detailed results. 

| **Demo Variance** | **LLM** | **Single Task**    |  | **Sequential Task**   |  | **Graph Task** |  |
|:---------------------------------------|:---------|:-------------|:---------|:-----------------|:--------:|-----------|-----|
|                                       |         | **Acc** ↑ | **F1** ↑  | **ED (%)** ↓ | **F1** ↑  | **F1** ↑ |
| 2                                     | GPT-3.5  | 43.31      | 48.29     | 71.27        | 32.15     | 43.42         |
| 2                                     | GPT-4    | 65.59      | 67.08     | 47.17        | 55.13     | 53.96         |
| 6                                     | GPT-3.5  | 51.31      | 51.81     | 60.81        | 43.19     | 58.51         |
| 6                                     | GPT-4    | 66.83      | 68.14     | 42.20        | 58.18     | 64.34         |
| 10                                    | GPT-3.5  | 52.83      | 53.70     | 56.52        | 47.03     | 64.24         |
| 10                                    | GPT-4    | 67.52      | 71.05     | 39.32        | 60.80     | 66.90         |

*Table 1. Evaluation of task planning in terms of the diversity of demonstrations. We denote demonstration variance as the number of different task types involved
in the demonstrations. (Same as Table 1 in Rebuttal PDF.)*


| **LLM**      | **Task Planning** |              | **Model Selection** |        | **Response Generation**  |
|--------------|-------------------|--------------|---------------------|--------|---------------------------|
|              | **Passing Rate** ↑  | **Rationality** ↑  | **Passing Rate** ↑  | **Rationality** ↑  | **Success Rate**↑     |
| Alpaca-13b   | 51.04      | 40.26             | -           | -                | 6.92                      |
| Vicuna-13b   | 79.41      | 66.15             | -           | -                | 15.64                     |
| GPT-3.5      | 99.42      | 91.03             | 93.89       | 84.29            | 63.08                     |

*Table 2. Human Evaluation. We design passing rate, rationality and success rate as the subjective metrics and invite human experts to evaluate them. (Same as Table 2 in Rebuttal PDF.)*


Meta-review: Summary of reviews:

The proposed framework, HuggingGPT, is a novel approach to using large language models (LLMs) to solve complex tasks. The framework consists of four steps: task planning, model selection, task execution, response generation.

Strengths:

- It is able to solve a wide variety of complex tasks, including tasks that require multiple modalities (e.g., text, images, and audio).
- It is able to leverage the strengths of LLMs, such as their ability to learn from large amounts of data and generate text outputs.
- It is able to select the appropriate models to execute each subtask, which can improve the performance of the framework.

Weaknesses:

- It is dependent on the availability of LLMs. This could be a limiting factor in some cases, as LLMs can be expensive to train and maintain.
- It is not yet clear how the framework would perform in real-world settings. 

Overall, the proposed framework is a promising approach to using LLMs to solve complex tasks. More research is needed to evaluate the framework in real-world settings and to address its weaknesses. I thank the authors for engaging with reviewers extensively to address their questions and provide additional results. Hence, I recommend this paper for acceptance.
Decision: Accept (poster)