OpenAGI: When LLM Meets Domain Experts
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Large Language Model
Artificial General Intelligence
AI Agents
Complex Task Solving
Open-domain Model Synthesis
Compositionality

Abstract: Human Intelligence (HI) excels at combining basic skills to solve complex tasks. This capability is vital for Artificial Intelligence (AI) and should be embedded in comprehensive AI Agents, enabling them to harness expert models for complex task-solving towards Artificial General Intelligence (AGI). Large Language Models (LLMs) show promising learning and reasoning abilities, and can effectively use external models, tools, plugins, or APIs to tackle complex problems. In this work, we introduce OpenAGI, an open-source AGI research and development platform designed for solving multi-step, real-world tasks. Specifically, OpenAGI uses a dual strategy, integrating standard benchmark tasks for benchmarking and evaluation, and open-ended tasks including more expandable models, tools, plugins, or APIs for creative problem-solving. Tasks are presented as natural language queries to the LLM, which then selects and executes appropriate models. We also propose a Reinforcement Learning from Task Feedback (RLTF) mechanism that uses task results to improve the LLM's task-solving ability, which creates a self-improving AI feedback loop. While we acknowledge that AGI is a broad and multifaceted research challenge with no singularly defined solution path, the integration of LLMs with domain-specific expert models, inspired by mirroring the blend of general and specialized intelligence in humans, offers a promising approach towards AGI. We are open-sourcing the OpenAGI project's code, dataset, benchmarks, evaluation methods, and the UI demo to foster community involvement in AGI advancement: https://github.com/agiresearch/OpenAGI.

OpenAGI: When LLM Meets Domain Experts
Yinggiang Ge Rutgers University
Wenyue Hua Rutgers University
Kai Mei Rutgers University
Jianchao Ji Rutgers University
Juntao Tan Rutgers University
Shuyuan Xu Rutgers University
Zelong Li Rutgers University
Yongfeng Zhang Rutgers University
Abstract
Human Intelligence (HI) excels at combining basic skills solve complex tasks. This capability is vital for Artificial Intelligence (AT) and should be embedded comprehensive Al Agents, enabling them to harness expert models for complex task-solving towards Artificial General Intelligence (AGI). Large Language Models (LLMs) show promising learning and reasoning abilities, and effectively external models, tools, plugins, or APIS to tackle complex problems. In this work, we introduce OpenAGI, an open-source AGI research and development platform designed for solving multi-step, real-world tasks. Specifically, OpenAGI uses a dual strategy, integrating standard benchmark tasks for benchmarking and evaluation, and open-ended tasks including more expandable models, tools, plugins, or APIs for creative mooblem-solving Tasks are presented as natural language queries the L.M,, which then selects and executes appropriate models. We also propose a Reinforcement Learning from Task Feedback (RLTF) mechanism that uses task results to improve the L.MM's task-solving ability, which creates a self-improving AI feedback loop. While we acknowledge that AGI is a broad and multifaceted research challenge with no singularly defined solution path, the integration of LLMs with domain -ssecffc expert models, inspired by mirroring the blend of general and specialized intelligence in humans, offers promising approach towards AGI. We are open-sourcing the OpenAGl project's 's code, dataset, benchmarks, evaluation methods, and the UI demo to foster community involvement in AGI advancement: https: /ttt::///ihuuae com htt.a//.cc.coopppeeeeeeeee/eeeea
Introduction
The acquisition and reuse of skills is a fundamental aspect of human intelligence that enables the formation of complex skills address novel or intricate problems [19] 57]. We posit that machine intelligence should incorporate this capacity to synthesize various skills by composing them into complex skills for complex solving. In computer science parlance, each skill is referred as domain expert "model" a reusable tool, module, network, plugin, or API with defined function. The domain expert models can be synthesized into a larger "plan" for performing more complex tasks. The model synthesis process is adaptable to the input or task, such that for a given task, the models are synthesized into the most suitable plan to address the task at hand. As a result, different inputs tasks may necessitate distinct synthesized models
plan for task-solving.
Recent advances in Large Language Models (LLMs) have showcased exceptional learning and reasoning capabilities, rendering them well-suited for selecting, synthesizing, and executing external expert models to address complex tasks. These LLMs, such as GPT series 332 2. LLaMA series
45 44
and T5 series [33 8), have exhibited a profound understanding of natural language and the
[yinggiang wtwp://..uuuaaiimii.iiaa....
@rutgers.ed edu
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Prompt
Instruction
You see plenzer who is cxpert coming up with to lis goen eljector execation el Ensure pontbll tach tace in memlaannt effectiv, shert and necesary. robot only allowed utiltee the proctded machin laariing medels do nech teak. Urrdoy to de los to wo tavve the ohyreive: Bestore low crdeatiooll maay, liier and propoeal aneges to regalar irrggn.
LLM
Rectora palay, reerealuutinedd
Mlnnyy to regulie images.
Frovided medilss
Sentiment Anaaytte shal when want anabyse the scatiment of sentenee. reoedveves seriencee inpet.
Teat Sunmmrtiattinn atlle neeful wlons
paragraph, recalves
Madiiee
sscld
tegett
pasition. eupeee quesiion lesel en givea costral.
ixput.
Domain Expert Models Plagin, A710))
Questien Anssectig: iedfal wdens you
T5
Image Elaeptrcatoo: ueeful whe you want know the thas tmage. It recetve tmage, path aa Ingett
dfject Detertionn ssellal when you want to detect the objects in phatts it restices lesign. path as legurt colerisation: oteee usefid whes you west to caletor a pheto. it peocives imagee pats input.
IP gaigg Tiee
A
traage Super Reeaalition uurfal whee you sost to create a nigh-rmullation Iauge from Ins-raselation trsige
trager Denetiing weffll when you to Genviar
image receioes
beblurring sastial when
daihar blarry Image recelves
Langhhaka
Viccal Opeettios
aeed
foard gives image
SAASAA
Captieming
photo. iaage deter wellll whes you to grneritee imaage board on glvvn desoripciin.
what
repet.
Plan
Plan Execution
Evaluation
1. Image Densining: Lae this medel remove the noter from the imagee, Neise refers the random varaation brightnen or coloe Infermation lrnapes, capturing
Predictions
Image beblurring: implement image mation, defocus,
the blarry images. klurviness
Traage Senolatig
this medel enhance the resolution the the msdded can, convert low reselution images high-reaclution veriins, giving them moee decaal and eelacing pisciatine.
Super-Reso Resolstii:::
Colertoation: the the colertzation medel to add colers the grayucale images Thi model will convect meuscheenee (black, white and gray) imagee colored images.
Imange Beblurring
Ground-outh
Iasage Super Besobtiie
Codectaation
Figure I: An example of benchmark tasks, which shows the OpenAGI pipeline. OpenAGI generates a solving plan for the input task described in natural language (using GPT-3.5 in this example, but can be other LLMss such as GPT-4, Vicuna, Flan-T5, Claude-2, and LLaMA-22,, executes the plan with domain expert models, tools, APIs, and then conducts evaluation for the plan execution results.
ability to generate coherent and contextually relevant responses. This has opened up new possibilities for their application in complex tasks involving multi- modality data, such as image and text processing. as well as the integration of domain-specific knowledge. In this process, LLMs play a crucial rolc as they can understand and generate natural language, which helps AI to better comprehend and handle various problems. By integrating knowledge and skills from different domains, Open-domain Model Synthesis (OMS) holds the potential to drive the development of artificial general intelligence (AGI), enabling AI to solve diverse array of problems and tasks. Despite acknowledging the complexity and lack of a defined path towards AGI, the combination of LLMs and domain domain-specific expert models, inspired by the interplay of general and specialized intelligence in humans, provides promising direction 19]. However, the current research field, despite initial attempts, presents several significant challenges: 1) Extensibility: Several existing works employ fixed number of models, such as WebGPT (25) and ToolFormer [40], resulting in difficulties when attempting to expand their capabilities; 2) Nonlinear Task Planning: The majority of current research is limited to solving tasks with linear task planning solutions [49] 18], meaning that each sub-task must be completed before the next sub-task can start. However, linear planning of models may not suffice for solving complicated tasks, besides, many tasks involve multiple multi-modal inputs; 3) Quantitative Evaluation: Many existing works only provide qualitative results, such HuggingGPT (42). This makes it difficult to assess the planning capabilities of LLMs to determine whether the strategies employed are optimal.
In order to mitigate the above limitations, we develop a platform that encompasses a diverse array of domain-specific expert models and intricate multi-step tasks with single or multiple multi-modal inputs. Furthermore, to promote the community's 's long-term advancement and assessment of AGI's

---

abilities, we open-source all code and datasets, and hence, name this platform OpenAGI. A toy example, showing the entire pipeline of OpenAGI, is depicted in Fig. [ Specifically, 1) a natural language instruction of a specific task is given; 2) the instruction is augmented by manually designed prompt and then fed as input into L.LM to generate a plan; 3) the expert models are selected and synthesized based on the generated plan, and subsequently executed to process the data samples; 4) the task-solving ability of the LLM can be evaluated by comparison between the output and the ground-truth labels or through human evaluation.
OpenAGI embodies a dual approach to address diverse requirements- saguireement--benmmaakk tasks and open- ended tasks. On the one hand, we have incorporated benchmark tasks, each supported by task-specific datasets and evaluation metrics. This inclusion provides researchers with a consistent platform to assess and compare the performance of various models, stimulating continuous improvement and competitive innovation. For benchmark tasks, as depicted in Fig- I we utilize a selection of expert models derived from esteemed libraries such as Hugging Face's 's transformers and diffusers, as well as from GitHub repositories, thereby easily facilitating the expansion of our model set. Additionally, the datasets have been meticulously selected to align with or resemble the training datasets of the respective models. We then implement a variety of data augmentation techniques to enhance these original datasets, enabling the construction of sophisticated multi-step tasks designed to assess the planning and task-solving capabiliiiies of a given LLM. On the other hand, OpenAGI also offers open-ended tasks that utilize variety of expandable models. These tasks open the door to creativity and imaginative groblem-solvimg enabling the exploration of innovative solutions that may not emerge within more constrained task frameworks. For open-ended tasks, depicted in Fig, 2 which is designed to accommodate a broader spectrum of needs, we further include LangChain to provide additional expert models, such Google Search, Wikipedia, Wolfram Alpha and so on. Indeed, relying solely on input text for learning proves insufficient for LLMs when faced with real-world tasks. In order to improve its performance, we introduce a mechanism referred to as Reinforcement Learning from Task Feedback (RLTF). This approach capitalizes on the performance feedback procured from tasks following the execution of the solution devised by the LLM. Consequently, the RLTF mechanism effectively refines the LLM's s planning strategy, resulting in an enhanced and more adaptive system. In summary, the key contributions of the work include:
We introduce OpenAGI, an AGI research platform, specifically designed to offer complex, multi- step tasks accompanied by their respective datasets, evaluation methods, and diverse range of extensible models which can be synthesized to effectively solve these tasks. The purpose of this platform is to aid in the quantification of the overarching planning and task-solving abilities of LLMs. OpenAGI embraces AGI by focusing on LLM-driven, (open-domain) model synthesis, predominantly utilizing models and datasets Hugging Face, GitHub and LangChain.
We propose the LLM+RLTF approach for OpenAGI, which leverages a Large Language Model as a controller select, synthesize and execute various external expert models for complex task-solving. The feedback obtained from these tasks is then employed to refine the LLM's planning strategy, thereby enhancing the LLM's overall performance and task-solving ability.
We evaluate both open-source and closed-source LLMs with differing scales under distinct learning schema and the OpenAGI pipeline. Our findings suggest that even smaller-scale LLMss, when paired with an appropriate learning schema such as RLTF, are able to possess the potential to outperform competitors that equip a significantly greater magnitude of model parameters.
2 Related Work
2.1 Large Language Model and AI Agents
With the advancement of highly parallelizable transformer architectures, pre-trained language models (PLMs) have demonstrated remarkable capabilities in comprehending, generating, and manipulating natural language [3] 24] These models are pre-trained on a large corpora of text data and commonly fine-tuned for specific downstream tasks. Shortly, the scaled-up PLMs, known as Large Language Models (LLMs) [44 27 6 55 45), encompassed substantially greater number of parameters and leveraged vast amounts of training data. Consequently. LLMs exhibited an enhanced capacity to learn intricate language patterns and structures, along with a notable reasoning ability, leading
to superior performance across diverse natural language processing tasks 2 45 55
30 14
52. Apart from the above superiority, L.LMs may occasionally produce seemingly plausible yet inaccurate predictions and face challenges when addressing problems that require specialized domain
3

---

expertise [23] Consequently, the emerging field of Augmented Language Models (ALMs) focuses on addressing the limitations of conventional LLMs 8 2 by equipping them with enhanced reasoning capabilities and the ability to employ external resources [33] The process of reasoning involves breaking down intricate assignments into smaller, more manageable sub-tasks that can be independently or collaboratively tackled by LLMs with the assistance of tools. What's more, LLMs can also invoke external tools or models to accomplish the relevant tasks, For example, ToolFormer [40) introduces external API tags within text sequences, facilitating LLMs access to external tools. Visual ChatGPT [5]] combines ChatGPT with Visual Foundation Models (VFMs) such as Transformers, ControlNet, and Stable Diffusion, which acts bridge between users, allowing them to communicate via chat and generate visuals. HuggingGPT [42] integrates the Hugging Face hub with task-specific models around ChatGPT to tackle AI tasks. ChatGPT for Robotics [47] employs ChatGPT for a wide array of robotics tasks through strategic prompt engineering. Besides, several open-sourced GitHub repositories are related to this topic, such as BabyAGI and AutoGPT. Notably, AutoGPT [15] is an automated agent, which is designed to set multiple objectives, break them down into relevant tasks, and iterate on these tasks until the objectives are achieved. Augmented language models may use these enhancements separately or joint them in a specific order to finish the specific task, which ultimately results in superior generalization capabilities.
Different from other works, we propose OpenAGI, an open-source AGI research and development platform designed to address the challenges commonly encountered in existing works, such as exten- sibility, nonlinear task planning, and quantitative evaluation. Furthermore, we introduce innovative methods into the learning schema of LLMs, including Reinforcement Learning from Task Feedback (RLTF) and nonlinear task planning, which aims to address challenges on generalization, optimal task planning. and Al's reff-improvement (please see Sec. | in supplemen- tary materials for an extended discussion on these problems). We hope the OpenAGl platform can facilitate the open and long-term development and evaluation of AGI abilities in the community.
(OOD)
2.2 Reinforcement Learning from Human Feedback (RLHF)
To better align Large Language Models (LLMs) with human values, Reinforcement Learning from Human Feedback (RLHF) has been introduced 77|58 (18, which fine-tunes LLMs by collected human feedback, effectively enhancing alignment criteria such as helpfulness, honesty, and harmlessness. At its core, RLHF deploys reinforcement learning (RL) algorithms, notably Proximal Policy Opti- mization (PPO) 41, to tailor LLMs to this feedback via a reward model. Importantly, this approach actively involves human oversight in the training process, exemplified by notable models such as InstructGPT 227 Nonetheless, the efficacy of RL.FF is contingent upon the quality of feedback from adept labelers, rendering its practical implementation challenging [1] 29] Consequently, there is an imperative to refine the RLHF framework to diminish the dependency on manual labeling and explore innovative, efficient annotation methodologies that ensure data integrity. Compared with RLHF, the proposed RLTF gets task feedback to supply information that guides the learning direction of LLMs, resulting in improved and more efficient strategies, which does not require human intervention.
3 The OpenAGI Platform
OpenAGl includes a wide range of features tailored to various needs. One key component is its benchmark tasks, detailed in Sec. [3.]] particularly valuable tool for researchers. These tasks come equipped with task-specifi datasets and evaluation metrics. This makes it possible for researchers to evaluate the performance of different LLMs in a structured and uniform manner, offering insights into their efficacy and potential areas for improvement. In addition to benchmark tasks, OpenAGI also offers open-ended tasks, detailed in Sec. 3.2 These tasks allow for a greater degree of creativity and imagination, breaking away from conventional constraints to enable more exploratory research. We believe this combination of structured benchmark tasks and flexible open-ended tasks makes OpenAGI robust and versatile platform that can cater to a diverse array of research requirements.
3.1 Benchmark Tasks
For benchmark tasks, our goal to provide the community a valuable tool to evaluate the planning abilities of LLMss for complex, multi-step tasks. Specifically, instead of building complicated, multi- step tasks from scratch, we first explore the domain expert models (Sec. 3.1.1 that can used as building blocks, then introduce how create such tasks based on them (Sec. 3.1.2

---

3.1.1 Domain Expert Model Set
We now present the domain tasks and the corresponding models that can be employed in our platform. This set is designed to be flexible, allowing users to easily incorporate their own domain tasks and models. Our domain tasks are as follows:
Sentiment Analysis classifies the sentiment polarity of given sentence [I: Text Summarization creates a text summary that represents the most important or relevant information within the original text content [20]; Machine Translation converts
Language-related Models:
sentence from a source language to a target language [3|; Fill Mask involves replacing masked
words within given text [22]; Question Answering (QA) provides textual answer of a question
based on the given context [39].
Vision-related Models: Image Classification aims comprehend entire image as whole and assign it to a specific label [2] Object Detection identifies and localizes specific objects within an image by detecting their instances of particular class [B: Colorization refers the technique of adding plausible color information to monochromatic photographs or videos [54]; Image Suppr-resolution generates a high-resolution (HR) image from a low-resolution (LR) image [T];; Image Denoising aims to remove unwanted noise from an image while preserving its important features [53] Image Deblurring aims to recover a clear image from a blurred input
image [53).
Language Models: Visual Question Answering (VQA) involves answering questions based on an image [8];; Image Captioning generates textual descriptions of the visual content depicted in an image; Text-to-Image Generation generates images from given input sentence or sequence of words [36].
The details of the corresponding models are shown Tab.
and A.3 in supplementary materials. After selecting the domain expert models, choosing the raw datasets becomes a more straightforward process, provided that we need to ensure proper alignment between the datasets and the domain expert models' training sets. Raw datasets are provided follows: ImageNet-1KK (38), Common Objects in Context (COCO) [21]. CNN/Daily Mail [26]. Stanford Sentiment Treebank (SST2) 28). TextVQA QA 43] Stanford Question Answering Dataset (SQuAD) [35] More details
about theses datasets can be found in Sec. A.2 in supplementary materials.
3.1.2 Multi-step Tasks and Corresponding Datasets Construction
multi-step task, as the name suggests, refers to a complex problem that cannot be solved one simple step. It necessitates several sub-processes stages, each requiring a particular type of problem- solving skill, in other words, domain expert model. order to construct such complex, multi-step tasks, introduce several commonly- data augmentation methods, which Gaussian Blur, Gaussian Noise, Grayscale, Low Resolution, Translation, Word Mask, to augment the raw dataset. More details about these methods can be found in Sec. A.3 in supplementary materials.
For the purpose of our study, we have sorted these tasks into six primary categories according to the modalities of their inputs and outputs:
Image in, image out: In these tasks, images undergo several transformation stages. An example could be a task that involves "Denoising and enhancing the resolution of a low-resolution, noisy image". Here, the multi-step process entails image denoising followed by super-resolution.
Image in, text out: These tasks usually involve interpreting the content of images. For example, "Detect objects in an image and describe them in sentence" requires object detection followed by text generation.
Text in, image out: Tasks under this category may include generating an image based on textual descriptions, such as "Create graphical representation of the room described in the given text". demanding text understanding and image generation steps.
Text in, text out: These tasks engage in text transformation or interpretation. For instance, "Translate a paragraph from English to German and summarize it in English" requires two steps translation and summarization.
Image-text puir in, text out: These tasks deal with complex interplay between visual and textual data. For example, "Given an image and question about the image in English, answer the question in German." This task includes image-text understanding, question answering, and translation.
5

---

Instruction
Prompt
LLM
You planner who expert coming with todo lot for givee elgective for the exccution af' robet.
traditonal chines style,
each rebot vtilice the provided tools whra needed.
featuring Can shan Luu shut as
Ensure the sa short is proublk, tuska in relevent, effoctire and
the theme and accempanted ports a piece clasied
Develop a detuiled te-de lst to schieve the oljrctive:
Creste a painting the texitiond chinese style, featuring 'ivo Shan Lin sard' the farme, and accompunnrd by a Chiness poets and a piece of clasical Chincse instrueestal musie.
muic.
T5
Provided male
Domain Expert Models
Search ideal for ancevttng quectians related current events gathering Information. The outpute fest.
A
Poluting:
when you gromst pointing using a given description.
Text to Miuter useful when you need generate musr using a given deanription.
Test to Porty: uehhil wien you need to groerate a peen using give decntprion.
O
Plan
Plan Execution
Result
1. the Search to research the thems "Gan shan Liu siba'. Understamd the meaning and the elements eften regresented with this theme.
Search research the truditional Chinese painting style, focusing on technigues, elementss and symboliam often
Based the information gathered step I detailed descripetin of palitting should llolk Easure that descrtition is representative Shui" theme and line traditional chinese painting style.
Teat to Priuting
Use the Text to Painting tnel to generate painting based on the description created in step
5. Use the Search tool to research Chinese postry styles and themes that would fit well with the "Gao Shan Liu shos" theme.
Search
6. Create a detuiled description for the poem based on the information from step 5, including its theme, style, and tone, ensuring they align with the "Gao Shan Lou Shui" theme,
D K.
groerate porm based descrigtion created step
the
Text Poetn
b
k
a. Use the Search tool to reaearch classical Chinase instromental music styles and instruments that would complement the "Gao shan Lao Shin' theme and the created painting.
Search
Create detailed description for the musk su Information step & incloding style, temsoo, and instruments used, ensuring they align with "Gao Shan Liu Shui" theme.
10. Use the Text to Music tool to generato piece munii based en the description created in step
Text Music
Figure 2: An example of open-ended tasks, which instructs OpenAGI to create artwork given the theme "Gao Shan Liu Shui" (translating to "High Mountain and Flowing Water" in English). OpenAGl generates non-linear struestruut plan for the task with GPT-3.5, and then executes the plan with expert models to create painting, a poem, and a piece of music for the theme.
Text-text pair in, text out: These tasks can involve comparison, synthesis, or information extraction from two text inputs. For instance, "Given two reviews of a movie in English, translate them into German and provide a summary."
In total, we have devised 185 multi-step tasks, of which 117 tasks maintain a linear task structure with steps following a simple sequence, while the remaining 68 tasks exhibit a non-linear task structure, where steps might be performed concurrently or in complex order. Among these categories, tasks such as Question Answering (QA) and Visual Question Answering (VQA), involving multiple or even multi-modal inputs, are notably complex and defy simple, linear task planning solutions. For a comprehensive view, we provide example tasks and their input and output data samples in Tab. [A.4] of the supplementary materials. Additionally, a complete list of the task descriptions, accompanied by their estimated difficulty levels, can be found Tab. A.5 within the supplementary materials.
3.1.3 Evaluation Metrics
Given that the benchmark tasks of OpenAGl comprise a diverse range of domain tasks with multi- modal data, we classify them according to domain tasks as well as input and output types. We then

---

assess their performance using the following three metrics based on their categories: CLIP Score [6]. BERT Score [56] and ViT Score (more details can be found supplementary). In particular. we employ the CLIPP Score only for Text-to-Imee -extet Generation-based tasks, the BERT Score is utilized to assess tasks with text outputs, and the VIT score applied to measure image similarity for the remaining tasks with image outputs. We also normalize the BERT and CLIP scores.
3.2 Open-ended Tasks
Open-ended tasks necessitate an elevated degree of creative and imaginative capacity, as they deviate from conventional constraints stimulate more exploratory research. These tasks designed to accommodate a broad spectrum of needs, as illustrated in Fig. [ To achieve this, LangChain integrated to provide additional expert models from renowned sources such as Google Search, Wikipedia, Wolfram Alpha, and more. Crucially, these models offer extendability, ensuring that open-ended tasks are confined specific guidelines or performance metrics. To exemplify this process, Fig. 2 elucidates how OpenAGI is directed to create traditional Chinese painting with "Gao Shan Liu Shui" (translating to "High Mountain and Flowing Water" in English) as its theme. The process is enriched with the addition of a generated ancient Chinese poem and a piece of music that harmonize with the painting. To effectively deliver on this instruction, OpenAGI first conducts an online search to comprehend the historical narrative of "Gao Shan Liu Shui". Sequentially, the painting, poem, and music are generated between expansive language models and domain-specific expert models. The final product coherent artistic ensemble of painting, poem, and music - successfully resonates with the underlying ancient narrative, demonstrating the efficacy of this approach in open-ended tasks. More examples are provided supplementary.
step-by-step fashion, leveraging the collaboration
Reinforcement Learning from Task Feedback (RLTF)
While learning solely from input text powerful method for training LLMs, it is not sufficient for handling real-world tasks that require deeper understanding of context and environment. One potential method to improve the capabilities of LLMs incorporate reinforcement learning (RL) techniques. By leveraging the strengths of RL, LLMs can gain additional insights from trial-and-error experiences. This leads to more robust and adaptive models, especially in situations where labeled data is scarce or when tasks involve physical interactions. In this work, we propose Reinforcement Learning from Task Feedback (RLTF). shown in Fig. 3 which utilizes task feedback to supply more information that guides the learning direction of LLMs, resulting in improved and more efficient strategies. We choose to use REINFORCE (50) in this work and more details about the algorithm are provided in Sec. A.6 in supplementary.
LIM
Planning
A
T5
O
Task Description
Human Evalution
Domain Expert Models
Metrics
Benchmack Evalation
Plan Execution
Results
Figure 3: An illustration of the RLTF mechanism.
5 Experiments
5.1 Backbone LLMs
GPT-3.5-turbo. The GPT (Generative Pre-trained Transformer) series 2 consists of advanced language models. In this work, we use the GPT-3.5-turbo-0301 snapshot.
7

---

Claude-2. Claude-2 [9] is an transformer LLM trained with unsupervised learning and RLHF,
GPT-4. GPT-4 is follow-up version GPT-3.5, which more powerful than predecessors. In this work, we use the GPT-4-0613 snapshot.
Flan-T5-Large. T5 8 a series of language models which are fine-tuned using a technique
called instruction fine-tuning. Flan-TS-Large has 770 million parameters.
Vicuna-7B. Vicuna S is an open-source chatbot trained by fine-tuning the LLaMA 45 model
with user-shared conversations. In this work, we use the 7-billion size model of Vicuna.
LLaMA-2. LLaMA-2 [46] is successor to the original LLaMA model, it is significantly more powerful. In this work, we use the 13-billion size model.
Overall, we include three closed-source LLMs (GPT-3.5-turbo, Claude-2 and GPT-4) well as three open-source LLMs
Vicuna-7B and LLaMA-2-13B).
5.2 Learning Schema of LLMs
We employ the following LLM learning schema for experimentation.
Zero-shot Learning (Zero) directly inputs the prompt to the LLM.
Few-shot Learning (Few) presents of high-quality demonstrations, each consisting of both input and desired output, on the target task. As the model first sees good examples, it can better understand human intention and criteria for what kinds of answers wanted.
Fine-tuning involves using manually labeled data samples as additional training signals to refine and adapt pre-trane pre-trained LLMs specific tasks or domains.
RLTFis is our proposed method in Sec. which further utilizes the RL method to tune the fine-tuned LLMs with human labelled data.
We employ Low-Rank Adaptation (LoRA) 11 to optimize all open-source LLMs across both the Fine-tuning and RLTF learning schema.
5.3 Planning Solution Parser
To transform the original LLM output into a viable task planning solution, we use a parser built on GPT-3.5. The prompt employed reads follows: "You are key phrase extractor who is able to extract potential module names from the given context. You have already known all the module names in the full module list. The full module list is: [Image Classification, Colorization, Object Detection, Image Deblurring, Image Denoising, Image Super Resolution, Image Captioning, Text to Image Generation, Visual Question Answering, Sentiment Analysis, Question Answering, Text Summarization, Machine Translation]. Given the following context: '()'. Please extract a module sequence from this context and remove module names which do not exist in the full module list from this sequence. Output the module sequence after filtering as the format of 'module: modulel, module: module2, module: module3, etc... Once this prompt executed on the LLM's original text output, a task planning solution will be generated which consists of a multi-step solution of the problem.
5.4 Datasets
Considering the fact that the imbalanced number tasks with different input and output modalities could lead to skewed measurement results, we select the tasks in OpenAGl to compose the training set. In particular, we randomly select 10% of tasks, along with their corresponding datasets, based on input and output modalities for training purposes. For few-shot, fine-tuning and RLTF, we supply manually curated, feasible solutions as ground-truth labels. In the case of RLTF, we employ the reasonable initialization for LLMSs and constrained beam search II 37 to reduce the likelihood of producing infeasible solutions (details can be found in Sec. A.7 in supplementary). Moreover, choose additional 10% of tasks, adhering to the same selection criteria as mentioned above, to serve as the test set.
fine-tuning checkpoint
5.5 Experimental Analysis
The main experimental results are tabulated in Tab. and showcasing the results for closed-source and open-source LLMs, respectively. The overall performance is calculated as the average of CLIP,

---

BERT and VIT scores. Here, only the task deseriptions of the benchmark tasks are fed into LLMs (additional information, such as the imput prompt and LLMs' outputs, is provided Fig. A.4 and A.5 in supplementary). Broadly speaking, closed-source LLMS demonstrate superior performance on OpenAGl tasks, with GPT-4 leading the pack under both zero- and few-shot scenarios. In the open-source category, LLaMA-2-1H 13B takes the lead, consistently posting top results across various learning schema-the performance possibly influenced by its larger model size. Notably, open-source LLMs significantly benefit from the tuning methods, particularly Fine-tuning and RLTF. These methods mark noticeable enhancements for Flan-TS-Large, Vicuna-7B, and LLaMA-2-13B when compared with zero-shot and few-shot learning schema. In fact, each of these open-source models hits its pinnacle under the RLTF approach. Conclusively, with RLTF tuning, the performance of LLaMA- 2- 13B approaches that of GPT-3.5, illustrating its potential.
Table OpenAGI toskiso performances under different settings for three closed- source LLMs.
Boldface denotes the highest score under each learning schema.
GPT-3.5-turbo
Claude-2
GPT-4
Metrics
Zero
Few
Zero
Few
Zero
Few
CLIP Score
0.0
0,0
0.0
0.2543
0.0
0.3055
BERT Score
0.1914
0.3820
0.2111
0.5038
0.2076
0.6307
VIT Score
0.2437
0.7497
0.4082
0.5416
0.5058
0.6480
Overall
0.1450
0.3772
0.2064
0.4332
0.2378
0.5281
Table 2: OpenAGI task- solving performances under different settings for three open-source LLMs. Boldface denotes the highest score under each learning schema.
Flan-1 LS-Lage
Vicuna-7B
LLaMA-2-13B
Metrics
Zero
Fow
Fine-tuoing
RLTF
Zero
Few
Fine-tuning
RLTF
Zero
Few
Fine-tuning
RLTF
CLIP Score
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0,0612
0.0608
0.1220
BERT Score
0,2488
0.0
0.0655
0.0513
0.1756
0.0986
0.2401
VIT Score
0.0
0.0
0.6316
0.6978
0.1704
0.4285
0,5507
0,7300
0.3614
0,2558
0.6723
0,7584
Overall
0.0829
0.2105
0,2544
0.0739
0.1428
0.2239
0.3018
0.1533
0.1817
0.2967
0.3735
5.6 Effect of Prompts
We design two types of prompts combined with different levels of model description to test LLMs' zero-shot performances. The first, Prompt-1, only combines the task description with the model names, while the second, Prompt-2, integrates the task description with comprehensive model descriptions, detailing model usage, input, and output types (additional information about these two prompts is provided in Fig. AA. in supplementary). We analyze the results in Tab. and 4 in conjunction with the previous zero-shot results in Tab. and 2 Compared to the original prompt that only uses task description to generate the results in Tab. [ and 2 is evident that in most cases, the closed-source LLMs, such GPT series and Claude-2, tend to outperform when provided with detailed model-related information as seen Prompt-1 and Prompt-2.
contrast, open-source LLMs, whose understanding and reasoning capacity may be weaker than those huge closed-sous source models, appear to be misled by the ambiguous details in Prompt-1 and Prompt-2 during the model selection process. Overall, detailed prompts can assist in improving the zero-shot performance to certain degree, depending on the specific model. However, they may not be as potent as other training scenarios for smaller size LLMs, such fine-tuning or RLTE.
5.7 Case Study of Non-linear Planning
We qualitatively evaluate LLMs' capability of non-linear task planning. Fig. Aillustrates the responses of GPT-3.5, Vicuna-7B and Flan-T5-Large to Prompt-2. The given task description requires the LLM to answer a query posed in English about a given noisy, blurry, and gray-scale image in German. It can be observed from the results that the performance of the models varies significantly. Flan- T5-Large, for instance, demonstrates a struggling comprehension of the query, while Vicuna-7B's answer incorporates all the provided models in an attempt to resolve the task. GPT-3.5 successfully comprehends the task and consequently delivers a reasonable plan. The plan generated by this model is notably non-linear, and it instructs to employ a Visual Question Answering (VQA) model with the English query and processed image as inputs steps and 2 in order to accomplish the task. Similarly, another task scenario is demonstrated in Fig. 2 which is an open-ended task with

---

Table 3: Zero-shot tosk-solving performances under various prompts for three closed-source LLMs.
GPT-b.5-turbo
Claude-2
GPT-4
Metrics
Prompt-1
Prompt-2
Prompt-1
Prompt-2
Prompt-1
Prompt-2
CLIP Score
0.0
0.0
0.0
BERT Score
0.2106
0.3013
0,4088
0.2333
0.4402
0.5595
VIT Score
0.0
0.2710
0,6816
0.7957
0.5497
0,5565
Overall
0.0702
0.1907
0.3635
0.3430
0.3299
0.3717
Table 4: Zero-shot tosk-no performances under various prompts for three open-source LLMs.
Flan-Tr T-------J Large
Vicuna-7B
LLaMA-2-13B
Metrics
Prompt-1
Prompt-2
Prompt-1
Prompt-2
Prompt-1
Prompt-2
CLIP Score
0.0
0.0
0.0
0.0
0.0
0.0
BERT Score
0.0
0.0603
0.0267
0.0971
0.1717
Score
0.2385
0.0
Overall
0.0
0.0
0.0201
0.0884
0.0323
0.0572
GPT being instructed to generate a painting in a traditional Chinese style that depicts "Gao Shan Liu Shui". Initially, GPT seems to lack understanding of what constitutes traditional Chinese style painting and it is also unfamiliar with the concept of "Gao Shan Liu Shui". As a remedy, GPT utilizes Google search in the initial two steps gather information these unfamiliar topics. Subsequently, it integrates the retrieved information to formulate a comprehensive prompt that instructs the Text-to-Iman Generation model to create the desired artwork
Use the "Image Denoising" model to reduce the noise in the given image. Use the "Image Deblurring" model to remove the blur from the denoised image. Use "Image Resolattso model to further enhance the quality of the image. Apply the "Visual Question Answering" model answer the English question based on the processed Image. Fimally, use "Machine Translation"
answer from English
1. Use Sentiment Analysia to analyze the sentiment of the Engliah question.
2. Use Text Summarization to summarize English question.
3. Use Machine Translation to translate the English question into German.
the 5. Use Question Answering to answer the German question based on the given context.
Mask
the masked position the
translation.
Use Image Classification to classify the grayscale image.
grayscale image. 8. Use Colorization to colorize the grayscale image.
7. Use Object Detection detact objects
9. Image Super Resolution
high-resolution image the grayscale image. 10. Use Image Denoising to denoise the grayscale image.
Use image Deblurring deblur the grayscale image.
12. Use Visual Question Answering to answer the German question based on the given image,
13. Use Image Captioning deseribe the contents of the grayscale image.
Use
Image Generation generate image based on the question.
T5
Answering a question German
Figure 4: An example of Non-linear Planning.
6 Conclusions and Future Work
In this work, we introduce OpenAGI, an open-source AGI research platform designed to facilitate the development and evaluation of LLMS in solving complex, multi-step tasks through manipulating vari- ous domain expert models, tools, plugins or APIs. OpenAGl provides a wide range of tasks, models, datasets, benchmarks, and evaluation methods. We also propose the LLM+RLTF approach, which combines LLMs with reinforcement learning to optimize task-solving performance. The evaluation of various LLMs using the OpenAGI pipeline and different learning schema demonstrates that smaller- scale LLMs can potentially outperform larger models when combined with the appropriate learning approach, such as RLTF. In the future, we aim to explore Human-in-thee in-the-loop agents, where LLM may prompt human experts for answers as one step of the mask-solving plan when suitable model is unavailable, thus enabling better Human-Al collaboration; 2) Trustworthy agents, which guarantee the safety and the ethical standard of agents during task-solving: and 3) Self-improving agents, which enable automated task generation and training that facilitate OpenAGI in independent exploration of tasks, empowering the self-reflection, self- aelmpproppiing and self- eef--improvement of intelligent agents.
10

---

References
[1] Dogu Araci. 2019. Finbert: Financial sentiment analysis with pre-trained language models. arXiv preprint 10Xiv:198.11066 (2019).
[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877-1901.
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16. Springer, 213-229.
[4] Hanxiong Chen, Yunqi Li, He Zhu, and Yongfeng Zhang. 2022. Learn Basic Skills and Reuse: Modularized Adaptive Neural Architecture Search (MANAS). In Proceedings of the 31st ACM International Conference on Information & Knowledge Management. 169-179.
[5] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* * ChatGPT Quality. https
//1mays // 2020/00232000 03
[6] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.00 023311 (2022).
[7] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems 30 (2017).
[8] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.1 1116 (2022).
[9] Claude-2. 2023. Model Card and Evaluations for Claude Models.
[10] Marcos V Conde, Ui-Jin Choi, Maxime Burchi, and Radu Timofte. 2022. Swin2SR: Swinv2 transformer for compressed image super-resolution and restoration. arXiv preprini arXiv:2209.11345 (2022).
[11] Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2020. Autoregressive entity retrieval. arXiv preprint arXiv:2010.00904 (2020).
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1999 (2020).
[13] Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. 2022. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.0788. (2022).
[14] Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. 2022. Recommen- dation as language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5). In Proceedings of the 16th ACM Conference on Recommender Systems. 299-315.
[15] Significant Gravitas. 2023. AutoGPT. Cetps::/newwrr agpt. CO/
[16] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. Clipscore: reference-free evaluation metric for image captioning. arXiv preprint arXiv:2 104.004400 04.08718 (2021).
11

---

[17] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprini 1rXiv:2006.06685 (2021).
[18] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning. PMLR, 9118-91111 9147.
[19] Daniel Kahneman. 2011. Thinking, fast and slow. macmillan.
[20] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising -o-eueeeee- pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv: 1910. /346/ (2019).
[21] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision. Springer, 740-755.
[22] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining approach. arXiv preprint 190. 1692 (2019).
[23] Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Y. Yu, Asli Celikyilmaz, et al. Augmented language models: a survey. arXiv preprint xxiv:2302.00844 (2023).
2023.
[24] Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heinz, and Dan Roth. 2021, Recent advances natural language processing via large pre-trained language models: A survey. arXiv preprint arXiv:2111.01243 (2021).
[25] Reüichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. Webgpt: Browser- assisted question- with human feedback. arXiv preprint arXiv:2112.09333 (2021).
[26] Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016. Abstractive text summarization using sequence-to-sequene mas and beyond. arXiv preprint 4rXiv:1602.06023 (2016).
[27] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35 (2022), 27730-27744.
[28] Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. arXiv preprint cx/0506075 (2005).
[29] Ethan Perez, Saffron Huang, Francis Song. Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. 2022. Red teaming language models with language models. arXiv preprini 2xXi::2202.03286 (2022).
[30] Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, et al. 2023. Tool learning with foundation models. arXiv preprint arXiv:2304.08354 (2023).
[31] Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. 2020. Pre-trained models for natural language processing: survey. Science China Technological Sciences 63, 10 (2020), 1872-1897.
[32] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et 2019. Language models are unsupervised multitask learners. (2019).
12

---

[33] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Texx Transformer. Journal of Machine Learning Research 21, 140 (2020), 1-67.
htttpp///mml..oog///pprrr////// 2200/////200000 html
[34] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, al. 2020. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res. 21, 140 (2020), 1-67.
[35] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv: (606.05254 (2016).
[36] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10684-10695.
[37] Ohad Rubin and Jonathan Berant. 2021. SmBoP: Semi-autoregressivg Bottom-up Semantic Pars- ing. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 311 311-324.
[38] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhi- heng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge. International Jour- nal of Computer Vision (IJCV) 115, 3 (2015), 212- 141263-015-0816
[39] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint 1910:19100011000 (2019).
[40] Timo Schick, Jane Dwivedi-Yuu Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761 (2023).
[41] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proxi- mal policy optimization algorithms. arXiv preprint arXiv: 1707.06347 (2017).
[42] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang, 2023. HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace. arXiv preprint arXiv:2303. 17580 (2023).
[43] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition. 8317-8326.
[44] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford Alpaca: An instruction-folloiing LLaMA model. https: / thub. www.:ottotttteeeeeee lab/ Sataanford_apaacee
[45] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. /3977 (2023).
[46] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. 2023. Llama Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023).
[47] Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor. 2023. Chatgpt for roboties: Design principles and model abilities. Microsoft Auton. Syst. Robot. Res 2 (2023), 20.
[48] Jianfeng Wang, Zhengyuan Yang. Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. 2022. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205.1411 (2022).
13

---

[49] Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. 2023. Describe, explain, plan and select: Interactive planning with large language models enables open-worhh world multi-task agents. arXiv preprini arXiv:2302.01500 (2023).
[50] Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning 8, (1992), 229-256.
[51] Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. 2023. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint 2303002303.04671 (2023).
[52] Ruosong Ye, Caiqi Zhang, Runhui Wang, Shuyuan Xu, and Yongfeng Zhang, 2023. Natural language is all graph needs. arXiv preprint arXiv:2308. 07134 (2023).
[53] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. 2022. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5728-5739.
[54] Richard Zhang, Lun-Yan Zhu, Phillip Isola, Xinyang Geng, Angela S. Lin, Tianhe Yu, and Alexci A. Efros. 2017. Real-Time Reme-Time User-Guided Image Colorization with Learned Deep Priors. ACM Trans. Graph. 36, 4, Article 119 (jul 2017), 11 pages. https://dbi.org:
3145/3072999.00733
[55] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0100 00006 (2022).
[56) Tianyi Zhang. Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv: (ate::904009997 (2019).
[57] Yongfeng Zhang. 2021. Problem Learning: Towards the Free Will of Machines. arXiv preprint arXiv:2109.00177 (2021).
[58] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv; 190.:08990099 (2019).
14

---

Supplementary Material for OpenAGI
Image Deblurring,
Colorization,
&. Image Denoising,
4. Image Supe-resolution
Input Images
Predictions
Predictions
(a) Examples of Diistiiiiiiitttionn Generalization ing the same task (task description the same as Fig. [ with images from different distributions. The places highlighted by red ellipses denote areas with significant discrepancies from the ground-truth images after executing the same image restoration madel sequence.
solv-
(b) Example of different model se- quences for solving the same task depicted in Fig. [ Both are valid model sequences but they result in very different task-solving quality.
Figure A.1: Research challenges when solving complex, multi-step tasks with augmented LLMs.
A.1 Research Challenges
Although the OpenAGl platform offers numerous advantages and enhanced accessibility, also gives rise to a variety of novel research challenges, such as:
nuttof-Diitribbution (OOD) Generalization. Domain-specific expert models may exhibit limited generalization ability due to their strong dependence on the distribution of the training data. As demonstrated in Fig. A.I (a) when processing images from disparate sources exhibiting distributional shift, the original model sequence to address the task in Fig. I becomes ineffective. In the majority of instances, only few colors are accurately restored, while most remain incorrect. Furthermore, noise and blurring persist, remaining highly perceptible to human observers.
Optimal Task Planning. There compositional number of ways to combine different models to generate solutions, which can make it difficult to identify the best approach. Additionally. it is possible for multiple valid solutions to exist for given task, but the quality of each solution can vary greatly. For instance, as depicted in Fig. A. (b) executing the same four models in a different sequence compared to Fig. [ can lead to noticeably different outcomes. The results from the second approach (Fig. A. (b) exhibit significantly more noise and color inconsistencies compared to the
first approach (Fig. I. Therefore, is crucial for the LLM to identify and implement the optimal task plan from among the various possibilities.
Nonlinear Task Structures. During model execution, a model may need more than one inputs and each input need to be produced by a prerequisite model, resulting in nonlinear (tree) structure for the solution. In this context, employing a nonlinear task planning may enable more effective integration of the diverse inputs and more efficient parallel processing of the models to achieve the desired outcome. However, incorporating such nonlinear task planning ability into L.MSs presents unique challenges beyond the LLM's existing task-solving capabilities.
In consideration of the first two challenges, we introduce a mechanism referred to as Reinforcement Learning from Task Feedback (RLTF). This approach capitalizes on the performance feedback
15

---

procured from tasks following the execution of the solution devised by the LLM. Consequently, the RLTF mechanism effectively refines the LLM's planning strategy, resulting in an enhanced and more adaptive system. Indeed, relying solely input text for learning proves insufficient for LLMs when confronted with real-world tasks. Task feedback, on the other hand, supplies additional information that steers the learning trajectory of LLMSs towards improved and efficient solutions. For the third challenge, we propose Nonlinear Task Planning, which utilizes beam search as an efficient semi-autoregrrssive decoding method I 37 such that for each decoding step in beam search, different hypotheses are treated as parallel actionable solutions for different inputs instead of competing hypotheses. If a task requires parallel processing for multiple imputs, such as both text and image, then in generation time, an actionable solution taking text as input and another solution taking image imput will be generated and executed in parallel.
Table A.I: Language-relater models
Domain Task
Input Modality
Output Modality
Model
Sentiment Analysis
Text
Text
FinBert
Text Summarization
Text
Text
BART
Machine Translation
Text
Text
T5
34
Fill Mask
Text
Text
DistilRoherta
Question Answering
Text, Text
Text
DistilBERT
Table A.2: Vision-related models
Domain Task
Input Modality
Output Modality
Model
Image Classification
Image
Text
ViT
Object Detection
Image
Text
DETR
Colorization
Image
Image
Colorizer
Image Super-Resolution
Image
Image
Swin2SR
10
Image Denoising
Image
Image
Restormer
Image Deblurring
Image
Image
Restormer
531
Table A.3: Vision-language models
Domain Task
Input Modality
Output Modality
Model
Visual Question Answering
Image, Text
Text
GIT
48
Image Captioning
Image
Text
Vision Encoder Decoder
[1]
Text-to-Image Generation
Text
Image
Stable Diffusion
36
https///uuginggfac http:///huginngffe...ooyyyyggggu codyyiyanghuuus/hhbeet-oomm
"nttpss/huuggingfacc.. co/gpt22
'httss//hugggingface. co/t5 base
"bttp::/huugggngffacccoo////
wttpa//a...itiisiooiettttaea basc
bttp:///wwwwabaasaaaaahhheeee 16-24
http://whugggigggaac..cocccccbbs 101
12
"
16

---

A.2 Original Datasets
ImageNet-IK 38 is a large-scale image dataset, derived from the broader ImageNet database, containing approximately million images. These images are categorized into 1,000 distinct classes, with each class representing a specific object or concept. The dataset has been instrumental in the development and evaluation of sate-of-the-arr deep learning algorithms for image classification, object recognition, and transfer learning.
Common Objects in Context (COCO) 121 is a large-scale, richly-annotated image dataset designed to advance the fields of object detection, segmentation, and captioning. Released in 2014, it contains over 200,000 labeled images with 1.5 million object instances from 80 different object categories. The dataset features complex, real-world scenes with multiple objects per image, various object scales, and diverse contexts.
CNN/Daily Mail [26] is valuable resource for text summarization, which consists of human- generated abstractive summaries, created by transforming news articles from CNN and Daily Mail websites into questions, with one entity concealed, and generating summaries from the corresponding passages. The authors have made available the scripts used to crawl, extract, and generate meesstion-nswwer pairs from these websites. The corpus contains 286,817 training pairs, 13,368 validation pairs, and 11,487 test pairs, as defined by the scripts. On average, the source documents in the training set span 766 words across 29.74 sentences, while the summaries are composed of 53 words and 3.72 sentences.
Stanford Sentiment Treebank (SST2) 28 is corpus with labeled parse that allows for the analysis of the compositional effects of sentiment in language. The corpus consists of 11,855 single sentences extracted from movie reviews. It was parsed with the Stanford parser and includes a total of 215,154 unique phrases from those parse trees, each annotated by human judges.
benchmark for evaluating visual reasoning based on text present in images. In order to answer questions pertaining to the images, TextVQA necessitates models to read and reason about the text contained within them. The incorporation of text as new modality in images demands that models be able to reason over this modality to address TextVQA queries. Thus, Tex poses unique challenge for models integrate both visual and textual cues to arrive at comprehensive answer.
TextVQA
4 serves
Stanford Question Answering Dataset (SQuAD) 35 is collection of question-answer pairs sourced from Wikipedia articles. distinguishing characteristic of SQuAD is that the correct answers to the questions be any sequence of tokens in the corresponding text. This flexibility is a result of the dataset's construction through crowd-sourcing, which results in diverse set of questions and answers compared other question-answering datasets.
A.3 Data Augmentation Methods
Upon determining the raw datasets, our next objective is augment them from various perspectives to construct complex, multi-step tasks. For instance, we can introduce noise and reduce the resolution of an image from ImageNet-IK to create new datasets that may require "Image Denoising" and "Image Huper-Resolution for initial recovery before performing classification. The data augmentation methods employed are as follows:
Gaussian Blur is prevalent image processing technique that involves convolving an image with a Gaussian filter kernel. This filter is applied smooth the image and reduce noise, yielding a blurred output image.
Gaussian Noise refers the addition of Gaussian-distribute noise.
Grayscale entails converting the colorful image to a grayscale image.
Low Resolution pertains to images with a reduced pixel density (pixels per inch, or ppi).
Translation denotes the process of converting text from one language, such as English, to another, such as German. In this work, we only use Aeglish-to-Geemmal translator for simplicity.
Word Mask randomly replaces single word in a given sentence with the "[MASK]" token.
17

---

Task description
Input Sample
Output Sample
Given low-resolutioned noisy blurry grayscale image, how to return the regular image step by step?
Given low-resolution noisy blurry grayscale image, how return the object names in English step by step?
bear
big burly grizzly bear is show [Mask] grass in the background.
Given clozed English text, how to translate the text in German step by step?
Ein kräftiger Grizzly Bär ist im Hintergrund mit Gras zu sehen.
Given noisy blurry grayscale image and clozed English query, how to answer the question in English step by step?
22
Question: what number is [Mask] the player's jersey?
Context: Super Bowl
5 was an American football game to
Given clozed English document and clozed English query, how to answer the question in German step by step?
Goldener Jahrestag
determine the champion of the National...
Question: What was the theme of Super [Mask] 50?
Table A.4: Examples of multi-step tasks and their augmented data samples.
A.4 Evaluation Metrics
reference-free frree metric used to assess the correlation between generated image caption and the actual content of the image. Research has shown that it has strong correlation with human judgment and is a reliable measure for evaluating image captioning performance [16].
CLIP Score [6
BERT Score"! 17 uses contextual embeddings from the pre-trained BERT model to compare words in candidate and reference sentences through cosine similarity. Studies have shown that is highly correlated with human evaluation at both sentence-level and system-level (56). Additionally, BERT Score calculates precision, recall, and measure, making a valuable tool for evaluating various language generation tasks. In this work, we use the value of Fl score.
VIT Score metric designed to assess the visual similarity between images. By caleu- lating the cosine similarity of their respective embeddings, which are generated using a Vision Transformer, the VIT Score offers quantitative measure of their likeness.
httss//hugginggfccc http:://wagiggggiec.cccccccceele wwwpp///coollleeeeeeeeeeeeeeeee. 18 htwp:///claa.eeaacch..... wtwpaaaaa...c..oocooooooooaiagio http:://wwobbbo.st..........ooop
18

---

A.5 Dataset Documentation and Data Samples for Benchmark Tasks
Our dataset is designed to evaluate LLM's planning ability of using domain expert models. To accomplish this, we enhance the standard CV/NLP datasets using various combinations of data aug- mentation methodologies. We have devised 185 multi-step tasks total, of which 117 tasks maintain a linear task structure with steps following a simple sequence, while the remaining 68 tasks exhibit non- non-linear task structure, where steps might be performed concurrently or in a complex order. Each benchmark task is accompanied by small dataset, which contains 100 augmented data samples. All benchmark datasets can be accessed, reviewed, and downloaded via ht https: / /drive google com/dri ve fo de 1A 7 TM MMXchhUBBG T E1 5SnCPR57e, which is committed to transparency and ease accessibility. for any rights violation related to this dataset. The data license is Creative Commons Attribution 4.0 International, ensuring all necessary permissions and regulations are stringently adhered to. We have
the authors, we affirm that we assume all responsibility
The dataset is hosted GitHub chosen this platform considering its robustness, reliability, and its proven track record for data hosting. We ensure that access to the data will be maintained consistently, possibly through curated interface. maintenance plan is in place to address potential issues, provide necessary updates, and ensure the data's 's long-term term-termm& availability and integrity.
thub com/ agi Rggireseeacch/peeie
We also offer several data samples to illustrate the structure of the datasets further. For example, consider the third row of Tab. A.4 which represents machine translation domain task (i.e., translating from English to German).
this case, we apply the "Word Mask" augmentation technique on the text inputs to create multi-step task, which can be described as "Given clozed English text, how can the text be translated into German step by step?" For instance, given an original data sample, "A big burly grizzly bear is shown with grass in the background", the word "with" has been chosen to be masked to generate the augmented data sample, "A big burly grizzly bear shown [MASK] grass in the background".
A.6 Details of RLTF
In the setup of RLTF, the environment the OpenAGI platform and the agent is the LLM L parameterized with Φ. The solution generated by the LLM can be seen set of instructions that solve the input task / and can be executed on the corresponding augmented dataset De. We can use the performance (provided in Sec. 3.1.3) that dataset the reward signal R and use reinforcement learning to fine-tune the LLM. More concretely, to find the optimal solution, we require the LLM to maximize its expected reward on the training set Ttrains represented by J(D):
( Exxrainn-TTTT. (+) [
(A.1)
Since the reward signal R is non-differentiable, we need update &. In this work, we use the REINFORCE in [50] as follows,
policy gradient method to iteratively
V (Φ) Ps (0) V log P (Strain|4) R.
(A.2)
An empirical approximation of the above quantity is:
VoJ ( 2
Σ
Vo log ())raii(()) R
(A.3)
Terain
(ETerava
The above update is unbiased estimate for our gradient, but has a very high variance. To reduce the variance this estimate, we employ a baseline function b, which the moving average of the previous reward signals:
V (J)
Σ tETirain
V log P(Strain b) (R b)
(A.4)
T Trrainl
A.7 Constrained Generation
To generate the solution for natural language task description, we require the L.M to generate an actionable solution consisting of sequences of model names. For tasks that require only one input,
19

---

the model only needs to generate one actionable sequence of models. For tasks that require multiple inputs. such as Visual Question Answering. the LLM needs multiple steps in order accomplish the task, where each step is either a sequence of models or parallel of several sequences of models. Towards this end, the LLM must satisfy three conditions: 1) only generate the model names without irrelevant tokens, 2) generate valid sequences of models, and 3) generate paralleled sequences of models for different inputs when necessary.
Condition 1: For the LLM to generate only model names, instead of tuning the model to teach it what names are available, we adopt constrained beam search [Л]. which only allows generating tokens from the model set M at every decoding step. More specifically, we define our constraints as a prefix trie such that each model name a path from the root to some leaf node. For each node in the tree, its children indicate all the allowed continuations from the prefix defined traversing the trie from the root to f. Thus in each decoding step, the next token can only be selected from either all possible continuations allowed based on the generated tokens or the first tokens of all possible next model names. For example, if "Text" already generated, based on the set of model names, the next tokens can only be either "Summarization" due to the "Text Summarization" model or "Generation" due to the "Text Generation" model, as shown in Fig. A.2
Summarization
Text
to Image Generation
Figure A.2: Model name based constrained generation.
For the LLM generate valid sequences of models, consecutive models should have input and output modalities matched. If the output modality of a model is text, then the next model can only be models that take text as input. This also achieved by constrained beam search such that when finishing generating one model, the constraint function will determine the output modality of this model and find out all possible next models in model set M, excluding the models that are already generated. It will dynamically construct a new trie for all these model names based on the output modality. For example, the first generated model name is "Text Summarization", then the next possible models can be "Sentiment Analysis". "Text Generation", etc., as shown in Fig. A.3
Condition 2:
Sentiment Analysis
Text to Image Generation
Text Summarization,
Machine Translation
Fill Mask
Figure A.3: Model type based constrained generation.
A.8 Zero- and Few-shot Schema
As in shown in Fig. A.4 in the zero-shot setting, most LLMS struggle generate valid task plans, let alone optimal solutions. In particular, GPT-3.5 tends to generate repetitive contents, which subsequently maps identical model names. Meanwhile, Vicuna-7B and -lan-e constrained by their zero-shot capabilities, fail to produce a reasonable plan. In the few-shot setting, we incorporate several manually labeled task plans as instructions to guide the generation, resulting in remarkable improvement in the quality of the task plans. As observed in Fig. AA55 all three LLMs can produce solutions that are semantically similar to the provided examples. fact, many solutions can be used directly, even without the need for mapping.
A.9 Broader Impacts, Limitations, and Future Visions
Just like any technology, the irresponsible use of AI techniques and intelligent systems may have detrimental effects on individuals and society as a whole. In particular, existing Large Language Models (LLMs) are not sufficiently designed to ensure their harmless usage, making them vulnerable to misuse by malicious parties. Consequently, it is important to address and mitigate the potential risks associated with L.MMs when used for complex task solving. Our constrained generation framework provides potential solution to this issue. By incorporating ethical constraints, such as an AI
20

---

Prompt
low-resolution, blurry, grayscale image, how return the regular image step step?
higher resolation version of the blurry image. Step 2: Use deblarring algorithm to remove blucriness and increase the sharpness of the image. Step 3: Apply denoising algorithm to reduce any noise the image that may have been introduced during deblurring process.
1: Collect data images
Step a: Use upsampling method to increase the resolation of the image to the desired level. enhancement technique make image more visually appealing. Step 6: Finally, apply sharpening techniques to add crispness and detail to the image.
Step 5: Apply
the image generale mask image. 2. Use the mask image to gemerate the masked image. Use the masked image generate the final image. 4. Use the masked image to generate the final image. Use
The blurry Image blurry image with blurry background. To return the regular the fallowing steps: First, you Then, you need to find the blurry image. Then, you need to find the regular image. Then, you need to find the blurry image. Then, you need find the regular image. Then, ...
use
T5
blurry image. Then, you
find regular image.
Figure A.4: An example of zero-shot schema.
Few-shot Prompt
Problem: Given low-resolutioned noisy blurry gray image, how return the regular image stop by step?
Solution:
Step Image Super Resolution,
Stop 2: Image Denoising,
Stop Image Deblurring,
Stop Colorization,
Problem:
Problem:
English text, to generate English step step?
Solution:
Step Fill Mask,
Step 2: Text Generation,
Problem: low-resolationed noisy blurry grayscale image, regular image step by step?
Soltuion:
1:
Image Super Resolution,
Step 2: Image Deblurring,
Stop 3: Emmage Denolsing.
Step 1: Image Super Resolution, Step 2: Image Deblurring,
Step 3: Colorization, Image Classifiction, Step 5: Machine Translation,
Image Super Resolution, Image Deblurring, Step 3: Colorization.
Step 1:
Stop 2:
T5
Step 4: Image Captioning, Step Machine Translation,
Figure A.5: An example of few-shot schema.
constitution, into the plan generation process, we can guide the agent to generate plans that are both ethically sound and benign while tackling complex tasks.
In the future, we can incorporate multiple models within each single-step task, thereby providing an expanded selection of options for LLMs to address complex problems. Additionally, we can integrate datasets from alternative modalities, such as video and audio, into OpenAGI platform. These datasets will facilitate the development of more sophisticated tasks to further investigate the planning capabilities of L.L.s. We will also explore multiple sask-solving agents interact with each other for complex problem solving. Another promising direction is to involve humans in the loop during the
21

---

resolution of complex tasks, In such scenarios, LLM may prompt human experts for answers one step of the task-solving plan when suitable model is unavailable, thus enabling better Human-Al collaboration. Lastly, we can explore automated task generation techniques that empower OpenAGI to generate complex tasks independently, facilitating self- self-prompting and self-improvement in its task-solving capabilities.
A.10 Computational Resources
For augmenting the data, we used devices equipped with Intel(R) (eon(R) Gold 6226R CPU @ 2.90GHz and 256 GB RAM. For training and testing the LLMs, used 4xA5000-24GB GPUs.
A.11
Training Details
In our experiments, we use Low-Rank Adaptation (LoRA) 19 17 for efficient fine-tuning of Flan- T5-Large, Vicuna-7B and LLaMA-2-13B under both Fine-tuning and RLTF schema, and the coniguraaion/yyyer-paraeee settings are shown in Tab. A.5
Table A.5: Configuration and parameter settings for Plan-T--aa Large,,, Vicuna-7B and LLaMA-2-13B
Flan-T5-Laam
Vicuna-7B
LLaMA-2-138
Confguraaioo//typee-praaaam
Fine-tuning
RLTF
Finc-tuning
RLTF
Fine-tuning
RLTF
Optimizer
AdamW
AdamW
AdamW
AdamW
AdamW
AdamW
Epochs
200
10
200
10
200
10
Training Batch Size Per GPU
8
5
1
1
Gradient Accumulation Steps
1
1
Learning Rate
le-5
Ic-5
5c-6
5c-6
lc-5
lc-5
Weight Decay
le-6
le-6
le-6
le-6
le-6
le-6
Warmup Ratio
0.1
0.1
0
0
Scheduler
Linear
Linear
Linear
Linear
Linear
Linear
LoRA_r
8
8
8
8
LoRA
16
16
16
16
LoRA. dropout
0.05
0.05
0.05
0.05
0.05
0.05
0.2
0.2
0.2
Decay Rate of
0.9
0.9
0.9
Beam Size
30
20
20
Num Outputs
30
20
Top k
5
40
40
Top
0.5
0.75
0.75
Temperature
0.9
0.2
0.2
Num Beam Groups
Table A.6: Task descriptions of all multi-step tasks in OpenAGI. The difficulty level is estimated by the size of human-labeled solutions, that is, the total number of models used in the human-labeled task solution. The higher the number, the more difficult the task.
Task Description
Difficulty Level
Given low-resolutioned noisy blurry grayscale image how to return the regular image step by step?
Given noisy blurry grayscale image
3
how to return the regular image step by step?
Given low-resoll rewouusolut blurry grayscale image
how to return the regular image step by step?
Given blurry grayscale image
2
how to return the regular image step by step?
Given low- resolutiod noisy grayscale image
3
how to return the regular image step by step?
Continued on next page
Panps//hugginggaee.c////ggg/
22

---

Table A.6 - continued from previous page
Task Description
Difficulty Level
Given noisy grayscale image
2
how to return the regular image step by step?
Given low-resolutioned grayscale image
2
how to return the regular image step by step?
Given grayscale image
how to return the regular image step by step?
Given low-resolttioned noisy blurry image
3
how to return the regular image step by step?
Given noisy blurry image
2
how to return the regular image step by step?
Given low-resolutioned blurry image
2
how to return the regular image step by step?
Given blurry image
1
how to return the regular image step by step?
Given low-resolutioned noisy image
2
how to return the regular image step by step?
Given noisy image
1
how to return the regular image step by step?
Given rowrrusoluto image
how to return the regular image step by step?
Given row-rssolt noisy blurry grayseale image how to return the caption in German step by step?
5
Given low-resolutionck noisy blurry grayscale image how to return the class label in German step by step?
6
Given low-resolutioned noisy blurry grayscale image how to return the object names in German step by step?
6
Given low-resolutioned noisy blurry grayscale image
5
how to return the caption in English step by step?
Given reoouuiott noisy blurry grayscale image how to return the class label in English step by step?
5
Given resouutolte noisy blurry grayscale image
5
how to return the object names in English step by step?
Given noisy blurry grayscale image
5
how to return the caption in German step by step?
Given noisy blurry grayscale image
5
how to return the class label German step by step?
Given noisy blurry grayscale image
5
how to return the object names in German step by step?
Given noisy blurry grayscale image
4
how to return the caption in English step by step?
Given noisy blurry grayscale image
4
how to return the class label in English step by step?
Given noisy blurry grayscale image
4
how to return the object names in English step by step?
Given low- rowrrssolu blurry grayscale image
5
how to return the caption in German step by step?
Given low-resolutioned blurry grayscale image
5
how to return the class label in German step by step?
Given low-resolutioned blurry grayscale image
5
how to return the object names German step by step?
Given resouuoolt blurry grayscale image
4
how to return the caption in English step by step?
Given low-tusolut blurry grayscale image
4
how to return the class label in English step by step?
Given low-resolutioned blurry grayscale image
4
how to return the object names in English step by step?
Given blurry grayscale image
4
how to return the caption in German step by step?
Given blurry grayscale image
4
how to return the class label in German step by step?
Continued on next page
23

---

Table A.6 - continued from previous page
Task Description
Difficulty Level
Given blurry grayscale image
4
how to return the object names in German step by step?
Given blurry grayscale image
3
how to return the caption in English step by step?
Given blurry grayscale image
3
how to return the class label in English step by step?
Given blurry grayscale image
3
how to return the object names in English step by step?
Given low-resolutioned noisy grayscale image
5
how to return the caption in German step by step?
Given low-resolutioned noisy grayscale image
5
how to return the class label in German step by step?
Given rewressolutoned noisy grayscale image
5
how to return the object names in German step by step?
Given low-resolutioned noisy grayscale image
4
how to return the caption in English step by step?
Given low-resolutioned noisy grayscale image
4
how to return the class label English step by step?
Given oowrresolutooned noisy grayscale image
4
how to return the object names in English step by step?
Given noisy grayscale image
4
how to return the caption in German step by step?
Given noisy grayscale image
4
how to return the class label in German step by step?
Given noisy grayscale image
4
how to return the object names in German step by step?
Given noisy grayscale image
3
how to return the caption in English step by step?
Given noisy grayscale image
3
how to return the class label in English step by step?
Given noisy grayscale image how to return the object names in English step by step?
3
Given low-resolutioned grayscale image
4
how to return the caption in German step by step?
Given low-resolutioned grayscale image
4
how to return the class label German step by step?
Given low-resolutioned grayscale image
4
how to return the object names in German step by step?
Given low-resolutioned grayscale image
3
how to return the caption in English step by step?
Given rewrrusolu grayscale image
3
how to return the class label in English step by step?
Given low-resolutioned grayscale image
3
how to return the object names in English step by step?
Given grayscale image
3
how to return the caption in German step by step?
Given grayscale image
3
how to return the class label in German step by step?
Given grayscale image
3
how to return the object names in German step by step?
Given grayscale image
2
how to return the caption in English step by step?
Given grayscale image
2
how to return the class label in English step by step?
Given grayscale image
2
how to return the object names in English step by step?
Given low-resolutioned noisy blurry image
5
how to return the caption in German step by step?
Given resouuoout noisy blurry image
5
how to return the class label in German step by step?
Continued on next page
24

---

Table A.6 - continued from previous page
Task Description
Difficulty Level
Given low-resolutioned noisy blurry image
5
how to return the object names in German step by step?
Given low-resolutioned noisy blurry image
4
how to return the caption in English step by step?
Given low-resolutioned noisy blurry image
4
how to return the class label in English step by step?
Given low-resolutioned noisy blurry image
4
how to return the object names in English step by step?
Given noisy blurry image
4
how to return the caption in German step by step?
Given noisy blurry image
4
how to return the class label in German step by step?
Given noisy blurry image
4
how to return the object names in German step by step?
Given noisy blurry image
3
how to return the caption in English step by step?
Given noisy blurry image
3
how to return the class label English step by step?
Given noisy blurry image
3
how to return the objecl names in English step by step?
Given row-rssuuti blurry image
4
how to return the caption in German step by step?
Given low-resolutionck blurry image
4
how to return the class label in German step by step?
Given low-resolutioned blurry image
4
how to return the object names in German step by step?
Given low-resolutioned blurry image
3
how to return the caption in English step by step?
Given reoouutott blurry image
3
how to return the class label in Fnglish step by step?
Given resouuiold blurry image
3
how to return the object names in English step by step?
Given blurry image
3
how to return the caption in German step by step?
Given blurry image
3
how to return the class label German step by step?
Given blurry image
3
how to return the object names in German step by step?
Given blurry image
2
how to return the caption in English step by step?
Given blurry image
2
how to return the class label in English step by step?
Given blurry image
2
how to return the object names in English step by step?
Given low-resolutioned noisy image
4
how to return the caption in German step by step?
Given low-resolutioned noisy image
4
how to return the class label in German step by step?
Given low-resolutioned noisy image
4
how to return the object names in German step by step?
Given resouutolt noisy image
3
how to return the caption in English step by step?
Given low-rssolutioned noisy image
3
how to return the class label in English step by step?
Given low-resolutioned noisy image
3
how to return the object names in English step by step?
Given noisy image
3
how to return the caption in German step by step?
Given noisy image
3
how to return the class label in German step by step?
Continued on next page
25

---

Table A.6 - continued from previous page
Task Description
Difficulty Level
Given noisy image
3
how to return the object names in German step by step?
Given noisy image
2
how to return the caption in English step by step?
Given noisy image
2
how to return the class label in English step by step?
Given noisy image
2
how to return the object names in English step by step?
Given low-resolutioned image
3
how to return the caption in German step by step?
Given low-resolutioned image
3
how to return the class label in German step by step?
Given reweesolutione image
3
how to return the object names in German step by step?
Given low-resolutioned image
2
how to return the caption in English step by step?
Given low-resolutioned image
2
how to return the class label English step by step?
Given low-resolutooned image
2
how to return the object names in English step by step?
Given clozed English text
2
how to generate a image step by step?
Given English text
how to generate a image step by step?
Given clozed English text
3
how to return the summarization in German step by step?
Given clozed English text
2
how to translate the text in German step by step?
Given clozed English text
3
how to return the sentiment in German step by step?
Given clozed English text
2
how to return the summarization in English step by step?
Given clozed English text
2
how to return the sentiment in English step by step?
Given English text
2
how to return the summarization in German step by step?
Given English text
how to translate the text in German step by step?
Given English text
2
how to return the sentiment in German step by step?
Given English text
1
how to return the summarization in English step by step?
Given English text
how to return the sentiment in English step by stop?
Given row-resolu noisy blurry grayscale image and clozed English query
6
how to answer the question in English step by step?
Given low-resolutioned noisy blurry grayscale image and clozed English query how to answer the question in German step by step?
7
Given low-resolutioned noisy blurry grayscale image and English query how to answer the question English step by step?
5
Given resouuool noisy blurry grayscale image and English query how to answer the question in German step by step?
Given noisy blurry grayscale image and clozed English query
5
how to answer the question in English step by step?
Given noisy blurry grayscale image and clozed English query
6
how to answer the question in German step by step?
Given noisy blurry grayscale image and English query
4
how to answer the question in English step hy step?
Given noisy blurry grayscale image and English query
5
how to answer the question in German step by step?
Continued on next page
26

---

Table A.6 - continued from previous page
Task Description
Difficulty Level
Given low-resolutioned blurry grayscale image and clozed English query how to answer the question in English step by step?
5
Given low-resolutioned blurry grayscale image and clozed English query
6
how to answer the question in German step by step?
Given low-resolutioned blurry grayscale image and English query
4
how to answer the question in Figlishh step by step?
Given low-resolutioned blurry grayscale image and English query
5
how to answer the question in German step by step?
Given blurry grayscale image and clozed English query
4
how to answer the question in English step by step?
Given blurry grayscale image and clozed English query
5
how to answer the question in German step by step?
Given blurry grayscale image and English query
3
how to answer the question in English step by step?
Given blurry grayscale image and English query
4
how to answer the question in German step by step?
Given low-resolutioned noisy grayscale image and clozed English query how to answer the question English step by step?
4
Given low-resolutioned noisy grayscale image and clozed English query how to answer the question in German step by step?
6
Given low-resolutioned noisy grayscale image and English query
4
how to answer the question in English step by step?
Given rewruusolutoned noisy grayscale image and English query
5
how to answer the question in German step by step?
Given noisy grayscale image and clozed English query
5
how to answer the question in English step by step?
Given noisy grayscale image and clozed English query how to answer the question in German step by step?
5
Given noisy grayscale image and English query how to answer the question in English step by step?
3
Given noisy grayscale image and English query how to answer the question German step by step?
4
Given low-resolutioned grayscale image and clozed English query how to answer the question in English step by step?
4
Given low-resolutioned grayscale image and clozed Englishh query
5
how to answer the question in German step by step?
Given low-resolutioned grayscale image and English query
3
how to answer the question in English step by step?
Given low-resolutioned grayscale image and English query
4
how to answer the question in German step by step?
Given grayscale image and clozed English query
4
how to answer the question in English step by step?
Given grayscale image and clozed English query
5
how to answer the question in German step by step?
Given grayscale image and English query
2
how to answer the question English step by step?
Given grayscale image and English query
3
how to answer the question in German step by step?
Given low-resolutioned noisy blurry image and clozed English query how to answer the question English step by step?
4
Given resouuoout noisy blurry image and clozed English query how to answer the question in German step by step?
5
Given low-resolutioned noisy blurry image and English query
4
how to answer the question in English step by step?
Given low-resolutioned noisy blurry image and English query
5
how to answer the question in German step by step?
Given noisy blurry image and clozed English query
4
how to answer the question in English step hy step?
Given noisy blurry image and clozed English query how to answer the question in German step by step?
5
Continued on next page
27

---

Table A.6 - continued from previous page
Task Description
Difficulty Level
Given noisy blurry image and English query
3
how to answer the question in English step by step?
Given noisy blurry image and English query how to answer the question in German step by step?
4
Given low-resolutioned blurry image and clazed English query
4
how to answer the question in English step by step?
Given low-resolutioned blurry image and clozed English query
5
how to answer the question in German step by step?
Given low-resolutioned blurry image and English query
3
how to answer the question in English step by step?
Given low-resolutioned blurry image and English query
4
how to answer the question in German step by step?
Given blurry image and clozed English query
3
how to answer the question in English step by step?
Given blurry image and clozed English query
4
how to answer the question in German step by step?
Given blurry image and English query
2
how to answer the question in English step by step?
Given blurry image and English query
3
how to answer the question in German step by step?
Given low-resolutioned noisy image and clozed English query
4
how to answer the question in English step by step?
Given rewrrssolt noisy image and clozed English query
5
how to answer the question in German step by step?
Given low-resolutioned noisy image and English query
3
how to answer the question in English step by step?
Given low-resolutioned noisy image and English query how to answer the question in German step by step?
4
Given noisy image and clozed English query how to answer the question in English step hy step?
3
Given noisy image and clozed English query
4
how to answer question German step by step?
Given noisy image and English query
3
how to answer the question in English step by step?
Given noisy image and English query
4
how to answer the question German step by step?
Given oow-resolutioned image and clozed English query
3
how to answer the question in English step by step?
Given low-resolutioned image and clozed English query how to answer the question in German step by step?
4
Given rewrresolu image and English query
2
how to answer the question in English step by step?
Given low-resolutioned image and English query
3
how to answer the question in German step by step?
Given clozed English document and clozed English query
4
how to answer the question German step by step?
Given clozed English document and clozed English query
3
how to answer the question in English step by step?
Given clozed English document and English query
3
how to answer the question in German step by step?
Given clozed English document and English query
2
how to answer the question in English step by step?
Given English document and clozed English query
3
how to answer the question in German step by step?
Given English document and clozed English query
2
how to answer the question in English step by step?
Given English document and English query
2
how to answer the question in German step by step?
Given English document and English query
1
how to answer the question in English step hy step?
28

---

Prompt-1
You planner who is expert coming up with to list for given objective for the execution of robot. Ensure the list is short as possible. Each task in relevant, effective, short and necessary. The robot only allowed to utilize the provided machine learning models do each task. Develop to list achieve the objective: Given noisy, blurry, grayscale image and English question related to that image, how to answer the question in German?
Provided models:
Sentiment Analysis
Text Summarization
Machine Translation
Fill Mask
Question Answering
Image Classification
object Detection
Colorization
Image ResputReeoo
Image Denoising
Image Deblurring
Visual Question Answering
Image Captioning
Text-to ti--to--maee Generation
Prompt-2
You are planner who is an expert at coming up with a to-do list for given objective for the execution of robot. Ensure the list is as short as possible, Each task in it is relevant, effective, short and necessary. The robot is only allowed to utilize the provided machine learning models to do each task. Develop to-do list to achieve the objective: Given noisy, blurry, grayscale image and English question related to that image, how to the question in German?
Provided models:
Sentiment Analysis: useful when you want to analyze the sentiment of sentence. It receives sentence as input. Text Summarization: useful when you want to summarize a sentence or paragraph. It receives text as input. Machine Translation: useful when you want to translate a sentence. It receives text as input.
Fill Mask: useful when you want to fill the sentence at the masked position. It receives text input.
Question Answering: useful when you need to
question based given context.
to know the class of the image. It receives image. path input. object Detection: useful when you want detect the objects a photo. receives image_path as input. Colorization: useful when you want to colorize photo. receives image f_tethhhhhhhh as input.
Image Classification; useful when you
Image Super-Resolution
Resolution: useful when you want to create high-resolution image from a wwwreeslution image.
Image Denoising: useful when you want to denoise noisy image. It receives image as input.
Image Deblurring: useful when you want to deblur blurry image. It receives image input.
Visual Question Answering: useful when you need to answer question based on given image.
Image Captioning: useful when you want to know what is inside the photo. It receives image input. Text-to-Image Generation: useful when you want generate image based given description.
Figure A.6: Prompts used for experiments in Tab. and Tab.
29

---

Instruction
Prompt
LLM
Write travel report trip to China, including recommendatioss for sights, activities, and local cuisine, well as practical information for travelers, such as how to stay safe and healthy and how navigate the country.
You are a planner who an expert at coming with todo list for a given objective. each task, uttlize one the provided only when needed.
Ensure the list short possible, and tasks Develop detailed to-do list to achieve the objective: Write travel report for a trip to chins, including
relevant, effective and short.
recommenddatioss sights, activities, local cuisine, well practical information for travelers, as stay safe and healthy and how navigate the country.
T5
Provided Loods:
Domain Expert Models
Search: ideal for auswering questions related to curvent
gathering information.
A1
image Captioning Add informative captions to a photo.
Machine Transhation: useful when you need translaic the original language into English. Google Places: Useful for when you need validate discover addresses from ambiguous
O
Plan Execution
Result
Below are some tips to help you stay safe while traveling in China...
1) Research safety tips & precautions while travelling in China (use search).
China, officially the People's ss Republic of China, country in East Asia...
2) Gather info on recommended sightseeing spots & popular attractions in China (search).
Search
Best Time to Travel to China While the best time travel weather-wise beeween May and June September and October, you are also likely to find crowds and delays during these periods.
3) koo best time to visit China based on weather conditions (search).
The forbidden city, Beijing
4) Find out about Chinese culture & customs (image captioning)
to Image
The Potala
5) Learn basic phrases in Mandarin that will be helpful during your journey (machine translation).
Hello Good morning Nice to meet you
Machine Translation
6) Plan restaurrsssssssss based local cuisines available in different regions of china (google places).
These eight culinary cuisines are Anhui, Cantonese, Fujian, Human,jiangsu, Shandong, Szechuan and Zhejiang.
7) Create budget plan for expenses like transportation, accommodation etc. (search).
Gongle Places
step-by guide to creating travel budget that works no matter where you're going and for how long! Plan your travel better & enjoy your holidays!
8) Prepare necessary documents required before leaving home (passport, visa, insurance etc.)search).
Required Documents Make two copies of all your travel documents in case of emergency.
Search
Figure A.7: Another example of open-ended task. OpenAGI is instructed to generate a travel report. The backbone LLM used in this example is Vicuna-7B.
30


Reviews:
[{"review_id": "uWNhSUBik1", "reviewer": "Reviewer_6ZWy", "summary": "", "questions": "", "limitations": "The discussion of limitations should be moved to the main text.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The topic of LLM-based agents is important and worth-exploring", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "mUoVZzQMES", "reviewer": "Reviewer_Zkzc", "summary": "", "questions": "", "limitations": "Please refer to \"Opportunities For Improvement\"\n", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The paper proposes an interesting open-source platform, OpenAGI, to evaluate the LLMs' ability to do planning and problem-solving for multi-step tasks.\n2. The quantitative benchmarks proposed offers a good way to evaluate the LLMs.\n3. The RLTF method is very interesting and effective based on the evaluation result. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "B7UYOlpn27", "reviewer": "Reviewer_fmdC", "summary": "", "questions": "", "limitations": "I think the authors could do a better job at addressing potential limitations and negative impacts. For example, I think more comprehensive testing could be done as to the negative things that can be done with the current framework. I also think the detrimental effects mentioned in the limitations should be significantly expanded upon and have research to back it up, especially if this technique is a path towards AGI.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "- RLTF is original, novel, and works well\n    - Although, it is very similar to RLHF, so I believe that RLHF should be discussed. It is still a novel contribution though.\n- Useful platform/tool for developers\n-Benchmarked existing LLMs", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "XH67vEOLDE", "reviewer": "Reviewer_9AkB", "summary": "", "questions": "", "limitations": "The authors have addressed the limitations. ", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The authors open-source the benchmark, includes code, data, evaluation methods and demo. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "eBkEsZamqY", "reviewer": "Reviewer_cNMX", "summary": "", "questions": "", "limitations": "The authors spend a short paragraph addressing the limitations in the supplementary material, but I think they can spend more effort discussing the limitations of the benchmark in terms of the kinds of tasks it considers, the lack of human evaluations for open-ended tasks, the lack of investigation into how RLTF performs well and what the outputs of T5-flan work, and whether the training method would perform well under distribution shift.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "Open-domain model synthesis is an exciting research area that definitely lacks benchmarks, and so this benchmark platform is very relevant and necessary. The paper is easy to read and mostly clear, especially in the graphical representation of the platform. There are quite a large range of tasks, and the method of creating tasks from data augmentation techniques and combining existing expert models is interesting and could be useful in other settings as well.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The paper proposes OpenAGI, a framework for testing and training existing and future AGI models (here, for example, those that can take in or output either text or images).
OpenAGI is based on the idea that large language models (LLMs) now can read task instructions in a sentence (e.g. "restore this image", and generate a list of steps for solving the task (akin to AutoGPT). The framework is a closed loop that starts from (1) reading task instructions; (2) generating a list of steps toward the solution; (3) utilizing existing tools to execute each step; and (4) generating results.
In such a closed loop, LLMs can be fine-tuned using a set of objective measures defined by the generated results, i.e. no humans in the loop or Reinforcement Learning from Task Feedback (RLTF).

This paper is more like ViperGPT and AutoGPT, yet, applied to testing and training AI models themselves.
The work is quite thorough: exploring 6 different input/output settings; 6 different LLMs. The framework also shows that significant performance improvements can be observed for Vicuna-7b and Flan-T5-Large when fine-tuning or using RLTF compared to zero-shot and few-shot learning methods.

The work is solid and thorough. The framework and its code can practically contribute to the community.
Decision: Accept (Poster)