SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yingqiang Ge
Wenyue Hua
Kai Mei
Jianchao Ji
Juntao Tan
Shuyuan Xu
Zelong Li
Yongfeng Zhang*
Human Intelligence (HI) excels at combining basic skills to solve complex tasks. This capability is vital for Artificial Intelligence (AI) and should be embedded in comprehensive AI Agents, enabling them to harness expert models for complex task-solving towards Artificial General Intelligence (AGI). Large Language Models (LLMs) show promising learning and reasoning abilities, and can effectively use external models, tools, plugins, or APIs to tackle complex problems. In this work, we introduce OpenAGI, an open-source AGI research and development platform designed for solving multi-step, real-world tasks. Specifically, OpenAGI uses a dual strategy, integrating standard benchmark tasks for benchmarking and evaluation, and open-ended tasks including more expandable models, tools, plugins, or APIs for creative problem-solving. Tasks are presented as natural language queries to the LLM, which then selects and executes appropriate models. We also propose a Reinforcement Learning from Task Feedback (RLTF) mechanism that uses task results to improve the LLM's task-solving ability, which creates a self-improving AI feedback loop. While we acknowledge that AGI is a broad and multifaceted research challenge with no singularly defined solution path, the integration of LLMs with domain-specific expert models, inspired by mirroring the blend of general and specialized intelligence in humans, offers a promising approach towards AGI. We are open-sourcing the OpenAGI project's code, dataset, benchmarks, evaluation methods, and the UI demo to foster community involvement in AGI advancement: https://github.com/agiresearch/OpenAGI.
The acquisition and reuse of skills is a fundamental aspect of human intelligence that enables the formation of complex skills to address novel or intricate problems [19, 4, 57]. We posit that machine intelligence should incorporate this capacity to synthesize various skills by composing them into complex skills for complex task-solving. In computer science parlance, each skill is referred to as a domain expert "model" - a reusable tool, module, network, plugin, or API with a defined function. The domain expert models can be synthesized into a larger "plan" for performing more complex tasks. The model synthesis process is adaptable to the input or task, such that for a given task, the models are synthesized into the most suitable plan to address the task at hand. As a result, different inputs or tasks may necessitate distinct synthesized models as a plan for task-solving.
Recent advances in Large Language Models (LLMs) have showcased exceptional learning and reasoning capabilities, rendering them well-suited for selecting, synthesizing, and executing external expert models to address complex tasks. These LLMs, such as GPT series [32, 2], LLaMA series [45, 44] and T5 series [33, 8], have exhibited a profound understanding of natural language and the
* {yingqiang.ge,wenyue.hua,kai.mei,jianchao.ji,juntao.tan,shuyuan.xu,zelong.li,yongfeng.zhang} @rutgers.edu
Figure 1: An example of benchmark tasks, which shows the OpenAGI pipeline. OpenAGI generates a task-solving plan for the input task described in natural language (using GPT-3.5 in this example, but can be other LLMs such as GPT-4, Vicuna, Flan-T5, Claude-2, and LLaMA-2), executes the plan with domain expert models, tools, APIs, and then conducts evaluation for the plan execution results.
ability to generate coherent and contextually relevant responses. This has opened up new possibilities for their application in complex tasks involving multi-modality data, such as image and text processing, as well as the integration of domain-specific knowledge. In this process, LLMs play a crucial role as they can understand and generate natural language, which helps AI to better comprehend and handle various problems. By integrating knowledge and skills from different domains, Open-domain Model Synthesis (OMS) holds the potential to drive the development of artificial general intelligence (AGI), enabling AI to solve a diverse array of problems and tasks. Despite acknowledging the complexity and lack of a defined path towards AGI, the combination of LLMs and domain-specific expert models, inspired by the interplay of general and specialized intelligence in humans, provides a promising direction [19]. However, the current research field, despite initial attempts, presents several significant challenges: 1) Extensibility: Several existing works employ a fixed number of models, such as WebGPT [25] and ToolFormer [40], resulting in difficulties when attempting to expand their capabilities; 2) Nonlinear Task Planning: The majority of current research is limited to solving tasks with linear task planning solutions [49, 18], meaning that each sub-task must be completed before the next sub-task can start. However, linear planning of models may not suffice for solving complicated tasks, besides, many tasks involve multiple multi-modal inputs; 3) Quantitative Evaluation: Many existing works only provide qualitative results, such as HuggingGPT [42]. This makes it difficult to assess the planning capabilities of LLMs to determine whether the strategies employed are optimal.
In order to mitigate the above limitations, we develop a platform that encompasses a diverse array of domain-specific expert models and intricate multi-step tasks with single or multiple multi-modal inputs. Furthermore, to promote the community's long-term advancement and assessment of AGI's
abilities, we open-source all code and datasets, and hence, name this platform OpenAGI. A toy example, showing the entire pipeline of OpenAGI, is depicted in Fig. 1. Specifically, 1) a natural language instruction of a specific task is given; 2) the instruction is augmented by manually designed prompt and then fed as input into LLM to generate a plan; 3) the expert models are selected and synthesized based on the generated plan, and subsequently executed to process the data samples; 4) the task-solving ability of the LLM can be evaluated by comparison between the output and the ground-truth labels or through human evaluation.
OpenAGI embodies a dual approach to address diverse requirements—benchmark tasks and open-ended tasks. On the one hand, we have incorporated benchmark tasks, each supported by task-specific datasets and evaluation metrics. This inclusion provides researchers with a consistent platform to assess and compare the performance of various models, stimulating continuous improvement and competitive innovation. For benchmark tasks, as depicted in Fig. 1, we utilize a selection of expert models derived from esteemed libraries such as Hugging Face's transformers and diffusers, as well as from GitHub repositories, thereby easily facilitating the expansion of our model set. Additionally, the datasets have been meticulously selected to align with or resemble the training datasets of the respective models. We then implement a variety of data augmentation techniques to enhance these original datasets, enabling the construction of sophisticated multi-step tasks designed to assess the planning and task-solving capabilities of a given LLM. On the other hand, OpenAGI also offers open-ended tasks that utilize a variety of expandable models. These tasks open the door to creativity and imaginative problem-solving, enabling the exploration of innovative solutions that may not emerge within more constrained task frameworks. For open-ended tasks, as depicted in Fig. 2, which is designed to accommodate a broader spectrum of needs, we further include LangChain to provide additional expert models, such as Google Search, Wikipedia, Wolfram Alpha and so on. Indeed, relying solely on input text for learning proves insufficient for LLMs when faced with real-world tasks. In order to improve its performance, we introduce a mechanism referred to as Reinforcement Learning from Task Feedback (RLTF). This approach capitalizes on the performance feedback procured from tasks following the execution of the solution devised by the LLM. Consequently, the RLTF mechanism effectively refines the LLM's planning strategy, resulting in an enhanced and more adaptive system. In summary, the key contributions of the work include:
- We introduce OpenAGI, an AGI research platform, specifically designed to offer complex, multi-step tasks accompanied by their respective datasets, evaluation methods, and a diverse range of extensible models which can be synthesized to effectively solve these tasks. The purpose of this platform is to aid in the quantification of the overarching planning and task-solving abilities of LLMs. OpenAGI embraces AGI by focusing on LLM-driven, (open-domain) model synthesis, predominantly utilizing models and datasets on Hugging Face, GitHub and LangChain.
- We propose the LLM+RLTF approach for OpenAGI, which leverages a Large Language Model as a controller to select, synthesize and execute various external expert models for complex task-solving. The feedback obtained from these tasks is then employed to refine the LLM's planning strategy, thereby enhancing the LLM's overall performance and task-solving ability.
- We evaluate both open-source and closed-source LLMs with differing scales under distinct learning schema and the OpenAGI pipeline. Our findings suggest that even smaller-scale LLMs, when paired with an appropriate learning schema such as RLTF, are able to possess the potential to outperform competitors that equip a significantly greater magnitude of model parameters.
2.1 Large Language Model and AI Agents
With the advancement of highly parallelizable transformer architectures, pre-trained language models (PLMs) have demonstrated remarkable capabilities in comprehending, generating, and manipulating natural language [31, 24]. These models are pre-trained on a large corpora of text data and commonly fine-tuned for specific downstream tasks. Shortly, the scaled-up PLMs, known as Large Language Models (LLMs) [34, 2, 27, 6, 55, 45], encompassed a substantially greater number of parameters and leveraged vast amounts of training data. Consequently, LLMs exhibited an enhanced capacity to learn intricate language patterns and structures, along with a notable reasoning ability, leading to superior performance across diverse natural language processing tasks [2, 45, 55, 6, 5, 30, 14, 52]. Apart from the above superiority, LLMs may occasionally produce seemingly plausible yet inaccurate predictions and face challenges when addressing problems that require specialized domain
expertise [23]. Consequently, the emerging field of Augmented Language Models (ALMs) focuses on addressing the limitations of conventional LLMs [8, 6, 2] by equipping them with enhanced reasoning capabilities and the ability to employ external resources [23]. The process of reasoning involves breaking down intricate assignments into smaller, more manageable sub-tasks that can be independently or collaboratively tackled by LLMs with the assistance of tools. What's more, LLMs can also invoke external tools or models to accomplish the relevant tasks. For example, ToolFormer [40] introduces external API tags within text sequences, facilitating LLMs' access to external tools. Visual ChatGPT [51] combines ChatGPT with Visual Foundation Models (VFMs) such as Transformers, ControlNet, and Stable Diffusion, which acts as a bridge between users, allowing them to communicate via chat and generate visuals. HuggingGPT [42] integrates the Hugging Face hub with task-specific models around ChatGPT to tackle AI tasks. ChatGPT for Robotics [47] employs ChatGPT for a wide array of robotics tasks through strategic prompt engineering. Besides, several open-sourced GitHub repositories are related to this topic, such as BabyAGI and AutoGPT. Notably, AutoGPT [15] is an automated agent, which is designed to set multiple objectives, break them down into relevant tasks, and iterate on these tasks until the objectives are achieved. Augmented language models may use these enhancements separately or joint them in a specific order to finish the specific task, which ultimately results in superior generalization capabilities.
Different from other works, we propose OpenAGI, an open-source AGI research and development platform designed to address the challenges commonly encountered in existing works, such as extensibility, nonlinear task planning, and quantitative evaluation. Furthermore, we introduce innovative methods into the learning schema of LLMs, including Reinforcement Learning from Task Feedback (RLTF) and nonlinear task planning, which aims to address challenges on out-of-distribution (OOD) generalization, optimal task planning, and AI's self-improvement (please see Sec. A.1 in supplementary materials for an extended discussion on these problems). We hope the OpenAGI platform can facilitate the open and long-term development and evaluation of AGI abilities in the community.
2.2 Reinforcement Learning from Human Feedback (RLHF)
To better align Large Language Models (LLMs) with human values, Reinforcement Learning from Human Feedback (RLHF) has been introduced [7, 58], which fine-tunes LLMs by collected human feedback, effectively enhancing alignment criteria such as helpfulness, honesty, and harmlessness. At its core, RLHF deploys reinforcement learning (RL) algorithms, notably Proximal Policy Optimization (PPO) [41], to tailor LLMs to this feedback via a reward model. Importantly, this approach actively involves human oversight in the training process, exemplified by notable models such as InstructGPT [27]. Nonetheless, the efficacy of RLHF is contingent upon the quality of feedback from adept labelers, rendering its practical implementation challenging [13, 29]. Consequently, there is an imperative to refine the RLHF framework to diminish the dependency on manual labeling and explore innovative, efficient annotation methodologies that ensure data integrity. Compared with RLHF, the proposed RLTF gets task feedback to supply information that guides the learning direction of LLMs, resulting in improved and more efficient strategies, which does not require human intervention.
3 The OpenAGI Platform
OpenAGI includes a wide range of features tailored to various needs. One key component is its benchmark tasks, detailed in Sec. 3.1, a particularly valuable tool for researchers. These tasks come equipped with task-specific datasets and evaluation metrics. This makes it possible for researchers to evaluate the performance of different LLMs in a structured and uniform manner, offering insights into their efficacy and potential areas for improvement. In addition to benchmark tasks, OpenAGI also offers open-ended tasks, detailed in Sec. 3.2. These tasks allow for a greater degree of creativity and imagination, breaking away from conventional constraints to enable more exploratory research. We believe this combination of structured benchmark tasks and flexible open-ended tasks makes OpenAGI a robust and versatile platform that can cater to a diverse array of research requirements.
3.1 Benchmark Tasks
For benchmark tasks, our goal is to provide the community a valuable tool to evaluate the planning abilities of LLMs for complex, multi-step tasks. Specifically, instead of building complicated, multi-step tasks from scratch, we first explore the domain expert models (Sec. 3.1.1) that can be used as building blocks, then introduce how we create such tasks based on them (Sec. 3.1.2).
3.1.1 Domain Expert Model Set
We now present the domain tasks and the corresponding models that can be employed in our platform. This set is designed to be flexible, allowing users to easily incorporate their own domain tasks and models. Our domain tasks are as follows:
- Language-related Models: Sentiment Analysis classifies the sentiment polarity of a given sentence [1]; Text Summarization creates a text summary that represents the most important or relevant information within the original text content [20]; Machine Translation converts a sentence from a source language to a target language [34]; Fill Mask involves replacing masked words within a given text [22]; Question Answering (QA) provides a textual answer of a question based on the given context [39].
- Vision-related Models: Image Classification aims to comprehend an entire image as a whole and assign it to a specific label [12]; Object Detection identifies and localizes specific objects within an image by detecting their instances of a particular class [3]; Colorization refers to the technique of adding plausible color information to monochromatic photographs or videos [54]; Image Super-resolution generates a high-resolution (HR) image from a low-resolution (LR) image [10]; Image Denoising aims to remove unwanted noise from an image while preserving its important features [53]; Image Deblurring aims to recover a clear image from a blurred input image [53].
- Vision-Language Models: Visual Question Answering (VQA) involves answering questions based on an image [48]; Image Captioning generates textual descriptions of the visual content depicted in an image; Text-to-Image Generation generates images from a given input sentence or sequence of words [36].
The details of the corresponding models are shown in Tab. A.1, A.2 and A.3 in supplementary materials. After selecting the domain expert models, choosing the raw datasets becomes a more straightforward process, provided that we need to ensure proper alignment between the datasets and the domain expert models' training sets. Raw datasets are provided as follows: ImageNet-1K [38], Common Objects in Context (COCO) [21], CNN/Daily Mail [26], Stanford Sentiment Treebank (SST2) [28], TextVQA [43], Stanford Question Answering Dataset (SQuAD) [35]. More details about theses datasets can be found in Sec. A.2 in supplementary materials.
3.1.2 Multi-step Tasks and Corresponding Datasets Construction
A multi-step task, as the name suggests, refers to a complex problem that cannot be solved in one simple step. It necessitates several sub-processes or stages, each requiring a particular type of problem-solving skill, in other words, domain expert model. In order to construct such complex, multi-step tasks, we introduce several commonly-used data augmentation methods, which are Gaussian Blur, Gaussian Noise, Grayscale, Low Resolution, Translation, Word Mask, to augment the raw dataset. More details about these methods can be found in Sec. A.3 in supplementary materials.
For the purpose of our study, we have sorted these tasks into six primary categories according to the modalities of their inputs and outputs:
- Image in, image out: In these tasks, images undergo several transformation stages. An example could be a task that involves "Denoising and enhancing the resolution of a low-resolution, noisy image". Here, the multi-step process entails image denoising followed by super-resolution.
- Image in, text out: These tasks usually involve interpreting the content of images. For example, "Detect objects in an image and describe them in a sentence" requires object detection followed by text generation.
- Text in, image out: Tasks under this category may include generating an image based on textual descriptions, such as "Create a graphical representation of the room described in the given text", demanding text understanding and image generation steps.
- Text in, text out: These tasks engage in text transformation or interpretation. For instance, "Translate a paragraph from English to German and summarize it in English" requires two steps - translation and summarization.
- Image-text pair in, text out: These tasks deal with complex interplay between visual and textual data. For example, "Given an image and a question about the image in English, answer the question in German." This task includes image-text understanding, question answering, and translation.
Figure 2: An example of open-ended tasks, which instructs OpenAGI to create an artwork given the theme "Gao Shan Liu Shui" (translating to "High Mountain and Flowing Water" in English). OpenAGI generates a non-linear (tree-structured) plan for the task with GPT-3.5, and then executes the plan with expert models to create a painting, a poem, and a piece of music for the theme.
- Text-text pair in, text out: These tasks can involve comparison, synthesis, or information extraction from two text inputs. For instance, "Given two reviews of a movie in English, translate them into German and provide a summary."
In total, we have devised 185 multi-step tasks, of which 117 tasks maintain a linear task structure with steps following a simple sequence, while the remaining 68 tasks exhibit a non-linear task structure, where steps might be performed concurrently or in a complex order. Among these categories, tasks such as Question Answering (QA) and Visual Question Answering (VQA), involving multiple or even multi-modal inputs, are notably complex and defy simple, linear task planning solutions. For a comprehensive view, we provide example tasks and their input and output data samples in Tab. A.4 of the supplementary materials. Additionally, a complete list of the task descriptions, accompanied by their estimated difficulty levels, can be found in Tab. A.5 within the supplementary materials.
3.1.3 Evaluation Metrics
Given that the benchmark tasks of OpenAGI comprise a diverse range of domain tasks with multimodal data, we classify them according to domain tasks as well as input and output types. We then
assess their performance using the following three metrics based on their categories: CLIP Score [16], BERT Score [56] and ViT Score $^2$  (more details can be found in supplementary). In particular, we employ the CLIP Score only for Text-to-Image Generation-based tasks, the BERT Score is utilized to assess tasks with text outputs, and the ViT score is applied to measure image similarity for the remaining tasks with image outputs. We also normalize the BERT and CLIP scores.
3.2 Open-ended Tasks
Open-ended tasks necessitate an elevated degree of creative and imaginative capacity, as they deviate from conventional constraints to stimulate more exploratory research. These tasks are designed to accommodate a broad spectrum of needs, as illustrated in Fig. 2. To achieve this, LangChain is integrated to provide additional expert models from renowned sources such as Google Search, Wikipedia, Wolfram Alpha, and more. Crucially, these models offer extendability, ensuring that open-ended tasks are not confined to specific guidelines or performance metrics. To exemplify this process, Fig. 2 elucidates how OpenAGI is directed to create a traditional Chinese painting with "Gao Shan Liu Shui" (translating to "High Mountain and Flowing Water" in English) as its theme. The process is enriched with the addition of a generated ancient Chinese poem and a piece of music that harmonize with the painting. To effectively deliver on this instruction, OpenAGI first conducts an online search to comprehend the historical narrative of "Gao Shan Liu Shui". Sequentially, the painting, poem, and music are generated in a step-by-step fashion, leveraging the collaboration between expansive language models and domain-specific expert models. The final product - a coherent artistic ensemble of painting, poem, and music - successfully resonates with the underlying ancient narrative, demonstrating the efficacy of this approach in open-ended tasks. More examples are provided in supplementary.
4 Reinforcement Learning from Task Feedback (RLTF)