SlowGuess/NeurIPS_2023_OCR huggingface dataset:
While learning solely from input text is a powerful method for training LLMs, it is not sufficient for handling real-world tasks that require a deeper understanding of context and environment. One potential method to improve the capabilities of LLMs is to incorporate reinforcement learning (RL) techniques. By leveraging the strengths of RL, LLMs can gain additional insights from trial-and-error experiences. This leads to more robust and adaptive models, especially in situations where labeled data is scarce or when tasks involve physical interactions. In this work, we propose Reinforcement Learning from Task Feedback (RLTF), shown in Fig. 3, which utilizes task feedback to supply more information that guides the learning direction of LLMs, resulting in improved and more efficient strategies. We choose to use REINFORCE [50] in this work and more details about the algorithm are provided in Sec. A.6 in supplementary.
Figure 3: An illustration of the RLTF mechanism.
5.1 Backbone LLMs
- GPT-3.5-turbo. The GPT (Generative Pre-trained Transformer) series [2] consists of advanced language models. In this work, we use the GPT-3.5-turbo-0301 snapshot.
$^{2}$ https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_sensitivity.ipynb
- Claude-2. Claude-2 [9] is an transformer LLM trained with unsupervised learning and RLHF.
- GPT-4. GPT-4 is a follow-up version of GPT-3.5, which is more powerful than its predecessors. In this work, we use the GPT-4-0613 snapshot.
- Flan-T5-Large. Flan-T5 [8] is a series of language models which are fine-tuned using a technique called instruction fine-tuning. Flan-T5-Large has 770 million parameters.
- Vicuna-7B. Vicuna [5] is an open-source chatbot trained by fine-tuning the LLaMA [45] model with user-shared conversations. In this work, we use the 7-billion size model of Vicuna.
- LLaMA-2. LLaMA-2 [46] is a successor to the original LLaMA model, it is significantly more powerful. In this work, we use the 13-billion size model.
Overall, we include three closed-source LLMs (GPT-3.5-turbo, Claude-2 and GPT-4) as well as three open-source LLMs (Flan-T5-Large, Vicuna-7B and LLaMA-2-13B).
5.2 Learning Schema of LLMs
We employ the following LLM learning schema for experimentation.
- Zero-shot Learning (Zero) directly inputs the prompt to the LLM.
- Few-shot Learning (Few) presents a set of high-quality demonstrations, each consisting of both input and desired output, on the target task. As the model first sees good examples, it can better understand human intention and criteria for what kinds of answers are wanted.
- Fine-tuning involves using manually labeled data samples as additional training signals to refine and adapt pre-trained LLMs to specific tasks or domains.
- RLTF is our proposed method in Sec. 4, which further utilizes the RL method to tune the fine-tuned LLMs with human labelled data.
We employ Low-Rank Adaptation (LoRA) [17] to optimize all open-source LLMs across both the Fine-tuning and RLTF learning schema.
5.3 Planning Solution Parser
To transform the original LLM output into a viable task planning solution, we use a parser built on GPT-3.5. The prompt we employed reads as follows: "You are a key phrase extractor who is able to extract potential module names from the given context. You have already known all the module names in the full module list. The full module list is: [Image Classification, Colorization, Object Detection, Image Deblurring, Image Denoising, Image Super Resolution, Image Captioning, Text to Image Generation, Visual Question Answering, Sentiment Analysis, Question Answering, Text Summarization, Machine Translation]. Given the following context:  $\{\}$  . Please extract a module sequence from this context and remove module names which do not exist in the full module list from this sequence. Output the module sequence after filtering as the format of 'module: module1, module: module2, module: module3, etc...'." Once this prompt is executed on the LLM's original text output, a task planning solution will be generated which consists of a multi-step solution of the problem.
5.4 Datasets
Considering the fact that the imbalanced number of tasks with different input and output modalities could lead to skewed measurement results, we select the tasks in OpenAGI to compose the training set. In particular, we randomly select  $10\%$  of tasks, along with their corresponding datasets, based on input and output modalities for training purposes. For few-shot, fine-tuning and RLTF, we supply manually curated, feasible solutions as ground-truth labels. In the case of RLTF, we employ the fine-tuning checkpoint as a reasonable initialization for LLMs and use constrained beam search [11, 37] to reduce the likelihood of producing infeasible solutions (details can be found in Sec. A.7 in supplementary). Moreover, we choose an additional  $10\%$  of tasks, adhering to the same selection criteria as mentioned above, to serve as the test set.
5.5 Experimental Analysis
The main experimental results are tabulated in Tab. 1 and 2, showcasing the results for closed-source and open-source LLMs, respectively. The overall performance is calculated as the average of CLIP,
BERT and ViT scores. Here, only the task descriptions of the benchmark tasks are fed into LLMs (additional information, such as the input prompt and LLMs' outputs, is provided in Fig. A.4 and A.5 in supplementary). Broadly speaking, closed-source LLMs demonstrate superior performance on OpenAGI tasks, with GPT-4 leading the pack under both zero- and few-shot scenarios. In the open-source category, LLaMA-2-13B takes the lead, consistently posting top results across various learning schema—the performance possibly influenced by its larger model size. Notably, open-source LLMs significantly benefit from the tuning methods, particularly Fine-tuning and RLTF. These methods mark noticeable enhancements for Flan-T5-Large, Vicuna-7B, and LLaMA-2-13B when compared with zero-shot and few-shot learning schema. In fact, each of these open-source models hits its pinnacle under the RLTF approach. Conclusively, with RLTF tuning, the performance of LLaMA-2-13B approaches that of GPT-3.5, illustrating its potential.
Table 1: OpenAGI task-solving performances under different settings for three closed-source LLMs. Boldface denotes the highest score under each learning schema.
Metrics	GPT-3.5-turbo	Claude-2	GPT-4
Zero	Few	Zero	Few	Zero	Few
CLIP Score	0.0	0.0	0.0	0.2543	0.0	0.3055
BERT Score	0.1914	0.3820	0.2111	0.5038	0.2076	0.6307
ViT Score	0.2437	0.7497	0.4082	0.5416	0.5058	0.6480
Overall	0.1450	0.3772	0.2064	0.4332	0.2378	0.5281
Table 2: OpenAGI task-solving performances under different settings for three open-source LLMs. Boldface denotes the highest score under each learning schema.
Metrics	Flan-T5-Large	Vicuna-7B	LLaMA-2-13B
Zero	Few	Fine-tuning	RLTF	Zero	Few	Fine-tuning	RLTF	Zero	Few	Fine-tuning	RLTF
CLIP Score	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0612	0.0608	0.1220
BERT Score	0.0	0.2488	0.0	0.0655	0.0513	0.0	0.1212	0.1756	0.0986	0.2281	0.1570	0.2401
ViT Score	0.0	0.0	0.6316	0.6978	0.1704	0.4285	0.5507	0.7300	0.3614	0.2558	0.6723	0.7584
Overall	0.0	0.0829	0.2105	0.2544	0.0739	0.1428	0.2239	0.3018	0.1533	0.1817	0.2967	0.3735
5.6 Effect of Prompts
We design two types of prompts combined with different levels of model description to test LLMs' zero-shot performances. The first, Prompt-1, only combines the task description with the model names, while the second, Prompt-2, integrates the task description with comprehensive model descriptions, detailing model usage, input, and output types (additional information about these two prompts is provided in Fig. A.6 in supplementary). We analyze the results in Tab. 3 and 4 in conjunction with the previous zero-shot results in Tab. 1 and 2. Compared to the original prompt that only uses task description to generate the results in Tab. 1 and 2, it is evident that in most cases, the closed-source LLMs, such as GPT series and Claude-2, tend to outperform when provided with detailed model-related information as seen in Prompt-1 and Prompt-2. In contrast, open-source LLMs, whose understanding and reasoning capacity may be weaker than those huge closed-source models, appear to be misled by the ambiguous details in Prompt-1 and Prompt-2 during the model selection process. Overall, detailed prompts can assist in improving the zero-shot performance to a certain degree, depending on the specific model. However, they may not be as potent as other training scenarios for smaller size LLMs, such as fine-tuning or RLTF.
5.7 Case Study of Non-linear Planning
We qualitatively evaluate LLMs' capability of non-linear task planning. Fig. 4 illustrates the responses of GPT-3.5, Vicuna-7B and Flan-T5-Large to Prompt-2. The given task description requires the LLM to answer a query posed in English about a given noisy, blurry, and gray-scale image in German. It can be observed from the results that the performance of the models varies significantly. Flan-T5-Large, for instance, demonstrates a struggling comprehension of the query, while Vicuna-7B's answer incorporates all the provided models in an attempt to resolve the task. GPT-3.5 successfully comprehends the task and consequently delivers a reasonable plan. The plan generated by this model is notably non-linear, and it instructs to employ a Visual Question Answering (VQA) model with the English query and processed image as inputs in steps 1 and 2 in order to accomplish the task. Similarly, another task scenario is demonstrated in Fig. 2, which is an open-ended task with
Table 3: Zero-shot task-solving performances under various prompts for three closed-source LLMs.
Metrics	GPT-3.5-turbo	Claude-2	GPT-4
Prompt-1	Prompt-2	Prompt-1	Prompt-2	Prompt-1	Prompt-2
CLIP Score	0.0	0.0	0.0	0.0	0.0	0.0
BERT Score	0.2106	0.3013	0.4088	0.2333	0.4402	0.5595
ViT Score	0.0	0.2710	0.6816	0.7957	0.5497	0.5565
Overall	0.0702	0.1907	0.3635	0.3430	0.3299	0.3717
Table 4: Zero-shot task-solving performances under various prompts for three open-source LLMs.
Metrics	Flan-T5-Large	Vicuna-7B	LLaMA-2-13B
Prompt-1	Prompt-2	Prompt-1	Prompt-2	Prompt-1	Prompt-2
CLIP Score	0.0	0.0	0.0	0.0	0.0	0.0
BERT Score	0.0	0.0	0.0603	0.0267	0.0971	0.1717
ViT Score	0.0	0.0	0.0	0.2385	0.0	0.0
Overall	0.0	0.0	0.0201	0.0884	0.0323	0.0572
GPT being instructed to generate a painting in a traditional Chinese style that depicts "Gao Shan Liu Shui". Initially, GPT seems to lack understanding of what constitutes a traditional Chinese style painting and it is also unfamiliar with the concept of "Gao Shan Liu Shui". As a remedy, GPT utilizes Google search in the initial two steps to gather information on these unfamiliar topics. Subsequently, it integrates the retrieved information to formulate a comprehensive prompt that instructs the Text-to-Image Generation model to create the desired artwork.
Figure 4: An example of Non-linear Planning.
Use the "Image Denoising" model to reduce the noise in the given image.
Use the "Image Deblurring" model to remove the blur from the denoised image.
Use the "Image Super-Resolution" model to further enhance the quality of the image.
Apply the "Visual Question Answering" model to answer the English question based on the processed image.
Finally, use the "Machine Translation" model to translate the answer from English to German.
1. Use Sentiment Analysis to analyze the sentiment of the English question.
2. Use Text Summarization to summarize the English question.
3. Use Machine Translation to translate the English question into German.
4. Use Fill Mask to fill the sentence at the masked position with the German translation.
5. Use Question Answering to answer the German question based on the given context.
6. Use Image Classification to classify the grayscale image.
7. Use Object Detection to detect the objects in the grayscale image.
8. Use Colorization to colorize the grayscale image.
9. Use Image Super-Resolution to create a high-resolution image from the grayscale image.
10. Use Image Denoising to denoise the grayscale image.
11. Use Image Deblurring to deblur the grayscale image.
12. Use Visual Question Answering to answer the German question based on the given image.
13. Use Image Captioning to describe the contents of the grayscale image.
14. Use Text-to-Image Generation to generate an image based on the German question.
Answering a question in German
In this work, we introduce OpenAGI, an open-source AGI research platform designed to facilitate the development and evaluation of LLMs in solving complex, multi-step tasks through manipulating various domain expert models, tools, plugins or APIs. OpenAGI provides a wide range of tasks, models, datasets, benchmarks, and evaluation methods. We also propose the LLM+RLTF approach, which combines LLMs with reinforcement learning to optimize task-solving performance. The evaluation of various LLMs using the OpenAGI pipeline and different learning schema demonstrates that smaller-scale LLMs can potentially outperform larger models when combined with the appropriate learning approach, such as RLTF. In the future, we aim to explore 1) Human-in-the-loop agents, where LLM may prompt human experts for answers as one step of the task-solving plan when a suitable model is unavailable, thus enabling better Human-AI collaboration; 2) Trustworthy agents, which guarantee the safety and the ethical standard of agents during task-solving; and 3) Self-improving agents, which enable automated task generation and training that facilitate OpenAGI in independent exploration of tasks, empowering the self-reflection, self-prompting and self-improvement of intelligent agents.
[49] Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. 2023. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2302.01560 (2023).
[50] Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning 8, 3 (1992), 229-256.
[51] Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. 2023. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671 (2023).
[52] Ruosong Ye, Caiqi Zhang, Runhui Wang, Shuyuan Xu, and Yongfeng Zhang. 2023. Natural language is all a graph needs. arXiv preprint arXiv:2308.07134 (2023).
[53] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. 2022. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5728-5739.
[54] Richard Zhang, Jun-Yan Zhu, Phillip Isola, Xinyang Geng, Angela S. Lin, Tianhe Yu, and Alexei A. Efros. 2017. Real-Time User-Guided Image Colorization with Learned Deep Priors. ACM Trans. Graph. 36, 4, Article 119 (jul 2017), 11 pages. https://doi.org/10.1145/3072959.3073703
[55] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 (2022).
[56] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019).
[57] Yongfeng Zhang. 2021. Problem Learning: Towards the Free Will of Machines. arXiv preprint arXiv:2109.00177 (2021).
[58] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 (2019).
Supplementary Material for OpenAGI
Figure A.1: Research challenges when solving complex, multi-step tasks with augmented LLMs.
A.1 Research Challenges
Although the OpenAGI platform offers numerous advantages and enhanced accessibility, it also gives rise to a variety of novel research challenges, such as:
- Out-of-Distribution (OOD) Generalization. Domain-specific expert models may exhibit limited generalization ability due to their strong dependence on the distribution of the training data. As demonstrated in Fig. A.1 (a), when processing images from disparate sources exhibiting a distributional shift, the original model sequence to address the task in Fig. 1 becomes ineffective. In the majority of instances, only a few colors are accurately restored, while most remain incorrect. Furthermore, noise and blurring persist, remaining highly perceptible to human observers.
- Optimal Task Planning. There is a compositional number of ways to combine different models to generate solutions, which can make it difficult to identify the best approach. Additionally, it is possible for multiple valid solutions to exist for a given task, but the quality of each solution can vary greatly. For instance, as depicted in Fig. A.1 (b), executing the same four models in a different sequence compared to Fig. 1 can lead to noticeably different outcomes. The results from the second approach (Fig. A.1 (b)) exhibit significantly more noise and color inconsistencies compared to the first approach (Fig. 1). Therefore, it is crucial for the LLM to identify and implement the optimal task plan from among the various possibilities.
- Nonlinear Task Structures. During model execution, a model may need more than one inputs and each input need to be produced by a prerequisite model, resulting in a nonlinear (tree) structure for the solution. In this context, employing a nonlinear task planning may enable more effective integration of the diverse inputs and more efficient parallel processing of the models to achieve the desired outcome. However, incorporating such nonlinear task planning ability into LLMs presents unique challenges beyond the LLM's existing task-solving capabilities.
In consideration of the first two challenges, we introduce a mechanism referred to as Reinforcement Learning from Task Feedback (RLTF). This approach capitalizes on the performance feedback
procured from tasks following the execution of the solution devised by the LLM. Consequently, the RLTF mechanism effectively refines the LLM's planning strategy, resulting in an enhanced and more adaptive system. Indeed, relying solely on input text for learning proves insufficient for LLMs when confronted with real-world tasks. Task feedback, on the other hand, supplies additional information that steers the learning trajectory of LLMs towards improved and efficient solutions. For the third challenge, we propose Nonlinear Task Planning, which utilizes beam search as an efficient semi-autoregressive decoding method [11, 37] such that for each decoding step in beam search, different hypotheses are treated as parallel actionable solutions for different inputs instead of competing hypotheses. If a task requires parallel processing for multiple inputs, such as both text and image, then in generation time, an actionable solution taking text as input and another solution taking image as input will be generated and executed in parallel.
Table A.1: Language-related models
Domain Task	Input Modality	Output Modality	Model
Sentiment Analysis	Text	Text	FinBert 3[1]
Text Summarization	Text	Text	BART 4[20]
Machine Translation	Text	Text	T5 5[34]
Fill Mask	Text	Text	DistilRoberta 6[22]
Question Answering	Text, Text	Text	DistilBERT 7[39]
Table A.2: Vision-related models
Domain Task	Input Modality	Output Modality	Model
Image Classification	Image	Text	ViT 8[12]
Object Detection	Image	Text	DETR 9[3]
Colorization	Image	Image	Colorizer 10[54]
Image Super-Resolution	Image	Image	Swin2SR 11[10]
Image Denoising	Image	Image	Restormer 12[53]
Image Deblurring	Image	Image	Restormer [53]
Table A.3: Vision-language models
Domain Task	Input Modality	Output Modality	Model
Visual Question Answering	Image, Text	Text	GIT 13[48]
Image Captioning	Image	Text	Vision Encoder Decoder 14
Text-to-Image Generation	Text	Image	Stable Diffusion 15[36]
<sup>1</sup>https://huggingface.co/yiyanghkust/finbert-tone
$^{2}$ https://huggingface.co/distilbert-base-cased-distilled-squad
3https://huggingface.co/facebook/bart-large-cnn
<sup>4</sup>https://huggingface.co/gpt2
<sup>5</sup>https://huggingface.co/t5-base
<sup>6</sup>https://huggingface.co/distilroberta-base
<sup>7</sup>https://huggingface.co/google/vit-base-patch16-224
<sup>8</sup>https://huggingface.co/facebook/deptr-resnet-101
<sup>9</sup>https://github.com/richzhang/colorization
10https://huggingface.co/caidas/swin2SR-classical-sr-x2-64
11 https://github.com/swz30/Restormer
12 https://huggingface.co/microsoft/git-base-textvqa
<sup>13</sup>https://huggingface.co/nlpconnect/vit-gpt2-image-captioning
$^{14}$ https://huggingface.co/CompVis/stable-diffusion-v1-4
A.2 Original Datasets
- ImageNet-1K [38] is a large-scale image dataset, derived from the broader ImageNet database, containing approximately 1 million images. These images are categorized into 1,000 distinct classes, with each class representing a specific object or concept. The dataset has been instrumental in the development and evaluation of state-of-the-art deep learning algorithms for image classification, object recognition, and transfer learning.
- Common Objects in Context (COCO) [21] is a large-scale, richly-annotated image dataset designed to advance the fields of object detection, segmentation, and captioning. Released in 2014, it contains over 200,000 labeled images with 1.5 million object instances from 80 different object categories. The dataset features complex, real-world scenes with multiple objects per image, various object scales, and diverse contexts.
- CNN/Daily Mail [26] is a valuable resource for text summarization, which consists of human-generated abstractive summaries, created by transforming news articles from CNN and Daily Mail websites into questions, with one entity concealed, and generating summaries from the corresponding passages. The authors have made available the scripts used to crawl, extract, and generate question-answer pairs from these websites. The corpus contains 286,817 training pairs, 13,368 validation pairs, and 11,487 test pairs, as defined by the scripts. On average, the source documents in the training set span 766 words across 29.74 sentences, while the summaries are composed of 53 words and 3.72 sentences.
- Stanford Sentiment Treebank (SST2) [28] is a corpus with labeled parse trees that allows for the analysis of the compositional effects of sentiment in language. The corpus consists of 11,855 single sentences extracted from movie reviews. It was parsed with the Stanford parser and includes a total of 215,154 unique phrases from those parse trees, each annotated by 3 human judges.
- TextVQA [43] serves as a benchmark for evaluating visual reasoning based on text present in images. In order to answer questions pertaining to the images, TextVQA necessitates models to read and reason about the text contained within them. The incorporation of text as a new modality in images demands that models be able to reason over this modality to address TextVQA queries. Thus, TextVQA poses a unique challenge for models to integrate both visual and textual cues to arrive at a comprehensive answer.
- Stanford Question Answering Dataset (SQuAD) [35] is a collection of question-answer pairs sourced from Wikipedia articles. A distinguishing characteristic of SQuAD is that the correct answers to the questions can be any sequence of tokens in the corresponding text. This flexibility is a result of the dataset's construction through crowd-sourcing, which results in a diverse set of questions and answers compared to other question-answering datasets.
A.3 Data Augmentation Methods
Upon determining the raw datasets, our next objective is to augment them from various perspectives to construct complex, multi-step tasks. For instance, we can introduce noise and reduce the resolution of an image from ImageNet-1K to create new datasets that may require "Image Denoising" and "Image Super-Resolution" for initial recovery before performing classification. The data augmentation methods employed are as follows: