SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Gaussian Blur is a prevalent image processing technique that involves convolving an image with a Gaussian filter kernel. This filter is applied to smooth the image and reduce noise, yielding a blurred output image.
- Gaussian Noise refers to the addition of Gaussian-distributed noise.
- Grayscale entails converting the colorful image to a grayscale image.
- Low Resolution pertains to images with a reduced pixel density (pixels per inch, or ppi).
- Translation denotes the process of converting a text from one language, such as English, to another, such as German. In this work, we only use English-to-German translator for simplicity.
- Word Mask randomly replaces a single word in a given sentence with the "MASK" token.
Task description	Input Sample	Output Sample
Given low-resolutiond noisy blurry grayscale image, how to return the regular image step by step?
Given low-resolution noisy blurry grayscale image, how to return the object names in English step by step?	bear
Given clozed English text, how to translate the text in German step by step?	A big burly grizzly bear is show [Mask] grass in the background.	Ein kräftiger Grizzly Bär ist im Hintergrund mit Gras zu sehen.
Given noisy blurry grayscale image and clozed English query, how to answer the question in English step by step?	Question: what number is [Mask] the player's jersey?	22
Given clozed English document and clozed English query, how to answer the question in German step by step?	Context: Super Bowl 5 was an American football game to determine the champion of the National...
Question: What was the theme of Super [Mask] 50?	Goldener Jahrestag
Table A.4: Examples of multi-step tasks and their augmented data samples.
- CLIP Score<sup>16</sup> is a reference-free metric used to assess the correlation between a generated image caption and the actual content of the image. Research has shown that it has a strong correlation with human judgment and is a reliable measure for evaluating image captioning performance [16].
- BERT Score<sup>17</sup> uses contextual embeddings from the pre-trained BERT model to compare words in candidate and reference sentences through cosine similarity. Studies have shown that it is highly correlated with human evaluation at both sentence-level and system-level [56]. Additionally, BERT Score calculates precision, recall, and F1 measure, making it a valuable tool for evaluating various language generation tasks. In this work, we use the value of F1 score.
- ViT Score<sup>18</sup> is a metric designed to assess the visual similarity between two images. By calculating the cosine similarity of their respective embeddings, which are generated using a Vision Transformer, the ViT Score offers a quantitative measure of their likeness.
$^{16}$ https://torchmetrics.readthedocs.io/en/stable/multimodal/clip_score.html
<sup>17</sup>https://huggingface.co/spaces/evaluate-metric/bertscore
<sup>18</sup>https://colab.research.google.com/github/huggingface/ notebooks/blob/main/examples/image_similarity.ipynb
A.5 Dataset Documentation and Data Samples for Benchmark Tasks
Our dataset is designed to evaluate LLM's planning ability of using domain expert models. To accomplish this, we enhance the standard CV/NLP datasets using various combinations of data augmentation methodologies. We have devised 185 multi-step tasks in total, of which 117 tasks maintain a linear task structure with steps following a simple sequence, while the remaining 68 tasks exhibit a non-linear task structure, where steps might be performed concurrently or in a complex order. Each benchmark task is accompanied by a small dataset, which contains 100 augmented data samples. All benchmark datasets can be accessed, reviewed, and downloaded via https://drive.google.com/drive/folders/1AjT6y7qLIMxcmHhUBG5IE1_5SnCPR57e, which is committed to transparency and ease of accessibility. As the authors, we affirm that we assume all responsibility for any rights violation related to this dataset. The data license is Creative Commons Attribution 4.0 International, ensuring all necessary permissions and regulations are stringently adhered to. The dataset is hosted on GitHub https://github.com/agiresearch/OpenAGI. We have chosen this platform considering its robustness, reliability, and its proven track record for data hosting. We ensure that access to the data will be maintained consistently, possibly through a curated interface. A maintenance plan is in place to address potential issues, provide necessary updates, and ensure the data's long-term availability and integrity.
We also offer several data samples to illustrate the structure of the datasets further. For example, consider the third row of Tab. A.4, which represents a machine translation domain task (i.e., translating from English to German). In this case, we apply the "Word Mask" augmentation technique on the text inputs to create a multi-step task, which can be described as "Given clozed English text, how can the text be translated into German step by step?" For instance, given an original data sample, "A big burly grizzly bear is shown with grass in the background", the word "with" has been chosen to be masked to generate the augmented data sample, "A big burly grizzly bear is shown [MASK] grass in the background".
A.6 Details of RLTF
In the setup of RLTF, the environment is the OpenAGI platform and the agent is the LLM  $\mathcal{L}$  parameterized with  $\Phi$ . The solution  $s$  generated by the LLM can be seen as a set of instructions that solve the input task  $t$  and can be executed on the corresponding augmented dataset  $\mathcal{D}_t$ . We can use the performance (provided in Sec. 3.1.3) on that dataset as the reward signal  $\mathcal{R}$  and use reinforcement learning to fine-tune the LLM. More concretely, to find the optimal solution, we require the LLM to maximize its expected reward on the training set  $\mathcal{T}_{train}$ , represented by  $J(\Phi)$ :
$$
J (\Phi) = \mathbb {E} _ {\mathbf {s} _ {\text {t r a i n}} \sim \mathcal {L} \left(\mathcal {T} _ {\text {t r a i n}} \mid \Phi\right)} [ \mathcal {R} ] \tag {A.1}
$$
Since the reward signal  $\mathcal{R}$  is non-differentiable, we need to use a policy gradient method to iteratively update  $\Phi$ . In this work, we use the REINFORCE in [50] as follows,
$$
\nabla_ {\Phi} J (\Phi) = \mathbb {E} _ {P \left(\mathbf {s} _ {\text {t r a i n}} \mid \Phi\right)} \left[ \nabla_ {\Phi} \log P \left(\mathbf {s} _ {\text {t r a i n}} \mid \Phi\right) \cdot \mathcal {R} \right] \tag {A.2}
$$
An empirical approximation of the above quantity is:
$$
\nabla_ {\Phi} J (\Phi) \approx \frac {1}{| \mathcal {T} _ {\text {t r a i n}} |} \sum_ {t \in \mathcal {T} _ {\text {t r a i n}}} \nabla_ {\Phi} \log P (s _ {\text {t r a i n}} | \Phi) \cdot \mathcal {R} \tag {A.3}
$$
The above update is an unbiased estimate for our gradient, but has a very high variance. To reduce the variance of this estimate, we employ a baseline function  $b$ , which is the moving average of the previous reward signals:
$$
\nabla_ {\Phi} J (\Phi) \approx \frac {1}{\left| \mathcal {T} _ {\text {t r a i n}} \right|} \sum_ {t \in \mathcal {T} _ {\text {t r a i n}}} \nabla_ {\Phi} \log P (s _ {\text {t r a i n}} | \Phi) \cdot (\mathcal {R} - b) \tag {A.4}
$$
A.7 Constrained Generation
To generate the solution for a natural language task description, we require the LLM to generate an actionable solution consisting of sequences of model names. For tasks that require only one input,
the model only needs to generate one actionable sequence of models. For tasks that require multiple inputs, such as Visual Question Answering, the LLM needs multiple steps in order to accomplish the task, where each step is either a sequence of models or a parallel of several sequences of models. Towards this end, the LLM must satisfy three conditions: 1) only generate the model names without irrelevant tokens, 2) generate valid sequences of models, and 3) generate paralleled sequences of models for different inputs when necessary.
Condition 1: For the LLM to generate only model names, instead of tuning the model to teach it what names are available, we adopt constrained beam search [11], which only allows generating tokens from the model set  $\mathcal{M}$  at every decoding step. More specifically, we define our constraints as a prefix trie such that each model name is a path from the root to some leaf node. For each node  $t$  in the tree, its children indicate all the allowed continuations from the prefix defined traversing the trie from the root to  $t$ . Thus in each decoding step, the next token can only be selected from either all possible continuations allowed based on the generated tokens or the first tokens of all possible next model names. For example, if "Text" is already generated, based on the set of model names, the next tokens can only be either "Summarization" due to the "Text Summarization" model or "Generation" due to the "Text Generation" model, as shown in Fig. A.2.
Figure A.2: Model name based constrained generation.
Condition 2: For the LLM to generate valid sequences of models, consecutive models should have input and output modalities matched. If the output modality of a model is text, then the next model can only be models that take text as input. This is also achieved by constrained beam search such that when finishing generating one model, the constraint function will determine the output modality of this model and find out all possible next models in model set  $\mathcal{M}$ , excluding the models that are already generated. It will dynamically construct a new trie for all these model names based on the output modality. For example, if the first generated model name is "Text Summarization", then the next possible models can be "Sentiment Analysis", "Text Generation", etc., as shown in Fig. A.3.
Figure A.3: Model type based constrained generation.
A.8 Zero- and Few-shot Schema
As in shown in Fig. A.4, in the zero-shot setting, most LLMs struggle to generate valid task plans, let alone optimal solutions. In particular, GPT-3.5 tends to generate repetitive contents, which subsequently maps to identical model names. Meanwhile, Vicuna-7B and Flan-T5-Large, constrained by their zero-shot capabilities, fail to produce a reasonable plan. In the few-shot setting, we incorporate several manually labeled task plans as instructions to guide the generation, resulting in a remarkable improvement in the quality of the task plans. As observed in Fig. A.5, all three LLMs can produce solutions that are semantically similar to the provided examples. In fact, many solutions can be used directly, even without the need for mapping.
A.9 Broader Impacts, Limitations, and Future Visions
Just like any technology, the irresponsible use of AI techniques and intelligent systems may have detrimental effects on individuals and society as a whole. In particular, existing Large Language Models (LLMs) are not sufficiently designed to ensure their harmless usage, making them vulnerable to misuse by malicious parties. Consequently, it is important to address and mitigate the potential risks associated with LLMs when used for complex task solving. Our constrained generation framework provides a potential solution to this issue. By incorporating ethical constraints, such as an AI
Zero-shot Prompt
Given a low-resolution, blurry, grayscale image, how to return the regular image step by step?
Figure A.4: An example of zero-shot schema.
Step 1: Collect more data or images to create a higher resolution version of the blurry image.
Step 2: Use a deblurring algorithm to remove the blurriness and increase the sharpness of the image.
Step 3: Apply a denoising algorithm to reduce any noise in the image that may have been introduced
during the deblurring process.
Step 4: Use an upsampling method to increase the resolution of the image to the desired level.
Step 5: Apply a contrast enhancement technique to make the image more visually appealing.
Step 6: Finally, apply sharpening techniques to add crispness and detail to the image.
1. Use the image to generate the mask image.
2. Use the mask image to generate the masked image.
3. Use the masked image to generate the final image.
4. Use the masked image to generate the final image.
5.Use ...
The blurry image is a blurry image with a blurry background. To return the regular image, you can use the following steps: First, you need to find the blurry image. Then, you need to find the regular image. Then, you need to find the blurry image. Then, you need to find the regular image. Then, you need to find the blurry image. Then, you need to find the regular image. Then, ...
Few-shot Prompt
Problem: Given low-resolutioned noisy blurry gray image, how to return the regular image step by step?
Step 1: Image Super Resolution,
Step 2: Image Denoising,
Step 3: Image Deblurring,
Step 4: Colorization,
Problem: Given clozed English text, how to generate the rest of text in English step by step?
Step 1: Fill Mask,
Step 2: Text Generation,
Problem: Given low-resolutioned noisy blurry grayscale image, how to return the regular image step by step?
Soluion:
Figure A.5: An example of few-shot schema.
Step 2: Image Deblurring,
Step 3: Image Denoising.
Step 3: Colorization,
Step 4: Image Classification,
Step 5: Machine Translation,
Step 4: Image Captioning,
constitution, into the plan generation process, we can guide the agent to generate plans that are both ethically sound and benign while tackling complex tasks.
In the future, we can incorporate multiple models within each single-step task, thereby providing an expanded selection of options for LLMs to address complex problems. Additionally, we can integrate datasets from alternative modalities, such as video and audio, into the OpenAGI platform. These datasets will facilitate the development of more sophisticated tasks to further investigate the planning capabilities of LLMs. We will also explore multiple task-solving agents to interact with each other for complex problem solving. Another promising direction is to involve humans in the loop during the
resolution of complex tasks. In such scenarios, LLM may prompt human experts for answers as one step of the task-solving plan when a suitable model is unavailable, thus enabling better Human-AI collaboration. Lastly, we can explore automated task generation techniques that empower OpenAGI to generate complex tasks independently, facilitating self-prompting and self-improvement in its task-solving capabilities.
A.10 Computational Resources
For augmenting the data, we used devices equipped with Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz and 256 GB RAM. For training and testing the LLMs, we used 4xA5000-24GB GPUs.
A.11 Training Details
In our experiments, we use Low-Rank Adaptation (LoRA) $^{19}$  [17] for efficient fine-tuning of Flant5-Large, Vicuna-7B and LLaMA-2-13B under both Fine-tuning and RLTF schema, and the configuration/hyper-parameter settings are shown in Tab. A.5.
Table A.5: Configuration and parameter settings for Flan-T5-Large, Vicuna-7B and LLaMA-2-13B
Configuration/Hyper-parameter	Flan-T5-Large	Vicuna-7B	LLaMA-2-13B
Fine-tuning	RLTF	Fine-tuning	RLTF	Fine-tuning	RLTF
Optimizer	AdamW	AdamW	AdamW	AdamW	AdamW	AdamW
Epochs	200	10	200	10	200	10
Training Batch Size Per GPU	8	5	1	1	1	1
Gradient Accumulation Steps	1	1	1	1	1	1
Learning Rate	1e-5	1e-5	5e-6	5e-6	1e-5	1e-5
Weight Decay	1e-6	1e-6	1e-6	1e-6	1e-6	1e-6
Warmup Ratio	0.1	0.1	0	0	0	0
Scheduler	Linear	Linear	Linear	Linear	Linear	Linear
LoRA_r	8	8	8	8	8	8
LoRA_α	16	16	16	16	16	16
LoRA_dropout	0.05	0.05	0.05	0.05	0.05	0.05
ε	-	0.2	-	0.2	-	0.2
Decay Rate of ε	-	0.9	-	0.9	-	0.9
Beam Size	-	30	-	20	-	20
Num of Outputs	-	30	-	20	-	20
Top k	-	5	-	40	-	40
Top p	-	0.5	-	0.75	-	0.75
Temperature	-	0.9	-	0.2	-	0.2
Num of Beam Groups	-	1	-	1	-	1
Table A.6: Task descriptions of all multi-step tasks in OpenAGI. The difficulty level is estimated by the size of human-labeled solutions, that is, the total number of models used in the human-labeled task solution. The higher the number, the more difficult the task.
Task Description	Difficulty Level
Given low-resolutioned noisy blurry grayscale image how to return the regular image step by step?	4
Given noisy blurry grayscale image how to return the regular image step by step?	3
Given low-resolutioned blurry grayscale image how to return the regular image step by step?	3
Given blurry grayscale image how to return the regular image step by step?	2
Given low-resolutioned noisy grayscale image how to return the regular image step by step?	3
Continued on next page
$^{19}$ https://huggingface.co/blog/lora
Table A.6 - continued from previous page
Task Description	Difficulty Level
Given noisy grayscale image how to return the regular image step by step?	2
Given low-resolutioned grayscale image how to return the regular image step by step?	2
Given grayscale image how to return the regular image step by step?	1
Given low-resolutioned noisy blurry image how to return the regular image step by step?	3
Given noisy blurry image how to return the regular image step by step?	2
Given low-resolutioned blurry image how to return the regular image step by step?	2
Given blurry image how to return the regular image step by step?	1
Given low-resolutioned noisy image how to return the regular image step by step?	2
Given noisy image how to return the regular image step by step?	1
Given low-resolutioned image how to return the regular image step by step?	1
Given low-resolutioned noisy blurry grayscale image how to return the caption in German step by step?	5
Given low-resolutioned noisy blurry grayscale image how to return the class label in German step by step?	6
Given low-resolutioned noisy blurry grayscale image how to return the object names in German step by step?	6
Given low-resolutioned noisy blurry grayscale image how to return the caption in English step by step?	5
Given low-resolutioned noisy blurry grayscale image how to return the class label in English step by step?	5
Given low-resolutioned noisy blurry grayscale image how to return the object names in English step by step?	5
Given noisy blurry grayscale image how to return the caption in German step by step?	5
Given noisy blurry grayscale image how to return the class label in German step by step?	5
Given noisy blurry grayscale image how to return the object names in German step by step?	5
Given noisy blurry grayscale image how to return the caption in English step by step?	4
Given noisy blurry grayscale image how to return the class label in English step by step?	4
Given noisy blurry grayscale image how to return the object names in English step by step?	4
Given low-resolutioned blurry grayscale image how to return the caption in German step by step?	5
Given low-resolutioned blurry grayscale image how to return the class label in German step by step?	5
Given low-resolutioned blurry grayscale image how to return the object names in German step by step?	5
Given low-resolutioned blurry grayscale image how to return the caption in English step by step?	4
Given low-resolutioned blurry grayscale image how to return the class label in English step by step?	4
Given low-resolutioned blurry grayscale image how to return the object names in English step by step?	4
Given blurry grayscale image how to return the caption in German step by step?	4
Given blurry grayscale image how to return the class label in German step by step?	4
Table A.6 - continued from previous page
Task Description	Difficulty Level
Given blurry grayscale image how to return the object names in German step by step?	4
Given blurry grayscale image how to return the caption in English step by step?	3
Given blurry grayscale image how to return the class label in English step by step?	3
Given blurry grayscale image how to return the object names in English step by step?	3
Given low-resolutiond noisy grayscale image how to return the caption in German step by step?	5
Given low-resolutiond noisy grayscale image how to return the class label in German step by step?	5
Given low-resolutiond noisy grayscale image how to return the object names in German step by step?	5
Given low-resolutiond noisy grayscale image how to return the caption in German step by step?	4
Given low-resolutiond noisy grayscale image how to return the class label in German step by step?	4
Given low-resolutiond noisy grayscale image how to return the object names in German step by step?	4
Given noisy grayscale image how to return the caption in German step by step?	4
Given noisy grayscale image how to return the class label in German step by step?	4
Given noisy grayscale image how to return the object names in German step by step?	4
Given noisy grayscale image how to return the caption in German step by step?	3
Given noisy grayscale image how to return the class label in German step by step?	3
Given noisy grayscale image how to return the object names in German step by step?	3
Given low-resolutiond grayscale image how to return the caption in German step by step?	4
Given low-resolutiond grayscale image how to return the class label in German step by step?	4
Given low-resolutiond grayscale image how to return the object names in German step by step?	4
Given low-resolutiond grayscale image how to return the class label in German step by step?	3
Given low-resolutiond grayscale image how to return the object names in German step by step?	3
Given low-resolutiond grayscale image how to return the class label in German step by step?	3
Given low-resolutiond grayscale image how to return the object names in German step by step?	3
Given low-resolutiond grayscale image how to return the class label in German step by step?	3
Given low-resolutiond grayscale image how to return the object names in German step by step?	3
Given low-resolutiond gray scale image how to return the class label in German step by step?	2
Given low-resolutiond grayscale image how to return the object names in German step by step?	2
Given low-resolutiond grayscale image how to return the class label in German step by step?	2
Given low-resolutiond grayscale image how to return the object names in German step by step?	5
Given low-resolutiond grayscale image how to return the class label in German step by step?	5