SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table A.6 - continued from previous page
Task Description	Difficulty Level
Given low-resolutioned noisy blurry image how to return the object names in German step by step?	5
Given low-resolutioned noisy blurry image how to return the caption in English step by step?	4
Given low-resolutioned noisy blurry image how to return the class label in English step by step?	4
Given low-resolutioned noisy blurry image how to return the object names in English step by step?	4
Given noisy blurry image how to return the caption in German step by step?	4
Given noisy blurry image how to return the class label in German step by step?	4
Given noisy blurry image how to return the object names in German step by step?	4
Given noisy blurry image how to return the caption in German step by step?	4
Given noisy blurry image how to return the object names in English step by step?	3
Given noisy blurry image how to return the caption in German step by step?	3
Given noisy blurry image how to return the object names in English step by step?	3
Given low-resolutioned blurry image how to return the caption in German step by step?	4
Given low-resolutioned blurry image how to return the class label in German step by step?	4
Given low-resolutioned blurry image how to return the object names in German step by step?	4
Given low-resolutioned blurry image how to return the caption in German step by step?	3
Given low-resolutioned blurry image how to return the class label in German step by step?	3
Given low-resolutioned blurry image how to return the object names in German step by step?	3
Given low-resolutioned blurry image how to return the caption in German step by step?	3
Given low-resolutioned blurry image how to return the class label in German step by step?	3
Given low-resolutioned blurry image how to return the object names in German step by step?	3
Given low-resolutioned blurry image how to return the caption in German step by step?	2
Given low-resolutioned noisy image how to return the object names in German step by step?	2
Given low-resolutioned noisy image how to return the caption in German step by step?	2
Given low-resolutioned noisy image how to return the class label in German step by step?	3
Given low-resolutioned noisy image how to return the object names in German step by step?	3
Given low-resolutioned noisy image how to return the caption in German step by step?	3
Given noisy image how to return the caption in German step by step?	3
Given noisy image how to return the class label in German step by step?	3
Table A.6 - continued from previous page
Task Description	Difficulty Level
Given noisy image
how to return the object names in German step by step?	3
Given noisy image
how to return the caption in English step by step?	2
Given noisy image
how to return the class label in English step by step?	2
Given noisy image
how to return the object names in English step by step?	2
Given low-resolutioned image
how to return the caption in German step by step?	3
Given low-resolutioned image
how to return the class label in German step by step?	3
Given low-resolutioned image
how to return the object names in German step by step?	3
Given low-resolutioned image
how to return the caption in English step by step?	2
Given low-resolutioned image
how to return the class label in English step by step?	2
Given low-resolutioned image
how to return the object names in English step by step?	2
Given clozed English text
how to generate a image step by step?	2
Given English text
how to generate a image step by step?	1
Given clozed English text
how to return the summarization in German step by step?	3
Given clozed English text
how to translate the text in German step by step?	2
Given clozed English text
how to return the sentiment in German step by step?	3
Given clozed English text
how to return the summarization in English step by step?	2
Given clozed English text
how to return the sentiment in English step by step?	2
Given English text
how to return the summarization in German step by step?	2
Given English text
how to translate the text in German step by step?	1
Given English text
how to return the sentiment in German step by step?	2
Given English text
how to return the summarization in English step by step?	1
Given English text
how to return the sentiment in English step by step?	1
Given low-resolutioned noisy grayscale image and clozed English query
how to answer the question in English step by step?	6
Given low-resolutioned noisy grayscale image and clozed English query
how to answer the question in German step by step?	7
Given low-resolutioned noisy grayscale image and English query
how to answer the question in English step by step?	5
Given low-resolutioned noisy grayscale image and English query
how to answer the question in German step by step?	6
Given noisy blurry grayscale image and clozed English query
how to answer the question in English step by step?	5
Given noisy blurry grayscale image and clozed English query
how to answer the question in German step by step?	6
Given noisy blurry grayscale image and English query
how to answer the question in English step by step?	4
Given noisy blurry grayscale image and English query
how to answer the question in German step by step?	5
Continued on next page
Table A.6 - continued from previous page
Task Description	Difficulty Level
Given noisy blurry image and English query how to answer the question in English step by step?	3
Given noisy blurry image and English query how to answer the question in German step by step?	4
Given low-resolutiond blurry image and clozed English query how to answer the question in English step by step?	4
Given low-resolutiond blurry image and clozed English query how to answer the question in German step by step?	5
Given low-resolutiond blurry image and English query how to answer the question in English step by step?	3
Given low-resolutiond blurry image and English query how to answer the question in German step by step?	4
Given low-resolutiond blurry image and English query how to answer the question in German step by step?	2
Given blurry image and clozed English query how to answer the question in English step by step?	3
Given blurry image and clozed English query how to answer the question in German step by step?	4
Given blurry image and English query how to answer the question in English step by step?	2
Given blurry image and English query how to answer the question in German step by step?	3
Given low-resolutiond noisy image and clozed English query how to answer the question in English step by step?	4
Given low-resolutiond noisy image and clozed English query how to answer the question in German step by step?	5
Given low-resolutiond noisy image and English query how to answer the question in English step by step?	3
Given low-resolutiond noisy image and English query how to answer the question in German step by step?	4
Given noisy image and clozed English query how to answer the question in English step by step?	3
Given noisy image and clozed English query how to answer the question in German step by step?	4
Given noisy image and English query how to answer the question in English step by step?	3
Given noisy image and English query how to answer the question in German step by step?	4
Given low-resolutiond image and clozed English query how to answer the question in German step by step?	3
Given low-resolutiond image and English query how to answer the question in German step by step?	2
Given low-resolutiond image and English query how to answer the question in German step by step?	3
Given clozed English document and clozed English query how to answer the question in German step by step?	4
Given clozed English document and clozed English query how to answer the question in English step by step?	3
Given clozed English document and English query how to answer the question in German step by step?	3
Given clozed English document and English query how to answer the question in English step by step?	2
Given English document and clozed English query how to answer the question in German step by step?	3
Given English document and English query how to answer the question in German step by step?	2
Given English document and English query how to answer the question in German step by step?	1
Prompt-1
You are a planner who is an expert at coming up with a to-do list for a given objective for the execution of a robot. Ensure the list is as short as possible. Each task in it is relevant, effective, short and necessary. The robot is only allowed to utilize the provided machine learning models to do each task. Develop a to-do list to achieve the objective: Given a noisy, blurry, grayscale image and English question related to that image, how to answer the question in German?
Provided models:
Fill Mask
Image Super-Resolution
Image Deblurring
Text-to-Image Generation
Prompt-2
Sentiment Analysis: useful when you want to analyze the sentiment of a sentence. It receives sentence as input.
Text Summarization: useful when you want to summarize a sentence or a paragraph. It receives text as input.
Machine Translation: useful when you want to translate a sentence. It receives text as input.
Fill Mask: useful when you want to fill the sentence at the masked position. It receives text as input.
Question Answering: useful when you need to answer a question based on a given context.
Image Classification: useful when you want to know the class of the image. It receives image_path as input.
Object Detection: useful when you want to detect the objects in a photo. It receives image_path as input.
Colorization: useful when you want to colorize a photo. It receives image_path as input.
Image Super-Resolution: useful when you want to create a high-resolution image from a low-resolution image.
Image Denoising: useful when you want to denoise a noisy image. It receives image as input.
Image Deblurring: useful when you want to deblur a blurry image. It receives image as input.
Visual Question Answering: useful when you need to answer a question based on a given image.
Image Captioning: useful when you want to know what is inside the photo. It receives image as input.
Text-to-Image Generation: useful when you want to generate an image based on a given description.
Figure A.6: Prompts used for experiments in Tab. 3 and Tab. 4.
Figure A.7: Another example of open-ended task. OpenAGI is instructed to generate a travel report. The backbone LLM used in this example is Vicuna-7B.