SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The work presented in this paper, Chameleon, has significant potential for positive societal impact. By augmenting large language models (LLMs) with plug-and-play modules for compositional reasoning, Chameleon can provide more accurate responses to complex, multi-modal tasks, making it a potentially valuable framework for various applications, including but not limited to education, finance, and decision support systems. Additionally, the system's ability to synthesize programs without requiring any training could democratize access to AI technology, enabling non-experts to leverage the power of AI in diverse fields. As research continues to advance in large language models and tool integration, we anticipate that our framework will serve as a foundation for further innovations in pursuing more generalizable and efficient solutions to complex reasoning tasks.
While there might be negative societal impacts associated with the Chameleon, such as misinformation and privacy concerns if data sources and external tools it utilizes are not curated meticulously, we believe these risks can be carefully managed and minimized. There's also a risk that excessive reliance on Chameleon's increased autonomy may undermine critical thinking skills or job functions. To effectively mitigate these issues, careful curation of data sources and external tools, along with a strong commitment to user data protection, are essential. Additionally, Chameleon's autonomy should be viewed as a means to augment, not replace, human capabilities. Therefore, the development of robust ethical guidelines, transparency mechanisms, and safeguards is critical, underlying our commitment to the socially responsible deployment of AI.
<sup>4</sup>https://www.usgs.gov/geology-and-ecology-of-national-parks/ecology-death-valley-national-park-0
$\triangleright$  Instruction for the planner model
You need to act as a policy model, that given a question and a modular set, determines the sequence of modules that can be executed sequentially can solve the question.
The modules are defined as follows:
Query_Generator: This module generates a search engine query for the given question. Normally, we consider using "Query_Generator" when the question involves domain-specific knowledge.
Bing_Search: This module searches the web for relevant information to the question. Normally, we consider using "Bing_Search" when the question involves domain-specific knowledge.
Image_Captioner: This module generates a caption for the given image. Normally, we consider using "Image_Captioner" when the question involves the semantic understanding of the image, and the "has_image" field in the metadata is True.
Text_Detector: This module detects the text in the given image. Normally, we consider using "Text_Detector" when the question involves the unfolding of the text in the image, e.g., diagram, chart, table, map, etc., and the "has_image" field in the metadata is True.
Knowledge_Retrieval: This module retrieves background knowledge as the hint for the given question. Normally, we consider using "Knowledge_Retrieval" when the background knowledge is helpful to guide the solution.
Solution_Generator: This module generates a detailed solution to the question based on the information provided. Normally, "Solution_Generator" will incorporate the information from "Query_Generator", "Bing_Search", "Image_Captioner", "Text_Detector", and "Knowledge_Retrieval".
Answer_Generator: This module extracts the final answer in a short form from the solution or execution result. This module normally is the last module in the prediction pipeline.
Below are some examples that map the problem to the modules.
$\triangleright$  In-context example(s)
Question: Compare the average kinetic energies of the particles in each sample. Which sample has the higher temperature?
Context: The diagrams below show two pure samples of gas in identical closed, rigid containers. Each colored ball represents one gas particle. Both samples have the same number of particles.
Options: (A) neither; the samples have the same temperature (B) sample A (C) sample B
Metadata: 'pid': 19, 'has_image': True, 'grade': 8, 'subject': 'natural science', 'topic': 'physics', 'category': 'Particle motion and energy', 'skill': 'Identify how particle motion affects temperature and pressure'
Modules: ["Text_Detector", "Knowledge_Retestival", "Solution_Generator", "Answer_Generator"]
Table 8: The prompt constructed for the planner model on the ScienceQA task. The prompt consists of the instruction that describes the role of the planner model, the in-context examples that map the problem to the module sequence, and the test example.
Program_Generator: This module generates a Python program that can solve the given question. It takes in the question and possible context and produces a program that can be executed by the "Program_Executor" module. Normally, we consider using "Program_Generator" when the questions and contexts involve complex computation, such as arithmetic operations over multiple numbers, or when the questions involve complex logical operations, such as "if-else" statements.
Program_Verifier: This module verifies whether the generated program from "Program_Generator" is valid and error-free. It checks for syntax errors, logical errors, and other potential issues that may arise during program execution.
Program Executor: This module executes the generated program from "Program_Generator" and produces an output that can be further processed by other modules, such as "Question_Answering".
Row_Lookup: This module returns the simplified table that only remains the rows that are relevant to the question. It takes in the question and a table and returns the simplified table. If all rows are relevant or there are only three rows or fewer, return the original table. Normally, we only consider using "Row_Lookup" when the table involves more than three rows and the question only requires a small number of rows to answer the question.
Column_Lookup: This module returns the simplified table that only remains the columns that are relevant to the question. It takes in the question and a table and returns the simplified table. If all columns are relevant or there are only two columns, return the original table. Normally, we consider using "Column_Lookup" when the table involves more than two columns and the question only requires a small number of columns to answer the question.
Table_Verbalizer: This module converts the table to a description that can be easily understood by the downstream modules, like "Program_Generator", "Solution_Generator", "Question_Answering". Normally, we consider using "Table_Verbalizer" when the table involves a small number of rows and columns and the table is domain-specific, such as steam-and-leaf plots, function tables, etc.
Knowledge_Retrieval: This module retrieves domain-specific knowledge for the given question and table. Normally, we consider using "Knowledge_Retrieval" when the question and table involve domain-specific knowledge, such as "steam-and-leaf plots", "function tables", "tax forms", etc.
Solution_Generator: This module generates a detailed solution to the question based on the information provided. Normally, we use "Solution_Generator" when the question and table involve simple computation, such as arithmetic operations over a single number.
Answer_Generator: This module extracts the final answer in a short form from the solution or execution result. This module normally follows the "Solution_Generator" or "Problem_Executor" module.
Table:
designer watch | $8,141
designer coat | $6,391
Question: How much more does a designer watch cost than a designer coat? (unit: $)
Modules: ["Program_Generator", "Program_Verifier", "Program Executor", "Answer_Generator"]
Table 9: The prompt constructed for the planner model on the TabMWP task. Similarly, the prompt consists of the instruction, the in-context examples, and the test example.
$\triangleright$  Instruction
Read the following question, and generate the background knowledge as the context information that could be helpful for answering the question.
Question: Which property do these three objects have in common?
Options: (A) hard (B) soft (C) yellow
Metadata: 'pid': 43, 'has_image': True, 'grade': 4, 'subject': 'natural science', 'topic': 'physics', 'category': 'Materials', 'skill': 'Compare properties of objects'
Detected text in the image: ['handkerchief', 'slippers', 'leisure suit']
Knowledge:
- This question is about comparing the properties of three objects: a handkerchief, slippers, and a leisure suit.
- The objects are related to the topic of physics and the skill of comparing properties of objects.
- Properties of objects can include physical characteristics such as color, texture, shape, size, weight, and material.
Table 10: The prompt constructed for the "Knowledge Retrieval" module on the ScienceQA task.
Read the following question and metadata, and generate the query for browser search as the context information that could be helpful for answering the question.
Question: Which property do these two objects have in common?
Options: (A) hard (B) bendable
Metadata: 'pid': 329, 'has_image': True, 'grade': 2, 'subject': 'natural science', 'topic': 'physics', 'category': 'Materials', 'skill': 'Compare properties of objects'
Detected text in the image: [(41, 183], [131, 183], [131, 199], [41, 199]), 'rubber gloves'), ([245, 183], [313, 183], [313, 197], [245, 197]), 'rain boots')]
Search Query: Common material properties of jump rope and rubber gloves
Table 11: The prompt constructed for the "Query Generator" module on the ScienceQA task.
Given the question (and the context), select the answer from the options ["A", "B", "C", "D", "E"]. You should give concise and step-by-step solutions. Finally, conclude the answer in the format of "the answer is [ANSWER]", where [ANSWER] is one from the options ["A", "B", "C", "D", "E"]. For example, "the answer is A", "the answer is B", "the answer is C", "the answer is D", or "the answer is E". If the answer is not in the options, select the most possible option.
Context: Select the better answer.
Metadata: 'pid': 6493, 'has_image': True, 'grade': 2, 'subject': 'natural science', 'topic': 'physics', 'category': 'Materials', 'skill': 'Compare properties of objects'
Image caption: A pair of scissors next to a pair of scissors.
Detected text with coordinates in the image: [([[53, 185], [121, 185], [121, 199], [53, 199]], 'jump rope'), ([[233, 183], [323, 183], [323, 201], [233, 201]], 'rubber gloves')
Retrieved knowledge:
- This question is about comparing the properties of two objects: rubber gloves and rain boots.
- The objects are related to the topic of physics and the skill of comparing properties of objects.
- Properties of objects can include physical characteristics such as color, texture, shape, size, weight, and material. In this case, the two objects have the property of being bendable in common.
Bing search response: The most common materials used for disposable gloves are Latex, Vinyl and Nitrile. Each material has its benefits and drawbacks. Latex Gloves are constructed from Natural Rubber Latex and are the most popular type of disposable glove.
Solution: An object has different properties. A property of an object can tell you how it looks, feels, tastes, or smells. Different objects can have the same properties. You can use these properties to put objects into groups. Look at each object. For each object, decide if it has that property. A bendable object can be bent without breaking. Both objects are bendable. A hard object keeps its shape when you squeeze it. The rubber gloves are not hard. The property that both objects have in common is bendable. Therefore, the answer is B.
Table 12: The prompt constructed for the "Solution Generator" module on the ScienceQA task.
▷ Instruction
Read the following table and question, and generate the domain-specific knowledge as the context information that could be helpful for answering the question.
▷ In-context example(s)
Table:
x | y
10 | 15
11 | 9
12 | 2
Question: The table shows a function. Is the function linear or nonlinear?
Knowledge:
- A linear function is a function whose graph is a straight line.
- A nonlinear function is a function whose graph is not a straight line.
- The equation of a linear function is y = mx + b, where m is the slope and b is the y-intercept.
- The equation of a nonlinear function is not y = mx + b.
Table 13: The prompt constructed for the "Knowledge Retrieval" module on the TabMWP task.
Read the following question and table. Each row is separated by a newline (\n') and each column is separated by a vertical bar ('l'). Return the simplified table that only remains the rows that are relevant to the question. If all rows are relevant, or the number of rows is fewer than three, return the original table.
Question: In preparation for graduation, some teachers and students volunteered for the various graduation committees. How many people are on the music committee?
Committee | Students | Teachers
Program | 5 | 17
Ticket | 20 | 5
Music | 20 | 15
Schedule | 15 | 20
Food | 18 | 2
Simplified Table:
Table 14: The prompt constructed for the "Row Lookup" module on the TabMWP task.
▷ Instruction
Read the following question and table. Each row is separated by a newline (‘\n’) and each column is separated by a vertical bar (‘l’). Return the simplified table that only remains the columns that are relevant to the question. If all columns are relevant, return the original table.
▷ In-context example(s)
Question: Look at the following schedule. When does Recess end?
Table:
Subject | Begin | End
Recess | 6:15 A.M. | 7:20 A.M.
Orchestra | 7:30 A.M. | 8:40 A.M.
Art | 8:45 A.M. | 9:35 A.M.
Handwriting | 9:45 A.M. | 10:20 A.M.
Gym | 10:30 A.M. | 11:15 A.M.
Choir | 11:20 A.M. | 12:25 P.M.
Science | 12:35 P.M. | 1:35 P.M.
Reading | 1:40 P.M. | 2:50 P.M.
Simplified Table:
Subject | End
Recess | 7:20 A.M.
Orchestra | 8:40 A.M.
Art | 9:35 A.M.
Handwriting | 10:20 A.M.
Gym | 11:15 A.M.
Choir | 12:25 P.M.
Science | 1:35 P.M.
Reading | 2:50 P.M.
Table 15: The prompt constructed for the "Column Lookup" module on the TabMWP task.
Read the following question and table. Write a textual description of the table. The description should keep the critical information in the table for answering the question. The description should not answer the question.
Table description: The table shows the number of students and teachers on each of the four graduation committees: Program, Ticket, Music, and Schedule. The Music committee has 20 students and 15 teachers.
Table 16: The prompt constructed for the "Table Verbalizer" module on the TabMWP task.
Read the following table and then write Python code to answer a question.
Price | Quantity demanded | Quantity supplied
$895 | 21,000 | 3.400
$945 | 17,200 | 7,400
$995 | 13,400 | 11,400
$1,045 | 9,600 | 15,400
$1,095 | 5,800 | 19,400
Questions: Look at the table. Then answer the question. At a price of $995, is there a shortage or a surplus? Please select from the following options: ['shortage', 'surplus'].
Code:
Python Code, return 'ans'. Make sure that 'ans' is a string selected from the options in the question quantity_demanded_at_price_955 = 13400 quantity-supplied_at_price_955 = 11400 if quantity_demanded_at_price_955 > quantity-supplied_at_price_955: ans = 'shortage' else: ans = 'surplus'
Table 17: The prompt constructed for the "Program Generator" module on the TabMWP task.
Read the following table and then answer a question.
$895 | 21,000 | 3,400
Question: Look at the table. Then answer the question. At a price of $995, is there a shortage or a surplus? Please select from the following options: ['shortage', 'surplus'].
Solution: At the price of $995, the quantity demanded is greater than the quantity supplied. There is not enough of the good or service for sale at that price. So, there is a shortage. The answer is shortage.
Table 18: The prompt constructed for the "Solution Generator" module on the TabMWP task.
Figure 7: Transitions between modules in programs generated by Chameleon (GPT-4) on ScienceQA. START is the start symbol, END is a terminal symbol and the others are non-terminal symbols.
Figure 8: Transitions between modules in programs generated by Chameleon (GPT-4) on TabMWP. START is the start symbol, END is a terminal symbol and the others are non-terminal symbols.
Figure 9: One more example from our Chameleon (GPT-4) approach on ScienceQA.
Figure 10: One more example from our Chameleon (GPT-4) approach on TabMWP.
$\triangleright$  Query
Question: Which better describes the Death Valley ecosystem?
Context: Figure: Death Valley. Death Valley is a desert ecosystem in eastern California.
Options: (A) It has dry, thin soil. It also has many different types of organisms. (B) It has a small amount of rain. It also has only a few types of organisms.
Metadata: 'pid': '18973', 'has_image': True, 'grade': 3, 'subject': 'natural science', 'topic': 'biology', 'category': 'Ecosystems', 'skill': 'Describe ecosystems'
$\triangleright$  Response
Generated Program: ["Query_Generator", "Bing_Search", "Solution_Generator", "Answer_Generator"]
Query Generator:
Characteristics of Death Valley ecosystem
Bing Search: (the first one from response candidates)
Visitors come to Death Valley to experience the stark and lonely vastness of the valley, the panorama of rugged canyons and mountains; the pleasures of the dry, moderate winter climate, the challenge of the hot, arid summer; the relief of the cooler mountains; and the reminders of frontier and Native American ways of life.
Solution Generator:
Death Valley is a desert ecosystem, which means it has specific characteristics related to its climate and the organisms that live there. A desert ecosystem typically has dry, thin soil and a small amount of rain. This environment supports only a few types of organisms that are adapted to survive in such harsh conditions. Therefore, the answer is B.
Answer Generator (Wrong prediction):
(B) It has a small amount of rain. It also has only a few types of organisms.
Correct Answer (Ground truth):
(A) It has dry, thin soil. It also has many different types of organisms.
Table 19: An example of failure cases in ScienceQA from Chameleon (GPT-4) where some modules perform inaccurately. The generated query from the "Query Generator" module is insufficiently specific, leading to a response from "Bing Search" that does not adequately detail the characteristics of the Death Valley ecosystem, particularly with regard to the types of organisms present. (From U.S. Geological Survey: Contrary to its name, Death Valley National Park is very much alive ... Despite the low precipitation and extreme summer temperatures, Death Valley is home to many diverse species, several of which are endemic (found nowhere else in the world).)
Question: Which property matches this object?
Options: (A) stretchy (B) yellow
Metadata: 'pid': '10338', 'has_image': True, 'grade': 2, 'subject': 'natural science', 'topic': 'physics', 'category': 'Materials', 'skill': 'Identify properties of an object'
helium balloons
Generated Program: ["Image_Captioner", "Solution_Generator", "Answer_Generator"]
Image Captioner:
A colorful kite is being flown in the sky.
Look at the image of the object, which is a colorful kite. A stretchy object can be stretched without breaking. Kites are typically made of materials that are not stretchy, such as nylon or polyester. A yellow object has a yellow color. The kite in the image is colorful and includes the color yellow. Therefore, the answer is B.
Answer Generator (Wrong prediction): (B) yellow
Correct Answer (Ground truth): (A) stretchy
Table 20: An example of failure cases in ScienceQA from Chameleon (GPT-4) where the generated programs are suboptimal. The LLM planner does not call the "Text Detector" and "Knowledge Retrieval" tools, resulting in an absence of semantic information from the image and a lack of related domain knowledge. Additionally, the employed image captioning model in providing an accurate textual description of the given visual context.
Context: Below is a food web from an ocean ecosystem in Monterey Bay, off the coast of California. A food web models how the matter eaten by organisms moves through an ecosystem. The arrows in a food web represent how matter moves between organisms in an ecosystem.
Options: (A) bat star (B) phytoplankton (C) kelp bass
Metadata: {'pid': '15175', 'has_image': True, 'grade': 6, 'subject': 'natural science', 'topic': 'biology', 'category': 'Ecological interactions', 'skill': 'Interpret food webs II'}
Generated Program: ["Text_Detector", "Knowledge_Retestual", "Solution_Generator", "Answer_Generator"]
Text Detector:
[[297, 91], 'orca'], [[653, 85], 'sea otter'),
([487, 191], 'kelp bass'), ([73, 221], 'phytoplankton'),
([625, 269], 'black rockfish'), ([323, 327], 'zooplankton'),
([149, 363], 'plainfin'), ([593, 361], 'kelp'),
([132, 380], 'midshipman'), ([401, 391], 'bat star'),
([271, 433], 'sea cucumber'), ([409, 471], 'sea urchin')
Knowledge Retrieval:
- The question is about a food web in an ocean ecosystem in Monterey Bay, California.
- A food web models how matter eaten by organisms moves through an ecosystem.
- The arrows in a food web represent how matter moves between organisms in an ecosystem.
- The organisms mentioned in the question are bat star, phytoplankton, and kelp bass.
- The target organism is the sea cucumber.
- In a food web, matter moves from one organism to another through consumption, predation, or decomposition.
- Understanding the relationships between organisms in a food web can help determine the flow of matter between them.
Analyze the food web to determine which organism's matter eventually moves to the sea cucumber.
- The sea cucumber consumes the bat star.
- The bat star consumes the sea urchin.
- The sea urchin consumes the kelp.
- The kelp gets its nutrients from the water, which contains matter from phytoplankton and other sources.
In this food web, the matter from the bat star eventually moves to the sea cucumber through consumption. Therefore, the answer is A.
Answer Generator (Wrong prediction): (A) bat star
Correct Answer (Ground truth): (B) phytoplankton
Table 21: An example of failure cases in ScienceQA from Chameleon (GPT-4) where the current module inventory falls short in addressing specific abilities. In this instance, the module inventory does not possess the necessary tools capable of parsing the arrows in the food web illustration.
Table:
x	y
4	15
19	18
11	10
11	1
Table: (semi-structured format)
x | y  
4 | 15  
19 | 18  
11 | 10  
11 | 1
Question: Look at this table. Is this relation a function?
Options: (A) yes (B) no
Generated Program: ["Knowledge_Retrieval", "Program_Generator", "Program_Verifier", "Program_Executor", "Answer_Generator"]
- A relation is a set of ordered pairs  $(x, y)$ .
- A function is a relation in which each input ( $x$ -value) is paired with exactly one output ( $y$ -value).
- To determine if a relation is a function, check if each  $x$ -value is paired with only one  $y$ -value.
- If an  $x$ -value is paired with more than one  $y$ -value, the relation is not a function.
Program Generator:
relation  $=$  { 4:15, 19:18, 11:10, 11:1 }   
x_values  $=$  list(relation.keys())   
unique_x_values  $=$  set(x_values)   
if len(x_values)  $= =$  len(unique_x_values): ans  $=$  'yes'   
else: ans  $=$  'no'
Program Verifier: True
Program Executor:
ans = 'yes'
Answer Generator (Wrong prediction): (A) yes
Correct Answer (Ground truth): (B) no
Table 22: An example of failure cases in TabMWP from Chameleon (GPT-4) where some modules perform inaccurately. In this case, the LLM-based program generator is unsuccessful in creating a program that correctly mirrors the mathematical relationship expressed in the provided query and the retrieved domain knowledge.
Question: This is Oliver's complete financial record for September. How much money did Oliver have after he purchased the basketball?
Table:
Date	Description	Received	Expenses	Available Funds
Balance: end of August	|	$154.90
9/1	notebook	| $6.95	$147.95
9/15	walking dogs	$15.00	| $162.95
9/21	basketball	| $11.35	$151.60