Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models
Authors: Pan Lu
Baolin Peng
Hao Cheng
Michel Galley
Kai-Wei Chang
Ying Nian Wu
Song-Chun Zhu
Jianfeng Gao
Venue: NeurIPS 2023 poster (2023)
Keywords: large language models
compositional reasoning
tool use
multi-modal reasoning
mathematical reasoning
TL;DR: This paper introduces Chameleon, an approach that enhances large language models with plug-and-play tools for compositional reasoning, thereby demonstrating significant improvements in complex reasoning tasks.

Abstract: Large language models (LLMs) have achieved remarkable progress in solving various natural language processing tasks due to emergent reasoning abilities. However, LLMs have inherent limitations as they are incapable of accessing up-to-date information (stored on the Web or in task-specific knowledge bases), using external tools, and performing precise mathematical and logical reasoning. In this paper, we present Chameleon, an AI system that mitigates these limitations by augmenting LLMs with plug-and-play modules for compositional reasoning. Chameleon synthesizes programs by composing various tools (e.g., LLMs, off-the-shelf vision models, web search engines, Python functions, and heuristic-based modules) for accomplishing complex reasoning tasks. At the heart of Chameleon is an LLM-based planner that assembles a sequence of tools to execute to generate the final response. We showcase the effectiveness of Chameleon on two multi-modal knowledge-intensive reasoning tasks: ScienceQA and TabMWP. Chameleon, powered by GPT-4, achieves an 86.54% overall accuracy on ScienceQA, improving the best published few-shot result by 11.37%. On TabMWP, GPT-4-powered Chameleon improves the accuracy by 17.0%, lifting the state of the art to 98.78%. Our analysis also shows that the GPT-4-powered planner exhibits more consistent and rational tool selection via inferring potential constraints from instructions, compared to a ChatGPT-powered planner.

Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models
Pan Lu Baolin Peng". Hao Cheng Michel Galley?
Kai-Wei Chang', Ying Nian Wu', Song-Chun Zhu', Jianfeng Gao University of California, Los Angeles Microsoft Research, Redmond
https://chameeeen--1 falm..gthuup github.
Image Captorer
Knowledge Retrieaal
Solaton Generator
Answer Gonerato
Whet the divection of this push??
(A) from the baseball bit loward the basetell
[164, 21j:
[244, 70]: plates now
Earthay paper
Arelyaa the advertisement determime which persuasive appeaal is used. The ad menoons that Earthay paper plates carry the Slarra Club seal of approval.
[172: 112) carry the Sierra
This imples that the product envirrmentaaly friendl and has been andorsed by a reputabll organization This appeal peiarlly hosed on the credibblity and authority the Sierr Club, which an evample of ethos (character).
PS11 150) Club seal of
[255, 198) approval
pathoss (emotion)
othos (charaztrr logns (nessio))
Text Datector
Knooledgee Retrieval
S
Solution Generator
Anewnr Gererater
Which the main persuasive
appeaal used in this ad??
qu question ahout idderilying the main persuesive appeal used in an aevertisemmnt types of persuasive appeals: ethos, pathos, and logos.
ethos (character)
are
Ethoos is an appeal to the speaker's or wrier's credibility, character, or authority.
Patthos appeal to audiencels amations, foolings, sympethies.
Logos an appeal oogic reason, rationallty using facls, statitics, arguments.
Which animal's skin adapted for surviall in cold places??
(A) Eurosian tynx
Image Captioner
Generator
Search
Solution Generator
@
Answer Generator
(B) Thomyy Devil
Figure I: Examples from our Chameleon approach with GPT-4 on ScienceQA [32], multi-modal question answering benchmark in scientific domains. Chameleon is adaptive to different queries by synthesizing programs to compose various tools and executing them sequentially to get final answers.
Abstract
Large language models (LLMs) have achieved remarkable progress in solving various natural language processing tasks due to emergent reasoning abilities. How- ever, LLMs have inherent limitations as they are incapable of accessing up-to-date information (stored the Web in task-specific knowledge bases), using external tools, and performing precise mathematical and logical reasoning. In this paper, we present Chameleon, an AI system that mitigates these limitations by augmenting LLMs with flug-andd-lay modules for compositional reasoning. Chame lon synthesizes programs by composing various tools (e.g., LLMs, off-the-shelf vi- sion models, web search engines, Python functions, and heuristic- based modules) for accomplishing complex reasoning tasks. At the heart of Chameleon. an LLM-based planner that assembles a sequence of tools to execute to generate the final response. We showcase the effectiveness of Chameleon on two multi-modal knowledge-intensing reasoning tasks: ScienceQA and TabMWP. Chameleon, powered by GPT-4, achieves an 86.54% overall accuracy on ScienceQA, improv- ing the best published few-shot result by 11.37%. On TabMWP, GPT-44powered Chamel eon leo improves the accuracy by 17.0%, lifting the state of the art to 98%... Our analysis also shows that the GPT-4-powered planner exhibits more consistent and rational tool selection via inferring potential constraints from instructions, compared to ChatGPT-powere planner.
This title draws inspiration from the chameleon's ability to adapt and blend into its surroundings, which parallels the adaptability and versatility of large language models in compositional reasoning tasks with external tools.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
Remarkable progress has been observed in recent large language models (LLMs) for various natural language processing tasks, with prominent examples such as GPT-3 [4], PaI.M [8], LLaMA [64], ChatGPT [41], and the recently developed GPT-4 [42]. LLMs have demonstrated emergent abilities, including in-context learning and chain-of-thought (CoT) reasoning [56]. These models are capable of solving diverse tasks in a zero-shot fashion [25] or with the aid of few examples [57], and they show great potential in planning and decision-making akin to human beings [17. 16]. Despite these capabilities, LLMs face inherent limitations, such as an inability to access up-to-date information [26]. perform precise mathematical reasoning [44, 35], utilize specialized models [49]. Therefore, enhancing current LLMs with the capability to automatically compose external tools for real-world task solving is critical to address these drawbacks.
Consider the example @ in Figure I: Which is the main persuasive appeal used in this ad?. To answer this question, one needs to: 1) infer that there is ad image containing text context and call a text decoder to understand the semantics; 2) retrieve background knowledge about persuasive appeals and the differences among three persuasive appeals; 3) generate solution based on the input query and intermediate results from previous steps; and 4) finally produce the answer in a task-specific format. On the other hand, when answering Which animal's 's skin is adapted for survival in cold places (3), one might need to call modules such as an image captioner to decipher image information and a web search engine to retrieve domain knowledge understand scientific terminologies. However, current tool-augmest -ugmeeted LLMs still face challenges when addressing these real-world queries across various scenarios. Most existing approaches are either limited to a small number of tools [39, 6, 55, 18, 43, 49] or relying on domain- specific tools [40, 60, 13, 59, 52], and thus are not easy to generalize to queries of new domains (see sections 2 and A. for further discussion). In this work, we study how to enable LLMs to synthesize programs to capture the logic of composing heterogeneous tools.
To address the challenges of existing work, we introduce Chameleon a plug-and-play compositional reasoning framework that leverages LLMs to synthesize programs and compose various tools for a wide range of tasks. Unlike existing tool-augmented LLMs [49, 40, 60, 13, 59, 52], Chameleon leon uses richer set of tools, including LLMs, off-the-shelf vision models, web search engines, Python functions, and heuristics-bair based modules. Moreover, Chameleon leverages the in-context learning ca- pabilities of LLMs and builds on an LLM as a natural language planner, without requiring any training or carefully curated rules. Prompted by tool descriptions and usage examples, the planner infers a pro- gram composed of a sequence of tools to execute in order to generate the final response for a user query. Instead of generating programs in Nomain-speciff languages [40, 52, 13], Chameleon generates natural- nntural-language like (NL) programs (e.g., Srext_Detectte
Retrievall Thowleddg__Retrieaal. for the second query Figure 1). The NL- like programs are easy to understand and debug by users with limited programming experience, and easily extendable to new modules. During each module's s execution, the module processes the query and cached context, returns result determined by the module itself, and updates the query and context for subsequent execution. Composing modules as a sequential program allows subsequent modules to leverage prior cached context and updated queries.
Soluta Lluuiooonneeren Generator:
We showcase the adaptability and effectiveness of Chameleon on two tasks: ScienceQA [32] and TabMWP [33]. ScienceQA is multi-modal question answering benchmark spanning multiple context formats and various scientific topics, while TabMWP is mathematical benchmark involving diverse tabular contexts. These two benchmarks serve as good testbed to evaluate Chameleon' ' ability to coordinate diverse tools across different types and domains. Notably, Chameleon with GPT-4 achieves an 86.54% accuracy on ScienceQA, significantly improving upon the best published few-shot model by 11.37% On TabMWP, using GPT-4 as the underlying LLM, Chame achieves improvement of 7.97% over chain-of-thought (CoT) prompted GPT-4 [57] and a 17.0% increase over the best-published model [6], lifting the state of the art to 98. 78%. Further studies suggest that using GPT-4 as planner exhibits more consistent and rational tool selection and is able to infer potential constraints given the instructions, compared to other LLMs like ChatGPT.
Our contributions are as follows: (1) We develop a plug-and and- and-play compositional reasoning framework, Chameleon, that effectively composes external tools to address inherent limitations of LLMs and tackle a broad range of reasoning tasks. (2) Relying on an LLM as natural language planner to generate programs, Chameleor successfully integrates various tools, including LLMss, the- vision models, web search engines, Python functions, and rule-based modules, to build a versatile and

---

Miles hiked
Knowledge Retrieval
This table showe the number of miles Wanda hiked each day camping The median the middle value the arranged order. find median, data must be arranged order from least greafest (or greatest to leesf) and then the middle value(s) Islare determined.
Day
Miles
Sunday 10
Monday
9
Program Generator
Tuesday 10
edles, Miked [18, 9, 18, 9]
Wednescay
niles_hiked sortee(fmilss_hikee
niddlet (len(ntles (iikel))e_____kk)) 11 //
Thursday
9
Program Ventfer
middle2 Jen(miles hikkee // 2
Wanda went on a camping trlp and logged the number of miles ahe hiked each day. What is the median of the numbers?
ans (eilos (idddd + miles hiked[/iddlel)) /
Program Evecutor
ane 9.0
Answer Generator
Location
Arrive
Depart
stadium
A.:20
10:25 A.M.
Location
Arrive
Depart
park
10:35 A.M.
10:45 A.M.
Row Lookup
station
12:25 P.M. 12:35
hotel
11:10 A.M.
A.M15A.M.
airport
12:05 M.
12
(Step 1) Find the train station the schedule. Find the departure
Irain station
12:25
P.M.
time for the station.
bus station
1:10
1:10 P.M.
Solution Generator
(Step Train stadion: 12:35 P.M. The departs from train station 12:35 P.M. (Step 3) The answer is 12:35 P.M.
Look the following schedule. When does the bus dipart from the frain station?
(c) 1:10 P.M. (D) 10:45 A.M.
12:35 P.M. 1:10
Answer Generator
12:35
Figure 2: Two examples from our Chameleon approach with GPT-4 on TabMWP [33], a mathemat- ical reasoning benchmark with tabular contexts. Chameleon demonstrates flexibility and efficiency in adapting to different queries that require various reasoning abilities.
adaptable Al system capable of answering real-world queries. (3) We demonstrate leon's effectiveness on two challenging benchmarks, significantly surpassing the state of the art.
2 Related Work
Compositional Reasoning Neural modular and compositional approaches have been explored to automatically perform desired sub-task decomposition, enhancing interpretability and adaptability across various reasoning tasks. Early work (2. 3] posits that complex reasoning tasks are fundamen- tally compositional and proposes neural module networks (NMN) to decompose them into subtasks. However, these methods rely on brittle off-the-shelf parsers and are limited by module configurations. Some later work [19, 15, 14, 21J, takes step further by predicting instance-specific network layouts in an end-to-end manner, without relying on parsers, using reinforcement learning [58] and weak supervised learning. In visual reasoning, models comprising a program generator and an execution engine have been proposed to combine deep representation learning and symbolic program execution [19, 61]. In the domain of mathematical reasoning, an interpretable solver has been developed to incorporate theorem knowledge as conditional rules and perform symbolic reasoning step by step [31]. Our work takes inspiration from neural module networks, yet it offers several distinct advantages. First, Chameleon does not require expensive supervision of task-specific programs for modeling training. Instead, it generates sequential programs, consisting of modules, that are easy to generalize to various domains and tasks, allowing the extension to new modules in plug-and-play manner. Second, Chame leon does not require any training, but uses the in-context learning capabilities of LLMs to generate programs prompted by natural language instruction and demonstrations.
Tool-Augmented Language Models In recent years, the development of large language models (LLMMs [48, 8, 9, 53, 41, 42] has made tremendous progress and has stimulated research in prompt learning [57, 33, 22] and instruction learning [53, 64, 46, 11]. Despite the impressive performance of LLMs, they suffer from inherent limitations, such as the inability to access up-to-date information [26]. utilize external tools [49], or perform precise mathematical reasoming [44, 35]. Recent benchmarks, such ScienceQA and TabMWP [32, 33, 7, 54, 51, 30J, have emerged to evaluate the capability of LLMS to tackle intricate reasoning challenges. especially those emphasizing the use of external tools. Concurrently, there has been growing interest in harnessing external tools and modular approaches to augment LLMs. These augmented LLMs can access real-time information aided by web search engines [40] and leverage -ppeifi knowledge from external resources [62]. Some work leverages the Python interpreter to generate complex programs to employ powerful computational resources, and execute logical reasoning tasks more effectively [55, 10, 6, 39, 18, 43, 36). For example, Toolformer [49] constructs tool-use augmented data to train language models to select five
3

---

Tool Use
Skill Dimension
Inference & Extension
Model
Size
?
Image
Web
Know,
Math
Table
Composition
Planning
Plug-n-Play
CoT
x
x
X
X
X
Lila [39]
X
X
x
X
x
PoT [6]
/
K
X
X
/
X
X
K
X
X
Code4Struct [55]
X
X
x
PAL [10]
X
x
/
X
X
X
x
X
X
MathPrompter [18]
2
/
X
X
X
V
x
X
ART [43]
4
/
X
X
V
X
V
X
/
X
V
X
Toolformer [49]
X
X
/
X
X
X
nutural lang- program
X
WehGiPT [40]]
10
X
X
X X
X
X
X
X
X
MM-ReAcl [60]
/
/
word match natural lang. program program natural lang.
Visual GPT [59]
>10
/
X X
x
X
X
ViperGPT [52]
510
VisProg [13]]
>10
X
HiggingGPP [50]
>10
/
/
X
X
X
X
Chameleon (ours)
(ours)
>10
natural lang.
Table 1: comparison of work that augments large language models with tool usage. We report the tool size and tool types, including OpenAI (Ø), Hugging Face (8). Github (0). Web search (b), and code (e). We compare the skills each method possesses, such as image understanding, browser search, knowledge retrieval, mathematical reasoning, and table understanding. Some models can compose various tools, propose a planner to infer the relevant tools for execution, or are inherently extendable new tools. The label "_" refers uncertain information in the literature.
tools. In the realm of visual tools, various approaches have been proposed to enhance the capabilities of large language models in handling visual tasks [60, 59, 52, 13, 50], augmented with Hugging Face models [50], Azure models [60], visual foundation models [59].
We compare Chameleon with other tool-augmented language models in Table 1. Many of these approaches are either constrained
small set of tools or limited to task-specific tools, which reduces their capabilities across various skill dimensions and hampers their generalizability to new tasks. A recent line of work relies on large amounts of supervision [49, 26] and focuses on generating commands [40] and programs [52, 13] to infer the choice of tools. However, this approach needs to carefully tailored prompts to specific tasks and particular tools, and is neither flexible nor adaptive. In contrast, Chameleon instructs LLMs with natural language instructions that simply describe the roles of each module and provide a few calling examples, climinating the need for additional training tool-specific prompts when learning to compose different tools. More importantly, Chame aeameleon offers users flexibility in terms of tool types and sources, updating the underlying LLMs, adding new tools, and adapting new tasks. Our work shares the same spirit of AutoGPT [47], an autonomous GPT-4 agent with the artificial general intelligence (AGI) ambition to incorporate numerous tools to achieve user-defined goals. While AutoGPT is still under development, our work is the first to instantiate the idea and verify its effectiveness on well-studied benchmarks.
3 General Framework: Chameleon
To address the limitations of current LLMs in utilizing diverse tools, we propose Chameleon, a novel fiirg-and-play compositional reasoning framework, synthesizing the composition of various tools to accommodate wide range of problems. Chameleon is comprised of module inventory that defines different types of tools and an LLM-based planner, whose purpose is to decompose the original problem into sub-tasks that can effectively solved by task-specific tools. Unlike existing tool-augmented L.M approaches [49, 13, 59, 50), our module inventory features multiple tool types as illustrated in Table 2, enabling Chameleon to exhibit various reasoning abilities, including image understanding. knowledge retrieval, web search, complex mathematical reasoning, and table understanding. Instead of generating domain-specific programs [40, 13, 52], Chame employs an LLM-based planner to create natural-language-- instructions, which is less error-prone, easily expandable to new modules, and user- friendiindll.
programs that follow natural language
We formalize our planner follows: given the input query the module inventory M, and constraints G, the natural language planner P selects set of modules that can be executed sequentially to answer the query via generating a program in natural- lanural-lngguagellike format. The module inventory M consists of a set of prilb modules: (M,). each corresponding to a tool of various types (Table 2). G are the constraints for the plan generation, for example, the concurrent relations and sequence

---

orders of modules, In our work, the planner P is an LLM prompted to generate a sequence of module names in few-shot setup. The planner prompted in natural language with a planning task instruction I, the descriptions of modules in M with corresponding constraints G, as well as few demonstration examples D. A T-length plan sampled from P can denoted as p = M....., MT, where M' represents an the t-th element in the generated plan and M E MM. Formally, given an input query (problem statement) zo. plan p is generated as follows:
p M(,
Given the generated plan, the corresponding modules for each step are then executed sequentially. The plan is a natural-language program where each module is bound simply via string matching. When evaluating the module M at time step t, the output of the execution y is calculated by:
y' + (x--1,,,
(2)
where x²-1 is the input for the current module M and c² the cached information (e.g.. image semantics, retrieved knowledge, generated programs) resulting from the execution history of modules. Both the problem input x' and cache et for the next module M are updated, respectively, by:
x¹ uppate_inputt("'"_
+ update_cache((-11
The opdate_input and update_cachi functions are hand-designed for each M,. Specifically, update_ing e_input is applied to elements in the input query, including the question, table context, and image. These elements are updated after module execution. apdate_cacht generation of new information, such description for the input image retrieved knowledge from external resources. Finally, the response r to the query is generated by the last module M
corresponds to the
yd t M (x (
(5)
4 Applications of Chameleon
We demonstrate the applications of Chameleon on two challenging tasks: ScienceQA [32] (section 4.2) and TabMWP [33] (section 4.3), using the module inventory introduced in section 4.1. Further experimental details can be found in appendix A.2.
4.1 Module Inventory
To accommodate various reasoning capabilities over a diverse range of queries, our system uti- lizes rich module inventory of various external tools. We provide high-level overview of this inventory here, with detailed implementations in specific experiments. The complete module in- ventory, M, is presented in Table 2. Each tool within the inventory is defined as follows:
Tool Types
Tools
Knowledge Retrieval, Query Generatur,
Row Lookup, Column Lookup,
OpenAl
Table Verbalizer, Program Generator,
Solution Generator
Hugging Face Image Captioner
Github
Text Detector
Web Scarch
Bing Scarch
This module retrieves ad- ditional background knowledge crucial for tack- ling complex problems. is especially beneficial for specialized domains like science and mathe- matics, providing context for the task. For example, if a query is about tax form table, this module could generate knowledge about tax procedures, offering valuable context.
Knowledge Retrieval:
Python
Program Verifier, Program Executor
Rule-based
Answer Generator
Table 2: Different tools in our module inventory.
Bing Search: Like "Knowledge Retrieval", the "Bing Search" module aims to provide wide-ranging task-relevant knowledge. contrast, it excels when broader or up-to-date information from multiple sources is required. Using the search engine API, this module returns relevant search results based on the input query, which are then parsed and used by subsequent modules to gather richer context information from diverse sources, enhancing problem-solving effectiveness.
Query Generator: Since the original problem typically lacks tailored query for retrieving task- relevant information, this module creates search engine queries based on the problem, which are then
5

---

used by the "Bing Search" module. Mostly, it is good strategy to use the "Query Generator" module before the "Bing Search". Coupled with the search engine tool, generating more targeted queries generally facilitates both the recall and precision of retrieved information.
Image Captioner: Designed generate captions for images, this module provides crucial supple- mentary context for queries. It is particularly valuable when understanding an image semantically, scene. Using pre-trained models, translates visual data into language, facilitating effective comprehension and reasoning about image content.
like identifying objects and interactions
Text Detector:
This module is designed to identify text within given image. Typically, the "Text Detector" is employed when a question requires the extraction of textual information from images containing diagrams, charts, tables, maps, or other visual elements. By effectively detecting text in various formats, this module aids in the analysis and understanding of image-based content.
This module is crucial when queries involve tabular context, as locating relevant cells often required. Large tables can distract the system, so "Row Lookup" simplifies the table by retaining only the rows relevant to the query. If all rows are pertinent, it returns the original table.
Row Lookup:
Column Lookup: Like the "Row Lookup" module, "Column Lookup" addresses questions involving tabular context by focusing on relevant columns. It simplifies the table by retaining only pertinent columns, or returns the original table if all columns are relevant.
Table Verbalizer: Converting structured tables into text is likely to enhance the comprehension of
tabular information by various downstream modules as shown by [37] for open-domain question answering, making this module a vital part of our system. It translates tables into easily understandable descriptions for modules like "Program Generator" and "Solution Generator", particularly useful for small, domain-specific tables like stem-and-leat plots or function tables.
Program-aided approaches are shown to enhance the logical and mathematical reasoning abilities of LLMs [55, 10, 6, 39, 18, 43]. The "Program Generator" generates Python programs to solve queries effectively, which is particularly beneficial for queries requiring complex computations or intricate logical operations, such as "if-else" statements.
Program Generator:
Program Verifier: Recent studies highlight the importance of verification to reduce hallucination [45, 38). Hence, "Program Verifier" ensures the validity and error-free nature of by "Program Generator". It checks for syntax and logical errors, and potential execution issues, enhancing the reliability and accuracy the solutions.
programs generated
Program Executor:
This module executes the program generated by "Program Generator" and produces the result, bridging the gap between program generation and final solution derivation.
This module generates a detailed solution to the input query using all the cached information. Employing chain-of tf-uuugghtttt prompting approach [57]. it ensures coherent and well-structured responses. The planner can directly employ this module instead of other functional modules if it can solve the query independently, especially for simpler ones.
Solution Generator:
This task-specific module uses a rule-based approach extract and normalize answers from the results of the "Program Executor" or "Solution Generator". Unlike the Solution Generator" that provides detailed multi-step solutions, "Answer Generator" serves as the final module in the pipeline, providing concise and task-specific answers.
Answer Generator:
4.2 Science Question Answering
Science Question Answering (ScienceQA [32]) is a diverse benchmark for multi-modal question answering over a range of scientific topics and contexts. As examples illustrated in Figure I, answering these questions requires various tools and skills like image captioning, text detection, knowledge retrieval, online resource search, and multi-clue visual reasoning. When generating programs for using tools, we limit the search space to the relevant inventory subset (Table in the appendix). Programs are deemed invalid and default to a "Solution Generator" and "Answer Generator" sequence if these are not the final two elements, following the chain-of-thought prompting baseline [57]. See Table 8 in the appendix for the constructed natural language planner prompt. The prompts for L.LM-based modules like "Knowledge Retrieval", "Query Generator". and "Solution Generator" are shown in Table 10, 11, and 12, respectively, in the appendix.

---

iTunndd Params
Model
ALL
NAT
SOC
LAN
TXT
IMG
G1-6
G7-12
Heuristic boselines
Random Choice [32]
39.83
40.28
46.13
29.25
47.45
40.08
33.66
39.35
40.67
Human [32]
88.40
90.23
84.97
87.48
89.60
87.50
88.10
91.59
82.42
Fine-tuned models
MCAN [63]
95M
54.54
56.08
46.23
58.09
59.43
51.17
55.40
51.65
59.72
Top-Down []]
70M
59.02
59.50
54.33
61.82
62.90
54.88
59.79
57.27
62.16
BAN [23]
112M
59.37
60.88
46.57
66.64
62.61
52.60
65.51
56.83
63.94
DFAF [12]
74M
60.72
64.03
48.82
63.55
65.88
54.49
64.11
57.12
67.17
ViT.T [24]
113M
61.14
60.48
63.89
60.27
63.20
61.38
57.00
60.72
61.90
Patch-TRM [34]
90M
61.42
65.19
46.79
65.55
66.96
55,28
64.95
58.04
67.50
VisualBERT [27, 28]
111M
59.33
61.18
62.71
62.17
58.54
62.96
$9,92
UnifiedQA [20]
223M
70.12
68.16
69.18
74.91
63.78
61.38
84
72.98
UnifiedQA C6T [32]
223M
74.11
71.00
76.04
78.91
66.42
66.53
81.81
77.06
68.82
MM-COT, [65]
223M
70.53
71.09
70.75
69.18
71.16
65.84
71.57
71.00
69.68
MM-COT [65]
223M
84.91
87.52
77.17
85.82
87.88
82,90
86.83
84.65
85.37
MM-COT [65]
738M
91.68
95.91
82.00
90.82
9:526
88.80
92.89
92.44
90.31
LLaMA-Adapkerz [64]
1,2M
78.31
79.00
73.79
80.55
78.30
70.35
83.14
79.77
75.68
LLaMA-Adapler [64]
1.8M
85.19
84.37
88.30
84.36
83.72
80.32
86.90
85.83
84.05
Few-shot GPT:3
GPT-3 [4]
74.04
75.04
66.59
78.00
74.24
65.74
79.58
76.36
69.87
GPT-3 [32)
75.17
75.44
74.68
67.43
79.93
78.23
69.68
Published results (Above)
Fhw-shot ChatGPT
ChatGPT CoT
0M
78.31
78.82
70.98
83.18
77.37
67.92
86.13
80.72
74.03
Chameleon (ChatGPT)
0M
79.93
81.62
70.64
84.00
79.77
70.80
86.62
81.86
76.53
Few-shot GPT-4
GPT-4 CoT
0M
83.99
85.48
72.44
90.27
82.65
71.49
92.89
86.66
79.04
Chameleon (GPT-4
86.54
89.83
74.13
89.82
88.27
77.64
92.13
88,03
83.72
Table 3: QA accuracy (%) on the test of ScienceQA [32]. We report the number of tuned parameters for this task and the overall accuracy, along with accuracy scores for different question types, including natural, social, and language sciences, text, image, and no context, as well as grades 1-6 and 7-12. The highest scores among models in each section and overall are highlighted in blue and red, respectively, and the results of our best model are marked bold.
4.3 Tabular Mathematical Reasoning
TabMWP [33] is mathematical reasoning task involving diverse tabular contexts like schedules, prices, tax forms, plots, and function relations (Figure 2). requires Al systems understand various table formats and perform precise numerical symbolic computations. Like ScienceQA. we constrain the program search space to focus on two tool types: 1) those helping LLMs better digest tabular information (e.g.. "Row Lookup". "Column Lookup". and "Table Verbalizer") and 2) those performing faithful symbolic computations (e.g.. "Program Generator". "Program Verifier", and "Program Executor") as listed in Table 6. The generated programs must meet certain constraints, such as including "Answer Generator" and placing "Program Generator" prior to both "Program Verifier" and "Program Executor". Non-compliant programs default to a sequence of "Program Generator", "Program Verifier", "Program Executor", and "Answer Generator". aligning with the ofogram-oofthhouht prompting baseline [6] with added verification.
Experiments
We assess Chameleon's effectiveness and adaptability on two complex reasoning tasks, ScienceQA [32] and TabMWP [33]. See experimental details in appendix A.2.
5.1 Experimental Results
ScienceQA. Table 3 presents the results of existing baselines and our approach Chameleon, with key results highlighted in Figure 3 (a). Employing ChatGPT [41] the base LLM, Chameleon
7

---

#Tunedd Params
Model
ALL
FREE
MC
INT
DEC
EXTR
BOOL
OTH
G1-6
G7-8
Heuristic baselines
Heuristic guess
15.29
6.71
39.81
8.37
0.26
30.80
51.22
26.67
17.55
12.27
Human performance
90.22
84.61
93.32
84.95
83.29
97.18
88.69
96.20
94.27
81.28
Fine-tuned models
UnifiedQAssAu [20]
41M
29.79
22.27
51.31
27.27
2.83
52.28
48.11 11
69.52
35.85
21.71
UnifiedQAsAse [20]
223M
43.52
34.02
70.68
40.74
7.90
84.09
55.67
73.33
53.31
30.46
UnieeedQAAovaL [20]
738M
57.35
48.67
82.18
55.97
20.26
94.63
68.89
79.05
65.92
45.92
TAPEX [29]
139M
48.27
39.59
73.09
46.85
11.33
84.19
61.33
69.52
56.70
37.02
T [29]
406M
58.52
51.00
80.02
59.92
16.31
95.34
64.00
73.33
67.11
47.07
Zem--shot GPT-3
GPT-3 [4]
0M
56.96
53.57
66.67
55.55
45.84
78.22
55.44
54.29
63.37
48.41
GPT-3 CoT [57]
0M
57.61
54.36
66.92
55.82
48.67
78.82
55.67
51.43
63.62
49.59
Few-shor GPT-3
GPT-3 [4]
0M
57.13
54.69
64.11
58,36
40.40
75.95
52.41
53.02
63.10 10
49.16
GPT-3 CuT
0M
62.92
60.76
69.09
60.04
63.58
76.49
61.19
67.30
68.62
55.31
GPT-3 CoT-PrompiPG [33]
0M
68 23
66.17
74.11
64.12
74.16
76.19
72.81
65.71
71.20
64.27
Codex* [5]
0M
59.4
Codex PoT [6]
0M
73.2
Codex PoT-SC* [6]
0M
81.8
Published results (Abuve) A
Few-shot CharGPT
ChatGPT CoT
0M
82.03
78.43
92.32
75.38
90.30
92.30
92.89
87.62
83.06
80.66
ChatGPT PoT
0M
89.49
90.24
87.35
89.31
93.82
92.10
85,5
55.24
90.60
88.00
Chameleon (ChatGPT)
0M
93.28
93.13
93.72
92.71
94.76
91.29
98.11
78.85
93.37
93.17
Few-shot GPT-4
GPT-4 CoT
0M
90.81
88.48
97.49
86.16
97.51
89.52
92.40
GPT-4
0M
96.93
97.40
95.58
93.22
96.25
98.00
68.57
96.97
96.87
Chameleon (GPT-4)
0M
98.78
98.95
98.29
99.34
97.42
98.58
98.56
93.33
98.95
98.54
Table 4: QA accuracy (%) on the test set of TabMWP [33]. We report the number of tuned parameters for this task and the overall accuracy, and accuracy of different question types, including free-text questions, multi-choice questions, integer answers, decimal answers, extractive answers, Boolean answers, other text answers, grades 1-6, and grades 7-8. refers subset of results.
Ours (few-shot)
Few-shot
Fine tue-tuuee
Ours (few-shot)
Few-shot
(GPT-4)
Chameleon (GPT-4)
98.8
GPT-4 CoT
06.0
GPT-4 PoT
96.9
Chamslean (ChalGPT)
-74.9
GPT-4 CoT
90.6.
CharGPT CoT
78.3
Chameleon (CrwiGPT)
JB.5
GPT.3 CoT
75.2
ChatGPT PoT
89.5
UnfiedDA CoT
700-
ChaigPT CoT
92.0
LLaWsA-Adapeer
Codox PoT-SC
81.8
MM-COT
24..3
CoT-PhompiPG
GPT-3
68
85.0
70.0
75.0
80.0
85.0
90.0
66.0
90.0
(a) Results on ScienceQA
(b) Results on TabMWP
Figure 3: Results of main baselines and Chameleon. Dashed lines represent human performance.
achieves a 79.93% accuracy, a 1.62% improvement over Chain-of-Thought (CoT) [57] prompted ChatGPT. Notably, Chameleon is generalized form of CoT, where the generated program is sequence "Solution Generator" and "Answer Generator". Chameleon benefits from additional tool usage, such as "Knowledge Retrieval", "Bing Search", "Image Captioner", and "Text Detector". When built upon GPT-4 [42], our model attains an accuracy of 86.54% outperforming GPT-4 CoT [32] by 2.55% and GPT-3 CoT by 11.37% creating the new state of the art in few-shot settings.
TabMWP. Table presents results with key models in Figure (b). Similarly, significant improve- ments are observed for Chameleon over both fine-tuned and few-shot models. It is worth noting that both CoT and Program-of-Thoughe (PoT) [6] can be viewed as special cases of Chameleon Apart from "Solution Generator" and "Answer Generator". CoT doesn't 't utilize any tool, while PoT

---

Called
called
Knowledge Retrieval
Image Captioner
Query Generator
Bing Search
Chameleon (ChatGPT)
9?
Chameleon (GPT-4)
Figure 4: Tools called in the generated programs from Chameleon on ScienceQA.
Called
Not called
Row Lookupp
Column Lookup
Table Verbalizer
Knowledge Retrieval
Program Generator
Chameleon (ChatGPT) 47
5i
96
97
16
B4
24
Chameleon (GPT-4)
26
74
93
75
Figure 5: Tools called in the generated programs from Chameleon TabMWP.
only relies on symbolic programming tools like "Program Generator" and "Program Executor". Chameleon (ChatGPT) outperforms ChatGPT CoT and ChatGPT PoT by 11.25% and 3.79%, respectively, emphasizing the advantage of our enriched tool set. With GPT-4, Chameleon gains an additional 5,50%, reaching a 98.78 78% accuracy. Notably, Chameleon (GPT-4) surpasses Codex PoT-SC [6], the best-published model, by 17.0% and human performance by 8.56%.
5.2 Qualitative Analysis
Tool use planning. The proportions of key tools called in the programs from Chameleon on ScienceQA and TabMWP visualized in Figure 4 and Figure respectively, Interestingly, Chat- GPT and GPT-4 exhibit different planning behaviors. Generally, ChatGPT has a strong bias toward using or not using certain tools, highly influenced by in-context examples. For instance, ChatGPT calls "Knowledge Retrieval" in 72% of queries but only calls "Bing Search" in 3% of cases on ScienceQA; on TabMWP, ChatGPT heavily relies on "Row Lookup" (47%) but calls "Column Lookup" less frequently (4%). However, GPT-4 acts more objectively and rationally in tool selection. For example, GPT-4 calls "Knowledge Retrieval" more frequently (81% vs. 72%) and calls "Bing Search" more than ChatGPT (11% vs. 3%) when answering scientific questions on ScienceQA. Impressively, GPT-4 consistently calls "Query Generator" and "Bing Search" simultaneously by observing the tool usage descriptions, while ChatGPT lacks such reasoning capability.
Ablation study with disabled modules. We study the accuracy decline of Chameleon when key modules in the generated programs are disabled (Table 5), using ChaptGPT the underlying L.MSs and 500 test exam- ples. The results reveal that "Knowledge Retrieval" plays a vital role in both tasks. Domain-speciff tools, such as the search engine and vision models for ScienceQA, and program tools for TabMWP, also prove to be important.
Module
A (ScienceQA) Á (TabMWP)
Knowledge Retrieval
-7.8%
-2.2%
Bing Search
-7.4%
Text Detector
-8.4%.
Image Captioner
-6.0%
Program Generator
-7.4%
Table Verbulizer
-0.2%
Table Score drop with disabled modules.
Module transitions. We visualize the transition graphs of modules for generated programs by Chameleon (GPT-4) on ScienceQA and TabMWP in Figure 7 and 8, respectively. The transition probabilities in these graphs computed from the tool transitions observed graphs show that the GPT-4 planner is able to make good decisions on how to sequence tools in a few- shot setup. For example, on ScienceQA, Chameleon often decides rely either "Knowledge Retriever" or "Bing Search", but rarely both. On TabMWP, we observe two main modes: either going through the solution generator module or via the program generator, verifier, and executor.
the test sets. These
5.3 Case Study
Visualization examples of ScienceQA. Examples from Chame (GPT-4) on ScienceQA are visualized in Figure Chameleon (GPT-4) is able to adapt to different input queries by generating programs that compose various tools and executing them sequentially to obtain accurate responses. For instance, to answer the first question (Q), What is the direction of this push?, the system calls the image captioner model to extract semantic information from the image and employs the knowledge retrieval model to gather background knowledge for multi-modal reasoning. In the second example (2), the natural language planner infers that text detector tool is needed to understand the context

---

of the ad. The third query (3; more details provided in Figure 9 in the appendix), Which animal's skin adapted for survival in cold places?, involves scientific terminology related animal survival. The planner decides to call the Bing search engine to access domain-specific knowledge, benefiting from the numerous online resources.
Visualization examples of TabMWP. The adaptability and versatility of Chame for various queries are also observed on TabMWP, as illustrated in the examples in Figure The first example (Q) involves mathematical reasoning on tax form. Chameleon (1) calls the knowledge retrieval model to recall basic knowledge that assists understanding this domain-specifie table, (2) describes the table in more readable natural language format, and (3) finally relies program-aided tools to perform precise computations. In the second example (@), the system generates Python code that closely aligns with the background knowledge provided by the knowledge retrieval model. The third example (8) requires the system to locate the cell in a large tabular context given the input query. Chame leon calls the row lookup model to help accurately locate the relevant rows and generate the language solution via an LLM model, instead of relying on program-bare based tools.
Failure cases and limitations. Failure examples from Chameleon (GPT-4) are illustrated in Tables 19 24 the appendix. Inaccurate responses may arise from the limitations of the current modules or from suboptimal programs generated by the planner. Additionally, the module inventory may lack tools capable addressing specific abilities. Future directions could involve upgrading the modules and the planner, expanding the module inventory to support a broader range of capabilities. Further limitations and broader impacts are respectively discussed in sections and C of the appendix.
5.4 Error Analysis
To examine the eтrоr sources of the base large language models and understand how our model reduces mistakes from different aspects, we conduct an error analysis, as shown Figure We select 50 mistake examples from the ChatGPT baseline on ScienceQA as the evaluation set. We count the number of mistake examples and analyze their corresponding mistake type categories for ChatGPT, our Chameleon (ChatGPT) approach, and Chameleon (GPT-4).
The results show that our Chameleon approach can substantially reduce the number of mistakes compared to Chat- GPT. Our model features tools for image captioning and knowledge retrieval, thus the mistakes made by ChatGPT in the cat- egory of image understanding are reduced to 10 and 19 from 32 by Chameleon (ChatGPT) and Chameleon (GPT-4);
Number of Mistake Examples Different Categories
ChatGPT
Number Mistakes Exampless
(ChatGPT) Chameleon (GPT-4)
40
35
32
30
28
while the mistakes made by ChatGPT the category of knowledge under- standing are reduced to 6 and 3 from 37 by Chameleon (ChatGPT) and Chameleon (GPT-4). Benefiting from the sequential execution of tools, the mis- takes caused by solution generation significantly reduced as well. Addition- ally, find that the task planning of GPT- 4 outperforms ChatGPT by a large margin.
20
37
10:
10
All
Tool Planning
Image Mistake Type Category
Knowledge
Solution
Figure 6: of mistake examples in different categories ScienceQA. Image: image captioning, Knowledge: knowledge understanding, Solution: solution generation.
Conclusion
conclusion, we introduce novel plug-and-plaa compositional reasoning framework, Chameleon. that addresses the limitations of current large language models by augmenting them with external tools plug-and-ple manner. Our approach employs diverse set of tools and demonstrates impres- sive adaptability and effectiveness on two challenging benchmarks, ScienceQA and TabMWP. By achieving significant improvements accuracy over existing state-of-the-ar models, Chameleon showcases its potential for addressing real-world queries across various domains.
10

---

Acknowledgment
We would like to thank Chunyuan Li, Qiuyuan Huang, and other members of the Deep Learning group at Microsoft Research for their valuable discussions. We also thank Fan Yin from University of California, Los Angeles, and Mingyang Sun from University of Electronic Science and Technology of China for their thorough review of our paper and constructive feedback. Pan Lu's research for this work was financially supported by Microsoft during his visit at Microsoft Research, and was also partially supported by the Amazon PhD Fellowship. Bloomberg PhD Fellowship, Qualcomm Innovation Fellowship, and UCLA Dissertation Year Fellowship. Kai-Wei was supported an ONR grant N00014-23-1-42 and as a Sloan Fellow.
References
[1] Peter Anderson, Xiaodong He, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2018.
[2] Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Learning to compose neural networks for question answering. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1545-1554, 2016.
[3] Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 39-48. 2016.
[4] Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. Advances neural information processing systems, 33:1877-1901, 2020.
[5] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03377 2021.
[6] Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research, 2023.
[7] Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Elaine Wan, Xueguang Ma, Jianyu Xu, Tony Xia, and Xinyi Wang. TheoremQA: A theorem-drive question answering dataset. In The 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023.
[8] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022.
[9] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Fric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction- cinettionn- language models, arxiv preprint Xiv::22000. 11466, 2022.
[10] Luyu Gao, Aman Madaan, Shuyan Zhou, Uni Alon, Pengfei Liu, Yiming Yang. Jamie Callan, and Graham Neubig. Pal: Program-aide aided language models. In International Conference on Machine Learning, pages 10764-10799. PMLR, 2023.
[11] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui Hc, Xiangyu Yue, Hongsheng Li, and Yu Qiao. LLaMA Addtter V2: Parameter-efficicnt visual instruction model. arXiv preprint arXiv:2304.1500
2023.
[12] Peng Gao, Zhengkai Jiang, Haoxuan You, Pan Lu, Steven CH Hoi, Xiaogang Wang, and Hongsheng Li. Dynamic fusion with intra-and moter-modality attention flow for visual question answering. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6639-6648. 2019.
[13] Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 14953-14962, 2023.
[14] Ronghang Hu, Jacob Andreas, Trevor Darrell, and Kate Saenko. Explainable neural computation via stack neural module networks. In Proceedings of the European conference on computer vision (ECCV), pages 53 69, 2018.
11

---

[15] Ronghang Hu, Jacob Andreas. Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 804-813, 2017.
[16] Wenlong Huang. Picter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pages 18 9147. PMLR, 2022.
[17] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chehotar, et al. Inner monologue: Embodied reasoning through planning with language models. In 6th Anmual Conference on Robot Learning, 2022.
[18] Shima Imani, Liang Du, and Harsh Shrivastava. Mathprompier: Mathematical reasoning using large language models. In ICLR 2023 Workshop on Trustworthy and Reliable Large-Scale Machine Learning Models, 2023.
[19] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Judy Hoffmman, Li Fei-Fei, C Lawrence Zit- nick, and Ross Girshick. Inferring and executing programs for visual reasoning. In Proceedings of the IEEE international conference computer vision (ICCV), pages
2017.
[20] Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannanch Hajishirzi. UnifiedQA: Crossing format boundaries with a single QA system. In Findings of the Association for Computational Linguistics (EMNLP), pages 1896-1907, 2020.
[21] Tushar Khot, Daniel Khashabi, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Text modular networks: Learning to decompose tasks in the language of existing models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational I Linguistics: Human Language Technologies, pages 1264-12799 2021.
[22] Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. In International Conference on Learning Representations (ICLR), 2023.
[23] Jin-Hwa Kim, Jachyun Jun, and Byoung-Tiae Tak Zhang. Bilinear attention networks. In Advances in Neural Information Processing Systems (NeurIPS), pages 1571-1581, 2018
[24] Wonjae Kim, Bokyung Son, and Ildoo Kim. ViLT: Vision-and language transformer without convolution or region supervision. In Proceedings of the 38th International Conference on Machine Learning (ICMI.), pages 5583-5594, 2021.
[25] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-222133, 2022.
[26] Mojtaba Komeili, Kurt Shuster, and Jason Weston. Internet-augmented dialogue generation. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 8460-847 8478, 2022.
[27] Liunian Harold Li. Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Wei Chang. VisualBERT: A simple and performant baseline for vision and language. arXiv preprint 19Xiv:1908.0333 03555, 2019.
[28] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsich, and Kai- Aei Chang. What does BERT with vision look at? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 5265-5275, 2020.
[29] Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and Jian-Guang Lou. TAPEX: Ta- ble pre-training via learning a neural sql exccutor. In International Conference on Learning Representations (ICLR), 2022.
[30] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li. Hannaneh Hajishirzi, Hao Cheng, Kai- Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating math reasoning in visual contexts with gpt-4v, bard, and other large multimodal models. arXiv preprint arXiv:2310.02255, 2023.
[31] Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang. Xiaodan Liang, and Song- Chng-Chun220 Zhu, Inter- GPS: Interpretable geometry problem solving with formal language and symbolic reasoning. In The 59th Anmual Meeting of the Association for Computational Linguistics (ACL), 2021.
12

---

[32] Pan Lu, Swaroop Mishra. Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord. Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering, In The 36th Conference on Neural Information Processing Systems (NeurIPS). 2022.
[33] Pan Lu, Liang Qiu, Kai-Wei Chang. Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for eemisstructured mathematical reasoning. In International Conference on Learning Representations (ICLR), 2023.
[34] Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia. Yizhou Zhao, Wei Zhang, Zhou Yu. Xiaodan Liang, and Song-Chun Zhu. IconQA: new benchmark for abstract diagram understanding and visual language reasoning. In The 35th Conference on Neural Information Processing Systems (NeurlPS) Track on Datasets and Benchmarks, 2021.
[35] Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai-Wei Wei Chang. A survey of deep learning for mathematical reasoning. In The 61st Annual Meeting of the Association for Computational Linguistics (ACL), 2023.
[36] Yujie Lu, Pan Lu, Zhiyu Chen, Wanrong Zhu, Xin Eric Wang, and William Yang Wang. Multimodal procedural planning via dual text-image prompting, arXiv preprint arXiv:2305.0179: 2023.
[37] Kaixin Ma, Hao Cheng, Xiaodong Liu, Eric Nyberg, and Jianfeng Gao. Open domain question answering with a unified knowledge interface. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 1605-1620. Dublin, Ireland, May 2022. Association for Computational Linguistics.
[38] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-Refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303. 22.3003.77753 77651, 2023.
[39] Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang. Sean Welleck, Chitta Baral, Tanmay Rajpuro- hit, Oyvind Tafjord. Ashish Sabharwal, Peter Clark, and Ashwin Kalyan. Lila: A unified benchmark for mathematical reasoning. In The 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022.
[40] Reiihhrro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. WebGPT: Browser-assisted question- question-answering with human feedback. arXiv preprint arXiv:21 2021.
[41] OpenAL. Chatgpt, 2022.
[42] OpenAL. GPT-4 technical report. ArXiv, abs/2303.08774 087774 2023.
[43] Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. ART: Automatic multi-step reasoning and tool-use for large language models. arXiv preprint arxiv:2333.09014, 2023.
[44] Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080-2094, 2021.
[45] Baolin Peng, Michel Galley, Pengeheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint ar3iv:2302. /2813, 2023.
[46] Baolin Peng, Chunyuan Li, Pengeheng Hc, Michel Galley, and Jianfeng Gao. Instruction tuning with GPT-4, arXiv preprint wrXivv2304.03222 2023.
[47] Toran Bruce Richards. Auto GPT: An experimental open-source attempt to make GPT-4 fully autonomous. https: // thub. com/ S gn f cant- SrrantGGavii tas AuttttoPPP 2023.
[48] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili Daniel Hesslow. Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint or ww.:::2211.000121 2022.
[49] Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint 200. 2023.
13

---

[50] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. HuggingGPT: Solving tasks with chatgpt and its friends in huggingface. arXiv preprini urXiv:2303.77580 2023.
[51] Liangtai Sun, Yang Han, Zihan Zhao, Da Ma. Zhennan Shen, Baocai Chen, Lu Chen, and Kai Yu. SciEval: multi-level large language model evaluation benchmark for scientific research. arXiv preprint arXiv:2308. 13149, 2023.
[52] Dídac Surís, Sachit Menon, and Carl Vondrick. ViperGPT: Visual inference via python execution for reasoning. arXiv preprint arXiv:2303.00 08128888. 2023.
[53] Hugo Touvron. Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Bapliste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprini arXiv 13971, 2023.
[54] Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun Loomba, Shichang Zhang. Yizhou Sun, and Wei Wang. SciBench: Evaluating college-level scientific problem- solving abilities of large language models. arXiv preprint wrXiv22307.0633 2023.
[55] Xingyao Wang, Sha Li, and Heng Ji. Code4Struct: Code generation for few-shot structured prediction from natural language. arXiv preprint wwXi:2220..281111 2022.
[56] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler. et al. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022.
[57] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, al. Chain-of- prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems. 35:24824 24837, 2022.
[58] Ronald J Williams. Simple statistical
algorithms for connectionist reinforcement
learning. Reinforcement learning, pages 5 32, 1992.
[59] Chenfei Wu, Shengming Yin. Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual ChatGPT: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.0467. 2023.
[60] Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azamasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. AMM-EACT: Prompting ChatGPT for multimodal reasoning and action. arXiv preprint ar3iv:2303. 11381, 2023.
[61] Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural- symbolic VQA: Disentangling reasoning from vision and language understanding. Advances in neural information processing systems (NeurIPS), 31, 2018
[62] Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. Generate rather than retrieve: Large language models are strong context generators. In International Conference on Learning Representationss (ICLR), 2023.
[63] Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. In Proceedings of the IEF.F. Conference on Computer Vision and Pattern Recognition (CVPR), pages 6281-6290, 2019.
[64] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Qiao Yu. LLaMA-Adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arX:::2333. 16199, 2023.
[65] Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal thain- reasoning in language models. arXiv preprint 2xXv:2330..0092, 2023.
14

---

Supplementary Materials for Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models
A Appendix
A.1 Current Tool-Augmented LLMs
To address the limitations LLMs, an active research direction involves augmenting language models with access to external tools and resources, well exploring the integration of external tools and plug-and-play modular approaches. For example, aided by web search engines and external knowledge resources, LLMs are able access real-time information and leverage domain-specific knowledge [40]. To enhance mathematical reasoning abilities, recent work uses L.LMs [5] to generate complex programs exploit powerful computational resources, and execute logical reasoning tasks more effectively [55, 10, 6, 39, 18, 43]. Another line of recent work, such as ViperGPT [52], Visual ChatGPT [59], VisProg [13], and HuggingGPT [50] incorporates a collection of foundation computer vision models to equip LLMs with the abilities to perform visual reasoning tasks.
A.2 Experimental Details
Module search space.
The inventory subsets for ScienceQA and TabMWP are shown in Table 6.
Tool Types
Tools used on ScienceQA
Tools used on TabMWP
Knowledge Retrieval, Query
Knowledge Retrieval, Row Lookup, Column Lookup, Table Verbalizer, Program Generator, Solution Generator
OpenAl
Generator, Solution Generator
Hugging Face
Image Captioner
Github
Text Detector
Web Scarch
Bing Search
Python
Program Verifier, Program Executor
Rule-based
Answer Generator
Answer Generator
Table Tools used on ScienceQA and TabMWP, respectively. Reusable tools are marked in green.
Planner implementations. We choose the gpt-3.5-turbe engine for ChatGPT and the gpt-4 engine for GPT-4 when constructing the LLM- b--ad planner. The maximum length for generated programs set to 128, and the temperature is set to for the most deterministic generation. The planner prompts for the ScienceQA and TabMWP are illustrated in Table Table 9, respectively.
Module implementations for SeienceQA. By default, the LLM--bssd models four in-context examples as demonstrations, have temperature setting of 0, and allow maximum of 512 tokens for completion. Additional specific implementation details are provided as follows:
Knowledge Retrieval: The prompt consists of demonstration examples and the template is shown in Table 10.
Query Generator: The prompt template shown in Table 11. The maximum number of tokens for completion is set as 64.
Solution Generator: The prompt consists of 2 demonstration examples and the template
shown in Table 12.
15

---

Image Captioner:
We use the captioning model' to generate textual descriptions for input images. The maximum length of generated captions is set 16, the number of beams and the maximum number of output tokens is 512.
Text Detector: This module based on the github model to extract the text contents with coordinates in the image.
Bing Search: This module calls the Bing Search API and returns the top three responses for the text query.
This module extracts the answer snippet from the result provided by the "Solution Generator" and selects the most similar option from the given choices.
Answer Generator:
Module implementations for TabMWP. Similar to ScienceQA, the LL--based modules by default use four in-context examples demonstrations, have a temperature setting of O, and allow maximum of 512 tokens for completion. Additional implementation details are provided as follows:
Knowledge Retrieval: The prompt consists of 5 demonstration examples and the template is shown in Table 13.
Row Lookup: It is enabled only when there are more than three rows and 18 table cells, in order to accelerate inference. The prompt consists of 7 demonstration examples and the template is shown in Table 14. The maximum number of tokens for completion is set as 256.
Column Lookup: Similarly, this module is enabled with two more columns and 18 or more table cells. The prompt consists of 6 demonstration examples and the template is shown Table The maximum number of tokens for completion set as 256.
Table Verbalizer: The prompt consists of 7 demonstration examples and the template is shown Table 16.
Program Generator:
The prompt template is shown in Table 17. The maximum number of tokens for completion is set as 256.
Solution Generator: The prompt consists of 16 demonstration examples and the template is shown in Table 18.
Answer Generator: is used to normalize answers with two-place precision for questions with numerical answers and select the most similar option for multiple-choie questions.
aopdate_cache. cache. @odate_input is triggered by the execution of specific tools, like "Row_Lookupps which alter replace elements in the input reflect the updated state. Tools such as Siaage_Captionerr, "Text_Deeector 'Knowledge_Retreee 'Web WWeb_Search' and 'Program Generation' generate elements. update_cache stores these new elements in the cache, making them accessible for later tools' execution.
Implementations of update input and update
A.3 Experimental Results
Chameleon utilizes the LLM-based natural language planner to generate programs, i.e., sequences of used modules (tools). We report the statistics of the number of unique generated programs and the average length of corresponding tool sequences by Chame leon in Table both ScienceQA and TabMWP, using GPT-4 as the base L.MM generates fewer distinct programs, i.e., more consistent programs, than using ChatGPT, even when given the exact same prompt in the planning model. Our results are consistent with the findings in [42], which observes that GPT-4 has superior capability of understanding long contexts, aligning with human instructions, and performing high-level reasoning compared to other LLMs such as ChatGPT.
Generated program statisties.
B Limitations
While Chameleon represents a significant stride in exploiting large language models (LLMs) for compositional reasoning in plug-and-play manner, there are few areas that could benefit from
co/n http:///loooocc//iiiieooo wttpttttttt-ttte- image caption captionint Pnttps://github wwt.//aio/a/aaaaaaaaa....o.. //aayy/Ce btttps:/www..iceeee wtt.e/ooooo...cooiiioooeoe.eeeee
16

---

Task
Model
# of different programs
Average program length
Chain-of-thought (CoT)
2
ScieneeQA
Chameleon (ChatGPT)
14
3.03
Chameleon (GPT-4)
3,40
Chain-of-thought (CoT)
Program-of thought (PoT)
3
TabMWP
Chameleou (ChatGPT)
28
4.17
Chameleon (GPT-4)
19
4.09
Table 7: The statistics the number different generated programs and the average length of generated programs by Chameleon respectively. Chain-of-thought (CoT) prompting and Program- of-thought (PoT) prompting are also compared as they are the special cases Chameleon.
further refinement. One such area is the expansion of its adaptability wider variety of tasks and domains, beyond the benchmarks presented. The LLM-based planner, responsible for synthesizing programs and determining the sequence of tools, introduces an innovative approach, yet also raises intriguing research questions about optimizing the process for tool selection and sequence. It is plausible in the current system design that the quality of the LLMM-ased planner could impact overall performance. Moreover, Chameleon generates the program at one step, without incorporating re- re-planning mechanism the modules in the program are processed. Furthermore, we make the assumption that the list of modules and their descriptions will fit within the context window of LLMs, which may not always be the case. As the task complexity increases and the module inventory expands, there might be corresponding surge in computational demands or limitations due to the context limit, indicating potential areas for future optimization. However, these potential areas for enhancement don't detract from the paper's central achievements, but instead provide valuable directions for future work and research.
C Broader Impacts
The work presented in this paper, Chameleon, has significant potential for positive societal impact. By augmenting large language models (LLMs) with plug-and-play modules for compositional reasoning, Chameleon can provide more accurate responses complex, multi-modal tasks, making it a potentially valuable framework for various applications, including but not limited to education, finance, and decision support systems. Additionally, the system's ability to synthesize programs without requiring any training could democratize access to AI technology, enabling non- con-experts leverage the power of AI in diverse fields. As research continues to advance in large language models and tool integration, we anticipate that our framework will serve as foundation for further innovations in pursuing more generalizable and efficient solutions to complex reasoning tasks.
While there might be negative societal impacts associated with the Chameleon such as misinforma- tion and privacy concerns data sources and external tools it utilizes believe these risks can be carefully managed and minimized. There's 's also risk that excessive reliance on Chameleon increased autonomy may undermine critical thinking skills or job functions. To effectively mitigate these issues, careful curation of data sources and external tools, along with strong commitment user data protection, are essential. Additionally, Chameleon should be viewed as a means to augment, not replace, human capabilities. Therefore, the development of robust ethical guidelines, transparency mechanisms, and safeguards is critical, underlying our commitment to the socially responsible deployment of AI.
curated meticulously, we
autonomy
17

---

Instruction for the planner model
You need act as a policy model, that given question and modular set, determines the sequence of modules that can be executed sequentially can solve the question.
The modules are defined as follows:
This module generates a search engine query for the given question. Normally, we consider using "Query_Geneeee Generator" when the question involves domain-specific knowledge. Bing Search: This module searches the web for relevant information the question. Normally, we consider using "Bing_ "earch" when the question involves comain--specife knowledge.
Capato_Cptioo::: This module generates a caption for the given image. Normally, we consider using "Image_ Captioner" when the question involves the semantic understanding of the image, and the "has image" field in the metadata is True.
Text Detector: This module detects the text in the given image. Normally, we consider using "Text_Detectov" when the question involves the unfolding of the text in the image, e.g., diagram, chart, table, map, etc., and the thas_image" field the metadata is True.
Knowledge Retrieval: This module retrieves background knowledge the hint for the given question. Normally, we consider using Ghowledge_Retrievvee when the background knowledge is helpful to guide the solution.
Solution Solution_Generatorr This module generates detailed solution to the question based on the information provided. Normally, "Solution_Generalle will incorporate the information from Snnery_Geeeattrr" "Bing Search". Iatage_Captionee"s "Text Detector", and "Knowl- edge Retrieval".
Answer Generator: This module extracts the final answer in short form from the solution or execution result. This module normally the last module in the prediction pipeline.
Below are some examples that map the problem to the modules.
> In-context example(s)
Compare the average kinetic energies of the particles in each sample. Which sample has the higher temperature?
Question:
The diagrams below show two pure samples of gas identical closed, rigid containers. Each colored ball represents gas particle. Both samples have the same number of particles.
Context:
Options: (A) neither; the samples have the same temperature (B) sample A (C) sample B
Metadata: 'pid': 19, 'has_in mass_immme': True, 'grade': 'subject': 'natural science', 'topic': 'physies',
'category': 'Particle motion and energy', 'skill': 'Identify how particle motion affects temperature and pressure'
Modules: @eText_Detecton "Knowledge_ oegrerreerrraallllltttooooeeeeer r", "Answer_GGenerator!!
Table 8: The prompt constructed for the planner model on the ScienceQA task. The prompt consists of the instruction that describes the role of the planner model, the in-context examples that map the problem the module sequence, and the example.
18

---

> Instruction for the planner model
You need to act as a policy model, that given question and modular set, determines the sequence of modules that can be executed sequentially can solve the question.
The modules are defined as follows:
Program Generator: This module generates a Python program that can solve the given question. It takes in the question and possible context and produces a program that can be executed by the "Program_Fss "xecutor"""" module. Normally, we consider using "Program "eantattreerr"r when the questions and contexts involve complex computation, such as arithmetic operations over multiple numbers, or when the questions involve complex logical operations, such as "if-else" statements. Program_Verineee Verifier: This module verifies whether the generated program from "Pro- gram_ "een__Geeratrr is valid and error-free. It checks for syntax errors, logical errors, and other potential issues that may arise during program execution.
Program__xecutor: This module executes the generated program from "Program_GGenrrtoor" produces an output that can be further processed by other modules, such as "Question Latis__rnnsn""" Now_Lookupp This module returns the simplified table that only remains the rows that are relevant to the question. It takes in the question and a table and returns the simplified table. If all rows are relevant there are only three using "Row Loo_Loo when the table involves more than three rows and the question only requires small number of rows to answer the question.
Executor:
Generator" and
fewer, return the original table. Normally, we only consider
Column Lookup: This module returns the simplified table that only remains the columns that are relevant to the question. It takes in the question and a table and returns the simplified table. If all columns are relevant or there only two columns, return the original table. Normally, we consider using "Column "ookup" when the table involves more than two columns and the question only requires small number of columns to answer the question.
Table Verbalizer: This module converts the table to a description that can be easily under- stood by the downstream modules, like tion Answering". Normally, we consider using "Table Verbalizer" when the table involves a small number of rows and columns and the table is domain-specific, such as steam-and-leaf plots, function tables, etc.
Generator". Sooutiio_GGenrrrn
"Ques-
Knowledge Retrieval: This module retrieves domain-specific knowledge for the given question and table. Normally, we consider using "Knowledge Retrieval" involve domain-specific knowledge, such as "steam-and-leaf plots", "function tables". "tax forms". etc.
Retrieevg when the question and table
Solution Generator:
This module generates detailed solution the question based on the information provided. Normally, we use "Solution "Gririttor"eer"""" when the question and table involve simple computation, such as arithmetic operations over single number.
Answer Generator: This module extracts the final answer in a short form from the solution or execution result. This module normally follows the " or "Problem Executor" module.
Below are some examples that map the problem to the modules.
In-context example(s)
Table:
designer watch $8,141 designer coat $6,391
Question: How much more does designer watch cost than a designer coat? (unit: $)
Modules:
"Frogrrrrn "Frogran_Veriifie" "Progrrrai "Program_Fxecutor
Table 9:
The prompt constructed for the planner model the TabMWP task. Similarly, the prompt consists of the instruction, the in in-context examples, and the test example.
19

---

b Instruction
Read the following question, and generate the background knowledge as the context information that could be helpful for answering the question.
> In-context example(s)
Question: Which property do these three objects have in common?
Options: (A) hard (B) soft (C) yellow
Metadata: 'pid': 43, 'has_in sossimmaee:: True, 'grade": 4, 'subject': 'natural science', 'topic": 'physics', "category': 'Materials', 'skill": "Compare properties of objects'
Detected text in the image: ('handkerchief", "slippers', 'leisure suit']
Knowledge:
This question is about comparing the properties of three objects: handkerchief, slippers, and a leisure suit.
The objects are related to the topic of physics and the skill of comparing properties of objects. Properties of objects can include physical characteristics such as color, texture, shape, size, weight, and material.
Table 10: The prompt constructed for the "Knowledge Retrieval" module the ScienceQA task.
b Instruction
Read the following question and metadata, and generate the query for browser search as the context information that could be helpful for answering the question.
> In-context example(s)
Question: Which property do these two objects have common?
Options: (A) hard (B) bendable
Metadata: 'pid': 329, 'has 'has_image image': True, 'grade': 2, 'subject': 'natural science', 'topic": 'physics', 'category": 'Materials', 'skill': 'Compare properties of objects"
Detected text in the image: [[[41, 183], [131, 183], [131, 199], [41, 199]], 'rubber gloves"). ([[245, 183], [313, 183], [313, 197], [245, 197]], 'rain boots')]
Search Query: Common material properties of jump rope and rubber gloves
Table 11: The prompt constructed for the "Query Generator" module on the ScienceQA task.
20

---

B Instruction
Given the question (and the context), select the answer from the options ["A", "B", "C", "D", "E"]. You should give concise and step-by-step solutions. Finally, conclude the answer in the format of "the answer [ANSWER]]" where [ANSWER] is one from the options ["A". "B", "C", "D", "E"]. For example, "the answer is A", "the answer is B", "the answer is C", "the answer is D", or "the answer is E". If the answer not in the options, select the most possible option.
> In-context example(s)
Question: Which property do these two objects have in common?
Context: Select the better answer.
Options: (A) hard (B) bendable
Metadata: 'pid': 6493, 'has image': True, 'grade': 2, 'subject': 'natural science', 'topic": 'physics', "category": 'Materials', 'skill': 'Compare properties of objects"
Image caption: A pair of scissors next to a pair of scissors.
Detected text with coordinates in the image: [[[[53, 185]. [121, 185]. [121, 199], [53, 199]]. 'jump rope'), ([[233, 183], [323, 183], [323, 201], [233, 201J). 'rubber gloves')]
Retrieved knowledge:
- This question about comparing the properties of two objects: rubber gloves and rain boots. The objects are related the topic of physics and the skill of comparing properties of objects. Properties of objects can include physical characteristics such as color, texture, shape, size, weight, and material. In this case, the objects have the property of being bendable in common.
Bing search response: The most common materials used for disposable gloves are Latex, Vinyl and Nitrile. Each material has its benefits and drawbacks. Latex Gloves are constructed from Natural Rubber Latex and are the most popular type of disposable glove.
An object has different properties. property of object can tell you how looks, feels, tastes, smells. Different objects can have the same properties. You can use these properties to put objects into groups Look at each object. For each object, decide it has that property. A bendable object can be bent without breaking. Both objects are bendable. A hard object keeps its shape when you squeeze it. The rubber gloves are not hard. The property that both objects have common is bendable. Therefore, the answer is B.
Solution:
Table 12: The prompt constructed for the "Solution Generator" module on the ScienceQA task.
> Instruction
Read the following table and question, and generate the domain-specific knowledge as the context information that could be helpful for answering the question.
b In-context example(s)
Table: x
10
1119
12
Question: The table shows a function. Is the function linear or nonlinear?
Knowledge:
A linear function is function whose graph is straight line.
nonlinear function is a function whose graph is not straight line.
The equation of linear function is y
+ b, where m is the slope and b is the y-intercept. The equation a nonlinear function is not y = mrr + b.
Table 13: The prompt constructed for the "Knowledge Retrieval" module on the TabMWP task.
21

---

> Instruction
Read the following question and table. Each is separated by newline ('In") and each column is separated by a vertical bar ('I"). Return the simplified table that only remains the rows that are relevant the question. If all rows are relevant, the number of rows is fewer than three, return the original table.
In-context example(s)
Question: In preparation for graduation. some teachers and students volunteered for the various graduation committees. How many people are on the music committee?
Table:
Committee Students Teachers
Program 5 17
Ticket 20 5
Music 201 15
Schedule 15 20
Food 18 2
Simplified Table:
Committee Students Teachers Music 201 15
Table 14: The prompt constructed for the "Row Lookup" module on the TabMWP task.
Instruction
Read the following question and table. Each row is separated by newline ('In") and each column is separated by a vertical bar ('I'). Return the simplified table that only remains the columns that are relevant to the question. If all columns are relevant, return the original table.
In-context example(s)
Question: Look at the following schedule. When does Recess end?
Table:
Subject Begin End
Recess 6:15 A.M. 7:20 A.M.
Orchestra 7:30 A.M. 88:40 A.M.
Art 8:45 A.M. | 99:5 A.M.
Handwriting 9:45 A.M. 10:20 A.M.
Gym 10:30 A.M. 11:15 A.M.
Choir 11:20 A.M. 12:25 P.M.
Science 12:35 P.M. 1:35 P.M.
Reading 1:40 P.M. 12:00 P.M.
Simplified Table:
Subject End
Recess 7:20 A.M.
Orchestra 8:40 A.M.
Art 9:35 A.M.
Handwriting 10:20 A.M.
Gym 11:15 A.M.
Choir 12:25 P.M.
Science 1:35 P.M.
Reading 2:50 P.M.
Table 15: The prompt constructed for the "Column Lookup" module on the TabMWP task.
22

---

Instruction
Read the following question and table. Write a textual description of the table. The description should keep the critical information in the table for answering the question. The description should not answer the question.
> In-context example(s)
Table:
Committee Students Teachers
Program 5 17
Ticket 20 5
Music 20 15
Schedule 15 20
Food
Table description: The table shows the number students and teachers on each of the four graduation committees: Program, Ticket, Music, and Schedule. The Music committee has 20 students and teachers.
Table 16: The prompt constructed for the "Table Verbalizer" module on the TabMWP task.
Instruction
Read the following table and then write Python code to answer question.
In-context example(s)
Table:
Price Quantity demanded Quantity supplied
$895 21,000 3,400
$945 17,200 7,400
$995 13,400 11,400
$1.045 9,600 15,400
$1,095 5,800 19,400
Questions: Look at the table. Then answer the question. At a price of $995, is there a shortage or surplus? Please select from the following options: ['shortage', 'surplus"].
Code:
# Python Codc, return ''as'.. Make sure that Yans' is a string selected from the opticns in the question
13400
trantity__upplliee_tt.cc quantity_demann
11400
berpstt__demmmddd_____rriec 955 > guantity_supplied__ letpeee_____ie______1 955: ans shortage"
alse:
surplus'
Table 17: The prompt constructed for the "Program Generator" module the TabMWP task.
23

---

Instruction
Read the following table and then answer a question.
> In-context example(s)
Table:
Price Quantity demanded Quantity supplied
$895 21,000 3,400
$945 17,200 7,400
$995 13,400 11,400
$1.045 9,600 11,400
$1,095 5,800 19,400
Question: Look at the table. Then answer the question. a price of $995, is there a shortage or a surplus? Please select from the following options: ['shortage', 'surplus"].
Solution: At the price of $995, the quantity demanded is greater than the quantity supplied. There is not enough of the good or service for sale at that price. So, there a shortage. The answer is shortage.
Table 18: The prompt constructed for the "Solution Generator" module on the TabMWP task.
START
0.11
query generator
0.3
1.0
0.06
0.53
bing_search
0.01
0.01
text, detector
0.02
image_ captioner
0.78
0.97
0.79
0.22
knowledge retrieval
0.21
1.0
solution ggnneratorrrrrr
answer generator
1.0
END
Figure 7: Transitions between modules in programs generated by Chameleon (GPT-4)
ScienceQA. START is the start symbol, END a terminal symbol and the others are non-terminal symbols.
24

---

START
0.26
0.05
row lookup
0.19
0.03
0.02
0.11
column lookup
0.24
0.26
0.02
0.27
0.39
0.25
knowledge retrieval
0.2
0.36
0.02
0,74
solution generator
0.63
table verbalizer
1.0
program program_generator
program verifier
1.0
program
generator
1.0
END
Figure 8: Transitions between modules in programs generated by Chameleon (GPT-4) TabMWP. START is the start symbol, END is a terminal symbol and the others are non-terminal symbols.
25

---

Image Captioner
A polar standing snow.
Query Generator
Animal adaptations cold environments
(Respanse Birds feathors, and most mammals hair for, layer the environment. Marine memmals whales bluther thick layer as heaky-duty
transfer
insutation.
weather, birds fuff their feethers and animels raice their to thicken the insu layer.
(Kesponse Adtatations:: Hove a thick fur coat for insu againet the cold. In winter, the coat is while provide camouflage against the ico and snow. In summer when the snow has mehod. they heve thin, dark
Bing Search
brown coal which provides
camouuagg againel rock and vegetalion.
(Response The polar bear adhapted to life cold climaaee Polar bears are adapted survival the Arcti Their adaptations indude; pray snow...
Which animal's sktn adapted
for survival in cold plecss?
appeerance camoullage
(A) Eurasian tynw
Thorry Devil
The context mentions that polar bears live in and around the cold Arctic Ocean, and their skin is adapted help them survive in cold places.
given octions, neither Eurasian lynx contoxt. Hovewer, mantianed with skin adapted bear memtooad
thomy
Solution Generator
survival places. Since the polar option choices. Eurasian lynx colder ragions compared to the thomy devil, which is desert-dwelling repile, Therefore, the answer is
given
Answer Generalor
Eurastan
Figure 9: One more example from our Chameleon (GPT-4) approach on ScienceQA.
This table shows Bronson's pay stub for the pay period March 16-31 Total eamingss represent the amount of money Bronson made bifore tools Federal incomee tax. state incomee tax. and other tases the amou nts deducted from Branson's is total camings for various tawes. Total taxes be calculated by adding the federal income tax, state income tax, and other taxee, Pay after eaxes the amount money Bronson on made after all taxes have been deducted from his total earnings. This can be calculated by subtractnng the total laxes from the lotal carnings.
Knowledge Rerieval
Employee
Pay period
Bronson Lo
March 16-31
Total sarnings
$1,200.00
The table shows Branson Le's pay stub for the period of March 16-31. includes his lotal eamings $1 11,200.0 00, federal incomee lex of 44, state income tox of 362. 10, and other tieess $o1. .0. The total taxes and after tases not provided but can be calculated by adding the tax amounts and subtracting them from the total earnings, respecively.
Federal inzore tax.
$130.44
Table Verbaiizer
State tax $62.10
Other taxes
Pay after taxes
Program Generator
total earnings 1200
feceral incone tax 113. .4
Look sl Bronson's stub. Bronson lives n state that has state income tax. How much
state incone tox 62. 18
othern takes - 01.00
after taxes?
Program Voriter
total taves federal, Incose, tax + stato_incore tax
other: tawes
ans total lerralleemningss total taxes
Program Executor
ans 913.45
Figure 10: One more example from Chameleon (GPT-4) approach TabMWP.
26

---

A Query Which better describes the Death Valley ecosystem?
Question:
Figure: Death Valley. Death Valley is a desert ecosystem in eastern California.
Context:
Options: (A) It has dry, thin soil. It also has many different types of organisms. (B) It has a small amount of rain. It also has only few types of organisms.
Metadata: 'pid': '18973', mhas_image': True, 'grade': 3, 'subject': 'natural science', 'topic': 'biology', 'category': 'Ecosystems', 'skill': 'Describe ecosystems*
> Response
Generated Program: "eQQuey__eneereeee tor",
Seanc_"earch", "Solution_Oenta
Query Generator:
Characteristics of Death Valley ecosystem
Bing Search: (the first one from response candidates) Visitors come to Death Valley to experience the stark and lonely vastness of the valley, the panorama of rugged canyons and mountains: the pleasures of the dry. moderate winter climate, the challenge of the hot, arid summer; the relief of the cooler mountains; and the reminders of frontier and Native American ways of life.
Solution Generator:
Death Valley is desert ecosystem, which means it has specific characteristics related to its climate and the organisms that live there. desert ecosystem typically has dry, thin soil and a small amount of rain. This enviromment supports only few types of organisms that are adapted to survive such harsh conditions. Therefore, answer is
Answer Generator (Wrong prediction):
(B) It has small amount of rain. It also has only few types of organisms.
Correct Answer (Ground truth):
(A) It has dry, thin soil. It also has many different types organisms.
Table 19: An example of failure cases in ScienceQA from Chameleon (GPT-4) where some modules perform inaccurately. The generated query from the "Query Generator" module is insufficiently specific, leading to a response from "Bing Search" that does not adequately detail the characteristics of the Death Valley ecosystem, particularly with regard to the types of organisms present. (From U.S. Geological Survey: Contrary to its name, Death Valley National Park very much alive Despite the low precipitation and extreme summer temperatures, Death Valley is home to many diverse species, several of which are endemic (found nowhere else in the world).)
27

---

b Query
Question: Which property matches this object?
Context: Select the better answer.
Options: (A) stretchy (B) yellow
Metadata: 'pid': '10338", 'has baa_image": True, 'grade': 'subject': 'natural science', 'topic': 'physics", 'category": 'Materials', 'skill': 'Identify proper- ties of object'
hellum balloons
Response
Generated Program: ["Image_Captionee "Solut.ionnGeeeratoo "Answer_c enerator"]
Image Captioner:
A colorful kite is being flown in the sky.
Solution Generator:
Look at the image of the object, which is colorful kite. A stretchy object can be stretched without breaking. Kites are typically made of materials that are not stretchy, such as nylon or polyester. A yellow object has yellow color. The kite in the image is colorful and includes the color yellow. Therefore, the answer is B.
Answer Generator (Wrong prediction): (B) yellow
Correct Answer (Ground truth): (A) stretchy
Table 20: An example of failure cases ScienceQA from Chameleon (GPT-4) where the generated programs are suboptimal. The LLM planner does not call the "Text Detector" and "Knowledge Retrieval" tools, resulting in an absence of semantic information from the image and a lack of related domain knowledge. Additionally, the employed image captioning model in providing an accurate textual description the given visual context.
28

---

Query
Compare the average kinetic energies of the partieles in each sample. Which sample has the higher temperature?
Question:
Context: Below is food web from an ocean ecosystem in Monterey Bay, off the coast of California. A food web models how the matter eaten by organisms moves through an ecosystem. The arrows in a food web represent how matter moves between organisms in an ecosystem.
Options: (A) bat star (B) phytoplankton (C) kelp bass
Metadata: ('pid': '15175', 'has sass_mmmme:: True, 'grade": 6, 'subject': 'natural science', 'topic": 'biology', 'category": 'Ecological interactions', 'skill'; 'Interpret food webs II')
Response
Generated Program:
" Retrieval", "Solution
Generator", "Answer answer_Generator!
Text Detector:
[([297, 91], 'orea'), ([653, 85), 'sea otter'),
([487, 191]. 'kelp bass' ), ([73, 221], 'phytoplankton',.
([625, 269], 'black rockfish'), ([323, 327], 'zooplankton',
[149,,63],, 'plainfin'), ([593, 361], 'kelp'), ([132, 380], 'midshipman', ([401, 391]. 'bat star"),
([271, 433],
'sea cucumber'), ) ([409, 471], 'sea urchin')
Knowledge Retrieval:
- TThe question is about food web in an ocean ecosystem in Monterey Bay, California.
food web models how matter eaten by organisms moves through ecosystem.
The arrows in a food web represent how matter moves between organisms in an ecosystem.
The organisms mentioned in the question are bat star, phytoplankton, and kelp bass.
The target organism is the sea cucumber.
In food web, matter moves from one organism to another through consumption, predation, decomposition.
Understanding the relationships between organisms in food web can help determine the flow of matter between them.
Solution Generator:
Analyze the food web determine which organism's matter eventually moves the sea cucumber.
The sea cucumber consumes the bat star.
The bat star consumes the sea urchin.
The sea urchin consumes the kelp.
The kelp gets its nutrients from the water, which contains matter from phytoplankton and other sources.
In this food web, the matter from the bat star eventually moves to the sea cucumber through consumption. Therefore, the answer is A.
Answer Generator (Wrong prediction): (A) bat
Correct Answer (Ground truth): (B) phytoplankton
Table 21: An example of failure cases in ScienceQA from Chameleon (GPT-4) where the current module inventory falls short in addressing specific abilities. this instance, the module inventory does not possess the necessary tools capable of parsing the arrows in the food web illustration.
29

---

P Query
Table:
19
18
11
10
11
Table: (semi-structurrd format)
x
y
4
15
19
18
11 10
11
Question: Look at this table. Is this relation function?
Options: (A) yes (B) no
Response
Generated Program: ["Knowledge www.lledee_eeeeeeaaeeeeee am_Verifier",
Generator", "Progr
Snswer__eneraten
Knowledge Retrieval:
relation is of ordered pairs (x, y).
A function is relation in which each input (z-valuee is paired with exactly one output (y-value).
To determine if relation is function, check if each valuue is paired with only one y-value.
- If an z-value is paired with more than one y-value, the relation is not function.
Program Generator:
relation
15,
19: 18,
11: 10,
11: 1
x_values 1ist (relation, keys ()) pniiue_x_values v_values set (X_values)
len (x values) 1en (unique x values):
ans = 'yes"
else:
n
Program Verifier: True
Program Executor:
ans
'yes'
Answer Generator (Wrong prediction): (A) yes
Correct Answer (Ground truth): (B)
Table 22: An example of failure cases in TabMWP from Chameleon (GPT-4) where some modules perform inaccurately. In this case, the LLM-base program generator is unsuccessful creating program that correctly mirrors the mathematical relationship expressed in the provided query and the retrieved domain knowledge.
30

---

D Query
This Oliver's complete financial record for September. How much money did Oliver have after he purchased the basketball?
Question:
Table:
Date
Description
Received
Expenses
Available Funds
Balance: end August
$154.90
notebook
$6.95
$147.95
9/15
walking dogs
$15.00
$162.95
9/21
basketball
$11.35
$151.60
Table:
(semi-structured format)
Date Description Received Expenses Available Funds Balance: end of August $$54.90
9/1 notebook
$6.95 $147.95
9/15
walking dogs $15.00 $162.95
9/21
basketball $11.35 $151.60
Response
Generated Program: ["Row_Looee "ookw__o "Solution_Generaai Generator", "Answer_Genen Genera
tor"]
Row Lookup:
Date Description Received Expenses Available Funds
9/21
basketball $11.35 $151.60
Solution Generator:
(Step 1) Subtract the cost the basketball from the available funds before the purchase.
(Step 2) $151.60 $$1$11.. 1.35 $140.25
(Step 3) Oliver had $140.25 after he purchased the basketball.
(Step 4) The answer is $140.25.
Answer Generator (Wrong prediction): 140.25
Correct Answer (Ground truth): 151.60
Table 23: An example of failure cases in TabMWP from Chameleon (GPT-4) where the generated programs are suboptimal. In the given example, the simplified table produced by the "Row Lookup" lacks the structural relations present in the original table's s context. These relations are crucial for the L.LM-based solution generator to accurately comprehend the domain-specific table.
31

---

Query
Table:
Tour bus schedule
historic house
10:15
10:30 A.M.
A.:45
11:00 A.M.
downtown
A.:
A.:15
A.:45A.M. A1:30A.M.
skyscraper
11:30
A.:4
12:00 P.M.
12:15 P.M.
old building
12:30
12:45
P.M.
governor's mansion
P.M.
Table Title:
Tour bus schedule
Table: (semi-structured format)
historic house 10:15 A.M. 10:30 A.M.
10:45 A.M. 11:00 A.M.
downtown 11:00 A.M.
11:15 A.M.
1::3 A.M.
11:45 A.M.
skyscraper 11:30 A.M.
11:45 A.M. 12:00 P.M. 12:15 P.M.
old building 12:30 P.M.
12:45 P.M. 1:00 P.M. 1:15 P.M.
governor's s mansion 1:00 P.M. 1:15 P.M. 1:30 P.M. 1:45 P.M.
This is Oliver's complete financial record for September. How much money did Oliver have after he purchased the basketball?
Question:
Options: (A) 2 hours and 15 minutes (B) 2 hours and 45 minutes (C) 15 minutes (D) 30 minutes
Response
Generated Program: ""Ro_LLokuup"" Sellution_Generaten "Answer Genera
"Answer_Genera
tor"]
Row Lookup:
historic house 10:15 A.M. 10:30 A.M. 10:45 A.M. 11:00 A.M.
governor's mansion 1:00 P.M. 1:15 P.M. 1:30 P.M. 1:45 P.M.
Solution Generator:
(Step 1) Find the last departure time from the historic house and the first arrival time at the governor's mansion:
Historic house last departure: 11:00 A.M.
Governor's mansion first arrival: 1:00 P.M.
(Step 2) Subtract the departure time from the arrival time:
1:00 P.M. 1:00 A.M. hours (Step 3) It takes 2 hours to get from the historic house to the governor's mansion.
(Step 4) The answer is hours and 15 minutes.
Answer Generator (Wrong prediction): (A) 2 hours and 15 minutes
Correct Answer (Ground truth): (B) hours and 45 minutes
Table 24: An example of failure cases in TabMWP from Chameleon (GPT-4) where some modules perform inaccurately and the generated programs are suboptimal. this instance, the LLMdbbsed solution generator struggles understand the bus schedule, which incorporates domain-specific knowledge. Furthermore, the LLM planner does not utilize tools like "Table Verbalizer" and "Column Lookup", which could enhance the LLM'' comprehension of the tabular context.
32


Reviews:
[{"review_id": "5JacCPr00Y", "reviewer": "Reviewer_yhWh", "summary": "The paper introduces Chameleon, a novel AI system that enhances large language models (LLMs) with plug-and-play modules for compositional reasoning. The authors argue that while LLMs have achieved remarkable progress in solving various natural language processing tasks, they have inherent limitations in accessing up-to-date information, using external tools, and performing precise mathematical and logical reasoning. To address these drawbacks, Chameleon synthesizes programs by composing various tools, such as LLMs, off-the-shelf vision models, web search engines, Python functions, and heuristic-based modules, for accomplishing complex reasoning tasks.\n\nAt the heart of Chameleon is an LLM-based planner that assembles a sequence of tools to execute to generate the final response. The authors showcase the effectiveness of Chameleon on two multi-modal knowledge-intensive reasoning tasks: ScienceQA and TabMWP. ScienceQA requires the system to answer science-related questions by integrating information from text, tables, and images, while TabMWP involves generating natural language descriptions of tables based on input queries. Chameleon, powered by GPT-4, achieves an 86.54% overall accuracy on ScienceQA, improving the best published few-shot result by 11.37%.\n\nThe authors also provide a detailed analysis of Chameleon's performance on various sub-tasks and demonstrate its ability to handle complex queries that require multiple tools and intermediate steps. They further discuss the potential of Chameleon for addressing real-world queries across various domains and improving the state of the art in multi-modal knowledge-intensive reasoning tasks. The paper concludes by highlighting the importance of enhancing current LLMs with the capability to automatically compose external tools for real-world task solving.", "questions": "I raised many questions in the weakness section, and the authors may want to resolve them and clarify some questions. The following is some minor comments:\n1. I don't quite get the connection between the name \"Chameleon\" and the proposed method. I also don't quite get the meaning of the colors in the name.\n2. I'd suggest reducing the table space, removing most of the unnecessary baselines. The only meaningful baselines probably are the STOA from previous works (mostly fine-tuned?) and GPT-4/ChatGPT. \n3. Considering some improvements on top of ChatGPT and GPT-4 are kind of marginal, I'd encourage the authors to conduct a thorough error analysis to figure out how the base ChatGPT/GPT-4 make mistakes and how the proposed method reduces such errors. ", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper probably is one of the first works in the field that explore automatic task planning and execution with GPT-4 as the agent. It further enhances the abilities of GPT-4 by leveraging various tools controlled by a GPT-4 based planner. Such a combination is straightforward but necessary for LLM-based agents.\n2. One merit I like about this paper is that it has clear evaluation metrics, not like other agent-based models that merely have a runnable model and use cases. The standard evaluation clearly helps us understand the position of this paper and the performance of the proposed method.\n3. The improvement on top of GPT-4/ChatgPT is significant and clearly shows the effectiveness of the proposed method. As we can see from Tables 3 and 4, the proposed method can mostly improve their direct baselines, i.e., ChatGPT and GPT-4. Also, qualitative analyses are helpful in understanding the model behavior.", "weaknesses": "1. Like the other recent works for LLM agents, the novelty of such kind of works is somehow ad-hoc and limited. There are many tool-augmented works, as demonstrated in Table 1, and their differences are petty nuanced, and it's hard to tell the real novelty behind them. Usually, each such kind of work would leverage the most recent LLMs, like GPT-4, to do the planning or task decomposition and call special tools to solve each subtask. Most powers come from OpenAI's model, esp. GPT-4, which is hardly claimed to be the credit of these works (but it's good to see the proposed methods can further improve GPT-4). This leaves us with an essential question, what are the core contributions of such works? Demonstrating GPT-4 can be augmented by external tools definitely is nontrivial but also engineering-heavy. For NeurIPs works, I may like to see more research values; for example, which designs of the proposed method really distinguish itself from other similar works? Is it the \"generating a program in a natural-language-like format\"? Or some special prompt format to trigger the planning and task execution abilities? etc etc.\n2. Although the paper names its method Chameleon, trying to highlight the adaptability and versatility of LLMs, ironically, the task scope and experiment settings in this paper are limited. The planning pipeline and proposed tools are specially designed for solving the two datasets. While conceptually, the idea of augmenting GPT-4 with some tools can definitely be applied to more general tasks, this generality comes from this simple idea but not from the proposed method. I hope the authors could dedicate more effort to make the proposed method easier to be applied to more tasks.\n3. Another weakness of the proposed method probably is the simple planning design, which is basically a straightforward CoT with a linear structure. As we can see from Table 3, there is still a large gap between fine-tuned STOA and the proposed method. One possible reason might be the weak planning ability of the proposed method. This is somehow related to the first weakness, meaning that most LLM agent works combine GPT-4 with tools in a very simple way. I hope the authors could explore more advanced planning algorithms to see if the performance can be further improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, \n\nThank you for your insightful comments and constructive suggestions. We're grateful for your acknowledgment of our work is among the first works in the field and the distinctiveness by clear evaluation metrics. Your insights on our significant improvements and qualitative analysis are invaluable.\n\nWe eagerly anticipate **further engagement** with you to strengthen our paper. Your thoughtful comments have been addressed as follows:\n\n### W1: Novelty of the work of LLM agents\n\nWe acknowledge the concern regarding the novelty in the field of LLM agents. It's important to note that many papers in Table 1 are **unpublished concurrent works**, underscoring this frontier research area. However, despite this surge in similar research, our work carves out a distinct niche due to distinct contributions:\n\n1. **Generating natural-language-like programs**: Unlike other LLM-agent works, our system produces programs in a human-readable, natural-language format. This design mitigates errors and is more intuitive for domain experts, fostering human-machine collaboration.\n\n2. **Versatility in API integration**: Our approach is distinctive to support a range of API types, allowing it to tackle various tasks. It sequentially executes these APIs, updating inputs and cache, without introducing complexities.\n\n3. **Robust evaluation metrics**: We move beyond just presenting a system with “runnable examples”. Our work undergoes rigorous evaluation against real-world challenges, showing its superior performance and practical applicability.\n\n4. **Addressing inherent LLM limitations**: Our efforts extend the LLMs’ capabilities, enabling them to navigate issues like outdated information and limitations in reasoning with non-textual data.\n\n5. **Unpublished concurrent works**: Many papers in Table 1 represent published concurrent works, underscoring this frontier research area.\n\n*For more information, please check our clarification in the general response.*\n\n### W2: Task scope and experiment settings\n\nThank you for your constructive feedback on the scope and adaptability of Chameleon.\n\n1. **Experiment settings**: The main contribution of this paper is the introduction of a framework that allows domain experts to seamlessly integrate LLMs with existing tools to enhance task performance. Therefore, under this assumption, we have validated the idea using two complex tasks. The same framework can be extended to other tasks, and additional tools can also be integrated into it.\n\n2. **Chameleon's versatility**: The name “Chameleon” was chosen to highlight our method's adaptability. While the current implementation focuses on specific datasets, the design is meant to be applicable to a broader array of tasks. We've recorded notable performance gains in tasks not covered in the paper, such as an 11.85% gain on FinQA and an 11.39% gain on NER, both of which are benchmarks in financial reasoning.\n\n3. **Future directions**: We value your suggestion. Our primary focus in this paper was to introduce the novel method and demonstrate its efficacy. To foster community adoption and further exploration, we will open-source the code (submitted in the supplementary), allowing other developers to utilize our framework in diverse scenarios and tasks. \n\n*For more information, please check our clarification in the general response.*\n\n### W3: Proposed planning design and other algorithms\n\nWe are grateful for your constructive feedback and appreciate the chance to clarify our design.\n\n1. **Why the current design**: Our method was designed to be efficient and straightforward, requiring only a single LLM API call per query, avoiding feedback loops, and simplifying task performance optimization. We believe that these design decisions contribute to Chameleon's effectiveness and robustness.\n\n2. **Comparison with existing methods**: It's worth emphasizing that Chameleon's performance is indeed commendable. It surpasses all baselines and concurrent work, including some state-of-the-art methods, which indicates that the straightforward planning design does not limit its capabilities.\n\n3. **Potential for advanced planning algorithms**: We acknowledge your insight into the potential benefits of more complex planning algorithms. Being among the first works in the field of automatic task planning and execution with GPT-4 as the agent, we focus on the design of a general framework that allows domain experts to easily incorporate various tools. Therefore, for the sake of simplicity, we applied a straightforward planning algorithm. As the framework is general, other complex planning algorithms can be incorporated in the future.\n\n4. **Acknowledgment of the field's challenges**: We recognize that combining GPT-4 with tools in a simple way is a common challenge in the field. Our work aims to provide a strong foundation upon which more complex strategies can be built, and your suggestions will guide our future research directions.\n\n### Q1: Connection between the name “Chameleon” and the method\n\nOur model symbolizes its adaptability to diverse tasks and queries, adjusting its strategies much like a chameleon changes its colors. The name “Chameleon” directly relates to the title, “[Com]positional [Re]asoning with Large [Lan]guage Models”. Different colors within the name represent our model's ability to transition between various queries, aligning with the colored examples shown in Figure 1.\n\n### Q2: Adjusting the experimental table spaces\n\nYour suggestion is highly appreciated. We will remove less performant and fine-tuned works.\n\n### Q3: Error analysis of ChatGPT/GPT-4 and Chameleon\n\nThank you for your valuable feedback. **We've added your suggested error analysis to the new PDF in the rebuttal period**.\n\nChameleon's tools for image captioning and knowledge retrieval reduce mistakes in those categories. Sequential tool execution further lessens errors in solution generation. Additionally, GPT-4's task planning significantly outperforms ChatGPT."}, {"review_id": "SmLtBhw8xf", "reviewer": "Reviewer_1xpZ", "summary": "This work proposes Chameleon, an approach to connect large language models to different tools for compositional reasoning. Chameleon is powered by a GPT-4 planner that chains different tools to accomplish the described goal. Chameleon achieve improved performance in comparison with strong baseline models.  ", "questions": "* How does Chameleon select the module inventory? What is the rule of thumb?\n* How are `update_input` and `update_cache` are implemented? Any examples?\n* Prompt-based tools such as `Row_Lookup` are powered by the same model as the planner; is that correct? ", "limitations": "* Be more accurate in the improvement presentation ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The presentation of the framework is great. The paper has nice figures, clear definitions of problems, clear demonstrations of results.\n* The experiments are solid. This work evaluates Chameleon on two challenging benchmarks with distinct purposes\n* The study on how to make LLMs such as GPT-4 plan reasonably is useful for future research. ", "weaknesses": "* The improvement on the two benchmarks is not as significant as described in the abstract. The abstract indicates that \"on ScienceQA, improving the best published few-shot result by 11.37%\" where the best-published result is a GPT-3 based approach while Chameleon is powered by GPT-4. Therefore, the comparison is not completely fair. The improvement does not necessarily come from the benefit of Chameleon. Instead, It could be as simple as GPT-4 being more powerful (as many tools are prompt-based). In Table 3, Chameleon (GPT-4) only shows a 2.55% improvement over GPT-4 CoT, and 1.62% when comparing Chameleon (GPT-3.5) with GPT-3.5 CoT.\n* The paper does not explain *how* to apply Chameleon to other tasks. Even though Chameleon is claimed to be plug-and-play, there are many *task-specific* designs. For instance, the `Row_Lookup`, `Column_Lookup` , `Table_Verbalizer` tools are highly specialized for table-related questions, and they are very fine-grained. How does a practitioner know whether to have both `Row_Lookup` and `Column_Lookup`, or a simple `Table_Lookup` tool? What are the insights on tool selections? Do we need a dev set to \"fine-tune\" the tool pool? These details need to be further explained to make Chameleon really plug-and-play.\n* Another concern is that converting the output of one tool (module) to fit the input format of another tool is pre-defined (L153-L157), indicating that it is potentially challenging to generalize Chameleon to *massive* number of tools due to the extensive human efforts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, \n\nThank you for your thoughtful analysis and encouraging words. We are pleased to know that you appreciated the great presentation of our framework. Your recognition of our solid experiments on two challenging benchmarks. Moreover, we are glad that you perceive our study on how to make LLMs plan reasonably as a useful contribution to future research.\n\nYour insights strengthen our understanding of our work's impact, and we look forward to **engaging further** with you to refine and improve our paper. Your valuable comments have been addressed as follows:\n\n### W1: More accurate in the improvement presentation\n\n1. **Improvement margins**: While the improvements of 2.55% and 1.62% in ScienceQA may appear modest, it's crucial to recognize that they were achieved over already highly optimized models like GPT-4 CoT (83.99%) and ChatGPT CoT (78.31%). Even minor gains can translate to significant improvements in complex tasks. Notably, for TabMWP, Chameleon improves by 9.97% and 11.25% over GPT-4 CoT and ChartGPT CoT.\n\n2. **Comparison with GPT-3**: The intention behind our comparison with GPT-3 was to showcase Chameleon's performance using the latest models available.\n\n3. **Clarifications including accurate presentation**: To address your concerns, we propose to: modify the wording for a more accurate performance representation; explain that comparisons are made with the latest models and justify their relevance; provide an in-depth discussion of the improvements; include experiments to compare Chameleon with GPT-3 to ensure a consistent assessment.\n\n### W2: Apply Chameleon to other tasks\n\n1. **Task-specific vs. plug-and-play**: We acknowledge the concern about the specialized nature of some tools. While these tools were indeed tailored for task-specific questions, Chameleon's plug-and-play design facilitates the straightforward addition of new modules that can be adapted to various tasks. To address this concern, we include a discussion illustrating its potential applicability to various domains.\n\n2. **Tool selection insight**: Our goal here is to provide a general framework that domain experts can easily select and incorporate relevant tools and modules to significantly boost the performance of the task. They understand which tools are useful, so they select a subset of the modules that align with the tasks. We comprehend the request for more insights into tool selection. To address this concern, we will enhance our paper by including guidelines and insights on tool selection, offering a clearer path for both specialized and generalized approaches.\n\n3. **Fine-tuning the tool pool**: The suggestion of having a development set to “fine-tune” the tool pool is valuable and aligns with our vision of making Chameleon a flexible and adaptable system. We appreciate this insight and will include discussions regarding this possibility in our revision.\n\n### W3: Converting the output format is pre-defined\n\nWe understand the concern about the potential difficulty of generalizing Chameleon to a massive number of tools. However, we would like to clarify that in Chameleon, **no special forms or extensive human effort are needed to convert the output of one tool to fit the input of another.**\n\nThe outputs of each tool in Chameleon are structured entirely in *natural language* and are stored with an associated index name and value. Subsequent tools can directly utilize this textual cache if they are prompt-based or refer to the related index for other tool types. This enables *easy integration and coordination* between various tools, even as the system scales up. New tools can be added by prompting the planner with their descriptions and demonstration examples, without extensive human efforts. \n\nThis thoughtful design not only distinguishes our approach from concurrent works of LLM agents but also ensures that the challenges in generalizing to a massive number of tools are mitigated.\n\n### Q1: How does Chameleon select the module inventory?\n\nThank you for this insightful question! Chameleon's module inventory is thoughtfully crafted to cover various reasoning abilities, such as image understanding, table comprehension, knowledge retrieval, web search, program generation, and solution generation.\n\nWhen adapting Chameleon for specific tasks, we assume that there are **domain experts** who want to apply LLMs to solve complex tasks, and they know certain tools are useful. So they select a subset of the modules that align with the tasks. This not only enhances planning accuracy but also optimizes computational efficiency. For example, in ScienceQA, tools related to program generation were excluded as unnecessary.\n\n### Q2: How are update_input and update_cache implemented?\n\nThey are described in L152-157, with codes detailed in the supplementary. We are pleased to elaborate further here. \n\n- `update_input`​​: ​​It is triggered by the execution of specific tools, like `Row_Lookup`, which alter or replace elements in the input to reflect the updated state.\n\n- `update_cache`: Tools such as `Image_Captioner`, `Text_Detector`, `Knowledge_Retrieval`, `Web_Search`, and `Program_Generation` generate new elements. `update_cache` stores these new elements in the cache, making them accessible for later tools' execution.\n\n### Q3: Are prompt-based tools powered by the same model as the planner?\n\nYes, in our current implementation, the planner and prompt-based tools utilize the same LLM base model for straightforward implementation and consistent comparison. \n\nHowever, Chameleon's flexibility allows them to employ different LLM base models. GPT-4 might serve as the planner for its superior planning, while ChatGPT could be employed in prompt-based tools, balancing performance and cost-effectiveness. Also, both the planner and tools can be upgraded to fine-tuned models or predefined tools for enhanced performance."}, {"review_id": "7yNWd0Zn8P", "reviewer": "Reviewer_GU39", "summary": "The paper presents Chameleon, an AI system that enhances large language models (LLMs) with plug-and-play modules for compositional reasoning. Chameleon uses various tools, including LLMs, vision models, web search engines, and Python functions, to perform complex reasoning tasks. The system, powered by GPT-4, significantly improves performance on two multi-modal reasoning tasks: ScienceQA and TabMWP, setting new state-of-the-art results. The authors also demonstrate that Chameleon's GPT-4-powered planner exhibits more consistent and rational tool selection compared to a ChatGPT-powered planner.", "questions": "Overall a good paper. No additional questions.", "limitations": "The authors have thoroughly demonstrated the limitations of the work in the appendix.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is overall well-written and easy to follow. Abundant figures and visualizations also help with reader understanding.\n2. The paper presents a novel system, Chameleon, that enhances large language models (LLMs) with plug-and-play modules for compositional reasoning. This innovative approach addresses the inherent limitations of LLMs, such as their inability to access up-to-date information, use external tools, or perform precise mathematical and logical reasoning.\n3. Chameleon achieves impressive performances (especially with GPT-4) on two QA datasets: ScienceQA and TabMWP, showing the effectiveness of the entire framework.\n4. Very detailed and thorough analysis of existing related works, which can largely help the community to explore the direction.", "weaknesses": "1. I might miss something. However, the novelty of the method seems to be limited. As a method focusing on compositional tool use, I am wondering what are the differences between Chameleon and HuggingGPT (https://arxiv.org/pdf/2303.17580.pdf), except for the application tasks and involved tools. As both works seem to leverage human-introduced ordering information between tools to compositionally use the tools.\n2. Both datasets seem to have strong dependencies between different tools. With a relatively small number of simple tools (compared with vision models and SQL interpreters, etc.), the compositional order among tools seems to be quite static and rigid. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, \n\nThank you for your comprehensive and positive remarks about our work. Your recognition of Chameleon's novel approach, addressing the inherent limitations of LLMs reaffirms the value of our work. Your commendation of the achievements emphasizes the effectiveness of our framework. Your appreciation for our detailed examination of existing works, and their potential to guide the community in exploring this direction, is sincerely valued.\n\nWe are inspired by your feedback and look forward to **engaging further** with you to refine our paper. Your thoughtful comments have been addressed as follows:\n\n### W1: Novelty of Chameleon; differences between Chameleon and HuggingGPT\n\nWe would like to highlight the **novelty** of Chameleon: (1) It is one of the first works in the field of LLM agents for task planning and execution, as recognized by Reviewer yhWh (R1); (2) It has the potential to be a general framework for compositional reasoning and is distinct from prior works in constructing a solution entirely within modules with natural-language-format program generation, as highlighted by Reviewer uov2 (R4). *For more information, please check out our clarification in the general response.*\n\nWe also appreciate the chance to clarify the **differences** between Chameleon and HuggingGPT. It's vital to note that *HuggingGPT is concurrent with our work and is not published; its latest version is published on 25, May, later than the NeurIPS deadline*, further emphasizing the independent innovation in Chameleon:\n\n1. **Tool composition mechanism**: Chameleon introduces a unique planner-driven approach that dynamically selects and composes tools based on the context of the given task. The planning algorithm accounts for not only the ordering of tools but also the specific context, input, and required reasoning. In contrast, HuggingGPT parses the user request into a task list and assigns appropriate models for each task.\n\n2. **Task flexibility**: Chameleon is designed to handle a broader array of tasks by employing various tools, such as LLMs, vision models, web search engines, and Python functions. This combination allows Chameleon to perform complex reasoning tasks, such as mathematical reasoning and science QA, beyond the visual understanding tasks that have been demonstrated in HuggingGPT.\n\n3. **Tool type**:  Chameleon is designed as a general framework, which supports Github models, Huggingface models, Python tools, web engines, and heuristic tools. It has a novel mechanism to update the inputs and cache during the sequential execution of different types of tools. In contrast, HuggingGPT only supports models in Hugging Face, with standard model configurations.\n\n4. **Plug-and-play nature**: Chameleon features a modular nature where each tool can be replaced and updated individually. The tool can be LLM-based, pre-trained neural networks, or software tools. A new tool is easily included in the module inventory by providing the tool description and demonstration examples to the planner prompt. This makes Chameleon more robust and flexible in handling unseen or complex scenarios.\n\n5. **Empirical validation**: Chameleon has been empirically validated on two challenging reasoning tasks, setting new state-of-the-art results. Instead, HuggingGPT implements runnable examples and focuses on case studies and qualitative results.\n\nWe believe these key differences underscore the novelty of Chameleon. We will enhance the related work section to clarify the distinctions between Chameleon and other methods.\n\n### W2: The compositional order among tools seems to be static\n\nWe appreciate your observation and would like to provide clarifications.\n\n1. **Tool interdependencies**: While it's true that there are dependencies between different tools, this design **reflects the complex nature of real-world reasoning tasks**. In practice, many reasoning processes require a series of interdependent steps, and our system is intended to mirror these real-world scenarios. As shown in Table 7 in the Appendix, there are 14 and 11 different compositions for Chameleon (ChatGPT) and Chameleon (GPT-4) in ScienceQA, and 28 and 19 different compositions respectively on TabMWP. The diverse dependencies between tools add richness to the reasoning process, allowing Chameleon to leverage various skills.\n\n2. **Compositional order**: We analyzed the compositional order among tools by visualizing the transition graphs between modules in programs generated by Chameleon (please see Figure 6 and Figure 7 in the Appendix). According to the transition graphs, **the compositional order is dynamic and extensible, rather than static and rigid**. The complex graphs reflect the diverse reasoning paths required for the tasks we address. Interestingly, some frequent paths (compositional orders) in graphs indicate that our planners can find consistent and rational patterns for solving specific types of problems. These paths provide insight into the reasoning strategies employed by the planners, further showcasing the system's ability to execute structured reasoning.\n\n3. **Scalability and flexibility**: Despite the relatively small number of tools in the presented work, Chameleon is designed to be scalable and can be extended with more complex tools, such as other vision models and SQL interpreters. The current selection of tools serves as a demonstration of the framework's capabilities, and we plan to explore more compositions in future works.\n\nBased on your feedback, we propose to include a detailed explanation in the paper. Thank you once again for your thoughtful comments. "}, {"review_id": "AkDivMFFzq", "reviewer": "Reviewer_uov2", "summary": "This work presents Chameleon, a system that uses a library of tools or modules to assist a LLM to answer questions. First, a LLM is prompted with a description of available modules, constraints on how the modules should be used, few-shot demonstration examples, and finally the question to answer. The LLM then generates a sequence of modules that should be used to process the question. The modules are used in sequence, keeping track of module outputs and changes to the input question. Finally, the last module outputs an answer to the original question. When Chameleon uses GPT-4 as the base LLM, it achieves +2.55% improvement over GPT-4 with Chain-of-Thought prompting on ScienceQA, and +1.85% improvement over GPT-4 with Program-of-Thought prompting on TabMWP.", "questions": "## Potential takeaways that could be explored in more detail\n\n* Potential takeaway 1: performing tool use entirely within the sequence of modules is better than interleaving tool use with LLM generation as in prior work, e.g., ToolFormer. This might be due to constraints in module ordering, heuristics in how modules update the context, using module-specific prompts instead of the original prompt which might be overloaded, or interleaved tool use “interrupting” the flow of text generation. I would love to see an analysis of these different factors using the same set of tools, for a more general understanding of the benefit of Chameleon’s “sequence of modules” approach.\n\n* Potential takeaway 2: the Chameleon framework is truly general-purpose, meaning the same system achieves high performance on a variety of datasets. This would represent an important step forward in question answering. But currently, we only see dataset-specific variations of Chameleon performing well on one dataset at a time, which is less impressive because the comparison is to GPT-4 with CoT or PoT which has no dataset-specific design or engineering (other than a few-shot prompt). How does Chameleon perform on each dataset with all modules available, a single setting of module ordering constraints, and a single default module sequence? (If the performance is then worse than GPT-4 CoT or PoT, then is Chameleon truly a generally applicable approach?)\n\n* Potential takeaway 3: if the goal is to achieve high accuracy on a single academic dataset, then one can find success using Chameleon with dataset-specific engineering and heuristics. This takeaway is already supported by the paper, but in my opinion, this is a weak result. There are no dataset-specific approaches in the comparison (at least no recent ones building upon ChatGPT or GPT-4). These are also *academic* datasets that are meant to serve as a proxy for measuring more general ability; raw performance on these datasets without generalization to other tasks is not as impressive. On the other hand, if Chameleon could get high performance on a more general and broad dataset, for example (hypothetically) a representative subset of queries to a search engine or AI assistant that are related to science/math/reasoning/etc., then it could be argued that Chameleon is general enough for practical applications, and then this takeaway would carry more weight.\n\nIn light of these, I'm not sure what readers are supposed to take away from the paper in its current state, and so I cannot recommend acceptance at this time. However I do think any of the potential takeaways above, if explored thoroughly, could lead to a much more impactful paper, so I encourage the authors to continue their investigations.\n\n## Writing that oversells the approach\n\n* The abstract mentions “improving the best published few-shot result by 11.37%” on ScienceQA and 17.0% on TabMWP, and these numbers are repeated throughout the text. While technically true, these numbers are misleading, simply because the performance of GPT-4 with CoT or PoT hasn’t been published before. Certainly the authors can’t claim any “ownership” of GPT-4 plus CoT or PoT, so they should be treated as prior work even if the authors are the first to measure their accuracy on these datasets. Considering this, it is more honest to say that Chameleon achieves **+2.55% on ScienceQA and +1.85% on TabMWP** over previous approaches without fine-tuning. (MM-COT Large is a fine-tuned model achieving +5.14% over Chameleon on ScienceQA, according to Table 3.) Lines 65-68 are carefully worded to avoid mentioning those bolded numbers and should be more transparent.\n\n* Throughout the text, Chameleon is said to produce “NL-like programs”. Is “*programs*” the most accurate term to use here? These are merely **sequences of modules**, without any other programming feature like loops, conditionals, variable binding, helper functions, and so on. I think the term “program” is overselling the complexity of the module sequence plans.\n\n* What does “plug-and-play” mean? This is a key term used in the paper’s title, but I actually think it hides complexity. To add a new module, one must: (1) implement the core module logic, (2) describe the module in the planner’s prompt, (3) describe constraints about when to use the module with respect to other modules in the planner’s prompt, (4) implement checks for those constraints so invalid plans can be rejected, (5) provide few-shot demonstrations of using the module in the planner’s prompt, (6) decide for which datasets the module should be available, (7) design the module’s update_input and update_cache functions. This seems like quite a bit of work -- an acceptable amount of work, but not quite “plug-and-play”. If easily adding new modules is a key feature of Chameleon, then at the very least, the authors should clearly describe all the steps involved in using a new module, but this is currently missing from the paper.\n\n* Why is MM-COT Large not included in Figure 3(a)? It would be misleading to omit the best approach from the “main baselines”.\n\n* Some modules are always used in a particular order. From Figure 6 and 7 from the appendix, the following modules are always used together: (Query Generator -> Bing Search), (Solution Generator -> Answer Generator), and (Program Generator -> Program Verifier -> Program Executor). Would it make sense to combine each group into a single module? Table 1 lists Chameleon as having >10 tools, but if these tools were combined, the count would be 9. Chameleon is described as using modules in a compositional way, but the number of tools is inflated compared to which compositions are actually used.\n\n## Minor questions and suggestions\n\n* Throughout the text: “**our** Chameleon” is awkward phrasing. It would be more natural to say “our approach Chameleon” or simply “Chameleon”\n* Line 97: correcting grammar - “has made tremendous progress and **has stimulated** research in **prompt** learning ...”\n* Line 125: extraneous space after Chameleon\n* Line 153: please provide examples of update_input and update_cache, since they are hand-designed for each module\n* Line 153: Is $M_t$ actually $M_i$? Or $M^t$?\n* Line 171 and so on: I didn’t notice any usage of the notation $M_{kr}$ elsewhere, is it necessary?\n* Line 233: missing quote before Knowledge Retrieval\n* Lines 246 - 248: this sentence is repeated\n* Line 251: “in the appendix” is repeated\n* Line 296: does the “solution-executor module” mean the “Solution Generator module”?\n\n---\n\nUpdated scores after the author-reviewer discussions:\n* Soundness: 3 -> 4\n* Contribution: 2 -> 3\n* Rating (\"overall score\"): 3 -> 4", "limitations": "The limitations (Appendix B) should mention that it takes more wallclock time and compute resources to process questions with modules, especially if those modules themselves contain LLM components. If the use of multiple LLM-based tools really proliferates, then this would be a broader societal impact (Appendix C) through an increase in the amount of energy required to process queries, leading to greater environmental impact.", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "I see 4 main strengths of this paper:\n1. The idea of performing tool use entirely within a sequence of modules (instead of interleaving tool use with prediction from the original LLM) is new and offers some specific advantages\n2. Achieving a small performance increase on two question-answering datasets\n3. The project being an example of how engineering effort can be organized to achieve high dataset-specific performance\n4. The paper writing is well organized and easy to read\n\nOriginality: The most original aspect of this work is that the solution is constructed entirely within the modules, whereas previous work on tool use (to my knowledge) interleaves modules with prediction from the original LLM. With the Chameleon approach, the necessary reasoning at each step can be more tightly controlled (e.g., with module-specific prompts and heuristics about what information to include or exclude for future modules), and constraints can be placed on the sequence of modules to use.\n\nQuality: The authors provide an extensive comparison to prior models on two datasets, along with an ablation study for the importance of various modules. The experiments show a slight improvement over previous approaches.\n\nClarity: The paper is well organized, and the supplemental material contains extensive additional details for the modules.\n\nSignificance: The research topic (how to best use LLMs with tools) is definitely significant.", "weaknesses": "Overall, this paper feels less scientific and more like a report on an engineering accomplishment in achieving high accuracy on the benchmark datasets, since the modules were created with the datasets in mind, a dataset-specific subset of modules is available at test time, each module includes hand-designed heuristics for how to update the input and cached information, there are hand-designed constraints on module ordering, and dataset-specific “default” module sequences are used if the predicted one does not meet the constraints. All of this points to my interpretation that Chameleon is a framework that can be instantiated, with *sufficient engineering effort*, into a system with high *dataset-specific* accuracy. However, there is little evidence that the Chameleon approach can produce a *single, general-purpose* question answering system with high accuracy on a variety of datasets.\n\nFrom my perspective, the main weakness is the lack of some general takeaway that can be applied to other scenarios. There are some avenues toward potential takeaways but currently they are not explored thoroughly enough to draw confident conclusions. In the Questions section below, I provide some potential takeaways that could be explored in more detail.\n\nAnother weakness of the paper is that the writing oversells the approach in various ways. Please see the Questions section below for specifics. I hope the authors edit the writing accordingly.\n\nOriginality: There is not much novelty other than the “sequence of modules” approach mentioned in the Strengths section. Although, if the “sequence of modules” approach is shown to be better than prior approaches through further experiments, then its novelty would be sufficient.\n\nQuality: It is unclear what the main takeaways of the paper should be, or how well-supported they are.\n\nClarity: The writing oversells the work in a few ways.\n\nSignificance: Some of the potential takeaways, if explored more thoroughly, could be quite significant. However, the actual conclusions one can draw from evidence in the paper are less significant right now.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, \n\nThank you so much for your thoughtful review. We are encouraged by your recognition of our paper's originality, extensive experiments, and significant research topic. We appreciate your concerns and your insightful comments to lead to a more impactful paper.\n\nWe are committed to addressing the concerns and **engaging further** with you to enhance our paper. Your valuable comments have been addressed as follows:\n\n### W1: Scientific contributions vs. engineering efforts\n\nThank you for bringing up your concerns. We'd like to emphasize the scientific contributions as follows:\n\n1. “Pioneering exploration in the field of LLM agents”: Our work represents one of the first efforts in automatic task planning and execution by the LLM agent. It addresses the inherent limitations of existing LLMs and enhances their capabilities. Many concurrent works in Table 1 are unpublished, underscoring the frontier nature of this research area.\n\n2. **Uniqueness in modularity**: The modular design allows for a systematic and transferable approach, where individual components can be reused or replaced to adapt to various tasks. \n\n3. **Blend of science and pragmatism**: While achieving high accuracy on benchmarks required engineering efforts, these efforts were grounded in the scientific exploration of enhancing LLMs for complex tasks. The general capabilities of Chameleon are maintained, and we believe the fine-tuning performed reflects an acceptable practice within ML to show how the system can be optimized. Comprehensive ablation studies and analysis further provide insights for future work.\n\n4. “Light engineering efforts”: The planner and LLM-based tools are prompted with descriptions and demonstrations. Thanks to the nature of the modular design and textual cache, modules can be designed, updated, or replaced individually. These modules, generated from natural-language programs and executed sequentially, ensure straightforward development.\n\n*For more information, please check out our clarification in the general response.*\n\n### W2: General vs. dataset-specific framework\n\n1. **General framework**: Our goal is to provide a general framework that allows domain experts to easily incorporate relevant tools, thereby significantly enhancing task performance. This generalization is focused on a modularized framework that facilitates the easy integration of new tools, rather than a single system applicable to numerous tasks.\n\n2. **Results on new tasks**: The principles guiding Chameleon's design enable it to function as a single, adaptable system capable of achieving high accuracy across various datasets. We've recorded notable performance gains in tasks not covered in the paper, including an 11.85% gain on FinQA and an 11.39% gain on NER, both benchmarks in financial reasoning. \n\n3. **Limitations in context length**: At the time of our research, the context length for LLMs like GPT-4 was limited to 8K tokens, necessitating the selection of a dataset-specific subset of modules. Recently, GPT-4 has been scaled to support 16K tokens, and Claude has a 100K token window. These technical advancements enhance Chameleon's ability to plan using a complete set of modules.\n\n### T1: Analysis of different factors in tool use\n\nConcurrent works, such as UCB-Gorilla and Google-LATM, also favor separating tool planning and execution from LLM generation, and your review insightfully touches on underlying reasons:\n\n1. Complex reasoning often requires a step-by-step process, and having constraints in module ordering **ensures that the logic unfolds effectively**.\n\n2. By allowing subsequent modules to benefit from updated inputs and cache, Chameleon can **create context-aware responses**.\n\n3. By generating the tool planning or calling separately, we can **avoid overloading the prompt** and maintain a more streamlined communication with the tools.\n\n4.  The interleaved approach might interrupt text generation, while Chameleon **preserves the coherence of the text**.\n\nWe appreciate your valuable suggestion and will incorporate an analysis in the revision.\n\n### T2: Chameleon is general-purpose\n\n**As noted in the global response**, ScienceQA and TabMWP were selected as general-purpose complex reasoning benchmarks because of their unique advantages. As discussed in **W2**, the context limitation of current LLMs necessitates dataset-specific variations. By updating the planner to GPT-4 16K or Claude 100K, Chameleon, Chameleon can accommodate a comprehensive and scale-up module inventory, supporting more general implementation. The variations on different datasets in this paper were tailored to optimize the use of LLM computing resources, not to limit its general applicability. \n\n### T3: High performance on general datasets\n\n1. **Dataset-specific approaches in comparisons**: We have compared all approaches available up to the time of submission. For instance, we compare LLaMA-Adapter in Table 3, one of the SOTA models proposed in late April. In our revised paper, we will include comparisons with the latest approaches after submission.\n\n2. **Generalization to other tasks**: Chameleon is a general framework of compositional reasoning that can achieve high performance on general datasets. Its adaptability is evidenced by an 11.85% gain on FinQA and an 11.39% boost on NER, both benchmarks in financial reasoning.\n\n3. **ScienceQA and TabMWP are diverse**: *As we claimed in the global response*, ScienceQA and TabMWP were selected due to diverse features and general applicability across questions, domains, contexts, and skills, which require a wide array of tool compositions.\n\n### Writing suggestions\n\n*Due to space limits, we addressed writing suggestions in the Comment Box below.*\n\nIf you believe that our clarifications addressed your concerns about the quality of our work, we wish that you would consider *raising the rating* to reflect that our contributions are properly recognized, which is crucial to us."}]

Author rebuttal:
We appreciate the efforts of all reviewers in reviewing our paper. Thank you for the insightful comments that recognize the novelty and contributions of our work, and for the valuable suggestions to enhance our paper. We would like to address the common concerns, emphasize the novelty and contribution of our paper, and the rules for benchmark selection as follows:

### Common concerns

Some critical concerns were shared by reviewers, including: 1) the general framework, 2) how easy it is to adapt to others and why, 3) why the two chosen datasets are useful to support our general argument, and 4) the reasoning for a dataset-specific module inventory at this time.

Regarding these concerns, both Reviewer yhWh (R1) and GU39 (R3) have provided comments regarding the substantial novelty in research present in our work. We are grateful to Reviewer yhWh (R1) and 1xpZ (R2)  for recognizing our research as a well-conducted technical study that presents comprehensive and robust results. Likewise, we appreciate R3's acknowledgment of the excellence of our model design.

We would like to emphasize that while expanding Chameleon to more datasets may seem like a relatively apparent next step, we are the first to come up with a concrete design and demonstrate its effectiveness. Publishing these results will put everyone else in the field one step ahead, and we believe that these findings are valuable for advancing this field.

### Novelty of our approach

Our Chameleon work represents a significant stride in the evolving landscape of LLMs, as acknowledged by the reviewers in various aspects of the novelty:

1. **One of the first works in the field of LLM agents**: As Reviewer yhWh (R1) recognized, our approach is pioneering in automatic task planning and execution using GPT-4 as the agent. The combination of GPT-4 with various controlled tools enhances the capabilities of existing LLMs, marking a significant advancement in this domain.

2.  **An innovative approach that addresses the inherent limitations**: Reviewer GU39 (R3) acknowledged our novel system, Chameleon, as a unique solution that addresses fundamental challenges faced by LLMs.

3. **A general-purpose framework of compositional reasoning**: The Chameleon framework is not confined to specific tasks but rather has the potential to be extended as a general application, as pointed out in the takeaway by Reviewer uov2 (R4).

4. **Modularity with natural-language-format program generation**: Reviewer uov2 (R4) also highlighted the distinction of our work in constructing a solution entirely within modules, indicating the advantages of performing more tightly controlled reasoning steps and introducing constraints on the sequence of modules to use.

### Scientific contribution of our approach

Our scientific contributions are summarized as follows:

1. **Exploration of a critical research gap**: Reviewer uov2 (R4) validates the significance of this exploration: “The research topic (how to best use LLMs with tools) is definitely significant.” Our research fills a notable gap in the literature, particularly regarding comprehensive evaluations of LLMs with tools. Costly baselines have been evaluated and compared with published results, ensuring robustness and credibility.

2. **Rigorous evaluations**: Our work stands out for its clear and standardized evaluation metrics, differentiating it from other agent-based models that may lack clear benchmarks, as recognized by Reviewer yhWh (R1).

3. **Demonstrable enhancements and benchmarking:**: The study has consistently demonstrated significant improvements over strong baseline models, as acknowledged by multiple reviewers.

4. **Detailed experimental and qualitative insights**: The paper's strength also lies in its comprehensive experimental approach and the qualitative analysis that provides deeper insights into related research, as acknowledged by all reviewers.

### Concerns regarding dataset selection

Our study's emphasis on **compositional reasoning and tool use** guided the careful selection of benchmarks. The following factors were key considerations in choosing ScienceQA and TabMWP as benchmarks:

1. ​​**Multimodal contexts and reasoning abilities**: Both ScienceQA and TabMWP involve multimodal contexts and require the use of varied tools for reasoning abilities. Unlike other popular datasets like GSM8K, which focus on text-only problems, these benchmarks assess more complex capabilities. They align closely with the core research goals of our study, examining LLM's abilities in compositional reasoning and tool use, rather than merely text processing or domain-specific tasks.

2. **Diversity of questions, domains, contexts, and skills**: They feature a rich and varied composition of tools, aligning with our focus on compositional reasoning. Other datasets that are domain or context-limited do not provide the same level of diversity and challenge.

3. **Far from saturation in performance evaluation**: These benchmarks are not yet saturated in terms of performance, as evidenced by the latest few-shot models' scores (83.99% in ScienceQA for GPT-4 CoT and 81.8% in TabMWP for Codex PoT-SC). Other datasets have reached saturation, such as ARC, HellaSwag, and GSM-8K, with GPT-4's high scores of 96.3%, 95.3%, and 92.0%, respectively.

4. **Recent introduction and reduced risk of contamination**: Proposed in late 2022, these benchmarks carry less risk of contamination to the latest LLM training, ensuring a more genuine evaluation of the models' capabilities.

5. **Clear evaluation metrics**: Their focus on multi-choice or numeric questions allows for clear, objective evaluation metrics, aligning with our commitment to rigorous assessment.

###  Error analysis of ChatGPT and Chameleon

As suggested by Reviewer yhWh (R1), we added an error analysis of ChatGPT and Chameleon, to the new PDF below in the rebuttal period.

Meta-review: The paper proposes a methodology for leveraging LLMs to automatically compose and combine results from a range of tools.  While the technical contributions are relatively straightforward, the topic is highly relevant to current LLM research, and the overall framework represents a substantial engineering effort that achieves state-of-the-art results on ScienceQA and TabMWP with additional results for two datasets reported in the rebuttal (among other extensive experimental analysis that has been further augmented on rebuttal to address reviewer concerns).  I found the failure analysis in supplementary material to be very helpful and suggestive of follow-on research directions

There were a number of concerns that were raised in reviews and discussed on rebuttal, mostly to the reviewers' satisfaction.  I think two concerns are worth mentioning here.  One major concern involves overlap with the HuggingGPT paper posted to Arxiv on March 30, 2023, which would be considered concurrent work according to the 2023 NeurIPS FAQ for Authors.  A second major concern for reviewers is that some aspects of Chameleon are dataset-specific (tool availability, ordering), which the authors claim to address in new experiments during the rebuttals, but which do represent a major change from the submitted version.

While the paper is recommended for acceptance, reviewers maintain concerns about significant revisions to the paper to incorporate rebuttal content and the authors are requested to carefully consider such changes at the final revision stage that will not undergo further review.
Decision: Accept (poster)