Title: Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: large language models

compositional reasoning

tool use

multi-modal reasoning

mathematical reasoning
Abstract: Large language models (LLMs) have achieved remarkable progress in solving various natural language processing tasks due to emergent reasoning abilities. However, LLMs have inherent limitations as they are incapable of accessing up-to-date information (stored on the Web or in task-specific knowledge bases), using external tools, and performing precise mathematical and logical reasoning. In this paper, we present Chameleon, an AI system that mitigates these limitations by augmenting LLMs with plug-and-play modules for compositional reasoning. Chameleon synthesizes programs by composing various tools (e.g., LLMs, off-the-shelf vision models, web search engines, Python functions, and heuristic-based modules) for accomplishing complex reasoning tasks. At the heart of Chameleon is an LLM-based planner that assembles a sequence of tools to execute to generate the final response. We showcase the effectiveness of Chameleon on two multi-modal knowledge-intensive reasoning tasks: ScienceQA and TabMWP. Chameleon, powered by GPT-4, achieves an 86.54% overall accuracy on ScienceQA, improving the best published few-shot result by 11.37%. On TabMWP, GPT-4-powered Chameleon improves the accuracy by 17.0%, lifting the state of the art to 98.78%. Our analysis also shows that the GPT-4-powered planner exhibits more consistent and rational tool selection via inferring potential constraints from instructions, compared to a ChatGPT-powered planner.

Chameleon : Plug-and-Play Compositional Reasoning with Large Language Models

Abstract

Large language models (LLMs) have achieved remarkable progress in solving various natural language processing tasks due to emergent reasoning abilities. However, LLMs have inherent limitations as they are incapable of accessing up-to-date information (stored on the Web or in task-specific knowledge bases), using external tools, and performing precise mathematical and logical reasoning. In this paper, we present Chameleon , an AI system that mitigates these limitations by augmenting LLMs with plug-and-play modules for compositional reasoning. Chameleon synthesizes programs by composing various tools (e.g., LLMs, off-the-shelf vision models, web search engines, Python functions, and heuristic-based modules) for accomplishing complex reasoning tasks. At the heart of Chameleon is an LLM-based planner that assembles a sequence of tools to execute to generate the final response. We showcase the effectiveness of Chameleon on two multi-modal knowledge-intensive reasoning tasks: ScienceQA and TabMWP. Chameleon , powered by GPT-4, achieves an 86.54% overall accuracy on ScienceQA, improving the best published few-shot result by 11.37%. On TabMWP, GPT-4-powered Chameleon improves the accuracy by 17.0%, lifting the state of the art to 98.78%. Our analysis also shows that the GPT-4-powered planner exhibits more consistent and rational tool selection via inferring potential constraints from instructions, compared to a ChatGPT-powered planner.

1 Introduction

Remarkable progress has been observed in recent large language models (LLMs) for various natural language processing tasks, with prominent examples such as GPT-3 [4], PaLM [8], LLaMA [64], ChatGPT [41], and the recently developed GPT-4 [42]. LLMs have demonstrated emergent abilities, including in-context learning and chain-of-thought (CoT) reasoning [56]. These models are capable of solving diverse tasks in a zero-shot fashion [25] or with the aid of a few examples [57], and they show great potential in planning and decision-making akin to human beings [17, 16]. Despite these capabilities, LLMs face inherent limitations, such as an inability to access up-to-date information [26], perform precise mathematical reasoning [44, 35], or utilize specialized models [49]. Therefore, enhancing current LLMs with the capability to automatically compose external tools for real-world task solving is critical to address these drawbacks.

Consider the example ② in Figure 1: Which is the main persuasive appeal used in this ad? . To answer this question, one needs to: 1) infer that there is an ad image containing text context and call a text decoder to understand the semantics; 2) retrieve background knowledge about persuasive appeals and the differences among three persuasive appeals; 3) generate a solution based on the input query and intermediate results from previous steps; and 4) finally produce the answer in a task-specific format. On the other hand, when answering Which animal's skin is adapted for survival in cold places ( ③ ), one might need to call modules such as an image captioner to decipher image information and a web search engine to retrieve domain knowledge to understand scientific terminologies. However, current tool-augmented LLMs still face challenges when addressing these real-world queries across various scenarios. Most existing approaches are either limited to a small number of tools [39, 6, 55, 18, 43, 49] or relying on domain-specific tools [40, 60, 13, 59, 52], and thus are not easy to generalize to queries of new domains (see sections 2 and A.1 for further discussion). In this work, we study how to enable LLMs to synthesize programs to capture the logic of composing heterogeneous tools.

To address the challenges of existing work, we introduce Chameleon , a plug-and-play compositional reasoning framework that leverages LLMs to synthesize programs and compose various tools for a wide range of tasks. Unlike existing tool-augmented LLMs [49, 40, 60, 13, 59, 52], Chameleon uses a richer set of tools, including LLMs, off-the-shelf vision models, web search engines, Python functions, and heuristics-based modules. Moreover, Chameleon leverages the in-context learning capabilities of LLMs and builds on an LLM as a natural language planner, without requiring any training or carefully curated rules. Prompted by tool descriptions and usage examples, the planner infers a program composed of a sequence of tools to execute in order to generate the final response for a user query. Instead of generating programs in domain-specific languages [40, 52, 13], Chameleon generates natural-language-like (NL) programs (e.g., [Text_Detector, Knowledge_Retrieval, Solution_Generator, Answer_Generator] for the second query in Figure 1). The NLlike programs are easy to understand and debug by users with limited programming experience, and easily extendable to new modules. During each module's execution, the module processes the query and cached context, returns a result determined by the module itself, and updates the query and context for subsequent execution. Composing modules as a sequential program allows subsequent modules to leverage prior cached context and updated queries.

We showcase the adaptability and effectiveness of Chameleon on two tasks: ScienceQA [32] and TabMWP [33]. ScienceQA is a multi-modal question answering benchmark spanning multiple context formats and various scientific topics, while TabMWP is a mathematical benchmark involving diverse tabular contexts. These two benchmarks serve as a good testbed to evaluate Chameleon 's ability to coordinate diverse tools across different types and domains. Notably, Chameleon with GPT-4 achieves an 86.54% accuracy on ScienceQA, significantly improving upon the best published few-shot model by 11.37%. On TabMWP, using GPT-4 as the underlying LLM, Chameleon achieves an improvement of 7.97% over chain-of-thought (CoT) prompted GPT-4 [57] and a 17.0% increase over the best-published model [6], lifting the state of the art to 98.78%. Further studies suggest that using GPT-4 as a planner exhibits more consistent and rational tool selection and is able to infer potential constraints given the instructions, compared to other LLMs like ChatGPT.

Our contributions are as follows: (1) We develop a plug-and-play compositional reasoning framework, Chameleon , that effectively composes external tools to address inherent limitations of LLMs and tackle a broad range of reasoning tasks. (2) Relying on an LLM as a natural language planner to generate programs, Chameleon successfully integrates various tools, including LLMs, off-the-shelf vision models, web search engines, Python functions, and rule-based modules, to build a versatile and

9

This table shows the number of miles Wanda hiked each day on her

camping trip. The

median

is the middle value in a set of data when

the data is

arranged in order

. To find the

median

, the data must

be

arranged in order from least to greatest (or greatest to least)

,

and then the

middle value(s) is/are determined

.

Answer Generator

Knowledge Retrieval

Program Generator

Program Verifier

Program Executor

Wanda went on a camping trip

and logged the number of miles

she hiked each day. What is the

median

of the numbers?

miles_hiked = [10, 9, 10, 5, 9]

miles_hiked =

sorted

(miles_hiked)

middle1 =

(len(miles_hiked) - 1) // 2

middle2 =

len(miles_hiked) // 2

ans =

(miles_hiked[middle1] + miles_hiked[middle2]) / 2

ans = 9.0

Day

Sunday

10

Monday

9

Tuesday

10

Wednesday

5

Thursday

9

Look at the following schedule. When does

the bus

depart

from the

train station

?

(A) 12:35 P.M. (B) 1:10 P.M.

(C) 1:10 P.M. (D) 10:45 A.M.

Location

Arrive

stadium

10:20 A.M.

10:25 A.M.

park

10:35 A.M.

10:45 A.M.

hotel

11:10 A.M.

11:15 A.M.

airport

12:05 P.M.

12:10 P.M.

train station

12:25 P.M.

12:35 P.M.

bus station

1:10 P.M.

1:10 P.M.

12:35 P.M.

Answer Generator

Row Lookup

Solution Generator

Location

Arrive

train station

12:25 P.M.

12:35 P.M.

(Step 1) Find the train station on the schedule. Find the

departure

time for the

train station

.

(Step 2)

Train station

: 12:35 P.M. The bus

departs

from the

train

station

at 12:35 P.M.

(Step 3)

The answer is 12:35 P.M.

1

1

2

2

adaptable AI system capable of answering real-world queries. (3) We demonstrate Chameleon 's effectiveness on two challenging benchmarks, significantly surpassing the state of the art.

2 Related Work

Compositional Reasoning Neural modular and compositional approaches have been explored to automatically perform desired sub-task decomposition, enhancing interpretability and adaptability across various reasoning tasks. Early work [2, 3] posits that complex reasoning tasks are fundamentally compositional and proposes neural module networks (NMN) to decompose them into subtasks. However, these methods rely on brittle off-the-shelf parsers and are limited by module configurations. Some later work [19, 15, 14, 21], takes a step further by predicting instance-specific network layouts in an end-to-end manner, without relying on parsers, using reinforcement learning [58] and weak supervised learning. In visual reasoning, models comprising a program generator and an execution engine have been proposed to combine deep representation learning and symbolic program execution [19, 61]. In the domain of mathematical reasoning, an interpretable solver has been developed to incorporate theorem knowledge as conditional rules and perform symbolic reasoning step by step [31]. Our work takes inspiration from neural module networks, yet it offers several distinct advantages. First, Chameleon does not require expensive supervision of task-specific programs for modeling training. Instead, it generates sequential programs, consisting of modules, that are easy to generalize to various domains and tasks, allowing the extension to new modules in a plug-and-play manner. Second, Chameleon does not require any training, but uses the in-context learning capabilities of LLMs to generate programs prompted by natural language instruction and demonstrations.

Tool-Augmented Language Models In recent years, the development of large language models (LLMs) [48, 8, 9, 53, 4, 41, 42] has made tremendous progress and has stimulated research in prompt learning [57, 33, 22] and instruction learning [53, 64, 46, 11]. Despite the impressive performance of LLMs, they suffer from inherent limitations, such as the inability to access up-to-date information [26], utilize external tools [49], or perform precise mathematical reasoning [44, 35]. Recent benchmarks, such as ScienceQA and TabMWP [32, 33, 7, 54, 51, 30], have emerged to evaluate the capability of LLMs to tackle intricate reasoning challenges, especially those emphasizing the use of external tools. Concurrently, there has been a growing interest in harnessing external tools and modular approaches to augment LLMs. These augmented LLMs can access real-time information aided by web search engines [40] and leverage domain-specific knowledge from external resources [62]. Some work leverages the Python interpreter to generate complex programs to employ powerful computational resources, and execute logical reasoning tasks more effectively [55, 10, 6, 39, 18, 43, 36]. For example, Toolformer [49] constructs tool-use augmented data to train language models to select five

Model

Tool Use

Inference & Extension

Size

Image

Web

Know.

Math

Table

Composition

Planning

Plug-n-Play

CoT [57]

1

✓

✗

✗

✗

✗

✗

✗

✗

✓

✗

✗

✗

✗

1

✓

✗

✗

✗

✓

✗

✗

✗

✓

✗

✗

✗

✗

PoT [6]

2

✓

✗

✗

✗

✓

✗

✗

✗

✓

✗

✗

✗

✗

Code4Struct [55]

1

✓

✗

✗

✗

✓

✗

✗

✗

✗

✗

✗

✗

✗

PAL [10]

2

✓

✗

✗

✗

✓

✗

✗

✗

✓

✗

✗

✗

✗

MathPrompter [18]

2

✓

✗

✗

✗

✓

✗

✗

✗

✓

✗

✗

✗

✗

ART [43]

4

✓

✗

✗

✓

✓

✗

✓

✗

✓

✗

✓

✗

✓

Toolformer [49]

5

✗

✗

✗

✓

✗

✗

✓

✗

✗

✗

✗

natural lang.

✗

WebGPT [40]

10

✓

✗

✗

✓

✗

✗

✓

✗

✗

✗

✓

program

✗

MM-ReAct [60]

>10

✓

✗

✗

✓

✗

✓

✓

✓

✓

✓

✓

word match

✓

Visual ChatGPT [59]

>10

✓

-

-

✗

✗

✓

✗

✗

✗

✗

✓

natural lang.

✓

ViperGPT [52]

>10

✓

-

-

✗

✗

✓

✗

✓

✓

✗

✓

program

✓

VisProg [13]

>10

✓

-

-

✗

✓

✓

✗

✗

✗

✗

✓

program

✓

>10

✓

✓

✗

✗

✗

✓

✗

-

✗

-

✓

natural lang.

✓

Chameleon

(ours)

>10

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

natural lang.

✓

tools. In the realm of visual tools, various approaches have been proposed to enhance the capabilities of large language models in handling visual tasks [60, 59, 52, 13, 50], augmented with Hugging Face models [50], Azure models [60], visual foundation models [59].

We compare Chameleon with other tool-augmented language models in Table 1. Many of these approaches are either constrained to a small set of tools or limited to task-specific tools, which reduces their capabilities across various skill dimensions and hampers their generalizability to new tasks. A recent line of work relies on large amounts of supervision [49, 26] and focuses on generating commands [40] and programs [52, 13] to infer the choice of tools. However, this approach needs to carefully tailored prompts to specific tasks and particular tools, and is neither flexible nor adaptive. In contrast, Chameleon instructs LLMs with natural language instructions that simply describe the roles of each module and provide a few calling examples, eliminating the need for additional training or tool-specific prompts when learning to compose different tools. More importantly, Chameleon offers users flexibility in terms of tool types and sources, updating the underlying LLMs, adding new tools, and adapting to new tasks. Our work shares the same spirit of AutoGPT [47], an autonomous GPT-4 agent with the artificial general intelligence (AGI) ambition to incorporate numerous tools to achieve user-defined goals. While AutoGPT is still under development, our work is the first to instantiate the idea and verify its effectiveness on well-studied benchmarks.

3 General Framework: Chameleon

To address the limitations of current LLMs in utilizing diverse tools, we propose Chameleon , a novel plug-and-play compositional reasoning framework, synthesizing the composition of various tools to accommodate a wide range of problems. Chameleon is comprised of a module inventory that defines different types of tools and an LLM-based planner , whose purpose is to decompose the original problem into sub-tasks that can be effectively solved by task-specific tools. Unlike existing tool-augmented LLM approaches [49, 13, 59, 50], our module inventory features multiple tool types as illustrated in Table 2, enabling Chameleon to exhibit various reasoning abilities, including image understanding, knowledge retrieval, web search, complex mathematical reasoning, and table understanding. Instead of generating domain-specific programs [40, 13, 52], Chameleon employs an LLM-based planner to create natural-language-like programs that follow natural language instructions, which is less error-prone, easily expandable to new modules, and user-friendly.

We formalize our planner as follows: given the input query x 0 , the module inventory M , and constraints G , the natural language planner P selects a set of modules that can be executed sequentially to answer the query via generating a program in a natural-language-like format. The module inventory M consists of a set of pre-built modules: { M i } , each corresponding to a tool of various types (Table 2). G are the constraints for the plan generation, for example, the concurrent relations and sequence

orders of modules. In our work, the planner P is an LLM prompted to generate a sequence of module names in a few-shot setup. The planner is prompted in natural language with a planning task instruction I , the descriptions of modules in M with corresponding constraints G , as well as a few demonstration examples D . A T -length plan sampled from P can be denoted as p = M 1 , . . . , M T , where M t represents an the t -th element in the generated plan and M t ∈ M . Formally, given an input query (problem statement) x 0 , a plan p is generated as follows:

Given the generated plan, the corresponding modules for each step are then executed sequentially. The plan is a natural-language program where each module is bound simply via string matching. When evaluating the module M t at time step t , the output of the execution y t is calculated by:

where x t -1 is the input for the current module M t , and c t -1 is the cached information (e.g., image semantics, retrieved knowledge, generated programs) resulting from the execution history of modules. Both the problem input x t and cache c t for the next module M t +1 are updated, respectively, by:

The update_input and update_cache functions are hand-designed for each M i . Specifically, update_input is applied to elements in the input query, including the question, table context, and image. These elements are updated after module execution. update_cache corresponds to the generation of new information, such as a description for the input image or retrieved knowledge from external resources. Finally, the response r to the query is generated by the last module M T :

4 Applications of Chameleon

We demonstrate the applications of Chameleon on two challenging tasks: ScienceQA [32] (section 4.2) and TabMWP [33] (section 4.3), using the module inventory introduced in section 4.1. Further experimental details can be found in appendix A.2.

4.1 Module Inventory

To accommodate various reasoning capabilities over a diverse range of queries, our system utilizes a rich module inventory of various external tools. We provide a high-level overview of this inventory here, with detailed implementations in specific experiments. The complete module inventory, M , is presented in Table 2. Each tool within the inventory is defined as follows:

Knowledge Retrieval : This module retrieves additional background knowledge crucial for tackling complex problems. It is especially beneficial for specialized domains like science and mathe-

matics, providing context for the task. For example, if a query is about a tax form table, this module could generate knowledge about tax procedures, offering valuable context.

Bing Search : Like 'Knowledge Retrieval', the 'Bing Search' module aims to provide wide-ranging task-relevant knowledge. In contrast, it excels when broader or up-to-date information from multiple sources is required. Using the search engine API, this module returns relevant search results based on the input query, which are then parsed and used by subsequent modules to gather richer context information from diverse sources, enhancing problem-solving effectiveness.

Query Generator : Since the original problem typically lacks a tailored query for retrieving taskrelevant information, this module creates search engine queries based on the problem, which are then

used by the 'Bing Search' module. Mostly, it is a good strategy to use the 'Query Generator' module before the 'Bing Search'. Coupled with the search engine tool, generating more targeted queries generally facilitates both the recall and precision of retrieved information.

Image Captioner : Designed to generate captions for images, this module provides crucial supplementary context for queries. It is particularly valuable when understanding an image semantically, like identifying objects and interactions in a scene. Using pre-trained models, it translates visual data into language, facilitating effective comprehension and reasoning about image content.

Text Detector : This module is designed to identify text within a given image. Typically, the 'Text Detector' is employed when a question requires the extraction of textual information from images containing diagrams, charts, tables, maps, or other visual elements. By effectively detecting text in various formats, this module aids in the analysis and understanding of image-based content.

Row Lookup : This module is crucial when queries involve tabular context, as locating relevant cells is often required. Large tables can distract the system, so 'Row Lookup' simplifies the table by retaining only the rows relevant to the query. If all rows are pertinent, it returns the original table.

Column Lookup : Like the 'Row Lookup' module, 'Column Lookup' addresses questions involving tabular context by focusing on relevant columns. It simplifies the table by retaining only pertinent columns, or returns the original table if all columns are relevant.

Table Verbalizer : Converting structured tables into text is likely to enhance the comprehension of tabular information by various downstream modules as shown by [37] for open-domain question answering, making this module a vital part of our system. It translates tables into easily understandable descriptions for modules like 'Program Generator' and 'Solution Generator', particularly useful for small, domain-specific tables like stem-and-leaf plots or function tables.

Program Generator : Program-aided approaches are shown to enhance the logical and mathematical reasoning abilities of LLMs [55, 10, 6, 39, 18, 43]. The 'Program Generator' generates Python programs to solve queries effectively, which is particularly beneficial for queries requiring complex computations or intricate logical operations, such as 'if-else' statements.

Program Verifier : Recent studies highlight the importance of verification to reduce hallucination [45, 38]. Hence, 'Program Verifier' ensures the validity and error-free nature of programs generated by 'Program Generator'. It checks for syntax and logical errors, and potential execution issues, enhancing the reliability and accuracy of the solutions.

Program Executor : This module executes the program generated by 'Program Generator' and produces the result, bridging the gap between program generation and final solution derivation.

Solution Generator : This module generates a detailed solution to the input query using all the cached information. Employing a chain-of-thought prompting approach [57], it ensures coherent and well-structured responses. The planner can directly employ this module instead of other functional modules if it can solve the query independently, especially for simpler ones.

Answer Generator : This task-specific module uses a rule-based approach to extract and normalize answers from the results of the 'Program Executor' or 'Solution Generator'. Unlike the Solution Generator' that provides detailed multi-step solutions, 'Answer Generator' serves as the final module in the pipeline, providing concise and task-specific answers.

4.2 Science Question Answering

Science Question Answering (ScienceQA [32]) is a diverse benchmark for multi-modal question answering over a range of scientific topics and contexts. As examples illustrated in Figure 1, answering these questions requires various tools and skills like image captioning, text detection, knowledge retrieval, online resource search, and multi-clue visual reasoning. When generating programs for using tools, we limit the search space to the relevant inventory subset (Table 6 in the appendix). Programs are deemed invalid and default to a 'Solution Generator' and 'Answer Generator' sequence if these are not the final two elements, following the chain-of-thought prompting baseline [57]. See Table 8 in the appendix for the constructed natural language planner prompt. The prompts for LLM-based modules like 'Knowledge Retrieval', 'Query Generator', and 'Solution Generator' are shown in Table 10, 11, and 12, respectively, in the appendix.

Model

#Tuned

Params

ALL

NAT

SOC

LAN

TXT

IMG

NO

G1-6

G7-12

Heuristic baselines

-

39.83

40.28

46.13

29.25

47.45

40.08

33.66

39.35

40.67

Human [32]

-

88.40

90.23

84.97

87.48

89.60

87.50

88.10

91.59

82.42

Fine-tuned models

MCAN [63]

95M

54.54

56.08

46.23

58.09

59.43

51.17

55.40

51.65

59.72

Top-Down [1]

70M

59.02

59.50

54.33

61.82

62.90

54.88

59.79

57.27

62.16

BAN [23]

112M

59.37

60.88

46.57

66.64

62.61

52.60

65.51

56.83

63.94

DFAF [12]

74M

60.72

64.03

48.82

63.55

65.88

54.49

64.11

57.12

67.17

ViLT [24]

113M

61.14

60.48

63.89

60.27

63.20

61.38

57.00

60.72

61.90

Patch-TRM [34]

90M

61.42

65.19

46.79

65.55

66.96

55.28

64.95

58.04

67.50

VisualBERT [27, 28]

111M

61.87

59.33

69.18

61.18

62.71

62.17

58.54

62.96

59.92

UnifiedQA [20]

223M

70.12

68.16

69.18

74.91

63.78

61.38

77.84

72.98

65.00

223M

74.11

71.00

76.04

78.91

66.42

66.53

81.81

77.06

68.82

MM-COT

T

[65]

223M

70.53

71.09

70.75

69.18

71.16

65.84

71.57

71.00

69.68

MM-COT [65]

223M

84.91

87.52

77.17

85.82

87.88

82.90

86.83

84.65

85.37

MM-COT

Large

[65]

738M

91.68

95.91

82.00

90.82

95.26

88.80

92.89

92.44

90.31

LLaMA-Adapter

T

[64]

1.2M

78.31

79.00

73.79

80.55

78.30

70.35

83.14

79.77

75.68

LLaMA-Adapter [64]

1.8M

85.19

84.37

88.30

84.36

83.72

80.32

86.90

85.83

84.05

Few-shot GPT-3

GPT-3 [4]

0M

74.04

75.04

66.59

78.00

74.24

65.74

79.58

76.36

69.87

GPT-3 CoT [32]

0M

75.17

75.44

70.87

78.09

74.68

67.43

79.93

78.23

69.68

Published results (Above)

▲

Few-shot ChatGPT

ChatGPT CoT

0M

78.31

78.82

70.98

83.18

77.37

67.92

86.13

80.72

74.03

Chameleon

(ChatGPT)

0M

79.93

81.62

70.64

84.00

79.77

70.80

86.62

81.86

76.53

Few-shot GPT-4

0M

83.99

85.48

72.44

90.27

82.65

71.49

92.89

86.66

79.04

Chameleon

(GPT-4)

0M

86.54

89.83

74.13

89.82

88.27

77.64

92.13

88.03

83.72

4.3 Tabular Mathematical Reasoning

TabMWP [33] is a mathematical reasoning task involving diverse tabular contexts like schedules, prices, tax forms, plots, and function relations (Figure 2). It requires AI systems to understand various table formats and perform precise numerical or symbolic computations. Like ScienceQA, we constrain the program search space to focus on two tool types: 1) those helping LLMs better digest tabular information (e.g., 'Row Lookup', 'Column Lookup', and 'Table Verbalizer') and 2) those performing faithful symbolic computations (e.g., 'Program Generator', 'Program Verifier', and 'Program Executor') as listed in Table 6. The generated programs must meet certain constraints, such as including 'Answer Generator' and placing 'Program Generator' prior to both 'Program Verifier' and 'Program Executor'. Non-compliant programs default to a sequence of 'Program Generator', 'Program Verifier', 'Program Executor', and 'Answer Generator', aligning with the program-of-thought prompting baseline [6] with added verification.

5 Experiments

We assess Chameleon 's effectiveness and adaptability on two complex reasoning tasks, ScienceQA [32] and TabMWP [33]. See experimental details in appendix A.2.

5.1 Experimental Results

ScienceQA. Table 3 presents the results of existing baselines and our approach Chameleon , with key results highlighted in Figure 3 (a). Employing ChatGPT [41] as the base LLM, Chameleon

84.9

85.2

70.1

75.2

78.3

79.9

84.0

86.5

MM-COT

LLaMA-Adapter

GPT-3 CoT

ChatGPT CoT

Chameleon

(ChatGPT)

Chameleon

(GPT-4)

65.0

70.0

75.0

80.0

85.0

90.0

Fine-tuned

Few-shot

Ours (few-shot)

(a) Results on ScienceQA

98.8

96.9

90.8

93.3

89.5

82.0

81.8

68.2

Chameleon

(GPT-4)

Chameleon

(ChatGPT)

ChatGPT PoT

ChatGPT CoT

Codex PoT-SC

GPT-3

CoT-PromptPG

60.0

65.0

70.0

75.0

80.0

85.0

90.0

95.0

Few-shot

Ours (few-shot)

(b) Results on TabMWP

achieves a 79.93% accuracy, a 1.62% improvement over Chain-of-Thought (CoT) [57] prompted ChatGPT. Notably, Chameleon is a generalized form of CoT, where the generated program is a sequence of 'Solution Generator' and 'Answer Generator'. Chameleon benefits from additional tool usage, such as 'Knowledge Retrieval', 'Bing Search', 'Image Captioner', and 'Text Detector'. When built upon GPT-4 [42], our model attains an accuracy of 86.54%, outperforming GPT-4 CoT [32] by 2.55% and GPT-3 CoT by 11.37%, creating the new state of the art in few-shot settings.

TabMWP. Table 4 presents results with key models in Figure 3 (b). Similarly, significant improvements are observed for Chameleon over both fine-tuned and few-shot models. It is worth noting that both CoT and Program-of-Thought (PoT) [6] can be viewed as special cases of Chameleon . Apart from 'Solution Generator' and 'Answer Generator', CoT doesn't utilize any tool, while PoT

Not called

Called

Chameleon

(ChatGPT)

Chameleon

(GPT-4)

Knowledge Retrieval

Text Detector

Image Captioner

Bing Search

19

72

17

30

81

10

6

2

11

3

11

70

83

28

94

90

98

89

97

89

Not called

Called

Chameleon

(ChatGPT)

Chameleon

(GPT-4)

Row Lookup

Column Lookup

Table Verbalizer

Knowledge Retrieval

Program Generator

74

47

4

6

26

3

7

16

25

71

72

94

96

53

93

97

84

75

29

28

only relies on symbolic programming tools like 'Program Generator' and 'Program Executor'. Chameleon (ChatGPT) outperforms ChatGPT CoT and ChatGPT PoT by 11.25% and 3.79%, respectively, emphasizing the advantage of our enriched tool set. With GPT-4, Chameleon gains an additional 5.50%, reaching a 98.78% accuracy. Notably, Chameleon (GPT-4) surpasses Codex PoT-SC [6], the best-published model, by 17.0% and human performance by 8.56%.

5.2 Qualitative Analysis

Tool use planning. The proportions of key tools called in the programs from Chameleon on ScienceQA and TabMWP are visualized in Figure 4 and Figure 5, respectively. Interestingly, ChatGPT and GPT-4 exhibit different planning behaviors. Generally, ChatGPT has a strong bias toward using or not using certain tools, highly influenced by in-context examples. For instance, ChatGPT calls 'Knowledge Retrieval' in 72% of queries but only calls 'Bing Search' in 3% of cases on ScienceQA; on TabMWP, ChatGPT heavily relies on 'Row Lookup' (47%) but calls 'Column Lookup' less frequently (4%). However, GPT-4 acts more objectively and rationally in tool selection. For example, GPT-4 calls 'Knowledge Retrieval' more frequently (81% vs. 72%) and calls 'Bing Search' more than ChatGPT (11% vs. 3%) when answering scientific questions on ScienceQA. Impressively, GPT-4 consistently calls 'Query Generator' and 'Bing Search' simultaneously by observing the tool usage descriptions, while ChatGPT lacks such reasoning capability.

Ablation study with disabled modules. We study the accuracy decline of Chameleon when key modules in the generated programs are disabled (Table 5), using ChaptGPT as the underlying LLMs and 500 test examples. The results reveal that 'Knowledge Retrieval' plays a vital role in both tasks. Domain-specific tools, such as the search engine and vision models for ScienceQA, and program tools for TabMWP, also prove to be important.

Module transitions. We visualize the transition graphs of modules for generated programs by Chameleon (GPT-4) on ScienceQA and TabMWP in Figure 7 and 8, respectively. The transition probabilities in these graphs are computed from the tool transitions observed on the test sets. These graphs show that the GPT-4 planner is able to make good decisions on how to sequence tools in a fewshot setup. For example, on ScienceQA, Chameleon often decides to rely on either 'Knowledge Retriever' or 'Bing Search', but rarely both. On TabMWP, we observe two main modes: either going through the solution generator module or via the program generator, verifier, and executor.

5.3 Case Study

Visualization examples of ScienceQA. Examples from Chameleon (GPT-4) on ScienceQA are visualized in Figure 1. Chameleon (GPT-4) is able to adapt to different input queries by generating programs that compose various tools and executing them sequentially to obtain accurate responses. For instance, to answer the first question ( ① ), What is the direction of this push? , the system calls the image captioner model to extract semantic information from the image and employs the knowledge retrieval model to gather background knowledge for multi-modal reasoning. In the second example ( ② ), the natural language planner infers that a text detector tool is needed to understand the context

of the ad. The third query ( ③ ; more details provided in Figure 9 in the appendix), Which animal's skin is adapted for survival in cold places? , involves scientific terminology related to animal survival. The planner decides to call the Bing search engine to access domain-specific knowledge, benefiting from the numerous online resources.

Visualization examples of TabMWP. The adaptability and versatility of Chameleon for various queries are also observed on TabMWP, as illustrated in the examples in Figure 2. The first example ( ① ) involves mathematical reasoning on a tax form. Chameleon (1) calls the knowledge retrieval model to recall basic knowledge that assists in understanding this domain-specific table, (2) describes the table in a more readable natural language format, and (3) finally relies on program-aided tools to perform precise computations. In the second example ( ② ), the system generates Python code that closely aligns with the background knowledge provided by the knowledge retrieval model. The third example ( ③ ) requires the system to locate the cell in a large tabular context given the input query. Chameleon calls the row lookup model to help accurately locate the relevant rows and generate the language solution via an LLM model, instead of relying on program-based tools.

Failure cases and limitations. Failure examples from Chameleon (GPT-4) are illustrated in Tables 19 to 24 in the appendix. Inaccurate responses may arise from the limitations of the current modules or from suboptimal programs generated by the planner. Additionally, the module inventory may lack tools capable of addressing specific abilities. Future directions could involve upgrading the modules and the planner, or expanding the module inventory to support a broader range of capabilities. Further limitations and broader impacts are respectively discussed in sections B and C of the appendix.

5.4 Error Analysis

To examine the error sources of the base large language models and understand how our model reduces mistakes from different aspects, we conduct an error analysis, as shown in Figure 6. We select 50 mistake examples from the ChatGPT baseline on ScienceQA as the evaluation set. We count the number of mistake examples and analyze their corresponding mistake type categories for ChatGPT, our Chameleon (ChatGPT) approach, and Chameleon (GPT-4).

The results show that our Chameleon approach can substantially reduce the number of mistakes compared to ChatGPT. Our model features tools for image captioning and knowledge retrieval, thus the mistakes made by ChatGPT in the category of image understanding are reduced to 10 and 19 from 32 by Chameleon (ChatGPT) and Chameleon (GPT-4); while the mistakes made by ChatGPT in the category of knowledge understanding are reduced to 6 and 3 from 37 by Chameleon (ChatGPT) and Chameleon (GPT-4). Benefiting from the sequential execution of tools, the mistakes caused by solution generation are significantly reduced as well. Additionally, we find that the task planning of GPT4 outperforms ChatGPT by a large margin.

6 Conclusion

In conclusion, we introduce a novel plug-and-play compositional reasoning framework, Chameleon , that addresses the limitations of current large language models by augmenting them with external tools in a plug-and-play manner. Our approach employs a diverse set of tools and demonstrates impressive adaptability and effectiveness on two challenging benchmarks, ScienceQA and TabMWP. By achieving significant improvements in accuracy over existing state-of-the-art models, Chameleon showcases its potential for addressing real-world queries across various domains.

All

Tool Planning

Image

Knowledge

Solution

Mistake Type Category

0

10

20

30

40

50

Number of Mistakes in 50 Exampless

50

0

32

37

27

35

13

10

6

17

28

1

19

3

8

Number of Mistake Examples in Different Categories

ChatGPT

Acknowledgment

We would like to thank Chunyuan Li, Qiuyuan Huang, and other members of the Deep Learning group at Microsoft Research for their valuable discussions. We also thank Fan Yin from University of California, Los Angeles, and Mingyang Sun from University of Electronic Science and Technology of China for their thorough review of our paper and constructive feedback. Pan Lu's research for this work was financially supported by Microsoft during his visit at Microsoft Research, and was also partially supported by the Amazon PhD Fellowship, Bloomberg PhD Fellowship, Qualcomm Innovation Fellowship, and UCLA Dissertation Year Fellowship. Kai-Wei was supported an ONR grant N00014-23-1-2780 and as a Sloan Fellow.

References

Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018.

Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Learning to compose neural networks for question answering. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 1545-1554, 2016.

Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR) , pages 39-48, 2016.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021.

Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research , 2023.

Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Elaine Wan, Xueguang Ma, Jianyu Xu, Tony Xia, and Xinyi Wang. TheoremQA: A theorem-driven question answering dataset. In The 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2023.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022.

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 , 2022.

Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning , pages 10764-10799. PMLR, 2023.

Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. LLaMA-Adapter V2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304.15010 , 2023.

Peng Gao, Zhengkai Jiang, Haoxuan You, Pan Lu, Steven CH Hoi, Xiaogang Wang, and Hongsheng Li. Dynamic fusion with intra-and inter-modality attention flow for visual question answering. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6639-6648, 2019.

Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14953-14962, 2023.

Ronghang Hu, Jacob Andreas, Trevor Darrell, and Kate Saenko. Explainable neural computation via stack neural module networks. In Proceedings of the European conference on computer vision (ECCV) , pages 53-69, 2018.

Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE international conference on computer vision , pages 804-813, 2017.

Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning , pages 9118-9147. PMLR, 2022.

Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. In 6th Annual Conference on Robot Learning , 2022.

Shima Imani, Liang Du, and Harsh Shrivastava. Mathprompter: Mathematical reasoning using large language models. In ICLR 2023 Workshop on Trustworthy and Reliable Large-Scale Machine Learning Models , 2023.

Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Inferring and executing programs for visual reasoning. In Proceedings of the IEEE international conference on computer vision (ICCV) , pages 2989-2998, 2017.

Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. UnifiedQA: Crossing format boundaries with a single QA system. In Findings of the Association for Computational Linguistics (EMNLP) , pages 1896-1907, 2020.

Tushar Khot, Daniel Khashabi, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Text modular networks: Learning to decompose tasks in the language of existing models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 1264-1279, 2021.

Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. In International Conference on Learning Representations (ICLR) , 2023.

Jin-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang. Bilinear attention networks. In Advances in Neural Information Processing Systems (NeurIPS) , pages 1571-1581, 2018.

Wonjae Kim, Bokyung Son, and Ildoo Kim. ViLT: Vision-and-language transformer without convolution or region supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML) , pages 5583-5594, 2021.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems , 35:22199-22213, 2022.

Mojtaba Komeili, Kurt Shuster, and Jason Weston. Internet-augmented dialogue generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 8460-8478, 2022.

Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. VisualBERT: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 , 2019.

Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. What does BERT with vision look at? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL) , pages 5265-5275, 2020.

Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and Jian-Guang Lou. TAPEX: Table pre-training via learning a neural sql executor. In International Conference on Learning Representations (ICLR) , 2022.

Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating math reasoning in visual contexts with gpt-4v, bard, and other large multimodal models. arXiv preprint arXiv:2310.02255 , 2023.

Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. InterGPS: Interpretable geometry problem solving with formal language and symbolic reasoning. In The 59th Annual Meeting of the Association for Computational Linguistics (ACL) , 2021.

Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS) , 2022.

Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. In International Conference on Learning Representations (ICLR) , 2023.

Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. IconQA: A new benchmark for abstract diagram understanding and visual language reasoning. In The 35th Conference on Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks , 2021.

Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai-Wei Chang. A survey of deep learning for mathematical reasoning. In The 61st Annual Meeting of the Association for Computational Linguistics (ACL) , 2023.

Yujie Lu, Pan Lu, Zhiyu Chen, Wanrong Zhu, Xin Eric Wang, and William Yang Wang. Multimodal procedural planning via dual text-image prompting. arXiv preprint arXiv:2305.01795 , 2023.

Kaixin Ma, Hao Cheng, Xiaodong Liu, Eric Nyberg, and Jianfeng Gao. Open domain question answering with a unified knowledge interface. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1605-1620, Dublin, Ireland, May 2022. Association for Computational Linguistics.

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-Refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651 , 2023.

Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and Ashwin Kalyan. Lila: A unified benchmark for mathematical reasoning. In The 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2022.

Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. WebGPT: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332 , 2021.

OpenAI. Chatgpt, 2022.

OpenAI. GPT-4 technical report. ArXiv , abs/2303.08774, 2023.

Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. ART: Automatic multi-step reasoning and tool-use for large language models. arXiv preprint arXiv:2303.09014 , 2023.

Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 2080-2094, 2021.

Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint arXiv:2302.12813 , 2023.

Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with GPT-4. arXiv preprint arXiv:2304.03277 , 2023.

Toran Bruce Richards. Auto-GPT: An experimental open-source attempt to make GPT-4 fully autonomous. https://github.com/Significant-Gravitas/Auto-GPT , 2023.

Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili´ c, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 , 2022.

Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761 , 2023.

Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. HuggingGPT: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303.17580 , 2023.

Liangtai Sun, Yang Han, Zihan Zhao, Da Ma, Zhennan Shen, Baocai Chen, Lu Chen, and Kai Yu. SciEval: A multi-level large language model evaluation benchmark for scientific research. arXiv preprint arXiv:2308.13149 , 2023.

Dídac Surís, Sachit Menon, and Carl Vondrick. ViperGPT: Visual inference via python execution for reasoning. arXiv preprint arXiv:2303.08128 , 2023.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. SciBench: Evaluating college-level scientific problemsolving abilities of large language models. arXiv preprint arXiv:2307.10635 , 2023.

Xingyao Wang, Sha Li, and Heng Ji. Code4Struct: Code generation for few-shot structured prediction from natural language. arXiv preprint arXiv:2210.12810 , 2022.

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions on Machine Learning Research , 2022.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems , 35:24824-24837, 2022.

Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Reinforcement learning , pages 5-32, 1992.

Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual ChatGPT: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671 , 2023.

Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. MM-REACT: Prompting ChatGPT for multimodal reasoning and action. arXiv preprint arXiv:2303.11381 , 2023.

Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neuralsymbolic VQA: Disentangling reasoning from vision and language understanding. Advances in neural information processing systems (NeurIPS) , 31, 2018.

Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. Generate rather than retrieve: Large language models are strong context generators. In International Conference on Learning Representations (ICLR) , 2023.

Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6281-6290, 2019.

Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Qiao Yu. LLaMA-Adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199 , 2023.

Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923 , 2023.

Supplementary Materials for Chameleon : Plug-and-Play Compositional Reasoning with Large Language Models

A Appendix

A.1 Current Tool-Augmented LLMs

To address the limitations of LLMs, an active research direction involves augmenting language models with access to external tools and resources, as well as exploring the integration of external tools and plug-and-play modular approaches. For example, aided by web search engines and external knowledge resources, LLMs are able to access real-time information and leverage domain-specific knowledge [40]. To enhance mathematical reasoning abilities, recent work uses LLMs [5] to generate complex programs to exploit powerful computational resources, and execute logical reasoning tasks more effectively [55, 10, 6, 39, 18, 43]. Another line of recent work, such as ViperGPT [52], Visual ChatGPT [59], VisProg [13], and HuggingGPT [50] incorporates a collection of foundation computer vision models to equip LLMs with the abilities to perform visual reasoning tasks.

A.2 Experimental Details

Module search space. The inventory subsets for ScienceQA and TabMWP are shown in Table 6.

Table 6: Tools used on ScienceQA and TabMWP, respectively. Reusable tools are marked in green .

Planner implementations. We choose the gpt-3.5-turbo engine for ChatGPT and the gpt-4 engine for GPT-4 when constructing the LLM-based planner. The maximum length for generated programs is set to 128, and the temperature is set to 0 for the most deterministic generation. The planner prompts for the ScienceQA and TabMWP are illustrated in Table 8 and Table 9, respectively.

Module implementations for ScienceQA. By default, the LLM-based models use four in-context examples as demonstrations, have a temperature setting of 0, and allow a maximum of 512 tokens for completion. Additional specific implementation details are provided as follows:

Knowledge Retrieval : The prompt consists of 3 demonstration examples and the template is shown in Table 10.

Query Generator : The prompt template is shown in Table 11. The maximum number of tokens for completion is set as 64.

Solution Generator : The prompt consists of 2 demonstration examples and the template is shown in Table 12.

Image Captioner : We use the captioning model 1 to generate textual descriptions for input images. The maximum length of generated captions is set to 16, the number of beams is 4, and the maximum number of output tokens is 512.

Text Detector : This module is based on the github model 2 to extract the text contents with coordinates in the image.

Bing Search : This module calls the Bing Search API 3 and returns the top three responses for the text query.

Answer Generator : This module extracts the answer snippet from the result provided by the 'Solution Generator' and selects the most similar option from the given choices.

Module implementations for TabMWP. Similar to ScienceQA, the LLM-based modules by default use four in-context examples as demonstrations, have a temperature setting of 0, and allow a maximum of 512 tokens for completion. Additional implementation details are provided as follows:

Knowledge Retrieval : The prompt consists of 5 demonstration examples and the template is shown in Table 13.

Row Lookup : It is enabled only when there are more than three rows and 18 table cells, in order to accelerate inference. The prompt consists of 7 demonstration examples and the template is shown in Table 14. The maximum number of tokens for completion is set as 256.

Column Lookup : Similarly, this module is enabled with two or more columns and 18 or more table cells. The prompt consists of 6 demonstration examples and the template is shown in Table 15. The maximum number of tokens for completion is set as 256.

Table Verbalizer : The prompt consists of 7 demonstration examples and the template is shown in Table 16.

Program Generator : The prompt template is shown in Table 17. The maximum number of tokens for completion is set as 256.

Solution Generator : The prompt consists of 16 demonstration examples and the template is shown in Table 18.

Answer Generator : It is used to normalize answers with two-place precision for questions with numerical answers and select the most similar option for multiple-choice questions.

Implementations of update_input and update_cache . update_input is triggered by the execution of specific tools, like 'Row_Lookup', which alter or replace elements in the input to reflect the updated state. Tools such as 'Image_Captioner', 'Text_Detector', 'Knowledge_Retrieval', 'Web_Search', and 'Program_Generation' generate new elements. update_cache stores these new elements in the cache, making them accessible for later tools' execution.

A.3 Experimental Results

Generated program statistics. Chameleon utilizes the LLM-based natural language planner to generate programs, i.e., sequences of used modules (tools). We report the statistics of the number of unique generated programs and the average length of corresponding tool sequences by Chameleon in Table 7. On both ScienceQA and TabMWP, using GPT-4 as the base LLM generates fewer distinct programs, i.e., more consistent programs, than using ChatGPT, even when given the exact same prompt in the planning model. Our results are consistent with the findings in [42], which observes that GPT-4 has a superior capability of understanding long contexts, aligning with human instructions, and performing high-level reasoning compared to other LLMs such as ChatGPT.

B Limitations

While Chameleon represents a significant stride in exploiting large language models (LLMs) for compositional reasoning in a plug-and-play manner, there are a few areas that could benefit from

further refinement. One such area is the expansion of its adaptability to a wider variety of tasks and domains, beyond the benchmarks presented. The LLM-based planner, responsible for synthesizing programs and determining the sequence of tools, introduces an innovative approach, yet it also raises intriguing research questions about optimizing the process for tool selection and sequence. It is plausible in the current system design that the quality of the LLM-based planner could impact overall performance. Moreover, Chameleon generates the program at one step, without incorporating a re-planning mechanism as the modules in the program are processed. Furthermore, we make the assumption that the list of modules and their descriptions will fit within the context window of LLMs, which may not always be the case. As the task complexity increases and the module inventory expands, there might be a corresponding surge in computational demands or limitations due to the context limit, indicating potential areas for future optimization. However, these potential areas for enhancement don't detract from the paper's central achievements, but instead provide valuable directions for future work and research.

C Broader Impacts

The work presented in this paper, Chameleon , has significant potential for positive societal impact. By augmenting large language models (LLMs) with plug-and-play modules for compositional reasoning, Chameleon can provide more accurate responses to complex, multi-modal tasks, making it a potentially valuable framework for various applications, including but not limited to education, finance, and decision support systems. Additionally, the system's ability to synthesize programs without requiring any training could democratize access to AI technology, enabling non-experts to leverage the power of AI in diverse fields. As research continues to advance in large language models and tool integration, we anticipate that our framework will serve as a foundation for further innovations in pursuing more generalizable and efficient solutions to complex reasoning tasks.

While there might be negative societal impacts associated with the Chameleon , such as misinformation and privacy concerns if data sources and external tools it utilizes are not curated meticulously, we believe these risks can be carefully managed and minimized. There's also a risk that excessive reliance on Chameleon 's increased autonomy may undermine critical thinking skills or job functions. To effectively mitigate these issues, careful curation of data sources and external tools, along with a strong commitment to user data protection, are essential. Additionally, Chameleon 's autonomy should be viewed as a means to augment, not replace, human capabilities. Therefore, the development of robust ethical guidelines, transparency mechanisms, and safeguards is critical, underlying our commitment to the socially responsible deployment of AI.

▷ Instruction for the planner model

You need to act as a policy model, that given a question and a modular set, determines the sequence of modules that can be executed sequentially can solve the question.

The modules are defined as follows:

Query_Generator : This module generates a search engine query for the given question. Normally, we consider using "Query_Generator" when the question involves domain-specific knowledge.

Bing_Search : This module searches the web for relevant information to the question. Normally, we consider using "Bing_Search" when the question involves domain-specific knowledge.

Image_Captioner : This module generates a caption for the given image. Normally, we consider using "Image_Captioner" when the question involves the semantic understanding of the image, and the "has_image" field in the metadata is True.

Text_Detector : This module detects the text in the given image. Normally, we consider using "Text_Detector" when the question involves the unfolding of the text in the image, e.g., diagram, chart, table, map, etc., and the "has_image" field in the metadata is True.

Knowledge_Retrieval : This module retrieves background knowledge as the hint for the given question. Normally, we consider using "Knowledge_Retrieval" when the background knowledge is helpful to guide the solution.

Solution_Generator : This module generates a detailed solution to the question based on the information provided. Normally, "Solution_Generator" will incorporate the information from "Query_Generator", "Bing_Search", "Image_Captioner", "Text_Detector", and "Knowl-edge_Retrieval".

Answer_Generator : This module extracts the final answer in a short form from the solution or execution result. This module normally is the last module in the prediction pipeline.

Below are some examples that map the problem to the modules.

▷ In-context example(s)

Question: Compare the average kinetic energies of the particles in each sample. Which sample has the higher temperature?

Context: The diagrams below show two pure samples of gas in identical closed, rigid containers. Each colored ball represents one gas particle. Both samples have the same number of particles.

Options: (A) neither; the samples have the same temperature (B) sample A (C) sample B

Metadata: 'pid': 19, 'has_image': True, 'grade': 8, 'subject': 'natural science', 'topic': 'physics', 'category': 'Particle motion and energy', 'skill': 'Identify how particle motion affects temperature and pressure'

Modules: ["Text_Detector","Knowledge_Retrieval","Solution_Generato r","Answer_Generator"]

Table 8: The prompt constructed for the planner model on the ScienceQA task. The prompt consists of the instruction that describes the role of the planner model, the in-context examples that map the problem to the module sequence, and the test example.

Table:

designer watch | $8,141

designer coat | $6,391

Question: How much more does a designer watch cost than a designer coat? (unit: $)

Modules: ["Program_Generator","Program_Verifier","Program_Executor ","Answer_Generator"]

Table 9: The prompt constructed for the planner model on the TabMWP task. Similarly, the prompt consists of the instruction, the in-context examples, and the test example.

▷ Instruction for the planner model

You need to act as a policy model, that given a question and a modular set, determines the sequence of modules that can be executed sequentially can solve the question.

The modules are defined as follows:

Program_Generator : This module generates a Python program that can solve the given question. It takes in the question and possible context and produces a program that can be executed by the "Program_Executor" module. Normally, we consider using "Program_Generator" when the questions and contexts involve complex computation, such as arithmetic operations over multiple numbers, or when the questions involve complex logical operations, such as "if-else" statements.

Program_Verifier : This module verifies whether the generated program from "Pro-gram_Generator" is valid and error-free. It checks for syntax errors, logical errors, and other potential issues that may arise during program execution.

Program_Executor : This module executes the generated program from "Program_Generator" and produces an output that can be further processed by other modules, such as "Question_Answering".

Row_Lookup : This module returns the simplified table that only remains the rows that are relevant to the question. It takes in the question and a table and returns the simplified table. If all rows are relevant or there are only three rows or fewer, return the original table. Normally, we only consider using "Row_Lookup" when the table involves more than three rows and the question only requires a small number of rows to answer the question.

Column_Lookup : This module returns the simplified table that only remains the columns that are relevant to the question. It takes in the question and a table and returns the simplified table. If all columns are relevant or there are only two columns, return the original table. Normally, we consider using "Column_Lookup" when the table involves more than two columns and the question only requires a small number of columns to answer the question.

Table_Verbalizer : This module converts the table to a description that can be easily understood by the downstream modules, like "Program_Generator", "Solution_Generator", "Ques-tion_Answering". Normally, we consider using "Table_Verbalizer" when the table involves a small number of rows and columns and the table is domain-specific, such as steam-and-leaf plots, function tables, etc.

Knowledge_Retrieval : This module retrieves domain-specific knowledge for the given question and table. Normally, we consider using "Knowledge_Retrieval" when the question and table involve domain-specific knowledge, such as "steam-and-leaf plots", "function tables", "tax forms", etc.

Solution_Generator : This module generates a detailed solution to the question based on the information provided. Normally, we use "Solution_Generator" when the question and table involve simple computation, such as arithmetic operations over a single number.

Answer_Generator : This module extracts the final answer in a short form from the solution or execution result. This module normally follows the "Solution_Generator" or "Problem_Executor" module.

Below are some examples that map the problem to the modules.

▷ In-context example(s)

▷ Instruction

Read the following question, and generate the background knowledge as the context information that could be helpful for answering the question.

▷ In-context example(s)

Question: Which property do these three objects have in common?

Options:

(A) hard (B) soft (C) yellow

Metadata: 'pid': 43, 'has_image': True, 'grade': 4, 'subject': 'natural science', 'topic': 'physics', 'category': 'Materials', 'skill': 'Compare properties of objects'

Detected text in the image:

['handkerchief', 'slippers', 'leisure suit']

Knowledge:

This question is about comparing the properties of three objects: a handkerchief, slippers, and a leisure suit.

The objects are related to the topic of physics and the skill of comparing properties of objects.

Properties of objects can include physical characteristics such as color, texture, shape, size, weight, and material.

▷ Instruction

Read the following question and metadata, and generate the query for browser search as the context information that could be helpful for answering the question.

▷ In-context example(s)

Question: Which property do these two objects have in common?

Options: (A) hard (B) bendable

Metadata: 'pid': 329, 'has_image': True, 'grade': 2, 'subject': 'natural science', 'topic': 'physics', 'category': 'Materials', 'skill': 'Compare properties of objects'

Detected text in the image: [([[41, 183], [131, 183], [131, 199], [41, 199]], 'rubber gloves'), ([[245, 183], [313, 183], [313, 197], [245, 197]], 'rain boots')]

Search Query: Common material properties of jump rope and rubber gloves

Table:

x | y

10 | 15

11 | 9

12 | 2

Question:

The table shows a function. Is the function linear or nonlinear?

Knowledge:

A linear function is a function whose graph is a straight line.

A nonlinear function is a function whose graph is not a straight line.

The equation of a linear function is y = mx + b , where m is the slope and b is the y -intercept.

The equation of a nonlinear function is not y = mx + b .

▷ Instruction

Given the question (and the context), select the answer from the options ["A", "B", "C", "D", "E"]. You should give concise and step-by-step solutions. Finally, conclude the answer in the format of "the answer is [ANSWER]", where [ANSWER] is one from the options ["A", "B", "C", "D", "E"]. For example, "the answer is A", "the answer is B", "the answer is C", "the answer is D", or "the answer is E". If the answer is not in the options, select the most possible option.

▷ In-context example(s)

Question:

Which property do these two objects have in common?

Context: Select the better answer.

Options: (A) hard (B) bendable

Metadata: 'pid': 6493, 'has_image': True, 'grade': 2, 'subject': 'natural science', 'topic': 'physics', 'category': 'Materials', 'skill': 'Compare properties of objects'

Image caption:

A pair of scissors next to a pair of scissors.

Detected text with coordinates in the image: [([[53, 185], [121, 185], [121, 199], [53, 199]], 'jump rope'), ([[233, 183], [323, 183], [323, 201], [233, 201]], 'rubber gloves')]

Retrieved knowledge:

This question is about comparing the properties of two objects: rubber gloves and rain boots.

The objects are related to the topic of physics and the skill of comparing properties of objects.

Properties of objects can include physical characteristics such as color, texture, shape, size, weight, and material. In this case, the two objects have the property of being bendable in common.

Bing search response: The most common materials used for disposable gloves are Latex, Vinyl and Nitrile. Each material has its benefits and drawbacks. Latex Gloves are constructed from Natural Rubber Latex and are the most popular type of disposable glove.

Solution: An object has different properties. A property of an object can tell you how it looks, feels, tastes, or smells. Different objects can have the same properties. You can use these properties to put objects into groups. Look at each object. For each object, decide if it has that property. A bendable object can be bent without breaking. Both objects are bendable. A hard object keeps its shape when you squeeze it. The rubber gloves are not hard. The property that both objects have in common is bendable. Therefore, the answer is B.

▷ Instruction

Read the following table and question, and generate the domain-specific knowledge as the context information that could be helpful for answering the question.

▷ In-context example(s)

▷ Instruction

Read the following question and table. Each row is separated by a newline ('\n') and each column is separated by a vertical bar ('|'). Return the simplified table that only remains the rows that are relevant to the question. If all rows are relevant, or the number of rows is fewer than three, return the original table.

▷ In-context example(s)

Question: In preparation for graduation, some teachers and students volunteered for the various graduation committees. How many people are on the music committee?

Table:

Committee | Students | Teachers

Program | 5 | 17

Ticket | 20 | 5

Music | 20 | 15

Schedule | 15 | 20

Food | 18 | 2

Simplified Table:

Committee | Students | Teachers

Music | 20 | 15

▷ Instruction

Read the following question and table. Each row is separated by a newline ('\n') and each column is separated by a vertical bar ('|'). Return the simplified table that only remains the columns that are relevant to the question. If all columns are relevant, return the original table.

▷ In-context example(s)

Question:

Look at the following schedule. When does Recess end?

Table:

Subject | Begin | End

Recess | 6:15 A.M. | 7:20 A.M.

Orchestra | 7:30 A.M. | 8:40 A.M.

Art | 8:45 A.M. | 9:35 A.M.

Handwriting | 9:45 A.M. | 10:20 A.M.

Gym | 10:30 A.M. | 11:15 A.M.

Choir | 11:20 A.M. | 12:25 P.M.

Science | 12:35 P.M. | 1:35 P.M.

Reading | 1:40 P.M. | 2:50 P.M.

Simplified Table:

Subject | End

Recess | 7:20 A.M.

Orchestra | 8:40 A.M.

Art | 9:35 A.M.

Handwriting | 10:20 A.M.

Gym | 11:15 A.M.

Choir | 12:25 P.M.

Science | 1:35 P.M.

Reading | 2:50 P.M.

Read the following question and table. Write a textual description of the table. The description should keep the critical information in the table for answering the question. The description should not answer the question.

▷ In-context example(s)

Table description: The table shows the number of students and teachers on each of the four graduation committees: Program, Ticket, Music, and Schedule. The Music committee has 20

▷ Instruction

Read the following table and then answer a question.

▷ In-context example(s)

Table:

Price | Quantity demanded | Quantity supplied

$895 | 21,000 | 3,400

$945 | 17,200 | 7,400

$995 | 13,400 | 11,400

$1,045 | 9,600 | 15,400

$1,095 | 5,800 | 19,400

Question: Look at the table. Then answer the question. At a price of $995, is there a shortage or a surplus? Please select from the following options: ['shortage', 'surplus'].

Solution: At the price of $995, the quantity demanded is greater than the quantity supplied. There is not enough of the good or service for sale at that price. So, there is a shortage. The answer is shortage.

START

knowledge_retrieval

0.53

text_detector

0.3

query_generator

0.11

image_captioner

0.06

solution_generator

1.0

0.78

0.22

bing_search

1.0

0.79

0.01

0.21

answer_generator

1.0

0.02

0.01

0.97

END

1.0

START

program_generator

0.39

row_lookup

0.26

knowledge_retrieval

0.19

solution_generator

0.11

column_lookup

0.05

program_veri

fi

er

1.0

0.2

0.24

0.26

0.03

table_verbalizer

0.27

0.63

0.36

0.02

answer_generator

1.0

0.74

0.02

0.02

0.25

program_executor

1.0

1.0

END

1.0

1.0

A

polar bear

is standing in the

snow

.

Animal skin adaptations

for

cold environments

Eurasian lynx

Answer Generator

Solution Generator

Image Captioner

(Response 1) Birds use feathers, and most mammals use hair or

fur, to trap a layer of air next to the

skin

and reduce heat transfer

to the environment. Marine mammals like whales use blubber, a

thick layer of fat, as a heavy-duty form of insulation. In

cold

weather

, birds fluff their feathers and animals raise their fur to

thicken the insulating layer.

(Response 2)

Adaptations

: Have a thick fur coat for insulation

against the

cold

. In winter, the coat is white to provide camouflage

against the ice and

snow

. In summer, when the

snow

has melted,

they have a thin, dark grey or brown coat which provides

camouflage against rock and vegetation.

(Response 3) The

polar bear

is

adapted

to life in a

cold climate

Polar bears

are well

adapted

for

survival

in the Arctic. Their

adaptations

include: a white appearance - as camouflage from

snow

...

The context mentions that

polar bears

live in and around the

cold

Arctic Ocean, and their

skin

is

adapted

to help them

survive

in

cold places

. Out of the given options, neither the

Eurasian lynx

nor the

thorny devil

is mentioned in the context. However, the

polar bear

is mentioned as an

animal

with

skin adapted

for

survival

in

cold places

. Since the

polar bear

is not listed as an

option, we must choose the most possible option from the given

choices. The

Eurasian lynx

lives in

colder regions

compared to

the

thorny devil

, which is a desert-dwelling reptile.

Therefore, the

answer is A.

Bing Search

Which

animal's skin

is

adapted

for

survival

in

cold places

?

(A) Eurasian lynx

(B) Thorny Devil

GO

交

Look at Bronson's pay stub. Bronson lives

in a state that has state income tax.

How

much did Bronson make after taxes

?

This table shows Bronson's pay stub for the pay period of March

16-31.Total earnings represent the amount of money Bronson

made before taxes.

Federal income tax

,

state income tax

, and

other taxes

are the amounts deducted from Bronson's total

earnings for various taxes.

Total taxes

can be calculated by adding

the

federal income tax

,

state income tax

, and

other taxes

.

Pay

after taxes

is the

amount of money Bronson made after all

taxes have been deducted from his total earnings

. This can be

calculated by subtracting the total taxes from the total earnings.

The table shows Bronson Le's pay stub for the pay period of March

16-31. It includes his total earnings of $

1,200.00

, federal income tax

of $

133.44

, state income tax of $

62.10

, and other taxes of $

91.00

.

The total taxes and pay after taxes are not provided but can be

calculated by adding the tax amounts and subtracting them

from the total earnings

, respectively.

913.46

Answer Generator

Knowledge Retrieval

Table Verbalizer

Program Verifier

Program Executor

ans = 913.46

Employee

Pay period

March 16-31

Total earnings

$1,200.00

Federal income tax

$133.44

State income tax

$62.10

Other taxes

$91.00

Pay after taxes

?

Program Generator

total_earnings = 1200

federal_income_tax = 133.44

state_income_tax = 62.10

other_taxes = 91.00

total_taxes =

federal_income_tax + state_income_tax +

other_taxes

ans =

total_earnings - total_taxes

Go

▷ Query

Question: Which better describes the Death Valley ecosystem?

Context: Figure: Death Valley. Death Valley is a desert ecosystem in eastern California.

Options: (A) It has dry, thin soil. It also has many different types of organisms. (B) It has a small amount of rain. It also has only a few types of organisms.

Metadata: 'pid': '18973', 'has_image': True, 'grade': 3, 'subject': 'natural science', 'topic': 'biology', 'category': 'Ecosystems', 'skill': 'Describe ecosystems'

▷ Response

Generated Program: ["Query_Generator","Bing_Search","Solution_Qenera tor","Answer_Generator"]

Query Generator:

Characteristics of Death Valley ecosystem

Bing Search: (the first one from response candidates)

Visitors come to Death Valley to experience the stark and lonely vastness of the valley, the panorama of rugged canyons and mountains; the pleasures of the dry, moderate winter climate, the challenge of the hot, arid summer; the relief of the cooler mountains; and the reminders of frontier and Native American ways of life.

Solution Generator:

Death Valley is a desert ecosystem, which means it has specific characteristics related to its climate and the organisms that live there. A desert ecosystem typically has dry, thin soil and a small amount of rain. This environment supports only a few types of organisms that are adapted to survive in such harsh conditions. Therefore, the answer is B.

Answer Generator (Wrong prediction):

(B) It has a small amount of rain. It also has only a few types of organisms.

Correct Answer (Ground truth):

(A) It has dry, thin soil. It also has many different types of organisms.

Table 19: An example of failure cases in ScienceQA from Chameleon (GPT-4) where some modules perform inaccurately . The generated query from the 'Query Generator' module is insufficiently specific, leading to a response from 'Bing Search' that does not adequately detail the characteristics of the Death Valley ecosystem, particularly with regard to the types of organisms present. (From U.S. Geological Survey: Contrary to its name, Death Valley National Park is very much alive ... Despite the low precipitation and extreme summer temperatures, Death Valley is home to many diverse species, several of which are endemic (found nowhere else in the world). ) 4

Question:

Which property matches this object?

Context:

Select the better answer.

Options:

(A) stretchy (B) yellow

Metadata: 'pid': '10338', 'has_image': True, 'grade': 2, 'subject': 'natural science', 'topic': 'physics', 'category': 'Materials', 'skill': 'Identify proper- ties of an object'

▷ Query

Question:

Which property matches this object?

Context:

Select the better answer.

Options:

(A) stretchy (B) yellow

Metadata: 'pid': '10338', 'has_image': True, 'grade': 2, 'subject': 'natural science', 'topic': 'physics', 'category': 'Materials', 'skill': 'Identify proper- ties of an object'

▷ Response

Generated Program: ["Image_Captioner","Solution_Generator","Answer_G enerator"]

Image Captioner:

A colorful kite is being flown in the sky.

Solution Generator:

Look at the image of the object, which is a colorful kite. A stretchy object can be stretched without breaking. Kites are typically made of materials that are not stretchy, such as nylon or polyester. A yellow object has a yellow color. The kite in the image is colorful and includes the color yellow. Therefore, the answer is B.

Answer Generator (Wrong prediction): (B) yellow

Correct Answer (Ground truth): (A) stretchy

Table 20: An example of failure cases in ScienceQA from Chameleon (GPT-4) where the generated programs are suboptimal . The LLM planner does not call the 'Text Detector' and 'Knowledge Retrieval' tools, resulting in an absence of semantic information from the image and a lack of related domain knowledge. Additionally, the employed image captioning model in providing an accurate textual description of the given visual context.

▷ Query

Question: Compare the average kinetic energies of the particles in each sample. Which sample has the higher temperature?

Context: Below is a food web from an ocean ecosystem in Monterey Bay, off the coast of California. A food web models how the matter eaten by organisms moves through an ecosystem. The arrows in a food web represent how matter moves between organisms in an ecosystem.

Options:

(A) bat star (B) phytoplankton (C) kelp bass

Metadata: {'pid': '15175', 'has_image': True, 'grade': 6, 'subject': 'natural science', 'topic': 'biology', 'category': 'Ecological interactions', 'skill': 'Interpret food webs II'}

▷ Response

Generated Program: ["Text_Detector","Knowledge_Retrieval","Solution_ Generator","Answer_Generator"]

Text Detector:

[([297, 91], 'orca'), ([653, 85], 'sea otter'),

([487, 191], 'kelp bass'), ([73, 221], 'phytoplankton'),

([625, 269], 'black rockfish'), ([323, 327], 'zooplankton'),

([149, 363], 'plainfin'), ([593, 361], 'kelp'),

([132, 380], 'midshipman'), ([401, 391], 'bat star'),

([271, 433], 'sea cucumber'), ([409, 471], 'sea urchin')]

Knowledge Retrieval:

The question is about a food web in an ocean ecosystem in Monterey Bay, California.

A food web models how matter eaten by organisms moves through an ecosystem.

The arrows in a food web represent how matter moves between organisms in an ecosystem.

The organisms mentioned in the question are bat star, phytoplankton, and kelp bass.

The target organism is the sea cucumber.

In a food web, matter moves from one organism to another through consumption, predation, or decomposition.

Understanding the relationships between organisms in a food web can help determine the flow of matter between them.

Solution Generator:

Analyze the food web to determine which organism's matter eventually moves to the sea cucumber.

The sea cucumber consumes the bat star.

The bat star consumes the sea urchin.

The sea urchin consumes the kelp.

The kelp gets its nutrients from the water, which contains matter from phytoplankton and other sources.

In this food web, the matter from the bat star eventually moves to the sea cucumber through consumption. Therefore, the answer is A.

Answer Generator (Wrong prediction): (A) bat star

Correct Answer (Ground truth): (B) phytoplankton

▷ Query Table: Table: (semi-structured format) x | y 4 | 15 19 | 18 11 | 10 11 | 1 Question: Look at this table. Is this relation a function? Options: (A) yes (B) no ▷ Response Generated Program: ["Knowledge_Retrieval","Program_Generator","Progr am_Verifier","Program_Executor","Answer_Generator"] Knowledge Retrieval: - A relation is a set of ordered pairs ( x , y ). - A function is a relation in which each input ( x -value) is paired with exactly one output ( y -value). - To determine if a relation is a function, check if each x -value is paired with only one y -value. - If an x -value is paired with more than one y -value, the relation is not a function. Program Generator:

Table:

(semi-structured format)

Date

| Description | Received | Expenses | Available Funds

| Balance: end of August | | | $154.90

9/1 | notebook | | $6.95 | $147.95

9/15 | walking dogs | $15.00 | | $162.95

9/21 | basketball | | $11.35 | $151.60

Correct Answer (Ground truth):

151.60

▷ Query

Question: This is Oliver's complete financial record for September. How much money did Oliver have after he purchased the basketball?

Table:

Table:

(semi-structured format)

Date

| Description | Received | Expenses | Available Funds

| Balance: end of August | | | $154.90

9/1 | notebook | | $6.95 | $147.95

9/15 | walking dogs | $15.00 | | $162.95

9/21 | basketball | | $11.35 | $151.60

▷ Response

Generated Program: ["Row_Lookup","Solution_Generator","Answer_Genera tor"]

Row Lookup:

Date | Description | Received | Expenses | Available Funds

9/21 | basketball | | $11.35 | $151.60

Solution Generator:

(Step 1) Subtract the cost of the basketball from the available funds before the purchase.

(Step 2) $151.60 - $11.35 = $140.25

(Step 3) Oliver had $140.25 after he purchased the basketball.

(Step 4) The answer is $140.25.

Answer Generator (Wrong prediction):

140.25

Correct Answer (Ground truth):

151.60

Table Title:

Tour bus schedule

Table:

(semi-structured format)

historic house | 10:15 A.M. | 10:30 A.M. | 10:45 A.M. | 11:00 A.M.

downtown | 11:00 A.M. | 11:15 A.M. | 11:30 A.M. | 11:45 A.M.

skyscraper | 11:30 A.M. | 11:45 A.M. | 12:00 P.M. | 12:15 P.M.

old building | 12:30 P.M. | 12:45 P.M. | 1:00 P.M. | 1:15 P.M.

governor's mansion | 1:00 P.M. | 1:15 P.M. | 1:30 P.M. | 1:45 P.M.

▷ Query

Table:

Table Title:

Tour bus schedule

Table:

(semi-structured format)

historic house | 10:15 A.M. | 10:30 A.M. | 10:45 A.M. | 11:00 A.M.

downtown | 11:00 A.M. | 11:15 A.M. | 11:30 A.M. | 11:45 A.M.

skyscraper | 11:30 A.M. | 11:45 A.M. | 12:00 P.M. | 12:15 P.M.

old building | 12:30 P.M. | 12:45 P.M. | 1:00 P.M. | 1:15 P.M.

governor's mansion | 1:00 P.M. | 1:15 P.M. | 1:30 P.M. | 1:45 P.M.

Question: This is Oliver's complete financial record for September. How much money did Oliver have after he purchased the basketball?

Options: (A) 2 hours and 15 minutes (B) 2 hours and 45 minutes (C) 15 minutes (D) 30 minutes ▷ Response

Generated Program: ["Row_Lookup","Solution_Generator","Answer_Genera tor"]

Row Lookup:

historic house | 10:15 A.M. | 10:30 A.M. | 10:45 A.M. | 11:00 A.M. governor's mansion | 1:00 P.M. | 1:15 P.M. | 1:30 P.M. | 1:45 P.M.

Solution Generator:

(Step 1) Find the last departure time from the historic house and the first arrival time at the governor's mansion:

Historic house last departure: 11:00 A.M.

Governor's mansion first arrival: 1:00 P.M.

(Step 2) Subtract the departure time from the arrival time:

1:00 P.M. - 11:00 A.M. = 2 hours

(Step 3) It takes 2 hours to get from the historic house to the governor's mansion.

(Step 4) The answer is 2 hours and 15 minutes.

Answer Generator (Wrong prediction): (A) 2 hours and 15 minutes

Correct Answer (Ground truth): (B) 2 hours and 45 minutes

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents Chameleon, a system that uses a library of tools or modules to assist a LLM to answer questions. First, a LLM is prompted with a description of available modules, constraints on how the modules should be used, few-shot demonstration examples, and finally the question to answer. The LLM then generates a sequence of modules that should be used to process the question. The modules are used in sequence, keeping track of module outputs and changes to the input question. Finally, the last module outputs an answer to the original question. When Chameleon uses GPT-4 as the base LLM, it achieves +2.55% improvement over GPT-4 with Chain-of-Thought prompting on ScienceQA, and +1.85% improvement over GPT-4 with Program-of-Thought prompting on TabMWP.\n\nSTRENGTHS:\nI see 4 main strengths of this paper:\n1. The idea of performing tool use entirely within a sequence of modules (instead of interleaving tool use with prediction from the original LLM) is new and offers some specific advantages\n2. Achieving a small performance increase on two question-answering datasets\n3. The project being an example of how engineering effort can be organized to achieve high dataset-specific performance\n4. The paper writing is well organized and easy to read\n\nOriginality: The most original aspect of this work is that the solution is constructed entirely within the modules, whereas previous work on tool use (to my knowledge) interleaves modules with prediction from the original LLM. With the Chameleon approach, the necessary reasoning at each step can be more tightly controlled (e.g., with module-specific prompts and heuristics about what information to include or exclude for future modules), and constraints can be placed on the sequence of modules to use.\n\nQuality: The authors provide an extensive comparison to prior models on two datasets, along with an ablation study for the importance of various modules. The experiments show a slight improvement over previous approaches.\n\nClarity: The paper is well organized, and the supplemental material contains extensive additional details for the modules.\n\nSignificance: The research topic (how to best use LLMs with tools) is definitely significant.\n\nWEAKNESSES:\nOverall, this paper feels less scientific and more like a report on an engineering accomplishment in achieving high accuracy on the benchmark datasets, since the modules were created with the datasets in mind, a dataset-specific subset of modules is available at test time, each module includes hand-designed heuristics for how to update the input and cached information, there are hand-designed constraints on module ordering, and dataset-specific “default” module sequences are used if the predicted one does not meet the constraints. All of this points to my interpretation that Chameleon is a framework that can be instantiated, with *sufficient engineering effort*, into a system with high *dataset-specific* accuracy. However, there is little evidence that the Chameleon approach can produce a *single, general-purpose* question answering system with high accuracy on a variety of datasets.\n\nFrom my perspective, the main weakness is the lack of some general takeaway that can be applied to other scenarios. There are some avenues toward potential takeaways but currently they are not explored thoroughly enough to draw confident conclusions. In the Questions section below, I provide some potential takeaways that could be explored in more detail.\n\nAnother weakness of the paper is that the writing oversells the approach in various ways. Please see the Questions section below for specifics. I hope the authors edit the writing accordingly.\n\nOriginality: There is not much novelty other than the “sequence of modules” approach mentioned in the Strengths section. Although, if the “sequence of modules” approach is shown to be better than prior approaches through further experiments, then its novelty would be sufficient.\n\nQuality: It is unclear what the main takeaways of the paper should be, or how well-supported they are.\n\nClarity: The writing oversells the work in a few ways.\n\nSignificance: Some of the potential takeaways, if explored more thoroughly, could be quite significant. However, the actual conclusions one can draw from evidence in the paper are less significant right now.\n\nQUESTIONS:\n## Potential takeaways that could be explored in more detail\n\n* Potential takeaway 1: performing tool use entirely within the sequence of modules is better than interleaving tool use with LLM generation as in prior work, e.g., ToolFormer. This might be due to constraints in module ordering, heuristics in how modules update the context, using module-specific prompts instead of the original prompt which might be overloaded, or interleaved tool use “interrupting” the flow of text generation. I would love to see an analysis of these different factors using the same set of tools, for a more general understanding of the benefit of Chameleon’s “sequence of modules” approach.\n\n* Potential takeaway 2: the Chameleon framework is truly general-purpose, meaning the same system achieves high performance on a variety of datasets. This would represent an important step forward in question answering. But currently, we only see dataset-specific variations of Chameleon performing well on one dataset at a time, which is less impressive because the comparison is to GPT-4 with CoT or PoT which has no dataset-specific design or engineering (other than a few-shot prompt). How does Chameleon perform on each dataset with all modules available, a single setting of module ordering constraints, and a single default module sequence? (If the performance is then worse than GPT-4 CoT or PoT, then is Chameleon truly a generally applicable approach?)\n\n* Potential takeaway 3: if the goal is to achieve high accuracy on a single academic dataset, then one can find success using Chameleon with dataset-specific engineering and heuristics. This takeaway is already supported by the paper, but in my opinion, this is a weak result. There are no dataset-specific approaches in the comparison (at least no recent ones building upon ChatGPT or GPT-4). These are also *academic* datasets that are meant to serve as a proxy for measuring more general ability; raw performance on these datasets without generalization to other tasks is not as impressive. On the other hand, if Chameleon could get high performance on a more general and broad dataset, for example (hypothetically) a representative subset of queries to a search engine or AI assistant that are related to science/math/reasoning/etc., then it could be argued that Chameleon is general enough for practical applications, and then this takeaway would carry more weight.\n\nIn light of these, I'm not sure what readers are supposed to take away from the paper in its current state, and so I cannot recommend acceptance at this time. However I do think any of the potential takeaways above, if explored thoroughly, could lead to a much more impactful paper, so I encourage the authors to continue their investigations.\n\n## Writing that oversells the approach\n\n* The abstract mentions “improving the best published few-shot result by 11.37%” on ScienceQA and 17.0% on TabMWP, and these numbers are repeated throughout the text. While technically true, these numbers are misleading, simply because the performance of GPT-4 with CoT or PoT hasn’t been published before. Certainly the authors can’t claim any “ownership” of GPT-4 plus CoT or PoT, so they should be treated as prior work even if the authors are the first to measure their accuracy on these datasets. Considering this, it is more honest to say that Chameleon achieves **+2.55% on ScienceQA and +1.85% on TabMWP** over previous approaches without fine-tuning. (MM-COT Large is a fine-tuned model achieving +5.14% over Chameleon on ScienceQA, according to Table 3.) Lines 65-68 are carefully worded to avoid mentioning those bolded numbers and should be more transparent.\n\n* Throughout the text, Chameleon is said to produce “NL-like programs”. Is “*programs*” the most accurate term to use here? These are merely **sequences of modules**, without any other programming feature like loops, conditionals, variable binding, helper functions, and so on. I think the term “program” is overselling the complexity of the module sequence plans.\n\n* What does “plug-and-play” mean? This is a key term used in the paper’s title, but I actually think it hides complexity. To add a new module, one must: (1) implement the core module logic, (2) describe the module in the planner’s prompt, (3) describe constraints about when to use the module with respect to other modules in the planner’s prompt, (4) implement checks for those constraints so invalid plans can be rejected, (5) provide few-shot demonstrations of using the module in the planner’s prompt, (6) decide for which datasets the module should be available, (7) design the module’s update_input and update_cache functions. This seems like quite a bit of work -- an acceptable amount of work, but not quite “plug-and-play”. If easily adding new modules is a key feature of Chameleon, then at the very least, the authors should clearly describe all the steps involved in using a new module, but this is currently missing from the paper.\n\n* Why is MM-COT Large not included in Figure 3(a)? It would be misleading to omit the best approach from the “main baselines”.\n\n* Some modules are always used in a particular order. From Figure 6 and 7 from the appendix, the following modules are always used together: (Query Generator -> Bing Search), (Solution Generator -> Answer Generator), and (Program Generator -> Program Verifier -> Program Executor). Would it make sense to combine each group into a single module? Table 1 lists Chameleon as having >10 tools, but if these tools were combined, the count would be 9. Chameleon is described as using modules in a compositional way, but the number of tools is inflated compared to which compositions are actually used.\n\n## Minor questions and suggestions\n\n* Throughout the text: “**our** Chameleon” is awkward phrasing. It would be more natural to say “our approach Chameleon” or simply “Chameleon”\n* Line 97: correcting grammar - “has made tremendous progress and **has stimulated** research in **prompt** learning ...”\n* Line 125: extraneous space after Chameleon\n* Line 153: please provide examples of update_input and update_cache, since they are hand-designed for each module\n* Line 153: Is $M_t$ actually $M_i$? Or $M^t$?\n* Line 171 and so on: I didn’t notice any usage of the notation $M_{kr}$ elsewhere, is it necessary?\n* Line 233: missing quote before Knowledge Retrieval\n* Lines 246 - 248: this sentence is repeated\n* Line 251: “in the appendix” is repeated\n* Line 296: does the “solution-executor module” mean the “Solution Generator module”?\n\n---\n\nUpdated scores after the author-reviewer discussions:\n* Soundness: 3 -> 4\n* Contribution: 2 -> 3\n* Rating (\"overall score\"): 3 -> 4\n\nLIMITATIONS:\nThe limitations (Appendix B) should mention that it takes more wallclock time and compute resources to process questions with modules, especially if those modules themselves contain LLM components. If the use of multiple LLM-based tools really proliferates, then this would be a broader societal impact (Appendix C) through an increase in the amount of energy required to process queries, leading to greater environmental impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents Chameleon, an AI system that enhances large language models (LLMs) with plug-and-play modules for compositional reasoning. Chameleon uses various tools, including LLMs, vision models, web search engines, and Python functions, to perform complex reasoning tasks. The system, powered by GPT-4, significantly improves performance on two multi-modal reasoning tasks: ScienceQA and TabMWP, setting new state-of-the-art results. The authors also demonstrate that Chameleon's GPT-4-powered planner exhibits more consistent and rational tool selection compared to a ChatGPT-powered planner.\n\nSTRENGTHS:\n1. The paper is overall well-written and easy to follow. Abundant figures and visualizations also help with reader understanding.\n2. The paper presents a novel system, Chameleon, that enhances large language models (LLMs) with plug-and-play modules for compositional reasoning. This innovative approach addresses the inherent limitations of LLMs, such as their inability to access up-to-date information, use external tools, or perform precise mathematical and logical reasoning.\n3. Chameleon achieves impressive performances (especially with GPT-4) on two QA datasets: ScienceQA and TabMWP, showing the effectiveness of the entire framework.\n4. Very detailed and thorough analysis of existing related works, which can largely help the community to explore the direction.\n\nWEAKNESSES:\n1. I might miss something. However, the novelty of the method seems to be limited. As a method focusing on compositional tool use, I am wondering what are the differences between Chameleon and HuggingGPT (https://arxiv.org/pdf/2303.17580.pdf), except for the application tasks and involved tools. As both works seem to leverage human-introduced ordering information between tools to compositionally use the tools.\n2. Both datasets seem to have strong dependencies between different tools. With a relatively small number of simple tools (compared with vision models and SQL interpreters, etc.), the compositional order among tools seems to be quite static and rigid.\n\nQUESTIONS:\nOverall a good paper. No additional questions.\n\nLIMITATIONS:\nThe authors have thoroughly demonstrated the limitations of the work in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes Chameleon, an approach to connect large language models to different tools for compositional reasoning. Chameleon is powered by a GPT-4 planner that chains different tools to accomplish the described goal. Chameleon achieve improved performance in comparison with strong baseline models.\n\nSTRENGTHS:\n* The presentation of the framework is great. The paper has nice figures, clear definitions of problems, clear demonstrations of results.\n* The experiments are solid. This work evaluates Chameleon on two challenging benchmarks with distinct purposes\n* The study on how to make LLMs such as GPT-4 plan reasonably is useful for future research.\n\nWEAKNESSES:\n* The improvement on the two benchmarks is not as significant as described in the abstract. The abstract indicates that \"on ScienceQA, improving the best published few-shot result by 11.37%\" where the best-published result is a GPT-3 based approach while Chameleon is powered by GPT-4. Therefore, the comparison is not completely fair. The improvement does not necessarily come from the benefit of Chameleon. Instead, It could be as simple as GPT-4 being more powerful (as many tools are prompt-based). In Table 3, Chameleon (GPT-4) only shows a 2.55% improvement over GPT-4 CoT, and 1.62% when comparing Chameleon (GPT-3.5) with GPT-3.5 CoT.\n* The paper does not explain *how* to apply Chameleon to other tasks. Even though Chameleon is claimed to be plug-and-play, there are many *task-specific* designs. For instance, the `Row_Lookup`, `Column_Lookup` , `Table_Verbalizer` tools are highly specialized for table-related questions, and they are very fine-grained. How does a practitioner know whether to have both `Row_Lookup` and `Column_Lookup`, or a simple `Table_Lookup` tool? What are the insights on tool selections? Do we need a dev set to \"fine-tune\" the tool pool? These details need to be further explained to make Chameleon really plug-and-play.\n* Another concern is that converting the output of one tool (module) to fit the input format of another tool is pre-defined (L153-L157), indicating that it is potentially challenging to generalize Chameleon to *massive* number of tools due to the extensive human efforts.\n\nQUESTIONS:\n* How does Chameleon select the module inventory? What is the rule of thumb?\n* How are `update_input` and `update_cache` are implemented? Any examples?\n* Prompt-based tools such as `Row_Lookup` are powered by the same model as the planner; is that correct?\n\nLIMITATIONS:\n* Be more accurate in the improvement presentation", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Chameleon, a novel AI system that enhances large language models (LLMs) with plug-and-play modules for compositional reasoning. The authors argue that while LLMs have achieved remarkable progress in solving various natural language processing tasks, they have inherent limitations in accessing up-to-date information, using external tools, and performing precise mathematical and logical reasoning. To address these drawbacks, Chameleon synthesizes programs by composing various tools, such as LLMs, off-the-shelf vision models, web search engines, Python functions, and heuristic-based modules, for accomplishing complex reasoning tasks.\n\nAt the heart of Chameleon is an LLM-based planner that assembles a sequence of tools to execute to generate the final response. The authors showcase the effectiveness of Chameleon on two multi-modal knowledge-intensive reasoning tasks: ScienceQA and TabMWP. ScienceQA requires the system to answer science-related questions by integrating information from text, tables, and images, while TabMWP involves generating natural language descriptions of tables based on input queries. Chameleon, powered by GPT-4, achieves an 86.54% overall accuracy on ScienceQA, improving the best published few-shot result by 11.37%.\n\nThe authors also provide a detailed analysis of Chameleon's performance on various sub-tasks and demonstrate its ability to handle complex queries that require multiple tools and intermediate steps. They further discuss the potential of Chameleon for addressing real-world queries across various domains and improving the state of the art in multi-modal knowledge-intensive reasoning tasks. The paper concludes by highlighting the importance of enhancing current LLMs with the capability to automatically compose external tools for real-world task solving.\n\nSTRENGTHS:\n1. This paper probably is one of the first works in the field that explore automatic task planning and execution with GPT-4 as the agent. It further enhances the abilities of GPT-4 by leveraging various tools controlled by a GPT-4 based planner. Such a combination is straightforward but necessary for LLM-based agents.\n2. One merit I like about this paper is that it has clear evaluation metrics, not like other agent-based models that merely have a runnable model and use cases. The standard evaluation clearly helps us understand the position of this paper and the performance of the proposed method.\n3. The improvement on top of GPT-4/ChatgPT is significant and clearly shows the effectiveness of the proposed method. As we can see from Tables 3 and 4, the proposed method can mostly improve their direct baselines, i.e., ChatGPT and GPT-4. Also, qualitative analyses are helpful in understanding the model behavior.\n\nWEAKNESSES:\n1. Like the other recent works for LLM agents, the novelty of such kind of works is somehow ad-hoc and limited. There are many tool-augmented works, as demonstrated in Table 1, and their differences are petty nuanced, and it's hard to tell the real novelty behind them. Usually, each such kind of work would leverage the most recent LLMs, like GPT-4, to do the planning or task decomposition and call special tools to solve each subtask. Most powers come from OpenAI's model, esp. GPT-4, which is hardly claimed to be the credit of these works (but it's good to see the proposed methods can further improve GPT-4). This leaves us with an essential question, what are the core contributions of such works? Demonstrating GPT-4 can be augmented by external tools definitely is nontrivial but also engineering-heavy. For NeurIPs works, I may like to see more research values; for example, which designs of the proposed method really distinguish itself from other similar works? Is it the \"generating a program in a natural-language-like format\"? Or some special prompt format to trigger the planning and task execution abilities? etc etc.\n2. Although the paper names its method Chameleon, trying to highlight the adaptability and versatility of LLMs, ironically, the task scope and experiment settings in this paper are limited. The planning pipeline and proposed tools are specially designed for solving the two datasets. While conceptually, the idea of augmenting GPT-4 with some tools can definitely be applied to more general tasks, this generality comes from this simple idea but not from the proposed method. I hope the authors could dedicate more effort to make the proposed method easier to be applied to more tasks.\n3. Another weakness of the proposed method probably is the simple planning design, which is basically a straightforward CoT with a linear structure. As we can see from Table 3, there is still a large gap between fine-tuned STOA and the proposed method. One possible reason might be the weak planning ability of the proposed method. This is somehow related to the first weakness, meaning that most LLM agent works combine GPT-4 with tools in a very simple way. I hope the authors could explore more advanced planning algorithms to see if the performance can be further improved.\n\nQUESTIONS:\nI raised many questions in the weakness section, and the authors may want to resolve them and clarify some questions. The following is some minor comments:\n1. I don't quite get the connection between the name \"Chameleon\" and the proposed method. I also don't quite get the meaning of the colors in the name.\n2. I'd suggest reducing the table space, removing most of the unnecessary baselines. The only meaningful baselines probably are the STOA from previous works (mostly fine-tuned?) and GPT-4/ChatGPT. \n3. Considering some improvements on top of ChatGPT and GPT-4 are kind of marginal, I'd encourage the authors to conduct a thorough error analysis to figure out how the base ChatGPT/GPT-4 make mistakes and how the proposed method reduces such errors.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}