SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Pan Lu<sup>1</sup>, Baolin Peng<sup>2</sup>, Hao Cheng<sup>2</sup>, Michel Galley<sup>2</sup>
Kai-Wei Chang $^{1}$ , Ying Nian Wu $^{1}$ , Song-Chun Zhu $^{1}$ , Jianfeng Gao $^{2}$
<sup>1</sup>University of California, Los Angeles <sup>2</sup>Microsoft Research, Redmond
https://chameleon-llm.github.io
Figure 1: Examples from our Chameleon approach with GPT-4 on ScienceQA [32], a multi-modal question answering benchmark in scientific domains. Chameleon is adaptive to different queries by synthesizing programs to compose various tools and executing them sequentially to get final answers.
Large language models (LLMs) have achieved remarkable progress in solving various natural language processing tasks due to emergent reasoning abilities. However, LLMs have inherent limitations as they are incapable of accessing up-to-date information (stored on the Web or in task-specific knowledge bases), using external tools, and performing precise mathematical and logical reasoning. In this paper, we present Chameleon, an AI system that mitigates these limitations by augmenting LLMs with plug-and-play modules for compositional reasoning. Chameleon synthesizes programs by composing various tools (e.g., LLMs, off-the-shelf vision models, web search engines, Python functions, and heuristic-based modules) for accomplishing complex reasoning tasks. At the heart of Chameleon is an LLM-based planner that assembles a sequence of tools to execute to generate the final response. We showcase the effectiveness of Chameleon on two multi-modal knowledge-intensive reasoning tasks: ScienceQA and TabMWP. Chameleon, powered by GPT-4, achieves an  $86.54\%$  overall accuracy on ScienceQA, improving the best published few-shot result by  $11.37\%$ . On TabMWP, GPT-4-powered Chameleon improves the accuracy by  $17.0\%$ , lifting the state of the art to  $98.78\%$ . Our analysis also shows that the GPT-4-powered planner exhibits more consistent and rational tool selection via inferring potential constraints from instructions, compared to a ChatGPT-powered planner.
This title draws inspiration from the chameleon's ability to adapt and blend into its surroundings, which parallels the adaptability and versatility of large language models in compositional reasoning tasks with external tools.
Remarkable progress has been observed in recent large language models (LLMs) for various natural language processing tasks, with prominent examples such as GPT-3 [4], PaLM [8], LLaMA [64], ChatGPT [41], and the recently developed GPT-4 [42]. LLMs have demonstrated emergent abilities, including in-context learning and chain-of-thought (CoT) reasoning [56]. These models are capable of solving diverse tasks in a zero-shot fashion [25] or with the aid of a few examples [57], and they show great potential in planning and decision-making akin to human beings [17, 16]. Despite these capabilities, LLMs face inherent limitations, such as an inability to access up-to-date information [26], perform precise mathematical reasoning [44, 35], or utilize specialized models [49]. Therefore, enhancing current LLMs with the capability to automatically compose external tools for real-world task solving is critical to address these drawbacks.
Consider the example ② in Figure 1: Which is the main persuasive appeal used in this ad?. To answer this question, one needs to: 1) infer that there is an ad image containing text context and call a text decoder to understand the semantics; 2) retrieve background knowledge about persuasive appeals and the differences among three persuasive appeals; 3) generate a solution based on the input query and intermediate results from previous steps; and 4) finally produce the answer in a task-specific format. On the other hand, when answering Which animal's skin is adapted for survival in cold places (③), one might need to call modules such as an image captioner to decipher image information and a web search engine to retrieve domain knowledge to understand scientific terminologies. However, current tool-augmented LLMs still face challenges when addressing these real-world queries across various scenarios. Most existing approaches are either limited to a small number of tools [39, 6, 55, 18, 43, 49] or relying on domain-specific tools [40, 60, 13, 59, 52], and thus are not easy to generalize to queries of new domains (see sections 2 and A.1 for further discussion). In this work, we study how to enable LLMs to synthesize programs to capture the logic of composing heterogeneous tools.
To address the challenges of existing work, we introduce Chameleon, a plug-and-play compositional reasoning framework that leverages LLMs to synthesize programs and compose various tools for a wide range of tasks. Unlike existing tool-augmented LLMs [49, 40, 60, 13, 59, 52], Chameleon uses a richer set of tools, including LLMs, off-the-shelf vision models, web search engines, Python functions, and heuristics-based modules. Moreover, Chameleon leverages the in-context learning capabilities of LLMs and builds on an LLM as a natural language planner, without requiring any training or carefully curated rules. Prompted by tool descriptions and usage examples, the planner infers a program composed of a sequence of tools to execute in order to generate the final response for a user query. Instead of generating programs in domain-specific languages [40, 52, 13], Chameleon generates natural-language-like (NL) programs (e.g., [Text_Detector, Knowledge_Retestval, Solution_Generator, Answer_Generator] for the second query in Figure 1). The NLlike programs are easy to understand and debug by users with limited programming experience, and easily extendable to new modules. During each module's execution, the module processes the query and cached context, returns a result determined by the module itself, and updates the query and context for subsequent execution. Composing modules as a sequential program allows subsequent modules to leverage prior cached context and updated queries.
We showcase the adaptability and effectiveness of Chameleon on two tasks: ScienceQA [32] and TabMWP [33]. ScienceQA is a multi-modal question answering benchmark spanning multiple context formats and various scientific topics, while TabMWP is a mathematical benchmark involving diverse tabular contexts. These two benchmarks serve as a good testbed to evaluate Chameleon's ability to coordinate diverse tools across different types and domains. Notably, Chameleon with GPT-4 achieves an  $86.54\%$  accuracy on ScienceQA, significantly improving upon the best published few-shot model by  $11.37\%$ . On TabMWP, using GPT-4 as the underlying LLM, Chameleon achieves an improvement of  $7.97\%$  over chain-of-thought (CoT) prompted GPT-4 [57] and a  $17.0\%$  increase over the best-published model [6], lifting the state of the art to  $98.78\%$ . Further studies suggest that using GPT-4 as a planner exhibits more consistent and rational tool selection and is able to infer potential constraints given the instructions, compared to other LLMs like ChatGPT.
Our contributions are as follows: (1) We develop a plug-and-play compositional reasoning framework, Chameleon, that effectively composes external tools to address inherent limitations of LLMs and tackle a broad range of reasoning tasks. (2) Relying on an LLM as a natural language planner to generate programs, Chameleon successfully integrates various tools, including LLMs, off-the-shelf vision models, web search engines, Python functions, and rule-based modules, to build a versatile and
Figure 2: Two examples from our Chameleon approach with GPT-4 on TabMWP [33], a mathematical reasoning benchmark with tabular contexts. Chameleon demonstrates flexibility and efficiency in adapting to different queries that require various reasoning abilities.
adaptable AI system capable of answering real-world queries. (3) We demonstrate Chameleon's effectiveness on two challenging benchmarks, significantly surpassing the state of the art.
**Compositional Reasoning** Neural modular and compositional approaches have been explored to automatically perform desired sub-task decomposition, enhancing interpretability and adaptability across various reasoning tasks. Early work [2, 3] posits that complex reasoning tasks are fundamentally compositional and proposes neural module networks (NMN) to decompose them into subtasks. However, these methods rely on brittle off-the-shelf parsers and are limited by module configurations. Some later work [19, 15, 14, 21], takes a step further by predicting instance-specific network layouts in an end-to-end manner, without relying on parsers, using reinforcement learning [58] and weak supervised learning. In visual reasoning, models comprising a program generator and an execution engine have been proposed to combine deep representation learning and symbolic program execution [19, 61]. In the domain of mathematical reasoning, an interpretable solver has been developed to incorporate theorem knowledge as conditional rules and perform symbolic reasoning step by step [31]. Our work takes inspiration from neural module networks, yet it offers several distinct advantages. First, Chameleon does not require expensive supervision of task-specific programs for modeling training. Instead, it generates sequential programs, consisting of modules, that are easy to generalize to various domains and tasks, allowing the extension to new modules in a plug-and-play manner. Second, Chameleon does not require any training, but uses the in-context learning capabilities of LLMs to generate programs prompted by natural language instruction and demonstrations.
Tool-Augmented Language Models In recent years, the development of large language models (LLMs) [48, 8, 9, 53, 4, 41, 42] has made tremendous progress and has stimulated research in prompt learning [57, 33, 22] and instruction learning [53, 64, 46, 11]. Despite the impressive performance of LLMs, they suffer from inherent limitations, such as the inability to access up-to-date information [26], utilize external tools [49], or perform precise mathematical reasoning [44, 35]. Recent benchmarks, such as ScienceQA and TabMWP [32, 33, 7, 54, 51, 30], have emerged to evaluate the capability of LLMs to tackle intricate reasoning challenges, especially those emphasizing the use of external tools. Concurrently, there has been a growing interest in harnessing external tools and modular approaches to augment LLMs. These augmented LLMs can access real-time information aided by web search engines [40] and leverage domain-specific knowledge from external resources [62]. Some work leverages the Python interpreter to generate complex programs to employ powerful computational resources, and execute logical reasoning tasks more effectively [55, 10, 6, 39, 18, 43, 36]. For example, Toolformer [49] constructs tool-use augmented data to train language models to select five
Model	Tool Use	Skill Dimension	Inference & Extension
Size	◎	◎	◎	b	◎	Image	Web	Know.	Math	Table	Composition	Planning	Plug-n-Play
CoT [57]	1	✓	X	X	X	X	X	X	X	✓	X	X	X	X
Lila [39]	1	✓	X	X	X	✓	X	X	X	✓	X	X	X	X
PoT [6]	2	✓	X	X	X	✓	X	X	X	✓	X	X	X	X
Code4Struct [55]	1	✓	X	X	X	✓	X	X	X	X	X	X	X	X
PAL [10]	2	✓	X	X	X	✓	X	X	X	✓	X	X	X	X
MathPrompter [18]	2	✓	X	X	X	✓	X	X	X	✓	X	X	X	X
ART [43]	4	✓	X	X	✓	✓	X	✓	X	✓	X	✓	X	✓
Toolformer [49]	5	X	X	X	✓	X	X	✓	X	X	X	X	natural lang.	X
WebGPT [40]	10	✓	X	X	✓	X	X	✓	X	X	X	✓	program	X
MM-ReAct [60]	>10	✓	X	X	✓	X	✓	✓	✓	✓	✓	✓	word match	✓
Visual ChatGPT [59]	>10	✓	-	-	X	X	✓	X	X	X	X	✓	natural lang.	✓
ViperGPT [52]	>10	✓	-	-	X	X	✓	X	✓	✓	X	✓	program	✓
VisProg [13]	>10	✓	-	-	X	✓	✓	X	X	X	X	✓	program	✓
HuggingGPT [50]	>10	✓	✓	X	X	X	✓	X	-	X	-	✓	natural lang.	✓
Chameleon (ours)	>10	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	natural lang.	✓
Table 1: A comparison of work that augments large language models with tool usage. We report the tool size and tool types, including OpenAI (®, Hugging Face (®, Github (®, Web search (b), and code (c)). We compare the skills each method possesses, such as image understanding, browser search, knowledge retrieval, mathematical reasoning, and table understanding. Some models can compose various tools, propose a planner to infer the relevant tools for execution, or are inherently extendable to new tools. The label “-” refers to uncertain information in the literature.
tools. In the realm of visual tools, various approaches have been proposed to enhance the capabilities of large language models in handling visual tasks [60, 59, 52, 13, 50], augmented with Hugging Face models [50], Azure models [60], visual foundation models [59].
We compare Chameleon with other tool-augmented language models in Table 1. Many of these approaches are either constrained to a small set of tools or limited to task-specific tools, which reduces their capabilities across various skill dimensions and hampers their generalizability to new tasks. A recent line of work relies on large amounts of supervision [49, 26] and focuses on generating commands [40] and programs [52, 13] to infer the choice of tools. However, this approach needs to carefully tailored prompts to specific tasks and particular tools, and is neither flexible nor adaptive. In contrast, Chameleon instructs LLMs with natural language instructions that simply describe the roles of each module and provide a few calling examples, eliminating the need for additional training or tool-specific prompts when learning to compose different tools. More importantly, Chameleon offers users flexibility in terms of tool types and sources, updating the underlying LLMs, adding new tools, and adapting to new tasks. Our work shares the same spirit of AutoGPT [47], an autonomous GPT-4 agent with the artificial general intelligence (AGI) ambition to incorporate numerous tools to achieve user-defined goals. While AutoGPT is still under development, our work is the first to instantiate the idea and verify its effectiveness on well-studied benchmarks.
3 General Framework: Chameleon
To address the limitations of current LLMs in utilizing diverse tools, we propose Chameleon, a novel plug-and-play compositional reasoning framework, synthesizing the composition of various tools to accommodate a wide range of problems. Chameleon is comprised of a module inventory that defines different types of tools and an LLM-based planner, whose purpose is to decompose the original problem into sub-tasks that can be effectively solved by task-specific tools. Unlike existing tool-augmented LLM approaches [49, 13, 59, 50], our module inventory features multiple tool types as illustrated in Table 2, enabling Chameleon to exhibit various reasoning abilities, including image understanding, knowledge retrieval, web search, complex mathematical reasoning, and table understanding. Instead of generating domain-specific programs [40, 13, 52], Chameleon employs an LLM-based planner to create natural-language-like programs that follow natural language instructions, which is less error-prone, easily expandable to new modules, and user-friendly.
We formalize our planner as follows: given the input query  $x_0$ , the module inventory  $\mathcal{M}$ , and constraints  $\mathcal{G}$ , the natural language planner  $\mathcal{P}$  selects a set of modules that can be executed sequentially to answer the query via generating a program in a natural-language-like format. The module inventory  $\mathcal{M}$  consists of a set of pre-built modules:  $\{M_i\}$ , each corresponding to a tool of various types (Table 2).  $\mathcal{G}$  are the constraints for the plan generation, for example, the concurrent relations and sequence
orders of modules. In our work, the planner  $\mathcal{P}$  is an LLM prompted to generate a sequence of module names in a few-shot setup. The planner is prompted in natural language with a planning task instruction  $\mathcal{I}$ , the descriptions of modules in  $\mathcal{M}$  with corresponding constraints  $\mathcal{G}$ , as well as a few demonstration examples  $\mathcal{D}$ . A  $T$ -length plan sampled from  $\mathcal{P}$  can be denoted as  $p = M^1, \ldots, M^T$ , where  $M^t$  represents the  $t$ -th element in the generated plan and  $M^t \in \mathcal{M}$ . Formally, given an input query (problem statement)  $x_0$ , a plan  $p$  is generated as follows:
$$
p \leftarrow \mathcal {P} \left(x _ {0}; \mathcal {I}, \mathcal {M}, \mathcal {G}, \mathcal {D}\right). \tag {1}
$$
Given the generated plan, the corresponding modules for each step are then executed sequentially. The plan is a natural-language program where each module is bound simply via string matching. When evaluating the module  $M^t$  at time step  $t$ , the output of the execution  $y^t$  is calculated by:
$$
y ^ {t} \leftarrow M ^ {t} \left(x ^ {t - 1}; c ^ {t - 1}\right), \tag {2}
$$
where  $x^{t - 1}$  is the input for the current module  $M^t$ , and  $c^{t - 1}$  is the cached information (e.g., image semantics, retrieved knowledge, generated programs) resulting from the execution history of modules. Both the problem input  $x^t$  and cache  $c^t$  for the next module  $M^{t + 1}$  are updated, respectively, by:
$$
x ^ {t} \leftarrow \text {u p d a t e \_ i n p u t} \left(x ^ {t - 1}, y ^ {t}\right), \tag {3}
$$
$$
c ^ {t} \leftarrow \operatorname {u p d a t e \_ c a c h e} \left(c ^ {t - 1}, y ^ {t}\right). \tag {4}
$$
The update_input and update_cache functions are hand-designed for each  $M_{i}$ . Specifically, update_input is applied to elements in the input query, including the question, table context, and image. These elements are updated after module execution. update_cache corresponds to the generation of new information, such as a description for the input image or retrieved knowledge from external resources. Finally, the response  $r$  to the query is generated by the last module  $M^{T}$ :
$$
r = y ^ {T} \leftarrow M ^ {T} \left(x ^ {T - 1}; c ^ {T - 1}\right). \tag {5}
$$
4 Applications of Chameleon
We demonstrate the applications of Chameleon on two challenging tasks: ScienceQA [32] (section 4.2) and TabMWP [33] (section 4.3), using the module inventory introduced in section 4.1. Further experimental details can be found in appendix A.2.
4.1 Module Inventory
To accommodate various reasoning capabilities over a diverse range of queries, our system utilizes a rich module inventory of various external tools. We provide a high-level overview of this inventory here, with detailed implementations in specific experiments. The complete module inventory,  $\mathcal{M}$ , is presented in Table 2. Each tool within the inventory is defined as follows:
Knowledge Retrieval: This module retrieves additional background knowledge crucial for tackling complex problems. It is especially beneficial for specialized domains like science and mathematics, providing context for the task. For example, could generate knowledge about tax procedures,
Bing Search: Like "Knowledge Retrieval", the "Bing Search" module aims to provide wide-ranging task-relevant knowledge. In contrast, it excels when broader or up-to-date information from multiple sources is required. Using the search engine API, this module returns relevant search results based on the input query, which are then parsed and used by subsequent modules to gather richer context information from diverse sources, enhancing problem-solving effectiveness.
Query Generator: Since the original problem typically lacks a tailored query for retrieving task-relevant information, this module creates search engine queries based on the problem, which are then
Tool Types	Tools
OpenAI	Knowledge Retrieval, Query Generator, Row Lookup, Column Lookup, Table Verbalizer, Program Generator, Solution Generator
Hugging Face	Image Captioner
Github	Text Detector
Web Search	Bing Search
Python	Program Verifier, Program Executor
Rule-based	Answer Generator
Table 2: Different tools in our module inventory.
e, if a query is about a tax form table, this module offering valuable context.
used by the "Bing Search" module. Mostly, it is a good strategy to use the "Query Generator" module before the "Bing Search". Coupled with the search engine tool, generating more targeted queries generally facilitates both the recall and precision of retrieved information.
Image Captioner: Designed to generate captions for images, this module provides crucial supplementary context for queries. It is particularly valuable when understanding an image semantically, like identifying objects and interactions in a scene. Using pre-trained models, it translates visual data into language, facilitating effective comprehension and reasoning about image content.
Text Detector: This module is designed to identify text within a given image. Typically, the "Text Detector" is employed when a question requires the extraction of textual information from images containing diagrams, charts, tables, maps, or other visual elements. By effectively detecting text in various formats, this module aids in the analysis and understanding of image-based content.
Row Lookup: This module is crucial when queries involve tabular context, as locating relevant cells is often required. Large tables can distract the system, so "Row Lookup" simplifies the table by retaining only the rows relevant to the query. If all rows are pertinent, it returns the original table.
Column Lookup: Like the "Row Lookup" module, "Column Lookup" addresses questions involving tabular context by focusing on relevant columns. It simplifies the table by retaining only pertinent columns, or returns the original table if all columns are relevant.
Table Verbalizer: Converting structured tables into text is likely to enhance the comprehension of tabular information by various downstream modules as shown by [37] for open-domain question answering, making this module a vital part of our system. It translates tables into easily understandable descriptions for modules like "Program Generator" and "Solution Generator", particularly useful for small, domain-specific tables like stem-and-leaf plots or function tables.
Program Generator: Program-aided approaches are shown to enhance the logical and mathematical reasoning abilities of LLMs [55, 10, 6, 39, 18, 43]. The "Program Generator" generates Python programs to solve queries effectively, which is particularly beneficial for queries requiring complex computations or intricate logical operations, such as "if-else" statements.
Program Verifier: Recent studies highlight the importance of verification to reduce hallucination [45, 38]. Hence, "Program Verifier" ensures the validity and error-free nature of programs generated by "Program Generator". It checks for syntax and logical errors, and potential execution issues, enhancing the reliability and accuracy of the solutions.
Program Executor: This module executes the program generated by "Program Generator" and produces the result, bridging the gap between program generation and final solution derivation.
Solution Generator: This module generates a detailed solution to the input query using all the cached information. Employing a chain-of-thought prompting approach [57], it ensures coherent and well-structured responses. The planner can directly employ this module instead of other functional modules if it can solve the query independently, especially for simpler ones.
Answer Generator: This task-specific module uses a rule-based approach to extract and normalize answers from the results of the "Program Executor" or "Solution Generator". Unlike the Solution Generator that provides detailed multi-step solutions, "Answer Generator" serves as the final module in the pipeline, providing concise and task-specific answers.
4.2 Science Question Answering