SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hongxin  $\mathrm{Li}^{*1,2}$ , Jingran  $\mathrm{Su}^{*3,4}$ , Yuntao Chen $^{\dagger 3}$ , Qing  $\mathrm{Li}^{\dagger 4}$ , and Zhaoxiang Zhang $^{\dagger 1,2,3,5}$
$^{1}$ School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)  
 $^{2}$ State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences  
 $^{3}$ Center for Artificial Intelligence and Robotics, HKISI, CAS  
 $^{4}$ The Hong Kong Polytechnic University  
 $^{5}$ Shanghai Artificial Intelligence Laboratory
Computer end users have spent billions of hours completing daily tasks like tabular data processing and project timeline scheduling. Most of these tasks are repetitive and error-prone, yet most end users lack the skill to automate these burdensome works. With the advent of large language models (LLMs), directing software with natural language user requests become a reachable goal. In this work, we propose a SheetCopilot agent that takes natural language task and control spreadsheet to fulfill the requirements. We propose a set of atomic actions as an abstraction of spreadsheet software functionalities. We further design a state machine-based task planning framework for LLMs to robustly interact with spreadsheets. We curate a representative dataset containing 221 spreadsheet control tasks and establish a fully automated evaluation pipeline for rigorously benchmarking the ability of LLMs in software control tasks. Our SheetCopilot correctly completes  $44.3\%$  of tasks for a single generation, outperforming the strong code generation baseline by a wide margin. Our project page: https://sheetcopilot.github.io/.
The ability to intuitively direct sophisticated software through natural language has long been an aspiration pursued across generations. With the emergence of Large Language Models (LLMs) that can comprehend and respond to human directives, this vision is within closer reach now than ever. LLMs augmented with tools [24, 25, 20, 26, 2, 27] and reasoning abilities [32, 18, 30, 31] have shown promising results in recent research.
While LLMs continue advancing rapidly, their ability to interoperate seamlessly with existing software tools remains under-explored and not fully understood. Enabling LLMs to harness the rich functionality of countless existing software tools could unlock nearly limitless potential [21].
Progress in endowing LLMs with the ability to direct complex software systems has been hindered by the lack of both a standardized framework for model-application interaction and a comprehensive benchmark for evaluating their performance. Several challenges exist in designing a framework to facilitate interaction between LLMs and sophisticated software applications, including 1) Translating the complex internal state and vast functionality of applications into text forms comprehensible for
* Equal contribution. E-mails: lihongxin2021@ia.ac.cn, jing-ran.su@connect.polyu.hk
† Equally advising corresponding authors. E-mails: zhaoxiang.zhang@ia.ac.cn, csqli@comp.polyu.edu.hk, chenyuntao08@gmail.com
models [32]. This requires determining how to systematically represent software interfaces and logic through natural language; 2) Enabling models to generate software commands and parameters accurately and safely [9, 2]. Mechanisms must exist for validating, debugging, and as needed, rejecting or revising model outputs to avoid undesirable operations or states; 3) Providing models with means of monitoring software state changes, exceptions, and errors during multi-step tasks so that these models can respond appropriately. Models are required to understand software feedback and diagnose issues, and adjust directives as needed to robustly accomplish goals. In addition, enabling LLMs to direct complex software also requires curating datasets that capture the diversity and ambiguity of real-world language use, as well as developing automated techniques to reliably evaluate model performance at scale [7].
To systematically investigate the substantial challenges in developing natural language interfaces for software control, a robust application platform is required; as the most pervasive and multifunctional productivity tool, the spreadsheet serves as an ideal substrate for this work. To this end, we propose a general framework for facilitating interaction between language models (LMs) and software applications, along with an agent called SheetCopilot. As shown in Fig. 1 SheetCopilot understands high-level spreadsheet manipulation requests expressed in natural language. It decomposes these complex requests into step-by-step plans, and issues commands to automatically carry out the necessary operations using the spreadsheet application. In addition to our spreadsheet-manipulating agent, SheetCopilot, we propose a dataset consisting of complex, interactive spreadsheet manipulation requests expressed through natural language and an evaluation framework with automated metrics to assess how accurately models comprehend requests, devise optimal plans, and perform operations through the spreadsheet interface. We believe robust measurement is key to accelerating progress in this area.
Our agent SheetCopilot achieved substantial capabilities for guiding spreadsheet software through natural language. It generated fully executable command sequences for  $87.3\%$  of problems in our benchmark suite and produced completely correct solutions for over  $44.3\%$  of tasks—surpassing the traditional programming approaches by a wide margin. To rigorously assess model performance, we curated a dataset of 221 representative spreadsheet tasks collected from superuser.com, including verified solutions created by the authors for each task.
We present three primary contributions to the goal of achieving sophisticated interfaces between language models and traditional software applications:
- We proposed a general framework for facilitating model-software interaction along with SheetCopilot, an agent specialized for spreadsheets that translates high-level, task-oriented requests expressed in natural language into executable command sequences.
- We developed comprehensive resources for systematically evaluating model and interface performance, including a benchmark suite of interactive spreadsheet tasks reflecting real-world requests and a fully automated pipeline for measuring how accurately models comprehend complex prompts, devise optimal plans and execute operations through the software interface.
- We conducted an in-depth assessment benchmarking the abilities of leading LLMs in this challenging domain. The experiments show that LLMs equipped with our method significantly outperform the strong code generation baseline.
Tool-augmented Large Language Models Recently, the impressive performance of LLMs has sparked significant interest in exploring the task-planning capabilities of LLMs in various fields. Benefitting from the internalized knowledge about the world [1], a number of works have managed to enable LLMs to solve tasks by following instructions. One line of research [14, 3, 16, 15, 8] has utilized prompt engineering to elicit a sequence of mid-level steps from LLMs for household robotic tasks. To ground LLMs in the real world, these works have used auxiliary models [3, 16, 15] or trained LLMs via mixing visual-language data and embodied data [8]. Another promising direction is to connect LLMs with external tools [25], such as a web browser [22], HuggingFace model hub [26], chemical software [2], PowerPoint [20], and even a tool library [25, 24]. These works employ LLMs to generate action sequences which are further parsed into API calls of the tools. Compared with
Figure 1: We maneuver SheetCopilot to control software such as Microsoft Excel, generate step-by-step solutions fulfilling the user's requirements. In each step, SheetCopilot plans an initial atomic action according to the sheet state and then revises this step using the external document which provides the action usage and examples. Finally, the action with its arguments is extracted from the revised step and submitted to the simulation environment for execution. The entire process on the right shows that SheetCopilot successfully solves the task specified in the instruction using the provided available atomic actions.
these works, our work is targeted at spreadsheet manipulation, which is a common demand in almost all scenarios (e.g. economics, management, and research). To the best of our knowledge, limited research has been conducted on benchmarking the capability of LLMs for spreadsheet manipulation.
Natural Language Processing (NLP) for Spreadsheets Several studies [12, 11, 28, 6, 17] have investigated the feasibility of using NLP methods to guide the manipulation of Excel sheets. An early work is Flash Fill [11], which automates string processing tasks using program synthesis by example. NLyze [12] utilizes a translation algorithm to convert a user's natural language instruction to a ranked set of likely programs. Inspired by the success of Codex [5] and AlphaCode [19], one recent study [17] has explored the use of LLMs for generating Excel formulas given natural language descriptions of the desired output. They compared the performance of several state-of-the-art LLMs, including GPT-3 and T5, and found that these models can generate accurate formulas with high efficiency. However, this study focused on formula generation rather than general sheet control tasks. In this paper, we aim to address this gap by benchmarking the capability of LLMs for sheet control tasks.
3 Dataset and Evaluation
Prior research on language interfaces for spreadsheet control [12, 6, 17] has focused primarily on limited subsets of tasks like formula generation and lacked comprehensive, standardized means of evaluation. To address this issue, we aim to construct a high-quality evaluation benchmark as a foundation for assessing the spreadsheet control capabilities of LLM-based agents.
Our dataset compilation procedure incorporates gathering tasks and worksheets from the Internet, filtering low-quality or irrelevant tasks, consolidating redundant entries, adapting seed tasks, and manually annotating a core set. The end product is a comprehensive and cleanly-labeled collection of spreadsheet-related tasks. We also report statistics and analysis to characterize the dataset properties,
Figure 2: Dataset overview. We present an overview of the core set by showing the wordclouds of the instructions and involved atomic actions. The two clouds show that the core set contains diverse tasks that involve various spreadsheet operations.
guide future work, and set initial baselines. Moreover, we develop an automated, reproducible evaluation framework closely tailored to our curated natural language spreadsheet control dataset. This enables systematically assessing model abilities, gaining insights into current limitations, and driving continued progress in this domain.
3.1 Diverse Seed Task and Workbench Collection
We first scrape all questions with spreadsheet-related tags on www . superuser . com and obtain a raw dataset comprising  $\sim 16k$  question and answer (Q&A) pairs. Sourcing questions from SuperUser ensures our task dataset is both comprehensive and representative. As not every question represents a sheet manipulation task we apply keyword-based and LLM-based filters to remove Q&A pairs unrelated to spreadsheet automation, resulting in a remain of  $\sim 13k$  pairs. To analyze the distribution of the dataset, we define six task categories: Entry and Manipulation, Management, Formatting, Charts, Pivot Tables, and Formulas. We label exemplar Q&A pairs with at least one category to prompt the language model to categorize each pair, as pairs may belong to multiple categories. To identify representative Q&A pairs, we embed and cluster pairs within each unique category combination. We then choose 67 pairs representing the clustering centers and involving operations supported by our evaluation environment. The spreadsheet tasks described in these pairs are regarded as the seed tasks which capture the most important patterns of our dataset.
To evaluate LLMs, we also collect 28 real-world spreadsheets as our workbench by 1) downloading practical sheets from the Internet, and 2) Generating typical daily-use sheets by hand. These sheets represent common uses such as analyzing sales data, calculating financial metrics, and visualizing data with charts.
3.2 Core Set Collection
The seed tasks cannot be directly used since their original sheets differ from the evaluation sheets. We propose collecting a core dataset by adapting and simplifying the seed tasks to bridge this gap.
Adaptation. Inspired by self-instruct [29], we prompt an LLM to adapt the seed tasks according to the detailed descriptions of the evaluation sheets. Specifically, GPT-4 is prompted to change the manipulated elements in the seed tasks to generate new tasks compatible with the evaluation sheets. For instance, GPT-4 can change the data types required to be set or ranges to be modified in the original seed task. In this step, 1669 new task instructions are generated (See Tab. D for examples).
Simplification. The adaptations are likely to mention specific formulas and operations. To address this issue, we prompt an LLM to simplify each task by replacing specific expressions with natural spoken language so that the task instruction reads like the fashion of a non-expert user. This step reduces the average token length from 47.1 to  $33.8^{1}$ .
<sup>1</sup>We followed the instruction on https://github.com/openai/tiktoken to count the token number using the model "gpt-3.5-turbo".
To collect a core set, the authors select several tasks for each category combination from the simplified tasks. The authors also act as non-expert users to compose more tasks to enrich the core set, obtaining 221 tasks in total. Finally, multiple reference solutions are collected as the ground truth for each task. See Fig. 2 for the dataset statistics and more in the appendix.
3.3 Task Evaluation by Execution
It is hard to evaluate solutions generated by LLMs through verbatim comparison, as it is likely that multiple solutions can successfully complete a task. A viable approach is assessing whether the final sheet state after executing the solution meets the task instruction. We only assess the necessary properties required for the ground truth spreadsheet's operation. For example, in the task "Plot a line chart with the X-axis showing the week and the Y-axis showing sales", we only consider properties related to the chart, ignoring other aspects. To assess an LLM-generated solution, we evaluate the consistency of the necessary properties between the spreadsheet resulting from executing this solution and the ground truth spreadsheet in our evaluation environment. If the necessary properties of the resulting spreadsheet fully match any potential ground truth candidate, the associated solution is deemed correct.
SheetCopilot enables natural language interactions with spreadsheets. It takes spreadsheets and user tasks described in natural language as input and generates plans to modify spreadsheet contents and create diagrams or tables. We adopt an in-context learning framework inspired by models such as GPT-3 [4]. We propose "atomic actions" - a set of virtual APIs representing common spreadsheet functions. These actions allow language models to accurately interact with the spreadsheet software. We also propose a state machine-based task planning framework to handle the complex, multi-turn interaction between the language models and the spreadsheets. The atomic actions and state machine-based planning framework enable language models to effectively and robustly direct spreadsheet software through natural language.
4.1 Prompting LMs as a SheetCopilot
We design a systematic prompt template to turn LMs into copilots as shown in the left of Fig. 1. Our prompt consists of a general role description, a list of atomic actions with arguments, a set of output requirements, and a multi-round interaction example between a user and an assistant.
The general role description serves as an anchor for enabling LMs to understand the context. A list of atomic actions provides LMs with the interface information needed for task planning. The output requirement tells LMs how to generate texts that can be programmatically extracted and executed. The multi-round example hints LMs how the observe-propose-revise-act loop appears and improves the overall planning quality.
4.2 Atomic Action as A Bridge for LMs and Software
State-of-the-art LMs have shown the superb ability to generate detailed plans for household tasks [16], software control [20], and debugging [5]. However, the generated plans are in natural language which is easy for humans to read but not directly admissible for machine execution.
To overcome the limitation mentioned above, we propose to model the functionalities of existing spreadsheet software as a set of virtual APIs called atomic actions. An atomic action is comprised of an API name, a typed argument list, a usage document string, and several usage examples. These atomic actions can be implemented on different spreadsheet platforms. The example implementations in Tab. H of the appendix show that the atomic actions involve cell value modification, formatting, sheet management, formula and functions, charts, and pivot tables.
Choosing proper atomic action granularity is crucial, as actions must be expressive yet concise to fit in the LM context windows. We determine our atomic actions as follows: 1) Extract all actions involved in the top SuperUser spreadsheet Q&As; 2) Embed and cluster the extracted actions into candidates; 3) Select the minimum set of actions covering all the tasks we collected in Sec. 3.1.
Relation to Agents Generating VBA Codes LMs are also capable of generating machine-readable codes [5]. This approach is especially tempting for Microsoft Excel as it comes with an embedded script language called Visual Basic for Applications(VBA). However, the code generation approach faces challenges from both the LMs side and the spreadsheet software side. On the code LMs side, the existing training corpus [10, 13, 5] for code LMs hardly contains VBA source files as it is only a niche programming language compared with  $\mathrm{C + + }$  or Python. On the spreadsheet software side, software such as Google Sheets, Libre Office, and WPS either do not support VBA at all (Google Sheets) or only support a limited subset of VBA functions (Libre Office and WPS). Therefore, we advocate a more software-agnostic approach that does not rely on embedded programming language support.
4.3 State Machine-based Task Planning
A normal spreadsheet task usually involves several steps, while a sophisticated one often requires over ten steps. Open-loop planning - directly generating a complete task plan from the instruction - becomes exponentially harder as steps increase. Each step changes the sheet state so the correct step  $T + 1$  relies on perfectly understanding how the sheet state changes after the previous  $T$  steps. As tasks become more complex, open-loop planning struggles.
We propose a state machine-based planner which revises the plan according to feedback from either LMs or software. Our planner is divided into observing, proposing, revising, and acting stages. The state transition between these stages will be described below. Due to the page limit, please refer to the appendix for examples of complete planning logs.
Observing Stage In this stage, we add a brief description of the sheet state  $S_{t}$  to the query, providing information such as the name of each column and the total number of rows for LMs to determine atomic action arguments. This allows LMs to generate solutions in a closed-loop manner by observing the previous actions' consequences without implicitly modeling sheet states.
Proposing Stage In this stage, we concatenate the system prompt  $P$ , the initial task instruction  $I$ , the sheet state  $S_{t}$  and the planning history  $H_{t}$  and ask the LMs to plan the next atomic action  $A_{t + 1}$ .
$$
A _ {t + 1} = \operatorname {V a l i d a t e} \left(R _ {t + 1}\right) = \operatorname {V a l i d a t e} \left(\text {L a n g u a g e M o d e l} \left(P, I, S _ {t}, H _ {t}\right)\right). \tag {1}
$$
The direct response  $R_{t + 1}$  from the language model is not always convertible to an admissible atomic action  $A_{t + 1}$ . Common errors found in the validating step include missing the format requirement, hallucinating undefined actions, and incorrectly determining action parameters.
Revising Stage Two ways are adopted to revise a proposed atomic action: a feedback-based one and a retrieval-based one. Feedback-based revision utilizes the error feedback from both the atomic action validation and the spreadsheet software execution. For example, if the atomic action validating step detects a hallucinated atomic action, a new prompt will be created to inform the LM of this error and to ask it to reiterate the available atomic actions. Additionally, we use retrieval-based revision to supply the LM with detailed external knowledge that does not fit in the system prompt due to the context window limit. For example, if the LM uses an atomic action with wrong arguments, a detailed document containing the argument descriptions and usage examples of this action is provided in the new prompt to enhance the probability of the LM correctly determining the atomic action arguments. This process resembles how a human programmer behaves when encountering less familiar APIs.
A special case in the revision stage is that after being supplied with more information about the initially proposed atomic action, the LM suddenly finds that it has chosen a wrong action and decides to return to the revising stage.
Acting Stage After the proposing and revising stages, the atomic action  $A_{t + 1}$  is submitted to the spreadsheet software for execution.
$$
S _ {t + 1} = \operatorname {S p r e a d S h e e t E n v} \left(A _ {t + 1}, S _ {t}\right). \tag {2}
$$
The planning history  $H_{t}$  is updated if the execution succeeds,
$$
H _ {t + 1} = H _ {t} \cup \left\{A _ {t + 1}, S _ {t + 1} \right\}. \tag {3}
$$
If the software reports a run-time error, the state machine will return to the proposing stage to prompt the LM to re-plan according to the error feedback.
Table 1: Performances of the compared LLMs and a VBA-based method. The three LLMs exhibit impressive Exec@1 and Pass@1, with GPT-3.5-Turbo achieving the highest Exec@1 and GPT-4 obtaining the best Pass@1 and efficiency. With our method, GPT-3.5-Turbo outperforms the method that generates and runs VBA code by a large margin.
Data	Models	Exec@1↑	Pass@1↑	A50↓	A90↓
10%	GPT-3.5-Turbo	85.0%	45.0%	2.00	4.50
10%	GPT-4	65.0%	55.0%	1.33	2.00
10%	Claude	80.0%	40.0%	1.50	4.40
100%	GPT-3.5-Turbo	87.3%	44.3%	1.50	3.00
100%	VBA	77.8%	16.3%	-	-
4.4 Hallucination Mitigation
To enable the state machine to less frequently return to the proposing stage due to hallucination-induced errors, we adopt the following means.
Output Formatting The underlying functions of atomic actions require precisely formatted planning results. However, we found that LMs probably generate semantically correct yet inadmissible action plans as shown in Fig. 1. Therefore, we require LMs to wrap actions with special tokens (e.g. @) and detect the tokens in the output to check whether the output is correctly formatted.
Atomic Action Disambiguation The internalized knowledge in LMs is likely to be confused with the atomic action definitions in the document. Due to this conflict, LMs are prone to self-delusion, which means that it hallucinates undefined actions or adds illegal action arguments [23, 14]. To tackle this problem, the atomic action names are substituted with a set of synonyms that are far away from the official names in an embedding space. For instance, Write and SetConditionalFormat are substituted with RangeInputValue and FormatWithRules, respectively (See the details in the appendix).