SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Models
Authors: Hongxin Li
Jingran Su
Yuntao Chen
Qing Li
Zhaoxiang Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: Large Language Model; Task Planning; Embodied AI; Robotics; Software Automation; Decision making
TL;DR: We propose an LLM-based autonomous agent that manipulates complex software by following natural language instructions.

Abstract: Computer end users have spent billions of hours completing daily tasks like tabular data processing and project timeline scheduling. Most of these tasks are repetitive and error-prone, yet most end users lack the skill to automate these burdensome works. With the advent of large language models (LLMs), directing software with natural language user requests become a reachable goal. In this work, we propose a SheetCopilot agent that takes natural language task and control spreadsheet to fulfill the requirements. We propose a set of atomic actions as an abstraction of spreadsheet software functionalities. We further design a state machine-based task planning framework for LLMs to robustly interact with spreadsheets. We curate a representative dataset containing 221 spreadsheet control tasks and establish a fully automated evaluation pipeline for rigorously benchmarking the ability of LLMs in software control tasks. Our SheetCopilot correctly completes 44.3\% of tasks for a single generation, outperforming the strong code generation baseline by a wide margin. Our project page: https://sheetcopilot.github.io/.

SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Models
Hongxin Li Jingran Su Yuntao Chent3, Qing Lit4, and Zhaoxiang 7hangt1.235 1.2.3.5
"School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)
State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences
Center for Artificial Intelligence and Roboties, HKISI, CAS
'The Hong Kong Polytechnic University Shanghai Artificial Intelligence Laboratory
Abstract
Computer end users have spent billions of hours completing daily tasks like tabular data processing and project timeline scheduling. Most of these tasks are repetitive and error-prone, yet most end users lack the skill to automate these burdensome works. With the advent of large language models (LLMs), directing software with natural language user requests become a reachable goal. In this work, we propose SheetCopilot agent that takes natural language task and control spreadsheet fulfill the requirements. We propose set of atomic actions as an abstraction spreadsheet software functionalities. We further design state machine- based task planning framework for LLMs to robustly interact with spreadsheets. We curate a representative dataset containing 221 spreadsheet control tasks and establish fully automated evaluation pipeline for rigorously henchmarking the ability LLMS software control tasks. Our SheetCopilot correctly completes 44.3% of tasks for a single generation, outperforming the strong code generation baseline by a wide margin. Our project page: hutps://sheetcopiioooghhiuu githubbbio
Introduction
The ability to intuitively direct sophisticated software through natural language has long been an aspiration pursued across generations. With the emergence of Large Language Models (LLMs) that can comprehend and respond to human directives, this vision is within closer reach now than ever.
LLMs augmented with tools [24] 25 20 26
27 and reasoning abilities 32
30
have
shown promising results in recent research.
While LLMs continue advancing rapidly, their ability to interoperate seamlessly with existing software tools remains under-explored and not fully understood. Enabling LLMs to harness the rich functionality of countless existing software tools could unlock nearly limitless potential (21).
Progress endowing LLMs with the ability to direct complex software systems has been hindered by the lack of both a standardized framework for model-application interaction and comprehensive benchmark for evaluating their performance. Several challenges exist in designing framework to facilitate interaction between LLMs and sophisticated software applications, including 1) Translating complex internal state and vast functionality of applications into text forms comprehensible for
Equal contribution. E-mails: lihongxin2021 @ia.ac.cn. jing-raa ran.sura.eee.......... uun.su@@oonenn connect.ooooo...bb Equally advising corresponding authors. E-mails: zhaoxiang zhang ac, esqli @ comp. pol..euoy... chenyuntao08 @ @gmail.com
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

This requires determining how to systematically represent software interfaces and logic through natural language; 2) Enabling models generate software commands and parameters accurately and safely 19 2. Mechanisms must exist for validating. debugging, and as needed, rejecting or revising model outputs to avoid undesirable operations or states; 3) Providing models with means of monitoring software state changes, exceptions, and errors during multi-step tasks so that these models can respond appropriately. Models are required to understand software feedback and diagnose issues, and adjust directives as needed to robustly accomplish goals. In addition, enabling LLMs to direct complex software also requires curating datasets that capture the diversity and ambiguity of real-world language use, well as developing automated techniques to reliably evaluate model performance at scale 77].
models [32]
To systematically investigate the substantial challenges in developing natural language interfaces for software control, a robust application platform is required; as the most pervasive and multi- functional productivity tool, the spreadsheet serves an ideal substrate for this work. To this end, we propose a general framework for facilitating interaction between language models (LMs) and software applications, along with an agent called SheetCopilot. As shown in Fig. SheetCopilot understands high-level spreadshect manipulation requests expressed in natural language. It decomposes these complex requests into step-by-step plans, and issues commands to automatically carry out the necessary operations using the spreadsheet application. In addition to our grreaddhhet-maniiuleeee agent, SheerCopilot, propose dataset consisting of complex, interactive spreadsheet manipulation requests expressed through natural language and an evaluation framework with automated metrics to assess how accurately models comprehend requests, devise optimal plans, and perform operations through the spreadsheet interface. We believe robust measurement is key to accelerating progress in this area.
Our agent SheetCopilot achieved substantial capabilities for guiding spreadsheet software through natural language. It generated fully executable command sequences for 57.3% of problems in our benchmark suite and produced completely correct solutions for over 44.3% of tarks-surpassing the traditional programming approaches by wide margin. To rigorously assess model performance, we curated a dataset of 221 representative spreadsheet tasks collected from superuser. com, including verified solutions created by the authors for each task.
We present three primary contributions to the goal of achieving sophisticated interfaces between language models and traditional software applications:
We proposed a general framework for facilitating model-software interaction along with SheetCopilot, an agent specialized for spreadsheets that translates high-level, task-oriented requests expressed in natural language into executable command sequences.
We developed comprehensive resources for systematically evaluating model and inter- face performance, including benchmark suite of interactive spreadsheet tasks reflecting real-world requests and fully automated pipeline for measuring how accurately models comprehend complex prompts, devise optimal plans and execute operations through the software interface.
We conducted an in-depth assessment benchmarking the abilities of leading LLMs in this challenging domain. The experiments show that LLMs equipped with our method significantly outperform the strong code generation bascline.
2 Related Works
Tool-augmented Large Language Models Recently, the impressive performance of LLMs has sparked significant interest in exploring the task pas--ilannin capabilities of LLMs in various fields. Benefitting from the internalized knowledge about the world M. number of works have managed
to enable LLMs to solve tasks by following instructions. One line of research [14] 16 15 g has utilized prompt engineering to elicit a sequence of mid-level steps from LLMs for household robotic
tasks. To ground LLMs in the real world, these works have used auxiliary models 3
16 [ or
trained LLMs via mixing visual-language data and embodied data g
Another promising direction is
to connect L.MSs with external tools 25 such as a web browser 22
HuggingFace model hub (26..
chemical software 22, PowerPoint (20,, and even a tool library 25 24 These works employ LLMSs
to generate action sequences which are further parsed into API calls of the tools. Compared with

---

Enstrucion: Please highlight Soles between 200 and 500.
#1 Progeiin Aroge
sy denning Strge
Pepereth
Torget Spreadsheet
1506 Dne
12:.0
0
APS decamast bused us the langnie aattast
Gopp fusted
made diferest
records mory tnalcat
Contuct: wrrdbak
104
Hars the AFE decement
Doarcer shf, ReleDhdees let, otherio iny?
Aieet
strea des socal femeriee del thuge- Asto
Falilades (PT) The of the fild to thu by. startig free
range
Fall Piin and Colcrtio
Stop Piter orlerts "vxtto" "'4977
decitldd aocument
heaters new rambers of sheets
Aocte procde shyy -tey sakttenn with caplone heme
Aancfii
by hould onl nature one step in time and each ply indid
thit the comrre belwosk the heo & cheractirs on faichiy coll.
Detubel Joc)
Fie
there oddeec subple shorts in nuribai
swee der
Reeiaed Paan
Seap the
(Ftrrd calls
shect. You comor are other varidiin
Fite
serructan:: calune
with Ancweting proft much Then
redii ceblorge
1300
Seffrr
Aslion Jel: @hintapwogen "houttbe"" value (Prularp
Piterpaacce saat FIDr Roldtdeendd
Cloeminy Strge
Sheat strte Sharl "Meett" caloees (riqaters are A There Tioesel)) mon E1
Eacare ganertee acrice
( y
rom))
Orvinamert
Rue
1999
010000
1
03:10
Solect calle iale berwere 100 ABC Ieare f
500
Figure 1: We maneuver SheetCopilot to control software such as Microsoft Excel, generate step-by by- step solutions fulfilling the user's 's requirements. In each step, SheetCopilot plans an initial atomic action according to the sheet state and then revises this step using the external document which provides the action usage and examples. Finally, the action with its arguments is extracted from the revised step and submitted to the simulation environment for execution. The entire process on the right shows that SheetCopilot successfully solves the task specified in the instruction using the provided available atomic actions.
these works, our work targeted at spreadsheet manipulation, which is common demand in almost all scenarios (e.g. economics, management, and research). To the best of our knowledge, limited research has been conducted on benchmarking the capability of LLMs for spreadsheet manipulation.
17 have investigated the feasibility of using NLP methods to guide the manipulation of Excel sheets. An early work is Flash Fill [II, which automates string processing tasks using program synthesis by example. NLyze 112 utilizes translation algorithm to convert a user's natural language instruction set of likely programs. Inspired by the success of Codex (5] and A AlphaCode [19], one recent study 17 has explored the use of L.MMs for generating Excel formulas given natural language descriptions of the desired output. They compared the performance of several state-of-the-ar L.MMs, including GPT-3 and T5, and found that these models can generate accurate formulas with high efficiency. However, this study focused on formula generation rather than general sheet control tasks. In this paper, we aim address this gap by benchmarking the capability of LLMs for sheet control tasks.
Natural Language Processing (NLP) for Spreadsheets Several studies 12
28
ranked
3 Dataset and Evaluation
Prior research on language interfaces for spreadsheet control 12 6. 17 limited subsets of tasks like formula generation and lacked comprehensive, standardized means of evaluation. To address this issue, we aim to construct a high-quality evaluation benchmark as a foundation for assessing the spreadsheet control capabilities of LLM--based agents.
has focused primarily on
Our dataset compilation procedure incorporates gathering tasks and worksheets from the Internet, filtering low-quality or irrelevant tasks, consolidating redundant entries, adapting seed tasks, and manually annotating a core set. The end product a comprehensive and cleanly-labeled collection of spreadshhet-relltedd tasks. We also report statistics and analysis to characterize the datasct properties,
3

---

Plot
Row
Type
Use
Cell
Week
Sun
Date
Teu
Value
Result
Vame
Title
Min
Not
Out
Fill
Line
New
I
Red Revenure
Sale
uprn
Proft
Label
Axis
Rate
Text
Year
Type
Axis
Title
Series
Add
Freeo
Cell
Chart
Logical
Update
Vale
Set
Text
reate
Date
Insert
Table Lookup Char
Pivot
Tabel
Line
ell
Paste
Math
Autofill
Value
Sort
Copy
Set
Sheet
Header
Show
hart
Deplay
Set
Functions
Candisional Shore
Format
Filter
Dataing
Werte
Create
Eolor
Color
Calculate
Product
Legend
Stazenical
Data
Sheet
Befeeence
Strgge
Uydar
Wordeloud the instructions
Wordeloud of the atomic actions
Figure 2: Dataset overview. We present an overview of the core set by showing the wordclouds of the instructions and involved atomic actions. The two clouds show that the core set contains diverse tasks that involve various spreadsheet operations.
guide future work, and set initial baselines. Moreover, develop an automated, reproducible evaluation framework closely tailored to our curated natural language spreadsheet control dataset. This enables systematically assessing model abilities, gaining insights into current limitations, and driving continued progress in this domain.
3.1 Diverse Seed Task and Workbench Collection
We first scrape all questions with spreadsheet-related tags on www. superuser com and obtain raw dataset comprising 166 question and answer (Q&A) pairs. Sourcing questions from SuperUser ensures our task dataset is both comprehensive and representative. As not every question represents a sheet manipulation task we apply keyword- and LLM-based filters to remove Q&A pairs unrelated to spreadsheet automation, resulting in remain of ~ 13k pairs. To analyze the distribution of the dataset, we define six task categories: Entry and Manipulation, Management, Formatting, Charts, Pivot Tables, and Formulas. We label exemplar Q&A pairs with at least category to prompt the language model to categorize each pair, as pairs may belong to multiple categories. To identify representative Q&A pairs, embed and cluster pairs within each unique category combination. We then choose 67 pairs representing the clustering centers and involving operations supported by our evaluation environment. The spreadsheet tasks described in these pairs are regarded as the seed tasks which capture the most important patterns of our dataset.
To evaluate LLMs, also collect 28 real-world spreadshects as workbench by 1) downloading practical sheets from the Internet, and 2) Generating typical daily-use sheets by hand. These sheets represent common uses such as analyzing sales data, calculating financial metrics, and visualizing data with charts.
3.2
Core Set Collection
The seed tasks cannot be directly used since their original sheets differ from the evaluation sheets. We propose collecting core dataset by adapting and simplifying the seed tasks to bridge this gap.
Adaptation. Inspired by self-instruct [29]. we prompt an LLM to adapt the seed tasks according to the detailed descriptions of the evaluation sheets. Specifically, GPT-4 is prompted to change the manipulated elements in the seed tasks to generate new tasks compatible with the evaluation sheets. For instance, GPT-4 can change the data types required to be set or ranges to be modified in the original seed task. In this step, 1669 new task instructions are generated (See Tab. D for examples).
The adaptations are likely to mention specific formulas and operations. To address this issue, we prompt an L.M to simplify each task by replacing specific expressions with natural spoken language so that the task instruction reads like the fashion of non-expert user. This step reduces the average token length from 47.1 to 33.8
Simplification.
'ee followed the instruction on http://gittub.ccmmopeemi/iiiiiit to count the token number using the model -gpttb..

---

To collect a core set, the authors select several tasks for each category combination from the simplified tasks. The authors also act as non-expert users compose more tasks to enrich the core set, obtaining 221 tasks in total. Finally, multiple reference solutions are collected as the ground truth for each task. See Fig. for the dataset statistics and more in the appendix.
3.3 Task Evaluation by Execution
is hard to evaluate solutions generated by LLMs through verbatim comparison, as it likely that multiple solutions successfully complete task. A viable approach is assessing whether the final sheet state after executing the solution meets the task instruction. We only assess the necessary properties required for the ground truth spreadsheet's operation. For example, in the task "Plot a line chart with the X-axis showing the week and the Y-axis showing sales", , we only consider properties related the chart, ignoring other aspects. To assess an LLM-generated solution, we evaluate the consistency of the necessary properties between the spreadsheet resulting from executing this solution and the ground truth spreadsheet in our evaluation environment. If the necessary properties of the resulting spreadsheet fully match any potential ground truth candidate, the associated solution is deemed correct.
4 Method
SheetCopilot enables natural language interactions with spreadsheets. It takes spreadsheets and user tasks described in natural language as input and generates plans to modify spreadsheet contents and create diagrams tables. We adopt an in-context learning framework inspired by models such as GPT-3 [4. We propose "atomic actions" a set of virtual APIs representing common spreadsheet functions. These actions allow language models to accurately interact with the spreadsheet software. We also propose a state machine-bass task planning framework to handle the complex, multi-turn interaction between the language models and the spreadsheets. The atomic actions and state machine- based planning framework enable language models to effectively and robustly direct spreadsheet software through natural language.
4.1 Prompting LMs as a ShectCopilot
We design a systematic prompt template to turn LMs into copilots as shown in the left of Fig. | Our prompt consists of general role description, a list of atomic actions with arguments, set of output requirements, and a multi-round interaction example between a user and an assistant.
The general role description serves as an anchor for enabling LMs to understand the context. A list of atomic actions provides LMs with the interface information needed for task planning. The output requirement tells LMs how to generate texts that can be programmatically extracted and executed. The multi-round example hints LMs how the observe- the overall planning quality.
cesservepropose-reee propose- revise--ct loop appears and improves
4.2 Atomic Action as A Bridge for LMs and Software
State-of-the-and LMs have shown the superb ability to generate detailed plans for household tasks [6], software control (20). and debugging (5). However, the generated plans are in natural language which is easy for humans to read but directly admissible for machine execution.
To overcome the limitation mentioned above, we propose to model the functionalities of existing spreadshect software as a set of virtual APIs called atomic actions. An atomic action is comprised of an API name, a typed argument list, usage document string, and several usage examples. These atomic actions can be implemented on different spreadsheet platforms. The example implementations in Tab, H of the appendix show that the atomic actions involve cell value modification, formatting, sheet management, formula and functions, charts, and pivot tables.
Choosing proper atomie action granularity is crucial, as actions must be expressive yet concise to fit in the LM context windows. We determine our atomic actions as follows: 1) Extract all actions involved in the top SuperUser spreadsheet Q& 2) Embed and cluster the extracted actions into candidates; 3) Select the minimum set of actions covering all the tasks collected in Sec. 3.1
5

---

Relation to Agents Generating VBA Codes LMs are also capable of generating
codes [5]. This approach is especially tempting for Microsoft Excel it comes with an embedded script language called Visual Basic for Applications VBA). However, the code generation approach faces challenges from both the LMs side and the spreadsheet software side. On the code LMs side, the existing training corpus [T] 13 5 for code LMs hardly contains VBA source files as it is only a niche programming language compared with C++ or Python. On the spreadsheet software side, software such as Google Sheets, Libre Office, and WPS either do not support VBA all (Google Sheets) or only support limited subset of VBA functions (Libre Office and WPS). Therefore, we advocate more software-agnostie agnostie approach that does not rely embedded programming language support.
4.3 State Machine-based Task Planning
normal spreadsheet task usually involves several steps, while a sophisticated one often requires over ten steps. Open-loop planning directly generating a complete task plan from the instruction becomes exponentially harder as steps increase. Each step changes the sheet state so the correct step T 4+1 1 relies on perfectly understanding how the sheet state changes after the previous T steps. As tasks become more complex, open-loop planning struggles.
We propose a state machine-based planner which revises the plan according to feedback from either LMs or software. Our planner is divided into observing, proposing, revising, and acting stages. The state transition between these stages will be described below. Due to the page limit, please refer to the appendix for examples of complete planning
logs.
Observing Stage In this stage, we add a brief description of the sheet state St to the query. providing information such as the name of each column and the total number of rows for LMs to determine atomic action arguments. This allows LMs to generate solutions in a closed-loop manner by observing the previous actions' consequences without implicitly modeling sheet states.
Proposing Stage In this stage, we concatenate the system prompt P, the initial task instruction the sheet state St and the planning history H,, and ask the LMs to plan the next atomic action A++1-
A++1 Validate(R(+1))
Vaiddte(LLnggggMMood(((....... H.))
The direct response Re+1 from the language model is not always convertible an admissible atomic action A441- Common errors found in the validating step include missing the format requirement, hallucinating undefined actions, and incorrectly determining action parameters.
Revising Stage Two ways are adopted to revise a proposed atomic action: a feedback- based one and retrieval-based one. Feedback- based revision utilizes the error feedback from both the atomic action validation and the spreadsheet software execution. For example, if the atomic action validating step detects ballucinated atomic action, new prompt will be created to inform the LM of this error and to ask it to reiterate the available atomic actions. Additionally, we use retrieval-based revision to supply the LM with detailed external knowledge that does not fit in the system prompt due to the context window limit. For example, if the LM uses an atomic action with wrong arguments, a detailed document containing the argument descriptions and usage examples of this action is provided in the new prompt to enhance the probability of the LM correctly determining the atomic action arguments. This process resembles how a human programmer behaves when encountering less familiar APIs.
special case in the revision stage that after being supplied with more information about the initially proposed atomic action, the LM suddenly finds that it has chosen a wrong action and decides to return to the revising stage.
Acting Stage After the proposing and revising stages, the atomic action A is submitted to the spreadsheet software for execution.
S44 SpreadSheclEnv(Ar4.... S.).
(2)
The planning history H, is updated if the execution succeeds,
H = H, U (A+11, S.+1)
If the software reports a run-time error, the state machine will return to the proposing stage to prompt the LM to re-ple re-plan according to the error feedback.

---

Table 1: Performances of the compared LLMs and a VBA-based method. The three LLMs exhibit impressive Exec@) @1 and Pass I, with GPT-3.5-Turbo achieving the highest Exec@@ and GPT-4 obtaining the best Pass @ and efficiency. With our method, GPT-3.5-Turbo outperforms the method that generates and runs VBA code by a large margin.
Data
Models
Exec@ I1
Pass It
A504
A90
10%
GPT-3.5-Turbe
85.0%
45.0%
2.00
4.50
10%
GPT-4
65.0%
55.0%
1.33
2.00
10%
Claude
80.0%
40.0%
1.50
4.40
100%
GPT-3.5-Turbo
87.3%
44.3%
1.50
3.00
100%
VBA
77.8%
16.3%
4.4 Hallucination Mitigation
To enable the state machine less frequently return the proposing stage due hallucinationn induced errors, we adopt the following means.
Output Formatting The underlying functions of atomic actions require precisely formatted planning results. However, we found that LMs probably generate semantically correct yet inadmissible action plans as shown in Fig. [ Therefore, detect the tokens in the output to check whether the output is correctly formatted.
require LMs wrap actions with special tokens (e.g. @) and
Atomic Action Disambiguation The internalized knowledge in LMs is likely to be confused with the atomic action definitions in the document. Due to this conflict, LMs are prone to self-delusion, which means that it hallucinates undefined actions adds illegal action arguments [23 [4]. To tackle this problem, the atomic action names are substituted with a set of synonyms that are far away from the official names in embedding space. For instance, Write and SetConditionalFommas with Rangelnput Value and Format WithRules, respectively (See the details in the appendix).
substituted
5 Experiments
The goals of our experiments are threefold: (i) compare representative LLMs on the proposed dataset; (ii) demonstrate that the proposed method improves the success rate and efficiency a simple baseline; (iii) show the flexibility and stability of our method.
5.1 Benchmark Protocol
Dataset The 221 tasks introduced in Sec. 3.2 are used to conduct the following experiments.
Metries Exec@1 measures the proportion of solutions executed without throwing exceptions. Pass I used to evaluate functional correctness (5] A generated plan is considered correct if the final sheet state completely fulfills the task requirements. Beyond correctness, we propose A50 and A90 scores measure solution efficiency. These divide the number of atomic actions generated plan by the number in the ground truth and then calculate the 50th and 90th percentiles over all tasks. Lower A50 and A90 scores indicate that the LLM tends use fewer actions complete a task.
Models We adopt leading large language models with public API access, including GPT-3.5- Turbo/GPT-4 from OpenAl and Claude v1 from Anthropic. Details of the models and ayper-asgummnts used for generation could be found in the appendix.
5.2 Comparing Task Planning Ability of Different LLMs
We compare the three LLMs on the proposed dataset with the same token limit of 4096. For less accessible LLM APIs like GPT-4 and Claude, only 10% of the dataset is used for evaluation. We have maintained the diversity of this subset to avoid data distribution shift (see appendix for details). The results in Tab. show that GPT-4 demonstrates its strong planning capability by significantly outperforming both GPT-3.5-Turbe and Claude in the Pass@1 I and A50/A90. To explain why GPT-4 is inferior in Exec @ 1, check the results and find that it is mainly because GPT-4 exceeds the
7

---

Own
Mang
Cher
Per
Prot Tole
Fosl Tabio
Exce@11
PasS11
A50L
A901
GPT-3.5-Tubo
GPT-4
Clnade
Figure 3: The four metries decomposed in the six categories. The two GPT models both achieve 100% Exec@1 and Pass@) in the Management and Entry & manipulation categories. The three models obtain their own best efficiency different categories, suggesting that it difficult for these models to excel in all task categories.
Table 2: Ablation studies of the observe-proy propose-revise-ac framework proposed in Sec. 4.3 The
sheet state and error feedback increase the Exee@1 and Pass@1 when individually applied (rows 3, vs. 2) and bring significant improvement when both are applied (row 7 vs. 2). Inserting the external atomic action document with usage examples boosts the Exec@ and Pass@) and increases efficiency (row 2 vs. 1 and row 7 vs. 5). The synergy of the four components witnesses a large increase (30.7%) in Exec @ over the baseline (row vs. 1).
No.
State feedback
Error feedback
External doc.
Usage examples
Exec@ I+
Pass 1+
A501
A90,
56.6%
18.6%
1.50
4,00
67.9%
18.6%
1.00
2.50
75.6%
24.4%
1.50
3.00
92.8%
23.5%
1.00
2.00
68.3%
18.1%
1.50
3.78
/
85.5%
28.1%
1.50
3,00
v
87.3%
44.3%
1.50
3.00
token limit when solving difficult tasks although it has generated correct mid-steps. In contrast, GPT--35--urbb and Claude generate short but incorrect plans for most of these difficult tasks. Claude slightly worse than GPT-3.5-Turbo in Exec and Pass but exhibits better A50/A90, which shows that Claude a strong competitor of GPT--35--Turo. See more detailed failure analysis in Sec. F
To evaluate the category-specífic performances, We further break down the subset into the six categories (defined in Sec. 3.I.. The four metrics in each category illustrated in Fig. 3 The radar charts demonstrate that the two GPT models both achieve 100% Exec@1 and Pass @ in the Management and Entry & manipulation categories. Interestingly, The three models exhibit different patterns of A50/A90: GPT-3.5- Turbo, GPT-4, and Claude reach their best efficiency in the Formula, Management, and Pivot Table category, respectively. This suggests that it is difficult for these models to excel all task categories.
5.3 Ablation Studies of State Machine
We conduct ablation studies for GPT-3.5- the full dataset to analyze the impact of the two types feedback and external document insertion. The results are shown in Tab. 2
A) Closed-loop control generally boosts functional correctness Individually adding the sheet state feedback the proposing stage increases the Exec@ I and Pass@1 I (rows 3 vs. row 2) since the model no longer needs to implicitly infer the sheet state. Individually adding error feedback obtains the highest Exec@ (row vs. row 2) as a longer context window can be used to re-plan without the sheet state feedback, increasing the probability of completely finishing plan. The combination of the two feedback types further improves Pass@) A90 (row 7 vs. row 2), probably because the model solves more difficult tasks but with plenty of steps. It noticeable that without the external document and usage examples, the improvement in Exec @ becomes narrow and Pass @ even drops slightly (row 5 row 1). This is because the
cost of slightly increasing A50 and

---

Table 3: Ablation study of the atomic action names. Utilizing the synonyms far away from the official names brings an increase in Pass @ and slightly better efficiency (lower A50).
Models
Exec@ If
Pass@
A504
A90
Official names
87.3%
44.3%
1.50
3.00
Synonyms
86.9%
45.2%
1.33
2.79
A90
Petvatage (%6
80
3.5
70
Exco@1
3,0
Pass@l
2.0
40
1.5
0.0
0.2
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Temperature
Temperature
Figure Stability experiments results obtained by conducting evaluation times for each temperature except 0.0. The line charts show that SheetCopilot achieves stable performances even if the GPT-3.5 API temperature changes from 0.0 to 1.0.
simple action list in the initial prompt (shown in Tab. fails to provide sufficient information about how to determine correct action arguments even if detailed sheet description provided.
B) Inserting the external document improves both functional correctness and efficiency Merely adding the external atomic action document enjoys clear improvements of 17.2% in Exec 1 and 10.0% Pass@1 (row vs. row 5). This result demonstrates that presenting this document to the model enables it less frequently hallucinate illegal actions (which improves Exec 1) and to more accurately determine the action arguments according to the sheet state (which improves Pass@1 1). Further adding usage examples in the document reaches the highest performance (row 7). Additionally, without any feedback, the improvements over the baseline become narrower (row vs. row since it hard to determine correct action arguments without knowing the exact properties of the spreadsheet elements even if more details of atomic actions are provided. Adding the external document and usage examples reduces A50/A90 (row 2 that more information about atomic action usage helps to avoid redundant steps.
and rows vs. row 5), showing
C) SheetCopilot surpasses the VBA-based method To further demonstrate the advantages of our method, we compare with a method that generates and runs VBA code. Tab. shows that SheetCopilot with GPT-3.5-Turbo margin, increasing the Exec@ @ by 9.5% and Pass@ by 28.0% % This result shows that prompting powerful LLMS with method to control spreadsheets better alternative directly translating natural language requests to VBA code.
its backend outperforms the VBA-based method by large
5.4 Ablation Study of Atomic Action Names
To inspect the potential confusion problem stated in Sec. 4.4 we conduct another ablation study for GPT-3.5-Turbo on the full dataset by comparing the impact of adopting the official names and the synonyms. The results in Tab. 3 surprisingly show that using the synonyms increases the Pass @ and obtains lower A50/A90, which means that the model learns to use the synonyms to generate more correct solutions with fewer steps. This result demonstrates possible for users to define their own atomic actions and prompt LLMs to use them.
flexibility of our method:

---

Table 4: Ablation study of atomic action granularity on the chart-related and format-related tasks. The results show that using an integrated CreateChart action achieves slightly lower A50 and A90 but encounters significantly inferior Exec@1 I and Pass@1 1. Additionally, splitting the SetFormat into finer-grained format-setting actions leads higher Exec@ and Pass @
Experiment
Method
Exec@ If
Pass @ 1
A504
A90
Ours full
91.7%
43.3%
1.25
1.67
Integrating CreateChart
Ours Integrated CreateChart
79.1%
37.2%
1.00
1.50
Ours full
70.7% 80.5%
9.8%
2.75
6.65
Splitting SetFormat
Ours Split SetFormat
12.2%
2.00
5.60
5.5 The Influence of LLM Temperature on Task Plan Generation
We evaluate the stability of our method by running the full method three times with temperatures from 0.0 1.0. The results in Fig. show that the metrics are stable with slight deviations from the values for temperature=0.0.
5.6 Atomic Action at Different Granularity
A natural question is how to determine the granularity of atomic actions, i.e. the number of workbook elements an action manipulates.
To investigate this question, two experiments are conducted: 1) The actions that set chart properties incorporated into the CreateChart action, and the original separate actions are removed, with the expectation that SheetCopilot will set chart properties when creating charts. 2) In another experiment, SetFormat, which is originally used to set multiple format properties, is split into finer-grained format-setting actions. Please refer to Sec. D.2 for details.
We conduct these two experiments with GPT-3.5- Turbo backend on the chart and format-related tasks. The results in Tab Show that using an integrated CreateChart action to handle chart creation and property setting simultaneously obtains slightly higher efficiency (lower A.50 and A90). However, this variant encounters significantly inferior Exec@ and Pass In contrast, splitting the original SetFormat action witnesses considerable gains in Exec @ and Pass@
After analyzing the results, we found that an integrated CreateChart encounters lower functional correctness its complex document makes it difficult for SheetCopilot to understand the action usage, thus being less able to correctly determine action arguments. In addition, the lengthy documentation of this integrated action frequently exceeds GPT-3.5's token limit. In contrast, after splitting SetFormat, the LLM can easily understand the simple finer- greingg actions, thereby encountering fewer hallucination cases. See Sec. D.2 for detailed analysis.
observed that
These results suggest that is more desirable to use finer finer-grained atomic actions instead of integrated high-level actions in terms of functional correctness.
6 Conclusion
We propose SheetCopilot, a spreadsheet agent based on the observe- peservvepropos--revisee-aa framework that decomposes high-level task into step-by-step solutions for software control. To evaluate our agent, we curate realistic and diverse dataset representing the typical demand of non-expert spreadsheet users. The experimental results show that SheetCopilot can perform spreadsheet manipulation tasks with a high pass rate and acceptable efficiency, outperforming VBA-based methods. The ablation studies show that the closed-loop control and external document insertion used by SheetCopilot bring clear improvements over the baseline and that adopting set of atomic action names dissimilar to the official names achieves a surprising performance gain. We also find that utilizing finer-grained atomic actions instead of integrated high-level actions can notably improve functional correctness. We hope our work provides a useful roadmap for researchers interested in the field of M.L.M-based autonomous agents and sheds light on future research.
10

---

Acknowledgments and Disclosure of Funding
This work was supported in part by the National Key R&D Program of China (NO. 20222D016010), the National Natural Science Foundation of China (No. 61836014, No. U21B2042, No. 62072457, No. 62006231), Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences, and the InnoHK Fund.
References
[1] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky. Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie. Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong. Kyle Hsu, Jing Huang. Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning. Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang. Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models, 2022.
[2] Andres M Bran, Sam Cox, Andrew D White, and Philippe Schwaller. Chemcrow: Augmenting large-language models with chemistry tools. arXiv preprini 3XXi:2334.003766 2023.
[3] brian ichter, Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, Dmitry Kalashnikov, Sergey Levine, Yao Lu, Carolina Parada, Kanishka Rao, Pierre Sermanet, Alexander T Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Mengyuan Yan, Noah Brown, Michael Ahn, Omar Cortes, Nicolas Sievers, Clayton Tan, Sichun Xu, Diego Reyes, Jarek Rettinghouse, Jornell Quiambao, Peter Pastor, Linda Luu, Kuang-Huei Lee, Yuheng Kuang, Sally Jesmonth, Kyle Jeffrey, Rosario Jauregui Ruano, Jasmine Hsu, Keerthana Gopalakrishnan, Byron David, Andy Zeng, and Chuyuan Kelly Fu. Do as i can, not as say: Grounding language in robotic affordances. In 6th Annual Conference on Robot Learning, 2022.
[4] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Rad- ford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv, abs/2005. 14165, 2020.
[5] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan. Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katíe Mayer,
11

---

Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021.
[6] Xinyun Chen, Petros Maniatis, Rishabh Singh, Charles Sutton, Hanjun Dai, Max Lin, and Denny Zhou. Spreadshectcoder: Formula prediction from stmi-structured context. In International Conference on Machine Learning, pages 1661-1672. PMLR, 2021.
[7]
Cheng-Han Chiang and Hung yi Lee. Can large language models be an alternative human evaluations?, 2023.
[8] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. In arXiv preprint arXiv:2303.0332 2023.
[9] Nouha Dziri, Sivan Milton, Mo Yu, Osmar Zaiane, and Siva Reddy. On the origin of hal- lucinations in conversational models: Is it the datasets or the models? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5271-5285, Seattle, United States, July 2022. Association for Computational Linguistics.
[10] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00m 2020.
[11] Sumit Gulwani. Automating string processing in spreadsheets using input-output examples. SIGPIAN Not., 46(1):33))))))) 46(1):317-330, jan 2011.
[12] Sumit Gulwani and Mark Marron. Nlyze: Interactive programming by natural language for spreadsheet data analysis and manipulation. In Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data SIGMOD '14, page 803-814, New York,
NY, USA, 2014. Association for Computing Machinery.
[13] Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Xiaodong Song, and Jacob Steinhardt. Measur- ing coding challenge competence with apps. ArXiv, abs/2105.09933. 2021.
[14] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero- shot planners: Extracting actionable knowledge for embodied agents. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari. Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pages
PMLR, 17-23 Jul 2022.
[15] Wenlong Huang, Fei Xia, Dhruv Shah, Danny Driess, Andy Zeng. Yao Lu, Pete Florence, Igor Mordatch, Sergey Levine, Karol Hausman, et al. Grounded decoding: Guiding text generation with grounded models for robot control. arXiv preprint 2.X.xv2333000853 2023.
[16] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Tomas Jackson, Noah Brown, Linda Luu, Sergey Levine, Karol Hausman, and brian ichter. Inner monologue: Em- bodied reasoning through planning with language models. In 6th Annual Conference on Robot Learning, 2022.
[17] Harshit Joshi, Abishai Ebenezer, José Cambronero, Sumit Gulwani, Aditya Kanade, Vu Le, Ivan Radiček, and Gust Verbruggen. Flame: A small language model for spreadsheet formulas, 2023.
[18] Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 22199-22213. Curran Associates, Inc., 2022.
[19] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson,
12

---

Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode. Science, 37866624) 2022.
[20] Yaobo Liang, Chenfei Wu, Ting Song. Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuai Lu, Lei Ji, Shaoguang Mao, et al. Taskmatrix. ai: Completing tasks by connecting foundation models with millions of apis. arXiv preprint arXiv:2303. 16434, 2023.
[21] Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ramakanth Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. Augmented language models: survey. ArXiv, abs/2302.078422, 2023.
[22] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112. 09332, 2021.
[23] Pedro A. Ortega, Markus Kunesch, Gr' egoire Del' etang, Tim Genewein, Jordi Grau-Moya, Joel Veness, Jonas Buchli, Jonas Degrave, Bilal Piot, Julien Pérolat, Tom Everitt, Corentin Tallec, Emilio Parisotto, Tom Erez. Yutian Chen, Scott E. Reed, Marcus Hutter, Nando de Freitas, and Shane Legg. Shaking the foundations: delusions in sequence models for interaction and control.
ArXiv, abs/2110. 10. 10819, 2021.
[24] Bhargavi Paranjape. Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. Art: Automatic multi-step reasoning and tool-use for large language models. arXiv preprint arXiv:2303.08 090144 2023.
[25] Timo Schick, Jane Dwivedi-YY Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.0476.1 2023.
[26] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends huggingface. arXiv preprint arXiv:2303. /7580, 2023.
[27] D'idac Sur'is is. Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. ArXiv, 0ts/2303.081122 2023.
[28] Xinyu Wang, Sumit Gulwani, and Rishabh Singh. Fidex: Filtering spreadsheet data using examples. In Proceedings of the 2016 ACM SIGPLAN International Conference on Object- Oriented Programming, Systems, Languages, and Applications, OOPSLA 2016, page 195-213, New York, NY, USA, 2016. Association for Computing Machinery.
[29] Yizhong Wang. Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instruc- tions. ArXiv, abs/22 22.1112.0660 2022.
[30] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[31] Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. ArXiv, abs/2303. 11381, 2023.
[32] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:22 2022.
13

---

7 Supplementary Material
A Details of Dataset Collection
A.1 Details of Seed Task Collection
The details of the keyword-based and LLM-based filters described in Section 3.1 are shown below: Keyword-base -asee filtering: Questions containing keywords about irrelevant spreadsheet control tasks (e.g., Visual Basic for Application (VBA), loading and saving files, keyboard and mouse shortcuts, using images and shapes) are removed. LLM-based filtering: To easily identify irrelevant question and answer (Q&A) pairs, we introduce the seventh task category, i.c. Invalid, apart from the six categories. With these categories, set of Q&A pairs are labeled as exemplars to prompt ChatGPT to assign at least one category label to each pair (See Tab. B for the used prompt). After classification, we remove 2432 pairs labeled as Invalid, thereby obtaining a clean dataset containing 13574 pairs annotated with task categories.
We notice that multiple Q&A pairs likely refer to similar tasks with different expressions. To identify representative questions, clustering is performed within each combination of task categories (note that a pair may be labeled with multiple task categories). Specifically, the title and text of each question are concatenated as single paragraph which is then embedded into a 768d vector" For speeding up computation, all embeddings within a category combination are further projected to 64d vectors using Principle Component Analysis. Finally, these embeddings are grouped into 10 clusters using K-means++,, which means that 10 representative questions are selected for each category combination. Clustering all category combinations, 220 representatives are selected from the 13574 pairs.
As our evaluation environment does not support excessively complicated tasks (e.g., consolidating data, setting page layout, connecting pivot tables), out of these representatives, we further select 67 that involve operations supported by the evaluation environment. The tasks described in these questions are regarded the seed tasks that capture the most important patterns of the whole clean dataset.
A.2 Details of Workbench Collection
To lessen the difficulties of collecting a large-scale task dataset, the evaluation workbooks in the workbench strictly follow three rules: (1) Data in every sheet starts from cell A1; (2) Blank columns within tables are not allowed; (3) Each column is provided with a header on the first row, which means that the records of each column are listed from the second row. To better utilize the internal knowledge of LLMs, we also provide short paragraph describing the context of each evaluation workbook. This context briefly introduces the usage and domain of the workbook and includes useful formulas that assist manipulating the workbook. See Tab. A for the contexts of all evaluation workbooks.
A.3 Details of Core Set Collection
Adaptation. During adaptation, GPT-4 prompted to change the manipulated sheet elements (e.g. ranges, charts, data types) mentioned in the seed tasks. To enable GPT-4 to generate clear task instructions relevant to the evaluation sheets, three types of descriptions are included in the prompt, i.e., a brief sheet summary, column headers, and row numbers. In addition, adaptation exemplars written by the authors are included in the input to teach GPT-4 in a few-shot manner. In the end, a batch of 10 seed tasks to be adapted is listed so that the token limit is fully utilized, which speeds up the adaptation process and saves the cost. See the used prompt in Tab. C and adaptation examples in Tab. D
Verification. Between adaptation and simplification, an extra verification step required since a number of the adapted tasks are unrealistic and irrelevant to the target sheet. To retain valid proportions, we prompt GPT-3.5 (GPT-4 not used it overly expensive) classify each task according to four criteria, i.e., realism, clarity, relevance, and completeness. A task is valid only simultaneously fulfills the four criteria. GPT-3.5 asked describe how each task fulfills the criteria before determining its validity as we empirically find that this method more accurately
Thh model used for embedding OpenAl text-embeddingg ada-0002
14

---

Table A: The names and contexts of the collected workbooks. A context describes the usage and domain of the corresponding workbook as well as useful formulas used manipulate the workbook.
Workbook Name
Context
Invoices
My workbork records many invoices made different dates.
SalesRep
My workhook records the momthly sales of all employees.
SulmerSales
My workbook records sales of my company in the summer
Salee
My workbook records the sales of my company in the summer.
ShippingCosls
My company needs deliver goods to cuskemers by iruck. My workbook records the distances
hetween my customers and four destinations. The ver-mile shipping charge 3. with minimum charge of 75.
EmiireShippingCooss
My company needs deliver the goods to customers by trick. workhook records the distances between my customers and four destinations. The per-mile shipping charge is 3,5 with a minimum charge of 80.
PricingTable
My workbook contains two tables: Sheet "Sheetl" "I records my transactional data which are the number rolls of f'nce sold on certain dates, Sheet 'Pricing Table" pricing
determine the price per roll according to the range the mill number falls in (The range is bounded by Units From and Unit To).
RoomerangSales
My workbook has two tables. Sheet "Sheet I" records the sales of a boomerang company. 'Retail Price" lists the retail prices for all products.
Sheet
WeeklySales
My workbook records weekly sales COGS the profit has
calculated. The necessary
formula is Profit Sales- COGS.
NetInsome
My workbook records revenue expense. Income Revenue Total Expences.
PeriodRate
My workbook records the annual rates of my investments, year can consist of several periods,
Tax
My workbook records the woekly sales of my company and is used to compute taxes. The necessary
formulas follows: Profit Before Tax = Sales Total Expenses Before Tax; Tax Expense Prolit
Before Tax Tax Rate.
MaturitDute
My workbook records my loans with lengtts in days.
StockChange
My workbook records the values of my stocks on two dates.
IncomeStatemmntt
My workbook records the yearly accounting data my company. necessary accounting formulas
are as follows: Gross Profit Net Sales Cost of Goods Sold (COGS); Operating Profit Gross
Profit- Operating Expenses: Net Profit = Operating Profit- Tax Expense.
IncomeStatemmen12
My workboek recurds the yearly accoumting data of my company. The necessury accounting formulas are as follows: Gross Profit = Not Sales - Cost of Goods Sold (COGS); Net sales = Sales Sales return Discounts and allowances; Cost goods sold Materials charges + Labor charges Overhead; Gros profit Net sales Cost of goods sold.
SmallBalanceSheet
My workbook records the total assets, liabiliries, and owner's equity. Here are necessary financial formulas: Assets Current Assets Fixed Asscts Other Assets; Liabilitics & Owner's Equity Current Lisbilities Long-term Liabilities + Owner's Equity.
SimpleCompsuudinneress
My workbook blank and used recurd the interests of my investment. The necessary formulas as follows: Simple Interest = Principle amount Year Interest rate; Compound Interest = Principle anount Interest rate) 'Year.
FutureValue
My workbook records several investments whose fuluure values need to he calculated according to the formula Future value- Present value Annual Interest Rate # Compound periods) "(Years Compound periods).
Presen/Value
My workbook records several investments whose present values need to calculated according to the formula Present value Future value (() Annual Interest Rate # Compound periods) "(Years * 4 Compound periods).
ExpenseReport
My workbook records aspects of expenses
not yei been completed. The necessary formulas are as follows: Tax Subtotal Tax rate; Total i Subtotal + Tax.
Demographiccrofie
My workbook records information of respondents.
GDPBreakdown
have sheets: Sheet "Sheetl" records economic indicators of countries across the years. Sheet
"Sheel2" records a list of chosen country names.
FasyGDPBreakdown
My workbook records the economic indicatons of countries across many years.
XYScatterPlol
My sheet shows how two variables (Range and Height) change along with the projection angle.
Velocity DisplaeeDtel
My sheel records velocity against displacement.
Dragging
My sheet records data from an experiment where one hanging block (m22) drags block (m (01.5
kg) on frictionless table via a rope around a frictionless and massless pulley.
RampUpAmiDDow
My sheel records the accelerations of block two physical scenarios but has not been completed.
One scenaria is in columns to B while the other is in C D.
15

---

identifies invalid tasks. Finally, 1515 valid tasks (90.7%) remain. See Tab. L.b.Fer for the used prompt and an example of using GPT-3.5 to verify the tasks.
Simplification. As the adapted task instructions are likely to refer to specific functions and operations built in Excel, these instructions need be simplified read like the tone and fashion of non-expert user. We establish the following rules for prompting GPT-3.5 to simplify the tasks: 1) Convert specific mentions natural language descriptions and remove redundant words while retaining the original intention and order of actions. 2) Avoid referring to existing columns by the column indices since it more natural to refer column by the column header. 3) Mention the insertion place and add column header when inserting new column to avoid ambiguity. 4) Finally, use domain-speciin knowledge diversify the expression of the generated instructions. See Tab. for the prompt and an example.
After these steps, a core set from the simplified tasks, six random tasks are selected for each category combination, resulting in 402 selected tasks. The authors polish these selected tasks by further revising them and discarding a fraction not supported by the simulator, resulting in 184 remaining tasks. To enrich non-expert users compose 37 more tasks. Lastly, 221 tasks exist
dataset containing diverse, natural, and realistic tasks is produced. To extract
the core set, the authors in the core set. See Fig. for the instruction length and atomic action distributions, Fig. proportions of the six task categories and the task diversity of the core set, and Fig. C for the numbers of each category combination.
[ B for the
The final step is prepare reference solutions for these tasks. To objectively judge the performance of L.MMs, we use GPT-3.5 to generate multiple solutions for each task and then retain the successful ones after verifying the final by hand. As number of solutions likely to exist for task, multiple reference solutions are collected as the ground truth for one task.
A.4 Selecting the 10% Datast Used for Comparing the LLMs
20 tasks are selected from the 221 1ttaks core set to approximately maintain the percentages of the six categories and the distribution of the numbers of atomic actions. This collection basically represents the pattern of the core set, avoiding data distribution shift to a large extent. See Fig. D for the statistics of this subset.
40
49
35
30
Instructions
25
Atomic
20-
10-
10
8
20
50
990
110
140
170
200 230 2M)
320 350 580 4110 4401 470 Instruction Length
500 530
number atomic involved
Figure A: The distributions of the instruction lengths and the numbers of atomic actions involved in each instruction. The two histograms demonstrate the diverse task complexity of the core set.
B Atomic Action Names
B.1 Collecting Atomic Actions
Selecting set of indivisible and understand atomic actions is necessary since LLMs need to generate an interpretable solution to the user-specified task. As existing atomic actions used in spreadsheet software have been curated and publicized, we obtain the names of these actions during classifying the SuperUser Q&A pairs (Described in Sec. A.I) Specifically, ChatGPT is prompted to determine which atomic actions each Q&& pair requires to complete the task, which resembles a text summarization problem. As multiple similar names are likely be generated for an atomic action,
16

---

Table B: An example of using the classification prompt to predict multiple category labels and extract the involved atomic actions for the Q&A pairs scraped from SuperUser. For clarity, the prompt components and an example ChatGPT response are marked with brief descriptions in blue.
(Requeements and cateagory definitions for classificatio)
I have a batch of Excel questions for you to clean. I need you to deterning which caregories each question belongs to according to the question titl, and its belong multiple colegortes invalves the stomic actions of these categoriss
question
The six categories are detined as below:
A: Eniry and Munipulation, lasks that enter and manipulate cell contents soch as editing cell values, splitinng tests, imsering and delecing cows/columns.
B: Management. Tacks that organize data such as sotting, niherrngg using slices fitter data, and moving mes/cnlumns.
Formatting, lasks that modity sheet cell formats such changing fost, adding drop-dewn
and merging cells.
Formula, Tisks related to using fomuulas such as performing calculations statistical analysis. Es Charts Tisks related to presenting data visually such as creating charts, customizing chart elements, and addlngg a trend line.
F: Pivor Table. lasks related to using Plyat Table such as creating and formatting Pivor Tables.
of six categories above or also need you desermine which atomic actions each valid question requires somplese task.
question does belong
vague, regarded "Iavalid".
(Clvvsificatioo exemplars collected by authors.) Below are example questions the form of Python dictionary:
Questions:
Q1: 'queestion Hill":"Hww
'How filter a call which containing data by other list", "tep snswer": "You have solect the Column Headers and "Data", "Fillee" and filter" NT.2003
"Daia", "Filler for XL2207 and probably 2010."|
Q?: (qqueation litle "Group nows by oolumn ihen soet by another columnn.. . "top answer 1/1 fourd the solution myself, First soet by the ID then I added a now column with the following formsla-NNDEXX -INDII (55211 MIN $152785877----- RR.--S ROW:5152::1577880 Replace the hounds necessery. then soet this
new column and then the column
duta the order (wntted.)) ")
Q3: ('question tille" " "How to delete infinite blank rows". "Top snower "Go to Home tib - Find and Select - Gi Special Select Blanks, Click OK, The
blank cells will he highligheed... "i
Q6: (' ('question tille" "'How
vertain and insert before vertain row excel" "wp answer"; TTighlight the row by clicking of the number then press press ChrleV
Cudlx then click the number helow the row were you intend
the
Q5: (' l'question tille" "Ilow do I add conditional formatting to vells containing 4NAA in Excel?", "up answer" "Select the first cell you want to highlight. (B6),
Click Home > Conditional Fermatting >> Manage Rules >> New Rule, Select Use a formula to determine which cells to format. In the field Formar values where this formala is trae, enter =I5NA(566) (5B6). Click Formal to set the cell formatting, then solect OX
Click again to crease
furmatting rule. the Conditional Formatting mule, which will mucch to trae and thus apply the Cormatting you want"" "
Manager, edit range under Applies to $D6:5BR) Select OK apply
Q6: (qquestion mile"" HHow do you anto masine colls in Escel"" "tp anower 1/1 have solved like this Select the cntire speadshers 2. Deuble click the line the the
between
rows. This sizes
that I clicked on.")
Q7: l'quesion SSum
in column
starting WED.IIDEEEDLNNIIISSSSS..
welting explicit last indes". unswer": Foee 1,09. For Excel 2007 or after:
Excel 2003 or before: -SUMISSINDEXIBBS BS: 865566 (K/0B---0118555) (018576),000.0) L.D)")
(BB-INDE MATCHITRUUETRRRRR CEEECC/TTTREEEEEEIII INDEX (OS.LLLLLAAAA ANK (BSB
QB: ('question Gille"
the dates the corment monday, wednesday and frida of the curmet week in excel"", "top answer" Thee following will display the Monday, Wednesday Friday
=NOWO- current
=NOWD- WEEKDAY(NOWO..
(NOOOOO))))))) =NOWO- WEEKDAY
Basically this is taking the time now, and subtracting the current wockday (which gives you Mondayy) then adds 2 days or 4 days to get Wednesday and Friday: 'I Exvel Table object: Select cell in
Q9: ('question Essel: how adjust range in pivot table automaticall"". 'top answer" FFint, your data table source range, click
keyboard shortcut Ctri-T...)
Q10: ('question rille Create a pie chart from distinct values one column by grouping data Excel", "top answer": "This is how 1 it: I. Add column and till it with (name it Count for example). 2. Select your data (both columns) and crease a Pivor Table: On the lnsert tab click on the PivotTable worksheet Q11: l'question ditle "How Change Oreentaion of Multi Level
Pivet Table
create on the
shoet) ...?;
Vertical Excel Chart?" - "top answer": "You can oaly control alignment of inner most sei of mult level axis labels. Except when you add a data table to the chart, then you have no cunnel over the aligmment, One thing you can consideris is to tumn off the multi-level category option!))
I'question Usle" PDF with intennal hypertinks.. l'question ride' What does TABLE function do?". "up answer": "
Q14: I (qquesiio tikk" 'How du you create a report in Word (ur other documentation software) that is linked direcily with Excel 2007 dasa?", "top answer" "" (Exemplar classifiction results consist of the category labels and invulved atomac actions of each Q&A pair)
Resule:
01: categoey"""" "8", "atomic actions" "Fiterr)
02: [" c'ueeooy""" U. D", "atomic actions" 1. SSct, Statistic functions, Lookup and reference functions")
Q3: (' ('uuegoy""" A", "atomic actions" ": DDeett rows")
Q4: c'ategooyy'
"stomaic action""""Uu paste.
rows']
Q5: ('cueeooy"""C "C" "atomic actions" 'CCnndtional fommatting"
Q6: ('calegey" CC "atomic actions" "Resize cells") )
"
Q7: /'categey""11""" "asomac actions Moth functions))
Q8: ''caeeooy""""" "atomic actions: D'ut and time functions")
Q9: ('aategoe"" A "atomic actions" "Creste Pivat Tables, Insert cows"]
910: |'category"""A A",,, k, E."," "itcessc actions" "ooons"""nnerr" ""neert columns, Crease Pivot Thbles, Create Pivor Charis")
Q11: /'category W.".. "atomic actions " Change axis label")
''category': "taggorry"""ovliid" "atomic actioss"
013: ('category" "lavalid"" "atomic actions" "None)
014: I' I'calegory" "lvvalid" "atomic actions" :None)
(Q&&A poirs to be classified)
Please process the questions below and give results in the same form as the etamples ahowe.
Questions:
Q1: ('questica titee: SUM calumn unless value is (A))-CCI-C7))
matches col "top aaswer" "You
following, antered aray fomuula with
-HIHT--ENTEE -UMLFICOUUUTIII
Q2: I" l'qucstion tite" Woord paste rable from Exeel has marks insteed of values", "top answer" answer","Canee Caanse: The colume "ooo sonoow to alllow showing the contem, accorling Word. Remedy: Move the
pointer slowly horizeetally over the column border and it will change shope some point ..!! mathenatical convention for the of expoeenistion" 'top answer" "Your teacher right Accondiag standard rules of algeben exponeats are solted from right to left, therefore... ...'
Q3: l'ueestion rille" :" Accoaddng 4-302 (213)22, Is really the
(GP1:335 response)
Q1: [" l'calegoey"" A' D', D'""attmii 'atomic actions " "Update cell value. Math functions") ) Q2: C'alegoy"""C "alomic actions" " FFurmating vells")
Q3: cattegoyy. Invalid"" 'atomic actions (Noue]
17

---

Table C: An example of using the adaptation prompt to adapt the seed tasks. For clarity, the prompt components and an example GPT-4 response are marked with brief descriptions in blue.
(Ragiisements adogeation)
As an Excel expert. you have been assignd the responsibilit of alpping a set task instructions for speciáic Excel workbooks. These instructions will be utilized to evaluate the Excol munipulation capubiliiees of large linguge models.
Requinements:
Find, ikentify individual stomic actions used the original instractions, then develoop new instructions incorporating these actions,
2. Use the detailed desoripionss of the provided workhooks to madity the original instractions so that they hecome computible with the workbooks Specifically, you
must change the munipulated objects (rages, sheets, and culumns) in the original inctrutiions. You must also chunge the way you stomic actions. For instance, types 3. Use scandard runge references, such as Sheee21A1 A2:E11", 'All'' column C, or row 16.
original insoruction the data type as accouming, you chango
uduptation.
4. Use different phrtasnn (e g. various senience structures and noun/verb forms) to crease diverse instractions.
5. Apply domzin-specilic knowl diversify the genersted instructions, For instance, use financial know calculate various metrics, demonetrate trends in product sales from different perspectives, visualize data using various types of charts, so
6. (Importam:!) The generated instructions must deseribe realisii tasks and involve the normal use of atomic actions acconding to the workbook descriptions.
7. every now instruction, new tables and Pivot tables must be crested in a new workshoet and start from A1. Only one new shoet is allowed to be created. Besides, the headers of columas/ros and the
new sheets
The instructions after sdpptaiin should look like
non exper Excel would say and should not menton any specificc functions or openations built Excel.
(Available atomic actions used to label the adaptation results)
Here are the asomic actions you can idenuify within the six categories:
A, Entry and manipustion: Update cell value, Deleie, Splic text culumns, Insert Insert column. Ausofill, Costep Copy-puste format, Capy sheet, Cui-paste, and replace, hypertink, Delete hypertink. Romove duplicates, Rename sheets, Insert checkbox, Insert textbox, Create sheet.
sheet, Clear B. Management: Switch sheet, Sort, Filler. Delete filter. Slicer. Move rows, Move columns, Group. Usgroup, liide rows. lide columns, Unhide ruws, Unhide columns. Hide sheet. Unhide sheet, Set possword. Tranposee Create named range. Delete nomedl range. Data consolidation, Freeze punes, Unfreesse pines, Split pines Formatting: Format cells, Set data type. Delete formar. Merge cells. Unmerge. Change Protect, Unpeutect. Deup down Daia validation,
layout, Set border, Resize vells, Conditional formutingg. Lock and unlock. checkbux. Display formulas, Wrap test. Unwnp test, Autofit
D. Charts: Create char,, Create Pivor Chart, Set chart tile. Set chart axis, Set chart has axis, Set chart legend, Set chart sype, Set chart color, Set chart source, Set chart marker, Resize chart, Set tredd lime, Add data libels. (leave out for clarity)
E. Pivul Tuble: Create Pivot Tuble. Remove Pivor Table. summary typs. Sort Pivot Table
Formaaaa: Date time functions, Logical functions, Lookup and reference functions, Muth functions, Statistical functions, Test functions, Hinancial functions
Restrictions for the atomic action panameten::
Chart type only be imacked/chssteeei columa chart. (stakkddclusteerd) bar (ID) pie chart. line chart (with smooth lines), (stacked) area chart. chart. Coll value type can only date, test, time, currency percentage, number, or general.
I will give you an example fist:
(Adaptaaiin exemplars writeen by the authars) Given on Excel workhook:
The shoet "Sheetl" records emplayee woeking hours coffee shop. how collmms (Headers are: A: "location", B: "name". C: "date", Dr "hours", Et hours", "hase pay". "ot pay", H. "total pay") and rows (inclading the header now). The cells the "location" column can "capitol hill", ; "quees anne". The cells in "saae" columa can he "Ammn", i "Bob", "Blanca", "Frank".
The original instructions be adapted:
I'd know if there's a formula combination fomuulas) sum Columa ONLY it equals Column A?
2. Column A contains milliple due dates and call B1 is 10/31/2022 Append a special choracter to the eells in column A depending on whether the due dute less or more than the date in B1. If neither applics. leave the cell unchanged.
Table separatc dates by month similar the quarierly function, This will show all dates the last year under the column lubel. Choose mouthly option mnder data fillers in columas label under data filiers.
Create
4. Freeze A1.810 so that no matter how scroll vertically or harizontally this mage is always frozen.
5. I have five groups of data each occurying two columns. I'd like to have them plotted all on one bur churt but with the seriex separited (ie.. not clustered) so that togother sharing the same y-ssis.
several column
Adaptations computible with the given workbook (Show the categorins involved in the generated instraction and list the stomic actions following the cateegury label);
1. Instructio:: In new column with hemler "Toaal pay each locution", use formala to sum the 'totall pay" (Colamn H) for each employee ONLY if their "location" (Column is "capitul hill". Categories (atomic actions): (Update
value):
functions)
2. Instruction: Create formala in new colamnn (Column 1) with header "Marked Due Dates" check the dates column are less or more than 10/31/2022. If the date is less, append a 1 to the cell in the some now in colamn C: if the date is more, append a "+", otherise, levee the cell inchunged. -Caneorics catomic actions): A (Updase cell value. Autofill:: I (Logical functions. Text functions)
"Shoe12" bused the data in 'Shetl' then summarive sum Canegeiis (steenic actions) (Crente sheer), F. (Creste Pivor Tiblle)
Instruction:
Froor Table
for location in
Table
4. Instruction: Freese the nage Al:Il so tuat the headers remain visible w when scrolling vertically or horizentally. Categories (atemic actions): C (Freeze panes)
5. Instruction: Create Picot Table in sheet nimed "Sheet?" to surn the hours for all dates and then plot line clart to show the trend of hours changing with dates. Categories (atomic actionste (Create sheet); E (Create Pivot Table); (Create chart)
(Seed tasks be adapted)
Now it's your tum.
Given an Excel warkbook:
My workhook records information of respondents. Sheet "Sheetl" has 4 columns (Houders are A: "Respondeats", B: "Sex", C: "Cvil Status", D: "Highesst Educationaa (including the header The cells "Respandenis" columa range froms 1.00 to 40.00, The cells in the "Sex" column "Male". column can be 'Single", :, "Widow". :, "Mariee", The the "llighet Educational Attainment column can "Elementary" "Senior High School", 'Jansce High School".
Attainment") and
"Female". The cells in "Civil
The original instructions be adapted:
know if there's a focmula combination fomuulas) to sum Column ONLY equals Column 2. I have a sheet I with à cobamns as shown below. I have snother sheet named Sheet 2 with only the fist numes in Column A Sheet 1. would like write focmula as column of Sheet which matches the first name in Columa A of Sheer agaitst the first name in Columa A of Sheet 1. Ir" is found is column Cf Sheet 1 then copy the information in columa of Sheet column Sheet 2.
Adegeations compaiill with the given workbook (Show the categaries invovvd in the generated instruction and list the stomic actions following the category label): (GPF1-4 response)
Instruction: find out if there's a fommul combination of them) that up values in "Highest Educational Attimmenn" column (Columa D) ONLY "Sax" 'Male' (Column By7- Categories (stomic actions): (Uidaate cell valuejy F (Muth functions, Logical functions) Instruction: In new columa (Column E) named "Geeder-Match Educctionn", find if the "Sex" in column B matches the "Highest Eddcctiinaa Atainmenn in Columa D. If theee''ss a match, copy the exrresponding" "Cvill Stans" from Column C. to Column F. Catogorizs (atomic actionss; A (Insor colamin, Upvlate cell value, Autofily;: F (Lookup and seference functicus)
18

---

Formatting
Entry
14.7%
&
Pivor Table
manipulation
37.2%
8.8%
CHeas
12.0%
Cherge
Charts
5,4%
21.9%
Management
Formula
Figure B: Left: The proportions of the six categories. Right: Diversity of the verb- noun phrases in the core set. We demonstrate the diversity of the core set by showing the top 10 most frequent root verbs (the inner circle) and their direct noun objects (the outer circle) in the instructions.
Neerating
28
Pany and ruripsstions Forral horrating
Fereda, ban maspelatio
Maugrewer
and nocipabrion
Tany and reavpelaion
13
Exky and maiipatio Pioct Talle Clara
Eay rasipdaaiss
Clarb
Essy and mazizahiin, Evrrals, Clata
bary and sunqvinser, Fommatin,
lirty
nuripabtive, Fremecls Plan Clans
Eary and manipataion Formatiay, Fine Take
Easy and ixcipslatio Miragerer, Clars
Matagesent
reniulaiee. Foratiin
Mangaseni Senstiing
Chara, Fosnating, Extry and cranpedaion
Mangenent masipelaiis,
Manggmmntt ban
Daey and rrappeitios, Mauggeser
Dornets Hormantag Eaery maspelstio
Pary recipstaionn
Forres Socreting
Ersy and manipakioon Chars, Fivo Take
Fonrala Easy and mxtipsitioon Chars
Poot Exsy maiipaative
Forada
5
15
25
35
Count
Figure C: The distribution of the category combinations.
19

---

Table D: Adaptation examples. The adaptations show that using the proposed prompt, GPT-4 is able to adapt the seed tasks for the evaluation sheet by changing the operated sheet elements and use of the atomic actions.
Adagtations
that agpear
new cell,
C)
'Sheeil' thas appear more than once.
Add leading zeros to every number in column A then append them to the column
In a new shoet nimed "Sheet2", format the years in column A from
"Sheet!"
append corresunndnng Gress Profit (Column
3) Apply conditional fommatiing oells column based the cell value
Apply conditional formatting in column based on two other columns.
the cell value in column is not "End" and the cell value in column D is
in column B. If the value in A is greater than the value B, display the
the value in column A displayed red.
value column A
occupying two them platted all one bur chart but with the series separated that several
groups
"Sheet2" distances for each customer to each destination, and then create a clustered
Crease
column chars stack logether that shure the same y-ukis.
bur chari represeni the distamces for each destitation on the same y-axts.
Table E: Simplification examples. The simplification results demonstrate three features: (1) using brief and clear references to operated ranges; (2) expressing the intention with spoken language instead directly mentioning Excel built-in functions; (3) considerably reducing the sentence length.
Adaptations
Simplifcction Results
Calculate the sam of "Quantity (Column E) in 'Sheell' ONLY if the
Find the total quanity sold for each product listed in the 'Retail Price' sheet.
"Praduct" (Column in 'Sheetl' matches the "Product" (Cilaaon
RReaal Price'
new sheet named "Sheet3", INDEX and MATCH to rotrieve
Retrieve sales information based on row IDs listed on another sheet display the results ou new sheet nameel "Sheet3".
solles information from "Sheet broed on the row IDs provide in list
noother sheet.
colurnn (Column Hi
calculate
Calculate
of days today "Dute"
days differenes between the "Date" (Column and today's
new column named "Days from Today."
Formatting
Enity
14.7%
5-
Pivot Table
A Atomic actions
manigulation
37.2%
3
12.0%
3
2
5,4%
2
21.9%
Management
Formula Caregory proportions
0
The number of atomic actions involved
Figure D: Statistics the subset used for comparing the LLMs.
Acting Stage
[Paitore
Runtime Error
operation
Observe thout state
JAdosintble autions)
Query
Observing Stage
Revising Stage
Unnvald actirad
Validation Error
Start
Mulnsaelbbee actiond Oeery with externa
Observe sheet
Query with requirement valid actions
Qbery for aest
End
[Denc]
Proposing Stage
Elnaald actions
Validation
Query with requirement valid actions
Figure E: The complete state machine for task planning. The transfer condition (orange texts with brackets) must be fulfilled before performing the corresponding operation (green texts) to transfer to state.
20

---

Table F: An example of using the verification prompt to classify the adapted tasks. For clarity, the prompt components and the GPT-3.5 response are marked with brief descriptions in blue and the redundant contents are substituted with ellipsis.
(Roquirements verification) I will give you a hatch Excel tisk instructions that are utilized to ecaluste the sprencsheet munipulation capubilities large language models. Please chock instructions according to the eriteri and the deseriptions of the given warkhooks.
Requirements
1. an instraction fullills all of the following forr criteria striclly at the some time, is valid.
A, Realism: Verity that the insoucricns are representative of real-world Fxcol prahlems encrentorol by expert users, Avaid including contriced ce overly complex trsks that would rarely be encouniered practice.
Relevanc:: Verify that
instuctionss are relevant the context the given workbooks. Reject instuctions that not relate to the content of the workhooks or are
not applicable in the contest of Escel.
Clurity: Verify that the instructions clear and easy understund, They should be free from grammatical and ambiguilies.
Completenes: Verify that the instractions
completed using the provded workbook and Escel features. addditiional data functionality noeded
accomplish a tekk, reject the inctrution.
2. Use your domain-speiffe knowlolge resson and make decisions, For eximple, it is unlikely to calculate monthly yearly averages some physical values because task impractical.
(Exemplars for verification. The sheet description provided accuuntely verify fuitillment the criteria)
will give you an example fins:
Given an Excel worktookk
My workbook records many invoicss made different dates, Sheet "Sheetl" has columes (Headers are "No.", "Dons" C: "Salesman", "Product", "Price", The the "Dute" column "201 1-05-22 28 0000:000 "2011-08" 20011-05-25 00:00:00" 2001--0--7 000000y" The cells in the "Salesman" column can he "Joc", "Chin", "Moe'. The cells in the "Product" column can be QQud", "Majestas, "Bellen', "Criooa", 'Aipine" The cells in the "Price" columa range from 22,00 to 32,00. The cells in the "Unis" column range from 4,00 to 25.00. The cells in the "Sales" columa range from 128.00 to 750.00.
rows (including header now). The
"No."
range 10500.00 to
Instractions to check:
1. Find the soles value conesponding each No. in new calame calledd "levoice Lockup'.
2. Compare
Salesman column find the closest mutch for each Put the results new column named "Salesman Marched". representative Sheeti.
3. Create sheet named "Sheet2" summarize the total
for each
4. Preyend leading zemes each sumber in the "No." colamn so that they have fised length of digits. Append the new results to the eocrespouding prodact names in
the "Produet" cotumas and put the results in new column named "Padded No.".
Find the coresponding date for each No.
column named "Lookup Dase" Sheetl. results
and display the sow values Sheetl based on the column,
sheet named "Sheer?".
7. Round the values in the "'ales" column to two decimal places in new column nimed "Kounded Sales", and display the results with trailing zeros.
8 Merge cells Al through CI with cill A2 through D2.
Add hyperlinks each in "No" columa that link its correspanding file.
Check results (Give bridf ressons the comments and tell me the instmetiion is valid):
1. Realism: Yes, Relevance Yes, Clarity: Yes, Completenzss: Yes. Comment: The instruction fultills the 4 criteria, so it valid.
relevant the of 3. Realism: Yes, Relevance: Yes, Clarity: Yes, Completenese: Yes, Comment: The instruction fultills the 4 critesia, so it valid.
2. Kenlism: No. Relevance:
Clariyy: No, Completeness: No. Commest: instruction is unczalistic and unclear and does given workbook, so it incaldd
4. Realism: No. Relevance: Yes, Clarity: Yes, Completenes: You. Comment: The instruction appendl numbers product names, which is not realistic requirement, invulid.
Realism: Yes, Relevance: Yes, Clantyy Yes, Compleseness: Comment: The instruction fultills criteria, so it valid.
6. Realism: Yos, Relevance: Yos, Clariy: No, Completmes: No. Comment: The instraction does not specify what values to display, so it invalid.
7. Realism: Yes, Relevance: Yes, Clarity: Yes, Completenese Yes. Comment: The instuction fultills the criterin, so it valid.
Realism: Relevance: No. Cladiy: Yes, Completeness: Yes. Comment. instraction merges cells, which destroys the original data and is menningless
contect the workbrok, so inalid.
9. Realiam: Yes, Relevance: Yes, Clirity: Yes, Completeness: No. Commeut: The instruciion does not refer to specific corragoandngg files, which is incomplete, so it invalid.
(Inemections be veriffed) Now it's your
Given an Escel warkbookk:
My workbook records all rspects of expenses but has not yet been completed. The necessary formulas are as follows Tax = Subuutal + Tax rase: Total = Subiona Tax. Sheet 'Sheell 1 has 5 columns (Headers A: "Duie". B: "VenkeeClien" C: "Expense Accoumi, 'Subtotil", E: "Tax", F: "Total") and 25 rows fincloding the header row). cells
"Dase" column be "2020-01 01 06 00:0000." '2020 01 0000:00" "2019 12:27 0000000" ... The cells "VendortCien" zolumn can "Comppnny A", "Comppny B", 'Client B", "Client A". The cells in the "Expense Accuunt columm can be "Material Purchase", "Meals", "[liring" "Gus", "Car Repain". . "Reveption" " "Acoommodssion". "light tickets". The culls in the 'Subtotal" column range from 127 127.00 1954.00.
Instractions check:
1. Calculate the sum of the Subional columa only for the rows that have "Compsay A" in the Vendor(Client colums.
2. For euch row in Sheet2, find its matching row in Sheetl, and if the Expense Account is 'Accoomodaioe" copy the Subsonal to column B of Sheet2.
3. Calculate the by multiplying the Subiocal column by and write the resulis
column with the header "Tax Calculation" Date columa and the curreat dute. Writ the results in new columo with the header "Days Since Expeesee..
number days between each date 5. Count the number of VendoeChent values that appear more than once in the data.
Calculate
6. For euch ruw, subtraci the Expense Account's 's lins chree chumaters from the Subtotal and put the result in new column numed 'Adjusted Subtotal"
Check results (Give brief ressons the comments and
the instruction is valid):
(G3FT:3.5 respanse)
Realism: Yes, Kelevunce: Yes, Clanty: Yes, Compleseness: Comment: The instruction fultills criteria, so it valid.
2. Realism: Yes. Relevance: Yos, Clarity: Yes, Completonese: Yes, Comment The instruction spacifles motching condition and copying of data hised on that
condition, valid.
Realism: Relevnce: Clarity: Yes, Completenese: Comment: The instruction caleulites based the Sohuttal column, is valid.
Realism: Yes, Releunne:: Yos, Clarity: Yes, Compltteses: Yes. Commert: The instruction calcalates the number days between datei, which practical tack, is valid.
5. Realism: Yes, Relevance: Yos, Clariy: Yes, Completenese: You, Comment: The insenuction requines couming the number duplicate values, which is common
tack, is valid.
Realisun Yes, Relevance: Yes, Clarny:: Yes, Completeness: Yos. Commant: The instruction specifies a colculation on the Expense Accoant column, so it valid.
21

---

Table G: An example of using the simplification prompt to rephrase overly specific task instructions. For clarity, the prompt components and an example ChatGPT response are marked with brief descriptions in blue and the redundant contents are substituted with ellipsis.
(Requeeements for simplifictioo)
You have been tasked with paraphrasing set of instructions for Excel tasks.
Requirements:
Paraphaase each gaven instruction that more instruction.
non-expert Extel user would syy while retaining the original inteation and order actions the
2. Do not memtin any Excel built-in functions in the paraphoases If the original instrction mentions the names specifie Exeel built-in functions, you should replace functions spoken language. For example. "Use CONCATENATE combine cells" menions "CUNCATENATE" should be rephrased "concatioane cells". "Use conditional formatting change background color of cells" mentions "conditional formatting" and should be rephrased
background color
_.. "Write a focmula to copy data' should be replatsed as "Copy data".
3. Do not refer to existing columas by their indices le.g., column A is not desired;; instcod, memiion columns by their headers. For instance, "In column A' should he rephrased 'In Year column..
not add colume references in beacketts. For instance, "suktracting Tocal Expenses (Column B) from Revenue (Column A)" should be rephrased 'Subrract
txpeesses from Revenue". :
5. When inserting new columns, the colume indices most he given avoid anhiguitt. Resides, each now oolamn mist be given a header, so need keep the the original instractions.
column headers
Use domain specific knowledge divessity the expression the generated insoructions not use the same expeession repeatedly.
(Esemplars writtnn by authors in-contest learning)
I will give you some examples fins:
Original instructions:
1. Convert the "Yerr column valuees (Column A) from the fomnet of yyyy-00 yoyy.
through in each now, including 3. The advanced filters in "Sheetl In display rows with sales datt for specific month (assuming weeks starting from the Ist each month). For example, lilter to show with weeks. "Week to "Week 4" for the first mouth.
a new column with header "Combined data",
CONCATENATE finction cumbine
headers, and then mutofil the formula the last row data.
line
"Shoet?"
weeks (Calumn
and (Columa the y-axis.
5. Apply conditional formatting to Column II (Net Profit) based on Columns F (Operating Profit) and G (Tox Expense). the well value in Column G is not and Column F greater than D, display the value in Culumn in red.
6. In sheet named "ShoelT", use the VL.OOKUP function match each row in "You" (Column A) "Sheet I" with the corrsponding value in Colame B ("Net Sales") "Sheet)'.
7. a new column (Colamn G) with the header "Txx Calculaionn, , use a formula calealute the tax by mulliplying the cornosponding "Subtotal" (Column D) by O.1.
Think about the flaws the original instructions before pamghessing
Think: The original instruction refors columns by indices brackets, which undesired.
Paraphrase: Convert the 'Year' volumn format from 3yy.00 to yyy..
2.
Think: The original instructio menioos Excel built-in functions CONCATENATE, which is undesired.
Paraphrase: Concatenase data columns through 2 for zows and the results new column named 'Combined data".
Think: The original instuction menions an Excel built-in operation fix., advanced filter),, which is undesined.
Paraphrase: Display only the zows with weeks from Week to Week 4.
Think: The original instruction refies columns by indices in brackets, which undevined.
Paraphrase: Cresse a line chart on new sheet with the Week column on the x axis and the Sales column as the axis.
5.
Think: original instruction mentions Eiteel
functions (i.c.. conditioaal formatting) and refors columns by indices, which undesited. Paraphrase: If the cell value in Tax Expense Column not 0 and that in Operating Profit Column > display the cell text in the Net Profit column in red.
6.
Think: The original instruction mentions Excel built functions (i... VLOOKUP) and refers columas indices brackets, which undesired.
Paraphrase: Match
Year column and return corresponding values the Net Sales Column. Put the results in new sheet.
7.
Think: The original instrutionn refers to columns by indices in brackets, which undesired.
Paraphrase: Calcuhtte the multiphynn the 'Subtutal" culumn by 0.1 column zamed "Tux Calculation.
(Tack instrutions be simplifted The sheet description provided to maintain the relevance)
Now your Please follow the ropirements paraphmse the original instrcctonss acporting the given worlbook descriptions.
Given Excel workbeok:
My woekbook records all espeets of expenses but his not yet been completed. The necessary formulas are as follows: Thx - Subotta "lax mec: Total - Subsotal Tlx. Sheer "Sheetl has 5 colames (Heders are A: "Tai"" B "VendeeClien"" C: "Fepense Accuim", Dr "Subtotil", F: "Tae", F. "Teeal") and 25 mos fincluling the hender "2001-01-08 00.00.00" "2020-01-02 00000000 "2019-12-31 00:00:0"" The cells the "VandorCClees" column
the 'Dite' column "Clieat A", "Client B", "Company A", "Compuyy The colls the "Expense Accouat" columa can "Reception", "Car Repais", " "Accommodatoo"" "Material Parchase", "Giss, "Meals", " "Flight tickess" 'Hiring. The cells in the "Sabooaal' column mage from 127.00 to 1954.00. Original instrctioon:
The
Calculate
the "Subiotal" column (Columa Dj ONLY the "VendorCCient" (Column B) equals "Company A".
new sheet samod "Sheei2", write a formols column match the "VendeeClen"" Collme "Shoe12" against "Vendor/CTien" in Colamm "Sheetl" If the "Expesse Accoun" (Columa C2) in "Sheer!" is "AAcommsdation"" copy the "Subooral" (Column D) from "Sheetl" to Column B of "Sheet2". 3. a column (Column G) with the header TTax Caleulation", calculate the tax by mulliplying the coresponding "Suhnotal" (Column D) 0.1.
Think about the flaws the original instructions before panpbeassng:
(GPT3.5 response)
1.
Think: The original instruction refers to oslumns by indices besckets, which undesired Flaw: The original insimutiin mentions the specific value "Compony A" for "VendonChien" column,
undesired.
Parsphrase: Calculate the sum of the "Subtotal" column if the "Vendan(Cient" is for "Company A".
2.
Think: The original instuction memions Excel built functions (i... MATCII and refers to columns by indices brackess which is undesired.
Paraphaase:
sheel namned "Sheet2", find the comespunding "Vendor"Cleet"" 'Sheetl" for row copy "Sabutal" 'Sheetl" new column "Sheet2' if the "Expense Account is 'Accommodatica""
3.
Think: The original instrution refors columns by indices in brackets, which undesined.
Paraphrase: Calculate the uax by multiplying the "Subintal" column by new column named "Tax Calculation'.
22

---

Table H: The official names and synonyms of the atomic actions used in our method. The Formula category contains the types of formulas used for calculation the core set tasks and does not represent specific operations.
Category
Official Name
Synonym
Write
RangelnpurValue
DiscurdRange
InsertRow
NewRowAttlddex
InsertColunn
ColumnCreation
AutoFill
RungeValueTransfrr
Entry and manipulation
CopyPaste
RoplicateRange
FindReplace
Alteritange
SctHyperlink
Link:RangcAssociater
RemoveDuuliicate
Distinetata
CreateSheet
WorksheeiCreation
Clear
EraseRangeContens
Sort
AdvancedRangeSort
Filter
SmartRaneSSllettr
Management
CreateNamedRange
SetRangeName
FreezePanes
LockRowsColumms
SetFormat
CustomizeFont
SetDataType
RamgeTyyeFFomuater
Merge
ConcatenateCells
Formatting
ResizcRowColuna
RangeeDincnsiooAdjusse
SeCConditionalFemmaat
FormatWithRule
SetCallLack
ProtestActiveSheet
CreateChart
GraphConsstructer
CreatePivatGraph
ScrCharTTitle
CharlTiteSettings
SerChartAxis
CustomizAAkiiAttribuuee
SeiCharlllaAAsis
AxisDisplayMaanaa Manager
Charts
SesChartLegend
LegendCConfiguratio
SeiChanType
CharTyyeeSwith
SetCharMarker
DefineGeriesMarker
SctCharTreadlin
TrendlinAAjustmenss
AddDataLahels
DisplayChartDttal.aae
AddEnorBars
ErrerBandnncegratinn
CreatePivoTTable
PiworlableCoosstuuttr
Pivol Table
PivoFamecionChhange
Date time functions
Logical functions
Lookup and reforence functions
Formula
Math functions
N/A
Statistical functions
Text functions
Financial functions
23

---

project all generated action names into 768d embeddings using the OpenAl embedding model and then reduce these embeddings to 64d vectors before performing Kmeans+ clustering. This process produces 80 clustering centers each of which represents group of semantically similar action names. After checking the names corresponding to these centers by hand, we find that these names are the official names used for the built-in features of Microsoft Excel. According to the potential use of atomic actions in the core set tasks and the implementation restrictions of the evaluation environment, proportion of these action names (44 out of 80) are used by our SheetCopilot (Listed in the initial prompt and used in the external document). These 44 atomic action names are used for conducting the comparison experiment (Sec. 5.2) state machine ablation study (Sec. 5.3). and the stability test experiment (Sec. 5.5) Please see Tab. for these atomic action names.
B.2 Collecting Synonyms for the Official Atomic Action Names
To investigate the confusion issue mentioned in Sec. 4.4 we attempt to adopt a different set of atomic action names. Specifically, we use GPT-4 generate 10 candidate synonyms for each atomic action name according to the low-level implementation of the action. Then, we adopt as the new name the candidate farthest from the official name in terms of the cosine similarity in the embedding space" These synonyms (shown in Tab. H are used in the ablation study in Sec. 5.4 investigate the impact of adopting different sets of names.
C Details of SheetCopilot Implementation
We introduce the implementation of the state machine our SheetCopilot is based on and the prompt used to query LLMs to generate task solutions.
C.1 State Machine Implementation
The Observe- PaeeeePProooeeeeeeeeeeee Revise-Act framework is implemented using a state machine shown in Fig. At the beginning of each step, the Observing stage enables the LLM to receive the sheet state before planning an action in the Proposing stage. Since the LLM is likely to output an incorrect action, an external document of the initially planned action is inserted into the query to prompt the LLM to revise the action. If the initially planned action is invalid, a validation error will occur and then the state machine will return to the Proposing stage; If the revised action is invalid, validation error will also occur but the state machine will return to the Revising stage. If the revised action has been validated, the action will be submitted to the Acting stage for execution. The revised action is still probably erroneous, causing run-time errors in the software. return to the Revising stage to prompt the LLM to re-plan according to the error information until the execution is finished. The entire process is repeated until the LLM outputs "Done!".
this case, the state machine will
C.2 Prompting LLMs to Generate Task Solutions
The initial prompt used to query the LLM to generate step-by-step solutions is shown in Tab. The prompt is formatted as a multi-turn dialog according to the usage of OpenAl ChatGPT API, with each turn comprising content field and role field. The prompt consists of an overall task goal, list of atomic actions, requirements, an exemplar, and finally a task be solved.
D Additional Ablation Studies
D.1 Qualitative Experiment of Missing Atomic Actions
We deliberately remove several atomic actions to test the robustness of our method. Specifically, we remove SerCondittonalFormat and test SheetCopilot on a task involving conditional formatting. We also remove another important atomic action CreatePivorTable- to see whether SheetCopilot is able to adopt other feasible solutions to analyze data. We use GPT-3.5-Turbe as the backend of our SheetCopilot. Fig. present surprising results: Example I shows that SheetCopilot uses filter to retain the rows fultiling the task requirement, set the formats of these rows, and finally cancel the
'The model used for embedding OpenAl hext-embedding-adaeeeeee
'The model used for embedding the same above.
24

---

Table I: The initial prompt used to query LLMs to generate step-by-step solutions. The prompt is formatted as multi-turn dialogue between user and agent. Note that the final turn of this prompt is an example task instruction and can be replaced with other task instructions to generate solutions corresponding to the task. For clarity, redundant contents are left out.
(eocsem:: You are a spreadsheet agent who can lin pruper action APIs from the API document based on the language insructions.
Here is the API documen:
Write a Arge: tringe: str, value: sn) Uage Write value into range The string value also be excel formulas.
CopyPaste Args: (stunce: destination: sei Usage: Copy source range and paste the destination range.
CuiPaste Args (source: str, destimatoo: stt) Usage: Cut the source rage and paste into the destination range.
Setttyperlink + Args (source: str, url: str) Usage: Set byperlink for the source range.
Removellyprrink Aups
Usage: Remove hyperfink for the
range.
AutFFill Args (source: sir, desination: str) Usage: Auto
destination range
source range.
Soat Rt Azps: (source: sir, key I; str ooder: strw use". orientation: strw columa') Usage: Soet the source raage by keyl.
Filter à Args: (source: str, heldinnes: in, criteria: str) Usage: Filter the source range based on fielilndes by crittria. DeletuFilter Args () Usage: Delete all fillers
MoveRow Args: (source:
destination: int) Usage: Move the source
the destination row.
MoveColumn el Args: (source: in, destinaton: int) Usage: Move the source column to the destination column.
RemoveLDopicnne Args: (source: st, key: int) Lsage: Remove duplicate values in the source range based on the key.
Seti-ermat (source: fort None, fontSizr: floas None, color: Noue, CollColor None, bold: bool None, calic: bool None, underline: boel borizontal/ligmeenn
Nome) Usage: Set formal
source range. you want set ple.se 'SeDDttTType' API.
SetDaaa Type Args: (source: str datilype: str) Usage: Set data type for the source range.
SetCellMeegg it Args: (source: sir, merge: booli Usage: Togele cell merge for the source range.
Delete Args (source: region: Usage: Deletes sell range cells. comtent
Clear Auge: (source: Usage:
formatting AutoFit 4 Args: (source: str) Usage: Auto in the width and height of all cells in the range.
Range.
ResizeRowColumma Args: (source: sir, wisth: floot = Nome, height: float None) Usage: Resize the width and height of all cellss in the range.
SetCondiionll-on onnonn Arps: (source: formular: bold: nool None, color. Nose, fillColor. Nose, italic: bool None, undectine: bool Nose) Usage: conditional formas
source range.
(Leave our for clacity)
FrecsePanes Args: fiource: sto) Usage: Freeze panes for current window.
CreateChart Args: (source: destSheet: chartEyp:: chartName: XFiekt int None, Wised: Listini) Usage: Create chart based the data from source range. Please note that you use from pivot tuble crease chart, use "CesneChartroonPPoottt instead.
(Leave our for clurity)
Requirements:
The will show you headers and row numbers of sheets you reference. explanations.
Please provide step- by step solutions
3. You can only be allowed to ue the action APIs listed above. You cannot use mny other APIs. Do not generate any new action APIs.
4. It should only renam one step a time and each step should only contaas one action API.
Please add both before and after each function call indicate that content between the characters function call, like Functionn)
2anction21j0..
6. The address of a raage must contain the sheet name, due to that there may be multiple sheets a wockbook:
7. You can only call the action APIs and directly use the values the sheet. You cannot use other variables.. sole system]
Cesteettllstrnettonn In column calculate profit for each week. 'hhnn format the nambers Accouninn Number Forrat
Sheet state: Sheet "Shoet"" (octive) has 3 columns (Headers are A: "Week", B. "Sales", C "CXCSS" and 11 rows, (the row I the header mww). Please provide the first step.
role
(ccetent/Step Creste new column
Action API: de Weiteerange- 'Sheett value="Proftt joe, role assistam]]
(heetem::Shet Sheet "Sheet!" columes (Headers A 'Week', "Sales", C: "COGS", D. "Prollt)) and II (the I the licader row). task not finished, please provide next step, otherwise, please type "Doae!". rolenser)
(ccetem:Step Profit is sales minus COGS
Action API, weWiterrangg- 'Sheetl 'D2", value- "=2--C2")0 mlleasista]]]
[eoetent:heet state: Sheet "Sheztl" has 4 columas (Headers are A. 'Week", B: "Sales", C: "CDGSS" D. "Prolt") and 11 nows (the row I the header row). If task is not finished, please provide the next step, otherwise, please "Doae!". role
(costemtStep Fill other rows.
Action API: @ Autooillll souoe= 'Sheet ID2",, "D2:miio=="DDDD1""" mlleasssttn))
(eeetem:Sheett Sheet "Sheet!" columas (Headers A. 'Week', "Sales", C: "COGS", D. "Prolt') and 11 (the I the ltcader row). If Frask not finishol, please provide the next step. otherwise, please type "Dome!". rale user)
(ccuten:Step 4. Change the formar of the results these are accomnting values. Action API: See SSyrDaaaTyyennnnngee "Sheet (D2:DII" daniType= "oinTyp=""crrncc"""" alleassistat]
Sheeten::heet state: Sheet "Sheet!" has 4 columas (Headers are A. "Week", B: "Sales", C. "COGSS" D: "Prolt") and 11 rows (the cow I is the header row). If task not finishol, please provide the next step, otherwise, please "Done!" role user)
Donelen:Denel. role
(The following tum shows the contest, instruction, and shret state of the task to be solved)
(vontentMM workbook has two sables. Sheel 'Sheetl" records the sales of boomerang company. Sheet "Retail Price' lists the retuil prices for all products. the sales table considesing corresponding price dascuunt Please help this the websites chart this new sheet for the pivol lable with churt tile "Sales frequency by website" und without legends.
Instraction: header 'Revenue". Then create a pivol tablle
calculate the
in new sheet
boomerangs were sold. Finally, plot a
Sheet state: Sheet 'Sheett' i has columas (Headers are A: "Dite Time', ' B: "Web Sine", C: "Product" D: "Type", t:: "Quaniiy", F: "Discount") and 36 rows (1 header Sheet 'Retail Prive" has columas (leenders Please provide first step.
and 35 data
'Prddctt, D: "Retail Price") " and 23 rows (1 header and 22 data
user)
25

---

Example
Instruction: Can you help me set fill color the Invoice column cells whose corresponding values Sales column range between 200 500 as green?
Solution actions:
Maado
971336
"ahe formularit 200 HillCooor 'green")
07000
02100
010
201/007000
07100
070.0
021.0
$19.30
Final sheet statc
Solation SattCanniiiinnaaooreeen missing:
538.00
1. Filterisurre- fiediinder",,erirriii==00000 (riiterii=="
2. SetFomat(source SS2:et::A::1199 filCoolor"greea)
2061
25-1500
110.0
895403
3. DeleteFiller)
105.00
162503
30 LG-08
37100
19010
50015
06013
2541522 E08
Qued
538.0
20-433
20.15/8
14000
10612
Final sheet slate
Example
Instruction:
Sort the data by the Date column in an ascending order. Then summarize the total roll numbers for each date new sheet. Then create a line chart in the new sheet that displays the trend over time.
Solution atomic actions:
1. Sontsounce- "Sheet 1A2:D26", "Syl="Shoe1AA2, order-"ase" orientationn "colunun")
CoomeShherrshhheeNaaaaa "Shoeet""")
CreatePivor Taalliioooaaassooeeee "Sheet "Sheet//SAS1 (SS1:5D526" destShes
name- "PivoTTable" RowFFildd"""" ColumnFFeldd-], PageField-[], DataField-("""))
"PivorTabll"" dest SherSheet"" "Seet=""hh"""""
chartName- Trend", chart"ype-Linne)
Final sheet state
Solation when CreatePhvolTable missing:
1.
SSmpoonce-SSeeellAA22 "A2:e21A2::220 key1 keyl="Sheet11A2 1A2", order""ase", orwentationn 'column")
CreseSheett
Sammary")
3. Writerrnge- 'Rol Summary1A11 AA]", value-"Date")
Witterange- 'Roll Sammary value="Total Rolls")
Writaraagee-RA 'Roll Summary!A2', value=" !A2")
Writefrange- Roll Summanyy (B2", SUMiII Sheee11AAAAAA1
AutoHillsouree- SummurytA2.88 82", destinutio="Roll SummuryAA2H222
8. CresseChan(sommre- "Rall Sammary\A1:9266 destSheet Roll Summary""
0736
chartType Line". chartNae- "Roll Trend")
SoChar SaCCharTalelchataaaa Talocchattaaee- 'Roil Tread", litlo-"Roll Trend Over Time")
238
Final sheet state
Figure F: Two example solutions generated with missing atomic actions. For each example, the instruction (top), solutions generated with full/missing atomic actions (left), and the final sheet states after implementing the two solutions (right) are displayed. The two examples show that SheetCopilot demonstrates strong robustness by using a combination of other atomic actions attain the same goal which can be more efficiently attained using the missing atomic action.
26

---

filter to restore the data. Example 2 shows that SheetCopilot cleverly utilizes the SUMIF function to substitute the pivot table. Despite redundant calculation, obtains functionally correct results.
D.2 Detailed Analysis of Atomic Action at Different Granularity
We provide further details about the ablation studies on the action granularity (Sec. 5.66in the main paper), In our design, the low-level implementations of the atomic action used by SheetCopilot are consistent with the atomic actions used by Excel in terms of behavior, which means that our atomic actions possess normal granularity and are familiar to average users. However, one may argue that high-level atomic action that combines several actions used by our SheetCopilot may bring better performances as using high-level actions possibly improves efficiency.
To investigate this interesting problem, we combine several chart-related atomic actions into one high-level chart manipulation action and conduct an ablation study on the tasks involving charts. Specifically, incorporate the actions that set chart properties into the CreateChart action and remove the original separate actions, expecting the SheetCopilot to set all required properties on the creation of the chart. Additionally, the usages of these actions are merged and new example of showing how to use this integrated CreateChart action is also added (See the documents of the separate actions and integrated action in Tab. D. Additionally, in another experiment, we split SetFormat which originally contains multiple arguments related to setting various format properties to see the impact of using finer- geeergg cermat-setting actions (See the documents of the SetFormat action and finer- seeerggainnd format-setting actions in Tab, K]. The SetConditionalf Format action possesses function overlapped with that of SetFormat, thus interfering with the experiment. Therefore, we remove the Be(Conditionaloomma action when conducting this experiment.
We conduct this ablation study with GPT-3.5-Turbe as the SheetCopilot backend on the chart-related tasks (43 out of 221) and format-related tasks (41 out of 221). The results in Tab. show that using high-level CreateChart action handle chart creation and chart property setting simultaneously obtains slightly higher efficiency (lower A50 and A90). However, it is surprising that this method en- counters significantly inferior functional correctness, with Exec@1 and Pass@1 decreasing by 12.6% and 6.1 %, respectively. Likewise. splitting the original SetFormat action witnesses considerable performance gains in Exec@ and @
After analyzing the chart-related results, we found that using an integrated CreateChart encounters lower functional correctness since its document is so complex that SheetCopilot struggles under- stand the action usage, thus being less able to correctly determine all action arguments as required. In addition, the document of this integrated CreateChart action is extremely lengthy, causing the query to frequently exceed the GPT-3.5-Turbe token limit. These results suggest that it is more desirable to use atomic actions instead of integrated high-level actions in terms of functional correctness. After analyzing the format-related results, we observed that using the original SetFormat action tends to result in argument hallucination. For example, the LLM frequently adds a non-existing argument such as "criteria=..." trying to use SetFormat in a way similar to the removed SetConditonalFomma The LLM outputs such erroneous action repeatedly even if the error feedback is provided, finally exceeding the token limit and causing an execution error. In contrast, after splitting the SetFormat action, the LLM able to easily understand the simple grei-gr actions, thereby encountering fewer hallucination cases.
SheetCopilot Manipulation Example
To better illustrate the strengths of our SheetCopilot, an example of using SheetCopilot (GPT-3.5) to produce solutions users' requests is shown in Fig. G SheetCopilot is asked complete the sales data, analyze the results, and visualize the trend. SheetCopilot accurately understands the intention of the user's instruction and correctly revises its solution utilizing the external document of the chosen atomic action and the error feedback provided by the evaluation environment.
Analysis of failure cases
To obtain a clearer view of the distinction between the L.MSs compared Sec. S.c.5.22 we perform an in-depth analysis of the failure cases for each LLM. The predefined failure types are as follows:
27

---

Table J: The documents of the separate chart-related atomic actions (left column) and the integrated CreateChart action (right column). The arguments and usages of each separate action are merged and a new example of using the integrated CreateChart is provided in the document of the integrated CreateChart action. For clarity, the redundant contents are substituted with ellipsis.
The docaments the sepantte chart-relited actions
document the integrated CreatoChart haart action
CreaseChart:
aege: "(oonre: str, destSheer: sir, cye str, chariName: str. XFickt int
None. YField: List[nn] [II'
args explamsion:
source (string): The range which contains the data used create the chart. destSheet (string:: The
of the sheet where the than will located. type of
'Amaa. 'AraStacked'' "Ba Blasteeterd"' , 'BaOOIPie,, "BarStacked'. 'Bubble'. "Colamn Clusteedd " ColumnStackeed 'Line", 'LineMarkers', : 'LineMurkenSticcee" 'LineS- tacked". "Pic', 'XYSeater', "XYSettter ines' N NoMakke'... "YYScunerSmocs" XYScatteSnnootNNoMaa
chariTppe (sting)::
'DDPie'.
NhmmtName (strnn):: The nime the chart to be created.
XField tint:: The index of the column which contains the X values, starting from XTield is None.
finst column will be used.
YField (List(int): The indices the columns which cuntain Y values. starting from 1. Ficld is II. all columns except the first columa will be used. usage: Create a chart based on the data from the source range. Please nose that if this from pivot table create chari, the API "CresteChurt- FromPivoTaable instead.
CreateChart:
arpe: "(soure:: deutSheat: str, chinTTye: chartNam: sir, XField: None, YFied: Listjiat] []. rile: None, titeeizze: float None, titkBood:
example:
None. legendPositio: air None, legendSize: flom = None. legendNames: list = II. histeroBBass bool None, heatheal bool - Nonc, markes markerStyle: Listaar] None, mikerSize: DindliseDipphiy Fquntio:: boal None, hendlineDisplly
None, titleColor:
None, batt.gged::
e Excampl 1: Cieate a chart in Sheer2 based on the data from Al to H10 'Chart'.
CreateChantssume 1A
Sheetl the chart
Typee" ColumnClusteree chartName- "Charl"" a After implomeming this action, a chart named 'Chartl' will be created in Sheet2 based on the data from Al B Sheet!.
SSeeeheet=Sheeee2',,
RSqurad: Renpiie Nose, haxXAxis: bool - None, XAxisTille: str - None, bactAsis: bool None, YAsisTille: None)"
None, trendlinx Type: Lis(str]] Nune.
hool
arga explanation...
usage: Create a chart
from the source range also soe
SetCharTTile:
properties at creation time Note that is not alkowad to set the propertios for exisling Chirt. Please noe that if you use data from a pivor table create ChesschhrlruumPivvttt
volor: None)"
"chartName: title:
fontSize: float Nose, bold: book None.
instead.
args explanation....
a Example I: Create a chart in Shear? hased on the data from Al to BIB in 'Chunt'.
examle::
example:
Sheetl and the churt CreuteChantfsmmees SSheet 1A1::110.
SouShoet="Sheerh"
Aart
"o-"-CCooCCuttttaa chortName- Churll")
args explanasion....
args: YcmartName: st,, axis: str, hasAkiis: bool)'
i Afler implemening this action, a chart numed 'Chartl' will be created in Shor12 hased en the data from Al to B10 Sheet
example:
based set the chart title 'Chrril Thlle".
Esample Create
data from
Sheetl and
SetChatAAii::
Creatshhrnjjameee Shheet1AAl 11A1:100.
deschhott SShott" chart-
args: "(chutNaae: str, oxis:: str, title: None, InbelUricutttin:: str - None. Valle: flout None, mini Value: Boot Noney" args explanation.... ... example:
Type= ColumnClusterrd; chartNamez' Charil', title Chartl Tilee) implemening action, chart Sheet based the data from B10and the churt tille will Tiile"
'Chrrtl' will created
Chortt
# Example 3: Creste chart named 'Chrrll' and set till, marker, X Y-axis legerd, trndline.
SesChartHad
args: (chartNase: str, hasLegond: bool)"
Chantshhass "AAee..AA::CO... descShoet= 'Sheel'' chart- Type= "ColumCClsstend closteedd chuntName= CCharl" XField=l, 1, (Fi3
aegs esplanation....
example:
titk== (Caar Tide'. hasLegend-Trae. legendPesition== bottom' mark- erStyle=] cincle, Irianglee). condllineType== peee:oollyommal"" ). trendiineDs- playEquation-Tro DrepllinDiilll R.quurad:True.
SesChard egend:
args. "cchrtNamz: str, pasition: str Noue, fomtiize str Nose, seriesName:
XAxisTikk='s X axis YasiAiisrrrre, MoxisTilee Y-xis')
list III
a Afler implemening this action. chart named 'Chrrtl' will be created in Sheet! based the data 'Chattl Title. The finst colume will be used as X values and the second and third volumss will be used as Y values. The legend will be displayed at the bottom. The first have inanggle marker. polynomial trendline the equation and R squared will be displayed. The X X-axis tide will be set to "X-axis" and the Y-axss tide will be set to "Y
args explanation....
Al
will
example:
SesChurTypp:
args: "cllanNNane: str, charttype: stry'
have
marker and the second
will added to chari and
args eh explanation.... ... example:
Addaaaa abdle:
aigh: Naame str)'
agg explanation.... exomppee:
AroChan/rroBarrs:
argh: "CchnnName: str)"
args explanation.....
example:
SetChartMurker:
"(chunNaae:: str, style: List[str] None, size: flost Nenoy"
segs explanation:...
usage: Sei marker for the churt.
eximple:
28

---

Table K: The documents of the finer- seee-grained format-setting actions (left column) and the original SetFormat action (right column). The argument and usage of the original SetFormat are both divided as the arguments and usages of the finer-graing actions. For clarity, the redundant contents are substituted with ellipsis.
documents finer grained setting actions
document original Formota action
SetFunt:
args: "courre: str, font: str''
args explanation:
(sting): The range set font.
font (string): The font mage: Set
the source range.
example:
i Example 1: Set fort for the (A1:R6) to 'Arial". fonte"
Set SedFunilsource= "Sheett
Afler implementing this action, the range (A (B) will be set to "Arial" font.
SetFomiizz::
args: "(souree: fontSize: floun)"
args explanation:
usgger Set font size for the source example:
# Exumple I: Set form size the ramge (A (BB66) 20,
SefFomSizessurreee Shoet IA 186", fom(Size:20)
After implementng this action, range (AI:B6) will be 20 font size.
SefFontColor:
args: (sourve: str, color: stry"
args explanation:
usage: Set font color for the source range. example:
SetHommat:
arga: "(smnre:: sir. fort: sir None, fintSiie:: float None, collet str = None, BilCoooor int Nonc, bokt. baol - None, talie: hool - None, underline: bool Nane)"
See LerFonCCluntsuunee "Sheeil!A :D" color="red") a Aferr implementing this action, the range (A (B6) will be set to "red" font
Example 1: Sons color for the (Al (-) 'red'.
None. botizotallligameen: args explanation:
source (string): The range to set format.
color.
SedFillCooo::
font (string): The foot set. fouiSine (feenty:
args: (soumre: CillColee
args explanation:
"yellow", "magenta"" 'eyan', 'dark rod", 'dark groen'. iinColoe (serin):: The fill color set. It he "black", "white", 'red', "green', "blue" "yellow". "mageata' "cyas', 'dark_red", 'dack grren"".. bold (buoly Whether to set hold. True means bold. False means hold. tilic (hool): Whether to set talie. Tiee means tilic, Falso means not talic. underline (booly. Whether set undertite. True means underline, False menos underfine.
color (string): The color
"black", 'white", 'red", "green", "blle',
usage: Set fill color for the raage. example:
Exumple I: fill color the range (A (BB 'red'.
bethillcollossoorree SSheet -ol- fittColor="red After implemening this action, the range (6) will set 'red"
BaiioettlAlignmeet (string': borizontal aligament set, can "caster". "right".
SetBold:
args: (sourve: str, bold: booly"
neage: Ser format for the sounce you want to set data type, please use "SerDheaaTpp""
arg explanation:
example:
example:
# Exsmplc 1: Write hold text "Summer sales (5)" with blue fill color and white text color in Al.
e Example 1: Set hold for the range (A).
Afler implementing this action. range (A1:B6) set Setitalie:
sourood 'Sheet!
beld-Thue)
SetFonmatt Sheel (hce hold=Tree #iCCoee: "ilue", coloe= "white")
Writo""Shed
'Summer sales (57)
# After implementing this action, the eell Al will contain bold text "Summer sales (5)" with blue fill color and while text color.
args: (source: str, italic: bool!"
args explanation:
Fxample Adjust columi Set Lefoomail" "SheeilCCC". font= 'Arill' underfine=:ruu True)
Arial font underline.
example:
i Aue implementngg this action. the column C will be adjusted to Arial font
e Example I: Set italie for the range (A1:86).
Settralic'soures 'Sheet 186 italice True)
with underfine.
After implementng this action, runge (AL:36) set italic.
"sounve: undertine: bool)*
arg explanation:
mage: Set underfine for the source range.
exanagl::
Extumple 1: underline range (A1:86)
Sest iettfffin/ssuueee- "SheeeeeeShhett... "Shaei!AA A1166 underlin:-Tre))
i After implamenting this action, the (A1:66) will be set underline.
SctHorioonlllignnee
args: (source: str, BoizootalAAlgnnnse str)"
args esplanation:
(string): The range horiccatal aligoamett.
borizontalAiggneent (stiing:: horizonral alignmest set, be 'left',
"canter". "right"
usge:: Set hocizcotal aligament for the souree range. exanapl::
Exumple 1: horizontal alignment for the mago (A1-66) to "left".
SetHorieetalAiignee aattallliigmmttissreee. 'Sheet 1:66. horiscatalAl pnroouaalAliggimee-e genggnnnnn"aff "left"
# fle implemening this action, the range (AI:B6) will be set "left' hori- zontal alignment.
29

---

Turpet sorcodlee
My nomthook Shpp "Sunl" ronde
onepa Shact RRcaa Piic urall prodacts
Date me Wobt b. Prohhct
Typ
Quietty
Wholess
12/1N11 18:10 armaooni Bellon
Frral
0013
Brcert 1255 enap comm Quaed
Wholhes
6/00015 16:0 atadietee kipea LD
Resal
Mha
T/V/1 18:6 onto Belen
Proci
to tristagge Shont Gi walie "Raviaa)
A11
Hous glain
Woilin Seaatitt 1000000 RRcal PrletAA 112
We Skt Peckat
Sype
Quarity
12/33/3 2828 coa--bezecn// Yanal
Wrodesc
002
Rycal
0001
72.85
24013 ES colccakk Grodkdd bennkouc
035
E0003 abayae
B00/12 2400 gel bocir Fin Apporie menoocck
0,07
12
11rtac7s 22th weeeeeooo
Couacheed Eullatttt luticae "Saetii KCa.
2r tocoeweg ccn
caechhane Chart", darloope Tol's
ded hert Jabeborerr Ber beil ppraii
Dasc!
Figure G: ShectCopilot example: Handling sales data. The left column shows that ShectCopilot generates a step-by-step solution according to the sheet state feedback and correctly revises its mistakes using the external atomic action document well as the error feedback. The incorrect arguments are marked with red rectangles. The right column demonstrates the state changes of the evaluation sheet corresponding to each step the left. For illustration clarity, only brief documents are displayed.
Apuroderpe
Sccroles antaten (1124))
Brinn Slrn
Iiceppor wwt.itteeeeeeteeee
Deshoy vep se Mi
Devieg modlecc Hah
Wrig Wwa.atintii
T.S
GPT-4
Clude
Figure H: Comparing the proportions of different failure cases of the three LLMs. The evaluation datasct is the 10% subset. The charts show that GPT-3.5-Turbe and GPT-4 share similar failure patterns while Claude exhibit a different one.
30

---

API inability (2.5%)
Incomplete solution (18.9%)
Wrong formula (25.4%)
Hallucinatic data
Repeated output (4.9%
Wrong action (14.8%)
Wrong range (29.5%)
Certain arguments unset (0.8%)
Disobey requirements (0.8%)
Figure The proportions different failure cases when evaluating GPT-3.5-Turbo on the full coreset.
Wrong action Use wrong atomic actions attain specific goal. For instance, hide columns using a filter (a filter can only be used hide rows) or perform meaningless actions to destroy the data.
Wrong range Select the wrong ranges to manipulate. For instance, fail to use absolute references when required or select incomplete ranges as chart data sources.
Certain arguments unset Perform the atomic actions required by the tasks but fail to all required arguments. For instance, set the cell background color but forget to set the cell font color.
Repeated output Output the same atomic action repeatedly until exceeding the token limit.
Hallucinate data Hallucinate data to manipulate while ignoring the real data.
Incomplete solution Terminate solution without fulfilling all task requirements.
Disobey requirements Perform atomic action in way that deviates from the task requirements. For example, insert a hyperlink different from the one specified in the task instruction.
API inability Failure caused by the limitations of the low-level implementations of the atomic actions. For instance, the low-level implementation conditional formatting does not support all types of formulas (e.g. array formulas or formulas available only in specific software version).
The categorized failure cases of the three LLMs are shown Fig. From an overall viewpoint, we obtain two interesting findings: GPT-3.5-Turbo and GPT-4 share similar patterns of the failure case proportions while Claude demonstrates largely different failure pattern. This finding suggests that the two GPT models possess almost identical alignment in terms of spreadsheet manipulation and that a gap exists between the alignment Claude and those of the GPT models. 2) Jointly analyzing the failure cases and the performances in Tab. by SheetCopilot (See the prompt in Tab. Mb is tuned for GPT-3.5-Turbe Claude exhibits competitive performances compared with GGP-3.55TTurb
that although the prompt used
Inspecting individual cases, we found that GPT-3.5-Turbo tends to use wrong formulas, showing that its ability to predict formulas is weak. GPT-4 is prone to incomplete solutions due to the limited context window. In contrast, Claude encounters no incomplete solutions and less frequent repeated output but tends to wrong formulas and wrong ranges. One common failure case among the three LLMs that they often use a combination UNIQUE and SUMIFAVVEAAGEIFCOUNNT
to replace pivot tables. This combination is technically correct but these three models forget to apply absolute reference when selecting manipulated ranges used by auto-filling, leading to incorrect calculation results.
also check the failure cases made by GPT-3.5-Turee on the full core set (See experimental results in Tab. [ and illustrate the percentages of these cases in Fig. [ The pie chart shows that GPT-3.5-Turbo performs poorly at predicting formulas and determining appropriately selected ranges. These two types of failure occupy over 50% of all cases. Inspecting the wrong formula cases,
31

---

found that OPP--3.5-Turbo often fails to use absolute references when necessary, uses self-reference, and uses wrong criteria for a filter or conditional formatting. Inspecting the wrong range cases, we found that GPTr-.5--Turbo tends to use wrong auto-filling ranges, use wrong chart data sources, and leave out the headers when copy-pasting columns. Another two prominent types of failure are Wrong action and Incomplete solution, which occupy 33.7% in total. After inspecting the wrong action cases, we found that GPT-3.5-Turbo misunderstands the usage the atomic actions incorrectly uses. For example, GPT-3.5-Turbo inserts an unnecessary column to fill in data, ignoring the existing column where the data should be written; GPT-3.5-Turbe also uses a filter to hide columns, which violates the actual usage of a filter. After inspecting the incomplete solutions, we found that GPT-3.5- fails to fulfill all task requirements in these cases although other steps are correct (e.g. forgets to set the chart title or to autofill after filling in the first row). In the cases where repeated output occurs, GPT-3.5-Turbb generates the same action repeatedly as it fails to revise its plan according to the given error feedback. GPT-335-TTubb occasionally misses setting required arguments disoheys the task requirements to set incorrect arguments which both make up a small fraction of the failure cases. A small proportion of the failure cases result from the implementation limitations of several atomic actions, which can be fixed by upgrading these implementations.
G Demo
Video demos for our SheetCopilot agent can be found on our project website https://Sheeccoolooogghhbbbbbbb
H
Limitations
Our SheetCopilot possesses the following limitations:
We have not optimized the number of tokens used to solve spreadsheet tasks. This means that SheerCopilot consumes large number of tokens and is only capable of solving short- horizon tasks. Future work is needed to focus on designing efficient prompting methods that save tokens or using L.LMs with a larger context window.
State feedback only consists of the basic information of cells, ignoring the state of charts, pivot tables, filters, and frozen panes. SheetCopilot probably creates charts and pivot tables repeatedly as it is unable to observe the outcome of creating these elements.
Our dataset is manually curated and is not fully fledged yet. More labor work is required to generate more ground truth as numerous correct solutions to each task probably exist. Moreover, the more solutions are collected to conduct evaluation, the more accurate and stable performances we can obtain.
Our evaluation environment does not support all Excel built-in features, which restricts the task categories our SheetCopilot able to handle. More effort is needed to continuously upgrade the evaluation system to accommodate more diverse tasks.
Broader Impact
Our proposed SheetCopilot possesses the potential to significantly simplify the process of spreadsheet control, which positively impacts productivity and efficiency. By enabling users to interact with spreadsheets using natural language, SheetCopilot reduces the time and effort required to perform wide range of tasks, such as simple data entry and complex data analysis.
However, it is imperative to consider potential negative impacts. One concern is that the increased case of spreadsheet control possibly leads to an over-reliance on automation, potentially reducing the need for human expertise in certain areas, which may cause unemployment in society. Additionally, there is a risk that the use of SheetCopilot exacerbates existing inequalities in the workforce as those who lack access to SheetCopilot who
proficient in using may be disadvantage.
Another potential risk of using SheetCopilot that a wrong operation likely to face undesirable consequences when controlling sensitive data forms, such as tax forms or financial statements. This risk highlights the importance of ensuring the robustness and accuracy of the agent's actions, as well the need for appropriate safeguards and oversight minimize the risk of errors.
32

---

Additionally, SheetCopilot relies on the capabilities of language models, which are known to tentially inherit biases present in the training data. As a result, SheetCopilot may inadvertently exacerbate existing biases when interacting with spreadsheets and processing data. It is crucial to acknowledge the potential for inherited bias from L.MMs in the development and deployment of SheetCopilot and to take proactive steps to mitigate any negative impacts on fairness and equity.
33


Reviews:
[{"review_id": "c0NfooE1Rr", "reviewer": "Reviewer_bHxV", "summary": "This paper introduces SheetCopilot, a model that aims to generate step-by-step executable command sequences for software control according to the natural language description. Besides, a benchmark dataset for evaluating software control tasks is collected. Experimental results based on the dataset are reported.", "questions": "Is the collected dataset publicly available?", "limitations": "None.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "-\tGenerating step-by-step executable command sequences for software control according to the natural language description is a valuable problem.\n-\tThe proposed framework SheetCopilot takes a set of atomic actions as an abstraction of spreadsheet software functionalities and contains a state-machine-based task planning framework for LLMs to interact with spreadsheets, aiming to translate high-level task description into executable command sequences, which generally makes sense and is actually novel.\n-\tExperimental comparison against baselines (including the state-of-the-art LLMs like GPT-3.5 & GPT-4) has shown the performance advantage of the proposed method. Besides, the ablation study indicates that the design of the proposed framework is beneficial for improving performance.\n-\tIn addition, a comprehensive dataset containing 221 spreadsheet-related tasks is collected in this work for performance evaluation, and this dataset could be further served as a benchmark dataset in this area.\n", "weaknesses": "-\tReporting the stability test results through the line chart may be clearer. Besides, there is still some space to report more empirical evaluation results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive and insightful comments. Thank you for pointing out the strengths of our paper. Your concerns are addressed in detail below:\n\n**W1: Reporting the stability test results through the line chart may be clearer.**\n\nThanks for the advice. We have conducted extra experiments at temperatures 0.4, 0.6, 0.8, and 1.0. The line charts in Figure (a) in the response PDF show that the difference between the highest Pass@1 (47.66%) at temperature = 0.4 and the lowest one (44.30%) at 0.0 is slight and the changes of the other metrics are also slight. These results suggest that SheetCopilot achieves stable performances even if the GPT-3.5 API temperature changes from 0.0 to 1.0.\n\n**Q1: Is the collected dataset publicly available?**\n\nYes. Our dataset has been publicized on PapersWithCode anonymously. The reviewer can also check our dataset (including the task spreadsheets, task instructions, and ground truths) in the supplementary zip file.\n\nWe thank the reviewer again for the thorough review and feedback."}, {"review_id": "foA93av8Lm", "reviewer": "Reviewer_XbAE", "summary": "The paper proposes a benchmark and a framework based on observe-propose-revise-act for tackling spreadsheet problems. ", "questions": "- It would be good to discuss the performance-latency tradeoff in terms of #queries to the LLM, and the memory and compute used for inference.\n\n- It would be good to have a user satisfaction metric as well, e.g. if the users were satisfied by the time it look for the model to accomplish the task, etc.\n\n - Line 112: What are LLM-based filters?\n\n- It would be good to show the improvements brought by the framework on top of relatively smaller LLMs.", "limitations": "There is no limitation discussed in the main paper. A discussion about the increased computing requirements during inference and generalization to complex settings should be included.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The paper is very well written and the ideas are neatly presented with figures and tables. It was a pleasant read!\n- The ablation studies are quite interesting. It is interesting to see the effects of external documents, usage examples, state of the spreadsheet, and error messages separately. I also liked the ablation with the synonym substitution for official atomic action names.\n- The paper proposes a comprehensive framework of observe-propose-revise-act, which even though shown to work on a restricted domain of spreadsheets, can be used as inspiration for other systems, though it is not clear how scaling it up to more complex scenarios will work in practice.\n\n", "weaknesses": "- My main concern is over the benchmark creation process. Since the problems are picked up from a public domain site, I think there is a high possibility that these LLMs might have already seen this data during pretraining. Under this assumption, it becomes hard to understand the generalization capability of the proposed framework for unseen tasks. The authors don't discuss any deduplication efforts they performed to ensure that the performance of the shown metrics is not bloated.\n\n- Parts of the framework have been proposed by other works as discussed in the related works. However, I would consider the framework as a whole as a novel contribution. At the same time, I believe that the tasks tacked by the paper can be considered too simplistic for these LLMs ( especially since it is shown that the LLM achieves 100% performance on some categories of these tasks). It is not clear to me how the proposed framework would scale to more complex settings like a Copilot for programming where it is difficult to retrieve the relevant state of the environment, external documentation, or usage examples. Even if there exists a way to get this information, it is unlikely that it will fit within the context length available in the input prompt.\n\n- From my understanding, there is no timeout used and the LLM is queried multiple times until it gets the right answer. If this were the case, this is not a realistic assumption when deployed in practice. Users would not wait for long periods until they get the required task accomplished.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful review and feedback. Please see below for answers to your questions.\n\n**W1: hard to understand the generalization capability … discuss any deduplication efforts** \n\nThank you for this constructive comment.\n\nA direct way to test generalization is to split our dataset into two parts - one containing tasks whose original questions were asked before 09/30/2021 and one after. Since the data cutoff for GPT-3.5 is 2021/09, this allows us to evaluate performance on truly unseen data. Conducting this experiment, we found SheetCopilot achieves Exec@1=89.1% and Pass@1=44.0% on the before split, and Exec@1=80.4% and Pass@1=45.7% on the after split. The similar performance across splits suggests SheetCopilot exhibits generalization capability.\n\nWe conducted deduplication when collecting our dataset (see Section A.3 in the supplementary). Through the adaptation process, task instructions differ from the original SuperUser questions. Tasks also apply to new sheets rather than those mentioned on SuperUser. Moreover, ground truth solutions are step-by-step, unlike the colloquial SuperUser answers.\n\nTo confirm our data can be considered unseen, we calculated the maximum similarity between each task and all SuperUser questions. Similarities ranged from 0.06 to 0.16 (avg 0.10), indicating slight semantics overlap.\n\nWe also calculated ROUGE scores between our tasks and SuperUser questions, finding ROUGE-1=0.38, ROUGE-2=0.17, and ROUGE -L=0.35. The low scores further confirm a slight overlap.\n\nIn summary, the dataset overlap with SuperUser is minor and SheetCopilot indeed owns generalization capability.\n\n**W2: How the proposed framework would scale to more complex settings**\n\nOur main focus is to address language-instructed spreadsheet manipulation. Actually, applying our method to coding tasks presents an interesting future research direction.\n\nAs a preliminary idea, we propose treating a program's logging and error flag status as part of its environment state and using API documentation for programming languages and libraries as external documents.\n\nRegarding the potential context limitation, techniques like retrieval-augmented generation based on vector databases could help mitigate this issue when working with large-scale external documents.\n\n**W3: Users would not wait for long periods**\n\nWe agree that fast response is beneficial for increasing users’ satisfaction.\n\nOne cause of long waiting is deadlock due to the repeated output of LLMs (this failure occupies 4.9% as shown by Figure I in the supplementary). However, SheetCopilot's solutions in these cases still contain partially correct mid-steps. Users can employ the interactive mode (see Figure (b) in the response PDF) to abort and relaunch queries, escaping deadlocks. In contrast, correcting mistakes in code-based (VBA) methods is more difficult without easy interaction.\n\nAnother cause is the long inference time. Fortunately, many methods of accelerating LLM inference, such as KV-cache [1], have been proposed. These could be applied to reduce query time when deploying SheetCopilot.\n\nWe will add timeout and acceleration mechanisms in future updates to improve response speed.\n\n**Q1: The performance-latency tradeoff in terms of #queries to the LLM**\n\nWe agree that latency should be optimized if we want to productize SheetCopilot.\n\nCurrently, the majority of inference latency stems from our lengthy system prompt (typically 2k~3k tokens), while each generated step seldom exceeds 100 tokens.\n\nA potential optimization for reducing latency is precomputing SheetCopilot's fixed prompt once and caching it via KV-cache [1] for subsequent requests, thereby reducing the query time to the time required to generate ~100 tokens.\n\nBesides, performant open-source LLMs like llama2-7b [2] could already run at a speed of >150 tokens/s on a single 4090. In the future, this could potentially enable SheetCopilot to interact with users within hundreds of milliseconds.\n\nOverall, leveraging the fast progress in open-source LLM inference infrastructure to boost SheetCopilot user experience presents an exciting direction for future work. We will incorporate a discussion of this topic in the final revision.\n\n**Q2: good to have a user satisfaction metric**\n\nThank you for the suggestion. While human evaluation is indispensable for human-computer interaction research, it is beyond this paper's scope, which focuses on assessing LLMs' capacity for complex software control.\n\nNevertheless, we have implemented SheetCopilot on Google Sheets (see Figure (b) in the response PDF). We will use this platform to conduct satisfaction experiments in future work.\n\n**Q3: What are LLM-based filters?**\n\nThe LLM-based filter is ChatGPT used to classify Q&A pairs (see Section A.1 in Supplementary). Specifically, we prompted ChatGPT to classify each pair. After classification, we remove ~2.4k pairs labeled as “Invalid”, thereby filtering out irrelevant pairs and obtaining a cleaner dataset.\n\nAlpaca [3] and Self-instruct [4] adopted this filtering approach and inspired our work.\n\n**Q4: the improvements on top of relatively smaller LLMs**\n\nThanks for the advice. Please refer to Q1 in the global response above.\n\n**Limitations: There is no limitation discussed in the main paper**\n\nThe limitations of SheetCopilot are enumerated in Section H of the supplementary. We also list them briefly here:\n\n- We have not optimized token usage for solving spreadsheet tasks. Future efforts should save tokens or use LLMs with a larger context window.\n- Due to the token limit, state feedback provides only essential cell information.\n- Our evaluation environment does not support all Excel built-in features (e.g. array formulae).\n\n**Ref**:\n\n[1]Efficiently scaling transformer inference, 2023.\n\n[2]Llama 2: Open Foundation and Fine-Tuned Chat Models, 2023.\n\n[3]Stanford alpaca: An instruction-following llama model, 2023.\n\n[4]Self-instruct: Aligning language model with self-generated instructions, 2022."}, {"review_id": "CqoTOGIJho", "reviewer": "Reviewer_TF2z", "summary": "This paper introduces an LLM based spreadsheet manipulation task system from high level human language to spreadsheet manipulation tasks. The work is in the area of tool augmented LLM systems where LLM systems are used to create a chain of actions representing a complex manipulation task in a spreasheet system. The authors create a curated dataset consisting of a list of realworld spreadsheet tasks. They also build an assessment system. They study 3 popular LLMs - GPT-4, GPT-3.5turbo and Claude. They surprisingly find that GPT3.5turbo to be the best performing and also through ablation studies find that close-control action loops improve functional correctness, bringing in external docs improve correctness and surprisingly find that using synonyms far from the actual names also improve ", "questions": "1. Reference to Table 1 seems off. The citation says something and the table is about something else.\n2. Is the dataset for creating tasks and benchmarking available to the public to compare.\n3. Did you try smaller open source LLMs and see if they can be useful or how far you can get with them. \n4", "limitations": "1. There is not much study on the nature of the dataset curated. But the paper is just at the beginning of something interesting. While it is unlikely that one may see issues with spreadsheet data in discrimination or bias or fairness it would be good to address them. \n", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The authors take on the challenging problem of automating complex modification task in spreadsheets rather than formula repair or generation. If it works can be of big benefit to spreadsheet users.\n2. The authors create  a curated dataset of tasks in a systematic way. \n3. They also create a systematic evaluation system for evaluating the performance of the different popular LLMs in the context of the spreadsheet software system. ", "weaknesses": "1. I would have liked to see a better discussion on the task colletions, the Q&A pair collection, how complex these tasks are, their categorization, why they are multicategorized, what do those \"practical realworld spreadsheets\" look like etc. \n2. The techniques using LLMs are fairly straightforward with minor modifications like using synonyms (not sure why this works) adding documents etc. What is specific to the worksheet manipulation scenario that makes LLM easy or hard or different to work with. Lack appreciation of that. \n3. Further investigation into why GPT-3.5Turbo is better than even GPT-4 would be useful as it is counterintuitive. \n4. Instead of just exec@1, pass@1 softer metrics would be useful in understanding where these LLMs make mistakes and where they are good. \n5. Instead of using a tag cloud for dataset evaluation something detailed metrics would have been more insightful. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed review and insightful comments. Below, we address the concerns:\n\n**W1: a better discussion on the task collections, the Q&A pair collection, how complex these tasks are, their categorization, why they are multi-categorized, what do practical realworld spreadsheets look like**\n\nSection A.1 of the Supplementary Material elaborates on task collection methodology.\n\nWe collected ~27k spreadsheet Q&A pairs from SuperUser, filtered irrelevant pairs by keywords, further cleaned them by prompting ChatGPT to label invalid pairs, and selected representatives through clustering, resulting in a clean Q&A pair dataset.\n\nFigure A of the Supplementary Materials visualizes task complexity, with instruction lengths ranging from 20 to 530 words. Our dataset encompasses simple and long-horizon tasks requiring up to 9 actions across sheets. The tasks are stored as a .xlsx file in the supplementary zip file for ease of viewing.\n\nTask categorization occurred concurrently with LLM-based filtering (see Section A.1). Specifically, we prompted ChatGPT to classify each Q&A pair into 7 categories (i.e., Entry and Manipulation, Management, Formatting, Charts, Pivot Tables, Formulas, and Invalid). Figure (c) in the response PDF presents an example of categorization.\n\nThe tasks were multi-categorized since most of them involved spreadsheet operations belonging to multiple categories. For example, a data analysis task probably requires predicting formulas (Formulas), Auto-filling by dragging down (Entry and Manipulation) and plotting charts (Charts). If singular labeling is used, ChatGPT is prone to classification ambiguity and it is harder to evaluate the diversity and complexity of our dataset.\n\nThe \"practical real-world spreadsheets\" was also included in the supplementary zip file. Two brief examples: (i) SummerSales: The spreadsheet records the sales of a company. (ii) GDPBreakdown: The spreadsheet catalogs economic indicators over time.\n\n**W2: The techniques using LLMs are fairly straightforward with minor modifications.**\n\nWe believe the state machine-based planning pipeline of SheetCopilot is a novel mechanism for solving complex spreadsheet tasks, because\n\n- Plain text chatbots cannot directly execute spreadsheet actions, unlike SheetCopilot.\n- The closed-loop control architecture (Observing Stage, line 205) enables LLMs to accomplish long-horizon, multi-sheet tasks.\n- Our proposed revising stage (line 214) leverages software feedback to recover from errors, substantially boosting execution success from 67.8% to 92.8% (row 2 vs. row 4 in Table 2). The experimental result indicates that our design is original and powerful.\n\nAction dependency and the complexity of combining spreadsheet operations make it hard for LLMs to work with spreadsheet software. Lines 196-200 in Section 4.3 explains that LLMs must comprehend how sheet state evolves after each step to succeed. For example, LLMs struggle to manipulate filtered data without understanding the resulting layout. SheetCopilot's specialized design overcomes this obstacle.\n\n**W3: Further investigation into why GPT-3.5Turbo is better**\n\nThanks for the suggestion. Please refer to Q2 in the global response at the top.\n\n**W4: softer metrics would be useful in understanding where these LLMs make mistakes and where they are good.**\n\nWe appreciate your perspective on the importance of softer metrics in understanding the performance of the LLMs. \n\nOur study does employ multifaceted soft metrics. These include success rates across task categories and fine-grained failure analysis.\n\nFigure 3 decomposes Exec@1 and Pass@1 across six categories, revealing tasks for which each LLM excels. GPT-3.5-Turbo, GPT-4, and Claude achieve peak Pass@1 on Management, Manipulation, and Manipulation tasks, respectively.\n\nMoreover, Section F of the Supplementary presents an in-depth failure analysis by tallying the number of eight predefined failure types.\n\nStatistics reveal GPT-3.5-Turbo often utilizes incorrect formulas, GPT-4 frequently generates incomplete solutions due to token limits, and Claude tends to misapply formulas and ranges. A common mistake is that they often fail to apply absolute references when auto-filling, causing calculation errors.\n\nWe hope this addresses your concern.\n\n**W5: Instead of using a tag cloud for dataset evaluation something detailed metrics would have been more insightful.**\n\nThe supplementary details our dataset metrics. Figure A depicts the instruction length and atomic action distributions; Figure B illustrates category proportions and diversity within the core set; Figure C enumerates category combination frequencies. These three figures together demonstrate the richness and diversity of our dataset.\n\n**Q1: Reference to Table 1 seems off.**\n\nRegarding your comment about the reference to Table 1, we thoroughly reexamined the manuscript and table in question and believe there may be a misunderstanding.\n\nTable 1 compares LLMs and a VBA-based method. Two references exist:\n\n(i) The first (line 264) cites Table 1 to demonstrate GPT-4's strong planning capability.\n\n(ii) The second (line 306) cites Table 1 to show SheetCopilot surpassing the VBA-based method.\n\nWe hope this addresses your concern. If there are further ambiguities, we would gladly address them.\n\n**Q2: Is the dataset for creating tasks and benchmarking available to the public to compare.**\n\nYes. The dataset has been publicized on PapersWithCode anonymously and also included in the Supplementary zip file for ease of viewing.\n\n**Q3: Did you try smaller open-source LLMs**\n\nYes. Please refer to Q1 in the global response above.\n\n**Limitations: good to address discrimination or bias or fairness issues with spreadsheet data**\n\nThank you for providing the suggestion to improve the ethical integrity of our work. Please refer to Q5 in the global response for our explanation.\n\n_We thank the reviewer again for the insightful review and feedback._"}, {"review_id": "WmtUL64zJL", "reviewer": "Reviewer_HU2o", "summary": "Authors create SheetCopilot that takes input as a natural language task and then controls spreadsheet to perform the task. To provide an interface between natural language and the spreadsheet, author use a set of atomic actions that abstract spreadsheet functions. SheepCopilot uses state machine to create spreadsheet actions, modify them, and apply them to the spreadsheet.\nAuthors also present a dataset to evaluate their tool. SheetCopilot outperforms baseline approaches", "questions": "Why do certain tasks fail? Are the failures in certain categories due to the tasks, some limitations of LLMs, selection of atomic actions, spreadsheet functionality, too simple state machine? (Also see limitations comments).\n\nI have read the author’s rebuttal.\nThe rebuttal addressed my concerns about understanding of why certain tasks fail by providing failure analysis.", "limitations": "It would be good to have a more in-depth understanding why certain tasks fail and where are the boundaries of LLM spreadsheet interactions. It is good that authors classified tasks into six categories and observed different failure rates across them. However, it would be good to go further and understand the failures in more depth (also see questions).\n\nI have read the author’s rebuttal.\nThe rebuttal addressed my concerns about understanding of why certain tasks fail by providing failure analysis.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Authors propose an approach to interface LLMs with spreadsheets potentially automating a large number of repetitive tasks\n- Intermediate language of atomic actions provides a good output structure for LLM\n- State machine based processing of tasks improves the results\n- Atomic action name substitution with synonyms that are far away from the official names in an embedding space is interesting approach (but see weaknesses)\n- Test dataset is useful for evaluating SheetCopilot and similar systems\n- Ablation studies show clearly the contributions of different parts of the system\n- Positive results compared to VBA generation baseline\n- Paper is well written", "weaknesses": "- Paper contributions are moderate:\n- Intermediate language similar to atomic actions is a known approach in software engineering, similar to macro or scripting languages.\n- Atomic actions have to be implemented in spreadsheet software which requires more effort to adopt this approach compared to existing interfaces to spreadsheets. (This is not a major weakness though, especially if current interfaces to spreadsheets are not well suited or universal).\n- Atomic action name substitution with synonyms seems like a hacky approach, a bit similar to security-through-obscurity. It may have negative consequences (e.g. making it harder for LLM to connect tasks and actions) and it is not clear if this is should be used or relied upon in general.\n\nI have read the author’s rebuttal.\nThe rebuttal partially addressed my concerns about atomic actions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for a thorough review that will help us improve the work. Please see below for answers to your questions.\n\n**W1: Paper contributions are moderate.**\n\nWe believe our work contributes to the field of tool-augmented LLM, for three primary reasons:\n\n- We propose a novel framework enabling model-software interaction. The closed-loop control (Section 4.3) boosts success from 67.8% to 87.3%  (row 2 vs. row 7 in Table 2). Our framework also substantially outperforms a VBA baseline (Table 1). This facilitates applying language models to software automation.\n\n- Our dataset contains a diverse range of daily spreadsheet tasks (e.g., formulas, charts, and pivot table tasks). We believe this is a meaningful contribution because our work contributes a procedure for collecting software task datasets as well as a new testbed for comparing the planning capabilities of language models. To our knowledge, no existing dataset offers equally broad spreadsheet tasks, making ours useful for research on LLM-based agents.\n\n- We thoroughly compare leading LLMs (Table 1 in the main paper and failure analysis in the Supplementary), elucidating the strengths and weaknesses of these well-known models from an aspect of task planning.\n\nIn summary, the proposed SheetCopilot, dataset, and experiments provide novel contributions. We would appreciate any suggestions to further communicate the value of our research to the readers.\n\n**W2: Intermediate language similar to atomic actions is a known approach** \n\nWe agree that abstraction is standard in software engineering. However, to the best of our knowledge, no prior work has designed a unified abstraction for diverse spreadsheet platforms. Our atomic actions pioneer such an interface between LLMs and spreadsheets, analogous to an intermediate language abstracting assembly languages.\n\nOur atomic actions offer two key advantages:\n\n- They model spreadsheet functionality as virtual APIs (Section 4.2), enabling cross-platform control, as evidenced by Excel and Google Sheets compatibility (Fig. (b) in the response PDF). In contrast, macro languages like VBA typically target specific platforms.\n\n- They can be used by LLMs to generate human- and machine-readable solutions (Figures F and G in the supplementary).\n\nIn summary, our atomic actions constitute a novel, unified spreadsheet abstraction.\n\n**W3: Atomic actions have to be implemented in spreadsheet software**\n\nIt is right that current spreadsheet interfaces lack universality. This forces LLMs to relearn new rules when controlling new software, complicating prompting and debugging.\n\nTo address this, we propose software-agnostic atomic actions (Section 4.2) as a high-level abstraction. This enables cross-platform compatibility, as evidenced by SheetCopilot for Excel and GoogleSheets (Fig. (b)).\n\nIn summary, the platform-agnostic nature of our atomic actions is a strength rather than a weakness.\n\n**W4: Atomic action name substitution seems hacky**\n\nWe would like to clarify that atomic action name substitution is a component of our method, instead of a standalone approach.\n\nThe goal is mitigating a well-known LLM issue - hallucination - where models generate confident response that is not justified by their training data or that contradicts input prompts [1][2]. For example, we observed GPT-3.5 hallucinating a non-existing argument such as “criteria=” and using SetFormat in a way similar to SetConditonalFormat).\n\nThe cause is that GPT-3.5 confuses the atomic action knowledge with its internal Excel knowledge. GPT-3.5, like other LLMs, generates responses based on patterns it learned during training [3]. It has no access to real-time data (e.g., the usage of atomic actions) and cannot absorb new concepts after its training is complete.\n\nTherefore, if language patterns of the atomic actions are similar to those it has learned about Excel operations, GPT-3.5 will probably use the actions in a way like Excel operations. Figure I in Supplementary reveals such hallucination causes failures.\n\nTo tackle this, we substitute official names with dissimilar synonyms, enabling LLMs to strictly follow the usage of the atomic actions (Section 4.4). This is supported by Table 3 showing substitution improves both Pass@1 and efficiency.\n\nIn conclusion, atomic action name substitution is part of our method, playing a vital role in alleviating hallucination.\n\n**Q1: Why do certain tasks fail & Limitations: need understanding of why certain tasks fail**\n\nWe appreciate your suggestion to analyze task failures.\n\nTo clarify, we conducted a comprehensive failure analysis (Section F, Supplementary). We found that:\n\n- GPT-3.5 often applies incorrect formulas. For instance, it failed to use an approximate match in VLOOKUP when querying a price table.\n- GPT-4 occasionally generates incomplete solutions due to token limit. For instance, GPT-4 output correct mid-steps but the tokens had run out.\n- Claude tends to disobey requirements, like summarizing quantity rather than the requested revenue for pivot tables.\n- All three LLMs struggle with absolute references when auto-filling.\n\nFigure H compares failure patterns, showing that GPT-3.5 and GPT-4 share similar patterns while Claude exhibits a different one.\n\nFigure I delineates GPT-3.5's failure proportions over the full dataset. Failures stemming from the model itself (wrong formula/range, hallucinating, etc) occupy 73.8% in total. The failures caused by the token limit of the OpenAI API (Incomplete solutions) occupy 18.9%. The failures caused by the atomic actions (API inability) make up 2.5%. This breakdown reveals incorrect GPT-3.5 predictions as the primary failure source.\n\nWe hope the detailed analysis in the supplementary can address your concern.\n\n**Ref:**\n\n[1]Survey of hallucination in natural language generation, 2023\n\n[2]Shaking the foundations: delusions in sequence models for interaction and control, 2021\n\n[3]Training language models to follow instructions with human feedback, 2022"}]

Author rebuttal:
**Global Response**

_The authors would like to thank all reviewers for their appreciation and instructive suggestions!_

The authors are encouraged to hear that the reviews commented that

- the studied spreadsheet automation task is **valuable** (bHxV) and **challenging** (TF2z)
- the proposed SheetCopilot makes sense and is **novel** (bHxV), **inspiring** (XbAE), and **of big benefit** to spreadsheet users (EedW)
- our dataset curation process is **systematic** (TF2z), and our dataset is **a useful benchmark** for similar works (bHxV and HU2o)
- the ablation studies are **quite interesting** (XbAE) and **clearly show** our contributions (HU2o)
- our paper is **very well written** (HU2o) and the ideas are **neatly presented** (XbAE)
- no ethical issues appear in our work (Hbb1 and DGUT)

We carefully considered all concerns and comments provided by reviewers and addressed all of them appropriately. Our responses are summarized below:

**Q1 (XbAE and TF2z): Performances of smaller open-source LLMs.**

We tested smaller LLMs (e.g. llama13b, 7b) but found them insufficiently capable of generating complete solutions. Nevertheless, it is an intriguing research direction to develop small LLMs capable of solving software automation tasks. 

**Q2 (TF2z and DGUT): Why GPT-3.5 is better than GPT-4.**

We investigated why GPT-3.5 is better than GPT-4 in detail (Please see lines 265 - 268 in the main paper; Supplementary Section F). 

To clarify, GPT-4 does surpass GPT-3.5 in task success rate (Pass@1) as shown in Table 1. The lower Exec@1 of GPT-4 results from the experimental setting. To match GPT-3.5's 4096 token limit, we capped GPT-4's prompt plus response at 4096 tokens. Exceeding the token limit causes execution failure. This renders GPT-4 prone to incomplete solutions, while GPT-3.5's short yet incorrect outputs suffer less from the limit. Consequently, GPT-3.5 achieves a higher execution success rate (Exec@1) despite GPT-4's superior Pass@1.

In summary, GPT-4 outperforms GPT-3.5 in Pass@1 but the token limit renders GPT-4 less performant in Exec@1.

**Q3 (XbAE and TF2z): A more detailed discussion about the benchmark creation process and the nature of the dataset.**

Section A (Supplementary) details our benchmark creation process. Briefly, we collected ~27k SuperUser Q&A pairs, filtered irrelevant pairs by keywords, further cleaned them by prompting ChatGPT to label invalid pairs, and selected representatives through clustering, resulting in a clean Q&A pair dataset.

Figures A, B, and C (Supplementary) demonstrate our dataset's diverse complexity across six critical task categories.

We took deduplication measures to guarantee that our dataset is unseen by LLMs like GPT-3.5 (Section A.3 in the supplementary). The cosine similarity and ROUGE-L between our dataset and the SuperUser where our raw data comes from are < 0.16 and  < 0.35, respectively. This suggests that the SheetCopilot performances shown in Tables 1 and 2 are solid, without data leakage issues.

Overall, we believe our benchmark enables assessing LLM planning capabilities and will inspire further research at the intersection of language models and software agents.

**Q4 (TF2z and HU2o): About further investigation and failure analysis.**

We performed failure analysis for the tested LLMs by categorizing failure cases (Section F in the supplementary).

Key findings:

- GPT-3.5-Turbo struggles with formula prediction.
- GPT-4 often generates incomplete solutions due to token limits.
- Claude rarely repeats output but misapplies formulas/ranges.
- All models frequently neglect absolute references when auto-filling ranges, causing calculation errors.

The failure statistics also demonstrate interesting findings: GPT-3.5-Turbo and GPT-4 share similar failure patterns while Claude differs. This finding suggests that the two GPT models possess almost identical alignment in terms of spreadsheet manipulation and that a gap exists between the alignment of Claude and those of the GPT models.

We believe these results help researchers better understand the strengths and weaknesses of the leading LLMs.

**Q5 (TF2z and DGUT) While it is unlikely that one may see issues with spreadsheet data in discrimination or bias or fairness it would be good to address them.**

Thank you for bringing this to our attention. We understand the importance of fairness and unbiasedness in data. Rigorous steps were taken to uphold these principles:

Firstly, we removed any personally identifiable information from the collected spreadsheets to prevent discrimination. For example, fake names (e.g. numbers and letters) are used in the spreadsheets. Besides, the financial data in the spreadsheets are fabricated to guarantee that the sheets do not involve real individuals' or companies' financial information. These help to reduce bias, as it prevents LLMs from being influenced by personal characteristics.

Additionally, the authors thoroughly audited data, eliminating any offensive or biased content like racism or regional discrimination. For example, we removed tasks that process staff data according to country. One such task is “organizing them by name, country, and region within that country” which was labeled as “invalid” and discarded.

In summary, we implemented measures to address ethical concerns around bias and discrimination. We hope this reply addresses the reviewers’ concerns, and we are open to further suggestions to improve the ethical integrity of our work.

_Finally, we again show our greatest gratitude to all reviewers for their considerable and insightful comments._

Meta-review: The submission presents a observe/propose/revise/act framework for using LLMs to interact with an existing spreadsheet application. The LLM is embedded in a simple state machine used to control the transition between different phases. The submission also describes a new dataset of spreadsheet-related tasks that is used for evaluation.

The importance of the problem was highlighted by reviewers, though some concerns about whether the implied latency of the presented system (requiring several LLM queries) makes it impractical. The paper is executed well, with reviewers highlighting the quality of writing, the ablation studies and the usefulness of the collected dataset.

Overall, reviews were positive (with the exception of TF2z, who did not engage with the rebuttal, but mainly raised weaknesses also covered by other reviewers) but not enthusiastic due to the somewhat incremental nature of the work.
Decision: Accept (poster)