SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The goals of our experiments are threefold: (i) compare representative LLMs on the proposed dataset; (ii) demonstrate that the proposed method improves the success rate and efficiency over a simple baseline; (iii) show the flexibility and stability of our method.
5.1 Benchmark Protocol
Dataset The 221 tasks introduced in Sec. 3.2 are used to conduct the following experiments.
Metrics Exec@1 measures the proportion of solutions executed without throwing exceptions. Pass@1 is used to evaluate functional correctness [5]. A generated plan is considered correct if the final sheet state completely fulfills the task requirements. Beyond correctness, we propose A50 and A90 scores to measure solution efficiency. These divide the number of atomic actions in a generated plan by the number in the ground truth and then calculate the 50th and 90th percentiles over all tasks. Lower A50 and A90 scores indicate that the LLM tends to use fewer actions to complete a task.
Models We adopt leading large language models with public API access, including GPT-3.5-Turbo/GPT-4 from OpenAI and Claude v1 from Anthropic. Details of the models and hyper-arguments used for generation could be found in the appendix.
5.2 Comparing Task Planning Ability of Different LLMs
We compare the three LLMs on the proposed dataset with the same token limit of 4096. For less accessible LLM APIs like GPT-4 and Claude, only  $10\%$  of the dataset is used for evaluation. We have maintained the diversity of this subset to avoid data distribution shift (see the appendix for details). The results in Tab. 1 show that GPT-4 demonstrates its strong planning capability by significantly outperforming both GPT-3.5-Turbo and Claude in the Pass@1 and A50/A90. To explain why GPT-4 is inferior in Exec@1, we check the results and find that it is mainly because GPT-4 exceeds the
Figure 3: The four metrics decomposed in the six categories. The two GPT models both achieve  $100\%$  Exec@1 and Pass@1 in the Management and Entry & manipulation categories. The three models obtain their own best efficiency in different categories, suggesting that it is difficult for these models to excel in all task categories.
Table 2: Ablation studies of the observe-propose-revise-act framework proposed in Sec. 4.3. The sheet state and error feedback increase the Exec@1 and Pass@1 when individually applied (rows 3, 4 vs. 2) and bring a significant improvement when both are applied (row 7 vs. 2). Inserting the external atomic action document with usage examples boosts the Exec@1 and Pass@1 and increases efficiency (row 2 vs. 1 and row 7 vs. 5). The synergy of the four components witnesses a large increase  $(30.7\%)$  in Exec@1 over the baseline (row 7 vs. 1).
No.	State feedback	Error feedback	External doc.	Usage examples	Exec@1↑	Pass@1↑	A50↓	A90↓
1	56.6%	18.6%	1.50	4.00
2	✓	✓	67.9%	18.6%	1.00	2.50
3	✓	✓	✓	75.6%	24.4%	1.50	3.00
4	✓	✓	✓	92.8%	23.5%	1.00	2.00
5	✓	✓	68.3%	18.1%	1.50	3.78
6	✓	✓	✓	85.5%	28.1%	1.50	3.00
7	✓	✓	✓	✓	87.3%	44.3%	1.50	3.00
token limit when solving difficult tasks although it has generated correct mid-steps. In contrast, GPT-3.5-Turbo and Claude generate short but incorrect plans for most of these difficult tasks. Claude is slightly worse than GPT-3.5-Turbo in Exec@1 and Pass@1 but exhibits better A50/A90, which shows that Claude is a strong competitor of GPT-3.5-Turbo. See more detailed failure analysis in Sec. F.
To evaluate the category-specific performances, We further break down the subset into the six categories (defined in Sec. 3.1). The four metrics in each category are illustrated in Fig. 3. The radar charts demonstrate that the two GPT models both achieve  $100\%$  Exec@1 and Pass@1 in the Management and Entry & manipulation categories. Interestingly, The three models exhibit different patterns of A50/A90: GPT-3.5-Turbo, GPT-4, and Claude reach their best efficiency in the Formula, Management, and Pivot Table category, respectively. This suggests that it is difficult for these models to excel in all task categories.
5.3 Ablation Studies of State Machine
We conduct ablation studies for GPT-3.5-Turbo on the full dataset to analyze the impact of the two types of feedback and external document insertion. The results are shown in Tab. 2.
A) Closed-loop control generally boosts functional correctness Individually adding the sheet state feedback at the proposing stage increases the Exec@1 and Pass@1 (rows 3 vs. row 2) since the model no longer needs to implicitly infer the sheet state. Individually adding error feedback obtains the highest Exec@1 (row 4 vs. row 2) as a longer context window can be used to re-plan without the sheet state feedback, increasing the probability of completely finishing a plan. The combination of the two feedback types further improves Pass@1 but at the cost of slightly increasing A50 and A90 (row 7 vs. row 2), probably because the model solves more difficult tasks but with plenty of steps. It is noticeable that without the external document and usage examples, the improvement in Exec@1 becomes narrow and Pass@1 even drops slightly (row 5 vs. row 1). This is because the
Table 3: Ablation study of the atomic action names. Utilizing the synonyms far away from the official names brings an increase in Pass@1 and slightly better efficiency (lower A50).
Models	Exec@1↑	Pass@1↑	A50↓	A90↓
Official names	87.3%	44.3%	1.50	3.00
Synonyms	86.9%	45.2%	1.33	2.79
Figure 4: Stability experiments results obtained by conducting evaluation 3 times for each temperature except 0.0. The line charts show that SheetCopilot achieves stable performances even if the GPT-3.5 API temperature changes from 0.0 to 1.0.
simple action list in the initial prompt (shown in Tab. I) fails to provide sufficient information about how to determine correct action arguments even if a detailed sheet description is provided.
B) Inserting the external document improves both functional correctness and efficiency. Merely adding the external atomic action document enjoys clear improvements of  $17.2\%$  in Exec@1 and  $10.0\%$  in Pass@1 (row 6 vs. row 5). This result demonstrates that presenting this document to the model enables it to less frequently hallucinate illegal actions (which improves Exec@1) and to more accurately determine the action arguments according to the sheet state (which improves Pass@1). Further adding usage examples in the document reaches the highest performance (row 7). Additionally, without any feedback, the improvements over the baseline become narrower (row 2 vs. row 1) since it is hard to determine correct action arguments without knowing the exact properties of the spreadsheet elements even if more details of atomic actions are provided. Adding the external document and usage examples reduces A50/A90 (row 2 vs. row 1 and rows 6, 7 vs. row 5), showing that more information about atomic action usage helps to avoid redundant steps.
C) SheetCopilot surpasses the VBA-based method To further demonstrate the advantages of our method, we compare it with a method that generates and runs VBA code. Tab. 1 shows that SheetCopilot with GPT-3.5-Turbo as its backend outperforms the VBA-based method by a large margin, increasing the Exec@1 by  $9.5\%$  and Pass@1 by  $28.0\%$ . This result shows that prompting powerful LLMs with our method to control spreadsheets is a better alternative to directly translating natural language requests to VBA code.
5.4 Ablation Study of Atomic Action Names
To inspect the potential confusion problem stated in Sec. 4.4, we conduct another ablation study for GPT-3.5-Turbo on the full dataset by comparing the impact of adopting the official names and the synonyms. The results in Tab. 3 surprisingly show that using the synonyms increases the Pass@1 and obtains lower A50/A90, which means that the model learns to use the synonyms to generate more correct solutions with fewer steps. This result demonstrates the flexibility of our method: it is possible for users to define their own atomic actions and prompt LLMs to use them.
Table 4: Ablation study of atomic action granularity on the chart-related and format-related tasks. The results show that using an integrated CreateChart action achieves slightly lower A50 and A90 but encounters significantly inferior Exec@1 and Pass@1. Additionally, splitting the SetFormat into finer-grained format-setting actions leads to higher Exec@1 and Pass@1.
Experiment	Method	Exec@1↑	Pass@1↑	A50↓	A90↓
Integrating CreateChart	Ours full	91.7%	43.3%	1.25	1.67
Ours + Integrated CreateChart	79.1%	37.2%	1.00	1.50
Splitting SetFormat	Ours full	70.7%	9.8%	2.75	6.65
Ours + Split SetFormat	80.5%	12.2%	2.00	5.60
5.5 The Influence of LLM Temperature on Task Plan Generation
We evaluate the stability of our method by running the full method three times with temperatures from 0.0 to 1.0. The results in Fig. 4 show that the metrics are stable with slight deviations from the values for temperature=0.0.
5.6 Atomic Action at Different Granularity
A natural question is how to determine the granularity of atomic actions, i.e. the number of workbook elements an action manipulates.
To investigate this question, two experiments are conducted: 1) The actions that set chart properties are incorporated into the CreateChart action, and the original separate actions are removed, with the expectation that SheetCopilot will set chart properties when creating charts. 2) In another experiment, SetFormat, which is originally used to set multiple format properties, is split into finer-grained format-setting actions. Please refer to Sec. D.2 for details.
We conduct these two experiments with GPT-3.5-Turbo backend on the chart and format-related tasks. The results in Tab. 4 show that using an integrated CreateChart action to handle chart creation and property setting simultaneously obtains slightly higher efficiency (lower A50 and A90). However, this variant encounters significantly inferior Exec@1 and Pass@1. In contrast, splitting the original SetFormat action witnesses considerable gains in Exec@1 and Pass@1.
After analyzing the results, we found that an integrated CreateChart encounters lower functional correctness as its complex document makes it difficult for SheetCopilot to understand the action usage, thus being less able to correctly determine action arguments. In addition, the lengthy documentation of this integrated action frequently exceeds GPT-3.5's token limit. In contrast, we observed that after splitting SetFormat, the LLM can easily understand the simple finer-grained actions, thereby encountering fewer hallucination cases. See Sec. D.2 for detailed analysis.
These results suggest that it is more desirable to use finer-grained atomic actions instead of integrated high-level actions in terms of functional correctness.
We propose SheetCopilot, a spreadsheet agent based on the observe-propose-revise-act framework that decomposes a high-level task into step-by-step solutions for software control. To evaluate our agent, we curate a realistic and diverse dataset representing the typical demand of non-expert spreadsheet users. The experimental results show that SheetCopilot can perform spreadsheet manipulation tasks with a high pass rate and acceptable efficiency, outperforming VBA-based methods. The ablation studies show that the closed-loop control and external document insertion used by SheetCopilot bring clear improvements over the baseline and that adopting a set of atomic action names dissimilar to the official names achieves a surprising performance gain. We also find that utilizing finer-grained atomic actions instead of integrated high-level actions can notably improve functional correctness. We hope our work provides a useful roadmap for researchers interested in the field of LLM-based autonomous agents and sheds light on future research.
This work was supported in part by the National Key R&D Program of China (NO. 2022ZD0160102), the National Natural Science Foundation of China (No. 61836014, No. U21B2042, No. 62072457, No. 62006231), Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences, and the InnoHK Fund.
A Details of Dataset Collection
A.1 Details of Seed Task Collection
The details of the keyword-based and LLM-based filters described in Section 3.1 are shown below: **Keyword-based filtering:** Questions containing keywords about irrelevant spreadsheet control tasks (e.g., Visual Basic for Application (VBA), loading and saving files, keyboard and mouse shortcuts, using images and shapes) are removed. LLM-based filtering: To easily identify irrelevant question and answer (Q&A) pairs, we introduce the seventh task category, i.e. Invalid, apart from the six categories. With these categories, a set of Q&A pairs are labeled as exemplars to prompt ChatGPT to assign at least one category label to each pair (See Tab. B for the used prompt). After classification, we remove 2432 pairs labeled as Invalid, thereby obtaining a clean dataset containing 13574 pairs annotated with task categories.
We notice that multiple Q&A pairs are likely to refer to similar tasks with different expressions. To identify representative questions, clustering is performed within each combination of task categories (note that a pair may be labeled with multiple task categories). Specifically, the title and text of each question are concatenated as a single paragraph which is then embedded into a 768d vector². For speeding up computation, all embeddings within a category combination are further projected to 64d vectors using Principle Component Analysis. Finally, these embeddings are grouped into 10 clusters using K-means++, which means that 10 representative questions are selected for each category combination. Clustering all category combinations, 220 representatives are selected from the 13574 pairs.
As our evaluation environment does not support excessively complicated tasks (e.g., consolidating data, setting page layout, connecting pivot tables), out of these representatives, we further select 67 that involve operations supported by the evaluation environment. The tasks described in these questions are regarded as the seed tasks that capture the most important patterns of the whole clean dataset.
A.2 Details of Workbench Collection
To lessen the difficulties of collecting a large-scale task dataset, the evaluation workbooks in the workbench strictly follow three rules: (1) Data in every sheet starts from cell A1; (2) Blank columns within tables are not allowed; (3) Each column is provided with a header on the first row, which means that the records of each column are listed from the second row. To better utilize the internal knowledge of LLMs, we also provide a short paragraph describing the context of each evaluation workbook. This context briefly introduces the usage and domain of the workbook and includes useful formulas that assist in manipulating the workbook. See Tab. A for the contexts of all evaluation workbooks.
A.3 Details of Core Set Collection
Adaptation. During adaptation, GPT-4 is prompted to change the manipulated sheet elements (e.g. ranges, charts, data types) mentioned in the seed tasks. To enable GPT-4 to generate clear task instructions relevant to the evaluation sheets, three types of descriptions are included in the prompt, i.e., a brief sheet summary, column headers, and row numbers. In addition, adaptation exemplars written by the authors are included in the input to teach GPT-4 in a few-shot manner. In the end, a batch of 10 seed tasks to be adapted is listed so that the token limit is fully utilized, which speeds up the adaptation process and saves the cost. See the used prompt in Tab. C and adaptation examples in Tab. D.
Verification. Between adaptation and simplification, an extra verification step is required since a number of the adapted tasks are unrealistic and irrelevant to the target sheet. To retain valid proportions, we prompt GPT-3.5 (GPT-4 is not used as it is overly expensive) to classify each task according to four criteria, i.e., realism, clarity, relevance, and completeness. A task is valid only if it simultaneously fulfills the four criteria. GPT-3.5 is asked to describe how each task fulfills the criteria before determining its validity as we empirically find that this method more accurately
2The model used for embedding is OpenAI text-embedding-ada-002.
identifies invalid tasks. Finally, 1515 valid tasks  $(90.7\%)$  remain. See Tab. F for the used prompt and an example of using GPT-3.5 to verify the tasks.
Simplification. As the adapted task instructions are likely to refer to specific functions and operations built in Excel, these instructions need to be simplified to read like the tone and fashion of a non-expert user. We establish the following rules for prompting GPT-3.5 to simplify the tasks: 1) Convert specific mentions to natural language descriptions and remove redundant words while retaining the original intention and order of actions. 2) Avoid referring to existing columns by the column indices since it is more natural to refer to a column by the column header. 3) Mention the insertion place and add a column header when inserting a new column to avoid ambiguity. 4) Finally, use domain-specific knowledge to diversify the expression of the generated instructions. See Tab. G for the prompt and an example.
After these steps, a raw dataset containing diverse, natural, and realistic tasks is produced. To extract a core set from the simplified tasks, six random tasks are selected for each category combination, resulting in 402 selected tasks. The authors polish these selected tasks by further revising them and discarding a fraction not supported by the simulator, resulting in 184 remaining tasks. To enrich the core set, the authors act as non-expert users to compose 37 more tasks. Lastly, 221 tasks exist in the core set. See Fig. A for the instruction length and atomic action distributions, Fig. B for the proportions of the six task categories and the task diversity of the core set, and Fig. C for the numbers of each category combination.
The final step is to prepare reference solutions for these tasks. To objectively judge the performance of LLMs, we use GPT-3.5 to generate multiple solutions for each task and then retain the successful ones after verifying the final s by hand. As a number of solutions are likely to exist for a task, multiple reference solutions are collected as the ground truth for one task.
A.4 Selecting the  $10\%$  Dataset Used for Comparing the LLMs
20 tasks are selected from the 221-task core set to approximately maintain the percentages of the six categories and the distribution of the numbers of atomic actions. This collection basically represents the pattern of the core set, avoiding a data distribution shift to a large extent. See Fig. D for the statistics of this subset.
Figure A: The distributions of the instruction lengths and the numbers of atomic actions involved in each instruction. The two histograms demonstrate the diverse task complexity of the core set.
B Atomic Action Names
B.1 Collecting Atomic Actions
Selecting a set of indivisible and easy-to-understand atomic actions is necessary since LLMs need to generate an interpretable solution to the user-specified task. As no existing atomic actions used in spreadsheet software have been curated and publicized, we obtain the names of these actions during classifying the SuperUser Q&A pairs (Described in Sec. A.1). Specifically, ChatGPT is prompted to determine which atomic actions each Q&A pair requires to complete the task, which resembles a text summarization problem. As multiple similar names are likely to be generated for an atomic action, we
Table B: An example of using the classification prompt to predict multiple category labels and extract the involved atomic actions for the Q&A pairs scraped from SuperUser. For clarity, the prompt components and an example ChatGPT response are marked with brief descriptions in blue.
(Requirements and category definitions for classification)
I have a batch of Excel questions for you to clean. I need you to determine which categories each question belongs to according to the question title, and its top answer. A question may belong to multiple categories if it involves the atomic actions of these categories.
The six categories are defined as below:
A: Entry and Manipulation. Tasks that enter and manipulate cell contents such as editing cell values, splitting texts, inserting and deleting rows/columns.
B: Management. Tasks that organize data such as sorting, filtering, using slicers to filter data, and moving rows/columns.
C: Formatting. Tasks that modify sheet or cell formats such as changing text font, adding drop-down lists, and merging cells.
D: Formula. Tasks related to using formulas such as performing calculations and statistical analysis.
E: Charts. Tasks related to presenting data visually such as creating charts, customizing chart elements, and adding a trend line.
F: Pivot Table. Tasks related to using Pivot Table such as creating and formatting Pivot Tables.
If the question does not belong to any of the six categories above or it is too vague, it is regarded as "Invalid".
I also need you to determine which atomic actions each valid question requires to complete the task.
(Classification exemplars collected by the authors.)
Below are example questions in the form of a Python dictionary:
Q1: {"question title": "How to filter a cell which containing data by other list", "top answer": "You have to select the Column Headers and click on "Data", "Filter" and "automatic filter" for XL2003
"Data", "Filter" for XL2007 and probably 2010."}
Q2: {"question title":"Group rows by column then sort by another column", "top answer":"I found the solution myself. First sort by the ID then I added a new column with the following formula:=INDEX($B$1:$B$278,MIN(IF($I$2:$I$278)=I4,ROW($I$2:$I$278))))}. Replace the bounds as necessary. I then sort by this new column and then the ID column and the data is in the order I wanted."
Q3: {"question title": "How to delete the infinite blank rows?", "top answer":"Go to Home tab  $\rightarrow$  Find and Select  $\rightarrow$  Go to Special Select Blanks. Click OK. The blank cells will be highlighted...}"
Q4: {"title title": "How to cut certain rows and insert before a certain row in excel", "top answer": "Highlight the row by clicking of the row number then press Ctrl+X then click on the row number below the row were you intend to insert the cut rows then press Ctrl+V."
Q5: {"question title":"How do I add conditional formatting to cells containing #N/A in Excel?", "top answer":"Select the first cell you want to highlight. (B6).
Click Home  $\rightarrow$  Conditional Formatting  $\rightarrow$  Manage Rules  $\rightarrow$  New Rule. Select Use a formula to determine which cells to format. In the field Format values where this formula is true, enter =ISNA(SB6). Click Format to set the cell formatting, then select OK.
Click OK again to create the formatting rule. In the Conditional Formatting Rules Manager, edit the range under Applies to (ex: $B6:$B8). Select OK to apply the rule, which will match to true and thus apply the formatting you want."
Q6: {"question title": "How do you auto resize cells in Excel?", "top answer": "I have solved it like this: 1. Select the entire spreadsheet. 2. Double click the line in the margin in between the two rows. This auto sizes every row not just the one row that I clicked on."
Q7: {"title title": "Sum contents in column from starting cell on down without setting an explicit last cell index", "top answer": "For Excel 2003 or before: =SUM(B5:INDEX(B5:B65536, MATCH(TRUE, INDEX(ISBLANK(B5:B65536),0,0),0)-1,0)) For Excel 2007 or after: =SUM(B5:INDEX(B5:B1048576, MATCH(TRUE, INDEX(ISBLANK(B5:B1048576),0,0),0)-1,0))"}
Q8: {"question title": "How to get the dates of the current monday, wednesday and friday of the current week in excel?", "top answer": "The following will display the dates for Monday, Wednesday and Friday of the current week:
=NOW() - WEEKDAY(NOW(),3), = NOW() - WEEKDAY(NOW(),3) + 2, = NOW() - WEEKDAY(NOW(),3) + 4
Basically this is taking the time now, and subtracting the current weekday (which gives you Monday), then adds 2 days or 4 days to get Wednesday and Friday."