SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1. Paraphrase each given instruction so that it is more like what a non-expert Excel user would say while retaining the original intention and order of actions in the instruction.
2. Do not mention any Excel built-in functions in the parentheses. If the original instruction mentions the names of specific Excel built-in functions, you should replace these functions with spoken language. For example, "Use CONCATENATE to combine cells" mentions "CONCATENATE" and should be rephrased as "concatenate cells". "Use conditional formatting to change the background color of cells" mentions "conditional formatting" and should be rephrased as "Set cell background color if ...". "Write a formula to copy data" should be rephrased as "Copy data".
3. Do not refer to existing columns by their indices (e.g., column A is not desired); instead, mention columns by their headers. For instance, 'In column A' should be rephrased as 'In the Year column'.
4. Do not add column references in brackets. For instance, "subtracting Total Expenses (Column B) from Revenue (Column A)" should be rephrased as "Subtract Total Expenses from Revenue".
5. When inserting new columns, the column indices must be given to avoid ambiguity. Besides, each new column must be given a header, so you need to keep the column headers mentioned in the original instructions.
6. Use domain-specific knowledge to diversify the expression of the generated instructions and do not use the same expression repeatedly.
(Exemplars written by the authors for in-context learning)
I will give you some examples first:
Original instructions:
1. Convert the "Year" column values (Column A) from the format of yyyy.00 to yyyy.
2. In a new column with the header "Combined data", use the CONCATENATE function to combine the data from columns A through Z in each row, including the headers, and then autofill the formula to the last row of data.
3. Use advanced filters in "Sheet1" to display rows with sales data for a specific month (assuming weeks starting from the 1st of each month). For example, filter to show rows with weeks "Week 1" to "Week 4" for the first month.
4. Create a line chart on a new sheet named "Sheet2" with weeks (Column A) on the x-axis and sales (Column B) as the y-axis.
5. Apply conditional formatting to Column H (Net Profit) based on Columns F (Operating Profit) and G (Tax Expense). If the cell value in Column G is not 0 and Column F is greater than 0, display the value in Column H in red.
6. In a new sheet named "Sheet3", use the VLOOKUP function to match each row in "Year" (Column A) in "Sheet1" with the corresponding value in Column B ("Net Sales") from "Sheet1".
7. In a new column (Column G) with the header "Tax Calculation", use a formula to calculate the tax by multiplying the corresponding "Subtotal" (Column D) by 0.1. Think about the flaws in the original instructions before paraphrasing:
Think: The original instruction refers to columns by indices in brackets, which is undesired.
Paraphrase: Convert the "Year" column format from yyyy.00 to yyyy.
Think: The original instruction mentions Excel built-in functions CONCATENATE, which is undesired.
Paraphrase: Concatenate the data from columns A through Z for all rows and write the results in a new column named "Combined data".
Think: The original instruction mentions an Excel built-in operation (i.e., advanced filters), which is undesired.
Paraphrase: Display only the rows with weeks from Week 1 to Week 4.
Paraphrase: Create a line chart on a new sheet with the Week column on the x-axis and the Sales column as the y-axis.
Think: The original instruction mentions Excel built-in functions (i.e., conditional formatting) and refers to columns by indices, which is undesired.
Paraphrase: If the cell value in Tax Expense Column is not 0 and that in Operating Profit Column  $>0$  display the cell text in the Net Profit column in red.
Think: The original instruction mentions Excel built-in functions (i.e., VLOOKUP) and refers to columns by indices in brackets, which is undesired.
Paraphase: Match cells in the Year column and return the corresponding values in the Net Sales Column. Put the results in a new sheet.
Paraphrase: Calculate the tax by multiplying the "Subtotal" column by 0.1 in column G named "Tax Calculation".
(Task instructions to be simplified. The sheet description is provided to maintain the relevance)
Now it's your turn. Please follow the requirements to paraphrase the original instructions according to the given workbook descriptions.
My workbook records all aspects of expenses but has not yet been completed. The necessary formulas are as follows:  $\mathrm{Tax} =$  Subtotal \* Tax rate; Total  $=$  Subtotal  $^+$  Tax. Sheet "Sheet1" has 6 columns (Headers are A: "Date", B: "Vendor/Client", C: "Expense Account", D: "Subtotal", E: "Tax", F: "Total") and 25 rows (including the header row). The cells in the "Date" column can be "2020-01-08 00:00:00", "2020-01-02 00:00:00", "2019-12-31 00:00:00", ... The cells in the "Vendor/Client" column can be "Client A", "Client B", "Company A", "Company B". The cells in the "Expense Account" column can be "Reception", "Car Repairs", "Accommodation", "Material Purchase", "Gas", "Meals", "Flight tickets", "Hiring". The cells in the "Subtotal" column range from 127.00 to 1954.00.
1. Calculate the sum of the "Subtotal" column (Column D) ONLY if the "Vendor/Client" (Column B) equals "Company A".
2. In a new sheet named "Sheet2", write a formula in column B to match the "Vendor/Client" in Column A of "Sheet2" against the "Vendor/Client" in Column B of "Sheet1". If the "Expense Account" (Column C) in "Sheet1" is "Accommodation", copy the "Subtotal" (Column D) from "Sheet1" to Column B of "Sheet2".
3. In a new column (Column G) with the header "Tax Calculation", calculate the tax by multiplying the corresponding "Subtotal" (Column D) by 0.1.
Think about the flaws in the original instructions before paraphrasing:
(GPT3.5 response)
Think: The original instruction refers to columns by indices in brackets, which is undesired. Flaw: The original instruction mentions the specific value "Company A" for the "Vendor/Client" column, which is undesired.
Paraphrase: Calculate the sum of the "Subtotal" column if the "Vendor/Client" is for "Company A".
Think: The original instruction mentions Excel built-in functions (i.e., MATCH) and refers to columns by indices in brackets, which is undesired.
Paraphrase: In a new sheet named "Sheet2", find the corresponding "Vendor/Client" in "Sheet1" for each row and copy the "Subtotal" from "Sheet1" to a new column in "Sheet2" if the "Expense Account" is "Accommodation".
Paraphrase: Calculate the tax by multiplying the "Subtotal" column by 0.1 in a new column named "Tax Calculation".
Table H: The official names and synonyms of the atomic actions used in our method. The Formula category contains the types of formulas used for calculation in the core set tasks and does not represent specific operations.
Category	Official Name	Synonym
Entry and manipulation	Write	RangeInputValue
Delete	DiscardRange
InsertRow	NewRowAtIndex
InsertColumn	ColumnCreation
AutoFill	RangeValueTransfer
CopyPaste	ReplicateRange
FindReplace	AlterRange
SetHyperlink	LinkRangeAssociator
RemoveDuplicate	DistinctData
CreateSheet	WorksheetCreation
Clear	EraseRangeContents
Management	Sort	AdvancedRangeSort
Filter	SmartRangeSelector
CreateNamedRange	SetRangeName
FreezePanes	LockRowsColumns
Formatting	SetFormat	CustomizeFont
SetDataType	RangeTypeFormatter
Merge	ConcatenateCells
ResizeRowColumn	RangeDimensionAdjuster
SetConditionalFormat	FormatWithRules
SetCellLock	ProtectActiveSheet
Charts	CreateChart	Graph Constructor
CreateChartFromPivotTable	CreatePivotGraph
SetChartTitle	ChartTitleSettings
SetChartAxis	CustomizeAxisAttributes
SetChartHasAxis	AxisDisplayManager
SetChartLegend	LegendConfiguration
SetChartType	ChartTypeSwitch
SetChartMarker	DefineSeriesMarker
SetChartTrendline	TrendlineAdjustments
AddDataLabels	DisplayChartDataLabels
AddErrorBars	ErrorBarsIntegration
Pivot Table	CreatePivotTable	PivotTableConstructor
SetPivotTableSummaryFunction	PivotFunctionChange
Formula	Date and time functions
Logical functions
Lookup and reference functions
Math functions	N/A
Statistical functions
Text functions
Financial functions
project all generated action names into 768d embeddings using the OpenAI embedding model $^3$  and then reduce these embeddings to 64d vectors before performing Kmeans++ clustering. This process produces 80 clustering centers each of which represents a group of semantically similar action names. After checking the names corresponding to these centers by hand, we find that these names are the official names used for the built-in features of Microsoft Excel. According to the potential use of atomic actions in the core set tasks and the implementation restrictions of the evaluation environment, a proportion of these action names (44 out of 80) are used by our SheetCopilot (Listed in the initial prompt and used in the external document). These 44 atomic action names are used for conducting the comparison experiment (Sec. 5.2), state machine ablation study (Sec. 5.3), and the stability test experiment (Sec. 5.5). Please see Tab. H for these atomic action names.
B.2 Collecting Synonyms for the Official Atomic Action Names
To investigate the confusion issue mentioned in Sec. 4.4, we attempt to adopt a different set of atomic action names. Specifically, we use GPT-4 to generate 10 candidate synonyms for each atomic action name according to the low-level implementation of the action. Then, we adopt as the new name the candidate farthest from the official name in terms of the cosine similarity in the embedding space<sup>4</sup>. These synonyms (shown in Tab. H) are used in the ablation study in Sec. 5.4 to investigate the impact of adopting different sets of names.
C Details of SheetCopilot Implementation
We introduce the implementation of the state machine our SheetCopilot is based on and the prompt used to query LLMs to generate task solutions.
C.1 State Machine Implementation
The Observe-Propose-Revise-Act framework is implemented using a state machine shown in Fig. E. At the beginning of each step, the Observing stage enables the LLM to receive the sheet state before planning an action in the Proposing stage. Since the LLM is likely to output an incorrect action, an external document of the initially planned action is inserted into the query to prompt the LLM to revise the action. If the initially planned action is invalid, a validation error will occur and then the state machine will return to the Proposing stage; If the revised action is invalid, a validation error will also occur but the state machine will return to the Revising stage. If the revised action has been validated, the action will be submitted to the Acting stage for execution. The revised action is still probably erroneous, causing run-time errors in the software. In this case, the state machine will return to the Revising stage to prompt the LLM to re-plan according to the error information until the execution is finished. The entire process is repeated until the LLM outputs "Done!".
C.2 Prompting LLMs to Generate Task Solutions
The initial prompt used to query the LLM to generate step-by-step solutions is shown in Tab. I. The prompt is formatted as a multi-turn dialog according to the usage of OpenAI ChatGPT API, with each turn comprising a content field and a role field. The prompt consists of an overall task goal, a list of atomic actions, requirements, an exemplar, and finally a task to be solved.
D.1 Qualitative Experiment of Missing Atomic Actions
We deliberately remove several atomic actions to test the robustness of our method. Specifically, we remove SetConditionalFormat and test SheetCopilot on a task involving conditional formatting. We also remove another important atomic action - CreatePivotTable - to see whether SheetCopilot is able to adopt other feasible solutions to analyze data. We use GPT-3.5-Turbo as the backend of our SheetCopilot. Fig. F present surprising results: Example 1 shows that SheetCopilot uses a filter to retain the rows fulfilling the task requirement, set the formats of these rows, and finally cancel the
3The model used for embedding is OpenAI text-embedding-ada-002.
4The model used for embedding is the same as above.
Instruction: Can you help me set the fill color of the Invoice No. column cells whose corresponding values in the Sales column range between 200 and 500 as green?
Solution of full atomic actions:
1. SetConditionalFormat source="Sheet!!A2:A19", formula=\\"=AND(\$G2>=200,\$G2=500)\", fillColor="green")
Final sheet state
Solution when SetConditionalFormat is missing:
1. Filter.source  $=$  "Sheet!A2:G19", fieldIndex  $= 7$  ，criteria  $= ^{n} = 200$ $<  500^{n}$    
2. SetFormat.source  $=$  "Sheet!A2:A19", fillColor  $=$  "green")
3. DeleteFilter()
Instruction: Sort the data by the Date column in an ascending order. Then summarize the total roll numbers for each date in a new sheet. Then create a line chart in the new sheet that displays the trend over time.
1. Sort.source="Sheet1!A2:D26", key1="Sheet1!A2", order="asc", orientation="column")
2. CreateSheetsheetName  $=$  "Sheet2")
3. Create PivotTablesource="Sheet1!\ $A\$1:\$ D\$26",destSheet  $=$  "Sheet2", name  $=$  "PivotTable",RowField  $=$  ["A"],ColumnField  $=$  [],PageField  $=$  [],DataField  $=$  ["B"]
4. CreateChartFromPivotTable(pivotTableName="PivotTable", destSheet="Sheet2", chartName="Roll Trend", chartType="Line")
Solution when CreatePivotTable is missing:
1.Sort.source="Sheet1!A2:D26",key1  $=$  "Sheet1!A2",order  $=$  "asc",orientation  $=$  "column")
2. CreateSheetsheetName="Roll Summary")
3. Write(range="Roll Summary!A1", value="Date")
4. Write(range="Roll Summary!B1", value="Total Rolls")
5. Write(range="Roll Summary!A2", value="Sheet1!A2")
6. Write(range="Roll Summary!B2", value="=SUMIFS(Sheet1!B:B,Sheet1!A:A,A2)")
7. AutoFill.source="Roll Summary!A2:B2", destination="Roll Summary!A2:B26")
8. CreateChart.source="Roll Summary!A1:B26", destSheet="Roll Summary", chartType="Line", chartName="Roll Trend")
9. SetChartTitle chartName="Roll Trend", title="Roll Trend Over Time")
Figure F: Two example solutions generated with missing atomic actions. For each example, the instruction (top), solutions generated with full/missing atomic actions (left), and the final sheet states after implementing the two solutions (right) are displayed. The two examples show that SheetCopilot demonstrates strong robustness by using a combination of other atomic actions to attain the same goal which can be more efficiently attained using the missing atomic action.
filter to restore the data. Example 2 shows that SheetCopilot cleverly utilizes the SUMIF function to substitute the pivot table. Despite redundant calculation, it obtains functionally correct results.
D.2 Detailed Analysis of Atomic Action at Different Granularity
We provide further details about the ablation studies on the action granularity (Sec. 5.6 in the main paper). In our design, the low-level implementations of the atomic action used by SheetCopilot are consistent with the atomic actions used by Excel in terms of behavior, which means that our atomic actions possess a normal granularity and are familiar to average users. However, one may argue that a high-level atomic action that combines several actions used by our SheetCopilot may bring better performances as using high-level actions possibly improves efficiency.
To investigate this interesting problem, we combine several chart-related atomic actions into one high-level chart manipulation action and conduct an ablation study on the tasks involving charts. Specifically, we incorporate the actions that set chart properties into the CreateChart action and remove the original separate actions, expecting the SheetCopilot to set all required properties on the creation of the chart. Additionally, the usages of these actions are merged and a new example of showing how to use this integrated CreateChart action is also added (See the documents of the separate actions and integrated action in Tab. J). Additionally, in another experiment, we split SetFormat which originally contains multiple arguments related to setting various format properties to see the impact of using finer-grained format-setting actions (See the documents of the SetFormat action and finer-grained format-setting actions in Tab. K). The SetConditionalFormat action possesses a function overlapped with that of SetFormat, thus interfering with the experiment. Therefore, we remove the SetConditionalFormat action when conducting this experiment.
We conduct this ablation study with GPT-3.5-Turbo as the SheetCopilot backend on the chart-related tasks (43 out of 221) and format-related tasks (41 out of 221). The results in Tab. 4 show that using a high-level CreateChart action to handle chart creation and chart property setting simultaneously obtains slightly higher efficiency (lower A50 and A90). However, it is surprising that this method encounters significantly inferior functional correctness, with Exec@1 and Pass@1 decreasing by  $12.6\%$  and  $6.1\%$ , respectively. Likewise, splitting the original SetFormat action witnesses considerable performance gains in Exec@1 and Pass@1.
After analyzing the chart-related results, we found that using an integrated CreateChart encounters lower functional correctness since its document is so complex that SheetCopilot struggles to understand the action usage, thus being less able to correctly determine all action arguments as required. In addition, the document of this integrated CreateChart action is extremely lengthy, causing the query to frequently exceed the GPT-3.5-Turbo token limit. These results suggest that it is more desirable to use finer-grained atomic actions instead of integrated high-level actions in terms of functional correctness. After analyzing the format-related results, we observed that using the original SetFormat action tends to result in argument hallucination. For example, the LLM frequently adds a non-existing argument such as "criteria=..." trying to use SetFormat in a way similar to the removed SetConditionalFormat. The LLM outputs such erroneous action repeatedly even if the error feedback is provided, finally exceeding the token limit and causing an execution error. In contrast, after splitting the SetFormat action, the LLM is able to easily understand the simple finer-grained actions, thereby encountering fewer hallucination cases.
E SheetCopilot Manipulation Example
To better illustrate the strengths of our SheetCopilot, an example of using SheetCopilot (GPT-3.5) to produce solutions to users' requests is shown in Fig. G. SheetCopilot is asked to complete the sales data, analyze the results, and visualize the trend. SheetCopilot accurately understands the intention of the user's instruction and correctly revises its solution utilizing the external document of the chosen atomic action and the error feedback provided by the evaluation environment.
F Analysis of failure cases
To obtain a clearer view of the distinction between the LLMs compared in Sec. 5.2, we perform an in-depth analysis of the failure cases for each LLM. The predefined failure types are as follows:
Table K: The documents of the finer-grained format-setting actions (left column) and the original SetFormat action (right column). The argument and usage of the original SetFormat are both divided as the arguments and usages of the finer-grained actions. For clarity, the redundant contents are substituted with ellipsis.
Figure G: SheetCopilot example: Handling sales data. The left column shows that SheetCopilot generates a step-by-step solution according to the sheet state feedback and correctly revises its mistakes using the external atomic action document as well as the error feedback. The incorrect arguments are marked with red rectangles. The right column demonstrates the state changes of the evaluation sheet corresponding to each step on the left. For illustration clarity, only brief documents are displayed.
Figure H: Comparing the proportions of different failure cases of the three LLMs. The evaluation dataset is the  $10\%$  subset. The charts show that GPT-3.5-Turbo and GPT-4 share similar failure patterns while Claude exhibit a different one.
Figure I: The proportions of different failure cases when evaluating GPT-3.5-Turbo on the full coreset.
- Wrong action Use wrong atomic actions to attain a specific goal. For instance, hide columns using a filter (a filter can only be used to hide rows) or perform meaningless actions to destroy the data.
- Wrong range Select the wrong ranges to manipulate. For instance, fail to use absolute references when required or select incomplete ranges as chart data sources.
- Certain arguments unset Perform the atomic actions required by the tasks but fail to set all required arguments. For instance, set the cell background color but forget to set the cell font color.
- Repeated output Output the same atomic action repeatedly until exceeding the token limit.
- Hallucinate data Hallucinate data to manipulate while ignoring the real data.
- Incomplete solution Terminate a solution without fulfilling all task requirements.
- Disobey requirements Perform an atomic action in a way that deviates from the task requirements. For example, insert a hyperlink different from the one specified in the task instruction.
- API inability Failure caused by the limitations of the low-level implementations of the atomic actions. For instance, the low-level implementation of conditional formatting does not support all types of formulas (e.g. array formulas or formulas available only in a specific software version).
The categorized failure cases of the three LLMs are shown in Fig. H. From an overall viewpoint, we can obtain two interesting findings: 1) GPT-3.5-Turbo and GPT-4 share similar patterns of the failure case proportions while Claude demonstrates a largely different failure pattern. This finding suggests that the two GPT models possess almost identical alignment in terms of spreadsheet manipulation and that a gap exists between the alignment of Claude and those of the GPT models. 2) Jointly analyzing the failure cases and the performances in Tab. 1, we can see that although the prompt used by SheetCopilot (See the prompt in Tab. I) is tuned for GPT-3.5-Turbo, Claude exhibits competitive performances compared with GPT-3.5-Turbo.
Inspecting individual cases, we found that GPT-3.5-Turbo tends to use wrong formulas, showing that its ability to predict formulas is weak. GPT-4 is prone to incomplete solutions due to the limited context window. In contrast, Claude encounters no incomplete solutions and less frequent repeated output but tends to use wrong formulas and wrong ranges. One common failure case among the three LLMs is that they often use a combination of UNIQUE and SUMIF/AVERAGEIF/COUNTIF to replace pivot tables. This combination is technically correct but these three models forget to apply absolute reference when selecting manipulated ranges used by auto-filling, leading to incorrect calculation results.
We also check the failure cases made by GPT-3.5-Turbo on the full core set (See the experimental results in Tab. 1) and illustrate the percentages of these cases in Fig. I. The pie chart shows that GPT-3.5-Turbo performs poorly at predicting formulas and determining appropriately selected ranges. These two types of failure occupy over  $50\%$  of all cases. Inspecting the wrong formula cases, we