InstructExcel: A Benchmark for Natural Language Instruction in Excel
Authors: Justin Payan
Swaroop Mishra
Mukul Singh
Carina Suzana Negreanu
Christian Poelitz
Chitta Baral
Subhro Roy
Rasika Chakravarthy
Benjamin Van Durme
Elnaz Nouri
Venue: EMNLP 2023 Findings (2023)
Keywords: Code generation
program synthesis
benchmark
large language models
TL;DR: We introduce a new benchmark for code generation from natural language instructions in Excel.

Abstract: With the evolution of Large Language Models (LLMs) we can solve increasingly more complex NLP tasks across various domains, including spreadsheets. This work investigates whether LLMs can generate code (Excel OfficeScripts, a TypeScript API for executing many tasks in Excel) that solves Excel specific tasks provided via natural language user instructions. To do so we introduce a new large-scale benchmark, InstructExcel, created by leveraging the ‘Automate’ feature in Excel to automatically generate OfficeScripts from users’ actions. Our benchmark includes over 10k samples covering 170+ Excel operations across 2,000 publicly available Excel spreadsheets. Experiments across various zero-shot and few-shot settings show that InstructExcel is a hard benchmark for state of the art models like GPT-4. We observe that (1) using GPT-4 over GPT-3.5, (2) providing more in-context examples, and (3) dynamic prompting can help improve performance on this benchmark.

InstructExcel: A Benchmark for Natural Language Instruction in Excel
Justin 2 2*
Swaroop Mishra 3*1 Mukul Singh'
Carina Negreanu'
Christian Poelitz*
Chitta Baral Subhro Roy-
Rasika Chakravarthy'
Benjamin Van Durme-
Elnaz Nouri-
Microsoff, UMass Amherst, 3 AArizona State University
Abstract
2016) it is important to further diversify the range of tasks Al assistants can support.
With the evolution of Large Language Mod- els (LLMs) we can solve increasingly more complex NLP tasks across various domains, in- cluding spreadsheets. This work investigates whether I.LMs can generate code (Excel Office- Scripts, a TTppeScrppt API for executing many tasks in Excel) that solves Excel specific tasks provided via natural language user instructions. To do so we introduce a new large-scale bench- mark, INSTRUCTEXCEL," created by leverag- ing the 'Automate' feature in Excel automati- cally generate OfficeScripts from users' actions. Our benchmark includes over 10k samples cov- ering 170+ Excel operations across 2,000 pub- licly available Excel spreadshects. Experiments across various zero-shot and few-shot settings show that INSTRUCTEXCEL
Meanwhile, the advent of large language mod- cls, together with the instruction learning paradigm (Mishra al., 2022b; Wei al., 2022; Ouyang et al., 2022: Sanh et al., 2022), has paved the way for non-experts accomplish many tasks just by providing instructions in natural language. Can we leverage this novel paradigm to let Excel users automatically execute complex tasks from NL in- structions?
Excel OfficeSeripts provide a versatile Type- Script API for most standard functions in Excel, giving us a simple intermediate representation for executing NL instructions in Excel. Our task is to convert a user's description of simple action or set of actions in Excel (such as editing formatting) into executable OfficeScript accomplishing the action(s) the given spreadsheet. Figure 1 shows an example input and output, with additional exam- ples in Table 5 of Appendix A.
hard bench- mark for state of the art models like GPT-4. We observe that (1) using GPT-4 over GPT--3.5, (2) providing in-context examples, and (3) dynamic prompting can help improve perfor- mance on this benchmark.
We introduce INSTRUCTEXCEL, benchmark to investigate the instruction paradigm for generat- ing OfficeScripts from NL instructions. We extract publicly available Excel sheets and ask crowdwork- ers to think of an action they want perform on the sheet and write a summary it in NL. Sub- sequently, we ask crowdworkers to execute that action in the sheet and use the Automate feature to record the action and generate the underlying code. Automatic code generation improves output quality and eliminates potential bugs prevalent in human- written code. INSTRUCTEXCEL contains over 10k samples covering 170++ OfficeSeripts operations across 2,000 publicly available Excel sheets.
1 Introduction
Spreadshect software, especially Microsoft Excel, is used every day by diverse set of users (from complete novices to highly sophisticated power users) to solve wide range of important tasks. To help users accomplish their tasks, Excel has introduced an array of features, including Analyze Data² where users can specify their task in natural language (NI.) for a limited scope of tasks. As even seemingly simple manipulations can often require knowledge of complex concepts like loops, conditionals, and regular expressions (Desai et al.,
We evaluate OpenAl's GPT-3.5 Turbo and GPT- 4 models (OpenAI, 2023) as well as T5 model
* Work done during internship at Microsoft Currently Google DeepMind 'INstRUCTExCEL. along with the code used our experiments, available at sttps://githtse com/ microsoft/Instrrce Exce1. 'happs//supportt.micee www.tttot.oooooooooooooooom www.////wwnsoooffcccooeeiiiaaaaa with-mmalyze-datt-al100 105 149- Lol.--44fdd-adddA577227777700
documentation: https: //1 lleern.iicrooof wwtps/sfffcccc//eooeeeeeeee api /ert/ccesssse ptsp atts/xxcelsst scr scripttiiieii ?viev= of fice scripsss
OfficeSeripts API

---

(Raffel et al., 2020), on INSTRUCTEXCEL and find it to be challenging benchmark even using state of the art tools, indicating significant room for future work on this task. We also find that GPT-4 im- proves over GPT-3.5, and dynamic prompting and more in-context examples improve performance, but additional in-context instructions do not help.
et al., 2022c), and Promptsource (Bach et al., 2022) are some popular instruction learning benchmarks, however they are focused on general NLP. In con- trast to prior works, we focus on the application of the instruction paradigm in the Excel domain.
In addition to evaluating capabilities of state- of-the-art models on the full benchmark, we also illustrate the broader utility of INSTRUCTEXCEL through a case study on conditional formatting rules. INSTRUCTEXCEL contains 660 examples that represent conditional formatting tasks, which are studied in isolation (Singh et al., 2022). Viewed through the lens of this particular task, INSTRUC- TEXCEL provides data source that can be fur- ther processed enable experiments with task- specific ground truth. Many other sub-tasks can be extracted from INSTRUCTEXCEL, in the same manner, such as charting/plotting (Lee et al., 2021) or formula creation (Chen et al., 2021b).
2.2 Program Synthesis
The recent success of large language models (LLMs) like Codex (Chen et al., 2021), GPT4 (OpenAI, 2023), PaLM (Chowdhery al., 2022), PaLM2 (Anil al., 2023), StarCoder (Li et al., 2023) and WizardCoder (Luo et al., 2023) has led to significant progress in program symthesis. LLMs have been found to generalize to many different code generation tasks. The models vary in size and training schemes and their success rate in per- forming code generation tasks (Xu et al., 2022). Several works leverage pre-trained models to map NL queries sequences of API elements or other intermediate representations, which can be trans- lated into code, though this may not be necessary with models that have been pre-trained on code (Hadi et al., 2022: Shin and Van Durme, 2022). They have also been used for in in-place data transfor- mations (Narayan et al., 2022). Pre-trained models have proven to be especially strong when combined with clever post-processing steps constrained de- coding, for example in Jigsaw (Jain et al., 2022), Synchromesh (Poesia et al., 2022), and PICARD (Scholak ct al., 2021). Chan et al. (2022) investi- gates the impact of training on a large number of different tasks. Other important aspects studied in- clude length generalization (Anil et al., 2022), com- positional generalization (Shi et al., 2022), reverse engineering (Pearce et al., 2022), and generating development tools (Bareiß al., 2022). The task of NI. to Code is broadly of interest to the semantic parsing literature (Kamath and Das, 2018; Zhong et al., 2022; Zhao et al., 2022).
We believe ISTRUCTEXCELL will encourage development of user-friendly methods to automate various Excel tasks and help non-expert Excel users in their daily workflow.
2 Related Work
2.1 Instruction Learning Paradigm
The instruction paradigm (Mishra et al., 2022b; Wei al., 2022; Sanh et al., 2022: Ouyang al., 2022) provides a user-friendly option to leverage MI. models just by providing instructions with- out requiring underlying technical knowledge. In- structions describe tasks in NL (Efrat and Levy, 2020; Weller et al., 2020), and are shown to help models generalize to unseen tasks (Mishra et al., 2022b; Wei et al., 2022; Ouyang et al., 2022; Wang et al., 2022b; Xu et al., 2023; Zhong al., 2021: Gupta et al.. 2023; Patel al., 2022; Puri et al., 2023; Mishra and Nouri, 2022; Chung et al., 2022). Recent developments
large language models (Brown et al., 2020; Chowdhery et al., 2022) have led to the successful use of instruction learning methods in various applications, such as dialog (Gupta et al., 2022), tabular question answer- ing (Luo et al., 2022), relation extraction (Chen al., 2021a), biomedical applications (Parmar al., 2022), NER (Wang et al., 2022a), program synthesis (Kuznia et al., 2022), and style trans- fer (Reif et al., 2022). Natural Instructions (Mishra et al., 2022b), Supernatural Instructions (Wang
Existing benchmarks for automated code gener- ation include CoNaLa (Yin et al., 2018), DIANGO (Oda et al., 2015), HumanEval (Chen et al., 2021), MBPP and MathQA-Python (Austin et al., 2021), APPS (Hendrycks tests (Li et al., 2022). In contrast to these bench- marks that target general-purposs programming languages, our work INSTRUCTEXCEL instead fo- cuses on a specialized Excel API, which LLMs are less likely to have encountered during pre-training
al., 2021), and CodeCon-

---

Metadata
and other Excel-specific details associated with the sheet. The input is typically a single line of text provided by the crowdworker.
Public URL
https://office. lasakovi. com/ com/excel/ee lexcel//rr
aty/netradicni dicni-graty---mnnn excelu/histogram excel.lisx
The output field contains the OfficeSeript code generated when the user records the actions taken to accomplish their task. OfficeSeripts are written in TypeScript, a superset of JavaScript. Each script must contain a main function with the ExcelScript. Workbook type as its first parame- ter. When the function runs, the Excel application invokes the main function by providing the work- book its first parameter.
Output
File
This docurnent is about Histogram of Procento obyvatel
Description
Vék
Zerr
Muži
NL Description
20
-23%
12%
30-<44
-32%
20%
Data
40
-18%
25%
50 < 60
-14%
25%
// Insert chart on sheet selectedSheet
Excel File Name Each entry includes link pointing to the Excel file. Although we do not re- lease the spreadsheets directly as part of the bench- mark, each spreadsheet can be downloaded from its associated URL.
officeScript Code
let charL_1 aadeeed ExcelSerpt Chart type barClustered, selectedShest.getiane Rettangge 922::031
Figure I: Schema of INSSTRUCTEECCL A sample input- output pair in INSTRUCTEXCEL. natural language description and the linearized spreadsheet data are the inputs and the Excel OfficeScript code is the desired output. We also have two additional pieces of metadata, the URL for the Excel file and description of the file's contents.
Excel File Description Since some Excel files very large and may not fit within the limited context size of smaller language models, we have another field where the crowdworker describes the contents of the Excel file in natural language. This is usually one or two lines of text.
3 INSTRUCTEXCEL
3.3 Constructing INSTRUCTEXCEL
this section, we describe the INSTRUCTEXCEL. benchmark. We start with the schema used to repre- sent the task, followed by our dataset construction procedure, and an overview of the dataset statistics,
Selecting Excel Files Starting from a dataset of 5,000 Excel documents from publicly available links on the web, we removed any files that are in languages other than English or that were password- protected, corrupted, or otherwise inaccessible. We also applied a size criteria (20 KB < filesize 30 KB) to avoid Excel files that were too small to have meaningful data or too large for human anno- tators to understand and study in the provided time. We eventually selected final set of 2,000 Excel documents to be further annotated.
3.1 Task
The user inputs actions such as clicking buttons, writing formulas, and inserting columns. These actions correspond to OfficeScript code which pro- duces equivalent effect on the Excel spreadsheet. In addition, the user deseribes the intended result of their actions using natural language description. Our task is to map the natural language description input, along with the contents of the spreadsheet, to the OfficeScript code output.
Data Creation Process We used crowdsourcing to record actions over the set Excel documents. We used crowdsoureing platform called the Uni- versal Human Relevance System. 4 We recruited English speaking annotators from India. We used an intermediate third party vendor which assures quality of the tasks through management of com- munication and training of the annotators. We quested our vendor to arrange pool of annotators who have basic familiarity with Excel applications. We also provided the vendor a list of popular Excel
3.2 Schema
Figure shows the schema of INSTRUCTÉXCEL, along with an example. has 4 elements: input, output, Excel file URL and Excel file description (see Table 5 in Appendix A for more examples).
Input Each input contains a natural language in- struction that specifies the desired manipulation of the Excel spreadsheet, c.g. "highlight the 1st row sheet 1." This can also contain references to cells
*UHRS located at: https: /tt:://prrr //ppdd..duu/ooooo uurs. plannn.. com/ COm/UERS/ /URRS/

---

operations which they must use. We ensured that our third party vendor had access the Automate feature in Excel Web version and that they could access the Excel files of our dataset through the Excel Web version.
Category
# of instances
Samples
10347
Excel files
2000
Operations
177
Avg. Ops./Sample 8.9
For each document the user was instructed to in- put natural language description of the file. They were then asked to type natural language instruc- tion, record themselves performing the relevant action for the instruction, and copy-paste the re- sulting code generated by the Automate feature in Excel Web, They repeated that process times per spreadshect. The full data collection process is il- lustrated in Figure 2. A screenshot of the interface for the human intelligence task (HIT) is provided in Appendix B.
Table 1: Statistics of INSTRUCTÉXCEL. Operations are distinct API methods that represent various Excel tasks.
EM ROUGE
BLEU
21.30
67.64
65.23
60.45
Table 2: Metrics measured between pairs of code out- puts belonging the same natural language queries. The relatively low exact match but relatively high ROUGE, FI, and BLEU scores reflect the fact that some queries have multiple solutions that vary only slightly.
Qualification, Crowdworker Compensation and Data Quality Check We provided rounds of pilot HITs to the vendor's annotators, reviewed their responses, and made clarifications to the in- structions based on the feedback. Specifically, we instructed annotators not to overuse or underuse certain actions after observing the action distribu- tion in pilot rounds. We provided various examples of queries which could be toggled on and off within the interface. We assessed the qualification of the crowdworkers based on the responses to the pilot HITs submitted to us. In addition to the hourly rate of 12 USD for the annotation work, we also covered the vendor's management and training fees to assure response quality.
ample, use of the word 'formula' corresponds with the method call 'setFormulalocaa! Many of the top methods require only one parameter. However, some of the methods require fairly complex pa- rameters. The 'setFormulaLocal method requires formulas input as strings, and 'setRule' requires complicated conditional format rule objects that must be constructed before being passed to the method call. Some common actions performed by users require sequences of multiple method calls; for instance, inserting shape requires specifying the location, type, and color of the shape.
We found 76 queries that were repeated at least once in the benchmark. Most of these queries general queries such "freeze umn We compute the exact match, ROUGE, FI, and SacreBLEU scores between all pairs of code blocks belonging to the same natural language user queries, displayed in Table 2. Although the re- peated queries often have only one correct code solution (for example, "freeze row 1" has only one solution), many queries can be satisfied with multi- ple code solutions (i.e., "Create chart on worksheet Sheet1" can be satisfied with any chart type).
Consent and Privacy Control Annotators had the choice of performing the requested tasks only after signing consent form. In addition, pro- vided explicit instructions in the HITs that disal- lowed sharing of any personal and identifiable in- formation when providing responses.
1" or "sort col-
3.4 Statistics
Table shows key statistics of INSTRUCTEXCEL. The most common words used the Excel sheet descriptions are 'year', 'name', 'form', 'calendar', 'table', and "content." These words represent com- mon Excel workflows. 'Formula', 'background', 'color', 'conditional', and 'formatting' are the most common words in the NL instructions.
4 Experiment
Figure shows the most common methods used in the recorded OfficeScripts. We see fairly broad distribution over important Excel functionalities. The top methods correspond with the most com- mon words used in the users' instructions for ex-
We conduct experiments with supervised, zero-shot and few-shot learning to assess the performance of popular I.LMs on INSTRUCTEXCEL. We use GPT- 3.5 Turbo and GPT-4 in our experiments (Ope- nAI, 2023), specifically the "gpt-3.5-ture 16k"

---

Ropested sprcacchezes
sprtedshect
Uterstates
fuaction nadilee sioee
this is data simse of Ta of HEDERAL IRVEIT ADMINIETRATION
Inseet callumn and applied Toxt to columin coll A38 on Table 5 shoee
"edectedtaad sertettthhttt nos ecirting
Inoe Coll mine
Accendins.
1/ text callers de rage anther sedlctsihhett (437))
Insce Gounee
"s) 10..ermaaaneeisaaaeeellle '1')
5 actior code pais collected per spreatheett
Figure 2: Steps in data collection.
3500
3000
2500
COUNT
2000
1500
1030
500
selFormuaa..om parFormal
earColor
gelFont
addConditionall Formar
getFill
serStyie
maiRangeSordes
SeiRude
self wwttariioonallliinnnnn
sefWeight
selNumberF Formatlocal
Gen gerExtendedRRe Ratenee
addChart
marga
setindenti.cm evvl
apply
gotTable
setBlld
Insert
Figure 3: Plot showing the distribution of the top 20 most-common Excel operations in our dataset. The X axis shows the unique operations, and the axis shows the frequency of the operations in INSTRUCTEXCEL.
and "gpt-4-32k" models." We also experiment with finetuning the T5 Large LM Adapt model (Raffel et al., 2020).
respectively. Considering the computational cost and rate limit associated with GPT usage, we also form test-set subset of 200 samples which we use in our experiments.
In all experiments, single example consists of a triplet with 1) natural language query, 2) the data from the spreadsheet converted into a string using a Pandas ExcelFile object, and 3) the ground truth or generated OfficeScripts code. We elaborate below on our full experiment setup.
4.2 Setup
4.2.1 Zero-shot
In this setting we do not provide any examples to the model. Instead, we provide the prompt *Gen- erate function with excel script to execute the action given below in NL.' In order to make the task easier, we also add the common boilerplate code which starts every OfficeScript the prompt as exemplified Figure The output is limited to 256 tokens, and we impose this limit in all other settings as well.
4.1
Data Splits
We exclude examples from the benchmark that have broken URLs for the purposes of our exper- iments, as some of the URLs have expired since data collection and annotation occurred." 7 We di- vide the remaining examples into train, dev and test splits containing 4033, 1000 and 1000 instances
4.2.2 Few-shot and Max-shot
'nttpe://piatforr opw.ai....mp........mmmoom...... openai com/ docs/ models
We use the in-context learning setup (Brown al., 2020), showing few examples before prompting the model respond to the input query. We experi- ment with two different sho- learning setups. First, we evaluate models 3-shot learning (3 ex-
"https://huggiinggacc co/
httpg//gfaacccoooogggle..... google/
t5-large-1ar adapt.
"We intend to continue to collect and provide additional data in our public repository, some natural URL expiry will not impact the utility of the dataset.

---

in the prompt. Both settings always include the in- tended 10 in-context examples in the prompt, and the data included the examples retain an aver- age of 19.36 lines (for GPT-3.5) and 15.36 lines (for GPT4+API). This typically guarantees that the headers and some data are included for each of the in-context examples and the query. We hypothesize that increasing context window size will result in performance improvements.
Excel Script Function:
function main (warkbook: Excllsrrpp:Workboo)
let selectedSheet wwrkbbokk.ebbcciieeooessseeee //
Figure 4: Standard code template included at the end of each prompt.
amples are presented), where in-context examples are included after the instructions but before the query. Second, we evaluate models in the Max- shot setup, in which we add 10 examples. Unless otherwise specified, the examples used prompt the model are randomly chosen and put in random order ahead of time, and we use the same examples for all test instances.
4,4 API in Context
To inform GPT of the API, we experiment with ap- pending the API to every prompt. The API consists of an alphabetically sorted list of all classes, enums, methods, and properties in the OfficeScripts API, along with the accepted parameters for methods and the fields of all enums. We append the API immediately under the instruction in the prompt, before the in-context examples and the query.
4.2.3 Max-shot=Instrucee
Motivated by the success of detailed instructions in improving model performance (Mishra et al., 2022b; Wei et al., 2022; Ouyang et al., 2022; Mishra et al., 2022a), we prompt with a longer NL instruction along with 10 examples. The longer instruction is listed in Appendix C.
4.5 Dynamic Prompting
One other method which can inform GPT of the relevant API elements for given input is dynamic prompting (Liu et al., 2022). For dynamic prompt- ing, given natural language input, we search for the most similar natural language inputs in the train- ing set, measured according to Fl score of the nor- malized text. We then append the top 3 or 10 exam- ples from the training set, including the linearized data from each example. This prompting strategy not only provides in-context examples that are mantically similar to the query, but also increases the likelihood that the correct elements of the API are directly exposed in the context.
4.2.4 Finetuning
We finetune only the T5 model, finetuning GPT- 3.5 Turbo and GPT-4 is costly. Details of finetuning and the parameters for T5 inference are listed in Appendix D.
4.3 Data Inclusion
Although we try to include as much data as pos- sible in the prompts, the spreadsheets can often be too large to fit all of them in the prompt. This problem is especially pronounced in the Max-shot setting. We first calculate the number of tokens in the prompt when using all available data for each in-context example and the query. If the number of tokens exceeds the context length for the model, we remove the second half of each data string. We repeat this process until the prompt is small enough to fit in context, or all data has been deleted. If all data has been deleted and the prompt is still too large, we revert to the full data for each ex- ample and simply truncate the string of in-context examples to make the prompt fit in context.
4.6 Evaluation
We rely on textual similarity- evaluation met- rics that can casily be reused: Exact Match (EM) comparing the normalized code prediction with the gold standard, F1, ROUGE (Lin, 2004) and SacreBLEU (Post, 2018) to estimate the textual similarity between predicted code and the gold stan- dard. We use the implementations of ROUGE and SacreBLEU from HuggingFacc Datasets," and we use the built-in stemmer and ROUGE-L metric for ROUGE. We remove the boilerplate code (shown in Figure 4) before evaluation.
To understand the impact of data truncation, we analyze the two most space-restricted settings in our experiments, the Max-shot+Instruecee settings with both the GPT3.5 model with 16k context size, and the GPT4 model with 32k context size + API
We also require a metric that measures semantic equivalence of code outputs without penalizing ar- bitrary choices like variable names or choices for
"pttps:/ i aace co/ docs/ www.d/dattaa.o datasetss index.

---

arbitrary parameter values. Some parameters and numbers cannot be directly inferred from the natu- ral language query or the spreadsheet data, and are sometimes (but not always) arbitrary choices that do not impact the correctness of the code. Queries such as "insert shape in sheet December 2021" or "Change the legends position on sheet data in chart" can be correctly satisfied using many different pa- rameter values for the shape type and legend posi- tion, while queries such as "Apply group to A7:A11 on sheetl" require using the specific parameter val- ues A7 Distinguishing these cases is a non- trivial task, and some use-cases are more parameter- sensitive than others. In Section 6, we analyze the subset of INSTRUCTEXCEL. corresponding to con- ditional formatting operators, which admit evalua- tion by execution equivalence. However, evaluating relevance to general NL queries automatically is far more challenging.
automated metries. However, GPT-4+DP outper- forms the finetuned T5 model when comparing manual annotation score. The relative performance gain of finetuning over in-context learning is there- fore unclear, though both appear effective.
GPT-4 outperforms GPT-3.55 Turbo For the Few-shot (3), Max-shot and Max-shot+Instruction settings, we observe that the performance of GPT- 4 is higher than GPT-3.5 Turbo (details in Ap- pendix E).
More In-context Examples Help Few-Shot Capa- bilities Across both GPT models, we observe that the Max-shot model performance is much higher than the Few-shot and Zero-shot performance (see Appendix E for details). This potentially indicates that addition of in-context examples can greatly help LLMs generate OfficeScript code.
To circumvent the general question semantic equivalence, in addition to computing metrics on the original predicted and gold standard code, we also report the value for each metric when ranges, numbers, and parameters are stripped from the ex- amples and replaced with placeholder values. We call this funetion-based evaluation, since this ap- proach enables us to study functional equivalence of the output programs while ignoring some details.
Detailed Instructions Do Not Help The Max- Inotuusstn baseline is not remarkably dif- ferent from the Max-shot baseline. Considering the sensitivity of model performance to instruction framing (Mishra et al., 2022a; Zhao et al., 2021), it would be beneficial for future work to experiment with other instruction variants.
API in Context Including the API in context im- proves performance in the Zero-shot case, but not in the Few-shot case. In the Max-shot (shown in Table 3) and Max-shot+ including the API in context harms overall per- formance. This is likely because the API has large number of tokens. Much less data can be included in the prompt with this method, and some in-context examples may need to be removed from the prompt entirely.
We also annotate all predicted outputs from the three models with the highest performance on auto- mated metrics. For each predicted code output, we give binary judgment on whether the predicted code output satisfies the request given by the user. To aid in annotation, we perform our annotations while looking the gold standard outputs, and we consult the spreadsheets when data from the spreadsheet necessary to judge correctness.
experiments,
Dynamic prompting is in- credibly effective, causing at least 40% rela- tive improvement in all 4 metrics. Despite this performance improvement, the model still often makes mistakes where API elements are halluci- nated. Since prompting with the full API directly seems to harm performance, solving the hallucina- tion of API elements under in-context learning is an important area for further study.
Dynamic Prompting
5 Results
Table 3 shows all metrics across various model- ing setups, but only for Max-shot prompting. We find that Max-shot outperforms Zero-shot, Few- shot, and Max-shot+Instruction (see full results in Appendix E) across all models, so we only report Max-shot in-context learning and finetuning results in this section. We summarize our main findings below.
5.1 Manual Analysis
Finetuning Improves over In-context Learning on Automated Metrics Model performance af- ter finetuning is notably higher than model perfor- mance in all other settings, when comparing on
Our manual annotation indicates that although the finetuned T5 model outperforms GPT4+DP on the automated metrics, in-context learning with GPT- 4 slightly outperforms the finetuned T5 model

---

Standard Eval.
Function-basee based Eval.
Manual
Eval.
Model
EM
ROUGE
F1
BLEU
EM
ROUGE
F1
BLEU
GPT3.5 Turbo
0.0
34.47
5.09
29.12
0.5
42.69
10.68
33.86
GPT4
3.0
52.70
42.62
40.08
15.5
63.25
52.73
52.15
GPT4+API
1.50
43.18
40.96
31.24
12.00
54.33
48.09
41.95
GPT4+DP 15.00
69.59
61.83
62.60
41.50
80.80
73.56
74.91
56.91
GPT4+API+DP
15.00
63.99
57.83
57.47
36.00
74.78
67.94
68.71
50.79
T3 (Finetuned) 17.00
76.73
72.06
68.81
45.50
87.58
83.46
81.70
52.38
Table 3: Model performance in different evaluation settings. All models are evaluated under Max-shot in-context learning, except the finetuned T5 model. The full results Zero-shot, Few-shot, and Sexx-shot+Instuuctin can be found in Appendix E. EM is normalized exact match. Settings with +API indicate where we append the description of the full OfficeScripts API the prompts, and settings with +DP indicate where dynamic prompting is applied. In the standard evaluation setting report metries for the full prediction against the original gold standard output, while in the function-basee evaluation setting we report metrics on normalized code predictions and outputs. Normalization replaces parameters, ranges, and numbers with placeholder values.
manual correctness judgments. Thus, although the automated metrics can help with rough compar- isons, they do not give complete picture of model performance.
range are also the most prevalent. Examples of errors made by GPT4+DP are included in Table 8 of Appendix F. In contrast to the T5 model, the GPT4+DP model has more errors where com- pletely different task is solved than what is queried. This is a potential artifact of in-context learning that should be addressed. Both models suffer from overwriting important data, as shown the final example of Table
We also perform a 2-sample, one-sided t-test to compare the value of each automated met- ric for manually-judged incorrect predictions vs. manually-judged correct predictions. This test has the alternative hypothesis that the metric has lower average when the example manually deter- mined to incorrect. We perform these tests for the top-performing models and all metrics. We re- ject the null hypothesis with a p-value of 0.01 in all cases. These tests indicate a statistically significant difference between the values of automated metrics for correct vs. incorrect predictions, suggesting convenient
Case Study: Formatting Rules in Excel
The INSTRUCTEXCEL benchmark contains a wide range of Excel tasks, including charting/plotting (Lee al., 2021), formatting (Singh al., 2022), and formulas (Chen et al., 2021 We can de- rive specific benchmarks for these individual down- stream tasks from INSTRUCTEXCE) In this case study, we present an analysis of conditional format- ting rules as a downstream task.
that automated metrics may (though incomplete) replacement for metrics based on execution equivalence or manual judgment.
We also perform a qualitatitive exploration of remaining error types. About 20% of the annotated examples simply do not compile for the top three scoring models, often because they hallucinate API elements or formulas. Among the examples that compile correctly, we identify three major remain- ing error types for the finetuned T5 model: mis- understanding the user's intention, targeting the wrong object or incorrect cell range, and accom- plishing the correct task in different way than intended. Full examples of these failure modes are shown in Figure 5. We also identify major error types for the GPT4+DP model, finding that misunderstanding the users' intentions/solving the wrong task and targeting the wrong object or cell
To focus on the task of learning conditional for- matting rules, extract all examples from the INSTRUCTEXCEL dataset that involve any format- ting operator. These extracted samples are then manually validated to ensure their relevance to for- matting tasks. We obtain a total of 660 valid CF tasks after filtering.
To facilitate benchmarking, we transform the ex- tracted conditional formatting tasks into an Interme- diate Language (IL) representation specifically de- signed for formatting operators (Singh al., 2022). Figure 6 showcases an example of this IL.
To evaluate the performance of various ap- proaches on this task, we employed the original natural language (NL) descriptions associated with

---

Output
Uver intention mikuddestood
Change Charl till sheat data
charl gel tini).nnVVillllettrunn
che Ligiiiiiiii seife
Titlel"):
Incorrect target for data operation
gricliiee on sheet1
selectedSS peeectedireet.eefaaa neefaangnn" aeetfoorrtt)..eeriiieee attruel:
Two of accomplishing same task
Merge and center now on sheet1
selectedd)) out, geiRange("11 (:1"":1""""))""""""""""""""""""" j.merge(talie))
selected5t gerRanget ngo( getfoormmtt.sstttrrooonnaa
selectedShecct.ggtaanggge Aa attoa//Ai11a
ixxceent pt torzootta Connzonnal/lignneenn wwwggnme:tneoneeeree
Dlggmment cinoooti wtw.ottallagnnnnnnnnnaa comeent.centrr,
Figure 5; Examples of errors made by the finetuned T5 model test examples. Each line shows example of a common type of error.
Model
Setup
EM
SM
ExM
"iightight
Colane, corditiaral formatiing
Excel Scrdpt:
Zero-shot
2.3
5.6
12.1
funclae wistworooo:: bxcuberapt Nandool)
condirlanatioo witt tng: Lace Sorint cooel tieatiaaan:
G.T
Few-shot (3)
42.3
42.2
64.3
Max-shot
50.8
59.8
67.2
y/ chenan call value free cange Eirius on ontrcressoeve
gethonge"""
concitiznnal ao hommatiing peccalieueeet ae gethormac) gotfontt t sccclled" aaaeaag
Zero-shot
3.7
6,8
14.2
GPT 4
Few-shot (3)
45.6
60.5
70.2
corept coreptaenal fomwatiing gestuioeuest) sethutes)
Comaaaui
Max-shot
53.5
63.4
73.3
Sperator: Bww.strr................. www.ppp.....eee www.ooowoo..o.oooooooooooooo....
Table 4: Baseline results the CF task extracted from INSTRUCTEXCEL. EM, SM and ExM indicate Exact Match, Sketch Match and Execution Match respectively,
Fermetting
treekerthe Dhe maae, 139))
Figure 6: Example of a CF task from INSTRUCTEXCEL and the associated Intermediate Language (IL) rule.
licly available Excel spreadshects. Our experi- mental results show that (1) GPT-4 (2) more in- context examples, and (3) dynamic prompting help improve performance on this benchmark, while more-detailed in-context instructions and includ- ing the API description do not help. Finetuning aids performance on automated metrics, though our manual annotation shows that the difference between finetuning and in-context learning not conclusive. INSTRUCTEXCEL is still a challenging benchmark for state of the art models like GPT- 4 and T5. We also demonstrate, through a case study on conditional formatting, that INSTRUC- TEXCEL can be used as source data for more spe- cific study of important sub-tasks in Excel. We hope that improvements INSTRUCTEXCEL will enable novice users to perform complicated actions in Excel quickly, boosting productivity and teach- ing important Excel functionality in the process.
the tasks generate simplified formatting rules. As these rules were already in a parsed format, we report the following evaluation metrics: (1) Ex- act Match, (2) Sketch Match over the AST of the parsed rules, and (3) Execution Match - Verifying the equivalence of the resulting formatting by exe- cuting the generated rules on data and comparing the outcomes (Singh et al., 2022, 2023).
Table 4 illustrates the performance of various basclines on the task of learning conditional for- matting rules. GPT-4 with Max-shot in in-context learning achieves the highest performance among the evaluated approaches. Our analysis showcases the feasibility of extracting relevant data, trans- forming it into intermediate format, and bench- marking various approaches for the specific task of conditional formatting. These findings serve compelling motivation for the broader usage of INStruCTExcEl dataset in advancing research and development within the Excel domain.
8 Limitations
Our focus in INSTRUCTEXCEL, is on instructions written in English. A typical non-expert user work- flow often involves languages other than English. The limitation not include local language in IN- STRUCTEXCEL. will addressed future work.
7 Conclusion
We introduce INSTRUCTÉXCEL, a benchmark to investigate the capability of models in generating Excel OfficeScript code from NL instructions. IN-
STRUCTEXCEL consists of over 10,000 samples covering 170+ Excel operations across 2,000 pub-
The programming language under study, Excel Of- ficeScripts, is specific to Microsoft Excel, so our

---

results may not generalize to other spreadsheet soft- ware, Our focus in this work is to release a targeted benchmark which, if performance is improved, will enable a core set of NL to code functionality in Mi- crosoft Excel. The potential risks from this work are minimal. Excel OfficeSeripts is language de- signed to be secure, targeted only at manipulating data inside Excel spreadsheets. Therefore, there is little risk that a model will produce overtly harmful code in this domain.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sas- try, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter. Philippe Tillet, Felipe Petroski Such, Dave Cum- mings, Matthias Plappert, Fotios Chantzis, Eliza- beth Barnes, Ariel Herbert- William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang. Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders. Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. 2021. Evaluating Large Language Models Trained on Code. arXiv e-prints, page mrXiv:2107.033
References
Cem Anil, Yuhuai Wu, Anders Johan Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Venkatesh Ra- masesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. 2022. Exploring length gen- eralization in large language models. Proceedings of the 36th Annual Conference on Neural Information Processing Systems (NeurIPS).
Xiang Chen, Xin Xie, Ningyu Zhang, Jiahuan Yan. Shumin Deng. Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2021a. Adaprompt: Adaptive prompt- based finetuning for relation extraction. arXiv e- prints, pages arXiv-2104.
Rohan Anil, Andrew M Dai. Orhan Firat, Melvin John- son, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa. Paige Bailey, Zhifeng Chen, et al. 2023. Palm 2 technical report. arXiv preprint 2rXiv:2305. 10403.
Xinyun Chen, Petros Maniatis, Rishabh Singh, Charles Sutton, Hanjun Dai. Max Lin, and Denny Zhou. 2021b. Spreadsheetcoder: Formula prediction from semi-structured context. In Proceedings of the 38th International Conference Machine Learning (ICML).
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoe Le, et al. 2021. Program synthesis with large language models. arXiv preprint arxi::2108.00 170.0772..
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, Parker Schub, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodku- mar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya. Sanjay Ghemawat, Sunipa Dev. Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai. Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz. Erica Oliveira Moreira, Rewon Child, Oleksandr Polozov, Kather- ine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Ja- son Wei, Kathleen S. Meier-Heilsterm, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language modeling with pathways. ArXiv. abs/2204.0234)
Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari. Thibault Févry, et al. 2022. Promptsource: An integrated development environment and repository for natural language prompts. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 93-104.
Patrick Bareiß, Beatriz Souza, Marcelo d' Amorim, and Michael Pradel. 2022. Code Generation Tools (Al- most) for Free? A Study of Few-Shot, Pre-Trained Language Models on Code. arXiv e-prints, page arXiv:2206.01335
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan. Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few- shot learners. Proceedings of the 34th Annual Con- ference on Neural Information Processing Systems (NeurIPS).
Hyung Won Chung. Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Fric Li, Xuezhi Wang, Mostafa Dehghami, Siddhartha Brahma, et al. 2022. Scaling instruction-fin finetuned language models. arXiv preprint arXiv:2210. 1416.
Jun Shern Chan, Michael Martin Pieler, Jonathan Jao, J'er'emy Scheurer, and Ethan Perez. 2022. Few-shot adaptation works with unpredictable data. ArXiv, 0bs/2208.01009

---

Aditya Desai, Sumit Gulwani. Vineet Hingorami. Nidhi Jain, Amey Karkare, Mark Marron, and Subhajit Roy. 2016. Program synthesis using natural language. In Proceedings of the 38th International Conference on Software Engineering (ICSE).
2023. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161.
Yujia Li, David Choi, Junyoung Chung. Nate Kush- man, Julian Schrittwieser, Rémi Leblond, Tom Be- cles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy. Cyprien de Mas- son d' Autume, Igor Babuschkin, Xinyun Chen, Po- Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. 2022. Competition-level code generation with alpha- code. Science, 376(6)24): 1092-10999991999
Avia Efrat and Omer Levy. 2020. The turking test: Can language models understand instructions? arXiv preprint arXiv:2010.1 1982.
Himanshu Gupta, Saurabh Arjun Sawant, Swaroop Mishra, Mutsumi Nakamura, Arindam Mitra, San- tosh Mashetty, and Chitta Baral. 2023, Instruction tuned models are quick learners. arXiv preprint arXiv:2306.05539.
Chin- Yow Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out.
Prakhar Gupta, Cathy Jiao, Yi-Ting Yeh, Shikib Mehri, Maxine Eskenazi, and Jeffrey P Bigham. 2022. Im- proving zero and few-shot generalization in dia- logue through instruction tuning. arXiv preprint arXiv:2205. 12673.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, William B Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for gpt-3? In Proceedings of Deep Learning Inside Out (Dee- LIO 2022): The 3rd Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures. pages 100-114.
Mohammad Abdul Hadi, Imam Nur Bani Yusuf, Fer- dian Thung, Kien Gia Luong, Lingxiao Jiang. Fate- meh Hendijani Fard, and David Lo. 2022. On the effectiveness of pretrained models for api learning. Proceedings of the 30th International Conference on Program Comprehension (ICPC).
Man Luo, Sharad Saxena, Swaroop Mishra, Mihir Par- mar, and Chitta Baral. 2022. Biotabqa: Instruction learning for biomedical table question answering. In CEUR Workshop Proceedings, volume 3 3180, pages 291-304. CEUR-WS.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Man- tas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Xiaodong Song, and Jacob Steinhardt. 2021. Measuring coding chal- lenge competence with apps. Proceedings of the 35th Annual Conference on Neural Information Process- ing Systems (NeurIPS).
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xi- ubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol- instruct. arXiv preprint wxXiv:2306.08566
Naman Jain, Skanda Vaidyanath, Arun Iyer, Nagarajan Natarajan, Suresh Parthasarathy, Sriram Rajamani, and Rahul Sharma. 2022. Jigsaw: Large language models meet program synthesis. In Proceedings of the 44thlnternational Conference on Software Engi- neering (ICSE).
Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannanch Hajishirzi. 2022a. Reframing instructional prompts to GPTk's ' language. In Find- ings of the Association for Computational Linguistics: ACI. 2022.
Aishwarya Kamath and Rajarshi Das. 2018. A survey on semantic parsing. In Automated Knowledge Base Construction (AKBC).
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022b. Cross-task generaliza- tion via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers).
Kirby Kuznia, Swaroop Mishra, Mihir Parmar, and Chitta Baral. 2022. Less is more: Summary of long instructions is better for program synthesis. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 4532- 4552.
Swaroop Mishra and Elnaz Nouri. 2022. Hclp me think: A simple prompting strategy for non-experts to cre- ate customized content with models. arXiv preprint w:Xiv:2208.082:::
Doris Jung-Lin Lee, Dixin Tang, Kunal Agarwal, Thyne Boonmark, Caitlyn Chen, Jake Kang, Ujjaini Mukhopadhyay, Jerry Song, Micah Yong, Marti A. Hearst, and Aditya G. Parameswaran. 2021. Lux: Always-on visualization recommendations for ex- ploratory dataframe workflows. Proc. VLDB Endow. 15(3):727-738.
Avanika Narayan, Ines Chami, Laurel Oir, and Christo- pher Ré. 2022. Can foundation models wrangle your data? Proceedings of the VLDB Endowment. 16(4):738-746.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al.
Yusuke Oda. Hiroyuki Fudaba, Graham Neubig, Hideaki Hata, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. 2015. Learning to generate

---

pseudo-code from source code using statistical ma- chine translation. In 2015 30th IEEE/ACM Interna- tional Conference on Automated Software Engineer- ing (ASE), pages 574-584, IEEE
Manan Dey, M Saiful Bari, Canwen Xu. Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, De- bajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries. Ryan Teehan, Tali Bers, Stella Biderman, Leo Gao. Thomas Wolf, and Alexander M. Rush. 2022. Mul- titask prompted training enables zero-shot task gen- eralization. Proceedings of the 10th International Conference on Learning Representations (ICLR).
OpenAI. 2023. Gpt-4 technical report.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models follow instruc- tions with human feedback. Advances in Nearal Information Processing Systems, 35:27730-27744,
Mihir Parmar, Swaroop Mishra, Mirali Purohit, Man Luo, Murad Mohammad, and Chitta Baral. 2022. In-boxbart: Get instructions into biomedical multi- task learning. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 112- 128.
Torsten Scholak, Nathan Schucher, and Dzmitry Bah- danau. 2021. PICARD: Parsing incrementally for constrained auto-regressive decoding from language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.
Kensen Shi, Joey Hong. Manzil Zaheer, Pengcheng Yin, and Charles Sutton. 2022. Compositional generaliza- tion and decomposition in neural program synthesis. In Deep Learning for Code Workshop.
Pruthvi Patel, Swaroop Mishra, Mihir Parmar, and Chitta Baral. 2022. Is a question decomposition unit all we need? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pages 4553-4569.
Richard Shin and Benjamin Van Durme. 2022. Few- shot semantic parsing with language models trained on code. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistiex: Human Language Technologies.
Hammond A. Pearce, B. Tan, Prashanth Krishnamurthy, Farshad Khorrami, Ramesh Karri, and Brendan Dolan-Gavitt. 2022. Pop quiz! can a large lan- guage model help with reverse engineering? ArXiv, abs/2202.01142.
Mukul Singh, José Cambronero, Sumit Gulwani, Vu Le, Carina Negreanu, Elnaz Nouri, Mohammad Raza, and Gust Verbruggen. 2023. Format5: Abstention and examples for conditional table formatting with natural language.
Gabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Chris Meek, and Sumit Gulwani. 2022. Synchromesh: Reliable code generation from pre-trained language models. Proceedings of the 10th International Conference on Learning Represen- tations (ICLR).
Mukul Singh, José Cambronero, Sumit Gulwani, Vu Le, Carina Negreanu, Mohammad Raza, and Gust Ver- bruggen. 2022. Cornet: Learning table formatting rules by example.
Matt Post. 2018. A call for clarity in reporting BLEU scores. Proceedings of the 3rd Conference on Machine Translation: Rexearch Papers.
Ravsehaj Singh Puri, Swaroop Mishra, Mihir Parmar, and Chitta Baral. 2023. How many data samples is an additional instruction worth? In Findings of the Association for Computational Linguistics: EACL. 2023, pages 1012-1027.
Liwen Wang. Rumei Li, Yang Yan, Yuanmeng Yan, Sirui Wang. Wei Wu, and Weiran Xu. 2022a. In- structionner: A multi task instruction based gener- ative framework for few-shot ner. arXiv preprint mxXiv:2203.0300
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 21(140):1 67.
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Al- isa Liu, Noah A Smith, Daniel Khashabi, and Han- nanch Hajishirzi. 2022b. Self-instruet: Aligning lan- guage model with self generated instructions. arXiv preprint arXiv:2212.1050
Emily Reif, Daphne Ippolito, Ann Yuan, Andy Coenen, Chris Callison- and Jason Wei. 2022. A recipe for arbitrary text style transfer with large language models. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 2: Short Papers).
Yizhong Wang, Swaroop Mishra, Pegah Alipoormo- labashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, An- jana Arunkumar, David Stap, et al. 2022c. Sujan reddy sumanta patro, tanay dixit, and xudong shen. 2022. nnturarraauannnttuutiio
Generalization via declarative instructions on 1600+ nip tasks. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 5085- 5109.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja,

---

Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoe Le. 2022. Finetuned lan- zero-shot learners. Proceedings
guage models of the 10th International Conference on Learning Representations (ICLR).
Orion Weller, Nicholas Lourie, Matt Gardner, and Matthew Peters. 2020. Learning from task de- scriptions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Can Xu, Qingfeng Sun. Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large lan- guage models to follow complex instructions. arXiv preprint STtt:2334.122444
Frank E. Xu, Uri Alon. Graham Neubig, and Vincent Jo- Hellendoorn. 2022. A systematic evaluation of large language models of code. Proceedings of the 6th ACM SIGPLAN International Symposium on Ma- chine Programmingg, MAPS 2022, page 1-10, New York, NY, USA. Association for Computing Machin- ery.
Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. 2018. Learning mine aligned code and natural language pairs from stack overflow. In Proceedings of the 15th interna- tional conference mining software repositories. pages 476-486.
Wenting Zhao, Konstantine Arkoudas, Weiqi Sun, and Claire Cardie. 2022. Compositional task-oriented parsing abstractive question answering. In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4418-4427.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning (ICML).
Ruiqi Zhong, Kristy Lee, Zheng Zhang, and Dan Klein. 2021. Adapting language models for zero-shot learn- ing by meta- tuniittttnnng on dataset and prompt collections. In Findings of the Association for Computational Linguistics: EMNLP 2021.
Ruiqi Zhong, Charlie Snell, Dan Klein, and Jason Fisner. 2022. Active programming by example with a natural language prior, arXiv preprini arXiv:2205.1244

---

A Full Schema Example
F Failure modes of GPT4+DP
Table shows multiple onput- examples. The examples include before and after images of the relevant regions of the Excel spreadsheet.
Table 8 shows major error modes for GPT4+DP. These error modes are similar to those seen for T5 in Figure 5.
B HIT Screenshot
Figure 7 shows screenshot of the interface shown to crowdworkers while inputting file descriptions, NL instructions, and copy-paster code from the Automate feature.
C Longer Instruction
The more detailed instruction used in the Max- Inotttnstooction setting is: 'Generate function with excel script to execute the action given be- low in NL. You also need to generate comment de- scribing the operation you are performing. Make sure to generate valid excel operation and pass appropriate parameters provided in the action information. Simple solution is preferred over complex one.'
D Finetuning Parameters
T5 training was done for 5, 000 steps, with learning rate increasing linearly from 0 to 1 10-4 for the first 1, 000 steps, and then linearly down to O for the remaining steps. We restrict both the input and output size to 1, 000 tokens. If inputs and outputs are denoted as I, and O; = (44 /== respectively. then the finetuning objective is to maximize the probability of the true output given the input,
(0)00)
10
4)) = I p( 11 41
At inference time, we use beam search with beam size 5, and select the top completion for final evaluation.
E Full Experimental Results
The full results for the Zero-shot, Few-shot, Max- shot, and Max-shot+Instruetion settings on all mod- els are reported in Table 6. In addition to the evalu- ation metrics reported in Table 6, we also report the metrics after replacing all parameters, ranges, and numbers in both the gold standard and pre- dicted outputs with placeholder values (function- based evaluation). These modified metrics are re- ported in Table Although the metrics are much higher in this setting, the same overall trends hold.

---

Query
Gold Standard Output
function main (workbook: ExcelSeript workbeok)
Find
unique
from column
1et selectedSheet workbook, -er.Attiveoorkshect
Method range C11:C42 on G17
// range G17 selleceddheett
on sheet Au
se eeceeeheett eeteedhhott.getaaagee (*E17") wwt.orrul
"=unique (C11: ");}
MS
NAA
NAA
FA
AAS
AAS
FA
AAS
AAS
FA
AAS
AAS
NAA
PR.AR
MS
PRAR
MS
PR.AR
FA
ES
FA
ES
AR
FA
ICP
FA
ICP
BLEG
FA
ES
FA
ES
(workbook: Exceiscript. Norkkbckk)
Sendttiona.Pormattinn ExceiScript Fonoirttoo..eraatt
le= selectediheet workbook workbook.gettrttiveeeee wet.sssttteeeeseaaeeeetetttteeee 11
1/ Change top bot from range R8:P22 on se selecteddh
conditionslPormattiig - selectedSheet. get.hheeeetgtttggeeeeg... (*D6:2222)
Find top value from range B6:P22 on sheet CVs
Exce Ser peript,,oom Condiit Fona Pornal Type. Top.
conditionas Pon.ddiitn......tinnn.... ptt.:aastttt...oopsstttt.o.m gettormatt
.geti'ont ssetColor #300000");
condstlealllrmatiiii.eeeco htt.:opactttoo...eeeeettttt -getsormat
.getFill aee (e0
|rank:2,
type
11
0.000
0.00
98.65
00.00
98.01
86.16
0.00
99.84 96.18
0.00
49.24
66.59
0.00
49.24
58
0.00
function main (workbook: Rxcel Script. Workbook)
replace all N/A with 0
1et selecredSheet workbook geteeeaeeeeeeeeeeeeeeeeet..ttt..
selectedSheet. roplacodll ("N/A", "0",
[completeMatch: false, matchCase false))
Table S1, Study participant COVI-11 Diaghosis and
Table S1, Study partlcipant COVID-19 Diagnosis and
Pilen Mortim
Palend Stawen
FC3
NM
0
P43
FCE
FCB
IM
PC9
FC7
MM
0092
00P3
00R)
45
function main (workbook:
ExcelScript Workbook)
replace all N/A with 0
let selectedSheet workbook www.tttiieewrsssateeeeeeeeeeeeee ()
selectedSheet.co getaange ("B4: """":ial"""..rrpaceeee ("N/A", "o",
[complett fa matchCase: false))
Table 5: Example natural language queries and gold standard code outputs. Examples include images of the Excel spreadshect before and after executing the gold standard code. Example 4 shows the same query on the same data as example 3, solving using slightly different code.

---

Excel Click Task
Previns Dedggn Oebug mode
bobut
Svarthee
Uaay Saurrory Micioa
Report Tcchiiar Tsan:
Excel sheet: https://microsoff
Type the operalion you want to perform]
Can you write why you went do this operator on this specitic sheet? (Optionnl)
Type aclion here Pasle
Type action Paule code here
Type action Paste code here
4. Type action here Postte mode here
5. Type action here Paste code here
Figure 7: Screenshot of the HIT given to crowdworkers who created the INSTRUCTEXCEL dataset.
Model
Setup
EM
ROUGE
F1
SacreBLEU
Zero-shot
0.0
29.46
5.44
21.75
Few-shot (3)
1.0
34.94
11.10
28.31
GPT 3.5 Turbo
Max-shot
0.0
34.47
5.09
29.12
Max-shott-Instruction
0.0
35.67
2.67
29.53
Zero-shot
0.0
21.59
6.75
16.90
Few-shot (3)
1.5
42.57
31.73
31.09
GPT 4
Max-shot
3.0
52.70
42.62
40.08
Max- Max-shot-Instractton
3.0
41.86
38.34
31.71
Zero-shot
0.00
30.57
10.66
20.32
Few-shot
3.50
41.79
31.63
30.56
GPT 4 + API
Max-shot
1.50
43.18
40.96
31.24
Max-shot-Inssrructon
1.50
33.15
30.95
23.80
Zero-shot
0.00
22.84
7.00
18.16
Few-shot
11.00
67.28
58.23
59.23
GPT 4+ DP
Max-shot
15.00
69.59
61.83
62.60
Max-shote Instruuttton
15.00
69.22
61.95
62.29
Zero-shot
0.00
30.37
10.28
20.15
Few-shot
11.00
62.88
55.05
55.71
GPT 4+ API DP
Max-shot
15.00
63.99
57.83
57.47
Max- SonsshootIInttruttion
13.50
64.61
60.15
57.94
T5
Finetuning
17.00
76.73
72.06
68.81
Table 6: Model performance in different evaluation settings. EM is normalized exact match.

---

Model
Setup
EM
ROUGE
F1
SacreBLEU
Zero-shot
0.0
36.67
4.13
25.65
Few-shot (3)
1.0
42.02
11.23
34.55
GPT 3.5 Turbo
Max-shot
0.5
42.69
10.68
33.86
Max-shot--nnsrrutiion
0.5
44.04
6.64
35.59
Zero-shot
0.0
26.81
5.77
22.56
Few-shot (3)
1.5
50.05
35.35
39.25
GPT
Max-shot
15.5
63.25
52.73
52.15
7.0
52.10
45.23
40.57
Zero-shot
0.00
35.06
9.90
23.49
Few-shot
3.50
49.79
35.75
38.31
GPT 4 API
Max-shot
12.00
54.33
48.09
41.95
Max-shot-Instruction
1.50
43.86
36.08
32.28
Zero-shot
0.00
28.26
6.88
22.97
Few-shot
33.50
77.91
68.91
71.19
GPT 4 + Dynamic Prompt
Max-shot
41.50
80.80
73.56
74.91
Max-shot-Instruttion
41.00
80.12
73.30
74.34
Zero-shot
0.00
34.91
9,96
23.22
Few-shot
32.00
73.47
64.68
66.86
GPT + API+ Dynamic Prompting
Max-shot
36.00
74.78
67.94
68.71
Max-shot+Insiin
38.00
76.09
70.81
70.03
T5
Finetuning
45.50
87.58
83.46
81.70
Table 7: Results ignoring parameters, ranges, and numbers (function-basee evaluation).

---

Error
Ex. Input
Pred. Output
Gold Std.
User intention
Change the legends position
selestedSheet eeheetteeheett.eddhher rt
chart chart__3.getieeee 3.getZegend
misunder-
ExcelSeript www.serrii..hhatttyye.... ChartType. columnClustered
.eetPosition
stood/incorrect
sheet data in chart
ExcelSeript.
task
se lesteet wtw.eeddeeettttttgggggggeoe Ret *x28:630"11>
Cha < itngeenddoiitio.tppp.
Incorrect target for data opera- tion
Data Validation for drop down for column Lab from range A11:A101 101 on sheet Au source: "=$A$1 SAS101")1)
se lectedSheet
selectedShest
("M13:231")
geLRange (eLRaage("AA11)
gettaraaallidttion/
getExtendedRange
.atRulee
ExcelScript.
[lit:: (
KeyboardDirect ion.roooodddddnn
.eetRule
list:
Incorrect for- Apply proper mula applied
se selecteSSeeet
selectedSheat
function to cell C4 in sheet Supplementary Data
getRange ("C4")
PetRaage("I10)
se Farrul a
seLForrulaToce
AAERRGGE (C5:
"=proper ")
Overwriting im- portant data
Apply isnumber function to cell B7 of column price of sheet 8.S1a Airline Demand
selectedSheet.
selectedSheet
getRange ("37")
getRange ("H9")
setFormulalocsl
setFormulcLocal (NUMMER((R))))) ");
TSNUMBER ")
Table Types of errors made by GPT4+DP model. The error type listed, along with NL query, predicted (incorrect) output by GPT4+DP, and the gold standard output. Code is shortened only include the relevant. erroneous function calls.


Reviews:
[{"review_id": "6RYOSzKUMg", "reviewer": "Reviewer_DQNq", "summary": "", "questions": "**A.** The manual annotation process revealed some disagreements among experts on the correctness of model outputs. How do you plan to refine the evaluation criteria to ensure more clarity and consistency in future iterations of the benchmark?\n\n**B.** While the benchmark focuses on Excel, many other software applications could benefit from NLP integration. Do you have plans to expand or adapt the benchmark to other software applications or platforms?\n\n**C.** How does the INSTRUCT EXCEL benchmark compare to other Code based instruction benchmarks in terms of complexity and diversity of tasks? Can you provide insights into its uniqueness or advantages over other benchmarks?\n\n**D.** Can you elaborate on the real-world applicability of the benchmark? How does performance on this benchmark translate to effectiveness in real-world Excel tasks?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s detailed and insightful feedback. We are encouraged by the reviewer’s acknowledgment of our contributions: “Performance Metrics”, “Model Setups”, “Function-Based Evaluation”, “Manual Annotation” and “Visual Aids”. We are motivated that the reviewer finds InstructExcel to be a “systematic benchmark”, appreciates the “broad evaluation spectrum” and “novel function-based evaluation approach”, and suggests our “research bridges language models and spreadsheet software, heralding a new research domain with potential real-world applications in industries” and “offers insights into the intricacies of manual annotation”.\n\nWe respond to major comments below:\n\nR1.1: “Ambiguity in Evaluation”: Excel is used by billions of users with diverse backgrounds consisting of experts and non-expert users. We selected annotators with diverse Excel skill levels to match the real world user distribution. Some of the paper's authors had extensive experience with OfficeScripts, while others had less experience with OfficeScripts (but more experience in training and evaluating LLMs). The less-experienced annotators looked for similarity of the model output with the gold answer, whereas the expert annotators allowed differences with the gold output as long as the API generated by the model compiles and does the required task. We can clarify this in the camera ready version. We will also describe two different guidelines for expert and less-experienced annotators.\n\nR1.2: “Limited Scope”: Our decision to focus on Excel is driven by the fact that Excel has over a billion users, and is still a hard task for state of the art models like GPT4. Excel workflow is part of daily life for many users and we would like to apply advancements in large language models to significantly improve their workflow. InstructExcel is a stepping stone in that direction. We hope the community will expand our benchmark to other tools in future.\n\nR1.3: “Lack of Comparative Analysis”, “real world excel tasks”: Our work is at the intersection of instruction learning and program synthesis. In the instruction learning space, the input is a natural language instruction and the output is an answer, story, poem, email etc. Natural Instructions [1], Supernatural Instructions [2], and Promptsource [3] are some popular instruction learning benchmarks, however they are focused on general NLP. Program synthesis benchmarks such as MBPP [4], APPS [5], CodeContests [6], CoNaLa [7], DJANGO [8], and HumanEval [9] on the other hand involve conversion of natural language to code, but this line of work has been predominantly focused on general purpose programming languages such as Python, and C++. Our benchmark InstructExcel instead focuses on Excel OfficeScripts, a TypeScript API which has not been explored in existing benchmarks.  We will ensure all relevant instruction learning and program synthesis benchmarks are included in the related work section.\n\nOur work can be immediately applied to a product used by billions of users, and our benchmark is based on real-world Excel files which provides unique practical advantages over other academic benchmarks on instruction following. Additionally, our task is more complex for average humans unlike conventional instruction following tasks such as writing a story, email, or joke. In addition to the main NL-to-code task, InstructExcel provides a reservoir for studying several other tasks e.g. the conditional formatting task, sub-task extraction (L083-092). During the creation of our benchmark, we emphasized usage of popular Excel operations. We worked directly with the vendor (Universal Human Relevance System) to provide diverse examples to the annotators, and train the annotators to create a wide distribution of realistic NL queries. We also introduced an additional field to further explain the users’ intentions (Figure 8). The 170+ operations used in our benchmark cover diverse use cases of the Excel software, with popular use cases dominant in the distribution (Figure 3).\n\nR1.4: “Potential Overemphasis on Metrics”, “detailed error analyses”: We conducted error analysis and qualitative evaluation. Due to space constraints, we moved this content to Appendices D and E. We will bring some of these to the main paper in the camera ready version upon getting additional space. If you would like to see additional analysis, please let us know what you suggest and we will incorporate it.\n\nReferences: \n\n[1] Mishra et al. 2022, Cross-task generalization via natural language crowdsourcing instructions.\n\n[2] Mishra et al. 2022, Super-naturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks.\n\n[3] Bach et al. 2023, PromptSource: An integrated development environment and repository for natural language prompts.\n\n[4] Austin et al. 2021, Program synthesis with large language models.\n\n[5] Hendrycks et al. 2021, Measuring coding challenge competence with APPS.\n\n[6] Li et al. 2022, Competition-level code generation with AlphaCode.\n\n[7] Yin et al. 2018, Learning to mine aligned code and natural language pairs from Stack Overflow.\n\n[8] Oda et al. 2015, Learning to generate pseudo-code from source code using statistical machine translation.\n\n[9] Chen et al. 2021, Evaluating large language models trained on code.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces \"INSTRUCT EXCEL\", a systematic benchmark for evaluating language models' interaction with Microsoft Excel.\n2. It provides a broad evaluation spectrum, testing various models like GPT and T5 across setups such as zero-shot, few-shot, and max-shot.\n3. A novel function-based evaluation approach is introduced, emphasizing models' functional understanding over exact output matching.\n4. The research bridges language models and spreadsheet software, heralding a new research domain with potential real-world applications in industries relying on tools like Excel.\n5. The study offers insights into the intricacies of manual annotation, underscoring the importance of collaborative efforts in clarifying output ambiguities and enhancing model evaluations.", "reasons_to_reject": "1. **Ambiguity in Evaluation**: The manual annotation process, as discussed in the paper, indicates that even experts found it challenging to agree on the correctness of model outputs. This could raise questions about the consistency and clarity of evaluation criteria.\n\n2. **Limited Scope**: While the focus on Excel is pertinent given its widespread use, the benchmark might be considered too niche. Some might argue for a more generalized benchmark that encompasses a broader range of software applications beyond Excel.\n\n3. **Lack of Comparative Analysis**: While the paper introduces a new benchmark and tests several models on it, there might be limited discussion on how these results compare to other existing benchmarks or real-world tasks, potentially reducing its impact.\n\n4. **Potential Overemphasis on Metrics**: While metrics like EM, ROUGE, and SacreBLEU are valuable, the paper might be critiqued for not diving deeper into qualitative insights or providing more detailed error analyses.", "questions_to_authors": "**A.** The manual annotation process revealed some disagreements among experts on the correctness of model outputs. How do you plan to refine the evaluation criteria to ensure more clarity and consistency in future iterations of the benchmark?\n\n**B.** While the benchmark focuses on Excel, many other software applications could benefit from NLP integration. Do you have plans to expand or adapt the benchmark to other software applications or platforms?\n\n**C.** How does the INSTRUCT EXCEL benchmark compare to other Code based instruction benchmarks in terms of complexity and diversity of tasks? Can you provide insights into its uniqueness or advantages over other benchmarks?\n\n**D.** Can you elaborate on the real-world applicability of the benchmark? How does performance on this benchmark translate to effectiveness in real-world Excel tasks?", "paper_topic_and_main_contributions": "1. **Performance Metrics**: The document uses multiple metrics to evaluate model performance. \n2. **Model Setups**: Various configurations and versions of models like GPT-3.5 Turbo, GPT-4, GPT-4 with API, GPT-4 with Dynamic Prompting, and T5 are tested. There are also variations such as zero-shot, few-shot, max-shot, and max-shot with instructions.\n3. **Function-Based Evaluation**: Table 5 suggests that there's an evaluation setting where parameters, ranges, and numbers were ignored, implying a more function-based assessment.\n4. **Manual Annotation**: There's a mention of manual error annotations performed by three authors. They used these annotations to verify the automated metrics. Table 6 seems to present a statistical analysis of this, using a one-sided t-test to compare manual judgments with automated metric scores.\n5. **Visual Aids**: The document includes several figures. Figure 6 shows the structure of various fields in the benchmark, Figure 7 displays Excel file contents, Figure 8 is a screenshot of the HIT given to crowdworkers, and Figure 9 provides examples of errors made on test set examples.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3CyypaBGM8", "reviewer": "Reviewer_6cHD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for acknowledging our work to contain a “challenging benchmark” and “overall good writing”. We address the two major concerns below:\n\nR2.1: “ traditional metrics based on distance to golden labels may not be a suitable evaluation… A better alternative may be to show the true accuracy….”: This was a very fruitful question, thank you. We manually annotated 40 examples for the bottom 3 experimental settings in Table 2: GPT4+DP, GPT4+API+DP, and Finetuned T5. For this manual annotation, we download the associated spreadsheets and compare the results of executing the predicted code against the natural language queries. We find that among these 40 annotated examples, GPT4+DP shows 65% accuracy, GPT4+API+DP shows 50% accuracy, and the Finetuned T5 model shows 57.5% accuracy. This small-scale analysis suggests that you are absolutely correct; although the automated metrics are useful coarse-grained measures of model accuracy, they cannot replace manual annotation for detailed model comparison. We intend to extend the manual annotation to all 200 test examples and report these results in the final version of the paper. This nuance of evaluation will be very important information for future users of our benchmark. \n\nAutomation of this process remains out of scope for the current work, as it is a non-trivial problem. It may be possible to use PowerAutomate to execute the predicted and gold standard code completions, and then load them into Python using Pandas to compare the results. However, it is unclear how to execute this workflow programmatically (search for “run PowerAutomate from command line” to see others struggling with the same problem). We will include more discussion of the difficulties of automated evaluation in the final version. \n\nR2:2: “average number of examples used in 3-shot and max-shot experiments, accounting for truncation. ”:  Given that our main contribution is the benchmark itself, our data truncation scheme was intended to be a baseline for future work to build upon. We analyzed the two most space-restricted settings in our experiments, the maxshot+instruction settings with both the GPT3.5 model with 16k context size, and the GPT4 model with 32k context size + API in the prompt. Both settings always include the intended 10 in-context examples in the prompt, and the data included in the examples retain an average of 19.36 lines in the first case and 15.36 lines in the second case. This typically guarantees that the headers and some data are included from the first spreadsheet in the Excel file, though occasionally later spreadsheets have important data as well. We hope that follow-up work will address this non-trivial issue. There has been a recent trend towards increasing input context length, so perhaps increased context size will make this problem irrelevant in the future.\n\nR2.3: Typos Grammar Style And Presentation Improvements: Thank you for your comments on improving the presentation of the experiments and examples. We will certainly implement these suggestions in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors provide a challenging benchmark for the important task of (generating code for) automatically solving spreadsheet tasks via natural language instructions.\n- The authors perform experiments on common methods which shed light on the capabilities of current popular models/approaches and can serve as baselines for future research.\n- Writing is overall good.", "reasons_to_reject": "- The use of traditional metrics based on distance to golden labels may not be a suitable evaluation metric for this task. The justification proposed by the authors in Appendix D does not convince me otherwise, as the hypothesis for the t-test is very weak (i.e., that the average scores of the automated metrics are lower for incorrect samples from a single experiment, i.e., GPT-4 with max-shot prompting). This does not justify the use of the proposed metrics *for comparing the performance between different models*. A better alternative may be to show the *true* accuracy determined by human annotators based on random-selected samples for *each* main baseline method, alongside the scores from automated metrics. In fact, there is no mention of true model accuracy, i.e., the portion of *acceptable* answers, determined by annotators.\n  - For this, my soundness score for the baseline experiment part would be 2/5. If the authors could show an automated metric that can faithfully represent the ability of models to solve the task, that would be a highly relevant contribution alongside the proposed benchmark. However, the paper seems to fall short of this in its current state. Other than the experiments, the benchmark itself looks sound.\n- It may be informative to show the statistics for the average number of examples used in 3-shot and max-shot experiments, accounting for truncation. I'm also concerned about the truncation of *table data* due to limited context sizes discussed in Line ~382, as this would be very detrimental to task performance. While I understand that this is not a trivial issue to solve, some discussion of the severity of data truncation or statistics may help readers consider this when reading the experimental results. While the exact truncation protocol is explained in a dedicated subsection (4.3), It seems that the consequences of truncation have not been thoroughly considered by the authors. *If data truncation does not affect a significant number of samples, it would be nice if this is mentioned.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors share a benchmark dataset that contains (1) links to ~2000 public spreadsheets, (2) instructions for ~10K sample actions that may be performed in those spreadsheets, as queries, and (3) OfficeScript code that executes those actions, as target labels, (obtained via the Automate feature on Excel which records annotator actions into code).\n\nThe authors perform simple baseline experiments to identify the efficacy of common approaches: (1) zero-shot, few-shot, and *dynamic* [1] prompting on GPT 3.5 & 4, and (2) fine-tuning T5 large models. \n\nThe authors also highlight the potential to derive fine-grained benchmarks using subsets of the proposed dataset, providing a case study for the popular task of conditional formatting rules.\n\n[1] Liu 2021, What Makes Good In-Context Examples for GPT-3?", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "*I'm not familiar with this line of work.*"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- The results section could be expanded to more effectively deliver key insights on baselines. E.g., the expanded performance table (Table 5) could be moved to the main text, with a more thorough summary of findings in the main text of Section 5. Some content such as experimental details and Figure 3 could be shortened.\n- Comprehensive examples figures for selected samples from the dataset may help the reader intuitively understand the content of the benchmark. I.e., it may be better to organize the input, output (from Figure 6), and data contents (Figure 7) of each sample into a single, well-organized, readable figure. It would also be nice to show *multiple*, say 3, of such figures, to show a wide range of samples."}}, {"review_id": "I3DK21BqNO", "reviewer": "Reviewer_em7S", "summary": "", "questions": "A. What is the \"inter-annotator agreement rate\"? Give a natural language description to another worker, how often can the other worker reproduce the same action? I believe such a measurement is important, since it is easy for an annotator to think that the description they came up with is unambiguous, while another annotator interpreted it differently.\n\nB. How did you make sure that the input descriptions are diverse and of high quality? Asking annotators to come up with their own description is different from the distribution of descriptions users would naturally generate. While I understand that it'd be difficult to collect data from the real users, providing more details in this respect in the main paper would be helpful.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are thankful to the reviewer for their encouraging feedback. We appreciate the reviewer's judgment that we provide a “useful dataset” with the “interesting” “idea of ‘recording annotator's action' to form the annotated output”. We address the reviewer’s concerns below.\n\nR3.1 “inter-annotator agreement rate”: This is an interesting question, thank you.\n\nWe ran a qualitative analysis on inputs which appear at least twice in the benchmark. We find that most of these inputs have very high “inter-annotator agreement rate”; for example, there is only one way to satisfy “freeze row 1”: “selectedSheet.getFreezePanes().freezeAt(selectedSheet.getRange(\"1:1\"));”. However, some of these inputs can be satisfied multiple ways, for example, we see “replace all N/A with 0” satisfied by both “selectedSheet.replaceAll(\"N/A\", \"0\", {completeMatch: false, matchCase: false});” and “selectedSheet.getRange(\"B4:B10\").replaceAll(\"N/A\", \"0\", {completeMatch: false, matchCase: false});”. We can include this qualitative analysis in the final version of the paper, as well as a report on the average value of the automated metrics computed pairwise within each query with at least 2 completions.\n\nR3.2: “details on how they controlled the quality of the dataset.”, “input descriptions are diverse and of high quality”: Thanks for the great question. Here are more details on our process to control data quality.\n\n1. Multiple rounds of iterations over the instructions with pilot studies\n2. Various examples provided to annotators to demonstrate a variety of actions (these could be toggled on and off in the annotation interface)\n3. Instructed annotators to not overuse or underuse certain sets of actions after observing the distribution of actions in pilot rounds\n4. Used annotators that are paid hourly ($12 USD per hour) for delivering high quality work and not annotators that are motivated to click on as many hits as possible in a short span of time \n5. Employed an annotation project manager to facilitate training of annotation team, and to facilitate later communication with team\n6. Inspected sample annotations\n7. Selection of a certain set of files that were most suitable for the task: large enough to have nontrivial contents, but not so large that it would be impossible for annotators to understand the content (files between 60KB and 100KB)\n8. Selected only English-language files\n9. Filtered for password and corrupted files\n\nWe emphasized getting diverse and high quality descriptions during our selection of annotation vendors and in our pilot studies. Our initial pilot study indeed had repetitive Excel operations which we had to reject. We requested our vendor to arrange a pool of annotators who have basic familiarity with Excel applications. We also provided the vendor a list of top popular Excel operations which they must use. Additionally, we introduced an input field asking users to mention the intent behind the operation they want to perform (Figure 8). Inspired by other benchmark papers, we periodically reviewed the data and plotted the histogram of Excel operations (Figure 3), and accordingly informed the vendor of (1) the Excel operations they should not use anymore since they are already overpopulated, and (2) the operations they are encouraged to use since they are missing/underpopulated. The 170+ operations our benchmark contains cover diverse use cases of the Excel software, with the popular use cases dominant in the distribution (Figure 3).\n\nBased on the reviewer's suggestion, we will use our extra page to bring content from the appendix to the main paper in the camera ready version.\n\nR3.3.: Typos Grammar Style And Presentation Improvements: Thanks for pointing this out, we will update this in the camera ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper provides a useful dataset for the community (instruction -> code);\n- The idea of \"recording annotator's action' to form the annotated output is interesting and was not widely used in the semantic parsing literature before.\n- The empirical results (e.g. detailed instructions do not help much) are interesting to the community.", "reasons_to_reject": "It'd be useful for the authors to provide further details on how they controlled the quality of the dataset. On the other hand I would not set a high bar for the inter-annotator agreement rate and the justification for diversity as long as they are reasonable and reported in the paper. I'm ready to raise my score on soundness.  \n\nP.S. Sorry that I don't have time to go through the details in the appendix if you provide those details there (and I believe that the main paper should be self-contained). Feel free to address them in your updated version when an additional page is available for camera ready.", "questions_to_authors": "A. What is the \"inter-annotator agreement rate\"? Give a natural language description to another worker, how often can the other worker reproduce the same action? I believe such a measurement is important, since it is easy for an annotator to think that the description they came up with is unambiguous, while another annotator interpreted it differently.\n\nB. How did you make sure that the input descriptions are diverse and of high quality? Asking annotators to come up with their own description is different from the distribution of descriptions users would naturally generate. While I understand that it'd be difficult to collect data from the real users, providing more details in this respect in the main paper would be helpful.", "paper_topic_and_main_contributions": "The paper contributes a code generation dataset for Excel: the input is a natural language instruction and the output is a typescript representing a series of Excel operations. The dataset contains 10k samples over 2,000 publicly available spreadsheets. Preliminary experiments show that GPT-4 performs poorly, indicating plenty of room for improvement. The authors also ran a series of baseline approaches to improve the performance, such as increasing in-context examples, dynamic prompting etc.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "L475: while I believe that GPT-4 is larger than GPT-3.5 and larger models are probably indeed better, the fact that GPT-4 performance is better than GPT-3.5 does not imply that larger models are better because they are blackbox APIs and can differ on many different axes (e.g., training data). You could draw the conclusion if you compared fine-tuned T5-large vs T5-medium (and many other T5-series) though."}}]

Author rebuttal:

Meta-review: The paper "INSTRUCT EXCEL: A Benchmark for Evaluating Language Models in Spreadsheet Software" has received positive feedback for its innovative benchmark dataset and the insights it provides into language model performance. However, concerns about evaluation criteria, limited scope, and the need for further details on dataset quality control and inter-annotator agreement need to be addressed. The average soundness score is strong (4), indicating a solid foundation, while the average excitement score is ambivalent (3), suggesting room for improvement.
Decision: Accept-Findings