SpreadsheetCoder: Formula Prediction from Semi-structured Context

1 INTRODUCTION . Spreadsheets are ubiquitous for data storage , with hundreds of millions of users . Support for helping users write formulas in spreadsheets is a powerful feature for data analysis . Although spreadsheet formula languages are relatively simpler than general-purpose programming languages for data manipulation , writing spreadsheet formulas could still be tedious and error-prone for end users ( Gulwani , 2011 ; Hermans et al. , 2012b ; Cheung et al. , 2016 ) . Systems such as FlashFill ( Gulwani , 2011 ; Gulwani et al. , 2012 ) help end-users perform string transformation tasks in spreadsheets using a few input-output examples by automatically synthesizing a program in a domain-specific language ( DSL ) . Recently , several learning approaches based on different neural architectures have been developed for learning such programs from examples , and have demonstrated promising results ( Parisotto et al. , 2017 ; Devlin et al. , 2017 ; Vijayakumar et al. , 2018 ) . All these previous works formalize the spreadsheet program prediction problem as a programming by example task , with the goal of synthesizing programs from a small number of input-output examples . We argue that this choice engenders three key limitations . First , this setup assumes that each data row is independent , and each formula is executed on data cells of the same row . However , real spreadsheets are less structured than this . Data in spreadsheets is typically organized as semi-structured tables , and cells in different rows could be correlated . As shown in Figure 1 , in the same table , different data blocks could have different structures , without a common schema . Formulas can take cell values in other rows as function arguments . Second , because spreadsheets are semi-structured , they also contain rich metadata . In particular , many spreadsheet tables include headers that provide high-level descriptions of the data , which could provide important clues for formula prediction . However , table headers are not utilized in prior work . Finally , programming-by-example methods output programs in a DSL , which is typically designed to facilitate synthesis , and is much less flexible than the language in which users write formulas . For example , the FlashFill DSL only covers a subset of spreadsheet functions for string processing , and it does not support rectangular ranges , a common feature of spreadsheet formulas . In contrast , spreadsheet languages also support a wide variety of functions for numerical calculation , while the argument selection is more flexible and takes the spreadsheet table structure into account . In total , these limitations can compromise the applicability of such prior efforts to more diverse real-world spreadsheets and to richer language functionality . Instead , we propose synthesizing spreadsheet formulas without an explicit specification . To predict a formula in a given cell , the context of data and metadata is used as an implicit ( partial ) specification of the desired program . For example ( Figure 1b ) , if predicting a formula at the end of a column of numbers labeled “ Score ” , and a cell in the same row contains the text “ Total ” , this context might specify the user ’ s intent to compute a column sum . Our problem brings several new challenges compared to related work in programming by example ( Gulwani , 2011 ; Bunel et al. , 2018 ; Balog et al. , 2017 ) , semantic parsing ( Popescu et al. , 2003 ; Zhong et al. , 2017 ; Yu et al. , 2018 ) and source code completion ( Raychev et al. , 2014 ; Li et al. , 2018 ; Svyatkovskiy et al. , 2019 ) . Spreadsheet tables contain rich two-dimensional relational structure and natural language metadata , but the rows do not follow a fixed schema as in a relational database . Meanwhile , our tabular context is more ambiguous as the program specification , and the spreadsheet language studied in this work is more flexible than in the program synthesis literature . In this paper , we present SPREADSHEETCODER , a neural network architecture for spreadsheet formula prediction . SPREADSHEETCODER encodes the spreadsheet context in its table format , and generates the corresponding formula in the target cell . A BERT-based encoder ( Devlin et al. , 2019 ) computes an embedding vector for each input token , incorporating the contextual information from nearby rows and columns . The BERT encoder is initialized from the weights pre-trained on English text corpora , which is beneficial for encoding table headers . To handle cell references , we propose a two-stage decoding process inspired by sketch learning for program synthesis ( Solar-Lezama , 2008 ; Murali et al. , 2018 ; Dong & Lapata , 2018 ; Nye et al. , 2019 ) . Our decoder first generates a formula sketch , which does not include concrete cell references , and then predicts the corresponding cell ranges to generate the complete formula . For evaluation ( Section 4 ) , we construct a large-scale benchmark of spreadsheets publicly shared within our organization . We show that SPREADSHEETCODER outperforms neural network approaches for programming by example ( Devlin et al. , 2017 ) , and achieves 42.51 % top-1 full-formula accuracy , and 57.41 % top-1 formula-sketch accuracy , both of which are already high enough to be practically useful . Moreover , SPREADSHEETCODER can predict cell ranges and around a hundred different spreadsheet operators , which is much more flexible than DSLs used in prior works . With various ablation experiments , we demonstrate that both implicit specification from the context and text from the headers are crucial for obtaining good performance . 2 PROBLEM SETUP . In this section , we discuss the setup of our spreadsheet formula prediction problem . We first describe the input specification , then introduce the language and representation for spreadsheet formulas . Input specification . We illustrate the input context in Figure 1 . The input context consists of two parts : ( a ) context surrounding the target cell ( e.g. , all cell values in rows 2–7 , and columns A–D , excluding cell D4 in Figure 1a ) , and ( b ) the header row ( e.g. , row 1 ) . In contrast to prior programming-by-example approaches ( Gulwani , 2011 ; Parisotto et al. , 2017 ; Devlin et al. , 2017 ; Vijayakumar et al. , 2018 ) , our input specification features ( a ) tabular input , rather than independent rows as input-output examples , and ( b ) header information . Tabular input is important for many cases where formulas are executed on various input cells from different rows and columns ( Figure 1 ) , and headers hold clues about the purpose of a column as well as its intended type , e.g , the header cell `` Score '' in Figure 1b is likely to indicate that the column data should be numbers . Note that we do not include the intended output of the target cell in our input specification , for three reasons . First , unlike programming-by-example problems , we do not have multiple independent input-output examples available from which to induce a formula , so providing multiple input-output examples is not an option . Second , even for our single input instance , the evaluated formula value may not be known by the spreadsheet user yet . Finally , we tried including the intended formula execution result in our specification , but it did not improve the prediction accuracy beyond what the contextual information alone allowed . The spreadsheet language . Our model predicts formulas written in the Google Sheets language1 . Compared to the domain-specific language defined in FlashFill , which focuses on string transfor- 1Google Sheets function list : https : //support.google.com/docs/table/25273 ? hl=en . mations , the spreadsheet language supports a richer set of operators . Besides string manipulation operators such as CONCATENATE , LOWER , etc. , the spreadsheet language also includes operators for numerical calculations ( e.g. , SUM and AVERAGE ) , table lookups ( e.g. , VLOOKUP ) and conditional statements ( IF , IFS ) . As will be discussed in Section 4 , around a hundred different base formula functions appear in our dataset , many more than the operators defined in the FlashFill DSL . We limit our problem to formulas with references to local cells in a spreadsheet tab , thus we exclude formulas with references to other tabs or spreadsheets , and absolute cell ranges . Formula representation . One of the key challenges in formula representation is how to represent cell references , especially ranges , which are prevalent in spreadsheet formulas . Naively using the absolute cell positions , e.g. , A5 , may not be meaningful across different spreadsheets . Meanwhile , a single spreadsheet can have millions of cells , thus the set of possible ranges is very large . To address this , we design a representation for formula sketches inspired by prior work on sketch learning for program synthesis ( Solar-Lezama , 2008 ; Murali et al. , 2018 ; Dong & Lapata , 2018 ; Nye et al. , 2019 ) . A formula sketch includes every token in the prefix representation of the parse tree of the spreadsheet formula , except for cell references . References , which can be either a single cell or a range of cells , are replaced with a special placeholder RANGE token . For example , the sketch of the formula in Figure 1a is IF < = RANGE 1 `` A '' IF < = RANGE 2 `` B '' IF < = RANGE 3 `` C '' IF < = RANGE 4 `` D '' `` E '' $ ENDSKETCH $ , where $ ENDSKETCH $ denotes the end of the sketch . Notice that the sketch includes literals , such as the constants 1 and `` A '' . To complete the formula representation , we design an intermediate representation for ranges , relative to the target cell . For example , B5 in Figure 1c is represented as $ R $ R [ 0 ] C [ 1 ] $ ENDR $ since it is on the next column but the same row as the target cell A5 , and range C2 : C6 in Figure 1b is represented as $ R $ R [ -5 ] C [ 0 ] $ SEP $ R [ -1 ] C [ 0 ] $ ENDR $ . The special tokens $ R $ and $ ENDR $ start and conclude a concrete range , respectively , and $ SEP $ separates the beginning and end ( relative ) references of a rectangular multi-cell range . A complete spreadsheet formula includes both the sketch and any concrete ranges ; e.g. , the formula in Figure 1b is represented as SUM RANGE $ ENDSKETCH $ $ R $ R [ -5 ] C [ 0 ] $ SEP $ R [ -1 ] C [ 0 ] $ ENDR $ EOF , where EOF denotes the end of the formula . In Section 3.2 , we will discuss our two-stage decoding process , which sequentially predicts the formula sketch and ranges . 3 SPREADSHEETCODER MODEL ARCHITECTURE . In this section , we present our SPREADSHEETCODER model architecture for spreadsheet formula prediction . We provide an overview of our model design in Figure 2 . 3.1 TABULAR CONTEXT ENCODER . Input representation . Our model input includes the surrounding data values of the target cell as a table , and the first row is the header . When there is no header in the spreadsheet table , we set the header row to be an empty sequence . We include data values in cells that are at most D rows and D columns away from the target cell , so that the input dimension is ( 2D + 2 ) × ( 2D + 1 ) , and we set D = 10 in our experiments . Row-based BERT encoder . We first use a BERT encoder ( Devlin et al. , 2019 ) to compute a rowbased contextual embedding for each token in the target cell ’ s context . Since our 2D + 1 + 1 rows contain many tokens and we use a standard BERT encoder of 512-token inputs , we tile our rows into bundles of three adjacent data rows , plus the header row , which is included in every bundle . Then we compute a token-wise BERT embedding for each bundle separately ; the BERT weights are initialized from a pre-trained checkpoint for English . Specifically , in our experiments where D = 10 , we concatenate all cell values for each row i in the context into a token sequence Ri , which has length L = 128 ( we trim and pad as needed ) . We combine rows in bundles Srb = [ Hr , R3b−1 , R3b , R3b+1 ] , for b ∈ [ −3 , 3 ] ; here Hr is the header row . We set the BERT segment IDs to 0 for the header tokens , and 1 for data tokens in each bundle . There are 2D + 1 = 21 rows of context , so each of the 21 data rows is covered exactly once by the seven bundles . The header row is assigned a different BERT representation in each bundle . To obtain a single representation of the header row , we average per token across the embeddings from all of the bundles . Column-based BERT encoder . As shown in Figure 1b , some formulas manipulate cells in the same column , in which case a column-based representation may be more desirable . Therefore , we also compute a column-based contextual embedding for all context tokens . We perform similar tiling as for the row-based BERT encoding , yielding column bundles Scb for b ∈ [ −3 , 3 ] . Unlike with row-wise tiling , where we include the header row Hr with every bundle , for column-wise tiling we use the column of the target cell , Hc = C0 , as the “ header column ” in every bundle . After obtaining all token embeddings from this tiled computation by the BERT encoder , we discard token embeddings of C0 in its role as header column , and only use its regular token embeddings from bundle Sc0 . Row-wise and column-wise convolution layers . Although the output vectors of BERT encoders already contain important contextual information , such as headers , nearby rows and columns , they still do not fully embed the entire input table as the context . To encode the context from more distant rows and columns , we add a row-wise convolution layer and a column-wise convolution layer on top of each BERT encoder . Specifically , the row-wise convolution layer has a kernel size of 1× L , and the column-wise convolution layer has a kernel size of ( 2D + 2 ) × 1 for row-based BERT , and ( 2D + 1 ) × 1 for column-based BERT . In this way , the convolution layer aggregates across BERT embeddings from different bundles , allowing the model to take longer range dependencies into account . For each input token , let eb be its BERT output vector , cr be the output of the row-wise convolution layer , and cc be the output of the column-wise convolution layer . The final embedding of each input token is the concatenation of the BERT output and the output of convolution layers , i.e. , e = [ cr + cc ; eb ] .

Summary: This paper presents an interesting formulation for spreadsheet formula synthesis. Instead of taking the input output pairs as input, as is done in the programming by example (PBE) approaches, the proposed approach takes the semi-structured tabular context as input for predicting a formula for the target cell. A neural network architecture is presented which uses a BERT-based encoder to leverage the natural language meta-data.