Binding Language Models in Symbolic Languages
Authors: Zhoujun Cheng
Tianbao Xie
Peng Shi
Chengzu Li
Rahul Nadkarni
Yushi Hu
Caiming Xiong
Dragomir Radev
Mari Ostendorf
Luke Zettlemoyer
Noah A. Smith
Tao Yu
Venue: ICLR 2023 notable top 25% (2023)
Keywords: semantic parsing
large language model
neural symbolic
code generation

Abstract: Though end-to-end neural approaches have recently been dominating NLP tasks in both performance and ease-of-use, they lack interpretability and robustness. We propose Binder, a training-free neural-symbolic framework that maps the task input to a program, which (1) allows binding a unified API of language model (LM) functionalities to a programming language (e.g., SQL, Python) to extend its grammar coverage and thus tackle more diverse questions, (2) adopts an LM as both the program parser and the underlying model called by the API during execution, and (3) requires only a few in-context exemplar annotations. Specifically, we employ GPT-3 Codex as the LM. In the parsing stage, with only a few in-context exemplars, Codex is able to identify the part of the task input that cannot be answerable by the original programming language, correctly generate API calls to prompt Codex to solve the unanswerable part, and identify where to place the API calls while being compatible with the original grammar. In the execution stage, Codex can perform versatile functionalities (e.g., commonsense QA, information extraction) given proper prompts in the API calls. Binder achieves state-of-the-art results on WikiTableQuestions and TabFact datasets, with explicit output programs that benefit human debugging. Note that previous best systems are all finetuned on tens of thousands of task-specific samples, while Binder only uses dozens of annotations as in-context exemplars without any training. Our code is available at anonymized.

Published as a conference paper at ICLR 2023
BINDING LANGUAGE MODELS IN SYMBOLIC LANGUAGES
Zhoujun Cheng" Tianbao Xie Peng A Chengzu Li
Rahul Nadkarni*
Yushi Hu Caiming Xiong Dragomir Radev* Mari Ostendor!*
Luke Zettlemoyer** Noah A. Smith o
Tao Y
The University of Hong Kong Shanghai Jiao Tong University
University of Washington
Allen Institute for AI University of Waterloo Salesforce Research
Yale University
Meea Al
ABSTRACT
Though end-to-end neural approaches have recently been dominating NLP tasks in both performance and ease-of-use, they lack interpretability and robustness. We propose BINDER, a training free neural-symbolic framework that maps the task input to a program, which (1) allows binding unified API of language model (LM) functionalities to a programming language (e.g., SQL, Python) to extend its grammar coverage and thus tackle more diverse questions, (2) adopts LM as both the program parser and the underlying model called by the API during execution, and (3) requires only a few in-context exemplar annotations. Specifically, we employ GPT-3 Codex as the L.M In the parsing stage, with only a few in- context exemplars, Codex able to identify the part of the task input that cannot be answerable by the original programming language, correctly generate API calls to prompt Codex to solve the unanswerable part, and identify where to place the API calls while being compatible with the original grammar. In the execution stage, Codex can perform versatile functionalities (e.g., commonsense QA, information extraction) given proper prompts in the API calls. BINDER achieves state-of-the-art results on BIKKTTBLLQUESTIITT programs that benefit human debugging. Note that previous best systems are all finctuned on tens of thousands of task-specific samples, while BINDER only uses dozens of annotations in-context exemplars without any training. Our code is available at htttt:://///
and TABFACT datasets, with explicit output
com wwwmm/huulp///mm nider
INTRODUCTION
Performance natural language processing tasks is dominated by neural end-to-end systems that directly map inputs to outputs (Devlin et al. 2019 Liu et al., 2019 Lewis et al. [2020] Raffel et al.) 2020 i.a.) These end-to-end approaches and robustness. This stands in contrast to symbolic approaches that produce explicit intermediate representations such as logical forms, reasoning paths, or program code, which might then be executed to derive a final output (Zetlemoyyer & Collins, 2003; Gulwani et al., 2017 Chen et al. 2019b i.a.). The intermediate form produced by these the resulting execution makes them more robust input changes. However, their semantic coverage is limited by the affordances of the grammar of the selected symbolic language (e.g., not being able to handle "North America?" in Fig. D leading to failures real-world diverse questions, and the intermediate form annotations require expert knowledge and researcher labour.
flexible and easy-to-use while lacking interpretability
A few works (Andreas al. 2016 Gupta et al. 2019: Khot al. 2021: Zhu et al. 2022 i.a.) have been proposed to combine neural modules and symbolic languages (neural-symbolic) to leverage advantages of both approaches. However, they require the elaborate human design of the symbolic language and the calibration of corresponding neural modules to tackle problems in a specific domain with large training data. More specifically, most of these works propose task-specific symbolic language and corresponding modules that cover only limited semantic phenomena in a specific task and domain. Therefore, new languages and neural modules have to be introduced when adapting them
Equal contribution. Authors in alphabetical order. Work mainly done at the University of Hong Kong. Moor resources https: //mm code- bi binder..... www.dr..iituu.ii..

---

Published as a conference paper at ICLR 2023
finder programe Plythh birding calle
Gaadee-rython
Which & the best seling shirm Americs with chemicals?
GFT-3
of waw. "Sadee" ascondtsspaalle))
Progert
Polo
Sinder-sq
Detaai
ostton
aIEar of Ni oheen callity Sarret) "yeer
Napundl 12
Proze
AMD #i North Ameriaa wwt.eaaaattttii.... Nustciin of "yess"
DROER: BY DESC LINIT
polyatter
Mexico
4:00
$369
SElAr DASE LIMIT
600
GPT-3
U.S.
Codox
yes
horth
Tabay
Inowleiite
Inowledgeldd
An overview of the BINDER pipeline of two stages: parsing and execution. (1) In the parsing stage, the language model (LM) maps the input to BINDER program given the question and (optional) knowledge sources. The expressions with blue background in the program are API calls to acquire external results. (2) In the execution stage, an LM serves to realize the API calls given the prompt and the return values feed back into the original programming language. A deterministic program interpreter executes the program without API calls to derive the final answer.
Figure 1:
to new tasks and domains. Their coverage still restricted by the customized symbolic language and neural modules. Moreover, they call for various and large training data to ensure all modules are well trained. Therefore, we expect a neural-symbolic system that supports flexible neural module calls that will enable higher coverage for the symbolic language, while only requiring few annotations,
We propose BINDER, training-free neural-symbolic framework that maps task inputs to an executable program in programming language (c.g.. SQL. Python) bound with a unified API to call language models (LMs; Brown et al. 2220 Chen et al. 2021) to perform versatile functionalities, i.e. a BINDER program(e.g.. SQL. Sinder-Python in Fig. D. with only few input-B BINDER program annotations as in-context exemplars. More specifically, BINDER first prompts Codex, a code-pretraie pretrained of GPT-3, to parse question input into a BINDER program, in which Codex has to decide (1) which parts in the input can be converted to the targel programming language (question parts highlighted in grey in Fig. D. (2) the corresponding task API calls (e.g. J( [(North America?", Made_in) to prompt Codex to resolve the other parts, and (3) where to insert the API calls in the BINDER program. Next, BINDER prompts Codex again to generate answers the task API calls (given the generated task prompts), integrates the generated results back to the programming language. Specifically as in Fig. [] the prompt (e.g.. "North America?") and data (e.g., column Made_in) in API calls are fed into Codex, and the output is a new column answering the prompt based on the input data (i.e., yes/no of whether a country in M ade_it in column is from North America). Finally, the program with standard programming languages is executed the derive the final answer.
In summary, BINDER enables flexible functionality integration to the programming language to improve its coverage and requires only few annotations. BINDER program replaces custom neural modules and task-specific languages with a unified prompting API call to Codex and general programming languages, respectively, to handle much more diverse task inputs in open domains without complex language and neural module design. BINDER is built on the advances of in-context learning with language models and does not require any training and large-scale annotations.
We demonstrate the effectiveness of the BINDER framework on WIKITABLEEQESTIOON (WIKITQ: Pasupat & Liang 2015) TABFACT (Chen al. 2019a), two structured knowledge grounding datasets that require complex reasoning on the tables. Using Codex (Chen et al., (02T) as the LM, BINDER achieves of-te-of-the-ure results on WIKITQ and TABFACT. Note that the previous state-of-the-ard methods all require fine-tuning on more than 10K annotated training examples or even massive amounts of task-related pretraining data, while our method requires only a dozen or so annotations without training. In further analysis, we find that BINDER provides the greatest performance gain on the questions that the original language grammar (SQL and Python) cannot support, indicating that BINDER effectively improves programming language coverage. We also demonstrate BINDER can be applied on multi-modal knowledge sources (text, table, images, and combined) with MUL- TIMODALQA dataset (Talmor et al. 2021). Moreover, we show that BINDER, compared with end-to-end approaches, is more interpretable when debugging the model, more scalable to very large inputs, and more robust to noisy inputs.
2

---

Published as a conference paper at ICLR 2023
2 APPROACH
Task Definition Given an NLP task that accepts natural language question/Statement Q and optional context(s) (e.g., passages, tables, images, or combination the above) as inputs, the goal is to output answer A based on the inputs to respond to Q correctly. For example, in passage question answering, Q is a question about the passage(s) D; in table fact verification, Q is a statement about the table(s) D.
2.1 BINDER FRAMEWORK
The BINDER framework to solve NLP tasks is defined as follows: given natural language input Q and optional context(s) D as the input, executable BINDER program Z is generated. Finally, the output answer A is derived by exccuting Z with a BINDER interpreter.
Overview
BINDER Parsing the parsing stage, the input natural language Q is parsed into a BINDER program Z. A BINDER program is expression in a symbolic language that optionally includes API calls where the core symbolic language fails to provide a desired functionalityy. We define the API call in the program as function ʃ(Q ; D) that accepts a question Q to be answered, and the context D to be queried on. Here Q is the unanswerable part of Q with the programming language only and Đ is the relevant contexts in D to answer Q. For example, "North America?" in Fig is Q and its corresponding contexts D to answer Q is the column Made_in Note Q Q or D D is also valid (if Q Q and D D, the program is equivalent solving the problem with LM in end-to-end manner). The output of API call f(Q; D) is the answer to Q and it is represented variable compatible with the symbolic language grammar so that the program can be executed.
BINDER Execution In the execution stage, the program Z is executed by BINDER interpreter to derive the answer A. The BINDER interpreter consists of a standard symbolic language interpreter and the model(s) realizing the API calls. The execution phase includes lexical analysis, syntax analysis, and program evaluation. In lexical and syntax analysis, ʃ(Q D) is added as a new identifier in the grammar, and the program is parsed as an abstract syntax tree (AST) based on this extended grammar. In program evaluation, the API calls are evaluated by calling the underlying neural models. The API call output is saved as a variable compatible with the standard symbolic language grammar, and thus the program derive the output. Specifically, BINDER extends the symbolic language production rules to facilitate parsing the BINDER program's 's AST. The AST evaluated in bottom-up order so that nested API calls are also supported, which allows different degrees of decomposition and free combination of language models for the complex question. We provide more details about the grammar extension and BINDER program AST in Appendix A.4
be finally executed by an off-the-shelf symbolie language interpreter to
2.2 IN-CONTEXT LEARNING FOR BINDER
Much recent work uses large language models for in-context learning (Brown et al. 2020 Chen et al., 2021] Chowdhery et al.) 2022b). Compared with fine-tuning, in-context learning (1) only takes few aunotationsddemonssaaiiisss model parameters, which are both longstanding issues of conventional semantic parsing. We base our experiments for BINDER on Codex, code- pretraindd version of GPT-3, which has been shown to perform proficiently on code generation tasks (Rajkumar et al. 2022 Chen et al. 2022) and even
a prompt, and (2) performs inference without training the
some veekly-relateed tasks like dialogue state tracking (Hu al. 2002)). We use Codex as both the semantic parser and the model to perform API call functionalities.
In the parsing stage, it is challenging to generate BINDER programs because their grammar is different from the original programming language grammar due to the inserted API calls. Thus, we take advantage of the few-shot generalization ability of Codex and find that it can learn the modified grammar effectively with only small number of in-context examples. In the execution stage, Codex as the underlying LM serves to give the output to the API calls by concatenating the API call input, Q and D as the language model prompt (the prompt style is described in Section 3.1). The Codex output result(s) stored as variables in the standard programming language so that programming
3

---

Published as a conference paper at ICLR 2023
language interpreter execute on the combination of these variables and the rest part of the program.
Specifically, apply in-context learning for BINDER in the following manner: the inputs are is in-context exemplars of {Q1,. D; Z,) and the inference example (Q.D). D). The & examples should balance the trade-off between the diversity of question types and the model's maximum input capacity, which can either be manually selected (fixed) or automatically retrieved (dynamic) according to the inference example. The model outputs are n candidate BINDER programs 2 = (Zi,..., Zn that aim to solve the inference question Q. Next, the programs 2 are executed by the BINDER interpreter producing n answers A {A1, .... An J. Finally, the output answer A is derived via majority voting strategy over the set of produced answers A (the voting method is similar to the one used by MBR-EXEC (Shi et al., 2022), which we elaborate on in Appendix (..3) The values of and n are hyperparameters in the process above.
2.3 BINDER IMPLEMENTATION
In this section, describe our implementation of BINDER with SQL and Python over structured knowledge as a demonstration. As BINDER is designed to be extensible to various programming languages and API call functionalities, we introduce pipeline for users to quickly adapt BINDER to a new domain in Appendix D
We implement two APIs - soo 1)) and [ 1). where ò = (c1...., 966 is a (sub- (sub-)table of a set of table columns, and c = {ri,..., 3 is a column filled with cell values. Based on Q, feet maps D into a column, and f maps D into a value, Since both return types, i.e., column and value, are compatible with the grammars of SQL and Python (with the Pandas package), feot and s APIs are inserted replace the columns and values form valid BINDER program.
Take the question "which is the best-selling shirt made in North America and with no chemicals?" in [ as an example. The country names in the Made in column do not provide enough information on their own to indicate what their continents are, and thus pure SQL cannot solve it (with this table's data). However, it is easy for a (large) language model to answer whether a country from North America, which is represented by the f "North america?":Made.. j_n)) expression in the position of column name standard SQL. Similarly, the expression foot "No chemicals?" (,hhitt) calls a (large) language model to identify whether the shirts consist of no chemicals (i.e., pure cotton in this case) based on the textual details in the Shirt column.
When (sub-)qu quub))question is too complex or infeasible to be solved by creating intermediate new column with feol, we turn to feat to directly derive the answer. For example, given the question "which shirt is the most suitable for a formal event?" on the table in Fig D new column of "formality value" followed by SQL "ORDER BY" clause. Thus, the expression will be Foail "The most formal?":Shirr"" that outputs value the answer. soat looks more like end-to-end QA, with two important differences: (1) it can be integrated into more compositional SQL queries using its result as a value, (2) it inputs the sub-table instead of the whole table which can mitigate the challenge of input capacity.
it is hard to map Shirt to
"The
EXPERIMENTS
3.1 EXPERIMENT SETUP
Datasets We evaluate our method on three knowledge grounding datasets which were all previously dominated by end-to-end methods: WIKITQ (Pasupat & Liang 2015) and TABFACT (Chen et al.) 2019a). WIKITQ requires complex table reasoning skills to answer the questions. Furthermore, according to SQUALL (Sh et al. 2020b), about 20% of WIKITQ questions are not answerable by pure SQL, either because of the need for extra knowledge or the limited coverage of the SQL grammar, both of which are issues that BINDER is designed to address. TABFACT is binary fact verification benchmark over small tables, on which end-to-end methods have a large advantage but offer no interpretability.
4

---

Published as a conference paper at ICLR 2023
Method
Dev.
Test
Method
Test
Finetuned
Finetuned
T3-3B (Xie et al.
2022
51.9
50.6
SASP (Ou & Liu 2022
77.0
Tapex (Luu et al. 2021)
60.4
59.1
BART-
Large LLewss et al.
2020
82.5
TaCube (hou al. 2022)
61.1
61.3
T5-3B
Xie al. 2022
85.4
Omni Tab Chang et al.
2022
63.3
Tapex
Iiu et al.
212T)
85,9
Without Finetuming
Without Finetuning
Codex end-to-end QA
72.6
Codex end-to-end QA
50.5
48.7
Codex SQL"
80.7
Codex SQL
60.2
61.1
Codex BINDER (Ours) with few-shot retriever
85.1
Codex BINDER (Ours)
65.0
64.6
86.0
Table 1: WIKITQ execution accuracy on de- velopment and test sets. t denotes a symbolic method that outputs intermediate languages.
Table 2: TABFACT accuracy on the official small test set. t denotes a symbolic method that outputs intermediate languages.
The evaluation metrics are execution accuracy (EA) for WIKITQ and TABFACT following common practice for these datasets. Program executions are likely be semantically correct but fail to match the gold answer exactly - for example, SQL outputs 1/0 for yes/no questions. Though this is considered correct according to human evaluation, it is regarded as incorrect by the exact match evaluator. Thus, we add a pre-matching check for these semantically correct cases in WIKITQ the official evaluator. For fair comparison, methods and re-evaluate them using the same evaluator. We provide more details on the evaluator in Appendix A.5 and list the results with the official evaluator for all baselines and our method in
Evaluation
compute the outputs of all baseline
Baselines We compare our method to series of strong published methods on these datasets, including Tapex (Liu al. 202T). Omni'abb (Tiang et al. 2022). TaCube (Zhou et al.) 2022), T5- 3B (Xie et al. 2022 Raffel et al. 2020), and SASP (Ou & Liu 2022). These baselines are fine-tuned on the full-size training set, and many of them (Liu et al. 2021 Zhou et al., (022) are even further
pretrained on a domain-relevant corpus of extra data which is specific to the target domain and task, while BINDER is training- and only requires a few in-context exemplar annotations.
To further demonstrate the effectiveness of BINDER, we also evaluate Codex with additional inference modes including: (1) end-to-end QA, i.c., directly outputting the answer based on the input question and table; and (2) semantic parsing with the standard SQL language. Due to page limits, refer readers interested in these baselines to their respective papers for details (Liu et al. 2021 Jiang et al.)
2022. Zhou et al.) 2022 Xie al. 2022 Raffel et al. 2020 Ou & Liu 2022). 2) and present complete list and evaluation results of more previous systems on these datasets in Appendix C.I
Implementation Details We use the OpenAl Codex (code-davinci-0002) AP model in our exper- iments as both the parser to generate programs and as the underlying model for API calls during the execution of each program. We annotate 14 in-context exemplars with BINDER programs for each dataset, which are selected considering the diversity of question types in demonstrations and the maximum token limit for Codex (8, 000 tokens). The prompt format mainly follows (Rajkumar et al., 2022), which inputs the table schema and the first three table rows. The detailed prompt templates we use for each dataset are listed Appendix A. On TABFACT, we further annotate a pool of 200 examples with BINDER programs from the training set using vote-k selective annotation (Su al., 2022), and use them by retrieving relevant few-shot examples for each inference example maximum inner-produce similarity of the SentenceBert (Clark et al. 2019) embeddings of the questions. The Codex hyperparameters for parsing and execution are provided in Appendix A.. Empirically, it takes about 6 hours average evaluate 1, 000 examples (i.e., generate and execute 20 programs per example) given that Codex allows 200 queries per minute. We evaluate on the official small test set (2, 000 samples) of TABFACT considering the time cost.
3.2 MAIN RESULTS
WIKITQ All results on the WIKITQ dataset are shown in Table Our Codex BINDER outperforms listed strong baseline systems by a large margin, surpassing the previous state-of-the the-trrrrt by absolute
https::/beta. ope.aaiapppe......
5

---

Published as a conference paper at ICLR 2023
Method
Program-unsolvable
Program-solvable
Overall
T5-3B [Xic ct al.
2022
37.6
56.0
51.9
Tapex (Lu et al.
2021
33.6
68.0
60.4
TaCube (Zhou
2022)
34.9
68.5
61.1
Codex end-lo-end QA
40.3
53.4
50.5
w/o table inpuls
14.2
11.9
12.4
Codex SQL
31.2
68.4
60.2
Codex BINDER (Ours)
41.3
71.8
65.0
Table 3: Decomposition of execution accuracy on WIKITQ development set. The questions annotated with SQL by SQUALL (Shi et al., (020b) dataset are denoted program--olvable, and the rest are program-unsolvable Omni UJang et 2022) didn't 't provide result on the development set.
1.3%. Note that all bascline methods require access to the full training dataset and further aie-tuning, while method can achieve state-of-the-an performance with only 14 example annotations. Codex BINDER improves over semantic parsing with standard SQL. by 3.5% on the test set, indicating BINDER indeed mitigates the coverage limitations of the original language. Furthermore, both Codex BINDER and Codex SQL deliver dramatic advantages (15.9% and 12.4%) over end-to-end QA, showing that semantic parsing with Codex is better default choice when comes to structured knowledge grounding and code-related tasks. A fine- the-grained analysis on why BINDER outperforms end-to-end QA and pure SQL provided in Section 4.1
TABFACT Table 2 presents the results on TABFACT's official small test set. Codex BINDER substantially surpasses the previous best symbolic method SASP by 8.1%. Note symbolic methods usually fall behind end-to-end manners in fact verification (binary classification) since the answer space is small. When retrieving a few relevant questions from the annotated pool 200 exam- ples, our method achieves new state-of-te soate-of-thh-arr results, outperforming the previous best fine-tuned method. However, the improvement provided by retrieving a few similar examples is comparatively small (0.9%). Effective sample selection methods for in-context learning remain open challenge. Within the Codex few-shot setting (no few-shot retriever). BINDER shows large advantage over standard SQL (4.4%) and end-to-end QA (12.5%), indicating the necessity of BINDER framework.
Besides the performance gain, BINDER has the additional advantages of (1) interpretability that benefits human debugging and (2) robustmess that makes it stable to large or noisy inputs. We elaborate on these in Sections 4.2 andd.33
4 ANALYSIS
4.1 ABLATION STUDY
Binding neural module API calls into a programming language can help solve queries that are unsolvable in that language alone. Therefore, we are particularly interested in the performance of BINDER on the unsolvable questions. According to SQUALL (Shi et al. 2020b), about 20% and refer
of WIKITQ questions cannot be annotated in SQL; call these grrogram-unsovvate to the annotated ones as program-ssolvable. As presented in Table Codex BINDER significantly outperforms Codex SQL by 10.1% program-unsolveen of our design. Our method even performs better than all end-to-end QA methods on the program- unsolvable part, which are supposed to be more robust this subset of the examples. We note that while SQL performs well on about 31.2% of unsolvable questions, many are spurious programs that derive the correct answer by accident. BINDER largely mitigates this phenomenon. We randomly pick 100 correct predictions in program- -utrraaa-nssllbblee set shared by SQL and BINDER and find that BINDER has a much lower spurious rate than SQL (12% vs. 33%). One thing to note is that BINDER also achieves higher score than SQL on program- moovaabe We find that this is because some tables are manually cleaned (e.g., extracting numerical parts from the text) to run the annotated SQLs by SQUALL, which means there also exist unsolvable questions even in the
questions, aligned with the motivation
solvvbbee subset. We elaborate on our manual statistics about the proportion of WIKITQ that can be solved with SQL in Appendix C.4 We also present to what extent Codex itself can answer the question only using its internal knowledge. Codex can only answer a few WIKITQ questions (12.4%) correctly, indicating not pretrained overfit to the downstream dataset questions.

---

Published as a conference paper at ICLR 2023
Error Type
Description
Proportion %
Syntax error
Incorrect program syntax (invalid grammar)
5%
Semantic error
Incorrect program semantics
64%
Token Structure BINDER usage
Incorrect missing column/valuc/operrii
10%
Incorrect program structure (valid grammar)
22%
Missing BINDER API usage
32%
Incorrect execution
Incorrect execution answer with the correct program
15 %
False negative
Incorrect annotations or misjudge in evaluator
16%
Table 4: Error types of 100 samples from WIKITQ development set of BINDER (SQL).
System
Program
Answer
to end QA
/967
SELECT year FROM WHERE "LA_teaa-" kanaas state' AND
SQL
ninama los_team 10
<ompty>
SELECT year FROM WHERE win sar___tamm"faaa kansas stats! AND ["Mooaass"iwin__een - ""Points?";ics_s:s"
BINDER
10
comply>
Table 5: An example from WIKITQ development set. The query is "When was the first game that kansas state won by double digits?" ?" and the gold answer 1926. The incorrect segment(s) of each output are marked in red. See Appendix E for full context of this example.
4.2 INTERPRETABILITY
An important advantage BINDER provides the improvement in interpretability over the end-to- end approaches, where the explicit program can assist human debugging and error analysis. We sample 100 error cases from WIKITQ dev set (approximately 10% of all incorrect examples) of the Codex BINDER (on SQL) for error analysis. We classify errors into syntax error, semantic error, incorrect execution, and false negative, as listed in Table More details about the classification and example demonstrations are listed in Appendix [ For WIKITQ, the errors mainly lie in the BINDER usage (32%) and structure errors (22%) of semantic errors, and incorrect execution (15%). This indicates that 32% of incorrect examples can be further corrected if BINDER used and 15% can be corrected through better execution, e.g., by leveraging more powerful LMs, using better in-context learning methods, and annotating more exemplars.
We also use example to show the advantages of using BINDER for debugging and interpreting results. As shown in the example in Table the result from the end-to-end system is 1967, which is incorrect but provides no clue to the reason behind this prediction. The result from BINDER is incorrect, since the program uses the incorrect value and operator regarding the win win__eam column from the table (should be "LIKE "Økansas state "*), and it can be potentially fixed in the future by adding similar in-context exemplars or fuzzy match postprocessing. For standard SQL, though we find the direct subtraction of win_Jeam los fos_team is incorrect (contain non-numerical text), it cannot be fixed for its limited grammar coverage. Thus, in this example, BINDER enables finding the the end-to-end approach) while also being expressive enough for users to find a way to fix it (an advantage over a pure symbolic system), making BINDER fit for debugging and enabling interpretability of system that uses it. We leave it future work develop more systematic evaluation methods that consider ease of debugging by estimating human effort to correct system errors.
source of the error (an advantage
4.3 ROBUSTNESS
4.3.1 SCALABILITY
great advantage of BINDER over end-to-end QA is the scalability, Lee, parsing and executing the symbolic language is always practical even when the knowledge source is too large (e.g., company databases, domain knowledge graphs) to fit into the model input capacity or memory, while end-to- end QA fails or degrades since it requires the whole table as input to predict the answer. We expand tables with 100, 200, and 500 rows from a WIKITQ development subset (random 150 samples) by prompting Codex populate the original table with content-consistent rows, and then have two annotators annotate question-ansee answer pairs manually. When the table is too large for Codex's maximum token limit (8, 000 tokens), truncate table rows until it fits. As shown in Figure Codex
7

---

Published as a conference paper at ICLR 2023
60
Codex end-ta-end GA
Codex end GA
Codex Bander
Codox Binder
70
Accurac
Aacuracy
60
50
40
Drgiinal
500
Original
Niney
Figure 2: Execution accuracy on WIKITQ with
Figure 3: Execution accuracy on WIKITQ with noisy content tables.
very large tables (original, 100, 200, 500 rows).
Method
F1
EM
Method
Program-unsolvable
Finetuned
Codex Python
30.7
Implicit-Decome Talmor et al. 2021
55.5
48.8
PReasM-Large Yoran et al 2022
65.5
59.0
Codex BINDER (Ours)
34.4
Withou Fineturing
Codex end-to-end QA Codex BINDER (Ours) with oracle retriever
55.4
48.0
Table 6:
WIKITQ accuracy on program-
57.1
51.0
unsolvable subset using Python as the program- ming language.
64.5
58.1
Table 7: MMQA FI/EM on development set.
end-to-end QA performance drops dramatically table size increases, while BINDER consistently outperforms it with only slight performance decreases. Note that the input of BINDER is only three table rows for all table sizes.
4.3.2 NOISY CONTENT
End-to-end methods are more brittle to noisy inputs, especially when there exists distractors that similar to the question-relevant (gold) contents. We build a noisy WIKITQ development subset (ran- dom 150 samples) with distractors in three steps: (1) the gold table cells and columns are found using a heuristic fuzzy match, (2) replace 15% cells in gold columns with either tex/ that has the smallest edit distance to the gold cell string in the WordNet corpus or number that equals to the gold cell value +1, (3) have one annotator check that question-answe pairs are valid after content disturbance. As shown in Figure 3 BINDER is stable confronting distractors (1.3% 4), while end-to-end QA is more likely to be confused by similar text and numbers (6.7% 4).
4.4 BINDER EXTENSION
BINDER with Python We design BINDER to be easily extensible to various programming languages. Thus, we explore using Python (with the Pandas package) as the BINDER language on WIKITQ. Similar to SQL., BINDER with Python is implemented by incorporating the ʃ ((::-) neural API into it. Since the neural API also calls language models with Python, we just use the original Python interpreter to execute our BINDER with Python. We evaluated on the program-unsolvabe subset of WIKITQ to test whether our method improves Python's 's capability. Though this subset is split based SQL, Python shares some similar weaknesses with SQL, such in cases with external knowledge requirements and unsupported functionality. As shown in Table 6 BINDER with Python effectively improves the Python coverage the difficult subset. The gap between Python and SQL performance on WiKITQ may lie in the properties of each programming language, as suggested by Guo et al. 22000). We conjecture that Codex is more familiar with semantic parsing SQL for tabular data, while Python is not as commonly used in this context.
MultiModal Application We further explore applying BINDER on the multi-modal dataset MUL- TIMODALQA (MMQA) across text, tables, and images. To input images into the LM program generation, images are converted into textual image captions with a vision-text pretrained model OFA (Wang al. 2022 in advance. For table-related questions, the BINDER program is based
8

---

Published as a conference paper at ICLR 2023
on SQL. For non-table questions, a program with fral is generated with the targeted passage or image title(s). We follow Yoran et al. (2022) to retrieve the question-relevant passages and images in preprocessing. As far we know, are the first to demonstrate that multi- QA across text, tables, and images can be handled with interpretable and executable programs. As listed in T 7 under the Codex few-shot setting, BINDER achieves better performance than end-to-end QA and the fine-tuned baseline Implicit-Decomp. showing the feasibility of BINDER on multi-modal knowledge sources. With the oracle retriever that assumes gold passages and images are given for each question, BINDER can achieve comparable performance with the ttate-of-the-an ahe-tht,,,, showing the potential of BINDER approach in the future.
RELATED WORK
Semantic Parsing Semantic parsing (Zelle & Mooney 1996 Zettlemoyer & Collins
2005) has
been a mainstay of symbolic methods that produce an executable program given natural language input, generate intermediate structures that assist wroblem-solving, and improve interpretability over the neural methods which generate solution directly that came later. Structured knowledge grounding tasks mainly adopt semantic parsing since symbolic languages like SQL., SPARQL can be executed
on them (Berant et al. 2013 Liang et al. [017]] Yin & Neubig 2017 Zhong et al., [018 Yu et al.
et al. 202 Scholak et al. ().1). Beyond structured knowledge, Chen et al. (2019) 9b) and Thome et al. (202 design domain-specific languages executable on text. Recently, CCen al. (202 l)
propose a generative pre-trained language model for code generation that require no additional human annotation. Many works propose methods based on this model and achieve great success (Rajkumar et al. 2022, Shi et al. 2022). However, the semantic parsing method is restricted by its grammar coverage, unable to solve problems requiring external knowledge functions.
Neural-Symbolic Methods Some works integrate neural modules with symbolic languages for the advantage of both approaches, i.e. good performance and interpretability. (Andrea al. (2016); Hu et al. (2177); Das et al. (2018) and Gupta et (2019) generate programs that are further softly executed by the corresponding neural modules. Khot et al. (202T) propose text module networks to solve complex tasks by decomposing them into simpler ones solvable by existing QA models and a symbolic calculator. BREAK (Wolfson et al., 2020 proposes a meaningful representation. QDMR, that decomposes the question into multiple steps. However, they require the elaborate design of functions to be used in corresponding task and the calibration of corresponding neural modules which require complicated training steps and large training data (normally tens of thousands) to tackle problems in a specific domain.
Compared with these methods, BINDER is expressive and flexible to handle real-world diverse questions since it is able to make proper API calls to enhance its functionalities. Moreover, BINDER is training-fres free and requires only dozens of annotations based certain symbolic language to perform on domain specific task while maintaining: excellent performance, ability in scaling on input, interpretabilltyyndd robustness over noisy content.
CONCLUSION
We propose BINDER, training-free ffree neural- neural-symbolic framework that maps the task input to program that allows binding a unified LM API for additional functionalities. BINDER aims to combine the strengths of end-to-end approaches (high coverage) and symbolic approaches (high interpretability). Using Codex as the LM, BINDER achieves state-of-the-art performance on WIKITQ and TA TABF with only dozens of in-context demonstrations and no additional training. contrast, the best existing systems are finetuned on thousands of task-specific training samples and may require further domain- specific pertaining. We also perform a series of analyses of BINDER, decomposing performance gains, examining robustness to large noisy inputs, applying it to multi-modal knowledge sources, and extending it to the Python language. We regard BINDER as a new, language model-focused attempt to integrate two wddely-adopted paradigms in NLP: end-to-end and symbolic approaches. With the recent powerful large language models, it has become feasible that BINDER programs are generated and executed correctly in training- free manner. In the future, we believe BINDER can be extended to many more scenarios with the appropriate programming language and functionalities, and we hope can inspire more creative ideas balancing model capability and interpretabilityy
9

---

Published as a conference paper at ICLR 2023
7 REPRODUCIBILITY
BINDER experiments are mainly based on OpenAl Codex (code-davinci-002) AP We provide (1) the input prompt templates we use in Appendix A.I] (2) Codex hyper-parameters for each dataset
adopt in Appendix A.2 (3) more implementation details in Appendix A Besides, we also upload our
source code in the materials making our results easy be reproduced.
https::/bbet..openaa comm
10

---

Published as a conference paper at ICLR 2023
REFERENCES
Rishabh Agarwal, Chen Liang, Dale Schuurmans, and Mohammad Norouzi. Learning to generalize from sparse and underspecified rewards. In International conference on machine learning, pp. 130-140. PMLR, 2019.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 39-48, 2016.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Freebase from question-answer answer pairs. In Proc. of EMNLP, 2013. URL httpsss/acclaatttooloo/o/
D13-1160
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Rad- ford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Proc. wwwprrisscccpppeeee
of NeurlPS, 2020. URL
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Tian-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests. arXiv preprint 22X.:22207.11397 2022.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.0333 2021.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. Tabfact: A large-scale dataset for table-based fact verification. arXiv preprint arXiv:1909.02 144, 2019a.
Xinyun Chen, Chen Liang, Adams Wei Yu, Denny Zhou, Dawn Song, and Quoc V Le. Neural symbolic reader: Scalable integration of distributed and symbolic representations for reading comprehension. In International Conference on Learning Representations, 2019b.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023111 2022a.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023: 2022b.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction finetuned language models. arXiv preprint arXiv:22 1416, 2022.
instruction-iiettueed
Elizabeth Clark, Asli Celikyilmaz, and Noah A. Smith. Sentence mover's similarity: Automatic eval- uation for multi-sentence texts. Proc. ACL, July 2019. URL httthttt//aaaaatttoottt
org/P19- 12647
Abhishek Das, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Neural modular control for embodied question answering. In Conference on Robot Learning, pp. 53-62. PMLR, 2018.
Pradeep Dasigi, Matt Gardner, Shikhar Murty, Luke Zettlemoyer, and Eduard Hovy. Iterative search for weakly supervised semantic parsing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language 'eecholoogiss Volume I (Long and Short Papers), pp. 8669-2680, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transfonmers for language understanding. In Proc. of NAACL, 2019. URL https:://arxinn www.:orrggs.ooooo 04805

---

Published as a conference paper at ICLR 2023
Julian Eisenschlos, Syrine Krichene, and Thomas Mueller. Understanding tables with intermediate pre- pre-training. In Findings of the Association for Computational Linguistics: EMNLP 2020, pP- 281-296, 2020.
Sumit Gulwani, Oleksandr Polozov, Rishabh Singh, et al. Program synthesis. Foundations and Trends@ in Programming Languages, 4(1-2):1-119, 2017.
Jiaqi Guo, Qian Liu, Jian-Guang Lou, Zhenwen Li, Xueqing Liu, Tao Xie, and Ting Liu. Benchmark- ing meaning representations in neural semantic parsing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 1520-1540, 2020.
Jiaqi Guo, Jian-Gune -uang Lou, Ting Liu, and Dongmei Zhang. Weakly supervised semantic pars- ing by learning from mistakes. In Findings of the Association for Computational Linguis-
fies: EMNLP 2021, pp. 2603-2617, Punta Cana, Dominican Republic, November 2021. As- sociation for Computational Linguistics. doi: 10.18653/v 1/202/1 1.200l.inniings--eiii findings-emnl..222 URL www.llanhh.oo..orrg//oooooo
fin...iiniigg. emrlp. 222
Nitish Gupta, Kevin Lin, Dan Roth, Sameer Singh, and Matt Gardner. Neural module networks for reasoning over text. arXiv preprint 1rXiv:192..04971, 2019.
Jonathan Herzig, P. Nowak, Thomas Müller, Francesco Piccinno, and Julian Martin Eisenschlos. Tapas: Weakly supervised table parsing via pre-training. In Proceedings of ACL, 2020.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Training compute-optimal large language models. arXiv preprint 2rXiv:2203 2022.
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 804-813, 2017.
Yushi Hu, Chia-Hsuan Lee, Tianbao Xie. Tao Yu, Noah A Smith, and Mari Ostendorf. In-context learning for few-shot dialogue state tracking. arXiv preprint arXiv:2203.08566 2022.
Zhengbao Jiang, Yi Mao, Pengcheng He, Graham Neubig, and Weizhu Chen. Omnitab: Pretraining with natural and synthetic data for few-shot muble-based question answering. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 932-942,, 2022.
Tushar Khot, Daniel Khashabi, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Text modular networks: Learning to decompose tasks in the language of existing models. Proceedings 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1264-1279, 2021.
gs of the
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916, 2022.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-seqqenee pre-training for natural language generation, translation, and comprehension. In Proc. of ACL, July 2020. URL hittps://arxiv/ wwtp:/ivv.r//abb//////o//
Chen Liang, Jonathan Berant, Quoe Le, Kenneth D. Forbus, and Ni Lao. Neural symbolic machines: Learning semantic parsers on Freebase with weak supervision. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 23-33, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-1000
Chen Liang, Mohammad Norouzi, Jonathan Berant, Quoe V Le, and Ni Lao. Memory augmented policy optimization for program synthesis and semantic parsing. Advances in Neural Information Processing Systems, 31, 2018.
Qian Liu, Bei Chen, Jiaqi Guo, Zeqi Lin, and Jian guang Lou. Tapex: Table pre-training via learning neural sqi executor, 2021.
12

---

Published as a conference paper at ICLR 2023
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 19X7 2019.
Arvind Neelakantan, Quoc V Le, and Ilya Sutskever. Neural programmer: Inducing latent programs with gradient descent. arXiv preprint arXiv:1511.0483 2015.
Suixin Ou and Yongmei Liu. Learning to generate programs for table fact verification via structure- aware semantic parsing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7624-7638, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10.11863/vvln 1tw.1//.222.....o...222222 URL 525
Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1470-1480, Beijing, China, July 2015. Association for Computational Linguistics. doi: 1..1155/1//15-- 1142. URL
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. JLMR, 21, 2020. URL htttt:///// jmlr..
jttet:::/mmlr..riti www.ll.org/papess//222 h21//2211200220 074. html
Nitarshan Rajkumar, Raymond Li, and Dzmitry Bahdanau. Evaluating the text-to-sql capabilities of large language models. arXiv preprint arXiv:2204. 200409004 2022.
Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. PICARD: Parsing incrementally for con- strained auto-regressive decoding from language models. In Proceedings the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 9895-9901, Online and Punta Cana, Do- minican Republic, November 2021. Association for Computational Linguisties. doi: 10. 18653/v/v emm 1p i 779
2021. m21..mmppp-min.79.. URL https: //acl antho wachoologge oogy org/ 2021
Peter Shaw, Ming-Wer Wei Chang, Panupong Pasupat, and Kristina Toutanova. Compositional gen- eralization and natural language variation: Can a semantic parsing approach handle both? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, August 2021. Association for Computational Linguistics.
Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida I Wang. Natural language to code translation with execution. arXiv preprint arXiv:2204. 11454 2022.
Qi Shi. Yu Zhang, Qingyu Yin, and Ting Liu. Learn combine linguistic and symbolic informa- tion for table-based fact verification. In Proceedings of the 28th International Conference on Computational Linguistics, pp- 5335-5346, 2020a.
Tianze Shi, Chen Zhao, Jordan Boyd-Graber, Hal Daumé III, and Lillian Lee. On the potential of lexico-logical alignments for semantic parsing to sqi queries. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1849-1864, 2020b.
Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, et al. Selective annotation makes language models better shw- learners. arXiv preprint 27X.1:2209.01975, 2022.
Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant. Multimodalqa: Complex question answering over text, tables and images. arXiv preprint arXiv:2104.0603: 2021.
James Thome, Majid Yazdani, Marzieh Saeidi, Fabrizio Silvestri, Sehastian Riedel, and Alon Halevy. 2021.
Database reasoning over text. arXiv preprint
Bailin Wang, Ivan Titov, and Mirella Lapata. Learning semantic parsers from denotations with latent structured alignments and abstract programs. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNL.--ICCNIP)), pp. 3774-3785, 2019.
13

---

Published as a conference paper at ICLR 2023
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through a simple tegueenceto-seqqencee learning framework. arXiv preprint arXiv:2202.0300 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022.
Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldherg, Daniel Deutch, and Jonathan Berant. Break down: A question understanding benchmark. Transactions of the Association for Computational Linguistics, 2020.
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien- Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang. Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. arXiv preprint arXiv: 00966.05..6... 2022.
Xiaoyu Yang. Feng Nie, Yufei Feng, Quan Liu, Zhigang Chen, and Xiaodan Zhu. Program enhanced fact verification with verbalization and graph attention network. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1810-7825, 2020.
Pengcheng Yin and Graham Neubig. A syntactic neural model for general-purpon code generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 440-450, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi:
/ataaaaaa///llaaooooooyy org/p17-1041
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. TaBERT: Pretraining for joint understanding of textual and tabular data, In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, July 2020.
Ori Yoran, Alon Talmor, and Jonathan Berant. Turning tables: Generating examples from semi- structured tables for endowing language models with reasoning skills. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 6016-6031, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 18653/ v1/2022.acl-long. 416. URL https: http://aclaattt http///caaaatttoooooooooaom org.oogg/////oo /202
416
Tao Yu, Rui Zhang. Kai Yang. Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 3911-39211 2018
Tao Yu, Chien-Shem Wu, Xi Victoria Lin, Yi Chern Tan, Xinyi Yang. Dragomir Radev, Caiming Xiong, et al. Grappa: Grammar- Conference on Learning Representations, 2020.
pre-training for table semantic parsing. In International
John M. Zelle and Raymond J. Mooney. Learning to parse database queries using inductive logic programming. In AAAI 1996, pp. 1050-1055, 1996.
Luke S. Zettlemoyer and Michael Collins. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. UAI, 2005.
Yuchen Zhang, Panupong Pasupat, and Percy Liang. Macro grammars and holistic triggering for efficient semantic parsing. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1214-1223, 2017.
Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. 2018
Wanjun Zhong, Duyu Tang, Zhangyin Feng, Nan Duan, Ming Zhou, Ming Gong, Linjun Shou, Daxin Jiang, Jiahai Wang, and Jian Yin. Logicalfactchheker: Leveraging logical operations for fact checking with graph module network. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6053-6065, 2020.
14

---

Published as a conference paper at ICLR 2023
Fan Zhou, Mengkang Hu, Haoyu Dong, Zhoujun Cheng, Shi Han, and Dongmei Zhang. Tacube: cre-computing data cubes for answering rumerical-reasoning questions over tabular data. arXiv preprint 27X11:2205.12662, 2022.
Zhaocheng Zhu, Mikhail Galkin, Zuobai Zhang, and Jian Tang. Neural-symbolic models for logical queries on knowledge graphs. arXiv preprint arXiv:2205. 10128, 2022.
15

---

Published as a conference paper at ICLR 2023
Appendices
A
MORE IMPLEMENTATION DETAILS
We present details in implementation of Codex BINDER for interested readers.
A.1 CODEX INPUT PROMPTS
We list prompts we use in main experiments for BINDER for each dataset. As shown, the prompt starts with an instruction describing the task, followed by few examples for in-context learning. We follow Rajkumar et al. 2022) table prompt format: (1) "CREATE TABLE" schema; (2) the first three rows of the table; (3) the question Q and parsed logic form BINDER program. Besides, we also add row_ke row_id and lower case all table contents, which we find will improve the Codex parsing performance. For the sample to be inferenced, we empty the BINDER program to Codex generate it, and input the full table to provide more information. If the full table is too large, will shrink number of in-context shots until the table fits within the Codex max token limits.
## WikiTQ
Cenerate SQL given the question and table to answer the question correctly.
CREATE TABLE Electoral district of Lachlan row id int, member text, party text, term text)
/*
example rows: SELECT * FROM w LIMIT 3;
row id member party term
john ryan none 1859-1864
1 james martin none 1864-1869
2 james watson none 1869-1880
*/
Q: of the members of the third incarnation of the lachlan, who served the longest?
Binder: SELECT member FROM ORDER BY
{"How long does it last?"; term) DESC LIMIT 1
## TabFact
Generate SQL given the statement and table to verify the statement correctly.
CREATE TABLE british records in athletics
row_id int, event text, data text, athlete text, date text, place text)
/*
example rows: SELECT FROM LIMIT 3; row id event data athlete date place 5 km t19:29 and drake 1990-05-27 00:00:00 søfteland norway
16

---

Published as a conference paper at ICLR 2023
1 5 miles 32:38 ian mccombie 1985-03-23 00:00:00 york united kingdom
10 km 40:17 chris maddocks 1989-04-33 00:00:00 burrator united kingdom */
Q: there be 8 different event that take place within the united kingdom
Binder: SELECT (SELECT COUNT (place) FROM WHERE
E("Is it united kingdom?"; place) 'yes')
## MMQA
Generate SQL given the question, table, passages, image captions to answer the question correctly.
CREATE TABLE 2018 Warrington Wolves season
row_id int,
player text, signed from text, contract length text,
announced text)
/*
example rows:
SELECT * FROM w LIMIT 3;
row_id player signed from contract length announced
sitaleki akauola penrith panthers 2 2017-08-01 bryson goodwin south sydney rabbitohs 2017-10-01 tyrone roberts gold coast titans 3 2017-10-01 */
1
CREATE TABLE Images
cow_id int,
gold coast titans text)
/*
All rows of the table:
SELECT * FROM w; row_id gold coast titans
a logo for the golden knights is painted on the beach.
*/
Q: What player was transferred from the team that has crossed
swords its logo to the Warrington Wolves in the 2018 season? Binder: SELECT player FROM w WHERE
{(Has crossed swords on its logo?"; 'signed from') 'ys''
In end-to-end QA setting, the prompt is almost the same, except the full table contents used in all the in-context examples to since in most cases QA must see the complete table to answer the question correctly. We empirically find that full table input Codex is necessity for end-to-end QA, but only small bonus for semantic parsing.
In the ablation study experiments, the prompt is also almost the same, except that under the very large table scalability setting, we truncate number of table rows instead of in-context shots to fit in Codex max token limits, order simulate realistic situation where only parts of the table can be input into the model.
A.2 CODEX HYPER-PARAMER PARAMETERS
In both two phases of our Codex BINDER pipeline, parsing and execution, the Codex api is called.
We set the Codex in-context learning hyper-parameters shown in Table
17

---

Published as a conference paper at ICLR 2023
Parsing TABFACT
Execution TABFACT
Hlyper-paramele
WIKITQ
MMQA
WIKITQ
MMQA
temperature
0.4
0.6
0.4
0.0
0.0
0.0
1.U
1.0
1.0
1.0
1.0
1.0
Wuak_oulptt_okeee tookens
512
512
512
1024
1024
1024
sampling_n
20
30
20
1
stop_tokens
Anin
In'm
Inin
in'n
kn/n
Inin
num shots
14
18
8
8
Table 8: Codex hyper-parameters we set in main experiments. Codex is used in two phases: Parsing and Execution.
Parsing In parsing phase, i.e. parsing questions into programming languages, we annotate dozens or examples from training set modified examples from development set as in-context demonstrations for each dataset. temperature suitable trade-off between fidelity and diversity of generated programs in most cases. sampling number of generations (programs) per sample. We heuristically set sampling as 20 in WIKITQ and MMQA to balance the time cost in execution and the potential performance gain over majority voting. For TABFACT, we increase temperature to 0.6 and 50, as we empirically find that for binary classification tasks (or tasks with small output space), more generations with diversity can better determine the final answer.
control randomness of generations. We find 0.4 is
Execution As mentioned in Section 2 the executor will call model(s) with various functionalities. In experiments, use Codex as the underlining model for all the neural modules since Codex itself is a very powerful large language model which can realize most functions on texts and codes. For images, they preprocessed into text via image captioning by OFA (Wang al., 2022). Note that on MMQA, replacing Codex (with image captions) with a specific VQA model is likely to improve the performance on images because image captioning is question-agnostee while VQA model can predict the answer based on the question.
We let Codex output only one answer with temperature 0.0 per input instead of multiple answers followed by majority vote considering the time cost. An interesting thing is that even when temperature is 0.0, the Codex outputs may different in two inferences. OpenAl team said the randomness inherent to GPU computations, and generally the outputs will be the consistent in most cases, which aligned with our observations.
We also find it better to give Codex some in-context demonstrations for the neural module functionality. 50 samples are annotated as a small retrieve pool (shared across datasets) for mapping a column to new one according to the question, i.e., f API in BINDER grammar. A simple BLEU score retriever will retrieve 8 similar items from the pool as in-context demonstrations for each coming sample. Below several examples demonstrating the formats of neural module functionality prompt. The output (sub-)table contains new mapped column named after the query, which will be further merged into the original table. Note it is important to formulate the output also (sub-)table, otherwise if Codex outputs a list of answers, it will be confused which row it is generating answer for and perform very poorly.
## 1
Give a database as shown below: Table: Highest mountain peaks of California
/*
row id prominence
0
10080 ft; 3072 m
1677 ft; 511 m
2
7196 ft; 2193 m
2894 ft; 882 m
4
9832 ft; 2997
2563 ft; 781 m
*/
Q: Answer question "What is the value of in " row by row.
QA mapê output:
18

---

Published as a conference paper at ICLR 2023
x
row id
prominence
What is the value of in feet?
10080 ft; 3072
10080
1
1677 ft; 511 m
1677
2
7196 ft; 2193
7196
3
2894 ft; 882 m
2894
9832 ft; 2997
9832
2563 ft; 781 m
2563
*/
## 2
Give a database as shown below:
Table: 2010-11 UAB Blazers men's basketball team
/*
row row_id hometown
chicago, i1, u.s.
oklahoma city, ok, u.s. montgomery, al, u.s.
2
greenville, ma, u.s. birmingham, al, u.s.
4
*/
Q:
Answer question "Is it from alabama?" row by row.
QA
mapê output:
x*
row id hometown
Is it from alabama?
chicago, i1, oklahoma city, ok, u.s. no montgomery, al, U.S. yes greenville, ms, U.8. no birmingham, al, u.s. yes
no
1
3
4.
*/
## 3
Give a database shown below:
Table: 1963 International Gold Cup
/*
row id driver
jim clark
1
richie ginther
graham hill
3
jack brabham
tony maggs
*/
0:
Answer question "What is his/her country?" row by row.
QA
mapê output:
row
driver
What is his/her country? scotland united states england australia south africe
jim clark
1
richie ginther
graham hill
jack brabham
tony maggs
x/
19

---

Published as a conference paper at ICLR 2023
A.3 MAJORITY VOTE STRATEGY
Majority vote is widely used to ensemble multiple candidate answers. A simple implementation of majority vote is to select the answer that appears most often in candidates the final answer. For example, given a candidate answer pool C' with n answers, the output answer Gout is derived by:
Gout arg (uax(count(a))
E set(C)
(1)
a
where count(-) function counts the number of occurrences of the input answer in C, and set(:) function de-duplicates the input list. In our case, n programs are generated per sample, and their execution results compose the candidate answer pool.
In experiments, we adopt two variants of majority vote strategies. The first one we call answer-biased. In TABFACT, we conjecture that is difficult validate statement of multiple sub-statements as entailed (i.e., answer 1) with SQL program, since minor error in program will cause the execution to output refuted (i.e., answer 0). In other words, we value when an answer 1 occurs because indicates the statement has a high probability of being entailed. Thus, we re-weight answers by assigning four votes for an answer 1, and one vote for an answer The second program- pvogram-biased. We assign larger weights to BINDER language when it occurs in the generated programs because BINDER consists of small portion few-shot prompt, and thus more generations will be standard programming language. Specifically, we we-weight BINDER program with ten votes, and the others with one vote in WiKITQ and MMQA.
A.4 BINDER GRAMMAR ADAPTION TO SQL
Quseet 1o0
60%
Wiech boll ingoolic made ArAmorii??
501
GELOCT
FOOR
14000
1O07
Tdentifier Table
colure
RREJECT shics
API Coluan
Coluzn Auu(Q) P)
VESRS 1150111 Colorty Satri) 1pea"
Apgragate
fuc(0)
CSEED
Salee LIMES
Frart Nerroaa
Figure 4: An illustration of BINDER program AST and extension to grammar. The blue types in the production rule are extended by BINDER.
BINDER extends the production rules of the programming language ensure the return values of API calls can be compatible with its original grammar. Take SQL as an example, in Figure 4 two APIs feel and feal are added as identifiers in the production rules to fit in as column and value in SQL. The AST of the shown BINDER program is quite similar to standard SQL's AST, except that API calls are placed in the column position.
A.5 EVALUATOR
Program executions are naturally more difficult to match gold answers exactly when the answer is not a span in the input knowledge source, while end-to-end models may match this pattern by finetuning infer phrases from the question. However, we find that in WIKITQ, some error cases of program execution outputs are correct according to humans. These cases mainly fall into two types. The first A or B choice question, e.g... "Are there at least 13 different components on the chart?" in Table The SQL program SELECT COUNT (component) > 13, returns 1 to indicate "yes" and 0 for "no". Generally in or B problem, we match with and with B. The second is number with unit question, e.g.. "what is the difference in years between constituency and 2" in Table where the gold answer contains unit years in it, while it is almost impossible for programs to derive years, and 4 also reasonable correct answer in this case. Therefore, we add pre-matching logics upon WIKITQ official evaluator for these two problems. Besides, we also normalize the dates with Recognizers package in Python. Table @ some examples to show the differences of three WIKITQ evaluators. For fair comparison, all previous baselines are re-evaluated with the aomantic-mated evaluator in experiments.
20

---

Published as a conference paper at ICLR 2023
Example
EM(TaPEx string match)
EM(WIKITQ official)
EM(Semantic march)
(Nornalization)
Question: What same problem
Bernard Collomb had as Innes Ireland?
Gold Answer: oil pressure
Prod Answer: oil
(56 laps)
(Float Precision)
Question: What is the difference between the
x
qualifying time 1967 1965?
Gold Answer: 7.45
Answer: 7.449999999999999
Choice)
Question: Are there al least 13
X
x
different components on the chart?
Gold Annwer: Yes
Pred Answer:
(Number with Units)
Question: What the difference in years
between constiuency and 2?
Gold Anrwer: years
Pred Annwer: 4
Table 9: Examples to illustrate differences among three exact match evaluators on WIKITQ. In experiments, we evaluate all baselines and our method with cemantic-match evaluator.
End-to-End
Chain-af-Thauge
Semantic Parsing/Code Generation
Binder (ours)
Input
Input
GPT-3
GFT-a
GPT-3
Reasening path
sg:
stnden-age
Pota Lucioland Timber Bay wre from no chemniicaas Luchyiena the Lustoland.
EisesCheirth mene ii yalle "yea"
ctbci Sable DESK LDdT
OREER BY Sabes DCSc LINET 2
GF1-3
Ludkyland
Luckyland
-Erppty
Uninserpettable not robust
Uniellable araaver antalllmant
Limited PL grammer coverage
Interpretable, robust, wide coverage
Figure 5: Comparison of the BINDER method( with other large language model usage paradigms: End-to-End, Chain-of-Thought, and Semantic Parsing/Code Generation.
B COMPARISON WITH OTHER LARGE LANGUAGE MODEL USAGE PARADIGMS
Besides work, some other works also focus how leverage large language models in creative ways to achieve better performance on downstream tasks and see improvements with proper designs. We compare Binder method with the three previous paradigms of large language model usage: End-End, Chain-of-Thouge and Semantic Parsing/Code Generation.
End-to-End End-to-End method (Brown et language models to generate final answers directly, often done by providing a task description and/or few examples for in-context learning. Despite being effective enough reach state-of-the-aar or comparable performance in a large number of benchmarks, it suffers from being uninterpretable and lacking in robustness.
2020 Hoffmann al. 2022 i.a.) aims to use large
sifin--f-thoughh Chain-of-thought methods (Wei et al. 2022 Chowdhery et al. 2022a Kojima
et al. 2022, Chung et al.] 2022] i.a.) improve the ability of large language models to perform complex reasoning by generating a series of intermediate reasoning steps. While achieving great success in various benchmarks. as a model-generated natural language, ohain-of-thouuhh suffers from unreliability and uncontrollability.
21

---

Published as a conference paper at ICLR 2023
Semantic Parsing/Code Generation Semantic Parsing/Code Generation methods (Chen et al.) 2021; Rajkumar et al. 2022] Shi et al. 2022)) aim to parse the question into pre-defined pro- gram SQL, Python, etc.), then execute it through the corresponding interpreter. It gains advantages in interpretability and robustness compared with End-to-End and Maiinof--Thoughe methods, but still suffers from the fixed grammar of the pre-defined programming language, making it inherently limited in coverage (Shi et al. 2020b).
BINDER BINDER is neural- -ouurllyymmolic paradigm that aims at mapping the question to a program that allows binding a unified LM API for additional functionalitiess This keeps the interpretability and robustness of the semantic parsing method while unlocking the grammar's limitations on its coverage.
C MORE EXPERIMENTAL RESULTS
C.1 RESULTS OF MORE PREVIOUS SYSTEMS
Due to page limits, we didn't 't list the complete previous systems in main results of WIKITQ and TABFACT. In this section, we present comparisons between more previous methods and ours, as shown in Table I0 and Table
Method
Dev
Test
Fineuned
Pasupat & Liang
2015
37.0
37.1
Neelakantan ct al. 2015
34.1
34.2
Zhang et al.
2017
40.6
43.7
Method
Small Test
Finetuned
Liang al. 2018
42.7
43.8
Dasign al.
2019
43.1
44.3
Chen
2019a
68.1
Agarwal ( 2019
43.2
44.1
Zhone et
2020
74.3
Wang 2019
43.7
44.5
Shi et al.
2020a
74.2
Herzig
2020
48.8
Yang et
2020
76.2
Xie ct al. (1122
51.9
50.6
Ou & Liu
2022
77.0
Yin of al.
2022))
53.0
52.3
Ju ct al.
212T
(BART-I
82.5
Yu al.
2020
51.9
52.7
Eusenschlos ct al.
2020
83.9
Guo ct al
2021
53.6
52.3
Xie et al.
2022
85.4
Liu et al.
2021
60.4
59.1
Liu
2021
(Tapex)
85.9
Zhou et al.
2022
61.1
61.3
Without Finetuming
Jiang et al.
2022
63.3
Codex end-to-end QA
72.6
Without Finetuning
Codex SQL'
80.7
Codex end-to-end QA
50.5
48.7
Codex BINDER ((Our)) with few-shot retriever
85.1
Codex SQL
60.2
61.1
86.0
Codex BINDER (Ours)
65.3
65.2
Table 11: TABFACT accuracy on small test set of more previous systems. means symbolic method that outputs intermediate languages.
Table 10: WIKITQ execution accuracy of more previous systems. means the result has been re-evaluated by the semantic-match evaluator in Table 9 means symbolic method that outputs intermediate languages.
C.2 RESULTS WITH WIKITQ OFFICIAL EVALUATORS
We list BINDER performance evaluated under WIKITQ official evaluator in Table 12 As shown, there exist performance drops compared with semantic-matel evaluator. The symbolic methods with SQL and BINDER see larger drops (about 2.5%) than end-to-end methods (about 1%) because the or B choice questions and number with units answers (illustrated in Table 9 are judged as incorrect.
C.3 ABLATION STUDY OF #MAJORITY VOTE CANDIDATES
In this section, we give an empirical study of how number of generations (i.e., candidate BINDER programs) affects majority voting performance. As shown in Figure 6 and Figure 7 increasing the
22

---

Published as a conference paper at ICLR 2023
Method
Dev
Test
Finetuned
T5-3B (Xie et al.
2022
51.6
50.3
Tapex (Liu et al.
2021
60.1
58.5
TaCube (Zhou el al. [2022]
60.9
OmniTab (Jang et al.
2022
62.9
Without linetuning
Codex end-to-end QA
49.3
47.6
Codex SQL
57.6
55.1
Codex BINDER (Ours)
62.6
61.9
Table 12: WIKITQ execution accuracy using its official evaluator, with normalizer recognize date.
#candidates of majority vote effectively improves the performance on WIKITQ and TABFACT. Em- pirically, TABFACT requires more candidates to determine entailed or refuted. We haven't it conducted experiments on WikITQ with more than 20 candidate programs now, which may potentially lift the current best performance a bit.
66
62
61
10
10
Figure 6: WIKITQ execution accuracy on dev Figure 7: TABFACT execution accuracy on small set with different numbers of generations (X-axis) test set with different numbers of generations (x- axis) used in majority vote.
used in majority vote.
C.4 WIKITQ SQL UNSOLVABLE PROPORTION BY OUR STATISTICS
In Table the decomposition execution accuracy on WIKITQ development set, we can see the BINDER method also gain advantage on Program-solvable subset (and pure SQL method still able to have certain performance on Program Program-unsolvan vubollvbbe part). It is surprisingly since this part could already be solved by SQL (according notation from SQUALL dataset), and thus approach does not theoretically improve the performance of this part. Further, we found this is because in SQUALL, authors normalized part of the table before doing the SQL annotation, and they also missed some samples that could be solved after normalizing the table. In turn, there is a significant portion of the data marked by SQUALL, and when we use the original table rather than the table modified by the authors, we find that it belongs to the part that cannot be resolved by the program, so that BINDER could gain some advantage on it. The same is true for the part that not marked by SQUALL. We re-executed the SQL annotation of the SQUALL dataset on the tables we normalized for each sample, check and re-judged whether the sample was a Program-sole solvable sample by whether the execution result matched the previous one, and obtained the new split of 1251 (44.19%) and 1580 and Program-solfe solvable set, respectively. (Previous split according
(55.81% 31%) for Coogram-unsolvalte to SQUALL is of 625 22.08%) and 2206 (77.92% examples. This reduces table content noise by artificially cleaning the table, but also underestimates the challenges posed by in in-the-wild data.) The new split will be released in code for future fine-grained analysis exploration.
23

---

Published as a conference paper at ICLR 2023
Ioman star
Good perfermance Teterpetsbiiity Sonability
Select progrorsengg
Areetate Binder prop
Genemte lincer progpens
4pply the anderfying modes
Exscute binder prograns
Figure 8: A pipeline to extend BINDER to new domain.
D PIPELINE TO EXTEND BINDER
In this section, we introduce a pipeline with Codex to extend BINDER a new domain with new the programming language and neural modules. Figure 8 illustrates the pipeline:
(1) A programming language is selected as the intermediate representation according to the domain. For example, SQL fits table domain, SPARQL, Cypher fit knowledge graph domain, and general-purposs purpose languages like Python, Lisp, and Java are more likely to be adapted to various domains.
Before parsing the text into BINDER programs, in-context examples required help Codex learn this task and the grammar udaption (inserted neural modules) from the original language. Empirically, since Codex has been heavily pretrained on code, dozen of BINDER program annotations are enough. Besides, annotating a pool of more examples (e.g., 100) and retrieving in-context examples for each inference sample has been proved to be effective way to improve parsing accuracy, we applied in TABFACT.
(3) Generate the BINDER programs with Codex given the in-context examples and the inference query. Codex syper-parameters can be adjusted based on the task.
Bind models with the neural module functionalities in BINDER program. In this paper, we set Codex as the only underlying model for all functionalities like commonsense QA, information extraction because large language models (LLM) are powerful and easy-to-use. If a single LLM is not enough to realize the required functionalities, e.g. VQA, bind the API with a specialized VQA model given the inputs are images.
(5) Execute the BINDER programs with the deterministic language executor and neural models.
E
ERROR ANALYSIS
ERROR TYPES
We divide the error type into:
(1) Syntax error: prediction not satisfied with the syntax of BINDER program.
(2) Semantic error:
Column: choose the wrong column, or miss the restriction from certain column.
Value: use the wrong value, or obfuscate the fuzzy match with the exact match.
Operator: use wrong operator or miss the restriction of operators.
Structure: The query produced overall structural error. Either the overall meaning of the query is misinterpreted; or the query only considers the question and is not applicable to this table all.
missing the usage of BINDER when needed, the command inside the BINDER cell is improper or ambiguous to execute.
BINDER usage:
(3) Incorrect execution:
The predicted program is correct, but the final result are wrong because wrong mid-term answers are predicted in the execution process(error in information extraction, error in handling the corner cases value etc.).
(4) False negative: the annotation of the gold answer is wrong, or the prediction is actually right while the executor misjudges it wrong due to limited ability.
24

---

Published as a conference paper at ICLR 2023
row id
year
site
winning team
losing team
scries
1902
lawrence
kansas 16
kansas state 0
ku 1-0
1903
lawrence
kansas 34
kamsas slate 0
ku 2-0
1904
manhattan
kansas 41
kansas state 4
ku 3-0
1905
lawrence
kansas 28
kansas state 0
ku 4-0
1906
manhattan
kansas state 6
kansas
ku 4-1
1907
lawrence
kansas 29
kansas state
ku 5-1
1924
manhatian
kansas
kansas
ku 17-2-3
1925
lawrence
kansas state 14
kansas
ku 17-3-3
23
1926
manhattan
kansas stare 27
kansas
ku 17-4-3
24
1927
lawrence
kansas state 13
kansas
ku 17-5-3
25
1928
manhattan
kansas 7
kansas state 0
ku 18-5-3
Table 13: The table of WIKITQ nt-1239 with title "Kansas-Kanss State football rivalry". We it for demonstration of interpretibility. Removed rows unimportant to get the answer. The query is When was the first game that kansas state won by double digits? and the gold answer is 7926.
E.2 ERROR EXAMPLE
The context(table, query, answer) of WIKITQ nt- nt-1239 is shown Table 13
F ANNOTATION INTERFACE
We build an annotation interface allowing real-time executions with huggingface space F1 SQL and BINDER with SQL are supported. We will release the source code which hopefully may benefit annotations of tasks requiring symbolic languages in the community, e.g., semantic parsing and code generation. Figure 9 and 10 demonstrate our interface.
Currently,
"htpss//tuggingffccco////mm cppppcccsssc
25

---

Published as a conference paper at ICLR 2023
Question: when was the last time kansas state lost with 0 points in manhattan?
Title: Kansas-Kansas State football rivalry
row_id
year
site
winning
losing
series
lawrence
kansas
lawrence
1904
manhattan
kansas
kansas state
3
1905
lawrence
kansas
kansas state
4-0
1906
manhattan
kansas state
kansas 4
ku 4-1
5
1907
lawrence
kansas
kansas state
ku 5-1
6
6
1908
lawrence
kansas 12
kansas state 6
ku 6-1
7
7
1909
manhattan
kansas 5
kansas state 3
ku 7-1
8
8
1911
manhattan
kansas 6
kansas state 0
ku 6-1
1912
Lawrence
kansas 19
kansas state
ku 9-1
Answer: ['1964']
Full info for this dataset item you need)
(...)
SQL
SELECT MAX(Year) FROM w WHERE QAd"map@Did Kansas State lost with 0 points?"; "losing team )
Execution Answer: [1964]
Download annotations
Figure 9: An example from WIKITQ annotation interface.
26

---

Published as a conference paper at ICLR 2023
Statement: the album powerhouse be release in 1968
Table:
caption
none
row_id id
year
title
from album
label
0
1962
2022-8-29tni/ sinnin' sam
lookin' ahead
pacific jazz
1
1962
the young rabbits / /song....
lookin' ahead
pacific jazz
2
1962
congolese sermon weat...
(only 45)
pacific jazz
3
1963
samba/ tough talk
tough talk
world pacific
1963
turkish black/ boopie
tough talk
world pacific
1963
spanish castles bluesette
jazz waltz (w les mccann)
world pacific
heat wave/ broadway
heat
world pacific
1964
remember 2022-8-30 lo...
stretchin' out
world pacific
1965
tough talk/ the thing
thing
world pacific
1965
anua
Answer: 1
means entailled means refuted.
Full info for this dataset item you need)
(...)
Neural SQL
SELECT (SELECT year FROM WHERE ffom album* "lighthouse '68")
Execution Answer: [1]
Submit
Download annotations
Figure 10: An example from TABFACT annotation interface.
27


Reviews:
[{"review_id": "OaFocbMmEb", "reviewer": "Reviewer_Pz8D", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "1) The paper proposed BINDER to incorporate symbolic component into large language models, and the main benefit of the method is not requiring any fine-tuning.\n2) BINDER outperforms state-of-the-art results on WIKITABLEQUESTIONS and TABFACT datasets \n3) useful analysis is presented to help readers understand the proposed approach.", "summary_of_the_review": "The proposed idea is exciting, the results are fairly strong, but the writing needs to be improved.\n[Edited: The author has improved their writing to make the paper much clearer.  I have thus raised my score from 6 -> 8.]", "strength_and_weaknesses": "Strength:\n1. Incorporating symbolic component into end2end learning systems is always an interesting idea\n2. The proposed approach achieved state-of-the-art results\n\nWeakness:\n1. The writing is not clear throughout.  Most importantly, the paper directly jumps in talking about the proposed method without first clearly introducing the problem setup.\n2. Again, presentation is not clear.  The main figure is confusing.  I do not understand how the prompting part is employed exactly in the system.  In particular, it is not clear how the prompting results are combined with the parsed results to feed in the program interpreter.\n3. The results are a bit over sold.  Even though the fine-tuning methods require more task annotations, but GPT3 codex is pretrained on many more data.  The improvement is not that surprising.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LLUpQGiEnq6", "reviewer": "Reviewer_8V6D", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a training-free two-stage method based on Codex model for QA tasks. The first stage uses few-shot learning to prompt Codex and converts a natural language question to a SQL or Python program, where difficult-to-resolve parts are represented as API calls. In the second stage, Codex is prompted again to generate answers to each said API call, and then the whole program is executed to obtain the answer. Experimental results show new state-of-the-art on WikiTQ and TabFact datasets, with particular performance gains on questions that are unsolvable by natural-language-to-SQL translation.", "summary_of_the_review": "The main idea is good and well executed.", "strength_and_weaknesses": "This paper is a creative usage of a code generation model. It also represents an interesting new approach to neural symbolic: code generation models can learn to divide a task into a high-level symbolic logical program and some low-level components for neural inference.\n\nThe experimental results are promising. With a pre-trained Codex model, without fine-tuning and with a small number of annotated examples, the proposed method is able to achieve state-of-the-art on two QA datasets. It achieves the desired behavior of a neural symbolic method, including better interpretability and answering some difficult questions.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "BSt1VEsAUOH", "reviewer": "Reviewer_S9TA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper propose BINDER, a training-free neural-symbolic framework that (1) maps the task input to a program using a pre-trained language model, where the model has to decide which part in the input can be converted to a target programming language and corresponding tasks API calls for further extension; (2) adopts an LM as both the program parser and the underlying model called by the API during the execution, where a language model serves to realize the API calls given the prompt and the return values feed back into the original programming language; (3) derive the final answer via a deterministic program interpreter executing the program without API calls. BINDER achieves state-of-the-art results on WikiTableQuestions and TabFact datasets, where previous work is all fine-tuned on a large-scale training set.", "summary_of_the_review": "- This paper is clear and well-organized. Lots of complementary materials are also provided in the Appendix to help us understand the technical details better.\n- The methods can be applied to a broad range of semantic parsing tasks since it use uniformed underlying APIs, i.e., SQL and Python, and require no further training process.\n- The experimental results are impressive. It outperforms previous work which typically use finetuning settings based on large training set.", "strength_and_weaknesses": "Strength:\n- The system does not require extra training but can achieve better performance compared to fine-tuning settings.\n- The in-depth analysis validates that BINDER can have significantly better performance on program-unsolvable questions.\n\nWeakness:\nNo obvious weakness was observed.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%