SCoRe: Pre-Training for Context Representation in Conversational Semantic Parsing
Authors: Tao Yu
Rui Zhang
Alex Polozov
Christopher Meek
Ahmed Hassan Awadallah
Venue: ICLR 2021 Poster (2021)

Abstract: Conversational Semantic Parsing (CSP) is the task of converting a sequence of natural language queries to formal language (e.g., SQL, SPARQL) that can be executed against a structured ontology (e.g.  databases, knowledge bases).  To accomplish  this  task,  a  CSP  system  needs  to  model  the  relation  between  the unstructured language utterance and the structured ontology while representing the multi-turn dynamics of the dialog. Pre-trained language models (LMs) are the state-of-the-art for various natural language processing tasks. However, existing pre-trained LMs that use language modeling training objectives over free-form text have limited ability to represent natural language references to contextual structural data. In this work, we present SCORE, a new pre-training approach for CSP tasks designed to induce representations that capture the alignment between the dialogue flow and the structural context. We demonstrate the broad applicability of SCORE to CSP tasks by combining SCORE with strong base systems on four different tasks (SPARC, COSQL, MWOZ, and SQA). We show that SCORE can improve the performance over all these base systems by a significant margin and achieves state-of-the-art results on three of them.

Published as a conference paper at ICLR 2021
SCORE: PRE-TRAINING FOR CONTEXT REPRESENTA- TION IN CONVERSATIONAL SEMANTIC PARSING
Tao Yu
Rui Zhang
Yale University tao. cao.yulya eelyylle.edi
The Pennsylvania State University 2mz22270pps. edu
Oleksandr Polozov, Christopher Meek, Ahmed Hassan Awadallah Microsoft Research {polozov, meex, Iassaran)eeilrrosoft.com
ABSTRACT
Conversational Semantic Parsing (CSP) is the task of converting sequence of natural language queries to formal language (e.g., SQL., SPARQL) that can be executed against a structured ontology (e.g. databases, knowledge bases). To accomplish this task, CSP system needs to model the relation between the unstructured language utterance and the structured ontology while representing the mulli-turn dynamics of the dialog. Pre-trained language models (LMs) are the state-of- the-art for various natural language processing tasks. However, existing pre-trained LMs that use language modeling training objectives over free-form text have limited ability represent natural language references to contextual structural data. In this work, we present SCORE, new pre-training approach for CSP tasks designed to induce representations that capture the alignment between the dialogue flow and the structural context. We demonstrate the broad applicability of SCORE CSP tasks by combining SCORE with strong base systems on four different tasks (SPARC, CoSQL, MWoZ, and SQA). We show that SCORE can improve the performance over all these base systems by significant margin and achieves state-of- -her results on three of them.
INTRODUCTION
The goal of task-oriented dialog systems to assist the user in completing a certain task by performing an action or retrieving relevant information (Tur & Mori 2011). They are often built top of structured ontology grounded in a knowledge base, a database, or a set of API calls. This in contrast to open-domain dialog systems (also referred to chit-chat systems) where the goal is to maximize engagement with users in open-ended conversations (Jafarpour et al. 2010 Ritter et al. T).
A key component of task-oriented conversational systems is Conversational Semantic Parsing (CSP), which converts each utterance in the dialog into a formal language query (e.g.. SQL., SPARQL) that can be executed against the structured ontology. CSP has been extensively studied in several academic and industrial research settings such as dialog systems (e.g., dialog state tracking in MWoZ (Budzianowski et al., 2018). interacting with physical agents (e.g., (Chai et al. 2018)),
context-dependert semantic parsing (e.g., SPARC (Yu et al. 2019b)), LQL-grounded state tracking (e.g.. CoSQL (Yu al.) (211a)), and sequential question answering (e.g.. SQA (lyyer et al., 2017)). These settings differ in some respect, but they share the same overall objective and key challenge: how to jointly represent the natural language utterances and underlying structured ontology while taking into consideration the multi-turn dynamics of the dialog.
Similar to many other natural language tasks, recent work in CSP has significantly benefited from advances in language model pre-training. However, existing general-purpose pre-trained language models, e.g. BERT (Devlin et al. (2019), are pre-trained on free-form text data using language model objectives. This limits their ability in modeling the structural context the multi-turn dynamics of the dialogs. This presents an opportunity to improve pre-trained LMs to specifically address these limitations for CSP tasks. Recent work has demonstrated the benefits of adapting pre-trained LMs

---

Published as a conference paper at ICLR 2021
Dialogs
Formal Programs
Database
NAP- sm lodtlooooooo SOL
Uar: Find the numras of the top 3 Higheal sales bookka.
SELECT FRDM book ORDER BV sal, arrount DESC LIVIT
Usr. Who are their outhors?
SSLECT Lile, nava FRos auther As JCHN besk A012 ChitU 12 Faasthr in oeass Br czasi, ancuure Disc
Press Inamre) oodiess
Also the their publishers
d' t3e ONDER SV viliaarrn DESC UMII3
ECTRL otte cloom 12 FROM sumher IDKN booe
Book (itc) outher i sale amourt
looking sheep
Retzueet sccches ansoncanter)
Dialog stan treeli
There chinese restouront colled
Daju Nococh Batr
Antraver laanches areancerte: carrnn leee-
Rectaurent romn price [ste]]
Simee
Usr. Yee please, for people at 18:30 on Thurscay
loweet I..I destiration
Uar: b to bask lood balwween the restaurant 20:30
Bolnanmi/arioors you arce-ondee rumenDup Noodl Sinenti Sasersseer-000: Ew-tlodion--Diieooooooo
Sya:
Figure I: Examples of conversational semantic parsing tasks from SPARC and MWoZ datasets.
to specific domains (Gururangan et al. 2020) or tasks (Zhang et al. 2019b via a second phase of pre-training. For example, open-domain dialogue language models such DialoGPT (Zhang et al.) 2020) and ConveRT (Henderson et al.] 2019) are pre-trained on the Reddit data and applied to dialog response generation and retrieval tasks.
In this paper, we introduce SCORE (Structured & Sequential Context Representation), a language model pre-training approach for CSP tasks. SCORE adapts general pre-trained LMs by introducing second phase of pre-training using multiple objectives that capture both multi-turn dynamics and the structural contexts in dialog. In contrast to open-domain dialogs, CSP datasets are usually much smaller due to the difficulty and expense of obtaining and labeling data (mapping natural language utterances to formal language). Unlike most prior work on contextualized LMs which are pre-trained on free text, according to the finding where questions in CSP tasks are more compositional than other free-text since they can be mapped into formal representations, we propose to train SCORE on synthesized conversational semantic parsing data with multiple training objectives that aim to ground utterances into the schema of the underlying ontology and to model the relationship between different utterances in the multi-tura conversation. In this way, SCoRE can effectively inject structural and conversational inductive biases in LMs that can translate to many CSP tasks. SCORE uses an order of magnitude smaller dataset for the second stage of pre-training, does not require changes to the pre-trained model architecture, can be used as a drop-in replacement of general pre-trained LMs with any semantic parsing model, and can be used out-of-the-box in many CSP tasks.
We apply SCORE to four different CSP tasks: (1) sequential text-to-SQL (SPARC), (2) conversational text-to-$QL (CoSQL), (3) dialog state tracking (MWoZ), and (4) weakly-superviser sequential question answering (SQA). The fours tasks represent different scenarios, types of ontologies, super- vision signals, system responses, and domains (see Table for detailed comparison and Figure [] for examples). We demonstrate that: (1) SCORE training objectives can effectively incorporate synthesized data, (2) a single pre-trained SCORE model can be used for several CSP tasks and can be combined with many baseline systems with different model architectures and (3) SCORE significantly improve all baseline systems and achieves new state-of-the-an results on three benchmarks (SPARC, SPARC, and MW and comparable performance to state-of-the-as results on the fourth (SQA).
2 APPROACH
The key challenge of CSP is to capture the relationship between the natural language utterance and the structured ontology in the multi-turn dialog dynamics. To this end, we inject structural and conversational inductive biases in SCORE by introducing two objective functions: Column Contextual Semantics (CCS) and the Turn Contextual Switch (TCS). Because the size of existing semantic parsing datasets is limited, we produce synthesized data for pretraining SCORE by sampling from the context-free grammar induced from complex texl-to-SQL examples in different domains. Moreover, to prevent SCORE from overfitting to the linguistic pattern of our synthesized data, we use the Masked Language Modeling (MLM) objective on numan-generate utterances as regularization.
2.1 PRELIMINARIES
Task Definition In CSP, at each turn t, we aim to produce formal representation qi given the current utterance 44 the interaction history h2 = [ui, U2..... 14-1], and the schema c (table and column names, slots, etc.) of the target database (ontology) d. To cover different problem variants,
2

---

Published as a conference paper at ICLR 2021
Structured
Annotation (Supervision)
Cross Domain
System Response
Dataset
Ontology
# Dialogs
Turns
SPARC
database
SQL (supervised)
x
4,298
12,726
CoSQL.
database
SQL (supervised)
3,0077
15,598
MWoZ
domain ontology
slot-value (supervised)
X
8,438
113,556
SQA
table
denotation sveakly-uupervieed)
6,066
17,553
Table 1: Comparison of CSP datasets. Examples from two of the datasets are shown in Figure Cross-domain means the train and test sets have different domains, MWOZ is not cross-domain,
consider four popular CSP tasks shown in Table ! SPARC (sequential text-to-SQL), CoSQL (con- versational text-to-SQUi SQL), MWoZ (dialogue state tracking), and SQA (weakly supervised sequential question answering). They have different target formal language and structured ontology:
For the utterance u, is the user question for SPARC and SQA, while for CoSQL and MWoZ, u is the combination of a user query and a system response.
For the database d, SPARC and CoSQL use multi-table databases; for MWoZ, the pre-defined ontology d can also be viewed as database; for SQA, d is single table.
For the formal representation q. is the SQL query SPARC and CoSQL; in MWoZ the saouevalue pairs that can be viewed as simple SQL queries consisting of SELECT and WHERE clauses; and for SQA, q is the latent program.
The base architecture of SCORE takes as input single turn of a CSP dialog (ue, ft) jointly with the underlying database schema c. Given this contextualized conversational imput C = (ut, he, c). SCORE encodes it into contextualized conversation representations S; for each token in Cz. The encoder architecture follows RoBERTa (Liu al. 2019b). it then followed by a linear layer and normalized (Ba al. 2016) produce final representations hy for each token:
Base Architecture
V St, E 3, (I) where GELU is activation by HHendrycks & Gimpel (2016) and W, is learned parameter matrix.
C = =(ue, hy, c). S, = ROBERTA(C)), h1 = DereeNooo(GELU(W///
To build Ct, we first concatenate current utterances u, and dialog history h, separated by a special token 
, this simple strategy has been shown effective in state-of-the-art CSP systems (Zhang et al. 2019c Wu et al. 2019) Liu et al. 2020)) Heck et al.) 2020). To incorporate the database schema, we follow Hwang et al. (2019) to concatenate all column names as a single sequence. Column names are separated by the special token 
 and prefixed by their corresponding table name.
2.2 SCORE PRE-TRAINING
SCORE addresses the challenges of CSP by pre-training a task-oriented language model contextual- ized by the conversational flow and the underlying ontology. In pre-training, the SCoRE model is self-supervised by two novel objectives in addition to the established Masked Language Modeling (MLM) objective. These objectives facilitate the accurate representation of the conversational flow between dialog turns and how this flow maps to the desired columns the ontology.
Column Contextual Semanties The first challenge of CSP is capturing the alignment between the natural language utterance and the underlying database schema. To address it, we optimize the SCORE model with the auxiliary objective of Column Contextual Semantics (CCS). For each column in the database schema c, CCS targets the operations that should be performed on this column in a given conversational turn. Specifically, each formal representation is decomposed into operations on columns and tables, e.g. GROUP BY and HAVING for SQL queries, or WHERE for the slot-value pairs. In this way, data covers 148 column operations. We use the encoding of the special token 
 right before each column or table name to predict its corresponding operations, and then compute the CCS loss:
(oes(C.)= E iec CrossEntropy (LayerNorm W2 h CCS(4))
(2)
where hf, is the contextualized representation of the 8th column's special token 
 in the contextu- alized input Ct. CCS(g:) returns the column operation label for the current formal representation qt. CrossEntropy 148 computes the 148-way cross-entropy between the column operation prediction and label, and Wa learned parameter matrix.
3

---

Published as a conference paper at ICLR 2021
TCB MOLTAATTOO
TCa
MLL
OCS
CCB
CCS
OCS
Transformer Encoder
aiao show namis thei pubraners
si vit sull vinua? books
<hi author wuller she aroun..
Gurrent Question
Dislog History
Database Scheme
Figure 2: Pre-training of SCORE encoder on a SPARC text-to-SQL example from Figure
I
Turn Contextual Switch The second challenge of CSP is capturing the conversational context flow and how is grounded into the formal representations. The TCS objective aims capture this grounding of context flow. To this end, it targets predicting the difference in formal representations between dialog turns based on the natural language utterance.
Based on the context-free grammar of SQL, we identify 26 possible turn difference operations that a conversational turn could elicit. They encode changes between different turns of user queries (the sys- tem response is not involved here) since we assume that most turn contextual shifts are from the user. For example, INS (WHERE) indicates inserting new WHERE condition and DEL (SELECT agg) indicates removing an aggregate operation from SELECT statement (e.g. when an utterance "Show all the ages instead." elicits change SELECT MAX (age) use the encoding of the special token 
 right before each turn to predict the context switch label between this turn and the previous history:
+ SELECT age ...). We
Lres (C2) = CrossEntropy (LayerNoorn W H;), TCS 9t-1))
(3)
where H; E [[(-1)xdd is the contextualized representation all previous turns in C2 with hidden dimension d, TCS(qr: 9--1) returns the turn difference operations from --11 to drs and W3 is learned parameter matrix. We don't 't use this objective to pre-train SCORE for MWoZ because the context switch label between turns is relatively simple in MWoZ (only select and where changes).
Masked Language Modeling As in prior work large-scale language models (Devlin al. 2019), we use the Masked Language Modeling (MLM) objective to facilitate contextual representation learning for natural language utterances. Importantly for regularization,
only apply this loss
on in-domain human- uaman-annottted natural language data. Namely, includes utterances in SPARC, CoSQL, and SQA well as nine task-oriented dialog datasets processed by Wu et al. (2200) for
MWoZ (see data statistics in Figure 4 Formally, the MLM loss is given by:
LMIM (C;) = Σ CrossEntropy vocab (LayerNorm (Layeenttp (Wah)
(4)
where hy" are the contextualized representations of the masked 15% of tokens in Cr, and W4 is learned parameter matrix.
Pre-Training Setup and Steps To summarize the pre-training steps, first collect dataset Dan of combined -uman-annotated natural language questions (without labels) from existing CSP tasks (as mentioned above), and create large synthesized conversational data Daya that is generated by grammar induced from a small set of SPARC annotated examples (See 2.3). After that, we incorporate both datasets in pre-training. More specifically, synthetic and natural examples are randomly sampled during pre-training. The total pre-training loss is the sum of the three objectives with CCS and TCS only applied Psya and MLM only to Dnati
C= -=EGEp (Cocs(C.) + (Tcs + (M,)
(5)
Figure 2 shows an overview of SCORE pre-training on an example SPARC dialogue from Figure
We report additional implementation details for pre-training SCORE in Section 3.3 Appendix C
4

---

Published as a conference paper at ICLR 2021
2.3 DATA SYNTHESIS
We re-use the synthetic dataset of 120k synthetic task-oriented dialogues for MWoZ, introduced by Campagna al. (2020). In this work, we introduce a complementary procedure synthesize data for conversational text-to-SQL dialogues. We use about 400k tables in WIKITABLES (Bhagavatula et al. 2015) (after filtering and cleaning), WikiSQL, and Spider datasets as underlying databases d. and then synthesize about one dialog for each table. Finally, we synthesize 435k text-to-SQL conversations in total. Table Appendix shows an example of the synthesized question-SQL. pairs and their corresponding templates in grammar.
To this end, we use only 500 dev exam- ples from SPARC induce two utterance- SQL generation grammars: (I) a single- turn context-free grammar G, for generating context-independent question-SQL pairs, and (2) follow-up context-free grammar G, for follow-up question-SQL pairs. The single-turn grammar Ge contains list of synchronous question-SQL templates where typed slots (COLUMNO, OPO, VALUEO,... represent tions of tables, columns, values, and SQL opera- tions. The follow-up grammar G, contains con- text switch labels and lists of follow-up question templates. For example, if the context switch 11:
Algorithm Data synthesis algorithm
1: h +
2: rs SAMPLE(G.)
3: No. 90 < -RANDASSIONSLOTSS(.rr
4: h+ =(ão, 40)
5: up. qp f- uo 9o
6: for & 3 1 to T do
represssen men-
7:
if RAND (0, 1) < 0.2 then Ts f SAMPLE(G.)
8:
9:
üz, qt < RANDASSIONSLOTS((.r
else
Te + SAMPLE(G.)
label is INS (SELECT col umm0), the corre-
if CONSTRAINTCHEECKKT 9p) then üt. 9: &- EDITASSIGN( Tc) h+ (ūt, re) up. Ap f- ur 9t
sponding question could be "How about show 13:
col umn too?". . To ensure generalization, only induce the grammars from the SPARC train- ing set. Appendix B shows examples of the return grammar rules and synthesized utterances.
15:
The data synthesis procedure using the two grammars is shown in Algorithm 1 Given database d and sampled single-turn question-SQU SQL template, the function RANDASSIGNSLOTT samples values (column names, cell values, and SQL operations) for typed slots in the template and returns the first synthesized question ing and the corresponding queyy ÿo- To generate T follow-up question-SQL. pairs, the function CONSTRAINTCHECK/ (re: 4p) checks the previous query Op satisfies constraints of the sampled template тe (e.g. contains its mentioned nonterminal). Finally, EDITASSIGN ra) edits the previous SQL Gp to generate the current follow-up SQL label q and samples values for typed slots in the template to generate the corresponding follow-up up question ūt
3 EXPERIMENT SETTINGS
3.1 DATASETS AND EVALUATION METRICS
We evaluate SCORE on four popular CSP tasks: SPARC (sequential text-to-SQL), CoSQL (conver- sational -ottto--SLL, MWoZ (dialogue state tracking), and SQA (sequential question answering). summarized in Table I
SPARC (Yu et al. 2019b is large collection of sequences of inter-related context-dependent question-SS SQL pairs. contains 4.3K questions sequences and 12k+ questions. CoSQL (Yu et al. 2019a is large conversational text-to-SQL. corpus, with 3k dialogues, collected under the Wizard- of-Oz (WOZ) setting. We focus on the SQI--goounde dialogue state tracking task which maps user intents into SQL queries if possible given the interaction history. Both SPARC and COSQL. cover 200 complex DBs spanning 138 domains.
htrras//aale-111yy.tthbbbbb io/sparc
https://yale-11ly..om 1
5

---

Published as a conference paper at ICLR 2021
MWoZ (Budzianowski et al. 2018 Eric et al. 2019 is corpus of over 10k human human written task-triented dialogs created through a WOZ crowdsourcing setting. We focus on the belief state tracking task in MWoZ which maps multi-turn user utterances to slot-value annotations.
SQA dyyer et al. 2017 is constructed from a subset of WikiTableQuestions (Pasupat & Liang 2015) by decomposing highly compositional questions into sequence of simple questions. The task because each resulting decomposed question is only annotated with answers one or more table cells, while the logic program is latent. It has 6,066 question sequences with 17,553 questions in total on 982 unique open-domain tables from Wikipedia.
is
We adopt the official metrics defined for each of the tasks. For SPARC and COSQL, report question match accuracy (QM): the exact set match accuracy (Yu et al. 2018b) over SQL templates and interaction match accuracy (IM): the ratio of interactions for which all questions are predicted correctly. For MWoZ, we report joint goal accuracy (JGA) which is similar to the IM accuracy used in SPARC and CoSQL. Finally, for SQA, we report denotation QM and IM accuracies.
3.2 BASE MODELS AND OTHER BASELINES
base model. Since it is originally developed for single-turn text-to-SQL, we extend it to a multi-turn setting by concatenating current utterances and dialog history (see Section 2.2). Note that RAT-SQL alone, without SCORE, achieves better or comparable results to state-of-thee the-art models developed for SPARC and COSQL.
For SPARC and CoSQL, we RAT-SQL (Wang al. 2020
For MWoZ, we employ Trippy (Heck et al., 2020). It achieves state-of-the-ar performance on MWoZ and uses BERT to encode user and system utterances and dialog history. We report higher results (around 2%) for Trippy than reported by Heck al. (2020) since we train for more epochs (25 vs. 10). To show the improvement of SCORE is not tied to specific base systems, we also experiment with another strong base model SOM-DST (Kim al. 2020) for MWoZ and follow the same experimental details to train it.
For SQA, we use
semantic parser proposed by Wang et al. (019). The model first generates an abstract program given input question and then instantiates it by searching for aligmments between slots in the abstract program and question spans. As is originally developed for single-turn questions, extend it to the multi-turn setting in the same way RAT-SQL.
We report additional implementation details for all base models in Appendix C In addition to reporting results for all base models with SCORE, also report original base models results (with BERT and/or ROBERTA) and several other state-of-theearr baselines for each task.
3.3 DATASET USAGE IN PRE-TRAINING
In our experiments and ablation study, we train several versions of SCORE with different objectives and datasets: (1) SCORE (MLM): pre-trained on annotated natural questions using MLM. (2) SCORE (CCS+TCS); pre-trained on only synthesized data, which achieves the best results on SParC, CoSQL, and SQA. (3) SCoRE (CCS+TCS+MLMY) pre-trained on the synthesized data using CCS+TCS and annotated natural questions using MLM.
Furthermore, note that the synthesized data is generated using grammar induced by about 500 examples from only SPARC. Therefore, no CoSQL or SQA data are seen in any pre-training steps. For MWoZ, Campagna et al. (2020) study only the dev examples induce the data synthesis grammar.
4 RESULTS AND ANALYSIS
Overall Results The results of SPARC and CoSQL, MWoZ, and SQA are in Table 2, and 4 respectively. We run each main experiment three times with different random seeds and report the mean. Overall, SCORE gains significant improvements over BERT and ROBERTA on all tasks, performances on SPARC, CoSQL, and MWoZ.
achieving state-of-the-art patt-
mu1 woo
http://ake.

---

Published as a conference paper at ICLR 2021
SPARC
CoSQL
Dev
Test
Dev
Test
Models
QM
QM
IM
QM
IM
QM
IM
SyntaxSQI. Yu al. 2018a
18.5
4.3
20.2
5.2
14.2
2.2
GAZP BERT Zhong
2020
48.9
29.7
45.9
23.5
42.0
12.3
39.7
12.8
EditSQ BERT Zhangg et al, 2019c
47.2
29.5
47.9
25.3
39.9
12.3
40.8
13.7
IGSQL BERT
50.7
32.5
51.2
29.5
44.1
15.8
42.5
15.0
R²SQL+ BERT
55.8
30.8
46.8
17.0
RAT-SQL BERT Wang al. 2019
56.8
33.4
48.4
19.1
+ ROBERTA
58.2
36.7
50.1
19.3
+ SCORE
62.2
42.5
62.4
38,1
52.1
22.0
51.6
21.2
Table The SPARC and CoSQL accuracy over all questions (QM) and all interactions (IM). The scores of IGSQL + BERT and RASQL BERT are from the official leaderboards.
Models
MWoZ 2.1
SQA
DST-reader Gao et al.
2019
36.40
Models
QM
IM
TRADE eW al. 2019
46.60
DS-DST Zhang al.
2019a
51.21
Pasupat & Liang
2015
33.2
7.7
SOM-DST Kim al. 2020
52.57
Ncelakantan ct al.
2017
40.2
11.8
DS-picklist Zhang
2019a
53.30
Tyyer et al. 2017
44.7
12.8
TripPy (Heck et al.
55,29
Sun el al. 2019a
45.6
13.2
Simple (Hossemn-
2020
55.72
Muller al. 2019
55.1
28.1
Herzig el al.
2020
67.2
40.4
TripPy (ours)
58.37
+ SCoRE
60.48
Wang et al.
2019
RoBERTa
62.8
33.2
Wang et al.
2019
SCORE
65.4
38.5
Table 3: Joint goal accuracies (JGA) on MWoZ 2.1 test set. All models use a BERT-like en- coder/GPT.
Table 4: Question (QM) and interaction (IM)
accuracy on the SQA test set.
For SPARC and CoSQL Table 2 compared with ROBERTA, SCORE boosts the performance by 4.0% QM / 5.8% IM SPARC, and 2.0% QM 2.7% IM CoSQL. This demonstrates the effectiveness of SCORE on contextual semantic parsing tasks. In addition, on MWoZ dialog state tracking task in Table TripPy achieves 60.5% % JGA by replacing BERT with SCORE, outperforming the prior state-of-the-art [Hosscini-Asl et al. 2020 by 4.8%. This indicates that dialog state tracking also benefits from SCORE. Finally, SCORE also achieves higher performance than EOBERTA on weakly supervised sequential question ans weriernng SQA task. As Table 4 shows, SCORE improves QM by 2.6% and IM by 4.9% over ROBERTA with Wang al. (2019) as the base model. This demonstrates that the enhanced ability of semantic parsing and context modeling in SCORE is transferable to denotation-ban based CSP tasks.
What is the effect of each pre- training objective? T shows an ablation study on different pre- training objectives. We find that
Learning Objective
SPARC
CoSQL
MWoZ
SQA
MLM only
37.0(40.3)
20.3(+1.0)
59.47(+1.10)
34.7(+1.5)
CCS only
41.3(+4.6)
21.2(+1.9)
59.32(-0.95)
32.7(-0.5)
CCS+TCS
42.5(45.8)
22.0(+2.7)
38.5(45.3)
the best SCORE results are achieved by pre-training on only synthesized data (CCS+TCS) without any nat- ural questions (MLM) on SPARC, CoSQL. and SQA but not on MWoZ. By adding MLM to CCS+TCS (CCS+TCS vs. CCS+TCS+MLMM) MLM actually hurts the performance (-3.9% on SParC. -0.3% on CoSQL, and 4.4% on SQA) while increases for MWoz. One possible reason is that questions in MWoZ are more diverse in language but less compositional while semantic compositionality and turn changes are more important in the other three CSP tasks. Also, the synthesized data used pre-train SCORE for SPARC and CoSQL is generated by the grammar induced by SPARC, which might overfit to SPARC. In addition, SCORE pre-trained with only MLM loss improves the performance 11.%) but not as large as CCS+TCS (+5.5% on SPARC, +1 +1.7% 7% on CoSQL, and +3.4% on SQA). Finally, we test the effectiveness of TCS on SPARC, COSQL, and SQA by adding TCS CCS (CCS only vs.
CCS+TCS+MLM
38.6(+1.9)
21.7(+2.4)
60.48(+2.11)
33.7(+0.5)
Table 5: The effect of SCORE pre-training objectives. Im- provements are shown the parentheses.
7

---

Published as a conference paper at ICLR 2021
CCS+TCS), SCoRE gains improvements of 1.2% SPARC and 0.8% CoSQL, and 4.4% on SQA.
Does SCORE improve question match accu- racy on individual turns? T de- tailed results of SCORE's question accuracy for individual conversation turns on the SPARC dev
QM
Q1
Q2
Q3
Q4
RAT-SQL BERT
56.8
71.1
53.6
47.8
31.8
+RoBERTa SCORE
58.2
68.7
58.5
48.9
35.2
62.2
70.6
63,5
52.6
45,5
set. SCORE provides a significant improvement for every conversation turn except the first (in which the task more similar to single-turn semantic parsing). CoSQL and SQA exhibit similar behavior and are presented in Appendix A
Table 6: Detailed results on the dev set of SPARC. Q: is the accuracy of the 7th conversation question.
What if we use the synthesized data to simply augment the training data? To answer this, we compare the results of the base models trained with without the synthesized data CoSQL and MWoZ. As shown in Table 7 the extra synthetic data does not significantly improve the performance, indicating that directly augmenting the synthetic data to the training set is not effective. The similar findings are reported in many recent work (Zhang al.) 2019c Herzig et al. 2020) Campagna et al. 2020 Zhong et al. 2020). In contrast, pre-training on the synthesized data with our objectives improves the performance on the downstream tasks.
CoSQL
MWoZ
no syn with syn
48.4
58.37
48.6
58.45
Table 7: Effect of synthetic data as training data augmentation.
How general is SCoRE and its synthetic gram- mar? For generalization in task settings, we have shown that the pre-training strategy of SCORE improve the performance over different CSP tasks including semantic parsing (SPARC and CoSQL). dialog state tracking (MWoZ), and weakly super- vised table question answering (SQA). addition, we demonstrate the effectiveness of SCORE on dif- ferent base models. To this end, we experiment with different base model SOM-DST for MWoZ. As shown in Table & SCORE can still improve the performance with a different base model on MWoz (SOM-DST+BERY vs. SOM-DST+SCORE on syn. MWoZ).
MWoz
SOM-DST BERT
52.57
+ SCORE on syn. text-to-SQL
53.57
+ SCORE on syn. MWoZ
54.61
Table 8: Performance of SCORE pre-trained
on different synthesized data on MWoZ.
To demonstrate the generalization in synthetic grammar and data, as shown in Table 2 and 4 SCORE (TCS+CCS) is pre-trained on the synthesized data of the grammar induced from SPARC only, and still improves the performance on CoSQL (+2.7%) and SQA (+4.9%) where any CoSQL and SQA annotated data is seen in any pre-training steps. Moreover, in Table g we show that SCORe pre-trained the text-to-SQL synthesized data could also surprisingly improve the performance on MWoZ. We expect that higher performance could be achieved with SCORE pre-trained on task-specific synthesized data. Finally, our pre-training approach can be applied to any existing LMs including larger seq2seq LMs (e.g., BART (Lewis et al. 2020), T5 (Raffel et al. 2020)).
Can SCORE deliver more value when in-domain data limited (e.g., in low-resource setting)? We want to answer this question similar to experiments other investigations of LMs as few-shot learn-
QM
IM
RoBERTa
53.3
21.2 26.1
ers (Wu al. 2020 Brown et al. 2020 Schick & Schütze 2020). To this end, we compare ROBERTA and SCORE under a few-shot setting on SQA when only 10% of training data is available. We choose SQA because its annotation is most different from the synthetic text-to-SQL dataset we use for pretraining. demonsseetsssstess that SCORE delivers even larger improvements compared to the ROBERTA baseline when only 10% training data is available (3.8% vs 2.6%).
SCORE
57.1
Table 9: Performance of SCORE on 10% training
data of SQA,
5 RELATED WORK
Conversational Semantic Parsing Conversational semantic parsing is one of the most important research topics in conversational Al and has been studied in different settings including task-orient oriented dialogues, question answering, and text-to-SQI. Task-oriented dialog systems (Henderson al.
8

---

Published as a conference paper at ICLR 2021
2014; Wen et al. 2016 Mrkšić et al., 2017 BBdziaanwssi al. 2018) aim to help users accomplish a specific task (e.g. flight booking) and often pre-define slot templates grounded in a domain- specific ontology. In comparison, several other datasets were recently introduced for cross-domain conversational text-to-SLL. tasks (SPARC and COSQL. Yu et al. 2019a (b) and sequential questions answers over tables (Iyyer al. 2017). While the previous work has achieved significant progress in different datasets separately, to the best of our knowledge, we are the first to study four different CSP tasks together (sequential text-to-SQL, conversational text-to-SQL, dialog state tracking, and weakly-supervisee sequential question answering) by addressing the shared key challenge of learning representations in pre-trained language models that capture the alignment between the dialogue flow and the structural context.
Conversational Language Model Pre-training Several recent efforts have demonstrated the value of adapting pre-trained LMs specific tasks using different pre-training objectives, e.g., summariza- tion (Zang et al. 2019b). knowledge inference (Sun et al. 2019b Liu et al. 2019a), etc. Closest to our work is adapting pre-trained LMs for open-domain chit-chat models and for tabular data representation. The former focuses on improving response generation on open ended dialogues by adding a pre-training step on open-domain conversations data, such as Reddit data (Zhang al.) 2020. Henderson et al. 2019). For example, WWu et al. (2020) introduced ToD-BERT,, pre-trained language model combining 9 high-quality human- humann task-oriented dialogue datasets to conduct language model and response selection pre-training. However, they use language modeling training objectives over free-form text and therefore have limited ability to represent structural data. The latter has focused on improving language model pre- pre-training for encoding tabular data (Yin et al., 2020 Herzig et al. 2020), but they focus on the single turn semantic parsing setting. Our approach is different from previous work because we address the challenge of conversational semantic parsing tasks by learning pretrained representation for both the multi-turn dynamics of the dialog and the relation between the unstructured language utterance and the structured ontology. Furthermore, our pre-training approach is much more data-efficien than prior LM pre-training work and saves lot of time and computing resources (Appendix DJfor more details). Our pre-training step can be done within only one day using 8 V100 GPUs
Using Synthesized Data for Semantic Parsing Synthesized data has been frequently used in semantic parsing to alleviate the challenge of labeled data scarcity. For example, Wang et al. (2015) proposed a method for training semantic parsers in new domains by generating logical forms and canonical utterances and then paraphrasing the canonical utterances via crowd-sourcing. Similar approaches were used to train semantic parsers in other domains and settings (Zhong et al. 2017
Su et al. 2017: Cheng al.
2018 Shah et al., 2018). Another line of work has proposed using synthesized data to adapt single turn semantic parsing models to new domains (Jia & Liang 2016 Yoo
al. 2018 Campagna et al.
2019) and task-oriented dialogues (Campaana al. 2222p. However,
they reported that combining synthetic data and the supervised data does not yield significant improvements, consistent with results by [Herzig et al. (2020). By contrast, we introduce new data synthesize procedure for conversational text-to-SQL. dialogues and use it in different way by pretraining language models to induce better representations for many CSP tasks. Our synthesized data can be easily generated without human involvement and the pre-trained models add value to different tasks simultaneously.
6 CONCLUSION
We presented SCORE a new pre-training approach for conversational semantic parsing. The training objectives of SCORE aim to induce natural language representations that capture the multi-tum dynamics, compositional semantic of the target language, and the references to the structural ontology appearing in the dialog. SCORE can be used with many semantic parsing models as a drop-in replacement for general pretrained LMs. We demonstrated SCORE effectiveness by using it as a feature representation encoder with strong baseline models for a wide range of CSP tasks. In particular, our empirical results on four different CSP tasks demonstrated that SCORE can be used to significantly improve the performance of existing strong baseline models by simply replacing an existing pre- trained LM with our SCORE pre- pre-trained model. Furthermore, we are able achieve state-of-te the-art results on three of these tasks. We hope SCORE will encourage further exploration of the benefits and limitations of pre-training approaches for CSP systems.

---

Published as a conference paper at ICLR 2021
REFERENCES
Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. ArXiv, abs/ 111116000000 06450, 2016.
Chandra Bhagavatula, Thanapon Noraset, and Doug Downey. Tabel: Entity linking in web tables. In International Semantic Web Conference, 2015.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. arXiv preprint arXiv:2005. 2400:22000.11155.. 2020.
Pawel Budzianowski. Tsung-Hsiin Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. Multiwoz - large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In EMNLP, 2018.
Giovanni Campagna, Silci Xu, Mehrad Moradshahi, Richard Socher, and Monica S. Lam. Genie: A generator of natural language semantic parsers for virtual assistant commands. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, pp- 394 410. Association for Computing Machinery, 2019.
Giovanni Campagna, Agata Foryciarz. Mehrad Moradshahi, and Monica Lam. Zero-shot transfer learning with synthesized data for multi-domain dialogue state tracking. In Proceedings of 58th Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), 2020.
Joyce Chai, Qiaozi Gao, Lanbo She, Shaohua Yang. Sari Saba-Sadiya, and Guangyue Xu. Language to action: Towards interactive task learning with physical agents. In IJCAI, pp. 2-9, 2018.
Jianpeng Cheng, Siva Reddy, and Mirella Lapata. Building a neural semantic parser from domain ontology. ArXiv, abs/1812. 18... 10037, 2018.
Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACI-HLT, 2019.
Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyag Gao, and Dilek Hakkani-Tur. Multiwoz 2:1: Multi-domain dialogue state corrections and state tracking baselines. arXiv preprint arXiv: 1907.016,,,. 2019.
Shuyang Gao, Abhishek Sethi, Sanchit Agarwal, Tagyoung Chung, and Dilek Hakkani-Tur. Dialog state tracking: A neural reading comprehension approach. In SIGDial, 2019.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't 't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, July 2020. Association for Computational Linguistics.
M. Heck, Carel van Nickerk, Nurul Lubis, Christian Geishauser, Hsien-Chin Lin, M. Moresi, and Milica Gavsi'c. c. Trippy: triple copy strategy for value independent neural dialog state tracking. In SIGdial, 2020.
Matthew Henderson, Blaise Thomson, and Jason D. Williams. The second dialog state tracking challenge. In SIGDIAL Conference, 2014.
Matthew Henderson, lñigo Casanueva, Nikola Mrkvsi c, P. Su, Tsung- Hsien, and Ivan Vulic. Convert: Efficient and accurate conversational representations from transformers ArXiv, 1.00688, 2019.
Dan Hendrycks and Kevin Gimpel. baseline for detecting misclassified and out-of--ditrribttio examples in neural networks. ArXiv, abs/1610.02136,, 2016.
Jonathan Herzig, P. Nowak, Thomas Müller, Francesco Piccinno, and Julian Martin Eisenschlos.
Tapas: Weakly supervised table parsing via pre-training. In ACL, 2020.
Ehsan Hosseini-Asl. B. McCann, Chien-Sheng Wu, Semih Yavuz, and R. Socher, A simple language model for task-oriented dialogue. ArXiv, abs/2005.00723 00796, 2020.
10

---

Published as a conference paper at ICLR 2021
Wonseok Hwang. Jinyeung Yim, Seunghyun Park, and Minjoon Seo. A comprehensive exploration on wikisql with aable-aware word contextualization. ArXiv, abs/1902.00169, 2019.
Mohit Iyyer, Wen-tau Yih, and Ming-Wei Chang. Search-based neural structured learning for the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), Vancouver, Canada, July 2017. Association for Computational Linguistics.
sequential question answering. In Proceedings the
Sina Jafarpour, Christopher Burges, and Alan Ritter. Filter, rank, and transfer the knowledge: Learning to chat. Advances in Ranking, 10:2329- 9290, 2010.
Robin Jia and Percy Liang. Data recombination for neural semantic parsing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), 2016.
Sung-Dong Kim, Sohee Yang, Gyuwan Kim, and Lee. Efficient dialogue state tracking by selectively overwriting memory. In ACL, 2020.
Diederik Kingma and Jimmy Ba. abs/1412.69800 2014.
Adam: A method for stochastic optimization. CoRR,
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising requence-to-sequeen pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, July 2020. Association for Computational Linguistics.
Q. Liu, B. Chen, Jiaqi Guo, Jian-Guang Lou, Bin Zhou, and Dongmei Zhang. How far are we from effective context modeling ? an exploratory study semantic parsing in context. ArXiv, abs/2002.00652, 2020.
Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. K-bert: Enabling language representation with knowledge graph, 2019a.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke S. Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11699 2019b.
Nikola Mrkšić, Diarmuid Ó Séaghdha, Tsung-Hsien Wen, Blaise Thomson, and Steve Young. Neural belief tracker: Data-driven dialogue state tracking. In Proceedings of the 55th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1777-1788. Association for Computational Linguistics, 2017.
Thomas Müller, Francesco Piccinno, Massimo Nicosia, Peter Shaw, and Yasemin Altun. Answering conversational questions on structured data without logical forms. In EMNLP/JCCNPP, 2019.
Arvind Neelakantan, Quoc V. Le, M. Abadi, A. McCallum, and Dario Amodei. Learning natural language interface with neural programmer. ArXiv, abs/161 1,09945, 2017.
Panupong Pasupat and Percy Liang. Compositional semantic parsing Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume I: Long Papers, pp. 1470- 14800 2015.
etni-structured tables. In
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-66) 2020.
11

---

Published as a conference paper at ICLR 2021
Alan Ritter, Colin Cherry, and William B. Dolan. Data-driven response generation in social media. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing. pp. 5-3-583333 Edinburgh, Scotland, July 2011. Association for Computational Linguisties.
Ohad Rubin and Jonathan Berant. Smbop: Semi-auttoegrressive bottom-up semantic parsing. arXiv preprint arXiv:2010. 12412, 2020.
Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. arXiv preprint arXiv:2009.07116 2020.
Pararth Shah, Dilek Hakkani- Tür,, Gokhan Tür, Abhinav Rastogi, Ankur Bapna, Neha Nayak, and Larry Heck. Building a conversational agent overnight with dialogue self-play, 2018.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Technologies, Volume 2 (Short Papers), New Orleans, Louisiana, June 2018. Association for Computational Linguisties.
Peter Shaw, Ming-War Wei Chang, Panupong Pasupat, and Kristina Toutanova. Compositional generaliza- tion and natural language variation: Can semantic parsing approach handle both? arXiv preprint arXiv:2010.11770 2020.
Yu Su, Ahmed Hassan Awadallah, Madian Khabsa, P. Pantel, M. Gamon, and Mark J. Encarnación. Building natural language interfaces web apis. Proceedings of the 2017 ACM Conference Information and Knowledge Management, 2017.
Yibo Sun, Duyu Tang, Nan Duan, Jingjing Xu, X. Feng, and B. Qin. Knowledge-aware conversational semantic parsing over web tables. In NLPCC, 2019a.
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. Emni:: Enhanced representation through knowledge integration, 2019b.
Gokhan Tur and Renato De Mori. Spoken language understanding: Systems for extracting semantic information from speech. John Wiley & Sons, 2011.
Bailin Wang, Ivan Titov, and Mirella Lapata. Learning semantic parsers from denotations with latent structured alignments and abstract programs. In Proceedings of EMNLP, 2019.
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. RAT-SQL: Relation-awant schema encoding and linking for text-to-SQL parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7567-7578, Online, July 2020. Association for Computational Linguistics. doi: 10. 20.18653/v1/202000 aacl--llmmm -main.677.
Yushi Wang, Jonathan Berant, and Percy Liang. Building a semantic parser overnight. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th Interna- tional Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1342, Beijing, China, July 2015. Association for Computational Linguistics. doi: 10.3115/v1/PM 1/11111/1111111199
Soung-Hsien Wen, David Vandyke, Nikola Mrksic, Milica Gasic, Lina M Rojas-Barahnn Pei-Hao Su, Stefan Ultes, and Steve Young. A network- bbaed end-to-end trainable task-oriented dialogue system. arXiv preprint arXiv:1604 100..0466.0 04562, 2016.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's 's transformers: State-of-the-ar natural language processing. ArXiv, abs/1910.03771. 2019.
Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. Transferable multi-domain state generator for task-oriented dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. Association for Computational Linguistics.
Chien-Sheng Wu, Steven C.H. Hoi, Richard Socher, and Caiming Xiong. TOD-BERT: Pre-trained natural language understanding for trikntrie dialogue. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
12

---

Published as a conference paper at ICLR 2021
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. TaBERT: Pretraining for joint understanding of textual and tabular data. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics. pp. 8413-84488 8426, Online, July 2020. Association for Computational Linguistics. doi: 1...8665/11/222.aa. ma..
Kang Min Yoo, Youhyun Shin, and Sang goo Lee. Data augmentation for spoken language under- standing via joint variational generation, 2018.
Tao Yu, Michihiro Yasunaga, Kai Yang. Rui Zhang, Dongxu Wang. Zifan Li, and Dragomir Radev. Syntaxsqlnet: Syntax tree networks for complex and cross-domain text-to-sql task. In Proceedings of EMNLP. Association for Computational Linguisties, 2018a.
Tao Yu, Rui Zhang, Kai Yang. Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In EMNLP, 2018b.
Tao Yu, Rui Zhang, He Yang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Chern Tan, Tianze Shi, Zihan Li, Youxuan Jiang, Michihiro Yasunaga, Sungrok Shim, Tao Chen, Alexander Fabbri, Zifan Li, Luyao Chen, Yuwen Zhang, Shreya Dixit, Vincent Zhang, Caiming Xiong, Richard Socher, Walter Lasecki, and Dragomir Radev. Cosql: A conversational text-to-sql challenge towards cross-domain natural language interfaces to databases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing. Association for Computational Linguistics, 2019a.
Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li, Irene Li Heyang Er, Bo Pang, Tao Chen, Emily Ji, Shreya Dixit, David Proctor, Sungrok Shim, Vincent Zhang Jonathan Kraft, Caiming Xiong, Richard Socher, and Dragomir Radev. Sparc: Cross-domain semantic parsing in context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, 2019b. Association for Computational Linguistics.
Jian-Guo Zhang, Kazuma Hashimoto, Chien-Sheng Wu, Yao Wan, Philip S Yu, Richard Socher, and Caiming Xiong. Find or classify? dual strategy for slot-value predictions on multi-domain dialog state tracking. arXiv preprint 1:XX::1900003444 2019a.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. Pegasus: Pre-training with extracted 2019b.
gap-sentences for abstractive summarization. arXiv preprint
Rui Zhang. Tao Yu, He Yang Er, Sungrok Shim, Eric Xue, Xi Victoria Lin, Tianze Shi, Caiming Xiong, Richard Socher, and Dragomir Radev. Editing- based sql query generation for cross-domain context-dependent questions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing. Association for Computational Linguistics, 2019c.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. Dialogpt: Large-scale generative pre-training for conversational response generation. In ACL, system demonstration, 2020.
Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, abs/1709.00103, 2017.
Victor Zhong, M. Lewis, Sida I. Wang, and Luke Zettlemoyer. Grounded adaptation for zero-shot executable semantic parsing. The 2020 Conference on Empirical Methods in Natural Language Processing, 2020.
13

---

Published as a conference paper at ICLR 2021
A DETAILED RESULTS
QM
IM
Q1
Q2
Q3
Q4
Q5
RAT-SQL BERT
48.4
19.1
54.6
48.4
47.5
43.9
31.0
+RoBERTa + SCORE
50.1
19.3
59.7
50.9
46.3
46.5
32.4
52.1
22.0
60.8
53.0
47.5
49.1
32.4
Table 10: Detailed results of CoSQL the dev set. Q is the accuracy of the 7th question in the conversation.
QM
IM
Q1
Q2
Q3
Wang et (0199
51.0
22.0
68.3
48.0
38.5
+RoBERIa
62.8
33.2
77.2
61.7
52.1
+SCORE
65.4
38.5
78.4
65.3
55.1
Few-Shot (10% training data)
Wang et al. (2019)
+RoBERIa +SCORF
53.3
21.2
71.0
52.5
36.6
57.1
26.7
74.6
56.7
40.7
Table 11: Detailed results of SQA on the test set. Q is the accuracy of the zth question in the conversation.
B SYNTHESIZED EXAMPLES & TEMPLATES
Table 12 shows example of the synthesized question-SQL. pairs and their corresponding templates in our grammars.
Turn Question-SQQ Template
Synthesized Question-SQL
"Find the number TABLEO with COLUMNO OPO VALUEÜ"
"FFind number football with hometown is murrieta, california?"
SELECT COUNT(") ORDER BY COLUMNO OPO VALUED
SELECT COUNT(") WHERE TEAM HOMETOWN 1= "MURRIETA, CALIFORNIA"
"Can you give me their COLUMN 1?" EEPLACC((ELECT.COLLIM DEL (SELECT.AGG)
"Can you give me their football team player?" SELECT FOOTBALL_TEAM, PLAYER != "MURRIETA, CALIFORNIA"
TCS:
TEAM_HOMETOWN
"How about only show those with ASO COL- UMN2?"
"How about only show those with the largest age?"
SELECT FOOTRALI_TEANE P_LAYERR WHERE TRA MOHOMMTOWN
TCS: ADD(ORDERBY.AR) WWRDDDERRBBASSO.CCOM ASO.COLUUO2C))
"MURRIETA, CALIFORNIA" ORDER BY AGE DESC LIMIT
"AS1?" 1
"The smallest?"
TCS: REPLACE(ORDERRYY... COLLUN)))
SELECT BENTBALLLTEMM.PLLTE WHERE TEAMLHOMETOWN MURRIETTA,
LIMIT
Table 12: An example of synthetic conversational text-to-SQL data.
C IMPLEMENTATION DETAILS
C.1 SCORE
For pre-training SCORE on synthesized text-to-SQL data, we use ROBERTA large and pre-train with batch size 12, gradient accumulation step 2, and maximum length 248. We use learning rate 1e-5 and gradually reduce the learning rate without a warm-up period using Adam (Kingmal & Ba 2014) with epsilon 1e-8, BERT base is used in pre-training SCoRE on synthesized MWoZ data because contains longer conversations. We set the maximum length to 512 and batch size 24. All SCoRE pre-trained for 30 epochs, which usually take less than half a day 8 V V100 GPUs.
14

---

Published as a conference paper at ICLR 2021
We experimented with SCORE pre- pre-trained for 5, 10, and 30 epochs and found that most of the best downstream performances occur when base systems incorporate with SCORE pre-traiii trained for less than 10 epochs. Our implementation is based on the Transformers library (Wolf et al. 2019).
C.2 BASE MODELS
For a fair comparison, all RAT-SQL experiments are trained for 40k steps. We adopt the same hyperparameters as Shaw et al. (2018) except for learning rates. We find that learning rates of le-4 and 1e-5 for RAT and BERT respectively produce more stable results.
RAT-SQL:
TripPy: We use the same hyperparameters for training TripPy on MWoZ as in (Heck et al. 2020)
except train it for 25 epochs (as opposed to 10 epochs reported in (Heck a 2020)). When we train TripPy for 25 epochs, we get new result that is higher (around 2%) than the one reported in (Heck et al. 2020). Similarly, when we train TripPy with SCORE, train it for 25 epochs.
SOM-DST: We use the same hyperparameters from Kim et al. (2020) for all SOM-DST experiments on MWoZ.
Wang ef al. 2 ; We use the same hyperparameters from Wang et al. 2019 for SQA experiments.
Note that Herzig al. (2020) outperform Wang et al. (2119) SQA because (1) they don't generate
logic forms but select table cells and applying aggregation operators. Waag et al. (2019) generate latent programs, yet the grammar of the latent program can only cover 87% questions. (2) They
reduce the search space by reusing the previous question answer. We choose Wang et al. (2019) as
our base model because generating symbolic programs has many practical advantages (even at cost of around 1% accuracy drop), such as showing interpretable reasoning steps, enabling formal reasoning, and operationalization without GPU/TPU accelerators.
D PRE-TRAINING COST
We test the performance of SCORE with respect to the number of pre-training epochs. Figure 3
Intersttoon eoursr CQM or 201)
shows that the best performance of the down- stream tasks is usually achieved in early epochs, more specifically for SPARC and CoSQL and 15 for oZ Longer pre-training time does not improve or even hurts the performance. One possible reason that longer pre-training makes
59
SCORE overfit to the synthesized data whose utterances are unnatural.
pre-training
As for the data, as shown in Table 5 even if SCORE pre-trained with only a relatively small amount of synthesized data (without the MI. loss), most of the tasks can achieve much higher performances. With a relatively smaller training corpus and shorter training time compared to other pre-trained language models, SCoRE is efficient in time and data.
Figure 3: The effect of pre-training time.
E ADDITIONAL RESULTS
Effect TCS We ran the TCS only experiment on SPARC, and will add TCS only results (including for other tasks) to Table in the final version. SCORE (TCS only) outperforms RoBERTa by 2.4% so far (note: training still going on) on SPARC (39.1% vs. 36.7%). Also, as discussed in Section 4 we also provide a secondary evidence by testing the effectiveness of TCS on SPARC, CoSQL, and SQA by adding TCS CCS (CCS only vs. CCS+TCS), SCORE (with TCS) gains improvements of 1.2% on SPARC and 0.8% on COSQL, and 4.4% on SQA.
Incorporating Additional Examples Used in Synthetic Grammar Induction As we mentioned in Section 2.3
used about 500 examples from SPARC to induce the grammar for data synthesis in pre- ure-training. For a fair comparison, we also report the results of incorporating the additional SPARC examples in CoSQL and SQA. More specifically, directly concatenate the additional SPARC
15

---

Published as a conference paper at ICLR 2021
examples CoSQL training set, and train RAT-SOL-ROBERTA on it, which slightly improves the performance (19.6% vs. 19.3%) but not as large as SCORE (22.0% vs. 19.3%).' Also, because SQA wuakly-supervised sequential question answering, which differs from SPARC, we first fine-tune ROBERTA on the additional SPARC examples using CCS, and then apply it to SQA. this way, the ROBERTA trained with additional SPARC examples achieves similar performance the original one (62.7% vs 62.8%).
Performance Comparison with ToD-BERT ToD-BERT is pre-trained on human-annotated ques- tions with both MIL and response contrastive objectives. To compare TOD-BERT with SCORE, we ran experiments of RAT-SQL. ToD-BERT on SPARC. SCoRE (62.2%) outperforms ToD-BERT (54.6%) by 7.6%.
Comparison with Finetuning Larger Language Models Based on our experiments and other published results, we didn't find existing larger LMs (BART (Lewis et al. 2020), T5 (Raffel et al.. 2020), GPT-2 (Radford et al., 2019) outperform custom models + BERT on CSP tasks. Our evidence based on Spider Yuu al. 2018b), which is the single-turn version of SParC and CoSQL. For T5, Shaw et al. (2020) applied 15 as seq2seq to Spider, and compared with RAT-SQL BERT-Large T5-Base performs much worse (57.1% vs. 69.6%). and T5-3B improves only 0.3, but is 6 times larger. Moreover, for Bart, we have performed experiments on Spider and we found that BART cannot outperform custom models + BERT: RAT-SQL + BERT 69.7%, RAT-SQL + BART encoder 67.8%, BART encoder decoder (406M, seq2seq task) 62.4%. In Rubin & Berant (2020). BART didn't 't outperform BERT either. As for GPT-2. Wu et al. (2220) and Hosseini-Ae Asl et al. (2020) found it does not outperform BERT MWoZ.
F TASK-ORIENTED DIALOGUE DATASETS
Name
Dialogue
Utterance
Åvg. Turn
# Domain
MetaLWOZ (Lee 2019)
37,884
432,036
11.4
Schema (Rasingi al., 2019)
22,825
463,284
20.3
17
Taskmaster (Byme 2019)
13.215
303,066
22.9
MWOZ (Budzianowski 2018)
10,420
71,410
6.9
MSR ESE (Li al., 2018)
10,087
74,686
7.4
SMD (Eric and Manning, 2017)
3,031
15,928
5.3
Frames (Asri et al., 2017)
1,369
19,986
14.6
WOZ (Mrkšić 2016)
1,200
5,012
4.2
CanRest676 (Wen al., 2016)
676
2,744
4.1
Figure 4: Data statistics of -amaan-nnnotttd task-oriented dialogue datasets used in Wu et al. (020).
16


Reviews:
[{"review_id": "slZZXsyZLa", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper introduces a semantic-parsing specific pretraining objective. The authors argue that general pre-training methods such as BERT do not have enough inductive bias for semantic parsing.\nSince a lot of data doesn’t exist for semantic parsing datasets,  the authors use synthetic data to better adapt to the ontology of the task. They use this pre-trained model on 4 semantic parsing tasks and show that their pre-training is indeed helping the SOTA models by establishing new SOTA for 3 of the 4 datasets.\n\nReasons for score: \nThis is a clearly written paper with the objective of making conversational semantic parsing better. The authors present a reasonable idea, do extensive experiments to show its merit on variety of tasks. The pre-trained checkpoints released by this paper will be useful to the community as a whole.\n\nPros:\n1. 2 new objective functions (TCS and CCS) specific to the task of semantic parsing. Authors clearly show that both those objectives help downstream tasks\n2. The paper was easy to follow with clear objectives.\n3. Strong performance of their pre-trained model which improves over previous SOTA\n4. Well done ablation studies and analysis.\n\nCons:\n1. Data synthesis steps are ad-hoc. Why were only 435k dialogues synthesized? It would be great to have a more detailed study of how the synthetic data looks, what is the effect on model performance.\n2. Due to synthetic data the method is not very general. Training larger models with more data will be non trivial to do.\n3. While the authors chose SOTA baselines for their task, stronger general pre-trained models (BART, T5 etc) might beat this method easily.\n\nPlease address and clarify the cons above \n\nTypos/Areas for improvement:\n1. Citations in table 3\n2. Mistake in Table 1 Multiwoz2.1 is multi domain.\n3. The number of utterances are not clear: 3.1 -> vanilla multiwoz2.1 does not have over 100k task oriented dialogues\n4. 3.1 SQA description not clear.\n5. In section 2.2 it would be great to have a clearer description of the steps for pre-training, it's hard to tease out the exact steps taken\n6. For table 7, what happens when you use the synthetic tuning data to first train and then fine-tune on the original task.\n7. For table 8: More studies on the generalizability would be good\n    1. If you simply drop in the network to something small like geo-query, does it help a general seq2seq model?\n    2. Beyond semantic parsing tasks - does this help in sentence classification too?", "extra_scores": {}, "extra_text": {}}, {"review_id": "rxQbNWIQsdA", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Summary]\nIn this paper, the authors proposed a pre-training strategy for Conversational Semantic Parsing (CSP) tasks. The pre-training is run on top of any existing LM (i.e., in this work RoBERTA has been used), and uses three additional loss functions to inject the CSP inductive bias into the LM: Column Contextual Semantics (CCS), Turn Contextual Switch (TCS) and Masked Language Modeling (MLM). Moreover, the authors proposed to use synthetically generated data in the pretraining. The results are presented in four well-know datasets for CSP: SPARC, COSQL, MWOZ, and SQA. \n\n[Pros]\n- the proposed pre-training strategy is novel\n- the performance of the proposed pre-training strategy is effective \n\n[Cons-Edited]\n- the paper claims that \"However, existing pre-trained LMs that use language modelling training objectives over free-form text have limited ability to represent natural language references to contextual structural data.\", there the authors have not compared the proposed strategy with large pre-trained LMs, especially for Seq2Seq training (e.g., BART, T5, GPT-2) and larger versions of these models. Independently by the number of parameters, which for instance has never been mentioned in the paper, LM trained on text-only can achieve similar or better performance in CSP (Check Author Response Comments), without the need of task-specific pre-trained (i.e. CCS and TCS).    \n- although few samples, only 500 samples or the dev set only, are used for generating the synthetic data, some of the datasets are used for the pre-training strategy. Moreover, there is a substantial human bias in the construction of the synthetic data, for instance, to create these data probably a human would need to read way more than 500 samples, or even with 500 samples, a human can pretty much guess the distribution of the data, especially for a grammar-based generation as SQL\n- the comparison made in the paper are over existing model instead over existing pre-training strategy, or larger models. \n\n[Minor-Cons]\n- In Eq. (1) the \"SCORE\" function is actually a RoBERTA encoder, if I understood correctly, else, this function is not defined anywhere. Why not using  RoBERTA or LM instead? \n- In Eq. (1) there is a typo I guess $v_t$ should be $h_t$\n- The explanation of the two pre-training loss CCS and TCS is very hard to understand, and Figure 2 doesn't help. I suggest showing more examples. \n\n\n[Reason to reject] The claim of the paper is not supported for lack of comparisons with different, larger and using different pre-training strategies, LMs. Moreover, the community should be encouraged to create as general as possible pre-trained models, instead of task-specific ones, and especially pre-trained models that use real and unlabeled data. \n\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "qDvZFFZi1hR", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview: the authors propose a new pre-training method for grounding LM representations in both structural/schematic information and also dialogue context. To explore the efficacy of this method, they experiment in 4 different conversational semantic parsing tasks, which are each different enough to demonstrate the usefuleness of their approach. \n\nContribution: modified pre-training method \n\nThe good: My overall impression is that this work makes sense, the paper is clearly written and flows just fine, and that the authors demonstrated the efficacy of their proposed method. The fact that the 4 baselines are different enough makes your claim convincing. \n\nThe bad: I feel that this paper is really lacking a driving motivation and the cohesive story is a bit weak/lacking. For example, I felt the Related Works section was very superficial, rather than contributing. Shouldn't the discussion/conversation be more about contextualization methods?  You are not the first to try contextualizing schema elements, for example. I think the paper could be significantly improved if the conversation is shifted towards discussion of contextualization, rather than \"CSP is a task. People use pre-training. You can generate some data and use it\". You have an interesting story in your hands, but you are not using it! How is your contextualization method different from other attempts? In what ways is it better, in what ways is it worse? That said, I still think the paper has good and interesting work, and I do think it should be accepted, but I would really encourage the authors to consider making some changes on this note because it will make the work much more interesting, thought provoking, and impactful. \n\nSmall things:\n* unless i click on the box to jump to Table 1 on the PDF, it took me some time to find Table 1 when I printed it out. Is there anyway to move this closer to where you point to it, or sepperate it from the figure below? \n\nClarifying question:\n* With regards to encoding the Turn Contextual Switch -- are you encoding the changes from user to agent, or just the changes from user to user, or both? Is it always user to user query changes, because some datasets have no system response? This is the only section in the paper that I found to be lacking a bit, and maybe a sentence or two more could add clarification for me. \n\nOverall:\nGood work :-) \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5O9wmJydn_", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to pretrain contextual semantic parsing models on synthesized data with two new training objectives: Column Contextual Semantics (CCS) and Turn Contextual Switch (TCS). The CCS objective predicts correct database operations based on corresponding columns in tables. The TCS aims to predict the labels of conversational turn switch patterns categorized based on differences in meaning representations between dialogue turns. The synthetic data is generated by apply two utterance-SQL generation grammars. They show that the new approach significantly outperforms te baselines on Sparc, CoSQL, and MultiWOZ.\n \nMy decision is just between marginally accepted and marginally rejected. I like the idea of pretraining with CCS and the empirical results show that the proposed approach outperforms all baseline systems on three out of four benchmark datasets. However, my major concern is the usefulness of TCS.\n \nPros: \n \n1. The paper finds out that CCS and synthetic data works in pre-training, despite the prior work finds synthetic data is not useful in a standard supervised setting. \n\n2. Overall, the paper is well written. I can easily follow the technical details of the proposed methods. \n\n3. This paper provides comprehensive experiments to justify the key contributions of this paper. The ablation study helps understand which technique works on the selected datasets. \n\nCons: \n \n1. The usefulness of TCS objective needs further justification. I suggest adding experiments of TCS only to Table 6. The TCS objective needs further investigation to understand in which cases it works.\n\n2. It is desirable to have an ablation study to investigate how effective is each grammar (with/without follow-up context-free grammar) for pretraining.\n\nQuestions:\n1. Could you explain why the proposed method works worse than herzig et al. (2020b) on SOA?\n \n2. Why does the system compare with the baselines also trained on 10\\% training data of SOA?", "extra_scores": {}, "extra_text": {}}, {"review_id": "1zLXUYZTfFb", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a pre-training approach to improve the performance in conversational semantic parsing.  The idea is to use the training data to learn how to generate contextual representations by combining the now commonly used masked language modelling pretraining objective (MLM) with two additional objectives, named column contextual semantics and turn contextual switch. Furthermore, additional synthetic data was generated.\n\nI found the paper easy to follow and the results convincing on the whole. 4 datasets and different parsers were used and in each case the proposed method improved the results, and in 3 out of 4 a new SOTA was reached. \n\nHowever, it is important to note that the objective propose needs labeled training data in order to learn the alignment between the utterances and the queries. Thus they allow us to exploit labeled data for other versions of the task, rather than exploit unlabelled data for the semantic parsing. Therefore, while I like the paper, I think a key comparison missing is to compare against training existing models using combinations of the datasets considered, to allow the models access to the same training data. This could be easy incases where the output is of the same form, e.g. SQL for SPARC and COSQL, or it could be multi-task training to combine it with the datasets fo the other two tasks. As things stand, the approach proposed indeed brings benefits, but it is only compared against methods using no additional labeled data beyond what is built for the task at hand. Having said this, I am not arguing that training data concatenation or multi-task training will work better; but I think such a comparison is needed.\n\nBeyond this, the other aspect of the paper that needs to be improved is the technical clarity. While I found the intuitive descriptions of equations 2 and 3 easy to understand, the equations themselves were not. In both cases the formal representation q (e.g. the SQL query) while it is needed to calculate the objective, it is not in the equations. Thus it would be very easy to have it interpreted differently and thus lead to ambiguity and people not being able to reproduce it. This is particularly important as how one does the decomposition of q is likely to matter to the results.\n\nPost-author response: I appreciate that the extra experiment I asked for was conducted and the equations were fixed. While I think Review5 raised some interesting discussion points which should be included in the final version, I still think the paper has merit, even if larger-scale pre-training would have improved the results. Thus I raised my score to 7.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to pre-train contextual semantic parsing models on synthesized data (using a small amount of additional supervised training data and grammar-based generalizations therefrom) with two new training objectives: Column Contextual Semantics (CCS), mapping text to database columns, and Turn Contextual Switch (TCS), to deal with the update semantics between turns. 

I thank the reviewers for their detailed engagement with this paper, and thanks the authors for their responsiveness in doing extra experiments and rewriting that made this paper better and the decision clearer.

Pros

The authors did such a great job of summarizing the pros, that I think I can just copy their summary: "We are glad that the reviewers appreciate the novelty and the effectiveness of our proposed approach (R5), find our experiments to be comprehensive and convincing by achieving SOTA on 3 out of 4 different tasks (R1, R2, R3, R4), ablation studies and analysis to be informative and well done (R2, R4), and think our paper is clearly written and easy to follow (R1, R2, R3, R4)."

Cons

- A somewhat specific and ad hoc data synthesis solution
- Stronger pre-trained contextual language models might beat assumed baselines or methods shown here (R4, R5)
- The story is weak and should be better motivated through discussion of contextualization of interpretation

In general the reviewers recommend accepting the paper, and I agree. However, it is perhaps not of the novelty, clarity, or impact size to qualify for more than a Poster. R5 has a good point about how strong pre-trained LMs are a general tool and should be preferred to the extent they work in 2020, but I think they are too opinionated to suggest this is a reason for rejection. Along with the other reviewers and the authors, I think it is most reasonable to accept work showing good progress using "medium-sized" pre-trained LMs -- really we thought BERT was big a couple of years ago! -- and this work has comprehensive experiments with good results. I would encourage the authors:

- To say more about the alternative strategy of instead using a bigger pre-trained LM, as has come out in the discussion on OpenReview, and the pros and cons of this approach (though maybe the results with BART are the only fairly comparable data point)
- To strengthen the presentation by orienting the paper more around the importance of contextualization in interpreting dialog turns in conversational semantic parsing (as opposed to the "one turn" nature of the original famous semantic parsing datasets).

p.s. One typo I noticed in the revised paper while reading: fours --> four

Decision: Accept (Poster)