Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: [Summary]
In this paper, the authors proposed a pre-training strategy for Conversational Semantic Parsing (CSP) tasks. The pre-training is run on top of any existing LM (i.e., in this work RoBERTA has been used), and uses three additional loss functions to inject the CSP inductive bias into the LM: Column Contextual Semantics (CCS), Turn Contextual Switch (TCS) and Masked Language Modeling (MLM). Moreover, the authors proposed to use synthetically generated data in the pretraining. The results are presented in four well-know datasets for CSP: SPARC, COSQL, MWOZ, and SQA. 

[Pros]
- the proposed pre-training strategy is novel
- the performance of the proposed pre-training strategy is effective 

[Cons-Edited]
- the paper claims that "However, existing pre-trained LMs that use language modelling training objectives over free-form text have limited ability to represent natural language references to contextual structural data.", there the authors have not compared the proposed strategy with large pre-trained LMs, especially for Seq2Seq training (e.g., BART, T5, GPT-2) and larger versions of these models. Independently by the number of parameters, which for instance has never been mentioned in the paper, LM trained on text-only can achieve similar or better performance in CSP (Check Author Response Comments), without the need of task-specific pre-trained (i.e. CCS and TCS).    
- although few samples, only 500 samples or the dev set only, are used for generating the synthetic data, some of the datasets are used for the pre-training strategy. Moreover, there is a substantial human bias in the construction of the synthetic data, for instance, to create these data probably a human would need to read way more than 500 samples, or even with 500 samples, a human can pretty much guess the distribution of the data, especially for a grammar-based generation as SQL
- the comparison made in the paper are over existing model instead over existing pre-training strategy, or larger models. 

[Minor-Cons]
- In Eq. (1) the "SCORE" function is actually a RoBERTA encoder, if I understood correctly, else, this function is not defined anywhere. Why not using  RoBERTA or LM instead? 
- In Eq. (1) there is a typo I guess $v_t$ should be $h_t$
- The explanation of the two pre-training loss CCS and TCS is very hard to understand, and Figure 2 doesn't help. I suggest showing more examples. 


[Reason to reject] The claim of the paper is not supported for lack of comparisons with different, larger and using different pre-training strategies, LMs. Moreover, the community should be encouraged to create as general as possible pre-trained models, instead of task-specific ones, and especially pre-trained models that use real and unlabeled data. 





review_title: Review (Edited after comments)

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes to pretrain contextual semantic parsing models on synthesized data with two new training objectives: Column Contextual Semantics (CCS) and Turn Contextual Switch (TCS). The CCS objective predicts correct database operations based on corresponding columns in tables. The TCS aims to predict the labels of conversational turn switch patterns categorized based on differences in meaning representations between dialogue turns. The synthetic data is generated by apply two utterance-SQL generation grammars. They show that the new approach significantly outperforms te baselines on Sparc, CoSQL, and MultiWOZ.
 
My decision is just between marginally accepted and marginally rejected. I like the idea of pretraining with CCS and the empirical results show that the proposed approach outperforms all baseline systems on three out of four benchmark datasets. However, my major concern is the usefulness of TCS.
 
Pros: 
 
1. The paper finds out that CCS and synthetic data works in pre-training, despite the prior work finds synthetic data is not useful in a standard supervised setting. 

2. Overall, the paper is well written. I can easily follow the technical details of the proposed methods. 

3. This paper provides comprehensive experiments to justify the key contributions of this paper. The ablation study helps understand which technique works on the selected datasets. 

Cons: 
 
1. The usefulness of TCS objective needs further justification. I suggest adding experiments of TCS only to Table 6. The TCS objective needs further investigation to understand in which cases it works.

2. It is desirable to have an ablation study to investigate how effective is each grammar (with/without follow-up context-free grammar) for pretraining.

Questions:
1. Could you explain why the proposed method works worse than herzig et al. (2020b) on SOA?
 
2. Why does the system compare with the baselines also trained on 10\% training data of SOA?
review_title: TCS needs further investigation

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a pre-training approach to improve the performance in conversational semantic parsing.  The idea is to use the training data to learn how to generate contextual representations by combining the now commonly used masked language modelling pretraining objective (MLM) with two additional objectives, named column contextual semantics and turn contextual switch. Furthermore, additional synthetic data was generated.

I found the paper easy to follow and the results convincing on the whole. 4 datasets and different parsers were used and in each case the proposed method improved the results, and in 3 out of 4 a new SOTA was reached. 

However, it is important to note that the objective propose needs labeled training data in order to learn the alignment between the utterances and the queries. Thus they allow us to exploit labeled data for other versions of the task, rather than exploit unlabelled data for the semantic parsing. Therefore, while I like the paper, I think a key comparison missing is to compare against training existing models using combinations of the datasets considered, to allow the models access to the same training data. This could be easy incases where the output is of the same form, e.g. SQL for SPARC and COSQL, or it could be multi-task training to combine it with the datasets fo the other two tasks. As things stand, the approach proposed indeed brings benefits, but it is only compared against methods using no additional labeled data beyond what is built for the task at hand. Having said this, I am not arguing that training data concatenation or multi-task training will work better; but I think such a comparison is needed.

Beyond this, the other aspect of the paper that needs to be improved is the technical clarity. While I found the intuitive descriptions of equations 2 and 3 easy to understand, the equations themselves were not. In both cases the formal representation q (e.g. the SQL query) while it is needed to calculate the objective, it is not in the equations. Thus it would be very easy to have it interpreted differently and thus lead to ambiguity and people not being able to reproduce it. This is particularly important as how one does the decomposition of q is likely to matter to the results.

Post-author response: I appreciate that the extra experiment I asked for was conducted and the equations were fixed. While I think Review5 raised some interesting discussion points which should be included in the final version, I still think the paper has merit, even if larger-scale pre-training would have improved the results. Thus I raised my score to 7.
review_title: Official Blind Review #3 (Edited post author response period)

final_score: rating: 7: Good paper, accept
review_content: Overview: the authors propose a new pre-training method for grounding LM representations in both structural/schematic information and also dialogue context. To explore the efficacy of this method, they experiment in 4 different conversational semantic parsing tasks, which are each different enough to demonstrate the usefuleness of their approach. 

Contribution: modified pre-training method 

The good: My overall impression is that this work makes sense, the paper is clearly written and flows just fine, and that the authors demonstrated the efficacy of their proposed method. The fact that the 4 baselines are different enough makes your claim convincing. 

The bad: I feel that this paper is really lacking a driving motivation and the cohesive story is a bit weak/lacking. For example, I felt the Related Works section was very superficial, rather than contributing. Shouldn't the discussion/conversation be more about contextualization methods?  You are not the first to try contextualizing schema elements, for example. I think the paper could be significantly improved if the conversation is shifted towards discussion of contextualization, rather than "CSP is a task. People use pre-training. You can generate some data and use it". You have an interesting story in your hands, but you are not using it! How is your contextualization method different from other attempts? In what ways is it better, in what ways is it worse? That said, I still think the paper has good and interesting work, and I do think it should be accepted, but I would really encourage the authors to consider making some changes on this note because it will make the work much more interesting, thought provoking, and impactful. 

Small things:
* unless i click on the box to jump to Table 1 on the PDF, it took me some time to find Table 1 when I printed it out. Is there anyway to move this closer to where you point to it, or sepperate it from the figure below? 

Clarifying question:
* With regards to encoding the Turn Contextual Switch -- are you encoding the changes from user to agent, or just the changes from user to user, or both? Is it always user to user query changes, because some datasets have no system response? This is the only section in the paper that I found to be lacking a bit, and maybe a sentence or two more could add clarification for me. 

Overall:
Good work :-) 

review_title: Interesting work, would benefit from improved discusison 

final_score: rating: 7: Good paper, accept
review_content: Summary:
This paper introduces a semantic-parsing specific pretraining objective. The authors argue that general pre-training methods such as BERT do not have enough inductive bias for semantic parsing.
Since a lot of data doesn’t exist for semantic parsing datasets,  the authors use synthetic data to better adapt to the ontology of the task. They use this pre-trained model on 4 semantic parsing tasks and show that their pre-training is indeed helping the SOTA models by establishing new SOTA for 3 of the 4 datasets.

Reasons for score: 
This is a clearly written paper with the objective of making conversational semantic parsing better. The authors present a reasonable idea, do extensive experiments to show its merit on variety of tasks. The pre-trained checkpoints released by this paper will be useful to the community as a whole.

Pros:
1. 2 new objective functions (TCS and CCS) specific to the task of semantic parsing. Authors clearly show that both those objectives help downstream tasks
2. The paper was easy to follow with clear objectives.
3. Strong performance of their pre-trained model which improves over previous SOTA
4. Well done ablation studies and analysis.

Cons:
1. Data synthesis steps are ad-hoc. Why were only 435k dialogues synthesized? It would be great to have a more detailed study of how the synthetic data looks, what is the effect on model performance.
2. Due to synthetic data the method is not very general. Training larger models with more data will be non trivial to do.
3. While the authors chose SOTA baselines for their task, stronger general pre-trained models (BART, T5 etc) might beat this method easily.

Please address and clarify the cons above 

Typos/Areas for improvement:
1. Citations in table 3
2. Mistake in Table 1 Multiwoz2.1 is multi domain.
3. The number of utterances are not clear: 3.1 -> vanilla multiwoz2.1 does not have over 100k task oriented dialogues
4. 3.1 SQA description not clear.
5. In section 2.2 it would be great to have a clearer description of the steps for pre-training, it's hard to tease out the exact steps taken
6. For table 7, what happens when you use the synthetic tuning data to first train and then fine-tune on the original task.
7. For table 8: More studies on the generalizability would be good
    1. If you simply drop in the network to something small like geo-query, does it help a general seq2seq model?
    2. Beyond semantic parsing tasks - does this help in sentence classification too?
review_title: Well written paper introducing a pre-training objective specifically for semantic parsing.

Meta-review: This paper proposes to pre-train contextual semantic parsing models on synthesized data (using a small amount of additional supervised training data and grammar-based generalizations therefrom) with two new training objectives: Column Contextual Semantics (CCS), mapping text to database columns, and Turn Contextual Switch (TCS), to deal with the update semantics between turns. 

I thank the reviewers for their detailed engagement with this paper, and thanks the authors for their responsiveness in doing extra experiments and rewriting that made this paper better and the decision clearer.

Pros

The authors did such a great job of summarizing the pros, that I think I can just copy their summary: "We are glad that the reviewers appreciate the novelty and the effectiveness of our proposed approach (R5), find our experiments to be comprehensive and convincing by achieving SOTA on 3 out of 4 different tasks (R1, R2, R3, R4), ablation studies and analysis to be informative and well done (R2, R4), and think our paper is clearly written and easy to follow (R1, R2, R3, R4)."

Cons

- A somewhat specific and ad hoc data synthesis solution
- Stronger pre-trained contextual language models might beat assumed baselines or methods shown here (R4, R5)
- The story is weak and should be better motivated through discussion of contextualization of interpretation

In general the reviewers recommend accepting the paper, and I agree. However, it is perhaps not of the novelty, clarity, or impact size to qualify for more than a Poster. R5 has a good point about how strong pre-trained LMs are a general tool and should be preferred to the extent they work in 2020, but I think they are too opinionated to suggest this is a reason for rejection. Along with the other reviewers and the authors, I think it is most reasonable to accept work showing good progress using "medium-sized" pre-trained LMs -- really we thought BERT was big a couple of years ago! -- and this work has comprehensive experiments with good results. I would encourage the authors:

- To say more about the alternative strategy of instead using a bigger pre-trained LM, as has come out in the discussion on OpenReview, and the pros and cons of this approach (though maybe the results with BART are the only fairly comparable data point)
- To strengthen the presentation by orienting the paper more around the importance of contextualization in interpreting dialog turns in conversational semantic parsing (as opposed to the "one turn" nature of the original famous semantic parsing datasets).

p.s. One typo I noticed in the revised paper while reading: fours --> four

Decision: Accept (Poster)