Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Pros
•	This paper incorporates the multi-domain domain-slot pairs into the bert input so that the relations between sentences, domain-slot are modeled.
Cons
•	It’s better to experiment on more dataset to prove the method’s effectiveness.
Comments
•	This paper   https://arxiv.org/abs/2006.01554 seems get better performance on the same dataset.
•	In the domain split/merge method, does the domain/slot used as vocabulary tag or word embedding?
•	The slot-gate classifier’s output is fed into the slot-value classifier, how does this affect the performance?

review_title: This paper studies the problem of multi-domain dialogue state tracking which is challenging.  It proposes a bert based model for multi-domain dialogue that effectively models the relationship among domain-slot pairs.  The experimental results show the model achieve sota performance on the MultiWOZ-2.1 dataset.

final_score: rating: 3: Clear rejection
review_content: Summary:

This paper proposed a new approach for modeling multi-domain dialogue state tracking by incorporating domain-slot relationship using a pre-trained language encoder. The proposed approach are based on using special tokens to mode l such relationship. Two kinds of special tokens are proposed to represent domain-slot pair, DS_merge token for each specific pair, and tokens for every domain and slots separately 



Comments:

1- what is the role of segment embedding (Fig3) for DST tank? Are two different segment used in the pretraining of model?

2- Table 1:
                  2-1 what type of label cleaning is used to compute joint goal accuracy? From SimpleTOD paper, each baseline has used different label cleaning
                  2-2: DS-split is bolded, while it is lower than SimpleTOD

3- Table 2: the results indicate that DST performance drops in total by blocking attention across different DS tokens. However, it is not clear how much of the performance drop belongs to turns with cross-domain related slots. The figure 4 only present one example of this case, which might not be correct for all wrong predictions. Also, it is helpful to report DST for single domain and to evaluate the importance of proposed approach. 

4- Since the proposed approach can be used on any pretrained encoder, the evaluation on BERT and/or Roberta is helpful to understand the robustness of approach to the choice of pretrained encoder.



--------------------------------------------------------------
Post rebuttal: 

Table 3:
the results indicates that only Albert-xxlarge achieve very high performance (222M). however, comparable models to other approaches, such as Roberta-base or albert-xlarge achieved around ~57% performance which is within margin of previous arts. for example, SimpleTOD with gpt2-base (124M) achieved 55.7% and ConvBert achieved 58%.
Therefor, it is unclear why Albert-xxlarge get so much higher performance compared to other encoders, since same tokenization and domain-slot relation is used. 
Based on results in Table 3, there is inconsistency in which domain-slot relation does not always results in better performance,  and it depends on the choice of encoder too. 

Overall, the proposed architecture is very similar to TRADE model, in terms of using an encoder for dialogue history, slot-gate and slot-value classifier. The only difference is in using a much powerful pretrained encoder. 
review_title: limited analysis 

final_score: rating: 7: Good paper, accept
review_content: Summary:
This paper showcases how pre-training can help with Dialogue State Tracking. The authors explicitly model
the relationship between domain-slot pairs. With their encoding and using strong pre-trained initializations
they are able to improve the joint goal accuracy by almost 1.5 points which is impressive.

Reasons for score: 
This is a very well written paper and will be a good resource for people working on the task of Dialogue State Tracking.
The authors show how they can model relationships between domain-slot pairs and how they can encode them effectively
using pre-trained representations.
I am hoping that the authors can address some of the cons during the rebuttal period.

Pros:
1. Good dialogue representation which helps with the task of state tracking
2. Simple model consisting of encoders and 2 classifiers which are well explained.
3. Clear ablation study showing the value of 1) pre-training and 2) modeling relationship between domain-slot values


Cons:
1. This approach, like other popular approaches, suffers from the problem of having a fixed output vocabulary for slot values - hence limiting its scalability. While this cannot be addressed in this work, this is a drawback of this approach.
2. Some of the design decisions are stated but not well explained
- Only one pre-training method compared
- Authors mention they drop "dontcare" from slot gating but don't show the affect with or without it.
- Not much details on the setup and how it was trained.
3. Not much qualitative analysis.

Please address and clarify the cons above 

Typos/Areas for improvement:
1. Section 3.2 and 3.3 can be shortened a lot. I would suggest showing more analysis.
- More examples of type of  mistakes fixed.
- Which turn in the dialogue does the error decrease the most.
- How much is the training time/ accuracy tradeoff
2. Adding another layer to make DS-split work should be trivial, there is no reason to leave that to future work.
Could you show how the results look with that?

------

Updating score based on authors' response.
review_title: Interesting approach using pre-trained representations to encode domain-slot pairs for DST

final_score: rating: 4: Ok but not good enough - rejection
review_content: [Summary]
In this paper, the authors proposed a multidomain state-tracking model that leverages the relationship among different domain-slot pairs. This is done by leveraging the full-attention step over the [CLS] special token and by providing all the domain-slot pairs as a special token to a pre-trained language model (Figure 2 is very clear). To predict the value of the slot $D_{i,j}$, the author concatenates the representation of the [CLS] token, share among all the domain-slots, and the $D_{i,j}$, provided as input, and use a gating mechanism, by only using $D_{i,j}$ representation, to decide whether require as value (i.e., prediction) or not (e.g. None). \

The authors experimented using ALBERT (Lan et al., 2019) as a pre-trained language model, on the well-known benchmark MultiWoZ 2.0  (Budzianowski et al., 2018)  2.1 (Eric et al., 2019). The authors studied different format to represent $D_{i,j}$ DS-merge (i.e., one token per domain-slot) and DS-split (i.e., one token per slot and one per domain, thus more scalable). The reported performance is state-of-the-art at the time of submission. 

[Pros]
- The paper reads well and it is easy to follow for people working on Task-Oriented dialogue.
- The proposed method is simple and effective, and it would be easy to reproduce. 

[Cons]
- The idea of using domain-pairs as input to a large pre-trained model is not novel (Wu et al., 2019; Zhang et al., 2019; Lee et al., 2019), as also pointed out by the authors, but the authors do not explicitly clarify this in the methodology section, leading the reader to believe that the domain-pairs is their own contribution. Same for the slot-gate (Wu et al., 2019)
- The authors claim to learn relations between slots, but the analysis section is very thin and it just shows an ablation by masking the attention between the slot. Two points: why not just removing the [CLS] token instead of removing the attention, and why just using on ALBERTA large. For instance, the authors said "For this experiment, we used the ALBERT configuration of large-v2,
for faster experimentation" which is contradictory since large-v2 is the slowest to run I guess. Can the authors show this ablation for all the model size? 
- Although, MWoZ is the current benchmark for DST in ToDs, there are also other datasets for this task that can be considered (e.g., Schema Guided Dialogue (SGD) (Rastogi et.al. 2019))


[Reason to Reject]
The main contribution of this paper is very thin, adding the [CLS] token as input, and the main technical contribution is not well explored (missing an in-depth ablation). 

[Reason to Accept]
State-of-the-art performance at the submission time. To be noted, (Mehri et.al. 2020) reported better performance in MWoZ and other datasets, but this paper was released after the ICLR submission deadline.

[Question]
-  Can the authors show the ablation for all the model size? 

[Suggestion]
- Figure 4 is very hard to read. I suggest to better format the dialogue. 


review_title: Review

Meta-review: The authors explore modeling the relationship between domain-slot pairs in multi-domain dialogue state tracking via use of special tokens in pre-trained contextualized word embeddings (i.e., one special token for each domain-slot pair or special tokens for the domain and the slot that are merged). Beyond this, the basic architecture is very similar to the TRADE architecture (and papers that build on this general slot-gate + slot-value classifier) for the fixed vocabulary setting. Experiments are conducted on the MultiWOZ 2.1/2.2 datasets, demonstrating impressive improvements over recent results.

== Pros ==
+ They demonstrate that domain-slot interdependencies can be modeled through special tokens for use with pre-trained embeddings.
+ The top-line empirical results are impressive.

== Cons ==
- Lack of a deep dive on the empirical analysis to show precisely why/where the proposed method is working better than existing work.
- The methodological advance is minimal beyond using better pre-trained embeddings.
- Only one dataset when others exist and this is largely an empirical paper.
- The writing is rushed and reads like a 'late-breaking' paper.

Evaluating along the specified dimensions:
* Quality: The quality of the work was the primary concern of the reviewers. Specifically, this reads like a 'late breaking' paper where the table of results is impressive, but there isn't significant examination of the empirical results showing why/when it works relative to competing methods. Focusing just on Tables 2 & 3, much of the improvement is ostensibly really due to the more powerful embeddings. Contextualizing this wrt {SimpleTOD, TRADE, DSTQA, Picklist}, this appears a minor methodological innovation centered around the input embeddings. The empirical results are impressive, but may very well be a result of the more powerful pre-trained embeddings -- additional empirical analysis and discussion might be able to convince the reader otherwise, but is lacking here.
* Clarity: This is a very simple idea, so it should be easily understood by most familiar with the research area. That being said, the paper seems very rushed in general.
* Originality: This applies ideas used in many NLP applications to the dialogue-state tracking problem. As previously stated, the architecture is similar to several existing DST formulations -- where the core idea is to model slot-value interdependencies through the contextualized embeddings using special token. While not a trivial idea, it also is something that many could/would have put together. Until it is abundantly clear that this isn't really a study of how to apply larger pre-trained embeddings to DST problems, it isn't clear that this is a significant dialogue systems advance beyond the strong performance.
* Significance: As stated, this isn't a significant methodological advance. However, the empirical results appear very impressive -- although the reviewers expressed some concerns regarding the evaluation. Since this is largely empirical, one of the reviewers pointed out that additional relevant datasets now exist, which would significantly strengthen the case.

In summary, the empirical results appear impressive, ostensibly setting the SoTA. However, there were several concerns regarding the novelty of the approach, if it is actually working better due to the reasons stated, sufficient analysis of the empirical results, amongst other things. Thus, despite the impressive results, the consensus evaluation was that this work is not ready for publication in its current form (even if the top-line results should be disseminated).

Decision: Reject