Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper presents an interesting approach to generate dialogs in a controllable fashion to evaluate a Dialog State Tracking system on a data distribution which is different from the training/test data. The proposed approach first generates a turn-level goal by adding or dropping a slot and then replacing slot values. In the second step, the proposed method generates counterfactual conversation conditioned on the dialog history and goal generated in the previous step. The authors show that evaluating current state-of-the-art DST model on MultiWOZ datasets with the generated counterfactuals results in significant performance drop. Additionally, human evaluation shows that the generated conversations perfectly reflect the underlying user goal. 

The paper is trying to tackle an important problem of evaluating robustness of a DST model when most of the available datasets has similar distribution in train and test splits. The proposed method is well explained and the effectiveness of the approach is substantiated by extensive results.

It is not clear what is the performance of utterance generation model. This model is somewhat different from other language model since it is conditioned on belief as well. Also, seems like the proposed approach can generate counterfactual conversation only for one turn which seems limited for the evaluation.

I would like to hear from authors on:
- In section 5.4, it is not clear if slot combination dictionary was also split such that slot combination in train and test data doesn’t overlap? If not, model can learn just the generated utterance pattern and perform better on the generated test set. 
- Performance of the utterance generation model in terms of diversity of the generated utterances since it will affect the evaluation’s robustness. 
review_title: Nice work on generating conversations to evaluate models for Dialog State Tracking 

final_score: rating: 4: Ok but not good enough - rejection
review_content: <Summary>

This paper addresses the problem of evaluating dialogue state trackers (DST)’s generalization ability to novel and realistic dialogue scenarios that do not exist in the dataset. It proposes a model-independent approach to evaluate DST systems with the idea of counterfactual conversation generation. The proposed approach is integrated with three recent DST models and evaluated on MultiWoZ dataset. 

<Strengths> 

1. The idea of counterfactual goal/conversation generation can be useful for the evaluation of DST systems, which is often quite challenging in practice. 

2. The proposed CoCo idea is tested with three recent DST systems including Trad, TripPy and SimpleTod and show its effectiveness on MultiWoZ dataset. 

<Weakness>

1. This paper proposes an interesting idea for DST evaluation but its implementation is largely ad-hoc and engineering intensive and thus bears little technical novelty.

(1) As described in section 4 and Fig.2, the proposed approach is simply a combination of multiple components with heuristic rules (e.g. the way of value substitution, and the use of some predefined operations). 

(2) The two filtering schemes are also ad-hoc, consisting of two slot-value match filter and classifier fitter.

(3) It is hard to find methodological novelty in the proposed method. Given that ICLR is a top premier ML venue, it could be a significant weakness to be a publishable work. Thus, this work may be better fit to a venue of NLP.

2. Experimental evaluation can be improved. 

(1) This work evaluates only on a single dataset – MultiWoZ 2.1. Admittedly, it is one of the most important benchmarks for DST tests, but one or more datasets are encouraged to use for better justification of generality, for example, MultiWoZ 2.0 or Taskmaster-1. 

(2) The experiments mainly focus on how much the proposed CoCo improves the three DST systems. However, comparative study is required such as how much CoCo is better than other alternatives or baselines. 

<Conclusion>

My initial decision is ‘reject’ mainly due to lack of technical novelty. Experiments could be improved for better evaluation. 

review_title: Review #1

final_score: rating: 6: Marginally above acceptance threshold
review_content: Update after reading author response:

Thank you for the thoughtful response.  I appreciate that you have added extra implementation details and am changing my score to a 6.  

Regarding the limitations in scope:  My apologies if my review was confusingly worded.  I just wanted to clarify that I had meant that the counterfactual generation method itself may have limitations (not the high-level DST task, which I agree has broad uses).  The concern is that the adversarial example generation strategy might be too domain-specific to transfer easily to other tasks, and that this might limit the impact of the proposed counterfactual generation method.  I think this is somewhat related to the parts of the methodology that R1 described as "ad-hoc", "engineering intensive", or reliant on heuristics.  I still have some reservations about the transferability of the proposed methods, though the authors' response to R1 did clarify a bit on this point.

-------------------------------------------------------------------------------------------------------------------------------------------

Original Review:

This paper presents CoCo, a new method for generating adversarial examples for DST tasks using “controllable counterfactual” generation.  Unlike other approaches for adversarial example generation, this approach is model agnostic.  They also demonstrate the effectiveness of the method by applying the Multi-woz dataset where s.o.t.a. Model performance drops by ~30 points.

I might lean towards rejecting this paper because there are several points in the methods that are still unclear to me.  However, I hope the authors may be able to clarify some of my questions (I’ve listed a few in the limitations section of my review).  Another potential drawback is that the methods described here are limited in scope to the dst domain, so it may not be as useful to other ML researchers more broadly.  

Strengths:
- CoCo is a very effective method.  Model performance drops by about 30 points, demonstrating that the adversarial examples generated are extremely challenging. 
- The human study demonstrates that the created examples are still human-like and are actually often rated “more correct” than the original Multi-Woz responses.

Limitations:
- The proposed methods are using a lot of pre-existing components and limited in scope to this one domain.  While the created counterfactuals are useful as a challenge dataset for DST systems, the overall approach in this paper may not be more broadly impactful outside of this task.
- There are a few points in the methodology that seem a bit unclear or should be described more fully:
  - Can you provide more details about the conditional generation model (p):  How is it being trained?  What architecture is being used?
  - The Slot-Value match filter: How are you judging whether the candidate “contains” the value? (Is this exact match?)
  - Classifier filter:  Can you provide more details on the classifier architecture and how it is being trained?  What is the precision and recall of its predictions?

Questions:
- In section 5.4: Any thoughts on why data augmentation is more effective for the Trippy model than the other two models?

Minor Feedback:  
- The CoCo name could be easily confused with Microsoft Coco, a commonly used computer vision dataset.  Authors might consider changing the name to avoid confusion.
- The font in Figures 3 and 4 is tiny and hard to read.  I recommend making it bigger.
review_title: Creating challenging examples for DST systems: interesting results but may need more clarification in the methods section

Meta-review: The paper proposes a method  to generate conversations for evaluate dialog systems using counterfactual generation. 

Pros:
- The reviewers agree that the paper makes a good contribution towards evaluation of DST models. 
- The paper adds to a growing body of work on robust evaluation of NLP models

Cons:
- One reviewer has commented on the lack of novelty. However, I believe that the authors have adequately addressed it. In particular, I do not see any harm in using heuristics/templates to generate counterfactuals as long as the final goal of robust evaluation is achieved. 
- It would have been good to evaluate the method on other datasets. However, I agree with the authors' rebuttal that this is indeed the most popular dataset for the task and most SOTA methods evaluate on this dataset. 

The authors have adequately addressed all reviewer concerns and have clearly highlighted their contributions and novelty.

I think of this as a valuable contribution and would like to see the paper accepted. 
Decision: Accept (Poster)