Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: 
The paper deals with robust evaluation of NLU in task-oriented dialog systems. They propose some augmentation strategies like synonym replacement, insertion of vocal words, and homophone replacement, etc. for making models more robust to slot filling and intent detection. Results show that these augmentations largely help curate a more challenging evaluation set mimicking real-life scenarios.

strengths: ASR error-related tasks, while predominant for Speech-to-text tasks, they have not been extensively studied for NLU. 
weaknesses: Augmentation strategies like synonym replacements are widely known in the literature of robustness. The proposed augmentations, in some form, have been introduced in earlier papers (see the comments in the related works). Discussion related to that, and comparison to those DA techniques are not provided in the experiments (Section 4.4 has some baseline DA techniques, but as highlighted in the Related works comments, DA techniques designed for ASR errors have not been rigorously compared to).
correctness: 
The technical details are largely correct, with no immediate issue.

clarity: The paper is well written.
relation to prior work: 
The discussion on related works is one of the weakest links in the present draft. Although some works related to generalization tests using test set alterations, and sensitivity towards superficial features have been discussed, there is no structured discussion on the efforts to create strong testbeds using strategies involving augmentation, human-in-the-loop adversarial testing sets (such as Dynabench or Genie.), etc.

Another major related area missing in the discussion is that of counterfactual augmentations. Papers like Learning The Difference That Makes A Difference With Counterfactually-Augmented Data, Evaluating Models' Local Decision Boundaries via Contrast Sets, etc. are popular works that discuss counterfactual tests (and training) sets to learn more robust and generalizable decision boundaries. With regards to task-oriented dialogs, the recent ICLR paper COCO: Controllable Counterfactuals for Evaluating Dialogue State Trackers demonstrate novel ways to automatically generate counterfactuals related to slot edits and counterfactual response generation. This paper does highlight that there is a significant overlap between slot values across training and testing sets and that models would exploit frequent patterns to get good results. 

Closely related works to this draft, that are not discussed here include the empirical study on ASR errors in open-domain dialog - Are Neural Open-Domain Dialog Systems Robust to Speech Recognition Errors in the Dialog History? An Empirical Study, where the authors demonstrate that training with ASR errors teach models to be more robust to these superficial lexical errors. There are also many works that attempt to simulate ASR errors through various means, such as interpolations between linguistic and acoustic embeddings (Simulating ASR errors for training SLU systems), homophones (Improving the robustness of speech translation.), similar pronunciation (Augmenting translation models with simulated acoustic confusions for improved spoken language translation.), confusion n- gram pairs harvested from aligned ASR-reference text pairs (Data Augmentation for Training Dialog Models Robust to Speech Recognition Errors) etc. Similar augmentation strategies (already seen in the literature)  are present in the current draft, such as speako. While these works are not targeted for slot-filling and intent-detection, their observations with speech recognition should broadly transfer to these NLU tasks too. Slot substitution using synonyms has been discussed in Simple is Better! Lightweight Data Augmentation for Low Resource Slot Filling and Intent Classification, but again, the paper does not make the comparison.

More systematic robustness checks for evaluating models have been proposed in works like Beyond Accuracy: Behavioral Testing of NLP Models with CheckList, or more recently, Robustness Gym: Unifying the NLP Evaluation Landscape. As these works propose robustness checks in various NLP tasks, there should be a discussion if SLU tasks contain any peculiarities that the proposed draft specifically discusses.

The above references are a non-exhaustive list. As such, any revision should also include a detailed discussion (with comparisons) on other important related works.

additional feedback: 
1) Please fix the formatting of the related works section as it has artificial line breaks and is presently not in the format of paragraphs.


2) Figure 1 demonstrates the failure of the model from - A stack-propagation framework with token-level intent detection for spoken language understanding.  What is the reasoning behind choosing this model? Are there more recent models that might be more robust?


3) In Table 5, why does the performance drop when augmentations are introduced on the original test set?
review_title:  Interesting work but needs better comparison to literature.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes a framework that uses some heuristic rules to change the original test set without semantic altering. The proposed framework is applied to three benchmarks and the performance of SOTA models are measured on the altered test set. Experiment results show that even SOTA models are not robust to some perturbations.
strengths: (1) The motivation of this paper is clear.
(2) They propose a framework to perturb a dataset, which may be useful for future research.
(3) Human evaluation is conducted to ensure the semantic equivalence between the altered test set and the original test set.
(4) Experiment results reveal that SOTA models are not robust enough to small perturbations to the input. Also, there are some interesting phenomena. For example, Pre-V fillers are less disruptive than other operators, which "may" suggest some syntax-level pattern independence.
weaknesses: (1) The resulting test set using the proposed framework may not be "natural ". If an utterance does not seem natural, I would expect the models to be more likely to fail. It's not clear whether it is meaningful to test a model's ability on unnatural text, given that it has been trained on natural text.
(2) Although the paper claims that its motivation is to explore the heuristic patterns exploited by different models. It lacks in-depth analysis of where the models fail and what specific patterns different models may have exploited when making their prediction. I do think that the data augmentation part could be replaced by some detailed analysis (e.g. error case analysis).

correctness: The dataset construction process is well-described. However, I do have the concern that the resulting text may not be natural. The experiment designs are appropriate and the evaluations are well-performed.
clarity: The paper is mostly clear. 

Minor issues:
(1) Sentence in Line 150 seems incomplete.
(1) The Hard setting is not clear to me.

relation to prior work: Missing reference: Naik, Aakanksha, et al. "Stress test evaluation for natural language inference." (2018).
additional feedback: I do think the whole story is a little plain, and I would expect the authors to add more insight into it through more in-depth analysis.
review_title: Interesting work, but not ready for publication

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper introduces a heuristic-based black-box attack technique on the slot filling and intent detection tasks. Specifically, three kinds of sentence-modifying operations are used, including adding semantic-neutral filters, replacing words with synonyms, and adding typos to words. Two heuristic strategies are used on the testing set, which include randomly apply operations on sentences and select the most misleading operation based on a surrogate JointBERT model. Both methods are applied on three benchmarks. Human-based evaluations are made to show the generated adversarial examples are label-preserving. Naturally trained models have shown significant decreases in the adversarial test sets. On the other hand, however, models augmented trained with the perturbed examples show a decrease in standard accuracy though enjoy better robust accuracy.
strengths: The paper introduces a useful adversarial attack technique for slot-filling and indent detection tasks. The results are claimed to be the first to show the deficiency of the current research in this area, and highlight that there is a lot of room for future research. 
weaknesses: The methods used to generate the adversarial examples are not novel and have been widely used in the NLP field, making the contributions a little incremental. For example, (Yin, et al.) attack sentence encoders with token substitutions, (Belinkov, et al.) attack neural translation systems with typoes, and (Jia, et al.) attack reading comprehension systems by adding semantic neutral sentences. With so many previous works, I am not surprised that NLP models are susceptible to the perturbations introduced in this work.

> Yin, Fan et al. “On the Robustness of Language Encoders against Grammatical Errors.” ArXiv abs/2005.05683 (2020): n. pag.

> Belinkov, Yonatan and Yonatan Bisk. “Synthetic and Natural Noise Both Break Neural Machine Translation.” ArXiv abs/1711.02173 (2018): n. pag.

> Jia, Robin and Percy Liang. “Adversarial Examples for Evaluating Reading Comprehension Systems.” ArXiv abs/1707.07328 (2017): n. pag.
correctness: The authors have made a human-based evaluation on the generated testing sets to ensure the added perturbations are reasonable.
clarity: The paper is basically well presented. It's worth mentioning that the line 400 - 4`10 should be deleted according to the instructions. For checklist 3. (c), where are the error bars in the paper? For 2. (a) (b), I don't find any theoretical results in the paper.
relation to prior work: The authors have discussed similar works in other tasks for NLP, like NLI, co-reference resolution, and grammar error correction. On the other hand, more discussions about the adversarial attack techniques in the NLP field should be included, like those I mentioned in *Weaknesses*. For reference, check this [list](https://github.com/thunlp/TAADpapers).
additional feedback: What does line 179 mean?
> So, we repeat this operation 10 times and calculate the mean score and variance.
review_title: Useful but Not Novel

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: Final rating: Initial rating was "Below acceptance threshold", have changed it now to "above acceptance threshold". Explanation in the comments of this thread. 

The paper argues that the state-of-the-art Intent detection models are not evaluated on sufficiently realistic samples that occur often in real-world applications. The spoken language introduces corruptions in the text (e.g. filler words) and models need to be robust to such corruptions. 

The paper shows that the state-of-the-art models are not robust and suffer a drop in performance under such corruptions. The texts in the existing datasets are automatically altered by introducing a fixed set of corruptions. The paper also introduces data augmentation techniques to improve the robustness of these models for these new test sets.

strengths: The new test set is well-motivated, it is helpful to benchmark Intent detection models and evaluate their robustness for real-world applications. The proposed data augmentation technique also improves the robustness of the state-of-the-art approaches.
weaknesses: One of the key aspects that is not evaluated is how well the new test set reflects the real-world scenario. Since the dataset is automatically curated, does it cover most of the cases that can happen in the real world? I did not find any discussion in the paper as to why the proposed corruption techniques represent real-world cases well. 
correctness: The claims by the authors are correct. 
clarity: The paper can be improved by describing the tasks such as intent detection, slot-filling in a couple of lines. This is crucial for understanding the paper, and these terms (especially the second one) are not common knowledge. 

Most of the footnotes can be part of the main text.
relation to prior work: Yes
additional feedback: Please check the Weaknesses section.
review_title: New test benchmarks to evaluate the robustness of Intent Detection models against possible corruptions in text due to variations in the spoken language. Some analysis on how well this synthetically curated test set reflects the real scenario is missing. Not sure if the code, data and the test sets will be released as it was mentioned that: “The code and the data are proprietary”. 

Meta-review: As pointed by reviewers, this work will require a series of additional work (better comparison to literature and justify novelty) to make it ready for publication.  I recommend rejecting the current version.
Decision: Reject