Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper presents NATURE, a set of simple spoken-language-oriented transformations applied to the evaluation set of datasets, that show the shortcomings of current models in real-world settings. While these transformations are natural human spoken language variations and preserve the semantics of an utterance, they can deteriorate model performance significantly (up to 40%) which shows the limitations of current approaches.
strengths: 1. This is an insightful paper that investigates several main shortcomings of current approaches.
2. The dataset is creatively constructed and large enough to draw robust conclusions.
3. The experiments are comprehensive and detailed. 
weaknesses: 1. How do you ensure that these transformations are actually natural? They seem natural but certain ones occur less likely than others (e.g. Speako does not happen that much in my opinion).
2. How do you choose these transformations? There could be many others that are more realistic but not covered in this benchmark.
3. How well can better ASR systems alleviate these problems? Are the drops in performance due to worse ASR or worse NLP models?
correctness: The claims are largely correct, but see several weakness points above.
clarity: Mostly. Although the part on human evaluation could be more detailed and more clear, I had trouble understanding that.

Several tables like Table 7 could also be improved in presentation, perhaps replaced with graphs/plots to show drops in performance. I'm not quite sure what the main takeaway from Table 7 is.
relation to prior work: There can be closer comparison to current work, such as using tables to show comparable dataset sizes, data sources, target studies, and so on.
additional feedback: none
review_title: review

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper presents NATURE a set of spoken language transformations to modify a sentence without changing the semantic meaning. Current deep learning models overfit on surface level patterns leading to poor generalization. NATURE is used to modify the data from several existing slot-filling and intention detection benchmarks and is compared to models trained on the original unmodified benchmarks. When evaluated end to end the performance of the models was severely diminished.
strengths: NATURE provides a way to automatically modify sentences in ways that maintain semantic meaning to add more natural spoken language characteristics.
The paper does show that current models are not capturing the semantic meanings, since changing data in ways that does not change the semantic meaning of the statements can significantly decrease their performance.

weaknesses: Incorporating the work done showing NATURE’s use as a tool to augment datasets into the main paper would make a stronger argument for its utility.

correctness:  The evaluation methods and experiments were appropriate and performed correctly. 

clarity:  The paper was clearly written and easy to follow.

relation to prior work: Previous work has suggested the use of human in the loop approach to add additional adverse data, while this work provides an automatic method.

additional feedback: None
review_title: A set of spoken language transformations that maintain semantic meaning to evaluate and improve generalization

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: 
This work presents a set of language operators/transformations called NATURE to transform the sentences in the evaluation/test set of different language datasets and evaluates the state-of-the-art models on the perturbed sentences for intent detection (ID) and slot filling (SF) tasks.

NATURE operators are categorized into three groups including *Fillers*, *Synonyms*, and *Speako (similar sounding).*
The authors hypothesize that a trained model should have the same performance when the NATURE operators are applied to the data.

The authors mention that researchers could simply remove fillers as a pre-processing step, but it is more interesting to study their effects.
They divide **fillers** into four categories:

1.  Beginning-of-sentence (BOS)
2.  End-of-sentence (EOS)
3.  Pre-verb
4.  Post-verb

The first two perturbations are easy to add, but the last two perturbations require finding the part-of-speech (POS) tags. They also have a fail-safe option to add the common filler &ldquo;*like*&rdquo; when no verb is found.

For **synonyms**, they randomly choose a POS tag and then find a word from the utterance that has this POS tag, and they replace this token with the [MASK] token. Finally, they feed this utterance to BERT which outputs a list of candidates with their corresponding probabilities. Some of these candidates are not appropriate substitutions. To remedy this, the authors remove candidates that do not have the same POS tag.
If an utterance doesn&rsquo;t contain the randomly chosen POS tag, they use the common *noun* POS tag.

For **speako**, they make a list of words that occur more than 1000 times in the whole English Wikipedia and convert each of them to their International Phonetic Alphabet (IPA) representation. Next, they randomly select one word from the sentence, convert it to its IPA format, and compute the similarity between it and all items in the list using Levenshtein distance. Finally, they replace the word from the utterance with the closest candidate from the list.

They only apply one of the transformations to each utterance to keep perturbed sentences recognizable and comparable to their original formats.
The authors design two different evaluation sets using NATURE operators which they call *Random* and *HARD*. In the Random design, the authors select one operator randomly for each utterance, repeat it ten times, and compute the average score. In the HARD design, they use the JointBERT model to select the most challenging operator, and for each utterance, they select the operator with the lowest confidence score.

The authors assume that applying NATURE operators on the utterances maintains the semantics, and the resulting utterances are similar to original utterances in terms of meanings and labeling. To measure this, they provide a multiple-choice survey. The result of the survey shows high scores in both groups of participants for slot-filling while a lower score for intent detection.

Authors also run another set of experiments with a general perturbation method, *the Checklist tool* which includes four simple perturbations; (1) adding or removing **punctuations**, (2) adding **typo** by swapping random characters with one of their keyboard neighbors, (3) **contraction** operator that swaps contracted words with their non-contracted version and vice versa (4) **NER** operator that substitutes names, location, and numbers with other named entities.

They test their method on three datasets including Airline Travel Information System (ATIS), SNIPS voice platform, and NLU-ED.
They evaluate the performance of slot-filling by reporting the F1 score, and the performance of intent detection by reporting accuracy. They also use an end-to-end accuracy (E2E) which counts all true positives for both slot-filling and intent detection and reports one metric for the two tasks of SF and ID.

The authors use two models; Stack-Prop+BERT and Bi-RNN.
Stack-Prop+BERT uses BERT to encode tokens and then feeds them into two BiLSTMs, one for the SF task and another for the ID task. The output of the SF BiLSTM is added to ID BiLSTM input to be able to generate word-level intent labels which are then averaged to produce a sentence-level label.

The results consistently show a lower performance on the perturbated data compared to the original utterances. This supports the claim that the state-of-the-art models cannot generalize well to small changes in language, and shows the utility of this work.


strengths: 1.  The data augmentation and transformations introduced by this paper are valid and can happen in real-life spoken and even written language.
2.  This paper as a benchmark can be an incentive for researchers to develop more generalizable language models.
3.  The evaluations and analysis are extensive and very well done.
4.  The low score when the EOS operator is used is interesting and can be further investigated.


weaknesses: 1.  The paper is already very good, but it would have been stronger if the authors could use models that have pre-trained versions available to ensure that implementation discrepancies do not affect the comparability of the performances.
2.  Adding a paragraph in the related work section to describe how this paper is different from the cited works could be helpful.

Some minor and easy-to-fix issues:

1.  Although it is mentioned in lines 222-223, it would be better to describe what the percentages are showing in the caption of table 6 so that reader can understand without searching for the description outside the caption.
2.  Table 5 is not referenced in the body of the paper. All figures and tables should be referenced to let the readers know when to look at them.


correctness: The evaluations are thorough and valid. The claims are supported by the results.
clarity: The paper is enjoyable and easy to read. The flow of the paper is good and keeps the reader engaged.

relation to prior work: Authors position their work very well with respect to related works in the area.

additional feedback: The link you provided in the comments does not work. I do not know whether it is the same material that you included in the GitHub repository or not.
review_title: A valid and fair set of operations on language datasets that decreases the performance of the SOTA models on two tasks 

Meta-review: The paper presents a set of semantics-preserving spoken-language transformations that can be applied to existing datasets. The goal is to provide a mechanism for evaluating the generalizability of learned models in the face of changes to the language that preserve meaning and are consistent with human speech behaviors and ASR-induced errors, in order to measure the overfit of especially deep learning methods on existing benchmarks. Extensive evaluations, comparisons, and analyses are provided, including baselines for other forms of language perturbation. The datasets provided are large enough and well-evaluated enough to be of use, and supports the basic claim that existing models are prone to overfitting on surface patterns. The proposed transformations are intuitive and the paper is well written.
Decision: Accept