Pay Attention to Real World Perturbations! Natural Robustness Evaluation in Machine Reading Comprehension

Authors: Yulong Wu, Viktor Schlegel, Riza Batista-Navarro
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Natural Perturbations, Robustness Evaluation, Machine Reading Comprehension

Abstract:
As neural language models achieve human-comparable performance on Machine Reading Comprehension (MRC) and see widespread adoption, ensuring their robustness in real-world scenarios has become increasingly important. Current robustness evaluation research, though, primarily develops synthetic perturbation methods, leaving unclear how well they reflect real life scenarios. Considering this, we present a framework to automatically examine MRC models on naturally occurring textual perturbations, by replacing paragraph in MRC benchmarks with their counterparts based on available Wikipedia edit history. Such perturbation type is natural as its design does not stem from an arteficial generative process, inherently distinct from the previously investigated synthetic approaches. In a large-scale study encompassing SQUAD datasets and various model architectures we observe that natural perturbations result in performance degradation in pre-trained encoder language models. More worryingly, these state-of-the-art Flan-T5 and Large Language Models (LLMs) inherit these errors. Further experiments demonstrate that our findings generalise to natural perturbations found in other more challenging MRC benchmarks. In an effort to mitigate these errors, we show that it is possible to improve the robustness to natural perturbations by training on naturally or synthetically perturbed examples, though a noticeable gap still remains compared to performance on unperturbed data.

Decision: reject
Reviewer scores: 6, 3, 3

Reviews and discussion:
<thread>
creation date:
1730417108632

thread_type:
official_review

<thread_object>
creator: reviewer_fabr

<content>
summary:
The authors study the impact of real world perturbations on MRC performance for various Transformer-based architectures. They propose a framework to create naturally-perturbed test sets from popular QA datasets using the revision history of Wikipedia. They perform experiments and analyses on the abovementioned models, followed by adversarial training of encoder-only models.

strengths:
* Comprehensive evaluation across multiple architecture types rather than only decoder models and multiple models of each type: encoder, decoder, encoder-decoder models
* Somewhat comprehensive evaluation across multiple QA datasets, with caveats: see below
* Paragraphs are generally well-written and easy to understand

weaknesses:
**TLDR**

The authors pose an interesting question, but the execution of the study contains unexpected design decisions that are not well-justified. The exact improvement of their claimed methodology over existing work is also unclear.

**Details:**
* The authors call out the similarities between their method and Belinkov & Bisk (2018), do not make clear the differences and improvements over the latter, if any. The claimed contribution ("novel Wikipedia revision history-based framework") also does not highlight the exact improvements over simply applying Belinkov & Bisk (2018)'s method to the MRC setting, and is therefore running the risk of over-claiming. I recommend the authors revise the framing to highlight the exact contribution over Belinkov & Bisk (2018) and other prior, similar work [1, 2].
* The encoder-only models were only evaluated on SQuAD datasets to draw conclusions, with no clear justification even though the authors are aware that they are no longer challenging benchmarks.
* The transition to experiments on the other architectures and datasets was a little strange (only evaluating on questions that the encoder models failed on) with no clear justification. There is then a transition back to encoder-only models and SQuAD for adversarial training, also without clear justification for excluding other datasets and architectures. 
* The authors appear to have missed relevant work on robustness evaluation in QA/MRC [3, 4] and other types of synthetic perturbations [5].


**Missing References:**
1. Mining Naturally-occurring Corrections and Paraphrases from Wikipedia’s Revision History. Max et al. 2010.
1. Robust Systems for Preposition Error Correction Using Wikipedia Revisions. Cahill et al. 2013.
1. It’s Morphin’ Time! Combating Linguistic Discrimination with Inflectional Perturbations. Tan et al. 2020.
1. Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglots. Tan et al. 2021.
1. From Hero to Zéroe: A Benchmark of Low-Level Adversarial Attacks. Eger et al 2020.

questions:
> Questions on which none of the encoder-only models fail under the perturbation are then removed.

Why was this decision made, rather than studying the effect of all perturbed questions on each architecture type? An analysis of the overlaps between architectures is good to have, but I would have preferred to see the former if I had to choose one. I may have missed a strong justification, if one has already been included.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730507248122

thread_type:
official_review

<thread_object>
creator: reviewer_qqoy

<content>
summary:
This paper introduces perturbed sets of several machine-reading comprehension datasets including SQuAD, using Wikipedia edits to create natural perturbations. Results show that encoder-only, decoder-only and encoder-decoder LMs suffer from this challenge. Adversarial training with perturbation is an effective defense strategy.

strengths:
1. It is interesting to use Wikipedia edit history to construct perturbation.
2. The perturbed set is verified by human to ensure that the perturbed examples are still valid.
3. Results show that natural perturbation is a powerful attack to LMs.

weaknesses:
1. It is unclear whether stronger model, e.g. gpt-4o would still suffer from this challenge. While weaker models like BERT suffers from the natural perturbations, it is important to show that it is still a challenge for recent stronger LLMs.
2. The perturbation method relies on Wikipedia edit history, limiting its applicability to non-Wikipedia based datasets. 
3. The performance drops on non-SQuAD datasets like DROP are relatively small, e.g. LLaMA-2 only exhibits less than 2 points drop, which could also potentially be remedied by adversarial training. Again, I'm concerned that this benchmark is not super challenging for recent LLMs anymore.

questions:
line 334-337: I have trouble understanding this long sentence with too many clauses. can you explain that?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731275063163

thread_type:
official_review

<thread_object>
creator: reviewer_6syj

<content>
summary:
This paper contributes to understanding and improving the robustness of Machine Reading Comprehension models by introducing a new evaluation framework based on naturally occurring text variations. Rather than relying on synthetic perturbations, the authors leverage Wikipedia edit history to generate realistic test cases that reflect how the text changes in the real world.

strengths:
- Proposes a framework using Wikipedia edit history to generate natural perturbations in MRC benchmarks

- Evaluate model performance across encoder-only, encoder-decoder, and decoder-only architectures

- Shows that natural perturbations can degrade performance and these errors transfer to larger models

- Demonstrate that adversarial training with both natural and synthetic perturbations can help mitigate these issues

weaknesses:
I am generally optimistic about the paper, and I have the following minor concerns. 

The analysis section could be more in-depth
1. Permutations and models

- No investigation of how perturbation magnitude affects performance and why certain permutations affect the model more than others;

- Missing analysis of the interaction between model size and robustness is extremely important as we see that some observations might not always be predictable and transferrable on smaller model sizes. 

- I am wondering why authors didn't consider ablation studies on the impact of Wikipedia edit types.

2. Wikipedia edit history

It feels to me that Wikipedia's edit history might suffer from being less realistic and potentially data contamination.

questions:
- why certain permutations affect the model more than others;

- Are certain types of changes over/under-represented?

- whether different pretraining approaches affect models' robustness to natural perturbations?

- the effectiveness of adversarial training vary with model size and architecture?

- the effectiveness of adversarial training vary with model size and architecture?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732209757863

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Changes to our manuscript

comment:
Dear Reviewers,

We again thank you for your efforts in reviewing our work and providing insightful feedback. Based on the weaknesses and questions raised by each reviewer, we made the following changes in the updated manuscript (changes are marked in blue):

**(Reviewer 6Syj)**
1. Investigated how perturbation magnitude affects performance *(Section 5.2 footnote 8)* and further clarified why certain permutations affect the model more than others *(Section 5.2 line 343-345)*.
2. Added analysis of the interaction between model size and robustness. *(Section 5.4 396-400)*
3. Further discussed the overall distribution of perturbation types *(Section 5.2 327-329)* and the effectiveness of adversarial training *(Section 6 481-484)*.

**(Reviewer qQoy)**
1. Added experimental results on GPT-4o. *(Section 4.2 227; Table 2 414; Table 3 447-448)*
2. Explained the applicability limitation in this work. *(Section 1 footnote 1)*
3. Described the challenge test set construction process in detail. *(Section 5.4 381-382; Appendix H)*

**(Reviewer fAbR)**

1. Highlighted the differences and improvements of our natural perturbation construction methodology over previous works. *(Section 1 075-081 and 083-085)*
2. Referenced the suggested related works. *(Section 1 045; Section 2 116-117)*
3. Added experimental design justification. *(Section 5 247-256; Section 6 455-457)*
4. Presented supplementary experiments to study the effect of all perturbed instances on each architecture type. *(Appendix E)*
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a new framework for evaluating the robustness of Machine Reading Comprehension (MRC) models by leveraging Wikipedia edit history -- naturally occurring edits to construct test sets that reflect real-world text variations. The paper evaluates how these naturally perturbed examples affect various Transformer architectures (encoder-only, encoder-decoder, and decoder-only) on multiple MRC datasets. In addition, the authors explore adversarial training with naturally perturbed data to help reduce accuracy drops.

**Strengths** (1) Using real-world perturbations is interesting and technically sound, complementing synthetic perturbations. (2) the study covers multiple architectures, which shows the breadth of evaluation.

**Weaknesses** (1) Reviewers highlight the insufficient analysis of various perturbation types, model sizes, and more challenging datasets & tasks; (2) The paper's experimental design for different architectures appears inconsistent as raised by some reviewers; (3) A deeper discussion of potential data contamination would strengthen the paper


**Decision** The paper received mixed reviews, and the author also addressed part of the questions of reviewers. However, while all reviewers agree the paper targets the important area of real-world robustness, there still limitations in the current form which the reviewers cannot agree on after the rebuttal. Therefore, I am leaning toward rejection.