Evaluating Paraphrastic Robustness in Textual Entailment Models
Year: 2023

Abstract:
We present P̂ aRTE, a collection of 1,126 pairs of Recognizing Textual Entailment (RTE) examples to evaluate whether models are robust to paraphrasing. We posit that if RTE models understand language, their predictions should be consistent across inputs that share the same meaning. We use the evaluation set to determine if RTE models’ predictions change when examples are paraphrased. In our experiments, contemporary models change their predictions on 8-16% of paraphrased examples, indicating that there is still room for improvement.

heading: 1 Introduction
text: Recognizing Textual Entailment (RTE), the task of predicting whether one sentence (hypothesis) would likely be implied by another (premise), is central to natural language understanding (NLU; Dagan et al., 2005), as this task captures “all manners of linguistic phenomena and broad variability of semantic expression” (MacCartney, 2009). If an RTE model has a sufficiently high capacity for reliable, robust inference necessary for full NLU (MacCartney, 2009), then the model’s predictions should be consistent across paraphrased examples.
We introduce P̂ aRTE, a test set to evaluate how reliable and robust models are to paraphrases (Table 1 includes an example). The test set consists of examples from the Pascal RTE1-3 challenges (Dagan et al., 2006; Bar-Haim et al., 2006; Giampiccolo et al., 2007) rewritten with a lexical rewriter and manually verified to preserve the meaning and label of the original RTE sentence-pair. We use this evaluation set to determine whether models change their predictions when examples are paraphrased.
While this may not be a sufficient test to determine whether RTE models fully understand language, as there are many semantic phenomena that RTE models should capture (Cooper et al., 1996; Naik et al., 2018), it is necessary that any NLU system be robust to paraphrases.
Our experiments indicate that contemporary models are robust to paraphrases as their predictions do not change on the overwhelmingly large majority of examples that are paraphrased. However, our analyses temper this claim as models are more likely to change their predictions when both the premise and hypothesis are phrased compared to when just one of the sentences is rewritten. We release P̂ aRTE1 to encourage others to evaluate how well their models perform when RTE examples are paraphrased.

heading: 2 Related Work
text: With the vast adoption of human language technology (HLT), systems must understand when different expressions convey the same meaning (paraphrase) and support the same inferences (entailment). Paraphrasing and entailment are closely connected as the former is a special case of the latter where two sentences entail each other (Nevěřilová, 2014; Fonseca and Aluísio, 2015; Víta, 2015; Ravichander et al., 2022). Para-
1https://dataverse.harvard.edu/dataset.xhtml? persistentId=doi:10.7910/DVN/HLMI23
880
phrasing has been used to improve RTE predictions (Bosma and Callison-Burch, 2006; Sun et al., 2021) and RTE has been used for paraphrase identification (Seethamol and Manju, 2017) and generation (Arora et al., 2022). Furthermore, both phenomena are key to NLU (Androutsopoulos and Malakasiotis, 2010) and work such as Zhao et al. (2018); Hu et al. (2019) have explored rewriting RTE examples to create more robust models.
We follow a long tradition of evaluating linguistic phenomena captured in RTE models (Cooper et al., 1996). Recent tests focus on evaluating how well contemporary RTE models capture phenomena such as monotonicity (Yanaka et al., 2019a,b), verb veridicality (Ross and Pavlick, 2019; Yanaka et al., 2021), presuppositions (Parrish et al., 2021) implicatures (Jeretic et al., 2020), basic logic (Richardson et al., 2020; Shi et al., 2021), figurative language (Chakrabarty et al., 2021), and others (Naik et al., 2018; Poliak et al., 2018a; Vashishtha et al., 2020). Unlike many of those works that evaluate models’ accuracy on examples that target specific phenomena, we use a contrastive approach (Prabhakaran et al., 2019; Gardner et al., 2020) to determine whether RTE models’ predictions change when examples are paraphrased.

heading: 3 P̂ aRTE
text: To explore whether these RTE models are robust to paraphrases, we create P̂ aRTE, a modified version of the Pascal RTE1-3 challenges (Dagan et al., 2005; Bar-Haim et al., 2006; Giampiccolo et al., 2007). P̂ aRTE contains 1,126 examples of an original unmodified RTE sentence-pair grouped with a sentence-pair with a modified premise, hypothesis, or both. We use the examples in RTE1-3 to create our test set, as opposed to other RTE datasets due to its long-standing history.

heading: 3.1 Paraphrase Generation & Verification
text: For each RTE premise-hypothesis pair (P-H), we created three paraphrased premises (P’) and hypotheses (H’) using a T5-based paraphraser2 finetuned on the Google PAWS dataset (Zhang et al., 2019). To ensure lexically diverse paraphrases, we filter out any paraphrases that have high lexical overlap with the original sentences using Jaccard index threshold of 0.75. Out of 14,400 generated sentences, 2,449 remained - 956 paraphrased
2We manually verified the quality of this paraphraser. See Appendix B for more details.
premises (P’) and 1,493 paraphrased hypotheses (H’). Next, we retained 550 paraphrased premises and 800 paraphrased hypotheses paraphrases that crowdsource workers identified as grammatical and similar in meaning to the original sentences.3 We include a grammatical check since an existing RTE evaluation set focused on paraphrases (White et al., 2017) contains hypothesis-only biases related to grammaticality (Poliak et al., 2018b).
If at least one P’ or one H’ passes this filtering process, we retain the original RTE example and pair it with a corresponding paraphrased example (i.e. P’-H’, P’-H, or P-H’). In the case where more than one P’ or H’ passes the filtering, we retained the P’ or H’ that crowdsource workers deemed most similar to the original sentence. Out of the original 2,400 RTE test pairs, we retain 914 pairs with a high-quality P’ or H’, resulting in 1,178 original and paraphrased RTE pairs.4

heading: 3.2 Overcoming Semantic Variability
text: MacCartney (2009) argues that in addition to being reliable and robust, RTE models must deal with the broad variability of semantic expression. In other words, though two sentences may be semantically congruent, it is possible that small variations in a paraphrased sentence contain enough semantic variability to change what would likely, or not likely be inferred from the sentence. Despite all P’ and H’ being deemed to be semantically congruent with their corresponding original sentences, the semantic variability of paraphrases might change whether H or H’ can be inferred from P’ or P.
Therefore, propagating an RTE label from an original sentence pair to a modified sentence pair might be inappropriate. We manually determined that this issue occurs in just 52 (4%) examples, and retained 1,126 examples. This ensures an evaluation set of high-quality examples that can be used to determine whether models are sensitive to paraphrases and change their prediction on paraphrased examples. Our dataset contains 402 examples with just a paraphrased premise P’, 602 with just a paraphrased hypothesis H’, and 122 with both a paraphrased premise and hypothesis.
3See Appendix B for a detailed description of this filtering process, including annotation guidelines.
4415 pairs where the premise is paraphrased, 631 pairs where the hypothesis is paraphrased, and 132 pairs where both are paraphrased.

heading: 4 Experimental Setup
text: We explore models built upon three different classes of sentence encoders: bag of words (BoW), LSTMs, and Transformers. Our BoW model represents premises and hypotheses as an average of their tokens’ 300 dimensional GloVe embeddings (Pennington et al., 2014b). The concatenation of these representations is fed to an MLP with two hidden layers. For the BiLSTM model, we represent tokens with GloVe embeddings, extract sentence representations using max-pooling, and pass concatenated sentence representations to an MLP with two hidden layers.
Our transformer-based models are pre-trained BERT (Devlin et al., 2019) and Roberta (Liu et al., 2020) encoders with an MLP attached to the final layer. Additionally, we use GPT-3 in a zero-shot setting where we ask it to label the relationship between a premise and hypothesis.5
The RTE training sets do not contain enough examples to train deep learning models with a large number of parameters. We follow the common practice of training models on MNLI and using our test set to evaluate how well they capture a specific phenomenon related to NLU. During testing, we map the MNLI ‘contradiction’ and ‘neutral’ labels to the ‘not-entailed’ label in RTE, following common practice (Wang et al., 2018; Yin et al., 2019; Ma et al., 2021; Utama et al., 2022, inter ailia).

heading: 5 Results
text: Table 2 report the results. The RTE and P̂ aRTE columns respectively report the models’ accuracy on the 1,126 unmodified and paraphrased sentence pairs.6 Comparing the difference in accuracy be-
5See Appendix A for more details, including hyperparameters, model sizes, and GPT-3 prompt design and configurations. Our code is available at https://github.com/ stonybrooknlp/parte
6Although there are just 914 unmodified sentence pairs, for the sake of a head-to-head comparison, we retain all instances
tween unmodified and paraphrased examples can be misleading. If the number of times a model changes a correct prediction is close to the number of times it changes an incorrect prediction, then the accuracy will hardly change. Figure 1 demonstrates why the accuracies do not change by much when models’ predictions change on paraphrased examples. Furthermore, if a model is robust to paraphrases, then it should not change its predictions when an example is paraphrased, even if the prediction on the original unmodified example was incorrect. Hence, our test statistic is the percentage of examples where a model’s predictions change (% ∆ P̂ aRTE column in Table 2) rather than a change in accuracy.
Compared to the Transformer based models, the BoW and BiLSTM models seem to be more sensitive, and less robust to paraphrasing, as they change their predictions on 15.27% and 16.69% respectively of the 1,126 examples. However, this might be associated with how word xembedding models only just outperform random guesses in and perform much worse on RTE compared to the Transformer models.
of the unmodified sentence pairs when computing accuracy.
Focusing on the Transformer models, we noticed that RoBERTa performs the best on the datasets and is the most robust to paraphrasing - changing its predictions on just under 8% of paraphrased examples. Interestingly, when the models are trained specifically to perform this task, the models change their predictions on fewer paraphrased examples as these models’ accuracy increases. However, improving performance alone might not automatically improve models’ robustness to paraphrases. GPT3’s accuracy noticeably outperforms BERT’s accuracy, but GPT-3 changes its predictions on more paraphrased examples compared to BERT.
P’-H’ compared to P-H’ or P’-H Figure 2 shows noticeable increases in the percentage of changed predictions when both premise and hypothesis are paraphrased compared to when just one of the sentences is paraphrased. Specifically, for BoW and BiLSTM we see an increase of 4.01 and 6.01 percentage points respectively, and for BERT, Roberta, GPT-3 increases of 4.97, 4.83, and 3.55. As the transformer-based models changed their predictions on 12-14% of examples where both sentences are paraphrased compared to 9-11% in general, this analysis further suggests that these models are not as robust to paraprhases as desired.
Entailed vs Not-entailed examples RTE analyses often differentiate how models perform on entailed vs not entailed examples (Liu et al., 2022). In Figure 3, we do not see meaningful differences in how models’ predictions change on paraphrased examples based on the gold label. This might suggest that our dataset does not contain statistical irregularities based on the RTE labels.
Correct vs Not-Correct Predictions Figure 4 shows that the Transformer models’ predictions is more likely to change when it’s prediction on an original example was incorrect (right red bars) compared to when the prediction for an original example was correct (left blue bars). For example, when RoBERTa’s prediction for an original RTE example was correct, the model changed its prediction on just 5.5% of the corresponding paraphrased examples. When RoBERTa’s predictions for an original RTE example were incorrect, RoBERTa’s predictions changed for 20.88% corresponding paraphrased examples. Analyzing differences in models’ confidences assigned to predictions might provide more insight (Marcé and Poliak, 2022). We leave this for future work.
Source Task RTE1-3 examples originated from multiple domains and downstream tasks, e.g. question-answering (Moldovan et al., 2006), information extraction (Grishman and Sundheim, 1996), and summarization (Evans et al., 2004; Radev et al., 2001). This enables researchers to evaluate how
RTE models perform on examples that contain different aspects of open domain inference necessary for the task (MacCartney, 2009). Figure 5 reports the changes in models’ predictions across the different sources of examples. We do not see consistent trends across the original data sources.

heading: 6 Conclusion
text: We introduced P̂ aRTE, a high-quality evaluation set of RTE examples paired with paraphrased RTE examples. We use our evaluation set to determine whether RTE models are robust to paraphrased examples. Our experiments indicate that while these models predictions are usually consistent when RTE examples are paraphrased, there is still room for improvement as models remain sensitive to changes in input (Jia and Liang, 2017; Belinkov and Bisk, 2018; Iyyer et al., 2018). We hope that researchers will use P̂ aRTE to evaluate how well their NLU systems perform on paraphrased data.
Limitations
Our results nor evaluation set cannot be used to indicate whether RTE models trained for other languages are robust to paraphrases. However, researchers can apply the methods we used to develop P̂ aRTE to build evaluation sets in other languages to test whether non-English NLU systems are robust to paraphrases.
Ethics Statement
In conducting our research on RTE model robustness to paraphrasing, we take great care to ensure the ethical and responsible use of any data and models involved. We adhere to the principles of fairness, transparency, and non-discrimination in
our experimentation and analysis. Furthermore, we take measures to protect the privacy and confidentiality of any individuals crowdsource workers. We also strive to make our evaluation set and methods openly available to the research community to promote further study and advancement in the field of Natural Language Processing.

heading: A Experimental Implementation Details
text: This section describes the model implementations for our experiments. For our work we trained/finetuned three different models - Bag of Words (BoW), BiLSTM, BERT-large with a classification head and RoBERTa-large with a classification head. Each model was trained on the MultiNLI training dataset (Williams et al., 2018) and validated on the paraphrased RTE dev set we created. Each model was implemented using PyTorch. All transformer based models were downloaded from HuggingFace.
A.1 BoW
The BoW model consisted of GloVe (300 dimension embeddings trained on 840B CommonCrawl tokens) (Pennington et al., 2014b) vectors as the embedding layer. The average of all word vectors for the input sequence is treated as its final representation. The representations for the hypothesis and premises were concatenated and passed through three fully connected layers with ReLU activation units after each layer. We concatenate the premise, hypothesis, their absolute difference and their product and pass it into the first layer of the classifier. This input to the first layer is of 4 * embedding dimension and the output is of embedding dimension. Each subsequent hidden layer’s input and output dimensions are embedding dimension * embedding dimension.
The model was trained with a vocabulary size of 50,000, a learning rate of 0.005, the maximum sequence length was 50 and a batch size of 32. We force all sentences to be of maximum sequence length using truncation or padding where applicable. We train the model for 15 epochs and select the one that achieves highest validation accuracy for our experiments.
A.2 BiLSTM
The BiLSTM model consisted of GloVe (300 dimension embeddings trained on 840B CommonCrawl tokens) (Pennington et al., 2014a) vectors as the embedding layer. The average of all word vectors for the input sequence is treated as its final representation. The word vectors were passed through an LSTM unit. This unit was bidirectional, with 64 hidden units and 2 stacked LSTM layers. The representations for the hypothesis and premises were concatenated and passed through three fully connected layers with ReLU activation units after
each layer. We concatenate the premise, hypothesis, their absolute difference and their product and pass it into the first layer of the classifier. This input to the first layer is of hidden units * embedding dimension and the output is of embedding dimension. Each subsequent hidden layer’s input and output dimensions are embedding dimension * embedding dimension.
The model was trained with a vocabulary size of 50,000, a learning rate of 0.005, the maximum sequence length was 50 and a batch size of 32. We force all sentences to be of maximum sequence length using truncation or padding where applicable. We train the model for 15 epochs and select the one that achieves highest validation accuracy for our experiments.
A.3 BERT
We fine tuned the BERT-large model available on HuggingFace 7. We added a classification head on top of the model using the AutoModel API on HuggingFace. The model was trained for 5 epochs with a learning rate of 3e-6 using the Adam optimizer. In order to simulate larger batch sizes on smaller GPUs, we used gradient accumulation as well. We simulated a batch-size of 32 by accumulating gradients over two batches of size 16. The model which achieved the highest validation accuracy was used for our experiments.
A.4 RoBERTa
We fine tuned the RoBERTa-large model available on HuggingFace 8. We added a classification head on top of the model using the AutoModel API on HuggingFace. The model was trained for 5 epochs with a learning rate of 3e-6 using the Adam optimizer. In order to simulate larger batch sizes on smaller GPUs, we used gradient accumulation as well. We simulated a batch-size of 32 by accumulating gradients over 8 batches of size 4. The model which achieved the highest validation accuracy was used for our experiments.
A.5 GPT-3
We used a temperature of 0.0 for all the experiments to select the most likely token at each step, as this setting allow for reproducibility. response = openai.Completion.create(
model="text-davinci-003", prompt=prompt,
7https://huggingface.co/bert-large-uncased 8https://huggingface.co/roberta-large
temperature=0, max_tokens=1, top_p=1.0, frequency_penalty=0.1, presence_penalty=0.0
)
We restricted the model outputs to just one token. Only “yes" or “no" are considered valid answers. The model did not generate any output apart from these in all our experiments. We used the following prompt template: Premise: {sentence1} Hypothesis: {sentence2}
Does the premise entail the hypothesis? Answer:

heading: B Dataset Creation
text: The following process describes how we create a vetted, paraphrased version of the RTE dataset that tests whether models’ are robust to paraphrased input. First, we use a strong T5-based paraphraser to create three re-written sentences for each premise and hypothesis in the 2,400 pairs in the RTE1-3 test sets, resulting in 14,400 new sentences. To generate these paraphrases, we use top-k sampling during decoding.9 The re-writer model was fine-tuned on the Google PAWS dataset and can be found on Huggingface 10. To evaluate its ability to generate gramatically correct paraphrases, we sampled 100 sentence pairs with at least one valid paraphrase and manually went through them. Upon checking for grammaticality, we found a grammatical error in <8% of the sentences.
Since we want to test paraphrastic understanding beyond simple lexical replacement, we discarded the re-written sentences that had at most a 25% lexical overlap with the corresponding original sentence. We use Jaccard index as a measure of lexical similarity (1) where τs are the tokens in the original sentence and τp are the the tokens in the paraphrase.
Score = τs ∩ τp τs ∪ τp
(1)
To ensure that the re-written sentences are indeed sentence-level paraphrases for the original sentences, we relied on crowdsource workers to remove low quality paraphrases. The Amazon Mechanical Turk HIT is described in detail in subsection B.2. We retain any paraphrases that get a similarity score above 75 out of 100.
9k=120; top-p=0.95 10https://huggingface.co/Vamsi/T5_Paraphrase_
Paws
B.1 Manual Verification Before crowd sourcing to get the best paraphrase generated for a given sentence, we conducted manual evaluation to understand the average error rate of the paraphraser model used. As mentioned above, we sampled 100 sentence pairs with each pair having atleast one valid paraphrase. The paraphrases for these sentences were evaluated for grammatical errors. Any semantic errors are handled during crowd-sourcing.
The errors can roughly be classified into roughly three categories - repetition errors, tense errors and incorrect punctuation. Examples of each type can be found in Figure 6. Overall, we found the error rate to be small enough to continue using the paraphraser. We also asked MTurk workers to mark paraphrases as grammatically incorrect to ensure that the final dataset does not have any grammatically incorrect sentences.
B.2 MTurk HIT We used Amazon Mechanical Turk to identify ungrammatical paraphrases rate how well a generated paraphrase preserved the meaning of the original sentence. No filtering criteria was applied to crowdsource workers and were paid roughly $14.20 an hour.
Each annotator was presented with a reference sentence, a corresponding paraphrased sentences, and tasked to judge on a scale of 0 to 100 how closely a paraphrased sentence retains the meaning of the reference sentence. A similarity score of 100 means that the paraphrase is the exactly the same in meaning as the reference, while a similarity score of 0 means that the meaning of the paraphrase is irrelevant or contradicts the reference sentence. Additionally, the MTurk workers were asked to judge the grammaticality of the paraphrase by selecting whether the paraphrase was grammatically correct or now. Figure 7 includes the instructions we showed crowdsource workers for judging similarity between sentences.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations section at the end of the paper
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 3
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section 3, Appendix

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Appendix
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3, Appendix
C 3 Did you run computational experiments? Section 4-5, Appendix

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Not applicable. Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Appendix

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Appendix

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Appendix
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: We leave this for future work. Source Task RTE1-3 examples originated from multiple domains and downstream tasks, e. g. question-answering (Moldovan et al. , 2006), information extraction (Grishman and Sundheim, 1996), and summarization (Evans et al. , 2004; Radev et al. , 2001). This enables researchers to evaluate how
RTE models perform on examples that contain different aspects of open domain inference necessary for the task (MacCartney, 2009). Figure 5 reports the changes in models’ predictions across the different sources of examples. We do not see consistent trends across the original data sources