Geometry matters: Exploring language examples at the decision boundary
Authors: Debajyoti Datta
Shashwat Kumar
Laura Barnes
Tom Fletcher
Venue:  (2021)
Keywords: Natural Language Processing
Text Classification
Information Geomtery
Sentiment Analysis

Abstract:   A growing body of recent evidence has highlighted the limitations of natural language processing (NLP) datasets and classifiers. These include the presence of annotation artifacts in datasets, classifiers relying on shallow features like a single word (e.g., if a movie review has the word "romantic", the review tends to be positive), or unnecessary words (e.g., learning a proper noun to classify a movie as positive or negative). The presence of such artifacts has subsequently led to the development of challenging datasets to force the model to generalize better. While a variety of heuristic strategies, such as counterfactual examples and contrast sets, have been proposed, the theoretical justification about what makes these examples difficult for the classifier is often lacking or unclear. In this paper, using tools from information geometry, we propose a theoretical way to quantify the difficulty of an example in NLP. Using our approach, we explore difficult examples for several deep learning architectures. We discover that BERT, CNN and fasttext are susceptible to word substitutions in high difficulty examples. These classifiers tend to perform poorly on the FIM test set. (generated by sampling and perturbing difficult examples, with accuracy dropping below 50%). We replicate our experiments on 5 NLP datasets (YelpReviewPolarity, AGNEWS, SogouNews, YelpReviewFull and Yahoo Answers). On YelpReviewPolarity we observe a correlation coefficient of -0.4 between resilience to perturbations and the difficulty score. Similarly we observe a correlation of 0.35 between the difficulty score and the empirical success probability of random substitutions. Our approach is simple, architecture agnostic and can be used to study the fragilities of text classification models. All the code used will be made publicly available, including a tool to explore the difficult examples for other datasets.
  

Under review as a conference paper at ICLR 2021
GEOMETRY MATTERS: EXPLORING LANGUAGE EXAM- PLES AT THE DECISION BOUNDARY
Anonymous authors Paper under double-blind review
ABSTRACT
A growing body of recent evidence has highlighted the limitations of natural language processing (NLP) datasets and classifiers. These include the presence of annotation artifacts datasets, classifiers relying on shallow features like a single word (e.g., if a movie review has the word "romantic", the review tends to be positive), or unnecessary words (e.g., learning a proper noun to classify a movie positive negative). The presence of such artifacts has subsequently led to the development of challenging datasets to force the model to generalize better. While variety of heuristic strategies, such as counterfactual examples and contrast sets, have been proposed, the theoretical justification about what makes these examples difficult for the classifier is often lacking unclear. In this paper, using tools from information geometry, we propose a theoretical way quantify the difficulty of example in NLP. Using our approach, we explore difficult examples for several deep learning architectures. We discover that both BERT. CNN and fasttext are susceptible word substitutions in high difficulty examples. These classifiers tend to perform poorly on the FIM test set. (generated by sampling and perturbing difficult examples, with accuracy dropping below 50%). We replicate our experiments on 5 NLP datasets (YelpReviewPolinn SogouNews, YelpReviewFull and Yahoo Answers). On YelpReviewPPlarity we observe correlation coefficient of -0.4 between resilience to perturbations and the difficulty score. Similarly we observe a correlation of 0.35 between the difficulty and the empirical success probability of random substitutions. Our approach is simple, architecture agnostic and can be used to study the fragilities text classification models. All the code used will be made publicly available, including a tool to explore the difficult examples for other datasets.
AGNEWS,
INTRODUCTION
Machine learning classifiers have achieved state-of-the-art success in tasks such as image classification and text classification. Despite their successes, several recent papers have pointed out flaws in the features learned by such classifiers. Geirhos et al. (2020) cast this phenomenon as shortcut learning, where classifier ends up relying on shallow features in benchmark datasets that do not generalize well to more difficult datasets or tasks. For instance, |Beery et al. (2018) showed that an image dataset constructed for animal detection and classification failed to generalize to images of animals in new locations. In language, this problem manifests at the word level. Poliak et al. (2018) showed that models using one of the two input sentences for semantic entailment performed better than the majority class by relying on shallow features. Similar observations were also made by Gururangan al. (2018), where linguistic traits such as "vagueness" and "negation" were highly correlated with certain classes
In order to study the robustness of classifier, is essential to perturb the examples the classifier's decision boundary. Contrast sets by Gardner et al. (2020) and counterfactual examples by Kaushik ct al. 2020) are two approaches where the authors aimed at perturbing the datasets to identify difficult examples. In contrast sets, authors of the dataset manually fill in the examples near the decision boundary (examples highlighted small circles in Figure I to better evaluate the classifier perfor- mance. In counterfactual examples, the authors use counterfactual reasoning along with Amazon Mechanical Turk to create the "non- grattiituss changes." While these approaches are interesting, it's still unclear if evaluating these will actually capture classifier's fragility. Furthermore, these

---

Under review as a conference paper at ICLR 2021
Original Points
Largesi eigen value of fisher matrix
Figure 1: Quantifying difficulty by using the largest eigenvalue of the Fisher information metric (FIM) a) We show that contrast sets and counterfactual examples aren't necessarily concentrated near the decision boundary as shown in this diagram. Difficult examples are the ones shown in green region (close to the decision boundary) and this is the region where we should evaluate model fragility. b) We sample points from a two-component Gaussian mixture model. We next train a classifier to separate the two classes. e) Dataset colored by the eigenvalue of the FIM, difficult examples with higher eigenvalue lie closer to the decision boundary.
approaches significantly differ from each other and it's important to come up with a common way to reason about them.
Motivated by these challenges, we propose a geometrical framework to reason about difficult examples. Using method, we are able to discover fragile examples for state of the art NLP models like BERTby Devlin et al. (2018) and CNN (Convolutional Neural Networks) by Kim (2014). Our experiments using the Fisher information metric (FIM) show that both counterfactual examples and contrast sets are, in fact, quite far from the decision boundary geometrically and not that different from normal examples (circles and triangles in Figure D As such, it more important to perform evaluation on the examples lying in the green region, which represent confusing examples for the classifier, where even small perturbation (for instance, substituting the name of an actress) can cause the neural network to misclassify. It is important to note that this does not depend solely on the classifier's ' certainty as adversarial examples can fool neural networks into misclassifying with high confidence, was shown by SZegedy et al. (2013).
We now motivate our choice of using the Fisher information metric (FIM) in order to quantify the difficulty of an example. In most natural language processing tasks, deep learning models are used to model the conditional probability distribution p(y æ) of a class label y conditioned on the input Here x can represent a sentence, while y can be a sentiment of the sentence. If we imagine neural network as a probabilistic mapping between inputs to outputs, a natural property to measure the Kullback-eeibler (KL) divergence between the example and an perturbation around that example. For small perturbations to the input, the FIM gives a quadratic form that approximates, up to second order, the change in the output probabilities of neural network. Zhao et al. (2019) used this fact to demonstrate that the eigenvector associated with the maximum eigenvalue of the FIM gives an effective direction to perturb an example to generate an adversarial attack in computer vision. Furthermore, from an information geometry viewpoint, the FIM is Riemannian metric, inducing manifold geometry on the input space and providing a notion of distance based on changes in the information of inputs. To the best of our knowledge, this is the first work analyzing properties of the fisher metric to understand classifier fragility in NLP.
The rest of the paper is organized as follows: In Section 2, we summarize related work. In Section 3, we discuss our approach of computing the FIM and the gradient- based perturbation strategy. In Section 4, we discuss the results of the eigenvalues of FIM in synthetic data and sentiment analysis datasets with BERT and CNN. Finally, in Section 5, we discuss the implications of studying the eigenvalues of FIM for evaluating NL. models.
2

---

Under review as a conference paper at ICLR 2021
Table 1: CNN, IMDb dataset: Unlike the difficult examples (larger eigenvalue), word substitutions ineffective in changing the classifier output for the easier examples (smaller eigenvalue). In difficult examples synonym or change of name, changes classifier label. In easy examples, despite multiple simultaneous antonym substitutions, the classifier sentiment does not change.
Perturbed sentiment
Word substitutions
Fosstive Negative
Goin this movie, heard good things
Coming out it. wasn't really anaazed nor disappointed Simon Poçz plays a ruther childish churacker much like his other muvies. There were couple of hughs here and there- nothing too funny. Prohubly my favorite - preferred parts of the movie is when be dances in the club sceme locally gota try that out time lind myself in aclab. couple of stars here and there incloding: Megan Fox, Kinsen Dunst that chick
difficult example
(Amaz =525
X-Files, and Bridges. found quite amusing see a cameo appeanace Thandic Newion in scee, She course being in a prosious movie with Simon Pigz. Run Fabboy Run. 1 see a loss up, you'll either enjoy it to an extent or find a Estle dull. might add, Kirsten Dunst -+ Nicole Kidman, Emma Stone, Megan Fox, Tom Cruise, Juhnny Depp. Robert Downey is akorable in this muvic. :3
Negalive -+ Neg- ative ensy example (Amaz =00000))
I missed this movic is the cinema but had some idea in the back of my head that it was worth look, so when sow it ou the shelves in DVD I thought "time to watch it. Big mistakelchr foette bit long list of stars conaot save this tureyy sumely
An incompeeheniblle exceptionally delivered and poorly - brilliantly presented. Peruus would have made more sense I'd Robbins' novel but unless the film is completely different to the novel, and with Robbins assistng in the screepploy I doubt the novel would have be exeruciating exciliing read well chr />ette ist hope the actors were well paid they looked emboraasee Morita, Even Thurman has the grace to look uncomfortable times-chr /s-hhe />Save yourself around 98 minutes your life for something more woethubikk, like diniiing your roenalls or sorting out your sock drswer. Even when you ice it in the "under $5 dhon-sway hin at your local stoee, resist the wrge!
the
best movies
comprehensiile ploe poorly
this of celluloid and lately DVD blanks, take for example Pat
2 RELATED WORK
In NLP, machine learning models for classification rely on spurious statistical patterns of the text and use shortcut for learning to classify. These can range from annotation artifacts, as was shown by Goyal et al. (2017); Kaushik and Lipton (2018) Gururangan et al. (2018), spelling mistakes as Glockner
in McCoy al. (2019). or new test conditions that require world knowledge Gllooeegellockeerrrrr al. (2018).
Simple decision rules that the model relies on are hard to quantify. Trivial patterns like relying on the answer "2" for answering questions of the format "how many" for the visual question answering dataset Antol et al. (2015), would correctly answer 39% of the questions. Jia and Liang] (2017)
showed that adversarially inserted sentences that did not change the correct answer, would cause state of the art models to regress in performance in the SQuAD Rajpurkar et al.) (2016) question answering dataset. Gloeknnr et al. (2018) showed that template-basee -base modifications by swapping just one word from the training set to create test set highlighted models' failure to capture many simple inferences. Dixon et al. (0188) evaluated text classifiers using a synthetic test set to understand unintended biases and statistical patterns. Using standard set of demographic identity terms, the authors reduce the unintended bias without hurting the model performance. [hhnn et al. showed that word substitution strategies include stylistic variations that change the sentiment analysis algorithms for similar word pairs. Evaluations of these models through perturbations of the input sentence are crucial to evaluating the robustness of models.
Another issue of language recently has been that static benchmarks like GLUE by Wang et al. (2018) tend to saturate quickly because of the availability of ever-increasing compute and harder benchmarks are needed like SuperGlue by Wang et al.] (2019). A more sustainable approach to this is the development of moving benchmarks, and one notable initiative in this area is the Adversarial NLI by Nie et al. (2119), but most of the research community hardly validate their approach against this sort of moving benchmark. In the Adversarial NLI dataset, the authors propose an iterative, adversarial human-and- wowddd--oo------------e
solution for Natural Language Understanding dataset collection, where the goal post continuously shifts about useful benchmarks and makes models robust by training the model iteratively difficult examples. Approaches like never-ending learning byMitchell al.) (2018) where models improve, and test sets get difficult over time is critical. A moving benchmark is necessary since know that improving performance newly collected datasets under the same condition Recht et al. (2019); Beery et al. (2018). Therefore, it is essential to find difficult examples in a more disciplined way.
constant test set may not generalize to
Approaches based on geometry have recently started gaining traction in computer vision literature. Zhao et al. (2019) al used a similar approach for understanding adversarial examples in images.
3

---

Under review as a conference paper at ICLR 2021
3 METHODS
neural network with discrete output can be thought of as a mapping between a manifold of inputs the discrete output space. Most traditional formulations treat this input space as flat, thus reasoning that the gradient the likelihood in input space gives us the direction which causes the most significant change in terms likelihood. However, by imagining the input as a pullback of the output, we obtain non-linear manifold where the euclidean metric no longer suffices, A more appropriate choice thus is use the fisher information Riemannian metric tensor.
We first introduce the Fisher Metric formulation for language. For the purposes of the derivation below the following notations are used.
Vector of input sentence. This an * sentence where n the number of words in the sentence and is the dimensionality of the word embedding.
Label of class, in our context that is the positive or the negative sentiment.
p(yxx) The conditional probability distribution between y and x.
L q) The KL divergence between distributions p and q for two sentences
ff(z) (x) Gradient of function of f w.r.t
V fxx) Hessian of f(x) w.r.t x
We apply the perturbation ry to modifying sentence create sentence (eg., counterfactual example). We can then see the effect of this perturbation y in terms of change in the probability distribution over labels. Ideally, would like find points where small perturbation can result in a large change in the probability distribution over labels.
(L(p(wl=)llol + +)) = -ogpll)))gp((ul) + Eptylm)lggp((I= + n)
We now perform Taylor expansion of the first term on the right hand side
Po (logp(y/x)) Vlogp(yla) "/" (loopp(yl))) +..) + Peooeslogp((yl)
Since the expectation of score is and the first and last terms cancel out, are left with.
= Gn
Where G is the FIM. By studying the eigenvalues of this matrix locally, we can quantify if small change in 1) can cause large change in the distribution over labels. We use the largest eigenvalue of the FIM as score to quantify the "difficulty" of an example. We now propose the following algorithm to compute the FIM:
After getting the eigenvalues of the FIM. we can use the largest eigenvalue Amaz to quantify how fragile an example is linguistic perturbation.
points with largest eigenvalues, smaller perturbations can be much more effective in changing the classifier output. These examples, thus, are also more confusing and more difficult for the model to classify.
4

---

Under review as a conference paper at ICLR 2021
Table 2: CNN, Counterfactual Examples: In difficult examples (larger eigenvalue), individual synonym/antony substitutions are effective in changing the classifier output. In easy examples (smaller eigenvalue) multiple antonym substitutions simultaneously have no effect on the classifier output.
Perturbed sentiment
Word substitutions
Fositive
Negative
move was on last night. puess time filer. because incrodible! The movie is just entervaimens piece to show some talent at the start and thruggouu. (Not bad talent at all). Bat the story too brillians. for words. The "wolf" of that is what you can call it, is hardly shown fully save his teeth. When is fully in view, you can clearly they had interss woeking on the CGL because the wall runs like he's like
difficult example
Amaz =4.3
CGI fur sput who going get killedslnsheteesteee next. Even if you like these kind of splatter movies you will be surprised, they good job at it-she ho-chr /=Don't I even get me started on the actoes... Very amazing lines and the girls hardly scream any thing. But then again. someone asked sign up bebb
treadmill,
asedd shiny tehr bebr /hhh
gore and blood, you
give few backs, then bey, where du I Sveerall exciting boring, extraordinary, uninteresting, exceptional frighsening horroe.
de good acting
Negative ative easy example
Neg-
couldits stand this movic is definite waste movic. It hils you with bocadom. This movic noe worth the wonth buying, should everyones Bosheas, Excellent movie gets only worse. is story cheesy romance, bud - good drama, action, and plenty of unfunny - Tunny lines to keep you rolling your eyes. hated lot of the quotes I use them all the time in mooking the film. They did not help keep shows that anyoss movin, but every time watch 1 deadr that as long as is. get so bored in it, that goes slow, I hated this movic. I nover went to watch it again.
long time.
Ameai -0.013
make you bocause everyone
mean
task of what I want to do. whise about until they get their way.
achieve
they have
Algorithm 1 Algorithm for estimating difficulty of an example
Input: x, f
Output: Amax
Calculate probability vector:
1: p =(x)
Calculate Jacobian of log probability w.r.x
2: J == alooo
Duplicate probability vector along rows to match J's shape
3: P. = duplicate( (p, [0]
Compute the FIM
4: G =PeJ.
Perform eigendecompositiion to get the eigenvalues
5: Xg Ug = eigendecomposition...
6: return max
3.1 GRADIENT ATTRIBUTION BASED PERTURBATION
In our perturbations, we rely on Integrated Gradients (IG) by Sundararajan et al. (2017), since IG satisfies both sensitivity (network gradients to focus on relevant input feature attributes the model with respect to the output) and implementation invariance of the neural network. IG allows us to assign relative importance to each word token the input. After computing the token attributions with IG, we only perturb those words for "easy" and "difficult" examples. Once we compute the feature attribution for each input word in the sentence, we find the most important words by thresholding the attribution score. We then test the classifier's fragility by substituting these words for their synonymssantonyms,, etc, and checking if the classifier prediction changes. We show that easy examples are robust to significant edits with multiple positive tokens like "excellent", "great" substituted simultaneously, and "difficult" examples change predictions from positive to negative with meaningless substitutions like names of actors and actresses.
4 DISCUSSION AND RESULTS
4.1 FIM REFLECTS DISTANCES FROM THE DECISION BOUNDARY
We first investigate the FIM properties by training a neural network on a synthetic mixture of gaussians dataset. The parameters of the two gaussians are μl 1-2,- -2] and #2 [3.5.3. 3.5). The covariances are Σ = eye(2) and Σn = [[.., 1.], [1., The dataset is shown in figure [ We train a 2-layered network separate the two classes from each other. We use algorithm to compute the largest.
5

---

Under review as a conference paper at ICLR 2021
eigenvalue of the FIM for each datapoint and use it to color the points. We also plot the eigenvector for the top 20 points.
As seen by the gradient of the colors in Figure 1, the points with the largest eigenvalue of the FIM lie close to the decision boundary. These points are indicative of how confusing the example is to the neural network since small shift along the eigenvector can cause a significant change in the KL divergence between the probability distribution of the original and new data points. These points with high eigenvalue are close to the decision boundary, and these examples most susceptible to perturbations.
4.2 FIM VALUES CAPTURE RESILIENCE TO LINGUISTIC PERTURBATIONS, QUALITATIVE EXPLORATION
4.2.1 CNN
For the difficult examples, we tried the following trivial substitutions one at a time: a) a synonym b) a semantically equivalent word c) an antonym d) substituting the name of actress present in the same passage. As seen in Table D either replacing favorite with preferred or "Kirsten Dunst" with any of the listed actors/actresses suffices to change the classifier's prediction. Note that, "Megan Fox's 's" name appears in the same review in the previous sentence. Similar in Table 3 for counterfactual examples, it's sufficient to replace "exciting" with either an antonym ("boring" or "uninteresting") or synonym ("extraordinary" or "exceptional"). We see the same pattern in contrast sets in Appendix.
For easy examples however, despite trying to replace four or more high attribution words simultane- ously with antonyms, the predicted sentiment did not change. Substitutions include "good" to "bad", "unfunny" to "funny", "factually correct" to "factually incorrect". Even though the passage included words like "boredom", word that is generally associated with negative movie review, the model did not assign it a high attribution score. Consequently, we did not try to substitute these words for testing robustness fragility of word substitutions.
YelpReviewPolarity
Perturbation Accuracy YelpaoviewPolarity
100
dassiifer jacuracy
example type easy
60
example type easy
dificuut
40
afficuuit
20-
0-
200
400
1000
1200
0.3
0.2
0.3
04
0.6
0.
num esammllss
weight wrighh_prtuubbttion
Figure 2: Left figure: The difficult examples on the FIM test set are challenging for the classifiers with accuracy between 1-5%. For the easy examples, (low FIM Amar) the accuracy ranges between 60- 100%. Right figure: We plot the classifier accuracy with respect to the 12 norm of the perturbation in embedding space. For difficult examples, the classifier accuracy drops below 5% at small perturbation strengths (12 norm < 0.3)
4.2.2 BERT
BERT: Transfer learned models like BERT capture rich semantic structure of the sentence. They are robust to changes like actor names and tend to rely semantically relevant words for classifying movie reviews. As we can see from Table difficull BERT examples, even with multiple positive words, tend to predict a negative sentiment when only one of the positive word is substituted. Even with words like "fantastic", "terrific" and "exhilarating", just changing "best" to "worst" changed the entire sentiment of the movie review. Easy examples for BERT require multiple simultancous word substitutions to change the sentiment as can be seen in Table Unlike CNN models, BERT is significantly more robust meaningless substitutions like actor names.

---

Under review as a conference paper at ICLR 2021
YelpReviewPolaanty p-value 946110076161179 r-value 0.393
YelpRevieePooaatyy pivalue L035219055196215e rvluue
percentage fippert perturtatioo
0.8
06
0.4
0.2
0.0
1og largest eigenssluess
log largent esgenwalless
Figure 3: On YelpReviewPPlaarty we observe correlation coefficient of -0.4 between minimum perturbation strength and the difficulty score. Similarly, we observe correlation of 0.35 between the difficulty score and empirical success of random word substitutions. This suggests that fim eigenvalue captures perturbation sensitivity in both embedding space and word substitutions.
Hetggram de
ofherce pppuutatte DEPESS Cigatind METUCI DN
Figure 4: a) Distribution of difference in largest eigenvalue of FIM of the original and the perturbed sentence in contrast set and counterfactual examples for BERT and CNN. With mean near 0, these perturbations are not difficult for the model. Adhoc perturbations are thus not useful for evaluating model robustness.
4.3 HIGH FIM EXAMPLES ARE CHALLENGING TO NLP CLASSIFIERS, QUANTITATIVE EXPERIMENTS
4.3.1 BAG OF TRICKS FOR EFFICIENT TEXT CLASSIFICATION
When the dataset size is increased, simple models perform competitively to models like BERT Wang et al. (2020). We use the bag of tricks for efficient text classification Joulin et al. (2017) for its inference and training speed and strong performance on text classification on 5 popular text classification datasets (YelpReviewPolarity. AGNEWS, SogouNews, YelpReviewFull and Yahoo Answers)
In order to investigate the relationship between classifier performance and FIM eigenvalues we select n-smallest and n-largest fim examples from the test set. We randomly chose perturbation strength between and and we perturb the example along the largest eigenvector proportional to the chosen perturbation strength. As can be seen from figure 2 left, despite sampling large range of difficult examples, between (125-1200,, the classifier performed poorly on the difficult set with the accuracy failing exceed 9%. On the easy set however, the classifier accuracy never falls below
7

---

Under review as a conference paper at ICLR 2021
57.5%. In figure 2b, we plot the classifier aceuracy function of perturbation strength (I2 norm of perturbation vector for difficult examples, even small perturbations upto 0.4 12 norm, cause a 17% reduction in accuracy. For easy examples however even upto a 12 norm of barely result in any drop in performance. More examples are Appendix A.
We randomly sample 500 examples and perturb the examples along the largest eigenvector. We use binary search to determine the minimum perturbation strength sufficient to flip the classifier output. As we can see from Figure 3, a correlation of -0.4 implies is possible misclassify larger FIM examples by applying a small perturbation strength. In order to investigate the relationship between FIM and random word substitutions, we flipped 10% of the words in document, for 500 randomly sampled examples. As seen in Figure 3b, a correlation of 0.35 implies that difficult examples, (larger FIM eigenvalues) are more susceptible to random word substitutions.
4.4 Do CONTRAST SETS AND COUNTERFACTUAL EXAMPLES LIE CLOSE TO THE DECISION BOUNDARY?
We first plot the distribution of eigenvalues of IMDb examples, counterfactual examples and contrast sets examples in Figure 2. If the goal evaluate examples near the decision boundary the contrast set/counterfactual eigenvalue distribution should he shifted to the right with very little overlap with the original examples. However, as we can see from Figure 2, we see a 69.38% overlap between contrast sets and IMDb examples as well as 73.58% overlap between counterfactual and IMDb examples for the CNN model that was only trained on IMDb training set. The presence of significant overlap between the three distributions indicate that counterfactual/contras examples are not more difficult for the model compared to the normal examples. Furthermore, since the FIM capture distance from the decision boundary, most counterfactual and contrast sets lie far away from the decision boundary as the original IMDb dataset examples.
We next quantify the effect of changing the original sentence to the perturbed sentence in counter- factual/contrast test sets. We plot the distribution of difference in eigenvalue of the original and perturbed sentence in Figure 2. 30% of the counterfactual examples (area the right of O) did not increase the FIM eigenvalue and the perturbation was ineffective. For contrast sets, this number is around 70% (area the right of 0), hinting that majority of perturbations are not useful for testing model robustness. A subtle point to note here, even though 70% of counterfactual examples increase the difficulty, it's because we have chosen a weak threshold (0) to quantify the usefulness of the perturbation. A more practical threshold like would lead to lesser number of useful examples for both counterfactual and contrast set examples.
5 IMPLICATIONS
5.1 CONSTRUCTING TEST SET BY PERTURBING HIGH FIM EXAMPLES
As evident from figure 2, high FIM examples are particularly challenging for NLP classifiers. This makes them useful to understand model performance. By sampling the high FIM examples and perturbing them slightly in embedding space, we can construct a new test set to evaluate NLP models. By repeating this process multiple times, more robust classifiers can be created.
5.2 NLP MODELS SHOULD BE EVALUATED AT THE EXAMPLES NEAR THE DECISION BOUNDARY
Deep learning models because of their high representation capacity are good at memorizing the training set. In the absence of sufficient variation in the test set, this can lead to an inflation in accuracy without actual generalization. However, examples near the decision boundary of classifier, show the fragilities of these classifiers. These are the examples that are most susceptible to shallow feature learning and thus are the ones that need to be tested for fragility and word substitutions. Our approach based on FIM score, provides a task and architecture agnostic approach to discovering such examples.
8

---

Under review as a conference paper at ICLR 2021
Table 3: Statistics of difference in largest FIM eigenvalue (pre and post perturbation) for counterfactual examples and contrast sets.
Architecture
Dataset
Type
Mean
Std
CNN
Counterfactual Examples
Dev -0.55
1.17
Test
-0.57
1.22
CNN
Contrast Sets
Dev
0.56
0.89
Test
0.65
1.15
BERT
Counterfactual Examples
Dev
0.004
0.10
Test
0.003
0.11
BERT
Contrast Sets
Dev
0.02
0.10)
Test
-0.01
0.11
5.3 INVESTIGATE IIIGI FIM EXAMPLES WITH INTEGRATED GRADIENTS TO UNDERSTAND MODEL FRAGILITY
For our models, difficult examples have mix of positive and negative words in movie review. The models also struggled with examples of movies that selectively praise some attributes like acting (e.g., "Exceptional performance of the actors got me hooked to the movie from the beginning") while simultaneously use negative phrases (e.g., "however the editing was horrible"). Difficult examples also have high token attributions associated with irrelevant words like "nuclear," "get," and "an." Thus substituting one or two words in difficult examples change the predicted label of the classifier. Similarly, easier examples have clearly positive reviews (e.g.. "Excellent direction, clever plot and gripping story"). Combining integrated gradients with high FIM examples can thus yield insights into the fragility of NLP models.
5.4 PERTURBATION SHOULD BE PERFORMED ALONG THE LARGEST EIGENVECTOR
The eigenvector corresponding to Amaz represents the direction of largest change in probability distribution over output. This makes it a useful candidate direction to evaluate model fragility while random word substitutions are easier to interpret, they might end up being orthogonal to the largest eigenvector and thus not capture model fragility effectively. One strategy to circumvent this issue is to find word substitutions with large projection on this eigenvector.
6 CONCLUSION
We have proposed a geometrical method to quantify the difficulty of an example in NLP. Our method identified fragilities in state of the art NLP models like BERT and CNN. By directly modeling the impact of perturbation in natural language through the Fisher information metric, examples close to the decision boundary are sensitive to meaningless changes. We also showed that counterfactual examples and contrast sets don't necessarily lie close the decision boundary. Furthermore, depending on the distance from the decision boundary, small innocuous tweaks to a sentence might actually correspond to a large change in the embedding space. Thus one has to be careful in constructing perturbations, and more disciplined approaches are needed to address the same.
showed that
As our methods are agnostic to the choice of architecture or dataset. in future we plan to extend this to other NLP tasks and datasets. We are also studying the properties of the decision boundary: whether different classifiers have similar decision boundary, are there universal difficult examples which confound multiple classifiers? We also plan to investigate strategies for automatic generation of sentence perturbations based on the largest eigenvector of the FIM. Even though we have explored difficult examples from a classifier's 's perspective, we are also interested in exploring the connections between FIM and perceived difficulty of examples by humans.
REFERENCES
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international

---

Under review as a conference paper at ICLR 2021
conference on computer viston, pages 2425-2433, 2015.
Sara Beery, Grant Van Hom, and Pietro Perona. Recognition in Terra Incognita. pages 456- 473, 2018. URL httttttt/ppppecceeeeooo Beery Recognition in Terra BCCV 2018 paper paper.html
thecvf. wtt.e/ccco/cooccecccoececo 201 8 8/html/
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800 2018,
Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 67-73, 2018.
Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Flazar, Ananth Gottumukkala, Nitish Gupta, Hanna Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou, Evaluating NLP Models via Contrast Sets. Stti:200040227000 [cs], April 2020. URL arXiv: 2004.02709.
http://arxiv. org/ abs 2004 02709
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut Learning in Deep Neural Networks. arXiv:2004. 07788 [cs, q-bio], April 2020. URL http:://www...oppp///p/
20t.//00000000000000 arXiv: 2004.07780.
Max Glockner, Vered Shwartz, and Yoav Goldberg. Breaking nli systems with sentences that require simple lexical inferences. arXiv preprint arXiv:1805.02266. 2018.
Yash Goyal, Tejas Khot, Douglas Summers-Star Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6904-6913, 2017.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. Annotation Artifacts in Natural Language Inference Data. arXiv:1803.0m 02324 [cs], arXiv: 1803.02324.
April 2018. URL
Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. arXiv preprint arXiv: arXiv: 1707.07328, 2017.
Armand Joulin, E. Grave, P. Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. ArXiv, abs/1607.017553 2017.
Divyansh Kaushik and Zachary C Lipton. How much reading does reading comprehension require? critical investigation of popular benchmarks. arXiv preprint arXiv: arXiv:/808.044 04926, 2018.
Divyansh Kaushik, Eduard Hovy, and Zachary C. Lipton. Learning the Difference that Makes a Difference with Coumerfaccuallyy-ngmeeeedd Data. arXiv:1909 12434 [cs, stat], February 2020. URL htt.:tt://aax...oo////o www.bb///..... 12434
arXiv: 1909.12434.
Yoon Kim. Convolutional neural networks for sentence classification. arXiv preprint arXiv: 1408 2014.
R Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. arXiv preprint arXiv:1902.01007 2019.
Tom Mitchell, William Cohen, Estevam Hruschka, Partha Talukdar, Bishan Yang, Justin Betteridge, Andrew Carlson, Bhanava Dalvi, Matt Gardner, Bryan Kisiel, al. Never-ending learning. Communications of the ACM, 103 2018.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial nli: A benchmark for natural language understanding. arXiv preprint arXiv: 14599, 2019.
10

---

Under review as a conference paper at ICLR 2021
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. Hypothesis Only Baselines in Natural Language Inference. Proceedings of the Seventh Joins Conference on Lexical and Computationnl Semantics, pages 180-191, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 1/118663/1/1118-0000 URL https
7/ 7/www.o ac web www.::w............ antho wow.tthhhoooyyyooooooooo 81
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang, Squad: 100,000+ questions for machine comprchension of text. arXiv preprint arXiv: 1rX.0::600.05250,, 2016.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? arXiv preprint arXiv: 1902. 10811, 2019.
Judy Hanwen Shen, Lauren Fratamico, lyad Rahwan, and Alexander M Rush. Darling or babygirl? investigating stylistic bias in sentiment analysis.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning-Voiee Volume 70, pages 3319- 3328. JMLR. org, 2017.
Christian Szegedy, Wojeiech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 2013.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint 180400000 2018.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in Neural Information Processing Systems, pages 3261-3275, 2019.
Sinong Wang. Madian Khabsa, and Hao Ma. To pretrain or not to pretrain: Examining the benefits of pretrainng on resource rich tasks. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2209-2213, 2020.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac. Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771, 2019.
Chenxiao Zhao, Thomas Fletcher, Mixue Yu, Yaxin Peng, Guixu Zhang, and Chaomin Shen. The adversarial attack and detection under the fisher information metric. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5869-5876, 2019.

---

Under review as a conference paper at ICLR 2021
APPENDIX A
CNN and BERT:
We train convolutional neural network (CNN) with 50dd GloVe embedding on the IMDb dataset, and calculate the eigenvalue of the FIM for each example. The accuracy on the IMDb test set around 85.4%. For all experiments in the paper the model examples in the original IMDb training split with 90% train and 10% valid split. For all experiments in the paper, we did not train on the counterfactual or contrast examples fairly evaluate the robustness to perturbations of contrast sets and counterfactual examples. For BERT we finetuned beet-base-uncased and achieved an accuracy of 92.6% using huggingface transformers by Wolf et al. (2019). We evaluated the largest eigenvalue of the dev and test sets of the contrast set and counterfactual examples datasets of IMDb.
trained the original 25000
Bag of Tricks for efficient text classification:
We use unigram and bigram word embeddings. The latent dimension We use torchtext datasets directly available for the training and test splits. We use the SGD optimizer with a learning rate of and train each model for epochs. We use a learning rate scheduler with gamma of 0.9.
12

---

Under review as a conference paper at ICLR 2021
APPENDIX B
YelpReviewFull
Perturbation Accuracy YelpReviewFull
example
easy dificuit
atcuracy
example easy difficuit
50
dass.ter
40
20
200
400
600
800
1000
1200
01
0.2
0.3
944
0.5
0.6
0.7
00
perturbetion
SogeuNews
Perturbation Accuracy SogouNews
100
90
Gassifer pecuracy
example type easy
70
dificutt
example type
easy
dificcltt
30
200
400
600
1000
1200
0.1
0.2
0.3
0.4
0.6
08
num, examples
Neight, perturbation
NEWS
Perturbation Accuracy
90
dasc-ieer aecuracy
80
exemple type easy
dificcut
60
example_ _ype
easy
dificclt
200
1000
1200
0.
0.6
num esamples
wright perturbation
13

---

Under review as a conference paper at ICLR 2021
YahooAnswers
Accuracy YahooAnswers
90
example type
example
easy dificutt
60
classter accuracy
50
80
10
30-
10
num examples
weight perturbation
Figure 5: Here we take the Joulin et al. (017)) text classification approach ( and plot classifier accuracy function of the step size of the perturbation. The number of examples in the x-axis represents the number of examples based on the cigenvalue. So, 200 refers to the 200 easy examples and 200 difficult examples. We then perturb these examples in the direction of the eigenvector and check if the classifier prediction flipped. As can be seen that the classifier accuracy remains very high for easy examples and significantly low for difficult examples. On the second diagram, for easy examples, the classifier still exhibits minimal performance drop when the weight of the perturbation along the eigenvector is increased. Thus our method dataset agnostic.
YelpReviewFult: v-vuu 743709911449907e 106 vvalue: O 0.302
SogouNews p-vauue 1.6126677760886272.2228 ravalue:
355
3.0
perfurbation
12 nome of perturtation
2.5
2.0
2.5
lag engenvalurss
largest eigenvalues
NEWS 2.744924485/1057744 value -0.103
12 mommme of perturbitton
largest eigenvalues
Figure 6: use the eigenvector with the largest eigenvalue as a perturbation embedding space. As see there is linear relationship between log lambda max and the perturbation strength needed to flip the classifier output. We use binary search between the range (0,6) to discover the minimal 12 norm perturbation along the largest eigenvector which can successfully flip the classifier's output. For each dataset, we select 500 random examples from the test set for the perturbation. Note the negative slope all the datasets. The p-luuue and the r-value are reported the top. The log of the largest fim eigenvalue is on the x-axis. Thus, an example with small eigenvalue requires larger perturbation flip classifier prediction than an example with large fim eigenvalue.
14

---

Under review as a conference paper at ICLR 2021
YelpileviewFull p-value 361480880098451188 4e-17 r-value:
SognuNews
31997742018870077 value
1.0
percentage tippee on perturbetion
seriirtaation
0.0
0.8
0.6
0.4
0.
10
14
16
14
16
log of largest fimm eigensations
log of largest fimm aigervaluts
AG. NEWS p value 1. 10510708171959983 132 r-value: 0
percentage Rupped on centurtation
0.8
0.4
0.0
largest eigervalues
Figure 7: We notice the linear relationship between the log of eigenvalue and the percentage of successful word flips. Since the length of each document varies drastically for these document classification problems, we decide the number words to flip as 10% of the document length with the vocabulary of the dataset. We then measure the percentage of predictions whose classification label changes and the log of fim eigenvalue. The hr www.rnnddeeeeeeeeererrrrooooaaaa
APPENDIX C
Litaprrr harpet of
teceer belween orgineledd servieed
Call bet 20
0/6
016
018
Figure a) Histogram of eigenvalues of dev sets original IMDb examples, contrast sets and counterfactual examples. The significant overlap between the three distributions indicates that the counterfactual and contrast set examples might not be as difficult as previously believed. The tail end of all three distributions contain the difficult examples. b) Distribution of difference in largest cigenvalue of FIM of the original and the perturbed sentence in contrast set and counterfactual examples. With mean near 0, these perturbations not necessarily more difficult for the model.
15

---

Under review as a conference paper at ICLR 2021
Table 4: Perturbing easy example on the IMDb dataset: The first row represents original sentence.
As we can see here, most perturbations are ineffective in changing the FIM eigenvalue and thus the difficulty of the example. Despite multiple substitutions (last row), we are only able to achieve modest increase in FIM score, indicating the resilience to perturbations.
Perturbed
Word substitutions
timent
Negative
Probably the worst Dolph film ever. There's nothing you'd want or expect here. Don't waste your time. Dolph plays miserable cop with no interests in life. His brother gets killed and Dolph tries to figure things out. The character is just plain stupid and stumbles around aimlessly. Pointless.
Negative
easy example
(Aoez 0000077
Negative
Prohahly the worst Dolph film ever. There's nothing - everything you't d want expect here. Don't t waste your time. Dolph plays miserable cop with interests in life. His brother gets killed and Dolph tries to figure things out. The character just plain stupid and stumbles around aimlessly. Pointless.
Negative mi-
nor change FIM value
(Ames 0.0008)
Negative Negative mi- nor change in FIM value Ovnez -0.0005)
Probably the worst Dolph film ever. There's 's nothing you't want or expect here. Don't waste your time. Dolph plays - portrays a miserable cop with no interests in life. His brother gets killed and Dolph tries to figure things out. The character is just plain stupid and stumbles around aimlessly. Pointless.
Negative Negative mi-
Probably the worst Dolph film ever. There's nothing you's 'd want or expect here. Don't waste your time. Dolph plays portrays miserable cop with interests life. His brother gets killed and Dolph tries + attempts figure things out. The character is just plain stupid and stumbles around aimlessly. Pointless.
nor change FIM value (Amaz -.00077
Negative
Probably best Dolph film ever. There's everything you'd want or expect here. Spend your time. Dolph portrays a miserable cop with lots of interests in life. His brother gets killed and Dolph attempts to figure things out. The character is just plain amazing.
Negative Significant in- crease in FIM (Amaz 1.56)
16

---

Under review as a conference paper at ICLR 2021
Table 5: Perturbing a difficult example on the IMDb dataset: The first row represents original sentence. Unlike easy examples, difficult examples tend to have a mixture of positive and negative traits. Furthermore, a minor perturbation like removal of a sentence (row 1) or substitution of word (row 2) causes a significant drop in FIM eigenvalue. The small FIM score indicates that the perturbed review is not difficult for the classifier.
Perturbed sen-
Word substitutions
timent
Negative +
It really impresses me that it got made. The dircetor//rritrraccoor must be really charismatic in reality. I can think of no other way itd pass script stage. What I want you consider this...wiile watching the films I was feeling sorry for the actors. It felt like being in a stand up comedy club where the guy is dying on his feet and your sitting there, not enjoying it, just feeling really bad for him coz hes of trying. Id really like to know what the budget is, guess it must have been low the film quality is really poor. want to write 'the jokes didn't appeal to me'. but the reality is for them to appeal to you. you'd have be the man who wrote them. or a retard. So imagine that in script fod this guy got THAT green lit. Thats impressive isn't it?
Negative diffi- cult example (hoass 55.77
Negative
It really impresses mc that it got made. The dircctor/wwrierraccor must be really charismatic in reality. I can think of no other way itd pass script stage. What I want you consider is this... watching the films I was feeling sorry for the actors. It felt like being in stand up comedy club where the guy is dying on his feet and your sitting there, not enjoying it, just feeling really bad for him coz hes of trying. Id really like to know what the budget is, guess it must have been low as the film quality is really poor. want to write 'the jokes didn't 't appeal to me'. but the reality is for them appeal to you, you't 'd have be the man who wrote them. or retard. So imagine that in script and this guy got THAT green lit. Thats impressive isn't it?
Negative
Significant change in FIM (Amaz 0.640)
Negative
It really impresses me that it got made. The director//rrier/cccoo must be really charismatic in reality. I can think of no other way itd pass script stage. What I want you consider hiis...hiile watching the films I was feeling sorry for the actors. It felt like being in a stand up comedy club where the guy is dying on his feet and your sitting there, not enjoying it, just feeling really bad for him coz hes of trying. Id really like to know what the budget is, guess it must have been low the film quality is really poor. want to write 'the jokes didn't appeal to me'. but the reality is for them appeal to you, you't 'd have be the man who wrote them. or retard. So imagine that in script for this guy got THAT green lit. Thats impressive + weird isn't ' it?
Negative
Significant change in FIM (Amux 0.445)
Table 6: BERT: Difficult Examples change sentiment with a single word substituted. Easy examples, however, retain positive sentiment despite multiple substitutions of positive words with negative words.
Perturbed seatiment
Word substitutions
Fositive
Negative
demograghic, and kinda ideundy sometimes lacky and meaningful dialogue seems realistic. and in budtr b-1 in trying as hard as possible not to trash this movie like the others did, but it's easy when the filmmakes were trying very hard <be b-cht The editing in this movie terifie! Possibly the best - worst editing I've ever seen editing eves go into the
OK, kinda sdea this movie. I'm
of movic would have been forgivable
liven
difficult example
Donaz 0.78)
movie! There things that you don't but adentitying a
film school learn, leaning
them.
is.dhr /b-che buAlio, the shot... my the shoes, just fantesti! details, but we sometimes just see random things popping up, and that, in conjunction with the editing will give you the most exhilirating film viewing experience <hr (5-bbe This movie being made en low or no budget with d casi and also. I've seen short films youtube
lot less artistic integrity!
Posétive - Pusi- live (Amaz -055)
This is the best and must original show seen years. The more I wasch it the more Talll in love with - hale st. The cnst is excellent terrible, the writing great bad. I personally loved + hated every chameter. However. everyone there good
example
backgrunds just like real life. helieve ABC his done great service to the writers, actors and the potentia auchence of this show, to cancel so quickly and not adverisc it enough nor give it real chance to goin following There are so fow shows I watch anymore as most TV awful. This show opinion right down there with favorites Greys Anaoomy and Brothers and Sisters. fact think audience Brothers Sisters hale show they even knew about
personaliiis
17

---

Under review as a conference paper at ICLR 2021
Table 7: Counterfactual perturbations cause reduction in difficulty: The original example is more difficult than the perturbed counterfactual example because of strong giveaway words like "amazing". The subtle changes make the sentence positive like changing the amount the rating almost makes no difference. Instead, the use of strong word "amazing" makes the sentence extremely casy for the model to classify as positive. The heavy reliance on a single word for the positive example makes this much easier to classify than the original sentence which used the word "odd" (a word that is not necessarily negative) as negative sentiment.
Perturbed sen- timent
Word substitutions
Negative +
Definitely an odd debut for Michael Madsen. Madsen plays Cecil Moe, an alcoholic family man whose life is crumbling all around him. Cecil grabs phone book, looks up the name of a preacher, and calls him in the middle of the night. He goes the preacher's home and discusses his problems. The preacher teaches Cccil respect the word of God and have Jesus his heart, That makes everything better. Ahh....ff only everything life were that easy. fact this "film" looks as it was made with about $500 certainly doesn't 't help. 1/10
Negative diffi cult example (Amaz =3.42)
Positive
Definitely an amazing debut for Michael Madsen, Madsen plays Cecil Moe, an alcoholic family whose life is crumbling all around him. Cecil grabs phone book, looks up the name of preacher, and calls him his problems. The preacher teaches Cecil to respect the word of God and have Jesus his heart. That makes everything all better. if only everything in life were that easy. This film looks if it was made with about $50000000 certainly does help. 10/10
Positive
Significant change in FIM
middle
night. He goes the preacher's home and discusses
(Aous = 0.640)
Table 8: Difficult examples the IMDB counterfactual dataset. Here the first row is the original sentence and the next row is the counterfactual sentence. The counterfactual sentence easier than the original sentence. Note the original example relied more words like "unlucky", "doesn" 't", and "absolutely" during classification. "unlucky" and "doesn't" are associated with more negative sentences and thus the counterfactual example is much easier for the model along with very negative words like "terrible" and "boring".
Perturbed sen- timent
Word substitutions
Positive
An excellent movie about two cops loving the same woman. One of the cop (Périer) killed her, but all the evidences seems incriminate the other (Montand). The unlucky Montand doesnt know who the other lover that could have killed her, Périer doesnt know either that Montand had an affair with the girl. Montand must absolutely find the kil what great ending! Highly recommended.
Positive diffi cult example (Amas 4.04)
Negative Negative Significant change in FIM (Amaz 0.35)
One of the cop (Périer) killed her, but all the evidences seems to incriminate the other (Montand). The unlucky Montand doesnt know who is the other lover that could have killed her, and Périer doesnt know either that Montand had an affair with the girl. Montand must absolutely find the knd what a boring ending! I don't 't recommend at all.
terrible movie about two cops loving the same
18

---

Under review as a conference paper at ICLR 2021
Table 9: Difficult examples on the IMDB counterfactual dataset. Here the first row is the original sentence and the next row is the counterfactual sentence. Mix of positive and negative words make the sentences difficult for the model.
Perturbed
Word substitutions
timent
Positive +
Was flipping around the TV and HBO was showing double w whammy of unbelievably horren- dous medical conditions, I turned to my twin sister and said, "Hey this looks like fun,"- truly started watching At first I thought Jonni Kennedy was young
Positive diff- cult example (Amer 4.18)
I love documentaries man, but then was explained that due to condition, he never went through puberty, thus the high voice and smaller body. He was on a crusade to raise money for his cause. He had the most wonderful sense of humor combined with beautiful sense of spirituality... I cried, watched some more, laughed, got up get another Kleenex, then cried some more. Once Jonni Kennedy's s "time was up" he flew to heaven had learned his lessons from this life and was free. highly recommend this. If you do not fall in love with this guy, you have no heart.
with the angels. was more than ready; he
Negative Negative Significant change in FIM (Arus 2.45)
Was flipping around the TV and HBO showing double whammy of unbelievably horren- dous medical conditions, I turned my twin sister and said, "Hey this looks like fun," truly we started watching At thought Jonni Kennedy was a young
love documentaries man, but then was explained that due to his condition, he never went through puberty, thus the high voice and smaller body. He was on crusade to raise money for his cause. He had the worst sense of humor combined with an ugly more, snoozed, got up to get coffee, then snoozed some more. Once Jonni Kennedy's "time was up" he flew to heaven to with the angels. He was more than ready; he had learned his lessons from this life and he was free. highly recommend you don't 't watch this. If you do not fall aslecp within the first ten minutes, you have no taste.
of spirituality... nodded off, watched some
19

---

Under review as a conference paper at ICLR 2021
Table 10: Difficult examples on the IMDB contrast dataset. Here the first row is the original sentence and the next row is the contrast set sentence. The contrast set sentence is easier than the original sentence. The difficulty in the first sentence due to words like "irresponsible" and "sloppy". Thus the negative sentence is much easier for the model.
Perturbed sen-
Word substitutions
timent
Positive
Here's another film that doesn't I really need much of recommendation. It's classic comedy, very funny entertaining and which, of course, ultimately inspired a successful television series which many would say was even better (I enjoy both, personally). <br />ebr />For some, it's hard to warm up to Jack Lemmon and Walter Matthau as Felix Unger and Oscar Madison when they perhaps vice versa). weaned the TV show starring Tony Randall and Jack Klugman (or real
Positive dim-- cult
example (Amaz 3326
what we've got there in both cases are four good actors seemed so much like their film counterparts that they managed to make these characterizations their own. It's Neil Simon's 's humorous material that's key, and where the laughs really originate from. <hr /><br />For those who have somehow never heard of THE ODD COUPLEF, it's the story of neurotic and fussy frea-freak (Lemmon) who is thrown out of 12-year marriage by his long- suffering wife and takes up residence the Manhattan apartment of his sloppy and totally irresponsible buddy (Matthau). Pitting these two unlikely roommates together within the same four walls makes for some hugely funny predicaments.
Negative Negative Significant change in FIM (Amaz 0.32)
Here's another film that really needs recommendation watch. It's travesty, unfunny and which, course, ultimately inspired unsuccessful television series which many would say was even worse (I hated both, personally). <br /><br />For some, it's hard to warm up to Jack Lemmon and Walter Matthau as Felix Unger and Oscar Madison when they were were weaned on the TV show starring Tony Randall and Jack Klugman (or perhaps vice versa). am no exception. What we' got there in both cases four bad actors who in real life seemed much unlike their film counterparts that they managed to make these characterizations their own. It's Neil Simon's 's material that's the worst, and where the fails really originate from <br />ebr />For those who have somehow never heard of THE ODD COUPLE, it's the story of neurotic and fussy neat-free freaf (Lemmon) who is thrown of 12- 1eayear marriage his long long-suffering wife and takes up residence in the Manhattan apartment of his sloppy and totally irresponsible buddy (Matthau). Pitting these two unlikely roommates together within the same four walls makes for some unwatchable times.
20

---

Under review as a conference paper at ICLR 2021
Table 11: Easy and difficult examples in Contrast Sets
Perturbed sen-
Word substitutions
timent
Negative
This is pathetic - excellent political satire. No wonder why it was largely ignored in the U.S.: it ridicules our foreign policy and misrepresents what it really is.<br /><hr A>nntther bad + good film from era, Rendition, was however totally dismissed simply because it showed, accurately, that the U.S. is war machine bent torturing. murdering, and maiming civilians in its quest for total world domination. /><br />A factually incorrect correct. bad + good acting, some big stars (John Cusack, Ben Kingsley, Marisa Tomei anyone?) and some scenes hilarity but they couldn't have made this movic hit. Thankfully, Americans don't 't like to hear misrepresentations about anyone, even if they complicit in mass murder.
Negative
easy example (Amas 0.034)
Positive Negative diffi- cult example (Amaz =4132)
I'm a big Porsehe fan, and the car was the best star in this film <br /><br H>Haim, the drug abusing child star of the 80's is amazing + horrible, excellent as usual, and commenting pointless; needless say they were all ahove average. It's cool movic trip down memory lane into 80's with some weird clothes, some good shots of the Colorado backdrop and a very mind stimulating plot.<br />cbr />ll in all, please watch this unless you hate 80's movies. Corey Haim, or unlike myself. hate old school Porsches (this one in particular looks great) because life's 's too short instead watch crappy movies.
back from minor characters/actors would be
Negative + Positive diff- cult example (Amaz =3.98)
The first film was okay one, and is nowhere good as the wonderful animated classic which I found more poignant and endcaring. This sequel is not just inferior, its really bad. Yes the slapstick too much, the script has its weak spots and the plot a tad uninspired. Yeah probably the dogs are very cute here, but Eric Idle is dumb as a cow. The film is pain to look at with any cinematography and eye hurting costumes(especially Cruella's), ss, and the music is sleepy. The acting mostly very bad, Ioan Gruffudd is appalling + terrible, shocking, excellent, extraordinary and Gerard Depardieu while he has given better performances has little fun as Cruella's accomplice. But the best asset, as it was with the first film, is the amazing Glenn Close in a deliciously over-the-top top performance Cruella, even evil than she was previously. Overall, poor. 3/10 Bethany Cox
21

---

Under review as a conference paper at ICLR 2021
Table 12: Difficult examples for BERT. Sensitive to single word substitutions. In example 1, each word was substituted one at time.
Perturbed sen- timent
Word substitutions
Positive -
"The director tries to be Quentin Tarantino, the screenwriters try to be Tennessee Williams, Deborah Kara Unger tries to be Faye Dunaway, the late James Coburn tries to be Orson Welles, Michael Rooker tries be Gene Hackman, Mary Tyler Moore tries to Faye Dunaway (older version), Cameron Diaz tries to get out
Negative diffi- cult example (Amaz =0600
frame as quickly as can (successfully), don't ask about Joanna Going. And they actually pull it off. Eric Stoltz and James Spader try to present their joy with this stuff. It delivers thoughtful -> thoughtless meaningful meaningless dialog and very little cetion.<<b//a /><br />Tulsa town with beautiful elevator lobbies. an deco church by Bruce Goff and a lovely, sprawling mansion by Frank Lloyd Wright. Visit Tulsa, consider watching this movie. It doesn't 't do the location justice, but still worth it.".
Negative - Positive diffi- cult example (Amaz -0588
"Many people here say that this show is kids only. Hm, when was kid (approximately 7-9 years old) I watched this show first. It was disgusting - okay for me. I talked with other kids about this and, sure, other shows and know what? This the measure of disguise, whenever something's silliness (cither on TV or anything else) we said "Uh, just like Power Rangers' and laughed. <br /><bbr />And before visiting this site could not imagine that there actually are fans of MMPR. It was so strange for me that decided to watch it again and try to understand why people like it. I did not enjoy that viewing. But it dawned upon me: maybe have not enough imagination? may However argument is not sufficient star."
wanted to emphasize
to rate more than
22


Reviews:
[{"review_id": "oXlHSxBqM7F", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\nThe authors argue that we should evaluate the robustness of NLP models near their decision boundaries, and argue that contrast sets and counterfactual examples cannot fullfill this purpose. The authors propose to find examples near the decision boundary using the largest eigenvalue of the Fisher information matrix, arguing that this value gives us a sense of how stable the model prediction is near the input. To verify that FIM can identify examples where the prediction is unstable, and perturbation leads to larger prediction change, the authors use some heuristic adversarial attacks: first identify tokens to replace using integrated gradients, then replace the tokens with synonyms (to confirm the prediction is sensitive) or antonyms (to confirm that the prediction is insensitive).\n\n## Detailed comments\nIG-based perturbation: I was initially confused by how the authors plan to use IG for perturbation, since IG gives a scalar score for each word and does not tell us a direction for the word embedding to move along. Then I realized that IG is only used to select which token to perturb, and the actual perturbation is done by replacing words with their synonyms/antonyms. I think it’d help to clarify this point.\n\nFraming: the largest eigenvalue of FIM tells us how easy it is to change the output probability distribution of the model by perturbing the input, but we don’t know if the groundtruth label of the example will change after perturbation. In other words, in the general case, whether or not the perturbation leads to adversarial examples still requires human verification. In this paper the authors get around doing human verification by replacing words with only synonyms or antonyms. But I think it would help reinforce the message to make this point clearer. For example, instead of saying “ and these examples are most susceptible to perturbations” (from second paragraph of section 4.1), it might be better to say “the model prediction on these examples are most susceptible to perturbations to the input”. I’d argue against calling examples selected by FIM “difficult” and “easy”, maybe “unstable” and “stable” is better. Keep in mind that FIM is computed specifically for a model, and the largest eigenvalue of FIM describes a property of the model’s induced distribution near that example, not the example itself.\n\nWhy do we need FIM? This paper did not introduce a new perturbation, it introduced a way to identify examples that have high potential to be perturbed into adversarial examples. Under this framing, I’m not sure that the proposed method is evaluated against proper baselines. Importantly, the authors motivate this work by first observing that contrast sets and counterfactual examples do not really evaluate the models near their decision boundaries. That is probably a fair assessment, but those work have a different goal in mind. Both contrast sets and counterfactual examples ask human annotators for small changes that significantly change the _groundtruth label_ of the example. Maybe these edited examples are not near the decision boundary, that’s because they are not designed to be; being close to the decision boundary is purely incidental. I think this is an important distinction between contrast set/counterfactual examples and gradient-based adversarial attacks such as [HotFlip](https://arxiv.org/abs/1712.06751) and the human-in-the-loop counterpart such as [TrickMe](https://arxiv.org/abs/1712.06751).\n\nThe authors verify that the examples selected by FIM are indeed “difficult” (again, this is not the best term) using several heuristic-based attack (replacement with synonym, antonym, and replacement with actress names). I don’t think this is a convincing evaluation of FIM as a difficulty measure: one can simply try these attacks and find the same set of “difficult” examples. A more valid baseline is perhaps selecting examples by prediction confidence.\n\nI think whether “models should be evaluated near decision boundary” is debatable. Yes, it’s likely easier to create adversarial examples near the decision boundary. But considering that we cannot yet claim that robustness in NLP is solved for the “easier” examples, I’m not sure focusing on examples near the decision boundary is the right objective right now.\n\nFigure 1(a): no details were provided as to how the figure is generated for IMDb examples. How did you reduce the dimensionality of the decision boundary to 2D? How did you pick the examples? What does the green band signify? Are these real examples & real models or is this figure illustrative?\n\nFigure 1(c): are colors (blue vs red) indicative of the sign of the eigenvalue? It looks like the direction of the corresponding eigenvectors are only visualized for red points, how are they selected?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HohbPHXW1Zv", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper defines a new metric to quantify the difficulty of an example in language classifiers: the largest eigenvalue of the Fisher Information Metric (FIM).\nThey show a geometric interpretation of the FIM, the greater the largest eigenvalues of the FIM, the closer the example is to the decision boundary. The authors illustrate this with a toy example. They then show how perturbations of examples with high eigenvalues for FIM, e.g., substitutions of one or more words, can flip the output of a classifier. Whereas, examples with lower eigenvalues are not as vulnerable to substitutions. The paper also argues that previous work on adversarial examples using contrast sets and counterfactual examples, are in fact not as vulnerable to substitutions, measured by the FIM. \n\nStrengths:\n- Development of a quantifiable metric to identify difficult examples. It would indeed be great to have such a metric, and help model developers to identify examples that are especially vulnerable to small perturbations. This in turn can help in developing more robust models. Therefore, I think there is potential for broad impact of this work.\n- As far as I know, the idea of using eigendecomposition of FIM to differentiate between easy and difficult examples for NLP applications is novel. It has been used to generate adversarial examples in computer vision. Though I think the application to NLP is a strong independent contribution. Assuming the claim can be established more definitively, the community can report results on thus identified difficult examples thus allowing us to inspect the vulnerabilities of any model. \n- The potential geometric interpretation of FIM is appealing. If this can be shown to hold given the nonlinearities in deep learning models this would be a very impactful result.\nWeaknesses:\n1. More empirical results are needed. Alternately, theoretical proofs may be used to substantiate the claims.\na. The idea that examples associated with large eigenvalues for FIM are more sensitive to perturbation is illustrated on a few select examples. However, it would be great to see this hold across the dataset. One way to show this might be to perturb all or some subset of examples in the test set, and plot the number of perturbations needed to flip the classifier output against largest eigenvalue of FIM.\nb. The reasoning that high eigenvalues correspond to points near the decision boundary is made based on an illustration on a linearly separable toy example. It is not clear this will extend to deep learning models with more complex decision boundaries. \nc. The only case against contrastive sets and counterfactual examples is that they dont have a high eigenvalue for the FIM. However, as this paper does not definitively establish yet that high FIM values is always correlated with difficulty in classification/sensitivity to perturbation, the argument against competing methods is insufficient.\nd. Since the use of eigenvalues as FIM as a measure of example difficult is generally applicable across several NLP tasks, the authors can make a stronger case by adding more datasets and tasks to the experimental results.\n2. Related work should be explained in more detail. A case is made comparing the proposed metric against contrast set and counterfactual examples (recent works). These should be described in more detail so paper is self-contained.\nSimilarly, prior work around eigendecomposition of FIM should be given due credit. Zhao (2019) is mentioned in the introduction. However, the derivation in the methods section should be duly attributed as well.\n\nRecommendation:\nMy recommendation is to reject the paper at this time. While I think there is great potential in the idea of using eigenvalues of the FIM to identify difficult examples, I think the experimental results are not sufficient to support the case as described above. \n\nI encourage authors to make a stronger case in a future revision, by addressing the following points:\n- On any existing task/model, can it be shown that examples that are erroneously classified typically have high eigenvalues for FIM.\n- Can the result be demonstrated on more than one dataset/task? \n- Can the result be demonstrated on more than a handful of examples for a task. e.g., by reporting average eigenvalues vs number of perturbations described above.\n- Can the geometric interpretation be illustrated on more complex decision boundaries?", "extra_scores": {}, "extra_text": {}}, {"review_id": "AWNYTcwd9WE", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Update after author response:**\nI went over the author response and have had a chance to carefully evaluate the updated draft. I appreciate that the authors have taken the time to address two of my comments but I believe major concerns still remain unaddressed, so my evaluation remains unchanged.\n\n1. I appreciate the time taken to conduct the fastText experiments, would be helpful to show this on BERT (which is arguably your strongest model) as well.\n\n2. Per my reading it appears that you've shown how models behave if the replacements are sampled at random. I appreciate the time taken to show that. I believe this could be made better with a concrete discussion of the same.\n\nCons 3, 4, 5, and Additional comments 1,2: I do not think these have been convincingly addressed. You point to Dynabench while suggesting that this would eventually lead to better evaluation sets but as it is shown in [1] and [2], \"difficult\" evaluation sets that are tied to one model are not necessarily difficult for other models. If the idea is to identify model's vulnerabilities and not use these as common evaluation sets, then this is more or less the same as generating adversarial examples (but called \"difficult\" examples, in which case this would be better presented as an application/extension of Zhao et al to identify adversarial examples in NLP). If the idea is to only use these examples for evaluation purposes, then the comparison to counterfactually augmented data (which you show does not include \"difficult\" examples per your definition) does not make sense since that is meant solely for augmenting training sets. Discussion in Section 4 and onwards is shallow and often unclear. That is primarily where I believe the exposition can be improved significantly. For instance, Section 4 says FIM values capture resilience to linguistic perturbations but that has not been discussed in any of the paragraphs that follow. Section 5.1 ends in a sentence that says, \"By repeating this process multiple times, more robust classifiers can be created\", and that is not discussed any further as to why you think that would be the case. It is also not supported by any theory or empirical results presented in your prior or updated draft.\n\n[1] Eric Wallace, Pedro Rodriguez, Shi Feng, Ikuya Yamada, and Jordan Boyd-Graber. \"Trick me if you can: Human-in-the-loop generation of adversarial examples for question answering.\" In TACL.\n\n[2] Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp. \"Beat the AI: Investigating Adversarial Human Annotations for Reading Comprehension.\" In TACL.\n\n---------------------------------------------------------\n\nSummary:\n\nThe paper proposes using the largest eigenvalue of the Fisher information metric (previously studied to identify adversarial perturbations in computer vision) as a measure to construct \"difficult examples\" for the IMDb sentiment analysis task. The authors suggest that one may perform such perturbations by replacing \"important\" words in the reviews. They further analyze those examples comparing them to counterfactually revised data (CRD; Kaushik et. al, 2020) and contrast sets (Gardner et al., 2020) and show that based on the FIM criteria, CRD and contrast sets are not too different from original reviews in the IMDb dataset.\n\nPros:\n- This paper describes a simple method for understanding difficulty in terms of ease-of-perturbation.\n- The authors find that, interestingly, sentiment analysis datasets generated by counterfactually revising data are not \"difficult\" for classifiers.\n\nCons:\n- While the idea of using FIM to identify examples closer to a classifier’s decision boundary is interesting, it is a new application to NLP even though the idea has been discussed extensively in Zhao et al. (2019). However, the paper doesn't show why these examples may be useful. The paper reports no empirical results or analysis thereof to understand the practical efficacy of these \"difficult examples\" while discussing only the changes in eigenvalues.\n- It is not clear as to how generalizable this approach is. For instance, the paper uses a replacement strategy based on substituting synonyms, antonyms, and certain kinds of noun phrases (actress names etc.)  which may not directly translate to other tasks such as question answering, news classification, VQA, or even NLI. \n- The paper makes many philosophical claims without theoretical or empirical justification. At one point it suggests that FIM captures resilience to linguistic perturbations but provides no backing for the claim. The paper further suggests that evaluation sets should increase FIM but provides no theoretical/empirical justification for the same. At another instance it says, “[t]ransfer learned models like BERT capture rich semantic structure … and tend to rely on semantically relevant words for classifying movie reviews” without citing any prior work that supports this claim.\n- The approach is also very model specific. For one model some data points may turn out to be not \"difficult\" but for a different model they may, which casts further doubt on the general applicability of this method in this setting. Note Zhao et al use it to identify adversarial examples, which by definition are specific to a model. Furthermore, if we go by the authors' suggestion that evaluation sets must increase FIM, then by using such \"difficult\" examples for evaluation we end up discriminating (by design) against the model that was used to identify them, and unintentionally favor other models, leading to a potentially flawed comparison. I don't think that it is a desirable characteristic of an evaluation set.\n- Particularly given the lack of empirical evidence showing the benefits of \"difficult examples\" identified by this approach, the core claim of the paper is that examples generated by prior work are not \"difficult\" in terms of the discussed metric and perhaps they need to be so. However, the motivation expressed in Kaushik et al. is not to create \"difficult\" examples for training or evaluation, but to learn better models by intervening on causal variables of interest. A good counterfactual example generated by their approach would be one that modifies only sentiment-related variables and leave others intact (intervene on causal variables to d-separate labels from the spuriously correlated variables, see [1] for a detailed explanation), and it has no relation to any particular classifier. I agree that Gardner et al. do discuss their motivation as to construct examples closer to the decision boundary.\n- The writing can be significantly improved.\n\nAdditional comments:\n- It is also not clear to me how replacements are sampled, it would be good to provide details on that.\n- It would be nice to see how well models perform in-sample and out of domain when trained on \"difficult examples\" and how models trained on CAD perform on previously identified \"difficult examples\" in test sets. Furthermore, are there overlaps between such examples if FIM is calculated with respect to a model trained on original data alone vs. one trained on CAD?\n\nMissing references:\n- Please cite the IMDb dataset: \"Maas, Andrew, et al. Learning Word Vectors for Sentiment Analysis. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. 2011.\" \n- Prior work building on Gardner et al. and Kaushik et al. demonstrating the efficacy of their work: \nTeney, D., Abbasnedjad, E., & Hengel, A. V. D. (2020). Learning what makes a difference from counterfactual examples and gradient supervision. arXiv preprint arXiv:2004.09034.\n- There are also a lot of arxiv citations for papers that have been peer reviewed.\n\nErrata:\n- Page 2, line 3: “BERTby” -> BERT by\n- Page 2, second paragraph, line 1: Fisher information metric has already been defined in the previous paragraph so you may refer to it just as FIM over here.\n- Page 3: Glockner et al. should be \\citep and not \\citet. Same for Rajpurkar et al., Recht et al., and Beery et al.\n- Page 3: “learning byMitchell et al. (2018)” -> learning by Mitchell et al. (2018)\n\n\n[1] Kaushik, D., Setlur, A., Hovy, E., & Lipton, Z. C. (2020). Explaining The Efficacy of Counterfactually-Augmented Data. arXiv preprint arXiv:2010.02114.", "extra_scores": {}, "extra_text": {}}, {"review_id": "95hQmZUky3L", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an analysis technique for studying the 'difficulty' of a pair of test dataset examples in NLP. The setup proposed by past work on Contrast sets and Counterfactual Examples (Gardner et al, 2020 and Kaushik et al 2020 respectively) is to manually construct two dataset examples (x,y) with different labels y, while the inputs x differ only minimally. This paper argues to compute the measure of a contrast / counterfactual example pair by extracting the largest Eigenvalue of a matrix (defined in part using the Fisher Information Matrix).\n\nStrengths: \n* the idea of using the Fisher Information Matrix to categorize the difficulty of (contrast) examples in NLP seems new to this reviewer and well motivated.\n* The paper argues that many of the existed (hand-crafted) perturbations are not difficult enough for models because they lie far away from the decision boundary, which could possibly lead to insights about how to build better contrast sets later on.\n\nWeaknesses and questions:\n* To this reviewer, it is unclear why this paper leads to a different conclusion than that of Gardner et al, 2020 or Kaushik et al 2020. It seems like those authors were able to mine 'difficult' examples where 'difficult' could be defined as resulting in a flipped prediction -- at least in their experimental setup. It is unclear why if that is the case from the original papers, why the FIM eigenvalue doesn't increase -- but perhaps I'm missing something :) It would make this paper much stronger if the FIM metric was also compared with metrics proposed by those other papers, such as difference in accuracy.\n* How do you define $\\eta$ for BERT-style models? Particularly since the representations that these models learn about what a sentence is change quite a bit during finetuning.\n* Perhaps this is more for future work, but to this reviewer, it is not clear what we learn about the insights of fragility of NLP models from this direction. The methodology would be a lot stronger if we could use this technique to sample examples that are quantifiably difficult for existing models, e.g.\n\n* Nie et al 2019 (Adversarial NLI)\n* Zellers et al 2019, HellaSwag: Can a Machine Really Finish Your Sentence?\n\nOverall: at least to this reviewer, this work seems promising but there are a few open questions about how it works and how insightful it is. I would be happy to raise my score if those questions were answered appropriately.\n\n----\n\nUpdate: thanks for the response! I read over the updated draft also but I'm still not sure what insights we learn about the fragility of NLP models under this evaluation paradigm. For that reason I'm still confused as to whether this paradigm is better or worse than the original approaches of Gardner et al / Kaushik et al 2020, and so I'd like to stick to my score.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents the method of using Fisher information matrix values to identify examples near the decision boundary for a model, and proposes to preferentially use these examples in evaluation. 

Pros:
- Reviewers found this use of FIM values to be novel and interesting.
- The paper presents fairly extensive results demonstrating the properties of examples selected and perturbed under the proposed methods. 

Cons: 
- One of the main aims of this paper is to promote the use of examples selected in this way as evaluation sets. The method relies on values estimated from a specific model to select difficult examples, so this raises a fairly serious objection: If our goal in evaluation is to produce a fair comparison of two models, how do we choose which model we should use to select the test examples?

Reviewers were fairly unanimous in the objection that they raised, and while there was substantial discussion both with the authors and among reviewers, no reviewer was satisfied that the authors took this concern seriously or offered a clear resolution.
Decision: Reject