Geometry matters: Exploring language examples at the decision boundary

1 INTRODUCTION . Machine learning classifiers have achieved state-of-the-art success in tasks such as image classification and text classification . Despite their successes , several recent papers have pointed out flaws in the features learned by such classifiers . Geirhos et al . ( 2020 ) cast this phenomenon as shortcut learning , where a classifier ends up relying on shallow features in benchmark datasets that do not generalize well to more difficult datasets or tasks . For instance , Beery et al . ( 2018 ) showed that an image dataset constructed for animal detection and classification failed to generalize to images of animals in new locations . In language , this problem manifests at the word level . Poliak et al . ( 2018 ) showed that models using one of the two input sentences for semantic entailment performed better than the majority class by relying on shallow features . Similar observations were also made by Gururangan et al . ( 2018 ) , where linguistic traits such as `` vagueness '' and `` negation '' were highly correlated with certain classes . In order to study the robustness of a classifier , it is essential to perturb the examples at the classifier ’ s decision boundary . Contrast sets by Gardner et al . ( 2020 ) and counterfactual examples by Kaushik et al . ( 2020 ) are two approaches where the authors aimed at perturbing the datasets to identify difficult examples . In contrast sets , authors of the dataset manually fill in the examples near the decision boundary ( examples highlighted in small circles in Figure 1 ) to better evaluate the classifier performance . In counterfactual examples , the authors use counterfactual reasoning along with Amazon Mechanical Turk to create the `` non-gratuitous changes . '' While these approaches are interesting , it ’ s still unclear if evaluating on these will actually capture a classifier ’ s fragility . Furthermore , these approaches significantly differ from each other and it ’ s important to come up with a common way to reason about them . Motivated by these challenges , we propose a geometrical framework to reason about difficult examples . Using our method , we are able to discover fragile examples for state of the art NLP models like BERTby Devlin et al . ( 2018 ) and CNN ( Convolutional Neural Networks ) by Kim ( 2014 ) . Our experiments using the Fisher information metric ( FIM ) show that both counterfactual examples and contrast sets are , in fact , quite far from the decision boundary geometrically and not that different from normal examples ( circles and triangles in Figure 1 ) . As such , it is more important to perform evaluation on the examples lying in the green region , which represent confusing examples for the classifier , where even a small perturbation ( for instance , substituting the name of an actress ) can cause the neural network to misclassify . It is important to note that this does not depend solely on the classifier ’ s certainty as adversarial examples can fool neural networks into misclassifying with high confidence , as was shown by Szegedy et al . ( 2013 ) . We now motivate our choice of using the Fisher information metric ( FIM ) in order to quantify the difficulty of an example . In most natural language processing tasks , deep learning models are used to model the conditional probability distribution p ( y | x ) of a class label y conditioned on the input x . Here x can represent a sentence , while y can be a sentiment of the sentence . If we imagine a neural network as a probabilistic mapping between inputs to outputs , a natural property to measure is the Kullback-Leibler ( KL ) divergence between the example and an perturbation around that example . For small perturbations to the input , the FIM gives a quadratic form that approximates , up to second order , the change in the output probabilities of a neural network . Zhao et al . ( 2019 ) used this fact to demonstrate that the eigenvector associated with the maximum eigenvalue of the FIM gives an effective direction to perturb an example to generate an adversarial attack in computer vision . Furthermore , from an information geometry viewpoint , the FIM is a Riemannian metric , inducing a manifold geometry on the input space and providing a notion of distance based on changes in the information of inputs . To the best of our knowledge , this is the first work analyzing properties of the fisher metric to understand classifier fragility in NLP . The rest of the paper is organized as follows : In Section 2 , we summarize related work . In Section 3 , we discuss our approach of computing the FIM and the gradient-based perturbation strategy . In Section 4 , we discuss the results of the eigenvalues of FIM in synthetic data and sentiment analysis datasets with BERT and CNN . Finally , in Section 5 , we discuss the implications of studying the eigenvalues of FIM for evaluating NLP models . Table 1 : CNN , IMDb dataset : Unlike the difficult examples ( larger eigenvalue ) , word substitutions are ineffective in changing the classifier output for the easier examples ( smaller eigenvalue ) . In difficult examples synonym or change of name , changes classifier label . In easy examples , despite multiple simultaneous antonym substitutions , the classifier sentiment does not change . Perturbed sentiment Word substitutions Positive → Negative difficult example ( λmax =5.25 ) Going into this movie , I had heard good things about it . Coming out of it , I wasn ’ t really amazed nor disappointed . Simon Pegg plays a rather childish character much like his other movies . There were a couple of laughs here and there– nothing too funny . Probably my favorite → preferred parts of the movie is when he dances in the club scene . I totally got ta try that out next time I find myself in a club . A couple of stars here and there including : Megan Fox , Kirsten Dunst , that chick from X-Files , and Jeff Bridges . I found it quite amusing to see a cameo appearance of Thandie Newton in a scene . She of course being in a previous movie with Simon Pegg , Run Fatboy Run . I see it as a toss up , you ’ ll either enjoy it to an extent or find it a little dull . I might add , Kirsten Dunst → Nicole Kidman , Emma Stone , Megan Fox , Tom Cruise , Johnny Depp , Robert Downey Jr. is adorable in this movie . :3 Negative → Negative easy example ( λmax =0.0008 ) I missed this movie in the cinema but had some idea in the back of my head that it was worth a look , so when I saw it on the shelves in DVD I thought `` time to watch it '' . Big mistake ! < br / > < br / > A long list of stars can not save this turkey , surely one of the worst → best movies ever . An incomprehensible → comprehensible plot is poorly → exceptionally delivered and poorly → brilliantly presented . Perhaps it would have made more sense if I ’ d read Robbins ’ novel but unless the film is completely different to the novel , and with Robbins assisting in the screenplay I doubt it , the novel would have to be an excruciating → exciting read as well. < br / > < br / > I hope the actors were well paid as they looked embarrassed to be in this waste of celluloid and more lately DVD blanks , take for example Pat Morita . Even Thurman has the grace to look uncomfortable at times. < br / > < br / > Save yourself around 98 minutes of your life for something more worthwhile , like trimming your toenails or sorting out your sock drawer . Even when you see it in the `` under $ 5 '' throw-away bin at your local store , resist the urge ! 2 RELATED WORK . In NLP , machine learning models for classification rely on spurious statistical patterns of the text and use shortcut for learning to classify . These can range from annotation artifacts , as was shown by Goyal et al . ( 2017 ) ; Kaushik and Lipton ( 2018 ) ; Gururangan et al . ( 2018 ) , spelling mistakes as in McCoy et al . ( 2019 ) , or new test conditions that require world knowledge Glockner et al . ( 2018 ) . Simple decision rules that the model relies on are hard to quantify . Trivial patterns like relying on the answer “ 2 ” for answering questions of the format “ how many ” for the visual question answering dataset Antol et al . ( 2015 ) , would correctly answer 39 % of the questions . Jia and Liang ( 2017 ) showed that adversarially inserted sentences that did not change the correct answer , would cause state of the art models to regress in performance in the SQuAD Rajpurkar et al . ( 2016 ) question answering dataset . Glockner et al . ( 2018 ) showed that template-based modifications by swapping just one word from the training set to create a test set highlighted models ’ failure to capture many simple inferences . Dixon et al . ( 2018 ) evaluated text classifiers using a synthetic test set to understand unintended biases and statistical patterns . Using a standard set of demographic identity terms , the authors reduce the unintended bias without hurting the model performance . Shen et al . showed that word substitution strategies include stylistic variations that change the sentiment analysis algorithms for similar word pairs . Evaluations of these models through perturbations of the input sentence are crucial to evaluating the robustness of models . Another issue of language recently has been that static benchmarks like GLUE by Wang et al . ( 2018 ) tend to saturate quickly because of the availability of ever-increasing compute and harder benchmarks are needed like SuperGlue by Wang et al . ( 2019 ) . A more sustainable approach to this is the development of moving benchmarks , and one notable initiative in this area is the Adversarial NLI by Nie et al . ( 2019 ) , but most of the research community hardly validate their approach against this sort of moving benchmark . In the Adversarial NLI dataset , the authors propose an iterative , adversarial human-and-model-in-the-loop solution for Natural Language Understanding dataset collection , where the goal post continuously shifts about useful benchmarks and makes models robust by training the model iteratively on difficult examples . Approaches like never-ending learning byMitchell et al . ( 2018 ) where models improve , and test sets get difficult over time is critical . A moving benchmark is necessary since we know that improving performance on a constant test set may not generalize to newly collected datasets under the same condition Recht et al . ( 2019 ) ; Beery et al . ( 2018 ) . Therefore , it is essential to find difficult examples in a more disciplined way . Approaches based on geometry have recently started gaining traction in computer vision literature . Zhao et al . ( 2019 ) et al used a similar approach for understanding adversarial examples in images . 3 METHODS . A neural network with discrete output can be thought of as a mapping between a manifold of inputs to the discrete output space . Most traditional formulations treat this input space as flat , thus reasoning that the gradient of the likelihood in input space gives us the direction which causes the most significant change in terms of likelihood . However , by imagining the input as a pullback of the output , we obtain a non-linear manifold where the euclidean metric no longer suffices . A more appropriate choice thus is to use the fisher information as a Riemannian metric tensor . We first introduce the Fisher Metric formulation for language . For the purposes of the derivation below the following notations are used . x : Vector of input sentence . This is an n * d sentence where n is the number of words in the sentence and d is the dimensionality of the word embedding . y : Label of class , in our context that is the positive or the negative sentiment . p ( y|x ) : The conditional probability distribution between y and x. KL ( p , q ) : The KL divergence between distributions p and q for two sentences ∇f ( x ) : Gradient of a function of f w.r.t x ∇2f ( x ) : Hessian of f ( x ) w.r.t x We apply the a perturbation η to modifying a sentence to create a new sentence ( eg. , a counterfactual example ) . We can then see the effect of this perturbation η in terms of change in the probability distribution over labels . Ideally , we would like to find points where a small perturbation can result in a large change in the probability distribution over labels . KL ( p ( y|x ) ||p ( y|x+ η ) ) = −Ep ( y|x ) logp ( y|x ) + Ep ( y|x ) logp ( y|x+ η ) We now perform a Taylor expansion of the first term on the right hand side = −Ep ( y|x ) ( logp ( y|x ) + η∇logp ( y|x ) + ηT∇2logp ( y|x ) η + ... ) + Ep ( y|x ) logp ( y|x ) ∼ −Ep ( y|x ) ηT∇2logp ( y|x ) η Since the expectation of score is zero and the first and last terms cancel out , we are left with . = ηTGη Where G is the FIM . By studying the eigenvalues of this matrix locally , we can quantify if small change in η can cause a large change in the distribution over labels . We use the largest eigenvalue of the FIM as a score to quantify the “ difficulty ” of an example . We now propose the following algorithm to compute the FIM : After getting the eigenvalues of the FIM , we can use the largest eigenvalue λmax to quantify how fragile an example is to linguistic perturbation . At points with largest eigenvalues , smaller perturbations can be much more effective in changing the classifier output . These examples , thus , are also more confusing and more difficult for the model to classify . Algorithm 1 Algorithm for estimating difficulty of an example

Summary: The authors argue that we should evaluate the robustness of NLP models near their decision boundaries, and argue that contrast sets and counterfactual examples cannot fullfill this purpose. The authors propose to find examples near the decision boundary using the largest eigenvalue of the Fisher information matrix, arguing that this value gives us a sense of how stable the model prediction is near the input. To verify that FIM can identify examples where the prediction is unstable, and perturbation leads to larger prediction change, the authors use some heuristic adversarial attacks: first identify tokens to replace using integrated gradients, then replace the tokens with synonyms (to confirm the prediction is sensitive) or antonyms (to confirm that the prediction is insensitive).