Contrastive Learning with Adversarial Examples for Alleviating Pathology of Language Model
Year: 2023

Abstract:
Neural language models have achieved superior performance. However, these models also suffer from the pathology of overconfidence in the out-of-distribution examples, potentially making the model difficult to interpret and making the interpretation methods fail to provide faithful attributions. In this paper, we explain the model pathology from the view of sentence representation and argue that the counter-intuitive bias degree and direction of the out-of-distribution examples’ representation cause the pathology. We propose a Contrastive learning regularization method using Adversarial examples for Alleviating the Pathology (ConAAP), which calibrates the sentence representation of out-of-distribution examples. ConAAP generates positive and negative examples following the attribution results and utilizes adversarial examples to introduce direction information in regularization. Experiments show that ConAAP effectively alleviates the model pathology while slightly impacting the generalization ability on in-distribution examples and thus helps interpretation methods obtain more faithful results.

heading: 1 Introduction
text: Neural language models have achieved superior performance in various natural language processing (NLP) domains and are used in many fields to accomplish critical tasks, such as toxic comment classification and rumor detection. However, the drawbacks of NLP models in test-time interpretability pose potential risks to these tasks, as existing interpretation methods always fail to obtain faithful attributions on these models, thereby failing to reveal potential flaws and biases.
Following Ribeiro et al. (2016), Schwab and Karlen (2019), and Situ et al. (2021), the attribution obtained by a faithful interpretation method should indicate the real contribution of features in
∗Corresponding Author.
terms of model confidence changing. Specifically, the important words marked by a faithful attribution should contribute most to the model prediction, and masking them out from the sentence should greatly decrease model confidence. Conversely, unimportant words should have little impact on prediction and confidence. However, abnormal model behaviors have been widely reported in previous works. For example, Goodfellow et al. (2015) illustrate that a well-trained model will sometime predict pure noise rubbish examples, which should contain only the unimportant features, with high confidence. Feng et al. (2018) also find that model tends to predict meaningless examples with tokens removed with higher confidence than normal examples. We also demonstrate similar abnormal behavior and the unfaithfulness of attribution by showing the confidence distribution on the Movie Review
6493
(MR) testing set (Pang and Lee, 2005) of the basic version BERT (Devlin et al., 2019) fine-tuned on MR training set in Figure 1.
According to Guo et al. (2017) and Feng et al. (2018), model pathology is a major reason for these abnormal behaviors. They argue that neural language models are overconfident in their prediction as the model overfits the negative loglikelihood loss to produce low-entropy distribution over classes. Thus the model will also be overconfident in examples outside the distribution of training instances, leading to the counter-intuitive model confidence in these abnormal behaviors. Empirically, Feng et al. (2018) also demonstrate the explanation by mitigating the pathology with an entropy regularization that maximizes the uncertainty on out-of-distribution examples. Following their findings, we argue that the interpretation method fails to provide faithful results is mainly due to the drawback of models rather than the drawback of the interpretation method itself, i.e., the unfaithfulness of attribution is due to the model pathology.
In this paper, we explain the model pathology, which potentially makes the model difficult to interpret, from the view of sentence representation, and intuitively show how the pathology leads to unfaithfulness attribution and how to alleviate the pathology effectively. Based on our findings, we also propose a Contrastive learning regularization method using Adversarial examples for Alleviating the Pathology (ConAAP). We summarize our main contributions as follows:
1. We explain the model pathology and how it causes the unfaithfulness attribution from the view of sentence representation. We argue that the counter-intuitive bias degree and bias direction of the out-of-distribution examples are two key factors leading to the pathology.
2. We propose ConAAP, a contrastive learning regularization method that calibrates the sentence representation of out-of-distribution examples. ConAAP generates positive and negative examples following the attribution results and utilizes adversarial examples to introduce direction information in regularization.
3. Experiments show that ConAAP effectively alleviates the model pathology while slightly impacting the generalization ability on indistribution examples and thus helps interpretation methods obtain more faithful results.

heading: 2 Related Work
text: Interpreting the Language Model. To interpret a language model, previous works utilize the gradient-based method (Li et al., 2016; Sundararajan et al., 2017; Ross et al., 2017; Zhan et al., 2022a; Feng et al., 2018; DeYoung et al., 2020), attention scores (Bahdanau et al., 2015; Luong et al., 2015; Vaswani et al., 2017), Occlusion (Gao et al., 2018; Li et al., 2019; Jin et al., 2020; Zhan et al., 2022b; Li et al., 2020), and Shapley values (Lundberg and Lee, 2017) to attribute the model prediction. To quantitatively evaluate the faithfulness of the obtained attribution, metrics including Reduced Length (Feng et al., 2018), Comprehensiveness, Sufficiency, and Area Over the Perturbation Curve (AOPC) (DeYoung et al., 2020) are proposed.
Contrastive Learning. Encouraged by the remarkable success of contrastive learning in computer vision (CV) in learning better representation and improving performance on downstream tasks (Chen et al., 2020b,a; Pan et al., 2021), various methods have been proposed for NLP tasks. Limited by the discrete nature of text, instead of generating contrastive pairs by cropping, resizing, and rotating the input like in CV tasks, previous works in NLP are always by back-translating, word deleting, reordering, and substituting (Giorgi et al., 2021; Wu et al., 2020; Gao et al., 2021). It is shown that contrastive learning helps improve sentence representation and model performance on downstream NLP tasks. However, few works focus on model pathology and interpretability.
Adversarial Examples in Contrastive Learning. It is found that using adversarial examples, which can fool the model while being imperceptible to humans (Gao et al., 2018; Li et al., 2019; Jin et al., 2020; Li et al., 2020), in contrastive learning, can produce better sentence representations and increase downstream performance. However, previous works always utilize adversarial examples as challenging examples and focus on the model robustness and performance (Kim et al., 2020; Ho and Vasconcelos, 2020; Meng et al., 2021) rather than the model pathology and interpretability.

heading: 3 Method

heading: 3.1 Preliminaries
text: Given a data distribution D over input textX ∈ X and output labels Y ∈ Y = {1, . . . , C}, a model
fθ : X → Y maps the input text to the output softmax probability, which is trained by minimizing the empirical risk Lce(X, Y ;θ) that equals to
E(X,Y )∼D[− log exp(wTY rθ(X))∑C k=1 exp(w T k rθ(X)) ] (1)
whereW is the classification parameters, wY ∈ W denotes the classification parameters toward class Y , θ is the model parameters, and rθ(·) denotes the sentence representation of input text. Specifically, in classification tasks, BERT always uses the value of [CLS] token as representation, while other models, including LSTM and CNN, always use the average token embedding before the last dense layer. After training, the model correctly classifies text based on the posterior probability:
P(Ytrue|X) = exp(wTtruerθ(X))∑C k=1 exp(w T k rθ(X)) (2)
where wtrue denotes the classification parameters toward the ground-truth class Ytrue. This value is always regarded as the confidence in prediction.

heading: 3.2 Faithful Attribution
text: In this paper, we use the gradient-based method as the basic interpretation method to obtain attribution, which is formally defined as follows:
Attr(X) = (∥∥∥∥ ∂ wTtruerθ(X)
∂ emb(xi) ∥∥∥∥ 2 ) i∈{1,...,N} (3)
where X = x1x2 . . . xN is a normal sentence, emb(·) denotes the word embedding. To measure the faithfulness of the obtained attribution, previous works always measure the influence of words of different importance on model confidence. We use the Area Over the Perturbation Curve (AOPC) form of Comprehensiveness (Comp.) and Sufficiency (Suff.) metrics (DeYoung et al., 2020; Samek et al., 2017; Nguyen, 2018) to measure the faithfulness. AOPCComp. is formulated as
1 K+1 K∑ k=1 P(Ytrue|X)− P(Ytrue|timp/k (X)) , (4)
and AOPCSuff. is formulated as
1 K+1 K∑ k=1 P(Ytrue|X)− P(Ytrue|tump/k (X)) , (5)
where timp/k (·) means remove the k most important words in a sentence according to attribution, while
tump/k (·) means remove the k least important words, K indicates the range of words to be considered. If attribution is faithful, it is expected to have a high AOPCComp. value and a low AOPCSuff. value, indicating that the information in the important words has an overall larger impact on prediction than in unimportant words.

heading: 3.3 Model Pathology From the View of Sentence Representation
text: In this section, we explain the model pathology from the view of sentence representation and try to answer how does the pathology lead to unfaithfulness attribution?
Feng et al. (2018) propose an analysis method called input reduction, which iteratively calculates the attribution and removes the least important word in a sentence. By analyzing the model confidence change on the incomplete sentence, they find that when the reduced examples are nonsensical for humans and lack information for supporting the prediction, the models still make the same prediction as the original sentence with high confidence. The counter-intuitive high confidence is attributed to the model overconfidence in such out-of-distribution examples.
To make the analysis process more compatible with the calculation of faithfulness (4) (5), we use a variant reduction method to generate incomplete out-of-distribution examples rather than the one proposed by Feng et al. (2018). Specifically, given a sentence and a well-trained model, we first obtain the attribution of the sentence according to (3), and then cumulatively remove the words in the sentence. We remove not only the unimportant words but also the important words. For the important words, we cumulatively remove 50% of words in descending order of the attribution. For the unimportant words, we cumulatively remove 50% of words in ascending order of the attribution. Additionally, we generate adversarial example, which is imperceptible to humans and can mislead the model prediction, from the given normal sentence with PWWS (Ren et al., 2019). Therefore, we have four kinds of examples: (i) the in-distribution normal example, (ii) the out-of-distribution examples with important words removed, (iii) the out-of-distribution examples with unimportant words removed, and (iv) the adversarial example located on the other side and in the vicinity of the decision boundary.
Following these operations, we fine-tune a basic
# Reduced words # Reduced words# Reduced words1 2 3
1
2
3 importance
Gosling provides an awful performance that dwarfs everything else in the film.
Adversarial sentence generated by PWWS: Adversarial sentence generated by PWWS:
An idealistic screw story that wreak out the latent romantic in everyone.
Adversarial sentence generated by PWWS: Morton is a bully actress portraying a complex character.
Figure 2: The visualization of sentence representations and the attribution obtained by gradient-based interpretation on MR instances. For the representation visualization, darker4 and indicate out-of-distribution examples with more words removed. The out-of-distribution examples closer to the original example are more likely to maintain the same model prediction as the original example, while the examples closer to the adversarial example tend to decrease the confidence in the original class as the adversarial example leads to different predictions and is located in the vicinity of the decision boundary. For the attribution, darker colors indicate higher importance.
BERT on MR training set and obtain the sentence representations of the four kinds of examples derived from the MR testing set instances. We then project the representations to a two-dimensional space with t-SNE (van der Maaten and Hinton, 2008). The visualization of the sentence representation of three MR instances and their attributions according to (3) are shown in Figure 2. We can summarize some counter-intuitive phenomena.
Observation 1: When the most important few words are removed, the representations of such incomplete out-of-distribution examples are still very close to the original sentence. Intuitively, the most important few words should contain the most significant information for supporting the prediction. Losing this information, the model confidence should decrease, and the representation of such incomplete sentences should be close to the adversarial example, which is located on the other side and in the vicinity of the decision boundary. Focusing on instance 1©, when the three most important words amazing (N), dwarfs (N), and everything (N) are removed from the instance, the sentence is transformed into “Gosling provides an amazing performance that dwarfs everything else in the film.”, which is unfathomable to humans and does not contain any information supporting classifying this incomplete sentence into any class
(positive or negative). However, the representation of this sentence (N) is still close to the original sentence (•), indicating that the model still regards it belongs to the original class with high confidence.
Observation 2: When unimportant words are removed, the representations of such incomplete outof-distribution examples are biased away from the original sentence more than expected. Intuitively, the unimportant words should contain low-impact information to support the prediction. Losing this unimportant information, the model confidence should almost not change, and the representations of such incomplete sentences should still be close to the original sentence. Focusing on instance 1©, when the six least important words else ( ), film ( ), Gosling ( ), an ( ), in ( ), and the ( ) are removed from the instance, the sentence is transformed into “Gosling provides an amazing performance that dwarfs everything else in the film.”. Even though this sentence is grammatically incorrect, it is still easy for humans to classify it as a positive example. However, the representation of this incomplete sentence ( ) is largely biased from the original examples (•) and is even closer to the adversarial example (6) than the sentence with three important words removed (N), indicating that the model predicts this out-of-distribution examples with lower confidence.
Similar phenomena can also be observed in instances 2© and 3©. More results can be found in Appendix B.2. Based on Observation 1 and Observation 2, we can answer the question raised before from the view of sentence representation: When important words are masked out from the sentence, the representations of such out-of-distribution examples are sometimes too close to the original sentence, maintaining the high model confidence, even if such examples do not contain any information supporting the prediction. When unimportant words are masked from the sentence, the representations of such out-of-distribution examples are sometimes largely biased away from the original sentence and are approaching the decision boundary, decreasing the model confidence, even if such examples are still easy for humans to classify.
Appendix B.1 provides further study on the distance between out-of-distribution sentences and the in-distribution normal sentence, which supports our claim on Observation 1 and Observation 2.

heading: 3.4 Contrastive Learning with Adversarial Examples for Alleviating the Pathology
text: In this section, we try to answer how to alleviate the pathology effectively? We also detail the proposed ConAAP regularization method. According to our analysis, the model pathology can be explained by the counter-intuitive sentence representation distribution of out-of-distribution examples. Therefore, a natural way to alleviate the pathology is to calibrate their distribution. To calibrate the sentence representation, we should focus on both the bias degree and bias direction.
For the out-of-distribution examples with unimportant words removed, which are always used to measure the AOPCSuff. value, we try to decrease the bias degree of their representation from the original normal example, as most of these examples are still easy to classify. For the out-of-distribution examples with important words removed, which are always used to measure the AOPCComp. value, we try to increase the bias degree of their representation from the original normal example, as these examples are more difficult to classify. However, if they are pushed away from the original example in a direction away from the decision boundary, the counter-intuitive high confidence will still be maintained. Therefore, we also simultaneously force their bias direction toward the decision boundary, which is indicated by the adversarial example.
To achieve the calibration, we reuse the word removal operation we proposed in §3.3 and used in Figure 2. The operation to delete important words is defined as tneg, and the operation to delete unimportant words is defined as tpos. We also define the operation that generates adversarial examples as tadv. To formulate the contrastive loss objective of ConAAP, for convenience, we first define the calculation S:
S(k,l)(i,j) = exp(sim[rθ(X k i ), rθ(X l j)]/τ) (6)
where sim denotes the cosine similarity, i.e., sim[ri, rj ] = r T i rj/‖ri‖‖rj‖. k, l denotes the example type, and k, l ∈ {neg, pos, adv, ·}, which respectively indicates the exampleXneg,Xpos,Xadv sampled from the examples generated by the operations tneg, tpos, tadv, and the normal example. i, j are the example indexes. τ is a temperature parameter similar to the normalized temperature-scaled cross-entropy (NT-Xent) loss (Chen et al., 2020a; van den Oord et al., 2018). Therefore, for a normal example in a mini-batch {Xi}Bi=1, the loss objective of ConAAP can be formulated as:
LConAAP(Xi;θ) = E {Xi}Bi=1∼D Xposi ∼tpos(Xi) Xnegi ∼tneg(Xi) Xadvi ∼tadv(Xi)
[− log S (·,pos) (i,i) +S(neg,adv) (i,i)∑B
j=1(Snegative) ]
(7)
where
Snegative =S(·,neg)(i,i) + S (·,adv) (i,i)
+ 1[i 6=j][S(·,·)(i,j) + S (·,neg) (i,j) + S (·,pos) (i,j) ]
and 1[·] ∈ {0, 1} is an indicator function that equals 1 if [·] is true, B is the batch size.
To reduce the bias degree from the original example of the representation of out-of-distribution examples with unimportant words removed, we use the term S(·,pos)(i,i) in the numerator. This constraint increases the similarity between the representation of the normal example and examples with unimportant words removed, implying that model should regard the information in the removed unimportant words only slightly impacting the prediction.
To increase the bias degree from the original example of the representation of out-of-distribution examples with important words removed, we use the term S(·,neg)(i,i) in the denominator. This constraint decreases the similarity between the representation of normal example and examples with important
words removed, implying that model should regard the information in the removed important words significant in prediction.
We simultaneously use the term S(neg,adv)(i,i) in the numerator to force the bias direction of outof-distribution examples with important words removed toward the decision boundary indicated by the adversarial example. We also use the term S(·,adv)(i,i) in the denominator to prevent the representation of normal example and adversarial example from collapsing together, ensuring that the adversarial example can always be utilized as a guide to locate the direction of decision boundary. It should be noted that ConAAP only focuses on alleviating the model pathology, and we leave improving the model robustness to future work.
The terms S(·,·)(i,j)+S (·,neg) (i,j) +S (·,pos) (i,j) in the denominator imply that the model should differentiate the various examples and their derived examples in a mini-batch, as the semantics of different examples should be different.
Finally, we use the LConAAP as regularization and combine it with the normal training method, which originally trains the model only with maximum likelihood. The overall objective can thus be formulated as follows:
minθ Lce(X, Y ) + α LConAAP(X) (8)
where α is a parameter balancing the two parts.

heading: 4 Experiment

heading: 4.1 Metrics
text: We measure the model pathology and the faithfulness of attribution with metrics AOPCComp. and AOPCSuff., and the parameter K in them is both set as the 40% of words for each sentence. We use AOPCDiff. to indicate the difference between AOPCComp. and AOPCSuff., i.e., the difference between the overall influence of words of different importance on prediction. Based on the Reduced Length (Feng et al., 2018), we also use IR# and UR# to measure the influence of Important and Unimportant words on prediction, measuring the number of important and unimportant words removed until the prediction changes. We use RDiff. to indicate the difference between IR# and UR#. Larger AOPCDiff. and RDiff. are expected for a nonpathological model and faithful attribution. We also use accuracy (ACC.) and confidence (P(Y |X)) on normal examples to measure the generalization ability of model on in-distribution examples.

heading: 4.2 Experiment Setup
text: Dataset. Focusing on the text classification, our experiments are performed on AG News (Zhang et al., 2015), MR (Pang and Lee, 2005), and IMDB (Maas et al., 2011). More details of datasets are provided in Appendix A.1.
Model. Three models in different architectures are adopted. For TextCNN, we reuse the architecture in (Kim, 2014) while replacing the embedding with the 300-dimensional GloVe (Pennington et al., 2014). For LSTM (Hochreiter and Schmidhuber, 1997), we connect a Bi-LSTM layer with 150 hidden units with a dense layer based on the 300-dimensional GloVe layer. For BERT (Devlin et al., 2019), we use the base uncased version.
Baseline. To show the effectiveness of ConAAP and empirically demonstrate the analysis of the bias degree and bias direction we provide in §3.3 and §3.4, we use the following baselines: (i) Normal: using (1) as objective. (ii) ConAAP: combining (7) with Normal method, using (8) as objective. (iii) Entropy: maximizing the model uncertainty on the reduced examples (Feng et al., 2018). Please see Appendix A.3 for more details on Entropy method. (iv) ConAAP w/o imp-dir: removing S(neg,adv)(i,i) ,S (·,adv) (i,i) in LConAAP, indicating removing the calibration on the bias direction of out-of-distribution examples with important words removed. (v) ConAAP w/o imp-deg-dir: removing S(·,neg)(i,i) ,S (neg,adv) (i,i) and S (·,adv) (i,i) in LConAAP, indicating removing the calibration on both the bias degree and direction of out-of-distribution examples with important words removed.
Implementation Details. The batch size is set as 64. For efficiency, we use a method called CharDelete to generate adversarial examples in tadv, which randomly deletes characters in the important words until the attack success. More details of CharDelete are in Appendix A.2. We use Adam (Kingma and Ba, 2015) as the optimizer. Most setting of learning rate / α / τ for LSTM, TextCNN, and BERT is 5e-4/1.2/0.1, 5e-4/1.2/0.1, 3e-5/1.2/0.01. All reported results are the average of five independent runs.

heading: 4.3 Main Results
text: ConAAP marginally impacts the generalization performance for in-distribution examples. Table 1 illustrates the accuracy and confidence results for in-distribution examples. Utilizing ConAAP
as regularization during training has a minimal impact on the model’s behavior for in-distribution examples, as evidenced by the marginally changed model accuracy and confidence. The accuracy difference between ConAAP and Normal training methods is within 1.11%, and the model confidence on normal examples P(Y |X) decreases by at most 0.04 compared to Normal method. These results demonstrate that imposing regularization on the sentence representations of out-of-distribution examples only slightly compromises the model’s generalization performance for in-distribution examples. Furthermore, bias degree and direction constraints in ConAAP also have only a minor impact on generalization capabilities.
ConAAP effectively alleviates model pathology. Table 2 illustrates the results on model pathology and attribution faithfulness. ConAAP consistently yields the largest values for RDiff. and AOPCDiff., indicating that the model considers the information in important words to have a more significant impact on predictions than that in unimportant words, and the attributions are more faithful. Moreover, when the calibration on the bias direction of outof-distribution examples with important words removed (w/o imp-dir) is removed, both RDiff. and AOPCDiff. decrease, indicating less faithful attributions and reduced effectiveness in alleviating model pathology. Removing the calibration on both the bias degree and direction of out-of-distribution examples with important words removed (w/o impdeg-dir) leads to further reductions in RDiff. and especially AOPCDiff. values, demonstrating the ef-
IR# UR# RDiff. ↑ AComp. ASuff. ADiff. ↑ AG News
LSTM Normal 26.59 28.74 2.15 0.07 0.03 0.04 Entropy 25.40 28.42 3.02 0.06 0.02 0.04 ConAAP 22.32 27.74 5.42 0.18 0.05 0.13
w/o imp-dir 22.97 27.50 4.53 0.18 0.09 0.09 w/o imp-deg-dir 23.05 27.76 4.71 0.16 0.09 0.07
TextCNN Normal 19.54 23.45 3.91 0.12 0.04 0.08 Entropy 19.57 24.50 4.93 0.16 0.03 0.13 ConAAP 18.68 24.59 5.91 0.22 0.04 0.18
w/o imp-dir 18.77 23.88 5.11 0.22 0.07 0.15 w/o imp-deg-dir 19.38 24.22 4.84 0.17 0.05 0.12
BERT Normal 27.74 34.72 6.98 0.03 0.01 0.02 Entropy 27.10 34.85 7.75 0.04 0.01 0.03 ConAAP 24.09 34.78 10.68 0.18 0.01 0.17 w/o imp-dir 27.04 34.68 7.64 0.12 0.02 0.10 w/o imp-deg-dir 28.08 34.92 6.84 0.05 0.01 0.04
MR
LSTM Normal 9.95 12.68 2.73 0.07 0.03 0.04 Entropy 9.17 12.03 2.86 0.09 0.04 0.05 ConAAP 8.94 12.46 3.52 0.13 0.03 0.10
w/o imp-dir 9.61 12.76 3.15 0.10 0.02 0.08 w/o imp-deg-dir 9.13 12.53 3.40 0.10 0.03 0.07
TextCNN Normal 11.42 13.40 1.98 0.08 0.04 0.04 Entropy 9.51 11.28 1.77 0.08 0.03 0.05 ConAAP 6.61 9.90 3.29 0.19 0.06 0.13
w/o imp-dir 7.09 9.88 2.79 0.19 0.08 0.11 w/o imp-deg-dir 7.23 10.28 3.05 0.17 0.07 0.10
BERT Normal 11.42 15.02 3.60 0.06 0.03 0.03 Entropy 10.43 14.91 4.48 0.05 0.02 0.03 ConAAP 10.31 15.08 4.77 0.20 0.02 0.18
w/o imp-dir 10.97 15.12 4.15 0.10 0.02 0.08 w/o imp-deg-dir 10.69 15.09 4.40 0.05 0.02 0.03
IMDB
fectiveness of ConAAP’s multi-view objective that simultaneously calibrates the bias degree and direction of the representations of various examples.

heading: 4.4 Further Analysis and Ablation Study
text: In this section, we conduct further analysis and ablation study on BERT and MR dataset.
Hyperparameter α. Figure 3(a) illustrates the influence of α. We find that AOPCDiff. begins to increase when α > 0.05 and stabilizes for α > 0.5. The accuracy is stable and will slightly increase as α continues to increase.
Temperature τ . Figure 3(b) illustrates the influence of τ . We find that ConAAP is sensitive to τ , and an appropriate τ contributes to both model accuracy and the effectiveness in alleviating the pathology. AOPCDiff. reaches its peak when
τ = 0.01, while the accuracy fluctuates by no more than 1.4% when τ is assigned different values.
Batch size. Figure 3(c) shows the influence of batch size on ConAAP. Larger batch sizes prove beneficial for ConAAP, as both model accuracy and the ability to alleviate pathology improve with an increase in batch size.
Attack method in tadv. Various attack methods can be utilized in ConAAP (Gao et al., 2018; Garg and Ramakrishnan, 2020; Li et al., 2019; Jin et al., 2020), and the influence of attack methods is shown in Figure 4. ConAAP remains effective in alleviating model pathology when utilizing different attack methods. It should be noted that adversarial examples in ConAAP are used to introduce direction information and are not intended to be nearly imperceptible to humans. Consequently, their quality is not of primary concern, and a fast CharDelete method suffices for our purposes.
Confidence Changing with Word Removal. Figure 5 illustrates the impact of word removal on model confidence. As more important words are removed, the confidence of the Normal method remains close to 1, while the label shift induced
by word removal causes the model’s confidence in the original class to approach 0 (Figure 5(a)). In contrast, the distribution of ConAAP is considerably smoother than the Normal method (Figure 5(c)). When more unimportant words are removed, the confidence for both ConAAP and the Normal method consistently concentrates in a high region (Figure 5(b)(d)).
Case study. The case study is shown in Figure 6. For the model trained with the Normal method, various interpretation methods show considerable divergence in word importance. Moreover, the model predicts the sentence with high confidence even after removing the two most important words (e.g.,
following Gradient attribution, the model predicts the sentence “One of the greatest movies ever” as positive with 87.21% confidence). In contrast, for the model trained with ConAAP, different interpretation methods show a more consistent result of word importance (e.g., important words are concentrated in greatest, movie; unimportant words are concentrated in one, of , the, ever), resulting in more faithful attributions. Specifically, when the two most important words are removed, the average confidence across different attributions is 41.43%. Conversely, when unimportant words are removed, the model can still make high-confidence predictions similar to the original examples.

heading: 5 Conclusion
text: In this paper, we argue that the failure of interpretation methods to provide faithful attributions for language models is due to the model pathology that models are overconfident in out-of-distribution ex-
amples when making predictions. We explain the model pathology from the perspective of sentence representation and propose ConAAP, a contrastive learning regularization method to calibrate the sentence representation of out-of-distribution examples. Experiments demonstrate the effectiveness of ConAAP in alleviating model pathology, which helps interpretation methods obtain faithful results. We hope that our work will provide a new perspective on research in the field of interpretability.
Limitations
We explain model pathology from a classification perspective, but the pathological nature may exist in language models for performing various tasks, such as reading comprehension, textual entailment, and visual question answering. Although our proposed regularization technique may be applicable to various tasks, we have only investigated its effectiveness in classification problems. Further evaluations are expected to be conducted in future works. The proposed method also leads to more time-consuming training, primarily due to the generation of adversarial examples, while only a minimal amount of time is spent on generating out-of-distribution examples.
Ethics Statement
This paper investigates model pathology from a sentence representation perspective and proposes a regularization technique to alleviate the pathology. It is possible that the proposed method can be used for both benign purposes, such as fixing the potential flaws and biases of models, and malign ones, such as exposing the vulnerabilities of models, which makes it easier for adversaries to generate malicious input. Despite these risks, we argue that studying model pathology openly is essential. Exploring the pathological nature of models will help us effectively control these potential risks and improve our understanding of the mechanics of natural language models. All datasets used in this paper are publicly accessible, and our work fully complies with their respective licenses.

heading: Acknowledgements
text: The authors would like to thank the anonymous reviewers for their thorough and constructive feedback. This research was supported by National Research and Development Program of China (No.2019YFB1005200).

heading: A Additional Experimental Details
text: A.1 Details on Dataset
AG News contains news articles in the areas of World, Sport, Business, and Science/Technology, with 120,000 for training and 7,600 for testing. MR contains movie reviews from Rotten Tomatoes labeled as positive or negative, with 8,530 for training and 1,066 for testing. IMDB contains binary polar movie reviews from the Internet Movie Database, with 25,000 for training and 25,000 for testing.
A.2 Details on CharDelete Attack Method
We use the CharDelete adversarial attack method in tadv to generate adversarial examples in our main experiments. The details of CharDelete are shown in Algorithm 1. ConAAP does not tend to generate high-quality adversarial examples that are imperceptible to humans and only utilizes adversarial examples to introduce direction information into regularization. This attack method meets our requirements, and a complex method is unnecessary.
Algorithm 1: CharDelete Algorithm input :Original sentence X = x1x2 . . . xN ,
model fθ , true label Ytrue output :Adversarial example Xadv
1 obtain the attribution of all input words Attr(X) by gradient-based attribution method in (3) 2 obtain the importance rankings (indexes) of input words R(X)← arg sort
i Attr(xi)i∈{1,2,··· ,N}
3 X′ ←X 4 for ri ∈ R(X) do 5 X′ ← randomly remove the letter in the ri-th
word of sentence X′
6 if argmax Y ∈Y P(Y |X ′) 6= Ytrue then 7 return X ′ as Xadv; /* Success */
8 return X; /* Fail */
A.3 Details on Entropy Method
The Entropy training method is proposed by (Feng et al., 2018). They use an entropy of the output distribution as a regularization term in the overall training objective. Specifically, the loss objective of the Entropy method is
Lentropy = ∑
(X,Y )∈(X ,Y) log(P (Y |X))
+ λ ∑ i∈{1,··· ,b} H ( P ( Y | tump/min(Xi) )) (9)
where λ is a parameter balancing the two terms, tump/min generates the sentences with multiple unimportant words reduced to the minimum length that can keep the model predictions by beam search, b is the beam width, H denotes the entropy. λ is set as 1e-3, in accordance with the original paper.

heading: B Additional Experimental Results
text: B.1 Distance Between Different Examples We also provide the aggregated results on the distance between out-of-distribution sentences and the in-distribution normal sentence in Figure 7-9. After removing important words, the density distribution of Euclidean distance between such sentence representations and the original sentences becomes smoother, with an increase in the maximum distance. However, most sentence representations remain close to the original ones (with Euclidean distance approaching 0). Intuitively, although the density distribution becomes smoother after important word removal, there is no significant horizontal shift (i.e., minimal distance changes), indicating that information from some important words does not have a sufficient impact on predictions. After removing unimportant words, the change in the density distribution of Euclidean distance between such sentence representations and the original sentence is less pronounced than when important words are removed. However, the representations of some sentences diverge considerably from the originals when only a few unimportant words are removed (e.g., distance greater than 10 in MR when only one unimportant word is removed), indicating that information from some unimportant words may have a much greater influence on predictions than expected.
B.2 Sentence Representation Distribution
Figure 10-11 show more visualization of the sentence representation and the attribution on instance sentences. Observation 1 and Observation 2 can also be observed in these examples.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: In Section Limitations.

heading: 3 A2. Did you discuss any potential risks of your work?
text: In Section Ethics Statement.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: In Abstract and Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts?

heading: In Section 3 and Section 4.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: In Section 1 and Section 4.2.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: In Section Ethics Statement.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? In Section Ethics Statement.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? All datasets utilized by us are widely adopted benchmark datasets.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? In Section 4.1, Section 4.2, and Appendix A.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. In Appendix A.1.
C 3 Did you run computational experiments?

heading: In Section 4.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? In Section 4.2.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? In Section 4.1 and Section 4.2.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? In Section 4.2.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? In Section 4.2.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Further evaluations are expected to be conducted in future works. The proposed method also leads to more time-consuming training, primarily due to the generation of adversarial examples, while only a minimal amount of time is spent on generating out-of-distribution examples