Using Interpretation Methods for Model Enhancement
Authors: Zhuo Chen
Chengyue Jiang
Kewei Tu
Venue: EMNLP 2023 Main (2023)
Keywords: interpretation methods
few-shot

Abstract: In the age of neural natural language processing, there are plenty of works trying to derive interpretations of neural models. 
Intuitively, when gold rationales exist during training, one can additionally train the model to match its interpretation with the rationales.
However, this intuitive idea has not been fully explored. In this paper, we propose a framework of utilizing interpretation methods and gold rationales to enhance models. Our framework is very general in the sense that it can incorporate various interpretation methods. Previously proposed gradient-based methods can be shown as an instance of our framework. We also propose two novel instances utilizing two other types of interpretation methods, erasure/replace-based and extractor-based methods, for model enhancement. We conduct comprehensive experiments on a variety of tasks. Experimental results show that our framework is effective especially in low-resource settings in enhancing models with various interpretation methods, and our two newly-proposed methods outperform gradient-based methods in most settings. Code is available at https://github.com/Chord-Chen-30/UIMER.

Using Interpretation Methods for Model Enhancement
Zhuo Chen, Chengyue Jiang, Kewei Tu
School of Information Science and Technology, ShanghaiTech University Shanghai Engineering Research Center of Intelligent Vision and Imaging {chenzhuo, frienzhuo,jiangchy tukw)@shanghaii tech. edu. cn
Abstract
the age neural natural language process- ing, there are plenty of works trying derive interpretations of neural models. Intuitively, when gold rationales exist during training, one additionally train the model to match its in- lerpretation with the rationales. However, this intuitive idea not been fully explored. In this paper, propose framework of utilizing interpretation methods and gold rationales to enhance models. Our framework is very gen- eral in the sense that can incorporate various interpretation methods. Previously proposed gradient-basel methods be shown as instance of our framework. We also propose two novel instances utilizing two other types of interpretation methods, erasure/replace-base based and extractor-based methods, for model en- hancement. We conduct comprehensive ex- periments variety of tasks. Experimental results show that our framework effective especially low-resource settings in enhanc- ing models with various interpretation meth- ods, and our two newly-proposed methods out- perform gradient-bact based methods in most tings. Code is available at https://e thub. com/Chord-Chen- Seeeld-Chenee///// 30/UIMER.
Input
Model
Output
Gold output
besk
Interpretations
Gradionn
Abeudd
-boued
Abaued
Interpretationn methods
Gold rationales
Figure Our framework illustration utilizing interpre- tation methods to enhance models. The dotted green line indicates how the parameters of the model are opti- mized.
Tokeens
add
step
to
ma
to the 50 clásicos playlist
Attribustan scores
0.7
0.3
0.1
0.1 0.3 0.0 0.1
Gold rationales
1
0
Figure 2: An example of attribution scores and gold rationales.
In some scenarios, we have access gold ratio- nales (input tokens critical for predicting correct outputs) during training, or have simple and fast ap- proaches to obtaining gold rationales. In that case, it is intuitively appealing to additionally train the model such that its most attributed tokens match gold rationales (see example in Fig. 2). In other words, when equipped with interpretation methods, can train the model on where to look at in the input, in addition the standard output-matching objective. This can seen as injecting external knowledge embodied rationales into the model and is especially beneficial to low-resource sce- narios with few training data. This intuitive idea, however, has not been fully explored. There are only few previous studies based on the gradient- based category of interpretation methods (Huang et al., 2021; Ghaeini al., 2019; Liu and Avci. 2019) and they neither compare the utilization of different interpretation methods on model enhance- ment nor experiment on a comprehensive range of
1 Introduction
Deep neural networks have been extensively used to solve Natural Language Processing (NLP) tasks and reach state-of-the-an performance. Due to the black-box nature of neural models, there are a lot of studies on how to interpret model decisions by giving attribution scores to input tokens, i.e., how much tokens in input contribute to the final pre- diction. We can roughly group these interpreta- tion methods into four categories, namely gradient- based (Ross et al., 2017; Smilkov et al., 2017), attention-baset (Vashishth et al., 2019; Serrano and Smith, 2019), erasure/replace-bur based (Prabhakaran et al., 2019; Kim et al., 2020) and extractor- extractor-based (De Cao et al., 2020; Chan et al., 2022) methods.
*Corresponding author.

---

tasks.
these various interpretation methods for model en- hancement. In these interpretation methods, at- tribution scores are calculated to indicate the im- portance of input tokens. According to different calculations of attribution scores, we can generally group interpretation methods into the following four categories.
In this paper, we first propose a frame- work named UIMER that Utilizes Interpretation Methods and gold Rationales to improve model performance, as illustrated in Fig. 1. Specifically, in addition to task-specific loss, we add a new loss that aligns interpretations derived from inter- pretation methods with gold rationales. We also discuss how the optimization of the task-specific loss and the new loss should be coordinated. For interpre-
Gradient- methods Gradient-based inter- pretation methods are quite popular and intuitive. Li et al. (2016) calculates the absolute value of the gradient w.r.t each input token and interprets it as the sensitiveness of the final decision to the input. Following that, Ghacini et al. (2019) extends the calculation of gradient to intermediate layers of deep models. Sundararajan et al. (2017) proposes a better method that calculates Integrated Gradients as explanations of inputs.
previous methods utilizing gradient- tation for model enhancement (Huang et al., 2021; Ghaeini et al., 2019), we show that they can be seen as instances of our framework.
We then propose two novel instances of our framework based extractor-basee based interpretation methods respectively. Specifically, in the first instance, we utilize Input Marginalization (Kim et al., 2020) as the interpre- tation method in our framework, which computes attribution scores by replacing tokens with a variety of strategies and measuring the impact on outputs, and design a contrastive loss over the computed attribution scores of rationales and nationaanssss. In the second instance, we utilize Rationale Ex- tractor (De Cao al., 2020) as the interpretation method in our framework, which is a neural model that is independent of the task model and trained with its own loss. We again design a contrastive loss over the attribution scores computed by the extractor and in addition. design new training process that alternately optimizes the task model (using the task-specific loss and our contrastive loss) and the extractor (using its own loss).
erasure/replace- based and
A Attention-basee methods The attention mech- anism calculates distribution over input tokens, and some previous works use the attention weights as interpretations derived from the model (Wang et al., 2016; Ghacini et al., 2018). However, there is no consensus as to whether attention is inter- pretable. Jain and Wallace (2019) alters attention weights and finds no significant impact predic- tions, while Vig and Belinkov (2019) finds that attention aligns most strongly with dependency lations in the middle layers of GPT-2 and thus is interpretable.
Erasure/replace-base methods The general idea is quite straightforward: erase or replace some words in a sentence and see how the model's 's pre- diction changes. Li et al. (2016) proposes method to analyze and interpret decisions from neural model by observing the effects the model of erasing various parts of the representation, such input word-vector dimensions, intermediate hidden units, or input words. Kim et al. (2020) gives a new interpretation method without suffering from the (OOD) problem by replacing words in inputs and it reaches better interpretability compared with traditional erasure-based methods.
In summary, our main contributions can be sum- marized as follows: (1) We propose a framework that can utilize various interpretation methods to enhance models. (2) We utilize two novel types of interpretation methods to enhance the model under our framework. (3) We comprehensively evaluate our framework on diversified tasks including classi- fication, slot filling and natural language inference. Experiments show that our framework is effective in enhancing models with various interpretation methods, especially in the low-resource setting.
out-of-distribune
Extractor-based methods An extractor-based method typically uses extra model to extract words that the task model pays attention De Cao et al. (2020) introduces "Differentiable Masking" which learns to mask out subsets of the inputs while maintaining differentiability. This decision is made with a simple model based on intermediate hid-
2
Related Work
2.1 Interpretation Methods
Interpretation methods aim to decipher the black- box of deep neural networks and have been well- studied recently. Our framework aims to utilize

---

den layers and word embedding layers of trained model. Chen and Ji (2020) proposes the variational word mask method to learn to restrict the infor- mation of globally irrelevant or noisy word level features flowing to subsequent network layers.
attribution scores a for input æ. can be defined on different levels of granularity. In common cases, 0g and T; are one-to-one and aj is defined by a measure calculated by the interpretation method based on the model indicating the importance of token 2; for the model in producing its output. For example, in gradient-based interpretation methods, a; is usually some function of the gradient of a; in the model. A higher magnitude of the gradient implies higher importance of input Zz.
2.2 Utilizing Interpretation Methods to Enhance Models
Some previous work studies whether interpreta- tion methods can be utilized to enhance models. Du al. (2019) is closely related one of our contributions (i.e., UIMER-IM in Sec. 3.2). How- ever, their model suffers from the OOD problem and does not consistently outperform baselines in their experiments. Ghacini et al. (2019) designs a simple extra objective that encourages the gra- dient of input to have a positive effect on ground truth. They find it useful in event-extraction and Cloze-Style question- nneswinn-answering tasks. To handle Non- Con-Imaprtnnt Rationales and exploit Potential Im- portant non-rationales, Huang et al. (2021) de- signs their method following the idea that ratio- nales should get more focus than non-rationales and only part of the rationales should attain higher focus. They focus on text classification tasks and find that, despite the rationale annotations being insufficient and indiscriminative, their method can bring improvements. However, these previous stud- ies neither compare the utilization of various in- terpretation methods on model enhancement nor conduct experiments on a comprehensive range of tasks.
Learning Objective Apart from the original task- specific learning objective Liasks our framework introduces an extra learning objective Line that em- bodies the idea of aligning attribution scores with gold rationales g. The overall objective on one ex- ample takes the form of:
Lg (1, y) a (a, g)
(1)
where is the model parameter and a is a coeffi- cient balancing the two objectives.
Warm-up Training Lint can be seen as measur- ing whether the model pays attention the gold ra- tionale words in the input. We deem that compared to teaching a randomly initialized model focusing more on the task task-specific rationale words, teaching a model with task knowledge is more effective and reasonable because it might be better at recogniz- ing task-specific rationales with the help of task knowledge rather than rote memorization. Thus, instead of optimizing Lg at the very beginning, our framework requires the model to be well at least halfway trained before training
3 Method
the objective Lint, and during warm-up training, only the objec- tive of the task is optimized. The empirical results (in Sec. 4.2) also support our intuition and show that warm-up training is effective.
We propose a framework UIMER that utilizes an interpretation method to enhance models based on gold rationales on training data.
Setup Consider training example with input and gold output y. We also have access to gold rationales g of input æ indicating the subset of tokens that are critical for predicting the correct output y. The gold rationales g can be annotated by humans or generated automatically from external knowledge sources.
In the following subsections, introduce three instances of our framework. The first utilizes gradient-based interpretation methods and sub- sumes a few previous studies as special cases. The second and third are new methods proposed by us utilizing erasure/replace-baae and extractor- interpretation methods respectively.
In our setup, g is encoded as a 0/1 vector with gi 1 indicating that 2; is gold rationale and l. = 0 otherwise. Our method, however, can be easily extended to handle g encoded with real num- bers indicating the importance of token.
3.1 Utilizing Gradient-Based Methods
As introduced in Sec. 2.2, some previous studies utilize gradient-based based (GB) interpretation methods to enhance models. They can be seen as instances of our framework, hence denoted as UIMER-
Given a model tasked with predicting output from input x, an interpretation method produces

---

In this type of methods, attribution score a is usu- ally defined by a function f of the gradient of input a::
3. Uniform: ((<%/)) [ where [ is the vo- cabulary size.
(aJ (JJ
a =
(2)
We follow one of the strategies sample S based on q(>r,,) 2, and define at as:
where usually J refers to the training objective or the task model's ss output.
0: log,
loga (od2((ms)(m)) (4)
In general, Lint is defined as a constraint on the gradients of gold rationales:
where
Lint (a, g) g)
(3)
m Σ
g(<l,) q(ac))ooulzz)))
where D is usually distance function that calcu- lates the enror of how approaches g.
a,c5
In Ghaeini et al. (2019)'s work, f is function that takes the sum of the gradients of each input embedding dimension and is to take the sum of the gradients of rationale words. In Huang et al. (2021)'s work, f is the L1 norm that sums up the absolute value of gradients over the input embed- ding dimensions and D is designed in various ways to give rationale words higher altribution scores.
odds(p) - pp(11
For inputs with only one gold rationale word, computing and optimizing the attribution score is easy. However, there might be more than one ratio- nale in general and the calculation of the attribution score of each rationale token becomes impractical when the input length and the number of rationales get larger. Therefore, bution score defined in the original IM method multi-token attribution score which can be more efficiently computed.
extend the token attri-
3.2 Utilizing Erasure/Replace- Based Methods
We incorporate an erasure/replaced-aased interpre- tation method, "Input Marginalization" (IM) (Kim et al., 2020), into our framework in this subsection and name this instance UIMER-IN We first de- fine the attribution score produced by IM and then define and introduce how to calculate Lint- Other crasure/replace-based methods can be integrated into our framework in similar way.
Formally, for input x with more than one ru- tionale word, denote x'R as a new sentence which all rationale words are replaced, and x'y- as a new sentence in which the same number of non-rationale words are replaced. 3 The way to gen- erate one æ'R is by replacing one rationale word time using the strategies mentioned before. We de- note the score of replacing x to x', as q q(2r22) and ((x()) is calculated as the average of the replacing probabilities of rationale words using a certain re- placing strategy Similarly. I'y is generated and (<xNx) can be defined. We repeat this generating process and denote the set of generated (xn(('r)) as (R(N with the size of (5 being hyper- parameter. Then the attribution scores ag for the entire set of rationales and a,y for the same number
3.2.1 Attribution Score by Input Marginalization
Define po(y)z) x) as the probability of the gold output that the model predicts. To calculate the attribution score of token F,, new set of sentences S needs to be generated, with the size of S being hyper- parameter. Denote x', new sentence with at; replaced by some other token u. Denote q(xx,, as the probability of replacing z; with u to obtain x'y- which can be determined by different strategies!;
1. BERT: Replace x; by [MASK] to obtain the masked language model probability of u.
"When c(x(.) is determined by BERT strategy, fol- low the method IM to construct S. Words with the highest probabilities selected rather than sampled replace
2. Prior: q(xr',) count(u)/N, where count(u) is the number of times token u appears in corpus and is the corpus size.
'The same number of non-rationales be replaced are chosen randomly from input.
"Here we deviate from the calculation of Kim et al. (2020) that multiplies the probabilities. If there are many rationale words in sentence and we take the product of probabilities. the sentence with the most probable words replaced by BERT will have dominant probahility compared with others, which often dogenerates the calculation of attribution score.
'In our experiment, we also include a strategy "MASK"" which means 2; is simply replaced hy the [MASK] token and ( =

---

of non-rationales defined as:
2. A term constrain the changes of task model's prediction after feeding the masked input (or hidden states).
aR = log2 (wddddddddppppseep logs (odd((((()))))
logg(odds(mm))
aN = ((<)))p(()(z'))
3.3.2 Definition of Lint in UIMER-DM
mR Σ
With attribution scores defined in DM, define contrastive loss Lint follows:
EBR
mN = Σ
((a'x)Dolylarr))
2
CSN
Lint a, g) =
Σ =g1=1
min
a max dj
1,
1-9.=0
3.2.2 Definition of Lint in UIMER-IM
(7)
For a given input æ and gold rationale g, with OR and aN defined, we expect the attribution score of rationale words to be higher than that of non- rationale words. Thus, we design contrastive margin loss:
Intuitively, we encourage the attribution score of rationale words to be higher than the attribution score of non-rationale words. The loss will be zero as long the maximum attribution score of non- rationale words is lower than the attribution score of any rationale word.
Line(a, g) mar(an OR 0,
3.3.3 Training in UIMER-DM
where is a positive hyperparameter controlling the margin. Here a is not defined w.r.t each token, and it refers to the attribution score for multi-tokens. Note that when calculating aN ag the term log2 (odds(ss(poo)))z)) is canceled out and does not need to be computed. We choose to the margin loss instead of simply maximizing ag and minimiz- ing o.v because many cases non-rationale words may still provide useful information and we do not want to eliminate their influence on the model.
When training UIMER-DM, there are two objec- tives and two sets of parameters, Lg (Eq. 1) for model parameter 0 and L for extractor parame- ter Φ. Intuitively, the two sets of parameters should not be optimized simultancously. That is because our framework requires accurate interpretation model (i.e., the extractor here). If the extractor is trained the same time with the model, then since the model keeps changing, there is no guarantee that the extractor could keep pace with the model, and hence its interpretation of the model may not match the latest model, breaking the requirement of our framework.
3.3 Utilizing Extractor-Based Methods
In this section, we incorporate "DiffMask" (DM) (De Cao et al., 2020), an extractor- based interpre- tation method into our framework and name this instance UIMER-DM DM. Other extractor-basee based inter- pretation methods can be integrated into UIMER in similar way.
We adopt the following training schedule to cir- cunvent the problem. First, we follow the warm- up strategy and train the model. After that, we alternate between two steps: (1) optimizing the extractor parameters & w.r.t. LDM with the model parameters 0 frozen: (2) optimizing the model pa- rameters A w.r.t. Lo with the extractor parameters frozen. The number of rounds that we alternately optimize & and 0 and the number of epochs in each round are hyperparameters.
3.3.1 Attribution Score by the Extractor
DM, the attribution scores a are produced by a simple extractor model Ext, parameterized by ø.
= Ext (Fn(())
(6)
where Enc(w) refers the encoding of input produced by an encoder model, and is composed of real numbers the range of 0 to 1. Here a is defined one-to-one w.r.t. each token æ. The extractor needs to be trained by its own objective LDM composed of 2 parts:
Experiment
4.1 Experimental Settings
Datasets To evaluate our framework. exper- iment with all the methods introduced in the pre- vious section on three tasks: Intent Classification (IC), Slot Filling (SF) and Natural Language In- ference (NLI). The three tasks take the forms single sentence classification, sequence labeling
1. A term to encourage masking the input (or hidden states) as much as possible.

---

Task
Dataset
Source Rationales
Rationales
IC SNIPS
Human annotation
add step to me to the 50 clásicos playlist
SNIPS
Regular Expression
rate the current essay 2 out 6
Premise: children smiling and waving at camera
NLI
e-SNLI
Given in the dataset
Hypothesis: the kids are frowning
Table Examples of rationales on the three tasks in our experiments.
and sentence pair classification respectively. For Intent Classification and Slot Filling, we adopt the SNIPS (Coucke al., 2018) dataset. For Natural Language Inference, we adopt the e-SNLI (Cam- buru et al., 2018) dataset. SNPIS is widely used in NLU research (Jiang et al., 2021 Chen et al., 2019). It is collected from SNIPS personal voice assistant. There are 13084, 700 and 700 samples in the training, development and test sets respec- tively, and there are 72 slot labels and intent types. e-SNLI a large dataset extended from the Stan- ford Natural Language Inference Dataset (Bowman annotated natural
lines. We also regard the two previously-proposse gradient-based methods introduced in Sec. 3.1 stronger baselines.
Training Settings We mainly focus on low- resource settings with limited training examples, which is when external resources such as rationales can be most beneficial. With less training data, there is increasing return of utilizzig rationales. For Slot Filling and Intent Classification, we com- pare our methods and baselines on 1-shot, 3-shot, 10-shot, 30-shot and 100% of training data. For Slot Filling, n-shot means that we sample train- ing examples such that each slot label appears at least times in the sampled examples. For Natural Language Inference, we compare our framework and baselines with 100 and 500 training samples from e-SNLI. Hyperparameters development set and report the average result of random seeds on the test set. Detailed data about hyperparameters shown in App. 8.3.
et al., 2015) to include language explanations of the entailment relations. There are 549367, 9842, and 9824 samples in the training, development and test sets respectively.
Rationales We give examples rationales and show how they are obtained in Table For the Intent Classification task, wc ask one annotator to construct set of keywords for each intent type based on the training set. This only takes less than 15 minutes. For the Slot Filling task, use 28 regular expressions with simple patterns which ref- erence Jiang al. (202Ib)² to match the sentences in the SNIPS dataset and regard matched tokens as rationales. The job takes less than 30 minutes (less than minute each on average). The complete ra- tionale sets for Intent Classification and Slot Filling task are shown in App. 8.1. For e-SNLI, we use the explanations provided by Camburu et al. (2018).
tuned on the
4.2 Main Results
We present the main results in all the settings in Table 2. First, from the Mean column, see that gradient-based methods (UIMER- Ghaeini et al. (2019); Huang et al. (2021)) reach better per- formance than the base model, and all variants of our proposed UIMER- IM and UIMER-DA meth- ods outperform both UIMER GB and base models. For UIMER-IM, its variants achieve the best performance in eight of the twelve settings and the second best in the rest of settings, suggesting its general applicability. The "+Uniform" variant of UIMER-IM can be seen to clearly outperform the other variants in the 1/3-shot settings on In- tent Classification and we analyze the potential reason behind this in App. 8.2. UIMER-IM with the "BERT (warm.)" variant brings 14.86% gain for the NLI 100-example setting.
Baselines For Intent Classification task, we choose BERT-base-uncasee unccsee + Softmax our base model?. For Slot Filling, we choose BERT-base- uncased CRF our base model. For Natural Language Inference, we prepare the data into the form "[CLS] premise [SEP] hypothesis [SEP]". feed into DERT-base-uncased and apply linear layer to the hidden state of the /CLS) token to score the entailment. These base models are natural base-
For UIMER- -DM, it achieves the best perfor- mance in the 1/3-shot settings and competitive the 10-shot setting on Slot Filling, which indicates
'hese regular expressions designed to extract slots. 'ht tps: //:///hue //tgggggggggaaa ace. co/bert base-uncased We implement it based on JoinBERT

---

IC (Aec.
SF (F1)
NLJ (Aec.)
Mean
Model
full
30
full
500
79.18
97.64
38,
81.70
73.11
UIMER-GB Glacini al. (2019)
79.14
97.92
37.77
51.69
67.57
95.99
67.13
75.07
Base
67.04
83.04
91.43
94,57
98. 21
39.02
50.66
67.11
80,30
95.
66.15
08.57
75.10
UIMER-GO
Gate
67.14
82.01
91.39
94.07
98.077
37, 84
51.63
67,34
80,89
95,50
66.06
68.31
75.02
Huang (2021)
81.82
38,
$1.08
95.39
75.05
(Gale+Onder)
67.71
80.25
92.11
94.39
98.39
38.86
51.73
67.76
80.55
95.77
65.10
65.84
74.87
Our Framesord
MASK
69.85
83.17
91.18
93.86
98,00
38.68
52.47
69.27
81.67
95.83
63.36
70,04
75.61
UIMER IM
BERT
70.61
83.93
91.78
94.61
97.86
39
51.96
69.22
81.85
95.77
62.08
69.03
75.67
Prior
83.93
76.06
Uniform
73.32
86.04
91.64
94,00
98,04
39.31
51.37
69.02
81.69
95.49
64,34
69.19
76.12
MASK (wam..)
70.82
82,96
92.43
94.04
98.25
38.60
31.55
67.93
82,25
94, 89
68.79
09.95
76.04
(warm.)
70.93
82.71
92.00
94.32
97.82
39.53
82,58
95.
76.28
UIMER-IM
Prior (warm.))
73.82
83.11
91,93
94.07
97.82
38.24
51.68
68.26
82.66
95.82
68.25
71,82
76.46
Uniform (warm))
75.79
86.29
91.67
94.32
98.11
38.71
52.18
68.21
82.17
95.39
67.58
69.79
76.68
One-pass
66.75
84.42
91.53
93.78
97.96
39.86
52.87
67.28
81,90
95.22
63.03
66.91
75.13
UIMER-DM
Multi-round
95.54
76.18
Table 2: Evaluation of our framework on three tasks. Underlines mark the results of our UIMER-IM/DE that outperform the base model and UIMER-GB methods. Boldface marks the best results among all the methods. The Mean column gives the average each row.
warm-pp
72
71,31
71
70.72
70.01
70.01
70
69
69.26
68.
68.21
68
67.28
67
Brachiiiel Tox
SAEMER Dm Dev
Cuo-Paox
20 Rounds
50
Figure 3: The curves of Lint with and without warm-up training on 1-shot Intent Classification over random seeds.
Figure 4: Ablation study of multi-round training on 10- shot Slot Filling.
shot setting of the Intent Classification task. particular, we inspect the value of objective Lint during training with and without warm-up training. as shown in Fig. 3. It can be seen that with warm- up training, the Tint objective starts with lower value and converges zero faster, verifying the benefit of warm-up training.
that an extra extractor might be more capable of locating rationales than other interpretation meth- ods for structured prediction problems. Applying multi-round training to UIMER- can be seen to clearly improve the performance in almost all the settings of all three tasks. We conduct an abla- tion study on the effect of multi-round training of UIMER-DE DD in Sec. 5.2.
5.2 Multi-Round Training
From the results, it is also evident that in general, the performance gap between any instance method our framework and the base model becomes larger with less data, implying the usefulness of rationales when training data cannot provide suffi- cient training signals for model enhancement.
In our method UIMER-D -M,, we propose an asyn- chronous training process that trains the model and the extractor asynchronously for multiple rounds. We conduct a study on the effectiveness of the multi-round training process on 10-shot Slot Fill- ing and the results are shown in Fig. 4. We observe that by iteratively and alternately training the ex- tractor and the model, performances on both the development and the test sets show an upward trend with more rounds. Note that we use fewer training epochs (around 10 for the extractor and 20 for the
5 Analysis
5.1 Warm-up Training
We conduct an ablation study on the effect warm- up training of our method UIMER- IM in the

---

add karusellen jazz brasileiro
Prediction
BaseModel
Oug 1.787 < 6x 2.355 (gg 2 6.098 ay 0.544
Play
UIMER-IM
AddToPlaylist
what the movie schedules
Prediction
MadeM
Oug < 0.
SearchCreative Work SeanchSrrreennggEEvn
UIMER-IM
(gg
ag 1.465
rate
current
chronic
stars
Prediction
BaseModel +DM
0.99
0.99
0.99
0.72
0.99
0.95
x ScarchCreativv Work
UIMER-DM
0.00
0.00
0.00
0.00
0.00
0.99
RateBook
play
the
newest
music
by
evil
jared
hasselhoff
Prediction
BaseModel +DM
0.99
0.99
0.99
0.99
0.99
0.00
0.99
0.99
PlayMusic
UIMER-DM
0.99
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Play Play
Table 3: A case study that analyzes the task performance and quality of interpretations of the base model and our methods UIMER-IM&S "BaseModel +IM/DM Att..: The attribution scores produced by base model and IM/DM interpretation method. UIMER-IM/DAN The attribution scores produced by our framework.
model) each round of multi-round training than one-pass training. 3-round training does not outper- form one-pass training simply because it has few total training epochs.
Acc.
Acc.
%
Acc.
S
BascModel +IM
68.86 77.99
37.86
81.57
65.71
UIMER-IM
36.59
85.33
75.79
BaseModel +DM UIMER-DM 75.75
72.44
52.44
58.57
65.71
5.3 Case Study
16.52
81.28
70.21
To show that our framework is able both enhance the model in task performance and give better in- terpretations, we conduct case study on 1-shot Intent Classification setting. From the first two ex- amples in Table 3, we can see that the base model can neither predict the correct intent label of the sentence nor produce good interpretations (the attri- bution scores of non-rationales are higher than the scores of rationales); in comparison, our UIMER IM fixes both problems. In the last two examples, we show that UIMER -D succeeds in lowering the attribution scores of irrelevant parts in the input and producing high scores for some or all of the rationales. It can also be seen that the extractor trained on the base model in 1-shot settings views most of the inputs being important, while the tractor UIMER-DMY is much more parsimonious and precise.
Table 4: Performances when the model gives higher attribution scores to rationale words or not.
rationales, as shown in the first and second columns in Table 4. For methods with the DM interpretation method, 0 is calculated by averaging the attribu- tion scores for all rationale words in and aN is calculated by averaging the attribution scores for all non-rationale words. We can see that when our UTMER-IM/DDI method correctly recognizes ra- tionale words, reaches higher accuracy than the base model, which suggests that helping models pay more attention to rationales can additionally improve the task performance.
Conclusion
Though many interpretation methods are studied for deep neural models, only sporadic works uti- lize them to enhance models. In this paper, we propose framework that utilize various in- terpretation methods enhance models. We also propose two novel instances utilizing two other types of interpretation methods for model enhance- ment. In addition, we discuss how the optimization of the task-specific loss and the new loss should
5.4 Relation Between Attribution Score and Performance
In this section, study how the model performs the test set when succeeds fails to give rationale words higher attribution scores. We con- duct experiments on I-shot Intent Classification and calculate the accuracy while giving rationale words higher or lower attribution scores than non-

---

be coordinated. Comprehensive experiments are conducted on variety of tasks including Intent Classification, Slot Filling and Natural Language Inference. Experiments show that our framework is effective in enhancing models with various inter- pretation methods especially in the low-resource setting, and our two perform gradient-based methods in most settings. For future work, plan to extend our framework utilize more forms of rationales and additional interpretation methods.
Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages
Qian Chen, Zhu Zhuo, and Wen Wang. 2019. Bert for joint intent classification and slot filling. arXiv preprint arXiv: /902: 10909.
methods out-
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbreehtt Francesco Calta- girone, Thibaut Lavril, al. 2018. Snips voice plat- form: an embedded spoken language understanding system for private-by srivate-by-desigee voice interfaces. arXiv preprint arXiv:1805. 100.000000000000
7 Limitations
Nicola De Cao, Michael Sejr Schlichtkrull, Wilker Aziz, and Ivan Titov, 2020. How do decisions emerge across layers in neural models? interpretation with differentiable masking. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 3243-3255
It can be inferred from the result that the two newly introduced methods do not give the best perfor- mance in rich-resource settings. We prospect that method UIMER-IM IM plays a role in incorporating the information of rationales by introducing more similar inputs to the model when the training data is scarce. However, when training data is sufficient enough for the task, the effect of this way to supply information on rationales is reduced. For method UIMER-DAE also does not perform the best in rich-resource settings. We attribute the ordinary performance of UIMER-DM to that with rich data, most knowledge can be implicitly learned by the model, and injecting gold rationale doesn't help.
Mengnan Du, Ninghao Liu, Fan Yang, and Xia Hu. 2019. Learning eredible deep neural networks with rationale regularization. In 2019 IEEF International Conference on Data Mining (ICDM), pages 150-159. IEEE.
Reza Ghaeini, Xiaoli Fern, Hamed Shahbazi. and Prasad Tadepalli. 2019. Saliency learning: Teaching the model where to pay attention. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 1 (Long and Short Papers), pages 4016-4025.
Acknowledgement
This work was supported by the National Natural Science Foundation of China (61976139).
Reza Ghaeini, Xiaoli Fern, and Prasad Tadepalli. 2018. Interpreting recurrent and attention-based neural models: a case study on natural language inference. In Proceedings of the 2018 Conference Empiri- cal Methods Natural Language Processing, pages 4952-4957.
References
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. large annotated corpus for learning natural language inference. In EMNLP.
Quzhe Huang, Shengqi Zhu, Yansong Feng, and Dongyan Zhao. 2021. Exploring esstatly-labblle rationales neural network models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 557 1-582....
Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-snli: Natu- ral language inference with natural language expla- nations. Advances Neural Information Processing Systems, 31.
Sarthak Jain and Byron C Wallace. 2019. Attention is not explanation. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, Volume (Long and Short Pa- pers), pages 3543-3556.
Aaron Chan, Maziar Sanjabi, Lambert Mathias, Liang Tan. Shaoliang Nie. Xiaochang Peng. Xiang Ren, and Hamed Firooz. 2022. Unirex: A unified learning framework for language model rationale extraction. International Conference on Machine Learning. pages 2867-2889. PMLR.
Chengyue Jiang, Zijian Jin, and Kewei Tu. 2021a. Neu- ralizing regular expressions for slot filling. In Pro- ceedings of the 2021 Conference Empirical Meth- ods Natural Language Processing, pages 9481 9498.
Hanjie Chen and Yangfeng Ji. 2020. Learning varia- tional word masks to improve the interpretability of neural text classifiers. In Proceedings of the 2020

---

Arrest my book of Chengyue Jiang, Zijian Jin, and Kewei Tu. 2021b. Neu- ralizing regular expressions for slot filling. In Con- ference Empirical Methods in Natural Language Processing.
Original sentence
Intent RateBook
Rate my current book out
Replace the rationale by BERT
Rating my current book 1 out of 6
Siwon Kim, Jihun Yi, Eunji Kim, and Sungroh Yoon. 2020. Interpretation of nip models through input marginalization. In Proceedings of the 2020 Con- ference Empirical Methods in Natural Language Processing (EMNLP), pages 3154-3167.
Give my current book 1 out of 6
RateBook
my current book out of
Replace the rationale randomly
Operate my current book out
RateRook
Sell my current book 1 out of
Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. Un- derstanding neural networks through representation erasure. arXiv preprint arXiv: 161..2260
Figure The underline marks a rationale. Replace the rationale by BERT v.s. randomly.
Frederick Liu and Besim Avci. 2019. Incorporating priors with feature attribution on text classification. In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics, pages 6274- 6283.
Yequan Wang, Minlie Huang, Xiaoyan Zhu, and Li Zhao. 2016. Attention-basse Istm for aspect-level sentiment classification. In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 606-615.
Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Appendix
Vinodkumar Prabhakaran, Ben Hutchinson, and Mar- garet Mitchell. 2019. Perturbation sensitivity analy- sis to detect unintended model biases. In Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3740-5745.
8.1 Patterns to match rationales
In experiments, adopt simple patterns match rationales. For Intent Classification, dic- tionary is constructed, as shown in Table (upper), mapping intent labels to rationales, and for Slot Filling, a group of Regular Expressions, as shown in Table 5 (below), is used to extract rationales. Both tasks are derived from SNIPS dataset that contains 13084 examples, and obtaining both the dictionary and the Regular Expressions is simple and fast. Please note that though the table below seems little complex, the gray parts are just the syntax of Regular Expressions. Only the black parts contain rational information.
Andrew Slavin Ross, Michael C Hughes, and Finale Doshi-Veles Velez. 2017. Right for the right reasons: Train- ing differentiable models by constraining their expla- nations. arXiv preprint arXiv: 1 703.03377.
Sofia Serrano and Noah A Smith. 2019. Is attention interpretable? In Proceedings of the 57th Annual Meeting of Association for Computational Lin- guistics, pages 2931-2951.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. 2017. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825
8.2 Why is "+Uniform" Good
In experimenting with Sec. 3.2, we found that re- placing rationales/non--ationnne. randomly, i.c. the "Uniform" strategy often produces better result even than the "BERT" strategy, despite the fact that compared replacing tokens randomly, pre- trained BERT can apparently produce more fluent sentences. Here we give an example in Intent Clas- sification task that possibly explains the cause of this fact in Fig. 5.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Interna- tional conference machine learning, pages 3319- 3328. PMLR.
Shikhar Vashishth, Shyam Upadhyay, Gaurav Singh Tomar, and Manaal Faruqui. 2019. Attention in- terpretability across alp tasks. arXiv preprint
100.:11999.11288.
As stated in Sec. 3.2, aim to minimize Lint defined in Eq. 5 in which want lower the latter term in aR. and it weighted sum of the predictions of these sentences generated by BERT on the ground truth intent. However, we see that BERT pretty much keeps the original meaning of
Jesse Vig and Yonatan Belinkov. 2019. Analyzing the structure of attention in transformer language model. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing Interpreting Neural Networks for NLP, pages 63-76.

---

the sentence in this example. Thus lowering the latter in ag seems be contradiction the origin task. In contrast, we look at the sentences generated by replacing the rationale randomly they contain much less information about the ground truth intent, and minimizing Lint seems to be more reasonable.
8.3 Hyperparameters
For Intent Classification and Slot filling task, the learning rate To is tuned among (8e-5, le-4, 2e-4, 4c-4) in 1/3/10/30-shot settings and {le-5, 2c-5) in full training data setting. For Natural Language Inference task, the learning rate of Lg tuned among (8e-5, le-4, 3e-4). In all three tasks, a is tuned in the range [0.001, 20).
tuned in the range [0.01, 10]. We use AdamW optimizer (Loshchilov and Hutter, 2018) and "linear schedule with warmup" scheduler. Detailed hyperparameters are shown in Table 6-8.
8.4 Result with std.
We show the result with unbiased estimation of standard deviation in Table in all few-shot set- tings.

---

Task
Intent Label (keys)
Rationales (values)
AddToplaylist
[add. playlist, album, list]
BookRestaurant
[hook, restaurant, reservation, reservations
GetWeather
[weather, forecast, warm, freezing, hot, cold] [play, music, song, hear]
IC
PlayMusic
RateBook
[rate, give, star, stars, points, rating, book]
SearchCreative Work
[find, show, called]
SearchScreeningvennt
[movie, movies, find, theatres, cinema, cinemas, film, films, show]
Task
Regular Expressions
[ forlshowidowwloodlgee)) (?Perationale I> ((Peraatioalee>>atth))) ((cllee)
* (?Penoe [0-5] htw:://wwwomddeeweeeeee//lllieie (?Pcrationaaee> pointsstttsssss
* (?Perationalc: c>>a rating of) (IPenonn- (031///ooooowooottttceooohh////
* ((Perratonaale-ratee) ratelggive) "("P<rationalel>e I> of).*
"[IPnnone>
(?P<rationale> of) DecnoneeIGlsixy
((hislcurrent)
schedulelmovie schedulessalbumlmoii schedules!
movie timeseessayteextooottt whwwsssggiaaallcooooooooooeeeeee showigamelpainting!
Ly
((Perationale>ade add) "(/Perationalel> tot).
(PP (Perrationll>-addlil "(?P<craionnaell>> 1>to ( (PPcrationlle:-paaalis).
PPerationales playy playlist).*
* ((P<rationale>my my)) * [[P"rrationalel>alaliii playlist).*
* (PPerratoonall>scc hww.aa/asaaaaaannnanaannssssnnna
(PPcrationalel>hyy..
*( @P www.tersssumlllllnessadllllooooe
((CP<rationalel>inn)
SF
* (IP<crationale>>et what is the weather).*
*( ieceatioonnewwwaaaheewwwwewe forecast).
*( *( (|Perratonall>bboo book) (?P<none>a).*
*(
truckleafeteria).*
*( *(P P nearestioooossstccchhydoeeee bylin the neighborhoodlin the area).*
*( [P<craiioallettt Pecaationlee-aabeellatssseesssss tabbeseattieeeevaioollsstuarttte
(([P<raaionalll>>oo)) (ttoo))"((cattionneeeeepppppp
* (PPrrationale b> movie houselcinemalmovie theatre).*
*( *2P (PPrrationale>>>> when islwhat time islfind melwhere istiskseelwatch).
* www.aaleeeeeefiiiiinsssroooooooo sharklgoogle musidderehhpriiyxxooyymmdllmaaaa
pandoomshhckeriiihaattineeellll fm).*
[[Perationale-aninaae animated movies/filmslfllm)
* (PPrrratonaale>ee forlat) heteee/iiid/eeeciiinicoeeeooooor
*(I (IP<rratonaale-ii highly meddlbeshhoppullrro-neeedg......
*(2 (PPPrrttinaaleccooai hwtpee/weeellllleolllleeeennnnne
*0 (
* OPPrraiionaee> bherelcurrent positionlcurrent locationleurrent placelcurrent spot).*
Table 5: The upper Table refers to the dictionary we construct match rationales for each intent type for Intent Classification task. The bottom refers to Regular Expressions to match rationales for the Slot Filling task. Tokens following <rationale*>> tag are annotated rationales.

---

Maetred
spechs
slphs
wanp gpockal
wop
UIMER-EN +RFECT
0.5
UIMER ESI +Pbor
12:1989:22553:42
0,0003
70
1
0.3
UTVER- -Ms Iniferr
15-996- 1283-12333-47
0.0012
24
70
0.6
1olel
UIMER-IM -MASK (wamm))
55 1988-12533:42
0.0034
70
0.5
Todla V160-5X022-3088
UIMER -II3CT (warm)
55 1989- 1255-4
0.0022
70
0.08
DIMER-IM
1963:
UIMER-DO One-peea
1989:
0,0005
UIMER DW Malti mund
55, 1988:125538
12553:22
0,0002
0.001
UIMER Is +MASK
55, 1989912333344
0.0004
70
0.6
0.5
UIMER Est -HIRTT
55 1255-12553-42
0.0003
24
70
0.6
0.5
UIMER 4Piior
25. 151891122333
0.0022
70
001
UIMER -nnifor
12353-23333-2
0.5
(womm)
V100-SYM2-9208
195612233342
UIMER -Ptor (xarm)
1257-12
0/0022
ULMER Ist Lailfom (warn)
55 1988:12533:42
0,0002
70
1
0.3
UIMER DM One poss
15 198--1255--42
0,0004
0.001
24
70
0.5
ULMER Des Mulli cound
25, 1988-12533:42
0.0002
0,0025
50
50
UIMER-EM -MASK
55 1988 12533-22
0,0002
70
0.1
0.5
UIMER BERT
0.1
UIMER-IM
1969:
0.1
ULMER-
10 shor
VINER- Eot -EEKT (wams)
UIMER-IM
(xoem)
55, 1988-12533.42
1233342
0,0022
70
0.1
0.01
Teda V100 SXM2 32GB
UIMER 151 -Prior (warm))
25 1986:12333342
0,0022
70
4
0.11
10
UIMER-IA EM Uaiffa (warm))
55 198812253:44
0,0022
70
0.01
tack tnited"
UIMER Dos
1288112333:42
Malis-mand
125811233344
198812238842
LIMER-BEE -Proor
LIMER-IM -inifor
SS. 1989: 12
0.0011
50
001
30 sbot
UTMER-BE FM RFFT (wamm))
UTMIRR Ist -SSAK (somm))
55 1988 1985-12551-47
SS 1969 12333-42
0.0011
0.0011
24
50
50
0.88
0.6
0.5
0.08
NVIDIA TITAN V
UIMER--EE
1988,
Tiiifem
12559-43
UIMER-Do Onc-pte
25
UIMER-DO Malt-mand
106%:
0.002
0.00001
UIMER Ett -MASK
0000001
30
0.1
0.5
LIMER- -BF3TT -IFSTT
15
0.00001
24
30
0.01
0.8
UIMER 15t +Pror
55
0,00011
30
0.1
0.5
UIMER iniform
000000
30
0.01
0.0
UIMER +MASK
000001
0.5
NVIDSA
ULMER
(wazaa)
0.5
UIMER Est -nniform (wam])
000000
0.000
0.8
LIMERR- FXM One-ppea
55
0.0000
0.001
10
0.1
UIMER De Multi round
55
0.0001
0.001
30
0.1
30
Table 6: Hyperparameters for task Intent Classification. task trained:: The origin task is firstly well trained, then objective is optimized.

---

Sentin
Method
seed
atpto
GPU
-MASK
0.0032
LIMER--SM -Pricr
188:
0.0032
0.6
UIMER Est -Uniform
12:199-12553:42
0.0002
24
0.6
05
TIMER-TM -MASK (oom)))
1255-1555:-2
0.0032
24
70
0.08
0.5
UIMER-IM -BEXT (wam)))
55 151188112333--
0,0022
24
0.6
0,1
NVIDIA TITAN
UIMER-IW -Srise
55 1969 12351-2
0,0022
0.5
UIMER-IM -Lhifoen
12383-2
pain
DIMER-Do: Malismond
12333-42
0.0032
0.001
UIMER-IW -MASK
1953-12
0.0031
24
UIMER Lst -BERT
55. 22119612553-42
0.0021
24
70
0.08
0.1
UIMER-IP MMMER-I---E -Prior
55: 12158212553--2
0.0031
24
70
0.1
UIMER Lst -Luifor
55 13198812333242
0.0021
24
70
3sho
UIMER Est -MASK (worm.)
UIMER-IM -BFST
55: 1989-12555:22
0.0001
0,005
NVIDIA TITAN
UIMER EM
(warm))
195-12333--2
000031
LIMER--T -Uniform (worm)
0.0021
UIMER Das One pas
55: 198-912533322
0.0021
0.013
24
70
0.1
UTMER-DE Dis Multimonnd
55 19-3199--7 12553-22
0.0001
0.003
24
70
3
UIMER Ltt -MASK
55. 12199912233-42
0.001
24
70
UIMER-IM -EETT
S5. 1233312333-2
0.001
24
70
0.01
UIMER
55 1986-12333-4
UIMER-IM -Uhifiam
55 1988-1233444
10-shat
UIMER- 1st
UIMER [st -BUR (wamm)
(waras)
1989-12533--22
1988-12333-4
0.001
24
0.01
NNIDIA TITAN
UIMER 1st -Piior (wesss))
55. 1988.12533.42
0,001
24
70
UIMER LM -Uniform (warm))
15:1988:12533:-
0.001
24
70
UIMER- Das Ove
S5 1988:12333-42
0.00201
0.0001
24
Mulii
198-12353-42
0,00201
0.0001
10
UIMER-IM -MASK
S5 1988.12333-4
10
UIMER-IM
UIMER-IM -Pricr
12388-2233344
0.0031
10
LIMER-IM Uniifom
0.0031
10
30-shot
UIMER-IM -MASK (waam.)
LIMER-- -H133CT coomm))
SS. 1938.12333.42
55 55:069:12730 12551-2
0,0001
0.00000
24
24
50
50
6001
10
0.08
0.05
10
10
NVIDIA TITAN V
UIMER-IM -Svior (stem.)
SS 1068:12333-44
0,00203
24
0.001
0.05
10
-Lailoun
10
UIMER-DD
SS: 12188422233:4
0.0001
0.08
UIMER-DM Muli-saand
55. 123331233
0.0021
0.0001
0.08
TIMER-IM
0.00001
UIMER Is -BUKT
55
0.00001
24
0.01
TIMER-FM -Prior
55
0.00001
24
70
011
UIMER IM -Uniform
55
0,00201
24
20
0.0001
TIMER-IM -MASK (xorm.)
UIMER -BLR
55
0,00001
0.00201
24
00000
NVIDRA TITAN
LTMER-
(woem.)
DIMER-DM -Uhiftem (osem))
0.00001
0.0001
LIMER- Dad One pois
0.00001
0.011
24
DIMER-DD Mult-mmnd
55
0.00001
0.001
24
20
50
Table 7: Hyperparameters for task Slot Filling. task trained*: The origin task is firstly well trained, then objective is optimized.
Setting
Methol
seal
bz
maa qpudis
sipho
quilia
WOEE epochs
Hmmlll-counck
cualy
GPU
UIMER IM #MASK
55: 95:19991223338 42
0.0023
32
UIMER-IA -BERT
55 $5.198.12233344
0.0033
0.01
UIMER IM +Prior
1998:12333-42
0.0033
DIMER- Iniform
15-1954-1993--2
0.0005
31.
100
LIMER- IM -11 BURT courn)
UIMER 1st +MASK (warm.)
55: 35:1995-12333.
55- 1253333333-3
0.0005
0,0003
32
32
50
1
0.03
0.1
0.1
0.1"
NVIDIA A40
UIMER 1st PPior (wara.))
55 1988-12333 42
0.0033
32
50
0.1
0.03
0.3
UIMER IM Uniifom
55 1969-12333-0
0.0055
0001
001
UIMER
pai
1988112333...
0.0038
0.001
1969:12333-42
0,0033
0.001
000
UIMER- AMASK
55
16
0.03
DIMER-IM -BERT
123881333344
0.0008
16
001
UIMER-IM &Prior
55.1985. 12383.2333.44
0,0001
0.03
8.I
TIMERRISM +1 Iniffrm
55: 1965:12385 3-333-42
0L0001
16
S00 UIMER-IA Ist +MASK (ocum.)
SS 12383. 42
0.00005
10
001
30
NVIIIT TTEAN X2
UIMER -BERT (war)))
55.
0.0001
16
30
UIMER-ISt
55 1988.12388.4
0.00005
30
UIMER- Lt UUiifom (wara))
55 1198,112333 1223612333-44
0,000055
30
DIMER-DA Cuc-qpaco
1988:12388844
0,0008
0.001
16.
001
UIMER DaI Mulli sound
55. 1965. 1253312333.4
0,00005
0.001
16
000
Table 8: Hyperparameters for task NLI. 0.1#: 10% of the whole mini-batches are used to do warm-up training.

---

7.38
2.51
2.76
4.22
2.76
4,22
5.22
4.70
2.99
2.20
0.78
1.51
3.57
1.30
4.77
500
62.84
09.77
68.57
68.33
68.57
04.57
65.84
70.04
69.03
70.32
69.19
69.95
71.82
69.79
66,91
67.60
70
69.
NLI
13.66
0.86
1.37
1.13
3.50
7.62
1.19
6.04
2.58
7.87
2.44
0.88
3.35
1.77
4,92
2.94
100
54.03
67.13
66.15
66.06
68.44
65.10
63.36
62.08
66.56
64.34
68. 79
68.89
68.25
67.58
63.03
65.44
0.12
0.57
2.64
2.24
1.25
1.30
1.90
1.02
1.9
1.38
0.96
1.32
1.54
1.40
1.54
30
81.70
82.16
80.20
60.39
81.08
80.55
81.67
81.85
81.97
$1.69
82,25
82.58
82.66
82.17
81.90
82.00
80.8
0.58
0.47
1.22
0.53
0.68
0.76
1.21
1.92
0.71
0.89
0.90
1.85
1.38
0.46
0.71
0.55
10
67.05
67.57
67.11
67.34
67.55
67.76
69.27
69.22
69.02
67.93
68.26
68.21
67.28
69.26
68.
SF
1.23
0.77
0.98
1.76
1.52
1.18
0.88
1.82
1.35
1.18
1.03
0.71
2.25
0.81
50.97
51.69
50.66
51.63
50.99
51.73
52.47
51.96
51.37
51.55
52.83
51.68
52.18
52.87
53.10
Table Result with std. on all few few-shot shot settings.
1.92
1.22
1.7
1.77
1.44
1.83
2.73
1.80
1.21
2.14
2.23
1.73
2.27
2.11
1.53
1.17
38.14
37.77
37.84
38. 18
38.68
39 28
38.07
99.31
38.60
99.53
38.24
38.71
39.86
41.32
39
38
0.62
1.30
0.20
1.02
1.38
1.07
1.32
0.64
0.43
0.60
0.54
0.11
0.74
0.80
0.62
0.60
30
93.79
94.18
94.57
94.07
94.36
94.39
93.86
94.61
93.96
94.00
94.04
94,32
94.07
94.32
93.78
94.00
1.40
0.18
1.14
0.05
1.21
1.46
0.50
1.16
1.48
1.59
1.64
1.08
1.21
0.58
0.92
10
91.00
91.82
91.43
91.39
50.52
92.11
91,18
91.78
91.00
91.64
92.43
92.00
91.93
91.67
91.53
91.92
IC
6.17
6.12
4,83
3.19
4.35
5.7
5.32
3.69
5.68
2.16
3.11
4.07
2.19
151
5.42
4,40
79.18
79.
83.04
82.01
11.82
80.25
83.17
83.93
83,93
86.04
82.96
82.71
83.11
86.29
85. 86
7.30
7.30
9.11
5.19
4.71
6.24
4.24
2.52
4.75
4.02
2.40
5.56
5.97
4.94
7.54
8.57
65.71
65.71
67.04
67. 14
65.25 15.28
67.71
69 85
70.61
73.71
73.32
70.82
70.93
73,82
75.79
66.75
70.21
MASK (warm.)
BERT (warm.)
Uniform (war))
+ Prior (warm..)
(gate+oorder)
UIMER GB Gitacini al. (2019)
Multi-round
+ Uniform
+ MASK
base
order
BERT
Prior
One-
Model
Huung et (2021)
UIMER Gis
UIMER-II Im
UIMER-Im
UIMER Dm
Baseline
Ou Commenee


Reviews:
[{"review_id": "7J1835lNkN", "reviewer": "Reviewer_xBwV", "summary": "", "questions": "[Q1] Baselines: Are there other stronger baselines for the tasks like intent classification? The practical impact of this paper might be decreased if a stronger baseline outperforms the combination of the proposed method + a weak base model. Moreover, it is still possible that the proposed method cannot bring an improvement to the stronger baseline model. \n\n[Q2] Experimental metrics. The metrics seem to include standard metrics like F1. However, the reviewer is wondering whether the generated interpretation is also measured and serves as another metric as shown in Figure 1 in Introduction? \n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your suggestions. As for your concern:\n\n[-] Experiment: \n\n- We indeed evaluated UIMER in rich-resource settings (i.e., full data of the IC and SF tasks, results shown in the two “full” columns in Table 2) and almost all methods in UIMER outperform the base model. Experiments on even larger-scale training data and out-of-distribution settings are interesting future work.\n\n[-] Model: \n\n- This work is not trying to solve specific “challenges”. Instead, we aim to 1) generalize previous work into a framework that can utilize various interpretation methods to enhance models, 2) propose two novel instances of the framework, and 3) perform a more comprehensive empirical evaluation.\n\nQ1:\n\n- No matter how strong a baseline is, there would always be tasks and (low-resource) settings in which additional knowledge embodied in rationales is helpful, and that’s when our framework can be useful. Our experiments showcase such settings using the most widely used baselines.\n\nQ2: \n\n- The third column (% $a_R > a_N$) of Table 4 in Sec. 5.4 shows an evaluation of generated interpretations. We can see that with UIMER-Im and -Dm, rationales in the test sets are better recognized (i.e., more rationale tokens are given higher attribution scores). For your reference, we further measured the same metric of UIMER-Im in the rich-resource setting on Intent Classification task and found a consistent conclusion:\n    |  |  | Acc. ($a_R > a_N$) | Acc. ($a_R < a_N$) | % |\n    | --- | --- | :---: | :---: | :---: |\n    | BaseModel +Im | 1shot | 68.86 | 37.86 | 81.57 |\n    | BaseModel +Im | full | 99.01 | 98.15 | 90.34 |\n    | UIMER +Im | 1shot | 77.99 | 36.59 | 85.33 |\n    | UIMER +Im | full | 99.46 | 33.33 | 99.46 |\n\n    Note that “Acc. ($a_R$ * $a_N$)” shown here is slightly different from Table 2 because the Acc. on samples without gold rationale cannot be included ($a_R$ and $a_N$ cannot be calculated without gold rationales annotated in the sample.)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "[+] Research topic. Model interpretation is an emerging topic. Also, utilizing interpretation to enhance model performance is an interesting direction.\n\n[+] Experiments and evaluation. Extensive experiments on three tasks with two datasets are conducted. Detailed analysis is provided with a concrete case study to demonstrate the effectiveness of the proposed framework. \n\n[+] Presentation. The paper is well-presented with proper figures and tables. The introduction is easy to follow to get the key idea in a short time.\n", "reasons_to_reject": "[-] Experiment: The experiments focus on low-resource setting. However, it would be interesting to see some results when there are large-scale of training data. Even though rationales might not help improve the overall performance in settings with lots of training data, it would be interesting to explore some other problems, e.g., would the performance on out-of-distribution samples can be improved with external resources? \n\n[-] Model: The challenges solved by the proposed model seems not introduced accordingly. Even though the authors argue that SoA methods do not compare different interpretation methods on different tasks, the missing of proper justification of challenges would decrease the technical contribution and novelty of this paper. \n\n[-] Presentation: In Table 1, it would be better to use full names (e.g., intent classification) of tasks because there is enough space to make it easier to read with full names.\n", "questions_to_authors": "[Q1] Baselines: Are there other stronger baselines for the tasks like intent classification? The practical impact of this paper might be decreased if a stronger baseline outperforms the combination of the proposed method + a weak base model. Moreover, it is still possible that the proposed method cannot bring an improvement to the stronger baseline model. \n\n[Q2] Experimental metrics. The metrics seem to include standard metrics like F1. However, the reviewer is wondering whether the generated interpretation is also measured and serves as another metric as shown in Figure 1 in Introduction? \n", "paper_topic_and_main_contributions": "This paper proposes a general framework for utilizing model interpretation to enhance model performance. The main contribution of this paper lies in the exploration of utilizing two novel interpretation methods to enhance model performance and extensive evaluation on various tasks is conducted. Compared with existing papers in this area, this study designs a more general framework and conducts experiments in more tasks and datasets. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iY7yGZ3W6C", "reviewer": "Reviewer_fRry", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed reply. As for your concern: \n\nReasons To Reject:\n\n- [1-7]:\n    \n    We want to clarify that our work is valuable and novel despite [1-7].\n    \n    [1,4] do not utilize interpretation methods and solely utilize rationales.\n    \n    [6,7] focus on improving model interpretability and do not leverage gold rationales for model enhancement, which does not match our motivation or setup.\n    \n    [2,3,5] are all instances of our framework. [2,3] belong to UIMER-Gb and are similar to previous work discussed in Sec. 3.1. As we mentioned in the paper, UIMER-Gb is not our contribution.\n    \n    [5] is similar to UIMER-Im+MASK and is the only work among [1-7] that is closely related to one of our main contributions (i.e., UIMER-Im). However, [5] has the OOD problem and does not outperform baselines in their experiments (Table III of [5]). In contrast, our UIMER-Im under various settings consistently outperforms the baseline in our experiments. \n    \n    Most of the works only experiment on classification tasks [1,2,5,6,7] in rich-resource settings [1-7] and we comprehensively evaluate classification, slot filling and natural language inference tasks in both low and rich-resource settings.\n    \n\n    Below we discuss each reference in more detail.\n    \n    - [1] is a work that utilizes information stored in rationales but not a work that utilizes interpretation methods. An extra module apart from the task model is responsible for extracting rationales and it is trained to align gold rationales. However: 1) there are no concepts of utilizing interpretation methods because the extra module just extracts tokens as how it is trained by rot memorization. \n    \n    - [2] and [3] can also be seen as instances as we discussed in Sec. 2.2, similar to Ghaeini et al. (2019) and Huang et al. (2021). In [2] the authors apply “Integrated Gradients” techniques that are also discussed in our paper (line 125). \n    \n    - [4] uses the “Context Decomposition (CD)” method to decompose logits $g(x)$ into two terms and view one of them ($\\beta$) “importance score”. CD was first introduced by “Murdoch et al., Beyond Word Importance: Contextual Decomposition To Extract Interactions From LSTMs”. Rieger et al. modify the method to apply to other model architectures (CNNs).\n    \n        1) Obviously, it is model-dependent while our framework is model-agnostic. \n    \n        2) The decomposition process still remains black-box: $\\beta$ is still not interpretable when it is defined. Though it can memorize some rationales after supervised training, it is not able to find the tokens that influence the model’s output the most. In our work, all interpretation methods are interpretable themselves without the need for supervision from gold rationales/explanations. \n    \n        3) We further propose “Multi-Round” training strategy (Sec. 5.2) that better coordinates task loss and interpretation loss (refer to the definition in Equation 1 in [4]). \n    \n    - [5] calculates the importance score $s^t$ by measuring the deviation of the prediction between the original input and the partial input $x$ with token $t$ omitted which suffers **Out-Of-Distribution (OOD) problem** discussed by “Kim et al., Interpretation of NLP models through input marginalization (the third paragraph in their Introduction)”. From Table III in [5], the method does not outperform the baseline methods in half of the settings. \n    \n    - [6] and [7] focus on improving model interpretability and do not leverage gold rationales for model enhancement. Compared to [6], we further propose “Multi-Round” training strategy that better aligns the “interpretation module” and “task model”, which results in better performance than “one-pass” training (Sec. 5.2). As for [7], they design a new layer that can discover interactions between words. However, we do not aim to develop new interpretation methods/modules, instead, we aim to utilize interpretation methods and gold rationales for model enhancement. \n    \n\n- High-quality rationales:\n    \n    The approaches we use to obtain rationales are simple and fast as stated in “Sec. 4.1, Rationales”. Despite the rationales being not perfect, we still find that just simple rationales and proper utilization of various interpretation methods are able to enhance models. \n    \n- Other model architectures:\n    \n    It is interesting to use UIMER on other base models, which we leave as future work.\n    \n- Performance:\n    \n    First of all, we would like to point out that our performance improvement over the baseline is quite consistent in low-resource settings (i.e., not using full data), as shown in Table 2 and also Table 9 (showing standard deviations). Second, a much smaller improvement on full data is to be expected. The table below shows that with full training data on Intent Classification, the base model can already achieve near-perfect task accuracy as well as very high interpretation accuracy (90.34%), so our framework has little room to further improve the task accuracy by increasing the interpretation accuracy. \n    \n    |  |  | Acc. (a_R > a_N) | Acc. (a_R < a_N) | % |\n    | --- | --- |:---: |:---: |:---: |\n    | BaseModel +Im | 1shot | 68.86 | 37.86 | 81.57 |\n    | BaseModel +Im | full | 99.01 | 98.15 | 90.34 |\n    | UIMER +Im | 1shot | 77.99 | 36.59 | 85.33 |\n    | UIMER +Im | full | 99.46 | 33.33 | 99.46 |\n    \n    Note that “Acc. ($a_R$ * $a_N$)” shown here is slightly different from Table 2 because the Acc. on samples without gold rationale cannot be included ($a_R$ and $a_N$ cannot be calculated without gold rationales annotated in the sample.)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-\tThe idea of utilizing interpretations to improve model performance is promising.\n-\tThe paper is well-written and easy to follow.\n-\tA comprehensive study (with different interpretation methods and training paradigms) is conducted.\n", "reasons_to_reject": "-\tWhile this work contributes valuable insights, its novelty appears limited. The utilization of interpretations to enhance model performance has been extensively explored in both supervised and unsupervised approaches [1-7]. \n-\tThe proposed method, which relies on gold rationales as additional supervision, may pose limitations in real-world applications due to the expense of collecting high-quality rationales.\n-\tOnly the BERT-base model is tested. It would be interesting to see how the proposed method generalizes to other model architectures.\n-\tThe performance improvement, particularly when using full data, appears to be marginal. Further investigation into the underlying reasons for this observation would be valuable.\n\n[1] Zhang et al., Rationale-augmented convolutional neural networks for text classification, 2016.\n\n[2] Liu and Avci, Incorporating priors with feature attribution on text classification, 2019.\n\n[3] Ross et al., Right for the right reasons: Training differentiable models by constraining their explanations, 2017.\n\n[4] Rieger et al., Interpretations are useful: penalizing explanations to align neural networks with prior knowledge, 2019.\n\n[5] Du et al., Learning credible deep neural networks with rationale regularization, 2019.\n\n[6] Chen and Ji., Learning Variational Word Masks to Improve the Interpretability of Neural Text Classifiers, 2020.\n\n[7] Sekhon et al., Improving Interpretability via Explicit Word Interaction Graph Layer, 2023.\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a framework called UIMER that utilizes interpretation methods and gold rationales to improve model performance. Interpretation methods (e.g., gradient-based methods, erasure/replace-based methods, and extractor-based methods) identify important features in the input, which are then aligned with gold rationales during training. The overall objective combines the original task-specific learning objective and the interpretation alignment objective. A BERT-base model is adopted as the baseline model and tested with UIMER using different interpretation methods on three tasks (Intent Classification, Slot Filling, and Natural Language Inference). Various training paradigms are investigated across few-shot and full-data settings.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Zhang et al., Rationale-augmented convolutional neural networks for text classification, 2016.\n\n[2] Liu and Avci, Incorporating priors with feature attribution on text classification, 2019.\n\n[3] Rieger et al., Interpretations are useful: penalizing explanations to align neural networks with prior knowledge, 2019.\n\n[4] Sekhon et al., Improving Interpretability via Explicit Word Interaction Graph Layer, 2023.\n\n[5] Du et al., Learning credible deep neural networks with rationale regularization, 2019.\n"}}, {"review_id": "IH98UDKUh2", "reviewer": "Reviewer_PcW8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your suggestions for our work. As for your concern:\n\nFramework:\n\n- We would like to argue that this is just a matter of wording and should not be a reason for rejection. We choose the word “framework” because different interpretation methods can be plugged into our “framework” following the same setup as defined at the beginning of Sec. 3. We could perhaps replace “framework” with “paradigm”.\n\nSec. 5.3:\n\nSec. 5.3 is just a **case study** and by nature it contains only a few representative examples. As for more rigorous support for our claims:\n\n1. Sec. 4.2 “Main Result” shows that our framework is able to enhance models in performance.\n2. Sec. 5.4 shows that our framework is also able to enhance interpretability (in particular, the third column (% $a_R > a_N$) of Table 4). We can see that with UIMER-Im and -Dm, rationales in the test sets are better recognized (i.e., more rationale tokens are given higher attribution scores).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper demonstrates how to use model interpretation techniques and gold rationales to improve the performance of the model on certain tasks. \n2. Their method produces results (on three datasets) that are better than existing techniques that try to use gold rationales. \n3. I appreciate that they talk about how they obtained the gold rationales since these aren't often present in datasets and can be a major practical limitation. It is promising to see that even simple techniques to create gold rationales led to good improvement in the results.", "reasons_to_reject": "While I appreciate that they demonstrate how to use three different model interpretation techniques in their framework, calling the idea a \"framework\" seems a bit misleading. This is mostly because frameworks often have multiple components in the system that are novel yet remain constant across different implementations. Whereas, in this paper, the main part of the framework, the \"L_int\" function changes for each model interpretation technique. \n\nSection 5.3 seems to be based on just a few examples and doesn't seem to justify the authors' claims that their method is able to enhance the model in performance as well as interpretability.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In situations where gold rationales can be obtained for a task, the authors demonstrate how to use these rationales as an additional source of supervision while training the models. Model interpretation techniques allow us to gauge the level of \"importance\" the model places on each token. These feature importances are used to gauge the model's rationale. When training the model, the authors add a new task to the objective function to measure the distance between the model's rationale (obtained by model interpretation techniques) and the gold rationale (either 0 or 1 for each token).\n\nTheir main contributions are - \n1. They demonstrate how to implement their idea of using gold rationales to provide additional supervision by using three different existing model interpretation techniques. They try different (minor) variations when using each model interpretation technique, providing the reader with some guidelines for how to squeeze out the best performance.\n2. Using three different datasets, the authors demonstrate that their technique outperforms the baseline (not using gold rationales) as well as existing techniques that use similar training strategies. Furthermore, they provide empirical evidence showing that the gain in performance is larger in low-resource settings.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Please consider increasing the label font size for the x and y axis labels in Figure 3."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a general framework, UIMER, which leverages model interpretation techniques and gold rationales to enhance model performance. The study explores various interpretation methods (e.g., gradient-based, erasure/replace-based, extractor-based) to identify key input features, aligning them with gold rationales during training. The framework combines task-specific objectives with interpretation alignment objectives. Experimental evaluation spans multiple tasks and datasets, showcasing its superiority over baseline models and existing techniques, particularly in low-resource scenarios. 

The paper tackles the promising area of using model interpretation to boost model performance, showing potential in this domain. It excels in conducting extensive experiments across diverse tasks and datasets, making it well-structured and informative.
Nonetheless, it predominantly focuses on low-resource settings, leaving questions about its scalability to larger training data and adaptability to out-of-distribution scenarios. Addressing these aspects could enhance the paper's relevance and impact.
Decision: Accept-Main