Breaking Free from MMI: A New Frontier in Rationalization by Probing Input Utilization
Authors: Wei Liu
Zhiying Deng
Zhongyu Niu
Jun Wang
Haozhao Wang
Zhigang Zeng
Ruixuan Li
Venue: ICLR 2025 Poster (2025)
Keywords: Interpretability
natural language processing
feature selection
Primary area: interpretability and explainable AI

Abstract: Extracting a small subset of crucial rationales from the full input is a key problem in explainability research. The most widely used fundamental criterion for rationale extraction is the maximum mutual information (MMI) criterion. In this paper, we first demonstrate that MMI suffers from diminishing marginal returns. Once part of the rationale has been identified, finding the remaining portions contributes only marginally to increasing the mutual information, making it difficult to use MMI to locate the rest. In contrast to MMI that aims to reproduce the prediction, we seek to identify the parts of the input that the network can actually utilize. This is achieved by comparing how different rationale candidates match the capability space of the weight matrix. The weight matrix of a neural network is typically low-rank, meaning that the linear combinations of its column vectors can only cover part of the directions in a high-dimensional space (high-dimension: the dimensions of an input vector). If an input is fully utilized by the network, it generally matches these directions (e.g., a portion of a hypersphere), resulting in a representation with a high norm. Conversely, if an input primarily falls outside (orthogonal to) these directions, its representation norm will approach zero, behaving like noise that the network cannot effectively utilize.  
Building on this, we propose using the norms of rationale candidates as an alternative objective to MMI. 
Through experiments on four text classification datasets and one graph classification dataset using three network architectures (GRUs, BERT, and GCN), we show that our method outperforms MMI and its improved variants in identifying better rationales. We also compare our method with a representative LLM (llama-3.1-8b-instruct) and find that our simple method gets comparable results to it and can sometimes even outperform it.

Published as a conference paper at ICLR 2025
BREAKING FREE FROM MMI: A NEW FRONTIER IN RATIONALIZATION BY PROBING INPUT UTILIZATION
Wei Liu', Zhiying Deng": Zhongyu Niu', Jun Wang?;
Haozhao Wang';
Zhigang Zeng". Ruixuan Li
School of Computer Science and Technology. HUST
2 Faculty of Artificial Intelligence in Education, Central China Normal University
$iWudao Tech "School of Artificial Intelligence and Automation, HUST
[ide_Iw.zy_mn .iiu, hz.wang, rxli} @ @hust.edu.cn
zhiyingdzy @ @gmail.com *jwang @wwudao.tcch, "zgzeng@hust.edun
ABSTRACT
Extracting a small subset of crucial rationales from the full input is key prob- lem in explainability research. The most widely used fundamental criterion for rationale extraction is the maximum mutual information (MMI) criterion. In this paper, we first demonstrate that MMI suffers from diminishing marginal returns. Once part of the rationale has been identified, finding the remaining portions con- tributes only marginally to increasing the mutual information, making it difficult to use MMI to locate the rest. In contrast to MMI that aims to reproduce the prediction, we seek to identify the parts of the input that the network can actu- ally utilize. This is achieved by comparing how different rationale candidates match the capability space of the weight matrix. The weight matrix of a neural network is typically low-rank, meaning that the linear combinations of its col- umn vectors can only cover part of the directions in a high-dimensional space (high-dimension: the dimensions of an input vector). If an input is fully utilized by the network, it generally matches these directions (e.g., portion of a hyper- sphere), resulting in a representation with a high norm. Conversely, if an input primarily falls outside (orthogonal to) these directions, its representation norm will approach zero, behaving like noise that the network cannot effectively uti- lize. Building on this, we propose using the norms of rationale candidates as an alternative objective to MMI. Through experiments on four text classifica- tion datasets and one graph classification dataset using three network architec- tures (GRUs, BERT, and GCN), we show that our method outperforms MMI and its improved variants in identifying better rationales. We also compare method with representative LLM (Iamma-3. 1-8b 1-8b-instruct) and find that our simple method gets comparable results it and can sometimes even outperform it. Code: pestps://githan www hoomm//jggeceegggz.// i ona lizat
INTRODUCTION
With the success of deep learning, there are growing concerns over the model interpretability. Ex- ploring the theory and technique of interpretable machine learning frameworks is of immense im- portance in addressing a myriad of issues. For instance, XAI techniques can aid in detecting model discrimination (fairness) (Pradhan et al., 2022), identifying backdoor attacks (security) (Li et al. 2022), and revealing potential failure cases (robustness) (Chen et al.) 2022 Zhang et al. 2024), among others. Post-hoc explanations, which are trained separately from the prediction process, may not faithfully represent an agent's decision, despite appearing plausible (Lipton) 2018). In contrast to post-hoc methods, ante-hoc (or self-explaining) techniques typically offer increased transparency (Lipton] 2018) and faithfulness (Yu et al. 2021), as the prediction is made based on the explanation itself. There is a stream of research that has exposed the unreliability of post-hoc explanations and called for self-explanatory methods (Rudin) 2019; Ghassemi et al. 2021 Ren et al. 2024).
"Corresponding authors.

---

Published as a conference paper at ICLR 2025
Encoder
RNN Torsfonn
Pooking
Linaer
Skgnzid
Enc(Z)
Y [0, 1]
greel lotel with eacellen: locaioon the breaifast buffet temific service wonderful.
Extrador
preet botel with sacellent locaioon the bnvaddaatt buifet tamifii. i-wiindaciil.
Prodicto
0.1
M
mming ,o H,.Y.
0.9
X
Z
The standard rationalization framework RNP. The task is binary sentiment classification about the hotel's location. X, Z, Y,Y.Y. represent the input, the extracted rationale candidate, the prediction and the ground truth label. respectively. M is sequence of binary masks. Enc( 2) is the encoder's 's final layer representation (like the term "embedding" in (Lee et al.) 2024 Bolukbasi et al.) 2021). DE, Op represent the parameters of the extractor and the predictor. He denotes cross-entropy.
Figure 1:
In this study, our primary focus is on investigating a general model-agnostic self-explaining frame- work called Rationalizing Neural Predictions (RNP, also known as rationalization) (Lei et al., (016), which with its variants has become one of the mainstream methods to facilitate the interpretability
of NLP models (Sha et al. [202]] Yu et al., 2021; Antognini et al. [202] Liu et al. 2022] 2023a c
2024b/aj Storek et al., 2023 Liu et al. 2024c| Zhao et al., (024)) and also holds the potential to be applied to image classification (Yuan et al.) 2022 and graph neural networks (Luo et al. 2020).
RNP utilizes a cooperative game involving an extractor and predictor. This game is designed with focus on "data-centric" (i.c., it is to explain the connection between text and the model-agnostic task label, rather than explaining the output of a specific model) feature importance. The extractor first identifies the most informative part of the input, termed the rationale. Subsequently, the ratio- nale is transmitted to the predictor to make predictions, as illustrated in Figure [ Apart from its use for interpretability, some recent studies find that rationalization can also serve method for data cleaning. The extracted (Z, Y) pairs can act as new dataset, and trained with such a cleaned dataset, a predictor may be more robust (Chen et al. 2022) and generalizable (Wu al. 2022 Gui et al. 2023), thanks to the removal of task-irrelevant, harmful information.
The commonly used objective for finding rationales is maximizing the mutual information between the rationale candidates and the task labels (e.g., by minimizing cross-entropy), which is called the maximum mutual information (MMI) criterion. In practice, however, MMI faces the problem of diminishing marginal utility. For example, if the rationale candidate consists of 80% the real rationales and 20% random noise, it might be informative enough to help predictor make the correct prediction (see a specific toy example in $4.1. and empirical verification in Figure (b).. In this case, replacing the left 20% noise with real rationales can only trivially improve the mutual information (or intuitively, the prediction accuracy. See $ for the theoretical perspective). As result, the gradient provided by MMI cannot guide to find the left 20% real rationales well.
To avoid the shortcomings of MMI, instead of following the traditional approaches of fixing MMI's flaws through various regularization terms, this paper aims to find an alternative objective to the MMI criterion. Since the MMI criterion has been the fundamental objective in the XAI literature for a long time, our finding is important as it opens a new avenue for extractive interpretability without the fundamental MMI. The overall idea can be summarized as follows: (1) The rationales are those the predictor can learn and utilize. (2) Neural networks usually have low-rank weight matrices (Kang et al. 2024), meaning that the linear combinations of its column vectors can only cover part of the directions in a
space (high (high-dimension: the dimensions of an input vector). (3) For features that the network does not learn, their directions are usually orthogonal to the learned directions (see Appendix 14), and the representation norms through the weight matrix will approach zero. On the contrary, weal-learned features usually interact with the learned directions of the weight matrix, resulting in representations with higher norms (Appendix A.15 provides toy example for intuitive understanding). This property is borrowed from recent advances (Kang et al.) 2024) in the -ut-o--distribution (OOD) detection field. Based the above properties, we attempt
2

---

Published as a conference paper at ICLR 2025
to find rationales by maximizing the norm (i.e., [Enc(Z)ll in Figure I of the rationale candidate's 's representation, whose motivation is further empirically supported by the results in FigureBic)
The key difference between our method and the mainstream MMI approaches can be summarized at a high level: MMI aims to find rationales that can reproduce the prediction, while our study seeks identify the parts of the input that the network can actually utilize (i.e., match the non-zero rank subspaces of the weight matrix), whose idea is more in line with the philosophy of ante-hoc explanation. Our work opens new eyes in the XAI literature, as almost all current mainstream XAI methods follow the MMI criterion or its variants. The idea to observe which parts of the input can be utilized by the network from the perspective of forward propagation is also novel, as this is the first time that an extractive XAI method has been freed from relying on the model's final output (e.g,, may be used in the future to probe task insk-irrelevant (i.e., not fine-tuned) pretrained encoders. But this is beyond the scope of this paper and is left for future work.).
The contributions can be summarized as follows: (1) We empirically find the diminishing marginal utility problem of identifying rationales with the MMI criterion (Figure B. Considering that MMI a widely used criterion for finding explanations, this empirical observation may somewhat remind the XAI community to rethink this fundamental criterion. (2) We formally analyze the reasons why the diminishing marginal utility can occur with the MMI criterion, providing insights for further researchers to better address this issue. (3) Based on a theoretical property borrowed from the OOD research, we propose an alternative objective to the MMI criterion. Empirical results on both text and graph data with three different encoders (GRUs, BERT, and GCN) show that our method not only outperforms the vanilla MMI, but also beats its several recently improved variants.
RELATED WORK
Extractive Rationalization. The self-explaining framework of rationalization named RNP (Lei ct al. 2016) is flexible and offers unique advantage: certification of exclusion, which means any unselected input is guaranteed to have no contribution to prediction, making it important to the NLP community YYu al. 2021). Based on it, many methods have been proposed improve RNP from different aspects. |ao et al. (2018) used Gumbel-softmax to do the reparameterization for binarized selection. Bastings et al. (2019) replaced the Bernoulli sampling distributions with rectified Ku- maraswamy distributions. Jain et al. (2020) disconnected the training regimes of the generator and predictor networks using a saliency threshold. Paranjape et al. (2020) imposed a discrete bottleneck objective to balance the task performance and the rationale length. De Young et al. (2020) proposed a benchmark that can be used for supervised rationale extraction. 3PL.AEER Yuu et al. 2019)) tries to squeeze the informative texts from the unselected parts to produce comprehensive rationales. DMR (Huang et al. 2 tries to align the distributions of rationale with the full input text in both the output space and feature space. A2R (Yu et al.) 202 endows the predictor with the information of full text by introducing a soft rationale. FR (Liu et al. 2022) folds the two players to regularize the predictor with the extractor (as the extractor can view the raw input) by sharing a unified encoder. Inter RAT (Yue et al., (023) tried to use backdoor adjustment to alleviate the spurious correlations in the raw dataset. |ernandes al. (2022) leveraged meta-learning techniques improve the qual- itt of the explanations. Havrylov et al. (2019) cooperatively trained the models with continuous and discrete optimisation schemes. (Hase al. 2020) explored better metries for evaluation. (Ra- jagopal et al. (02T)) used phrase-based concepts to conduct a self-explaining model. Other methods like data augmentation with pretrained models (Plyler et al., 202T). training with
rationales (Chan et al., 2022). injecting noise to the selected rationales (Storek et al.] (2023), using attack techniques to inspect spurious correlations (Liu et al., 2024c), have also been tried.
All of these previous studies take MMI as the fundamental criterion of finding rationales, and the diminishing marginal utility problem has been overlooked. The purpose of this paper is to analyze the diminishing marginal utility problem and to alleviate it, which is orthogonal to previous research.
The properties of the network's 's learned inputs. Some previous research has found that complex neural networks typically have low-rank weight matrices (Aghajanyan et al. 2021). Kang et al. 2024) shows both theoretically and empirically that the weight matrices and network representa- tions associated with the learned inputs often occupy dineesnnnio subspaces with high overlap. However, when the network encounters unlearned OOD inputs, their associated representations tend to have less overlap with the weight matrices compared to those the network has learned. As re-
3

---

Published as a conference paper at ICLR 2025
sult, the feature representations corresponding to the unlearned OOD inputs tend have smaller norms than those the learned inputs, resulting in less signal being propagated from the input. Our method is inspired by this theoretical property.
Generative explanation with LLMs. Generative explanation is a research line that is close but orthogonal to research extractive explanation. With the great success of LLMs, new search line for explanation is chain-of-thought. By generating (in contrast to selecting) intermediate reasoning steps before inferring the answer, the reasoning steps can be seen kind of explanation. The intriguing technique is called chain-of-thougle (CoT) reasoning (Wei et al. 2022). However, LLMs sometimes exhibit unpredictable failure modes (Krcıman et al. 2023) or hallucination rea- soning (Ji et al.) (023)), making this kind of generative explanation not trustworthy enough in some high-stakes scenarios. Also, some recent research finds that L.LMs are not good extractive tasks
QQin et al. 2023 Li et al. 2023 Ye et al. 2023)
The potential impact of rationalization in the era of LLMs. Compared to traditional "model- centric" XAI methods which solely focus on the model's learned information, "data-centric" ap- proaches primarily aim to extract model-agnostic patterns inherent in the data. So, apart from im- proving interpretability, rationalization can serve as method of data cleaning /Siler (2023).
Dpmaain-ppecficc large models often require supervised fine-tuning using domain-specifh data. Un- cleaned data may contain harmful information such biases and stereotypes (Sun et al. 2024) Recent research suggests that training predictors with extracted rationales can remove irrelevan harmful information, enhancing robustness (Chen et al. 2022) and generalization (Wu et al. 2022 Gui et al. 2023). Considering that small models are sufficient for simple supervised tasks and are more flexible and cost-effective for training on single datasets (e.g., searching hyperparameters and adding auxiliary regularizers), using small models for rationalization on a single dataset and then using the extracted rationales for supervised fine-tuning might prevent large models from learning harmful information from new data. Additionally, shorteming input texts can also reduce the memory required for fine-tuning (Guan al.) 2022). recent study also finds that training small model for data selection (although not the same as rationale selection) and producing a small subset is useful for fine-tur tinettnnig L.MS (Xia al., (024).
We compare our method against a representative LLM (Ilama-3.1 1aama-3.1-8b-nnee 1---b--nssttuet), in Appendix A.7 and demonstrate that our approach achieves comparable results, sometimes even surpassing it.
PRELIMINARIES
3.1 THE RATIONALE EXTRACTION TASK
We consider the text classification task, where the input is text sequence X=[#1, 22, æv] with ar being the i-th token and being the number of tokens. Y represents the classes dataset D. The standard rationalization framework RNP LLi et al. 2016) consists of extractor fE(+) and a predictor fP)),, with 0g and 8p representing the parameters of the extractor and predictor. For (X,Y)- D. the extractor first outputs sequence of binary mask M fx( (0,1) (in practice, the extractor first outputs a Bernoulli distribution for each token and the mask for each token is independently sampled using gumbel- Then, it forms the rationale candidate Z by the element- wise product of X and M:
(', mm]]
Z= M o X (mi01,"., . "mm
To simplify the notation, we denote fx( Z in the following sections, i.e., fx( Z. With the extractor's ' selection, we get set of (Z,Y) samples, which are generally considered to represent the distribution P(Y The rationale Z is searched by maximizing the mutual information I(Y;))
Z' = arg max I(Y;Z) = arg max H(Y) - -H(12)) = arg min H(YIZ), s.l., Z fx(X).
2
z
(2)
In practice, the entropy H (Y is commonly approximated by the minimum cross-entropy ming, HY, with = fr(Z) representing the output of the predictor. It is essential to note that the minimum cross-entropy is equal to the entropy (please refer to Appendix A.5)
4

---

Published as a conference paper at ICLR 2025
As result, the predictor uses the eross-entropy objective to do the classification, and the extractor also uses the cross-entropy objective to find good rationales:
Extractor: min H.(Y,5p(2)22
8. Predictor: min (=(()5((2)1z))
Op
s.l., Z JE(X), (X,Y)~D)
Replacing Z with fx(X), the extractor and the predictor are trained cooperatively to minimize the eross-entropy. Here rewrite Equation (3) for better conciseness and clarity:
min Ho(Y, (.C.,IP(T((X))(x((()) s.l., (X,Y)-D. 0.00p
(4)
To make the selected rationale human-intelligible, rationalization methods usually constrain the rationales by compact and coherent regularization terms. In this paper, we use the most widely used constraints proposed by Chang et al. (2020):
M|
?(M) A
A2
m, ma-1/
(5)
The first term encourages that the percentage of the tokens being selected as rationales is close to a pre-defined level s. The second term encourages the rationales to be coherent.
3.2 THE PROPERTIES OF THE NETWORK'S UTILIZATION ON DIFFERENT INPUTS
It has been found that neural networks usually have low-rank weight matrices (Kang et al. 2024 For these low-rank weight matrices, they correspond to low-dimensional subspaces. And the loca- tion and magnitude of these subspaces are determined by the information they learn (Kang et al.) 2024]. If network just learns (or is trained on) some uninformative noise, the rank will be low and the corresponding subspace will be narrow. On the contrary, if the network learns from informative knowledge, the subspace will be wider. From a localization perspective, the learned inputs often occupy these low-dimensional subspaces with high overlap. And the unlearned inputs tend to have little overlap. As result, if an input is informative and the knowledge is learned by the network, the representation of it through the network will have high l2 norm. If an input does not contain the knowledge learned by the network, the norm will approach 0 (Kang et al. 2024).
Simply put, the rank of the weight matrix is determined by the knowledge learned by the network and is reflected in the directions on the hypersphere that can be occupied by the combination of the column vectors (including position and size). These areas occupy subspace in the high-dimensional space, which we refer to as the capability subspace of the weight matrix. If an input contains features that the network has learned, learned directions, leading to representation with higher norm. On the other hand, if an input does not contain the learned information, it is likely to fall outside the capability subspace and be orthogonal to the column vectors (sce Appendix 14), thus its representation norm will be very low, behaving like noise. At the same time, the network does not learn any informative knowledge, the capability subspace of the weight matrix itself will be very low and all inputs will have low norms.
highly likely to fall within the capability subspace and match the
This property was first used by Kang et al. (024) to remove duitt--distriuution inputs reinforce- ment learning. But we think it also be used for interpretability. By observing the norms of the representations of different rationale candidates, we can determine how well they match the network, and thus identify which parts of the full input the network is actually utilizing.
4 THE LIMITATIONS OF MINIMIZING CROSS-ENTROPY AND OUR METHOD
4.1 THE DIMINISHING MARGINAL RETURNS
Although using MMI to identify rationales has almost become the default choice, we find that it faces the problem of diminishing marginal return.. Once the majority of the rationale components have
5

---

Published as a conference paper at ICLR 2025
0.00
acc
Closs ent/009
norm
0.0-
0.72
0.70
0.70
0.6-
3.2
0.03
0.0
055-
0.00
5.2 0.6
U. The rabio mertona
058
0.2
0.4
0.8
02 04 The rationale
1.0
The rationale
(b)
(c)
Figure 3: The (a) prediction accuracy, (b) cross-entropy loss, and (c) the norm of the representation (i.e., ||Enc(( 2| through the neural network vary with the proportion of true rationale components in the rationale candidate input within trained standard RNP predictor. The dataset is Beer-Aroma. The results of more datasets shown in Appendix A.8
been identified, discovering the left rationale components has minimal effect on further reducing cross-entropy.
In this section, first provide an intuitive toy example help readers understand the diminishing marginal returns from a high-level intuition. Then, we show the origin of the diminishing marginal returns problem. Finally, we provide empirical evidence from real-world datasets to verify the exis- tence of this problem in practice.
An intuitive toy example. Consider comment about food "...... The food is very delicious, and I like it very much. ....." and we need to predict its sentiment label. Both Rı ="The food is very delicious" and R₂ ="and I like very much" can indicate a positive enough sentiment tendency. However, if either R₁ or R2 is given, finding another component will not contribute much to the sentiment polarity- This creates an obstacle to finding the complete rationale by MMI (minimizing the cross-entropy in practice). A more visual example is provided in Figure The distance between Rí and R& &22 is non-trivial, so the extractor needs to pay considerable effort to move from the state of selecting Ri to the state of selecting both Rt and Rg. However, the payoff' for this effort is minimal, resulting in small gradients provided by the gradient descent algorithm for this move.
The reasons of this problem lie in two aspects. One is the gradient saturation problem of the sigmoid function before the predictor's output V. which quite intuitive and can be illustrated by the example in Figure 2 (softmax is similar). Aside from the sigmoid function, the problem can also be introduced by the diminishing marginal returns of mutual information itself.
The theoretical perspective. The high-level understand- ing of mutual information T(Y; Z) is that, how much the uncertainty of decreases when given Z.
f,IR) f(8,882)
We consider perfect rationale R composed of R, and Rg. The problem is that MMI does not satisfy additiv- ity, which means that we cannot promise [(Y; Ri R2) I(Y; R1) I(Y: R2). The combined effect of R₁ and R2 on reducing the uncertainty of Y may be less than the
Fjhoise))
sum of the individual effects of Rú and R2 on reducing the uncertainty of Y. Formally, we consider the situations where
100000
0g
Rg5Rg
((Y; R1, R2) (YY; R1) I(Y;
(6)
Figure 2: The diminishing marginal re- turns Sigmoid function.
Please refer to Appendix A.6 for more detailed discus- sions about Equation (6).
From Equation
6), we can further get
T(Y; R2 R1) = T(Y; R2, ((Y; I(Y; RY)
[[(Y; II);;R)))( [oY;
= R2).

---

Published as a conference paper at ICLR 2025
It means that, although Rg is informative and can reduce the uncertainty of it can be less effective conditioned on another informative part R1.
Empirical verification of the diminishing marginal returns. Figure 3 shows how the prediction accuracy, cross-entropy loss, and the norm of the representation (i.c., ||Ene{ 2)1l2 in Figure I vary with the proportion of true rationale components in the rationale candidate input within trained standard RNP predictor. The dataset is a text classification dataset Beer-Aroma. The results of more datasets are shown in Appendix A.8
The test set contains human-annotated ground-truth rationales. We base our inputs on the manually annotated ground-truth gold rationales in the original full text. A certain proportion of tokens in the gold rationales are replaced with tokens randomly selected from the non-rationale part of the same full text. The x-axis represents the proportion of tokens that have not been replaced, where O indicates all tokens are random, and I indicates the complete ground-truth rationale. It can be observed that when the rationale candidate used as input contains more than 60% true rationale components, the decrease in cross-entropy loss (Figure 3b)) and the increase in accuracy (Figure 3(a) slow down. Subsequent addition of new rationale components has diminished effect on loss reduction, making it difficult to identify the left rationale components.
This raises a question: is there an objective function that can indicate rationale, and does not involve either the mutual information or the sigmoid function of the neural network's 's output layer, thereby allowing for the possibility of avoiding diminishing marginal returns?
Empirical observation on the norm of intermediate representation. From the theoretical prop- erty mentioned in $3.2 we know that the representation' 's norm is an indicator for the degree of the network's 's utilization moid function. We then empirically observe whether it faces the problem of diminishing marginal returns. Figure (c) shows how the norm varies with the proportion of true rationale components in the rationale candidate input within a trained standard RNP predictor. We see that it indeed does not face the problem of diminishing marginal returns. Instead, grows even faster as the proportion of rationale components grows.
input. Besides, it does not involve either mutual information the sig-
4.2 THE PRACTICAL METHOD
Unlike existing mainstream rationalization methods that add extra auxiliary modules, we do not change the model architecture of the vanilla RNP, thus preserving its conciseness and flexibility. The only needed modification is to replace the extractor's 's cross-entropy loss with the norm loss.
Compared to MMIIbased methods (Equation 3). we remove the cross-entropy loss from the extrac- tor's parameters 6g and replace it with the norm:
Extractor: min- (onn-loo(]En()))ll
(8)
9. Predictor: min H. (Y,
(9)
0p
s.t., Z - fx(X), (X,Y)
D. ~D.
(10)
The predictor is trained do the classification, and the extractor is trained identify the rationales.
During training, (9) and (8) are alternated. The practical implementation with Pytorch is in Appendix
A.3 We call this method N2R (norm rationale). Note that when we say MMI (see Equation (2), it refers only to the objective of the extractor, and does not include the predictor. No matter how the extractor's objective changes, the predictor is always trained with the cross-entropy as it needs to do the classification.
To further show the potential and scalability of our N2R, we also verify the possibility of combining MMI and N2R (corresponding to the supplementary experiments in A.. This combination is inspired by the empirical results of Figure 3 In the initial stage of training, the extractor has not yet identified enough true rationale components, leading to small gradients provided by the norm objective (Figure 3 c)), which is not efficient enough to guide the extractor in finding the rationale. However, at this stage, the MMI objective provides larger gradients
B()).. In the later stages of training, the situation is reversed. Overall, the MMI and norm objectives complement each other, resulting in improved performance. We maintain the objective of the vanilla RNP (Equation 3) and

---

Published as a conference paper at ICLR 2025
Table 1: Results datasets from the BeerAdvocate Benchmark. We report the average results of
five random seeds. Values in "0" are the standard deviations. InterRRAT Yue et al. (2023). NIR:
Storek et al. 2023
A2I: Liu (022c).
Datasets
Beer-Appeannce
Bear-Arema
Methods
S
Acc
P
F1
S
Acc
P
FI
RNP
14.7 (0.7)
(3.3)
(0.5
59.7 (3.1) (2.)
81.7 (1.2
(1.0
64.7 (8.8)
(10.4)
Inter RAT
(1.1
NAA
(5.3)
46.9 (2.3)
512
(6.1
NAA
(2.5 (244)
(2.1)
15.4
A2I N2R (ours)
(0.3
(1.0
75.2 (0.)
60.6
148 61.1
82.7 (2.3)
69.4 (2.5)
(2.9
(2.6
148 (0.5) (1.8
81.9
(2.)
(2.)
(.4) (4.9
67.2(1.3) (1.)
68.7
add the norm to the extractor's objective:
Extractor: min|
(11)
8,
Predictor: min
(12)
8p.
s.t., Z fx(X), (X,Y) D.
(13)
We call this method MMI+N2R. The practical implementation with Pytorch is in Appendix A.4
5 EXPERIMENTS
5.1 SETTINGS
Baselines. The main baseline for direct comparison the vanilla MMI-based rationalization frame- work RNP (Lei al. 2016), as RNP and our N2R match in selection granularity, optimization algorithm, and model architecture, which helps us to focus on our claims rather than some poten- tially unknown mechanisms. To show the competitiveness of our method, we also include several recently published methods that improve MMI with various regularizers: Inter RAT (Yue et al., 2023). NIR (Storek al. 2023). CR (Zhang et 2023), and A2I (Liu al. 2024c). all of which have been discussed in 22
Datasets. Although the rationale extraction process is unsupervised, the rationalization task requires comparing the rationale quality extracted by different models. This necessitates that the test set includes ground-truth rationales, which imposes special requirements on the datasets. Following the conventional setup in the field of rationalization, widely used benchmarks. Apart from the text data, we also include a graph classification dataset.
employ four text classification datasets from two
The text classification datasets are Beer-Appearance Beer-Aroma (collected from the BeerAd- vocate benchmark (McAuley et al. 2012)). Hotel-Service, Hotel-Cleanliness (collected from the HotelReviews benehmark (Wang al. 20100). We also use graph classification dataset, called BA2Motifs (Ying et al. (2019) to verify generalizability. All of these datasets contain human- annotated ground-truth rationales the set, making it convenient to compare different methods' performance fairly. More details are in Appendix A.I
Implementation details. Both the extractor and the predictor are composed of an encoder (NNWTrannffrmer//GCNN and linear layer. We use three types of encoders: GRUs (following Inter RAT and A2I, table and 2). bert-base-uncased (following CR, table 3 and GCN (for the BA2Motifs dataset). For NIR and our N2R, considering they are both variants of the standard RNP, we first manually tune the hyperparameters for RNP. and then apply the hyperparameters NIR and N2R. For Inter RAT, since it has originally been implemented on the beer-related datasets, we apply its original hyperparameters but only adjust the sparsity regularizer in Equation (5). For CR. we just keep the major settings peeet-bas-unnaseee asity 10%) the same as and copy its results from its original paper. We report the average results of five random seeds. More details are in Appendix A.2
both
the Beer-Appearance dataset, and the spr-
Metrics. Following the previous research of Inter_RAT and A2I, we mainly focus on the rationale quality, which is measured by the overlap between the annatttnd rationales and the model- selected tokens. The terms P. R, F1 denote precision, recall, and Fl score respectively. These metrics the most frequently used in rationalization. The term S represents the average sparsity of
8

---

Published as a conference paper at ICLR 2025
Table 2: Results datasets from the HotelReviews Benchmark.
Datasets
Hotel-Service
Hotel-Cleanlines
Methods
RNP
(0.3
(1.5)
41.0(1.))
54,6(1.1
468(144
(..3
(7.2
28.1
48.7 41.7(1.))
(0.6
(1.9
38.1 (1.9) 328(1.))
(1.1
NIA
44.1 (2.2(2.1)
(2.4)
(..)
NIR
(5.0
(0.9
53.5 (1.)
(6.3(14)
(0.4)
96.7 (0.9)
(8.0
(9.2 (0.6)
A2I
(04)
96.7
(1.4
(1.6
47.1
15.3 (0.3)
(1.0
49.7 28.8
(0.7)
N2R (ours)
97.4 (0.2)
(0.5)
42.8 (0.5)
(1.0
48.6 (0.6
14.8 (0.2)
97.4 (0.4)
(..)
53.4 (0.))
39.8 (0.6)
Table 3: Results with BERT encoder. " * P: the results of baselines are obtained from the paper of CR (Zhang et al. 2023).
Datasets
Beer-Appearance
Becr-Aroma
Methods
Acc
Acc
RNP
(n/a)
(1.7) (1.0(1.4) 20.3
25.2
10.0 (a/g)
49.1 (2.0
22.7
(2.5
A2R
10.0 (p/a)
91.5 (2.2) (0.0
(1.
34.4
(0/a
81.((19)
61.3 (2.8
(3.8
(1.2
INVRAT
10.0 (n/a) (3.1 (2.5
(1.3
(2.)
(n.0
(0.0
(3.1)
27.5
33.2
(m/a)
(1.
31.(1.)
39.
(2.9)
42.0
49.1 (2.8)
N2R (ours)
10.8 (0.3) 93.5 (1.)
79.7
(1.)
(2.5
10.0
91.0
74.3
57.6 (3.7
the selected rationales, that is, the percentage of selected tokens in relation to the full text. Since the sparsity of ground-truth rationales on these datasets is around 10% 20%, we adjust in Equation (5) to make S about 15% (since Equation (5) is only a soft constraint, cannot strictly limit S to be exactly 15%.). Ace stands for the predictive accuracy.
5.2 RESULTS
Results on standard benchmarks. Tables and 2 show the results on the four text classification datasets. In terms of the rationale quality (F1 score), our N2R significantly outperforms the standard MMIIbased method (i.e., RNP) and also beats its improved variants. Compared to the second- best results of previous methods, the relevant improvements of our N2R on these four datasets are 8.3% (= 67.1
22.7-67.1) 1.6% (= 68.7-67.6 5.1% (= 49.6-47.1), and 9.0% (= 39.8-36.6, showing the
68.5
47.1
36.5
competitiveness of replacing the MMI-based objective with our norm objective.
We also compare with a representative LLM, 1lama-3.1-8h 1-8b-insttruct, Table Appendix A.7 and find that our simple N2R gets comparable results and can sometimes
outperform it.
Results with BERT encoder. We also follow CR to conduct experiments with pretrained bert- base-uncased as supplement. Since some methods become highly sensitive to hyperparameters after switching to an over wver-parameterized BERT model (also supported by Remark 6.1 in (Zhang eet al.) 2023)), and our computational resources insufficient for extensive hyperparameter tun- ing for these methods, we primarily compare our approach with methods that have already been implemented using BERT. The results shown in Table 3 Our N2R still outperforms previous MMI-based methods significantly.
Results with GCN encoder. To show generalizability of our method, we expand the RNP frame- work to graph neural networks to conduct a supplement experiment. Since Inter RAT, NIR, and CR are methods specifically designed for text data and not suitable for graph tasks, we only compare our N2R with the standard RNP on the BA2Motifs dataset to show its effectiveness rather than competitiveness. For this dataset, we select set of nodes for each graph as the rationale. The results are shown in Table 4 We see that our method is still effective when applied to graph neural networks. Note that our method is very simple and has the potential be combined with more advanced methods in the future. However, since the interpretability of graph neural networks is not the focus of this paper, we leave for future work.
N2R can be further improved when combined with MMI. To further verify the scalability of our N2R, we implement a variant it by combining N2R and MMI criterion together, which is introduced at the end of $4 The comparison between vanilla MMI, N2R, and N2R+MMI on the datasets from the BeerAdvocate benchmark and HotelReviews benchmark are shown in Figure ((a) and Figure 4(b) respectively.

---

Published as a conference paper at ICLR 2025
Table 4: Results with GCN encoder BA2Motifs.
Methods
P
F1
RNP
(2.3) (1.9)
(5.5
36.5 (2.2)
(3.4
N2R
20.1 (1.2) 96.0 (1.9)
40.1 (2.1) 40.4 (4.4)
40.2 (3.2)
85
MMI (RMP))
50-
49.5
48,6
MMI IRNP)
45,8
MM N2I
75,2
8
75
45
71,3
1 70
66.5
68.7
40
40.4
39.8
35.6
60
GeenAApeerannce
Beer-Aroma
Hotel-Service
Hotel Citan-Cleaniieess
(b)
Figure 4: The comparison between vanilla MMI, N2R, and N2R+MMI the datasets from (a) BeerAdvocate benchmark and (b) HotelReviews benchmark.
We observe that, N2R its already significantly outperforms MMI, yet when combined with MMI, its performance improves even further. As compared to the vanilla MMI, the the relevant improvements of MMI+N2R these four datasets are 13.1%(( 75.2-66.5 5.8%(=
8.4%( 71.3-65.8 65.8
66.5
46.8 49.5-46.8 ). and (= 40.4-35.6
35.6 respectively. This improvement of MMI+N2R aligns
with the empirical results shown in Figure In the initial stage of training, the extractor has not yet identified enough true rationale components, leading to small gradients provided by the OOD objective (Figure
which is insufficient effectively guide the extractor in finding the rationale. However, at this stage, the MMI objective provides larger gradients (Figure ())) In the later stages of training, the situation is reversed. Overall, the MMI and OOD objectives complement each other, resulting in enhanced performance.
CONCLUSION AND FUTURE WORK
In this paper, we first analyze the diminishing marginal return limitation of the fundamental MMI- based objective in the XAI literature. Then, we propose use the norm of the intermediate repre- sentation of rationale candidates to replace the MMI objective, which inspired by OOD detection techniques. Our OOD-inspired objective not only outperforms the vanilla MMI, but also beats sev- eral recent variants. What's ' more, it can easily combined with MMI, further validating its seala- bility and potential. Our work represents pioncering attempt to bridge the fields of OOD detection and interpretability. This could potentially inspire researchers in the OOD field to adapt more OOD detection techniques to the XAI domain, thereby further advancing the development of XAI.
Most existing methods find explanations by reconstructing the model's final output, whereas we focus identifying which parts of the input are utilized by the network during forward propagation. This is new direction that can free explanation algorithms from relying on the model's final output. This property could have broader implications, such as potentially being used in the future to explain lask-agnostic (i.e., not fine-tuned) pretrained encoders.
ACKNOWLEDGMENT
This work is supported by the National Key Research and Development Program of China under grant 2024YFC3307900; the National Natural Science Foundation of China under grants 62376103. 62302184, 62436003 and 62206102; Major Science and Technology Project of Hubei Province under grant 2024BAA008; Hubei Science and Technology Talent Service Project under grant 2024DJC078; and Ant Group through CCF-Ant Research Fund. The computation is completed in the HPC Platform of Huazhong University of Science and Technology
10

---

Published as a conference paper at ICLR 2025
We sincerely thank the reviewers and the editor for their valuable feedback and efforts during the review process. They have helped a lot in improving the quality of this paper. We thank Lao Gao and Yuankai Zhang for providing the results on LLMs and image data.
REFERENCES
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume I: Long Papers). Virtual Event, August 1-6, 2021, pp. 7319-7328. Association for Computational L Linguistics, 2021. doi: 1001165511122211ALLLLLL0
URL
https://6ol.orr/100..
acl oog. 568
Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha S. Srinivasa. Git re-basin: Merging mod- els modulo permutation symmetries. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openrewt
net
Diego Antognini, Claudiu Musat, and Boi Faltings. Multi-dimensional explanation of target vari- ables from documents. In ThitttyFFfth AAAI Conference on Artificial Intelligence, AAAI 202 I, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 12507-12515. AAAI Press, 2021. doi: 10. 10.1609/aami.v35111 17483.
/tt.:ooo.ooggoooo 10. 1609 aaai. 135114.17483
Yujia Bao, Shiyu Chang, Mo Yu, and Regina Barzilay. Deriving machine attention from human rationales. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pp. 1903-19133 Association for Computational Linguistics, 2018. doi: 10.18653/11/118-11
URL http::://ddoooooo ///g:///...//.......ooo org/10.
18653/v1/d18 1216,
Jasmijn Bastings, Wilker Aziz, and Ivan Titov. Interpretable neural predictions with differentiable binary variables. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp- 2963-2977, Association for Computational Linguisties, 2019. doi: 10. 1/29911/1 URL https://doi.org/11 1tt.://.....133//1pp/11ll 1284
Tolga Bolukbasi, Adam Pearce, Ann Yuan, Andy Coenen, Emily Reif, Fernanda B. Viégas, and Martin Wattenberg. An interpretability illusion for BERT. CoRR, abs/2104.07143, 2021. URL
https::/aarxt http/rxxiv.rg//aa/.. atsh///s///// 07143
Tony Cai, Jianqing Fan, and Tiefeng Jiang. Distributions of angles in random packing on spheres.
Journal of Machine Learning Research, 14(136): (036):1837-111 837-1864, 2013.
Aaron Chan, Maziar Sanjabi, Lambert Mathias, Liang Tan, Shaoliang Nie, Xiaochang Peng, Xiang Ren, and Hamed Firooz. UNIREX: unified learning framework for language model rationale extraction. In International Conference Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 Proceedings of Machine Learning Research, pp. 2867- 2889. PMLR, 2022. http://ttt::ppoeeeeeeaee
mlr www.sprsssvll.... ch/n22a html
Shiyu Chang, Yang Zhang, Mo Yu, and Tommi S. Jaakkola. Invariant rationalization. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 1448-1458. PMLR, 2020. URL
mlr. htt.:/eeiiinnnm.....rrpssees/ssa httpss///v11//ccch22....t. html
Howard Chen, Jacqueline He, Karthik Narasimhan, and Danqi Chen. Can rationalization improve robustness? In Proceedings of the 2022 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, NAACL 2022, Seatlle, WA, United States, July 10-15, 2022, pp. 3792-3805. Association for Computational Linguisties, 2022. doi: 10.18653/v1/20222 naacl- 278. URL https: /ttpp:///do.. www.oo.gg/o.... 2022. naac -ma i 278

---

Published as a conference paper at ICLR 2025
Jay De Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. ERASER: A benchmark to evaluate rationalized NLP models. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 4443-4458, Online, July 2020. m08. https:::
Association for Computational Linguistics. doi: 10.
//aclanthology wwwp://oooo...g./2..ooooo
wwwp//gg.oooooo.oo acl main.
Patrick Fernandes, Marcos Treviso, Danish Pruthi, André Martins, and Graham Neubig. Learn- ing to scaffold: Optimizing model explanations for teaching. Advances in Neural Information 2022.
Processing Systems,
Marzyeh Ghassemi, Luke Oakden-Rayner, and Andrew L Beam. The false hope of current ap- proaches explainable artificial intelligence health care. The Lancet Digital Health, 3(11): c745-e7500 2021.
Yue Guan, Zhengyi Li, Jingwen Leng, Zhouhan Lin, and Minyi Guo. Transkimmer: Transformer learns layer-wis, wise skim. Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 7275-7286. Association for Computational I Linguistics, 2022. doi: 10. AC- 1ee.l.ogg/00.1833311.. act.://222.........aa...aa
URL https:
502
Shurui Gui, Meng Liu, Xiner Li, Youzhi Luo, and Shuiwang Joint learning of label and environment causal independence for graph out-of-distribution generalization. arXiv preprint arXiv:2306.01100 2023.
Peter Hase, Shiyue Zhang, Harry Xie, and Mohit Bansal. Leakage-adjusted simulatability: Can models generate non-trivial explanations of their behavior in natural language? In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4351-433660 2020.
Serhii Havrylov, Germán Kruszewski, and Armand Joulin. Cooperative learning of disjoint syntax and semantics. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 1118-1128. Association for Computational Linguistics, 2019. doi: 10.18653/11/19--111 URL https:
7/doi. org. 10. 18..186656 18653/666111 1115
Yongfeng Huang, Yujun Chen, Yulun Du, and Zhilin Yang. Distribution matching for rationalization. In Thirty-Fifth AAAI Conference on Artificial intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Ed- ucational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 13090-13097. AAAI Press, 2021. URL https: /ojs aaai www.oorgngnns.ee..eoeeooe..e php/ AAAI/
articles article/vii ew 17547
Sarthak Jain, Sarah Wiegreffe. Yuval Pinter, and Byron C. Wallace. Learning to faithfully ra- tionalize by construction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, ACL 2020, Online, July 5-10, 2020, pp. 1459-4473. Association for Computational Linguistics, 2020. doi: 20.18653/v120220.cr m09. URL hetps:
7/doi aeoi.oog//00 10. 10.00.8665 (1888663//2000 acl...ccl-1 aww.aaalmmaa......... main. 409
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1 2023.
Katie Kang, Amrith Setlur, Claire Tomlin, and Sergey Levine. Deep neural networks tend extrap- olate predictably. In The Twelfth International Conference on Learning Representations, 2024.
Emre Kıcıman, Robert Ness, Amit Sharma, and Chenhao Tan. Causal reasoning and large language models: Opening new frontier for causality. arXiv preprint a305v: 230500000, 2023.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. /tt.ttpp://arttt.orrr/b////////
6980
12

---

Published as a conference paper at ICLR 2025
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catan- zaro, and Wei Ping. Nv-embed: Improved techniques for training Ilms as generalist em- bedding models. CoRR, abs/2405.174228, 2024. doi: 10.48500AARXIV.245..1445 URL
htttps://do. wtt.dii.rrg/ii/ www.orr...... 48550/ / 2405. 17428
Tao Lei, Regina Barzilay, and Tommi S. Jaakkola. Rationalizing neural predictions. In Proceed- ings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pp. 107-117. The Association for Computational Linguisties, 2016. doi: 10. 10.18533/1/d11--0111 d1 416-1011
URL
Bo Li, Gexiang Fang, Yang Yang, Quansen Wang, Wei Ye, Wen Zhao, and Shikun Zhang. Evaluat- ing chatgpt's information extraction capabilities: An assessment of performance, explainability, calibration, and faithfulness. arXiv preprint arXiv:2304. 2023.
Yiming Li, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor leamnigg A survey. IEEE Transac- tions on Neural Networks and Learning Systems, 2022.
Zachary C Lipton. The mythos of model interpretability: In machine learning, the concept of inter- pretability is both important and slippery. Queue, 16(3):31-57,20 2018.
Wei Liu, Haozhao Wang, Jun Wang, Ruixuan Li, Chao Yue, and YuanKai Zhang. FR: Folded rationalization with a unified encoder. Advances in Neural Information Processing Systems, 2022. TRLLttps:: //tp::////ppeeeeeeeeeee net fttp//oormmmimiee wtt.oummmm.22yysaakk...
Wei Liu, Haozhao Wang, Jun Wang, Ruixuan Li, Xinyang Li, Yuankai Zhang, and Yang Qiu. MGR: multi-generator based rationalization. Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics /Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 20233 pp. 12771- 12787. Association for Computational Linguisties, 2023a. doi: 1118853/1122233.ccl/ URL
Anna Rogers, Jordan L. Boyd-Graber, and Naoaki
aww.0022..11oooooo.. 715
Wei Liu, Jun Wang, Haozhao Wang, Ruixuan Li, Zhiying Deng, Yuankai Zhang, and Yang Qiu. D-separation for causal self-explanation. In Advances Neural In- formation Processing Systems 36: Annual Conference on Neural Information Pro- cessing Systems 2023, NeurlPS 2023, New Orleans, IA, USA, December 10 16, 2023, 2023b. URL http:/ /papers nips. cc/paper Files/paper//2/ /hashh/ 01836678806655522399222222222283 ot
Abstract Conference html
Wei Liu, Jun Wang, Haozhao Wang, Ruixuan Li, Yang Qiu, Yuankai Zhang, Jie Han, and Yixiong Zou. Decoupled rationalization with asymmetric learning rates: flexible lipschitz restraint. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2023, Long Beach, CA, USA, August 6 60, 2023, pp. 1535-1547. ACM, 2023c. doi: 10. 1145/3580305.3599999, URL hitps: 2o1 org 1t1.//00111111333000033303553 3599299
Wei Liu, Zhiying Deng, Zhongyu Niu, Jun Wang. Haozhao Wang, YuanKai Zhang, and Ruixuan Li. Is the mmi criterion necessary for interpretability? degenerating non-causal features to plain noise for ratiiaatiinaliatinnn. In The Thirty-eight Anmual Conference on Neural Information Processing Systems, 2024a.
Wei Liu, Haozhao Wang, Jun Wang. Zhiying Deng, Yuankai Zhang, Cheng Wang, and Ruixuan Li. Enhancing the rationale-input alignment for self-explaining rationalization. In 40th IEEE International Conference on Data Engineering, ICDE 2024, Utrecht, The Netherlands, May 13- URL https:
16, 2024, pp. 2218 2230. IEEE 2024b. doi: //doi.org//
Wei Liu, Jun Wang, Haozhao Wang, Ruixuan Li. Zhiying Deng, and YuanKai Zhang. Attacking for inspection and instruction: Debiasing self-explaining text classification, 2024c. URL https: //openrevi ew. hett/forrmm.ddsssdoUuoaaaa
13

---

Published as a conference paper at ICLR 2025
Dongsheng Luo, Wei Cheng. Dongkuan Xu, Wenchao Yu, Bo Zong, Haifeng Chen, and Xiang Zhang. Parameterized explainer for graph neural network. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neu- ral Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, vir-
tual, 2020. URL https::/pproceed: neur ips cc//paperr2222/haat)
e 3TD083331533bbb 7b858 Abstract .attraactttmmml
Julian J. McAuley, Jure Leskovec, and Dan Jurafsky. Learning attitudes and attributes from multi- aspect reviews. In 12th IEEE International Conference on Data Mining. ICDM 2012, Brussels, Belgium, December 10-13, 2012, pp. 1020-1025, IEEE Computer Society, 2012. doi: 10.1109/ ICDM.2002.110 URL /ttpp////////o..../.........../.
wttp:///....11//....... 2012. 110,
Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summa- rization based on minimum cuts. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACI-044), pp- 271-278, 2004.
Bhargavi Paranjape, Mandar Joshi, John Thickstun, Hannaneh Hajishirzi, and Luke Zettlemoyer. An information bottleneck approach for controlling conciseness in rationale extraction. In Proceed- ings of the 2020 Conference on Empiricul Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 1938-1953 Association for Computational Linguis- www.eeennamaaaeee wwi.. 153, URL https:: oro.o...r/10...8555 v1 01/2020.ro emn www..mii....1.....
tics, 2020, doi: 10. 10.118633/1120000
Mitchell Plyler, Michael Green, and Min Chi. Making (counterfactual) difference one rationale at a time. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14. 2021, virtual, pp. 28701-28713, 2021. URL.
ine htmll
hash/ Mash/FOf800eee Fw0F0000000021110 91
Romila Pradhan, Jiongli Zhu, Boris Glavic, and Babak Salimi. Interpretable data-based explanations for fairness debugging. In SIGMOD '22: International Conference on Management of Data, Philadelphia, PA, USA, June 12 17, 2022, pp. 261. ACM, 2022. doi: 10.1145/3514221.
3517886. URL httpt: /ttp:::///dooooooo wwwpor/g..oooooooooooo//...//ooe 31112/55511222..33
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang- Is chatgpt general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476, 2023.
Dheeraj Rajagopal, Vidhisha Balachandran, Eduard H Hovy, and Yulia Tsvetkov. SELFEXPLAIN: A self-explaining architecture for neural text classifiers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pp. 836-856 850, Online and Punta Cana, Dominican Republic, November 2021, Association for Computational Linguisties. doi: 10. 1865// v1/2021 www.llmmmnppan....m URL https: //acl hn I ogy. org/ 2021 emm 1ommmlpmmt p0 nn. 64
Qihan Ren, Jiayang Gao, Wen Shen, and Quanshi Zhang. Where we have arrived in proving the emergence of sparse interaction primitives DNNs. The Twelfth International Confer- ence on Learning Representations, 2024. URL hnttps://openreview wet.eeewweweeeeeeeeeeeeeeeeeeeee feee//forumiii
33WSLBMy6B 6B
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 105):206-215, 2019. doi: 10. 1038/542256-019-0048.. X.
1038/ 0111////222225011-0025--03
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. arXiv preprint arXiv: 191 1.08731,,, 2019.
Benjamin B Seiler. Applications of Cooperative Game Theory to Interpretable Machine Learning. PhD thesis, Stanford University, 2023.
Lei Sha, Oana-Maria Camburu, and Thomas Lukasiewicz. Learning from the best: Rationalizing predictions by adversarial information calibration. In Thirty- Fifth AAAI Conference on Artifi- cial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial
14

---

Published as a conference paper at ICLR 2025
Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelli- gence, EAA/ 2021, Virtual Event, February 2-9, 2021, pp. 13771-13779. AAAI Press, 2021. URL httpss:/ojs aaai. org/ index. php/ AAAI A.AI/artticll//vew//111...
Adam Storek, Melanie Subbiah, and Kathleen R. McKeown. Unsupervised selective rationalization with noise injection. In Proceedings of the 61st Annual Meeting of the Association for Compu- tational Linguristies (Volume Long Papers), ACL 2023, Toronto, Canada. July 9-14. 2023, pp. 12647-12659) Association for Computational Linguisties, 2023. doi: 1o.188631112222..ul 707. URL. htt.:htttt////tt http://.....rr/oooooo orgg10 18653 201655/v12220 /t////1//2222 acl-ol2n..clolnno 707
Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wen- han Lyu, Yixuan Zhang, Xiner Li, Zhengliang Liu, Yixin Liu, Yijue Wang, Zhikum Zhang, Bhavya Kailkhura, Caiming Xiong, Chao Zhang, Chaowei Xiao, Chumyuan Li, Eric P. Xing, Furong Huang. Hao Liu, Heng Ji, Hongyi Wang, Huan Zhang, Huaxiu Yao, Manolis Kellis, Marinka Zitnik, Meng Jiang, Mohit Bansal, James Zou, Jian Pei, Jian Liu, Jianfeng Gao, Jiawei Han, Jieyu Zhao, Jiliing Tang, Jindong Wang, John Mitchell, Kai Shu, Kaidi Xu, Kai-Wei Chang, Lifang He, Lifu Huang, Michael Backes, Neil Zhenqiang Gong, Philip S. Yu, Pin-Yu Chen, Quanquan Gu, Ran Xu, Rex Ying, Shuiwang Ji, Suman Jana, Tianlong Chen, Tianming Liu, Tianyi Zhou, William Wang, Xiang Li, Xiangliang Zhang, Xiao Wang, Xing Xie, Xun Chen, Xuyu Wang. Yan Liu, Yanfang Ye, Yinzhi Cao, and Yue Zhao Trustllm: Trustworthiness in large language models. CoRR, abs/2401.05555 2024, doi: 1.48550/ARXIV2240.055555 URL https: / //doi. wtw./0../00....... 48550/ 20.50//rXivi 2401 05561
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Hongning Wang, Yue Lu, and Chengxiang Zhai. Latent aspect rating analysis review text data:
rating regression approach. In Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, July 25-28, 2010, pp. 783- doi: 10.1 00.1145/1835804.1835000
792. ACM, 2010.
URL https: /tt. org 10. 1r./10.1155// 1145/
1835804 1835903
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le. and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurlPS, 2022. URL http:://pppprrree nips ct/ wttppppeeppppelepeppegaaaeeeee
2022 httphh/h/cccc500155222eeeeaaaaaa otg.aaaaaaaaaaaaaaaooaoaaaaaaaaa www.aaaaaaaaabtaatttooooeeeeeee html
Yingxin Wu, Xiang Wang, An Zhang, Xiangnan He, and Tat-Seng Chua. Discovering invari- ant rationales for graph neural networks. In The Tenth International Conference on Learning Representations, ICLR 2022, Viriual Event, April 25-29, 2022. OpenReview.net, 2022. URL attps://openrevale
net
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. LESS: selecting influential data for targeted instruction tuning. CORR, abs/2402.04333, 2024. doi: 10. 48500AARXIV.442204333 URL httttstt::/// //doi org/10. 48550 araiv. 2402. 24333343333333
Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, al. A comprehensive capability analysis of gpt-3 and gpt-3.5 series models. arXiv preprini arXiv:2303. 10420, 2023.
Zhitao Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec. Gn- nexplainer: Generating explanations for graph neural networks. In Advances in Neu- ral Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC. Canada, pp. neur ips. cc pttppppppp/r//aaa/ii//// html
9240-9251, 2019. URL /tt.1//10100.eeee.3034244411111
Mo Yu, Shiyu Chang, Yang Zhang, and Tommi S. Jaakkola. Rethinking cooperative rationalization: Introspective extraction and complement control. In Proceedings the 2019 Conference Em- pirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural I Language Processing, EMNI.P-ICCILP 2019, Hong Kong, China, November 3-7, 2019,
15

---

Published as a conference paper at ICLR 2025
pp. URL https:://ol..ooo//...85////////
for Computational Linguistics. 2019. doi: 10.18653/v 1420
Mo Yu, Yang Zhang, Shiyu Chang, and Tommi S. Jaakkola. Understanding interlocking dynam- ics of cooperative rationalization. In Advances in Neural Information Processing Systems 34; Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 12822-12832 2021. wtw.tttstt//eetctcttteeeee http:ttpps..//prcceeeiii.gggggg mewpriipp neurips. cc/
6a orapa///222222//llaaaaaaaoaaara3 .html
Hao Yuan, Lei Cai, Xia Hu, Jie Wang, and Shuiwang Ji. Interpreting image classifiers by generating discrete masks. IEEE Trans. Patten Anal. Mach. Intell,, 44(4):2019-2030, 2022. doi: 10.1
TPAML2020.30287
http::::: doi. org//0.......ooooo..oooooo...... 1109/TPAM.
Linan Yue, Qi Liu, Li Wang, Yanqing An, Yichao Du and Zhenya Huang. Interventional tionalization. In Proceedings of the 2016 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2023, Singapore, December 6 -10, 2023, 2023. URL https:
Wenbo Zhang, Tong Wu, Yunlong Wang, Yong Cai, and Hengrui Cai. Towards trustworthy ex- planation: On causal rationalization. In Proceedings of the 40th International Conference on Machine Learning (ICML'23), volume 202 of Proceedings of Machine Learning Research, pp. 41715-41736. PMLR, 23-29 Jul 2023. URLhttps: awxiv. 230g:///s//333311111
Yuankai Zhang, Lingxiao Kong, Haozhao Wang, Ruixuan Li, Jun Wang, Yuhua Li, and Wei Liu. Adversarial attack for explanation robustness of rationalization models. In ECAI 2024 27th European Conference on Artificial Intelligence, 19-24 October 2024, Santiago de Compostela, Spain Including 13th Conference on Prestigious Applications of Intelligent Systems (PAIS 2024), volume 392 of Frontiers in Artificial Intelligence and Applications, pp. 2290-2297,, IOS Press, 2024. doi: 10.3233/FAIA240752 UtLhtttss://// /ttp::///oooooooo 2ttp///...3///AA2220000000
Yunxiao Zhao, Zhiqiang Wang. Xiaoli Li, Jiye Liang, and Ru Li. AGR: reinforced causal agent- guided self-explaining rationalization. In Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics, ACL 2024 Short Papers, Bangkok, Thailand, August 11-16, 2024, pp. 510-518. Association for Computational Linguisties, 2024. URL https: //aclanthology. wwt.:yyy.rrg//20/0 acl acl-short.47
16

---

Published as a conference paper at ICLR 2025
Table 5: Statistics of datasets used in this paper
Train
Dev
Annotation
Datasets
Pos
Neg
Pos
Neg
Pos
Neg
S
Appearance
16891
16891
6628
2103
923
13
18.5
Beer
Aroma
15169
15169
6579
2218
848
29
15.6
Service
50742
50742
6344
6344
101
99
11.5
Hotel
Cleanliness
75049
75049
9382
9382
99
101
8.9
A MORE DETAILS
A.1 DATASETS
The statistics of the datasets are in Table Pos and Neg denote the number of positive and negative examples each set. S denotes the average percentage tokens in human-annotated rationales to the whole texts.
A.2 EXPERIMENTAL DETAILS
The code and detailed running instructions will be made publicly available on GitHub after the paper is accepted. The code is now in an anonymous repository: https : / anonymoos. 4oper. science/zz x /22R--R---- OESE, The anonymous repository will be closed in Nov 13, as we do not want people other than the reviewers to get the code.
The maximum sequence length is set to 256. We use the Adam optimizer Kingma & Ba (2013) with its default parameters, except for the learning rate (the learning rate is 0.0001). The temperature for gumbel-softmax the default value 1. We implement the code with Pytorch on a RTX4090 GPU. We report the average results of five random seeds, and the seeds are [1,2,3,4,5].
For NIR and our N2R, considering they are both variants the standard RNP, we first manually tune the hyperparameters for RNP, and then apply the hyperparameters to both NIR and N2R. For all datasets, we use a learning rate of 0.0001. The batchsize is 128 for the beer-related datasets and 256 for the hotel-related datasets. These hyperparameters are found by manually tune the standard RNP and are applied to both NIR and our N2R.
The core idea of NIR is to inject noise into the selected rationales. We use RNP its backbone. A unique hyperparameter of NIR is the proportion of noise. Following the method in its original paper, we searched within [0.1, 0.2, 0.3] and found that 0.1 yielded the best results on most datasets, hence we adopted 0.1 for it.
We found that the training of Inter RAT very unstable. To avoid potential unfair factors, our main settings are determined with reference to it. Except for the part about sparsity, we used its original hyperparameters for
For A2I, we contact its authors and get the code and hyperparameters of it.
For CR, we just keep the major settings sprasity of 10%) the same as it and copy its results from its original paper.
the Aeer-Appear dataset, and the
A.3 IMPLEMENTATION DETAILS OF N2R
For batch of (X,Y) Y), we first send X to the extractor and get the rationale Z:
fg fx))
(14)
Then, get copy of Z with the pytorch function "torch.detach("::
z' = torch.detachi ZZ),
(15)
such that the following computation does not involve the extractor's gradients. Then, we send the rationale Z' to the predictor and get the prediction Y:
Φ fr(Z').
(16)
17

---

Published as a conference paper at ICLR 2025
Table 6: The quality of rationales extracted by Ilama-3. 1----------ssttucee
Datasets
Beer-Appeuracce
Becr-Aroma
Methods
P
N2R (ours)
14.8 (0.5)
65.3 (2.7
72.7 (2.1)
14.9 (0.4)
70.2
67.3
68.7 (L.I)
Hamm-3. 3. 8h (linetune)
n/a
86.3 (n/a)
46.2 (m/a)
60.2(nn/)
n/a
73.2 (m/a)
50.6 (n/a)
59.8 (n/a)
Ilama-3. 1.1aaa...1--- 1-8b (2 shot)
15.4 (a/a)
16.0 (a/a)
15.7 (o/a)
n/a
17.9 (a/a)
24.2 (m/a)
20.6 (o/a)
Datasets
Herel-Service
Hotel-Cleenline Cleanliness
Methods
FI
P
N2R (ours)
15.1 (0.2)
42.8 (0.5)
56.3 (1.0)
48.6 (0.6)
14.8 (0.2)
53.4 (0.5)
39.8 (0.6)
11am-33114 1-Bb (finetune)
n/a
77.3 (n/u)
40.6 (m/a)
53.3 (n/a)
n/u
54.9 (n/u)
31.3 (m/a)
39.9 (n/a)
Ilama-3. 3, 1-8h (2 shot)
n/a
45.3 (n/a)
51.7 (m/a)
48.3 (m/a)
n/a
39.3 (m/a)
43.0 (m/a)
41.1 (naa
Then, we update the predictor with the cross-entropy loss.
min H.(Y,Y)
(17)
8y
Note that this updating process with cross-entropy will not influence the extractor, since have used "torch.detachh" for Z.
Then, we fix the parameters of the predictor, and only update the extractor. We first get the rationale candidate Z with
- - f And we then send it to the predictor's 's encoder to get |n l222. Then, we update the extractor with
(18)
min min-tog((EEne())1a
(19)
Then, we get into the next round to update the extractor and the predictor again.
A.4 IMPLEMENTATION DETAILS OF N2R+MMI
The implementation details are similar to those Appendix A.3 The only difference is that we no
more use "torch.detach".
A.5 THE MINIMUM
IS EQUAL TO THE ENTROPY
The cross-entropy consists of two parts:
HY(Y,YiZ) H() (2
(20)
When minimizing the cross-entropy H,(Y, by adjusting the predictor's parameters, in fact minimizing (xL((((12))IP(()12)) And know that if the predictor is trained ideally, we have min (PP1)))))))))))) Then, we have
min H.(Y, Ý1Z) HY) 0p
(21)
A.6 THE DETAILED DISCUSSION ABOUT EQUATION 6
In any cases, we have
( I(Y; IYY;; - -I Ra)R)- I(Y; R2).
(22)
(23)
The magnitude relationship between T(Y; (1) and I(Y; R2) is arbitrary. In other words, there exists some scenarios where 1 < <{;)))<(())))) R In such cases, we have
I(Y: R2, < 222x <I(Y;;;)) ([Y;R;)+((* +();
(24)
In these cases, the mutual information faces the diminishing marginal returns problem.
A.7 THE RATIONALES EXTRACTED BY LLAMA-3. 3. 3..---B-NSTTTUCCT
18

---

Published as a conference paper at ICLR 2025
To further show the potential impact of rational- ization in the era of LLMs, here we present the results of the experiments conducted with the 1aama-3.--bbinnstee
Task: Sentiment classification about Beer's appearance Input: Pours a rather crisp yellow almost orange with a thin head. The aroma dominated by sweet malts with just slight hoppiness dancing in the background. The taste does have surprssng amount hoppiness for Pilsner. There is good maltiness to it as well, but citrus hops just slightly overpower The beer is very light and refreshing. This makes for excellent summer Expected output: 1lpours a rather crisp yellow almost orange with a thin head. Ilama-3 oulput: 1puur a rather crisp yellow almost orange
model. We perform both 2-shot prompting and supervised fine-tuning.
For 2-shot prompting, we provide the model with negative text with its cor- responding rationale, and a positive text with its corresponding rationale. For supervised fine-tuning, the super- vison label is the classification label, since we perform unsupervised ratio- nale extraction.
beer.
An example of Ilama's 's output. Here "I" means that the class label Y positive. And the words after "I" represent the rationale.
Figure 5:
We use 4*RTX 4090
24GB GPUs and LoRA to fine tune the models. We provide detailed docu- ment in our anonymous code repository (https: / //aonmoous include all the details (including the prompt templates, LoRA fine-tuning parameter settings, and more).
of 1ims pdf) to
In most cases, the model output the ratio- nale in the correct format. 5 shows an example. But in 2-shot prompting, the model sometimes outputs additional parts along with the rationale (through manual observation, this situation does not occur frequently.). Figure another example. such cases, we use gpt-3.5- turbo to extract the content within the quotation marks.
Task: Sentiment classification about appearance Input: 22oz bottle pouted into Goblet: Opeque orange with light, white, creamy head that was not all that well retained but full carbonation, but settle into small thin cap. The Sweet and malty. The taste a very nice balance of the two styles. A little more hops, but balanced very nice with sweetness the malt and fruit. to full body, perhaps little thick for my taste, but still good. The beer had a nice bitter dry aftertaste and was well carbonated. The beer was fairly easy to drinks give the A BV, after the 22oz, was protty well done. Overall, good beer and probably the first one of the side projects that I think the brewery should consider brewing on regular basis.
was Belgian triple than IPA,
beer had medium
The results are shown in Table LLMs are not good at counting, so we did not constrain the percentage length (i.e., sparsity) of the rationale extracted by the model. Compared to our N2R, Ilama-3.1 does not have crushing advantage. On two out of four datasets, our N2R outper- forms 1lama-3.1. And the left two datasets, N2R achieves comparable results to Ilama-3. 1. Besides, our N2R can be applied to graph data, while it is not easy to do so for LLMss
Expected output: lopeqee orange with light, white, creamy head Uhat was not all that well retained but full carbonation, but did settle into small thin cap. Ilama-3.1 outpul: positivefT he overall tone of the review is positive, with phrases such 'a very nice balance of the two styles", "nice bitter dry aftertaste", "well carbonated" and "overall, a good beer' indicating a favorable opinion of beer. JPT-3.5rrestructured output: 1 very belance the two styles nice bitter dry aftertaste well carbonated overall, a good beer
A.8 MORE RESULTS OF DIMINISHING MARGINAL RETURNS
Figure shows the results more datasets.
Figure 6: An example of llama fails to output the rationale in the right format.
A.9 A VISUALIZED EXAMPLE EXTRACTED RATIONALES
Figure g dataset.
shows a visualized example of rationales extracted by different methods on the Beer-Aree Aroma
Figrue shows a visualized example of rationales extracted by different methods on the CUB (see Appendix 10p dataset.
A.10 THE RESULTS ON CUB DATASET
TL;DR. The results are in Table 7
The CUB dataset (Wah et al. 201T) contains photographs of birds annotated by species, totaling around 1 11.7k samples. In experiment, follow SSaawaa al. (2099 categorize them into
19

---

Published as a conference paper at ICLR 2025
0.00
acc
4.50
nom
0.63
4.75
0.78
4.70
0.76
4.50
0.78
4.35
0.72
4.45
0.4
0.6
0.5
0.5
60
0,
0.6
0.5
1.2
0.0
0.2
2.4
0.6
1.
The rabio me
The rabi real sattenals
The ratio
(b)
(c)
0.90
2.0
4.2
0.8
4.0-
0.00
0.70
2.5
0.65
2.4
0.00
0.55
7.0
0.50
2.8
0/4
12
0'0
The raflo of real rarlonal
The nit of resl attonal
The rato of real ratenaale
(e)
(f)
2.50
0.05
225
entrooy
0.00
0.75
1.75
6.70
152-
4.2
0.05
1.0-
0.00
0.59
050
0'1
0.8
54 0. 3.0
The raalaaanna
The
The resioalle
(h)
Figure 7: The prediction accuracy, cross-entropy loss, and the norm of the representation through the neural network vary with the proportion of true rationale components in the rationale candidate input within trained standard RNP predictor. (a)())(e): the Beer- Apperappee dataset. (f)( the Hotel-Service dataset. (g)(h)i): the Hotel-Cleanliness dataset.
two categories: waterbird (albatross, auklet, cormorant, frigatebird, fulmar, gull, jaeger, kittiwake, pelican, puffin, tern, gadwall, grebe, mallard, merganser, guillemot, or Pacific loon) and landbird (ani, blackbird, bobolink, bunting, cardinal, catbird, Yellow Breasted Chat, Eastern Towhee, Chuck Will Widow, cowbird, Brown Creeper, crow, cuckoo, finch, Northern Flicker, flycatcher, goldfinch, grackle, grosbeak, hummingbird), (Sagawa et al. (2019) annotated each image with the bird's silhou- ette in order to study the overfitting problem, which can serve as the ground-trun rationale.
The classification task on this dataset is challenging due to severe overfitting problems. Since other baseline rationalization methods are specifically designed for text data and are not suitable for image data, we only compare with vanilla MMI (i.e., RNP) to validate our method's effectiveness rather than competitiveness. We also compare with a vanilla classifier (ResNet18) to verify the effective- ness of our method in improving classification accuracy.
Details and metries. For this image classification dataset, the extractor and predictor in Figure [ are different from those used in performing text tasks. The goal of the extractor is to select a portion of the pixels from image as the rationale, which can be seen as performing binary classification on each pixel, similar to image segmentation. So, we use an U-Net to be the extractor. And for the predictor, use ResNet18. The meaning of F1 is the same as the one used in Table between the extractor selected pixels and ground truth pixels). We add an additional metric, IoU (a common metric for image segmentation), to further measure the rationale quality.
[ablee((th) (the overlap
20

---

Published as a conference paper at ICLR 2025
Label Asomma
Labd
(Beee Aroma)
Ammal:
Labe (eee-Ameel
Postivee
Postlive
Postire.
Postive.
Positive
Prodiction Positive.
Predictios Positive
Prodictoon Posilive
Predicioon Poclive
Predictions Fosttive
Impot: medium bewwn.
Imput modium brown
Impat: medrum beown.
Impat: medium brown,
lmpu:: medium beows, faity clear light
Eudy clear higt
Sady clear light
farly clear and light
facty cler and light
penetmole. small thin
penerable. thin
penstmilee small ring
prommle small thin ring
penetrall. small than rang of while form
foom. sweet and nshumalt mery very nuzy
foom sweet and ath malt smalls very zutty mosted almond. pozmn
ric malt snala, yes nunr
noh malt smells, very mayy mostad almond, pecan
malt smells roased
muserd almond. pezon
modeed almond, pezan
pecan)
with some commel. nch almond and toffee bittor
with some cammel rich almond and toffee slight as sweet es nose implies, some dryness and alcohol present [unknowm)
with some commal nch
with some carmell meh
with some caramne. tich almond and tocfne notes,
almond and todfee bitter
slmond and taffze
slight as sweel as nose impties.
bitter
slight
some bitter coroa or slihtt espresso may be not the nose implire,
expreso maybe.
oxpre oa mayhe
expresso moyhe sweel as nose implite,
tsprssoo maybe.
sweet the nose impliee, some ress and aloshol prosont. [unkmown)
some dry dryness and alcohol present. [unknown]
some dry ness and alcohol present. I [unknown))
some dryness alcohol presntt [unknown)
alcobulic. laily strog carbonation,
alcobulic. Eairly hewvy cartonation, mouthfoel. poett diniiable..
aicoholic faidy stropg earbonstion, mouthris protty drikable..
alouholic. Juidy
alosholtc. fairyy strong curbonation. medium heavy mouthreet pestty drinkabio.
heavy mouthfoee. pratty drnkable...
besvy
cortonation. medium leavy mouthried pentyy drnkable.
[a] RNP
Inter RAT
(c) NTR
(6) A
N2R (ours)
Figure 8: Visualized example rationales on the Beer-Appearance dataset. The underlined words are annotated ground-trum tound-truth rationales. Model-selecter rationales are highlighted with different colors.
ground truth rationale
selected
mask
mask
rationale
image
(a) RNP
(b) N2R
Figure 9: A Visualized example of rationales on the CUB dataset.
Results. The results are shown in Table On this challenging image classification dataset, N2R still outperforms the MMI-based method RNP. As for the improvement in classification accuracy compared the vanilla classifier ResNet18. one possible reason is that our N2R retains informa- tion relevant to classification while removing irrelevant noise (however, the poor performance of RNP may be due its failure to find comprehensive useful information). This phenomenon is con- sistent with the findings of Wu et al. (2022). which suggest that extracting rationales can enhance generalizability to some extent.
We also provide visualized example for this dataset in Figure 9 small part of the bird but a large part of the water (shortcut) to classify this image as "waterbird". While the N2R-selected rationale includes the whole bird.
We find that RNP selects only
Table 7: The results on the CUB dataset.
Methods RNP N2R (ours)
Ace
P
R
F1
loU
15.5 (1.1)
81.3 (0.7)
33.3 (2.1)
37.0 (4.7)
35.0 (3.2)
21.3 (2.4)
15.8 (1.1)
86.5 (0.6)
46.1 (5.4)
52.5 (4.8)
49.1 (4.9)
32.6 (4.2)
Classifier (ResNet18)
n/a
85.0 (0.8)
n/a
n/a
n/a
n/a
21

---

Published as a conference paper at ICLR 2025
Table The results on the MovieReview dataset. "*": The results the baselines are copied from
Inter RAT(Yue et al. 2023) (they did not report the classification accuracy).
Methods
S
Acc
R
F1
RNP*
20.0
35.6
21.1
24.1
A2R*
20.0
48.7
31.9
34.9
INVRAT
20.0
33,9
24.3
28.3
Inter-RAT
20.0
35.7
35.8
35.7
N2R
20.3 (2.2)
88.4 (2.1)
45.6 (2.1)
31.7 (1.6)
37.4 (1.9)
Classifier
n/a
86.4 (2.3)
n/a
n/a
n/a
Table 9: Results with BERT encoder. The dataset the most widely used Beer-Appearance. results obtained from the paper of CR (Zhang et al. 2023).
Datasets
Beer-Appcarance
Beer-Aroma
Methods
Ace
P
F1
Acc
FI
RNP
(0/a)
(1.7
(1.4 20.3 (1.9)
(1.7
(m/a)
(2.0
(3.2
28.7 (2.2)
(2.5)
A2R*
10.0 (n/a)
(2.2)
(0.0
(1.6
(1.3
10.0 (n/a)
81.5
(2.8
34.8 (3.1)
(1.)
(mfa)
(3.1
(1.2
(mla)
90.0
(9.6
(1.9
(2.2
CR"
10.0 (n/a)
(1.7
(4.1) 59.7 (1.9)
(1.6 39.0
10.0 (n/a)
(2.5
68.0
(3.0
(2.1
N2R (ours)
(1.8)
(2.5
(..1)
91.0
(3.7 (5.3
(4.6
Classiierr
m/u
92.6
n/a
m/a
m/a
m/
91.6
a/a
ma
n/a
RESULTS ON MOVIEREVIEW DATASET A.11
TL;DR. The results are in Table g
MovieReview (Pang & Lee 2004) is text classification dataset with much longer texts (the average length 774 words) as compared to the datasets used in Table and 2 Recently, Young et al. 2020) annotated rationales for this dataset so that it can be used for the rationalization task.
We follow the settings of Inter_RAt RAT (Yue et al. 2023) to conduct experiments on this dataset. The encoders for extractor and the predictor are both GRUs. And the word embedding is VloVe-100d. The maximum sentence length is set to be 1024.
We compare with the baselines already implemented by Inter and copy the results from it. We also compare with vanilla classifier (GloVe+GRU) to see the classification performance.
The results are shown in Table 8 this long-text challenging dataset.
We see that our N2R still outperforms MMI-based methods
A.12 COMPARISON WITH BERT CLASSIFIER.
compare the classification accuracy with vanilla classifier implemented with BERT on Aeer-Appe and Beer-Aroma datasets. The results are shown in Table 9 Our N2R gets even better accuracy than the BERT classifier. One possible reason that our N2R retains information relevant to classification while removing irrelevant noise. This phenomenon is consistent with the findings of Wu et al. (2022), which suggest that extracting rationales can enhance generalizability to some extent.
A.13 THE CONVERGENCE SPEED
Figure 10 shows comparison of convergence speed between RNP and N2R. The batchsize is 128 and the learning rate 0.0001. We see that N2R converges much faster than RNP, which means that train N2R with fewer steps and thus saving the computational costs.
22

---

Published as a conference paper at ICLR 2025
72
10
62-
22-
40-
40-
32-
RNP
RNP
22-
N2R
N2R
15
40
20
150
10
40
130
120
1/0
Training epochs
Tra ining epochs
(a)
Figure 10: The comparison of convergence speed between RNP and N2R. (a) The Beer-Appearance
dataset. (b) The Beer-Arone dataset.
A.14 THEORETICAL SUPPORT FOR USING NORMS TO DISTINGUISH BETWEEN RATIONALES
Lemma (CCai et al. 2013)) Let U and V be two random points on a p-dimensional unit hyper- sphere RP, and O is the origin. Let be the angle between vector OU and vector OV, then
T "1 2) K K Pr([O
(25)
for all p22 2 and e e (0, 1/2). where K is a universal constant.
Lemma Dis the Proposition of (Cai at. 2013). And it tells us that "all high-dimensional random vectors are almost always nearly orthogonal each other" (Cai et al. 2013). As the dimension p increases, O gradually converges to π 7/2
If two vectors are orthogonal to each other, then their dot product will be zero.
We consider a simple case. Consider a FFN layer without the bias term. And its weight matrix is W _g and the input is X I The output of the FFN layer is XW R" Each dimension in X W is the dot product of X and a column vector of W. The column vectors of W represent some directions the model handle. For well-trained FFN, the directions are determined by its training data and represent the information of it learns.
We use the lower case denote a specific input.
Consider an input 2 does not contain any information that has been learned by W. For example, W is part of classifier trained to distinguish between cell phones and water cups, and 21 is photo of a grassy. Although 21 is not truly random noise, since W has never been trained to recognize grassland, 21 acts like random noise to W. So it is likely that 21 is orthogonal those column vectors in W. So, every dimension of æ W will approach zero and thus ||r, WI|2 will also approach zero.
If an input 22 is full of the information learned by W, then it probably matches the directions of W. And thus || W|la will be high.
Then, what would be the intermediate state of xı and x2? We use V 1m to denote an arbitrary column vector in W. We assume an input x3. The first dimensions of æ3 (denoted as 1ik) contain useful information, and the last dimensions represent uninformative noise. We have that
Z3
x3
l:k
l:k
+ 23
1on b
(26)
J++ I:m consists of uninformative noise, we have 25 Imn k Ion 0. Thus D3 V x.1ik V
We assume that the useful information is uniformly distributed each dimension of and V. For an input x, denote æ² as the i-th dimension of it. Formally:
23

---

Published as a conference paper at ICLR 2025
Assumption If input contains k-dimennsinnal useful information (for simplicity, always assume that the useful information is in the first lo dimensions), we assume that the informativeness between different dimensions is the same:
x² V V2, Vi k,j k.
(27)
With this assumption in place, quantitatively analyze why our method is not affected by the problem of diminishing marginal returns.
Consider that the proportion of the gold rationale components in extractor-selected rationale candidate is Aa (i.e., there are ka of dimensions in aa represent useful information). When we add more gold rationale components to it to make the proportion be kg and the new rationale candidate be x. We will have that
I Wlk ka 1/2 Wil
(28)
Proof.
Ta W = [xa V1, Fa Vn]
= [I V.lik. , x x1kko V22ika]
(29)
Lol Vi m 2t V
where the third equation is from Assumption Similarly, we have
1
W Ko 1/1 "1,," x/2 V.
(30)
ry is got from replacing the uninformative part of xa (i.e., the last few dimensions) with useful information, so we have =² = x Thus we have W.W. x W
and Ir_ Wl W.
K
Conclusion. Ideally, the norm metric designed should increase approximately linearly as the proportion of gold rationale the rationale candidates grows, thus avoiding the problem of dimin- ishing marginal returns like MMI-based methods.
Although we made an assumption that may not hold in reality, it was made to facilitate a better quantitative analysis, and the conclusions drawn at least have qualitative trend. The trends in Figure 7 (f)(()) also verify this trend.
A.15 A TOY EXAMPLE FOR BETTER INTUITIVE UNDERSTANDING OF OUR METHOD
Consider network consists of only linear layer (without bias) and has low-rank weight matrix M that, after elementary row transformations, takes the following form:
1
0
0
1
0
A and B represent inputs with and without the model's Icarned information respectively. After undergoing corresponding transformations (cxisting research indicates that after performing certain elementary transformations the weight matrix, corresponding transformation can be found for the inputs, resulting in identical outputs (Ainsworth et al.) 2023)), they are likely to take the forms like: A 0,0, B [0,0, 1j. The informative input A usually matches more with the non- zero parts of M. And ||AM]] v > ||BM]]2 0. (The column vectors of the weight matrix represent certain directions in high- righ-dimensional space (determined by the learned information), but random noise B generally does not fall along these directions does not contain the correspond- ing information.)
24


Reviews:
[{"review_id": "qv6e7pso9g", "reviewer": "Reviewer_f8iZ", "summary": "+ The paper first identifies an issue of current MMI methods: once part of the rationales are detected, it is hard to find the remaining parts due to diminishing marginal returns.\n\n+ Targeting this issue, the paper empirically analyzes the norm of the rationale's representation in the training process, considering the relationship between the information richness and the representation norm.\n+ Based on the analysis results, the paper proposes to use/add the log representation norm to the Extractor's training loss, and the proposed method achieves better performance compared to several baselines.", "questions": "+ Comparing Table 1 and Table 3, I found that BERT-based methods (i.e., taking BERT as the encoder) seem to perform worse than GRU-based methods on Beer-Appearance. It is not intuitive for me as BERT is expected to be a stronger model. Do you have any explanations?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ The identified issue of current MMI methods is meaningful and insightful.\n+ The proposed method makes sense intuitively. It is also a simple way to tackle the identified issue and seems to be effective.", "weaknesses": "+ Missing some key evaluation results\n  + What's the accuracy of the Predictor (i.e., task accuracy) if taking as input the rationale extracted by BERT-based methods and llama-3.1-8b-instruct?\n  + Why did you evaluate BERT-based methods on ONLY Beer-Appearance? Is there any specific reason for not showing results on the other three datasets?\n+ It would be better to study what the improvement in the numbers actually means. Humans may have very different annotations when asked to extract a rationale. For instance, in the toy example (line 266), I personally prefer taking only R2 as the rationale as it is quite enough to make the prediction (and thus I can give a more concise rationale than R1+R2). Therefore, it is unclear whether the number improvement really means a better rationale (e.g., a rationale that avoids missing information that is actually essential). One possible way is to do some case study or human analysis to compare rationales extracted by different methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FAvgTlMhGD", "reviewer": "Reviewer_hZKo", "summary": "This work suggests extracting crucial rationales from an input of language models using the weight matrices of these language models. The proposed method is an alternative to the more traditional MMI method. It should help us move towards more explainable AI by showing which parts of the input are crucial for making a decision.", "questions": "- What does x axis of Figure 2 mean?\n- What do notations in Equation 2 mean? (they are not defined in the text)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The work investigates an important topic: XAI.\n- Authors repeat each experiment several times and report standard deviation.", "weaknesses": "It is very hard to understand the contribution:\n\n- The method description in abstract, introduction and many other places of the paper seems to be simply wrong. Weight matrices of LLMs (excluding \"attention weights\"; but the paper doesn't tell anything about attention weights) are not dependent on the particular input, so they can not be used for estimating the parts of inputs. The method description in the paper should be rewritten completely.\n\n**UPDATE**: The authors have addressed the previously mentioned weakness and clarified their intended meaning. They made a revision of the paper. It is now very clear from the revised text that they studied the interaction between the weights and the input representations from the previous layer of the model (see discussion below).\n\n- I suppose that the authors could mean components of the embedding matrix, not the weight matrix? But this suggestion still leads to confusion: I looked into the code and it still wasn't clear how exactly the model's embeddings were used in calculations. E.g. the function \"`get_embeddings`\" from embeddings.py is not used in other scripts at all. At the same time, \"`get_glove_embedding`\" function was used; but there is not even a single mention of GloVe in the paper itself, so there is a contradiction again. I urge the authors to make their code more clear and consistent with the paper text.\n\n**UPDATE**: The authors have clarified this inconsistency in their comments, and the revised paper now mentions GloVe.\n\n- The reasoning about the rank of \"weight matrices\" also sounds unclear and not founded enough. It is again unclear, whether weight matrices or embedding matrices are considered; besides, in the context of neural networks, it is hardly possible to talk about \"zero matrix rank\" for sure in any case. In a strict sense, \"zero rank\" would mean that the whole matrix is exactly zero which is very rare case in practice. Probably authors mean some approximation of the matrix rank, but the exact formula for that approximation was never written down.\n\n**UPDATE**: The authors provided somewhat more justified explanation for this part (see the discussion below and the revised paper).\n\n- Figure 2 doesn't look correct, it's meaning is unclear (see \"Questions\" part). Its description also looks unclear and not founded enough.\n\nOther issues:\n\n- If we assume again that authors mean embeddings and not weight matrices (as follows from common sense and from \"4.2 The practical method\" part of the paper), it is still not clear, what will happen if the input isn't connected to the task at all. E.g. if the input is like \"John went for a walk\", and the task is to estimate, whether he liked the beer aroma or not. It seems that the suggested approach (or, at least, my, very vague understanding of it), isn't able to correctly handle such situations.\n\n**UPDATE**: Figure 2 and OOD example were commented in the discussion.\n\nMinor issues:\n\n- Citations in lines 167-173 should be put into braces.\n\n**UPDATE**: After a thoughtful discussion with the authors and reviewing the revised version, I now have a clearer understanding of the method, and it seems reasonable to me. As a result, I have increased my main score from 3 to 6. The other scores have been adjusted accordingly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wQVFVBzrwf", "reviewer": "Reviewer_meUo", "summary": "The study introduces an alternative approach to Maximum Mutual Information (MMI) for extracting rationales in explainable AI (XAI). Traditionally, MMI-based methods have been used to extract relevant features by maximizing information shared between input rationales and predictions. However, the authors argue that MMI suffers from diminishing marginal returns, limiting its efficacy in fully capturing critical rationale components. They propose using the norms of input representations, aligning extracted rationales with utilized parts of the network's weight matrix. This novel method Norm to Rationale (N2R), reportedly outperforms MMI-based methods across several datasets (text and graph-based). Authors use GRUs, BERT and GCN as an encoders, and several text classification and one graph classification dataset.", "questions": "I find the approach interesting, but I have a few questions and suggestions that could further improve the study:\n\n- How does this approach actually improve classification quality in real-world scenarios? The paper mentions MMI limitations, but do these truly impact final performance? I understand that classification performance isn’t the primary goal of explainable AI, but it would be interesting to see if there’s a noticeable effect.\n- You claim generalizability across domains, yet the experiments are limited to text and graphs. Could this method be extended to other domains, such as speech or even image classification?\n- It is unclear if there are specific limitations regarding encoder and extractor architectures. Could you clarify this? Additionally, it would be interesting to see how the approach scales with (a) an increasing number of encoder parameters and (b) longer input text lengths. I believe such experiments would enhance the work, making it more valuable for tasks like in-context learning and RAG prompt improvements.\n\nMinor Remarks and Questions:\n\n- It is unclear why you use the logarithm in Eq. (8) and subsequent equations. Is this related to Fig. 3c?\n- Figure 3 is somewhat confusing: there are no y-labels, and the caption lacks references to panels (a), (b), and (c).\n- Figure 2 appears after Figure 3; it may be better to reorder them for clarity.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The use of encoder output norms for rationale extraction is novel, and the experiments provide convincing motivation for using this approach over MMI-based methods.\n- The theoretical analysis of MMI-based methods is clear and engaging.\n- The proposed method is promising due to its simplicity and potential for generalizability.\n- The authors conduct experiments with multiple random seeds and report results with standard deviations, which significantly improves the rigor of the work and reinforces confidence in the performance of the approach.", "weaknesses": "- The experiments are limited to datasets with annotated rationales, and the authors do not present results showing how their approach improves classification quality in more advanced real-world datasets.\n- The experiments focus solely on text and graph domains. The claim of generalizability would be strengthened by including additional domains.\n- The impact of norm-based rationale selection on computational resources, particularly for larger models, is not discussed. Additionally, the authors omit details and experiments regarding the properties of the encoder and extractor, such as whether the encoder should have a specific architecture or how scaling the encoder affects the method.\n- While the empirical results are compelling, the theoretical foundation for norm-based rationale extraction could benefit from additional depth. More detailed mathematical insights into why norms better capture input utility and how the input interacts with the encoder's weight matrix could strengthen the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v5kki7BTFt", "reviewer": "Reviewer_3eT9", "summary": "The authors propose a novel approach for extracting rationales in a select-predict-like pipeline, replacing the traditional MMI loss criterion  with a new objective that is independent of the output prediction.\n\nThey claim that MMI suffers from diminishing marginal returns, that is, the model can select a portion of the input as rationales and still predict the true label with insignificantly lower accuracy.\n\nTherefore, instead of relying on the output prediction for evaluating the rationales extraction, they prob the weight matrix of neural networks norm to detect snippets that the model truly utilize.", "questions": "- I was wondering what are the metrics when you pass gold rationales to the predictor directly. This would indicate the maximum performance you could reach.\n- I was wondering what is the performance of your framework w.r.t. OOD data. This helps to understand if the model learn to rely on plausible rationales instead of statistical shortcuts\n- BeerAdvocate is more sentence level rationales, evaluating on token level rationales (like e-SNLI) which is more challenging would be interesting.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- Model agnostic method (They propose a new metric for rationale extraction which is regardless of the model itself as long as you have access to the weights).\n- Their method falls in the self-explanation category which guarantees faithful rationales.\n- Potential to be extended to image classification and graph neural network.\n- Mixing N2R and MMI improves the performance even better as MMI works better in the early stages of finding rationales and N2R works better when the models already detected few.", "weaknesses": "- Works like UNIREX already discussed the problem you mentioned about MMI by introducing \"sufficiency\" and \"comprehensiveness\" metrics.\n- The extractor and the predictor are not trained cooperatively (end-to-end) due to the non-differentiability of rationale space.\n- You should put the task performance in the results. \n- Visualization of some extracted rationale would be great.\n- You didn't provide the hyperparameter (for example in Eq. 5) in your experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)