DARE: Disentanglement-Augmented Rationale Extraction
Authors: Linan Yue
Qi Liu
Yichao Du
Yanqing An
Li Wang
Enhong Chen
Venue: NeurIPS 2022 Accept (2022)
Keywords: Rationale Extraction
Disentanglement
Mutual Information

Abstract: Rationale extraction can be considered as a straightforward method of improving the model explainability, where rationales are a subsequence of the original inputs, and can be extracted to support the prediction results. Existing methods are mainly cascaded with the selector which extracts the rationale tokens, and the predictor which makes the prediction based on selected tokens. Since previous works fail to fully exploit the original input, where the information of non-selected tokens is ignored, in this paper, we propose a Disentanglement-Augmented Rationale Extraction (DARE) method, which encapsulates more information from the input to extract rationales. Specifically, it first disentangles the input into the rationale representations and the non-rationale ones, and then learns more comprehensive rationale representations for extracting by minimizing the mutual information (MI) between the two disentangled representations. Besides, to improve the performance of MI minimization, we develop a new MI estimator by exploring existing MI estimation methods. Extensive experimental results on three real-world datasets and simulation studies clearly validate the effectiveness of our proposed method. Code is released at https://github.com/yuelinan/DARE.

DARE: Disentanglemen---ugmmnette Rationale Extraction
Linan Yue 12 Qi Liu
Yichao Dul Yanqing 12 Li Wang ,2,3 Enhong Chen
Bangl.233
1: Anhui Province Key Laboratory of Big Data Analysis and Application, University Science and Technology of China
2: State Key Laboratory of Cognitive Intelligence
3: ByteDancc
(Inyue, , duyichao, anyq, w1063) @mail. uste. edu..eduucoom cn; {qiliuql cheneh) @ustc. edu. en
Abstract
Rationale extraction can be considered as straightforward method of improving the model explainability, where rationales are subsequence of the original inputs, and can be extracted to support the prediction results. Existing methods are mainly cascaded with the selector which extracts the rationale tokens, and the predictor which makes the prediction based on selected tokens. Since previous works fail to fully exploit the original input, where the information of non-selected tokens Rationale
is ignored, this paper, we propose a Areentanglement-Augeenntt Extraction (DARE) method, which encapsulates more information from the input to extract rationales. Specifically, first disentangles the input into the rationale representations and the non-ration non-rationale ones, and then learns more comprehensive rationale representations for extracting by minimizing the mutual information (MI) between the two disentangled representations. Besides, to improve the performance of MI minimization, we develop a new MI estimator by exploring existing MI estimation methods. Extensive experimental results on three real-world datasets and simulation studies clearly validate the effectiveness of our proposed method. Code is released at https: //g::///iiii h/ggttgttb..////////////g com/yuelin comm/yeelinnnDARE.
Introduction
Although the performance of deep neural networks (DNNs) has significantly improved across range of natural language understanding tasks [43, 49, 50], the inability of DNNs to provide the explainability for their predictions still remains a serious risk. To this end, several researchers [35, 26, 28] have focused on improving the explainability of DNNs. Among them, the rationale extraction method [24, 4] can provide an intuitive explanation by identifying the important features of inputs. Specifically, it extracts short and coherent part of original inputs (i.e., the rationale) as an explanation to support the prediction results when yielding them.
Traditional rationale extraction approaches [24, 4, 44, 33] cascade the selector and the predictor. As shown in Figure I(a), illustrate this type of approaches with an example of the charge prediction, where charges are automatically predicted based on the case fact. Specifically, the selector first selects subsequence of the fact description (i.e.. the bolded tokens in Figure 1(a). Then, the predictor yields the charge result based on the selected tokens. And the extracted subsequence is defined as the rationale. However, since the extractive rationales only depend on the comparison between the prediction results and labels, this kind of approach may fail to fully exploit the information of the original input. To solve that, Sha et [37] add an external guider (i.e., Figure 1(b)) which takes the
"Corresponding Author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

whole text as the input and generate the accurate but uninterpretable representations as most DNNs do to predict the result. Then, the guider utilizes the above representations to guide the predictor to yield more comprehensive task-related representations with an adversarial-basse method. Since this method fails to utilize the information of the original text, where the non-rationale tokens are ignored, we argue that this "guidance pattern" can be further explored to improve the rationale extraction.
Along this research line, in this paper, we propose a " self- guided method, Disentanglement-Augmeentt Extraction (DARE), which augments traditional rationale extraction approaches with the disentangled representations learning. Different from the previous model that requires external guidance, DARE aims to guide itself to extract more comprehensive rationales by squeezing more information from the input. Specifically, we first disentangle the origi- nal inputs into two parts: the rationale tokens and the non- rationale ones by the selector, where the two type tokens should be independent ideally. Then, in the predictor, consist- ing of encoder and classifier, these two type tokens are passed into a shared encoder to obtain the corresponding rep- resentations. Finally, DARE utilizes rationale representations to yield results. Meanwhile, the guider adopts non-rat non-rationale representations to guide the rationale ones to be more com- prehensive by reducing the dependency between the two rep- resentations (i.e., minimizing their Mutual Information (MI)).
nrightll caet fett cxtwein the vees let cotteder vaated wate yen
Rationale
Rer dcinde and wake
Adtaale disted de veree win
whic
wvs wile wostle yol
ando by pllwwen
aidendan darket the dicee wilh
the rulles sucadio
predictor
Charge: Dirac of tesccood
Chagge Chipe BEEE
(b)
Figure I: Schematic of existing ratio- nale extraction approaches presented in this paper. (a) illustrates a tradi- tional process of rationale extraction that utilizes the selected rationale (in bold) to yield results. (b) shows an external guider to guide the rationale selection based on the whole input.
Besides, to improve the performance of minimizing MI, based on the state-of-the-ar MI minimization method, Contrastive Log-ratio Upper Bound (CLUB) [11, I 12], we propose a new implementation of CLUB (i.e., CLUB_NC (c_E)) by exploring the relationship between CLUB and InfoNCE (a classical maximizing MI method [32]). Following the experimental setup of [34, 11]. we evaluate the effectiveness of CLUB _CE on two simulation studies by comparing with other MI minimization methods.
To validate the effectiveness of our proposed DARE, we perform extensive experiments on the BeerAdvocate dataset [29] which is utilized to the multi-aspec sentiment analysis, movie review dataset [48]. and legal judgment prediction dataset (CAIL2018) 8) [41] which is adopted to predict judgment results including charges, law articles and terms of penalty. The experimental results empirically show that DARE has achieved better performance on both extracting rationales and predicting task results compared to other state-of-the-art rationale extraction methods.
2 Dissntanglement--Aggneeeee Rationale Extraction
2.1 Problem Definition
Given a text input x = {21, ,2, consisting of n tokens and the ground truth y, our goal is first to learn a mask variable m (m,, m.,.... m,, m,), and then learn model which adopts the rationale z = m@r = {ml I,, m2- 22,... mp ] to yield the prediction results. Take the case in Figure for ex- ample, based on the fact description x, DARE aims to yield the charge result y while extracting the rationale to support this result. Next, we will present the details of our proposed " self-guided method, DARE, which consists of the selector, predictor and guider.
prodicirr
croole
cailee
guider
Figure 2: Architecture of DARE consisting of the selector, predictor and guider the white boxes indicate the rationales related tokens and the black are non-related ones.
2.2 Architecture of DARE
To utilize the non-rationale representations to enhance the rationale ones, and further extract more accurate rationales, propose the DARE method which is shown in Figure 2. In DARE, we first

---

utilize the selector to select the rationale tokens and the non-rationn non-rationale ones, and adopt the shared encoder to generate the corresponding representations. Then, the classifier in the predictor can yield the prediction results based on the rationale representations. Meanwhile, the guider considered as a regularizer minimizes the MI between the above two types of representations to ensure they are explicitly disentangled, which can make the rationale representations more comprehensive.
2.2.1 Selector
To select the rationale tokens, the selector takes x: {x1, I2,...,,an] the input generate binary variable m = (m1, m2,.... mm,, and adopts m to sample the rationale and non-rationale tokens. Specifically, we first utilize the selector generate a probability distribution p(m[2z which represents the probability of sampling each x; as the part of the rationale. To ensure this operation of sampling is differentiable, we adopt HardKuma reparameterization trick [4] generate the p(m|æz);
kj = =F F Kuma (u;; by): ++ (r - 1) x kj. p(my(()) min (1, max (0, (0)))))
(1)
where Uj is random sampled from the uniform distribution 10, FRuma (u; a, b) =
(1 (1 //) 1/a is the inverse c.d.f. of the Kumaraswamy distribution [23], and b are pa- rameterized by DNNs which takes x as the input, and and I are fixed hyperparameters. Finally, we can derive the rationale tokens as 21 = m @ x and the non-rationale ones as 22 = (1 - m) @ Z.
2.2.2 Predictor
The predictor qu (1)21) which consists of encoder and classifier outputs the prediction results based on the rationale tokens z1. Specifically, we first re-embed the 21 into continuous hidden states by the encoder to obtain the rationale representations Uz Rd where d is the hidden size. Then, we pass the vs, into the classifier to yield the results y which is predicted based exclusively on the rationale tokens extracted by selector, and the task loss function can be formulated as
Ltask (ssppo(s)l)) [- log 9 (y) 21)]
where Pa is the neural network of selector.
2.2.3 Guider
While predicting results, to guide the model itself to extract more comprehensive rationales, we augment the disentangled representations learning (DRL) into the guider. Specifically, we pass non-rationale tokens to the shared encoder described in section 2.2.2 to obtain the non-rationale representations V/z. Then, we hope V2 and U22 should be independent ideally so that the vz can contain rich rationale information. To achieve this goal, we propose the mutual information minimization method which reduces the dependency between V22 and U2a to disentangle them (i.c., min Z i Besides, following the application of MI in DRL [12, 36, 20). we should maximize [VV U2 U22) (a reconstruction loss) to ensure U2, and U22 encapsulate information from the input x sufficiently, where U. is the original input representations. However, since Us, and Vsg obtained from the separate input, we argue that Us, and U22 have preserved the original input itself, and can remove this regularization. In section 2.3, we will describe the MI minimization method in detail.
2.3 Mutual Information Minimization
In this section, first review some concepts of MI, and then present MI minimization method (i.e., CLUB_NCE) by discussing the relationships between InfoNCE and CLUB.
2.3.1 Mutual Information
In probability theory and information theory, the mutual information (MI) of two random variables (e.g., X and Y) is a measure of the mutual dependence between the two variables T(X;Y) = Ep(r.y) log pla.y) p(=)p(()] However, directly computing MI values is difficult [5], especially when X and Y are continuous and high-dimensional. To approximate MI, for MI maximization tasks, Oord et al. [32] derive a lower-bound MI estimation (InfoNCE):
N
N
N Σ
N e (an)s))
1
Ince
log
f (I v.)
log
(3)
N
N
N
N
1=1
2==1
1=1
j=1
3

---

where {(r, 11) is a batch of sample pairs and f (x, 3) is learnable score function. For MI min- imization tasks, Cheng et al. [11, 12] propose a Contrastive Log-ratio Upper Bound (CLUB) method:
N Σ
N
N
logp(uiles) N
log ((5)(1))))),
(4)
N
where p(y (z) is conditional distribution.
2.3.2 CLUB NCE
Although p(yla) in CLUB can be any neural network, intuitively, an easy way to parameterize p(y(æ) with an MLP. practical, p(y|z) is always parameterized by a Gaussian family?, where one potential reason for not employing MLP is that it is difficult to converge in CLUB. Therefore, it significantly to design a function that can parameterize p(y(z) (z) with any network and be easy to converge. To this end, we first discuss the relationship between InfoNCE and CLUB. Specifically, by applying the Jensen's 's inequality in the Eq (3), we can get:
N
N
N
N E 1=1
N
N
Tnee
1 N
f (a.
1 N
log [ (as
I (2t. U5),
N
N2
(5)
where we note this inequality is similar to Eq (4) and the main difference is the score function. Inter- estingly, we find for any critic faz y) in InfoNCE, the optimal critic is (" v) = log(p(y x)) [34] which is equivalent to the critic function in Eq (4).
Inspired by this observation, we combine CLUB
Algorithm 1 MI Minimization with CLUB NCE
with InfoNCE empirically and propose a new implementation of CLUB (i.e., CLUB _NCE) to minimize MI, which first adopts the trained f(r y) by InfoNCE to replace the log(p(v | x)) in CLUB to calculate the value of Torub and then minimizes it minimize MI.
for each training iteration do
Sample {(r,, })) N from Is (2,y). Compute the InfoNCE loss as:
exp( Ince N EN log شكك exp(fa(a;;3)))
Update the fa (x, y) by maximizing Lnce. for i 1; N +i+++ do
We illustrate the process of MI Minimization with CLUB NCE in Algorithm 1. At each train- ing iteration, we first sample { 1 from the generator Io (I, v). Then, we update the fo (I, by maximizing Ince. Next, the value of Lolub can calculated as described in Eq (5) and 9o (Ir, y) can be updated. The neural net- work fe and IJ are trained alternately. Specifically, in DARE, we set the shared encoder as the gø. and the guider which is parameterized with full connection layer as the fe-
U, So (3i, vi)
Σ== (zi, 2g))-
end for
Update the Io (x, y) by minimizing:
Lolub end for Em Uz.
2.4 Training
During training, following the setup of [4], we first penalize Lsh = - E p(m; 0|æ²)] to ensure the model extracts short rationales, where p(m, 0fæ²) is the probability of the token not being selected. Then, add regularizer to encourage coherence of selected tokens?:
n
Cre
p(m, = (1- p(mp+1 = x)) - (1 - p(m; = 0 x)) p(m5+1 = x)).
n
(6)
By combining Eq (2), Leh, Lre and the Ictub calculated by CLUB NCE between E2 and Uzy. final objective of DARE is defined as L = Ltakk + Xg Can - 1, + X + Aq where Å and Xa are fixed hyperparameters, X2 and A are Lagrangian multipliers which will be updated while training, and 4, represents that how many tokens DARE will select.
Pnttpa://githin htttge//ggtthh...m///e.aeeeee com/Linearg9/CLLB
For more details of Lsh, Lre and Hurd refer [4].

---

Table 1: Precision, Recall and Fl of selected rationales for three aspects. Among them, "% selected" represents the average proportion selected tokens in the original text.
Methods
Appearance
Smell
Palate
Precision
Recall
F1
%i selected Precision
Recall
F1
% selected Precision
Recall
F1
% selected
Bemoulli
96.3
56.5
71.2
HardKumu
98.1
65.1
78.3
13
96.8
31.5
47.5
7
89.8
48.6
63.
InfoCal IB
97.3
67.8
79.9
13
94,3
34,5
50.5
7
89.6
51.2
65.2
InfoCal(HK)
97.9
71.7
82.8
94.8
42.3
58.5
89.4
56,9
69.5
DARE (LIOut)
91.5
26.7
41.3
13
84.0
38.0
52.3
7
55.4
57.0
56.2
DARE (CLUB)
93.7
73.0
82.1
13
90.9
42.9
58.3
7
88.7
54.3
67.4
DARE
95.1
73.5
82,9
88.6
46.8
61.2
7
85.6
59.0
69.9
(std)
+0.2
+0.3
+0.1
+0.8
+0.6
-0.6
+116
+0.5
+0.2
3 Experiments
In this section, we first compare DARE with some baselines on beer reviews dataset, movie reviews dataset, and a legal judgment prediction dataset. Then, to demonstrate the effectiveness of L_BB_CE, we compare it with some classical MI minimization methods on both simulation studies and real-world datasets.
3.1 Beer Reviews Prediction
Beer Reviews Prediction is formulated as multi-aspect sentiment analysis task, which predicts the ratings (on a scale of stars) for multiple aspects (e.g., smell, palate) based on the beer reviews. We use the BeerAdvocate [29] dataset containing more than 220,000 beer reviews as our dataset. For the fair comparison, we replicate the pre-process [24] and [37] where the dataset has been divided into three aspects including appearance, smell, and palate. We normalize the ratings to [0, 1] and adopt them as the ground truth for the regression. For testing, we take 994 reviews for three aspects our test set, where the aspect-related rationales are annotated by human.
3.1.1 Comparison Methods and Experimental Setup
Bernoulli [24] generates rationales by yielding the Bernouli distribution of each token.
HardKuma [4] proposes HardKuma distribution for reparameterized gradient estimates while selecting rationale tokens.
Info [37] introduces an adversarial-basee method to guide the selector-predictor model to extract
rationales. Since InfoCal proposed new LM regularizer to encouraging the consecutiveness of rationales, for a fair comparison, we adopt one of the versions of InfoCal (i.e., InfoCal (HK*) ' in [37]), which applies the same regularizers Lih. and Eye ensure the rationales are short and coherence.
InfoCal IB [37] adopts an Information Bottleneck regularizer to manage the trade-off between yielding short rationales and accurate results by removing the adversarial module in InfoCal.
We replicate the setup of [37] where the architecture selector and encoder are RCNN and the loss function is the rqean-s error (MSE) loss. Besides, we adopt the precision, recall, and F1-score to evaluate the performance of the selected rationales. Among them, the precision defined as how many the selected tokens are in annotated rationales, and the recall represents how many annotated rationale tokens are selected. For a fair comparison, we set the 4 as (0.13, 0.07, 0.07) in three aspect datasets, respectively, and initialize the epoch as 50, which are both consistent with the previous methods [4, 37]. Several results baselines in Table are directly taken from [37].
3.1.2 Experimental Results
To demonstrate the effectiveness of DARE, we compare it with baselines the beer review prediction task, and experimental results are shown in Table 1. From the results, we can find DARE achieves better performance on the three aspects of BeerAdvocat datasets, especially recall and F1 scores, which demonstrates that DARE can select more comprehensive rationales. Specifically, compared with the traditional significantly outperforms them, indicating the effectiveness of our method on utilizing the information
models (i.c.. Bernoulli, HardKuma and InfoCal IB), DARE
"HK represents adopting the regularizers (i.e., Lsh and Cre) in HardKuma.
5

---

of non-roationn ratr-aatioal tokens, where this type tokens are ignored in previous models. As the selector and predictor in DARE are based on HardKuma, we can consider HardKuma is variant of DARE, which removes the disentanglement operation. Our model performs better than it, we can conclude that our disentanglement augmented method with MI minimization is effective.
Besides, we analyze the experimental results between DARE and InfoCal(HK) which is also a "guided" model but with an external guider (i.e.. Figure 1(b)). DARE does not outperform the IHH) on all metrics and some results are close (e.g.. Fl in Palate prediction). This is probably because InFoCal(HK) exploits the strength of the external guider, a black-box neural network that can yield more accurate but uninterpretable representations. However, perform as well as our model, which demonstrates the effectiveness of squeezing the non-rationale information from imput. We also analyze the effectiveness of CLUB NCE by comparing other MI minimization methods (e.g.. CLUB [11] and 1.1Out [34)), and the results are shown in section 3.4.4.
whole, InfoCal(HK) does not
In the section 2.2.3, we argue that the separable tokens have preserved the information of the original input and DARE does need the reconstruction loss. To validate this argument, we implement DARE with the reconstruction loss by maximizing / ((22 U2r U2y). Among them, the token Fl scores are 82.6, 61.4 and 69.8 for appearance, smell and palate, respectively, and so far the results are similar to DARE. Although DARE with reconstruction loss improves on some metrics over DARE, the improvement very limited. moreover, DARE with reconstruction loss increases the parameters of the model. Therefore, we consider that the reconstruction loss is not necessary.
Since RCNN is used as the selector-predictn architecture, to illustrate DARE is agnostic of the model structure, we implement HardKuma and DARE with pretrained models (i.e., adopting BigBird [47] to replace RCNN) the appearance aspect. The token F1 of HardKuma is 84.4 and DARE is 87.5, where DARE still outperforms HardKuma, demonstrating the effectiveness of the DARE structure.
3.2 Movie Reviews Prediction
Besides the BeerAdvocate dataset, we also make experi- ments on a movic review dataset [48] of the ERASER bench- mark [15], which also contains token-level human rationale annotations. The movie reviews prediction task is formulated as binary classification, yielding the sentiment of movie re- views (i.e., positive or negative). Table 2 gives results on this task. From the observation, we find DARE achieves the best performance on the three metrics, which further validates the effectiveness of DARE.
Table 2: Results on movie review.
Methods
Movie Recall
Precision
Bernoulli HardKuma InfoCal(HK)
35.6
26.0
28.0
31.1
28.3
27.0
36.7
30.3
33.2
DARE (std)
30.7
36,6
33,4
#3.1
#2.1
+0.7
3.3 Legal Judgment Prediction
Legal Judgment Prediction (LJP) can be formulated as text classificatinn problem, which adopts the case fact to yield the judgment results (i.e., charge, law articles, and terms of penalty). We conduct our experiments on publicly available datasets of the Chinese AI and Law challenge CAIL2018 [41]. CAIL2018 contains criminal cases which consists of fact description and corresponding charges, law articles, and terms of penalty results. Following the data process of [46), we divide the terms into non- overlapping intervals. Then, the LJP task can be formulated as multi-class classification problem.
In addition to comparing the methods in the rationale extraction, we also compare with some classical baselines in LJP task including FLA [27], TopJudge [51], LADAN [42] and NeurJudge [46]. The above baselines are trained with multi-task framework which exploits the dependence among the sub-tasks in LJP. Besides, we conduct experiments on one of versions of CAIL2018 (denoted by CAIL) which contains 134,739 cases [46]. We set the 1, as (0.14, 0.14, 0. 0.14) for training. For evaluating, as the CAIL dataset does not contain annotations of rationales, we first adopt the accuracy (Acc), macro-precision (MP), macro-recall (MR), and macro-F1 (F1) to evaluate the performance of yield judgment results. Then, we provide a qualitative analysis on the rationale extraction. Detailed description of comparison methods and experimental setups can be found in Appendix A.
3.3.1 Experimental Results
To evaluate the performance of our model on LJP, we show the experimental results from three aspects. First, comparing with the traditional rationale extraction methods, can find DARE performs well

---

Table 3: Judgment prediction results on CAIL. Therein, the underlined scores are the state-of-thee thetth--rt performances in LJP but lacking explainabilityy and the results in bold performs second only to NeurJudge but with explainability. Results of LJP baselines in Table 3 are quoted from [46].
Methods
Charges
Law Articles
Terms of Penalty
Acc
MP
MR
F1
Acc
MP
MR
F1
Acc
MP
MR
FLA
84.72
83.71
73.75
75.04
85.63
83,46
73.83
74.92
35.04
33.91
27.14
24.79
TopJudge
86.48
84.23
78.39
80.15
87.28
85,81
76.25
78.24
38.43
35.67
32.15
31.31
LADAN
88.28
86.36
80.54
82.11
88.78
85,15
79.45
80.97
38.13
34,04
31.22
30.20
NeurJudge
88.89
86.96
85.42
85.73
89.71
86.68
83.92
84.97
41.03
39.52
36.82
36.35
Bernoulli
85.76
82.71
78.04
79.38
86.45
81.94
75.92
77.35
38. 19
33.05
32. 10
30.32
HardKuma
86.26
86.22
77.83
80.04
85.76
80.92
76.85
77.96
35.94
35.26
27.52
26.09
InfoCal IB
87.14
84.86
80.70
82.11
87.58
83.55
78.49
80.13
36. 60
33.12
28. 20
26.82
(HK))
87.85
86,79
82.53
83.87
87.65
84.88
80.09
81.67
38.69
35.68
31.61
31.21
DARE (L1Out)
83.20
80.92
73.74
75.10
85 89
82.76
76.54
78.
24.09
11.76
9.14
3.72
DARE (CLUB)
82.87
83.19
73.17
74.84
86.48
84.14
77.05
78.83
37.45
33.41
28.99
27.88
DARE
88.29
86,58
83,29
84.36
88.51
85.62
81.35
82.66
38.43
36.77
33.25
30.65
(std)
+0.20
+0.37
+0.34
+0.31
+0.28
+0.40
+0.31
+0.38
-0.22
+0.17
+0.41
-0.27
HardKuma
The People's Procussorate alleped that defondast Toway public proeecution agency
fancllly somel relationss with die victie the roow foote regard, staterserrs oxptanations
tollowing endense: cugiure hitwyy Mendgjtd that defendast Tomny used vislence and verhel (Broats others jaroblly hove sexual rolatiour with the strate bearing, count
noom
The People's Procuestorate allopod Gua the deferdant Tomay and Bob farcilly bad scaual relations with the violle the room of Clotel In this requed, the
InfoCai(II))
public prosecution agreey cited the fnllwwing cordene coptun: festery and dntemeres ongierations of the defenddnnt After heaning our count itencifeed that the defendant Tommy waol rielonce and voctall trreet 2ith oden forcibli bax scunal ediatios =iil the thatie the Jotel room
rictia room of cChotel this regard, following exieesee: coptire histoyy -.. miteeeerss exglazations defendunt hearing, our count Mentiiee tuat defendant Tommyy soed violence and vanhal dereats with others to forcilly hav sexual relutions
The People's Procusiorate alleped Shat defendast Tounty and Bob foecilly Aad pouse! refations
DARE
public prosecution agency
victiee bottl room
Figure Visualized selective rationales with different methods. The underlined tokens repre- sent the real rationales which is used to support the practical charge (i.e., rape), and the blue is the predicted rationales. More examples can be found in Appendix B.
on LJP tasks, which further validates the effectiveness of our proposed model based on the previous beer reviews prediction experimental results. Meanwhile, the analysis of the comparison between CLUB and other methods of minimizing MI are shown in section 3.4.4.
Then, by comparing with methods in LJP (c.g.. NeurJudge). we observe DARE still achieves good overall performance, second only to NeurJudge (the sfate-of-the-art model in LJP) in most metrics. It is worth noting that these LJP methods are modeled based on the multi-task learning, where the charge, article and term of penalty prediction tasks are related and can enhance each other while DARE is a single-task model (i.e.. we need to train separate model for each task). It demonstrates that DARE can extract effective rationale tokens for prediction. Besides, although these LJP methods achieve promising performance, the judgment results still remain unreliable and difficult to explain. In contrast, DARE can achieve trade-off between the accuracy and explainability.
As there exist no annotations of rationales in CAIL dataset, we provide a qualitative analy- sis on rationales extracted by DARE. First, Fig- ure 3 shows an example of the extracted rationale for the charge prediction with different methods, where the underlined tokens represent the practi- cal rationales annotated by human judge. From the observation, DARE can select key rationales which could be adopted to support the prediction results. Besides, comparing with other extracted performance of basclines, DARE can seleet more rationale tokens, which indicates the strength of utilizing the non-rationale representations.
-13
(6) HarnKuuu
(b) DARE
Figure Disentanglement visualizations of ra- tionale representations.
Then, to verify that DARE has effectively disentangled the textual input into the rationale represen- tations and the non-rationale ones, we visualize these representations spaces using t-SNE [40] in Figure 4. Specifically, we visualize the same case (in Figure 3) predicted with HardKuma and DARE. Figure 4(a) shows the visualization of HardKuma. Among them, the blue circle selected represents the tokens that are predicted correctly by HardKuma; and the red circle selected denotes the tokens that are predicted incorrectly, and the position of the red circle is very close to the blue circle. For Figure 4(b), the blue circle selected represents the tokens that are predicted correctly by DARE; and
7

---

the red circle selected denotes the tokens that are predicted incorrectly by HardKuma, where our DARE identify that these tokens are non-rationale tokens. From Figure 4(b), we can find the tokens predicted incorrectly by HardKuma are accurately predicted to be non-rationale tokens. It can be seen that after the disentanglement operation, these tokens can be obviously separated and be farther away from the tokens circled in blue. Besides, we calculate the Euclidean(EU)) distance between the blue circle and the red circle in Figure 4(a) and 4(b) for measuring their distance. Among them, the EU distance in Figure 4(a) is 7.62. and in Figure 4(b) is 12.18 which is higher than it in Figure 4(a). The above visualizations demonstrate that DARE can well disentangle rationale representations.
Furthermore, we add human evaluation to further evaluate extracted rationales in LJP. We sampled 100 examples for the charge prediction task by comparing DARE with HardKuma and (HK). Specifically, following [37], we evaluate the rationales with three metrics: usefulness (U), completeness (C), and fluency (F). Among them, each scored from (lowest) to (e.g, 2.0 and 4.9). The scoring standard for human annotators can be found in Appendix A.3. Besides, we ask people to evaluate the extracted rationales and the results are report in the table. From the observation on Table 4, we can find DARE outperforms Kard and
Table 4: Human evaluation on
charge prediction.
Charge
Methods
C
F
HardKuma InfoCal(HK)
3.95
3.30
3,45
4.45
3.78
4.25
DARE
4.58
3.92
4.32
InfoCal(HK) in all metries, illustrating the effectiveness of DARE.
3.4 MI Estimation Quality
In this section, we first evaluate CLUB_NCE on simulation studies with two toy tasks. Then, we show the performance of CLUB __CE and other MI minimization methods on two real-world applications which described previously.
3.4.1 Simulated Studies
Here, introduce the experimental simulated datasets and comparison methods. First, following the setup from [34, 11], we experiment a dataset where x and y are sampled from a correlated Gaussian distribution: ~ N (0, Ia). y N (pz, (1 p²) fa). where p is the correlation and d is the dimensionality. In the first toy task (denoted by Gaussian), we sample each dimension of (z:; y) from this simulated dataset, and the second task (Cubic) is the same as the first while applying the transformation y + y² (i.e., sample (x:y²) from the same Gaussian distribution). Meanwhile, the value of this mutual information is invariant (i.e., (ar, y) = [a,, y²) [22]. Under Gaussian distributions, the ground truth of MI is tractable and can be calculated (ax, y)
log (1 (1-p²) [34]. For both of tasks, we set the dimension to d = 20, hidden size to 128 and batch size to 64. Besides, set the initial MI value and increase by 2 every 4k iterations, for total of 20k iterations.
Then, we present the comparison methods including InfoNCE [32], CLUB [11], VarUB [1]. and LIOut [34]. Among them, CLUB and InfoNCE have been described in section 1.3.1, and VarUB (uaruu Ep(e.)) log p(ulæ) r()) ) and L1Out (Itiout = Σ> log - Ezu: P(,. are both
.N
p(us las))
MI minimization methods which are adopted to estimate the upper bound of MI, where p(y|æ) is commonly parameterized by a Gaussian family [11] and r(y) is fixed as standard normal distribution.
3.4.2 Simulated Studies
Here, we introduce the experimental simulated datasets and comparison methods. First, following
the setup from Poole et al. [34] and Cheng et al. [11], we experiment a dataset where x and
are sampled from a correlated Gaussian distribution: x A (0, Ta), y ~ N (pz. (1 p²) Ta).
where p is the correlation and d is the dimensionality. In the first toy task (denoted by Gaussian). we sample each dimension of (x; from this simulated dataset, and the second task (Cubic) is
the same as the first while applying the transformation y - yd (i.e., sample (z; y²r from the
same Gaussian distribution). Meanwhile, the value of this mutual information is invariant (i.e.,
[(x, v) = [xx, )²)) [22]. Under Gaussian distributions, the ground truth of MI is tractable and can be calculated [(x, 2) log (1 - p²) [34]. For both of the tasks, we set the dimension to d 20.
8

---

1200
Figure 5: Performance of MI estimation approaches the Gaussian task. Among them, the true MI values are the black line. The estimated values are drawn as light shadow curves while the dark shadow ones represent the local averages of estimated values, with bandwidth 200.
hidden size to 128 and batch size 64. Besides, set the initial MI value as 2 and increase it by every 4k iterations, for a total of 20k iterations.
Then, we present the comparison methods including InfoNCE [32], CLUB [11], VarUB [1], and LIOut [34]. Among them, CLUB and InfoNCE have been described in section 2.3.1, and VarUB
(Faarub Bp(e..) log p(ylz) r(y) ) and 1.10ut (Jtouu N
.V MI minimization methods which are adopted to estimate the upper bound of MI, where p(y|æ) (z) is
E>= log
p(u (m:)
both
x T بوي p(s))1))
commonly parameterized by Gaussian family [11] and r(y) is fixed as standard normal distribution.
3.4.3 Experimental Results on Simulated Studies
In this section, we mainly report the experimental anal- ysis on the Gaussian task, the analysis on Cubic is sim- ilar and can be found in Appendix B.2. Figure 5 shows the performance of our CLUB CLUB_NCE and other MI es- timation approaches. Among them, the joint critic is single Multilayer Perceptron (MLP) and takes [x; y] as the input. where "; represents the concatenate opera-
50 Ya
HI Valee the sqgeeaooooci
10:heeentcom ente
Figure 6: Variance of the CLUB NCE on the Gaussian with different batch sizes in the range (8, 16, 32, 64, 128).
tion, and the separable critic is f(x, y) = 91 (x)² 92 (v), where I1 and 91 are two different MLPs. The gaus- sian critic is parameterized by a Gaussian family (i.e., N (y p(z), o (x:) I)). where the mean and variance are calculated by neural networks. We can observe that the InfoNCE values are under the true MI while CLUB __NE are above it, which illustrates CLUB_NC _CE is an upper bound of MI. The estimation of VarUB has low value, indicating this method could not remain an upper bound in some cases. The CLUB and L1Out has high bias and variance when the ground truth MI high, whereas CLUB CLUB_NCE NCE has the lower ones, especially the variance. It might have benefited from the strength of InfoNCE which has low variance.
To further validate the influence of InfoNCE on CLUB NCE, we first analyze the relationship between them theoretically, and then conduct a comparison among the CLUB NCE with different batch sizes. Specifically, we review the InfoNCE shown Eq (3):
N
(( log
N log
(arcs)))
Tnoe
1=1 23=1 e((on)))) + log N,
N
IEN N N1 e
N
1=1
where the first term is less than O and Tnee is upper bounded by log N [32]. Therefore, the performance of InfoNCE is influenced by batch sizes, where the larger the batch size N, the lower variance between the estimated MI and the true one [34]. In other words, since CLUB _CE adopts the trained critic by InfoNCE (Algorithm 1), the value of Letub should have lower variance as the batch size increases. Therefore, we conduct an experiment to estimate CLUB CLUB_NCE II by adopting various batch sizes. As shown in Figure 6, we calculate the variance between the estimated MI values Ietab and the theoretical MI ones on the Gaussian task with different batch sizes. From the observation, that CLUB NCE with larger batch size achieves the lower variance, which supports the previous theoretical analysis.
conclude
3.4.4 Real-world Applications
We compare CLUB_NCE NCE with CLUB, L1Out and VarUB on beer reviews prediction and legal judgment prediction. Specifically, we replace CLUB NCE in DARE with other MI minimization

---

methods, and denote them as DARE (CLUB), DARE (LlOut) and DARE (VarUB). The experimental results are reported in Table and Table Among them, as DARE (VarUB) fails to converge within epochs in the experimental setup, we do not present its result in the table. From the observation, DARE performs the best which indicates our CLUB B_CCEEECCCCCC is more effective than others methods at minimizing MI on real-world applications. Besides, these variants of DARE fail to improve on some tasks (e.g.. DARE (LlOut) in Palate prediction and Terms of Penalty prediction), and are even less effective than other selector-predicto models, illustrating it is crucial for DARE to choose superior MI estimator to effectively minimize the MI.
4 Related Work
Rationale Extraction.
Rationale extraction methods can provide the model explainability by extract-
ing important features of inputs and many researchers [24, 4, 44, 8, 9, 39, 45] have been attracted. Among them, Lei et al. [24] proposed classical framework for rationales extraction with a selector and predictor. Following this framework, Bastings et al. [4] introduced a HardKuma distribution for reparameterized gradient estimates and facilitated end-to-end differentiability of this framework. Paranjape et al. [33] and Chen et al. [10] managed trade-off between extracting sparse rationales and generating accurate results by the information bottleneck theory. Since this type of framework failed to exploit the information of the original text, several guider methods [37, 7, 19] have been proposed, which adopted the whole text to guide the rationale extraction. Therein, Sha et al. [37] introduced an adversarial-baat
technique to make the selector-pre -erectoo-preector model learn from the guider model. Cao et al. [7] minimized the KI. divergence hetween the two types models to reduce their difference. Although extensive research has been carried out on the rationale extraction, few considered the information of non-rationale tokens containing rich information for extracting rationales.
Disentangled Representation Learning. Disentangled representation learning (DRL) [6. 18, 16, 3] maps different aspects of data into independent latent vectors by adding constraints on the embedding space and has attracted much attention. Recently, information theory has been widely adopted in the applications of DRL. Among them, Sanchez et al. [36], Cheng et al. [12] and Huang et al. [20] maximized MI between the original input and its different aspects to retain as much as possible information of the input. And Cheng et al.[11] ] and Colombo al. [14] reduced the dependency among the different aspects of the input to make them independent by minimizing the MI.
Mutual Information. Mutual information (MI) is adopted measure the dependency between random variables. Since computing the values of MI is challenging, and many researches utilized the MI regularizer in loss functions, and not concerned with its precise value, recent works focused on MI maximization and minimization. Nguyen et al. [31] studied a lower bound of MI based on the MI f-divergence representation for maximizing MI. Belghazi al. [5] proposed mutual information estimator (MINE) which estimated the lower bound of MI. Similarly, InfoNCE was introduced by Oord al. [32], which adopted the negative sampling estimation to maximize MI. For MI minimization, Poole et al. [34] studied an upper bound of MI estimator LlOut based on reviewing the existing variational bounds on MI. And a contrastive log-ratio upper bound (CLUB) was proposed by [12. 11] to minimize MI. Besides, based on the KL. divergence and Renyi divergence, Colombo et al. [14] derived a MI estimator on the basis of a variational upper bound.
5 Conclusion
In this paper, we proposed diisenaagglement-aggmeene squeezed more information from the original input by exploiting the non-rationale tokens. To be specific, DARE disentangled the textual input into two parts (i.e.. the rationale representations and the non-rationale ones), and minimized the mutual information (MI) between the two disentanglements for extracting more comprehensive rationales. Besides, to improve the performance of minimizing MI, we introduced a new MI minimization method CLUB __NE based on exploring the relationship between InfoNCE and CLUB. Experimental results three real-world datasets and a simulated dataset demonstrated the effectiveness of our proposed method.
rationale extraction method (DARE) which
Acknowledgements. This research was partially supported by grants from the National Key Research and Development Program of China (No.2021 YFF0900003) the National Natural Science Foundation of China (Grants No. 69.611 and U20A20229).
10

---

References
[1] Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. Deep variational information bottleneck. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
[2] Yanqing An, Qi Liu, Han Wu, Kai Zhang, Linan Yue, Mingyue Cheng, Hongke Zhao, and Enhong Chen. Lawyerpan: a proficiency assessment network for trial lawyers. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 5-13, 2021.
[3] Cher Bass, Mariana da Silva, Carole Sudre, Petru-Daniel Tudosiu, Stephen Smith, and Emma Robinson. Icam: Interpretable classification via disentangled representations and feature attribution mapping. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[4] Jasmijn Bastings, Wilker Aziz, and Ivan Titov. Interpretable neural predictions with differ- entiable binary variables. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), July 2019.
[5] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International Conference Machine Learning, pages 531-540. PMLR, 2018.
[6] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8): 1798-1828, 2013.
[7] Nicola De Cao, Michael Sejr Schlichtkrull, Wilker Aziz, and Ivan Titov. How do decisions emerge across layers in neural models? interpretation with differentiable masking. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, 2020.
[8] Shiyu Chang, Yang Zhang, Mo Yu, and Tommi S. Jaakkola. A game theoretic approach to class-wise selective rationalization. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, 2019.
[9] Shiyu Chang, Yang Zhang, Mo Yu, and Tommi S. Jaakkola. Invariant rationalization. In Proceedings of the 37th International Conference on Machine Learning, (ICML), 2020.
[10] Hanjie Chen and Yangfeng Ji. Learning variational word masks to improve the interpretability of neural text classifiers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 4236 4251, 2020.
[11] Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. Club: contrastive log-ratio upper bound of mutual information. In International Conference on Machine Learning (ICML), pages 1779-1788. PMLR, 2020.
[12] Pengyu Cheng, Martin Renqiang Min, Dinghan Shen, Christopher Malon, Yizhe Zhang, Yitong Li, and Lawrence Carin. Improving disentangled text representation learning with information- theoretic guidance. In Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics (ACI.), 2020.
[13] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using encoder- decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724-1734, 2014.
[14] Pierre Colombo, Pablo Piantanida, and Chloé Clavel. A novel estimator of mutual information for learning to disentangle textual representations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/JCNLP 2021, Virtual Event, August 1-6, 2021, 2021.
11

---

[15] Jay De Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. ERASER: A benchmark to evaluate rationalized NLP models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACI. 2020, Online, July 5-10, 2020, 2020.
[16] Shuyang Gao, Rob Brekelmans, Greg Ver Steeg, and Aram Galstyan. Auto-encoding total correlation explanation. In The 22nd International Conference on Artificial Intelligence and Statistics. pages 1 1157-1166. PMLR, 2019.
[17] Junliang Guo, Zhirui Zhang. Linli Xu, Hao-Ran Wei, Boxing Chen, and Enhong Chen. Incorpo- rating BERT into parallel sequence decoding with adapters. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information P Processing Systems 2020, NeurlPS, 2020.
[18] Irina Higgins, David Amos, David Pfau, Sébastien Racanière, Loic Matthey, Danilo J. Rezende, and Alexander Lerchner. Towards definition of disentangled representations. CoRR, 2018.
[19] Yongfeng Huang. Yujun Chen, Yulun Du, and Zhilin Yang. Distribution matching for rational- ization. Thirty-Fifth AAAI Conference on Artificial Intelligence, (AAAI), 2021.
[20] Zhenya Huang, Xin Lin, Hao Wang, Qi Liu, Enhong Chen, Jianhui Ma, Yu Su, and Wei Tong. Disenqnet: Disentangled representation learning for educational questions. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery Data Mining, pages 696-704, 2021.
[21] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference Learning Representations (ICIR), 2014.
[22] Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical review E, 69(6))066138, 2004.
[23] Ponnambalam Kumaraswamy. A generalized probability density function for double-bounded random processes. Journal of hydrology, -6(1-2)779-88, 1980.
[24] Tao Lei, Regina Barzilay, and Tommi Jaakkola. Rationalizing neural predictions. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2016.
[25] Xiaoya Li, Jingrong Feng, Yuxian Meng, Qinghong Han, Fei Wu, and Jiwei Li. A unified MRC framework for named entity recognition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 5849-5859, 2020.
[26] Yi-Shan Lin, Wen-Chuan Lee, and Z Berkay Celik. What do you see? evaluation of explainable artificial intelligence (xai) interpretability through neural backdoors. Proceedings of the 27th ACM SIGKDD Conference Knowledge Discovery & Data Mining, 2021.
[27] Bingfeng Luo, Yansong Feng. Jianbo Xu, Xiang Zhang, and Dongyan Zhao. Learning to predict charges for criminal cases with legal basis. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2017.
[28] Siwen Luo, Hamish Ivison, Caren Han, and Josiah Poon. Local interpretations for explainable natural language processing: survey. arXiv preprint arXiv:2103.11072, 2021.
[29] Julian J. McAuley, Jure Leskovec, and Dan Jurafsky. Learning attitudes and attributes from multi-aspect reviews. In 12th IEEE International Conference on Data Mining, ICDM 2012, Brussels, Belgium, December 10-13, 2012.
[30] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111-3119 2013.
[31] Xuani Long Nguyen, Martin J Wainwright, and Michael I Jordan. Estimating divergence func- tionals and the likelihood ratio by convex risk minimization. IEEE Trunsactions on Information Theory, 56 (6(11))5847-588 2010.
12

---

[32] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 18Xv:18877.0378. 2018.
[33] Bhargavi Paranjape, Mandar Joshi, John Thickstun, Hannaneh Hajishirzi, and Luke Zettlemoyer. An information bottleneck approach for controlling conciseness in rationale extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), November 2020.
[34] Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning (ICML), pages 5171-5180. PMLR, 2019.
[35] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 2016.
[36] Eduardo Hugo Sanchez, Mathieu Serrurier, and Mathias Ortner. Learning disentangled represen- tations via mutual information estimation. In Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXII, 2020.
[37] Lei Sha, Oana-Maria Camburu, and Thomas Lukasiewicz. Learning from the best: Rationalizing prediction by adversarial information calibration. In Proceedings of the 35th AAAI Conference on Artificial Intelligence (AAAI), 2021.
[38] Maosong Sun, Xinxiong Chen, Kaixu Zhang, Zhipeng Guo, and Zhiyuan Liu. Thulac: An efficient lexical analyzer for chinese, 2016.
[39] Marcos V. Treviso and André F. T. Martins. The explanation game: Towards prediction explainability through sparse communication. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, 2020.
[40] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11). 2008
[41] Chaojun Xiao, Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Zhiyuan Liu, Maosong Sun, Yansong Feng, Xianpei Han, Zhen Hu, Heng Wang. et al. Cail2018: A large-scale legal dataset for judgment prediction. In arXiv preprint
2018.
[42] Nuo Xu, Pinghui Wang, Long Chen, Pan, Xiaoyan Wang, and Junzhou Zhao. Distinguish confusing law articles for legal judgment prediction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3086-3095. Association for Computational Linguistics, 2020.
[43] Adams Wei Yu, David Dohan, Minh-Thhng Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoe Le. Qanet: Combining local convolution with global self-attention for reading comprehension. In 6th International Conference Learning Representations, ICLR 2018, 2018.
[44] Mo Yu, Shiyu Chang, Yang Zhang, and Tommi S Jaakkola. Rethinking cooperative rational- ization: Introspective extraction and complement control. In Empirical Methods in Natural Language Processing (EMNLP), 2019.
[45] Mo Yu, Yang Zhang, Shiyu Chang, and Tommi Jaakkola. Understanding interlocking dynamies of cooperative rationalization. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
[46] Linan Yue, Qi Liu, Binbin Jin, Han Wu, Kai Zhang, Yanqing An, Mingyue Cheng, Biao Yin, and Dayong Wu. Neurjudge: A circumstance-aware neural framework for legal judgment prediction. In Proceedings of the 44th International ACM SIGIR Conference Research and Development in Information Retrieval, 2021.
[47] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 33, 2020.
13

---

[48] Omar Zaidan and Jason Eisner. Modeling annotators: A generative approach to learning from annotator rationales. Proceedings of the 2008 conference on Empirical methods in natural language processing, pages 31-40, 2008
[49] Kai Zhang, Qi Liu, Zhenya Huang, Mingyue Cheng, Kun Zhang, Mengdi Zhang, Wei Wu, and Enhong Chen. Graph adaptive semantic transfer for cross-domain sentiment classification. 2022.
[50] Kai Zhang, Kun Zhang, Mengdi Zhang, Hongke Zhao, Qi Liu, Wei Wu, and Enhong Chen. Incorporating dynamic semantics into pre-trained language model for aspect-based sentiment analysis. In Findings of the Association for Computational Linguistics: ACL 2022, May 2022.
[51] Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Chaojun Xiao, Zhiyuan Liu, and Maosong Sun. Legal judgment prediction via topological learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We discuss the limitations in Appendix C.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We discuss the broader impacts in Appendix C.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Please see Section 2.
(b) Did you include complete proofs of all theoretical results? [Yes] Please see Section 2.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] Please see Table 1, Table 2 and Table
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We run all experiments on single V100 GPU.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] Although the original BeerAdvocate has been removed by the authors of this dataset, we can obtain BeerAdvocate from http://people, csail.mis wwt.:mmittttee...eemmmmmmmmm wdu.taaleiibeer// which published by [24]. CAIL.2018 has been published in https://ggt thwbttggttttbttt/e com/ China-ai-lav-da www.--a--ccaallnn//AAALLL-.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 'rure using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifi- able information or offensive content? [Yes] To our best knowledge, their exists no personally identifiable information in BeerAdvocate, MovicReview and CAIL.
14

---

5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "ij1CkahNmMW", "reviewer": "Reviewer_V57r", "summary": "This paper focuses on the problem of rationale extraction for classification tasks. The proposed method addresses a hypothesized limitation in prior work, namely failing to fully exploit the original input to represent selected rationale tokens. The proposed method jointly optimizes the classification objective and a new objective which explicitly minimizes the mutual information between the representations of the selected rationales and the ones from the rest of the input to extract more information out of it.  Results on two rationale extraction tasks show improvements over existing rationale extraction methods. The developed mutual information objective is shown to be better than alternative ones on simulated experiments. \n", "questions": "The point that existing methods ignore information from the non-selected tokens in the abstract is not entirely clear at first. Since bidirectional representations are typically used to represent the tokens this statement creates confusion. It would be helpful to clarify early on that the disentanglement goal essentially aims to encourage dependence among tokens of the same type and independence for tokens of different types  (e.g. rationale/non-rationale).\n\nRegarding the statement in lines 201-204: is the score reported with HardKuma using the paper's implementation or it is reported from prior work (line 189)? To understand the differences it is also important to know what is the number of parameters for each of them.\n\nThe evaluation is missing details on how the model and training objective hyper-parameter optimization was performed exactly. What values were examined for each hyper-parameter and what is their effect? Also, it is unclear how some design choices were made (shared encoder, MI minimization that depends on trained InfoNCE).   \n\nIn Table 2, was there a reason why an experiment using plain InfoNCE with DARE was not included? It'd be useful to report performance with this objective for completeness and for clearly demonstrating the added value of the selected one. \n\nThe evaluation does not provide sufficient evidence about the quality of the extracted rationales for the second dataset since it is based on a handful of examples and t-sne. A systematic human evaluation study would be required to measure quality.  \n\nIn Eq. 6, could you elaborate on how is the coherence encouraged by this regularizer? Any examples of what happens if this is not enforced would be most helpful. ", "limitations": "Yes. There is a section in the Appendix. Limitations are not sufficiently investigated and discussed (see above). ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths** \n- The narrative/structure of the paper was easy to follow and presentation was generally good. \n- Proposes an approach for rational extraction which combines ideas from existing selection methods and objectives to minimize mutual information between selected rationales and other parts of the input. \n- Presents results that show classification moderate improvements over previous rationale extraction methods on two  classification tasks.  \n\n**Weaknesses** \n- Even though writing was fair in most places, some of the model design and evaluation choices lacked clarity and proper motivation/justification.  \n- The proposed objective to minimize mutual information appears to be incremental and the comparison between alternatives incomplete (e.g. lack of comparison with plain InfoNCE on extraction task). \n- Several design choices are not well justified and the description lacks clarity in a few places (e.g. shared encoder/using trained InfoNCE in section 2.3, coherence regularizer, objective coefficients section 2.4). There are no ablation studies that demonstrate the source of improvement or contribution from each individual objective to the final performance. \n- The evaluation is lacking details for understanding the results and the logic behind the design choices. Also, the quality of the extracted rationales for the judgment dataset is based only on a handful of examples and a t-sne visualization. \n- The scope of the paper in the present form is limited to findings on two datasets and exploration beyond them would be required to be of interest to a broader audience. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "hgBZnjal4jf", "reviewer": "Reviewer_1CAu", "summary": "This work aims at proposing a rationale extraction method that disentangles the rationale and non-rationale spans. Specifically, apart from predicting the label from the rationale span, the proposed method DARE adds another main objective that minimizes the mutual information (MI) between the representations of the rationale and non-rationale spans. Modified upon previous MI minimization work InfoNCE and CLUB, the authors propose a new method CLUB_NCE and use it in DARE. Experiments on two natural language datasets, BeerAdvocate and Legal Judgment Prediction show a performance advantage of DARE over the baselines.", "questions": "Please see my concerns in the previous section. Also, RCNN is used as the selector and encoder architecture. Can pretrained transformer models be used instead? If so, would the re-embed operation in Sec 2.2.2 (line 101) lead to any issues?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper presents an overall novel and effective method for rationale extraction. The comparison between different MI minimization methods is aided by simulated experiments. The proposed method DARE is introduced rather clearly. Experiments on two natural language datasets show advantages of DARE over a few baseline methods.\n\nThe concerns that some readers might raise and complain include:\n1- At line 88 and 111, a main argument for adding a regularizer that minimizes the MI between rationale and non-rationale representations, is that it can make the rationale more “comprehensive” and “rich” information. Can this argument be supported more intuitively with a discussion? (the connection between A and B’s disentanglement and A being “comprehensive”)\n\n2- In Sec 2.3.2, CLUB_NCE is mainly CLUB with the approximation network p(y|x) replaced by the InfoNCE scoring function. While there are performance gains as shown in the experiments, the potential motivation and rationale behind this modification is not fully clear and can benefit from a further discussion.\n\n3- The experiment presentation in Sec 3.2.2 needs improvements. The description and figure for the t-SNE experiment look confusing. How were the blue and red circles selected in a formal way? Can the distance between clusters be measured? The scale of 5(a) and 5(b) is different, making it harder for a comparison.\n\n4- At line 295, the L1Out method was in previous tables but was not introduced until this stage. This should probably appear earlier in the text with more descriptions.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Kog0sOKx8L", "reviewer": "Reviewer_wZNR", "summary": "This paper aims to extraction rationales for text classification tasks. They propose to divide the sentence into two separate parts: rationale tokens and non-rationale tokens. They design a loss based on mutual information to encourage the embeddings of rationale tokens and embeddings of non-rationale tokens to be disentangled and expect more useful information can be squeezed into the rational part. Experiments and analysis are presented to support their design.", "questions": "Suggestions\n- I feel like the reconstruction loss is needed. It takes both rationale embeddings ($v_{z_1}$ in Figure 2) and non-rationale embeddings ($v_{z_2}$ in Figure 2) as the input with the goal to reconstruct the original input. This loss encourages the model to encode as much information to $v_{z_1}$ and $v_{z_2}$ as possible while your MI loss makes $v_{z_1}$ and $v_{z_2}$ disentangled. Otherwise, without this reconstruction loss, the encoder can just learns to encode $v_{z_2}$ as random vectors. Those random $v_{z_2}$ can still have low mutual information with $v_{z_1}$, which is obviously not the expected situation. In this case $v_{z_2}$ encodes nothing and your approache will reduce the common selector + predictor approach. Although the current design somehow works and the above-mentioned case does not happen that much, I still think the reconstruction loss is needed.\n- For Table 1, some of the improvements are small. Adding t-test results may help.\n\nQuestions:\n- For Table 1 and Table 2, how many runs of experiments for the mean and std? \n- What would happen if you also add LM regularizer just like the original InfoCal?", "limitations": "The authors adequately addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The writing is clear and easy to understand.\n- The design is simple and seems to be effective.\n- Comprehensive analysis and studies for the proposed model.\n\nWeaknesses\n- No loss to ensure that the information preserved by the non-rational embeddings are meaningful (see more details in the \"Questions\" section below). ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FgXG4q9mweZ", "reviewer": "Reviewer_GAz6", "summary": "This paper studies a method for rationale extraction for natural language processing (NLP), focusing on text classification tasks. The overall architecture involves a selector to select rationale tokens from an input text, a predictor to consume the selected rationale tokens to obtain a representation for predictions, and a guider to obtain a representation for the non-rationale tokens that will be regularized to minimize the mutual information (MI) with the rationale token-based representation. Compared to previous work, this paper argues that it is helpful to use information for non-rationale tokens to improve the selection of rationale tokens. In addition, to minimize mutual information between two representations, the paper presents a new approach (CLUB_NCE) that combines two existing methods of InfoNCE and CLUB. Experiments are conducted over two text classification datasets to demonstrate better prediction performance and more reasonable rationale for the method. The paper also provides a simulated experiments to compare the proposed MI minimization method with previous baselines and a visualization to demonstrate the disentangled representations for rationale and non-rationale tokens.", "questions": "Please consider the weakness presented above.", "limitations": "The paper includes a discussion on broader impact and limitations in the appendix. However, the limitation discussion is limited to only selection of the l_r parameter. It might be helpful to further consider the limitations as described in the weakness section.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength\n\n+The paper considers an important problem of rationale extraction to improve interpretability for NLP models.\n\n+The idea of using non-rationale information to improve rationale extraction is interesting and the introduced MI minimization method demonstrates good empirical results.\n\n+The proposed method achieves competitive performance on two datasets that can provide useful baselines for future research.\n\n**Weakness\n\n-While interesting, the guider component (with non-rationale tokens) and the CLUB_NCE for MI minimization seem to be incremental extension of existing methods for rationale extraction and MI estimation. The method mostly involves modification and combination of closely related techniques (e.g., InfoNCE and CLUB).\n\n-As indicated in the Appendix, the model only uses word2vec embeddings to encode texts and does not consider current pre-trained language models (e.g., BERT) that has significantly advanced different NLP tasks. The key question is whether the proposed method can still maintain its claimed benefits when applied with pre-trained language models. In addition, this important implementation information with word2vec and Bi-GRU should be presented in the main paper to make it easier to understand the methods.\n\n-Although Section 3.3 provides empirical experiments to compare different MI estimation methods, it is helpful to better motivate for the designed MI method CLUB_NCE. In particular, Section 2.3.2 mainly describes the implementation of the method. Without a discussion on limitations of previous methods and motivation to address the limitations, it is difficult to understand the benefits of the designed method.\n\n-The performance of the proposed method is very close and sometime worse than the closely related baseline InfoCal(HK), making the effectiveness of the method less convincing. It might be helpful to provide significant scores to strengthen the results.\n\n-Due to the coherence regularization, the selected rationale tokens tend to form consecutive sequences, causing remaining non-selected tokens to involve nonconsecutive words that might lead to unnatural texts. As such, the learned representations for rationale and non-rationale tokens might be quite different, making the MI minimization less reasonable. As such, is it better to obtain non-rationale representation by subtracting rationale representation from the representation for the entire input text to avoid unnatural inputs? It might be helpful to explore this approach to better justify the design choice of the model in this work. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept