Understanding Interlocking Dynamics of Cooperative Rationalization
Authors: Mo Yu
Yang Zhang
Shiyu Chang
Tommi S. Jaakkola
Venue: NeurIPS 2021 Poster (2021)
Keywords: Rationalization
interlocking
cooperative game
model explainability
natural language processing

Abstract: Selective rationalization explains the prediction of complex neural networks by finding a small subset of the input that is sufficient to predict the neural model output. The selection mechanism is commonly integrated into the model itself by specifying a two-component cascaded system consisting of a rationale generator, which makes a binary selection of the input features (which is the rationale), and a predictor, which predicts the output based only on the selected features. The components are trained jointly to optimize prediction performance. In this paper, we reveal a major problem with such cooperative rationalization paradigm --- model interlocking. Inter-locking arises when the predictor overfits to the features selected by the generator thus reinforcing the generator's selection even if the selected rationales are sub-optimal. The fundamental cause of the interlocking problem is that the rationalization objective to be minimized is concave with respect to the generator’s selection policy. We propose a new rationalization framework, called A2R, which introduces a third component into the architecture, a predictor driven by soft attention as opposed to selection. The generator now realizes both soft and hard attention over the features and these are fed into the two different predictors. While the generator still seeks to support the original predictor performance, it also minimizes a gap between the two predictors. As we will show theoretically, since the attention-based predictor exhibits a better convexity property, A2R can overcome the concavity barrier. Our experiments on two synthetic benchmarks and two real datasets demonstrate that A2R can significantly alleviate the interlock problem and find explanations that better align with human judgments.

Understanding Interlocking Dynamics of Cooperative Rationalization
Mo Yu!
Yang Zhang 'MIT-IBM Watson Al Lab
Shiyu Chang 1.2
Tommi Jaakkola CSAIL MIT
UC Santa Barbara
yumθus. ibm. com
yang. zhang2@iba com
chang870ucsb. edu
tommi @csail aettallmii......mmmm
Abstract
Selective rationalization explains the prediction of complex neural networks by finding small subset of the input that is sufficient to predict the neural model output. The selection mechanism is commonly integrated into the model itself by specifying two-component cascaded system consisting of rationale generator, which makes binary selection of the input features (which is the rationale), and predictor, which predicts the output based only the selected features. The components are trained jointly to optimize prediction performance. this paper, reveal major problem with such cooperative rationalization paradigm model interlocking. Interlocking arises when the predictor overfits to the features selected by the generator thus reinforcing the generator's selection even if the selected rationales are sub-optimal. The fundamental cause of the interlocking problem is that the rationalization objective be minimized is concave with respect the generator's 's selection policy. We propose a new rationalization framework, called A2R, which introduces third component into the architecture, a predictor driven by soft attention as opposed to selection. The generator now realizes both soft and hard attention over the features and these are fed into the different predictors. While the generator still seeks to support the original predictor performance, it also minimizes gap between the two predictors. As we will show theoretically, since the attention- based predictor exhibits better convexity property, A2R can overcome the concavity barrier. Our experiments on two synthetic benchmarks and two real datasets demonstrate that A2R can significantly alleviate the interlock problem and find explanations that better align with human judgments
Introduction
27 29 46 explains the prediction of complex neural networks by finding small subset of the input rationale that suffices on its own to yield the same outcome as the original data. To generate high-quality rationales. existing methods often train cascaded system that consists of two components, i.e., rationale generator and a predictor. The generator selects subset of the input explicitly (a.kaa,, binarized selection), which is then fed the predictor. The predictor then predicts the output based only on the subset of features selected by the generator. The rationale generator and the predictor are trained jointly optimize the prediction performance. Compared to many other interpretable methods [5] 2x] 45 38 mechanism as proxy of models' explanation, selective rationalization offers unique advantage: certification of exclusion, i.e., any unselected input is guaranteed to have no contribution to prediction.
Selective rationalization 8 10
that rely attention
Authors contributed equally to this paper. Work was done when SC was at MIT-IBM Watson Al Lab. release our code httpg////ggtttatcccoooeeeee/eeee
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

However, binarized selective rationalization schemes are notoriously hard to train B 46 To overcome training obstacles, previous works have considered using smoothed gradient estimations (e.g. gradient straight-through [9] or Gumbel softmax [2I), introducing additional components to control the complement of the selection IC 46 adopting different updating dynamics between the generator and the predictor I using rectified continuous random variables to handle the constrained
optimization in training S etc. In practice, these solutions are still insufficient. They either still
require careful tuning are cost of reduced predictive accuracy.
In this paper, we reveal a major training problem of selective rationalization that has been largely overlooked - model interlocking. Intuitively, this problem arises because the predictor only sees what the generator selects during training. and tends to overfit to the selection of the generator. As a result, even if the generator selects sub-optimal rationale, the predictor can still produce lower prediction loss when given this sub-optimal rationale than when given the optimal rationale that has never seen. As a result, the generator's ss selection of the sub-optimal rationale will be reinforced. In the end, both the rationale generator and the predictor will trapped in a sub-optimal equilibrium, which hurts both model's predictive accuracy and the quality of generated rationales.
By investigating the training objective of selective rationalization theoretically, we found that the fundamental cause of the problem of interlocking is that the rationalization objective we aim to minimize is undesirably concave with respect to the rationale generator's 's policy, which leads many sub-optimal corner solutions. On the other hand, although the attention iitention-based models (i.e., via soft selection) produce much less faithful explanations and do not have the nice property of certification of exclusion, their optimization objective has a better convexity property with respect to the attention weights under certain assumptions, and thus would not suffer from the interlocking problem.
Motivated by these observations, we propose a new rationalization framework, called A2R (attention- to-rationale), which combines the advantages of both the attention model (convexity) and binarized rationalization (faithfulness) into one. Specifically, our model consists of a generator, and two predictors. One predictor, called attention-based predictor, operates on the soft-attention, and the other predictor, called binarized predictor, operates on the binarized rationales. The attention as used by the inrention-basee predictor is tied to the rationale selection probability as used by the binarized predictor. During training, the generator aims to improve both predictors' performance while minimizing their prediction gap. As we will show theoretically, the proposed rationalization scheme can overcome the concavity of the original setup, and thus can avoid being trapped in sub- optimal rationales. In addition, during inference time, we only keep the binarized predictor to ensure the faithfulness of the generated explanations. We conduct experiments on two synthetic benchmarks and two real datasets. The results demonstrate that our model can significantly alleviate the problem of interlocking and find explanations that better align with human judgments.
2 Related Work
Selective rationalization: h proposes the first generator-pree prreditttr framework for rationalization. Following this work, new game-theoretic frameworks were proposed to encourage different desired properties of the selected rationales, such as optimized Shapley structure scores [4], comprehen-
siveness [o, multi-aspect supports 4 [ and invariance [12]. Another fundamental direction is to
overcome the training difficulties. 6 replaces policy gradient with Gumbel softmax. [H[ proposes
to first pre-train the predictor, and then perform end-to-end training. [II] adopts different updating
dynamics between the generator and the predictor. [X] replaces the Bernoulli sampling distributions with rectified continuous random variables to facilitate constrained optimization.
39
proposes to
enhance the training objective with an adversarial information calibration according to a black-box predictor. However, these methods cannot address the problem of interlocking.
35 could serve as a proxy of the rationale. Although attention is easy to obtain, it lacks faithfulness. An input associated with low attention weight can still significantly impact the prediction. In addition, recent works 38
Attention as proxy of explanation: Model's attention 5
[4] also find that the same prediction on an input could be generated by totally different attentions, which limits its applicability to explaining neural predictions. To improve the faithfulness of attentions, 33 [
7 20 35
regularize the hidden representations on which the attention is computed over; 17 applies attention weights on losses of pre-defined individual rationale candidates' predictions. Nevertheless, rationales remain to be more faithful explanations due to their certification of exclusion.

---

X17
Generator
M
O
Rationale-based Predictor
S. Lez
Rationale
Mask
Figure A conventional selective rationalization framework.
+1 so as to promote the faithfulness of their attention rationales. The interlocking problem still persists in this framework, because the loss landscape remains concave (refer to our arguments in Section (333) Specifically, since the predictor would not see the sentences that receive 0 attention weights, it tends to underfit these sentences. As result, the generator does not have the incentive to assign positive weights to the sentences that are previously assigned zero weights, thus is prone selecting the same sentences.
18 12
force the sparsity of the attention with sparsemax
Model interpretability beyond selective rationalization: There are other popular interpretability frameworks besides selective rationalization. Module networks p R [2]
compose appropriate neural modules following logical program to complete the task. Their applicability is relatively limited, due to the requirement of pre-defined modules and programs. Evaluating feature importance with gradient is another popular method. Though ? discusses several advantages of gradient-basee based methods over rationalization, they are post-hoc and cannot impose structural constraints on the explanation. Other lines of work that provide post-hoc explanations include local perturbations 25 :0:; locally fitting interpretable models [I :36]; and generating explanations in the form of edits to inputs that change model prediction to the contrast case 37
information 7
2 40
Selective Rationalization and Interlocking
In this section, will formally analyze the problem of interlocking in conventional selective rationalization frameworks. Throughout this section, upper-cased letters, i.e., A and A, represent random vectors (bolded) and random values (unbolded) respectively; lower cased letters, i.e., and represent deterministic vectors (bolded) and values (unbolded) respectively. Vectors with a colon subscript, i.e., dr.7, represent a concatenation of a ar, i.e., [a
3.1 Overview of Selective Rationalization
Consider a classification problem, (X, Y), where X = X1.r the input feature, and Y is the discrete class label. In NLP applications, X1.7 be understood of selective rationalization to identify a binary mask, M E {0, 1)", that applies to the input features to form rationale vector, Z, as explanation of Y. Formally, the rationale vector is defined as
series of I words/sentences. The goal
Z M X [M, X,,....... Mr)
Conventionally, Z is determined by maximizing the mutual information between Z and Y, i.e.,
max T(Y; M X), s.t. M E M. M
where M refers to constraint set, such as the sparsity constraint and a continuity constraint, requiring that the selected rationale should be a small and continuous subset of the input features.
One way of learning to extract the rationale under this criterion is to introduce a game-theoretic framework (see Figure [] consisting of two players, a rationale generator and a predictor. The rationale generator selects subset of input rationales and the predictor makes the prediction based only on the rationales. The two players cooperate to maximize the prediction accuracy, so the rationale generator would need select the most informative input subset.
Specifically, the rationale generator generates a probability distribution, π, for the masks, based on the input features X. Then, the mask M is randomly drawn from the distribution To simplify our exposition, we focus on the case that X, represents sentence and only one of the 7 sentences is selected as rationale. In this case, M is one-hot vector, and is multinomial distribution. Formally, the mask M generated as follows
M (X) [T (X). ........ TTTTTX (X)), where n, (X) p(M = e; XX),
and e, denotes 7 --dimensional one-hot vector, with the i-th dimension equal to one. The generaliza- tion to making multiple selections will be discussed in Section 4.1

---

After the mask is generated, the predictor, f-(:: 0.) (the subscript stands for rationale to differentiate from the attention-buus baseod predictor introduced later), then predicts the probability distribution of Y based only on Z M i.e.,
fz; = (o = 112), .(Y =c Z)],
where ý represents a predicted distribution, and 0. denotes the parameters of the predictor.
The generator and the predictor trained jointly to minimize the cross-entropy loss of the prediction:
min (T, 0r), where (T, 0.) Ex,Y [e(Y, f-M X; M-+(())
x+),0r
Der denotes the training set; E(-,-) denotes the cross entropy loss. It can be shown K3 that, if (.)
and (+ 0-) both have sufficient representation power, the globally optimal (XX) of Equation would generate masks M that are globally optimal under Equation
3.2 Interlocking: A Toy Example
Despite the nice guarantee of its global optimum solution, the rationalization framework in Equa- tion f suffers from the problem of being easily trapped into poor local minima, a problem we refer to as interlocking. To help readers understand the nature this problem, we would like to start with toy example, where the input consists of two sentences, X1 and X2. We assume that X1 is the more informative (in terms of predicting Y) sentence between the two, so the optimal solution for the rationale generator is always select X1 (i.e. m and 0).
However, assume, for some reason, that the generator is initialized so poorly that it only selects X2. and that the predictor has been trained to make the prediction based only on X,. In this case, we will show that very hard for the generator-predictte fails to converge to the globally optimal solution of selecting X1. Since the predictor underfits to X will produce large prediction error when X, is fed. As a result, the rationale generator would stick with selecting X2 because X2 yields a smaller prediction error than X1. The predictor, in turn, would keep overfitting to X₂ and underfitting to X,. In short, both players lock the other player from escaping from the poor solution, hence the name interlocking.
to escape from this poor local minimum thus
The problem interlocking can also be mani- fested by an accordance game, where the gen- erator has two strategies, select X; and select X2, and the predictor also has two strategies, overfit to X2 and overfit to X2. An example payoff table is shown in Table IT As can be seen, (select X1, overfit to X1) has the high- est payoff, and thus is the optimal solution for both players. However, (select X2, overfit to X2) also constitutes Nash equilibrium, which is locally optimal.
Table 1: An example payoff (negative loss) table of the accordance game between the generator (Gen) and the pre- dictor (Pred), where the interlocking problem is manifested as multiple Nash Equilibria.
Pred.
Overfit to X1
Overfit to X₂
Select Xi
(-1,-1) (-20, -20)
(-10. - 10)
Gen.
Select X2
(-2, -2)
3.3 Interlocking and Concave Minimization
To understand the fundamental cause of the interlocking problem, rewrite the optimization problem in Equation into a nested form:
min C,( 0.) min min L.( 0.) min (TT, O,, (Tr)). * 1,00
(5)
#())
x()
where (T))) = argmin Cr 0,).
(6)
6.
Furthermore, denote
Then, the problem of finding the optimal rationale boils down to finding the global minimum of ()(T), In order to achieve good convergence properties, (TT) would ideally be convex with respect to n. However, the following theorem states the opposite.
Theorem 1. (T)) is concave with respect to

---

Can
Desired
Lp
Interlocking
1
Unfaithful
01
TT a1
(a)
(b)
Figure 2: Example loss landscapes the two-sentence scenario. (a) An example loss landscape of rationale- based explanation (Equation (). which is concave and induces interlocking dynamics towards a sub-optimal local minimum. (b) An example loss landscape of attention-hased explanation (Equation )), which is convex but with unfaithful global minimum. (c) The two loss landscapes share common end points. Desirable landscapes should lie in between.
The proof is presented in Appendix A.I Theorem | iimpies the cooperative rationalization objective contain many local optima at the corners. Going back to the two-sentence example, Figure & a) plots an example (T)) against Since there are two sentences, = 0 implies that the generator always selects X2, and mi implies the generator always selects X As shown in the figure, since X1 is more informative than X2, the global minimum is achieved at observed that m = also local minimum, and therefore the rationalization framework can be undesirably trapped into the rationalization scheme that always selects the worse sentence of the two.
1. However, it can be
3.4 Convexity of Attention Explanation
Knowing that the selective rationalization has an undesirable concave objective, we now turn to another class of explanation scheme, i.e. attention- based explanation, which uses soft attention, rather than binary selection, of the input explanation. Specifically, would like to investigate whether its objective has a more or less desirable convexity property than that selective rationalization.
Formally, consider attention), which almost identical to the rationalization predictor Equation ( except that the binary mask M is replaced with soft attention weight (x where each dimension sums to one. So the optimization objective becomes
attention-based predictor, (xx @ X; 0a) (the subscript stands for
min (cc, 0.), where (a, 0a) = Ex,y - (((()) @ X: 0.).))
(8)
Similar to Equations
to 7 define C. (x) L(a, (a))), where (2 argmin La(a, 0.).
(9)
6,
The following theorem shows that (a) has more desirable convexity property. Theorem 2. (a) is convex with respect to a, if
1. (a, 0.) is p-strongly convex with respect to a with & distance metric, v0.:
2. (a (a(')) has a bounded regret with the optimal loss, i.e., when a' a, with & norm:
La(a,,0(aa)) Ca (x)))) [ [Ila(x) - a'x))l22 Va(+),
(10)
3. p
The proof is presented in Appendix A.2 where also discuss the feasibility of the assumptions. special case where the predictor has sufficient representation power is discussed in Appendix A.3 Figure (b) plots an example (a) against an, again under the same two-sentence toy scenario. Note that a1 = 0 means Xx gets all the weight; an = 1 means X; gets all the weights. As can be observed, C. (a) now convex function, which makes more desirable in terms of optimization. However, the example in Figure b) also shows why such attention faithful. Even though X, is better sentence of the two, the global minimum C. (.(()) is achieved at the point where X₂ gets larger weight than X; does. The reason why the global minimum is usually achieved in the interior < 01 < 1) rather than the corner (a, = 0 or 1) that the predictor would have access more information if both X₁ and X2 get non-zero attention weights.
scheme is sometimes not
5

---

X
Generator
1:T Attention Weight
Attention-based Predictor
Sa
Loe
X1T
Las
Forward < Backward
Hard Selection
Mir Rationale Mask
Rationale-based Predictor
f
Lee
Figure 3: Our proposed rationalization architecture.
3.5 Comparing Binary Selection and Soft Attention
Figure [c) puts together the two loss landscapes, (Tr) and (2))), with the rationale selection probability tied to the attention weights, i.., π a. There are two important observations. First, the two loss functions take the same values the two corners, #1 = on and # an 1, because either corner case, both binary selection and soft attention schemes would exclusively select one of the two sentences, hence yielding the same loss, both predictors have the same architecture and parameterization. Second. the binary selection and soft attention have complementary advantages. The former has faithful global minimum but concave; the latter is convex but the global minimum is not faithful. Therefore, both advantages can be simultaneously achieved if we can design a system with loss landscape that lies between the two loss functions, as shown by the gray curve.
4
The Proposed A2R (Attention-to-Ratiinaal Framework
4.1 The A2R Architecture
Our proposed A2R aims to combine the merits of selective rationalization and attention based explanations. shswws the architecture of A2R. A2R consists of three modules, a rationale generator, rationale-based predictor, and altention-based predictor.
The rationale generator generates a soft attention, xxX). The same soft attention also serves as the probability distribution from which the rationale selection mask, M, is drawn. i.c., M cxX).. The rationale-base predictor, f- 0,), predicts the output Y based on the input masked by M. The attention-bat based predictor, fa(; O.), predicts the output Y based on the representation weighted by xX)). ). 0, and 0. denote the parameter of the two predictors, respectively. Formally,
f+MM O 0,),), fa(X, oe (x));
Note that, instead having the input form of (xX) @ X to the attention- based predictor (as in Section 3.4), write X and (X)) two separate inputs, to accommodate broader attention mechanisms that weight on the intermediate representations rather than directly on the input. In the experiments, we implement this general framework following some common practices in the NLP community, with details deferred in Section 5.2
It is worth emphasizing that the output of the rationale generator, a (XX), is just one set of attention weights, but has two uses. First, it is used to directly weight the input features, which is fed to the attention- based predictor. Second, it is used to characterize the distribution of the rationale mask M. The rationale mask is applied to the input feature, which is then fed to the rationale-based predictor.
So far. our discussion has focused on the case where only one of the input features is selected as the rationale. A2R can generalize to the case where multiple input features are selected. this case, the rationale mask M can have multiple dimensions equal to one. In our implementation, M is determined by retaining 4% largest elements of a(X), where is preset sparsity level.
4.2 The Training Objectives
The three components have slightly different training objectives. The rationale-based predictor minimizes its prediction loss, while reducing the gap between the two predictors, i.e.
min (T 0.) AL O,, On).
(11)

---

where (T 0.) is the prediction loss of the rationale-based predictor defined in Equation + Las(a, 0r, 0.) is the Jensen-Shannin Shannon divergence between the two predicted distributions, defined as
LJS(TT 0, 0a) = =E X--rer [JS(f.(M & fo (X, Nut (x)
We select the JS divergence because it matches the scale and gradient behavior of the other loss terms.
Both the rationale generator and the attention- iitention-based predictor try to minimize the prediction loss of the attention- based predictor, while again reducing the gap between the two predictors, i.e.,
min ( 0.) + sC 0,, 0s), T(1,0%a
(12)
where (a( is the prediction loss of the attention- -baeed predictor defined in Equation 3 Note that both Equation T and (12 can be optimized using standard wradient-descent-bn techniques.
The gradient of the aationale-basel predictor does not prapagate back to the generator.
4.3 How Does A2R Work
Essentially, A2R constructs loss landscape that lies between those of the rationale- based predictor and the attention-basee predictor. To better show this, we would like to return to the toy scenario illustrated Figure A (c) the À in Equation (2) is zero, then the loss for the rationale generator would be exactly the lowest curve (i.e., C.))) As À increases, the attention based loss curve would shift upward towards the rationale-based loss. As result, the actual loss curve for the generator will resemble the gray curve in the middle, which addresses the concavity problem and thus the interlocking problem, without introducing unfaithful solutions. We use only the attention-batt predictor to govern the generator, rather than passing the gradient of both predictors to the generator, because the gradient of La is much more stable than that of Lr, which involves the policy gradient.
Experiments
5.1
Datasets
Two datasets are used in our experiments. Table in Appendix B shows their statistics. Both datasets contain human annotations, which facilitate automatic evaluation of the rationale quality. To our best knowledge, neither dataset contains personally identifiable information or offensive content.
BeerAdvocate: BeerAdvocate from 22 is multi-aspect sentiment prediction dataset, which has been commonly used in the field of rationalization 6 [T 27 46). This dataset includes sentence-level annotations, where each sentence is annotated with one multiple aspect labels.
MovieReview:
The MovieReview dataset is from the Kraser benchmark [16]. MovieReview is sentiment prediction dataset that contains phrase-level rationale annotations.
5.2 Baselines and Implementation Details
We compare the original rationalization technique RNP achieve ofate--o--hh-artt results on real-world benchmarks, which include 3PLAYER 46). HARD- KUMA S and BERT-RNP [6]. 3PLAYER model builds upon the original RNP and encourages the completeness of rationale selection. HARDKUM UMA is a token-level method that optimizes the depen- dent selection of RNP to encourage more human- interan-tterretaable extractions. BERT-RNP re-implements the original RNP with more powerful BERT generator and predictor. RNP is our main baseline to directly compare with, as RNP and our A2R match granularity of selection, optimization algorithm and model architecture. We include the other baselines to show the competitiveness of our A2R.
2 and several published models that
We follow the commonly used rationalization architectures bidirectional gated recurrent units (GRU) (5S] in the generators and the predictors for both our A2R and our reimplemented RNP. For A2R. we share the parameters of both predictors' GRU while leaving the output layers' parameters separated. Our rationale predictor f. encodes the masked input M @ X into the hidden states, followed by max-poolia moo-looling. The attention- predictor f. encodes the entire input X into hidden states, which is then weighted by a.
in implementations: We use

---

Table 2:
Results Beer-Skew (top) and Bear-Biased (bottom). P, and Fl indicate the token-level precision, recall, and Fl rationale selection. X1 % refers to ratio first sentence selection (lower better). The aroma and palate aspects have 0.5% and 0.2% of the testing examples with groundtruth rationales located in the first sentence, respectively. Bold numbers refer to the better performance between RNP and A2R in each setting.
RNP
A2R
Aspect Setting
Acc
F1
X1% %
Acc
F1
X₃ %
Skew Skew 15 Skew20
82.6
68.5
63.7
61.5
14.5
84.5
78.3
70.6
69.2
10.4
Aroma
80.4
54.5
51.6
49.3
31.2
81.8
58.1
53.3
51.7
35.7
76.8
10.8
14.1
11.0
80.5
80.0
51.7
47.9
46.3
41.5
Skew 10
77.3
5.6
7.4
5.5
63.9
82.8
50.3
48.0
45.5
27.5
Palate
Skew 15
77.1
1.2
2.5
1.3
83.1
80.9
30.2
29.9
27.7
58.0
Skew20
75.6
0.4
1.4
0.6
100.0
76.7
0.4
1.6
0.6
97.0
Biased0.7 7
84.7
71.0
65.4
63.4
12.6
85.5
77.9
70.4
69.0
12.2
Aroma
Biased0.75
84.4
58.1
54.5
52.3
25.3
85.3
68.4
61.7
60.5
20.9
Biased0.8
83.3
2.6
6.0
3.4
99.9
85.8
59.7
54.8
53.2
29.8
Biased0.7
83.9
51.4
50.5
47.3
24.3
83.5
55.0
52.9
50.1
18.8
Palate
Biased0.75 75
80.0
0.4
1.4
0.6
100.0
82.8
52.7
50.7
47.9
22.0
Biased0.8
82.0
0.4
1.4
0.6
100.0
83.6
47.9
46.2
43.5
29.6
All methods initialized with 100-dimension Glove embeddings 32! The hidden state dimensions 200 for Beer Advocate, and 100 for MovieReview. We use Adam 24 as the default optimizer with learning rate of 0.001. The policy gradient update 0.2. The aforementioned hyperparameters and the best models to report are selected according to the development accuracy. Every compared model trained on a single V100 GPU.
learning rate of 1e-4. The exploration rate
5.3 Synthetic Experiments
To better evaluate the interlocking dynamics, we first conduct two synthetic experiments using the BeerAdvocate dataset, where deliberately induce interlocking dynamics. We compare A2R with RNP, which is closest to our analyzed framework in Section that suffers from interlocking.
Beer-Skewed: In the first synthetic experiment, we let the rationale predictor overfit the first sentence of each example at the initialization. the appearance aspect of the beer, and thus is rarely the optimal rationale when the explanation target the sentiment for the aroma or palate aspects. However, by pre-training rationale predictor on the first sentence, expect induce interlocking dynamics toward selecting the sub-optimal first sentence. Specifically, we pre-train the rationale predictor for epochs by only feeding the first sentence. Once pre-trained, we then initialize the generator and train the entire rationalization pipeline. We set & to be 10, 15, and 20 for our experiments.
BeerAdvocate dataset, the first sentence usually about
Table 2 (top) shows the result in the synthetic Beer-Skewed setting. The k in 'Skewk' denotes the number of pre-training epochs. The larger the k, the more serious the overfitting. X, % denotes the percentage of the test examples where the first sentence is selected as rationale. The higher X, % is, the worse the algorithm suffers from interlocking. There are two important observations. First, when the number of skewed training epochs increases, the model performance becomes worse, i.e., it becomes harder for the models to escape from interlocking. Second, the RNP model fails to escape in the Sooma-Skew20 setting and all the palate settings (in terms of low Fl scores), while our A2R can rescue the training process except for S.eete For the other settings, both models can switch to better selection modes but the performance gaps between the RNP and our methods are large.
We further study the failure the Saete- setting with another experiment where we set A=0 to degrade our system a soft-attention system, which in theory would not suffer from interlocking. In the mean time still generates the hard mask as rationales and trains the rationale-based predictor. This results in 2.2% Fl score, with 97.3% X; selection i.e., the soft model also fails. This suggests that the failure of A2R may not be ascribed to its inability to cope with interlocking, but possibly to the gradient saturation of the predictor.
8

---

Table 3: Full results Beer Review. Our A2R achieves best results all the aspects. Note that the appearance aspect does not suffer from interlocking so all approaches performs similarly.
Appearance P
Aroma
Palate
Acc
R
F1
Acc
P
R
F1
Acc
P
R
F1
HardKuma
S
86.0
81.0
69.9
71.5
85.7
74.0
72.4
68.1
84.4
45.4
73.0
46.7
RNP
85.7
83.9
71.2
72.8
84.2
73.6
67.9
65.9
83.8
55.5
54.3
51.0
3PLAYER
85.8
78.3
66.9
68.2
84.6
74.8
68.5
66.7
83.9
54.9
53.5
50.3
Our A2R
86.3
84.7
71.2
72.9
84.9
79.3
71.3
70.0
84.0
64.2
60.9
58.0
(std)
#0.2
+1.2
-0.7
#0.8
#0.1
+:0.3
#:033
+0.4
=0.2
+:077
#0.4
#0.5
BeerAdvocate Pulate Aspect
pours dark brown almost black color there is minimal head that goes away almost immediately with only a little lacing, smell is little subdued . dark coffee malts are the main smell with slight bit hops also taste is mostly of coffee with a little dark chocolate it starts sweets but ends with the dry espresso taste mouthfeel is thick and chewy like a stout should be but prefer a smoother feet drinkability is nice very good representation for its style.
Figure 4: Examples of generated rationales on the palate aspect. Human annotated words are underlined. A2R and RNP rationales are highlighted in blue and red colors, respectively.
Beer-Biased:
The second setup considers interlocking caused by strong spurious correlations. We
follow similar setup in [12] to append punctuation "." and "." at the beginning of the first sentence with the following distributions:
p(append, Y = 1) = p(append. Y = = a; p(append |Y =Y=] = p(append. YY= = 0) = 1 a.
We set to 0.7, 0.75, and 0.8 for our experiments, which are all below the achievable accuracy that selecting the true rationales. Intuitively, since sentence one now contains the appended punctuation, which is an easy-to-capture clue, we expect induce interlocking dynamics towards selecting the first sentence, even though the appended punctuation is not as predictive as the true rationales.
Table 2 (bottom) shows the result in the synthetic Beer-Biased setting. The result is similar to that in the Beer-Skewed setting. First, the higher correlated bias makes it more difficult for the models to escape from interlocking. Second,
model can significantly outperforms the baseline across all the settings. Third, the RNP model fails to escape in the Aroma-Biased0. 88 and the Baate-Baased settings with biases ratios of 0.75 and 0.8, while our A2R can do well for all of them.
5.4 Results on Real-World Settings
BeerAdvocate: Table 3 gives results on the standard beer review task. Our A2R achieves new state-of-the-art on all the three aspects, in terms of the rationale Fl scores. All three baselines generate continuous text spans as rationales, thus giving a similar range of performance. Among them, the state-of-the-an method, HardKuma, is not restricted to selecting single sentence, but would usually select only 1~2 long spans as rationales, due to the dependent selection model and the strong continuity constraint. Therefore, the method has more freedom in rationale selection compared to the sentence selection in others, and gives high predictive accuracy and good rationalization quality.
A ARR achieves consistent performance advantage over all the baselines on all three aspects. In addition, we have observed evidence suggesting that the performance advantage is likely due to A2R's 's superior handling of the interlocking dynamies. More specifically, most beer reviews contain highly correlated aspects, which can induce interlocking dynamics towards selecting the review of a spuriously correlated aspect, analogous to the appended punctuations in the Beer-Biased synthetic setting. For example, when trained on the aroma or the palate aspect, RNP has the first epochs selecting the "overall" reviews for more than 20% of the samples. On the palate aspect, RNP also selects the aroma reviews for more than 20% samples in the first 6 epochs. Both of these observations indicate that RNP trapped in interlocking convergence path. On the appearance aspect, we do not observe severe interlocking trajectories in RNP; therefore for this aspect, we do not expect huge improvement in our proposed algorithm. The aforementioned training dynamics explain why our approach has larger performance advantage aroma and palate aspects (4.5% and 7.4% in Fl

---

respectively) than on appearance. Figure gives an example where the RNP makes a mistake of selecting the "overall" review. More examples can be found Appendix D
Table gives results on the movie review task. Since the human rationales are multiple phrase pieces, we make both RNP and A2R perform token-level selections to better fits to this task. We follow the standard setting 6 2 to use the sparsity and continuity constraints to regularize the selected rationales for all methods. For fair comparisons, we use a strong constraint weight of 1.0 to punish all algorithms that highlight more than 20% of the inputs, or have more than 10 isolated spans. These numbers are selected according the statisties of the rationale annotations.
MovieReview:
Different from BeerAdvocate, the annotations of MovieReview at the phrase-level, which are formed as multiple short spans. In addition, these
Table 4: Results movie review.
annotated rationales often tend to be "over-complete", ie.. they contain multiple phrases, all of which are
Fl
individually highly predictive of the output. Because of this, the advantage of HARDKUMA becomes less obvious compared other baselines. Yet still out- performs two different implementations of RNP (i.e.,
RNP impl by
26
13.9
BERT-RNP [6
32.2
HARDKUMA S
31.1
28.3
27.0
RNP
35.6
21.1
24.1
3PLAYER
38.2
26.0
28.0
the published result in 26.. and our own implemen- tation). Our A2R method consistently beats all the baselines including the strong BERT- based approach.
Our A2R
48.7
31.9
34.9
Sensitivity of X: In the previous experiments, we set A=1.0. 1.0. This is a natural choice because the two loss terms of the same scale. To understand the sensitivity of the À selection, we add the analysis as follows: we re-run the experiments following the setting in Table 3 with the value of A varying from le 10. Figure summarizes the results, As can be seen, A2R performs reasonably well within a wide range of À ~ [0.1, 2.0), within which the two loss terms are of comparable scales.
Finally, we would like discuss the possible future direction of annealing instead of using a fixed value. Intuitively, since the soft model does not suffer from interlocking, it may help if at the beginning of train- 70 ing we give the soft branch more freedom arrive a position without interlocking, then control the con- sistency to guarantee faithfulness. This corresponds to first set small À and then gradually increase it. However, our preliminary study shows that simple implementation does not work. Specifically, we start with À= and then gradually increase À to 1.0 by the 10-th epoch. This gives slightly worse results in almost all settings, except for the malate-BiasedO case, where a slight increase is observed.
73.5
73.8
73.9
73.9
74.1
73.6
70.7
70.6
Toven-LLvel Score
60.4
89.2
66.9
85
Appearance Aroma Palate
60
58.6
57.7
55.6
55,2
55
1e-3
1e-2
0.1
1.0
5.0
10.0
Figure 5: Analysis of the sensitivity of X.
6 Conclusion and Societal Impacts
In this paper, we re-investigate the training difficulty in selective rationalization frameworks, and identify the interlocking dynamics an important training obstacle. It essentially results from the undesirable concavity of the training objective. We provide both theoretical analysis and empirical results to verify the existence of the interlocking dynamics. Furthermore, we propose to alleviate the interlocking problem with a new A2R method, which can resolve the problem by combining the complementary merits of selective rationalization and attention-based explanations. A2R has shown consistent performance advantages over other baselines on both synthetic and real-wor world experiments. A2R helps to promote trustworthy and interpretable AI, which major concern in society. We do not identify significant negative impacts on society resulting from this work.
pointed out the lack of inherent interpretability in rationalization models, because the black-box generators are not guaranteed produce causally corrected rationales. Our A2R framework can alleviate this problem as the soft training path and the attention- based rationale generation improves the interpretability. which suggests potential towards fully interpretable rationalization models in the future.
Our proposed A2R has advantages beyond alleviating interlocking. Recent work 19 4
10

---

References
[1] David Alvarez-Melis and Tommi S Jaakkola. Towards robust interpretability with self-explaining neural networks. arXiv preprint 1.K.:18000077333 2018.
[2] Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Learning to compose neural networks for question answering. Proceedings of NAACL-HLTT, pages 1545-1554, 2016.
[3] Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein, Neural module networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 39-48, 2016.
[4] Diego Antognini and Boi Faltings. Rationalization through concepts. arXiv preprint arXiv:2105.04837 2021.
[5] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv: 1409.044, 2014.
[6] Yujia Bao, Shiyu Chang, Mo Yu, and Regina Barzilay. Deriving machine attention from human rationales. arXiv preprint 1 1808 2018.
[7] Jasmijn Bastings and Katja Filippova. The elephant in the interpretability room: Why use attention as explanation when we have saliency methods? In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 149 155, 2020.
[8] Joost Bastings, Wilker Aziz. and Ivan Titov. Interpretable neural predictions with differentiable binary variables. arXiv preprint urXiv:1905.08160 2019.
[9] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:/308.33332 2013.
[10] Samuel Carton, Qiaozhu Mei, and Paul Resnick. Extractive adversarial networks: High-recall explanations for identifying personal attacks in social media posts. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3497-3501, 2018.
[11] Shiyu Chang. Yang Zhang, Mo Yu, and Tommi Jaakkola. A game theoretic approach wise selective rationalization. In Advances in Neural Information Processing Systems, pages 10055-10065, 2019.
[12] Shiyu Chang, Yang Zhang, Mo Yu, and Tommi Jaakkola. Invariant rationalization. International Conference on Machine Learning, pages 1448-1458. PMLR, 2020.
[13] Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan. thcoretic perspective on model interpretation. In International Conference on Machine Learning, pages 882-891, 2018.
Learning to explain: An information-
[14] Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan, L-Shapley and C-Shapley: Efficient model interpretation for structured data. arXiv preprint 180vv1808.0020 2018
[15] Junyoung Chung. Caglar Gulcehre, Kyunglluun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint 112.5514, 2014.
[16] Jay Younn Sarthak Jain, Nazneen Fatema Rajani, Fric Lehman, Caiming Xiong, Richard Socher, and By- ron C Wallace. Fraser: benchmark evaluate rationalized nip models. arXiv preprint arXiv: /9/ 1.03429, 2019.
[17] Max Glockner, Ivan Habemal, and Iryna Gurevych. Why do you think that? exploring faithful sentence- level rationales without supervision. arXiv preprint arXiv:2010.03384, 2020.
[18] Nuno Miguel Guerreiro and André FT Martins. Spectra: Sparse structured text rationalization. arXiv preprint 100.:220990
[19] Alon Jacovi and Yoav Goldberg, Aligning faithful interpretations with their social attribution. Transactions of the Association for Computational Linguistics, 9:294-310, 2021.
[20] Sarthak Jain and Byron C Wallace. Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 3543-3556, 2019.
[21] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint 111.:11111111111111 144, 2016.
11

---

[22] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Inferring and executing programs for visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2989-2998, 2017.
[23] Yoon Kim, Carl Denton, Luong Hoang. and Alexander M Rush. Structured attention networks. arXiv preprint wVXV:1702200888..20 2017.
[24] Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6940 2014.
[25] Igor Kononenko al. An efficient explanation of individual classifications using game theory. Journal of Machine Learning Research, 1 11 2010.
[26] Eric Lehman, Jay DeYoung. Regina Barzilay, and Byron C Wallace. Inferring which medical treatments work from reports clinical trials. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 3705-3717, 2019.
[27] Tao Lei, Regina Barzilay, and Tommi Jaakkola. Rationalizing neural predictions. arXiv preprint 160ivv660.04410 2016.
[28] Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. Visualizing and understanding neural models in NLP. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 68 1-911 2016.
[29] Jiwei Li, Will Monroe, and Dan Jurafsky. Understanding neural networks through representation erasure. arXiv preprint arXiv:1612.000 010.12222222222.... 2016.
[30] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, pages 4765-4774, 2017.
[31] Andre Martins and Ramon Astudillo. From softmax to sparsemax: A sparse model of attention and multi-label classification. In International conference on machine learning. pages 1614-1623, PMLR, 2016.
[32] Julian McAuley, Jure Leskovec, and Dan Jurafsky. Learning attitudes and attributes from multi-aspect reviews. In 2012 IEEE 12th International Conference Data Mining, pages
IEEE, 2012.
[33] Akash Kumar Mohankumar, Preksha Nema, Sharan Narasimhan, Mitesh M Khapra, Balaji Vasan Srini- vasan, and Balaraman Ravindran. Towards transparent and explainable attention models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages 4206 021644 2020.
[34] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word repre- sentation. Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543, 2014.
[35] Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, and Zachary C. Lipton. Learning deceive with attention- based explanations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4782-4793, 2020.
[36] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should I trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1144. ACM, 2016.
[37] Alexis Ross, Ana Marasović, and Mathew E Peters. Explaining nlp models via minimal contrastive editing (mice). arXiv preprint arXiv:2012. 13985, 2020.
[38] Solia Serrano and Noah A Smith. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2931-2951, 2019.
[39] Lei Sha, Mana-Maria Camburu, and Thomas Lukasiewicz. Learning from the best: Rationalizing prediction by adversarial information calibration. arXiv preprint arXiv:0012. 2020.
[40] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv: 1312.6034, 2013.
[41] Mukund Sundararajan, Ankur Taly, and Qigi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, pages 2319-3328 JMLR. org.
2017.
12

---

[42] Marcos Treviso and André FT Martins. The explanation game: Towards prediction explainability through sparse communication. In Proceedings of the Third BlackboxNLP Workshop Analyzing and Interpreting Neurul Networks for NLP, pages 107-118, 2020.
[43] Martin Tutek and Snajder. Staying true to your word:(how) can attention become explanation? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 131-142, 2020.
[44] Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1 1-200, 2019.
[45] Kelvin Xu, Jimmy Ba. Ryan Kiros, Kyunghyun Cho, Aaron Courville. Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pages 2048-2057, 2015.
[46] Mo Yu, Shiyu Chang, Yang Zhang, and Tommi S Jaakkola. Rethinking cooperative rationalization: Introspective extraction and complement control. arXiv preprint arXiv:1910.13294, 2019.
[47] Viming Zheng, Serena Booth, Julie Shah, and Yilun Zhou, The irrationality neural rationale models. arXiv preprint arXiv:2 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] In the experiments in Sec- tion 5.4 we show our approach does not show advantage if the task has significant interlocking dynamics.
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] Please see Appendix A
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Please find the code the supplemenal material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Please see Section 5.2
(c) Did you report error bars (e-g., with respect to the random seed after running experi- ments multiple times)? [Yes] Please see Table Bable and 4
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Described at the end of Section 5.2
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] The BeerAdvocate needs to be obtained by emailing the creators, so we did not mention the license in the paper. The MovicReview data is publicly available at hhttp://www www.w...wbbb..c.......... com/
(c) Did you include any new assets either in the supplemental material or as URL? [es]] The only new asset is our codebase, which is included in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] Discussed in Section 5.1
13

---

5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "k5wyviQNujz", "reviewer": "Reviewer_x1nk", "summary": "The paper analysis the problem of interlocking in cooperative rationalization, provide both theoretical and practical analysis for this problem, and derive a potential way to solve it:\n- The paper first shows theoretically, that, when formulating selective rationalization as a two player cooperative game, where a rationale generator generates a mask, and a predictor uses the masked input to predict the output, a problem of interlocking can happen if both the predictor and the rationale generator initially have a correlated (but potentially invalid) bias as to which part of the input best explains the output. In such cases, as it has been mostly train on one, potentially suboptimal, subpart of the input, the predictor will incur greater errors when using other (potentially better) parts of the inputs, which, in turn, will reinforce the generator in selecting the suboptimal subpart. The paper analyses this problem theoretically, by showing that the prediction loss provided optimal predictor is concave in the policy, meaning that some vertices of the simplex might be local optima.\n- In turn, the authors show that when using soft-attention as an explanation scheme instead of hard attention provides a convex prediction loss provided optimal predictor, but loses the property of being faithful.\n- The authors propose a methods that combine the soft-attention and hard-attention explanation scheme to better optimize the hard attention one, by training both an attention predictor and a rationale predictor, using the attention predictor to train the rationale generator, while regularizing the attention predictor to keep it close to the rationale predictor.\n- The authors validate the existence of the interlocking problem on a toy dataset (by manually favoring one of the subparts of the inputs by hand), and show that there approach partly alleviate interlocking. They proceed to demonstrate that their approach achieves state of the art on real world datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Disclaimer: I am quite unfamiliar with the cooperative rationalization literature, and thus can hardly compare this work to potential related papers.\n\nThat being said, I found this paper to be quite an interesting read. The presentation is overall very clear:\n- The general framework of Cooperative Rationalization is clearly formulated.\n- A simple yet effective intuition of the interlocking problem is provided, and interlocking seems like a natural hypothesis to explain failure modes of cooperative rationalization.\n- The subsequent formalization of interlocking in term of concavity of the prediction loss in the policy, provided optimal predictor, is natural. The derivations are, to the best of my understanding, correct.\n- The proposed solution which boths regularize the soft-attention predictor toward the hard-attention one, while using the soft-attention predictor to train the generator is quite interesting.\n- The experimental validation of the existence of the interlocking problem is convincing, as well as the more real world experiments.\n\nMy only concern is with the dependency of the algorithm on the \\lambda parameter. One of the effects of lambda is to trade how much the soft-attention model is going to be regularized by the hard-attention model. With a low value of lambda, one falls back to the interlocking problem occurring with pure hard-attention models. With a high value of lambda, the attention model is only loosely regularized by the hard attention model, and the generator is then incited to pick sub-optimal rationales, since those rationale provide better explanation in a soft-attention setting, but not in a hard-attention one.\n- As using a bad value of lambda could lead to potentially bad results, would it be possible to provide an analysis of the sensitivity of results to lambda?\n- In the same vein, interlocking seems to be susceptible to happen mostly at the beginning of learning. Would it make sense to progressively anneal lambda (and flow gradients from the rationale predictor into the generator), to avoid convergence to a suboptimal solution?\n\n# Post rebuttal edit:\nThe authors provided the additional ablation on lambda that I was interested in, and the results seems reasonably robust to a large range of lambdas. I am consequently raising my score from 6 to 7, given the clarity and interest of the proposed analysis and of the provided experimental results.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Both limitations and societal impacts are adequately addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "YQmhrSVipAD", "reviewer": "Reviewer_rr7z", "summary": "The paper uncovers an issue in training rationale extraction models, which the authors call _interlocking dynamics_. When training a rationalizer, it turns out the optimization problem is concave, which means that in certain conditions the model can converge into a sub-optimal local minimum. When the model relies instead on a soft-attention as a rationalization, the problem is convex but the global minimum is not faithful. To solve this, A2R is proposed. A2R uses both hard-choice and soft attention rationales to obtain a better loss landscape. A2R is compared to other rationale extractors and obtains state-of-the-art in two datasets.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "_Originality_: The idea presented is novel and well-defined in relation to related work. However, I think [this paper](https://arxiv.org/pdf/2004.13876.pdf) should be cited (see comments below for more discussion about this).\n\n_Quality_: This is a great submission. The contributions are technically sound and the experiments support them. The ablation shown in Table 2 is very interesting. I found no critical weakness to recommend rejection for this paper.\n\n_Clarity_: This paper is very well-written. Explanations are thorough and it is well-organized.\n\n_Significance_: The paper reveals an issue in the training of most rationale extraction models that the community should be aware of. The model proposed aims to solve the issue and advances the state-of-the-art on two datasets.\n\n__Typos__\n\n- Line 36 and 74: soft-max --> softmax\n- Line 36: \"introducing additional component to\" --> introducing an additional component to (...) / introducing additional components to (...)\n- Line 62: \"During the training\" --> During training\n\n__Comments__\n\nHow do you think the [paper mentioned above](https://arxiv.org/pdf/2004.13876.pdf) handles the unfaithfulness of the attention-based models? Since the attention is sparse, I would expect this solution can lie near the grey line in your Figure 1c.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3h"}, "extra_text": {}}, {"review_id": "YLd5qzf3Qxg", "reviewer": "Reviewer_edqT", "summary": "This paper describes \"interlocking dynamics\" that lead to local optima in selective rationalization techniques. The idea is simple: if the rational generator selects suboptimal features, you risk the predictor overfitting to these features, thereby removing a path to the optimal solution (since the generator cannot improve the predictor's loss by selecting better features if the predictor has overfit to suboptimal features). The paper builds intuition through a simple coordination game example, and describes sufficient conditions for their attention-augmented approach to overcome the coordination problem.", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents well-written description of an intuitive problem with nice simple examples to make the problem clear, but it's let down by the description of their solution. The toy example (section 3.2) and theorems (section 3.3) do a good job of explaining what can go wrong in selective rationalization approaches, and these are supported with synthetic examples (section 5.3) that show that these issues can occur in (sufficiently adverse) practical settings, but then there is very little in the presentation of their method that explains why their solution achieves the desired loss landscape shown in figure 1 (c), or why that particular loss was chosen?\n\nIt seems like with different choices of $\\lambda$, you could interpolate between the situation in fig 1 (a) and (b) and the paper provides no guidance on how to choose $\\lambda$? The explanation given in section 4.3 says, \n\n>If the $\u0000\\lambda$ in Equation (12) is zero, then the loss for the rationale generator would be exactly the lowest curve. As $\\lambda$ increases, the attention-based loss curve would shift upward towards the rationale-based loss. As a result, the actual loss curve for the generator will resemble the gray curve in the middle [of fig 1 (c)], which addresses the concavity problem and thus the interlocking problem, without introducing unfaithful solutions.\n\nBut the conclusion that \"the actual loss curve for the generator will resemble the gray curve in the middle\" is not supported by the the experiments in 5.3 which clearly show that their method is still sensitive to interlocking under sufficient skew. Of course, this is unsurprising given that there's no a priori method for selecting $\\lambda$, but it shouldn't be presented as though it solves the problem.\n\nAdditionally:\n - I would have like to see more discussion on why the particular loss was chosen: why do we need to constrain the two predictors to output the same distribution? Why do we use JS divergence to enforce this (and not some other metric / divergence)? \n - The discussion on the feasibility of the assumptions for theorem 2 should really be in the main text. Having read that discussion, it seems like theorem 2 is more a statement about nice conditions under which $\\mathcal{L}_a$ is convex, rather than an argument that it is generally convex. E.g. you state \"as long as the strong convexity of the loss function dominates the non-convexity of $f_a$\"- fine, but we have no way of checking that. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I've described the limitations of the work above; I have no concerns about societal impact - generally methods such as these make machine learning more transparent which is positive.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "J4sNlo99tS", "reviewer": "Reviewer_aHUq", "summary": "The traditional selective rationalization consists of one rationale generator and one predictor. This cooperative rationalization structure reveals a major problem, i.e., model interlocking. To tackle this problem, in this paper, the authors propose a new rationalization framework, which introduces a third component into the architecture, a predictor driven by soft attention as opposed to selection. The additional predictor enables the framework to exhibit a better convexity property. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The research topic is interesting. In the traditional rationalization framework, the predictor only sees what the generator selects during training, and tends to overfit the selection of the generator. The authors found that the fundamental cause of the problem of interlocking is that the rationalization objective is undesirably concave with respect to the rationale generator’s policy, which leads to many sub-optimal corner solutions.\n2. The proposed A2R method is inspiring and effective. It combines the advantages of both the attention model (convexity) and binarized rationalization (faithfulness) into one. The proposed rationalization scheme can overcome the concavity of the original setup, and thus can avoid being trapped in sub-optimal rationales. \n3. The claims are well justified and proved, and the experiments are well designed.\n\nWeakness:\n1. The paper is not easy to follow for readers not familiar with selective rationalization and NLP. I suggest using more real-world examples to explain the problem, the challenges, and the advantages of the proposed framework.\n2. The experimental improvement seems trivial in some of the designed scenarios. E.g., Apperance and Aroma in Table 3.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I donot think this work has potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the problem of selective rationalization, revealing that the usual cooperative rationalization paradigm suffers from model interlocking — this arises when the predictor overfits to the features selected by the generator thus reinforcing the generator's selection even if the selected rationales are sub-optimal. To sidestep this, the paper proposes a new rationalization framework (A2R), which introduces a third component into the architecture, a predictor driven by soft attention as opposed to selection. Experiments on two synthetic benchmarks and two real datasets demonstrate that A2R can significantly alleviate the interlock problem.

This is a solid paper that proposes a simple strategy to solve an important practical problem with rationalizers. While reviewers pointed out some weaknesses (lack of clarity and need for better motivation, no sensitivity analysis for the lambda coefficient), some of these concerns have been successfully alleviated in the rebuttal, with new results for dependency on lambda. Therefore I recommend acceptance. I urge the authors to take into account the reviewers’ comments when preparing the final version. 
Decision: Accept (Poster)