Understanding Interlocking Dynamics of Cooperative Rationalization
Year: 2022

Abstract:
Selective rationalization explains the prediction of complex neural networks by finding a small subset of the input that is sufficient to predict the neural model output. The selection mechanism is commonly integrated into the model itself by specifying a two-component cascaded system consisting of a rationale generator, which makes a binary selection of the input features (which is the rationale), and a predictor, which predicts the output based only on the selected features. The components are trained jointly to optimize prediction performance. In this paper, we reveal a major problem with such cooperative rationalization paradigm — model interlocking. Interlocking arises when the predictor overfits to the features selected by the generator thus reinforcing the generator’s selection even if the selected rationales are sub-optimal. The fundamental cause of the interlocking problem is that the rationalization objective to be minimized is concave with respect to the generator’s selection policy. We propose a new rationalization framework, called A2R, which introduces a third component into the architecture, a predictor driven by soft attention as opposed to selection. The generator now realizes both soft and hard attention over the features and these are fed into the two different predictors. While the generator still seeks to support the original predictor performance, it also minimizes a gap between the two predictors. As we will show theoretically, since the attention-based predictor exhibits a better convexity property, A2R can overcome the concavity barrier. Our experiments on two synthetic benchmarks and two real datasets demonstrate that A2R can significantly alleviate the interlock problem and find explanations that better align with human judgments.2

heading: 1 Introduction
text: Selective rationalization [8, 10, 11, 13, 14, 17, 27, 29, 46] explains the prediction of complex neural networks by finding a small subset of the input – rationale – that suffices on its own to yield the same outcome as to the original data. To generate high-quality rationales, existing methods often train a cascaded system that consists of two components, i.e., a rationale generator and a predictor. The generator selects a subset of the input explicitly (a.k.a., binarized selection), which is then fed to the predictor. The predictor then predicts the output based only on the subset of features selected by the generator. The rationale generator and the predictor are trained jointly to optimize the prediction performance. Compared to many other interpretable methods [5, 23, 45, 38] that rely on attention mechanism as a proxy of models’ explanation, selective rationalization offers a unique advantage: certification of exclusion, i.e., any unselected input is guaranteed to have no contribution to prediction.
⇤Authors contributed equally to this paper. Work was done when SC was at MIT-IBM Watson AI Lab. 2We release our code at https://github.com/Gorov/Understanding_Interlocking.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
However, binarized selective rationalization schemes are notoriously hard to train [8, 46]. To overcome training obstacles, previous works have considered using smoothed gradient estimations (e.g. gradient straight-through [9] or Gumbel softmax [21]), introducing additional components to control the complement of the selection [10, 46], adopting different updating dynamics between the generator and the predictor [11], using rectified continuous random variables to handle the constrained optimization in training [8], etc. In practice, these solutions are still insufficient. They either still require careful tuning or are at a cost of reduced predictive accuracy.
In this paper, we reveal a major training problem of selective rationalization that has been largely overlooked — model interlocking. Intuitively, this problem arises because the predictor only sees what the generator selects during training, and tends to overfit to the selection of the generator. As a result, even if the generator selects a sub-optimal rationale, the predictor can still produce a lower prediction loss when given this sub-optimal rationale than when given the optimal rationale that it has never seen. As a result, the generator’s selection of the sub-optimal rationale will be reinforced. In the end, both the rationale generator and the predictor will be trapped in a sub-optimal equilibrium, which hurts both model’s predictive accuracy and the quality of generated rationales.
By investigating the training objective of selective rationalization theoretically, we found that the fundamental cause of the problem of interlocking is that the rationalization objective we aim to minimize is undesirably concave with respect to the rationale generator’s policy, which leads to many sub-optimal corner solutions. On the other hand, although the attention-based models (i.e., via soft selection) produce much less faithful explanations and do not have the nice property of certification of exclusion, their optimization objective has a better convexity property with respect to the attention weights under certain assumptions, and thus would not suffer from the interlocking problem.
Motivated by these observations, we propose a new rationalization framework, called A2R (attentionto-rationale), which combines the advantages of both the attention model (convexity) and binarized rationalization (faithfulness) into one. Specifically, our model consists of a generator, and two predictors. One predictor, called attention-based predictor, operates on the soft-attention, and the other predictor, called binarized predictor, operates on the binarized rationales. The attention as used by the attention-based predictor is tied to the rationale selection probability as used by the binarized predictor. During training, the generator aims to improve both predictors’ performance while minimizing their prediction gap. As we will show theoretically, the proposed rationalization scheme can overcome the concavity of the original setup, and thus can avoid being trapped in suboptimal rationales. In addition, during inference time, we only keep the binarized predictor to ensure the faithfulness of the generated explanations. We conduct experiments on two synthetic benchmarks and two real datasets. The results demonstrate that our model can significantly alleviate the problem of interlocking and find explanations that better align with human judgments.

heading: 2 Related Work
text: Selective rationalization: [27] proposes the first generator-predictor framework for rationalization. Following this work, new game-theoretic frameworks were proposed to encourage different desired properties of the selected rationales, such as optimized Shapley structure scores [14], comprehensiveness [46], multi-aspect supports [4, 11] and invariance [12]. Another fundamental direction is to overcome the training difficulties. [6] replaces policy gradient with Gumbel softmax. [46] proposes to first pre-train the predictor, and then perform end-to-end training. [11] adopts different updating dynamics between the generator and the predictor. [8] replaces the Bernoulli sampling distributions with rectified continuous random variables to facilitate constrained optimization. [39] proposes to enhance the training objective with an adversarial information calibration according to a black-box predictor. However, these methods cannot address the problem of interlocking.
Attention as a proxy of explanation: Model’s attention [5, 23, 45] could serve as a proxy of the rationale. Although attention is easy to obtain, it lacks faithfulness. An input associated with low attention weight can still significantly impact the prediction. In addition, recent works [7, 20, 35, 38, 44] also find that the same prediction on an input could be generated by totally different attentions, which limits its applicability to explaining neural predictions. To improve the faithfulness of attentions, [33, 43] regularize the hidden representations on which the attention is computed over; [17] applies attention weights on losses of pre-defined individual rationale candidates’ predictions. Nevertheless, rationales remain to be more faithful explanations due to their certification of exclusion.
[18, 42] force the sparsity of the attention with sparsemax [31], so as to promote the faithfulness of their attention as rationales. The interlocking problem still persists in this framework, because the loss landscape remains concave (refer to our arguments in Section 3.2&3.3). Specifically, since the predictor would not see the sentences that receive 0 attention weights, it tends to underfit these sentences. As a result, the generator does not have the incentive to assign positive weights to the sentences that are previously assigned zero weights, thus is prone to selecting the same sentences.
Model interpretability beyond selective rationalization: There are other popular interpretability frameworks besides selective rationalization. Module networks [2, 3, 22] compose appropriate neural modules following a logical program to complete the task. Their applicability is relatively limited, due to the requirement of pre-defined modules and programs. Evaluating feature importance with gradient information [7, 28, 40, 41] is another popular method. Though [7] discusses several advantages of gradient-based methods over rationalization, they are post-hoc and cannot impose structural constraints on the explanation. Other lines of work that provide post-hoc explanations include local perturbations [25, 30]; locally fitting interpretable models [1, 36]; and generating explanations in the form of edits to inputs that change model prediction to the contrast case [37].

heading: 3 Selective Rationalization and Interlocking
text: In this section, we will formally analyze the problem of interlocking in conventional selective rationalization frameworks. Throughout this section, upper-cased letters, i.e., A and A, represent random vectors (bolded) and random values (unbolded) respectively; lower cased letters, i.e., a and a, represent deterministic vectors (bolded) and values (unbolded) respectively. Vectors with a colon subscript, i.e., a1:T , represent a concatenation of a1 to aT , i.e., [a1; · · · ;aT ].

heading: 3.1 Overview of Selective Rationalization
text: Consider a classification problem, (X, Y ), where X = X1:T is the input feature, and Y is the discrete class label. In NLP applications, X1:T can be understood as a series of T words/sentences. The goal of selective rationalization is to identify a binary mask, M 2 {0, 1}T , that applies to the input features to form a rationale vector, Z, as an explanation of Y . Formally, the rationale vector Z is defined as
Z = M X ⌘ [M1X1, · · · ,MTXT ]. (1)
Conventionally, Z is determined by maximizing the mutual information between Z and Y , i.e., max M I(Y ;M X), s.t. M 2 M, (2)
where M refers to a constraint set, such as the sparsity constraint and a continuity constraint, requiring that the selected rationale should be a small and continuous subset of the input features.
One way of learning to extract the rationale under this criterion is to introduce a game-theoretic framework (see Figure 1) consisting of two players, a rationale generator and a predictor. The rationale generator selects a subset of input as rationales and the predictor makes the prediction based only on the rationales. The two players cooperate to maximize the prediction accuracy, so the rationale generator would need to select the most informative input subset.
Specifically, the rationale generator generates a probability distribution, ⇡, for the masks, based on the input features X. Then, the mask M is randomly drawn from the distribution ⇡. To simplify our exposition, we focus on the case that Xi represents a sentence and only one of the T sentences is selected as a rationale. In this case, M is a one-hot vector, and ⇡ is a multinomial distribution. Formally, the mask M is generated as follows
M ⇠ ⇡(X) = [⇡1(X), · · · ,⇡T (X)], where ⇡i(X) = p(M = ei|X)], and ei denotes a T -dimensional one-hot vector, with the i-th dimension equal to one. The generalization to making multiple selections will be discussed in Section 4.1.
After the mask is generated, the predictor, fr(·;✓r) (the subscript r stands for rationale to differentiate from the attention-based predictor introduced later), then predicts the probability distribution of Y based only on Z = M X, i.e.,
fr(Z;✓r) = [p̂(Y = 1|Z), · · · , p̂(Y = c|Z)], (3)
where p̂ represents a predicted distribution, and ✓r denotes the parameters of the predictor.
The generator and the predictor are trained jointly to minimize the cross-entropy loss of the prediction:
min ⇡(·),✓r Lr(⇡,✓r), where Lr(⇡,✓r) = EX,Y ⇠Dtr M⇠⇡(X) [`(Y,fr(M X;✓r))]. (4)
Dtr denotes the training set; `(·, ·) denotes the cross entropy loss. It can be shown [13] that, if ⇡(·) and fr(·;✓r) both have sufficient representation power, the globally optimal ⇡(X) of Equation (4) would generate masks M that are globally optimal under Equation (2).

heading: 3.2 Interlocking: A Toy Example
text: Despite the nice guarantee of its global optimum solution, the rationalization framework in Equation (4) suffers from the problem of being easily trapped into poor local minima, a problem we refer to as interlocking. To help readers understand the nature of this problem, we would like to start with a toy example, where the input consists of two sentences, X1 and X2. We assume that X1 is the more informative (in terms of predicting Y ) sentence between the two, so the optimal solution for the rationale generator ⇡ is to always select X1 (i.e. ⇡1 = 1, and ⇡2 = 0).
However, assume, for some reason, that the generator is initialized so poorly that it only selects X2, and that the predictor has been trained to make the prediction based only on X2. In this case, we will show that it is very hard for the generator-predictor to escape from this poor local minimum thus it fails to converge to the globally optimal solution of selecting X1. Since the predictor underfits to X1, it will produce a large prediction error when X1 is fed. As a result, the rationale generator would stick with selecting X2 because X2 yields a smaller prediction error than X1. The predictor, in turn, would keep overfitting to X2 and underfitting to X1. In short, both players lock the other player from escaping from the poor solution, hence the name interlocking.
The problem of interlocking can also be manifested by an accordance game, where the generator has two strategies, select X1 and select X2, and the predictor also has two strategies, overfit to X1 and overfit to X2. An example payoff table is shown in Table 1. As can be seen, (select X1, overfit to X1) has the highest payoff, and thus is the optimal solution for both players. However, (select X2, overfit to X2) also constitutes a Nash equilibrium, which is locally optimal.

heading: 3.3 Interlocking and Concave Minimization
text: To understand the fundamental cause of the interlocking problem, rewrite the optimization problem in Equation (4) into a nested form:
min ⇡(·),✓r Lr(⇡,✓r) = min ⇡(·) min ✓r Lr(⇡,✓r) = min ⇡(·)
Lr(⇡,✓⇤r (⇡)), (5)
where ✓⇤r (⇡) = argmin ✓r Lr(⇡,✓r). (6)
Furthermore, denote L⇤r(⇡) = Lr(⇡,✓⇤r (⇡)). (7)
Then, the problem of finding the optimal rationale boils down to finding the global minimum of L⇤r(⇡). In order to achieve good convergence properties, L⇤r(⇡) would ideally be convex with respect to ⇡. However, the following theorem states the opposite. Theorem 1. L⇤r(⇡) is concave with respect to ⇡.
The proof is presented in Appendix A.1. Theorem 1 implies the cooperative rationalization objective can contain many local optima at the corners. Going back to the two-sentence example, Figure 2(a) plots an example L⇤r(⇡) against ⇡1. Since there are two sentences, ⇡1 = 0 implies that the generator always selects X2, and ⇡1 = 1 implies the generator always selects X1. As shown in the figure, since X1 is more informative than X2, the global minimum is achieved at ⇡1 = 1. However, it can be observed that ⇡1 = 0 is also a local minimum, and therefore the rationalization framework can be undesirably trapped into the rationalization scheme that always selects the worse sentence of the two.

heading: 3.4 Convexity of Attention-based Explanation
text: Knowing that the selective rationalization has an undesirable concave objective, we now turn to another class of explanation scheme, i.e., attention-based explanation, which uses soft attention, rather than binary selection, of the input as an explanation. Specifically, we would like to investigate whether its objective has a more or less desirable convexity property than that of selective rationalization.
Formally, consider an attention-based predictor, fa(↵(X) X;✓a) (the subscript a stands for attention), which is almost identical to the rationalization predictor in Equation (3), except that the binary mask M is replaced with a soft attention weight ↵(X) where each dimension sums to one. So the optimization objective becomes
min ↵(·),✓a La(↵,✓a), where La(↵,✓a) = EX,Y ⇠Dtr [`(Y,fa(↵(X) X;✓a))]. (8)
Similar to Equations (5) to (7), define L⇤a(↵) = L(↵,✓⇤a(↵)), where ✓⇤a(↵) = argmin
✓a
La(↵,✓a). (9)
The following theorem shows that L⇤a(↵) has a more desirable convexity property. Theorem 2. L⇤a(↵) is convex with respect to ↵, if
1. La(↵,✓a) is µ-strongly convex with respect to ↵ with `2 distance metric, 8✓a;
2. La(↵,✓⇤a(↵0)) has a bounded regret with the optimal loss, i.e., when ↵0 = ↵, with `2 norm:
La(↵,✓⇤a(↵0)) La(↵,✓⇤a(↵))  l 2 E ⇥ k↵(X) ↵0(X)k2 ⇤2 , 8↵(·),↵0(·); (10)
3. µ l.
The proof is presented in Appendix A.2, where we also discuss the feasibility of the assumptions. A special case where the predictor has sufficient representation power is discussed in Appendix A.3 Figure 2(b) plots an example La(↵) against ↵1, again under the same two-sentence toy scenario. Note that ↵1 = 0 means X2 gets all the weight; ↵1 = 1 means X1 gets all the weights. As can be observed, La(↵) is now a convex function, which makes it more desirable in terms of optimization. However, the example in Figure 2(b) also shows why such attention-based scheme is sometimes not faithful. Even though X1 is a better sentence of the two, the global minimum La(↵) is achieved at the point where X2 gets a larger weight than X1 does. The reason why the global minimum is usually achieved in the interior (0 < ↵1 < 1) rather than the corner (↵1 = 0 or 1) is that the predictor would have access to more information if both X1 and X2 get non-zero attention weights.
X2
1 2
Hard selection
z1 zNz2
XT
N
Rationale Selector
Attention-based Predictor
x1 x2 xN
ys
Rationale Predictor
z1 x1 zN xNz2 x2
yh Lce

heading: 3.5 Comparing Binary Selection and Soft Attention
text: Figure 2(c) puts together the two loss landscapes, L⇤(⇡) and L⇤a(↵), with the rationale selection probability tied to the attention weights, i.e., ⇡ = ↵. There are two important observations. First, the two loss functions take the same values at the two corners, ⇡1 = ↵1 = 0 and ⇡1 = ↵1 = 1, because at either corner case, both binary selection and soft attention schemes would exclusively select one of the two sentences, hence yielding the same loss, if both predictors have the same architecture and parameterization. Second, the binary selection and soft attention have complementary advantages. The former has a faithful global minimum but concave; the latter is convex but the global minimum is not faithful. Therefore, both advantages can be simultaneously achieved if we can design a system with a loss landscape that lies in between the two loss functions, as shown by the gray curve.

heading: 4 The Proposed A2R (Attention-to-Rationale) Framework

heading: 4.1 The A2R Architecture
text: Our proposed A2R aims to combine the merits of selective rationalization and attention-based explanations. Figure 3 shows the architecture of A2R. A2R consists of three modules, a rationale generator, a rationale-based predictor, and an attention-based predictor.
The rationale generator generates a soft attention, ↵(X). The same soft attention also serves as the probability distribution from which the rationale selection mask, M , is drawn. i.e., M ⇠ ↵(X). The rationale-based predictor, fr(·;✓r), predicts the output Y based on the input masked by M . The attention-based predictor, fa(·;✓a), predicts the output Y based on the representation weighted by ↵(X). ✓r and ✓a denote the parameter of the two predictors, respectively. Formally,
fr(M X;✓r), fa(X,↵(X);✓a).
Note that, instead of having the input form of ↵(X) X to the attention-based predictor (as in Section 3.4), we write X and ↵(X) as two separate inputs, to accommodate broader attention mechanisms that weight on the intermediate representations rather than directly on the input. In the experiments, we implement this general framework following some common practices in the NLP community, with details deferred in Section 5.2.
It is worth emphasizing that the output of the rationale generator, ↵(X), is just one set of attention weights, but has two uses. First, it is used to directly weight the input features, which is fed to the attention-based predictor. Second, it is used to characterize the distribution of the rationale mask M . The rationale mask is applied to the input feature, which is then fed to the rationale-based predictor.
So far, our discussion has focused on the case where only one of the input features is selected as the rationale. A2R can generalize to the case where multiple input features are selected. In this case, the rationale mask M can have multiple dimensions equal to one. In our implementation, M is determined by retaining q% largest elements of ↵(X), where q is a preset sparsity level.

heading: 4.2 The Training Objectives
text: The three components have slightly different training objectives. The rationale-based predictor minimizes its prediction loss, while reducing the gap between the two predictors, i.e.
min ✓r Lr(⇡,✓r) + LJS(⇡,✓r,✓a), (11)
where Lr(⇡,✓r) is the prediction loss of the rationale-based predictor defined in Equation (4). LJS(⇡,✓r,✓a) is the Jensen-Shannon divergence between the two predicted distributions, defined as
LJS(⇡,✓r,✓a) = E X⇠Dtr M⇠↵(X) [JS(fr(M X;✓r)kfa(X,↵(X);✓a))] .
We select the JS divergence because it matches the scale and gradient behavior of the other loss terms.
Both the rationale generator and the attention-based predictor try to minimize the prediction loss of the attention-based predictor, while again reducing the gap between the two predictors, i.e.,
min ⇡(·),✓a La(⇡,✓a) + LJS(⇡,✓r,✓a), (12)
where La(⇡,✓a) is the prediction loss of the attention-based predictor defined in Equation (8). Note that both Equation (11) and (12) can be optimized using standard gradient-descent-based techniques. The gradient of the rationale-based predictor does not prapagate back to the generator.

heading: 4.3 How Does A2R Work
text: Essentially, A2R constructs a loss landscape that lies between those of the rationale-based predictor and the attention-based predictor. To better show this, we would like to return to the toy scenario illustrated in Figure 2(c). If the in Equation (12) is zero, then the loss for the rationale generator would be exactly the lowest curve (i.e., L⇤a). As increases, the attention-based loss curve would shift upward towards the rationale-based loss. As a result, the actual loss curve for the generator will resemble the gray curve in the middle, which addresses the concavity problem and thus the interlocking problem, without introducing unfaithful solutions. We use only the attention-based predictor to govern the generator, rather than passing the gradient of both predictors to the generator, because the gradient of La is much more stable than that of Lr, which involves the policy gradient.

heading: 5 Experiments

heading: 5.1 Datasets
text: Two datasets are used in our experiments. Table 5 in Appendix B shows their statistics. Both datasets contain human annotations, which facilitate automatic evaluation of the rationale quality. To our best knowledge, neither dataset contains personally identifiable information or offensive content.
BeerAdvocate: BeerAdvocate from [32] is a multi-aspect sentiment prediction dataset, which has been commonly used in the field of rationalization [6, 11, 27, 46]. This dataset includes sentence-level annotations, where each sentence is annotated with one or multiple aspect labels.
MovieReview: The MovieReview dataset is from the Eraser benchmark [16]. MovieReview is a sentiment prediction dataset that contains phrase-level rationale annotations.

heading: 5.2 Baselines and Implementation Details
text: We compare to the original rationalization technique RNP [27], and several published models that achieve state-of-the-art results on real-world benchmarks, which include 3PLAYER [46], HARDKUMA3 [8], and BERT-RNP [16]. 3PLAYER model builds upon the original RNP and encourages the completeness of rationale selection. HARDKUMA is a token-level method that optimizes the dependent selection of RNP to encourage more human-interpretable extractions. BERT-RNP re-implements the original RNP with more powerful BERT generator and predictor. RNP is our main baseline to directly compare with, as RNP and our A2R match in granularity of selection, optimization algorithm and model architecture. We include the other baselines to show the competitiveness of our A2R .
We follow the commonly used rationalization architectures [8, 27] in our implementations: We use bidirectional gated recurrent units (GRU) [15] in the generators and the predictors for both our A2R and our reimplemented RNP. For A2R, we share the parameters of both predictors’ GRU while leaving the output layers’ parameters separated. Our rationale predictor fr encodes the masked input M X into the hidden states, followed by max-pooling. The attention-based predictor fa encodes the entire input X into hidden states, which is then weighted by ↵.
3https://github.com/bastings/interpretable_predictions.
All methods are initialized with 100-dimension Glove embeddings [34]. The hidden state dimensions is 200 for BeerAdvocate, and 100 for MovieReview. We use Adam [24] as the default optimizer with a learning rate of 0.001. The policy gradient update uses a learning rate of 1e-4. The exploration rate is 0.2. The aforementioned hyperparameters and the best models to report are selected according to the development set accuracy. Every compared model is trained on a single V100 GPU.

heading: 5.3 Synthetic Experiments
text: To better evaluate the interlocking dynamics, we first conduct two synthetic experiments using the BeerAdvocate dataset, where we deliberately induce interlocking dynamics. We compare our A2R with RNP, which is closest to our analyzed framework in Section 3 that suffers from interlocking.
Beer-Skewed: In the first synthetic experiment, we let the rationale predictor overfit the first sentence of each example at the initialization. In the BeerAdvocate dataset, the first sentence is usually about the appearance aspect of the beer, and thus is rarely the optimal rationale when the explanation target is the sentiment for the aroma or palate aspects. However, by pre-training rationale predictor on the first sentence, we expect to induce an interlocking dynamics toward selecting the sub-optimal first sentence. Specifically, we pre-train the rationale predictor for k epochs by only feeding the first sentence. Once pre-trained, we then initialize the generator and train the entire rationalization pipeline. We set k to be 10, 15, and 20 for our experiments.
Table 2 (top) shows the result in the synthetic Beer-Skewed setting. The k in ‘Skewk’ denotes the number of pre-training epochs. The larger the k, the more serious the overfitting. X1% denotes the percentage of the test examples where the first sentence is selected as rationale. The higher X1% is, the worse the algorithm suffers from interlocking. There are two important observations. First, when the number of skewed training epochs increases, the model performance becomes worse, i.e., it becomes harder for the models to escape from interlocking. Second, the RNP model fails to escape in the Aroma-Skew20 setting and all the palate settings (in terms of low F1 scores), while our A2R can rescue the training process except for Palate-Skew20. For the other settings, both models can switch to better selection modes but the performance gaps between the RNP and our methods are large.
We further study the failure in the Palate-Skew20 setting with another experiment where we set =0 to degrade our system a soft-attention system, which in theory would not suffer from interlocking. In the mean time it still generates the hard mask as rationales and trains the rationale-based predictor. This results in a 2.2% F1 score, with 97.3% X1 selection – i.e., the soft model also fails. This suggests that the failure of A2R may not be ascribed to its inability to cope with interlocking, but possibly to the gradient saturation of the predictor.
and RNP rationales are highlighted in blue and red colors, respectively.
Beer-Biased: The second setup considers interlocking caused by strong spurious correlations. We follow a similar setup in [12] to append punctuation “,” and “.” at the beginning of the first sentence with the following distributions:
p(append , |Y = 1) = p(append . |Y = 0) = ↵; p(append . |Y = 1) = p(append , |Y = 0) = 1 ↵.
We set ↵ to 0.7, 0.75, and 0.8 for our experiments, which are all below the achievable accuracy that selecting the true rationales. Intuitively, since sentence one now contains the appended punctuation, which is an easy-to-capture clue, we expect to induce an interlocking dynamics towards selecting the first sentence, even though the appended punctuation is not as predictive as the true rationales.
Table 2 (bottom) shows the result in the synthetic Beer-Biased setting. The result is similar to that in the Beer-Skewed setting. First, the higher correlated bias makes it more difficult for the models to escape from interlocking. Second, our model can significantly outperforms the baseline across all the settings. Third, the RNP model fails to escape in the Aroma-Biased0.8 and the Palate-Biased settings with biases ratios of 0.75 and 0.8, while our A2R can do well for all of them.

heading: 5.4 Results on Real-World Settings
text: BeerAdvocate: Table 3 gives results on the standard beer review task. Our A2R achieves new state-of-the-art on all the three aspects, in terms of the rationale F1 scores. All three baselines generate continuous text spans as rationales, thus giving a similar range of performance. Among them, the state-of-the-art method, HardKuma, is not restricted to selecting a single sentence, but would usually select only 1⇠2 long spans as rationales, due to the dependent selection model and the strong continuity constraint. Therefore, the method has more freedom in rationale selection compared to the sentence selection in others, and gives high predictive accuracy and good rationalization quality.
A2R achieves a consistent performance advantage over all the baselines on all three aspects. In addition, we have observed evidence suggesting that the performance advantage is likely due to A2R’s superior handling of the interlocking dynamics. More specifically, most beer reviews contain highly correlated aspects, which can induce interlocking dynamics towards selecting the review of a spuriously correlated aspect, analogous to the appended punctuations in the Beer-Biased synthetic setting. For example, when trained on the aroma or the palate aspect, RNP has the first 7 epochs selecting the “overall” reviews for more than 20% of the samples. On the palate aspect, RNP also selects the aroma reviews for more than 20% samples in the first 6 epochs. Both of these observations indicate that RNP is trapped in a interlocking convergence path. On the appearance aspect, we do not observe severe interlocking trajectories in RNP; therefore for this aspect, we do not expect a huge improvement in our proposed algorithm. The aforementioned training dynamics explain why our approach has a larger performance advantage on aroma and palate aspects (4.5% and 7.4% in F1
respectively) than on appearance. Figure 4 gives an example where the RNP makes a mistake of selecting the “overall” review. More examples can be found in Appendix D.
MovieReview: Table 4 gives results on the movie review task. Since the human rationales are multiple phrase pieces, we make both RNP and A2R perform token-level selections to better fits to this task. We follow the standard setting [6, 27] to use the sparsity and continuity constraints to regularize the selected rationales for all methods. For fair comparisons, we use a strong constraint weight of 1.0 to punish all algorithms that highlight more than 20% of the inputs, or have more than 10 isolated spans. These numbers are selected according to the statistics of the rationale annotations.
Different from BeerAdvocate, the annotations of MovieReview are at the phrase-level, which are formed as multiple short spans. In addition, these annotated rationales often tend to be “over-complete”, i.e., they contain multiple phrases, all of which are individually highly predictive of the output. Because of this, the advantage of HARDKUMA becomes less obvious compared to other baselines. Yet it still outperforms two different implementations of RNP (i.e., the published result in [26], and our own implementation). Our A2R method consistently beats all the baselines including the strong BERT-based approach.
Sensitivity of : In the previous experiments, we set =1.0. This is a natural choice because the two loss terms are of the same scale. To understand the sensitivity of the selection, we add the analysis as follows: we re-run the experiments following the setting in Table 3, with the value of varying from 1e3 to 10. Figure 5 summarizes the results. As can be seen, A2R performs reasonably well within a wide range of ⇠ [0.1, 2.0], within which the two loss terms are of comparable scales.
Finally, we would like to discuss the possible future direction of annealing instead of using a fixed value. Intuitively, since the soft model does not suffer from interlocking, it may help if at the beginning of training we give the soft branch more freedom to arrive at a position without interlocking, then control the consistency to guarantee faithfulness. This corresponds to first set a small and then gradually increase it. However, our preliminary study shows that a simple implementation does not work. Specifically, we start with = 0 and then gradually increase to 1.0 by the 10-th epoch. This gives slightly worse results in almost all settings, except for the Palate-Biased0.8 case, where a slight increase is observed.

heading: 6 Conclusion and Societal Impacts
text: In this paper, we re-investigate the training difficulty in selective rationalization frameworks, and identify the interlocking dynamics as an important training obstacle. It essentially results from the undesirable concavity of the training objective. We provide both theoretical analysis and empirical results to verify the existence of the interlocking dynamics. Furthermore, we propose to alleviate the interlocking problem with a new A2R method, which can resolve the problem by combining the complementary merits of selective rationalization and attention-based explanations. A2R has shown consistent performance advantages over other baselines on both synthetic and real-world experiments. A2R helps to promote trustworthy and interpretable AI, which is a major concern in society. We do not identify significant negative impacts on society resulting from this work.
Our proposed A2R has advantages beyond alleviating interlocking. Recent work [19, 47] pointed out the lack of inherent interpretability in rationalization models, because the black-box generators are not guaranteed to produce causally corrected rationales. Our A2R framework can alleviate this problem as the soft training path and the attention-based rationale generation improves the interpretability, which suggests a potential towards fully interpretable rationalization models in the future.

Review summaries:
The paper analysis the problem of interlocking in cooperative rationalization, provide both theoretical and practical analysis for this problem, and derive a potential way to solve it: - The paper first shows theoretically, that, when formulating selective rationalization as a two player cooperative game, where a rationale generator generates a mask, and a predictor uses the masked input to predict the output, a problem of interlocking can happen if both the predictor and the rationale generator initially have a correlated (but potentially invalid) bias as to which part of the input best explains the output. In such cases, as it has been mostly train on one, potentially suboptimal, subpart of the input, the predictor will incur greater errors when using other (potentially better) parts of the inputs, which, in turn, will reinforce the generator in selecting the suboptimal subpart. The paper analyses this problem theoretically, by showing that the prediction loss provided optimal predictor is concave in the policy, meaning that some vertices of the simplex might be local optima. - In turn, the authors show that when using soft-attention as an explanation scheme instead of hard attention provides a convex prediction loss provided optimal predictor, but loses the property of being faithful. - The authors propose a methods that combine the soft-attention and hard-attention explanation scheme to better optimize the hard attention one, by training both an attention predictor and a rationale predictor, using the attention predictor to train the rationale generator, while regularizing the attention predictor to keep it close to the rationale predictor. - The authors validate the existence of the interlocking problem on a toy dataset (by manually favoring one of the subparts of the inputs by hand), and show that there approach partly alleviate interlocking. They proceed to demonstrate that their approach achieves state of the art on real world datasets.

The traditional selective rationalization consists of one rationale generator and one predictor. This cooperative rationalization structure reveals a major problem, i.e., model interlocking. To tackle this problem, in this paper, the authors propose a new rationalization framework, which introduces a third component into the architecture, a predictor driven by soft attention as opposed to selection. The additional predictor enables the framework to exhibit a better convexity property. 

The paper uncovers an issue in training rationale extraction models, which the authors call _interlocking dynamics_. When training a rationalizer, it turns out the optimization problem is concave, which means that in certain conditions the model can converge into a sub-optimal local minimum. When the model relies instead on a soft-attention as a rationalization, the problem is convex but the global minimum is not faithful. To solve this, A2R is proposed. A2R uses both hard-choice and soft attention rationales to obtain a better loss landscape. A2R is compared to other rationale extractors and obtains state-of-the-art in two datasets.

This paper describes "interlocking dynamics" that lead to local optima in selective rationalization techniques. The idea is simple: if the rational generator selects suboptimal features, you risk the predictor overfitting to these features, thereby removing a path to the optimal solution (since the generator cannot improve the predictor's loss by selecting better features if the predictor has overfit to suboptimal features). The paper builds intuition through a simple coordination game example, and describes sufficient conditions for their attention-augmented approach to overcome the coordination problem.