Understanding Interlocking Dynamics of Cooperative Rationalization

1 Introduction . Selective rationalization [ 8 , 10 , 11 , 13 , 14 , 17 , 27 , 29 , 46 ] explains the prediction of complex neural networks by finding a small subset of the input – rationale – that suffices on its own to yield the same outcome as to the original data . To generate high-quality rationales , existing methods often train a cascaded system that consists of two components , i.e. , a rationale generator and a predictor . The generator selects a subset of the input explicitly ( a.k.a. , binarized selection ) , which is then fed to the predictor . The predictor then predicts the output based only on the subset of features selected by the generator . The rationale generator and the predictor are trained jointly to optimize the prediction performance . Compared to many other interpretable methods [ 5 , 23 , 45 , 38 ] that rely on attention mechanism as a proxy of models ’ explanation , selective rationalization offers a unique advantage : certification of exclusion , i.e. , any unselected input is guaranteed to have no contribution to prediction . ⇤Authors contributed equally to this paper . Work was done when SC was at MIT-IBM Watson AI Lab . 2We release our code at https : //github.com/Gorov/Understanding_Interlocking . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . However , binarized selective rationalization schemes are notoriously hard to train [ 8 , 46 ] . To overcome training obstacles , previous works have considered using smoothed gradient estimations ( e.g . gradient straight-through [ 9 ] or Gumbel softmax [ 21 ] ) , introducing additional components to control the complement of the selection [ 10 , 46 ] , adopting different updating dynamics between the generator and the predictor [ 11 ] , using rectified continuous random variables to handle the constrained optimization in training [ 8 ] , etc . In practice , these solutions are still insufficient . They either still require careful tuning or are at a cost of reduced predictive accuracy . In this paper , we reveal a major training problem of selective rationalization that has been largely overlooked — model interlocking . Intuitively , this problem arises because the predictor only sees what the generator selects during training , and tends to overfit to the selection of the generator . As a result , even if the generator selects a sub-optimal rationale , the predictor can still produce a lower prediction loss when given this sub-optimal rationale than when given the optimal rationale that it has never seen . As a result , the generator ’ s selection of the sub-optimal rationale will be reinforced . In the end , both the rationale generator and the predictor will be trapped in a sub-optimal equilibrium , which hurts both model ’ s predictive accuracy and the quality of generated rationales . By investigating the training objective of selective rationalization theoretically , we found that the fundamental cause of the problem of interlocking is that the rationalization objective we aim to minimize is undesirably concave with respect to the rationale generator ’ s policy , which leads to many sub-optimal corner solutions . On the other hand , although the attention-based models ( i.e. , via soft selection ) produce much less faithful explanations and do not have the nice property of certification of exclusion , their optimization objective has a better convexity property with respect to the attention weights under certain assumptions , and thus would not suffer from the interlocking problem . Motivated by these observations , we propose a new rationalization framework , called A2R ( attentionto-rationale ) , which combines the advantages of both the attention model ( convexity ) and binarized rationalization ( faithfulness ) into one . Specifically , our model consists of a generator , and two predictors . One predictor , called attention-based predictor , operates on the soft-attention , and the other predictor , called binarized predictor , operates on the binarized rationales . The attention as used by the attention-based predictor is tied to the rationale selection probability as used by the binarized predictor . During training , the generator aims to improve both predictors ’ performance while minimizing their prediction gap . As we will show theoretically , the proposed rationalization scheme can overcome the concavity of the original setup , and thus can avoid being trapped in suboptimal rationales . In addition , during inference time , we only keep the binarized predictor to ensure the faithfulness of the generated explanations . We conduct experiments on two synthetic benchmarks and two real datasets . The results demonstrate that our model can significantly alleviate the problem of interlocking and find explanations that better align with human judgments . 2 Related Work . Selective rationalization : [ 27 ] proposes the first generator-predictor framework for rationalization . Following this work , new game-theoretic frameworks were proposed to encourage different desired properties of the selected rationales , such as optimized Shapley structure scores [ 14 ] , comprehensiveness [ 46 ] , multi-aspect supports [ 4 , 11 ] and invariance [ 12 ] . Another fundamental direction is to overcome the training difficulties . [ 6 ] replaces policy gradient with Gumbel softmax . [ 46 ] proposes to first pre-train the predictor , and then perform end-to-end training . [ 11 ] adopts different updating dynamics between the generator and the predictor . [ 8 ] replaces the Bernoulli sampling distributions with rectified continuous random variables to facilitate constrained optimization . [ 39 ] proposes to enhance the training objective with an adversarial information calibration according to a black-box predictor . However , these methods can not address the problem of interlocking . Attention as a proxy of explanation : Model ’ s attention [ 5 , 23 , 45 ] could serve as a proxy of the rationale . Although attention is easy to obtain , it lacks faithfulness . An input associated with low attention weight can still significantly impact the prediction . In addition , recent works [ 7 , 20 , 35 , 38 , 44 ] also find that the same prediction on an input could be generated by totally different attentions , which limits its applicability to explaining neural predictions . To improve the faithfulness of attentions , [ 33 , 43 ] regularize the hidden representations on which the attention is computed over ; [ 17 ] applies attention weights on losses of pre-defined individual rationale candidates ’ predictions . Nevertheless , rationales remain to be more faithful explanations due to their certification of exclusion . [ 18 , 42 ] force the sparsity of the attention with sparsemax [ 31 ] , so as to promote the faithfulness of their attention as rationales . The interlocking problem still persists in this framework , because the loss landscape remains concave ( refer to our arguments in Section 3.2 & 3.3 ) . Specifically , since the predictor would not see the sentences that receive 0 attention weights , it tends to underfit these sentences . As a result , the generator does not have the incentive to assign positive weights to the sentences that are previously assigned zero weights , thus is prone to selecting the same sentences . Model interpretability beyond selective rationalization : There are other popular interpretability frameworks besides selective rationalization . Module networks [ 2 , 3 , 22 ] compose appropriate neural modules following a logical program to complete the task . Their applicability is relatively limited , due to the requirement of pre-defined modules and programs . Evaluating feature importance with gradient information [ 7 , 28 , 40 , 41 ] is another popular method . Though [ 7 ] discusses several advantages of gradient-based methods over rationalization , they are post-hoc and can not impose structural constraints on the explanation . Other lines of work that provide post-hoc explanations include local perturbations [ 25 , 30 ] ; locally fitting interpretable models [ 1 , 36 ] ; and generating explanations in the form of edits to inputs that change model prediction to the contrast case [ 37 ] . 3 Selective Rationalization and Interlocking . In this section , we will formally analyze the problem of interlocking in conventional selective rationalization frameworks . Throughout this section , upper-cased letters , i.e. , A and A , represent random vectors ( bolded ) and random values ( unbolded ) respectively ; lower cased letters , i.e. , a and a , represent deterministic vectors ( bolded ) and values ( unbolded ) respectively . Vectors with a colon subscript , i.e. , a1 : T , represent a concatenation of a1 to aT , i.e. , [ a1 ; · · · ; aT ] . 3.1 Overview of Selective Rationalization . Consider a classification problem , ( X , Y ) , where X = X1 : T is the input feature , and Y is the discrete class label . In NLP applications , X1 : T can be understood as a series of T words/sentences . The goal of selective rationalization is to identify a binary mask , M 2 { 0 , 1 } T , that applies to the input features to form a rationale vector , Z , as an explanation of Y . Formally , the rationale vector Z is defined as Z = M X ⌘ [ M1X1 , · · · , MTXT ] . ( 1 ) Conventionally , Z is determined by maximizing the mutual information between Z and Y , i.e. , max M I ( Y ; M X ) , s.t . M 2 M , ( 2 ) where M refers to a constraint set , such as the sparsity constraint and a continuity constraint , requiring that the selected rationale should be a small and continuous subset of the input features . One way of learning to extract the rationale under this criterion is to introduce a game-theoretic framework ( see Figure 1 ) consisting of two players , a rationale generator and a predictor . The rationale generator selects a subset of input as rationales and the predictor makes the prediction based only on the rationales . The two players cooperate to maximize the prediction accuracy , so the rationale generator would need to select the most informative input subset . Specifically , the rationale generator generates a probability distribution , ⇡ , for the masks , based on the input features X . Then , the mask M is randomly drawn from the distribution ⇡ . To simplify our exposition , we focus on the case that Xi represents a sentence and only one of the T sentences is selected as a rationale . In this case , M is a one-hot vector , and ⇡ is a multinomial distribution . Formally , the mask M is generated as follows M ⇠ ⇡ ( X ) = [ ⇡1 ( X ) , · · · , ⇡T ( X ) ] , where ⇡i ( X ) = p ( M = ei|X ) ] , and ei denotes a T -dimensional one-hot vector , with the i-th dimension equal to one . The generalization to making multiple selections will be discussed in Section 4.1 . After the mask is generated , the predictor , fr ( · ; ✓r ) ( the subscript r stands for rationale to differentiate from the attention-based predictor introduced later ) , then predicts the probability distribution of Y based only on Z = M X , i.e. , fr ( Z ; ✓r ) = [ p̂ ( Y = 1|Z ) , · · · , p̂ ( Y = c|Z ) ] , ( 3 ) where p̂ represents a predicted distribution , and ✓r denotes the parameters of the predictor . The generator and the predictor are trained jointly to minimize the cross-entropy loss of the prediction : min ⇡ ( · ) , ✓r Lr ( ⇡ , ✓r ) , where Lr ( ⇡ , ✓r ) = EX , Y ⇠Dtr M⇠⇡ ( X ) [ ` ( Y , fr ( M X ; ✓r ) ) ] . ( 4 ) Dtr denotes the training set ; ` ( · , · ) denotes the cross entropy loss . It can be shown [ 13 ] that , if ⇡ ( · ) and fr ( · ; ✓r ) both have sufficient representation power , the globally optimal ⇡ ( X ) of Equation ( 4 ) would generate masks M that are globally optimal under Equation ( 2 ) .

Summary: The paper analysis the problem of interlocking in cooperative rationalization, provide both theoretical and practical analysis for this problem, and derive a potential way to solve it: - The paper first shows theoretically, that, when formulating selective rationalization as a two player cooperative game, where a rationale generator generates a mask, and a predictor uses the masked input to predict the output, a problem of interlocking can happen if both the predictor and the rationale generator initially have a correlated (but potentially invalid) bias as to which part of the input best explains the output. In such cases, as it has been mostly train on one, potentially suboptimal, subpart of the input, the predictor will incur greater errors when using other (potentially better) parts of the inputs, which, in turn, will reinforce the generator in selecting the suboptimal subpart. The paper analyses this problem theoretically, by showing that the prediction loss provided optimal predictor is concave in the policy, meaning that some vertices of the simplex might be local optima. - In turn, the authors show that when using soft-attention as an explanation scheme instead of hard attention provides a convex prediction loss provided optimal predictor, but loses the property of being faithful. - The authors propose a methods that combine the soft-attention and hard-attention explanation scheme to better optimize the hard attention one, by training both an attention predictor and a rationale predictor, using the attention predictor to train the rationale generator, while regularizing the attention predictor to keep it close to the rationale predictor. - The authors validate the existence of the interlocking problem on a toy dataset (by manually favoring one of the subparts of the inputs by hand), and show that there approach partly alleviate interlocking. They proceed to demonstrate that their approach achieves state of the art on real world datasets.