Understanding Interlocking Dynamics of Cooperative Rationalization

1 Introduction . Selective rationalization [ 8 , 10 , 11 , 13 , 14 , 17 , 27 , 29 , 46 ] explains the prediction of complex neural networks by finding a small subset of the input – rationale – that suffices on its own to yield the same outcome as to the original data . To generate high-quality rationales , existing methods often train a cascaded system that consists of two components , i.e. , a rationale generator and a predictor . The generator selects a subset of the input explicitly ( a.k.a. , binarized selection ) , which is then fed to the predictor . The predictor then predicts the output based only on the subset of features selected by the generator . The rationale generator and the predictor are trained jointly to optimize the prediction performance . Compared to many other interpretable methods [ 5 , 23 , 45 , 38 ] that rely on attention mechanism as a proxy of models ’ explanation , selective rationalization offers a unique advantage : certification of exclusion , i.e. , any unselected input is guaranteed to have no contribution to prediction . ⇤Authors contributed equally to this paper . Work was done when SC was at MIT-IBM Watson AI Lab . 2We release our code at https : //github.com/Gorov/Understanding_Interlocking . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . However , binarized selective rationalization schemes are notoriously hard to train [ 8 , 46 ] . To overcome training obstacles , previous works have considered using smoothed gradient estimations ( e.g . gradient straight-through [ 9 ] or Gumbel softmax [ 21 ] ) , introducing additional components to control the complement of the selection [ 10 , 46 ] , adopting different updating dynamics between the generator and the predictor [ 11 ] , using rectified continuous random variables to handle the constrained optimization in training [ 8 ] , etc . In practice , these solutions are still insufficient . They either still require careful tuning or are at a cost of reduced predictive accuracy . In this paper , we reveal a major training problem of selective rationalization that has been largely overlooked — model interlocking . Intuitively , this problem arises because the predictor only sees what the generator selects during training , and tends to overfit to the selection of the generator . As a result , even if the generator selects a sub-optimal rationale , the predictor can still produce a lower prediction loss when given this sub-optimal rationale than when given the optimal rationale that it has never seen . As a result , the generator ’ s selection of the sub-optimal rationale will be reinforced . In the end , both the rationale generator and the predictor will be trapped in a sub-optimal equilibrium , which hurts both model ’ s predictive accuracy and the quality of generated rationales . By investigating the training objective of selective rationalization theoretically , we found that the fundamental cause of the problem of interlocking is that the rationalization objective we aim to minimize is undesirably concave with respect to the rationale generator ’ s policy , which leads to many sub-optimal corner solutions . On the other hand , although the attention-based models ( i.e. , via soft selection ) produce much less faithful explanations and do not have the nice property of certification of exclusion , their optimization objective has a better convexity property with respect to the attention weights under certain assumptions , and thus would not suffer from the interlocking problem . Motivated by these observations , we propose a new rationalization framework , called A2R ( attentionto-rationale ) , which combines the advantages of both the attention model ( convexity ) and binarized rationalization ( faithfulness ) into one . Specifically , our model consists of a generator , and two predictors . One predictor , called attention-based predictor , operates on the soft-attention , and the other predictor , called binarized predictor , operates on the binarized rationales . The attention as used by the attention-based predictor is tied to the rationale selection probability as used by the binarized predictor . During training , the generator aims to improve both predictors ’ performance while minimizing their prediction gap . As we will show theoretically , the proposed rationalization scheme can overcome the concavity of the original setup , and thus can avoid being trapped in suboptimal rationales . In addition , during inference time , we only keep the binarized predictor to ensure the faithfulness of the generated explanations . We conduct experiments on two synthetic benchmarks and two real datasets . The results demonstrate that our model can significantly alleviate the problem of interlocking and find explanations that better align with human judgments . 2 Related Work . Selective rationalization : [ 27 ] proposes the first generator-predictor framework for rationalization . Following this work , new game-theoretic frameworks were proposed to encourage different desired properties of the selected rationales , such as optimized Shapley structure scores [ 14 ] , comprehensiveness [ 46 ] , multi-aspect supports [ 4 , 11 ] and invariance [ 12 ] . Another fundamental direction is to overcome the training difficulties . [ 6 ] replaces policy gradient with Gumbel softmax . [ 46 ] proposes to first pre-train the predictor , and then perform end-to-end training . [ 11 ] adopts different updating dynamics between the generator and the predictor . [ 8 ] replaces the Bernoulli sampling distributions with rectified continuous random variables to facilitate constrained optimization . [ 39 ] proposes to enhance the training objective with an adversarial information calibration according to a black-box predictor . However , these methods can not address the problem of interlocking . Attention as a proxy of explanation : Model ’ s attention [ 5 , 23 , 45 ] could serve as a proxy of the rationale . Although attention is easy to obtain , it lacks faithfulness . An input associated with low attention weight can still significantly impact the prediction . In addition , recent works [ 7 , 20 , 35 , 38 , 44 ] also find that the same prediction on an input could be generated by totally different attentions , which limits its applicability to explaining neural predictions . To improve the faithfulness of attentions , [ 33 , 43 ] regularize the hidden representations on which the attention is computed over ; [ 17 ] applies attention weights on losses of pre-defined individual rationale candidates ’ predictions . Nevertheless , rationales remain to be more faithful explanations due to their certification of exclusion . [ 18 , 42 ] force the sparsity of the attention with sparsemax [ 31 ] , so as to promote the faithfulness of their attention as rationales . The interlocking problem still persists in this framework , because the loss landscape remains concave ( refer to our arguments in Section 3.2 & 3.3 ) . Specifically , since the predictor would not see the sentences that receive 0 attention weights , it tends to underfit these sentences . As a result , the generator does not have the incentive to assign positive weights to the sentences that are previously assigned zero weights , thus is prone to selecting the same sentences . Model interpretability beyond selective rationalization : There are other popular interpretability frameworks besides selective rationalization . Module networks [ 2 , 3 , 22 ] compose appropriate neural modules following a logical program to complete the task . Their applicability is relatively limited , due to the requirement of pre-defined modules and programs . Evaluating feature importance with gradient information [ 7 , 28 , 40 , 41 ] is another popular method . Though [ 7 ] discusses several advantages of gradient-based methods over rationalization , they are post-hoc and can not impose structural constraints on the explanation . Other lines of work that provide post-hoc explanations include local perturbations [ 25 , 30 ] ; locally fitting interpretable models [ 1 , 36 ] ; and generating explanations in the form of edits to inputs that change model prediction to the contrast case [ 37 ] . 3 Selective Rationalization and Interlocking . In this section , we will formally analyze the problem of interlocking in conventional selective rationalization frameworks . Throughout this section , upper-cased letters , i.e. , A and A , represent random vectors ( bolded ) and random values ( unbolded ) respectively ; lower cased letters , i.e. , a and a , represent deterministic vectors ( bolded ) and values ( unbolded ) respectively . Vectors with a colon subscript , i.e. , a1 : T , represent a concatenation of a1 to aT , i.e. , [ a1 ; · · · ; aT ] . 3.1 Overview of Selective Rationalization . Consider a classification problem , ( X , Y ) , where X = X1 : T is the input feature , and Y is the discrete class label . In NLP applications , X1 : T can be understood as a series of T words/sentences . The goal of selective rationalization is to identify a binary mask , M 2 { 0 , 1 } T , that applies to the input features to form a rationale vector , Z , as an explanation of Y . Formally , the rationale vector Z is defined as Z = M X ⌘ [ M1X1 , · · · , MTXT ] . ( 1 ) Conventionally , Z is determined by maximizing the mutual information between Z and Y , i.e. , max M I ( Y ; M X ) , s.t . M 2 M , ( 2 ) where M refers to a constraint set , such as the sparsity constraint and a continuity constraint , requiring that the selected rationale should be a small and continuous subset of the input features . One way of learning to extract the rationale under this criterion is to introduce a game-theoretic framework ( see Figure 1 ) consisting of two players , a rationale generator and a predictor . The rationale generator selects a subset of input as rationales and the predictor makes the prediction based only on the rationales . The two players cooperate to maximize the prediction accuracy , so the rationale generator would need to select the most informative input subset . Specifically , the rationale generator generates a probability distribution , ⇡ , for the masks , based on the input features X . Then , the mask M is randomly drawn from the distribution ⇡ . To simplify our exposition , we focus on the case that Xi represents a sentence and only one of the T sentences is selected as a rationale . In this case , M is a one-hot vector , and ⇡ is a multinomial distribution . Formally , the mask M is generated as follows M ⇠ ⇡ ( X ) = [ ⇡1 ( X ) , · · · , ⇡T ( X ) ] , where ⇡i ( X ) = p ( M = ei|X ) ] , and ei denotes a T -dimensional one-hot vector , with the i-th dimension equal to one . The generalization to making multiple selections will be discussed in Section 4.1 . After the mask is generated , the predictor , fr ( · ; ✓r ) ( the subscript r stands for rationale to differentiate from the attention-based predictor introduced later ) , then predicts the probability distribution of Y based only on Z = M X , i.e. , fr ( Z ; ✓r ) = [ p̂ ( Y = 1|Z ) , · · · , p̂ ( Y = c|Z ) ] , ( 3 ) where p̂ represents a predicted distribution , and ✓r denotes the parameters of the predictor . The generator and the predictor are trained jointly to minimize the cross-entropy loss of the prediction : min ⇡ ( · ) , ✓r Lr ( ⇡ , ✓r ) , where Lr ( ⇡ , ✓r ) = EX , Y ⇠Dtr M⇠⇡ ( X ) [ ` ( Y , fr ( M X ; ✓r ) ) ] . ( 4 ) Dtr denotes the training set ; ` ( · , · ) denotes the cross entropy loss . It can be shown [ 13 ] that , if ⇡ ( · ) and fr ( · ; ✓r ) both have sufficient representation power , the globally optimal ⇡ ( X ) of Equation ( 4 ) would generate masks M that are globally optimal under Equation ( 2 ) .

Summary: This paper describes "interlocking dynamics" that lead to local optima in selective rationalization techniques. The idea is simple: if the rational generator selects suboptimal features, you risk the predictor overfitting to these features, thereby removing a path to the optimal solution (since the generator cannot improve the predictor's loss by selecting better features if the predictor has overfit to suboptimal features). The paper builds intuition through a simple coordination game example, and describes sufficient conditions for their attention-augmented approach to overcome the coordination problem.