AGR: Reinforced Causal Agent-Guided Self-explaining Rationalization (2024)

Abstract: Most existing rationalization approaches are susceptible to degeneration accumulation due to a lack of effective control over the learning direction of the model during training. To address this issue, we propose a novel approach AGR (Agent-Guided Rationalization), guiding the next action of the model based on its current training state. Specifically, we introduce causal intervention calculus to quantify the causal effects inherent during rationale training, and utilize reinforcement learning process to refine the learning bias of them. Furthermore, we pretrain an agent within this reinforced causal environment to guide the next step of the model. We theoretically demonstrate that a good model needs the desired guidance, and empirically show the effectiveness of our approach, outperforming existing state-of-the-art methods on BeerAdvocate and HotelReview datasets.

heading: 1 Introduction
text: To explain the prediction of neural networks, selective rationalization task (Lei et al., 2016; Yu et al., 2019, 2021) has been studied in recent years. As shown in Figure 1, it aims to select a small and human-intelligible subset (i.e., rationale) from the input to support and explain the prediction results when yielding them. As an interpretable diagram, rationalization holds significant potential for elucidating the decision-making process of predictive models, building trust, and deriving insightful and pertinent insights (Yuan et al., 2020; Zhang et al., 2023; Deng et al., 2023).
Various approaches have been proposed for rationalization, spanning from early rationale sampling-based methods (Bao et al., 2018; Bastings et al., 2019; Paranjape et al., 2020) to the extra-component-based methods (De Cao et al., 2020; Huang et al., 2021; Yu et al., 2021; Liu et al., 2022; Yue et al., 2022; Liu et al., 2023a). These
*Corresponding author
methods predominantly concentrate on improving the performance of rationalization models by either refining the sampling directly or aligning additional information beyond the rationale, resulting in impressive results. However, to the best of our knowledge, the current methods are prone to degeneration accumulation1 since they usually do not discern whether the generator during training has produced unmeaningful or flawed rationales; instead, they directly pass them to the predictor even if generated rationales are degraded.
For instance, the underlined rationale in Figure 1 is degraded, as the word
:::::::::: appearance alone does not
reliably determine the sentiment polarity of input X . But the predictor overfits to this uninformative rationale and classifies the sentiment according to whether “appearance” is included in the rationale. Consequently, when the predictor receives degraded rationales, it steers the model towards an undesirable direction (aka., learning bias). Thus, optimizing this bias during training is crucial for ensuring the model’s generalization performance.
The proposed methods (Chang et al., 2020; Zhang et al., 2023; Yue et al., 2023) fall short in considering rationalization optimization comprehensively, neglecting existing causality during rationale learning. Although they often employ causal theory to uncover relationships between rationale pieces, they struggle to directly optimize
1Degeneration over rationalization is a highly challenging problem, which means the predictor may overfit to meaningless rationales generated by the not yet well-trained generator (Yu et al., 2019; Liu et al., 2023b,d).
510
the cooperative game dynamics between the generator and predictor during training. As shown in Figure 1, optimizing rationale from “appearance” to “appearance: light yellow to almost clear” necessitates evaluating the causal impact on target prediction, guiding the model’s subsequent optimization. Thus, if we could construct a guiding signal to reward or penalize the learning behavior of the model, this would significantly reduce the model’s learning bias during training, alleviating the problem of degeneration accumulation.
To address the above problems, we propose a novel rationalization method named AGR (AgentGuided Rationalization), which leverages a reinforced causal agent to guide the cooperative game optimization during rationale training, as shown in Figure 2. In particular, 1) we quantify the causal effects in the rationale optimization process, and design a reinforcement learning (RL) process (e.g., Markov decision) to refine the learning bias during training. 2) We further pretrain an agent within reinforced causal environment to guide next actions by a system of rewards. We also theoretically illustrate that a robust model needs the desired guidance. 3) Experimental results demonstrate the effectiveness of our approach, surpassing state-of-the-art methods on BeerAdvocate and HotelReview datasets.

heading: 2 Problem Formulation
text: Notation. Following previous research (Liu et al., 2023b,c,d), we consider the classification problem and denote the generator and predictor as fG(·) and fP (·), with θg and θp representing their parameters. The input text X = [x1, x2, ..., xl](1 ≤ i ≤ l) consists of tokens xi, where l is the number of tokens. The label of X is a one-hot vector Y ∈ {0,1}c, where c is the number of categories. Cooperative game for rationalization. The fG(·) selects the most informative pieces from X by a sequence of binary mask M = [m1, ...,ml] ∈{0,1}l. Then, it forms the rationale Z = M ⊙ X = [m1x1,m2x2, ...,mlxl], where the informativeness of Z is measured by the negative cross entropy −H(Y, Ŷ ). Consequently, the fG(·) and fP (·) are optimized cooperatively by
min θg ,θp H(Y, Ŷ ∣ fG(X)), s.t.Ŷ = fP (fG(X)). (1)
In addition, rationales are usually constrained by compact and coherent regularization terms Ω(M) = λ1 ∣ ∣∣M ∣∣1l − s∣+λ2∑t∣mt −mt−1∣ (Chang et al., 2020), where s is a pre-defined sparsity level.

heading: 3 Reinforced Causal Agent
text: In this section, we present our reinforced causal agent, considering both causal effect and learning bias of degeneration during rationale training.

heading: 3.1 Rationale Causal Attribution
text: Formally, we construct a rationale Z∗k by maximizing an attribution metric A(·) in rationalization
Z∗K = argmaxZK⊆XA(ZK ∣ŷc), (2) where A(·) measures the contribution of each candidate ZK to the target prediction ŷc.
However, A(ZK ∣ŷc) needs to be quantified. To this end, we introduce causal intervention calculus do(·), including do(Z = ZK) and do(Z = ∅)(Pearl, 2009; Pearl et al., 2016), and reformulate the causal contribution from ∅ to ZK by mutual information, A(ZK ∣ŷc) = I(ŷc, do(ZK)) − I(ŷc, do(∅)). (3)

heading: 3.2 Markov Decision Process as RL
text: Equation 3 illustrates the procedure for derivingZK from an initial state of zero training. However, it may generate degraded rationales at step i, where 0 < i < K. Thus we need to seek for quantifiable objectives between Zi and Zi+1, Zi+1 = argmaxZi+1∈{X/Zi}A(Zi+1∣Zi, ŷc). (4)
According to Equation 3, we have the causal contribution between Zi and Zi+1: A(Zi+1∣Zi, ŷc) = I(ŷc, do(Zi+1)) − I(ŷc, do(Zi)). So, A(Zi+1∣Zi, ŷc) = −H(ŷc∣Zi+1) +H(ŷc∣Zi)= −H(ŷc∣{Zi ∪ {zi+1}}) +H(ŷc∣Zi) = −pθ(ŷc∣Z)log pθ(ŷc∣Zi)
pθ(ŷc∣{Zi ∪ {zi+1}}) , (5)
where H(ŷc∣Zi) is the term of conditional entropy. As a result, Equation 5 explicitly quantifies Zi+1’s effect with previously obtained rationale Zi.
To further promote the cooperative game, we model the training process of rationale as a Markov decision process M = {S,A,P,R}, where S = {si} represents set of states abstracting the process of optimizing rationale during training, and A = {ai} indicates the set of actions. In particular, The transition dynamics P(si+1∣si, ai+1) specify how the state si+1 is updated from the prior state si by taking action ai+1. Besides, R(si, ai+1) quantifies the reward obtained after taking action ai+1 based on the prior state si. Therefore, cooperative training for rationale can be depicted as the sequence process (s0, a1, r1, s1, ..., aK , rK , sK), where the state si can be formulated by si = Zi in the i-th update; s0 = Z0 can be initiated by generator fG(·).
Nevertheless, the above process exhibits a limitation in its inability to detect learning bias at any given state si. To address this, we reformulate the sequence process as (<s0 ,̃a0,r̃0,s̃0>, a1, r1, <s1 ,̃a1, r̃1,s̃1>, ..., aK , rK , <sK , ãK , r̃K , s̃K>), where <si ,̃ai,r̃i,s̃i> indicates process of transitioning from state si to s̃i in the i-th update.
Given the state si = Zi, we derive the available action space: Ãi = {X/Zi}. The searched action can be represented as ãi = z̃i, (6) where z̃i ∈ {X/Zi} indicates candidate rationale in action space. Having made the action ãi, the state transition is to merge z̃i into Zi, i.e., Z̃i = Zi∪{z̃i}.
To assess the effectiveness of the action ãi in mitigating the learning bias of the model, the reward R̃i(s̃i, ãi) at state si can be formulated as follows:
R̃i = { A(z̃i∣Zi, ŷ∗c ) + 1, iffP (Zi ∪ {z̃i}) = ŷ∗cA(z̃i∣Zi, ŷ∗c ) − 1, otherwise. (7) According to Equation 5, although we can quantify the probabilities at states s̃i and si, and present the relevant reward R̃i, obtaining y∗c poses a challenge.

heading: 3.3 Pretrained Agent
text: To address the limitation, we propose a reinforced causal agent in the aforementioned causal and reinforcement learning framework to better align the probability distribution of the target prediction and theoretically justify the creation of an auxiliary agent targeting ŷc.
Pretrained Embedding. We pretrain the auxiliary agent, denoted as fA(·), with
θ∗A = argmin θA H(Y, Ŷ ∣X), s.t.Ŷ = fA(X), (8)
where θA represents the parameters of the agent, and θ∗A denotes the optimal solution.
Theorem Analysis. Assuming X , Z, Y , and A as random variables in rationalization representing the input, rationale, label, and auxiliary variable, respectively, we propose: Lemma 1. Given X , Z, Y , Ŷ = fP (fG(X)). Existing a guiding variable A could enable the predictor fP (·) to achieve good predictions. That is, a solution for A exists, and X is a solution of A.
The proof is provided in Appendix A. Lemma 1 suggests that constructing an auxiliary variableA aligned with X for rationalization contributes to the learning of a good prediction.

heading: 4 Agent-Guided Rationalization
text: As depicted in Figure 2, following the establishment of the environment for the reinforced causal agent, we delineate the construction and training of the policy network qϕ.

heading: 4.1 Policy Network Architecture
text: It takes the pair of intermediate state Zi and ŷc provided by fA(·) as input. Formally, z̃i ∼ qϕ(Zi, ŷc), (9) where θϕ is the trainable parameters of the policy network, and z̃i is generated according to the probability of next action Pϕ(z̃i∣Zi, ŷc).
Representation learning of action candidates. With the space of action candidates Ãi = X/Zi, our policy network first learns the representation for each action candidate ã(j)i (0 < j < N), where N is the number of candidates.
Then, we employ the encoder to encode X/Zi for obtaining the action representation of z̃i by ez̃i = encoder(X/Zi), (10) utilizing bidirectional Gated Recurrent Units (GRUs) (Cho et al., 2014) as the encoder.
Sampling of action. The policy network aims to select a singular action ãi = z̃i from the search space, prioritizing its relevance to the current state si = Zi. This selection process is modeled as: pz̃i =MLP ([ez̃i ;eZi]), (11) where eZi indicates the current rationale’s representation. The selection probability for each action candidate within Ãi is computed using Pϕ(z̃i∣Zi, ŷc) = softmaxÃi(pz̃i), (12) where ϕ is the parameters collected of MLP.

heading: 4.2 Policy Gradient Training
text: Since discrete sampling within the policy network blocks gradients, we adopt policy gradient-based training framework REINFORCE (Sutton et al., 1999). The objective maxΩ(L) is as follows: max ϕ
EZi∈ÃiEi[R̃(Zi, z̃i)logPϕ(z̃i∣Zi, ŷc)]. (13) The final task loss is a jointly optimized objective: min θg ,θp H(Y, Ŷ )+Ω(M)−Ω(L), s.t.Ŷ = fP (fG(X))
(14)

heading: 5 Experiments

heading: 5.1 Datasets, Baselines and Evaluation Metrics
text: Datasets. We compare AGR using BeerAdvocate (McAuley et al., 2012) and HotelReview (Wang et al., 2010) datasets, which are two multi-aspect sentiment classification datasets widely used in rationalization. Following existing work, we obtain the data in the same way as Yue et al. (2023) for BeerAdvocate, and we preprocess HotelReview dataset in the same way as Huang et al. (2021) and Liu et al. (2023b). Baselines. We compare with eight models for BeerAdvocate, including three sampling-based methods: RNP (Lei et al., 2016), HardKuma (Bastings et al., 2019), Information Bottleneck (IB) (Paranjape et al., 2020), and three extra-component-based methods: DARE (Yue et al., 2022), FR (Liu et al., 2022), MGR (Liu et al., 2023b), and two causalbased methods: INVRAT (Chang et al., 2020),
Inter-RAT (Yue et al., 2023). For HotelReview dataset, we compare with five models, including RNP (Lei et al., 2016), CAR (Chang et al., 2019), DMR (Huang et al., 2021), A2R (Yu et al., 2021), and MGR (Liu et al., 2023b). Evaluation Metrics. Following (Huang et al., 2021; Yu et al., 2021; Yue et al., 2023; Liu et al., 2023b), we focus on the quality of rationales, and adopt Precision (P), Recall (R), and F1-score (F1) as metrics. We perform the best results on the validation set before testing on the test set. The Appendix B provides further details in this section.

heading: 5.2 Performance Comparison
text: Results on BeerAdvocate. As shown in Table 1, our proposed method AGR outperforms all the eight baselines in terms of three aspects for BeerAdvocate dataset. Furthermore, in sparsity experiments (Table 2), AGR consistently outperforms the latest state-of-the-art results, affirming its effectiveness for selective rationalization. Results on HotelReview. Table 3 shows that our model once again obtains the best performance
Table 4: Examples of generated rationales. Human-annotated rationales are underlined. Rationales from three models are highlighted in blue and are denoted as Z1, Z2 and Z3 respectively.
FR (2022) MGR (2023b) AGR (Ours) Aspect: Beer-Appearance Aspect: Beer-Appearance Aspect: Beer-Appearance Label: Positive, Pred: Positive Label: Positive, Pred: Positive Label: Positive, Pred: Positive Text: i picked this beer up on a whim as i was in the mood for a good coffee stout and the siren-like figure somehow told me this is the beer for you . a bit freaky , but i went with it . i was impressed from the very first pour . like any stout , the color is a dark molasses black . but ... the head was thick and dense with good retention . the coffee aroma was intense ! the roasted goodness almost overwhelms my sense of smell .the roasted coffee flavors are the first things that i could taste along with hints of chocolate . however , i can tell there ’s more complexity here than my palette can decipher . the coffee flavors bring bitterness but it ’s not over powering as the sweetness of the malt cuts the bitterness quite nicely the beer has carbonation but once the bubbles have escaped the beer gives a creamy , velvety feel and finish . the alcohol was very well hidden in this beer which is scary ... Text: i picked this beer up on a whim as i was in the mood for a good coffee stout and the siren-like figure somehow told me this is the beer for you . a bit freaky , but i went with it . i was impressed from the very first pour . like any stout , the color is a dark molasses black . but ... the head was thick and dense with good retention . the coffee aroma was intense ! the roasted goodness almost overwhelms my sense of smell .the roasted coffee flavors are the first things that i could taste along with hints of chocolate . however , i can tell there ’s more complexity here than my palette can decipher . the coffee flavors bring bitterness but it ’s not over powering as the sweetness of the malt cuts the bitterness quite nicely the beer has carbonation but once the bubbles have escaped the beer gives a creamy , velvety feel and finish . the alcohol was very well hidden in this beer which is scary ... Text: i picked this beer up on a whim as i was in the mood for a good coffee stout and the siren-like figure somehow told me this is the beer for you . a bit freaky , but i went with it . i was impressed from the very first pour . like any stout , the color is a dark molasses black . but ... the head was thick and dense with good retention . the coffee aroma was intense ! the roasted goodness almost overwhelms my sense of smell .the roasted coffee flavors are the first things that i could taste along with hints of chocolate . however , i can tell there ’s more complexity here than my palette can decipher . the coffee flavors bring bitterness but it ’s not over powering as the sweetness of the malt cuts the bitterness quite nicely the beer has carbonation but once the bubbles have escaped the beer gives a creamy , velvety feel and finish . the alcohol was very well hidden in this beer which is scary ...
Methods AppearanceS P R F1 AGR 20 83.7 87.5 85.6 -w/o causal. 20 81.5 87.8 84.5 -w/o embedd. 20 81.9 86.9 84.3 -w/o both 20 74.3 85.2 79.4
Table 5: Ablation studies on the BeerAdvocate.
across all multi-aspects datasets consistently. Ablation Studies. To further verify the effectiveness of AGR, we conduct the ablation experiments. As depicted in Table 5, removing either the optimized objective of causal effectiveness (referred to as causal.), the pretrained agent embedding (referred to as embedd.), or both, results in a notable decline in AGR’s performance, underscoring the critical roles played by our proposed key components in AGR method. Further Analyses. Firstly, we compare AGR with FR and MGR, providing the visualized examples. For example, we can observe from Table 4 that although all three methods are able to focus on the appearance aspect, FR and MGR still exhibit some degeneration (since the selective rationale still has some distance from the target prediction). However, AGR utilizes causal calculus to capture the causal variations between Z1 and Z2, as well as between Z2 and Z3, regarding the target prediction,
thereby gradually mitigating this degeneration during the training process. The Appendix C presents more visualized examples. Secondly, similar to (Liu et al., 2023b), we also compare the complexity of AGR with other models. As shown in Table 6, we can see that the complexity of AGR has been somewhat improved compared to latest work; however, there is still room for further improvement. This will be a key focus of future research.

heading: 6 Conclusion
text: In this paper, we propose AGR, a reinforced causal agent-based rationalization approach to guide the cooperative game optimization during rationale training. Our theoretical insights underscore the necessity of this guidance signal for accurate predictions. Empirical evaluations on two widely-used benchmarks indicate the effectiveness of our proposed approach, surpassing existing state-of-the-art methods for selective rationalization.
Limitations
There are still some limitations that need further improvement in the future. Firstly, optimizing cooperative game of rationalization during training brings great significance to the model performance, but how to more efficiently search for meaningful actions within a larger search space for good rationales remains the next direction to explore. Nextly, this work does not involve the debiasing techniques of data-level. Considering the debiasing technique may be a good way to further improve the results. In addition, as the latest research (Chen et al., 2022; Liu et al., 2023a,b) has shown that it is still a challenging task to finetune pretrained language models on the cooperative game framework. Therefore, how to incorporate the cooperative framework and (large) language models is a research interest.
Ethics Statement
This paper does not involve the presentation of a new dataset and the utilization of demographic or identity characteristics information.

heading: Acknowledgements
text: We would like to thank all the anonymous reviewers for their valuable feedback. This work was supported by the National Natural Science Foundation of China (Nos.62376144, 62272285, 62076155) and the Science and Technology Cooperation and Exchange Special Project of Shanxi Province (No.202204041101016).

heading: A Proof of Lemma 1
text: Given random variables X , Z, Y , and A, where A is drawn from the distribution of X . According to Section 2, to obtain a good predictor, we have
min θg ,θp H(Y, Ŷ ) = min θg ,θp H(Y, fP (Z)), (15)
where Z = fG(X). It means that we need to minimize H(Y,Z) (Liu et al., 2023b), i.e., to reduce more uncertainty and indicate the label Y . We assume that exist variable A could make to reduce the uncertainty of learning Y , then our goal is to make H(Y,A) ≤H(Y,Z).
According to the mutual information formula, we can obtain:
H(Y ) −H(Y,A) ≥H(Y ) −H(Y,Z), (16) so,
I(Y,A) ≥ I(Y,Z). (17) Next, since we have X = {Z,X/Z} where X/Z
denotes the text derived from X and unrelated to the rationale, so we can obtain mutual information between X and Y ,
I(Y ;X) = I(Y ;{Z,X/Z})= I(Y ;Z) + I(Y ;X/Z ∣Z) (18) According to the non-negativity of mutual information, we have I(Y ;X/Z ∣Z) ≥ 0, so I(Y,X) ≥ I(Y,Z) (19)
Further, we denote I(Y,X) = ε0 ≥ ε1 ≥ I(Y,Z) ≥ ε2, where ε1 and ε2 indicate the upper and lower bounds of I(Y,Z), respectively.
Therefore, we can obtain that when A = X ,the equation I(Y,A) = ε0 ≥ ε1 ≥ I(Y,Z) is satisfied. That is to say, a solution for A exists, and X is a solution of A.
The proof of Lemma 1 is completed.

heading: B Experiment Details
text: B.1 Baselines We compare AGR with the following baselines: RNP (2016), a original RNP sampling method. HardKuma (2019), a kumaraswamy-distributionbased sampling method. CAR (2019), a game theoretic-based approach to class-dependent rationalization. Information Bottleneck (IB) (2020), a model utilizing IB objective for balancing performance and rationale length. INVRAT (2020), a method that introduces an environment-agnostic predictor.
DMR (2021), which proposes a teacher-student distillation framework to align input distribution. A2R (2021), a method that introducing a soft rationale to predictor. DARE (2022), which introduces a guider into predictor to encapsulate more information from the input. FR (2022), a method using a unified encoder for generator and predictor. Inter-RAT (2023), which develops an interventional rationalization to discover the causal rationales. MGR (2023b), a method leveraging multiple generators to select rationales.
B.2 Datasets Following previous research (Huang et al., 2021; Yue et al., 2023; Liu et al., 2023b), we obtain BeerAdvocate and HotelReview datasets. BeerAdvocate (McAuley et al., 2012) and HotelReview (Wang et al., 2010) are publicly available from existing work. As shown in Table 7, the specific splitting details of the two datasets are presented.
B.3 Implementation To fairly compare with previous works and validate the effectiveness of the approach proposed, we utilize the 100-dimension Glove (Pennington et al., 2014) as the word embedding and the 200- dimension GRUs (Cho et al., 2014) encoder to build the generator fG(·) in the AGR architecture. Further generator fG(·) follows Equation 1 for cooperative optimization with predictor fP (·). Meanwhile, we construct the policy network qϕ(·) to collaborate with the generator fG(·) and predictor fP (·) to learn candidate actions in different training states, including the representation learning of action candidates and the sampling of actions. We use Adam (Kingma and Ba, 2015) as the optimizer.

heading: C Additional Examples
text: As shown in Table 8, we provide more examples of selected rationale from the Beer-Aroma and HotelLocation two aspects, where their sparsity is set to be about 20% and 10%, respectively.

heading: Future Work
text: Limitations
There are still some limitations that need further improvement in the future. Firstly, optimizing cooperative game of rationalization during training brings great significance to the model performance, but how to more efficiently search for meaningful actions within a larger search space for good rationales remains the next direction to explore. Nextly, this work does not involve the debiasing techniques of data-level. Considering the debiasing technique may be a good way to further improve the results. In addition, as the latest research (Chen et al. , 2022; Liu et al. , 2023a,b) has shown that it is still a challenging task to finetune pretrained language models on the cooperative game framework. Therefore, how to incorporate the cooperative framework and (large) language models is a research interest