Inverse Reinforcement Learning for Text Summarization
Authors: Yu Fu
Deyi Xiong
Yue Dong
Venue: EMNLP 2023 Findings (2023)
Keywords: Abstractive Summarization
Inverse Reinforcement Learning
Reward Function Optimization
TL;DR: We introduce inverse reinforcement learning (IRL) for abstractive summarization, demonstrating improvements over MLE and RL baselines across diverse datasets and model sizes

Abstract: We introduce inverse reinforcement learning (IRL) as an effective paradigm for training abstractive summarization models, imitating human summarization behaviors. Our IRL model estimates the reward function using a suite of important sub-rewards for summarization and concurrently optimizes the policy network.  Experimental results across datasets in different domains (CNN/DailyMail and WikiHow) and various model sizes (BART-base and BART-large) demonstrate the superiority of our proposed IRL model for summarization over MLE and RL baselines. The resulting summaries exhibit greater similarity to human-crafted gold references, outperforming MLE and RL baselines on metrics such as ROUGE, coverage, novelty, compression ratio, factuality, and human evaluations.

Inverse Reinforcement Learning for Text Summarization
Yu Fu
Deyi Xiong" Tianjin University dyxiongetjj. edu. cn
Yue Dong
University of California Riverside
University of California Riverside yue. dengducrrc.ee edu
yfu093eucr.cm edu
Abstract
other quality aspects like fluency, coherence, paraphrasing. Li et al. (2019) and Pasunuru and Bansal (2018) later integrated other types of re- wards such as BERTScore (Zhang et al., 2019) or multiple rewards into the RL training process. However, as reward components increase, their weights must be set manually, relying heavily on the author's experience and making generalization into new domains difficult.
We introduce inverse reinforcement learning (IRL) as effective paradigm for training ab- stractive summarization models, imitating hu- summarization behaviors. Our IRL model estimates the reward function using suite of important sub-rewards summarization and concurrently optimizes the policy network. Ex- perimental results across datasets different domains (CNN/DailyMas Mail and WikiHow) and various model sizes (BART-base and BART- large) demonstrate the superiority our pro- posed IRI. model for summarization and RL baselines. The resulting summaries ex- hibit greater similarity to human-crafted gold references, outperforming MLE and RL base- lines on metrics such as ROUGE, coverage, novelty, compression ratio, factuality, and hu- man evaluations.
In contrast to RL, we argue that inverse rein- forcement learning (IRL) may be more suitable for text summarization. IRL focuses estimat- ing an agent's reward function based on their ob- served behavior, rather than predefining it (Arora and Doshi, 2021). Consequently, IRL can be ad- vantageous in situations where the reward function is not explicitly known (Ghosh et al., 2021) or chal- lenging to define through interactions (Bahdanau et al., 2019). Our experimental results suggest that by using IRL to automatically learn weights over combined summarization subrewards and imitate human generations/cxpert demonstration, we can jointly optimize the reward function and policy network, yielding superior summaries as measured by both automatic and human evaluations.
MLE
1 Introduction
Most fine-tuned abstractive summarization systems (Rush al., 2015; Dou et al., 2021) )) are trained using maximum likelihood estimation (MLE) and the negative likeliicld (NLL) loss. Previous research has demonstrated that MLE training pos- sesses certain vastdvan (1) object mismatch (Ding and Soricut, 2017), where the NLL loss con- centrates on word-level matches, neglecting token rearrangement and paraphrases; (2) exposure bias (Ranzato et al., 2016), the discrepancy between training and inference regarding reference tokens.
More specifically, inspired by Shi et al. (2018) 0018); Ghosh et al. (2021), we integrate IRL into text Summarization, which estimates the reward func- tion for summarization and optimizes the model accordingly. By employing IRL, we gain the abil- ity to dynamically learn the weights of various sub-reward components crucial to the summariza- tion task based on the training data. Once the sub- rewards are defined, the training process with IRL consists of two alternttig phases: the reward up- date phase, which focuses on learning the reward function, and the policy update phase, which aims to optimize the model to maximize the reward. Fig- ure presents an overview of our approach.
To address these issues, reinforcement learning (RL), which optimizes policy networks by directly maximizing the discrete reward, has emerged as an alternative training paradigm for summarization (Paulus et al., 2018; Yadav et al., 2021). Typically. RL-trained summarization models require pre- defined reward function and common practice (Paulus et al., 2018) is to use ROUGE (Lin, 2004). ROUGE-base reward does not, however, consider
Compared to the models trained with MLE or RL, our empirical results on multiple summariza-
"Corresponding author.

---

Summarziation
Model
Reward
Summary
ROUGE
Update Pollcy
Novelty
Expert
Coverage
Compression
Update Reward
Summary
Article
Reward Model
Figure 1: Our proposed framework of inverse reinforcement learning with multiple reward components for summarization. Our model is trained with two alternative phases: (1) Policy Update: using the reward model (left) to update the summarization model (right). (2) Reward Update: using expert demonstration (middle) and trained policy (right) to adjust the reward model (left).
tion datasets and different model sizes suggest that the IRL-trained agent produces summaries that significantly more similar the human-written ref- erence summaries across multiple evaluation met- rics, including ROUGE, coverage, novelty, and compression ratio. In addition, although we did not explicitly incorporate faithfulness as a sub-reward during training, our empirical results indicate that our models exhibit higher level of abstraction, with notably lower decline rate in hallucination. This particular characteristic closely aligns with the behavior observed in human human-generaan generated summaries. Our contributions can be summarized as follows:
for both abstractive summarization (Rush et al., 2015; See al., 2017; Dou et al.. 2021) and extrac- tive summarization (Nallapati al., 2017: Zhong et al., 2020). In spite of the differences in genera- tion paradigms, these models trained with the NLL loss have similar limitations, including exposure bias and objective mismatch (Paulus et al., 2018). To address these challenges, reinforcement learning has emerged as popular alternative for training abstractive (Paulus et al., 2018; Yadav et al., 2021; Dou al., 2021) and extractive summarization sys- tems (Zhong al., 2020; Bian et al., 2022: Zhang et al., 2022). RL-based approaches are designed to optimize a discrete reward, often chosen heuristi- cally, such as ROUGE. For instance, Yadav et al. (2021) propose reward function specifically tai- lored for consumer health question (CHQ) datasets, while Fabbri et al. (2022) construct reward func- tion based on textual entailment and coverage of summaries in semantic spaces. However, manually designing reward functions can limit their general- izability beyond specific domains. PA notable work similar to ours is the model introduced by Böhm et al. (2019), which focuses on learning reward functions rather than manually selecting them for policy optimization. In contrast to our approach, they train neural networks to learn directly from human ratings instead of expert demonstrations, which may result in lower interpretability.
We introduce inverse reinforcement learning into text summarization and define suite of rewards that are important for summarization optimization, which, to the best of our knowl- edge, is the first attempt at this task.
We demonstrate that IRL method is effective at learning the weights for combining different sub-rewards for summarization. allowing to train policies based the training datasets in- stead of manually determining these weights.
By simultaneously estimating the reward func- tion and optimizing the summarization agent with expert demonstrations, we show that the model trained with IRL produces summaries that closely follow human behaviors, in terms of better ROUGE, coverage, novelty, compres- sion ratio and factuality when compared the baselines trained with MLE and RL.
Hallucination The increasing flexibility of text generation models has given rise to concern- ing phenomenon known hallucination (Narayan et al., 2021: Dong al., 2022; Cao al., 2022), where models generate text unsupported by source documents (Maynez et al., 2020). Moreover, it has been observed that conventional evaluation met-
2 Related Work
Summarization Extensive research has been conducted on fine-tuned deep learning approaches

---

rics, such ROUGE, do not effectively capture factuality (Maynez et al., 2020). To tackle issues related faithfulness and factuality in generated summaries, numerous methods have been proposed to enhance the preservation of entity-level informa- tion. For example, Narayan et al. (2021) employ entity chains an intermediate planning stage, while Dong et al. (2022) incorporate entity-level external knowledge into summarization. Instead of designing additional components to explicitly ad- dress hallucination, we adopt an approach similar to Wang and Sennrich (2020) and leverage IRL that naturally mitigates hallucinations by addressing the exposure bias. This perspective offers a novel direc- tion to enhancing the factuality of summarization systems, complementing existing strategies that fo- on entity-level information preservation.
Problem Formulation The task of abstractive summarization can be viewed as a conditional lan- guage generation task. Given a source document x = { ,,,,,..,,y,, with n tokens, the summa- rization task involves learning a conditional prob- abilistic model pa (y|x) that produces a summary
(yn,..., Myy)), where ya is chosen from pre- defined vocabulary V and denotes the parameters of the summarization model. po po(y/x) can be fur- ther decomposed into the product of conditional probabilities for each token based on the previously generated context:
ly] II po y<txx).
po(y x)
Generally, po in Eqn. (1) is trained using the maximum likelihood estimation (MLE) objec- tive with teacher forcing (Williams and Zipser, 1989), which aims to maximize the likelihood of the human-written reference summary y" (vi.... ... Dsy): y log puullvi ... (2)
Inverse Reinforcement Learning The backbone of our approach is inverse reinforcement learning, which has been widely used in a diverse range of ar- eas, including computer vision (Zheng et al., 2021) and NLP (Shi et al., 2018; Wang et al., 2018; Ghosh et al., 2021; Ghosh and Srivastava, 2021; Hao et al., 2022). In NLP, Wang et al. (2018) explore ex- pert demonstrations to train neural networks with learned reward functions for visual story genera- tion. Hao et al. (2022) employ the trained teacher- forcing model the reward function, generating step-wise rewards for text generation. However, these approaches implicitly provide the reward function for policy optimization. Of the closely related works, both Ghosh et al. (2021) and Ghosh and Srivastava (2021) incorporate IRL into their proposed methods. These studies focus on table- to-text generation and program generation from natural language instructions, respectively, while research centers on the summarization task. Given the task differences, we propose distinct sub- reward components tailored specifically for text summarization. Furthermore, we provide a com- prehensive analysis highlighting the advantages of IRL in summarization, particularly concerning n- grams, entities, and hallucinations.
[MLE
. vi-1)
where ly"l denotes the length of the target se- quence.
Reinforcement Learning Due to exposure bias (Ranzato et al., 2016) in MLE with teacher fore- ing, RL has emerged as alternative for training neural summarization models (Paulus et al., 2018). It offers the advantage of directly optimizing dis- crete metries such as ROUGE (Lin, 2004), which considers a certain degree of flexibility for token rearrangement and paraphrasing.
In RL. training, models typically require predefin- ing the reward Rt The reward function R(y) is established by comparing the output sequence y with the ground truth sequence y* using evaluation metries such as the average of ROUGE -1, -2, -L, and F1 scores with respect to the gold references (Narayan et al., 2018; Dong et al., 2018). The ob- jective of RL is to learn a policy that maximizes the predefined discrete metric:
3 Method
This section presents an overview and formulation of summarization, along with discussion of re- inforcement learning (RL.) and our approach to training summarization models using inverse rein- forcement learning (IRL).
CRL = R(y") Σ log p(yllv vi-1)
(3)
where y" is obtained by sampling from the prob- ability distribution in the current policy at each

---

decoding time step, with m' representing the length of y"
for the reward components. For IRL, we assume that the summary is sampled from distribution (o(y), which defined as:
To stabilize training and reduce variance in text generation, the commonly used self-critical policy gradient training algorithm (Rennie et al., 2017) is employed. In this algorithm, two separate output sequences, y" and y", are generated during each training iteration. These sequences represent the sampled output and the baseline output, respec- tively. The baseline output obtained by maxi- mizing the output probability distribution at each time step, which essentially involves performing a greedy search. With this baseline formulation, the reward R can be calculated as R = R(y")- - R(y").
1
poly) - exp(Rs(y)),
(5)
R is defined in Eqn. (4), and Z Jy exp( is the partition function. The training objective, denoted by [()), is to update the weights of sub- rewards in order to maximize the liglliidldoodd of the probability defined in Eqn. (5), computed as:
N
1 N
log Pa (y").
(6)
3.1 Training with Inverse Reinforcement Learning
For a reward component Cj. the gradient can be calculated as follows (Ziebart et al., 2008):
We apply the Maximum Entropy IRL algorithm (Ghosh et al., 2021) to train our summarization agent. The objective is to establish effective re- ward function derived from expert demonstrations, which, in our context, take the form of human- authored reference summaries. We identify crucial reward components for text summarization, includ- ing salience, hovelty/araphrrasngg. compression ra- tio, and content coverage. It's worth noting that we do not claim optimality for the defined sub-rewards, and exploration to better match human preferences is left for future work. Instead, we demonstrate improvements in the summarization agent across various critical measures by defining set of sub- rewards and training the IRL. agent to optimize a linear combination of these sub-rewards.
o5 ( = Ey-ppatt V Vay Ro(l)
(7)
Byrpoly) Va, Ry)
To estimate Eqn. (7), which involves expecta- tions over all possible summaries, we employ im- portance sampling. Specifically, we estimate it by sampling N summaries from the expert demonstra- tions distribution Pdata and M summaries from the policy distribution po(y):
N
V Φ0
(()))
V
Ry
N
M
Bm Vo, os Ro(y
Em Bm
m=1
Training an agent with IRI. involves two phases that are performed alternatively: (1) the reward update phase that focuses on learning the reward function and (2) the policy update phase that fo- cuses finding the optimal agent. During the reward update phase, we utilize the fixed, learned policy to generate a summary. We then update the weights of different sub-reward components by considering the reference summary in the train- ing pair. In the policy update phase, we fix the reward function and employ it to update the policy gradients, refining the agent's performance.
(8)
where
exp Ras (y
3m x
Po py(ym)
y and y are drawn from Pdata and the (y) re- spectively. The full training procedure is illustrated in Algorithm 1. More mathematical details can be found in Shi et al. (2018) and Ghosh et al. (2021).
Note that we employed mixed training (see Ap- pendix A for hyperparameter details) for both RL and IRL training to expedite convergence while maintaining a consistent setting for comparison. While setting the reward function is always chal- lenge in RL, IRL can simply learn a paradigm from expert demonstration, which in turn reduces the inductive bias brought by humans. Our re- sults and analyses in section demonstrate the improvements from multiple perspectives, includ- ing salience, coverage, and faithfulness.
The base of our IRL method consists of sub- reward components C {Ci, C2..... Cx), as claborated in Section 4.2. The IRL reward function is a weighted sum of these components:
(y) = C
(4)
where Φ- - (611 .2.,,,,,0 ..)} is the weight vector

---

Algorithm IRL Training for Summarization
Dataset
Method
R-1f
R-21
R-L
BS
Input:
BART-basc
Pretrained policy (summarization) model po()). Initital reward model (y)
MLE
42.02
19.46
39.04
60.24
CNN/DM
44.47
21.05
41.89
61.62
Labelled samples yx',y") 1i=1-
RL (Equal)
43.42
20.39
40.71
61.08
Policy learning rate a,, reward learning rate 3.
IRI.
44.61
20.14
42.21
62.19
Training epoch H, reward update frequency F.
MLE
40.30
16.76
39.16
69.24
Output:
WikiHow
RL
42.43
16.79
41.04
69.21
RL (Equal)
41.36
16.39
39.92
69.74
Optimal policy model and reward model.
IRL
42.76
16.92
41.29
69.57
1: for < to H do
BART-large
2:
if h%FF then
MILE
44.19
21.27
41.24
61.61
3:
for Φ E 02,,,........ } do Get Vo,J(() according to Eqn. (8) and update the reward model:
CNN/DM
44.81
21.07
41.93
60.86
4:
RL (Equal)
41.47
20.39
38.83
60.67
IRI.
46.12
21.98
43.15
63.05
5:
3j - Φ VV T(b))
MLE
42.47
19.02
41.25
70.33
6: end for
WikiHow
42.91
19.18
41.67
70.35
7: fix reward model (y).
RL (Equal)
41.86
18.69
40.61
69.78
8: end if
IRI.
43.59
17.52
42.12
69.64
9: for mini-batch B from fx', y )i= do
Table 1: Comparison summarization models trained
10: Use reward model (y(y) and get LRL
with different learning methods CNN/DM and Wiki- How test sets. We report R (ROUGE)-1,22-4 and BS (BERTScore) measuring the salience model.
according to Eqn. (3) to update thepolicy model:
θ= A VoLL.
end for 13: end for
4.2 Sub-Reward Components
This part provides detailed definition of sub- reward components used in the IRL in Eqn. (4). The sub-reward components encourage the agent to generate summaries that closely align with human- written reference summaries in terms of salience. novelty, coverage, and compression ratio.
4 Experiments and Results
We conducted extensive experiments to examine the effectiveness of the proposed IRL-based sum marization approach against traditional summariza- tion methods. This section will provide details about the datasets, baselines, reward components, experiment settings and results.
ROUGE (Lin, 2004): Encourages generations to match the references in terms of salience, with focus on ROUGE-L as the sub-reward.
Novelty (Chen al., 2020): Ensures similar level of novelty the generation to reference, measured by novel n-grams in the summary.
4.1 Datasets and Baselines
BART-base and BART-large (Lewis et al., 2020) were used as the backbone model for the experi- ments with the Hugging Face MLE implementa- tion. RL (Equal) means using equal weight for every sub-reward components defined in section 4.2 as the final training reward. We carried out experiments on both CNN/DailyMail (See et al., 2017) and WikiHow (Koupaee and Wang, 2018) datasets. Further training and evaluation details are presented in Appendix A.
Coverage (Grusky et al., 2018): Ensures that the generated summaries cover a similar amount of content as the reference, calculated using the word overlap rate between the sum- mary and the original article.
Compression Ratio (Grusky et al., 2018): Maintains balanced word count ratio be- tween the generated/reference summary and the original article.
4.3 Main Results
The results demonstrate consistent superiority the IRL models over both RL and MLE models
tree/v4.9,

---

Dataset
Method
R-L
Nov
Cov
Comp
IRL model generates more abstractive summaries while maintaining a similar coverage to the refer- ence.
REF
100
78.02
83,99
13.48
The only exception the RL model, which achieves better compression results on the Wiki- How dataset. As we train both RL. and IRL models concurenntyy with the MLE model, we consider the MLE result reference point for both models.
MLE
39.01
14.07
99.24
15.22
CNN/DM
RI
41.87
25.97
98.83
16.41
IRI.
42.19
58.78
96.72
13.81
REF
100
95.63
73.54
18.07
MLE
39.16
81.13
89,30
12.70
RL models consistently achieve higher compres- sion results than MLE models, as they primarily optimize for the final ROUGE score. However, our IRL model allows us to adjust the MLE re- sult towards the reference (CNN/DM: 25.22223..,, 13.81; WikiHow: 12.70-15.88, 18.07).
WikiHow
RL
41.03
83.08
91.79
15.88
IRL
41.29
91.32
89.19
14.80
Table 2: Test results on different reward components on BART-base. REF refers to the results of the human- written reference summaries. Matching the REF closely is better bolded. IRL reward components include R-L, Nov (Novelty), Cov (Coverage), and Comp (Com- pression).
4.5 Human Evaluation
In addition, we conducted human evaluation com- paring BART-base trained with IRL versus RL on the CNN/DM dataset. The human judges were presented with reference summaries and genera- tions from different summarization systems in a random and anonymized order. The judges were asked to evaluate which system's 's summary was more similar to the reference overall. They were instructed read the source article only when they were unable to decide needed additional infor- mation."
across datasets and model sizes in the majority of measures. Particularly, the performance improve- ment of the BART-large model is notably signifi- cant for the CNN/DM dataset. On the other hand, is observed that the BART-large model on the Wik- iHow dataset adopted a distinct strategy, leading to a notable improvement in ROUGE-1 at the expense of decline in ROUGE-2 and BERTScore ?. Nev- ertheless, IRL. model consistently outperforms models trained with MLE and RL across most met- ries by effectively balancing and utilizing different sub-reward components across various datasets and models.
IRI. vs. RL
Judge
Judge
Judge 3
Avg.
IRL preferred
56%
53%
58%
55.67%
Table 3: Human evaluation results on 100 randomly sampled examples, accompanied by generations from BART-base trained with IRL RL, presented in a ran- dom and anonymized order. Each example was inde- pendently annotated hy three annotators, resulting in an average pairwise inter-annotator agreement of 57.33%.
4.4 Component-wise Results
The objective of MaxEnt IRI. is to learn reward function that aligns with human behaviors and train the policy generate summaries similar to expert demonstrations. To assess the effectiveness of IRL in optimizing each dimension of summarization, as identified in prior work as characteristic of effective summarization, we conducted experiments to eval- uate the fine-grained performance of BART-base using various training strategies.
Table 3 presents the human evaluation results. With confidence level of 95% and one-sided A/B tests, IRL exhibits significantly higher sim- ilarity to camman-generated reference summaries (p = 0.0246). Furthermore, the preference for IRL (55.67%) surpasses that of RL (47.67%) by a no- table margin of 16.78% Additionally, pairwise inter-annotator agreement was measured, yielding agreement percentages of 55%, 60%, and 57% for the respective evaluations. These findings provide strong support for the IRL. method, highlighting its
We present the results for each sub-reward com- ponent in Table 2. which demonstrate that our IRL model closely aligns with the references in each component, indicating the successful fulfillment of the IRL training objective. Additionally, the cover- age results in Table 2 suggest that models trained with MLE and RL tend to prefer directly copy- ing words from the source article. In contrast, our
3 All judges are native English speakers with minimum of a bachelor's degree and were compensated rate //..5/h. "We made the decision to make reading the source article optional for the judges in order to prevent creating significant cognitive burden and to encourage them to take shortcuts.
htt http:/tttuu.cco/ugggggggggga aegpaangggcee/ttttttttteeee 1.15. (1551//mtriccsberrscooo

---

Method
1-gram
2-gram
3-gram
4-gram
FactCC
KL
CNNDM
WikiHow
MLE: 0.1371 RL: 0.0500
MLE: 0.0126
RL: 0.0165
IRL: 0.0078
IRL: 0.0097
CNNDD
REF
20.61
57.87
75.91
83.94
41,43
0.006
CNNDM
RL
MLE
1.66
8.30
14.25
18.78
82.70
IRL
2.61
15.25
34.30
75.13
0.004
TRL.
5.67
37.33
58.61
70.81
57.21
WikiHow
REF
47.21
85,16
94.90
90.72
88.66
25
50
75
100
125
150
175
200
MLE
31.15
65.80
80.97
72.69
92,93
0.010
RL 25.86
63.69
82.91
85.17
91.55
IRL
29.00
73.26
91.18
90.40
92.21
WkkHow
0.008
0.006
Table 4: Test results in terms of n-gram novelty and FacICC scores. Test results that are more similar the references are bolded.
0.002
255
50
75
100
125
150
175
200
facc
Figure 3: The copy fragments' position/location distri- bution original articles for different models.
80
MLE
70
IRL
CNNDM
Cao et al., 2022; Wan and Bansal, 2022).
Without considering the abstractive level of the generations, it seems that models trained with MLE have the highest faithfulness scores according to Ta- ble 4. However, we also notice that reference sum- maries have lower FactCC scores. The decrease in FactCC might be rooted in reference summaries being more abstract, indicated by the novel n-gram measures (left column). Still, TRL. models tend generate summaries more closely aligned with the reference in terms of novelty and FactCC.
40
93
92
WikiHoww
91
90
86
88
94
96
Novelty
To further measure the abstractiveness vs. faith- fulness trade-off, plot the trade-off curve similar Ladhak et al. (2022) by using the REF the chor. As Figure 2 shows, the curve for our IRL model significantly above both the MIE and RL model, which demonstrates that our IRL model tends to be "abstractive" with the slowest decline of faithfulness.
Figure 2: The FactCC/Novelty curves the CNN/DM and WikiHow dataset. We use the REF result as an anchor to judge three models for trade-off between abstractiveness and faithfulness.
capacity to enhance the quality of summarization outputs.
5 Analysis
5.2 Coverage Analysis
This section provides in-depth analyses of the pro- posed framework from different perspectives, in- cluding faithfulness, coverage, entity-level analysis and weight analysis.
One limitation of the Coverage metric from Grusky et al. (2018) is its disregard for the position of copied text in the original source. We believe that the position of these copied fragments is also cru- cial because a different copy position may increase the Coverage score but widen the gap between the generated summary and the reference summary.
5.1 Results on Hallucination Reduction
Following Wang and Sennrich (2020)'s findings correlating exposure bias with hallucination in NMT, we investigated if IRL can alleviate this problem summarization. We utilized the popu- lar FactCC (Kryšciński et al., 2019) for measuring faithfulness and hallucination (Dong et al., 2022;
The position//ocation distribution can be seen in Figure 3. We limited the maximum location 200, as the remaining locations make up small percent- age. It is clear that the IRL model is more closely aligned with the REF, particularly on the CNN/DM

---

CNNDDM dataset
WikiHow dataset
Novely
Novety Govwrage Compression
Goverage
Compression
weggt
weight
0.2
0.0
2
10 epech
14
38
10
20
0.2
opech
Figure 4: The weight tendency curves during the training with IRL. The x-axis is the training epoch and the y-axis is the weight of corresponding reward. Initially, every reward is equally weighted.
dataset. To further gauge the similarity of the distri- butions, computed the KL-divergence between the models and REF and present the results in Fig- ure A lower score indicates more similar dis- tribution to REF. On both CNN/DM and WikiHow datasets, the IRL. model had the lowest KL score, indicating that it learned policy that closely mim- ies the expert policy used to generate REF, which aligns with its intended purpose.
Method
Precision
Recall
Fl
Length
CNN/DM
MLE
36.97
41.18
41.51
78.68
RL
37.74
45.79
42.99
86.19
IRL
39.68
42.74
43.21
73.94
WikiHow
MLE
7.61
6.63
61.95
46.90
RL
8.25
7.35
59.86
57.36
5.3 Entity-level Analysis
IRL
8.47
6.93
62.25
53.19
Summarization aims condense the main ideas of article into brief summary. We conducted additional analyses evaluate the ability of the models to preserve important information at the entity level, as presented in Table 5. By compar- ing the entities extracted from the generated sum- maries, reference summaries, and original articles, we obtained insightful results. Notably, the IRI. model achieved the highest F1 score, indicating its proficiency in retaining important information. Additionally, the IRL model produced shorter maries with higher concentration of information on the CNN/DM dataset, making an effective approach for summarization.
Table 5: Entity match based spacy. Precision uses the entity number in the model's output as the denomi- nator, while Recall uses the entity number in reference as the denominator. Length represents the average sum- marization length of the corresponding model.
On both datasets, the ROUGE and Novelty com- ponents maintain consistently positive weight throughout the IRL training process. In contrast, the Coverage component gradually diminishes in importance over time. Notably, the IRL models have acquired distinct compression strategies spe- cific to each dataset. In comparison to the MLE results, the IRL models generate shorter summaries for the CNN/DM dataset, while producing longer summaries for WikiHow. Importantly, these vised summaries closely match the reference sum- maries, indicating improved performance.
5.4 Weight Analysis
To gain insights into how the IRL. model effec- tively manages diverse sub-reward components, we plot the weight curves associated with these com- ponents during the IRI. training phase, depicted in Figure 4. The figure reveals compelling ob- servations, illustrating that the weight curves for CNN/DM and WikiHow exhibit similar trends that align with the primary objective of summarization, while also showcasing distinct patterns that corre- spond to the unique characteristics of each dataset.
Conclusion
We introduce inverse reinforcement learning into text summarization and demonstrate the efficiency of this method. Using IRL., we can train a policy to better match human behaviors by learning from expert demonstrations. Our experimental results in-

---

dicate that IRL can improve the summary quality in a variety of measures, including ROUGE, novelty, Coverage, compression ratios, and factuality. Thus, our empirical results suggest that IRL can better fit into the goal of summarization, in addition to providing more interpretable training.
Meng Cao, Yue Dong, and Jackie Chi Kit Cheung. 2022. Hallucinated but factual! inspecting the factuality of hallucinations, in abstractive summarization. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 2350-3354.
Yiran Chen, Pengfei Liu, Ming Zhong, Zi-Yi Dou, Dan- qing Wang, Xipeng Qiu, and Xuanjing Huang. 2020. Cdevalsumm: An empirical study of cross-dataset evaluation for neural summarization systems. arXiv preprint arXiv:0010.05139.
7 Limitations
We only considered four sub-rewards to fit into the summarization task for interpretable results, How- ever, IRL allows for the use of more sub-rewards during training, and as such, there is potential for further exploration in this area. Secondly, we use self-critical policy gradient training as the back- bone RL algorithm, but other advanced algorithms such as PPO (Schulman et al., 2017) could be incor- porated into IRL and summarization in the future. IRL does not restrict the choice of the backbone RL algorithm during the reward update phase.
Nan Ding and Radu Soricut. 2017. Cold-start rein- forcement learning with softmax policy gradient. Ad- vances Neural Information Processing Systems, 30.
Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit Cheung. 2018. Bandit- Sum: Extractive summarization as a contextual ban- dit. In Proceedings of the 2018 Conference on Empir- ical Methods in Natural Language Processing, pages 3739-3748, Brussels, Belgium. Association for Com- putational Linguistics.
8 Acknowledgments
Yue Dong, John Wieting, and Pat Verga. 2022. Faithful to the document or to the world? mitigating hallu- cinations via linked knowledge in abstractive summarization. arXiv preprint arXiv:2204. 102004413766 13761.
The present research was partially supported by the Natural Science Foundation of Xinjiang Uygur Autonomous Region (No. 2022D01D43) and Zhe- jiang Lab (No. B022KHOABO1). We would like to thank the anonymous reviewers for their insightful comments.
Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhengbao Jiang, and Graham Neubig. 2021. GSum: A gen- eral framework for guided neural abstractive summa- rization. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, pages
References
Online. Association for Computational Linguistics.
Saurabh Arora and Prashant Doshi. 2021. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, 297: 103500.
Alexander Fabbri, Xiaojian Wu, Srini Iyer, Haoran Li, and Mona Diab. 2022. AnswerSumm: A manually- curated dataset and pipeline for answer summariza- tion. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, pages 2508 22500 Seattle, United States. Asso- ciation for Computational Linguistics.
Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Seyed Arian Hosseini, Pushmeet Kohli, and Edward Grefenstette. 2019. Learning to understand goal specifications by modelling reward. In 7th In- ternational Conference on Learning Representations. ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Sayan Ghosh, Zheng Qi, Snigdha Chaturvedi, and Shashank Srivastava. 2021. How helpful is inverse reinforcement learning for table-to-text generation? In Proceedings of the 59th Annual Meeting of the As- sociation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 71-79.
Junyi Bian, Xiaodi Huang, Hong Zhou, and Shan- feng Zhu. 2022. Gosum: Extractive summariza- tion of long documents by reinforcement learning and graph organized discourse state. arXiv preprint arXiv:2211. 10247.
Florian Böhm, Yang Gao, Christian M. Meyer, Ori Shapira, Ido Dagan, and Iryna Gurevych. 2019. Bet- ter rewards yield better summaries: Learning to sum- marise without references. In Proceedings of the 2019 Conference on Empirical Methods Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) pages 3110-3120, Hong Kong, China. Association for Computational Linguistics.
Sayan Ghosh and Shashank Srivastava. 2021. Map- ping language to programs using multiple reward components with inverse reinforcement learning. In Findings of the Association for Computational Lin- guistics: EMNLP 2021, pages 1449-1462.
Max Grusky, Mor Naaman, and Yoav Artzi. 2018 Newsroom: A dataset of 1.3 million summaries

---

with diverse extractive strategies. arXiv preprint vrXiv:804.111288
Human Language Technologies, Volume 1 [Long Pa- pers), pages 1747-1759, New Orleans, Louisiana. Association for Computational Linguisties.
Yongchang Hao, Yuxin Liu, and Lili Mou. 2022. Teacher forcing recovers reward functions for text generation. arXiv preprint arX1v:2210006708.
Shashi Narayan, Yao Zhao, Joshua Maynez, Gonçalo Simöes, Vitaly Nikolaev, and Ryan McDonald. 2021. Planning with learned entity prompts for abstractive summarization. Transactions of the Association for Computational Linguisties, 9: 11445-14499
Mahnaz Koupaee and William Yang Wang. 2018. Wiki- how: A large scale text summarization dataset, arXiv preprint arXiv:1810.09300
Ramakanth Pasumuru and Mohit Bansal. 2018. Multi- reward reinforced summarization with saliency and entailment, arXiv preprint arXiv:1800 06451.
Wojciech Kryściński, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Evaluating the factual consistency of abstractive text summarization. arXiv preprint 1tt.:i::111./28/000
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive sum- marization. International Conference on Learning Representations.
Faisal Ladhak, Esin Durmus, He He, Claire Cardie, and Kathleen McKeown. 2022. Faithful or extractive? on mitigating the Bithfulless-aastraaiiveesss trade- off in abstractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pages 1410-1421, Dublin, Ireland. Association for Computational Linguistics.
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech 7aremba. 2016. Sequence level train- ing with recurrent neural networks. In 4th Inter- national Conference on Learning Representations, ICLR 2016.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequene pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880. Online, Association for Computa- tional Linguistics.
Steven J Rennie, Etienne Marcheret, Youssef Mroueh. Jerret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 7008-7022
Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sen- tence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Lan- guage Processing, pages 379-389, Lisbon, Portugal. Association for Computational Linguisties.
Siyao Li, Deren Lei, Pengda Qin, and William Yang Wang. 2019. Deep reinforcement learning with dis- tributional semantic rewards for abstractive summa- rization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-JJCNLP) pages 6038-6044, Hong Kong, China. Association for Com- putational Linguistics.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proxi- mal policy optimization algorithms. arXiv preprint 1uXiv:1707.06344
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer- generator networks. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 1073- 1083, Vancouver, Canada. Association for Computa- tional Linguistics.
Chin-Yew Lin. 2004. ROUGE: package for auto- matic evaluation of summaries. In Text Summariza- fion Brunches Out, pages 74-81, Barcelona, Spain. Association for Computational IaaaaliLing Linguistics.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factu- ality in abstractive summarization. arXiv preprint arXiv:2005.0066..
Zhan Shi, Xinchi Chen, Xipeng Qiu, and Xuanjing Huang. 2018. Toward diverse text generation with inverse reinforcement learning. arXiv preprint 1rX.v:1804.11255
Ramesh Nallapati. Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of doc- uments. In Thirty-first AAAI conference on artificial intelligence.
David Wan and Mohit Bansal. 2022. Factpega- sus: Factuality-awar pre-training and fine-tuning for abstractive summarization. arXiv preprint arXiv:2205.07880
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Ranking sentences for extractive summariza- tion with reinforcement learning. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics:
Chaojun Wang and Rico Sennrich. 2020. On exposure bias, hallucination and domain shift neural ma- chine translation. arXiv preprint arXiv:2005.00 03344.

---

Xin Wang, Wenhu Chen, Yuan-Fang Wang, and William Yang Wang. 2018. No metrics are perfect: Adversarial reward learning for visual storytelling. arXiv preprint arXiv: AXXv:/804.09160
Ronald Williams and David Zipser. 1989. learn- ing algorithm for continually running fully recurrent neural networks. Neural computation, 122):700-20..
Shweta Yadav, Deepak Gupta, Asma Ben Abacha, and Dina
2021. Reinforcement learning for abstractive question summarization with question-aware semantic rewards. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1th International Joint Conference Natural Language Processing (Volume Short Papers), pages 245-255, Online. Association for Computational Linguistics.
Haopeng Zhang, Xiao Liu, and Jiawei Zhang. 2022. Hegel: Hypergraph transformer for long document summarization. arXiv preprint arXiv:2210.04126.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Eval- uating text generation with bert. arXiv preprint arXiv:9904.0675..
Guanjie Zheng, Hanyang Liu, Kai Xu, and Zhenhui Li. 2021. Objective-are aware traffic simulation via in- reinforeement learning. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 3771-3777 Interna- tional Joint Conferences Artificial Intelligence Organization. Main Track.
Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2020. Extractive summarization text matching. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6197-6208, Online. Association for Computational I. Linguistics.
Brian D. Ziebart, Andrew Maas, J. Andrew Bag- nell, and Anind K. Dey. 2008. Maximum entropy inverse reinforcement learning. In Proceedings of the Twenty-Third AAAI Conference Artificial In- telligence, AAAI 2008, Chicago, Illinois, USA, July 13-17, 2008, pages 1433 1438. AAAI Press.

---

A Appendix
A.1 Mixed Training
For abstractive summarization, ROUGE metric is widely used to evaluate the performance of the summarization model. For reinforcement learning, if we simply use ROUGE the reward with only the RL. loss, it may cause too many repetitions in the final output. Following Paulus et al. (2018), we use the MLE loss together with the RL loss for training, as:
Lix = (1 (11-7)) CRL + MLLLE
where is papereparameter. In other words, the LRI used in the paper is actually LMix We set >=0.0016 for the CNN//ailyMial dataset as in (Paulus al., 2018). Similarly, we also set 0000006 for the WikiHow dataset.
A.2 Training and Evaluation Details
We used the BART-base" model as our backbone model. It has around 140M parameters. We per- formed the MLE, RL, and IRL training on four GeForce RTX 2080Ti GPUs with 11 GB of mem- ory each. For the MLE training, scripts provided by the transformers? package. For the RL training, as using the full dataset requires too much time, following (Pasunuru and Bansal, 2018), we used the first 10K examples in the dataset to train the model. The training epoch was set to 20, and the policy learning rate a was set to le-6 for both RI. and IRI. Additionally, for IRL training, we set N M 100 in Eqn. (8) to update the reward. The update frequency used in Algorithm was 1. For all of the training methods, chose the best model based on the ROUGE-L score on the validation set.
followed the
For evaluation, we used the Hugging Face dataset package to get both ROUGE and BERTScore.
"nttpss:/huugiingaace htttss//iigggiaaaeeeeeeeeeeeeeer http://gghuuu..ooom/ngggggggg///
tree/v4.9, 2tt.a///e////yorrrts////eeeeeeee
1.15. 1/datasss


Reviews:
[{"review_id": "oRwAf2TDuy", "reviewer": "Reviewer_B2Fc", "summary": "", "questions": "A. Related to weakness 1: What reward function does the baseline RL model use?  \nB. Related to weakness 3: Other than the task and reward functions, are there any other aspects of the method that differ from the IRL methods in previous studies (Shi et al., 2018; Ghosh et al., 2021)?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Rej1:\n\nIn fact, we conducted experiments assigning equal weights to different reward components, and the results showed a significant disparity from Inverse Reinforcement Learning (IRL). We list the results below:\n\n| Dataset    | Method             | R-1       | R-2       | R-L       | BS        |\n| ---------- | ------------------ | --------- | --------- | --------- | --------- |\n| BART-base  |                    |           |           |           |           |\n| CNN/DM     | MLE                | 42.02     | 19.46     | 39.04     | 60.24     |\n|            | RL                 | 44.47     | **21.05** | 41.89     | 61.62     |\n|            | *RL(equal_weight)* | *43.42*   | *20.39*   | *40.71*   | *61.08*   |\n|            | IRL                | **44.61** | 20.14     | **42.21** | **62.19** |\n| WikiHow    | MLE                | 40.30     | 16.76     | 39.16     | 69.24     |\n|            | RL                 | 42.43     | 16.79     | 41.04     | 69.21     |\n|            | *RL(equal_weight)* | *41.36*   | *16.39*   | *39.92*   | *68.74*   |\n|            | IRL                | **42.76** | **16.92** | **41.29** | **69.57** |\n| BART-large |                    |           |           |           |           |\n| CNN/DM     | MLE                | 44.19     | 21.27     | 41.24     | 61.61     |\n|            | RL                 | 44.81     | 21.07     | 41.93     | 60.86     |\n|            | *RL(equal_weight)* | 41.47     | 20.39     | 38.83     | 60.67     |\n|            | IRL                | 46.12     | 21.98     | 43.15     | 63.05     |\n| WikiHow    | MLE                | 42.47     | 19.02     | 41.25     | 70.33     |\n|            | RL                 | 42.91     | **19.18** | **41.67** | **70.35** |\n|            | *RL(equal_weight)* | *41.86*   | *18.69*   | *40.61*   | *69.78*   |\n|            | IRL                | **43.46** | 16.31     | 41.53     | 69.30     |\n\n*RL(equal_weight)* in the table represent using the fixed weights for every component and update model. We will add the corresponding experimental results in the next version. As those results show, the improvement in IRL effectiveness doesn't simply stem from utilizing multiple diverse rewards, but rather from its ability to automatically determine the weights of various reward components and genuinely obtain an effective reward.\n\nRej2:\n\nThe motivation behind DORB is similar to ours – both aim to balance among multiple rewards. However, algorithmically, we employ the **MaxEnt** framework from IRL, while DORB is based on a bandit algorithm. The key distinction between MaxEnt and Bandit lies in how they update the rewards used in updating the model at each time step.\n\nIn the MaxEnt framework, the rewards used for the model at each time step are a mixture of the rewards learned up to that point. On the other hand, in the DORB approach, when updating the text generation model at a specific time step, only a single reward from the set of rewards is used. Moreover, it's important to note that for the bandit algorithm, the reward used to control bandit updates and choose arm is the average of all reward components, unlike MaxEnt, which continuously learns and adjusts the reward function as it evolves.\n\nTherefore, the **ultimate goal of DORB** remains the maximization of the fixed weight sum of all rewards, rather than obtaining precise weights corresponding to individual reward components as in IRL and combining them into a final reward function.\n\nRej3&wek1:\n\nCompared to the previous work, we have conducted more comprehensive experiments in the field of text summarization. We have also undertaken in-depth analysis of the performance throughout the entire training process. We believe that thoroughly analyzing a framework and dissecting its underlying principles is also a meaningful endeavor.\n\nWek2:\n\nIn the baseline RL approach, we utilized the ROUGE-L score as the reward, aligning with the training setup outlined in the previous paper[1].\n\nTypos Error:\n\nYes! We will fix this error(Eqn.(3)) in the next version.\n\n[1]Romain Paulus Caiming Xiong and Richard Socher. 2018. A Deep Reinforced Model for Abstractive Summarization. In ICLR", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper empirically shows the effectiveness of IRL's automatic weighting of multiple reward functions in text summarization.\n2. This study conducts extensive analysis, detailing in what aspects IRL with multiple rewards improves performance.", "reasons_to_reject": "1. This paper appears to lack a comparison to a baseline model that maximizes multiple rewards using fixed weights. While the paper does have a comparison to the baseline using RL, it does not go into detail about the rewards in this model, which is likely a ROUGE single reward.\nIt has already been reported that maximizing multiple rewards itself is effective in improving performance. For example, Pasunuru and Bansal (2018) shows that simply maximizing different rewards for each mini-batch improves performance more than maximizing a single reward. With the current results, it is not possible to determine whether the proposed IRL model's performance is achieved by maximizing multiple rewards or automatically determining the weights, which is claimed to be the main contribution of the proposed method.\n\n2. There is a missing citation of a closely related work. Pasunuru et al. (2020) uses a bandit algorithm to dynamically determine the optimal weights for multiple rewards. There should be a mention of this study and an explanation of the advantages of the proposed method over it.\n\n3. The proposed method does not seem to differ from the methods in the previous studies mentioned in the paper (Shi et al., 2018; Ghosh et al., 2021), except that the task was changed to text summarization, and the reward function was changed accordingly. Although it is a finding that IRL is effective even in text summarization, the contribution is somewhat small if this is the only difference.\n\nReferences  \n* Ramakanth Pasunuru and Mohit Bansal. 2018. Multi-reward reinforced summarization with saliency and entailment. In NAACL.\n* Ramakanth Pasunuru, Han Guo, and Mohit Bansal. 2020. DORB: Dynamically Optimizing Multiple Rewards with Bandits. In EMNLP.\n* Zhan Shi, Xinchi Chen, Xipeng Qiu, and Xuanjing Huang. 2018. Toward diverse text generation with inverse reinforcement learning. In IJCAI.\n* Sayan Ghosh, Zheng Qi, Snigdha Chaturvedi, and Shashank Srivastava. 2021. How Helpful is Inverse Reinforcement Learning for Table-to-Text Generation?. In ACL.", "questions_to_authors": "A. Related to weakness 1: What reward function does the baseline RL model use?  \nB. Related to weakness 3: Other than the task and reward functions, are there any other aspects of the method that differ from the IRL methods in previous studies (Shi et al., 2018; Ghosh et al., 2021)?", "paper_topic_and_main_contributions": "This paper applies inverse reinforcement learning (IRL) to text summarization. In the text summarization task, there can be multiple reward functions, but it is expensive to manually determine how to weight them. The proposed method uses IRL to learn the optimal weighting of the multiple reward functions automatically.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "* Ramakanth Pasunuru, Han Guo, and Mohit Bansal. 2020. DORB: Dynamically Optimizing Multiple Rewards with Bandits. In EMNLP.", "Typos_Grammar_Style_and_Presentation_Improvements": "* Isn't Eqn. (3) missing minus sign?"}, "extra_text": {}}, {"review_id": "S6uC7nwzuL", "reviewer": "Reviewer_k2JP", "summary": "", "questions": "The contribution is unclear. The IRL algorithm already exists and reward addition is more like hyperparameter tuning.\n\nThe experiments are very less to showcase the effectiveness.\n\nHow does IRL perform on other LLMs?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Rej1&Rej2:\n\nOur fundamental approach involves utilizing IRL to adjust the weights of various reward components. Traditionally, the weight settings for combinations of multiple rewards were treated as hyperparameters, which often posed challenges since the same hyperparameters couldn't be effectively used across different datasets. However, IRL enables the learning of optimal reward weight settings across various datasets and empowers the trained model to achieve performances akin to human-crafted datasets, all while alleviating the need for manual hyperparameter tuning based on the trainer's experience. On the other hand, we focus on interpretability and conduct an in-depth analysis of the IRL method. Our ultimate goal is to attain a model that performs on par with human-crafted datasets, the improvement in the final ROUGE score is an additional gain resulting from obtaining a reward that possesses good interpretability.\n\nCompared to RL, IRL involves an additional step solely for updating reward weights. The time required for this step is negligible in comparison to the overall training time of RL. As a result, the training efficiency of IRL remains consistent with that of RL.\n\nRej3:\n\nWe have achieved significant results on different datasets and models of different scales . This is already sufficient to demonstrate the effectiveness of the IRL method and its future potential. Therefore, conducting additional experiments will not impact the final conclusion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors propose a new model which uses IRL during training. The experiments conducted in the paper cover datasets -- CNN/DailyMail and WikiHow, and they also consider variations in model sizes using BART-base and BART-large architectures. \n\nThe results consistently demonstrate that the proposed IRL model outperforms both Maximum Likelihood Estimation (MLE) and Reinforcement Learning (RL) baselines in the benchmarks discussed.", "reasons_to_reject": "The method of IRL already exists, and the contribution is only towards adding sub-rewards which is not novel enough for a long paper.\n\nSimple contrastive learning already attains an R1 of 46.67 on CNN/Dailymail. So, how is your model better? IRL will take a lot of toll on the training and inference, making it inefficient. \n\nThe benchmarks are only provided over BART. How about other LLMs? Pegasus already attains a higher Rouge score over these datasets.\t", "questions_to_authors": "The contribution is unclear. The IRL algorithm already exists and reward addition is more like hyperparameter tuning.\n\nThe experiments are very less to showcase the effectiveness.\n\nHow does IRL perform on other LLMs?", "paper_topic_and_main_contributions": "The paper presents a novel approach called Inverse Reinforcement Learning (IRL) for training abstractive summarization models. The key contribution of the paper is the introduction of the IRL model, which estimates a reward function using a set of sub-rewards designed specifically for summarization tasks. The model is benchmarked over CNN/Dailymail and Wikihow.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "74VA5Qz1xQ", "reviewer": "Reviewer_D2vy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Rej1:\n\nIndeed, the FactCC score's proximity to REF doesn't necessarily indicate better results. In Section 5.1, we noted a contradiction in our findings, where the IRL model yielded lower FactCC scores, contrary to prior observations [1]. As outlined in [2], there exists a strong correlation between FactCC scores and abstractive tendencies. To address this, Figure 2 presents an abstractiveness vs. faithfulness trade-off curve. This curve helps elucidate why the FactCC scores from our IRL model are relatively lower in absolute terms compared to MLE. Moreover, it demonstrates that our IRL model inclines towards being \"abstractive\" resembling REF more closely while maintaining the slowest decline in faithfulness.\n\nChaojun Wang, Rico Sennrich. 2020. On exposure bias, hallucination and domain shift in neural machine translation. In ACL. [1]\nFaisal Ladhak, Esin Durmus, He He, Claire Cardie and Kathleen McKeown. 2022. Faithful or Extractive? On Mitigating the Faithfulness-Abstractiveness Trade-off in Abstractive Summarization. In ACL.[2]\n\nRej2:\n\nWe provide the training details in Appendix A.2. In the next version, we will provide additional details about the training process and make the source code along with the execution scripts available for reproducible experiments. Additionally, Figure 4 illustrates the weight variation curves of different rewards throughout the entire training process. From these curves, the stability of the training process is evident.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The author proposes the novel approach of using inverse reinforcement learning to enhance the performance of text summarization systems, which is a rare contribution in this field.\n2. The author conducts comprehensive experiments to demonstrate the superiority of their method and provides analysis and reasoning to support their claims. \n", "reasons_to_reject": "1. It is uncertain whether the fact that the FactCC scores of the IRL method are closer to the reference (REF) indicates better fact consistency. The statement suggests that the IRL method performs well regarding fact character but does not explicitly guarantee it. \n2. The article seems to have provided limited details about the training process, such as the hyperparameter settings and training stability. If the source code is available, it would enhance the reproducibility of the study and provide more insights into the training specifics. \n", "questions_to_authors": "", "paper_topic_and_main_contributions": "For text generation tasks such as text summarization, the issue of exposure bias is an unavoidable topic. Previous articles have used reinforcement learning and contrastive learning to address this issue. In this paper, the author proposes using inverse reinforcement learning (IRL) to solve this problem. They set up four types of sub-rewards to combine into a reward function. The approach involves optimizing the weights of the reward function first and then optimizing the behavioral policy based on the reward function to generate higher-quality summaries.\nThe author designs experiments comparing their method with maximum likelihood estimation (MLE) and reinforcement learning (RL) on the CNN/DM and WikiHow datasets to validate the performance of their approach. Through their experiments, the summaries generated by the IRL method are closer to the reference summaries in the four selected evaluation metrics. The IRL method also performs better regarding n-gram novelty and FactCC, while MLE and RL may achieve higher FactCC scores.\nReinforcement learning has been widely applied in summarization, while inverse reinforcement learning in text generation is relatively novel, especially in the context of outline.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper under review introduces a novel approach that applies Inverse Reinforcement Learning (IRL) to text summarization tasks. The main contributions of the paper are the introduction of IRL for text summarization, the design of sub-rewards, and empirical validation on datasets like CNN/Dailymail and Wikihow. While reviewers generally acknowledge the novelty of applying IRL to text summarization and the extensive empirical evidence to support its effectiveness, concerns were raised about the novelty of sub-rewards, the computational efficiency, and some missing details.
Decision: Accept-Findings