Multi-Dimensional Optimization for Text Summarization via Reinforcement Learning (2024)

Abstract: The evaluation of summary quality encompasses diverse dimensions such as consistency, coherence, relevance, and fluency. However, existing summarization methods often target a specific dimension, facing challenges in generating well-balanced summaries across multiple dimensions. In this paper, we propose multiobjective reinforcement learning tailored to generate balanced summaries across all four dimensions. We introduce two multi-dimensional optimization (MDO) strategies for adaptive learning: 1) MDOmin, rewarding the current lowest dimension score, and 2) MDOpro, optimizing multiple dimensions similar to multi-task learning, resolves conflicting gradients across dimensions through gradient projection. Unlike prior ROUGE-based rewards relying on reference summaries, we use a QA-based reward model that aligns with human preferences. Further, we discover the capability to regulate the length of summaries by adjusting the discount factor, seeking the generation of concise yet informative summaries that encapsulate crucial points. Our approach achieved substantial performance gains compared to baseline models on representative summarization datasets, particularly in the overlooked dimensions.

heading: 1 Introduction
text: Determining a "good summary" extends beyond a single factor, generally embracing multiple dimensions such as coherence, consistency, fluency, and relevance (Kryscinski et al., 2019; Zhong et al., 2022; Liu et al., 2022b; Wang et al., 2023b; Liu et al., 2023a). Despite the remarkable advancements in abstractive summarization, challenges persist in addressing issues such as factual inconsistency, which generates inaccurate information, and irrelevance, which involves omitting crucial details.
Recently, there have been ongoing efforts to focus on such inferior dimensions (Pasunuru and
∗Equal contribution
Bansal, 2018; Gunasekara et al., 2021; Cao et al., 2022; Berezin and Batura, 2022; Wan et al., 2023; Liu et al., 2023b; Nan et al., 2021; Wang et al., 2023b; Chern et al., 2023), and reinforcement learning (RL) is applied as one strategy. Most existing RL approaches use a single reward of the ROUGE score (Lin, 2004), which measures the overlap with the reference summary. However, its subpar quality across various datasets has been frequently underscored (Liu et al., 2023c; Zhang et al., 2024; Goyal et al., 2023).
Pointing out the limitations of ROUGE scores in detecting hallucinations, various studies have focused on addressing this issue. Pasunuru and Bansal (2018) assigned weights to each word to overcome shortcomings of ROUGE, Roit et al. (2023) provided a reward with the natural language inference (NLI) entailment relationship between generated summary and the document, and Gunasekara et al. (2021) provided rewards via Question Answering (QA) model. However, those methods cannot capture summary-intrinsic dimensions, such as fluency or coherence. Addressing shortcomings in one dimension often leads to unintended drawbacks in other dimensions; thus, achieving a high-quality summary generation by balancing multiple dimensions remains challenging (Figure 1).
1 5858
In this work, we introduce multi-objective RL, aiming to generate solid summaries that are coherent, factually consistent, fluent, and relevant. Our RL approach is based on a proximal policy optimization (PPO) (Schulman et al., 2017), and we incorporate four dimensions of a unified multidimensional evaluation metric, UniEval (Zhong et al., 2022), as multiple rewards. We suggest two strategies for optimal rewarding with multiple objectives, namely MDOmin and MDOpro. MDOmin fosters adaptive learning by selecting the lowest dimension score as the reward at each iteration. Meanwhile, MDOpro projects gradient onto the normal plane to handle conflicting gradients in multi-task RL, leveraging a PCGrad (Yu et al., 2020) optimizer. By effectively projecting the gradients of multiple rewards, our method can adjust the learning direction for optimal training. Both strategies aim to enhance deficient dimensions while preserving superior ones during training.
In summarization tasks, unlike typical PPO usage that rewards at each step, the score for a generated summary is obtained only at the end of the episode when the entire summary is produced. KL-penalty replaces the reward per token during episodes; hence, the discount factor can be crucial in obtaining an optimal policy (Kim et al., 2022). Consequently, we investigate how adjusting the discount factor affects the generated summaries, particularly in length.
Our MDO strategies outperform the baseline model in experiments using the representative CNN/DM and BillSum summarization datasets. Notably, our methods significantly enhance the previously inferior relevance dimension, supporting competitive results in other dimensions. Additional examinations, measuring whether the contents of the generated summaries are from the original articles, reveal around 90% coverage with a shorter average length. This outcome implies the capacity of the MDO to create brief yet pertinent summaries.
Our contributions are summarized as follows: • We propose two multi-dimensional optimiza-
tion methods for multi-objective RL, introducing multiple UniEval dimensions as rewards.
• We have empirically verified improvements in deficient dimensions while maintaining competitiveness in superior dimensions across two datasets, outperforming naive MDO methods.
• We find that adjusting a discount factor can
control the generated summary length.

heading: 2 Related Work
text: Dimension-specific text summarization Previous studies have mostly focused on improving specific dimensions of text summarization, such as generating consistent summaries by resolving hallucinations. Wang et al. (2023b) involves a two-stage process where key entities are first extracted in the initial stage, followed by the integration of these entities to generate summaries in the second stage. Wan et al. (2023) altered the decoding strategy using a ranker and lookahead approach to produce the token with the highest faithfulness score. Their methods only considers to generate faithful summaries but overlooks other various dimensions. RL for abstractive text summarization RL methods for text summarization have predominantly utilized the ROUGE score as a reward (Narayan et al., 2018; Chen and Bansal, 2018; Pasunuru and Bansal, 2018; Kryściński et al., 2018; Dong et al., 2018; Paulus et al., 2018; Wang et al., 2018; Parnell et al., 2022). However, recent studies emphasized that the ROUGE score fails to evaluate summaries adequately due to the revealed poor quality of reference summaries in summarization tasks (Liu et al., 2023c; Zhang et al., 2024; Goyal et al., 2023). Moreover, the ROUGE score only calculates the word overlap with the reference summary, failing to evaluate whether sentences are natural or consistent.
Therefore, some researchers have explored the application of the NLI model (Roit et al., 2023) or QA model (Gunasekara et al., 2021) as a reward, which does not solely rely on the ROUGE score. Roit et al. (2023) employs reinforcement learning with an NLI reward, aiming to maintain high consistency by using the entailment relationship between the summary and the document as a reward. Gunasekara et al. (2021) generate questions from both the document and the summaries using a QA model to verify the presence of answers, aiming to enhance precision and recall related to consistency and relevance. Yet, these methods do not comprehensively consider diverse quality dimensions. Multi-objective RL RL with multiple rewards can lead to more efficient model training (Dann et al., 2023). However, multi-reward application in text summarization has not been extensively explored. Pasunuru and Bansal (2018) employ multi-
2
ple rewards such as ROUGE-L, ROUGE-Sal (which weighs vital information), and entailment, but they simply approach as multi-task learning without consideration for finer optimization. Su et al. (2023) utilize multiple RL policies to summarize multiple documents by constructing individual policy models for importance, redundancy, and length. They aim to concisely summarize multiple documents, preventing content overlap and including only salient information. Yet, they did not aim for a comprehensive summary of a single document, as only the importance feature was considered. Unlike their exclusive focus on enabling the model to capture the essential or relevant content, we explore the optimal strategies for multi-objective RL, aiming for well-balanced summarization.

heading: 3 Method Description
text: Throughout the RL process, it is crucial to maintain the fundamental summarization capabilities of the fine-tuned model while simultaneously improving scores across various dimensions. To achieve this goal, we employ proximal policy optimization (PPO) (Schulman et al., 2017) for RL application, utilizing a supervised, parameter-frozen reference model to guide the policy. In our pursuit of multiobjective RL in summarization, we adopt UniEval (Zhong et al., 2022), a metric that evaluates scores across different dimensions using a QA model. Incorporating four dimensions in the rewarding process, we introduce two optimal MDO methods to guide RL policy updates effectively. The entire process is illustrated in Figure 2. Multi-rewards UniEval leverages a QA module for a unified multi-dimensional assessment in the rewarding process. The dimensions tackled by the UniEval closely align with human preferences, evaluating summaries based on key quality indicators. They include coherence (the structural coherence of the summary), consistency (the absence of discrepancies with the main text), fluency (the natural flow of sentences within the summary), and relevance (the inclusion of only important content from the document). PPO PPO stands out as a well-established policy gradient model, renowned for its efficiency and stability attributed to its clipping surrogate objective. This object mitigates abrupt changes during policy updates, ensuring overall stability and avoiding divergence. Given the clipped surrogate objective, 𝐿𝐶𝐿, the value loss, 𝐿𝑉 𝐹 , and the entropy, 𝑆, the full PPO loss at timestep 𝑡 is defined as follows: L𝑡(𝜃) = ?̂?𝑡 [ LCL𝑡 (𝜃) − 𝑐1L VF 𝑡 (𝜃) + 𝑐2𝑆[𝜋𝜃](𝑠𝑡) ] Unlike typical PPO applications that provide rewards at each time step, the summary can only be evaluated once when the entire sentences are generated in the summarization task. Thus, in line with the approach proposed by Stiennon et al. (2020), we employ a supervised fine-tuned summarization model as the policy 𝜋RL. The value model shares parameters with 𝜋RL, with an additional value head. Again, we utilize a reference model 𝜋FT, which is also a fine-tuned summarization model but with frozen parameters, to maintain the summarization performance of the 𝜋RL. In particular, rewards for each action, except for the generation of the last token, is the KL penalty between the policy 𝜋𝑅𝐿 and the reference model 𝜋𝐹𝑇 . This process ensures that the 𝜋𝑅𝐿 does not diverge too far from the supervised fine-tuned summarization model during the RL process. For the final action, which is the selection of the last token of the summary, a total
3
reward is assigned by a reward model, 𝑟(𝑥, 𝑦), for the entire summary: R(𝑥, 𝑦) = 𝑟(𝑥, 𝑦) − 𝛽 log[𝜋RL𝜃 (𝑦|𝑥)∕𝜋FT(𝑦|𝑥)] Generalized advantage estimation (GAE) (Schulman et al., 2016) is used for advantage estimation. Finely adjusting the influence of future reward in GAE is facilitated by employing the discount factor 𝛾 alongside parameter 𝜆 . 𝑥 and 𝑦 denote the document and summary, respectively. The state 𝑠 is the current token, the action 𝑎 is the selection of the next token by the 𝜋𝑅𝐿, and the action space is the vocabulary of the 𝜋𝑅𝐿, 𝑉 .
In our multi-objective setting, the score for each dimension 𝑑𝑘 corresponds to a reward 𝑟𝑘(𝑥, 𝑦). The key focus of our two MDO strategies lies in optimizing these multi-rewards to train the policy effectively. We use online learning, similar to the previous methods (Stiennon et al., 2020), which demonstrated strong performance across various domains (Fan et al., 2023). Algorithm 1 Calculation of MDOmin Input: documents={𝐷1, 𝐷2,… , 𝐷 },1: policy 𝜋𝜃 , model parameter 𝜃, 𝐸𝑣𝑎𝑙𝑢𝑎𝑡𝑜𝑟2: hyperparameter 𝛽, 𝜆, discount factor 𝛾 , 3: 𝐷𝑖𝑚𝑠 ← {“coh”, “con”, “flu”, “rel”} 4:  ← 𝑙𝑒𝑛𝑔𝑡ℎ(𝐷𝑖𝑚𝑠) 5: for 𝑖 = 1 to  do 6: 𝐿 ← 0 7: // Generate a summary 8: 𝑆𝑖 = 𝜋𝜃(𝐷𝑖)9: // Calculate rewards 10: for 𝑗 = 1 to  do 11: 𝑟𝑗 = 𝐸𝑣𝑎𝑙𝑢𝑎𝑡𝑜𝑟(𝐷𝑖𝑚𝑠[𝑗])12: end for 13: 𝑟 = argmin1≤𝑚≤ 𝑟𝑚(𝐷𝑖, 𝑆𝑖) 14: 𝑅 = 𝑟(𝐷𝑖, 𝑆𝑖) − 𝛽 log ( 𝜋𝑅𝐿𝜃 (𝑆𝑖|𝐷𝑖) 𝜋𝐹𝑇 (𝑆𝑖|𝐷𝑖) ) 15: // Estimate advantage ?̂? using GAE 16: 𝛿 ← 𝑟𝑡 + 𝛾𝑉 (𝑠𝑡+1) − 𝑉 (𝑠𝑡) 17: ?̂?𝑡 ← 𝛿𝑡 + 𝛾𝜆𝛿𝑡+1 +⋯ + (𝛾𝜆)𝑇−𝑡+1𝛿𝑇−1 18: 𝐿 ← PPO loss for ?̂?𝑡, 𝑅, 𝜋𝜃19: update Δ𝜃 20: end for

heading: 3.1 MDOmin
text: Focusing on the most vulnerable dimensions, we suggest MDOmin, which selects a minimum dimension score as the reward, 𝑟(𝑥, 𝑦), among the evaluated four-dimensional scores. This approach intuitively aims to uplift the performance of the inferiorquality dimensions. By adopting the minimum score, the model is prompted to perform policy gradients to address the weakest dimension, achieving a balanced summary generation. The same model
evaluates all four dimensions; thus, no scaling is required, and the lowest-rated dimension is directly utilized as the reward. The details of the MDOmin is explained in Algorithm 1. Algorithm 2 Calculation of MDOpro Input: documents={𝐷1, 𝐷2,… , 𝐷 },1: policy 𝜋𝜃 , model parameter 𝜃, 𝐸𝑣𝑎𝑙𝑢𝑎𝑡𝑜𝑟2: hyperparameter 𝛽, 𝜆, discount factor 𝛾 , 3: 𝐷𝑖𝑚𝑠 ← {“coh”, “con”, “flu”, “rel”} 4:  ← length(𝐷𝑖𝑚𝑠) 5: for 𝑖 = 1 to  do 6: 𝐿 ← 0 7: // Generate a summary 8: 𝑆𝑖 = 𝜋𝜃(𝐷𝑖)9: // Calculate rewards 10: for 𝑗 = 1 to  do 11: 𝑟𝑗 = 𝐸𝑣𝑎𝑙𝑢𝑎𝑡𝑜𝑟(𝐷𝑖𝑚𝑠[𝑗]) 12: 𝑅𝑗 = 𝑟𝑗(𝐷𝑖, 𝑆𝑖) − 𝛽 log ( 𝜋𝑅𝐿𝜃 (𝑆𝑖|𝐷𝑖) 𝜋𝐹𝑇 (𝑆𝑖|𝐷𝑖) ) 13: // Estimate advantage ?̂? using GAE 14: 𝛿 ← 𝑟𝑡 + 𝛾𝑉 (𝑠𝑡+1) − 𝑉 (𝑠𝑡) 15: ?̂?𝑡 ← 𝛿𝑡 + 𝛾𝜆𝛿𝑡+1 +⋯ + (𝛾𝜆)𝑇−𝑡+1𝛿𝑇−1 16: 𝐿 ← 𝐿 + PPO loss for ?̂?𝑡, 𝑅𝑗 , 𝜋𝜃17: end for 18: 𝑔𝑚 ← ∇𝜃𝐿(𝜃) ∀𝑚 ∈ 𝐷𝑖𝑚𝑠19: 𝑔𝑃𝐶𝑚 ← 𝑔𝑚 ∀𝑚20: // Project conflict gradient 21: (𝑝, 𝑞) ← select (𝑝, 𝑞) ∈ 𝐷𝑖𝑚𝑠 ×𝐷𝑖𝑚𝑠 where 𝑝 ≠ 𝑞 22: if 𝑔𝑃𝐶𝑝 ⋅ 𝑔𝑞 < 0 then 23: 𝑔𝑃𝐶𝑝 ← 𝑔𝑃𝐶𝑝 − 𝑔𝑃𝐶𝑝 ⋅𝑔𝑞 ‖𝑔𝑞‖2
𝑔𝑞 24: end if 25: update Δ𝜃 = 𝑔𝑃𝐶 = ∑𝑚 𝑔𝑃𝐶𝑚26: end for

heading: 3.2 MDOpro
text: While rewards can be adaptively provided based on individual dimension scores, it may prove insufficient if there exists an inherent trade-off relationship between dimensions. For instance, attempting to improve consistency by including entities from the main document in the summary could potentially reduce the fluency between sentences within the summary. Consequently, finding a Pareto improvement becomes challenging when faced with such inherent trade-offs.
To overcome the intrinsic trade-off relationship, we suggest an MDO𝑝𝑟𝑜, which projects multiple conflicting gradients onto a plane, utilizing the PCGrad optimizer (Yu et al., 2020). Treating multiple dimensions as distinct tasks, the optimizer projects each task’s gradient onto the normal plane of the gradient of other tasks with conflicting gradients. In cases where gradients from multiple losses oppose each other, the learning may become ineffective. The PCGrad optimizer alleviates interference between the gradients of different dimensions by en-
4
suring that the gradient of one dimension does not adversely affect the gradient of others. The detailed process is outlined in Algorithm 2.

heading: 4 Experimental Setup
text: Datasets We utilize two text summarization datasets considering potential influences of source document complexity: the BillSum dataset for legislative content and the CNN/Daily Mail dataset for news summarization. BillSum comprises an 18.9K training set and a 3.2K test set, while CNN/DM has a 287K training set and an 11.5K test set. In light of studies indicating poor quality of reference summaries in the datasets (Liu et al., 2023c; Zhang et al., 2024; Goyal et al., 2023), we use an enhanced version of CNN/DM test set introduced by Wang et al. (2023b). Baseline models As baseline models, we employ encoder-decoder models commonly used for the text summarization task, including BART (Lewis et al., 2020) and T5 (Raffel et al., 2020). For additional comparison, we report PEGASUS (Zhang et al., 2020a) and BRIO (Liu et al., 2022a) results. To ensure comparability, we fine-tune BART-base,
BART-large, and T5-base under the same hyperparameter settings: a batch size of 4, a learning rate of 5e-5, and 10 epochs. For PEGASUS and BRIO models, we utilized already fine-tuned versions on the Billsum1 and CNN/DM23.
In addition, we compare our model with LLMs, GPT-3-CoT (Wang et al., 2023b) and GPT-4 (OpenAI et al., 2024). GPT-3-CoT is a 2-stage chain-ofthought approach where the first stage extracts the core elements, and the second stage integrates them to address the issue of LLMs not sufficiently incorporating elements in generated summaries in the news datasets. We used GPT-4-turbo for GPT-4.
Hyperparameters for RL For RL, we use a batch size of 4, a learning rate of 1.41e-6, discount factor 𝛾 = 0.9, and randomly select only 10K samples from the training set of each dataset. We conduct experiments with three different seeds and report the average scores.
1https://huggingface.co/google/pegasus-billsum 2https://huggingface.co/google/pegasus-cnn_dailymail 3https://huggingface.co/Yale-LILY/brio-cnndm-cased
5
Unimportant contents are highlighted in yellow , and unnatural or structurally disruptive ones are marked in green .
Metrics We use various evaluation metrics for multi-dimension assessment, such as UniEval, ChatGPT, and human evaluations. For detailed measurements on each dimension, we also use QuestEval (Scialom et al., 2021) and BERTSCore (Zhang et al., 2020b). QuestEval assesses precision by generating questions from summaries using a question generation model and checking if the answers are in the document. It generates questions from the document and verifies whether the answers are in the summaries for recall. The overall QuestEval score is an F1 score based on precision and recall. We use precision value for the BERTScore, which calculates the similarity between the token vectors in the generated summaries and those in the reference summaries based on BERT embeddings.

heading: 5 Results
text: Main results In Table 1, our multi-objective optimization techniques, MDOmin and MDOpro, have consistently demonstrate enhanced performance across all UniEval dimensions. Notably, applying to the BART-base exhibit significant advancements in the lowest-quality dimension, relevance, with MDOmin and MDOpro showing increases of 0.027
and 0.034, respectively. Similarly, in the dimension of consistency, also had inferior quality, MDOmin and MDOpro lead to notable improvements of 0.024 and 0.020, respectively. Our methods consistently yield modest yet discernible enhancements even in dimensions with relatively high baseline scores. The same trend is evident in the evaluation of the BART-large model, with considerable strides made in dimensions that initially exhibited lower performance, accompanied by marginal but discernible improvements in dimensions already featuring high scores. This underlines adaptive learning capabilities our methods, enabling the model to dynamically adjust its focus and balance diverse dimensions with the overall enhancements. In the assessment using alternative metrics such as QuestEval and BERTScore, the BART-large+MDOmin model stands out. These results highlight that our generated summaries maintain competitive quality even when measured based on the original document and the reference summaries. The standard deviation is specified in Appendix A.1.
We extend our experiments to include the CNN/DM dataset. As illustrated in Table 2, training with multi-dimensional optimized methods enhances the performance on the CNN/DM dataset akin to those observed on the BillSum dataset. Notably, substantial score improvements are recorded in the dimensions of fluency and relevance, registering increases of 0.046 and 0.048, respectively, addressing areas where the quality was initially deficient. Still, the scores remained comparable or slightly lower in dimensions where the model already demonstrated high proficiency, such as coherence and consistency. Consequently, RL with MDO has resulted in well-balanced summaries across various dimensions.
As LLMs have demonstrated superior performance in summarization tasks (Zhang et al., 2024; Goyal et al., 2023; Pu et al., 2023), we compare our
6
model with the latest LLMs, GPT-3+CoT (Wang et al., 2023b) and GPT-4 (OpenAI et al., 2024). Despite the smaller model size, our method exhibits comparable performance to the larger and more expensive GPT-4 with only 0.018 differences in BillSum (Table 1). Moreover, it shows higher BERTScore in BillSum and CNN/DM (Table 2).
Figure 3 illustrates actual changes in summaries as multi-dimensional scores increase with our model. The initial models frequently incorporated irrelevant details and awkwardly constructed sentences. In contrast, our model, fine-tuned to enhance each dimension through MDO, effectively omits non-essential information and improves the natural flow of sentences. Qualitative observations suggest a positive link between improving UniEval scores and producing high-quality summaries. ChatGPT evaluation Recent studies informed that ChatGPT’s evaluation capabilities closely align with human judgments (Gao et al., 2023; Chiang and Lee, 2023; Wang et al., 2023a). To further verify with indicators other than the QA-based metrics, we include ChatGPT evaluation with four dimensions identical to those in UniEval. Inputting the document and its summaries into ChatGPT, we request evaluations for each dimension on a scale ranging from 0 to 5 (the highest) using detailed prompts. As depicted in Figure 4, the model with MDOmin and MDOpro exhibits improvements across all evaluated dimensions compared to the baseline model, particularly demonstrating a noteworthy 11.1% and 8.3% increase in the lowest-rated dimension, consistency. The prompts are shown in Appendix B. Human evaluation Given that the Englishwritten BillSum dataset has congressional information, we hired three experts who are native English speakers and possess extensive experience with government documents via Upwork4. We follow the evaluation criteria outlined in Roit et al. (2023), which employed NLI-based RL: comprehension, attribution, salience, and conciseness. comprehen-
4https://www.upwork.com
sion assesses the ease of understanding the summary, attribution gauges the consistency of the summary with the main document, salience determines whether the summary includes only the most important information, and conciseness evaluates the brevity of the summary. As outlined in Table 3, our model surpasses the baseline across all dimensions. Moreover, evaluators preferred summaries generated by our model over those produced by the baseline model, as depicted in Figure 5. To verify our methods, we conduct significance tests on the BillSum dataset for both human evaluation results and the UniEval overall score. The results of the two-tailed paired t-test, with p-values < 0.05, demonstrate statistically significant performance differences in MDOmin and MDOpro compared to the baseline model, BART. Mechanical analysis Recent studies (Liu et al., 2023c; Zhang et al., 2024; Goyal et al., 2023) pointed out that reference summaries generally exhibit low quality; thus, ROUGE, which solely relies on overlap with reference summaries may not accurately capture the true quality of the summaries. Nevertheless, we assess our model using traditional evaluation metrics, including ROUGE, coverage, and the average summary length. Summaries of our model show relatively lower ROUGE scores (Table 4), yet the comparative coverage, which
7
calculates the proportion of tokens in the generated summary that are present in the document.
Meanwhile, models with MDO produce shorter summaries compared to those generated by base models. Comprehensive results of the substantial coverage, high relevance and salience scores (Table 1, 3) imply that our shorter summaries concisely encapsulate only the essential contents from the document. In contrast, summaries generated by PEGASUS average around 193 words, which is excessively long for a summary. As demonstrated by Guo and Vosoughi (2023), lengthy summaries are favorable in mechanical metrics like ROUGE. Further, Roit et al. (2023) reported a decrease in entailment percentage as the token length increases. The observed trends persist in our results, where PEGASUS, producing the longest summaries, shows the highest ROUGE scores.

heading: 6 Discussions
text: Summary length varies by text complexity In text summarization tasks, concisely encapsulating only the critical information is crucial. However, the optimal length of a summary depends on the document’s informational content, resulting in varying ideal lengths across datasets. When a document
contains rich information, its summary tends to be longer; conversely, a document with less information leads to a shorter summary. The CNN/DM news dataset includes less information, allowing for the essential contents to be sufficiently covered in a shorter length. On the other hand, the legislative dataset, BillSum, characterized by longer texts and a greater volume of information, tends to yield longer summaries for all models, as revealed in Figure 6. Remarkably, our models consistently produce short yet concise summaries for both datasets, while the PEGASUS model outputs severely lengthy summaries when the data complexity increases. Discount factor affects summary length We investigate the impact of a discount factor 𝛾 on the length of the generated summaries. A clear pattern is found in our empirical experiments on the BillSum dataset – a larger discount factor results in shorter summaries (see Figure 7). This phenomenon can be attributed to the training process of the policy model, particularly its emphasis on the relevance dimension. When estimating the advantage 𝐴, a larger 𝛾 places more emphasis on future rewards. As the reward for the last token is determined using UniEval, and relevance often receives the lowest score among dimensions, the training focus may lean heavily towards optimizing relevance. Consequently, the model tends to anticipate higher
8
scores by generating concise summaries that mostly include only the most crucial sentences, aligning with relevance’s evaluation criteria of containing essential information. Thus, a larger discount factor is expected to generate shorter summaries in this specific context. Comparison with naive approaches When using RL in Language Models, careful attention should be paid to training, as models have the potential to diverge easily, and the value model may fail to converge properly. Considering the intricacy of multi-reward optimization, we conduct additional experiments, emphasizing the need for specialized optimization for multiple rewards. We explore straightforward optimization strategies, such as summing the rewards for each aspect score to formulate the final reward (MDOsum-r) and aggregating the losses for each aspect score, akin to conducting multi-task training (MDOsum-l). However, employing the MDOsum-r method amplifies the performance gap between dimensions, making the superior ones better while the inferior ones (fluency) worse, thereby boosting the imbalance. MDOsum-l, a naive multi-task approach, shows improved results over the baseline but fails to outperform MDOmin and MDOpro (Table 5). These findings highlight the importance of our adaptive optimization strategies for multi-objective RL compared to simple multi-rewarding.

heading: 7 Conclusion
text: This work aims to elevate the summary quality on diverse dimensions by introducing optimized multi-objective RL strategies. With the adoption of UniEval, we incorporate the assessed fourdimensional scores of summaries for rewarding. In particular, we propose two multi-dimensional optimization (MDO) strategies, aiming to learn the optimal policy during the multi-objective RL process. Our MDO strategies exhibited improved performance across all dimensions, and human-evaluated results further proved the capacity to generate balanced summaries. Comparisons with the naive summing of rewards or losses imply that our finer optimization strategies facilitates the efficacy of RL in summarization.

heading: Limitation
text: In this work, we solely utilize UniEval, an opensource evaluation metric, for multi-dimensional evaluation due to its strong correlation with human judgment. However, our approach could be extended and applied if additional evaluation metrics for multiple dimensions become available. As a future work, combining multiple metrics for each single dimension can be further considered as in Wan et al. (2023). We explored the relationship between the discount factor and summary length, yet did not investigate how it practically affects performance enhancement. Observing how performance varies by adjusting the discount factor could be an intriguing topic. Also, we employ MDO on the open-source small encoder-decoder models, considering their cost-effectiveness. This choice is attributed to our main goal of showcasing the applicability of multi-objective RL in summarization tasks. However, given the model-agnostic nature of MDO, implementation with other LLMs is feasible; thus, our method can be extended in future works.

heading: Ethical Statement
text: We utilized public datasets such as BillSum, CNN/DM, and CNN/DM element-aware test sets in our research. For the human evaluation conducted through Upwork, we compensated fairly for the assessments. A total of $50 was paid per person as a fixed prize for evaluating three summaries per document across ten documents, covering four dimensions and preference assessments.

heading: Acknowledgements
text: This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2023-00217286), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.RS-2019II191906, Artificial Intelligence Graduate School Program (POSTECH)), and the Technology Innovation Program (20015007, Development of Digital Therapeutics of Cognitive Behavioral Therapy for treating Panic Disorder) funded By the Ministry of Trade, Industry & Energy (MOTIE, Korea).

heading: A Detailed Experimental Results

heading: A.1 Standard deviation
text: We evaluated the standard deviation for the experiments in Table 1 and Table 2. The standard deviation results for each dataset are reported in Table 6 and Table 7, respectively. A.2 Performance variation according to the
size of the value model We investigated whether the size of the policy and the value models influence the performance improvement extent in MDO. The UniEval, used as
our reward, is based on the T5-large with 770M parameters. Compared to the reward model, the value models of BART-base (139M) and BART-large (406M) have smaller parameters. Consequently, it might be challenging for the value model to accurately predict rewards due to its relatively smaller size than the reward model. As shown in Figure 8, the closer the value model’s size to the reward model’s size, the higher the performance improvement over the baseline. A.3 Performance differences based on the
base optimizer of PCGrad In the MDOpro, we utilized Adam as the base optimizer for PCGrad. The Adam optimizer adjusts the size of parameter updates based on the gradient magnitude, which results in significantly better performance compared to the SGD optimizer in the MDOpro method that involves gradient projection (Table 8).

heading: A.4 Details of used metrics
text: • UniEval (Zhong et al., 2022): Evaluation model, which evaluates four dimensions with a single model. Each dimension is trained with questions and answers using T5. Scores for each dimension are calculated by inserting a prompt along with the summary.
• QuestEval (Scialom et al., 2021): Utilizes a question generation model to create questions from the document and checks if the answers to these questions are present in the summary, calculating recall. Conversely, it generates questions from the summary to check if the answers to these questions are present in the text, calculating precision.
• BERTScore (Zhang et al., 2020b): Calculates precision and recall through the cosine similarity between the token embeddings of the generated summary and the reference summary.
• Coverage: Measures whether each token of the generated summary is present in the document. Unlike exact copy, this metric is finely calculated through lemmatization and case conversion using the NLTK5 library.
• ROUGE6: Counts the number of overlapping words between the generated summary and the reference summary.
5https://www.nltk.org 6https://huggingface.co/spaces/evaluate-metric/rouge
12
• Summary length: Counts the total word of the summary.

heading: A.5 Hardware usage
text: For MDO, we used NVIDIA A100-SXM4-80GB, and for fine-tuning the baseline models on text summarization, we utilized NVIDIA RTX A5000.

heading: B Detailed Evaluation Setup

heading: B.1 ChatGPT evaluation
text: For the ChatGPT7 evaluation, we specified how each summary should be assessed. Providing a detailed description of the dimensions enables Chat-
7https://chat.openai.com
GPT to assess each dimension properly. Scores were assigned on a scale from 0 to 5 (the highest) points. When given detailed prompts to evaluate each dimension, ChatGPT provides scores for each dimension along with explanations for its evaluations. For instance, if the summary includes incorrect information, such as hallucinations, ChatGPT will measure a low consistency score and provide an explanation for this assessment. The details of prompts are in Table 9.
13

heading: B.2 Human evaluation
text: For our human evaluation, we hired three Englishnative experts through Upwork. We provided detailed scripts on how each dimension should be evaluated. Instead of using the dimensions of coherence, consistency, fluency, and relevance measured by UniEval, which we used as rewards, we followed the human evaluation dimensions used by Roit et al. (2023). As the four dimensions used for our rewards are core elements in assessing the summary quality, we assumed that optimizing all four core elements would likely lead to positive evaluations in other unused dimensions as well. The detailed description we provided for human evaluation is illustrated in Table 10.
14

heading: Future Work
text: As a future work, combining multiple metrics for each single dimension can be further considered as in Wan et al. (2023). We explored the relationship between the discount factor and summary length, yet did not investigate how it practically affects performance enhancement. Observing how performance varies by adjusting the discount factor could be an intriguing topic. Also, we employ MDO on the open-source small encoder-decoder models, considering their cost-effectiveness. This choice is attributed to our main goal of showcasing the applicability of multi-objective RL in summarization tasks. However, given the model-agnostic nature of MDO, implementation with other LLMs is feasible; thus, our method can be extended in future works