Human-in-the-loop Abstractive Dialogue Summarization
Year: 2023

Abstract:
Abstractive dialogue summarization has received increasing attention recently. Despite the fact that most of the current dialogue summarization systems are trained to maximize the likelihood of human-written summaries and have achieved significant results, there is still a huge gap in generating high-quality summaries as determined by humans, such as coherence and faithfulness, partly due to the misalignment in maximizing a single human-written summary. To this end, we propose to incorporate different levels of human feedback into the training process. This will enable us to guide the models to capture the behaviors humans care about for summaries. Specifically, we ask humans to highlight the salient information to be included in summaries to provide the local feedback, and to make overall comparisons among summaries in terms of coherence, accuracy, coverage, concise and overall quality, as the global feedback. We then combine both local and global feedback to fine-tune the dialog summarization policy with Reinforcement Learning. Experiments conducted on multiple datasets demonstrate the effectiveness and generalization of our methods over the stateof-the-art supervised baselines, especially in terms of human judgments1.ive dialogue summarization has received increasing attention recently. Despite the fact that most of the current dialogue summarization systems are trained to maximize the likelihood of human-written summaries and have achieved significant results, there is still a huge gap in generating high-quality summaries as determined by humans, such as coherence and faithfulness, partly due to the misalignment in maximizing a single human-written summary. To this end, we propose to incorporate different levels of human feedback into the training process. This will enable us to guide the models to capture the behaviors humans care about for summaries. Specifically, we ask humans to highlight the salient information to be included in summaries to provide the local feedback, and to make overall comparisons among summaries in terms of coherence, accuracy, coverage, concise and overall quality, as the global feedback. We then combine both local and global feedback to fine-tune the dialog summarization policy with Reinforcement Learning. Experiments conducted on multiple datasets demonstrate the effectiveness and generalization of our methods over the stateof-the-art supervised baselines, especially in terms of human judgments1.

heading: 1 Introduction
text: Abstractive conversation summarization, which aims at processing, organizing, and distilling human interaction activities into natural, concise, and informative text (Murray et al., 2006; Wang and Cardie, 2013), is one of the most challenging and interesting tasks in text summarization. Growing attention has been paid to neural abstractive conversation summarization through a variety of designs including transferring document summarization models (Gliwa et al., 2019; Yu et al., 2021;
1The data and codes are available at https: //github.com/SALT-NLP/Human_in_the_Loop_ Conversation_Summarization
Jia et al., 2022), utilizing conversational structures (Chen and Yang, 2020; Feng et al., 2020b; Zhu et al., 2020a; Chen and Yang, 2021b; Liu et al., 2019b; Lin et al., 2022; Zhang et al., 2022; Liu et al., 2021), introducing conversational data augmentation (Chen and Yang, 2021a), incorporating controllable signals (Narayan et al., 2021b; Wu et al., 2021) and pre-training conversation models (Zhong et al., 2021). Most of them are trained with supervised learning, which maximizes the log probability of human written summaries. While they have gained impressive performances, there are still huge gaps in generating high-quality summaries as determined by humans such as coherence or faithfulness(Chen and Yang, 2021b), largely due to a misalignment between the fine-tuning objective (maximizing the likelihood of single human-written summary) and the actual needs (generating more human-favored summaries) (Ziegler et al., 2019).
To train the summarization models on objectives that can more closely capture the behaviors humans care about, Reinforcement Learning (RL) has been used to directly optimize the rewards learned and constructed from human feedback (Ziegler et al., 2019; Stiennon et al., 2020; Böhm et al., 2019; Ye and Simpson, 2021). Different kinds of feedback have been explored to construct the reward functions such as human ratings over CNN/DM summaries (Böhm et al., 2019), overall preferences among pairs of summaries (Ziegler et al., 2019), and the similarity-redundancy matrix (Böhm et al., 2019). While achieving promising performances, they are mainly designed for document summarization with a single reward function learned from overall assessments on summaries(Böhm et al., 2019; Ziegler et al., 2019). As a result, they might not be directly adapted to dialogue summarization because of the intrinsic differences between documents and conversations. Compared to documents, conversations are generally less structured and more complex (Chen and Yang, 2020). There
9176
are diverse interactions between multiple speakers and complex structures such as interruptions, discourse relations, and speaker roles in dialogues (Chen and Yang, 2020). Therefore, more subtle levels of human feedback with the consideration of conversation structural information is needed to provide more comprehensive, consistent, and generalizable rewards, which may lead to better performances for dialogue summarization.
To fill in this gap, we introduce Human-In-TheLoop (HITL) abstractive dialogue summarization with different levels of human feedback to leverage various conversation structures. Specifically, we incorporate two levels of human feedback: (1) Local Feedback, which consists of highlighted words or phrases in dialogues to capture the salient structural information, including but not limited to speaker’s intents, identifiable events/topics, and discourse relations (e.g., causal relationships and important emotions), and (2) Global Feedback, which includes dimensions like Coherence, Accuracy, Coverage, Concise and the Overall Quality, to provide more comprehensive human preferences on the given summary. We hire and train human annotators to provide the introduced two levels of human feedback on 1,000 randomly sampled conversations from the DialogSum dataset (Chen et al., 2019). With the collected human feedback, we construct the local reward (rl) based on the similarities between the generated summaries and the annotated highlights and learn the global reward (rg) models via supervised learning which predict the human preferences. Finally, we train the summarization policy via RL to maximize the rewards predicted by rl and rg. Specifically, the policy generates a token of text at each time step and is updated using the PPO algorithm (Ziegler et al., 2019) based on the reward given to the entire generated summary. We conducted extensive experiments and ablation studies in different settings on the recent conversation summarization dataset, DialogSum (Chen et al., 2019) and SAMSum (Gliwa et al., 2019), to demonstrate the superiority of our methods compared to the state-of-the-art supervised learning baselines, especially in terms of human judgments and generalization abilities.
To summarize, our contributions are: (1) we introduced and collected the local and global feedback tailored for abstractive conversation summarization; (2) we designed the HITL to learn better conversation summarization policies via rein-
forcement learning where different levels of human feedback are directly optimized; (3) we performed extensive experiments to study the effectiveness and generation abilities of our HITL methods on DialogSum and SAMSum datasets.

heading: 2 Related Work

heading: 2.1 Abstractive Dialogue Summarization
text: Neural abstractive dialogue summarization has received intensive attention recently with the introduction of large-scale datasets (Gliwa et al., 2019; Chen et al., 2019; Tuggener et al., 2021). Besides directly transferring documents summarization methods to conversations (Gliwa et al., 2019), models tailored for conversation have been proposed to achieve better performances (Zhao et al., 2019; Zhu et al., 2020b; Feng et al., 2021), which make use of the rich structured information in conversations such as dialogue acts (Goo and Chen, 2018), key point/entity sequences (Liu et al., 2019a; Narayan et al., 2021a), topic segments (Liu et al., 2019c; Li et al., 2019), stage developments (Chen and Yang, 2020), discourse relations (Chen and Yang, 2021b; Feng et al., 2020a), action mentions (Zhang et al., 2022), and coreferences (Liu et al., 2021). Recent work has also explored learning in a data-efficient way through data augmentation and semi-supervised learning (Chen and Yang, 2021a), generating more controllable summaries (Wu et al., 2021; Narayan et al., 2021b). Moreover, external information such as commonsense knowledge is incorporated to help understand the global conversation context (Feng et al., 2020b). Zhong et al. (2021) pre-trained a language model on conversational data to help the summarization as well.
Most of the current dialogue summarization systems are still trained to maximize the likelihood of human-written text and have led to significant performances, but there is still a huge gap in generating high-quality summaries as determined by humans such as coherence, faithfulness, conciseness, and concreteness (Chen and Yang, 2020). This is mainly due to the misalignment between the training objective and model evaluation. For example, models never plan and look ahead for overall summarization goals. To fill in this gap, we directly learn the summarization policy that maximizes the rewards constructed from human feedback via Reinforcement Learning to generate more human-favored summaries.

heading: 2.2 Learning with Human Feedback
text: Recent research has started to explore incorporating human feedback into the training process to achieve human-preferred systems in different tasks such as dialogue generation (Jaques et al., 2019; Yi et al., 2019; Hancock et al., 2019), story generation (Zhou and Xu, 2020), document summarization (Ziegler et al., 2019; Stiennon et al., 2020; Böhm et al., 2019) and etc. Our work is most related to previous work which utilizes human feedback to train document summarization models with Reinforcement Learning (RL) (Ziegler et al., 2019; Stiennon et al., 2020; Böhm et al., 2019; Ye and Simpson, 2021), where human ratings/comparisons over summaries are usually used to learn the reward models to serve as the value networks in RL. Despite the effectiveness, it is challenging to directly apply them to conversation summarization, largely due to the complex structures in conversations, which requires subtle reward design.
Inspired by these prior work, we introduce two levels of human feedback to guide the dialogue summarization model to generate more human-favored summaries instead of only collecting pairwise-comparing binary global rating annotations, including the (1) Local Feedback which highlights the important conversation structures to summarize and the (2) Global Feedback which consists of different fine-grained dimensions to provide more comprehensive judgments. Our work is also related to using RL to optimize automatic metrics
for summarization, such as ROUGE (Ranzato et al., 2015; Wu and Hu, 2018; Gao et al., 2019; Parnell et al., 2021), while we are directly optimizing human preferences with RL.

heading: 3 Methods
text: In this section, we introduce our Human-in-theLoop conversation summarization (HITL) pipeline (in Figure 1) where we incorporate two levels of human feedback, the local and global feedback, into the learning process. Inspired by Stiennon et al. (2020), our pipeline for abstractive conversation summarization includes 3 stages: (1) Collecting two levels of human feedback from conversationsummary pairs where summaries are generated with baseline models; (2) Learning and designing reward models from two levels of human feedback; (3) Learning the summarization policy which could generate higher-quality summaries as judged by humans against the reward model.

heading: 3.1 Datasets
text: We utilize DialogSum (Chen et al., 2019), a recent large-scale dialogue summarization dataset emphasizing real-life daily conversations, to study humanin-the-loop conversation summarization. We selected DialogSum because the summaries in DialogSum are less extractive with more novel ngrams. The summaries are more compressed compared to the other conversation summariza-
tion datasets (Chen et al., 2019)2, which makes the datasets more challenging and requires human knowledge to generate better summaries.

heading: 3.2 Collecting Human Feedback
text: Here we describe the process of getting the desired global and local human feedback.

heading: 3.2.1 Annotation Setup
text: Sampling Dialogues From this DialogSum dataset, we randomly sample 1,000 dialogues from 13,360 dialogues to collect our designed two levels of human feedback. As the data statistics shown in Table 1, the distribution of our sampled examples is close to that of the original DialogSum dataset.
Baseline Summaries We generate a set of baseline summaries with different models for the global feedback annotation. Specifically, for every dialogue, we generate 4 summaries with 4 different summarization systems: (1) BART-large fine-tuned on SAMSum and XSUM 3 with a 30.4/11.5/24.8 ROUGE score on DialogSum, (2) DistilBART finetuned on CNN/Daily Mail and SaumSUM 4 with a 33.8/13.6/27.8 ROUGE score , (3) BART-large fine-tuned on SAMSum 5 and with a 33.0/13.5/27.0 ROUGE score (4) BART-large-xsum 6 fine-tuned on SAMSum 7 with a 26.6/10.2/21.4 ROUGE score. These different summaries are then compared by humans to provide global feedback.
Hiring and Training Annotators We hire two annotators through Upwork8 and provide them
2The data statistics are shown in Table 7 in the Appendix. 3https://huggingface.co/Salesforce/
bart-large-xsum-samsum 4https://huggingface.co/philschmid/ distilbart-cnn-12-6-samsum 5https://huggingface.co/linydub/ bart-large-samsum 6https://huggingface.co/facebook/ bart-large-xsum 7https://huggingface.co/knkarthick/ meeting-summary-samsum 8https://www.upwork.com
with extensive training for the task. During multiple training sessions, we explain how to highlight salient information and compare summaries using our interfaces. We go through selected example dialogues and discuss with them to resolve inconsistencies and disagreements. To further reaffirm the training, we also perform test runs on the sampled dialogues. From these test cases, we make sure that they annotate the data properly and achieve good agreements. We pay the annotators $25 per hour. We get 41.67 hours of work for the first member and 39.83 hours for the second member 9.

heading: 3.2.2 Local Feedback
text: For the local feedback, we ask annotators to highlight the salient information in the provided dialogues. The highlighted information needs to be helpful in generating a summary. The information can be phrases, sentences, or a couple of words in the given dialogue. Specifically, we ask annotators to look for some important aspects including (1) speaker’s intents, (2) identifiable events/topics, (3) discourse relations such as causal relationships and (4) important emotions in the conversation. For every conversation, we ask the annotator to annotate 3 to 8 highlights. After 3 rounds of training sessions, we examine the quality by asking them to annotate the same set of 50 dialogues and computing the agreement scores between the two annotators (0.865 BERT-score between their annotated spans) 10. We also make sure the highlights match the important information in ground truth summaries (0.792 BERT-score between annotated spans and corresponding summaries) 11. Annotators then annotate the remaining dialogues by themselves independently. After annotation, we collect 6.1 spans for every dialogue with 59.5 words on average.

heading: 3.2.3 Global Feedback
text: After highlighting the salient information, we provide the annotators with 3 pairs of summaries sampled from the set of baseline summaries. We then ask them to make comparisons in terms of Coherence, Accuracy, Coverage, Conciseness, and Overall Quality. For every comparison between summary A and summary B, the annotators need to grade on a scale of 5 points: summary A is mostly
9The interface is shown in the Appendix 10The BERTScore for randomly sampled pairs of spans is 0.573. 11The BERTscore for randomly sampled pairs of utterance and summary is 0.469.
better, summary A is partially better, equal, summary B is partially better, the summary B is mostly better. We provide detailed guidelines to the annotators about those different dimensions12. After 3 rounds of training sessions, we show the annotators 50 dialogues with 150 pairs of summaries and ask them to make comparisons, resulting in 150 comparisons. We then calculate the Fleiss Kappa scores to measure the agreements among different annotators. In the end, we obtain an average score of 0.342 for Coverage, 0.381 for Coherence, 0.376 for Conciseness, 0.373 for Accuracy, and 0.369 for Overall Quality, indicating moderate agreement (Landis and Koch, 1977). Annotators then annotate the remaining dialogues by themselves independently. In total, we collect 3000 pairs of comparisons for every dimension.

heading: 3.3 Method
text: This section focuses on how to incorporate the annotated feedback into the training process to assist the summarization systems in generating more human-favored summaries.

heading: 3.3.1 Rewards Modeling
text: We first describe how to train the reward models and compute the rewards for any given conversation-summary pairs.
Local Rewards Our goal is to encourage the summarization systems to generate summaries that cover the important information mentioned in the dialogues while avoiding redundant information. Thus here we propose to model the local rewards based on these highlights from annotators. For a given conversation C with a set of human-annotated salient spans M = M1:m (e.g., phrases/sentences/words in the dialogues), suppose the model would generate a summary s. We view the list of highlights M annotated by humans as information needed by the summaries, and the other sentences without highlights as possible redundant information set N = N1:n = C − M . We then calculate the local coverage rewards rl(C, s,M) by calculating the cosine distances between the embeddings of the summary and the information in the dialogues:
rl(C, s,M) =
m∑
i
cos(s,Mi)− n∑
j
cos(s,Nj)
(1)
12The detailed guidelines are shown in the Appendix.
Here we embed the summaries and the dialogue information utilizing sentence-transformers (allmpnet-base-v2) 13 (Reimers and Gurevych, 2019).
Global Rewards Generating high-quality summaries with better human preferences is essential for building better summarization systems. To this end, we design the global rewards by learning human preferences from their annotations. For a given set of annotated conversations C = {C1, ..., Cn} with baseline summaries S = {(s11, s12, s13, s14), ..., (sn1 , sn2 , sn2 , sn3 )} with different dimensions of global human feedback, we first learn a set of reward models rgj (C, s; θe, θj) to measures the quality or impact of the dimension j on summary s for a conversation C. Here, θe are the parameters to encode the conversation and summaries; θj stands for the parameters of linear heads for the dimension j, which outputs a scalar on top of θe. Specifically, we initialize θe with a BART-large model fine-tuned on the DialogSum dataset, and randomly initialize θj for every dimension in the global feedback. During training, we train the model to predict which summary in a summary pair {sin, sim} of conversation Ci is preferred by humans, by optimizing the following objective:
L = −E(Ci,sin,sim)∼(C,S)Σj [log(σ(rgj (Ci, s i n;
θe, θj)− rgj (Ci, sim; θe, θj)))]
where sin is the summary preferred by humans. Implementations are shown in Section 4.2. We select the hyper-parameter based on the loss on the validation set (8:2 split), and further evaluate the learned reward models in Section 4.4.
We then combine different dimensions to provide the global rewards rg(C, s):
rg(C, s) = Σjrgj (2)

heading: 3.3.2 HITL Summarization Policy Learning
text: Here we train a summarization policy with human feedback for generating high-quality outputs as judged by humans. We utilize reinforcement learning to learn the summarization policy πRLϕ . Specifically, we initialize the policy with a supervised learning BART-large baseline model πB fine-tuned on DialogSum. We use the PPO algorithm (Schulman et al., 2017) to maximize the rewards from the above local and global reward models rl and rg,
13https://github.com/UKPLab/ sentence-transformers
where each time step is a BPE token 14. The full reward R(C, s,M) is:
R(C, s,M) = wlrl(C, s,M) + wgrg(C, s)
− β log [ πRLϕ (s|C) πB(s|C) ] (3)
We introduce a KL divergence term between the HITL policy and the supervised baseline model (Jaques et al., 2019). This term could prevent the learned policy generating outputs that are too different from the supervised models and encourage the learned policy to explore instead of collapsing to a single model (Stiennon et al., 2020). wl, wg and β are weights to balance different sub-rewards.
Following Stiennon et al. (2020), we use a Transformer with separate parameters from the policy for the PPO value function. And we initialize the value function to the parameters of the reward model. In our experiments, the reward model, policy and value function are the same size.
14The reward model would give the rewards after the entire summary generated. Each episode terminates when the policy outputs the EOS token, and the discount factor γ = 1.

heading: 4 Experiments

heading: 4.1 Baselines
text: We compare our models with several baselines:
• BART-large (Lewis et al., 2020): We utilized BART-large as our backbone model as well as the supervised learning baseline. Utterances are separated by a special token.
• HITL-synthesis: We use heuristics to approximate the local and global feedback, via which we then learn synthesized reward models and the HITL summarization policy. Specifically, for the local feedback, we utilize a greedy algorithm (Nallapati et al., 2016; Zhang et al., 2022) to obtain the synthesis highlights based on ground truth. For the global feedback, we
utilize the randomly sampled utterances as negative summaries compared to the ground truth summary.

heading: 4.2 Implementation Details
text: For the supervised baseline, we initialize the model with BART-large and fine-tune it on the full DialogSum for 10 epochs with a 3e-5 learning rate and 120 warm-up steps. We use a batch size of 8. For the global reward models, we set the hidden size of the linear head 256. We use a batch size of 8 and train the reward model for 2 epochs with a 3e-5 learning rate. For PPO, we initialize our policies with the supervised baseline and our value functions with the reward models. We set γ = 1 and λ = 0.95 for the advantage estimation (Schulman et al., 2015), do 4 epochs of optimization with a batch size of 8 and run for 5,000 episodes. We set wl = 1, wg = 1.5 and β = 0.05 based on grid search among {0.05, 0.5, 1, 1.5, 2, 2.5} for the full reward R. All experiments were performed on 8 NVIDIA V100 (32GB memory).

heading: 4.3 Automatic Evaluation
text: We first evaluated all the models with the widely used automatic metric ROUGE(Lin and Och, 2004) and reported ROUGE-1, ROUGE-2, and ROUGEL in Table 2. We found that the performances were not better for synthesis data when there were less
training data. When there was plenty of synthesis feedback, (HITL-synthesis with Full data) can help improve over the supervised baseline, where the local reward was more important compared to the global reward. After incorporating ground-truth human feedback, our HITL-(rg + rl) model with both global and local rewards achieved the best performances even with less training data compared to synthesis baselines. The local rewards consistently brought in more performance boost because the conversation structural information in local rewards can help the systems more directly capture the important factors in the conversation. This indicates the effectiveness of our HITL framework for conversation summarization, as the human judgements were directly guiding the learning process.

heading: 4.4 Human Evaluation
text: Following Böhm et al. (2019) and Stiennon et al. (2020), we randomly sampled 200 conversations
from the DialogSum test set and asked annotators from Amazon Mechanical Turk to select the preferred summary from pairs of summaries generated by BART-large and HITL-(rg + rl). Turkers were asked to judge coherence, accuracy, coverage, and conciseness 15. To increase the annotation quality, we required Turkers to have a 98% approval rate and at least 10,000 approved tasks for their previous work. Each conversation was rated by three workers, and we used majority voting to decide the preferred summaries. The pay rate was 0.5$ per hit. We measured the agreement by computing the Intra-Class Correlation (ICC) was 0.693, showing moderate agreement (Koo and Li, 2016).
Main Results From Table 3, we observed that summaries from our introduced (HITL-(rg + rl)) are much more preferred (favored in 82% cases) by human compared to supervised baseline (BARTlarge). These significant improvements came from comparably small amount of annotations (1000 dialogues). These indicated that the systems (HITL(rg + rl)) that directly learn from a small amount of global and local human feedback could generate higher-quality summaries with better human preferences compared to supervised baselines.
Evaluating the Global Reward Models Based on human preferences, we further examined the global reward model and compared it with its subdimensions as well as the ROUGE metric. Basically, we assume that the reward model agrees with human preferences when the model is assigning higher scores to these human-preferred summaries. As shown in Table 4, reward models learned from humans generally agree well with humans, where our global reward rg receives the highest agreement rate. This showed the high quality and effectiveness of our global feedback collection as well as the global reward models. As a result, our HITL-(rg+ rl) model achieves better performances compared to baselines. The results also showed the potential of our global reward models to be used to better automatically evaluate the summaries (Fabbri et al., 2020).

heading: 4.5 Generalization
text: We then evaluated the generalization abilities of our HITL summarization system and our learned global reward model rg. We transferred the knowledge
15The guidelines are the same as the guidelines for global feedback annotations described in the Appendix.
learned on DialogSum to another corpus, SAMSum (Gliwa et al., 2019) which summarizes messengerlike conversations about daily topics, such as arranging meetings and discussing events. The good generalization shown below also lower the amortized cost (Rajani et al., 2021) of our methods.
Generalization of HITL models We first directly applied the whole HITL-(rg + rl) models trained on DialogSum to the SAMSum corpus. The results were visualized in Table 5. The zero-shot evaluations on SAMSum got lower ROUGE scores compared to the models trained on SAMSum data, while our best model, HITL-(rg + rl), achieved better performances compared to the supervised baseline model (BART-large). This showed that our policy empowered with human feedback can better generalize to other domains compared to supervised learning models because our policy was learned from rewards that explicitly indicated human preferences. Such rewards are more general to different domains compared to supervised learning objectives which are specific to one dataset.
Generalization of the Global Reward Model We then re-trained the HITL-(rg) policy on SAMSum corpus while we directly utilized the global reward model rg(C, s) learned from human feedback on DialogSum data as the reward functions. We reported the results in Table 5 and observed that the HITL-(rg) outperformed the supervised BARTlarge model on SAMSum in terms of ROUGE scores. This showed that our global reward models rg can be directly applied to other conversation summarization datasets to provide reinforcement learning rewards and boost performance because the global rewards learned from human feedback are implying the qualities of summaries in general rather than being limited to one specific domain.

heading: 4.6 Ablation Study
text: Here we performed two ablation studies to further study the impact of the quality and the quantity of human feedback in our HITL pipeline.
Perturbing the Qualities of Annotations We compared HITL-(rg + rl) policy trained with annotations on the same 400 dialogues of three levels of qualities: (1) Synthesis Annotations as described in Section 4.1, (2) Noisy which was the annotation from annotators without extensive training sessions, (3) Clean which was the annotation after the training sessions. We visualized the comparisons in
Table 6, and found that the performances were better with higher quality annotations. This suggests that the quality of human feedback matters.
Increasing the Annotations We then varied the number of annotations from 400 to 1000 in our HITL-(rg + rl) model in Table 2. The ROUGE scores were higher with more human annotations because of better reward learning and policy learning with more training data. This implies the importance of enough human feedback to learn and design better rewards.

heading: 5 Conclusion
text: In this work, we introduced two levels of conversation human feedback into the abstractive conversation summarization to generate human-preferred summaries. Specifically, we first collected local and global human feedback to design the corresponding reward functions. We then learned the summarization policies via reinforcement learning to optimize the designed rewards. Extensive experiments in different settings and ablation studies on DialogSum and SAMSum corpus via both automatic and human evaluations demonstrated the effectiveness and generalization of our introduced HITL pipeline. For future work, we would like to explore incorporating human feedback in natural languages which are more general and explicit to indicate how to summarize conversations to improve the abstractive conversation summarization.

heading: 6 Limitation
text: In this work, we collect extensive and comprehensive human feedback with high qualities to facilitate our human-in-the-loop conversation summarization framework. While the learned rewards and models are showing good generalization abilities, further attention is still needed to deeply understand what types of feedback or what amount of feedback is necessary. Our current work only considers human feedback collected using the required forms (i.e., rankings and highlighting). We encourage future work to explore how to incorporate human preferences with more open-ended feedback such as through natural languages. Furthermore, we mainly focus on conversation summarization with human feedback in this work, and other types of summarization tasks (e.g., multi-document summarization, email to-do summarization, meeting summarization and etc.) could be further explored to incorporate human knowledge.

heading: Acknowledgements
text: We thank members of the SALT Lab and the anonymous reviewers for their helpful feedback. This work was supported in part by an Amazon Faculty Research Award and an NSF grant IIS-2247357.

heading: A Data Statistics for DialogueSUM and SAMSum

heading: B The Annotation Interface
text: Since we hired and trained our own set of annotators, rather than using a crowd sourcing website such as Amazon Mechanical Turk, we built
our own website to allow for a standardized, customized user interface for all annotators. The website contains the information for highlighting, summary comparisons as well as detailed instructions.
From here we collect local and global guidance. For local guidance, we display one of the dialogues on the website. We ask the user to highlight salient information and then press next. Afterward, we
display 3 pairs of summaries and ask the user to compare the pairs of summaries in 5 different dimensions. Screenshots from the website are shown in Figure 3. Data collected from the website can be easily ported into a central database containing all of our human data.

heading: C Global Feedback Guidelines
text: We provide the annotators with 3 pairs of summaries sampled from the set of baseline summaries, and ask them to make comparisons in terms of Coherence, Accuracy, Coverage, Conciseness, and Overall Quality. For every comparison between summary A and summary B, the annotators need to grade upon a scale of 5 points: summary A mostly better, summary A partially better, equal, summary B partially better, summary B mostly better. We provide detailed guidelines to the annotators about those different dimensions:
• Coherence: Summary is easy to understand and free of English errors. For comparing summaries against each other in Coherence, we ask the annotators to compare the number and severity of grammatical, syntax, and spelling errors of each summary against each other.
• Accuracy: Information that stated in the summary is accurate and does not incorrect information. Summary is not misleading and has too much errors. For comparing summaries against each other in Accuracy, we ask the annotators discover the amount and severity of inaccurate statements that occur in the summaries against each other.
• Coverage: Mentions main information of the conversations. It conveys the most salient information from the dialogue. For comparing summaries against each other in Coverage, we ask the annotators to look at the number of events in each summary. Also taking into the factor of importance of events, we ask the annotator to compare the number of events against the pair of summaries.
• Conciseness: Summary is short and to the point. It does not have too much unimportant information that is not included in the salient information. For comparing summaries against each other in Conciseness, we ask the annotators to mainly look at the length
of the summaries. Then we check if any information doesn’t fit, and penalize as such.
• Overall Quality: We ask the annotator to use all of the above information and other related context to give an overall rating. Even though we asked the annotator to consider all the information, we asked the annotator to factor coverage and accuracy more into their decision for Overall Quality. This is because it is of at most importance for a dialogue summary to accurately summarize the salient information of the dialogue.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 6
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? Not applicable. Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Not applicable. Left blank.
C 3 Did you run computational experiments? Section 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Section 3
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 3
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Section 3
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: We encourage future work to explore how to incorporate human preferences with more open-ended feedback such as through natural languages. Furthermore, we mainly focus on conversation summarization with human feedback in this work, and other types of summarization tasks (e. g. , multi-document summarization, email to-do summarization, meeting summarization and etc. ) could be further explored to incorporate human knowledge