Improving Summarization with Human Edits
Authors: Zonghai Yao
Benjamin J Schloss
Sai P Selvaraj
Venue: EMNLP 2023 Main (2023)
Keywords: Human-aligned AI;  Unlikelihood Training;  Human Edits;  Imitation Edits;  Human Feedback
TL;DR: We propose Sequence Alignment (un)Likelihood Training (SALT), a novel technique to use both the human-edited and model-generated data together in the training loop.

Abstract: Recent work has shown the promise of learning with human feedback paradigms to produce human-determined high-quality text. Existing works use human feedback to train large language models (LLMs) in general domain abstractive summarization and have obtained summary quality exceeding traditional likelihood training. In this paper, we focus on a less explored form of human feedback -- Human Edits. We propose Sequence Alignment (un)Likelihood Training (SALT), a novel technique to use both the human-edited and model-generated data together in the training loop. In addition, we demonstrate simulating Human Edits with ground truth summaries coming from existing training data -- Imitation edits, along with the model-generated summaries obtained after the training, to reduce the need for expensive human-edit data. In our experiments, we extend human feedback exploration from general domain summarization to medical domain summarization. Our results demonstrate the effectiveness of SALT in improving the summary quality with Human and Imitation Edits. Through additional experiments, we show that SALT outperforms the conventional RLHF method (designed for human preferences) -- DPO, when applied to human-edit data. We hope the evidence in our paper prompts researchers to explore, collect, and better use different human feedback approaches scalably.

Improving Summarization with Human Edits
Zonghai Yao i
University of Massachusetts, Amherst
zonghaiyao@umasse edu
Benjamin J Schloss
Sai P. Selvaraj Abridge Al Inc.
Abridge Al Inc.
ben. wws.s...chhs@@@mmm @aaillssssssmmmooooooo com
aps www.paaaraaaaaaii..... com
Abstract
models to specific task, they are usually fine- tuned to maximize the likelihood of human-written text. While this strategy has led to markedly im- proved performance in many metrics, models still cannot consistently produce human-determined high-quality output. The NLP community has pointed out some key drawbacks of traditional fine- tuning. First, important errors (e.g. hallucinations) and unimportant errors (e.g. minor grammar errors) equally contribute to the final loss. Second, the model weighs the loss equally on all labeled data of different types, qualities, and difficulties. Third, distribution shifts in new data degrade performance (catastrophic (Kinkpati)KKkkkaatttkkkkk et al., 2017).
Recent work has shown the promise of learn ing with human feedback paradigms to pro- duce human-determined high-quality text. Ex- isting works use human feedback to train large language models (LLMs)) in general domain abstractive summarization and have obtained summary quality exceeding traditional likeli- hood training. In this paper, we focus on a less explored form of human feedback Hu- man Edits. We propose Sequence Alignment Lunelikelihood Training (SALT), a novel tech- nique to use both the human-edited and model- generated data together in the training loop. addition, we demonstrate simulating Hu- man Edits with ground truth summaries com- ing from existing training data - Imitation edits, along with the model-generated summaries ob- tained after the training, to reduce the need for expensive human-edit data. In our experi- ments, we extend human feedback exploration from general domain summarization to medical domain summarization. Our results demon- strate the effectiveness of SALT in improving the summary quality with Human and Imita- tion Edits. Through additional experiments, we show that SALT outperforms the conventional RLHF method (designed for human prefer- DPO. when applied to human-edit data. We hope the evidence in our paper prompts researchers to explore, collect and better use different human feedback approaches scalably.
CC
Ground truth summary
DR: Plus ribavirin, roughly based your weight. Like 3 pills in the morning, 3 pills in the evening.
Start ribavirin 3 pills twiee a day
CCUser
SAT
S
DR: Uh, and have you had any more chest pain?
chest pain
Confirms chest pain.
PT: I did, yeah. I do.
DR: Uh, and have you had any more chest pain?
chest pain
Denies chest pain.
PT: Not really. No.
DR: And
have gemtibrozil
Hish oil.
Fish capsales twice day.
600 mg twice a day.
DR: Fish oil, you do 2 capsules a day.
Table I: Example of conversation-us to notes summariza- tion data from Clinician Conversations (CC) dataset and corresponding human-edit dataset, CCUser, where user- edited summaries- Sg. made from the Al-generated ones- SAA from our SOAP generation pipeline.
1 Introduction
Some works tackle these problems with human feedback (HF). Specifically, they fine-tune lan- guage models with HF using reward learning (Sti- ennon et al., 2020; Ziegler et al., 2019). With a large amount of HF data, these works demonstrate that large-scale LMs, such GPT-3 (Brown al., 2020), have a text generation quality exceeding traditional likelihood training. However, the acqui- sition cost of large-scale HF is high, and whether smaller LMs can also benefit is not fully studied.
Large-scale language model pretraining has be- come increasingly prevalent to achieve high per- formance on various natural language processing (NLP) tasks (Brown et al., 2020; Sanh et al., 2021; Chowdhery et al., 2022; Longpre et al., 2023; Ope- nAI, 2023; Cai et al., 2023). When applying these
Code and the public dataset (Appendix A.2) is at https: earnrroouummnndrii Work was done during internship at Abridge Al Inc.

---

addition, because LLMs are often provided in the form of third-party APIs and are too large for many companies' and labs' infrastructure to host, smaller models (e.g., T5 family (Raffel et al., 2020)) still play important roles in many domains (e.g., medi- cal), where privacy issues and pragmatic economics dominate decision-making strategies.
technique, Sequence Alignment (un)Likelihood Training (SALT), which uses Human Edits and unlikelihood objectives together with the standard likelihood training paradigm to improve the sum- mary quality. Unlikelihood training was proposed to reduce the probability of unlikely tokens pre- dicted by models (Welleck et al., 2019).
In our first experiment, we use the Human Edits from physicians editing Al-generated clinical sum- maries from medical conversations to improve the summarization models. In our second, we explore how we can get similar benefits with pre-existing ground-truth human summaries that are not written as edits to the Al-generated summaries, which we call Imitation Edits. We refer to Al-generated sum- mary SAI human-edit summary Sg, and imitation- edit summary S₁. We show how the unlikelihood objective can be generalized improve the sum- mary quality together with (SAI. Sg) and (S S1) pairs. In addition, our results show that SALT stably improves summary quality for T5 (small and large) summarization models with Human and Imi- tation Edits. Further experiments show how SALT can address the catastrophic forgetting problem arising from the distribution difference between Sa and Sg with the help of RSALT, which is an improved version of the Replay-based methods in Continual Learning (Rebuffi et al., 2017).
Our goal in this paper is to explore methods to train language models to improve the summary quality with HF inexpensively. HF for summa- rization can come in different forms. One is to obtain human scores for the summaries. Previous work (Stiennon et al., 2020; Ziegler et al., 2019) focuses on training a reward function through HI' data and using such rewards as training objectives by comparing different summaries' scores. More recently, this is used by generative AI works (e.g., ChatGPT and GPT4 (Ouyang et al., 2022: OpenAl, 2023)), and they call the method RLHF. Another HF is obtaining edits make the summary correct. natural way to collect
The second approach feedback from users in workflows where users may be working off of an Al-generated summary in their workflow. For example, the summaries Sy. in Table I are the results of clinicians/scribes mod- ifying our Al-generated EHR summaries SAI In addition, the second approach might be more data efficient in improving the summarization models than the first, as it conveys more granular infor- mation than a score for the entire summary. Hu- man Edits from the second approach can also be converted to scores with simple rules like the per- centage of edits, although this has not been studied extensively. Hence, from an ML data point of view, the second approach has certain unique ad advantages. Furthermore, large-scale expert feedback is hard get using annotation ways in RLHE, consider- ing the expert/user's time, cost, and willingness. But, Human Edits, which can be obtained from the users using the AI summaries for their work, may become a more reasonable alternative in various domains.
Finally, to compare SALT and RLHF, we experi- ment with SALT and Direct Preference Optimiza- tion (DPO) (Rafailov et al., 2023) on human edit data and demonstrate the superiority of SALT on this type of human feedback.
To conserve space constraints, we have relegated specific contents to the appendix. Appendix A.1 and A.2, we provide definitions of the SOAP Struc- ture and implementation details. In Appendix A.3, we focus on the utilization of Imitation Edits and SALT for training on publicly available datasets, accompanied by the experimental results. Lastly, in Appendix A.4, we have more discussion about the relation between SALT and various other RLHF's.
In summary, our contributions are as follows: To our knowledge, we are the first to extend current HF trends in summarization research to the automatic clinical note-generatinn task.
We explore how to use Human Edits to improve summary quality. In addition to general domain summarization, we also focus on a medical domain summarization task in automatic clinical note gen- eration from doctor-patient conversations, which is understudied due privacy and data inaccessi- bility problems. Table provides an example of a Clinician Conversation from our dataset (CC). We present our work from two experiments on a novel
Different from the form HF used in previ- ous work, we explore Human Edits to improve summary quality in this paper.
We show how to construct Imitation Edits to reduce the need for expensive HF data.

---

We show SALT extends unlikelihood train- ing into a general framework using sequence alignment and further combines SALT and Replay-based methods (Rebuffi et al., 2017) into RSALT for tackling catastrophic forgetting.
Subsection For S. A.
OC
CCUse
Family Medical History
9.155
9.131
Surgical History
6.070
6,957
Review of Systems
8.043
7.183
Chief Complaint
4,199
4,162
Allergies
6.000
7.523
Medical History
Finally, we show that SALT achieves better per- formance than DPO on human-edit feedback.
Social History
8.631
9.880
Medications
6,618
3.762
Immunizations
2
Related Work
Laboratory and Imaging Reselts
8.352
8.544
Assesament
29.31
33.85
Most directly related to our work is research on automatic clinical note generation from doctor- patient conversations (Schloss and Konam, 2020;
Diagnostics and Appointments
9,724
10:73
Prescriptions and Therapeutics
10:42
7.928
Table 2:
Average words in CC and CCUser.
Ramprasad et al., 2023; Krishna et al., 2020;
can be extended to Imitation Edits to reduce the need for HF data which can be expensive to get.
Abacha et al., 2023a; Ben Abacha et al., 2023; Yim al., 2023; Wang et al., 2023), and the difference is that those works focus on training a summarization model with pre-labeled data, while we focus on using HF further to improve the summary quality of the trained models.
Dataset
3.1 Clinician Conversations (CC) Dataset
This dataset is collection of 63000 consented doctor-patient de-identification conversations with human transcripts with an average duration of 9 minutes. We segmented the dataset to create train- ing, validation, and test sets of 52,000, 5,000, and 6,000 files each while controlling important char- acteristics of the distribution in each split. The transcripts of the conversations were annotated ac- cording to the traditional SOAP format 2. A SOAP note can contain numerous observations that are grounded to shorter excerpts from the transcript via timestamps that relate back the original au- dio. There are several sections and subsections the SOAP structure, each of which needs specific information and is written a different format. Ta- ble shows the average length span of different subsections is large.
Previous work used HF to train summarization models with reinforcement learning (RL) (Böhm al., 2019; Ziegler al., 2019; Stiennon al., 2020) and used GPT-2 and GPT-3 optimize HF across various summarization tasks. These RL- based methods focus on training a reward function through HF data and use such rewards as train- ing objectives by comparing different summaries (RLHF). Recently, some RLHF variants collect use rewards more flexibly and stably (Akyürek et al., 2023; Dong et al., 2023; Zhao et al., 2023; Yuan al., 2023). We introduce unlikelihood train- ing as an additional learning objective supervised learning. Our technique aims to decrease the proba- bility of unlikely sequences, defined as those which appear in the AA but not in Sy, and increase the probability of verified sequences, which are in SA and reinforced by Se, as well as novel sequences which do not appear in SAT but do appear in Sg. Unlikelihood training (Welleck al., 2019) in- volves adding unlikelihood loss lower the prob- ability of negative candidates. Previous work has explored many scenarios with various negative can- didates for unlikelihood training, including: style transfer (Devaraj et al., 2021), repetition, copy- ing, and contradictions (Li et al., 2019), factuality (Cao and Wang, 2021), text degeneration (Su et al., 2022), and clinical summarization (Adams al., 2022). In this work, we align the Sg with SAT identify negative candidates and train different tokens with unlikelihood and likelihood loss. We also show that our experiments on Human Edits
3.2 CCUser Dataset
In order to generate SOAP notes from doctor- patient conversations, our pipeline follows (Ram- prasad et al., 2023; Krishna et al., 2020). We first record the clinical conversation, then tran- scribe it either using humans or using Google's medical-connversations Automatic Speech Recog- nition (ASR) service. Then, using our proprietary models, we classify utterances into SOAP sections. Finally, using our section conetiioooodiiioneed summariza- tion model trained on the CC dataset, we generate summaries for each of the utterance clusters be- longing to each section.
2SOAP structure detuils can be found in the Appendix A.I.

---

We use our pipeline extract SOAP summaries for our clinician users who record their conversa- tions with their patients via mobile app. The generated summaries were edited by scribes and doctors using our dashboard for their documenta- tion tasks. The dashboard is built for doctors and scribes to check and fix Al-generated summaries in their regular workflow quickly. Hence, we didn't t enforce any trainingginstructiions that might make the data more useful for research, and the users were free to use the dashboard as they saw fit.
(ii) Reinforce the probability of producing words that the user didn't change in Sat and are retained in Sg-
(iii) Increase the probability of producing words that the new user added in Sg.
The loss functions to train the summarization model with SA and Sg:
L8AI = Σ (t)- War-o lp(a, t)
ZESAI
(1)
w-- Wa (x,
Σ [Lx=c(0) Wn-e Lp(a, +
The distribution of the CCUser dataset differs from the CC dataset in the following ways. First, CC uses human-writted transcripts training in- puts, while CCUser uses our pipeline's s inputs from ASR transcripts rather than human-tagged utter- ances. Second, the average length of conversation was 20 min for CCUser compared to 9 min for CC dataset, which could mean more complex conver- sations. The dataset has 215 ASR transcripts with Al-generated notes (along with the Human Edits) from 10 physicians. We randomly select 70 notes from 7 physicians as training dataset, 10 for each physician, and divide the remaining 145 notes into evaluation and test sets. Finally, our dataset is split as trainteval:test 1279:14571 1457:1458 (utterance cluster, edited summary, AI summary) triplet.
(2)
L LE-Ne (1) WB-NO Ly (r, ()]
Lp(x, = log (1 po U))
(3)
Lr(x, 1) log pa (eelaeee,, U)
(4)
Where:
1. U is the utterance cluster used as input
2. and NC mean "changed" and "not changed" tokens when we align Sar and SE sequences.
3. 1 (t) and AC-NNc(t) are the indicator function to signify if the token at in SA is changed or not-changed by the user. Similarly,
1 (() and 1 ( corresponds to Sw. 4. wy are the loss weights, for example, WAA is the weight to penalize tokens that are SAA but not in Sy.
4 Methods
5. L, (xr, t) and L,(x, t) are the likelihood and unlikelihood loss functions
Given a tokenized utterance cluster input U .lennnt the CC summariza- tion model M generates summary SAI [// 1/2 Veensal for it. The user edits this summary from SA to Sg, where Sg = [y, 32, 1/. -VVeSSS.. We aim to update parame- ters in M based on both SAI and Sp. Let, lenU, IEnSAI, and lenSp be the number of tokens in U, SAI, and Sg respectively.
The losses LS and Lsg used in the (SAI, Sg) pair are used to train the summarization model. The indicator functions used in the above equa- tions can be found by tracking the user changes as they edit the summary or by aligning Sy to S using a sequence alignment algorithm. We use sequence alignment (the Boadleman-Wunseee Al- gorithm (Needleman and Wunsch, 1970)) in this work because our dashboard doesn't 't log the users' keystrokes. Assume have a pair from SA and the corresponding Sp. "patient takes one aspirin daily" and "patient doesn't want to take aspirin".
4.1 Sequence Alignment Likeliodd Training (SALT) using SAI and Sg
When a user edits summary from SAT to Sg. they can modify or delete span of tokens, insert a new span of tokens, or not change anything to span of tokens. We want to use these Human Edits to improve our summarization models and produce outputs that are closer to the user's modified sum- mary than before. We do this using both Sa and Sg in the training. We train the model to: (i) Lower the probability of producing words that the user deleted or modified in SAI-
We can align these two sentences as below:
patient putient
takes one aspirin daily wont to take
doesn't
aspirin
C
S D C
Where "C" is "Correspondence" (matching), "T" is "Inserted", "D" is "Deleted"' and "S" is "Sub- stituted".. Note that do on the token level in the implementation. For SA word list ("pattent", "takes", "one", "aspirin", "daily"], the correspond- ing indicator function in Equation are:

---

1 Ar- (1) 10,1,1,0,1]
improve the summary quality, subjecting the model to the catastrophic forgetting problem because of the distribution differences between them. We use the traditional Replay- methods, (Rebuffi et al., 2017). which sample a part of the data from the seen dataset (c.g., CC) and add it to the unseen data (e.g.. CCUser), to address the catastrophic forgetting problem. Here, the likelihood loss is calculated for both sampled seen data (ssen) and human-edie data S S[lnnscen) with the loss function I. M (. + M M.ESggunneenn where we use Maximum Likelihood Estimation for the loss. Following Section 4.1, can use both S (unseen)) and (((anseen) to do SALT training. Following Section 4.2, for the sampled previously seen data, we can also get (SAr(seen) S[(seen) pairs and do SALT training. According to Equa- tions 1, 2, 5, the loss function with RSALT is
1 NN- (t) 01.000
For SE word list ["patient", """ "want", "to", "take", "aspirin"], the corresponding indicator function in Equation 2 are:
1 R (t) = [0, 1, 1, 1
1 No (t) = [1 0,0,0,00000
4.2 Imitation Edits
Sg is a special kind of ground truth summary from the user. SE is obtained by the user using U and SAT SE = Fn(U, SAT). An interest- ing question is whether we can approximate the edited summary Sy (Imitation Edits), and use it to improve the models in the absence of actual Hu- man Edits with SALT. In our work, we use the pre-existing ground-truth summaries as S₁ even though they were not explicitly written as edits to SAI- Leveraging such data has several advantages. First, Sg is not easy to obtain, approximating Sg with S) can increase the amount of data available for unlikelihood training. And we will be able to use SALT even without human-edit data or any new annotations. Second, under the premise of ensuring that the Imitation Edits are of high quality, combin- ing Human Edits and Imitation Edits can further improve the model's performance since both of them bring effective data points for training. Third, Imitation Edits can be used to solve the forgetting problem when we do SALT training with SAZ and Sp, we show this in the next section.
LSALT AAA
Lgg(unseen)
(6)
LRSALT LSAA(eee)) LSs
L LSALT + LRSALT
5 Metrics
ROUGE and UMIS-F1 Models are evaluated with full-length Fi-scores of ROUGE (Lin, 2004). We use QuicKUMLS to extract medical concepts from both model-generated and ground truth sum- maries and then calculate Fl-scores for these two lists of concepts, which is named UMLS- F1 (Adams et al., 2023; Ramprasad et al., 2023).
To imitate Human Edits, we assume the original ground truth summary is generated from Sa and its utterance cluster U (even though the ground truth notes were written independently). Similar to the above setting with SAI and Sg, we use the alignment algorithm to align SA and Sy. Then we calculate Lsy-
GPT4 & Human preference Recent work shows higher correlation between human and GPT4 eval- uation than traditional metrics (Moramarco et al., 2022; Gao al., 2023; Fu et al., 2023), we also use GPT4 preference as measurements to evaluate summary quality. Specifically, we instruct GPT4 give preference ranking on diffeerent Al-generated summaries based on the conversation snippet and reference summary' 4 Similarly, we asked 2 med- ical students to rate summaries from CC based on the same information, for privacy reasons, we did not evaluate CCUser with humans. We discuss the Mean Reciprocal Rank (MRR) (Radev et al., 2002) of different models Section 6.4. Generally, a higher MRR value implies that evaluators have more preference over an approach.
Ls, = 111 on( z
(5)
xo L, L.
where 1I [ (1) and 11, ve (t) signify if the token Dy in S7 is changed nha-cha compared to SAL. and wa are the loss weights.
4.3 Replay-based SALT (RSALT) for Catastrophic Forgetting Problem
We continue training the model M that has con- verged in the original summarization dataset (c.g., CC) on the Human Edits dataset (e.g., CCUser) to
https//gittbb.oom/GGrrgtttto cow/Georgetown-.... Www.-lR--Lbbu/uUUAAA Prompts can be found in Appendix.
hhtps://githu...
Both with hospital internship experience

---

SAGE ROUGE and UMLS-FI measure the de- gree of "likelihood," i.., they evaluate whether or not the model can generate something closer to some references. However, we don't 't just want to know how much "closer to Sg" is newly generated summary, but also how "far away from the bad part of SA^" spans that are changed by the Human Edits. To address this problem, we design evalu- ation method to measure how likely machines are make the same mistakes as before and how likely they are to generate summaries more like the target users (as identified during the editing process). We call this System output Against the Generated and Edited sentence (SAGE). Given the evaluation data (U, SAr, Sg), where SA is generated by the model trained by the original summarization dataset (e.g.. CC) and Si is edited by human based on (U, SAr),
CC
U-f
R1
U-f
M
36.07
48.97
SALTI
57.77
61.02
34.27
46.45
SALTa,
57.70
61 1..06
34,46
46.58
SALT,
57.84
60.81
34.68
46.77
SALT,
57.57
61.09
34,47
46.64
S
58.39
62.13
34.79
47.06
RSALE,
59.57
62.52
35.55
48.25
SALT RSALT
59.60
62.57
35.43
48,20
SALT, RSALT:
59.88
62.60
36.24
48.42
SALT:4 BSALT:4
60.43
63 63.44
36.26
48.6
Human Edits results. Compared to the likeli- hood training S,LT, our proposed SALT has better performance on both new human-edit CCUserevat and the model's prior training CC dataset, when using just CCUsereval for training (Section 6.1.1). Further, we show that the catastrophic forgetting problem can be addressed with Replay- -baee argumentation to our method- RSALT (Section 6.3).
Table 3:
we can get the new summary Snew generated by the new model trained by Human Edits dataset (c.g.. CCUser). Using (Snews SAI, Sg), we can define three groups of words after removing stop words and punctuation in Snow:
ready converged on the CC dataset. All meth- ods below are initialized from M and continue training on SE. St, and SAI-
1. Gm)((A-F)) = wwlw E SAI A w & SE}
2. SALT;: the baseline, which is only based on likelihood training on Sg or S₁
2. Gu2(E-A[) [w|w & SAI A Sg) 3. Gu3(AInE) = [w|w E Sg A w E S
3. SALT (or SA likelihood training on Se or Sy, but with decreased (or increased) weights for 1 E-C or I-C tokens
By training on HF, we aim to have Snew closer
to Se while avoiding the mistakes found in SAT- So SAGE counts how many words in Snew are in GwI(Ar E). Gwz(E AI), and Gun(A We call this word level SAGE Ew) Similarly,
4. SALT,; only unlikelihood training on SAT
5. SALT both likelihood (on Sg or Sy) and unlikelihood (on S
we can define Ga(Ar-E)) E), Gra(E-AN) AS). Ges(ArDE) and make Concept-level SAGE (SAGF)) based on UMLS concept overlap in Snew. SAI. and Sg. We have two assumptions regarding SAGE:
6. SALT,; all the above SALT variations
7. SALT is the traditional replay- based method. When continuing to train M with different SALT variations on new data, this method will sample part of the data from the dataset that M has already seen and use them for training with likelihood loss.
1. users can accept machines making some mis- takes, but they can't t tolcrate machines making the same mistake, again and again.
2. users will be more satisfied if the model, over time, learns to generate outputs more similar to the user's edited summaries
8. &RLT,+ following Section 4.3, RSALT treats sampled data from the replay- based method as imitation-edit data and uses both likelihood and unlikelihood training.
According to Assumption 1 and 2, a model trained on HF should be able to generate less con- tent belonging to G1 (Gw1 and Gat), and more con- tent belonging to G2 (Gwz and Go2). The model should also be able generate G3 (Gua and G.3) since G3 represents human-verified information.
6.1
SALT in human-edit dataset
6.1.1
Analyzing the behavior of SALT
In Table 3, the evaluation on the CCUser shows compared to the regular likelihood training
6 Experiments
"There are no scores for M on human-edit evaluation data (CCUser) here. Because human-edit data is directly modified from M's SAr, so it is unfair to calculate the scores of its Sar and human-cdit data and compare with other methods.
We use the following symbols:
1. M refers to models that are trained and al-

---

SALT. SALTI
SAGE
FAGE
6.1.2
Reducing the forgetting problem
Gui
Gu2
Gua
Get
Ga2
Ges
In Table we see dip in evaluation metrics for SALT; in the old evaluation dataset CCevol when we train the model trained on the CCUser catastrophic forgetting. The reason could be the distribution difference between CCUser and CC dataset described in Section 3.2. Both SALT, and S have different degrees of improvement in ROUGE-1 and UMLS-F1 on CCeval data. This result shows that SALT training also alleviates the forgetting problem certain extent.
SALE
1
S
0,982
0,599
1,05
1,022
1011
1.011
S
0.992
1.043
1,02
1,076
1,800
1.099
SALT
0,835
0.824
0.917
0.891
0,986
0:811
0,926
1,029
0.990
1,088
1,06
Table 4: Word-level and concept-level SAGE for CCUser,wol normalize by SALT, as the baseline.
(SALT)), changing loss weights for 1l E-C tokens in likelihood training (SALT or SALT,,) can bring changes to their performance. Predictably in Table 4, that SALT, produces higher Gw2 than i and the trends in other columns are not as pronounced since SAT isn't considered. Simi- larly, SALT, produces lower Gw1 than the others. However, + achieves significantly higher performance on both CC and CCUser. We further show how we can manipulate a model's behaviors using different SALT through SAGE in Table 4. First, SALT, only uses Sg, and all tokens in Se contribute to the loss equally. SALT can increase or decrease the emphasis of the model on 1g-c through different weights on the loss function. In- creasing the loss weight of LE-C will make the model generate more words/concepts belonging to LE-C (Gw2 and G22), which follows our SAGE Assumption 2. While reducing the loss weight of Ig- C will make the model generate fewer words and concepts belonging 1g-C (Gwz2 and G the same time it can also reduce the genera- tion of words/concepts belonging to AT- (G.r1 and G.1), which satisfies our SAGE Assumption 1.
One widely used and effective technique re- duce catastrophic forgetting is the replay-base method, which mixes in the seen data the model was trained on (e.g., CC). In this work, we set the ratio of CCUser and CC data to 2:1. That is, assum- ing that there are CCUser data, we will sample 0.5 CC data to train together' ? Table 3 shows that -ALT,+RRAIE is effective in helping the model reduce the catastrophic forgetting problem. Adding the sampled seen data improves the model's performance in both the new - CCUser and the original CC data. However, we still see a reduc- tion in the performance of CC dataset compared with M, which shows that the traditional replay-based method cannot com- pletely solve this problem. In Section 6.3, show how address the problem further with SALT, imitation-edii data, and RSALT.
see
in the
6.2 SALT in imitation-edit dataset
SALT uses the relationship between Sg and SA to get better performance than using just Sy and likelihood training. In this section, we show that we can directly improve the summarization model M using a similar relationship between Sy (the ground truth data) and SAI without new human-ede data or additional annotation, i.e., by assuming that the Sy is the output of human-edit data on SAI. Simu- lating Human Edits this way lets us 1) demonstrate the effectiveness of SALT on a public dataset that does not have the human-edit component in them," 8 and 2) reduce the amount of Human Edits needed as it is hard to get.
So SALT, and SALT, make the model better for users according to the SAGE metric.
Second, unlike the three above SALT variations, SALT, only uses SA but it knows which tokens in SAT belong AI-C and AI-NC respectively. So SALT, v significantly reduces the words and con- cepts belonging 1 A However, because the data of 1E-NC has not been seen, SALT,, a rarely generates related words and concepts.
Finally, SALTH+' has more granular information- that tokens belonging to AI- 1I As NC. NNNC. EE C. and HE NC in Sa (Sx) through their corresponding loss weights. Therefore, SALTH=u can learn the more suitable distribution, which decreases the generation of words and concepts belonging to 1 A c while increasing the generation of words and concepts belonging to
Although both come from humans, SE and S are fundamentally different in their relationship with SAI. The former modified from SA while humans generate the latter from scratch. There-
? "Adjusting this ratio will bring some improvements in certain SALT., but we found that 2:1 has relatively good performance in most SALT,, so we use this ratio uniformly. "Due to the space limit, we put results in the Appendix.
1l AAA- AC-NC- Eg- and 1l NC-

---

M
11
4
1.
1 u
the seen dataset even if the model is already con- verged (on the seen/original training data). Third, different from previous human-edit results, SALT, of CC is better than SALT(+w- We think this is because M has started to overfit on CC data, so continuing to add likelihood to the original training data reduces the scores.
RI
36.07
35.77
35.76
35.65
37.39
36.16
U-f
48.97
48.86
48.60
48.97
49.45
49.24
Table 5: SALT results for imitation-edit experiments. The imitation-edi data come from the training dataset which the model M has already seen by assuming the ground truth generated by editing the model's output.
CCicat-r
CC
6.2.2 Imitation Edits using unseen data
R1
U-f
RI
U-f
We use a part of the test dataset (not used in the evaluation) from CC to experiment with the effects of SALT and Imitation Edits on unseen data. In Table 6, we take M (trained on CC-train) and train it with a part of CC-test as the imitation-edit data with SALT. We take the remaining test data of the CC-test evaluate the model performance in new imitation-edit data and then use CC-eval evaluate the model performance in the original data. In imitation-edi evaluation results (CCtest -) of Table 6. SALT,+u has better performance than the baseline method SALT,, which is consistent with our results using human-edit data in Table 3. In the original data evaluation results (CCeval) of Table 6, although there from distribution shift, SALT still has a higher score than the baseline model SALT,-
M
36.01
58.15
36,07
48.97
SALT:
57.55
48.50
36.57
58.12
36.28
48.84
36.73
57.48
36.61
48.61
SALT R.
36.74
58.48
36.65
48.77
Table 6: Imitation Edits experiments. Here the imitation- edit data comes from subset of the corresponding test dataset (we don't use them in the table for metrics),
which M has never seen before. We use CC-test for SALT and CC-train for RSALT during training.
fore, Sg is directly dependent on Saz, but S₁ is not. Consequently, even though SE and Sy are de- pendent on the same data as input, the differences between SA and S are likely to be larger than between Sa and Sy. We can see this difference in the average percentage of changed tokens LE-C and 1 C is 1, the former (6. 17%) is much lower than the latter (45.59%). Hence, after do se- quence alignment between S₁ and SAI, perform a two-step post-processing operation to ensure the training stability, which helps us to reduce the per- centage of changed tokens from 45.59% to 19.07% with an acceptable amount of data lost (21.38%).
forgetting problem arising
6.3 Solving forgetting problem with RSALT
Through previous analysis, we see that SALT helps M to continue training on human-edit data or imitation-edit data. In Section 6.1.2 and 6.2.2, we observed that the traditional replay-based method cannot completely solve the catastrophic forgetting problem, so the performance of SALT, +RSALT, on Table and is still lower than M's perfor- mance if there are distribution differences.
6.2.1 Imitation Edits using seen data
We use the training data from CC to experiment with the effects of SALT and Imitation Edits on seen data. First, for the CC dataset, the results in Table 5 show that continuing to use likelihood loss on the training dataset to train the already con- vergent M does not improve the performance and leads to overfitting. However, when we use S, as imitation-edit data and do SALT training on it with SAT. we can see an improvement. Second, we see similar results for the CNN dataset. Lven though there is performance degradation arising from overfitting for SALT,, doing SALT training with Sj and Sa can improve the performance more than using just the likelihood training. These results show that we can get additional improvement on the model by continuing to train it with SALT on
We report the results of SALT, B in Table 3 and 6. We find that SALT,+RSAIE, does not have the forgetting problem when continu- ing to train with human-edit data. We attribute this result to the data augmentation that RSALT brings to the traditional replay- -bsed method. RSALT not just reuses the seen data to prevent the model from forgetting the learned distribution but also uses the output generated by the model itself with SALT to expand the effective training data points further.
6.4 Preference Evaluation
In CC dataset, GPT4 (on 500 data points) ranks SALT, x +RSALT, higher than other variations (SALT, and $ and M. To verify the GPT ranking, we performed human evaluation on a
"The details are in Appendix A.

---

Reward
RI
Meteor
00.52
22.63
SALT;
0.368
0.381
0.203
0.371
0.292
07.48
39.05
SALTE
0.591
0.394
0.215
0.383
0.320
43
DPO
0.484
0.379
0.210
0.369
0.301
DPO
0,532
0.372
0.191
0.361
0.291
CCUser (GPT4)
CC (GPT4)
cc (Human)
Table 7: SALT and DPO results CCUser with GPT-2
Figure 1: CCUser&CC GPT4 preference. We instructed GPT4 give preference ranking for 4 Al-generated summaries (on 500 data points): M (not trained on
for B {0.1, 0.5} on GPT-2 (117M parame- ters), with Rouge, Meteor, and Reward Accuracy (Reward Acc) the CCUser test dataset. Reward Accuracy used in DPO is the ratio of data points for which chosen reward > rejected reward.
CCUser), i SALT +RSATtT (1)
( is most preferred by GPT4 on CCUser, (2)
while SALT + L4 is most preferred by GPT4
on CC. (3) CC on human preference (on 25 data points) for M. SALT, SALT and +ALT
chosen reward = (mo(S))U) (Sa((S) (12)
rejected reward 8 (S 87 (S.r(SA((()) (13)
smaller set (25 data points). Human ranking agrees with the GPT4 ranking. In CCUser, GPT4 (on 500 data points) ranks SALT,; a higher than other vari- ations, which expected as +RST++ is also trained on the replay dataset. Because of pri- vacy reasons, we did not do the human evaluation on CCUser. In Appendix Table 12, we show the prompt used with GPT4 for ranking the summaries. We show all the MRR scores for different models in our work in Figure
We find that DPO is better than SALT, which is just equivalent to likelihood training on Sg. This is expected since DPO also uses SĄI. However, DPO gets lower performance than S SALTi+a. When we change hyper-parameter B to get higher Reward Accuracy, others (ROUGE, and Meteor) degrade, and vice versa. We think this is because, DPO penalizes the entire rejected summary, which is not suitable for human edit feedback, because most words in SAI and Sş are the same. DPO does not explicitly consider such cases, and hence, it might be difficult for DPO to learn an implicit reward through SAI and Sg without using the fine fine-grained relationship between their tokens. It is interesting to see that Reward Accuracy is higher for SALT than DPO, even though the SALT loss function does not explicitly maximize chosen and rejected log probability like DPO.
7 Diseussion: SALT vs RLHF
First, we argue that Human Edits is more natu- ral way to collect feedback from users as they fix AI- Al-generated text for their workflow to improve generation. Collecting other forms of feedback that are not directly tied to the user's workflow will not scale as much, this is especially true in domains requiring expert domain knowledge and with nu- anced user goals. Considering the cost, time, and availability of the experts, it is important to collect HF from the expert's 's daily workflow.
It should be noted that DPO was developed for using comparisons and not human edit feedback. For human edits feedback, a straightforward way improve DPO could be to modify the loss function to use only the "negative tokens" in the rejected summary, which aligns with our SALT ideas.
Second, we experiment with Direct Preference Optimization (DPO) (Rafailov et al., 2023) to com- pare the difference between RLHF and SALT while using human edit feedback dataset. The train- ing setup of DPO and SALT are similar, they are trained directly on the human preference dataset without training explicit reward models. We use SaI as the rejected summary and Sg as the cho- sen summary and calculate the DPO loss - LDPO. between them to train the model.
Conclusion
In this work, we explore improving language models with Human Edits feedback, which can be collected scalably than others. Specifically, we propose the SALT training objective based on se- quence alignment and unlikelihood training and show how design Imitation Edits to reduce the need for expensive HF, We further show on human edits data, SALT performs better than a straightfor- ward RLHF (DPO) approach.
(Ration = log To [5x U)- log no UU)
(9)
[Ratiores = log nref (Sg U\) - log Tre (Sx(UU)
(10)
LDPo llg (3 (TRatios - (Ratiorer))
(1I)
We used GPT because, the time of this paper, DPO is only implemented on decoder-only models in Hugging Face https://hugiinggaae wtw.ocssr//mmmiinnn/oooo /rriinn trainer
where and f are the current and original model
parameters. Table 7 shows the performance of DPO

---

9 Limitations
quence alignment the medical entities (Luo al., 2022) or jargon (Kwon et al., 2022) help the model focus more on important tokens during the training process to reduce hallucination further.
In our experiments, we find that our method im- proves relatively smaller language models like T5. Due to the limitation of computational resources, we are not able to try our methods on larger lan- guage models. So we don't t understand which HF (human feedback or human edit data) is better on LLMs. But like what we discussed in Section 1, Human-Edits have many unique advantages from an ML data point of view. Given that it's natural way to collect feedback from users as they fix our Al-generated summaries for their workflow, many products in the industry can more easily use this HF approach and our SALT method to improve their text generation quality without too much ex- tra effort. In addition, other HF methods should be explored more in various domains and models of various sizes so as to help the NLP community find the most suitable HFF method in various scenarios.
10 Ethics Statement
The methods related to unlikelihood training are very dependent on the quality of negative candi- dates. In this paper, we propose a very general framework to provide negative candidates, that is, to caleulate the sequence alignment between Sar and Human-Edits Imitation- There will be some potential problems in actual deploy- ment: First of all, for Human-Edits, we don't know whether the user is modifying because of some kind of error in SAr or because of the user's per- sonal preference. These two behaviors need to be distinguished in future research or actual deploy- ment because the former data is more suitable for improving the problems of the model itself (such some factual errors), and the latter data more suit- able for user-personi aeer-personalized training data. Secondly, whether for Iluman-Edits or Imitation- Edits, when a large number of complex Edits appear, the se- quence alignment algorithm we currently use may not be able to get the correct negative candidates, resulting in rewards or penalties for wrong tokens. In the experiments in this paper, use some filters control the quality of the training data provided for unlikelihood training, but the reality will be very complicated. In addition to using similar fil- ters in this paper, another solution is to dircetly track the users' changes as they edit the summary on the product, and the subsequent training steps will not change. But this will add lot of extra overhead to the product engineering.
Another point that has not been explored in this paper is LLM-in-the-loop. With the emergence of GPT3.5 and ChatGPT, LLM has shown level close to beyond human beings in many domains. In this paper, did not L.Ms to conduct ex- periments similar to Human Edits (that is, treat the LLM as human to modify Sa to get SE(LLM)) Ideally, this would provide better Imitation- Edits to reduce HF costs. In addition to time and resource constraints, we discussed in Section 1, data pri- vacy issues make it hard for many practitioners in the industry to input their data into these third-party APIs service websites for related experiments. LLM--in-te-llopp is undoubtedly a worthwhile next step in the future, and we will study how deal with related data privacy issues. This will also be a problem to be solved for many other tasks in medical and other privacy-oriented domains.
Acknowledgements
The current implementation of our methods also has some room for improvement. Our code cur- rently only tries one global sequence alignment algorithm, the Keedleman-Wunsse Algorithm. In fact, there are many alternatives that can help the model improve in different aspects. For example, how to improve factuality during LM's summaries is one key topic for both NLP and BioNLP com- munity (Tang et al., 2022; Abacha et al., 2023b; Chang et al., 2023). Some previous work explor- ing language models and knowledge has shown that insufficient knowledge may lead to factual er- rors (Petroni et al., 2019; Sung et al., 2021; Yao et al., 2022a,b). So we can limit the scope of se-
We thank the Abridge AI for CC and CCUser data, well as the professionals who performed the human evaluations. In addition, we also thank UMass BioNLP Lab for producing and providing us with a large batch of publicly available GPT Edits data for related work 12.
References
Asma Ben Abacha, Wen-wai Yim, Yadan Fan, and Thomas Lin. 2023a. An empirical study of clini- cal note generation from doctor patient encounters.
12This part of the data and code will appear on our GitHub https::/giitub..oo http://www..ooeesssooooooooeeeee waw.oooonnaaaoaaannaaia

---

In Proceedings of the 17th Conference of the Euro- pean Chapter of the Association for Computational Linguistics, pages 2283-2294.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko. Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay. Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Mor- eira, Rewon Child, Oleksandr Polozov. Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. PaLM: Scaling language modeling with pathways. arXiv preprint w001:2204.003111.
Asma Ben Abacha, Wan Yim, George Michalopou- los, and Thomas Lin. 2023b. An investigation of evaluation metrics for automated medical note gener- ation. arXiv preprint 2r0vv 17364.
Griffin Adams, Han-Chin Shing, Qing Sun. Christo- pher Winestock, Kathleen McKeown, and Noémie Elhadad. 2022. Learning to revise references for faithful summarization. arXiv preprint arXiv:2204. 102900
Griffin Adams, Jason Zucker, and Noémie Elhadad. 2023. meta-evaluation of faithfulness metries for long-form hospital-course summarization. arXiv preprint arXiv:2303.033 039448.
Afra Feyza Akyürek, Ekin Akyürek, Aman Madaan, Ashwin Kalyan, Peter Clark, Derry Wijaya, and Niket Tandon, 2023. RI4f: Generating natural language feedback with reinforcement learning for repairing model outputs. arXiv preprint arXiv:2305.0880 08844.
Ashwin Devaraj. Jain Marshall, Byron C Wallace, and Junyi Jessy Li. 2021. Paragraph-level simplification of medical texts. arXiv preprint arXiv:2104.05767
Asma Ben Abacha, Wen-wai Yim. Griffin Adams, Neal Snider, and Meliha Yetisgen. 2023. Overview of the mediqa-chat 2023 shared tasks on the summarization and generation of doctor- doctor-patient conversations. In ACI-ClimicalNI.P 2023.
Hanze Dong. Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. 2023. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767
Florian Böhm, Yang Gao, Christian M Meyer, Ori Shapira, Ido Dagan, and Iryna Gurevych. 2019, Better rewards yield better summaries: Learning to summarise without references. arXiv preprint arXiv:190.001114
Jinlan Fu, See-Kiong Ng. Zhenghao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. arXiv preprint WwXii:2302.04160
Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Ship- ing Yang, and Xiaojun Wan. 2023. Human-like sum- marization evaluation with chatgpt. arXiv preprint 2xXi:2330400255:
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sustry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Ag- nieszka Grabska- Barrinska, et al. 2017. Over- coming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, l1 (14(5))352133526
Pengshan Cai, Zonghai Yao, Fei Liu, Dakuo Wang, Meghan Reilly. Huixue Zhou, Lingxi Li, Yi Cao. Alok Kapoor, Adarsha Bajracharya, et al. 2023. Paniniqa: Enhancing patient education through interactive question answering. arXiv preprint arX1v.338..32533.
Kundan Krishna, Sopan Khosla, Jeffrey P Bigham, and Zachary Lipton. 2020. Generating soap notes from doctor-patient conversations using mod- ular summarization techniques. arXiv preprint 2rX
Shuyang Cao and Lu Wang. 2021. Cliff: Con- trastive learning for improving faithfulness and fac- tuality in abstractive summarization. arXiv preprint
arXiv:2109.09209
Sunjae Kwon, Zonghai Yao, Harmon S Jordan, David A Levy, Brian Corner, and Hong Yu. 2022. Medjex: A medical jargon extraction model with wiki's hyper- link span and contextualized masked language model score. arXiv preprint 1000022..
Haw-Shiuan Chang, Zonghai Yao, Alolika Gon, Hong Yu, and Andrew McCallum. 2023. Revisiting the architectures like pointer networks to effi- ciently improve the next word distribution, sum- marization factuality, and beyond. arXiv preprint arXiv:2305.1 12289.
Margaret Li, Stephen Roller, Ilia Kulikov, Sean Welleck, Y-Lan Boureau, Kyunghyun Cho, and Jason Weston.

---

2019. Don't t say that! making inconsistent dialogue unlikely with unlikelihood training. arXiv preprint arXiv:1911 1911.0360.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 21(140):1-67.
Chin- Yew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text summariza- tion branckes out, pages 74-81.
Sanjana Ramprasad, Elisa Ferracane, and Sai P Selvaraj. 2023. Generating more faithful and consistent soap notes using sptribteesppccfie parameters.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung. Yi Tay, Denny Zhou, Quoe V. Lc. Barret Zoph. Jason Wei, and Adam Roberts. 2023. The flan collection: Designing data and methods for effective instruction tuning.
Sylvestre-Alviss Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. 2017. icarl: In- cremental classifier and representation learning. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001-2010.
Ling Luo, Po-Ting Lai,
Wei, Cecilia N
Arighi, and Zhiyong Lu. 2022. Biored: a rich
biomedical relation extraction dataset. Brieffiggs in Bioinformatics, bbbs222
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M. Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablami, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang. Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Tr- ishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Tee- han, Stella Biderman, Leo Gao, Tali Bers, Thomas Wolf, and Alexander M. Rush. 2021. Multitask prompted training enables zero-shov shot task generaliza- tion. CoRR, abs/21 10.08207.
Francesco Moramarco, Alex Papadopoulos Korfiatis, Mark Perera, Damir Jurie, Jack Flann, Ehud Re- iter, Anya Belz, and Aleksandar Savkov. 2022. Hu- man evaluation and correlation with automatic met- ries in consultation note generation. arXiv preprint arXiv:2204.00447.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't ' give me the details, just the summary! topic-aware convolutional neural networks for ex- treme summarization. ArXiv, abs/1808. 100.0888..8755.
Saul Needleman and Christian D Wunsch. 1970. general method applicable to the search for simi- larities in the amino acid sequence of two proteins. Journal of molecular biology, 4843):443-453.
Benjamin Schloss and Sandeep Konam. 2020. Towards an automated soap note: classifying utterances from medical conversations. In Machine Learning for Healthcare Conference, pages 310-631. PMLR.
OpenAI. 2023, Gpt-4 technical report. arXiv preprint arXiv:2303.08774
Abigail See, Peter J. Liu, and Christopher Manning. 2017. Get to the point: Summarization with pointer- generator networks. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 1073- 1083, Vancouver, Canada. Association for Computa- tional Linguisties.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.022 02555
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler. Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei. and Paul Christiano. 2020. Learn ing to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008- 3021.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, An- ton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2019. Language models knowl- edge bases? arXiv preprint (ttp::::9990000000
V Podder, V Lew, and S Ghassemzadeh. 2021. Soap notes. (updated 2021 sep 2]. StatPearls [Interntt) StatPearls Publishing. Available from; https://www. ncbi. nim. nih. Nov/books/NBKK42266.
Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Ling- peng Kong, and Nigel Collier. 2022. A contrastive framework for neural text generation. arXiv preprint X2Xi:22002.06417.
Dragomir R Radev, Hong Qi, Harris Wu, and Weiguo Fan. 2002. Evaluating web-based question answering systems. In LREC. Citeseer.
Mujeen Sung, Jinhyuk Lee, Sean Yi, Minji Jeon, Sung- dong Kim, and Jaewoo Kang. 2021. Can language models be biomedical knowledge bases? arXiv preprint arXiv:2109.07154.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305. 18290.
Liyan Tang, Tanya Goyal, Alexander R Fabbri, Philippe Laban, Jiacheng Xu. Semih Yahvuz, Wojeiech Kryšciáski, Justin F Rousseau, and Greg Durrett.

---

2022. Understanding factual errors in summariza- tion: Errors, summarizers, datasets, error detectors. arXiv preprint arXiv:2205. 12854.
A
Appendix
A.1
SOAP Structure
The SOAP (Subjective, Objective, Assessment, and Plan) structure is commonly used by providers (Podder et al., 2021).
Junda Wang, Zonghai Yao, Avijit Mitra, Samuel Osebe. Zhichao Yang. and Hong Yu. 2023.
UMASS BioNLP at MEDIQA-chat 2023: Can
LLMs generate high-quality synthetic note-oriented doctor-patient patient conversations? Proceedings of the 5th Clinical Natural Language Processing Workshop,
* The Chief Complaint section is a brief descrip- tion of patient's ' conditions and the reasons for the visit.
doctor-
pages 460-471, Toronto, Canada. Association for Computational Linguistics.
* The Subjective section is detailed report of the patient's current conditions, such as source, onset, and duration of symptoms, mainly based on the patient's self-report. This section usually includes history of present illness and symptoms, current medications, and allergies.
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Di- nan, Kyunghyun Cho, and Jason Weston, 2019. Neu- ral text generation with unlikelihood training. arXiv preprint (iXv:II98.00431
Zonghai Yao, Cao, Zhichao Yang. Vijeta Deshpande, and Hong Yu. 2022a. Extracting biomedical fac- tual knowledge using pretrained language model and electronic health record context. arXiv preprint arXiv:2209.07859
The Objective section documents the results of physical exam findings, laboratory data, vital signs, and descriptions of imaging results.
Zonghai Yao, Yi Cao, Zhichao Yang, and Hong Yu. 2022b. Context variance evaluation of pretrained lan- guage models for prompt-based biomedical knowl- edge probing, arXiv preprint arXiv:221 102655.
The Assessment section typically contains medical diagnoses and reasons that lead medical diagnoses. The assessment is typi- cally based on the content of the chief com- plaint and the subjective and objective sec- tions.
Wen-wai Yim, Yujuan Fu, Asma Ben Abacha, Neal Snider, Thomas Lin, and Meliha Yetisgen. 2023. Aci- bench: a novel ambient clinical intelligence dataset for benchmarking automatic visit note generation. Submitted Nature Scientific Data.
The Plan section addresses treatment plans based on the assessment.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang. Songfang Huang, and Fei Huang. 2023. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05300
A.2 Implementation Details
Due data privacy issues, we cannot disclose our CC and CCUser datasets. But for the repro- duction of our methods, in the Appendix, we also use two general domain summarization datasets, CNN/Daily Mail (CNN) (See et al., 2017) and Extreme Summarization (XSum) (Narayan et al.. 201 8) to test the imitation-edit experiments.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. 2023. Slic-hf: Se- quence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 10425
Daniel M Ziegler. Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. 2019. Fine-tuning lan- guage models from human preferences. arXiv preprint arXiv: T90iv:1909.00000 080..0893.
The summarization model used in this paper is based the publicly available T5-small !o and T5-large Note that the experimental results of our 15-large-based model are not real human edit feedback for the summaries it generates, because of some deployment and privacy issues, we can only collect the CCUser data (Human Edits) for t5- wawlll-ased-moonl www.-mmooolll-etteeteeeee generated summaries via our mobile app. Therefore, we put t5-large-related re- sults only in the appendix. All the results in Section 6.1 are for our (5-small-based model. But overall, the patterns and findings are consistent on both t5-small and large
httpss//huuggiggffccc.ot//o/o/o/ soll "hunp://hugginnffccc.otttsirg

---

Section
Subsection
Definition
Subjective
Chief Complaint
Patient's primary motivation for the visit and type visit Patient's report of system-related health and symptoms Patient's reported diagnoses/condition (when and what, excluding laboratory and imaging results and surgeries)
Review of Systems
Past Medical History
Past Surgical History
Patient's ' reported prior surgeries (what, when, where) Conditions affecting patient's 's close genetic relatives Patient's 's alcohol, tobacco, and drug-related behaviors Patient's list of medications (not prescribed during visit) Patient's list of allergies (primarily medicinal)
Family Medical History
Social History
Medications
Allergies
Miscellaneous
Patient's clinically relevant social and other cireumstances
Objective
Immunizations
Vaccination record (not frequently discussed)
Laboratory and Imaging Results
Clinician's 's discussion of laboratory/imaging results
Assessment
Assessment
Syuthesis of the reason for the visit and pertinent diagnosis
Plan
Diagnostics & Appointments
Plan for future tests, appointments, or surgeries Plan for medications and therapeutics
Prescriptions & Therapeutics
Table 8: Details of the SOAP structure used in our CC and CCUser datascts.
15 and for Ar in Equations 1, 5. 5. We trained Mcc the annotated Clinician Conversations (CC) dataset for 10000 steps and McNN on CNN data with 100000 steps (batch size of 8). We then initialized the CCUser models SALT, with M and trained them on 70 human-edit notes for 1000 steps (batch size of 1.
In this paper, we used '-1' for w ar--c:
tation Edits with 1000 steps. We took the remain- ing test data of CC-test, CNN-test, and XSum- test (Narayan et al., 2018) respectively Imitation Edits evaluation data, and then used CC-eval and CNN-eval to evaluate the performance of the model in the original data.
In all our evaluations, used beam size of 4, wwweeettngrammeeee mizam- and minimum length and maximum length of sentences were set as (10, 100). We used five different random seeds to sam- ple training data for all our experiments, and the scores reported in the tables are the average of these random seeds.
In Section 6.2.1 and A.3.2, ran M on both CC and CNN datasets' training data (See al., 2017) to get the generated summaries SAL and we use the ground truth data as Imitation Edits on the seen data S₁- We then initialized SALT, from Moe and MONN separately and trained on corresponding Imitation Edits with 1000 steps. We used CC-eval and CNN-eval to evaluate the models' performance.
A.3 Imitation Edits Experiments
A.3.1 Imitation Edits smoothing function
Although both come from humans, Sø and S are fundamentally different in their relationship with SAI- The former is modified from SAI while humans generate the latter from scratch. There- fore, SE is directly dependent on SAI, but S₁ is not. Consequently, even though Sg and S are de- pendent the same data input, the differences between SA and S are likely be larger than between SAI and Sg. We can see this difference in the average percentage of changed tokens 1 c and 1 is 1, the former (6. (7%17% is much lower than the latter (45.59%). Hence, after we do se-
In Section 6.2.2 and A.3.3, we sampled 3000 CC test data summaries (11812 data in total), 3000
CNN test data (114 data in total), and 3000 XSum test data (1 1334 data in total) as Imitation Edits the unseen data since we don't have the unseen training data in these datasets. Similarly, we initialized SALT, from M and trained on Imi-
For W. War-o- we used -1.2 for SALTI, and -0.5 for SALT,,, and all other SALT, and RSALT, use -1. "Wo did all the experiments with 1 NVIDIA Tesla P100 GPU - 16 GB memory, with Adam optimizer be- (a..90.,,0,,99), epsilon=lc-08, learning rato=5c-05..

---

CCUser
CCoval
RI
U-f
R1
U-f
SALTI
07.77(#0.28)
61.02(#1.06)
34.27(#0.21)
46.45(#0.52)
SALT
57.70(#0.25)
61.06(+0.86)
34.46(=0.31)
46.58(+0.33)
SALT,
37.84(#0.36)
60.81(#0.79)
34.68(#0.25)
46.77(#0.71)
SALT,
57.57(#0.66)
61.09(+1.33)
34.47(=0.44)
46.64(+0.51)
SALT
58.39(#0.57)
62.13(#1.03)
34.79(#0.30)
47.06(#0.47)
;ALT,+RSALT,
59.57(#0.47)
62.52(40.98)
35.55(#0.32)
48.25(#0.60))
SALT - *
59.60(#0.52)
62.57(#1.34)
35.43(#0.23)
48.20(#0068)
SALT, I+a
59.88(+0.43)
62.60(+0.85) 60.43(#0.61)
48.42((0.41) 36.24(#0.36)
BALTI&++RSALL
63.44(+0.92)
36.26(+0.40)
48.69(+0.59)
Table 9: 95% Coufidence interval results for Table 3
CCUseT
CCamt
ing to use likelihood loss on the training dataset to train the already convergent M does not improve the performance and leads to overfitting. However, when we use Sy as imitation-edit data and do SALT training on it with SAI, we can see improvement. Second, we see similar results for the CNN dataset. Even though there is no performance degradation arising from overfitting for SALTi, doing SALT training with Sy and Saı can improve the perfor- mance more than using just the likelihood training, These results show that we can get additional im- provement on the model by continuing to train it with SALT on the seen dataset even if the model is already converged (on the seen/original training data). Third, different from previous human-edit results, SALT, of CC better than S SALT(4 We think this because M has started to overfit on CC data, so continuing to add likelihood to the original training data reduces the scores.
R1
U-f
R1
U-f
M
40.48
42.22
37.21
47.12
SALT;
64.28
64.90
36.67
48.51
63.47
64.95
37.10
48.66
62.49
62.86
37.58
49.87
SALT: -RSALT
64.71
64.53
37.87
50.02
Table 10: Ta-large results on CCUser dataset. Ta-large is also first fine-tuned on the CC dataset and then fine- tuned on the CCUser dataset. Note that CCUser data is collected only for T5-small, so it's real Human Edits dataset for Ta-large.
quence alignment between S₁ and SAI. perform a two-step post-processing operation to ensure the training stability, First, we only penalize consecu- tive tokens (> 1) in Sar that are not aligned with S), for eg., the 1) No (t) = [1, 0, 1, 1,1,0,00 0, 1], becomes [1, 1, 1, 0, 1]. and the corresponding change is made to I1 Ar-e (1). This smoothing is to reduce the impact of less important negative tokens, for e.g., punctuation and word plural, they more frequently present in such single negative to- kens. On the contrary, consecutive negative tokens are more likely to represent important errors (e.g., hallucination and missing information). Second, we discard data with more than 60% of the tokens being 0 in the indicator function 1 A which helps us to reduce the percentage of changed to- kens from 45.59% to 19.07% with acceptable amount of data lost (21.38%).
A.3.3 Imitation Edits using unseen data
We use a part of the test dataset (not used in evaluation) from CC, CNN and XSum to experi- ment with the effects of SALT and Imitation Edits unseen data. In Table 6, show three exper- imental results. In the first experiment, we take M (trained on CC-train) and train with part of the CC-test as the imitation-edit data with SALT. In the second experiment, we take M (trained on CNN-train) and train with a part of the CNN-test as imitation-edi data. In the third experiment, we take M (trained on CNN-train) and train it with part of the XSum-test imitation-cdit data. In the three experiments, we took the remaining test data of the CC-test, CNN-test, and XSum-test, spectively, to evaluate the model performance in new imitation-ede data and then used CC-eval and CNN-eval to evaluate the model performance
A.3.2 Imitation Edits using seen data
We use the training data from two datasets, CC and CNN, to experiment with the effects of SALT and Imitation Edits on seen data. First, for the CC dataset, the results in Table show that continu-

---

(CC-test) <CC-train>
(CNN-test) CNNN-triin>>>
(XSum-test) <CNN -train
Cr
CConal
CNNteat-r
CNN eval
XSum
CNN.val
R1
U-f
R1
U-f
RI
R1
R1
R2
M
36.01
58.15
36.07
48.97
36.44
15.28
36.99
15:49
17:40
2.36
36.24
15.35
SALTI
36.09
57.55
36.14
48.50
36.97
15.39
37.29
15.29
26.56
7.22
26.02
8.79
SALT:44
36.57
58.12
36.28
48.84
37.59
16.04
38.17
16.35
27.03
7:27
28.64
9.27
36.61
37.57
38.39
16.65
25.56
15.57
SALT +RSALT
36.74
36.65
37.73
38.42
26.10
Table 11: Imitation Edits experiments: Here the imitation-edit data comes from a subset of the corresponding test dataset (we don't use them in the table for metries) which M has never seen before. () and <>> show the training data used by SALT and RSALT respectively.
the original data.
In this task, we for your expertise annotating the quality system-generated SOAP notes by machine learning models. Mainly we provide conversation snippet and a human- written reference SOAP note for the respective snippet, along with for your preference.
In imitation-edit evaluation results teetest-rs CN Niestrr, X Sumtest-r) of Table 6, SALT has better performance than the baseline method SALT; in all three experiments, which is con- sistent with our results using human-edit data in Table 3. In the original data evaluation results (CCeval CNNa of Table 6, although there was forgetting problem in the first two experiments, SALT still has a higher score than the baseline model S SALT. In the third experiment, we success- fully imitated the forgetting problem similar to CC and CCUser by using the distribution difference between CNN and XSum. Similar to the results in Table 3, S SALTi4u can alleviate the forgetting problem to a certain extent while improving the performance the new dataset.
summaries, and ask
Output your ranking for system- senee-egeeeatedd summaries. Use the following format, and do not add any other text.
Some examples: > b>edd
> <>bb>a
Conversation snippet: [conversation]
Human written reference SOAP note for the respec- tive snippet: [reference]
System- generated summaries:
1. [summary/]
2. [summary2]
A.4 More Discussion
3. [summary3]
4. [summary4]
Why does SALT work? First. SALT makes good use of the Sa data. From the perspective of data augmentation, SE provides a new ground truth summary from the user, and the users also ver- ify the remaining tokens in Sai- SALT helps the model to use all the tokens in both Sg and SAL. which greatly improves the utilization of human- edit data. Second, SALT gives the model more objectives. Using SAZ in SALT makes the model not just "be close the correct distribution" in the likelihood training, but also "be far away from a negative distribution".. Thus, we can teach the model to avoid making the same mistakes again, which has a special meaning for the user (Assump- tion 1).
Now, output your ranking:
Table 12: GPT4 Prompt for preference ranking.
Human Edits. Using Imitation Edits is essentially a kind of data augmentation method during training. But, when have edits to model's original output from real users, have the unique opportunity to improve model output according to their individual expectations. SALT can model such information during the training and help the model have more appropriate behaviors to serve the users better in more data-efficient way.
SALT and RLHF the Section Regarding the relationship between SALT and other RLHI's, we have some preliminary discussions here, and they need follow-up work demonstrate. It seems that SALT keeps most the advantages and disadvantages of DPO against
We discuss SALT and DPO in
Human Edits and Imitation Edits Even though SALT can be used with human-edit data or imitation-edit data to improve the summarization models, our experiments are not enough to con- clude that Imitation Edits can completely replace

---

PPO. Often, no reinforcement learning means more stable and easy training (and hyper-tuning). Our human eval also shows that SALT can make mod- els more aligned with human preference without explicit reward models, which the same with DPO. Also, it's questionable whether a good ex- plicit reward model can be learned from Sai and SE since it's not easy as positive negative movie reviews distinguish. For limitation, How does the SALT model generalize tion, compared with PPO with an explicit reward function? For example, standard RLHF methods can leverage additional unlabeled prompts by label- ing LM generations with the learned reward model. Can training with self-labeling from the SALT sim- ilarly make effective use of unlabeled prompts? Other papers like RAFT and RRHF use explicit reward models filter high-score data points for SFT. Whether we can train good reward model as a good filter also a big question here. Another difference that we will make full use all data points (SAr + Sg) during the training, but they will only use high-quality ones (Sg) and discard the rest (SAr). So theoretically, we use data more efficiently and model more information from SAI-
of distribu-


Reviews:
[{"review_id": "XGDib2y4aO", "reviewer": "Reviewer_6R2L", "summary": "", "questions": "Question A: The model training details are missing. How the base model M is trained, for how long, and what is the stop criteria? How do you make sure the model is fully trained without overfitting? \n\nQuestion B: How does the unlikelihood learning compared to other learning method like RL and contrastive learning? \n\nQuestion C: Does human edits always improve the quality of the summary in the dataset? it would be nice to show such statistics, on what edited, how the quality shift.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Reasons To Reject:\n\n**Q1:** The paper does not compare to other methods mentioned in literature review on incorporate Human Edits as Feedback to improve text summarization, for example how the used unlikelihood learning method is compared to Reinforcement learning based and constrastive learning based. The title seems to be a bit overselling. \n\n**Answer:** Thanks to the reviewer's point here. In order to solve this question, we experimented on the most representative work of RLHF recently, DPO (Direct Preference Optimization), on our CCUser dataset, because the settings of the two are similar, that is, training directly on the human preference dataset , without training explicit reward model. Specifically, we use S_AI as the rejected summary, S_E as the chosen summary and calculate the DPO loss to train the model.\n\n|**Results with GPT2**|**Rewards ACC**|**Rouge1**|**Rouge2**|**Rougel**|**Meteor**\n|-------|-------|-------|-------|-------|-------|\n|**SALT(l)**|**0.368**|**0.381**|**0.203**|**0.371**|**0.292**|\n|**SALT(l_d)**|**0.430**|**0.388**|**0.210**|**0.376**|**0.302**|\n|**SALT(l_i)**|**0.390**|**0.389**|**0.214**|**0.376**|**0.320**|\n|**SALT(u)**|**0.522**|**0.391**|**0.216**|**0.380**|**0.310**|\n|**SALT(l+u)**|**0.591**|**0.394**|**0.215**|**0.383**|**0.320**|\n|**DPO(beta=0.1)**|**0.484**|**0.379**|**0.210**|**0.369**|**0.301**|\n|**DPO(beta=0.5)**|**0.532**|**0.372**|**0.191**|**0.361**|**0.291**|\n\nHere are our findings:\n\n- It is difficult for DPO to learn an implicit reward through S_AI and S_E because most words in S_AI and S_E are the same. So we find the reward metrics (reward ACC or margin) and other metrics (ROUGE, Factuality score) become negatively correlated. If we change hyper-parameters to get high reward metrics, the ROUGE and Factuality score becomes extremely bad, and its final generated summary becomes a mess. On the contrary, if we want to keep a ROUGE, Factuality score, the reward metrics will not improve at all (similar to SFT training). It’s easy to explain from the DPO loss function:\n\n\t> pi_logratios = policy_chosen_logps - policy_rejected_logps\n\n\t> ref_logratios = reference_chosen_logps - reference_rejected_logps\n\n\t> logits = pi_logratios - ref_logratios\n\n\t> DPO loss = --F.logsigmoid(self.beta * logits)\n\nSo DPO considered penalizing the whole rejected summary, which is not a suitable behavior with human edit feedback. Again, most words in S_AI and S_E are the same, so DPO does not consider such cases.\n\n- SALT can be treated as an extension or variance of DPO, especially for human edit feedback. One straightforward solution to improve DPO on human edit feedback is to target “negative tokens'' in the rejected summary and calculate DPO loss, which aligns with our SALT ideas. After doing sequence alignment to find negative tokens, we use likelihood and unlikelihood loss for different tokens, but DPO loss can also be a good choice. We also calculate DPO rewards when we train the model with SALT, and interestingly, we find that, even though SALT loss function doesn’t explicitly maximize chosen and rejected logps, we still see similar improvement for DPO rewards metrics (e.g., ACC or margin).\n\n\t> chosen_rewards = self.beta * (policy_chosen_logps - reference_chosen_logps)\n\n\t> rejected_rewards = self.beta * (policy_rejected_logps - reference_rejected_logps)\n\nWe will update these small experiments in the appendix and discuss our findings. We also treat this as our future work with more deep and comprehensive experiments.\n\n- We don’t have enough time to try all recent RLHF methods, but here is some thoughts:\n\n\t> It seems that SALT keeps most of the advantages and disadvantages of DPO against PPO. Often, no reinforcement learning means more stable and easy training (and hyper-tuning). Our human eval also shows that SALT can make models more aligned with human preference without explicit reward models, which is the same with DPO. Also, it’s questionable whether a good explicit reward model can be learned from S_AI and S_E since it’s not as easy as positive/negative movie reviews to distinguish. For limitation, How does the SALT model generalize out of distribution, compared with PPO with an explicit reward function? For example, standard RLHF methods can leverage additional unlabeled prompts by labeling LM generations with the learned reward model. Can training with self-labeling from the SALT similarly make effective use of unlabeled prompts?\n    \n\t> Other papers like RAFT and RRHF use explicit reward models to filter high-score data points for SFT. Whether we can train a good reward model as a good filter is also a big question here. Another difference is that, we will make full use of all data points (S_AI + S_E) during the training, but they will only use high quality one (S_E) and discard the rest (S_AI). So theoretically we use data more efficiently and model more information from S_AI.\n    \nIt’s amazing to see the RLHF community grow up so quickly! We will explore and try more experiments to compare SALT with all these new methods (DPO/RAFT/RRHF) in the future.\n\n\n**Q2:** The numbers shown in the experiment results look every close, some claims might not be true given the unclear significance. For example, in Table 3, the claims about comparing different variant of the proposed methods (SALT_x)\n\n**Answer:** 95% Confidence interval results:\n\n|**Methods**|**CCUser_R1**|**CCUser_Uf**|**CC_R1**|**CC_Uf**|\n|-------|-------|-------|-------|-------|\n|**SALT(l)**|**57.77** (±0.28)|**61.02** (±1.06)|**34.27** (±0.21)|**46.45** (±0.52)|\n|**SALT(l_d)**|**57.70**  (±0.25) |**61.06** (±0.86)|**34.46**  (±0.31)|**46.58** (±0.33)|\n|**SALT(l_i)**|**57.84**  (±0.36)|**60.81** (±0.79)|**34.68** (±0.25)|**46.77** (±0.71)|\n|**SALT(u)**|**57.57** (±0.66)|**61.09** (±1.33)|**34.47** (±0.44)|**46.64**(±0.51)|\n|**SALT(l+u)**|**58.39** (±0.57)|**62.13** (±1.03)|**34.79** (±0.30)|**47.06** (±0.47)|\n|**SALT(l) +RSALT(l)**|**59.57** (±0.47)|**62.52** (±0.98)|**35.55** (±0.32)|**48.25** (±0.60)|\n|**SALT(l+u) +RSALT(l)**|**59.60** (±0.52)|**62.57** (±1.34)|**35.43** (±0.23)|**48.20** (±0.68)|\n|**SALT(l) +RSALT(l+u)**|**59.88** (±0.43)|**62.60** (±0.85)|**36.24** (±0.36)|**48.42** (±0.41)|\n|**SALT(l+u) +RSALT(l+u)**|**60.43** (±0.61)|**63.44** (±0.92)|**36.26** (±0.40)|**48.69** (±0.59)|\n\n\n**Q3:** The evaluation majorly rely on automatic metric (ROUGE) to evaluate the summarization system, which has been proved to unreliable in many areas. The human evaluation is just a quick sanity check with very small (25) samples. It would be nice to expand this to larger one, and also show some case analysis on the model behaviors.\n\n**Answer:** Thanks to the reviewer for this suggestion. Unfortunately, since CCUser is the product data, we can't even do human eval due to the privacy restrictions, that's why we only do human eval with CC (human annotated data). But we did GPT4 eval on both CC and CCUser, and the results were as expected. At the same time, due to data privacy restrictions, we cannot provide specific cases of different behavior after SALT training on CCUser in the paper.\n\n**However**, we know the importance of reproduction and these case studies to the community, so we will release our SALT code, scores, and output results of different models on the public dataset. We couldn't find a big enough public dataset with (doctor-patient conversation, note) and human edits information to train a summarization model, so we used another clinical summarization task on MIMIC-III, after-visit-summary generation task for our public dataset experiments. In addition, we use GPT4 to edit the model output to get \"human edits\". The specific experimental results are given in the next point.\n\n\n**Q2:** The paper mainly focus on single datasets, it would be nice to bring some experiments on other public dataset in the appendix to the main text.\n\n**Answer:** Thanks to the reviewer's point here. In order to solve this question, we experimented on the MIMIC-III after-visit-summary generation dataset [1] [2]. The dataset has 31,895 (Clinical note, AVS) pairs for this summarization task. We first fine-tune the gpt2 on this dataset, and then ask GPT4 to edit the trained-gpt2 generated summaries based on the input clinical notes and reference AVS summaries. GPT4 can easily provide such “human-edits” without too much prompt engineering. So we have (Clinical note, gpt2 generated summary, GPT4-edit summary) for SALT/DPO. Here are the results:\n\n|**Results with GPT2**|**Rewards ACC**|**Rouge1**|**Rouge2**|**Rougel**|**Meteor**\n|-------|-------|-------|-------|-------|-------|\n|**SALT(l)**|**0.596**|**0.500**|**0.366**|**0.487**|**0.455**|\n|**SALT(l_d)**|**0.586**|**0.512**|**0.378**|**0.498**|**0.471**|\n|**SALT(l_i)**|**0.677**|**0.502**|**0.369**|**0.489**|**0.457**|\n|**SALT(u)**|**0.860**|**0.462**|**0.317**|**0.447**|**0.406**|\n|**SALT(l+u)**|**0.777**|**0.511**|**0.375**|**0.497**|**0.468**|\n|**DPO(beta=0.1)**|**0.654**|**0.491**|**0.355**|**0.477**|**0.444**|\n|**DPO(beta=0.5)**|**0.827**|**0.485**|**0.347**|**0.470**|**0.438**|\n\nThe results here are in line with paper results, we will release codes, results and model output on this public task for the community.\n\n[1] Generation of Patient After-Visit Summaries to Support Physicians \nhttps://aclanthology.org/2022.coling-1.544/\n\n[2] Learning to Revise References for Faithful Summarization\nhttps://aclanthology.org/2022.findings-emnlp.296.pdf\n\n### Questions For The Authors:\n**Question A:** The model training details are missing. How the base model M is trained, for how long, and what is the stop criteria? How do you make sure the model is fully trained without overfitting?\n\n**Answer:** We talk about these in the Appendix (A.2 Implementation Details) to save some space. We trained the model for 10000 steps (about 10 epochs), we saved the model according to the best 3 rouge1 score, finally we use the epoch 8 checkpoint since it has the best rogue 1 score.\n\n**Question B:** How does the unlikelihood learning compared to other learning method like RL and contrastive learning?\n\n**Answer:** The answer here is the same with point1 in Reasons To Reject.\n\n**Question C:** Does human edits always improve the quality of the summary in the dataset? it would be nice to show such statistics, on what edited, how the quality shift.\n\n**Answer:** This is a very good question and future research topic, we also mentioned this in the Limitation and Ethics Statement sections that “There will be some potential problems in actual deployment: First of all, for Human-Edits, we don't know whether the user is modifying because of some kind of error in S_AI or because of the user's personal preference. These two behaviors need to be distinguished in future research or actual deployment, because the former data is more suitable for improving the problems of the model itself (such as some factual errors), and the latter data is more suitable for user-personalized training data.”. We will keep explore these directions in our future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method is novel, by using unlikelihood learning to learn human edits as feedback to improve text summarization. The proposed method is generic and could be extended text generation in many other domains. The study also has a broad interest to the research in the community of human AI collaboration.  \n2. The study of unlikelihood with imitation edits raise interesting findings: (1) it can help further tuning the text generation model with less overfitting; (2) it could help eliminate forgetting issue when using replay-based method. \n3. The experiments with careful analysis are clear and looks solid to prove the advantage of proposed method.", "reasons_to_reject": "1. The paper does not compare to other methods mentioned in literature review on incorporate Human Edits as Feedback to improve text summarization, for example how the used unlikelihood learning method is compared to Reinforcement learning based and constrastive learning based.  The title seems to be a bit overselling. \n2. The numbers shown in the experiment results look every close, some claims might not be true given the unclear significance. For example, in Table 3, the claims about comparing different variant of the proposed methods (SALT_x)\n3. The evaluation majorly rely on automatic metric (ROUGE) to evaluate the summarization system, which has been proved to unreliable in many areas. The human evaluation is just a quick sanity check with very small (25) samples. It would be nice to expand this to larger one, and also show some case analysis on the model behaviors. \n4. The paper mainly focus on single datasets, it would be nice to bring some experiments on other public dataset in the appendix to the main text.", "questions_to_authors": "Question A: The model training details are missing. How the base model M is trained, for how long, and what is the stop criteria? How do you make sure the model is fully trained without overfitting? \n\nQuestion B: How does the unlikelihood learning compared to other learning method like RL and contrastive learning? \n\nQuestion C: Does human edits always improve the quality of the summary in the dataset? it would be nice to show such statistics, on what edited, how the quality shift.", "paper_topic_and_main_contributions": "This paper proposed to use unlikelihood learning together with normal likelihood learning to incorporate human edits as feedbacks to improve text summarization. \nThe learning method use sequence alignment to identify the token changes the human make on AI-generated summary, such token changes are then encoded in the cross-entropy loss function to either penalize or promote their likelihood in the generation. \nThe experiments on clinical note generation (and other public data as presented in Appendix) show that unlikelihood learning with human edits can perform better than baseline that simple continual training on human edited data. \nThe experiments with imitation where human edits are simulated between reference ground truth summary and AI-generated summary, and confirm such data also help improve performance, meanwhile reduce the needs of costly human edits. \nThe experiments also show the unlikelihood learning with imitation edits can serve as a way of data augmentation to improve catastrophic forgetting of simply replay-based method.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[An Exploration of Post-Editing Effectiveness in Text Summarization](https://aclanthology.org/2022.naacl-main.35) (Lai et al., NAACL 2022)"}, "extra_text": {}}, {"review_id": "7ATCyqEphi", "reviewer": "Reviewer_ZppW", "summary": "", "questions": "I find the data split not very clear. In 243-246, \"we randomly select 70 notes from 7 physicians as training dataset, 10 for each physician, and divide the remaining 145 notes into 245 evaluation and test sets.\" Why is it important to mention the number of physicians covered in the training data? I also wonder if the training and test sets overlap in terms of being produced by the same physician. If so there might be implicit bias such as the use of words or the edit style.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Reasons To Reject:\n\n**Q1:** Further, the paper does not have a sufficient comparison with related works through experiments either.\n\n**Answer:** Thanks to the reviewer's point here. In order to solve this question, we experimented on the most representative work of RLHF recently, DPO (Direct Preference Optimization), on our CCUser dataset, because the settings of the two are similar, that is, training directly on the human preference dataset , without training explicit reward model. Specifically, we use S_AI as the rejected summary, S_E as the chosen summary and calculate the DPO loss to train the model.\n\n|**Results with GPT2**|**Rewards ACC**|**Rouge1**|**Rouge2**|**Rougel**|**Meteor**\n|-------|-------|-------|-------|-------|-------|\n|**SALT(l)**|**0.368**|**0.381**|**0.203**|**0.371**|**0.292**|\n|**SALT(l_d)**|**0.430**|**0.388**|**0.210**|**0.376**|**0.302**|\n|**SALT(l_i)**|**0.390**|**0.389**|**0.214**|**0.376**|**0.320**|\n|**SALT(u)**|**0.522**|**0.391**|**0.216**|**0.380**|**0.310**|\n|**SALT(l+u)**|**0.591**|**0.394**|**0.215**|**0.383**|**0.320**|\n|**DPO(beta=0.1)**|**0.484**|**0.379**|**0.210**|**0.369**|**0.301**|\n|**DPO(beta=0.5)**|**0.532**|**0.372**|**0.191**|**0.361**|**0.291**|\n\nHere are our findings:\n\n- It is difficult for DPO to learn an implicit reward through S_AI and S_E because most words in S_AI and S_E are the same. So we find the reward metrics (reward ACC or margin) and other metrics (ROUGE, Factuality score) become negatively correlated. If we change hyper-parameters to get high reward metrics, the ROUGE and Factuality score becomes extremely bad, and its final generated summary becomes a mess. On the contrary, if we want to keep a ROUGE, Factuality score, the reward metrics will not improve at all (similar to SFT training). It’s easy to explain from the DPO loss function:\n\n\t> pi_logratios = policy_chosen_logps - policy_rejected_logps\n\n\t> ref_logratios = reference_chosen_logps - reference_rejected_logps\n\n\t> logits = pi_logratios - ref_logratios\n\n\t> DPO loss = --F.logsigmoid(self.beta * logits)\n\nSo DPO considered penalizing the whole rejected summary, which is not a suitable behavior with human edit feedback. Again, most words in S_AI and S_E are the same, so DPO does not consider such cases.\n\n- SALT can be treated as an extension or variance of DPO, especially for human edit feedback. One straightforward solution to improve DPO on human edit feedback is to target “negative tokens'' in the rejected summary and calculate DPO loss, which aligns with our SALT ideas. After doing sequence alignment to find negative tokens, we use likelihood and unlikelihood loss for different tokens, but DPO loss can also be a good choice. We also calculate DPO rewards when we train the model with SALT, and interestingly, we find that, even though SALT loss function doesn’t explicitly maximize chosen and rejected logps, we still see similar improvement for DPO rewards metrics (e.g., ACC or margin).\n\n\t> chosen_rewards = self.beta * (policy_chosen_logps - reference_chosen_logps)\n\n\t> rejected_rewards = self.beta * (policy_rejected_logps - reference_rejected_logps)\n\nWe will update these small experiments in the appendix and discuss our findings. We also treat this as our future work with more deep and comprehensive experiments.\n\n- We don’t have enough time to try all recent RLHF methods, but here is some thoughts:\n\n\t> It seems that SALT keeps most of the advantages and disadvantages of DPO against PPO. Often, no reinforcement learning means more stable and easy training (and hyper-tuning). Our human eval also shows that SALT can make models more aligned with human preference without explicit reward models, which is the same with DPO. Also, it’s questionable whether a good explicit reward model can be learned from S_AI and S_E since it’s not as easy as positive/negative movie reviews to distinguish. For limitation, How does the SALT model generalize out of distribution, compared with PPO with an explicit reward function? For example, standard RLHF methods can leverage additional unlabeled prompts by labeling LM generations with the learned reward model. Can training with self-labeling from the SALT similarly make effective use of unlabeled prompts?\n    \n\t> Other papers like RAFT and RRHF use explicit reward models to filter high-score data points for SFT. Whether we can train a good reward model as a good filter is also a big question here. Another difference is that, we will make full use of all data points (S_AI + S_E) during the training, but they will only use high quality one (S_E) and discard the rest (S_AI). So theoretically we use data more efficiently and model more information from S_AI.\n    \nIt’s amazing to see the RLHF community grow up so quickly! We will explore and try more experiments to compare SALT with all these new methods (DPO/RAFT/RRHF) in the future.\n\n\n**Q2:** Empirically, the improvements observed are marginal: for a baseline ROUGE of ~50, the improvement is ~0.5 (1%). The result possibly does not strongly encourage the adoption of the proposed method.\n\n**Answer:** Thanks to the reviewer's point here. We add 95% Confidence interval results here:\n\n|**Methods**|**CCUser_R1**|**CCUser_Uf**|**CC_R1**|**CC_Uf**|\n|-------|-------|-------|-------|-------|\n|**SALT(l)**|**57.77** (±0.28)|**61.02** (±1.06)|**34.27** (±0.21)|**46.45** (±0.52)|\n|**SALT(l_d)**|**57.70**  (±0.25) |**61.06** (±0.86)|**34.46**  (±0.31)|**46.58** (±0.33)|\n|**SALT(l_i)**|**57.84**  (±0.36)|**60.81** (±0.79)|**34.68** (±0.25)|**46.77** (±0.71)|\n|**SALT(u)**|**57.57** (±0.66)|**61.09** (±1.33)|**34.47** (±0.44)|**46.64**(±0.51)|\n|**SALT(l+u)**|**58.39** (±0.57)|**62.13** (±1.03)|**34.79** (±0.30)|**47.06** (±0.47)|\n|**SALT(l) +RSALT(l)**|**59.57** (±0.47)|**62.52** (±0.98)|**35.55** (±0.32)|**48.25** (±0.60)|\n|**SALT(l+u) +RSALT(l)**|**59.60** (±0.52)|**62.57** (±1.34)|**35.43** (±0.23)|**48.20** (±0.68)|\n|**SALT(l) +RSALT(l+u)**|**59.88** (±0.43)|**62.60** (±0.85)|**36.24** (±0.36)|**48.42** (±0.41)|\n|**SALT(l+u) +RSALT(l+u)**|**60.43** (±0.61)|**63.44** (±0.92)|**36.26** (±0.40)|**48.69** (±0.59)|\n\nIn the appendix, **our human eval and gpt4 eval** also support the experimental results of SALT.\n\n**Q3:** Due to the way the data is processed, which uses humans, Google APIs, and the in-house model, it is hard to reproduce the work.\n\n**Answer:** Thanks to the reviewer's point here. Unfortunately, since CCUser is the product data, we can't release this patient-doctor-conversation-to-note data for reproduction due to the privacy restrictions.\n\nHowever, we know the importance of reproduction to the community, so we will release our SALT code, scores, and output results of different models on the public dataset. We couldn't find a big enough public dataset with (doctor-patient conversation, note) and human edits information to train a summarization model, so we experimented on the MIMIC-III after-visit-summary generation dataset [1] [2] for all SALT experiments. In addition, we use GPT4 to edit the model output to get \"human edits\". The dataset has 31,895 (Clinical note, AVS) pairs for this summarization task. We first fine-tune the gpt2 on this dataset, and then ask GPT4 to edit the trained-gpt2 generated summaries based on the input clinical notes and reference AVS summaries. GPT4 can easily provide such “human-edits” without too much prompt engineering. So we have (Clinical note, gpt2 generated summary, GPT4-edit summary) for SALT/DPO. Here are the results:\n\n|**Results with GPT2**|**Rewards ACC**|**Rouge1**|**Rouge2**|**Rougel**|**Meteor**\n|-------|-------|-------|-------|-------|-------|\n|**SALT(l)**|**0.596**|**0.500**|**0.366**|**0.487**|**0.455**|\n|**SALT(l_d)**|**0.586**|**0.512**|**0.378**|**0.498**|**0.471**|\n|**SALT(l_i)**|**0.677**|**0.502**|**0.369**|**0.489**|**0.457**|\n|**SALT(u)**|**0.860**|**0.462**|**0.317**|**0.447**|**0.406**|\n|**SALT(l+u)**|**0.777**|**0.511**|**0.375**|**0.497**|**0.468**|\n|**DPO(beta=0.1)**|**0.654**|**0.491**|**0.355**|**0.477**|**0.444**|\n|**DPO(beta=0.5)**|**0.827**|**0.485**|**0.347**|**0.470**|**0.438**|\n\nThe results here are in line with paper results, we will release codes, results and model output on this public task for the community.\n\n[1] Generation of Patient After-Visit Summaries to Support Physicians \nhttps://aclanthology.org/2022.coling-1.544/\n\n[2] Learning to Revise References for Faithful Summarization\nhttps://aclanthology.org/2022.findings-emnlp.296.pdf\n\n\n\n\n### Questions For The Authors:\n**Question** I find the data split not very clear. In 243-246, \"we randomly select 70 notes from 7 physicians as training dataset, 10 for each physician, and divide the remaining 145 notes into 245 evaluation and test sets.\" Why is it important to mention the number of physicians covered in the training data? I also wonder if the training and test sets overlap in terms of being produced by the same physician. If so there might be implicit bias such as the use of words or the edit style.\n\n\n**Answer:** With regard to the number of doctors, we just report some information of our data truthfully. We try to make the sampling of data random. The data of train and test may come from the same doctor, but they will not come from the same note. We admit the existence of implicit bias, in fact this is a very interesting topic, we also mentioned this in the Limitation and Ethics Statement sections that “There will be some potential problems in actual deployment: First of all, for Human-Edits, we don't know whether the user is modifying because of some kind of error in S_AI or because of the user's personal preference. These two behaviors need to be distinguished in future research or actual deployment, because the former data is more suitable for improving the problems of the model itself (such as some actual errors), and the latter data is more suitable for user-personalized training data.”. We will keep exploring these directions in our future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea of (un)likelihood training using the original output as well as human edits is reasonable and could inspire other NLP applications.\n2. The formation of gold reference as imitation edited data is interesting, although it might be overkill.", "reasons_to_reject": "1. Due to the way the data is processed, which uses humans, Google APIs, and the in-house model, it is hard to reproduce the work.\n2. Empirically, the improvements observed are marginal: for a baseline ROUGE of ~50, the improvement is ~0.5 (1%). The result possibly does not strongly encourage the adoption of the proposed method.\n3. Further, the paper does not have a sufficient comparison with related works through experiments either.\n\n (Update: I think 2 and 3 are partially addressed as the authors added more experiments to compare with existing literature, and reported significance testing of existing results in the rebuttal.)", "questions_to_authors": "I find the data split not very clear. In 243-246, \"we randomly select 70 notes from 7 physicians as training dataset, 10 for each physician, and divide the remaining 145 notes into 245 evaluation and test sets.\" Why is it important to mention the number of physicians covered in the training data? I also wonder if the training and test sets overlap in terms of being produced by the same physician. If so there might be implicit bias such as the use of words or the edit style.", "paper_topic_and_main_contributions": "This work proposes to apply unlikelihood training to human-edited summary data in the clinical domain. The author in addition proposed to experiment with gold reference as an imitation edited data. There are a good amount of experiments and evaluations, but the improvement is not consistent or significant.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Between 075 and 098, the paradigm of learning from human edits is explained. I feel that some related works could be mentioned here, in the domain of summarization, or perhaps other NLP tasks."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. 256-257: The length symbol is confusing - I initially thought it meant length \"minus\" the output.\n2. The presentation of the first paragraph in section 6 and the experiment table should be improved. Why are different font sizes used in different rows in Table 3 and Table 4?\n3. The \"Ethics Statement\" does not discuss the ethical concerns of this work and data used, but rather reads like an extended part of the limitations."}}, {"review_id": "uvF0zv2om2", "reviewer": "Reviewer_88MQ", "summary": "", "questions": "Your work is great, yet proposing your own models, on your own dataset, with your own metrics, could be seen as siloed.\nConnecting your work to prior work is an important step towards giving the reader of your work a nuanced understanding of how your work fits within the research space. Do you know of (external) baselines you could include which would contextualize your work?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Reasons To Reject:\n\n**Q1:** The experiment section lacks real external comparisons. Currently the only systems that are compared are the full system, and several ablated versions of the method. In order to understand the method, it would be important to compare to prior work. Prior work in edit-based model could interesting to compare (the following tutorial from NAACL 2022 might be a good place to find pointers: https://text-editing.github.io/ ). In essence, a reader needs to understand: is this method an efficient way to integrate human feedback/edits compared to other methods, not just which components of the proposed system work/do not work (ablation study).\n\n**Answer:** Thanks to the reviewer's point here. In order to solve this question, we experimented on the most representative work of RLHF recently, DPO (Direct Preference Optimization), on our CCUser dataset, because the settings of the two are similar, that is, training directly on the human preference dataset , without training explicit reward model. Specifically, we use S_AI as the rejected summary, S_E as the chosen summary and calculate the DPO loss to train the model.\n\n|**Results with GPT2**|**Rewards ACC**|**Rouge1**|**Rouge2**|**Rougel**|**Meteor**\n|-------|-------|-------|-------|-------|-------|\n|**SALT(l)**|**0.368**|**0.381**|**0.203**|**0.371**|**0.292**|\n|**SALT(l_d)**|**0.430**|**0.388**|**0.210**|**0.376**|**0.302**|\n|**SALT(l_i)**|**0.390**|**0.389**|**0.214**|**0.376**|**0.320**|\n|**SALT(u)**|**0.522**|**0.391**|**0.216**|**0.380**|**0.310**|\n|**SALT(l+u)**|**0.591**|**0.394**|**0.215**|**0.383**|**0.320**|\n|**DPO(beta=0.1)**|**0.484**|**0.379**|**0.210**|**0.369**|**0.301**|\n|**DPO(beta=0.5)**|**0.532**|**0.372**|**0.191**|**0.361**|**0.291**|\n\nHere are our findings:\n\n- It is difficult for DPO to learn an implicit reward through S_AI and S_E because most words in S_AI and S_E are the same. So we find the reward metrics (reward ACC or margin) and other metrics (ROUGE, Factuality score) become negatively correlated. If we change hyper-parameters to get high reward metrics, the ROUGE and Factuality score becomes extremely bad, and its final generated summary becomes a mess. On the contrary, if we want to keep a ROUGE, Factuality score, the reward metrics will not improve at all (similar to SFT training). It’s easy to explain from the DPO loss function:\n\n\t> pi_logratios = policy_chosen_logps - policy_rejected_logps\n\n\t> ref_logratios = reference_chosen_logps - reference_rejected_logps\n\n\t> logits = pi_logratios - ref_logratios\n\n\t> DPO loss = --F.logsigmoid(self.beta * logits)\n\nSo DPO considered penalizing the whole rejected summary, which is not a suitable behavior with human edit feedback. Again, most words in S_AI and S_E are the same, so DPO does not consider such cases.\n\n- SALT can be treated as an extension or variance of DPO, especially for human edit feedback. One straightforward solution to improve DPO on human edit feedback is to target “negative tokens'' in the rejected summary and calculate DPO loss, which aligns with our SALT ideas. After doing sequence alignment to find negative tokens, we use likelihood and unlikelihood loss for different tokens, but DPO loss can also be a good choice. We also calculate DPO rewards when we train the model with SALT, and interestingly, we find that, even though SALT loss function doesn’t explicitly maximize chosen and rejected logps, we still see similar improvement for DPO rewards metrics (e.g., ACC or margin).\n\n\t> chosen_rewards = self.beta * (policy_chosen_logps - reference_chosen_logps)\n\n\t> rejected_rewards = self.beta * (policy_rejected_logps - reference_rejected_logps)\n\nWe will update these small experiments in the appendix and discuss our findings. We also treat this as our future work with more deep and comprehensive experiments.\n\n- We don’t have enough time to try all recent RLHF methods, but here is some thoughts:\n\n\t> It seems that SALT keeps most of the advantages and disadvantages of DPO against PPO. Often, no reinforcement learning means more stable and easy training (and hyper-tuning). Our human eval also shows that SALT can make models more aligned with human preference without explicit reward models, which is the same with DPO. Also, it’s questionable whether a good explicit reward model can be learned from S_AI and S_E since it’s not as easy as positive/negative movie reviews to distinguish. For limitation, How does the SALT model generalize out of distribution, compared with PPO with an explicit reward function? For example, standard RLHF methods can leverage additional unlabeled prompts by labeling LM generations with the learned reward model. Can training with self-labeling from the SALT similarly make effective use of unlabeled prompts?\n    \n\t> Other papers like RAFT and RRHF use explicit reward models to filter high-score data points for SFT. Whether we can train a good reward model as a good filter is also a big question here. Another difference is that, we will make full use of all data points (S_AI + S_E) during the training, but they will only use high quality one (S_E) and discard the rest (S_AI). So theoretically we use data more efficiently and model more information from S_AI.\n    \nIt’s amazing to see the RLHF community grow up so quickly! We will explore and try more experiments to compare SALT with all these new methods (DPO/RAFT/RRHF) in the future.\n\n\n**Q2:** Some reorganization of the content, and improving priorities within the main paper might be beneficial. For instance, the authors mention a human evaluation, yet there is no space in the paper for its results. When going into the Appendix, one finds out that such results are positive and interesting, even though they are themselves buried in a Table with GPT4-based experiments. Why not give the human evaluation results importance? Isn't this the gold standard?\n\n\n**Answer:** Thanks to the reviewer for this suggestion. Yes, human eval is most important in generation task. Unfortunately, since CCUser is the product data, we can't do human eval on it due to the privacy restrictions, that's why we only do human eval with CC (human annotated data). But we did GPT4 eval on both CC and CCUser, and the results were as expected. At the same time, due to data privacy restrictions, we cannot provide specific case studies of different behavior after SALT training on CCUser in the paper. These reasons made us decide to just put them in the appendix when submitting.\n\n\n**Q3:** In general with the method/evaluation, it seems like the authors are reinventing the wheel a bit. The method is quite complex, with many variants (when in the end, the variant with all components seem to win it all), the SAFE metric proposed seems a bit ad-hoc. Why not use well-known, existing edit-based metrics like SARI (or newer variants) from edit-based tasks (such as simplification), they fulfill similar objectives (measuring distance from an bad reference, and similarity to a good reference).\n\n**Answer:** Thanks to the reviewer for this suggestion. Actually, we find DPO defined reward ACC is also a good and simple metric (try to see whether chosen_rewards > rejected_rewards) for a similar goal, and we report above, which also supports our previous results. We will explore and try more related metrics for this goal.\n\n\n\n\n### Questions For The Authors:\n**Question** Your work is great, yet proposing your own models, on your own dataset, with your own metrics, could be seen as siloed. Connecting your work to prior work is an important step towards giving the reader of your work a nuanced understanding of how your work fits within the research space. Do you know of (external) baselines you could include which would contextualize your work?\n\n\n\n**Answer:** The answer here is the same with point1 in Reasons To Reject.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper addresses a novel task within summarization: how to effectively integrate user feedback -- in the form edits -- to improve system output summaries.\n- The method is complex and seem to yield gains on several metrics. In particular, the experiments reveal that with RSALT, the model is able to maintain (or close) performance on the original dataset, while yielding improvements on the new task.", "reasons_to_reject": "- The experiment section lacks real external comparisons. Currently the only systems that are compared are the full system, and several ablated versions of the method. In order to understand the method, it would be important to compare to prior work. Prior work in edit-based model could interesting to compare (the following tutorial from NAACL 2022 might be a good place to find pointers: https://text-editing.github.io/ ). In essence, a reader needs to understand: is this method an efficient way to integrate human feedback/edits compared to other methods, not just which components of the proposed system work/do not work (ablation study).\n- Some reorganization of the content, and improving priorities within the main paper might be beneficial. For instance, the authors mention a human evaluation, yet there is no space in the paper for its results. When going into the Appendix, one finds out that such results are positive and interesting, even though they are themselves buried in a Table with GPT4-based experiments. Why not give the human evaluation results importance? Isn't this the gold standard?\n- In general with the method/evaluation, it seems like the authors are reinventing the wheel a bit. The method is quite complex, with many variants (when in the end, the variant with all components seem to win it all), the SAFE metric proposed seems a bit ad-hoc. Why not use well-known, existing edit-based metrics like SARI (or newer variants) from edit-based tasks (such as simplification), they fulfill similar objectives (measuring distance from an bad reference, and similarity to a good reference).", "questions_to_authors": "Your work is great, yet proposing your own models, on your own dataset, with your own metrics, could be seen as siloed.\nConnecting your work to prior work is an important step towards giving the reader of your work a nuanced understanding of how your work fits within the research space. Do you know of (external) baselines you could include which would contextualize your work?", "paper_topic_and_main_contributions": "The paper proposes a method to finetune a summarization model leveraging human edits (or imitation edits) that can steer an initial system summary towards a better reference. The method, named SALT, relies on improving the likelihood of tokens inserted in the edit, while lowering the likelihood of tokens removed in the edit. Further, to avoid catastrophic forgetting, the model is jointly finetuned on the original dataset (replay-based SALT, or RSALT). The model presents results based on ROUGE, UMLS-F1 (a medical-domain specific score), and SAGE (to focus on the removal of undesired tokens). These experiments are conducted on both the original CC dataset as well as a newly introduced (and smaller) CCUser. Experiments confirm that RSALT results leads to metric improvements.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zlI3aCKdhe", "reviewer": "Reviewer_sCPx", "summary": "", "questions": "1) Is there a thought about doing this process repeatedly? Getting human feedback, retraining the model, and then getting further feedback? \nCan this iterative process help to reduce costs by being more data efficient?\n\n2) Section 3: This presentation can be drastically improved. a) Lacks citations - unclear where the datasets are coming from, b) Not sure what the AI is in Section 3.2, c) The naming of CC and CCUser can be improved, d) Section 3 needs some introduction that connects the datasets to how they are being used.\n\n3) Section 4: a) A figure might help. b) What exactly is the point of E-C and E-NC? This does not correspond to the 3 points described in 269-275. Why would one want to reduce the prob. of E-C tokens (as per Eq 2)? What am I missing here?\n\n4) Line 491: Was any statistical test performed? Otherwise, the usage of the word might be misleading.\n\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Reasons To Reject:\n\n**Q1:** The presentation of Sections 3 and 4 is currentlnotesor. I have questions about the datasets and the approach itself (see questions / suggestions below).\n\n**Answer:** Thanks to the reviewer's point here. We will revise these two sections in our final version based on your suggestions.\n\n\n**Q2:** The imitation edits part is not convincing to me - given the diversity in generation, what is the point of aligning the AI-generated output to the ground truth? How to ensure that the AI-generated tokens that are not in ground truth are actually bad? The current experiments should include some statistics around this alignment between AI-generated summaries and the ground truth. Mhuman editsg some constraints, like only considering entities and their missing attributes in the AI-generated summary, etc. might help.\n\n\n**Answer:** Thanks to the reviewer for this suggestion. In lines 570 to 587 of the paper, we discussed how to use a simple filter to control the quality of such imitation edits. Lines 707-715 of the limitation section already refer to your suggestion of \"only considering entities and their missing attributes in the AI-generated summary\". \n\nIn general, we agree that the current imitation edits have significant shortcomings, in addition to the above two points, we found that GPT4 can provide \"human-like edits\" data very well, so using GPT4 edits as imitation edits is another option. Here we provide the experimental results based on GPT4 edits on the public dataset (also for reproduction). Here are the details:\n\nWe couldn't find a big enough public dataset with (doctor-patient conversation, note) and human edits information to train a summarization model, so we experimented on the MIMIC-III after-visit-summary generation dataset [1] [2] for all SALT experiments (We will release our SALT code, scores, and output results of different models on this public dataset). In addition, we use GPT4 to edit the model output to get \"human edits\". The dataset has 31,895 (Clinical note, AVS) pairs for this summarization task. We first fine-tune the GPT2 on this dataset, and then ask GPT4 to edit the trained-GPT2 generated summaries based on the input clinical notes and reference AVS summaries. GPT4 can easily provide such “human-edits” without too much prompt engineering. So we have (Clinical note, GPT2 generated summary, GPT4-edit summary) for SALT. Here are the results:\n\n|**Results with GPT2**|**Rewards ACC**|**Rouge1**|**Rouge2**|**Rougel**|**Meteor**\n|-------|-------|-------|-------|-------|-------|\n|**SALT(l)**|**0.596**|**0.500**|**0.366**|**0.487**|**0.455**|\n|**SALT(l_d)**|**0.586**|**0.512**|**0.378**|**0.498**|**0.471**|\n|**SALT(l_i)**|**0.677**|**0.502**|**0.369**|**0.489**|**0.457**|\n|**SALT(u)**|**0.860**|**0.462**|**0.317**|**0.447**|**0.406**|\n|**SALT(l+u)**|**0.777**|**0.511**|**0.375**|**0.497**|**0.468**|\n\nThe results here are in line with paper results.\n\n[1] Generation of Patient After-Visit Summaries to Support Physicians \nhttps://aclanthology.org/2022.coling-1.544/\n\n[2] Learning to Revise References for Faithful Summarization\nhttps://aclanthology.org/2022.findings-emnlp.296.pdf\n\n\n\n\n### Questions For The Authors:\n**Question 1** Is there a thought about doing this process repeatedly? Getting human feedback, retraining the model, and then getting further feedback? Can this iterative process help to reduce costs by being more data efficient?\n\n**Answer:** Thanks to the reviewer for this suggestion, this is a very interesting topic and a project we are currently working on. Specifically, we used the human eval data collected from the product before, so it was impossible to collect human edits for the new coming batch to update the model parameters using SALT in real-time, and then collect human edits for the next batch (continuously loop until the human is satisfied). This is an experiment we want to conduct in the future, but it may require a lot of manpower to really participate in the human-in-the-training loop. \n\nThe high-quality edits of GPT4 above allow us to use GPT4 to simulate the above process and achieve GPT4-in-the-training-loop (GPT4 can be regarded as a teacher model here). This is our ongoing experiment to support the evidence that we let experts enter the training loop in the future, and the relevant results will be shown in the next paper.\n\n\n\n**Question 2** Line 491: Was any statistical test performed? Otherwise, the usage of the word might be misleading.\n\n**Answer:** Thanks to the reviewer's point here. We add 95% Confidence interval results here:\n\n|**Methods**|**CCUser_R1**|**CCUser_Uf**|**CC_R1**|**CC_Uf**|\n|-------|-------|-------|-------|-------|\n|**SALT(l)**|**57.77** (±0.28)|**61.02** (±1.06)|**34.27** (±0.21)|**46.45** (±0.52)|\n|**SALT(l_d)**|**57.70**  (±0.25) |**61.06** (±0.86)|**34.46**  (±0.31)|**46.58** (±0.33)|\n|**SALT(l_i)**|**57.84**  (±0.36)|**60.81** (±0.79)|**34.68** (±0.25)|**46.77** (±0.71)|\n|**SALT(u)**|**57.57** (±0.66)|**61.09** (±1.33)|**34.47** (±0.44)|**46.64**(±0.51)|\n|**SALT(l+u)**|**58.39** (±0.57)|**62.13** (±1.03)|**34.79** (±0.30)|**47.06** (±0.47)|\n|**SALT(l) +RSALT(l)**|**59.57** (±0.47)|**62.52** (±0.98)|**35.55** (±0.32)|**48.25** (±0.60)|\n|**SALT(l+u) +RSALT(l)**|**59.60** (±0.52)|**62.57** (±1.34)|**35.43** (±0.23)|**48.20** (±0.68)|\n|**SALT(l) +RSALT(l+u)**|**59.88** (±0.43)|**62.60** (±0.85)|**36.24** (±0.36)|**48.42** (±0.41)|\n|**SALT(l+u) +RSALT(l+u)**|**60.43** (±0.61)|**63.44** (±0.92)|**36.26** (±0.40)|**48.69** (±0.59)|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) Well-motivated. I am convinced that human edits are a better form of incorporating human feedback than ratings. It especially holds value since the data can be naturally obtained from the user workflows.\n\n2) Using unlikelihood training to incorporate the human edits (269-275) makes sense to me and is sound - the presentation can be improved though (see below).\n\n3) The experiments are comprehensive, with sufficient analysis and ablations.", "reasons_to_reject": "1) The presentation of Sections 3 and 4 is currently poor. I have questions about the datasets and the approach itself (see questions / suggestions below).\n\n2) The imitation edits part is not convincing to me - given the diversity in generation, what is the point of aligning the AI-generated output to the ground truth? How to ensure that the AI-generated tokens that are not in ground truth are actually bad? The current experiments should include some statistics around this alignment between AI-generated summaries and the ground truth. Maybe putting some constraints, like only considering entities and their missing attributes in the AI generated summary, etc. might help.", "questions_to_authors": "1) Is there a thought about doing this process repeatedly? Getting human feedback, retraining the model, and then getting further feedback? \nCan this iterative process help to reduce costs by being more data efficient?\n\n2) Section 3: This presentation can be drastically improved. a) Lacks citations - unclear where the datasets are coming from, b) Not sure what the AI is in Section 3.2, c) The naming of CC and CCUser can be improved, d) Section 3 needs some introduction that connects the datasets to how they are being used.\n\n3) Section 4: a) A figure might help. b) What exactly is the point of E-C and E-NC? This does not correspond to the 3 points described in 269-275. Why would one want to reduce the prob. of E-C tokens (as per Eq 2)? What am I missing here?\n\n4) Line 491: Was any statistical test performed? Otherwise, the usage of the word might be misleading.\n\n", "paper_topic_and_main_contributions": "The paper presents an approach (referred to as SALT) to incorporate human edits for improving summarization of clinical texts. The approach relies on a combination of likelihood and unlikelihood training to reduce the probability of tokens that are edited and increase the probability of tokens that are retained and that of the new tokens that are added by the human user. The authors perform experiments using two datasets and show improvements in summarization across a variety of metrics.\n\nOverall, I believe that the idea is interesting and naturally suited for the problem. However, I have concerns about some parts of the approach and the presentation.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1) 04: Change to “Existing work in general domain abstractive summarization uses human feedback….”\n\n2) 295: add “respectively”.\n\n3) 341: combing -> combining\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces SALT, a novel method utilizing unlikelihood training with human edits to enhance clinical text summarization. Reviewers find the concept of incorporating human edits well-motivated and appreciate the comprehensive experiments and ablations. However, concerns about presentation clarity in certain sections and the effectiveness of imitation edits are raised. Reviewers suggest broader comparisons with prior work, the adoption of established edit-based metrics, and a deeper exploration of alignment between AI-generated summaries and ground truth.
Decision: Accept-Main