FaMeSumm: Investigating and Improving Faithfulness of Medical Summarization
Authors: Nan Zhang
Yusen Zhang
Wu Guo
Prasenjit Mitra
Rui Zhang
Venue: EMNLP 2023 Main (2023)
Keywords: Medical Summarization
Factuality
Faithful Summarization
Abstractive Summarization
TL;DR: We introduce FaMeSumm as a faithfulness improvement method for medical summarization and achieve state-of-the-art performances on several benchmarks.

Abstract: Summaries of medical text shall be faithful by being consistent and factual with source inputs, which is an important but understudied topic for safety and efficiency in healthcare. In this paper, we investigate and improve faithfulness in summarization on a broad range of medical summarization tasks. Our investigation reveals that current summarization models often produce unfaithful outputs for medical input text. We then introduce FaMeSumm, a framework to improve faithfulness by fine-tuning pre-trained language models based on medical knowledge. FaMeSumm performs contrastive learning on designed sets of faithful and unfaithful summaries, and it incorporates medical terms and their contexts to encourage faithful generation of medical terms. We conduct comprehensive experiments on three datasets in two languages: health question and radiology report summarization datasets in English, and a patient-doctor dialogue dataset in Chinese. Results demonstrate that FaMeSumm is flexible and effective by delivering consistent improvements over mainstream language models such as BART, T5, mT5, and PEGASUS, yielding state-of-the-art performances on metrics for faithfulness and general quality. Human evaluation by doctors also shows that FaMeSumm generates more faithful outputs. Our code is available at https://github.com/psunlpgroup/FaMeSumm.

FAMESUMM: Investigating and Improving Faithfulness of Medical Summarization
Nan Zhang', Yusen Zhang', Wu Guo?, Prasenjit Mitral Rui Zhang'
'Te Pennsylvania State University
CChildren's Hospital Affiliated of Zhengzhou University $L3S Research Center (niz5124,yf25488, prata, rmz5277 jepsuu edu, guoscho@ 126.com
Abstract
2021): a summary is unfaithful if it contains in- trinsic error (the fact that contradicts the source) or extrinsic error (the fact that cannot be directly inferred from the source text), while a faithful sum- mary should be free from both. Unfaithful sum- maries will pose significant healthcare risks by mis- leading patients and medical providers. However, to the best of our knowledge, only few papers in- vestigate the faithfulness of medical summarization by providing systematic categorization of errors for medical domain (Otmakhova et al., 2022; Adams et al., 2022). Additionally, recent investigation and faithfulness improvement (Zhang et al., 2020b; Alambo et al., 2022) works are limited to certain types of medical text such as radiology reports.
Summaries of medical text shall be faithful by being consistent and factual with source inputs, which an important but understud- ied topic for safety and efficiency in health- care. In this paper, we investigate and im- prove faithfulness in summarization on broad range of medical summarization tasks. Our investigation reveals that current summariza- tion models often produce unfaithful outputs for medical input text. We then introduce FAMESUMM, framework improve faithful- ness by fine-tuning pre-trained language mod- els based on medical knowledge. FAMESUMM performs contrastive learning on designed sets of faithful and unfaithful summaries, and it in- corporates medical terms and their contexts encourage faithful generation of medical terms. We conduct comprehensive experiments on three datasets in two languages: health question and radiology report summarization datasets in English, and a patient-doctor dia- logue dataset in Chinese. Results demonstrate that FAMES UMM is flexible and effective by delivering consistent improvements over main- stream language models such as BART, T5, mT5, and PEGASUS, yielding state-of-the-an performances on metrics for faithfulness and general quality. Human evaluation by doctors also shows that UAME:SUM generates more faithful outputs. Our code available at https: //gi thub. com/ http://grrou//aaMMMuuu
Moreover, as many summarization approaches are based on language models that are pretrained general domain text, they inadequate gen- erate faithful summaries due to the lack of medical domain knowledge. For example, although He et al. (2021) and Zhang et al. (2021) leveraged pre- trained models such as T5 (Raffel et al., 2019) and BART (Lewis et al., 2020), their performance is suboptimal because they did not incorporate medi- cal knowledge into their models.
In this work, we aim to both investigate and im- prove the faithfulness of medical summarization. We first provide taxonomy of faithfulness error types appearing in medical summaries and analyze several competitive baselines. Our investigation in Table shows that current summarization mod- els make a significant amount of faithfulness er- rors. Then, we introduce MASEUUM to improve Fathfulness for Medical Summarization. FAME- SUMM is general-purpose framework applica- ble to various language models many medical summarization tasks. It adopts objectives that finetune pre-trained language models explicitly model faithfulness and medical knowledge. The first one uses contrastive learning (Khosla et al., 2020; Cao and Wang, 2021). FAMESUMM adopts
1 Introduction
Summarizing medical text a key step towards improving the efficiency in healthcare (Liu et al., 2019; Krishna et al., 2020), including applications in summarizing medical dialogues (Joshi et al., 2020), clinical notes (Kanwal and Rizzo, 2022), health questions (He al.. 2021), and radiology reports (Dai et al., 2021).
An important but understudied issue of medical text summarization faithfulness, as defined in recent literature (Maynez et al., 2020; Huang et al.,

---

Category
Description
Example
Percentage
Intrinsic Errors
Entity Rela-
Expressing wrong relations be-
Source: ::: .. IA ME BENT.
5.25%
tionship Entity
tween two entities or actions,
Prediction: KM
E....
Generating wrong entities.
Source: -. 20 A
5 176emm 1661.kg...
3.00%
Prediction: 2 1011 ff.. 3kg...
Negation
Ignoring adding negation words.
Source: ::::1: dE 1
my E Ne (9, 1997
3.50%
33 #:: 7 Cr
Prediction: 7: IA DRA oi WH
Question
Misinterpreting question into statement vice versu.
Source: # A A MT th BARREMET
0.75%
Prediction: 60 th
Extrinsic Errors
Template
Generating statement that fol- lows the patterns (template) of training samples.
Prediction: No evidence of acute cardiopulmonnrr pracess." Gold: Minimal blunting of the right costophrenic angle, suggest- ing small pleural effusion. No focal infiltrate.
18.00%
Extraneous Fact
Add additional facts that are not the source.
Prediction: No evidence of active latent tuherculosis.
14.50%
Gold: Unremarkahle radiographic examination of the chest. No
radiographic evidence of tuberculosis.
Low Speci- licity
Not specific enough to describe the siluation.
Prediction: What are the treatments for vomiting? Gold: What can I give my 8-month-old for conslant vomiting?
2.00%
Table 1: A taxonomy faithfulness error categories of the baseline models. The translation Chinese texts is provided in Appendix A. Words marked in waves are the errors and those marked underlines are the evidence from the source or reference summary to infer the errors. This sentence is common pattern in the training set and thus is template.
much simpler heuristics (as straightforward as rule- based copying and manipulating source texts) than other contrastive learning baselines and achieve state-of-the-art performances. The second objec- tive learns medical knowledge by modeling med- ical terms and their contexts in the loss function. We show that directly modeling context tokens of medical terms is an effective design for faithfulness improvement, which offers enrichment for existing approaches to learning medical knowledge (Joshi al., 2020; Michalopoulos al., 2022).
edge. Our contributions are: (1) We investigate existing summarization models for medical text to reveal their faithfulness issues in the medical domain; (2) We propose the FAMESUMM UMM frame- work with two fine-tuning strategies to improve faithfulness; (3) We conduct a comprchensive set
of experiments to demonstrate the effectiveness FAMESUMM by improving mainstream language models and achieving state-of-the-art performances on several benchmarks.
We apply FAMESUMM to variety of back- bone pretrained language models on three datasets including health question summarization, radiol- ogy report summarization, and medical dialogue summarization. We compare FAMESUMM with baselines including backbone models, faithfulness- based models, state-of-the-es art medical summariz- ers, and GPT-3 as an example of large language models (LLMss). FAMESUMM generates 16% more faithful summaries than GPT-3 based on doc- tors' evaluation, and it provides consistent score improvements over baselines according to auto- matic metrics. Unlike recent methods (Adams al., 2022; Chen al., 2022) that train addi- tional models for faithfulness. as a cost-efficient method, FA MAMMUMM demonstrates the possibil- ity of maximizing faithfulness by designing simple contrastive sets and incorporating medical knowl-
2 Investigating Faithfulness of Medical Summarization
Comprehensive error analysis can improve the un- derstanding of the severity of faithfulness issues, and help researchers better design model for im- proving faithfulness. However, little research has studied the faithfulness issues in medical summa- rization. Therefore, we provide a systematic tax- onomy of faithfulness errors. Specifically, we first randomly pick 100 samples from HQS, RRS, and MDS datasets (Section 4. 4.1), respectively, form- ing 300 samples in total. We use fine-tuned trans- former models and the pointer-generator network as baselines to generate summaries for each sample, and we manually check and categorize the faithful- ness errors of the summaries. Inspired by Maynez et al. (2020), we first classify the faithfulness errors

---

Reference: What drinks do not contain Vitamin K
Positive Example: they all have Vitamin K
Source Contexts: "not", "contain", "Vitamiin K"
Prediction
UMLS
Contrast
CcL
Compare
LCE
Encourage Generation of Medical Knowledge
Negative Example:
What do not contain sugar?
MKK
Reference
Learning from Contrastive Medical Summary
Medical Knowledge Incorporation
Cross Entropy
FaMeSumm L
LCL.
MMKI
CCR
Traditional Models
CCF
Figure 1: Diagram of FAMESUMM architecture with an example reference summary. The underlined part in the reference contains a medical term ("Vitamin K"") and its context ("do not contain") that are modeled by FAMESUMM.
found in the 300 samples into intrinsic errors and extrinsic errors. Then, we further cluster the error cases in finer granularity based on their main char- acteristics: (1) The intrinsic error includes entity error, entity relationship error, negation error, ques- tion error, and modifier error, and (2) the extrinsic error contains template error, low specificity error, and extraneous fact error. Complementing the er- types in Pagnoni al. (2021) with types that are specific the medical domain (e.g., low specificity error), Table shows examples of each error type produced by our baselines along with their popularity. As shown the table, 47.00 .7.00% of the samples have faithfulness errors, demonstrating that current summarization models make signif- icant amount of faithfulness errors. Among all errors, entity relationship/lemplate errors (5.25%, 18.00% are the most common intrinsic/extrinsic errors, respectively.
3.1 Learning from Contrastive Summaries
For our first objective, we design positive and neg- ative sets of summaries guided by medical entities, and we use contrastive learning to train models to distinguish faithful and unfaithful summaries. Medical terms defined in Section 4.4.
We build positive and negative set for every training instance implement contrastive learning (Cao and Wang, 2021). A positive set (P) contains faithful summaries of the source text while a nega- tive set (N) contains unfaithful and suspected un- faithful summaries (references with unmentioned medical terms).
Designing Contrastive Medical Summaries We use two steps to build the positive set. Reference summary with faithfulness valida- tion. Reference summaries have been found to contain unfaithful contents (Maynez et al., 2020; Dhingra et al., 2019), and thus we perform a faith- fulness validation before adding them to the posi- tive set. Specifically, we apply simple heuristic algorithm that a reference summary will be in the positive set if all its medical concepts show up the corresponding source. Although a reference summary with unmentioned medical terms does not necessarily make itself unfaithful, we do no- tice the faithfulness issues between source and reference in both our dataset and existing litera- ture. Since having unmentioned medical terms is a good indicator of extraneous information, we put a reference with unmentioned medical terms into the negative set'.
3 Improving Faithfulness of Medical Summarization
We propose FAMESUMM, faithfulness optimiza- tion framework guided by medical domain knowl- edge for pretrained language models. As shown in Figure 1, the first objective LaL uses contrastive learning (CL) over set of curated faithful and unfaithful summaries created by perturbing medi- cal entities and concepts (Section 3.1); the second objective LMKI explicitly performs medical knowl- edge incorporation (MKT) and encourages generat- ing accurate medical terms from source texts (Sec- tion 3.2). These strategies complement each other during the fine-tuning stage in addition the cross-entropy loss LCE (Section 3.3). FAMESUMM is flexible because is widely applicable to various backbone pretrained language models.
Data augmentation by sentence extraction. We further augment positive summaries by extract-
Wee experiment with placing all references into the pos- itive sets but notice suboptimal faithfulness performance as descrihed in Section 5.

---

ing sentences from the source. Depending on the placement of the reference summary from the previous strategy, we extract one or several sen- tences (or utterances in dialogue) from the source as positive example(s) to meet the minimum size requirement of the positive set. We extract con- tent in the following order of priority: (I) extract sentence that contains medical terms showing up in both source and reference if there is one, (2) extract the longest sentence (or the last utterance in dialogue), (3) extract the first utterance (usu- ally a question posted by patient) if the data is dialogue, and (4) apply back- translation by trans- lating the sentence from (1), (2), or (3) to German and then back to English as paraphrase. The negative set is constructed by manipulating sentences from source text by the following:
where his hj. h& are representations for summaries Vi. 9ý. and yk (we use outputs produced by the last layer of the decoder from a language model, and the outputs are averaged over all sequence to- kens). cos(:;:) computes cosine similarity, and r is temperature. By minimizing LCL. we finetune the model to generate summary representations to maximize the discrepancy between contrastive sets and develop a preference for more faithful outputs.
3.2 Incorporating Medical Knowledge
Our second objective aims to incorporate medi- cal knowledge in the reference to generate more faithful summaries. To this end, we design a loss term LMKi to encourage models to maximize the likelihood of the medical terms in reference by in- creasing their generation probability.
Reference summary failing faithfulness vali- dation. Based on the verification of reference in positive set construction, reference summary with any unmentioned medical terms will be a negative example.
We first identify all medical terms in reference summaries, since this type of term is one of the most important carriers of medical information. For each medical concept, its context including (1) two tokens preceding the medical term and (2) any negative unigram (Ap- pendix H) in the reference. Then, we maintain a vector bn with the same length as the vocabulary size to record the frequency of the context tokens and medical terms. For example, if token of inter- est appears two times in reference summary, its corresponding value in 6m is 2. Let p denote model prediction scores of all vocabularies, i.e., logits of all vocabulary tokens averaged across all reference tokens before the softmax layer. Since bm and p have the same dimension, LMKI is computed as the dot product of 4m and
further consider
Replacing and appending medical concepts. When reference summary contains a medical term showing up in the source, we replace that medical term with a new one. We also add new medical concept the start or the end of the ref- erence summary. These newly introduced terms break the faithfulness of the original reference summary because they are randomly selected and do not appear in the source or the reference.
Changing attribute value. If a reference sum- mary contains numerical attributes, we change the number to another random value, e.g., "5 doses" will be changed "6 doses".
LMKi = bm
Entity swap. To break the relationships among entities in a reference summary, we first apply named entity recognition and swap all recognized entities the original summaries.
In this way, LMKI encourages model maxi- mize the prediction scores of the medical terms and their context words during generation improve summary faithfulness.
Logic inversion. We invert the logic of a refer- ence summary in our Chinese dataset so that its affirmative sentence changes to negative one vice versa. To achieve this, we first select pair of positive and negative unigrams (Appendix H). Whenever we see reference that has the posi- tive or the negative unigram in this pair, we then change it to invert the logic.
3.3 Overall Fine-tuning Objective
As described above, LCL and ΣMKI incorporate medical knowledge to promote faithfulness. Fur- thermore, we add the cross-entropy loss Lee to obtain our final training objective:
L - XCL LCL + AMKI LMKI + LCE (2) where ÀCL and λиKI serve as weights. Note that we encourage all the medical terms in reference summarics due to LMKI while discouraging unmen- tioned medical terms due to Lai.
Given the positive and negative sets, the con- trastive learning loss function LCL. is:
Σ log (hp 1/T) W.JEEP NA
LeL
explcos(he, kW77]

---

Disease Distribution
Summarization Task
Source
Language
Split (# Instances)
Heart
Liver
Brain
Kidney
Respiration
Stomach
Others
Health Question (HQS)
NLM
English
1000/50/100
51
19
37
28
970
Radiology Report (RRS)
Hospital
English 5444/4000600
34461
1069
4277
303
76433
6505
11195
Medical Dialogue (MDS)
Telemedicine
Chinese
1346/288/288
106
162
102
150
124
1322
Table 2: Statistics of three datasets for evaluation. We show the number of train/dev/test examples in the split.
4
Experiment Setup
strict: a summary is faithful or coherent when all three doctors agree.
4.1
Datasets
To demonstrate the applicability of our approach as a general-purpose summarizer, we use three differ- ent datasets in Table 2 including Health Question Summarization (HQS) from the MEDIQA 2021 shared task I (Ben Abacha et al., 2021; Ben Abacha and Demner- Fushma,, 2019), Radiology Report Summarization (RRS) from the MEDIQA 2021 shared task 3 (Johnson et al., 2019a,b; Demner- Fushman et al., 2016), and Medical Dialogue Sum- marization (MDS) (Song et al., 2020).
4.3 Baselines
Five types of baselines considered: Backbone pretrained language model We fine- tune Pegasus-large (Zhang et al., 2020a), BART-large (Lewis et al., 2020), BioBART-large (Yuan et al., 2022), T5-base (Raffel et al., 2019). and mT5-small (Xue et al., 2021) as our baselines. Different variants of FAMESUMM To sort out the factors that affect performance, we present two FAMESUMM variants on HQS including training with contrastive learning only and training with positive sets that accept all references. When using mT5-small on MDS, we also conduct an ablation study to showcase the effectiveness of individual fine-tuning objectives.
4.2 Evaluation Metrics
We adopt holistic evaluation scheme in aspects of faithfulness, general quality, and human evaluation. Faithfulness Metries To measure faithfulness, we report three metrics from existing literature based on question answering, medical entity, or textual entailment: QuestEval (Scialom et al., 2021), FaR (Shing et al., 2021), and SummaC (Laban et al.. 2022). We also use Concept F1 (C F1) to mea- sure the coverage of medical concepts in reference summaries (Joshi et al., 2020).
Faithfulness-based model Since contrastive learn- ing is a major part of FAMESUMM, CLIFF (Cao and Wang. 2021) and QFCL (Zhang et al., 2022) are the closest baselines we find. CLIFF is shown to be effective for faithfulness improvement on news-related datasets while QFCL is built specifi- cally for medical question summarization. QFCL is different from CLIFF in terms of negative set construction method.
General Quality We report ROUGE-1/2/L (Lin, 2004) and SERTScore (Zhang et al., 2019) to assess the general summarization quality.
Human Evaluation by Doctors We conduct a hu- man evaluation by medical experts to judge coher- ence and faithfulness. Six doctors are recruited: three of them evaluate 100 test instances in HQS while the other three are on 100 test instances in MDS. Doctors are asked to identify incoherent and unfaithful summaries produced by candidate models. The order of summaries generated by models is randomized and anonymous to doctors. When all doctors complete their evaluation, they discuss how to resolve major disagreements while still holding their distinctive opinions towards in- dividual examples. We then count the number of faithful and coherent summaries through vote of three doctors. Two types of voting are reported: (1) consensus: a summary is considered faithful or coherent if at least two doctors think so, and (2)
A bbstrcctvve medical summarizer We select Joshi et al. (2020) as the medical dialogue summarizer that fits MDS. Other related work (Krishna et al., 2021; Zhang et al., 2021) target special types of medical dialogue (e.g., extremely long dialogue), which not the focus of this paper.
LLM To make model comparison fair, we do not consider zero-shot summarization from LLMs. We fine-tune GPT-3 (Brown et al., 2020) on 10 random training instances of HQS as a baseline.
4.4 Implementation Details
FAMESUM relies on the identification of medical terms. For English datasets, use Unified Medi- cal Language System (Bodenreider, 2004) iden- tify them. For MDS in Chinese, we recruit human annotators to tag them as described in Appendix B.

---

Faithfulness
General Quality
Model
QuestEval
FaR
SummaC
CFI
R1
R2
RL
BERTScore
PEGASUS (He et al., 2021)
0.3069
0.3188
0,4279
26.24
30.19
11.93
28.52
0.7427
CLIFF (Cao and Wang. 2021)
0.3145
0.3217
0.4225
30.18
29.92
11.40
27.67
0.7401
+ QFCII.Zhang (Zhang et al., 2022)
0.3193
0.3042
0,4322
28.00
29.27
11.01
27.32
0.7396
FAMESUMM (CI. only)
0.3157
0.3222
0.4355
30.20
29.31
10.82
27.39
0.7352
FAMESUMM (All ref in Pjj
0,3105
0.3267
0,4138
28.90
31.14
12.29
29.09
0,7489
FAMESUMM
0.3134
0.3492
0.4401
30.92
30.84
12.19
28.99
0.7456
BART (He et al., 2021)
0.3120
0.3300
0.4062
30.02
31.51
10.57
29.84
0.7512
FAMEESUM
0.3127
0.3233
0,4640
40.42
31.76
11.71
29.64
0.7491
T5 (He et al., 2021)
0.3
0.3237
0.4125
27.38
30.11
11.40
27.54
0.7460
+ +FAMESUMM
0.3186
0.3467
0,4342
32.79
30.19
11.00
27.91
0.7470
BioBART (Yuan et al., 2022)
0.3116
0.3487
0,4810
30.15
32.38
11.91
29.51
0.7486
FAMESUMM
0.3239
0.3595
0.4740
33.33
32.99
12.57
30.56
0.7544
Table 3: Results for HQS. We rerun baselines compute all metrics.
places all references into positive sets.
Faithfulness
General Quality
Model
QuestEval
FaR
SummaC
C F1
R1
R2
RL
BERTScore
ORS-Indiame
PEGASUS (Dai et al., 2021)
0.2413
0.0682
0.2350
46.12
44.35
29.67
43.87
0.7999
+ FAMESUMM
0.2441
0.0741
0.2769
47.37
46.69
33.13
46.15
0.81
RRS- Stanfrtnf
PEGASUS (Dai et al., 2021)
0.2892
0.2146
0.4907
40.74
40.97
26.37
38.45
0.7755
+FAMESUMM
0.2884
0.2324
0.4931
42.86
41.21
26.90
38.86
0.7768
Table 4: Results for RRS. We rerun baselines to compute all metrics. We show splits separately.
Model
FaR
FI
RL
BERTScore
and analysis aim to answer the following research questions:
Joshi et al. (2020)
0.0619
32.53
19.99
0.6937
mT5+
0.3784
36.96
32.43
0.7505
RQ 1: How can FAMESUMM improve faithful-
+CL*
0.4040
37.85
32.73
0.7518
ness of backbone models (5.1)?
*MKI*
0.4072
39.30
32.27
0.7492
RQ 2: What are the effects of the two loss func-
+FAMESUMM
0.4177
38.57
33.05
0.7520
tions in FAMESUMM (5.2 and 5.5)?
Table 5: Results for MDS. For models with *, we report their scores that are averaged over random seeds (seed values 42, 0, and I) due unstable training dynamics of transformer- based models (Mosbach et al.. 2021). Scores based on a single seed (single-run scores) are reported in Appendix
RQ 3: How does FAMESUMM compare with other types of baselines including faithful sum- marization models (5.1), state-of-the-art medical
summarization (5.3), and LLMs (5.4)?
RQ 4: performing faithfulness validation on reference summarics better than accepting all them into the positive sets (5.1)?
More implementation details are in Appendix C. Due to differences on language and dataset struc-
RQ 5: How do doctors judge the faithfulness of medical summarization and its correlation with current faithfulness metrics (5.4)?
ture, we customize the contrastive set construction for each dataset, and the details are in Appendix F and G. Hyperparameters of each model are in Ap- pendix J.
5.1 Overall Results
Our overall results for three datasets are presented in Table 3, 4, and 5.
5 Results and Analysis
All performance scores in this section are evalu- ated on the test set of each dataset. Our results
Improvements over Backbone Models As dis- cussed Section 4.2, the effects learning from

---

Dataset
Model
R1
R2
RL
Consensus
Strict
Model
Fleiss' kappa
Sünger al. (2021)
33.40
15.99
31.49
C
# F
# C
#
HQS
He et al. (2021)
35.14
16.08
31.31
FAMESUMM Ensemble
35.35
14.86
33.06
HQS
Mahajan et al. (2021)
67.72
58.81
66.57
GPT-3
100
60
100
37
0.4675
RRS-Indiana
Dai et al. (2021)
68.34
59.56
67.17
PEGASUS
100
71
100
54
0,4295
FAMESUMM Ensemble
68.30
59.68
67.46
+ CLIFF
100
70
59
0.6508
Mahajan al. (2021)
38.84
22.84
36.11
+ FAMESSUMM
100
73
100
60
0.4976
RRS-Stanford
et al. (2021)
43.12
27.69
40.14
+ FAMESUMM
100
76
100
66
0.4697
FAMESUMM PEGASUS
43.34
Joshi et (2020)
22.81
MDS
MDS
mTS
34.20
19.95
Joshi (2020)
100
22
0.7316
FAMESUMM mT5
34.92
20.56
33.05
mT5
100
82
100
77
0.7479
+ MKI
100
84
100
81
01.946
Table 6: Ensembling results of FAMESUMM. We report ROUGE compared with previous aate-of-thh-art on each benchmark.
+ FAMESUMM
100
89
83
0.6351
Table 7: Human evaluation results. placess all refer- ences into positive sets. "# C": Coherent. "# Faithful. We report Fleiss' kappa faithfulness.
contrastive medical summaries are demonstrated by improved automatic faithfulness measures. The effects of medical knowledge incorporation are mainly demonstrated through C F1. With only three exceptions QuestEval, FaR, and SummaC metrics (minimal decrease), FAMESSUM provides consistent improvements over the corresponding backbone models on all automatic faithfulness met- rics. Therefore, we show that the proposed fine- tuming strategies are effective at learning medical knowledge and generating more faithful outputs. ROUGE and BERTScore are not compromised with FAMESUMM, since most of them are higher when compared against corresponding backbone models.
Spearman's
Model
Dataset
QuestFFval
FaR
SummaC
FAMESUMM +PEGASUS
HQS
0.067
0.171
0.312
Table 8: and three faithfulness metrics.
Correlation analysis between human evaluation
and QFCL based on our observation in Section 2. The advantage of our CL demonstrates the util- ity of designing more diverse and accurate trastive sets to learn decision boundary- In Ta- ble FAMES UMM and all mT55based models are better than Joshi et al. (2020) due the success of pre-trained transformer models.
Advantage of Faithfulness Validation on Ref- erences Comparing the last two rows of PEGASUS bbseed models in Table 3, we see that faithfulness validation on reference summaries en- ables FAMESUMM to dominate all faithfulness metrics. ROUGE and BERTScore are slightly lower after adding this validation, since we disvalue many references by placing them into the negative sets. We include this validation trastive sets, since the decrease trivial and this paper mainly deals with improving faithfulness.
Complementary Roles of CL and MKI formance gap between FAMESUMM and CLIFF (or QFCL) becomes much more sigmificant after we add MKI, which is shown through the last row of PEGASUS-baat models in Table 3. Most faithful- ness metrics are positively impacted when we train with the complete version of FAMESUUM (FaR ex- periences the greatest increase), and general qual- ity metrics of FAMESUMM are also significantly higher than CLIFF and QFCL. Although CL and MKI have different focuses, their complementary roles enable performance boost across a variety of metrics.
The per-
step of building con-
Improvements over Faithful Summarization Baselines Compared with CLIFF or QFCL in Table 3, FAMESUMM (CL only) offers a consistent improvement on all faithfulness metrics despite a small decrease on QuestEval, which demonstrates the advantage of our contrastive learning method. There are two reasons: (1) both CLIFF and QFCL do not enforce any validation of references, and this is a flawed design based on Adams et al. (2022) and the discussion above, and (2) our CL considers much more comprehensive error types than CLIFF
5.2 Ablation Study of Two Loss Functions
As discussed above, both fine-tuning strategics in FAMESUMM contribute the overall faithfulness improvement. In Table 5, we iteratively add fine- tuning strategy to mT5. As a result, FaR score keeps increasing when we add more strategies. The overall quality as measured by RL and BERTScore

---

Source: ...3 iAtRi Wt..eiiiiaaaeeee.. 1a 20 S0
Translation: D. worry! Concussion will devalop Parkimson discuse!...
Learning from contrastive medical
& 7018
X Negation emor
Translation: Concussion will develop into Parkinson's disease later!
summaries
FaMeSumm (8 Rotti 1i101)
Nogative unigram captured
Translation: Concussion will not develop into Parkinson's discnse!
Ref summary:
on tusigna
baby birth defects?
Medical knowledge incorporation
PEGASUS: What the risks of tasigna?
Missing importans medical term
FaMeSamm + PEGASUS Where can I find information on tasigna hirth defects?
+
Genceoling of an important medical term
Figure 2: Two examples (from MDS and HQS datasets) show the effect of FAMESUMM finetuned on mT5 and PEGASUS. We mark the medical terms in blue in the second example.
is also the highest with both strategies. C F1 is slightly lower on the last row of Table 5, indicating that our pipeline is learning a trade-off between our two strategies.
A Although this paper mainly deals with faithful- ness concerns in healthcare, we show that improv- ing faithfulness can also yield better ROUGE scores due to the fact that FAMES output stays con- sistent with the source text.
We can see that MKI is most effective on entity- based faithfulness metrics. It yields the highest and the second highest FaR. Relative to MKI, CL improves faithfulness more comprchensively in a mixture of question answering, entailment, and entity aspects. CL has better RL and BERTScore than MKI, which indicates that improving faithful- ness in comprchensive ascepts has the potential of improving general summarization quality. CL also has better scores than plain mT5.
5.4 Human Evaluation
We report our human evaluation results in Ta- ble 7. FAMESUMM outputs the most faithful sum- maries on both HQS and MDS, which reconfirms its strongest capability of improving faithfulness. Echoing the analysis in Section 5.1, FAMESUMM without faithfulness validation has suboptimal per- formance. Joshi et al. (2020) suffers from incoher- ent summaries and only has faithful summary under the strict standard (doctors will mark a summary as faithful if it is incoherent). We find moderate to substantial inter-annotator agreement for faithfulness judgments. Fleiss' kappas on MDS are higher than those on HQS (except for CLIFF), both of which are above 0.4.
5.3 Ensembling for Comparison with State-of-the-art Medical Summarization
Since the best papers HQS and RRS adopt model ensembling, we perform ensembling over multiple FAMESUMM results compare with those works. Table displays this comparison, which demonstrates that our proposed fine-tuning strategics can yield statc-of-the-art performance. In general, we could achieve better performance than the best paper on each benchmark of MEDIQA with much fewer base models or ad-hoc techniques. For example, as the best paper on HQS, He et al. (2021) utilized ensembling based four models along with training on the validation set and error correction for misspellings. Our ensembled outputs (with FAMESUMM) surpass the best score on R1 and RL without using the validation set or any error correction techniques. More ensembling details are provided in Appendix E. On MDS, we see constant ROUGE increase from Joshi et al. (2020) to FAMESUMM. One reason is due the superi- ority of pre-trained language models over pointer- generator networks in Joshi et al. (2020). Besides that, FAMESUMM yields better scores than plain mT5 because of FAMESUMM' 's capability of gener- ating domain-specific and more faithful summaries.
Compare with GPT-3 GPT-3 only has 37 faith- ful outputs under the strict protocol. FAMESUMM generates 16% more faithful summaries than GPT- based on consensus protocol. Doctors find that GPT-3 generates more extrinsic errors than other models (e.g., extraneous fact error in Table 1). Its linkage to external knowledge allows its outputs to be more diverse, but it also yields more unrelated or even incorrect information.
We show the correlation between human evaluation and the three faithful- ness metrics in Table 8. All correlation scores are small. SummaC has the strongest correlation while QuestEval has very weak correlation. Our find- ings of the low correlation are echoed in Wang et al. (2023) and call for future research efforts on faithfulness metrics for medical summarization. This also reflects our motivation for conducting doctors' evaluations.
Correlation Analysis

---

5.5 Case Study
Faithfulness and Factuality in Abstractive Sum- marization Recent work has proposed several factual evaluation metrics (Fabbri et al., 2021; Scialom et al., 2021; ; Wang et al., 2020; Fabbri et al., 2022; Laban et al., 2022; Shing et al., 2021). Cao and Wang (2021) utilized contrastive learning (Khosla et al., 2020) with a design of contrastive sets in the news domain for faithfulness enhance- ment. Several recent efforts (Goyal et al., 2022; Liu et al., 2022; Tam al., 2022) investigated summarization factuality of L.MMs such as GPT- (Brown et al., 2020) and BLOOM (Scao et al.. 2022), yet they only used news-related datasets. Few papers (Zhang et al., 2020b; Alambo et al.. 2022) that directly modeled faithfulness in health- care (by adding inductive bias informed by faithful- ness through model optimization) focused on single specific tasks such as radiology report summariza- tion. By contrast, we show that it is important design domain -specfi contrastive sets and offer a general purpose medical summarizer.
We present two examples in Figure 2 to show the effectiveness of the two loss functions in FAMESUMM. In the contrastive learning exam- ple, FAMESUMM copied key sentence from the source while capturing an important negative uni- gram ("not"). However, plain mT5 ignored it and thus generated negation error FAMESUMM is more likely capture this kind of negative uni- gram than its baseline because we applied logic inversion when we built the negative sets for the MDS dataset.
In the second example, FAMESUMM success- fully generates an important medical term "birth defects", while the baseline failed. This term makes its summary more specific and consistent with the reference. Incorporating medical knowledge, FAMESUMM is more likely to generate medical concepts than its baselines to improve faithfulness.
-arrpose
6 Related Work
7 Conclusion
Medical Summarization There are different medical summarization datasets on medical di- alogues (Joshi et al., 2020; Zeng et al., 2020; Krishna et al., 2021; Yim and Yetisgen, 2021; Navarro et al., 2022; Zhou al., 2021) and multi- documents YDeuuung et al., 2021; Katsimpras and Paliouras, 2022). Some of them are pri- vate. MEDIQA 2021 shared tasks (Ben Abacha al., 2021) tackle three summarization tasks: con- sumer health question summarization (HQS), multi- answer summarization (MAS), and radiology re- port summarization (RRS).
In this paper, we investigate faithfulness issues on abstractive medical summarization in current sum- marization models. We propose FAMESUMM con- sisting of two fine-tuning objectives that can be applied to mainstream pretrained language mod- els across many datasets. Our results based on automatic metrics and doctor evaluation show that our method mitigates the faithfulness concern medical summarization and delivers state-of-the-ar performances on several benchmarks.
Limitations
As for medical summarization models, Joshi et al. (2020) and Enarvi et al. (2020) constructed reinter-generator networks, they did not ben- efit from the success of pre-trained transformer models. Zhang et al. (2021) fine-tuned BART and developed a multistage approach to handle long conversations. The length of conversations is not the concern of our work. The focus of Krishna et al. (2021) is to generate SOAP notes (Subjective information, Objective observation, Assessments made by doctors, and Plan for future care), and their work aims have summaries divided into most 15 sections. By contrast, FAMESUMM serves as a general-purpose fine-tuning strategy for abstractive summarization in healtheare with wide applicability.
Since FAMESUMM involves learning from con- trastive summaries, it sets relatively high require- ment GPU memory, which is common draw- back of existing contrastive learning framework. For any dataset used for training, adding the con- trastive learning component of FAMESUMM will demand more memory than training without it. As a result, the training batch size of FAMESUMM may be set to small value. For example, fine- tuning Pegasus- large with FAMESUMM on RRS- Indiana takes about 48GB memory using training batch size of 2. Note that FAMESUMM is more efficient than many recent works such Chen al. (2022) in terms of memory consumption and/or other training requirements as described in Sec- tion I, and we never need to go beyond batch size

---

of 4 for all experiments on FAMESUMM. Our code supports multi-GPU training to allow users try larger batch sizes.
Asma Ben Abacha, Yassine Mrabet, Yuhao Zhang, Chai- tanya Shivade, Curtis Langlotz, and Dina Demner- Fushman, 2021. Overview of the MEDIQA 2021 shared task on summarization in the medical domain. In Proceedings of the 20th Workshop on Biomedical Language Processing, pages 74-85, Online. Associa- tion for Computational Linguistics.
Ethics Statement
Improving the faithfulness of abstractive sum- maries is an active research area. Although meth- ods proposed in this work are shown to be effective at reducing unfaithful errors in Section 5, com- pletely eliminating these errors has not yet been reached. Since unfaithful errors may still exist in our model outputs, we emphasize that FAME- SUMM is designed to be used under supervision from at least one medical practitioner. This medi- cal practitioner needs to check the faithfulness of any summaries generated from FAMESUMM be- fore providing them to patients in order to avoid risks. We used detailed statistics as well as a case study in Section 5 to showcase the performance of FAMESUME UMM in terms of faithfulness.
Olivier Bodenreider. 2004, The unified medical lan- guage system (umls): integrating biomedical termi- nology. Nucleic acids research, 32(suppl 1):D267- D270.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901
Shuyang Cao and Lu Wang. 2021. CLIFF: Contrastive
learning for improving faithfulness and factuality in abstractive summarization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6633-6649, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
All medical datasets used in this work were de- identified before we accessed them.
Xiuying Chen, Mingzhe Li, Xin Gao, and Xiangliang Zhang. 2022. Towards improving faithfulness in abstractive summarization.
Acknowledgements
We thank Tianyang Zhao, Xiangyu Dong, Yilun Zhao, Yiyang Feng, Fangxu Yu, Yunxiang Li, Xue- qing Zhang, and Wei Chen for their significant as- sistance on data annotation. This research was par- tially funded by the Federal Ministry of Education and Research (BMBF), Germany under the project LeibnizKILabor with grant No. 01DD20003.
Songtai Dai, Quan Wang, Yajuan Lyu, and Yong Zhu. 2021. BDKG MEDIQA 2021: System report for the radiology report summarization task. Proceed- ings of the 20th Workshop on Biomedical Language Processing, pages 103 111, Online. Association for Computational Linguistics.
Dina Fushman, Marc D Kohli, Marc B Rosen- man, Sonya E Shooshan, Laritza Rodriguez. Sameer Antani, George R Thoma, and Clement McDon- ald. 2016. Preparing a collection of radiology ex- aminations for distribution and retrieval. Journal of the American Medical Informatics Association. 23(2):304-310.
References
Griffin Adams, Han-Chin Shing, Qing Sun, Christo- pher Winestock, Kathleen McKeown, and Noémie Elhadad. 2022. Learning to revise references for faithful summarization. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2022, pages 4009-4027, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Jay De Young, Iz Beltagy, Madeleine van Zuylen, Bailey Kuchl, and Lucy Lu Wang. 2021. MS* Multi- document summarization of medical studies. In Pro- ceedings of the 2021 Conference Empirical Meth- ods in Natural Language Processing, pages 7494- 7513, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Amanuel Alambo, Tanvi Banerjee, Krishnaprasad Thirunarayan, and Mia Cajita. 2022. Improving the factual accuracy of abstractive clinical text summa- rization using multi-objective optimization. arXiv e-prints, pages Xiv--2204
Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming- Wei Chang, Dipanjan Das, and William Cohen. 2019. Handling divergent reference texts when evaluating table-to-ter generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4884-4895, Florence, Italy. Asso- ciation for Computational Linguistics.
Asma Ben Abacha and Dina Demner-Fushmant 2019. On the summarization of consumer health questions. In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics, pages 2228- 2234, Florence, Italy, Association for Computational L Linguistics.
Seppo Enarvi, Marilisa Amoia, Miguel Del-Agua Teba, Brian Delaney, Frank Diehl, Stefan Hahn, Kristina

---

Harris, Liam McGrath, Yue Pan, Joel Pinto, Luca Ru- bini, Miguel Ruiz, Gagandeep Singh, Fabian Stem- mer, Weiyi Sun, Paul Vozila, Thomas Lin, and Ran- jani Ramamurthy, 2020. Generating medical reports from patient-doctor conversations using sequence-to- sequence models. In Proceedings of the First Work- shop on Natural Language Processing for Medical Conversations, pages 22-30, Online. Association for Computational Linguistics.
Neel Kanwal and Giuseppe Rizzo. 2022. Attention- based clinical note summarization. In Proceedings of the 37th ACM/SIGAPP Symposium Applied Computing, SAC '22, page 813-820, New York, NY, USA. Association for Computing Machinery.
Georgios Katsimpras and Georgios Paliouras. 2022. Pre- dicting intervention approval in clinical trials through multi nouti-document summarization. In Proceedings of the 60th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 1947-1957, Dublin, Ireland, Association for Computational Linguistics.
Alexander Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. QAFactEval: Improved QA- based factual consistency evaluation for summariza- tion. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, pages 2587-2601, Seattle, United States. Asso- ciation for Computational Linguistics.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Su- pervised contrastive learning. In Advances in Neural Information Processing Systems, volume 33, pages 18661-18673. Curran Associates, Inc.
Alexander R. Fabbri, Wojeiech Kryšciński, Bryan Me- Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating Summariza- tion Evaluation. Transactions of the Association for Computational Linguistics, 9:391-409.
Kundan Krishna, Sopan Khosla, Jeffrey Bigham, and Zachary C. Lipton. 2021. Generating SOAP notes from doctor-patient conversations using modular summarization techniques. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing ( (Vol- ume 1: Long Papers), pages 4958 4972, Online. As- sociation for Computational Linguistics.
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. News summarization and evaluation in the era of gpt-3, arXiv preprint arXiv:2209. 12356.
Shiyi Han, Yuhui Zhang, Yunshan Ma, Cunchao Tu, Zhipeng Guo, Zhiyuan Liu, and Maosong Sun. 2016. Thuocl: Tsinghua open chinese lexicon. Tsinghua University.
Kundan Krishna, Amy Pavel, Benjamin Schloss, Jef- frey P. Bigham, and Zachary C. Lipton. 2020. Ex- tracting structured data from physician-patient con- versations by predicting noteworthy utterances.
Yifan He, Mosha Chen, and Songfang Huang. 2021. damo_nip at MEDIQA 2021: Knowledge-hased pre- processing and coverage-oriente reranking for med- Proceedings of the 20th Workshop on Biomedical Language Processing, pages 112-118 Online. Association for Computa- tional Linguistics.
ical question summarization.
Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2022. SummaC: Re-visiting NLI- based models for inconsistency detection in summa- rization. Transactions of the Association for Compu- tational Linguistics. 10:163-177.
Yichong Huang, Xiachong Feng, Xiaocheng Feng, and Bing Qin. 2021. The factual inconsistency problem abstractive text summarization: survey. arXiv preprint 2rXiv:2104.14833
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequenee for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguisties. pages 7871-7880 Online. Association for Computa- tional Linguistics.
pre-training
AEW Johnson, TJ Pollard, SJ Berkowitz, R Mark, and S Homg. 2019a. Mimic-exr database (version 2.0. 0). physionet.
Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih- ying Deng, Roger G Mark, and Steven Horng. 2019b. Mimic-exr. a de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 661):1-8.
Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- fion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Anirudh Joshi, Namit Katariya, Xavier Amatriain, and Anitha Kannan. 2020. Dr. summarize: Global sum- marization of medical dialogue by exploiting local structures. In Findings of the Association for Com- putational Linguistics: EMNLP 2020, pages 3755- 3763, Online. Association for Computational Lin- guistics.
Yixin Liu, Alexander R Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, et al. 2022. Re- visiting the gold standard: Grounding summariza- tion evaluation with robust human evaluation. arXiv preprint 221.v22112.099.

---

Zhengyuan Liu, Hazel Lim, Nur Farah Ain Suhaimi. Shao Chuen Tong, Sharon Ong, Angela Ng. Shel- don Lee, Michael R. Macdonald, Savitha Ramasamy, Pavitra Krishnaswamy, Wai Leng Chow, and Nancy F. Chen. 2019. Fast prototyping a dialogue compre- hension system for ourse-patiens conversations on symptom monitoring. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Industry Papers), pages 24-31, Minneapolis, Minnesota. Association for Computational Linguistics.
factuality metries. Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 4812-4829, Online. As- sociation for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts. Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-test trans- former. CoRR, abs/1910.
Mario Sänger, Leon Weher, and Ulf Leser. 2021. WBI at MEDIQA 2021: Summarizing consumer health questions with generative transformers. In Proceed- ings of the 20th Workshop on Biomedical Language Processing, pages 86-95, Online. Association for Computational Linguistics.
Diwakar Mahajan, Ching- Tsou, and Jennifer J Liang. 2021. IBMResearch at MEDIQA 2021: To- ward improving factual correctness of radiology re- port abstractive summarization. In Proceedings of the 20th Workshop on Biomedical Language Processing, pages 302-310, Online. Association for Computa- tional Linguistics.
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: 176b- parameter open-access multilingual language model. arXiv preprint arXiv:221 1.05100.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factu- ality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919, On- line. Association for Computational Linguistics.
Thomas Seialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021. QuestEval: Summariza- tion asks for fact-based evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 6594-6604, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
George Michalopoulos, Kyle Williams, Gagandeep Singh, and Thomas Lin. 2022. MedicalSum: A guided clinical abstractive summarization model for generating medical reports from patient-doctor con- versations. In Findings of the Association for Com- putational Linguistics: EMNLP 2022, pages 4741- 4749, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Han-Chin Shing, Chaitanya Shivade, Nima Pour- damghani, Feng Nan, Philip Resnik, Douglas W. Oard, and Parminder Bhatia. 2021. Towards clin- ical encounter summarization: Learning to com- pose discharge summaries from prior notes. CORR, abs/2 101.104...00 13498.
Marius Mosbach, Maksym Andriushchenko, and Diet- rich Klakow. 2021. On the stability of fine-tuning bert: Misconceptions, explanations, and strong base- lines.
Yan Song, Yuanhe Tian, Nan Wang, and Fei Xia. 2020. Summarizing medical conversations via identifying important utterances. In Proceedings of the 28th International Conference on Computational Linguis- rics, pages 717-729, Barcelona, Spain (Online). In- ternational Committee on Computational Linguisties.
David Fraile Navarro, Mark Dras, and Shlomo Berkovsky. 2022. Few-shot fine-tuning SOTA sum- marization models for medical dialogues. In Proceed- ings of the 2022 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies: Student Research Workshop, pages 254-266, Hybrid: Seattle, Washington + Online. Association for Computational Linguistics.
Derek Tam, Anisha Mascarenhas, Shiyue Zhang. Sarah Kwan, Mohit Bansal, and Colin Raffel. 2022. Eval- uating the factual consistency of large language models through summarization. arXiv preprint 2:Xi::2211.08111
Yulia Otmakhova, Karin Verspoor, Timothy Baldwin, and Jey Han Lau. 2022. The patient is more dead than alive: exploring the current state of the multi- document summarisation of the biomedical literature. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( VVolume
Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the fac- tual consistency of summaries, In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 5008-5020, Online. Asso- ciation for Computational Linguistics.
1: Long Papers), pages 5098-5111, Dublin, Ireland. Association for Computational Linguisties.
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality abstrac- tive summarization with FRANK: A benchmark for
Lucy Lu Wang, Yulia Otmakhova, Jay De Thinh Hung Truong, Bailey E. Kuehl, Erin Bransom, and Byron C. Wallace. 2023. Automated metrics

---

for medical multi-document summarization disagree with human evaluations.
Yuhao Zhang, Derek Merek, Emily Tsai, Christopher D. Manning, and Curtis Langlotz. 2020b. Optimizing the factual correctness of summary: A study of summarizing radiology reports. In Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguistics, pages 5108-5120, Online. As- sociation for Computational Linguisties.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale. Rami AI-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap-
ter of the Association for Computational Linguisties: Human Language Technologies, pages 483-498, On- line. Association for Computational Linguistics.
Meng Zhou, Zechen Li, Bowen Tan, Guangtao Zeng- Wenmian Yang. Xuehai He, Zeqian Ju, Subrato Chakravorty, Shu Chen, Xingyi Yang. Yichen Zhang, Qingyang Wu, Zhou Yu, Kun Xu, Eric Xing, and Pengtao Xie. 2021. On the generation of medical dialogs for COVID-19. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 1 11th International Joint Confer- ence on Natural Language Processing (Volume 2: Short Papers), pages 886-896, Online. Association for Computational Linguistics.
Wen-wai wai Yim and Meliha Yetisgen. 2021. Towards automating medical scribing Clinic visit Dia- logue2Note sentence alignment and snippet summa- rization. In Proceedings of the Second Workshop on Natural Language Processing for Medical Conversa- rions, pages 10-20, Online. Association for Compu- tational Linguistics.
Hongyi Yuan, Zheng Yuan, Ruyi Gan, Jiaxing Zhang, Yutao Xie, and Sheng Yu. 2022. BioBART: Pretrain- ing and evaluation of a biomedical generative lan- guage model. Proceedings of the s Workshop on Biomedical Language Processing, pages 97-109, Dublin, Ireland. Association for Computational Lin- guistics.
A Translation of Chinese Examples for The Taxonomy of Faithfulness Errors
In Table 9, we provide English translation of the Chinese texts in Table 1.
B Details of Constructing Medical Dialogue Summarization Dataset
Guangtao Zeng, Wenmian Yang. Zeqian Ju, Yue Yang. Sicheng Wang, Ruisi Zhang, Meng Zhou, Jiaqi Zeng, Xiangyu Dong, Ruoyu Zhang, Hongchao Fang. Penghui Zhu, Shu Chen, and Pengtao Xie. 2020. MedDialog: Large-scale medical dialogue datasets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9241-9250, Online. Association for Computa- tional Linguistics.
We follow Song et al. (2020) construct our med- ical dialogue summarization dataset. During web crawling, we find that the URLs of all the dia- logues on Chunyu- Doctor start with a common part?. Most of the dialogues have summaries (lype- A summaries) that are concatenations of original utterances from the doctors, and only a small group of them contains summaries (type-B summaries) written by the doctors that may have novel words. We set the type-B summary as the reference sum- mary and only collect the dialogues that contain type-B summaries. A type-B summary always preceded by an identifier called "based on this inquiry, the doctor updates the summary and sug- gestion:".
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe- ter Liu. 2020a. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In In- ternational Conference on Machine Learning, pages 11328-11333 11339. PMLR.
Longxiang Zhang. Renato Negrinho, Arindam Ghosh, Vasudevan Jagannathan, Hamid Reza Hassanzadeh, Thomas Schaaf, and Matthew R. Gormley. 2021. Leveraging pretrained models for automatic summa- rization doctor- pettor-patient conversations. In Find- ings of the Association for Computational Linguis- rics: EMNLP 2021, pages 3693-3712, Punta Cana. Dominican Republic. Association for Computational Linguistics.
We look for URLs that start with the common part described above, and we only download their HTML pages if their source dialogues contain the identifier of the type-B summary. Due to constant changes of the URL of each dialogue on the plat- form, we download its source HTML immediately after we find it. Then, we scrape dialogue and sum- mary from every HTML document collected and discard the dialogues that have images or audio inputs. We also throw away those that contain only one utterance, because they are technically not con- versations between two characters. Finally, some
Ming Zhang. Shuai Dou, Ziyang Wang, and Yunfang Wu. 2022. Focus-driven contrastive learning for med- ical question summarization. In Proceedings of the 29th International Conference on Computational Lin- guistics, pages 6176-6186, Gyeongju, Republic of Korea, International Committee on Computational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. Bertscore: Evalu- ating text generation with bert.
Chttps://wwwr bhunyuyisheng..com/c//////////

---

Category
Example (English Translation)
Source: Doctor: ...to test for mycoplasma antibodies and treat with erythromycin if the result is positive.
Entity Relationship
Prediction: receive therapy. you can get shecked for erythromycin at the hospital...
Source: Patient: am a 1--year-old male, with a height of 176cm and weight 61.3kg...
Entity
Prediction: The patient with height of 1717cm and weight of 61.3kg...
Source: Patient: Can pneumoconiosis detected during its latency period? (Male, years old)
Negation
Doctor: Hello. It cannot detected.
Prediction: Pneumoconinsis can be detected during its latency period.
Source: Patient: Does it mean that if it's not well controlled, it can also lead cerebral palsy in the baby? If it's not well controlled, it can also lead to cerebral palsy. the baby.
Question
Prediction:
Table 9: English translation of Chinese examples in Table 1. Words marked in waves are the errors and those marked in underlines the evidence from the source to infer the errors.
patients ask follow-up questions after conversations are summarized. We truncate these conversations to ensure they only contain the parts before the follow-up. Following these steps, we obtain 2251 dialogues with reference summaries. Our collected data anonymous.
specific template to its source conversation?
2. Are there any medical terms found in the sum- mary but not in the dialogue?
3. Are there any medical terms found in both summary and dialogue?
After an exploratory analysis, we notice 2 things to be completed for data cleaning. The first thing is about the specificity of the reference summary. Some reference summaries written by doctors are not directly related to their source dialogue, so we treat them nonspecific templates. For ample, in conversation about heart disease, summary like "please follow my account to ask further questions" are not directly related to the conversation. We need to filter out these nonspe- cific summaries before training our model. Another thing is to identify the medical concepts in the ref- erence summarics, because utilize these terms for fine-tuning FAMESUMM. Any terms that refer disease names, treatment methods, drug names, and biomedical vocabularies are considered as med- ical concepts in this dataset. To maximize medical term coverage, we recruit annotators who are na- tive speakers of Chinese to manually tag medical terms, because existing glossarics of Chinese medi- cal terms do not fully cover those in our dataset. For instance, although Unified Medical Language Sys- tem (Bodenreider, 2004) and THU Open Chinese Lexicon (Han et al., 2016) own large collections of Chinese medical concepts, many drug names from our dialogues such as "Yinaoning" (a Chinese patent drug for the growth of brain) are not found in this collection. Annotators are asked to evalu- ate the specificity of each reference summary and manually tag medical terms. Every dialogue comes with 3 annotation questions:
After the completion of annotation, we discard the dialogues that have nonspecific templates summaries and consider the rest our dataset for dialogue summarization. Every dialogue this dataset comes with two sets of medical terms that correspond to annotation questions and We apply 70/15/15 random split on the collected data to construct our dialogue summarization dataset.
C More Implementation Details
FAMESUMM relies on medical terms, and the def- inition of medical terms or concepts depends on the different ways we use to identify them. For example, if the tool we use to tag them is Unified Medical Language System (Bodenreider, 2004), any named entity that has a CUI (Concept Unique Identifier) or is an alias of another CUl-term is con- sidered as medical term. We also recruited human annotators for the Chinese dialogue to do medical term identification due the lack of promising tools. We do not align the difference of this defi- nition, because our contributions are based on pro- vided medical terms.
We perform grid search on the validation set of each dataset for hyperparameters tuning. For RRS (both Indiana and Stanford) and Chinese dialogue, tune hyperparameters
their validation sets based ROUGE scores. Because the validation data of RRS is a combined set that comes from dif- ferent sources (the data distributions of training and combined validation sets are similar but dif- ferent), we train models of both RRS benchmarks
1. Does the reference summary serve as a non-

---

using the same training set but different valida- tion set. Specifically, we use "validation Indiana" for the Indiana benchmark and use the combined validation set of RRS ("validation MIMIC" and "validation Indiana") for Stanford, since Stanford does not come with its own validation data and we hope a greater data variety would help in this case. Hyperparameter tuning is more challenging on HQS, since its training, development, and test sets are all different in terms of summarization style (He et al., 2021). Thus, we compute: (1) Rouge-2 between model output and source text of the vali- dation set, and (2) string length of the source text in validation divided by the string length of the out- put. The weights of these two values are 1 and 0.5 respectively. We use their weighted sum to tune hyperparameters for training and decoding.
outputs. In contrast, we rerank the generated sum- maries from FAMeSUUM (fine-tuned based on PEGASUS with different hyperparameters) outputs based on the observations from us and Dai et al. (2021).
On RRS-Stanford, the previous late-offthhe-ar (Dai al., 2021) fine-tuned 16 PEGASUS (with different seeds) on the training set with output nor- malization. We reach better performance by just using one FAMESUMM output. This FAMESUMM model is the same model as the last row of Table 4, except that length penalty parameter is additionally tuned on the validation set in order to encourage model to generate longer output.
F Customization of Positive Sets for Each Dataset
More Example Outputs from FAMESUMM and Baselines
For HQS dataset, we apply rule (1), (2), and (4) for sentence extraction. If the reference summary of training instance is placed into the positive set, either (1) (2) will be used in order get one more positive example. Otherwise, (1) or (2) will be used to get the first positive example, and (4) will used to augment it to get the second. For the experiment that places all references into the positive sets, only (4) is utilized to augment a reference summary to obtain the second positive example.
In Figure 3, we show two more examples of faith- fulness improvement brought by FAMESUMM in addition to the case study in Section 5.5. For the two examples in HQS, we copy the source text (a health question posted by patient) and the outputs from CLIFF, QFCL, GPT-3, and FAMESUMM. In the first example, both CLIFF and QFCL misinter- pret the source text and make faithfulness errors. In the second example, GPT-3 makes an extrancous fact error by mentioning "a skin infection caused by staphylococcus bacteria" that is never asked in the source. FAMESUMM shows strong capabilities of protecting information integrity in both examples.
For RRS dataset, we use the same way to build positive sets as HQS.
For MDS dataset, we apply rule (1), (2), and (3) for sentence extraction. All three rules exccuted to ensure end up with at least two positive ex- amples for each training instance. Rule (4) not used, since we do not find comparable machine translation tool for Chinese.
E Model Ensembling for HQS and RRS
We provide more ensembling details here. On HQS benchmark, following the previous state-of-the-art (He et al., 2021), we rerank four system outputs (FAMESUMM fine-tuned with BioBART, T5, PE- GASUS, and BART) based on three features: fi- delity, consensus, and wellformedness. Computing a weighted sum of the three features, we pick the best summary from the four outputs for every test instance. Since each of our single systems has better performance as shown in Table 3, our ensem- bling result achieves the new state-of-the-art.
G Customization of Negative Sets for Each Dataset
For HQS dataset, ulation rules except logic inversion, since we find that negation error is more popular in our Chinese MDS dataset. For the experiment that places all references into the positive sets, no reference sum- maries will be placed into the negative set, so "refer- ence summary failing faithfulness validation" will also not be used.
all the sentence manip-
On RRS-Indiana, the previous state-of-the-art (Dai et al., 2021) fine-tuned 16 PEGASUS models (with different seeds) on the union of training and validation data and performed ensembling based on mutual similarity scores among any two system
Since RRS dataset the largest one we use, we choose not to use "changing attribute value", entity swap, and logic inversion to reduce training cost.

---

covid-13 how long to quarantine ufter being positive how long are you contagious if tested positive for covid-19, 19, how long before i can safely return to work afler a positive covid 19 test
Source:
CLIFF PEGASUS: What the incubation period for covid-197
Example
QFCL PEGASUS: What are the treatments for covid-19?
GPT-3: How long are you contagious if tested positive for Covid-19
FaMeSamm PEGASUS: How long before you can safely return to work after positive covid 19 test?
Source: outbreak red raise on back shoulder arms from shoulder wrist with raised bulbs of liquid. treatment and maintenance
CLIFF PEGASUS: What the treatments for raised bulbs of liquid?
Example
QFCI. + PEGASUS: What are the treatments for raised bulbs of liquid?
GPT-3: What are the treatments for skin infection caused by staphylococcus bacteria
FaMeSumm PEGASUS: What are the treatments for raised bulbs of liquid?
Figure 3: Two more examples of improvement brought by FAME:SUMM on HQS. We mark the faithfulness emros in red.
Model
FaR
C
RL
BERTScore
of models based on mT5. Therefore, in Table 5, these models are finc-tuned three times with seed values 42, O, and 1. Note that all the models on the held-out validation set present the same trend as in Table 5 based on single run (seed value 42), including Joshi et al. (2020). We report all the single-run scores the test set of MDS in Table 10. is clear see that the trend the same in Table 5 (e.g.. MKI has the largest F1, and FAMESUMM has the best scores on other metrics), which reinforces our analysis.
Joshi et al. (2020)
0.0619
32.53
19.99
0.6937
mTT
0.3973
37.26
32.80
0.7512
+CL
0.4133
37.50
32.99
0.7534
+MKI
0.4041
40.00
32.26
0.7497
+FAMESUMM
0.4264
38.24
33.28
0.7540
Table 10: Results for MDS based on single seed.
For MDS dataset, we use the sentence manip- ulation rules except entity swap.
Positive and Negative Unigrams Used by Each Dataset
Hyperparameters
As a step towards building negative sets for our Chinese dataset (MDS), we select "#J [ and "7 DJ as the pair of positive and negative unigrams to invert the logic of reference summaries. Their English translations are "can" and "cannot" respec- tively.
PEGASUS HQS. The key hyperparameters are learning rate, warmup steps, and training batch size. Their values are 0.00005, 700, and 8 respectively.
CLIFF PEGASUS on HQS. The key hyperpa- rameters are learning rate, ÀCL warmup steps, and training batch size. Their values are 0.00003, 1.0, 600, and respectively.
As step towards incorporating medical knowl- edge for our English datasets (HQS and RRS), we select "no", "nopee, "doesn't", "don't", and "not" as the negative unigrams to model medical con- cepts. As for our Chinese dataset (MDS), we select "7 """" "JE", "Y"", and "#E=" as the negative unigrams. All these Chinese negative unigrams mean "no" in English.
QFCL + PEGASUS on HQS. The key hyperpa- rameters are learning rate, ACL warmup steps, and training batch size. Their values are 0.000025, 1.0. 600, and 4 respectively.
I Single-run Scores
MAMMMMM (CL only) PEGASUS on HQS.
Due to relatively unstable training dynamics, we do not want to report "lucky" or "unlucky" scores
The key hyperparameters are learning rate, ACL. warmup steps, and training batch size. Their values

---

are 0.00003, 1.0, 600, and respectively.
Their values are 0.00006, 2.0, 0.0014, and 2 respectively.
FAMESUMM (All ref in P) + PEGASUS on HQS. The key hyperparameters are learning rate, ACL, λMKI, warmup steps, and training batch size. Their values are 0.00004, 1.0, 0.001, 600, and 4 respectively.
PEGASUS on RRS-Stanford. The key hyper- parameters are learning rate, warmup steps, and training batch size. Their values are 0.00006, 1000. and 1 respectively.
FAMESUMM + PEGASUS on HQS. The key hyperparameters are learning rate. ÀCL, AMKI- warmup steps, and training batch size. Their values are 0.00003, 1.0, 0.001, 600, and respectively.
FAMESUMM PEGASUS RRS-Stanford. The key hyperparameters are learning rate, ÄCL. λMKI, warmup steps, and training batch size. Their values are 0.00004, 0.8, 0.0014, 600, and respectively.
BART on HQS. The key hyperparameters are learning rate, warmup steps, and training batch size. Their values 0.000045, 1000, and 2 respectively.
Joshi et al. (2020) on MDS. The key hyperpa- rameters are learning rate, Am (for magnitude of medical concept modeling), A.. (for magnitude of negation modeling). ō (for controlling the strength of generation probability). dimension of hidden states, and dimension of input embeddings. Their values are 0.15, 1.0, 0.1, 1.0, 256, 28 respectively.
FAMESUMM + BART on HQS. The key hyper- parameters are learning rate, λCL. λMKI, warmup steps, and training batch size. Their values are 0.00003, 1.0, 0.0011, 1000, and 2 respectively.
mT5 on MDS. The key hyperparameters are learning rate, warmup steps, and training batch size. Their values are 0.00055, 1000, and 4 respectively.
T5 on HQS. The key hyperparameters are learning rate, warmup steps, and training batch size. Their values are 0.0004, 1700, and 2 respectively.
CL mT5 on MDS. The key hyperparameters are lcarning rate, ÅCL warmup steps, and training batch size. Their values are 0.0005, 1.0, 1200, and 4 respectively.
FAMESUMM + T5 on HQS. The key hyperparam- eters are learning rate, Act. AMKI, warmup steps, and training batch size. Their values are 0.00055, 1.0, 0.0013, 1500, and 2 respectively.
MKI mT5 on MDS. The key hyperparameters learning rate, AMKL, warmup steps, and training batch size. Their values are 0.0005, 0.0014, 1200, and 4 respectively.
BioBART on HQS. The key hyperparameters are learning rate, warmup steps, and training batch size. Their values are 0.00008, O, and 2 respectively.
FAMESUMM + mTS on MDS. The key hyper- parameters are learning rate, ÀCL- ΛMKI- warmup steps, and training batch size. Their values are 0.0005, 1.0, 0.001, 1000, and 4 respectively.
FAMESUMM BioBART on HQS. The key hyperparameters are learning rate, >CL. AMKI- warmup steps, and training batch size. Their values are 0.00003, 0.95, 0.0011, 1000, and 2 respectively.
PEGASUS on RRS-Indiana. The key hyper- parameters are learning rate, warmup steps, and training batch size. Their values are 0.00006, 0. and 4 respectively.
FA FAMESUMM + PEGASUS on RRS-Indiana. The key hyperparameters are learning rate, XCL. AMKL- warmup steps, and training batch size.


Reviews:
[{"review_id": "KfZppiMqyK", "reviewer": "Reviewer_BQfn", "summary": "", "questions": "A. Is L_CL equivalent to CLIFF with a different selection strategy for positive/negative examples?\n\nB. For table 5, what is the relative performance of CLIFF? What about using only your L_CL?\n\nC. You describe FaMeSumm as \"cost-efficient\" (line 105) because only one model must be trained, but also state that it requires 48 GB of memory to finetune PEGASUS with batch size 2 (in Limitations). It's not clear to me that training 1 big model with contrastive learning is more efficient than prior approaches (finetuning without contrastive learning and training a second, smaller model as a faithfulness detector). Can you quantify the difference in cost (memory or GPU-hours) between your method and prior work? \n\nD. Your results in Table 6 are generally very small increases in ROUGE. Are these results from single runs or averaged over multiple runs? And did you perform statistical testing?\n\nE. The validation of reference summaries removes summaries with medical terms not mentioned in the input, but the datasets include health question summarization and dialogue summarization. The patients posing questions/speaking to the doctor may not use the correct medical terms, but that does not make the reference that uses those terms unfaithful. Did you notice any decrease in performance over test set examples that require the model to generate medical terms not found in the input? If so, do you consider this to be a positive or negative impact of the method?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful feedback!\n\nWeakness1\n\n  To sort out the factors that drive performance increase, we report new experiment results by training FaMeSumm CL without MKI on both HQS and MDS datasets, and the performance is reported in the first two tables below (the first table is about HQS, and the second one is about MDS). Combining with the scores reported in the original paper, on HQS, we see that FaMeSumm CL has higher scores than CLIFF on all faithfulness metrics, which demonstrates the advantage of our proposed CL method. Its ROUGE and BERTScore are lower than CLIFF, because we disvalue many reference summaries by placing them into the negative sets. Paying the price of this trivial performance decrease, we obtain a consistent increase on faithfulness measures. Since FaMeSumm CL is not good at improving entity-based faithfulness, we see a marginal increase from FaMeSumm CL over CLIFF on FaR and C F1. We add MKI here to gain a more significant increase. In conclusion, FaMeSumm CL is better than CLIFF.\n\n  Analyzing the second table, we aim to analyze the complementary roles of CL and MKI. We can see that FaMeSumm CL has better RL and BERTScore than MKI only, which shows that improving faithfulness from question answering and entailment aspects has the potential of improving general summarization quality. It also has better scores than plain mT5.\n\n  Contrary to FaMeSumm CL, MKI is most effective on entity-based faithfulness metrics as shown in the second table. It yields the highest C F1 and the second highest FaR. We can see that MKI has the strongest impact on C F1 and FaR. \n\n\n|                                              | QuestEval  | FaR        | SummaC     | C F1      | R1        | R2        | RL        | BERTScore  |\n|----------------------------------------------|------------|------------|------------|-----------|-----------|-----------|-----------|------------|\n| PEGASUS                                      | 0.3069     | 0.3188     | 0.4279     | 26.24     | 30.19     | 11.93     | 28.52     | 0.7427     |\n| PEGASUS + CLIFF                              | 0.3145     | 0.3217     | 0.4225     | 30.18     | 29.92     | 11.40     | 27.67     | 0.7401     |\n| PEGASUS + FaMeSumm                           | 0.3134     | **0.3492** | **0.4401** | **30.92** | **30.84** | **12.19** | **28.99** | **0.7456** |\n| PEGASUS + Zhang et al. 2022 (**new experiment**) | **0.3193** | 0.3042     | 0.4322     | 28.00     | 29.27     | 11.01     | 27.32     | 0.7396     |\n| FaMeSumm CL only (**new experiment**)            | 0.3157     | 0.3222     | 0.4355     | 30.20     | 29.31     | 10.82     | 27.39     | 0.7352     |\n\n\n\n\n|                                    | FaR        | C F1      | RL        | BERTScore  |\n|------------------------------------|------------|-----------|-----------|------------|\n| mT5*                               | 0.3784     | 36.96     | 32.43     | 0.7505     |\n| FaMeSumm CL only* (**new experiment**) | 0.4040     | 37.85     | 32.73     | 0.7518     |\n| +MKI*                              | 0.4072     | **39.30** | 32.27     | 0.7492     |\n| +FaMeSumm*                         | **0.4177** | 38.57     | **33.05** | **0.7520** |\n\n\nWeakness 2: We appreciate this insight! In the first table, we report the performance of Zhang et al. 2022. This paper claims that it is different from CLIFF in terms of negative sample construction method, so it shares the same positive sets as CLIFF. It indeed leads CLIFF on QuestEval and SummaC, which means that Zhang et al. 2022 is a stronger baseline in terms of question answering and entailment aspects of faithfulness. The reason behind is likely due to its manipulation on question focus for the medical question summarization task. Our FaMeSumm CL still outperforms Zhang et al. 2022 on most metrics. Except QuestEval, combining MKI with FaMeSumm CL provides us a greater performance boost. Note that FaMeSumm is designed to handle various summarization tasks in healthcare while Zhang et al. 2022 has a narrower scope and only conducted experiments on question summarization.\n\nWeakness 3: We apologize for the misunderstanding! We did not mean to ignore the ample amount of prior work. In line 49, we meant to say that not many works **“investigated”** faithfulness by providing systematic analysis and categorization of faithfulness errors on medical domain as what we did in Section 2. In line 552, we meant to say that not all works **“directly modeled”** faithfulness by adding inductive bias informed by faithfulness through model optimization, as many papers improve faithfulness through modifying training data or other effective means. We are more interested in those papers that directly modeled faithfulness during training, since they are the direct competitor of our work. We will clarify these points about prior work in our final version.\n\nWeakness 4: Thanks for pointing this out! Yes, the taxonomy proposed in Pagnoni et al. 2021 is similar to ours. We both have some same error types such as entity error. While Pagnoni et al. 2021 is on general domain, we propose new error types that are specific to the medical domain. For example, low specificity error in our taxonomy is particularly important for the medical domain, since ignoring certain specifiers can be detrimental in healthcare. We recommend combining all kinds of taxonomies in order to comprehensively investigate unfaithful errors.\n\nQ (A): Yes. We provided a direct comparison between these two as discussed above.\n\nQ (B): It is hard to accurately implement CLIFF contrastive sets on MDS because it is a Chinese dataset. For example, CLIFF utilized machine translation for positive set construction (same technique as the one in line 215), but we did not find a comparable translation tool that works at the similar level of performance on Chinese data. Note that we do not need to apply machine translation on MDS as a part of FaMeSumm, since we could already meet the minimum size requirement of the positive set by following the prior data augmentation techniques discussed in lines from 201 to 217. As for using only our L_CL, its performance is discussed above for weakness 1.\n\nQ (C): Thanks for your question! We would like to clarify that the 48GB requirement mentioned in the Limitations section applied only when we used PEGASUS-large or BART-large. For all of our experiments reported in this paper that use PEGASUS-large or BART-large, we never needed to go beyond batch size of 4, so the most memory we possibly needed was 48*2=96GB. However, in Chen et al. 2022, they claimed that 4 NVIDIA A100 GPUs are needed for their experiments based on BART-large and pegasus-xsum. Although we are not sure whether each of their A100 has 40GB or 80GB, even the most optimistic estimate for them (or the most pessimistic estimate for us) is that FaMeSumm would save at least 40*4-96=64GB of GPU memory.\n\n\nQ (D): Thanks for your question! On HQS benchmark, following the previous state-of-the-art (He et al., 2021), we rerank 4 system outputs (FaMeSumm fine-tuned with BioBART, T5, PEGASUS, and BART) based on four features: fidelity, length, consensus, and wellformedness. Computing a weighted sum of the four features, we pick the best summary from the 4 outputs for every test instance. Note that the previous state-of-the-art trained 4 systems on the union of training and validation data and utilized some ad-hoc techniques such as misspelling correction (mentioned in line 445 and 448), but **we do not need those to reach better performance**. The reason is that each of our single systems has better performance as shown in Table 3.\n\nOn RRS-Indiana, the previous state-of-the-art (Dai et al., 2021) fine-tuned **16** PEGASUS (with different seeds) on the union of training and validation data and performed ensembling based on mutual similarity scores among any two system outputs. In contrast, we leverage the observations conducted on the training set in that paper and rerank system output out of **7** FaMeSumm (fine-tuned based on PEGASUS with different seeds) outputs.\n\nOn RRS-Stanford, the previous state-of-the-art (Dai et al., 2021) fine-tuned **16** PEGASUS (with different seeds) on the training set with output normalization (this normalization is based on an observation mentioned in the previous paragraph). We reach better performance by **just using one** FaMeSumm (fine-tuned based on PEGASUS with different seeds) output. This FaMeSumm model is the same model as the last row of Table 4, except that length penalty parameter is additionally tuned on the validation set in order to encourage our model to generate longer output.\n\nWe do not aim to make a huge improvement on ROUGE in Table 6. Instead, we achieved better scores with fewer base models and ad-hoc techniques, since each of our base models (presented in Table 3 and 4) has better performance than its baseline. We will describe these implementations with more details in our final version and share our ensembling code for each dataset.\n\nQ (E): We appreciate this insight! As mentioned in our footnote, our decision of validating and removing reference summaries with novel medical terms is based on our initial experiments about analyzing whether to place all references into positive sets. To give more details, we provide a third table below on experiments on the HQS dataset. As you pointed out, devaluing some references (placing them into the negative sets when they have novel medical terms) will hurt general summarization quality to a small extent with a performance decrease on ROUGE and BERTScore. However, we see a consistent and a more significant improvement on faithfulness scores as shown below. Paying the price of a trivial performance decrease on ROUGE and BERTScore, we view our validation of reference summaries as a contribution to related work. When patients do not use the correct medical terms, our method can be further extended to incorporate these examples that require the model to generate medical terms not in the input, and we will leave this as future work.\n\n|                                                                            | QuestEval  | FaR        | SummaC     | C F1      | R1        | R2        | RL        | BERTScore  |\n|----------------------------------------------------------------------------|------------|------------|------------|-----------|-----------|-----------|-----------|------------|\n| PEGASUS + FaMeSumm (all references are placed into positive sets as CLIFF) | 0.3105     | 0.3267     | 0.4138     | 28.90     | **31.14** | **12.29** | **29.09** | **0.7489** |\n| PEGASUS + FaMeSumm                                                         | **0.3134** | **0.3492** | **0.4401** | **30.92** | 30.84     | 12.19     | 28.99     | 0.7456     |\n\nRegarding style and presentation improvement, we will move Section 4.4 earlier in our final version to eliminate the confusion and modify our phrasing in lines 54-56 to “as many summarization approaches do not add inductive bias informed by faithfulness through model optimization”. Further elaboration on footnote 1 is provided above in Q (E).\n\n\nThanks a lot for bringing up those missing references. We will add them into Section 6 in our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "S1. The observation in footnote 1 (that contrastive learning is less effective when the positive examples include gold reference summaries that use medical terms not found in the input) is surprising, and a useful insight. (Though the authors do note that a similar issue is discussed in Adams (2022).). \n\nS2. The additional loss term L_MKI appears novel and has a clear intuitive justification. The same underlying concept of upweighting relevant domain terms during training could be useful across domains.\n\nS3. The authors perform a detailed human evaluation, and in this evaluation, the method improves factuality over CLIFF.", "reasons_to_reject": "W1. The ablations of the role of each loss term are incomplete. In particular, this work introduces two ideas: a different strategy for selecting/excluding examples for contrastive learning and an additional loss that rewards generating medical terms from the input. However, the new selection strategy is not compared to prior selection strategies (CLIFF) on its own, and so it is impossible to disentangle whether the improvement over prior work comes from the selection strategy or the additional loss term L_MKI. \n\nW2. Zhang et al. 2022 would be a stronger baseline than CLIFF, as it is also contrastive learning approach focused on medical summarization: \n>Ming Zhang, Shuai Dou, Ziyang Wang, and Yunfang Wu. 2022. Focus-Driven Contrastive Learning for Medical Question Summarization. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6176–6186, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.\n\nW3. In several places, the authors indicate that \"only a few papers\" (line 49, 552) focus on faithfulness in medical summarization, but there is ample prior work in this area. I feel the authors are slightly misrepresenting the state of the research in this direction. Additionally, the paper would benefit from an additional section in the related work discussing the prior work on contrastive learning for summarization.  \n\nW4. The taxonomy of error categories in Table 1 is similar to the taxonomy proposed in Pagnoni et al 2021, and would benefit from comparing/engaging with that taxonomy: \n\n> Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4812–4829, Online. Association for Computational Linguistics.\n\n**Edit after rebuttal**: The authors have run additional experiments to address W1 and W2, and I have updated my soundness score accordingly (2->3).", "questions_to_authors": "A. Is L_CL equivalent to CLIFF with a different selection strategy for positive/negative examples?\n\nB. For table 5, what is the relative performance of CLIFF? What about using only your L_CL?\n\nC. You describe FaMeSumm as \"cost-efficient\" (line 105) because only one model must be trained, but also state that it requires 48 GB of memory to finetune PEGASUS with batch size 2 (in Limitations). It's not clear to me that training 1 big model with contrastive learning is more efficient than prior approaches (finetuning without contrastive learning and training a second, smaller model as a faithfulness detector). Can you quantify the difference in cost (memory or GPU-hours) between your method and prior work? \n\nD. Your results in Table 6 are generally very small increases in ROUGE. Are these results from single runs or averaged over multiple runs? And did you perform statistical testing?\n\nE. The validation of reference summaries removes summaries with medical terms not mentioned in the input, but the datasets include health question summarization and dialogue summarization. The patients posing questions/speaking to the doctor may not use the correct medical terms, but that does not make the reference that uses those terms unfaithful. Did you notice any decrease in performance over test set examples that require the model to generate medical terms not found in the input? If so, do you consider this to be a positive or negative impact of the method?", "paper_topic_and_main_contributions": "This work proposes a customization of contrastive learning for medical summarization. They design an alternate strategy for constructing positive and negative examples for contrastive learning and introduce an additional loss that rewards the generation of medical terms from the input. They conduct automatic and manual evaluation of models trained with their three-part loss, and find that these models have higher factuality than models trained with only cross-entropy loss or a different contrastive learning setup.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "This would be a stronger baseline:\n* Ming Zhang, Shuai Dou, Ziyang Wang, and Yunfang Wu. 2022. Focus-Driven Contrastive Learning for Medical Question Summarization. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6176–6186, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.\n\nMore work on contrastive learning for summarization (though not necessarily medical summarization):\n* Zhichao Geng, Ming Zhong, Zhangyue Yin, Xipeng Qiu, and Xuanjing Huang. 2022. Improving Abstractive Dialogue Summarization with Speaker-Aware Supervised Contrastive Learning. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6540–6546, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.\n* Yixin Liu and Pengfei Liu. 2021. SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 1065–1072, Online. Association for Computational Linguistics.\n* Jinpeng Hu, Zhuo Li, Zhihong Chen, Zhen Li, Xiang Wan, and Tsung-Hui Chang. 2022. Graph Enhanced Contrastive Learning for Radiology Findings Summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4677–4688, Dublin, Ireland. Association for Computational Linguistics.\n* Chenxin An, Ming Zhong, Zhiyong Wu, Qin Zhu, Xuanjing Huang, and Xipeng Qiu. 2022. CoLo: A Contrastive Learning Based Re-ranking Framework for One-Stage Summarization. In Proceedings of the 29th International Conference on Computational Linguistics, pages 5783–5793, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.\n\nRelated to the idea of permuting attribute values to get negative examples (line 234): \n* Zheng Zhao, Shay B. Cohen, and Bonnie Webber. 2020. Reducing Quantity Hallucinations in Abstractive Summarization. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2237–2249, Online. Association for Computational Linguistics.\n\nTaxonomizing factual errors in summarization models:\n* Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4812–4829, Online. Association for Computational Linguistics.", "Typos_Grammar_Style_and_Presentation_Improvements": "line 54-56:\n> as many summarization approaches are based on language models that are pretrained on general domain text, they are inadequate\n\nAs I understand it, your method is also based on pretrained models (with additional loss terms customized for the domain). This phrasing suggests to me that your method does not use models pretrained on general domain text.\n\nSection 4.4 could be moved earlier in the paper-- around line 268-270, I was wondering how you performed the identification of medical terms. It would be nice to have that information in the first place where you describe using medical terms.\n\nFootnote 1 is interesting, and further elaboration on this point would be appreciated."}}, {"review_id": "b2PZduIItQ", "reviewer": "Reviewer_Ptjw", "summary": "", "questions": "(A) Are the backtranslated/data augmentation instances also verified for factuality?\n\n(B) What is the final summary representation? Several works describe options from an average over tokens, an average over named entity tokens, or using the last token. Line 254 “last layer of the decoder from a language model” is not a clear description - that could be the last token or an average of all tokens.\n\n(C) Why  are only a limited number of models evaluated for Tables 4 -8 (as opposed to all the models mentioned/evaluated in this work)?\n\n(D) How was the hyperparameter search  conducted? Grid? Some other optimization? The appendix only specifies the objective (a weighted average of Rouge-2 and length ratio), but not the actual method of search. \n\n(E) Similarly, is this tuning (particularly for the lambda weights) performed separately for the  ablations over loss functions? Repeating all parameter selection steps (particularly for the loss weightings, perhaps less so for learning rates and other hyperparameters)is required to answer the implicit question in ablation of “what if we only used  this particular setting”..   \n\n(F) What was the Fleiss Kappa on the reference?\n\n(G) Table 6: what is the FAMESUMM ensemble over? My best guess is “all models” but not all models always have results shown.\n\n(H) Are there any meaningful qualitative differences between summaries from the different models/configurations? As the automated metrics are (mostly) not capturing large differences between models and configurations, a qualitative evaluation of the actual changes they have on model text would be useful.\n\n(I) What are the GPT-3 ROUGE scores?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful feedback!\n\nWeakness (A): The reason that we used averaged scores on transformer-based models is to try to be fair for all models in Table 5. Based on our observation and a recent work (Mosbach et al., 2021), we noticed that the stability of transformer-based models is significantly lower than the baseline based on the pointer-generator network (Joshi et al., 2020). Since we do not want to report “lucky” (or “unlucky”) scores for our models, we report averaged scores in Table 5. Moreover, we did not choose a single score out of 3. Instead, we compute an averaged score for every metric in Table 5.\n\nWeakness (B) and (C): We address them in detail through the following questions.\n\nQ(A): When we constructed our positive sets, we had manually verified at least 20 examples for each dataset through random sampling, and we did not notice any unfaithful errors. Considering the fact CLIFF also utilized this method, we think the translated instances generally provide high-quality positive examples. As there is no theoretical guarantee on their faithfulness, we rank this method as the last method (discussed in line 208) if we are still behind the minimum size requirement of positive sets (there has to be at least 2 positive examples). We will stop the construction of positive sets whenever we meet this requirement, so most training instances do not have this kind of translated data.\n\nQ(B): Thanks for pointing out this ambiguity! The outputs produced by the last layer of the decoder are averaged over all tokens as the final summary representation.\n\nQ(C): Our experiments are designed to answer our research questions as listed in the beginning of Section 5, and we believe the current set of experiments are sufficient to reach the claimed contribution. In addition, we will add more experimental results in our final version as shown in the response to Reviewer 1 and Reviewer 3.\n\nQ(D): Yes, we performed grid search on the validation set of each dataset.\n\nQ(E): Yes, we performed tuning on those lambda weights separately. All lambda values can be viewed in Appendix C.\n\nQ(F): Thanks for your question. Table 7 aims to evaluate different models’ performance by doctors, so we did not ask doctors to evaluate reference summaries, and we do not have the Fleiss Kappa on reference. Our human evaluation is purely conducted by doctors. It is comprehensive by covering two datasets and five baselines while showing substantial improvements in faithfulness ratings (e.g., 6% to 7% improvement over CLIFF on HQS).\n\nQ(G): Thanks for your question! We provide more ensembling details here. On HQS benchmark, following the previous state-of-the-art (He et al., 2021), we rerank 4 system outputs (FaMeSumm fine-tuned with BioBART, T5, PEGASUS, and BART) based on four features: fidelity, length, consensus, and wellformedness. Computing a weighted sum of the four features, we pick the best summary from the 4 outputs for every test instance. Note that the previous state-of-the-art trained 4 systems on the union of training and validation data and utilized some ad-hoc techniques such as misspelling correction (mentioned in line 445 and 448), but **we do not need those to reach better performance**. The reason is that each of our single systems has better performance as shown in Table 3.\n\nOn RRS-Indiana, the previous state-of-the-art (Dai et al., 2021) fine-tuned **16** PEGASUS (with different seeds) on the union of training and validation data and performed ensembling based on mutual similarity scores among any two system outputs. In contrast, we leverage the observations conducted on the training set in that paper and rerank system output out of **7** FaMeSumm (fine-tuned based on PEGASUS with different seeds) outputs.\n\nOn RRS-Stanford, the previous state-of-the-art (Dai et al., 2021) fine-tuned **16** PEGASUS (with different seeds) on the training set with output normalization (this normalization is based on an observation mentioned in the previous paragraph). We reach better performance by **just using one** FaMeSumm (fine-tuned based on PEGASUS with different seeds) output. This FaMeSumm model is the same model as the last row of Table 4, except that length penalty parameter is additionally tuned on the validation set in order to encourage our model to generate longer output.\n\n\nQ(H): For qualitative analysis, Figure 2 and Section 5.5 present a case study (translation provided) to showcase the effectiveness of FaMeSumm compared with baselines such as mT5 and PEGASUS. Our takeaway message about this case study is that FaMeSumm can generate more accurate domain-specific terms, entities, and their relationships such as negation to reduce popular unfaithful errors mentioned in Table 1, which improves summarization faithfulness. We will add more examples in the Appendix in our final version.\n\nQ(I): We just computed the ROUGE scores for GPT outputs, and the R1, R2, and RL of GPT-3 on HQS are 30.51, 11.61, and 27.85. These scores are lower than FaMeSumm in Table 3. We will add these in our final version.\n\nTypo (A): Thanks for pointing this out. We have translation in Figure 2,and we will surely add translation for Table 1 as well.\n\nTypo (B): Good point! We will rephrase line 377 to “Does FaMeSumm…”.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper runs a reasonable set of experiments, with methods not previously applied as thoroughly or as well to this domain, is mostly clear on the experimental details, and shows improvements on a human evaluation.", "reasons_to_reject": "(A) Table 5 - Averaging scores over three random seeds is not a fair evaluation. The paper must specify how a single answer is chosen, whereas the average conflates aggregates over three scores.\n\n(B) The paper leaves many open questions, some of which might be addressable via some editing, appendix entries, and/or automated methods (questions (A), (B), (C), (D), (E), (G), (I)), while others might require more experimental work ((F), (H)).\n\n(C) Question (H), regarding finding any qualitative differences, is an important question for understanding what the methods in this work actually do.", "questions_to_authors": "(A) Are the backtranslated/data augmentation instances also verified for factuality?\n\n(B) What is the final summary representation? Several works describe options from an average over tokens, an average over named entity tokens, or using the last token. Line 254 “last layer of the decoder from a language model” is not a clear description - that could be the last token or an average of all tokens.\n\n(C) Why  are only a limited number of models evaluated for Tables 4 -8 (as opposed to all the models mentioned/evaluated in this work)?\n\n(D) How was the hyperparameter search  conducted? Grid? Some other optimization? The appendix only specifies the objective (a weighted average of Rouge-2 and length ratio), but not the actual method of search. \n\n(E) Similarly, is this tuning (particularly for the lambda weights) performed separately for the  ablations over loss functions? Repeating all parameter selection steps (particularly for the loss weightings, perhaps less so for learning rates and other hyperparameters)is required to answer the implicit question in ablation of “what if we only used  this particular setting”..   \n\n(F) What was the Fleiss Kappa on the reference?\n\n(G) Table 6: what is the FAMESUMM ensemble over? My best guess is “all models” but not all models always have results shown.\n\n(H) Are there any meaningful qualitative differences between summaries from the different models/configurations? As the automated metrics are (mostly) not capturing large differences between models and configurations, a qualitative evaluation of the actual changes they have on model text would be useful.\n\n(I) What are the GPT-3 ROUGE scores?", "paper_topic_and_main_contributions": "This work reports an NLP Engineering experiment for improving medical summarization systems (and reproduces several baselines) via training with loss functions targeted to increase faithfulness (against constructed negative samples) and topicality with respect to the inputs (as measured by weighted term mentions). It includes several model variants (based in PEGASUS, BART, T5), a comparison against GPT-3, an ablation over the objective modifications, several automated metrics (QuestEval, FaR, SummaC, C(oncept) F1, and ROUGE variants), and a human evaluation. The automated metrics show a possibly meaningful difference in SummaC and C F1, while all other automated metrics show only small (mostly marginally improved) results. The human metrics tell a more interesting story, with a clear improvement from adding the new objectives and a lack of general utility from many of the automated metrics.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "(A) I do not read any Chinese - since it seems the authors are bilingual , it may be beneficial for readers to include a translation of the examples in the Appendix. I have not counted this against the paper.\n\n(B) RQ1 (line 377) - this should be “Does FAMESUMM” not “How can” as the how of the method causing improvements is speculative. Rephrasing this makes it clear that RQ1 and RQ3 are not actually different."}}, {"review_id": "oTIkUeOuYI", "reviewer": "Reviewer_fbJn", "summary": "", "questions": "- How is ensembling performed?\n- Are the logits before softmax averaged across all reference tokens or is the MKI dot product computed at the token-level?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful feedback!\n\n  To clarify the confusion about weakness 1 and 2, we organize our response into the following 4 points:\n\n**1. Comparison between our CL and CLIFF**\n\nWe choose HQS to compare FameSumm with CLIFF because: (1) On MDS, it is hard to accurately implement CLIFF contrastive sets because it is a Chinese dataset. We would like to humbly point out that we compared CLIFF with human evaluation only on HQS, not MDS. (2) On RRS, our focus is to compare with other state-of-the-art models.\n\n  As shown in Table 7, our pair-wise human evaluation between CLIFF and FameSumm on HQS shows that the improvement of FameSumm is significant with 6% to 7% increase in Faithfulness rating.\n\n  To sort out the factors that drive performance increase, we report new experiment results by training FaMeSumm CL without MKI on both HQS and MDS datasets, and the performance is reported in the first two tables below (the first table is about HQS, and the second one is about MDS). Combining with the scores reported in the original paper, on HQS, we see that FaMeSumm CL has higher scores than CLIFF on all faithfulness metrics, which demonstrates the advantage of our proposed CL method. Its ROUGE and BERTScore are lower than CLIFF, because we disvalue many reference summaries by placing them into the negative sets. Paying the price of this trivial performance decrease, we obtain a consistent increase on faithfulness measures. Since FaMeSumm CL is not good at improving entity-based faithfulness, we see a marginal increase from FaMeSumm CL over CLIFF on FaR and C F1. We add MKI here to gain a more significant increase. In conclusion, FaMeSumm CL is better than CLIFF.\n\n**2. Complementary roles of CL and MKI**\n\n  Analyzing the second table, we aim to analyze the complementary roles of CL and MKI. We can see that FaMeSumm CL has better RL and BERTScore than MKI only, which shows that improving faithfulness from question answering and entailment aspects has the potential of improving general summarization quality. It also has better scores than plain mT5.\n\n Contrary to FaMeSumm CL, MKI is most effective on entity-based faithfulness metrics as shown in the second table. It yields the highest C F1 and the second highest FaR. We can see that MKI has the strongest impact on C F1 and FaR. \n\n  We also trained the simpler version as suggested in the review. Technically speaking, this “simpler” version does not differ much from our MKI in terms of implementation difficulty and training efficiency (e.g., GPU memory consumption and training time used for 1 epoch). The reason is that the PyTorch implementation of cross entropy loss requires us to construct a weight vector that has the same dimension as the vocabulary of the backbone model (see our answer to question 2; these 2 implementations are very similar). Then, we compare their performance in the second table and find that MKI is significantly better. We offer two reasons to explain this phenomenon: (1) our MKI considers modeling the context tokens (2 tokens before medical terms and negative unigrams) as our unique contribution to this domain, and this greatly facilitates the generation of medical terms, since almost all existing language models generate tokens in an autoregressive fashion so modeling context tokens helps; (2) optimizing logits instead of weighted cross entropy loss has a more direct impact on the decoding process of language models.\n\n**3. Ablation of different types of contrastive sets**\n\n In this third table, it is clear to see the reason why we disvalue some reference summaries for positive sets construction as mentioned in footnote 1. The last row dominates all faithfulness metrics while having slightly lower ROUGE and BERTScore. Since this paper mainly deals with improving faithfulness and the decrease is trivial, we place some reference summaries into negative sets if they fail our check described in Section 3.1.\n\n|                                              | QuestEval  | FaR        | SummaC     | C F1      | R1        | R2        | RL        | BERTScore  |\n|----------------------------------------------|------------|------------|------------|-----------|-----------|-----------|-----------|------------|\n| PEGASUS                                      | 0.3069     | 0.3188     | 0.4279     | 26.24     | 30.19     | 11.93     | 28.52     | 0.7427     |\n| PEGASUS + CLIFF                              | 0.3145     | 0.3217     | 0.4225     | 30.18     | 29.92     | 11.40     | 27.67     | 0.7401     |\n| PEGASUS + FaMeSumm                           | 0.3134     | **0.3492** | **0.4401** | **30.92** | **30.84** | **12.19** | **28.99** | **0.7456** |\n| FaMeSumm CL only (**new experiment**)            | **0.3157**     | 0.3222     | 0.4355     | 30.20     | 29.31     | 10.82     | 27.39     | 0.7352     |\n\n|                                                       | FaR    | C F1  | RL    | BERTScore |\n|-------------------------------------------------------|--------|-------|-------|-----------|\n| mT5*                                                  | 0.3784 | 36.96 | 32.43 | 0.7505    |\n| +CL* (**new experiment**)                                 | 0.4040 | 37.85 | 32.73 | 0.7518    |\n| +MKI*                                                 | 0.4072 | **39.30** | 32.27 | 0.7492    |\n| +FaMeSumm*                                            | **0.4177** | 38.57 | **33.05** | **0.7520**    |\n| Increase CE weights on medical terms* (**new experiment**) | 0.4024 | 37.79 | 31.73 | 0.7485    |\n\n\n|                                                                            | QuestEval  | FaR        | SummaC     | C F1      | R1        | R2        | RL        | BERTScore  |\n|----------------------------------------------------------------------------|------------|------------|------------|-----------|-----------|-----------|-----------|------------|\n| PEGASUS + FaMeSumm (all references are placed into positive sets as CLIFF) | 0.3105     | 0.3267     | 0.4138     | 28.90     | **31.14** | **12.29** | **29.09** | **0.7489** |\n| PEGASUS + FaMeSumm                                                         | **0.3134** | **0.3492** | **0.4401** | **30.92** | 30.84     | 12.19     | 28.99     | 0.7456     |\n\n**4. Motivation, explanation, and empirical evidence of MKI**\n\nThe motivation of MKI is to facilitate the generation of summaries with more accurate entities, terms, and their logical relationships such as negation captured by context words, thus improving faithfulness in the medical domain. To this end, we design a loss term L_MKI to encourage models to maximize the prediction scores of the medical terms as well as their context words. Specifically, in Equation 1, we use the dot product between b_m and p which have the same dimension as the vocabulary size. Here, b_m records the frequency of medical terms and their context tokens, while p is the logits of medical terms and their context by averaging across all reference tokens. Our motivation has been further supported by our empirical results in Table 5, Table7, and new experiments above, demonstrating that MKI is most effective on entity-based faithfulness metrics. \n\nQ1: Thanks for your question! We provide more ensembling details here. On HQS benchmark, following the previous state-of-the-art (He et al., 2021), we rerank 4 system outputs (FaMeSumm fine-tuned with BioBART, T5, PEGASUS, and BART) based on four features: fidelity, length, consensus, and wellformedness. Computing a weighted sum of the four features, we pick the best summary from the 4 outputs for every test instance. Note that the previous state-of-the-art trained 4 systems on the union of training and validation data and utilized some ad-hoc techniques such as misspelling correction (mentioned in line 445 and 448), but **we do not need those to reach better performance**. The reason is that each of our single systems has better performance as shown in Table 3.\n\nOn RRS-Indiana, the previous state-of-the-art (Dai et al., 2021) fine-tuned **16** PEGASUS (with different seeds) on the union of training and validation data and performed ensembling based on mutual similarity scores among any two system outputs. In contrast, we leverage the observations conducted on the training set in that paper and rerank system output out of **7** FaMeSumm (fine-tuned based on PEGASUS with different seeds) outputs.\n\nOn RRS-Stanford, the previous state-of-the-art (Dai et al., 2021) fine-tuned **16** PEGASUS (with different seeds) on the training set with output normalization (this normalization is based on an observation mentioned in the previous paragraph). We reach better performance by **just using one** FaMeSumm (fine-tuned based on PEGASUS with different seeds) output. This FaMeSumm model is the same model as the last row of Table 4, except that length penalty parameter is additionally tuned on the validation set in order to encourage our model to generate longer output.\n\nWe will describe these implementations with more details in our final version and share our ensembling code for each dataset.\n\nQ2: Thanks for your question! In Equation 1, both b_m and p have the same dimension as the vocabulary size. b_m records the frequency of medical terms and their context tokens, while p is the logits of medical terms and their context by averaging across all reference tokens. Therefore, the MKI dot product in Equation 1 is computed at vocabulary-level, not token-level. We will clarify this in our final version.\n\n\n\nThanks a lot for bringing up these missing references. All these publications have related but different focuses than our work. We will add them into Section 6 in our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is very clearly and well-written. It is nicely polished with relevant related work, good diagrams, and thorough experimental analysis.\n\n- The decision to not always include a reference as a positive example is smart and an easy win. That is a simple yet clear improvement over the perturbations in CLIFF. The other modifications are decided based on a qualitative error analysis, which is the right way to go about it.\n\n- They test their methods on many downstream datasets and conduct a human evaluation of outputs.", "reasons_to_reject": "- The experimental design doesn't adequately test the benefits of FaMeSumm over other CL methods. CLIFF is only tested against automatic metrics for one dataset (HQS). While it is compared with CLIFF on human annotations for 2 datasets (HQS and MDS), the results seem marginal.  I think it would help to run a pair-wise comparison annotation between CLIFF and FaMeSumm to see if it gets stronger results. It would also be helpful to directly compare FaMeSumm CL against CLIFF without the MKI loss, as it's hard to suss out what's diving the performance increase. For MKI, it's not very well motivated or compared to other methods which aim to boost the generation of salient content.  I would want to see a simpler version which just places a higher CE weight on medical terms.  It would be nice to also see an ablation of all the types of corruptions made to positive and negative sets.\n\n- The MKI objective could be better motivated and explained. See questions below.", "questions_to_authors": "- How is ensembling performed?\n- Are the logits before softmax averaged across all reference tokens or is the MKI dot product computed at the token-level?", "paper_topic_and_main_contributions": "They add two objectives to fine-tuning on medical summarization models.  The first is a contrastive objective (which is based on the CLIFF objective) but they use different methods for generating negative and positive contrast sets. The second is a novel objective (MKI) which encourages the model to generate medical concepts in the references.\n\nThey show improved performance over baseline models (BART, T5, mT5, and PEGASUS) when fine-tuning with their multi-task objectives.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "For contrastive learning:\n\nFeng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero Nogueira dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen McKeown, and Bing Xiang. 2021. Entity-level Factual Consistency of Abstractive Text Summarization. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2727–2733, Online. Association for Computational Linguistics.\n\nGriffin Adams, Bichlien Nguyen, Jake Smith, Yingce Xia, Shufang Xie, Anna Ostropolets, Budhaditya Deb, Yuan-Jyue Chen, Tristan Naumann, and Noémie Elhadad. 2023. What are the Desired Characteristics of Calibration Sets? Identifying Correlates on Long Form Scientific Summarization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10520–10542, Toronto, Canada. Association for Computational Linguistics.\n\nFor clinical text evaluation:\n\nAdams, Griffin, Jason Zucker, and Noémie Elhadad. \"A meta-evaluation of faithfulness metrics for long-form hospital-course summarization.\" arXiv preprint arXiv:2303.03948 (2023).\n\nFor modeling:\n\nLiu, Fenglin, et al. \"Retrieve, reason, and refine: Generating accurate and faithful patient instructions.\" Advances in Neural Information Processing Systems 35 (2022): 18864-18877."}}, {"review_id": "2DQO6E8T0g", "reviewer": "Reviewer_UEQF", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful feedback!\n\nWeakness1: Regarding the change in summary length of positive sets, first of all, we do notice a significant length difference between reference summaries and positive examples for some datasets. For example, in HQS dataset, its reference summaries used for training have an average number of 10.1 tokens per instance, but its positive examples have an average number of 33.2 tokens per instance. As demonstrated in Table 3, 4, 5, and 7, this difference has a positive effect on faithfulness and summarization quality, because combining reference summaries with these longer positive examples form a diverse training data which helps FaMeSumm to better learn a decision boundary about generating tokens.\n\nWe will add translation for examples in Table 1 and a clarification about the change in summary length of positive sets in our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper propose a proven effective solution to a real problem in current summarization systems. The solution is well described and well evaluated. Results are good and adequately discussed.", "reasons_to_reject": "Section 3.1  is not clear enough. In particular, the paragraph on \"Data augmentation by sentence extraction\" is modifying  reference summaries by adding new sentences from the source. It is not clear how this change in summary length will impact the capacity of summarizers.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper describes the augmentation of recent summarization systems with methods to improve their fidelity in respecting the meaning of the original documents.\nIt starts by studying the level of faithfulness errors in current systems and then propose several means to improve them (creating more contrastive summaries, incorporating medical knowledge and cross entropy loss). Previous systems are the fine-tuned using the proposed learning objective.\nResults on several datasets show the effectiveness of the proposed fine-tuning methods.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper should include translations of examples in Table 1 to help non-Chinese reading readers to understand the errors."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper explores methods for improving the faithfulness of medical summarization. It offers a solution to a tangible issue in summarization systems. The paper is commendable for its clear writing, inclusion of related work, and comprehensive experimental analysis. However, reviewers also pointed out concerns, including an omission of prior work on contrastive learning for summarization, similarities between the error taxonomy in Table 1 and Pagnoni et al. 2021, insufficient testing of FaMeSumm's benefits compared to other CL methods, and several unanswered questions that may need further experimentation.
Decision: Accept-Main