Assessing Privacy Risks in Language Models: A Case Study on Summarization Tasks
Authors: Ruixiang Tang
Gord Lueck
Rodolfo Quispe
Huseyin A Inan
Janardhan Kulkarni
Xia Hu
Venue: EMNLP 2023 Findings (2023)
Keywords: Summarization
Membership Inference Attack
Privacy
Language Model

Abstract: Large language models have revolutionized the field of NLP by achieving state-of-the-art performance on various tasks. However, there is a concern that these models may disclose information in the training data. In this study, we focus on the summarization task and investigate the membership inference (MI) attack: given a sample and black-box access to a model's API, it is possible to determine if the sample was part of the training data. We exploit text similarity and the model's resistance to document modifications as potential MI signals and evaluate their effectiveness on widely used datasets. Our results demonstrate that summarization models are at risk of exposing data membership, even in cases where the reference summary is not available. Furthermore, we discuss several safeguards for training summarization models to protect against MI attacks and discuss the inherent trade-off between privacy and utility.

Assessing Privacy Risks in Language Models: A Case Study on Summarization Tasks
Ruixiang Tang* Gord Lueck Rodolfo Q Huseyin A Inan
Janardhan Kulkarni Xia Hu*
Department of Computer Science, Rice University, TX, USA Microsoft, Redmond, WA, USA
Microsoft Research, Redmond, WA, USA
edu [gordonl, edquispe, huseyin. inan, jakul jakul)@@ic cwwiiiroooo..ommm com
Abstract
dataset. This problem can be formulated as an ad- versarial scenario, with Bob acting as the attacker and Alice as the defender. Bob proposes methods to infer the membership, while Alice attempts to make membership indistinguishable. In the past, researchers proposed various attacking and defend- ing techniques. However, most of them focus on the computer vision classification problem with fixed set of labels. Little attention has been given to comprehending MI attacks Seq2Seq models.
Large language models have revolutionized the field of NLP by achieving state-of-the-an -het--rrtt---tttt per- formance on various tasks. However, there is concern that these models may disclose infor- mation in the training data. In this study, we focus on the summarization task and investigate the membership inference (MI) attack: given sample and black-box access to model's API, it possible to determine if the sample was part of the training data. We exploit text similarity and the model's 's resistance to document modi- fications as potential MI signals and evaluate their effectiveness widely used datasets. Our results demonstrate that summarization mod- els are risk of exposing data membership. even in cases where the reference summary is not available. Furthermore, we discuss several safeguards for training summarization models to protect against MI attacks and discuss the inherent trade-off between privacy and utility.
This paper focuses on the summarization task and investigates the privacy risk under membership inference attacks. Inspired by previous research in MI literature(Shokr et al., 2017; Hisamoto et al., 2020), we pose the problem as follows: given black- box access to summarization model's 's API, can we identify whether document- pair used train the model?. Compared to membership inference attacks on fixed-label classification tasks, text generation tasks present two significant chal- lenges: (1) The process of generating summaries in- volves a sequence of classification predictions with variable lengths, resulting in complex output space. (2) Existing attacks heavily rely on the output prob- abilities (Shokri et al., 2017; Mireshghallah et al., 2022), which is impractical when utilizing APIs of Seq2Seq models. Therefore, it remains uncertain whether the methodologies and findings developed for classification models can be applied to language generation models.
1 Introduction
Text summarization seeks condense input docu- ment(s) into a shorter, more concise version while preserving important information. The recent large language models have significantly enhanced the quality of the generated summaries (Rothe et al., 2021; El-Kassas et al., 2021: Chung et al., 2022). These models have been applied to many sensitive data, such as clinical and finance reports (Zhang al., 2020b; Abacha et al., 2021). Given these high-stakes applications, it is critical to guarantee that such models do not inadvertently disclose any information from the training data and that data remains visible only to the client who owns it.
A pertinent question emerging from the study is the rationale behind the efficacy of MI attacks on summarization models. The key insight lies in the training objective of these models, which aims to minimize the discrepancy between the generated and reference summary (See et al., 2017). Conse- quently, samples with significantly lower loss (in- dicating similarity between the generated and ref- erence summaries) are more likely be part of the training dataset. Based on this concept, we propose a baseline attack method that utilizes the similarity
To evaluate the potential memorization of spe-
cific data by a model, membership inference (MI) attack (Shokri al., 2017) has become the de facto standard, owing to its simplicity (Murakonda et al., 2021). Given a model and sample (input label pair), the membership inference attack wants to identify whether this sample was in the model's training

---

between the generated and reference summaries differentiate between training and non- con-training samples. One limitation of the baseline attack that Bob requires access to both the documents and reference summaries to launch the attack, rendering the attack less practical for summarization tasks. Building upon this, our study introduces more general document-only attack: Given only a doc- ument and black-bos access to the target model's APIs, can Bob infer the membership without ac- cess to the reference summary? We tackle this problem by examining the robustness of generated summaries in response to perturbations in the input documents. According to the max-margin princi- ple, training data tends to reside further away from the decision boundary, thus exhibiting greater re- silience to perturbations, which aligns with observa- tions from prior research the adversarial domain (Tanay and Griffin, 2016; Choquette-Chie Choo et al., 2021). Consequently, Bob can extract fine-grained membership inference signals by evaluating data robustness toward perturbations. Remarkably, we show that Bob can estimate the robustness without reference summaries, thereby enabling document- only attack. In summary, this work makes the fol- lowing contributions to the language model pri- vacy:
between training and non-training samples. How- ever, this approach feasible for Seq2Seq mod- els. For each generated token in these models, the output probability over the word vocabulary often comprises tens of thousands of elements-for in- stance, c.g., the vocabulary size for BART is 50,265 (Lewis et al., 2020). As such, most public APIs do not offer probability vectors for each token but rather furnish overall confidence score for the sequence, calculated based on the product of the predicted tokens' probabilities.
Natural Language Privacy. An increasing body of work has been conducted on understanding pri- vacy risk in NLP domain. (Hayes et al., 2017; Mee- han et al., 2022; Chen et al., 2022; Ponomareva et al., 2022). Pioneering research has been dedi- cated to studying MI attacks in NLP models. The study by (Hisamoto et al., 2020) examines the black-box membership inference problem of ma- chine translation models. They assume Bob can access both the input document and translated text and use BLEU scores as the membership signal, which is similar to our baseline attack. (Song and Shmatikov, 2019) investigate a white-box MI at- tack for language models, which assume Bob can obtain the probability distribution of the generated token. Different from previous work, our attack is under the black-box setting and considers more general document-onis only attack in which Bob only needs input documents for membership inference.
1. Defined the blx MI attack for the sequence-to-scquennee model. Experiments on sum- marization tasks show attackers can reliably infer the membership for specific instances.
3 Preliminaries
2. Explored data robustness for MI attacks and found that the proposed approach enables attackers to launch the attack solely with the input document.
3.1 Problem Definition
We introduce two characters, Alice and Bob, in the membership inference attack problem. Alice (Defender) trains a summarization model on private dataset. We denote a document as f and its corresponding reference summary 8. Alice provides API to users, which takes document as input and returns a generated summary s. Bob (Attacker) has access to data similar Al- ice's data distribution and wants to build a binary classifier g(-) to identify whether a sample is in Al- ice's 's training data, Atrain The sample comprises a document f and its reference summary s. Together with the API's output 8. Bob uses g(.) to infer the membership. whose goal is to predict:
3. Evaluate factors impacting MI attacks, such dataset size, model architectures, etc. We also explore multiple defense techniques and discuss the privacy-utility trade-off.
2 Background and Related Works
Membership Inference Attacks. In a typical black-box MI attack scenario, as per the literature (Shokri et al., 2017; Hisamoto et al., 2020), it is posited that the attacker, Bob, can access a data distribution identical to Alice's 's training data. This access allows Bob to train shadow model, us- ing the known data membership of this model as ground truth labels train an attack classifier. Bob can then initiate the attack by sending queries to Al- ice's model APIs. Most previous studies leverage disparities in prediction distributions to distinguish
( = Otherwise
If (f, 8) E Atrain
g(S, 8,8)
(1)

---

m is the ground truth membership, the Attack Ac-
Tost st
Train s())
curacy (ACC) is defined as:
Ain
Aout
Bin
Bout
F [F El(ts ml, (2)
1
ACC F)
Ataii
where an accuracy above 50% can be interpreted as a potential compromise of privacy. Following a similar definition, we can define other commonly used metrics, such as Recall, Precision, AUC, etc. Previous literature mainly uses accuracy or AUC to evaluate the privacy risk (Song and Shmatikov, 2019; Hisamoto et al., 2020; Mahloujifar et al., 2021; Jagannatha et al., 2021). However, these metrics only consider an average case and are not enough for security analysis (Carlini et al., 2022). Consider comparing two attackers: Bob; perfectly infers membership of 1 1% of the dataset but suc- ceeds with a random 50% chance on the rest. Bob succeeds with 50.5% on all dataset. On average, two attackers have the same attack accuracy or AUC. However, Bobj demonstrates exceptional potency, while Bob₃ is practically ineffective. In order to know if Bob can reliably infer the mem- bership in the dataset (even just a few documents). we need to consider the low False-Positive Rate regime (FPR), and report an attack model's True- Positive Rate (TPR) at a low false-positive rate. this work, we adopt the metric TPRg.1% which is the TPR when FPR = 0.1 1..
Figure Data Splitting.
3.2 Shadow Models and Data Splitting
In this work, we follow the typical settings in the MI attack (Shokri al., 2017; Hisamoto et al.,
2020; Jagannatha al., 2021; Shejwalkar al., 2021) and assume Bob has access to the data from the same distribution as Alice to train their shadow models. Subsequently, Bob utilizes the known data membership of the shadow models as training la- bels to train an attack classifier g()), whose goal is to predict the data membership of the shadow model. If the attack the shadow model proves successful, Bob can employ the trained attack clas- sifier to attempt an attack on Alice's model.
As depicted in Figure 1, we follow the previous work setting (Hisamoto et al., 2020) and split the whole dataset as Agll and Bau, with Alice only having access to Autt and Bob only has access to Ban. For Alice, Aoul is further split into two parts: Atrain and Aout, where Agrain is utilized for train- ing a summarization model and Aout serves hold-out dataset that is not used. (Note that Atrain includes the data used for validation and testing, and we use Atrain to specify the data used to train the model). In the case of Bob, Ball is further split into Bin and Bout, where Bob employs B;, to train shadow models and Bunt serves as hold-out dataset. To construct the attack classifier g(-). Bob train 9-)) with the objective of differentiating samples in Bin and Bout-
4 MI Attacks for Summarization Tasks
4.1
A Naive Baseline
The bascline attack is based on the observation that the generated summaries of training data often ex- hibit higher similarity to the reference summary, i.e., lower loss value (Varis and Bojar, 2021). In an extreme case, the model memorizes all training reeeeenntrrfeerecce summary pairs and thus generate perfect summaries for training samples (s 3). Hence, it is natural for Bob exploit the similarity between and s as a signal for member- ship inference.
3.3 Evaluation Protocols
We adopt the following evaluation protocols to evaluate g(-) attack performance on Alice's model. Given a document and its corresponding reference summary (f, s), selected from Atrain or Aout, Bob sends f to Alice's s API and gets the output sum- mary &. Then Bob employs the trained classifier g(-) to infer whether the pair sf))) is present in Al- ice's training data. Since Atrain is much larger than Aouto we make the binary classification task more balanced by sampling a subset Ain from the Atran with the same size as Aout- Given a set F of test samples (f, 8, 8, m), where (f)s) s) E Ain U Apntr
There are multiple approaches to quantify- ing text similarity. Firstly. Bob utilizes the human-desigh metrics ROUGE-1, ROUGE-2, and ROUGE-L scores to calculate how many semantic content units from reference texts are covered by the generated summaries (Lin, 2004; Lin and Och, 2004). Additionally, Bob can adopt neural-based language quality scores, c.g., sentence transformer ScereeReimers and Gurevych, 2019), to capture the

---

semantic textual similarity, Finally, we follow stud- ies in the computer vision domain and also leverage the confidence score, such as the perplexity score, as the MI attack feature. Bob then concatenates all features to one vector, i.e., [ROUGE-I, ROUGE-2, ROUGE-L, Transformer Score, Confidence Score], and employs classifiers, such as random forest and multi-layer perceptron, to differentiate training and non-training samples. The baseline attack can be written follows:
4.2.1 Document-only MI Attack
Existing attack methods need Bob to access both the document and its corresponding reference summary 8 to perform membership inference. However, is challenging for Bob to obtain both of these for summarization tasks. Here, we propose a low-resource attack scenario: Bob only has docu- ment and aims to determine whether the document is used to train the model. Under this scenario, the
previously proposed attacks cannot be applied there is reference summary available.
Base Base ()))
However, the concept of evaluating sample ro- bustness offers potential solution as we can ap- proximate the robustness without relying on ref- erence summaries. To address this, we modify the g)))uug Instead of calculating the similarity scores between generated summaries (6, .., %1) and reference summaries 8, Bob replaces the ref- erence summary with the generated summary 3, and estimate the document robustness by calculat- ing the similarity scores between and perturbed documents' summaries (61, ..., $4). The proposed document only MI attack can be written as follows:
where sim represents the function that takes two summaries as input and returns a vector of selected similarity evaluation scores.
4.2 Document Augmentation for MI
The baseline attack is limited to relying solely on text similarity information from a single query. However, if Bob has the capability to send mul- tiple queries to Alice's API, can Bob potentially explore more nuanced membership signals? Build- ing upon this concept, robustness of output summaries when subjected to perturbations in input documents as the attack fea- ture. Following a max-margin perspective, samples that exhibit high robustness are training data points (Tanay and Griffin, 2016; Hu et al., 2019; Deniz et al., 2020; Choquette-Chon et al., 2021). For our task, the assumption is that documents in the train- ing dataset are more robust to perturbations and will have less change in the output summaries. Bob consider various augmentation methods, such as word synonym replacement and sentence swap- ping. We use D to denote the set of augmentation methods. Given a document f. Bob chooses one augmentation method d E D and generates n new documents, i.e. f fd. Bob then queries the API using the augmented documents and obtains the output summaries (J, ( ..., (%) -> (8, .i.... sd)) To train the classifier, Bob uses similarity scores between all summaries with the reference summary as the feature, which can be written as follows:
propose utilizing the
g D only g([sim(${) ..., sim(s, (8g, 8)]). (5)
Compared to g(-)ang. the proposed g(-) ob- tains robustness information for the document only with the generated summary 8. Our experiments show that this approximate robustness contains valuable membership signals, and g(-) effectively infer the membership of specific sam- ples using only the documents as input.
Experiment Setup
5.1 Dataset
We perform our summarization experiments on three datasets: SAMsum, CNN//ailyMail (CN- NDM), and MIMIC-cxr (MIMIC).
SAMsum (Gliwa al., 2019) is dialogue summarization dataset, which is created by ask- ing linguists to create messenger-like conversa- tions. Another group of linguists annotates the reference summary. The original split includes 14,732/81 (..732/818/819 dialogue-summan summary pairs for train- ing/validation/tess.
A(g) g( ([ssim(8, s), sim( (f, s),.., .... sim(3d s)D
CNNDM (Hermann et al., 2015) is news article summarization dataset. The dataset collects news articles from CNN and DailyMail. The summaries are created by human annotators. The original split includes 287,227/13,368//
(4)
Compared eq. 3, the proposed g( g(-)Aug can ad- ditionally use the summaries' robustness informa- tion for MI, e.g., the variance of similarity scores var(($ s)... sim()()s)).
summary pairs for 111.3////111190001 news article-

---

out
ROUGE-1
ROUGE-1
ROUGE-1
ROUGE-2 Precision
ROUGE-2
ROUGE-2
ROUGE-L
ROUGE-L
ROUGE-L
Confidenze
Trunsfommer Scure
Soore
Figure 2: Distribution of similarity scores of Ain and Aout of SAMsum dataset.
SAMsum
CNNDM
MIMIC
ACC
AUC
TPRg.1%
ACC
AUC
1P5g.1%
ACC
AUC
TPR
RF
61.10
64.72
1.31
53.48
55.38
0.83
65.41
66.37
2.58
LR
61.15
65.88
1.05
51.24
53.88
0.05
66.73
68.64
2.20
SVM
61.67
65.45
2.03
50.30
52.72
0.03
65.90
69.24
2.34
MLP
61.73
65.84
2.15
52.33
55.84
1.17
67.11
70.71
3.05
RoBERTa 60.10
63.27
1.26
50.01
51.71
0.05
66.08
68.01
2.05
Table Baseline Attack Results. Bob tried different classifiers, including Random Forest (RF), Logistic Regression (IRR), Support Vector Machine (SVM), and Multi-layer Perceptron (MLP). Following the evaluation protocol in Section 3.3, we show membership attack performance on Asn and Aout.
MIMIC is public radiology report summariza- tion dataset. We adopt task 3 in MEDIQA 2021 (Abacha et al., 2021), which aims to generate the impression section based on the findings and background sections of the radiology report. We choose the MIMIC-exr as the data source, and the original split includes 91544/2000 medical report- impression pairs for training/validation.
We adopt Adam (Kingma and Ba, 2014) as the optimizer. For SAMsum, CNNDM, and MIMIC, the batch size is set 10/4/4, and the learning rate is set 2e 5. 2r le During inference, set the length penalty as 2.0, the beam search width as 5, and the max/min generation length as 60/10, 140/30, and 50/10. Alice chooses the best model based on the validation ROUGE-L score. Bob randomly splits Baul into two equal parts: Bin and B Bob employs Bi, to train shadow mod- els and chooses the best model based on the vali- dation ROUGE-L performance. We only trained one shadow model in the experiment. Our imple- mentation is based on the open- open-source PyTorch- transformer repository. All experiments are re- peated 5 times and report the average results.
As we discussed in Sec. 3.2, we reorganized the datasets into three disjoint sets: Airain, Ands and Bau- We assume Bob can access around 20% of the dataset to train shadow models and 9()). In Table 2, show the details number for each split.
Atrain
Ain
Aout
Bau
SAMsum
13,369
1,000
1,000
2,000
CNNDM
252,085
20,000
20,000
40,000
5.3 Augmentation Methods
MIMIC
78.544
10,000
10,000
20,000
We consider three augmentation methods: word synonym (WS), sentence swapping (SW), and back translation (BT)²: word synonyms randomly choose 10% of words in a document and change to their synonym from WordNet(Miller. 1995), sen- tence swapping randomly chooses a sentence and swap it with another sentence, back translation first translates the document to French, Spanish, and
Table 2: Each dataset is divided in to three disjoint sets: Atrain Aont and Ball- Asn is sampled from Arrasn with same size Agut-
5.2 Models and Training Details
our experiments, we adopted widely used summarization models: BART-base (Lewis al.,
'hups://gittubb. www.//ugginnffacertaasaooooo
2020) and FLAN-T5 base (Chung et al., 2022) (Re- sults of FLAN-T5 are detailed in the Appendix).
"Our implementation based on Data Augmentation by Back back-translation (DAB). Githuh: https///ithuh..om/vicai//

---

SAMsum
CNNDM
MIMIC
ACC
AUC
TPR
ACC
AUC
p
ACC
AUC
Base
61.10
64.72
1.31
53.48
55.38
0.83
65.41
66.37
2.58
WS
62.11
65.23
1.45
54.69
56.21
0.95
63.33
66.53
2.61
RF
SW
62.51
65.81
1.61
54.51
56.83
1.01
66.44
66.48
2.65
BT
61.07
64.41
1.31
53.30
54.96
0.78
65.22
65.98
2.49
Base
61.15
65.88
1.05
51.24
53.88
0.05
66.73
68.64
2.20
WS
61.23
65,90
1.05
52.25
53.94
0.13
67.01
68.74
2.25
LR
SW
62.03
66.70
1.20
52.58
54.92
0.21
67.59
69.15
2.86
BT
60.14
65.13
1.01
52.13
53.95
0.06
66.78
68.99
2.21
Base
61.67
63.45
2.03
50.30
52.72
0.03
65.90
69.24
2.34
WS
62.26
66.03
2.20
50.45
52.87
0.11
66.93
70.41
2.43
SVM
SW
62.75
66.55
2.41
51.67
53.89
0.13
66.87
70.59
2.52
BT
61.70
65.51
2.05
50.41
52.81
0.05
65.85
68.94
2.27
Base
61.73
65.84
2.15
52.33
55.84
1.17
67.11
70.71
3.05
WS 62.13
66.21
2.51
53.11
56.21
1.25
68.24
71.12
3.15
MLP
SW 62.85
67.00
2.49
53.53
56.26
1.36
68.18
71.33
3.57
BT
62.01
66.81
2.17
52.40
55.91
1.19
67.01
70.69
3.02
Table 3: Document Augmentation Attack Results. Base shows the baseline attack results in Table 1.
SAMsum
CNNDM
MIMIC
ACC
AUC
1P
ACC
AUC
TPR
ACC
AUC
TPRo.1%
RF
57.11
58.24
1.27
51.07
53.09
0.52
60.17
61.25
2.13
LR
57.03
57.85
1.10
50.89
52.83
0.11
57.72
59.44
2.13
SVM
57.05
57.11
1.89
50.41
52.63
0.09
59.15
61.22
1.91
MLP
57.21
57.05
1.97
51.30
53.11
1.07
60.07
60.21
2.67
Table 4: Document only Attack Results based sentence swapping augmentation.
German, and then back translates English. In our main experiments, we generate augmented samples for WS. SW, and for BT.
vious classification models (Shokri et al., 2017). is useless for summarization model. Further- more, we fine-tune a RoBERTa model and use the raw texts to differentiate generated summaries of Bin and Bout referred to as RoBERTa in the table. However, the results indicate that raw text is infe- rior the similarity score features, with the MLP model using similarity scores as features achieving the best performance.
6 Experiment Results
Baseline Attack. We present the results of the baseline attack in Table 1. Our analysis reveals that the attack successful in predicting mem- bership, as the accuracy and AUC results on the three datasets are above 50%. Furthermore, the attack AUC on the MIMIC and SAMsum datasets is above 65%, which highlights a significant pri- vacy risk to Alice's 's model. In Figure 2, we exam- ine the feature distribution of A,n. and Aout Our key observation is that A,n exhibits notably higher ROUGE-1, ROUGE-2, and Transformer Scores than Aours indicating that the model's behavior is distinct on training and non-training samples. Ad- ditionally, our study discovered that the confidence score, which has been found to be useful in pre-
In addition to the AUC and ACC scores, we also evaluate the performance the attacks in the high- confidence regime. Specifically, we report the true positive rate under a low false positive rate of 0. %, referred to TPR in the table. Our results demonstrate that the model samples with high confidence. For example, the MLP model achieves TPR of 3.05% on the MIMIC dataset, which means that the model successfully detects 305 samples in Air with only 10 false positives in Aput-
reliably identify

---

Document Augmentation MI Attack. In this sec- tion, we investigate the effectiveness of evaluating the model's robustness against document modifica- tions as the MI feature. The results of the attack are presented in Table We observe a consis- tent improvement in attack performance across all datasets compared to improvement in TPRo, indicates that the robust signal allows the attacker detect more samples with high confidence. We find that sentence swap- ping achieves the best attack performance across all datasets. Figure, we show the standard deviation distribution of ROUGE-L FI scores, 3, calculated as SD(R-L($, R-L(6 ..., R-L(S s)). We find that the variance of training data is notably lower than non-training data, indicating that train- ing samples more robust against perturbations.
0.1
034
000
0.0
5.0
6.00
Rouge-1
Rouge-l
Rouge-L
Specifically, the
Figure 4: ROUGE-L Standard deviation of g() g(-)p_onty
Impact of Overfitting. In Figure we show the tack AUC and validation ROUGE-L Fl score under varying training steps on the SAMsum dataset. We find that the attack AUC increases steadily as the number of Alice's ss training steps increases, which is consistent with previous research (Shokri et al., 2017). Moreover, early stopping by ROUGE score (5 epochs) cannot alleviate the attack. The AUC curve indicates that the model gets a high attack AUC this checkpoint. A better early stop point is epochs, which significantly reduces the MI at- tack AUC without a substantial performance drop. However, in practice, hard to select proper point without relying on an attack model.
012-
OUR
0.64
Rouge- Recall
Rouge-L L Precision
Rouge-L Score
Figure 3: ROUGE-L Standard deviation of gp
80
Allack AlC
40
70-
ROUIEE- #1
39
Attack AUC
Document-only Attack. In this section, we present the results of our document-only attack. As previously discussed in Section 4.2.1, the at- tack classifier g(-) does not have access reference summaries. Instead, Bob estimates the model's robustness by using generated sum- maries. In Figure 4, we show the standard de- viation distribution of ROUGE-L scores, calcu- lated as SD(R-L(S)) 8), R-L(sg) ..., R-L( $)). Similar to the results in Figure 3, the variance of training data is lower than that of non-training data but with smaller differences. Reflecting results, observe lower attack performance for document-only attacks (Table 4) compared to Base Table 1. However, attack accuracy and AUC are above 50%, indicating a privacy risk even under this low-resource attack. More importantly, the TPR..1% results show that Bob can still infer certain samples' membership with high confidence.
ROUGE
40-
30
30
Epochs
Figure 5: Attack AUC under different epochs.
Impact of Dataset Size. In this study, we assess the impact of dataset size on MI attacks. To do this, we train our model with 10% to 100% of the total dataset. Our results, as depicted in Figure 6, indi- cates that the size of the training set increases, the AUC of MI attacks decreases monotonically for both SAMsum and CNNDM dataset. This suggests that increasing the number of samples in the train- ing set can help to alleviate overfitting and reduce the MI attack AUC. Some recent studies have high- lighted the issue of duplicate training samples in large datasets (Lee et al., 2022). This duplication can escalate the privacy risks associated with these samples and should be taken into consideration when employing large datasets.
the
7 Ablation Studies
In this section, we will investigate several impact factors MI attacks. All experiments were con- ducted using the baseline attack with the MLP clas- sifier. more detailed analysis is in the Appendix.
Impact of the Model Architecture. In previous experiments, we assumed that Bob uses the same

---

We now investigate some approaches that aim to limit the model's ability to memorize its training data. Specifically, we try two approaches: differ- ential privacy SGD (DP-SGD) (Dwork, 2008; Machanavajjhala et al., 2017: Li et al., 2021) and L2 regularization (Song al., 2019). For DP-SGD, is the privacy budget, where lower indicates higher privacy. We conduct experiments on the SAMsum dataset. As shown in Table 5, we find that as the À increase and c decrease, the attack AUC stably drops. Particularly, when e = 8.0 and À = 12.0, the AUC drops to about 50%. However, find defense methods cause notable perfor- mance drop the ROUGE-L F1 score. Indicating that there is privacy-utility trade-off.
SAMsum CNNOK!
50
75
Attack
60-
55-
10%
22%
30%
40%
50%
60%
70%
50%
90%
100%
Figure 6: Attack AUC across different dataset sizes.
architecture as Alice to train shadow models. In this section, we further explore the attack transfer- ability attack across different model architectures. As shown in Figure 7, Bob and Alice can choose different model architectures, we evaluate the trans- ferability metries for various models on SAMsum Dataset, including BART, BertAbs (Liu and Lap- ata, 2019), PEGASUS (Zhang et al., 2020a), and FLAN-T5 (Chung et al.. 2022). The results in- dicate that the attack AUC is highest when both Bob and Alice employ the same model. However, even when Bob and Alice utilize different models, the MI attack exhibits considerable transferability across the selected model architectures. These find- ings suggest that the membership signal exploited by the attack classifier demonstrates generalizabil- ity and effectiveness across various architectures.
DP-SGD
200.0
100.0
8.0
AUC
64.12
54.51
50.46
ROUGE-L
37.21
32.32
27.31
Regularization
0.0
6.0
12.0
AUC
65.84
59.34
52.64
ROUGE-L
37.35
34.32
29.11
Table 5: Defense Performance on DP-SGD and L2 Reg- ularization with different privacy strengths.
Limitations
In this work, we demonstrate that the MI attack is effective. However, it remains unclear what proper- ties make samples more susceptible to MI attack. In other words, given model and dataset, we cannot predict which samples are more likely to be memorized by the model. We find that the detected samples under TPR0.1% have an average shorter reference length, but further research is needed fully answer this question. Additionally, it is impor- tant to note that while the MI attack is a commonly used attack, its privacy leakage is limited. Other attacks pose more significant threat in terms of information leakage (Carlini al., 2021). The evaluation of these attacks in summarization tasks should be prioritized in future studies.
S.I. SUSVSnS .IIIV Bertbbs
Bob Model Architecture
62.77
60.21
Attack AUC
67.31
66.32
59.00
60.03
67.01
BertAbs BART PEGASUS T5
Alice Model Architecture
Figure 7: Architecture Transferability.
10 Conclusion
8
Defense Methods
In this paper, we investigated the membership in- ference attack for the summarization task and ex- plored two attack features: text similarity and data robustness. Experiments show that both features contain fine-grained MI signals. These results veal the potential privacy risk for the summariza- tion model. In the future, we would like to explore advanced defense methods and alleviate the trade- off between privacy and utility.
*Our implementation is hased on dp-transformers. Github: http:///hwhc.ccmmmicccc/oo//o/aa

---

Acknowledgement
Hyung Won Chung, Le Hou, Shayne Longpre. Bar- ret Zoph. Yi Tay. William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. 2022. Scaling cntrructionfineture language models. arXiv preprint arXiv:2210.111 111.1116..
The authors thank the anonymous reviewers for their helpful comments. The work is in part sup- ported by NSF grants NSF CNS-1816497, IIS- 1849085, and IIS-2224843. The views and con- clusions contained in this paper are those of the authors and should not be interpreted as represent- ing any funding agencies.
Oscar Deniz, Anibal Pedraza, Noelia Vallez, Jesus Salido, and Gloria Bueno. 2020. Robustness to ad- versarial examples can be improved with overfitting. International Journal of Machine Learning and Cy- bernetics, ((4)9935-99 994.
Ethics Statement
Cynthia Dwork. 2008. Differential privacy: A survey of results. In International conference on theory and applications of models of computation, pages 1-19. Springer.
In this study, we have utilized publicly available datasets and models to ensure the transparency and reproducibility of our research. We ensure that these datasets do not disclose any private, protected information relating to individuals. We wish to clar- ify that the content encompassed in these datasets does not reflect our personal views or opinions. Our analysis and findings are solely based on the data provided and aim to enhance understanding of language generation models and their security implications.
Wafaa S BI-Kassas, Cherif R Salama, Ahmed A Rafea, and Hoda K Mohamed, 2021. Automatic text sum- marization: A comprehensive survey. Expert Systems with Applications, 165: 165:113679
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Alek- sander Wawer. 2019. Samsum corpus: A human- annotated dialogue dataset for abstractive summa- rization. EMNLP-IJCNLP 2019, page 70.
Jamie Hayes, Luca Melis, George Danezis, and ED Cristofaro. 2017. Logan: Evaluating information leakage of generative models using generative adver- sarial networks. arXiv preprint arXiv: 1et..17055..6...
References
Karl Moritz Hermann, Tomas Kocisky, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. Advances in neural information processing systems, 28.
Asma Ben Abacha, Yassine M'rabet, Yuhao Zhang. Chaitanya Shivade, Curtis Langlotz. and Dina Demner-Fushman. 2021. Overview of the mediqa 2021 shared task on summarization in the medical domain. In Proceedings of the 20th Workshop on Biomedical Language Processing, pages 74-85.
Sorami Hisamoto, Matt Post, and Kevin Duh. 2020. Membership inference attacks on sequence-to- to- sequence models: Is my data in your machine trans- lation system? Transactions of the Association for Computational Linguistics, 8:49-63.
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. 2022. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pages 1897-1914. IEEE
Shengyuan Hu, Tao Yu, Chuan Guo, Wei-Lun Chao, and Kilian Q Weinberger. 2019. A new defense against adversarial images: Turning weakness into a strength. Advances in Neural Information Process- ing Systems, 32.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss. Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, al. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633-2650.
Abhyuday Jagannatha, Bhanu Pratap Singh Rawat, and Hong Yu. 2021. Membership inference attack suscep- tibility of clinical language models. arXiv preprint 1rXiv2104408800
Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong. Binxing Jiao, Daxin Jiang, Haoyi Zhou, Jianxin Li, and Furu Wei. 2022. The-x: Privacy-preservine trans- former inference with homomorphic encryption. In Findings of the Association for Computational Lin- guistics: ACI. 2022, pages 3510-3520.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint 1:X2::4412.6980
Katherine Lec, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch,, and Nicholas Carlini. 2022. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers). pages 8424-8445.
Christopher A Choquette- -Choo, Florian Tramer. Nicholas Carlini, and Nicolas Papernot. 2021. Label- only membership inference attacks. In International conference on muchine learning. pages 1964-1974, PMLR.

---

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and I Luke yeutlemoyer. 2020. Bart: Denoising pruuencc-to-sequemee
Nils Reimers and Iryna Gurevych. 2019, Sentence-bert, Sentence embeddings using siamese bert-networks. arXiv preprint 1908. 10084.
pre-training for nat- ural language generation, translation, and comprehen- sion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881-7880.
Sascha Rothe, Joshua Maynez, and Shashi Narayan. 2021. A thorough evaluation of task-specific pre- training for summarization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 140-145.
Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. 2021. Large language models can be strong differentially private learners. arXiv preprint arXiv:2110.0569.
Abigail See, Peter Liu, and Christopher Manning. 2017. Get to the point: Summarization with pointer- generator networks. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1073- 1083.
Chin- Yew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain.
Association for Computational Linguistics.
Virat Shejwalkar, Huseyin A Inan, Amir Houmansadr, and Robert Sim. 2021. Membership inference attacks against nlp classification models. In NewrIPS 2021 Workshop Privacy in Machine Learning.
Chin-Yew Lin and FJ Och. 2004. Looking for a few good metries: Rouge and its evaluation. In Nteir workshop.
Reza Shokri, Marco Stronati, Congzheng Song, and Vi- taly Shmatikov. 2017. Membership inference attacks against machine learning models. In 2017 IEEE sym- posium security and privacy (SP), pages 3-18. IEEF
Yang Liu and Mirella Lapata. 2019. Text summariza- tion with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Juint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730-3740.
Congzheng Song and Vitaly Shmatikov. 2019. Audit- ing data provenance in text gexeraeneottion models. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, pages 196-206.
Ashwin Machanavajhall, He, and Michael Hay. 2017. Differential privacy in the wild: A tutorial on current practices & open challenges. In Proceed- ings of the 2017 ACM International Conference on Management of Data, pages 1727-1730.
Liwei Song, Reza Shokri, and Prateek Mittal. 2019. Privacy risks of securing machine learning models against adversarial examples. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pages 241-257.
Saced Mahloujifar, Huseyin A Inan, Mclissa Chase, Esha Ghosh, and Marcello Hasegawa. 2021. Mem- bership inference word embedding and beyond. arXiv preprint arXiv:2106. 1 11384.
Casey Mechan, Khalil Mrini, and Kamalika Chaudhuri. 2022. Centence-leve. privacy for document embed- dings. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume Long Papers), pages 3367-3380.
Thomas Tanay and Lewis Griffin. 2016. A boundary tilting persepective on the phenomenon of adversarial examples. arXiv preprint 1608.07600769
Dusan Varis and Ondřej Bojar. 2021. Sequence length is domain: Length-based overfitting in transformer models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pages 8246-8251 -257.
George A Miller 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11) (8111)399-44 41.
Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg-Kirkpatte Kirkpatrick, and Reza Shokri. 2022. Quantifying privacy risks of masked language models using membership inference attacks. arXiv preprint 229909.....
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe- ter Liu. 2020a. Pegasus: Pre- tre-training with extracted gap-sentences for abstractive summarization. In In- ternational Conference on Machine Learning, pages 11328-11339 PMLR
Sasi Kumar Murakonda, Reza Shokri, and George Theodorakopoulos. 2021. Quantifying the privacy risks of learning high-dimensional graphical models. In International Conference on Artificial Intelligecce and Statistics, pages 2287-2295. PMLR.
Yuhao Zhang. Derek Merck, Emily Tsai, Christopher D Manning, and Curtis Langlotz. 2020b. Optimizing the factual correctness of summary: study of summarizing radiology reports. In Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguistics, pages 5108-510 5120.
Natalia Ponomareva, Jasmijn Bastings, and Sergei Vas- silvitskii. 2022. Training text-wo-text transformers with privacy guarantees. In Findings of the Associa- tion for Computational Linguistics: ACL 2022, pages 21 2182-2193.

---

A Experiments on More Models
despite its crucial role in MI attacks within the com- puler vision domain, proves be insignificant in the requence-to-sequee
our primary experiments, our focus on the BART model; however, we are also interested in exploring other commonly used models. Among these models, we consider the Fan-T5 base (Chung et al., 2022), where we maintain the same experi- mental setup but replace the Alice and Bob model with Flan-T5. Table presents the baseline attack results, which are consistent with findings in the BART model. Our analysis demonstrates the successful prediction of membership the Flan- T5 model, as evidenced by the accuracy and AUC results across all three datasets exceeding 50%. Furthermore, the attack AUC for the MIMIC and SAMsum datasets exceeds 66%, highlighting sig- nificant privacy risk Alice's model.
model. This could be attributed to the beam search process, which in- variably samples sentences with high confidence, thereby rendering this feature redundant.
1ouge2-12
10ugo2
Transformer Score
rougel-12
mupetlpreecsion
rugett
--ecall
rouget precaiion
Confiendee score
Additionally, Table showcases the results of the Document augmentation attack, revealing a con- sistent improvement in attack performance across all datasets compared to g(-) Notably, the enhancement in TPRo.1% indicates that the robust signal enables the attacker to detect more samples with high confidence. Our findings indicate that sentence swapping yields the most effective attack performance across all datasets and metrics.
Figure 8: Feature Importance in MI attack on BART model.
Huuge210
rougeD-vecal
rougel-12
Transfomer
Morcover, in Table 9, we observe lower at- tack performance for document-only attacks com- pared to 9 in Table 7. Nevertheless, the attack accuracy and AUC remain above 50%, signi- fying privacy risk even in the context of this low- resource attack. Most importantly, the TPR results demonstrate that Bob can still infer the mem- bership of certain samples with high confidence.
muugt2pprecsion
tougel-12
nouget-rocall
rougel-recal
muppet-greeiiion
Confiendee score
0.10
0.1
0.70
Figure 9: Feature Importance in MI attack on FL.AN-TS model.
To conclude, our findings are consistent across both the Flan-T5 and BART models, indicating that these summarization models have the ability memorize training data and pose valid threat leaking membership information.
C More on Ablation Studies
In this section, will add more ablation studies. Firstly, will study the impact of overfitting, and dataset size on the FLAN-T5 dataset. Then we will introduce the impact of query numbers. All ex- periments were conducted with the baseline attack method, employing the MLP classifier.
B Feature Importance
In this section, we delve into the feature impor- tance of the baseline MI attack, specifically target- ing the SAMsum dataset. Figure and 9 displays the feature importance 4 determined by the Random Forest classifier for the baseline attack. The ROUGE-2 score emerges as the most valu- able feature. On the contrary, the confidence score,
Impact of Overfitting. We study the impact of overfitting in MI attacks on the FLAN-T5 model. Figure 10 shows the attack AUC and validation ROUGE-L Fl score under varying training steps on the SAMsum dataset. We observe that the attack AUC increases steadily as the number of Alice's s training steps increases. Early stopping by ROUGE
"The importance score based on the scikit-lcam package: https://scike http///iiiittttt leaae oaggn..or////eee org/stable, e/auto, pauut__eemmplleee examples/ ensemble/plot_fg forest. forest_impor importances. tances. html

---

score (4 epochs) cannot alleviate the attack.
number necessitates more queries the API, con- sequently escalating the attack cost. Additionally, we attempted to combine data from the two aug- mentation methods, with the results documented in the 'Comb' row. Interestingly, merging data from the two augmentation methods did not further enhance the attack performance.
30
40
Attack AUC ROUSE-L
70
39
Attack AUC
80
38
ROUGE
50
37
40
36
BART
12
24
30
35
2
4
6
8
WS
66.21
66.34
66.39
Epochs
SW
67.00
67.14
67.31
Comb
66.65
66.94
67.13
Figure 10: Attack AUC under different epoches.
FLAN-T5
Impact of Dataset Size. In this study, we assess the impact of dataset size MI attacks the FLAN-T5 model. Specifically, train the model with 10% to 100% of the total dataset. As de- picted Figure 11, we found that as the size of the training set increases, the AUC of MI attacks decreases monotonically for both SAMsum and CNNDM datasets. This outcome aligns with our observations from the BART model and suggests that increasing the number of samples in the train- ing set can help to alleviate overfitting.
12
24
WS
67.01
67.19
67.35
SW
68.22
68.29
68.37
Comb
67.27
67.33
67.84
Table 6: Different Augmentation Number.
20
43
20
BART
FLAN-TS
Attack
Figure 12: Architecture Transferability.
65
60-
D More on Architecture Transferability
10%
20%6
30%
406
50%
50%
70%
60%
90%
100%
As discussed in Section 7, when Alice and Bob em- ploy different architectures, the MI attack exhibits robust architecture transferability. This implies that even when the attacker's classifier is trained on a shadow model distinct from Alice's 's model, the learned attack signal is generalizable across differ- ent architectures. In this section, we aim to explain this transferability. Our key observation is that the MI signal remains remarkably consistent, even across different architectures. As demonstrated in Figure 12, we exhibit the distribution of the ROUGE-L Fl score in Alice's model, using both the BART and FLAN-T5 architectures. The ob- servation reveals a consistent pattern: the ROUGE score for the training data is notably higher than that for non-training data. This trend persists across both architectures. which lends insight into the at- tack's transferability.
Figure 11: Attack AUC across various dataset sizes.
Impact of Augmentation Numbers. In our pri- mary experiments involving document augmenta- tion for MI attacks, generated augmentations for word synonym (WS) and sentence swapping (SW), and for back translation (BT). In this sec- tion, we extend our investigation to the impact of augmentation quantity, focusing on WS and SW, as they exhibit a significant increase over the baseline attack. Table 6 presents the attack AUC for aug- mentations of 6, 12, and 24 on the SAMsum dataset. Notably, we observe slight improvement in attack performance with increased number of augmen- tation samples, which aligns with the notion that more augmented data can help the classifier better evaluate the sample's ' robustness. However, it's im- portant to acknowledge that a higher augmentation

---

SAMsum
CNNDM
MIMIC
ACC
AUC
TPRo. 1%
ACC
AUC
TPRo.19
ACC
AUC
RF
61.87
65.01
1.22
53.01
56.99
0.81
66.23
67.11
2.63
LR
62.43
66.15
1.11
52.13
55.01
0.11
67.32
69.51
2.30
SVM
63.14
67.00
1.95
52.21
54.41
0.06
66.41
68.71
2.13
MLP
63.10
67.01
2.13
53.14
56.32
1.17
67.91
72.05
3.18
RoBERTa
60.45
64.33
1.27
50.48
53.28
0.75
65.71
69.83
2.37
Table FLAN-TS Baseline Attack Results. Following evaluation protocol Section 3.3, we show Base performance Aen and Aout.
SAMsum
CNNDM
MIMIC
ACC
AUC
pPR
ACC
AUC
TPR
ACC
AUC
TPR
Base
61.87
65.01
1.22
53.01
56.99
0.81
66.23
67.11
2.63
WS
62.43
66.55
1.24
55.21
56.52
0.77
68.03
68.03
2.71
RF
SW
63.00
66.84
1.66
55.11
56.25
1.13
67.91
68.17
3.31
BT
61.22
65.18
1.20
55.13
56.77
0.95
66.37
67.00
2.43
Base
62.43
66.15
1.11
52.13
55.01
0.11
67.32
69.51
2.30
WS
61.53
66.21
1.18
53.45
55.31
0.20
68.13
69.07
2.43
LR
SW
63.35
67.99
1.28
54.00
55.27
0.33
68.55
71.35
2.83
BT
61.43
66.01
1.13
53.01
54.22
0.19
68.13
70.59
3.05
Base
63.14
67.00
1.95
52.21
54.41
0.06
66.41
68.71
2.13
WS
63.54
66.35
2.24
52.77
55.45
0.19
67.10
69.58
2.33
SVM
SW
63.17
67.12
2.12
52.59
55.32
0.27
68.15
70.83
3.01
BT
62.15
67.08
2.14
50.75
54.55
0.13
65.85
68.11
1.99
Base
63.10
67.01
2.13
53.14
56.32
1.17
67.91
72.05
3.18
WS
62.22
67.01
2.77
54.34
57.12
1.21
68.56
71.22
4.01
MLP
SW
63.15
68.22
3.71
54.99
58.18
2.04
68.33
73.55
3.88
BT
62.22
67.21
2.20
53.74
57.73
1.56
67.13
72.14
3.40
Table Document Augmentation Attack Results on FLAN-TS. Base shows baseline attack results Table
SAMsum
CNNDM
MIMIC
ACC
AUC
TPRo.1%
ACC
AUC
TPRo.1%
ACC
AUC
TPRp
RF
56.30
57.74
1.17
50.99
54.08
0.57
60.55
61.65
2.22
LR
57.31
59.88
1.05
51.22
53.76
0.39
55.01
58.83
2.26
SVM
57.15
58.86
1.85
51.03
54.07
0.12
59.33
62.25
1.91
MLP
57.60
57.35
2.08
52.73
54.49
1.21
61.15
62.77
2.77
Table 9: Document only Attack Results based on sentence swapping augmentation on FLAN-TS. Base shows the best baseline attack results in Table with full knowledge.


Reviews:
[{"review_id": "azLHBSj30P", "reviewer": "Reviewer_Eh5X", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** The author(s) employ BART-base and Flan-T5 models for summarization. Perhaps they could use more powerful models.\n\n**R1:** Thank you for your constructive suggestion.\n\n- We selected Flan-T5 and BART-base models as **they are commonly employed as foundational models in summarization research and demonstrate commendable performance.** Thus, we consider these models to be representative when assessing the membership inference attack risk for summarization tasks.\n- According to your suggestion, we tested our method on the recent large language model Vicuna-7B [1], recognized as one of the leading open-source models. We fine-tuned Vicuna on the Samsum and CNNDM datasets and applied the baseline and document augmentation attacks, utilizing MLP as the classifier. For augmentation, we selected the sentence swap method, which proved the most effective for membership inference attacks in our study. **As Table 1 depicts, Vicuna-7B is vulnerable to membership inference attacks,** evidenced by an attack AUC greater than 50%. Additionally, the TPR_0.1\\% results indicate that the attack can identify specific training data with high accuracy and confidence. We will add these results in the Appendix.\n\n**Table 1: MI Attack Performance on Vicuna-7B Model**\n|MI Attack $\\downarrow$| Dataset $\\downarrow$|ACC|AUC|TPR_0.1\\%|\n|:---:|:---:|:---:|:---:|:---:|\n|Baseline Attack| SAMsum | 61.87 | 65.43 | 1.23 |\n|| CNNDM| 52.15 | 56.86| 0.29|\n|Document Augmentation Attack | SAMsum |62.17 | 65.79 |1.50 |\n||CNNDM | 53.01 | 57.85 | 0.81 |\n\n**Q2:** They tried three augmentation strategies: Word Synonym, Sentence swapping, and Back Translation. However, these strategies seem similar to each other. Perhaps they could have employed slightly different strategies, such as attempting to summarize from overlapping segments of the input document, etc.\n\n**R2:** Thanks for the insightful comment. \n- We have chosen three distinct augmentation strategies, **each offering a unique perspective on linguistic perturbation: lexical, structural, and contextual.** The Word Synonym technique targets lexical variations, Sentence Swapping addresses the structural sequencing of information, while Back Translation introduces nuanced phrasing differences. Collectively, we believe these methods encapsulate the majority of perturbations relevant to input documents. \n- Although there are other potential augmentation strategies, including summarizing from overlapping document segments, **it may require a different training approach and modification of the inference strategy.** Our current focus is on document perturbations, as these are most practical for countering real-world black-box attacks. However, we appreciate the suggestion and are open to exploring alternative strategies in future research.\n\n\n**Q3:** There are some typos in the paper which can be improved. For example, line 87 contains the word ‘lunch’ where as it should have been ‘launch’.\n\n**R3:** Thank the reviewer for pointing out the typos. We have refined this typo in our updated version and will thoroughly review the paper.\n\n**Q4:** This paper presents an approach to denoise weakly supervised labels. They focus on enhancing the quality of weak labels using k-fold cross-validation.\n\n**R4:** Thank you for taking the time to review our paper. We appreciate your comments, but it seems there may be a misunderstanding. The focus of our work is not on denoising weakly supervised labels using k-fold cross-validation. We wonder if this particular feedback might have been intended for another paper. We would be happy to address any concerns that are directly related to our research.\n\n\n**References**\n\n[1] Chiang, Wei-Lin, et al. \"Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.\" See https://vicuna. lmsys. org (accessed 14 April 2023) (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Novel task formulation \n\n- Extensive experimentation and ablation studies highlighting the impact of various factors \n\n- Variety of datasets employed in the study.  \n\n- Propose plausible defense mechanisms.", "reasons_to_reject": "- Not the most up to date models employed.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents approaches to determine Membership Inference (MI) in the context of text summarization. MI is a form of attack on LLMs to determine the risk of exposing training data. They present studies in two scenarios: (a) when the attacker has access to the reference summary using one or more API calls and (b) when the attacker has access to only the model and not the reference summary. They show that current summarization models are at risk of exposing data membership even in cases where the reference summary is not available. \n\nStrengths:  \nThey perform extensive experiments across three datasets to determine MI attack in summarization models. The results from the paper seem generalizable. \nTheir ablation studies highlight the impact of overfitting, data size and model architcture.  \nThey also discuss the plausible defense strategies against such attacks.  \nThe paper is written clearly and is easy to comprehend.  \n \nWeaknesses:  \nThey employ BART-Base and Flan-T5 models for summarization. Perhaps they could use more powerful models.  \n(not really a weakness?) They tried three augmentation strategies: Word Synonym, Sentence swapping and Back Translation. However, these strategies seem similar to each other. Perhaps they could have employed slightly different strategies such as attempting to summarize from overlapping segments of the input document, etc.  \nThere are some typos in the paper which can be improved. For example, line 87 contains the word ‘lunch’ where as it should have been ‘launch’", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xhK6IpC2Re", "reviewer": "Reviewer_NwCg", "summary": "", "questions": "(A)\tAre there any reasons that this paper focuses on summarization compared with previous work focusing on MT translation? The reviewer thinks this paper’s method can be applied to MT.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** The reason why this paper focuses on summarization is not presented. Previous work focused on Machine Translation (MT).\n\n**R1:** Thank you for your insightful comments. \n\n- In the introduction section, we emphasized our focus on the summarization task for two primary reasons: **1) Summarization stands as a representative task for generative models**. This makes it a crucial domain to investigate, especially given the increasing adoption of such models in various applications. and **2) Given the widespread deployment of summarization models in critical sectors like healthcare**, there exists a potential risk of privacy breaches. Documents in such domains often contain Personally Identifiable Information (PII). Despite the potential for privacy risks, there is a lack of prior research specifically addressing these concerns in the context of summarization models.\n- The proposed membership inference attack methods and threat models are transferrable to many other sequence-to-sequence tasks, such as machine translation and generative question answering. However, adapting them requires the exploration and analysis of domain-specific MI features. We believe that the incorporation of additional domains within this study might overextend the scope of the paper, making it overly lengthy and potentially overwhelming readers. We plan to explore other domains in our future research.\n\n**Q2:** The reviewer thinks description of more specific applications is appreciated by readers. What situation is this paper’s method used for?\n\n**R2:** Thank you for the constructive suggestion.\n- As we emphasized in the paper, membership inference attacks are standard tests for privacy vulnerabilities. **From the attacker's perspective**, these attacks can identify whether a specific document is used for training the summarization model. **Conversely, for model developers,** the proposed attacks act as evaluative tools, allowing stakeholders to assess the potential privacy risks associated with their models. \n- As a concrete example, consider the hospital environment where summarization models are often employed to distill extensive patient histories and reports into concise summaries for efficient review by healthcare practitioners [1, 2]. However, there exists a tangible risk: **a summarization model trained on a comprehensive dataset of patient records might inadvertently remember and leak information about patients,** potentially violating regulations like HIPAA and GDPR. Our method helps in detecting whether such a summarization model can inadvertently leak specific details from its training data. **By evaluating these risks beforehand, appropriate privacy defense mechanisms can be implemented prior to model deployment.**\n\n**References**\\\n[1] Krishna, Kundan, et al. \"Generating SOAP notes from doctor-patient conversations using modular summarization techniques.\" ACL 2020.\\\n[2] Jain, Raghav, et al. \"A survey on medical document summarization.\" arXiv preprint arXiv 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Tackling a problem of a MI attack for summarization\n- Proposal of input document only MI attack for summarization using input documents modified with word synonym replacement, sentence swapping and back translation.\n", "reasons_to_reject": "-\tThe reason why this paper focuses on summarization is not presented. Previous work focused on Machine Translation (MT).", "questions_to_authors": "(A)\tAre there any reasons that this paper focuses on summarization compared with previous work focusing on MT translation? The reviewer thinks this paper’s method can be applied to MT.", "paper_topic_and_main_contributions": "This paper tackles a concern that LLMs may disclose information in the training data. This paper focuses on the summarization task and investigate the membership inference (MI) attack. In this setting, this paper uses a sample and black-box access to a model to determine if the sample was part of the training data. This paper exploits text similarity between an output and the reference summary of its input text and the stability of model outputs to input text as modified with operations such as word synonym replacement and sentence swapping. Evaluation of their effectiveness on SAMsum, CNN and Daily Mail, and MIMIC  demonstrates that summarization models are at risk of exposing data membership, even in cases where the reference summary is not available. Furthermore, this paper also discusses safeguards for training summarization models to protect against MI attacks.\n\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "\n-\tThe reviewer thinks description of more specific applications is appreciated by readers. What situation is this paper’s method used for? For example, Hisamoto et al., 2020 described them in their paper for the MT case.\n"}}, {"review_id": "Dvb8ChBS58", "reviewer": "Reviewer_9rho", "summary": "", "questions": "1) Is it possible to extend the authors' method to perform membership inference attacks on autoregressive generative language models? I'm intrigued to explore whether some publicly available models, displaying impressive capabilities, were trained on legally questionable data sets, such as non-permissive codebases. \n\n2) Have the authors conducted experiments in this direction? If there is existing research pertaining to membership inference attacks on generative models, please consider adding it to the related work section.\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** At times the main inference was hard to follow and justify logical jumps, however, the abundance of figures/tables covered those minor discrepancies.\n\n**R1:** Thank you for your feedback. We will enhance the narrative clarity and ensure it aligns well with the figures and tables for better understanding.\n\n**Q2:** Is it possible to extend the authors' method to perform membership inference attacks on autoregressive generative language models? I'm intrigued to explore whether some publicly available models, displaying impressive capabilities, were trained on legally questionable data sets, such as non-permissive codebases.\n\n**R2:** Thank you for your insightful suggestions. \n- Indeed, the proposed attack can be applied to autoregressive generative language models. Based on your suggestion, we tested our method on Vicuna-7B [1], which is recognized as one of the leading open-source autoregressive generative language models. We fine-tuned Vicuna on the Samsum and CNNDM datasets and applied the baseline and document augmentation attacks, utilizing MLP as the classifier. For augmentation, we selected the sentence swap method, which proved the most effective for membership inference attacks in our study. **As Table 1 depicts, Vicuna-7B is vulnerable to membership inference attacks,** evidenced by an attack AUC significantly greater than 50%. Additionally, the TPR_0.1\\% results indicate that the attack can identify specific training data with high accuracy and confidence.\n- We agree with you that **membership inference attacks have the potential to help identify if specific protected content has been utilized in pretraining/fine-tuning generative models,** such as non-permissive codebases and copyrighted literature. Using membership inference attacks to safeguard intellectual property presents a compelling research opportunity, and we plan to delve deeper into it in our upcoming work.\n\n\n**Table 1: MI Attack Performance on Vicuna-7B Model**\n|MI Attack $\\downarrow$| Dataset $\\downarrow$|ACC|AUC|TPR_0.1\\%|\n|:---:|:---:|:---:|:---:|:---:|\n|Baseline Attack| SAMsum | 61.87 | 65.43 | 1.23 |\n|| CNNDM| 52.15 | 56.86| 0.29|\n|Document Augmentation Attack | SAMsum |62.17 | 65.79 |1.50 |\n||CNNDM | 53.01 | 57.85 | 0.81 |\n\n**Q3:** Have the authors conducted experiments in this direction? If there is existing research pertaining to membership inference attacks on generative models, please consider adding it to the related work section.\n\n**R3:** Thank you for your insights. \n- To the best of our knowledge, **limited research exists specifically on membership inference attacks for text generative models.** We have referenced two prior studies that delve into this subject. However, these studies either require both data and its corresponding label for membership inference or demand white-box access to the model. Additionally, both of them focus on relatively small models, and membership inference attack on recent large generative models is under-studied. \n-  There are also studies that focus on membership inference attacks in different domains, such as image generation [2] and text-image generation [3]. We appreciate the suggestion and plan to include these references in our updated paper.\n\n**References**\\\n[1] Chiang, Wei-Lin, et al. \"Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.\" 2023.\\\n[2] Chen, Dingfan, et al. \"Gan-leaks: A taxonomy of membership inference attacks against generative models.\" CCS 2020.\\\n[3] Wu, Yixin, et al. \"Membership inference attacks against text-to-image generation models.\" arxiv 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors present a simple but effective method for MI attacks that works specifically with summarization. Unlike previous methods that focused on classification tasks, this approach doesn't need scores or probabilities and can work with just black-box API access. The method can even be used without the summarized document, making it very useful for real-world situations.\n\nThe paper also includes extra tests and analysis, such as how well the method can be transferred to different setups. The authors have thought about how to protect against these kinds of attacks and the balance between keeping data private and making it useful. Overall, this is a solid paper that adds valuable insights to the field of natural language processing and information security. It's a good read for anyone interested in these subjects.", "reasons_to_reject": "At times the main inference was hard to follow and justify logical jumps, however, the abundance of figures/tables covered those minor discrepancies.", "questions_to_authors": "1) Is it possible to extend the authors' method to perform membership inference attacks on autoregressive generative language models? I'm intrigued to explore whether some publicly available models, displaying impressive capabilities, were trained on legally questionable data sets, such as non-permissive codebases. \n\n2) Have the authors conducted experiments in this direction? If there is existing research pertaining to membership inference attacks on generative models, please consider adding it to the related work section.\n", "paper_topic_and_main_contributions": "This paper takes a close look at membership inference (MI) attacks for large language models, focusing on summarization tasks. It's well-written, easy to read, and the definitions and notations are clear. The figures in the paper are well-designed, adding to the understanding of the topic.\n\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an interesting, basic and effective method to identify whether models enclose information from training data. The method can work on a blackbox model and provides extensive experiments on multiple dataset. 

Apart from minor improvements in the writing, reviewers point out that the method is tested on small models and it could be the case that it works less well for larger ones, so this is a nice first step, and that the choice of summarisation where most other papers investigate machine learning, it is harder to compare to other methods. 

Overall, the paper presents a nice first step that can be explored further (does it also work with larger models, how does it compare to other methods when applied to ML) in the future.
Decision: Accept-Findings