Membership Leakage in Pre-trained Language Models
Authors: Yuan Xin
Zheng Li
Ning Yu
Michael Backes
Yang Zhang
Venue: Submitted to ICLR 2023 (2023)
Keywords: membership leakage
pre-trained language models
natural language processing

Abstract: Pre-trained language models are becoming a dominating component in NLP domain and have achieved state-of-the-art in various downstream tasks. Recent research has shown that language models are vulnerable to privacy leakage of their training data, such as text extraction and membership leakage. However, existing works against NLP applications mainly focus on the privacy leakage of text generation and downstream classification, and the privacy leakage of pre-trained language models is largely unexplored. In this paper, we take the first step toward systematically auditing the privacy risks of pre-trained language models through the lens of membership leakage. In particular, we focus on membership leakage of pre-training data in the exposure of downstream models adapted from pre-trained language models. We conduct extensive experiments on a variety of pre-trained model architectures and different types of downstream tasks. Our empirical evaluations demonstrate that membership leakage of pre-trained language models exists even when only the downstream model output is exposed, thereby posing a more severe risk than previously thought. We further conduct sophisticated ablation studies to analyze the relationship between membership leakage of pre-trained models and the characteristic of downstream tasks, which can guide developers or researchers to be vigilant about the vulnerability of pre-trained language models. Lastly, we explore possible defenses against membership leakage of PLMs and propose two promising defenses based on empirical evaluations.

Under review as a conference paper at ICLR 2023
MEMBERSHIP LEAKAGE IN PRE-TRAINED LANGUAGE MODELS
Anonymous authors Paper under double-bling review
ABSTRACT
Pre-trained language models are becoming dominating component in NLP do- main and have achieved state-of-the-an various downstream tasks. Recent re- search has shown that language models are vulnerable to privacy leakage of their training data, such as text extraction and membership leakage. However, exist- ing works against NLP applications mainly focus on the privacy leakage of text generation and downstream classification, and the privacy leakage of pre-trained language models is largely unexplored. In this paper, we take the first step toward systematically auditing the privacy risks of pre-trained language models through the lens of membership leakage. In particular, we focus membership leakage of pre-training data the exposure of downstream models adapted from pre-traines language models. We conduct extensive experiments on variety of pre-trained model architectures and different types of downstream tasks. Our empirical evalu- ations demonstrate that membership leakage of pre-trained language models exists even when only the downstream model output is exposed, thereby posing more severe risk than previously thought. We further conduct sophisticated ablation studies to analyze the relationship between membership leakage of pre-trained models and the characteristic downstream tasks, which can guide developers or researchers to be vigilant about the vulnerability of pre-trained language models. Lastly, we explore possible defenses against membership leakage of PLMs and propose two promising defenses based on empirical evaluations.
INTRODUCTION
Nowadays, pre-trained language models (PLMs), represented by BERT (Devlin et al. 2019), have revolutionized the natural language processing community (Wolf et al. 2019 Vaswani et al. 2017 Munikar et al. 2019). PLMs are typically pre-trained on large-scale corpora to learn some universal linguistic representations and are then fine-tuned for downstream domain-speciff tasks (Sun et al.) 2019 Shen et al.) 2021). Concretely, downstream model owners add only few task-specific layers on top of the PTMs to adapt to their own tasks, such as text classification, named entity recognition (NER), and Q&A. This training paradigm not only avoid training new models from scratch, but also form the basis of sfate-of-the-as results across NLP.
Despite its novel advantage in adapting downstream tasks, PLMs are essentially DNN models. Re- cent studies (Erkin et al.) 2009) Liu et al.] 2022 Choo et al. 2021: Li et al.) g shown that machine learning models (e.g., image classifiers) are vulnerable privacy attacks, such tribute and membership inference attacks. Yet, existing privacy attacks against language models have mainly focused on text generation and downstream text classification (Song & Shmatikov] 2019 Shejwalkar et al. (021). To our knowledge, the potential privacy risks of pre-training data for PLMs have never been explored.
To fill this gap, we take the first step towards systematically audit the privacy risks of PLMs through the lens of membership inference: An adversary aims to infer whether a data sample is part of PLMs' training data. In particular, given the realistic and common scenario that downstream service providers are more likely to build models adapted from PLMs, we consider that the adversary access only these downstream service models deployed online. Here, PLMs allow adding any task sseciici layers to fit any type of downstream task, such as classification (Shejwalkar et al., 2021). NER (Mc-
Callum & Li 2003), and Q&A (Bordes al. 2014). We further consider another realistic scenario

---

Under review as a conference paper at ICLR 2023
where no additional information about the target PLMs is available to the adversary other than the output, ie., black-box setting. We perform an extensive measurement study of membership infer- ence, jointly, over four different PLMs architectures (BERT.LBBERT. XLNet) and five different downstream datasets that refers to three downstream tasks. Our evaluations show mem- bership leakage of pre-training data exists even when only the output of the downstream model is exposed, regardless of the PLMs architecture and downstream tasks,and thereby pose more severe risk than previously thought. We further analyze the relationship between membership leakage and the characteristic of downstream tasks. We also conduct sophisticated ablation studies guideline in- ventors to be vigilant about the vulnerability of pretrained NLP models Lastly, we explore possible defenses that can prevent membership leakage of PLMs and propose two promising defenses based on empirical evaluation results.
Contributions.
We pioneer to conduct the first investigation on membership leakage of PLMs' pre-training data with only downstream model output exposed.
We conduct extensive experiments of downstream tasks. Our empirical evaluations demonstrate that membership leakage of PLMs exists even when only the downstream model is exposed.
variety of PLMs architectures and different types
We conduct sophisticated ablation studies to analyze the relationship between membership leakage and the characteristic of downstream tasks.
We explore possible defenses against membership leakage of PLMs and propose two promising defenses.
2 BACKGROUND/ / RELATED WORK
2.1 PRE-TRAINED LANGUAGE MODELS
Pre-trained Encoder. Nowadays, large-scale pre-trained language models (PLMs) are pushing
natural language processing to new era. They typically pre-trained large corpora by self- supervised learning and fine-tuned for different types of downstream tasks. The first generation of masked language modeling
PLMs was BERT Devlin al. (2019), whose pre-training tasks (MLM) and next sentence prediction (NSP). Since then, many variants of BERT have emerged to improve the learning ability of language models, such as ALBERT (Lan et al. 2020). RoBERTa (Liu et al. 2019), XLNet (Yang et al. 2019).
Downstream Tasks. PLM trained on a large corpus can learn generic linguistic representations, which can benefit in wide range of downstream tasks such as text classification. named entity recognition (NER), and question and answering (Q&A). PLMs unify these tasks into common pre-training and fine-tuming pipeline and achieves superior performance across them (Gururangan let al.) 2020 Qiu al. 2020), and thereby the pre-training and fine-tuning pipeline has become the most widely applied downstream model construction paradigm.
PLMs, despite their high level of generic linguistic representations, also rely on large-scale corpora that contains private/sensitive pre-training data, such as phone numbers, addresses, and biomedical data (in BioBERT (Lee et al.. 2020)). Therefore, the vulnerability of PLMs to privacy leakage de- serves attention, as well proactive assessment. Further, this is also important in light of the latest regulations under the EU General Data Protection Regulation (GDPR) umbrelle which re- quire data owners to have greater control their data. In addition, since PLMMs tasks, this means that downstream models adapted from PLMS are actually more common in the real world. Therefore, these concerns and realities drive attention to the privacy leakage that exists in PLMS and that only the downstream models on them are exposed.
downstream
2.2 MEMBERSHIP INFERENCE ATTACKS
Membership inference attack is kind of data inference attack which aims to infer whether the data sample was used to train the target machine learning model (Hu et al.) 2021 Carlini et al. 2021
https://gdpr-infott
2

---

Under review as a conference paper at ICLR 2023
Ye et al.] 2021). The core idea of membership inference is to leverage overfitting to conduct the attack. Usually machine learning models will overfit on its training data, so when querying the
well-trained machine learning model with both member and non-member data, the model will have
different behaviors, which could be leveraged to differentiate between member and non-members (Li
& Zhang 202T He et al. 2022 Li al. 2022 Carlini et al. 2021 Rahimian et al.
2020
He al.
2022).
Existing attacks have been extensively studied in computer vision (Salem et al. 2019 He et al.
2022 Ye et al., 2021; Humphries et al.] 2020; Liu et al., [02T]] Yang et al. (020), while only a
al.) 202T Mireshghallah et al. [022] Jagannatha et al. [202]). Especially in the NLP domain, existing work has focused on a specific downstream task, with Song & Shmatikov (2019) working in sequence models and (Shejwalkar et al. (202T) working in text classification models. Mireshghallah et al. (202); Jagannatha et al. (2021) work on masked language models but they can directly get access to target encoder's s output, this assumption is not realistic. Further, all existing works can access models that are exactly trained or fine-tuned for the data they aim to infer without any inconsistency. In contrast, consider more realistic and challenging scenario where do not have direct access to the original PLMs that is purely applicable to the pre-training data we aim to infer, i.c., there is actually an inconsistency between the PLMs now and the final output we can access. Second, this inconsistency is further amplified by the fact that the fine-tuning process will modify the parameters of the PLMs fit the new downstream task. These inconsistencies make it more difficult to evaluate whether model is truly vulnerable to membership inference or not, which may lead to premature claims of privacy for PLMs.
few works have been done in NLP domain (Song & Shmmatikoy 2019; Shejwalkar
3 PROBLEM STATEMENT
3.1 PRE-TRAINED LANGUAGE MODELS
Here, we formulate pre-trained language model. A pre-trained language model (PLM) that trained on large-scale pre-training dataset D can map textual inputs to embeddings. The downstream service provider then adds several layers The model M is then fine-tuned to fit their own downstream dataset by minimizing a predefined loss function using some optimization algorithms. Finally, the downstream model M can map textual inputs / to new types of outputs, such as posteriors for text classification or embeddings for Q&A.
top of the PLM to build new downstream model M.
3.2 THREAT MODEL.
Attacker's Goal. The goal of the attacker is make binary prediction of whether the given textual input is used to train the target pre-trained language model Munger. A textual input is called a member (or non-member) of the Marrger if it is in (or not in) the target pre-training data D Durrer--
Attacker's Background Knowledge. Typically, the background knowledge in membership infer- ence domain is along two dimensions: the architecture of target PLMs and the distribution of target pre-training dataset Dunget. In the first dimension, we assume that the attacker has no knowledge of the architecture of the target PLM and has only black-box access to the downstream model Manzer adapted from the target PLM. The is the most realistic scenario as service providers typically share task-specific models adapted from PLMs to the public as "machine learning as a service", leading the attackers to access the output of the downstream model Maarger directly rather than the output of the target PLMS to launch membership inferences.
Besides, we assume that the attackers has very small subset from the member training data of pre- training data Drrget. as well as a small set of local non- non-member data Deon member The adversary then local
xm E Dimember xmm E Dioa momber Ithat constructs an auxiliary dataset D Dausittary = {x.T U nm can be used to train the attack model A. Note that, the assumption of auxiliary dataset also holds for previous works (Liu et al.) (002). In addition, since the PLMs are usually trained on billions of pre-training data, it is very likely that an attacker could collect a very small fraction of such the large-scale pre-training data, which is also more reasonable and efficient than collecting another billions of local corpora to build a local shadow model. Last but not least, attacks on real-world models are also more realistic than existing works in laboratory settings.
3

---

Under review as a conference paper at ICLR 2023
4 METHODOLOGY
4.1 INTUITION
The key intuition of our work a general observation of PLMs memorization on pre-training data. Given that PLMs can map textual inputs t to embedding vectors for the benefit of downstream tasks, PLMs are typically trained deeply on pretrained data Diorer to learn general language repre- sentations, i.e., PLMs memorize pre-training data deeply. By further analogy with the fact that the confidence scores of members in image classification are usually larger than non me[b this yields the following key intuition about membership information:
Members and non-members of PL.Ms behave differently on embeddings.
Due to deep memorization, such behavioral differences on embedding vectors will be re- tained in the downstream model output.
Thus, the retained differences reflected in the downstream model output (although more likely not as much as the embedding vectors directly output by the PI.Ms) to distinguish between members and non-members of the PLMs.
4.2 ATTACK METHODOLOGG
Here, propose the methodology to evaluate the membership leakage of PLMs.
Auxiliary Dataset Construction. As aforementioned, we assume that an attacker can collect very small subset contained in the pre-training data Diarger i.e., the members of PLMs. The attacker then constructs an auxiliary dataset by combining all these collected datasets
Attack Training Dataset Construction. The attacker feeds all samples of the auxiliary dataset to the downstream model and records all outputs. The attacker then constructs the training dataset for the attack model by labeling these recorded outputs members
non-members, respectively.
Attack Model Training. The attacker build an attack model, i.e.. a binary classifier, and then train it with the auxiliary dataset in conjunction with classical training techniques.
Membership Inference. Finally, the attacker can feed a candidate text input into a well-trained attack model, and then the attacker determines that the candidate text input is member member based on the predictions of the attack model.
5 EVALUATIONS
5.1 EXPERIMENTAL SETUP
PLM Architectures and Pre-training Datasets. We consider the four recent state-of-the-art PLMs: BERT (Devlin et al.) 2019). ALBERT (Lan et al. 2020), RoBERTa (Liu et al. 2019). XL- Net YYang et al., (019).. These PLMs are constructed from different loss functions and training schemes well architectures. Note that due the resource limitation, ie., insufficient GPU resource, we are unable to train our PLMs on billions of corpora. Thus, we use well-trained PLMs publicly available online as attack target Actually, this is realistic than almost all the existing works only in the laboratory environment.
Each PLM application benchmarks its huge amount of pre-training data. Here, we emphasize that two datasets, i.c., Wikipedia and BooksCorpus (Devlin et al. (2119), are the most popular and used across all these four PLMs, and consider them members our work.
Downstream Models and Fine-tuning Datasets. As mentioned earlier, we considered the most re- alistic and challenging scenario where we can only access the output of downstream models adapted from PLMs and can design downstream models for any type of downstream task. Without losing generality, we considered six benchmark datasets cover three trendy topics, namely SST/AG's News/Yelp Review Full for text classification (Socher et al. 2013 Zhang et al.) 2015), CONLL2003
All four models we consider in this paper are downloaded from https: https://huggiiggaaee co../eac.ccccc...eee1
4

---

Under review as a conference paper at ICLR 2023
1.0
1.0
0.8
0.6
Recal
Dow merestaammmmmmmmmm Daka
Dowtsbeam laaca
SST
BST
0.2
AC's Novs
02
AOS Nows
Tepp Slecte Full
Yeep Decie Tuxt
NEIT
NEI
0.0
3.0
(a) Precision
(b) Recall
Figure 1: Attack performance text classification, NER and Q&A tasks. SST/AG's News/Yelp Re- view Full are three text classification tasks with 2/4/5 classes; CoNLL2003 is NER dataset; SQuAD v1.0. is Q&A dataset.
for Named Entity Recognition (NER) (Li et al. 2020 and SQuADv.10 & one for Q&A (Bordes)
et al. 2014).
Auxiliary Dataset and Attack Model. To build the auxiliary dataset, select very small subset from the pre- pre-training data used for all these PLMs, i.e., 0.23% of Wikipedia and 0.02% of BooksCorpus, members. We then select third-party datasets non-member datasets, which is different from both the pre-training and fine-tuning datasets. The non-member datasets are IMDB, CoLA and AX, which from the GLUE benchmark dataset (Devlin et al. 2019). We use mix of the three datasets as non-members for better generalization.
For attack model, we build three-layer MLP the attack model, which accepts the output of downstream models as input. Due the various of downsteram task, the dimentions of fist layer of attacks is also different across different tasks.
Evaluation Metrics. We use precision and recall as evaluation metrics for attack performance, as used in previous works (Salem et al. 2019 He al. 2022).
Attack Implementation. We train the attack model for 100 epochs with a learning rate of le-5, and the optimizer we use is the Adam (Kingma Ba 2015)) optimizer. We split the auxiliary dataset into training and testing data for the attack model in a 5:1 proportion. We also further investigate the effect of the size or type of the auxiliary dataset on the attack performance in Section [Section 5.2
5.2 EXPERIMENTAL RESULTS
Attack Performance. First, we report the attack performance, i.c., precision and recall, in Figure We can observe that membership inference achieves significant performance across all PLMs and different downstream tasks compared to random guessing. (i.e., 0.5). For instance, the precision 0.774/0.875/0.877 and the recall is 1825/00862/0) 8700
of BERT fine-tuned on SST/NER/Q&A which is much higher than the random guess. These results indicate that membership leakage in PLMS exists regardless of the type of downstream task, which poses much more serious threat to PLMs than that shown by existing work. These empirical results verify key intuition quantita- tively.
Membership Leakage Analysis. The above results fully demonstrate the vulnerability of PLMs to membership leakage. Here, we delve more deeply into the reason behind that from a visualization method. We study the pre-trained BERT that is then fine-tuned AG's News dataset. We ran- domly sample 1000 member samples and 1000 non-member samples. We first feed these samples to the original pre-trained BERT and embed the output into 2D space using t-Distributed Stochastic
5

---

Under review as a conference paper at ICLR 2023
(a) Per-trained BERT
(b) Fine-tuned BERT
(c) Downstream model
Figure 2: TSNE visualization on BERT fine-tuned on AG's News. Each plot means that TSNE visualization is based the embedding of this model.
Rocal
Precbion
W00goors BroisCroppu Milliee
Wiggessy Broie Melure
Widgens Brosee Minure
1.733
0.844
04112
5.000
0.3113
0.200
04113
01411
1.134
ED109
3.352
00832
L.925
0.901
0.815
0,787
00942
00/10
Walpeda GedsaCorrodgg..om
Walke
WaR DusssCarliaggdooo
(a) Text Classification
(b) NER
Figure 3: Attack performance with relaxation of member datasets, the x-axis represents the datasets used for training attack model, the y-axis represents the datasets used for testing attack model
Neighbor Embedding (t-SNE) as shown Figure ((a) Similarly, for the downstream model, i.e., by adding few layers on top of the pre-trained BERT and fine-tuning on AG's News, we obtain t- SNE visualizations of the BERT and downstream model outputs, as shown Figure 2(b) Figure respectively. From these results, we can make the following observations:
2(c)
Members and non-members on original PLMs behave differently.
Members and non-members on fine-tuned PLMs still behave differently.
Encouragingly, members and non-members on the fine-tuned downstream model also be- have differently, though not as much as on PLMs.
These observations qualitatively reveal membership leakage of PLMs' pre-trained data persists, although only downstream models are exposed and PLMs are not, and again verify key intuition.
Relaxation of Assumptions on Auxiliary Dataset. As aforementioned in Section 5.1 we use mixture of Wikipedia and BooksCorpus datasets datasets as our non-menn non-members, and then construct the auxiliary dataset. Here, we relax the assumption of members and non non-members the adversary access:
members, mixture of three third-party
Relaxat i ion-I: The adversary obtains only one of the members, e.g., Wikipedia, and uses to train the attack model and finally evaluate the attack performance on the other unseen members i.e., BooksCorpus.
Lexxation-Iie The adversary obtains only one of the non-members, e.g. IMDB, and uses train the attack model and finally evaluate the attack performance the other unseen non nom-merers, i.e., AX or CoLA.
Note that when we focus on the relaxation of one of the assumptions, we keep the other assumptions unchanged described in /Section 5.1
We report the attack performance for Re iollxxation-i I in Figure 3 First, we can observe that the attack performance remains very high for all different types of downstream tasks. These results

---

Under review as a conference paper at ICLR 2023
GoLA
CaLA
CHLA
13002
MM
AX
0/811
02010
AX
t9995
1993
3.915
0:811
025-4
0.99
IM0g
(a) Text Classification
(b) NER
Figure 4: Attack performance with relaxation of non-member datasets, the x-axis represents the
datasets used for training attack model, the y-axis represents the datasets used for testing attack model
show that whatever the type of members, they behave differently from non-members, and that such differential behavior between different types of members and meemmermbes share the same similar boundary. In other words, even if an adversary collects only one certain types of members, he/she can still obtain significant attack performance, which extends the scope of member inference attack.
We report the attack performance for relaxation-Ie in Figure 4 We can observe that the attack performance is still very high for all different types of downstream tasks. Similarly, these results show that whatever the type of non-menn monmmebers,, they behave differently from members, and that such differential behavior between different types of non-members and members share the same or similar boundary. Even if adversary collects only one certain types of non-members, he/she can still obtain significant attack performance, which further extends the scope of member inference.
In summary, these empirical results demonstrate that membership leakage of PLMs is much more severe than previously thought.
BERT
0.95
DERT
RoBERTa
RoBERTa
Predsion
0.85
Recall
0.65
1k
2k
4k
8k
16k 32%
64k 128k 256k
1k
2k
4k
8k
16k 32k
64k 128k 256k
Data Size
Data Size
(a) Precision
(b) Recall
Figure 5: Attack performance versus the size of member dataset
Effeet of Member Data size. We further investigate the effects of size of member data the ad- versary collect. In our default setting, we adopt 0.23% of Wikipedia and 0.02% of BooksCorpus as members, i.e., 1.5w and 1.5w, respectively. Here, we evaluate the attack performance with different size of members by varying it from 1K to 256k. We report the attack performance across different PLMs fine-tuned on AG's News. As we can see in Figure 3] there precision and recall as the member size increases. These results actually consistent with the expectation that more training data leads to high performance for an ML model, whereas the at- tack model is essentially members is only 1k, that is, 500 from Wikipedia and 500 from BooksCorpus, the attack performance still very high. Note that 500 samples only account for 0. 0.0077% of Wikipedia and 0.00067% of
clear trend of increasing
ML model. Encouragingly,
observe that though the size of

---

Under review as a conference paper at ICLR 2023
BooksCorpus. Such low proportion gives developers researchers warning that membership inference against PLMs is easier to start, which once again poses a serious threat. Besides, they should carefully release their pre-training data to prevent their training information from leaking.
Effeet of Number of Classes. Then we
study the relationship between membership in- ference and the number of classes of down- stream tasks. The data set Review Full Dataset (Zhang et al. 2013), which an online review data set for business and has five classes. Here, we construct 2-class dataset, 3-class dataset, 4-class dataset and class dataset extracted from Yelp Review Full dataset. The only difference between these ex- tracted datasets is the number of classes, and the size of these datasets the same. As shown in Figure 6 the attack performance increases with the number of classes, which indicates that membership infor-
here is Yelp
1.0
score
0.8
Model
BERT
ALBERT RoBERTa
the more classes, the mation leaked by the downstream model. A straightforward reason that the dimension of the downstream model's output increases with the number of classes, which brings more in- formation about membership, thus leading to higher performance of membership inference.
XLNet
0.5
Number of class
Figure 6: Attack performance versus number of classes.
1.0
1.0
0.9
Precision
0.8
0.8
Recall
0.7
Model
Model
ALBERT
ALBERT
RoBERTa
RoBERTa
XLNet
XLNot
3000K
256K
Data Size
Data Size
(a) Precision
(b) Recall
Figure 7: Attack performance versus different fine-tuning data size
Effect of the Size of Fine-tuning Dataset. Here, study the effect of the size of fine-tuning dataset. We random sample different sizes of samples from Amazon dataset to fine-tune the all the PLMs we consider this paper. The attack performance in Figure shows that attack performance drops with the increase of fine-tuning data size. The reason is that adding more fine-tuning data to fine-tune the model equal to adding additional knowledge to the PIMs, and PLMs will change to greater extent, the membership information may be laid over by the fine-tuning process, which leads to worse attack performance.
Effect of Number of Epochs. Lastly, we study the effect of the number of fine-tuning epochs. As shown in Figure 8 attack performance decreases as the number of epoch increases. PLMs change to larger extent the number of fine-tuning epochs increases, pre- are-training data is more likely to be forgotten, leading to poorer attack performance. However, in practice, download service providers typically fine-tune their task-specific models within more
membership information of
8

---

Under review as a conference paper at ICLR 2023
1.0
1.0
Precision
Recall
0.7
0.7
0.6
0.6
Model
Model
0.5
ALBERT
0.5
ALBERT
RoBERTa
RoBERTa
0.4
0.4
28
28
32
Epoch
Epoch
(a) Precision
(b) Recall
Figure 8: Attack performance versus different number of fine-tuning epochs
than 10 epochs, which means that membership leakage of PLMs is resisted after the fine-tuning process.
POSSIBLE DEFENSE
To mitigate the threat of membership leakage, large body of defense mechanisms have been pro- posed in the computer vision domain, such as data augmentation, Differential privacy (Acs et al.] 2017 Chen al. 2020 Bun & Steinke||2011
adversary regularization (Nasr et al. 2018), Mem- Guard (Jia et al. 2019). Note that due to resource limitation, i.e., insufficient GPU resource, we unable to train PLMs billions of corpora,
adopt the pre-trained language models publicly available online. Therefore, we cannot evaluate the first three defenses that need to be ap- plied in the pre- pre-training process. For MemGuard that changes the confidence score of classification task, it definitely achieves good defense performance in downstream classification tasks. However, for NER and Q&A, the output vectors contain enough information for their own downstream tasks, and MemGuard that changes the output vectors can lead to performance degradation.
In addition to existing defenses, here explore possible defenses based on the above evaluation in Section 5.2 i.e., increasing the size of the fine-tuning dataset and increasing fine-tuning epochs. As shown in Figure 7 and Figure 8 we can observe that the attack performance decreases with the increase of the size of fine-tuning dataset and fine-tuning epochs. Both can change PLMs to greater extent, so they can override the membership information through fine-tuning process, which leads to the reduction of membership leakage. In addition, both possible defenses also benefit the downstream task, i.e., improving the utility performance of the downstream model. Lastly, we leave it future work to explore effective defenses against membership leakages of PLMs.
CONCLUSION
In this work, we take the first step to systematically evaluate the membership leakage of pre-training data of PLMs. In particular, we consider realistic scenario where the adversary can only get access to the output of downstream models adapted from PL.Ms. We conduct extensive experiments on a va- riety PLMs architectures and different types of downstream tasks. The experiment results shows membership leakage of PLMs exists even when only the downstream model is exposed, showing severe threat than previously thought. To further study which factors that affect membership leak- age, conduct in-depth ablation study from different angles, which direct model inventors to alert to vulnerabilities of PLMs. Finally, we explore possible defenses against membership leakage of PLMSs and propose two promising defenses.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Gergely Acs, Luca Melis, Claude Castelluccia, and Emiliano De Cristofaro. Differentially Private Mixture of Generative Neural Networks. International Conference on Data Mining (ICDM), pp. 715-720. IEEE, 2017.
Antoine Bordes, Sumit Chopra, and Jason Weston. Question Answering with Subgraph Embed- dings. In Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 615-620. ACL, 2014.
Mark Bun and Thomas Steinke. Concentrated Differential Privacy: Simplifications, Extensions, and Lower Bounds. In Theory of Cryptographt Conference (TCC), pp. 335-658. Springer, 2016.
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song. Andreas Terzis, and Florian Tramèr. Mem- bership Inference Attacks From First Principles. CoRR abs/2112.03570, 2021.
Dingfan Chen, Tribhuvanesh Orekondy, and Mario Fritz. GS-WGAN: A Gradient-Sanitizze Ap- proach for Learning Differentially Private Generators. In Annual Conference on Neural Informa- fion Processing Systems (NeurIPS). NeurIPS, 2020.
Christopher A. Choquette Choo, Florian Tramèr, Nicholas Carlini, and Nicolas Papernot. Label- Only Membership Inference Attacks. In International Conference on Machine Learning (ICML), pp. 1964-1974 PMLR, 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, In Conference of the North Ameri- can Chupter of the Association for Computational Linguisties: Human Language Technologies (NAACI-HI.T), pp. 4171-4186. ACI., 2019.
Zekeriya Erkin, Martin Franz, Jorge Guajardo, Stefan Katzenbeisser, Inald Lagendijk, and Tomas Toft. Privacy-Preservve Face Recognition. Privacy Enhancing Technologies Symposium, 2009.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't ' Stop Pretraining: Adapt Language Models to Domains and Tasks. In Annual Meeting of the Association for Computational Linguistics (ACL), pp. 8342-8360. ACL, 2020.
Xinlei He, Zheng Li, Weilin Xu, Cory Cornelius, and Yang Zhang. Membership-Doctor: Com- prehensive Assessment of Membership Inference Against Machine Learning Models. CoRR abs/2208. 10445, 2022.
Hongsheng Hu, Zoran Salcic, Lichao Sun, Gillian Dobbie, Philip S. Yu, and Xuyun Zhang, Mem- bership Inference Attacks on Machine Learning: A Survey. ACM Computing Surveys, 2021.
Thomas Humphries, Matthew Rafuse, Lindsey Tulloch, Simon Oya, Ian Goldberg, and Florian Kerschbaum. Differentially Private Learning Does Not Bound Membership Inference. CoRR abx/2010.12112, 2020.
Abhyuday Jagannatha, Bhanu Pratap Singh Rawat, and Hong Yu. Membership Inference Attack Susceptibility of Clinical Language Models. CoRR abss2/04.08.30/ 2021.
Jinyuan Jia, Ahmed Salem, Michael Backes, Yang Zhang, and Neil Zhenqiang Gong. MemGuard: Defending against Black-Box Membership Inference Attacks via Adversarial Examples. ACM SIGSAC Conference Computer and Communications Security (CCS), pp. 259-274. ACM, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations (ICLR), 2015.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. ALBERT: Lite BERT for Self-supervised Learning of Language Representations. In International Conference Learning Reprexentations (ICLR), 2020.
10

---

Under review as a conference paper at ICLR 2023
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jae- woo Kang. BioBERT: pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 2020.
Jiacheng Li, Ninghui Li, and Bruno Ribeiro. Membership Inference Attacks and Defenses in Classi- fication Models. In ACM Conference on Data and Application Security and Privacy (CODASPY), pp. 5-16. ACM, 2021.
Xiaoya Li, Jingrong Feng, Yuxian Meng. Qinghong Han, Fei Wu, and Jiwei Li. A Unified MRC Framework for Named Entity Recognition. In Annual Meeting of the Association for Computa- fional Linguistics (ACI.), pp. 5849-5859 ACI., 2020.
Zheng Li and Yang Zhang. Membership Leakage in Label-Only Exposures. In ACM SIGSAC Conference on Computer and Communications Security (CCS), pp. 880-895, ACM, 2021.
Zheng Li, Yiyong Liu, Xinlei He, Ning Yu, Michael Backes, and Yang Zhang. Auditing Membership Leakages of Multi-Exit Networks. CoRR abs/2208. / /1000 2022.
Jiaxiang Liu, Simon Oya, and Florian Kerschbaum. Generalization Techniques Empirically Outper- form Differential Privacy against Membership Inference. CoRR abs/21 2021.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pre- training Approach. CoRR abs/1907.11692, 2019.
Yugeng Liu, Rui Wen, Xinlei He, Ahmed Salem, Zhikun Zhang. Michael Backes, Emiliano De Cristofaro, Mario Fritz, and Yang Zhang. ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning Models. In USENIX Security Symposium (USENIX Security). USENIX, 2022.
Andrew McCallum and Wei Li. Early results for Named Entity Recognition with Conditional Ran- dom Fields, Feature Induction and Web-Enhanced Lexicons. In Conference on Computational Natural Language Learning (CoNLL), pp. 188-11 191. ACL., 2003.
Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg- KorgKKiikpatrick, and Reza Shokri. Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks. CoRR (bs/2203.03929g 2022.
Manish Munikar, Sushil Shakya, and Aakash Shrestha. Fine-grained Sentiment Classification using BERT. CoRR abs/19110034477 2019.
Milad Nasr, Reza Shokri, and Amir Houmansadr. Machine Learning with Membership Privacy using Adversarial Regularization. In ACM SIGSAC Conference Computer and Communications Security (CCS), pp. 634-646. ACM, 2018
Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. Pre-trained Models for Natural Language Processing: Survey. CoRR abs/2003.080 08227,, 2020.
Shadi Rahimian, Tribhuvanesh Orekondy, and Mario Fritz. Sampling Attacks: Amplification of Membership Inference Attacks by Repeated Queries. CoRR 0bb/22009000395 2020.
Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Ma- chine Learning Models. In Network and Distributed System Security Symposium (NDSS). Internet Society, 2019.
Virat Shejwalkar, Huseyin A Inan, Amir Houmansadr, and Robert Sim. Membership Inference Attacks Against NLP Classification Models. In PriML Workshop (PriML). NeurIPS, 2021.
Lujia Shen, Shouling Ji. Xuhong Zhang, Jinfeng Li, Jing Chen, Jie Shi, Chengfang Fang. Jian- wei Yin, and Ting Wang. Backdoor Pre-trained Models Can Transfer to All. In ACM SIGSAC Conference Computer and Communications Security (CCS), pp. 3141-3158. ACM, 2021.
11

---

Under review as a conference paper at ICLR 2023
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In Conference Empirical Methods in Natural Language Processing (EMNLP), pp- 1631-1642. ACI,, 2013.
Congzheng Song and Vitaly Shmatikov. Auditing Data Provenance in Text-Generation Models. In ACM Conference on Knowledge Discovery and Data Mining (KDD), pp. 196-206. ACM, 2019.
Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. How to Fine-Tune BERT for Text Classifica- tion? In China National Conference on Chinese Computational Linguistics (CCL), pp. 194-206. Springer, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Annual Conference on Neural Information Processing Systems (NIPS), pp. 5998-6008. NIPS, 2017.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. HuggingFace's Trans- formers: State-of-the the-trte art Natural Language Processing. CoRR abs//991.03371,.005
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V.
Le. XLNet: Generalized Autoregressive Pretraining for Language Understanding. Annual Conference Neural Information Processing Systems (NeurlPS), NeurlPS, 2019.
Ziqi Yang, Bin Shao, Bohan Xuan, Ee-Chien Chang, and Fan Zhang. Defending Model Inversion and Membership Inference Attacks via Prediction Purification. CoRR abs/2005.0320 2020.
Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, and Reza Shokri. Enhanced Membership
Inference Attacks against Machine Learning Models. CoRR abs/21 2021.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level Convolutional Networks for Text Classification. In Annual Conference on Neural Information Processing Systems (NIPS), pp- 649-657. NIPS, 2015.
12


Reviews:
[{"review_id": "Rqmm_58sLB", "reviewer": "Reviewer_h5pk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper runs inference attacks against fine-tuned language models to evaluate membership leakage of pre-training data. The attacker is assumed to have access to a small subset of the pre-training dataset. The attacker trains an attack model whose training data is the subset of pre-training data and some non-member data.", "summary_of_the_review": "My recommendation is mainly because the attack in this paper is not a qualified membership inference attack. It is more like a text classification task because of the choice of training data for the attack model.", "strength_and_weaknesses": "Evaluating the privacy risk of pre-training data through fine-tuned models is an interesting problem. I think in reality it is possible that the pre-trained language models are held as secret and the attacker  could one access fine-tuned APIs. Exploring this direction would help the community to understand how the privacy risk of pre-training data changes after fine-tuning.\n\nHowever, the ‘membership inference attack’ in this paper is not a qualified attack because of the choice of attack model’s training data. The member/non-member data used to train attack model have to be from **the same distribution**. The non-membership data in this paper are data from the GLUE benchmark which are not from the same distribution as the pre-training data. The ‘attack’ then becomes a standard text classification task. A proper non-member data should be from the same source as the pre-training data, e.g., the test set of the pre-trained model.\n\n\nHere are some other suggestions.\n\n1. “To our knowledge, the potential privacy risks of pre-training data for PLMs have never been explored.” There is a line of works focus on the privacy risk of pre-trained models. A representative one is Carlini et al., 2021[1]. It directly attacks pre-trained models instead of fine-tuned ones. Therefore, this claim is inaccurate.\n\n2. How do you set the number of member datapoints to train the attack model? You choose $0.23\\%$ of Wikipedia and $0.02\\%$ of BookCorpus. Why these two specific numbers\n\n\n[1]: Extracting Training Data from Large Language Models, https://arxiv.org/abs/2012.07805.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "J0W4rAkkbz2", "reviewer": "Reviewer_qzMP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the leakage of pre-training data, after fine-tuning for down-stream tasks, through the downstream task API. In other words, this paper studies how much pre-training data a fine-tuned model leaks, if black-box access is provided to a fine-tuned pre-trained model. The paper assumes access to 'auxiliary' data, which is a combination of a few samples of pre-training data, and non-member data, which are all fed to the down-stream model, logits collected and then used as 'training data' for an attack model, which has the binary task of guessing membership based on downstream model's logits.  The paper then mounts this attack on different scenarios with different setups and studies memorization and shows that pre-training data does indeed leak from the downstream models.", "summary_of_the_review": "I have provided thorough feedback in the strengths/weaknesses box which includes my questions. I am willing to increase my score if my questions are answered, specifically the confusion about where membership is measured over fine-tuning data and where it is measured over pre-training data. ", "strength_and_weaknesses": "Strengths:\n1. The paper studies downstream fine-tuning tasks, and measures leakage of pre-training data on those which I haven't really seen studied before. I have seen papers measuring memorization of fine-tuning data after pre-training, and papers studying pre-training alone, but not this scenario which is interesting. \n\n2. I like how the authors study and ablate the effect of different things, such as training epochs, classes, dataset size, etc. \n\n\nWeaknesses: \n1. The paper needs to clarify/narrow its claims and assumptions a bit better. More specifically, the following issues need to be clarified:\n\na.\tUse of the term pre-trained LMs, or PLMs: The opening paragraph of the paper states \"... (PLMs), represented by BERT\" which is not accurate, as BERT only represents MLMs as pre-trained models, whereas there is a whole different class of generative (auto-regressive), pre-trained models, such as the GPT family, whose memorization and leakage has been widely studied before [1-3]. I think it is really important that  the paper clarifies what it means by pre-training early on and not use the term pre-trained LM solely to BERT-based models (also encoder-decoder pre-trained models such as T5 and BART are also fairly common and are referred to as pre-trained LMs).\n\nb.\tClaiming prior work has not studied privacy risks/memorization in pre-training: I think this paper is very interesting and the problem is novel, however the positioning right now is inaccurate and needs to be improved. the claim \"Pre-training has never been studied\" needs to be better clarified: In terms of memorization of pre-training data,  Carlini et al. [1] study it for the GPT models, [4-5] study it for MLMs, and [2] studies memorization in LLMs. There is also work studying the memorization of fine-tuning LMs [3,6]. What the paper is studying is leakage of pre-training data through downstream tasks. This needs to be better clarified. \n\nc.\tIn the sentence \"PLMs memorize pre-training data deeply. By further analogy with the fact that the confidence scores of members in image classification are usually\", instead of image classification, NLP papers can be used as evidence as there are already many NLP attack papers. \n\n\n2. The writing and representation of the results is a bit confusing. For example, it is really hard for me to figure out what exactly is the downstream task in Figure 2? Another issue is what results are reported over the downstream task data and what membership results is over pre-training data? For example, the number of classes experiment at the end of the paper, is it the effect of number of downstream classes on pre-training data memorization/forgetting? It seems not. However, the number of training epoch experiments seems to study pre-training data forgetting as fine-tuning goes on. If the paper focuses on pre-training data, it should make that clear and present results accordingly.\n\n3. I also don't understand how Figures 3 and 4 show \"In summary, these empirical results demonstrate that membership leakage of PLMs is much more severe than previously thought.\", as in what was previously thought? The fact that fine-tuning/downstream classification memorizes samples a lot was shown before so I am not exactly sure what this conclusion is. \n\n4. In Figure 2's explanation, the paper says: \"Encouragingly, members and non-members on the fine-tuned downstream model also be- have differently, though not as much as on PLMs.\", however, the points actually overlap a lot. I would actually be interested in seeing the precision-recall numbers for this experiment, as I think this is the most important experiment in the paper, which directly studies leakage of pre-training data after fine-tuning. \n\n\nMinor:\n1.\t”machine learning as a service”, the opening quotation marks need to be fixed\n2.\t\"that the attackers has a very small subset\" -> have\n3.\t\"Page 4: larger than non-members[][][],\" references are missing\n\nReferences:\n[1]  Carlini, Nicholas, et al. \"Extracting training data from large language models.\" 30th USENIX Security Symposium (USENIX Security 21). 2021.\n\n[2] Tirumala, Kushal, et al. \"Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models.\" arXiv preprint arXiv:2205.10770 (2022).\n\n[3] Mireshghallah, Fatemehsadat, et al. \"Memorization in NLP Fine-tuning Methods.\" arXiv preprint arXiv:2205.12506 (2022).\n\n[4] Mireshghallah, Fatemehsadat, et al. \"Quantifying privacy risks of masked language models using membership inference attacks.\" arXiv preprint arXiv:2203.03929 (2022).\n\n[5] Lehman, Eric, et al. \"Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?.\" arXiv preprint arXiv:2104.07762 (2021).\n\n[6] Shejwalkar, Virat, et al. \"Membership inference attacks against nlp classification models.\" NeurIPS 2021 Workshop Privacy in Machine Learning. 2021.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "C53K5Ubr8QL", "reviewer": "Reviewer_EhJ8", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a membership inference attack against a fine-tuned language model where the targets are the sentences from the pre-train corpus. The attack trains a simple classifier with known members and non-members using the outputs of the fine-tuned model. The empirical evaluation using Wikipedia and BookCorpus as a positive set and SST, AG News, and Yelp Review as a negative set. The classifier could identify these two sets with higher precision and recall than random guess.", "summary_of_the_review": "This paper considers an interesting problem. Membership inference in other domains show the task is actually very difficult. In contrast, the experimental result in this paper looks promising despite the additional fine-tuning step. However, I doubt the cause is properly understood and the variables are correctly controlled. The paper unfortunately states that pre-training cannot be controlled due to the resource, but proper evaluation can be only done when the pre-train corpus is at least split into a train dataset for pre-train, a train dataset for the membership classifier, and a membership test dataset, to control the style, category and overall semantic of the test datasets. Without these, the membership classifier can look at which dataset the test sentence came from using the output of the fine-tuning tasks, which conveniently related to the chosen negative dataset. There was no attempt to understand or compare the outputs of the fine-tuned model either, making the verification of the claim not possible.", "strength_and_weaknesses": "Strengths.\n- S1. The proposed classifier achieves better precision and recall than random guess.\n- S2. The experiment considers dimensions such as the number of epochs and the size of the fine-tuning dataset.\n- S3. A defense is discussed.\n\nWeaknesses.\n- W1. The claim of higher certainty by the fine-tuned model with a member sample is not verified. Instead, a black box classifier is trained. This hinders readers from understanding how this is made possible.\n- W2. The attack model might be classifying the sentence type (movie review, Q&A, ...) rather than the membership, as they are consistently coming from certain sources, and they have distinct styles, semantics, and category. So the experiment is not well controlled to say the cause is the membership. For example, the positive dataset includes Wikipedia which is very general and encyclopedic, but IMDB is a review dataset. The fine-tuned model with SST will classify the IMDB data into positive and negative, while Wikipedia sentences often do not have clear polarity. Similarly, the output of the fine-tuned model with AG's News can provide the category of the input sentence, and this setting of Wiki/Book vs. Movie review, etc dataset can be easily classified with such category information. Again, knowledge vs. review data can have very different semantics and styles. Thus, the cause of the precision and recall is unclear. For more accurate assessment, a pre-training has to be done on the same type of corpora, or even better, by dividing a single corpus into two at the pre-train stage to properly test them.\n- W3. The absolute number for the fraction of the known members 0.23% and 0.02% is not given initially, which is seemingly very large considering the size of Wikipedia and BookCorpus.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject