Fragments to Facts: Partial-Information Fragment Inference from LLMs
Authors: Lucas Rosenblatt
Bin HAN
Robert Wolfe
Bill Howe
Venue: ICML 2025 poster (2025)
Keywords: memorization
extraction
membership inference
attacks
LLMs
large language models
privacy
Primary area: deep_learning->large_language_models
TL;DR: Fine-tuned LLMs can leak sensitive data; we propose a threat model using partial, unordered fragment sets to attack LLMs and propose effective inference methods under this model.

Abstract: Large language models (LLMs) can leak sensitive training data through memorization and membership inference attacks. Prior work has primarily focused on strong adversarial assumptions, including attacker access to entire samples or long, ordered prefixes, leaving open the question of how vulnerable LLMs are when adversaries have only partial, unordered sample information. For example, if an attacker knows a patient has "hypertension," under what conditions can they query a model fine-tuned on patient data to learn the patient also has "osteoarthritis?" In this paper, we introduce a more general threat model under this weaker assumption and show that fine-tuned LLMs are susceptible to these fragment-specific extraction attacks. To systematically investigate these attacks, we propose two data-blind methods: (1) a likelihood ratio attack inspired by methods from membership inference, and (2) a novel approach, PRISM, which regularizes the ratio by leveraging an external prior. Using examples from medical and legal settings, we show that both methods are competitive with a data-aware baseline classifier that assumes access to labeled in-distribution data, underscoring their robustness.

Fragments to Facts: Partial-Informatio Fragment Inference from LLMs
Lucas Rosenblatt
Bin Han *2
Robert Wolfe
Bill Howe?
Abstract
Pariill Information
Memorization Attack
Emggeet Inference
Large language models (LLMs) can leak sensitive training data through memorization and member- ship inference attacks. Prior work has primarily focused on strong adversarial assumptions, in- cluding attacker access to entire samples long. ordered prefixes, leaving open the question of how vulnerable LLMs are when adversaries have only partial, unordered sample information. For example, if an attacker knows a patient has "hy- pertension," " under what conditions can they query model fine-tuned on patient data learn the patient also has "osteoarthritis?" In this paper, we introduce more general threat model under this weaker assumption and show that fine-tuned LLMs are susceptible to these fragment-specific extraction attacks. To systematically investigate these attacks, we propose two data-blind methods: (I) a likelihood ratio attack inspired by methods from membership inference, and (2) novel ap- proach, PRISM, which regularizes the ratio by leveraging an external prior. Using examples from medical and legal settings, we show that both methods are competitive with a data-aware base- line classifier that assumes access to labeled in- distribution data, underscoring their robustness.
public frigments
The pattere a Moyyammoll the clinte today originally for hypestensinn
to
known prefi
fosteoarth it i hypartans 3
memoriani
and check, She has hEstory of hypes Sons os ostoogorosss.
osteopozosis
wit
pávau Enagreenss
Figure I. Comparing the PIFI LLM threat model to the memo- rization threal model in a medical scenario. PIFI uses unordered, publicly available fragments from sample (like a patient record) to infer private fragments (like sensitive medical condition). Memorization assumes access to an ordered prefix of the sample, and checks for verbatim generation by LLM of the suffix.
they' 've become susceptible to attacks targeting private, sen- sitive information (Carlini et al., 2021: 2024).
A growing body of work has studied membership inference attacks which aim determine if a specific example appears in the training set (Jagannatha al., 2021; Carlini al., 2022a; Mireshghallah et al., 2022; Shi et al., 2023; Mat- tern et al., 2023; Morris et al., 2023; Duan et al., 2024) and memorization attacks which attempt to reconstruct verbatim training examples (Carlini et al., 2021; 2022b; 2024; Lukas et al., 2023). Membership inference often targets outlier samples, while memorization attacks focus on content that is repeated ("inlier" samples (Dionysiou & Athanasopoulos, 2023)), and each approach typically assumes strong adver- sarial knowledge: at least access to ordered prefixes, and typically access to entire training samples (for example, in the case of copyright infringement cases (Grynbaum & Mac, 2023; Small, 2023)). These assumptions overlook more re- alistic and common scenarios which an attacker possesses only partial, unordered information about target.
1. Introduction
Instruction-tune language models (LLMs) have trans- formed how users interact with AI, offering personalized assistance in domains ranging from fact-checking to postpar- tum care (Tang et al., 2024; Antoniak et al., 2024; Wolfe & Mitra, 2024a:b). This tight feedback loop between end users and developers also enables continuous fine-tuning of LLMs on the very data collected from user interactions (Poddar et al., 2024). Yet, as these models grow in size and scope,
We consider a malicious adversary who seeks to acquire sensitive fragments of information from an individual's 'ss data. For example, consider an attacker who learns that an indi- vidual has hypertension from a social media post, and wants to learn co-morbid conditions that apply to that individual in order to target sales, deny insurance, or even aid in identity theft. Using this small subset of the patient's 's data (i.e., a single diagnosed condition), can the attacker extract those co- co-morbidities by prompting fine-tuned language model? Though they remain under-specified and under-studied, such attacks are increasingly plausible, especially in the case of a clinic or hospital offering a public or internal chatbot cre- ated by fine-tune fine-tuning on the organization's own medical data
Eqqual contribution 'New York University, New York, USA University Washington. Scattle, USA. Correspondence to: Lu- cas Rosenblatt (11287@@ny.cduuu
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Partial-Information Fragment Inference from LLMs
(McDuff et al., 2023; Wu et al.; Zhou et al., 2024).
2. Preliminaries
In this research, we propose new partial-information frag- ment inference threat model that unifies insights from mem- bership inference and extractable memorization attack strate- gies under weaker adversarial assumptions. Notably, we show that if an attacker only knows a small set of text frag- ments from an original training sample (e.g.. that "hyperten- sion" and "beta blockers" are in a patient's 's medical notes), they can infer additional, potentially sensitive fragments (e.g., "osteoporosis") that are also in the training sample.
We provide relevant background on LLMs and data privacy attacks, and we introduce necessary notation. We then dis- cuss related adversarial threat models: sample membership inference and training data extraction.
Large Language Models We study generative (also known as "autoregressive" or "causal") language models, which are trained to predict the next word in a sequence by max- imizing the negative log-likelihood of the model over to- kens (words or subwords) in a vocabulary (Radford, 2018). We specifically examine generative language models that have undergone the process of instruction tuning (Wei et al., 2021), meaning our models use chat-based format wherein the model and user take turns exchanging messages, and the model responds in accordance with human preferences (Ouyang et al., 2022). Examples of such models include ChatGPT (OpenAI, 2022), Llama (Dubey et al., 2024; Tou- vron al., 2023), and Mistral (Jiang et al., 2023) models that are particularly relevant because of their case of use and accessibility both to lay users and potential adversaries.
Contributions Our findings reveal that even weak adver- saries with access to only a few unordered fragments of an individual's sample can pose a privacy threat by comparing results between models to infer additional fragments. Based on our results, we suggest the need for improved defenses that mitigate not just sample memorization or membership inference, but also partial, fragment-level inference vul- nerabilities, before deployment of public-facing fine-tuned models in sensitive domains like medicine or law. More specifically, we make the following contributions:
A Novel Threat Model. We formalize a threat model where an adversary only has access only publicly avail- able text fragments for a target individual, as opposed to that individual's 's full sample.
Notation We adopt several typical notational conventions when discussing language models and extraction attacks. Consider a token sequence (2,,, ,T) = x E X where X²" is the space of possible input sequences of length T. Each an is drawn from set of vocabulary tokens V (i.e., T, is typically single word or word subsequence in
2 Effective Data-Blind Attacks. We show that LR attkttckk, a straightforward Likelihood Ratio approach, surprisingly competitive, often rivaling more powerful classifier baseline that assumes access to labeled, in- distribution examples. We also propose PRISM (Posterior- Refined Inference for Subset Membership), a method that uses the LR-Attack score to update a posterior likelihood by using a prior, ultimately reducing false positives.
V). We refer to the dataset of token sequence samples as D = {x,]/=ı we will also let D ~ D, where D is a probability distribution over X specific to some task (e.g.. summarization of medical notes).
Formally, a generative language model is fit by learning to predict the next token over X by expanding,
3 Empirical Validation. We conduct experiments in a medical sunmarization context, first fine-tuning an LLM for summarization on medical notes. We then reduce each note to set of fragments (in this case, medical terms), and we simulate the efficacy of our attacks in the hands of an adversary who possesses only a small amount of information about an individual. Our experiments show that fine- -uned LLMs are vulnerable to extraction attacks under even these inttrmttinoo conditions; we observe 9.5% TPR on Qwen-2-7B at 2% FPR using LR- ARR-Attack, and an 11.5% TPR on 1-ama-3-8B- at 5% FPR using PRISM, for example.
T
Pr(11,22..... .., (T) = II Pr(z: 21, ................................ 21-1)
E=1
We refer to a generative language model fine-tuned on a dataset fo.D (read as model f with pre-trained parameters 0, further fine-tuned on D). We then denote the probability of a sequence of tokens under a given model as Ja,n(x) fe, (Zn, Zi...., In-1). FI....,Tn-1) sequence of tokens under a specific prompt sequence x, we write f x) or
To denote the probability of a
Vyn 21,..., Tm).
Threat Model Specific Notation
We describe the problem setting and notation in $2, present our threat model in $3, attack approaches in $4, and de- scribe experimental setup on real-world scenarios in $5. We introduce the LLMs in $6, and articulate experimental re- sults in $7. We then discuss related work, implications, and challenges from our work, encouraging further ex exploration of strengths and limitations of this new family of attacks.
We also adopt several non-standard notational conventions. To help distinguish between an arbitrary training sequence and a sequence associated with individual, we use the shorthand (s1...., Sx2) = s to refer to a sequence associated with a single individual in D (e.g., s refers to John Doe's medical note). We refer to short, ordered sequences of tokens as fragments. Note that a fragment
2

---

Partial-Information Fragment Inference from LLMs
can be a single word (e.g., ostcoporosis), yet consist of an ordered sequence of tokens in an LLM embedding matrix (e.g., 'oste,' "opor," 'osis"). Additionally,
Carlini et al. (2022b) define extractable memorization as follows: for black-box target model Jo.D, D, a sample suffix Skm is considered extractable if there exists length-(k 1) prefix. S1:k-1, such that the concatenation [$1:k-1 || Skm] is in D and fo.D produces 8km when decoded greedily under prompt S1:k-1- Conversely, given black-box access to a target model fa.r2 and complete target sequence s, mem- bership inference attack produces a likelihood that s was in D, mediated by the output of fo.D (Carlini et al., 2022a). MI attack performance is based on using that likelihood to make classification (i.e., predicting 1 or O for included or not included, respectively). Despite differing objectives, the threat models share an assumption: they assume a priori access to s, a complete and ordered target sample of interest.
We let the function A represent our adversarial pub- lic information assumption, which maps an ordered sample sequence s = ($1....., 8n) to a potentially unordered set 8 of short fragments, typically key- words that carry sensitive information but are dis- coverable through public information. For example, an attacker might learn from social media that John Doe contracted covid and experiences hypertension. While the fragments "covid" and "hypertension" are a subset of John Doe's complete medical chart, no prefix or other significant component of the chart itself is available. Formally, let, A n P (X*), so that S = A(s). Usually we assume S C Frag(s), where Fsag(s) denotes the collection of all fragments contained in S, but |S| << [Frag[s)] and the frag- ments in S are words or short phrases.
3. Partial-Information Fragment Inference
Assuming a priori access to a complete sample s is plausible in some settings, like measuring the likelihood of copyright infringement of one's own data, but not in cases where an adversary attacks a model without access to the origi- nal training data. We thus propose the fragment inference (PIFI, pronounced "piffy") adversarial threat model for information extraction under weaker as- sumption; namely, under PIFI, I, an adversary seeks to infer specific and sensitive fragment-level information about in- dividuals based on a small subsel of public fragments (e.g.. between 4 and 30) assumed to be present in sample s.
2 We y denote single, private candidate fragment targeted under our threat model.
We use the conventional term shadow model (Carlini al., 2022a) discuss model fine-tuned on data from distribution D, but where we are sure that the sample of interest was not included. Formally, let D' denote a shadow dataset, where s & D': both D and D' are drawn from D. Thus, finetun- ing on D' yields a shadow model, fe.D.. Finally, we introduce notation for a world model that al- lows us to query k arbitrary models to estimate the probability of a sequence x under any model. We thus denote the world model folworid as follows: fe.eori( =
3.1. Capabilities of the Adversary
The PIFI threat model makes the same black-box model access assumptions as most MI and EM attacks. Specifically, PIFI assumes access to output probabilities produced by fe.D. fo.D' and arbitrary additional models fo.woria, but not access model weights, embeddings, training param- eters. Unlike other threat models, PIFI does not assume access to complete sample s associated with an individ- ual; instead, the adversary only has access to a fragment set S C Frag(s). The adversary can then investigate a candi- date target fragment y", where S is useful for investigating y* if S is likely to be a subset of the tokens in the original sample. Consider the realistic scenario wherein an attacker has learned something specific about an individual's medical history from their social media: PIFI enables the attacker to use that limited information as S to infer additional sen- sitive data y" about the individual from an LLM fo.D-
k 1= (x).
2.1. Prior Threat Models
We outline prior work that induces language models to ex- pose specific, private information learned from training data.
Standard Language Model Attacks Two prominent at- tack threat models have emerged in the literature on LLM privacy. The first, extractable memorization (EM) (Carlini et al., 2019; 2021; 2022b). (sometimes called discoverable memorization (Wang et al., 2024)), asks whether prompting model with part of a sample from its training data will lead the model to output the rest of the sample (approximately (Yu et al., 2023) or exactly (Lukas et al., 2023)). The sec- ond, membership inference (MI) (Carlini et al., 2022a), asks whether one can infer sample membership in the training data from model outputs (usually as a binary classification task). We describe both attacks to motivate our approach.
3.2. Goal of the Adversary
The adversary would like to know if the candidate target fragment y" is in an individual's sample s, given that s was included in training data D. Why is it important to condition on an individual's sample actually being included in D? If the adversary had reason to believe that s was not in D. then
3

---

Partial-Information Fragment Inference from LLMs
4. Approaches to PIFI
Figure 2. An illustration of our threat model. An I.IM is fine- tuned, egg. with private medical notes. Then, an adversary prompts
In this section, we describe the class of INFER functions for producing the likelihood f. Then, we define the three specific functions from this class that we consider in this work: Classifier, LR-Attack, and PRISM.
the fine-tuned LLM with relevant fragments of information (e.g., from target individual's 's medical records) to infer unknown frag- ments associated with the individual.
private samples s,,
may Sy to finetune feD
The patsert is a 76 76-yoax old shite Semale who peesents to to the clinds today originally for typertensian and aed exteopernnti.
4.1. The Classifier Baseline
hilenary
To produce a strong baseline, we train a binary classi- fier on the same three-dimensional vector of probabili- ties [po, Po', Puoriel used in the computation of our LR-At and PRISM attacks, with the goal being to predict whether a given y" was in the data D on which a target LLM was trained. For training, however, a classifier requires labeled data. Labeled data access an unrealis- tic assumption under the PIFI threat model; the attacker explicitly does not have access any amount of the train- ing data D for target model Se.D. and may not even have access to samples suitably similar to target s. Of course, if the attacker somehow did have labeled training data, they could adjust their INFER approach using standard machine learning tools to fit the distribution over labels, leading to strong attack. This makes Classi fier, which we call data-aware method, a very strong reference point for the performance of AR-Attack and PRISM, which are data- blind methods. Generally, we expect it to upper-bound their performance, assuming the training distribution generalizes for a particular distribution over text fragments. Note that for the empirical evaluations included in this paper, we employ Light Gradient Boosting Machine ("LighiGBM")) model (Ke et al., 2017). a highly performant option for binary classification with data, and do standard cross-validatio when training. Overall, Classifi iier is an important vali- dation of the PIFI threat model and a target for the data- blind attacks; C1 ass fier validates that there is signal in the vector [PD: Po', Puoria] for attacks under the P I threat model (assuming one can specify the right data-blind INFER function to exploit it).
Adversarial Actor
Seeks to infer facts a bout patient from L.M feo trained on
Gleass a set of fragremm Frag (8) sia public sources
S be ta blockers hypertension)
Use S' to infer lilelibood of tanget fragmeent y" for patiinnt with black box access to fa.
y" =stoorrrattriiss ? S
their inference might be about individuals with samples like s, and not about the specific individual associated with s. For example, any individual with osteoarthritis may also have increased odds of being a smoker, however, this does not mean that Jane Doe, who has osteoarthritis, is also smoker, unless we have reason to believe that her sample (e.g.. her medical note) says this is the case.
Algorithm A Class of P IIFI Attack Models
Input: Private fragment public fragment set A(s) = S for an individual, target language model fo shadow and world models fe.D', foworid. decision threshold T, Output: (0.1}
1: Po = fo. Prompt(S))
2: Po = fo Pr)mpt(S))
3: Psoria = fo.worlal"" Prompt(SS)
4: + INFER([ Pps, PD', Peeria] }), where scores the likelihood that y E givense D.
5: Return [e 7].
4.2. The LR-Attack approach
In Algorithm 1, we formalize the inputs and outputs for the PIFI threat model, and we constrain the class of INFER functions that we consider in this paper. Specif- ically, we consider adversaries that use their access to models fo.D and fo.worid to compute quantities like, pn = fo, PS) where Prompt(S) = [x), $1, $1. Sm, xy] is a simple prompt in which the frag- ments possessed by the adversary are embedded. Ac- cordingly, using only information available in the vec- tor of probabilities (po, Po', P.artal. the attacks PRISM, LR LR-Attack, and Classifier each specify an INFER function to produce a likelihood E corresponding to the belief that y" is in s, given that s is in D, which can be converted to binary prediction with a decision threshold r.
The Neyman- Pearson Lemma is a classical insight from hypothesis-resting literature that generally underpins statisti- cal hypothesis testing and likelihood based attacks, like the data-blind LR- Atc we propose (Neyman & Pearson, 1933). Intuitively, the Lemma states that, given two distri- butions, the optimal way to distinguish between them at a fixed false-positive rate is to threshold their likelihood ratio. Carlini et al. (2022a) reinterpret this in the context of mem- bership inference: if Qia (s) the distribution over models trained with and Qout (s) is that over models trained with- out s. then the Seyman-Pearsoo Lemma implies we should estimate A (=)) where p(- Qun/ (s)) denotes the probability density of models under each distribution.
4

---

Partial-Information Fragment Inference from LLMs
As Carlini et al. (2022a) note, we never have direct access to these true distributions, we must approximate them empirically (e.g. using shadow models).
that may not hold but captures the intuition that if y is in s, then po will be higher relative to pp'. Hence, we choose to treat Pr(s E D) as random event M with some prior belief 6 = Pr(M), and then apply function in the shape of a Bayesian update using the LR-At AR-Attack ck statistic, estimat- ing Pr(M | 6) == Pr(f | E D) Pr(M). We pragmatically Pr(()
Unlike the ciambership-infereee threat model, however, PIFI does not assume access to the full sample s. Nonethe- less, the Neyman-Pearson Lemma still suggests that com- paring probabilities obtained from a target model potentially fine-tuned with against those from model fine-tuned without s can be informative. Concretely, for a candidate fragment y" and known (public) fragment set S, we mea- sure PD = fy PS) and PD' = fe.D-(y" (y" PS). We then define = Pn/Por which approxi- mates the ideal likelihood-ratio statistic. Large i indicates that the target model assigns an unusually high probability to y" relative the shadow model, in the context of the known fragments S. In the extreme case where we already know that y' truly appears in s, distinguishing between {s E D} and {s & D} reduces to membership inference but in the PIFI setting, the goal is to infer whether y* in s, using only our partial knowledge S to guide that inference. Thus, while é does not directly compute the "true" membership-beee based ratio, can expect it to sufficiently reflect that ratio to serve as effective attack statistic.
assume Pr(e I D) x & and Pr(( s & D) x 1/é. We plug é and 1/ in directly obtain a closed-form score we believe is correlated with the true Pr(M) é).
As we make many assumptions and approximations here, and it is difficult to reason about these distributions in the context of large language models, we investigated PRISM using a small, known language model under the P PIFT threat model. In particular, we tested PRTSM against LR-At t tack on a simple trigram language model with a small, character-level vocabulary that we could fully spec- ify (see Appendix C); this allowed us actually calculate the above probabilities that we can only hope to approxi- mate with larger language models. In this controlled setting, PRI SM outperformed LR-Attack this further motivated us to evaluate it alongside AR-Attack on L.MSs.
5. Experimental Setup & Datasets
4.3. The PRISM approach
In this section, we describe the experimental setup and datasets for the empirical evaluation of our at- tacks in real-world scenarios (medical in $5.1; le- gal in $5.2). Code for this project is available at git com/ wwt//eoee//aaeeeeeeeeeee Beee///eaaaaaaeeeeeeeeeeeeeaaeee
Recall the example of Jane Doe in Section 3.2, where the attacker could mistakenly infer that Jane is a smoker sim- ply because she has osteoarthritis. LR-Attack may suf- fer from such false positives: we cannot fully distinguish whether y" is associated with S specifically because Jane's personal record is in D, or because individuals with in their samples generally also include y.
5.1. Model Instantiation: Medical Summarization
Medicine is a highly sensitive domain, where language mod- els are actively being trained on privately identifiable infor- mation (Das et al., 2024). Thus, we assess the efficacy of our attacks on a medical summarization task. We use the MTS- Dialog dataset (Abacha al., 2023; Yim et al., 2023; Han et al., 2023), which includes 1,700 doctor- ceator-ppateet dialogues, with corresponding summaries. To construct fragment sets for our adversarial public information assumption, we run a biomedical Named Entity Recognition (NER) tool from SciSpacy' on each medical dialogue. Our fragments are then medically specific terms and phrases (e.g., "constipation" or "acute cholecystitis"; see Appendix D for examples). We filter out dialogues without any extracted entities, leaving us 948 train, 69 validation, and 312 test samples. This pro- cess yields a variety of types of fragments for each sample, including medical conditions, drug names, and symptoms. Evaluating the PIFI threat model means completely disre- garding the original dialogues and focusing exclusively on
To remedy this, we consider how to incorporate addi- tional "world model" probability, Pworias into our likelihood score. We observe that one way to do this is to take a prior on sample membership, and try to derive the proba- bility of interest. Suppose D* ~ D is an unknown dataset used to train a language model, and let us write S short for PSompt(S) here for clarity of presentation. Then, the probability that fo assigns to y' given S, conditional on s E D", can be expanded
Pr(fo.D- (y' 8) Is (s
Pr(fa.D- (y S)) Pr(fo.D (y" (S] | PT) & D*)
Pr(s D*)
If D' = D, then Pr(Jo.D-Xx (y* | S) s & D*) can be approximated by Po'. Similarly, we approximate Pr(fe.D- (y- S)) with Proria. using queries to several "world" models that capture the general association between S and y". It remains estimate Pr(s DD*). We connect this to our AR-Attace statistic, i = pp//pn. We assume L is strongly correlated with Pr(s E D) - an approximation
We use en an NER model trained on the BC5CDR corpus, which contains annotated chemicals and dis- cases, https://allens github. 10/ /et//cttapat spacy/.
5

---

Partial-Information Fragment Inference from LLMs
the medical fragment sets for inference.
(Wei et al., 2021). Specifically, the models used as target and shadow models in the present work include: Llama- 1.1-8B-Instrue & 1--ma---2--B-Instruee Qwen-2-7B- Instruct, and Nisttrl-7B-Insttuutt-b.. See Appendix F for more details on these models, and a discussion of our construction of the world model, for which we employ three LLMs not fine-tuned on either or D'.
We are careful to include a variety of false positive examples in our testing; see Figure 6 for an overview of attack failure modes. To construct negative examples, we sample a second disjoint set of unseen fragments, i.e., fragments from notes that did not appear in the fine-tuning subset and which are not part of the samples that we test as true positives. The at- tacker must distinguish between true positive y' fragments (i.e., this y" co-occurred alongside fragments from S in Prompt(S), given the from which S was derived is in the training data D) and false positive fragments (anything else). Light experimentation led us to believe that the attack is not particularly sensitive to the choice of Prompt(-), so long as describes the fine-tuning task (i.e., says something about a summary). We thus formulate a consistent, task-appropriate Prompt(-) follows: Consider a patient whose medi cal summary contains: (diabetes, hypertension, ... That individual's s medical surmary also includes: {y"}. By inserting fragments, we obtain output probabilities from each model which serve as inputs for our LR-Attack and PRISM attacks, as well as our Classifie: baseline.
Fine- MunnnT We employ two approaches to fine-tuning the target model and shadow model LLMs for our experi- ments. For Llama-3.1 1--na-3.--B--nsttuucc, Qwen-2-7B-Instruuct and Mistral-7B--nstrrutt-O... we em- play full fine-tuning: updating the weights of the model itself. To test the robustness of our method to LoRA, we also quantize Llama-3. 188--Insruut to FP8 precision and fine-tune the model using low-rank adaptation (LoRA) (Hu ct al., 2022: Dettmers et al., 2023). To further test the ro- bustness of our method, we consider both models that have seen the data only once (i.e., undergone epoch of fine- tuning) and models that saw the data repeatedly, until loss convergence (e.g., were fine-tuned for 10 or more epochs).
Evaluation Metries Similarly to membership inference, attacks with high false positive rates are not meaningful in practical settings (Carlini et al., 2022a). Thus, we evaluate the effectiveness of our PIFI attacks by closely examining the Receiver Operating Characteristic (ROC) curve. The ROC curve demonstrates trade-offs between TPR and FPR for all possible choices of decision thresholds т, while the Area Under the Curve (AUC) metric integrates over the ROC curve aggregate performance across FPRs. We do consider overall AUC, but specifically focus on the true pos- itive rate at fixed, low false-positives rates (TPR @ %FPR, e.g., 2% and 5%). Additionally, we use a log-scale to dis- play the ROC curve, which follows Carlini et al. (2022a)'s s example and highlights results from the low FPR regime.
5.2. Model Instantiation: Legal Setting
To evaluate the effectiveness of our attacks in another domain, we experiment with legal case summarization task. We use legal data from the Free Law project", filtering for sentencing/ / criminal possession data using the built in Nomic topic modeling tool. We extract criminal behaviors (e.g., criminal possession of a weapon) and personal identi- fiable information (e.g., five years in prison) as fragments. Ultimately, we have 748 train, 188 validation, and 235 test samples. The attack procedure is identical to the medical setting. We adopt the following prompt for the legal task: Consider an individual whose legal document summary contains the following criminal behaviors and identifiable information: fcriminal possession of a weapon, J..}. That individual's legal document summary then also includes: (y*). We expected the legal domain to be more difficult for PIFI; legal language overlaps more with everyday language than medical, which is a known challenge under other threat models (Lukas et al., 2023; Duan et al., 2024).
Expectations for Success The PIFI threat model relies on strictly weaker adversarial assumptions (i.e., no complete samples) when compared to direct memorization attacks. Prior ablation results those attacks have suggested they are sensitive to conditioning versions of the target sample prefix (Ippolito et al., 2022; Lukas et al., 2023). As PIFT is analogous to strongly ablating a sample prefix, we do not expect extremely high TPRs in most settings. Nevertheless, even a modest TPR (at a very low, fixed FPR, like 2%) can have real impact applied at scale, as an attacker would likely query large numbers of individuals while being conservative with their inferences.
approximate or permuted
6. Models & Evaluation
Models We study our attacks using four autoregressive, decoder-only large language models (Radford, 2018), each of which is pretrained on large text corpus and subse- quently instruction-tuned to respond to user instructions
7. Results & Discussion
We present the results of applying the attacks enabled by our partial-information fragment inference (PIFT) threat
http:e////hauiiggscccceoeeeeeeee free- www. cor
6

---

Partial-Information Fragment Inference from LLMs
Method
Proh. Access
TPR @ 2% FPR
TPR @ 5% FPR
ROC-AUC
PD
Pn'
Pvorid
Liana 3 8B
Qwen 2 7B
Mistral 7B
Liana 3 SB
Qwen 2 7B
Mistral 7B
Llama 8B
Qwen 2 7B
Mistral 7B
Classifier
6.3%
8.2%
5.5%
14.9%
16.3%
12.8%
0.68
0.73
0.68
LR-Attack
5.3%
9.5%
2.9%
10.6%
16.2%
8.8%
0.64
0.67
0.59
PRISM
4.4%
4.3%
4.2%
11.5%
11.0%
11.3%
0.67
0.66
0.63
Table 1. Main Results Across Models (Llama, Qwen, Mistral), for LLMs fine-tuned to convergence on the medical summarization task. Color-coding is Olympic medal standard by column (Gold is best, Silver second, Bronze third).
Llama 8B (1 Epach, medical)
Llama 3 8B (Cons. medical)
7.1. (TI) Attacks Are Effective Across Diverse Models
0.5
We attack three LLMs 1--8B Qwen-2-7B, and Mistral-7B-vv02) instruction- for medical note sum- marization with - tack, PRISM, and Classifier under the PIFI threat model. As shown in Table 1, all three attack methods exceed a random- guesss baseline by considerable margin (doubling and sometimes quadrupling TPRs at low FPRs), indicating that the PIFI threat model is effective against three families of LLMs.
1994
0.02 0.05 0.1
TP%
0,02 0.05 U.1
Classifier (ALC 0.06)
Classlarr (AUG- 0.85)
LRAtlack 0.54)
LR.Atack (AUC 0.641
PRISM (AUC- 0,681
PRISM IADC- 0:57)
0.01
0.02
0.05 0.1
0.5
0.01
0.02
0.05 0.1 FPR
0.5
FPR
Figure 3. Left: single epoch fine-tuned L 8B model. Right: convergence fine-tuned Llama-3 8B model. More fine-tuning consistently increases the success rates of PIFI attacks.
7.2. (T2) Repeated Exposure Increases Vulnerability
Next, we examine whether increased data exposure is posi- tively correlated with P PIFI success when attacking a model. We compare models fine-tuned for a single epoch (model updates based on a single pass on the data) against those fine-tuned for ten or more epochs (i.e., many passes, to convergence). Figure 3 shows that more fine-tuning con- sistently increases the success rates of PIFI I attacks. This aligns with prior studies of membership inference (Carlini et al., 2021; 2022a) in LLMs, which found that repeated exposure to a sample increases the chances of memorization. However, even a single epoch of fine-tuning is sufficient to achieve a non-trivial TPR, demonstrating privacy risks posed by even lightly fine-tuned LLMs.
Llama 6B
Llama 8B LoRA Finetuned (Conv., medical)
(Cone, medical)
0.5
1998
0.05 0.1
TP%
0.02 0.05 0.1
Chaslllee (AUC 0.000
Classtiier (AUA) - 0.66
0.02
1R-Atack (AtI)
PRISM (ACC 0,871
PRISM 0:7)
0.01
0.02
0.05 0.1 0.5
0.01
0.02 0.05 0.1 FPR
0.5
FPR
Figure 4. Left: fully fine-tuned Llama-3 8B model. Right: LoRA fine-tuned Llama-3 8B mode. LoRA-fine-tueed models exhibit less vulnerability than their fully fine-tuned counterparts.
Llama 3B LoRA Finetuned
Llama 8B LoRA Finetuned (Cons, medical)
(Cony, medical)
7.3. (T3) LoRA Fine-Tuning Still Leaks Information
We also investigate whether parameter-efficient methods like LoRA (Hu et al., 2022) mitigate fragment-level leakage. In keeping with prior work (Jiang et al., 2024). Figure 4 demonstrates that nerability than fully fine-tuned counterparts. However, the difference TPR @ FPR modest, and PRISM SM retains high recall, indicating some memorization has still occurred.
1998
0.02 0.05 0.1
0.02 0.05 0.1
models exhibit less vul-
Chattfle (AUC 0.62)
Classtieer (AUC 0.60
18vkuark
PRSM (AUIC dxa))
PRISM (AUCc datt)
0.01 0.02
0.05
0.1
0.5
0.01
0.02 0.05 0.1
0.5
FPR
FPR
Figure 5. Left: LoRA fine-tuned Llama-3 3B model. Right: LoRA fine-tuned of the Llama-3 8B model. Larger parameter counts confer greater capacity for memorizing training examples.
7.4. (T4) Vulnerability of Larger Models
Comparing models across parameter scales within the same family reveals that the larger models are more vulnerable to PIFI attacks, as we obtain higher AUC for an 8-billion pa- rameter Llama model than for a 3-billion parameter Llama model when both are fine-tuned for 10 epochs, as shown in Figure 5. This outcome is again consistent with prior work demonstrating that larger parameter counts confer greater capacity for memorizing training examples (Car-
model the medical summarization task (mostly deferring results on the more difficult legal task to the appendix), focusing on factors that most influence fragment-level in- ference. We report results for IR-Attack, PRISM, and Classiffi fiee attacks, providing evidence for substantial privacy risk across model families and hyperparameter set- tings. We structure results as takeaways, denoted T#.

---

Partial-Information Fragment Inference from LLMs
lini et al., 2021). We also evaluated our attack on 70B- parameter Llama model LoRA-fine-tuned for 10 epochs (the largest model we could finetune given compute con- straints). The comparison of its performance with the 8B Llama model is visualized in Figure 10. While large L.MSs deliver improved performance, their higher memorization capacity also raises the risk of fragment-level data leakage.
Method
TPR @ 2% FPR
TPR @ 5% FPR
ROC-AUC
Classifier LR-Attack
10.8%
13.7%
0.65
17.5%
34.4%
0.77
PRISM
2.8%
4.2%
0.57
Classifier
7.1%
17.0%
0.74
T.R-Attack
2.5%
5.3%
0.61
PRISM
5.6%
13.5%
0.73
Table 2. (Top) Attack performance when considering fragments that occur once in the fine-tuning data ("rare" set). (Bottom) Performance when considering fragments that occur mulriple times in the fine-tuning data ("common" " set) (Note: model attacked convergence fine-tuned Llama-3 8B model).
7.5. (T5) Noising the Prompt Has Minor Impact
Introducing noise in the form of inaccurate tokens does not significantly degrade the performance of our attacks. We find that, for a Llama-3-8B model fine-tuned for one epoch, replacing 75% of the accurate fragments possessed by the adversary with inaccurate fragments (corresponding to false assumptions on the part of the adversary) reduces AUC from .68 to .65 for PRI SM and from .66 to .61 for Classi fier, while leaving LR-Attack unchanged at .54. We provide additional detail in Appendix B.2.
When the candidate fragment is unique, the simple likelihood-ratis statistic of LR-Attack is markedly more sensitive (TPR@ @ 2%FPR = 17.5%). Because such tokens are nearly absent in the shadow model's 's fine-tuning, the ratio Po/Po: spikes, yielding a strong signal. But, as fragment frequency rises, many individuals in D share the same to- ken. Here the world-model prior in PRI SM suppresses false positives and overtakes the other methods (AUC 0.73), reflecting its design goal of balancing sensitivity with preci- sion in classifier remains competitive across both strata, one of the two data-blind attacks always matches or exceeds its recall at low-FPR underscoring that meaningful leakage persists even without labeled in-distribution examples. Taken to- gether, these results corroborate the complementary nature of the two data-blind attacks: LR-Attack excels when the adversary probes for idiosyncratic, sparsely represented fragments, whereas PRISM is better suited for fragments that occur more widely.
7.6. (T6) Legal Setting Attacks Are More Challenging
The legal domain presents a more difficult challenge for our methods. Though the attacks still outperform chance (often by 2:1 TPR:FPR ratio), AUC for a Qwen2-7B- Insttrcct model fine-tuned for 10 epochs falls .55 for PRISM, to .61 for LR-Attack, and to .59 for Classifier. with similar declines observed for a Llama-3 model. Note that this finding accords with expectations, as outlined in Section 5.2. We provide additional detail in Appendix B.3.
regimes. Although the data-aware
7.7. (T7) LR-Attack Excels for Rare Fragments, PRISM for Common Ones
We find that L.R-Attack performs best for rare fragments (e.g.. "brachial plexopathy"), while PRISM performs best for more common fragments (e.g.. "parkinsons"). This sug- gests that incorporating a prior from the world model helps to eurb false positives and improve precision, an important consideration for likelihood ratio P.D. of te-Attack increases sharply for rare data, rendering the method more sensitive in these cases. We provide additional detail in Appendix B.4.
Potential of DP Fine-Tuning as a Defense Differentially private (DP) fine-tuning of LLMs (Yu al., 2021; Li et al., 2021) has been proposed as strategy to defend against privacy vulnerabilities in LLMs. Recent work shows that DP fine-tuning helps to protect against memorization in LLMs, and that sentence-level privacy can help to reduce (but not eliminate) PII leakage in LLMs (Lukas et al., 2023).
fragments. However, the
pn'
To that end, we additionally fine-tuned the Llama-3.2-31B Instruct model with sample-level differential privacy (DP).* 4 Below we report results with 3, common setting to balance utility and privacy. In terms of utility, Table 3 shows how DP fine-tuning improves over the base model, but does not fully recover the gains of non non-private fine-tuning, in line with prior work (Lukas et al., 2023). terms of de- fense against the PIFI I threat model, while Classifier and PRISM roughly double the performance of random guessing, DP suppresses the performance of LR-ATTACK
7.8. Additional Experiments
Rare vs. Common Fragments To clarify why the ranks of the three attacks sometimes trade places in Table 1, we stratify the 1,034 candidate fragments in the medical task by their corpus frequency.² 3 Exactly 47% of the fragments appear once in the entire training set, while the remainder surface two or more times. Table 2 reports attack perfor- mance on these two disjoint partitions.
Frequency is computed on the fine-tming split only, mirroring what an attacker would implicitly exploit.
"Ten epochs, opacus set to achieve under (c, 10-5 uses dp-transformers library (Wutschitz et al., 2022).
8

---

Partial-Information Fragment Inference from LLMs
Metric
Base model
DP (c = 3) / Non-DP
the concatenation [p || ||s] is in D and Sa produces when decoded greedily under prompt p (Carlini al., 2022b). Modifications to extraction attacks may seek to alter an LLM's text generation strategy to more easily extract data. Yu et al. (2023) benchmark techniques to enhance extraction attacks, employing techniques like top-k, nucleus, and typi- cal sampling to address distribution discrepancies, resulting in substantial improvements in data extraction. Similarly, Nasr et al. (2023) design attacks that can cause production model like ChatGPT to diverge from its instruction-tuned objective and emit training data at a higher rate.
ROUGE-Luu 1
0.0963
0.0969
0.1004
BERTScore FI
0.7140
0.7186
/ 0.7299
Method
TPR @ 2% FPR
TPR @ 5% FPR
ROC-AUC
Classifier
4.4%
10.0%
0,64
LR-Attack
0.9%
2.4%
0.51
PRISM
4.0%
9.6%
0.54
Table (Top) Summarization quality on the medical task. (Bot- tom) PIFI results on Alama-3.2-3BB Instructt model fine-tuned with DP-SGD, showing some vulnerability.
Membership Inference Membership inference attacks at- tempt to learn whether a given sample was present in the training data for a machine learning model (Shokri et al., 2016). Prior work demonstrates that LLMs are vulnerable to membership inference attacks both after initial pre-training (Duan et al., 2024) and after additional fine-tuning (Fu et al., 2023). More complex membership inference attacks may leverage additional information about a model, such as info gained using model-stealing techniques (Wang et al., 2024). In addition to their relevance from a privacy perspective, MI attacks are used in analyses of model memorization (Shi et al., 2023) and test set contamination (Oren et al., 2023).
Method
TPR @ 2% FPR
TPR @ 5% FPR
ROC-AUC
Classificr
7.0%
13.4%
0.69
LR-Attack
5.3%
10.6%
0.64
PRISM
5.2%
11.6%
0.70
Table 4. Results using DeepSeck for world probabilities.
(TPR@2% FPR= 0.9%).
Improved World Models We experimented with using world probabilities from two large open-source models, DeepSeek-V3. V3 and RI (averaging the two). Table gives the performance of our attacks on the Illam-3-8B model (with 10 epochs of fine-tuning) using DeepSeek world prob- abilities. We observe slight improvements each of our three attack metrics compared to results obtained when us- ing the three smaller L.Ms as world models. These results suggest that incorporating additional high-quality models in the world model ensemble would further enhance the per- formance of our attacks. However, as noted previously, the log-probabilities for powerful closed models are generally not available or significantly restricted, which limits our ensemble to open-source models.
9. Limitations
We acknowledge that this work does not extensively eval- uate the efficacy of DP fine-tuning against PIFI attacks; a promising line of future work would be large scale ex- ploration of the defensive effectiveness of DP-fine-tuiing. However, while DP fine-tuning may be effective for privacy, this approach degrades the utility of the fine-tuned model, sometimes substantially (Mireshghalaah et al., 2021). Ad- ditionally, DP fine-tuning of L.MMs remains uncommon in practice due to the significant technical and financial bur- dens - e.g. necessitating -RMM GPUs and large batch sizes, even with gLoRA. Consequently, assessing PIFI with- out DP is both reasonable and practically important, since the vast majority of fine-tuned LLMs lack DP defenses.
8. Related Work
The PIFI threat model builds on prior research on LLM privacy. Here we consider related work, focusing on memo- rization and membership inference attacks predominant approaches in the field.
10. Conclusion
Memorization LLMs may produce sensitive data through user interactions; for example, by prompting a model with "Jane Doe's social security number is...' one might elicit a social security number (Carlini et al., 2019). This rep- resents memorization attack: formally, given model fe and prefix x, can we extract target sequence that we believe is in the training data exactly? Work by Carlini et al. (2021) introduced this concept of memorized extractabil- ity as k-eidetic memorization, where k gives the number of times s appeared in D. In subsequent work, a similar definition was given: s is considered extractable if given k tokens of context there exists a length-k' string p such that
We introduce a novel threat model PIFT, which provides a realistic approach to fragment-level extraction from LLMs. The LR-Attack and PRISM attacks we construct under the data and model availability assumptions of PIFI re- veal the vulnerability of LLMs to extraction of private data, even when an attacker possesses only fragmented and/or unordered data. Our work suggests a need for research on LLM privacy that adopts weaker (and thus more realis- tic) assumptions about scenarios in which these models are vulnerable privacy attacks.

---

Fragment Inference from LLMs
Impact Statement
Conmy, A., al. Stealing part of production language model. arXiv preprint arXiv:2403.066.: 2024.
Our work advances the science of LLM privacy attacks by introducing new threat model and two novel attacks on chat-based language models, pointing to the vulnerabilities embedded in an increasingly common form of digital infras- tructure. The intended impact of this research is equip organizations and individuals training language models with a better understanding of their vulnerabilities, and to ad- vance the knowledge available to the research community in producing effective methods to circumvent these attacks. We acknowledge that our methods may equip real-world bad actors with the ability draw private information from publicly available LLMs. However, suggest that fore- warned is forearmed: these vulnerabilities likely already exist in many production language models, and our work can help responsible organizations to prepare an appropriate defense.
Das, B. C., Amini, M. H., and Wu, Y. Security and privacy challenges of large language models: A survey. ACM Computing Surveys, 2024.
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.1431/1 2023.
Dionysiou, A. and Athanasopoulos, E. Sok: Membership inference is harder than previously thought. Proceedings on Privacy Enhancing Technologies, 2023.
Duan, M., Suri, A., Mireshghallah, N., Min, S., Shi, W., Zettlemoyer, L., Tsvetkov, Y., Choi, Y., Evans, D., and Hajishirzi, H. Do membership inference attacks work on large language models? arXiv preprint arX(v:2402.07884, 2024.
References
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Abacha, A. B., Yim, W.-w., Fan, Y., and Lin, T. An empiri- cal study of clinical note generation from doctor- pattiet encounters. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 2291-23022 2023.
Fu, W., Wang. H. Gao. C., Liu, G., Li, Y. and Jiang, T. Practical membership inference at- tacks against fine-tuned large language models via self-prompt calibration. ArXiv, abs/2311.06062. 2023. URL https://aag http://appp..eaaiiiesseeemeeeess semanticscholee 2rs//orrpsDD::251288788
Antoniak, M., Naik, A., Alvarado, C., Wang, L. L., and Chen, I. Y. NIp for maternal healthcare: Perspectives and guiding principles in the age of llms. In The 2024 ACM Conference on Fairness, Accountability, and Trans- parency, pp. 1446-1463, 2024.
Grynbaum, M. and Mac, R. The times sues openai and microsoft over a.i. use of copyrighted work. The New York Times, Dec 2023.
Carlini, N., Liu, C., Erlingsson, U. Kos, J., and Song, D. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX se- curity symposium (USENIX security 19), pp. 267-284, 2019.
Han, B., Zhu, H., Zhou, S., Ahmed, S., Rahman, M. M., Xia, F., and Lybarger, K. Huskyscribe at mediqa-sum 2023: Summarizing clinical dialogues with transformers. CLEF (Working Notes), pp. 1488-1509, 2023.
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert- Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2633 (633-2650, 2021.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint arXiv:2009. 00000000030,....0000000000000000 2020.
Carlini, N., Chien, S., Nasr, M., Song, S., Terzis, A., and Tramer, Membership inference attacks from first prin- ciples. In 2022 JEEE Symposium on Security and Privacy (SP), pp. 1897-1914. IEEE, 2022a.
Hu, E. J., Shen, Y., Wallis, P.. Allen-Zhuu Z., Li, Y.. Wang, S., Wang, I.., and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https: openreview net t 1iiiidd
Carlini, N., Ippolito, D., Jagiclski, M., Lee, K., Tramer, F., and Zhang, C. Quantifying memorization across neu- ral language models. arXiv preprint 0724:22002.07646, 2022b.
Ippolito, D., Tramèr, F., Nasr, M., Zhang. C., Jagielski, M., Lee, K., Choquette-Chooe C. A., and Carlini, N. Prevent- ing verbatim memorization in language models gives a false sense of privacy. arXiv preprint arXiv:22 2022.
Carlini, N., Paleka, D., Dvijotham, K. D., Steinke, T.,
Hayase, J., Cooper, A. F,, Lee, K., Jagielski, M., Nasr, M.,
10

---

Partial-Information Fragment Inference from LLMs
Jagannatha, A., Rawat, B. P. S., and Yu, H. Membership in- ference attack susceptibilitty of clinical language models. arXiv preprint arXiv:2104.08303 2021.
Morris, J. X., Zhao, W., Chiu, J. T., Shmatikov, V., and
Rush, A. M. Language model inversion, 2023.
Nasr, M., Carlini, N., Hayase, J., Jagielski, M., Cooper, F., !ppolito, D., Choquette- Choo, C. A., Wallace, E., Tramèr, F., and Lee, K. Scalable extraction of training data from (production) language models. arXiv preprint arXiv:2311.17033 2023.
Jiang, A. Q. Sablayrolles, A., Mensch, A.. Bamford, C.,, Chaplot, D. S., Casas, D. d. I., Bressand, F., Lengyel, G., Lample, G., Saulmier, L., et al. Mistral 7b. arXiv preprint TrXiv:2310.066: 2023.
Jiang, T., Huang, S., Luo, S., Zhang, Z., Huang, H., Wei, F., Deng, W., Sun, F., Zhang, Q. Wang, D., and Zhuang, F. Mora: High-rank updating for parameter-efficien fine-tuning. ArXiv, abs32405. 2024. URL https://ap...eeaniicciiii semaamicschhhhll.. ptc/CozpusID:2699199
Neyman, J. and Pearson, E. S. The testing of statistical hypotheses in relation to probabilities a priori. In Math- ematical proceedings of the Cambridge philosophical society, volume 29, pp. 492-510. Cambridge University Press, 1933.
OpenAL. Introducing chatgpt. index/ chatgpt/ 2022. [Accessed 19-01-2024).
com/
Ke, G., Meng, Q.. Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q. and Liu, T.-Y. Lightghm: A highly effi- cient gradient boosting decision tree. In Neural Informa- tion Processing Systems, 2017. URL https: / /api. semanti www.ccschlll.oo ar aalla...r. .rrg Cctp/Corrpuppsaa : 5895.
Oren, Y., Meister, N., Chatterji, N. S., Ladhak, F,, and Hashimoto, T. Proving test set contamination in black box language models. ArXiv, abs/2310. 2023, URL https: /apii. semanticscholar. org
Li, X. Tramèr, F. Liang, P., and Hashimoto, T. B. Large language models can be strong dif- ferentially private learners. ArXiv, abs/2110.05679, 2021. URL attps://apps /ppi. attps///pi..emaatiiceeeeee.. semanticscholar. Drr/Corpus 22863328822
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., al. Training language models follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744 2022.
Lukas, N., Salem, A., Sim, R., Tople, S., Wutschitz, L., and Zanella-Béguelin, S. Analyzing leakage of personally identifiable information language models. In 2023 IEEE Symposium on Security and Privacy (SP). pp. 346- 363. IEEE, 2023.
Poddar, S., Wan, Y., Ivison, H., Gupta, A., and Jaques, N. Personalizing reinforcement learning from human feedback with variational preference learn- ing. ArXiv, abs/2408.10075. 2024. URL https: / / //api. www.ssmmcacicchoole.....o wwt.oooro//ooooooie 271904022.
Mattern, J., Mireshghallah, F., Jin, Z. Schölkopf, B., Sachan, M., and KirkKKirkpatrick, T. Membership infer- ence attacks against language models via neighbourhood comparison. arXiv preprint 23055 /8462, 2023.
Radford, A. Improving language understanding by genera- tive pre-training. 2018.
McDuff, D., Schaekermann, M., Tu, T., Palepu, A., Wang, A., Garrison, J., Singhal, K., Sharma, Y., Azizi, S., Kulka- mi, K., et al. Towards accurate differential diagnosis with large language models. arXiv preprint arX1v:2312.00164, 2023.
Shi, W., Ajith, A., Xia, M., Huang, Y., Liu, D., Blevins, T., Chen, D., and Zettlemoyer, L. Detecting pretrain- ing data from large language models. arXiv preprint 20. 2023.
Shokri, R., Stronati, M., Song, C., and Shmatikov, Membership inference attacks against machine learn- ing models. 2017 IEEE Symposium on Security and Privacy (SP), pp. 3-18, 2016. URL https://aapp semanticscholat org/ Cttp///rpppcoooeeeeeee :00.s1II::0888755
Mireshghallah, F., Inan, H. A., Hasegawa, M., Ruhle, V., Berg-Kirkpatrick, T., and Sim, R. Pri- vacy regularization: Joint privacy-utility optimiza- tion in languagemodels. In North American Chap- ter of the Association for Computational Linguistics, 2021. URL https: /api..pp./// semanticscholar: htc//CorpuSID:2223333
Small, Z. Sarah silverman sues openai and meta over copy- right infringement. The New York Times, Jul 2023.
Mireshghallah, F., Goyal, K., Uniyal, A., Kerg-Kirkpatrick, T., and Shokri, R. Quantifying privacy risks of masked language models using membership inference attacks. arXiv preprint arXiv:2203.0300 2022.
Tang, L., Laban, P., and Durrett, G. Minicheck: Elfi- cient fact-checking of llms on grounding documents. Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing. Association

---

Partial-Information Fragment Inference from LLMs
for Computational Linguisties, 2024. URL https:
R., Lin, R., Wang, S., Bai, S., Tan, S., Zhu, T., Li, T.,
//axxiv. .rg. http:/dd//224CC 10774.
Liu, T., Ge, W. Deng, X., Zhou, X., Ren, X., Zhang,
X., Wei, X., Ren, X., Liu, X., Fan, Y., Yao, Y., Zhang,
Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C.,, Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahri- ari, B., Ramé, A., et al. Gemma Improving open language models at practical size. arXiv preprint arXiv:2408.00 00/ 18, 2024.
Y. Wan, Y., Chu, Y., Liu, Y., Cui, Z., Zhang, Z., Guo,
Z., and Fan, 7. Qwen2 technical report, 2024. URL
https:
Yim, W.-w,, Abacha, A. B., Snider, N., Adams, G. and Yetisgen, M. Overview of the mediqa-sum task at image- clef 2023: Summarization and classification of doctor- patient conversations. CLEF, number, pp. 2023.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Ham- bro, E., Azhar, F,, Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation language models, ArXiv, abs/2302.13971, 2023. URL https:/ api semanticscholl. / ID: 25721 9404.
Yu, D., Naik, S., Backurs, A., Gopi, S., Inan, H. A., Ka- math, G., Kulkarni, J., Lee, Y. T., Manoel, A., Wutschitz, L., Yekhanin, S., and Zhang, H. Differentially private fine-tuning of language models. ArXiv, abs/21 10.022000 2021. URL https: semanticscholar. o org/ /
Wang. J. G., Wang, J., Li, M., and Neel, S. Pandora's white- box: Increased training data leakage in open llms. arXiv preprint ar arXiv:2402. 17122, 2024.
Yu, W., Pang, T., Liu, Q., Du, C., Kang, B., Huang, Y., Lin, M., and Yan, S. Bag of tricks for training data extraction from language models. In International Conference on Machine Learning, pp. 40306-40320. PMLR, 2023.
Wei, J., Bosma, M., Zhao, V., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners. ArXiv, abs/2109.01652, 2021. URL https: appi. semanticscholoe. org/ Drg/CozpuSID:: 237416585.
Zhou, S., Xu, Z., Zhang, M., Xu, C.,, Guo, Y., Zhan, Z., Ding, S., Wang. J., Xu, K., Fang, Y., et al. Large language models for disease diagnosis: A scoping review. arXiv preprint 1xXiv:2409.00099 2024.
Wolfe, R. and Mitra, T. The impact and opportunities of generative ai in fact-checking, In The 2024 ACM Confer- ence on Fairness, Accountability, and Transparency, pp-
1531-1543, 2024a.
Wolfe, R. and Mitra, T. The implications of open generative models in human-centered data science work: A case study with fact-checking organizations. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 7, pp. 1595-1607, 2024b.
Wu, X., Zhao, Y., Zhang, Y., Wu, J., Zhu, Z., Zhang, Y., Ouyang, Y., Zhang. Z., Huimin, W., Lin, Z., et al. Med- journey: Benchmark and evaluation of large language models over patient clinical journey. In The Thirt Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
Wutschitz, L., Inan, H. A., and Manoel, A. dp-transformers: Training transformer models with differential pri- vacy. https: http::/wwww.mmirosteoo htapearrch/pppoeeeeeggtg-earrroo gust 2022.
cwm/enee//////e en-u//
Au-
Yang, A., Yang, B., Hui, B., Zheng. B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., Wang, J., Yang, J., Tu, J., Zhang, J., Ma, J., Yang, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J. Dang, K., Lu, K., Chen, K. Yang, K., Li, M., Xue, M., Ni, N., Zhang, P., Wang, P., Peng, R., Men, R., Gao,
12

---

Partial-Information Fragment Inference from LLMs
A. Experimental Setup
illustrate our experimental setup below. Where a sample s the target dataset and the target fragment y" the sample s, attacks under the PIFI threat model can succeed by successfully identifying the fragment y". Scenarios where an attack does succeed under PIFI include those where a target fragment y" identified but y S, or s not
in D, is not in D and y" is not in
3 using girea sccess to fen
["hypartandian"
wwwteaatthrttsstttoooo bl blockere??
Outcomes Secui X Takax Modar
Phonpe eith cmmp and non eurple fngeuem,, curput Red Food
#in
S: Dopertersson,, beta blockevsi
y" oioooarttiiiii
Dypecterudon, bets blackazsj
y" casces
notin
in
S: Driskwyy disenee, fever)
y" dasios
Figure 6. Successful and failed attack scenarios. An attack is only considered successful when the sequence the dataset D, the target fragment y" is in the sequence s, and we accurately infer the target fragment's 's presence in s. Any other scenario is considered as failed attack - (1) the target fragment NOT in sequence. (2) the sequence NOT the data (3) the sequence NOT the data D and the target fragment is NOT in the sequence.
A.1. Illustrative Examples
We probed 4,302 fragment instances for the medical summarization task, of which 1,034 were unique. The empirical frequency distribution of these fragments is highly skewed: the most common fragments were common medical terms, such as pain (124 occurrences), fever (69), shortness of breath (55), diarrhea (47), and chest pain (44). By contrast, 75% of all fragments appeared fewer than five times, and 47% occurred exactly once. These rare fragments include some highly specific conditions and medications, such as (selected at random) Vincristine, colitis, Baunorubicing Naprosyn, Xalatan, and lumbar spinal stenosis. Their rarity renders these fragments appropriate targets for the likelihood-ratie attack (Table 2). For more concrete examples, Table 5 presents five successes obtained using LR--Attcc and PRISM scores. In each case, the adversary begins with handful of publicly available fragments S and is able to infer an additional sensitive fragment y that indeed appears in an individual's 's record.
Public fragments provided to the model
Inferred (record)
fragment
anxiety disorder, arthritis, Morton's neurcmas,
hypothyroidism (train_679)
migraines
shortness of breath, Coumadin, lightheadedness, chest pain, Cardizem, pain, vertigo
atrial
fibrillation (crain.873)
toxicity, breast cancer, Ixempra, tumor, neuropathy, Avastin, cancer, Faslodex, Zometa, ixabepilone swelling, rashes, vomiting, pain
Aromasin (train.107)
seizures
(train_84)
weakness, headaches, stroke, sudden loss of consciousness, seizures, tremors
epilepsy (train.577)
Table Illustrative PIFI attack successes. Each row lists the attacker's public fragment set (left) and the private fragment y inferred, together with the record identifier (right).
13

---

Partial-Information Fragment Inference from LLMs
B. Additional Results
B.1. (TT) Attacks Are Effective Across Diverse Models (cont.)
We attack three LLMs (Llama-3.1-8B, Qwen-2-7B, and Mistral-7B) instruction-ture for medical note summarization. Table 1 in the main body presents the quantitative results, while Figure 7 shows the ROC curves for all models, allowing a more comprehensive inspection different FPR values. We observe that the PIFI threat model is effective against three families of LLMs.
Llama 3 8B (Conv., medical)
Qwen2 7B (Conv., medical)
Mistral 7B (Conv., medical)
50
0.5
TPR
0.02 0.05 0.1
TPR
0.02 0.05 0.1
7777
0.02 0.06 0.1
Classifirr (AUC- 0.68)
Classifier (AUC- 0.73)
Classiier (AUC 0.68)
LRAttack (AUC 0.64)
ARttttack (AUC 0.67)
IR-Attakk (AUC 0.59)
PRISM (AUC 0,67)
PRISM (AUC- 0.66)
PRISM (AUC. 0.63)
0.01 0.02
0.05
0.1
0.5
0.01
0.02
0.05 0.1
0.5
0.01
0.02
0.05 0.1
0.5
FPR
FPR
FPR
Figure 7. Left: fully fine-tuned Llama-3-88 model. Middle: fully fine-tuned Qwen2-7B model Right: fully fine-tuned Mistral-7B model. We observe that SM maintains consistent performance with respect to Classi fier and LR-At tack tcck for each model, indicating its robustness to different model families.
B.2. (T5) Noising the Prompt Has Minor Impact
Because our partial-information threat model relies on only a handful fragments, one might wonder whether, if an attacker includes incorrect or unassociated fragments, this would hurt attack performance. In our experiments, permuting the listed fragments (e.g., replacing true fragment set "hypertension, diabetes" with partially true set "hypertension, osteoporoxis") has negligible effect on attack success rates. Although prompts containing more true fragments achieve slightly higher AUC, the improvement is less pronounced than one might expect. This demonstrates that the fine-tuned model has become highly attuned to associations between individual pairs of fragments, and that it is robust to noise introduced by fragments in the prompt not associated with the individual.
Llama 3 8B
Llama 3 RB (50% Permuted, Epech, medical)
Llama 3 BB (25% Permuted, Epach, medical)
Llama 3 8H (1 Epoch, medicall
(75% Permuted, Epach, medical)
0.5
0.5
TPR
T7
0.02 0.00 0.7
0.02 0.05 TVR
TFR
0.02 0.05 0.3
0.02 0.08
Classffe CALC- - 3.61)
Classdar (ALCC 8.622
Classifue UALC 0.63)
Classtler (AUC 66
LR (AUC- - 0.54)
LK Attad 0.54)
LR Aitack (AUC - 0.55)
LR-Attack (AUI - 0.54)
PRISM (UCC 0.63)
PRISM (AUC
PRISM (AUC 0.67)
PRISM
0.01
0.02
0.05
0.1
0.5
0.01
0.02
0.05
0.1
0.5
0.01
0.02
0.05
0.1
0.5
0.01
0.02
0.05 0.1
0.5
FPR
FPR
FPR
FPR
Figure& 8. Results of an ablation wherein only k% of fragments used in the attack prompt are accurate, demonstrating that PRISM remains a viable method for an attacker who possesses only a small number of correct fragments. Results also show that PRISM outperforms other methods, including learned classifier, when LLM only lightly fine-tuned (i.e., for single epoch), suggesting the effectiveness of the prior incorporated by the attack.
B.3. (T6) Attacks in the Legal Setting Are More Challenging
We also validate our PIFI threat model in a legal case summarization task (see Section 5.2 for details). Here, the fragments often correspond to crimes committed or other identifiable information (see Appendix D for examples and categories). We find that our methods still outperform chance, often by 2:1 TPR:FPR ratio, in identifying whether a sensitive legal
14

---

Partial-Information Fragment Inference from LLMs
Liama (25% Permuted, Cons, medical)
(75% Permuted, Cons. medical)
(30% Permuted, Conk, medical)
(Conv, medicall
0.
0.5
TPR
TPR
0.02 0.05 0.1
TPC
TPR
0.02 0.03 0.1
0.00 0,05
0,00 0.05
Clossifee
Ll-ktackk (AUC-
LitAttack (AUC 0.66)
LR-Atteck (AUIC
Lit-Alteck (AUIC 0.64)
PRUSK (AUC - 0,56)
PRUSK (AUC 0,96)
PRUSK (AUC - 0.57)
PHUSK (AUC 0.57)
0.01 0:05
0.01
0.1
Figure 9. Results of our ablation wherein only k% of prompt fragments are accurate; the figure illustrates results for a Llama-3 model trained for 10 epochs, and attacked 25%, 50%, 75%, and 100% accurate fragments..
Liama 3B LoRA Finetuned (Conv, medical)
Llama 8B LoRA Finetuned (Conv., medical)
Llama 70B LoRA Finetuned (Conv., medical)
0.5
0.5
0.5
TPR
0.1
TPR
0.11
TPR
0.1
0.02 0.05
0.02 0.05
0.02 0.05
Classifier (AUC 0.62)
Classifier (AUC 0.64)
Classfier (AUC 0.67)
LR-Atack (AUC 0.46)
LR Attack (AUC 0.53)
LR-Atlack (AUC 0.64)
PRISM (AUC 0.68)
PRISM (AUC 0.67)
PRISM (AUI 0.65)
0.01
0.02
0.05 0.1
0.5
0.01
0.02 0.05 0.1
0.5
0.01
0.02
0.05 0.1
0.5
FPR
FPR
FPR
Figure 10. Left: LoRA fine-tuned Llama-3 3B model. Middle: LoRA fine-tuned of the Llama-3 8B model. Right: LoRA fine-tuned of the Llama-3 70B model. Larger parameter counts confer greater capacity for memorizing training examples.
text fragment was in the training set; however, the attacks are noticeably less successful than the medical context. We hypothesize that, because legal keywords are more frequent general pre-training data, the value of fragment-specific signal is diluted. Consequently, distinguishing target fine-tuned fragment from the baseline text distribution becomes harder. Nevertheless, P PIFT attacks remain feasible, indicating that L.LMs fine-tuned in the legal setting may leak partial information, even though the language distribution is closer to everyday speech.
15

---

Partial-Information Fragment Inference from LLMs
Qwen2 7B
Llama 8B
(Conv., law)
(Conv., law)
0.5
0.5
TPR
TPR
0.02 0.05 0.1
0.02 0.05 0.1
Classifier (AUC = 0.59)
Classifier (AUC = 0.55)
LR-Attack (AUC 0.61)
LR-Attack (AUC 0.58)
PRISM (AUC 0.55)
PRISM (AUC 0.51)
0.01 0.02 0.05
0.1
0.5
1
0.01 0.02
0.05 0.1
0.5 1
FPR
FPR
Figure 11. The legal setting proves more challenging for attacks. AUC does not exceed .61 in Qwen-2-7B or .58 in Llama-3-8b, with the strongest result obtained by LR-Attack both cases, suggesting that its sensitivity to unusual fragments particular benefit setting where most fragments are relatively frequently occurring words LLM pretraining data.
B.4. (T7) LR-Attack Excels for Rare Fragments, PRISM for Common Ones
We analyze how fragment frequency affects attack performance. For rare tokens token sequences (e.g., specialized diagnosis in medical dataset, lab value), the simple LRRAAtack often performs best, presumably because the likelihood ratio P2, spikes sharply when the model sees unusual fragment learned from training data (e.g.. "brachial Ppy plexopathy" or "darvocet", from the @Biologicalsstruttue text corpora (e.g., "alzheimer," or "parkinsons", from the "Disease disorder" "Disease_disorder" category), PRISM 's incorporation of prior from the world model helps curb false positives and improves precision. These observations support our claim that partial-informationn inference uniform across fragments: modeling both the in-domain shadow distribution and an external "world" distribution can improve performance higher- frqqueccyy tokens.
category). By contrast, when fragments are more common
16

---

Partial-Infoo Inforal-nfoormation Fragment Inference from LLMs
B-Blologimi stricture #p=89/126 (medical)
B-Coreforence 11/108 (medical)
B-Datailed description +F=61/336 (medical)
B-Diagnoatie pracedure +P=35/245 (medical)
0.5
0.1
TPR
0.02 0.05 0.1
IP%
0.02 0.05 0.1
TPR
0.05
0.05
0.02
Classiferr (AUC- 0.72)
Aitak 0.74)
Classifar (AUC - 0.7k)
AStaccck IAUC 0.65)
Classifar (AUC 0.65)
LR
0.02
Classifer (AUC - 0.54)
Attack IAUC 0.59)
PRIEM IAUC 0.09)
PRIEM IAUC 0.63)
0.01
0.02
0.05
0.1 FPR
0.5
0.01
0.02
0.05 0.1
0.5
0.01
0.02
0.05
0.1 FPR
0.5
0.01
0.02
0.05 0.1 FPR
0.5
FPR
B-Disease disurder
B-Lab value #P-8/103 (medical)
B. Mediccatio
B-Sign symptom #P=451/1 585 (medical)
99-1918866 (medical)
#F=53/117 (medical)
0.5 1
0.5
0.5
0.5
199
0.02 0.05 0.1
0.02 0.05 0.1
0.02 0.05 0.1
0.02 0,05 0.1
Classiterr (AUC- 0.73)
Classier (AUC- - 0.72)
Classiteer (AUC 0.700
Classifer (AUC- - 0.63)
LB Attack IAUC 0.66)
L2 Attack (AUC 0.86)
LB Attack (AUC - 0.05)
Attack IAUC 0.01)
(AUC
0.01
0.02
0.05
0.1
0.5
0.01
0.02
0.05
0.1
0.5
0.01
0.02
0.05 0.1
0.5
0.01
0.02
0.05
d.1
0.5
FPR
FPR
FPR
FPR
Figure/ 12. Our findings indicate that LR-Attack is more effective for fragments, while PRISM is more effective for common fragments. For example, PRISK outperforms LR LR-Attack for the names of diseases (bottom left, B-Disease B-Disease_disonnder) _disonder) which contains the commons names of diseases (like "parkinsons"), with AUC equal .72 compared AUC of .66 for LR-Attack. On the other hand, UR-Attack outperforms PR P IMM for fragments related to more technical biological structure terms (B-Riological structure, top left, which contains uncommon terms such as "brachial plexopathy").
C. Synthetic Setting
We compare the PRISM and LR-Attack partial-informatie extraction attacks in toy setting using a synthetic trigram language model. This allows to validate insights given in Section 4.3, as will allow ourselves access known distribution over tiny vocabulary.
Toy Data Let V= (vo, [.....;-[[]] be vocabulary of size | (in toy experiments, we use a small set of letters, V= (a, b, c, dd. the vocabulary.). We generate a target dataset D of size N, where cach s D is ordered sequence (rI, U2,,, Uz.) four letters, such as "abd", "aaa", "acb." We then create shadow dataset D' of the same size and distribution, ensuring that a controlled subset of samples does not appear in D', so that D' serves as an fout-of-distribution reference for the target s. Then we combine target set and shadow set as the world dataset: D., = @(D, D'). The world dataset represents a comprehensive knowledge set. We generate test dataset D, with the same parametrization (V, N, T) as D for evaluation.
of length T. For example, sequence with T = 3 letters could be any permutation of the three out of
Toy Models We fit three toy trigram models, fe.D. fa.p. fe. on each corresponding dataset. Each model estimates ((ug/0/-2 14-1) via counts of observed trigrams in the respective dataset. For example, if we observe "ab" ten times and "abb" three times, then probability P(bjab) = 0.3. To cover the case where trigram has occurrences, a small smoothing value le- is applied. Thus, we can get the probability of any sequence under of the models by multiplying the probabilities of all trigrams in the sequence: P(e
... . II', P(u (()))1)-20--1).
Synthetic Attack For each sequence s (v1, (2 .....,, Uz) E D,. we consider conditional length C, C
We randomly select C' vocabularies, without replacement, and in order, from (v), 02,.... 1z,-1) 1) as the conditional sequence. Then we append the last vocabulary uy to the new sequence as sc. The intuition behind this shorter conditional length is that we only have partial information of the sequence, which is being used to inform the probability of the whole sequence. In our setup, the shorter the conditional length, the harder the prediction task is. Then we calculate the probability of the conditional sequence sc from the three models: Pso.D, D. Pac.D,Psc.n Pse,D' Psc.D.. The LR-Attack attack calculated as described in Section 4.2, and the PRI SM attack is calculated as described in Section 4.3.
17

---

Partial-Information Fragment Inference from LLMs
Experimental Setup & Results We experimented with the following parametrization: |V] [4, 1, 6, T [4, 5). N [V](7-1) We report AUC in Figure 13. We observe that (1) when vocabulary size increases, the task is more difficult, because there are more permutations of different letters, increasing the volume of information. Fixing vocabulary size, increasing the sequence length makes the task more challenging, as longer sequences contain more complicated information. Fixing sequence length and increasing conditional length makes the prediction task easier, because a longer conditional sequence provides more information, and (2) other than the case of (V), T, ()=,,,,,) PRISM almost always outperforms the ARR-Attack method. The effect more salient when conditional length is longer, i.e., when probabilities used for the attack are conditioned on more information.
LBAAtack
PHISII
Seg Length
Wbrab Size
Sag Length 4| Voral Size
Sag Length Marab Sioe
Seq Length 4| Merah Size
080
Seg Eength Size
Seg Zength Size
Tength
Length
0.70
0.60
0.50
Coed Length
Coadd Langth
Coadd Tength
Gred Length
Figure 13. Synthetic attack results. AUC scores are visualized for two methods, Each square shows the result of parametrization.
18

---

Partial-Information Fragment Inference from LLMs
D. Data Examples
Medical Note Summarization Example
System:
You are helpful medical assistant! Please help me summarize dialogues between doctors and patients.
User:
Doctor: What brings you back into the clinic today, miss? Patienr: I came in for a refill my blood pressure medicine.
Doctor: It looks like Doctor Kumar followed up with you
last time regarding your hypertension, osteoarthritis
ostcoporosis hypothyroidism, allergic rhinitis and kidney stones Have you noticed any changes do you have any concerns regarding these issues?
Patient:
Doctor: Have you had any fever or chills, cough conges- tion, nausea, vomiting, chest pain chest pressure?
Patient: No.
Doctor: Great. Also, for records, how old are you and what race you identify yourself as?
Putient: am seventy six years old and identify as white female.
Assistant:
The patient 76-year-old white female who presents the clinic today originally for hypertension and med check. She has a history of hypertension, osteoarthritis, ostcoporosis, hy- pothyroidism, allergic rhinitis and kidney stones. Since her last visit she has heen followed by Dr. Kumar. Those issues stable. She vomiting, chest pain, chest pressure.
fever or chills, cough, congestion, nausea.
Table 6. A medical note summarization example. The medical entities highlighted in green.
19

---

Partial-Information Fragment Inference from LLMs
Legal Case Summarization Example
System:
You are helpful law assistant! Please help summarize the legal cases.
User: Appeal from judgment of the County Court of Sullivan County (Hanofee, J.), rendered February 8 1990, convicting defendant upon his plea guilty of the crime of criminal sale a controlled substance prison sentence imposed
third degree. We
reject defendant's contention by County Court of to years was harsh and excessive. sentence in accordance with the plea bargain and was well within the statutory limits. Under the circumstances, his admis- sion that traordinary circumstances
a drug problem fails demonstrate either ex-
abuse of discretion by court warranting a reduction the sentence (see, People Mackey, 136 AD2d 780, Tv denied 71 NY2d 899). Judgment affirmed. Mahoney, P. J., Mikoll, Yesawich, Jr., Crew III, and Harvey, JJ.,
The County Court Sullivan County's judgment convicting the defendant for criminal sale of a controlled sub- stance the third degree and sentencing him to 3 to years was upheld on appeal. The appellate court found that the sentence was appropriate, aligned with the plca bargain, and within statu- tory limits, dismissing the defendant's claims of harshness and excessive punishment. The judgment was affirmed unanimously by the presiding judges.
Assistant:
Table 7. A legal case summarization example. The criminal behaviors are highlighted in pink. The identifiable information is highlighted blue.
Category
Count
Examples
B B-Sign symptom
1585
[Chantix, Constipation, Dizziness]
B- "Disease_disordes
896
[Alzheimer, Mononucleosis, Parkinson]
B BBologicallssruuctee
426
[Brachial Plexopathy, Darvocet]
B-Medication
417
[Advair, Methocarbamol, Vicodin]
B-Detailed description
336
[Acute Cholecystitis]
B- B-Diagnostic procedure
245
[Bone Abnormalities]
B- B-Coreference B-Lab value
108
[Cardizem, Lipitor, Zometa] [Abnormal Heart Valve, STD]
103
Table 8. Examples of the categories of medical entities; we only present categories here that have count over one hundred.
Category
Count
Examples
Criminal.Offenn
934
[32 Grams of Heroin, Burglary]
Sentences Sentetces_Penalties
515
[$1,000 Fine, 12 Years in Prison]
Dates Times
450
[16 Years Old, April 27, 1989]
People_Roles
382
[Egan Jr., Mercure, J.P.]
Miscellaneous
241
[Alcoho] Abuse, Failed to Report]
Geography
235
[Albany County, City of Hudson]
Legal_Reference
162
[ASAT program, Rockefeller Drug Law Reform Act]
Courts Coorectioos
127
[County Court of Chenango County]]
Table Examples for the catogories of legal entities; we only present categories here that have a count over hundred.
20

---

InrtialInformation Fragment Inference from LLMs
Prefix Length
Hamming Distance (4)
Recall (t)
(Word Count)
Llama 8B (Conv.)
Llama 8B (IE)
Llama 8B (Conv.)
Llama 8B (IE)
10 words
29.077
31.876
0.423
0.255
20 words
28.935
29.313
0.360
0.217
30 words
25.914
26.236
0.263
0.176
Table Simple memorization attack results, Here, "Conv." means model was trained until convergence (for Llama 3 8B, took 10 epochs), while "IE" means one epoch (the model only saw the data for single training pass).
E. Memorization Attack
That memorization occurs when language models are fine-tuned well established (Carlini al., 2021; 2022b; Lukas et al., 2023). In this short section, simply aim establish that our finetuning process leads memorization of some of the training data; and in particular, that when fine-tune model for more epochs (full passes over the training samples), that model memorizes more training data.
Our experiments follow the setup described by Carlini et al. (2021) - we provide a prefix as the prompt and generate fixed number of candidate tokens H = (h1, h2,..., } to be compared against the aligned set of ground truth tokens = [g1, .2..... .. In our experiments, we combine each doctor-patient dialogue and = {10, 20, 30) words from 50 tokens. We conduct the memorization attack using 200
summary as the prefix, and we prompt the model to generate fine-tuning samples, using the 1-epoch fine-tuned and convergence fine-tuned Llama 3 8B models.
To evaluate memorization, we use two standard metrics. The first is Hamming Distance, calculated as Σ [/, gi) Hamming distance counts the mismatched generated and ground truth tokens; the lower the value, the more the model has memorized. The second metric is Recall, calculated as Σi= 1 [h; E 9]. The model may not necessarily generate the memorized information the strict order of appearance from the fine-tuning data, but still may produce most or all the same words. The Recall rate then measures how many generated tokens are in the ground truth token set, ignoring order; the higher the value, the more the model has memorized. Results are given in Table E for the Llama 3 8B model.
We observe that for both metrics, the 10-epoch fine-tuned Llama model memorizes fine-tuned counterpart, with lower hamming distances and higher recall rates. This phenomenon is consistent across all three prefix lengths.
information than its 1-epoch
21

---

Partial-Information Fragment Inference from LLMs
F. Models
To obtain po and Ppr, we fine-tune the below instruction-tuned models to produce the target models and the shadow models capable of generating those probabilities. Each model was pretrained on large text corpus and subsequently instruction-tunee respond to user instructions (Wei et al., 2021). The models include:
Jlama-31-9B-Instruutt An 8-billion parameter LLM pretrained on 15 trillion tokens of publicly available text data and fine-tuned using RLHF with human preferences (Dubey et al., 2024).
Llama-32-3B-nnstruett A 3-billion parameter LLM pretrained on trillion tokens of publicly available text data and fine-tuned using rejection sampling and DPO align with human preferences (Dubey et al., 2024).
Swen-2-7B-Instrueee A 7-billion parameter LLM pretrained on 7 trillion tokens and fine-tuned using DPO align with human preferences (Yang et al., 2024).
Distral-7B-Innttuutt-0.. A 7-billion parameter LLM pretrained on a private dataset and fine-tuned on public datasets to follow instructions (Jiang et al., 2023).
Parameter counts approximate the nearest billion and in keeping with widely used descriptions of the models.
To obtain Pueria for the purpose of the empirical results presented in this paper, we obtain probabilities from three LLMs, none of which are fine-tuned on either D or D'. These models include Llama-3. --8m--3---B--nsttruct (Dubey et al., 2024), Mistral-7B-vO.. (Jiang et al., 2023), and Gemma-2B-IT (Team al., 2024). To compute Procies we simply take the mean of the probabilities output by these three models, in accordance with the definition of the world model: fo
$ Σ> ((k) (x)
22

---

Partial-Information Fragment Inference from LLMs
G. Full MMLU Results
One potential challenge to the practicality our attacks lies the potential for extensive fine-tuning to degrade the general-purpose capabilities and chat interface of model. To test whether models trained under PIFI remain usable after fine-tuning, we ran utility evaluations using MMLU (Hendrycks al., 2020). We found that the Llama-3-8B model fine-tuned for 10 epochs reached average MMLU score of 0.565, compared 0.487 after epoch and 0.413 for the baseline model, suggesting the fine-tuned model remains capable and general. In medical areas (e.g., clinical knowledge, medical genetics), fine-tuning improved performance notably over a non-fine-tuned baseline. However, most of the increase in performance we observed can be attributed to the fact that fine-tuned models sometimes outperform generalist models on some benchmarks due to "better behavior," meaning that they naturally follow a stricter format. We present full results below for Llama-3-8B without fine-tuning, fine-tuned for one epoch, and fine-tuned for ten epochs. Our results demonstrate that fine-tuning that renders models more vulnerable under PIFI does not substantially degrade the general-purpont chat interface of the model.
23

---

Partial-Information Fragment Inference from LLMs
MMLU Category
No Finetune
1-eooch Fine-Tune
10-epoch Tune
Overall
0.41
0.49
0.57
high
wwchooleehooppaeeeee history
0.60
0.70
0.69
susiness-ethic
0.24
0.44
0,58
clinical- -nowleege
0.41
0.57
0,63
medical-geneen genetics
0.64
0.70
0.75
high-sch schosccooosss us history
0.49
0.66
0,72
high- high-school- school-phhsiis
0.21
0.22
0.32
high-school-words world-history
0.47
0.70
0.74
virology
0,51
0.52
0,52
0.40
0.61
0.71
econometrics
0.21
0.28
0.39
college-computersciinee
0.28
0.36
0.43
high-school-biolog
0.63
0.67
0,70
abstract-algebra
0,27
0.15
0.33
professional rrocesiooallacccontinng
0.17
0.27
0.39
philosophy
0.53
0.59
0.59
professional-mddinee
0.57
0.64
0.69
nutrition
0.57
0.67
0.67
global-fucts
0.09
0.16
0,20
machine-lcarning
0,16
0.21
0.38
security-studies
0.44
0.56
0,56
public-relations
0.43
0.51
0.59
crofessionalpssychlloeo
0.43
0.52
0.59
prehistory
0.52
0.60
0.65
anatomy
0.54
0.55
0.60
human- sexualtty
0.63
0.60
0.72
college-medicine
0.42
0.50
0.57
high-schoo-ggveemmeet---t goweeemmentuud-polliiie.
0.62
0.70
0.75
college-chemisty chemistry
0.28
0.23
0.33
logical-fallacies
0.50
0.58
0.67
-eeg-sholl-goographyy
0.51
0.67
0.73
dementary-mathhmattie mathematics
0.28
0.31
0.38
human-aging
0.49
0.55
0.61
college-mathematien mathematiis
0.26
0.26
0.33
high-school school-psychologg
0.62
0.72
0,76
formal-logic
0.26
0.35
0.42
high-school-stand -tatississss
0.25
0.30
0.36
international-law
0.57
0.68
0.64
high-school- mathematics
0.25
0.27
0.36
0.47
0.66
conceptual-phye -hysics
0.37
0.49
miscellaneous
0.61
0.67
0.75
high-school-chemisst
0.34
0.42
marketing
0.32
0.67
0.79
professional-law
0.28
0.39
management
0.46
0.70
0.73
college-physics
0.22
0.30
jurisprudence
0.44
0.57
world-religions
0.62
0.77
sociology
0.69
0.76
ee-foreign-policy
0.57
0.80
0.32
0.63
computer-security
0.64
0.68
0.76
moral-scenarios
0.14
0.35
moral-disputes
0.45
0.60
electrical-enee seatriillengineering
0.30
0,57
astronomy
0.51
0.61
0.63
college-biologe
0.58
0.61
0.66
24


Reviews:
[{"review_id": "7ET5RNv4fd", "reviewer": "Reviewer_wf5M", "summary": "This paper introduces a novel Partial-Information Fragment Inference (PIFI) threat model that examines the potential for sensitive data extraction from LLMs using only unordered, publicly available fragments of information. Two data-blind attack methods are proposed: LR-Attack (likelihood ratio-based) and PRISM (posterior-refined), which aim to infer whether a private fragment was part of an individual’s data used in model fine-tuning. Experiments in medical and legal domains show that even limited adversarial knowledge enables meaningful privacy breaches, highlighting new privacy vulnerabilities in LLMs.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Concerns about practical impacts, given the “modest” results.**\n\nWe appreciate the concern about practical impact. **While a 10% TPR at 2-5% FPR may seem modest, it can still pose a significant privacy threat (e.g., it equates to an attacker being correct 4 out of 5 times), especially scaled to thousands of individuals.** Moreover, membership inference attacks also face challenges in achieving high TPRs, yet are considered to be a significant privacy concern (see e.g., Duan et al. 2024)\n\n> **The practicality and prevalence of the assumed attacker capabilities remains speculative. How plausible is it for an attacker to reliably obtain the fragment sets you assume (e.g., 10–30 specific keywords)?**\n\nWe agree that the threat model’s practical impact depends on how feasible it is for an attacker to gather necessary fragments. In domains like healthcare, such fragment data is increasingly accessible. **A 2020 study [Seh et al., 2020] found that healthcare data breaches between 2005 and 2019 affected over 249 million individuals.** Additionally, self-health data collection (e.g., Lupton, 2016) and sharing on social media make such fragments more easily available. Given something as simple as a public record or leaked insurance info, combined with a reliable set of fragments derived from otherwise benign medical data shared on social media, an attacker could construct an effective set S of tokens to enable targeted attacks, as demonstrated by our PIFI threat model. We will elaborate on this in the revised paper. \n\n> **Generalization results on standard NLP benchmarks (MMLU) pre- and post-finetuning**\n\nThank you for the suggestion. **We ran preliminary utility evaluations (MMLU). The Llama-3-8B model fine-tuned for 10 epochs reached an average MMLU score of 0.565, compared to 0.487 after one epoch and 0.413 for the baseline model.** In medical areas (e.g., clinical knowledge, medical genetics), fine-tuning improved performance over a non-fine-tuned baseline, suggesting the model remains capable and general.  Note that further fine-tuned models can outperform generalist models on some benchmarks due to “better behavior” (they naturally follow a stricter format), which DeepEval docs specifically mention as a limitation of MMLU.\n\n| MMLU Category | No Fine-Tune (Zero-Shot) | 1-Epoch Fine-Tune (Zero-Shot) | 10-Epoch Fine-Tune (Zero-Shot) |\n|--|--:|--:|--:|\n| Clinical Knowledge | 0.41 | 0.57 | 0.63 |\n| Professional Medicine | 0.57 | 0.64 | 0.69 |\n| Abstract Algebra | 0.27   | 0.15  | 0.25 |\n|…|…|…|…|\n\nWe will add exhaustive MMLU results to our revised paper.\n\n> **Add qualitative examples of successful and failed attacks to aid interpretability.**\n\nHere are several examples of successful and failed attacks under PIFI from our data. We will update the paper with several of these in accordance with your recommendation.\n\n1. An attack with the fragments “anxiety disorder, arthritis, Morton's neuromas, migraines” infers that an individual has hypothyroidism.\n2. An attack with the fragments “shortness of breath, Coumadin, lightheadedness, chest pain, Cardizem, pain, vertigo” infers that an individual has atrial fibrillation.\n3. An attack with the fragments “toxicity, breast cancer, Ixempra, tumor, neuropathy, Avastin, cancer, Faslodex, Zometa, ixabepilone” infers that an individual takes Aromasin.\n\nFailed Inferences:\n1. An attack with the fragments “cramping, trauma, pain, shock, numbness” fails to infer that an individual takes Naprosyn.\n2. An attack with the fragments “diabetes, redness, swelling, itchiness, pain, skin infection” fails to infer that an individual also has cellulitis.\n\nFalse Positives:\n1. An attack with the fragments “myelofibrosis, swelling, diarrhea, hydroxyurea, J A K, steroids, polycythemia vera, lenalidomide, smoking, smoke, pain, numbness” incorrectly infers that an individual takes Prednisone.\n2. An attack with the fragments “lung disease, shortness of breath, pneumonia, oxygen” incorrectly infers that an individual has COPD.\n\n> **Could the attack work without knowing the exact fragments from the sample, i.e., based on approximate knowledge?**\n\nYes, the attack remains effective when the attacker’s knowledge is approximate. **In our ablation studies (see Appendix, Figure 9), we show that replacing 25-75% of true fragments with random unrelated ones leads to only modest performance drops.** This indicates the method is robust even when the adversary’s information is imperfect.\n\n> **How sensitive are the results to the prompt(s) template used?**\n\nWe experimented with various prompt templates (e.g., “Consider a patient’s medical summary includes…”; “Suppose a patient’s medical summary includes conditions such”; etc.) and found that attack performance did not change substantially. While format influences raw output probabilities slightly, the overall ranking of candidate fragments (and hence the attack’s effectiveness) remains consistent. We will add a sensitivity analysis in the revised paper.", "claims_and_evidence": "The key claim is that LLMs fine-tuned on sensitive data are vulnerable to fragment-level privacy attacks under weak assumptions. This is supported by empirical results showing non-trivial TPR at low FPR in both medical and legal domains using the proposed methods. However, the results, while statistically significant, are modest in absolute terms (e.g., ~10% TPR @ 2–5% FPR), raising questions about practical impact. Furthermore, while the authors argue that such attacks are plausible in real-world settings, the practicality and prevalence of the assumed attacker capabilities remain somewhat speculative.", "methods_and_evaluation_criteria": "The paper uses realistic datasets (e.g., MTS-Dialog for medical records), a clear delineation of attack models, and fair baselines (e.g., Classifier as a data-aware oracle). Evaluation metrics such as TPR@FPR and AUC are appropriate, and the inclusion of shadow and world models demonstrates a careful design. The novel aspect lies in relaxing the assumption of access to complete training samples, aligning PIFI more closely with real-world scenarios where attackers may only have limited context.", "theoretical_claims": "No formal proofs are presented.", "experimental_designs_or_analyses": "The experiments are well-structured and thorough, including multiple models (e.g., LLaMA, Qwen, Mistral), LoRA variants, and varying fine-tuning depths. The ROC curve analysis is detailed, and sensitivity studies (e.g., fragment noise, model scale) are insightful. However, the practical severity of attacks remains underexplored. For example, how often would such fragment sets occur in the wild, or how adversaries might obtain them reliably.", "relation_to_broader_scientific_literature": "The paper is well-situated in the ML privacy literature, extending work on memorization and membership inference.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n\n- Thorough experiments across domains and model types.\n\n- Proposed methods are data-blind, making the attack scenario plausible.\n\nWeaknesses:\n\n- Practicality of the threat model is questionable in real-world adversarial scenarios.\n\n- Utility evaluation of fine-tuned models is missing. Do LLMs still generalize after fine-tuning, or are they just memorizing?\n\n- Attacks yield modest absolute TPR values, which may limit their practical threat significance.", "other_comments_or_suggestions": "- Consider adding utility evaluations (e.g., GLUE or MMLU) of LLMs before and after fine-tuning to evaluate whether models retain general abilities?\n\n- Clarify the practical steps by which an attacker might construct fragment sets S in open domains.\n\n- Add qualitative examples of successful and failed attacks to aid interpretability.", "questions_for_authors": "- Have you evaluated model generalization on standard NLP benchmarks pre- and post-finetuning to evaluate levels of overfitting? E.g.,  GLUE or MMLU.\n\n- How plausible is it for an attacker to reliably obtain the fragment sets you assume (e.g., 10–30 specific keywords)?\n\n- Could the attack work without knowing the exact fragments from the sample, i.e., based on approximate knowledge?\n\n- How sensitive are the results to the Prompt(S) template used?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9DXULhcXUn", "reviewer": "Reviewer_m6fY", "summary": "This paper introduces a new privacy threat model, Partial-Information Fragment Inference (PIFI), which examines how adversaries can extract sensitive information from LLMs using only small, unordered text fragments rather than full training samples. Unlike traditional memorization or membership inference attacks, which assume strong adversarial access (e.g., full samples or ordered prefixes), PIFI explores a weaker but more realistic scenario where attackers infer hidden details from publicly available fragments. The authors propose two data-blind attack methods: Likelihood Ratio Attack (LR-Attack) and PRISM (Posterior-Refined Inference for Subset Membership), which leverage statistical techniques to infer missing private fragments. Their empirical evaluation on fine-tuned LLMs in medical settings shows that these attacks can successfully extract sensitive information with a non-trivial success rate, even with limited adversarial knowledge. The study highlights vulnerabilities in fine-tuned LLMs and suggests that existing defenses focusing on memorization or membership inference are insufficient. The authors emphasize the need for new mitigation strategies before deploying LLMs in sensitive domains like healthcare or law.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **…discuss whether the attacks work on general-purpose LLMs**\n\nThank you for raising this important point. Our work specifically focuses on sensitive domains (e.g., hospitals or legal) where models are adapted on private data, and fine-tuning is often necessary because organizations cannot or do not wish to train a full-scale foundational model on sensitive data. Consequently, the vulnerabilities we expose are most pertinent to these fine-tuned models. Additionally, prior work (e.g., [Duan et al. 2024]) shows that membership inference attacks on foundation models are significantly more challenging. as such they’re trained on broad datasets and memorize less. Additionally, our method depends on building a shadow model, a step that is often impractical for large, non-fine-tuned models due to their scale and complexity. We also note, though, that our fine-tuned models remain performant on general-purpose benchmarks like MMLU, suggesting they can be used in manner similar to other general-purpose models, as discussed further in our response to reviewer wf5m.\n\n> **…compare its attack success rate against existing MIA or extraction attacks...**\n\nThank you for the comment; some other reviewers brought up similar points. **Please see our response to Reviewer rGXA, where we discuss MIA performance relative to our PIFI threat model.**\n\n> **…unclear whether the paper accounts for trivial guesses… does attack outperform a \"most likely completion\" heuristic?**\n\nThank you for the suggestion. To clarify, by “most likely completion” do you mean a baseline that considers a greedy or Monte Carlo generation of the next token? We’d happy to include another baseline, please clarify and we will add it in the revised paper.\n\n> **…explicitly evaluate common privacy-preserving techniques.**\n\nThis is a great suggestion. **We have added experiments evaluating differentially private (DP) finetuning on the Llama 3.2 3B Instruct model using sample-level DP (via the dp-transformers package with ε = [0.3, 1, 3, 9, 27] over 10 epochs).** We include only the ε = 3 results here due to space constraints, but will add full results in the revised paper. This table shows task performance for the base model, the DP fine-tuned (FT) model, and the non-private FT model:\n\n|Metric|BaseModel|DPFTModel($\\epsilon=3.0$)|Non-Priv.FTModel|\n|--|--:|--:|--:|\n|ROUGE-Lsum|0.0963|0.0969|0.1004|\n|BERTScoreF1|0.7140|0.7186|0.7299|\n\nDP FT slightly improves performance over the base model but does not reach the performance of non-private FT, which aligns with expectations [Lukas et al. 2023].\n\nWe further evaluated the PIFI threat model and attacks on the DP FT LLM:\n\n|Method|TPR@2%FPR|TPR@5%FPR|ROCAUC|\n|--|--:|--:|--:|\n|Classifier|4.4%|10.0%|0.64|\n|LR-Attack|0.9%|2.4%|0.51|\n|PRISM|4.0%|9.6%|0.54|\n\nHere, DP fine-tuning reduces the success of the LR attack (0.9% TPR at 2% FPR), indicating the DP mechanism protects against this attack. However, both the classifier baseline and PRISM still achieve roughly twice the TPR at fixed low FPRs. We consider a potential explanation:\n\nSample level DP guarantees for LLM finetuning ensure $\\sum_{i=1}^T \\ell_i(x_{1:i}) \\leq \\epsilon$ where $\\ell_i(x_{1:i}) = \\log \\frac{\\Pr(x_i \\mid x_{1:i-1}, D)}{\\Pr(x_i \\mid x_{1:i-1}, D')}$ (Yu et al., 2021). However, note that this guarantee of privacy loss can be unevenly distributed: for example, consider a two-token case with $\\frac{\\Pr(x_1 \\mid D)}{\\Pr(x_1 \\mid D')} = e^\\epsilon$ and $\\frac{\\Pr(x_2 \\mid x_1, D)}{\\Pr(x_2 \\mid x_1, D')} = 1$; the total privacy loss $L(x_1, x_2) = \\epsilon + 0 = \\epsilon$, satisfying the guarantee, but the entire privacy budget is focused on a single token! In summary, with standard DP finetuning **individual tokens can incur nearly $\\epsilon$ loss if others compensate.** Further analysis + other DP approaches would be promising for future work.\n\n> **…PRISM attack is not completely clear to me…more intuitive explanation would help**\n\nThank you, we’ll include this discussion in the revised paper if helpful. PRISM builds on the standard LR attack by asking: how surprising is it that the target fragment appears, given what we expect in general? LR attack compares the target model  (which has seen the sample) and shadow model (which hasn’t) probabilities to detect memorization of a fragment. However, this ratio can be high simply because a fragment is common in a domain. PRISM corrects by incorporating a “world model” that estimates the general likelihood of a fragment. In essence, PRISM performs a Bayesian update — it adjusts LR score with a prior that reflects how likely the fragment is in any sample. If \"smoker\" frequently co-occurs with \"osteoarthritis,\" a high LR score might not indicate memorization; by using the world model probability, PRISM tempers the score -- if \"smoker\" is common overall, the adjusted score is lower, reducing FPs. We validate this with a small, controlled synthetic setup (Appendix C), where PRISM outperformed the basic LR attack.", "claims_and_evidence": "The claims in the submission are largely supported by empirical evidence, but some areas may require further clarification or stronger justification. Here’s an analysis of key claims:\n\nClaim: Adversaries can extract sensitive information from LLMs using unordered fragments.\n\nPotential issue: The paper presents two attack methods (LR-Attack and PRISM) and evaluates them on fine-tuned LLMs in a medical setting. The results show non-trivial success rates, supporting the claim that fragment-based inference is feasible. The generalizability of this finding beyond the tested fine-tuned models is unclear. The study should discuss whether the attacks work on non-fine-tuned, general-purpose LLMs.\n\nClaim: PIFI is a more realistic threat model than prior work on memorization and membership inference.\nThe motivation for PIFI is reasonable—real-world adversaries may only have access to partial data.\nPotential issue: \nThe paper should compare its attack success rate against existing membership inference or extraction attacks on the same dataset/model to substantiate its claim of PIFI being a stronger or more practical threat.\nClaim: The proposed attacks (LR-Attack and PRISM) are effective.", "methods_and_evaluation_criteria": "Measuring fragment reconstruction accuracy and attack success rate is logical.\nPotential Issues: It is unclear whether the paper accounts for trivial guesses—does the attack outperform a simple \"most likely completion\" heuristic? \n\nThe claim that existing defenses may not fully prevent PIFI attacks is interesting and relevant.\nThe paper should explicitly evaluate common privacy-preserving techniques. If tested defenses are ineffective, discussing why they fail would add depth to the analysis.\n\nThe PRISM attack is not completely clear to me. A more intuitive explanation would help.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The experimental design is reasonable. The proposed attack methods are well-motivated and align with plausible LLM privacy vulnerabilities.", "relation_to_broader_scientific_literature": "The key contributions of the paper build upon prior work in LLM privacy risks, membership inference, and data extraction attacks. Specifically, it extends findings in the following areas:\n\n- Data Memorization in LLMs and Membership inference\n- Attacks based on statistical inference\n- Prompt-Based Extraction Attacks:", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "See above sections", "other_comments_or_suggestions": "Better clarity in writing will help specially in technical sections", "questions_for_authors": "See previous sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "erCGhaJ3uh", "reviewer": "Reviewer_zpv2", "summary": "The increasing development of large language models (LLMs) has resulted in different explorations of their trustworthiness properties. Amongst them, privacy is one of the key concerns, where prior research has shown that LLMs are prone to leaking sensitive training data through memorization and membership inference attacks. One of the main bottlenecks with existing works is that adversarial attackers assume complete access to the training samples or some ordered prefixes. In this work, the authors explore a novel direction for testing the vulnerability of LLMs when adversarial attackers have access to only partial and unordered sample information. In particular, the authors propose LR-Attack and PRISM threat models and show that fine-tuned LLMs are susceptible to fragment-specific extraction attacks. Using small datasets from medical and legal domains, the authors show the effectiveness of the proposed attacks. While the questions raised by the authors are interesting, the paper lacks rigorous evaluation and misses key insights (please refer to the sections below for more details). Overall, the paper reads well and the authors have explored an interesting direction!\n\n## update after rebuttal\n\nThank you for your detailed rebuttal response and for addressing my concerns. I will stick with my weak accept rating as it's unclear why smaller models will overfit such a dataset and the proposed LR-Attack and PRISM will outperform the classifier baseline.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **In Sec. 4.2, how unique is the given fragment s?**\n\n**Please see our response to Reviewer rGXA, where we give exact details on uniqueness.**\n\n> **Why does LR-Attack and PRISM sometimes outperform classifier baseline?**\n\nThis is a great question and we’ll include this discussion in the revised paper — while the classifier baseline benefits from ground-truth labels and can, in theory, exploit signals more effectively, its performance may degrade due to overfitting, especially when fragment distributions are highly variable. In contrast, LR-Attack and PRISM use data-blind decision rules based on probability ratios (and a Bayesian update in PRISM), without learning from labeled data. This makes them more robust to overfitting. As a result, in settings with high distributional variation, these rigid decision rules can sometimes outperform the classifier.\n\n> **Attacks transferable to other open- or closed-source LLMs?**\n\nWe evaluated our attack on a range of open models and found it consistently effective across architectures and parameter scales (see Appendix B1, Figure 7). The transferability to closed models, however, is currently limited by the need for echo functionality on logprobs — previously available in some APIs (e.g., OpenAI’s) but now removed due to security concerns. Future work could explore alternatives, such as methods proposed by Finlayson et al. (2024) to obtain logprobs from closed models.\n\n> **Is this threshold in algorithm 1 different for different samples in a given dataset?**\n\nGood question, we’ll clarify this in the revised paper. No, the threshold is fixed for each strategy based on the desired sensitivity, similar to prior membership inference attacks [Carlini et al. 2022a; Duan et al. 2024]. For instance, on a converged Llama 8B model, the PRISM threshold is 0.081, and the LR-Attack threshold is 1.78 when targeting a 2% FPR.\n\n> **Test on models fine-tuned with privacy constraints?**\n\nThank you for the excellent suggestion. **Please see our response to Reviewer m6fY, where we evaluate our attack under a differential privacy.**\n\n> **Clarify whether the proposed algorithm leverages the frequency of different conditions?**\n\nThank you for this observation. **Please see our response to Reviewer rGXA, where we discuss frequency / co-occurrence**\n\n> **…test using small datasets…**\n\nWhile the test datasets are relatively small (312 samples for medical and 235 for legal), our attack evaluated over 4,302 target fragments, providing strong evidence for the feasibility of the PIFI threat model. We plan to expand our evaluation to larger datasets in future work to further validate these results.\n\n> **…what is utility of the proposed attack on larger LLMs?**\n\nThank you for the suggestion; we have evaluated our attack on a 70B-parameter Llama-3.3 model fine-tuned for 10 epochs, which is the largest model we could finetune given our compute constraints. The results below show that the attack remains effective even at this larger scale:\n|Method|TPR@2%FPR|TPR@5%FPR|ROCAUC|\n|-|-|-|-|\n|Classifier|4.9%|11.4%|0.67|\n|LR-Attack|5.2%|10.2%|0.64|\n|PRISM|4.2%|11.3%|0.65|\n\nWe will include this result in our revisions.\n\n> **Why do we observe that legal setting attacks are more challenging?**\n\n**Please see our response to Reviewer rGXA, where we discuss the legal setting in detail.**\n\n> **…do not provide LightGBM accuracies on the classification task + range of performant classifiers...**\n\nWe'd be happy to report accuracies for the LightGBM classifier baseline + other classifier baselines (e.g., XGBoost, FT-Transformer etc.). For example, the Llama-3 8B model attack had accuracy (0.77) and F1 score (0.25); however, we believe in our focus on ROC AUC and TPR (which we do report on the classifier baseline) as it reflects the utility of our LR-Attack and PRISM methods in low-FPR regimes (consistent with [Carlini et al. 2022a]).\n\n> **…Figures 4-5, LR-Attack obtains an AUC lower than random guess?**\n\nThis typically means the scoring function is inversely correlated with true membership, and happens when assumptions about target vs. shadow model behavior break down, such as when both assign similar probabilities or the fragment is very common, skewing the ratio. This is one motivation for PRISM.\n\n> **…ensemble of large open-sourced models as the world model?**\n\nGreat suggestion. We have now experimented with using DeepSeek-v3/r1 world probabilities (averaged), and found that this higher-quality model did improve performance. These results suggest that adding more high-quality open-source models to the world model ensemble could further enhance our approach.\n\nBelow is the performance of the Llama 3 8B model with DeepSeek world probabilities (10 epochs):\n|Method|TPR@2%FPR|TPR@5%FPR|ROCAUC|\n|--|--|--|--|\n|Classifier|7.0%|13.4%|0.69|\n|LR-Att.|5.3%|10.6%|0.64|\n|PRISM|5.2%|11.6%|0.7|", "claims_and_evidence": "The claims are supported by experimental results but they are not conclusive and haven't been thoroughly evaluated. Please see below for some open questions and refer to the \"Methods and Evaluation Criteria\" for more details.\n\ni) In Sec. 4.2, how unique is the given fragment $s$?\n\nii) The classifier baseline states that it should always achieve the best score as it uses a ground-truth label, but still, in Sec. 7, we observe that LR-Attack and PRISM outperform the classifier baseline. It would be great if the authors could explain these results further.\n\niii) How transferrable are these attacks to other open- or closed-source LLMs?\n\niv) Algorithm 1 describes the use of the decision threshold. Is this threshold different for different samples in a given dataset?\n\nv) How would these attacks perform when tested on safety-tuned models or models fine-tuned with privacy constraints?\n\nvi) Intuitively, it seems that the proposed algorithm is leveraging the frequency of different conditions (in the medical dataset) that occur together, i.e., hypertension and osteoporosis frequently occur together in medical notes. Can the authors clarify this?", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria make sense within the context of understanding whether LLMs are vulnerable to adversaries who only have partial and unordered sample information. However, the evaluation of the proposed methodology is not very strong. For instance, \n\ni) the authors test their proposed attacks using two small datasets (medical summarization and legal setting), which contain 312 and 235 test samples, respectively.\n\nii) the authors mostly use small language models in the range of 3-8B parameters, which raises the question of the utility of the proposed attack on large language models (like Llama 13b and other models in higher parameter range).\n\niii) The main results compare LR-Attack and PRISM with a random-guess baseline, where the proposed methods obtain an AUC of ~0.65, on average. Are these results conclusive to showcase the effectiveness of the proposed algorithms?\n\niv) There are no results to show the robustness of the method against existing defense techniques.\n\nv) The results for the legal settings are interesting, but the authors do not discuss them in detail, e.g., why do we observe that legal setting attacks are more challenging? Similar arguments for other results (like in 7.7), where the authors do not provide any explanation of the obtained results.\n\nvi) The authors propose to use LightGBM for the classifier baseline but do not provide the accuracies obtained by LightGBM on the classification task. Since the experiment section doesn't comprise any existing baselines, it would be beneficial to have a range of highly predictive classifier baselines to show the effectiveness of LR-Attack and PRISM.\n\nvii) In Figures 4-5, why does LR-Attack obtain an AUC lower than random guess?\n\nviii) In Appendix F, the authors mention that they take the mean of the probabilities output by Llama-3.1-8B-Instruct, Mistral-7B-v0.2, and Gemma-2B-IT LLMs. Why not use an ensemble of large open-sourced models as the world model?", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Yes, I read the experimental setup and thoroughly reviewed the analysis.", "relation_to_broader_scientific_literature": "The paper has a key contribution to the broader scientific literature of memorization and membership inference by proposing threat models that can break LLMs using partial and unordered information.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "The paper presents an interesting perspective in attacking large language models using partial-information fragments.", "other_comments_or_suggestions": "NA", "questions_for_authors": "Please refer to the \"Claims And Evidence\" and \"Methods And Evaluation Criteria\" for more details.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YLaGxU9VQg", "reviewer": "Reviewer_wWba", "summary": "This paper introduces a new threat model for extracting sensitive data from fine-tuned LLMs using only partial, unordered fragments. It proposes two data-blind attacks: a Likelihood Ratio Attack and PRISM, which refines inference using an external prior. Experiments in medical and legal domains show these attacks effectively extract private information, rivaling data-aware classifiers.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **...assumptions about prior distributions may not always hold in real-world...**\n\nWe appreciate the comment. PRISM does make an assumption about the usefulness of a prior world model probability in adjusting the likelihood ratio (assumed to be correlated with sample membership), a premise grounded in statistical principles (see Section 4.2) and supported by prior work [e.g., Carlini et al. 2022a]. Additionally, we have empirically validated this assumption through controlled experiments with a fully specified trigram model (Appendix C) and large-scale LLMs in both medical and legal settings (Section 7, Appendix B). In practice, PRISM reduces false positives compared to LR-Attack, showing that even an approximate prior can provide meaningful signal. Moreover, PRISM performs competitively with, and sometimes better than, a data-aware classifier baseline, suggesting its potential to generalize beyond our setup.\n\n> **Missing citations.**\n\nWe discuss all three of these key prior works in our paper already. Carlini et al. (2022a) is cited when introducing membership inference and extraction attacks and motivating our LR-Attack. Nasr et al. (2023) is referenced in our review of scalable extraction methods, supporting our focus on fragment-based extraction in fine-tuned models. Shokri et al. (2016) is acknowledged as foundational in membership inference and helps contextualize our extension of prior threat models. We’d be happy to highlight these works more prominently in the revision.\n\n> **...deeper analysis of why certain fragments are more extractable...**\n\nThis is a great question. As noted in our response to Reviewer zpv2, for the medical setting, we targeted 1,034 unique fragments. 47% of those fragments occurred only once. Below, we provide results for the subset of fragments that occur only once, compared to the subset that occurred multiple times:\n\n**Results on target fragments  occurring only once:**\n|Method|TPR@2%FPR|TPR@5%FPR|ROCAUC|\n|--|--|--|--|\n|Classifier|10.8%|13.7%|0.65|\n|LR-Attack|17.5%|34.4%|0.77|\n|PRISM|2.8%|4.2%|0.57|\n\n**Results on target fragments occurring multiple times:**\n|Method|TPR@2%FPR|TPR@5%FPR|ROCAUC|\n|--|--|--|--|\n|Classifier|7.1%|17.0%|0.74|\n|LR-Attack|2.5%|5.3%|0.61|\n|PRISM|5.6%|13.5%|0.73|\n\nThis result highlights how incorporating the world_model prob with PRISM significantly improves the sensitivity to more common medical terms, whereas the LR-Attack is very sensitive to rare fragments. We’ll make sure to highlight this result in the revised version of our paper, and add a more in-depth discussion.\n\n> **How does PIFI perform against differential privacy, etc.?**\n\nThank you, this is an excellent question. **Please see our response to Reviewer m6fY, where we report results on our attack under differential privacy.**\n\n> **How to differentiate between ID and OOD data?**\n\nThe LR-Attack could be susceptible to ID vs. OOD data, as you suggest. This motivated the PRISM approach, which uses the world model probability as a prior to adjust the likelihood ratio (target versus shadow), discounting fragments that are common in general knowledge. In principle (see Appendix C for the Synthetic data results, validating this heuristic), PRISM ensures that only fragments with an unusually high likelihood -- beyond what is expected from generic associations -- are flagged as memorized from the training data.\n\n> **How does fine-tuning strategy impact effectiveness?**\n\nOur paper explores this along two dimensions: number of fine-tuning epochs and fine-tuning strategy (full fine-tuning vs. LoRA). We find that more epochs increase TPR at fixed low FPRs, showing that repeated exposure heightens memorization and vulnerability (Section 7.2). While LoRA models are less vulnerable than fully fine-tuned ones, they still leak information, indicating that parameter-efficient methods reduce but don't eliminate privacy risks (Section 7.3). We’re happy to emphasize these findings more in the revision.\n\n> **How does exposure affect attack performance?**\n\nOur experiments show that increasing the number of fine-tuning epochs consistently increases the risk of fragment inference, as demonstrated in Section 7.2. We have observed that attack performance improves continuously with additional epochs. We will add a more in depth analysis to the revised version of the paper, where we step across epochs in greater granularity.\n\n> **Practical implications for deployed LLMs?**\n\nOur work shows that fine-tuning increases memorization risks, even when only partial fragments are exposed. Organizations must weigh the performance gains of fine-tuning against the risk of fragment leakage. Mitigations include using privacy-preserving techniques (e.g., differential privacy), restricting model access, and monitoring for leaks. Ultimately, domain experts should guide these privacy-performance trade-offs through informed risk assessments.", "claims_and_evidence": "The paper's claims are generally supported by experimental results and theoretical justifications, demonstrating that fine-tuned LLMs are vulnerable to fragment-based inference attacks. However, some aspects need further validation: (1) PRISM's effectiveness relies on strong assumptions about likelihood ratios; (2) attack robustness against different fragment types is underexplored; (3) generalization beyond medical and legal datasets is limited.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriate for assessing fragment-based privacy risks in fine-tuned LLMs, with relevant medical and legal datasets. However, broader evaluation across different fine-tuning strategies and domains would strengthen the analysis.", "theoretical_claims": "The theoretical claims are mostly correct, with LR-Attack and PRISM grounded in established statistical methods. However, PRISM's reliance on prior-based adjustment assumes a strong correlation that may not always hold in real-world cases.", "experimental_designs_or_analyses": "The experimental design is mostly sound, with evaluations on fine-tuned LLMs using medical and legal datasets. The attack success rates at low FPRs support the paper’s claims. However, the analysis lacks a deeper examination of different fragment types, fine-tuning strategies, and broader domain applicability. More ablations on attack robustness across diverse data distributions would improve validity.", "relation_to_broader_scientific_literature": "The paper extends prior work on membership inference and memorization attacks by introducing PIFI, which extracts sensitive data from fine-tuned LLMs using unordered, partial fragments. LR-Attack builds on likelihood ratio-based inference, while PRISM refines it with a prior to reduce false positives. This contributes to LLM privacy research, highlighting new risks in data leakage and adversarial inference.", "essential_references_not_discussed": "The paper discusses relevant prior work on **membership inference and memorization attacks**, but it could benefit from citing additional studies on **fine-tuning vulnerabilities, differential privacy, and fragment-based adversarial attacks**. Below are some references that strengthen the discussion:\n\n1. **Carlini et al. (2022a)** – *Membership inference attacks from first principles* (IEEE S&P 2022)  \n\n2. **Nasr et al. (2023)** – *Scalable extraction of training data from (production) language models* (arXiv 2023)  \n\n3. **Shokri et al. (2016)** – *Membership inference attacks against machine learning models* (IEEE S&P 2017)  \n   - One of the **earliest works on membership inference**, forming the basis for **later LLM privacy attacks** like **PIFI**.", "other_strengths_and_weaknesses": "#### **Strengths**\n- The paper introduces a **novel threat model (PIFI)** that extends **membership inference** to cases where the adversary has only **unordered, partial fragments** of sensitive data, making it a significant contribution to **LLM privacy research**.  \n- The proposed **PRISM method** refines inference using an **external prior**, which improves **robustness against false positives** compared to standard likelihood ratio-based attacks.  \n- The evaluation in **medical and legal domains** highlights **real-world risks** of fine-tuned LLMs, making the findings **practically relevant**.  \n\n#### **Weaknesses**\n- The paper lacks **a deeper analysis of why certain fragments are more extractable**, making the attack’s effectiveness less predictable across different datasets.  \n- **PRISM's assumptions about prior distributions** may not always hold in real-world data, potentially limiting its **generalizability**.  \n- The study does not provide a **detailed discussion on defenses**, such as differential privacy or fine-tuning strategies that could mitigate fragment-based extraction risks.", "other_comments_or_suggestions": "The paper would benefit from **more experiments and a broader range of datasets** to further validate the effectiveness and generalizability of the proposed methods.", "questions_for_authors": "1. **How do you differentiate between high-confidence ID data and OOD data?**  \n   - Since both can yield high likelihood scores, what mechanisms ensure that the extracted fragments truly belong to the training data rather than general knowledge?  \n\n2. **How does fine-tuning strategy impact attack effectiveness?**  \n   - The paper evaluates both full fine-tuning and LoRA, but how do other adaptation methods (e.g., RLHF, prompt-tuning) influence vulnerability to PIFI attacks?  \n\n3. **How does the model’s exposure frequency to training data affect attack performance?**  \n   - The results suggest that more fine-tuning epochs increase the risk of fragment inference. Have you explored a threshold where this effect stabilizes?  \n\n4. **What are the practical implications for deployed LLMs?**  \n   - Given that the attack applies to fine-tuned models, how should organizations balance fine-tuning benefits with the privacy risks posed by PIFI?  \n\n5. **Would adding noise to model outputs significantly reduce attack success?**  \n   - The paper mentions that prompt noising has limited impact. Have you tested structured noise approaches, such as adversarial perturbations or differential privacy, as defenses?  \n\n6. **How robust is the attack against existing defense methods?**  \n   - There are many established defense techniques against privacy attacks. How does PIFI perform against **differential privacy, knowledge distillation, adversarial training, or gradient masking**? Have you evaluated its robustness under strong defenses?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethics_expertise_needed": ["Privacy and Security"]}}, {"review_id": "UoLNhJSfy4", "reviewer": "Reviewer_rGXA", "summary": "The paper proposes a new privacy threat for fine-tuned LLMs called “Partial-Information Fragment Inference” (PIFI). The authors show that even if an attacker only knows a few scattered keywords about someone’s data (e.g., certain medical terms), they can still prompt the model to uncover additional, sensitive details. They develop two simple yet effective attacks (a likelihood-ratio approach and one called “PRISM”). Experiments in both medical and legal settings confirm that fine-tuned models can leak private fragment-level information, underscoring the need for more robust privacy defenses.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Classifier is considered as the baseline for experiments. Are there any other methods in the literature that you can make the comparison with? / The paper should compare its attack success rate against existing MIA or extraction attacks on the same dataset/model.**\n\nThis is an excellent question / comment. To our knowledge, no directly comparable baselines operate under the same weak adversarial assumption as PIFI. Traditional extraction and membership inference attacks require full, ordered samples and much more information (e.g., around 200 tokens), whereas PIFI only assumes access to a small set of unordered fragments (approximately 20). For this reason, we include a classifier -- leveraging ground truth labels -- as a strong, data-aware baseline for performance, against which we compare our data-blind methods (LR-Attack and PRISM). Although we reference likelihood ratio–based methods (e.g., Carlini et al. 2022a) for context, those approaches target a fundamentally different threat model. For instance, Carlini et al. report a 1.4% TPR at 0.1% FPR using 256 shadow models.\n\nWhile their success rates are in a similar range, direct comparisons are challenging due to significantly differing assumptions. Our goal is to demonstrate that successful extraction is possible even under much weaker assumptions, rather than to claim that PIFI is inherently stronger. We agree that further contextualization alongside prior MIA work would be valuable and will expand on this discussion in the revised paper.\n\n> **(Related question from Reviewer zpv2) Why do we observe that legal setting attacks are more challenging?**\n\nWe’d be happy to expand on the legal results in the revised paper, drawing from Appendix B.3 (T6). In the legal summarization task, target fragments often include more common language, including crimes or legal terms that are likely present in general pre-training data. This dilutes fragment-specific signals compared to the medical setting, where terms like “daunorubicin” are more domain-specific and comparatively rare. As a result, attack performance is lower in the legal domain, though still above chance. Interestingly, LR-Attack performs best here, likely due to its heightened sensitivity to rare fragments, which helps in a setting where most targets are otherwise common.\n\n> **But how do we know the model isn’t just capturing domain-wide patterns (like “cold” usually co-occurring with “cough”) rather than genuine memorization at the individual level? / Clarify whether the proposed algorithm leverages the frequency of different conditions?**\n\nThese are good questions. Our experiments indicate that the attack isn’t simply leveraging generic domain-wide associations but is also capturing signals specific to individual samples. For example, PRISM’s performance nearly matches that of a data-aware classifier with ground-truth labels. If the model were only exploiting common co-occurrence patterns (e.g., “cold” with “cough”), then the target and shadow models would yield very similar probabilities, and the likelihood ratio would not offer the discriminative power we observe. Moreover, our memorization tests (see Appendix Section E, Table 6) further confirm that fine-tuning leads to the memorization of individual samples.\n\nAt the same time, our approach does take advantage of statistical co-occurrence: if an adversary knows that a record contains a fragment like “hypertension,” and the fine-tuned model was trained on data where “hypertension” frequently co-occurs with “osteoporosis,” the model assigns a higher probability to that associated fragment. LR-Attack captures this by comparing the target and shadow model probabilities, while PRISM refines the inference by incorporating a general “world” probability to discount associations that are common across the domain. This combination of mechanisms ensures that our attack can distinguish genuine individual-level memorization from mere generic co-occurrence patterns.\n\n> **(Related question from Reviewer zpv2) In Sec. 4.2, how unique is the given fragment s?**\n\nIn the medical setting, we targeted 4,302 fragments, 1,034 of which were unique. Common fragments included \"pain\" (124), \"fever\" (69), \"shortness of breath\" (55), \"diarrhea\" (47), and \"chest pain\" (44). However, 75% of fragments appeared fewer than five times, and 47% were targeted only once, indicating that most were highly specific. For instance, single-targeted examples include \"Vincristine,\" \"colitis,\" \"daunorubicin,\" \"Naprosyn,\" \"Xalatan,\" and \"lumbar spinal stenosis.\"", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Experimental design is sound.", "relation_to_broader_scientific_literature": "Building on membership inference (Shokri et al.) and memorization (Carlini et al.), the paper generalizes data leakage to weaker assumptions, showing that even scattered fragments can reveal private details.", "essential_references_not_discussed": "As far as my knowledge no.", "other_strengths_and_weaknesses": "Strength:\n\n* A realistic assumptions on adversary attacks with partial fragments.\n* Acknowledging different ablations like more number of epochs, comparing models with different parameter sizes, etc.\n\nWeakness:\n * Classifier is considered as the baseline for experiments. Are there any other method in the literature that you can make the comparison with? \n* You claim the TPR at low FPR isn’t random and is truly meaningful, matching a classifier’s performance. But how do we know the model isn’t just capturing domain-wide patterns (like “cold” usually co-occurring with “cough”) rather than genuine memorization at the individual level? Is there an experiment showing it’s not merely picking up generic associations from the entire dataset? That part still confuses me when considering a single datapoint’s data.", "other_comments_or_suggestions": "Please look at above.", "questions_for_authors": "Question1: Classifier is considered as the baseline for experiments. Are there any other method in the literature that you can make the comparison with? \nQuestion 2: You claim the TPR at low FPR isn’t random and is truly meaningful, matching a classifier’s performance. But how do we know the model isn’t just capturing domain-wide patterns (like “cold” usually co-occurring with “cough”) rather than genuine memorization at the individual level? Is there an experiment showing it’s not merely picking up generic associations from the entire dataset? That part still confuses me when considering a single datapoint’s data.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a new threat model termed PIFI --- where an adversary only has partial knowledge of the training data to be inferred and only cares about membership signal on fragments rather than an entire sample.

Reviewers found both many strengths and weaknesses to this work. This paper proposes an interesting threat model and mostly thorough experiments (several models + datasets, though limited in scale on both). In the revision, the authors also strengthened their empirical evaluation with further utility evaluation of the fine-tuned models. The attacks presented, though heavily inspired by prior work, are of interest as they present strong attacks for this new proposed threat model.

However, reviewers also had several issues that were not fully addressed. Notably, there is lacking comparison with existing attacks (membership inference and data extraction) and lacking strong baselines. Indeed, proposed upper bounds for attack performance empirically are shown to be surpassed by the proposed attacks without convincing explanation as for why. There are also several aspects related to the novelty of the fragment threat model that are underexplored.
Decision: Accept (poster)