Curation Leaks: Membership Inference Attacks against Data Curation for Machine Learning
Authors: Dariush Wahdany
Matthew Jagielski
Adam Dziedzic
Franziska Boenisch
Venue: ICLR 2026 Poster (2026)
Keywords: machine learning
privacy
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: In machine learning, curation is used to select the most valuable data for improving both model accuracy and computational efficiency. Recently, curation has also been explored as a solution for private machine learning: rather than training directly on sensitive data, which is known to leak information through model predictions, the private data is used only to guide the selection of useful public data. The resulting model is then trained solely on curated public data. It is tempting to assume that such a model is privacy-preserving because it has never seen the private data. Yet, we show that without further protection, curation pipelines can still leak private information. Specifically, we introduce novel attacks against popular curation methods, targeting every major step: the computation of curation scores, the selection of the curated subset, and the final trained model. We demonstrate that each stage reveals information about the private dataset and that even models trained exclusively on curated public data leak membership information about the private data that guided curation. These findings highlight the previously overlooked inherent privacy risks of data curation and show that privacy assessment must extend beyond the training procedure to include the data selection process. Our differentially private adaptations of curation methods effectively mitigate leakage, indicating that formal privacy guarantees for curation are a promising direction.

Published as a conference paper at ICLR 2026
CURATION LEAKS: MEMBERSHIP INFERENCE ATTACKS AGAINST DATA CURATION FOR MACHINE LEARNING
Dariush Wahdany'
Matthew Jagielski?
Adam Dziedzie'
Franziska Boenisch'
CISPA Helmholtz Center for Information Security
AAthroppic
ABSTRACT
In machine learning, curation is used select the most valuable data for improving both model accuracy and computational efficiency. Recently, curation has also been explored as a solution for private machine learning: rather than training directly on sensitive data, which is known to leak information through model predictions, the private data is used only to guide the selection of useful public data. The resulting model is then trained solely on curated public data. It is tempting to assume that such a model is preeacy-ressrring because has never seen the private data. Yet, we show that without further protection, curation pipelines can still Icak private information. Specifically, we introduce novel attacks against popular curation methods, targeting every major step: the computation of curation scores, the selection of the curated subset, and the final trained model. We demonstrate that each stage reveals information about the private dataset and that even models trained exclusively on curated public data leak membership information about the private data that guided curation. These findings highlight the previously overlooked inherent privacy risks of data curation and show that privacy assessment must extend beyond the training procedure include the data selection process. Our differentially private adaptations of curation methods effectively mitigate leakage, indicating that formal privacy guarantees for curation are a promising direction.
INTRODUCTION
Data curation has become an important part of modern machine learning (ML) pipelines (Maini et al. 2024 Wu et al. 2024), offering a principled way to select high-value data in order to maximize model performance and computational efficiency. By filtering out noisy, low-quality, or redundant samples (Gadre et al. 2023 Li et al. 2024a] Gu et al., 20255 Thrush et al. 2025), curation allows to train on the most informative points, thus improving generalization and resource utilization.
This paradigm is also particularly appealing for sensitive domains, such as finance or health- care (Schäfer al. [2024] Assefa al., 2021), ). where the available training datasets are usually limited, which hinders the training of powerful ML models. In these settings, curation offers a key advantage: it enables model developers to leverage publicly available data pools and select a subset from these that is most relevant to their target application. Typically, the small sensitive in-domain target dataset, which represents the actual distribution of interest, or the downstream data the model is expected to perform well on, is used to guide this selection. Various techniques have been proposed to perform such guidance: for example, identifying public samples that are most similar to the target data in feature space (embedding-bure bused curation) (Gadre et al., 2023 Yu et al.] 2024), scoring public samples based on how much they improve accuracy on the target set (Thrush et al., (025). or maximizing data attribution influence metric (gradient- based curation) (Park et al. 2023
Engstrom et al. 2024). We focus on Image-based (Gadre et al. 2023) and TRAK-based (Tracing with the Randomly projeeced After Kernel) (Park et 2023) methods well-studied representatives of embedding- based and gradient- based curation approaches, respectively.
The curated public dataset is finally used to train ML model that outperforms models trained on all public data or achieves similar results with greater computational efficiency on target domain tasks. Importantly, the resulting model is never directly exposed to the sensitive in-domain target dataset.

---

Published as a conference paper at ICLR 2026
Due to these advantages, curation is widely used in practice. Curated datasets routinely released to the public (Penedo et al. 2023 Li et al. 2024b] Penedo et al., 2024), and in some cases, even the intermediate quality scores made available (Together Computer 2023). Furthermore, there exists a growing market of data curation as a service, where datasets, scores, and subsets are exchanged between organizations (DatalogyA]] |Snokell| (ScaleTT) Yet, the privacy risks for the target data under such practices are, to date, not well understood. To address this gap, in this paper, we provide the first systematic study of privacy risks in curation pipelines. Therefore, we design and carry out custom membership inference attacks (Shokri et al., 20177, ( et al., (222a) on data points from the target set at every stage of the pipeline, and demonstrate that each step can leak private information.
Concretely, we design and eval- uate attacks against 1) the cu- ration methods' released scores, 2) the selected public subsets, and 3) the final trained model, shown in Figure T To attack curation scores, we employ Likelihood Ra- tio Attack (LiRA) (Carlini et al. 2022a) with signal filtering (Jagiel- ski et al. 2023) and custom at- tacks based on voting schemes. We show that Image-based curation's s nearest-neighbor mechanism creates high vulnerability for samples with non-zero influence, while TRAK's ss averaging provides better protection but remains vulnerable for small target datasets. To attack the curated public subset, we adapt LiRA (which typically operates on continuous model outputs) to the binary setting, where the adversary only knows if sample was picked by the curation or not. Additionally, we design a custom attack that iteratively performs membership inference by exploiting the deterministic behavior of the curation mechanism. Finally, attack the trained model, we insert a few fingerprinted samples F into the public pool whose curation score is highly influenced by a particular target, and which imprint measurable signal in the model if trained on. The addition of such samples represents a realistic setup where the public pool data is often scraped from the internet, and Carlini et al.) (0224) have shown that it is possible to introduce targetly manipulated data points into ML model training pipelines this way. Because our attacks succeed with only a small number of inserted samples, they hold the potential to pose a risk in real-world curation settings where public data is reused. Additionally, we explore differential privacy as defense and show that it can mitigate these risks, though a thorough investigation of the privacy-utility trade-off remains an important direction for future work. Overall, our findings underscore that duration-guider approaches demand privacy-aware design to prevent leakage of private target data information.
Public Data
Scores for D
URA Custom Attacks
Curation
Public Subeet 1
Binary LiRA Custom Altacks
Private Data T
Trained Model M
Custom End to- End Attacks
We attack private data T used to curate a public dataset D. We show that the scores top-scoring subsets 'D and even trained models M leak membership information.
Figure I:
To summarize, we make the following main contributions:
1. We present the first comprehensive privacy analysis of data curation pipelines, highlighting that curation leaks private information at each step: the scores, curation sets, and the final model.
We design custom membership attacks for each curation step, showing that both curation scores and curated datasets leak membership information without pipeline modifications.
3. We show that even end-to-end attacks on the final model can leak target data information by inserting only a small number of crafted samples into public datasets.
4. Our empirical evaluation on six datasets and two curation methods shows that, while TRAK is more robust than image-based based curation, it remains highly vulnerable for small curation target datasets, the very scenario motivating curation in sensitive domains.
2 BACKGROUND AND RELATED WORK
Data Attribution. The influence of training samples on the behaviour of trained ML model on some target data point is determined via data attribution methods. The DataModels approach (yyas) et al. 2022) trains many models different training data subsets and then fits an influence estimator. TRAK (Park et al.) 2023) peforms data attribution using closed-form influence functions for Logistic Regression. To make that applicable to deep learning, they formulate the model training
2

---

Published as a conference paper at ICLR 2026
logistic regression on gradients of trained model. With that approach, TRAK is more efficient than DataModels, but still requires training in the order of ~ 20 models and another backward pass for each on the full training set. Iyyss & Engstrom (2025) introduce an alternative which requires training just one model, but requires a full backward pass on the entire training dataset for each target sample. By selecting those training samples with the highest positive attribution scores, data attribution methods can serve as methods for data curation (Engstrom et al. (024)).
Data Curation. The goal of data curation is to select the most valuable data for training. Gadre et al. (2023) introduced DataComp, an ML challenge of obtaining the highest accuracy just by modifying the training data. They also developed image-basee based filtering for selecting appropriate images based on semantic similarity. Thrush et al.] (225)) propose attributing training data performance to target utility by computing correlations on the loss of the training data and the target utility. Notably, this allows computing correlations for data that the models have not been trained on, sparing the computationally expensive setup that e.g., TRAK- or DataModels- based curation requires.
In this work, we focus on two representative curation methods, namely image embedding- curation (Gadre et al. 2023) and TRAK (Park et al. 2023). In both cases, we have a public dataset ((,)=r We call curating for TT when we try to obtain
D= = [ J and a private dataset T
subset D C D s.t. the performance and/or compute efficieny for utility on T is improved by training on D instead of the full D.
Image Embedding- based Curation. Image embedding- based curation assigns scores based on the cosine similarity of image-embeddings. For each image x; in the public curation pool D, the score is the maximum similarity to a sample 1, in the private target set T, iee., s(x;) = maxjex (1)()()))) ((t,)) where ((-) is the embedding function.
TRAK. TRAK computes attribution scores via projected gradients to identify influential pool samples. Following Engstrom et al. (2024), in this work, we compute our scores the average attribution score on ie., s(.,) - >3 + G2 where GT are the gradients of T. & are the TRAK features. We obtain them from the gradients X of D as + X(XTx)-IQ Q are the scaling factors, determined as the gradient of model output to loss.
A Assessing Privacy Risks in ML. The de facto standard for assessing the privacy risks in ML systems is to rely on Membership Inference Attacks (MIAs) (Shokri et al. 2017 Carlini et al. 2022a). A popular attack perform membership inference LiRA (Carlini et al. 2022a). It trains shadow models on various subsets of the target data and then fits distributions to the observed behaviour of the models with a target vs. those without a target. Then, to attack model, compares the likelihoods of the observed behaviour under the distribution of the target being in the training vs. being not in the training data. The ratio these likelihoods serves as the membership score. In this work, we adapt LiRA, but instead of training shadow models, we perform curation random subsets of the target data to obtain shadow sets, i.e., the curation results for the random subsets. We then select high-signal measurements similar to Dagielski et al. (2023), i.e., those that differ most significantly for specific target, and compute the membership inference scores as the likeliiid ratios (Section 3.2)
ATTACKING DATA CURATION PIPELINES
In this section, explore privacy leakage from the different stages of data curation pipelines outlined in Figure We start off by outlining the threat model and adversary goal and capabilities in Section 3.1 Then, attack three progressively harder threat models: (I) continuous curation scores (Section 3.3). , (2) binary selection masks (Section 3.4). and (3) final trained models (Section (355).
Finally, we summarize attacks in Section 3.6
3.1 THREAT MODEL
Adversary Goal. Our attacks infer membership in the private target set Teel C T used for curation, Le., for a given target whether E TTet. A target sample i can influence the curation scores 8, and through that, subsequently, the curated public dataset D and, finally, the model M trained on D. Unlike in classical MIA (Shokri al., 2017: Carlini et al. 2022ab / never part of the training data of the model M directly, iee., DNT
0.
3

---

Published as a conference paper at ICLR 2026
Attack Surface
Goal
Knowledge
Capabilities
Observations
Scores (Section
Passive observation
Scores & E RR/P
Public pool DD
Subset (Section
3.4
Passive observation
Selection m. E {0, 1}IP)
Infer
target set T, curation algorithm
Final Model (Section
3.5
Tsa. C
Inject fingerprinted samples JF into D
Trained model M (black- box query access)
before curation
Table Threat model summary. All adversaries aim to infer membership in the private curation target set Tad (not training-set set membership). Scores provide fine- fine-grained ranking information, subsets reveal only binary selection, and final models require active poisoning with detectable fingerprints.
Adversary Capabilities and Knowledge. Across all pipeline stages, we assume the following ad adversary knowledge; (1) The full public pool 'D. Since such pools are often web-scale dataset from the internet, we assume the adversary can obtain it. (2) The target dataset T. This follows standard assumptions in MIAs (Shokri et al. 2017) where the adversary knows the target samples whose membership they want to infer. (3) The curation algorithm used (e.g., Image-based or TRAK). Curation methods are often open-source or disclosed in model documentation. For adversary capabilities we assume the adversary can observe only the outcome of the respective curation stage, i.e., the scores the selection mask or (through black-box query access) the trained model M. Only for attacks on the final models we assume small part of the public pool can be poisoned, which Carlini al. (2024) have shown to be realistic. Table summarizes threat model.
3.2 ADAPTING LIRA FOR CURATION
LiRA (Carlini et al. 2022a) is theoretically grounded membership inference test based on likelihood ratios. We adapt as our principled baseline across all curation stages. The key adaptation is to replace LiRA's shadow models with shadow curation sets. Concretely, we sample m different random subsets from the target dataset T and perform curation based on each them. Each target is in exactly half of the random subsets, ensuring an unbiased estimate of the in/out distributions. The resulting shadow sets {s; E R 1 with N = [D] of curation outputs take the role of the shadow models in the original LiRA (see Section B. for details). For each shadow set, the adversary has ground truth membership information.
A key challenge that the public pool typically very large (e.g. N 12.8M for CommonPool), so each target's 's membership signal is diluted across millions of curation outputs. Inspired by [agielski al. (2023), who address an analogous signal-dilution problem for language model distillation, select only the single most informative public sample per target, namely the one whose curation output shows the largest difference between the member and non-member distributions:
ke arg max EsaeT, (5) EjaaTT, (8) [s() RE(N]
(1)
Given Ae", LiRA computes membership inference score vtt) log- ligeliillihood ratio over the in/out distributions of the selected sample's s curation output. The exact distributional form depends on the
observation type-continuous scores (Section 3.3 vs. binary selection masks (Section 3.4 and is detailed in the respective sections.
All attacks produce scalar membership inference score v(() per target higher values indicate stronger evidence of membership. Beyond the principled LiRA adaptations, we also design custom attacks that more targetedly exploit specific curation algorithm characteristics.
3.3 SCORE-BASER ATTACKS
We first analyze the privacy risks that arise from access to continuous curation scores.
LiRA for Scores. Using the shadow sets from Section 3.2 we observe continuous score vectors. For the selected public sample k* (t), we fit Gaussian distributions to the in-member and out-member score populations and compute the membership inference score as
UIRA (1) log.(5k=<(e [N oi))
log ((5k(()) [N (uml) ddu))
4

---

Published as a conference paper at ICLR 2026
where (c)(() is the score of the most informative public sample identified via Equation
Custom Voting (Image- based), Image-based curation's is deterministic nearest-neighhor structure (s(x) max; cos(Ø(x), (t))) enables reverse-engineering: for each public sample, identify which target t" = arg min, {(()) -sim (()), o(t)) was responsible, increment (tore(t)) (positive evidence), and decrement votes for all t where sim()(z), ((t)) > s(x) (negative evidence, i.e., if t had been present, s(x) would be higher). The membership inference scores are the number of votes Vyote-
Least Squares (TRAK). TRAK- TRAK-based curation computes scores as s(x) + E. ((x) G. Because the score is linear combination of per-target contributions, we can directly solve for the unknown membership mask that best explains the observed scores. We denote m {0, 1/TT|)
the masked mean operator. To recover membership signals, we then solve
minimize me R
(x) T G.m - =s
(3)
The membership scores are the optimal weights Ulasy" = m. App. B.4.2 further details this.
3.4 SUBSET SELECTION ATTACKS
Assessing the privacy risks of the curated public subset is significantly more challenging than attacking the curation scores. This is because the scores yield a fine-grained ranking of the public samples whereas the curated dataset itself can be considered as a binary mask that only indicates for each public data point in the pool D whether it was included into the curated set D.
LiRA for Subsets. While the original LiRA is designed to operate on continuous output logits, we need our attack to operate on binary selection observations. Using the shadow sets from Section we binarize the curation outputs to top-k masks {s; E 10, J fj=1 representing which public samples were selected.
In naïve setup, we directly model these binary outcomes wweethe each public sample was selected (I) or not (0) in each shadow curation-usine Bernoulli distributions. For each E we compute Min E IR-V as the average of 8; for every Ti and joot E R where L E T. These are the frequencies of public samples being the top set depending on whether was part of the curation target. Using the kn filtering from Section 3.2 select for each target the pool sample 2, that most indicative of t's presence, i.e., shows the highest difference between Hin, and Mour... We compute the membership inference score looking only the binary signal of whether 2; is present not as the log- eigliiellhood ratio for the Bernoulli distribution
P( Min,e)
M (1 (111-1mmn)) MMn,,) 1
UBnaayy LiRA (1) log
log
P(x., Momu,)
Moui, [ - Monn, 1-
We also experiment with fitting the Bernoulli distribution with continuous labels (App.
B.2
basen Membership Inference (Image-based). As an alternative to LiRA for Image- based curation, we exploit its deterministic nearest-neighbor structure. We define overweighted samples as those selected by our hypothesis but not by the target (æ E D, but x & D), suggesting our hypothesis includes incorrect targets. Conversely, underweighted samples (x D but x & D2) suggest missing targets in our hypothesis. We initialize To = {0} and vote accumulator Unerative (t) = 0 for each t T. At iteration é:
= CurateeD D, Tr), O DD D. u, = D\ D.
(5)
Update votes for each target E T. where (x) = arg max sim sim() o(t') denotes the nearest target to .::
Ultrrtiive (t) += Σ = Σ [x==(()))
(6)
seld.
#EO,
Keep targets with positive of votes
Tr E T, Untrrtive (t) 0}.
(7)
5

---

Published as a conference paper at ICLR 2026
10a
10
10'
10
107
12
10P
107
10
101
185
100
1000
2000
1000
2000
1.000
2000
5000
(a) CIFAR-10
(b) RESISC45
(c) PCAM
(d) STLI0
Figure 2: Influence sparsity in Image-based curation. Distribution of how many public samples have each target their nearest neighbor. The concentration at zero demonstrates that most targets have no direct influence on curation scores, necessitating our fingerprinting approach.
The algorithm terminates when the curated set from our hypothesis closely matches the observed curated set (J(D,, D) 2 8), where -): denotes Jaccard similarity. We note that multiple target sets can produce identical curated outputs due to the many-to-on to-e mapping from targets to selected samples. The final vote counts Ulterative (1) serve as membership inference scores: higher votes indicate stronger evidence that target was present in the private curation set.
For TRAK-based curation, we rely on the Binary LiRA scores UBinary LiRA from above, as TRAK's averaging does not expose the per-target structure that the voting attack exploits. We also evaluate variant with continuous shadow labels (App. B.2).
3.5 FINAL MODELS ATTACK
Finally, we assess privacy leakage of the target data from the final model M trained purely on the curated public data. This represents the highest threat as such end-to-end attack could actually be instantiated against exposed models that are trained based on curated data.
To successfully extract membership information in this end-to-end setup. we rely on inserting a few modified samples F into the large curation pool. This is realistic setup, as Carlini al. (2024) have shown various practical ways of putting adversarial examples into web-scale training data. Samples in F must satisfy two requirements:
1. Selective triggering: Each sample must be selected during curation and only if a specific target t ' is present in the private target This requires crafting samples whose curation scores sensitive to individual targets rather than the aggregate properties of TT
Detectable fingerprint: Selected samples must imprint measurable signal in the trained model M that the adversary can later detect to infer membership.
We empirically validate that fingerprinted samples imprint a detectable signal in the final model (Figures 10 and IT in App. (.2)): the signal does not degrade model utility but leads to elevated probabilities on semantically unrelated concepts. For as few as fingerprint samples, the signal remains constant for training dataset sizes up to (0000,000(ppossooooo
rate 0.0005%).
Since training large models for each experiment is intractable, we validate the signal once and reduce the end-to end-to-end attack to observing whether the fingerprinted samples selected during curation (i.e.. f e D). We now detail how construct JF and compute membership scores for each curation method.
In image-Based Curation, selection from the pool depends solely on image embeddings. Hence, we can modify text captions arbitrarily without affecting curation scores. Being able to arbitrarily modify captions has been shown sufficient for imprinting measurable signal in trained models (Carlini & Terzis 2022). However, achieving selective triggering remains challenging due to the sparse influence structure of nearest-neighhor selection. As illustrated in Figure 2 in the unmodified setup, most target samples have zero influence on public sample scores under nearest-neighte
curation. E.g. for PCAM and STL10, 98.28% and 34.16% of target samples are not the nearest neighbor for any public sample, respectively. This influence sparsity creates both a challenge (most targets are unattackable by default) and an opportunity (influenced samples have strong membership signal).
This is the maximum evaluated, not necessarily where the signal breaks down.

---

Published as a conference paper at ICLR 2026
Algorithm 1 TRAK Membership Inference via Fingerprint Detection
Require: Pool data X, target gradients Y, fingerprints F, threshold assumption Ensure: Membership scores for each target
G - X + = [Cholesky decomposition)
1 - E|], + Cov[Y] [Target statistics)
+ FG Y (Signal matrix]
1/ + V " G ft for all (Noise scales)
Initialize score, 0 for all targets
for each target j E [n] do
(j) < arg max; [S,y/m (Best fingerprint] Compute 2H, via Aherman-Morrisoe
PHo PH, + percentile ranks of 2H. 2H, confidence, [clip(pn, , P. 100) clip(pm, P. 100)
end for
for each observed fingerprint ʃ E D do
for each target j where {]) = do
PH. I p then |Fingerprint crosses threshold) if PH, > PHo
score, <- scorej + confidence,
end if end for end for return {score,) }
We construct candidates for F from images in D paired with semantically unrelated captions (e.g., "ratatouille" for CIFAR-10 targets). Unrelated captions ensure detectability: if selected, these fingerprints imprint measurable signal in the trained model that would not appear otherwise. For each candidate fingerprint f and target t,, we compute a correspondence score balancing two objectives. Let s(a, b) = sim(Ø(a). ((b)) denote embedding similarity, then
(1 (1-maxx(f:t) C'tt
score(f,t)
sfiti)
+(1 +(1-a)
(8)
attraction f
repulsion from t'ght4
where E [0, 1] trades off between proximity to the intended target and separation from samples the target set. Our attack inserts as fingerprints F farg max score t4)lt, T). i.e. the
FED
highest-scoring sample for each target (not always uniquely, i.e., IT)).
Given the observed fingerprint selections, we compute membership scores as follows: Let pá denote the baseline percentile rank of fingerprinted sample f when scored against the full target T. The adversary does not know the selection threshold r but can conservatively assume they will pick from the top 50% of the pool. We model the probability of selecting ff without its corresponding target as:
Po(f) = + exp(-(p6 7)/a)*
where a controls the transition sharpness. For each fingerprinted sample we measure to be in we set the membership inference score of the corresponding target(s ( to the surprise of the measurement of that sample under Ho divided by the number of targets that share that fingerprint.
UEEE Inn. (t.) 1 selected] ]
P( selected Po(J))
(10)
Nsharing (ff)
For TRAK-based Curation, face the challenge that TRAK explicitly penalizes mislabeled samples through gradient alignment scoring (Park et al., 2023), rendering direct caption manipulation ineffective. However, discover that appending semantically orthogonal information to correct
Target(s) in plural, as the mapping is necessarily unique.
7

---

Published as a conference paper at ICLR 2026
URA
Custom
Random
Binary LIA
Sigmold Brary LIA
Custem
Rando
10a
10"
10"
10"
Tue Pasative Bate
Tue Positive Rate
20-1
20
10-1
10-1
20-2
10
20
10-2
10
10
False Positive Rate
False Poctive Rate
False Positive Rate
Fillsn Positive Rate
Image-based
(b) TRAK
(d) Image-based
TRAK
(c) Scores
Without Scores
Figure 3: Attack success for curation scores and subsets. Image-based curation's 's nearest-neighhor
mechanism is highly vulnerable, while TRAK's gradient averaging shows almost no leakage.
captions (e-g., "an image of airplane and ratatouille") preserves high TRAK scores while enabling detectable model changes (Figure[I]) This works because TRAK evaluates gradient alignment along task-relevant dimensions; orthogonal additions contribute negligible projection onto these directions.
We construct fingerprint candidates by copying target samples and augmenting their captions with orthogonal signals. For each candidate-taie target pair, we compute signal-to-noiss ratio measuring how sensitively the fingerprint's score responds to that target's 's presence versus interference from other targets. We select the highest-SNR, fingerprint for each target to form F. The membership score for target depends whether its fingerprint crosses the selection threshold when is present but not when absent:
confidence,
if fr D otherwise
UE2E Trak Tra)()))
(11)
where confidence, quantifies the percentile shift between two hypotheses. Algorithm provides the complete TRAK attack procedure; full mathematical details appear in App. C.3 attack procedures are detailed in Algorithms and
The Image-based
3.6 SUMMARY OF ATTACKS
We developed attacks across the three pipeline stages. Our Score-Based Attacks (Section 3.3 infer membership information from the curation scores. The (1) LiRA-based attack adapts likelihood ratio tests by running shadow curation processes and comparing score distributions between member and non-member scenarios for both image-based and TRAK-basee methods. (2) Voting Scheme exploits Image-based curation by reverse-enginee reverse-engineeriin nearest-neighbor relationships through voting across shadow predictions. (3) Least Squares solves linear systems that relate TRAK-bast curation scores to target membership. Our Subset Selection Attacks (Section 3.4) target the binary selection decisions made by curation. (4) Binary LiRA adapts the likelihood ratio framework binary subset selection by modeling selection probabilities as Bernoulli distributions and comparing likelihoods under member versus non-member hypotheses. (5) Iterative Voting Scheme iteratively refines a hypothesis of the target set by running curation on candidate sets and adjusting the hypothesis until the resulting selection matches that of the target set. Finally, our End-to-End Model Attacks (Section 3.5) exploit the downstream model trained on curated data. (6) Fingerprinting (Image- based) uses mislabeled captions as fingerprints, selecting them via correspondence scoring to ensure they appear only when specific private examples are present in curation. (7) Fingerprinting (TRAK) employs benign captions with added orthogonal information as fingerprints, selecting them based on membership signal-to-noise ratio and selection chance.
EVALUATION
Experimental Setup. We evaluate attacks using CommonPool (small) (Gadre al. 2023) the public dataset D, containing 12.8M samples. We evaluate across six diverse target datasets spanning natural images, medical imaging, and satellite imagery cover wide range of curation setups:
8

---

Published as a conference paper at ICLR 2026
Binary LIRA
LIRA
TPR 1% FRX
Voting Scheme
TPr 1% PPR
Sig. Binary LIRA
10-1
Random
10"4
Ittrativve Reconstruction
Random
10
Score-based attacks. LiRA and custom voting achieve high success all datasets.
(a)
10
(b) Binary selection attacks. Adapted LiRA variants and iterative reconstruction re- main effective.
otarto
foodll
poam
clarto
o16r200
pram
reacds
nio
Dataset
Dataset
Figure 4: Attack success correlates with influence patterns from Figure Cross-dataset compari- son shows TPR at 1% FPR inversely correlates with the percentage of zero-influence targets.
1.5
10
FPA
TPR @ 1% FPR
199 7PR
1% TRR
10-44
30-1
20.0
10-1
TPR @
TFR
10
30-4
19rl
Attack
10"
107
10"
10a
10
16a
102
10
Private Target
Private Target Size
Privao Target
Privaze Target Size
(a) CIFAR-10
(b) RESISC45
(c) CIFAR-I0
(d) RESISC45
Image-based
TRAK
Figure 5: End-to-end membership inference success. Image-basse based curation shows consistent partial leakage while TRAK exhibits size-dependent vulnerability.
CIFAR-10/100 (Krizhevsky) 2009), STL-10 (Coates et al. (0011), RESISC45 (Cheng al. 2017), PatchCamelyon (Veeling et al. 2018), and Food101 (Bossard al.) 2014). For end-to-end attacks, train models following DataC ommp small-scale (see Section C for full details). We obtain image embeddings from OpenAl's CLIP ViT-L/14 model (Radford et al. 2021]. Gradients for TRAK are obtained from model trained the full pool dataset ID using contrastive training in Gadre et al. (2023). For LIRA attacks, we use 256 shadow sets per configuration (see Section C for details).
4.1 ATTACK PERFORMANCE ON CURATION SCORES
Image-based curation exhibits high attack success rates. The dataset-specific variations in attack success for Image-based curation align with our analysis from Figure namely that datasets with higher influence concentration (fewer targets affecting many public samples) exhibit greater vul- nerability. This holds regardless the underlying data type-satellite imagery (RESISC45) with less zero-influence samples is more attackable and medical images (PatchCamelyon) with more zero-influence samples less attackable than CIFAR 10. In contrast, TRAK demonstrates natural protection through its averaging mechanism, giving near-random attack performance (AUC z 0.5) as individual target contributions become diluted through aggregation and dimensionality reduction.
4.2 ATTACK PERFORMANCE ON BINARY SELECTIONS
Even when restricted to observing only binary selection patterns, Image-based curation remains vulnerable membership inference attacks (Section 4p. Our iterative reconstruction algorithm successfully recovers the private target set for all samples with non-zero influence, though the large fraction of vero-influence samples provides natural protection for the remaining targets. This success generalizes over various datasets (Figure 4b).
4.3 END-TO-END ATTACKS ON TRAINED MODELS
The end-to-end attack depends significantly on the target dataset size, evaluate TPR at 1% FPR for different target dataset sizes [T] and various datasets in Figure 5 We show that Image- based curation leads to moderate information leakage across all target dataset sizes, with TPR values up to 21.4% at 1% FPR for RESISC45 at |T] = 100. Targets with non-zero influence are exposed at every target size, creating bimodal privacy distribution where most samples remain protected while

---

Published as a conference paper at ICLR 2026
subset faces significant exposure. TRAK exhibits different behavior, demonstrating strong decrease in attack success with growing [7]. This size-dependent vulnerability suggests TRAK may be suitable for large-scale applications but poses significant risks for small, sensitive datasets.
4.4 ABLATIONS
We perform further ablations on various aspects of our cu- ration setups and analyze their impact on the attack success. We conduct these ablations on attacks against scores, which yield the strongest leakage signal and thus provide the clear- est insights into the factors affecting attack success. First, we analyze the impact of the number of dimensions used for Image-based embeddings and TRAK-based gradient projec- tions. Our results in Figure 37 suggest that for image-based curation, there is sweet spot. with the highest leakage at 128 dimensions. TRAK requires enough dimensions (2 1,024), below which the attack success drops. Additionally, we show how target dataset size impacts attack success. Our results in Figure 32 suggest differences across datasets: e.g., at- tack success against STL-10 remains near constant while for CIFAR-10, we observe that the success drops as the target dataset size grows. Finally, Figure (36] shows that the number of shadow models for LiRA has a varying effect on improving the attack success on different datasets for Image-based curation: e.g., double the number of shadow models from 128 to 256 for CIFAR- 10 increases AUC by (abs.) 10% while for PCAM only 1%. For TRAK, we do not observe any improvement in attack success when adding more shadow models. In App. E. provide: (1) full ROC curves across all six datasets for score-based (Figures [4] and I6 , selection-based (Figures 15 and 17) and end-to-end attacks (Figures 18 and and Figures 26 to 3T). (2) target size ablations showing how larger target datasets provide a shielding effect (Figures 32 and 34). (3) shadow model scaling showing different improvements for different datasets (Figure 36). (4) embedding dimension ablations revealing optimal attack configurations (Figure 37). and (5) caption design ablations for TRAK fingerprints (App. C.5)
TRAKK
based
TPR @ 1% FPR 196)
101
102 10" 10a
Privacy Budget
Figure 6: Attack success versus for CIFAR-I0 (77) 1000).
5 MITIGATING PRIVACY LEAKAGE
Having demonstrated that data curation pipelines pose real privacy risks (Section 4 we now turn to mitigations. Differential privacy (DP) is the de facto standard for providing formal privacy guarantees (Dwork et al. 2006a). We propose DP adaptations of both curation methods to yield (e. d)- DP guarantees via the Gaussian mechanism (Dwork et al. 2006b). For DP Image-based curation we add calibrated Gaussian noise to each per-target similarity before selecting the maximum (Report Noisy Max (Dwork et al.] 20144). For DP TRAK-based curation we privatize the mean gradient computation, similar to Abadi et al. (216)). App. D.I details the algorithms. Our DP adaptations mitigate leakage (App. 4p: at E 100, TPR 1% FPR drops from 98. 08.4% to 5.4% for Image-based and from 100% to 33.2% for TRAK, which requires stricter guarantees due to its higher-dimensional gradient space: at = 10, both drop to near- baseline (1.19 1..1% %% and 1.7%, respectively).
We further show in App. D.2 that simply removing the most vulnerable samples does not prevent leakage, exhibiting privacy onion effect (Carlini et al. 2022b) for Image-based curation.
DISCUSSION AND CONCLUSION
Our work demonstrates that data curation pipelines can leak membership information about the target datasets, exposing privacy risks at every stage, from curation scores and curated subset to the final trained model. Image-based nearest-neighhor methods are particularly vulnerable, and even state-of-the-art approaches like TRAK expose privacy risks for small target sets. Our discovered risks become practically relevant when adversaries can introduce manipulated samples in the curation pool, which is the when the pool is simply crawled from the internet. This highlights that, as curation becomes central to ML, we need novel curation methods with dedicated safeguards. such as Differential Privacy (DP), to reduce potential leakage throughout the entire data curation pipeline.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
We would like to acknowledge our sponsors, who support our research with financial and in-kind contributions: OpenAl and G-Research. We also thank members of the SprintMI ggoupp for their feedback. Responsibility for the content of this publication lies with the authors. This work was supported by the Helmholtz Association's 's Initiative and Networking Fund on the HAICORE@IZI partition.
ETHICS STATEMENT
Our work demonstrates privacy vulnerabilities in data curation methods. We conduct this research to identify and quantify these issues before they can be exploited maliciously, enabling the development of privacy-preserinng methods. Our attacks require specific capabilities like injecting target-speciffe and modified samples into public datasets, which limit their immediate applicability and provide time to implement countermeasures.
REPRODUCIBILITY STATEMENT
Reproducing our results is possible from the information provided, and we support reproduction through the release of all our attack implementations in the supplemental code, as well as providing more detail the algorithms in the appendix. What hinders reproduction is the computational cost associated with running curation and training algorithms on dataset sizes that are relevant to the topic of data curation. We argue that this is to extent unavoidable and b) our introduction of suitable proxy metrics that eliminate the need to train models improves reproducibility significantly.
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, llya Mironov, Kunal Talwar, and Li Zhang. Deep Learning with Differential Privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. -18-,18, Vienna Austria, 2016. ACM. ISBN 978 978-1-4503-413944
doi: 10. 10.1145/2976749.29788 org. org/ doi /10. 1145/21767740 0...2997313
URL https
7/dl. acm.
Meenatchi Sundaram Muthu Selva Annamalai, Borja Balle, Emiliano De Cristofaro, and Jamic
Hayes. To Shuffle or not to Shuffle: Auditing DP-SGD with Shuffling, 2024. URL https
//axxiv.org/abs: htlp//bs/224// /411, 10614
Samuel A. Assefa, Danial Dervovic, Mahmoud Mahfouz, Robert E. Tillman, Prashant Reddy, and Manuela Veloso. Generating synthetic data in finance: Opportunities, challenges and pitfalls. In Proceedings of the First ACM International Conference on Al in Finance, ICAIF '20, pp. 1-8, New York, NY, USA, October 2021. Association for Computing Machinery. ISBN 978- --500--5887- doi: (0.1145/3383455.4422555
Kyriakos Axiotis and Maxim Sviridenko. Iterative hard thresholding with adaptive regularization: Sparser solutions without sacrificing runtime. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 1175-1197, PMLR, 2022. URL httph: mlr. press/vlll httss//v112axxxottooo
html
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food- 101 Mining Discriminative Components with Random Forests. David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision - ECCV 2014, pp. 446-461, Cham, 2014. Springer International Publishing. ISBN 978-3-319-10599-4, doi: 10. (0007/978-3-19-11 -1777----1----09942221
Nicholas Carlini and Andreas Terzis. Poisoning and backdooring contrastive learning. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29.
hhpps///prrnnml..cm
11

---

Published as a conference paper at ICLR 2026
2022. OpenReview.net, 2022. URL /tt.:p/te////eeeeeree.eee//////a 1Mp
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramèr. Membership inference attacks from first principles. In 43rd IEEE Symposium Security and Privacy, SP 2022, San Francisco, CA, USA, May 22-26, 2022, pp. 1897-1914. IEEE, 2022a. doi: 2022.9
http://111111///11 10.10009SP44211.222..99.. /tt.:///000....///////////...... 833649
URL httL/httt////oii.
Nicholas Carlini, Matthew Jagielski, Chiyuan Zhang, Nicolas Papernot, Andreas Terzis, and Florian Tramer. The privacy onion effect: Memorization is relative. Advances Neural Information Processing Systems, 35:13263-13276 2022b.
Nicholas Carlini, Matthew Jagielski, Christopher A. Choquette-Chov Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramèr. Poisoning Web-Scale Training Datasets is Practical. In 2024 IEEE Symposium Security and Privacy (SP), pp. 407- 425, 2024, doi: 10.1SS99//544 2024.0017.. tt httpttss//eeeeeppeeeeeeeeee org rast /at.a/////occceee///666666111111
ISSN: 2375-1207.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 2017. doi: 10. 10.100UJPK0C.20122279999
Mehdi Cherti and Romain Beaumont. Clip benchmark, 2025. URL https:t//o...org///////// 281/zenodo. 15403103
Adam Coates, Andrew Ng, and Honglak Lee. An Analysis of Single-Layer Networks in Unsuper-
vised Feature Learning. In Proceedings of the Fourteenth International Conference Artificial
Intelligence and Statistics, pp. 215-223, JMLR Workshop and Conference Proceedings, 2011 1.
URL URRLhttts://ppcceeeiii
mlr wwwpppresssssieeee wttp:///////allleaaeeeeaeaeeaeee wtt.tttsslaaallllllll..
ISSN: 1938-7228.
DatalogyAL DatologyAl: Train Better Models, Faster and Smaller. URL https: /www datolo
ggai. com
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, arXiv:2010.11929 [cs].
June 2021. URL http: arxiv. wttp://////....11//99993
Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our Data, Ourselves: Privacy Via Distributed Noise Generation. In Serge Vaudenay Advances in Cryptology EUROCRYPT 2006, pp. 486-503, Berlin, Heidelberg, 2006a. Springer. ISBN 978-3-540-34547-3. doi: 10. 10.1007/11761679)
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating Noise to Sensitivity in Private Data Analysis. In Shai Halevi and Tal Rabin (eds.), Theory of Cryptography, Lecture Notes in Computer Science, pp. 265-284, Berlin, Heidelberg, 2006b. Springer. ISBN 978-3-540-32732-5. doi: 10.1007/116818788. 14.
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and trends@ in theoretical computer science, 93344):211 1-407, 2014.
Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: Model-aware dataset selection with datamodels. In Forty-first International Conference on Machine Learning, ICML 2024, Viemna Austria, July 21-27, 2024. OpenReview.net, 2024. URL um. Lun21d=GGBHKKe.BB
net/ for
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah M. Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander J. Ratner. Shuran Song, Hannanch Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets. In Alice Oh, Tristan Naumann, Amir
12

---

Published as a conference paper at ICLR 2026
Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Anmual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA. USA, December 10 16. 2023, 2023. |RL: fi 2alpaaaaea...ccacccaaaaaaa222ae. httpa///weappppao////aaaaaaaaaaa
ni waips.com http:/ppppee_iieepppepeeeeeeeeee http:pppprr/22/aaaaaaaaaaaae
8 Albee-AAstracct-aaaasses
Baw_annn____herrr______ html
Xin Gu, Gautam Kamath, and Zhiwei Steven Wu. Choosing public datasets for private machine learning via gradient subspace distance. In IEEE Conference on Secure and Trustworthy Machine Learning, SaTMI. 2025, Copenhagen, Denmark, April 1,11, 2025, pp. 879-900, 900. IEEE, 2025. doi: htRlhttps:/ doi org /00 11w.1119///T..5522.....
025. 00054
Andrew Ilyas and Logan Engstrom. MAGIC: Neai-Optimal Data Attribution for Deep Learning, 2025. URL https://p: org. 0tbbs//504.16433
Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclere, and Aleksander Madry. Data- models: Predicting Predictions from Training Data, 2022. URL httpps://ax.....rrrtt////
Matthew Jagielski, Milad Nasr, Katherine Lee, Christopher A. Choquette- Choo, Nicholas Carlini, and Florian Tramèr. Students parrot their teachers: Membership inference model distillation. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16. 2023, 2023. URL ht wwwpsspppprrsaaaaaae nips. paper_apppppe Saapper__illeeeee www.sieeepppeeeee wttpe//aapeeee/2/////////// b07 207722246433622e77711eee hatpr//eaaaaaaaaaaaaaaa--oorreee html
Abst recttraact-aoo
Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. 2009.
Jeffrey Li, Alex Fang, Georgios Smymis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee F. Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng- Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah M. Pratr, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jicyu Zhang, Khyathi Raghavi Chandu, Thao Nguyen, Igor Vasiljevic, Sham M. Kakade, Shuran Song. Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El-Nouby, Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alex Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. Datacomp-lm: In search of the next generation of training sets for language models. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver. BC, Canada, December 10 15, 2024, 2024a. htR.ttp::/ ps cc httppapppre...pppppppcappppepeee www.ffileessseee http:///paapeeeeeeee/e/e/ee////e and hat_resss__neeeehhheaeeeee Benchmarchhhs
Track.html
Zejian Li, Chenye Meng, Yize Li, Ling Yang. Shengyuan Zhang, Jiarui Ma, Jiayi Li, Guang Yang, Changyuan Yang, Zhiyuan Yang, Jinxiong Chang, and Lingyun Sun. LAION-SG: An enhanced large-scale dataset for training complex image-text models with structural annotations, 2024b.
URL httpt::tta/aiiiiiee org/ abs/ /etes/212222 08580 arXiv: 2412.08580 [cs.CV].
llya Loshchilov and Frank Hutter Decoupled Weight Decay Regularization, January 2019. URL arXiv: 17111 1.01 101101 [cs].
http:
Saeed Mahloujifar, Luca Melis, and Kamalika Chaudhuri. Auditing STS-Differential Privacy in One Run, 2024. URL
Pratyush Maini, Skyler Seto, Richard He Bai, David Grangier. Yizhe Zhang, and Navdeep Jaitly. Rephrasing the web: A recipe for compute and data-efficient language modeling. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACI. 2024, Bangkok, Thailand,
13

---

Published as a conference paper at ICLR 2026
August 11-16. 2024, pp. 14044-1444 14072. Association for Computational Linguistics, 2024. doi: 11066////2224ALLLLLONN fLong.751
B. K. Natarajan. Sparse Approximate Solutions to Linear Systems. SIAM Journal on Computing, 24(2):227-333 2344, 1995. ISSN 0097-5397. doi: 10. 10.11177/5009753399244440 URtstttss: //epubs. s siam. org/ do abs/ /1....11//00007333772220000 for Industrial and Applied Mathematics.
Publisher: Society
Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclere, and Aleksander Madry. TRAK:
attributing model behavior at scale. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 27074-27113, PMLR, 2023. URL htttps:/pproceeddngg wle.spres/ee wttp://v22//ppee http://w2222paaa23...3em html
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The RefinedWeb dataset for falcon LLM: Outperforming curated corpora with web data, and web data only, 2023.
URL UtLhttps::/ http:pp:://aaiivai http///wwww...r////////////// 01 01116 arXiv: 2306.011 16 [cs.CL].
Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The FineWeb datasets: Decanting the web for the finest text data at scale, 2024. /ML.appss/axx.....cr//////////// 2406.17557 [Cs.CL].
www.wx.ooo//ab///eeea/o 17557 arXiv:
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML. 202 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8766 PMLR, 2021. URL http: httt:::/proceedinng mlrr www.ppessssili. htte//rdfoorr2d/// 8 ml
ScaleAl. Scale AI URL
com
Raphael Schäfer, Till Nicke, Henning Höfener, Annkristin Lange, Dorit Merhof, Friedrich Feuerhake, Volkmar Schulz, Johannes Lotz, and Fabian Kiessling. Overcoming data scareity in biomedical imaging with a foundational multi-task model. Nature Computational Science, 4(7))495-500 July 2024. ISSN 2662-8457.
doi: 00.10388/433888024-006666
Reza Shokri, Marco Stronati, Congzheng Song. and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 20/7 IEEE Symposium on Security and Privacy, SP 2017, San Jose, CA. USA. May 22 2017, pp. 3-18. IEEE Computer Society, 2017. doi: 10.1109/SP.2011 41. btRtsttps:/ doi. org /t.zg/11..100 10.1109/S2 2017.41
Snorkel. Snorkel A1 Helping model providers and Al development teams push the boundaries of AI. URL https::/snookel.....
Tristan Thrush, Christopher Potts, and Tatsunori Hashimoto. Improving pretraining data using perplexity correlations. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL /ttttt//////eeeee//// huukoov@
ew
Together Computer. RedPajama: an open dataset for training large language models, October 2023. URL htttss://
com htt.uub....ooooottttteteee togethe www.rrctooeeeeeteoeeeeo htwp://oooeeeeeeeteeeeeeeeeeee
jama Data
Bastiaan S. Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivariant cnns for digital pathology.
Alejandro F. Frangi, Julia A. Schnabel, Christos Davatzikos, Carlos Alberola-López. and Gabor Fichtinger (eds.), Medical Image Computing and Computer Assisted Intervention MICCAI 2018 21st International Conference, Granada, Spain, September 16-20, 2018. Proceedings, Part II, volume 11071 of Lecture Notes in Com- puter Science, pp. 210-218. 218. Springer, 2018 doi: 10. 1007/978- 0---0000-000 00934-21 24. URL 1000..1077700 000077778-00 0000079977---000000----------0-0
httpp://ddo
14

---

Published as a conference paper at ICLR 2026
Shanshan Wu, Zheng Xu, Yanxiang Zhang, Yuanbo Zhang, and Daniel Ramage. Prompt Public Large Language Models to Synthesize Data for Private On-device Applications, 2024. URL
https://arxiv. org/ 2404. 04360
Da Yu, Sivakanth Gopi, Janardhan Kulkarni, Zinan Lin, Saurabh Naik, Tomasz Lukasz Religa, Jian Yin, and Huishuai Zhang. Selective pre-training for private fine-tuning, Trans. Mach. Learn. Res., 2024, 2024. URL htttps://oppnreee///s net/ fot.eorrmmmmyyyy3000p
15

---

Published as a conference paper at ICLR 2026
A
CURATION METHODS
Table 2: Overview of Curation Methods
Method
Description
Score Function
Image-based
Identifies relevant pool data using image embeddings
s(x)
maxieT cos( cos()((r), e()))
where Φ6-) is the embedding func- tion
TRAK
Computes attribution scores via pro- jected gradients to identify influen- tial pool samples
<xx) 17 Eier ((x) G, where + are projected and out-to-loss- scaled features
A.1 IMAGE-BASED SCORING
We employ Image-based nearest neighbor distances inspired by Gadre et al. (2023). Following the DataComp methodology, we use embeddings from OpenAl's pretrained CLIP ViT-L/14 model (Rad) ford et al. 2021 to compute the image representations o(-). For each pool sample z; D, the curation score is computed the maximum cosine similarity to any target sample in T, i.e. the pool sample is scored based on its closest (most similar) target sample. This nearest-neighhor mechanism creates deterministic relationship where each pool sample's score is determined by exactly one target sample.
A.2 TRAK
The TRAK algorithm (Park et al. 2023) for computing influence scores is formally presented in Algorithm
Algorithm 2 TRAK Algorithm for Influence Computation
Require: Training dataset D. target dataset T, model fo. projection dimension dpeuj
Ensure:
Influence scores of training samples target samples
Girain foD (Gradients for training data)
Qrrain + V 5o C(D, fe) (Output-to-loss gradients}
Graret + VT) (Gradients for target data]
P GandommProjctionnMann (atatriiiiimmi dproj) (Typpccally Rademacher]
G + Guain P |Project training gradients)
G Gurget +- Grarget P (Project target gradients)
X
X G. Girain (Compute Gram matrix)
Guain (X (X) (Compute features)
+ Ginge, [Raw scores)
Sraw
[7
Sraw ④ Okurain (Scale by output-to-loss gradients)
return S (Final influence scores)
B ATTACKING CURATION SCORES
B.1 ATTACK DESIGN RATIONALE
We design our attacks exploit the specific mathematical structure of each curation method. While no attack can be proven universally optimal without strong distributional assumptions, our approaches are theoretically grounded and, in the case of Image-based scoring, exploit fundamental limits of what can be inferred from the exposed information.
Method-Agnostii Attacks via LiRA. For purpose membership inference, we employ the Likelihood Ratio Attack (LiRA) (Carlini al. 2022a). This attack is theoretically motivated by the
16

---

Published as a conference paper at ICLR 2026
lemma, which establishes that the likelihood ratio test is optimal for binary hy- pothesis testing (membership vs. non-memeen non-membership) under the assumption that the score distributions known or be accurately estimated. LiRA provides principled baseline that makes minimal assumptions about the curation method's internal structure.
We make two key adaptations for curation pipelines. First, replace shadow models with shadow curation runs,ie. instantiations of the curation algorithm using different random subsets of T, s.t. each target sample is part of exactly half of the random subsets. Second, following Jagielski al. (2023), we filter the curation scores to select only the public sample per target with the maximum difference expected scores between member and non- nn-member hypotheses (Equation
This preserves the essential statistical properties needed for LiRA: we can empirically model P(s|t e T) and P(s]t & T) from shadow obscrvations. Figure 7a illustttts the resulting in and score distributions. We can then formulate a likelihood ratio attack on the curation scores by computing the log-ratio of the in and out score distributions (Equation (2). Figure 7bjillustrates the mapping from measured scores to the resulting membership inference score. Given sufficient shadow sets and correct distributional assumptions (e.g., Gaussian for continuous scores, Bernoulli for binary selections), LiRA approximates the theoretically optimal likelihood ratio test.
0.8
Out
Membership Score
0.6
0.5
Density
0.4
0.0
0.2
-1.0
0.0
-1.5
-2
Score
Observed Curation Score
(a) In and out score distributions.
(b) Membership inference score.
Figure 7: LiRA be adapted to curation pipelines. (a) We can empirically model P(s|t e T) and P(s]t & T) from shadow observations. (b) We can then compute membership inference scores based on likelihood ratio attack by computing the log-ratio of the in and out score distributions.
Image-Based Curation: Exploiting Deterministic Structure. For Image- baaee- scoring with nearest- neighbor retrieval (s(x) = maXTeT cos(Ø(z)) (os))))))(())) the max operation creates a deterministic function where each pool sample's ' score is determined by exactly one target sample: its nearest neighbor. This structure enables perfect reverse-engineering: given s(x) and the embeddings, we can identify which t" T was responsible via t" arg min, cos((( o(t)) s(x)]-
Our custom voting attack exploits this theoretical limit, since only nearest- -eirhst relationships are exposed, and our attack recovers exactly these relationships. Positive votes accumulate for targets that are nearest neighbors to pool samples, while negative votes identify targets that would have produced higher scores if present. This is fundamentally more powerful than LiRA because it does not rely on distributional assumptions or shadow set approximations: it deterministically extracts the membership signal embedded in the nearest-neighhor structure.
The attack's 's effectiveness is constrained only by: (1) the sparsity of nearest-neighhor relationships (many targets may not be nearest neighbors to any pool sample, as shown in the main paper), and (2) numerical precision in matching scores to similarities. Unlike LiRA, which requires many shadow sets to approximate distributions, our deterministic attack needs only the scoring function's 's mathematical structure and embedding access.
B. ORACLE ATTACK SETUP
To establish upper bound attack performance for Image- based scoring, define oracle attack that leverages perfect knowledge of the curation mechanism. The oracle operates under the following membership scoring scheme for each target sample & T:
17

---

Published as a conference paper at ICLR 2026
10a
1.00
True Positive Rate
TPR 1% FPR
10
0.75
0.50
10
Dracle
LIR
LRA
0.25
10
Random
0.00
Rendom
clfr10
cifar100
food101
pcam
reslsc45
stl10
10
10
10-?
10"
False Positive Rate
Dataset
(a) CIFAR-10
(b) Multiple Datasets
Figure 8: Our custom attack on Image-based scores matches oracle performance. With access to the scores, the attack achieves the same utility the oracle attack.
if is the nearest neighbor of any pool sample
membership__cooe(()
0
if Fr E D cos()(r), x()) > s(x)
(12)
0.5
otherwise (no influence scores)
The rationale for this scoring scheme is as follows:
target sample is responsible for the observed score of least one pool sample (i.e., it is the nearest neighbor in T for that pool sample), then the oracle has definitive evidence that E Teel-
Score (Member):
Score :Non-member): the oracle can identify pool samples where the similarity to exceeds the observed score, this indicates that t would have produced higher score if it were in Tag. Since it did not, the oracle can definitively conclude that t Teel-
Score 0.5 (Unknown): For target samples that have observable influence any pool scores-neither as nearest neighbors nor as potential higher-scoring alternatives-even the most powerful oracle only guess their membership status uniformly random.
This oracle attack represents the theoretical maximum achievable performance given the information exposed by the Image-basee based scoring mechanism. As shown Figure g our deterministic voting attack matches oracle performance when scores are available, confirming that it successfully extracts all membership information embedded the nearest-neighhor structure.
TRAK: Compressed Sensing for Averaged Signals. TRAK scores are computed as s(z) IT EieT ((x) Gt, averaging gradient contributions projected low-dimensional space from all targets. This averaging diffuses individual membership signals, preventing the deterministic eereee-nngini that succeeds for Image-based methods.
Recovering which targets contributed to averaged scores is a sparse recovery problem: given the observation E RIP) and the measurement matrix + G P|x[7], we seek the sparse indicator vector Tua {0, 13171 such that & GT This is compressed sensing problem that has been extensively studied in signal processing and optimization literature.
We employ Orthogonal Matching Pursuit (OMP) and Iterative Hard Thresholding (IHT) (Axiotis & Sviridenko 2022) greedy algorithms known to provide near-optimal solutions under certain conditions. These represent the state-of-the-art for practical sparse recovery. Theoretically optimal attacks would require solving the fg-minimization problem exactly, which is NP-hard (Natarajan) 1995). Brute-force search over all (ITI) possible target subsets is computationally infeasible for realistic dataset sizes (e.g., 5000
7000 possibilities).
In practice, we find that even these theoretically snorrttcally-groundeed compressed sensing attacks fail to mean- ingfully outperform LiRA TRAK. This empirical result suggests that the averaging operation in TRAK provides inherent protection by spreading signal across many dimensions, making sparse recovery ill-conditioned. The attack success becomes dependent on [T]: smaller target sets (common
18

---

Published as a conference paper at ICLR 2026
in sensitive domains) concentrate signal and become more vulnerable, while larger sets benefit from stronger averaging.
B.2 METHOD-AGNOSTIC ATTACKS
To attack the curation methods, we employ the likelihood-ratio attack (LiRA) from Carlini et al. (2022a). We initialize N 256 shadow models. Each of those uniformly curates for random subset of the target dataset, s.t., each target sample is used in exactly half the shadow models. Furthermore, we perform 25 independent curation runs on random subsets that will attack.
To analyze the attacakbility of a target sample, we group the shadow model scores into those that contained the target sample Sn and those that did not Sout We then average over the shadow models, giving us two vectors of pool scores Sin and Sout As many of those scores will probably not give us meaningful membership signals (Jagielski et al. 2023). we first find the index of the pool score that has the largest average difference, i.e., i = arg max |Sin Soutl- We then fit Gaussian to the distribution of St,i and Sr,ut, giving us two distributions (in(s) and Powl (s). We can then compute the log-ratio of the of the ith pool score to the two distributions, giving the LiRA score.
To perform the attack without access to the scores, we replace the scores with binary signals, indicating whether a particular pool sample would be in the selected partition of the shadow models. The binarized LiRA computes the mean of those binary signals and computes the difference in bernoulli log- probbabbiyy mass functions.
Soft Binarization. While the adversary only observes binary selections for the target, since they set up the shadow curations themselves, they have access to the underlying continuous scores for those. Rather than discarding this information, we apply sigmoid transformation 3 = on (sx) = 1/(1 + exp(-x> (s; Tx) T)) preserve boundary proximity information. Empirically, we show that this improves attack success when samples cluster near the selection threshold but provides no benefit when scores are clearly separated (Figure
B.3 IMAGE-BASED SCORING
We recall that for Image- based scoring, the score for each sample in D only influenced by the nearest neighbour in T. Therefore, we analyse how many pool samples each target sample is the nearest neighbour for.
Given pool sample E DD with target score (Ix), we can determine which target sample E T was responsible for this score through the following attack:
Normalize the embeddings: & d(x) [o(=) and i
<(t) |o(00]2 for all
2. Compute similarities: sim(æ, = a for all / E T
3. Find target with matching score: t = arg mingeT [sim(z, t) s(x)
4. For each target t, assign votes:
= if sim (z, > s(x)
vtt)
{
-1
otherwise
The voting scheme reveals membership T through positive votes: a target sample receives positive votes, it was used to compute the score of at least one pool sample. Negative votes indicate that this target sample could have given a higher score than what was observed, implying it was not used in the curation.
Selection-Basee Based Attack. When we only have access to the selected pool samples D C D but not their scores, we can still reconstruct the target dataset through an iterative elimination process:
1. Initialize To = as the full target dataset, R = @ as the set of removed samples, and voting scores
19

---

Published as a conference paper at ICLR 2026
Algorithm 3 Image-based Scores MIA
Input:
Dataset embeddings fo(x)) target embeddings (Ø(t))ET, target scores Output: Votes v(() for each target / E
Initialize VieT : =(() 0
Normalize all target embeddings:
o(t) for all E T [[+())l2
for each sample æ E D do
Normalize the query embedding: x [lø(z)]2
Compute similarities; sim(2, () x t for all & ET
Find target with matching score: 1" arg mit [sim(x, () - s(x) for each target t T do
if then
v(t) v(t) +
else if sim xr. t) > 8(z) then
utt) (t) -
end if
end for
end for
2. At each iteration i:
(a) Identify target-missing and target-exclusive samples:
Target-missing: M, DD Diagee (selected by us but not target)
Target-exclusive: & Piargtt Dr (selected by target but not us)
(b) Find nearest neighbors in remaining targets:
For target-missing: N₁-- NN(M,, 7:)
For target-exclusive: N₁+ = NN(E, T:)
(c) Update votes:
v[V] + vWW, (negative for likely non-members)
v[N+]] + v(N,'] + (positive for likely members)
(d) Update removed set: R = {1 [ T: v[t] < 0}
(e) Update remaining targets: T++1 T R
(f) Recompute pool selection: Di+1 = TopK (NN-Sim(2) (D,T,+)))))) T.))),k)
(g) Compute Jaccard similarity: Jj 12 Deso
11:
3. Stop when either:
Perfect reconstruction: J4
No improvement: J, Ji-p <. e for patience p and threshold
The attack maintains voting score for each target sample. Negative votes suggest the target sample is causing overselection of pool samples that are no in Tsel. while positive votes indicate the target sample helps select pool samples that are also in Tsa The ROC-AUC these votes against true membership provides measure of attack success.
B.4 TRAK
TRAK computes the scores for all pool samples E RlD as
1 Σ TTG, 1ET
(13)
Following the DsDm methodology (Engstrom et al. 2024), we first train CLIP model on the full pool dataset D to obtain reference model. We then obtain the gradients G, and & from this model by computing the sampl zero-shot gradients the gradients obtained by minimizing the classification loss of the linear zero-shot classifier obtained by embedding caption-templates (Cherti)
20

---

Published as a conference paper at ICLR 2026
The curation score for each pool sample is the average of its TRAK attribution scores across all target samples t E T, capturing how influential that pool sample would be for improving performance the target dataset.
& Beaumont 2025)
Since the target model does not use the entire target dataset, but instead an unknown subset Twe C T. the target scores are computed
T
T G.
(14)
T
TE
where GT Rdx[]Tal is the matrix of the target gradients for the selected target samples.
Recovering Tal from Sy constitutes a subset-sum problem, which is NP-hard (Natarajan 1995). We know about Tael that it is subset T.
B.4.1 ORTHOGONAL MATCHING PURSUIT
Hence, we can formulate this as a standard compressed sensing problem with the formulation
Taa 4 GT
(15)
where Tel E R is TTall sparse vector that performs the equivalance of masked mean compu- tation. We then recover the standard compressed sensing problem
min sealll T
s "G STa
(16)
where se lo is the lo- fo-mnorm" of set, i.e. the number of non-zero elements in sel We approach this problem using the greedy orthogonal matching pursuit (OMP) algorithm and adaptively regularized iterative hard thresholding (IHT) (Axiotis & Sviridenko 2022
B.4.2 LEAST SQUARES FORMULATION
We solve for the membership indicator via least squares. Let X R denote the pool features (④b
and Y E Raxd the target gradients (G The TRAK scores satisfy
Σ XY =Y- k
(17)
where m E {0,1/K 1/k}" " is the mean operator encoding membership, with k = (Teil non-zero entries indicating which targets are in Twl. To avoid materializing XYT E R multiply both sides from the left by X
s XY m.
(18)
Since X Raxd and (X"X)YT R tractable, avoid materializing the (N, n) matrix. We solve the least squares problem
minimize m
X'XY m x
(19)
for m. The membership scores then given by Bistsg m,, where m is the optimal solution.
B.5 COMBINING VOTING AND LIRA ATTACKS
We analyze whether the voting- based attack for Image-based scoring and the LiRA attack provide complementary signals that can be combined for improved membership inference. The attacks exploit different aspects of the curation mechanism: voting exploits the deterministic nearest-neighhor structure, while LiRA models score distributions from shadow curation runs. Correlation analysis reveals that LiRA and voting scores have low Pearson correlation (p R 0.3).
We combine the two attack scores via weighted averaging:
Scombined (1) (1.RA(() (1 w) &yoting (t)
(20)
21

---

Published as a conference paper at ICLR 2026
1.0
Oracle
1.00
1.00
0.27
@ 1% FPR
0.8
Pearson
Voting Scheme-
1.00
1.00
0:27
10-1
0.6
TPR
0.5
LIRA
0.27
0.27
1.00
10-2
Combinstions Vettng-Schema
-1.0
Orarie
Voting Voting-Schemi Scheme
LIRA
0.0
10-3
10
Weight LiRL, 1=Voting- Scheme)
FPR
(a) Score correlation analysis
(b) Combined ROC
(c) Combined TPR at 1% FPR
Figure 9: Combining voting and LiRA does not improve attack performance. (a) Scatter plot
showing correlation between LiRA and voting scores reveals low correlation (p 0.27). (b) ROC curves for the combined attack and (c) TPR 1% FPR for different weightings the two attacks. We show that the combined attack does not improve over voting alone.
where SI.RA (1) and Svoting (1) are normalized to [0, IJ, and E [0, controls the weighting. We evaluate w E (0, 0.25, 0.5, 0.75, 1.0) identify the optimal combination.
Figure shows ROC curves for the individual and combined attacks. The combined method does not improve over voting alone. Since LiRA substantially underperforms voting on Image-based scoring (AUC x 0.6 vs. x 0.95) and the voting-based attack already matches the oracle attack performance (Figure LiRA contributes mostly noise when combined.
C ATTACKING MODELS TRAINED ON CURATED DATA
C.1 MODEL. TRAINING DETAILS
All end-to-end attack experiments train CLIP models following the DataComp small-scale bench- mark (Gadre et al. 2023). We provide the complete training configuration below: We use the Vision Transformer VIT-B 32 architecture (Dosovitskiy et al. (202]) for all experiments. Models are trained with the AdamW optimizer (Loshchilov & Hutter 2019) with the following hyperparameters:
Learning rate: 5 10 with cosine decay schedule
Linear warmup: 500 steps
Weight decay: 0.2
Beta coefficients: 31 = 0.9, Bg 0.98
Gradient clipping: maximum norm 1.0
Batch size: 1024
Precision: Automatic mixed precision (AMP)
Training Budget. All models are trained for fixed budget of 10M samples. The number of epochs adjusted based on the curated pool size, e.g. pool of 100k samples results in 100 epochs, while pool of IM samples results in 10 epochs. This ensures all models receive equivalent amounts of training regardless curation pool size.
Training Data. Models trained exclusively the curated subset D selected from CommonPool only benefit from curation
(small). They are never trained on the private target data T performed using T.
22

---

Published as a conference paper at ICLR 2026
10-2
10-2
Model Prediction
Model Prediction
10-7
10
10 12
10
Baseline With Fingerprints
10-17
10
10
10
10c
5
10
Top-k (log scale)
Epoch
(a) Effect of training size
(b) Signal evolution during training
Figure 10: Mislabeled images leave a measurable fingerprint in the trained model. We fix the number of fingerprints at 5 and measure probability on the mislabeled concept. (a) The signal remains detectable even as the training set grows to 1,000,000 samples (0.0005% fingerprint rate), though variance increases. Non-fingerprinted models consistently produce near-zero signal, enabling reliable fingerprint detection. (b) The fingerprint signal strengthens with additional training epochs.
C.2 IMAGE-BASED
Image-based curation scores each pool sample DD based its maximum similarity to any target sample:
s(x) max cos(ó(x), o(t)
(21)
rET
where (--) is the CLIP embedding function. This creates one-to-many correspondence mapping between targets and pool samples.
The key insight: When a target t is included in the victim's 's subset Tvictim all pool samples for which is the nearest neighbor will receive higher scores and may cross the selection threshold.
We note that by default most target samples do not affect any target samples, as Figure we assume the adversary can inject samples with embeddings { TM} Rd into the pool to probe membership.
2 shows. But,
Algorithm 4 Image-based Correspondence Mapping
Input: Candidate embeddings &c, target pool embeddings ọ7, mixing parameter Phase 1: Find correspondences
For each target t E
Find i candidate nearest neighbors: NN&(t) = fz11............................ xb
Compute attraction score: O; = cos( (d()), o(x;)) )) (similarity to target)
Compute repulsion score: ri maxe yt cos(<(())) o(x;)) (similarity to other targets)
Combined score: 8; (1 a) ag + a r3
Select best match: x=' arg max 8t
Store mapping: MM()) x=
Phase 2: Identify correspondence uniqueness
For each selected candidate {x* xx MM()) for some l]:
Count targets mapping to it: (M-+ = (M:MM()) = x)1
Output: Correspondence mapping M and uniqueness counts |M (x))
For each candidate sample compute its baseline percentile against the full target pool:
Po (æ) = percentile
max cos()))) tET
(22)
This represents the expected ranking when the target randomly selects their subset Tsel.-
23

---

Published as a conference paper at ICLR 2026
Algorithm 5 Image-based End-to-End MIA
Input: Fingerprinted samples F, selection observations {selected, x F), baseline percentiles selection selection rate
Compute selection threshold: (1 1 p) 100 (e.g., T50)
For each target E :
Initialize: surprise, 0, count,
For each fingerprinted sample a E F where t (x):
Compute expected selection probability:
(z) L+exp(-(08-rr//1 (sigmoid model)
Observe: selected, (whether fingerprinted sample æ was selected)
Compute surprise:
If selected, true: ò 1 Pexp (ar (high surprise if unlikely selection)
If selected, false: 5 = Pexp (x) (negative surprise if expected selection missed)
Weight by uniqueness: w = M/I-M (x)
surprise, - surprise, + w Update: count, t count, +1
Update:
Average: MIA, = surprise,/ max(count,, 1)
Output: MIA scores [MIA,)eT
The adversary then marks samples for monitoring based on information potential: For sample æ with targets M M=)(
InfoScore(a x) CrossingScore(r)
UniquenessScore(z)
(23)
where:
po < if po 2
CrossingScore(()) =
1+el pole 1075
(24)
0.1
UniquenessScone(())
(25)
[M-1
The adversary marks the top-scoring samples that are most likely to reveal membership through threshold crossing. Having stronger a-priori bounds the percentile the target will use for curation, will improve this attack. In practice, we assume 7 to be 50%. meaning the target will probably not pick from the worse half of public data.
Using this information, the adversary selects the highest-scoring candidates to form the fingerprinted set F &c. These samples then inserted into the pool D with altered captions enable detection.
Algorithm shows the full algorithm.
C.3 TRAK
The TRAK system computes influence scores = | ΣLET Ø' where + are the projected pool features and G, target gradients. However, when the target secret subset Tsa.. S T. the actual scores become:
STa.
""G.
(26)
TTet.
DETx.
We assume the adversary can inject samples with gradients C C= (cI...., cM3 C Rd into the pool to probe membership. Obviously, gradients cannot be chosen arbitrarily. Instead, we obtain these as gradients of target samples with modified captions.
When contrastive gradient e; appended the pool matrix X, the Sherman gives its score as:
Sherman-Morrison Morrison formula
G.
Znew
quew
1 c. G-le;
(27)
24

---

Published as a conference paper at ICLR 2026
where G (X + A is the regularized inverse Gram matrix and y Eees Us is the secret average.
The key insight is that including target yj in the secret subset creates change in scores of the samples:
c cT
()())
Inew m
(28)
This membership signal is proportional to the preconditioned inner product c. G- and inversely proportional the subset size m, making smaller subsets more vulnerable.
Assuming target gradients have mean μ and covariance E, the signal-to-noise ratio for detecting target V/ using contrastive o;
lc G- Uy
(CR)(C)) Vm O G-1
(29)
For each target yy, the adversary selects the contrastive maximizing SNR:
1c GTGGTy 33l
8" (i) = arg max dE[M]
(30)
Vc, G EG-
The attackability score A₁ max; (c) quantifies how detectable each target's membership is.
he attack succeeds when the contrastive gradient crosses the curation selection threshold. Let 7ie be the score of the k-th ranked pool sample. The attack successful if:
Under Hg (target not in subset): 2Ho < Tk (not selected)
Under Hi (target in subset): 2H, 2 Th (selected)
This binary change in selection status provides clear membership signal. The attack is most effective when the contrastive score under Hg is just below Tke requiring only small membership signal to cross the threshold.
To maximize the sensitivity to our target samples we experiment with adding copies of the target data to the pool. These should naturally have a high utility. To imprint a measurable membership signal in the model, need add some additional information, though. Mislabeling the samples as Carlini & Terzis (022) did would serve that purpose, but those samples would not get picked by the TRAK algorithm, it specifically rejects mislabeled samples Park al. (2023) have shown. So instead, we add harmless and orthogonal information, i.e., we retain the original and useful caption and add a suffix, which mostly preserves the TRAK signal (see Section C.5). Figure] [1 shows the measurable difference in models trained on such samples with additional text appended to the caption.
C.4 COMPUTATIONAL AND QUERY COST
We analyze the computational and query costs of our proposed attacks. Unlike traditional membership inference attacks that require training on the order of hundreds of shadow models (Shokri et al. [2017] Carlini al. 2022a), our attacks require few curation runs, are training-free, and scale with the curation algorithms themselves.
Computational Complexity. Let N = [D] denote the pool size, n = |T| the target dataset size. Our attacks require running only the respective curation algorithms or operations of similar complexity, therefore scaling linearly with the number of pool samples, as do the curation methods. For Image-based scoring, curation costs O(N)) for computing cosine similarities. For TRAK. this requires training M model checkpoints and computing gradients for all N pool samples, with complexity O(M N- F) where F is the number of operations forward pass.
Score-based attacks require one curation run.
Subset-hased attacks require approximately 10 iterative curation until convergence.
25

---

Published as a conference paper at ICLR 2026
10a
10a
Model Predictioo ction
10.5
Model Prediction
10-12
10-12
Baseline With Fineeprrints
10
10-19
10
10
10: 30
10a
10c
20
Top-k (Log Scale)
Epoch
(a) Effect of training size
(b) Signal evolution during training
Figure 11: Samples with harmless orthogonal information imprint a measurable signal in the model. In this case, we insert samples from CIFARIO with an appropriate caption from CLIP- Benchmark with an additional 'nnd ratatouille' in the end. We then measure the model zeroshot probabaility for the concept 'ratatouille'. This effect has more variance than Figure [hossws for mislabeled images, but still distinguishes models trained with and without the fingerprints.
End-to-end attacks require one curation run to identify fingerprints.
This makes our attacks practical at any scale where curation is practical in the first place.
Query Budget. For end-to-end black-box attacks using fingerprinting (Section we analyze the number of queries required the target model.
To attack single target sample, we need at most black-box queries, one for each fingerprint sample identified during the curation phase. In practice, some fingerprints contribute to multiple targets when they share nearest-neighhor relationships. This causes the query budget to scale at most linearly with the number of targets and on expectation sublinearly with respect [T].
C.5 CAPTION PROPERTIES IN CURATION POISONING
To understand which caption should be used for TRAK fingerprints, we analyze the effect of altering the captions on the curation score.
Experimental Design. We systematically ablate over three key dimensions of caption design: (I) concept type: correct class label baseline (ID 0), wrong in-distribution CIFAR- classes (IDs 1-4), and out-of-distribution concepts (IDs 5-19); and (2) caption templates: simple, article, imageof, photoof, and combined variants (IDs 8-13). Table shoows the complete set of 20 caption configurations tested. We also compare different OOD concepts (ratatouille, shader, medical scan, abstract art) and combined templates that pair an OOD concept with the correct class label from the image to understand detectability and curation signal strength trade-offs.
For each caption configuration in Table 3 we analyze gradient correlations with the baseline configu- ration (ID 0) to understand how caption properties impact TRAK curation behavior.
OOD concepts (IDs 5-7) with explicit templates like photoof and imageof (IDs 10-11) alter TRAK gradients too drastically, resulting in low curation signals that undermine attack effectiveness. find that combines templates (IDs 12-19) offer the most practical trade-off: by pairing an OOD concept with the correct class label, their gradients correlate highly with the baseline (ID 0), preserving the TRAK curation behavior while planting detectable information in the model. This makes combined captions the optimal choice for practical curation poisoning attacks.
C.6 MEASURING WORST-CASE PRIVACY LEAKAGE
While the above attacks already effective in showing privacy leakage from curation pipelines in natural setups, we further want to approximate the worst-case privacy leakage that can occur. There- fore, we rely on inserting canaries into the private data T. Canaries sample specifically crafted
26

---

Published as a conference paper at ICLR 2026
Table 3: Caption templates used in ablation study. We test variations across ID (in-distribution) and OOD (out-of-distribution) concepts using different template styles. ID O uses the correct class label (baseline, denoted as class). IDs 1-4 use wrong ID classes (e.g., labeling dog image as "airplane"). IDs 5-11 use pure OOD concepts. Combined templates (IDs 12-19) pair an OOD concept with the correct class label (e.g. "dog and ratatouille" a dog image).
ID
Concept
Type
Template
Caption
0
class
id
photoof
photo of class
1
airplane
photoof
photo of airplane
2
automobile
photoof
photo of automobile
3
bird
photoof
photo of bird
4
cat id photoof
photo of cat
ratatouille ood shader ood 6
5
photoof
photo of ratatouille
photoof
photo of shader
7
medical scan ood photoof
photo of medical
8
ratatouille ood ratatouille
simple
ratatouille
9
article
ratatouille
10
ratatouille
imageof
an image of ratatouille photo of ratatouille
11
ratatouille
ood
photoof
12
ratatouille
+ ood
mmbbined_imagens combined_
an image of class ratatouille photo of class ass and ratatouille an image of c a.s and shader
13
ratatouille
+ ood
14
shader
id + ood
15
shader id ood
combined photoof
photo of class and shader medical scan photo of class and medical sean an image of class and abstract art photo of class and abstract art
16
medical scan
+ ood
combined imageof
an image of class
17
medical scan
+ ood
combined phntooe
18
abstract art
id + ood
combined_imageof
19
abstract art
ood
combined__hotoorf
1.0
Correlation to Baseline
0.5
0.0
-0.5
ID Wrong
OOD Wrong
Templates
Combined
-1.0
2
3
5
8
10
11
12
13
14
15
16
17
18
19
Variant
ID
Figure 12: Gradient correlation with baseline configuration. Combined templates (IDs 12-19) that pair an OOD concept with correct class label maintain the highest gradient correlation with the baseline (ID 0), preserving natural TRAK curation behavior. Other caption variants, while potentially more detectable, alter gradients too drastically and reduce attack signal strength.
to be attackable and have been used succesfully to assess empirical privacy leakage (Mahloujifar et al. 2024) and verifiy formal privacy guarantees (Annamalai et al. 2024).
For Image-basse based Curation To craft Image-based curation canary and probe p, any two similar and very low-scoring (e.g., Ti) samples from D. The low score ensures that they are far away from any sample in T. our probe p will not be selected during curation by default. With high similarity between and c, we can also reliably ensure that e inserted into T, p will score high.
first select
For TRAK-based Curation, we recount that TRAK computes the score of a training sample æ X
for the average f of T grads(T) (XT)) X - nq.
27

---

Published as a conference paper at ICLR 2026
( = p00:==p"GGG1
Similarly to the attack Section 3.5 we derive (in more detail in Section C.6.1) the Signal-to-Noise- Ratio (SNR) between target canary and pool sample p
(pp, c) (p)
1p c
SNR(p, e)
GD
We insert those canary pairs that yield the highest SNR.
C.6.1 CRAFTING TRAK CANARIES
TRAK computes the score of a training sample E X for the average of T grads(T) as =(X"X)- pq. we append a probe p Rd X, the score that new row (using the Sher- man-Morrison formula) is
p G
nnew
1 G-
Unew
If we formulate hypothesis test whether canary gradient Rd is part of the target gradients (H,) or is not part (Ha), we can define the following statistics
y a
(Ho), or y
1
c
(11))
m
with
E|s] O, Cov(s) R E,
For a fixed probe we can define the baseline influence b, the denominator d and the canary influence
c as
b(p) G ,e,
d(p) = 1 G p.
s(p,c) = ccc
Then under Ho
b(p)
so(p) Qnew
d(p)
Under H1 adding leads change in score for p of
s(p,c) c)
A(p, Inew
m d(p)
The standard deviation of the probe in either case is
a pop==-------
C.7 CANARY ANALYSIS
For Image-based curation, we manage find canaries that will reliably leak their presence in the shaders21k dataset. For TRAK, our canaries achieve similar success rates as the original attack on the final trained model. We hypothesize that TRAK's resistance canary attacks stems from the limited adversary control. Despite optimizing for maximum signal-to-noise ratio, canaries provide only marginal improvements under realistic threat scenario. Since the canary gradients are obtain contrastively with thousands of other samples the adversary cannot control, then randomly projected into lower-dimensional space, and afterwards averaged with all other gradients, it remains challenging craft a strong signal.
D MITIGATING PRIVACY LEAKAGE
We investigate how mitigate the privacy risks outlined in our work. Section D.I loutlines how employ DP to protect the target dataset. Section D.2 shows the effect of removing the most vulnerable samples.
28

---

Published as a conference paper at ICLR 2026
D.1 DIFFERENTIAL PRIVACY
We provide detailed methodology and experimental results for the differentially private (DP) adaptions of curation methods introduced in Section 5
D.1.1 DP BAGE--ASSD CURATION
In standard Image-based curation each pool sample's score is determined by its nearest neighbor in the target set. This deterministic nearest-neighhor structure creates direct correspondence between scores and individual target samples, making the method highly vulnerable to MIAs.
DP Noisy Max. The most natural way to privatize the nearest-neighbor scoring is the Report Noisy Max mechanism (Dwork et al. 2014): for each pool sample, compute the cosine similarity to every target embedding, add calibrated Gaussian noise to each similarity, and then take the maximum:
SNM (z) max (cos(((x), ((t)) + nd,
Th ~ ~_(00,
(31)
SET
Since cosine similarity bounded in [-1, 1], adding or removing single target sample changes the set of queries by one entry with sensitivity A = 2. The noise is calibrated as:
lo.(5/)))
(32)
This preserves the nearest-neighhor semantics-ean each pool sample is still scored by its best match in the target set-while providing (s, DDDP.
DP Mean. A stronger alternative replaces the nearest-neighhor operation entirely with a distance to the DP mean of all target embeddings:
spp(x) cas(((2)) 0r where का 17| Eier (t)) is the mean target embedding, and 17 (N(0, o²I) Gaussian noise calibrated via the Gaussian mechanism (Dwork et al. 2006b).
pr
(33)
The fz-sensitivity of the mean is:
2 [T]
D = [T|
max (1(()) - o(t')12 S 6,
(34)
for normalized embeddings [|o(t)]l2 = 1. To achieve (E, DD- we set:
A llg(5/)))
2 T
(1.25)))/))
(35)
Comparison. Table 5 compares both approaches. The DP mean provides stronger privacy at equivalent e because averaging already obscures individual target contributions, while the noisy max retains the per-target nearest-neighhor structure. However, the noisy max may be preferable when the neirest-neighhor semantics are important for downstream curation quality.
D.1.2 DP TRAK CURATION
TRAK-based based curation computes scores via s(x) [b) where g | Eier G. is the mean
gradient over target samples. While averaging provides some natural privacy protection, small target sets ([T) < 5000) remain highly vulnerable to membership inference attacks.
We privatize the mean gradient computation:
JDP
Σ
G ~~ N(0,001
(36)
T
where GÜ are the clipped gradients
- G. min (1 ( (LIGGHT
C
(37)
with clipping threshold C' and sensitivity A₂ 2C/[T]. The noise scale is
[ / log( 1.25/8).
2C
29

---

Published as a conference paper at ICLR 2026
Table 4: Attack success on curation scores for DP Image-based and TRAK curation.
Privacy Guarantee
TPR @ 1% FPR
(e, = le 5)
Image-Basee Based Curation
TRAK
non-private
0,9842 # 0.0013
1.0000 0:0000
1000
0.0160 = 0.0000
1.0000 0,0000
100
0.0155 = 0.0000
0.3324
10
0.0134 # 0.0000
0,0165 = 0.0072
0.0107 # 0.0000
0,0136 0.0071
2
0,0105 = 0.0000
0,0125 0.0065
Table 5: Impact of Nearest-Neighbor vs. Mean on the Image-Based Privacy Leakage.
Privacy Guarantee
TPR @ 1% FPR
(c, 8 = 10-
Image-Based Curation (Mean)
Image- Based Curation (Nearest Neighbor)
non-private (inf)
0.0160 + 0.0067
0.9842 = 0.0013
1000
0.0160 = 0.0069
0.8591 = 0.0158
100
0.0155 #
0.0542 = 0.0090
10
0.0134 # 0.0069
0.0110 = 0.0050
5
0,0107 = 0.0065
0.0109 = 0.0057
2
0.0105 = 0.0045
0,0106 = 0.0057
D.1.3 EVALUATION
We test (0.1, 0.5, 1.0, 2.0, 5.0, 10.0) with d 10-5, We attack the curation scores with LiRA, fitted on shadow curation scores with no DP.
Table 4 shows the attack success on the curation scores for DP Image-based and TRAK curation. The results show that these measures are effective at mitigating privacy leakage. For Image-Based Curation, just replacing the nearest neighbor operation with mean drastically reduces the attack success. For TRAK-bain based curation we saw higher attack success, despite using mean, because the gradient projection dimension (32,768) is significantly higher than the embedding dimension for Image-based curation (768). This aligns with our ablation results for TRAK, where the attack success drops for less than 2, 2,048 dimensions. This explains why even the non-private mean is hard to attack, and subsequently why DP guarantee of 1000 shows low attack success. Table 5 analyzes this further, comparing the mean and nearest neighbor-based curation for various privacy guarantees. The results show that the mean is harder attack than the nearest neighbor-based curation for various privacy guarantees.
D.2 REMOVING THE MOST VULNERABLE SAMPLES
We detail our approach and results of removing the most vulnerable samples from the target dataset.
We conduct our experiments on the curation scores for CIFAR-10. For the TRAK-based analysis, we use a target dataset size of 25,000 samples and employ the least squares attack. The attack is evaluated over 16 seeds. For the Image-based analysis, we a target dataset size of 5,000 samples and employ the LiRA attack. The attack is evaluated over 16 seeds with 256 shadow models. In both experiments, we define the "vulnerable" set as the top 5% samples with the highest membership inference attack success rate. For comparison, we also remove 5% of samples at random. We report AUC only on the remaining samples.
We then evaluate four scenarios:
Baseline: Full target dataset.
Ideal: Post-hoc removal of vulnerable samples from ROC computation.
Vulnerable Removal: Re-run experiment after removing vulnerable samples from target dataset.
Random Removal: Re-run experiment after removing random samples from target dataset.
30

---

Published as a conference paper at ICLR 2026
Findings. We investigate whether removing the most vulnerable samples from the target dataset could serve as an effective defense against membership inference. The results show that for both methods, removing samples increases overall attack success.
For Image-based curation, Figure 13a shows that the attack success increases significantly when removing the most vulnerable samples. Removing samples random increases the attack success only marginally. The 5% most vulnerable target samples are the score-determining nearest neighbors for 31.1% of the pool. Their removal exposes 1.8% of previously shielded targets and increases vulnerability for over 80% of target samples, which are now on average the nearest neighbor for 83 additional pool samples. Therefore, removing the most vulnerable samples actually exposes many previously protected samples. This is a strong case of the Privacy Onion effecr (Carlini et al. 2022b).
For TRAK-based curation, Figure 13b shows that removing vulnerable samples increases overall attack success similarly to removing random samples. Removing the most vulnerable samples (AUC 0.9400) reduces attack success marginally more than removing random samples (AUC 0.9439). But our ablations have shown that the attack success is very sensitive to target dataset size (Fig. 34-35 in Appendix E.5), so the reduction in target dataset size outweighs the benefit removing any samples.
Our findings suggest that simple sample removal is not robust defense strategy. For image-based curation, onion effect negates the benefit removing outliers. For TRAK-based curation, while no onion effect is present, the attack remains effective, indicating that privacy Icakage is not confined to small subset of "vulnerable" points but is a systemic property of the curation process.
E EXTENDED EVALUATION
E. IMAGE-BASED SCORING ATTACKS
We show the results of the attack Image-based scoring in Figure [4 LiRA exhibits strong and consistent performance across all datasets. The custom attack is better for all datasets at FPR > 10-4 For lower FPRs, performance is highly variable. Figure 13 shows the results of the attack Image- based scoring without access to the scores. The custom attack and LiRA achieve similar results, with the custom attack being slightly more successful in the low FPR region.
31

---

Published as a conference paper at ICLR 2026
10o
100
True Positive Rate
Positive Rate
10-1
10-1
Baseline
(AUC=0.890)
Random removal (AUC-0.301
(AUC=0. 896)
Ideal (AUC=
Vulnerable removal (AUC=0.950)
Actual (AUC-0. 956)
Random guess
Random guess
10-2
10-2
10 -2
10-1
10a
10-2
10-1
10
False Positive Rate
False Positive Rate
Actual expect effect
Vulnerable random sample removal
Image-based
100
100
True Positive Rate
True Positive
Baseline
(AUC=0.90
Random reroova (AUC=0.35) 0.944)
(AUC=0.900) 0,905 Ideal (AUC-0.900)
Vuln erable removal (AUC=0. 0.940)
Actual (AUC=0.940)
Random guess
Random guess
10-2
10-2
10-2
10-1
10a
10-2
10-1 False Positive Rate
10
False Positive Rate
Actual expect effect
Vulnerable random sample removal
(b) TRAK
Figure 13: Removing the most sensitive samples does not prevent Icakage. We show attack success increases when removing the most vulnerable samples. a) Image-based curation shows strong privacy onion effect. Removing the most vulnerable samples increases attack success significantly, while removing random samples has negligible effect. b) TRAK-based curation shows no privacy onion effect, but since attack success is highly sensitive to target dataset size, removing samples increases it.
32

---

Published as a conference paper at ICLR 2026
109
10
10
10-1
10-4
10-4
Positive
True Positive
True Pocitive
10-3
10-3
10-3
4
10-4
10-4
10
10-4
10-4
10"
10-4
10-2
False Positive Rate
False Positive Rate
False Positive Rate
(a) CIFAR-10
(b) CIFAR-100
(c) STL-10
LIRA
Custom
Random
10-1
10-1
10-1
Positive
Positive
Positive
10-2
10-2
10-2
10-1
10-3
10
10-4
10-4
10-4
10-²
10a
10
10-4
10a
10-4
10-2
10"
False Positive Rate
False Positive Rate
False Positive Rate
RESISC45
(c) PCam
Food101
Figure 14: Image-based Attack success with access to the scores for various datasets and curation
methods given access to the full set of pool scores. We find that while none of the methods satify DP only Image-base scoring is attackable.
107
10P
107
Footthee Dave
2:0 Pobbikve Ra72
30-1
10
10
10
(a) CIFAR-10
(b) CIFAR-100
(c) STL-10
10"
30F
109
Fraitie Rite
15
10
15
Poa
Positive
(d) RESISC45
(e) PCam
(f) Food101
Binary LIRA
Sigmold Binary URA
Custom
Random
Figure 15: Image-based Attack success without access to the scores for various datasets and curation methods given access to the full set pool scores. We show that just the selection of the pool samples reveals membership information about the targets.
33

---

Published as a conference paper at ICLR 2026
E.2 TRAK SCORING ATTACKS
Figures 16 and 17 show that TRAK is hardly attackable, both with access to the scores and without.
34

---

Published as a conference paper at ICLR 2026
109
10a
10c
10-1
10-4
10-4
Postéive
Tue Positive
The Positive
10-4
10-3
10-3
10-3
10
10-4
10
10-4
10"²
10c
10-4
10
10c
10"2
False Positive Rate
False Posktive Rate
False Positive Rate
(a) CIFAR-IE 10
(b) CIFAR-100
(c) STL-10
LIRA
Least Squares
10a
10
10
Random
10-4
10-1
10-4
Positive
Thu Positive
10-4
10-4
10-4
10-3
10
10
10-4
10
10-4
10-2
10
10
10-2
10a
10-*
10-2
10"
False Positive Rate
False Posiivve Rate
False Positive Rate
RESISC45
(c) PCam
Food101
Figure 16: TRAK Attack success with access to the scores.
Rate
The Roothee Rate
Tru Postvv Rate
40-1
10-2
10-2
40-2
10
20
10-1
10-4
10-4
10-4
10
10
10a
10
10"
100
10
10
10"
Padte Posilive ve Rate
Taloe Poslltve Rat
Palce Poohive Rate
(a) CIFAR-10
(b) CIFAR-100
(c) STL-10
10-1
10-2
1.0m
lvee
Trus Poe
20-4
2012
10-2
10a2
10T4
101
10-h
10-4
10t4
10-2
10P
10-1
10
107
10:4
10th
16
Tahe Posllive Rate
Taloe Pocitivw Pate
Dilne Rosttive Rate
(d) RESISC45
(e) PCam
(I) Food101
Binary LIRA
Sigmoid Binary LIRA
Random
Figure 17: TRAK Attack success without access to the scores.
E.3 END-TO-END ATTACKS
Figure 18 shows that models trained on Image-based curation leak membership information over all target dataset sizes. Figure 19 shows that for small target datasets, even models curated with TRAK are highly attackable, more so than Image-based based curation. This success diminishes as the target dataset size is increased though.
35

---

Published as a conference paper at ICLR 2026
10
10a
10
TPR @ 1% FPR
10
10-1
10"
TPR @
TPR
10-1
10
10
10a
10a
10a
10a
107
10a
Private Target Size
Target
Private Target Size
(a) CIFAR-10
(b) CIFAR-100
(c) STL-10
Attack
Random
10a
10a
10a
TPR @ 1% FPR
TPR FPR
1%
10
10
10-2
10-2
10-2
10
10a
10a
10a
10a
10a
10²
10a
10a
Private Target Size
Private Target Size
Private
Target Size
(d) RESISC45
(c) PCam
(f) Food101
Figure 18: Image-based attack success with access only to the models trained curated data.
109
109
109
TPR 19 FPR
1% FPR
FPR
10-1
10-1
10-1
TPR @
10-2
10-2
10-²
10a
10
10a
10a
10
10a
10a
Private Target Size
Private Target Size
Private Target Size
(a) CIFAR-10
(b) CIFAR-100
(c) STL-10
Attack
Random
10"
10"
TRR EPR
TPR @ 1% FPR
1PP 1% FPR
10-1
10-2
10-2
10-2
10a
10a
10a
Private Target Size
Private Target Size
Private Target Size
(d) RESISC45
(c) PCam
(f) Food101
Figure 19: TRAK attack success with access only to the models trained curated data.
E.4 DETAILED ROC CURVES
Figures 20 to show full ROC curves for configurations of datasets and target dataset sizes for the end-to-ene end attack, showing the full TPR-FPR tradeoff. We furthermore add AUC plots, to enhance comparison with other work that has reported these numbers. These results show that, when the attack is successful, succeeds across wide range of the trade-off curve.
36

---

Published as a conference paper at ICLR 2026
10a
10
Thue Positive Rate
Thus Positive Rate
Thue Positive Rate
10-1
10-2
10-1
10-1
10"2
10
10
10
10-1
10
10-2
10-2
109
10
10-2
10a
10a
10a
10*
False Positive Rate
False Positive Rate
False Positive Rate
Private Target Size
(a) Size 10
(b) Size 50
(c) Size 100
(d) AUC vs. Size
Figure 20: TRAK end-to-end attack on CIFAR-10: ROC curves for different target dataset sizes and AUC vs. size summary.
10a
10
Tue Positive Rate
Tue Positive Rate
Tre Positive Rate
10:
10:2
10:2
1
10
1
10-1
10-1
10-1
10-1
10-1
10
10
10
10
100
10
10
10a
10a
10a
10*
False Positive Rate
False Positive Rate
False Positive Rate
Private Target Size
(a) Size 10
(b) Size 50
(c) Size 100
(d) AUC vs. Size
Figure 21: TRAK end-to-end attack on CIFAR-100: ROC curves for different target dataset sizes and AUC vs. size summary.
10a
10a
10a
Thue Postiive Rate
True Positive Rate
True Posttive Rate
10-2
10-1
10-2
10-1
0.7
0.6
10-1
10-2
10-a
10-1
10
10-2
10-2
10a
10
10-2
10:
10c
False Positive Rate
False Positive Rate
False Positive Rate
Private Target Size
(a) Size 10
(b) Size 50
(c) Size 100
(d) AUC vs. Size
Figure 22: TRAK end-to-end attack Food101: ROC curves for different target dataset sizes and AUC vs. size summary.
10a
10F
10a
Tiue Postiive Rate
True Positive Rate
True Posttive Rate
10-1
10-1
10-1
10-2
10-1
0.6
10-2
10-2
10-1
4
10a
10a
False Positive Rate
False Positive Rate
False Positive Rate
Private Target Size
(a) Size 10
(b) Size 50
(c) Size 100
(d) AUC vs. Size
Figure 23: TRAK end-to-end attack PCam: ROC curves for different target dataset sizes and AUC vs. size summary.
37

---

Published as a conference paper at ICLR 2026
10a
10"
10"
0.9
True Postive Rate
True Postive Rate
10":
10-3
True Posiiive
0.7
0.6
10-1
10
10-1
0.5
10-1
10-1
10-1
0.4
10
10-²
10"
10
10-a
109
10-4
10-2
10
101
10'
10
10
False Positive Rate
Faise Positive Rate
False Positive Rate
Private Target Size
Size 10
(b) Size
(c) Size 100
AUC vs. Size
Figure 24: TRAK end-to-end attack on RESISC45: ROC curves for different target dataset sizes and AUC vs. size summary.
10a
10a
10
True Posttive Rate
True Positive Rate
True Posttive Rate
10-3
10
10-2
AUC
10-2
10-3
10-1
10-1
10
10
10
10"
10
10a
10
10"
10a
10a
10a
False Positive Rate
False Positive Rate
False Positive Rate
Private Target Size
(a) Size 10
(b) Size
(c)
100
(d) AUC vs. Size
Figure 25: TRAK end-to-end attack on STL-10: ROC curves for different target dataset sizes and
AUC vs. size summary.
10a
10a
10a
True Positive Rate
True Positive Rate
True Positive Rate
10-1
10-1
10-1
ROC AUC
0.55
10-2
10-2
10-2
0.50
10"3
10
0.45
10
10
10-1
10-1
10
10a
10-1
10a
10a
False Positive Rate
False Positive
False Positive Rate
Private Target Size
(a) Size 100
(h) Size 500
(c) Size 1000
(d) AUC vs. Size
Figure 26: image-based end-to-end attack on CIFAR-10: ROC curves for different target dataset sizes and AUC vs. size summary.
10a
10"
0.7
True Positive Rate
True Positive Rate
True Positive Rate
10-1
10
10-1
ROC AUC
0.6
10-2
10-2
10-2
10
10
10
10
0.4
10
10
10a
10a
False Positive Rate
False Positive
False Positive Rate
Private Target Size
(a) Size 100
(b) Size 500
(c) Size 1000
(d) AUC Size
Figure 27: Image-based end-to-end attack on CIFAR-100: ROC curves for different target dataset sizes and AUC vs. size summary.
38

---

Published as a conference paper at ICLR 2026
10"
10"
10"
True Positive Rate
True Positive Rate
True Positive Rate
0.55
ROC AUC
0.50
10-2
10-2
10
10
10
10
10
10
10
0.40
10
10
10"
10-1
10
10"
10-1
10
10"
10a
10a
False Positive Rate
False Positive Rate
False Positive Rate
Private Target Size
(a) 100
(h) Size
(c) Size 1000
(d) AUC vs. Size
Figure 28: Image-based end-to-end attack on Food101: ROC curves for different target dataset
sizes and AUC vs. size summary.
10a
10a
10a
True Positive Rate
True Positive Rate
10-1
10-1
10-1
ROC AUC
True Positive
0.50
10-2
10-2
0.45
10"3
10-3
10
10"4
10
10
0.40
10-1
10
10"
10-1
10
10"
10a1
10
10"
10a
10a
False Positive Rate
False Positive Rate
False Positive Rate
Private Target Size
(a) 100
(h) Size
(c) Size 1000
(d) AUC vs. Size
Figure 29: Image-based end-to-end attack on PCam: ROC curves for different target dataset sizes and AUC vs. size summary.
10a
10a
10a
True Positive Rate
True Positive Rate
0.8
10
10-1
True Positive
0.7
10"2
10-3
10-3
10
0.5
10-4
10-4
10
10-1
10-2
10"
10-4
10-2
109
10-t
10-2
10"
10a
False Positive Rate
False Positive Rate
False Positive Rate
Private Target Size
(a)
100
(b) Size
(c) Size 1000
(d) AUC vs. Size
Figure 30: Image-based end-to-end attack on RESISC45: ROC curves for different target dataset
sizes and AUC vs. size summary.
10a
10a
10a
True Positive Rate
True Positive Rate
True Positive Rate
0.7
10
10
10
ROC
10
10
10
0.5
10
10
10
0.4
10
10-2
10a
10
10-2
109
10-1
10-2
10a
10a
10a
False Positive Rate
False Positive Rate
False Positive Rate
Private Target Size
(a) 100
(b) Size
(c) Size 1000
(d) AUC vs. Size
Figure 31: Image-based end-to-end attack STL-10: ROC curves for different target dataset sizes and AUC vs. size summary.
39

---

Published as a conference paper at ICLR 2026
10a
10"
100
TPR. @ 195 FPR
TPR 15 FPR
TPR. @ 15 FPR
10
10
10
10"
10
10'
Target
Target
Target
(a) CIFAR-10
(h) CIFAR-100
(c) STL.-10
Custom
10a
10"
10"
Random
TPR. @ 1% FPR
TPR 1% PPR
TPA. 1% FPR
10-1
10-2
10-2
10-
10a
10
10*
Target Dataset Size
Target
Target
(d) RESISC45
(c) PCam
(f) Food101
Figure 32: Image-based attack success with access to scores ablated over target dataset size.
10
10a
TPR 15% FPR
@ 1%
10
@
10-2
10
10
10c
10a
10a
Target Dataset Size
Target Dataset Size
Target Dataset Size
(a) CIFAR-10
(b) CIFAR-100
(c) STL-100
Binary LIRA
Sigmoid Binary LIRA
10"
10a
Random
TPR @ 1% FPR
10-1
10-1
10-1
@
@
10-2
10-2
10-2
10a
10a
10a
Target Datasat Size
Target Dataset Size
Target Dataset Size
(d) RESISC45
(c) PCam
(f) Food101
Figure 33: Image-based attack success without access to scores ablated over target dataset size.
E.5 TARGET DATASET SIZE ABLATION
Figures 32 to 35 show the effect of target dataset size on attack success. For Image-based curation, the leakage considerable all dataset sizes. Since only the nearest neighbors determine the scores, a larger dataset size means fewer samples are exposed, but the sensitivity of the exposed samples remains constant. For TRAK b-aseed curation, small dataset sizes all samples are exposed, but at larger sizes the averaging has shielding effect for all samples as well.
40

---

Published as a conference paper at ICLR 2026
TPR @ 1% FPR
TPR 1N FPR
TPR. @ 1% FPR
10-1
10-1
10-4
Dataset Size
Target Dataset
Target Dataset Size
(a) CIFAR-10
(h) CIFAR-100
(c) STL--10
LIRA
-
Least Squares
10a
10"
10"
Random
TPR @ 1% FPR
TPR @ 1% FPR
TPR. @ 1% FPR
10-1
10-1
10-4
10-2
10-2
10-1
10
10a
10a
10a
102
10a
10a
10a
10
Target
Target
Target
(d) RESISC45
(c) PCam
(f) Food101
Figure 34: TRAK-based attack success with access to scores ablated over target dataset size.
TPK 136 FPR
10-1
10-1
10-1
3
@
10:2
10f
Target Dataset
Target Dataset Size
Target Dataset Size
(a) CIFAR-10
(b) CIFAR-100
(c) STL-10
Binary LIRA
Sigmoid Binary LIRA
10a
10a
Random
TPR @ 33% FPR
10-1
10-1
10-1
@P
TPR @
10-2
10-2
10-2
102
102
10a
10a
103
10f
10a
10a
10"
Target
Target
Target
(d) RESISC45
(e) PCam
(1) Food101
Figure 35: TRAK TRAK-based attack success without access to scores ablated over target dataset size.
41

---

Published as a conference paper at ICLR 2026
50
1.25
TPR @ 1% FPR (%)
TPR @ 1% FPR (%)
40
1.00
30
PCAM
0.75
CIFAR-10
20
0.50
CIFAR-100
STL-10
10
0.25
Food-101
RESISC45
0
0.00
16
32
64
128
256
16
32
64
128
Number of Shadow Models
Number of Shadow Models
(a) Image-based based
(b) TRAK
Figure 36: Shadow models ablation. (a) For Image-based curation we show that the improvement of adding more shadow models varies drastically between datasets. (b) For TRAK, we do not observe any improvement in attack success when adding more shadow models.
100
102
FPR (%)
1% FPR (%)
75
101
PCAM
50
CIFAR-10
TPR @
TPR
CIFAR-100
25
10
STL-10
Food-101
512
1024
2048
4096
8192
16384
32768
RESISC45
2
16
64
128
256
768
Embedding Dimension
Gradient Dimension
(a) Image-hased
(b) TRAK
Figure 37: Dimension ablation. We show that the number of (a) embedding dimensions for Image-
based curation and (b) projection dimensions for TRAK have a moderate effect on attack success on the scores.
E.6 SHADOW MODELS ABLATION
Figure 36 shows that the improvement adding more shadow models varies drastically between datasets. For Image-based curation, the improvement is connected to the sparsity of the signals that Figure shows. For TRAK, the improvement is more consistent across datasets.
E.7 DIMENSION ABLATION
Figure 37 shows the effect of changing the number of dimensions. For Image-Based curation, show that the attack success is highest for 128 dimensions using our voting based attack (for LiRA 64 dimensions). Because attacks on Image-based curation exploit one-to-one mappings of measurements between target and pool samples, having fewer dimensions can increase the reliability of those measurements. Only extremely few dimensions (e.g., two) does the attack success drop to that of random guessing, For TRAK, we need sufficient number of dimensions (< 1024), below which the attack success drops significantly.
F
LLM USAGE
We have used LLMs to speed up the process typesetting figures, algorithms, and formulas in 1TTEX. We further employed LLM--based agents for writing code, especially to generate figures.
42


Reviews:
[{"review_id": "ERgO81lQ51", "reviewer": "Reviewer_Ju1r", "summary": "This paper presents the first comprehensive privacy analysis of data curation pipelines in machine learning. The authors challenge the common assumption that training models on curated public data inherently protects private target datasets used for curation. Through systematic membership inference attacks across three pipeline stages (curation scores, selected subsets, and final models), the paper demonstrates that each stage leaks information about the private target set.", "questions": "- Can you please provide a single, clear definition of the threat model?\n\n- Can you please provide a more detailed justification for your 'shadow' methodology? Specifically, why is re-running the curation algorithm on random subsets of the private data $\\mathcal{T}$  a sound way to construct the in/out distributions for a LiRA attack? This seems to be a custom algorithm that borrows the LiRA name, and its statistical validity is not obvious.\n\n- Can you provide any evidence that this assumption of selection equals detection holds? For example, can you show that a fingerprint sample, when selected, always creates a detectable signal in the final model, even when trained as part of a large curated set?\n\n- Could you please analyze the query budget/cost or computational cost of the attack? How would it affect the scalability of the method?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- This is the first systematic analysis of privacy leakage in data curation pipelines, addressing a genuine blind spot in the community. it systematically evaluates privacy leakage across three critical stages of the curation pipeline (curation scores, selected data subsets, and the final trained models) using diverse datasets and curation methods.\n\n- The analysis of influence sparsity in image-based curation provides valuable insights into why certain methods are more vulnerable.\n\n- The paper delivers actionable insights: simple max-based curation is fundamentally insecure, while average-based curation (like TRAK) provides innate robustness except in the small-dataset regime .", "weaknesses": "- The threat model is not clearly defined. The adversary's assumed capabilities and knowledge are not clearly stated upfront and appear to change depending on the attack. The adversary knowledge could be escalates to extreme, white-box levels. For the end-to-end TRAK attack, the adversary is assumed to have profound knowledge of the curation mechanism, including the model architecture, the ability to compute gradients, and the need to calculate the Gram matrix $G$. This level of white-box access is not clearly stated or justified.\n\n- The paper's core attack methodology is its adaptation of LiRA. This adaptation is (a) confusingly explained and (b) methodologically questionable. Running an attack algorithm on subsets of the private data to build in/out distributions  is not standard and is not justified as a sound method for simulating the true 'in' and 'out' worlds.\n\n- The paper's strongest end-to-end attack is weakened by its reliance on a proxy metric. The authors state the full end-to-end attack is 'computationally intractable'. Their solution is to 'assume that the adversary can measure whether $f \\in \\tilde{D}$.' This assumption effectively equates selection with detection, completely ignoring the possibility that ML training dynamics (e.g., SGD noise, and the influence of thousands of other samples) could 'drown out' the fingerprint’s signal, making it undetectable in the final model even if it had been selected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YVmvJLeSXU", "reviewer": "Reviewer_AAT3", "summary": "This paper investigate the privacy risks of data curation and show effective membership inference attacks on all the major steps of data curation. This includes similarity score evaluation of the data points in the public dataset (which is the easiest), and the construction of the curated public dataset (which is harder), and the training of the ML model on the curated dataset. The paper builds upon popular and well-known membership inference attacks and provided versions that works regardless of the data curation method used. Furthormore, custom attack methods are designed for image similarity score-based curation and TRAK-based curation. Experiments show that while data curation does not train any model on the private dataset, but all its major steps still leak the membership of the private domain dataset, and it is worse for smaller domain dataset. This reveals new privacy risks of data curation.", "questions": "1. Some important parameter and implementation details are not explained. For example, how many shadow models $m$ are used in the attacks? What are the parameters in the data curation procedure and what is the setting of the model training procedure? When training the model, is regularization used? If so, how heavy is that?\n2. The paper does not discuss the effects of $m$ on the attack performance. I would like to see the relation between the compute spent by the attacker and the attack effectiveness.\n3. For end-to-end model MIAs, all results are when the selected target subset is 1% of the target set. I understand that model training is time consuming, but I would like to see the attack performance for more TPRs.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper reveals new privacy risks of data curation via membership inference attacks.\n2. Systematic review of MIAs in all steps in the data curation pipeline, and attacks are proposed and validated for all of them.\n3. Existing MIAs work for agnostic with modifications, but the paper also proposes custom attacks are proposed to target concrete data curation methods, including TRAK and image-based data curation.", "weaknesses": "1. Choice of parameters are not clear or discussed in the main text.\n2. Computational complexity of the attacks is not discussed.\n3. Limited experiments for end-to-end model MIAs.\n\nPlease see the questions listed in the section below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FMrZNvYmdT", "reviewer": "Reviewer_zhSF", "summary": "The paper argues that “privacy via curation” (only using sensitive data to select public data and then training solely on the public subset) is not automatically private. They design membership inference attacks against: (i) curation scores; (ii) the selected public subset; and (iii) the final model (via a small number of crafted public “fingerprint” samples).The results show that image-embedding/nearest-neighbor style curation is much more vulnerable; TRAK-style gradient-averaged curation is more robust but still leaky for small target sets.", "questions": "Please address the “Weaknesses” above in your rebuttal—especially by (1) stating the problem crisply, (2) explaining why it matters in realistic deployments, and (3) laying out clear threat models for all three attack stages. If those pieces are clarified and make sense, I’m inclined to raise my score.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The problem itself seems interesting.", "weaknesses": "1. I’m familiar with LiRA and recognize you use an online variant, but the three attack surfaces are hard to follow because the threat model isn’t explicitly stated up front. Please spell out—on one page—(i) the adversary’s goal (membership in the private target set used for curation vs. classic training-set membership), (ii) what the adversary can observe at each stage (scores, selection mask, final model), and (iii) what the adversary can do (e.g., can they inject public items?). A single “who-sees-what” diagram for the 3 stages would save readers a lot of guesswork. Also, there are many typos in the main text and appendix; a thorough copy edit would help.\n\n2. The paper does not convincingly explain why leakage at the scoring and subset-selection stages matters when, in many realistic deployments, only the final model is exposed. In such cases, standard membership inference on the trained model already demonstrates risk. The authors should clarify concrete scenarios (and access assumptions) where Sections 3.1 and 3.2 introduce new or additional threat surface beyond what model-only exposure already entails.\n\n3. The end-to-end attack hinges on the ability to insert crafted samples into the public pool. This reads as logically circular: relying on an existing injection threat to establish a new leakage threat. The paper should justify the realism and scope of this assumption (e.g., where such insertion is feasible, at what rates, and under what defenses or deduplication) and disentangle the novelty of the proposed leakage from the prerequisite capability.\n\n4. The underlying question is important: if a model is trained on curated public data, where the curation used a private target set, does the deployed model leak membership about that private set? However, the current presentation makes the answer hard to find. A concise figure (or two) that lays out the threat model for all three stages—attacker view, defender assets, and leakage channel—would greatly improve readability. With a clearly defined problem and threat model, the empirical results would be far more persuasive; without that foundation, even sophisticated methods risk failing to convince readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UPmf4hesud", "reviewer": "Reviewer_hymw", "summary": "This paper uncovers a novel and important privacy vulnerability in machine learning pipelines: membership leakage during data curation. The authors demonstrate both theoretically and empirically that data curation, even before model training, can expose sensitive information about which samples were part of the raw dataset.", "questions": "1.Clarify the assumption\n2.Add necessary ablation studies.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper addresses a new angle on privacy leakage. Most membership inference works focus on model outputs; This work focus shifts to data preprocessing and curation.\n2. The problem is well motivated and clearly defined.\n3. The empirical results are convincing and well-presented.", "weaknesses": "1. While the proposed curation leak is novel and conceptually important, its real-world applicability is limited by the assumption that the adversary can access or infer the curation outcome. In many production pipelines, the raw uncurated data or discarded samples are not observable.\n2. The paper briefly mentions DP sampling and randomization but does not propose new defense mechanisms.\n3. Ablation studies (e.g., effect of dataset size or embedding dimension) are not included.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)