Empirical Comparison of Membership Inference Attacks in Deep Transfer Learning
Authors: Yuxuan Bai
Gauri Pradhan
Marlon Tobaben
Antti Honkela
Venue: Accepted by TMLR ()

Abstract: With the emergence of powerful large-scale foundation models, the training paradigm is increasingly shifting from from-scratch training to transfer learning. This enables high utility training with small, domain-specific datasets typical in sensitive applications.
Membership inference attacks (MIAs) provide an empirical estimate of the privacy leakage by machine learning models. Yet, prior assessments of MIAs against models fine-tuned with transfer learning rely on a small subset of possible attacks. We address this by comparing performance of diverse MIAs in transfer learning settings to help practitioners identify the most efficient attacks for privacy risk evaluation. We find that attack efficacy decreases with the increase in training data for score-based MIAs. We find that there is no one MIA which captures all privacy risks in models trained with transfer learning. While the Likelihood Ratio Attack (LiRA) demonstrates superior performance across most experimental scenarios, the Inverse Hessian Attack (IHA) proves to be more effective against models fine-tuned on PatchCamelyon dataset in high data regime.

Published in Transactions on Machine Learning Research (10/2025)
Empirical Comparison of Membership Inference Attacks in Deep Transfer Learning
Yuxuan Bai
yuzuan.bai@@ellinnk..f
Deportment of Computer Science University of Helsinki
Gauri Pradhan
gourk.prudhannhhhtsssk...f
Departmend of Computer Science University of Helsinki
Marlon Tobaben
murtoo..tobbbern@hccssiikkkk
Department of Computer Science University of Helsinki
Antti Honkela
antti. mntfihoonkelahhessinktt
Department of Computer Science Universily of Helsinki
Reviewed on OpenReview: https: // openrevie.. net/ forum? id= TUCgdd
Abstract
With the emergence of powerful large-scale foundation models, the training paradigm is in- creasingly shifting from from-scratch training to transfer learning. This enables high utility training with small, domain-specific datasets typical in sensitive applications. Membership inference attacks (MTAs) provide an empirical estimate of the privacy leakage by machine learning models. Yet, prior assessments of MIAs against models fine-tuned with transfer learning rely on a small subset of possible attacks. We address this by comparing perfor- mance of diverse MIAs in transfer learning settings help practitioners identify the most efficient attacks for privacy risk evaluation. We find that attack efficacy decreases with the increase in training data for score-based MIAs. We find that there is one MIA which captures all privacy risks in models trained with transfer learning. While the Likelihood Ratio Attack (LiRA) demonstrates superior performance across most experimental scenar- ios, the Inverse Hcssian Attack (IHA) proves to be more effective against models fino-tuned on PatchCamelyon dataset high data regime.
1 Introduction
As foundation models increasingly power modern AT systems, their adaptation through transfer learning raises privacy concerns. Recent rescarch has demonstrated that fine-tuned models can inadvertently mem- orize their training data rather than learning generalizable patterns (Chu et. al.) 2025 creating potential privacy vulnerabilities.
Membership inference attacks (MIAs) (Shokri et al. 2017 have emerged as critical tool for quantifying such privacy leakage by determining whether specific data points were used during model training. These attacks not only provide empirical lower bounds on privacy guarantees of a training algorithm, but also expose privacy vulnerabilities in model training strategics. Despite significant advances in MIA methodologics, their evaluation has predominantly focused on models trained from scratch. Figure I shows the varying privacy vulnerabilities exploitable by MTAs between fine-tuned and from-scratch trained models, even when both

---

Published in Transactions on Machine Learning Research (10/2025)
10"
10
TPR
10
MIA against from-scratch training
XIA against trander learning
Random Classifier
10 8
10
10-2
10
FPR
Figure 1: MIA efficacy as measured using LiRA (Carlini et al. 2022 against WideRosNett50-22 Zagoruyko & Komodakis 2016 trained from scratch using CIFAR-10 versus the same model pre-trained on ImageNet- Ik (Deng et al. 2009) when only the last linear layer of the model is fine-tuned on CIFAR-10. The results are averaged over 3 repeats, with each repeat using M + 1 target models (M = 64) that share the same optimized hyperparameters obtained through hyperparamcter optimization (HPO). The errorbars represent the interquartile range (IQR) of corresponding TPR at FPR. The plot demonst that the attack does not behave similarly across the 2 training paradigms, highlighting the need to investigate the performance of different MIA approaches against foundation models fine-tuned using deep transfer learning to ensure that strong attack is used to evaluate their privacy risks.
trained on identical datasets. This divergence suggests that MIA efficacy in transfer learning fundamentally differs from that in from-scratch training.
Earlier works (Carlini et al. 2019: 2021 Lee et al. 2022 Kandpal et al., 2022) mainly focused on examining
memorization pre-training data by large foundation models. [Meus et al. (2025) recommend using fine- tuned versions of large language models to evaluate memorization using MIAs but do not conduct any experiments in this setting. Other studies using MIAs in transfer learning (Tobaben et al.] [0233] 2024 Pradhan et al.) 2025) have typically used limited selection of attacks. A related study on the efficccy of MIAs against machine unlearning by Hayes et al. (2025) shows that the use of weaker versions of MTA overestimates the privacy protection provided by existing unlcarning techniques. Thus, it is crucial to establish the relative elficacy of different. MIAs to ensure that weaker allacks, which underestimate privacy risks, are not used to evaluate membership privacy in transfer learning scenarios.
Our Contributions In this work, we conducted a systematic evaluation of existing score- bosee MIAs
(Yeom et al. 2018 Salem al. 2019 Ye et al. 2022 Liu al. 2022 Bertran et al. 2023 Li et. al. 2024
Suri et al. 2024) in transfer lcarning context.
Motivated by the work by Tobaben et al. (2024). we investigate the relationship between MIA efficacy and fine-tuning dataset size using consistent experimental setups. Our results confirm that MIA efficacy gencrally decrcases as the number of examples per class increases for most score-based attacks in transfer learning, consistent with the power-law relationship previously observed. However, we identify a notable exception: the white-box Inverse Hessian Attack (IHA) (Suri et al., 2024) exhibits markedly different be- havior, demonstrating superior performance high-shot regimes on PatchCamelyon compared to black-box methods. Additionally, we analyze the effects of changing the training paradigm and properties of the attacks MTA efficacy.
2

---

Published in Transactions on Machine Learning Research (10/2025)
2
Related Work
Deep Transfer Learning Deep transfer learning has been widely adopted in machine learning, leveraging knowledge from source tasks to enhance performance on target tasks with limited data (Yosinski et al. 2014) The process involves pre-training on large-scale datasets to learn general-purpose feature representations, followed by fine-tuning on smaller, task-spocific datasets, reducing data requirements and computational costs. However, this approach introduces privacy vulnerabilities. Models may memorize patterns from source datasets [Tramer et al., 2024). Additionally, fine-tuning on small target datasets often leads to overfitting, increasing vulnerability to privacy attacks that can extract information about individual training samples. Rescarchers commonly use pre-trained models like ResNet (He et al.) 2016; Kolesnikov ct al. 2020) and Vision Transformer (ViT) (Dosovitskiy et al. 2021) | due to computational constraints. Therefore, evaluating and mitigating privacy leakage during fine-tuning on sensitive downstream tasks forms a practical motivation for privacy rescarch.
Membership Inference Attacks Membership inference attacks (MIAs) aim to determine whether specific data sample used in training dataset of a target model. These attacks exploit differences in model behavior when responding to samples used for training the model (member samples) versus non- member samples, thereby compromising the privacy of sensitive data. MTAs are typically categorized based on the adversary's 's knowledge and access to the target model (Hu et al.) 2022b). In the white-box sctting, attackers have full access to the model's learned parameters, gradients, and architect details. In contrast, black-box attacks operate with limited information, typically requiring knowledge of the data distribution and potentially access to model architecture and hyperparameters. Black-box MIAs further diverge into 2 primary variants: score-based MIAs that exploit the model's confidence scores, and label-only MIAs that function only with the predicted class labels (Li & Zhang. 2021 Choquelle-Choo Chooe et al. 2021 Peng et al., 2024). Our work mainly focuses on score- score-based MIAs, as they represent the most potent yet practically feasible attacks.
Shadow-model-Dasse vs. Shadow-model-freee MIAs Score-based MTAs can be further divided into shadow and shadow-model-frre approaches. Mladow-model-basee MIAs depend on shadow training (Shokri et al.
2017), technique where the attacker trains surrogate models that mimic the behavior of the target model. T These include met hods such MT-Leeas (Adversary 1) (Salem et al. 2019). Trajectory-
MIA LLu et al. 2022 Sequritial-Metric based MIA (SeqMIA) (Li et al. 2024), Likelihood Ratio Attack
(LiRA) (Carlini et al. 2022), and Robust MIA (RMIA) (Zarifaadeh et al. 2024) Despite their effectiveness,
MIAs require substantial computational resources, especially since their attack efficiency relies on training additional models. This limitation has motivated the development of more computationally efficient shadow-model alternatives, including LOSS attack (Yeom et al. 2018 Attack-P Yee et al.
shadow- hatee--oddl--sseee
2022 and QuA (QMIA) (Bertran et al. 2023
Standard MIA Threat Model In the standard MIA threat model, formulated by Sablayrolles et al. (2019), the allacker is assumed to know the data distribution and the specifications of the model including the training procedure, model architecture, training hyperparameters, etc. Most the attacks listed in Table follow this threat model, except the Inverse Hessian Attack (IHA) proposed by Suri et al. (2024). IHA relies on 2 key assumptions: for a given target sample, assumes knowledge of all n records n-sized training dataset except for the target sample, and access to the target model's parameters (see Section 3.3 in Suri et al. (2024)). Thus, IHA does not follow the standard threat model for MIAs. Table [ summarizes the threat models for all MIAs used in this paper.
3 Score-based MIAs
In this section, we describe the different MIAs employed in this paper and highlight how they differ in their approach to estimate membership privacy.
Preliminaries Let D be a dataset sampled from data distribution æ. This dataset used to train a machine learning model M with parameters 0. Next, we establish the probability notations used in the
3

---

Published in Transactions on Machine Learning Research (10/2025)
Table 1: Sunmarizing MIAs in terms of the auxiliary information about the target model available to the attacker. Training Data Access implies that the attacker can access all but the target record in the training dataset.
Target Model Access
Attack
Data Distribution
Architecture
Hyperparameters
Model Parameters
Training Data Acness
LOSS
Yoom et al.
2018
Attackr
QMIA
[Bertean el a
LRRA
RMIA
Zarizadel el al
2024
MI-Laaxs
Trajectory-MI
Lan el ml
20122
IHA
Suri
paper. Pr(O(z) denotes the probability of observing parameters 0 when in included in the training set, while Pr(0]z) denotes the probability of observing A if x is not in the training set. Conversely, Pr(z|0) represents the probability that x was part of the training set that produced 0.
3.1 Shadow-model-basee MIAs
MI-Leaks MT--Leaks (Adversary 1) (Salem et al. 2019) refines the shadow training approach (Shokri et al., 2017p. It begins by training shadow model using the dataset Dahadow sampled from и. However, instead of using the full prediction vector, MI-Leaks extracts only the top 3 posterior probabilities (or top for binary-class datasets), ordered from highest to lowest for each sample in Dshadow. Following this, it uses the trimmed probability vectors as inputs to train the attack model. This attack model can then be deployed to compute the membership scores for samples in D.
LiRA LiRA is hypothesis testing framework for membership inference proposed by Carlini al. (2022) For given target model, it trains M shadow models, such that the target sample, x, is included in the training dataset for 1/2 of them (IN models) whereas it is excluded from the training dataset of the remaining M/2 models (OUT models). Using the predicted and logit-scaled confidence scores from these IN and OUT shadow models, the attacker can build the IN and OUT Gaussian distributions. Following this, the attacker can employ a likelihood ratio (LR) test (Neyman & Pearson
1933) to compare Pr(0]ar) against Pr(0]z):
Pr(f(z)
(R)
(1)
Pr(0
The attacker use LRo(x) as the membership score to train binary classifier to differentiate between member and non-member samples.
MIA Liu et al. 2022 proposed Trajectory-MIA as an efficient alternative to LiRA. This is because Trajectory-Min MIA uses single shadow model compared to LiRA's multiple shadow models' based approach. The method leverages knowledge distillation (Hinton et al.) 2015) to simulate the target model's training trajectory. Specifically, it performs distillation on both the target and shadow models, minimizing Lullbeck-Leibler (KL) divergence between the student and teacher models' outputs. By recording the per- example training loss trajectory across distillation epochs, Trajectory-MIA captures temporal patterns that differ between member and non-member samples. These loss trajectories serve feature vectors for MLP classifier to predict memberships.
RMIA RMIA (Zarifzadeh et al. 2024 further refines LiRA by incorporating knowledge about the pop- ulation data ~ æ in the likelihood ratio. It introduces a pairwise LR test that explicitly incorporates population samples 2, computing the probability that the pairwise LRR exceeds preset threshold 7:
Pr (I,o( 2) Pr Pr(0
7)
(2)
Pr Pr(0|2)

---

Published in Transactions on Machine Learning Research (10/2025)
This pairwise LR formulation captures the relative relationship between a potential member sample and known non-member samples z drawn from the population. By composing these pairwise comparisons, Zarifzadeh et al. (2024) contend that RMIA achieves greater robustness to distribution shifts between mem- bers and non-members.
3.2 Shadow-model MIAs
LOSS attack (Yeom et al. 2018 uses loss on the target sample ((M(z), y) as membership signal. Since the objective of training a machine Iearning model is usually to minimize their loss on the training samples, it compares the loss the larget sample against fixed threshold to infer membership. Attack-P (Ye et al.) 2022) is an improvised version of the LOSS attack, which constructs empirical cumulative distribution function (CDF) from the population samples. Unlike the LOSS attack which uses fixed threshold Attack-P compares the loss of the target sample to the distribution of losses from known non-members, calculating what percentage of non-member losses fall below the target's 's loss. Specifically, these attacks compare Pr(x(0) with Pr(z|0) as the threshold to determine the membership of x.
LOSS Attack/ Attack-P
QMIA QMIA (Bertran et al. 2023 determines per-sample thresholds by performing quantile regression on the distribution of confidence scores obtained from known non-member data. By training a regression model with pinball loss to predict these thresholds at a desired false positive rate, QMIA creates a muanced decision boundary that adapts to individual sample. This model-agnostic approach was proposed as an effective alternative to shadow-model- MIAs black-box scttings where only API access is available.
IHA Proposed by Suri et al. (2024), the inverse hessian attack (IHA) builds on recent advancements in discrete-time SGD dynamics (Liu et al. 2021 Ziyin et al. 2022). They show that optimal membership inference requires white-box access to the model parameters post-training and knowledge of all but the target record in the training dataset, rather than relying only on output predictions. IHA relies on local similarity assumption, which posits that models trained with without specific data point converge to similar local minima. Under this assumption, the Hessian matrices at the respective optima share similar structure: H. Ho(w)) = H1 (w)), where wi and wi represent the optimal paramcters for models trained without and with the target sample, respectively. In addition, the loss functions achieve similar values at these local minima: La = Lo(w)) = L1 (w]). This assumption allows IHA to approximate the optimal membership inference by formulating the MIA scoring function using terms dependent on gradients and model parameters.
4 Experimental Setup
Datasets We use CIFAR-10, CIFAR-100 (Krizhevsky 2009), and PatchCamelyon (Veeling et al. 2018) in our experiments. CIFAR-10 and CIFAR-100 are common benchmark datasets for MIA evaluation. Patch- Camelyon, including only 2 classes, enables experiments with substantially larger number of shots S (exam- ples per class), providing greater insight into how training set size affects MIA efficacy.
Models We use VIT-B/16 (Dosovitskiy et al. 2021 and BiT-M-R50x1 (R-50) (Kolesnikov et al. 2020 the backbone models for fino-tuning, both pre-trained on ImageNet-21k (Deng et al. 2009).
Parameterization We employ' 3 schemes for parameterization: (i) Head-only, where only the classification layer is replaced by trainable linear layer, with initial weights set to O, while the feature extraction backbone remains frozen; (#) ALL, where all parameters are trainable during fine- fine-tuning, with initialization from the pre-trained weights; and (Mi FILM, where FiLM adapters (Perez et al.) 2018) are introduced throughout the network alongside a trainable classification head. This paraneler-effice convolutional and transformer architectures, enables more expressive adaptation while minimizing trainable parameters compared full fine-tuning. Although alternatives such as LoRA (Hu et al., (202aa). and CaSE (Patacchiola al.] 2022) exist, FiLM selected due to its demonstrated effectiveness parameter-officient
technique, applicable to both
few-shot transfer learning (Shysheya et al. 2023 Tobaben et. al. 2023
5

---

Published in Transactions on Machine Learning Research (10/2025)
CIFAR-10
CTFAR-100
PatchCamelyy
0.1
0.01
TPR =0.001
0.001
0.0001
0.001
0.0001
128
256
512
1024
2048
16
32
64
128
256
256
1024
4006
16384
65536
S (Shots)
(Shols)
(Shots)
LIRA (M-64)
MI-Leaka (M=1)
Attack-P
Shadow-model-baae (Max)
RMIA (M=64)
Trajectory MIA (M=3)
QMIA
Random Classifirr
Shadow-model free (Max)
Figure 2: MIA efficacy against VIT-B/16 (Head-only) models as a function of S (shots). Upper: Shadow-model-basse attacks using M shadow models. Lower: Shadow-model-free attacks. The errorbars represent the interquartile range (IQR) of the estimated TPR at fixed FPR and the dotted lines represent the maximum of the median MIA efficacy of Shadow-modell-aaee (THA excluded). Shadow-model--esse atlacks generally demonstrate more stable and stronger MIA elficacy compared to shadow-model-free attacks. In the high-shot regime of PatchCamelyon, however, the white-box IHA has a considerable advantage over other MIAs in terms of MIA efficacy, leveraging on its access to all but the target record in the training datasct. Results are averaged over 10 repeats and we use target model per repeat.
and black-box shadow-model-fre attacks
Hyperparameter Optimization Before model training, we perform hyperparameter optimization (HPO) identify the optimal set of hyperparameters parameters to train both the target and the shadow model(s). We begin by sampling 1/2 of the training dataset (D) for HPO. In each HPO trial, we use 70% of the data for training the model while the remaining 30% is used as validation dataset. We run the HPO for 20 trials to explore the hyperparameter space. We implement HPO using Optuna (Akiba et al. 2019) with Tree-structured Parzen Estimator (TPE) algorithm Bergstra et al. 2011). Table 2 summarizes the hyperparameters and their corresponding search ranges used in our experiments.
train the target model. We use the same set of hyper-
Table 2: Hyperparameter search ranges used for Bayesian optimization with Optuna.
Hyperparameter
Parameterization
Range
Head-only
[1, 200]
Epoch
FiLM
[1, 40)
Train Batch Size
Hlad-only/FiLL.
[10, 1000]
Learning Rate
Tead-only/FiLM [10-7, 10-2]
Metrics The metrics used to evaluate the efficacy of MIAs include: (t) log-scaled ROC curves to visualize the trade-off between true positive rale (TPR) and false iositive rate (FPR), with emphasis on the low FPR region, (i) TPR at low FPR to measure MIA efficacy when the false positives must be minimized, better representing realistic attack scenarios (Carlini al. 2022), and (ūi) Interquarttle Range (IQR) serving as the difference between the 25th and 75th percentile of set of values and provides trimmed estimation that is less sensitive to outliers compared to standard
statistical measure to quantify uncertainty.

---

Published in Transactions on Machine Learning Research (10/2025)
deviation. This helps mitigate the impact of extreme values that may occur due to particularly favorable unfavorable random initializations, providing a more reliable assessment of variation in MTA efficacy across different experimental repeats.
Unless otherwise specified, all our results presented are averaged over 10 exper- imental repeats. Within each repeat, we sample a new subset of the population dataset (e.g. CIFAR-10). We sample the datasets to train the target and shadow models from the selected subset such that for each sample in the target model's training dataset, we have 1/2 of the shadow models trained with a whereas the remaining are not training with Additionally, we run the HPO algorithm for each experimental repeat to find optimal hyperparameters to train the target and shadow models. In order to ensure fair comparisons between different MIAs, we restrict our experiments to 1 target model per repeat as more target models are computationally expensive. With LiRA it is possible to use the so-called efficient LiRA implementation proposed by Carlini et al. (2022) that uses every shadow model also as a target model (sce Scction 5.3 While it computalionalyy cheap to implement the same also for RMIA, is computationally expensive Lo do the same for the other attacks.
Experimental Protocol
5 Results
We comprehensively analyze the factors that influence MIA efficacy of various attacks transfer learning.
In Section 5.1 we explore the effect of the number of shots on the performance of different attacks.
Section 2
we study how the performance of attacks varies for different parameterization schemes. Tn
Section
5.3 and Section 5.4 study the impact of number of shadow models and data augmentation
the most powerful MIAs, namely, LiRA and RMIA.
In addition, we investigate the impact of attack-specific parameters on MTA efficacy by evaluating how choice of attack threshold, 7, for RMIA (Appendix AA.I] and distillation set size for Trajectory-MIA (Appendix A affect their respective performance. These attacks have been proposed as more efficient alternatives to LiRA but we find their performance to be sensitive the choice of hyperparameters, such as ? in R.IA, introduced by the attack design.
5.1 Effect of Training Dataset Properties
Figure 2 demonstrates the relationships between the MIA efficacy and the number of cxamples per class, or shots (S), across all datasets, confirming the power law relationship proposed by Tobaben el al. (2024) Detailed numerical results are presented in Tables A2 to A4 and model performance is shown in Figure A3 Among all examined attacks, LiRA consistently outperforms other approaches across most experimental settings, exhibiting remarkable stability as evidenced by the narrower confidence intervals for it. The per- formance advantage of LiRA is particularly pronounced at lower S, though this advantage diminishes as $ increases.
While most attacks show monotonic degradation in MIA efficacy with increase in S, IHA displays non- monotonic patterns with multiple fluctuations. This could likely be due to the non-standard MTA threat model underlying IHA where the attacker has knowledge of all the remaining records (except the target sample) in the training dataset of the target model. This demonstrates that while increasing training data volume can effectively reduce average vulnerability to attacks in the standard MIA threat model (as shown by Tobaben et al. (2024)) ) the effect does not extend to attacks assuming a non-standard threat model, such as IHA.
5.2 Effect of Training Paradigm
Figure 3 illustrates the difference in MIA elficacy between Head-only, FiLM and ALL parameterizations for R-50 models fine-tuned using CIFAR-10 across multiple MIAs. Overall, we find that the strongest MIAs demonstrate minimal change in MIA efficacy due change parametcrization. Given that LiRA maintains consistent performance across both parameterization schemes, this suggests that practitioners should choose parameterization scheme based primarily on utility considerations rather than privacy concerns, as the fine-
7

---

Published in Transactions on Machine Learning Research (10/2025)
tuning choice does not significantly alter the efficacy of the most effective MIA approaches. Complete tabular data are provided in Table A5 and the model performance results are shown in Figure A4
LiRA (M-61)
RMIA (M-61)
MT-Leaks
0.1
0.01
TPR@FPR= 001
0.001
0.0001
Trajectory-Mi MIAA
Attack-P
QMIA
0.1
0.01
0.0001
61
256
1021
256
1024
256
1024
S (Shots)
S (Shots)
S (Shots)
Head-only
FiLM
ALL
Random Classifier
Figure 3: Comparison of MIA efficacy against R-50 fine-tuned on CIFAR-10 across 4 different S (shots) with different parameterization strategies: Head-only, FiL,, and ALL. The errorbars represent the interquartile range (IQR) of the estimated TPR at fixed FPR. Results averaged over 5 repeats with 1 target model in each repeat. For the strongest attacks, there is no considerable difference in MIA efficacy across the 3 parameterization schemes. Some points for Trajectory-MIA with ALL fine-tuning are not visible in the plots due to poor performance or OOM issues.
5.3 Number of Shadow Models
Figure 4 illustrates the relationship between the number of shadow models M and MIA efficacy for LiRA and RMIA, 2 of the strongest shadow-modellbesee attacks discussed in this paper. To ensure a statistically robust evaluation, we employ an efficient implementation of LiRA and RMIA proposed by Carlini et al.
(2022). This approach involves sampling M + datasets from the training datasct DD which contains C x S samples (C' classes with S examples per class), such that each sample has 0.5 probability of being selected for any given dataset. We then train models each of these datasets and evaluate attacks against each model while using the remaining M models shadow models.
LiRA exhibits significant sensitivity to variations in M across both low-data (S = 16) and data-rich (S 1024) scenarios but is robust to increase in M beyond M 2 64. While RMTA demonstrates robustness to changes in M, its efficacy does not exceed LiRA in either of the scenarios. The performance for both the attacks stabilize beyond M 2 64 suggesting it be cost-efficient choice for the number of shadow models in deep transfer learning setting. Full data are presented in Table A6
5.4 Effect of Using Data Augmentation During Fine- Tine-Tunig
Prior research has demonstrated that both LiRA and RMTA benefit. from querying each sample multiple times when attacking models trained from scratch using data augmentation (Carlini et al. 2022 Zarifzadeh et al. 2024). These altacks achieve improved performance by using not only the original sample but also augmented versions of it during the inference process. To determine whether similar improvements occur in deep transfer learning scenarios, we fine-tune target models using training datascts augmented with simple transformations, including mirror flipping and pixel shifting. This approach is employed in both from-scratch training PPerez & Wang (017)) and transfer learning (Mehta et al. 2023) to improve model generalization, particularly when working with limited data.
Following the same efficient implementation as described in Section 5.3 multiple augmented queries generated using subset of transformations applied during training. For LiRA,
we evaluated MIA efficacy using
8

---

Published in Transactions on Machine Learning Research (10/2025)
(a) CIFAR-10
Shots=16
Shots=1024
TPROFPR- PR--0.00 -0.0000 0.001
0.01
0.001
0.0001
16
32
128
256
16
32
128
256
(b) PatchCamelyon
Shots=256
Shots= 1024
TPROFPR-O. 0.001
0.1
0.01
0.001
0.0001
M (# of Shadow Models)
LIRA
RMIA
Random Claasifter
Figure 4: Relationship between MIA efficacy and the number of shadow models (M) for LiRA and RMIA against ViT-B/16 B/6 model with Head-only fine-tuned (a) CIFAR-10 and (b) PatchCamelyon. Results demonstrates MIA efficacy in low data availability (shots S = 16 for CIFAR-10 and S = 256 for Patch- Camelyon) and high data availability (S = 1024 for both CIFAR-10 and PatchCamelyon) scenarios. For each configuration, we train M + models per repeat, using cach model as the target while the remaining M serve as shadow models. We compute the average MIA elficacy (TPR at fixed FPR) across M target models per repeat, then construct boxplots using these average TPR from 5 independent repeats, LiRA dominates in terms of efficacy over RMIA despite the latter's performance being more robust to the variations in M.
the membership signal is averaged over multiple queries directly, while for RMIA, follow the majority voting scheme recommended by Zarifzadeh et al. [002]). Figure shows that data augmentation produces negligible performance improvements for both attacks when targeting Head-only fine-tuned models. This finding represents a significant departure from the from-scratch training findings, suggesting different vulner- ability patterns in transfer learning. Furthermore, the test accuracy of the models trained with and without augmentations remain relatively close. Based on these results, we employ the non-augmented version of both attacks in all experiments for computational efficiency. Table A7 provides complete tabular data.
6 Discussion
Our findings largely corroborate the claim made by Tobaben al. (2024) which states that increasing the number of examples per class generally reduces MIA efficacy. However, do not find this behavior to be consistent across all the attacks. For example, MIA efficacy of IHA increases substantially larger shots observed in Figure 2 IHA docs not use standard MIA thrcat model common to other attacks discussed in this paper. For a given larget sample, it assumes knowledge of all n - 1 records in a n-sized training dataset except for the target sample whereas in the standard MLA threat models, attacks rely on no such prior knowledge of other data records. The power law relationship between datasct size and the MIA performance proposed by Tobabee al. (2024) holds for the standard MIA threat model and thus, does not extend THA.

---

Published in Transactions on Machine Learning Research (10/2025)
LiRA
RMTA
1.00
0.98
TPR@FFR=000M
Test. Accuracy
0.1
0.96
0.01
0.94
0.001
0.92
0.90
0.0001
1024
64
256
1024
S (Shots)
S (Shots)
S (Shots)
augmentation
Mirror
Shift
Random Classifier
Accuracy
Figure 5: Impact of data augmentation on MIA efficacy across S (shots) for VIT-B/16 models Hcad-only fine-tuned on CIFAR-10 with dala tugmenlations. We compare 2 augmentation stralegies: + Mirror (where original image plus a horizontally flipped copy of it are used to train the target model) and + Slift (where horizontally flipping and/or =1-pixel shifts are applicd to the original image), with No augmentation as the bascline. The errorbars represent the interquartile range (IQR). Left and middle pancls show MIA efficacy for LiRA and RMTA, respectively, while the right panel shows test accuracy. Results averaged over 5 repeats and we use M + 1 target models (M = 64) per repeat.
For the most powerful attacks (e.g. LiRA) we find that different parameterization schemes, such as Head-only and FiLM, show minimal differences in terms of MIA efficacy. One notable exceptions Trajectory-MLL which shows increased vulnerability against Head-only fine-tuning. However, Trajectory-MIA is weaker and less stable compared to LiRA, which maintains consistent performance across both parameterization schemnes. This implics that the parameterization choices for fine-tuning could be guided primarily by the utility as they do not significantly affect the performance of the strongest attacks.
Carlini et al. (2022: Zarifzadeh et al. (2024) suggest that data augmentation can be utilized to improve MIA efficacy against. models trained from scratch. However, Section 5.4 we observe no significant improvement in MIA efficacy due to augmentation against models with the last linear layer subject to fine-tuning.
No single MIA is able to capture all vulnerabilities in fine-tuned models. LiRA provides robust auditing capabilities but shows decreased efficacy as dataset sizes grow. THA shows potential to detect vulnerabil- ities missed by black-box attacks, particularly with datasets that have different characteristics from the pre-training data, such as PatchCamelyon (Goyal et al. 2023 Choi et al.] 2024 Thaker et al.) 2024). Com- prehensive privacy auditing requires multi-faceted approach that combines both black- box and white-bon methods.
Limitations
We focus on balanced datasets in our experiments to evaluate the relationship between examples per class (S) and MIA efficacy for different attacks because this design choice enables clear comparison across different data availability scenarios. Future work could extend this analysis to imbalanced datasets commonly found in real-world deployments.
To ensure a fair comparison across different attacks in sections 5.1 and 5.2 we restrict our ex- periments to having target model per repeat. This differs from Carlini et al. (2022)* 's efficient implementation of LiRA where they reuse all the M + 1 trained models as the target model and average the MIA efficacy over all of them. This is because attacks such as Trajoctory-MIA require training additional distilled models for each target model- shadow model pair, which makes the attack computationally infeasible if the number of target models per repeat is set to be the
10

---

Published in Transactions on Machine Learning Research (10/2025)
as LiRA (M + +). Similar computational constraints are associated with IHA where approximat- ing iHVPs per target sample per model will be prohibitively expensive for large numbers of target models.
Another limitation of our paper is that we do not offer detailed comparison between the perfor- mance of different MIAs against models trained with finc-tuning and from-scratch beyond Figure 1 This is because running these experiments for models trained from scratch would be computationally expensive.
7 Conclusion
In this work, we evaluated and compared the performance over a large set MIAs in transfer learning setting. We found that the attack strength deteriorates as the dataset size increases for MIAs in the stan- dard threat model. This agrees with the power law postulated by [Tobaben et al. (2024). However, this relationship is not guaranteed to hold for all the attacks discussed in this paper, such as IHA operates under threat model distinct from other attacks. This shows that there is single existing attack that can fully quantify the privacy leakage in deep transfer learning. In addition, we found no significant difference in MIA efficacy between Head-only and FiLM parameterization strategies for most attacks, implying that choice of parameterization can be made with a utility-first perspective. However, MIAs are sensitive to the choice of attack propertics, such as the number of shadow models used for the attack. These empirical findings provide guidance for practitioners seeking assess privacy risks using MIAs in deep transfer learning applications.
The code for our experiments is available at: https://github: /ttp://w...appaaaaaaooooooooooar tacks by adapting the code provided by prior works' authors |
learningg We implement the at-
Acknowledgments
This work was supported by the Research Council of Finland (Flagship programme: Finnish Center for Ar- tificial Intelligence, FCAI, Grant 356499 and Grant 359111), the Stratogic Research Council at the Research Council of Finland (Grant 358247) as well as the European Union (Project 101070617). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union the Europcan Commission. Neither the European Union the granting authority be held respon- sible for them. The authors wish to acknowledge CSC IT Center for Science, Finland, for computational resources (Project 2003275).
References
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A Next- generation Hyperparameter Optimization Framework. In Proceedings of the 25th ACM SIGKDD Inter- national Conference on Knowledge Discovery & Data Mining, pp. 9623-2631. Association for Computing Machinery, 2019.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for Hyper-Parameli Opti- mization. In Proceedings of the 25th International Conference on Neural Information Processing Systems, volume 24, pp. 2546 2554. Curran Associates Inc., 2011.
Martin Bertran, Shuai Tang, Aaron Roth, Michael Kearns, Jamie H Morgenstern, and Steven Z. Wu. Scalable Membership Inference Attacks via Quantile Regression, In Proceedings of the 37th International Conference on Neural Information Processing Systems, volume 36, pp. 314-330, 2023.
Suri et al. 2024
bttpp://ggchub..co/prrraaaanrrtp prtp:/yypppppryyccccccommm moter tteeeeeeeeeeeee /ww.em/restterchhhhhhhaccca
Ye et al. 2022 Zarifzadeh et al.
2024
http::///g....ccocooooooo//////a Bertran
2023
Eon--Explitingg Loss Tra Trajectory
Liu et al.
2022
1aiir_a0000 2021
Carlini et 2021
http:///ghhiicccm///imaaalae/aat Salem 2019

---

Published in Transactions on Machine Learning Research (10/2025)
Nicholas Carlini, Chang Liu, Ülfar Erlingsson, Jernej Kos, and Dawn Song. The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks. In Proceedings of the 28th USENIX Conference on Security Symposium, pp. 267-284. USENIX Association, 2019.
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Warbert-Voss.. Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ülaar Erlingsson, Alina Oprea, and Colin Raffel. Extracting Training Data from Large Language Models. In 30th USENIX Securily Symposium (USENIX Securily 21), pp.
USENIX Association, 2021.
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership Inference Attacks From First Principles. In 2022 IEEE Symposium. Security and Privacy (SP), pp- 1897 1914. IEEE Computer Society, 2022.
Caroline Choi, Yoonho Lee, Annic S Chen, Allan Zhou, Aditi Raghunathan, and Chelsea Finn. AutoFT: Robust Fine-Tuming by Optimizing Hyperparameters on OOD Data. In NeurIPS 2023 Workshop on Distribution Shifts: New Frontiers with Foundation Models, 2024.
Christopher A. Choquette-Choo Florian Tramer, Nicholas Carlini, and Nicolas Papernot. Label-Only Mem- bership Inference Attacks. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pp. 1964-1974. PMLR, 2021.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Sergey Levine, and Yi Ma. SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training. In The Second Conference on Parsimony and Learning (Recent Spollight. Track), 2025.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNel: A Large-Scale Hierarchical Image Database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248 2482. IEEE Computer Socicty, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations, 2021.
Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune Like You Pretrain: Improved Finetuning of Zero-Shot Vision Models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19338-19347, 2023.
Jamic Hayes, Ilia Shumailov, Eleni Triantafillon, Amr Khalifa, and Nicolas Papernot. Inexact Unlcarning Needs More Careful Evaluations to Avoid a False Sense of Privacy. In 2025 IEEE Conference Secure and Trustworthy Machine Learning (SaTML), pp. 497-519. IEEE Computer Society, 2025.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778. IEEE Computer Sociely, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in a Neural Network. arXiv Preprint arXiv: 1503.02531. 2015.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Chen Weizhu. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations, 2022a.
Hongsheng Hu, Zoran Salcie, Lichao Sun, Gillian Dobbie, Philip S. Yu, and Xuyun Zhang. Membership Inference Attacks Machine Learning: A Survey. ACM Comput. Surv. 54(11s), 2022b.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating Training Data Mitigates Privacy Risks Language Models. In Proceedings of the 39th International Conference on Machine Learning, volume 162, pp. 10797-10707. PMLR, 2022.
12

---

Published in Transactions on Machine Learning Research (10/2025)
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big Transfer (BiT): General Visual Representation Learning. In Compuler Vision - ECCV 2020, pp. 491- 507. Springer-Verlag, 2020.
Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. Technical report, University of Toronto, 2009.
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burde and Nicholas Carlini. Deduplicating Training Data Makes Language Models Better. In Proceedings of the 60th Annual Meeting of the Associalion for Computalional Linguistics (Volume 1: Long Papers), pp- 8424 8445. Association for Computational Linguistics, 2022.
Hao Li, Zheng Li, Siyuan Wu, Chengrui Hu, Yutong Ye, Min Zhang, Dengguo Feng, and Yang Zhang, SoqMIA: Sequential-Motric Based Membership Inference Attack. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, CCS '24, pp. 3496-30 3510. Association for Computing Machinery, 2024.
Zheng Li and Yang Zhang. Membership Leakage in Label-Ouly Exposures. Proceedings of the 2021 ACM STGSAC Conference on Computer and Communications Security, CCS '21, pp. 880-895 Association for Computing Machinery, 2021.
Kangqiao Liu, Liu Ziyin, and Masahito Ueda. Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descent. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 7045-70555 PMLR, 2021.
Yiyong Liu, Zhengyu Zhao, Michael Backes, and Yang Zhang. Memnershhi Inference Attacks by Exploiting Loss Trajectory. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communicatiins Security, CCS '22 pp. 2085-2098, Association for Computing Machinery, 2022.
Matthicu Meeus, Igor Shilov, Shubham Jain, Manuel Faysse, Marck Rei, and Yves-nlex de Montjoye. SoK: Membership Inference Atlacks on LLMs are Rushing Nowhere (and How to Fix It). In 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pp. 281-401. IEEE Computer Society, 2025.
Harsh Mehta, Abhradeep Guha Thakurta, Alexey Kurakin, and Ashok Cutkosky. Towards Large Scale Trans- fer Learning for Differentially Private Image Classification. Transactions Machine Learning Research, 2023.
Jerzy Neyman and Egon Pearson. IX. On the Problem of the Most Efficient Tests of Statistical Hypotheses. Philosophical transactions of the Royal Sociely of London, (694-7000)):288 337, 1933.
Massimiliano Patacchiola, John Bronskill, Aliaksandra Shysheya, Katja Hofmann, Sebastian Nowozin, and Richard Turner. Contextual Bqueero-and-Fxxcitttan for Efficient Few-Shot Image Classification. In Pro- ceedings of the 36th International Conference on Neural Information Processing Systems, volume 35, pp. 36680 36692. Curran Associates, Inc., 2022.
Yucfeng Peng, Jacchul Roh, Subhransu Maji, and Amir Houmansadr. OSLO: One-Shot Label-Only Mcm- bership Inference Attacks. In Proceedings of the 38th International Conference on Neural Information Processing Systems, volume 37, pp. 62310-62388 Curran Associates, Inc., 2024.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville, FILM: Visual Reasoning with a General Conditioning Layer. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
Lais Perez and Jason Wang. The Effectiveness of Data Augmentation in Image Classification using Deep Learning, arXiv preprint ar arXi::1712.0,40 2017.
Gauri Pradhan, Joonas Jälkö, Marlon Tobaben, and Antti Honkela. Hyperparamcters in Scoro-Basok Membership Inference Attacks. In 2025 IEEE Conference on Secure and Trustworthy Machine Learn- ing (SaTM..), pp. 362 384. IEFE Computer Society, 2025.
13

---

Published in Transactions on Machine Learning Research (10/2025)
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Hervé Jégou. White-bon Black-box: Bayes Optimal Strategies for Membership Inference. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 5558 5567. PMLR, 2019.
Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. MT-Leeks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models. In Proceedings of 26th Annual Network and Distributed System Security Symposium (NDSS). Internet Society, 2019.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmaaikkv. Membership Inference Attack Against Machine Learning Models. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 3-18.
TEEE Computer Society, 2017.
Aliaksandra Shysheya, John F Bronskill, Massimiliano Patacchiola, Sebastian Nowozin, and Richard E Turner. FiT: Parameter Efficient Few-shot Transfer Learning for Personalized and Federated Image Clas- sification. In International Conference on Learning Representations, 2023.
Anshuman Suri, Xiao Zhang, and David Evans. Do Parameters Reveal More than Loss for Membership Inference? Transactions on Machine Learning Research, 2024.
Pratiksha Thaker, Amrith Setlur, Zhiwei Steven Wu, and Virginia Smith. On the Benefits of Public Repre- scntations for Private Transfer Learning under Distribution Shift. In Proceedings of the 38th International Conference on Newral Informalion Processing Systems, volume 37, pp. 27088 27120. Curran Associales, Inc., 2024.
Marlon Tobaben, Aliaksandra Shysheya, John F Bronskill, Andrew Paverd, Shruti Tople, Santiago Zanella- Beguelin, Richard E Turner, and Antti Honkela. On the Efficacy of Differentially Private Few-shot Image Classification. Transactions on Machine Learning Reseorch, 2023.
Marlon Tobaben, Hibiki Tto, Joonas Jälkö, Gauri Pradhan, Yuan He, and Antti Honkela. Impacc of Dataset Properties on Membership Inference Vulnerability of Deep Transfer Learning, arXiv preprint arXiv:2402.000 06674, 2024.
Florian Tramèr, Gautam Kamath, and Nicholas Carlini. Position: Considerations for Differentially Private Learning with Large-Scale Public Pretraining. In Proceedings of the 41st Internalional Conference Machine Learning, volume 235, pp. 48453 48467. PMLR, 2024.
Bastiaan S. Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling, Rotation Equivariant CNNs for Digital Pathology. In Medical Image Computing and Computer Assisted Intervention -MICAI 2018, pp. 210-218. Springer-Verlag, 2018.
Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri. Enhanced Membership Inference Attacks against Machine Learning Models. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security CCS '22 pp. 3093-3106. Association for Com- puting Machinery, 2022.
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting. Tn 2018 TEFE 31st Computer Security Foundations Symposium (CSF), pp. 268-282. IEEE Computer Society, 2018.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Proceedings of the 28th International Conference on Neural Information Processing Systems, volume 27, pp. 3320 3328. Curran Associates, Inc., 2014.
Sergey Zagoruyko and Nikos Komodakis. Wide Residual Networks. In Proceedings of the British Machine Vision Conference 2016. BMVA Press, 2016.
Sajjad Zarifzadch, Philippe Liu, and Reza Shokri. Low-Cost High-Power Membership Inference Attacks. In Proceedings of the 41st International Conference on Machine Learning, volume 235, pp. 58244-58282. PMI.R, 2024.
14

---

Published in Transactions on Machine Learning Research (10/2025)
Liu Ziyin, Kangqiao Liu, Takashi Mori, and Masahito Ueda. Strength of Minibatch Noise in SGD. In International Conference on Learning Represenlations, 2022.
A.1 Influence of y in RMIA Scoring Function
As shown in Equation 2), RMIA uses (2 1) as the threshold for the likelihood ratio lest. It determines how much higher should the likelihood of observing model parameters 9 be a target sample z was in the training dataset relative random population sample to pass the membership tost. As such, y is critical parameter in the RMIA. Following the same efficient. implementation as described in Section 5.3 we conduct sensitivity analysis varying 7 from to 64 and report the MTA efficacy to evaluate attack performance. While RMIA efficacy against models trained from scratch is robust to the choice of as shown by Zarifzadeh ct al. (024),, this robustness does not extend to few-shot transfer learning setting. For ViT-B/ 16 models fine-tuned on CIFAR-10 with the Head-only setting, only RMTA with ? = 2 performs consistently across varying the number of shots. Despite wide error bars, its median performance remains stable.
Shots=16 16
Shots=1024
TPRGFPR=0000
0.1
0.01
0.001
0.0001
20
21
22
23
24
2
2
20
21
22
2
24
25
2
7
7
LIRA
Random Classifier
Figure A1: RMIA efficacy function of the threshold parameter 7. Results show MIA efficacy against ViT-B/16 models Head-only fine-tuned on CIFAR-10. We train M + models (M = 64) per repeat, using each model as the target while the remaining M serve as shadow models. We compute the average MIA efficacy (TPR at fixed FPR) across all M + target models per repeat, then construct boxplots using the average TPR from 10 independent repeats. RMIA shown to be sensitive to the value of 7 in deep transfer learning setting. The blue dashed line represents the median MIA efficacy achieved by LiRA under identical conditions.
A.2 Effect of Distillation Set Size in Trajectory-MIA
Trajectory-MTA requires an auxiliary dataset for knowledge distillation simulate the target model's training process. Following Liu al. (2022) we split the given dataset into target, shadow, and distillation datasets, and we construct our distillation datasets using all data that is not part of the target and shadow datasets.
Figure A2 illustrates that the distillation set size has to be sufficiently large for -raAectory-MIA work effectively. Smaller distillation sets prevent the student model from learning sufficiently diverse examples to generalize lo unseen data. However, | > 20000 does not. lead to any significant. improvement in Trajectory-MIA efficacy. For CIFAR-10, using [D*| = 20000 for knowledge distillation proves to be more effective than using all available data (~50000 samples) very low-shot regimes like 16. As the number of shots increases, the performance difference between [D*] = 20000 and |D*| All decreases, because the available data for distillation become increasingly similar in both scenarios.
15

---

Published in Transactions on Machine Learning Research (10/2025)
Shots=18 16
Shots= 1024
TPRGFPR= 0.0011
0.1
0.01
0.001
0.0001
20
200
2000
20000
A11
20
2000 20000
A11
DR (Distillation Set Size)
LIRA
Random Guessing
Figure A2: Sensitivity of Trajectory-MIA efficacy to distillation set size. MIA efficacy is evaluated against ViT-B/10 models Head-only fine-tuned on CIFAR-10. |DK| represents the distillation set size, with All representing using all available data not part of the fine-tuning datasets. The errorbars represent the in- terquartile range (IQR) associated with the estimated TPR at fixed FPR. The blue dashed line represents the median MIA efficacy achieved by LiRA under identical conditions. We use LiRA with 64 shadow models as the upper bound on MIA efficacy. Results are averaged over 10 repeats and use 1 target model per repeat.
A.3
Additional Results
A.3.1 Model Performance
The training and test accuracies for the target models are plotted in Figure A3 for the ViT-B/16 models used in Sections 5.1 and 5.3 and in Figure A4 for the R-50 models used in Section 5.2 Figure A3 indicates that datasets exhibiting a large distribution shift from the pretraining data (such as PatchCamelyon) are associated with lower training and test accuracies compared to datasets that closely match the pre-training distribution (such as C1)AR-10). This is consistent with degraded feature transferability under distribution shift. Figure A4 shows that Head-only, FILM, and ALL fine-tuning achieve comparable difference in model performance on the training and test sets, which explains why the choice of training paradigm has little effect on MIA efficacy for the strongest attacks.
CIFAR-10
CIFAR-100
PatchCamelyon
1.0
Accuracy
0.9
0.8
0.7
Trsin
Test
0.6
128
256
512
1024
2018
16
32
64
128
256
256
1024
4096
16381
65536
S (Shots)
S (Shots)
S (Shots)
Figure A3: Training and test accuracies for ViT-B/16 models pre-trained on ImageNet-21k and Head-only tune with varying (shots). The errorbars represent the standard errors. Results are averaged over repeats and we use accuracies from random models per repeat.
16

---

Published in Transactions on Machine Learning Research (10/2025)
Head-only
FiLM
ALL
1.00
Accuracy
0.90
0.85
Train
Test
0.80
16
64
256
1024
16
64
256
1024
16
64
256
1024
S (Shots)
S (Shots)
S (Shots)
Figure A4: Training and test accuracies for R-50 models pre-trained on ImageNet-21k fine-tuned on CIFAR- across varying S (shots) and different parameterization strategies: Head-only, FiLM, and ALL. The errorbars represent the standard errors. Results are averaged over 10 repeats and we use accuracies from 10 random models per repeat. The gap between training and test accuracy decreases S increases, indicating roduced overfitting. This aligns with the observed decline in MIA efficacy, overfitting is strongly correlated with at lack elficacy.
17

---

Published in Transactions on Machine Learning Research (10/2025)
A.3.2
ROC Plots Comparing MIA Efficacy Across Different Attacks
Figures
A5
10
A7
represent full ROC curves for allacks used in Figure 2
S=128
S5256
S=512
10a
1071
TPR.
10
10
10-3
10-2
10-1
10
10-3
10-2
10-1
10
10-3
10-2
10-1
10
FPR.
FPR
FPR
S=1024
$=2048
10a
10
TPR
10
10
10-3
10
10
10a
10-3
10
10
10a
FPR
FPR
LiRA (M=64)
-
Attack-P
QMIA
- IHA
RMIA (M=64)
ML-Lenks (M=1)
Trajectory-Mil MIA (M=3)
Random Classifier
Figure A5: ROC curves for ViT-B/16 models Head-only fine-tuned on CIFAR-10 with varying S (shots). The errorbars represent the interquartile range (IQR) associated with the estimated TPR at fixed FPR. Results are averaged over 10 repeats and we use larget model per repeat.
18

---

Published in Transactions on Machine Learning Research (10/2025)
S-8
S=16
S=32
100
10
TPR
10
10-3
10-3
10
10
10
10 10-3
10
S=64
S= 128
S
10
10-1
TPR
10
10
10
10
10
10t
10
10
10
10a
10
109
FPR
FPR
FPR
-
LIRA (M=61)
-
Attack-P
QMIA
-
IHA
RMIA (M=64)
ML-Leaks (M=1)
Trajectory-MIA (M=3)
Random Classifier
Figure A6: ROC curves for VIT-B/16 models Head-only fine-tuned on CIFAR-100 with varying 5 (shots). The errorbars represent the interquartile range (IQR) associated with the estimated TPR at fixed FPR. Results are averaged over repeats and we use target model per repeat.
19

---

Published in Transactions on Machine Learning Research (10/2025)
S-256
S-512
S-1024
100
10
TPR
10
10
10
10
10
10
10
10-2
10
10a
10
10
10
10
S=2048
S-4096
--81
100
10
TPR
10
10
10
10
10
10"
10
10-2
10
100
10
10
10
109
S 16384
S=32768
665336
10
TPR
10
10
10
10
10
10a
10
10
10
10
10
10
10
10
FPR
FPR
FPR
LIRA (M-64)
- AttackeP
QMIA
IHA
RMIA (M=61)
MI-Leaks (M=1)
Trajoctory-MIA (M=3)
Random Classifler
Figure A7: ROC curves for ViT-B/16 models Head-only fine-tuned on PatchCamellyn with varying S (shots). The errorbars represent the interquartile range (IQR) associated with the estimated TPR fixed FPR. Results averaged over 10 repeats and target model per repeat.
20

---

Published in Transactions on Machine Learning Research (10/2025)
A.3.3 Effect of Domain Shift MIA Efficacy
To assess MIA elficacy when larget models are line-tuned datasets that are out-of-distribution (OOD) relative to the pre-training data, we perform the same analysis as in Section 5.2 using PatchCamelyon. Results are presented in A.333 and Ta Consistent with the findings in Section we observe that the strongest MIAs show minimal sensitivity to parameterization changes. Note that Trajectory-MIA results are omitted due to CUDA memory limitations while training the distilled versions of shadow and target model required for this attack.
LiRA (M-64)
RMIA (M-64)
ML-Leaks
TPROFPR- 000001
0.1
0.01
0.001
0.0001
256
512
1024
2048
4096
256 512 1024 2048 4096
256
512
1024 2048 4096
S (Shots)
S (Shots)
(Shots)
Attack-P
QMTA
TPROFPR=0000 0.0001
0.01
0.001
0.0001
256
512
1024 2018 4096
256
512 1024 2018 4096
S (Shots)
$ (Shots)
Head-only
Classifier
Figure A8: Comparison of MIA efficacy against R-50 fine-tuned on PatchCamelyon with Head-only versus FiLM parameterization across 4 different S (shots). Some points are not visible the plots due to poor performance (TPR< 0.0001).
Table A1: MTA efficacy against R-50 models fine-tuned PatchCamelyon with Head-only FiLM parameterization across 4 different S (shots).
Shots
Attack
Parameterization
TPR@FPR=0.1
TPR@FPR=0.01
TPR@FPR=00001
LARA (M 64)
Head-only
0.340
0.185
0.073
FILM
0.332
0.135
0.085
RMIA [M 64)
Head-only
0.093
0.093
0.093
FiLM
0.108
0.108
0.108
MI.LLaass (M = I)
Head-only
0.093
0.000
0.000
FiLM
0.000
0.000
Attack-P
Head-only
0.085
0.000
0.000
FILM
0.087
0.000
0.000
QMIA
Head-only
0.132
0.000
0.000
FiLM
0.137
0.000
0.000
LIRA (M = 64)
Head-only
0.158
0.030
0.003
FILM
0.213
0.049
0.022
RMIA (M 64)
Head-only
0.128
0.001
FILM
0.174
0.031
0. 0.002
Continued
page
21

---

Published in Transactions on Machine Learning Research (10/2025)
Table A1 continued from previous page
Shots
Attack
Parameterization
1PR#FPR=0.1
1PR@FPR=0.01
TPR@FPR=0.001
ML-Leaks (M 1)
Head-only
0.114
0.005
0.001
FiLM
0.112
0.007
0.001
1024
Attack-P
Hoad-only
0.115
0.005
0.000
FiLM
0.113
0.006
0.000
QMIA
Head-only
0.122
0.009
0.001
LiRA (M 64)
Head-only
0.178
0.046
0.004
FiLM
0.178
0.035
0.006
RMIA (M 64)
Head-only
0.116
0.029
0.013
2048
FiLM
0.157
0.046
0.011
ML-Leaks (M 1)
Head-only
0.100
0.008
0.000
FiLM
0.097
0.008
0.000
Attack-P
Head-only
0.001
QMIA
Head-only
0.126
0.004
FIL.
0.118
0.015
0.004
LIRA (M 64)
Head-only
0.148
0.028
0.006
FiL.M
0.169
0.029
0.008
RMIA [M 64)
Head-only
0.128
0.042
0.007
4096
FIL.
0.186
0.034
0.007
Meaks (M = 1)
Head-only
0.002
Attack-P
Head-only
0.107
0.013
0.000
FIL.M
0.099
0.012
0.000
QMIA
Head-only
0.096
0.011
0.000
FIL.M
0.099
0.010
0.001
22

---

Published in Transactions on Machine Learning Research (10/2025)
A.3.4 Additional Effect of Training Dataset Properties
provide tabular results corresponding to Figure 2 demonstrating how MIA efficacy varies function of S (shots) across three datasets: CLEAR-10, CIFAR-100, and PatchCamelyon. The tables present numerical data for both shadow-model-based attacks (using varying numbers of shadow models M) and shadow-model-free attacks, with results obtained using ViT-B/16 models pre-trained on ImageNet-21k and fine-tuned with Head-only setting.
Tables A2
A4
Table A2: MIA efficacy as a function of (shots) for ViT-B/16 models pre-trained on ImageNet-21k and fine-tuned on CIFAR-10 (Head-only). Values represent medians with interquartile ranges (IQRs). Results are averaged over 10 repeats and we use target model per repeat.
IPR@FPR=0.1
TPR@FPR=0001
TPR@FPR=00001
Shots
Attack
Median
IQR
Median
IQR
Median
IQR
LiRA (M 64)
0.33
0.05
0.12
0.03
0.06
0.05
RMIA 64)
0.04
0.00
8 128
ML-Leaks (M = 1)
0.00
Trajectory-Mol MIA (M = 3)
0.12
0.02
0.01
0.00
0.00
0.00
Attack-P
0.13
0.02
0.02
0.01
0.00
0.00
QMIA
0.24
0.02
0.03
0.02
0.01
0.01
IHA
0.12
0.04
0.02
0.02
0.00
0.01
LIRA (M 64)
0.25
0.06
0.10
0.04
0.06
0.02
RMIA 61)
0.11
0.07
0.06
0.04
0.03
8 256
ML-Leaks (M
0.01
0.00
Trajectory-MIA (M = 3)
0.17
0.18
0.06
0.10
0.02
0.04
Attack-P
0.11
0.01
0.01
0.01
0.00
0.00
QMIA
0.19
0.06
0.02
0.02
0.01
0.00
IHA
0.11
0.02
0.01
0.01
0.00
0.00
LIRA (M 6:1)
0.21
0.04
0.08
0.02
0.05
0.01
RMIA (M 64)
0.15
0.04
0.05
0.02
0.03
0.02
5 = 512
ML-Leaks (M 1)
0.01
0.00
Trajectory MIA 3)
0.04
0.04
Attack-P
0.11
0.01
0.01
0.00
0.00
0.00
QMIA
0.15
0.03
0.03
0.01
0.00
0.00
IHA
0.12
0.03
0.02
0.01
0.01
0.01
LIRA (M 64)
0.17
0.01
0.05
0.01
0.03
0.01
RMIA (M 64)
0.12
0.10
0.04
0.01
0.02
0.01
S 1024
ML-Leaks (M
0.10
0.01
0.01
0.00
0.00
0.00
'Irajcctory-Me MIA 3)
0.01
Attack-P
0.10
0.00
0.01
0.00
0.00
QMIA
0.13
0.02
0.03
0.01
0.00
0.00
IHA
0.11
0.03
0.02
0.01
0.00
0.00
LIRA (M 64)
0.15
0.01
0.04
0.01
0.02
0.01
RMIA (M 64)
0.11
0.04
0.04
0.01
0.02
0.01
ML-Leaks (M
0.10
0.00
0.01
0.00
0.00
0.00
2048
Trajectory-MI. 3)
0.02
0.01
Attack-P
0.01
0.01
0.00
0.00
QMIA
0.12
0.01
0.02
0.00
0.00
0.00
IHA
0.12
0.01
0.02
0.01
0.00
0.01
23

---

Published in Transactions on Machine Learning Research (10/2025)
Table A3: MIA efficacy as a function of (shots) for ViT-B/16 models pre-trained on ImageNet- and fine-tuned on CIFAR-100 (Head-only). Values represent medians with interquartile ranges (IQRs). Results are averaged over 10 repeats and we use target model per repeat.
TPR&FPR=000
0.1
TPRØFPR=
TPR@FFR=
Shots
Attack
Median
IQR
Median
IQR
Median
IQR
LIRA (M 64)
1.00
0.00
0.97
0.03
0.84
0.20
64)
S
ML-Leaks (M 1)
0.02
0.37
0.07
Trajectory-Mia (M 3)
1.00
0.01
0.47
0.46
0.16
0.40
Attack-P
0.99
0.02
0.33
0.17
0.03
0.04
QMIA
0.98
0.02
0.54
0.16
0.20
0.20
IHA
1.00
0.00
0.43
0.30
0.09
0.13
LRRA (M 64)
0.97
0.04
0.71
0.12
0.54
0.21
RMIA 64)
= 16
MI.-Taaks (M 1)
Trajectory-oe MIA (M = 3)
0.92
0.11
0.56
0.46
0.18
0.37
Attack-P
0.52
0.23
0.07
0.02
0.01
0.01
QMIA
0.77
0.13
0.15
0.16
0.04
0.03
THA
0.90
0.22
0.16
0.11
0.05
0.09
LIRA (M 64)
0.87
0.04
0.57
0.04
0.39
0.10
RMIA 64)
32
ML.Leaks (M 1)
0.00
Trajectory-Mo MIA (M = 3)
0.82
0.10
0.51
0.13
0.36
0.12
Attack-P
0.24
0.03
0.03
0.01
0.00
0.00
QMIA
0.55
0.07
0.07
0.03
0.01
0.00
IHA
0.40
0.13
0.04
0.03
0.01
0.02
LIRA (M 64)
0.68
0.03
0.42
0.06
0.30
0.04
RMIA 61)
0.03
0.31
0.20
0.04
ML-Leaks (M
64
Trajectory-Mi. (M = 3)
0.62
0.05
0.35
0.04
0.16
0.06
Attack-P
0.14
0.01
0.02
0.01
0.00
0.00
QMIA
0.29
0.04
0.04
0.01
0.01
0.01
IHA
0.20
0.09
0.04
0.07
0.03
0.07
LIRA (M 64)
0.51
0.03
0.31
0.05
0.21
0.06
RMIA (M 64)
0.48
0.03
0.22
0.02
0.12
0.01
5 128
ML-Leaks (M
0.00
Trujectory MIA 3)
0.10
Attack-P
0.12
0.01
0.01
0.00
0.00
0.00
QMIA
0.19
0.03
0.02
0.01
0.00
0.00
IHA
0.18
0.09
0.05
0.07
0.01
0.07
LIRA (M 64)
0.39
0.01
0.19
0.01
0.11
0.02
RMIA (M 64)
0.36
0.02
0.14
0.01
0.07
0.01
5 256
ML-Leaks (M
0.11
0.02
0.01
0.00
0.00
0.00
Trraecctoy--Me MIA 3)
0.01
0.07
Attack-P
0.11
0.01
0.01
0.00
0.00
QMIA
0.14
0.01
0.02
0.00
0.00
0.00
IHA
0.14
0.04
0.02
0.01
0.01
0.01
24

---

Published in Transactions on Machine Learning Research (10/2025)
Table A4: MIA efficacy as a function of S (shots) for ViT-B/16 16 models pre-trained on ImageNet-21k and fine-
tuned on PatchCamelyon (Heead-nnly). Values represent medians with interquartile ranges (IQRs). Results are averagod over 10 repeats and we use target model per repeat.
TPR&FPR=
0.1
TPRØFPR=
TPR@FPR=0.000
Shots
Attack
Median
IQR
Median
IQR
Median
IQR
LIRA (M 64)
0.29
0.08
0.10
0.05
0.06
0.07
64)
5 256
ML-Leaks (M 1)
0.02
0.00
Trajectory-MiA (M 3)
0.11
0.03
0.01
0.01
0.00
0.00
Attack-P
0.10
0.02
0.01
0.01
0.00
0.00
QMIA
0.19
0.06
0.04
0.03
0.01
0.01
IHA
0.13
0.03
0.03
0.04
0.01
0.01
LRRA (M 64)
0.24
0.10
0.05
0.08
0.03
0.03
RMIA 64)
8 612
MI.-Teaks (M 1)
0.00
Trajectory-oe MIA (M = 3)
0.11
0.03
0.01
0.02
0.00
0.00
AttackeP
0.11
0.01
0.01
0.00
0.00
0.00
QMIA
0.15
0.05
0.04
0.03
0.01
0.01
THA
0.13
0.06
0.02
0.02
0.00
0.00
LIRA (M 64)
0.20
0.04
0.07
0.02
0.02
0.02
RMIA 64)
S 1024
ML.Leaks (M 1)
0.00
Trajectory--o MIA (M = 3)
0.12
0.07
0.01
0.02
0.00
0.01
Attack-P
0.10
0.01
0.01
0.01
0.00
0.00
QMIA
0.15
0.04
0.03
0.01
0.00
0.00
IHA
0.13
0.05
0.01
0.02
0.00
0.01
LIRA (M 64)
0.17
0.02
0.04
0.01
0.02
0.01
RMIA 61)
0.16
0.02
0.04
0.01
0.01
S = 2048
ML-Leaks (M
Trajectory-Mil (M = 3)
0.15
0.04
0.03
0.02
0.01
0.01
Attack-P
0.11
0.01
0.01
0.00
0.00
0.00
QMIA
0.14
0.04
0.02
0.02
0.01
0.01
IHA
0.16
0.05
0.03
0.02
0.00
0.00
LIRA (M 64)
0.14
0.03
0.02
0.01
0.01
0.00
RMIA (M 64)
0.13
0.02
0.03
0.01
0.00
0.00
S = 4096
ML-Leaks (M
0.00
Trajectory MIA 3)
0.04
0.00
Attack-P
0.10
0.01
0.01
0.00
0.00
0.00
QMIA
0.13
0.02
0.02
0.00
0.00
0.00
IHA
0.13
0.04
0.02
0.03
0.00
0.00
LIRA (M 64)
0.13
0.01
0.02
0.00
0.00
0.00
RMIA (M 64)
0.13
0.01
0.03
0.00
0.00
0.00
S 8192
ML-Leaks (M
0.10
0.01
0.01
0.00
0.00
0.00
'IrajectoryMMl MIA 3)
0.00
Attack-P
0.10
0.01
0.01
0.00
0.00
QMIA
0.12
0.01
0.02
0.01
0.00
0.00
IHA
0.17
0.03
0.03
0.05
0.01
0.01
LRRA (M 64)
0.11
0.01
0.01
0.00
0.00
0.00
RMIA (M 64)
0.12
0.01
0.02
0.01
0.00
0.00
Continued on next
25

---

Published in Transactions on Machine Learning Research (10/2025)
Table A4
continued from previous page
TPRSFPR=0.1
TPR@FPR=0.01
TPROFFR=00001
Shots
Attack
Median
IQR
Median
IQR
Median
IQR
ML-ksaks (M =
0.10
0.00
0.01
0.00
0.00
0.00
S 16384
Trajoctory-Mil (M m 3)
0.11
0.01
0.01
0.00
0.00
0.00
Attack-P
0.10
0.00
0.01
0.00
0.00
0.00
QMTA
0.11
0.01
0.02
0.00
0.00
0.00
IHA
LIRA (M 64)
0.11
0.01
0.01
0.00
0.00
0.00
RMIA (M 64)
0.11
0.02
0.02
0.00
0.00
0.00
32768
ML-keaks (M = 1)
0.10
0.00
0.01
0.00
0.00
0.00
Trajectory-Mil MIA (M = 3)
0.11
0.00
0.01
0.00
0.00
0.00
Attack-P
0.10
0.00
0.01
0.00
0.00
0,00
QMIA
0.11
0.01
0.01
0.00
0.00
0.00
IHA
LIRA (M 64)
0.10
0.01
0.01
0.00
0.00
RMIA (M = 64)
0.11
0.01
0.01
0.00
0.00
0.00
65536
MLkLeaks (M = 1)
0.10
0.00
0.01
0.00
0.00
0.00
Trajectory-Mil MIA (M = 3)
0.10
0.00
0.01
0.00
0.00
0.00
Attack-P
0.10
0.00
0.01
0.00
0.00
0.00
QMIA
0.10
0.00
0.01
0.00
0.00
0,00
THA
0.14
0.05
0.04
0.04
0.01
0.02
A.3.5
Additional Effect of Training Paradigm
Table A5 provides tabular results shown in Figure 3 comparing MIA efficacy against R-50 models fine-tuned
CIFAR-10 using three different parameterization strategies (Head-only, FiLM, and ALL) across four shots (S = 18, 64, 256, 1024).
Table A5: MIA efficacy against R-50 fine-tuned on CIFAR-10 across 3 parameterization strategics (Head- only, FiLM, and ALL) and 4 different S (shots). Values represent medians with interquartile ranges (IQRs). Results are averaged over 5 repeats with target model in each repeat.
TPROFPR=0.1
TPRØFPR=0.01
TPR#FPR=0000
Shots
Attack
Parameterization
Median
IQR
Median
IQR
Median
IQR
LiRA (M 64)
Head-only
0.89
0.20
0.51
0.37
0.51
0.37
PILM
0.90
0.07
0.73
0.12
0.73
0.12
ALL
1.00
0.00
0.98
0.02
0.98
0.02
RMIA (M 64)
Head-only
0.16
0.37
0.10
0.16
0.10
0.16
FILM
0.90
0.11
0.67
0.15
0.07
0.15
ALL
1.00
0.00
0.95
0.04
0.95
0.04
MI-L.oaks (M ===] 1)
Head-only
0.39
0.23
0.04
0.00
PILM
0.14
0.18
ALL
0.99
0.01
0.83
0.24
0.83
0.21
Trjjsttry-MIA (M 3)
Head-only
0.31
0.26
0.02
0.04
0.00
0.00
FILM
0.33
0.06
0.01
0.01
0.01
0.01
ALL
0.51
0.25
0.11
0.03
0.11
0.03
Head-only
0.40
0.19
0.05
0.01
0.00
0.00
Continued on next
26

---

Published in Transactions on Machine Learning Research (10/2025)
Table A5 continued from previous page
TPR@PPR=0.1
TPROFPR=0.01
1PR9FPR=0.001
Shots
Attack
Parameterization
Medlan
IQR
Medlan
IQR
Medlan
IQR
Attack-P
FILM
0.53
0.14
0.01
0.16
0.00
0.02
ALL
0.99
0.01
0.60
0.17
0.06
0.02
Head-only
0.60
0.29
0.36
0.11
0.15
0.09
QMIA
PIT.M
0.56
0.04
0.08
0.05
0.06
0.08
LiRA (M 64)
Head-only
0.57
0.07
0.22
0.08
0.13
0.01
FILM
0.55
0.07
0.26
0.10
0.15
0.09
ALL
0.94
0.03
0.72
0.06
0.42
0.20
RMIA (M = 64)
Head-only
0.12
0.19
0.05
0.09
0.03
0.08
FILM
0.41
0.07
0.24
0.02
0.14
0.01
ALL
0.70
0.05
0.42
0.10
0.24
0.12
ML-Loaks (M = 1)
Head-only
0.00
PiLM
0.01
ALL
0.17
0.06
0.01
0.02
0.02
Trajectory-MIA (M = 3)
Head-only
0.27
0.39
0.02
0.21
0.00
0.17
FILM
0.15
0.03
0.02
0.02
0.01
0.01
ALI
NaN
NaN
NaN
NaN
NaN
NaN
Head-only
0.16
0.03
0.02
0.02
0.00
0.00
Attack-P
PIL.M
0.14
0.06
0.02
0.01
0.00
0.00
ALL
0.41
0.07
0.02
0.01
0.01
Head-only
0.01
QMIA
FILM
0.17
0.05
0.03
0.02
0.01
0.00
ALL
0.42
0.08
0.08
0.21
0.02
0.15
LiRA (M 64)
Head-only
0.25
0.06
0.10
0.04
0.06
0.02
FILM
0.29
0.03
0.14
0.01
0.08
0.01
ALL
0.44
0.11
0.20
0.05
0.13
0.03
RMIA (M 64)
Head-only
0.11
0.07
0.06
0.04
0.03
0.03
PiLM
0.23
0.06
0.01
0.01
256
MI-Loaks 1)
Head-only
0.11
0.01
0.01
0.00
0.00
0.00
FILM
0.11
0.01
0.01
0.00
0.00
0.00
ALL
0.15
0.02
0.02
0.00
0.00
0.00
Trajectory-MIA (M 3)
Head-only
0.17
0.18
0.06
0.10
0.02
0.04
PIL.
0.16
0.04
0.04
0.02
0.02
0.01
ALL
0.12
0.01
0.01
0.01
0.00
0.00
Head-only
0.01
0.00
Attack-P
FILM
0.01
0.00
0.00
0.00
ALL
0.17
0.02
0.01
0.00
0.00
0.00
Head-only
0.19
0.06
0.02
0.02
0.01
0.00
QMIA
FILM
0.13
0.00
0.02
0.00
0.00
0.00
ALL
0.17
0.03
0.04
0.03
0.01
0.01
LiRA (M = 64)
Head-only
0.17
0.01
0.05
0.01
0.03
0.01
FILM
0.01
0.01
0.01
0.04
0.01
0.01
RMIA (M 64)
Head-only
0.12
0.10
0.04
0.01
0.02
0.01
PILM
0.17
0.01
0.06
0.00
0.03
0.01
ALL
0.23
0.04
0.13
0.03
0.05
0.00
Continued on next
27

---

Published in Transactions on Machine Learning Research (10/2025)
Table A5 continued from previous page
TPR@FPR=0.1
TPROFPR=0.01 IPR9FPR=0.001
Shots
Attack
Parameterization
Medlan
IQR
Medlan
IQR
Medlan
IQR
Head-only
MI-Loals (M 1)
FILM
0.10
0.01
0.01
0.00
0.00
0.00
ALL
0.11
0.02
0.01
0.00
0.00
0.00
Irajectory-MIA (M =3)
Head-only
0.16
0.01
0.05
0.01
0.02
0.01
S 1021
FILM
0.12
0.01
0.02
0.00
0.00
0.00
ALL
0.15
0.06
0.03
0.02
0.01
0.01
Head-only
0.10
0.00
0.01
0.00
0.00
0.00
Attack-P
PiLM
ALL
0.01
0.00
0.00
0.00
Head-only
0.13
0.02
0.03
0.01
0.00
0.00
QMIA
FILM
0.12
0.01
0.01
0.00
0.00
0.00
ALL
0.22
0.09
0.12
0.06
0.06
0.03
A.3.6 Additional Effect of Number of Shadow Models
Table A6 provides tabular results investigating the relationship between M (the number of shadow models) and MIA efficacy for LiRA and RMIA. The experimental setup employs each of M + 1 trained models as the larget with the remaining M models serving as shadow models, allowing for a robust evaluation of the effect of shadow model scaling on VIT-B/ HIT-B/16 16 models with Head-only fine-tuned on CIFAR-10.
Table A6: MTA efficacy as a function of the number of shadow models (M) for LiRA and RMTA against VIT- B/16 model with Head-only fine-tuned on CIFAR-10. Values represent medians with interquartile ranges (IQRs). For each configuration, train M + 1 models per repeat, using each model as the target while the remaining M serve as shadow models. MIA efficacy is averaged over 5 repeats and across all M + 1 targel models per repeat.
TPRØFPR-0.1
TPR@FPR=0.01
TPRSFPR=00001
Shots
# of Shadow Models
Attack
Median
IQR
Median
IQR
Median
IQR
M
LIRA
0.38
0.12
0.06
0.02
0.05
0.02
RMTA
0.22
0.12
0.06
0.05
0.04
0.05
M
LIRA
0.83
0.13
0.40
0.09
0.39
0.09
RMTA
0.19
0.12
0.03
0.06
0.03
0.06
S 16
LIRA
0.87
0.53
0.11
0.53
0.14
M 32
0.17
0.06
M = 64
LIRA
0.89
0.22
0.51
0.45
0.51
0.45
RMIA
0.12
0.03
0.02
0.14
0.02
0.14
M 128
LIRA
0.90
0.09
0.60
0.11
0.60
0.11
RMIA
0.16
0.08
0.05
0.04
0.06
0.04
LIRA
0.90
0.08
0.59
0.12
0.59
0.12
RMIA
0.15
0.07
0.05
0.04
0.05
0.04
M
0.00
0.29
0.09
0.12
0.06
0.06
M 16
LiRA
0.52
0.05
0.18
0.04
0.08
0.01
RMIA
0.23
0.15
0.11
0.09
0.08
0.07
LIRA
0.55
0.05
0.25
0.01
0.14
0.01
M 32
Continned on next: page
28

---

Published in Transactions on Machine Learning Research (10/2025)
Table A6 continued from previous page
TPROFPR=0.1
TPRØFFR=0.01
TPR3FPR=00001
Shots
# of Shadow Models
Attack
Median
IQR
Medlan
IQR
Median
IQR
IMIA
0.21
0.13
0.11
0.04
0.08
0.04
M 64
LJRA
0.57
0.05
0.24
0.08
0.14
0.00
RMIA
0.27
0.19
0.12
0.12
0.09
0.07
LiRA
0.59
0.01
0.27
0.02
0.17
0.03
RMIA
0.07
M 256
0.03
RMIA
0.18
0.09
0.10
0.05
0.07
0.05
M
LIRA
0.14
0.01
0.01
0.00
0.00
0.00
ItMIA
0.16
0.04
0.06
0.03
0.03
0.02
M
LiRA
0.23
0.04
0.06
0.01
0.02
0.00
RMIA
0.12
0.07
0.06
0.04
0.03
0.03
9 256
LiRA
M 32
0.03
M 64
0.28
0.11
0.04
0.07
RMIA
0.13
0.02
0.07
0.04
0.03
0.03
LIRA
0.28
0.05
0.12
0.03
0.06
0.02
M 128
RMIA
0.10
0.05
0.05
0.03
0.03
0.03
LIRA
0.29
0.05
0.12
0.03
0.07
0.02
RMIA
0.11
0.05
0.05
0.03
0.03
0.03
M
0.01
M 16
LiRA
0.15
0.00
0.03
0.00
0.01
0.00
RMIA
0.10
0.03
0.04
0.00
0.02
0.01
1024
LiRA
0.16
0.01
0.04
0.00
0.02
0.00
M 32
RMTA
0.09
0.05
0.04
0.01
0.02
0.01
LiRA
0.18
0.01
0.05
0.00
0.03
0.01
RMIA
0.12
0.08
0.04
0.00
0.02
0.00
128
0.09
0.00
0.02
M 256
LiRA
0.18
0.01
0.06
0.00
0.03
0.00
RMIA
0.09
0.05
0.04
0.00
0.02
0.01
A.3.7 Additional Effect of Using Data Augmentation During Fine- Sine-Tuning
Table A7 provides tabular results shown Figure examining the effect of data augmentation on MIA
efficacy for ViT-B/16 models fine-tuned on CIFAR- 10. The table compares two augmentation methods (+ Mirror and Shift) across different shot values (S 16, 64, 256, 1024).
29

---

Published in Transactions on Machine Learning Research (10/2025)
Table A7: Impact of data augmentation on MIA efficacy across (shots) for ViT-B/16 models Head-only fine-tuned on CIFAR-10 with data augmentations. We compare 2 augmentation strategies: + Mirror (where original image plus horizontally flipeed copy of it are used train the target model) and Shift (where horizontally flipping and/or #1-pixel shifts applied to the original image). Values represent medians with interquartile ranges (IQRs). Results are averaged over 5 repeats and we use M + 1 target models (M 64) per repeat.
TPR@FPR=0.1
TPR@FPR=0.01
TPR@FPR=0000
Shots
Attack
Augmentation
Median
IQR
Median
IQR
Median
IQR
LiRA (M 64)
No augmentation
0.89
0.03
0.55
0.07
0.55
0.07
+ Mirror
0.89
0.03
0.55
0.07
0.55
0.07
16
+ Shift
0.91
0.03
0.58
0.07
0.58
0.07
RMIA (M 64)
No augmentation
0.28
0.40
0.10
0.23
0.10
0.23
Mirror
0.09
0.20
+ Shift
LIRA 64)
No sugmentation
0.42
0.01
0.15
0.02
0.08
0.00
+ Mirror
0.42
0.04
0.15
0.02
0.08
0.00
64
+ Shift
0.43
0.04
0.16
0.03
0.09
0.01
RMIA (M = 64)
No sugmentation
0.13
0.02
0.04
0.04
0.02
0.04
Mirror
0.12
0.02
0.04
0.03
0.02
0.03
+ Shift
0.13
0.04
0.04
0.02
0.03
LIRA (M 64)
augmentation
0.03
+ Mirror
0.19
0.02
0.06
0.01
0.08
0.01
S 256
+ Shift
0.20
0.02
0.06
0.01
0.03
0.01
RMLA (M = 64)
No augmentation
0.10
0.01
0.03
0.01
0.00
0.00
Mirror
0.09
0.01
0.03
0.01
0.01
0.00
+ Shift
0.10
0.01
0.03
0.01
0.01
0.01
LifLA (M 61)
No augmentation
0.15
0.00
0.04
0.00
0.02
0.00
+
0.02
S 1024
+
0.00
0.04
0.02
0.00
RMTA 64)
No sugmentation
0.12
0.01
0.04
0.00
0.02
0.00
+ Mirror
0.11
0.01
0.04
0.00
0.02
0.00
+ Shift
0.11
0.01
0.04
0.00
0.02
0.00
30


Reviews:
[{"review_id": "Y1kVYnECOz", "reviewer": "Reviewer_3qR7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Solid and comprehensive empirical comparison of popular practical MIAs.\n- The finding on the comparison to white-box attack (IHA) is interesting.\n\nWeaknesses:\n- I found the main contribution of this work to be the solid experiment in comparing MIAs. Yet this contribution is significantly diminished as the authors do not provide their code.\n- The major findings are known to the literature, and the findings also do not seem to have strong relations to transfer learning settings.\n\n### Detail comments\nI feel this is a solid empirical comparison of popular and practical MIA approaches in the literature. I am happy to see the authors include not only classical approaches (i.e., LiRA), but also SOTA methods such as RMIA. The experiment seems comprehensive, and the corresponding design is done nicely. While the major findings are aligned and known to the literature, I still think it would be a nice addition to the literature. \n\nHowever, several drawbacks prevent me from supporting this work. First, I feel the major contribution of this work is its nice experiment design and setup. This contribution is diminished due to the fact that the authors do not provide their code. Another drawback is that the work does not have strong connections to transfer learning settings, which is one main selling point of the work and thus a bit misleading.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors compare various popular practical membership inference attacks, as well as one white-box inverse Hessian attack in the fine-tuning setting. The major results are consistent with existing literature, while they also found different behavior of inverse Hessian attack compared to the black-box MIAs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would suggest the authors consider the following changes.\n- Make the code publicly available. I will be more inclined to support the acceptance of this work if the code is open-sourced. \n- Either clarify what the unique findings are in the transfer learning setting compared to the standard setup. Or the authors should at least explain what should be taken care of when considering the transfer learning setting compared to the standard setup. If none of these can be justified, I would suggest removing the emphasis on transfer learning.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wft1NfdGZI", "reviewer": "Reviewer_82yg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- strengths:\n1. This work investigates a problem with practical implications, as transfer learning is widely deployed in practice. The paper is well-written and easy to follow.\n\n2. This work performs hyper-parameter optimization to identify the optimal set of hyper-parameters to train the target model for a fair comparison.\n\n3. This work provides analysis for the factors that influence MIA efficacy to understand the properties of different attacks.\n\n- weaknesses:\n1. It is nice to have Table 1 to demonstrate the target model access. In addition, it would also helpful to include the data access assumption. For example, according to Suri et al.2024, \"IHA assumes knowledge of all the other n − 1 records in an n-sized dataset\" and therefore presents IHA as privacy auditing tool. It would be helpful to also discuss the data access of these attacks.\n\n2. A key characteristic of transfer learning is the domain shift between pre-training and fine-tuning. It would be helpful to perform the analysis in Section 5.2, 5.3, 5.4 for task with more domain shift such as PatchCamelyon to understand if there is any discrepancy between ID setting (the CIFAR10 in the current manuscript) and OOD setting.\n\n3. For the data augmentation result, the representation learning in pre-training aims to learn the same representation for the same image with different augmentations, This could be the reason that the data augmentations cannot improve the MIAs in transfer learning. On the other hand, this also indicates that different augmentations in transfer learning does not bring additional information gain and might not improve the utility. I therefore have concerns about the discussion in Section 6 \"practitioners can leverage augmentation techniques to improve model utility without substantially compromising its privacy.\" that needs further justification.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the effectiveness of membership inference attacks under transfer learning setting. This work consider score-based MIAs including both shadow-model-based attacks and shadow-model-free attacks. This worked evaluated several MIAs including LiRA, RMIA,  ML-Leaks, Trajectory-MIA, Attack-P, QMIA, and IHA for two transfer learning settings: head-only and FiLM adapters. This work provides the  empirical evaluation of these MIAs on CIFAR10, CIFAR100 and PatchCamelyon datasets, and provides analysis for the factors that influence MIA efficacy including number of shots, FiLM vs Head-only,  number of shadow models. effect of using data augmentation.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address my concerns in weaknesses.", "broader_impact_concerns": "NA.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cJvSk21Dt5", "reviewer": "Reviewer_uhSE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- results are evaluated across a very comprehensive set of MIA\n- confirms prior literature with strong empirical results\n\n**Weaknesses**\n\n- main results only confirm existing knowledge\n- missing full-parameter fine-tuning (see requested changes)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors present extensive evaluations of existing membership inference attacks on transfer learning settings, mainly showcasing that attack performance generally decreases as the fine-tuning dataset size increases. However, the authors also find that this general rule of thumb is not always true, and that there exist attacks/setup configurations where increasing fine-tuning dataset size does not reduce performance (IHA on binary dataset). The authors also find that across the attacks they evaluated, model training choices like head-only, FiLM strategies, and data augmentations do not make a significant difference in attack performance, while attack parameters, like # of shadow models have more affect.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Low-shot model performance clarification**\n\nIn figures 2, 3, 4, it is implied that the authors trained models using at the lowest, only 4 samples per class in the fine-tuning set. Is the fine-tuned model able to learn the fine-tuning datasets with that little amount of data? Can the authors report test scores of their fine-tuned models?\n\n**Missing ROC Curves**\n\nIn section 4, it is indicated that the authors would report ROC curves in addition to their other metrics. However I do not see any ROC curves in the manuscript.\n\n**Full parameter fine-tuning**\n\nA key missing component of this paper is that full-parameter fine-tuning does not appear to be considered. Can the authors explain why? or add results for full-parameter fine-tuning?", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)