Impact of Dataset Properties on Membership Inference Vulnerability of Deep Transfer Learning

## Abstract

Abstract We analyse the relationship between privacy vulnerability and dataset properties, such as examples per class and number of classes, when applying two state-of-the-art membership inference attacks (MIAs) to fine-tuned neural networks.
We derive per-example MIA vulnerability in terms of score distributions and statistics computed from shadow models.
We introduce a simplified model of membership inference and prove that in this model, the logarithm of the difference of true and false positive rates depends linearly on the logarithm of the number of examples per class.
We complement the theoretical analysis with empirical analysis by systematically testing the practical privacy vulnerability of fine-tuning large image classification models and obtain the previously derived power law dependence between the number of examples per class in the data and the MIA vulnerability, as measured by true positive rate of the attack at a low false positive rate.
Finally, we fit a parametric model of the previously derived form to predict true positive rate based on dataset properties and observe good fit for MIA vulnerability on unseen fine-tuning scenarios.

## 1 Introduction

Machine learning models are prone to memorising their training data, which makes them vulnerable to privacy attacks such as membership inference attacks (MIAs; Shokri et al., ; Carlini et al., ) and reconstruction attacks (e.g. Balle et al., ; Nasr et al., ).
Differential privacy (DP; Dwork et al., ) provides protection against these attacks, but strong formal protection often comes at the cost of significant loss of model utility.

Finding the correct balance between making models resistant to attacks while maintaining a high utility is important for many applications. In health, for example, many European countries and soon also the EU within the European Health Data Space have requirements that models trained on health data that are made publicly available must be anonymous, i.e. they must not contain information that can be linked to an identifiable individual. On the other hand, loss of utility of the model due to privacy constraints may compromise the health benefits that might be gained from it.

In this paper, our aim is to theoretically understand and apply two state-of-the-art MIAs systematically, to help understand practical privacy risks when fine-tuning deep-learning-based classifiers without DP protections.
We focus on transfer learning using fine-tuning because this is increasingly used for all practical applications of deep learning.
Our case study focuses on understanding and quantifying factors that influence the vulnerability of non-DP deep transfer learning models to MIA.
Since the earliest MIA methods, there has been evidence that classification models with more classes are more vulnerable to MIA (Shokri et al., ).
Similarly, there is evidence that models trained on fewer samples can be more vulnerable (Chen et al., ; Németh et al., ).
Feldman & Zhang () showed that neural networks trained from scratch can memorise a large fraction of their training data, while the memorisation is greatly reduced for fine-tuning.

In this paper, we theoretically study the relationship between examples per class, which we denote as shots ($S$), and MIA vulnerability (true positive rate tpr at fixed false positive rate fpr) for a simplified model of fine-tuning and derive a power-law relationship in the form

$$ $\displaystyle\log(\textsc{tpr}-\textsc{fpr})=-\beta_{S}\log(S)-\beta_{0}.$ (1) $$

We complement the theoretical analysis with extensive experiments over many datasets with varying sizes in the transfer learning setting for image classification tasks and observe the same power law.

Previously, Tobaben et al. () and Yu et al. () have presented limited studies of how dataset properties affect the MIA vulnerability.
Tobaben et al. () reported how the MIA vulnerability of few-shot
image classification is affected by the number of shots (i.e. the number of examples
available per class).
Yu et al. () studied how the MIA vulnerability of a class in an image classifier trained from scratch depends on the average individual privacy parameters of that class.
Our work significantly expands on both of these by a more systematic study and modelling of dataset properties.

List of contributions
We analyze the MIA vulnerability of deep transfer learning using two state-of-the-art MIAs LiRA (Carlini et al., ) and RMIA (Zarifzadeh et al., ). Both MIAs are based on the assumption that the attacker can train so-called shadow models (Shokri et al., ) which imitate the model under attack. We first analyze both MIAs theoretically by introducing a simplified model of membership inference (Section 3). We support our theoretical findings by an extensive empirical study on the MIA vulnerability of
deep learning models. We focus on a transfer learning setting for image classification tasks, where a large pre-trained neural network is fine-tuned on a
sensitive dataset, and the MIA is performed on the final classifier. We find that the MIA vulnerability has a strong correlation with certain dataset properties (Section 4).

- 1.
Per-example vulnerability: We derive per-example LiRA and RMIA vulnerability (tpr at fixed fpr) in terms of MIA attack score distributions and statistics computed from shadow models (see Section 3.3).
- 2.
Power-law in simplified model of MI: We formulate a simplified model of membership inference to quantitatively relate dataset properties and MIA vulnerability. For this model, we prove a power-law relationship between the per-example LiRA and RMIA vulnerability and the number of examples per class. We then extend the the power-law relationship to the average-case LiRA and RMIA vulnerability (See Section 3.4).
- 3.
Few-shot MIA: We conduct a comprehensive study of MIA vulnerability (tpr at fixed low fpr) in the transfer learning setting for image classification tasks with target models trained using many different datasets with varying sizes and confirm the theoretical power law between the number of examples per class and the vulnerability to MIA (see Figure 1).
- 4.
Regression model:
We utilise our empirical observations to
fit a regression model to predict MIA vulnerability
($\log(\textsc{tpr}-\textsc{fpr})$ at fixed low fpr) based on examples per class ($\log S$) and
number of classes ($\log C$), which follows the functional form of the theoretically derived power-law. We show both very good fit on
the training data as well as good prediction quality
on unseen data from a different feature extractor
and when fine-tuning other parameterisations
(see Figure 4).

## 2 Background

Notation for the properties of the training dataset $\mathcal{\mathcal{D}}$:
1 $C$for the number of classes
2 $S$for shots (examples per class)
3 $\mathcal{|\mathcal{D}|}$for training dataset size ($\mathcal{|\mathcal{D}|}=CS$).
We denote the number of MIA shadow models with $M$.

Membership inference attacks (MIAs) aim to infer whether a particular sample was part of the training set of the targeted model (Shokri et al., ). Thus, they can be used to determine lower bounds on the privacy leakage of models to complement the theoretical upper bounds obtained through differential privacy.

Likelihood Ratio attack (LiRA; Carlini et al., )
While many different MIAs have been proposed (Hu et al., ), in this work we consider the Likelihood Ratio Attack (LiRA). LiRA is a strong attack that assumes an attacker that has black-box access to the attacked model, knows the training data distribution, the training set size, the model architecture, hyperparameters and training algorithm.
Based on this information, the attacker can train so-called shadow models (Shokri et al., ) which imitate the model under attack but for which the attacker knows the training dataset.

LiRA exploits the observation that the loss function value used to train a model is often lower for examples that were part of the training set compared to those that were not.
For a target sample $(x,y_{x})$, LiRA trains the shadow models:
1 with $(x,y_{x})$ as a part of the training set ($(x,y_{x})\in\mathcal{D}$) and
2 without $x$ in the training set ($(x,y_{x})\notin\mathcal{D}$)
.
After training the shadow models, $(x,y_{x})$ is passed through the shadow models, and
based on the losses (or predictions) two Gaussian distributions are formed:
one for the losses of $(x,y_{x})\in\mathcal{D}$ shadow models, and one for the
$(x,y_{x})\notin\mathcal{D}$.
Finally, the attacker computes the loss for the point $x$ using the model under
attack and determines using a likelihood ratio test on the distributions built
from the shadow models whether it is more likely that $(x,y_{x})\in\mathcal{D}$ or $(x,y_{x})\notin\mathcal{D}$. We use an optimization by Carlini et al. () for performing LiRA for
multiple models and points without training a computationally infeasible number
of shadow models. It relies on sampling the shadow datasets in a way that each sample is in expectation half of the time included in the training dataset of a shadow model and half of the time not. At attack time each model will be attacked once using all other models as shadow models.

Robust Membership Inference Attack (RMIA; Zarifzadeh et al., ) Recently Zarifzadeh et al. () proposed a new MIA algorithm called RMIA, which aims to improve performance when the number of shadow models is limited. Similar to LiRA, RMIA is based on shadow model training and computing the attack statistics based on a likelihood ratio. The main difference to LiRA is that RMIA does not compute the likelihood ratio based on aggregated IN/OUT statistics, but instead compares the target data point against random samples $(z,y_{z})$ from the target data distribution. After computing the likelihood ratios over multiple $(z,y_{z})$ values, the MIA score is estimated as a proportion of the ratios exceeding a preset bound. This approach makes RMIA a more effective attack when the number of shadow models is low.

Measuring MIA vulnerability Using the chosen MIA score of our attack,
we can build a binary classifier to predict whether a sample belongs to the
training data or not. The accuracy profile of such classifier
can be used to measure the success of the MIA. More specifically, throughout
the rest of the paper, we will use the true positive rate (tpr) at a
specific false positive rate (fpr) as a measure for the vulnerability.
Identifying even a small number of examples with high confidence is considered
harmful (Carlini et al., ) and thus we focus on the regions of small fpr.

## 3 Theoretical analysis

In this section, we seek to theoretically understand the impact of the dataset properties on the MIA vulnerability.
It is known that different data points exhibit different levels of MIA vulnerability depending on the underlying distribution (e.g.  Aerni et al., ; Leemann et al., ).
Therefore, we start with analysing *per-example* vulnerabilities for LiRA and RMIA.
In order to quantitatively relate dataset properties to these vulnerabilities, a simplified model is formulated.
Within this model, we prove a power-law between the per-example vulnerability and the number $S$ of examples per class.
Finally, the per-example power-law is analytically extended to *average-case* MIA vulnerability, for which we provide empirical evidence in Section 4.

### 3.1 Preliminaries

First, let us restate the MIA score from LiRA as defined by Carlini et al. (). Denoting the logit of a target model $\mathcal{M}$ applied on a target data point $(x,y_{x})$ as $\ell(\mathcal{M}(x),y_{x})$, the LiRA computes the MIA score as the likelihood ratio

$$ $\displaystyle\mathrm{LR}(x)=\frac{p(\ell(\mathcal{M}(x),y_{x})\mid\mathbb{Q}_{ \text{in}}(x,y_{x}))}{p(\ell(\mathcal{M}(x),y_{x})\mid\mathbb{Q}_{\text{out}}( x,y_{x}))},$ (2) $$

where the $\mathbb{Q}_{\text{in}/\text{out}}$ denote the hypotheses that $(x,y_{x})$ was or was not in the training set of $\mathcal{M}$. Carlini et al. () approximate the IN/OUT hypotheses as normal distributions. Denoting $t_{x}=\ell(\mathcal{M}(x),y_{x})$, the score becomes

$$ $\displaystyle\mathrm{LR}(x)=\frac{\mathcal{N}(t_{x};\hat{\mu}_{\mathrm{in}}(x) ,\hat{\sigma}_{\mathrm{in}}^{2}(x))}{\mathcal{N}(t_{x};\hat{\mu}_{\mathrm{out} }(x),\hat{\sigma}_{\mathrm{out}}^{2}(x))},$ (3) $$

where the $\hat{\mu}_{\text{in/out}}(x)$ and $\hat{\sigma}_{\text{in/out}}(x)$ are the means and standard deviations for the IN/OUT shadow model losses for $(x,y_{x})$. Larger values of $\mathrm{LR}(x)$ suggest that $(x,y_{x})$ is more likely in the training set and vice versa. Now, to build a classifier from this score, the LiRA tests if $\mathrm{LR}(x)>\beta$ for some threshold $\beta$.

Next, let us restate how RMIA (Zarifzadeh et al., ) builds the MIA score. RMIA augments the likelihood-ratio with a sample from the target data distribution to calibrate how likely you would obtain the target model if $(x,y_{x})$ is replaced with another sample $(z,y_{z})$. Denoting the target model parameters with $\theta$, RMIA computes

$$ $\displaystyle\mathrm{LR}(x,z)=\frac{p(\theta\mid x,y_{x})}{p(\theta\mid z,y_{z })},$ (4) $$

and the corresponding MIA score is given as

$$ $\displaystyle\text{Score}_{\text{RMIA}}(x)=\Pr_{(z,y_{z})\sim\mathbb{D}}( \mathrm{LR}(x,z)>\gamma),$ (5) $$

where $\mathbb{D}$ denotes the training data distribution.
Similar to LiRA, the classifier is built by checking if $\text{Score}_{\text{RMIA}}(x)>\beta$. In the following, we will use the direct computation of likelihood-ratio as described in Equation 11 of Zarifzadeh et al. () which approximates $\mathrm{LR}(x,z)$ using normal distributions.

### 3.2 Computing the tpr for LiRA and RMIA

Using the LiRA formulation of Equation 3, the tpr for the target point $(x,y_{x})$ for LiRA is defined as

$$ $\textsc{tpr}_{\mathrm{LiRA}}(x)=\Pr_{\mathcal{D}_{\mathrm{target}}\sim{\mathbb {D}}^{|\mathcal{D}|},\phi^{M}}\left(\frac{\mathcal{N}(t_{x};\hat{\mu}_{\mathrm {in}}(x),\hat{\sigma}_{\mathrm{in}}(x)^{2})}{\mathcal{N}(t_{x};\hat{\mu}_{ \mathrm{out}}(x),\hat{\sigma}_{\mathrm{out}}(x)^{2})}\geq\beta\mid(x,y_{x})\in \mathcal{D}_{\mathrm{target}}\right),$ (6) $$

where $\beta$ is a threshold that defines a rejection region of the likelihood ratio test, $\hat{\mu}_{\mathrm{in}}(x),\hat{\mu}_{\mathrm{out}}(x),\hat{\sigma}_{\mathrm{
in}}(x)$ and $\hat{\sigma}_{\mathrm{out}}(x)$ are LiRA statistics computed from shadow models, and $\phi^{M}$ denotes the randomness in shadow set sampling and shadow model training (see Appendix A for derivation).

For theoretical analysis of RMIA, we focus on the direct approach that is an approximation of the efficient Bayesian approach, as Zarifzadeh et al. () empirically demonstrates that these approaches exhibit similar performances.
Let $\hat{\mu}_{a,b}$ and $\hat{\sigma}_{a,b}$ denote, respectively, the mean and standard deviation of $t_{b}$ estimated from shadow models, where $a$ denotes which of $(x,y_{x})$ and $(z,y_{z})$ is in the training set.
By Equation 11 in (Zarifzadeh et al., ), the per-example performance for RMIA is given as

$$ $\textsc{tpr}_{\mathrm{RMIA}}(x)=\\ \Pr_{\mathcal{D}_{\mathrm{target}}\sim{\mathbb{D}}^{|\mathcal{D}|},\phi^{M}} \left(\Pr_{(z,y_{z})\sim{\mathbb{D}}}(\mathrm{LR}(x,z)\geq\gamma)\geq\beta\mid (x,y_{x})\in\mathcal{D}_{\mathrm{target}}\wedge(z,y_{z})\notin\mathcal{D}_{ \mathrm{target}}\right)$ (7) $$

$$ $\mathrm{LR}(x,z)=\frac{\mathcal{N}(t_{x};\hat{\mu}_{x,x},\hat{\sigma}_{x,x}^{2 })\mathcal{N}(t_{z};\hat{\mu}_{x,z},\hat{\sigma}_{x,z}^{2})}{\mathcal{N}(t_{x} ;\hat{\mu}_{z,x},\hat{\sigma}_{z,x}^{2})\mathcal{N}(t_{z};\hat{\mu}_{z,z},\hat {\sigma}_{z,z}^{2})},$ (8) $$

where $t_{z}$ is the score on $z$ similar to $t_{x}$ and $\phi^{M}$ denotes the randomness in shadow set sampling and shadow model training (see Appendix A for derivation).

We define the average-case tprs for LiRA and RMIA by taking the expectation over the data distribution:

$$ $\displaystyle\overline{\textsc{tpr}}_{\mathrm{LiRA}}=\mathbb{E}_{(x,y_{x})\sim {\mathbb{D}}}[\textsc{tpr}_{\mathrm{LiRA}}(x)]$ (9) $\displaystyle\overline{\textsc{tpr}}_{\mathrm{RMIA}}=\mathbb{E}_{(x,y_{x})\sim {\mathbb{D}}}[\textsc{tpr}_{\mathrm{LiRA}}(x)]$ (10) $$

### 3.3 Per-example MIA vulnerability

Although LiRA models $t_{x}$ by a normal distribution, we consider a more general case where the true distribution of $t_{x}$ is of the location-scale family.
That is,

$$ $t_{x}=\begin{cases}\mu_{\mathrm{in}}(x)+\sigma_{\mathrm{in}}(x)t&\text{ if }(x ,y_{x})\in\mathcal{D}_{\mathrm{target}}\\ \mu_{\mathrm{out}}(x)+\sigma_{\mathrm{out}}(x)t&\text{ if }(x,y_{x})\notin \mathcal{D}_{\mathrm{target}},\end{cases}$ (11) $$

where $t$ has the standard location and unit scale, and $\mu_{\mathrm{in}}(x),\mu_{\mathrm{out}}(x)$ and $\sigma_{\mathrm{in}}(x),\sigma_{\mathrm{out}}(x)$ are the locations and scales of IN/OUT distributions of $t_{x}$.
We assume that the target and shadow datasets have a sufficient number of examples.
This allows us to also assume that $\hat{\sigma}(x)=\hat{\sigma}_{\mathrm{in}}(x)=\hat{\sigma}_{\mathrm{out}}(x)$ and $\sigma(x)=\sigma_{\mathrm{in}}(x)=\sigma_{\mathrm{out}}(x)$, where $\hat{\sigma}(x)$ is the standard deviation of $t_{x}$ estimated from shadow models and $\sigma(x)$ is the true scale parameter of $t_{x}$. (See Appendix B for the validity of these assumptions).
The following result reduces the LiRA vulnerability to the location and scale parameters of $t_{x}$.

###### Lemma 1 (Per-example LiRA vulnerability) .

Suppose that the true distribution of $t_{x}$ is of location-scale family with locations $\mu_{\mathrm{in}}(x),\mu_{\mathrm{out}}(x)$ and scale $\sigma(x)$, and that LiRA models $t_{x}$ by $\mathcal{N}(\hat{\mu}_{\mathrm{in}}(x),\hat{\sigma}(x))$ and $\mathcal{N}(\hat{\mu}_{\mathrm{out}}(x),\hat{\sigma}(x))$.
Assume that an attacker has access to the underlying distribution ${\mathbb{D}}$.
Then for a large enough number of examples per class and infinitely many shadow models, the LiRA vulnerability of a fixed target example is

$$ $\textsc{tpr}_{\mathrm{LiRA}}(x)=\begin{cases}1-F_{t}\left(F_{t}^{-1}(1-\textsc {fpr}_{\mathrm{LiRA}}(x))-\frac{\mu_{\mathrm{in}}(x)-\mu_{\mathrm{out}}(x)}{ \sigma(x)}\right)&\text{ if }\hat{\mu}_{\mathrm{in}}(x)>\hat{\mu}_{\mathrm{out }}(x)\\ F_{t}\left(F_{t}^{-1}(\textsc{fpr}_{\mathrm{LiRA}}(x))-\frac{\mu_{\mathrm{in}} (x)-\mu_{\mathrm{out}}(x)}{\sigma(x)}\right)&\text{ if }\hat{\mu}_{\mathrm{in} }(x)<\hat{\mu}_{\mathrm{out}}(x),\end{cases}$ (12) $$

where $F_{t}$ is the cdf of $t$ with the standard location and unit scale.

###### Proof.

See Section C.1.
∎

Here we assume that an attacker trains shadow models with the true underlying distribution.
However, in real-world settings the precise underlying distribution may not be available for an attacker.
We relax this assumption in Appendix B so that the attacker only needs an approximated underlying distribution for the optimal LiRA as in Lemma 1.

Next we focus on the per-example RMIA performance.
As in the case of LiRA, we assume that $t_{x}$ and $t_{z}$ follow distributions of the location-scale family.
We have

$$ $\displaystyle t_{x}$ $\displaystyle=\begin{cases}\mu_{x,x}+\sigma_{x,x}t&\text{ if }(x,y_{x})\in \mathcal{D}_{\mathrm{target}}\wedge(z,y_{z})\notin\mathcal{D}_{\mathrm{target} }\\ \mu_{z,x}+\sigma_{z,x}t&\text{ if }(x,y_{x})\notin\mathcal{D}_{\mathrm{target} }\wedge(z,y_{z})\in\mathcal{D}_{\mathrm{target}}\end{cases}$ (13) $\displaystyle t_{z}$ $\displaystyle=\begin{cases}\mu_{x,z}+\sigma_{x,z}t&\text{ if }(x,y_{x})\in \mathcal{D}_{\mathrm{target}}\wedge(z,y_{z})\notin\mathcal{D}_{\mathrm{target} }\\ \mu_{z,z}+\sigma_{z,z}t&\text{ if }(x,y_{x})\notin\mathcal{D}_{\mathrm{target} }\wedge(z,y_{z})\in\mathcal{D}_{\mathrm{target}}.\end{cases}$ (14) $$

It is important to note that $\mu_{a,b}$ and $\sigma_{a,b}$ denote, respectively, a location and a scale, while previously defined $\hat{\mu}_{a,b}$ and $\hat{\sigma}_{a,b}$ are, respectively, a mean and a standard deviation.
As for the analysis of LiRA, we assume that the target and shadow sets have a sufficient number of examples per class, and that $\sigma_{x}=\sigma_{x,x}=\sigma_{z,x}$, $\sigma_{z}=\sigma_{x,z}=\sigma_{z,z}$, $\hat{\sigma}_{x}=\hat{\sigma}_{x,x}=\hat{\sigma}_{z,x}$ and $\hat{\sigma}_{z}=\hat{\sigma}_{x,z}=\hat{\sigma}_{z,z}$, where $\sigma_{x}$ and $\sigma_{z}$ are, respectively, the true scales of $t_{x}$ and $t_{z}$, and $\hat{\sigma}_{x}$ and $\hat{\sigma}_{z}$ are, respectively, standard deviations of $t_{x}$ and $t_{z}$ estimated from shadow models (see Appendix B for the validity of these assumptions).

###### Lemma 2 (Per-example RMIA vulnerability) .

Suppose that the true distributions of $t_{x}$ and $t_{z}$ are of location-scale family with locations $\mu_{x,x},\mu_{z,x},\mu_{x,z},\mu_{z,z}$ and scales $\sigma_{x},\sigma_{z}$, and that RMIA models $t_{x}$ and $t_{z}$ by normal distributions with parameters computed from shadow models.
For a large enough number of examples per class and infinitely many shadow models, the RMIA vulnerability of a fixed target example is bounded by

$$ $\textsc{tpr}_{\mathrm{RMIA}}(x)\leq\begin{cases}1-F_{t}\left(F_{t}^{-1}(1- \alpha)-\frac{\mathbb{E}_{(z,y_{z})\sim{\mathbb{D}}}[q]}{\mathbb{E}_{(z,y_{z}) \sim{\mathbb{D}}}[A]}\right)&\text{ if }\ \mathbb{E}_{(z,y_{z})\sim{\mathbb{D} }}[A]>0\\ F_{t}\left(F_{t}^{-1}(\alpha)-\frac{\mathbb{E}_{(z,y_{z})\sim{\mathbb{D}}}[q]} {\mathbb{E}_{(z,y_{z})\sim{\mathbb{D}}}[A]}\right)&\text{ if }\ \mathbb{E}_{(z ,y_{z})\sim{\mathbb{D}}}[A]<0,\end{cases}$ (15) $$

for some constant $\alpha\geq\textsc{fpr}_{\mathrm{RMIA}}(x)$, where

$$ $\displaystyle q$ $\displaystyle=\frac{(\mu_{x,x}-\mu_{z,x})(\hat{\mu}_{x,x}-\hat{\mu}_{z,x})}{ \hat{\sigma}_{x}^{2}}-\frac{(\mu_{x,z}-\mu_{z,z})(\hat{\mu}_{x,z}-\hat{\mu}_{z ,z})}{\hat{\sigma}_{z}^{2}}$ (16) $\displaystyle A$ $\displaystyle=\frac{\sigma_{x}}{\hat{\sigma}_{x}^{2}}(\hat{\mu}_{x,x}-\hat{\mu }_{z,x})+\frac{\sigma_{z}}{\hat{\sigma}_{z}^{2}}(\hat{\mu}_{x,z}-\hat{\mu}_{z, z}).$ (17) $$

###### Proof.

See Section C.2.
∎

Note that here we must assume that the attacker has access to the underlying distribution for the optimal RMIA as the Equations 16 and 17 depend on the parameters computed from shadow models.

###### Lemma 1 (Per-example LiRA vulnerability) .

###### Proof.

###### Lemma 2 (Per-example RMIA vulnerability) .

###### Proof.

### 3.4 A simplified model of membership inference

Now we construct a simplified model of membership inference that streamlines the data generation and shadow model training.

We sample vectors on a high dimensional unit sphere and classify them based on inner product with estimated class mean. This model is easier to analyse theoretically than real-world deep learning examples.
We generate the data and form the classifiers (which are our target models) as follows:

- 1.
For each class, we first sample a true class mean $m_{c}$ on a high dimensional unit sphere that is orthogonal to all other true class means ($\forall i,j\in\{1,\dots,C\}:m_{i}\perp m_{j}\lor i=j$).
- 2.
We sample $2S$ vectors $x_{c}$ for each class. We assume that they are Gaussian distributed around the the true class mean $x_{c}\sim\mathcal{N}(m_{c},s^{2}I)$ where the $s^{2}$ is the in-class variance.
- 3.
For each ”target model” we randomly choose a subset of size $CS$ from all generated vectors and compute per-class means $r_{c}$.
- 4.
The computed mean is used to classify sample $x$ by computing
the inner product $\langle x,r_{c}\rangle$ as a metric of similarity.

The attacker has to infer which vectors have been used for training the classifier. Instead of utilising the logits (like in many image classification tasks), the attacker can use the inner products of a point with the cluster means.

This simplified model resembles a linear (Head) classifier often used in transfer learning when adapting to a new dataset. We also focus on the linear (Head) classifier in our empirical evaluation in Section 4. In the
linear classifier, we find a matrix $W$ and biases $b$, to optimize
the cross-entropy between the labels and logits $Wv+b$, where $v$
denotes the feature space representation of the data. In the
simplified model, the rows of $W$ are replaced by the cluster means
and we do not include the bias term in the classification.

Now, applying Lemma 1 to the simplified model yields the following result.

###### Theorem 3 (Per-example LiRA power-law) .

Fix a target example $(x,y_{x})$. For the simplified model with arbitrary $C$ and infinitely many shadow models, the per-example LiRA vulnerability is given as

$$ $\textsc{tpr}_{\mathrm{LiRA}}(x)=\Phi\left(\Phi^{-1}(\textsc{fpr}_{\mathrm{LiRA }}(x))+\frac{|\langle x,x-m_{x}\rangle|}{\sqrt{S}s||x||}\right),$ (18) $$

where $m_{x}$ is the true mean of class $y_{x}$. In addition, for large $S$ we have

$$ $\log(\textsc{tpr}_{\mathrm{LiRA}}(x)-\textsc{fpr}_{\mathrm{LiRA}}(x))\approx- \frac{1}{2}\log S-\frac{1}{2}\Phi^{-1}(\textsc{fpr}_{\mathrm{LiRA}}(x))^{2}+ \log\frac{|\langle x,x-m_{x}\rangle|}{||x||s\sqrt{2\pi}}.$ (19) $$

###### Proof.

See Section C.3.
∎

An immediate upper bound is obtained from Theorem 3 by the Cauchy-Schwarz inequality:

$$ $\log(\textsc{tpr}_{\mathrm{LiRA}}(x)-\textsc{fpr}_{\mathrm{LiRA}}(x))\leq- \frac{1}{2}\log S-\frac{1}{2}\Phi^{-1}(\textsc{fpr}_{\mathrm{LiRA}}(x))^{2}+ \log\frac{||x-m_{x}||}{s\sqrt{2\pi}}.$ (20) $$

This implies that if $||x-m_{x}||$ is bounded, then the worst-case vulnerability is also bounded.
Hence we can have the LiRA vulnerability arbitrarily close to zero in this non-DP setting by simply increasing $S$.
Similarly, employing Lemma 2 and the simplified model, we obtain the following upper bound for RMIA performance.

###### Theorem 4 (Per-example RMIA power-law) .

Fix a target example $(x,y_{x})$. For the simplified model with infinitely many shadow models, the per-example RMIA vulnerability is given as

$$ $\textsc{tpr}_{\mathrm{RMIA}}(x)\leq 1-F_{|t|}\left(F_{|t|}^{-1}(1-\alpha)- \frac{\psi(x,C)}{\sqrt{S}s}\right),$ (21) $$

where $\alpha\geq\textsc{fpr}_{\mathrm{RMIA}}(x)$ and

$$ $\psi(x,C)=\\ \frac{\mathbb{E}_{(z,y_{z})\sim{\mathbb{D}}}\left[2||x-z||^{2}\mid y_{z}=y_{x} \right]+(C-1)\mathbb{E}_{(z,y_{z})\sim{\mathbb{D}}}\left[(||x-m_{x}||^{2}+||z- m_{z}||^{2})\mid y_{z}\not=y_{x}\right]}{\mathbb{E}_{(z,y_{z})\sim{\mathbb{D}} }\left[2||x-z||\mid y_{z}=y_{x}\right]+(C-1)\mathbb{E}_{(z,y_{z})\sim{\mathbb{ D}}}\left[(||x-m_{x}||+||z-m_{z}||)\mid y_{z}\not=y_{x}\right]}.$ (22) $$

In addition, for large $S$ we have

$$ $\log(\textsc{tpr}_{\mathrm{RMIA}}(x)-\textsc{fpr}_{\mathrm{RMIA}}(x))\leq- \frac{1}{2}\log S-\frac{1}{2}\Phi^{-1}(\alpha)^{2}+\log\frac{\psi(x,C)}{\sqrt{ \pi/2}}.$ (23) $$

###### Proof.

See Section C.4.
∎

As for the LiRA power-law, bounding $||x-z||,||x-m_{x}||$ and $||z-m_{z}||$ will provide a worst-case upper bound for which the power-law holds.
Now the following corollaries extend the power-law to the average-case MIA vulnerabilities.
We will also empirically validate these results in Section 4.

###### Corollary 5 (Average-case LiRA power-law) .

For the simplified model with arbitrary $C$, sufficiently large $S$ and infinitely many shadow models, we have

$$ $\log(\overline{\textsc{tpr}}_{\mathrm{LiRA}}-\overline{\textsc{fpr}}_{\mathrm{ LiRA}})\approx-\frac{1}{2}\log S-\frac{1}{2}\Phi^{-1}(\overline{\textsc{fpr}}_ {\mathrm{LiRA}})^{2}+\log\left(\mathbb{E}_{(x,y_{x})\sim{\mathbb{D}}}\left[ \frac{\langle x,x-m_{x}\rangle}{\sqrt{2\pi}||x||s}\right]\right).$ (24) $$

###### Proof.

See Section C.5.
∎

###### Corollary 6 (Average-case RMIA power-law) .

For the simplified model with sufficiently large $S$ and infinitely many shadow models, we have

$$ $\log(\overline{\textsc{tpr}}_{\mathrm{RMIA}}-\overline{\textsc{fpr}}_{\mathrm{ RMIA}})\leq-\frac{1}{2}\log S-\frac{1}{2}\Phi^{-1}(\alpha)^{2}+\log\left( \mathbb{E}_{(x,y_{x})\sim{\mathbb{D}}}\left[\frac{\psi(x,C)}{\sqrt{2\pi}} \right]\right).$ (25) $$

###### Proof.

See Section C.6.
∎

###### Theorem 3 (Per-example LiRA power-law) .

###### Proof.

###### Theorem 4 (Per-example RMIA power-law) .

###### Proof.

###### Corollary 5 (Average-case LiRA power-law) .

###### Proof.

###### Corollary 6 (Average-case RMIA power-law) .

###### Proof.

## 4 Empirical evaluation of MIA vulnerability and dataset properties

In this section, we investigate how different properties of datasets affect the MIA vulnerability. Based on our observations, we propose a method to predict the vulnerability to MIA using these properties.

### 4.1 Experimental setup

We focus on a image classification setting where we fine-tune pre-trained models on sensitive downstream datasets and assess the MIA vulnerability using LiRA and RMIA with $M=256$ shadow/reference models. We base our experiments on a subset of the few-shot benchmark VTAB (Zhai et al., ) that achieves a test classification accuracy $>80\%$ (see Table A3).

We report results for fine-tuning a last layer classifier (Head) trained on top of a Vision Transformer ViT-Base-16 (ViT-B; Dosovitskiy et al., ), pre-trained on ImageNet-21k (Russakovsky et al., ). The results for using ResNet-50 (R-50; Kolesnikov et al., ) as a backbone can be found in Section F.1. We optimise the hyperparameters (batch size, learning rate and number of epochs) using the library Optuna (Akiba et al., ) with the Tree-structured Parzen Estimator (TPE; Bergstra et al., ) sampler with 20 iterations (more details in Section E.2). We provide the the code for reproducing the experiments in the supplementary material.

Measuring the uncertainty for tpr
The tpr values from the LiRA-based classifier can be seen as maximum
likelihood-estimators for the probability of producing true positives among the
positive samples. Since we have a finite number of samples for our estimation,
it is important to estimate the uncertainty in these estimators. Therefore, when
we report the tpr values for a single repeat of the learning algorithm, we
estimate the stochasticity of the tpr estimate by using Clopper-Pearson
intervals (Clopper & Pearson, ). Given tp true positives
among p positives, the $1-\alpha$ confidence Clopper-Pearson interval for the
tpr is given as

$$ $\begin{split}&B(\alpha/2;\textsc{tp},\textsc{p}-\textsc{tp}+1)<\textsc{tpr}\\ &\textsc{tpr}<B(1-\alpha/2;\textsc{tp}+1,\textsc{p}-\textsc{tp}),\end{split}$ (26) $$

where $B(q;a,b)$ is the $q$th-quantile of $\text{Beta}(a,b)$ distribution.

### 4.2 Experimental results

Using the setting described above, we study how the number of classes and the number of shots affect the vulnerability (tpr at fpr as described in Section 2) using LiRA. We make the following observations:

- •
A larger number of $S$ (shots) decrease the vulnerability in a power law relation as demonstrated in Figure 1(a). We provide further evidence of this in the Appendix (Figures A.2, A4 and A5).
- •
Contrary, a larger number of $C$ (classes) increases the vulnerability as demonstrated in Figure 1(b) with further evidence in Figures A.3, A6 and A7 in the Appendix. However, the trend w.r.t. $C$ is not as clear as with $S$.

Figure: (a) as function of $S$ (shots)
Refer to caption: x1.png

RMIA
In Figure 2 we compare the vulnerability of the models to LiRA and RMIA as a function of the number of $S$ (shots) at $\textsc{fpr}=0.1$. We observe the power-law for both attacks, but the RMIA is more unstable than LiRA (especially for lower fpr). More results for RMIA are in Figures A.6, A.7 and A.8 in the Appendix.

Figure: Figure 2: LiRA and RMIA vulnerability ($(\textsc{tpr}-\textsc{fpr})$ at $\textsc{fpr}=0.1$) as a function of shots ($S$) when attacking a ViT-B Head fine-tuned without DP on different datasets. For better visibility, we split the datasets into two panels. We observe the power-law for both attacks, but the RMIA is more unstable than LiRA. The lines display the median over six seeds.
Refer to caption: x3.png

### 4.3 Model to predict dataset vulnerability

The trends seen in Figure 1 suggest the same power law relationship that we derived for the simplified model of membership inference in Section 3. We fit a linear regression model to predict $\log(\textsc{tpr}-\textsc{fpr})$ for each $\textsc{fpr}=10^{-k},k=1,\dots,5$ separately using the $\log C$ and $\log S$ as covariates with statsmodels (Seabold & Perktold, ). The general form of the model can be found in Equation 27, where $\beta_{S},\beta_{C}$ and $\beta_{0}$ are the learnable regression parameters.

$$ $\log_{10}(\textsc{tpr}-\textsc{fpr})=\beta_{S}\log_{10}(S)+\beta_{C}\log_{10}( C)+\beta_{0}$ (27) $$

In Section F.2, we propose a variation of the regression model that predicts $\log_{10}(\textsc{tpr})$ instead of $\log_{10}(\textsc{tpr}-\textsc{fpr})$ but this alternative model performs worse on our empirical data and predicts $\textsc{tpr}<\textsc{fpr}$ in the tail when $S$ is very large.

We utilise MIA results of ViT-B (Head) (see Table A4) as the training data. Based on the $R^{2}$ (coefficient of determination) score ($R^{2}=0.930$ for the model trained on $\textsc{fpr}=0.001$ data), our model fits the data extremely well. We provide further evidence for other fpr in Figure A.4 and Table A9 in the Appendix. Figure 3 shows the parameters of the prediction model fitted to the training data. For larger fpr, the coefficient $\beta_{S}$ is around $-0.5$, as our theoretical analysis predicts. However, the coefficient value decreases for small fpr. This is perhaps because the power-law in Equation (24) only holds for large $S$, and for small fpr Equation (24) significantly underestimates the vulnerability in small-$S$ regime (see Appendix D).

Figure: Figure 3: Coefficient values for different fpr when fitting a regression model based on Equation 27 fitted on data from ViT-B (Head) with LiRA (Table A4). The error bars display the 95% confidence intervals based on Student’s t-distribution.
Refer to caption: x4.png

##### Prediction quality on other MIA target models

We analyse how the regression model trained on the ViT-B (Head) data generalizes to other target models. The main points are:

- •
R-50 (Head): Figure 4(a) shows that the regression model is robust to a change of the feature extractor, as it is able to predict the tpr for R-50 (Head) (test $R^{2}=0.790$).
- •
R-50 (FiLM): Figure 4(b) shows that the prediction quality is good for R-50 (FiLM) models. These models are fine-tuned with parameter-efficient FiLM (Perez et al., ) layers (See Section E.1). Tobaben et al. () demonstrated that FiLM layers are a competitive alternative to training all parameters. We supplement the MIA results of Tobaben et al. () with own FiLM training runs. Refer to Table A8 in the Appendix.
- •
From-Scratch-Training: Carlini et al. () provide limited results on from-scratch-training. To the best of our knowledge these are the only published LiRA results on image classification models. Figure 4(b) displays that our prediction model underestimates the vulnerability of the from-scratch trained target models. We have identified two potential explanations for this
1 In from-scratch-training all weights of the model need to be trained from the sensitive data and thus potentially from-scratch-training could be more vulnerable than fine-tuning.
2 The strongest attack in Carlini et al. () uses data augmentations to improve the performance. We are not using this optimization.

Figure: (a) Predicted LiRA vulnerability ($(\textsc{tpr}-\textsc{fpr})$ at $\textsc{fpr}=0.001$) as a function of $S$ (shots). The dots show the median tpr for the train set (ViT-B; Table A4) and the test set (R-50; Table A5) over six seeds (datasets: Patch Camelyon, EuroSAT and CIFAR100). The linear model is robust to changing the feature extractor from ViT-B to R-50.
Refer to caption: x5.png

## 5 Discussion

Under the GDPR and similar legal regimes, machine learning (ML) models trained on personal data that memorise the data are personal data and need to be carefully protected.
Our work analyses in which cases trained models would most likely be personal data and in which cases they might not be.
This will help in evaluating the risk of different kinds of models, favouring less risky models when possible and paying extra attention to more risky cases.

As the best means of protecting privacy, differential privacy, reduces the utility of models, it is important to understand when it is necessary.
Our results highlight that models are most vulnerable to MIA when the number of examples per class is low.
This reinforces the well-known unfortunate dilemma: differential privacy seems most important in the regime where it likely reduces the model utility the most.

One major reason for MIA vulnerability can be memorisation of the training data. Feldman & Zhang () experimentally test memorisation in neural network training, and find that according to their definition, a large fraction of the training data are memorised when training from scratch, while only few are when fine-tuning. This is aligned with our results that indicate from scratch training to be more vulnerable than fine-tuning.

Our model singles out the number of examples per class $S$ as the key predictor of model vulnerability.
While our results are based on balanced datasets, it seems likely that minority classes and outliers would be more vulnerable as predicted by our model given their smaller $S$.
Studying the vulnerability in unbalanced data and in the presence of outliers is an important area of future work.

##### Limitations

Our work measures the vulnerability through running two particular MIAs. There is no guarantee that models that are not vulnerable to this attack might not be vulnerable to some stronger attacks. Formal bounds on attack susceptibility would require something like differential privacy. Both our theoretical and empirical analysis focus on deep transfer learning using fine-tuning. Models trained from scratch are likely to be more vulnerable.

## Acknowledgments

This work was supported by the Research Council of Finland (Flagship programme: Finnish
Center for Artificial Intelligence, FCAI, Grant 356499 and Grant 359111), the Strategic Research Council
at the Research Council of Finland (Grant 358247) as well as the European Union (Project
101070617). Views and opinions expressed are however those of the author(s) only and do not
necessarily reflect those of the European Union or the European Commission. Neither the
European Union nor the granting authority can be held responsible for them.
The authors wish to thank the CSC – IT Center for Science, Finland for supporting this project with computational and data storage resources.
We thank Mikko A. Heikkilä and Ossi Räisä for helpful comments and suggestions and John F. Bronskill for helpful discussions regarding few-shot learning.

## Appendix A Formulating LiRA and RMIA for Section 3

Let $\mathcal{M}$ be our target model and $\ell(\mathcal{M}(x),y_{x})$ be the loss of the model on a target example $(x,y_{x})$.
The goal of MIA is to determine whether $(x,y_{x})\in\mathcal{D}_{\mathrm{target}}$.
This can be formulated as a hypothesis test:

$$ $\displaystyle H_{0}:(x,y_{x})\notin\mathcal{D}_{\mathrm{target}}$ (A1) $\displaystyle H_{1}:(x,y_{x})\in\mathcal{D}_{\mathrm{target}}.$ (A2) $$

### A.1 LiRA

Following (Carlini et al., ), we formulate the Likelihood Ratio Attack (LiRA).
LiRA exploits the difference of losses on the target model under $H_{0}$ and $H_{1}$.
To model the IN/OUT loss distributions with few shadow models, LiRA employs a parametric modelling.
Particularly, LiRA models $t_{x}$ by a normal distribution.
That is, the hypothesis test formulated above can be rewritten as

$$ $\displaystyle H_{0}:t_{x}\sim\mathcal{N}(\hat{\mu}_{\mathrm{out}}(x),\hat{ \sigma}_{\mathrm{out}}(x)^{2})$ (A3) $\displaystyle H_{1}:t_{x}\sim\mathcal{N}(\hat{\mu}_{\mathrm{in}}(x),\hat{ \sigma}_{\mathrm{in}}(x)^{2}).$ (A4) $$

The likelihood ratio is now

$$ $\mathrm{LR}(x)=\frac{\mathcal{N}(t_{x};\hat{\mu}_{\mathrm{in}}(x)^{2},\hat{ \sigma}_{\mathrm{in}}(x)^{2})}{\mathcal{N}(t_{x};\hat{\mu}_{\mathrm{out}}(x), \hat{\sigma}_{\mathrm{out}}(x)^{2})}.$ (A5) $$

LiRA rejects $H_{0}$ if and only if

$$ $\mathrm{LR}(x)\geq\beta,$ (A6) $$

concluding that $H_{1}$ is true, i.e., identifying the membership of $(x,y_{x})$.
Thus, the true positive rate of this hypothesis test given as

$$ $\textsc{tpr}_{\mathrm{LiRA}}(x)=\Pr_{\mathcal{D}_{\mathrm{target}}\sim{\mathbb {D}}^{|\mathcal{D}|},\phi^{M}}\left(\frac{\mathcal{N}(t_{x};\hat{\mu}_{\mathrm {in}}(x),\hat{\sigma}_{\mathrm{in}}(x)^{2})}{\mathcal{N}(t_{x};\hat{\mu}_{ \mathrm{out}}(x),\hat{\sigma}_{\mathrm{out}}(x)^{2})}\geq\beta\mid(x,y_{x})\in \mathcal{D}_{\mathrm{target}}\right),$ (A7) $$

where $\phi^{M}$ denotes the randomness in the shadow set sampling and shadow model training.

### A.2 RMIA

By modelling $t_{z}$ by a normal distribution, Zarifzadeh et al. () approximate the *pairwise* likelihood ratio as

$$ $\mathrm{LR}(x,z)=\frac{p(\theta\mid x,y_{x})}{p(\theta\mid z,y_{z})}\approx \frac{\mathcal{N}(t_{x};\hat{\mu}_{x,x},\hat{\sigma}_{x,x}^{2})\mathcal{N}(t_{ z};\hat{\mu}_{x,z},\hat{\sigma}_{x,z}^{2})}{\mathcal{N}(t_{x};\hat{\mu}_{z,x}, \hat{\sigma}_{z,x}^{2})\mathcal{N}(t_{z};\hat{\mu}_{z,z},\hat{\sigma}_{z,z}^{2 })},$ (A8) $$

where $\hat{\mu}_{a,b}$ and $\hat{\sigma}_{a,b}$ are, respectively, the mean and standard deviation of $t_{b}$ estimated from shadow models when the training set contains $a$ but not $b$.
Then RMIA exploits the probability of rejecting the pairwise likelihood ratio test over $(z,y_{z})\sim{\mathbb{D}}$:

$$ $\mathrm{Score}_{\mathrm{RMIA}}(x)=\Pr_{(z,y_{z})\sim{\mathbb{D}}}\left(\mathrm {LR}(x,z)\geq\gamma\right).$ (A9) $$

Thus, RMIA rejects $H_{0}$ if and only if

$$ $\Pr_{(z,y_{z})\sim{\mathbb{D}}}\left(\mathrm{LR}(x,z)\geq\gamma\right)\geq\beta,$ (A10) $$

identifying the membership of $x$.
Hence the true positive rate of RMIA is given as

$$ $\textsc{tpr}_{\mathrm{RMIA}}(x)=\Pr_{\mathcal{D}_{\mathrm{target}}\sim{\mathbb {D}}^{|\mathcal{D}|},\phi^{M}}\left(\Pr_{(z,y_{z})\sim{\mathbb{D}}}\left( \mathrm{LR}(x,z)\geq\gamma\right)\geq\beta\mid(x,y_{x})\in\mathcal{D}_{\mathrm {target}}\wedge(x,y_{x})\notin\mathcal{D}_{\mathrm{target}}\right),$ (A11) $$

where $\phi^{M}$ denotes the randomness in the shadow set sampling and shadow model training.

## Appendix B On the assumptions in Section 3

### B.1 The assumption of shared variances

In Section 3 we assumed that for LiRA $\sigma_{\mathrm{in}}=\sigma_{\mathrm{out}}$ and $\hat{\sigma}_{\mathrm{in}}=\hat{\sigma}_{\mathrm{out}}$, and that for RMIA $\sigma_{x}=\sigma_{x,x}=\sigma_{z,x}$, $\sigma_{z}=\sigma_{x.z}=\sigma_{z,z}$, $\hat{\sigma}_{x}=\hat{\sigma}_{x,x}=\hat{\sigma}_{z,x}$ and $\hat{\sigma}_{z}=\hat{\sigma}_{x.z}=\hat{\sigma}_{z,z}$.
Utilising the simplified model formulated in Section 3.4, we show that for large enough number $S$ of examples per class these assumptions are reasonable.

From the proof of Theorem 3 (see Section C.3), the LiRA scores are

$$ $\displaystyle s_{y_{x}}^{\mathrm{(in)}}$ $\displaystyle=\langle x,\frac{1}{S}\left(\sum_{i=1}^{S-1}x_{y_{x},i}+x\right) \rangle=\langle x,\frac{1}{S}\sum_{i=1}^{S}x_{y_{x},i}\rangle+\langle x,\frac{ 1}{S}(x-x_{y_{x},S})\rangle$ (A12) $\displaystyle s_{y_{x}}^{\mathrm{(out)}}$ $\displaystyle=\langle x,\frac{1}{S}\sum_{i=1}^{S}x_{y_{x},i}\rangle,$ (A13) $$

respectively, when $(x,y_{x})\in\mathcal{D}_{\mathrm{target}}$ and when $(x,y_{x})\notin\mathcal{D}_{\mathrm{target}}$. Then, since for the simplified model scores follow Gaussian distributions, we have

$$ $\displaystyle\sigma_{\mathrm{in}}^{2}$ $\displaystyle=\hat{\sigma}_{\mathrm{in}}^{2}=\operatorname{Var}(s_{y_{x}}^{ \mathrm{(in)}})=\frac{1}{S}\left(1-\frac{1}{S}\right)||x||^{2}s^{2}$ (A14) $\displaystyle\sigma_{\mathrm{out}}^{2}$ $\displaystyle=\hat{\sigma}_{\mathrm{out}}^{2}=\operatorname{Var}(s_{y_{x}}^{ \mathrm{(out)}})=\frac{1}{S}||x||^{2}s^{2}.$ (A15) $$

Thus, the differences $\sigma_{\mathrm{in}}-\sigma_{\mathrm{out}}$ and $\hat{\sigma}_{\mathrm{in}}-\hat{\sigma}_{\mathrm{out}}$ are negligible for large $S$.
Similarly, we have

$$ $\displaystyle\sigma_{x,x}^{2}$ $\displaystyle=\hat{\sigma}_{x,x}^{2}=\operatorname{Var}(s_{y_{x}}^{(x)}(x))= \frac{1}{S}\left(1-\frac{1}{S}\right)||x||^{2}s^{2}$ (A16) $\displaystyle\sigma_{z,x}^{2}$ $\displaystyle=\hat{\sigma}_{z,x}^{2}=\operatorname{Var}(s_{y_{x}}^{(z)}(x))= \begin{cases}\frac{1}{S}\left(1-\frac{1}{S}\right)||x||^{2}s^{2}\quad&\text{ if }y_{x}=y_{z}\\ \frac{1}{S}||x||^{2}s^{2}\quad&\text{ if }y_{x}\not=y_{z}\end{cases}$ (A17) $\displaystyle\sigma_{x,z}^{2}$ $\displaystyle=\hat{\sigma}_{x,z}^{2}=\operatorname{Var}(s_{y_{z}}^{(x)}(z))= \begin{cases}\frac{1}{S}\left(1-\frac{1}{S}\right)||z||^{2}s^{2}\quad&\text{ if }y_{x}=y_{z}\\ \frac{1}{S}||z||^{2}s^{2}\quad&\text{ if }y_{x}\not=y_{z}\end{cases}$ (A18) $\displaystyle\sigma_{z,z}^{2}$ $\displaystyle=\hat{\sigma}_{z,z}^{2}=\operatorname{Var}(s_{y_{z}}^{(z)}(z))= \frac{1}{S}\left(1-\frac{1}{S}\right)||z||^{2}s^{2}.$ (A19) $$

Therefore, the differences $\sigma_{x,x}-\sigma_{z,x}$, $\sigma_{x.z}-\sigma_{z,z}$, $\hat{\sigma}_{x,x}-\hat{\sigma}_{z,x}$ and $\hat{\sigma}_{x.z}-\hat{\sigma}_{z,z}$ are negligible for large enough $S$.
Hence as long as the simplified model approximates classification tasks to which Lemmas 1 and 2 are applied, these assumptions are reasonably justified.

### B.2 Relaxing the assumption of Lemma 1

In Lemma 1 we assume that an attacker has access to the true underlying distribution.
However, this is not necessarily the case in real-world settings.
Noting that the Equation 12 mainly relies on the true location parameters $\mu_{\mathrm{in}}(x),\mu_{\mathrm{out}}(x)$ and scale parameter $\sigma(x)$, we may relax this assumption.

Notice that if we completely drop this assumption so that an attacker trains shadow models with an arbitrary underlying distribution, then we may not be able to choose a desired $\textsc{fpr}_{\mathrm{LiRA}}(x)$.
From Equation A36 in the proof of Lemma 1 we have

$$ $\displaystyle\frac{\hat{\sigma}^{2}\log\beta}{\sigma(\hat{\mu}_{\mathrm{in}}- \hat{\mu}_{\mathrm{out}})}+\frac{\hat{\mu}_{\mathrm{in}}+\hat{\mu}_{\mathrm{ out}}}{2\sigma}-\frac{\mu_{\mathrm{out}}}{\sigma}=\begin{cases}F_{t}^{-1}(1- \textsc{fpr}_{\mathrm{LiRA}}(x))&\text{ if }\hat{\mu}_{\mathrm{in}}>\hat{\mu}_ {\mathrm{out}}\\ F_{t}^{-1}(\textsc{fpr}_{\mathrm{LiRA}}(x))&\text{ if }\hat{\mu}_{\mathrm{in}} <\hat{\mu}_{\mathrm{out}}\end{cases}$ (A20) $\displaystyle\log\beta=\begin{cases}\frac{\hat{\mu}_{\mathrm{in}}-\hat{\mu}_{ \mathrm{out}}}{\hat{\sigma}^{2}}\left(\sigma F_{t}^{-1}(1-\textsc{fpr}_{ \mathrm{LiRA}}(x))-\frac{\hat{\mu}_{\mathrm{in}}+\hat{\mu}_{\mathrm{out}}}{2}+ \mu_{\mathrm{out}}\right)&\text{ if }\hat{\mu}_{\mathrm{in}}>\hat{\mu}_{ \mathrm{out}}\\ \frac{\hat{\mu}_{\mathrm{in}}-\hat{\mu}_{\mathrm{out}}}{\hat{\sigma}^{2}}\left (\sigma F_{t}^{-1}(\textsc{fpr}_{\mathrm{LiRA}}(x))-\frac{\hat{\mu}_{\mathrm{ in}}+\hat{\mu}_{\mathrm{out}}}{2}+\mu_{\mathrm{out}}\right)&\text{ if }\hat{ \mu}_{\mathrm{in}}<\hat{\mu}_{\mathrm{out}},\end{cases}$ (A21) $$

where we abuse notations by denoting $\mu_{\mathrm{in}}$ to refer to $\mu_{\mathrm{in}}(x)$ and similarly for other parameters.
Since we need to choose a rejection region of the likelihood ratio test such that $\beta\geq 1$, we have

$$ $\begin{cases}\frac{\hat{\mu}_{\mathrm{in}}-\hat{\mu}_{\mathrm{out}}}{\hat{ \sigma}^{2}}\left(\sigma F_{t}^{-1}(1-\textsc{fpr}_{\mathrm{LiRA}}(x))-\frac{ \hat{\mu}_{\mathrm{in}}+\hat{\mu}_{\mathrm{out}}}{2}+\mu_{\mathrm{out}}\right) \geq 0&\quad\text{ if }\ \hat{\mu}_{\mathrm{in}}>\hat{\mu}_{\mathrm{out}}\\ \frac{\hat{\mu}_{\mathrm{in}}-\hat{\mu}_{\mathrm{out}}}{\hat{\sigma}^{2}}\left (\sigma F_{t}^{-1}(\textsc{fpr}_{\mathrm{LiRA}}(x))-\frac{\hat{\mu}_{\mathrm{ in}}+\hat{\mu}_{\mathrm{out}}}{2}+\mu_{\mathrm{out}}\right)\geq 0&\quad\text{ if }\ \hat{\mu}_{\mathrm{in}}<\hat{\mu}_{\mathrm{out}}.\end{cases}$ (A22) $$

Therefore, the sufficient condition about attacker’s knowledge on the underlying distribution for Lemma 1 to hold is

$$ $\begin{cases}\sigma F_{t}^{-1}(1-\textsc{fpr}_{\mathrm{LiRA}}(x))-\frac{\hat{ \mu}_{\mathrm{in}}+\hat{\mu}_{\mathrm{out}}}{2}+\mu_{\mathrm{out}}\geq 0&\quad \text{ if }\ \hat{\mu}_{\mathrm{in}}>\hat{\mu}_{\mathrm{out}}\\ \sigma F_{t}^{-1}(\textsc{fpr}_{\mathrm{LiRA}}(x))-\frac{\hat{\mu}_{\mathrm{in }}+\hat{\mu}_{\mathrm{out}}}{2}+\mu_{\mathrm{out}}\leq 0&\quad\text{ if }\ \hat{\mu}_{\mathrm{in}}<\hat{\mu}_{\mathrm{out}}.\end{cases}$ (A23) $$

We summarise this discussion in the following:

###### Lemma 7 ( Lemma 1 with a relaxed assumption) .

Suppose that the true distribution of $t_{x}$ is of location-scale family with locations $\mu_{\mathrm{in}}(x),\mu_{\mathrm{out}}(x)$ and scale $\sigma(x)$, and that LiRA models $t_{x}$ by $\mathcal{N}(\hat{\mu}_{\mathrm{in}}(x),\hat{\sigma}(x))$ and $\mathcal{N}(\hat{\mu}_{\mathrm{out}}(x),\hat{\sigma}(x))$.
Assume that an attacker estimates parameters $\hat{\mu}_{\mathrm{in}}(x),\hat{\sigma}(x),\hat{\mu}_{\mathrm{out}}(x)$ and $\hat{\sigma}_{\mathrm{out}}(x)$ with an approximated underlying distribution such that

$$ $\begin{cases}\sigma F_{t}^{-1}(1-\textsc{fpr}_{\mathrm{LiRA}}(x))-\frac{\hat{ \mu}_{\mathrm{in}}(x)+\hat{\mu}_{\mathrm{out}}(x)}{2}+\mu_{\mathrm{out}}(x) \geq 0&\quad\text{ if }\ \hat{\mu}_{\mathrm{in}}(x)>\hat{\mu}_{\mathrm{out}}(x )\\ \sigma F_{t}^{-1}(\textsc{fpr}_{\mathrm{LiRA}}(x))-\frac{\hat{\mu}_{\mathrm{in }}(x)+\hat{\mu}_{\mathrm{out}}(x)}{2}+\mu_{\mathrm{out}}(x)\leq 0&\quad\text{ if }\ \hat{\mu}_{\mathrm{in}}(x)<\hat{\mu}_{\mathrm{out}}(x).\end{cases}$ (A24) $$

Then the LiRA vulnerability of a fixed target example is

$$ $\textsc{tpr}_{\mathrm{LiRA}}(x)=\begin{cases}1-F_{t}\left(F_{t}^{-1}(1-\textsc {fpr}_{\mathrm{LiRA}}(x))-\frac{\mu_{\mathrm{in}}(x)-\mu_{\mathrm{out}}(x)}{ \sigma(x)}\right)&\text{ if }\hat{\mu}_{\mathrm{in}}(x)>\hat{\mu}_{\mathrm{out }}(x)\\ F_{t}\left(F_{t}^{-1}(\textsc{fpr}_{\mathrm{LiRA}}(x))-\frac{\mu_{\mathrm{in}} (x)-\mu_{\mathrm{out}}(x)}{\sigma(x)}\right)&\text{ if }\hat{\mu}_{\mathrm{in} }(x)<\hat{\mu}_{\mathrm{out}}(x),\end{cases}$ (A25) $$

where $F_{t}$ is the cdf of $t$ with the standard location and unit scale.

###### Lemma 7 ( Lemma 1 with a relaxed assumption) .

## Appendix C Missing proofs of Section 3

### C.1 Proof of Lemma 1

See 1

###### Proof.

We abuse notations by denoting $\mu_{\mathrm{in}}$ to refer to $\mu_{\mathrm{in}}(x)$ and similarly for other statistics.
We have

$$ $\displaystyle\log\frac{\mathcal{N}(t_{x};\hat{\mu}_{\mathrm{in}},\hat{\sigma}) }{\mathcal{N}(t_{x};\hat{\mu}_{\mathrm{out}},\hat{\sigma})}\geq\log\beta$ (A26) $\displaystyle-\frac{1}{2}\left(\frac{t_{x}-\hat{\mu}_{\mathrm{in}}}{\hat{ \sigma}}\right)^{2}+\frac{1}{2}\left(\frac{t_{x}-\hat{\mu}_{\mathrm{out}}}{ \hat{\sigma}}\right)^{2}\geq\log\beta$ (A27) $\displaystyle\frac{1}{2\hat{\sigma}^{2}}(2t_{x}\hat{\mu}_{\mathrm{in}}-\hat{ \mu}_{\mathrm{in}}^{2}-2t_{x}\hat{\mu}_{\mathrm{out}}+\hat{\mu}_{\mathrm{out}} ^{2})\geq\log\beta$ (A28) $\displaystyle\frac{1}{2\hat{\sigma}^{2}}(\hat{\mu}_{\mathrm{in}}-\hat{\mu}_{ \mathrm{out}})(2t_{x}-\hat{\mu}_{\mathrm{in}}-\hat{\mu}_{\mathrm{out}})\geq\log\beta$ (A29) $\displaystyle\begin{cases}t_{x}\geq\frac{\hat{\sigma}^{2}\log\beta}{\hat{\mu}_ {\mathrm{in}}-\hat{\mu}_{\mathrm{out}}}+\frac{\hat{\mu}_{\mathrm{in}}+\hat{\mu }_{\mathrm{out}}}{2}&\text{ if }\hat{\mu}_{\mathrm{in}}>\hat{\mu}_{\mathrm{out }}\\ t_{x}\leq\frac{\hat{\sigma}^{2}\log\beta}{\hat{\mu}_{\mathrm{in}}-\hat{\mu}_{ \mathrm{out}}}+\frac{\hat{\mu}_{\mathrm{in}}+\hat{\mu}_{\mathrm{out}}}{2}& \text{ if }\hat{\mu}_{\mathrm{in}}<\hat{\mu}_{\mathrm{out}}.\end{cases}$ (A30) $$

When $(x,y_{x})\notin\mathcal{D}_{\mathrm{target}}$, $t_{x}\sim\mathcal{N}(\mu_{\mathrm{out}},\sigma)$.
Therefore, if $\hat{\mu}_{\mathrm{in}}>\hat{\mu}_{\mathrm{out}}$, in the limit of infinitely many shadow models

$$ $\displaystyle\textsc{fpr}_{\mathrm{LiRA}}(x)$ $\displaystyle=\Pr_{t}\left(\mu_{\mathrm{out}}+\sigma t\geq\frac{\hat{\sigma}^{ 2}\log\beta}{\hat{\mu}_{\mathrm{in}}-\hat{\mu}_{\mathrm{out}}}+\frac{\hat{\mu} _{\mathrm{in}}+\hat{\mu}_{\mathrm{out}}}{2}\right)$ (A31) $\displaystyle=\Pr_{t}\left(t\geq\frac{\hat{\sigma}^{2}\log\beta}{\sigma(\hat{ \mu}_{\mathrm{in}}-\hat{\mu}_{\mathrm{out}})}+\frac{\hat{\mu}_{\mathrm{in}}+ \hat{\mu}_{\mathrm{out}}}{2\sigma}-\frac{\mu_{\mathrm{out}}}{\sigma}\right)$ (A32) $\displaystyle=1-F_{t}\left(\frac{\hat{\sigma}^{2}\log\beta}{\sigma(\hat{\mu}_{ \mathrm{in}}-\hat{\mu}_{\mathrm{out}})}+\frac{\hat{\mu}_{\mathrm{in}}+\hat{\mu }_{\mathrm{out}}}{2\sigma}-\frac{\mu_{\mathrm{out}}}{\sigma}\right),$ (A33) $$

and if $\hat{\mu}_{\mathrm{in}}<\hat{\mu}_{\mathrm{out}}$, similarly,

$$ $\displaystyle\textsc{fpr}_{\mathrm{LiRA}}(x)$ $\displaystyle=\Pr_{t}\left(\mu_{\mathrm{out}}+\sigma t\leq\frac{\hat{\sigma}^{ 2}\log\beta}{\hat{\mu}_{\mathrm{in}}-\hat{\mu}_{\mathrm{out}}}+\frac{\hat{\mu} _{\mathrm{in}}+\hat{\mu}_{\mathrm{out}}}{2}\right)$ (A34) $\displaystyle=F_{t}\left(\frac{\hat{\sigma}^{2}\log\beta}{\sigma(\hat{\mu}_{ \mathrm{in}}-\hat{\mu}_{\mathrm{out}})}+\frac{\hat{\mu}_{\mathrm{in}}+\hat{\mu }_{\mathrm{out}}}{2\sigma}-\frac{\mu_{\mathrm{out}}}{\sigma}\right).$ (A35) $$

Thus

$$ $\frac{\hat{\sigma}^{2}\log\beta}{\sigma(\hat{\mu}_{\mathrm{in}}-\hat{\mu}_{ \mathrm{out}})}+\frac{\hat{\mu}_{\mathrm{in}}+\hat{\mu}_{\mathrm{out}}}{2 \sigma}-\frac{\mu_{\mathrm{out}}}{\sigma}=\begin{cases}F_{t}^{-1}(1-\textsc{ fpr}_{\mathrm{LiRA}}(x))&\text{ if }\hat{\mu}_{\mathrm{in}}>\hat{\mu}_{\mathrm {out}}\\ F_{t}^{-1}(\textsc{fpr}_{\mathrm{LiRA}}(x))&\text{ if }\hat{\mu}_{\mathrm{in}} <\hat{\mu}_{\mathrm{out}}.\end{cases}$ (A36) $$

It follows that if $\hat{\mu}_{\mathrm{in}}>\hat{\mu}_{\mathrm{out}}$,

$$ $\displaystyle\textsc{tpr}_{\mathrm{LiRA}}(x)$ $\displaystyle=\Pr_{t}\left(\mu_{\mathrm{in}}+\sigma t\geq\frac{\hat{\sigma}^{2 }\log\beta}{\hat{\mu}_{\mathrm{in}}-\hat{\mu}_{\mathrm{out}}}+\frac{\hat{\mu}_ {\mathrm{in}}+\hat{\mu}_{\mathrm{out}}}{2}\right)$ (A37) $\displaystyle=\Pr_{t}\left(t\geq\frac{\hat{\sigma}^{2}\log\beta}{\sigma(\hat{ \mu}_{\mathrm{in}}-\hat{\mu}_{\mathrm{out}})}+\frac{\hat{\mu}_{\mathrm{in}}+ \hat{\mu}_{\mathrm{out}}}{2\sigma}-\frac{\mu_{\mathrm{in}}}{\sigma}\right)$ (A38) $\displaystyle=1-F_{t}\left(F_{t}^{-1}(1-\textsc{fpr}_{\mathrm{LiRA}}(x))-\frac {\mu_{\mathrm{in}}-\mu_{\mathrm{out}}}{\sigma}\right).$ (A39) $$

If $\hat{\mu}_{\mathrm{in}}<\hat{\mu}_{\mathrm{out}}$, then

$$ $\displaystyle\textsc{tpr}_{\mathrm{LiRA}}(x)$ $\displaystyle=\Pr_{t}\left(\mu_{\mathrm{in}}+\sigma t\leq\frac{\hat{\sigma}^{2 }\log\beta}{\hat{\mu}_{\mathrm{in}}-\hat{\mu}_{\mathrm{out}}}+\frac{\hat{\mu}_ {\mathrm{in}}+\hat{\mu}_{\mathrm{out}}}{2}\right)$ (A40) $\displaystyle=\Pr_{t}\left(t\leq\frac{\hat{\sigma}^{2}\log\beta}{\sigma(\hat{ \mu}_{\mathrm{in}}-\hat{\mu}_{\mathrm{out}})}+\frac{\hat{\mu}_{\mathrm{in}}+ \hat{\mu}_{\mathrm{out}}}{2\sigma}-\frac{\mu_{\mathrm{in}}}{\sigma}\right)$ (A41) $\displaystyle=F_{t}\left(F_{t}^{-1}(\textsc{fpr}_{\mathrm{LiRA}}(x))-\frac{\mu _{\mathrm{in}}-\mu_{\mathrm{out}}}{\sigma}\right).$ (A42) $$

∎

###### Proof.

### C.2 Proof of Lemma 2

See 2

###### Proof.

We abuse notations by denoting probabilities and expectations over sampling $\mathcal{D}_{\mathrm{target}}\sim{\mathbb{D}}^{|\mathcal{D}|}$ and $(z,y_{z})\sim{\mathbb{D}}$ by subscripts $\mathcal{D}_{\mathrm{target}}$ and $z$.
Recall that

$$ $\displaystyle\textsc{tpr}_{\mathrm{RMIA}}(x)$ $\displaystyle=\Pr_{\mathcal{D}_{\mathrm{target}}}\left(\Pr_{z}\left(\mathrm{LR }(x,z)\geq\gamma\right)\geq\beta\mid(x,y_{x})\in\mathcal{D}_{\mathrm{target}} \wedge(x,y_{x})\notin\mathcal{D}_{\mathrm{target}}\right)$ (A43) $\displaystyle\textsc{fpr}_{\mathrm{RMIA}}(x)$ $\displaystyle=\Pr_{\mathcal{D}_{\mathrm{target}}}\left(\Pr_{z}\left(\mathrm{LR }(x,z)\geq\gamma\right)\geq\beta\mid(x,y_{x})\notin\mathcal{D}_{\mathrm{target }}\wedge(x,y_{x})\in\mathcal{D}_{\mathrm{target}}\right),$ (A44) $$

where

$$ $\mathrm{LR}(x,z)=\frac{\mathcal{N}(t_{x};\hat{\mu}_{x,x},\hat{\sigma}_{x}^{2}) \mathcal{N}(t_{z};\hat{\mu}_{x,z},\hat{\sigma}_{z}^{2})}{\mathcal{N}(t_{x}; \hat{\mu}_{z,x},\hat{\sigma}_{x}^{2})\mathcal{N}(t_{z};\hat{\mu}_{z,z},\hat{ \sigma}_{z}^{2})}.$ (A45) $$

We have

$$ $\displaystyle\lambda:=$ $\displaystyle\log(\mathrm{LR}(x,z))$ (A46) $\displaystyle=$ $\displaystyle\log\left(\frac{\frac{1}{\sqrt{2\pi}\hat{\sigma}_{x}}\exp\left(- \frac{1}{2}\left(\frac{t_{x}-\hat{\mu}_{x,x}}{\hat{\sigma}_{x}}\right)^{2} \right)\frac{1}{\sqrt{2\pi}\hat{\sigma}_{z}}\exp\left(-\frac{1}{2}\left(\frac{ t_{z}-\hat{\mu}_{x,z}}{\hat{\sigma}_{z}}\right)^{2}\right)}{\frac{1}{\sqrt{2 \pi}\hat{\sigma}_{x}}\exp\left(-\frac{1}{2}\left(\frac{t_{x}-\hat{\mu}_{z,x}}{ \hat{\sigma}_{x}}\right)^{2}\right)\frac{1}{\sqrt{2\pi}\hat{\sigma}_{z}}\exp \left(-\frac{1}{2}\left(\frac{t_{z}-\hat{\mu}_{z,z}}{\hat{\sigma}_{z}}\right)^ {2}\right)}\right)$ (A47) $\displaystyle=$ $\displaystyle-\frac{1}{2}\left(\frac{t_{x}-\hat{\mu}_{x,x}}{\hat{\sigma}_{x}} \right)^{2}+\frac{1}{2}\left(\frac{t_{x}-\hat{\mu}_{z,x}}{\hat{\sigma}_{x}} \right)^{2}-\frac{1}{2}\left(\frac{t_{z}-\hat{\mu}_{x,z}}{\hat{\sigma}_{z}} \right)^{2}+\frac{1}{2}\left(\frac{t_{z}-\hat{\mu}_{z,z}}{\hat{\sigma}_{z}} \right)^{2}$ (A48) $\displaystyle=$ $\displaystyle\frac{1}{2\hat{\sigma}_{x}^{2}}(2t_{x}\hat{\mu}_{x,x}-\hat{\mu}_{ x,x}^{2}-2t_{x}\hat{\mu}_{z,x}+\hat{\mu}_{z,x}^{2})+\frac{1}{2\hat{\sigma}_{z} ^{2}}(2t_{z}\hat{\mu}_{x,z}-\hat{\mu}_{x,z}^{2}-2t_{z}\hat{\mu}_{z,z}+\hat{\mu }_{z,z}^{2})$ (A49) $\displaystyle=$ $\displaystyle\frac{\hat{\mu}_{x,x}-\hat{\mu}_{z,x}}{2\hat{\sigma}_{x}^{2}}(2t_ {x}-\hat{\mu}_{x,x}-\hat{\mu}_{z,x})+\frac{\hat{\mu}_{x,z}-\hat{\mu}_{z,z}}{2 \hat{\sigma}_{z}^{2}}(2t_{z}-\hat{\mu}_{x,z}-\hat{\mu}_{z,z}).$ (A50) $$

When $(x,y_{x})\in\mathcal{D}_{\mathrm{target}}$, $t_{x}=\mu_{x,x}+\sigma_{x}t$ and $t_{z}=\mu_{x,z}+\sigma_{z}t$, and thus $\lambda$ becomes

$$ $\displaystyle\lambda_{x}:=$ $\displaystyle\frac{\hat{\mu}_{x,x}-\hat{\mu}_{z,x}}{2\hat{\sigma}_{x}^{2}}(2 \mu_{x,x}+2\sigma_{x}t-\hat{\mu}_{x,x}-\hat{\mu}_{z,x})$ (A51) $\displaystyle+\frac{\hat{\mu}_{x,z}-\hat{\mu}_{z,z}}{2\hat{\sigma}_{z}^{2}}(2 \mu_{x,z}+2\sigma_{z}t-\hat{\mu}_{x,z}-\hat{\mu}_{z,z}).$ (A52) $$

Similarly, when $(x,y_{x})\notin\mathcal{D}_{\mathrm{target}}$, we have $t_{x}=\mu_{z,x}+\sigma_{x}t$ and $t_{z}=\mu_{z,z}+\sigma_{z}t$. Then $\lambda$ is now

$$ $\displaystyle\lambda_{z}:=$ $\displaystyle\frac{\hat{\mu}_{x,x}-\hat{\mu}_{z,x}}{2\hat{\sigma}_{x}^{2}}(2 \mu_{z,x}+2\sigma_{x}t-\hat{\mu}_{x,x}-\hat{\mu}_{z,x})$ (A53) $\displaystyle+\frac{\hat{\mu}_{x,z}-\hat{\mu}_{z,z}}{2\hat{\sigma}_{z}^{2}}(2 \mu_{z,z}+2\sigma_{z}t-\hat{\mu}_{x,z}-\hat{\mu}_{z,z})$ (A54) $\displaystyle=$ $\displaystyle\left(\underbrace{\frac{\sigma_{x}}{\hat{\sigma}_{x}^{2}}(\hat{ \mu}_{x,x}-\hat{\mu}_{z,x})+\frac{\sigma_{z}}{\hat{\sigma}_{z}^{2}}(\hat{\mu}_ {x,z}-\hat{\mu}_{z,z})}_{A}\right)t$ (A55) $\displaystyle+\underbrace{\frac{\hat{\mu}_{x,x}-\hat{\mu}_{z,x}}{2\hat{\sigma} _{x}^{2}}(2\mu_{z,x}-\hat{\mu}_{x,x}-\hat{\mu}_{z,x})+\frac{\hat{\mu}_{x,z}- \hat{\mu}_{z,z}}{2\hat{\sigma}_{z}^{2}}(2\mu_{z,z}-\hat{\mu}_{x,z}-\hat{\mu}_{ z,z})}_{B}$ (A56) $\displaystyle=$ $\displaystyle At+B.$ (A57) $$

Notice that $A$ and $B$ are functions of $z$ and independent of $t$.
Also note that taking probability over $t$ corresponds to calculating probability over $\mathcal{D}_{\mathrm{target}}\sim{\mathbb{D}}^{|\mathcal{D}|}$.
Thus, Since we can set $\gamma>1$, using Markov’s inequality and the triangle inequality, we have in the limit of infinitely many shadow models

$$ $\displaystyle\textsc{fpr}_{\mathrm{RMIA}}(x)=$ $\displaystyle\Pr_{t}\left(\Pr_{z}(\lambda_{z}\geq\log(\gamma))\geq\beta\right)$ (A58) $\displaystyle\leq\Pr_{t}\left(\Pr_{z}(|\lambda_{z}|\geq\log(\gamma))\geq\beta\right)$ (A59) $\displaystyle\leq\Pr_{t}\left(\frac{\mathbb{E}_{z}[|\lambda_{z}|]}{\log(\gamma )}\geq\beta\right)$ (A60) $\displaystyle=\Pr_{t}\left(\mathbb{E}_{z}[|\lambda_{z}|]\geq\beta\log(\gamma)\right)$ (A61) $\displaystyle\leq\Pr_{t}\left(\mathbb{E}_{z}[|A|]|t|+\mathbb{E}_{z}[|B|]\geq \beta\log(\gamma)\right)$ (A62) $\displaystyle=\Pr_{t}\left(|t|\geq\frac{\beta\log(\gamma)-\mathbb{E}_{z}[|B|]} {\mathbb{E}_{z}[|A|]}\right)$ (A63) $$

Therefore, we can upper-bound $\textsc{fpr}_{\mathrm{RMIA}}(x)\leq\alpha$ by setting

$$ $\alpha=1-F_{|t|}\left(\frac{\beta\log(\gamma)-\mathbb{E}_{z}[|B|]}{\mathbb{E}_ {z}[|A|]}\right).$ (A64) $$

That is,

$$ $\frac{\beta\log(\gamma)-\mathbb{E}_{z}[|B|]}{\mathbb{E}_{z}[|A|]}=F_{|t|}^{-1} (1-\alpha).$ (A65) $$

Now let

$$ $q=\lambda_{x}-\lambda_{z}=\frac{(\mu_{x,x}-\mu_{z,x})(\hat{\mu}_{x,x}-\hat{\mu }_{z,x})}{\hat{\sigma}_{x}^{2}}+\frac{(\mu_{x,z}-\mu_{z,z})(\hat{\mu}_{x,z}- \hat{\mu}_{z,z})}{\hat{\sigma}_{z}^{2}}.$ (A66) $$

Note that $q$ is also independent of $t$, thereby $\mathbb{E}_{z}[|q|]$ being a constant. By the similar argument using Markov’s inequality and the triangle inequality, we have

$$ $\displaystyle\textsc{tpr}_{\mathrm{RMIA}}(x)=$ $\displaystyle\Pr_{t}\left(\Pr_{z}(\lambda_{x}\geq\log(\gamma))\geq\beta\right)$ (A67) $\displaystyle\leq\Pr_{t}\left(\Pr_{z}(|\lambda_{x}|\geq\log(\gamma))\geq\beta\right)$ (A68) $\displaystyle\leq\Pr_{t}\left(\frac{\mathbb{E}_{z}[|\lambda_{x}|]}{\log(\gamma )}\geq\beta\right)$ (A69) $\displaystyle=\Pr_{t}\left(\mathbb{E}_{z}[|\lambda_{x}|]\geq\beta\log(\gamma)\right)$ (A70) $\displaystyle\leq\Pr_{t}\left(\mathbb{E}_{z}[|A|]|t|+\mathbb{E}_{z}[|B|]+ \mathbb{E}_{z}[|q|]\geq\beta\log(\gamma)\right)$ (A71) $\displaystyle=\Pr_{t}\left(|t|\geq\frac{\beta\log(\gamma)-\mathbb{E}_{z}[|B|]} {\mathbb{E}_{z}[|A|]}-\frac{\mathbb{E}_{z}[|q|]}{\mathbb{E}_{z}[|A|]}\right)$ (A72) $$

Hence we obtain

$$ $\textsc{tpr}_{\mathrm{RMIA}}(x)\leq 1-F_{|t|}^{-1}\left(F_{|t|}^{-1}(1-\alpha) -\frac{\mathbb{E}_{z}[|q|]}{\mathbb{E}_{z}[|A|]}\right).$ (A73) $$

∎

###### Proof.

### C.3 Proof of Theorem 3

See 3

###### Proof.

Let $\mathcal{D}_{\mathrm{target}}=\{(x_{j,1},j),...,(x_{j,S},j)\}_{j=1}^{C}$. Then the LiRA score of the target $(x,y_{x})$ is

$$ $\displaystyle s_{y_{x}}^{\mathrm{(in)}}$ $\displaystyle=\langle x,\frac{1}{S}\left(\sum_{i=1}^{S-1}x_{y_{x},i}+x\right) \rangle=\langle x,\frac{1}{S}\sum_{i=1}^{S}x_{y_{x},i}\rangle+\langle x,\frac{ 1}{S}(x-x_{y_{x},S})\rangle$ (A74) $\displaystyle s_{y_{x}}^{\mathrm{(out)}}$ $\displaystyle=\langle x,\frac{1}{S}\sum_{i=1}^{S}x_{y_{x},i}\rangle,$ (A75) $$

respectively, when $(x,y_{x})\in\mathcal{D}_{\mathrm{target}}$ and when $(x,y_{x})\notin\mathcal{D}_{\mathrm{target}}$.
Thus, in the limit of infinitely many shadow models we obtain

$$ $\displaystyle\mu_{\mathrm{in}}-\mu_{\mathrm{out}}$ $\displaystyle=\hat{\mu}_{\mathrm{in}}-\hat{\mu}_{\mathrm{out}}=\mathbb{E}[s_{y _{x}}^{\mathrm{(in)}}-s_{y_{x}}^{\mathrm{(out)}}]=\frac{1}{S}\langle x,x-m_{x}\rangle$ (A76) $\displaystyle\sigma^{2}$ $\displaystyle=\operatorname{Var}(s_{y_{x}}^{\mathrm{(out)}})=\frac{1}{S} \operatorname{Var}(\langle x,x_{y_{x},i}\rangle)=\frac{1}{S}||x||^{2}s^{2}$ (A77) $$

Noting that the LiRA score follows a normal distribution, by Lemma 1 we have

$$ $\displaystyle\textsc{tpr}_{\mathrm{LiRA}}(x)$ $\displaystyle=\begin{cases}1-\Phi\left(\Phi^{-1}(1-\textsc{fpr}_{\mathrm{LiRA} }(x))-\frac{\mu_{\mathrm{in}}-\mu_{\mathrm{out}}}{\sigma}\right)&\text{ if } \hat{\mu}_{\mathrm{in}}>\hat{\mu}_{\mathrm{out}}\\ \Phi\left(\Phi^{-1}(\textsc{fpr}_{\mathrm{LiRA}}(x))-\frac{\mu_{\mathrm{in}}- \mu_{\mathrm{out}}}{\sigma}\right)&\text{ if }\hat{\mu}_{\mathrm{in}}<\hat{\mu }_{\mathrm{out}},\end{cases}$ (A78) $\displaystyle=\begin{cases}1-\Phi\left(\Phi^{-1}(1-\textsc{fpr}_{\mathrm{LiRA} }(x))-\frac{\langle x,x-m_{x}\rangle}{\sqrt{S}s||x||}\right)&\text{ if } \langle x,x-m_{x}\rangle>0\\ \Phi\left(\Phi^{-1}(\textsc{fpr}_{\mathrm{LiRA}}(x))-\frac{\langle x,x-m_{x} \rangle}{\sqrt{S}s||x||}\right)&\text{ if }\langle x,x-m_{x}\rangle<0,\end{cases}$ (A79) $$

By the symmetry of a normal distribution, this simplifies to

$$ $\textsc{tpr}_{\mathrm{LiRA}}(x)=\Phi\left(\Phi^{-1}(\textsc{fpr}_{\mathrm{LiRA }}(x))+\frac{|\langle x,x-m_{x}\rangle|}{\sqrt{S}s||x||}\right),$ (A80) $$

where $\langle x,x-m_{x}\rangle\not=0$ by the assumption that $\hat{\mu}_{\mathrm{in}}\not=\hat{\mu}_{\mathrm{out}}$.
This completes the first half of the theorem.

Now we have

$$ $\displaystyle\textsc{tpr}_{\mathrm{LiRA}}(x)$ $\displaystyle=\Pr\left(\eta\leq\gamma_{0}+\frac{|\langle x,x-m_{x}\rangle|}{ \sqrt{S}s||x||}\right),$ (A81) $\displaystyle\textsc{fpr}_{\mathrm{LiRA}}(x)$ $\displaystyle=\Pr(\eta\leq\gamma_{0}),$ (A82) $$

where $\gamma_{0}$ is a tunable constant and $\eta\sim\mathcal{N}(0,1)$.
Thus for large enough $S$ we have

$$ $\displaystyle\textsc{tpr}_{\mathrm{LiRA}}(x)-\textsc{fpr}_{\mathrm{LiRA}}(x)$ $\displaystyle\approx p_{\eta}(\gamma_{0})\frac{|\langle x,x-m_{x}\rangle|}{ \sqrt{S}||x||s}$ (A83) $\displaystyle=\frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}\Phi^{-1}(\textsc{fpr}_{ \mathrm{LiRA}}(x))^{2}}\frac{|\langle x,x-m_{x}\rangle|}{\sqrt{S}||x||s},$ (A84) $\displaystyle\log(\textsc{tpr}_{\mathrm{LiRA}}(x)-\textsc{fpr}_{\mathrm{LiRA}} (x))$ $\displaystyle\approx-\frac{1}{2}\log S-\frac{1}{2}\Phi^{-1}(\textsc{fpr}_{ \mathrm{LiRA}}(x))^{2}+\log\frac{|\langle x,x-m_{x}\rangle|}{||x||s\sqrt{2\pi}}.$ (A85) $$

∎

###### Proof.

### C.4 Proof of Theorem 4

See 4

###### Proof.

To apply Lemma 2, we will calculate $\mathbb{E}_{z}[q]$ and $\mathbb{E}_{z}[A]$.
Let $s_{y_{x}}^{(x)}(x)$ (resp. $s_{y_{x}}^{(z)}(x)$) denote the score of the target $x$ for class $y_{x}$ when the dataset contains $(x,y_{x})$ but not $(z,y_{z})$ (resp. when the dataset contains $(z,y_{z})$ but not $(x,y_{x})$).
Let $s_{y_{z}}^{(x)}(z)$ and $z_{y_{z}}^{(z)}(z)$ be corresponding scores of example $z$.
Then we have

$$ $\displaystyle s_{y_{x}}^{(x)}(x)$ $\displaystyle=\frac{1}{S}\langle x,\sum_{i=1}^{S}x_{i}+x-x_{S}\rangle$ (A86) $\displaystyle s_{y_{x}}^{(z)}(x)$ $\displaystyle=\begin{cases}\frac{1}{S}\langle x,\sum_{i=1}^{S}x_{i}+z-x_{S} \rangle\quad&\text{ if }y_{x}=y_{z}\\ \frac{1}{S}\langle x,\sum_{i=1}^{S}x_{i}\rangle\quad&\text{ if }y_{x}\not=y_{z }\end{cases}$ (A87) $\displaystyle s_{y_{z}}^{(x)}(z)$ $\displaystyle=\begin{cases}\frac{1}{S}\langle z,\sum_{i=1}^{S}z_{i}+x-z_{S} \rangle\quad&\text{ if }y_{x}=y_{z}\\ \frac{1}{S}\langle z,\sum_{i=1}^{S}z_{i}\rangle\quad&\text{ if }y_{x}\not=y_{z }\end{cases}$ (A88) $\displaystyle s_{y_{z}}^{(z)}(z)$ $\displaystyle=\frac{1}{S}\langle z,\sum_{i=1}^{S}z_{i}+z-x_{S}\rangle$ (A89) $$

where $x_{i}$ are samples with label $y_{x}$ and $z_{i}$ are samples with label $y_{z}$ when $y_{x}\not=y_{z}$. Thus we obtain

$$ $\displaystyle\mu_{x,x}$ $\displaystyle=\langle x,m_{x}\rangle+\frac{1}{S}\langle x,x-m_{x}\rangle$ (A90) $\displaystyle\mu_{z,x}$ $\displaystyle=\begin{cases}\langle x,m_{x}\rangle+\frac{1}{S}\langle x,z-m_{x} \rangle\quad&\text{ if }y_{x}=y_{z}\\ \langle x,m_{x}\rangle\quad&\text{ if }y_{x}\not=y_{z}\end{cases}$ (A91) $\displaystyle\mu_{x,z}$ $\displaystyle=\begin{cases}\langle z,m_{x}\rangle+\frac{1}{S}\langle z,x-m_{x} \rangle\quad&\text{ if }y_{x}=y_{z}\\ \langle z,m_{z}\rangle\quad&\text{ if }y_{x}\not=y_{z}\end{cases}$ (A92) $\displaystyle\mu_{z,z}$ $\displaystyle=\langle z,m_{x}\rangle+\frac{1}{S}\langle z,z-m_{x}\rangle\quad$ (A93) $\displaystyle\sigma_{x}$ $\displaystyle=\frac{1}{\sqrt{S}}s||x||$ (A94) $\displaystyle\sigma_{z}$ $\displaystyle=\frac{1}{\sqrt{S}}s||z||,$ (A95) $$

where $m_{z}$ is the true class mean of $y_{z}$ (see Appendix B for derivation of $\sigma_{x}$ and $\sigma_{z}$).

Now recall that

$$ $\displaystyle q=$ $\displaystyle\frac{(\mu_{x,x}-\mu_{z,x})(\hat{\mu}_{x,x}-\hat{\mu}_{z,x})}{ \hat{\sigma}_{x}^{2}}+\frac{(\mu_{x,z}-\mu_{z,z})(\hat{\mu}_{x,z}-\hat{\mu}_{z ,z})}{\hat{\sigma}_{z}^{2}}$ (A96) $\displaystyle A=$ $\displaystyle\frac{\sigma_{x}}{\hat{\sigma}_{x}^{2}}(\hat{\mu}_{x,x}-\hat{\mu} _{z,x})+\frac{\sigma_{z}}{\hat{\sigma}_{z}^{2}}(\hat{\mu}_{x,z}-\hat{\mu}_{z,z})$ (A97) $$

In the limit of infinitely many shadow models, estimated statisitcs converge to true parameters, thereby yielding

$$ $\displaystyle q=$ $\displaystyle\left(\frac{\mu_{x,x}-\mu_{z,x}}{\sigma_{x}}\right)^{2}+\left( \frac{\mu_{x,z}-\mu_{z,z}}{\sigma_{z}}\right)^{2}$ (A98) $\displaystyle A=$ $\displaystyle\frac{\mu_{x,x}-\mu_{z,x}}{\sigma_{x}}+\frac{\mu_{x,z}-\mu_{z,z}} {\sigma_{z}}.$ (A99) $$

Using the law of total expectation, we have

$$ $\displaystyle\mathbb{E}_{z}[|q|]=$ $\displaystyle\Pr_{z}(y_{z}=y_{x})\mathbb{E}_{z}[|q|\mid y_{z}=y_{x}]+\sum_{j=1 ,j\not=y_{x}}^{C}\Pr_{z}(y_{z}=j)\mathbb{E}_{z}[|q|\mid y_{z}=j]$ (A100) $\displaystyle=$ $\displaystyle\frac{1}{C}\mathbb{E}_{z}\left[\left.\left(\frac{\langle x,x-z \rangle}{\sqrt{S}s||x||}\right)^{2}+\left(\frac{\langle z,x-z\rangle}{\sqrt{S} s||z||}\right)^{2}\ \right|\ y_{z}=y_{x}\right]$ (A101) $\displaystyle+\frac{C-1}{C}\mathbb{E}_{z}\left[\left.\left(\frac{\langle x,x-m _{x}\rangle}{\sqrt{S}s||x||}\right)^{2}+\left(\frac{\langle z,z-m_{z}\rangle}{ \sqrt{S}s||z||}\right)^{2}\ \right|\ y_{z}\not=y_{x}\right]$ (A102) $\displaystyle=$ $\displaystyle\frac{1}{CSs^{2}}\mathbb{E}_{z}\left[\left.\frac{\langle x,x-z \rangle^{2}}{||x||^{2}}+\frac{\langle z,x-z\rangle^{2}}{||z||^{2}}\ \right|\ y _{z}=y_{x}\right]$ (A103) $\displaystyle+\frac{C-1}{CSs^{2}}\mathbb{E}_{z}\left[\left.\frac{\langle x,x-m _{x}\rangle^{2}}{||x||^{2}}+\frac{\langle z,z-m_{z}\rangle^{2}}{||z||^{2}}\ \right|\ y_{z}\not=y_{x}\right],$ (A104) $$

and

$$ $\displaystyle\mathbb{E}_{z}[|A|]=$ $\displaystyle\Pr_{z}(y_{z}=y_{x})\mathbb{E}_{z}[|A|\mid y_{z}=y_{x}]+\sum_{j=1 ,j\not=y_{x}}^{C}\Pr_{z}(y_{z}=j)\mathbb{E}_{z}[|A|\mid y_{z}=j]$ (A105) $\displaystyle=$ $\displaystyle\frac{1}{C}\mathbb{E}_{z}\left[\left.\left|\frac{\langle x,x-z \rangle}{\sqrt{S}s||x||}+\frac{\langle z,x-z\rangle}{\sqrt{S}s||z||}\right|\ \right|\ y_{z}=y_{x}\right]$ (A106) $\displaystyle+\frac{C-1}{C}\mathbb{E}_{z}\left[\left.\left|\frac{\langle x,x-m _{x}\rangle}{\sqrt{S}s||x||}+\frac{\langle z,z-m_{z}\rangle}{\sqrt{S}s||z||} \right|\ \right|\ y_{z}\not=y_{x}\right]$ (A107) $\displaystyle=$ $\displaystyle\frac{1}{C\sqrt{S}s}\mathbb{E}_{z}\left[\left.\left|\frac{\langle x ,x-z\rangle}{||x||}+\frac{\langle z,x-z\rangle}{||z||}\right|\ \right|\ y_{z}= y_{x}\right]$ (A108) $\displaystyle+\frac{C-1}{C\sqrt{S}s}\mathbb{E}_{z}\left[\left.\left|\frac{ \langle x,x-m_{x}\rangle}{||x||}+\frac{\langle z,z-m_{z}\rangle}{||z||}\right| \ \right|\ y_{z}\not=y_{x}\right].$ (A109) $$

Hence we obtain by the Cauchy-Schwarz inequality

$$ $\displaystyle\frac{\mathbb{E}_{z}[|q|]}{\mathbb{E}_{z}[|A|]}=$ $\displaystyle\frac{1}{\sqrt{S}s}\cdot\frac{\mathbb{E}_{z}\left[\frac{\langle x ,x-z\rangle^{2}}{||x||^{2}}+\frac{\langle z,x-z\rangle^{2}}{||z||^{2}}\mid y_{ z}=y_{x}\right]+(C-1)\mathbb{E}_{z}\left[\frac{\langle x,x-m_{x}\rangle^{2}}{| |x||^{2}}+\frac{\langle z,z-m_{z}\rangle^{2}}{||z||^{2}}\mid y_{z}\not=y_{x} \right]}{\mathbb{E}_{z}\left[\left|\frac{\langle x,x-z\rangle}{||x||}+\frac{ \langle z,x-z\rangle}{||z||}\right|\mid y_{z}=y_{x}\right]+(C-1)\mathbb{E}_{z} \left[\left|\frac{\langle x,x-m_{x}\rangle}{||x||}+\frac{\langle z,z-m_{z} \rangle}{||z||}\right|\mid y_{z}\not=y_{x}\right]}$ (A110) $\displaystyle\leq$ $\displaystyle\frac{1}{\sqrt{S}s}\cdot\frac{\mathbb{E}_{z}\left[2||x-z||^{2} \mid y_{z}=y_{x}\right]+(C-1)\mathbb{E}_{z}\left[(||x-m_{x}||^{2}+||z-m_{z}||^ {2})\mid y_{z}\not=y_{x}\right]}{\mathbb{E}_{z}\left[2||x-z||\mid y_{z}=y_{x} \right]+(C-1)\mathbb{E}_{z}\left[(||x-m_{x}||+||z-m_{z}||)\mid y_{z}\not=y_{x} \right]}.$ (A111) $$

Now Lemma 2 yields

$$ $\displaystyle\textsc{tpr}_{\mathrm{RMIA}}(x)$ $\displaystyle\leq 1-F_{|t|}\left(F_{|t|}^{-1}(1-\alpha)-\frac{\mathbb{E}_{z}[| q|]}{\mathbb{E}_{z}[|A|]}\right)$ (A112) $\displaystyle\leq 1-F_{|t|}\left(F_{|t|}^{-1}(1-\alpha)-\frac{\psi(x,C)}{\sqrt {S}s}\right)$ (A113) $$

where $F_{|t|}$ is the cdf of the folded normal distribution and

$$ $\psi(x,C)=\frac{\mathbb{E}_{z}\left[2||x-z||^{2}\mid y_{z}=y_{x}\right]+(C-1) \mathbb{E}_{z}\left[(||x-m_{x}||^{2}+||z-m_{z}||^{2})\mid y_{z}\not=y_{x} \right]}{\mathbb{E}_{z}\left[2||x-z||\mid y_{z}=y_{x}\right]+(C-1)\mathbb{E}_{ z}\left[(||x-m_{x}||+||z-m_{z}||)\mid y_{z}\not=y_{x}\right]}.$ (A114) $$

This completes the first half of the theorem.

Now that from Lemma 2 we have

$$ $\displaystyle\textsc{tpr}_{\mathrm{RMIA}}(x)$ $\displaystyle=\Pr_{t}\left(\Pr_{z}(\lambda_{z}+q\geq\log\gamma)\geq\beta\right)$ (A115) $\displaystyle\leq\Pr_{t}\left(\frac{\mathbb{E}_{z}[|\lambda_{z}|]+\mathbb{E}_{ z}[|q|]}{\log(\gamma)}\geq\beta\right)$ (A116) $\displaystyle=\Pr_{t}\left(|t|\geq F_{|t|}^{-1}(1-\alpha)-\frac{\mathbb{E}_{z} [|q|]}{\mathbb{E}_{z}[|A|]}\right)$ (A117) $\displaystyle\textsc{fpr}_{\mathrm{RMIA}}(x)$ $\displaystyle=\Pr_{t}\left(\Pr_{z}(\lambda_{z}\geq\log\gamma)\geq\beta\right)$ (A118) $\displaystyle\leq\Pr_{t}\left(\frac{\mathbb{E}_{z}[|\lambda_{z}|]}{\log(\gamma )}\geq\beta\right)$ (A119) $\displaystyle=\Pr_{t}\left(|t|\geq F_{|t|}^{-1}(1-\alpha)\right).$ (A120) $$

We claim that the bound for $\textsc{fpr}_{\mathrm{RMIA}}(x)$ (Equation A119) is tighter than that for $\textsc{tpr}_{\mathrm{RMIA}}(x)$ (Equation A116). Since this is not straightforward, we numerically show this in Table A1.
Now, for large enough $S$ we obtain

$$ $\displaystyle\textsc{tpr}_{\mathrm{RMIA}}(x)-\textsc{fpr}_{\mathrm{RMIA}}(x)$ $\displaystyle\leq\Pr_{t}\left(|t|\geq F_{|t|}^{-1}(1-\alpha)-\left|\frac{ \mathbb{E}_{z}[q]}{\mathbb{E}_{z}[A]}\right|\right)-\Pr_{t}(|t|\geq F_{|t|}^{- 1}(1-\alpha))$ (A121) $\displaystyle\leq\Pr_{t}\left(|t|\geq F_{|t|}^{-1}(1-\alpha)+\frac{\psi(x,C)}{ \sqrt{S}s}\right)-\Pr_{t}(|t|\leq F_{|t|}^{-1}(1-\alpha))$ (A122) $\displaystyle\approx p_{|t|}(F_{|t|}^{-1}(1-\alpha))\frac{\psi(x,C)}{\sqrt{S}}.$ (A123) $$

Since $|t|$ follows the standard folded normal distribution with mean 0 0 and scale $1$, we have

$$ $p_{|t|}(F_{|t|}^{-1}(1-\alpha))=\frac{1}{\sqrt{\pi/2}}\exp\left(-\frac{1}{2}F_ {|t|}^{-1}(1-\alpha)^{2}\right).$ (A124) $$

Hence we obtain

$$ $\log(\textsc{tpr}_{\mathrm{RMIA}}(x)-\textsc{fpr}_{\mathrm{RMIA}}(x))\leq- \frac{1}{2}\log S-\frac{1}{2}F_{|t|}^{-1}(1-\alpha)^{2}+\log\frac{\psi(x,C)}{ \sqrt{\pi/2}}.$ (A125) $$

∎

**Table A1: The gap in the upper bound for $\textsc{tpr}_{\mathrm{RMIA}}(x)$ in Equation A116 minus the gap in the upper bound for $\textsc{fpr}_{\mathrm{RMIA}}(x)$ in Equation A119. 5 random target examples are sampled from the simplified model distribution with $C=2,s=0.1,d=50$. Simulation is done with 100000 samples of test points $z$, 1000 samples of $t$ and $\gamma=1.0001$. Due to computational constraints, results are not stable especially for small $S$.**
| S | FPR=0.2 | FPR=0.1 | FPR=0.01 | FPR=0.001 |
| --- | --- | --- | --- | --- |
| 10 | 0.000 | 0.000 | 0.000 | 0.000 |
| 100 | 0.000 | 0.000 | 0.000 | 0.000 |
| 1000 | 0.000 | 0.000 | 0.041 | 0.068 |
| 10000 | 0.329 | 0.353 | 0.365 | 0.368 |

###### Proof.

### C.5 Proof of Corollary 5

See 5

###### Proof.

By theorem 3 and the law of unconscious statistician, we have for large $S$

$$ $\displaystyle\overline{\textsc{tpr}}_{\mathrm{LiRA}}-\overline{\textsc{fpr}}_{ \mathrm{LiRA}}$ $\displaystyle=\int_{\mathbb{D}}\Pr(x)(\textsc{tpr}_{\mathrm{LiRA}}(x)-\textsc{ fpr}_{\mathrm{LiRA}}(x))dx$ (A126) $\displaystyle\approx\int_{\mathbb{D}}\Pr(x)\frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2 }\Phi^{-1}(\overline{\textsc{fpr}}_{\mathrm{LiRA}})^{2}}\frac{\langle x,x-m_{x }\rangle}{\sqrt{S}||x||s}dx$ (A127) $\displaystyle=\frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}\Phi^{-1}(\overline{\textsc{ fpr}}_{\mathrm{LiRA}})^{2}}\frac{1}{\sqrt{S}}\int_{\mathbb{D}}\Pr(x)\frac{ \langle x,x-m_{x}\rangle}{||x||s}dx$ (A128) $\displaystyle=\frac{1}{\sqrt{S}}e^{-\frac{1}{2}\Phi^{-1}(\overline{\textsc{fpr }}_{\mathrm{LiRA}})^{2}}\mathbb{E}_{(x,y_{x})\sim{\mathbb{D}}}\left[\frac{ \langle x,x-m_{x}\rangle}{\sqrt{2\pi}||x||s}\right].$ (A129) $$

Note that here we fixed $\textsc{fpr}_{\mathrm{LiRA}}(x)=\overline{\textsc{fpr}}_{\mathrm{LiRA}}$ for all $x$.
Then we obtain

$$ $\log(\overline{\textsc{tpr}}_{\mathrm{LiRA}}-\overline{\textsc{fpr}}_{\mathrm{ LiRA}})\approx-\frac{1}{2}\log S-\frac{1}{2}\Phi^{-1}(\textsc{fpr})^{2}+\log \left(\mathbb{E}_{(x,y_{x})\sim{\mathbb{D}}}\left[\frac{\langle x,x-m_{x} \rangle}{\sqrt{2\pi}||x||s}\right]\right).$ (A130) $$

∎

###### Proof.

### C.6 Proof of Corollary 6

See 6

###### Proof.

By theorem 4 and the law of unconscious statistician, we have for large $S$

$$ $\displaystyle\overline{\textsc{tpr}}_{\mathrm{RMIA}}-\overline{\textsc{fpr}}_{ \mathrm{RMIA}}$ $\displaystyle=\int_{\mathbb{D}}\Pr(x)(\textsc{tpr}_{\mathrm{RMIA}}(x)-\textsc{ fpr}_{\mathrm{RMIA}}(x))dx$ (A131) $\displaystyle\leq\int_{\mathbb{D}}\Pr(x)\frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2} \Phi^{-1}(\alpha)^{2}}\frac{\psi(x,C)}{\sqrt{S}}dx$ (A132) $\displaystyle=\frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}\Phi^{-1}(\alpha)^{2}}\int_{ \mathbb{D}}\frac{\psi(x,C)}{\sqrt{S}}dx.$ (A133) $\displaystyle=\frac{1}{\sqrt{S}}e^{-\frac{1}{2}\Phi^{-1}(\alpha)^{2}}\mathbb{E }_{(x,y_{x})\sim{\mathbb{D}}}\left[\frac{\psi(x,C)}{\sqrt{2\pi}}\right].$ (A134) $$

Hence we obtain

$$ $\log(\overline{\textsc{tpr}}_{\mathrm{RMIA}}-\overline{\textsc{fpr}}_{\mathrm{ RMIA}})\leq-\frac{1}{2}\log S-\frac{1}{2}\Phi^{-1}(\alpha)^{2}+\log\left( \mathbb{E}_{(x,y_{x})\sim{\mathbb{D}}}\left[\frac{\psi(x,C)}{\sqrt{2\pi}} \right]\right).$ (A135) $$

∎

###### Proof.

## Appendix D LiRA vulnerability for Small FPR

In Section 3.4 we proved for the simplified model that for LiRA $\log(\textsc{tpr}-\textsc{fpr})\approx-\frac{1}{2}\log S$ ignoring additive constants when fpr is fixed (Corollary 5).
In Section 4.3 we observed that the coefficient $\beta_{S}$ for $\log S$ is around $-0.5$ for larger fpr, aligning with the theoretical value.
However, for smaller fpr the coefficient is smaller than $-0.5$.
To understand this phenomenon, it is important to note that the power-law (Theorem 3 and Corollary 5) only holds for sufficiently large $S$.
Thus, it can be explained that the difference of coefficient values $\beta_{S}$ for small and large fpr comes from the small-$S$ regime as follows.

In the proof of Corollary 5 and Theorem 3 the only approximation that could introduce some bias is Equation A84. That is,

$$ $\displaystyle\textsc{tpr}_{\mathrm{LiRA}}(x)-\textsc{fpr}_{\mathrm{LiRA}}(x)$ $\displaystyle=\Pr\left(\eta\leq\Phi^{-1}(\textsc{fpr}_{\mathrm{LiRA}}(x))+r \right)-\Pr\left(\eta\leq\Phi^{-1}(\textsc{fpr}_{\mathrm{LiRA}}(x))\right)$ (A136) $\displaystyle\approx p_{\eta}\left(\Phi^{-1}(\textsc{fpr}_{\mathrm{LiRA}}(x)) \right)r,$ (A137) $$

where $\eta\sim\mathcal{N}(0,1)$, $r$ is a shift that scales $O(1/\sqrt{S})$ and $p_{\eta}$ is the pdf of $\eta$.
Figure A.1 numerically illustrates this approximation.
It can be observed that for small $S$ the approximation does not hold, underestimating the true vulnerability.
Particularly, this effect is remarkably larger for small $\textsc{fpr}_{\mathrm{LiRA}}(x)$.
In other words, the simplified model overestimates the coefficient value $\beta_{S}$ for small $\textsc{fpr}_{\mathrm{LiRA}}(x)$ by underestimating the true vulnerability in the small-$S$ regime.

Figure: (a) $\textsc{fpr}_{\mathrm{LiRA}}(x)=0.1$
Refer to caption: x7.png

## Appendix E Training details

### E.1 Parameterization

We utilise pre-trained feature extractors BiT-M-R50x1 (R-50) (Kolesnikov et al., ) with 23.5M parameters and Vision Transformer ViT-Base-16 (ViT-B) (Dosovitskiy et al., ) with 85.8M parameters, both pretrained on the ImageNet-21K dataset (Russakovsky et al., ). We download the feature extractor checkpoints from the respective repositories.

Following Tobaben et al. () that show the favorable trade-off of parameter-efficient fine-tuning between computational cost, utility and privacy even for small datasets, we only consider fine-tuning subsets of all feature extractor parameters. We consider the following configurations:

- •
Head: We train a linear layer on top of the pre-trained feature extractor.
- •
FiLM: In addition to the linear layer from Head, we fine-tune parameter-efficient FiLM (Perez et al., ) adapters scattered throughout the network. While a diverse set of adapters has been proposed, we utilise FiLM as it has been shown to be competitive in prior work (Shysheya et al., ; Tobaben et al., ).

#### E.1.1 Licenses and access

The licenses and means to access the model checkpoints can be found below.

- •
BiT-M-R50x1 (R-50) (Kolesnikov et al., ) is licensed with the Apache-2.0 license and can be obtained through the instructions on [https://github.com/google-research/big_transfer](https://github.com/google-research/big_transfer).
- •
Vision Transformer ViT-Base-16 (ViT-B) (Dosovitskiy et al., ) is licensed with the Apache-2.0 license and can be obtained through the instructions on [https://github.com/google-research/vision_transformer](https://github.com/google-research/vision_transformer).

### E.2 Hyperparameter tuning

Our hyperparameter tuning is heavily inspired by the comprehensive few-shot experiments by Tobaben et al. (). We utilise their hyperparameter tuning protocol as it has been proven to yield SOTA results for (DP) few-shot models.
Given the input $\mathcal{\mathcal{D}}$ dataset we perform hyperparameter tuning by splitting the $\mathcal{\mathcal{D}}$ into 70$\%$ train and 30$\%$ validation.
We then perform the specified iterations of hyperparameter tuning using the tree-structured Parzen estimator (Bergstra et al., ) strategy as implemented in Optuna (Akiba et al., ) to derive a set of hyperparameters that yield the highest accuracy on the validation split.
This set of hyperparameters is subsequently used to train all shadow models with the Adam optimizer (Kingma & Ba, ).
Details on the set of hyperparameters that are tuned and their ranges can be found in Table A2.

**Table A2: Hyperparameter ranges used for the Bayesian optimization with Optuna.**
|  | lower bound | upper bound |
| --- | --- | --- |
| batch size | 10 | $|\mathcal{D}|$ |
| clipping norm | 0.2 | 10 |
| epochs | 1 | 200 |
| learning rate | 1e-7 | 1e-2 |

### E.3 Datasets

Table A3 shows the datasets used in the paper. We base our experiments on a subset of the the few-shot benchmark VTAB (Zhai et al., ) that achieves a classification accuracy $>80\%$ and thus would considered to be used by a practitioner. Additionally, we add CIFAR10 which is not part of the original VTAB benchmark.

**Table A3: Used datasets in the paper, their minimum and maximum shots $S$ and maximum number of classes $C$ and their test accuracy when fine-tuning a non-DP ViT-B Head. The test accuracy for EuroSAT and Resics45 is computed on the part of the training split that is not used for training the particular model due to both datasets missing an official test split. Note that LiRA requires $2S$ for training the shadow models and thus $S$ is smaller than when only performing fine-tuning.**
| dataset | (max.) | min. | max. | test accuracy | test accuracy |
| --- | --- | --- | --- | --- | --- |
|  | $C$ | $S$ | $S$ | (min. $S$) | (max. $S$) |
| Patch Camelyon (Veeling et al., ) | 2 | 256 | 65536 | 82.8% | 85.6% |
| CIFAR10 (Krizhevsky, ) | 10 | 8 | 2048 | 92.7% | 97.7% |
| EuroSAT (Helber et al., ) | 10 | 8 | 512 | 80.2% | 96.7% |
| Pets (Parkhi et al., ) | 37 | 8 | 32 | 82.3% | 90.7% |
| Resics45 (Cheng et al., ) | 45 | 32 | 256 | 83.5% | 91.6% |
| CIFAR100 (Krizhevsky, ) | 100 | 16 | 128 | 82.2% | 87.6% |

#### E.3.1 Licenses and access

The licenses and means to access the datasets can be found below. We downloaded all datasets from TensorFlow datasets [https://www.tensorflow.org/datasets](https://www.tensorflow.org/datasets) but Resics45 which required manual download.

- •
Patch Camelyon (Veeling et al., ) is licensed with Creative Commons Zero v1.0 Universal (cc0-1.0) and we use version 2.0.0 of the dataset as specified on [https://www.tensorflow.org/datasets/catalog/patch_camelyon](https://www.tensorflow.org/datasets/catalog/patch_camelyon).
- •
CIFAR10 (Krizhevsky, ) is licensed with an unknown license and we use version 3.0.2 of the dataset as specified on [https://www.tensorflow.org/datasets/catalog/cifar10](https://www.tensorflow.org/datasets/catalog/cifar10).
- •
EuroSAT (Helber et al., ) is licensed with MIT and we use version 2.0.0 of the dataset as specified on [https://www.tensorflow.org/datasets/catalog/eurosat](https://www.tensorflow.org/datasets/catalog/eurosat).
- •
Pets (Parkhi et al., ) is licensed with CC BY-SA 4.0 Deed and we use version 3.2.0 of the dataset as specified on [https://www.tensorflow.org/datasets/catalog/oxford_iiit_pet](https://www.tensorflow.org/datasets/catalog/oxford_iiit_pet).
- •
Resics45 (Cheng et al., ) is licensed with an unknown license and we use version 3.0.0 of the dataset as specified on [https://www.tensorflow.org/datasets/catalog/resisc45](https://www.tensorflow.org/datasets/catalog/resisc45).
- •
CIFAR100 (Krizhevsky, ) is licensed with an unknown license and we use version 3.0.2 of the dataset as specified on [https://www.tensorflow.org/datasets/catalog/cifar100](https://www.tensorflow.org/datasets/catalog/cifar100).

### E.4 Compute resources

All experiments but the R-50 (FiLM) experiments are run on CPU with 8 cores and 16 GB of host memory. The training time depends on the model (ViT is cheaper than R-50), number of shots $S$ and the number of classes $C$ but ranges for the training of one model from some minutes to an hour. This assumes that the images are passed once through the pre-trained backbone and then cached as feature vectors. The provided code implements this optimization.

The R-50 (FiLM) experiments are significantly more expensive and utilise a NVIDIA V100 with 40 GB VRAM, 10 CPU cores and 64 GB of host memory. The training of 257 shadow models then does not exceed 24h for the settings that we consider.

We estimate that in total we spend around 7 days of V100 and some dozens of weeks of CPU core time but more exact measurements are hard to make.

## Appendix F Additional results

In this section, we provide tabular results for our experiments and additional figures that did not fit into the main paper.

### F.1 Additional results for Section 4

This Section contains additional results for Section 4.

#### F.1.1 Vulnerability as a function of shots

This section displays additional results to Figure 1(a) for fpr $\in\{0.1,0.01,0.001\}$ for ViT-B and R-50 in in Figures A.2, A4 and A5.

Figure: (a) ViT-B Head $\textsc{tpr}-\textsc{fpr}$ at $\textsc{fpr}=0.1$
Refer to caption: x9.png

Table: Table A4: Median MIA vulnerability over six seeds as a function of $S$ (shots) when attacking a Head trained without DP on-top of a ViT-B. The ViT-B is pre-trained on ImageNet-21k.

Table: Table A5: Median MIA vulnerability over six seeds as a function of $S$ (shots) when attacking a Head trained without DP on-top of a R-50. The R-50 is pre-trained on ImageNet-21k.

#### F.1.2 Vulnerability as a function of the number of classes

This section displays additional results to Figure 1(b) for fpr $\in\{0.1,0.01,0.001\}$ for ViT-B and R-50 in in Figures A.3, A6 and A7.

Figure: (a) ViT-B Head $\textsc{tpr}-\textsc{fpr}$ at $\textsc{fpr}=0.1$
Refer to caption: x15.png

Table: Table A6: Median MIA vulnerability over 12 seeds as a function of $C$ (classes) when attacking a Head trained without DP on-top of a ViT-B. The Vit-B is pre-trained on ImageNet-21k.

Table: Table A7: Median MIA vulnerability over 12 seeds as a function of $C$ (classes) when attacking a Head trained without DP on-top of a R-50. The R-50 is pre-trained on ImageNet-21k.

#### F.1.3 Data for FiLM and from scratch training

Table: Table A8: MIA vulnerability data used in Figure 4(b). Note that the data from Carlini et al. () is only partially tabular, thus we estimated the tpr at fpr from the plots in the Appendix of their paper.

#### F.1.4 Predicting dataset vulnerability as function of S 𝑆 S italic_S and C 𝐶 C italic_C

This section provides additional results for the model based on Equation 27

Table: Table A9: Results for fitting Equation A138 with statsmodels Seabold & Perktold () to ViT Head data at fpr $\in\{0.1,0.01,0.001,0.0001,0.00001\}$. We utilize an ordinary least squares. The test $R^{2}$ assesses the fit to the data of R-50 Head.

Figure A.4 shows the performance for all considered fpr.

Figure: (a) ($(\textsc{tpr}-\textsc{fpr})$ at $\textsc{fpr}=0.1$
Refer to caption: x21.png

### F.2 Simpler variant of the prediction model

The prediction model in the main text (Equation 27) avoids predicting $\textsc{tpr}<\textsc{fpr}$ in the tail when $S$ is very large. In this section, we analyse a variation of the regression model that is simpler and predicts $\log_{10}(\textsc{tpr})$ instead of $\log_{10}(\textsc{tpr}-\textsc{fpr})$. This variation fits worse to the empirical data and will predict $\textsc{tpr}<\textsc{fpr}$ for high $S$.

The general form this variant can be found in Equation A138, where $\beta_{S},\beta_{C}$ and $\beta_{0}$ are the learnable regression parameters.

$$ $\log_{10}(\textsc{tpr})=\beta_{S}\log_{10}(S)+\beta_{C}\log_{10}(C)+\beta_{0}$ (A138) $$

Table A10 provides tabular results on the performance of the variant.

Table: Table A10: Results for fitting Equation A138 with statsmodels Seabold & Perktold () to ViT Head data at fpr $\in\{0.1,0.01,0.001,0.0001,0.00001\}$. We utilize an ordinary least squares. The test $R^{2}$ assesses the fit to the data of R-50 Head.

Figure A.5 plots the performance of the variant similar to Figure 4(a) in the main text.

Figure: (a) tpr at $\textsc{fpr}=0.1$
Refer to caption: x26.png

### F.3 Empirical results for RMIA

Figures A.6, A.7 and A.8 report additional results for RMIA Zarifzadeh et al. ().

Figure: (a) $(\textsc{tpr}-\textsc{fpr})$ at $\textsc{fpr}=0.1$
Refer to caption: x31.png

Figure: (a) $(\textsc{tpr}-\textsc{fpr})$ at $\textsc{fpr}=0.1$
Refer to caption: x34.png

Figure: (a) $(\textsc{tpr}-\textsc{fpr})$ at $\textsc{fpr}=0.1$
Refer to caption: x37.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1.	As the authors mention in the introduction, prior work has already explored the impact of the number of examples per class (shots) on MIA vulnerability. How does this study differ from previous work, and what new insights does it provide beyond existing findings?
2.	While LiRA and RMIA represent state-of-the-art approaches in MIA, it is also essential to assess the generalizability of the presented results to other MIA techniques. How might other methods differ in vulnerability patterns under the same dataset conditions?
3.	As a widely used method for privacy protection, it is necessary to discuss the impact of differential privacy on the arguments presented in this paper.
soundness: 3
strengths: 1.	The paper formalizes the effect of dataset properties on membership inference vulnerability for two advanced MIA methods (LiRA and RMIA), enriching the analytical framework for MIA.
2.	The study includes a comprehensive and well-designed set of experiments across multiple datasets.
summary: This paper investigates the impact of dataset properties on membership inference attacks (MIAs). The authors analyze two state-of-the-art MIA methods, LiRA and RMIA, by deriving a relationship between dataset properties (e.g., number of examples per class) and MIA vulnerability. They introduce a simplified model for membership inference and demonstrate that a power-law relationship exists between per-example MIA vulnerability and the dataset's properties. The authors also fit a regression model to predict MIA vulnerability, providing insights into privacy risks for fine-tuned classifiers.
weaknesses: 1.	In the abstract and introduction, the authors emphasize that their focus is on the privacy analysis of fine-tuned image classification models. However, sections 2 and 3 lack specific analyses that are tailored to fine-tuned neural networks, relying instead on a general theoretical model.
2.	The authors need to clarify the differences in membership inference vulnerability between standard image classification models and fine-tuned models. This would strengthen the argument for fine-tuning as a key focus in their work.
3.	As the regression model is presented as one of the main contributions, the methods section would benefit from a more detailed description of this model, its design, and its implementation process.
4.	Although LiRA and RMIA are currently the best methods, it is also necessary to analyze other MIA methods to validate the generalizability of the proposed arguments.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 4
questions: NA
soundness: 3
strengths: First, I'd like to note that I did enjoy reading the paper: it is clearly written, nicely presented, and well executed. Particularly the plots are extremely visually appealing.

Overall, the attempts to provide some interesting theoretical underpinning for a long known and well studied problem.
summary: The paper proposes a theoretical result that links the success of membership inference attacks (MIAs) to the number of training data points per class. This theoretical link is shown for the RMIA and LiRA attack, and empirical evaluation is performed through fine-tuning vision encoders on multiple downstream datasets.
weaknesses: The core weaknesses of the paper can be grouped as follows:

**Studying a well established problem, while not taking prior work into account**

The underlying causes of MIA risks have been studied in a long line of work since 2017. The findings made in the paper, showing that vulnerability decreases as you have more samples per class, and with fewer classes have been made already. Hence, the expected impact and relevance of the presented findings is negligible. Given the enormous amounts of prior work, it is particularly surprising that the paper does not even have a related work section.

This section could start at early work. E.g. [1] (2021) has an extremely related approach where they assign privacy risk scores that implicitly expresses a similar phenomenon: with the probability of a sample being part of the training set yielding a per point privacy risk: here, it is over classes. Additionally, they add a very valuable angle (not considered in this present work here excessively): Classes with less members have on average lower probability: "the class labels with high generalization errors tend to have higher privacy risk scores".
Given the results by [2], this seems extremely crucial to analyze. For the given work, I am wondering how much this power law connection really holds when data is not well curated. We can imagine a really "malformed" class that consists of extremely rare and outlier data points with high variance. How much will having more of those really help reduce membership risk?
Starting from these works, a related work section could help put the given work into context.

Additionally, another line of work that needs to be taken into account is the one around [3,4,5] where it has been shown that with less data points per class, the class is more vulnerable and that privacy attacks have disparate success.

**Generality of Results**

While LiRA and RMIA are *current* state-of-the-art, it is, undoubtably reasonable to derive the bounds for those concrete attacks. However, the results are presented "as is", i.e., without embedding them into any broader context and without generalization. This makes it questionable, how well the results will age, i.e., how much can we benefit from them in 2-3 years, when LiRA and RMIA are obsolete and have been replaced by new and more powerful attacks.

In a similar vein, a suggestion would be to improve the writing in Section 3.4. The results are interesting, but it is not discussed what they mean: "Hence we can have the LiRA vulnerability arbitrarily close to zero in this non-DP setting by simply increasing S" -> this seems to mean that if every class has more and more train data, the membership risk decreases. Formulating such intermediate "checkpoints" in plain English would be genuinely helpful to position the results.

**Evaluation done for fine-tuning might skew the results**

Overall, I am not convinced that the chosen evaluation setup is sensible for the presented results. The paper considers transfer learning where the heavy lifting is done by a pre-trained vision encoder. However, the pre-training data itself has a distribution that will play a role for the vulnerability of the fine-tuning data. Data points that have high overlap with the classes / concepts present in the pre-training will have a significantly different vulnerability than data from an entirely different distribution. Hence, to underline the theoretical results, the correct approach, would have been training from scratch.

While the paper tries to argue that the pre-training encoder has no influence ("Figure 4a shows that the regression model is robust to a change of the feature extractor"), this statement is too broad and not backed up by results: The experiments were conducted on only two encoders, both pre-trained on ImageNet. These encoders will have the same biases, and will expose the fine-tuning data equally. If any, to make such a statement, it would be required to assess more encoders, especially ones trained on entirely different distributions, including non-object centric dataset.

The fact, that the expectations do not match the from-scratch training ("underestimates the vulnerability of the from-scratch trained target models") is another indicator that comparing with fine-tuning is not the right approach. 


**Minor Comments**

- In the intro, the paper says: "and apply two state-of-the-art MIAs", it would be good to cite them there already for the readers to understand what to expect
- Would it make sense to add the values from the theoretical model in Figure 3, such that one can compare how much the empirical one deviates?

**References**

[1] Song, L., & Mittal, P. (2021). Systematic evaluation of privacy risks of machine learning models. In 30th USENIX Security Symposium (USENIX Security 21) (pp. 2615-2632).

[2] Yeom, Samuel, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. "Privacy risk in machine learning: Analyzing the connection to overfitting." In 2018 IEEE 31st computer security foundations symposium (CSF), pp. 268-282. IEEE, 2018.

[3] Suriyakumar, Vinith M., Nicolas Papernot, Anna Goldenberg, and Marzyeh Ghassemi. "Chasing your long tails: Differentially private prediction in health care settings." In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 723-734. 2021.

[4] Bagdasaryan, Eugene, Omid Poursaeed, and Vitaly Shmatikov. "Differential privacy has disparate impact on model accuracy." Advances in neural information processing systems 32 (2019).

[5] Kulynych, Bogdan, Mohammad Yaghini, M. Cherubin, G. Veale, and C. Troncoso. "Disparate vulnerability: On the unfairness of privacy attacks against machine learning." In 22st Privacy Enhancing Technologies Symposium. 2022.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see Weaknesses above.
soundness: 4
strengths: 1. Membership inference is an important attribute that can explain vulnerability of machine learning models to privacy attacks. The extensive theoretical results help provide generalizable and explainable interpretations of impacts of multiple characteristics of datasets on model vulnerability. 

2. Theoretical results are presented in a manner that is easy to follow for an informed reader, and supported by detailed proofs in the Appendix. 

3. Extensive experiments on multiple datasets validate theoretical claims. The datasets have been carefully chosen to encompass different numbers of classes and image domains. 

4. The work presented has the potential for high impact beyond traditional computer science, for example, balancing the requirements of various regulatory regimes such as GDPR with a need to obtain `useful' information from possibly sensitive data.
summary: This paper investigates the relationship between inherent properties of a dataset, such as the number of classes and the number of samples belonging to each class, and its impact on privacy. Privacy is examined through the perspective of membership inference attacks (MIAs), which seek to determine if a sample has been used to train a model of interest. The authors present theoretical results that establish a power-law relation between dataset properties and vulnerability to MIAs, and support their claims with experimental evaluations.

****
Updated score after examining author response.
*****
weaknesses: 1. The authors indicate that the paper considers two types of MIAs. There have been a wide variety of MIAs examined in the literature, including some which may not rely on confidence scores (e.g., label-based MIAs). Is there a claim (perhaps speculative) that the authors can make about similar relationships for other classes of MIAs? 

2. The experiments focus on datasets and models related to image classification. Can the results be extended to other domains, such as audio or video? Multi-modality can strengthen the scope of the proposed contribution. 

3. The authors point out that analyses and experimental results rely on an assumption that the datasets are balanced. Some perspective on what might happen if this is not the case can shed more light on real-world applicability of the work. 

4. A more detailed discussion or comparison of the effect of dataset characteristics on other aspects of privacy- e.g., differential privacy, which has an extensive literature and supported by theoretical guarantees- will strengthen the submission. Could the authors comment on this?

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: see above
soundness: 3
strengths: - The work presents comprehensive experiments on privacy vulnerabilities using two SOTA MIA methods. It conducts a fine-grained analysis of vulnerabilities at the individual sample level. 
- The findings are robustly validated across diverse architectures and datasets.
- The study provides theoretical analyses of a power-law relationship between dataset properties (e.g., shots per class) and MIA vulnerability.
summary: The study investigates privacy vulnerabilities through Membership Inference Attacks (MIA), with a particular focus on how dataset size affects model privacy. Through detailed per-sample evaluations, the researchers discovered that models become more resistant to privacy attacks as the number of examples per class increases, following a power law distribution. To further examine this phenomenon, the researchers evaluated the findings using few-shot learning and developed a regression model to predict a model's vulnerability to MIA based on dataset properties, such as the number of classes and examples per class.
weaknesses: 1. The paper needs to clarify its motivation. While it discusses transfer learning and few-shot learning, it's unclear whether these techniques are studied as potential privacy protection methods or whether the authors simply examine their privacy risks. The introduction introduces these learning approaches but doesn't explicitly position them against existing differential privacy (DP) defenses. This makes it difficult to understand whether the main goal is to propose new privacy protection strategies or to analyze privacy vulnerabilities in these methods.

2. The paper's organization should be reversed. It would be clearer to show the experimental results first and then explain the theory behind them rather than the other way around. 

3. The authors need to revise their claimed contributions. For example, their first contribution about analyzing privacy attacks at the individual sample level isn't new - other researchers have already done similar work (such as [1], [2]).

4. It is unsurprising that larger datasets exhibit lower overall vulnerability to MIA, as models trained on larger datasets tend to rely less on individual samples, reducing the risk of memorization. 

5. The authors predicted the overall dataset vulnerability using a regression model with properties such as the number of classes and examples per class. While it offers useful insights,  individual-level vulnerabilities are often more actionable for user privacy protections. It's much more meaningful to investigate the per-sample vulnerabilities. However, considering varying dataset sizes, it may be challenging to derive consistent per-sample results, as the same sample could behave as an outlier in some datasets but not in others.

[1] Carlini, Nicholas, et al. "The privacy onion effect: Memorization is relative." Advances in Neural Information Processing Systems 35 (2022): 13263-13276.

[2] Wen, Rui, Michael Backes, and Yang Zhang. "Understanding Data Importance in Machine Learning Attacks: Does Valuable Data Pose Greater Harm?." arXiv preprint arXiv:2409.03741 (2024).

Meta-review: Summary: This paper studies how the number of examples per class and the number of classes impact the membership inference attacks (MIA), using two state-of-the-art MIA attacks, LiRA and RMIA. The paper finds a power-law relationship between those properties and MIA vulnerability. Empirical evaluation is performed through fine-tuning vision encoders on multiple downstream datasets.

Strengths:
1. What matters in MIA vulnerability is an important topic to the ICLR community. The paper identifies the number of examples per class and the number of classes as the key factors, with theoretical underpinning.
2. The paper is well-written and organized, with rich experimental results.

Weaknesses:
Reviewers have common concerns on the paper:
1. Though LiRA and RMIA are two state-of-the-art MIA methods, there are more attacks in the literature. The paper does not test on other MIA methods to validate the generalizability of the proposed arguments.
2. The paper highlights that it focuses on fine-tuned models. Reviewers have concerns on it as reviewers believe the training data of the pre-trained models might also impact MIA vulnerability. Instead, the authors are encouraged to do the test on models trained from scratch.
3. Discussion of related work is missing in the paper. Reviewers raised concern as they find previous work has sent a similar message as this paper (e.g., the number of examples per class and the number of classes matter in MIA vulnerability). Therefore, reviewers believe the paper fails to provide enough new messages to the community.

Three out of four reviewers vote for strong rejection with a score of 3. Following the majority vote, AC would recommend rejection. AC would encourage the authors to take the above weaknesses into account in the next revised version.

Decision: Reject