TransMIA: Membership Inference Attacks Using
Transfer Shadow Training
Seira Hidano

Takao Murakami

Yusuke Kawamoto

KDDI Research, Inc., Saitama, Japanse-hidano at kddi-research.jp

AIST, Tokyo, Japan

AIST, Tokyo, Japan

Abstract—Transfer learning has been widely studied andgained increasing popularity to improve the accuracy of machinelearning models by transferring some knowledge acquired indifferent training. However, no prior work has pointed outthat transfer learning can strengthen privacy attacks on machine learning models. In this paper, we propose TransMIA(Transfer learning-based Membership Inference Attacks), which
use transfer learning to perform membership inference attacks
on the source model when the adversary is able to access theparameters of the transferred model. In particular, we propose atransfer shadow training technique, where an adversary employs
the parameters of the transferred model to construct shadow
models, to significantly improve the performance of membership
inference when a limited amount of shadow training data isavailable to the adversary. We evaluate our attacks using two real
datasets, and show that our attacks outperform the state-of-theart that does not use our transfer shadow training technique. We
also compare four combinations of the learning-based/entropybased approach and the fine-tuning/freezing approach, all ofwhich employ our transfer shadow training technique. Thenwe examine the performance of these four approaches basedon the distributions of confidence values, and discuss possible
countermeasures against our attacks.
Index Terms—Privacy, membership inference attack, transfer
learning, deep learning, shadow training

I. I NTRODUCTION
A number of recent machine learning (ML) systems arebuilt upon a large amount of training data. In many practicalapplications, however, we may not have sufficient training
data due to high expense or difficulty in data acquisition.
Transfer learning [1]–[6] has been widely studied to improve
the accuracy of machine learning systems by incorporatingsome knowledge gained through solving a different problem.
For example, we may transfer a part of an ML model trainedby a data-rich organization (e.g., big company, large hospital)
to another ML model in a different organization (e.g., startup
company, small hospital) that has only a small amount of data
and wants to significantly improve the accuracy of the lattermodel. We refer to the former model as a source model, andthe latter as a target model.
In many practical ML systems, training data involve sensitive personal data; e.g., face images, medical data. A large
dataset in the source data-rich organization can also include
some sensitive personal data; e.g., face images in a large-scale
image dataset for object recognition. When a source modelis trained using sensitive personal data, transfer learning canraise a serious privacy concern. In particular, General Data

Protection Regulation (GDPR) [7] requires an adequate levelof protection when an organization transfers personal data to
another organization. Therefore, protecting privacy of users inthe source model is an important issue in transfer learning
across organizations.
Privacy in transfer learning has been recently studied in theliterature. For example, privacy-preserving transfer learningalgorithms have been proposed in [8]–[12]. A privacy attack
against transfer learning has also been studied in a recent
paper [13]. Specifically, the authors in [13] studied a riskof membership inference [14], in which an adversary infers
whether or not personal data of a particular user is used fortraining a source model. They showed by experiments that theadversary who has black-box access to a target model (i.e.,
who can obtain the target model’s output upon an arbitrary
input) infers membership information in the target model withhigh accuracy, but cannot infer membership information in thesource model in the black-box access setting.
However, it remains open whether the adversary can infermembership information of the source model in the white-box
access setting, in which the adversary can access the architecture and parameters of the transferred model (e.g., a partof the source model). The privacy risk analysis in the whitebox access setting is important to understand the “adequate”
level of protection in transfer learning across organizations.
The analysis in the white-box setting is also essential in that
all the parameters of the transferred model may be leaked by
the target organization via illegal access or internal fraud. Thisissue cannot be overlooked, since the number of data breaches
is rapidly increasing in recent years [15], [16].
In this paper, we focus on deep learning, one of the mostpopular approaches in machine learning, and study the risk ofmembership inference against transfer learning systems in thewhite-box access setting. Specifically, we focus on networkbased deep transfer learning [2], [3], [6], which trains a deepneural network in the source domain (i.e., source model) andreuses a part of the neural network to construct another neuralnetwork in the target domain (i.e., target model). Then we
model and analyze an adversary who has white-box access tothe transferred part of the source model.
In this analysis, we assume that the source organization does
not reveal all parameters of the whole source model to general consumers, since otherwise membership inference attacks
would be performed efficiently [17]. Then the adversary does

not have white black-box access to the whole source model.
Instead, we assume that the adversary has white-box accessonly to the transferred part of the source model to analyze
the privacy risk in model transfer across organizations; e.g.,
the privacy risk when the target organization, which obtains
the transferred model by signing a contract with the source
organization, leaks the transferred model by illegal access orinternal fraud. In this setting, the adversary obtains the source
model’s output by querying arbitrary input, and infers whether
or not personal data of a particular user have been used to train
the source model.
One might think that if the adversary has the white-box
access to the transferred part of the source model, then theadversary can easily perform the membership inference attack
without the black-box access to the whole source model.
However, this is not the case. Specifically, Nasr et al. [17]showed that a white-box membership inference attack thatuses the outputs of individual layers does not provide better
accuracy than the black-box attack. Then they proposed awhite-box attack using the gradient of the prediction loss
with regard to all parameters, and showed that the proposed
attack outperforms the black-box attack. However, this attack
cannot be performed in our setting where the adversary does
not have a white-box access to the whole source model,
because the adversary cannot calculate the gradient by the
back-propagation algorithm in this case. For this reason, we
explore a new membership inference algorithm under theassumption of the white-box access to the transferred modeland the black-box access to the whole source model.
Under this assumption, we show that the adversary can alsotransfer the knowledge in the same way as the target organization to significantly improve the accuracy of membership
inference attacks. Specifically, we propose a new techniquecalled transfer shadow training, which transfers the parametersof the transferred model to a shadow model [14] that imitates
the behavior of the source model and for which the adversary
knows the training dataset. This technique is helpful for theadversary especially when the amount of shadow training data
is small. For example, the adversary may not have sufficient
shadow training data due to high expense or difficulty inacquiring personal data (e.g., face images, medical data), justlike the target organization. By experiments using two real
datasets, we demonstrate that our new technique enables suchan adversary to perform the membership inference attack
against transfer learning with high accuracy.
Contributions. Our main contributions are as follows:• We propose TransMIA (Transfer learning-based Membership Inference Attacks), new attacks that use transfer
learning to perform membership inference attacks on thesource model. In particular, we propose a transfer shadow
training technique, which transfers the parameters of thetransferred model to a shadow model, to significantly
improve the accuracy of membership inference when
the amount of shadow training data is small. To ourknowledge, this work is the first to use transfer learning
to strengthen privacy attacks on the source model. Zou etal. [13] proposed a privacy attack against transfer learning(in a black-box access setting), whereas we propose a
privacy attack using transfer learning (in a white-box
access setting).
• We evaluate our membership inference attacks by experiments using two real datasets, and show that our membership inference attacks are realistic and effective threats
to the source model especially when the adversary has
a limited prior knowledge on the datasets. Specifically,
our attacks outperform a state-of-the-art method [14] that
does not use our transfer shadow training technique.
• We compare four types of our membership inference
attacks to investigate the advantages and disadvantagesof these attacks. Specifically, we evaluate the four combinations of the learning-based/entropy-based approachand the fine-tuning/freezing approach, all of which aretrained with our transfer shadow training technique. Then
we show that the learning-based approach has muchhigher accuracy than the entropy-based approach. We also
show that the attack with the fine-tuning approach may
overfit shadow training datasets when the shadow training
datasets are small. Furthermore, we discuss the results ofthese four approaches on the basis of confidence values,
and discuss possible countermeasures for our attacks.
A. Related Work
Membership inference attacks. Shokri et al. [14] studied amembership inference attack which, given an individual data
record and black-box access to a model, determines if therecord was used to train the model. Here they proposed a
shadow training technique, which trains shadow models that
imitate the behavior of the target model. Inspired by [14], anumber of studies have been made on this attack [17]–[23].
For example, Jayaraman et al. [23] assumed a realistic scenario
where the candidate pool from which the adversary samples
records is imbalanced, and proposed a membership inference
attack based on the direction of change in loss of a queryrecord when it is perturbed with a small amount of noise.
They showed that their attack outperforms previous attacks inimbalanced prior settings. However, their attack still considers
only the black-box access to the model.
Nasr et al. [17] evaluated white-box membership inference
attacks against deep learning algorithms. They first showedthat a straightforward extension of the black-box attack to thewhite-box setting, which uses the outputs of individual layers(rather than the outputs of the last layers), does not provide
better accuracy than the black-box attack. Then they proposed
a white-box attack using the gradient of the prediction loss
with regard to all parameters, and showed that it outperforms
the black-box attack. Their proposed attack cannot be usedin our setting where the adversary does not have a whitebox access to ht , because the adversary cannot calculate thegradient by the back-propagation algorithm in this case.
Recently, Song et al. [24] proposed a membership inference
attack based on modified prediction entropy (MPE). They
showed that the MPE-based attack outperforms the DNN-

based attack in their experiments. On the other hand, theMPE-based attack results in low accuracy in our experiments,
because the distributions of confidence values are similar
between training and test datasets and a single threshold does
not separate the two distributions.
Defences against membership inference attacks. Defences
against membership inference attacks have also been widely
studied; e.g., regularization-based defenses [14], [25], [26], DP(Differential Privacy)-based defenses [10], [27]–[29], adding
noise to (or transforming) a confidence score vector [30],[31], and DAMIA [32]. The regularization-based defenses use
regularization techniques to reduce overfitting; e.g., l2 -norm
regularization [14], dropout [26], and adversarial regularization [25]. The DP-based defenses aim to prevent membership inference attacks from adversaries with any backgroundknowledge. The regularization-based defenses and the DPbased defenses have no formal utility-loss guarantees, and candecrease the utility when we want to achieve high privacy.
In the DP-based defenses, the privacy budget  tends to belarge to achieve high utility. For example, Rahman et al. [33]reported that centralized deep learning in [27] requires  ≥ 4
to achieve high utility, and that it is vulnerable to membership
inference attacks in this case.
Jia et al. [30] proposed MemGuard, which adds noise to aconfidence score vector based on adversarial example. Theirdefense bounds the loss of utility and provides better privacyutility trade-off than the state-of-the-art regularization-based
defenses and DP-based defenses. However, this defense cannot
be applied to the whole parameters in the transferred model
gs . The same applies to the defense in [31].
Finally, Huang et al. [32] proposed DAMIA based on DA(Domain Adaptation), which uses non-sensitive training datafrom a source domain to obfuscate sensitive training data froma target domain. Specifically, it works as follows: (i) remove
labels of sensitive training data; (ii) train a model via DA. In
step (ii), it uses as input sensitive training data without labels
and non-sensitive training data with labels.
Our work totally differs from [32] in that ours uses transfer
learning to increase the performance of the membership inference attack. Another crucial difference lies in source and target
tasks; since DA assumes that the source and target tasks are thesame [4], DAMIA is limited to the setting where two datasets
are collected for the same task (e.g., two-class classificationof cats and dogs). In contrast, the source and target tasks canbe different in our setting; e.g., two datasets can be used forclassification of different image labels.
Transfer learning. Transfer learning has been widely studied
over the past few decades (see [4]–[6] for surveys). Privacypreserving transfer learning has also been studied in recent
years; e.g., a method based on homomorphic encryption andsecret sharing [8], methods based on DP [9]–[12].
However, no studies have used transfer learning tostrengthen privacy attacks on the source model, to our knowledge. We proposed a transfer shadow training technique, andshowed that transfer learning can be used for such purposes‫ܦ‬ୱ୲୰ୟ୧୬

݂ୱ ݃ୱ

݄ୱ
outputinput
transfer‫୲୲ܦ‬୰ୟ୧୬

݂୲ ݃୲

݄୲
outputinput

Fig. 1: Deep transfer learning. After a source model fs istrained using a dataset Dstrain , the freezing approach initializes
gt as gs and trains only ht using another dataset Dttrain(gt = gs ). In contrast, the fine-tuning approach initializes gt
as gs and updates the whole network (gt 6= gs ).
when the amount of shadow training data is limited.
II. P RELIMINARIES
A. Basic Notations and Machine Learning
In this paper we deal with supervised learning models withclassification tasks. We define an input space X as a set offeature vectors, an output space Y as a set of class labels. Let
P be the distribution of all data points over X × Y. Then alearning algorithm seeks a function that maps a feature vectorx ∈ X to its class label y ∈ Y. We denote by DY the set of
all distributions over Y. Then each distribution in
can bePDY|Y|
represented as a real vector (p1 , . . . , p|Y| ) with i=1 pi = 1.
A classification model is a function f : X → DY that maps
a feature vector x to a prediction vector f (x) that associates
each class label y ∈ Y with
P a classification confidence value
f (x)y ∈ [0, 1] such thaty∈Y f (x)y = 1. The final classification result is the class label with the largest confidence
value, i.e., argmaxŷ f (x)ŷ .
To evaluate how badly the classification model f fits eachdata point (x, y), we employ a real-valued loss function l (e.g.,cross entropy). Given a training dataset Dtrain ⊆ X × Y, thetraining aims to find a classification model f that minimizes
the expected loss LDtrain (f ) = E(x,y)∼Dtrain [l(f (x), y)]. To
evaluate the model f , a test dataset Dtest ⊆ X × Y is used tocompute LDtest (f ) = E(x,y)∼Dtest [l(f (x), y)].
B. Deep Transfer Learning
Transfer learning [1]–[6] is an approach to improving theperformance of a machine learning task on a target domain bytransferring some knowledge from a source domain. Networkbased deep transfer learning [2], [3], [6] is a branch of transfer
learning that pre-trains a deep neural network fs from a large
training dataset Dstrain in the source domain and reuses a partof fs to construct another network ft with a significantly small
dataset Dttrain in the target domain. See Fig. 1 for an overview.
Formally, we define network-based deep transfer learning
as follows. Let X be an input space, Ys (resp. Yt ) be a source(resp. target) output space, and Ps (resp. Pt ) be the distribution

of the data points over X × Ys (resp. X × Yt ). Then a sourceclassification model is a deep neural network fs : X → DYs
that outputs a prediction vector v providing each label y with aclassification confidence value vy ∈ [0, 1], and the source task
is defined as t s = (Ys , fs ). As illustrated in Fig. 1, a source
model fs can be regarded as the cascade of the two networks
gs and hs corresponding to the shallow and deep layers; i.e.,
fs = hs ◦gs . The shallow layers gs extract task-independent lowlevel features (e.g., edges, shapes), while the deep layers hs
represent task-specific high-level features (e.g., eyes, mouths).
Analogously to the source domain, we also define a target
model ft = ht ◦ gt and a target task t t = (Yt , ft ).
In network-based deep transfer learning, a source model fs
is first trained using a dataset Dstrain ⊆ X × Ys and then its
shallow layers gs are reused in the training of a target modelft with another dataset Dttrain ⊆ X × Yt (See Fig. 1). We call
Dstrain a source training dataset, and Dttrain a target training
dataset. In later sections we also use a source test dataset
Dstest ⊆ X × Ys and a target test dataset Dttest ⊆ X × Yt .
In this work we deal with two approaches: freezing andfine-tuning. In the freezing approach, we initialize and fix theshallow layers gt as gs and update only the deep layers in the
training of the target model ft . Then the training aims to finda ht that minimizes the expected loss LPt (ht ◦ gs ). In contrast,in the fine-tuning approach, we initialize the shallow layers gt
as gs and update the whole network in the training of ft . Thenthe training aims to find a ht ◦ gt that minimizes LPt (ht ◦ gt ).
Empirically, the fine-tuning approach has better accuracy whentarget task labels are plentiful, whereas the freezing approachis better for scarce target task labels [34], [35].
C. Membership Inference Attack
We next review a privacy attack on machine learning models, called a membership inference attack (MIA) [14], wherean adversary is not given a training dataset Dtrain and attempts
to reveal membership information on Dtrain . Specifically, theadversary in this attack aims to determine whether a specificdata point (x, y) is included in a training dataset Dtrain used
to build a classification model f .
The membership inference attack exploits a different behavior of the model f when a given data point (x, y) has been used
to train f . To build an adversary A against f , we attempt tolearn some statistical relevance between the distribution f (x)of confidence values and the membership (x, y) ∈ Dtrain .
In this attack, the adversary A is provided access to themodel f and some dataset disjoint from the training dataset
Dtrain . An adversary is said to have black-box access to f if
it can query data x to f and obtain their prediction vectors
f (x). In contrast, white-box access to f allows the adversary
to obtain the internal structure of f itself (e.g., the weights ofthe connections between nodes when f is a neural network).
Formally, an adversary in the black-box membership inference attack on a model f is a function A : X × Y × DY →{in, out} that given a data point (x, y) and its prediction
vector f (x), outputs one of the labels in and out, where in(resp. out) represents that A predicts (x, y) ∈ Dtrain (resp.

(x, y) 6∈ Dtrain ). To evaluate the performance of the membership inference attack, we measure the accuracy, precision,and recall of the attack by using a test dataset Dtest ⊆ X × Y
disjoint from the training dataset Dtrain .
We deal with two approaches to constructing a membership
inference adversary: the learning-based approach [14] andthe entropy-based approach [24]. The former constructs anadversary A as a classification model obtained by supervised
learning using a dataset other than Dtrain . The latter approachcalculates a variant of the Shannon entropy of the prediction
vector (defined in Section II-D) and determine the membershipby checking whether this entropy exceeds a certain threshold.
D. Modified Prediction Entropy
Finally, we review a recently proposed measure called
modified prediction entropy [24], which can be used to build amembership inference attack that may outperform the existing
DNN-based attacks in some settings shown in [24]. Intuitively,
this measure is a variant of the Shannon entropy of a prediction
vector f (x) that also takes into account the correct class label
y. Formally, the modified prediction entropy mpe(v, y) of aprediction vector v ∈ DY w.r.t. a label y ∈ Y is defined by:P
mpe(v, y) = −(1 − vy ) log(vy ) − y0 6=y vy0 log(1 − vy0 ).
III. ATTACK S CENARIOS
We consider a small startup company (or small hospital)
Csmall that has only small datasets of images (e.g., face images,
X-ray images) and wants to achieve higher performance ofimage classification. Then a big company (or large hospital)
Cbig , that possesses large datasets of images, provides a partof their ML model to the startup Csmall . Specifically, Cbig pretrains a deep neural network fs from a large training dataset
Dstrain in the source domain, and the startup Csmall reuses theshallow layers gs of fs to construct another network ft using
a significantly small dataset Dttrain in the target domain. The
big company Cbig also offers a machine learning as a service(MLaaS) to general consumers via a black-box API. This
service enables the general consumers to query an arbitrary
input to obtain fs ’s output. Note that the big company Cbig does
not allow white-box access to the whole source model fs to
the general consumers, because otherwise a lot of membership
information can be leaked by the parameters of fs [17].
Our attack scenario is that the startup company Csmall , whichobtains the transferred model gs by signing a contract with the
big company Cbig , leaks the shallow layers gs e.g., by illegal
access, malware infection, or internal fraud. Note that even
if the startup company Csmall trains ft with the fine-tuning
approach, gs can be leaked by Csmall . Then we consider anadversary A who has black-box access to the deep neuralnetwork fs through an API and has all the parameters of theshallow layers gs for the purpose of transfer learning (Fig. 2).
The adversary A performs a black-box membership inference
attack on fs to learn whether a specific sensitive image x
is included in the training dataset Dstrain . Although gs alone
extracts only task-independent low-level features and may notdirectly be used to construct membership inference attacks, we‫ܥ‬ୠ୧୥ (black-box API)݂ୱ ݃ୱ‫ܥ‬ୱ୫ୟ୪୪
݃ୱ݄ୱ
outputinput

transfertransfer‫ܥ‬ୱ୫ୟ୪୪

୲୰ୟ୧୬,௜‫ܦ‬ୱ୦ୟୢ୭୵

݃ୱleakage

Adversary ‫ܣ‬Fig. 2: Our attack scenario. After the big company Cbigtransfers the shallow layers gs to the small startup company
Csmall , the adversary A obtains gs from Csmall (e.g., via illegal
access). The adversary A also obtains the output of fs via theblack-box API provided by Cbig . The adversary A performs
membership inference attack on fs using these information.
show that gs can be used to strengthen the shadow training
to mount a stronger adversary against fs . This implies thattransfer learning can increase the vulnerability to the privacy
breach from the source model fs .
IV. M EMBERSHIP I NFERENCE ATTACKS BASED ON THE
T RANSFER S HADOW T RAINING T ECHNIQUE
In this section we propose new membership inference
attacks TransMIA where attackers perform transfer learning in
the attack scenarios in Section III. In particular, we propose
a technique called transfer shadow training to significantly
improve the performance of the attacks when only a small
amount of shadow training data is available to the adversary.
A. Threat Model
We investigate a membership inference attack against a deepneural network fs that consists of shallow layers gs and deep
layers hs ; i.e., fs = hs ◦ gs . (See Fig. 1.) This attack aims
to determine whether a specific data point (x, y) is includedin the source training dataset Dstrain . In Section IV-C, weconstruct an adversary A who has (1) black-box access tothe network fs through an API, (2) white-box access to theshallow layers gs for the purpose of transfer learning, (3) noaccess to the deep layers hs , and (4) may not necessarily
know hs ’s underlying architecture or hyper parameters. In
our experiments in Section V, we construct shadow models
that have the same architecture as hs . However, as shown inprevious work such as [14], the knowledge of the architecture
is not necessary to construct shadow models.
We assume that the adversary A possesses a shadow training
train
dataset Dshadow⊆ X × Ys to train (possibly multiple) shadow
train
models. The elements of Dshadow
are drawn from the same
distribution Ps as the source training dataset Dstrain . Since thegoal of the attack is to reveal membership information on
Dstrain , the adversary A has no prior knowledge of Dstrain itself,train
hence of whether a data point (x, y) ∈ Dshadow
is included௜݂ୱ୦ୟୢ୭୵
௜݃ୱ୦ୟୢ୭୵

input௜݄ୱ୦ୟୢ୭୵

outputFig. 3: Overview of the transfer shadow training.
train
in Dstrain or not. Typically, Dshadow
is disjoint from Dstrain .
Note that such a threat model is also assumed in most of theprevious work on membership inference attacks (e.g., [14]).
We also assume that the adversary A possesses a shadow
test
train
test dataset Dshadow⊆ X ×Ys disjoint from Dshadow
and Dstest .
train
The adversary A uses the shadow training dataset Dshadow andtest
the shadow test dataset Dshadow
to train an attack model.
train
Finally, we assume that both the shadow datasets Dshadow
test
and Dshadow consist of a small amount of data. This assumption is reasonable, because it is expensive or difficult for
the adversary to acquire a large amount of personal data,(e.g., face images, medical data), just like the small startup
train
company Csmall . When Dshadow
is small, it is challenging forthe adversary to perform membership inference attacks withhigh accuracy. To address this issue, we propose a techniquecalled transfer shadow training, which we describe below.

B. Transfer Shadow Training
We now propose a transfer shadow training technique. As
shown in Fig. 3, the adversary A obtains the shallow layers gs
from the small startup company Csmall (e.g., via illegal access),
as described in Section III. Then the adversary A transfers
this information to shadow models to obtain stronger attack
performance with a small amount of shadow training data.
Specifically, we train multiple shadow models to simulate
the behavior of the original classification model fs underattacks. We denote by cshadow ∈ N the number of the shadowimodels. For each i = 1, 2, . . . , cshadow , let fshadow
be the ii
th shadow model. We regard fshadow as a cascade of shallow
i
i
i
i
◦
layers gshadow
and deep layers hshadow; i.e., fshadow= hshadow
train,ii
train
gshadow . We use a dataset Dshadow ⊆ Dshadow to train the i-thishadow model fshadow, and refer to it as the shadow trainingidataset for fshadow . As with [14], the training datasets fordifferent shadow models may intersect.
i
To construct each shadow model fshadow, our transferishadow training technique initializes the shallow layers gshadowi
as gs . Then it trains the shadow model fshadow by using thetrain,idataset Dshadow. In other words, we perform the network-based
deep transfer learning to construct the shadow model. As wewill show in our experiments, this significantly improves theaccuracy of membership inference even when the amount oftrainshadow training data Dshadow
is small. In the transfer shadowitraining we may take the freezing approach (gshadow= gs ) ori
the fine-tuning approach (gshadow 6= gs ).

C. Attack Algorithms
Next we describe algorithms for TransMIA (Transfer
learning-based Membership Inference Attacks), membership
inference attacks using our transfer shadow training technique.
Here we deal with two types of adversaries: learning-basedand entropy-based. This is because in a recent work [24],entropy-based membership inference attacks are proposed andshown to outperform the DNN-based membership inference
attack in their experiments. We denote our membership inference attacks with the learning-based approach and the entropybased approach by TransMIAL and TransMIAE , respectively.
To construct the attack algorithms, we introduce a shadow test
test,itest
idataset Dshadow⊆ Dshadowfor each shadow model fshadow. Aswith the shadow training datasets, the test datasets for different
shadow models may also intersect.
First, we construct a learning-based adversary using ourtransfer shadow training technique in the following three steps.
TransMIAL (learning-based adversary)i(L1) We first train multiple shadow models fshadow
for i =
1, 2, . . . , cshadow . Specifically, we initialize the shallowiilayers gshadow
as gs , and then train fshadow
by using thetrain,itrain
dataset Dshadow ⊆ Dshadow .
train(L2) We next obtain a dataset Dattacklabeled with membership information (in or out) by applying each shadow
train,iimodel fshadow
to the shadow training dataset Dshadow
test,itest
and the shadow test dataset Dshadow ⊆ Dshadow . Speciftrain
ically, we define an attack training dataset Dattackias the set of the records (x, y, fshadow (x), in) for eachtrain,ii(x, y) ∈ Dshadow(x0 ), out) for eachand (x0 , y 0 , fshadow
test,i0 0(x , y ) ∈ Dshadow where i = 1, 2, . . . , cshadow .
(L3) We finally construct a membership inference adversarymodel A : X × Ys × DYs → {in, out} as a classification
model (e.g., DNN, SVM) using the attack training
train
dataset Dattack
constructed in (L2).
Given an input (x, y) ∈ X × Ys , the learning-based adversary
TransMIAL obtains fs (x) by black-box access to the source
model fs and outputs the label A(x, y, fs (x)).
Next, we construct an entropy-based adversary with transfer
shadow training in the following three steps.
TransMIAE (entropy-based adversary)i(E1) We first train multiple shadow models fshadow
for i =
1, 2, . . . , cshadow in the same way as in (L1).
i(E2) We next employ each shadow model fshadow
and calcuilate the modified prediction entropy mpe(fshadow(x), y)train,itest,ifor each (x, y) ∈ Dshadow ∪ Dshadow . Then for each classlabel y ∈ Ys , we select a threshold τy that maximizes:
Pcshadow
train,iii=1 |{(x, y) ∈ Dshadow : mpe(fshadow (x), y) ≤ τy }|test,ii+ |{(x, y) ∈ Dshadow: mpe(fshadow(x), y) > τy }|.

Then τy maximizes the accuracy of the attack.
We finally define a membership inference adversary
A : X × Ys × DYs → {in, out} by A(x, y, fs (x)) = in if
mpe(fs (x), y) ≤ τy , and A(x, y, fs (x)) = out otherwise.
Given an input (x, y) ∈ X × Ys , the entropy-based adversary
TransMIAE obtains fs (x) by black-box access to the source
model fs and outputs the label A(x, y, fs (x)).
In each of TransMIAL and TransMIAE , the transfer shadow
training may take the freezing or fine-tuning approach. Therefore, there are four types of our membership inference attacks,
which we compare in our experiments. As a membership
inference adversary model A in TransMIAL , we use a deepneural network (DNN) and a support vector machine (SVM).
(E3)

V. E VALUATION
A. Datasets and Source Models
We conducted experiments to evaluate our attacks by using
two real datasets that are typically used as source datasetsfor experimental evaluation: ImageNet and VGGFace2 [36].
While several pre-trained classification models based on thesedatasets are provided, we construct the source models fromscratch. This is because the exact source training datasets used
for constructing the models are required for the evaluation.
In this section we explain these datasets and the sourceclassification models on which we perform attacks.
ImageNet. ImageNet is a large-scale image dataset designed
for visual object recognition research. We especially use the
ImageNet Large Scale Visual Recognition Challenge dataset
2012 (ILSVRC2012) [37], which is a trimmed ImageNet
dataset consisting of 1,000 categories of images. We randomly
select 100 categories from all categories with more than 800
images. We split the dataset into four disjoint datasets: atraining dataset Dstrain , a test dataset Dstest , a shadow training
train
test
, and a shadow test dataset Dshadow. Both Dstrain
dataset Dshadow
test
and Ds consist of 50 images for each category, and 5,000
train
test
images in total. The shadow datasets Dshadow
and Dshadow
have 62,013 and 62,010 images, respectively. We constructa classification model fs with ResNet18 [38]. We iterate thetraining of fs over 120 epochs, and save the updated model
every epoch. We finally select a model with the best accuracy
from the saved models as fs . The accuracy of the model fs for
Dstrain and Dstest are 0.298 and 0.242, respectively.
VGGFace2. VGGFace2 is a large-scale face recognition
dataset with more than 9,000 identities. We randomly select100 identities from the identities with more than 500 images.
As with the ImageNet dataset, we split the dataset into four
train
test
disjoint datasets Dstrain , Dstest , Dshadow, and Dshadow. Both
train
test
Ds
and Ds consist of 50 images for each identity, andtrain
5,000 images in total. The shadow datasets Dshadow
andtest
Dshadow have 22,623 and 22,680 images, respectively. We
assume a task to predict the identity for a given face image.
We construct a model fs with ResNet18 [38]. We iterate thetraining of fs over 120 epochs, and select the best model in thesame way as the ImageNet dataset. The accuracy of the modelfs for Dstrain and Dstest are 0.855 and 0.752, respectively.

C. Experimental Results
We first present experimental results on our transfer
learning-based attack TransMIAL(DNN) with the DNN andcompare it with the baseline approach BaseMIA.
Fig. 4 shows accuracy, precision, and recall. Here weevaluated three approaches: BaseMIA, TransMIAL(DNN) withthe freezing approach, TransMIAL(DNN) with the fine-tuning
approach. “Baseline5000” (dotted line) indicates the result of
train,iBaseMIA when each shadow training dataset Dshadow
consistsof 5,000 images, i.e., when the adversary utilizes shadow
training datasets with the same size as Dstrain .

0.60

0.70

0.58

0.65

0.56
0.54

BaselineFreezing
Fine-tuning

0.52
0.50

200

400

600

800

Accuracy

Accuracy

Baseline5000
Baseline5000

0.60
0.55

BaselineFreezing
Fine-tuning

0.50
0.45

1000

200

600

800

1000

Size of shadow training dataset(a) Accuracy in ImageNet.

(d) Accuracy in VGGFace2.

0.75

0.80
0.75

0.70
0.65

Baseline5000

0.60

BaselineFreezing
Fine-tuning

0.55
0.50

200

400

600

800

1000

Precision

0.70

Precision

400

Size of shadow training dataset

Baseline5000

0.65
0.60

BaselineFreezing
Fine-tuning

0.55
0.50
0.45

200

400

600

800

1000

Size of shadow training dataset

Size of shadow training dataset(b) Precision in ImageNet.

(e) Precision in VGGFace2.
1.0

0.6

0.8

Baseline5000

Recall

0.7

0.4
0.3

BaselineFreezing
Fine-tuning

0.2
0.1

BaselineFreezing
Fine-tuning

0.9

0.5

RecallB. Experimental Setup
In this section we describe the shadow models and attack
models. We assume a scenario where all the layers of the target
model fs except the last fully connected layer are published as
gs for transfer learning. The adversary leverages gs for transfer
shadow training. We apply the same settings to both target
models generated with the ImageNet and VGGFace2 datasets.
We construct shadow models with three learning approaches: baseline, freezing, and fine-tuning. The baseline
approach, denoted by BaseMIA, is the original membership
inference attack proposed by Shokri et al. [14], where we do
not utilize any layers of the transferred model gs to constructshadow models. The freezing and fine-tuning approaches are
instances of our proposed methods described in Section IV-B.
In our experiments, we construct 100 shadow models foreach learning approach (i.e., cshadow = 100). For each shadow
train,iimodel fshadow, we prepare a shadow training dataset Dshadow
test,iand a shadow test dataset Dshadow
by randomly selecting
train
test
images of Dshadow and Dshadow , respectively. Each shadowimodel fshadow
is trained with the corresponding shadow traintrain,itrain,iing dataset Dshadow. In our experiments, the size of Dshadowis100, 200, 300, 400, 500, 600, 800, or 1,000. We also constructtest,iia shadow test dataset Dshadow
for fshadow
with the same size
train,itrain
as Dshadow . As a result, the attack training dataset Dattackhas
the same number of records with the in and out labels.
We perform our transfer learning-based membership inference attack with the learning-based approach and the entropybased approach. For learning-based approach, we construct
two types of attack models: a deep neural network (DNN) anda support vector machine (SVM), denoted by TransMIAL(DNN)
and TransMIAL(SVM) respectively. In our experiments, weevaluate TransMIAL(DNN) , TransMIAL(SVM) , and TransMIAE .
The neural network has three fully-connected hidden layers.
The numbers of neurons for the hidden layers are 50, 30, and5. For each class label y, we obtain an attack training dataset
train
Dattack
and construct the attack model (i.e., 100 attack models
in total). We train each attack model with 50 epochs.
We evaluate attack performance with accuracy, precision,and recall. For each dataset size, we construct 10 attack
train
training datasets Dattack
while varying images selected from
train
test
Dshadow and Dshadow . We then perform each attack 10 times
with different attack training datasets. We average the resultsof 10 attacks for each performance metric. We use Dstrain andDstest as test datasets for the attack models.

200

400

600

800

1000

0.6

Baseline5000

0.5
0.4
0.3
0.2
200

400

600

800

1000

Size of shadow training dataset

Size of shadow training dataset(c) Recall in ImageNet.

(g) Recall in VGGFace2.

Fig. 4: Accuracy, precision, and recall of the baseline attack
BaseMIA and our attacks TransMIAL(DNN) with the learningbased approach (DNN). In TransMIAL(DNN) , we evaluate bothfreezing and fine-tuning approaches. Figures (a)–(c) (resp. (d)–(f)) are the results in the ImageNet (resp. VGGFace2) dataset.
Each line indicates the average of results for 100 classes.
Error bars indicate the standard deviation of results on 10
attacks with different shadow training datasets. The dotted line
annotated as “Baseline5000” indicates the result of BaseMIA
when the size of the shadow training dataset is 5,000. The
dashed lines and outline markers in Figures (c) and (f) indicatethat the corresponding precision value is less than 60%.

Fig. 4 shows that our attack TransMIAL(DNN) with thefreezing/fine-tuning approach achieves higher accuracy andprecision than the attack with the baseline approach BaseMIA.
In addition, the accuracy and precision of our attack with thefreezing approach are over the values of baseline5000 whentrain,ithe size of each shadow training dataset Dshadow
is greaterthan or equal to 300. These results show that transfer shadow
training is effective for the attack with a small shadow traintrain,iing datasets Dshadow. In our experiments with the ImageNet
dataset, the accuracy of the fine-tuning approach is worse thanthat of the freezing approach. The reason is that the fine-tuning
approach empirically tends to overfit the target training dataset
when the dataset is small [34], [35].
It should be noted that although the recall of the baseline
approach BaseMIA is high at the small sizes, the corre-sponding precision is very low: less than 0.6 (and therefore,
we show the recall in this case using the dashed line).
When the precision is close to 0.5, the adversary fails toinfer membership information, irrespective of the recall. For
example, consider an adversary who randomly outputs “in”
(member) with probability p ∈ [0, 1] and “out” (non-member)with probability 1 − p. The accuracy, precision, and recall of
this random guess are 0.5, 0.5 and p, respectively. Thus, thisrandom guess achieves recall = 1 when p = 1. However,it completely fails to infer membership information (in fact,
precision is 0.5). Similarly, BaseMIA outputs “in” many times
and this explains high recall of BaseMIA. However, BaseMIA
behaves like the random guess with high p, and fails to infermembership information (precision is close to 0.5).
Fig. 4 shows that the recall of our membership inference
attack TransMIAL(DNN) is low at the small sizes. This istrain,ibecause shadow training datasets Dshadow
include few data
points that are similar to source training data. However, therecall of TransMIAL(DNN) increases with an increase in thesize of the shadow training dataset. When the size is 1,000,
the recall of the freezing approach is close to the value ofbaseline5000. Since the size of 1000 is lower than the fullsize (i.e., 5,000), our attack is effective in terms of recall as
well. Note that our attack also achieves high accuracy andprecision; i.e., when the size is 1,000, TransMIAL(DNN) withthe freezing approach achieves higher accuracy and precision
than baseline5000, and almost the same recall as baseline5000
in both the ImageNet and VGGFace2 datasets.
We next present experimental results on our attacks with
the DNN, SVM, and the entropy-based approach, i.e., TransMIAL(DNN) , TransMIAL(SVM) , and TransMIAE . Fig. 5 shows
the accuracy of our attacks with the three different attack
models. Fig. 5 shows that the learning-based approaches
TransMIAL(DNN) and TransMIAL(SVM) have much higher accuracy than the entropy-based approach TransMIAE . However, TransMIAL(SVM) with the fine-tuning approach results in
low accuracy for the ImageNet dataset. As explained above,ieach shadow model fshadow
constructed with the fine-tuning
approach on the ImageNet dataset overfits shadow training
train,idataset Dshadow. In this case, the decision boundary between
in and out data points included in the attack training dataset is
distorted. It is not easy to construct a well-trained model fromsuch a complicated structure of data with a linear model suchas an SVM. As a result, the attack results in low accuracy.
The accuracy of the entropy-based approach TransMIAE
is below 55% for any settings. While the entropy-based
approach achieves high accuracy for some applications [24],the learning-based approach is more effective than the entropybased approach in our transfer learning setting.
D. Discussions on our attacks based on the confidence values
In this section, we discuss the results of our attacks shownin Section V-C based on the distributions of confidence values.
Fig. 6 shows the distributions of confidence values of theisource model fsi and shadow models fshadow
for a singleiclass in the VGGFace2 dataset. The shadow models fshadow

are constructed with shadow training datasets consisting of
1,000 images. The graphs (b), (c), and (d) in Fig. 6 present
confidence values for all 100 shadow models. In these graphs,
the shadow models are constructed with BaseMIA, TransMIAL(DNN) with the freezing approach, and TransMIAL(DNN)with fine-tuning approach, respectively. Each graph consistsof both distributions for training and test data.
Freezing vs fine-tuning. Fig. 6 shows that our transfer
shadow training (i.e., the freezing and fine-tuning approaches)trains shadow models better than the baseline. The distributions on the freezing approach are the closest to those on thesource model. As a result, the freezing approach achieves thebest accuracy among three shadow training approaches.
The distributions in the fine-tuning approach (Fig. 6-(d)) are
different between training and test datasets when compared tothose in the freezing approach (Fig. 6-(c)). More specifically,
most of the training data have the confidence value of about 1.0
for the correct label y in the fine-tuning approach (Fig. 6-(d)).
In other words, the shadow models constructed with the finetuning approach overfit shadow training data. This explains
the results of Figs. 4 and 5, where the freezing approach
outperforms the fine-tuning approach in many cases.
Thus we conclude that the freezing approach is more
appropriate for transfer shadow training than the fine-tuning
approach when the amount of shadow training data is small.
Learning-based vs entropy-based. Fig. 6-(a) shows thatthe source model is well trained in that the distributions
for training and test datasets are similar. In this case, theconfidence value-based approaches such as MPE (modified
prediction entropy) cannot achieve high accuracy, because asingle threshold does not separate the two distributions.
On the other hand, the learning-based approach can deal
with localized features of the distribution. This is true especially when we use the whole prediction vector as input to the
DNN. Consequently, the learning-based approach can achieve
high accuracy even though the distributions for training andtest datasets are similar. Most of source models in the realworld transfer learning are also well-trained, as large-saledatasets are utilized as the source training dataset. This explains the results of Fig. 5, where the learning-based approach
outperforms the entropy-based approach.
Note that in [24], the MPE (modified prediction entropy)based membership inference attack outperformed the DNNbased membership inference attack in their experiments. Whenthe distributions are very different between training and test
datasets (i.e., when the model overfits the training dataset),
the MPE-based attack can directly use the difference between
the two distributions. In that case, the MPE-based attack canoutperform the DNN-based attack, because it is not easy tofind optimal hyperparameters (e.g., hidden layers, learningrate) in the DNN, as pointed out in [24].
However, when the distributions are similar between training and test datasets, the entropy-based approach cannot
provide high accuracy, as explained above. In this case, weshould use the learning-based approach.

0.58

0.56
0.54

DNN
SVM
MPE

0.52
0.50

200

400

600

800

1000

0.65

0.56
0.54
0.52

0.70

DNN
SVM
MPE

0.60
0.55
0.50

0.50

200

400

600

800

1000

DNN
SVM
MPE

0.65

Accuracy

0.58

0.70

DNN
SVM
MPE

Accuracy

0.60

Accuracy

Accuracy

0.60

0.60
0.55
0.50

0.45

200

400

600

800

1000

0.45

200

400

600

800

1000

Size of shadow training dataset

Size of shadow training dataset

Size of shadow training dataset

Size of shadow training dataset(a) Freezing for ImageNet.

(b) Fine-tuning for ImageNet.

(c) Freezing for VGGFace2.

(d) Fine-tuning for VGGFace2.

Fig. 5: Accuracy of our attacks TransMIAL(DNN) , TransMIAL(SVM) , and TransMIAE with three different attack models: DNN,
SVM, and MPE (modified prediction entropy). The left figures (a) and (c) shows our attacks with the freezing approach,whereas the right figures (b) and (d) shows our attacks with the fine-tuning approach. Figures (a) and (b) (resp. (c) and (d))
show the results in the ImageNet (resp. VGGFace2) dataset. As with Fig. 4, each line indicates averages for 100 classes, anderror bars indicate standard deviations for 10 attacks.

10.0
7.5
5.0

600
400
200

0.0

0.2

0.4

0.6

0.8

1.0

0

0.0

0.2

0.4

0.6

0.8

1.0

150
100

0

Train
Test

500

50

2.5
0.0

Train
Test

200

Frequency

12.5

250

Train
Test

800

Frequency

Frequency

15.0

Frequency

Train
Test

17.5

400
300
200
100

0.0

0.2

0.4

0.6

0.8

1.0

0

0.0

0.2

0.4

0.6

0.8

1.0

Confidence value

Confidence value

Confidence value

Confidence value(a) Source model.

(b) Shadow models (baseline).

(c) Shadow models (freezing).

(d) Shadow models (fine-tuning).

Fig. 6: Distributions of confidence values of source and shadow models for a single class in the VGGFace2 dataset. Figure (a)shows the results on the source model. Figures (b), (c), and (d) show the results on the shadow models constructed with
BaseMIA, TransMIAL(DNN) with the freezing approach, and TransMIAL(DNN) with fine-tuning approach, respectively. Eachgraph shows the frequency of confidence values for the training and test data in that class.

E. Discussions on Countermeasures
We finally discuss possible countermeasures for our transfer
learning-based membership inference attacks.
Zou et al. [13] considered a privacy attack against transfer
learning, and showed through experiments that an adversary
who has black-box access to the target model ft cannot infermembership information about the source training dataset
Dstrain . In [39], we also show that the adversary who has blackbox access to the target model ft cannot infer membership
information about Dstrain using the ImageNet dataset.
We argue that both those results, and our results in Section V
are important to understand the “adequate” level of protection
in transfer learning across organizations (described in Section I). Specifically, if a big company (or large hospital) Cbigtransfers a part of the source model gs to a small company (orsmall hospital) Csmall , then the small company Csmall needs tosufficiently protect the transferred model gs so that the original
parameters of gs are not leaked by Csmall . Otherwise, the riskof membership inference for Dstrain would be increased by ourtransfer learning-based membership inference attacks.
On the other hand, the small company Csmall may allow third
parties or end users to have black-box access to the trainedtarget model ft . In other words, Csmall may provide a machinelearning as a service (MLaaS) via an API. In this case, theadversary who has black-box access to the target model ft
cannot infer membership information about Dstrain , as shown in[13], [39] only by experiments without theoretical guarantees.
We finally note that the big company Cbig may use
regularization-based defenses [14], [25], [26] or DP (Differential Privacy)-based defenses [10], [27]–[29] to obfuscate theparameters of the transferred model gs before providing it
to Csmall . If gs ’s parameters are sufficiently obfuscated, themembership inference attack can be prevented, even if theobfuscated parameters are leaked via illegal access or internal
fraud (just like the local privacy model [40], [41]). However,these approaches have no formal utility-loss guarantees [30]and can deteriorate the utility when we achieve strong privacy.
Furthermore, defenses by adding noise to (or transforming) a
confidence score vector [30], [31] cannot be applied to thewhole parameters of gs . Obfuscating the parameters of gs withhigh utility and privacy is left as future work.
VI. C ONCLUSION
In this paper, we proposed new membership inference
attacks TransMIA where attackers use transfer learning toperform membership inference attacks. In particular, we proposed a transfer shadow training technique, which transfers

the parameters of the transferred model to a shadow model, tosignificantly improve the accuracy of membership inference
when the amount of shadow training data is small. To ourknowledge, this work is the first to show that transfer learning
can strengthen privacy attacks on machine learning models.
As future work, we would like to develop obfuscation
methods for the parameters of the transferred model withhigh utility and privacy, e.g., by extending a game-theoretic
approach [42]. Another line of future work would be toevaluate TransMIA using other types of datasets than image.
ACKNOWLEDGMENT
This work was supported by JSPS KAKENHI Grant Number JP19H04113, by ERATO HASUO Metamathematics for
Systems Design Project (No. JPMJER1603), JST, and by Inria
under the project LOGIS. In our experiments, we used computational resources of the AI Bridging Cloud Infrastructure(ABCI) provided by National Institute of Advanced Industrial
Science and Technology (AIST).
R EFERENCES[1] R. Caruana, “Learning many related tasks at the same time withbackpropagation,” in Proc. NIPS’94, 1994, pp. 657–664.
[2] Y. Bengio, F. Bastien, A. Bergeron, N. Boulanger-Lewandowski, T. M.
Breuel, Y. Chherawala, M. Cissé, M. Côté, D. Erhan, J. Eustache,
X. Glorot, X. Muller, S. P. Lebeuf, R. Pascanu, S. Rifai, F. Savard,
and G. Sicard, “Deep learners benefit more from out-of-distribution
examples,” in Proc. AISTATS’11, 2011, pp. 164–172.
[3] Y. Bengio, “Deep learning of representations for unsupervised andtransfer learning,” in Proc. ICML 2011 Unsupervised and Transfer
Learning Workshop, 2012, pp. 17–36.
[4] S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Trans.
Knowl. Data Eng., vol. 22, no. 10, pp. 1345–1359, 2010.
[5] K. R. Weiss, T. M. Khoshgoftaar, and D. Wang, “A survey of transfer
learning,” J. Big Data, vol. 3, p. 9, 2016.
[6] C. Tan, F. Sun, T. Kong, W. Zhang, C. Yang, and C. Liu, “A survey on
deep transfer learning,” CoRR, vol. 1808.01974, 2018.
[7] The EU General Data Protection Regulation (GDPR), https://eurlex.europa.eu/eli/reg/2016/679/oj, 2016 (accessed in 2020).
[8] D. Gao, Y. Liu, A. Huang, C. Ju, H. Yu, and Q. Yang, “Privacypreserving heterogeneous federated transfer learning,” in Proc. IEEEBig Data’19, 2019, pp. 2552–2559.
[9] X. Guo, Q. Yao, W. Tu, Y. Chen, W. Dai, and Q. Yang, “Privacypreserving transfer learning for knowledge sharing,” CoRR, vol.
1811.09491v1, 2018.
[10] N. Papernot, M. Abadi, U. Erlingsson, I. Goodfellow, and K. Talwar,“Semi-supervised knowledge transfer for deep learning from private
training data,” in Proc. ICLR’17, 2017, pp. 1–16.
[11] Y. Wang, Q. Gu, and D. Brown, “Differentially private hypothesis
transfer learning,” in Proc. ECML PKDD’18, 2018, pp. 1–16.
[12] Q. Yao, X. Guo, J. Kwok, W. Tu, Y. Chen, W. Dai, and Q. Yang,“Privacy-preserving stacking with application to cross-organizational
diabetes prediction,” in Proc. IJCAI’19, 2019, pp. 4114–4120.
[13] Y. Zou, Z. Zhang, M. Backes, and Y. Zhang, “Privacy analysis ofdeep learning in the wild: Membership inference attacks against transfer
learning,” CoRR, vol. abs/2009.04872, 2020.
[14] R. Shokri, M. Stronati, C. Song, and V. Shmatikov, “Membership
inference attacks against machine learning models,” in Proc. IEEE
S&P’17, 2017, pp. 3–18.
[15] C. Morris, “Hackers had a banner year in 2019,”
https://fortune.com/2020/01/28/2019-data-breach-increases-hackers/,
2020.
[16] T.
Seals,“Data
breaches
increase
40%
in2016,”
https://www.infosecurity-magazine.com/news/data-breaches-increase40-in-2016/, 2017.

[17] M. Nasr, R. Shokri, and A. Houmansadr, “Comprehensive privacy
analysis of deep learning: Passive and active white-box inference attacks
against centralized and federated learning,” in Proc. IEEE S&P’19, 2019,
pp. 739–753.
[18] J. Hayes, L. Melis, G. Danezis, and E. D. Cristofaro, “LOGAN: Evaluating privacy leakage of generative models using generative adversarial
networks,” PoPETs, vol. 1, pp. 133–152, 2019.
[19] B. Hilprecht, M. Härterich, and D. Bernau, “Monte carlo and reconstruction membership inference attacks against generative models,” PoPETs,
vol. 4, pp. 232–249, 2019.
[20] L. Song, R. Shokri, and P. Mittal, “Privacy risks of securing machinelearning models against adversarial examples,” in Proc. ACM CCS’19,
2019, pp. 241–257.
[21] J. Li, N. Li, and B. Ribeiro, “Membership inference attacks anddefenses in supervised learning via generalization gap,” CoRR, vol.
abs/2002.12062, 2020.
[22] A. Pyrgelis, C. Troncoso, and E. D. Cristofaro, “Knock knock, who’s
there? membership inference on aggregate location data,” in Proc.
NDSS’18, 2018, pp. 1–15.
[23] B. Jayaraman, L. Wang, K. Knipmeyer, Q. Gu, and D. Evans, “Revisiting membership inference under realistic assumptions,” CoRR, vol.
abs/2005.10881, 2020.
[24] L. Song and P. Mittal, “Systematic evaluation of privacy risks of machinelearning models,” CoRR, vol. abs/2003.10595, 2020.
[25] M. Nasr, R. Shokri, and A. Houmansadr, “Machine learning withmembership privacy using adversarial regularization,” in Proc. ACM
CCS’18, 2018, pp. 634–646.
[26] A. Salem, Y. Zhang, M. Humbert, P. Berrang, M. Fritz, and M. Backes,“ML-Leaks: Model and data independent membership inference attacks
and defenses on machine learning models,” in Proc. NDSS’19, 2019.
[27] M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, “Deep learning with differential privacy,” in Proc.
ACM CCS’16, 2016, pp. 308–318.
[28] R. Shokri and V. Shmatikov, “Privacy-preserving deep learning,” in Proc.
ACM CCS’15, 2015, pp. 1310–1321.
[29] L. Yu, L. Liu, C. Pu, M. E. Gursoy, and S. Truex, “Differentially private
model publishing for deep learning,” in Proc. IEEE S&P’19, 2019, pp.
332–349.
[30] J. Jia, A. Salem, M. Backes, Y. Zhang, and N. Z. Gong, “MemGuard:
Defending against black-box membership inference attacks via adversarial examples,” in Proc. ACM CCS’19, 2019, pp. 259–274.
[31] Z. Yang, B. Shao, B. Xuan, E. Chang, and F. Zhang, “Defending model
inversion and membership inference attacks via prediction purification,”
CoRR, vol. abs/2005.03915, 2020.
[32] H. Huang, W. Luo, G. Zeng, J. Weng, Y. Zhang, and A. Yang, “DAMIA:
leveraging domain adaptation as a defense against membership inference
attacks,” CoRR, vol. abs/2005.08016, 2020.
[33] M. A. Rahman, T. Rahman, R. Laganière, N. Mohammed, and Y. Wang,“Membership inference attack against differentially private deep learningmodel,” Trans. Data Privacy, vol. 11, pp. 61–79, 2018.
[34] P. Agrawal, R. B. Girshick, and J. Malik, “Analyzing the performance ofmultilayer neural networks for object recognition,” in Proc. ECCV’14,
Part VII, 2014, pp. 329–344.
[35] R. B. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature
hierarchies for accurate object detection and semantic segmentation,” inProc. IEEE CVPR’14, 2014, pp. 580–587.
[36] Q. Cao, L. Shen, W. Xie, O. M. Parkhi, and A. Zisserman, “VGGFace2:
A dataset for recognising faces across pose and age,” in Proc. IEEE
FG’18, 2018.
[37] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma,
Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. FeiFei, “ImageNet Large Scale Visual Recognition Challenge,” IJCV, vol.
115, no. 3, pp. 211–252, 2015.
[38] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image
recognition,” CoRR, vol. abs/1512.03385, 2015.
[39] S. Hidano, Y. Kawamoto, and T. Murakami, “TransMIA: Membership Inference Attacks Using Transfer Shadow Training,” CoRR, vol.
abs/2011.14661, 2020.
[40] J. C. Duchi, M. I. Jordan, and M. J. Wainwright, “Local privacy andstatistical minimax rates,” in Proc. IEEE FOCS’13, 2013, pp. 429–438.
[41] S. P. Kasiviswanathan, H. K. Lee, K. Nissim, and S. Raskhodnikova,“What can we learn privately?” in Proc. IEEE FOCS’08, 2008, pp. 531–
540.

ୠ୧୥
௦ 

௦

ℎ௦
RXWSXWLQSXW

WUDQVIHU

௧ 
௧

ℎ௧
RXWSXW

LQSXWୱ୫ୟ୪୪ EODFNER[$3,$GYHUVDU\ ‫ܣ‬Fig. 7: Membership inference attack on the source model fs
by black-box access to the target model ft .
[42] M. S. Alvim, K. Chatzikokolakis, Y. Kawamoto, and C. Palamidessi,“Information leakage games,” in Proc. GameSec’17, 2017, pp. 437–457.
[43] A. Krizhevsky, “Learning multiple layers of features from tiny images,”
Tech. Rep., 2009.
[44] A. Khosla, N. Jayadevaprakash, B. Yao, and L. Fei-Fei, “Novel dataset
for fine-grained image categorization,” in Proc. CVPR 2011 workshop
on FGVC, 2011.

A PPENDIX

train
a shadow training dataset Dshadow⊆ X × Ys and a shadow
test
test dataset Dshadow ⊆ X × Ys whose elements were drawn
from the same distribution Ps as the source training dataset
Dstrain . Since the goal of the attack is to reveal informationon the membership to the source training dataset Dstrain , theadversary A has no prior knowledge of Dstrain itself, hence of
train
test
∪ Dshadow
is included
whether a data point (x, y) ∈ Dshadow
train
in Ds
or not. To consider the strongest possible adversary,
we assume that A possesses the target training dataset Dttrain
and the target test dataset Dttest . We also assume that A knows
whether the small company Csmall constructs a target model ft
with the freezing approach or the fine-tuning approach.

B. Attack Algorithms
Next we present our membership inference attacks against
the target model ft . Here we deal with the learning-based
adversary, which is shown to outperform the entropy-based
adversary in Section V. To construct the attack algorithms, wetrain,itrain⊆ Dshadow
and aintroduce a shadow training dataset Dshadow
test,itest
shadow test dataset Dshadow ⊆ Dshadow for each shadow modeli
fshadow. As with the previous work (e.g., [14]), these shadow
training datasets (resp. shadow test datasets) for different
shadow models may intersect.
We construct a learning-based adversary with black-box
access to the target model ft in the following four steps.

In this appendix, we evaluate membership inference attacks MIAfLt (learning-based adversary accessing ft )on the transfer learning where an adversary has black-box (TL1) We first construct multiple shadow source models
access to the target model ft and attempts to reveal informationi
to simulate the behavior of the source model
fs,shadowon the membership to the source training dataset Dstrain .
fs as follows. For each i = 1, 2, . . . , cshadow , we train a
Section A illustrates an attack scenario where an adversaryishadow source model fs,shadowusing the shadow training
performs membership inference attacks based on black-box
train,ii
dataset Dshadow . Let gs,shadow
be the shallow layers
access to the target model ft , and introduces a threat modelobtained by removing the last fully connected layer fromassumed in this attack. Section B presents the details ofifs,shadow.
the attack algorithms. Finally, Section C shows experimental(TL2) We next construct multiple shadow target modresults.
i
els ft,shadow
by transfer learning to simulate the behavior of the target model ft as follows. For each
A. Attack Scenarios and Threat Modeli = 1, 2, . . . , cshadow , we build a shadow target model
We assume another scenario where the small company (oriias theft,shadow
by initializing its shallow layers gt,shadow
small hospital) Csmall offers an API allowing for a black-box
i
networkg
obtained
in
(TL1)
and
then
trainings,shadow
access to the target model ft that they developed by the transferift,shadow
with the target training dataset Dttrain .
learning. Then an adversary A, who has black-box access to
train
the target model ft , infers whether a specific sensitive image x (TL3) We then obtain a dataset Dattack labeled with membership information (in or out) by applying each shadow
is included in the big company Cbig ’s source training dataseti
train
target model ft,shadow
to the shadow training dataset
Ds(Fig. 7).
train
test
D
and
the
shadow
test dataset Dshadow. SpecifWe investigate a membership inference attack that aims toshadow
train
ically,wedefine
D
as
the
set
of
the
records
determine whether a specific data point (x, y) is included inattack
i
train
train
(x,
y,
f
(x),
in)
for
each
(x,
y)
∈
D
the source training dataset Ds . The adversary A has blackt,shadow
shadow and
0
0 i
0
0 0
test
(x , y, ft,shadow (x ), out) for each (x , y ) ∈ Dshadow
box access to the target network ft through an API. Note that
train
where i = 1, 2, . . . , cshadow . We call Dattack an attack
we assume that the adversary A does not obtain the output of
training dataset.
the source model fs upon an arbitrary input (i.e., A does nothave black-box access to fs , and have black-box access to only (TL4) We finally construct a membership inference adversary
A : X × Ys × DYt → {in, out} as a classification modelft ), because we want to investigate whether the membership(e.g., deep neural network, support vector machine)
information about Dstrain is leaked by only the output of ft .
train
using the attack training dataset Dattack
constructed in
As with the previous work on the membership inference(TL3).
attacks (e.g., [14]), we assume that the adversary A possesses

Freezing
Fine-tuning

0.54
0.52
0.50
0.48

400

600

800

1000

200

Stanford Dogs (Dogs). Dogs is a dataset that has 120 breedsof dog images. We randomly select 100 breeds from all thebreeds. In the same manner as for the CIFAR-100 dataset, weconstruct 8 target training and test datasets, and then train atarget model ft with each training dataset. Table I also shows
the accuracy of each target training model in this dataset.
2) Experimental Setup: We assume a scenario where an
adversary A possesses the target training dataset Dttrain as
described in Section A. In this scenario, the adversary A trainsi1a shadow source model fs,shadow
to obtain its shallow layers
iigs,shadow . Then A constructs a shadow target model ft,shadowi
trainfrom gs,shadow by transfer learning using Dt
as explainedin Section B. We apply the same settings to both the target
models generated with the CIFAR-100 and Dogs datasets.
In our experiments, we construct 100 shadow target models
ift,shadowfor each target model ft by the freezing or fine-tuning
train
approach. Then we produce an attack training dataset Dattack
1 This is the same as the shadow source model f is,shadow constructed in theexperiments of Section V.

400

600

800

1000

Size of shadow training dataset

Size of shadow training dataset(a) Accuracy in CIFAR-100.

(d) Accuracy in Dogs.

Freezing
Fine-tuning

0.54

0.52

Precision

PrecisionFreezing
Fine-tuning

0.54

0.50
0.48

0.50
0.48

0.46

0.46
200

400

600

800

1000

200

400

600

800

1000

Size of shadow training dataset

Size of shadow training dataset(b) Precision in CIFAR-100.

(e) Precision in Dogs.

0.90

0.85

0.85

0.80

0.80

RecallCIFAR-100. CIFAR-100 is an image dataset that has 100
classes each consisting of 600 images. We randomly selectthe same number of images from each class, and construct 8
target training datasets with different sizes. We set the size ofa target training dataset Dttrain to 100, 200, 300, 400, 500, 600,
800, or 1,000. We also construct a target test dataset Dttest that
corresponds to each target training dataset Dttrain and consistsof the same number of images as Dttrain . We train a target
model ft from each target training dataset with the freezing andfine-tuning approaches. In the training of each model, we use
all the layers of fs except the last fully connected layer as thetransferred source model gs . We iterate the training over 120
epochs, and select a model with the best accuracy as ft . Table I
shows the accuracy of the model ft for the target training andtest datasets.

0.50

0.46
200

Recall1) Datasets and Target Models: We construct a target
model ft with the source model fs pre-trained in the experiments of Section V. We especially reuse the model constructedfrom the ImageNet dataset as fs .
As target training datasets Dttrain , we use two real datasets:
CIFAR-100 [43] and Stanford Dogs (Dogs) [44]. In thissection we explain these datasets and the target classification
models ft .

0.52

0.48

0.46

0.52

C. Experimental Evaluation

Freezing
Fine-tuning

0.54

Accuracy

Accuracy

Given an input (x, y) ∈ X ×Ys , the learning-based adversary
MIAfLt obtains ft (x) by black-box access to the target model ft
and outputs the label A(x, y, ft (x)).
Since the adversary A knows whether Csmall constructs atarget model ft with the freezing approach or the fine-tuning
approach (as described in Section A), A uses the same approach as Csmall . That is, if Csmall uses the freezing (resp. finetuning) approach, then A also uses the freezing (resp. finetuning) approach. As a membership inference adversary model
A in MIAfLt , we use a deep neural network (DNN), because it
outperformed a support vector machine (SVM) in Section V.

0.75
0.70

Freezing
Fine-tuning

0.65
0.60

200

400

Freezing
Fine-tuning

0.75
0.70
0.65

600

800

1000

0.60

200

400

600

800

Size of shadow training dataset

Size of shadow training dataset(c) Recall in CIFAR-100.

(f) Recall in Dogs.

1000

Fig. 8: Accuracy, precision, and recall of our attacks TransMIAL(DNN) with the learning-based approach (DNN). In
TransMIAL(DNN) , we evaluate both freezing and fine-tuning
approaches. Figures (a)–(c) (resp. (d)–(f)) are the results in
the CIFAR-100 (resp. Dogs) dataset. Each line indicates theaverage of results for 100 classes. The dashed lines and outline
markers in Figures (c) and (f) indicate that the corresponding
precision value is less than 60%.

as described in Section B. For each class label y, we construct
an attack model, i.e., 100 attack models in total.
We perform membership inference attacks with thelearning-based approach using a DNN. We denote byt
MIAfL(DNN)
the attack with the DNN. The DNN has three fullyconnected hidden layers with 50, 30, and 5 neurons, and istrained with 50 epochs. We evaluate attack performance withaccuracy, precision, and recall.
3) Experimental Results: We present experimental resultston the learning-based membership inference attack MIAfL(DNN)with a DNN. Fig. 8 shows the accuracy, precision, and recallof the attacks with the freezing and fine-tuning approaches.
Those graphs indicate that the accuracy and precision oft
MIAfL(DNN)
are much lower than those of TransMIAL(DNN) ,and those values are very close to 0.5. As mentioned in
Section V-C, when the accuracy and precision are close to0.5, the adversary fails to infer membership information, eventhough the corresponding recall is high.
From these results, we argue that even if the small company

TABLE I: Accuracy of target models in the CIFAR-100 and Dogs datasets.
Size
100
200
300
400
500
600
800
1,000

CIFAR-100
Freezing
Fine-tuning
Train
Test
Train
Test
0.810
0.080
0.500
0.070
0.635
0.105
0.610
0.095
0.473
0.107
0.707
0.090
0.545
0.115
0.703
0.105
0.450
0.118
0.602
0.110
0.555
0.133
0.850
0.128
0.334
0.146
0.914
0.159
0.494
0.165
0.920
0.183

Csmall allows the adversary to have black-box access to thetrained target model ft , the adversary fails to infer membership
information about Dstrain . It is left as future work to provide
a formal proof that the membership information of the source
model fs is not leaked by the output of the trained target
model ft .

Dogs
Freezing
Fine-tuning
Train
Test
Train
Test
0.360
0.040
0.770
0.050
0.505
0.055
0.765
0.055
0.507
0.057
0.407
0.053
0.525
0.065
0.765
0.078
0.392
0.062
0.646
0.074
0.183
0.057
0.280
0.058
0.276
0.075
0.330
0.075
0.316
0.065
0.891
0.091