On the Efficacy of Differentially Private Few-shot Image Classification
Authors: Marlon Tobaben
Aliaksandra Shysheya
John F Bronskill
Andrew Paverd
Shruti Tople
Santiago Zanella-Beguelin
Richard E Turner
Antti Honkela
Venue: Accepted by TMLR ()

Abstract: There has been significant recent progress in training differentially private (DP) models which achieve accuracy that approaches the best non-private models. These DP models are typically pretrained on large public datasets and then fine-tuned on private downstream datasets that are relatively large and similar in distribution to the pretraining data. However, in many applications including personalization and federated learning, it is crucial to perform well (i) in the few-shot setting, as obtaining large amounts of labeled data may be problematic; and (ii) on datasets from a wide variety of domains for use in various specialist settings. To understand under which conditions few-shot DP can be effective, we perform an exhaustive set of experiments that reveals how the accuracy and vulnerability to attack of few-shot DP image classification models are affected as the number of shots per class, privacy level, model architecture, downstream dataset, and subset of learnable parameters in the model vary. We show that to achieve DP accuracy on par with non-private models, the shots per class must be increased as the privacy level increases. We also show that learning parameter-efficient FiLM adapters under DP is competitive with learning just the final classifier layer or learning all of the network parameters. Finally, we evaluate DP federated learning systems and establish state-of-the-art performance on the challenging FLAIR benchmark.

Published in Transactions on Machine Learning Research (12/2023)
On the Efficacy of Differentially Private Few-shot Image Classification
Marlon Tobaben' University of Helsinki
marlon.tobbben@@hckssskii
Aliaksandra Shysheya"
as2975@cam.ccm uk
University of Cambridge
John Bronskill University of Combridge
jfb52 @cam.ac......m uk
Andrew Paverd
endrew. srddbrurpneer@@mmicosssf.tcommm
Microsoft
Shruti Tople
shrutt.oople@mmiccoooftooo
Microsoft
Santiago Béguelnn Microsoft
santiago@microsoft..omm
Richard Turner
ret26@com.ccc uk
University of Cambridge
Antti Honkela
antti.honkela@heessnkiij
University of Helsinki
Reviewed on OpenReview: https: // openreview. net/ forum? id- hFsr59I
Abstract
There has been significant recent progress in training differentially private (DP) models which achiove accuracy that approaches the best non-private models. These DP models are typically pretrained on large public datasets and then fine-tuned private downstream datasets that are relatively large and similar in distribution the pretraining data. However, in many applications including personalization and federated learning, it is crucial to perform well (i) in the few-shot sctting, as obtaining large amounts of labeled data may be problematic; and (ii) on datasets from a wide variely of domains for use in various specialist settings. To understand under which conditions few-shot DP can be effective, we perform an exhaustive set of experiments that reveals how the accuracy and vulnerability to attack of few-shot DP image classification models are affected the number of shots per class, privacy level, model architecture, downstream dataset, and subset of learnable parameters in the model vary. We show that to achieve DP accuracy par with non-private models, the shots per class must be increased as the privacy level increases. We also show that learning parameter-efficien FiLM adapters under DP is competitive with learning just the final classifier layer or learning all of the network parameters. Finally, we evaluate DP fedcrated learning systems and establish state-of-the-art performance on the challenging FLAIR benchmark.
*These authors contributed equally

---

Published in Transactions on Machine Learning Research (12/2023)
1 Introduction
It is well known that neural networks trained without formal privacy guarantees can be attacked to expose a subset of the training data (Carlini et al. 2021 Balle et al. 2022). For applications where training data are sensitive (Abowd 2018 Cormode et al.) 2018 it has become increasingly common to train under Differential Privacy (DP) (woork ct al.) 2006) which is considered to be the gold standard for protecting the privacy of individual training examples. Training with DP stochastic gradient descent (DP-SGD) (Rajkumar &
Agarwal 2012 Song et al. 2013 Abadi et al. 2016), which adapts SGD to guarantee DP, typically impairs model performance due gradient clipping and the addition of noise during training in order to mask the contribution of individual examples to model updates. However, there has been significant recent progress in training DP models which achieve accuracy that approaches the best non-private models in both NL.P LLi
et al. 2022b
Yu et al. 2022) and computer vision (Kurakin et al. 2022 De et al. 2022
Mehta et al. 2022
Cattan et al. 2022
The majority of these approaches are based on transfer learning where the models have been pretrained on large public datasets and then fine-tuned (Yosinski et al. 2014) on a private downstream dataset with DP- SGD, as transfer learning has been shown to be highly effective on non-private data (Kolesnikov el al. 2020
Shysheya et al.) 2022p. In the non-private setting, the subset of model parameters to fine-tune ranges from all model parameters Kolesnikov ct al.; 2020) to only the final layer, with the tuning of parameter-officient
adapters (Perez et al. 2018 Houlsby et al. 2019 Mahabadi et al. 2021) becoming increasingly prevalent.
Transfer learning has also proven successful in the DP setting with YYu et al. 2022) and without (Mehta
et al. 2022) adapters.
However, strong DP results have only been demonstrated with relatively large datasets, with no exlensive DP few-shot studies performed. The few-shot setting is crucial to any application where obtaining large amounts of labeled data is problematic. It is particularly significant in federated learning, where global model is trained using data from multiple distributed users, and personalized federated learning, which involves customizing a federated learning model with a specific user's 's data. In such scenarios, each user's data may be sensitive and of limited size, such as medical images (Sheller et al. 2020), personal photos (Massiceti et al, 2021), or confidential personal data or actions entered on a mobile device (Differential Privacy Team 2017 Ding et al. 2017
In addition, the strong DP transfer learning results that have recently been reported have largely considered the case where the data dist ribution of the downstream dataset is similar to the pretraining data distribution (Tramèr et al. 2022). A more demanding test is out-of-domain transfer where more information needs to be extracted from the downstream datasct, making private learning more challenging. Support for differing data distributions is essential for frequently encountered specialist settings such as medical imaging, Earth imaging, or personalized object recognition.
In this work, we answer the question: Under what conditions differentially private few-shot image classification effective? Our contributions are:
We provide the first comprehensive study the efficacy of DP few-shot image classification. In particular, in the centralized setting we perform an exhaustive set of experiments that reveals how the accuracy of DP and non-private models are alfected as the number of shols per class, privacy level, downstream dataset, model architecture, and the subset of learnable parameters in the model vary. We also investigate whether the trends observed in the centralized setting apply to federated learning. Novel insights include:
Amount of data required: It is known that classification accuracy under DP decreases as the level of privacy increases and the amount of data decreases, however: (i) we quantify how much more data is required under various levels of DP to match non-private accuracy. In particular, we found that the number of shots per class must be increased significantly to match non-private performance, depending on the subset of learnable parameters; and (ii) we show that accuracy under DP is strongly related to the difficultly of the transfer learning task.
Model porameterization: We show that fine-tuning parameter-efficien FILM adapters in addition to the final linear classifier layer performs close to better than fine-tuning all parameters in the model
2

---

Published in Transactions on Machine Learning Research (12/2023)
fine-tuning only the final layer under few-shot DP. This is demonstrated by superior accuracy for the FiLM configuration on the challenging VTAB-1k benchmark and establishing stale-of-the-art in terms of accuracy (macro average precision increased from 44.3% to 51.9%) and communicttion efficiency (cost reduced from 11.9M to 0.017M parameters per round) on the largo-scale FLAIR federated learning benchmark.
Characlerization of few-shot DP learning dynamics: We show that non-private few-shot transfer learners are generally in the interpolating regime where they achieve 100% training accuracy. Under strong DP, trained networks are generally in the regularization regime where test and train accuracies are comparable.
We assess the vulnerability of DP few-shot models with a strong membership inference attack (MIA) and find that non-private models are highly susceptible and the privacy level must. be increased to a high level to mitigate them.
Finally, we establish recommended practice guidelines for training DP few-shot models.
2 Background
In this section, we provide background information, definitions, and nomenclature required for subsequent sections. We focus our analysis on few-shot transfer learning based image classifiess that rely on large backbones pretrained non-private data.
Preliminaries We denote input images æ and image labels y E (1......C) where C' is the number of image classes indexed by c. Assume that we have access to a model f(x) - (b)(())) that outputs class- probabilities for an image p(yl2, 0, () = [(xr, 0, Φ) and comprises a feature extractor backbone bg : R2 - &d with parameters pretrained on a large upstream public dataset such as Imagenet- 21K (Russakovsky et al.) 2015) where d is the input image dimension and dg is the output fcature dimension, and a lincar layer classifier head he Rdd RC with weights Φ. Let D= {(am: (n)) x=1 be the privale downstream dataset that we wish to fine-tune the model f to. We denote the number of training examples per class or shot as S.
Learnable Paramcters In all experiments, the head parameters & are initialized to zero and are always learned when fine-tuning on D. For the backbone weights 0, we consider three options: (i) Head: 0 are fixed at their pretrained values and do not change during fine-tuning, only the head parameters & are updated; (ii) All: are initialized with pretrained values, but can be updated during fine-tuning in addition to the head; and (fii) FILM: using FiLM (Perez et al.) 2018)) layers. There exists myriad of adaptors for both 2D convolutional and transformer networks including FiLM, Adapter (Huulbyy et al. 2019). LoRA (Hu et al.,
2022a VPT (Jia ct al., 2022). AdaptFormer (Chen et al. 2022cp, NOAH (Zhang et al.
2022 Convpass 2022 that enable
(Jie & Deng 2022). Model Patch (Mudrakarta et al. 2019), and CaSE (Patacchiola et al.
a pretrained network to adapt to a downstream dataset in a parameter-efficien manner. In this work, we use FiLM due to its simplicity, high performance, and low paramcter count (Shysheya ct al., 2022). though another adapter could be used. A FiLM layer scales and shifts the activations aij arising from the j"h output of layer in the 7th block of the backbone as FiLM(a.):TK)),BB)) = Tij + Big, where Yi and Pig are scalars. We implement FiLM by fixing O at their pretrained values except for a subset of the scale and offset parameters utilized in the backbone normalization layers (e.g. GroupNorm, LayerNorm, etc., see Appendix A.3.1 for details), which can update during fine-tuning. For example, in a ResNet50, there are only 11 648 learnable FiLM parameters, which is fewer than 0.05% of O.
Transfer Difficulty (TD) The overlap between the distributions of the pretraining data and the downstream dataset as well other factors such as the number of classes in the downstream dataset are key determinants of the case and success of transfer lcarning. We mcasure the transfer difficulty (TD) as the relative difference between the accuracy of the All and Head learnable parameter configurations for a non-private model: TD= = 100 (Accsu AcCHead) /AccAu. This simple metric captures how different the downstream dataset is from the pretraining data as well as other factors that complicate transfer learning such as the number of classes C in the downstream dataset and its size [D]. If transfer learning is easy (i.e. TD is low), then only adapting the head of the network is sufficient. Tf transfer learning is more difficult (i.e. TD is high), then the backbone must also be adapted. Table I provides the TD values for all of the datasets used in the paper.
Differential Privacy (DP) DP Dwork al. 2006) is the gold standard for protecting sensitive data against privacy attacks. A stochastic algorithm is differentially private if it produces similar output distributions
3

---

Published in Transactions on Machine Learning Research (12/2023)
similar datasets. More formally, (c with privacy budget c 2 0 (lower means more private) and additive error 5 E [0 bounds how much the output distribution can diverge adjacent datasets. We use add/ add/remove adjacency, where two datasets are adjacent if one can be obtained from the other by adding or removing one data record, which could be a single datapoint in case of example-love privacy or data belonging to single user case of user-level privacy. The additive error is typically chosen such that 1/ We refer to Dwork & Roth (2014) for a thorough introduction to DP.
DP-SGD (Rajkumar & Agarwal 2012 Song et al., 2013 Abadi et al. 2016) adapts stochastic gradient descent (SGD) to guarantee DP. DP-SGD selects
using Poisson sampling, clips the Ez norm of per-example gradients, and adds isotropic Gaussian noise to the sum of mini-batch gradients. The level of privacy ((e, 6)-DP) is controlled by the noise multiplier o² which scales the variance of the added noise, the number of steps, and the sampling ratio (the Poisson sampling probability, ie., expected batch bize/|D),
Membership Inference Attacks (MIAs) MIAs aim to determinc if particular example was used in the training set of a model (Shokri et al. 2017). MIAs can be used to derive lower bounds to complement the theoretical upper bounds of (e, d)-DP for trained models. While there are many types of MIA (Hu et al.) 2022b), in this work we consider attacks that operate in the black-box mode (i.e. only model outputs can be observed) and can evaluate the loss on particular training or test examples (Carlini et al. 2022, Ye et al. 2022). In addition, we assume that attacks have access to images from the training data distribution and know the training algorithm used and its hyperparameters. To evaluate the effectiveness of a MIA, we examine the Receiver Operating Characteristic (ROC) curve which plots the attack true positive rate (TPR) against its false positive rate (FPR). We focus on the TPR at low FPR regime since a MIA is harmful if it can infer membership of even small number of training examples with high confidence (Carlini et al. 2022).
3 Related Work
DP Transfer Learning S describes various works where DP transfer learning using models pretrained on large public datasets achieve accuracy close to non-private approaches. However, to the best of our knowledge, there are no comprehensive studies on few-shol transfer learning under DP. The closest work to ours is Luo et al. (2021 | where the authors evaluate DP fine-tuning of a sparse subset of the parameters of models pretrained on public data on a small number of few-shot downstream datasets. Their work employs a relatively small backbone (ResNet18), pretrained on a small public dataset (minilmageNet)), with limited analysis. In contrast, our work utilizes large backbones, large public pretraining set, a wider range of privacy levels and downstream datascts, in addition to asscssing vulncrability to attacks and the federated learning selting. Tramer el al. (2022) point out that current DP benchmarks rely excessively on downstream datasets with a high level of overlap with the pretraining data. Our work addresses this issue by evaluating on datasets with a wide range of TD.
Federated Learning (FL) and Transfer Learning There has been a recent surge of interest in using large pretrained models as initialization for training decentralized models in both NLP (Lin et al. 2022 Stremmel & Singh| 2021 Weller et al.] 2022 Tian et al.) 2022) and computer vision (Chen et al. 2022b Tan et al.
2022 Qu et al. 2021 Chen et al. 2022a Nguyen et al. 2022 Liu et al.) 2022). Most of these works were able
to improve upon state-of-the-art results under different tasks and settings within FL. as well as showing that
the client data heterogeneity problem often seen in FL can be partially mitigated with pretrained networks.
FL and DP Even though the server in FL does not have access to raw user data, the privacy of users may still be compromised if (i) the server is untrusted (Huang et al. 2021) or (ii) a third party has access to the model after training (Geiping al.) 2020 Carlini et al.) 2022). Cryptographic techniques like secure aggregation Goryezka et al. (2013) can partially mitigate the former issue, while to fully tackle it as well as the latter, DP adaptations of the FL aggregation algorithms are needed McMahan et al. (2018). Similarly to DP-SGD, DP-FedAvg (McMahan et al. 2018) is an adaptation of the bascline FL algorithm FedAvg (McMahan et al. 2017), which provides user-level DP guarantees by applying the Gaussian mechanism to parameter updates sent to the server. Recently, a few studies have investigated the use of large pretrained models for FL under DP constraints in NLP Basu et al. (2021), representation learning Xu et al. (2022), and image classification Song et al. (2022). The closest work to ours is [Snn et al. (2022) who introduce FLAIR, a few-shot federated learning image classification dataset, which they use to perform relatively small evaluation
4

---

Published in Transactions on Machine Learning Research (12/2023)
of pretrained models (only ResNet18 was used) fine-tuned using FL under DP. However, to the best of our knowledge, there are no other studies on how large pretrained models fine-tuned via FI. aggregation algorithms behave under DP constraints for transfer-learned image classification. In this work we aim to fill this gap and evaluate these methods on real-world datasets.
4 Centralized Learning Experiments
In our experiments, we endeavor to answer the question: "Under what conditions is differentially private few-shot image classification effective?" We focus on transfer learning approaches that utilize large backbones pretrained on public data. We do this empirically by varying the: (i) number of shots S; (ii) set of learnable parameters in f (All, Head, FiLM); (iii) downstream dataset D (with varying TD); and (iv) network architecture: BiT-M-R50x1 (R-50) (Kolesnikov et al. 2020) with 23.5M parameters, Vision Transformer VIT-Base-16 16 (VIT-B) (Dosovitskiy et al. (021) with 85.8M parameters, both pretrained on the ImageNet-21IK dalaset. In all experiments, we assumee that the pretraining data is public and the downstream data is privale. Source code for all experiments can be found at: https://github. ctm/ httm//ambridgg mwg.mmlggdd... few-d---ewwoottt
Datasets For the experiments where S is varied, we use the CIFAR-10 (low TD) and CIFAR-100 (medium TD) datasets (Krizhevsky] (209)) which are commonly used in DP transfer learning, and SVHN (Netzer et al. 2011) which has a high transfer difficulty and hence requires greater degree of adaptation of the pretrained backbone. We also evaluate on the challenging VTAB-1k transfer Icarning benchmark (Zhai et al. 2019) that consists of 19 datasets grouped into three distinet categories (natural, specialized, and structured) with training set size fixed at 'D 1000 and widely varying TD.
Training Protocol For all contralized experiments, we first draw D of the required size (ID) = CS (i.c. the number of classes C' multippied by shot S) for varying shot or [D] = 1000 for VTAB- from the entire training split of the current dataset under evaluation. For the purposes of hyperparameter tuning, we then split 'D into 70% train and 30% validation. We then perform 20 iterations of Bayesian optimization based hyperparameter tuning (Bergstra et al. 2011) with Optuna Akiba et al. (2019) to derive set of hyperparameters that yield the highest accuracy on the validation data. This set of parameters is subsequently used to train a final model on all of DD. We evaluate the final, tuned model on the entire test split of the current dataset. Details on the set of hyperparameters that are tuned and their ranges can be found in Appendix A.3.2
For DP fine-tuning on D, we use Opacus (Yousefpour et al. 2021 and compute the required noise multiplier depending on the targeted (c. We report the results over three runs. For all experiments, we set = 1/|P| and report (c, dDP computed with the RDP accountant (Mironov 2017). Note that because we often change the dataset size (D) in our experiments this may make certain comparisons difficult since d will also vary. Similarly to previous work (De et al. 2022 Mehta et al. 2022 Sander et al. 2022) we do not account for privacy loss originating from the tuning of the hyperparameters. See Appendix
A.3 for additional training
details.
4.1 Few-shot DP Data Requirements
Fig. 1 depicts the performance of transfer learning under DP when varying S, c, and TD. Tabular results can be found in Tables 2 to 7 We see that accuracy decreases as S and decrease and as TD increases. For S < 10, accuracy is poor under DP. However, if the TD is low or medium, a moderate number of shots (S R 100) is sufficient to approach the accuracy of the non-private setting. For example, at S = 100, the model achieves better than 90% accuracy on CIFAR-10 using only 2% of the full training split at c 1. On the other hand, if TD is high, learning is more challenging and more shots are required to approach non-private accuracy. For example, for S 100 and 2, SVHN achieves just over 20% accuracy and falls well short of non-private levels even at S 500. Note that 5 changes based on [D]. Tables 23 and 24 provide (e,5) guarantees computed for when 5 is fixed and thus independent of [D].
Fig. shows the multiplier on the number of DP shots to match non non-private accuracy (see Appendix A.2.2 for additional figures and details). On the left we average over S E (5, 10), datasots, and network architectures. For all configurations, at c= 8, S must be increased by approximately - 8x to meet non-private accuracy and 20 35x é 1. In effect, as the privacy level increases, the required multiplier increases in an
5

---

Published in Transactions on Machine Learning Research (12/2023)
CIFAR-11 10 (low TD)
CIFAR 100 (medium TD)
SVHN (high TD)
100
90
8
60
4
50
2
10
25 shots (5)
100 250 500
25 shots (5)
100 250
25 50 shots (5)
100
250
Figure 1: Classification accuracy as function of shots and for CIFAR-10, CIFAR-100 and SVHN. Backbone is VIT-B and the best performing configuration out of All, FiLM and Head is used for each combination of and S, with 0= 1/[D). The accuracy is reported over three seeds with the line showing the median and the band reporting the lowest and highest accuracy. Analysis: Classification accuracy decreases as S and decrease and TD increases.
32
32
Muttipierr
Multiplier
8
8
4
All
SVHN
FiLM
CIFAR-10
2
2
Head
CIFAR-100
1
E=1
6=2
6=4
8=8
E=1
e=2
e=4
8=8
Figure 2: Multiplier of shots required to reach non- non-private accuracy. Left: Average over S {5, 10), datasets, and network architectures. Right: Average over S E {5, 10), network architectures, and learnable parameters. The data obtained using linear interpolation. See Appendix A.2.2 = 1/|D). Analysis: S must be increased by approximately 20 35x to meet non-private accuracy at 6 = 1 and
al 8
exponential manner. The multipliers are lower for simpler forms of adaptation (c.g. Head requires 20 x S at e= than for more complex forms (e.g. All requires 35 x S at e 1). On the right we average over S (5, 10), network architectures, and learnable parameters. Even though high TD datasets require more data for good accuracy, the multiplier values are similar and independent of the TD of the datasct (around 30x at e and 6x at 8). Fig. 3 shows the classification accuracy function of TD at 5 100. The accuracy gap between non-private and private training increases as TD increases.
4.2 Characterization of Learning Under Few-Shot DP
In this section, provide empirical evidence to highlight the different traits of private and non-private learning. Fig. 4 shows snapshots at E (1,8, of the train and test. accuracies as function of S for CIFAR-100 (medium TD) and SVHN (high TD) (see Figs. 14 and for versions with additional values of e). The three snapshots for each dataset can be viewed as discrete points on a continuum from low to high e. We see that learning under DP fundamentally different from non-privee noivprivate Non-private models with sufficient capacity operate in the interpolating regime and attain close to 100% training accuracy at all values of S, but have substantially lower test accuracy when low. In contrast, models that are trained with DP-SGD learning under heavy regularization and thus the training and test accuracies are significantly lower, but similar in value. When S is low, test accuracy is relatively poor and as S increases, test accuracy steadily improves. The point at which accuracy begins to improve varios with TD (CIFAR-100 test accuracy improvement starts much earlier than for SVHN). Independent of S, for low the train-lest, gap is very small, with the train accuracy indicative the test performance. As e increases, the train accuracy grows as the amount of rogularization pressure from DP roduced, ultimately entering the interpolating regime. For SVHN with 8o, Head leaves the interpolating regime for S 100, as there not enough capacity adapt to high TD dataset. As increases and S remains low, the test accuracy does not increase quickly

---

Published in Transactions on Machine Learning Research (12/2023)
100
E
80
00
70
60
50
30
10
10
20
30
40
50
Transfer Difficulty (TD)
Figure 3: Classification accuracy function of Iransfer difficulty (TD) and for CIFAR-10 (TD= 1.0), EuroSAT (TD=1.7), CIFAR-100 (TD=7.8) and SVHN (TD= 52.9) at = 100. EuroSAT has been chosen because the result for $ 100 can be casily taken from the VTAB results (Tables g to 13 due to C = 10. Backbone is VIT-B and the best performing configuration out of All, FiLM and Head is used for each e, with 8 = 1// The accuracy reported over three seeds with the line showing the median and the band reporting the lowest and highest accuracy, Analysis: The accuracy gap between non-private and private training increases as TD increases.
as the train accuracy and the accuracy gap grows. However, as increases, test accuracy starts catch up with train accuracy, reducing the gap. Fig. 21 shows the train and test accuracies for all 19 VTAB datasets function of e, where the general trends noted in Fig. 4 are also evident. While the results of Section 4.1
CIFAR-100
SVHN
Accuracy (%) Accuracy (%) Accuracy (%)
100
Accuracy (%) Accuracy (%) Accuracy (%)
100
80
60
60
6=1
Train/Test gap grows as privacy decreases
40
Train/Test gap grows as privacy decreases
40
20
20
100
All
100
FiLM
Head
40
20
test
20
train
100
100
6=
6=
20
20
100
250
500
500
shots
shots
Train/Test gap grows decreases
Train/Test gap grows decreases
Figure 4: Snapshots at E (1, oo), 8 8= 1/ of the train/test accuracies function of S for CIFAR-100 and SVHN. The trends in accuracy gap are shown with red arrows. At low S, the gap grows as increases, at high S gap decreases Lo nearly zero, and the gap grows as S decreases. Analysis: In the non-private set (e = 20), learning operates in the interpolation mode (i.e. train accuracy is 100%, yet accuracy continues to increase as S increases). As the privacy level increasos, learning operates learn under heavy regularization and the gap between train and test accuracy reduces.
indicate that both private and non-private accuracy benefit from additional training data, is evident that their learning behavior is significantly different.

---

Published in Transactions on Machine Learning Research (12/2023)
4.3
Few-shot DP Model Parameterization
CIFAR-10 (low TD)
CIFAR-100 (medium TD)
SVHN (high TD)
100
90
FILM
80
70
Head
60
50
40
30
20
6=2
10
25 50
250 500
25 50 shots (5)
100
250
500
10 shots (5)
100 250 500
(5)
Figure 5: Classification accuracy as function of shots and learnable parameters on VIT-B for CIFAR-10, CIFAR-100 and SVHN for c E {2, x with 5 8 1//D|. The accuracy is reported over three seeds with the line showing the median and the band reporting the lowest and highest accuracy. Analysis: FILM is comparable or better than All and Hend in terms of accuracy despite fine-tuning fewer than 0.05% of the parameters in the backbone.
Fig. 5 depicts classification accuracy function of S, two different values of c, and learnable parameters. FILM is comparable to or better than All and Head in terms of accuracy despite fine-tuning fewer than 0.05% of the parameters in the backbone. When the TD is low, training only Head is competitive with FILM and All, but when TD is medium or high, Head falls short as it cannot adapt the backbone to dataset that has a different data distribution. These observations have two implications: (i) FiLM able adapt to differing downstreeam datasets under DP and serves as a computationally efficient alternative to All; (ii) The result. provides empirical support for the observations of Li et al. (2022a) that the number of parameters has little effect the privacy utility trade-off when fino-tuning large pretrained models. Prior theory (Chaudhuri et al. 2011 Bassily et al. 2014) suggested that All should perform worse under DP compared to conligurations with fewer parameters.
11.1
71.
11.3
713
68.9
11.
71.
70
66.7
66.7
M6.7
06.7
VIT-B Head
R-50 Head
Accuracy (%)
60
58.9
59.3
58.0
99.5
58.9
99
50.8
599
10.2
53.6
51.8
ALM
45.7
43.8
VTT- FILM
37.5
35.1
34.3
36.4
VIT-B
27.5
29.
27.0
26.3
25.2
20.8
20
==1
=
=
E =
Figure 6: Average classification accuracy over all VTAB-1k datasets as a function of backbone, learnable parameters, and privacy level (s) at 10-3 Colored columns indicate results under DP, light gray indicates non-private accuracy for the corresponding configuration. Analysis: DP classification accuracy (colored columns) decreases significantly
is decreased and always falls short of non-private accuracy (gray columns). For non-private settings, the All learnable parameters setting outperforms FILM which outperforms Head. In contrast, for DP settings, All performs worst, FiLM and Head perform similarly, though FiLM is better in the majority of cases.
Fig. shows average classification accuracy all of the datasets in the VTAB- 1k benchmark (tabular results are in Tables to and comprehensive graphical results are in Figs. 19 and 20). We see that DP classification accuracy decreases siguificantly as is decreased and always falls short of non-privale accuracy. For non-private settings, the All learnable parameters setting outperforms FXLM which outperforms Head. contrast, for DP settings, All performs worst, FILM and Head perform similarly, though FiLM better the majority of cases. One explanation for this is that under DP low S, All requires more data compared to Head and FiLM for accuracy to progress beyond random chance be seen in Fig.
8

---

Published in Transactions on Machine Learning Research (12/2023)
Fig. 7 shows the difference between the accuracy of FILM and Head for VTAB-1k datasets function of The datasets are ordered from low to high TD (see Table 1. At é = 1, Head has an advantage over FiLM on several datasets. FiLM shows a significant advantage when the TD increases and as increases. Refer to Appendix A.2.5 for additional heat maps.
1
0.3
9.1
1.0
1.7
-12
2.2
0.1
-07
-0.
1.9
1.7
40
better Blue: Head better
2
0.8
1.6
3.3
-3.1
4.0
-04
-0.3
-0.8
1.5
0.3
3.7
0.3
0.3
-0.1
-0.5
0.4
1.5
0.7
4.4
20
4
0.1
0.0
0.0
2.0
0.6
1.0
-1.6
2.5
2.1
1.4
2.3
0.5
0.8
0.3
1.4
2.5
1.8
12.3
8
-0.5
-066
4.6
2.6
17.1
69
0.7
0.2
0.8
11
-0.4
1.8
-1.0
4.6
6.2
2.7
11.3
3.1
9.5
7.8
2.9
16.6
17.5
36.0
49.0
-40
retinopathy
flowers
caltach
pets
dtd
eurosat
camelyon
resics
cifar100
sun
kittl
dmlab
clevr-c
snorb el
snorb-at
dsprites-o
clevr-d
svhn dsprites-1
Figure 7: Heat map showing the accuracy difference between FiLM and Head for the VTAB-lk datasets as a function e. Backbone is VIT-B. Darker red indicates FiLM better. Darker blue indicates Head is better. Datasots ordered from low to high TD. 8 10-3 Analysis: At = 1, Head has an advantage over FiLM on several datasots. FiLM shows significant advantage when the TD increases and as increases.
4.4 Membership Inference Attacks
We use the state-of-the-ar Likelihood Ratio Attack (LiRA) (Carlini et al. 2022
allack models trained CIFAR-100 100 with varying S and privacy level using 256 shadow models. Refer to Appendix A.3.5 for additional detail. Excerpts from attack results are shown in Fig. g The complete set of attack ROC curves are shown in Figs. 22 and 23 while Table 14 reports TPR at several low FPR va values, AUC score, and the maximum membership inference advantage (defined as TPR - FPR by Yeom et al. (018)) achieved over the curve. Key observations are:
Non-private (e oc) models are extremely vulnerable MIAs (see Fig. middle). For example, in the case of bo, = 10, Head configuration, 82.2% of the examples can be successfully identified with false positive rate of only 0.1%.
Vulnerability of non-private (c= oo) models decreases S increases. Also, the FiLM configuration is consistently less vulnerable than Head (see Fig. S middle). We hypothesize that FiL.M generalizes better, so training examples do not stand out as much as in the Head configuration.
When S is fixed, vulnerability to MIAs greatly decreases with decreasing (see Fig. 8 right). Already with c 2, when 5 10 and FILM the vulncrability to MIA is substantially reduced, 2.5% of the examples can
be successfully identified with an FPR 1% and 0.3% of the examples with 0.1% FPR (see Table 14).
Under DP, there appears to be little or no difference between the vulnerability of the FiLM and Head configurations at the same e (sce Fig. right).
5 Federated Learning Experiments
In this section, we investigate how imposing user-level DP influences the performance of large pretrained models fine-tuned via federated aggregation. In evaluation, we use FLAIR Song al. 2022). which is recently proposed real-world dataset for multi-label image classification. It has around 50k users (overall around 400k images) with heterogencous data as well as a long-tailed label distribution, making it particularly appealing for benchmarking federated learning both in non-private and private seltings. Comprising mainly natural image data, FLAIR is low to medium TD dataset. As (Song et al. 2022). 8 is set to --1 where N = 41131 the number of training clients, and 2. We also perform experiments on CIFAR- 100 and Federated EMNIST, which have many fewer training users, but are widely used for benchmarking federated learning. Those results in Appendix A.2.8

---

Published in Transactions on Machine Learning Research (12/2023)
5 50
10P
100
10
10a
10
F
Readr- 021.0.10
5-20 1PR
10
Hezd 5-20, T.R
HMM (m-0000
20
R. G-00 1P 00
RESI SHOR, TRS-
Readand 155-0.000
Head: 5-50 178w 603
HASE S-5t, TARLO
10"1
5-00
16-3
R 2..222 TP6.4.00
Ht 198 9.003
NUX: Smic0, 171-0.227
Headre:3 195-0.00
Head 200,
198 2.037
HAse: 700, TPO 0.21
FMM 170.00002
mUl 5-200
11t 2000
RUM:
Headr 0.000
10TH
30Th
10"1
1014
1074
2011
10th
10"0
30"1
10"H
10"2
10"4 10
10 3079
10tH
10tH
20"H
30" 30'
Figure 8: ROC curves for LiRA (Carlini et al. 2022 on CIPAR-100 with R-50 backbone for two values of (2 and 0o) where S varies and for S 50 where varies. TPR values in legends are measured at FPR=0.001. Complete results Table 14 and Figs. 22 and 23
1/(1005). Analysis: Middle Non-private models are extremely vulnerable to MIAs. For 00, 10, Head configuration, 82.2% of the examples can be successfully identified with FPR 0.1%. Also, vulnerability decreases as S increases. Right: Increasing the privacy level reduces the vulnerability of the model as expected, when S 10 with = 2 and FiLM, only 2.5% of the examples can be successfully identified with FPR of 1%.
80
70.2
70.0
74.7
729
102
12.1900
23.54M
00.00
70
Communication cost
FILM
Head
Macro AP (9))
60
57.9
51.9 59.7
6071
60.5
59.
572
107
All
50.4
69.7
10a
old SOTA: R-18 All
R.18
30
10a
000090
C.035M
0.045M
0.0L3n
6.053M
40
30
8-18
R:50
VIT-B
R.18
R.50
VIT.B
Figure 9: Left: Private (colored) and non-private (gray) FL performance on FLAIR function of backbone and learnable parameters. = 2,6 41131-1.1, We use Macro-AP the primary metric to report accuracy for FLAIR. The R-18 All result on FLAIR is taken from Song et al. (2022). Our FLAIR results set new -tate-of-the-art. Right: FLAIR communication cost - the number of parameters sent at every user-server communication round. Analysis: We set a new state-of-the-art. result on the FLAIR benehmark by using FILM on R-18 by increasing Macro-AP from 47.2% to 51.9% and drastically reducing the communication cost from 11.9M parameters per round to 17K parameters. We further improve those results by using bigger backbones (R-50, VTT-B) with corresponding decreases in communication cost.
As in the centralized experiments, use R-50 and VIT-B, both pretrained ImageNet-21K. We also perform experiments on smaller architecture, ResNet18 (R-18) (He et al. 2016) pretrained on ImageNet-1K with 11.2M parameters, as it was initially used to achieve SOTA results on FLAIR.
For FL experiments, user-level DP is considered. We use FedADAM Reddi et al. 2021 aggregation, which was shown to have better empirical performance than standard FedAvg (MeMahan et al. 2017 We do not use Bayesian optimization for hyperparameter tuning, as cach FL run prohibitively expensive. Instead, we perform a small grid search over the server and client learning rates. Refer to Appendix AA336 for the hyperparameter ranges searched. For fair comparison on FLAIR, we fixed the other training hyperparameters to the values in the original paper Song et al. (2022).
Fig. g (left) shows the performance of different model configurations on FLAIR with (color) and without (grey) DP. We report macro average precision (Macro-AP) results here, while additional metrics are shown in Tables 15 and 16 As communication cost important in FL, in Fig. g (right) we report the number of parameters required be transmitted for each model configuration in one user-server interaction. Summarizing
Fig. g key observations are:
10

---

Published in Transactions on Machine Learning Research (12/2023)
With R-18 used in the original paper, achieve state-of-the-art performance under DP with FILM, improving Macro-AP from 44.3% to 51.9%. This improvement comes with a ioduction in communication cost from 11.2M parameters per each user-server interaction to only 17k.
With VIT-B we further improve the state-of-the-art result on FLAIR in both DP and non-private settings. Under DP, the Macro-AP increases to 59%, while for non-private, the Macro-AP increases from 62.1% to 74.7%.
Head is more robust under DP than All or FiLM. Head has the smallest relative drop in performance of around 10% for any model configuration.
Although FILM is outperforming Head and All on R-18, it is not always the case for other backbones. However, taking into account both test performance and communication cost, we can clearly see that either FiLM or Head is preferred. FiLM performs better for smaller backbones (R-18 and R-50), while Head is slightly better for VIT-B.
6 Discussion and Recommendations
Our work shows that DP few-shot learning works surprisingly well in the low TD setting, while the high TD setting is more difficult. Alternative strategies may include side-stepping privacy costs by leveraging the zero-shot capabilities of large pretrained models such as CLIP [Radford et al., 2021] or utilizing public data
in addition to private data in the fine-tuning process as well GGolatkar et al. 2022 in order to improve utility. In summary, our experiments show that:
How much additional data is required under few-shot DP? Image classification accuracy decreases as e and S decrease, and as TD increases. As a result, one should expect to use roughly 4 4- 8x larger S for 8 and 20 35x larger S for e 1 under DP to achieve accuracy comparable to non-private. (Note that 8 = 1/|D| for these multipliers.) The multipliers are surprisingly similar across different TD levels.
Transfer lcarning dynamics under DP are fundamentally different from non-private Non-private models with sufficient capacity operate in the interpolating regime and attain close to 100% training accuracy at all values of S, but have substantially lower test accuracy when S is low. In contrast, models that are trained with DP-SGD are learning under heavy regularization and thus the training and test. accuracios are significantly lower, but similar in value.
Parametor-effiiien FILM adapters perform well under DP FILM is comparable to or better than All and Head in terms of accuracy, demonstrating its ability to adapt to differing downst datasets despite fine-tuning fewer than 0.05% of the parameters in the backbone. When the TD easy, Head is competitive with FiLM and All, but when TD difficult, Head falls short as it cannot adapt the backbone to downstream dataset that has different data distribution. FiLM is also effective in the DP FI. setting, achieving state-of-the-art accuracy on the FLAIR benchmark while reducing communication cost by orders of magnitude.
Non-private Few-Shot Models Are Particularly Vulncrable to MIAS The vuincrability of non- privale few-shot models increases as S decreases. DP significantly mitigates the effectiveness of MIAs, e.g., we found that DP few-shot models can expose 2.5% the examples with a 1% FPR when e = (on CIFAR-100 with S=10, FILM on R-50) which is substantially less vulnerable than the non-private models.
Limitations We identify the following limitations in this work: (i) We focused exclusively on few-shot. transfer learning from relatively large pretrained models and did not consider meta-learning approaches or training from scratch. (ii) We used FiLM adapters exclusively and did not consider other parfmeter-effiiient adapters. Based on our experience, adapters do not have large effect on the overall trends that we observed (in comparison to the items that we did vary), and making fair comparison on a reasonable set of adapters would have exceeded our computational resources. (iii) The Transfer Difficulty (TD) metric is not ideal as it depends on the network architecture and training hyperparameters, but in practice it aligns extremely well with the empirical difficulty of adapting to a downstream datasct. (iv) We always set 8 = 1/D. While this is standard practice, some experiments in the paper where D varies, fair comparisons of results can be difficult and in that case an alternative would have been to choose a small constant value of 8. (v) For each
11

---

Published in Transactions on Machine Learning Research (12/2023)
experiment, used hyperparameter tuning to set constant learning rate, but this learning rate was not annealed as is often done in non-private training.
Broader Impact Statement Few-shot learning systems hold much positive potential from personalizing object recognizers for people who are blind (Massiceti et al. 2021 to rendering personalized avatars (akharov et al. 2019) (see (Hospedales et al.) 2020) for a full review). These systems, however, also have the potential used in adverse ways for example, in few-shot recognition in militarry/urveillaaae applications. We demonstrate how to execute highly successful membership inference attacks against few-shot learning models which could be employed in harmful manner. However, also show how to effectively mitigate such attacks by training with DP.
Acknowledgments
Marlon Tobaben and Antui Honkela are supported by the Research Council of Finland (Flagship programine: Finnish Center for Artificial Intelligence, FCAI; and grant 356499), the Strategic Research Council at the Rescarch Council of Finland (Grant 358247) well the European Union (Project 101070617). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Commission. Neither the European Union nor the granting authority can be held responsible for them. Aliaksandra Shysheya, John Bronskill, and Richard E. Turner are supported by an EPSRC Prosperity Partnership EP/ EP/ TT005366/1 between the EPSRC, Microsoft Research and the University of Cambridge. This work has been performed using resources provided by the CSC IT Center for Science, Finland, and the Finnish Computing Competence Infrastructure (FCCT), as well as the Cambridge Tier-2 system operated by the University of Cambridge Research Computing Service https: htwww/.wpttt cam. funded by EPSRC Tier-2 capital grant EP HPPP020259/1 /1. We thank Joonas Jälkö, Lukas Wutschitz, Stratis Markou, Massimiliano Patacchiola and Runa Eschenhagen for helpful comments and suggestions.
References
Martín Abadi, Andy Chu, Ian J. Goodfellow, H. Brendan McMahan, Πya Mironov, Kunal Talwar, and Zhang. Deep learning with differential privacy. In Edgar R. Weippl, Stefan Katzenbeisser, Christopher Kruegel, Andrew C. Myers, and Shai Halevi (eds.), Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria, October 24-28, 2016, pp. 308-318, ACM, 2016. doi: 10.1145/2900 2977779.277881111 URL https:://oii: htt.e//............111.. 299814///7777799277788888
John M. Abowd. The U.S. census bureau adopts differential privacy. In Yike Guo and Faisal Farooq (eds.), Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018, pp. 2867. ACM, 2018. doi:
URL
attps://dov org/10.11111 1tt.1111//22919992255555
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next- generation hyperparameter optimization framework. In Ankur Teredesai, Vipin Kumar, Ying Li, Rómer Rosales, Evimaria Terzi, and George Karypis (eds.), Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019, pp. 2623-2631. ACM, 2019. doi:
URL https:://oi...... org/ /0/. /144/129///00
3330701
Galen Andrew, Om Thakkar, Brendan McMahan, and Swaroop Ramaswany. Differentially private learn- ing with adaptive clipping. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NewrIPS 2021, December 6- 14, 2021, virtual, pp. 17455-17466, 2021. URL
wwtpseccppppppp/eeee/ 2021/hah//
Borja Balle, Giovanni Cherubin, and Jamie Hayes. Reconstructing training data with informed adversaries. 43rd IEFF Symposium Securily and Privacy, SP 2022, San Francisco, CA, USA, May 22-26, 2022, pp-
12

---

Published in Transactions on Machine Learning Research (12/2023)
IEEE, 2022. doi: 00.100/S74621..022.988883 URL http:://ao1.0rg/10.1110033
2022. 9833677
Raef Bassily, Adam D. Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 55th IEEE Annual Symposium on Foundations of Computer Science, FOCS 2014, Philadelphia, PA, USA, October 18-21, 2014, pp. 464 473. IEEE Computer Society, 2014. doi:
10.1109/FOCS.201 CECCS..014.566 URL httts:://oi..rrg:// 1tt.:/....119/// 110./FOCS. 2014.56
Priyam Basu, Tiasa Singha Roy, Rakshit Naidu, Zümrüt Müftüoglu, Sahib Singh, and Fatemehsadat Mireshghallah. Benchmarking differential privacy and federated learning for BERT models. ArXio preprint, abs/2106. 2021. URL
13973
Charles Beattic, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancy, Simon Green, Victor Valdés, Amir Sadik, et al. Deepmind lab. ArXiv preprinl, abs/1612.0380) 2016. URL. https://arxiv org/
brgg/bs//aa awwaaass...../// 03801
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for rrper-prrameter op- timization. In John Shawe- Taylor, Richard S. Zemel, Peter L. Bartlett, Fernando C. N. Percira, and Kilian Q. Weinberger (eds.), Advances in Neural Information Processing Systems 24: 25th Annuual Con- ference on Neurol Information Processing Systems 2011. Proceedings of meeting held 12-14 December 2011, Granada, Spain, pp. 2546 2554, 2011. URL https::/ www.spe. wwtppppee///////a/a/...a Abstract. html
Sebastian Caldas, Peter Wu, Tian Li, Jakub Konečný, H. Brendan McMahan, Virginia Smith, and Ameet Talwalkar. LEAF: A benchmark for federated settings. ArXiv preprint, abs/1812 bbs/1812.01197, 2018. URL
https://arxiv.. wwwgabs/////eeeeo.eeet...tt.aeet aaa//////// 1////1111.0001
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Song, Ülfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. In Michael Bailey and Rachel Greenstadt (eds.), 30th USENIX Security Symposium, USENIX Security 2021, August 11-13, 2021, pp. 2633 2650. USENIX Association, 2021. URL
https://wwws usenix.e.............
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1897-1914. TEFE, 2022.
Yannis Cattan, Christopher A. Choquette-Chone Choo, Nicolas Papernot, and Abhradeep Thakurta, Fine-tuning with differential privacy necessitates an additional hyperparameter search. CoRR, abs/2210.02156 2022.
doi: 10.48550/ar 2t.55///XXi..2220221555 URL /ttp:::///.....r///0...55.// http://awvvv............ 2210.02155
Kamalika Chaudhuri, Claire Monteleoni, and Anand D. Sarwate. Dillerentially private empirical risk minimization. J. Mach. Learn. Res., 12:1069 -100999-19,9,,--111111111119-190 2011. doi:
URL https :
//dl. awm. orgoorg//ooo do1/10 1at.a/.1..555//// 5555/1955553 202211900308222222266663
Hong- You Chen, Cheng- Hao Tu, Ziwei Li, Han Wei Shen, and Wei-Lun Chao. On the importance and applicability of pro-training for federated learning, 2022a. URL
htgg//bs/2000 /20.. 11488
Jinyu Chen, Wenchao Xu, Song Guo, Junxiao Wang, Jie Zhang, and Haozhao Wang. Fodtune: deep divc into efficient federated fine-tuning with pre-trained transformers, 2022b. URL htttps///rxiv..o wrwpviivoor/baaa. /bs/
2211.08025
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang. Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. ArXiv preprint, bbss/2205.1335 2022c. URL
attps://arxiv.cm org/aba/ http//bb/2200..... 13535
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEFE, 105(10):1865-188 1883, 2017.
13

---

Published in Transactions on Machine Learning Research (12/2023)
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In 2014 IEFF Conference on Compuler Vision and Pallern Recognilion, CVPR 2014, Columbus, OH, USA, June 23-28, 2014, pp. 3606 3613. IEEE Computer Society, 2014. doi: 10.1109/ D0.1199/CVPR.20 CVPR.2014.461. URL httpps://dll htte/di..ogg/////
org/ 10. 1109/CVPR..... 2014.4611
Grahai Cormode, Somesh Jha, Tejas Kulkarni, Ninghui Li, Divesh Srivastava, and Tianhao Wang. Privacy scale: Local differential privacy in practice. In Gautam Das, Christopher M. Jermaine, and Philip A. Bernstein (eds.), Proceedings of the 2018 International Conference on Management of Data, SIGMOD Conference 2018, Houston, TX, USA, June 10-15, 2018, pp. 1655 1658. ACM, 2018. doi: 10. 10.1145/3183713. 3197390. URL
Soham De, Leonard Berrada, Jamie Hayes, Samuel T.. Smith, and Borja Balle. Unlocking high-accuracy differentially private image classification through scale. CoRR, 1bs/2204.13650, 2022. doi: 1-48550/arXiin 2204.13650. URL https://bol..rgg/0.44666665
10.p 48550/ arxiv. 2204. 13650
Apple Differential Privacy Team. Learning with privacy at scale. http:///oooo-asses..e pdf 2017.
Bolin Ding, Janardhan Kulkarni, and Sergey Yekhanin. Collecting telemetry data privately. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Gar- nett (eds.), Advances in Newral Information Processing Systems 30: Annual Conference on Neural Informa- 2017. URI,fttpp www.:/9oo9-9ttttttttta...
lion Processing Systems 2017, December 4-9, 2017, Long Bench, CA, USA, pp.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 9-7, 2021. OpenReview.no net, 2021. URL https://openreevie htt.pppnrrrrieeeettoooortooioooo mett:fforrmmddddiccFdTaT
Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Foundations and Trendso in Theoretical Computer Science, 9(3 4):211-4077 2014. TSSN 1551-305X. doi: 10.1561 10.1561/040000 /040000002..
URL http:// wttp://xx......./....o....... wttp:////...1110000000000000
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam D. Smith. Calibrating noisc to scnsitivity in private data analysis. In Shai Halevi and Tal Rabin (eds.), Theory of Cryplography, Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006, Proceedings, volume 3876 of Lecture Notes in Computer Science, pp. 265 284. Springer, 2006. doi: 10.1007/11681888 14. URL https::/doi.org ort//or///////. 1t.gg/10.1000 10011/116611888..11 14
Li Fei-Fei, Rob Fergus, and Pictro Perona. One-shot learning of object categories. IEEE transactions pattern analysis and machine intelligence, 28(4):594 611, 2006.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset.
The International Journal of Robotics Research, 32(11):1231-1233 2013.
Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradients how easy is it to break privacy in foderated lcarning? In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florini Balcan, and Hsuan-Tien Lin (eds.), Advances in Newral Information Pro- cessing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020. URL https::/proceeednng/ httproceeeinngn.eeiis...cceooeee www.ripss.ccppper///a/... Abs.rssssbttttttttc.a
Aditya Golatkar, Alessandro Achille, Yu-Xiang Wang, Aaron Roth, Michael Kearns, and Stefano Soatto. Mixed differential privacy in computer vision. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 8366 8376. IEEE, 2022. doi: PL110//CVPR566822222200 202.0/888822200881.0 URL http:p////aaaiorrr//.0////////// 202.2226688222200083222 00819
14

---

Published in Transactions on Machine Learning Research (12/2023)
Google. Tensorflow federated: Machine learning decentralized data. http:://www.tensorfllll
federated 2019a.
Google.
Tensorflow privacy: Library for training machine learning models with privacy for training data'.
atttps://githin www.//eessoc..lo///rvvccc/////// 2019b.
Sivakanth Gopi, Yin Tat Lee, and Lukas Wutschitz. Numerical composition of diffcrential privacy. In Mare' Aurelio Ranzato, Alina Beygekzimer Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference Neurol Information Processing Systems 2021, NewrlPS 2021, December 6- 6-14, 2021, virtual, pp. 11631 11642, 2021. URL https 7/proceedings www.rnnnnippp...... httpp/////////////o///////////0/ -tw.eee1111666444--aa-rt Abstract. html
Slawomir Goryczka, Li Xiong, and Vaidy Sunderam. Secure multiparty aggregation with differential privacy: A comparative study. In Proceedings of the Joint. EDBT/ICDT 2013 Workshops, EDBT '13, pp. 155-163, New York, NY, USA, 2013. Association for Computing Machinery. ISBN 9781450315999. doi: 10.1145/ 2457317.2457343 URL
http:///.o....//...////. 115.11111/247111111 0tt.:a...00011111111/44751111111 2457343
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEFE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770 778. IEEE Computer Society, 2016. doi: CLL19//CVP.2216..00 URL https: /tops://doi..oo orggi.org/11 10. 1109/CVPR. 2016.90 90
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2220) 2019.
Timothy Hospcdales, Antreas Antoniou, Paul Micaclli, and Amos Storkey Meta-lcarning in neural networks: A survey. ArXiv preprint, abs/ Abs/2004.055339, 2020. URL
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2790 2799.. PMLR, 2019. URL http://pproceeednngs mll wwwpp/eeeevv v97 http//999//h/uuu houlsby19 html
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.nee 2022a. URL httppp//ppaaaarree..eee//o/ootee
Hongsheng Hu, Zoran Salcic, Lichao Sun, Gillian Dobbie, Philip S Yu, and Xuyun Zhang. Membership inference allacks on machine learning: A survey. ACM Compuling Surveys (CSUR), 54(11s):1 37, 2022b.
Yangsibo Huang, Samyak Gupla, Zhao Song, Kai Li, and Sanjeev Arora. Evaluating gradient inversion attacks and defenses in federated learning. In Marc' Aurelio Ranzato, Alina Beygekimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Newral Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NewrIPS 2021, December 6- 14, 2021, virtual, pp. 7232 7241, 2021. URL. html
3b3 0bff6666844499911118603388833833 haarsaaacccccccaeaaooaaaaaaaaaaa
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardic, Serge J. Belongic, Bharath Hariharan, and Ser- Nam Lim. Visual prompt luning. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision ECCV 2022 17th European Conference, Tel Avio, Israel, October 23-27, 2022, Proceedings, Part XXXIII, volume 13693 of Lecture Notes in Computer Science, pp. 709-727. Springer, 2022. doi: 000077998-3-011-111 978-3-031--1827-4444
41. URL https:://doo http::///oooooo/1.11000000 org/10.1007//
15

---

Published in Transactions on Machine Learning Research (12/2023)
Shibo Jie and ChinHong Deng. Convolutional bypasses are better vision transformer adapters. ArXiv preprint, abs/22077.07055 2022. URL. https://aaxiv: wttp://w.wrr/bb///////// 2207007039
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, Juby 21-26, 2017, pp. 1988 1997. IEEE Computer Sociely, 2017. doi: 10.1109 10.1109/CVPR.200 2CV..C URL
sttps://doi.. org. 1109/CVPR.
Kaggle and EyePacs. Kaggle diabetic retinopathy detection. teabeei--rttinnaaabyyddtttttttt
1e. cwm..com/oommmmm.mme c/
2015.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun ), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL. http: org/ abs/
abs/aaass.../////////////aatatta 111/11111..111 6980
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica. Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michacl Frahm (eds.), Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 29-28, 2020, Proceedings, Part V,, volume 12350 of Lecture Notes in Computer Science, pp. 491 507. Springer, 2020, doi: 00000079777--000-55555558 7 29. URL http8: 7_29
7/doi . 11111111000009---0000-----0
Alex Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009.
Alexey Kurakin, Steve Chien, Shuang Song, Roxana Geambasu, Andreas Terzis, and Abhradeep Thakurta. Toward Training at ImageNet Scale with Differential Privacy. ArXiv preprint, abs/2201.12222 2022. URL
https://arxiv org/abs/ 2222..2...222
Yann LeCun, Fu Jie Huang, and Leon Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 2, pp. 11 104. IEEE, 2004.
Wei Li and Andrew McCallum. Pachinko allocation: Dag-structured mixture models of topic correlations. In William W. Cohen and Andrew W. Moore (eds.), Machine Learning, Proceedings of the Twenty- Third International Conference (ICML 2006), Pillsburgh, Pennsylvania, USA, June 25-29, 2006, volume 148 of ACM International Conference Proceeding Series, pp. 577-584. ACM, 2006. doi:
URL https://dol. org/10 10. 1145/1143844 1143917
Xuechen Li, Daogao Liu, Tatsunori Hashimoto, Huseyin A. Tnan, Janardhan Kulkarni, Yin Tat Lee, and Abhradeep Guha Thakurta. When does differentially private learning not suffer in high dimensions? CoRR, 2022a. doi: 10. 2tt.5////XXi........000 URL hattps://ol lerddi..rg//ii org/10. /000 48550/arXix 2207
00160
Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022b. URL htttps://ppenrevlee.. www.sewww.ee/fooetreeeao
id=
Bill Yuchen Lin, Chaoyang He, Zihang Ze, Hulin Wang, Yufen Hua, Christophe Dupuy, Rahul Gupta, Mahdi Soltanolkotabi, Xiang Ren, and Salman Avestimehr. FedNLP: Benchmarking federated learning methods
for natural language processing tasks. In Findings of the Association for Computational Linguislies: NAACL 2022, pp. 157-175, Seattle, United States, 2022. Association for Computational Linguisties. doi: URL httpps://aclaanhoooooo.oo/////// http:///o....idinnnnmmama...ll
10.18653/vl http:/2222..nnnnsnneeeeeee
Zicheng Liu, Da Li, Javier Fernandez-Marques, Stefanos Laskaridis, Yan Gao, Lukasz Dudziak, Stan Z. Li, Shell Xu Hu, and Timothy Hospedales. Federated learning for inference at anytime and anywhere, 2022. URL https://arxx:: httppp////axiiioorrr//////1/1/1 04084
16

---

Published in Transactions on Machine Learning Research (12/2023)
Zehun Luo, Daniel J. Wu, Ehsan Adeli, and Li Fei-Fei. Scalable differential privacy with sparse net- work finetuning. In IEEE Conference on Computer Vision and Pallern Recognilion, CVPR 2021, virtual, June 19-25, 2021, Pp. 5059 5068. Computer Vision Foundation / LEEE, 2021. doi: 10. /C/CCCCVRR4433.2222222 Scalable Differentaa Privacy With Sparse Network Finetuning CVPR_ 2aa___22211____________ patpe..tmml html
URL
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hy- percomplex adapter layers. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NewrIPS 2021, December 6-
14, 2021, virtual, pp. 1022-1033 2021. URL
081 0 Aeboobbbbttttt....... html
Daniela Massiceti, Luisa M. Zintgraf, John Bronskill, Lida Theodorou, Matthew Tobias Harris, Edward Cutrell, Cecily Morrison, Katja Hofmann, and Simone Stumpf. ORBIT: A real-world few-shot dataset for teachable object recognition. In 2021 ZEEE/CVE International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, Oclober 10-17, 2021, pp- 117798-10800 TEEF, 2021. doi: 10.1109/TCCC448922.2222 2021 .01064.
URL httpps://dii org/10. /0. w1t...1111////11122.22.. 2021. 01064
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset, 2017.
Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Agūera y Arcas. Communication-
efficient learning of deep networks from docentralized data. In Aarti Singh and Xiaojin (Jerry) Zhu (eds.), Proceedings of the 20th International Conference on Artificial Intelligence and Stalisties, AISTATS 2017, 20- April 2017, Fort Lauderdale, FL, USA, volume 54 of Proceedings of Machine Learning Research, pp- 1273 1282. PMLR, 2017. URL http://procceedings mlr /rress//54//cmaaanla....
H. Brendan McMahan, Daniel Ramage, Kunal Talwar, and Zhang. Learning differentially private recurrent language models. In 6th International Conference on Learning Representations, ICLR 2018, Vancouwer, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.nee net, 2018. URL net/forum BIdJ0FFF100
https://openxeeieov wtwp:r/wwwwwweteoooouoouuuuouuoe
Harsh Mehta, Abhradeep Thakurta, Alexey Kurakin, and Ashok Cutkosky. Large scale transfer learning for differentially private image classification. CoRR, hbs/2205.00973, 2022. doi: URL https://doi.org: htt///00.855/////// 48550/arXiv. 2205. 2205.02973
Hya Mironov. Rényi differential privacy. In 30th IEEE Computer Securily Foundalions Symposium, CSF 2017, Santa Barbara, CA, USA, August 21-25, 2017, pp. 263-275. IEEE Computer Society, 2017. doi: 10.1109/CS5 11110//.FF.011..1... URL httpp:://ol. org/ 10. 11F9/CSF..... 2017.11 11
2017.11.
Pramod Kanshik Mudrakarta, Mark Sandler, Andrey Zhmoginov, and Andrew G. Howard. K for the price of 1: Parameter-effiieent multi-task and transfer learning. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.nee net, 2019. URL https://openreviev.. htt.:/ee//eeeeeee htw.://www.............rr////aa.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
John Nguyen, Kshitiz Malik, Maziar Sanjabi, and Michael Rabbat. Where to begin? exploring the impact of pre-training and initialization in federated learning, 2022. URL https://aaxxi.orr/aaas.
15387
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classos. In 2008 Sizth Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722-729. IEEE, 2008.
17

---

Published in Transactions on Machine Learning Research (12/2023)
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pottern Recognition, Providence, RI, USA, June 16-21, 2012, pp. 3498 3505. IEEE Computer Society, 2012. doi:
URL https:
11/0o/0///... 2012. 6248092
Massimiliano Patacchiola, John Bronskill, Aliaksandra Shysheya, Katja Hofmann, Scbastian Nowozin, and Richard E Turner. Contextual squeeze-and--xxiitaiion for efficient few-shot image classification. ArXiv preprint, abs/ abs//220..0983 00843,, 2022. URL
wwwpeiiww....r.iiee.oo www//a/......//// 09843
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. Film: Visual reasoning with a general conditioning layer. In Sheila. A. McIlraith and Kilian Q. Weinberger (ods.), Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 3942-3951. AAAT Press, 2018. URL httpps///wtt aaai oww.org//..o.oooo.ooo.o index. paadeex.hp///aae http////AAAAAAA////pp/rrre//////
Liangqiong Qu, Yuyin Zhou, Paul Pu Liang, Yingda Xia, Feifei Wang, Li Fei-Fei, Ehsan Adeli, and Daniel L. Rubin. Rethinking architecture design for tackling data heterogeneity in federated learning. 2022 IEEE/CVE Conference on Compuler Vision and Poltern Recognilion (CVPR), pp- 10051-10061, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Adilya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8748 8763. PMLR, 2021. URL http:://procedding mlr, paress/v13//ee ord21a httmla.mmlm
Arun Rajkumar and Shivani Agarwal. A differentially private stochastic gradient descent algorithm for multiparty classification. In Neil D. Lawrence and Mark A. Girolami (eds.). Proceedings of the Fifteenth International Conference on Artificial Intelligence and Slalistics, AISTATS 2012, La Palma, Canary Islands, Spain, April 21-23, 2012, volume 22 of JMLR Proceedings, pp. 933 941. JMLR.org, 2012. URL
mlr. press/ pwwps/////rra//eeeeeeeeeeeeeee v22/ra wwmaraawauaaaarrrr 12. .ttll
Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Koncčný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.no net, 2021. URL https:///penrrviee..eee..
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3): 1522, 2015. doi: 10. 120077/6112033-11-0081177
Tom Sander, Pierre Stock, and Alexandre Sablayrolles. TAN without burn: Scaling laws of DP-SGD. CoRR, abs/2210.034003, 2022. doi: 10.48550/arXiv22210..033 URL https: htttps///ooi.org/00..6665/////// 2210. 03408
Micah J Sheller, Brandon Edwards, G Anthony Reina, Jason Martin, Sarthak Pati, Aikaterini Kotrotsou, Mikhail Milchenko, Weilin Xu, Daniel Marcus, Rivka R Colen, et al. Federated learning in medicine: facilitating multi-inse multi-institutional collaborations without sharing patient data. Scientific reports, 10(1):1-12, 2020.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and pricecy (SP), pp. 3 18. IEEE, 2017.
Aliaksandra Shysheya, John Bronskill, Massimiliano Patacchiola, Sebastian Nowozin, and Richard E. Turner. Fit: Parameter efficient few-shot transfer learning for personalized and federated image classification. ArXiv preprint, /2////2260067., 2022. URL http:///aaravv.rrraa//2200000..
18

---

Published in Transactions on Machine Learning Research (12/2023)
Congzheng Song, Filip Granqvist, and Kunal Talwar. Flair: Federated learning annotated image repository. Xrxiv preprinl, abs/2207.00 /////2207.088699 2022. URL
Shuang Song, Kamalika Chaudhuri, and Anand D. Sarwate. Stochastic gradient descent with differentially private updates. In IEEE Global Conference Signal and Information Processing, GlobolSIP 2013, Austin, TX, USA, December 9-5, 2013, pp. 245-248. IEEE, 2013. doi: 1ttp////GooollS111001077000 https://doi org/10. 10///10.110 1109//1o 2013.02222222 6736861
URL
Joel Stremmel and Arjun Singh. Pretraining federated text models for next word prediction. In Kohei Arai (ed.), Advances in Information and Communication, pp. 477-488, Cham, 2021. Springer International Publishing. ISBN 978-3--33--3110--8
Yue Tan, Guodong Long, Jie Ma, Lu Liu, Tianyi Zhou, and Jing Jiang. Federated learning from pre-trained models: A contrastive learning approach. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Yuanyishu Tian, Yao Wan, Lingjuan Lyu, Dezhong Yao, Hai Jin, and Lichao Sun. Fedbert: When federated learning meets pre-training, ACM Trans. Intell. Syst. Technol. 13(4), 2022. ISSN 2157-6904. doi: 10.1145/3510083 URL https:://doo http::://dii.or//0...o 1r1.://11///1.11//1110033 1145/3510033
Florian Tramèr, Gautam Kamath, and Nicholas Carlini. Considerations for diffcrentially private learning with large-scale public pretraining. CoRR, abs/ (/88/2212.06470 2022. doi: 10.48550/ /8855//aaXi.22222.00000 URL
https: http://oi.orgg//// 48550//5550//1 2212.0647777
Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivariant cnns for digital pathology. In International Conference on Medical image computing and computer-ossisted intervention, pp. 210 218. Springer, 2018
Orion Weller, Marc Marone, Vladimir Braverman, Dawn Lawrie, and Benjamin Van Durme. Pretrained models for multilingual fedcrated learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Compulational Linguislics: Human Language Technologies, pp. 1413 1421, Seatule, United States, 2022. Association for Computational Linguistics. doi: 10.18653/v11222 http://1/222eeeee www.:/aaaaaaaamaa.a..a.. m0in.101. URL httpps://aclaattholoogg www.ooloooyo.og////0 org/2022. naw.::aanaammmmi.n...m
Jianxiong Xiao, James Hays, Krista A. Fhinger, Aude Oliva, and Antonio Torralba. SUN database: Large- scale scene recognition from abbey to zoo. In The Twentyy-Thrrd IEEE Conference on Computer Vision and Pallern Recognition, CVPR 2010, San Francisco, CA, USA, 13-18 June 2010, pp. 3485 3492. IEEE Computer Society, 2010. doi: 10.1109 10.1109/CVPR..C /C///C.221....3309000 URL htttp:://ooii http://///ooooo/1...111/ 111.///....9//PP..... 2010 5539970
Zheng Xu, Maxwell Collins, Yuxiao Wang. Liviu Panait, Sewoong Oh, Sean Augenstein, Ting Liu, Florian Schroff, and H. Brendan McMahan. Learning to generate image embeddings with user-level differential privacy, 2022. URL https://arxiv org/abs. 10844
Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschacdler, and Reza Shokri. Enhanced membership inference allacks against machine learning models. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, CCS '22, pp. 2022. Association for Computing Machinery. ISBN 9781450394505. doi: httpp://doi.
New York, NY, USA, URL
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha Privacy risk in machine learning: Analyzing the connection to overfitting. In 31st IEEE Computer Security Foundations Symposium, CSF 2018, pp. 268 282. TEFE Computer Society, 2018.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural net- works? In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger (ods.), Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pp. 3320-3323 3328, 2014. httpstsps // Awt.oo---ssttttttt.emmacao html
19

---

Published in Transactions on Machine Learning Research (12/2023)
Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Gosh, Akash Bharadwaj, Jessica. Zhao, Graham Cormode, and Nly Mironov. Opacus: User-friendly differential privacy library in pytorch. ArXiv preprint, abs/2109.12298, 2021. URL https:
//axxiv.rrg/abs 2tt./2119.1111 12298
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huscyin A. Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang. Differentially private fine-tuning of language models. In The Tenth International Conference on Learning Representations, net/
ICLR 2022, Virtual Event, April 25-29, 2022. OpenRevicw.nct, 2022. URL
forum? 2o-2fm2
Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor S. Lempitsky. Few-shot adversarial learning of realistic neural talking head models. In 2019 SEE/CVE International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pp. 1458-9463 IEEE, 2019. doi: 10.1109/1CCV. 010100///CCV.221.00955..., 2019.00995. URL https://doi. http//0.1109/IIC..... 2019
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. ArXiv preprinl, abs/1910.04867,, 2019.
URL http:////axxxvooo//bbbbaa/ 19117.48677777
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search, ArXiv preprint, abs/2206. bbs/2206.04673, 2022. URL https://arxiv org/ /at/a/bb///.....eee/ee
20

---

Published in Transactions on Machine Learning Research (12/2023)
A Appendix
A.1 Transfer Difficulty
Table shows the transfer difficulty for each of the datasets used in our experiments.
Table 1: Transfer dilficulty for the 19 VTAB-1k datasets (plus CIFAR-10). The Score column is computed the difference between the accuracy of the All learnable parameter configuration and the Head configuration, normalized by the All accuracy, and then scaled by 100. The lower the score, the lower the difficulty of the transfer. In the TD column, we map the score into three buckets: a score of 0-5 is low, 5-10 medium, and greater than high. To compute the scores, we use the VTT-B backbone and use accuracies from the VTAB-Ik experiments for the VTAB-1k datasets and the accuracies from the results of the effect of shots and DP experiments at S 100 for CIFAR-10.
Dataset
Score
TD
Caltech101
Fei et al.
2006
0.4
low
CIFAR1O
Krizhevsky
2009)
1.0
low
CIFAR100
Krizhevsky
2009
7.8
modium
Flowers 102
Nilsback
Zisserman
2008
0.2
low
Pets
Parkhi
2012
1.1
low
Sun397
[iaao al.
2010
8.8
medium
SVHN Netzer et al.
201
52.9
high
DTD
Cimpoi et a..
2014
1.3
low
AuroSAT
Helber al.
2019
1.7
low
Resics45
Cheng el al. 2017
6.7
medium
Patch Camelyon Veeling al.
2018
3.8
low
Retinopathy (Kaggle & Eyel Pacs
2015
0.2
low
RLCoRtt (Johnson et al.)
2017
26.2
high
CLEVR- Johnson et al.
2017
38.7
high
dSprites-loc Matthey al.
201
71.4
high
dSprites-ori Matthey et al. 2017
37.7
high
OmalNNORB-ar (LeCun et al.
2004
33.3
high
SmalNNORB-ele (LeCun et a.
2004
28.2
high
DMLab Beattic et al. 2016)
21.9
high
KITTI-dis Geiger el al. 2013
13.6
high
21

---

Published in Transactions on Machine Learning Research (12/2023)
A.2
Additional Results
A.2.1
Additional Effect of Shots per Class and Results
Tables lo 7 depict labular results for different backbones (R-50, VIT-B), different learnable parameter sets (Head, FILM, All), different numbers of shots per class (S= 1, 10, 25, 50, 100, 250, 500) and various privacy levels (c 1,2, 8, o0), all at 5 = 1/(D)).
Table 2: Classification accuracy as a function of e, S and learnable parameters for CIFAR-10. Backbone is R-50 pretrained on ImageNet-211 Accuracy figures are percentages and the sign indicates the 95% confidence interval over runs with different seeds.
15
5S
10S
25S
505
1005
250.
500S
=
10.0+:.00
9.8+0.3
9.9+0.2
29.3:4.33
54.1#7.4
74.3+1.2
86.1+1.7
90.5+10.7
2
9.8.1..3
14.0=7.99
9.8:10.2
50.213..5
71.21:2.3
83.3+199
90.410.6
91.6110.1
All
= 4
10.0
9.810.3
9.810.3
69.2:14.1
81.913.1
88.241.0
91.510.3
92.010.3
=
10.0++.0
20.1
28.0++35.9
53.0++26.
85.9+22.0
90.1+0.88
91.8+00.5
92.8++0.8
OX
51.3+6.0
78.7+33.0
86.3+00.7
89.3+1.1
91.9++0.6
93.5++0.3
94.6+0.4
95.5++0.3
6
11.1+00.6
18.8+3.5
21.4+44
45.6+3.4
68.343399
78.3+44.6
89.6+0.9
92.4+-0.8
= 2
13.343.3
21.0=5.6
31.71:..6
63.443.0
80.912.4
86.9+0.9
92.3=0.4
93.6:L0.7
FiLM
= 4
14.9+4.3
30.3+8.5
52.4+3.4
76.1
86.2+1.7
89.1#1.2
93.4=0.00
94.4+0.3
= 17.115.3
40.115.0
67.015.8
80.911.2
89.211.7
92.410.7
94.010.3
94.7:10.3
48.113.6
79.416.3
86.5+2.6
91.7+10.4
94.1110.3
94.910.2
95.510.2
95.810.2
11.7+..00
18.7+1.6
20.5+8.3
44.7+4.7
68.7+-3.3
82.0+1.6
87.3+0.2
88.9+0.8
= 2
11.9433.
1..
31.148.4
62.24.8.8
80.1+1.2
86.24.0.6
89.3+0.4
90.0+00.7
Head
=
13.1+2.6
30.0=5.2
45.5+12.1
79.4+:.8
84.7+0.7
87.7
89.8#0.2
90.8:10.4
16.844.4
41.5-6.5
65.5++3.1
83.2+1.7
86.240.7
89.1
90.1+0.3
91.240.1
= DO
49.214.7
75.216.0
83.510..4
87.210.66
89.010.2
90.310.2
91.410.2
92.110.3
Table 3: Classification accuracy function of c, S and learnable parameters for CIFAR-100. Backbone is R-50 pretrained on ImageNet-2lk: Accuracy figures are percentages and the + sign indicates the 95% confidence interval over 3 runs with different seeds.
15
5S
105
25S
50.5
100S
250S
500S
= 1.0+0.0
1.7+00.8
1.7+1.4
7.9+
23.9++0.7
48.5+2.1
50.6+0.3
62.4+55.2
6 2
1.3+0.6
1.2+0.6
6.8+00.2
23.1+0066
46.0+2211
59.4+2299
61.9+6.0
65.7+5..2
All
4
1.0+0.0
1.6+1.33
15.3+11.1
43.7+3.99
57.6+2.8
64.8+:0.4
63.0+0.9
68.1
6 8
1.0+0.00
5.117.9
31.1422.
55.912.6
59.6+1..7
67.3+0..3
67.610.8
74.142.3
= D
28.342.4
51.915.6
59.749.9
71.940.5
76.0+0.8
79.9
82.2+3.0
85.8
= 1
1.040.4
1.8+0.5
3.7+0.2
14.2+0.2
34.2+1.8
59.2+1.0
75.3+00.
80.1+0.8
= 2
1.40.55
2.940.3
9.340.6
33.4+0.8
55.0+3.2
72.210.3
80.210.4
82.8+0.5
FiLM
E = 4 1.740.4
7.2+1.3
22.4+1.2
53.940.9
70.140.6
77.9+0.6
82.310.4
84.2+0.5
ε = 8
2.210.3
18.620.3
38.713.7
65.3+1.6
75.610.5
80.510.8
83.510.5
85.240.4
25.816.1
64.8-3.2
71.8+1.1
77.6+0.1
81.410.4
82.910.4
83.810.8
83.411.2
= 1.3:0.6
2.3+0.99
4.411.2
13.5
32.3+0.99
52.1+0.7
66.910.8
71.940..7
E 2
1.440.7
3.940.8
8.7+1.4
31.6+1.2
51.0+11.
63.1+1.0
71.610.4
75.240.3
Head
= 4
1.8+0.66
7.5+1.8
22.1+1.33
46.3+0.7
61.7+1.6
68.440.8
74.4+0..3
77.040.1
= 8
2.540.7
17.211.7
38.911.9
58.910.3
67.510.4
72.110.5
76.310.1
78.410.3
= DO
25.615.66
56.410.3
62.8+114
69.6+0.2
72.940.3
75.8+10.3
78.540.2
78.8
22

---

Published in Transactions on Machine Learning Research (12/2023)
Table Classification accuracy as a function of c, S and learnable parameters for SVHN. Backbone R-50 pretrained on ImageNet-21k. Accuracy figures are percentages and the + sign indicates the 95% confidence interval over 3 runs with different seeds.
1S
5$
105
25.
508
100.5
250S
500$
6
7.122.0
12.916.3
8.841.2
9.0+1122
8.842.3
20.543..
25.720.4
32.119.5
6
7.242.0
9.0
9.4+1.9
8.640.7
12.3:6.3
23.1+4.9
35.5+2.9
46.115.0
All
4
7.242.0
7.342.3
7.641.77
7.811.8
9.5+14
27.8+3.7
42.9+6.7
65.116.9
= 8
7.222.0
8.542.3
8.711.1
9.040.66
22.448.9
39.6
60.9
78.1
= 90
14.41.5
19.6#12.8
42.2=4.1
76.1
84.1+5.5
86.845.0
93.1+0.5
94.6-0.2
= 1
12.643.6
9.510.66
11.412.8
12.311.4
13.9
18.612.1
25.1=1.1
33.8
= 11.34.0
10.8+1.3
12.0#1.6
15.0
16.3:...5
21.7+1.4
32.913.5
45.3+2.9
FiLM
9 =
9.310.4
11.422.0
13.5=1.3
17.0.1.8
20.711.8
27.612.1
39.813.4
61.112.6
8 9.911.0
11.941.6
16.6=1.4
21.241.8
25.611.9
33.312.1
51.413.9
67.312.0
= 13.840.3
20.2#1.1
25.7=1.7
37.814.1
42.441.8
58.746.2
71.525.5
84.413.1
10.0+0.9
8.7+1.0
10.8+0.1
11.1+1.2
13.3+0.9
18.1+1.4
24.7=110
31.2+00.7
= 2
9.9+0.5
9.1+1.4
11.2=1.4
13.9+1.6
17.1
21.2+2.0
29.6+1.5
36.1
Head
é = 4
10.340.8
10.2+0.8
13.5#2.1
17.4+0.2
19.310.7
24.911.1
35.3#12
40.9#1.0
6
10.320.8
11.111.
15.023.1
19.91..8
23.34:.3
29.311.0
39.721.3
45.211.2
13.840.5
18.5+1.6
21.2=3.4
28.7+11.4
32.841.6
38.0+1.4
47.120.3
48.5+3.2
Table 5: Classification accuracy function of S and learnable parameters for CIFAR-10. Backbone is VIT-B pretrained on ImageNet-21k.. Accuracy figures are percentages and the + sign indicates the 95% confidence interval over 3 runs with different seeds.
1S
55
10S
25$
505
100S
2505
500S
= 1.3
20.9++1.2
18.8-7.4
64.9+13.2
70.3+11.11
84.2++10.9
93.04-2.0
95.3++1.0
6 12.7+1100
9.1-2.77
24.8=24.0
76.9+1.4
88.4+1.3
93.1+009
95.4+1.0
97.0=10.9
All
E = 4
12.6+1.4
42.7+3.2
59.2=5.2
86.8+1.5
91.8=11.0
95.3+0.7
96.910.7
97.6++0.5
6 12.711.7
51.719.6
54.9.10.7
86.2.10.5
90.914.9
96.6.10.8
97.310.33
98.110.3
64.3+12.8
91.4+1.6
95.1=0.88
97.2+0.3
97.6+0.4
97.9+0.3
98.3+0011
98.4+0.1
< 10.3+3.1
15.0+3.9
23.8=1.7
57.7+8.7
81.9+1.6
89.6+1.35
95.5+1.0
96.9+0.22
= 2
11.412.9
21.518.0
37.516.4
74.514.8
91.7+0.2
93.5111.
96.110.9
97.310.1
FILM
4
13.312.0
37.745.4
58.625.9
82.815.5
93.1=1.0
94.411.2
96.910.4
97.521.11
= 8
16.41.1
5144+0.99
71.5+2.0
89.444.4
94.6+111
96.0+1100
97.140.1
97.6+-0.6
67.047.4
92.142.77
95.3+2.1
97.2+1.1
97.9++0.6
98.0+004
98.6++0.1
98.7++0.1
14.612.7
19.1#5.0
30.326.7
56.6+2.1
81.5+1.7
90.611.5
95.310.33
96.4+0.2
= 2
15.7+2.9
23.7+4.9
44.7-12.3
74.949.3
86.2+2.1
93.9+0.2
96.310.4
96.9+0.1
Head
c
17.312.9
31.919.8
59.719.3
85.3.6..5
94.410.6
95.111.4
96.7.0.33
97.0.10.4
8 19.5+4.11
42.242.8
74.7-9.3
91.7+1.6
92.9+5.5
95.4+0.7
97.0+00.2
97.1+0.3
c 00
66.0+5.7
74.8+5.6
90.7-4.7
95.142.6
96.5+0.44
97.0+0.1
97.3+0.1
97.4+0.1
23

---

Published in Transactions on Machine Learning Research (12/2023)
Table 6: Classification accuracy as a function of e, S and learnable parameters for CIFAR-100. Backbone is VIT-B pretrained on ImageNet- 22kk Accuracy figures are percentages and the + sign indicates the 95% confidence interval over runs with different seeds.
15
5S
10S
25.5
50S
100S
250S
500S
=
1.140.3
1.0+0.33
3.44-2.1
18.74-3.3
41.04-2.0
62.74-2.0
80.2+-4.9
85.7+-2.9
1.110.1
3.2+1.9
11.911.4
39.91:2.5
60.81:2.2
78.0
87.310.1
89.5:10.6
All
4
0.9+0.2
10.5=2.1
24.3+222
56.4+3.5
68.5+15.8
82.9+5.3
89.0#1.3
90.2+1.2
= 8
1.3+0.4
17.3+7.7
18.8+2.9
60.8+18.8
84.2++0.3
86.9+22.4
90.3+0033
91.2+00.2
26.2414.5
78.1#1.0
85.310.6
88.420.7
89.620.3
90.9
92.1=0.2
93.0=0.00
= 1
1.34011
2.1:1.2
5.111.3
22.440.4
53.5+4.1
71.6+2.5
84.910.2
89.4
1.640.2
4.5
15.5+0.7
51.422.7
69.1+1.3
82.4+2.0
89.1
90.3
FiLM
= 4
1.610.11
11.21:19
35.314.0
66.213.2
82.010.8
87.0:10.4
90.610.4
92.0:1..2
C = 8
2.740.7
25.622.4
53.314.6
77.611.8
83.613.5
88.1+1.2
91.611011
92.3:10.5
42.142.5
79.1+3.1
84.2+2.8
89.4+0.5
90.6+00.5
91.6++0.4
91.9+188
90.9+1.3
f 1.3+0.4
2.8+0.3
5.740.77
24.2+0.9
49.0+3.22
70.9+0.33
82.1+0.5
10..3
= 2 1.40.22
5.720.1
12.742.
48.721.1
70.2+1.3
78.542.4
85.120.2
87.0.:0.3
Head
= 4 2.240.4
11.7=0.9
29.5+5.0
65.743.9
76.0+5.0
83.7.0.66
86.910.4
88.1
= 3.240.5
21.7=2.0
51.944.6
74.5+4.2
82.0+0.3
85.440.5
87.2
88.5+0099
35.8112.2
72.214.5
78.713.0
84.310.8
86.110.3
87.410.4
88.010.8
88.410.4
Table Classification accuracy as a function c, S and learnable parameters for SVHN. Backbone is VIT-B pretrained on ImageNet-224 21k. Accuracy figures are percentages and the + sign indicates the 95% confidence interval over 3 runs with different sceds.
5S
10S
25.S
50S
100S
250S'
500S
6 11.911.9
9.542.1
9.7122.4
9.2+1.5
10.311.4
14.114.6
22.412.8
33.52:14.9
= 2 11.7+1.7
10.1+0.3
9.9+1.9
8.6+-0.5
12.6+55.5
22.8+0.4
37.9+8.4
55.242009
All
6 = 4
10.7+0.5
10.9:2.1
10.8=1.7
9.7+2.0
15.616.2
28.625.3
45.8+17.7
66.1422.0
8
10.5+0.5
10.5#00.
9.120.55
14.315.1
25.5+4.0
36.6+18.3
64.6+314
84.4+5..3
= 2
10.5.1.2
15.6.11..2
28.4112299
63.0.126.9
86.1.5.00
91.2.110
93.011.1
91.211.0
= 11.7.3.0
10.712.4
11.1114
10.0.11.2
11412,4
17.0.11.7
26.4.1.11
43.7.14.5
11.642.8
12.240.6
10.3+0.9
13.1+2.8
14.4+4.2
23.5+0.6
41.0+3.6
68.6+4.00
FILM
= 4
10.3+2.2
11.1+22
12.5+2.2
15.8+4.0
20.5+1.9
30.3+4.4
64.8+4.66
77.3+2.3
6 9.141.
13.111.4
14.4=2.1
20.910.9
23.412.1
53.414.8
74.311.2
83.710.5
12.6+2266
22.1+1.2
31.0+3.6
65.9+21.8
83.743.1
87.7+5.9
92.2+0.7
93.6+0.7
9.1+0.5
10.0+2.9
10.9+1.5
11.5+0.3
12.5+0.4
15.5+0.9
21.5+1.6
31.1+1.5
= 9.240.7
11.742.9
12.1
12.4#1.6
15.8114
22.0=2.1
28,842.1
37.4
Head
4
9.740.5
12.2+4.1
12.9=0.1
15.5+2.2
19.5+0.5
24.8+2.1
35.5+1.2
42.9+1.0
c = 9.3.0..8
10.7.1..8
14.5.0.2
18A10.55
23.212.1
29.6.1..3
40.5.0..9
16.3.11.1
12.712.0
14.6.12.8
23.5.10.9
29.6.11.6
33.8.2.6
38.8.11.3
47.6.11.7
52.6.11.1
24

---

Published in Transactions on Machine Learning Research (12/2023)
A.2.2 Additional Shot Multiplier Variations
We compute the multiplier for a configuration and dataset at e as follows: using the median accuracy obtained through the experiments depicted in Tables 2 to 7 (S 5, 10, 25, 50, 100, 250, 500) we linearly interpolate the median accuracy in the complete S [1, 500] grid. We determine the minimum S required to reach least the same accuracy for non-private at S (5, 10) using the S [1, 500] grid. The multiplier is then the minimum 8 required for DP divided by the S for non-private.
The Figs. 10 and
11
display the same analysis as Fig. 2 for all backbones (VIT-B, R-50) and non-private The Figs. 12 and 13 display the same analysis grouped by datasets.
shots of S (5, 10).
VIT-B (All) for non-private 5
VIT-B (FILM) for non-private $
32
32
Multipller
16
Multiplier
16
8
SVHN
8
CIFAR-10
CIFAR-100
c=1
E=2
6=4
E=8
E=1
6=2
6=4
E=8
VIT-B (Head) for non-private S 5
R-50 (All) for non-private =
32
32
Multiplier
16
Multiplier
4
4
2
2
E=1
2=2
E
6=1
e=2
c=
E=
R-50 (FiLM) for non-private $ m
R-50 (Head) for non-private S
32
32
Muttiplier
16
Multiplier
16
1
6=1
6=2 6=4
E=8
6=1
6=2
6=4
6=8
Figure 10: Multiplier of shots required to reach same accuracy as non-private with S = 5 for VIT-B and R-50 on CIFAR-10, CIFAR-100 and SVHN with 8 = 1//D). The data is obtained using linear interpolation of the median results of the experiments of Appendix 1.2.1 The multiplier is 1 for all for ViT-B with All parameters on SVHN at S = 5 (top left plot) because non-DP achieves a random accuracy and achieving random accuracy requires 5 1 for all configurations in the experiment.
25

---

Published in Transactions on Machine Learning Research (12/2023)
VIT-B (All) for non-private S 10
VIT-B (FILM) for non-private 5 = 10
32
32
Multiplier
16
Multiplier
16
8
SVHN
8
CIFAR-10
CIFAR-100
E=1
E=2
e=4
E=
E=1
e=2
E=
E=
VIT-B (Head) for non-private 5 10
R-50 (All) for non-private 5 = 10
32
32
Multipller
16
Multiplier
16
4
4
2
2
E=1
e=2
e=8
e=1
e=2
e=
e=8
R-50 (FiLM) for non-private 5 10
R-50 (Head) for non-private 5 = 10
32
32
Multiplier
16
Multiplier
16
6=1
6=2
E=
e=8
6=1
E=
6=
6=8
Figure 11: Multiplier of shots required to reach same accuracy as non-private with S 10 for VIT-B and R-50 on CIFAR-10, CIFAR-100 and SVHN with 8 = 1//D). The data is obtained using linear interpolation of the median results of the experiments of Appendix A.2.1
26

---

Published in Transactions on Machine Learning Research (12/2023)
VIT-B (SVHN) for non-private S = 5
VIT-B (SVHN) for non-private S = 10
32
32
Multiplier
16
Muttiplier
16
8
All
All
FiLM
FiLM
Head
Head
E=1
6=2
6=4
6=8
E=1
E=2
E=
6=8
VIT-B (CIFAR-10) non-private 5
VIT-B (CIFAR-10) for non-private 5
10
32
32
Multipler
16
Multiplier
16
All
All
FiLM
FILM
2
Head
Head
E=1
2
E=4
E=8
E=1
E=2
c
E=8
VIT-B (CFFAR-100) for non-private S =
VIT-B (CIFAR- 100) for non-private S = 10
32
32
Multiplier
Multiplier
16
All
All
4
4
FiLM
FiLM
2
Head
Head
6=1
6=2
E=4
E=8
c=1
6=2
6=4
6=8
Figure 12: Multiplier of shots required to reach same accuracy as non-private with $ E (5, 10] for VIT-B on CIFAR-10, CIFAR-100 and SVHN with 5 = 1//D). The data is obtained using linear interpolation of the median results of the experiments of Appendix A.2. The multiplier is 1 for all e for ViT-B with All parameters on SVHN at S 5 (top left plot) because non-DP achieves a random accuracy and achieving random accuracy requires 5 1 for all configurations in the experiment.
27

---

Published in Transactions on Machine Learning Research (12/2023)
R-50 (SVHN) for non-private 5 5
R-50 (SVHN) for non-private 5 10
32
Multipleer
16
Multiplier
16
8
All
All
4
FiLM
FiLM
Head
Head
E=1
E=4
E=8
c=1
E=2
E=8
R-50 (CIFAR-10) for non-private $ =
R-50 (CIFAR-10) 10) for non-private $ = 10
32
32
Multiplier
16
Multiplier
16
All
All
FiLM
FiLM
Head
Head
E=1
E=2
E=4
6=8
E=1
6=2
E=4
6=8
R-50 (00F for non-private S 5
R-50 (CIFAR-100) for non-private S
10
32
32
Multiplier
16
Multiplier
16
All
All
FiLM
FiLM
2
Head
Head
E=1
6=2
E=4
E=8
E=1
6=2
E=4
E=8
Figure 13: Multiplier of shots required to reach same accuracy as non-private with S E (5, 10) for R-50 on CIFAR-10, CIFAR-100 and SVHN with 5 = D/ The data obtained using linear interpolation of the median results of the experiments of Appendix A.2.1
28

---

Published in Transactions on Machine Learning Research (12/2023)
A.2.3
Additional versions of Fig.
E =
2
All
100
90
FILM
Accuracy [%)
Head
test
train
10
10
25
50
100
250
500
10
25
50
100
250
500
10
25
50
100
250
500
shots
shots
shots
8
100
90
Accuracy (%)
80
50
40
30
20
10
25
100
250
500
5
10
25
100
250
500
shots
shots
Figure 14: Test and train classification accuracy as a function of shots and learnable parameters (All, FiLM and Head) on VTT-B for SVHN for different with 6 1//D). The accuracy reported for the median runs of Table
6
E
100
All
90
FiLM
Accuracy (%)
80
Head
test
train
10
25
50
100
250
500
10
25
50
100
250
500
10
25
50
100
250
500
shots
shots
shots
=
8
90
Accuracy (%)
80
70
60
40
30
20
10
5
10
25
50
100
250
500
5
10
25
50
100
250
500
shots
shots
Figure 15: Test and train classification accuracy a function of shots and learnable parameters (All, FiLM
and Head) on VIT-B for CIFAR-100 for different with 8 1/[P]. The accuracy reported for the median runs of Table 6
29

---

Published in Transactions on Machine Learning Research (12/2023)
A.2.4
Comparison of Backbones for Effect of Shots and
Fig- 16
compares the backbones (VIT-B, R-50) using their best performing configuration. The VIT-B
backbone achieves comparable or better performance.
CIFAR-10 TD)
CIFAR-100 (medium TD)
SVHN (high TD)
Accuracy (%)
100
80
VIT-B
60
R-50
40
20
40
25
50
100
250
500
25
50
100
250
500
10
25
50
100
250
500
shots (S)
shots (S)
shots (S)
CIFAR-10 TD)
CIFAR-100 (medium TD)
SVHN (high TD)
Accuracy (%)
100
80
60
40
20
6
40
25
50
100
250
500
6
40
25
50
100
250
500
10
25
50
100
250
500
shots (5)
shots (S)
shots (S)
CIFAR-10 TD)
CIFAR-100 (medium TD)
SVHN (high TD)
Accuracy (%)
100
80
4
40
20
10
25
50
100
250
500
10
25
50
100
250
500
10
25
50
100
250
500
shots (S)
shots (S)
shots (5)
CIFAR-10 TD)
CIFAR-100 (medium TD)
SVHN (high TD)
Accuracy (%)
100
80
= 8
60
40
20
10
25
50
100
250
500
10
25
50
100
250
500
10
25
50
100
250
500
shots (S)
shots (S)
shots (5)
CIFAR-10 (low TD)
CIFAR-100 (medium TD)
SVHN (high TD)
Accuracy (%)
100
20
10
26
50
100
250
500
10
26
50
100
250
500
10
26
50
100
250
500
shots (5)
shots (S)
shots (S)
Figure 16: Classification accuracy for different as function of S and backbone (VTT-B, R-50) for CIFAR-10, CIFAR-100 and SVHN. TD (low, medium, high) refers to the transfer difficulty and is computed as in Appendix A.I The best performing configuration out All, FiLM and Head for each combination of c, S and backbone used. The accuracy reported over three seeds with the line showing the median and the band reporting the lowest and highest accuracy.
30

---

Published in Transactions on Machine Learning Research (12/2023)
A.2.5 Advantage of FiLM Function of Shots
Figs. 17 and 18 show the difference between the mean classification accuracy of FiLM and Head. Darker red
indicates FiLM is better. Darker blue indicates Head bet
CIFAR-10 TD)
CIFAR-100 (medium TD)
SVHN (high TD)
4.1
1.1 0.4 -1.0 0.2
-0.7
-0.6
-1.7
4.4
0.7 2.8 4.3
2.6 0.7 0.2 -1.1 1.5
4.9
12.6
40
-43
22
-72
-0.4
5.5 -0.4 -0.3 0.4
0.2
-1.
2.8
2.8
-1.1
3.8 4.0 3.3
2.4 0.4 -1.8
0.7
-1.4 1.5
12.131.2
20
-4.0
2.8
-1.1
-2.5 -1.3 -0.7
0.1 0.5
-0.6 -0.5
5.9
0.5 6.9 3.3
3.7 3.9
0.6 -1.1 -0.4
0.3
1.0
5.5
29.334.5
-33.1
9.2
-3.2
-2.3
1.7
0.2
-0.5
3.0
1.6
2.8
4.4 3.8
-0.2 2.4
-0.1
23.9
33.8333.
20
8
17.3
2.2 1.4 1.0 1.3
5.0 4.5 4.2 4.5
4.2
-0.0
7.5 7.5
36.3
49.9
4..444...44
48.9
-40
5
10
25
50
100
250
500
10
25
50
100
250
500
5
10
25
50
100
250
500
shots
shots
shots
Figure 17: Heat map showing the accuracy advantage FiLM over Head for CIFAR-10, CIFAR-100 and SVHN as a function of Backbone is VIT-B. Darker red indicates FiLM better. Darker blue indicates Head is better. Datasets ordered from low high TD.
CIFAR-10 (low TD)
CIFAR-100 (medium TD)
SVHN (high TD)
-06
0.2
0.9
0.9 -0.4 -3.7 2.3 3.5
-.2 -0.5 -0.6
0.7 1.9 7.1 8.4 8.2
2.6 0.8 0.5 1.2 0.7 0.5 0.4
2.5
40
1.4
-277
0.5
1.2
0.8 0.7 3.0 3.6
-0.0
1..
0.6
1.8
4.0
9.0 8.6 7.6
1.4 1.7 0.8
1.0
-0.8
0.5
3.2
9.2
0.2
-3.3
1.5
3.5
0011
-0.3 0.3
7.5
8.5
9.5
7.9 7.2
1.0 1.2 -0.0 -0.4
1.3
2.7
4.5
20.2
0
0.2
-1.4
1.5
-2.3
3.0
3.3
3.8
3.5
-0.3 1.3
-0.2
6.4
8.2
8.4
7.2 6.8
-0.4 0.8
1.6
1.2
2.3
4.0
11.8
20
8
-11.1
4.1
2.9 4.5 5.1 4.6 4.1
3.7
0.2
8.4 9.0 7.9 8.5 7.2 5.3 4.6
0.1
1.7 4.5 9.1 9.6 20.7 24.4
400
5
25
100
250
50
100
250
500
100
250
500
shots
shots
shots
Figure 18: Heat map showing the accuracy advantage of FiLM over Head for CIFAR-10, CIFAR-100 1000 and SVHN as function of c. Backbone is R-50. Darker red indicates FiLM is better. Darker blue indicates Head is better. Datasets ordered from low to high TD.
31

---

Published in Transactions on Machine Learning Research (12/2023)
A.2.6
Additional VTAB-1k Results
Tables to 13 depict tabular results for different backbones (R-50, ViT-B), different learnable parameter sets (Head; FiLM,, All), and various privacy levels (e 1, 2, 4, 8, oo), all 10-3
Table 8: Classification accuracy as function of for each of the datasets in the VTAB-lk benchmark. Backbone is R-50 pretrained on ImageNet-21k. Learnable parameters are Head. Accuracy figures are percentages and the sign indicates the 95% confidence interval over 3 runs with different seeds.
dataset
classes
6 1
=
c= 4
6
8
c 0e
Caltech101
Fei-Fei et al.
2006
102
11.8#6.9
30.0
57.113.6
69.311.4
87.9=0.2
CIFAR100
Krizhevsky
2009)
100
4.211.2
10.6#1.0
20.841.6
34.7+2.2
61.52066
Flowers
Nilsback Zisserman
2008
102
11.31:277
3316.4
73.120.7
89.8+2.1
98.4=0.1
Pets
Parkhi et al.
2012
37
28.6:15.4
5042.4
65.611.5
73.4+1.0
84.4=0.3
Sun397
(iiao et al.
2010
397
4.7+00.2
8.440.22
13.4+1.2
21.5+00.7
46.2=0.4
SVHN Netzer
201
23.0=t1.0
26.8+2.55
30.611.9
34.9#2.0
41.9=2.2
DTD Cimpoi et a..
2014
47
19.6#4.0
36.241.3
51.211.2
61.3+0.9
72.0=0.4
EuroSAT Heller al.
2019
77.2:11.9
85.3
88.4+1
91.1+0.0
94.3=0.2
Resics45 Cheng et al.
2017
45
19.3:3.5
33.442.8
48.642.5
60.9
78.5=0.2
Patch Camelyon Veeling et al.
2018
2
77.6+22.
79.2+=100
80.8+1.6
80.6+0..3
81.2=0.2
Retinopathy Kaggle & EyePacc Pacs
2015
73.2+0.6
73.7+0..3
73.4+0.6
74.0+0.4
75.2-0.1
CLEVR-count (Johnson et al.
2017
27.5++1.5
30.14119
33.6+1155
36.8+-211
51.2+1.2
CLEVR-dist Johnson et al.
2017
26.41:2.2
28.7++0.7
29.8+0.9
30.9
36.2=0.9
dSprites-loc Matthey al.
2017
16
6.640.1
6.8:0..7
7.640..5
7.511.2
18.9
dSprites-ori Matthey et al. 2017
16
9.340.9
10.840.5
13.240.7
16.210.2
45.9=2.5
SmalINORB-aa (LeCun et al. 2004
18
6.120.5
7.520.55
8.140.4
8.720.7
11.7=0.1
SmallNORB-elee (LeCun et al.
2004
1713.3
19.8#1.1
22.5+0.9
24.520.7
31:0.5
DMLab (Beattie et al. 2016
26.9
28.8+0.66
31+0.2
32.4+0.1
34.6=3.7
KITTI-dist Geiger ct al. 2013
54.5:12.5
59.643..5
66.911.66
65.4+1.3
69.2-0.7
All
27.6
34.7
43.0
48.1
58.9
Natural
14.7
27.9
44.5
55.0
70.4
Specialized
61.8
67.9
72.8
76.7
82.3
Structured
21.8
24.0
26.6
27.8
37.3
32

---

Published in Transactions on Machine Learning Research (12/2023)
Table 9: Classification accuracy function of for cach of the datasets in the VTAB-1k benchmark. Backbone is R-50 pretrained ImageNet-21k. Learnable backbone parameters are FiLM. Accuracy figures are percentages and the + sign indicates the 95% confidence interval over 3 runs with different seeds.
dataset
classes
1
2
4
8
Caltech 101
Fei-Fei et al.
2006
102
11.3+1.3
35.8+44.1
55.7+0.5
72.2+1.9
88.8+0.5
CIFARI00
Krizhevsky
2009
100
3.440.8
10.24+0.5
23.2+0.8
3..
71.7=113
Flowers 102
Nilsback
Zisscrman
2008
102
10.410.6
34.245.4
70.4+1.0
89.310.3
98.7=0.1
Pets (Parkhi et al.
2012
28.411.4
48.843.0
64.111.1
75-1.1
8810.4
Sun397
Xiao et al.
2010
397
4.2+0.55
8.0+0.1
14.1+0.8
21.7+00.8
46.8+0.7
SVHN Netzer et al.
20T
23.3+-144
28.140.99
32.9+0.7
38.643.22
56.6-2.4
DTD Cimpoi et al.
2014
20.8::2.7
36.7+1.5
50.314.55
61.4+1.3
72.4-0.4
EuroSAT (Helber et al.
2019
10
79.2:10.8
85.141.5
88.8122.2
02.1
95:10.1
Resics45 Cheng et al. 2017
21.11117
35.240.6
49.511.6
61.310.8
81.9=0.1
Patch Camelyon (Veeling et al.
2018
2
76.840.8
77.342.7
79.111.2
79.4+0.4
81.3-0.1
Retinopathy Kaggle & EyePacs
2015
73.420.3 5
73.540.1
73.910.5
74.400.2
74.0=3.2
CLEVR-count (Johnson al.
2017
29.1=11.5
31.040..
34.61:33
38=1.4
73+1.3
CLEVR-dist Johnson et al.
2017
26.7#1.3
28.7+0.77
30.5+0.6
31.8+0.66
49.3=1.6
dSprites-loc Matthey et al.
2017
6.6+0.3
6.440.3
6.7+0..5
8.5+144
64.0-8.7
dSprites-ori Matthey al. 2017
16
9.1+2.0
11.2+1.7
12.3+1.0
16.7+0.8
56.8+3.8
SmalINORB--an (LeCun et al.
2004
18
6.440.4
7.3+0.8
8.240.8
9.4+0.5
14.6=0.2
SmalINORB-clev
(LeCun et al.
2004
17.6411.0
20.8::0.3
22.71111
25.611.33
32.0=4.0
DMLab Beattii al. 2016
25.8+0.7
28.7+t0.7
30.5+0.9
32.1
41.8=0.4
KITTI-dist Geiger et al. 2013
4
56.3+11.6
60.7+
63.4+2..5
6..5+1.6
80.4+0.5
All
27.9
35.1
42.7
49.2
66.7
Natural
14.6
28.8
44.4
56.7
74.7
Specialized
62.6
67.8
72.8
76.8
83.1
Structured
22.2
24.3
26.2
28.8
51.5
33

---

Published in Transactions on Machine Learning Research (12/2023)
Table 10: Classification accuracy as a function of e for each of the datasets the VTAB- 1k benchmark.
Backbone is R-50 pretrained on ImageNet-21k. All parameters are learnable. Accuracy figures are percentages and the + sign indicates the 95% confidence interval over runs with different seeds.
dataset
classes
2
Caltech101 Fei et al.
2006
8.218.5
17.617.0
26.013.9
33.013.9
86.812.0
CIFAR100
Krizhevsky
2009
100
1.0+0.1
2.3+1.5
6.942.9
13.3+2.7
59.3+7.0
Flowers102
Nisbbck
Zisserman
2008
102
6.24288
6.927.1
33.7211.6
69.717.3
95.811.8
Pets (aarkh al.
2012
37
12.8+0.55
22.7+-2.2
32.4+4.1
21.74+9.0
83.0=0.22
Sun397
Xiao et al.
2010
397
3.340.4
3.0
2.7+0.4
3.41122
38.3=0.8
SVHN Netzer et
201
19.210.8
23.615..5
26.817.00
37.212.3
88.512.5
DTD Cimpoi et al.)
2014
47
13.7+3.6
21.6+2.3
27.7+3.8
34.0+:1.9
72.4+0.1
EuroSAT Helber et al.
2019
10
49.8+
69.2
72.7=1.2
32.4+
96.0=1.0
Resics45 Cheng et al. 20017
45
11.91144
13.816.4
24.311.0
26.816.8
84.111.1
Patch Camelyon Veeling et al.
2018
65.9+15.7
70.5+20.1
80.5+1.2
79.5+2.9
85.0+0.8
Retinopathy Kaggle & EyePacs
2015
73.640.0 5
73.610.0
73.6=0.00
73.610.0
76.0=1.3
CLEVR-comtt Johnson al.
2011 17
18.144.6
26.3+-2.0
36.2422.7
41.4+7.0
93.2+-0.2
CLEVR-dist Johnson et a...
2017
23.8+:14
22.7+2.2
25.2+1.7
36.913.3
62.1=1.7
dSprites-loc Matthey
2017
16
6.210.11
6.4.0.3
6.310.1
6.210.11
89.113.7
dSprites-ori Matthey et al.
2017
16
7.5+0.0
6.6
7.2+0.1
8.8+2.9
61.0=3.2
OmalINORB azi (LeCun a..
2004
5.410.2
5.710.1
5.710.3
6.240.8
21.913.3
SmallNORB-eleer [LeCun et al.
2004
12.3+1.2
13.8+2.44
21.2+2.7
22.7+-5.5
39.5+6.3
DMLab [Batttie et al.] 2016
22.510033
24.913.1
28.5=1.4
30.315.2
48,420.8
KITTI-dist (Geiger et al. 2013
31.4.6..7
46.9
55.2-10.5
60.64
81.1
All
20.8
25.2
31.2
36.4
71.7
Natural
9.2
13.9
21.1
28.7
73.3
Specialized
50.3
56.8
61.1
65.6
85.3
Structured
16.3
19.2
23.2
26.6
62.0
34

---

Published in Transactions on Machine Learning Research (12/2023)
Table 11: Classification accuracy function of for cach of the datascts in the VTAB-1k benchmark. Backbone is VIT-B pretrained ImageNet-2lk.. Learnable backbone parameters are Head. Accuracy figures are percentages and the + sign indicates the 95% confidence interval over 3 runs with different seeds.
dataset
classes
1
2
6= 4
8
Caltech 101
Fei-Fei et al.
2006
102
20.8++2.1
39.7+5.7
65.6+1.1
79.9+0.33
93.3-0.3
CIFARI00
Krizhevsky
2009
100
7.0+1.4
15.9422.7
33.
49.9+2.3
77.6=2.4
Flowers 102
Nilsback
Zisscrman
2008
102
13.743.0
47.241..
85.4+1.8
93.512.6
99.3-0.3
Pets (Parkhi et al.
2012
38.3122.
65.620.2
76.013.9
81.1#2.4
90.7=0.1
Sun397
Xiao et al.
2010
397
3.5+0.5
6.9+0.8
13.2+1..5
24.0+0.3
51.0+3.4
SVHN Netzer et al.
20T
23.3+114
27.24+1.5
31.6+1.2
35.3+0.3
43.1+0.4
DTD Cimpoi et al.
2014
20.41:2.6
37.043.1
49.9
61.613.2
75.7=0.3
EuroSAT (Helber et al.
2019
10
81.3.1..3
87.0+1.0
80.910.9
01.61.11
94.6-0.4
Resics45 Cheng et al. 2017
23.2422.
41.412.1
58.012.7
67.9
82.5=0.5
Patch Camelyon (Veeling et al.
2018
2
79.8+2.9
78.542.1
81.611.8
82.8+0.4
83.8-0.7
Retinopathy Kaggle & EyePacs
2015
73.3 5
72.6+1..3
73.6
74.0+0.2
73.8=2.3
CLEVR-count (Johnson al.
2017
25.5+10.9
27.7+1.
30.810.4
33.3+0.5
42.5=0.5
CLEVR-dist Johnson et al.
2017
26.1+0.77
27.5+00.5
30.1+0.3
31.5+0.5
35.1=0.3
dSprites-loc Matthey et al.
2017
6.9+0.5
7.8+0.77
8.7+0.11
9.4+0.6
19.1=2.7
dSprites-ori Matthey al. 2017
16
11.2+0.9
13.3+1.2
15.6+0.9
18.9+1.5
31.2-0.6
SmalINORB-azr (LeCun et al.
2004
18
6.940.6
7.8+0..5
8.141.3
9.0+0.88
12.2+0.1
SmalINORB-clev
(LeCun et al.
2004
16.94114
19.340.4
20.4+1.1
22.7+1.8
27.5=0.4
DMLab Beattii al. 2016
29.2+1.77
33.0+:1.6
35.0+1.0
37.3+1
40.2=0.6
KITTI-dist Geiger et al. 2013
4
51.3+8.4
57.1+
61.2+00.6
61.4+330
65.7+3.2
All
29.4
37.5
45.7
50.8
59.9
Natural
18.1
34.2
50.7
60.8
75.8
Specialized
64.4
69.9
75.8
79.1
83.7
Structured
21.7
24.2
26.2
27.9
34.2
35

---

Published in Transactions on Machine Learning Research (12/2023)
Table 12: Classification accuracy function of for cach of the datascts in the VTAB-1k benchmark. Backbone is VIT-B pretrained on ImageNet-21k. Learnable backbone parameters are FiLM. Accuracy figures are percentages and the + sign indicates the 95% confidence interval over 3 runs with different seeds.
dataset
classes
1
2
6 4
8
Caltech 101
Fei-Fei et al.
2006
102
11.7+6.00
42.9+4499
65.7+3.0
78.7+-2.6
94.1-0.8
CIFARI00
Krizhevsky
2009
100
7.140.2
1..4+11.
35.4+2.00
52.9+2.00
83.8=0.6
Flowers 102
Nilsback
Zisscrman
2008
102
16.0::2.8
48.8:5.22
85.311.7
96.420.7
99.5-0..0
Pets (Parkhi et al.
2012
39.31:222
62.543.6
78.010.9
83.612.4
91.8=0.4
Sun397
Xiao et al.
2010
397
2.7+0.4
7.1+1.2
14.6+110
23.1+0.5
53.7-2.0
SVHN Netzer et al.
20T
25.1+1.5
28.0+-1.1
33.440077
52.4+7.4
79.1+2.6
DTD Cimpoi et al.
2014
17.542..0
33.0
50.5+1.4
61.7+1.1
75.3=3.9
EuroSAT (Helber et al.
2019
10
79.6
86.6:12.2
90.910.2
01.01..5
96.5-0.2
Resics45 Cheng et al. 2017
22.0:3.11
40.642.3
55.513.9
66.1
87.0=0.5
Patch Camelyon (Veeling et al.
2018
2
76.612.7
78.142.2
80.111.6
80.620.4
82.8=1.0
Retinopathy Kaggle & EyePacs
2015
73.520.1 5
73.440.4
73.5
73.5+0.9
74.5=0.6
CLEVR-count (Johnson al.
2017
25.6:11.5
28.0+:..
31.610.55
33.7=0.6
52.0=3.3
CLEVR-dist Johnson et al.
2017
26.5+0.99
29.0+1..7
32.6+1.3
38.0+3.2
52.6=6.4
dSprites-loc Matthey et al.
2017
8.0+1.6
12.2+1.1
20.946.1
29.8+5.4
68.1
dSprites-ori Matthey al. 2017
16
9.3+0.7
13.7+1.5
17.0+1.4
21.5+1.1
47.8+3.9
SmalINORB-azr (LeCun et al.
2004
18
6.640.3
7.3+0.5
8.440.3
9.1+0.4
15.1=1.5
SmalINORB-clev
(LeCun et al.
2004
16.241.6
19.241.3
21.5+:1.2
22.911.4
35.3-44.6
DMLab Beattii al. 2016
29.8+11.7
33.3+:0.5
35.5+
36.5+0.9
43.3=3.6
KITTI-dist Geiger et al. 2013
4
53.64+2.6
60.8+.2.1
63.5+
66-2.9
76.9
All
28.8
38.0
47.1
53.6
68.9
Natural
17.0
34.3
51.9
64.1
82.4
Specialized
62.9
69.7
75.0
77.8
85.2
Structured
21.9
25.4
28.9
32.2
48.6
36

---

Published in Transactions on Machine Learning Research (12/2023)
Table 13: Classification accuracy function of for cach of the datascts in the VTAB-1k benchmark. Backbone is VIT-B pretrained on ImageNel-21k: All parameters are learnable. Accuracy figures are percentages and the = sign indicates the 95% confidence interval over 3 runs with different seeds.
dataset
classes
1
2
6 4
8
Caltech
Fei-Fei et al.
2006
102
16.1+5.2
34.9+2.1
55.3+110
69.9+2.77
93.7-0.4
CIFARI00 Krizhevsky
2009
100
7.140.4
14.34.0.7
24.2+115
36.2+4.0
84.2=0.3
Flowers 102
Nilsback
Zisscrman
2008
102
10.6::2.
3314.9
77.316.9
96=1.2
99.5-0..0
Pets (Parkhi et al.
2012
26.7:6.00
56.917.0
76.013.7
84.2+0.7
91.7=0.2
Sun397
Xiao et al.
2010
397
2.442.1
5.7+11.5
7.740.4
11.6+3.1
55.9-0.2
SVHN Netzer et al.
20T
22.9+1155
28.84.0.7
344-5.6
44.3+9.00
91.6-0.8
DTD Cimpoi et al.
2014
17.3
29.34:244
41.1+1.2
51.7+5.0
76.7=0.5
EuroSAT (Helber et al.
2019
10
74.3
78.912.2
8611.4
01.611.6
96.3
Resics45 Cheng et al. 2017
16424
2811.6
45.713.3
60.8+2.1
88.4=0.4
Patch Camelyon (Veeling et al.
2018
2
74.141.5
76.611.4
78.912.1
76.245.3
87.1-0.7
Retinopathy Kaggle & EyePacs
2015
5
73.420.5
73.1
73.6+0.1
73.6+0.1
74.0=1.3
CLEVR-count (Johnson al.
2017
21.5:15.6
28.841.5
33.612.4
38.220.7
57.6=8.7
CLEVR-dist Johnson et al.
2017
27.0=
36.443.5
42.2+3.2
45.
57.2=2.5
dSprites-loc Matthey et al.
2017
6.440.5
7.9+3.4
22.7+2.6
37.645.0
66.8-5.2
dSprites-ori Matthey al. 2017
16
7.9+2.1
11.1+3.7
13.5+6.5
19.9+2.99
50.1+1.1
SmalINORB--ae (LeCun et al.
2004
18
5.940.7
7.940.8
8.540.4
11.4+2.3
18.3+0.7
SmalINORB-clev
(LeCun et al.
2004
14.5.11.0
17.043.8
18.7144
26.740.1
38.3=2.9
DMLab Beattii al. 2016
29.2+1.3
32.7+11.4
35.4+:2.0
39.3#1.1
51.5=1.9
KITTI-dist Geiger et al. 2013
41.7=+5.8 4
51.24:3.4
57.9+8.6
68.9+0.3
76.0=0.7
All
26.1
34.3
43.8
51.8
71.3
Natural
14.7
29.0
45.1
56.3
84.8
Specialized
59.4
64.2
71.0
73.6
86.4
Structured
19.3
24.1
29.1
36.0
52.0
37

---

Published in Transactions on Machine Learning Research (12/2023)
Figs. 19 and 20 depict the complete set of VTAB-1k accuracy results function of dataset, privacy level (c), backbone, and learnable parameters. The datasets are ordered increasingly by transfer difficulty (TD). Although classifiers for the Retinopathy dataset appear to perform equally well independently of e, a closer inspection reveals that this dataset is unbalanced and learned classifiers predict the most common class in all set settings.
100
All, =
FiLM, c=
Accuracy (%)
80
Head, 6
60
All, c=1
40
FiLM, c=1
Head, c=1
0
retinopathy
flowers
caltech
pets
dtd
eurosat
camelyon
resics
cifar
sun
kitti
dmlab
clevr-c
snorb-el
shorb-az
dsprites-o
clevr-d
svhn dsprites-1
100
All, c
FiLM, E=
Accuracy (%)
80
Head, == 0
60
All, c=2
40
FiLM, c=2
20
Head, c=2
retinopathy
flowers
caltech
pets
did
eurosal
camelyon
resics
cifar
sun
kitti
dmlab
clevr-c
snorb-el
snorb-az
dsprites-o
clevr-d
svhn dsprites-!
100
All,
FiLM, 6=
Accuracy (%)
80
Head, E 00
60
All, e=
40
FiLM, <=4
20
Head, e=4
retinopathy
flowers
caltech
pets
dtd
eurosat
camelyon
resics
cifar
sun
kittl
dmlab
clevr-c
snorb-et
snorb-az
dsprites-o
clevr-d
svhn dsprites-1
100
All,
FiLM, c=
Accuracy (%)
80
Head, t oo
60
All, e=8
40
FiLM, c=8
20
Head, e=8
0
retinopathy
flowers caltech
pels
dtd
eurosal
camelyon
resics
cifar
sun
kitl
dmlab
clevi-c
snorb-el
snorb-az
dsprites-o
clevr-d
swhn dsprites-
Figure 19: Classification accuracy for VTAB-1k datasets function of privacy level (c) and learnable parameters. Backbone is R-50. Dashed lines in all plots indicate non-private accuracy reference. The datasets are ordered increasingly by transfer difficulty (TD).
38

---

Published in Transactions on Machine Learning Research (12/2023)
100
All, 6=
FiLM, c=
Accuracy (%)
80
Head,
60
All, 6=1
40
FiLM, c=1
20
Head, 1=1
0
retinopathy
flowers
caltech
pets
dtd
eurosat
camelyon
resics
cifar
sun
kitti
dmlab
clevr-c
snorb-et
shorb-at
diprites-o
clevr-d
svhn dsprites-4
100
-Mp All, E= 0o
FILM, c=
Accuracy (%)
80
Head,
60
All,
40
FiLM, e=2
20
Head, c=2
0
retinopathy
flowers
caltech
pets
dtd
eurosat
camelyon
resics
cifar
sun
kitti
dmlab
clevr-c
snorb-et
snorb-az
dsprites-4
clevr-d
svhn dsprites-
100
All, c0
FiLM, c=
Accuracy (%)
Head, 00
All, e=4
40
FiLM, c=4
20
Head, e=4
0
retinopathy
flowers
caltech
pets
dtd
eurosat
camelyon
resics
cifar
sun
kitti
dmlab
clevr-c
snorb-el
snorb-az
dsprites-o
clevr-d
svhn dsprites-
100
All, c=
FILM, c=
Accuracy (%)
Head, e= oa
All, e=8
40
FiLM, e=8
20
Head, e=8
0
retinopathy
flowers caltech
pets
did
eurosal
camelyon
resics
cifar
sun
kitti
dmlab
clevr-c snorb-el
snorb-az
dsprites-o
clevr-d svhn
dsprites-1
Figure 20: Classification accuracy for VTAB-1k datasets as a function of privacy level (c) and learnable
parameters. Backbone VIT-B, Dashed lines in all plots indicate non private accuracy as a reference. The datascts are ordered increasingly by transfer difficulty (TD).
39

---

Published in Transactions on Machine Learning Research (12/2023)
Fig. 21 depicts the final training and test accuracy function of and learnable parameters for all 19 VTAB-1k datasets with the ViT-B backbone. Although classifiers for the Retinopathy dataset appear to perform equally well independently of closer inspection reveals that this dataset unbalanced and learned classifiers predict the most common class in all settings.
Retinopathy
Flowers102
Cattech101
Pets
100
Accurapy (N)
DTD
EuroSAT
Patch Camelyon
Resics45
Accuracy
CIFAR100
Sun397
KITTI-dist
DMLab
CLEVR-count
SmallNORB-elev
SmallNORB-azi
dSprites-ori
38:88
Accuraty (%)
058858830 0888888888
CLEVR-dist
SVHN
dSprites-loc
Accuraty (%)
FILM
Head
accuracy
train
random accuracy
Figure 21: Test and train classification accuracy as a function of and learnable parameters (All, FiLM and Head) on VIT-B for all VTAB datasets with 8 = 1/[D]. The accuracy reported for the median run of Tables to 13 The datasets in order of increasing transfer difficulty (TD) from left-to-right and top-to-bottom.
40

---

Published in Transactions on Machine Learning Research (12/2023)
A.2.7
Additional Membership Inference Attack Results
Fig. 22 depicts the complete set of ROC curves for LiRA on CIFAR-100 with the R-50 backbone for various privacy levels (e) and learnable parameters Hend and FiLM al a fixed S.
Fig. 23 depicts the complete set of ROC curves for LiRA CIFAR-100 with the R-50 backbone for various shots S at fixed privacy levels (c) and learnable parameters Head and FILM.
Table 14 presents the True Positive Rates (TPR) at various False Positive Rates (FPR), Area Under Receiver
Operating Curve (AUC), and Attack Advantage (Attack Adv) (Yeom et al. 2018 for various privacy levels
(e) and shots per class (S) corresponding to the plots in Figs. 22 and 23
S = 10
S = 25
10a
109
10
10-1
10
10-2
TPR
TPR
FILM:E=>, TPR=0.525
10-3
FiLM:s=,, 4PR:0.449
10-3
Head:e=ee, TPR=0.622
Head:e==, TPR=0.539
FILM:e=8, TPR=0.015
FiLM:e=8, TPR=0.011
Head:e=8, TPR=0.010
Heads TPR=0.012
10
10
FILM:E=4, TPR=0.006
FiLM:e=4, TPR=0.005
Head:e=4, TPR=0.007
Head: TPR=0.003
FiLM:e=2, 0PR=0.003
FiLM:==2, TPR=0.003
10-5
Head:r=2, TPR=0.004
Head: 0PR=00003
1
1iLM:e=1, 000000002
FiLM:-=1, TPR-0.0022
Head:e=1, TPR=0.002
Head: TPR=0.002
10-6
10-5
10
10
10
10
10
10
10
10
10
10
10
10
FPR
FPR
S = 50
S = 100
10a
10a
10
10-1
10-2
10-2
TPR
10-3
FiLM:e=>, TPR=0.227
TPR
10-3
FiLMss=,, TPR=0.077
Head: TPR=0.410 TPR-0.00
Head:e=,, TPR=0.241
FiLM:e-8, TPR-0.009
TPM-00000
10
FiLM:--8, TPR-0.008
10-4
Head:e-8, TPR-0.010
Headte-8, TPR-0.008
FiLM:e-4, TPR-0.004
FiLMie-4, TPR-0.004
Head:e-4, TPR-0.004
10-5
Head:e-4, TPR-0.004
1
FiLM:e-2, TPR-0.002
FiLM:e-2, TPR-0.002
Head:e=2, TPR-0.002
Head:e-2, Moad:==2.TPR-00002 TPR-0.002
TPR-0.002 TP8-0.022
10-8
FiLMie-1, TPR-0.002
1
Head:e-1, TP8-0.002
Head:e-1, TPR-O.02 TPR-0.002
10-6
10-5
10-4
1
10-2
10-1
10-5
10-4
10-2
10-1
FPR
FPR
Figure 22: ROC curves for LiRA Carlini et al.
2022 on CIFAR-100 with R-50 backbone for various privacy
levels (e) and backbone configurations Head FiLM at fixed TPR values in legends are measured FPR=0.001.
41

---

Published in Transactions on Machine Learning Research (12/2023)
6=1
6=2
10a
10-1
10
10-2
10
TPR.
10-3
TPR.
10
10
50 TPR=0.002
Head: 5-20. TPR-0.004
FILM: 5-20. TPR=0.002
10
ALM: S-20, TP8-0.003
5-50, TPR=0.002
Head: TPR-0.003
10
FILM 5-50. TPR-0.002
FILM: 5=50, TPR=0.003
5-100, TPR=0.002
10
S-100. TPR=0.002
FILM: 5-100, TPR-0.002
RLM: 5-100, TPR-0.000
10
5-200, TPR-0.002
Head: 5-200, TPR-0.002
FILM: 5=200, TPR=0.002
10
FILM: S=200, TPR=0.002
5
10
10-1
10
10
10
30
10
10
109
FPR
FPR
==
E=
109
10a
10-1
10
10
1PA
10
TPR
10"
10
Head: S=20. TPR=0 007
10
Head: S=20, TPR=0.010
FILM: 5-20. TPR-0.00
5-20. TPR-0.015
Head: 5-50, TPR-0.003
Head:S-50 5-50, TPR-0.012
5-50, TPR-0.0US
10
TPR=0.011
Head: 5-100, TPR-0.004
Head: S-100, TPR-0.010
5-100,
5-100. TPR-0.009
Head: 2=200, 0PR=00004
S=20: TP8-0.000
S=200, TPR=0.0000
10-6
10
10
10
10-7
10
10a
10th
10-5
10
10-
10-2
10-1
10
FPR
FPR
E
10
10
10-3
10a4
Head: 5=20, TPR=0.822
Head: 5-50, TP8-0.539
10
FUM: S=50. 1PR=0.449
Head: 5-100, -PR-0.410
RUM: 5-100, TPR-0.2P2
10
5-200, TRR
FLM: 5-200, -PR-0.077
10
10
10
10
10
10-1 10"
FPR
Figure 23: ROC curves for LiRA CCrlini et al. fixed privacy levels (e) and backbone configurations Head and FiLM. TPR values in legends are measured at FPR-0.001.
2022 CIFAR-100 with R-50 backbone for various S at
42

---

Published in Transactions on Machine Learning Research (12/2023)
Table 14: True Postive Rates (TPR) various False Positive Rates (FPR), Area Under Receiver Operating Curve (AUC), and Attack Advantage (Yom et al. 2018) for varions privacy levels (c) and shots class (S)
corresponding to the plots in Figs. 22 and ImageNet-21k.
Dataset (D) is CIFAR-100. Backbone is R-50 pretrained on
TPR (%) @ 0.1% FPR
TPR (%) @ 1% FPR
TPR (%) @ 10% FPR
AUC
Attack Adv
S
Head
FiLM
Head
FILM
Head
FILM
Head
FILM
Head
FiLM
10
0.20
0.22
1.85
1.88
14.71
15.10
01566
0.572
0.092
0.106
25
0.17
0.17
1.52
1.61
13.51
13.77
0.550
0.552
0.070
0.074
50
0.16
0.16
1.50
1.50
13.04
12.99
0.541
0.541
0.058
0.058
100
0.16
0.15
1.43
1.41
12.54
12.26
0.535
0.531
0.049
0.042
10
0.38
0.30
2.62
2.56
18.76
18.80 13.83
0.610
0.613
0.164
0.166
25
0.27
0.27
2.19
2.12
17.12
16.83
0.593
0.593
0.138
0.138
0.23
2.05
15.95
0.579
100
0.23
1.81
15.11
14.47
0.566
0.557
0.094
0.080
0.65
0.56
1.20
4.14
26.06
26.19
0.678
0.677
0.262
0.260
25
0.33
0.49
3.00
3.53
21.07
22.77
0.637
0.646
0.203
0.214
50
0.42
0.41
3.21
3.02
21.27
20.28
0.629
0.617
0.187
0.167
100
0.39
0.36
2.90
2.68
19.33
18.30
0.606
0.595
0.148
0.131
10
1.01
1.47
7.06
8.23
36.20
37. 58
0.748
0.753
0.370
0.378
25
1.20
1.14
6.95
6.17
33.19
31.66
0.717
0.702
0.316
0.294
50
1.00
0.88
5.81
5.33
29.40
27.31
0.688
0.667
0.267
0.234
0.76
5.00
4.62
26.01
23.98
0.636
0.221
0.180
82,22
52.50
90.37
78.78
97.17
93.35
0.992
0.981
0.905
0.846
53.92
44.68
57.58
84.76
76.00
0.959
0.748
0.606
50
41.00
22.72
52.96
38.84
71.46
58.63
0.913
0.854
0.616
0.491
100
21.09
7.72
37.19
20.15
56.11
41.80
0.845
0.777
0.172
0.362
43

---

Published in Transactions on Machine Learning Research (12/2023)
A.2.8 Additional Federated Learning Results
Table 15 shows the non-private performance on the FLAIR dataset, while Table shows the same performance under DP guaranties with e
Table 15: Non-private Federated Learning performance FLAIR. function of backbone be and learnable parameters. C stands for averaged per-class metrics (Macro) and O denotes overall metrics (Micro). P, R and AP denote precision, recall, and average precision, respectively. The R-18 All result is taken from the original paper Song et al. (2022). Due to the significant computational requiremenls, only a single random seed was used in all experiments on FLAIR.
e
C-P
O-P
C-R
O-R
C-F1
O-F1
C-AP
O-AP
All
00
71.8
83.5
48.6
76.0
58.0
79.5
62.1
88.8
R-18
FiLM
0
73.8
82.0
44.8
74.4
55.7
78.0
59.7
87.7
Head
00
71.0
79.9
43.8
72.9
54.1
76.2
57.9
85.8
All
00
76.9
85.2
62.0
82
68.6
83.6
72.3
91.9
R-50
FiLM
0
78.3
83.8
57.9
80.0
66.6
81.9
70.2
90.6
Head
00
76
82.3
42.7
71.3
54.6
76.4
60.5
86.7
AIl
00
79.6
86.8
57.4
82.9
66.7
84.8
72.9
93.1
VIT-B
FiLM
00
81.9
86.8
59.3
81.6
68.8
84.1
74.7
92.7
Head
00
81.6
83.7
52
72.2
63.4
77.5
70.0
87.6
Table 16: Federated Learning performance FLAIR under DP with < 2 as a function of backbone by and learnable parameters. C stands for averaged per-class metries (Macro) and O denotes overall metrics (Micro). P, R and AP denote precision, recall, and average precision, respectively. The R-18 All result taken from the original paper Song et al. (2022). Due to significant computational requirements, only single random seed was used in all experiments with FLAIR.
bg
c
C-P
O-P
C-R
O-R
C-F1
O-F1
C-AP
O-AP
All
2
47.3
77.5
32.3
64.3
38.4
70.3
44.3
80.2
R-18
FiLM
2
59.0
81.0
39.1
70.3
47.0
75.3
51.9
85.2
Head
2
47.6
81.4
34.2
66.4
39.8
73.1
47.2
83.4
All
2
56.2
83.1
38.1
70.9
45.4
76.6
52.3
86.6
R-50
FiLM
2
59.7
79.3
39.4
69.9
47.5
74.3
51.3
84.2
Head
2
57.0
79.8
38.0
68.5
45.6
73.7
50.4
83.8
2
47.8
82.3
37.5
71.0
42.1
76.2
49.7
86.1
VIT-B
FiLM
2
58.1
84.2
42.5
76
49.1
79.9
57.2
89.2
Head
2
67.1
83.4
39.8
68.9
50.0
75.5
59.0
85.9
CIFAR-100 and Federated EMNIST Additionally, we perform experiments on CIFAR-100 and Federated EMNIST, which are commonly used to benchmark FL methods. We opt for these datasets as they have different degree TD: CIFAR-100 has medium TD, while Federated EMNIST had high TD. For CIFAR-100, use 500 training clients and 100 test clients, with each client having 100 samples and no clients sharing any data. To introduce more client heterogencity, the data distributed using the Pachinko Allocation Method (Li & McCallum 2006) as in Reddi et al. (2021). Federated EMNIST Caldas et al. (2018) is a dataset of black-and white bandwritten symbols from 62 classes grouped according to the writer. EMNIST is a highly out-of-distribution dataset (i.e. high TD) with respect to the ImageNet-221K pretraining data. As the number of training users CIFAR-100 (500 users) and Federated EMNIST (3400 users) is relatively low, we need increase e from 2 8, such that the amount of added noise during aggregation not excessive. 8 set - where N is the number of training clients. For CIFAR-100 and Federated EMNIST, we report standard test classification accuracy. All training details and hyperparameters in Appendix A.3.6
44

---

Published in Transactions on Machine Learning Research (12/2023)
Fig. 24 shows the performance of different model configurations CIFAR-100 and Federated EMNIST with and without DP. Table 17 illustrates private with 8 and non-private performance CIFAR-100 and Federated EMNIST. These tables present a tabular version of the results in Fig. 9
90.8
FILM
Head
Accuracy (%)
59.0
80.3
50.9
04.7
Accuracy
65.4 65.3
38.0
14.0
7844
57.0
75.9
63.5
65.8
19.9
72.7
62.6
73.6
68.4
All
40
41.2
20
18.3
15.6
21.3
20
R-18
R-50
VIT-B
R-18
R-50
VIT-B
CIFAR-100,E
EMNIST,E 8, 3400
Figure 24: Private (e=8 8, colored) and non-private (e === 00, gray) FL performance CIFAR-100 100 (left) and Federated EMNIST (right) as function of backbone and learnable parameters. We report accuracy on test clients. R-18 backbone is pretrained on ImageNet-lk, VIT-B and R-50 are pretrained on ImageNet-21k.
Table 17:
Federated Learning performance on CIFAR-100 and EMNIST with (c= 8) and without (c= oo) DP as a function of backbone by and learnable parameters. Accuracy (in %) is reported. R-18 backbone is pretrained on ImageNet- 1k, VIT-B and R-50 are pretrained on ImageNet-211. The + sign indicates the 95% confidence interval over runs with different seeds.
R-18
R-50
VIT-B
Dataset
Head
FiLM
All
Head
FiLM
Head
FiLM
CIFAR-100
00
63.3+0.2
69.8+0.3
72.8+0.7
59.1
79.8+0.55
83.0+0.1
84.6+0.1
90.2+0.3
90.8+00.3
27.1+1.4 1.4
18.3+0.9
15.6+1.0
20.9++06
21.3+10
23.5+1.3
50.8++0.1
40.2+-2.3
41.2+3.4
EMNIST
00
65.4+0.1
74.020.9
78.4+1.1
66.240.4
75.940.4
79.940.4
72.7+0.2
78.6+0.1
80.5+0..1
8
58.0+0.4
66.3+0.5
65.5+00.2
57.0=0.3
63.3++0.1
65.840.3
62.6+001
68.4+0.2
69.7+0.33
45

---

Published in Transactions on Machine Learning Research (12/2023)
A.3 Training and Evaluation Details
A.3.1 FiLM Layer Implementation
Table 18 details the locations and count of the parameters that are updateable for the FiLM configuration in each the backbones used the experiments.
Table 18: Backbone parameter count, FILM parameter count, FiLM parameter count as a percentage of the backbone parameter count, and FiLM parameter locations within the backbone for each of the backbones used the experiments.
Backbone Backbone Count
FiLM Count
FiLM (%)
Locations
R-18
11.2M
7808
0.07
GroupNorm Scale and Bias that follows each 3x3 Conv layer
R-50
23.5M
1644
0.05
GroupNorm Scale and Bias that fallows each 3x3 Conv
layer
Final GroupNorm Seale and Bias before Head
VIT-B
85.8M
38400
0.04
All LayerNorm Stale and Bins
A.3.2 Hyperparameter Tuning
For all centralized experiments, we first draw D of the required size ([D) = CS, [D] = 1000 in the case of VTAB-1k) from the entire training split of the current dataset under evaluation. For the purposes of hyperparameter tuning, we then split ② into 70% train and 30% validation. We then perform 20 iterations of hyperparameter tuning using the trec-structured parzen estimator (Bergstra et al. 2011 strategy as implemented in Optuna (Akiba et al. 2019) to derive a set of hyperparameters that yield the highest accuracy on the validation split. This set of parameters are subsequently used train final model on all of D. We the evaluate the final, tuned model on the entire test split of the current dataset. Details on the set of hyperparameters that are tuned and their ranges can be found in Table 19 For DP training, we compute the required noise multiplier depending the target 5)DD guarantee. The hyperparameter ranges are purposely broad and have been empirically derived. We fine-tune models for at most 200 epochs to limit the amount of compute necessary.
Table 19: Hyperparameter ranges used for the Bayesian optimization.
lower bound
upper bound
EPOCHS LEARNING RATE
200
1E-7 10
1E-2
BATCH SIZE
|D]
CLIPPING NORM
0.2
10
NOISE MULTIPLIER
BASED ON TARGET 6
A.3.3 Effect of Shots per Class and Experiments
For cach evaluated configuration, we draw [D] 00=00 CS examples from the datasct training split, tune hyper- parameters as described in Appendix A.3.2 and then test on the entire test. split of the dataset. We use the DP-Adam optimizer implemented in Opacus (Yousefpour et al.) 2021) for all private experiments. For non-private experiments, we used the Adam (Kingma & Baj 2015) optimizer for the Head and FiLM parameter configurations and the SGD optimizer for the All configuration. No data augmentation was used and images were scaled to 224x224 pixels.
All of the effect of S and experiments were carried out on (for Head and FiLM) and up to (for All) NVIDIA V100 GPUs with 32GB of memory. The runtime for executing the whole experiment depends on the the size of the few-shot training set and the number of parameters resulting from the choice of the backbone
46

---

Published in Transactions on Machine Learning Research (12/2023)
and the number of learnable parameters (All > FILM Head). For CIFAR-10 and SVHN the runtime for one configuration ranges from less than 5 GPU minutes (S = + Head) to 60 GPU hours (S = 500 + All). For CIFAR-100, the range is from 15 GPU minutes (S = 1 + Head) to over 700 GPU hours (S = 500 + All).
A.3.4 VTAB-1k Experiments
For each evaluated configuration of each of the 19 datasets in the VTAB-1k benchmark, we draw |D] = 1000 examples from the dataset training split, tune hyperparameters as described in Appendix A..222 and then test on the entire test split of the datasct. We use the DP-Adam optimizer as implemented in Opacus (Yousefpour) et al. 2021) for all private experiments. For uou-private experiments, we used the Adam (Kingma & Baj] 2015 optimizer for the Head and FILM parameter configurations and the SGD optimizer for the All configuration.
No data augmentation image size was 32x32 pixels less, in which case the images were scaled to 224 224 pixels. For the VIT-B backbone, images were scaled to x224 pixels.
used. For the R-50 backbone, images were scaled to 384x388 pixels unless the
All of the VTAB-Ik transfer learning experiments were carried out on a single NVIDIA A100 GPU with 80GB of memory. Processing times for each configuration of each dataset will vary with the selected hyperparameters and the size of the test split, but approximate times are listed Table 20
Table 20: Approximate time to tune, train, and test a single configuration of parameters on single VTAB-1k dataset for various backbones and parameter configurations. Units are wall clock GPU hours.
Parameter Configuration
Backbone
None
FiLM
All
R-50
0.6
0.9
2.7
VIT-B
1.3
2.4
6.5
A.3.5 Membership Inference Attacks Experiments
For each setting of S and e, we first sample 2|D] examples (recall [D] = CS = 1005) from the CIFAR-100 training set, and then train 257 different models (1 target model plus 256 shadow models) where each sample for the training set is randomly selected with 50% probability from the 2/D| examples. This ensures that approximately half of the models are trained on each example and half are not so that we can create distributions over the losses for each example being in and out of the training set as described in the LiRA algorithm (Carlini et al. 2022). We use each of the trained models in turn the target model and then accumulate the attack predictions over all 257 targets to produce the ROC curve for the attack. Due to the extreme computation demand in training a large number of shadow models for each setting of S and e, we restrict the attacks to the R-50 backbone and the Head and FiLM parameter configurations.
Our implementation based on code from the TensorFlow Privacy library (Google 2019b). All of the VTAB-1k transfer learning experiments were carried out on a single NVIDIA A100 GPU with 80GB of memory. When training the 257 models for each attack configuration, we do not perform hyperparameter tuning, instead we used the hyperparameter set from the CIFAR-100 experiments in Table 3 that yielded the highest accuracy for the particular configuration. Approxinate training limes for all 257 models in each configuration are listed on Table 21 The value of e did not alter the training times to significant degree.
A.3.6 Federated Learning Experiments
All experiments were performed in TensorFlow using censorflow-fedderaeed Google (2019a) for federated aggrega- tion and tensorflow-privaee Google (019b) for privacy accounting and the adaptive clipping algorithm Andrew
et al. b.21b. CIFAR-100 and Federated EMNIST datasets were taken from tensorflow-federatee..
FLAIR Each model configuration is trained for 5000 rounds with cohort size of 200. Each sampled user trains the model locally with SGD for 2 epochs with local batch size set to 16. The maximum number of
47

---

Published in Transactions on Machine Learning Research (12/2023)
Table 21: Approximate time to train 257 models for a single configuration of parameters for a LiRA attack on the CIFAR-100 datasct for various parameter and shot configurations. Units are wall clock GPU hours.
Shot (S)
Parameter Configuration
10
25
50
100
Head FiLM
12
46
25
49
96
images for each user is set to 512. For DP, 6 2, N-1.1, where N is the number of training users. As in the original paper, uniformly per round to simulate the noisc-level with cohort size of 5000.
L2 norm quantile to 0.1 for adaptive clipping and we use 200 users sampled
For the non-private setting we perform the grid search over:
server learning rale E {0.01, 0.05, 0.1}
client learning rate {0.01, 0.05, 0.1}
For the private setting (c (c=2) 2) fixed the client learning rate to the optimal value found for the non-private run and perform grid search over the server learning rate in the set (u/2,a/10 a/50, a/100}, where is the optimal server learning rate found for the non-private setting.
Processing times for each configuration on FLAIR are given in Table 22
Table 22: Approximate time to train and test a single configuration of parameters on FLAIR dataset for various backbones and parameter configurations. Units are wall clock GPU hours.
Parameter Configuration
Backbone
Head
FiLM
All
R-18
18
30
R-50
30
43
60
VIT-B
40
60
75
CIFAR-100 and Fedcrated EMNIST Each model configuration is trained for 500 rounds with cohort size of 20. Each sampled user trains the model locally with SGD for 5 epochs with local batch size set to 100. The maximum number of images for each user is set to 512. For DP, e = 8, = N-1.1, where N is the 3400 for Federated EMNIST). As in the original paper, we set L2 norm quantile 0.1 for adaptive clipping and we use 20 users sampled uniformly per round to simulate the noise-level with cohort size of 100.
number of training users (N = 500 for CIFAR-100,
For the non-private setting perform the grid search over:
server learning rate E {0.05, 0.1, 0.5}
client learning rate (0.01, 0.05, 0.1)
For the private setting (e 8) fixed the client. learning rate to the optimal value found for the non-private and perform grid search
learning rate (a/2, a/10, a/50, a a/100], where a is the optimal server learning rate found for the nou-private setting.
quantile for adaptive elipping bound E (0.1,0.5.0.8)
48

---

Published in Transactions on Machine Learning Research (12/2023)
A.3.7 On the (c, accounting
the centralized experiments we compute the (e, 6DDD guarantees using the RDP accountant (Mironov 2017) with 1/|D| where D where [D] CS (i.e. the number classes C multiplied by shot S). Setting 1/[2| is standard choice and simplifies comparisons with other papers. To allow for an easier comparison among different [D] provide Table 23 which illustrates the change of computed using the RDP accountant for == le
Additionally, we recompute the (e, 6)-DP guarantees with the PRV accountant (Gopi et al. 2021), which is a accurate numerical accountant and results in slightly smaller e than the RDP accountant given the same privacy parameters and 6. Table 24 shows the results for that.
Table 23: Recomputed e at == le-6 as function $ for the datasets CIFAR-10, CIFAR-100 and SVHN and original e E {1, 4, 8) that was computed originally at 6 = 1/ The computation is done using the RDP accountant (ironnov] 2017) provided opacus (Youselpour et al. 2021 The ranges of result from the fact that there not direct mapping from the original the recomputed but the recomputed depends the used privacy parameters (noise multiplier, subsampling ratio and number of steps).
original
15
55
105
255
505
100.
2505
500.
1
3.30-3.32
2.20-2.33
1.94-2.20
1.69-1.71
1.54-1.56
1.43-1.46
1.30-1.34
1.22-1.24
CIFAR-10
2
5.41-5.43
3.95-4.49
3.56-3.60
3.18-3.22
2.95-2.97
2.76-2.92
2.54-2.66
2.41-2.50
9.14-9.16
7.14-8.25
6.57-6.78
5.99-6.11
5.61-5.73
5.31-5.68
4.96-5.31
4.73-487
8
15.80-15.82
13.02-13.84
12.19-13.41
11.35-11.71
10.72-11.57
10.25-10.83
9.67-10.50
9.28-9.51
1
1.94-1.97
1.54-1.65
1.43-1.44
1.30-1.34
1.22-1.23
1.16-1.17
1.08-1.09
1.03-1.44
CIFAR-100
2
3.56-3.64
2.95-2.98
2.75-2.77
2.54-2.55
2.41-2442
2.29-2.30
2.16-2.17
2.07-2.08
4
6.60-6.81
5.63-5.73
5.32-5.42
4.96-5.03
4.73-4.94
4.53-1.55
4.29-4.30
4.14-1.55
12.26-122 12.74 8
10.76-10.86
10.25-10.48
9.66-9.85
9.28-9.39
8.94-9.03
8.52-8.55
8.25-8.29
1
3.30-3.32
2.19-2.33
1.94-1.96
1.69-1.71
1,54-1566
1.43-1.00
1.30-1311
1.22-1.21
SVHN
2
5.41-5.43
3.95-4.03
3.56-3.62
3.17-3.23
2.94-2.97
2.75-2.91
2.54-2.56
2.41-2.43
4
9.14-9.16
7.14-7.44
6.60-7.52
5.99-6.50
5.62-5.93
5.32-5.68
4.96-5.01
4.73-4.77
8
15.80-15.82
13.08-13.73
12.19-12.72
11.34-11.85
10.76-11.02
10.25-10.45
9.67-0.82
9.28-9.40
Table 24: Recomputed at d = 1e-5 as function of S for the datasets CIFAR-10, CIFAR-100 and SVHN and original E 2, 8) that was computed originally at 5 8= 1/DD). The computation is done using the PRV accountant (Gopi et al. 2021 provided in opacus (Yousefpour et al. 2021). The ranges of e result from the fact that there not direct mapping from the original to the recomputed but the rocomputed depends on the used privacy parameters (noise multiplier, subsampling ratio and number of steps).
original
5S
10S
25$
50S
100S
250S
500S
3.05-3.07
2.04-2.13
1.79-1.97
1.56-1.57
1.43-1.44
1.32-1.34
1.20-1.22
1.13-1.14
CI
5.02-5.01
3.66-1.01
3.30-3.33
2.94-2.96
2.73-2.74
2.55-2662
2.35-2.38
2.23-2.21
8.52-8.54
6.64-7.45
6.11-6.24
5.56-5.64
5.21-5.28
4.93-5.11
4.60-4.68
4.39-4.41
14.80-14.81
12.17-12.74
11.39-12.17
10.57-10.79
10.00-10.48
9.54-9.83
9.00-9.08
8.63-8.68
1.79-1.81
1.43-1488
1.32-1.33
1.20-1.22
1.13-1.14
1,07-1008
1.00-1.01
0.96-0.96
CIFAR-100
3.30-3.35
2.73-2.75
2.55-2.56
2.35-2336
2.23-2.21
2.12-2.13
2.00-2.00
1.92-1.92
6.12-6.27
5.22-5.28
4.93-4.98
4.60-4.62
4.39-4.46
4.20-4.20
3.98-3.99
3.83-3.84
11.43-11.74
10.02-10.08
9.54-9.65
9.00-9.06
8.63-8.66
8.31-8.32
7.92-7.93
7.61-7.68
3.05-3.07
2.03-2.13
1.79-1.81
1.56-1.58
1.431.44
1.32-1.35
1.20-1.21
1.13-1.14
SVHN
5.02-5.04
3.66-3.71
3.30-3.34
2.94-2.97
2.72-2.74
2.55-2.62
2.35-2.36
2.23-2.24
8.52-8.54
6.64-6.85
6.12-6.71
5.56-5.83
5.22-5.37
4.93-5.11
4.60-1.62
4.38-4.40
14.80-14.81
12.18-12.65
11.39-11.73
10.57-10.86
10.02-10.16
9.54-9.64
9.00-9.05
8.64-8.66
49


Reviews:
[{"review_id": "hWOzXizK7N", "reviewer": "Reviewer_wjZ8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think this paper is a great fit for TMLR. The paper does not necessarily provide any groundbreaking insights or \"surprising\" results, but the experiments are extensive and thorough, proper scientific practice is followed (error bars, ablation studies, and raw tables are all provided), and I think having a list of best practices is a valuable resource for the community.\n\nQuestions:\n\n- Is there an intuitive reason why one should expect the roughly linear trend (or i guess logarithimic, since the x axis is on a log scale) shown in Figure 2?\n\n- From what I can tell from the figures, it seems as though even though FiLM does better than Head under DP, a large part of this could be attributed to FiLM having higher accuracy to start with (i.e., in the non-private regime). Is this an accurate interpretation? E.g., in Figure 5, it seems like the \"drop\" for Head models is smaller than the drop for FiLM, but FiLM starts higher up.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper performs a large-scale study of few-shot classification under differential privacy (DP) constraints. The authors explore how changing the privacy level, training routine, and various other parameters affects the performance of the downstream model. The authors synthesize these experiments into several high-level takeaways that seem to hold robustly across different experimental setups. As a function of these findings, the authors put forward a set of recommendations for building differentially private few-shot machine learning systems.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think the paper is deserving of acceptance, but would recommend the changes below to strengthen it:\n\n- Given that TD is a quantitative metric, it would be nice to see a plot with TD on the x axis and K-shot accuracy (for some fixed value of K) on the y axis, and then have different lines for different privacy levels. This could reinforce the authors' point about the \"gap\" between private and non-private models increasing with transfer difficulty.\n\n- I suspect that the correlation between the two quantities (TD and few-shot accuracy under privacy constraints) will not be perfect, i.e., TD is not quite what dictates how hard a dataset is to learn under DP. It would be nice if the authors could discuss this a bit in their paper, and maybe provide some hypotheses about what the underlying factor of variation is.\n\n- It would be interesting to see whether the choice of efficient adapter changes the results significantly---i.e., is it just about the number of parameters, or is there something about the structure of the FiLM adapter that makes it better for DP?\n\n- The authors should increase the font size on the labels of some of the figures (e.g., Figure 3) as they are a bit hard to read.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rqqEqWRF4U", "reviewer": "Reviewer_drm9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe paper's experiments are well-detailed, comprehensive, and straightforward to replicate.\nThe paper is well-written and all terminology is clearly defined before use, making it easy to follow.\nThe subject matter is timely and important, because privacy is important and few-shot learning is especially relevant.\n\n**Weaknesses**\n\n*Major*\n\nFirst and foremost the authors need to address the issue of the choice of $\\delta$ in the experiments. This work sets $\\delta = 1/dataset size$ but when the dataset size is small, such as CS=10 (which is the case in the key few-shot setting the authors are interested in) this means $\\delta = 1/10$. That is, the DP training has a $10$% chance to 'catastrophically fail', or provide no privacy guarantees. \n\nI feel the paper really lacks novelty. Most of the analysis and findings, such as the privacy-utility tradeoff and the need for more data in DP, has been covered in prior works. What lesson do we as the readers take away from this work? I really wonder what information a reader who is, e.g., a researcher in privacy or a practitioner in industry, would take away from this paper given all the limitations (which to give credit to the authors are clearly stated and discussed) and limited new findings. \n\nWhile the paper does mention the effectiveness of parameter-efficient methods like FiLM, it does not compare these with other PEFT methods, limiting the depth of the analysis. \n\nI will go over some specific claims in the paper and why I am not satisfied with them. The following claims are not incorrect, but they are known from prior work, so it's not clear why the paper's analysis is stating these as conclusions rather than referencing them from prior work.\n\n\"Additional data is required under DP\" This point has already been made in well-known work ([1])\n\n\"Transfer learning under DP is fundamentally different from non-private\" This point has already been made in many prior works, see [2]\n\n\"The point at which accuracy begins to improve varies with TD\" This is known even non-privately, see the well-cited work done with the Stanford WILDS dataset [3]\n\n[1] https://arxiv.org/abs/2011.11660\n\n[2] https://arxiv.org/abs/2205.02973\n\n[3] https://arxiv.org/pdf/2202.10054.pdf\n\n\"We see that learning under DP is fundamentally different from non-private.\" \n\nThen there are some claims that can be interesting but are under-supported.\n\n\"As ϵ increases, the train accuracy grows as the amount of DP regularization is reduced, ultimately entering the interpolating regime\" There are two issues with this statement. First, without knowing the specific hyperparameters searched for, it's not necessarily the case that the amount of noise (what I assume the authors mean by DP regularization) is reduced, it could be that the number of iterations is increased. Second, I don't see any evidence on the plots that the models are in the interpolating regime. \n\n\"For SVHN with ϵ = ∞, Head leaves the interpolating regime for S > 100, as there is not enough capacity to adapt to a high TD dataset. \" It's not at all clear that this is an issue of capacity, because there is prior work in non-private (and even private) fine-tuning for linear probing (what the community would typically call the 'head' setting) that shows good performance on SVHN.\n\n*Minor*\n\nI didn't see an explicit definition for CS in the text. The term appears at the top of page 5. My understanding is that it's the number of classes times the number of shots.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper looks at DP few-shot learning for the context of image classification. The authors provide an empirical evaluation of this setting. They demonstrate that, while it is feasible to achieve reasonable accuracy via fine-tuning pretrained models in a DP-few-shot setting, a larger number of examples is required compared to non-private settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The authors should instead fix delta to something small such as $\\delta=1e-5$ for all experiments and rerun all the experiments with the $\\varepsilon$ corresponding to that $\\delta$.\n\nThe authors should provide more comparison to other PEFT methods.\n\nThe paper needs to clearly state its contributions because it's very unclear what is being claimed by this work and what is shared with prior work.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MHPvg9l5fg", "reviewer": "Reviewer_RPAK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n1. The paper conducted comprehensive experiments to study the effects of different factors.\n2. The authors seem to achieve the new SOTA results in DP few shot classification tasks. \n\nWeaknesses:\n\n1. I cannot quickly obtain the conclusions for each ablation study. It would be better for the authors to make a conclusion about the effect of each factor before illustrating the experimental results.\n2. Why not try to use different adaptors? Besides, It would be better to report the performance and the amount of extra tunable parameters of the adaptors of different adaptors.\n3. The metric of “transfer difficulty (TD)” could be problematic, which leads to possibly wrong conclusions. TD is related to the performance of ALL fine-tuning and HEAD fine-tuning. Fine-tuning, especially HEAD fine-tuning is sensitive to hyperparameters. Therefore, the TD can be affected by different hyperparameters.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper empirically studies the effect of different factors (including the number of shots per class, privacy level, model architecture, difficulty of downstream dataset, and learnable parameters) in the performance of DP few shot classification tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please kindly refer to the comments in the section Weaknesses and make corresponding changes.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The key contribution this paper aims to make is to offer an empirical insight into few-shot learning under differentially private constraints. A large-scale study was conducted to search for empirical insights into the choice of # of shots (per class), privacy level, downstream dataset, model architecture, and the subset of learnable parameters.

Both Reviewer wjz8 and RPAK were positive about having an empirical study of differentially private learning in various contexts; thus, I put little weight on their recommendation. Reviewer drm9 was a solicit reviewer and provided a valuable point regarding the results: the empirical study should justify their hyper-parameter choices well and the meaningful connections to existing practices.

My impression of the paper’s empirical results is not super-surprising and significant. However, according to TMLR's acceptance criteria (https://www.jmlr.org/tmlr/acceptance-criteria.html): 

> Crucially, it should not be used as a reason to reject work that isn't considered “significant” or “impactful” because it isn't achieving a new state-of-the-art on some benchmark. Nor should it form the basis for rejecting work on a method considered not “novel enough”, as novelty of the studied method is not a necessary criteria for acceptance. We explicitly avoid these terms (“significant”, “impactful”, “novel”), and focus instead on the notion of “interest”. If the authors make it clear that there is something to be learned by some researchers in their area from their work, then the criteria of interest is considered satisfied

I am leaning toward accepting this paper with minor revision.

**To authors:** Reviewer drm9 has a valid point. Before camera-ready, please tone down the claims with respect to the settings of $(\epsilon, \delta)$ in the experiments. For example, "We also show that learning parameter-efficient FiLM adapters under DP is competitive with and often superior to learning just the final classifier layer or learning all of the network parameters" in the abstract.
Decision: Accept (poster)