FiT: Parameter Efficient Few-shot Transfer Learning for Personalized and Federated Image Classification
Authors: Aliaksandra Shysheya
John F Bronskill
Massimiliano Patacchiola
Sebastian Nowozin
Richard E Turner
Venue: ICLR 2023 poster (2023)
Keywords: few-shot learning
transfer learning
federated learning

Abstract: Modern deep learning systems are increasingly deployed in situations such as personalization and federated learning where it is necessary to support i) learning on small amounts of data, and ii) communication efficient distributed training protocols. In this work, we develop FiLM Transfer (FiT) which fulfills these requirements in the image classification setting by combining ideas from transfer learning (fixed pretrained backbones and fine-tuned FiLM adapter layers) and meta-learning (automatically configured Naive Bayes classifiers and episodic training) to yield parameter efficient models with superior classification accuracy at low-shot. The resulting parameter efficiency is key for enabling few-shot learning, inexpensive model updates for personalization, and communication efficient federated learning. We experiment with FiT on a wide range of downstream datasets and show that it achieves better classification accuracy than the leading Big Transfer (BiT) algorithm at low-shot and achieves state-of-the art accuracy on the challenging VTAB-1k benchmark, with fewer than 1% of the updateable parameters. Finally, we demonstrate the parameter efficiency and superior accuracy of FiT in distributed low-shot applications including model personalization and federated learning where model update size is an important performance metric.

Published as a conference paper at ICLR 2023
FIT: PARAMETER EFFICIENT FEW-SHOT TRANSFER LEARNING FOR PERSONALIZED AND FEDERATED
IMAGE CLASSIFICATION
Aliaksandra Shysheya' University of Cambridge as29750cam, ac..eaamm............mm uk
John Bronskill University of Cambridge j fb540 cam. ac uk
Massimíliano Patacchiola University of Cambridge mp ( @cam. ac uk
Sebastian Nowozin
Richard E. Turner University of Cambridge ret 2 6êcam. ac. uk
nowozin@gmail.com com
ABSTRACT
Modern deep learning systems are increasingly deployed in situations such as personalization and federated learning where it is necessary to support i) learning on small amounts of data, and ii) communication efficient distributed training protocols. In this work, develop FiLM Transfer (FIT) which fulfills these requirements in the image classification setting by combining ideas from transfer learning (fixed pretrained backbones and fine-tuned FiLM adapter layers) and meta- learning (automatically configured Naive Bayes classifiers and episodic training) to yield parameter efficient models with superior classification accuracy low-shot. The resulting parameter efficiency is key for enabling few-shot learning. inexpen- sive model updates for personalization, and communication efficient federated learning. We experiment with FrT on wide range of downstream datasets and show that achieves better classification accuracy than the leading Big Transfer (BiT) algorithm at low-shot and achieves state-of-the art accuracy on the chal- lenging VTAB-IK benchmark, with fewer than 1% of the updateable parameters. Finally, demonstrate the parameter efficiency and superior accuracy of FiT in distributed low-shot applications including model personalization and federated learning where model update size is important performance metric.
INTRODUCTION
With the success of the commercial application of deep learning in many fields such computer vision (Schroff et al. 2015), natural language processing (Brown et al. 2020). speech recognition (Xiong al. (0188) and language translation (Wu et al., (016)), an increasing number of models being trained on central servers and then deployed on remote devices, often to personalize model to a specific user's needs. Personalization requires models that can be updated inexpensively by minimizing the number of parameters that need to be stored and or transmitted and frequently calls for few-shot learning methods as the amount of training data from an individual user may be small (Massiceti et al., 2021). At the same time, for privacy, security, and performance reasons, it can be advantageous to use federated learning where a model trained on an array of remote devices, each with different data, and share gradient parameter updates instead of training data with a central server (McMahan et al. 2017). In the federated learning setting, in order to minimize communication cost with the server, it also beneficial to have models with small number of parameters that need to be updated for each training round conducted by remote clients. The amount of training data available to the clients is often small, again necessitating few-shot learning approaches.
In order to develop data-efficient and parameter-efficient learning systems, we draw on ideas devel- oped by the few-shot learning community. Few-shot learning approaches can be characterized in
Authors contributed equally. Work performed while Microsoft Research.

---

Published as a conference paper at ICLR 2023
Classification (VTAB-1K)
Persona lization (ORBIT)
Federated Learning (CIFAR100)
10
78-
BiT
10
HT
107
10
All Accurscy 1911
Paramoters
Video Accurscy (151
Parameters
500 Glents Accurncy (8))
Commun. CosT4
Figure I: FIT is significantly more parameter efficient than BÝT. Results summary for FiT and BiT in classification, personalization, and federated learning scenarios using the BiT-M-R50x: backbone. The Parameters plots refer to the typical number of updatcable parameters in each model, while the Cost plot refers to the total client-server communication cost during federated training. In all settings, FrT achieves similar or better classification accuracy using orders of magnitude fewer updateable parameters and communication cost. Refer to Table Table 4 and Table 5 for more detail.
terms of shared and updateable parameters. From a statistical perspective, shared parameters capture similarities between datasets, while updateable parameters capture the differences. Updateable pa- rameters are those that are either recomputed or learned as the model is updated or retrained, whereas shared parameters are fixed. In personalized or federated settings. it is key to minimize the number of updateable parameters, while still retaining the capacity to adapt.
Broadly, there are two different approaches to few-shot learning: meta-learning (Hospedales et al. 2020) and transfer learning (fine-tuning) (Yosinski et al. 2014). Meta-learning approaches provide methods that have small number of updatable parameters (Requeima al. 2019). However, while meta-learners can perform strongly on datasets that are similar to those they are meta-trained on, their accuracy suffers when tested datasets that are significantly different (Dumoulin et al. (222)) Transfer learning algorithms often outperform meta-learners, especially on diverse datasets and even low-shot (Dumoulin al. [2021] Tian al.) (020)). However, the leading Big Transfer (BiT) (Dumoulin et al. [202]] Kolesnikov et al. 2019) algorithm requires every parameter in large network be updated. In summary, performant transfer learners parameter- inefficet,nnfffccint,, and parameter-efficient few-shot learners perform relatively poorly.
In this work propose FiLM Transfer FÝT, a novel method that synthesizes ideas from both the transfer learning and meta-learning communities in order to achieve the best of both worlds parameter efficiency without sacrificing accuracy, even when there are only a small number of training examples available. From transfer learning, we take advantage of backbones pretrained on large image datasets and the use of fine-tuned parameter efficient adapters. From meta-learning, we take advantage of metric learning based final layer classifiers trained with episodic protocols that we show are more effective than the conventional linear layer classifier.
We experiment with FrT wide range of downstream datasets and show that it achieves better classification accuracy at low-shot with two orders magnitude fewer updateable parameters when compared to BiT (Kolesnikov et al., 2019) and competitive accuracy when more data are available. We also demonstrate the benefits FiT on a low-shot real-world model personalization application and in demanding few-shot federated learning scenario. A summary of results is shown in Fig. where we see that FrT has superior parameter efficiency and classification accuracy compared to BYT in multiple settings. Our contributions:
A parameter and data efficient network architecture for low-shot transfer learning that (i) utilizes frozen backbones pretrained on large image datasets; (ii) augments the backbone with parameter efficient FiLM (Perez et al. 2018)) layers order to adapt to a new task; and (iii) makes novel use of an automatically configured Naive Bayes final layer classifier instead of the usual linear layer, saving a large number of updateable parameters, yet improving classification performance;
A meta-learning inspired episodic training protocol for low-shot fine-tuning requiring no data augmentation, no regularization, and minimal set of ppper-parameters;
Superior classification accuracy at low-shot on standard downstream datasets and state-of- of-the-art results on the challenging VTAB- benchmark (74.9% for backbones pretrained on ImageNet-2Ikk, while using R 1% of the updateable parameters when compared to the leading method BiT;
Demonstration of superior parameter efficiency and classification accuracy in distributed low-shot personalization and federated learning applications where model update size is key performance metric. We show that the FiT communication cost is more than orders of magnitude lower than BiT (7M versus 1 14B parameters transmitted) in CIFAR 100 federated learning experiment.
2

---

Published as a conference paper at ICLR 2023
2 FILM TRANSFER (FIT)
In this section we detail the FrT algorithm focusing on the few-shot image classification scenario.
Rehx where W is the width, H the height, ch the number of channels and image labels y E {1..... ., C} where C' is the number of image classes indexed by c. Assume that we have access to a model ((x) h (bg (bp(2)) that outputs class-pro plass-probbbilities for an image ply = c/ar, 0, Φ) for c 1, C. and is comprised of a feature extractor backbone (ax)) E IRds with parameters / that has been pretrained
Preliminaries We denote input images
large upstream dataset such Imagenet where de is the output feature dimension and final layer classifier or head h (-) E R with weights Φ. Let D = {(... yn. ) be the downstream dataset that we wish to fine-tune the model f to.
FIT Backbone For the network backbone, we freeze the parameters 0 to the values learned during upstream pretraining. To enable parameter-efficien and flexible adaptation of the backbone, we add Feature-wise Linear Modulation (FiLM) (Perez et al., 2018) layers with parameters &b at strategic points within bg. A FiLM layer scales and shifts the activations aij arising from the jth channel of a convolutional layer in the 7 block of the backbone as F FiLM((; Visj. Bsy) = asza. + Bij. where Jiy and Bi are scalars. The set of FiLM parameters & = {Tky Bby) } is learned during fine-tuning. We add FiLM layers following the middle 3 x 3 convolutional layer in each ResNetV2 (He et al. 2016b) block and also one at the end of the backbone prior to the head. Fig. A. illustrarss a FiLM layer operating on convolutional layer, and I illustrates how FiLM layer can be added to a ResNetV2 network block. FiLM layers can be similarly added to other backbones. An advantage of FiLM layers is that they enable expressive feature adaptation while adding only a small number of parameters (Perez et al.; 2018). For example, in a ResNet50 with a FiLM layer in every block, the set of FiLM parameters +/ account for only 1 16648 parameters which is fewer than 0.05% of the 4 that FiLM layers allow the model to adapt to a broad class of
parameters in bg. We show in Section[4] datasets. Fig. A.2a and Fig. A.2b show the magnitude of the FiLM parameters as function of layer for FiT-LDA on CIFAR100 and SHVN, respectively. Refer to Appendix AA. for additional detail.
FIT Head For the head of the network, we use a specially tailored Gaussian Naive Bayes classifier. Unlike a linear head, this head can be automatically configured directly from data and has only a small number of free parameters which must be learned, ideal for few-shot, personalization and federated learning. We will also show that this head is often more accurate than standard linear head. The class probability for test point x=' is:
= (bb )He. Ec))
p'y" = =c|bo, v (x") T. He E)
(I)
Σe TeN s (z*) (e"")l(e Ec))
No 1 ΣE ba. (z;), E. N Me
1 Eb (x.) (bb (zx) Me)
1
where Te
No
No
are the maximum likelihood estimates, Nr is the number of examples of class c in D, and (2|p,, E) is a multivariate Gaussian over with mean μ and covariance Σ.
Estimating the mean Hr for each class e is straightforward and incurs a total storage cost of Cd₁, However, estimating the covariance E, for each class e is challenging when the number of examples per class No is small and the embedding dimension of the backbone de is large. In addition, the storage cost for the covariance matrices may be prohibitively high if de is large. Here, we use three different approximations to the covariance in place of Eg in Eq.
Fisher 1936 Duda et al. 2012):
Quadratic Discriminant Analysis (QDA): VODA = e, Volose + Eg>inas + eeg
Linear Diseriminant Analysis (LDA): ΣIDA e2 Exask + +
ProtoNets (Snell et al. (017):: Epp = I; i.c. there is no covariance and the class representation is parameterized only by Me and the classifier logits are formed by computing the squared Euclidean distance between the feature representation of a test point bo. w (x*) and each of the class means.
In the above, Eelase is the computed covariance of the examples in class e in D, Σtack is the computed covariance of all the examples in D assuming they arise from a single Gaussian with a single mean, e = {ez, E2, ey} are weights learned during training, and the identity matrix I is used as a regularizer. QDA mainly serves as a baseline since it has very large set of updateable parameters arising from the fact that it stores a covariance matrix for each class in the dataset. LDA is far more parameter efficient than QDA, sharing a single covariance matrix across all classes. We show that LDA leads to
3

---

Published as a conference paper at ICLR 2023
Table I: Shared and updateable parameters for the transfer learning methods considered. The Example column contains the updateable parameters for all methods using a BiT-M-R50x1 backbone with
[0] 23, 500, 352, & 11, 648, d, 2048, and C = 10.
Method
Shared
Updateable
Example
BiT
0
0
+ Cd6
23,520,832
FIT QDA
2/
+
+
1
+ Cds + C (o +3 21,013,891
2
FIT LDA
0
//
+
Σ
le|
+ +((d 1) +
32,140
FIT ProtoNets
+
|
+ Cdd.
32,
7
Test Accuracy
Toe Accurary
43-
515
Sere
A1
1i
Leatnable Paramneeess Backbone
CHARR10 Training Iteration
SWIN Training Iteration
(a) LDA outperforms linear head.
(b) FIT-LDA converges more quickly than BiT.
Figure 2: (a) Average accuracy on VTAB-IK for linear and LDA heads versus learnable parameters in the backbone. (b) Test accuracy versus training iteration for CIFAR 100 and SVHN on VTAB-1k.
very similar performance to QDA. The number of model shared and updateable parameters for the three FTT variants as well as the BiT algorithm are detailed in Table Refer to Appendix A.2 for details on the parameter calculations.
An empirical justification for the use of the FIT-LDA head is shown in Fig. 2a where it outperforms linear head in the case FiLM and when all the backbone parameters are learned. Refer to Table for details. In Fig. 2b we see for both datasets, FIT-LDA converges significantly faster than BiT, which uses a linear head. The primary limitation of the Naive Bayes head is the higher (versus linear) computational cost due having to invert a d, dy covariance matrix each training iteration.
FIT Training We learn the FiLM parameters &b and the covariance weights e via fine-tunine (parameters are fixed from pretraining). One approach would be apply standard batch training on the downstream dataset, however was hard to balance under- and over- fett-fitiin using this setup. Instead, found that approach inspired by episodic training (Vinyals et al.) (016)) that is often used in meta-learning yielded better performance. We refer to this approach as episodic fine tuning and it works as follows. Note that we require 'training' data to compute æ, μ., Σ to configure the head, and 'test' set to optimize 2b and via gradient ascent. Thus, from the downstream dataset D, we derive two sets Dtruin and D Drest. If D is sufficiently large (D) 1000), such that overfitting not an issue, we set Derain. = DDest D. Otherwise, in the few-shot scenario, we randomly split 'D into Dtrain and Dtest such that the number of examples or shots each class e are roughy equal in both partitions and that there is at least one example of each class in both. Refer to Algorithm[A.. A. for details. For each training iteration, we sample a task T consisting a support set D drawn from Dtrain with S, examples and query D7, set drawn from Diest with Q, examples. First, D formed by randomly choosing a subset classes selected from the range of available classes in Dirain- Second, the number of shots of available examples in each class of Dirain with the goal of keeping the examples per class as equal as possible. Third, D is formed by using the classes selected for D and all available examples from Drest in those classes Episodic fine-tuning is crucial achieving the best classification accuracy with the Naive Bayes head. If all of Dirain and Drest are used for every iteration, overfitting occurs, limiting accuracy (see Table and Table A.4). The support set D/ is then used to compute π, μ., and Σ and we then use T
for each selected class is randomly selected from the range
to limit of 2000 examples. Refer to Algorithm A.2 for details.
Table
Do = (1x 3y 1-=1 7=1 to train ab and e with maximum likelihood. We optimize the following:
Q- log (15* (b (D,), M(D)), E(D)),(("))
e)
ΣΣ g=1
(2)
4

---

Published as a conference paper at ICLR 2023
FIT training hyper-pare parem-errmeterrs include learning rate, [Pī], and the number of training iterations. For the transfer learning experiments in Section 4 these are set to constant values across all datasets and do not need to be tuned based validation set. We do not augment the training data. In the 1-shot case, we do not perform episodic fine-tuning and leave the FiL.M parameters at their initial value of 7 1, B and
(0.5, 0.5, 1.0) and predict as described next. In Section 4 we show this can yield results that better those when augmentation and training steps are taken on I-shot data.
FIT Prediction Once the FiLM parameters vb and covariance weights have been learned, we use "D for the support set to compute wes Me, and E. for each class and then Eq. (1) can be used to make a prediction for any unseen test input.
RELATED WORK
We take inspiration from residual adapters (Rebuffi al., 2017 2018) where parameter efficient adapters are inserted into ResNet with frozen pretrained weights. The adapter parameters and the final layer linear classifier are then learned via fine-tuning. More recently, a myriad of additional parameter efficient adapters have been proposed including FiLM, Adapter (Houlsby et al., 2019).
LoRA (Hu et al., 202T). VPT (Jia et al. 2022). AdaptFormer (Chen et al.) (022). NOAH Zhang et al. 2022). Convpass (Jie & Deng. (022). (Mudrakarta et al.) 2219),, and CaSE (Patacchiola et al. 2022). For FIT we use FiLM as it is the most parameter efficient adapter. yet it allows for expressive adaptation, and can be used in various backbone architectures including ConvNets and Transformers.
To date, transfer learning systems that employ adapters use a linear head for the final classification layer. In meta-learning systems it is common to use metric learning heads (e.g. ProtoNets (Snell et al., 2017)), which have no or few learnable parameters. Meta-Learning systems that employ a metric learning head are normally trained with episodic training regime (Vinyals et al., (016). Some of these approaches (e.g. TADAM (Oreshkin et al.] 2018). FLUTEE (Triantafillou et 2021 ) and Simple CNAPs (Bateni al. 2020 use both a metric head and FiLM layers to adapt the backbone. FrT differs from all of the preceding approaches by using a powerful Naive Bayes metric head that uses episodic fine-tuning in the context of transfer learning, as opposed the usual meta-learning. We show in Fig. [2 and Section 3 that the episodically fine-tuned Naive Bayes head consistently outperforms a conventional batch trained linear head in the low-shot transfer learning setting.
EXPERIMENTS
In this section, we evaluate the classification accuracy and updatcable parameter efficiency of FIT in a series of challenging benchmarks and application scenarios. In all experiments, we use Big Transfer (BiT) (Kolesnikov
al., 2019). leading, scalable, general purpose transfer learning algorithm as a point of comparison. First, we compare different variations of FrT to BiT on several standard downstream datasets in the few-shot regime. Second, we evaluate FIT against BiT on VTAB-1K (Zhai et al., 2019), which is arguably the most challenging transfer learning benchmark. Additionally, we compare FiT to the latest vision transformer based methods that have reported the highest accuracies on VTAB-Ik to date. Third, we show how FIT can be used in personalization scenario on the ORBIT (Massiceti et al. 2021) dataset, where a smaller updatcable model is important evaluation metric. Finally, we apply FIT few-shot federated learning scenario where minimizing the number of parameter updates and their size is a key requirement. Training and evaluation details are in Appendix A.1I Source code for experiments can be found at: bctps://gitiin com/ cambr wnweeriiigeee.. mmm-mmmgmmm mlg/fi. fft
4.1 FEW-SHOT RESULTS
Fig. shows the classification accuracy as a function of updateable parameters for FIT-LDA and
BîT on four downstream datasets (CIFARIO, CIFARI00 (Krizhevsky et al., 2009). Pets (Parkhi et al.
2012), and Flowers (Nilsback & Zisserman, 2008)) that were used to evaluate the performance of BiT (Kolesnikov et al. 2019). Table A. contains complete tabular results with additional variants of
FiT and BiT. All methods use the BiT-M R50x1 (Kolesnikov et al. 2019) backbone that is pretrained on the ImageNet-21KK (Russakovsky et al. 2015) dataset. The key observations from Fig. are:
For <10 shots (except 1-shot on CIFAR100 100), FrT-LDA outperforms BiT, often by large margin.
5

---

Published as a conference paper at ICLR 2023
CIFAR10
CIFAR100
Flowers
Pets
Accuracy (%)
90
80
95
80
80
BIT
90
FIT LDA
70
60
85
60
40
80
0.1
1
10
10
10
0.1 1 10
Model Updateable Parameters (Millions)
Figure 3: FiT-LDA outperforms BiT at low-shot. Classification accuracy as a function of the number of updateable parameters (log scale) and shots per class for FIT-LDA and BiT on four downstream datasets. Classification accuracy is the vertical axis and is the average of 3 runs with different data sampling seeds. The dot size from smallest to largest indicates the number of shots per class 1, 2, 5, 10, and All. tabular version with results for additional variants Table A.1
On of datasets, FIT-LDA outperforms BiT even when all of D is used for fine-tuning. FiT-LDA outperforms BiT despite BiT having more than 100 times as many updateable parameters.
To avoid overfitting when small, Table indicatts that is better to split D into two disjoint partitions Draii and Drest and that Dī and D/2 should be randomly sub-sampled as opposed to using all of the data in each training iteration.
The datasets used in this section were similar in content the dataset used for pretraining and the performance of FrT-QDA and FiT-LDA was similar, indicating that the covariance per class was that useful for these datasets. In the next section, we test on wider variety of datasets, many of which differ greatly from the upstream data.
4.2 VTAB-IK RESULTS
The VTAB-Ik benchmark (Zhai et al. 2019) a low to medium-sho transfer learning benchmark that consists of 19 datasets grouped into three distinct categories (natural, specialized, and structured). From each dataset, 1000 examples are drawn at random from the training split to use for the downstream dataset D. After fine-tuning, the entire split is used evaluate classification performance. Table 2 shows the classification accuracy and updatcable parameter count for the three variants of FiT and BiT (see Table A.3 for eиror bars). The key observations from our results are:
Both FIT-QDA and FIT-LDA outperform BiT on VTAB-Ik.
The FIT-QDA variant has the best overall performance, showing that the class covariance is important achieve superior results datasets that differ from those used in upstream pretraining (e.g. the structured category datasets). However, the updateable parameter cost is high.
FIT-LDA utilizes two orders of magnitude fewer updateable parameters compared to BiT, making it the preferred approach.
Table A.4 shows that it best to use all of 'D for each of Dtrain and Drest (i.e. no split) and that D3 and 2 should be episodically sub-sampled as opposed to using all of the data in each iteration.
Table 2: FIT outperforms BiT on VTAB-1k. Classification accuracy and updateable parameter count (with 10 classes) for FrT variants and BiT on VTAB- 1k with BiT-M-R50xt backbone. Accuracy figures are percentages. Bold type indicates the highest scores. Green indicates summary columns.
Natural
Specialized
Structored
Paras (ND
Ovenll Ace
Chlcch011
CIFARIO0
Flowers 02
Su
SVHN
DTD
EurosA AL
Redees
Chundyyo
Metioqulhy
dSprees loe
200R8-boo
DMLah kh
KITT1-ds1
Method
BiT
23.5
68.3
$3.0
70.1
98.6
28.4
48.0
73.0
72.7
95.3
X5.9
68.3
77.2
54.6
47.9
91.6
65.9
18.7
25.8
47.1
M0.1
PD-QDA
21,0
78.6
90.3
74.1
99.1
91.0
51.1
75.1
70.9
95.6
82.6
8017
70.4
87.1
38.1
77.1
36.7
18.9
40.4
43.8
77.5
FIT Procoblets
FITEDA
0.03
0.03
60.3
90.4
39.6
74,2
98.5
99.0
90.5
31.5
51.6
74.3
68.2
70.9
93.8
95.1
82.5
77.0
82.5
79.9
57.9
66
88.7
85.6
883
96
689
748
34.2
51.
135
16.2
35.0
37.0
39,3
41.6
75.3
77.7
Table shows that FIT-LDA achieves otate-of-the-art classification accuracy when compared to leading transfer learning methods pretrained on ImageNet-2]) while requiring the smallest number of updateable parameters and using the smallest backbone. All competing methods use linear head.

---

Published as a conference paper at ICLR 2023
Table 3: FIT achieves SOTA on VTAB-1k. Classification accuracy (%) for the 3 VTAB-IK categories (Natural, Specialized, and Structured) and mean accuracy over all 19 datasets (Overall Acc) and updateable parameter count (Params) for leading transfer learning methods using various backbones
(backbone parameter count shown in parentheses) (Kolesnikov al. 2019 Dosovitskiy et al. 2020
Tan & Le 2021) pretrained on ImageNet-2Ik. ViT-Base- 16 results from JJie & Deng 2022 BiT
results from Kolesnikov et al. 2020 Green indicates results summary columns.
Method
Backhone
Params (M)
Overall Acc
Natural
Specialized
Structured
Kolesnikov al.
2019
BiT-M-R (382M)
72.7
85.8
59.4
BIT Kolesbikov el
2019
BTT.M R (928M)
928
73.5
80.8
85.7
61.1
el
(85.8M)
82.4
Aduper
[Houkhy ef
2019
ViT-Base- 1asse11611 (85.8M)
0.2
71.4
79.0
84.1
58.5
Adaap/oomee
PU22
ViT-Buse-I1 (85.8M)
72.3
80.6
84.9
58.8
ViT-Base-I (85.8M)
84.6
NOAH Zhingg el d 2022
WTT 16 (85.8M)
0.4
73.2
80.3
84.9
61.3
Convpass 01 Deng
ViT-Base-16 (85.8M)
ITT-LDA (auss)
EfficientNe (52.9M)
0.15
74.9
82.2
84.3
63.7
4.3 PERSONALIZATION
In the personalization experiments, we use ORBIT' (Massiceti et al. 2021), real-world few-shot video dataset recorded by people who are blind/low-vision, A user collects a series short videos of objects that they would like to recognize. The collected videos and associated labels are then uploaded to a central service to train a personalized classification model for that user. Once trained, the personalized model is downloaded to the user's smartphone. In this setting, models with smaller number of updateable parameters are preferred in order to save model storage space on the central The personalization performed by taking
and in transmitting any updated models large pretrained model and adapting it using user's individual data. We follow the object recognition benchmark task proposed by the authors, which tests personalized model on two different video types: clean where only a single object is present and clutter where that object appears within realistic, multi-object scene.
In Table 4 we compare FiT-LDA to several competitive transfer learning and meta- meta-learning methods that benchmark on the ORBIT dataset. We use the LDA variant FIT, achieves higher accuracy in comparison to the ProtoNets variant, while using far fewer updateable parameters than QDA. As baselines, for transfer learning, we include FineTTuner (Yosinski al.) 2014), which freezes the weights in the backbone and fine-tunes only the linear classifier head meta-learning approaches, we include ProtoNets (Snell et al. (2017) and Simple CNAPs (Bateni et al. 2020), which are meta-trained on Meta-Dataset (Dumoulin al. [222T) In the lower part of Table 4 we compare FiT-LDA and BiT. Both models use the same BiT-M-R50x! backbone pretrained on ImageNet-21K: Training and evaluation details are in Appendix (..end [A. For this comparison, show frame and video accuracy, averaged over all the videos from all tasks across all test users (17 test users, 85 tasks in total). We also report the number of shared and individual updateable parameters required to be stored or transmitted. The key observations from our results are:
individual's 's data. For
FIT-LDA outperforms competitive meta-learning methods, Simple CNAPs and ProtoNets.
FiT-LDA also outperforms Tuner in terms of the video accuracy and performs within error bars of it in terms of the frame accuracy.
The number of individual parameters for FIT-LDA is far fewer than in Simple CNAPs and BiT, and is of the same order of magnitude as Funer and ProtoNets.
FIT-LDA pretrained on ImageNet-21 performs on par with BiT, while having orders of magnitude fewer updatable parameters.
4.4 FEW-SHOT FEDERATED LEARNING
We now show how FIT can be used in the few-shot federated learning setting, where training data are split between client nodes, e.g. mobile phones or personal laptops, and each client has only a handful of samples. Model training is performed via numerous communication rounds between server and clients. In each round, the server selects a fraction of clients making updates and then sends the current model parameters to these clients. Clients update models locally using only their personal data and then send their parameter updates back to the server. Finally, the server aggregates information from all clients, updates the shared model parameters, and proceeds to the next round until
7

---

Published as a conference paper at ICLR 2023
Table 4: FiT outperforms competitive methods on ORBIT. Average accuracy (95% confidence interval) over 85 test tasks. (x) is the backbone used. EN-IK(-MD) is EfficiemtNet-B0 pretrained on ImageNet (Meta-Dataset) RN-21K is BiT-M-R50xI pretrained on ImageNet-21K, Shared the number of parameters shared among all users. Per User is the number of parameters stored for each user with C' classes. Averuge is the mean number of individual user parameters over ORBIT.
Clean Videos
Clutter Videos
Parameters
MODEL
bo(w)
FRAME ACC f VIDEO ACC
FRAME ACC
VIDEO ACC
SHARED 1
PER USER
AVERAGE J
FineTuner
EN-IK 78.1 (2.0)
85.9 (2.3)
63.1 (1.8)
66.9 (2.4)
4.01M
(d. - 1JC
0.01M
Simple CNAPS EN-MD 73.1 (2.1)
80.7 (2.6)
(1.8)
67.1 (2.4)
5.67M
7.63M
ProtoNets
EN-MD 71.6 (2.2)
(2.7)
63.0 (1.8)
67.3 (2.4)
4.01M
dC
0.01M
FYT-LDA
EN-IK 81.8 (1.8)
(1.9)
65.7 (1.8)
70.7 (2.3)
4.01M
(do + 1)C + [66] 0.03M
BIT
RN-21K (1.8)
89.7 (2.0)
67.7 (1.8)
73.9 (2.2)
D
(do + 13,C' + |0] 23.5M
FYT-LDA
RN-21K (1.7)
90.1 (2.0)
67.4 (1.8)
73.5
23.5M
(do + 1JC + [69] 0.03M
convergence, In this setting, models with smaller number of updateable parameters are preferred in order to reduce the client-server communication cost which is typically handwidth-limited The configuration and training protocol of FIT help significantly reduce communication cost compared to standard fine-tuning protocols, as i) only FiLM layers are transferred at each communication round, ii) contrast to standard linear head, the Naive Bayes head is not transferred and is constructed locally for each client. Table 5 shows communication parameters savings of FiT compared to BiT.
Experiments We use CIFARI00 (Krizhevsky et al. 2009). a relatively large-scale dataset compared to those commonly used to benchmark federated learning methods (Reddi et al.) |02 Shamsian et al. 2021). We employ the basic F FedAvą (McMa et al. 2016) algorithm, although other algorithms can also be used (Table A.7) We train all models for 60 communication rounds, with 5 clients per round and 10 update steps per client. Each client has 10 classes, which are sampled randomly before the start of training. We use ProtoNets head for simplicity, although QDA and LDA heads could also be used. More specific training and evaluation details are in Appendix [1.3
We evaluate FIT in two scenarios, global and personalized. In the global setting, the aim is to construct a global classifier and report accuracy on the CIFAR test set. We assume that the server knows which classes belong each client, and constructs a shared classifier by taking mean over prototypes produced by clients for a particular class. In the personalized scenario, we test personalized model on test classes present in the individual's 's training set and then report the mean accuracy over all clients. As opposed to the personalization experiments on ORBIT, where a personalized model is trained using only the client's local data, in this experiment we initialize a personalized model with the learned global FiL.M parameters and then construct ProtoNets classifier with an individual's data. Thus, the goal of the personalized setting is to estimate how advantageous distributed learning can be for training FiLM layers to build personalized few-shot models.
To test the global scenario, we compare FIT to BiT. As the training protocol used in BiT cannot be directly applied in the federated learning context, we trained BiT with constant learning rate. We do not provide comparison on the personalized scenario, as BiT uses a linear classification head trained globally, as opposed to using only client's local data. As, to the best of our knowledge, there are no suitable federated learning systems to compare to, we define baselines which form an upper and lower bounds on the model performance. For the global scenario, we take a FIT model trained centrally on all available data as the upper bound baseline. To get the lower bound baseline, we train FIT model for each client with their local data, then average FiLM parameters of these individual models and construct global ProtoNets classifier using the resulting FiLM parameters. The upper bound is therefore standard batch training. the performance of which we hope federated learning can approach. The lower bound is a simplistic version of federated learning with a single communication round which federated averaging should improve over. For the personalized setting, the upper bound baseline is as in the global scenario from which we form personalized classifier by taking a subset of classes belonging to a client from a global 100-way classifier. The lower bound baseline is set to a FrT model trained for each client individually. The upper bound is again standard batch training and the lower bound is derived from locally trained models which do not share information and therefore should be improved upon by federated learning.
Results 3 shows the comparison of FIT to BiT. Fig. 4 shows global and personalized classifi- cation accuracy as a function of communication cost for different numbers of clients and shots per
8

---

Published as a conference paper at ICLR 2023
Table 5: FIT is significantly more parameter efficient than BiT for federated learning. Compari- son of BiT and FIT in few-shot federated setting on CIFARI00 for different numbers clients and shots per client. Global setting is used. Accuracy figures percentages and the + sign indicates the 95% confidence interval over 3 runs. Parameter cost indicates the number of parameters sent in server-client communication. Per round is the number of parameters transmitted in each communica- tion round and Overall is the number of parameters transmitted during the whole training.
Parameter cost
50 Clients
100 Clients
500 Clients
Models
Per round
Overall
2-Shot
5-Shot
10-Shot
2-Shot
5-Shot
10-Shor
2-Shot
5-Shot
10-Shor
0.IM
68.520.3
73.840.7
77.1
74.420.3
76.5a0.2
78.420.1
77.9.0.4
77.620.1
79.220.3
BIT
237M
14B
69.0&1.4
73.660.6
78.910.6
73.241.9
78.640.3
80.220.3
69.920.3
77.320.9
78.621.3
Global Clients
Global 50 Clients
Global 100 Ciients
Global 500 Clients
70
75
60
70
70
Shots
50
Shots Upper Bound
lccuracy (96
60
65
Shots
Shots
Personalized 10 Clients
Personalized Clients
Personalized 100 Clients
Persoralizedd Clients
Shots Upper
95
95
Shots Lower Bound
90
10 Shots
05
90
90
90
10 Shots Upper Bound
85
85
85
10 Shots Lower Bound
80
80
8D
75
75
75
2.3M
4.7M
7M
2.3M
4,7M
7M
2.3M
4.7M
2.3M
4.7M
7M
Communication Cost
Figure 4: Global and personalized classification accuracy as function of communication cost over 60 rounds for different numbers of clients and shots per client on CIFAR100. Classification accuracy is on the vertical axis and is the average of 3 runs with different data sampling seeds. The color of the line indicates the number shots per class. The solid line shows the federated learning model, while dashed and dotted lines indicate the upper and lower bounds baselines, respectively.
client. By communication cost we mean the number parameters transmitted during training. The key observations from our results are:
FIT and BiT show comparable performance in terms of test accuracy, however FIT is much more parameter efficient and requires transmission of only 7M parameters through training, in contrast 14B parameters for BiT. This makes FrT highly suitable for federated learning applications.
In the global setting, the federated learning model is only slightly worse 5%) than the upper bound baseline, while outperforming the lower bound model, often by large margin. This shows that FIT can be efficiently used in federated learning settings with different configurations.
In the personalized scenario, for a sufficient number of clients (2 50) the gap between the federated learning model and the upper bound model is significantly reduced with the increase in number of shots. Federated training strongly outperforms the lower bound baseline, surprisingly even in the case of 10 clients with disjoint classes. This provides empirical evidence that collaborative distributed training can be helpful for improving personalized models the few-shot data regime.
In Appendix A.8 we show that distributed training of a FIT model can be efficiently used to learn from more extreme, non-natural image datasets like Quickdraw (Jongejan et al. 2016
5 DISCUSSION
In this work, we proposed FIT, parameter and data efficient few-shot transfer learning system that allows image classification models to be updated with only a small subset of the total model parameters. We demonstrated that FrT can outperform BiT using fewer than 1% of the updateable parameters and achieve state-of-the-ar accuracy on VTAB- lk. We also showed the parameter efficiency benefits of employing FrT in model personalization and federated learning applications.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
Aliaksandra Shysheya, John Bronskill, Massimiliano Patacchiola and Richard E. Turner are supported by an EPSRC Prosperity Partnership EP/T005386/1 between the EPSRC, Microsoft Research and the University of Cambridge. This work has been performed using resources provided by the Cambridge Tier-2 system operated by the University of Cambridge Research Computing Service httpss // 7/www. hpc cam. ac. uk funded by EPSRC Tier-2 capital grant EP/P020259/11. We thank the anonymous reviewers for key suggestions and insightful questions that significantly improved the quality of the paper. We also thank Aristeidis Panos and Siddharth Swaroop for providing helpful comments and suggestions.
Reproducibility Statement Source code for experiments can be found at: https: /ttp::://gittii
com/cambridger ml mlgfitt The README file details the data preparation steps and includes the command lines configure and run the experiments. Appendix I. details the training and evaluation procedures and hyperparameter settings for all of the experiments including few-shot and VTAB-IK Ik transfer learning experiments, personalization on ORBIT experiments, and the federated learning experiments.
REFERENCES
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated learning with personalization layers. CoRR, Abs/1912.000818 2019. URL http:
7/ //axxiv.org/abs/111/ 00818
Peyman Bateni, Raghav Goyal, Vaden Masrani, Frank Wood, and Leonid Sigal. Improved few-shot visual classification. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14493-14502, 2020.
Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Mareus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, et al. Deepmind lab. arXiv preprint arXiv:1612.03801, 2016.
John F Bronskill, Daniela Massiceti, Massimiliano Patacchiola, Katja Hofmann, Sebastian Nowozin, and Richard E Turner. Memory efficient meta-learning with large images. In Thirty-Fifth Confer- ence on Neural Information Processing Systems, 2021. URL
Forum forum?id-x2ppl x2pF Set__Tt_S____ _Su
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. arXiv preprint arXiv:2205.13: 13535, 2022.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10):1865-188) 2017.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describ- ing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4606-3613 2014.
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared rep- resentations for personalized federated learning. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. volume 139 of Pro- ceedings of Machine Learning Research, pp. 2089-2099. PMLR, 18-24 Jul 2021. URL http::s//proceeiiieo
mlmmmrr...... wwt.mmrr.........mm 39/ insllinnnall...mm ......m...mm....
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
10

---

Published as a conference paper at ICLR 2023
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Richard O Duda, Peter Hart, and David Stork. Pattern classification. John Wiley & Sons, 2012.
Vincent Dumoulin, Neil Houlsby, Utku Evci, Xiaohua Zhai, Ross Goroshin, Sylvain Gelly, and Hugo Larochelle. Comparing transfer and meta learning approaches unified few-shot classification benchmark. arXiv preprint arXiv:2 04.02668, 2021.
Li Fei-Fei, Rob Fergus, and Pictro Perona. One-shot learning of object categories. IEEE transactions on pattern analysis and machine intelligence, 28(4):594-611 511, 2006.
Ronald A Fisher. The use of multiple measurements in taxonomic problems. Annals of eugenics, 7 (2):179-188, 1936.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(1 12111311311 2013.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer. 2016b.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-221 2019.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. arXiv preprint wwX.x:2000005439, 2020.
Neil Houlsby, Andrei Giurgiu. Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799. PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 106.068866 2021.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. arXiv preprint ar arXiv:2203.1211 2022.
Shibo Jie and Zhi-Hong Deng. Convolutional bypasses are better vision transformer adapters. arXiv preprint ar orxiv:2207 Xiv:2207.07033, 2022.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017.
J. Jongejan, H. Rowley, T. Kawashima, J. Kim, and N. Fox-Gieg. The quick, draw! a.i. experiment. htttps:://qiikdiiee www.owwihhhgoogeeee htt www.ooo/aatta. htt.t//ttggggogee.comoaaaaa/a//a 2016.
Kaggle and EyePacs. Kaggle diabetic retinopathy detection. detecttooo https:
https::/www.kagleeeceoo
2015.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference Learning Representations (ICLR), 2015.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. arXiv preprint 1912. 6(2):8, 2019.
11

---

Published as a conference paper at ICLR 2023
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung. Sylvain Gelly, and Neil Houlsby. Official repository for the "Big Transfer (BiT): General Visual Representation Learning" paper.
rewearch http:////oppoooeeeeeeeeeeeeebbba httpchhhiiggtttteteeeeeeeeeeeeee 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Fu Jie Huang, and Leon Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 2, pp. 11-104. IEEE, 2004.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In I. Dhillon, D. Papail- iopoulos, and V. Sze (eds.), Proceedings of Machine Learning and Systems, volume 2. pp. 429-450, 2020. URL
wtwps/s.....ssss..o.mo.o Bttpr///ppppe22///1LLLL pdf
Paul Pu Liang, Terrance Liu, Ziyin Liu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Think lo- cally, act globally: Federated learning with local and global representations. CoRR, 0bs//2001.0123,, 2020. /RRp:
orq/ als 2001. C1523
Daniela Massiceti, Luisa Zintgraf, John Bronskill, Lida Theodorou, Matthew Tobias Harris, Edward Cutrell, Cecily Morrison, Katja Hofmann, and Simone Stumpf. ORBIT: Real-World Few- Shot Dataset for Teachable Object Recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset, 2017.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efffcienn learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 2273-1282. PMLR, 2017.
H. Brendan McMahan, Eider Moore, Daniel Ramage, and Blaise Agüera Arcas. Federated learning of deep networks using model averaging. CoRR, abs/1602.05629, 2016. URL http
www.xiv.org/bba... 05629 05629
Pramod Kaushik Mudrakarta, Mark Sandler, Andrey Zhmoginov, and Andrew Howard. K for the price of I: Parameter efficient multi-task and transfer learning. In International Conference Learning Representations, 2019, URL
net
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 201 1.
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722-729. IEEE, 2008
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. Advances in neural information processing systems, 31, 2018.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEEE, 2012.
Massimiliano Patacchiola, John Bronskill, Aliaksandra Shysheya, Katja Hofmann, Sebastian Nowozin, and Richard E Turner. Contextual suueeze-anddexcitalii for efficient few-shot image classification. arXiv preprint arXiv:2206. 099443, 2022.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual reasoning with a general conditioning layer. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI), 2018.
12

---

Published as a conference paper at ICLR 2023
Krishna Pillutla, Kshitiz Malik, Abdel-Rahman Mohamed, Mike Rabbat, Maziar Sanjabi, and Lin Xiao. Federated learning with partial model personalization, In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 17716-17758. PMLR, 17-23 Jul 2022. URL htttttttt:///oceeeeee press/v1122ppliotti2222 html
ngs mlr
Sylvestre-Alvise Rebuffi. Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. Advances in Neural Information Processing Systems, pp. 506-516, 2017.
Sylvestre-Alviss Rebuffi, Hakan Bilen, and Andrea Vedaldi. Efficient parametrization multi- domain deep neural networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 8119-8127, 2018.
Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In International Conference on Learning Representations, 2021. URL htlihtpp:: wttp://eeeerrree..ecmommm
/e/etpps/////eeeeeeeer....... http:/nee/foooo forum?
MG1-LFF33BB1 305
James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. In Proceedings of the 33rd Annual Conference on Neural Information Processing Systems (NeurIPS), pp. 7957-7968, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV). 115 1--22, 2015. doi: 10.1007/s| 1263-015-0816-y;
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2015.
Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hypemeetorks. In ICML, 2021.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Proceedings of the 31 Annual Conference on Neural Information Processing Systems (NeurIPS), pp. 4077-4087, 2017.
Mingxing Tan and Quoe Le. Efficientnetv2: Smaller models and faster training. In International Conference on Machine Learning, pp. 10096-10106. PMLR, 2021.
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B. Tenenbaum, and Phillip Isola. Rethinking few-shot image classification; good embedding is all you need? arXiv preprint arXiv:2003.1153, 2020.
Eleni Triantafillou, Hugo Larochelle, Richard Zemel, and Vincent Dumoulin. Learning a universal template for few-shot dataset generalization, In International Conference on Machine Learning, pp. 10424-10433, PMLR, 2021.
Bastiaan S Veeling, Jasper Linmans, Jim Winkens. Taco Cohen, and Max Welling, Rotation equiv- ariant cnns for digital pathology. In International Conference on Medical image computing and computer-assisted intervention, pp. 210-218. Springer, 2018.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In Proceedings of the 30th Anmual Conference on Neural Information Processing Systems (NeurIPS), pp. 3630-3638, 2016.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc Le. Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, al. Google's neural machine translation sys- tem: Bridging the gap between human and machine translation. arXiv preprint 1rXiv:1609.08// 160.008144, 2016.
13

---

Published as a conference paper at ICLR 2023
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485-3492 IEEE, 2010.
Wayne Xiong, Lingfeng Wu, Fil Alleva, Jasha Droppo, Xuedong Huang, and Andreas Stolcke. The microsoft 2017 conversational speech recognition system. In 2018 IEEE international conference on acousties, speech and signal processing (ICASSP). pp. 1934-5938. IEEE, 2018.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Proceedings of the 28th Annual Conference on Neural Information Processing Systems (NeurIPS), pp. 3320-3328, 2014.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04800 2019.
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search. 27X6::2206.04663, 2022.
arXiv preprint
14

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 FILM LAYER PLACEMENT
Fig. A. illustrates a FiLM layer operating on a convolutional layer, and Fig. illustrtst how FiLM layer can be added to a ResNetV2 network block. FiLM layers can be similarly added to EfficientNet based backbones, amongst others.
Aa
CN
MaCUU
2x3
CIV
IteLU
3ed
GN
RIM
ReLU
Ix]
block
(ng Sig
(a) A FiLM layer.
(b) A ResNet basic block with FiLM layers.
Figure A.I: (Left) A FiLM layer operating on convolutional feature maps in layer i and channel
j. (Right) How FiLM layer placed within a basic Residual network block (He et al. (0166)). GN is Group Normalization layer, ReLU is Rectified Linear Unit, and I x 1, and 3 x 3 are 2D convolutional layers with the stated kernel size.
A.2 MODEL PARAMETERS
In this section, we provide details on the updateable parameter calculations for the LDA and QDA variants of BiT. Refer Table
For FIT-QDA and FIT-LDA, the means and covariances contribute to the updateable parameter count. We use mean for every class which contributes C'd, updatcable parameters. A covariance matrix has d, x de values, however it can be represented in Cholesky factorized form which results lower triangular matrix and thus can be represented with (d, + 1)/2 values, with the rest being zeros.
In the case of FiT-LDA, where the covariance matrix is shared across all classes, a more compact representation is possible, resulting in considerable savings in updateable parameters:
(bo [Hor ELDA))
ply" = clbba (x"), π., EIDDA)
Ee Te w (x*) Hoo, ELDA)
ne (2t ETDDA) exp
(-1 (ao mo)) EIDA (bo mel)
Ee Tee det (2
- (bo,w(x')) mee) E (bo, - pre
exp
(A.1)
1 bo 1b (x )T DIDA (x, + n1. EIIAAbo Te exp (-
211 - DIDAMc)
ΣS aee exp -- (x ) TT v (x*) + - bo
1 I-1 EIDAMe)
exp WTT b (x") 1 DDbAMe)
Ee nee exp (n WEETDA bo (xx" 1/ ET (DAMe)
From Eg. A.I. it follows that to compute the probability of classifying test point x", need to store p. ELDA which has dimensionality de and n -LETTAHe which has dimensionality 1 for each class resulting only C(d, 1) parameters required for the FIT-LDA head. Since the covariance matrix is not shared in the case of FrT-QDA, no additional savings are possible in that case.
A.3 VTAB-1K DATASETS
The VTAB-IK 1k benchmark (Zhai et al. 2019 is a low to mediumm-hoo transfer learning benchmark that consists of 19 datasets grouped into three distinct categories (natural, specialized, and structured).
15

---

Published as a conference paper at ICLR 2023
The natural datasets are: Caltech101 (Fei-Fei et al. 2006). CIFAR100 (Krizhevsky et al.
2009).
Flowers 102 (Nilsback & Zisserman 2008), Pets (Parkhi et al. 2012). Sun397 (Xiao et al. SVHN (etzer et al. 2011), and DTD (Cimpoi et al. 2014).
2010).
The specialized datasets are: EuroSAT (Helber et al., 2019), Resics45 (Cheng ct al. 2017 Patch Camelyon (Veeling al. 2018), and Retinopathy (Kaggle & EyePacs 2015).
The structured datasets are: CLEVR-count (Johnson et al. 2017). CLEVR-dist JJonsson et al. 2017)
dSprites-loc (Matthey et al. 2017). dSprites-ori (Matthey al. (017)) SmallNORB Bzi (LeCun un et al.
2004). SmallNORB-ellev LeCun et al. 2004), DMLab (Beattie et al.
2016), KITTI-dist (Geiger
al. 2013
A.4 ADDITIONAL RESULTS
This section contains additional results that would not fit into the main paper, including tabular versions of figures. Note, in results, which is the ratio of the number of updateable parameters in one model to another. In our experiments, we measure RMUS relative to the number of updatable parameters in BiT model.
metric Relative Model Update Size or RMUS,
A.4.1 ADDITIONAL FEW-SHOT RESULTS
Table A.I shows the tabular version of Fig. 3 In addition. Table A..1 includes results for an additional variant of BiT (BiT-FiLM), and two additional variants of FIT (FIT-QDA and FiT-ProtoNets). BiT-FiLM is a variant of BiT that uses the same training protocol as the standard version of BiT, but the backbone weights are frozen and FiLM layers are added in the same manner FIT. The FiLM parameters &b and the linear head weights & are learned during training. The results are shown in Table A.1 and the key observations are:
In general, at low-shot, the standard version of BiT outperforms BiT-FiLM. However, as the shot increases, especially when training on all of D, BiT-FiLM is equal in classification accuracy.
The above implies that FiLM layers have sufficient capacity to accurately fine-tune to downstream datasets, but the FIT head and training protocol are needed to achieve superior results.
While the accuracy of FiT-QDA and FrT-LDA is similar, the storage requirements for a covariance matrix for each class makes QDA impractical if model update size an important consideration.
The accuracy of Pettooos is slightly lower than FrT-LDA, but often betters BiT, despite BiT having more than 100 times many updateable parameters.
Table A.1: Classification accuracy and Relative Model Update Size (RMUS) for all three variants of
FrT and two variants of BiT on standard downstream datasets as a function of shots per class. The backbone is BiT-M-R50x with 0 23, 500, 352, v/ 11, 648, and d, 2048. Accuracy figures are percentages and the + sign indicates the 95% confidence interval over 3 runs. Bold type indicates the highest scores (within the confidence interval).
BIT
FILM Transfer (FI)
Standard
FILM
QDA
LDA
ProtoNets
Dataset
Shot
Accuracy
RMUS
Accursy
RMUS
Acturacy
RMUS
Accumcy
RMUS
Accuracy
RMUS
10
$2.7=3.9
1.0
47.8x14.1
0.0014
54.0:5.7
0.893
54.:5.77
0.0014
52.9.5M1
0.0044
10
2
69.962.6
LO
63.41366
0.0014
73.0:8.8
0.893
74,2485
0,0014
68.9:9.1
0.0014
10
5
83.563.0
1,0
82.8+1.9
0.0014
85.43..9
0.893
66.444.2
0.0014
$1.865.1
0.0014
Krizhessky et 2009
10
10
88.1=2.0
LO
46.1239.7
0.0144
89.5=1.3
0.893
90.621.0
0.0144
$7.382.3
0.0014
10
95.920.2 All
L.0
96.0g20.
0.0014
96.450.0
0.893
96.320.1
0,0014
96.08011
0.0014
100
36.481.3
1.0
50.242.7
0.0091
33 13.8+0.8
$.850
33, 8:0.8
0.0091
30.7607
0.0091
100
48, 18.820.2
LO
43.0+2.9
0,0091
55.1s1.1
5,850
$4.51.0
0,0091
50.61
0.0991
C0
100
65 65.3
LO
39 69.6
0.0091
69.0:0.7
$,850
69.5
0.0091
67.95077
0.009
Rinheesiy 2009
100
10
72.360.9
LO
50 1811
0:091
75.6:0.6
5,800
75.3
0,0011
74.7
01099
100
All
16.100.1
L.0
X2.6al12
030091
32.1.0.1
5,850
X2.6
0,0011
30.3804
01009
182
79 8e0.7
1,0
79.3+2.5
030993
89.600.9
9.056
19.62019
0,0093
85.1500
0L0093
112
89 A9.41.
1.0
71.76113
B3093
95.600.5
9.056
U5. V5.621177
0.095
93,000
110093
Plowers 102
192
5
96. 800.
1O0
96.32112
03093
98.3
9.056
98. 98.4
0,0955
98,0
0L0993
Ni bubacck Ziseermon
2008
102
10
07. 3+0.7
1/0
96 960.
00093
99.060.1
9.056
96. 98.8+0.1
0.0033
98 6-011
0,993
H02
All
96.66 6.0.
1,0
9699
0,0933
99.000.1
9.056
98.
0,0093
98 A20
0.0093
S7
38, 3+2.3
1.0
36.+555
0/0037
50.2228
3.207
50.1+3.1
0.0097
46.6111
D,087
37
61 222.66
1.0
60 1s3.0
0/0037
74881.4
3,297
768811
0.0037
7288227
0L0657
76.7=2.0
76.811.5
0.0037
80060.2
3.297
82.560.4
0.0037
78 4344
0,057
Parkini
201
10
78 666.5.. -
79 79.266.4
0/0037
862+1.0
3.297
86.91.5
0.0087
88.860.4
0.067
All
90. 1+0.6
140.
0/0037
91.760.3
3,297
91.960.3
0.0037
91.2a01
0.067
16

---

Published as a conference paper at ICLR 2023
A.4.2 FEW-SHOT TASK ABLATIONS
Table A.2 shows the shw-shot results for all three variants of FiT with different ablations on how the downstream dataset D is allocated during training. No Split indicates that Dtrain is not split into two disjoint partitions and Dtrain = Dieat = D. However, Dirain and Deest are sampled to form episodic fine-tuning tasks detailed in Algorithm A.2 Split indicates that Derain is split into two disjoint partitions as detailed in Algorithm A.1 and then sampled into tasks as described in Algorithm A.2 Use All indicates that D Droain = D best D (i.e. 'D is not split) and that D Dirain and Diest are not sampled and that D = D = D for all tasks T.
Table A.2 shows that Use All is consistently the worst option. In general, in the few-shot case, Split either outperforms No Split (CIFARIO, Pets) or achieves the same level of performance (CIFAR100, Flowers 102). As result, we use the Split option when reporting the few-shot results.
Table A.2:
Classification accuracy for all three variants of FIT as a function of shots per class and how the downstream dataset D is utilized during training on standard datasets. The backbone is BiT- M-R50x with |0| 23, 500, 352, 2 11, 648, and d, 2048. Accuracy figures are percentages and the + sign indicates the 95% confidence interval over 3 runs.
QDA
LDA Split
ProtoNets Split
Dataset
Shot
No Split
Split
Use All
No Split
Use All
No Split
Use All
42.512.2
54.015.7
37.213.9
36.71:1.3
54.025.7
28.512.3
53.025.1
52.915.0
53.015.1
62.844.8
73.0&8.8
68.2455.
60.4116
20.8
65.2s4.5
68.919.4
65.2a4.5
CIPARID
79.7+0.9
85.+3.9
79.6+1.0
86.63355
86.4-4,2
09.43.5
76.1-4.2
81 11.8+511
75.6+9.3
Krizhevaky et al. 2009
10
84.2+0.1
19.5+1.3
84.0+0.3
92 12.3
90.6=1.0
87 47.9
84.5=3.8
87.3+2.3
81.9s4.5
96.640.1
96.4000
36.240.0
9.6100
96.3-0.1
96.000
96.00.11
95.460.0
33.0+3.9
33.8+0
0H0.8
32.8+2.7
3388
83
30.7:0.7
30.7+0.7
30.7+0.7
54.512.0
15.1
15.612.0
$5.61122
$1.511.0
15.113.1
$2.312.0
50.611.8
40.216.1
CIFARIDO
69.7s1.0
69.050.7
57.910.4
69 0911::00
69.521.3
57 61
68.721.5
67.910.7
$6.614.2
Krizhevaky et
75.5.0.3
75.610.6
67.647.9
75.610.2
75.320.6
67.013.2
74.8:0.2
74.7
All
82.4+0.1
82, 1+0.1
71. 17.240.1
82.6+0.2
82.6+0.2
81 1+0.1
80.6-0.2
80.3+0.4
78.2+0.1
86 1+0.5
89 39.6+0.9
89. 1+0.8
82, 1+0.8
89.6+0.9
89 690
85.1=0.9
85, 1+0.9
85.1+0.9
25.2:10.6
65.510.5
6-00.7
54.9
93.9:1.0
Flowers 102
98.4+13.
98, 3+0.3
2+0.4
98 540.2
98 4-003
98,1-0.4
9800003
96.6+0.6
Nilsback& & Zisserman 2008
10
99.010.0
99.010.1
98. 98.910.1
98.910.1
98.810.1
96, $10.0
98.610.0
98. 6.10
96.710.0
All
99. 19011
99.06001
98 8a0.0
99.060.11
98 98..
98, 4:00
9%.8
98.610.2
96. 750.0
29.4+3.1
50.2=2.8
50.2+2.8
17:266.3
103.1
50.0=3.2
46.652.0
53.0+2.1
7+.8+14
64 2+0.7
49.4+5.7
76.8-1.8
53.4+20
60.1-0.4
72. 72.8+2.7
66 10.1
Pees
81.2+22.3
$0.0=0.2
73.6+1.1
82, 1+2.2
82.5+0.4
71 11.033.1
83.0=2.4
78.4+0.7
67, 67.466.2
Parkhi 2012
87.3+1.0
86.2+1.0
W6.9-1.5
87.1=1.2
85.841)
79.2+2.0
92. 140.2
91.7+0.3
77.8+0.0
91.840.2
91.9+0.3
88.3+011
91.8-0.2
91.240.1
82.4+0.4
A.5 ADDITIONAL VTAB-IK RESULTS
Table A.3
depicts a more comprehensive version of Table 2 including error bars and RMUS for each dataset.
A.5.1 VTAB-IK TASK ABLATIONS
Table A.4 A.4 shows the VTAB- 1k results for all three variants of FIT with different ablations on how the downstream dataset D is allocated during training. Refer to Appendix (A.4.2 for the meanings of No Split, Split, and Use All. With some minor exceptions, the Use All case performs the worst. The performance of the No Split and Split options very close, with No Split being slightly better when averaged over all of the datasets. As result, we use the No Split option when reporting the VTAB-Ik results.
Table A.5 compares the classification accuracy of the LDA and linear heads as a function of the learnable parameters in the backbone. When all parameters in the backbone are learnable and when FiLM layers are employed, the Naive Bayes LDA head outperforms the linear head. In the case when all of the parameters in the backbone are frozen, the linear head is superior since in that case the LDA head has only 2 learnable parameters (the covariance weights e).
17

---

Published as a conference paper at ICLR 2023
Table A.3: FIT outperforms BiT on VTAB-IK, Classification accuracy and Relative Model Update Size (RMUS) for all three variants of FIT and BiT on the VTAB-1k benchmark. The backbone BiT-M-R50xI. Accuracy figures are percentages and the + sign indicates the 95% confidence interval over runs. Bold type indicates the highest scores (within the confidence interval).
QDA
LDA
ProtoNets
EHINetV2-M
Dutaset
Accuracy"
RMUS
Accuracy!
RMUS,
Accurayy
RMUS.
Accuracy
RMUS!
Accuracy!
Ca
102
88 080020.2
1.0
90.3s0..8
9.04
90.4:0.8
0.0093
89.610.2
0.0093
96.1=0.1
CIFAR100
$1.360.2
Flowers102
102
98.610.0
1.9
99.110.1
9,01
99.010.1
0,0093
98.61000
0,0093
99.610.0
Pers
37
88 4+0.2
1.0
91.0+0.3
3,30
90.5+0.0
0.0037
90.8:0.2
0.0037
93.0t=0.1
Sun397
397
48.0
1.0
$1.1:0.7
34,29
$1.610.5
0.0339
51.511.4
0.0339
47,9a0.5
DTD
47
72.720.3
1.0
70.9+0.1
4.18
70.9+0.1
0.0046
68.2611
0.0046
72,480.5
EumSAT
10
95. 3+0.1
1,0
95.620.1
0.89
95. 1+0.1
0.0014
93 8+0.1
0.0014
96.0=0..1
Resäco45
45
85.910.0
1.0
82 620.
4,01
52.22
0.0044
77 0.0.000
0.0044
86, 120.4
Parch Camelyon
1.9
0,0007
0,0077
Retinopathy
77.220.6
1.0
70.4+0.
0.45
66.240.5
0.0009
57 57.9+03
0.0009
72.180.6
CLEVR-coue -oconnt
54.6+7.11
1.0
87. 1=0.3
0.71
85 640.
0.0012
88.750.3
0.0012
94, 9+0.5
47
1.0
1+08
61.--20
dSprites-Ine
1.9
dSprites-ori
16
63.920.3
1.0
56,720.3
1.43
51.340.7
0.0019
34.240.8
0.0019
60.81111
SmallNCRRBapol
18
18.7
1.0
18.9:10.6
1.61
16.24
(JIIZ1
13.5+011
(.0021
16.9+1.0
SmalINORB-ele
25, S+10.9
1.0
40.4+0.2
37.0406
0.0013
35,04016
0.0013
$3.5+1.3
DMLab
41
0,0010
0,0010
KFTTI-dist
80.1+00.
1.0
77.5+0.7
0.36
77.740.8
(LOMNSS
75. 75.3+112
(L0008
82,2+0
All
68.3
70.6
69.3
65.5
74.9
Natural
77.0
78.7
74.7
82.2
Specialized
81.9
82.3
81.5
77.1
84.3
Strectured
54.0
57.5
55.0
51.6
63.7
Table A.4: Classification accuracy for all three variants of FIT on the VTAB-IK benchmark as a function of how the downstream dataset D is utilized during training. The backhone is BiT-M-R50x1. Accuracy figures are percentages and the = sign indicates the 95% confidence interval over 3 runs. Bold type indicates the highest scores.
QDA
LDA
ProtoNels Split
Dataset
No
Split
Use
No Split
Split
Use
Splir
Use
Caltech101
Rei-Fel
90.4+0.8
90.44.7
80.7-00.
CIPAR100
2000
74 1#0.1
74.8+0.3
43.400.0
74.2=0.5
74.140.4
69.040.0
73.920.
73.7=0.6
65.5+002
Flowers 102
Nilbbock &
mamm 2008
19.1.0.1
99.0s0
98.8g0.0
99.0g0.1
98 98.940.1
98.5a0.0
98.62000
98.6=0.1
96.720.0
Pets Purkhi
2012
910040.3
90.410.4
75.620.0
90.5.0.0
90.540.5
87.210.1
90.810.2
90.1.0.2
85.610.0
Sun307
$1.1
19.3.0.7
51.620.5
42.713.4
$1.511.4
50.4.0.8
42.41..2
SVHN
Nelee
A011
1s1.3
73.3+114
2=0.1
Sa0.
0
50.1+2.3
47.4=1.4
1s0.1
DXTD
Compor or 2014
70940.1
70.240.
72,8+0.0
70.960.1
70
66.540.
68.2+1.1
68.4-1.0
61.340.2
FaroSAT Helber et al 1019
95.640.1
94.740.6
03,5+0.
95.1
94 3406
3+00
93.8+0.1
92.7-0.2
89 4+0.1
Resies45
62.6+0.1
82,06033
77.5+00
$2.5=0.2
80340.2
78.340.1
77,050.0
76.4-0.8
9H0.1
Canelyon
44.440.5
78.5=2.11
Retinopathy (Kgglk
are 2015
70.410.1
67.5a1.0
25.5.0.0
66.220.5
63.400.3
25 15.040.1
57.910.3
58.420.6
17.0.0.0
CLEVR-
Johnson et 2017
$7.120.3
$4.9211.
40.6g0.1
85 6.0.9
84 $4.34
82.0a0.1
88.7
85.420.7
$7.420.1
CLEVR-dist
58.1
68/410.6
39.120.1
$6.1.0.8
55.711.7
57,710.4
$8.310.6
55.0.1.2
33.7.0.2
dSprisesloo
Mal
77.142.0
dSpritesori
Marthey
56.11
55.600.7
39.6m11.8
51.3
48.0H0.1
34,240.
32.4=1.2
760.3
SmallNORB--21
(1.2 mi
2014
18940..6
19,840.6
14.060.1
16.20.11
13.1+1.3
15,840.
13.5+0.1
13.0=0.6
13. 1+0.0
SmallNORB-elee
T22 ar
40.460..
40.5++1.4
29.2+0.1
37.06
38.5e11 161.6
36, 140.3
35,080.6
34.0=0.9
26 Se0.1
DMLab /Beatlic
43.840.3
41.2&1.3
33.750.1
S11.3
38.910.4
39.3a0.3
38.6:0.2
28 2a0.1
Goige
77.511.3
74.312.9
All
70.6
69.9
69.3
68.3
64.1
65.5
61.4
57.1
Natural
78,8
78.5
67.4
78.7
78.2
74.1
74.7
74.1
67.0
Specialized
82.3
81.4
65.6
$1.5
79.7
70.0
77.1
76.5
61.8
Stracturd
18

---

Published as a conference paper at ICLR 2023
Table A.5: Comparison of the classification accuracy of LDA and linear heads on VTAB-IK function of the learnable parameters in the backbone. "None" indicates that all of the parameters in the pre-trained backbone are frozen (i.e. no learnable parameters). "FiLM" indicates the that the only learnable parameters in the backbone are the FiLM layers. "All" indicates that all the parameters in backbone are learnable and that FiLM is not used. The backbone is BiT-M-R50x. Accuracy figures are percentages and the +: sign indicates the 95% confidence interval over 3 runs. This is the full version of Fig.
LDA
Linear
Dataset
None
FiLM
All
None
FiLM
All
Caltech101 (Fei-Fei et al.
2006
0+0.0
90.4+0.8
90,420.5
87.740.2
89.3+0.2
CIFAR100 Knzhevsky
2009
84.00
74.2+0.5
63.3+0.9
61.420.1
70.2+0.1
1
Flowers102 Nilsback
2008
98.7=0.0
99.0+0.1
98.920.2
98.6+0.0
98.7=0.1
98.6=0.0
Pets (Parkhi al. 2012
85.7+0.1
90.5+0.0
90.7+0.4
84.7+0.2
87,420.0
88.4+0.2
Sun397
Xrao
2010
47.7=0.0
51.6+0.5
38.6+1.4
46,8 46.8
46.820.2
48.0=0.1
SVHN Netzer al.
201
40.9+0.1
74.2+0.9
85.1
42.8+1.2
54.9
73.0=0.2
DTD Cimpor al.
2014
72.0+0.1
70.9=0.1
71.7=1.1
72.0+0.2
72.3+0.1
72.7=0.3
EuroSAT (Helber al.
2019
93.7+0.1
95.1
95.9+0.2
94.4+0.1
95.1+0.1
95.3+0.1
Resics45 (Cheng 2017
75.820.0
82.5+0.2
84.820.2
78.4+0.0
81.520.1
85.920.0
Patch Camelyon Veeling al.
2018
81.7+0.1
82.5+0.7
84.0#0.5
1.00
81.3+0.6
69.3+0.8
Retinopathy (Kaggic & HycPacc
2015
49.5+0.2
66.2+0.5
76.8+0.1
75.0=0.4
75.2=0.6
77.2=0.6
CLEVR-count JJohnson
2017
38.1
85.640.9
94.2+0.
48.3+0.8
18.18
$4.6+7.1
CLEVR-dist (Jonsson
33.620.1
56.1
$7.2+2.7
34.8+1.5
47.7+1.2
47.9=0.8
dSprites-loc Matthey 2017
21.6+1.1
74.8+1.4
91.6&119
12.7+1.4
49.7+0.6
91.6+1.1
dSprites-ori Matthey
2017
43.2+1.9
51.3+0.7
62.6#0.7
31.7#0.8
52.7=0.7
65.9=0.3
SmalINORB-uu [LeCun
2004
11.9=0.0
16.2+0.1
19.5#0.2
11.5+0.9
15.0=0.2
18.7=0.2
SmalINORB-eley (LeC
2004
140.0
37.0+0.6
44.2+0.6
29.3+
34.3+0.7
25.8
DMLab (Beattie at
2016
34.3+0.0
41.6+0.6
46.940.8
37.3+1..
41.8+0.7
1
KITTI-dist (Geiger ct
2013
66.0+0.0
77.7+0.8
82.9+0.2
69.2+0.7
78.1+1.5
10.1
All
56.5
69.3
72.6
57.5
65.3
68.3
Natural
70.8
78.7
77.0
70.6
74.2
77.0
Specialized
75,2
81.5
85.3
81.2
83.3
81.9
Structured
34.6
55.0
62.4
34.4
48.4
54.0
19

---

Published as a conference paper at ICLR 2023
A.6 DISCUSSION OF THE EMPIRICAL RESULTS
FiLM works well for low shots per class because it has a small number of parameters to adapt - this gets you long way with large pre-trained backbones and it also prevents over-fitting compared to adapting all the parameters in the backbone. Evidence: eventually as dataset size increases, full least as good as FiLM. For larger models, this transition happens larger number of data points and so we expect adapters to be especially useful as the field transitions to using large foundation models. Fig. provides an empirical justification of this argument, showing that FiT-LDA outperforms BiT at low-shot.
body adaptation
The linear head more flexible and works best methods in high data, but in the low- to medium-shot setting the linear head appears to lead to increased overfitting and the meta-trained LDA head performs best in this region. See Fig. for empirical justification.
The FIT approach suffers when there are fairly large number of data points and the dataset is very far from the pretraining data (ImageNet) e.g. see the DSprites results in Table where BiT beats FiT-LDA by large margin. In addition, the LDA and QDA variants of the Naive Bayes head are more computationally expensive compared to a linear head since LDA and QDA require inverting d, d, matrices for each training iteration.
A.7 FILM LAYER PARAMETER VARIATION
Fig. A.2a and Fig. A.2b show the magnitude of the FiLM parameters as function of layer for on CIFART00 and SHVN, respectively, in the VTAB- Ik setting. We see that for a dataset that differs from the pretraining data (SHVN), the FiLM layers required learn greater degree of adjustment compared to dataset that is similar to the pretraining data (CIFARI00) http://eppriniaaccoor////77//1
A.8 ADDITIONAL. FEW-SHOT FEDERATED LEARNING RESULTS
Table A.6: Few-shot Federated Learning Results CIFARI00 for different numbers of clients and shots per elient. Accuracy figures are percentages and the = sign indicates the 95% confidence interval over 3 runs. Global stands for the global setting, while Personalized stands for the personalized scenario. FL indicates Federated Learning training.
Global
Personalized
Clients
Shot
Lower Bound
FL
Upper Bound
Lower Bound
FL
Upper Bound
2
42.6#1.9
49.3+1.3
$2.5=0.9
73.2+1.4
80.1
81.7+1.2
10
5
55.6#1.6
64.1
68.7=0.7
82.6#0.4
88.8+0.2
90.6#0.4
10
60.7+1.2
71.540.8
74.7=0.3
86.640.2
91.8+0.5
92.620.6
2
59.9=0.6
68.5+0.3
73.2+602
75.4+1).9
84.6#0.5
93.4+0.2
50
5
63.0=0.8
73.8+0.7
78.0=0.1
83.5+0.5
91 1..00.4
94.9=0.2
10
65.8#1.1
77.1+0.3
79.2+0.3
87.420.6
93.4+0.3
95.2+0.1
2
65.620.6
74,4+0.3
77.5+0.2
75.7+0.9
85.3+0.2
94, 640.2
100
5
66.010.3
76.5+0.2
79.5+0.2
83.1
91.340.2
95.3+0.1
10
66.8+0.3
78.4+0.1
80.2+0.1
87.4+0.5
93.6+0.2
95.5+0.1
2
68.8+0.1
77.9+0.4
80.1
75.5+0.2
85.8+0.2
95.6+0.1
500
5
67.6+0.1
77.6+0.1
80.6+0.2
83.2+0.3
91 .1.00.2
95.7=0.1
10
67.7+0.1
79.240.3
80.8+0.2
87.7+0.1
94.310.1
96.2+0.1
Table shows the tabular version of Fig. 4 For the Federated Learning results, it includes only the resulting accuracy after training for 60 communication rounds. Refer to Section 4.4 for analysis.
In general, FIT can operate well with other FL aggregation methods, as they are agnostic to the neural network architectures used to learn the parameters. However, head instead of training a linear head via SGD, some of the theoretical convergence results of the methods mentioned may not hold. Despite this fact, FIT has shown to have quite fast convergence in practice. Table A.7 we provide the results of training FiT with Federated Averaging (FedAvg) and FedProx(Li et al. 2020
are constructing Naïve Bayes
20

---

Published as a conference paper at ICLR 2023
0.75
0.50
3.0
8
-0.25
0.5
0.50
0.0
-0.75
10
(a) CIFAR 100 film layer plot. Leff 7 and right
1.90
00
brta
0.00
-0.25
-0.75
9 layar
10 11
layar
(b) SVHN film layer plot. Left is and right is
Figure A.2: (Top) Box plots the film parameter magnitudes as a function of layer in the feature extractor.
To test distributed training of a FiT model more extreme, non-natural image dataset, also include the results for federated training of FrT on the Quickdraw dataset. As there is no pre-defined train/test split for the Quickdraw dataset, classes and use them for testing. We train all federated training models for 120 communication rounds, with 5 clients per round, and 10 update steps per client. Since Quickdraw is than CIFAR 100, it requires more communication rounds for training. Each client has 35 classes, which are sampled randomly at the start of training. In our experiments, we omit the 10-clients case, as the overall amount of data in the system is not enough to even train a robust global upper bound baseline model.
randomly choose 100 samples from each of the 345
difficult dataset
Fig. A.3 shows global and personalized classification accuracy as a function of communication cost for different numbers clients and shots per client for Quickdraw. while Table A.9 shows the tabular version of this figure.
Similarly to CIFAR 100 experiments, we compare FrT to BiT on the Quickdraw dataset. We train both models for 60 communication rounds. The results are presented in Table A.8 For 5 and 10 shots per class, FIT performs similarly to BiT for all numbers of clients tested. However, for 2-shot, BiT outperforms FIT for all numbers of clients tested, but the performance gap is reduced as the number of clients is increased. We attribute this behavior to the use of the ProtoNets classifier. During training each client's 's optimization step, construct N-way local ProtoNets classifier using only one image per class, which may result a classifier not robust enough for reasonable optimization of the FiLM layers. This hypothesis is also supported by Fig. A.3 where
huge gap between
21

---

Published as a conference paper at ICLR 2023
Table A.7: FrT trained with different FL algorithms on CIFAR100 for different numbers of clients and shots per client. Federated averaging (FedAvg) and FedProx methods were used. For FedProx μ = 0.01 is used. Accuracy figures are percentages and the + sign indicates 95% confidence interval over 3 runs. Global stands for the global setting, while Personalized stands for the personalized scenario.
Global
Personalized
Clients
Shot
FedProx
FedAvg
FedProx
FedAvg
50.1+2.2
49.3+1.3
80.5#1.1
80.1+0.8
10
6..3#1.6
14.1
88.9+0.3
88.8+0.2
10
71.5+0.8
71.5+008
91.9+0.3
91.8+0.5
68.8+0.5
38.5+0.3
84.8+0.6
84.6+0.5
50
74.0=0.5
73.8+0.7
91.2+0.4
91.0=0.4
10
77.120.1
77.140.3
93.520.2
93.4+0.3
74.5#0.2
74,4+0.3
85.3+0.2
85.3+0.2
100
76,9+0.3
76,5+0.2
91,4+0.3
91.3+0.2
10
78.420.2
78.4+0.1
93.7 0.2
93.6+0.2
78.1
77.940.4
86.1+0.3
85.8+0.2
500
77.9+0.1
77.6+0.1
91.1+0.2
91.0+0.2
10
79.3+0.3
79.2+0.3
94.3+0.1
94.3+0.1
Table A.8: Comparison of BiT and FIT in few-shot federated setting on Quickdraw for different numbers of clients and shots per client. Global setting is used. Accuracy figures are percentages and the sign indicates the 95% confidence interval over runs. Both models were trained for 60 communication rounds. Parameter cost indicates the number of parameters sent in server-client communication. Per round the number parameters transmitted each communication round and Overall is the number of parameters transmitted during the whole training.
Parameter cost
50 Clients
100 Clients
500 Clients
Models
round
Overall
2-Shot
10-Sho
2-Shot
S-Shot
10-Shor
2-Shot
10-Shot
26.6+1.0
40.8606
45.5402
32.022.
44.0H0.7
46.640.2
39.41.9
45.680.1
47.900.
BiT
241M
14,58
37.5+2.2
42.4&177
44.40077
40.420.5
44,2
46.420.9
41.1
45.7a2.6
47.5&1.7
22

---

Published as a conference paper at ICLR 2023
the performance of Federated Learning and centralized learning (the upper bound) for 2-shot. In contrast, the linear head in BiT is not local and is trained using all clients' data, thus avoiding this pitfall. This leads to the following observation if there is enough local client data to construct robust metric- baaric classifier, then Naïve Bayes head helps to significantly reduce communication cost without sacrificing the final model quality. However, if there enough local client data, then the use of linear classification head may be more appropriate.
Shots Upper Bound
5 Shots
10 Shots
Upper Bound
Upper
Shots Lower Bound
Shots Lower Bound
Shots Lower Bound
Global 50 Clients
Global 100 Cients
Global 500 Clients
40
40
40
30
35
Accuracy (96)
Personalized Clients
Personalized Clients
Personalized
70
70
60
60
50
50
2.3M
4.7M
7M
9.3M 11.6M
14M
2.3M 4.7M 7M 9.3M 11.6M
14M
2:3M 4.7M
7M 9.3M 11.6M 14M
Communication Cost
Figure A.3: Global and personalized classification accuracy function of communication cost over 120 rounds for different numbers clients and shots per client on Quickdraw. Classification accuracy is the vertical axis and the average of 3 runs with different data sampling seeds. The color of the line indicates the number of shots per class. The solid line shows the federated learning model, while dashed and dotted lines indicate the upper and lower bounds baselines, respectively.
Table A.9: Few-shot Federated Learning Results on Quickdraw for different numbers of clients and shots per client. Accuracy figures are percentages and the + sign indicates the 95% confidence interval over 3 runs. Global stands for the global setting, while Personalized stands for the personalized scenario. FL indicates Federated Learning training.
Global
Personalized
Clients
Shot
Lower Bound
FL
Upper Bound
Lower Bound
FL.
Upper Bound
2
26.3+0.4 4
29.0+0.5
32.920.6
36.920.2
43 33.4
64.2+0.8
50
5
30.6+0.3
40.7+0.6
43.9=0.5
48.8++0.4
63.6+0.3
74.3+0.4
10
34.3+0.7
44.9+0.1
47.2+0.2
56.5+0.8
71.1+0.1
76.6+0.2
2
28.5+0.1
32.4+0.8
43.0=1.0
35.9+0.3
43.9+1.0
73.140.4
100
5
32.810.4
44..4
47.420.1
48.620.9
64.920.8
76.420.3
10
3
16.8+0.1
48.7+0.2
55.520.4
71.540.5
77.240.3
2
32.0+0.3
40.3+2.4
49.1
36.2+0.1
49.2+2244
77.610.3
500
5
34.020.1
46.1+0.1
48.6#2.2
48.7+0.2
65.8+0.3
77.0=1.7
10
36.1
48.1
48.2+1.2
55.7+0.1
72.2+0.2
76.7+0.9
A.9
CONNECTIONS TO PERSONALIZED FEDERATED LEARNING
Partial model based personalized federated learning (PFL) (Collins et al. 2021 Arivazhagan et al. Liang al. 2020 Pillutla et al. 2022 is related to the personalized setting in our federated learning experiments, but is different in important regards. With regard to the similarities. the local Naive Bayes head parameters in approach may be considered "personal" parameters (there personalized parameters for each client), while the FiLM parameters could be viewed as "shared" parameters. However, there few major differences between the ideas:
2019
23

---

Published as a conference paper at ICLR 2023
The personalized parameters in setting, i.e. the ProtoNets head, does not require optimization loop to be learned. This simplifies deployment significantly.
The partial model personalization literature (Collins et al. 2021 Arivazhagan et al. 2019) Liang et al. 2020, Pillutla al., 2022) is concerned with proposing stable federated training algorithms that would work within heterogeneous settings, where clients have diverse data and standard FedAvg algorithm would fail, thus necessitating need to introduce 'personal' parameters. In contrast, personalized heads are required in our setup as each user has a different classification task perform. Moreover, we propose particular architecture that would be highly suitable for federated learning applications, as the number of parameters required to be transmitted is small, while most methods PFL. propose architecture agnostic optimization algorithms. Our model can then be trained with an arbitrary Federated Learning algorithm.
Another distinctive difference between the model personalization federated learning literature and our work that the former most of the methods train model's parameters, while in our work we used a deep pretrained network and fine-tune tune only the FiLM layers.
A.10 FIT TRAINING ALGORITHMS
Algorithm A.I and Algorithm A. detail how episodic fine-tuning tasks are split and sampled, respectively, for use in the FIT training protocol.
Algorithm A.1 Splitting the downstream dataset D
Require: = (( Bn) Bn) = (a, y}: downstream dataset
Require: unique () function that returns a list of unique classes and list of counts of each class
Require: select_by.clat asss) = function that extracts samples a specified class from a dataset
1: procedure SPLIT(D)
2: Dirain [-]
P- Create an empty list to hold D Deroin Create an empty list to hold Deest
3: Diest []
4: classes, class.counts + unique(y)
5: for all c classes do
6: assert(class counts(c) > 1)
Require minimum of 2 shots per class.
7:
train_count (ca
8: Do +- select.by.class(d
Select examples of class from D
9:
Dtrain f- Derain + Del: crun]
> Add train.count examples to D Dtroin
10: Dtest t- Drest Deftrain.coontt
Add remaining examples to Drest
11: end for
return Dirains Drest
13:
end procedure
A.11 TRAINING AND EVALUATION DETAILS
In this section, we provide implementation details for all of the experiments in Section 4
A.11.1 FEW-SHOT AND VTAB-IK TRANSFER LEARNING EXPERIMENTS
FIT All of the FIT few-shot and VTAB-1K 1k transfer learning experiments were carried out on single NVIDIA A100 GPU with 80GB of memory. The Adam optimizer (Kingma & Ba, 2015) with a constant learning rate 0.0035, for 400 iterations, and [D3|=100 was used throughout. No data augmentation was used and images were scaled to 384 x x84 pixels unless the image size was x2x32 pixels or less, in which case the images were scaled to 224 224 pixels. In each iteration of episodic fine tuning, perform single step of gradient ascent D These hyper-paramedt were empirically derived from a small number of runs.
FrT-QDA, FrT-LDA, and FrT-ProtoNets take approximately 12, 10, and 9 hours, respectively, to fine-tune on all 19 VTAB datasets and 5, 3, and hours, respectively, to fine tune all shots on the low-shot datasets.
For the FIT-LDA results using the EfficientNet22-M backbone Table 3 we used 1000 iterations instead of 400 and ran the experiments 4 NVIDIA 1100 GPUs, each with 80GB of memory.
24

---

Published as a conference paper at ICLR 2023
Algorithm A.2 Sampling task
Require: Drrain = {1 Us)) )
s=1 (æss ys}: train portion of downstream dataset
Require: Drest (xx M/ 1 (#q, 3 test portion of downstream dataset
Require: support.set.size size of the support set [D3]
Require: unique () = function that returns a list of unique classes and list of counts of cach class
Require: randint (min, max) function that returns a random integer between min and max choice(range, count) = function that returns random list of count. integers from
Require:
range
1: procedure SAMPLE, Drest support.set.size)
2: D +
>> Create an empty hold D countss
3: -
Create an empty list to hold D
4: test.classes, test_class.counts < uniqque(oo)
train class counts unique(ys)
5:
6: min.way + )
7:
+ www.aagniiileeeeaaiccceeeesssspr way + waw.aaainntchhh........a.a.aaa selected.clasees+ + http://ccseeeeooooooecaaaaaaaaaa
8:
Classification way to for this task
9:
b List of classes to use in
this task
10: balances shots
max(round(support set size
lenkselectedcclaseee)). 1)
11:
12:
for d0
13:
14:
sbcolsccusssaaiclcaaasoccbaooaaa suuse) Support shot list
15: selecrcddcbccsrecchhoeecclaaaaa shots_to
16:
+ D
& Add examples to D3
cl llassshotss-tttst..llascccooooo
18:
shots shots.te .uuee) Query shot list
20:
D/= - D/ + D D
Add examples to D
21:
end for
22: return D3, D
23: end procedure
25

---

Published as a conference paper at ICLR 2023
BiT For the BiT few-shot experiments, used the code supplied by the authors (Kolesnikov et al., 2020) with minor augmentations to read additional datasets. The BiT few-shot experiments were run on single NVIDIA V100 GPU with 16GB.
For the BiT VTAB- Ik experiments, we used the three tue-tuned models for each of the datasets that were provided by the authors (Kolesnikov et al. 2020). We evaluated all of the models on the respective test splits for each dataset and averaged the results of the three models. The BiT-HyperRule (Kolennkovv al. 2019) was respected in all runs. These experiments were executed on a single NVIDIA Geforce RTX 3090 with 24GB of memory.
A.11.2 PERSONALIZATION ON ORBIT EXPERIMENTS
The personalization experiments were carried out on a single NVIDIA GeForce RTX 3090 with 24GB of memory. It takes approximately 5 hours to train FIT-LDA personalization models for all the ORBIT (Massiceti et al. [021]) test tasks. We derived all hyperparameters empirically from a small number of runs. We used the ORBIT codebase in our experiments, only adding the code for splitting test user tasks and slightly modifying the main training loop to make it suitable for FrT training.
In the comparison of FrT to standard benchmarks on ORBIT (upper part of Table 4 all methods use an EfficientNet-BO (dg 1280) the feature extractor and image size of 224 x 224. FIT-LDA, FineTuner (Yosinski et al. 2014) and Simple CNAPs (Bateni et al. 2020) use a backbone pretrained on ImageNet (Deng al. 2009), while ProtoNets (Snell et al.) (217)) meta-trained the weights of the feature extractor on Meta-Dataset (Dumoulin et al. 2021). ). The task encoder in Simple CNAPs (Bateni
et al.
2020) is meta- eeeta-trained on Meta-Dataset. The results for all models (FineTunnr (Yosinski et al.
2014).
Simple CNAPs (Bateni et al.
(020) and ProtoNets (Snell et al. (017)) 7)) are from (Bronskill
al.
2021 FiLM layers in FIT-LDA are added the feature extractor as described in Section 2
resulting in
|bb = 20544.
For the comparison of FIT and BiT (lower part of Table 4), we use the BiT-M-R50x1 backbone pretrained on ImageNet-2IK. BiT was trained using the HyT-rrueeule (Kolesnikov et al., 2019) on
every user task. For experiments with BiT, we used the code supplied by the authors (Kolesnikoy et al. 2019) with minor augmentations to read the ORBIT dataset.
We follow the task sampling protocols described in (Massiceti et al.) (022), and train the FiT model for 50 optimization steps using the Adam optimizer with a learning rate of 0.007 for EfficientNet-BOO and 0.003 for BiT-M-R50x1. The ORBIT test tasks have slightly different structure comparison to standard few-shot classification tasks, so in Algorithm A.3 provide modified version of the data splitting for the classifier head construction. In particular, each test user has a number of objects (classes) they want to recognize, with several videos recorded per object. Each video is split into clips, consecutive 8-frame parts of the video. A user test task is comprised of these clips, randomly sampled from different videos of the user's objects, and associated labels. Since clips sampled from the same video can be semantically similar, we split the test task so that clips from the same video can only be in either the support query set, except for the cases when there only one video of an object available.
A.11.3 FEDERATED LEARNING EXPERIMENTS
For each local update new Adam optimizer is initialized. In each communication round, 5 clients are randomly chosen for making model updates. All of the federated learning experiments were carried out on a single NVIDIA A 100 GPU with 80GB of memory. In all experiments we use FTT with the BiT-M-R50x (Kolesnikov et al. 2019) backbone pretrained on the ImageNet-211 K (Russakovsky al. 2015) dataset and ProtoNets head. We derive all hyperparameters empirically from small number of runs.
BiT As the training protocol proposed in BiT cannot be directly applied to the federated learning setting, we simplify it and train the BiT model for 60 communication rounds with a constant learning rate of 0.003.
26

---

Published as a conference paper at ICLR 2023
Algorithm A.3 Splitting test task D for ORBIT personalization experiments
Require: D: downstream dataset; D {Da) where C' the number classes in test task, Da is data of class c; D. = {V
Ne i=1 where is the number of videos in class Vai is the set
of clips from ith video of class c; Vii ((F)a)))))]== where nai is the number of clips in ith video of class Keij is the jth clip from video Vai
Require: batch_size ze: size of context split
Require: choose (n, m) function that randomly samples different integers from set {i)
Require: select by index(D, i) function that extracts samples of indices from a dataset D
Require: diff(a,, b) function that computes set difference between sets and b
Require: range(n) = function that returns a set of values {i)
procedure SPLIT (ABKT))
2: Dirain <-
3: Diest +
4: num.clips +- floor (batch.size/C)
5:
for to C' do
6:
num context videos + ceil (nc/2)
7:
context..videos.ind ces t choose(n,, num.context..videss))
8:
num.clips..er..vdee floor (
9:
if n. = then
10:
11:
- hatp://aeeeaaalloottttccoooooaao
12:
Drrain train + bo context coipe indiccenndiess
13:
Drest +- Diest select by target wtw.aetttcegggennneeeeele...eeee
14:
else
15:
for j + conesst.videss.nddicosdd
context clips indices chooselm,, wwwppwwpp..cc..cc.c..ppt....p..t wtr.ipppper.iieee
17:
Dirain + Drrain + coleetttby...eeeyyc...oooot..... end for for j ht
19:
do
20:
Drest f- Drest + Vaj
21:
end for
22:
end
23:
end for
24: return Derain, Dtest
25:
end procedure
27

---

Published as a conference paper at ICLR 2023
CIFAR100 We train all federated learning models with different number of clients and shots per client for 60 communication rounds, with clients per round and 10 update steps per client. Each client has 10 classes, which are sampled uniformly before the start of training. In our experiments, used 2, 5 or 10 shot per class. For the global setting, we used the CIFAR 100 test split for the global model evaluation. For the personalized setting,
each client using the CIFAR C1FAR100 test split, but using only the classes that a particular client owns. This results in 10-class classification task. We then report the average accuracy across all clients. We use learning rate of 0.003 at the start of the training, decaying it by 0.3 every 20 communication rounds. Upper and lower bound baselines for both the global and personalized scenarios were trained for 400 epochs using the Adam optimizer with constant learning rate of 0.003. It takes around 20 minutes to train federated learning models, with slightly more training time required the models with a larger number of shots.
Quickdraw Each client has 35 classes, which are sampled uniformly at the start of training. As there is no pre-defined train/test split for the Quickdraw dataset, we randomly choose 100 samples from each of the 345 classes and use them for testing. This test data is then used to test both the personalized and global settings as described in the paragraph above. We train all federated training models for 120 communication rounds, with 5 clients per round, and 10 update steps per client. We use constant learning rate of 0.006 for training all federated learning models. Upper bound baseline models, which require training a global model using all available data, were trained for 3000 steps using the Adam optimizer with a constant learning rate of 0.006. Lower baseline models, requiring training personalized model for each individual, were trained for 400 steps using the Adam optimizer with learning rate of 0.006. It takes around 2 hours train federated learning models, with slightly more training time required for the models with a larger number of shots.
28


Reviews:
[{"review_id": "heB9LkCd8c", "reviewer": "Reviewer_BNuH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes FiLM Transfer framework (FIT) for image classification tasks via incorporating the ideas from transfer learning and meta learning. The proposed method aims to address the learning with small data and the communication efficiency in distributed learning, in the context of the personalization and federated learning. The FIT method basically consists of a pretrained backbone with an automatically configured Naive Bayes final layer classifier. The outperformances are demonstrated through the experiments in few shot, personalization and federated learning tasks.\n\n", "summary_of_the_review": "The merits and concerns are presented in the  main review. The main concerns are about the novelty of the network design, as well as the experimental design in federated settings. ", "strength_and_weaknesses": "Strengths:\n1. The motivation is well-elaborated with regard to the limited data and communication efficiency. \n2. The writing is easy to follow. \n\nWeakness:\n1. The design of the network architectures are basically based on the ideas from the transfer learning (frozen backbone) and meta learning (Naive Bayes final layer). Is the any difference of the design in FIT when compared with the original ideas in transfer learning and meta learning? or just simply using the two ideas without modification? Please clarify it.\n2. There are two personalization concepts in this work. One is in the evaluation on the ORBIT dataset. Another one is in the federated learning. What is the difference and the relationships between these two? \n4. In the experiments of few shot federated learning, the defined upper bound and lower bound for reference are useful for performance analysis during the evaluation of different settings. However, the details of model evaluation/test are not provided, for the global and personal models, for example, the setup and partitions of the datasets used in global or local test which are important in federated learning.  \n5. In the first sentence of the last paragraph in Page 8, the authors mentioned that, no suitable federated learning system for comparison. As the key idea of the proposed method in the scenarios of federated learning is based on partial model sharing/update, why the authors did not include the comparison of some partial model transmission-based federated learning (FL) methods or in personalized FL (PFL) scenarios, e.g., the sparisification based FL and PFL in which the communication efficiency and the performance of personal models are investigated. \n6. In FL settings used on Tab. A.7 and Fig. A.3, the performance of conventional models based FL methods should be provided, with regard to accuracy and communication efficiency. It will help to demonstrate the effectiveness. \n7. In the writing, more discussion should be provided around the proposed method in the federated settings, such as the advantages and the difference from the conventional partial model based PFL methods.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bnLBhl3Zq6", "reviewer": "Reviewer_YJfE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose FiLM Transfer (FiT) which combines transfer learning and meta-learning. FiT aims to achieve parameter efficiency under a limited number of training samples without degrading accuracy. This potentially results in inexpensive model updates for personalization and communication-efficient federated learning. \nFiT backbone is a frozen pre-trained network that is augmented with parameter-efficient FiLM layers (for adapting to the new tasks), and a Naive Bayes classifier layer with fewer updateable parameters rather than the linear layer.", "summary_of_the_review": "The paper proposes a technique, FiT to yield parameter efficient models that outperform BiT by combining transfer learning with meta-learning.", "strength_and_weaknesses": "**Strengths**\n1) The proposed FiT has superior parameter efficiency and classification accuracy using orders of magnitude fewer updateable parameters and communication cost than BiT.\n2) The paper is well-organized with detailed experiments and comparisons across various downstream tasks in few-shot learning.\n\n\n**Weakness**\n1) The paper combines ideas from FiLM (to learn from little data and generalize to more complex and/or substantially different data) layers and meta-learning-inspired episodic training protocol [Dumoulin'2021] to outperform transfer learning baseline BiT.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_ohtXDXAXWq", "reviewer": "Reviewer_8TEt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "   The paper proposed a parameter and data efficient architecture for low-shot transfer learning consisting of an automatically configured Naive Bayes classifier and FiLM layers that are used to adapt a fixed, pretrained backbone to a downstream dataset. ", "summary_of_the_review": "   This paper proposed Feature-wise Linear Modulation layers, which can be easily inserted into Resnet or Efficient networks, and only a few parameters are added. The use of an automatically configured Naive Bayes classifier to replace the traditional classification layer significantly improves the performance of the few shot and image/video classification.", "strength_and_weaknesses": "  Strength: The method proposed in this paper is simple, efficient and easy to implement. The adequate experimental results on few-shot,VTAB-1k benchmark, Personalization video dataset and Few-shot Federated Learning verify the efficiency of FiT.\n\n\nWeaknesses: The motivation for using a Naive Bayes classifier is not clear, why it would be better than a traditional MLP classification layer.The design form of Feature-wise Linear Modulation (FiLM) layers is very simple and direct. Why is such design effective?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JCKI2Inbgx", "reviewer": "Reviewer_upHx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel idea to improve few-shot learning by synergizing ideas from the transfer learning and meta-learning communities. Specifically, from transfer learning, it leverages the backbones pretrained on large image datasets and fine-tuned parameter-efficient adapters, while from meta-learning, it uses metric learning-based final layer classifiers trained with episodic protocols. The authors conducted experiments on several downstream datasets, illustrating that their proposed method, FIT, achieved superior accuracy at low-shot on benchmarking datasets. In addition, they demonstrated the method's parameter efficiency in distributed few-shot learning including model personalization and federated learning tasks.", "summary_of_the_review": "This paper proposes a novel method combining existing ideas from transfer learning and meta-learning, which enables a parameter and data-efficient network architecture for low-shot learning. The authors conducted a range of experiments on standard datasets to illustrate the effectiveness of their proposed method. This strategy could be inspiring to the community and potentially generalize beyond imaging classification tasks.", "strength_and_weaknesses": "Strengths: The paper introduces a novel strategy to combine ideas from transfer learning and meta-learning communities for developing data-efficient and parameter-efficient learning systems. The authors motivated the proposed method well. The method achieved superior performance at low-shot on several benchmarking datasets. In addition, the authors further demonstrated FIT's parameter efficiency and superior accuracy in distributed low-shot personalization and federated learning applications. The evaluation is relatively convincing.\n\nWeaknesses: The paper includes a range of quantitative evaluation results, but it would be better to also provide a more in-depth explanation of the experimental results and the authors' insights on why their proposed method works, and in which settings it would suffer.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FH0qm9lfvJL", "reviewer": "Reviewer_DFdq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a parameter and data efficient network architecture and the corresponding training procedure for personalization and federated learning. The proposed architecture includes FiLM layers in the pretrained backbones and takes Naive Bayes classifier as the head layer. ", "summary_of_the_review": "Overall, it's a good paper, I'd suggest to accept this paper for publication. ", "strength_and_weaknesses": "The paper is well motivated as both personalization and federated learning require parameter efficient and training data efficient model. \nThe proposed idea is succinct and to me it has potential to be generalized to applications other than CV.\nThe paper provides sufficient and convincing experiment results to show the superior performance of the proposed architecture. \n\nSome details need to clarify or fix:\n1. During the episodic fine tuning, in each iteration, do you optimize \\phi and e via gradient ascent for only one step, or optimize them by running the gradient ascent for multiple steps? \n2. In section 4.1, for the one-shot learning, how do you split the downstream dataset D as D_train and D_test?\n3. In the second paragraph on page 8, it's not clear to me why Naive Bayes head is not transferred to the server. Do you assume that there are no shared classes among clients? \n4. There is a typo in the 4th paragraph on page 3, \"b_d\" should be \"d_b\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster