Contextual Squeeze-and-Excitation for Efficient Few-Shot Image Classification
Authors: Massimiliano Patacchiola
John F Bronskill
Aliaksandra Shysheya
Katja Hofmann
Sebastian Nowozin
Richard E Turner
Venue: NeurIPS 2022 Accept (2022)
Keywords: few-shot learning
meta-learning
transfer learning

Abstract: Recent years have seen a growth in user-centric applications that require effective knowledge transfer across tasks in the low-data regime. An example is personalization, where a pretrained system is adapted by learning on small amounts of labeled data belonging to a specific user. This setting requires high accuracy under low computational complexity, therefore the Pareto frontier of accuracy vs. adaptation cost plays a crucial role. In this paper we push this Pareto frontier in the few-shot image classification setting with a key contribution: a new adaptive block called Contextual Squeeze-and-Excitation (CaSE) that adjusts a pretrained neural network on a new task to significantly improve performance with a single forward pass of the user data (context). We use meta-trained CaSE blocks to conditionally adapt the body of a network and a fine-tuning routine to adapt a linear head, defining a method called UpperCaSE. UpperCaSE achieves a new state-of-the-art accuracy relative to meta-learners on the 26 datasets of VTAB+MD and on a challenging real-world personalization benchmark (ORBIT), narrowing the gap with leading fine-tuning methods with the benefit of orders of magnitude lower adaptation cost.

Contextual Squeeze-and-Excitatoo for Efficient Few-Shot Image Classification
Massimiliano Patacchiola University of Cambridge mp20080cam. ac. uk
John Bronskill University of Cambridge jfb540cam.. ac uk
Aliaksandra Shysheya University of Cambridge as29750cam, ac uk
Katja Hofmann
Sebastian Nowozin' nowozin@gmail.com com
Richard E. Turner University of Cambridge ret268cam ac. uk
Microsoft Research
aahofman@microsoon com
Abstract
Recent years have seen a growth in user-centric applications that require effective knowledge transfer across tasks in the low-data regime. An example is personaliza- tion, where a pretrained system adapted by learning on small amounts labeled data belonging to specific user. This setting requires high accuracy under low computational complexity, therefore the Pareto frontier of accuracy vs. adaptation cost plays crucial role. In this paper we push this Pareto frontier in the few-shot image classification setting with a key contribution: a new adaptive block called Contextual Squeeze-and- axd-txxcitation (CaSE) that adjusts a pretrained neural network on a new task to significantly improve performance with single forward pass of the data (context). We meta-trained CaSE blocks conditionally adapt the body of network and fine-tuning routine to adapt linear head, defining a method called UpperCaSE. UpperCaSE achieves new state-of-the-as accuracy relative to meta-learners on the 26 datasets of VTAB+MD and on a challenging real-world personalization benchmark (ORBIT), narrowing the gap with leading fine-tuning methods with the benefit of orders of magnitude lower adaptation cost.
Introduction
In recent years, the growth of industrial applications based recommendation systems (Bennetti et al. (2007), speech recognition (Xiong et al., (2018), and personalization (Massiceti et al. 2021) has sparked interest in machine learning techniques that able adapt model on small amounts of data belonging to a specific user. A key factor in many of these applications the Pareto frontier of accuracy vs. computational complexity (cost adapt). For example, real-time classification task on a phone, pretrained model must be personalized by exploiting small amounts of data on the user's device (context). In these applications the goal is twofold: maximize the classification accuracy on unseen data (target) while avoiding any latency and excessive use of computational resources.
Methods developed to face these challenges in the few-shot classification setting be grouped in two categories: meta-learning and fine-tuning. Meta-learning is based on the idea cearning-how-to learn by improving the algorithm itself (Schmidhuber, 1987 Hospedales et al. 2020). Meta-learners are trained across multiple tasks to ingest a labeled context set, adapt the model, and predict the class membership of an unlabeled target point. Fine-tuning methods adjust the parameters of pretrained
neural network on the task at hand by iterative gradient-update (Chen et al. 2019 Triantaillouu et al. 2019
Tian al. 2020 Kolesnikov al. 2020 Dumoulin al. 2021).
Work done while the author was at Microsoft Research Cambridge (UK)
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

We can gain an insight on the differences between those two paradigms by comparing them in terms of accuracy and adaptation cost. Figure [ iillssratees this comparison by showing on the vertical axis the average classification accuracy on the 18 datasets of the Visual Task Adaptation Benchmark (VTAB, [Dumoulin et al. 2021), and on the horizontal axis the adaptation cost measured as the number of multiply-accumulan operations (MACs) required to adapt on single task (see Appendix C.1 for details). Overall, fine-tuners achieve a higher classification accuracy than meta-learnen but are more expensive to adapt. The comparison between two olate-of-the-e art methods for both categories,
Big Transfer (BiT. Kolesnikov et al. 2020) and LITE (Bronskill et al., (02T), shows a substantial performance gap of 14% in favor of the fine-tuner but at a much higher adaptation cost, with BiT requiring 526 x 10 MACs and LITE only 0.2 x 10 MACs.
It is crucial to find solutions that retain the best of both worlds: the accuracy of fine-tuners and low adaptation cost of meta-learners. The main bottleneck that hampers the adaptation of fine-tuners is the need for multiple gradi- ent adjustments over the entire set of network parameters, Restricting those adjustments to the last linear layer (head) significantly speeds up fine-tuning, but it harms perfor- mance (e.g. see experiments in Section 5.I Finding a way to rapidly adapt the feature extractor (body) is there-
Mela-leamers
Fine- tuners
Hybrids
70
BIT
65
UpperCaSE
Accuracy on VTAB
60
(ours)
MD-Transfer
14%
55
fore the main obstacle to bypass. In this paper we propose hybrid solution to this issue. exploiting meta-lcarned adapters for rapidly adjusting the body and a fine-tuning routine for optimizing the head.
LITE
CTX
50
ProtoNets
ProtoMAML
45
SUR
At the core of our approach is a novel extension of the popular Squeeze-and-Exeitate
40
Low
High
block proposed by Hu et al. (018)) to the meta-learning setting that we call Contextual Squeeze-and- iairezzz-anndExiiiatiionn (CaSE). We exploit CaSE as building block of hybrid training protocol called UpperCaSE which is based on the idea of adjusting the body of the network in a single forward pass over the of expensive fine-tuning
Adaptation cost (lower is better)
Figure 1: Accuracy and adaptation cost VTAB for meta-learners (blue), fine- tuners (red), and hybrids (blue-red). Black dotted-line the previous Pareto front across categories. UpperCaSE nar- rows the gap with the leading fine-tuning method and represents the best trade-off in terms of accuracy/adaptation-com
context, and reserving the routines for the linear head, similarly to methods like MetaOptNet (Lee et al. 2019)), R2D2 (Bertinetto et al.) 2018). and ANIL (Raghu et al. 2019). Figure [] shows how UpperCaSE substantially improves the performance in the low-cost regime, outperforming meta-learners, fine-tuners such as MD-Transfer (Triantafi al., 2019), and reducing the gap with the current state of the art (BiT). When adaptation cost is critical, UpperCaSE is the best method currently available since it can provide substantial computation savings and compelling classification performance.
Our contributions can be summarized as follows:
1.
We introduce a new adapter called Contextual Squeeze-and sruueze-and-Exxiiaaiin (CaSE), based on the popular Squeeze-and-Err Excitatttiiion model proposed by Hu et al. (2018), that outperforms other adaptation mechanisms (e.g. the FiLM generators used in Bronskill et al. 202I) in terms of parameter efficiency (a 75% reduction in the number of adaptation parameters) and classification accuracy (a 1.5% improvement on MetaDataset and VTAB). The code is released with an open-source license |
2. We use CaSE adaptive blocks in conjuction with fine-tuning routine for the linear head in a model called UpperCaSE, reporting an improved classification accuracy compared to the SOTA meta-learner (Bronskill et al. 2021) the 8 datasets of MDv2 (+2.5% on average) and the 18 datasets of VTAB (+6.8% on average), narrowing the gap with BiT (Kolesnikoy et al. 2020) with the benefit of orders of magnitude lower adaptation cost.
3. We showcase the potential of UpPerCaSE in a real-world personalization task on the ORBIT dataset (Massiceti al., 2021), where compares favorably with the leading methods in the challenging cross-domain setting (training on MDv2, testing on ORBIT).
http::///gthbbbccoommmtccaaaa/tt squueeeeeeetuaasseseeeeeeeeeeeee and-excitaiioo

---

2 Contextual Squezzeaand-Excitation (CaSE)
Problem formulation In this paragraph we introduce the few-shot learning notation, as this will be used to describe the functioning of a CaSE adaptive block. Let us define a collection of meta-training tasks as D = [T....., TD} where т (C,, Ti) represents generic task composed of a context set C, = {xx, 31,,.............. (x, v) / and target set T: = {xx y)...... (x. y) of input-output pairs. Following common practice we use the term shor to identify the number of samples per class (e.g. 5-shot is 5 samples per class) and the term way to identify the number of classes (e.g. 10-way is 10 classes per task). Given an evaluation task T. = [C.,,,, the goal to predict the true label y. of the unlabeled target point x, conditioned on the context set C..
In fine-tuning methods, we are given a neural network fo(.). with parameters estimated via standard supervised-eerningg on a large labeled dataset (e.g. ImageNet). Given a test task Ta adaptation consists of minimizing the loss C(-) via gradient updates find the task-specific parameters
0-, f G(e, L, T., fo), where e is learning rate, and G(-) is a functional representing an iterative routine that returns the adapted parameters 0,, (used for prediction). This procedure is particularly effective because it can exploit efficient mini mini-batching, parallelization, and large pretrained models.
In meta-learning methods training and evaluation are performed episodically (Vinyals al., 2016), with training tasks sampled from a meta-train dataset and evaluation tasks sampled from an unseen meta-test dataset. The distinction in tasks is exploited
define hierarchy. The parameters divided in two groups: & task-common parameters shared across all tasks (top of the hierarchy), and 2b task-specific parameters estimated on the task at hand as part of an adaptive mechanism (bottom of the hierarchy). The way & and xb come into play is method dependent; they can be estimated via gradient updates (e.g. MAML, Finn et al. 2017). learned metrics (e.g. ProtoNets, Snell al. (2017).
or Bayesian methods (Gordon et al. 2018 Patacchiola et al. 2020 Sendera et al. 2021).
Standard Squeeze-Exeite (SE) We briefly introduce standard SE (Hu et al. 2018). we going to build on top of this work. SE is an adaptive layer used in the supervised learning setting to perform instance based channel-wise feature adaptation, which is trained following a supervised protocol together with the parameters of the neural network backhone. Given a convolutional neural network, consider subset of T. layers and associate to each one of them Multi-Layer Perceptron (MLP), here represented as function 96 (-). The number of hidden units in the MLP is defined by the number of inputs divided by a reduction factor. Given a mini-batch of B input images, each convolution produces an output of size B x C x H X W where C' is the number of channels, H/ the height, and W the width of the resulting tensor. For simplicity we split this tensor into sub-tensors that are grouped into a set [H...... Ha with H, E RCx X To avoid clutter, we suppress the layer indexing when possible. SE perform a spatial pooling that produces tensor of shape B C' x 1; this can be interpreted as set of vectors (hi,... ,hn) with h; E CRC For each layer 1, the set is passed to the associated MI.P that will generate an individual scale vector 7 E TRC,, where
!
(1)
h
! B
(4)
(h!)
(1)
7 I
9
An elementwise product is then performed between the scale vector and the original tensor
H ( H x H
( B
(2)
with the aim of modulating the activation along the channel dimension. This operation can be interpreted soft attention mechanism, with the MLP conditionally deciding which channel must be attended to. A graphical representation of SE is provided in Tigure[(lett)
Contextual Squeeze- Excite (CaSE) Standard SE is an instance-based mechanism that is suited for i..dd data in the supervised setting. In meta-learning setting we can exploit the distinction in tasks to define a new version of SE for task-based channel-wise feature adaptation. For a task (C, T), consider the N images from the context set C, and the tensors produced by each convolution in the layers of interest (H,.... HN) } with H, E RCX As in standard SE, we first apply spatial pooling to each tensor H. which produces N vectors (hj,... hy) of shape h, E RC. Then context pooling is performed: this corresponds to an empirical mean over (h....., hy) } (see Appendix A for more details about context pooling). The pooled representation is passed to the associated MLP to produce a single scale-vector for that layer
7())
(1) g
(1))
with
h
b2
(3)
3

---

Squeeze-and-Exittee (SE)
Contextual Squeeze-and-Excitale (CaSE)
Learning/ Inference
Adaptative mode (context set)
Inference mode (larget set)
Convolution
x
Convolution
Convolution
MxCxHxx
Activation
B DxCxxxW
Activation
XxCxx
Activation
xxxxIxx
Spatial Pooling Cxcx1x1
Spatial Pooling
NxCxIx1
Context Pooling
1x0x1x1
MLP
Gxcxix1
MLP
LxCxix1
Cx111
Convolution
BxOx W
Convolution
N x EXCXHXX
Convolution
M x xxxCHH C
Figure 2: Comparison between the standard Squeeze- (left) and the proposed Contextual Squeeze-Excite (right). Red frames highlight the two key differences between SE and CaSE: context pooling and scale transfer from context target. B mini-batch size, C' channels, H height, W = width, N = context-set size, M target-set size, elementwise multiplication.
which is then multiplied elementwise by the original tensors
H 0 = H 10 H = H * ,7())
(4)
The scale vector is estimated in adaptive mode and transferred to the target points in inference mode (no forward pass on the MLPs), as shown in the rightmost part of Figure In synthesis, the three major differences between SE and CaSE are: (i) CaSE uses a contextual pooling with the aim of generating an adaptive vector per-task instead of per-instance as in SE; (ii) CaSE distinguishes between an adaptive mode and an inference mode that transfers the scale from context to target, while SE does not make such distinction; and (iii) CaSE parameters are estimated via episodic meta- meta-training while SE parameters via standard differences are fundamental to achieve superior performance in the few-shot setting. A representation of CaSE block is reported in Figure (right),,))ht),, (right), additional technical details are provided in Appendix A
snervised-trainingg In Section [. we show that those
Comparison with other adapters Popular adaptation mechanisms for few-shot learning are based on Feature-wise Linear Modulation layers (Perez |Perez et al. 2018). Those mechanisms perform adaptation using separate convolutional set-encoder to produce
embedding of the context set. The embedding is forwarded to local MLPs to produce the scale and shift vectors of the FiLM layers that modulate pretrained model. Variations of this adapter have been used in several methods, such as TADAM (Oreshkin et al.) 2018), CNAPs (Requeima et al., 2019). SimpleCNAPs (Bateni et al.. 2020), CAVIA (Zintgraf et al. 2019), and LITE (Bronskill al. 2021). We will use the generic term FiLM generator to refer to these adapters and the term FiLM to refer to the scale and shift vectors used modulate the activations. There are two key differences between FiLM and CaSE: (i) CaSE exploits context pooling to aggregate the activations of the backbone instead of a separate set-encoder in FilM generators (see Appendix[A ( for details) which is more efficient in terms of parameter count and implementation overhead; and (1I) FiLM uses scale and shift to modulate the activations, CaSE only the scale, therefore 50% less parameters are stored in memory and transferred during inference. In Section 5.1 we compare CaSE and the FiLM generators used in recent SOTA method (LITE, Bronskill al. 2021), ) showing that CaSE is superior in terms of accuracy while using fraction of the amortization parameters.
3 UpperCaSE: system description and optimization protocol
We exploit CaSE blocks part of UpperCaSE, hybrid training protocol based Coordinate- Descent (CD). We call this protocol hybrid because it combines a meta-training procedure to optimize the CaSE parameters (body) with a fine-tuning routine to estimate the task-specific parameters (head).
Preliminaries We are given a feature extractor (body) pretrained with supervised learning on a large dataset (c.g. ImageNet), defined as be (-) where 0 are the pretrained parameters. CaSE blocks,

---

parameterized by Φ, are added to the model at specific locations to give by b-) (see Appendix A for details about this step). We are interested in learning the CaSE parameters ø keeping constant the pretrained parameters O (omitted from here to keep the notation uncluttered). At training time, we are given series of tasks r = {C, T) D, where 'D is the training set. The number of classes (way) is calculated from the context set and used to define linear classification head Aw (-) parameterized
by ,b,. The complete model is obtained by nesting the two functions as hes (-)( We indicate a
forward pass through the body over the context inputs with the shorthand (C=) {zi..... ZN J.
where 2n is the context embedding for the input Xn. All the context embeddings and the associated labels are stored in .M= {(Z,, Vo ] I==
Optimization challenges We have two sets of learnable parameters, Φ the CaSE parameters, and + the parameters of the linear head for the task T. While Φ is shared across all tasks (task (ask-commonn) / must be inferred on the task hand (task-specific). In both cases, the objective the minimization of classification loss L. There are some challenges in optimizing the CaSE parameters in the body, as shown by the decomposition of the full gradient
dc dΦ
ULr dub
0c, 0Φ
Σ
Jup, d$
The first term / (sensitivity of the loss w.r.t. the head) and the direct gradient ac/06 (sensitivity of the loss w.r.t. the adaptation parameters with a fixed head) can be obtained with auto-differentiation as usual. The second term d (sensitivity of the head w.r.t. the adaptation parameters) is problematic because bb, is obtained iteratively after a sequence of gradient updates. Backpropagating the gradients to & includes backpropagation through all the gradient steps per- formed to obtain the task-specific ,/,, Previous work has showed that this produces instability, vanishing gradients, and high memory consumption (Antoniou al. 2018 Rajeswaran et al. 2019).
Meta-training via Coordinate- A potential solution to these issues the use of implicit gradients (Chen et al. 2020; Rajeswaran et al. 2019 Chen et al. 2022). The main problem with implicit gradients is the computation and inversion of the Hessian matrix as part of Cauchy's implicit function theorem, which is infeasible when the number of parameters in the linear head is large. Another possible solution the use of an intennatig-optmmizati
scheme, similar to the one proposed in number of recent methods such as MetaOptNet (Lee et al. 2019). R2D2 (Bertinetto) al.] 2018), and ANIL (Raghu et al., 2019). These methods share the idea of wwwee-looo-hhhoouueee loop-body meta-training, and they find the parameters of the linear head with closed form solutions or by stochastic optimization. Starting from similar assumptions we propose a simple yet effective scheme, which we formalize using Coordinate- The idea behind CD is to consider the minimization of a complex multi variate function as a set of simpler objectives that can be solved one at a time. In case, can consider the joined landscape w.r.t. Φ and + as composed of two separate sets of coordinates (block CD. Wrightt 2015). By minimizing - first, reach local minimum where OC,/ /åbp ~ O. Therefore CD induces direct optimization objective w.r.t. Φ. with Equation (5) reducing to OL, /C+/00 /0 (no red term). The time complexity of this method is only affected by the number of classes but constant w.r.t. the number of training points due to the use of mini- bati-hat which scales well with large tasks (e.g. those in MetaDataset and VTAB). See Appendix B for more details.
(CD) (Wright) 2015).
In practice, at each training iteration we sample a task T = (C, ~ D, perform a forward pass on the body (with CaSE in adaptive mode) to get
(c²) - +Z), }.
(6)
The context embeddings are temporarily stored in buffer with their associated labels M {(zn: yn) N to avoid expensive calls to by (-). We then set the head parameters to zero, and solve the first minimization problem (inner-loop), obtaining the task parameters s,, via hw.)
where a is learning rate, and G(.) is functional representing an iterative gradient-descen routine for parameter estimation (e.g. maximum likelihood estimation or maximum a posteriori estimation). Note that the iterative routine in Equation 7 only relies on the head hep (-) and not on the body (-). which is the primary source of memory savings and the crucial difference with common fine-tuning methods. Moreover, the inner-loop is agnostic to the choice of optimizer, it can handle many gradient steps without complications, exploit parallelization and efficient mini-batching
5

---

We then turn our attention to the second coordinate: the task lask-common parameters of the CaSE blocks in the body. For a single task, the update consists of single optimization step w.r.t. Φ (outer-loop) given support/target points and the task-specific parameters bb, identified previously. The final form of the equation depends on the optimizer, for a generic SGD the update is given by
ト Φ VV (C" U U bb),
(8)
where 3 is a learning rate. CaSE blocks must be in adaptive mode to allow the backpropagation of the gradients the MLPs. The process repeats, alternating the minimization along the two sets of coordinates. The pseudo-code for train and test is provided in Appendix
Inference unseen tasks After the training phase,
given unseen task (C., x.)
where x, is a single target input and y. the associate true label to estimate. Inference consists of three steps: (i) forward pass the body for all the context inputs with CaSE set adaptive mode as in Equation (6) and embeddings/labels stored in M,, (ii) estimation of the task-specific parameters 2b. via iterative updates in Equation 7 and (iii) inference of the aarget-point membership via forward pass over body and head ÿ-- lt.. (bx(x)) with CaSE in inference mode.
4 Related work
Meta-learning There has been a large volume of publications related to meta-learning. Here we focus on those methods that are the most related our work, and refer the reader to a recent survey for additional details (Hospedales et al. 2020). LITE (Bronskill et al. (222]) is protocol for training meta-learners on large images, that achieved SOTA accuracy on VTAB+MD. LITE particularly relevant in this work, as its best performing method is based on Simple CNAPs (Bateni et al.) 20220) that exploits FiLM for fast body adaptation. We compare against LITE in Section [5.2] showing that UpperCaSE is superior in terms of classification accuracy and parameter efficiency.
Fine-tuning Chen et al. (019)) were the first expose the potential of simple fine-tuning baselines for transfer learning. -TT-nssseer has been proposed in Triantafilouu et al. (2019) as an effective fine-tuning baseline for the MetaDataset benchmark. More recently Kolesnikov et al. (2020) have presented Big Transfer (BiT), showing that large models pretrained on ILSVRC-2012 ImageNet and the full ImageNet-21k are very effective transfer learning. MD-Transfer and BiT differ in terms of classification head, learning schedule, normalization layers, and batching. Fine-tuning only the last linear layer can be effective (Bauer et al. 2017 Tian et al., (2020). We compare against this baseline in Section 5.1 showing that adapting the body via CaSE significantly boosts the performance. Overall, fine-tuners have consistently outperformed meta-learners terms of classification accuracy, only under particular conditions (e.g. strong class-imbalance) the trend is reversed (Ochal et al.) 202 b).
Hy Hybrids Hybrid methods are trained episodically like meta-learners but rely on fine-tuning routines for adaptation. Model Agnostic Meta-Learning (MAML, Minn et al. (017)) finds a set of parameters that good starting point for adaptation towards new tasks in a few gradient steps. MAML
has been the inspiration for a series of other models such as MAML++ (Antoniou et al. 2018).
PMo (Triantafillou et al. 2019, and Reptile (Nichol al. 2018).
Dynamic networks CaSE blocks belong to the wider family of dynamic networks, models that can adapt their structure or parameters different inputs (Han al.
2021). Adaptive components have been used in a variety of applications, such as neural compression (Veit and Belongie, 20188) Wu et al. 2018). generation of artistic styles (Dumoulin al., 2016 Huang and Belongie| 2017). or routing (Guo et al. (019). Residual adapters (Rebaffi et al. 2017 2018) have been used in transfer learning (non few-shot) but they rely on fine-tuning routines which are significantly slow during adaptation. More recently. Li et al. (2022) have used serial and residual adapters in the few-shot setting, with the task-specific weights being adapted from scratch on the context set. This approach has similar limitations, since it requires backpropagation to the task-specific weights in the body of the network which is costly. In Sun et al. (2019) the authors introduce a Transfrr Learning (MTL) method for the few-shot setting. In MTL a series of scale and shift parameters are meta-learned across tasks and then dynamically adapted during the test phase via fine-tuning. This method suffers of similar limitations, as the fine-tuning stage is expensive during adaptation. Moreover, MTL relies on scale and shift vectors to perform adaptation whereas CaSE only relies on scale vector, meaning that needs to store and transfer 50% less parameters at test time.

---

reduction=32 (0.9M params)
adaptation
(1.7M params)
quartle
CaSE with reduction-32 (0.8M) parame))
CaSE with reduction=16 16 (1.8M params)
Whiskers: 1.5 quartii range
2.0
CoSi activation value
1.5
50
1.0
1
0.5
specall structur.
natural spocial.
stnactur
input
Dupput
Benchmerk
Adaplaann Benchmark
EfficientNelBo layers depth
Figure 3: Left: CaSE guueeze-and-Exciane 84 inputs, mahalanobis-distame head). CaSE outperforms SE in all conditions. Center: CaSE FiLM generators (Bronskill et al., 2021)) and baseline with no body adaptation (all methods use EfficientNetBO, 84 x 84 inputs, Mahalanobis-distanee head). CaSE outperforms FiLM generators in all conditions. Right: boxplot of CaSE activations different depth of an EfficientNetBO for 800 tasks sampled from the MDv2 test set (224 x 224 inputs, UpperCaSEE). The modulation of CaSE is minimal at early stages for general-purpose filters and increases deeper stages.
(SE) (both methods EfficientNetBO, 84 x
Experiments
In this section we report on experiments on VTAB+MD (Dumoulin et al. 2021) and ORBIT (Massiceti al. 2021). VTAB+MD has become the standard evaluation protocol for few-shot approaches, and it includes a large number of datasets (8 test dataset for MD, 18 for V. For a description of ORBIT, see Section 5.2 In all experiments we used the following pretrained (on ImageNet) backbones: EfficientNerB0 from the official Torchvision repository; ResNet50xx released with BiT (Kolesnikov et al. 2020). We used three workstations (CPU cores, 110G of RAM, and a Tesla V100 GPU), the meta-training protocol of Bronskill et al. (202T) (10K training tasks, updates every 16 tasks), the Adam optimizer with a linearly-decayed learning rate in [10-3 10-5] for both the CaSE and linear-head. The head is updated 500 times using random mini-batch of size 128. MD test results are averaged over 1200 tasks par-dataset (confidence intervals in appendix). We did not use data augmentation. Code to reproduce the experiments is available at httpp://gtt thub. cww coom/mpatacchiia wttp:///ooooooaaaaotceeeeeoeeeee
exciitatit
5.1 Analysis of CaSE blocks
In this sub-section report empirical results related to CaSE blocks in three directions: 1) compared standard SE (Hu et al. 2018)) and CaSE on MDv2 and VTAB, confirming that a) adaptation helps over not adapting, b) contextual adaptation (CaSE) outperforms instance based adaptation (SE); 2) we compare CaSE against a SOTA FiLM generator (Bronskill et al. 2021), showing that CaSE is significantly more efficient using 75% fewer parameters while boosting the classification accuracy average by +1.5% on VTAB and MD2; and 3) we provide an insight on the effectiveness of CaSE blocks with series of qualitative analysis.
Comparing SE vs. CaSE We compare standard SE and the proposed CaSE on VTAB and MD- v2. For a fair comparison we keep constant all factors of variation (backbone, training schedule, hyperparameters, etc.) and use the same reduction of 32 (0.8M adaptive parameters). In order to compare the results with the other experiments in this section, we use Mahalanobis-distaren head as in Bronskill et al. (202I), I), reporting results with a linear head in the appendix. We summarize the results in Figure (left) and add tabular breakdown in Appendix C2 CaSE outperforms SE in all conditions, confirming that a contextual adaptation mechanism is fundamental to transfer knowledge effectively across tasks.
Comparing adaptation mechanisms perform a comparison on VTAB+MD CaSE against FiLM generators (Bronskill et al. (021). and baseline that uses pretrained model but no adaptation of the body. Methods are compared in identical conditions, using a Mahalanobis-distame an EfficientNetBO backbone, and same training schedule. We show summary of the results in
head,

---

Figure (center) and provide full breakdown in the appendix. CaSE is able to outperform FiLM generators all conditions. In Appendix C.3 we report the results for CaSE with reduction 64 (0.4M parameters) showing that it is able to outperform FiL.M generators (1.7M parameters) using fraction of the parameters. The comparison with the baseline with no adaptation, shows that in all but one condition ( (VTAB specialized) adaptation is beneficial. This is likely due to the strong domain shift introduced by some of the specialized datasets.
Role of CaSE blocks To examine the role of CaSE blocks we analyze the aggregated activations at different stages of the body for 800 tasks sampled from the MDv2 test set using an EfficientNetBO trained with UpperCaSE on 224 x 224 images. In Figure (right) we report the aggregated distribution as boxplots, and Appendix C.5 we provide per-dataset breakdown. Overall the median is close to 1.0 (identity) which is the expected behavior on average we aim at exploiting the underlying pretrained model. The variance small early stages, indicating that CaSE has learned to take advantage of general-purpose filters that are useful across all tasks. In deeper layers the variance increases, showing a task- task-specific modulation effect. In Appendix [C.5 we also include a plot with per-channel activations for all datasets at different depths, showing that the modulation is similar across datasets at early stages and it diverges later on. An ablation study of different factors (e.g.
reduction, number of hidden layers, activation functions) is reported in Appendix C.4
5.2 Performance evaluation of UpperCaSE.
In this sub-section we analyze the performance of UpperCaSE in settings: 1) comparison on the VTAB+MD benchmark against SOTA fine-tuners and meta-learners, where we show that UpperCaSE able to outperform all the meta-learners, narrowing the gap with Big Transfer (BiT) on VTAB; 2) we show application of UpperCaSE in real-world personalization task on the challenging ORBIT dataset (Massiceti et al., 2021) for the cross-domain condition MDBITT where we achieve the best average-score in most metrics, although these improvements are within the error bars.
Table 1: UpperCaSE outperforms fine-tuners MDv2 and narrows the gap VTAB with the leading method (BiT) with a much lower adaptation cost. Average accuracy on the 26 datasets of VTAB+MD. RN=ResNet, EN=EfficientNet. Img: image size. Param.: total parameters (no adapters) in millions. Cost: MACs to adapt on a task (10-shot, 100-way), in Teras. Best results in bold.
Cost
MDv2
VTAB
Method
Protocol
Net
Img
Param.
MACs
all
all
natur.
spec.
struc.
-rD-sfanffer
fine-tuning
RN18
126
11.2
118.6
63.4
55.6
52.4
72.9
49.3
SUR
fine-tuning
RN50
224
164.6
28.8
71.3
43.7
50.9
66.2
27.2
Big Transfor
fine-tuning
RN50
224
23.5
526.3
73.3
65.4
69.4
81.0
54.5
UpperCaSE
hybrid
RN50
224
23.5
0.8
74.9
56.6
66.3
80.
37.6
UpperCaSE
hybrid
ENBO
4.0
0.2
76.1
58.4
69.1
80.3
39.4
Table 2: UaPE outperforms all meta-learningghyyrii methods and uses the lowest number of parameters per adaptive blocks. Average accuracy on the 26 datasets of VTAB+MD. Re=ResNet, FN=FffficienNNe.. Img: image size. Param.: total parameters (excluding adapters). Adapt.: total adaptive parameters in millions. Best results bold.
Adapt. J count
MDv2 1
VTAB t
Method
Protocol
Net
Img
Param.
all
natur.
spec.
struc.
ProtoMAMI.
hybrid
RN18
126
11.2
n/a
64.2
45.0
45.7
70.7
31.5
CTX
meta-learning
RN34
224
21.3
n/a
71.6
50.5
61.1
67.3
34.0
ProtoNet
meta-learning
ENBO
224
4.0
n/a
72.7
46.1
60.9
64.2
25.9
LITE
meta-lcarning
ENBO
224
4.0
1.7
73.8
51.4
65.2
71.9
30.8
UpperCaSF UpperCaSE
hybrid
RN50
224
23.5
0.8
74.9
56.6
66.3
80.1
37.6
hybrid
ENBO
224
4.0
0.4
76.1
58.4
69.1
80.3
39.4
Comparison on MD We compare UpperCaSE against fine-tuners, meta-learners, and hybrids on the 18 datasets of VTAB and the 8 datasets of MetaDataset-v2 (MDv2) and report the results

---

Table ORBIT: UpperCaSE obtains the best average-score in most metrics, being within error bars with leading methods. Average accuracy and 95% confidence interval for frames, videos, and frames to recognition (FTR). Cost: average MACs over all tasks (Teras). Results and setup from Massiceti et al. (202T): meta-train on MetaDataset and test on ORBIT, image-size 84 x 84, ResNer18 backbone, 85 test tasks (17 test users, tasks per user). Best results (within error bars) in bold.
Cost
Clean Video Evaluation (CLE-VE)
Clutter Video Evaluation (CLU-VE)
Method
M
frame acc.t
FTR
video acc.
frame acc.
FTR
video acc.
ProtoNet
3,2
59.0:12.2
11.5.:.88
69.2.13.0
47.0
20.4
52.8:1..5
CNAPs
3.5
51.9:2,5
20.8+2.3
60.8+3.2
11,6=19
30.7+2.1
43.0:t2.55
MAML
95.3
42.5++2.7
37.3++3.0
47.0++3.2
24.3=+1.8
62.3442.3
25.7:+2.2
FineTTuner
317.7
61.01:222
11.51:88
72.622299
48,4=1199
19.1
54.1
UpperCaSE
3.5
63.0++2.2
8.8+1.66
74.4+-2.8
48.1+1.8
18,2+1.77
54.5++2.5
in Table and Table 2 UpperCaSE outperforms all methods (including BiT) on MDv2 with an accuracy of 74.9% (ResNet50) and 76.1% (EfficientNetBO). On VTAB, UpperCaSE outperforms most methods, narrowing the gap with BiT. closer look at the differences in performance on VTAB between UpperCaSE and BiT (see T shows that the gap is narrower on the natural and specialized splits (+3.11% and +0.9%) but larger on structured (+16.9%).
The breakdown by dataset reported in Appendix C.6 shows that the major performance drops are on tasks that require localization and counting (e.g. dSprites, SmalINORB). Similar issues are encountered by methods such as LITE (Bronskill et al. (021)) which are based on FiLM generators, suggesting that those tasks may introduce a strong domain shift w.r.t. the meta-training set that is difficult to compensate without fine-tuning the body. It is not clear whether transfer learning is beneficial on these datasets the first place. The results in terms of adaptation cost (see Table T over synthetic task (10-shot, 100 way) show that UpperCaSE. is orders of magnitude more efficient (0.2 x 10 MACs) than all fine-tuners, with BiT being the most expensive method overall (526.3 x 10 MACs). The comparison against meta-learners in terms of number of adaptive parameters (see Table 2) shows that UpperCaSE requires a fraction of the parameters (0.4 vs 1.7 millions for an EfficienNNeBB)) compared to LITE (Bronskill et al. 202T) which is based on FiLM generators.
Comparison on ORBIT We compare UpperCaSE to other methods on ORBIT (Massiceti al.. 2021), rorldworld dataset for teachable object recognizers. ORBIT consists of 3822 videos of 486 objects recorded by 77 blind/low-vision people on their mobile phones. The dataset challenging because objects are poorly framed, occluded, blurred, and in wide variation of backgrounds and lighting. The dataset includes two sets of target videos, one for clean video evaluation (CLE-VE) with well-centered objects, and another for clutter video evaluation (CLU-VE) with objects in complex, cluttered environments. We consider a hard transfer-learning condition where classifiers are meta-trained on MetaDataset and tested on ORBIT.
UpperCaSE outperforms all other methods (on average) on most metrics, being within error bars with the two leading methods. Comparing UpperCaSE with Fineruune, the gap in favor of UpperCaSE is marginal on CLU-VE but substantial on CLE-VE (frame accuracy + +2,, video accuracy + 1.8%, and FTR - 2.77). Comparison in terms of adaptation cost (average MACS over all tasks) shows that UpperCaSE is orders of magnitude more efficient than FineTuner and close to the leading method (ProtoNet).
Results are reported in Table 3
Conclusions
We have introduced new adaptive block called CaSE, which is based on the popular Squeeze-and and Excitation (SE) block proposed by Hu et al.] (2018). CaSE effective at modulating pretrained model in the few-shot setting, outperforming other adaptation mechanisms. Exploiting CaSE we have designed UpperCaSE, hybrid method based on a Coordinate-Descent training protocol, that combines the performance of fine-tuners with the low adaptation cost of meta-learners. UpperCaSE achieves SOTA accuracy w.r.t. meta-learners on the 26 datasets of VTAB+MD and it compares favorably with leading methods in the ORBIT personalization benchmark.

---

Limitations There are two limitations that are worth mentioning: (i) UpperCaSE requires iterative gradient updates that herdware-dependent and may be slow/unaviilable some portable devices; (ii) breakdown VTAB results per-dataset shows that the method falls short on structured datasets. This indicates that fine-tuning the body may necessary for high accuracy when the shift w.r.t. the meta-training set is large.
Societal impact Applications based CaSE and UpperCaSE could be deployed in few-shot classifi- cation settings that can have a positive impact such as: medical diagnosis, recommendation systems, object detection, etc. The efficiency of our method can reduce energy consumption and benefit the environment. Certain applications require careful consideration to avoid biases that can harm specific groups of people (e.g. surveillance, legal decision-makinn making).
Acknowledgments and Disclosure of Funding
Funding in direct support of this work: Massimiliano Patacchiola, John Bronskill. Aliaksandra Shysheya, and Richard E. Turner are supported by an EPSRC Prosperity Partnership EP/T005386/1 between the EPSRC, Microsoft Research and the University of Cambridge. The authors would like to thank: anonymous reviewers for useful comments and suggestions; Aristeidis Panos, Daniela Massiceti, and Shoaib Ahmed Siddiqui for providing suggestions and feedback on the preliminary version of the manuscript.
References
Antoniou, A., Edwards, H., and Storkey, A. (2018). How to train your maml. arXiv preprint arXiv: 1810.0950..
Bateni, P., Goyal, R., Masrani, V., Wood, F,, and Sigal, L. (2020). Improved few-shot visual classification. In Proceedings the IEEE Conference Computer Vision and Pattern Recognition.
Bauer, M., Rojas-Carulla, M., Swiatkowski. J. B., Schölkopf, B., and Turner, R. E. (2017). Discrimi- native k-shot learning using probabilistic models. arXiv preprint arXiv:1706. Sokiv:1706000222 7736000666.
Bennett, J., Lanning, S., et al. (2007). The netflix prize. Proceedings of KDD cup and workshop.
Bertinetto, L., Henriques, J. F., Torr, P. H., and Vedaldi, A. (2018). Meta-learning with differentiable closed-fome solvers. arXiv preprint
Bronskill, J., Massiceti, D., Patacchiola, M., Hofmann, K., Nowozin, S., and Turner, R. (2021). Memory efficient meta-learning with large images. Advances in Neural Information Processing Systems.
Chen, W., Tripp, A., and Hernández-LLobtto J. M. (2022). Meta-learning feature representations for adaptive gaussian processes via implicit differentiation. arXiv preprint arXiv:2205.02777 00200..
Chen, W.- Liu, Y.-.., Kira, Z., Wang, Y.-C. F., and Huang, J.-B. (2019). closer look few-shot classification. arXiv preprint 1eXivv1904.000 042332.
Chen, Y., Friesen, A. L., Behbahani, F., Doucet, A., Budden, D., Hoffmmnn, M., and de Freitas, N. (2020). Modular meta-learning with shrinkage. Advances in Neural Information Processing Systems.
Dumoulin, V., Houlsby, N., Evei, U., Zhai, X., Goroshin, R., Gelly, S., and Larochelle, H. (2021). Comparing transfer and meta learning approaches on unified few-shot classification benchmark. arXiv preprint ar 30Xiv:2104.02638
Dumoulin, V., Shlens, J., and Kudlur, M. (2016). A learned representation for artistic style. arXiv preprint arXiv:/610.07629.
Finn, C., Abbecl, P., and Levine. S. (2017), Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning.
Garnelo, M., Schwarz, J., Rosenbaum, D., Viola, F., Rezende, J., Eslami, S., and Teh, W. (2018). Neural processes. arXiv preprint arXiv:1807.01622.
10

---

Gordon, J., Bronskill, J., Bauer, M., Nowozin, S., and Tumne, R. E. (2018). Meta-learning probabilistic inference for prediction. arXiv preprint 1xXvv:880500991..
Guo, Y., Shi, H., Kumar, A., Grauman, K., Rosing, T., and Feris, R. (2019). Spottune: transfer learning through adaptive fine-tuning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
Han, Y., Huang, G., Song, S., Yang, L.,, Wang, H., and Wang. Y. (2021). Dynamic neural networks:
A survey. IEEE Transactions on Puttern Analysis and Machine Intelligence.
Hospedales, T., Antoniou, A., Micaelli, P., and Storkey, A. (2020). Meta-learning in neural networks: A survey. arXiv preprini arXiv:2004.05439.
Hu, J., Shen, L.,, and Sun, G. (2018). Squeeze-ande excitacttn networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
Huang, X. and Belongie, S. (2017). Arbitrary style transfer in real-time with adaptive instance normalization. In Pruceedings of the IEEE/CVF International Conference on Computer Vision.
Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N. (2020). Big transfer (bit): General visual representation learning. European conference on computer vision.
Lee. K., Maji, S., Ravichandran, A., and Soatto, S. (2019). Meta-learning with differentiable convex optimization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
Li, W.-H., Liu, X., and Bilen, H. (2022). Cross-domain few-shot learning with task-specific adapters. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
Massiceti, D., Zintgraf, L., Bronskill, J., Theodorou, L.., Harris, M. T., Cutrell, E., Morrison, C., Hofmann, K., and Stumpf, S. (2021). Orbit: A real-world few-shot dataset for teachable object recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision.
Nichol, A., Achiam, J., and Schulman, J. (2018). On first-order meta-learning algorithms. arXiv preprint Xrv: /80300990.
Ochal, M., Patacchiola. M., Storkey, A., Vazquez, J., and Wang, S. (2021a). Few-shot learning with class imbalance. arXiv preprint arXiv:2101.02523.
Ochal, M., Patacchiola, M., Storkey, A., Vazquez, J., and Wang, S. (2021b). How sensitive are meta-learners to dataset imbalance? arXiv preprint arXiv:2104005344
Oreshkin, B., Rodríguez López, P., and Lacoste, A. (2018). Tadam: Task dependent adaptive metric for improved few-shot learning. Advances in neural information processing systems, 31.
Patacchiola. M.. Turner, J., Crowley, E. J., O' O'Boyle, M., and Storkey, A. J. (2020). Bayesian meta- learning for the few-shot setting via deep kernels. Advances in Neural Information Processing Systems.
Perez, E., Strub, F., De Vries, H., Dumoulin, V., and Courville, A. (2018). Film: Visual reasoning with general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence.
Raghu, A., Raghu, M., Bengio, S., and Vinyals, O. (2019). Rapid learning feature reuse? towards understanding the effectiveness of maml. arXiv preprint 190900999
Rajeswaran, A., Finn, C., Kakade, S. M., and Levine, S. (2019). Meta-learning with implicit gradients. Advances in Neural Information Processing Systems.
Rebuffi, S.-A., Bilen, H., and Vedaldi, A. (2017). Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems.
Rebuffi, S.-A., Bilen, H., and Vedaldi, A. (2018) 8). Efficient parametrization of multi-domal deep neural networks. Proceedings of the IEEE Conference on Computer Vision and Pottern Recog- nition.
11

---

Requeima, J., Gordon, J., Bronskill, J., Nowozin, S., and Turner, R. E. (2019). Fast and flexible multi- task classification using conditional neural adaptive processes. Advances in Neural Information Processing Systems.
Schmidhuber, J. (1987). Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-... hook. PhD thesis, Technische Universität München.
Sendera, M., Tabor, J., Nowak, A., Bedychaj, A., Patacchiola, M., Trzcinski, T., Spurek, P., and Zieba, M. (2021). Non-gaussian gaussian processes for few-shot regression. Advances in Neural Information Processing Systems.
Snell, J., Swersky, K., and Zemel, R. (2017). Prototypical networks for few-shot learning. Advances in Neural Information Processing Systems.
Sun, Q. Liu, Y., Chua, T.-.., and Schiele, B. (2019). Meta teta-transfer learning for few-shot learning, In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition.
Tian, Y., Wang, Y., Krishnan, D., Tenenbaum, B., and Isola, (2020). Rethinking few-shot image classification: a good embedding is all you need? In European Conference on Computer Vision.
Triantafillou, E., Zhu, T., Dumoulin, V., Lamblin, P., Evei, U., Xu, K., Goroshin, R., Gelada, C., Swersky, K., Manzagol, P.-A., al. (2019). Meta-dataset: dataset of datasets for learning to learn from few examples. arXiv preprint arXiv: arXiv:/903.0320 03096.
Veit, A. and Belongie, S. (2018). Convolutional networks with adaptive inference graphs. In Proceedings of the European Conference on Computer Vision (ECCV).
Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D., et (2016). Matching networks for one shot learning. Advances in Neural Information Processing Systems.
Wright, S. J. (2015). Coordinate descent algorithms. Mathematical Programming, 151.
Wu, Z., Nagarajan, T., Kumar, A., Rennie, S., Davis, L. S., Grauman, K., and Feris, R. (2018). Blockdrop: Dynamic inference paths in residual networks. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition.
Xiong, W., Wu, L., Alleva, F., Droppo, J., Huang, X., and Stolcke, A. (2018). The microsoft 2017 conversational speech recognition system. In JEEE international conference on acoustics, speech and signal processing (ICASSP).
Zintgraf, L., Shiarli, K., Kurin, V., Hofmann, K., and Whiteson, S. (2019). Fast context adaptation meta-learning International Conference on Machine Learning.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
dedicated paragraph in Section 6
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)) [Yes] Included in the supplementary material.
12

---

(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section [
(c) Did you report error bars (e.g.. with respect to the random seed after running exper- iments multiple times)? [Yes] Error bars have been reported for ORBIT and for the breakdonwn of the results on MetaDataset (MD) in the supplementary material. The aggregated results on MD in the paper do not report emror bars they are the average across many datasets. On VTAB error bars are not provided as the benchmark is only used for evaluation (this is in line with the original protocol).
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See setup description at beginning of Section [
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "c6fqIOKGfrq", "reviewer": "Reviewer_FFac", "summary": "This paper mainly focuses on few-shot learning. The authors advocate the importance of an efficient finetuning algorithm for FSL. To the end, they propose a new module based on SE block, which generates scaling parameters using task-specific information. Furthermore, they propose to leverage coordinate-descent in meta-training to solve the problems of instability, vanishing gradients and high memory consumption.", "questions": "1. The main concern comes from the proposed CaSE. This module takes episode-level features as input and provides channel-wise scaling to the final features, which reminds me of MTL [1]. The MTL is also aimed to meta-train new modules, that produce scaling and shifting parameters, and use them directly in testing phase. I find it very similar to the proposed method. It would be better if the authors could explain the exact difference between two methods and the significance.\n2. Beyond MTL, there are other works that can fulfill the goal of finetuning only few parameters on each episode, such as [2]. The authors should provide more discussion and comparison.\n3. I wonder if the authors can provide results on smaller dataset like miniImageNet so that the method can be compared with more state-of-the-art methods.\n\n\n[1] Sun Q, Liu Y, Chua T S, et al. Meta-transfer learning for few-shot learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 403-412.\n\n[2] Li W H, Liu X, Bilen H. Cross-domain Few-shot Learning with Task-specific Adapters[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 7161-7170.", "limitations": "Detailed comments are in the questions. In summary, I find the proposed module similar to the existing method in FSL, which is the main problem of the method.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. The authors conduct extensive experiments to show the effectiveness of their method.\n2. The proposed coordinate-descent for meta-learning is interesting and can be a potential plug-in solver for other meta-learning methods.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SuYESltz5Dj", "reviewer": "Reviewer_7nRY", "summary": "Authors correctly point out the tradeoff between adaptation costs and performance on novel datasets. Motivated by this tradeoff, authors introduce a novel adaptation layer, based on squeeze-and-excitation, that performs task-based feature modulation and is meta-learned in combination with a per-task learned linear head. The proposed method performs well on a variety of large-scale and difficult few-shot adaptation benchmarks at a reasonable computation cost. ", "questions": "Given CaSE’s conceptual similarity to FiLM and TADAM, which I consider to be far more appropriate baselines than SE, what exactly do the authors consider to be the true contribution and proper source of novelty here? \n\nGiven the above answer, how can this position be strengthened and made clearer in the body text of the paper?\n", "limitations": "Limitations are discussed clearly and fairly. Societal impacts are addressed very briefly, though since the impacts match those of few-shot learning more broadly, this is sensible. ", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS:\n\nA well-motivated, well-reasoned and widely applicable approach. Results are convincing and the model clearly accomplishes what it sets out to accomplish, in the way that it claims to accomplish it. Paper is clearly written, if not entirely well-organized or -focused (see below). \n\nWEAKNESSES:\n\nThe paper suffers from a misplaced focus in its presentation. Many of the named concepts presented as novel are not, and the truly novel contribution is somewhat limited and discussed only briefly. \n\n1. CaSE is introduced as a novel layer but is identical to SE but for the context pooling (discussed below) and the final activation layer (an implementation detail discussed in supplementary). \n\n2. Related to above, the adaptive mode / inference mode (described as “fundamental” on pg4 line 142) of CaSE is identical to TADAM (TADAM: Task Dependent Adaptive Metric for Improved Few-Shot Learning, NeurIPS2018), and also just a common-sense approach to handling a support vs query set. \n\n3. I hesitate to call the UpperCaSE meta-training scheme particularly novel or coordinate-descent-based. While optimization does switch back and forth between CaSE and head parameters, the head parameters reset with every new batch. In actuality the CaSE parameters are being meta-learned, while head parameters are being set in the inner loop, and UpperCaSE is just a straightforward and common-sense approach to meta-training. This procedure (train linear head to convergence, propagate gradient into meta-learned layers) is also the exact same procedure already introduced by MetaOptNet (Meta-Learning with Differentiable Convex Optimization, CVPR2019).\n\nIn my eyes the true novelty of CaSE is in taking the FiLM/TADAM approach to task-conditioning, and replacing the redundant task encoder layers with the appropriate intermediate network activations. This crucial difference is discussed only briefly in related work (pg6 line 237-238). I also consider this a somewhat limited conceptual contribution, empirical results aside. \n\nIn my eyes, this calls for a fairly substantial text revision, where the contribution is mainly a novel _approach_ to efficient adaptation rather than a “new adaptive block” (lines 7-8), and the conceptual linkages to TADAM/FiLM are explored and discussed rather than the similarities/differences relative to SE, which are much less relevant in this context. I recognize this could be a pretty idiosyncratic and overly specific take though – I’ll be curious to see what other reviewers think on the novelty/contribution. \n\nLess importantly, CaSE is conceptually quite similar to TADAM, and above proposed revisions aside, TADAM is at least worth a mention in related work, as a FiLM-derivative approach to few-shot learning. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7lNVTUig2h1", "reviewer": "Reviewer_9nir", "summary": "The work proposes a novel method for few-shot image classification. The method is named UpperCaSE and is based on adaptation of the Squeeze-and-Excitation block to learn task contextual parameters. Their approach is hybrid in terms of a combination between meta-learning and fine-tuning of the network. The proposed hybrid approach aims to bridge the gap between fine-tuning approaches which are more accurate, and the meta-learning approaches which have lower adaptation cost. Optimization protocol is based on Coordinate-Descent between the CaSE blocks in the network body (cross task parameters) and the task specific parameters of the head (last linear layer). The approach achieves new sota results for meta-learners on the VTAB+MD and ORBIT.", "questions": "I would have like to know how much the context pooling layer in the CaSE block is sensitive to the design choices. For example, have you tried to use several layers? adding a BN layer or other thing that either did not change the outcome or did not work at all.", "limitations": "I believe that the authors adequately addressed the limitations and potential negative societal impact of their work.\nI appreciate for being honest about the lower results on structure datasets and the hypothesis that this case might require fine-tuning of the network's body.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper is written in a clear and easy to follow manner.\n- The method requires only a single forward pass over the context.\n- The methods is simple and novel yet achieves very good results compared to other meta-learners.\n- Experimentation study is comprehensive and also shows the downside on structured tasks.\n\n**Weaknesses**\n- I would have liked to see some ablation study and discussion on the CaSE block architecture choice. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept