Memory Efficient Meta-Learning with Large Images
Authors: John F Bronskill
Daniela Massiceti
Massimiliano Patacchiola
Katja Hofmann
Sebastian Nowozin
Richard E Turner
Venue: NeurIPS 2021 Poster (2021)
Keywords: meta-learning
few-shot learning
few-shot image classification

Abstract: Meta learning approaches to few-shot classification are computationally efficient at test time, requiring just a few optimization steps or single forward pass to learn a new task, but they remain highly memory-intensive to train. This limitation arises because a task's entire support set, which can contain up to 1000 images, must be processed before an optimization step can be taken. Harnessing the performance gains offered by large images thus requires either parallelizing the meta-learner across multiple GPUs, which may not be available, or trade-offs between task and image size when memory constraints apply. We improve on both options by proposing LITE, a general and memory efficient episodic training scheme that enables meta-training on large tasks composed of large images on a single GPU. We achieve this by observing that the gradients for a task can be decomposed into a sum of gradients over the task's training images. This enables us to perform a forward pass on a task's entire training set but realize significant memory savings by back-propagating only a random subset of these images which we show is an unbiased approximation of the full gradient. We use LITE to train meta-learners and demonstrate new state-of-the-art accuracy on the real-world ORBIT benchmark and 3 of the 4 parts of the challenging VTAB+MD benchmark relative to leading meta-learners. LITE also enables meta-learners to be competitive with transfer learning approaches but at a fraction of the test-time computational cost, thus serving as a counterpoint to the recent narrative that transfer learning is all you need for few-shot classification.

Memory Efficient Meta-Learning with Large Images
John Bronskill University of Cambridge jfb540cam.ad ac. uk
Daniela Massiceti' Microsoft Research
Massimiliano Patacchiola" University of Cambridge mp2008@cam.as ac. uk
dmassicet @mi crosoft.com com
Katja Hofmann
Sebastian Nowozin
Richard E. Turner University of Cambridge ret26@cam ac uk
Microsoft Research kahofman@miicr crosott..coooooooooooooooooo com
Microsoft Research
senovoz@miccroofft.com
Abstract
Meta learning approaches to few-shot classification are computationally efficient at test time, requiring just few optimization steps or single forward pass to learn a new task, but they remain highly memory-intensive to train. This limitation arises because a task's s entire support set, which can contain up to 1000 images, must be processed before optimization step can be taken. Harnessing the performance gains offered by large images thus requires either parallelizing the meta-learner across multiple GPUs, which may not be available, trade-offs between task and image size when memory constraints apply. We improve on both options by proposing LITE, general and memory efficient episodic training scheme that enables meta-training on large tasks composed of large images on a single GPU. We achieve this by observing that the gradients for task can be decomposed into a sum of gradients over the task's ' training images. This enables us to perform a forward pass on a task's entire training by back-propagatee unbiased approximation of the full gradient. We LITE train meta-learners and demonstrate new state-of-te tfite-of-theer tetthheart art accuracy on the real-world ORBIT benchmark and of the 4 parts of the challenging VTAB+MD benchmark relative leading meta- meta-learners. LITE also enables meta-learners to be competitive with transfer learning approaches but at fraction of the test time computational cost, thus serving as a counterpoint to the recent narrative that transfer learning is all you need for few-shot classification.
but realize significant memory savings only a random subset of these images which we show is an
Introduction
Meta-learning approaches to few-shot classification are very computationally efficient. Once meta- trained, they learn new task test time with few 1-5 optimization steps [1] 2 single forward pass through the model (3/5] and with minimal or no contrast, transfer learning approaches based on fine-tuning typically rely large pre-trained feature extractor, and instead take 100s-1000 1000s of optimization steps at test time order to learn a task [6], thus incurring high computational cost for each new task encountered. This makes meta-learned solutions attractive in compute compute-constrained deployments, or scenarios where the model must learn multiple different tasks update on-the-fly (e.g. in continual and online learning settings 17)10).
tuning. In
However, a crucial barrier to progress is that meta-learning approaches are memory-intensive to train and thus cannot easily leverage large images for performance boost, recent fine-tuning approaches have done. This limitation arises because a meta-learner must back mack-propagate through all
Authors contributed equally
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Figure 1: LITE enables meta-learners to be trained on large images with GPU thereby significantly improv- ing performance while retaining their test time computational efficiency. The schematic shows test time efficiency (the number of steps and number of Multiply- Accumulate operations (MACs) needed to learn a new task at test time) and whether the method can be trained on large images (required for good performance). Exist- ing meta-learners are cheap to adapt but trained on small images (multiple GPUs are required for large images), transfer learning methods are expensive to adapt but trainable on large images. Meta- learners with LITE get the best of both worlds. Note: SC + LITE is Simple CNAPs trained with LITE.
Meta-Learnine with small images
Meta-Learnine with large images via LITE (single GPU)
Meta-Learning with large images (multi-PPU)
Transfer Learning with large images
of MACs to Adapt (lower better)
High
SUR [22]
BiT [6]
MD-Transfer [13]
Medium
CTX [24]
Proto-MAML [13]
MAML [1]
SC LITE
SC [5]
Low
ProtoNets LITE
ProtoNets [3]
Single-step
Few-steps
Multi-step
Stny
of Sleps Adapt (lower is better)
the examples in task's 's support set (i.e. the task's 's training set) that contribute to a prediction on query example. In some cases, this can be as many as 1000 images (TT). As a result, the amount of memory required for the computational graph grows linearly with the number of support images, and quadratically with their dimension. In contrast, transfer learning approaches can employ standard batch processing techniques to scale to larger images when under memory constraints feature which has contributed significantly their recent success on few-shot benchmarks [I].
Current solutions for training meta-learners on large images include 1) parallelizing the model across multiple GPUs, which may not be available or convenient, considering tasks with fewer checkpointing methods [12 which incur longer training times and still fall short the task sizes required in key benchmarks, Instead, most existing work [1 2 [13 4 [4] has opted for training on large tasks but small images which translates poorly into real-world applications and limits competitiveness on few-shot benchmarks.
support images, and 3) employing gradient/activattn
In this work, we improve on these alternatives by proposing LITE, a Large Image and Task Episodic training scheme for meta-learning models that enables training with large images and large tasks, on a single GPU. We achieve this through the simple observation that meta-learners typically aggregate a task's support examples using a permutation invariant operation. This structure ensures invariance to the ordering of the support set. Consequently, the gradients for a task can be decomposed as a sum of gradient contributions from the task's support examples. This enables to perform forward pass on a task's i entire support set, but realize significant savings in memory by back propkaarpgggting only random subset of these images which we show is an unbiased approximation of the true gradient. Fig. | Iillustrates the key trade-offs, with LITE enabling meta-learners to benefit from large images for improved classification accuracy but still remain computationally efficient at test time.
We use LITE to train key meta-learnin methods and show that our best performing instantiation Simple CNAPs [5] with LITE achieves oftthe----e--tttttt results relative to all meta-learners
two challenging few-shot benchmarks: VTAB+MD [1I] an extensive suite of both meta-learning and transfer learning tasks, and ORBIT [14], an object recognition benchmark of high-variation real-world videos. Our results showcase the unique advantage of meta-learning methods that when properly trained they can be competitive with transfer learning approaches in terms of accuracy for a fraction of the computational cost at test time and they serve as a counterpoint to the recent narrative that transfer learning is all you need for few-shot classification.
Our contributions
1. LITE, general and memory -ffrcieet episodic training scheme which enables meta-learning models to be trained on large images and large tasks on a single GPU.
2. A mathematical justification for approximating the true gradient with random subset of
task's support examples which applies to common classes of meta-learning methods.
3. Instantiations of LITE on key classes of meta-learners to demonstrate its versatility.

---

4. State-of-the performance using Simple CNAPs with LITE compared to other leading meta-learners on two challenging few-shot benchmarks, VTAB+MD and ORBIT
2 Why Meta-Learning with Large Images and Tasks is Difficult
Meta-learning preliminaries In few-shot image classification, the goal is to recognize new classes when given only few training (or support) images of each class, Meta-learners typically achieve this through episodic training []]. Here, an episode or task 7 contains a support set D = { y, ) where (ær, y) is an image-label pair, N, is the number of
and a query set 2 = (a (labeled) support elements given to learn the classes, and M, is the number of query elements requiring predictions. Note that in given task, elements in 2 are drawn from the same set of classes as the elements in Dz. For brevity we may use the shorthand D3 = { y} and Do = {x",""}
During meta- meta-training, meta-learne is exposed to large number of training tasks {r). For each task r, the meta-learner takes as input the support set Ds and outputs the parameters of classifier that has been adapted to the current task 0 = (s((s)) The classifier can now make task-specific probabilistic predictions f(x", 0" = 0g (Ds)) for any query input x (see Fig. 2 function f ())) computes the loss between the adapted classifier's predictions for the query imput and the true label y" which is observed during meta-trainn meta-training. Assuming that L. f. and 0& (Ds) are differentiable, the meta-learner can then be trained with stochastic gradient descent by back-propagating the loss and updating the parameters Φ
At meta-test time, the trained meta- reta-leamerr is given set of unseen test tasks, which typically contain classes that have not been seen during meta-traie teta-training. For each task, the meta-learner is given its support set Ds, and is then evaluated on its predictions for all the query inputs x' (Fig. 2 left).
[x
L
{xm
f
L
O
{
H
Ds J....
Dsn
0
H
for in
Figure 2: Left: Canonical meta-learner. Right: Meta-learner with LITE. The red dotted line shows the back-propagated gradients. Refer to Algorithm [
for nomenclature.
Large memory requirements for meta-training The primary bottleneck to using large images (i.e. 2 224 x 224 pixels) in meta-learning approaches the large amount (GPU) memory required to process a task's support set Ds during meta-training. Specifically, the meta-learner (Ds)) must perform a forward pass with a task's entire support set before can pack-propagate the loss for query elements (æ* y)) E Da (and release the computation graph), thus preventing the use of conventional batch processing. The amount of memory required scales linearly with the number of support images N and quadratically with their dimensions. If NT is large (c.g. the recent VTAB benchmark [II requires a task's support set to be as large as 1000 images), memory on a single GPU is thus quickly exceeded for large images.
Note, the number of query elements the task M is not a bottleneck when using large images as the loss decomposes over elements of the query set DD and is therefore amenable to bini-bbtchhin, By contrast, as the classifier itself is non-linear function and so it is not obvious how to apply similar ideas to allow scaling of DDg in principled way.
support set, the loss does not decompose
Current ad hoc solutions this problem are: (i) parallelize the meta-learner across multiple GPUs which may not be convenient or available and can involve significant engineering effort; (ii) train on tasks with smaller (or sub-sampled) support sets which may adversely affect performance tasks with more classes and/or large numbers of samples per class; (iii) train on tasks with smaller images (e.g. 84 84 pixels in minlImageNet
test
[16) which limits performance and translates poorly to many real-world applications; or (iv) trade memory usage for additional computation [12 by
"Source code for ORBIT experiments is htw.ltaoooooooo/gooooaooooooooor -ataset
and for the VTAB+MD experiments at httttps: thtp://gttttttute cow/ www.acanbbrideeaee LLITE

---

Algorithm LITE for meta-training task
Require:
Ds: task support set; Do: task query set; N: number of support examples in Ds; H:
number of elements in Dg back-propagate; M: number of query examples in Do: My: batch size for Do; backward () function to back-propugate loss; step () function to update parameters with a gradient step.
1: B MMM))
number of query batches
2: for all E 1...., do
3:
Dq, t- fmmm Fm m=1
M
> get query batch from Do >> H. to back pack-propagate
4:
H. 1 { Uns Mi where (nAln= ~ N)
5:
H < DsnH
> H to not back-propagate
6:
DS. + HUH
7:
0T t (Ds.))
8:
LB Mr. N.Me Llym I (x )))
get loss of query batch > back-propagate loss on query batch
9:
backward(Li)
10: end for
Φ s N/H)
update using weighting factor N/H
employing activation/gradient checkpointing (i.e. during training store only a subset of intermediate activations in a network needed for backpropagation and recompute the rest with additional forward computations when needed) which allows for training on larger tasks at the expense of training time, but still falls well short of the memory needed accommodate the task sizes required for key benchmarks (e.g. VTAB+MD).
Although training meta-learners has large memory requirements, meta-testing is generally memory efficient, requiring only a small number of gradient operations, or none at all, compared to transfer learning approaches that would perform large numbers of gradient-based based updates at test time.
3 Large Image and Task Episodic (LITE) training
In this section, we introduce our general and memory-efficient solution for training meta-learners episodically tasks with large support sets and large images. We call our approach Large Image and Task Episodic training or LITE. In Section 3.1 we describe how LITE can be applied to key classes of meta-learners.
Approach The fundamental idea underlying LITE to perform a forward pass using the entire support set Ds, but to compute the gradient contribution only small random subset of the examples in the support set. By doing this, we realize large savings in memory that includes gradicnts, activations, and the computation graph for the elements of Dy that not back prkpppaaagatdd This is an approximation of the true gradient that would result if mack-propagation was performed on all of the examples in Dg. In the following section, show that this approximation is unbiased estimate of the true gradient. The approach for general meta-leamer is detailed in Algorithm [ and shown diagrammatically in Fig. 2
Mathematical justification expected loss over all tasks.
The parameters of the meta-learner & are found by minimizing the
T M,
argmin
Σ C (yTT,
(x 0z (D))))
(I)
In most meta-learning approaches, the support set enters into the loss through sum over the N individual contributions from each data point it contains. This structure enables the meta-learners to be invariant the ordering of the support set and allows all members of the support to the adapted parameters (unlike altermative permutation invariant operators like max or min). Below show in blue how this arises in popular brands of meta-learners.
contribute
In amortization methods (e.g. CNAPS and VERSA (7),, the aggregation of support set points built in directly via deep set encoder Ebi (-). This encodes the support set into embedding vector

---

which is mapped to the classifier parameters by a hyper- Eup (-).
0Φ (Ds) goo Σe cpp (I Vn)
(2)
In gradient-basee based methods (e.g. MAML D. the classifier parameters are adapted from an initial value Φn using sum of derivatives an inner-loop loss computed for each data point in the support set. The derivatives play the role of the deep set encoder in amortization methods.
N Σ
(D(ss) Φ0 + Φ1
Cymerryrrs f (In, x)) dep
(3)
4=
Metric-based methods (e.g. ProtoNets (3) comprise body formed of feature extractor and a head formed from distance-based classifier. The classifier's body parameters are not adapted in task $.c (Ds) Φo. The classifier's head is adapted by averaging the activations for each class the support set to form prototypes. Letting ke denote the number of support examples of class the adapted head parameters are given by
specific way 0 (hoddy
Re
(head) 0.0
(Ds)
f(z 000
1 No
=c) 60).
Ke
Query points can then be classified using their distance from these prototypes d d(f(x,, do). 0 (
We have established that in many meta-learners, each support set affects the classifier parameters and therefore the loss through a sum contributions from each of its elements. We now focus on the consequences of this structure on the gradients of the loss with respect to the meta-learner's parameters. To reduce clutter, we consider the contribution from just a single query point from a single task and suppress the dependence of the loss on the classifier and the query data point, writing
Aly",((* (+ = L (cs(Ds) where ca
Ca Vnn
(5)
As consequence of the summation, the derivative of the loss is given by
(e(1))
de (n) &
d a Cles(Ds)) (De (((((D)))((((((Ss))) dΦ
dl(e)) de (Dea(Ds)
Σ
where C'(cs(Ds))
dØ
which is product of the sensitivity of the loss to the encoding of the data points and the sensitivity of the contribution to the encoding from each data point w.r.t. the meta-learner's parameters. This second term is the source of the memory overhead when training meta-learnent but importantly, it can be rewritten as an expectation w.r.t. a uniform distribution over the support set data-point indices,
de (n)
d ((e)(Ds) = N L' (c{((D))) En-U(L,N)
(7)
dø
dø
We can now define the LITE estimator of the loss-derivative by approximating the expectation by Monte Carlo sampling H times.
Σ de (n.) d -6 L(es(D))) where {mn) UI, (8)
N dΦ ((es ~ C(e((D)))
A=1 dΦ
This estimator is unbiased, converging to the true gradient H - oo. The estimator does not simply involve subsampling the support set - parts of depend on all the support set data points Dş- and this is essential for it to be unbiased. The expectation and variance of this estimator are
dc dΦ
dc
dc dΦ
N2 (C')² V (m.)-U(LN)) H
+ de (na)
B( (UL.N)
and V(Ra)-u((L)) dΦ
dΦ
In Section 5.3 we empirically show that the LITE gradient estimate is unbiased and that its standard deviation is smaller than that of the naive estimator formed by sub-sampling the full support set. LITE provides memory savings by subsampling H examples from the support set, with H < N, and back proppprrpgggaiig only them. Crucially, a forward pass still performed with the complementary set of points, with cardinality N H, but these are not back propaatd...
5

---

3.1 Applying LITE to key meta-learning approaches
To demonstrate its versatility, we now describe how to apply LITE to models within some of the main classes of meta-learnes: CNAPs A and Simple CNAPs [5] for amortization- based methods and ProtoNets (3] for metric-based methods. Note, these are a few possible instantiations. L.ITE can be applied to other meta-learning methods in a straightforward manner.
In the descriptions below, we consider just one query batch Dq. (i.e. one iteration of the for-loop in Algorithm!]] []. Note. in practice, whenever H. is passed through a module, pack-propagation is enabled, while for H., pack-propagation is disabled? Furthermore, since typically [7 < (72), we can forward H. in a single batch, however, we need to split H. into smaller batches. Since H. does not require gradients to be computed, this can be done without significant impact memory.
CNAPS 4 Simple CNAPs 55 + LITE (Appendix A.1) CNAPS variants are amortization- based methods whose hyper-networks take a task's 's support set as imput and generate FiLM layer [[8] parameters which modulate fixed feature extractor. The classifier head can also be generated (CNAPs [4) or adopt a metric-based approach (Simple CNAPs (5]), thus both variants can be adapted with just single forward pass of the support set at test time. Meta-training them with LITE involves passing H. and then H. through their set-encoder E and then averaging all the low-dimensional embeddings to get an embedding for the task. The task embedding is then input into set of MLPs which generate FiLM layer parameters. H. is passed through this configured feature extractor, followed by H. to get the task-adapted features for all support examples in Ds,, For CNAPS, the task-adapted features of H. and H are pooled by class and fed into second MLP which generates the parameters of the fully-connected classification layer. For Simple CNAPs, the task-adapted features of H and H. are instead used compute class-wise distributions (i.e. class mean and covariance matrices). With back-propagaan propagation enabled, the query batch Dq, is then passed through the conffuiigr feature extractor and classified with the task-configured classifier (for CNAPs), or with the Mahalanobis distance [19] to the class-wise distributions (for Simple CNAPs). The query batch loss is computed, and only back- proopppppggtted for H. Note that the feature extractor pre-trained and frozen, and only the parameters of the set-encoder and generator MLPs are learned.
ProtoNets 3 LITE (Appendix A.2) ProtoNets [3] is a metric-based approach which computes a set of class prototypes from the support set and then classifies query examples by their (e.g. Euclidean) distance to these prototypes. Like CNAPs variants, it requires only single forward pass to learn a new task. Meta-training ProtoNets with LITE involves passing H. through the feature extractor with baok-propag enabled, followed by H with back-propagation disabled, to obtain features for all support examples Ds.- These features are averaged by class to compute the prototypes such that (with back proppagatogntonn enabled) the query batch Ds, can be passed through the feature extractor and classified based on the Euclidean distance. The loss of the query batch is computed and only back propagated for H. Note that here all the parameters of the feature extractor are learned.
4 Related work
We review the two main approaches to few-shot learning: transfer learning methods which are easy to scale to large images but are costly to adapt at test time, and meta-learning methods which are harder to scale but cheap adapt (see Fig. I. Note, we do not cover methods already described above.
Transfer learning approaches have demonstrated state-of-the-ar performance on challenging few-shot benchmarks [.. However, they incur high computational cost at test time as they rely on large (pre-trained) feature extractors which are fine-tuned with many optimization steps. MD-Transfer [13] fine-tunes all the parameters in a ResNetl 8 feature extractor with cosine classifier head for 200 optimization steps. BiT 66] fine- tne-ttnes& a feature extractor (pre-trained on 300M images in the JFT- 300M dataset (2l)) with linear head, in some cases for up to 20,000 optimization steps to achieve its state-of-thed ti--hhe-rttt results on the VTAB (0 benchmark. SUR 22 instead trains 7 ResNet-50 feature extractors, one for each training dataset. At test time, the predictions from each are concatenated and fine-tuned to minimize a cosine-similarity loss. All of these approaches involve on the order of teras to petas of Mulinply-Accuulla
operations (MACs) to learn a single new test task, and this must
'In PyTorch, this can be achieved by setting torch. enabbll enabled True when passing H. and torch. grad. enabled False when passing H.

---

be repeated for each new task encountered. Furthermore, for each new task type, transfer learning approaches may need to be tuned on a validation set obtain the optimal hyper-parameters.
In comparison, meta-learning approaches 23 generally require orders of magnitude fewer MACs and
steps to learn new task at test time. Popular approaches include CNAPs 4 Simple CNAPs [S]
ProtoNets [3. and MAML [1] and are discussed in Section [.1] Others include ProtoMAML [13
which fuses ProtoNets and MAML by initializing the classifier weights with the prototypes and then, like MAML, takes a few optimization steps to tune the weights to the task. It therefore incurs a similar cost to adapt as MAML, except it must additionally compute the prototypes. Finally, CTX replaces the final average pooling layer of ProtoNets with a transformer layer that generates series of prototypes which are spatially aware and aligned with the task. Like CNAPs, it requires just single forward pass, however, requires more MACs to adapt since it uses a larger feature extractor (ResNet-34) and an attention module. CTX one of the few meta-learning approaches that has been meta-trained on 224 x 224 images, but requires days of training on 8 GPUs.
Memory efficient variants of MAML have been developed. First-order MAML avoiding the estimate of second-order derivatives. This is also done in Reptile 25 that additionally avoids unrolling the computation graph, performing standard gradient descent each adaptation step. Implicit MAML [26), decouples the meta-gradient from the inner loop and is able to handle many gradient steps without memory constraints. In addition [27], proposes methods to reduce the computation overhead of meta- veea-training MAML for large tasks. Note that, in all these cases, savings arise from working around the limitations of MAML, while LITE more general.
M saves memory by
5 Experiments
In this section, we demonstrate that meta-learners trained with LITE achieve state-of-the perfor- mance among meta-learners on two challenging few-shot classification benchmarks: (i) ORBIT [14
which is real-world few-shot object recognition dataset for teachable object recognizers; and
(ii) VTAB+MD [[] which is composed of the Visual Task Adaptation Benchmark (VTAB) [20] and Meta-Dataset (MD) [ and combines both few-shot and transfer learning tasks. We compare LITE meta-learners with state-of-the-an meta-learning and transfer learning methods in terms of classification accuracy, computational cost/time to learn task, and number of model parameters.
5.1 ORBIT Teachable Object Recognition Benchmark
Training meta-learners on large images with LITE achieves state-of-the-aar accuracy with low test time adaption cost on ORBIT. Results are reported as the average (95% confidence interval) over 85 test tasks (5 tasks per test user, 17 test users). I is image size. is model trained with/without LITE. RN-18 18 is ResNet- 8. EN-BO is EfficientNer-B
Table 1:
is x10 MACs. F forward pass. FB is forward-backward pass. Time is average wall clock time per task in seconds.
Clean Videos
Clutter Videos
Test-time adaption
MODEL
FRAME
FRAME VIDEO 1
MACS
STEPS
TIME
69.5 (2.2)
79.7 (2.6)
(1.7
(2.4
50FB
11.17M
FineTuner
28
224 RN-18
72.2 (2.2)
(2.6)
56.7 (2.0)
(2.5
546.57T
50FB
96.236
11.18M
EN-BO
(2.0)
(2.3)
63.1 (1.8)
66.9 (2.4)
12L.2T
50FB
139.99s
4.01M
84 RN 18
70.6
80.9 (2.6)
51.7
(2.5
95.31T
15FB
36.98s
11.17M
MAMI.
RN-18
75.7
86. (2.3)
(1.9)
15PB
224 EN BO
(1.99
(2.5)
64.6 (1.9)
69.4 (2.3)
36.31T
15FB
117.89s
4.01M
RN-18
(2.2
81.9 (2.5)
50.3 (1.7)
59.9
3.8TT
1F
0.73s
ProtoNets H
224 R8 LITE
76.7 (1.9)
86.4 (2.2)
61.4 61.4 (1.8)
68.5 (2.4)
5.47T
IF
1.07%
11. 18M
LITE
(1.7)
(1.9)
(1.8)
(2.3)
4.01M
84 RN-18 LITE
62.2.22.1)
(2.6)
(1.8)
(2.5
348T
IF
0.98s
12.75M
CNAPs 4
84.9 (1.9)
(1.9)
(2.5
1F
2.118
224 EN-BO
LITE
79.6
(2.6)
(1.9)
69.2 (2.3)
3,38T
IF
2.85s
10.59M
RN-18
(2.3
(2.5
(1.8)
(2.5)
IF
1,01s
11.97M
Simple CNAPs 3
224 RN-18
LITE
76.5 (2.0)
86.4 (2.2)
57.5 (1.9)
64.6 (2.4)
7.64T
1F
2.14g
11.97M
EN-BO
LITE
(1.7)
(1.8)
(1.9)
71.9 (2.3)
3.39T
1F
2.92s
5.67M
ORBIT is highly realistic few-shot video dataset collected by people who are blind/low-vision. It presents an object recognition benchmark task which involves personalizing (i.c. adapting)
7

---

recognizer to each individual user with just few (support) videos they have recorded of their objects. To achieve this, the benchmark splits data collectors into disjoint train, validation, and test user sets along with their corresponding objects and videos. Models are then meta meta-trained on the train users, and meta-tested on how well they can learn a test user's objects given just their videos (on a user-by-user basis). The benchmark has two evaluation modes: how well the meta-trained model can recognize a test user's objects in clean videos where there is only that object present, and in clutter videos where that object appears within a realistic, multi-object scene.
Experiments We meta-train ProtoNets [3]. CNAPs and Simple CNAPs [S] with LITE on tasks composed of large (224 x 224) images. We also meta-train first-order MAML on large images as baseline. Since first-order MAML can process task support sets batches, we simply reduce the batch size and do not need to use LITE. We compare all of the above to meta-training on tasks of small (84 x 84) images (i.e. the original baselines (4).. We also include transfer learning approach, Fine'Tuner (28]. which freezes a pre-trained feature extractor and fine-tunes just the linear classifier for 50 optimization steps. For each model, we consider a ResNet-18 (RN-18) and EfficientNNtt-O0 (EN-BO) feature extractor, both pre-trained on ImageNet [29]. We follow the task sampling protocols described in [14] (see Appendices B and C.I for details). We also include analyses on meta-training with small tasks of large images in Appendix D.3.
Results In Table we report frame accuracy and video accuracy, averaged over all the query videos from all tasks across all test users (17 test users, 85 tasks in total), along with their corresponding 95% confidence intervals. We also report the computational cost to learn a new task at test time in terms of the number of the wall clock time to adapt in seconds. See Appendix C.I for results on additional metrics. The key observations from our results are:
operations (MACs), the number of steps to adapt, and
Training on larger (224 x 224) images leads to better performance compared to smaller (84 x 84) images. The boost is significant for both clean and clutter videos, though absolute performance remains lower on clutter videos. This suggests that object detection or other attention based mechanisms may be required to further exploit large images in more complex/cluttered scenes.
All meta-learners + LITE set new state-of-the- on clean videos, and perform competitively with the FineTuner on cluttered videos, using an EfficientNet-BO backbone.
Meta-learners are competitive with transfer learning approaches in accuracy but are almost two orders of magnitude more efficient in the number of MACs and the time to learn a new task, and one order of magnitude smaller in the number of steps to adapt.
5.2 VT
VTAB+MD | combines revised versions of the Meta-Dataset (MD) and VTAB datasets and is one of the largest, most comprehensive, and most challenging benchmarks for few-shot learning systems. The MD-v2 part of the benchmark involves testing on 8 diverse datasets while VTAB-v2 involves testing on 18 datasets grouped into three different cataegories (natural, specialized, and structured).
Results Fig. 3 compares our best meta-learner, Simple CNAPS + LITE, with 6 other competitive approaches on VTAB+MD: BłT, MD-Transfer, and SUR are transfer learning based methods while ProtoMAML. ProtoNets, and CTX are meta-lcarning based. Note, comparisons are not always like- for-like as methods use differing backbones, image sizes, and pre-training datasets. Appendix D. D.2 provides this information along with the precise numbers and 95% confidence intervals. Appendices B and C.2 summarize all implementation and experimental details, and Appendix D.3 includes further analyses on the impact of task size meta-traing treiiin.. The key observations from our results are:
On MD-v2, Simple CNAPS + LITE has the highest average score and sets new ttate-of-the-art. On VTAB-v2, BiT scores highest overall, but among meta-learners, Simple CNAPs + LITE is the best overall and on the natural and specialized sections. It falls short of CTX on the structured section due to poor performance on dSprites which involves predicting the position and orientation of small white shapes on black background a task quite different to image classification.
These results are significant given that CTX takes 7 days to train on 8 GPUs, whereas Simple CNAPS + LITE trains in about 20 hours on single 16GB GPU. In addition, Simple CNAPs LITE uses a relatively small pre-trained backbone (4.0M parameters) compared to SUR's pre-trained ResNet-50s (one for each MD-v2 training set, plus ImageNet).
8

---

MD-Transter
SUR
ProtaNets
ProtoMAML
CTX
SC(84)
SC-UTE
80
70
60
Accunecy
40
30
MD-2
VTAB
such as VTAB (spacialized
VTAB
Figure 3: Summary of results on VTAB+MD. Simple CNAPs with LITE (SC LITE, black bar) trained on 224 x 224 images achieves state-of-the-art accuracy on Meta-Dataset (MD-v2), and state-of-the-an accuracy among meta-learners on 3 of 4 parts of VTAB. As reference, we have included transfer learning methods (red bars), other meta-learning methods (blue bars), and Simple CNAPs without LITE trained on small images (84 x 84, gray bar). Competitive results from [1] See Table D.2 for tabular results on individual datasets.
Simple CNAPS LITE using 224x224 images significantly outperforms Simple CNAPS using 84x84 images, except for when the dataset images are small (e.g. Omniglot, QuickDraw, dSprites). This demonstrates that using large images and approximation to the support set gradients achieves superior results compared to using small images with exact gradients.
MD-Transfer and BiT perform strongly across VTAB+MD, however, Simple CNAPS + LITE is significantly faster to adapt a new task requiring only forward pass of the support set with no pperrparreettrr tuning whatsoever. The transfer learners instead perform 100s of optimization steps adapt new task and may require a human-in the-loop to tune the learning rate and number of optimization steps.
5.3 Effect of varying [H]]
Table 2 shows the effect of varying [H], the number of examples brok-prop per task, on the VTAB+MD benchmark. Performance consistent across different [H]] an expected result since LITE provides an unbiased estimate of the true gradient. For both Simple CNAPs and ProtoNets + LITE, the results at the lowest values of |H| are respectable, though they fall short of what can be achieved at [7|| 40. Thus, the support gradient information does improve the solution, albeit by only 1-2
0.006
LITE
Subsampled Small Tasks
Average RMSE
0.004
0.003
0.002
percentage points. Note, report the lowest setting as
[H] = 1 for Simple CNAPs but [ = 0 for ProtoNets. This is because Simple CNAPs's adaptation network (which processes the support set) shares no parameters with the feature extractor and thus will not be learned the support gradients are completely ignored. On the other hand, ProtoNets' adaptation network shares all its parameters with the feature extractor, thus can be meta- learned even when the support gradients are neglected. Finally, in the two rightmost columns, we compare the classification accuracy for | = |Dsl (i.e.
0.000
10
20
30
40
50
60
70
80
90
[H1]
Figure 4: Average Root Mean Square Error (RMSE) w.r.t. the true gradients versus HH| for LITE and sub-sampled tasks on 84 x 84 images (10-way, 10-shot, |Dsl = 100).
using the full support set gradient) to | = 40 (i.e. using LITE). Due memory constraints, we do this at image size 84 x 84. Here we see that the difference in accuracy is significant. We expect that performance will smoothly interpolate as |H] is increased from 40 the size of the largest support set (at which point the full gradient is computed). This validates how LITE can be used to off GPU memory usage for classification accuracy by varying [7]]. Furthermore, we conduct an empirical analysis (see Table D.7) which shows that the LITE gradient estimates and the gradients when using smaller sub-sampled tasks are unbiased w.r.t. the true gradients. However, Fig. shows that LITE offers a significantly lower root mean square error w.r.t. the true gradients compared to using sub-san sampled tasks at all but the highest values of [P]. Refer to Appendix D.4 for additional details.

---

Table 2: Classification accuracy results in percent on VTAB+MD using Simple CNAPs and ProtoNets
with varying values of | Image sizes are 224 x 224 and 84 x 84 pixels. For [7] > 40, we used gradient/activation checkpointing methods [2] in addition to LITE. For full results see Appendix D.4
Model
Simple CNAPs
ProtoNets
Simple CNAPs
Image Size
224 224
224 x 224
84 x 84
[H]
10
20
30
40
100
10
20
30
40
40
Del
MD-v2
72.8
73.7
73.3
73.8
73.9
74.3
71.0
72.0
72.0
72.5
72.7
63.6
68.4
VTAB (all)
51.2
51.0
50.5
51.1
51.4
51.2
45.1
45.8
46.2
46.2
46.1
42.7
44.7
VTAB (natural)
64.5
65.3
64.1
65.8
65.2
66.0
58.5
60.0
60.6
60.8
60.9
47.7
49.5
VTAB (specialized)
71.8
71.4
70.5
71.3
71.9
71.6
63.9
64.2
64.5
64.2
61.0
63.8
VTAB (structured)
31.0
30.0
30.3
29.9
30.8
29.9
26.0
26.2
26.4
26.1
25.9
29.9
31.7
Discussion
We propose LITE, general and memory-efficient episodic training scheme for meta-leamnrs that enables them to exploit large images for higher performance with limited compute resources. LITE's significant memory savings come from performing a forward pass on a task's full support set, but back-propagee propagatinn only a random subset, which we show is an unbiased estimate of the full gradient. We demonstrate that meta- meat-lesmerrs trained with LITE are state-of oftee-of-hhe-rr among meta-learners on two challenging benchmarks, ORBIT and MTAB+MM,, and are competitive with transfer learning approaches at fraction of the test time computational cost.
This offers a counterpoint to the recent narrative that transfer learning approaches are all you need for few-shot classification. Both classes of approach are worthy pursuits (and will need to exploit large images in real-world deployments) but careful consideration should be given to the data and compute available test time determine which class best suited to the application under consideration. it involves learning just single task type (e.g. classifying natural images) with ample data and no compute or time constraints, then a fine-tuning approach would suffice and perform well. However, multitude of task types will be encountered at test time, each with minimal data, and new tasks need be learned on anoourc-cconsrainge devices (e.g. a mobile phone or a robot) or quickly/repeatedly (e.g. in continual or online learning settings), then meta-learning solution will be better suited.
Finally, the machine learning community grapples with greener solutions for training deep neural networks, LITE offers a step in the right direction by allowing meta-learners to exploit large images without an accompanying increase in compute. Future work may look toward applying the basic concept of LITE to other types of training algorithms to realize similar memory savings.
Limitations As discussed in Section 3 LITE can be applied
wide range of meta-learners provided that they aggregate the contributions from task's 's support set via a permutation inntuatttionninaiiiatt operation like sum. Because only subset of the support set is pack-propagated, however, the gradients can be more noisy and meta-training may require lower learning rates. Furthermore, LITE memory-efficient scheme for training meta-learners episodically and has not been tried with meta-learners trained in other ways (e.g. with standard supervised learning) or non-image datasets.
Societal impact Few-shot learning systems hold much positive potential from personalizing object recognizers for people who are blind [14] to rendering personalized avatars [30] (see [23 for a full review). These systems, however, also have the potential to be used in adverse ways - for example, in few-shot recognition in military/surveillancee applications. Meta-trained few-shot systems may also pose risks in decision making applications as uncertainty calibration in meta-learning models has not yet been extensively explored. Careful consideration of the intended application, and further study of uncertainty quantification in meta- meta-learning approaches will be essential in order to minimize any negative societal consequences of LITE if deployed in wolld applications.
Acknowledgments
We thank the anonymous reviewers for key suggestions and insightful questions that significantly improved the quality of the paper. Additional thanks go to Vincent Dumoulin for providing the Table D.2.
tabular results for SUR used in Fig. Table
10

---

Funding Transparency Statement
Funding in direct support of this work: John Bronskill, Massimiliano Patacchiola and Richard E. Turner are supported by an EPSRC Prosperity Partnership EP/T005386/1 between the EPSRC, Microsoft Research and the University of Cambridge.
References
[1] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In Proceedings of the 34th International Conference on Machine Learning (ICML), pages 1126-1 1135, 2017.
[2] Luisa M. Zintgraf, Kyriacos Shiarlis, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning In Proceedings of the 36th International Conference on Machine Learning (ICML). 2019.
[3] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Proceedings of the 31 Annual Conference on Neural Information Processing Systems (NeurlPS), pages 4077-4087 2017.
[4] James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E Turner. Fast and flexible multi-tast classification using conditional neural adaptive processes. In Pro- ceedings of the 33rd Annual Conference on Neural Information Processing Systems (NeurIPS), pages 7957-7968, 2019.
[5] Peyman Bateni, Raghav Goyal, Vaden Masrani, Frank Wood, and Leonid Sigal. Improved few-shot visual classification. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 14493-14502, 2020.
[6] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. arXiv preprint 1911v:1912.11377 6(2):8, 2019.
[7] Mark B Ring. CHILD: A first step towards continual learning. Machine Learning, 28(1): 77-104, 1997.
[8] David Saad. On-line learning in neural networks. Number 17. Cambridge University Press, 2009.
[9] Steven CH Hoi, Doyen Sahoo, Jing Lu, and Peilin Zhao. Online learning: comprehensive survey, arXiv preprint
2018
[10] Antreas Antoniou, Massimiliano Patacchiola, Mateusz Ochal, and Amos Storkey. Defining benchmarks for continual few-shot learning. arXiv preprint arXi::2004.11961 2020.
[11] Vincent Dumoulin, Neil Houlsby, Utku Evci, Xiaohua Zhai, Ross Goroshin, Sylvain Gelly, and Hugo Larochelle. Comparing transfer and meta learning approaches on a unified few-shot classification benchmark. arXiv preprint arXiv:2104.0222 104.02638. 2021.
[12] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint 1604..1604.00 06174,16147744 2016.
[13] Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta-Dataset: Dataset of Datasets for Learning to Learn from Few Examples. In Proceedings of the 8th International Conference on Learning Representutions (ICLR), 2020.
[14] Daniela Massiceti, Luisa Zintgraf, John Bronskill, Lida Theodorou, Mathew Tobias Harris, Edward Cutrell, Cecily Morrison, Katja Hofmann, and Simone Stumpf. ORBIT: A Real- World Few-Shot Dataset for Teachable Object Recognition. In Proceedings of the IEFF/CVF International Conference Computer Vision (ICCV), 2021.

---

[15] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. Proceedings of the 30th Annual Conference on Neural Information Processing Systems (NeurlPSS, pages 3630-3638, 2016.
[16] Sachin Ravi and Hugo Larochelle. Optimization model for few-shot learning. In Proceed- ings of the 5th International Conference on Learning Representations (ICLR), 2017.
[17] Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard Turner. Meta-learning probabilistic inference for prediction. Proceedings of the 7th International Conference Learning Representations (ICLR), 2019.
[18] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual reasoning with a general conditioning layer. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI), 2018.
[19] Prasanta Chandra Mahalanobis. On the generalized distance in statistics. In Proceedings of the National Institute of Science of India, volume 12, pages 59,55, 1936.
[20] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study representation learning with the visual task adaptation benchmark. arXiv preprint arXiv: 1910.04867, 2019.
[21] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 843 852, 2017.
[22] Nikita Dvomik, Cordelia Schmid, and Julien Mairal. Selecting relevant features from a multi- domain representation for few-shot classification. In Proceedings of the European Conference on Computer Vision (ECCV), pages 769-786, 2020.
[23] Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey, arXiv preprint arXiv:2004. 05439, 2020.
[24] Carl Doersch, Ankush Gupta, and Andrew Zisserman. CrossTransformers: spatially-aware few-shot transfer. arXiv preprint arXiv:2007.11499, 2020.
[25] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint
2018
[26] Aravind Rajeswaran, Chelsea Finn, Sham Kakade, and Sergey Levine. Meta-learning with implicit gradients. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2019.
[27] Jaewoong Shin, Hae Beom Lee, Boqing Gong, and Sung Ju Hwang. Large-scale meta-learning with continual trajectory shifting. In Proceedings of the International Conference on Machine Learning (ICML), 2021.
[28] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Proceedings of the 28th Annual Conference on Neural Information Processing Systems (NeurIPS), pages 3320-3300 3328, 2014.
[29] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li. and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
[30] Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky. Few-shot ad- versarial learning of realistic neural talking head models. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 9459-9468 2019.
[31] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
12

---

[32] Mingxing Tan and Quoc Le. EfficientNet: Rethinking model scaling for convolutional neural networks. Proceedings of the 36th International Conference on Machine Learning (ICML), pages 6105-61 14, 2019.
[33] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 4510-4520, 2018.
[34] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3)211--55)) 2015. doi: 10. 1007/s 123-01
[35] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR), 2015.
[36] TensorFlow Datasets, collection of ready-to-use datasets. https://wwwt www.stttssoro.o......o.o
datasets, 2020.
13


Reviews:
[{"review_id": "XsSWeD80ZE4", "reviewer": "Reviewer_ijTZ", "summary": "This paper considers the problem of training image classification for meta-learning/few-shot learning models with large images. The authors point out that many categories of meta-learning algorithms have difficulty using large images because of memory constraints brought up by computation of gradients computed during meta-training. They decompose 3 major categories of meta-learning models (CNAP, MAML, and Proto-Nets) into a common gradient update that is troublesome w/r/t memory. Their idea to make training more feasible is to use all the support set items in the forward pass but approximate the backward pass using a small random subsample of the examples in the support set. They compare training their versions of memory-efficient, large input meta-learning models against recent work on ORBIT (a few-shot object recognition task) and VTAB+MD (a newer benchmark for few-shot classification that is based on Meta-Dataset and VTAB datasets). On both benchmarks, they show that training using larger images using their proposed method improves performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: the paper is considering an interesting aspect of the meta-learning problem (holding constant the method but using larger input images) that as far as I know hasn't been considered before.\n\nQuality: the proposed method and its motivating analysis seem sound. The experimental comparison is also fair and highlights the benefit of using the large image input based on their proposed approximation. However, I believe the experiments could've been expanded. Why not consider mini-ImageNet or tiered-ImageNet benchmarks in which most work uses 84x84 inputs? It would be interesting to see the benefit of larger inputs on those commonly used benchmarks. Additionally, I think an ablation experiment showing how the quality and speed of training varies with the approximation parameter H would also have been useful to evaluate the method. Lastly, I also think first-order maml with a larger input would be a useful baseline to see the effect of using larger inputs in a different way than what the authors propose.\n\nClarity: the paper is very well-written. It motives the problem well and has a clear description of the proposed solution. The experiments also seem to be conducted fairly and are informative.\n\nSignificance: I think the results in the paper could be useful to researchers as they show a way to improve quality of meta-learning models simply by increasing the image resolution. One concern I have is that the authors include MAML in the class of methods for which the proposed approximation applies but in the end, do not implement or show results for the method since \"...because of the memory required to retain the inner loop’s computation graph, training may still need to be scaled to smaller images or smaller tasks.\" I am wondering if its actually useful to include discussion of MAML then in the paper if the paper doesn't have a concrete proposal/implementation of the approximation of second-order MAML that allows it to be trained on larger images?\f", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "PlqLJuqWSF", "reviewer": "Reviewer_qib5", "summary": "The authors propose a method that enables few-shot learners to be trained on 1 GPU without memory issues. This is very important since most few-shot learners need to process the whole train set of a task before evaluating the loss on the task. They show mathematically that their method gives an unbiased estimation of the true gradient (if all samples in training were back-propagated to update the model). They evaluate their method on CNAPS and ProtoNets and show that it can achieve better results since it can be trained with larger images.  ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written and well-structured. The problem that they try to solve is very important and also very challenging. The idea seems to be generalizable to different meta-learning techniques. They show it for CNAPS and ProtoNets. Their method is based on the idea to update the parameters of the model with the whole train set of the task but backpropagate through only a subset of the task. I think the mathematical study of this really helps the paper. Since this is mathematically correct, it might open new directions for new meta-learning algorithms. \n\nI have two concerns. First, optimization-based algorithms like MAML that require several inner steps are still cannot get much improvement from this method. I looked at the supplementary material and the part that discussed MAML, however, I could not understand why it is not possible to limit the back-propagation on only a few samples from the train set. Can you please explain more about this?\n\nMy second concern is that some other related work like (Meta-learning with implicit gradients) can be used to solve the same problem. It seems that related work did not thoroughly cover other approaches for solving the memory issue in meta-learning. Has there been any other work that tackles this issue? I am aware that it might not be possible to compare directly, however, it is good to discuss them.  \n\n======Post rebuttal======\n\nI thank the authors for clarifying reviewers' questions. I strongly suggest the authors modify the paper based on reviewers' requests. Especially, point 5 in reviewer wfg9's comment and experiments that were done for reviewer ES9N's comments. After discussing with other reviewers and reading their comments, I decided to stick with my initial review score of 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors described the limitations and societal impacts of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "NYMOP-k8qG-", "reviewer": "Reviewer_ES9N", "summary": "Many meta-learning approaches have been trouble with the scalability issue due to the large memory-required computation during the training process. To overcome such a problem, this paper proposes LITE that is memory efficient episodic training scheme allowing meta-learning methods to use the benefit of large-size images of tasks on a single GPU. On two benchmark datasets (ORBIT and VTAB+MB), meta-learning methods combined with LITE outperform meta-learning methods without LITE and Simple CNAPs (meta-learning method) + LITE shows competitive performance compared with transfer learning methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think this paper is nice since it tackles the memory issue that has been a problem for a long time in the meta-learning field and the proposed solution is generally applicable for many meta-learning methods. However, I wish some clarity problems and analysis are improved during the rebuttal period.\n\n\n- Strength\n    - This paper is well written and easy to follow.\n    - This paper tackles an important problem in the meta-learning domain, which is limited scalability due to the memory-intensive computation of training of support sets of tasks.\n    - By overcoming the above issue, this paper achieves the SOTA performance on two few-shot benchmark datasets.\n\n\n- Comments (Weakness)\n\n    (order of importance)\n\n    - Could the authors report memory savings and whether the performance drops for the same image sizes (e.g., 84) when we use LITE?\n    - I wonder how much we can enlarge the image size or task size by using LITE with a single GPU?\n    - In Table 1, could the authors report GPU hours (or Wall Clock Time) at the test time of each method? I wonder how much we can save time when we use the meta-learning methods compared with transfer learning as the authors described in the introduction.\n    - In Table 1, could the authors report the results with 224 EN-B0?\n    - Could the authors add discussion on [1] that is a method for large-scale meta-learning?\n\n    [1] Shin et al. Large-Scale Meta-Learning with Continual Trajectory Shifting, ICML 2021.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please refer to Comments in Main Review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "DYhTcKH_4D8", "reviewer": "Reviewer_wfg9", "summary": "Meta-learning algorithms are not memory friendly since they have to load all the support samples during the inner-loop. When the support data are high-resolution images, or need huge support samples (i.e. 1000), the meta-training is hard to work on a single GPU. As a result, this paper aims to propose an efficient meta-training algorithm which only rely on the gradients of subsets of the support data. And they show the can achieve competitive results against transfer learning approaches but at a fraction of the test-time computational cost on multiple benchmarks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The motivation of the paper is clear and the paper is easy to follow. The proposed algorithm is extremely simple and can be applied to almost all the meta-learners. The results on benchmarks which need large support samples seems promising. It is interesting to make the meta-learners more efficient but I am not sure how significant the problem is. In addition, I think more experiments are needed to show the effectiveness of the proposed algorithm.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1)  Are the performance of the meta-learners sensitive to the choice of H?  Can we have a table or a figure about that, like accuracy v.s. different choice of H. I am interested in how meta-learning performs when H is the full support size or when H=0 (Just detach the gradient of the support samples).\n\n2) Which one is more important to achieve a better performance? A larger support size or the gradient we needed for the support samples? Can we have the results where we have only the maximum number of support samples with 224 * 224 resolution to fit the memory?\n\n3) The experiments only show that the proposed algorithm works, but lacks experiments like how this unbiased estimator performs against the full gradient. In addition, we need to show the gradient of the support samples are important. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is addressing the problem of meta-learning with large-scale images. In this setting, many of the existing algorithms require an intractable amount of memory. The proposed method addresses this challenge by simply sub-sampling the support set in the backward pass. Although the method is sensible, the empirical gain is marginal. All reviewers agree on the merit of the paper to be accepted after an active discussion phase.
Decision: Accept (Poster)