Memory Efficient Meta-Learning with Large Images
Year: 2021

Abstract:
Meta learning approaches to few-shot classification are computationally efficient at test time, requiring just a few optimization steps or single forward pass to learn a new task, but they remain highly memory-intensive to train. This limitation arises because a task’s entire support set, which can contain up to 1000 images, must be processed before an optimization step can be taken. Harnessing the performance gains offered by large images thus requires either parallelizing the meta-learner across multiple GPUs, which may not be available, or trade-offs between task and image size when memory constraints apply. We improve on both options by proposing LITE, a general and memory efficient episodic training scheme that enables meta-training on large tasks composed of large images on a single GPU. We achieve this by observing that the gradients for a task can be decomposed into a sum of gradients over the task’s training images. This enables us to perform a forward pass on a task’s entire training set but realize significant memory savings by back-propagating only a random subset of these images which we show is an unbiased approximation of the full gradient. We use LITE to train meta-learners and demonstrate new state-of-the-art accuracy on the real-world ORBIT benchmark and 3 of the 4 parts of the challenging VTAB+MD benchmark relative to leading meta-learners. LITE also enables meta-learners to be competitive with transfer learning approaches but at a fraction of the test time computational cost, thus serving as a counterpoint to the recent narrative that transfer learning is all you need for few-shot classification.

heading: 1 Introduction
text: Meta-learning approaches to few-shot classification are very computationally efficient. Once metatrained, they can learn a new task at test time with as few as 1-5 optimization steps [1, 2] or a single forward pass through the model [3–5] and with minimal or no hyper-parameter tuning. In contrast, transfer learning approaches based on fine-tuning typically rely on a large pre-trained feature extractor, and instead take 100s-1000s of optimization steps at test time in order to learn a task [6], thus incurring a high computational cost for each new task encountered. This makes meta-learned solutions attractive in compute-constrained deployments, or scenarios where the model must learn multiple different tasks or update on-the-fly (e.g. in continual and online learning settings [7–10]).
However, a crucial barrier to progress is that meta-learning approaches are memory-intensive to train and thus cannot easily leverage large images for a performance boost, as recent fine-tuning approaches have done. This limitation arises because a meta-learner must back-propagate through all
∗Authors contributed equally
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
the examples in a task’s support set (i.e. the task’s training set) that contribute to a prediction on a query example. In some cases, this can be as many as 1000 images [11]. As a result, the amount of memory required for the computational graph grows linearly with the number of support images, and quadratically with their dimension. In contrast, transfer learning approaches can employ standard batch processing techniques to scale to larger images when under memory constraints — a feature which has contributed significantly to their recent success on few-shot benchmarks [11].
Current solutions for training meta-learners on large images include 1) parallelizing the model across multiple GPUs, which may not be available or convenient, 2) considering tasks with fewer support images, and 3) employing gradient/activation checkpointing methods [12] which incur longer training times and still fall short of the task sizes required in key benchmarks. Instead, most existing work [1, 3, 2, 13, 4, 14] has opted for training on large tasks but small images which translates poorly into real-world applications and limits competitiveness on few-shot benchmarks.
In this work, we improve on these alternatives by proposing LITE, a Large Image and Task Episodic training scheme for meta-learning models that enables training with large images and large tasks, on a single GPU. We achieve this through the simple observation that meta-learners typically aggregate a task’s support examples using a permutation invariant sum operation. This structure ensures invariance to the ordering of the support set. Consequently, the gradients for a task can be decomposed as a sum of gradient contributions from the task’s support examples. This enables us to perform a forward pass on a task’s entire support set, but realize significant savings in memory by back-propagating only a random subset of these images which we show is an unbiased approximation of the true gradient. Fig. 1 illustrates the key trade-offs, with LITE enabling meta-learners to benefit from large images for improved classification accuracy but still remain computationally efficient at test time.
We use LITE to train key meta-learning methods and show that our best performing instantiation – Simple CNAPs [5] with LITE – achieves state-of-the-art results relative to all meta-learners on two challenging few-shot benchmarks: VTAB+MD [11], an extensive suite of both meta-learning and transfer learning tasks, and ORBIT [14], an object recognition benchmark of high-variation real-world videos. Our results showcase the unique advantage of meta-learning methods – that when properly trained they can be competitive with transfer learning approaches in terms of accuracy for a fraction of the computational cost at test time — and they serve as a counterpoint to the recent narrative that transfer learning is all you need for few-shot classification.
Our contributions
1. LITE, a general and memory-efficient episodic training scheme which enables meta-learning models to be trained on large images and large tasks on a single GPU. 2. A mathematical justification for approximating the true gradient with a random subset of a task’s support examples which applies to common classes of meta-learning methods. 3. Instantiations of LITE on key classes of meta-learners to demonstrate its versatility.
4. State-of-the-art performance using Simple CNAPs with LITE compared to other leading meta-learners on two challenging few-shot benchmarks, VTAB+MD and ORBIT 2

heading: 2 Why Meta-Learning with Large Images and Tasks is Difficult
text: Meta-learning preliminaries In few-shot image classification, the goal is to recognize new classes when given only a few training (or support) images of each class. Meta-learners typically achieve this through episodic training [15]. Here, an episode or task τ contains a support setDτS = {(xτn, yτn)} Nτ n=1 and a query set DτQ = {(xτ∗m , yτ∗m )} Mτ m=1, where (x, y) is an image-label pair, Nτ is the number of (labeled) support elements given to learn the new classes, and Mτ is the number of query elements requiring predictions. Note that in a given task, elements inDτQ are drawn from the same set of classes as the elements in DτS . For brevity we may use the shorthand DS = {x, y} and DQ = {x∗, y∗}. During meta-training, a meta-learner is exposed to a large number of training tasks {τ}. For each task τ , the meta-learner takes as input the support setDS and outputs the parameters of a classifier that has been adapted to the current task θτ = θφ(DS). The classifier can now make task-specific probabilistic predictions f(x∗,θτ = θφ(DS)) for any query input x∗ (see Fig. 2). A function L(y∗, f(x∗,θτ )) computes the loss between the adapted classifier’s predictions for the query input and the true label y∗ which is observed during meta-training. Assuming that L, f , and θφ(DS) are differentiable, the meta-learner can then be trained with stochastic gradient descent by back-propagating the loss and updating the parameters φ.
At meta-test time, the trained meta-learner is given a set of unseen test tasks, which typically contain classes that have not been seen during meta-training. For each task, the meta-learner is given its support set DS , and is then evaluated on its predictions for all the query inputs x∗ (Fig. 2, left).
Large memory requirements for meta-training The primary bottleneck to using large images (i.e. ≥ 224× 224 pixels) in meta-learning approaches is the large amount of (GPU) memory required to process a task’s support set DS during meta-training. Specifically, the meta-learner θφ(DS) must perform a forward pass with a task’s entire support set before it can back-propagate the loss for query elements (x∗, y) ∈ DQ (and release the computation graph), thus preventing the use of conventional batch processing. The amount of memory required scales linearly with the number of support images Nτ and quadratically with their dimensions. If Nτ is large (e.g. the recent VTAB+MD benchmark [11] requires a task’s support set to be as large as 1000 images), memory on a single GPU is thus quickly exceeded for large images.
Note, the number of query elements in the task Mτ is not a bottleneck when using large images as the loss decomposes over elements of the query set DQ and is therefore amenable to mini-batching. By contrast, as the classifier itself is a non-linear function of the support set, the loss does not decompose and so it is not obvious how to apply similar ideas to allow scaling of DS in a principled way. Current ad hoc solutions to this problem are: (i) parallelize the meta-learner across multiple GPUs which may not be convenient or available and can involve significant engineering effort; (ii) train on tasks with smaller (or sub-sampled) support sets which may adversely affect performance on test tasks with more classes and/or large numbers of samples per class; (iii) train on tasks with smaller images (e.g. 84× 84 pixels in miniImageNet [16]) which limits performance and translates poorly to many real-world applications; or (iv) trade memory usage for additional computation [12] by
2Source code for ORBIT experiments is available at https://github.com/microsoft/ORBIT-Dataset and for the VTAB+MD experiments at https://github.com/cambridge-mlg/LITE.
Algorithm 1 LITE for a meta-training task τ Require: DS : task support set; DQ: task query set; N : number of support examples in DS ; H:
number of elements in DS to back-propagate; M : number of query examples in DQ; Mb: batch size for DQ; backward() ≡ function to back-propagate a loss; step() ≡ function to update parameters with a gradient step.
1: B ← ceil(M/Mb) . number of query batches 2: for all b ∈ 1, . . . , B do 3: DQb ← {x∗m, y∗m} Mb m=1 . get query batch from DQ 4: H ← {(xnh , ynh)}Hh=1 where {nh}Hh=1 ∼ U(1, N) .H to back-propagate 5: H ← DS ∩H .H to not back-propagate 6: DSb ← H∪H 7: θτ ← θ(DSb) 8: Lb ← 1MbΣ Mb m=1L(y∗m, f(x∗m,θτ )) . get loss of query batch
9: backward(Lb) . back-propagate loss on query batch 10: end for 11: φ← step(φ, N/H) . update φ using weighting factor N/H
employing activation/gradient checkpointing (i.e. during training store only a subset of intermediate activations in a network needed for backpropagation and recompute the rest with additional forward computations when needed) which allows for training on larger tasks at the expense of training time, but still falls well short of the memory needed to accommodate the task sizes required for key benchmarks (e.g. VTAB+MD).
Although training meta-learners has large memory requirements, meta-testing is generally memory efficient, requiring only a small number of gradient operations, or none at all, compared to transfer learning approaches that would perform large numbers of gradient-based updates at test time.

heading: 3 Large Image and Task Episodic (LITE) training
text: In this section, we introduce our general and memory-efficient solution for training meta-learners episodically on tasks with large support sets and large images. We call our approach Large Image and Task Episodic training or LITE. In Section 3.1, we describe how LITE can be applied to key classes of meta-learners.
Approach The fundamental idea underlying LITE is to perform a forward pass using the entire support set DS , but to compute the gradient contribution on only a small random subset of the examples in the support set. By doing this, we realize large savings in memory that includes gradients, activations, and the computation graph for the elements of DS that are not back-propagated. This is an approximation of the true gradient that would result if back-propagation was performed on all of the examples in DS . In the following section, we show that this approximation is an unbiased estimate of the true gradient. The approach for a general meta-learner is detailed in Algorithm 1 and shown diagrammatically in Fig. 2.
Mathematical justification The parameters of the meta-learner φ are found by minimizing the expected loss over all tasks.
argmin φ T∑ τ=1 Mτ∑ m=1 L (yτ∗m , f (xτ∗m ,θφ(DτS))) . (1)
In most meta-learning approaches, the support set enters into the loss through a sum over the N individual contributions from each data point it contains. This structure enables the meta-learners to be invariant to the ordering of the support set and allows all members of the support set to contribute to the adapted parameters (unlike alternative permutation invariant operators like max or min). Below we show in blue how this sum arises in popular brands of meta-learners.
In amortization methods (e.g. CNAPS [4] and VERSA [17]), the aggregation of support set points is built in directly via a deep set encoder eφ1(·). This encodes the support set into an embedding vector
which is mapped to the classifier parameters by a hyper-network tφ0(·).
θφ(DS) = tφ0 ( N∑ n=1 eφ1(xn, yn) ) . (2)
In gradient-based methods (e.g. MAML [1]), the classifier parameters are adapted from an initial value φ0 using a sum of derivatives of an inner-loop loss computed for each data point in the support set. The derivatives play the role of the deep set encoder in amortization methods.
θφ(DS) = φ0 + φ1 N∑ n=1 d dφ Linner(yn, f(xn,φ)) ∣∣∣ φ=φ0
(3)
Metric-based methods (e.g. ProtoNets [3]) comprise a body formed of a feature extractor and a head formed from a distance-based classifier. The classifier’s body parameters are not adapted in a task specific way θ(body)φ,c (DS) = φ0. The classifier’s head is adapted by averaging the activations for each class in the support set to form prototypes. Letting kc denote the number of support examples of class c, the adapted head parameters are given by
θ (head) φ,c (DS) =
1
kc kc∑ i=1 f(x (c) i , φ0) = 1 kc N∑ n=1 1(yn = c)f(xn, φ0). (4)
Query points can then be classified using their distance from these prototypes d(f(x∗n, φ0),θ (head) φ,c ).
We have established that in many meta-learners, each support set affects the classifier parameters and therefore the loss through a sum of contributions from each of its elements. We now focus on the consequences of this structure on the gradients of the loss with respect to the meta-learner’s parameters. To reduce clutter, we consider the contribution from just a single query point from a single task and suppress the dependence of the loss on the classifier and the query data point, writing
L (y∗, f(x∗,θφ(DS))) = L (eφ(DS)) where eφ(DS) = N∑ n=1 eφ(xn, yn) = N∑ n=1 e (n) φ . (5)
As a consequence of the summation, the derivative of the loss is given by
d
dφ L(eφ(DS)) = L′(eφ(DS))× ( N∑ n=1 de (n) φ dφ ) where L′(eφ(DS)) = dL(e)) de ∣∣∣ e=eφ(DS) (6)
which is a product of the sensitivity of the loss to the encoding of the data points and the sensitivity of the contribution to the encoding from each data point w.r.t. the meta-learner’s parameters. This second term is the source of the memory overhead when training meta-learners, but importantly, it can be rewritten as an expectation w.r.t. a uniform distribution over the support set data-point indices,
d
dφ L(eφ(DS)) = N L′(eφ(DS)) En∼U(1,N)
[ de
(n) φ
dφ
] . (7)
We can now define the LITE estimator of the loss-derivative by approximating the expectation by Monte Carlo sampling H times,
d
dφ L(eφ(DS)) ≈
N H L′(eφ(DS)) H∑ h=1 de (nh) φ dφ = d dφ L̂(eφ(DS)) where {nh}Hh=1 ∼ U(1, N). (8)
This estimator is unbiased, converging to the true gradient as H →∞. The estimator does not simply involve subsampling of the support set – parts of it depend on all the support set data points DS – and this is essential for it to be unbiased. The expectation and variance of this estimator are
E{nh}∼U(1,N) [ dL̂ dφ ] = dL dφ and V{nh}∼U(1,N) [ dL̂ dφ ] = N2 H (L′)2 V{nh}∼U(1,N) [ de (nh) φ dφ ] .
In Section 5.3, we empirically show that the LITE gradient estimate is unbiased and that its standard deviation is smaller than that of the naive estimator formed by sub-sampling the full support set. LITE provides memory savings by subsampling H examples from the support set, with H < N , and back-propagating only them. Crucially, a forward pass is still performed with the complementary set of points, with cardinality N −H , but these are not back-propagated.

heading: 3.1 Applying LITE to key meta-learning approaches
text: To demonstrate its versatility, we now describe how to apply LITE to models within some of the main classes of meta-learners: CNAPS [4] and Simple CNAPS [5] for amortization-based methods and ProtoNets [3] for metric-based methods. Note, these are a few possible instantiations. LITE can be applied to other meta-learning methods in a straightforward manner.
In the descriptions below, we consider just one query batch DQb (i.e. one iteration of the for-loop in Algorithm 1). Note, in practice, whenever H is passed through a module, back-propagation is enabled, while for H, back-propagation is disabled.3 Furthermore, since typically |H| |H|, we can forwardH in a single batch, however, we need to splitH into smaller batches. SinceH does not require gradients to be computed, this can be done without a significant impact on memory.
CNAPS [4], Simple CNAPS [5] + LITE (Appendix A.1) CNAPS variants are amortizationbased methods whose hyper-networks take a task’s support set as input and generate FiLM layer [18] parameters which modulate a fixed feature extractor. The classifier head can also be generated (CNAPS [4]) or adopt a metric-based approach (Simple CNAPS [5]), thus both variants can be adapted with just a single forward pass of the support set at test time. Meta-training them with LITE involves passing H and then H through their set-encoder eφ1 , and then averaging all the low-dimensional embeddings to get an embedding for the task. The task embedding is then input into a set of MLPs which generate FiLM layer parameters. H is passed through this configured feature extractor, followed by H, to get the task-adapted features for all support examples in DSb . For CNAPS, the task-adapted features ofH andH are pooled by class and fed into a second MLP which generates the parameters of the fully-connected classification layer. For Simple CNAPS, the task-adapted features of H and H are instead used to compute class-wise distributions (i.e. class mean and covariance matrices). With back-propagation enabled, the query batch DQb is then passed through the task-configured feature extractor and classified with the task-configured classifier (for CNAPS), or with the Mahalanobis distance [19] to the class-wise distributions (for Simple CNAPS). The query batch loss is computed, and only back-propagated forH. Note that the feature extractor is pre-trained and frozen, and only the parameters of the set-encoder and generator MLPs are learned.
ProtoNets [3] + LITE (Appendix A.2) ProtoNets [3] is a metric-based approach which computes a set of class prototypes from the support set and then classifies query examples by their (e.g. Euclidean) distance to these prototypes. Like CNAPS variants, it requires only a single forward pass to learn a new task. Meta-training ProtoNets with LITE involves passingH through the feature extractor with back-propagation enabled, followed byH with back-propagation disabled, to obtain features for all support examplesDSb . These features are averaged by class to compute the prototypes such that (with back-propagation enabled) the query batch DSb can be passed through the feature extractor and classified based on the Euclidean distance. The loss of the query batch is computed and only back-propagated forH. Note that here all the parameters of the feature extractor are learned.

heading: 4 Related work
text: We review the two main approaches to few-shot learning: transfer learning methods which are easy to scale to large images but are costly to adapt at test time, and meta-learning methods which are harder to scale but cheap to adapt (see Fig. 1). Note, we do not cover methods already described above.
Transfer learning approaches have demonstrated state-of-the-art performance on challenging few-shot benchmarks [20, 11]. However, they incur a high computational cost at test time as they rely on large (pre-trained) feature extractors which are fine-tuned with many optimization steps. MD-Transfer [13] fine-tunes all the parameters in a ResNet18 feature extractor with a cosine classifier head for 200 optimization steps. BiT [6] fine-tunes a feature extractor (pre-trained on 300M images in the JFT300M dataset [21]) with a linear head, in some cases for up to 20,000 optimization steps to achieve its state-of-the-art results on the VTAB [20] benchmark. SUR [22] instead trains 7 ResNet-50 feature extractors, one for each training dataset. At test time, the predictions from each are concatenated and fine-tuned to minimize a cosine-similarity loss. All of these approaches involve on the order of teras to petas of Mulitply-Accumulate operations (MACs) to learn a single new test task, and this must
3In PyTorch, this can be achieved by setting torch.grad.enabled = True when passing H, and torch.grad.enabled = False when passing H.
be repeated for each new task encountered. Furthermore, for each new task type, transfer learning approaches may need to be tuned on a validation set to obtain the optimal hyper-parameters.
In comparison, meta-learning approaches [23] generally require orders of magnitude fewer MACs and steps to learn a new task at test time. Popular approaches include CNAPS [4], Simple CNAPS [5], ProtoNets [3], and MAML [1] and are discussed in Section 3.1. Others include ProtoMAML [13] which fuses ProtoNets and MAML by initializing the classifier weights with the prototypes and then, like MAML, takes a few optimization steps to tune the weights to the task. It therefore incurs a similar cost to adapt as MAML, except it must additionally compute the prototypes. Finally, CTX [24] replaces the final average pooling layer of ProtoNets with a transformer layer that generates a series of prototypes which are spatially aware and aligned with the task. Like CNAPS, it requires just a single forward pass, however, requires more MACs to adapt since it uses a larger feature extractor (ResNet-34) and an attention module. CTX is one of the few meta-learning approaches that has been meta-trained on 224× 224 images, but requires 7 days of training on 8 GPUs. Memory efficient variants of MAML have been developed. First-order MAML [1] saves memory by avoiding the estimate of second-order derivatives. This is also done in Reptile [25] that additionally avoids unrolling the computation graph, performing standard gradient descent at each adaptation step. Implicit MAML [26], decouples the meta-gradient from the inner loop and is able to handle many gradient steps without memory constraints. In addition [27], proposes methods to reduce the computation overhead of meta-training MAML for large tasks. Note that, in all these cases, savings arise from working around the limitations of MAML, while LITE is more general.

heading: 5 Experiments
text: In this section, we demonstrate that meta-learners trained with LITE achieve state-of-the-art performance among meta-learners on two challenging few-shot classification benchmarks: (i) ORBIT [14] which is a real-world few-shot object recognition dataset for teachable object recognizers; and (ii) VTAB+MD [11] which is composed of the Visual Task Adaptation Benchmark (VTAB) [20] and Meta-Dataset (MD) [13] and combines both few-shot and transfer learning tasks. We compare LITE meta-learners with state-of-the-art meta-learning and transfer learning methods in terms of classification accuracy, computational cost/time to learn a new task, and number of model parameters.

heading: 5.1 ORBIT Teachable Object Recognition Benchmark
text: ORBIT [14] is a highly realistic few-shot video dataset collected by people who are blind/low-vision. It presents an object recognition benchmark task which involves personalizing (i.e. adapting) a
recognizer to each individual user with just a few (support) videos they have recorded of their objects. To achieve this, the benchmark splits data collectors into disjoint train, validation, and test user sets along with their corresponding objects and videos. Models are then meta-trained on the train users, and meta-tested on how well they can learn a test user’s objects given just their videos (on a user-by-user basis). The benchmark has two evaluation modes: how well the meta-trained model can recognize a test user’s objects in clean videos where there is only that object present, and in clutter videos where that object appears within a realistic, multi-object scene.
Experiments We meta-train ProtoNets [3], CNAPs [4] and Simple CNAPs [5] with LITE on tasks composed of large (224× 224) images. We also meta-train first-order MAML on large images as a baseline. Since first-order MAML can process task support sets in batches, we simply reduce the batch size and do not need to use LITE. We compare all of the above to meta-training on tasks of small (84× 84) images (i.e. the original baselines [14]). We also include a transfer learning approach, FineTuner [28], which freezes a pre-trained feature extractor and fine-tunes just the linear classifier for 50 optimization steps. For each model, we consider a ResNet-18 (RN-18) and EfficientNet-B0 (EN-B0) feature extractor, both pre-trained on ImageNet [29]. We follow the task sampling protocols described in [14] (see Appendices B and C.1 for details). We also include analyses on meta-training with small tasks of large images in Appendix D.3.
Results In Table 1, we report frame accuracy and video accuracy, averaged over all the query videos from all tasks across all test users (17 test users, 85 tasks in total), along with their corresponding 95% confidence intervals. We also report the computational cost to learn a new task at test time in terms of the number of Multiply-Accumulate operations (MACs), the number of steps to adapt, and the wall clock time to adapt in seconds. See Appendix C.1 for results on additional metrics. The key observations from our results are: • Training on larger (224× 224) images leads to better performance compared to smaller (84× 84)
images. The boost is significant for both clean and clutter videos, though absolute performance remains lower on clutter videos. This suggests that object detection or other attention-based mechanisms may be required to further exploit large images in more complex/cluttered scenes.
• All meta-learners + LITE set a new state-of-the-art on clean videos, and perform competitively with the FineTuner on cluttered videos, using an EfficientNet-B0 backbone.
• Meta-learners are competitive with transfer learning approaches in accuracy but are almost two orders of magnitude more efficient in the number of MACs and the time to learn a new task, and one order of magnitude smaller in the number of steps to adapt.

heading: 5.2 VTAB+MD
text: VTAB+MD [11] combines revised versions of the Meta-Dataset (MD) and VTAB datasets and is one of the largest, most comprehensive, and most challenging benchmarks for few-shot learning systems. The MD-v2 part of the benchmark involves testing on 8 diverse datasets while VTAB-v2 involves testing on 18 datasets grouped into three different cataegories (natural, specialized, and structured).
Results Fig. 3 compares our best meta-learner, Simple CNAPS + LITE, with 6 other competitive approaches on VTAB+MD: BiT, MD-Transfer, and SUR are transfer learning based methods while ProtoMAML, ProtoNets, and CTX are meta-learning based. Note, comparisons are not always likefor-like as methods use differing backbones, image sizes, and pre-training datasets. Appendix D.2 provides this information along with the precise numbers and 95% confidence intervals. Appendices B and C.2 summarize all implementation and experimental details, and Appendix D.3 includes further analyses on the impact of task size on meta-training. The key observations from our results are: • On MD-v2, Simple CNAPS + LITE has the highest average score and sets a new state-of-the-art. • On VTAB-v2, BiT scores highest overall, but among meta-learners, Simple CNAPS + LITE is
the best overall and on the natural and specialized sections. It falls short of CTX on the structured section due to poor performance on dSprites which involves predicting the position and orientation of small white shapes on a black background – a task quite different to image classification.
• These results are significant given that CTX takes 7 days to train on 8 GPUs, whereas Simple CNAPS + LITE trains in about 20 hours on a single 16GB GPU. In addition, Simple CNAPS + LITE uses a relatively small pre-trained backbone (4.0M parameters) compared to SUR’s 7 pre-trained ResNet-50s (one for each MD-v2 training set, plus ImageNet).
• Simple CNAPS + LITE using 224×224 images significantly outperforms Simple CNAPS using 84×84 images, except for when the dataset images are small (e.g. Omniglot, QuickDraw, dSprites). This demonstrates that using large images and an approximation to the support set gradients achieves superior results compared to using small images with exact gradients.
• MD-Transfer and BiT perform strongly across VTAB+MD, however, Simple CNAPS + LITE is significantly faster to adapt to a new task requiring only a forward pass of the support set with no hyper-parameter tuning whatsoever. The transfer learners instead perform 100s of optimization steps to adapt to a new task and may require a human-in the-loop to tune hyper-parameters such as the learning rate and number of optimization steps.
5.3 Effect of varying |H|
Table 2 shows the effect of varying |H|, the number of examples back-propagated per task, on the VTAB+MD benchmark. Performance is consistent across different |H|, an expected result since LITE provides an unbiased estimate of the true gradient. For both Simple CNAPs and ProtoNets + LITE, the results at the lowest values of |H| are respectable, though they fall short of what can be achieved at |H| = 40. Thus, the support gradient information does improve the solution, albeit by only 1-2 percentage points. Note, we report the lowest setting as |H| = 1 for Simple CNAPs but |H| = 0 for ProtoNets. This is because Simple CNAPs’s adaptation network (which processes the support set) shares no parameters with the feature extractor and thus will not be learned if the support gradients are completely ignored. On the other hand, ProtoNets’ adaptation network shares all its parameters with the feature extractor, thus can be metalearned even when the support gradients are neglected.
Finally, in the two rightmost columns, we compare the classification accuracy for |H| = |DS | (i.e. using the full support set gradient) to |H| = 40 (i.e. using LITE). Due to memory constraints, we do this at image size 84× 84. Here we see that the difference in accuracy is significant. We expect that performance will smoothly interpolate as |H| is increased from 40 to the size of the largest support set (at which point the full gradient is computed). This validates how LITE can be used to trade-off GPU memory usage for classification accuracy by varying |H|. Furthermore, we conduct an empirical analysis (see Table D.7) which shows that the LITE gradient estimates and the gradients when using smaller sub-sampled tasks are unbiased w.r.t. the true gradients. However, Fig. 4 shows that LITE offers a significantly lower root mean square error w.r.t. the true gradients compared to using sub-sampled tasks at all but the highest values of |H|. Refer to Appendix D.4 for additional details.

heading: 6 Discussion
text: We propose LITE, a general and memory-efficient episodic training scheme for meta-learners that enables them to exploit large images for higher performance with limited compute resources. LITE’s significant memory savings come from performing a forward pass on a task’s full support set, but back-propagating only a random subset, which we show is an unbiased estimate of the full gradient. We demonstrate that meta-learners trained with LITE are state-of-the-art among meta-learners on two challenging benchmarks, ORBIT and VTAB+MD, and are competitive with transfer learning approaches at a fraction of the test time computational cost.
This offers a counterpoint to the recent narrative that transfer learning approaches are all you need for few-shot classification. Both classes of approach are worthy pursuits (and will need to exploit large images in real-world deployments) but careful consideration should be given to the data and compute available at test time to determine which class is best suited to the application under consideration. If it involves learning just a single task type (e.g. classifying natural images) with ample data and no compute or time constraints, then a fine-tuning approach would suffice and perform well. However, if a multitude of task types will be encountered at test time, each with minimal data, and new tasks need to be learned on resource-constrained devices (e.g. a mobile phone or a robot) or quickly/repeatedly (e.g. in continual or online learning settings), then a meta-learning solution will be better suited.
Finally, as the machine learning community grapples with greener solutions for training deep neural networks, LITE offers a step in the right direction by allowing meta-learners to exploit large images without an accompanying increase in compute. Future work may look toward applying the basic concept of LITE to other types of training algorithms to realize similar memory savings.
Limitations As discussed in Section 3, LITE can be applied to a wide range of meta-learners provided that they aggregate the contributions from a task’s support set via a permutation-invariant operation like a sum. Because only a subset of the support set is back-propagated, however, the gradients can be more noisy and meta-training may require lower learning rates. Furthermore, LITE is a memory-efficient scheme for training meta-learners episodically and has not been tried with meta-learners trained in other ways (e.g. with standard supervised learning) or non-image datasets.
Societal impact Few-shot learning systems hold much positive potential – from personalizing object recognizers for people who are blind [14] to rendering personalized avatars [30] (see [23] for a full review). These systems, however, also have the potential to be used in adverse ways – for example, in few-shot recognition in military/surveillance applications. Meta-trained few-shot systems may also pose risks in decision making applications as uncertainty calibration in meta-learning models has not yet been extensively explored. Careful consideration of the intended application, and further study of uncertainty quantification in meta-learning approaches will be essential in order to minimize any negative societal consequences of LITE if deployed in real-world applications.

heading: Acknowledgments
text: We thank the anonymous reviewers for key suggestions and insightful questions that significantly improved the quality of the paper. Additional thanks go to Vincent Dumoulin for providing the tabular results for SUR used in Fig. 3 and Table D.2.
Funding Transparency Statement
Funding in direct support of this work: John Bronskill, Massimiliano Patacchiola and Richard E. Turner are supported by an EPSRC Prosperity Partnership EP/T005386/1 between the EPSRC, Microsoft Research and the University of Cambridge.

Review summaries:
Meta-learning algorithms are not memory friendly since they have to load all the support samples during the inner-loop. When the support data are high-resolution images, or need huge support samples (i.e. 1000), the meta-training is hard to work on a single GPU. As a result, this paper aims to propose an efficient meta-training algorithm which only rely on the gradients of subsets of the support data. And they show the can achieve competitive results against transfer learning approaches but at a fraction of the test-time computational cost on multiple benchmarks.

The authors propose a method that enables few-shot learners to be trained on 1 GPU without memory issues. This is very important since most few-shot learners need to process the whole train set of a task before evaluating the loss on the task. They show mathematically that their method gives an unbiased estimation of the true gradient (if all samples in training were back-propagated to update the model). They evaluate their method on CNAPS and ProtoNets and show that it can achieve better results since it can be trained with larger images.  

This paper considers the problem of training image classification for meta-learning/few-shot learning models with large images. The authors point out that many categories of meta-learning algorithms have difficulty using large images because of memory constraints brought up by computation of gradients computed during meta-training. They decompose 3 major categories of meta-learning models (CNAP, MAML, and Proto-Nets) into a common gradient update that is troublesome w/r/t memory. Their idea to make training more feasible is to use all the support set items in the forward pass but approximate the backward pass using a small random subsample of the examples in the support set. They compare training their versions of memory-efficient, large input meta-learning models against recent work on ORBIT (a few-shot object recognition task) and VTAB+MD (a newer benchmark for few-shot classification that is based on Meta-Dataset and VTAB datasets). On both benchmarks, they show that training using larger images using their proposed method improves performance.

Many meta-learning approaches have been trouble with the scalability issue due to the large memory-required computation during the training process. To overcome such a problem, this paper proposes LITE that is memory efficient episodic training scheme allowing meta-learning methods to use the benefit of large-size images of tasks on a single GPU. On two benchmark datasets (ORBIT and VTAB+MB), meta-learning methods combined with LITE outperform meta-learning methods without LITE and Simple CNAPs (meta-learning method) + LITE shows competitive performance compared with transfer learning methods.