Context-Aware Meta-Learning
Authors: Christopher Fifty
Dennis Duan
Ronald Guenther Junkins
Ehsan Amid
Jure Leskovec
Christopher Re
Sebastian Thrun
Venue: ICLR 2024 poster (2024)
Keywords: meta-learning
in-context learning
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Large Language Models like ChatGPT demonstrate a remarkable capacity to learn new concepts during inference without any fine-tuning. However, visual models trained to detect new objects during inference have been unable to replicate this ability, and instead either perform poorly or require meta-training and/or fine-tuning on similar objects. In this work, we propose a meta-learning algorithm that emulates Large Language Models by learning new visual concepts during inference without fine-tuning. Our approach leverages a frozen pre-trained feature extractor, and analogous to in-context learning, recasts meta-learning as sequence modeling over datapoints with known labels and a test datapoint with an unknown label. On 8 out of 11 meta-learning benchmarks, our approach---without meta-training or fine-tuning---exceeds or matches the state-of-the-art algorithm, P>M>F, which is meta-trained on these benchmarks.

Published as a conference paper at ICLR 2024
CONTEXT-AWARD META-LEARNING
Christopher Fifty', Dennis Duan 1.2 Ronald G. Junkins', Ehsan Amid", Jure Leskovee', Christopher Ré', Sebastian Thrun' Stanford University, GGoggl, Google DeepMind fi fiftyêcs. stanford. com
ABSTRACT
Large Language Models like ChatGPT demonstrate remarkable capacity to learn new concepts during inference without any fine-tuning. However, visual models trained to detect new objects during inference have been unable to replicate this ability, and instead either perform poorly or require meta-training and/or fine- tuning on similar objects. In this work, we propose a meta-learning algorithm that emulates Large Language Models by learning new visual concepts during inference without fine-tuning. Our approach leverages frozen pre-trained feature extractor, and analogous to in-context learning, recasts visual meta-learning as sequence modeling over datapoints with known labels and test datapoint with an unknown label. On 8 out of 11 few-shot image classification benchmarks, our approach- -iithou meta-training or tne-tuning-exceeeds exceeds matches the state-of- the-art algorithm, P>M>F, which is meta-traine meta-trained on these benchmarks. Our code is available
INTRODUCTION
Meta-learning refers to a capacity to learn new concepts from small number of demonstrations (Lake al., 2015). In a decade of remarkable advances to machine intelligence, it remains an area where human performance continues to surpass that of machines (Brown et al., 2020). To match human capabilities, and towards developing machines that can learn and think like humans, we must develop machine intelligence capable of learning novel concepts from only few examples (Lake et al., 2017).
Many applications of deep learning apply a learning algorithm to a large set of training data; however, learning from a very small number of training examples poses a challenge (Lake et al., 2017; Gamelo et al., 2018). This challenge led to two predominant evaluation settings: in-domain and cross-domain. The in-domain setting evaluates meta-learner's ability to quickly adapt to new tasks after training on similar tasks within a specific domain. Models designed for this setting are often extremely fast but exhibit poor generalization to tasks outside the target domain (Chen et al., 2019). Meanwhile, the cross-domain setting evaluates a meta-learner's 's ability to adapt to tasks in previously unseen domains. Methods designed for this setting are highly adaptable but slow during inference as they require fine-tuning on the support set (Guo et al., 2020; Oh et al.. 2022; Hu et al., 2022). Critically, meta-learners in both settings differ from a human's capacity to quickly generalize new tasks.
The problem of simultaneously fast and general meta-learning has recently been addressed in Natural Language by Large Language Models (LLMs). LLMs like ChatGPT can quickly generalize to new tasks through an ability termed in-context learning (Brown et al., 2020). However, it remains an open problem in Computer Vision. Even the best visual meta-learning algorithms cannot be deployed to a ChatGPT-llik system because such systems require models that can (1) generalize to broad set of tasks unknown at training time and (2) so in real-time, without the time allowance for finetuning the model. LLMs have shown remarkable ability to do both; however, current visual meta-learners may only satisfy one requirement or the other (Hu et al., 2022).
To measure progress towards this goal of fast and general visual meta-learners, we develop an evaluation paradigm that we call universal meta-learning. Universal meta-learning measures a model's 's capacity to quickly learn new image classes. It evaluates models across a diverse set of meta- meea-lcarning benchmarks spanning many different image classification tasks without meta-train treaingaining on any of the benchmarks" training sets fine-tuning the support set during inference. We focus on

---

Published as a conference paper at ICLR 2024
the application of few-shot image classification opposed to dense prediction tasks like in in-painting or segmentation as the universal setting has already been explored for these applications (Bar et al., 2022: Zhang et al., 2023; Wang et al., 2023; Kim et al., 2023; Butoi et al., 2023).
Beyond benchmarking methods in the universal setting, we present a meta-learner that achieves strong universal performance. Drawing inspiration from in-context learning in LLMs, we reformulate n-way-k-shot image classification as non-causal sequence modeling over the support set and an unknown query image. Specifically, given n-way classification with k-examples from each class, we train a non-causal model over { 3:) J (image, label) support set pairs, and an unlabeled query image 2nk+1, to predict the label of the query image. This formulation causes the meta- meaaeee to extrapolate to new classes in its parameter space, enabling it to learn new visual concepts during inference without fine-tuning. Due to its capacity to learn visual information "in-context", we term our approach Context-Aware Lera-neg (CAML).
In summary, our contribution is two-fold. First, we develop a meta-learning evaluation paradigm that approximates the performance of visual meta-learners in ChatGPT-like application. Second, we design a meta-learning algorithm that works well in this setting. Our empirical findings show that CAML outperforms other meta-learners in the universal setting. Remarkably, CAML's performance in the universal setting often matches- and even exceeds- tte in-domain performance of the state-of- tre-art meta- meta-learning algorithm, P>M>F (Hu et al., 2022), that is directly trained on each down-stream benchmark.
RELATED WORK
Meta-Learning as Causal Sequence Modeling. Several of the earliest meta-learning algorithms were formulated causal sequence modeling problems. Hochreiter et al. (2001) leverage a LSTM (Hochre- iter & Schmidhuber, 1997) to model extensions to semi-linear and quadratic functions, and two decades later, Graves et al. (2014); Santoro et al. (2016); Kaiser et al. (2017) build upon this approach by integrating a form of external memory that the LSTM can read to and write from memory to develop Neural Turing Machines. With the advent of self-attention (Vaswani et al., 2017), Mishra et al. (2017) predict the labels of query images by first composing a sequence of (image, label) pairs and then feeding through a stack of interleaved causal self-attention and temporal convolution layers. Kirsch et al. (2022) replaces the stack of interleaved causal oeff-attention and temporal convolution layers with Transformer encoder; however, their approach is also causal in the input sequence by composing a sequence of (image, label of previous image) pairs. Both Mishra et al. (2017) and Kirsch et al. (2022) are conceptually similar to our work; however, the causal property of both approaches breaks an important symmetry in meta-learning, namely invariance to permutations of the support set (Garnelo et al., 2018; Müller et al., 2021). In Section 5.2, we observe a performance gap between both approaches and CAML and hypothesize the causal approach actually forces a subtly more difficult modeling problem by imposing a causality inductive bias on a fundamentally non-causal prediction task.
Cross- Domain Meta-Learning. paradigm where the meta-training and inference-time data distributions are significantly differ- ent (Chen et al., 2019). Recent work finds that leveraging self-supervised pre-training-6 oor foun- dational model feature extractors-can significantly improve cross-domail performance (Hu et al., 2022; Zhang et al., 2021). Moreover, fine-tuning with respect to the support set almost always outperforms meta-learning without fine-tuning in this setting (Guo et al., 2020; Oh al., 2022; Phoo & Hariharan, 2020; Islam et al., 2021). While effective, fine-tuning is prohibitive to deploying visual meta-learning models in a manner similar to LLMs like ChatGPT the latency and memory cost to fine-tune model's parameters each user query is untenable. Accordingly, we propose the universal setting to measure a meta-learner' ability to learn to classify any task seen during inference without fine-tuning.
meta-learning refers to a challenging evaluation
In-Context Learning for Dense Prediction Tasks, Many recent works have explored in-context learning for other applications of computer vision. Bar et al. (2022) casts in-context learning as image in- painting by first concatenating demonstration images with a query image and then using vision model to fil-in-thhebband within this concatenated image. Building this work, Zhang et al. (2023) explores what demonstrations lead to strong in paianniiig performance and Wang et al. (2023) generalizes the approach by formulating other visual applications like segmentation, depth
2

---

Published as a conference paper at ICLR 2024
Paramelers Frozen
Parameters Trained
Query Image
Support Set Images
Class
P(Class)
MLP
01,0,0]
0.03
[0 1,
0.06
[0 0.0.11
0.91
Pre-Trained Image Encoder
Concatenate Embeddings
Mon-Causal Sequence Model
ELMES Class Encoder
Unknown [class] Embedding
n.
o.
[0.0.1]
Support Set Classes
Figure I: Overview CAML. Query and support set images are encoded with pre-trained feature extractor then concatenated with their corresponding ELMES label embeddings. We feed the resulting sequence of concatenated vectors into non-casual sequence model and extract the query vector from the output sequence to predict its class.
estimation, etc. as in-painting. Other approaches explore in-context learning for applications like scene understanding (Balazevic et al., 2024), medical image segmentation (Butoi et al., 2023), and more generally dense prediction tasks (Kim et al., 2023). Like these approaches, we study visual in-context learning; however, this work focuses on few-shot image classification rather than dense prediction tasks.
APPROACH
We adapt the ideas underpinning in-context learning in LLMs JLMssnnmely learning to classify query from context of support set demonstrations in single forward pass-to image classification. However, dissimilar from in-context learning, visual meta-learners should be non-causal: placing one example before another in the support set does not entail a causal relationship (Garnelo et al., 2018; Müller et al., 2021).
Architecture. An overview of CAML is shown in Figure 1. It consists of three different components: (1) frozen pre-trained image encoder, (2) fixed Equal Length and Maximally Equiangular (ELMES) class encoder. and (3) non-causal sequence model. While pre-trained image encoders and non-causal sequence models are well-known, to encode label information we introduce an ELMES encoder. An ELMES encoder bijective mapping between the labels and set of vectors that equal length and maximally equiangular. Historically, labels have been encoded with one-hot vectors; however Section 4, prove that ELMES encoding of mutually exclusive classes allows the sequence model to maximally identify classes within the support set.
As visualized in Figure 1, CAML first encodes query and support images using frozen pre-trained feature extractor. Crucially, the pre-trained image encoder's 's embedding space distills images into low-dimensional representations
that images with similar content and visual characteristics have similar embeddings. Classes of the support set are encoded with an ELMES class encoder; however as the class of the query is unknown, we use special learnable "unknown token" embedding that learned during large-scale pre-training. CAML then concatenates each image embedding with its corresponding query embedding to form an input sequence.
Progressing through Figure 1, this sequence is fed into non-causal sequence model, i.e. Trans- former encoder, to condition the output representations on the full context of query and support set points. This enables dynamic and real-time classification; visual characteristics from query and support set images can be compared with each other to determine the specific visual features- content, textures, etc. used classify the query. From the output sequence of the non-causal sequence model, we select the element at the same position as the query in the input sequence, and pass this vector through shallow MLP predict the label of the query.
Large-Scale Pre- Training, As our focus universal meta-learning and CAML may encounter any new visual concept during inference- pre-train CAML's non-causal sequence model on few-shot

---

Published as a conference paper at ICLR 2024
image classification tasks from ImageNet-lk 1k (Deng al., 2009), Fungi (Schroeder & Cui, 2018), MSCOCO (Lin et al., 2014), and WikiArt (Salch & Elgammal, 2015). We chose these datasets because they span generic object recognition (ImageNet- Ik, MSCOCO), fine- inee-graind image classification (Fungi), and unnatural image classification (WikiArt). To avoid distorting the pre-trained image encoder's embedding space, we freeze this module and only update the sequence model's parameters during pretraining. Similarly, since an ELMES minimizes the entropy of detecting classes within the support set, the label encoder is also frozen. In the context of pre-training, meta-training, and fine-tuning, CAML only requires pre-training and avoids meta-tran meta-training on the train/validation splits of meta-learnig benchmarks or fine-tuning on the support set during inference.
THEORETICAL ANALYSIS
In this section, we motivate our choice of the ELMES Class Encoder by considering the symmetries desirable in meta-learning algorithms. Two important symmetries are (1) invariance to the assignment of support set classes to numeric labels and (2) invariance to permutations in the ordering of the input sequence. The first invariance implies the class embeddings must be equiangular and equal norm, with an ELMES configuration minimizing the entropy of learnable model parameters detecting any given class. Later, show ELMES also satisfies the second symmetry. Due to space constraints, all proof's and many definitions, properties, lemmas, and theorems are allocated to Appendix A.I. We begin with formal definition of ELMES.
4.1 EQUAL LENGTH AND MAXIMALLY EQUIANGULAR SET OF VECTORS
Definition 1. An Equal Length and Maximally Equiangular Set (ELMES) is a set of non-zero vectors {0, Ad 1- 0j E IR++ for some Z O and d > 1, such that Vjj * j'. ı = 1 and -1 (0) )5) 2-1- Simply, all vectors in this set are equal length and maximally equiangular.
An Equal Angle and Maximally Equiangular Set (ELMES) of vectors has connections to both Equiangular Tight Frames in representation theory (Welch, 1974; Fickus et al., 2018) as well as the Simplex Equiangular Tight Frames highlighted in recent neural collapse works exploring softmax- layer geometry at the terminal phase of training (Papyan et al., 2020; Yang et al., 2022). We offer additional discussion comparing these structures in Appendix A. well as provide an intuitive view of an ELMES as a regular d-simplex immersed in Rd+k
4.2 LABEL SYMMETRY
Symmetry in the assignment of support classes to numeric labels is an important property of meta- learning algorithms. For example, if we have the support set classes (tower, bear, tree). the mapping of (bear -> 1, tower -> 2, tree -> 3} should produce the same prediction for a query point as a different mapping (bear 2, tower -> 3, tree -> 1). To explore this symmetry, we examine how class embeddings may be used by the model.
represent demonstration vector concatenation of an image embedding and a label embedding Φ: [p Φ). This vector is directly fed into the self-attention mechanism, where matrix multiply with key, query, and value self-attention heads. Taking only one of these matrices for simplicity with head-dimension k:
From formulation in Section
T
[s 1k
P
= 10, Pi)
(o. [a)]+ 110b vi)
(6.
(I)
The output of this transformation will be the sum of two vectors: one composed the inner products between the image embedding P and the learnable (T.}=_ and the other composed of the class embedding and the learnable { Note that Equation (1) implies that CAML is not invariant to the assignment of labels to support set classes due to the addition between (p Т2) and (Ø vx); constrain the geometry of the class embeddings {0) to in principle respect label symmetry. Specifically for i + k. (d: Φz) (ò: Φx) and ||+.1| ||,
however,
Similar to a convolutional filter learning to match a pattern within an image, our analysis assumes the learnable (g/1 ... e'x] will converge to vectors that maximize the inner product with a single
4

---

Published as a conference paper at ICLR 2024
class embedding subject to certain constraints. Under this assumption, ask what geometry of the d-class embeddings {6}]_1 allows a learnable y/, vector to most unambiguously detect single class embedding. To answer this question, define probability mass function for each 4h over the set of d classes so that maximizing the probability of the j class aligns with maximizing (dj tbs) and equally minimizing (Øx 1%) for
Definition 2. Let X be discrete Random Variable taking on values in {1,2 2.., d]. For learnable vec- tor Uh define probability mass function Po. = j) as the softmax over K( v2) (0s v.)l that:
ell [ cos(0
Pa. (X =)
ellvs (ls(,)))
where 0z, the angle between 6j and 1/s-
We say vi learns detect class j when Pa, (X= > Pv: (X for S d with k o j. By symmetry in the assignment of class embeddings to support classes, we can assume that the number of v; learned to detect class is similar the number of e) learned to detect class j for all pairs (i. We also leverage symmetry in the assignment of labels to support set classes to make the following assumptions. justification for each assumption is located in Appendix A.1.
Assumption 1. Suppose { H are learnable class detectors of unit norm with at least one 1/:
detecting each class 1 S i S d. The probability P4; (X = j] Pv. (X = =i) S i, S d.
Assumption 2. Define Pv. (X +)){(d)) set of vectors {Øj) 1m
as the probability of th, detecting @s from the d. Then the probability pes (X -))(( Po, (X
a) ()((1)(((mm)) for $ i.j S m and m 2 2.
Assumption 3. When v. = de To.1l Pv. (X = i) is maximized.
When Assumption 1, Assumption and Assumption 3 hold, the set of class embeddings that maximize the probability of a learnable 1h detecting class i is necessarily an ELMES.
Theorem 1. The set of class embeddings (0, 1/ Vj. 1 S S that maximizes Pv, (X = is necessarily an ELMES.
Alternatively when viewed through the lens of information theory, we can interpret ELMES the class embedding that minimizes the entropy t/; detecting class Informally, ELMES causes th: to have the least uncertainty when detecting class i.
Proposition 1. Let H (X) be the entropy of Pv. (X). An ELMES minimizes H (X).
4.3 PERMUTATION INVARIANCE.
In addition to label symmetry, it is also desirable for the output prediction of CAML to not depend on the order of demonstrations in the sequence. For example, if have the support set classes (tower, bear, tree). the sequence ((bear -> 1), (tower - 2), (tree -> 3)} should produce the same output as the permuted sequence {(tree -> 3). (bear -> 1), (tower -> 2)}. Building on the prior work of Kossen et al. (2021); Fifty et al. (2023), it suffices to show to show that the ELMES label encoder is equivariant to permutations in the input sequence show that CAML is invariant to permutations.
Proposition 2. Consider n-sequence of one-hot labels stacked into a matrix S E RRxx and an ELMES label encoder denoted by W E IR wxd with w denoting "way" and d the dimension of the label embedding. The label embedding SW is equivariant to permutations.
EXPERIMENTS
To quantify universal image classification performance, we evaluate a diverse set of meta-learning benchmarks divided across 4 different categories:
1. Generic Object Recognition: mini- mini-ImageNer (Vinyals et al., 2016), tiered- ImageNet (Ren et al., 2018), CIFAR-fs (Bertinetto et al., 201 8), and Pascal VOC (Everingham al.)
5

---

Published as a conference paper at ICLR 2024
Table I: MinilmageNett & CIFAR-Is mean accuracy and standard error across 10,000 test epochs. indicates the pre-trained image encoder backbone was frozen during training.
CIFAR-fs
MinilmageNet
Method (Backbone)
5w-1s
5w-5s
5w-1s
5w-5s
In-Domain [Meta- Teainnrai P>M>F Hu et al. (2022)
84.3
92.2
95.3
98.4
Universal Meta-Learnnig;
No Meta-Trainin or Finetuning
ProtoNet (Snell al., 2017)
62.
79.7
92.1
97.1#:0
ProtoNet!
57.7
81.0
85.3
96.01.1
MetaOpt (Lee et al., 2019)
53.1
73.1
78.5
91.61t11
MetaOpt
61.72.2
83.1
86.94.2
96.5
MetaQDA (Zhang et al., 2021)
60.41.2
83.2
88.24.2
97.41.0
GPICL (Kirsch al., 2022)
41.5#.4
78.3
95.6
98.24.1
SNAIL (Mishra et al., 2017)
62.1
71.1
93.6
98.1+.0
CAML
70.8+
85.5+
96.2++.1
98.6++.
Table 2: Pascal & Paintings mean accuracy and standard error across 10,000 test epochs. indicates the the pre-trained image encoder backbone was frozen during training.
Pascal Paintings
Paintings
Pascal
Method (Backbone)
5w-1s
5w-5s
5w-1s
5w-5s
5w-1s
5w-5s
In-Domain [Meta-Training]
P>M>F
60.7
74.4
53.2
65.8
72.2
84.4
Universal Meta-Learning
ProtoNet
49.6
63.5
38.3+.2
48.2
77.9
87.3
ProtoNet
70.6
48.3+.2
64.1
72.2+.2
84.3+.2
MetaOpt
38.2
58.21.11
31.61.2
48.0
63.7
81.7
MetaOpt!
53.2
74.8
49.3#.2
65.9
72.8+.2
84.4
MetaQDA
53.8+
74.1+.1
49.4+.2
66.6
73.5+
85.2+
GPICL
62.6
74.6
51.61.2
61.02.1
81.74.2
88.2+.2
SNAIL
62.5+
77.6
51.9
65.8+.1
79.7+.2
88.0+.2
CAML
63.8
78.31.1
51.1
65.22.1
82.6
89.7
Fine-Grained Image Classification: CUB (Wah et al., 2011), Aireraft (Maji et al., 2013), meta- iNat (Wertheimer & Hariharan, 2019), and tiered meta-iNat (Wertheimer & Hariharan, 2019)
3. Unnatural Image Classification: ChestX (Guo et al., 2020) and Paintings (Crowley & Zisserman, 2015)
Inter-Domain Image Classification: Pascal++aintings (Everingham et al.; Crowley Zisserman, 2015).
Generic object recognition, fine- fine-grained image classification, and unnatural image classification are standard benchmarking tasks meta-learning literature (Chen al., 2020; et al., 2022: Wertheimer et al., 2020; Guo et al., 2020). Beyond this, compose a challenging new inter-domain category by combining Pascal VOC with Paintings so that each class is composed both natural images and paintings. This allows to evaluate the ability of meta-learning algorithms to generalize across domains within the same class. For example, the support image for the class "tower" may be Van Gogh's 's The Starry Night, while the query may be picture of the Eiffel Tower. Humans have the ability to generalize visual concepts between such domains; however, meta-leamiin algorithms struggle with this formulation (Jankowski & Græbezewski, 201
5.1 BASELINES
We evaluate the performance of CAML, Prototypical Networks (ProtoNet) (Snell et al., 2017), MetaOpt (Lee et al., 2019), MetaQDA (Zhang et al., 2021), SNAIL (Mishra al., 2017), and GPICL (Kirsch et al., 2022) universal meta-learning setting by pre-training them with a ViT- base (Dosovitskiy et al., 2020) feature extractor initialized with weights from CLIP (Radford et al.,

---

Published as a conference paper at ICLR 2024
Table 3: meta-iNat & tiered meta-iNat & ChestX mean accuracy and standard error across 10,000
test epochs. f indicates the the pre-trained image encoder backbone was frozen during training.
meta-iNat
tiered meta-iNat
ChestX
Method (Backbone)
5w-1s
5w-58
5w-1s
5w-5s
5w-1s
5w-5s
In-Domain (Meta-Training
P>M>>
91.2
96.1
74.8
89.9
27.0
32.1
Universal Meta-Learning
ProtoNet
78.4
89.4
66.34.2
82.2#.2
22.41:1
25.3
ProtoNet
84.5+.2
94.8#t.1
73.8+.2
89.5#.1
22.7
25.8
MetaOpt
53.01.2
77.74.2
37.31.2
63.01.2
20.8::.1
23.0
MetaOpt
85.54.2
95.511.1
75.14.2
91.91.1
23.0
27.4
MetaQDA
86.31.2
95.91.1
76.0
92.41.1
22.6#.1
27.0
GPICL
90.0+t.2
95.1
60.8+
87.6+
20.1
20.9
SNAIL
89.1
94.81.11
77.34.2
86.51.2
20.2
20.0
CAML.
91.2-
96.31.1
81.9+
91.6
21.5+.1
22.2
Table CUB tiered- Ieaee--m Aircraft mean accuracy and standard error across 10,000 test epochs. f indicates the the pre-trained image encoder backbone was frozen during training.
CUB
ticred- ImageNet
Aireraft
Method (Backbone)
5w-1s
5w-5s
5w-1s
5w-5s
5w-1s
5w-5s
In-Domain (Meainnrain
P>M>F
92.3
97.0
93.5
97.3
79.8
89.3
Universal Meta- Meta-Learning
ProtoNet
$9.4
77.3+.2
93.5=
97.4
37.9
$2.5
ProtoNet
87.0:1.2
97.11111
87.32.2
96.11.1
62.41.3
82.0:1.2
MetaOpt
71.5+.2
41.2+t22
76.6
89.6
41.6+.2
26.7+
MetaOpt
87.9+.2
97.2+.1
88.2+.2
96.5+.1
64.8+.2
82.6
MetaQDA
88.31.2
97.41.1
89.4=.2
97.0
63.61..3
83.0
GPICL
75.1
94.5
94.6=.1
97.21.1
19.8
61.81.3
SNAIL.
87.5+.2
92.8+.2
93.1
97.3+.1
48.9
35.8+.3
CAML
91.82.2
97.11.1
95.41:
98.12.1
63.3
79.1
2021). Pre-training runs over few-shot classification tasks from ImageNet-1k, Fungi, MSCOCO, and WikiArt, and during evaluation the set of 11 meta-learning benchmarks, models are not meta-trained or fine-tuned. We compare with ProtoNet, MetaOpt, and MetaQDA as they achieve sfate-of-the-ad results when paired with pre- trained feature extractor (Hu et al., 2022). As sequence modeling underpins CAML, we also compare with SNAIL and GPICL to evaluate the performance of past formulations of causal sequence- sequence-based meta-learning algorithms in the universal setting.
To assess the gap between universal and in-domain leaanne performance, we benchmark the current tf-ee-of meta-learning algorithm P>M>F (Hu al., 2022). Similar the universal setting, P>M>F uses a ViT-base feature extractor initialized with weights from DINO (Caron et al., 2021); however, it meta-trains on the training set of each benchmark before evaluating on that benchmark's ' test set.
When pre-training all models in the universal setting, do not perform any hyperparameter tuning in order to match the practices used by P>M>F. We use early stopping with a window size of epochs during pre-training and the code release of Hu et al. (2022) to benchmark P>M>F with the training settings and hyperparameters described in their work.
the learning rate fixed 10-5 and
5.2 RESULTS
Our findings are summarized Table 1, Table 2, Table 3, and Table 4 and indicate that CAML sets a new state-of-the state-of-the-art for universal meta-learning by significantly outperforming other baselines 14 of 22 evaluation settings. For the other 8 evaluation settings, CAML matches- or nearly matches the best performing baseline. Remarkably, CAML. also performs competitively with
7

---

Published as a conference paper at ICLR 2024
P>M>F on 8 out of 11 meta-learning benchmarks, even though P>M>F meta- meta-trains on the training set of each benchmark.
This result suggests that the amount of new visual information Iearned during inference through visual in-context learning can be comparable to the amount learned when directly meta-training on in-domain data. This capacity may unlock new applications in the visual space, just as the emergence of in-context learning in LLMs has enabled many new applications in natural language.
Benchmarks Where CAML Underperforms. The 3 datasets where P>M>F outperforms CAML are CIFAR-fs, Aircraft, and ChestX. CIFAR-fs is generic object recognition benchmark containing CIFAR images downsampled to 32x32 resolution. As CAML, and CLIPP pre-train on 224x224 resolution images, downsampling by factor of 49 likely induces a distribution shift that was not learned by CAML during large-scale pre-training. In the cases of Aircraft and ChestX, we postulate that the CLIP embedding structued so images with similar captions have similar embeddings- struggles to effectively differentiate between the fine- aen-ggrined, specialized classes in these tasks. For example, while Boeing 737 and Airbus A380 have different labels in the Aircraft dataset, the scraped CLIP captions for those images may not reach that level of granularity. This corroborates the findings from Radford et al. (2021), which found that in a zero-shot setting, CLIP underperforms in specialized or complex tasks.
Our ablation study into non-CLIP pre-trained feature extractors in Tables 5 to 8 of Appendix C shows CAML's performance on Aircraft can drastically improve. Specifically, 5w-1s performance increases from 63.3 to 81.8 and 5w-5s performance increases from 79.1 to 92.1 when ViT-Huge pre-trained on Laion-2b (Schuhmann et al., 2022) initializes the weights of the image encoder rather than CLIP.
Fine-tuning CLIP Backbone. Our findings in Tables 1 to 4 indicate that updating the CLIP image encoder during pre-training hurts the performance of ProtoNet and MetaOpt. We observe that these methods tend to overfit during pre-training, and our empirical results show similar pattern: pre- training with these methods often helps the performance of benchmarks similar to ImageNet (i.e. Pascal, MinilmageNet, fiered-ImageNet), but significantly hurts the performance of out-of-domain tasks (i.e. Aircraft, CUB, Paintings) as shown in Tables to 4. We believe that further training the CLIP backbone distorts the structure of its embedding space, leading to catastrophic forgetting on out-of-domain tasks. Conversely, CAML, MetaQDA, SNAIL, and GPICL all of which freeze the parameters of the CLIP feature extractor- benefit greatly from large-scale episodic pre-training on ImageNet- Ik, Fungi, MSCOCO, and WikiArt.
ANALYSIS
To better understand how CAML learns during inference, we analyze its ability to dynamically update its representations. Due to casting meta-learning as non-causal sequence modeling, CAML considers the full context of query and support set to predict the label of the query. Specifically, the query dynamically influences the representation of support set points, and the support set points dynamically influence the representation of the query this sequence is passed through the layers of non-causal sequence model. This property enables universal weta-lcarning by allowing the model to update the support and query representations based on the context of the task, not only the contents of the images, within the parameter space of the sequence model.
An example where the query dynamically influences the support set is visualized in Figure Given only the 5 support examples, the prediction task is ambiguous. However, the nature of the query determines the prediction task. The query image of a tower in Figure 2a reduces the task to generic object recognition: classify the query based on the object portrayed in the image. On the other hand, and as visualized in Figure 2b, the query image of embroidery reduces the prediction task to texture identification: classify the query based on artistic medium.
To analyze how dynamic representations affect CAML, we examine the representations of the support set and query vectors at the input to and output of the non-causal sequence model. For both examples visualized in Figure 2a and Figure 2b, the non-causal sequence model learns to separate support set vectors by class identity and group the query representation with the correct support set example.
We find the frozen CLIP image embeddings are actually antagonistic for the classification-be sertoon-b-ttttturr task visualized in Figure 2b: the query image embedding closest the support set example for
8

---

Published as a conference paper at ICLR 2024
Query
Support
CLIP Embeddings
CAML Embeddings
Chicken
Second Principal Component
Class
Second Principal Component
Class 2
Class
Flower
Human
Query (Cless 4)
First. Principal Component
First Principal Component
(a) Left: An example task mask-lassify images by the objects depicted. Center: image emheddings output from the Image Encoder (CLIP) in CAML. Right: joint image-label representations output by the non-causal sequence model in CAML for the same task.
Query
Support
CLIP Embeddings
CAML Embeddings
Embroidery
3D Model
Pencil Sketch
Class
Second Principal Component
Second Principal Componeut
Class
Class 4
Paisting
Glass
Query (Class
Embroidery
First. Principal Component
First Principal Component
(b) Left: An example task classify images by the artistic medium used. Center: CLIP image embeddings output from Image Encoder (CLIP) CAML. Right: joint image-label representations output by the non-causal sequence model in CAML for the same task.
Figure 2:
Two sample tasks over the same support images but utilizing different criteria define classes. The nature the query image informs task being presented, e-g. classification by object (top) vs. classification by texture (bottom). For both tasks, the output of the non-causal sequence model provides better separation among class representations than CLIP embeddings and groups the query representation with the proper task, even when projected into 2D space by PCA.
the second class, "oil painting". Unsurprisingly, the baseline methods that rely on frozen CLIP MetaQDA, ProtoNet and MetaOpt! and therefore misclassify this example. On the other hand, as CAML considers the full context the query and support set, it develops representations of the query in the context of the support set-and and the support set in the context of the query- to group the query with the "embroidery" support set image they share the same texture, thereby correctly classifying this example.
seneedddngg--pciffiiiin
the query with "oil painting"
CONCLUSION
In this work, we develop universal meta-learning approximate the performance of visual meta- leamnrrs deployed to a ChatGPT-like application and present CAML: : a meta- meeaa-leanning algorithm that emulates in-context learning in LLMs by learning new visual concepts during inference without fine-tuning. Our empirical findings show that MAMI--wihout meta-training or matches the performance of the current ofate-of-the-ar meta-learning algorithm on 8 out 11 benchmarks. This result indicates visual meta- meta-learning models ready for deployment in manner similar to LLMs, and we hope this work recalibrates our sense limitations for the universal meta-learning paradigm.
Nevertheless, there are areas where CAML struggles. Specifically, the performance of CAML on highly out-of-distribution images e.g. chest x-rays-and and varying image resolutions e.g. rescaled behind that of the best in-domain approaches. Developing methods for the universal setting that robust to these cases is promising direction for future work.
CIFAR

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We thank Mayee Chen, Dan Fu, Jerry Liu, and Benjamin Spector for their invaluable feedback and help during revisions of this work. We also thank Chelsea Finn for helping us improve the related work, Victor Butoi for constructive dialogue over Twitter, and the Hazy Group at Stanford as a whole their support throughout the research process. We gratefully acknowledge the support of NIH under No. U54EB020405 (Mobilize), DARPA under Nos. N66001 1924033 (MCS), NSF under Nos. OAC-1835598 (CINES), CCF-1918940 (Expeditions), DMS-2327709 (IHBEM), Nos. CCF2247015 (Wardware-Aware) CCF1763315 (Beyond Sparsity), CCF1563078 (Volume to Veloc- ity), and 1937301 (RTML); US DEVCOM ARL under Nos. W911NF-23-2-0184 (Long-context) and W911NF-21-2-022 (Interactive Human-Al Teaming); ONR under Nos. N00014231 12633 (Deep Signal Processing); Stanford HAI under No. 247183; NXP, Xilinx, LETI-CEA, Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASF, Accenture, Eriesson, Qualcomm, Analog Devices, Google Cloud, Salesforce, Total, Wu Tsai Neurosciences Institute, Chan Zuckerberg Initiative, Ama- zon, Genentech, GSK, Juniper Networks, KDDI, UCB, the HAI-GCP Cloud Credits for Research program, the Stanford Data Applications Initiative, and the Stanford Data Science Initiative (SDSI). The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions or recom- mendations expressed in this material are those of the authors and do not necessarily reflect the views, policies, or endorsements, either expressed or implied, of NIH, ONR, or the U.S. Government.
REFERENCES
Ivana Balazevic, David Steiner, Nikhil Parthasarathy, Relja Arandjelović, and Olivier Henaff. Towards in-context scene understanding. Advances in Neural Information Processing Systems, 36, 2024.
Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. Visual prompting via image inpainting. Advances in Neural Information Processing Systems, 35:25005- 2022.
Luca Bertinetto, Joao F Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with differen- tiable closed-form solvers. arXiv preprint arXiv:1805.0813 2018.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Victor Ion Butoi, Jose Javier Gonzalez Ortiz, Tianyu Ma, Mert Sabuncu, John Guttag, and Adrian V Dalca. Universeg: Universal medical image segmentation. pp. 21438-2145)) 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Wei-Y Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. closer look at few-shot classification. arXiv preprint 1rXiv:/904.044 04232, 2019.
Wei- Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at shw-shot classification, 2020.
Elliot Crowley and Andrew Zisserman. In search of art. In Computer Vision-ECCV 2014 Workshops: Zurich, Switzerland, September 6-7 and 12, 2014, Proceedings, Part 13, pp. 54-70. Springer, 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10. (09//CVR.20099500000
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 11929, 2020.
10

---

Published as a conference paper at ICLR 2024
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascal.
Matthew Fickus, John Jasper, Emily J King, and Dustin G Mixon. Equiangular tight frames that contain regular simplices. Linear Algebra and applications, 555:98-138, 2018.
Christopher Fifty, Jure Leskovec, and Sebastian Thrun. In-context learning for few-shot molecular property prediction, 2023.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International conference on machine learning, pp- 7704-7113 PMLR, 2018.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint 140.54440 2014.
Yunhui Guo, Noel C Codella, Leonid Karlinsky, James V Codella, John R Smith, Kate Saenko, Tajana Rosing, and Rogerio Feris. A broader study of cross-domain few-shot learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Parl XXVII 16, pp. 124-141. Springer, 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to learn using gradient descent. In Artificial Neural Networks Betworks-ICANN 2001: International Conference Vienna, Austria, August 21-25, 2001 Proceedings 11, pp. 87-94. Springer, 2001.
Shell Xu Hu, Da Li, Jan Stühmer, Minyoung Kim, and Timothy M. Hospedales. Pushing the limits of simple pipelines for few-shot learning: External data and fine-tuning make difference, 2022.
Ashraful Islam, Chun-Fu Richard Chen, Rameswar Panda, Leonid Karlinsky, Rogerio Feris, and Richard J Radke. Dynamic distillation network for cross-domain few-shot recognition with unlabeled data. Advances in Neural Information Processing Systems, 14:358435595 2021.
Norbert Jankowski and Krzysztof Grabczewski. Universal meta-learning architecture and algorithms.
Meta-learning in computational intelligence, pp. 1-76, 2011.
Lukasz Kaiser, Ofir Nachum, Aurko Roy, and Samy Bengio. Learning to remember rare events. arXiv preprint 110.:v1770003122 2017.
Donggyun Kim, Jinwoo Kim. Seongwoong Cho, Chong Luo, and Seunghoon Hong. Univer- sal few-shot learning of dense prediction tasks with visual token matching. arXiv preprint arXiv:2303. 14969, 2023.
Louis Kirsch, James Harrison, Jascha Sohl-Dickstein, and Luke Metz. perrroe-purpose in-context learning by meta-learning transformers. arXiv preprint arXiv:22/2.04448, 2022.
Jannik Kossen, Neil Band, Clare Lyle, Aidan N Gomez, Thomas Rainforth, and Yarin Gal. Self- attention between datapoints: Going beyond individual input-output pairs in deep learning. Ad- vances Neural Information Processing Systems, 34:28742 2275222-266... 2021.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-leved concept learning through probabilistic program induction. Science, (3N26266):133211333 2015.
Brenden M Lake, Tomer D Ullman, Joshua Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behuvioral and brain sciences, 40:e253, 2017.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization, 2019.

---

Published as a conference paper at ICLR 2024
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Part V 13, pp. 740-755. Springer, 2014.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv: 2013.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- 2017.
learner. arXiv preprint
Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transformers can do bayesian inference. arXiv preprint arXiv:2112.10510, 2021.
Jaehoon Oh, Sungnyun Kim, Namgyu Ho, Jin-Hwa Kim, Hwanjun Song, and Se- Young Yun. Understanding cross-domain few-shot learning based on domain similarity and few-shot difficulty. Advances in Neural Information Processing Systems, 35:2622-2636 2022.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 16652-246633 2020.
Cheng Perng Phoo and Bharath Hariharan. Self-training for few-shot transfer across extreme task differences. arXiv preprini urXiv:2010.07734, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-learning for semi- eemi-supervised few-shot classification. arXiv preprint arXiv: 0066000766 2018
Babak Salch and Ahmed Elgammal. Large-scale classification of fine-art paintings: Learning the right metric on the right feature. arXiv preprint arXiv:1505.0083 2015.
Adam Santoro, Sergey Bartumov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memery-augmented neural networks. In International conference on machine learning, pp. 1842-18500 PMLR, 2016.
Brigit Schroeder and Yin Cui. FGVCx fungi classification challenge 2018. github com/ http://cccxxouuioiieeeeeeaa
2018.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 15:25278-25209 2022.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning, 2017.
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2 10270, 2021.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
12

---

Published as a conference paper at ICLR 2024
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-201 dataset. 2011.
Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6830-6839, 2023.
Manfred K Warmuth and Dima Kuzmin. Randomized online pea algorithms with regret bounds that are logarithmic in the dimension. Journal of Machine Learning Research, ((Oct):2287-2322) 2008.
Lloyd Welch. Lower bounds the maximum cross correlation of signals (corresp.). IEEE Transac- fions on Information theory, 2033)397--39, 1974.
Davis Wertheimer and Bharath Hariharan, Few-shot learning with localization in realistic settings. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6558-6567, 2019.
Davis Wertheimer, Luming Tang, and Bharath Hariharan. Few-shot classification with feature map reconstruction networks, 2020.
Yibo Yang. Liang Xie, Shixiang Chen, Xiangtai Li, Zhouchen Lin, and Dacheng Tao. Do we really need learnable classifier at the end of deep neural network? arXiv e-prints, pp. arXiv 2203, 2022.
Xueting Zhang, Debin Meng, Henry Gouk, and Timothy Hospedales. Shallow bayesian meta learning for real-world few-shot recognition, 2021.
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. What makes good examples for visual in-context learning? arXiv preprint arXiv:2301 10222331111770000 13670, 2023.
13

---

Published as a conference paper at ICLR 2024
A
APPENDIX
A.1
SUPPLEMENTARY THEORETICAL ANALYSIS
We offer additional insight into the theoretical analysis presented in Section 4 and provide the omitted remarks, properties, lemmas, and proofs.
A.1.1 EQUIANGULAR TIGHT FRAMES
Papyan et al. (2020) coin the term Simplex Equianguar Tight Frame to describe a set of vectors { such that the minimum angle between any two pairs of vectors maximized and all vectors have equal norm. Formally,
Definition 3. Let T be d-dimensional inner product space over lR. with the Euclidean inner product. Simplex ETF is a set of vectors {Ør})== 03 E TRd, specified by the columns of I-T (1s - (117)
where Rdxd is the identity matrix and gdx1 is the ones vector. Somewhat contradictory, Simplex Equiangular Tight Frame is not an Equiangular Tight Frame (Welch, 1974) as this set of vectors does not form a tight frame Rd
Definition 4. Let I be a d dd-imensional space over with the Euclidean inner product. An Equiangular Tight Frame (ETF) is set of non-zero, equal norm vectors {0; (011ー=1 n 2 d, that achieves the Welch lower bound:
160 Φ3 | 1
d d(n 1)
max j7.
V
is well-known that set of non-zero equal-norm vectors satisfies the Welch lower bound if and only if that set of vectors is equiangular and also tight frame for IRd (Fickus et al., 2018).
Definition 5. A set of non-zero, equal norm vectors (Ø) ) is equiangular if # j Kéj 2,y)
for some c E R, c 0.
Definition 6. {0; is tight frame for Rd if. Vo E Rd. AA > 0 such that Allu|² Ej=1 [(0) v)
Remark 1. Simplex Equiangular Tight Frame is not frame.
Proof. Observe that for any finite d, for { equal to the columns of V V 1117). it
is the that Φj -1 th. So (06 D do not span Rdd and therefore, cannot be tight
frame.
Similarly, Simplex ETF is not d simplex.
Remark 2. Simplex Equiangular Tight Frame is not simplex.
Proof. A simplex in R. requires points.
To align terminology with properties, we generalize Simplex ETF to ELMES Definition 1: set of d vectors in a (d + -)ddimensional ambient space with 2 O. Observe that a regular simplex special type of ETF in which the number of vectors in the is one more than the dimension of the space that they span (Fickus et al., 2018). Building on this observation, intuitive view of ELMES is regular d simplex immersed in BR++
Consider a centered d-dimensional regular simplex with vertices {Øg] &3 E RRd+1 Let kean be the canonical inclusion map: Bd - Rd+1 tean (I1, T2, ..., Fd) (I1, ..., I. Fd, 0at1).
Remark 3.
then (rean ( dd1 is ELMES.
14

---

Published as a conference paper at ICLR 2024
1.0
0.5
0.0
-0.5
0.5
0.0
-1.0
0.5
X
-1.0
0.0
1.0
1.0
y
Figure 3:
A visualization a ELMES 1 Observe endpoints of vectors ELMES on the vertices of a centered regular tetrahedron.
Proof. The criteria of ELMES maximally equiangular and equal length. As all vertices of centered regular d d-simeee equal length from the origin, )+ equal length and therefore {eea (Øy) } must also have equal length.
Similarly. from Lemma 10 of Papyan et (2020), know the cosine of the angle between any two vectors in a +d+ 1) dimennsooa ELMES is 7. It is known that for a d-dimensionall regular simplex in R centered at the origin, the angle subtended by any two verticies through the origin is cos(O) = - Immersing {
dd+1 Φj E Rd, into Rd+11 via the canonical inclusion operator tean does not change the pairwise angle between vectors in this set: (0) Ozr) (Vean tean (amn(o) ()))).
As (Rean (0)) j=1 equal length and maximally equiangular, it forms ELMES.
We now show that an ELMES immersed in a higher dimension remains an ELMES. Taken with Remark 3, we can view high-dimensional ELMES in R& composed of + vectors {Ø)))=1 >>> n + 1, as simply a n simplxx immersed in Rd via the canonical inclusion operator.
Lemma 1. Let tean Idd + pd+k If [o an ELMES, then {1ean (o))) is an ELMES.
Proof. This reduces to proving that the maximum angle between a set of d equiangular points Rd is the maximum angle between set of d equiangular points in dd+ Let {Ø, ) be an ELMES such that Φy T and (ıb 1d be an ELMES such that th Rd++ Then { lie in didimensional subspace of Rddk, Eyi,... Ja and basis vectors 61, ed such that Vaby E (w
1 Ed me. Therefore, Vj + j', (W bby) S (0, Q5)) as (0,55- are an ELMES for Rdd
A.1.2 ELMES ROTATIONAL SYMMETRY
There are infinitely many ELMES by rotating one such set of vectors about the origin.
Remark 4. Let {Ør]5 I be an ELMES in Bddt for some & 2 O. Let o Rd+k - dd k be an operator from the special orthogonal group SO(d + k). Then {o(0;)}$_ is also ELMES
15

---

Published as a conference paper at ICLR 2024
Proof. Length is preserved operations in SO(d + k) have determinant and angles are similarly preserved as operations in SO + k) are unitary (i.e. preserving inner product).
A.1.3 A SET OF ORTHONORMAL BASIS VECTORS Is NOT AN ELMES
A final remark relates to the common misconception that a set of orthonormal basis vectors {0/ ( is ELMES. While {d5))/= is ETF in Ra since this set realizes the Welch lower-bound in Definition 4, these vectors are not maximally equiangular: (Vi 1br) = > T-.
A.2 ELMES MAXIMIZES pby (X - j)
Justification of Assumption 1. This property is implied by symmetry in the assignment of class embeddings to support classes. As the assignment is arbitrary, all learnable di class detectors should have equal probability of detecting their respective class. For simplicity of notation, we say the learns to detect class embedding o; rather another class embedding Φx. k +
Justification of Assumption 2. Informally, this property states that, for any m-subset of classes { the probability of 3 detecting class is equal to the probability of ei detecting class i. This is again implied by symmetry in the assignment of class embeddings to support classes as meta-learning algorithms may predict among subset of m classes in the support set rather than the maximum number of classes d.
L
Justification of Assumption 3. Recall in TRd, (v/ Φ) = || cos(O) where 0 is the angle between t and Φ;. Then this assumption constrains our set {Ø,} 1d so that relative norm of Φ; with respect to øj is lower bounded by cos()..;) || (0.s))
Informally, the { 1 are sufficiently spread out in the ambient space so that the learnable s/y that maximizes Po:, (X = i) is Φb itself: vi = | This constraint helps us avoid degenerative cases like {o,} 1 all equal. For example, o, aæ,, + j with 0 is such degenerative case where one class embedding vector is stacked on a different class embedding, but with higher norm.
Proof of Theorem I. Taken with Assumption 1, Assumption 2, and Assumption 3, it suffices to show Theorem and Lemma 4 to prove Theorem 1.
Theorem Pvi (X 1) Pa (X = 2) = Pva (X = dd + { are equiangular and equal norm.
To show the forward
(=) direction, it suffices to first show ps, (X = 1) = Por =
Pv. (X = d) + {Ø,) { are equal norm and then show Po: (X = 1) = Pab (X = 2)
Peu (X = d) + و are equiangular.
Lemma 2. Po., (X = 1) = pv. (X = 2) =.. -...:=Pa(AA (X = d) = (0r 3و= are equal norm.
Proof. This implication holds when 2:
elli1ll
|d>l|
Per (X 1) = ellا ll ellozi cos(01,2)
elleall 11 + ellexl cos(01.2 Pes (X = 2)
ollo (ellll) + ellen (0)())))))
@ + elloall c00(012))))
elder || n0s(01,2)
elldall- 1102 cos((
| + cos(( (0,2)) = 1l + cos (01.))
| |Ø2||
Suppose d > 2 and Pv, (X 1)
Px.(X d). By Assumption 2, all m-combinations (G) of {pv, (X 1), ..., Poa (X (X d)} are equal. This implies all 2-combinations are equal:
Pe. (X = Poy (X = j) || =||0/|| Therefore, || || = =||baa||-
n
16

---

Published as a conference paper at ICLR 2024
Lemma 3. Pei (X 1) Pvd (X =) Pv. (X d) = {Øك}ラ= equiangular.
Proof. This implication is trivially true when d = 2 (see the proof of Lemma 2), and we show it is similarly true when d 3. Following the steps in the proof of Lemma we arrive at the following pairs of equalities:
(1) elleal( (l0xll))+ + ellór I+ Iloall co [10xal(1+caxx00.0)) + e dall +all++ +ldal cos(01.a)
(2) e ((oxll((+ooo().a)) + elló, cos(02.3) = cl(bal(1=coo(9..3)) + ds +|ll+ cos
(3) [l + ell ellés ++ |1 com(0,,))
elløall(1
+ ++11+ cos(01,
From Lemma 2, Peb (X 1) Paz (X 2) = Pvs (X 3) + || = | |0sll. so the above pairs of equalities reduce to:
(1) cos (0s,)) = c0s(01.3)
(2) cos(02,3) = cos(O1,a)
(3) cos(O1,3) = (0s((1.2)
and when 3, [ are equiangular.
Suppose > 3 and Pvd (X = 1) = = Pv. = d). By Assumption 2, all m cocombiiatiinns (m) of {po, (X = 1). Px d)} are equal. This implies all 3-combinations are equal:
Pv. (X i) = Pv. (X = = Pvz (X = k)) = fi. = 0,, = 85.k. Therefore, all angles are equal
OLj Om. for i, v d.
Proof of Theorem 2. (+) Suppose Pth, (X = 1) Pvz (X =2) = -..=-uuu = d).
By Lemma 2 and Lemma 3, Pv. (X 1) = p (X = 2) equiangular and equal norm.
Pou (X = d) = { are
(+) Suppose {ør ورززهه are equiangular and equal norm. Let |ol|| be the norm of any vector in our set and cos(@) be the pairwise angle between any two vectors. Then
elliil
Ps. (X = (x=
ellell + (d - 1)e col()))()) = Pvs =
for any i d.
Lemma 4. For a set of equiangular and equal norm vectors, maximum equiangularity maximizes J (X, j).
Proof. The maximum pairwise angle between two vectors in Rd is and from Theorem 2
eliill
Ps (X= = (X = ])= = elliil + (d cos(()
for all 1 < أ S d. Increasing the angle 0 decreases cos(O). Decreasing cos(O) only decreases the denominator, which in turn, increases Pv. (X = i). Therefore, maximizing the pairwise angle between all vectors maximizes Pe,, (X i) for all 1 S
A.2.1 AN ELMES MINIMIZES H (X)
Proof of Lemma J. Equal norm and equiangular { are bounded norm, and thus, the set of
probability distributions we obtain {po, (1), Pv. (2), ..., Pv. (d)} belong to a capped simplex (Warmuth
!lo??
& Kuzmin, 2008) A = {p E A max; Pv. (k) < c} where e हाا (dd1)ellil Clearly, among such probability vectors, the minimum entropy is achieved at the boundary where cos( is minimized, i.e., when the [6,) maximally equiangular.
17

---

Published as a conference paper at ICLR 2024
A.2.2 AN ELMES MAINTAINS PERMUTATION INVARIANCE
Proof of Proposition 2. This follows from row-wise equivariance to permutations in matrix mul- tiplication. For any permutation π [I...., n!] + [_,...: n| applied to the rows of S", we have (S)) = (SS).
B EXPERIMENTAL SETTINGS
In this section, we describe our experimental settings, and further, we direct readers interested in reproducing or using any of the methods we benchmark in this work to our released code. Unless stated otherwise, all universal meta-leurning baselines use CLIP feature extractor to encode images.
Large-Scale Pre- Training. All methods evaluated in the universal meta-learning setting adhere to the same pre-training paradigm. For each large-scale image classification dataset, we reformulate the objective from typical supervised image classification to both a 5-way-I-shot and a 5-way-5-shot episodic prediction tasks. Within a dataset, examples from different classes are randomly sampled to compose a batch of episodes, and after exhausting iterating through every training example, this process is repeated with the next dataset. Iterating through each dataset in our set of ImageNet- Ik, Fungi, MSCOCO, and WikiArt then constitutes a single epoch of training.
ProtoNet and MetaOpt Implementations. For the ProtoNet and MetaOpt algorithms, we evaluate two settings. The first freezes the CLIP backbone and then applies the metric-learning objective- cosine distance for ProtoNet and SVM for MetaOpt-1o classify the query image from the unmodified CLIP embeddings. The second emulates P>M>F Hu et al. (2022) by fine-tuning the CLIP backbone during surge-scale pre-training with the metric-learning objective function. During inference, the metric-learning objective is applied to the fine-tuned CLIP embeddings to classify query images.
MetaQDA Implementation. We follow the MetaQDA algorithm presented in Zhang al. (2021). Specifically, we freeze the CLIP feature extractor backbone and train the MetaQDA classifier during large-scale episodic pre-training.
SNAIL Implementation. We use the architecture presented in Mishra et al. (2017) but with the hidden dimension of the Attention and Temporal Convolution Blocks adapted to CLIP embeddings rather than the ResNet embeddings used in the original implementation. As in this Mishra et al. (2017), we freeze the CLIP feature extractor and train the SNAIL, model parameters during large-scale pre-training.
GPICL Implementation. We adapt the GPICL algorithm presented by Kirsch et al. (2022) for episodic meta-training with an ELMES label encoder. Specifically, we represent image feature vectors as CLIP embeddings and the label embeddings with an ELMES. Following Kirsch et al. (2022), we form sequence by concatening the current CLIP image embedding with the previous example's ELMES label embedding and add learnable positional embeddings so the model can use positional information of elements the sequence classify the query point in causal-like fashion. We set the Ceneral-Purposs In-Context Learning Transformer model to a ViT-Large (Dosovitskiy et al., 2020) with leranable positional embeddings.
CAML Implementation. The image encoder is set to CLIP and the label encoder is an ELMES. For the non-causal sequence model, we use ViT-Large as described in Table of Dosovitskiy et al. (2020). This size is chosen as it has a hidden dimension of 1,024 and the CLIP output embedding vectors have hidden dimension of size 768. Choosing a non-causal sequence model with a large hidden dimension allows to concatenate the label embedding to the CL.IP embedding; in this case, the label embedding a 256 dimensional ELMES. In total, the implementation of CAMI. used for empirical evaluation has 302 million trainable parameters.
Optimization Settings. Following the recommendation of training Vision Transformers (Steiner et al., 2021) well as standard practices, all universal meta-learning approaches use cosine learning rate schedule with 9,600 warmup steps increasing linearly from 0 to le-5 followed by cosine decay to 1 over the subsequent 360,000 steps. Given the size of our pre-training datasets, apply dropout, attention dropout, or weight decay regularization. We select a batch size of 525 so the 5-way-1-shot episodes contain 520 query predictions and the 5-way-5-shot episodes contain 500
do not
18

---

Published as a conference paper at ICLR 2024
Table 5: MinilmageNet & CIFAR-Is mean accuracy and standard error across 10,000 test epochs.
CITAR-
MinilmageNet
Method
Sw-1s
5w-58
5w-1s
5w-5s
CAML [ELMES Class Embedding]
70.8+.2
85.5
96.2+.1
98.6t.00 98.7++.0
CAMI. [I camaale Class Fmheeding]]
71.1+.2
85.9+.1
96.1+.1
Table 6: CUB & tiered-ImageNet & Aircraft mean accuracy and standard error across 10,000 test epochs.
CUB
tiered-Imagedn ImageNet
Aireraft
Method
5w-1s
5w-5s
5w-ls
5w-5s
5w-1s
5w-5s
CAML [ELMES Class Embedding]
91.8 2
97.1
95,4.1
98,1
63.3:..3 66.3+t.2
79.1 80.6+.2
CAMI. [I.earnable Class Embedding)
91.8+.2
97.1+.1
95.3+.1
98.3+t11
Table 7: Pascal & Paintings mean accuracy and standard error across 10,000 test epochs.
Pascal Paintings
Paintings
Pascal
Method
5w-1s
5w-5s
5w-1s
5w-5s
5w-1s
5w-5s
CAML [ELMES Class Embedding]
63.8+.2
78.3+
51.1+.2
65.2+t11 65.022.1
82.6 2
89.7
CAML [Learnable Class Embedding]
63.1#.2
78.01.1
51.31.2
82.1
89.71.1
Table 8: meta-iNat & tiered meta-iNat & ChestX mean accuracy and standard error across 10,000 test epochs.
meta-iNat
tiered meta-iNat
ChestX
Method
5w-1s
5w-5s
5w-1s
5w-5s
5w-1s
5w-5s
CAML [ELMES Class Embedding] CAML [Learnable Class Embedding]
91.2+ 2
96.3+.1
81.9
91.6+t.1
21.5
22.2+1:1
91.4:. 2
96.41.11
82.11.2
91.8
21.51.1
22.61.11
query predictions. Given the seale of the pre-training datasets-an the computation to train single model- we do not conduct any hyperparameter tuning.
P>M>F Tretn-rainn We follow the settings used by Hu et al. (2022) to evaluate P>M>E. Specifically, P>M>F uses DINO (Caron et al., 2021) feature extractor rather than a CLIP feature extractor as the authors of P>M>F found a DINO feature extractor to be preferrable. We refer readers Hu et al. (2022) for this comparison. For meta-training, we use the code released by Hu et al. (2022) and simply switch out the datasets to evaluate the In-Domain setting. Both the in-domain and universal meta-learning settings use the same test-set data; the difference is that P>M>F meta-trains on each training dataset before evaluating on the testing dataset of each benchmark.
C SUPPLEMENTARY ANALYSIS
ELMES Ablation. To supplement our theoretical analysis in Section 4, we train a version of CAML with learnable class embedding vectors in place of the fixed ELMES encoder. Given our analysis in Section 4, it is perhaps unsurprising we find whtt-w any constraints or limitations the class embeddings converge an ELMES. The average pair-wise angle between embedding vectors is 1.77 + 0.02 radians whereas the expected pairwise angle from an ELMES is 1.82. Similarly, the average norm of the learnable class embeddings converges to 1.34 + 0.02 whereas the learned norm of the ELMES model is 1.32.
An evaluation comparing CAML with learnable class embeddings to the approach with a fixed ELMES encoder is presented in Table 5, Table 6, Table 7, and Table 8 of the Appendix. In summary, the performance is approximately the same on each benchmark with the exception of Aircraft. In this case, the learnable embedding model significantly outperforms the ELMES model, and moreover, surpasses all other universal meta-learning baselines on the 5-way-1-shie 1-sot split with an accuracy of 66.3 + 22. Nevertheless, given the similarity between both approaches on the remaining 10 datasets,
19

---

Published as a conference paper at ICLR 2024
Table 9: MinilmageNet CIFAR-Is mean accuracy and standard error across 10,000 test epochs. indicates mean and standard error across 2,500 test epochs.
CIFAR-fs
MinilmageNett
Method
5w-1s
5w-5s
5w-1s
5w-5s
CAML (ResNet34)
61.8 +
79.4 +
94.7 + .1
98.
CAML (ViT-busc)
70.8+.2
85.5+.1
96.2+
98.6+.0
CAML (ViT-huge))
83.3+.4
93.5+.2
98.6#t11
99.6+.0
Table 10: CUB & tiered-ImageNet & Aireraft
mean accuracy and standard error across 10,000 test epochs. o indicates mean and standard error across 2,500 test epochs.
CUB
tiered licred-ImageNet
Aircraft
Method
5w-1s
5w-5s
5w-1s
5w-5s
5w-1s
5w-5s
CAML (ResNet34)
75.4 + .2 91.8+.2
88.3 + .1
96.1 + .1
98.5 .0
45.1 + .2
58.7 + .2
CAML (ViT-base)
97.1
95.41:1
98.1#.11
63.3:t.3
79.1
CAML (ViT-huge))
95.8+1.2
98.71.11
96.8
98.8
81.8
92.1
Table 11: Pascal & Paintings mean accuracy and standard error across 10,000 test epochs. indicates mean and standard error across 2,500 test epochs.
Pascal + Paintings
Paintings
Pascal
Method
5w-1s
5w-5s
5w-1s
5w-5s
5w-1s
5w-5s
CAML (ResNet34)
57.5 + 63.8
71.0 +
46.1 +
57.3 .11
77.4 + .2
86.8 +
CAML (ViT-base)
78.3
51.1
65.2
82.6
89.7
CAML (ViT-huge))
66.4
81.0
54.7.1.
69.9
83.41.4
90.1#t.3
Table 12: meta-iNat & tiered meta-iNat & ChestX mean accuracy and standard error across 10,000 test epochs. indicates mean and standard error across 2,500 test epochs.
meta iNa
tiered meta-iNat
ChestX
Method
5w-ls
5w-5s
5w-ls
5w-5s
5w-Is
5w-58
CAML (ResNet34)
82.4 + .2 91.2:1.2
91.4 + .1
72.3 +
84.6 +
21.8 + .1
23.6 + .1
CAML (ViT-base)
96.3:..1
81.9
91.61.1
21.5
22.2
CAML (ViT-huge)
94.6:t.3
97.9+t.1
89.3
95.6
21.6
22.0+
and the learnable class embeddings actually forming an ELMES, attribute the difference in Aircraft performance to stochasticity in training the model, suggesting that the fixed ELMES encoder indeed optimal.
Image Encoder Ablation. To evaluate how the performance of CAML is affected by the pre-trained image encoder, evaluate CAML with a ResNet-34 image encoder pre-trained on ImageNet- Ik, ViT-base image encoder pre-trained with CLIP, and a ViT-huge image encoder that is pre-trained on Laion-2b (Schuhmann et al., 2022). We use the open source models released by Hugging Face in our evaluation.
As indicated in Table 9, Table 10, Table 11, and Table 12, the performance CAML scales with the strength of the feature extractor. Specifically, the performance with ResNet-34 feature extractor significantly worse than the performance with CLIP ViT-base feature extractor, and turn, the performance with CLIP ViT-base is significantly worse than the performance with Laion-2b ViT-huge feature extractor. However, its unclear what facet the improved feature extractor is relevant for CAML, especially out-of-distribution tasks like Aireraft where the most benefit is seen. Moreover, it is unclear why there is no improvement on another out-of oft-of-distree distrribbttoonnn dataset, ChestX.
20

---

Published as a conference paper at ICLR 2024
t-SNE Plots of Image Encoder Embeddings
VIT-b CLIP ChestX
VīT-b CLIP Aireraft
75
25
20
-50
-0
-50
-40
-20
20
-40
-30
-20
-10
10-
20
ViT-h Laion-2b ChestX
ViT-h Laion-2b Aircraft
50-
0-
-25
-20
-100
-40
Figure 4: I-SNE projections of different image embeddings of various benchmark datasets with embeddings colored class identity. We see ViT-huge trained with Laion-2b better separates the Aircraft dataset than does ViT-base trained with CLIP. However, both image encoders unable separate ChestX.
To investigate this dimension, we visualize the image embeddings of both Aircraft and ChestX using t-sne (Van der Maaten & Hinton, 2008) dimensionality reduction. Figure 4 visualizes these embeddings colored by class identity. We find the ViT-huge model pre-trained on Laion-2b better separates the Aircraft dataset than the ViT-base model pre-trained using the CLIP objective; however, both models do not reasonably separate ChestX. We postulate that an image encoder that can capture the axes of variability among image embeddings crucial for strong CAML performance, and the reason we observe significantly improved results on Aircraft but not ChestX when using a Laion-2b ViT-h image encoder.
Taken together, these results indicate CAML is modular: as foundational model feature extractors continue improve, CAML will be able to capture these advances to improve its own performance.
Assignment of Labels to Support Set Classes Analysis. Symmetry to the assignment of labels support set classes is a desirable property few-shot learning algorithms. For instance, the predictions for [(bear, 1), (tower, 2), (tree, 3)] should be the same if the labels are permuted to [(bear, 3), (tower 1), (tree, 2)). CAML. not invariant to permutations in the assignment of classes to support set examples implied by eq. (1) in Section 4.2; however,
empirically find it is robust them. Label symmetry distinct from the permutation invariance property of CAML that is discussed in Section 4.3. Tangibly for the sequence [(bear, 1), (tower, 2), (tree, 3)). permutation invariance ensures the predictions are the same as if the order of demonstrations is permuted to [(tower, 2), (tree, 3), (bear, 1)].
In Figure 5(left), we visualize the histogram of the correct class probability for the example presented in Figure 2a after permuting the assignment of labels support-set images for all 120 permutations
21

---

Published as a conference paper at ICLR 2024
Histogram Plots of Variability to Permutations in Label Assignment
Distribution of P(Tower) after Permuting Label Assignment from Task in Figure 2a
Average Std. of Correct Class Probability after Permuting A Assignment in mini-ImageNet
30
25
89
20
Counc
15
20-
20
5
0.575
20
10-2
10
Sufbnax probabiitt correct
correceclass probabiltty
Figure 5: (Left) histogram of the correct class probability for the example presented in Figure 2a after permuting the assignment of labels to support-set images for all 120 permutations the 5-way-1-sher task. (Right) histogram of the average standard deviation of all 120 permutations of the 5-way- I-sht task for 1,000 samples from mini- Amai-lmaeeNet.
of the 5-way-1-shot task. In Figure 5(right), visualize the average standard deviation of all 120 permutations of the -way-1-shor task for 1,000 samples from mini-Image ImageNet. The mean of this statistic is 0.004 0.0004. Taken together, this indicates CAML is empirically robust to permutations in the assignment of labels to support set classes.
D DISCUSSION
Weaknesses of CAML. Despite its strong empirical performance, CAML presents several weaknesses. First, the maximum number of classes present the support set any point during inference must be known at pre-training to instantiate d-way ELMES. Further, at least one dataset during pre-training must use d-way classification setting so the th class detectors referenced Section 4 are trained within the Transformer encoder's attention layers.
Why does CAML not fine-tune the image encoder during pre-training? do not fine-tune image encoder because it is not advantageous for universal meta-learning.
Our goal to develop a meta-learning algorithm that may function in a ChatGPT-like application; should be able
in-context learning any set of images. Foundational image models are trained for exactly this purpose: they are pre-trained on billions of images to form well-structured image embedding space that is robust augmentations, occlusions, such as the presence of objects, textures, etc. of an image are encoded into the structure of the embedding space so that the of variability among the embeddings encode variation in specific visual attributes.
Moreover, valuable characteristics
Fine-tuning the image encoder corrupt this embedding space; especially since the datasets use for pre-training are orders of magnitude smaller than the ones used to train the Foundational model. This hypothesis is supported by our experiments with ProtoNet and MetaOpt in Tables Specifically, we find fine-tuning the backbone during pre-training leads to performance degradation on many of our benchmarks when evaluated in the universal meta-learning setting.
22


Reviews:
[{"review_id": "iUAwFrS2Ys", "reviewer": "Reviewer_8Akm", "summary": "This paper addresses a gap in the field of visual meta-learning, where models have traditionally struggled to learn new visual concepts during inference without fine-tuning, a capability that Large Language Models (LLMs) like ChatGPT have demonstrated in the textual domain. The authors introduce a novel meta-learning algorithm inspired by the in-context learning of LLMs. This approach treats n-way-k-shot image classification as a sequence modeling over known labeled data points and an unknown test data point.", "questions": "Please see the Weaknesses", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written.\n2. The problem studied in this paper is interesting and valuable.\n3. The theoretical work of this paper is sufficient, which improves the value of the paper.", "weaknesses": "1. The authors utilize the CLIP model to encode both images and labels. An area of potential exploration is why they didn't attempt to encode context and images directly, especially using datasets like MSCOCO.\n2. In the experiments, CAML's performance on out-of-domain tasks is notably weak. This might be primarily due to the treatment of unseen categories, which are uniformly encoded as \"Unknown [class] Embedding\".\n3. The study lacks ablation experiments for its various modules, and there's an absence of quantitative analysis for hyperparameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iHOPYtdXLp", "reviewer": "Reviewer_GdKg", "summary": "The paper introduces a new “universal meta-learning“ setup that “avoids meta-training on the train/validation splits of meta-learning benchmarks or fine-tuning on the support set during inference.” Instead, the paper attempts to recast meta-learning as a sequence modelling problem, where meta-testing on new tasks is analogous to in-context learning in large language models. The proposed approach CAML, context-aware meta-learning, uses CLIP image representations, together with one-hot label encoding dubbed as “Equal Length and Maximally Equiangular Set (ELMES) encoding,”  to represent each in-context learning example. The base sequence model is a Transformer encoder. It is trained to predict the query class label given in-context examples that are comprised of labelled support examples and a query example. The model is pre-trained on ImageNet-1k, Fungi, MSCOCO, and WikiArt and evaluated on 11 meta-learning benchmark datasets. Empirical results show that the proposed approach outperforms other “universal meta-learning“ baselines on 15 of 22 evaluation settings.", "questions": "1. Please summarize the novelty of this paper in relation to [1, 2].\n    \n2. Please respond to Weakness 2: The dichotomy between “meta-training“ and ”universal meta-learning.”\n    \n3. Please elaborate on how the transformer encoder is implemented and the rough scale of parameters it has.\n    \n4. Please elaborate on the pre-training dataset of CAML in “we pre-train CAML’s Transformer encoder on few-shot image classification tasks from ImageNet-1k.” How many examples are included in the pre-training set? Note that one of the benchmarks, miniImageNet, is a subset of ImageNet. Would this pretraining dataset result in task leakage?\n    \n5. Please elaborate on how the ProtoNet baseline is implemented. Is it trained on the same pre-training dataset but with the ProtoNet loss objective?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "1. The paper is well-motivated on the need for in-context learning by drawing analogies with large language models. I also liked the analysis in Fig. 2, which illustrates how dynamic in-context examples impact representation learning\n    \n2. Theoretical analysis of the “Equal Length and Maximally Equiangular Set (ELMES) encoding” presents an interesting analysis of label symmetry and permutation invariance in meta-learning.\n    \n3. The paper presents competitive empirical results on various meta-learning baselines.", "weaknesses": "1. Novelty: the paper does not discuss previous work that also formulated meta-learning as a sequence, or set, modelling problem [1, 2]. The problem formulations in [1, 2] are highly similar to the proposal in this paper, except for architectural differences in implementation. This weakens the novelty of this paper.\n\n2. The dichotomy between “meta-training“ and ”universal meta-learning”: the paper attempts to make the distinction between ”universal meta-learning” and \"meta-training\" in that the proposed CAML approach does not perform “meta-training“ or “fine-tuning on the support set.“ Instead, ”universal meta-learning” only performs pre-training. However, I think this dichotomy is not well-defined. Pre-training in the CAML fashion can be understood as learning across many different tasks in the pre-training dataset, i.e., meta-training, and in-context learning can be understood as performing implicit gradient descent based on the in-context examples. This dichotomy also implicates the comparisons in empirical results as CAML was mostly compared with other “universal meta-learning“ methods, i.e., ProtoNet, MetaOpt, and MetaQDA.\n\nAdditional related work:\n\n[1] Mishra, Nikhil, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. \"A simple neural attentive meta-learner.\" *arXiv preprint arXiv:1707.03141* (2017).\n\n[2] Lee, J., Lee, Y., Kim, J., Kosiorek, A., Choi, S. &amp; Teh, Y.W.. (2019). Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks. <i>Proceedings of the 36th International Conference on Machine Learning</i>, in <i>Proceedings of Machine Learning Research</i> 97:3744-3753 Available from https://proceedings.mlr.press/v97/lee19d.html.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0EVGJhmreR", "reviewer": "Reviewer_yxC1", "summary": "The paper proposes a meta-learning algorithm that learns new visual concepts during inference without fine-tuning. The method performs well on several benchmark datasets.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper draws a new perspective for meta-learning: learning to classify a query from a context of support set, imitating the way in LLMs. \n2. The framework is straightforward and clean. The reason (proved theoretically) for using the specific ELMES embedding is presented well.\n3. Extensive experiments and analysis are provided.", "weaknesses": "Apologies in advance I am not an expert in meta-learning. But I still have the following questions:\n1. What is the unknown class embedding initialization for the ELMES Class Encoder?\n2. As discussed by the authors themselves in section 5.3, the number of classes need to be known in advance and the frozen encoders limit the learning ability. However in my view, the need of number of classes is an inherent problem in few-shot learning. But finetuning more modules could be further discussed.\n3. There is a lack of experimental details (especially training details).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S2kK6aeuY8", "reviewer": "Reviewer_Hb5r", "summary": "This work introduces a novel meta-learning algorithm that allows visual models to learn new concepts during inference, mimicking the capabilities of LLMs such as ChatGPT. The technique utilizes a static pre-trained feature extractor and treats meta-learning similarly to sequence modeling with labeled and unlabeled data points. When evaluated on 11 benchmarks, the proposed method, without any meta-training or fine-tuning, outperformed or matched the leading P>M>F algorithm in 8 of those benchmarks.", "questions": "- Could the authors elaborate on what they consider to be the primary innovative contribution of their study?\n- It would be beneficial if the authors could include experiments to demonstrate the label symmetry and permutation invariance capabilities of the demonstrations as claimed.\n\n[1]. General-Purpose In-Context Learning by Meta-Learning Transformers, NeurIPs 2022\n\n[2]. Small Visual Language Models can also be Open-Ended Few-Shot Learners, Arxiv 2023.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Intriguing Research Question:** This paper delves into a significant question in meta-learning. The authors note that meta-learning traditionally involves pretraining, meta-learning, and fine-tuning. However, their approach seeks to bypass meta-learning and fine-tuning by transforming the learning process into a sequence modeling task and applying the in-context learning objective function.\n\n**Rigorous Numerical Performance:** The paper's data shows the model's performance to be both impressive and robust.\n\n**Novel ELMES Class Encoder:** The class embedding presented here appears to be quite innovative. I've noticed no other usage of this type of class embedding in in-context learning in NLP or VLM literature unless other reviewers bring up some status quo. This is also in alignment with some recent studies on Arxiv that indicate that class embeddings might be unnecessary, as arbitrary words, numbers, or first names can be used to label images (see [2]).", "weaknesses": "I will provide the paper's weaknesses in the following. \n- The concept of using the ICL objective function to pre-train a transformer model isn't a new one [1]. Unlike [1], which pre-trained a transformer from scratch within a meta-learning framework, this paper adapts the pre-training objective to Clip image embeddings, which doesn't significantly enhance novelty.\n- The primary innovative aspect highlighted in this paper is the ELMES Class Encoder. While this feature is intriguing, it narrows the scope of innovation in the study.\n- Some claims in the paper lack experimental backing. For instance, the assertion that the ELMES Class Encoder upholds label symmetry and is invariant to the permutation of demonstrations isn't convincingly proven with data.\n- The mathematical explanations in Section 4 are challenging to follow. A clearer, more comprehensible presentation of this section would be beneficial. Until then, I'm relying on other reviewers to verify the accuracy of the mathematical derivations presented.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)