Meta Learning to Bridge Vision and Language Models for Multimodal Few-Shot Learning
Authors: Ivona Najdenkoska
Xiantong Zhen
Marcel Worring
Venue: ICLR 2023 poster (2023)
Keywords: multimodal
few-shot learning
meta-learning
transformers
vision and language models

Abstract: Multimodal few-shot learning is challenging due to the large domain gap between vision and language modalities. Existing methods are trying to communicate visual concepts as prompts to frozen language models, but rely on hand-engineered task induction to reduce the hypothesis space. To make the whole process learnable, we introduce a multimodal meta-learning approach. Specifically, our approach decomposes the training of the model into a set of related multimodal few-shot tasks. We define a meta-mapper network, acting as a meta-learner, to efficiently bridge frozen large-scale vision and language models and leverage their already learned capacity. By updating the learnable parameters only of the meta-mapper, it learns to accrue shared meta-knowledge among these tasks. Thus, it can rapidly adapt to newly presented samples with only a few gradient updates. Importantly, it induces the task in a completely data-driven manner, with no need for a hand-engineered task induction. We evaluate our approach on recently proposed multimodal few-shot benchmarks, measuring how rapidly the model can bind novel visual concepts to words and answer visual questions by observing only a limited set of labeled examples. The experimental results show that our meta-learning approach outperforms the baseline across multiple datasets and various training settings while being computationally more efficient.

Published as a conference paper at ICLR 2023
META LEARNING TO BRIDGE VISION AND LANGUAGE MODELS FOR MULTIMODAL FEW-SHOT LEARNING
Ivona Najdenkoska', Xiantong Zhen' Marcel Worring' AIM Lab, University of Amsterdam, IInception Institute of Artificial Intelligence fi.najdenkoska, x. x.zhen, m. n.worring) @uva.n1
ABSTRACT
Multimodal few-shot learning is challenging due to the large domain gap between vision and language modalities. Existing methods are trying communicate visual concepts as prompts to frozen language models, but rely on sand-engincered task induction to reduce the hypothesis space. To make the whole process learnable, we introduce a multimodal meta-learning approach. Specifically, our approach decom- poses the training of the model into set of related multimodal few-shot tasks. We define a meta-mapper network, acting as meta-learner, to efficiently bridge frozen large-scale vision and language models and leverage their already learned capacity. By updating the learnable parameters only of the meta- mapper. it learns to accrue shared weta-knowledde among these tasks. Thus, it can rapidly adapt to newly presented samples with only a few gradient updates. Importantly, it induces the task in a completely data-driven manner, with no need for a hand-engineered task induction. We evaluate our approach recently proposed multimodal few-shot benchmarks, measuring how rapidly the model can bind novel visual concepts words and answer visual questions by observing only limited set of labeled examples. The experimental results show that our meta-learning approach out- performs the baseline across multiple datasets and various training settings while being computationally more efficient.
INTRODUCTION
Learning quickly by observing few examples in a multimodal environment is an integral part of human intelligence (Schmidhuber, 1987; Bengio et al., 1991). Yet, it is quite challenging for current vision and language models to deal with a limited labeled space, when performing multimodal few-shot learning (Tsimpoukelli al., 2021; Alayrac et al., 2022). On the contrary, language-only models have already flourished over the past years, especially when being transferred to a limited labeled space (Brown et al., 2020; Perez et al., 2021; Min et al., 2022), result of the large-scale pre-training and huge model capacity. Such advances in natural language processing inspired similar efforts in the vision domain, yielding large vision models with impressive few-shot and zero-shot image classification capabilities (Radford et al., 2021; Zhai et al., 2021; Jia et al., 2021). However, due the large domain gap between vision and language modalities. it is non-trivial to directly embody few-shot capabilities into multimodal settings, which is the main motivation of this work.
One of the main challenges for bridging this gap is finding proper mechanism for communicating visual concepts to a language model, by accruing shared knowledge from sequences of multimodal tasks. The Frozen model (Tsimpoukelli et al.. 2021) is the first multimodal few-shot learner which tackles this challenge by taking inspiration from language models able to do in-context learning (Brown et al., 2020). This requires task induction provided as sentence followed by context data samples, to reduce the hypothesis space for open-ended image interpretation. This might not be a shortcoming for simpler tasks, like binary decisions; however it becomes an obstacle for more complicated tasks (Li & Liang, 2021), since the task induction has to be engineered each time. We hypothesize that the task can be induced from the data itself in a completely learnable manner.
Meta-learning or learning to learn (Schmidhuber, 1987; Thrun & Pratt, 2012; Andrychowicz et al., 2016) comes as a natural solution to any few-shot settings. While it has been extensively studied in unimodal settings, particularly for few-shot image classification (Vinyals et al., 201 6; Ravi &

---

Published as a conference paper at ICLR 2023
Meta-training
Inference (Meta-test)
Ti.1 27
Support sct
217
support SEt
Support EeE
pupport
NO VaNe
shott
viy Labar!
steddngh
back ensorrpell
Tell
Theil gaden
fy
fe
fo
fo
souncies
Motia Joumeer
campleiis
fø
fg
f
&. Golcen retriaver.
rats upoia
updar
Learrer
query
euery carolla
A-2 ragped vin Anade wihojaade
Iuc.
Figure I: Multimodal few-shot meta-lcarning task for an example of 2-way I-shot setting, with two categories (ways) present in the support set images, each represented with one sample (shot). Given batch of tasks Tr, the support set is first used obtain task-specific model parameters 6, for each task by few gradient-step updates, which are then used together with the query set samples to perform a meta-update step for updating the meta-parameters After the meta-training is finished, for new given task, the meta-trained model is used for inference by further adapting the meta-learned model with the support set, and measuring the performance on unseen query samples.
Larochelle, 2017; Finn et al., 2017; Raghu et al., 2019; Ye et al., 2020), meta-learning remains almost unexplored for multimodal few-shot settings. Similar to unimodal settings, empowering multimodal few-shot learner with the ability to accrue knowledge across tasks, would assist in building internal multimodal representations broadly suitable for many tasks. These representations could serve bridge between the different modalities and assist in learning quickly new tasks by observing only limited labeled examples.
Motivated by this, we define a novel multimodal few-shot meta-learning approach, to bridge the gap between vision and language modalities, illustrated in Figure Specifically, our approach adopts publicly available pre-trained large vision encoders (Radford et al., 2021) and language models (Brown et al., 2020), which are kept frozen to take advantages of their already-learned reasoning capability (Tsimpoukelli et al., 2021; Mokady et al., 2021; Zhou et al., 2022; Jia et al., 2022; Tewel al., 2021; Zhai et al., 2021). doing so, our method avoids the huge computational resources required by those models during training and their dependency on large datasets. Unlike prior multimodal few-shot learners (Tsimpoukelli et al., 2021; Alayrac et al., 2022), our approach decomposes the training of the model into sequential observing of multimodal few-shot tasks, in the spirit of meta-learming. During the meta- meta-training stage, the model translates the visual representations into visual prefix for the language model, by using lightweight meta-mapper network. This network serves as a multimodal bridge between the large vision and language models and is entirely built from self-attention layers (Lee al., 2019). Essentially, the aim of the meta-mapper is to collect shared meta-knowledge from related tasks, by mapping the visual representation into the latent space of the language model. Then, during inference, or meta-test according to the meta-learning parlance, the model is able to induce the task in a fully data-driven manner by observing few labeled examples and thus entirely removes the need for aand-engineered task inductions.
In summary, we contribute in three major aspects: Conceptually: We introduce meta-learning to multimodal few-shot learning, which enables fast adaptation and efficient learning of multimodal few- shot tasks. To that end, we design new setting for multimodal few-shot learning by re-organizing existing datasets and following suitable benchmarks. Methodologically:: We present a multimodal meta learner by using lightweight meta-mapper, which learns to bridge a large frozen vision and language backbones. To the best of our knowledge, this is the first meta-learning based model to solve multimodal few-shot tasks. Empirically: We demonstrate through systematic experiments those benchmarks that our model, using only the small trainable meta- mapper with frozen backbones, yields strong multimodal few-shot performance, while being computationally more efficient.
2

---

Published as a conference paper at ICLR 2023
2 RELATED WORKS
Large-scale language models emerged since the introduction of attention (Bahdanau al., 2015) and Transformers (Vaswani et al., 2017) to successfully deal with long-range dependencies in sequences. The field has seen significant progress over the last years (Devlin et al.. 2019; Radford al., 2019; Brown et al., 2020), which also initiated the development of similar strategies for vision (Dosovitskiy et al., 2021; Radford et al., 2021; Zhai et al., 2021) and multimodal models (Lu et al., 2019; Li et al., 2020; Wang et al., 2021; Jia et al., 2021; Tewel et al., 2021; Mokady et al., 2021; Eichenberg et al., 2021; Shen et al., 2021). In current few-shot scenarios, the language models are steered into producing desired output by using the concept of prompting (Brown et al., 2020; Tsimpoukelli et al., 2021; Alayrac et al., 2022). The approach is to prepend fixed task induction with a few examples as a prompt and then generate the output from the language model. Instead of optimizing over fixed set of task inductions and examples, also known as in-context learning, we aim to optimize the context representation (Li & Liang, 2021; Lester et al., 2021), in the form of meta-learned visual prefix.
Meta-learning for few-shot learning (Ravi & I Larochelle, 2017; Santoro et al., 2016b; Finn et al., 2017; Vinyals et al., 2016; Snell al., 2017) addresses the fundamental challenge of generalizing across tasks with limited labelled data. The aim is learning to acquire inductive biases and perform fast adaptation to new tasks (Grant et al., 2018). Meta-learning algorithms are typically categorized as: (i) metric- based, focusing on learning common embedding space and deriving prototypes as meta-knowledge (Vinyals et al., 2016; Snell et al., 2017: Sung et al., 2018) (fi) memory-based, using an external memory meta-knowledge to quickly adapt to new tasks (Mishra et al., 2018; Santoro al., 201 6a; Munkhdalai & Yu, 2017; Munkhdalai er al., 2018) and (iit) optimization-bn to learn good model initialization across tasks meta-knowledge. which can be used to efficiently adapt to new samples (Ravi & Larochelle, 2017; Finn et al., 2017; Grant et al., 2018). Our approach is positioned in this last category, it offers greater flexibility and is modality
aiming
Multimodal few-shot learning across vision and language modalities has emerged recently with the introduction of the first multimodal few-shot learner Frozen (Tsimpoukelli et al., 2021), followed by other innovative prompting and in-context learning strategies (Song et al., 2022; Jin et al., 2021). Particularly, Frozen performs in-context learning, which is considered as of the possible approaches to deal with multimodal few-shot learning settings. Another recently proposed model named Flamingo (Alayrac et al., 2022) follows similar in-context learning paradigm, by using risual-language model of 70 billion parameters, which is proven to be successful due to its scale. Our goals differ since our aim is to perform fast adaptation to new tasks by acquiring across tasks, with a lightweight model of less than two million trainable parameters. In particular, we define optimization- baaase meta-learning steps (Ravi Larochelle, 2017: Finn et al., 2017; Grant et al., 2018), by using the context samples to adapt a small trainable mapper and evaluate on the query samples.
knowlege
3 METHODOLOGG
We aim to train model that learn and quickly adapt to new multimodal tasks with limited labeled data under the meta-learning setting. In the next sections, we will first define the multimodal meta-learning setting, then explain our architecture, and finally describe how it is used during training and inference time.
3.1 PROBLEM FORMULATION
We start by defining our approach for multimodal few-shot settings, following a meta-learning approach. Different from standard supervised learning, in meta-learning we are dealing with a collection of meta meta-datasets, which are split into disjoint D Detta-train and D D partitions. We define a meta-training stage, where the model is trained on the meta-train Dmeta-train partition, and separate meta-test stage where the performance is measured on the Dmeta- partition. In particular, the meta-tras train D Dmeta-train set consists of imta-ddttsse datasets D,, each one containing a pair of separate inner-train set D.",, also referred to support set and inner-test set D referred to as a query set. This means that D Dimeta-tra train = {(D] Di").... (Drr D(.)) We refer to each meta-dataset pair (Dfr, D1*) meta-task T,, following Finn et al. (2017).
3

---

Published as a conference paper at ICLR 2023
Mapper
visual profix
Vision encoder
Language model
Language model Generator
This golden
This is golden retriever.
encoding
decsiing
The architecture of the multimodal meta few-shot learner. It consists of three parts: frozen vision encoder na; frozen language model with a text embedder Ie and a generator Iw: and a meta- mapper with trainable meta-parameet 0. In the example shown, the model is generating the last word retriever, an autoregressive manner.
Figure 2:
When considering k-shot, N- -Vwayy setting, the support set D.rT of single meta-task Tr consists of labeled samples for each of the N N-ways, where N is the number of object categories i.c. ways in few-shot parlance, meaning D.r {(z]. vi)... (i ) where 2} represents the image and yj represents the caption. The query set D is defined D {(2), 9i af ... (it an a }}, where o is the image, Im is an optional slot for an input text representation, like a question, and a is the output text representation,
an answer to the question.
We consider a distribution over tasks p(T), where a single batch of tasks T: ~ P(T) ) for the k-shot N-way problem, sampled the following fashion: N object categories are randomly sampled from the pool of all meta-train trrin samples. For the support set there are l samples chosen for each of the categories, and for the query set there are m samples per category, with m >
3.2 MODEL ARCHITECTURE
The architecture that present in this paper is modular and consists of three components: a vision encoder, the meta-mapper and a language model, illustrated in Figure 2. is trained in fully autoregressive manner, meaning that it offers flexibility for the choice of downstream generative multimodal task.
Vision encoder network The vision encoder is defined function Ug. with fixed parameters Rdd taken from pre-trained vision encoder. The input is raw image x and the outputs are the extracted visual features Uo (x) = 31 T..
Meta-mappen network To map the visual encoding into the latent space of the language model in our multimodal few-shot setting, we use set of learnable parameters E Rd= These parameters also called visual prefix for the language model, with the same dimension d, the language embeddings. We prepend this visual prefix to the encoded visual features, yielding the following sequence (P1 ...... T1.... 2n). Particularly, we view this representation as an ordered of elements and we employ self-attention to simultaneously encode the whole set. For this purpose, we adopt the set muet attention block (Lee al., 2019), which has the role of meta-mapper with trainable meta-pare parameters 0, and is defined as:
(1) where the pairwise dot-product QK" measures the similarity amongst features, and is used for feature weighting computed through an activation function a. Intuitively, each feature of V will get more weight if the dot- dot-product between and K is larger. In Eq. 1, we have Q K V [pi... Pt, I.,... Jn meaning that the input of the meta-mapper is the sequence of features, and the output vector of learned parameters pi PK. namely the visual prefix for the language model, as expressed in:
MetaMap, (Q, V) o(QKT)+ +
Pi ... p MetaMape([PL
Pt. ...
21... xx,))
The self-attention layer, as the building block of this module, allows to retrieve meaningful information from the visual features T1. Fn. and accumulate it into pi ... due to its inherent weighting mechanism pairwise similarities between elements in the sequence. The meta marameers of the meta-mapper are learned and shared across all tasks T in D metta- train-
4

---

Published as a conference paper at ICLR 2023
Language model The language model is defined function Iw, which parametrizes probability distribution over a text sequence y. It consists of an embedding function Jg. with 3/ E R to embed each word token Yi of the sequence into a word token embedding t, followed by Transformer block conducting an autoregressive text generation. More precisely, the language model receives the visual prefix pi--. ... P% concatenated with the token embeddings f1, l. and outputs the next tokens conditioned on the prefix in an autoregressive manner:
4 g ... pi ti...... t.)) i m.
Since approach aims to learn in a limited labeled space which might distort the learned parameters we update them, we initialize the w parameters with pre-trained parameters from large-scale language model and keep them entirely frozen.
3.3 META-TRAINING & INFERENCE
To meta-train the model, we sample batches of multimodal few-shot learning tasks Ti p(T), which consist of a support set denoted as D and a query set D Here, for simplicity, we assume that our full model described in 3.2 is defined as a function fo. which receives an image x as input and produces y as output. The loss function, optimized per task during training, a cross-entropy loss, defined as:
(To(
Σ xiyYnD!!"
log Jo(x)) + (1 log - fo(x')).
When adapting new task T; the trainable meta parameters @ become task uesk-specific parameters. namely 6.. These task-specific parameters are computed with N gradient-step updates, similar as in (Finn et al., 2017), with the following rule for one gradient update: 0 = aVaLT, (fe). This is referred as the inner-loop update, where is the hyperparameter for the step size. Next, the model meta-parameters 0 are optimized for the performance of fs;- using the query set D samples and the
lask-specific parameters 0g as initialization of the model:
min
Σ CT. = Σ
LT. LT(lo-aveer,(1)))
(5)
X D!!
x1 XDDDD
which is called outer-loop optimization. The meta-optimization across all tasks Tú is performed using stochastic gradient descent update rule, as follows: 0 + + - V Exi LT, (fo;), where is the step size hyperparameter.
In the meta-test stage, we consider new multimodal few-shot tasks Tn with previously unseen objects, which also have support set: D for fast adaptation by fine-tuning the model meta- ceta-parammttess 0 to a given task, and a query set D to evaluate the model on this task. Note that the generation of the answer for each query sample is done in open-ended autoregressive manner. Specifically, use top-k nucleus sampling (Holtzman et al., 2019) to sample from the language model given the visual prefix of the image.
EXPERIMENTS
4.1
EXPERIMENTAL SETUP
Datasets To design a meta-learning setting for multimodal few-shot learning, the datasets have to be structured into sequences of tasks, as explained in section 3. The datasets introduced in Frozen (Tsimpoukelli et al., 2021) are representative examples for this type of multimodal few-shot datasets, where each task consists of few labeled instances as support set and unlabeled ones as query set. In practise, any dataset can be suited for few-shot meta-learning, as long as there is an available object information based on which the tasks can be constructed. Therefore, for meta-training, we use the COCO2017 captioning dataset (Lin et al., 2014 4) and restructure it to construct tasks in N-way, k-shot manner based on the N object categories present in the images. For meta-test, we use the four mentioned datasets, namely, Real-Name minilmageNet, Real-Fast VQA. Sspen-Ended minilmageNet and Fast- VQA datasets, as they test the characteristics that make good multimodal few-shot learner, such fast binding of visual concepts to words and adaptation
tasks.
5

---

Published as a conference paper at ICLR 2023
Table 1: Comparison with the Frozen (Tsimpoukelli al., 2021) baselines on Real-Name and Open-Ended minilmageNet 2- and 5-way setting; expressed in accuracy(%)). ANIL (Raghu et al., 2019) is used as an upper bound, since it is a discriminative approach as opposed to our open-ended generative one. Our episodically trained models are outperforming the Frozen baselines, both with and without domain-shift. The overall best performance is denoted bold, whereas the same settings as the baseline are denoted in italic & bold.
Real-Name 2-way
Open-Enden 2-way
Real-Name 5-way
Open-Ended 5-way
Methods
episodic
cross domain
1-shot
5-shots
I-shot
5-shots
1-shot
5-shots
1-shot
5-shots
Prozen
1.7
29.0
18.0
Frozen st task ind
/
33.7
66.0
53.4
58.9
14.5
33.8
20.2
21.3
x
35.6
63.7
50.2
$7.5
15.2
39.6
18.9
22.0
Ours
37.3
66.0
52.5
59.0
19.2
40.3
20.9
25.0
/
45.3
69.8
53.6
63.4
24.7
41.8
24.8
28.5
x
48.2
72.3
58.7
65.8
29.0
43.2
25.1
29.6
ANIL upperbound
73.9
84.2
45.5
62.6
Table 2: Comparison with the Frozen (Tsimpoukelli et al., 2021) baselines on 2- and 5-way Real- Name and Open-Ended minilmageNet for the I-shot setting, with different number repeats (reps); expressed in accuracy(%), Increasing the repeats in the support set does not show huge benefit compared to when the number of shots are increased. The overall best performance denoted in bold, whereas the same settings the baseline denoted in italic & bold.
Real-Name 2-way
Open-Ender 2-way
Real-Name 5-way
Open- 5-way
Methods
episodic
cross domain
1-rep
5-reps
I-rep
5-reps
I-rep
5-reps
1-rep
5-reps
Frozen w/ tsk ind
/
63.0
63.7
51.1
58.5
33.8
32.8
21.4
20.9
x
36.0
36.7
$1.2
52.0
15.5
15.9
18.5
19.0
Ours
/ 46.8
47.5
56.4
58.0
26.7
27.1
26.5
28.1
x
62.1
64.3
59.8
60.8
31.3
33.7
26.8
27.5
Training & inference procedures We consider two different training procedures: (i) standard mini-batched training, which we refer to as a non-episodic; (ii) our proposed procedure by observing batches of multimodal tasks, which we refer to episodic training: These training procedures are considered within two scenarios determined by the domain shift emerging during training and inference: (i) cross-domain multimodal few-shot learning, when the training and test datasets are of different distribution. Here the model is trained on COCO2017 (both in episodic and non-episodic manner) and the performance is measured the full multimodal few-shot datasets; (ii) in-domain multimodal few-shot learning (standard meta-learning! setup, where the training and test partitions are of the same domain. This means that the training and test partitions, namely meta- meta-training and meta-test tasks are constructed from the same multimodal few-shot dataset.
Implementation details The vision encoder implemented as CLIP (Radford et al., 2021) with the Vision Transformer (ViT/B-32) (Dosovitskiy al., 2021) as backbone model, yielding visual features of size 512. The language model is implemented as GPT-2 (Radford et al., 2019), with word length with dimensions
embeddings of size 768. The learnable visual prefix is empirically of 768, which are specific to the language model. The meta- mapa-ma is initialized following Xavier uniform initialization (Glorot & Bengio, 2010). For the meta-learning specific hyperparameters, we empirically determined to have five inner-loop gradient-updates with learning rate of 0.01. The meta-update performed with Adam (Loshchilov Hutter, 2018) with meta-learning rate of 0.001 and 4 tasks in each batc. All hyperparameters are tuned using the query sets in the meta-training partition which acts as a validation set.
Regarding the computational resources, our model is trained end-to-end on one NVIDIA GTX 1080Ti GPU, in less than 2 hours, showing the benefit of the lightweight framework. The total number of trainable parameters is approximately two million, which is order of magnitudes lower than the Frozen model. The full implementation is in PyTorch and HuggingFace (Wolf et al., 2020) and is publicly realized at https: / / thub. com/ vona denkoskk / /mul t modaa -me learn.

---

Published as a conference paper at ICLR 2023
Table Comparison with the Frozen baseline (Tsimpoukelli al., 2021) on 2-way Real-Fast VQA and Fast-VQU in accuracy($)). Our episodically trained models outperform their counterparts, both with and without domain shift.
Real-Fan Reat-Fast
VQA
Fast- Fast-VQA
Methods
episodic
cross-domain
I-shot
5-shots
I-shot
5-shots
Frozen
X
7.8
10.5
2.8
7.9
X
5.4
9.1
2.5
7.1
X
6.9
10.7
3
8
Ours
8.5
13
5.2
8.6
9.7
13.2
5.7
9.3
Table 4: Benefit of the meta- meta-knowledge in accuracy(%) on the four multimodal few-shot datasets
under the 2-way setting, Accruing meta- weeta-knowledge crucial for our approach.
Real-Name
Open-Ended Ended
Real-Fast VQA
VQs
Variants
I-shot
5-shots
1-shot
5-shot
I-shot
5-shots
1-shot
5-shot
w/o meta-knowledge
0.0
0.06
0.0
0.05
0.0
0.03
0.0
0.06
w/ meta meta-knowledge
48.2
72.3
58.7
65.8
9.7
13.2
5.7
9.3
4.2 RESULTS & DISCUSSION
Binding of visual concepts and words The experiments conducted on Real-Name and Open- minilmageNet measure to what extent the multimodal meta-learner is able to bind visual concepts to words. Table I shows the 2-way and 5-way accuracy in and 5 shots both datasets. We observe that our multimodal meta-learner is able to largely outperform Frozen, even without using an engineered task induction. This shows the advantage of having meta-learned visual prefix. in contrast to just reshaping the vision encoder output as a prefix to language models. Specifically, the meta-learned prefix is able collect shared meta-knowledge from related instances in the tasks, which useful for narrowing down the search space in learnable manner, instead of using an engineered task instruction. Having this adaptable component, makes easier for the model later adapt to unseen visual concepts, without being explicitly instructed to do so. Also, similar to Frozen, we observe an upward trend when increasing the number of shots, for both the 2-way and 5-way settings, which confirms that having more distinct examples of the task increases the performance.
We believe that the open- open-ended approach is promising due to its flexibility reasoning about visual concepts in an unconstrained manner, instead of relying a pre-defined closed set of concepts. However, due to the magnitudes larger hypothesis space in the open-ended text generation (the full vocabulary of the language model), compared to the finite set of possible answers of conventional classifiers, such as ANIL (Raghu et al., 2019), it is not possible to perform a fair comparison to them. Therefore, use their results upper bound to our approach.
Increasing the number of repeats of support samples with the number of times each shot is repeated in the support set presented to the model, during inference i.e. the meta-test time. We observe from Table that increasing the repeats in the support set, does not show as large benefit as increasing the number of distinct shots. This means that the meta- learner able to accumulate more useful keta-knnowedde by seeing different examples of images, only once, rather than multiple repetitions of the same ones. Contrary to this observation, Frozen benefits more from these repetitions since they are used in combination with a hand- -ani-neneeeered task induction, which is crucial for their method.
an additional analysis, we experiment
Visual question-answering with limited labeled samples The aim of the experiments on the Real-Fast and VQA 2-ways benchmarks, presented in Table 3 with 1 and 5 shots, is to evaluate the abilities of the multimodal meta-learner to answer more complex queries about the objects in the image. There is an implicit testing of the binding of visual concepts and words, since the query samples are designed in such a way to contain both categories from the support set in the query image, while the question is addressed in one of them. Moreover, the emphasis is the ability of the meta-learner to reason about particular object properties, without being explicitly meta-trained to do As observe from the results, our multimodal meta-leamer achieves improvements over the
7

---

Published as a conference paper at ICLR 2023
This is i??
Q: Thia #? Ground-truths This trifle. Ours This trifle.
what breed cst this? Ground-truthe Fersian. Persian bread.
@: What does the dog ridu? Ground-truth blue surtboard. Ours: surfboard.
Ground -tuutthth This electric
electrio
Figure 4: Qualitative examples of query set images from Real-Name minilmageNNe (first two) and Real-Fast VQA (last two). with their question, ground-trun -ound and answers generated by our model.
Frozen baseline, showing once more the benefit of accruing meta-knowledge across tasks. Again, there is an upward trend in the performance when increasing the number of shots.
Episodic training of multimodal few-shot learners Another important observation from Tables 1 and 3 is that the episodic training improves the overall performance. The simple reason for this having matched training and inference time conditions, as in standard supervised learning. More precisely, the multimodal few-shot datasets that are employed at inference time, are structured into tasks, so training the model in a similar manner by observing the tasks improves the performance.
Learning across domains Tables and 3 provide insights into the more challenging cross-domain multimodal few-shot setting. Essentially, meta-training on the COCO2017 captioning dataset, helps the model to bind visual concepts with richer semantic descriptions, compared to when training on the multimodal few-shot datasets (which have simpler captions). However, according to the evaluation procedure that we follow, also employed by Frozen, the model expected to generate the exact word as the ground-truth, penalizing any paraphrasing of words. Therefore, COC02017 7 meta- trained model exhibits lower accuracy from a quantitative perspective when transferred to a minilmageNet-basee dataset. However, in many cases the generated sentences are qualitatively better and more detailed, as we discuss the next paragraph.
Qualitative analysis In Figure 4, we show a examples of query images with the questions and answers infer- ence time, by the best version of our approach in Tables I and 3. The capability of the multimodal meta-learner to bind visual concepts to words is apparent: the model is able connect the visual concepts in the image not only to electric guitar, as stated in the ground-truth but also the word player. This demonstrates that the model is correctly steered by the meta-learned visual prefix to leverage additional information about image, not necessar- ily present in the ground-truth. An important observation from these examples is that the generated answers have some differences w.r.t the truth, which is however penalized during evaluation, since we count as correct only words that match, to be able to compare Frozen. The evaluation should, however, measure whether the gener- ated sentence matches the image in more general manner, ideally without using foound-truth references (Hessel et al., 2021), which is left for future work.
Reai-ame munimacafitt [hvaes]
05
Aocume Th0
20
Figure 3: Relationship between consecu- tive steps of suddient-based updates and the accuracy on Real-Name minilma- geNet on 5-way tasks, performed during the meta-test stage.
4.3 ABLATION STUDY
What is the benefit of accruing meta-knowledge? To test how useful the meta-knowledge is, we run an ablation experiment where the meta-knowledge accrued from the previous batch of tasks is erased before observing the next one. This essentially means to randomly initialize the meta-mapper, which collects the meta-knowledde. before observing the next batch of tasks. As it can be observed in Table 4, this erasing yields deteriorating performance, across all datasets, ways and shots. This shows that training the meta-mappe to accumulate useful representations from sequences of tasks is crucial for the success of meta-approach.
8

---

Published as a conference paper at ICLR 2023
Table 5: Benefit of the self-attention structure of the meta-mapper in accuracy (%) on the four multimodal few-shot datasets under the 2-way setting. Self-attention meta-mapper is more critical than the MLP-based one for our approach.
Real-Name
Open-Ended
Real-Fast VQA
Fast- Fast-VQA
Variants
1-shot
5-shots
1-shot
5-shot
I-shot
5-shots
I-shot
5-shot
MLP-based meta- seta-mapper
42.5
62.6
45.3
58.3
4.3
11.9
4.5
7.5
Self-attention meta-mapper
48.2
72.3
58.7
65.8
9.7
13.2
5.7
9.3
Table 6: Effect of using a fixed, hand-engineered task induction vs. only learning it from support sets, in accuracy (%) on the four multimodal few-shot datasets under the 2-way setting. Learning the task induction in data-driven manner improves the performance.
Real-Name
Open-Ended
Real-Fast VQA
Fast-VOr VQA
Variants
1-shot
5-shots
1-shot 5-shot
1-shot
5-shots
1-shot 5-shot
w/ fixed task ind
47.9
71.9
58.0
65.0
9.5
13.2
5.0
8.9
w/o fixed task ind
48.2
72.3
58.7
65.8
9.7
13.2
5.7
9.3
What the effect of the multimodal inner loop optimization? To analyse the effect of the gradient-based updates during test time, namely the inner loop optimization, we illustrate the rela- tionship between the update steps and the accuracy Figure 3. We observe increased performance as the number of gradient-based steps increases, for {1, 3, 5) shots on 5-way Real-Name minilm- ageNet tasks. This demonstrates that the mechanism of the meta-learning framework to adapt the meta-parameters on support samples, allows the model to respond promptly to new task, which yields increased performance on the query samples.
Does the structure of the meta-mapper influence the performance? To test the performance of our approach with a different structure of the meta-mapper, we run experiments with an alternative to the proposed self eeff-attention-bsse one. Specifically, we implement a multi-layer perceptron (MLP) with one hidden layer. can be observed from Table 5, that the self-attention meta-mapper brings notable improvements over MLP. This is attributed to the fact that, different from MLP, self-attention layers are able to select and extract the most relevant features from an input representation.
Is there an effect of fixed task induction in the meta-learning setting? We analyse the effect of a fixed hand-engineered task induction, as opposed learning the induction a data-driven manner. In particular, we choose the best variant of our meta-learner from Table 1, and add fixed sentence describing the task in natural language, for instance, "Answer with lion or dog. similar as Frozen. It can be observed from Table 6. that there is no significant increase in the performance, meaning that only fine-tune tuning the meta- mapper on the support set is good enough to induce the task.
CONCLUSION
We present a novel meta-learning approach for multimodal few-shot learning. We design our model by introducing lightweight trainable meta-mapper that acts as a bridge between large frozen vision and language models. This meta-mapper accrues shared meta- meta-knowledge by sequentially observing tasks into a learnable visual prefix, which is then used to steer the language model into generating relevant linguistic interpretations. Unlike prior works, we use the support samples to adapt our model to unseen tasks, which shows to be beneficial as it improves performance without using hand-engineered task induction. Moreover, it operates with low computational costs, it smartly combines large pre-traine pre-trained models without the need to train them from scratch. This makes our architecture flexible enough to incorporate additional modalities, which is left to tackle in future work. Last but not least, our experiments verify the effectiveness of our approach by outperforming the baseline on several benchmarks and fostering further exploration of multimodal few-shot meta-learning.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
Our proposed model uses pre-trained large-scale models, both for the vision encoder and language decoder modules, namely CLIP (Radford et al., 2021) and GPT-2 (Radford et al., 2019). So, our model inherits any problematic biases and limitations that these pre-trained models may contain, meaning we should be careful when using the results especially in sensitive settings. More specifically,
model has generative nature with the ability to produce language, which might lead to generation of disinformation or potential hallucinations. But, as our model conditional on the visual features, it enforces the text generation follow the image contents. This serves control mechanism of the otherwise free text generation, and can mitigate the generation linguistic disinformation. Additionally, can be seen in some of the qualitative examples in Figure 4, our proposed approach takes advantage of the complementarities of the two backbone models. This suggests that might be useful to mitigate biases potentially present in one of the pre-trained trained models.
REPRODUCIBILITY STATEMENT
To help the reproducibility of the paper, we provide the formal definition of the meta-learning setup in Section I. the implementation details in Section 4. additional details about the design of the setting and the formal definition of the algorithms during the meta-training and inferences stages in Appendix. Also, we release the source code to reproduce the results and evaluate the performance of the model at: https: / /tttt/tthh//t// com/ vvona daanassaaaaaaaaaaaaaaaaaaa imoda -met a-
ACKNOWLEDGMENTS
This work is financially supported by the Inception Institute of Artificial Intelligence. the University of Amsterdam and the allowance Top consortia for Knowledge and Innovation (TKIs) from the Netherlands Ministry of Economic Affairs and Climate Policy.
REFERENCES
Lean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204. 14198, 2022.
Marein Andrychowiez, Misha Denil, Sergio Gomez, Matthew Hoffman, David Pfau, Tom Schaul, Brendan Shillingford. and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, 29, 2016.
Dzmitry Bahdanau, KyungHyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. ICLR, 2015.
Y. Bengio, S. Bengio, and J. Cloutier. Learning a synaptic learning rule. In IJCNN-91-Seatle International Joint Conference on Neural Networks, volume pp. 969 vol.2-, 1991. doi: 10.1109 19CNN,1991.1553 1556211.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Jacob Devlin, Ming-War Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference the North, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021.
10

---

Published as a conference paper at ICLR 2023
Constantin Eichenberg. Sidney Black, Samuel Weinbach, Letitia Parcalabescu, and Anette Frank. multimodal Magma-multimodal augmentation generative models through udapter-based finetuning, arXiv preprint arXiv:21122052553 2021.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agmostic meta-learning for fast adaptation of deep networks. International conference on machine learning. pp. 1126-1133 PMLR, 2017.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth International Conference Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pp. 249-256, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient- based meta-leariigg as hierarchical bayes. ICLR, 2018.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference- free evaluation metric for image captioning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021. doi: 10. 1/253/v1/20211 wttp:/2212.mmmm..mmmmmmmmm
comnlp-main.55 URL http:: / /xx. doi. orgg 0 /613/vll3vv/lle /2021 emn 1 1--pmmmm ma i n.m.mmm... 595.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint
2019.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- lsiounlanguage representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904-4916. PMLR, 2021.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. arXiv preprint arXiv:2203.12119, 2022.
Woojeong Jin, Yu Cheng, Yelong Shen, Weizhu Chen, and Xiang Ren. A good prompt is worth millions parameters: Low-resource prompt-based learning for vision-language models, 2021.
Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set trans- former: A framework for coantion-based permutation-invariann neural networks. In International Conference Machine Learning, pp. 3744-3753. PMLR, 2019.
Brian Lester, Rami A1-Rfou, and Noah Constant. The power of scale for parameter-efffcient prompt tuning. Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, http::///// dx doi 1ttpg//...115/vv1
2021. doi: 10. 1/2021 wtt.://.222.ppppp...... .wwnleemnmmmmm. www.lpmmai.n25......... 2021. caannlp-mainn 243.
Xiang Lisa Li and Perey Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00100 2021.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vasion-language tasks. In European Conference on Computer Vision, pp. 121-137. Springer, 2020.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In International Confer- ence on Learning Representations, 2018.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining agno-tii visiolinguistic representations for vision-and and-language tasks. Advances in neural information processing systems, 32, 2019.
11

---

Published as a conference paper at ICLR 2023
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannanch Hajishirzi. Metaicl: Learning to learn in context. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2022.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- leamner. ICLR, 2018.
Ron Mokady, Amir Hertz, and Amit Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint (tl.::22111007200 2021.
Tsendsuren Munkhdalai and Hong Yu. Meta networks. In ICML, pp- 2554 2563, 2017.
Tsendsuren Munkhdalai, Xingdi Yuan, Soroush Mehri, and Adam Trischler. Rapid adaptation with conditionally shifted neurons. ICML, 2018.
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. Advances in Neural Information Processing Systems, 14:11004-1107, 2021.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ily Sutskever. Language models unsupervised multitask learners. 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International Conference on Machine Learning, pp. 8748 1763.876.. PMLR, 2021.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. In International Conference Learning Representations, 2019.
Sachin Ravi and H. Larochelle. Optimization as a model for few-shot learning. In ICLR, 2017.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory-augment neural networks. 2016a.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory-augmented neural networks. In ICML, 2016b.
Jürgen Schmidhuber. Evolutionary principles in self-referential learning, learning how learn: the meta-meta-.. hook. PhD thesis, Technische Universität München, 1987.
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit vision-and and-language tasks? In International Conference Learning Representations, 2021.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
Haoyu Song. Dong, Wei-Nan Zhang, Ting Liu, and Furu Wei. Clip models are few-shot learners: Empirical studies vqa and visual entailment, 2022.
Flood Sung, Yongxin Yang. Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1199-1208. 2018.
Yoad Tewel, Yoav Shalev, Idan Schwartz, and Lior Wolf. Zero-shot image-to-text generation for visual-semantic arithmetic. arXiv preprint axXiv22111.14447, 2021.
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 2012.
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34:200-212, 2021.
12

---

Published as a conference paper at ICLR 2023
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. ArXiv, abs/2108. 10904, 2021.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-an art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro- cessing: System Demonsirations, pp. 38-45, Online, October 2020. Association for Computational Linguisties. URL httpt::////.. wwww ac lweb. org/ anw.aooohooooootooooooo http:///...2.2.....mmeemm....mmm
6.
Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pp. 8808-8817, 2020.
Xiaohua Zhai, Xiao Wang. Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. arXiv preprint arXiv:2111.07991, 2021.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, 2022.
13


Reviews:
[{"review_id": "_5yfpoP27E", "reviewer": "Reviewer_idxn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper propose a meta mapper network to tackle the task of multi-modal few-shot learning. Particularly, the proposed meta mapper contextualizes the soft visual prompts with the features of a pre-trained/frozen visual encoder and language models, for adapting the language model to output for the target task. During the inference, only the meta-mapper is fine-tuned via gradient based optimization, to adapt the model for the target multi-modal learning tasks. To train this meta-mapper, MAML-style of meta learning routine is employed for training. With this key idea, this paper achieves strong results on Real-Name mini-ImageNet and Open-Ended mini-ImageNet.", "summary_of_the_review": "Overall I think this paper is making a solid contribution to the multi-model few-shot learning. The proposed method is interesting enough and effective. ", "strength_and_weaknesses": "Strength:\n\n+ The main idea of combining soft-prompt tuning and MAML is very straightforward, and novel to some extent.\n+ The empirical results are strong, and show that the proposed method outperforms the previous state-of-the-art in multimodal few-shot learning.\n+ The ablation studies are quite comprehensive, and most of my questions about the methods was addressed. \n\nWeakness:\n- I don't see very obvious weakness of the methods. \n\nQuestions:\n\nOne question I have is about the variance of the model's performance, with respect to the few-shot support examples. Figure 3 has shown same hints but it could be interesting to more variance analysis on realistic VQA data. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EL2G6WfQ64s", "reviewer": "Reviewer_c2Z3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new approach to few-shot multimodal learning using a meta-learning approach that relies on generating visual prefixes that leverage prior training with other tasks. The visual prefix is produced by the proposed meta-mapper that meta-learns from previous tasks. The prefix approach enables seamless bridging of the textual and visual modalities. The proposed approach is computationally lightweight since it uses frozen backbone visual and text encoders (i.e. does not require retraining those), and clearly advances over the state of the art in its results. The paper also presents several insightful ablation results that establish the significance of the proposed approach to meta-learning.", "summary_of_the_review": "Well written paper. Clear advancement of the state of the art with both the approach and experimental results. Insightful results from ablation. Good paper.", "strength_and_weaknesses": "Strengths:\n1. Thorough and insightful literature survey that provides a sound motivation for the proposed approach.\n2. Clear explanation of both the proposed approach and the experimental results.\n3. Sound technical approach that is computationally lightweight and applicable to multiple multimodal tasks.\n4. Experimental results show clear advance over the state of the art.\n5. Insightful ablation results show the advantage of the proposed approach.\n\nWeaknesses\n1. No significant weaknesses. A suggestion - have you thought of how you would bridge three modalities? That is not required for this paper but I would be curious to know your thoughts. For example, how would you bridge visual, text and audio modalities? My guess is that your approach will scale up easily to accomodate more modalities but do think about it.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Bkw7rN3CYw", "reviewer": "Reviewer_zhwM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to combine meta learning with few-shot adaptation of language models, by essentially building upon methods such as Frozen (Tsimpoukelli et al., 2021) which maps image representations into language model token space, but adding a meta-mapper network, acting as a meta-learner, to more efficiently perform this mapping. The proposed approach can rapidly adapt to newly presented samples with only a few gradient updates, and shows strong results on recently proposed multimodal few-shot benchmarks.", "summary_of_the_review": "The novelty of the paper is limited, since it seems to be a combination of Frozen (Tsimpoukelli et al., 2021) and standard meta-learning approaches to learn the mapping from image representations into language model token space.", "strength_and_weaknesses": "Strengths:\n1. The paper is well motivated and described. The ideas are clear and experiments are on several large multimodal datasets.\n2. The paper is very clear with clear figures and exposition.\n3. Experiments are comprehensive and results are strong.\n\nWeaknesses:\n1. The novelty of the paper is limited, since it seems to be a combination of Frozen (Tsimpoukelli et al., 2021) and standard meta-learning approaches to learn the mapping from image representations into language model token space.\n2. There should be more comparisons in both performance and complexity - the proposed method does better but certainly uses more computation so the tradeoff should be analyzed.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AgeVNcUWskN", "reviewer": "Reviewer_VtRH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed to use meta learning for multimodal few-shot learning. Unlike previous methods that may train fully or partially the vision or language models, this paper freezes both vision and language models while only training the learnable parameters for the meta-mapper. Thanks to the frozen backbones, the meta-learning process is fast and able to make use of the pre-trained knowledge. The experimental results show that the proposed method outperformed the existing Frozen multimodal learning baseline. \n", "summary_of_the_review": "This paper proposed a simple idea to bridge pre-trained vision and language backbones via meta-learning. The training is parameter efficient and fast thanks to frozen backbones. Concerns regarding the \"cross-domain\" comparisons lead to borderline rating. \n", "strength_and_weaknesses": "Strength: \n* The idea itself is very simple and easy to understand, for an impactful research direction. Making use of already pre-trained vision/language models could help save computational cost.\n* This paper is very well written and is very easy to understand. The illustrations of the idea are very clear. \n* Good qualitative results and ablation tests.\n\nWeaknesses:\n* It is appreciated to mark both \"episodic\" and \"cross-domain\" in Table 1 and Table 2. It is clear that the proposed \"episodic\" training helps on the quality significantly on target tasks. However, \"cross-domain\" leads to unfair comparisons, where the Frozen baseline only reported results with \"cross-domain\" setup while this paper highlights results with non \"cross-domain\" (\"in-domain\") setups. I am concerned by this point and if we look at \"episodic\" and \"cross-domain\" row in Table 2, this paper falls behind Frozen overall. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster