MetaReVision: Meta-Learning with Retrieval for Visually Grounded Compositional Concept Acquisition
Authors: Guangyue Xu
Parisa Kordjamshidi
Joyce Chai
Venue: EMNLP 2023 Findings (2023)
Keywords: Compositional Learning
Meta-Learning
Retrieval-enhance Learning
Visual-Language Models

Abstract: Humans have the ability to learn novel compositional concepts by recalling primitive concepts acquired from past experience and generalizing these primitive concepts to novel compositions. 
Inspired by the above human’s compositional learning procedure, in this paper, we propose MetaReVision, a retrievalenhanced meta-learning model to solve the visually grounded compositional concept learning problem. 
The proposed MetaReVision consists of a retrieval module and a meta-
learning module which are designed to incorporate retrieved primitive concepts as supporting set to meta-train visual-language models for grounded compositional concept recognition. 
Through meta-learning from episodes constructed by the retriever, MetaReVision learns a generic compositional representation that can be fast updated to recognize novel composi tional concepts. 
We create CompCOCO and CompFlickr to benchmark the grounded compositional concept learning. 
Our experimental results show MetaReVision outperforms other competitive baselines and the retrieval module does plays an important role in this compositional learning process.

Vision: Meta-Learning with Retrieval for Visually Grounded Compositional Concept Acquisition
Guangyue Xu Michigan State University
Parisa Kordjamshidi Michigan State University kordjams@msu. edu
Joyce Chai University of Michigan chaijy@umich, edu
xuguang3emsu.edt edu
Abstract
is driving
Humans have the ability learn novel compo- sitional concepts by recalling and generalizing primitive concepts acquired from past expe- riences. Inspired by this observation, this paper, we propose MetaReVision a retrieval- enhanced meta-learning model address visually grounded compositional concept learn- ing problem. The proposed MetaReVision consists of retrieval module and meta- learning module which are designed to incorpo- rate retrieved primitive concepts as a support- ing set meta-train vision-language models for grounded compositional concept recogni- tion. Through meta- meta-lcarning from episodes con- structed by the retriever, PetaReVisson learns a generic compositional representation that can be fast updated to recognize novel composi- tional concepts. We create CompCOCO and CompFlickr to benchmark the grounded com- positional concept learning. Our experimental results show that MetaReVision outperforms other competitive baselines and the retrieval module plays an important role in this compo- sitional learning process.
down the
man tuble
with red
bean sating togather.
Learnt Element Concept: red,
Compositional Concoct chalr
An illustration Grounded Compositional Concept I. Learning(GCCI.). For example, given concepts (red, bus) and (old, chair) in the training data, the goal learn to predict novel compositional concepts (red, chair) masked token prediction at test time.
Figure 1:
chair, from the training data. The trained models then applied to predict novel compositional con- cepts e.g., red chair in the testing phase although these concepts were never seen during training.
The ideal vision-language system should have the compositional ability to solve the GCCL prob- lem. Recently, significant efforts have been made to the development of pre-training vision- language models (VLMs) (Tan and Bansal, 2019; Su et al., 2020; Radford et al., 2021). These VLMs have demonstrated impressive performance in various downstream tasks, including Visual Question An- swering (VQA) (Li et al., 2020), Vision-Language Navigation (VLN) (Hao et al., 2020) and image captioning (Zhou et al., 2020). Despite their suc- cess related fields, remains unclear whether these models can truly perceive the world a com- positional manner or generate language compo- sitionally to cooperate with humans in a shared physical world. Such composition-relater ques- tions important from both the theory and the application perspectives. From the theory perspec- tive, compositional learning allows the model process and understand objects by breaking them down into smaller, interpretable units. Therefore, compositional learning helps improve large models'
1 Introduction
Learning to compose from previous experience is an important integral part of human intelligence (Fodor and Pylyshyn, 1988; Biederman and Vessel, 2006). Generally, compositional learning refers to the ability to learn a of basic primitives and gen- eralize these primitives in novel scenario different from training time (Kemp and Tenenbaum, 2009; Ontanón et al., 2021). It includes various learning aspects, such as systematic generalization, produc- tivity and substitutivity (Hupkes et al., 2020). In this work, focus systematic generalization within the multi-modal setting and propose multi- modal compositional problem: Grounded Com- positional Concept Learning (GCCL). As shown in Figure 1, in the GCCL setting, the models are trained with primitive concepts, such as red and

---

efficiency and generalization (Andreas et al., 2016). From the application perspective, it is not realistic to give the model all possible compositions in train- ing data. For example, in Vision Language Navi- gation (VLN), is not feasible to observe a sofa with all possible colors c.g. red sofa and blue sofa. The vision-language models applied in VLN are ex- pected to recognize these compositions after learn- ing the element concepts Compositional learning can be viewed as special case of gero-shot learn- ing problems. Moreover, the domain-shiff problem is commonplace in zero-shot learning because the statistical distribution the data in the training set (seen compositions) and the testing set (novel com- positions) could be significantly different. While compositionality can be reliably interpreted by hu- mans, State-of-the-ee VLMs, which are trained on vast amounts of image-text pairs and employ di- verse loss functions, still encounter challenges in compositional learning (Ma et al., 2023b; Thrush et al., 2022).
learning from the episodes constructed by the re- triever, MetaReVision learns a generalized com- positional representation that can be fast updated for novel compositional recognition. We evaluate MetaReVision on the proposed CompFlickr and CompCOCO datasets. The empirical results show that coupling retrieval and meta-learning performs better in GCCL compared with previous baselines. Contributions of this work can be summarized as follows:
This work explores a novel angle of retrieval- enhanced compositional concept learning. The model relics on retricval to construct episodes for meta-learning. It addresses the domain-shift problem in compositional learn- ing by learning from the retrieved instances.
Two datasets are created to serve as bench- marks for grounded compositional concept learning. These datasets enrich existing zero- shot vision-language tasks, from the end-task level to the token-level.
To address these limitations, this paper takes a closer look at the compositionality in VLM with an attempt to improve its ability. More specifically, we create two grounded compositional concept learn- ing datasets, CompFlickr and CompCOCO curated from MSCOCO (Chen et al., 2015) and Flickr30K (Plummer et al., 2015), for VLMs' token-level compositional analysis. Moreover, MetaReVision Meta-Learning with Retrieval for Visually Grounded Compositional Concept Acquisi- tion, retrieval-enhanced meta-learning framework for compositional concept acquisition, which intro- duces retriever into GCCL. The retrieval mecha- nism plays crucial role in human learning. It facil- itates long-term retention, understanding enhance- ment, and knowledge transfer during the learning process, which have been discussed by large body of studies in cognitive science (Karpicke and Blunt, 2011; I; Karpicke, 2012). To mimic such human's retrieving behavior(Roediger and Butler, 2011; Karpicke and Roediger III, 2008), MetaReVision retrieves relevant primitive concepts from pre- constructed concept database and provides them support evidence to do meta-learning for com- positional concept learning. MetaReVision fol- lows Seear--Rtrrivval-CC Compose framework. It
Our experiments show that MetaReVision demonstrates stronger performance in GCCL, especially in the novel setting. This empir- ically shows the effectiveness of combining retrieval and meta-learning techniques in the context of grounded compositional learning.
present
2 Related Works
Meta-Learning also known as learning to learn, aims to solve a low-resource problem by leverag- ing the learned experience from a set of related tasks. Meta-learning algorithms deal with the prob- lem of efficient learning so that they can learn new concepts or skills fast with just few seen exam- ples (few-shot setting) even without seen exam- ples (zero-shot setting). Different from the typical meta-learning scenario where the training and test episodes are given in advance in few-shot learn- ing (Sung et al.. 2018: Snell et al., 2017: Nichol et al., 2018a; 8a; Finn et al., 2017), in GCCL, we need to construct episodes to employ meta-learning methods for compositional concept learning. In MetaReVision, we introduce retriever to actively construct episodes to help compositional concept learning. During the test time, with additional retrieved support items, MetaReVision can fur- ther fast-update VLMs for current compositional concept recognition in the query set. This test- time fine-tuning is different from previous works
shares the compositional learning burden be- tween VLMs and the retriever. Through meta-
Flement concepts are also called primitive concepts in our setting. We use them interchangeahly in this work.

---

which apply meta-learning in the zero-shot set- ting(Conklin et al., 2021).
Met MetaReVision explores the retrieval method solve the compositional problem.
Retrieval-Enhancee Learning. Retrieving related instances from a database, either the training set or external knowledge base, has been widely applied in tasks such as language modeling (Khandelwal et al., 2019), reinforcement learning (Goyal et al.. 2022) and language tasks such as NER (Wang et al., 2021). Instead of distilling all training information into the model's parameters through gradient up- dates, retrieval-enhanced learning introduces re- triever find related instances and based on these instances conduct further learning. For example, KNN-LM (Khandelwal et al., 2019) extends the pre-trained language model by linearly interpolat- ing its next word distribution with a retrieval mod- ule. This design shows effective domain adaptation ability. Wang et al. finds external contexts for the target instance by retrieving set of semantically relevant texts fine-tune the CRF module ad- dress the NER problem. These studies highlight the significance of actively recalling information from a database to enhance learning outcomes. The general scheme of such methods is to combine a parametric model with a non-parametric retrieval system (Long et al., 2022). Different from these set- tings, in GCCL, train own concept retriever and show retrieval's importance in compositional learning.
Grounded Compositional Concept Learning (GCCL)
We start by introducing the settings of Grounded Compositional Concept Learning (GCCL) and fur- ther introduce the benchmarks we curated for this problem in this section.
3.1 Problem Definition
Existing VLMs try to learn a generic representation for multi-modal tokens in different contexts. These VLMs are expected to obtain generic token repre- sentations that have strong transfer ability for down- stream tasks. We consider a setting that directly examines whether VLMSs have the ability to acquire compositional meanings of tokens through the lens of language modeling. Different from the task- level compositional studies, GCCL approaches the compositional problem from the token-level and investigates whether VLMs possess the capability to acquire the compositional meanings of tokens.
Training
Comp, Testing
Red apple is hadet
staxding
Seen Coup, Terting
parking road
Blue has parking at coraer
Compositional Learning. Recent research sug- gests that compositionality remains challenge for state-of-the-art (SoTA) neural models such as Transformers and Graph Ncural Networks (Niko- laus et al., 2019; Hupkes et al., 2020; SHAO et al., 2023). To tackle this challenge, inspired by sym- bolic AI, some works try to add structural con- straints into neural models (Bergen et al., 2021). There also some attempts generate new data for the compositions (Naeem et al., 2023; Xian al., 2018). Also, there have been noteworthy advancements in vision-language benchmarks that focus on probing and enhancing VLM's 's composi- tional abilities recently (Eisenschlos et al., 2023; Thrush et al., 2022; Ruis et al., 2020; Ma et al., 2023b). Nevertheless, these works build end tasks in compositional manner. They emphasize the performance of these compositional end tasks with- out giving consideration the token-level compo- sitional ability. However, GCCL. targets VL.'s leven-level compositional ability. Moreover, dif- ferent from symbolic and autment solutions,
Figure 2: GCCL task definition. Red highlights seen compositional concepts and blue highlights novel com- positional concepts.
Figure shows an example of the GCCL task. Given a set of image-caption pairs with the compo- sitional concepts masked out from the caption, the model is tasked to learn the concept representations and predict the masked compositional concept ditioned the contextual information. The learned model is then applied in the testing phase on both novel compositions as well as seen compositions. The model evaluated based on its ability to learn novel compositions while maintaining (i.e., not for- getting) seen compositions.
Formally, given a set of text-image pairs (Ieap: Zimg) |i= where Zing E I is the image with annotated bounding boxes, Teap T is the caption with the compositional concepts replaced by MASK. The objective of GCCL to predict the masked tokens based on the contextual informa- tion (M et al., 2023a; Jin et al., 2020). Therefore,

---

for BBoxes, only the locations are considered as input, not their label information. A model capable of solving GCCL can be described as a functional f x - Vattr x Vayj. where Vattr X Vanj is the target compositional concepts which could be either adjective + noun pairs or noun + verb pairs. Based on whether Vattr x Vobj have been seen during training, GCCL can be categorized into seen compositional testing and novel compositional testing. The desired compositional VLMs should achieve improved novel performance without sacri- ficing the seen performance.
in Figure 3. The retrieval module learns to find similar element concepts from the training data, The meta-learner organizes the retrieved items as a pseudo task to meta-tune VLMs for composi- tional learning. In this part, we will discuss the base VLMs, retrieval modulc, and meta-learning module in detail and answer two key questions in MetaReVision's design: 1) How retrieve related items, 2) How to utilize the retrieved items in the context of meta-learning
4.1 Vision-Language Models (VLMs)
VLBERT (Su al., 2020) and LXMERT (Tan and Bansal, 2019) are two representative VLMs that are suitable in our GCCL setting. They represent one-stream and two-stream VLMs separately. The difference is that two-stream VLMs have additional self-attention layers before cross-attention layers. We conduct experiments using these two types of VLMs to show the general effectiveness of the pro- posed framework. Moreover, all VLMs are trained from scratch to make sure that they do not see novel compositions during their training time.
3.2 GCCL Dataset Creation
We build GCCL's benchmarks, CompFlickr and CompCoco, from MSCOCO (Chen et al., 2015) and Flickr30K (Plummer et al., 2015). We use the same data split introduced by Nikolaus et al.. Their work studies the composition ability of image captioning systems by selecting 24 pairs as novel compositions by removing all images related to these 24 pairs from the training dataset. This en- sures that novel compositions have never been seen during training. Other works adapt the same data split for compositional learning studies. For exam- ple, Jin et al. utilized this split to check current VL models' compositional ability phrases under the continual learning setting. However, in Jin et al. 's work, most of the extracted phrases are in the form of article + noun, like the car and a man, They are single objects instead of compositional concepts. Such phase evaluation is not a good setting for compositional learning.
4.2 Retriever and Element Concept Database
Given the compositional concepts, the ideal re- triever is expected to retrieve the training examples that are the most beneficial for the target compo- sitional concept learning. It is usually assumed that the examples that are the nearest neighbors of query examples are more likely to be beneficial ones for generalizing (Long et al., 2022). GCCL retriever needs an encoder to encode the element concept, construct a database to organize these ele- ment concepts' information, and retrieve relevant concepts.
In order to evaluate the token-level composi- tional ability, we develop two benchmarks Compt- COCO and ComptFlickr to address the above limi- tation. Concretely, after paring the captions using Stanta (Qi et al., 2020), we use a number of rules to collect and mask the compositional concepts, the details are in the Appendix C. Finally, the dataset is divided into 4 parts: training set without novel compositions, validation set with both seen and novel compositions for hyper-parameter tuning and model selection, seen test set, and novel test set. The detailed statisties of novel compositions for these two datasets are shown in Appendix D.
Element Concept Encoder. Given the linguistic and visual clues for the compositional concepts, the encoder is acting maps a MASK concept
function fx Ting) that
fixed-length vector Rd Then for each primitive concept in the target com- positions, f(-) can help retrieve related primitive concepts. MetaReVisiin relies on these retrieved concepts to conduct further compositional learn- ing. In this way, MetaReVision enhances its own compositional capability by augmenting the input through the retrieval procedure. The encoding func- tion f()) is the key component for the retriever. In traditional vision-language tasks, like VQA and Visual Entailment(Song et al., 2022), CLIP (Rad- ford et al., 2021) is usually used as the encoder to encode the whole visual or textural input and
4 Meta-Learning with Retrieval for GCCL (MetaReVision)
MetaReVision mainly consists of two modules: the retrieval model and the meta-learner as shown

---

Retriever Construction
Meto- Training
Test Phose
Input:A A (MA3K] (MAS)) crring road. Outputt Blus
Query
Novel
Input larg (MASK] [MAS presing parked cars
Input: People siting on the [MASK] [MASK]
Label: White bus
VLM-Encoder
VLM-Encoder
VLM-Encoder
FAISS Indexer
FAISS Indexer
FAISS Indexer
[Hlue] [Dos] driving
Element Concept DB
Sagaott tum
[blue] [bus] driving on the road
Key: Denov Ventor D.S, Value:
[white] jplone) on the table
A [red] [bruck] traveingg
Elemmnt cancent: Navr
inteructionn
Element type: Adjective
Sentence: blu bus driing the road 7616
Updating:
uge CLANSIR Renteree(5)).
vhice Alg/R, 5) -0-00-00000 0-07,
Fine-Tune and Predict: Red Bus
Figure 3: MetaReVision Architecture. The whole system includes two modules: retrieve and meta-trained VLM. During testing, MetaReVision retrieves related instances to fast-update VLM for novel compositional learning.
help build the retriever. However, GCCL's token- level compositional setting, we focus on the token's 's representation and therefore use the VLMs as an encoder to extract MASK concept's representation for further compositional learning. These vectors are used keys to construct the Element Con- cept Database and perform approximate nearest neighbor search augment compositional learning. We add a two-layer MLP and adopt Masked Lan- guage Modeling (MLM) to train Aision-language retriever. For the encoder's training, since we fo- cus on concept acquisition, words in compositional concepts are masked with probability of 1.0, and others are not masked during training.
retrieved DB item's value, and Ret the retrieved DB item set.
After adding the retrieval module into GCCL, the problem can be re-formulized as:
p(v x) p(u Ret(x)) pl p(Ret()) x
(2)
Learner
Retrieval
where v is the MASK compositional concept's pre- diction, x E Rd is the maksed concept's encoded vector and Ret(r) is the retrieved DB items based on its vector learning happens in two levels: 1) retrieve related items from DB based the encoding vector, 2) learn conditioned on contextual information and the retrieved items.
Equation 1. The compositional
Element Concept Database. The element concept datastore DB = (ki, ut}}, which is constructed offline using the above-trained vision- language en- coder, consists of dense representations of masked element concepts k = Enc Strupp:Timg C Rd is as keys and the corresponding (Feap; Ling) values. To efficiently access this database, we implement the dense retriever for GCCL by an off the-the-shlf-- retriever engine FAISS (Johnson al., 2019) with a flat index (IndexFlatIP)) without any training. Then given a masked concept, we can retrieve the top- K DB items by calculating the cosine similarity scores between the /MASK] concept with all DB items in nearly real-time as follows:
4.3 Meta-Learning for GCCL
Given the retrieved items, there are several ways exploit these examples to facilitate compositional learning. The most direct method is to fine-tuning (FT). However, because the retrieved items are noisy and FT often faces over- over- fitting issues when they learn from a few labeled examples, FT does not help GCCL. Another choice in in-context learn- ing (Wei et al., 2022). However, as GCCL is a multi-moda. problem. We have multiple image- caption pairs in the contextual input, current large multi-modals, like LLaVA (Liu et al., 2023) and GPT-4 (gpt, 2023), can not be applied directly here. In MetaReVision we choose meta- meta-lcarning frame- work to utilize the retrieved items for GCCL. Meta- learning here is to train the base VLM with the ability to accumulate knowledge across episodes?
Rkt(k) = {k1, Val1). (KM, Valar)
(I)
where k is the mask concept's embedding vector, k; the DB item's key, Val, = (Zeap, Timg.) is the
Repisodes also called tasks in meta-lcarning

---

and build internal generic representations for to- kens that are suitable for compositional learning. Moreover, we introduce the verbalizer module to enforce the predicted concept for the query set com- ing from the retrieved support items. The verbal- izer helps mitigate the memorization problem in meta-learning (Yin et al., 2019). In the following part, we will discuss episode construction, the de- tails about MAMI, and verbalizer module used in MetaReVision.
MAML
Target Compositional Items
Retrieved Support Items
Yins liu
0'
Meta
Meta-Train
Maaly (Fe00))
-- (Fei5))
Episode Constructions. We construct GCCL tasks T₁ for meta-learning as follows:
Figure MAML's computing procedure
which unrolls the optimizing process and tries find good initial parameter configuration for all compositions.
Duupport Dqueer)
(3)
where Dippport indicates the support set and Diueery indicates the query set. Specifically, for one task, we randomly select the query set. Then we retrieve a small number of examples that are similar the query concepts. These retrieved items make up the support set. Meta-Icarning's objective in GCCL is to predict the compositional concepts in the query set after learning the element concepts in the support set, Here, episodes help VLMs to accumulate composi- tional knowledge and learn generic compositional representation for masked concepts from the task- level instead of instance-level.
Verbalizer. MAML's classical application is in few-shot learning, where class-to-label assignment needs to be conducted within each episode, that is, the same class has different labels among different episodes. Without such re-assignment, the models can memorize the class information and conduct prediction directly without considering the items in the support set. This is known as memorization problem in MAML discussed in (Yin et al., 2019) 9), To help MetaReVision learn from the retrieved instances, we introduce the verbalizer module into MetaReVision. It enforees prediction for the query by selecting concepts from the support shown in Figure In this way, Me MetaReVision will rely on the retrieved element concepts rather than memorizing the labels to do compositional learning. This helps alleviate the MAML's memorization problem.
compositional concept as
Meta-Learner. We use MAML (Finn et al., 2017) as our meta-learning algorithm. As an optimization- based method, MAML. has two optimizing steps within each episode: the meta-train step and the meta-test step. In the meta-train step, MAML learns a task-specific learner 0' based on the cur- rent parameter 0 and retrieved support items S. In the meta-test step, MAML updates the parameter f based on the fast-updated parameter 6' and the compositional query items Q as shown in Figure Moreover, MAML can be solved by formulating as bi-level optimization problem. Equation 2 can be extended to Equation 4.
Cross-Entrogy/ Loss
D
Retrieved Concept
Rea!
Yellow
Selection
Vocebulary
Biux
Thois
Concent Predict Head
Heree
FLM Eacoder
Retrieved Supporting Cuncepts
poaxxsoaaa (Bal] maad.
[Alg (0, Retriever (8)),Q) min 0 where Alg(0, 5) = aVo((A,))
Figure 5: Verbalizer helps VLM consider retrieved in- stances when learning.
(4)
where 0 is the learnt parameters, Retriever(S)) stands for the retrieved DB items, Q is target com- positional concept and Alg represents the optimiza- tion algorithm adapting to the support instances. There are different versions regarding Alg (Nichol et al., 201 8b; Finn et al., 2017). We use MAML
4.4 Inference
During inference time, consider each com- positional concept as a query item and retrieve rel- evant instances from concept DB as support in- stances. Therefore, we construct a specific task for

---

the current compositional concept. Instead of ap- plying the general model 0 directly, MetaReVision retrieves support instances to fast-update the model to adapt to current compositions and make pre- dictions as 02 = argmax P(v), where the prediction comes from the retrieved concepts. In MAML's testing, it is observed that larger num- ber of updates can give considerable performance boost. Thus, we choose the inner loop updates to 20 before testing.
HIGHER to calculate the higher gradients. The code for this paper will be released at A.
5,4 Baselines
We use two types of baselines in this evaluation. The first is the train-from-scratte baseline which trains VLMs from random initialized parameters. Another baseline is MAML without retriever. In this setting, VLMs are meta-trained using the same retrieved tasks, but VLMs not access the sup- port set. It predicts directly during test time. This baseline used to show the importance of the re- triever during test time for GCCL. Moreover, we also compare two variants of MetaReVision, in- cluding Top 4 and Div Top 4 retrieves top 4 similar concepts, which may contain duplicated concepts. The same concept could have different vector representation which is affected by different visual and textual contexts. For example, car could have different vector values when modified by red or blue. Div retrieves the top distinct similar concepts expecting that the true primitive concept will be in the retrieved set.
5 Experiments
this section, we introduce the GCCL's datasets, demonstrate the implementing details of MetaReVision and compare its results with other baselines. Ultimately, we empirically analyze the retriever importance in MeaReVision
5.1 Dataset
CompCOCO is constructed from MSCOCO (Chen al., 2015) using its 2014's 4's split. In this split, COCO-captions has 103175 training images and 15112 validation images (Chen al., 2015). Be- cause MSCOCO does not provide test data, we use the validation data as the testing data Com- pCOCO. Moreover, we did some minor synonym modifications described in the Appendix A to ex- tract more clean concepts.
5.5 Main Results
We report the performance under both novel and seen settings as shown in Table I and Table 2. From the two tables, we can see that MetaReVi siion does help compositional learning, especially in the novel setting. Novel Compositions. As shown Table 1. MetaReVision improves the performance on the novel setting compared to the pre-trained model and MAML models. This suggests that MetaReVision captures a generic representation which is beneficial for compositional learning through meta-learning on the retrieved tasks. How- ever, compared with seen compositions (i.e.. Ta- ble 2), the performance on novel pairs drops signifi- cantly across the board. MetaReVision's accuracy drops by about 20% on CompCOCO dataset novel setting compared with the seen setting. This indicates that such compositional generalization is still very difficult and open task for current VL models.
CompFlickr is constructed from Flickr30k En- tities (Plummer et al., 2015). Flickr30k con- tains 276k manually annotated bounding boxes for 31, 783 images and a total of 158, 158,915 En- glish captions (five per image). We use the given train/val/test split to construct CompFlickr.
5.2 Evaluation Metrics.
We use accuracy as our primary metric to measure the GCCL performance and report object, attribute, and compositional accuracy separately. Jin et al. uses perplexity the forgetting metric in continual learning which is not appropriate in our work due MetaReVision offline setting.
5.3 Implementation Details
Seen Compositions. Table 2 shows the perfor- mance the seen setting. From the table, we can see that all models have similar accuracy in the seen setting. One possible reason is that all the
The implementation of MetaReVision uses the HuggingFace Transformers library (Wolf et al., 2020). For MAML, we use Adam optimizer (Kingma and Ba, 2014) as both inner and outer optimizers. We set the inner learning rate to 5e 5, the outer learning rate to le 5, and based on

---

VL-Model
VLBERT
LXMERT
Metric
Pair Accu.t
Attr. Accu.t 25.88%
Obj. Accu.1
Pair Accu.
Attr. Accu.
Obj. Accu.
Seiin-Srrachh
7.73%
50.74%
8.14%
26.36%
55.06%
COCO
MAML w/o Ret
9.03%
27.08%
50.04%
9.04%
27.01%
56. 19%
Ours(Top 4)
11.15%
29.84%
50.17%
12.01%
29.36%
58.81%
Ours(Div 4)
13.50%
31.85%
50.92%
13.79%
33.76%
59.87%
Train-Scratch
6.04%
17.53%
65.21%
5.12%
18.10%
61.68%
Flickr
MAML w/o Ret
8.60%
22.06%
64.38%
7.52%
18.45%
64.55%
Ours(Top 4)
10.7%
24.58%
65.54%
9.38%
20.45%
65. 10%
Our 4)
11.50%
25.49%
66.58%
10.58%
22.45%
65.15%
Table 1: Results on Novel Compositional Concept.
VL-Model
VLBERT
LXMERT
Metric
Pair Accu.t
Attr. A
Obj. Accu.1
Pair Accu.1
Attr. Accu.
Obj. Accu.
Train-Scratch
32.45%
49.06%
60.03%
34.12%
50.33%
61.96%
COCO
MAML w/o Ret
32.23%
49.05%
59.20%
34.09%
49.97%
61.93%
Ours(Top 4)
32.27%
49.15%
59.98%
34.02%
49.90%
61.90%
Ours(Div 4)
32.46%
50.01%
60.05%
34.15%
50.32%
62.00%
Train-Scratch
24.34%
42.72%
52.53%
22.68%
40.86%
50.1
Flickr
MAML w/o Ret
23.73%
41.92%
49.01 %
22.15%
41.21%%
49.97%
Ours(Top 4)
23.75%
41.95%
49.04%
22.75%
41.19%
50.01 %
Ours(Div 4)
26.52%
46.11%
53.23%
23.41%
42.02%
51.61% %
Table Results on Seen Compositional Concept.
models have been fully trained using the seen com- positional concepts. MAML-based methods do not hurt the in-domain performance during this meta- learning phase.
Retrieval Accuracy
Rovel
5.6 Empirical Analysis of Retriever
Retrieval Accuracy. Figure shows the retriever's top-4 accuracy for attributes, objects, and pairs un- der both seen and novel settings. Attribute recog- nition is the key challenge compared with object recognition in GCCL, even in the retrieval phase. In GCCL, the learned VLMs are biased to the seen attributes that need to be adjusted for effective com- positional learning.
Figure Retriever accuracy comparison between seen pairs and novel pairs in CompCOCO.
Conclusions and Future Work
Importance of diverse sampling. Retrieving true concepts into the support set is important for GCCL. this part, we assume an oracle situation where we can always select the true element concepts into the support set during test time. We study potential advantages that can be derived under this config- uration. From Figure we can see that the true concept in the support set does help the composi- tional learning. also explains the importance of diverse sampling which increases the probability selecting the correct elemental concepts.
In this work, we propose MetaReVision, which combines retrieving method and meta-learning to train VLMs for grounded compositional concept learning. Our work highlights the significance of retrieval in compositional learning. Our empiri- cal results and CompFlickr, have shown that MetaReVision consistently outperforms conventional VLMs and meta-lcarning methods without retriever, especially in novel settings. However, GCCL is still a chal-
proposed datasets, CompCOCO

---

DB during the meta-learning time. Training both the learner and the retriever in an end-to-end man- ner could improve the performance for GCCL and other retrieval-enhancel models.
Oradle
24.13%
Div-4
References
Top-4
22.015
2023. Openai: Gpt-4 technical report.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein, 2016. Neural module networks. Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 39-48.
Figure 7: MetaReVision accuracy on CompCOCO using different retrievers.
Leon Bergen, Timothy O'Donnell, and Dzmitry Bah- danau. 2021. Systematic generalization with edge transformers. Advances in Neural Information Pro- cessing Systems, 34: 14:1319-4
lenging open problem and many problems remain. Our future work will explore more cognitively plau- sible models and explicitly address the grounding ability in compositional concept learning.
Irving Biederman and Edward A Vessel. 2006. Percep- tual pleasure and the brain: A novel theory explains why the brain craves information and seeks it through the senses. American scientist, 94(3):247-253.
7
Acknowledgement
This project supported by National Science Foun- dation (NSF) CAREER award 2028626 and par- tially supported by the Office of Naval Research (ONR) grant N00014-20-1-2009 Any opinions, findings, and conclusions or recommendations ex- pressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation nor the Office of Naval Re- search. We thank all reviewers for their thoughtful comments and suggestions.
Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakr- ishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015, Microsoft coco captions: Data collection and evaluation server. arXiv preprint 503.50355.
Henry Conklin, Bailin Wang, Kenny Smith, and Ivan Titov. 2021. Meta-learning to compositionally gener- alize. arXiv preprint arXiv:2106.04252.
Julian Martin Eisenschlos, Jeremy R. Cole, Fangyu Liu, and William W. Cohen. 2023. WinoDict: Probing language models for in-context word acquisition. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Lin- guistics.
8 Limitations
The limitations of the proposed MetaReVision in- clude 1) Grounding limitation. Currently, we rely on VLM's attention mechanism to do grounding. We do not have an explicit grounding design to align the textual concepts and visual regions. This could be an interesting direction for future GCCL works. 2) SoTA generative model comparisons. Currently, we can not directly apply SoTA gener- ative models, such as BLIP-2 and MiniGPT, on GCCL due to the following reasons. One rea- son is the GCCL problem setting. In GCCL, is not easy to transform the supporting items, in- cluding multiple images and captions, into con- textual input for these generative models. An- other reason is controlled evaluation which means that these huge generative models may have al- ready seen the novel compositions during training and it not fair comparison with other mod- cls. 3) Updating retriever. We construct our elc- ment concept DB in advance and not updating this
Chelsea Finn, Picter Abbeel, and Sergey Levine. 2017. Model-an -adeos-agnosti meta-learning for fast adaptation of deep networks. In International conference on ma- chine learning, pages 122-1135. PMLR.
Jerry A Fodor and Zenon W Pylyshyn. 1988. Connec- tionism and cognitive architecture: A critical analysis. Cognition, 28(1-2):3-74
Anirudh Goyal, Abram Friesen, Andrea Banino, Theo- phane Weber, Nan Rosemary Ke, Adria Puig- domenech Badia, Arthur Guez, Mehdi Mirza, Pe- ter C Humphreys, Ksenia Konyushova, al. 2022. Retrieval- -agmeette reinforcement learning. ternational Conference on Machine Learning, pages 7740-7765. PMLR.
Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. 2020. Towards learning generic agent for vision-and-languaage navigation via pre- training. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition. pages 13137-13146.

---

Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. 2020. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67:757-795.
Ziqiao Ma, Jiayi Pan, and Joyce Chai. 2023a. World- to-words: Grounded open vocabulary acquisition through fast mapping in vision-language models. arXiv preprint arXiv:2366.0865
Xisen Jin, Junyi Du, Arka Sadhu, Ram Nevatia, and Xiang Ren. 2020. Visually grounded continual learning of compositional phrases. arXiv preprint wrXiv:2005.00785
Zixian Ma, Jenry Hong. Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. 2023b. Crepe: Can vision-language foundation models reason compositionally? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 10910-10921.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scal similarity search with GPUs. IEEE Transactions on Big Data, 733:5355-477.
Muhammad Ferjad Naeem, Muhammad Gul Zain Ali Khan, Yongqin Xian, Muhammad Zeshan Afzal, Di- dier Stricker, Luc Van Gool, and Federico Tombari. 2023. 12mffommr: Large language model generated multi-view document supervision for zero-shot im- age classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 15169-15179.
Jeffrey D Karpicke. 2012. Retrieval based learning; Ac- tive retrieval promotes meaningful learning. Current Directions in Psychological Science, 21(3):15--133
Jeffrey D Karpicke and Janell R Blunt. 2011. Re- trieval practice produces more learning than elab- orative studying with concept mapping. Science, 331(6018):72-7755
Alex Nichol, Joshua Achiam, and John Schulman. 2018a. On first-order meta-learning algorithms. arXiv preprint arXi::
Jeffrey D Karpicke and Henry L Roediger III. 2008. The critical importance of retrieval for learning, science, 319(5865):966- 968.
Alex Nichol, Joshua Achiam, and John Schulman. 2018b. On first-order meta-learning algorithms. arXiv preprint arXiv:1803 02999.
Charles Kemp and Joshua B Tenenbaum. 2009, Strue- tured statistical models of inductive reasoning, Psy- chological review, 16.)):20.
Mitja Nikolaus, Mostafa Abdou, Matthew Lamm, Rahul Aralikatte, and Desmond Elliout. 2019. Composi- tional generalization in image captioning. arXiv preprint arXiv:1909.04400.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv:1911.0017.
Santiago Ontanón, Joshua Ainslie, Vaclav Cvicek, and Zachary Fisher. 2021. Making transform- ers solve compositional tasks. arXiv preprint arXiv:2108.04378.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint 14Xiv:1412.6900
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image- to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641-20 1644..
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020. Oscar: Object- semantics aligned pre-training for vision-language tasks. In Computer Dision-ECCV 2020: 16th Euro- pean Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16, pages 121-137. Springer.
Peng Qi. Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics: System Demonstrations.
Haotian Liu, Chumyuan Li, Qingyang Wu, and Yong Jac Lee. 2023. Visual instruction tuning. arXiv preprint arXiv:2304.08-83
Alexander Long, Wei Yin, Thalaiyasingam Ajanthan, Vu Nguyen, Pulak Purkait, Ravi Garg, Alan Blair, Chunhua Shen, and Anton van den Hengel. 2022. Retrieval augmented classification for long-tai tail vi- sual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 6959-6969.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International confer- ence on machine learning, pages 8748-8763. PMLR.
Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. 2018. Neural baby talk. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 721777197777
Henry L Roediger and Andrew C Butler. 2011. The crit- ical role of retrieval practice in long-term retention. Trends in cognitive sciences, 15(1:200-27.

---

Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, and Brenden M Lake. 2020. A bench- mark for systematic generalization in grounded lan- guage understanding. Advances in neural informa- tion processing systems, 33: 13:19981-1987.
Yongqin Xian, Tobias Lorenz, Bernt Schiele, and Zeynep Akata. 2018. Feature generating networks for zero-shot learning. In Proceedings of the IEEE conference computer vision and pattern recogni- lion, pages 5542-5551.
NAN SHAO, Zefan Cai, Hanwei xu, Chonghua Liao, Yanan Zheng, and Zhilin Yang. 2023. Compositional task representations for large language models. In The Eleventh International Conference on Learning Representations.
Mingzhang Yin, George Tucker, Mingyuan Zhou, Sergey Levine, and Chelsea Finn. 2019. Meta- learning without memorization. arXiv preprint arXiv:1912.03820.
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. 2020. Unified vision- language pre- pre-training for image captioning and vqa. In Proceedings of the AAAT conference on artificial intelligence, volume 34, pages 13041-13044
Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning. arXiv preprint arXiv: 1rxiiv1703.000 100..7073..5555
Haoyu Song, Li Dong, Wei-Nan Zhang, Ting Liu, and Furu Wei. 2022. Clip models are few-shot learners: Empirical studies vqa and visual entailment, arXiv prepriat urXiv:2203.07191
Weijie Su, Xizhou Zhu, Yue Cao. Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2020. V1-bert: Pre-training of generic visual-linguistic representations. In Inter- national Conference Learning Representations.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. 2018. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1199-1281 1208.
Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross-modality encoder representations from trans- formers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanprect Singh, Adina Williams, Douwe Kiela, and Candace Ross. 2022. Winoground: Probing vision and lan- guage models for lsiouting compositionality. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Patlern Recognition, pages 5238- 5248.
Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Kewei Tu. 2021. Improving named entity recognition by external context retrieving and cooperative learning. arXiv preprint arXiv:2 1o0.033553.444
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistae, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing, In Proceedings of the 2020 con- ference on empirical methods natural language processing: system demonstrations, pages 38-45.

---

A Modified MSCOCO Synonym List
In order to extract more compositional concepts, we modify (Lu et al., 2018)'s category and change the drier synonym list as: hair drier, hairdryer, hair dryer, blow dryer, blow drier.
B Episode Examples
Table 3 shows episode examples constructed in MetaReVision. From the table, we can see that MetaReVision can retrieve true element concepts for target compositional concepts, such as white truck. bird fly, boy cat. But there also exist cases can not find true element concepts in the retrieved support set, such as blue bus. In this example, MetaReVision can retrieve many similar objects, but has a chal- lenge to retrieve the true color blue. Also, from these randomly sampled episodes, we can see that GCCL, objects are easier to be retrieved compared to objects.
Turget Context
Turget Concepts
Retrieved Contest
Retrieved Concepts
Several bikes parkell next a while van.
White
man in soit poses by an colored truck.
Colored Track
white parked frost of house iuut heing
White Track
woman smiling in front
big
Big
People waiting on the side of the road for the yellow bus.
Yellow Bus.
geese are flying
newr trees.
Geese Fly
Two bawks flying near snow coveral mo motaiiin.
Hawk Fly
couple of bircs Bling through a cloudy sky.
hind
Two birds sit grass
other.
Bird
Two black hinds are sittnng on top of mountain.
Black Bind.
young bay enjoying his pizza
dinner
Boy Enjoy
The little girl cating lunch and having milk.
Girl Eat
a small boy is cating from green plate
boy cat
The woman cating her meal at the table by lieself
Woman Fat
elderly couple having a snack
Couple
A while and black dog laying top of yellow beat.
Black Dog
A brown dog is on the dock a bost on water.
Brown Dog
black
some green grass
houses
Beuwn
The black and white puppy is playing with a small toy.
Dog Play
white black animal lays bench
red bus driving down a sireet in front of red double decker bus.
Rol Bus
has lurge sign the it.
Blue Bus
red driving down city mad cloudy day
Red
A red bus driving next to an orange and green bos
Green Bus
red Almble docker has regular bux and low track outdoor.. Two men in suils stand froe of blue and white semi truck. white and black bus with rainbow colorod flag on the front friends stand large blue RV parked outside large brick building.
Regular Bas
blue bus parked front of an axure buikling,
Bluc Truck
Black Bux.
Bluc Rus
orange vun.
Orange Van
Blue RV
Table 3: Episode examples constructed by MetaRe 's retrieval modules.
Compositional Extracting Rules
After parsing by Stanza, we can extract compositional pairs using the following rules. Compared with Jin et al.' phase extracting rule, MetaReVision extracts more reasonable compositional pairs.
Gono
Noun
Ver
large passenger airplane flying through the air..
brown
horse middle the city cating grass.
ANIO
ACURECL
An orange blue and white bus and brown round structure behind It.
Moun
Werb
AMGO
airplane that is either, landing or just taking off.
black
Inside white
NSUBI
Noun
Verb
yellow.
kitten is sitting
dish
table.
(a) Rules to extract adj-noun pairs.
(b) Rules to extract verb-noun pairs.
Figure 8: Extracting rules to Construct CompFlickr and CompCOCO.

---

D GCCL Data Statistics
Table shows the statistics the extracted novel compositional concepts. From the table, we can see that CompCOCO has more novel pairs than Compflickr. And CompCOCO is more reliable evaluation for novel compositional learning than And CompFlickr
MSCOCO
FlIckr33KK
Train Img.
Train Caps.
Test Img.
Test Caps.
Trai Imn.
Train Caps.
Val lin.
Val Caps.
Test Img.
Test Caps.
black
205
323
122
190
17
24
0
2
3
small dog
G81
1067
481
360
612
12
33
white beat
373
261
196
131
69
85
0
3
8
big
38
eat horse
212
378
106
187
2
0
0
stand child
1285
1556
577
741
1048
1475
57
26
36
white horse
25-4
500
151
300
51
10U
4
big cat
184
216
103
108
0
0
blue bus
276
506
143
243
11
16
0
0
small table
261
296
1:14
154
48
54
1
1
1
bold child
stand bird
632
831
260
106
13
21
0
0
brown dog
613
878
291
430
934
1838
31
61
29
58
small cal
252
325
149
183
0
white truck
262
420
121
176
35
42
2
2
2
big plane
1345
5
0
ride woman
595
674
300
330
266
5:37
17
9
23
0
1
black cat
840
1760
448
940
15
27
D
big 213
291
123
169
34
red hus
566
1212
232
474
20
small plane
481
833
158
279
13
20
0
lie woman
301
388
144
194
145
278
2
4
Table 4: Novel Pair Statistics for both CompCOCO and CompFlickr. We the same 24 pairs verify the compositional generalization.


Reviews:
[{"review_id": "eFOOCFPwPy", "reviewer": "Reviewer_SB8Y", "summary": "", "questions": "Question A: The verbalizer is only introduced in section 4.3 and briefly explained using a figure (Figure 5, line 461-465) but the concrete implementation remains unclear.\n\nQuestion B: The implementation of the baselines is not directly inferrable from the name: how does MAML work without the retrieved concepts as the support set?\n\nQuestion C: What are duplicated concepts (line 532)? Does one concept have multiple vector representations in the element concept DB?\n\nQuestion D: The experiment section should be extended to include more SOTA models to demonstrate the utility and improvement of the proposed framework.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q_A: About the verbalizer**\n\n**Ans:** The verbalizer constructs a smaller vocabulary based on the retrieved items to limit the predictions to those labels. We will make sure to refer to the verbalizer in the beginning of section 4 and clarify the implementation further in section 4.3. The introduction of verbalizer is to enforce the learner to make prediction considering  both the retrieved items and current parameters, which solve the *memorization* problem in meta-learning.\n\n-------\n\n**Q_B: how does MAML work without the retrieved concepts as the support set?**\n\n**Ans:**  As discussed in Sec. 5.4 (line 525-528), we use the same retriever to construct the episodes. However, during test time, we do not have a support set, we just do normal *[MASK]* prediction using the meta-trained VL learner.  This result difference is used to show the importance of retriever in GCCL. GCCL is a zero-shot problem, the retriever help release the learning burden from VL learner to retriever.\n\n--------\n\n\n**Q_C:** What are duplicated concepts (line 532)? Does one concept have multiple vector representations in the element concept DB?\n\n**Ans**: Yes. One concept could have multiple vector representations [when combined with different attributes/objects]. For example, car could have a different vector representation when modified by 'red' or 'blue'. The encoding is affected by different visual and textual contexts.\n\n\n--------\n\n**Q_D:** Extend to SOTA models.\n\nReally appreciate for the suggestion. We also consider using large pre-trained VL models (if that is what the reviewer means by SOTA), like BLIP-2 and MiniGPT, in GCCL. However, we stick to LXMERT and VLBERT based on following considerations:\n\n- *GCCL problem setting*. IN GCCL, we want to explore VL model's compositional learning ability (recognizing attr-obj and obj-verb pairs), given the contextual visual and textual hints and we formulate GCCL as a [MASK] token predicting problem. However, SOTA models, like BLIP-2 and MiniGPT, are generative models which are not suitable in GCCL setting.\n- *Controlled evaluation*. As reviewer mentioned, the compositional evaluation in GCCL is controlled and we only train from scratch using our own training data that excludes the novel compositions to avoid muddy evaluation.\n- *Low performance*. This shows that GCCL is a challenging problem. Although large VL models achieve success in downstream tasks, like VQA or retrieval, they have difficulty in this fine-grained compositional generalization. MetaReVision propose a possible direction, combining retreval and mata-learning, to enhance compositional learning for GCCL, but far from solving it.\n\n-----\n\n**Typos and Presentation Improvements**\n\nThank you for pointing the typos out and we will modify all in the revised version.\n\n-----", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. As far as the reviewer's knowledge, the proposed framework is novel and the problem is well-motivated.", "reasons_to_reject": "1. Implementation of part of the proposed framework is not explained clearly (Question A);\n2. Some experimental details are unclear (Question B, C);\n3. The work is mainly empirical, but the experiment section is generally weak: only numerical performances are compared with a limited number of models, and the utility of the proposed framework remains in question (Question D);\n4. There is very limited documentation of the proposed benchmark datasets.", "questions_to_authors": "Question A: The verbalizer is only introduced in section 4.3 and briefly explained using a figure (Figure 5, line 461-465) but the concrete implementation remains unclear.\n\nQuestion B: The implementation of the baselines is not directly inferrable from the name: how does MAML work without the retrieved concepts as the support set?\n\nQuestion C: What are duplicated concepts (line 532)? Does one concept have multiple vector representations in the element concept DB?\n\nQuestion D: The experiment section should be extended to include more SOTA models to demonstrate the utility and improvement of the proposed framework.", "paper_topic_and_main_contributions": "The paper proposed a novel meta-learning framework with retrieval for the Grounded Compositional Concept Learning (GCCL) problem. Specifically, a retriever is used to help construct the support set with similar concepts as the query item for the meta-learner to learn a generic representation. The paper also proposed two benchmark datasets for GCCL and analyzed the proposed framework on these two benchmarks.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "It is strongly suggested to work on correcting the typos throughout the paper (line 38, 217, 457, 489, 527, etc)."}, "extra_text": {}}, {"review_id": "rlXVkNnswi", "reviewer": "Reviewer_c8Aa", "summary": "", "questions": "Line 406/607, how do you know GPT-4 cannot take multiple images?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Stronger VL models.**  The models used in this paper are LXMERT and VLBERT, which were proposed 3 or 4 years ago. It's better to evaluate more advanced models like BLIP-2. I agree that the large-scale model may have seen the novel concepts during training, which may muddy the experiments. But the absolute numbers of pair predictions in the paper look very low (10\\%), making me question the proposed method's effectiveness.\n\n**Ans:** Really appreciate for the suggestion. We also consider using SOTA models, like BLIP-2 and MiniGPT, in GCCL. However, we stick to LXMERT and VLBERT based on following considerations:\n- *GCCL problem setting*. IN GCCL, we want to explore VL model's compositional learning ability (recognizing attr-obj and obj-verb pairs), given the contextual visual and textual hints and we formulate GCCL as a *[MASK]* token predicting problem. However, SOTA models, like BLIP-2 and MiniGPT, are generative models which are not suitable in GCCL setting.\n- *Controlled evaluation*. As reviewer mentioned, the compositional evaluation in GCCL is controlled and we only train from scratch using our own training data that excludes the novel compositions to avoid muddy evaluation.\n- *Low performance*. This shows that GCCL is a challenging problem. Although large VL models achieve success in downstream tasks, like VQA or retrieval, they have difficulty in this fine-grained compositional generalization.  MetaReVision propose a possible direction, combining retreval and mata-learning, to enhance compositional learning for GCCL, but far from solving it.\n\n\n-------\n\n**Generalized to more complicated compositions.** This task/data only evaluate the composition between one adjective and one object. It could make the paper stronger to consider other types/numbers of compositions. \nIt is also not clear to what's the difference between this paper and previous work on compositional concepts like \\textit{Winoground}.\n\n**Ans:** We agree with the reviewer that there are much more complex compositional learning settings. However, as the reviewer kindly acknowledged our contribution, this GCCL problem is still very challenging and worth independent investigation.\nThe **Winoground** task is indeed an interesting task designed totally differently. It covers the problem of compositionality, testing does not necessarily considers novel compositions. They have a multichoice setting that include exact same words with different orders. The order changes the compositional meaning. As reviewer suggested, retrieval-based meta-learning could also be applied this Winoground task with carefully designed retriever design and learner design, which is for our future work.\n\n\n--------\n\n**Paper writing**\n\n**Ans:** Thank you for pointing all these out and we will modify all of them in the new version.\n\n---------\n\n**Question:** Line 406/607, how do you know GPT-4 cannot take multiple images?\n\n\n\n**Ans:** Here, we mean that we can't input multiple images to GPT-4 *in one round* as contextual visual information for GCCL. We hope to get the compositional prediction in one round. But interacting/talking with GPT-4, we can input one image in one round and then several images in several round to prompt GPT-4 for GCCL.  But this is another GCCL setting.\n\n\n-------------------", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The task/dataset is designed neatly to study the understanding of compositional concepts.\n2. The proposed retrieval-enhanced meta-learning is intuitive and proved effective in novel concept recognition.", "reasons_to_reject": "1. Stronger VL models.\nThe models used in this paper are LXMERT and VLBERT, which were proposed 3 or 4 years ago. It's better to evaluate more advanced models like BLIP-2. I agree that the large-scale model may have seen the novel concepts during training, which may muddy the experiments. But the absolute numbers of pair predictions in the paper look very low (10%), making me question the proposed method's effectiveness.\n\n2. Generalized to more complicated compositions.\nThis task/data only evaluate the composition between one adjective and one object. It could make the paper stronger to consider other types/numbers of compositions. It is also not clear to what's the difference between this paper and previous work on compositional concepts like Winoground.\n\n3. Paper writing.\nThere are some typos in the paper (see later comments). Some figures can be improved, e.g., in Fig 6, hard to see the points of novel pairs.", "questions_to_authors": "Line 406/607, how do you know GPT-4 cannot take multiple images?", "paper_topic_and_main_contributions": "This paper aims to study models' generalization ability to combine primitive concepts into compositional concepts. For example, see the primitive concepts of \"red\" and \"chair\" during training and can understand the novel composition of \"red chair\" during testing. This paper introduced two datasets based on MSCOCO and Flicker to conduct the compositional learning. The proposed MetaReVision method consists of a retrieval module to retrieve relevant primitive concepts and a meta-learning module to train the model with retrieved data. The experiments show that the MetaReVision method can improve the performance of novel compositional concepts.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 315 reresentive - > representative\nLine 380: Val inconsistent with the equation\nLine 444: inconsistent symbols in the equation and text\nLine 458: memory -> memorize"}, "extra_text": {}}, {"review_id": "ot50D7KZoA", "reviewer": "Reviewer_tj2B", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**About minor issues**\n\n**Ans:** Thank you for pointing all these out and we will modify all of them in the new version.\n\n--------\n\n\n**Domain Shift Problem:** In the contribution of this paper, it is mentioned to deal with the domain-shift problem in compositional learning. Which existing works have this problem, or do all the compositional learning have it? It is not mentioned above, and it is better to explain it earlier.\n\n\n**Ans**: Yes, all compositional learning has domain-shift problem, because we have not seen the novel compositions during training time. The statistical distribution of the data in the training set (seen compositions) and the testing set (novel compositions) are significantly different. Compositional learning can be viewed as a special case of zero-shot learning problem and the domain shift problem is commonplace in Zero-Shot Learning. We will mention that earlier in the revised version.\n\n-------\n\n\n**difference between Compositional Learning and Grounded Compositional Concept Learning in one sentence**\n\n**Ans:** GCCL is a specific case of compositional leaning which focus on the compositional learning's systematic aspect and the grounding part. Compsitional learning in the vision and language setting is mostly concerned with grounding that is why we call it grounded compositional concept learning. However, there are compositional learning settings that are defined only based on a single modality like natural langue. For example working on longer length text when trained on shorter text, etc.\n\n\n-----\n\n**If the difference between the training data and the test data is too large, will the retriever still work?**\n\n**Ans:**  The performance will degenerate as shown in Fig. 6. This is caused by the domain-shift problem and the challenge of GCCL.\n\n\n--------\n\n**difference between the retriever and the attention mechanism**\n\n**Ans:** Retriever uses the attention mechanism to calculate the primitive concept's embedding and construct the element concept database.\n\n-----", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The task of Grounded Compositional Concept Learning is useful. \n- The scale of experiments is decent, and the empirical evaluation is solid.  ", "reasons_to_reject": "Minor issues:\n- Misspell: the first paragraph in the Introduction, `propose a multi-modal copositional problem`\n- Misspell: in the first contributions, `address the domain-shift problem in compositioanl learning...`\n- Misspell: the last sentence in Compositional Learning, `compptional problem`\n- Misspell: Figure 2 ... `predict the masked conpositional concept conditioned`\n\nMajor issues:\n- In the contribution of this paper, it is mentioned to deal with the domain-shift problem in compositional learning. Which existing works have this problem, or do all the compositional learning have it? It is not mentioned above, and it is better to explain it earlier. \n- Please explain the difference between Compositional Learning and Grounded Compositional Concept Learning in one sentence. \n- If the difference between the training data and the test data is too large, will the retriever still work?\n- Please briefly explain the difference between the retriever and the attention mechanism in this paper. \n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focuses on systematic generalization within the multi-modal setting, proposes a multi-modal compositional problem, Grounded Compositional Concept Learning (GCCL), and proposes two benchmarks: CompFlickr and CompCOCO, which enrich zero-shot vision-language tasks, from end-task level to token-level. MetaReVision is proposed to solve this problem.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MtVIfDELF4", "reviewer": "Reviewer_6gSR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**It is not clear what practical applications can be associated with the GCCL task and the proposed MetaReVision method is too complicated, which is hard to follow.**\n\n**Ans**: We thank the reviewer for acknowledging the novelty of our approach for the integration of retrieval and meta-learning. \nTo the first question on the application of GCCL, we believe this approach can leverage various applications in VL area. The straightforward one is image captioning where we want to include detailed composition explanations of objects and attributes. \nFor the reading difficulty, it would be great if the reviewer could provide comments on the unclear parts that we can improve.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It is novel to integrate retrieval and meta-learning.", "reasons_to_reject": "It is not clear what practical applications can be associated with the GCCL task, and the proposed MetaReVision method is too complicated, which is hard to follow.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper defined the Grounded Compositional Concept Learning (GCCL) task, collected two datasets (CompFlickr and CompCOCO) for this task, and proposed a MetaReVision method to address this task, which is based on retrieval and meta-learning.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper defined a new Grounded Compositional Concept Learning (GCCL) task, modified two existing datasets into CompFlickr and CompCOCO for this task, and proposed a method based on retrieval and meta-learning for this task,

All reviewers were in consensus that the paper is sound with sufficient experiments backing up their main contribution of combining meta-learning techniques with retrieval methods.

On the flip side, reviewers were also concerned regarding the practical application of the proposed approach, since the main experiments are conducted on synthetically modified versions of existing image-text datasets. There were also concerns that experiments were tested primarily on relatively old methods, and as a result there were no reviewers who felt particularly excited about the new practical applications of the methods proposed in the paper. I would suggest the authors try to tailor their problem setting so that the more recent (generative) vision language models are compatible, which should lead to a more accurate reflection of the difficulty of the problem and can pave the way for more impactful methods designed for the problem. It would also help to have one key experiment on a truly real-world task (not modified from other datasets) in order to showcase how realistic the task actually is.
Decision: Accept-Findings