CoLLIE: Continual Learning of Language Grounding from Language-Image Embeddings
Authors: Gabriel Skantze
Bram Willemsen
Venue: ICLR 2022 Submitted (2022)
Keywords: Continual Learning
Language Grounding
Language-Image Embeddings
Multimodal Distributional Semantics
Reference Resolution

Abstract: This paper presents CoLLIE: a simple, yet effective model for continual learning of how language is grounded in vision. Given a pre-trained multimodal embedding model, where language and images are projected in the same semantic space (in this case CLIP by OpenAI), CoLLIE learns a transformation function that adjusts the language embeddings when needed to accommodate new language use. Unlike traditional few-shot learning, the model does not just learn new classes and labels, but can also generalize to similar language use. We verify the model's performance on two different tasks of continual learning and show that it can efficiently learn and generalize from only a few examples, with little interference with the model's original zero-shot performance. 

Under review as a conference paper at ICLR 2022
COLLIE: CONTINUAL LEARNING OF LANGUAGE GROUNDING FROM LANGUAGE-IMAGE EMBEDDINGS
Anonymous authors Paper under double-blind review
ABSTRACT
This paper presents CoLLIE: a simple, yet effective model for continual learning of how language is grounded in vision. Given pre-trained multimodal embed- ding model, where language and images are projected in the same semantic space (in this case CL.IP by OpenAI), CoI.LI learns a transformation function that ad- justs the language embeddings when needed to accommodate new language use. Unlike traditional few-shot learning. the model does not just learn new classes and labels, but can also generalize to similar language use. We verifyy the model's s performance on two different tasks of continual learning and show that it can ef- ficiently learn and generalize from only a few examples, with little interference with the model's 's original zero-shot performance.
INTRODUCTION
Any artificial agent interacting with an environment, using vision, and communicating with other agents (such as humans), using language, needs to be able to ground the meaning of language with the visual properties of the environment. One approach language into a joint semantic embedding space (Frome et al. 2013 Bruni et al. 2014). In such a model, a visual stimulus and a language construct that have similar representations are supposed to have similar meanings. In order to name a given object with certain visual features, the agent should try to generate referring expression that has a similar embedding as the visual features of the object, and in order to understand what referring expression is denoting, it should look for objects that have a similar visual feature embedding as that of the referring expression.
this problem to project vision and
Recent developments in multimodal representation learning using large amounts of data have given impressive results. An example of model integrating language and vision is CLIP by OpenAI (Radford et al., 2021), which was trained using constrastive learning 400 million pairs of images and their captions. Images and texts are embedded (separately) using state-of-the-art computer vision and language processing pipelines into a 512 dimensional vector. By calculating the dot product of the two embeddings, it is possible to determine how similar an image is to a text (or an image an image, or text text), as illustrated in Figure Db. The model was shown to be very effective at so-called zero-shot learning. which for CLIP means that the model can do image retrieval by ranking the similarity of images to a given label (such as "a black cat"). This can be contrasted with traditional image classification, where the model is specifically trained to classify images into a predefined set of categories (e.g., Deng et al. 2009). In addition to being more flexible (as it can use virtually infinite set of categories), CLIP was also shown to be more robust against noise and variations in the images, compared to supervised image classification (Radford et al. 2021).
While such model is potentially very useful for agents that need to ground language in vision, it limited in that trained once, without any mechanism for updating its representations in light of new data, unless the entire model is retrained and the number of new examples is sufficient. This is clearly limiting the model's usefulness in real-life application scenarios for agents interacting in dynamic environment. Not only will new objects with new properties emerge, but the way humans talk about objects changes over time. As has been shown repeatedly in experiments on human-human interaction, this is not only a long-term issue, but the exact meaning of language may often be negotiated and evolve during the course of a single interaction, and then develop into partner-specific language use (Brennan & Clark 1996; Shore & Skantze 2018). This phenomenon has been referred to as conceptual pacts (Brennan & Clark 1996), or more generally as alignment

---

Under review as a conference paper at ICLR 2022
(a) Zero-shot learning
(b) Few-shot learning
(c) CoLLIE
red
score
score
the monstor
0.01
the blue barn
0.01
dof product
1.00
Classifier
Transformation
CLIP lext
CLIP Image
CLIP image
CLIP
CLIP Xeat
Image
"the red monitor"
"the red monitor"
Unlimitad categories
Unlimited categories
Unlimited catagoriss
Continual learning
Continual leaming
Continual learning
Figure 1:
Comparison of CoLLIE Zero-shot and Few-shot learning. Green boxes show where continual learning is taking place.
in communication (Pickering & Garrod 2006). For example, if hard-to-describe object is being referred to, the partners might establish a new name for it and then continue using that name for similar objects.
The ability to continually learn over time has been a long-standing challenge for machine learning and artificial intelligence, and this area of research has been referred to as continual or lifelong learning (Parisi et al. 2019). There are several problems involved in this. First, humans can learn new concepts using only few examples, in contrast machine learning models that typically need several orders of magnitude more examples. Second, computational models have been shown to be prone to so-called catastrophic forgetting (Parisi et al., 2019). Unless the model re-trained entirely from scratch (which is infeasible for large models like CLIP), the updates to the model's parameters might interfere with previously learned knowledge, resulting abrupt performance drops. This is also referred to as the stability-plasticl dilemma (Parisi et al. 2019).
Learning from few examples greatly depends on having powerful enough representations. Thus, the first problem has been addressed using transfer learning. where fixed base model learns rich general representations from other (but related) tasks. This base model is then used as input to simple classifier with only few parameters (such as logistic regression), requiring only few training examples. This is often referred to as few-shot learning (Wang et al. (0200). Since the CLIP model is trained learn powerful general representations, it was also shown to be fairly good base model for few-shot learning (Radford et al. (021). In principle, an agent that sees a new object and hears being referred as red monitor" by human could train few-shot classifier be able to identify such objects in the future, as illustrated in Figure Пp. However, a problem with this form of few-shot learning is that it is based on the same principles as the conventional supervised image classification discussed above, where labels do not have any inherent meaning, but are instead treated as atomic symbols. If agent using a language-image image embedding model (such as CLIP) would learn a new label using this approach, it is not clear when it should use its hase model to resolve language-image relationships (as in Figure [T), and when should apply the newly learned classifier for the specific label (as in Figure [p). Moreover, those new categories would have no relationship to other (previously known or newly acquired) categories. For example, if the agent would learn how the term "the red monitor" is used in specific situation, it would not be able to infer that "the red display" might be used in similar way. In addition, it is unclear how it should be able to make use of semantic compositionality i.e., combine (in principled way) the newly acquired language with the language it already knows in compositional manner, to understand expressions such as "the blue monitor".
In this paper, we propose COLLIE, a simple, yet effective, model for Continual learning of Language grounding from Language-Image Embeddings. The general principle of CoLLIE is illus- trated in Figure Te. Instead of learning new model for each new concept (as in few-shot learning),
2

---

Under review as a conference paper at ICLR 2022
the model relies on base model of tnnguage-image embeddings in joint embedding space (CLIP in our case). with zero-shot capabilities. We then use and update a separate transformation model that makes adjustments to the language embedding to better fit the new concepts that are being learned, when needed. Thus, the continual learning is only taking place in this transformation model, while the base model is fixed. Our aim is achieve the following characteristics:
Sample efficient: We want to be able learn new language-image mappings quickly with only few examples.
The transformation model is very lightweight and relatively cheap to retrain. While ideally continual learning should happen without any retraining (so-called "rehearsal") and without keeping training data in memory (Parisi et al. 2019). we accept rehearsal given that only the transformation model needs to be re-trained. We will return to this issue in the discussion section.
Computationally efficient:
Generalizable: We want to be able to use the newly learned concepts to understand new related concepts.
As the model learns new concepts, it should continue to perform equally well on tasks it could do before, and newly learned concepts should not interfere with each other.
Robust:
2 RELATED WORK
Language grounding is a core problem of AL, and is related to the more general problem of symbol grounding. i.c., how the symbols used by AI system get their meaning in terms of how they are anchored to the external world (Harnad 1990). In the field of computational linguistics, there is a long history of research reference resolution), as well as how to generate referring expressions, based on the visual properties of the target referent and potential distractors (Krahmer & van Deemter 2012 Qiao et al. 2020). In the field of computer vision, the related problems of image captioning (generating a text describing an image) and image retrieval and object detection (from natural language descriptions) have also been studied extensively (c.g.. You et al. 2016; Hu et al. 2016). Recently, there has also been lot of research done the areas of Visual Question-Answering (VQA) and Visual Dialog, where visual language understanding and generation is combined and treated in an end-to-end fashion (Kafle) & Kanan 2017 Das et al. 2017). However, these studies typically assume that a fixed model of language grounding can be trained, and that the language use does not change after that.
how to infer the referents of referring expressions (so-called exophoric
Previous research on continual learning in image classification has mainly studied the effect of adding new classes the model (Kemker & Kanan, 2018; Kemker et al.) 2018). Our work is different, in that it is not based on image classification with a limited set of classes, but rather on adjusting model that can do zero-shot image retrieval, where the language can form virtually endless number of "classes".
The challenge of how achieve continual learning without catastrophic forgetting has been studied for a long time, since early studies of continual learning in neural networks showed that this was indeed serious problem (MeCloskey & Cohen 1989). Parisi al. (2019) outline three basic ap- proaches to alleviate catastrophic forgetting for continual learning in neural networks: First, various regularization approaches may used to impose constraints on the update of the model's parame- ters. Second, it is possible to allow the architecture of the network to change, c.g., by adding neurons or layers. Third, complementary learning systems are inspired by the human brain, in that they rely on an interplay between episodic memory (specific experience) and semantic memory (general structured knowledge), where learning first happens in the former, and is eventually consolidated with the latter (during "sleep"). CoLLIE does not fit squarely into any of these, but comes closest to complementary learning in its use of a base model (where parameters are fixed) and a dynamic model (where learning happens).
3 DATA, TASK AND METRIC
For our evaluations, assume that the task is rank set of candidate referents based how well they match a referring expression. As our metric, we use the Mean Reciprocal Rank (MRR),
3

---

Under review as a conference paper at ICLR 2022
which is equal to 1 divided by the assigned rank of the correct candidate, yielding score between 0 and 1. Thus, an MRR of corresponds to ranking the correct candidate first and 0.5 corresponds to ranking it second (which can still be considered quite good if the number of candidates is large). The reason we choose MRR instead of accuracy is that it does not only take the top-ranked candidate into account, and therefore can be considered to be more nuanced metric.
In this paper, we use two datasets. First, we use the LAD dataset (Large-scale Attribute Dataset) by Zhao et al. (2018), from which we selected a set of 200 categories belonging to the super-categories animals, fruits, electronics and vehicles, with a total of 68247 images. To verify CLIP's zero- shot performance on this datasef" we did 20 iterations where we randomly selected one image per category (i.e., 200 images) and performed the ranking task using the LAD labels of the categories as referring expressions, yielding an MRR of 0.773. We think this confirms CLIP's impressive zero-shot performance on these types of images.
To study a more challenging set of images (for CLIP), we also use the images from the KTH Tangrams dataset (Shore et al., 2018). Orginially, these images were used in an experiment where participants were asked to take turns referring to tangram figures (using spoken language) while the other participant was trying to identify them on common game board (Shore & Skantze (018)) To assess CLIP's zero-shot performance on these tangram figures, we took the 17 shapes used in the study and made colored versions of them (red, green, blue, yellow and purple), constituting set of 85 candidate referents. The referring expressions were constructed by combining the color with the name of the shape used by the authors of the paper (e.g., "the blue giraffe"). As expected, CLIP's zero-shot performance on these referents is not as good, only yielding an overall MRR of 0.31. The MRR for the individual shapes are shown in the Appendix (Figure 77. While some shapes are identified correctly ("mountain", "barn"), most of them are not. This is of course understandable, given that these images are not representative of CLIP's training data.
In fact, it was not that easy for the human participants in the experiment to do this task either, at least not for their initial attempts. However, they soon started to invent names for the different shapes, forming conceptual pacts after repeated interactions and making the interactions more efficient over time. If an artificial agent should be able to engage in such a task, would clearly have to be able to apply some form of continual learning in the way outlined in the introduction. For this to work, CLIP still needs to have a good representation of the images. To investigate whether this is the case, performed a t-SNE analysis (Van Der Maaten & Hinton 2008 the CLIP embeddings of the colored tangram images to reduce the 512 dimensions to dimensions, as illustrated in Figure i in the Appendix. As can be seen, the shapes and colors seem form clusters and to be handled in somewhat consistent fashion, which indicates that it should indeed be possible to learn names for them.
THE COLLIE TRANSFORMATION
The idea behind CoLLIE is to learn transformation function, T == T(t)) R512 - R512, which takes the CLIP embedding of the text, f, and returns another transformed embedding, т, that better represents the new language use, and is closer the CLIP image embedding i, as illustrated in It is important to note that in order to retain the zero-shot performance, T should in most cases return a similar output as input, unless the text has a domain-specific meaning that the model should correct for.
Figure 2
The transformation function is modelled as T(t) = ++ A(t) x S(t), where A(L) R5122 -> R222 is an adjustment function, and S(t) R512 + 10 is a scaling function. As we will see, this scaling function helps to retain the zero-shot performance of the model. It can be noted that this principle is similar to that of Residual connections in neural networks (He et al. 2016) and Gated Linear Units (Dauphin et al., 2017).
Training examples are stored as pairs of text and image embeddings (1, i). and thus have limited footprint (512+512 floats). A(t) (f) is then trained to estimate the difference vector i-t, using the accu- mulated training examples. We learn A using linear regression: A(t) 3t m, B E g512x512 m e R512 To avoid overfitting (given the limited number of training examples) we use ridge regression
For the experiments in this paper. we use the publicly released trentr CLIP model with the V/T-B/32 Vision Transformer architecture Phttps://gite www.t/...coooaaaooaoooooo/eee wwwm/ppp/a//c.11
4

---

Under review as a conference paper at ICLR 2022
similarlyy
A(t.) =
7(t) [ A(0 S(t)
Dol product
Transformed
Predicled Scalling
Language Embedding
Adjustment
512
A(l)
Embodding
512
Original Language Embedding
4 = tat the
"hh red monitor")
CLIP
512
image
4 CLIP mage!
CLIP
tedt
4 = CLIP, ind
(" blue monitor")
4 = CLIP mag!!
"the red monitor"
5 T()) = All)) x Stl,)
Figure 2: The CoLLIE transformation.
Figure 3: A principled illustrated of the intu- ition behind CoLLIE.
(L2 regularization with A 0.001). The objective of S is to return a value close to when the input is text that should be transformed (i.e., close to any example in the training set), and close to O otherwise. We learn S using support vector regression (SVR) with an RBF kernel (forced in the range [0, 1). The accumulated training examples are used as positive examples (with training target 1). As negative examples (with training target 0), we simply use a list of the 1000 most common nouns in English (representing expressions that should not be transformed), which seemed to work relatively well in our initial tests.
Figure illlutraats the intuition behind the model: Given that we have a reference an image ("the red monitor"), ), we encode it using CLIP and get an embedding 41 As can be seen, in this case the text embedding not very close to the embedding of the corresponding image 11. and will thus retrieve the wrong referent. To teach the model to make better predictions in the future, we add the pair (11, in) as training example. Using the accumulated training examples, we train A to approximate the difference vector between the embedding of the image and the text (61 f1). Now, when new referring expression is to be resolved, "the blue monitor", the expression encoded by CLIP into t2- Again, directly using this embedding would result in poor match for this domain. If we now apply the learned adjustment function A(t2), is likely to return a similar vector as 12 - t (given that by is relatively close to t and that we do not have any other, more similar, training examples). Similarly, S(t2) is likely to return a value close to 1. We now get a new vector T2 t2 + A(22) x S(22). which indeed closest the true referent i2.
While our choice of models for A(t) and S(t) are just two examples of classes of functions that could be used in the CoLLIE transformation, we chose them for demonstrating the efficiency of the approach, even when these functions are very simple. Other adjustment and scaling functions could of course be explored in future work. However, given that want to be able to learn these functions with very few examples, it is important that we use models requiring relatively few parameters.
5 EXPERIMENTS
5.1 EXPERIMENT I: LEARNING SHERDO-WORDS FOR REALISTIC OBJECTS
We first devised an evaluation scheme to see whether the model can learn new words for pho- tographic images from the LAD dataset, and monitor the retained zero-shot performance during training. We randomly select set of N categories, Ctrasns (out of the 200 categories) for which want to teach the model new names. We then assign gories, using randomly selected pseudo-words from the Novel Object and Unusual Name (NOUN) Database ("boskot", "derd", "tust", etc.) (Horst & Hout 2016). Training is then performed over five
name for each of these N cate-
5

---

Under review as a conference paper at ICLR 2022
(A) Testingleaming new pseudo-words Guan
(B) Testing existing Cuai
(C) Testing existing Cuter
0.8
0.6
MAR
CLIP
0.4
COLLIE
COLLIE (w/o scaling, neg. ex.)
COLLIE (w/o scaling)
Few-shot learner
2
5
round
round
round
roke
shede
virdex
printer
laplop
computer monitor
keyboard
refrigerator
phone
Figure 4: Performance of the model over five rounds training on the LAD dataset (averaged over 50 iterations, 95% CI), where [Ctrain| = 50. One training example per category is added per round. (A) shows continual learning performance. (B) and (C) show retention performance. Few-shot learner is only applicable for pane (A).
and testing is performed over six rounds (the initial round of testing is performed without training, which reflects the model's zero-shot performance). At the beginning of each round, we randomly select image for each of the 200 categories, without ever reusing images between rounds. We then let the model rank the 200 images potential referents for each pseudo-word, and the MRR is computed as described in Section 3 At the end of each round, we add the images from Ctrain and their associated pscudo-words as training examples (i.c., one example per category) the model, and retrain it. This whole procedure is repeated over 50 iterations (with new pseudo-words and categories randomly selected and assigned), order get a smooth average performance per round. We evaluate and compare the performance using (1) the CoLLIE model, (2) the fixed CLIP model, and (3) few-shot classifier based on logistic regression (implemented
the same way as in Rad- ford et al. (02T). For the few-shot classifier, each pseudo-word is treated as a class. To study the effect of the scaling function, we also add (4) the COLLLI model without the scaling function, and (5) the CoLLIE model without scaling function, but where the negative examples (common nouns) are added as training examples to the adjustment function to produce zero-length vectors (i.c., no adjustment).
The results are shown in Figure 4(A), where = 50. As can be seen, CoLLIE quickly learns the new pseudo-words and reaches fairly good performance (0.618) already after one round (i.e., when it has only been provided with one example per category), increasing to 0.750 at the final round (where five examples have been provided), which is quite close to CLIP's zero shot performance of 0.773 for the original words on this dataset. Here, the scaling function has very little effect. Since the CLIP model not doing any learning, it obviously has very poor zero-shot performance on these new words. However, the few-shot classifier has a slightly better performance than CoLLIE, especially after five examples are added (0.805). This is perhaps not very surprising, given that it is optimizing this classification task, rather than transforming the embedding space. Also, in this specific task, CoLLIE does not benefit from generalization the learned words, as they are arbitrarily assigned and there is no semantic compositionality effect.
To study the retained zero-shot performance the model during training, we also plot the perfor- mance of the models when using the original words for the 50 categories in Ctroins in Figure BB) As can be seen, the original names for those categories can still be resolved by CoLLIE with a slight (but not catastrophic) drop in performance compared to the static CLIP model (0.699 the final round), even though the model has also learned new words for them. In this case, the scal- ing function is important and without it the performance drops considerably (to 0.260). Similarly, for each round, we also study the models' retained zero-shot performance on 50 randomly selected categories, Cother, which Figure CC). Again, when using the scaling function, CoLLIE does not seem to interfere much with CLIP's s original zero-shot performance (0.719 vs. 0.769 at the final round), while there drastic
0.760
not part of Ctrain using their original names. This is shown in

---

Under review as a conference paper at ICLR 2022
Performance of the model (MRR) on the LAD dataset (averaged over 50 iterations), with different numbers of classes/pseulowoorde to learn (N). (A) shows continual learning performance and (B) shows retention performance. n/s negative samples used for the scaling function.
Table 1:
Ctrain
10
50
100
150
10
50
100
150
(A) New pseudo-words
Cerain
After
example
Civain:
After 5 examples
CoLLIE (I000 n/s)
0.616
0.618
0.636
0.634
0.770
0.750
0.746
0.751
CoLLIE (100 n/s)
0.639
0.629
0.639
0.636
0.770
0.750
0.746
0.752
CoLLIE (w/o scaling)
0.654
0.632
0.641
0.637
0.774
0.751
0.746
0.753
Few-shot learner
0.639
0.648
0.660
0.660
0.802
0.805
0.810
0.815
CLIP
0.033
0.030
0.030
0.028
0.028
0.029
0.030
0.031
(B) Original words
Ctrain
After 5 examples
Cother:
After 5 examples
CoLLIE (1000 n/s)
0.790
0.699
0.606
0.519
0.761
0.719
0.608
0.515
CoLLIE (100 n/s)
0.769
0.621
0,450
0.333
0.732
0.638
0,443
0.314
CoLLIE (w/o scaling)
0.515
0.260
0.127
0.078
0.361
0.242
0.112
0.054
CLIP
0.798
0.760
0.768
0.773
0.779
0.769
0.783
0.774
drop in performance (to 0.242) when the scaling function not used. Since the few-shot learner has no zero-shot performance to retain, its performance is not plotted in pane B-C.
As these experiments show, the scaling function is important for the performance of the model, and simply adding the negative examples to the adjustment function does not have the same effect (as seen in Figure 4). In the Appendix (Table 2). we also investigate different implementations of the scaling function, of which SVR has overall favorable performance.
To further investigate the performance, we also run experiments with different numbers of classes/pseudo-woote words to learn (N [Ctrain and different numbers of negative examples in the scaling function. The results are shown in Table [ performance is relatively stable for different values of N. However, retention performance is clearly affected as N increases. This is especially true when only 100 negative examples are used for the scaling function. Thus, is likely that the drop in retention per- formance could be mitigated by adding even more negative examples, as N increases. As seen in (A), the number of negative examples does not seem to have a big effect on the continual learning performance.
As can be seen in (A), the continual learning
seen in (B), the zero-shot
5.2 EXPERIMENT II: LEARNING LANGUAGE FOR TANGRAM FIGURES
The biggest expected benefit of CoLLIE comes from its ability to generalize from the language learning, which was not addressed in Experiment I. We thus devised an evaluation scheme to how quickly the model can learn to identify the colored tangram shapes (introduced in Section 33) for which it clearly had a very poor zero-shot performance. Here, we expect the model benefit from the compositionality of the referring expressions. As discussed earlier, given that it has learned what concept associate with the phrase "the blue rock", should be able to extrapolate this understanding to "the red rock".
Again, the task is to rank the 85 potential referents, given referring expression. Similar Exper- iment I, the model starts out with no training examples (round 0). We then train the model over 30 rounds. each round, one random referent is picked, the model's performance (in terms of MRR) on this referent is assessed, the image-text pair of the referent is added to the training set, the model retrained, and a new round begins. This whole procedure is then repeated over 3000 iterations, resetting the model after each iteration. The MRR per round (over all iterations) is illustrated in Fig- ure 5 We do a similar comparison with other models as in Experiment I. Here, we let the few-shot learner (again, logistic regression classifier) fall back CLIP when it is presented with referring expression it has not seen before. We then add the new referring expression as a new class for the few-shot learner, and retrain it.
As can be seen, COLLIE quickly learns the names for the tangrams, reaching an MRR of 0.860 after 30 rounds. Note that the 85 images are all unique in terms of shape-color combinations. which means that the model must be able to generalize in order to achieve this performance. In
7

---

Under review as a conference paper at ICLR 2022
0.9
0.8
CoLLIE (without scaling)
0.7
CoLLIE
Few shot leamer
0.6
MRR
0.5
0.4
0.3
0.2
0
5
10
15 round
20
25
30
Figure 5: Performance of the model over 30 rounds of training (averaged over 3000 iterations) when training on the colored tangrams (95% CI). One new training example is added per round.
contrast, the few-shot learner has much are introduced in most rounds in the beginning (unless the exact same object happened to be picked twice), and has no way of generalizing from already learned classes. This confirms the hypothesis that CoLLIE should be able to benefit from the compositionality of language: After being taught what "red giraffe" looks like, CoLLIE is now better identifying a "blue giraffe", combining the base representation of "blue" with the learned meaning of "giraffe".
performance,
referring expressions (classes)
To further confirm this ability, we also performed an experiment where we first train the model on all 17 shapes of one random color, and then evaluate it on the same shapes with different random colors. This was iterated 100 times. Whereas the CLIP baseline model (and the few-shot learner, which has to fall back on the CLIP model) only had an average MRR of 0.317 on these unseen combinations, CoLLIE achieved an MRR of 0.857.
The intuition behind why this works was illustrated in Figure 3 CoLLIE learns to predict the dif- ference vector that needs to applied. Thus, the color dimensions in the CLIP embedding were already aligned between the language and the image, there will not be any need to adjust those di- mensions is only the dimensions related to the shape that need to adjusted. The fact that this works despite CLIP's representation being entirely distributed is interesting. The steady im- provement also indicates that the learning of each concept does not interfere with learning other concepts. However, as can seen, without the scaling function, the model has drastic drop in performance for the first rounds, which is likely because the newly learned adjustments are added generously unrelated referring expressions.
As further (limited) test to verify the model's ability to generalize, we substituted the names of the "mricken">"hhen"" etc.). This way, we formed referring
tangrams with synonym ("barn"-"" expressions such as "the blue hen". Using the CoLLIE model trained for 30 rounds as described above, we then evaluated these expressions (over all 3000 iterations). The MRR for these was 0.602, which is clearly better than the baseline of 0.290 (using CLIP directly), providing further evidence for the model's ability to generalize. Given that many of the names had no obvious synonyms, the individual performance of them varied greatly (MRR 0.056-0.920). A breakdown of these results can be found in the Appendix (Figure
DISCUSSION
Several previous studies have addressed the problem of incremental class learning in image classifi- cation (Kemker & Kanan 2018 Kemker et al. 2018). However, to the best of our knowledge, the
TTaken from thesaurus.com
8

---

Under review as a conference paper at ICLR 2022
problem of continual adjustment of language-iman embeddings to learn new language grounding has not been addressed before. Thus, we do not have any results from prior work to compare our performance with. However, we hope that this work can serve as benchmark for future studies and alternative models.
Returning to the four characteristics we aimed to achieve, we think that the model has shown to be sample efficient, as it seems to reach a fairly high performance with only one training exam- ple per new category. Second, Experiment II showed that the model was able to generalize from the newly learned language use, thanks to the semantic compositionality of the referring expres- sions. Third, Experiment showed that the model was fairly robust, as despite slight drop in the model's 'ss original zero-shot performance, it did not exhibit catastrophic forgetting. Finally, the model fairly computationally efficient, seeing as the transformation model uses very simple models with few parameters and the stored training examples have very small footprint. Nevertheless, the transformation model needs to be retrained when new examples are added, so there are limits to its scalability. Whether this is a problem, however, depends entirely on the use case scenario. Regardless, the continual learning of the transformation function without storing examples is also an interesting topic for future work.
As we have seen, the scaling function plays a very important role in retaining the model's s zero-shot performance, making sure that only the newly learned terms are adjusted. However, given how the scaling function was trained here (simply using common nouns as negative examples), this will not always work, and therefore still saw slight drop in zero-shot performance, especially as the number of new concepts to be learned increases. The scaling function could of course be more or the
less restrictive. For example, could require an exact match with a training example scale to I, and O otherwise. This would retain all of the zero-shot performance, at the expense of being able to generalize the learning to similar language use. Exploring more sophisticated scaling functions that provide a good balance between retention and generalization is an interesting topic for future work. For example, Shin et al.) (017) explore the use of a generative model to generate samples for rehearsal, which alleviates the need for storing training examples.
Of course, CoLLIE's performance also relies on CLIP already having good representations of the "new" categories, and so it can be argued whether the categories themselves are really new - it is rather the transformed embedding of the language that better maps to this region in the embedding space. In other words, it might be fair to say that the model learns domain-specifer language use. CoLLIE's performance is thus limited by the performance of the base model (CLIP in this case) to accurately represent the landscape of visual properties of objects. As pointed out by RRafford et al.) (202), CLIP's representations are limited in certain aspects, including counting objects in an image or representing detailed attributes.
An interesting topic for future work is how to consolidate the learned transformation function into the base model, and then learn new transformation function on top of this, or use different transformation functions in different contexts, as language use is highly context dependent. Another line of future work is to incorporate the model into a system that learns through interaction. Given the small number of examples needed to learn new language use, the model should be interesting for studies on continual language grounding in the context of human-robon interaction (e.g.. Chai et al. 2016).
7 CONCLUSION
We have presented CoLLIE: simple, yet effective model for continual learning of how language is grounded in vision. Given a pre-trained language-image embedding model capable of zero-shot image classification (CLIP), CoLLIE learns a transformation function that adjusts the language em- beddings when needed to accommodate new language use. The transformation function learns the difference vector that needs to be applied to the embedding, and uses scaling function to retain em- beddings that should not be adjusted. We establish new benchmarks with novel metrics to capture the trade-off between continual learning, retention (avoiding catastrophic forgetting), and general- ization. The evaluation showed that the model can learn new language use with very few examples. Unlike traditional few-shot learning, the model does not just learn new labels, but can also generalize to similar language use, and benefit from the semantic compositionality of language.

---

Under review as a conference paper at ICLR 2022
8 REPRODUCABILITY STATEMENT
The models were implemented using scikit-learn dard parameters unless otherwise stated. The code used for running the experiments and reproducing the results in this paper is provided as supplementary material, including necessary data or pointers data.
with stan-
REFERENCES
Brennan and Herbert Clark. Conceptual pacts and lexical choice in conversation. Journal of Experimental Psychology, 22(6))1482-1444 1493, 1996.
Elia Bruni, Nam Khanh Tran, and Marco Baroni. Multimodal Distributional Semantics. Journal of Artificial Intelligence Research, 49: 1-47, 2014.
Joyce Y. Chai, Rui Fang, Changsong Liu, and Lanbo She. Collaborative language grounding toward situated human-robot dialogue. Al Magazine, 37(4):32 2016.
Abhishek Das. Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M.F. Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In Proceedings 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, pp. 1080-1089, 2017.
Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference Machine Learn- ing- Volume 70. pp. 933-941,220 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 JEEE Conference Computer Vision and Putlern Recognition, pp. 248-255, 2009.
Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. Advances in Neural Information Processing Systems 26 (NIPS 2013), 2013.
Stevan Harnad. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1-3):335- 346, 1990.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
J. S. Horst and M. C. Hout. The novel object and unusual name (noun) database: A collection of novel images for use experimental research. Behavior research methods, 1994 2016.
Ronghang Hu, Huazhe Xu, Mareus Rohrbach, Jiashi Feng. Kate Saenko, and Trevor Darrell. Natural language object retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Kushal Kafle and Christopher Kanan. Visual question answering: Datasets, algorithms, and future
challenges. Computer Vision and Image Understanding, 163:3-20, 2017. ISSN 1090235X.
Ronald Kemker and Christopher Kanan. Fearnet: Brain-inspired model for incremental learning. In Proceedings of ICLR 2018, 2018.
Ronald Kemker, Mare McClure, Angelina Abitino, Tyler Hayes, and Christopher Kanan. Measuring catastrophic forgetting in neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Emiel Krahmer and Kees van Deemter. Computational Generation of Referring Expressions: A Survey. Computational Linguisties, 38(1):173-218, 2012.
10

---

Under review as a conference paper at ICLR 2022
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of learning and motivation, 14:109-165, 1989.
German Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019.
M Pickering and S Garrod, Alignment as the Basis for Successful Communication, Research on Language and Computation, 4:203-228, 228, 2006.
Yanyuan Qiao, Chaorui Deng. and Qi Wu. Referring expression comprchension: A survey of meth- ods and datasets. IEEE Transactions on Multimedia, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. arXiv:2103.000200 2021.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual Learning with Deep Genera- tive Replay. In NIPS' 17: Proceedings of the 31 International Conference on Neural Information Processing Systems, pp. 2994-994499 3003, 2017.
Todd Shore and Gabriel Skantze. Using Lexical Alignment and Referring Ability to Address Data Sparsity in Situated Dialog Reference Resolution. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 2288-2297, , 2018
Todd Shore, Theofronia Androulakaki, and Gabriel Skantze. Kth Tangrams: A dataset for research on alignment and conceptual pacts in task-oriented dialogue. LREC 2018 11th International Conference on Language Resources and Evaluation, pp. 768-775, 2018.
Laurens Van Der Maaten and Geoffrey Hinton. Visualizing data using t-SNE, Journal of Machine Learning Research, 2008.
Yaqing Wang, Quanming Yao, James Kwok, and Lionel Ni. Generalizing from few examples: A survey few-shot learning. ACM Computing Surveys (CSUR), 53(3):1-34, 2020.
Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. Image captioning with semantic attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Zhao, Yanwei Fu, Rui Liang, Jiahong Wu, Yonggang Wang, and Yizhou Wang. large-scale attribute dataset for zero-shot learning, arXiv: 043.144444.44,,0000 2018.

---

Under review as a conference paper at ICLR 2022
A
APPENDIX
X
X
N
Figure 6: t-SNE dimensionality reduction of the colored tangram CLIP embeddings.
Table 2: Performance of the model (MRR) on the LAD dataset (averaged over 50 iterations), with different implementations of the scaling function, where |Ctrain] 50.
Learning
new for
Testing existing
Testing words for Cother
existing
escudo-words
words for Crratn
Ctroin
Round
5
SVR (RBF)
0.630
0.758
0.721
0.711
0.711
0.693
SVR (sigmoid)
0.574
0.756
0.759
0.690
0.749
0.672
SVR (linear)
0.614
0.758
0.736
0.684
0.729
0.667
SVR (poly)
0.635
0.758
0.711
0.715
0.701
0.698
Logistic regression
0.633
0.756
0.656
0.679
0.649
0.657
Linear regression
0.640
0.759
0.646
0.644
0.640
0.624
12

---

Under review as a conference paper at ICLR 2022
Shape
HITOW
barn
boat
bridge 0,097 0.924
chevron 0.338 0.902
chicken 0,108 0.841
crown 0.843 0.883
giraffe
CLIP
0.114
1.000
0.095
0.050
0.053
COLLIE
0.848
0.829
0.806
0.837
0.864
Synonym
pointer
shed
ship
overpass
rank 0.599 0.597
tiara
camel
0.073
1.000
0.130
0.109
0.616
0.059
0.059
CoLLIE
0.251
0.807
0.761
0.920
0.727
0.852
0.325
0.456
V
Shape CLIP
head
lozenge 0.351
monitor 0.207
mountain
spikes 0.392
temple
wedge
0,099
1.000
0.162
0.290
0.096
COLLIE
0.893
0.892
0.778
0.990
0.914
0.835
0.753
0.845
Synonym
skull
troche
screen
hill
stone
spears
church
chock
CLIP
0.134 0.651
0.382
0.331
0.672 0.880
0.162
0.166
0.211
0.124
CoLLIE
0.807
0.362
0.877
0.310
0.597
0.056
Figure 7: Performance (MRR) on individual tangram shapes and their synonyms. Both the original zero-shot performance of CLIP (over the 85 candidates), and the performance of CoLLIE (after training on 30 examples with the original names), are shown.
13


Reviews:
[{"review_id": "e7bEGvMI4wX", "reviewer": "Reviewer_qatX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is unique in its problem setting because it combines continuous learning with language and image embedded representations. Although the proposed method is simple, the experimental results show its ability to deal with new categories while maintaining the original performance.\n\nOn the other hand, this paper adopts only CLIP as an existing representation learning method. Although CLIP is a modern and excellent representation learning model, the authors should have conducted similar experiments with other representation learning to show that the superiority of the proposed method is a general contribution to existing representation learning methods. Otherwise, the advantage of the proposed method may exist only when applied to CLIP.\n\nIn addition, the proposed method is in the form of a product of an adjustment function that performs a linear transformation and a scaling function that outputs [0,1], which is added to the original representation. It is just a special form of the combination of Gated Linear Units (GLU) (Dauphin et al., 2017) and residual connections. The authors state that the proposed method is simple, but they do not mention that they propose the same as GLU.\n\nFurthermore, the adjustment function and the scaling function are trained with different regression methods. However, there is no explanation why the scaling function, in particular, was implemented with support vector regression. In linear support vector regression, the output of [0,1] as defined by the authors as S(t) cannot be satisfied, and negative values and values larger than one may be output. As described above, the Transformation module is regarded as a combination of the GLU and the residual connection. Similar effects may have been obtained when the module is trained so that the vector representation of the text in the negative sample is output as it is and that the vector representation of the text in the positive sample matches the vector representation of the image. It is unclear why ad hoc learning as proposed is necessary.\n\nFinally, as the authors also state, the rationale for learning the 1000 most common nouns as negative examples is questionable. It may have worked in this case, but what would happen if these nouns were also included in the Few-Shot learning text?\n\nThe following are minor concerns.\n- The proposed method's behavior would have been better understood if the output of the scaling function alone had been reported, along with Figure 5, for the new pseudo-words and the existing words.\n- The authors claim that the experimental results in section 5.2 show the hypothesis that the proposed method can learn blue giraffes better if it learns red giraffes. However, section 5.2 only shows that MRR as a macroscopic retrieval accuracy has improved. Other experiments and evaluations are needed to show whether this hypothesis truly holds. For example, we could train the few-shot training with only red giraffes and then evaluate whether the blue giraffes can be retrieved in zero-shot.\n- Finally, CoLLIE is only described in this abbreviation, but its official name is unknown.", "summary_of_the_paper": "This paper proposes a simple new method for continuous learning of language-image embeddings. The proposed method, CoLLIE, employs a Transformation module designed to work only on the samples undergoing few-shot learning. It consists of a scaling function that determines whether the text is close to the sample undergoing few-shot training and an adjustment function that estimates the difference between the embedded representation of the paired text and the embedded representation of the image. The experimental results show that the proposed method can deal with new categories by few-shot learning while retaining the zero-shot learning performance of existing embedded representations (CLIP).", "summary_of_the_review": "### Original Summary\nThis paper proposes a simple and effective method for a unique problem setting. On the other hand, the novelty of the method is much less than the authors may think, and a similar method is not cited. The rationality of the design of the proposed method is also questionable, and its versatility has not been experimentally demonstrated. The reviewer recommends that the authors expand the experiments and resubmit the revised paper to another international conference.\n\n### Final Rating\nThe reviewer has read the other reviews, the responses from the authors, and the revised manuscript. The first score (3) has been upgraded a little to (5), but the reviewer still leans to reject the paper. The reasons are described in the last post of this thread.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MACnBgkGCpq", "reviewer": "Reviewer_jzf6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# strengths and weaknesses\n- Strengths\n   - Clear and thoughtful discussion of the motivation, namely the importance of grounded language learning and continual learning. I enjoyed reading it!\n   - Straightforward description of the main technical contribution\n - Weaknesses\n   - As presented, I'm not sure if the presented approach has signicant technical novelty. The proposed solution boils down to finding a linear transformation for the frozen CLIP representations. I would argue that this is already present in any system that uses a foundational model, e.g., BERT or CLIP, to produce input representations for a downstream task (see [1] for broad overview of typical practices). It's common practice for the model to include a fully-connected feed-forward network for transformation of these input embedding.\n   - As presented, I don't think the given approach can properly be called \"continual learning\". CoLLIE re-purposes the CLIP representations for a specific task, but the CLIP model itself is unchanged, and no transfer-learning or fine-tuning happens on the CLIP weights. In sec. 6 its mentioned that future work could explore incorporating the \"learned transformation function into the base model.\" I agree that this direction is worth exploring and I think it would yield results that would better fit the description of \"continual learning\".\n   - In a similar vein, it's unclear how this approach could be scaled to multiple tasks, which is a vital part of any continual learning system (see comments below)\n\n# Comments and clarifying questions\n- sec. 4: Regarding the function $A(t) = \\beta t + m$, could the authors give a brief explain for their choice of a affine transformation function here? Was this motivated by some previously observed property of the CLIP embedding space? Why should an affine transform be preferred over any other non-linear transform, e.g. quadratic, affine with a non-linear activation etc.?\n- sec. 4: Can the authors comment on how this approach can be scaled to deal with multiple distributional shifts? What happens if the descriptions for some of the tangrams change? Would you propose creating a second $A'$ and $S'$? If so, how would they interact with $A$ and $S$?\n- sec 5.1: What is the \"few shot classifier\"? If it is a multi-class classifier, then how can it be compared to CoLLIE, which is learning a fundamentally different task, namely a mapping between embedding spaces?\n- sec 5.1: Shouldn't the CoLLIE (without scaling) be trained with negative examples in order to give a fair comparison? More exactly, the paper mentions an ablation where the classifier $S(.)$ is removed from the system, so the system only consists of the transformation $A(.)$. Figure 5 is claimed to show that $S(.)$ is necessary to prevent catastrophic forgetting. However, the training for the ablated system only consists of (pseudo-words, image) pairs. So the transformation $A$ does not have any negative examples, which $S$ would normally have access to. \n- sec 6: \"stored training examples have a very small footprint\": If I understand correctly, these training examples are not stored in memory during prediction, so is it very important that their footprint be small?\n\n# Suggestions\n- Sec 1: \"Unless the model is re-trained entirely from scratch (which is infeasible for large models like CLIP), the updates to the model’s parameters might interfere with previously learned knowledge, resulting in abrupt performance drops.\"\n  - As I understand, this reasoning explains why transfer-learning or fine-tuning on CLIP is undesirable. However, is there reason to believe that catastrophic forgetting will occur for the settings considered in this paper (nonce-words, tan-gram descriptions)? The tasks don't seem too dissimilar from what CLIP sees ordinarily during training. I think it would strengthen your case if Figure 5 or Table 1 had results that show how forgetting occurs when the original embedding model is fine-tuned. I understand that CLIP isn't available for fine-tuning right now, but maybe a conceptually similar multi-modal transformer could be used.\n- Sec 3: Could you report the mean squared error between the transformed vectors $T(t)$ and the target vectors?\n  - I ask because it's possible that the similarity ranking for a given embedding may be high relative to the other candidates, but could still be a large distance away from the target.\n  - In that vein, do you have a sense of what images your transformed embeddings correspond to? Can images be decoded for them?\n- Sec 5.2: It is claimed that the results in figure 6 demonstrate that the model can leverage compositional knowledge to generalize. Given that results are averaged over 3000 runs, I am inclined to believe this claim. But it could also be shown explicitly by creating a held-out test-set of unseen color+shape combinations. This would increase confidence in the claim that the model is exhibits some type of systematic generalization.\n\n# citations\n[1] Bommasani, Rishi, et al. \"On the opportunities and risks of foundation models.\" arXiv preprint arXiv:2108.07258 (2021).\n[2] Magassouba, Aly, Komei Sugiura, and Hisashi Kawai. \"CrossMap Transformer: A Crossmodal Masked Path Transformer Using Double Back-Translation for Vision-and-Language Navigation.\" arXiv preprint arXiv:2103.00852 (2021).\n", "summary_of_the_paper": "This paper presents a way to take general-purpose joint text+image representations, e.g., from CLIP, and transform them for use on a new task, while still preserving performance on the old task.\n\nThe gist of the idea is this: given a new task, a classifier is trained to decide whether or not a given input belongs to the new task. If it does, then that input's representation is transformed according to an affine transformation.\n\nMore exactly, the proposed approach can be broken into two pieces. The first piece is an affine transformation $A$ that is used to transform an original CLIP embedding $t$ into a new representation $A(t)$ which is better suited for the new task. The second piece is a scaling function $S(t):\\mathbb{R}^{512}\\rightarrow [0,1]$ that predicts how likely it is that the input $t$ belongs to the new task. For a given input $t$, the transformed vector is then given by $t+A(t)\\times S(t)$.\n\nTwo experiments are described: (1) learning to associate nonce word descriptions with photographic images and (2) learning to associate descriptions with colored tan-grams which were not seen by CLIP in pre-training.\n\nFor both experiments, $S$ is a support vector machine that is trained to identify whether a given descriptions belongs to the new task, and $A$ is trained using linear regression. Performance is scored based on how highly the target image is ranked among a set of other images, where the ranking is done by dot-product similarity.\n\nIn both experiments, they find that their system performs well on the new task, while still retaining the performance on other examples.", "summary_of_the_review": "Overall, the paper is well-written and outlines good directions for future work. The motivations are described clearly and thoughtfully. However, as is, I do not think it represents a significant enough contribution. The approach described is already present in most systems that use a large pre-trained model to provide embeddings for the input. That is, it's common for most existing approaches to take an off-the-shelf model, freeze the weights, and then use the produced embeddings for some down-stream task (see for example [2]). Finally, I'm not sure if the proposed solution can properly be called \"continual learning\" since it's unclear how it should scale with the addition of new tasks.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9J_MenSTGZ-", "reviewer": "Reviewer_tBdi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: the experiment shows that a small network and a simple difference vector are enough to achieve similar performance to the designed few-shot learner.   \n\nWeakness: This paper tries to use a small network to achieve continued learning. The main idea to achieve it is to predict a difference vector between exit-word and pseudo-word. This idea is somewhat similar to the idea of GLoVe, which also has a similar difference vector between multiple word pairs. From the experiment, the idea works for the 2 test cases. However, the test data is constructed (like “the red monitor\" and “the blue monitor\") and thus the reference image-language pairs can easily be found. I wonder that if the text queries are more complex, like a long sentence or a paragraph, will the idea still work. Usually, in a real case, a more complex text is more common than well-defined constructed phrases or words.\n\nIn the experiment, the author mentions a few-shot learner, which is not fully described in the paper. How does the author designs the few-shot learner may also influence the performance? Few-shot learning is now a popular topic and many papers are proposed. I think more experiments for the different few-shot methods on the test dataset will be more convenient.\n\n", "summary_of_the_paper": "This paper target on continual learning for the language-vision ground task with an additional small network. The network is attached on a CLIP model and it contains 2 linear layers to predict a difference vector of the original text embeddings and scaling value. The network are optimized by support vector regression (SVR). The method is test on pseudo-word and  tangram figures tasks. The result shows the model with proposed method can achieve better performance of original CLIP model. \n", "summary_of_the_review": "I think the idea of the difference vector is similar to some previous work. For the experiment, the data is somewhat constructed and simple for the method to find the relative object or description. However, It does not show that if the method can also be applied to the more complex case, which may be more common in the real life. And the comparison is not so sufficient. The author may compare with more few-shot methods to support their contributions. \n\nIn general, I think the paper still needs to be improved and I recommend rejecting it this time.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4FqkQwdZ5OU", "reviewer": "Reviewer_Y4pM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method is simple but effective for the setting it was evaluated in. If the “Related Work” section is accurate, it is introducing a novel and interesting problem setting. The problem itself is well motivated.\n\nMy main concerns are the following:\n\nC1) The method experiences catastrophic forgetting, as the number of classes increases (as shown in Table 1). The authors propose to mitigate it by keeping more negative examples, however this is not desirable for a continual learning setting.\n\nC2) While the authors make an argument that this is a continual method, there are only 2 tasks at any given point - the source task that the original model was pre-trained on, and the new task, consisting of all the classes that need to be classified. Also, it appears that the solution to this setting is based on MTL - having examples from all tasks (examples from the source task are used as negative examples for the gating function) and training the free parameters on them simultaneously. As a result, I’m hesitant to support this being called a continual learning solution.\n\nNote: there's a typo in section 3 using MMR instead of MRR.\n\nOverall, it appears that this paper presents a simple method in an interesting direction, and can server as a useful baseline for further research.", "summary_of_the_paper": "The paper augments a pre-trained multi-modal model, which encode language and images into a shared latent space. Said model needs adjustments in order to account for previously unseen links between phrases and images. The paper adjusts this by learning a linear transformation of the embedding space. A threshold function is used in order to apply the new linear transformation only to relevant inputs.", "summary_of_the_review": "Overall, it appears that this paper presents a simple method in an interesting direction, and can server as a useful baseline for further research.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This manuscript presents a method built on top of CLIP which transforms language embeddings to understand new phrases while maintaining the original abilities of CLIP.

1. The technical novelty of this work is limited. That being said, if the task is of wide interest, a straightforward approach that performs well is not just good, it's even preferable to one that is complex. The authors bring up the fact that reviewers are asked to rate both technical and empirical novelty. This is true. Yet, reviewers were unconvinced both by the method and the setting.

  The manuscript does not explain why this setting provides additional challenges or value compared to the many other continual learning or zero-shot settings that exist in the literature. What the authors say "we are not aware of any previous work that learns a direct transformation of the representations themselves to accomplish both continual learning and retained zero-shot use of those representations with the same model" seems to be undisputed by the reviewers. But is it critically important to future ML research that a single model does both? Or that a model learns a direct transformation to do so? Overall, this task seems very constrained and tailored to this one approach, while usually the more general a setting is, the more it is valued by the community because it will be more likely to stand the test of time and lead to new advances. Reviewers also could not point to a compelling immediate practical need for such a model, which would be another reason for considering a novel setting.

  While in the responses the authors acknowledge that they do not consider their method to be the ultimate solution, that the method has significant limitations, and that this really should be considered a strong baseline, this is not how the work is presented. Relatively little is said in the manuscript about any of these topics.

2. In response to requests for experiments (such as exploring the space of transformations and exploring alternate models to CLIP) the authors put forward that space limitations preclude such experiments. I would encourage authors not to rely on this argument going forward as it does not serve their cause well. Between the unlimited appendix and the possibility of linking to an anonymized website space cannot be a constraint. Science is complex these days and it's not unusual to have to report extensive additional experiments outside the main body of the manuscript. I encourage authors to consider that these requests by multiple reviewers are likely going to be the first questions that the readers of their work will also want answers to. Exploring other transformations and models is critical to understanding the value and impact of the work.

Minor point: I did not see this in the reviews but Figure 1(a) has the labels for CLIP text and image flipped.

If the authors round out the experiments and demonstrate either that their idea is more general-purpose, i.e., that it can be applied to other settings and problems, or that this setting is of great value on its own, this could be a strong contribution.
Decision: Reject