Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper augments a pre-trained multi-modal model, which encode language and images into a shared latent space. Said model needs adjustments in order to account for previously unseen links between phrases and images. The paper adjusts this by learning a linear transformation of the embedding space. A threshold function is used in order to apply the new linear transformation only to relevant inputs.
main_review: The method is simple but effective for the setting it was evaluated in. If the “Related Work” section is accurate, it is introducing a novel and interesting problem setting. The problem itself is well motivated.

My main concerns are the following:

C1) The method experiences catastrophic forgetting, as the number of classes increases (as shown in Table 1). The authors propose to mitigate it by keeping more negative examples, however this is not desirable for a continual learning setting.

C2) While the authors make an argument that this is a continual method, there are only 2 tasks at any given point - the source task that the original model was pre-trained on, and the new task, consisting of all the classes that need to be classified. Also, it appears that the solution to this setting is based on MTL - having examples from all tasks (examples from the source task are used as negative examples for the gating function) and training the free parameters on them simultaneously. As a result, I’m hesitant to support this being called a continual learning solution.

Note: there's a typo in section 3 using MMR instead of MRR.

Overall, it appears that this paper presents a simple method in an interesting direction, and can server as a useful baseline for further research.
summary_of_the_review: Overall, it appears that this paper presents a simple method in an interesting direction, and can server as a useful baseline for further research.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a way to take general-purpose joint text+image representations, e.g., from CLIP, and transform them for use on a new task, while still preserving performance on the old task.

The gist of the idea is this: given a new task, a classifier is trained to decide whether or not a given input belongs to the new task. If it does, then that input's representation is transformed according to an affine transformation.

More exactly, the proposed approach can be broken into two pieces. The first piece is an affine transformation $A$ that is used to transform an original CLIP embedding $t$ into a new representation $A(t)$ which is better suited for the new task. The second piece is a scaling function $S(t):\mathbb{R}^{512}\rightarrow [0,1]$ that predicts how likely it is that the input $t$ belongs to the new task. For a given input $t$, the transformed vector is then given by $t+A(t)\times S(t)$.

Two experiments are described: (1) learning to associate nonce word descriptions with photographic images and (2) learning to associate descriptions with colored tan-grams which were not seen by CLIP in pre-training.

For both experiments, $S$ is a support vector machine that is trained to identify whether a given descriptions belongs to the new task, and $A$ is trained using linear regression. Performance is scored based on how highly the target image is ranked among a set of other images, where the ranking is done by dot-product similarity.

In both experiments, they find that their system performs well on the new task, while still retaining the performance on other examples.
main_review: # strengths and weaknesses
- Strengths
   - Clear and thoughtful discussion of the motivation, namely the importance of grounded language learning and continual learning. I enjoyed reading it!
   - Straightforward description of the main technical contribution
 - Weaknesses
   - As presented, I'm not sure if the presented approach has signicant technical novelty. The proposed solution boils down to finding a linear transformation for the frozen CLIP representations. I would argue that this is already present in any system that uses a foundational model, e.g., BERT or CLIP, to produce input representations for a downstream task (see [1] for broad overview of typical practices). It's common practice for the model to include a fully-connected feed-forward network for transformation of these input embedding.
   - As presented, I don't think the given approach can properly be called "continual learning". CoLLIE re-purposes the CLIP representations for a specific task, but the CLIP model itself is unchanged, and no transfer-learning or fine-tuning happens on the CLIP weights. In sec. 6 its mentioned that future work could explore incorporating the "learned transformation function into the base model." I agree that this direction is worth exploring and I think it would yield results that would better fit the description of "continual learning".
   - In a similar vein, it's unclear how this approach could be scaled to multiple tasks, which is a vital part of any continual learning system (see comments below)

# Comments and clarifying questions
- sec. 4: Regarding the function $A(t) = \beta t + m$, could the authors give a brief explain for their choice of a affine transformation function here? Was this motivated by some previously observed property of the CLIP embedding space? Why should an affine transform be preferred over any other non-linear transform, e.g. quadratic, affine with a non-linear activation etc.?
- sec. 4: Can the authors comment on how this approach can be scaled to deal with multiple distributional shifts? What happens if the descriptions for some of the tangrams change? Would you propose creating a second $A'$ and $S'$? If so, how would they interact with $A$ and $S$?
- sec 5.1: What is the "few shot classifier"? If it is a multi-class classifier, then how can it be compared to CoLLIE, which is learning a fundamentally different task, namely a mapping between embedding spaces?
- sec 5.1: Shouldn't the CoLLIE (without scaling) be trained with negative examples in order to give a fair comparison? More exactly, the paper mentions an ablation where the classifier $S(.)$ is removed from the system, so the system only consists of the transformation $A(.)$. Figure 5 is claimed to show that $S(.)$ is necessary to prevent catastrophic forgetting. However, the training for the ablated system only consists of (pseudo-words, image) pairs. So the transformation $A$ does not have any negative examples, which $S$ would normally have access to. 
- sec 6: "stored training examples have a very small footprint": If I understand correctly, these training examples are not stored in memory during prediction, so is it very important that their footprint be small?

# Suggestions
- Sec 1: "Unless the model is re-trained entirely from scratch (which is infeasible for large models like CLIP), the updates to the model’s parameters might interfere with previously learned knowledge, resulting in abrupt performance drops."
  - As I understand, this reasoning explains why transfer-learning or fine-tuning on CLIP is undesirable. However, is there reason to believe that catastrophic forgetting will occur for the settings considered in this paper (nonce-words, tan-gram descriptions)? The tasks don't seem too dissimilar from what CLIP sees ordinarily during training. I think it would strengthen your case if Figure 5 or Table 1 had results that show how forgetting occurs when the original embedding model is fine-tuned. I understand that CLIP isn't available for fine-tuning right now, but maybe a conceptually similar multi-modal transformer could be used.
- Sec 3: Could you report the mean squared error between the transformed vectors $T(t)$ and the target vectors?
  - I ask because it's possible that the similarity ranking for a given embedding may be high relative to the other candidates, but could still be a large distance away from the target.
  - In that vein, do you have a sense of what images your transformed embeddings correspond to? Can images be decoded for them?
- Sec 5.2: It is claimed that the results in figure 6 demonstrate that the model can leverage compositional knowledge to generalize. Given that results are averaged over 3000 runs, I am inclined to believe this claim. But it could also be shown explicitly by creating a held-out test-set of unseen color+shape combinations. This would increase confidence in the claim that the model is exhibits some type of systematic generalization.

# citations
[1] Bommasani, Rishi, et al. "On the opportunities and risks of foundation models." arXiv preprint arXiv:2108.07258 (2021).
[2] Magassouba, Aly, Komei Sugiura, and Hisashi Kawai. "CrossMap Transformer: A Crossmodal Masked Path Transformer Using Double Back-Translation for Vision-and-Language Navigation." arXiv preprint arXiv:2103.00852 (2021).

summary_of_the_review: Overall, the paper is well-written and outlines good directions for future work. The motivations are described clearly and thoughtfully. However, as is, I do not think it represents a significant enough contribution. The approach described is already present in most systems that use a large pre-trained model to provide embeddings for the input. That is, it's common for most existing approaches to take an off-the-shelf model, freeze the weights, and then use the produced embeddings for some down-stream task (see for example [2]). Finally, I'm not sure if the proposed solution can properly be called "continual learning" since it's unclear how it should scale with the addition of new tasks.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a simple new method for continuous learning of language-image embeddings. The proposed method, CoLLIE, employs a Transformation module designed to work only on the samples undergoing few-shot learning. It consists of a scaling function that determines whether the text is close to the sample undergoing few-shot training and an adjustment function that estimates the difference between the embedded representation of the paired text and the embedded representation of the image. The experimental results show that the proposed method can deal with new categories by few-shot learning while retaining the zero-shot learning performance of existing embedded representations (CLIP).
main_review: This paper is unique in its problem setting because it combines continuous learning with language and image embedded representations. Although the proposed method is simple, the experimental results show its ability to deal with new categories while maintaining the original performance.

On the other hand, this paper adopts only CLIP as an existing representation learning method. Although CLIP is a modern and excellent representation learning model, the authors should have conducted similar experiments with other representation learning to show that the superiority of the proposed method is a general contribution to existing representation learning methods. Otherwise, the advantage of the proposed method may exist only when applied to CLIP.

In addition, the proposed method is in the form of a product of an adjustment function that performs a linear transformation and a scaling function that outputs [0,1], which is added to the original representation. It is just a special form of the combination of Gated Linear Units (GLU) (Dauphin et al., 2017) and residual connections. The authors state that the proposed method is simple, but they do not mention that they propose the same as GLU.

Furthermore, the adjustment function and the scaling function are trained with different regression methods. However, there is no explanation why the scaling function, in particular, was implemented with support vector regression. In linear support vector regression, the output of [0,1] as defined by the authors as S(t) cannot be satisfied, and negative values and values larger than one may be output. As described above, the Transformation module is regarded as a combination of the GLU and the residual connection. Similar effects may have been obtained when the module is trained so that the vector representation of the text in the negative sample is output as it is and that the vector representation of the text in the positive sample matches the vector representation of the image. It is unclear why ad hoc learning as proposed is necessary.

Finally, as the authors also state, the rationale for learning the 1000 most common nouns as negative examples is questionable. It may have worked in this case, but what would happen if these nouns were also included in the Few-Shot learning text?

The following are minor concerns.
- The proposed method's behavior would have been better understood if the output of the scaling function alone had been reported, along with Figure 5, for the new pseudo-words and the existing words.
- The authors claim that the experimental results in section 5.2 show the hypothesis that the proposed method can learn blue giraffes better if it learns red giraffes. However, section 5.2 only shows that MRR as a macroscopic retrieval accuracy has improved. Other experiments and evaluations are needed to show whether this hypothesis truly holds. For example, we could train the few-shot training with only red giraffes and then evaluate whether the blue giraffes can be retrieved in zero-shot.
- Finally, CoLLIE is only described in this abbreviation, but its official name is unknown.
summary_of_the_review: ### Original Summary
This paper proposes a simple and effective method for a unique problem setting. On the other hand, the novelty of the method is much less than the authors may think, and a similar method is not cited. The rationality of the design of the proposed method is also questionable, and its versatility has not been experimentally demonstrated. The reviewer recommends that the authors expand the experiments and resubmit the revised paper to another international conference.

### Final Rating
The reviewer has read the other reviews, the responses from the authors, and the revised manuscript. The first score (3) has been upgraded a little to (5), but the reviewer still leans to reject the paper. The reasons are described in the last post of this thread.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper target on continual learning for the language-vision ground task with an additional small network. The network is attached on a CLIP model and it contains 2 linear layers to predict a difference vector of the original text embeddings and scaling value. The network are optimized by support vector regression (SVR). The method is test on pseudo-word and  tangram figures tasks. The result shows the model with proposed method can achieve better performance of original CLIP model. 

main_review: Strength: the experiment shows that a small network and a simple difference vector are enough to achieve similar performance to the designed few-shot learner.   

Weakness: This paper tries to use a small network to achieve continued learning. The main idea to achieve it is to predict a difference vector between exit-word and pseudo-word. This idea is somewhat similar to the idea of GLoVe, which also has a similar difference vector between multiple word pairs. From the experiment, the idea works for the 2 test cases. However, the test data is constructed (like “the red monitor" and “the blue monitor") and thus the reference image-language pairs can easily be found. I wonder that if the text queries are more complex, like a long sentence or a paragraph, will the idea still work. Usually, in a real case, a more complex text is more common than well-defined constructed phrases or words.

In the experiment, the author mentions a few-shot learner, which is not fully described in the paper. How does the author designs the few-shot learner may also influence the performance? Few-shot learning is now a popular topic and many papers are proposed. I think more experiments for the different few-shot methods on the test dataset will be more convenient.


summary_of_the_review: I think the idea of the difference vector is similar to some previous work. For the experiment, the data is somewhat constructed and simple for the method to find the relative object or description. However, It does not show that if the method can also be applied to the more complex case, which may be more common in the real life. And the comparison is not so sufficient. The author may compare with more few-shot methods to support their contributions. 

In general, I think the paper still needs to be improved and I recommend rejecting it this time.
review_title: null

Meta-review: This manuscript presents a method built on top of CLIP which transforms language embeddings to understand new phrases while maintaining the original abilities of CLIP.

1. The technical novelty of this work is limited. That being said, if the task is of wide interest, a straightforward approach that performs well is not just good, it's even preferable to one that is complex. The authors bring up the fact that reviewers are asked to rate both technical and empirical novelty. This is true. Yet, reviewers were unconvinced both by the method and the setting.

  The manuscript does not explain why this setting provides additional challenges or value compared to the many other continual learning or zero-shot settings that exist in the literature. What the authors say "we are not aware of any previous work that learns a direct transformation of the representations themselves to accomplish both continual learning and retained zero-shot use of those representations with the same model" seems to be undisputed by the reviewers. But is it critically important to future ML research that a single model does both? Or that a model learns a direct transformation to do so? Overall, this task seems very constrained and tailored to this one approach, while usually the more general a setting is, the more it is valued by the community because it will be more likely to stand the test of time and lead to new advances. Reviewers also could not point to a compelling immediate practical need for such a model, which would be another reason for considering a novel setting.

  While in the responses the authors acknowledge that they do not consider their method to be the ultimate solution, that the method has significant limitations, and that this really should be considered a strong baseline, this is not how the work is presented. Relatively little is said in the manuscript about any of these topics.

2. In response to requests for experiments (such as exploring the space of transformations and exploring alternate models to CLIP) the authors put forward that space limitations preclude such experiments. I would encourage authors not to rely on this argument going forward as it does not serve their cause well. Between the unlimited appendix and the possibility of linking to an anonymized website space cannot be a constraint. Science is complex these days and it's not unusual to have to report extensive additional experiments outside the main body of the manuscript. I encourage authors to consider that these requests by multiple reviewers are likely going to be the first questions that the readers of their work will also want answers to. Exploring other transformations and models is critical to understanding the value and impact of the work.

Minor point: I did not see this in the reviews but Figure 1(a) has the labels for CLIP text and image flipped.

If the authors round out the experiments and demonstrate either that their idea is more general-purpose, i.e., that it can be applied to other settings and problems, or that this setting is of great value on its own, this could be a strong contribution.
Decision: Reject