Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper addresses continual learning in joint vision and language model. Specifically, the paper analyses the causes for task/dataset forgetting in a continuous training regime from a geometric point of view. The analysis works as follows:

1. First, CLIP is trained on COCO to obtain a baseline CLIP0 model. 
2. Next, Flicker 30K is divided into 5 subsets and the CLIP0 model is further trained sequentially on each of the subsets to obtain CLIP1-CLIP5 models. These models are also denoted as CLIP_ct models, where ct stands for continual training. 
3. In addition, the authors train CLIP on the entire COCO and Flicker 30K datasets to obtain CLIP_jt, where jt stands for joint training, which serves as an upper bound to CLIP1-5 performance. 
4. The authors measure the performance of the various models on Flicker and COCO test set, demonstrating a gradual degradation in performance of the CLIP_cp models on COCO tests set and an improvement on Flicker test set, which makes sense as the models are trained (gradually) on the Flicker dataset. They name this phenomenon "Cognitive Disorder". 
5. Next, the paper considers the angles between all pairs of visual representations from different models, i.e.: the angle between the visual representation extracted using CLIP_t for a sample i and the visual representation extracted using CLIP_(t+1) for the same sample i. This analysis is done on the test set and denoted as Self Angle relationship Matrix (SAM). This measures to what extent the representations are rotated during continual training, with respect to themselves. They find that those angles change slowly, therefore argue to not be the reason for the performance degradation. The change in SAM is denoted as Intra-modal rotation. 
6. The paper then turns to analyse how angles between pairs of visual embeddings from different samples change during continual learning. Specifically, they consider a matrix composed of the angles of representations from different samples i,j extracted from the same CLIP_t model. This matrix is denoted as Rotation Angle Matrix (RAM). This analysis is done for the textual embeddings as well.
7. The authors find that the RAM matrix changes more dramatically for visual embeddings than the angle between off-diagonal elements of the textual embeddings. The change in RAM is denoted as Inter-modal deviation. 
8. From this analysis, the paper argues that the inter-modal deviation (i.e.: the fact that the two modalities are not synchronised in the amount of rotation between off-diagonal elements in different training steps) is the cause for the degradation in performance during continual learning (which is denoted Cognitive Disorder as I explained above). 

Now, following this analysis, the paper proposes to alleviate "cognitive disorder" by aligning the amount of rotation of off-diagonal elements between the two modalities during training. This works as follows: first, given the trained model on sub dataset t , we consider the contrastive matrix M_t, i.e.: the matrix whose element in the i,j position is the similarity score of the i'th sample vision embedding and the j'th sample text embedding. The method add a KL divergence term between M_(t-1) to M_t to the loss. Samples (rows) for which the model in "timestamp" t-1 are not corrected classified are replaced in M_(t-1) by their values from M_t.

The paper presents two experiments to evaluate the performance of the proposed method. The first experiment follows the same setting as the analysis above and in the second experiment, the authors trained a CLIP0 model on COCO and Flickr datasets and gradually trained 8 additional models on 8 subsets of SBU-1M. The experiments demonstrate that using the proposed method, the resulting models only suffer from a small degradation of performance on the "baseline" dataset while performing well on the "new" gradually added sub-datasets, compared to the baseline CLIP0 model and even perform better on the "new" dataset than the baseline continual training model. 
 



strength_and_weaknesses: Strengths:
1. The paper is well written and easy to follow. 
2. The paper provides a thorough analysis on the causes for performance drop in vision and language models from an original point of view.
3. The proposed method performs very well in practice, as demonstrated in the experiments. 
4. The paper showcases a real problem, provides an analysis to its cause, proposed a solution and provides experiments supporting the claims made and demonstrating superior performance of the proposed solution. 


Weaknesses and questions:
1. First, I strongly recommend to change the terminology of "Cognitive Disorder". The term has little connection to the problem it describes, it's a bit far-fetched to borrow a term originally used to describe a mental health disorder and really adds nothing to the paper. A more technical term (dataset forgetting, degradation from distribution shift,...) would be much better.
2. The analysis on the causes of the performance degradation is done on the test set. However, the proposed solution is of course implemented during training. We obviously can't add a loss term during training and optimise it on the test set, but how much of this analysis holds when you consider the training set? I would expect the model to overfit to the training set, would there be inter-modal deviation between subset t and subset t-1 during training? 
3. In table 3 alpha is set to 20. An analysis of the effect of alpha and to what extent the method is robust to the choice of alpha would be interesting to see. Of course, as alpha is increased the effect of "alignment" between models is increased, but I think the paper should explain how alpha was selected to be 20 and how sensitive this value is (e.g. how dramatically the results would change if a different, less ideal value would be used). 
4. A drawback of the method is that it required to store the vision and language embeddings from model t-1 during training of model t (or re-evaluate them during step t). I'm not sure how much it would be applicable to training on huge scale datasets (e.g. LAION dataset). 
clarity,_quality,_novelty_and_reproducibility: The paper is clear, novel and the method is easy to reproduce. 
summary_of_the_review: Strong paper in my opinion. The paper identifies a continual learning performance degradation as drawback of joint vision and language embedding models. It demonstrates this drawback by training a baseline model on COCO, "finetuning" it on Flick and evaluating on both Flickr and COCO test set, indicating a clear degradation on COCO. The paper analyses the cause for it and finds it to be Intra modal deviation, i.e.: misalignment between the amount of rotation of diagonal elements during training between the two modalities and designs a solution via adding a term to the loss to encourage intra modal alignment. The paper provides experiments that demonstrate the affect of the solution in mitigating the drawback. I mentioned a few weaknesses and question I would appreciate if the authors could address in their rebuttal.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a new method based on the CLIP model for continual vision-language representation learning. The authors find that simple continual training in the CLIP model degenerates the performance on multimodal retrieval tasks, named Cognitive Disorder (CD). To tackle CD, they suggest Mod-X that selectively aligns the off-diagonal information distribution of the contrastive matrices utilizing the CLIP models from current and past timesteps.
strength_and_weaknesses: **strength**
- Introduce and analyze challenging issues for continual vision-language representation learning, named intra-modal rotation and inter-modal deviation
- outperform baselines in both image-text and text-image retrieval.

**weakness**
- Missing detailed and explicit descriptions for many notations (e.g., ct, jt, the subscript (task index) i, the superscript T in Section 3.2.3., the capital letter S in figure 6, etc
- The difference between Cognitive disorder and Catastrophic forgetting (or representational forgetting, referred to in Madaan et al. 2022) needs to be clarified. 
- Intra-modal rotation may happen to all non-iid task-based problems, including streaming learning and conventional continual learning, which is not new, and the following analyses are a bit straightforward in the continual learning field.
- Including Appendix 7.1 for the mathematical derivation, inter-modal deviation is not explicitly related to cognitive disorder (the performance degeneration of retrieval tasks). It tends to, but in a strict sense, the manuscript doesn't provide evidence that increasing inter-modal deviation in each sample is always harmful to "cognitive disorder".
- Only naive and old baseline (EWC) is used for validation, which is hard to evaluate how the proposed model achieves superior performance compared to recent strong methods for multi-modal continual learning for retrieval tasks.
clarity,_quality,_novelty_and_reproducibility: The manuscript is easy to read since the idea is very simple, but the overall writing, including the descriptions for notations, is sometimes unclear.
summary_of_the_review: The direction of the submission is auspicious and interesting, but explanations might be required to be more kind and include the details. Also, I recommend further comparing with competitive methods for multimodal continual learning methods for solving retrieval tasks.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper investigates continual learning of a multi-modal text-image model like CLIP. It investigates the angle shift of the learned representations throughout different learning sessions, and it also proposes a knowledge distillation type of loss to address the issue of catastrophic forgetting. Results showing that using knowledge distillation can significantly improve the CLIP continual learning. However, the paper’s model and analyses are disconnected and there is concern on whether the model is specific to CLIP or contrastive learning in general.
strength_and_weaknesses: The strength of the paper is that it shows an interesting phenomenon of angle shifting – the relative angle between a positive pair didn’t move as much as the absolute angle rotates. This suggests that the representation space is undergoing constant rotation during continual learning, which is an interesting insight. Also the experimental results clearly show that the proposed model is significantly better than baselines.

The main weakness of the paper is that it reads very disconnected. There is no intuition why knowledge distillation addresses angular shift, and neither are there results confirming that using knowledge distillation helps this issue by showing the angular shift before and after distillation. 

The proposed model seems generic to any contrastive learning algorithm, and the whole paper could have been benchmarked on image-only continual unsupervised learning (e.g. Madaan et al. 2021), instead of image-text learning. There is nothing specific to multi-modality here, and you could treat the two vectors coming from either an image-text pair or a pair of two views of the same image.

Moreover, the paper claims novelty on “cognitive alignment”, but it is basically a knowledge distillation loss (i.e. LwF Learning without Forgetting, Li & Hoiem 2016) applied directly in the context of contrastive learning (i.e. predicting image instance labels instead class labels). Since no gradient is backproped into the old model, the KL loss is equivalent to a cross entropy loss between old and new prediction. One minor difference is that the paper manually edits the old model’s prediction if the label is wrong. This seems an ad hoc operation and there is no empirical evidence presented in the paper showing that it is necessary. In summary, the paper may run into a concern of over-claiming, as the proposed method is LwF/knowledge distillation re-purposed in the context of contrastive learning, but it is instead branded with a bunch of “cognitive” terms (“cognitive disorder”, “cognitive deviation”, “cognitive align”) with no real cognitive science intuitions or references.

Madaan et al. Representational continuity for unsupervised continual learning. ICLR 2022.
Li & Hoiem. Learning without Forgetting. ECCV 2016.
clarity,_quality,_novelty_and_reproducibility: A bunch of “cognitive” terms aside, the paper is clear to read. The novelty is not as strong as it might seem due to the similarity of a knowledge distillation loss. I cannot judge on reproducibility.
summary_of_the_review: The introduction of a multi-modal continual learning benchmark is interesting, and the analyses are insightful, but unfortunately the model disconnects from the analyses. I suggest the authors show a direct link why the analyses inform such a model, and also evaluate whether it is possible to run on image-only continual learning benchmarks or there are things that are specific to image+text. Overall I think the paper is slightly below the bar as it is hard to find a clear hypothesis throughout.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies a continual learning problem based on the previous CLIP model. COCO and Flicker datasets are investigated to determine the representation variations when continually trained on one dataset and tested on the other.  They found that the retrieval performance on one dataset would drop significantly when continually trained. They correlate this phenomenon with representation variations. To alleviate this, they introduce a new training loss to contain the predictions between current and previous predictions. 
strength_and_weaknesses: Strength
1. The continual learning of CLIP model is challenging but very interesting.  
2. Authors correlate the degrading performance with the representation variations. 
3. The manuscript's structure is well-organized and easy to follow.

Weakness
1. The conclusion can not drawed from the provided observations and experiments in the paper. First, only two retrieval datasets are considered, the representation differences may vary in different datasets. Second, the correlations between the retrieval performance and representation variations are not evaluated.
2. For the proposed method, the data are different during different stages, but the training objective is modified on different training iterations. Why would the new loss alleviate the forgetting issue? 
3. The experiments are not sufficient to demonstrate the claims in the introduction. 
clarity,_quality,_novelty_and_reproducibility: Clarity. 
The paper clearly introduces the re-training issue of the CLIP model. The authors investigate what correlates with performance drop on the image-text retrieval task.

Quality.
The formwork is clear and well-organized. 

Novelty.
They investigate the proposed problem from the representations and record the difference across several continual learning stages, which is new. But the proposed method seems irrelevant to the findings.

Reproducibility.
Code is not provided. 

summary_of_the_review: Clarity. 
 The authors investigate the image-text retrieval continual learning problem based on the CLIP model. They provide several interesting findings and modify the training objectives. But the experimental results are not sufficient to support the claims of the present paper. Therefore, I tend to reject the current version.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper investigates the feasibility of training CLIP model continuously through streaming data and find the reason about cognitive disorder in continual CLIP training and proposes a new framework Mod-x to alleviate model's cognitive disorder. While the paper presents some interesting new ideas for continual learning in a new area, reviewers had some major concerns about the weak empirical studies, the lack of comparison with strong continual learning baselines (given this topic has been so much extensively studied), weak technical novelty,  etc. While the authors have attempted to address some review questions in their rebuttal, the manuscript requires a significant revision before it can be accepted in a top venue.
Decision: Reject