Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes CLiMB, a continual learning benchmark on vision-and-language tasks. CLiMB includes several CL algorithms and a modified ViLT model. It is presented to benefit for CL in multimodal tasks with deployment to multi- and unimodal tasks. The paper is well-motivated and the evaluation metrics are well-clarified.
strengths: 1. To the best of my knowledge, this paper is the first CL benchmark on multimodal tasks. It can facilitate research on multimodal task adaptation and mitigating "catastrophic forgetting".
2. The paper is well-written, and tasks and evaluation metrics are clarified.
3 The experiments provide insightful results, that is existing CL strategies do not generalize well to sequences of multimodal tasks. Such results also demonstrate the necessacity of this benchmark.
weaknesses: As a benchmark, more experiments (not only ViLT-based model) are expected to be conducted.
The motivation for mitigating "catastrophic forgetting" in the multimodal scenario (between different multimodal tasks) is weak and need further discussion.
correctness: Yes.
clarity: This paper is well written but the copyright of images is expected to be clarified.
relation to prior work: To the best of my knowledge, this work properly discusses previous related contributions.
additional feedback: None.
review_title: Well-motivated benchmark for multimodal continual learning.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper identifies that the current CL benchmarks limit to the vision-only or language-only tasks, and there lacks a benchmark for vision-language models.  It proposes CLiMB, a benchmark that contains upstream vision-language pretraining and downstream multimodal tasks.  The paper compares and evaluates different CL approaches with ViLT model, and find that common CL methods do help mitigate catastrophic forgetting during multitask pretraining, but they are not complimentary.
strengths: - The paper identifies an interesting problem on the continual learning with vision-language pretraining, and constructs different data configuration settings to study this problem.
- A unified toolkit is built for studying the proposed task.
weaknesses: - The paper proposes a multimodal continual learning benchmark, evaluates the baseline approaches and identifies several issues with the current approaches.  However, it fails to explain why the identified problem is valuable for the community to solve: why do we need to tackle this problem specifically in the multimodal settings?  If it is specific to multimodal data (but not to unimodal data), the authors need to explain the status quo for the unimodal data.  If not, the authors need to analyze why it is better to tackle the problem in multimodal settings instead of unimodal settings.
    - The paper calls for future work exploring cross-task knowledge transfer for vision-language models (L11-13), without much of the evidence why this line of work is promising.  For example, is there any line of work in vision-only / language-only showing that different tasks in continual learning are complimentary.  If so, this would make the argument stronger; and if not, the authors need to explain why this would be promising in vision language — after combing the vision and language modality.
    - The author explains the performance drop of the VCR dataset with the large distribution shift between upstream and downstream tasks.  This seems to be a general problem with the continual learning.  Similarly, is there any evidence in any modality suggesting that this is solvable, or is there any arguments on why this can be easier with multimodal data?
correctness: The experiments design are appropriate and are performed correctly.
clarity: The paper is well written and is easy to follow.
relation to prior work: The author has discussed the related works.
additional feedback: See weakness section above.  I would be willing to increase the rating if the above concerns are addressed.
review_title: It identifies several problems in the multimodal continue learning setting, but needs more justifications on its significance.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper starts off with a simple preposition: it would serve as a benchmark to gauge the effectiveness of knowledge transfer and mitigated forgetfulness in cross-task modeling using both multi-modal and unimodal frameworks. They abstract these tasks into two phases: phase 1, upstream continual learning tasks, which considers the degree of forgetfulness and knowledge transfer across language and vision tasks trained sequentially, and phase 2, downstream continual learning tasks, which considers performance of low shot transfer capability of the models used in the prior phase.

Through the benchmark, the authors conclude that in the upstream phase, CL models do not lead to knowledge transfer AND that while some tasks mitigate forgetting, model performance pails in comparison to a full tuning of a model to a specific task. There are a couple interesting caveats to that, though: using Adapters is a strategy that the authors found outperforms other methods for forgetting mitigation and VCR is one task that greatly degrades performance on forgetting and knowledge transfer. They are able to show that current CL models are not able to appropriately generalize to tasks or be particularly performant for downstream low shot tasks. 



strengths: The authors do a great job at defending their thesis. I was pleasantly surprised by a number of the findings, including the relatively higher performance of the Adapter method in mitigating forgetting over the Freezing Bottom 9 Layers method. This is definitely a finding that will get a lot of attention and conversations started on really digging into why this could be. As such, the paper lays a great foundation to future research into the use of Adapters as a preferred method to improve CL tasks. 
weaknesses: There are no obvious weaknesses so I will simply share some of my "Would Have Been Nice Haves".
They go down particular strategies and the performance of said strategies on downstream task performance showing that all methods lead to a negative knowledge transfer capability except one: Adapters. This is where my interest get particularly peaked. Adapters not only are more performant than other forgetful mitigation strategies explored in this paper, but they also perform comparably to full model tuning. As well, there is a moment in the paper where the authors contend that model forgetfulness is also seemingly related to the sequence by which the models are being trained on each task. They show that one model in particular, VCR, leads to the most extreme level of forgetting and negatively affects knowledge transfer. I wish the authors would have spent more time on that. They only spend 1 sentence speculating on why this may be and I was left wanting more. Same for the Adapters observation, they quickly overlook the higher performance in the breakdown of the paper's results. 
This brings me to another thing, ViLT-BERT. The authors do not discuss ViLT-BERT specifically until page 6. Before this, they focus the crux of their analysis around downstream task performance on how well the CL models can generalize to ViLT. This is fine, but I wish the authors did not bury the lede there. ViLT-BERT outperforms the low shot accuracy of ViLT which is extremely significant because it shows what the authors correctly point out in the paper: strong priors assist with downstream low shot performance. Considering the lengths the authors went into abstracting CL tasks into two phases -upstream and downstream- it would have been nice to see them speculate the relationship between upstream task performance and downstream task accuracy on both ViLT and ViLT-BERT keeping the idea of strong priors in mind throughout more of the paper. 
correctness: Yes
clarity: Yes
relation to prior work: The authors do a good job at presenting prior contributions while showing a need for the addition of their work to present research. 
additional feedback: No additional feedback.
review_title: CLiMB Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper combines existing datasets (five language and vision datasets, five vision datasets, and five language datasets) to create a continual learning benchmark.  The paper then evaluates a standard language vision transformer model and six continual learning algorithms (including simple algorithms like sequential fine tuning and more complex algorithms such as including task adapters).  The paper then analyzes whether knowledge transfers between language and vision tasks, whether tasks are forgotten when updating models for new tasks, and whether models exhibit transfer to language and vision only tasks.

The main contributions are: continual learning benchmark for multimodal tasks and analysis on a variety of continual learning algorithms in the multimodal setting.

strengths: Clarity: This paper was clear and easy to follow for me.

Quality: Experiments were well done; the paper includes a variety of continual learning methods and presents a variety of different analyses.  The results lead to some interesting insights as well.  For example, L232 mentions that experiments indicate that current language and vision models do not learn complex language reasoning skills.  This makes sense given the data they are trained on, but it is nice to see quantitatively.  I appreciate that the paper includes many different continual learning algorithms.  

Originality:  Continual learning is well studied in machine learning.  The originality in this paper comes from crafting a multimodal continual learning benchmark and providing analysis on common continual learning algorithms in a new domain.

Significance:  Continual learning is an important research area and creating a benchmark for multimodal continual learning will likely spur interesting research in this direction.

weaknesses: Clarity: One small thing I did not understand is how to read Figure 3b; it was not clear to me which lines correspond to which task order and how to draw conclusions from the chart.

Quality: The paper could be stronger by applying continual learning algorithms to more than one language and vision model (ViLT is used in the paper).  However, I do not think this is necessary for acceptance.

Originality: As one of the main contributions of this paper is presenting a continual learning multimodal task set up, I would have liked to have seen more discussion on why the task set up is a good one, perhaps by motivating with a “real world” example.  Whereas in the continual learning setup I am the most familiar with (adding new class labels sequentially) has an intuitive motivation that a model might come across new classes continuously, I was less sure of the motivation for adding new tasks sequentially.  Furthermore, though the author’s mention that unimodal tasks can benefit from multimodal tasks, the reverse can also be true (e.g,. many of our models include pretrained detectors).  Another way to set up multimodal continual learning would be to see if learning additional concepts in vision only or language only tasks transfers to tasks like VQA or NLVR (similar to what was done in some older work on “novel object captioning” see [here](https://arxiv.org/abs/1511.05284)).   I would like to see more explanation motivating the particular setup chosen in this paper.

Significance:  Though the paper is likely to spur more interest in multimodal continual learning, I am not completely sold on what is measured (e.g., upstream to downstream task transfer) and would be a bit concerned that others might adopt the dataset without questioning decisions made in this paper.

correctness: As far as I can tell, the paper is correct.

clarity: Good.
relation to prior work: Good – authors mention papers that show multimodal work can benefit unimodal tasks; another paper that could be cited is the [vokenization paper](https://arxiv.org/pdf/2010.06775.pdf).
additional feedback: n/a
review_title: CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: The paper describes a benchmark to assess catastrophic forgetting of various CL methods on multi-modal data sets. It also studies how knowledge transfer is present when shifting to unimodal problems or to low-shot tasks.
The discoveries are very important and show how CL is still an unsolved problem. It's also interesting to see that adapters can mitigate forgetting while freezing the encoder part.

strengths: - The paper tackles understudied problems, namely multi-modal CL and knowledge transfer from bi-modal to uni-modal continual learning.
- The discoveries are of high importance to the field.

weaknesses: The paper proposes an important benchmark to help researchers perform CL on bi- and uni-modal problems. However, this benchmark is poorly documented. I could identify these drawbacks:
- The implemented code misses any type of documentation: missing Docstrings, typing.
- The only thing that explains the benchmark is a readme file in the repository. There should be an illustrative tutorial showing how to perform and evaluate each setting.
- There should be a tutorial that explains how the different classes can be instantiated and used.
- There is no class or function diagram to show how the whole framework is constructed.
- There is no explanation of how the framework can be extended using a new CL method.
- There is no explanation of how to incorporate a new multi- or uni-modal data set.
- No maintenance plan.
- To be useful, the benchmark should facilitate a clear and easy way of creating reports that consist of plots and tables.



correctness: The paper and the benchmark are limited to very few measures presented in the paper. The authors should take into account the measures proposed in Chaudhryet al.
Chaudhry, Arslan, et al. "Riemannian walk for incremental learning: Understanding forgetting and intransigence." Proceedings of the European Conference on Computer Vision (ECCV). 2018.

clarity: The details of the proposed performance measures are put in Table 2, which is fine. However, they should be explained better. Is random selection the same as a random classifier?

relation to prior work: Missing relation to former CL benchmarks.

additional feedback: - At the end of Page 3, why do you compare with the performance of S_{PT}^{LS(i)}? Shouldn't the comparison be against the task T^i_M itself?
review_title: Great insights into CL challenges but a poorly documented benchmark

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: A benchmark to study continually learning in multimodal tasks.
They show that CL strategies do not generalize well when trained on a sequence of multimodal tasks, or enable few-shot transfer. 
Designed to be extensible for new tasks, architectures, and CL algorithms.

strengths: “Continually learning” or adaptation is an important and timely challenge, especially in the rise of more tasks and models. 
Extensive evaluation of tasks (multimodal, visual, textual) and CL algorithms.
Motivation new research of continual learning strategies for vision-language tasks. 
An extensible framework. 

weaknesses: The main weakness is that they evaluate only one model (ViLT), and the insights could be different when using other models. 
Another weakness is pointing only to the problem without hinting at solutions. 
I do think that this is a good paper. However, dealing with these 2 weaknesses can make it great. 

There could also be a better evaluation of the order of the tasks, but I do understand that doing so is limited since there are many options and permutations of tasks. 

correctness: See the main weakness from the previous comment: they evaluate only one model (ViLT), and the insights could be different when using other models. 
clarity: The paper is well written. I could use a few more examples to better understand the formulations in S3.1, perhaps in the Appendix. 
Additionally I found some of the plots difficult to understand with the formulations, perhaps it will be helpful to explain a single point (e.g., Fig 4)

relation to prior work: Not enough. They call the subject “continual learning”, but literature also refers to it as “online learning” or “domain adaptation”, and there are many more previous works and strategies to consider.
additional feedback: I hope that the benchmark can indeed be easily extended. 
This is an important challenge. Identifying a problem is important, but I hope that your framework will pave a path to better solutions.
review_title: An important challenge - good paper 

Meta-review: metareview: Overall, the reviewers have expressed positive support for the submission. It proposes a continuous learning framework for multi-modal models. The reviewers' concerns were quickly addressed in the responses. The primary concern appears to be a weak motivation for the problem being tackled. The authors propose a stronger framing I am hoping to see in the final version of this paper. I am in support of accepting the paper.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept