Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors propose a new continual video-language modeling (CVLM) setting, where models are supposed to be sequentially trained on five widely-used video-text datasets.
The authors propose the BMU-MoCo, a cross-modal MoCo-based model with a novel bidirectional momentum update (BMU) strategy.
To further boost our BMU-MoCo, a pair of global momentum encoders are maintained by the same BMU strategy.
The proposed method is tested on the proposed benchmark, and shows promising results. 
strengths_and_weaknesses: Strength:
On the newly promised benchmark CVLM, the authors implemented several previous well-known baselines. This is a good effort. 

Weakness:
The proposed method is only tested on the proposed benchmark. The authors may want to first make the proposed benchmark solid, or to show effectiveness of the proposed method on other continual learning benchmarks.
questions: 1. Equation (16) between Line 185 and 186: how is this contrastive loss implemented? Typically, such cross-entropy loss is implemented with log-softmax function to stablize the computation. However, in Equation (16), it seems not straightforward to use log-softmax to compute this loss. If computed by softmax and then log, is it stable?

2. LIne 290, "even outperforms ER-ring using 10% memory (about 200GB under our CVLM setting)". I'm curious that how this baseline that requires 200GB memory was implemented. Could the authors provide more details on the implementations of all these baselines. From my perspective, it would be a better contribution if the authors can make the CVLM benchmark and its baselines solid, compared with the newly proposed method. 
limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper incorporates the video-language modeling into continual learning. To overcome the challenges in continual learning, this paper proposes a novel cross-modal MoCo-based model with bidirectional momentum update (BMU).
strengths_and_weaknesses: strengths: 
1) This paper provides the a new continual video-language modeling setting.
2) The extensive experiments demonstrate the effectiveness of the proposed method.

weaknesses: 
1) This paper focuses on the video-language modeling in continual learning, however, the experiments only conducted on the text-to-video retrieval task. Since the proposed framework is two-stream architecture and is suitable for adjusting to the other video-language tasks, the experiments could be further investigated into multi-task transfer learning[1] in VLM domain, instead of on the single task.
2) The setting of two momentum encoder branches(local and global) is too complicated and the gains from global branch in table1 seems not significant.

[1] A deep hierarchical approach to lifelong learning in Minecraft, AAAI17
questions: Could the author conducted more experiments on the multi-task transfer learning part?
limitations: yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors introduce a new setting of the Video-Language Modeling task in a continual learning scenario. They propose an evaluation schema based on text2video (cross-modal) retrieval with 5 different tasks which correspond to separate popular video-language modeling datasets. Moreover, the authors propose their own method, namely BMU-MoCo, based on MoCo self-supervised representation learning approach, and compare it against other continual learning methods. The proposed method gives the best results given the evaluation metrics that measure the performance degradation between consecutive tasks, particularly ‘Forgetting rate’ and ‘Harmonic Mean’.
strengths_and_weaknesses: Strengths:
1.	The Paper is well-structured, and the contribution is clearly stated and supported in the experimental section.
2.	The authors propose a new approach to evaluate the effectiveness of VLM models in a continual learning setup with new metrics to evaluate the task.
3.	The proposed method gives the best results among other continual learning-based approaches.

Weaknesses:
1.	The method section is a bit messy and becomes hard to follow.
2.	Notations in 3. are definitely over complex, and could be simplified, e.g. by dropping ‘i’ in Eq. 1-6.
3.	The method is not entirely free from additional resources contrary to what the authors claim, since it requires an additional model (momentum encoder- local + global) to be trained and stored, as well as the queue for the training.
4.	Some typos and minor editing issues.
5.	The conclusions on the m hyperparameter are unclear.
6.	Both figures 1 are non-informative and confuse the reader.
7.	Lack of clear motivation why CVML among other cross-modal tasks and evaluation only on one of them seems to be quite limited.

questions: In Figure 1a) what do colors represent? What are the current models and the final model?

Could you explain in detail what happens when the queue size Nq is larger than Nb?

What does FR for Task 1 indicate? Is there a Task 0 then? The comparison is not clear.

Is 0.5 additional GB of memory for BMU-MoCo local only or both?

How are frames sampled and fed to ViT and how is the averaging over the whole video being performed?

limitations: The authors very briefly mention the limitations of their work, with no discussion of potentially negative impact. In my opinion, the limitations of this work are two-fold. First, as the authors mention, they only tackle the CVML task, however, to fully address this task, the results of state-of-the-art approaches on particular datasets should also be included, showing that they indeed struggle with catastrophic forgetting. Otherwise, it would be beneficial to address other cross-modal tasks.
review_title: null

Meta-review: metareview: This work presents a study on continual video-language modeling. In addition to the modeling side of things (BMU-MoCO), the authors construct a new benchmark in which they compare a number of existing methods. While I think it's great that the authors came up with a new benchmark, it's always a somewhat difficult analysis when a paper comes up with both a new benchmark and a method that beats the previous methods on this new benchmark. This is a shared concern with at least one of the reviewers. I do note that computational limitations make it difficult for the authors to thoroughly test on many other benchmarks. The authors do provide some UCL results in the rebuttal, which strengthen their case. 

All in all, I have to agree with reviewer ZHhB that the method is somewhat complicated and that the gains from the global branch seem overstated. I found the methods part hard to follow as well. All in all, the work does seem interesting and important, but perhaps I am more convinced about the benchmark rather than the method. As is, I would still recommend it for acceptance, but I feel it would need a good amount of work to improve clarity of the exposition and ideally more robust empirical evidence (that doesn't involve benchmarks created by the authors themselves).


confidence: Less certain
Decision: Accept