BMU-MoCo: Bidirectional Momentum Update for Continual Video-Language Modeling
Authors: Yizhao Gao
Nanyi Fei
Haoyu Lu
Zhiwu Lu
Hao Jiang
Yijie Li
Zhao Cao
Venue: NeurIPS 2022 Accept (2022)
Keywords: Video-language modeling
continual learning
catastrophic forgetting
representation learning

Abstract: Video-language models suffer from forgetting old/learned knowledge when trained with streaming data. In this work, we thus propose a continual video-language modeling (CVLM) setting, where models are supposed to be sequentially trained on five widely-used video-text datasets with different data distributions. Although most of existing continual learning methods have achieved great success by exploiting extra information (e.g., memory data of past tasks) or dynamically extended networks, they cause enormous resource consumption when transferred to our CVLM setting. To overcome the challenges (i.e., catastrophic forgetting and heavy resource consumption) in CVLM, we propose a novel cross-modal MoCo-based model with bidirectional momentum update (BMU), termed BMU-MoCo. Concretely, our BMU-MoCo has two core designs: (1) Different from the conventional MoCo, we apply the momentum update to not only momentum encoders but also encoders (i.e., bidirectional) at each training step, which enables the model to review the learned knowledge retained in the momentum encoders. (2) To further enhance our BMU-MoCo by utilizing earlier knowledge, we additionally maintain a pair of global momentum encoders (only initialized at the very beginning) with the same BMU strategy. Extensive results show that our BMU-MoCo remarkably outperforms recent competitors w.r.t. video-text retrieval performance and forgetting rate, even without using any extra data or dynamic networks.

BMU-MoCo: Bidirectional Momentum Update for Continual Video-Language Modeling
Yizhao Ga Nanyi Feil.2 Haoyu Lu Zhiwu 12, Hao Jiang? Yijie Li Zhao Cao
Gaollng School of Artificial Intelligence, Renmin University of China, Beijing, China Beijing Key Laboratory of Big Data Management and Analysis Methods Huawei Poisson Lab, Hangzhou, Zhejiang, China {gaoyizhao, luzhimu)@ruc.ede edu.c..ddcccm.
Abstract
Video-language models suffer from forgetting old/leamed knowledge when trained with streaming data. this work, we thus propose a continual video-language mod- eling (CVLM) setting, where models are supposed to be sequentially trained on five widely-used video-text datasets with different data distributions. Although most existing continual learning methods have achieved great success by exploiting extra information (e-g., memory data of past tasks) or dynamically extended networks, they cause enormous resource consumption when transferred to CVLM setting. To overcome the challenges (i.e., catastrophic forgetting and heavy resource con- sumption) in CVLM, we propose novel cross-modal MoCo-based model with bidirectional momentum update (BMU), termed BMU-MoCo. Concretely, our BMU-MoCo has two core designs: (1) Different from the conventional MoCo, we apply the momentum update to not only momentum encoders but also encoders (i.e.. bidirectional) each training step, which enables the model to review the learned knowledge retained in the momentum encoders. (2) To further enhance our BMU-MoCo by utilizing earlier knowledge, we additionally maintain a pair of global momentum encoders (only initialized at the very beginning) with the same BMU strategy. Extensive results show that our BMU-MoCo remarkably outper- forms recent competitors w.r.t. video-text retrieval performance and forgetting rate, even without using any extra data or dynamic networks.
Introduction
Existing video-language modeling (VLM) methods have achieved promising performance for video- text retrieval [59, 39, 61, 30, 24, 53, 28, 4] with non-streaming data. However, in real-world application scenarios, VLM models need to evolve with streaming data (e.g., collected from the Internet [39, 42]) to accommodate more tasks. Under this setting, since it costs too much resource to retrain the model with both old and new data for each task, a common practice to fine-tune VLM models with only the newly- wwwiied data. Note that such model fine-tunine leads to severe performance degradation on previous tasks. This is a well-documented phenomenon called catastrophic forgetting [18, 38] under the conventional continual learning setting [48, 45, 34, 16, 23, 60].
Therefore, in this work, we propose a continual languaa modeling (CVLM) setting to better simulate the realistic scenario. Under our CVLM setting, models are supposed to be sequentially trained on five widely-used video-text datasets: VATEX [54], ActivityNet [25], MSR- -TTT [55]. DiDeMo [20], and MSVD [10]. An evaluation protocol is also established for CVLM, which contains three metrics to respectively measure the text-to-video retrieval performance (Recall @ 1, shortened as
'The corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

current models
final model
60
extra memory
localgglobal momestum encoders fve
R@1 (%)
40
Task t-1
Task
update frx
update
20
BMU- MoCo
BMU- MoCo
encoders f
0
Task 1
Task
Task 3 (a)
Task
dyzamic network
(b)
Figure 1: Illustration of the catastrophic forgetting problem in CVLM and the core design for our BMU-MoCo. (a) The catastrophic forgetting problem in CVLM. We train a basic cross-modal MoCo model on five tasks and present the comparative results of the final model and current models on learned tasks (Task 1-4). Note that there is no catastrophic forgetting on Task 5 and thus this task is omitted here. (b) The core design of BMU-MoCo. Different from the conventional MoCo, update not only momentum encoders but also encoders through the bidirectional momentum update (BMU) strategy without extra memory dynamic network across all tasks.
@ 1), forgetting rate (FR), and harmonic mean (HM) performance (see more details of the evaluation protocol in Sec. 1). Moreover, we implement a basic cross-modas MoCo [19] model (Base-MoCo) our baseline method since it has shown superior performance on video- aado-language modeling [33, 35]. As illustrated in Figure (a)), we can observe that in spite of achieving great R @ results with current models (evaluated right after trained on each task), the performance of the final Base-MoCo model (trained across all five tasks) drops significantly (i.e., catastrophic forgetting).
To tackle the catastrophic forgetting problem, most recent continual learning works attempt to preserve the learned knowledge from a variety of perspectives: (1) Maintaining a memory buffer to save and exploit data from previous tasks [46, 5, 34, 3, 8, 45]; (2) Generating pseudo data of learned tasks [51, 27, 43, 57]; (3) Extending the network architecture dynamically as each new task arrives [48, 16, 1, 7, 11. However, when these methods are transferred to our CVLM setting, the resource consumption is enlarged rapidly as the number of tasks grows, due to the characteristic of video data. In addition, another branch of continual learning works focus imposing a regularization constraint with quadratic penalty [23, 60, 14,45 49] or knowledge distillation [32, 2, 21, 44, 22], which leads to an unwanted rofe on the performance of old and new tasks with limited neural resources [40]. Therefore, it is a long-standing and arduous challenge to train a video-language modeling network under the CVLM setting with both effectiveness and efficiency taken into consideration.
To overcome this challenge under the CVLM setting, we devise BMU-MoCo, a cross-modal MoCo- based model with novel bidirectional momentum update (BMU) strategy. As shown in Figure 1(b), our BMU-MoCo needs neither extra memory data nor dynamically extended neural networks, which is quite different from previous continual learning methods. Concretely, similar to cross-modal MoCo applied in [33, 35], our proposed BMU-MoCo has a video encoder (i.e., ViT-Base [13]) and a text encoder (i.e., BERT- Base [12)), followed by the momentum video/text encoders. Different from the original MoCo [19] and its cross-modal versions [15, 33, 35] that utilize momentum update for only momentum encoders
maintain large consistent queue, our BMU strategy imposes momentum update on both momentum encoders and (video/text) encoders. As a result, at each training step, the encoders of our BMU-MoCo learn the new knowledge by end-to-end update with pack-propagttion whilst reviewing the old knowledge directly from the parameters of momentum encoders by momentum update. In our opinion, our BMU-MoCo outperforms existing methods for two main reasons: (1) Momentum encoders are initialized by current encoders at the beginning of each new task and then progress slowly, which helps our model preserve adequate old knowledge without sacrificing the performance on new tasks; (2) Since there is no category information under the CVLM setting, learning from memory data or distilling with a batch of new data only absorbs part of previous knowledge while our BMU-MoCo learns holistic old knowledge directly from the parameters of momentum encoders. To further enhance our proposed BMU-MoCo, we also maintain pair of global (cross-task) momentum encoders with the same BMU strategy, which are only initialized at the very beginning of the whole training process and thus preserve earlier knowledge than the normal local (task-specific) momentum encoders.

---

The main contributions of this paper are four-fold: (1) We propose a new continual video-language modeling (CVLM) setting, where models are supposed to be sequentially trained on five widely-used video-text datasets. (2) To effectively and efficiently overcome the catastrophic forgetting problem under the CVL.M setting, we devise BMU-MoCo, a cross-modal MoCo-based model with a novel bidirectional momentum update (BMU) strategy. For the first time, our BMU strategy can review holistic old knowledge directly from the parameters of momentum encoders while learning on new tasks, without extra memory or dynamic network across all tasks. (3) To bring further improvements to our proposed BMU-MoCo, a pair of global momentum encoders are maintained by the same BMU strategy to preserve and review earlier knowledge under the CVLM setting. (4) Extensive experimental results demonstrate that our BMU-MoCo outperforms recent continual learning methods by large margins w.r.t. both text-to- video retrieval performance and forgetting rate, even without any extra memory data or dynamically extended networks.
2 Related Work
Video-Language Modeling. Most existing methods for video-language modeling follow two paradigms: (1) Single-stream methods [36, 61, 52, 29, 28, 56] typically include multi-modal trans- former to achieve fine-grained cross-modal interaction between the video and language modalities. Although achieving great performance, they suffer from the huge time complexity caused by the pair- wise inputs during inference, which makes them unsuitable for practical applications. (2) Two-stream methods [17, 41, 4, 33, 35J learn video and text representations independently, and align them after encoding. To ensure the inference efficiency, both the baseline method (i.e., Base-MoCo) and our BMU-MoCo for CVLM
set to be two-stream methods. Importantly, different from Base-MoCo, our BMU-MoCo has a novel BMU strategy to address the catastrophic forgetting problem and two extra global momentum encoders to further boost the model performance.
Continual Learning. Conventional continual learning methods mainly focus on image classification tasks. They can be roughly categorized into three groups: (1) Rehearsal-basse methods apply extra memory to store sampled data [45, 34, 46, 9, 5, 3, 8, 6] or generate pseudo data [51, 27, 43, 57] from previous tasks. The memory size and the training complexity tend to be enlarged significantly as the number of tasks grows. (2) Expansion- -asod methods either add extra extended networks for new tasks [48, 16, 7, 31] or select partial model parameters to update for different tasks [58, 50J. They need more computational resources especially for a long sequence of training tasks (e-g., under CVLM setting). (3) Regularization-busel methods modify the model parameters with quadratic loss penalty [23, 60. 14, 49] or knowledge distillation constraints [32, 2, 21, 44, 22]. Although succeeded in image classification tasks, they still face large challenge in balancing the model performance between old and new tasks when applied to our CVLM setting. Although our BMU-MoCo can be classified as a regularization- based method, it has a vital difference from existing regularization-base methods: benefiting from the bidirectional momentum updating process, our BMU-MoCo can directly utilize the holistic previous knowledge from the parameters of momentum encoders for model training (i.e., updating the encoders), and simultaneously update the momentum encoders at each training step to accommodate new tasks.
3 Methodology
3.1 Preliminary
We propose new continual video-language modeling (CVLM) setting, where models are supposed to be sequentially trained on n video-text datasets D = [Di, D , Da]. For each task t, it contains dataset D,- [V. T2 N.-1 with No video-text pairs, where Vú2 denotes video with S, frames and Ti represents an English text description. The target of CVLM is to learn video encoder fey and a text encoder for. which can respectively project the input video and its related text description into a joint embedding space with nearest metric distance. Different from the classical VLM setting which only considers the model performance on the current dataset Dj, our CVLM setting requires the models to prevent the catastrophic forgetting on the previously-used datasets [D1, Dz. ... D.-1] (t > 1) while also performing well on the current dataset D,. Note that our proposed BMU-MoCo for CVLM is a memory-free method which only utilizes the current dataset Dr for each task i (without the need of reloading the other datasets). Therefore, without particular statement, we only consider task & with D, in the following subsections for formulation simplicity.
3

---

3.2 Network Architecture
Video Encoder. We follow the most recent video- language modeling works [28, 56, 33, 35] to learn video representation by fusing the image embeddings of sampled frames per video. Concretely, given each video V, with S; frames, we randomly sample frames (s S,) and embed them with an image encoder Sing (i.e., ViT-Base [13]) to obtain the frame embeddings:
F (x,), 1,2, ,,,
where oa denotes the j-th sampled frames of video V, and Fing denotes its image embedding encoded by Sang. Then we project F by Linear layer sproj:
prog = Spros (Fing) ).j = 1, ..
(2)
where proj E Ie denotes the projected d-dimensional image embedding of Fim. Following COTS [35] and HiT [33], we obtain the final video embedding of V; by adopting fusing layer faeg to aggregate the image embeddings (Pig) }:
FV Sang
(F proj> proj"
(3)
where favg denotes an Average Pooling layer and FV- E I is the video embedding of V,. In summary. our video encoder för encodes the video inputs by adopting fimg. fpros and faeg Eqs. (1)-
Text Encoder. For the language modality, we adopt BERT-Base [12] as our backbone to encode each input text T,, In detail, we first tokenize Ti into sequence of text tokens 14, ..................... (7), where denotes the length of T. Then we obtain the text token embeddings through the backbone foem:
F
fbert (1, 12, ,K(')
where Peett denotes the token embeddings of T, obtained by Sbert- We then project them by a Linear layer fproj into the d-dimensional space as:
Fy
Proo [ proos [1), Fre (2) Frt Pros [ri]]
= Foros Berrlll, F (2], .-. Fbbrelril)
(5)
where Fie denotes the j-th element of F bert: and pros [i] E Rdd represents the projected text embedding of token j in T, (which has the same dimension dl as video embedding Ff). To obtain the final text embedding of Ti we apply an Average Pooling layer fang:
F} = favg (f [4], F proj [2], ...
proy (ril),
(6)
where Fj. E Rdd denotes the text embedding of T; In summary, our text encoder for encodes the text inputs by adopting foert, Sproj. and favg in Eqs. (4)-(6).
3.3 BMU-MoCo
Cross-Modal MoCo. Similar the original single-modal MoCo [19], recent state-of-the-an video- language modeling works COTS [35] and HiT [33] construct a cross-modal MoCo architecture to maintain video/text momentum encoders by the same momentum update mechanism, which creates consistent queues for cross-modal contrastive learning objectives. As illustrated in Figure 2, our BMU-MoCo follows this paradigm and further transfers to our CVLM setting. Concretely, for a mini-batch NB video-text pairs B = [V,, [V), N, we first obtain the query embeddings q.' 9 of V,, T, by video encoder fey and text encoder fey:
for of = for
(7)
Then we maintain two momentum encoders fer... for... (termed local momentum video/text en- coders in Figure 2) for both video and text modalities, whose parameters fy.m. fr,m are initialized by Ov, Or the beginning of each task i. During the training process, Ov,m, 0r,m are continuously updated by By, Or with the momentum update strategy:
0y.m Oy.m (1 m) 0v, Or 0r.m + (1 m) Pr.
(8)

---

bidirectional
Lrzvv
Lyar
momentum update
similarty
similartty
Rg
Rj "., MY R$ Ry
gradient
gradient
Task
queve
quese
global momentum encoder
local momentum encoder
global momentum
momentum video encodes
video encoder
text encoder
encoder
Bv
Vawery
Trpary
0
8y.m
global video encoder
local video encoder
local momentum lext encoder
global text encoder
Task t-1
video encoder
text encoder
Figure 2: Schematic illustration of our BMU-MoCo. The momentum update strategy applied to both encoders and momentum encoders (i.e., bidirectional). To exploit earlier knowledge, we further maintain pair of global momentum encoders with the same BMU strategy, whose parameters are inherited across tasks and only initialized at the very beginning.
where m is the coefficient of momentum update. To form the contrastive learning loss of cross-modie MoCo, we need two consistent queues preserve the negative video/text samples. detail, the key embeddings ky ki of V.. T; are firstly acquired by momentum video and text encoders:
kY
Sev. (Vi), bo S0r (Ti).
We then respectively push ky and h?? into the negative video queue QV and the negative text queue Q" (after computing loss), where Q" {kY, ky ky ... k and QT {KT kg .g.... KNg (Nq is the queue size). The contrastive losses of cross-modal MoCo (Basc-MoCo) are:
No
Lv2
log
exp
(10)
NB
k?
NT
exp
exp
Ne
LT2
log
exp
(11)
NB
T.
Σ No
exp
exp
where is the temperature. Note that the queue size NQ is decoupled from the batch size Nn. Therefore, it can take large value for better representation of the data distribution.
Bidirectional Momentum Update. Although achieving great success with non-streaming data (e.g., single video-text dataset), the original cross-modal MoCo has difficulty in coping with the catastrophic forgetting problem under our CVLM setting. To overcome this difficulty, propose a novel bidirectional momentum update (BMU) strategy for cross-modal MoCo to review the old knowledge retained in momentum encoders at each training step. Concretely, for video/text encoders ffor in addition to the end-to-end update by back mack-propagation, we further update their parameters By, 0T using the parameters v 0r,m of momentum encoders fov.m. for... m by momentum update:
Oy r Ov + (1-m)- Ov.m 0r m 0T + (1 m) OT
(12)
where mn is a momentum coefficient, and 0y.m, fr.m are simultaneously updated by Eq. (8). Together, Eq. (8) and Eq. (12) compose our BMU strategy. Note that the advantages of BMU lie two aspects: (1) At the beginning of each new task f, 0y.m and 0r.m are respectively initialized by By and Or, which makes the knowledge of task
1 be preserved. (2) During the training process, 0y,m and 0r,m are constantly and slowly updated by the momentum update strategy, which enables model to review the old knowledge but without sacrificing the performance on new tasks.
Global Momentum Encoders. To further enhance our BMU-MoCo, we propose to maintain pair of global momentum encoders which can preserve earlier knowledge. shown Figure they only initialized at the very beginning of the whole training process under our CVLM setting,
5

---

and their parameters are transmitted across tasks. Formally, let fäv... and fōr... denote the global momentum video and text encoders, respectively. Their parameters 8v. and fr, are updated by the BMU strategy along with the parameters By and 0r of encoders:
Oy = r Ov + (1 11-m)) f)) Öv,m, Or = m Or + (1 - (1) Ör.
(13)
By.m = m 8y.m + (1 m) 8r, Br.m = m 0 + (1 m) fT.
(14)
Note that Eq. (12) and Eq. (13) are implemented subsequently. For each video-text input (Vi, Ti we obtain new group of key embeddings ky, Ke with the global momentum encoders föy... Jō
RY Sōv... (Vi), b.f So (T:)-
(15)
We push Ry and ET respectively into two negative queues Q and QT. where Q {b ky ....... -. NNo Q" = {&T ky, kg. ....... ....,, kKgg. Note that each query embedding (e.g., g has two corresponding positive embeddings (NY" ky and two corresponding negative queues (Q",0) (Q)). The cross-modal contrastive losses are defined as:
N
of AT. exp ( q AI + exp
1
Lvrr-
log
(16)
NB
2 K." + exp
of k?
1=1
exp
+ lexp
tepp
Ng
1. J.F.Y
1
exp
expl
LT2V
log
(17)
NB
of
kV
of
RV
exp
cxpp
lexp
exp
where T is the temperature. Now we have the final loss of BMU-MoCo for our CVLM setting:
Liiaa
Lver + Lrev.
(18)
The full (pseudocode) algorithm of our BMU-MoCo is presented in the supplementary material.
4 Experiments
4.1 Experimental Setup
Our CVLM setting defined over a sequence five video-text datasets: (1) VATEX [54] is a large-scale open-domain dataset, which has 25,991 videos with 250K text descriptions for training, 3,000 videos for validation and 6,000 videos for testing. (2) ActivityNet [25] is an action domain dataset, which consists of 20K YouTube videos with 100K text descriptions. We follow the standard setting in [4, 28] to use 10K videos for training and 4.9K for test (the vall split), where all texts of each video are concatenated into one query paragraph. (3) MSR- VTT [55] contains 10K videos, with 20 text descriptions per video. We follow the Ik-A split in recent works [28, 4, 56, 33] with 9K training videos and IK test videos. (4) DiDeMo [20] consists of 10K Flickr videos with 40K text annotations. Following [4, 28), we train and evaluate our model on paragraph-to-videe
Datasets.
retrieval (the same setting for AetivityNet). (5) MSVD [10] has 1,200 videos with 48K texts for training. 100 videos for validation and 670 ones for testing. Overall, there are around 50K videos with 500K text descriptions in all five datasets (i.e., each dataset has 100K video-text pairs in average).
Evaluation Metrics. Similar to the standard video-language modeling setting, we evaluate the text-to-video retrieval performance of model on Recall @ (shortened as R@ 1). R@I refers the percentage of text queries that correctly retrieve the ground-truth candidate at top- For our CVLM setting, we further define two evaluation metries: forgetting rate (FR) and harmonic mean (HM). Formally, let M; denote the model after trained on task and A (t < i) denotes the R @ 1 result of M, on task t. The overall @ 1 (2 Σi_ AA) the average R @ results of the final model M,, on all tasks. Based on these notations, we then define the FR and HM as follows:
(1) Forgetting rate (FR) of M, on task / (1 < i) is the performance decrease between M, and M₁: FR = A4 Ai, where a lower FR indicates the model forgets less knowledge. Note that there is no catastrophic forgetting (FR = 0) when = i. The Overall FR is obtained by just summing the FR results of the final model M,, across all tasks: Overall FR Σr= (Af AT).

---

Table 1: Comparative results obtained by the final model M, on five video-text datasets/tasks under our CVLM setting: VATEX [54] (i.e., Task1), ), ActivityNet [25] (i.e., Task2), MSR-VTT [55] (i.e., Task3), DiDeMo [20] (i.e. Task4), MSVD [10] (i.e. Task5). For fair comparison, all baseline models are re-implemented based on the same cross-modal MoCo architecture for our CVLM setting. denotes applying extra encoders, including encoders from the last task (e.g., LwF [32]) and global momentum encoders (e.g, our BMU-MoCo). 'Mem.' denotes applying memory buffer during training. 'BMU-MoCo (local)' denotes BMU-MoCo without global momentum encoders.
Task1
Task2
Task3
Task4
Task5
Overall
Method
Mem.
R@1t
FR.
R@11
FR.
R@11
FR.
R@1
It
FR.
R@11
R@1
FRI
IMM
Base-MoCo [35]
No
38.99
15.30
18.61
15.23
28.00
5.60
28.22
7.27
40.28
30.82
43,40 34.67
LWF! [32]
No
42.02
12.27
19.95
14.87
28.90
6.00
29.91
6.98
37.91
32.24
40.12 35,81
ER-ring [9]
Yes
41.99
12.30
22.09
11.79
29.80
5.40
30.31
5.08
38.53
32.54
34.57 35.67
DER [5]
Yes
40.15
14.14
21.35
12.65
28.80
5.00
30.71
4.09
39.96
32.19
35.88 35.39
Co2L [6]
Yes
41.23
13.06
21.74
13.06
27.50
5.30
30.41
5.38
39.29
31.58
34.48 35,06
LUMPf [37]
Yes
40.16
14.13
21.78
12.37
30.50
3.00
29.91
4.99
39.39
32.45
34.49
35.56
BMU-MoCo (local)
No
46.82
7.47
23.27
10.84
30.00
3.40
31.21
4.08
41.94
34.65
25.79
37.05
BMU-MoCo!
No
48.48
5.81
23.45
10.43
30.80
2.90
32.80
3,49
41.83
35.47
22.63
37.59
(2) Harmonic mean (HM) calculates the harmonic mean of the overall R @ (current) and the overall R@1 1 (final), where the overall R @ (current) denotes the average of the R@1 values obtained by each current model M; (i = 1, 2, ... n) on each current task i, and the overall R@ (final) denotes the
2. EE A:.1 1. AE average R @ for the final model M, on all tasks. Formally, we have: HM = 1 Li=: A Σ 2 A. Note that HM can alleviate the trade-off problem between overall R@ (current) and overall R@1 (final), which is otherwise an inherent limitation of FR. Specifically, when a model has a lower overall @@ (current) and lower overall R@! 1 (final), it could also have better/lower FR (which is unsatisfactory) but still lead to a worse/lower HM (see Figure 4(c)).
Implementation Details. Recent works on vision + language demonstrate that video-language models benefit from image-text pre-training [28, 4], which can accelerate the model convergence and is more suitable for the real-world application scenarios. We thus apply ViT-Base [13]/BERT- Base [12] as our image/text encoder, and follow the recent sfate-of-the-art MoCo-based model COTS [35] to pre-train our model with 5.3M image-text pairs. We then sequentially train all the models (BMU-MoCo and all competitors) on five video-texi datasets/tasks. For each task, we train model for 10 epochs and choose the best trained one w.r.t. the validation R @ results. For those competitors using a memory buffer during training (e.g.. ER-ring [9]), we set the memory size to 10% of the average data size 100K (i.e., 10K video-text pairs). Note that the percentage 10% is larger than the buffer size of most recent sidearssl-based continual learning methods [9. 5, 37.. More implementation details are given follows: (1) In the training phase, all sampled frames of each video are resized to 384x x 384 and augmented by gray- gray-scaling and color-jitter. (2) For the first epoch of each task under CVLM setting, we set the learning rate to 5e-5 and decay it to 5e-6 afterwards. (3) We select the two momentum coefficients m = 0.99, i. = 0.99, and the temperature r = 0.07. We set the batch size NB to 48 and the queue size No 1,440. (4) The total training time on five tasks is around 20 hours with 8 Tesla V100 GPUs for each model.
4.2
Main Results
Table summarizes the comparative results in terms of text-to-video retrieval (R @ 1), forgetting rate and harmonic mean (HM) obtained by the final model M,, (per method) on five datasets. We re-implement five recent continual learning methods (fused with cross-modal MoCo) under our CVLM setting, including rehearsal-baad -based methods (ER-ring [9]. DER [51), regularization-baa
methods (LwF [32]) and their combinations (Co2L [6], LUMP [37]). We can observe that: (I) Our BMU-MoCo outperforms recent methods by large margins without using any extra memory or dynamically extended networks. Concretely. our method achieves the best R@ and FR results on all tasks, and outperforms the second best by 2.93% for overall R R@1 1, 1 11.85% for overall FR and 1.78% for overall HM. (2) Without applying global momentum encoders, our BMU-MoCo (local) also beats all competitors, directly showing the effectiveness of our BMU strategy. (3) The improvements over Base-MoCO obtained by utilizing knowledge distillation (e.g., LwF [32]) or extra memory data (e.g..
Co2L [6]) are limited due to the lack of category information under our CVLM setting.
7

---

Base-MoCo
ER-ring
Co2L
BMU-MoCo
60
40
55
VATEX
35
ActivityNet
R@1 (%)
R@1 (%)
30
50
25
45
20
40
15
35
10
1
2
3
4
5
2
3
5
Taskindex
Task index
Figure 3: Detailed comparative results for text-to-video retrieval (R@) obtained by each model M; (per method) on the first two tasks: VATEX and ActivityNet. Note that the gap between the beginning and the end of each line denotes the forgetting rate.
Table 2: Ablation study results for our BMU-MoCo. 'Local' denotes applying the local momentum encoders, while 'Global' denotes applying the global momentum encoders. Results for text-to-video retrieval @R@ 1), forgetting rate (FR) and harmonic mean (HM) reported.
Task1
Task2
Task3
Task4
Task5
Overall
BMU Local Global
R@ It
FR.
R@1
FR.
R@I
R@It
FR,
R@
If
FR, R@I 1f
R@11
FRI
HM
38.99
15.30
18.61
15.23
28.00
5.60
28.22
7.27
40.28
30.82
43.40 34.67
38.95
15.97
18.51
15.45
30.10
4.60
28.91
6.78
39.89
31.47
42.75 35.16
41.44
12.85
21.68
13.22
29.40
4.90
29.31
6.08
39.96
32.35
36.95 35.67
V
46.82
7.47
23.27
10.84
30.00
3.40
31.21
4.08
41.94
34.65
25.79 37.05
46.35
7.94
23.16
10.99
30.60
3.70
31.41
5.28
41.70
34.64
27.91 37.22
V
48.48
5.81
23.45
10.43
30.80
2.90
32.80
3.49
41.83
35.47
22.63 37.59
Figure 3 shows more detailed comparative results for text-to-video retrieval (R@) @ 1) obtained by each model M, (per method) on the first two datasets (VATEX [54] and Activity [25]). We compare our BMU-MoCo with three representative competitors, including Base-MoCo [19], ER-ring [9], and Co2L [6]. Concretely, the left sub- fugufigure presents the results of M 1 ~ Ms (per method) on task (VATEX), i.e., A (1 i < 5). The right sub-figure presents the results of M2 ~ MM, (per method) on task (ActivityNet) i.., A (2 < S 5). It can be observed that: (1) For task (VATEX), the performance of our BMU-MoCo drops the most slowly after it is trained on the following tasks (task 2 to task 5). (2) For task (ActivityNet), our BMU-MoCo also leads to the slowest performance drop after trained the following tasks (task 3 to task 5). Overall, our BMU-MoCo indeed significantly alleviates the performance decrease problem during the whole training process.
4.3 Ablation Study
We first conduct ablation study to demonstrate the contributions of the BMU strategy. the local momentum encoders and the global momentum encoders applied in our BMU-MoCo. The ablative results are shown in Table It can be clearly seen that: (1) With our BMU strategy, our model achieves remarkable improvements (4th row vs. 1st row). (2) Simultaneously applying local and global encoders better than using only one of them (3rd row vs. 1st/2nd row), which indicates that the knowledge preserved in local and global momentum encoders are quite different and thus complementary to each other. (3) Our BMU strategy helps our model to excavate old knowledge preserved in different momentum encoders (6th row 3rd row) and achieve the best performance (6th row vs. 4th/5th row), which further validates the effectiveness of our BMU.
Considering the core role of BMU, we thus analyze the impact of the momentum coefficient m utilized in our BMU-MoCo. According to COTS [35], the other momentum coefficient m of our model is fixed at 0.99 (only the value of m is changed). Figure 4 shows the results for overall R I (current), overall R@1 (final), and overall HM, respectively. We find that the value of vi cannot be too big or too small. Concretely, when in is too big (e.g.. 0.999 and 1), the knowledge preserved in momentum encoders cannot be well-rev by our model. When vi is too small (e.g., 0.9), the end-to-end update by back poop-aropagttion is influenced too much, which leads to bad results for
8

---

Overal R@o (cumer))
40
40
39.00
40.02
39.63
Ovenal n@a (fhra)
36
35.47
Ovenall HM
33
36,82
37.59
36.19
39
38.39
34
33.04
36
34.67
30.82
34
37
30
32
0.9
0.99
0.909
1
0.999
0.9
0.99
0.999
Ait for BMU-MoCo
A for BMU-MoCo
m for BMU-MoCO
(a)
(b)
(c)
Figure Comparative results of our BMU-MoCo with different momentum coefficient m. The results for overall R @ (current), overall R @@1 (final) and overall HM are reported in (a-c), respectively.
ER- ER-ring
BMU-MoCo
ER-ring
BMU-MoCo
36
35.47
45
42.77
Overall R@1 (firal)
35
oqual space
40
34
36,04
34.57
35
33
32.54
32.01
30
32
31
30.96
25
requal space
22.63
30
20
5
10
memory size (%)
memory size (%)
Figure 5: Comparative results obtained by different memory size. The red dotted line denotes our BMU-MoCo (with memory) while the blue line denotes the representative rehearsal-based method ER-ring. The green line suggests that the storage space consumption of our BMU-MoCo (with global momentum encoders) is equal to ER-ring (with 0.05% memory to store videos).
overall R@1 (current) and overall HM. It is worth mentioning that the model with smaller rn (0.9) has lower/better FR (14.95) since sacrifices the model performance on overall @ (current). This phenomenon demonstrates the necessity of utilizing the overall HM to measure the overall (trade-off) model performance. Therefore, we set the momentum coefficient in to 0.99 in all our experiments, which helps our model review old knowledge while learning well on new tasks.
4.4 Further Evaluation
To demonstrate both the efficiency and effectiveness of our BMU-MoCo under our CVLM setting, we compare our model with a representative rehearsal-based method ER-ring [9] by different memory sizes in Figure 5. Note that our B.MU-MCCo has two global momentum encoders that need 0.5GB more storage space than the original cross-modas MoCo (used by all competitors including ER-ring). As shown in Figure 5, when the memory size of ER-ring becomes 0.05%, it equals to the size of extra storage space used by our BMU-MoCo (but our model performs significantly better). In real- world application scenarios, the memory size of rehearsal-barte the number of tasks grows, while the fixed extra space size (0.5GB) of our BMU-MoCo negligible. More importantly, our BMU-MoCo (with a fixed 0.5GB sapce size) even outperforms ER-ring using 10% memory (about 200GB under our CVLM setting) by large margins on both overall @ (final) and overall FR. This directly indicates the efficiency and effectiveness of our BMU-MoCo.
methods like ER-ring enlarges rapidly as
Note that the key factor our BMU-MoCo is to learn holistic knowledge from momentum encoders by momentum update. This momentum update strategy has two characteristics: (1) encoders are updated on each iteration by momentum update and (2) momentum encoders are simultaneously updated. We thus consider two alternative strategies to further validate the effectiveness of our BMU-MoCo in Table (1) SMU-MoCo-ensemblle updates the encoders by adding the parameters of the momentum encoders at the end of each task (i.e., ensemble), which is different from the peraaion-by-iteraee
strategy our BMU-MoCo. can be observed that BMUU-MoCoeensemble learns better than the baseline method (Base-MoCo) but still has large performance gap compared with our BMU-MoCo. (2) BMU-MoCo-fnee fixes the momentum encoders during each task (only re- initialized by encoders at the beginning of each task) and utilizes it to update the encoders in

---

Table 3: Different momentum update strategies for our BMU-MoCo. Results for text-to-video retrieval (R@1) 1), forgetring rate (FR) and harmonic mean (HM) are reported.
Task1
Task2
Task3
Task4
Task5
Overall
Method R@ 11
FRI
R@1
1
FR.
R@
11
FRI
R@1
11
FR.
R@11
1
R@1
FR,
HM
Base-MoCo
38.99
15.30
18.61
15.23
28.00
5.60
28.22
7.27
40.28
30.82
43.40 34.67
BMU-MMCoo-ensemble
41.44
12.85
21.68
13.22
29.40
4.90
29.31
6.08
39.96
32.35
36.95 35.67
BMU-MoCo-iixed
51.36
2.93
23.12
7.38
25.10
1.50
30.11
2.00
38.19
34.07
13.81 35.12
BMU-MoCo 48.48
5.81
23.45
10.43
30.80
2.90
32.80
3.49
41.83
35.47
22.63 37.59
Table 4: Comparative results of unsupervised continual learning (UCL). "Multi-Task" represents the upper-bound method which is based on multi-task learning (over all tasks). "Finetune" the lower-bound method that directly fine-tunes the model across all tasks without other strategies.
Split CIFAR-10
Split CIFAR-100
Split Tiny-
Method
Accuracyt
Forgetting
Accuracyt
Forgetting!
Accuracy!
Forgetting.
Multi-Task
95.76
86.31
82.89
Finetune
90.11
5,42
75.42
10.19
71.07
9.48
PNN [48]
90.93
66.58
62.15
SI [60]
92.75
1.81
80.08
5.54
72.34
8.26
DER [5]
91.22
4.63
77.27
9.31
71.90
8.36
LUMP [37]
91.00
2.92
82.30
4.17
76.66
3.54
BMU-MoCo
92.80
1.98
83.81
3.69
77.45
3.39
the same iteration by-iteration way of our BMU-MoCo. It can be observed that BMU-MoCo-fixed focuses reviewing the old knowledge but sacrifices the model performance current tasks, which leads to worse overall R @@ (final) and HM. Therefore, considering all evaluation metrics @R@1 1, FR, and HM), the update strategy (for encoders) of our BMU-MoCo is better than its alternatives.
Finally, to show the transferability of our BMU-MoCo under other continual learning settings, we apply it to the unsupervised continual learning (UCL) setting which is originally proposed in
LUMP [37]. This UCL setting includes three benchmark datasets for training and evaluation: (1)
Split CIFAR-10 10 [26] consists of five tasks with 2 random classes per task; (2) Split CIFAR- -00 [26] consists of 20 tasks with 5 random classes per task; (3) Split Iiny-ImageNet consists of 20 tasks with 5 random classes (out of 100 classes from ImageNet [11]) per task. The comparative results of UCL
are shown in Table 4. We can observe that our BMU-MoCo outperforms all the competitors including PNN [48], SI [60], DER [5], and LUMP [37], showing the effectiveness of our BMU-MoCo under the UCL setting. This also demonstrates the general applicability of our BMU strategy.
5 Conclusion
In this paper, we propose new continual video-language modeling (CVLM) setting, where models are supposed to sequentially trained on five widely-used video-text datasets. To overcome the catastrophic forgetting and heavy resource consumption challenges, we propose a novel framework BMU-MoCo, which cross-modal MoCo-based model with bidirectional momentum update (BMU). We maintain both local and global momentum encoders with our BMU strategy to review broader old knowledge while lcarning on new tasks. Extensive experimental results show that our BMU-MoCo outperforms recent competitors by large margins, even without using extra memory data or dynamically extended networks. The limitation of our work lies in that we have only evaluated BMU-MoCo under the CVLM setting, and thus need transfer it to other continual learning settings (e.g., continual image-text pre-training for comprehensive study.
Acknowledgments and Disclosure of Funding
This work was supported in part by National Natural Science Foundation of China (61976220 and 61832017), Beijing Outstanding Young Scientist Program (BJJWZYJHO) 2019 100020099)99), and the Research Seed Funds of School of Interdisciplinary Studies, Renmin University of China.
10

---

References
[1] Ferran Alet, Tomás Lozano-Pérez, and Leslie P Kaelbling. Modular meta-learning. 856-868. 2018
In CoRL, pages
[2] Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with network of experts. In CVPR, pages 3366-3375, 2017.
[3] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In NewrlPS, pages 11817-11826, 2019.
[4] Max Bain, Arsha Nagrani, Gul Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV, pages
2021.
[5] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. In NewrlPS, pages 15920-15930, 2020.
[6] Hyuntak Cha, Jacho Lee, and Jinwoo Shin. Co21.: Contrastive continual learning. In ICCV, pages 9516-9525, 2021.
[7] Michael B Chang, Abhishek Gupta, Sergey Levine, and Thomas Griffiths. Automatically composing representation transformations as a means for generalization. arXiv preprint arXiv:/807 2018.
[8] Arslan Chaudhry, Albert Gordo, Puneet K Dokania. Philip Torr, and David Lopez-Paz. Using hindsight to anchor past knowledge in continual learning, arXiv preprint arXiv:2002. 08160, 3, 2020.
[9] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv: 10486, 2019.
[10] David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In ACL, pages 190-200, 201
[11] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li. Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. CVPR. pages 248 223,22552222 2009.
[12] Jacob Devlin, Ming-We Mei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HILT, pages 4171-4186, 2019.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenhorn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakoh Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
[14] Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In AISTATS, pages 3762-3773 2020.
[15] Nanyi Fei, Zhiwu Lu, Yizhao Gao. Guoxing Yang. Yugi Huo, Jingyuan Wen, Haoyu Lu, Ruihua Song, Xin Gao, Tao Xiang. Hao Sun, and Ji-Rong Wen. Towards artificial general intelligence via multimodal foundation model. Nature Communications, 13:3094, 2022.
[16] Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori 7wols, David Ha, Andrei Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint 1711::1001.87744 2017.
[17] Simon Ging, Mohammadreza Zolfaghari, Hamed Pirsiavash, and Thomas Brox. COOT: Cooperative hierarchical transformer for video-text representation learning. In NeurlPS, pages 22605-22618, 2020.
[18] Ian Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based based neural networks. arXiv preprint arXiv: / /3/2.62/11 1,, 2013.
[19] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsuper- vised visual representation learning. In CVPR, pages 9726-9735. 2020.
[20] Anne Lisa Hendricks, Oliver Wang, Eli Shechtman, Josef Sivie, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In ICCV, pages 5804-5813, 2017.
[21] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Lifelong learning via progressive distillation and retrospection. In ECCV, pages 437-453 437-452, 2018.

---

[22] Saihui Hou, Xinyu Pan, Chen Change Loy. Zilei Wang. and Dahua Lin. Learning unified classifier incrementally via rebalancing. In CVPR, pages 831-839, 2019.
[23] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, forgetting in neural networks. PNAS, 114((1))3521-3353 2017.
Grabska-Barwinkka, et al. Overcoming catastrophic
[24] Bruno Korbar, Fabio Petroni, Rohit Girdhar, and Lorenzo Torresani. Video understanding as machine translation. arXiv preprint arXiv:2006.07200 2020.
[25] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niehles. Dense-captioning events in videos. In ICCV, pages 706-715, 2017.
[26] Alex Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Tront, 2009.
[27] Frantzeska Lavda, Jason Ramapuram, Magda Gregorova, and Alexandros Kalousis. Continual classification learning using generative models. arXiv preprint 111.:::8100.061111 2018.
[28] Lei, Linjie Luowei Zhou, Zhe Gan, Tamara Berg, Mohit Bansal, and Jingjing Liu. Less is more: ClipBERT for vidco-and- and--aggagge learning via sparse sampling. CVPR, pages 7331-7341, 2021.
[29] Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. Unicoder- VL: A universal encoder for vision and language by cross-modal pre-training. In AAAI, pages 11336-1 1344, 2020.
[30] Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. HERO: Hicrarchical encoder for vidco+ language renieeepresnntttinnn pre-training. EMNLP, pages 2046-2065, 2020.
[31] Xilai Li, Yingho Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. In ICMI., pages 3925-3934, 2019.
[32] Zhizhong Li and Derek Hoiem. Learning without forgetting. TPAMI, (0((1))993322994 2017.
[33] Song Liu, Haoqi Fan, Shengsheng Qian, Yiru Chen, Wenkui Ding, and Zhongyuan Wang. HiT: Hierarchical transformer with momentum contrast for video-text retrieval. In ICCV, pages 11915-11925, 2021.
[34] David Lopez-Paz and Mare" Aurelio Ranzato. Gradient episodic memory for continual learning. NeurlPS, pages 1470-6479, 2017.
[35] Haoyu Lu, Nanyi Fei, Yuqi Huo, Yizhao Gao, Zhiwu Lu, and Ji-Rong Wen. COTS: Collaborative two- stream vision-language pre-training model for cross-modal retrieval. In CVPR, pages 15692-15701, 2022.
[36] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lec. ViLBERT: pretraining task-agnostic visiolinguistic representations for vision-and-languaa languagge tasks. In NeurlPS, pages 2019.
2019.
[37] Divyam Madaan, Jachong Yoon, Yuanchun Li, Yunxin Liu, and Sung Ju Hwang. Representational continuity for unsupervised continual learning. ICLR, 2021.
[38] Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of Learning and Motivation, volume 24, pages 109-165. Elsevier, 1989.
[39] Antoine Miech, Dimitri Zhukov, Jean- Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. HowTo100M: Learning text-video embedding by watching hundred million narrated video clips. In ICCV, pages 2630-2640, 2019.
[40] German Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019.
[41] Mandela Patrick, Po-Yao Huang, Yuki Asano, Florian Metze, Alexander G Hauptmann, Joao F Henriques, and Andrea Vedaldi. Support-set bottlenecks for video-text representation learning. In ICLR, 2020.
[42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, pages 8748 8763, 2021.
[43] Jason Ramapuram, Magda Gregorova, and Alexandros Kalousis. Lifelong generative modeling. Neuro- computing, 404:381 400, 2020.
12

---

[44] Amal Rannen, Rahaf Aljundi. Matthew B Blaschko, and Tinne Tuytelaars. Encoder based lifelong learning. In ICCV, pages 1320-1328, 2017.
[45]
Sylvestre-Alvise Rehuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In CVPR. pages 2001-2010, 2017.
[46] Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv:1810.1 2018.
[47] Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint urXiv:1711.01239, 2017.
[48] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Ko- ray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606. 04671, 2016.
[49] Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. arXiv preprint or 1i:i:: 2021.
[50] Joan Serra, Didae Suris. Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In ICML, pages 4548-4557, 2018.
[51] Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In NewrIPS, pages
2017.
[52] Hao and Mohit Bansal. LXMERT: learning cross-modality encoder representations from transformers. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, EMNLP-IJCNLP, pages 5099-51 5100, 2019.
[53] Xiaohan Wang, Linchao Zhu and Yi Yang. T2VLAD: global-local sequence alignment for text-video retrieval. In CVPR. pages 5079-5088, 2021.
[54] Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. VaTeX: A large-scale, hugl-quality multilingual dataset for video-and and--nd--aaa and-lagga research. In ICCV, pages 4590, 2019.
[55] Jun Xu, Tao Mci, Ting Yao, and Yong Rui. MSR-VTI VTT: A large vidco description datasct for bridging vidco and language. In CVPR, pages 5296, 2016.
[56] Jianwei Yang. Yonatan Bisk, and Jianfeng Gao. Taco: Token-aware cascade contrastive learning for video-text alignment. In ICCV, pages 11562-11572, 2021.
[57] Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In CVPR, pages 8715 8724, 2020.
[58] Jachong Yoon, Eunho Yang, Jeongtac Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In ICLR, 2018.
[59] Youngjae Yu, Jongseok Kim, and Gunhee Kim. A joint sequence fusion model for video question answering and retrieval. In ECCV, pages 487-503, 2018.
[60] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In ICML, pages 3987-3995, 2017.
[61] Linchao Zhu and Yi Yang. ActBERT: Learning global-loca video-text representations, In CVPR, pages 8743-8752, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Please see Section 5.
(c) Did you discuss any potential negative societal impacts of your work? [No]
13

---

(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Please see Section 4.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please see Section 4.1.
4. If you using existing assets (e-g.. code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cited the creators of all five datasets in Section
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you's 're
using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "srJqej-zfHT", "reviewer": "Reviewer_ZHhB", "summary": "This paper incorporates the video-language modeling into continual learning. To overcome the challenges in continual learning, this paper proposes a novel cross-modal MoCo-based model with bidirectional momentum update (BMU).", "questions": "Could the author conducted more experiments on the multi-task transfer learning part?", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths: \n1) This paper provides the a new continual video-language modeling setting.\n2) The extensive experiments demonstrate the effectiveness of the proposed method.\n\nweaknesses: \n1) This paper focuses on the video-language modeling in continual learning, however, the experiments only conducted on the text-to-video retrieval task. Since the proposed framework is two-stream architecture and is suitable for adjusting to the other video-language tasks, the experiments could be further investigated into multi-task transfer learning[1] in VLM domain, instead of on the single task.\n2) The setting of two momentum encoder branches(local and global) is too complicated and the gains from global branch in table1 seems not significant.\n\n[1] A deep hierarchical approach to lifelong learning in Minecraft, AAAI17", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "pwAgPw6jSiY", "reviewer": "Reviewer_6RoA", "summary": "The authors propose a new continual video-language modeling (CVLM) setting, where models are supposed to be sequentially trained on five widely-used video-text datasets.\nThe authors propose the BMU-MoCo, a cross-modal MoCo-based model with a novel bidirectional momentum update (BMU) strategy.\nTo further boost our BMU-MoCo, a pair of global momentum encoders are maintained by the same BMU strategy.\nThe proposed method is tested on the proposed benchmark, and shows promising results. ", "questions": "1. Equation (16) between Line 185 and 186: how is this contrastive loss implemented? Typically, such cross-entropy loss is implemented with log-softmax function to stablize the computation. However, in Equation (16), it seems not straightforward to use log-softmax to compute this loss. If computed by softmax and then log, is it stable?\n\n2. LIne 290, \"even outperforms ER-ring using 10% memory (about 200GB under our CVLM setting)\". I'm curious that how this baseline that requires 200GB memory was implemented. Could the authors provide more details on the implementations of all these baselines. From my perspective, it would be a better contribution if the authors can make the CVLM benchmark and its baselines solid, compared with the newly proposed method. ", "limitations": "Yes. ", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nOn the newly promised benchmark CVLM, the authors implemented several previous well-known baselines. This is a good effort. \n\nWeakness:\nThe proposed method is only tested on the proposed benchmark. The authors may want to first make the proposed benchmark solid, or to show effectiveness of the proposed method on other continual learning benchmarks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "H3SiN9K4NZ_", "reviewer": "Reviewer_oj9S", "summary": "The authors introduce a new setting of the Video-Language Modeling task in a continual learning scenario. They propose an evaluation schema based on text2video (cross-modal) retrieval with 5 different tasks which correspond to separate popular video-language modeling datasets. Moreover, the authors propose their own method, namely BMU-MoCo, based on MoCo self-supervised representation learning approach, and compare it against other continual learning methods. The proposed method gives the best results given the evaluation metrics that measure the performance degradation between consecutive tasks, particularly ‘Forgetting rate’ and ‘Harmonic Mean’.", "questions": "In Figure 1a) what do colors represent? What are the current models and the final model?\n\nCould you explain in detail what happens when the queue size Nq is larger than Nb?\n\nWhat does FR for Task 1 indicate? Is there a Task 0 then? The comparison is not clear.\n\nIs 0.5 additional GB of memory for BMU-MoCo local only or both?\n\nHow are frames sampled and fed to ViT and how is the averaging over the whole video being performed?\n", "limitations": "The authors very briefly mention the limitations of their work, with no discussion of potentially negative impact. In my opinion, the limitations of this work are two-fold. First, as the authors mention, they only tackle the CVML task, however, to fully address this task, the results of state-of-the-art approaches on particular datasets should also be included, showing that they indeed struggle with catastrophic forgetting. Otherwise, it would be beneficial to address other cross-modal tasks.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.\tThe Paper is well-structured, and the contribution is clearly stated and supported in the experimental section.\n2.\tThe authors propose a new approach to evaluate the effectiveness of VLM models in a continual learning setup with new metrics to evaluate the task.\n3.\tThe proposed method gives the best results among other continual learning-based approaches.\n\nWeaknesses:\n1.\tThe method section is a bit messy and becomes hard to follow.\n2.\tNotations in 3. are definitely over complex, and could be simplified, e.g. by dropping ‘i’ in Eq. 1-6.\n3.\tThe method is not entirely free from additional resources contrary to what the authors claim, since it requires an additional model (momentum encoder- local + global) to be trained and stored, as well as the queue for the training.\n4.\tSome typos and minor editing issues.\n5.\tThe conclusions on the m hyperparameter are unclear.\n6.\tBoth figures 1 are non-informative and confuse the reader.\n7.\tLack of clear motivation why CVML among other cross-modal tasks and evaluation only on one of them seems to be quite limited.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept