Expectation-Maximization Contrastive Learning for Compact Video-and-Language Representations
Authors: Peng Jin
Jinfa Huang
Fenglin Liu
Xian Wu
Shen Ge
Guoli Song
David A. Clifton
Jie Chen
Venue: NeurIPS 2022 Accept (2022)
Keywords: Expectation-Maximization algorithm
contrastive learning
video-and-language representation learning
cross-modal retrieval

Abstract: Most video-and-language representation learning approaches employ contrastive learning, e.g., CLIP, to project the video and text features into a common latent space according to the semantic similarities of text-video pairs. However, such learned shared latent spaces are not often optimal, and the modality gap between visual and textual representation can not be fully eliminated. In this paper, we propose Expectation-Maximization Contrastive Learning (EMCL) to learn compact video-and-language representations. Specifically, we use the Expectation-Maximization algorithm to find a compact set of bases for the latent space, where the features could be concisely represented as the linear combinations of these bases. Such feature decomposition of video-and-language representations reduces the rank of the latent space, resulting in increased representing power for the semantics. Extensive experiments on three benchmark text-video retrieval datasets prove that our EMCL can learn more discriminative video-and-language representations than previous methods, and significantly outperform previous state-of-the-art methods across all metrics. More encouragingly, the proposed method can be applied to boost the performance of existing approaches either as a jointly training layer or an out-of-the-box inference module with no extra training, making it easy to be incorporated into any existing methods.

Expectation-Maxiniaatoon Contrastive Learning for Compact Video-and-Languaa Representations
Peng Jin 133
Jinfa Huang 1.
Fenglin Liu
Xian Wu
Shen Ge
Guoli Song"t
David Clifton Jie Chen
School of Electronic and Computer Engineering, Peking University, China
PPngg Cheng Laboratory, Shenzhen, China
3 AI for Science (A14S) -reeer-de Program, Peking University Shenzhen Graduate School, China
Department of Engineering Science, University of Oxford, UK Teencett JARVIS Lab, China
"Oxford-Szzhon Centre for Advanced Research, Suzhou, China
(jp21, jinfahuang | @stu. www.upku.edccco (songgl, chenj) ) @pel.ac.cn ac.en
(shenge, kevinxwu) @tencent.com [fenglin.liu, david. clivf.cliftoo// | @eeng.ox.cc..o
Abstract
Most video-and-language representation learning approaches employ contrastive learning, e.g., CLIP [53], to project the video and text features into common latent space according to the semantic similarities of text-video pairs. However, such learned shared latent spaces are not often optimal, and the modality gap between visual and textual representation can not be fully eliminated. In this paper, we propose Expectation-Me Satrtttiiii--aaiizziiiii Contrastive Learning (EMCL) to learn compact video-and-le and-language representations, Specifically, we use the Expectation- Maximization algorithm find a compact set of bases for the latent space, where the features could be concisely represented as the linear combinations of these bases. Such feature decomposition rank of the latent space, resulting in increased representing power for the semantics. Extensive experiments on three benchmark text-video retrieval datasets prove that our EMCL can learn more discriminative video-and- language representations than previous methods, and significantly outperform previous state-of-the-art methods across all metrics. More encouragingly, the proposed method can be applied to boost the performance existing approaches either as a jointly training layer or inference module with extra training, making it easy be incorporated into any existing methods..
aadeo-and-languse
representations reduces the
Introduction
Text-video retrieval [76], which aims to fetch relevant videos using textual queries vice versa, an important yet challenging cross- modal task. The dominating paradigm text-video retrieval contrastive learning [67, 24, 9, 23, 11, 10], which is commonly adopted framework for video- and-language representation learning. The core idea of contrastive learning is to pull the textual and visual representations of matched text-video pairs together and push the representations of unmatched text-video pairs apart. In this manner, contrastive learning enables neural networks to learn discriminative iido-and-language representations.
Equal contribution.
I Corresponding author: Guoli Song, Jie Chen. Coode https://ittub..cm/jphuu /tE.o////ppp/////LLIL.L 7/EMCL
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Video
Text
20 Different Classes
Hyperplane between videos and texts
Text guy driving car.
Text A man riding a surfboard.
Text: horse jumps inside his stable.
Text A man is riding surtboard.
Text A guy driving a
Text: horse
jumps inside his stable
(b)
Figure 1: Visualization of video and language representations learned by (a) recent standard contrastive learning method [21] and (b) our anpectation-maximizatti contrastive learning method. (a) shows that although the representations belonging the same class are well-clustered, there is still a clear dividing line (see Hyperplane) between video representations and language representations. Besides, the representations with different semantics (i.e., from different semantic classes) overlap with each other. On the contrary, (b) shows that our method can learn more discriminative video-and- language representations. In detail, the addo-and-laad and-language representations with the same semantics (i.e., belonging to the same semantic class) are rell-grouped; and there are clear dividing spaces between different semantic classes.
However, standard contrastive learning has intrinsic limitations for text-video retrieval tasks, since the success of contrastive learning largely depends on the volume and variety of negative samples [9]. Without adequate negative samples, it would be hard to guide the direction of sample learning, and the features could not be contrasted well. As shown in Figure la, we randomly select three videos (cach video has 20 text captions) from the benchmark dataset [70] and analyze the entire feature space of recent standard contrastive learning method [21]. Here we visualize the feature space with t-SNE [43]. In an ideal feature space, the instances of the same semantic classes should be close to each other. However, shown in Figure la, we notice that the feature space learned by standard contrastive learning fails to preserve inter-modal semantic relatedness, where videos and texts with the same class semanties are still far away. In other words, the semantic-relevant but modal-different instances can not be grouped together in this feature space. To improve the performance of contrastive learning, recent image-text representation learning methods either increase batch sizes or maintain large data memory banks [9, 11, 44, 67, 23]. These works target to collect sufficient negative samples for contrastive learning. However, such text-video retrieval approaches are relatively expensive due to the incurred extra computational costs, especially on large-scale datasets [57]. So an urgent challenge here is to find an efficient method to learn a semantically relevant feature space.
To efficiently bridge the modality gap and group visual and textual representation according to semantics, we propose to learn low-rank and compact latent feature space. For this purpose, propose a novel method named SespectationMMxxiiizze uses the same parametric model to abstract and reconstruct both textual and visual representations. In detail, we find a set of reconstruction bases for subspace representation by estimating the parameters in Expectation-Maxinn features are represented with the distributions over the same sets of hidden variables, which can preserve strong semantic relations across modalities. As shown in Figure 1b, our EMCL effectively learns a semantically related subspace which has smaller intra-class variance and larger inter-class variance compared to the previous contrastive learning method (Figure la). We further propose EMCL-Net based on the EMCL and apply EMCL-Net to the task of text- video retrieval. In particular, for better adapting the proposed EMCL into the downstream task, our EMCL-Net introduces a parameter initialization strategy of the EM algorithm. Experimental results on three text-video retrieval benchmark datasets, i.e., MSR-VTT [70], ActivityNet [27], and LSMDC [55], show the advantages of the proposed EMCL. The main contributions are as follows:
Contrastive Learning (EMCL), which
(EM) algorithm [16]. In the learned subspace, both video and text
We identify the intrinsic limitation of contrastive learning for text-video retrieval, i.e., the cross-modal representation bias could not be fully eliminated via standard contrastive learning approaches.
To alleviate such limitation, we reformulate the contrastive learning for video-and- lann-aae representations into expectation expectation-maxiizzalii maximization iteration manner and propose a plug-and

---

play feature projection module named Expectation ExpectaioonMMxximizztioon Contrastive Learning (EMCL), which learns the subspace that aims to become semantic-relevant representation.
Based on our EMCL, we further propose the EMCL-Nett, which introduces a parameter initialization strategy for the EM algorithm. Experiments show that our approach achieves state-of-the-ar results on three text-video retrieval datasets. More encouragingly, our method can be easily applied to boost the performances of existing approaches either as jointly training layer or an out-of-the-box inference module with no extra training.
Related Work
Cross-modal learning is widely studied in many areas, including cross-modal retrieval [76, 67, 24],
transfer learning [50, 45], domain adaptation [56, 34J, and captioning [39] in which different modali- ties/domains come from different distributions. The main challenge of cross-modal learning is to use given risionaand-laguaae pairs learn common representations shared between modalities [38]. Most existing works of text-video retrieval [9, 23, 22] map text and video to the same latent space, where the similarity between them can be directly calculated [20, 6, 63, 17, 65. 66, 7, 15, 73, 52]. In detail, CE [41] introduces a mixture-of-expert method that mixes the features of many different pre-training experts; MMT [21] shows multi-modal transformer with multiple self-attended layers for video embedding. Recently, contrastive learning methods, e.g., CLIP [53], show great success in advancing the stateoof-the-ar performances of cross-modal tasks [42]. Contrastive learning meth- ods [19, 5, 23, 44, 10, 40] try to learn data representations from positive and negative pairs, making the representations of positive pairs (usually data augmentations that retain semantic information) have high similarity, and negative pairs (different semantic examples) have low similarity, Inspired by the great success contrastive learning, we employ CLIP [53] for learning sideo-and-language representations. In this paper, a positive pair is matching text-video pair, and negative pair contains non-matching text and video. However, due to the multi- nature and apatial-tempora evolution of video [12, 37], it essential capture the most important semantic concept for the video. Meanwhile, the visualization in Figure 1 also shows that the feature space of video-and -nnguuggg representations has semantically irrelevant redundancy which leads to non-ideal contrast in the feature space. [35] analyzes the non-optimal feature space with modality gap impacts model's performance and fairness. Therefore, we propose to conduct contrastive learning in pnpectation-maximizatiin iteration manner to eliminate the non-optimal semantically redundant dimensions, acquiring compact representations.
3 Approach
In this section, we first introduce how to reformulate the Contrastive Learning into expectation- maximization itcration manner, i.c., Expectation-Maxxine Maximization Contrastive Learning (EMCL). Then, we introduce how to incorporate the proposed EMCL into the neural network for vidco-and- language representations, which are used to perform the text-video retrieval task.
3.1 Preliminaries
Expeetation-Maxiiizztoon Algorithm The Expeetation-Maxiinzaaioo (EM) algorithm [16] is an iterative optimization strategy, which was originally designed to solve the problem when data are missing in the process of parameter estimation. Brietly, given unobserved hidden vari- ables Z = {21, 22, ZN) } and observed data sets X = {21, x2,,..,NN' with N samples, the goal of EM algorithm is to estimate the maximum likelihood solution of model parameters i = arg max Ei=1 log Ex, p(x,, 0). In step E, the EM algorithm calculates the conditional probability expectation (x(z,) = p(zz 0). In the M step, the likelihood function is maximized to get the revised parameter = arg max EME. (((z))log log p(....0))
Q((z.)
Gaussian Mixture Model The Gaussian Mixture Model (GMM) [54] combines multiple sin- gle Gaussian models. Assuming that GMM consists of K Gaussians, given the input X =
{2), D2, ..., AN} with hidden variables Y, the probability of GMM is as follows:
N K
p(X (X,Y:p,E.x)
II
II
(Te. (ZalpR. Ex))
(1)
1=1
3

---

pnpectation-Maxiizaaaion Maximization Con- trastive Learning, with T iterations of routing. Typically,
Algorithm
The proposed Expectation-
Haddene
Stop Bases
Require: E IR P (Video features and text features), M E Rx (Initial Value Maintcnance)
1: Initialize À E R22x with splice of M copied 2B times
2: for routing iteration 1 = 2.,..... do
Y X X/a
Step
4: for = 1, D do
5: [y,1,,.. vs.k + Softmax (BU,,, My,K)) # Eq. (6)
Input
Bases
Initial value
Output
6:
end for
7: À < XY
Inibitize
subspace 2,,, do 9: Ar // ////5/111 Ui.k
Figure Module overview. For clarity, we show the case when the number of
# #Eq. (7)
10: end for
subspaces K 3. The input features come from samples, which are colored differently. EMCL projects input fea- tures into K = 3 subspaces to filter out the redundancy in features.
11: end for
12: Reconstruct X E 1R with X AYT
Eq. (8)
13: Update M with my = amng + (1 a) 7.
#Eq. (9)
14: return X R2BED, .M R
where N (*|UU, Ex) is the probability function of the kin Gaussian and is the prior probability. Through the E step of the EM algorithm, the estimated value of yn.k be obtained by:
NeN (xr Mix Ex)
Go.k
Σ k= TNN (Nnnuun Ex)
In the M step of the EM algorithm, the parameters of the Gaussian mixture model are updated as:
Σ N
Dn.k
Ex
>N Un - PT)"T((n 1ss))
Σ
tk
N
E Dn.k
ma Do.k
3.2 Expeetation-Maxxinzzaaoo Contrastive Learning (EMCL)
In this section, we introduce our EMCL in detail. Representing the features low-dimensional space fundamental method to eliminate the inherent redundancies in features, which is also the goal of our method. We leverage these compact features for contrastive learning to efficiently bridge the gap and improve their performance. The overview of our EMCL is shown in Figure 2 and Algorithm 1.
Mathematically, the subspace can be defined as follows: let X B the features, where the sample size, D is the dimension of the feature. Suppose that these features are distributed in many unknown linear subspaces, where the intrinsic dimensions of the linear subspaces are less than D. We define the union of K semantically related oowdiinesiional linear subspaces the semantic subspace. To this end, our method includes two parts: Firstly,, we use the EM algorithm [16] to find the bases of the K optimal subspaces, which is called "Maximum Probability Projection", Secondly, need to re-represent the features in these subspaces, which is called "Feature Reconstruction" Finally, in the "Contrastive Learning", the compact features re-represent used in contrastive learning to bridge the gap and improve their performance.
Maximum Probability Projection Given original video features C. B and original text features C, E BDD, we combine both video features and text features to be the input data X 22BxD, where 2,,, is the ju coding bit of the in sample. In method, each coding bit aa is assigned to a specific subspace, and meanwhile we introduce the hidden variable Y. where Vy.k represents whether the kn subspace is selected by Tı We take the base À of the subspace as the parameter of EM algorithm [16], where X E k228* represents the distribution of samples in K semantically related low-dimensional linear subspaces. Given input data E x2BxD and hidden variable Y BDx the complete likelihood of the input feature X can be expressed as:
(p(as.), Ai.k))
(4)

---

where P(*) is the probability function and p(*) is a kernel function. Here we assume that zi.j can be represented by shared Gaussian distributions like GMM [54]. For simplicity, we freeze Σ and π and only consider the mean X = la of the Gaussian model.
There many choices for the kernel function p(. f v). such linear kernel ( + c). polynomial kernel (a." y + e)d, Gaussian kernel exp(--)l@ / and on. We find that different kernel functions have slightly different impacts on the final results. For easy implementation, we use Gaussian kernel and rewrite it in form similar the attention model [2, 60]. The formula is:
2B Lin
Ai,k
p( ).,k) = exp
(5)
where a is a syper-parameter adjust the distribution, similar to the mean and covariance in the Gaussian distribution. In this paper, we define T.j to be the In column of X, and A,, to be the knn column of A.
Reviewing the EM algorithm [16], estimates the parameters of the model by continuously executing steps and M steps. In step E, it calculates the conditional probability expectation with current parameters. In step M, it maximizes the likelihood function to update the parameters.
In step E, referring to the Eq. (2), the estimated value of Mj,k can be obtained by:
pl A.A.)
Mj.A
(6)
Σ p( X.R))
In the M step, we update base A according Eq. (3), which is formulated
N Σー HiaTis
Σ 3i.k
By repeatedly iterating step E and step M, our algorithm forces this set of optimal subspaces to represent the original video features and original text features at the same time. After several iterations, we could keep the information that appears in both the video and the text to remove the redundancy.
Feature Reconstruction When reconstructing features, we use Y and À to linearly reconstruct the original features. Learning from the meaning of Y and À mentioned above, Yj.& indicates whether the km subspace is selected by æ. and λ represents the base of the kip subspace. The formula for feature reconstruction is:
Fi
(8)
As shown in Eq. (8), we only use the corresponding base λi,e (for all k E KJ) to reconstruct the dh sample Zz.j- Therefore, we estimate different Zij from K subspaces.
Contrastive Learning The above feature reconstruction method generates the low-rank and compact features stripped of redundancy. Therefore, we can leverage these compact features for contrastive learning to efficiently bridge the modality gap and improve task performance.
3.3 EMCL for Cross-Modal Learning
In this section, we discuss in detail how to incorporate the EMCL into the neural network for cross- modal downstream tasks. Targeting on the cross cross-modal tasks, we propose the ECML-Net, which introduces parameter initialization strategy for the EM algorithm that establish the connection between batches. At last, we take an important yet challenging cross- modal task, i.c., the text-video retrieval, as example to illustrate how employ our EMCL-Net to perform downstream tasks.
Initial Value Maintenance Considering that corpus of video-basse cross-modal tasks usually contains thousands of text-video pairs, it is often not enough to only establish connections between samples within single batches. Therefore, our module builds an inter-batch information transfer mechanism by maintaining separate initial value M E R In implementations, each time new batch of features is fed into the model, the EM algorithm needs to initialize the X. Rather than randomly initializing À for each batch, use M as the initial value of the A. For all i, k
5

---

initialize À with Ai, = my. We update M using an average moving method similar to the Batch
Normalization (BN) layer [25]. The moving method is formulated as:
2B
1
my = am, + (1 a)
X
2B
where E [0, 1] is the momentum. Similar the average moving method in the BN layer, we don't update M in the inference stage. Because the initial value is crucial to the stability of the EM algorithm [1, 33], we need to limit the value range of the initial value. In each iteration, we use the L2 normalization operation to limit the value range of X.
EMCL-Net For fair comparisons, we follow common practice [42, 13, 62] to extract the video representations of input videos and the language representations of input texts. In detail, for video representations, first extract the frames from the video clip the input sequence of video; Then we use ViT [18] to encode the frame sequence, by exploiting the transformer architecture to model the interactions between image patches; Followed by the CLIP [53], the output from the [class] token is used as the frame embedding: Finally, we aggregate the embedding of all frames and obtain the video representation C. IR BXD For text representation, directly use the text encoder of CLIP to acquire the text representation C₂ E R Bx
As plug- aadg--nd-plle play module, our proposed EMCL is inserted at the end of the video-text encoders. We concatenate video features C, and text features Ce. generating the input data of our method X = [Cu; Ct) 22BxD Following [14], we add reconstructed features fEMCI (X) by the EMCL
and original features X to obtain final text- representations X = |Ce: C.] = fEECLL (X) + X E 22 D where 3 is the scale factor. Here, 3 is the scale factor which is used to balance the original and new embeddings. By adjusting 3. we can add some flexibility in the reconstructed video-and-language representations for the robustness of the model.
Training Objective During the training stage, the EMCL module is trained with the neural network. We use M to initialize the X. Then the components Y and À are updated in unsupervised way Moreover, we update the initial value M using an average by iteration, shown in Algorithm moving method. The goal of the EMCL-Net is to map text and video into a joint representation space to measure the text-video similarity. Following common practice, we use cosine similarity s(t, v) = o7 cu
|.ll as the similarity measure between text and video v, where é represents the reconstructed text representation of text I; and è represents the reconstructed video representation of video v. We train our model with InfoNCE loss [59]:
B log 1=1 Σj exp(s(t(,,)))
exp(s(t,, u.)TT) + B
B exp(s(tr.u.// log B Σj exp(s(t,,,://)))
C=
(10)
i=1
where B is the batch size and 5 is the temperature hyper--aramenn parameter. This loss function maximizes the similarity of positive pairs s(t,, v)) and minimizes the similarity of negative pairs.
During the inference stage, given set of queries (text/video) and set of candidates (videos/texts), we use the trained M to initialize the X. Then Y and À can be updated in an unsupervised way by iteration. The goal of cross-modal retrieval task is to map the query and candidate into a joint video-and-languaal
representation space to measure the query-candidate similarity stt. v). In this way, we can output the similarity scores for all the input candidates, and take the candidates with the 1o5-1/5/10/0 similarity scores with input query as the final prediction.
4 Experiments
Datasets, Metrics and Implementation Details Datasets. We conduct the experiments on three popular text- video retrieval datasets, i.e., MSR-VTT [70], ActivityNet Captions [27], LSMDC [55], and follow common practice [42, 13, 62] to pre-process the datasets for fair comparison. In detail, MSR-VTT [70] contains 10,000 videos, each with 20 text descriptions; We follow the 1k-A split [41] with 9,000 videos for training and 1,000 for testing. ActivityNet Captions [27] contains 20,000 videos with multiple sentence descriptions; We report results on the "vall" split (10,009 training, 4,917 testing) as in [21]. LSMDC [55] contains 18,081 video clips from 202 movies; We follow the split of [21] with 1,000 vidcos for testing.

---

.e- Rose Model (Vido-
+MMCI (Vde:))idee)
.a. Rase Model (Vidio-Tiic)
EMCT. [Vidio-Tiest]
50
50-
16
68--
46.-
-4-
44-
0.3
1.5
2.0
The number subspaces
Iteration number I (b)
Hypennerrmmdunn perometor
(a)
(c)
Figure 3:
Effect of (a) the number of subspaces K; (b) the iteration number T; (c) the Hyper-
parameter a for our proposed EMCL module with inverted softmax [13,
Metrics. We choose standard retrieval metrics: Recall at K (R@K) and Median Rank (MdR) to evaluate the text-to-video and video-to-text retrieval performance.
Implementation Details. We utilize the CLIP (ViT-B/32) [53] equipped with Temporal Trans- former [42] as pre-trained Bi-Encoder (Base Model). Following previous works [42], the frame length and caption length are 12 and 32 for MSR-VTT and LSMDC, For ActivityNet, a long video retrieval dataset, we the frame length 64 and caption length 64. We follow training schedules from previous works [42, 13, 62]. Concretely, we use the Adam optimizer [26] with a linear warmup. The initial learning rate is 1 for text encoder and video encoder and le-4 for other modules. We set the temperature T = 0.01, a 1, the momentum a = 0.9, the number of iterations is set to 9 and the parameter K set to 32. The network is optimized with the batch size of 128 in epochs. During the training stage, the EMCL. module is trained with the neural network. Each time new batch of features is fed into the model, we use M to initialize the X. Then the components Y and are updated by iteration. Moreover, we update the initial value M using an average moving method. During the inference stage, given a set of queries (text/video) and set of candidates (videos/texts), we use the trained M to initialize the A. Then the components Y and À can be updated by iteration. When the EMCL module is incorporated into trained baselines as an out-of-the-box inference module with no extra training, À is randomly initialized. Then the components Y and À can be updated in an unsupervised way by iteration.
Comparisons to State-of-the-art Table 1 shows the results of our method on three text-video retrieval datasets. As we can see, our ECML-Net method consistently outperforms the recently proposed state-of-the-art methods on both text-to-video retrieval and video-to-text retrieval tasks across all datasets and metrics.
Since the input and output dimensions of the EMCL module are the same, it is model-agnostic and can be applied to features extracted from any language and video encoders. Therefore, we further equip our EMCL with three strong baseline models, i.e., MMT [21], CLIP4Clip [42]. DCR [62], and evaluate the performance of these models on the MSR- VTT datasets. Specifically, we insert the EMCL module at the end of the video-text encoders. Table 2 shows that our EMCL can be applied successfully boost all baselines either as jointly training layer or an out-of-the-box inference module with no extra training. Overall, our approach can boost the baselines with the most significant improvement up to 3.5% and 4.2% for text-to- text-to-video task and video-to-tes task in terms of R @ metric, respectively. The significant improvements demonstrate the generalization ability of EMCL.
Generalization Analysis
Comparisons to Other Baseline Methods We further compare our method with other baseline methods in Table 3. PCA [58] is popular method for finding salient features shared by the two modalities so that the modality gap can be reduced to a certain extent. "Transformer" represents using a shared transformer between two modalities, which can explicitly project the data from two modalities into a shared space. "Fully Connected Layers" represents using two shared fully connected layers between two modalities. "Sparse Autoencoders" is a common sparse autoencoder [46], which reduces the average response of the encoding layer and learns the compact representation. Different from these representative dimensionality reduction methods, the motivation of our method is to use contrastive learning to preserve video-text semantic relatedness. By maximizing the joint posterior probability of video and text, we find semantically related subspace for compact video-text representation. In contrast, PCA and its variants are feature dimension reduction methods, which aim to maximize the variance of projected data and cannot guarantee semantic relationships. Sparse autoencoder reduces the average response of the encoding layer for sparsity, which may potentially
7

---

Table 1: Comparisons current state-of-the th-t methods on the MSR- VTT [70], ActivityNet [27] and LSMDC [55] datasets. "1" denotes higher is better. "4" denotes lower is better. ff denotes employing inverted softmax [13, 4J.
(a) Retrieval performance on the Text-> Video task.
Pre-trained
VTT
ActivityNer [27]
LSMIX [55]
Methods
weights
RØ Re1R@55RR R 101 MdR, R@ R R 501 MdR
R@
R R 101 MdR.
JSFusion [75]
10.2
31.2
43.2
13.0
21.2
34.1
36.0
CE [-1]
GPT-1
20.9
48.8
62.4
6.0
18.2
47.7
91.4
6.0
11.2
26.9
34.8
25.3
MMT [21]
BERT-Base
24.6
54.0
67.1
4.0
22.7
54.2
93.2
5.0
13.2
29.2
38.8
21.0
Support-Set [49]
T5-Buse
27.4
56.3
67.7
3.0
26.8
58.1
93.5
3.0
T2VLAD [64]
BERT-Base
29.5
59.0
70.1
4.0
23.7
55.5
93.5
4.0
14.3
32.4
42.2
16.0
CLIP4Clip [42]
CLIP (ViT-B/32)
44.5
71.4
81.6
2.0
40.5
72.4
98.1
41.0
49.1
11.0
EMCL-Net (Ours)
CLIP (VIT-B/32)
46.8
73.1
83.1
2.0
41.2
72.7
98.1
42.4
50.9
10.0
EMCL-Net (Ours)
CLIP (VIT-B/32)
51.6
78.1
85.3
1.0
50.6
78.7
98.1
25.9
46.4
53.7
(b) Retrieval performance the Vidco->Text task.
Pre-trained
MSR-VTT [70]
ActivityNet [27]
LSMDC [55]
Methods
weights
R@ V@ItR@51H 5t R MdR. R@
R@ @ MdR
R R 101 MdR.
CL [41]
GPT-1
20.6
50.3
64.0
90.9
MMT [21]
BERT-Base
24.4
56.0
67.8
4.0
22.9
54.8
93.1
4.3
12.1
29.3
37,9
22.5
Support-Se[[9]
T5-Base
26.6
55.1
67.5
3.0
25.5
57.3
93.5
3.0
T2VLAD [64]
BERT-Base
31.8
60.0
71.1
3.0
24.1
56.6
94.1
4,0
14.2
33.5
41.7
17.0
CLIP4Clip [42]
CLIP (VIT-B/32)
42.7
70.9
80.6
2.0
42.5
74.1
98.1
2.0
20.8
39.0
48.6
12.0
EMCL-Net (Ours)
CLIP (VIT-B/32)
46.5
73.5
83.5
2.0
42.7
74.0
98.3
2.0
22.2
40.6
49.2
12.0
EMCL--Net (Ours)
CLIP (VIT-B/32)
51.8
80.2
88.0
1.0
50.6
78.9
98.4
1.0
26.7
44.7
54.4
8.0
Table Generalization analysis of our EMCL on the MSR-VTT dataset [70]. We equip our EMCL with three strong contrastive learning baselines. denotes our own re-implementation of baselines; I denotes the EMCL trained jointly with the baselines from scratch; 8 denotes the EMCL is incorporated into trained baselines an out-of-the inference module with no extra training. We conducted runs with different seeds for all experiments, the t-tests indicate that p < 0.01. The (+Number) denotes the absolute improvements.
Text->Video R@5t
Video->Text R@5t
Methods
R@
RØ R@10f
R@11
R@101
MMT [21]*
25.9
54.8
68.5
26.0
58.2
69.3
+ EMCL (Ours)
26.2 (+0.3)
57.2 (42.4)
70.8 (+2.3)
27.2 (+1.2)
59,8 (+1.6)
(+1.1)
+ EMCL (Ours)
27.1 (+1.2)
57.6 (+2.8)
70.5 (+2.0)
27.8 (+1.8)
59.3
69.8 (+0.5)
CLIP4Clip [42]
43.4
70.4
78.5
42.4
68.6
79.2
FMCL (
44.6 (+1.2)
71.4 (+1.0)
79.5 (+1.0)
45.0 (+2.6)
71.2 (+2.6)
79.2 (+0.0)
EMCL (Ours)
46.9 (+3.5)
72.5 (+2.1)
81.8 (+3.3)
46.6 (+4.2)
73.3
82.3 (+3.1)
DCR [62]
46.8
72.6
82.6
72.1
82.1
+ EMCL (Ours)
47.5 (+0.7)
74.9 (42.3)
83.8 (+1.2)
45.9 (+0.1)
73.6 (+1.5)
83.5 (+1.4)
+ EMCL (Ours)
48.0 (+1.2)
73.7 (+1.1)
83.0 (+0.4)
45.9 (+0.1)
73.5 (+1.4)
83.5 (+1.4)
discard semantic information. addition, our method has linear complexity O(BDK) and does not require additional training. In contrast, the complexity of PCA is (D3) (D > B and D K), and the autoencoder requires additional training.
A Analysis Effeett of the parameter initialization strategy. As shown in Table 4, with or without parameter initialization strategy, our method can successfully promote the base model. is worth noting that the EM algorithm [16] is sensitive to initial values 33]. In other words, the convergence of EM algorithm depends mainly on initial parameters. Random initialization leads to large fluctuations in convergence results. This fatal flaw limits the performance of our module. Fortunately, with the proposed parameter initialization strategy, our EMCL method surpasses the base model by large margin with 3.5% R@1 I and 1.4% R@ I on the text-to-video and video-to-text tasks, respectively, proving the effectiveness our parameter initialization strategy used for EMCL-Net.
Effect of the number of subspaces. In Figure 3a, we show the effect of the number of subspaces On the one hand, we find that fewer subspaces mean fewer semantic centers, which limits our module's ss ability to reconstruct the features. On the other hand, larger number of subspaces requires more training data, which increases the cost of our model learning. We set the center size K = to achieve the best performance in practice.
8

---

Table Comparisons other baseline methods on MSR-VTT dataset [70]. We perform the anal- the sample
Table 4: Effect of the Parameter Initialization Strategy in our EMCL. tf denotes employing inverted softmax [13, 4].
ysis on the Text->Video task. size. D is the dimension of the original feature. K' the number of subspaces. 11 denotes em-
Text-> Video
Methods
Initialization
R@if
R@51
R 10f MdR
ploying inverted softmax [13, 4].
Base Model
42.4
70.8
80.6
2.0
+ +EMCI
43.3
71.8
81.6
2.0
Complexity
Mcthods
R@
EMCL
46.8
73.1
83.1
2.0
Time
Space
EMCL
51.6
78.1
85.3
1.0
Base Model
42.4
Transformer
PCA
O(B2D) O(D²)
O(D*) O(D*)
36.0
41.3
Methods
Initialization
R@ 11
R@ R@ 101 MdRL
Video->
Fully Connected Layers
(BBK)) O(DK)
42.1
Base Model
43.2
70.0
81.1
2.0
+ Sparse Autoencuders
((DDK) O(DK)
43.8
+MCC
45.1
72.8
82.3
2.0
EMCL (Ours)
OBBKK) O(DK)
46.8
EMCL
46.5
73.5
83.5
2.0
+ EMCL (Qurs)
((DKK) O(DK)
51.6
EMCL
51.8
80.2
88.0
1.0
Table 5: Comparisons with state-of-the-art methods for video captioning on MSR-VTT dataset [70]. We report BLEU-
Table Comparisons with state- of-the-art methods for video ques- tion answering on MSRVTT-QA dataset [69]]
4 [48]. METEOR [3], ROUGE-L [36]. and CIDEr [61] met-
rics. All methods only use image modality input.
Methods
BLEU-4t
METEOR
ROUGE-LÍ
CIDEr
Methods
Accuracy
STG-KD [47]
40.5
28.3
60.9
47.1
ClipBERT [28]
37.4
ORG-TRL [79]
43.6
28.8
62.1
$U.9
VGT
39.7
MGCMP [8]
41.7
28.9
62.1
51.4
VQA-T [71]
41.5
OpenBook [78]
42.8
29.3
61.7
52.9
SiaSamRea [74]
41.6
ARB-ACL [32]
42.6
28.9
61.5
$1.3
MERLOT [77]
43.1
DCD [?2]
43.4
29.6
61.8
52.8
Ca-Tokenization [S1]
45.7
modelcap
45.2
29.8
63.0
54.6
Base modelge
45.0
EMCL (Ours)
45.3
30.2
63.2
54.6
EMCL (Ours)
45.8
Efffect of the iteration number. In Figure 3b, we show the influence of EM iteration number T. Overall performance improves slightly before leveling off. We find that the algorithm has converged when the number of iterations is 9, so we set the number of iterations to 9 as default.
Soyper-paramett selection. The parameter a is hyper-parameter to adjust the distribution (Eq. 5), similar to the mean and covariance in the Gaussian distribution. We evaluate the scale range setting [0.3, 2.0] shown in Figure 3c. We find that R @ is improved from 43.8% to 51.6% when σ = 0.7 and saturated with o = 1. As a result, we adopt = to achieve the best performance.
Generalize to other tasks Video captioning. The purpose of video captioning is to describe the content of the video in fluent sentences. "Base modeloap" uses CLIP [53] to extract video features and trained with cross-entropy loss. To generate higher-quality sentences, video features and ground-truth text features. As shown in Table 5, using EMCL brings significant improvements caption quality, e.g., gaining relative improvement of 0.4% METEOR.
apply EMCL between
Video question answering. Visual question answering requires the model predict an answer using visual information [30, 29]. We use the target vocabulary for MSRVTT-QA dataset [69], and train fully connected layer on top of the final language features classify the answer. "Base modelga" uses CLIP [53],a mansformer-basee [18, 31] visual-language pre-training model, to extract video- and-language features and trained with cross-entropy loss. To learn compact video-and- languag representations, we apply EMCL. between video features and question features. Table 6 shows that EMCL can be applied to boost video question answering successfully and boost the bascline with an improvement up to 0.8%.
Qualitative Analysis Analysis of EMCL iterative process. As shown in Figure 4a, redueing the intra-class variance makes videos and texts belonging to the same semantic class gather, and increasing the inter-class variance makes those belonging different semantic classes separate from each other. The two conclusions we get from Figure 4b and Figure 4c are follows. (1) EMCL module reduces the intra-class variance the same semantic classes and increases the inter-class variance of different semantic classes. (2) The number K of subspaces has great influence on the EMCL module. The effect of the FMCL module limited if K is too small, while the intra-class variance increases if K is too large.

---

Video
Text
e+e- Different Classes
K=8
K=16
K=32
K=64
K=128
0.024
1.6
0.022
1.5
1.4
0.020
0.018
Intrr-class
1:2
1.3
0.016
1.0
Reduce the intra class
0.014
Incresse inter class variance
The iteration number
(a)
(b)
(G)
Figure 4: Visualization of EMCL iterative process. (a) shows the influence of reducing the intra-class
variance and increasing the inter-class variance on the feature space. (b) shows the relationship
between the intra-class variance and the iteration number on the MSR- VTT dataset. (c) shows the
relationship between the inter-class variance and the iteration number on the MSR-VTT dataset.
The original data
X-Y view
X-Y view
X-Y
view
view
X-Y-Z view
Y-Z view
X-Y-Z view
Y-Z view
The oulputs of EMCL
X-Y view
X-Y view
X-Y view
vie-
Y-Z view
X-Y-Z view
v-Z.view
X-Y-Z
Y-Z view
Figure 5: Visualization of our EMCL. We randomly generate a set of 3D data containing types of samples corresponding to 3 different text-video semantic centers. The figure the top is the visualization of the original data, and the figure on the bottom is the visualization of EMCL's outputs.
Visualization. To better understand EMCL,, we provide visualization form of both the original representations and the filtered representations. We notice that the EMCL module eliminates the redundant dimensions, and the features reconstructed by our module are very compact in the feature space. As shown in Figure 5, even though features have intense noise in the redundancy dimension. the EMCL module still learns semantic information. The EMCL module forces the differences between classes in subspace to be more obvious than original, which is helpful for the contrastive learning to learn the semantic centers shared by videos and texts.
5 Conclusion
In this paper, we studied the intrinsic limitation of classic contrastive learning for text-video retrieval. We found that the contrastive method in the entire representation space fails to preserve motrllmodal semantic relatedness, which makes the features gather separate in the subspace which is irrelevant semantics. mitigate this effect, we propose to directly learn subspace that is related to shared semantics and do contrastive learning in it. By learning the subspaces related to semantics, we are able to learn the common semantic center of video and text the semantic subspace. Further, is worth noting that method could be applied for other contrastive learning tasks, which include similar samples containing redundant dimensions or with limited number of negative samples.
Acknowledgements This work is supported by the Nature Science Foundation of China (No. 61972217, 62081 Foundation NNo.2019B151112000 2020B111 1340056), Also, this work supported part by the National Institute for Health Research (NIHR) Oxford Biomedical Research Centre; an InnoHK Project the Hong Kong Centre for Cerebro-cardiovasculaa Health Engineering; and the Pandemic Sciences Institute, University of Oxford, Oxford, UK.
2208136011 1601122, 62006133, 32071459), Guangdong Basic and Applied Basic Research and Guangdong Science and Technology Department (No.
10

---

References
[1] Maryam Abdolali and Nicolas Gillis. Beyond linear subspace clustering: A comparative study of nonlinear manifold clustering algorithms. Computer Science Review, 42: 10043503, 2021.
[2] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR 2015: International Conference on Learning Representations, 2015.
[3] Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings the ACL workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65-72,
2005.
[4] Simion-Vias Bogolin, loana Croitoru, Hailin Jin, Yang Liu, and Samuel Albanie. Cross modal retrieval with querybank normalisation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5194-5205, 2022.
[5] Meng Cao, Tianyu Yang, Junwu Weng, Can Zhang, Jue Wang, and Yuexian Zou. VTP: Video-text pre-trane vre-training for temporal localization. arXiv preprint arXiv:2207.10362 2022.
[6] Shugiang Cao, Bairui Wang, Wei Zhang, and Lin Ma. Visual consensus modeling for video-text retrieval. Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 167-175, Jun. 2022.
[7] Jiacheng Chen, Hexiang Hu, Hao Wu, Yuning Jiang, and Changhu Wang. Learning the best pooling strategy for visual semantic embedding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15789-15798, 2021.
[8] Shaoxiang Chen and Yu- Yu-Gang Jiang. Motion guided region message passing for video cap- tioning. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages October 2021.
[9] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML 2020: 37th International Conference on Machine Learning, volume pages 1597-1607, 2020.
[10] Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He. Improved baselines with momen- tum contrastive learning. arXiv preprint arXiv:2003.04297, 2020.
[11] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. arXiv preprint arXiv:2/04. 02055. 2021.
[12] Zhenfang Chen, Lin Ma, Wenhan Luo, and Kwan- Kenneth Wong. Weakly-superviet spatio-temporally grounding natural sentence in video. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1884-1894, 2019.
[13] Xing Cheng, Hezheng Lin, Xiangyu Wu, Fan Yang, and Dong Shen. Improving video- text retrieval by multi-stream corpus alignment and dual softmax loss. arXiv preprint arXiv:2109.04200 2021.
[14] Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio De Rezende, Yannis Kalantidis, and Diane Larlus. Probabilistic embeddings for cross-modal retrieval. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8415-8424 2021.
[15] Ioana Croitoru, Simion- Vlad Bogolin, Marius Leordeanu, Hailin Jin, Andrew Zisserman, Samuel Albanie, and Yang Liu. Teachtext: Crossmodal generalized distillation for text-video retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1 11583-11599 2021.
[16] Arthur P. Dempster, Nan M. Laird, and Donald B. Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society series b-methodological 1--2), 1977.
11

---

[17] Jianfeng Dong, Xirong Li. Chaoxi Xu, Xun Yang, Gang Yang. Xun Wang, and Meng Wang. Dual encoding for video retrieval by text. IEEE Transactions on Pattern Analysis and Machine Intelligence, (48)):40654088 2022.
[18] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR 2021: International Conference on Learning Representations, 2021.
[19] Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discrimi- native unsupervised feature learning with convolutional neural networks. In Advances in Neural Information Processing Systems 27, volume 27, pages 766-774, 2014.
[20] Valentin Gabeur, Arsha Nagrani, Chen Sun, Karteek Alahari, and Cordelia Schmid. Masking modalities for cross-modal video retrieval. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1766-1775, 2022.
[21] Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. Multi-modal transformer for video retrieval. In Proceedings of the European Conference on Computer Vision, volume 12349, pages 214-229,10 2020.
[22] Satya Krishna Gorti, Noël Vouitsis, Junwei Ma, Keyvan Golestan, Maksims Volkovs, Animesh Garg, and Guangwei Yu. X-pool: Cross- language-video attention for text-video retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 5006-5015, 2022.
[23] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9733 2020.
[24] R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildee Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR 2018: International Conference on Learning Represen- lations, 2018.
[25] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings The 32nd International Conference on Machine Learning, volume 1, pages 448-456 2015.
[26] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR 2014: International Conference on Learning Representutions, 2014.
[27] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense- captioning events in videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 706-715, 2017.
[28] Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L. Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for lann-uaed learning via sparse sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7331-7341, 2021.
[29] Hao Li, Jinfa Huang, Peng Jin, Guoli Song, Qi Wu, and Jie Chen. Toward 3d spatial reasoning for human-like text-based visual question answering. arXiv preprint arXiv:2209, 10326, 2022.
[30] Hao Li, Xu Li, Belhal Karimi, Jie Chen, and Mingming Sun. Joint learning of object graph and relation graph for visual question answering. arXiv preprint arXiv:2205.041888 2022.
[31] Kehan Li, Runyi Yu, Zhennan Wang, Li Yuan, Guoli Song, and Jie Chen. Locality guidance for improving vision transformers on tiny datasets. arXiv preprint arXiv:2207.10026 2022.
[32] Shanhao Li, Bang Yang, and Yuexian Zou. Adaptive curriculum learning for video captioning. IEEE Access,
2022.
12

---

[33] Xia Li, Zhisheng Zhong, Jianlong Wu, Yibo Yang, Zhouchen Lin, and Hong Liu. Expectation maximization attention networks for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 91 9166-9175 2019.
[34] Jian Liang, Kaixiong Gong, Shuang Li, Chi Harold Liu, Han Li, Di Liu, Guoren Wang, et al. Pareto domain adaptation. In Proceedings of the 35th International Conference on Neural Information Processing Systems, volume 34, pages 12917-12929, 2021.
[35] Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. arXiv preprint arXiv: 02055, 2022.
[36] Chin- Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Proceedings of Workshop on Text Summarization of ACL, Spain, 2004.
[37] An-An Liu, Yu- Ting Su, Wei-Z Vei-Zhi Nie, and Mohan Kankanhalli. Hierarchical clustering multi- task learning for joint human action grouping and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(1):102-114, 2017.
[38] Fenglin Liu, Yuanxin Liu, Xuancheng Ren, Xiaodong He, and Xu Sun. Aligning visual regions and textual concepts for semantic -rrannic-grouned image representations. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 0847-6857, 2019.
[39] Fenglin Liu, Xuancheng Ren, Xian Wu, Shen Ge, Wei Fan, Yuexian Zou, and Xu Sun. Prophet attention: Predicting attention with future attention. In Proceedings of the 34th International Conference Neural Information Processing Systems, 2020.
[40] Fenglin Liu, Changchang Yin, Xian Wu, Shen Ge, Ping Zhang, and Xu Sun. Contrastive attention for automatic chest x-ray report generation. In ACL/JCNNPP (Findings), pages 269- 280, 2021.
[41] Yang Liu, Samuel Albanie, Arsha Nagrani, and Andrew Zisserman. Use what you have: Video retrieval using representations from collaborative experts. arXiv preprint arXiv: /907. 13487, 2019.
[42] Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval. arXiv preprint 204.00000 2021.
[43] Laurens Van Der Maaten. Accelerating t-sne using tree-based algorithms. Journal of Machine Learning Research, 15(1):322133255, 2014.
[44] Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant rep- resentations. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 6707-6717, 2020.
[45] Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in trans- fer learning? In Proceedings of the 34th International Conference on Neural Information Processing Systems, volume 33, pages 512-523, 2020.
[46] Andrew Ng. Sparse autoencoder. CS294A Lecture notes, 72(2011):1-19, 2011.
[47] Boxiao Pan, Haoye Cai, De-An Huang, Kuan-Hui Lee, Adrien Gaidon, Ehsan Adeli, and Juan Carlos Niebles. Spatio-temporal graph for video captioning with knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10870-10879, 2020.
[48] Kishore Papineni, Salim Roukas, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 31 311-318, 2002.
[49] Mandela Patrick, Po-Yao Huang, Yuki Asano, Florian Metze, Alexander G Hauptmann, Joao F. Henriques, and Andrea Vedaldi. Support-set bottlenecks for video-text representation learning. In ICLR 2021: International Conference on Learning Representations, 2021.
13

---

[50] Trung Phung, Trung Le, Tung-Long Vuong, Toan Tran, Anh Tran, Hung Bui, and Dinh Phung. On learning Nomain-invariann representations for transfer learning with multiple sources. In Proceedings of the 35th International Conference on Neural Information Processing Systems, volume 34, pages 27720-27733, 2021.
[51] AJ Piergiovanni, Kairo Morton, Weicheng Kuo, Michael S Ryoo, and Anelia Angelova. Video question answering with iterative video-text co-tokenization. arXiv preprint arXiv:2208.00934 2022.
[52] Mengshi Qi, Jie Qin, Yi Yang, Yunhong Wang, and Jiebo Luo. Semantics-aware spatial- temporal binaries for cross-modal video retrieval. IEEE Transactions on Image Processing. 30:2989-300 3004, 2021.
[53] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML 2021: 38th International Conference Machine Learning, pages
2021.
[54] Sylvia. Richardson and Peter J. Green. On bayesian analysis of mixtures with an unknown number of components (with discussion). Journal of The Royal Statistical Society Series B-statistical Methodology, 59(4):731-792 1997.
[55] Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schiele. dataset for movie de- scription. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 3202-3212, 2015.
[56] Petar Stojanov, Zijian Li, Mingming Gong, Ruichu Cai, Jaime Carbonell, and Kun Zhang. Domain adaptation with invariant representation learning: What transformations to learn? In Proceedings of the 35th International Conference on Neural Information Processing Systems, volume 34, pages 24791-2499 2021.
[57] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7464-7473, 2019.
[58] Michael E Tipping and Christopher M Bishop. Probabilistic principal component analysis. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 61):))) 1-622, 1999.
[59] Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv e-prints, pages arXiv-1807, 2018
[60] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference Neural Information Processing Systems, volume 30, pages 5998- 6008, 2017.
[61] Ramakrishna Vedantam, Lawrence Zitnick, and Devi Parikh. Cider: Consensus-base image description evaluation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4566 4575, 2015.
[62] Qiang Wang, Yanhao Zhang, Yun Zheng, Pan Pan, and Xian-Sheng Hua. Disentangled repre- sentation learning for text-video retrieval. arXiv preprint 1rXiv:2203.07111, 2022.
[63] Wei Wang, Junyu Gao, Xiaoshan Yang, and Changsheng Xu. Many hands make light work: Transferring knowledge from auxiliary tasks for video-text retrieval. IEEE Transactions on Multimedia, 2022.
[64] Xiaohan Wang, Linchao Zhu, and Yi Yang. T2VLAD: Global-local sequence alignment for text-video retrieval. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 5079-5088, 2021.
14

---

[65] Jiwei Wei, Yang Yang, Xing Xu. Xiaofeng Zhu, and Heng Tao Shen. Universal weighting metric learning for cross-modal retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[66] Michael Wray, Hazel Doughty, and Dima Damen. On semantic similarity in video retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3650-3660, 2021.
[67] Zhirong Wu, Yuanjun Xiong, Stella Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance-level discrimination. arXiv preprint
2018.
[68] Junbin Xiao, Pan Zhou, Tat-Seng Chua, and Shuicheng Yan. Video graph transformer for video question answering. arXiv preprint arXIv:220.005342, 2022.
[69] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference Multimedia, pages 1645-1653, 2017.
[70] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Mst-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 5288-5296, 2016.
[71] Antoine Yang, Antoine Miech, Josef Sivie, Ivan Laptev, and Cordelia Schmid. Just ask: Learning to answer questions from millions of narrated videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1686-1697, 2021.
[72] Bang Yang and Yuexian Zou. Clip meets video captioners: Attribute aware representation learning promotes accurate captioning. arXiv preprint arXiv:2111.15162, 2021.
[73] Jianwei Yang, Yonatan Bisk, and Jianfeng Gao. Taco: Token-aware cascade contrastive learning for video-text alignment. Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 1562-11572, 2021.
[74] Weijiang Yu, Haoteng Zheng, Mengfei Li, Lei Ji, Lijun Wu, Nong Xiao, and Nan Duan. Learning from inside: Self-driven siamese sampling and reasoning for video question answering. In Proceedings of the 35th International Conference Neural Information Processing Systems, volume 34, pages 26462-26474 2021.
[75] Youngjae Yu, Jongseok Kim, and Gunhee Kim. A joint sequence fusion model for video question answering and retrieval. In Proceedings of the European Conference Computer Vision, pages 471-487 2018.
[76] Youngjae Yu, Hyungjin Ko, Jongwook Choi, and Gunhee Kim. End-to-end concept word detection for video captioning, retrieval, and question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3261-3269, 2017.
[77] Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. In Proceedings of the 35th International Conference on Neural Information Processing Systems, volume 34, pages 23634-23655 2021.
[78] Ziqi Zhang, Zhongang Qi, Chunfeng Yuan, Ying Shan, Bing Li, Ying Deng, and Weiming Hu. Open-book video captioning with retrieve-copy- generate network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9837-9846, 2021.
[79] Ziqi Zhang, Yaya Shi, Chunfeng Yuan, Bing Li, Peijin Wang, Weiming Hu, and Zheng-Jun Zha. Object relational graph with teacher-recommendeer learning for video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15278-13288, 2020.
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 4 and Appendix A.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Appendix A.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 3.1 and Section 3.2.
(b) Did you include complete proofs all theoretical results? [Yes] See Section 3.1 and Section 3.2.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Section 4 and Appendix B.
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix B.
4. If you using existing assets (e.g., code, data, models) or curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent obtained from people whose data you's 're using/curating? [Yes]
(c) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
16


Reviews:
[{"review_id": "XAtgMmjvCN", "reviewer": "Reviewer_jsU2", "summary": "Video-and-Language representation learning methods represent the data from both modalities into a shared latent space. However due to separate independent encoders for the two modalities, there is a gap between the representations of the two modalities. To bridge the modality gap, this paper proposes to learn a shared latent space by representing the data from both the modalities into the same latent space defined by basis vectors that are learned using an EM style algorithm. The paper evaluates the proposed approach and the baselines on three text-video retrieval benchmark datasets.", "questions": "Questions:\n\n1. As pointed out in Weakness 1, the modality gap phenomenon is arising because the representation heads/encoders are separate for the modality in the CLIP4Clip approach. This results in two independent spaces and the contrastive loss tries to reduce the distance for a given pair of text and video. Even though the contrastive loss reduces the distances for many such pairs of text and video, it still lacks in reducing the distance between the two latent spaces. However, passing the representations from the two encoders through a common set of nonlinear layers can explicitly project the data from two modalities into a shared space. Applying the contrastive loss on top of it can make the embeddings of the text-image pair to be closer to each other in this shared space while explicitly forcing the two modalities to share the common latent space. This simplistic approach is analogous to the proposed approach and thus should be compared as a baseline.\n2. Lines 186-191, the paper describes the plug-n-play module which misses a key detail. Without training EMCL, how are the components Y and \\lambda learned? I think the EMCL model needs to be trained to get the values of these variables. The results in the section “Generalization Analysis” also don’t shed any light into the method of plugging in the EMCL module.\n3. Continuing on Question 1, in lines 186-191 the alternatively be implemented using a Residual layer that preserves the original embeddings along with the new embeddings from the shared space.\nIn the section “Comparisons to other Dimensionality Reduction Methods”, the experiment setup is not clearly explained. What is “Linear”?", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper points out a very important phenomenon of the modality gap when representing the data into a shared latent space.\n2. The paper proposes a way to represent the modalities into a shared latent space of learned basis vectors.\n3. The proposed method is a plug n play framework that can be applied to any representation learning algorithm\n4. The results show an improvement in performance when the proposed method is added to standard representation learning methods.\n\nWeakness:\n1. The paper adopts a very complicated approach to address the phenomenon of modality gap, by selecting a set of basis vectors and then representing the data using a linear combination of those vectors. A simplistic baseline can be passing the representations through a common feed-forward network which explicitly forces the modality to be represented in the same space. The comparison to such a simplistic baseline is missing.\n2. The proposed approach has a few critical hyperparameters (number of basis vectors, number of EM steps) which if not properly set can degrade the performance.\n3. The writing of the method description lacks clarity with some aspects of the proposed approach not explained clearly.\nThe intuition behind selecting the baselines are not explained. Moreover, the baselines are not explained at all. Mere mentioning the names of the baselines in the Table significantly reduces the readability of the paper.\n\nIn summary, the paper points out to an important phenomenon, although the proposed method with extra hyperparameters need to be compared to relevant simple baselines. In addition, the writing needs to be greatly improved by including proper explanation of baselines, the intuition behind choosing them and clear explanations for all the aspects of the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WuDrI-nePsG", "reviewer": "Reviewer_f2Q1", "summary": "This paper addresses the task of bidirectional text-to-video retrieval and the limitation of contrastive learning that is commonly used to learn discriminative visual and text features in a shared latent space. The authors qualitatively show that state-of-the-art approaches still do not learn a strong alignment between the visual and text modalities and propose a reformulation of the standard contrastive learning as an expectation-maximization project to bridge the gap between both modalities. Finally, the paper introduces a parameter initialization strategy for the proposed expectation-maximization approach, which results in significant improvements on the task across three commonly-used datasets.", "questions": "It is not very clear from the writing why learning low rank representations will lead to a stronger alignment between both modalities in the shared embedding space. This might be out of scope, but a proof may be helpful in conveying this intuition.", "limitations": "1) Line 128: y_{j,k} should be y_{i,j,k} instead? The i-th index is not used in this notation.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1) The paper is largely well-written and easy to understand. In particular, the mathematical definitions that are provided are very helpful for understanding the proposed approach.\n\n2) A known limitation of contrastive learning is that it requires a large number of negative samples during training to learn discriminative features that are effective for bidirectional text-to-video retrieval. Furthermore, these negative samples have to be hard negatives, which can be very difficult to discover without fine-grained and costly annotations. The proposed expectation-maximization approach helps to mitigate this issue to a large degree by learning a low-rank shared latent feature space, which removes redundancy. Interestingly, this results in a much stronger alignment between the learnt visual and text representations. As such, this is a significant contribution to this task and multimodal representation in general. While the expectation-maximization algorithm is not new, its application to the problem of learning more compact and effective multimodal representations is original.\n\n3) The authors also conducted extensive ablation experiments over the variables including the number of subspaces as well as number of iterations used in the expectation-maximization algorithm. These are especially helpful for the reader to quickly determine the contributions of each component of the proposed approach.\n\nWeaknesses:\n\n1) Some of the figures are pretty small and the numbers can be a little hard to read, even in the magnified state.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Mtz-1dg7HX", "reviewer": "Reviewer_yWjj", "summary": "The paper presents an approach that improves cross-modal contrastive learning by jointly estimating more compact sub-spaces for both modalities using the well-known EM algorithm. The authors show improvements on three video-text retrieval tasks, and multiple baseline models, to which the proposed sub-space embedding method is being applied. The proposed method works by linearly interpolating the original features with some that have been reconstructed in a shared space, generated using EM estimation with radial Gaussians. This improves similarity between the two spaces, and AFAIK represents a novel idea. For efficient implementation, the authors transfer GMM initializations across mini-batches, which further improves performance.\n", "questions": "I am wondering if and how diagonal or full covariance Gaussians would affect the results? Would the results improve further, or would they deteriorate because the subspace fits better to the data, and therefore less dimensionality reduction and mapping is taking place?\nHave you experimented with not just using a single codebook in a video, but multiple codebooks, so that e.g. one could cover indoor and one outdoor, one does cartoons, one does real footage, etc?\nAre there any architectures on which the method does not improve? I.e. is there something to learn about which methods does EMCL improve upon, and which ones it does not?\n", "limitations": "The authors work in an established framework, therefore I think there are no specific limitations to discuss.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThe paper presents a wealth of results and meaningful ablations. Authors include the code in the submission, and commit to making it available (please provide a clear license) upon acceptance of the paper.\nIn addition to text-video contrastive learning, the authors apply their algorithm to other tasks such as visual representation learning and w shot classification, and observe similar improvement.s\n\nWeaknesses\n\nThe paper does not discuss computational requirements, other than stating that the algorithm has constant complexity.\nThe writing is sometimes a bit awkward, as in \"our EMCL\", but generally accessible.\nThe paper does not demonstrate improvements on the latest SOTA (but given the number of different architectures presented, it seems likely it would give improvements)\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GChm--Hc3TX", "reviewer": "Reviewer_widu", "summary": "The paper proposes Expectation-Maximization Contrastive Learning (EMCL) to reduce the modality gap exists in multi-modal representations. The EM operation is applied offline before contrastive learning, making EM have low computation cost and a plug-and-play module. The visualization demonstrates this algorithm's ability to close the modality gap, and the video-text retrieval studies indicate how effective it is.", "questions": "1. Is the ECML module trained with the neural network or frozen during contrastive learning? My current understanding is that ECML determines bases offline before the contrastive learning, but lines 69 - 70 say it can also be a jointly training layer, which makes me a bit confused, so it would be great if you could elaborate on this.\n2. Follow the previous question, if the module is always frozen, does it mean that the vision and language encoders can not be trained when fine-tuning? If yes, this might limit the usage of ECML since end-to-end training of the whole model is the mainstream approach for now, and it usually leads to better performance.\n3. If the EM part can train with contrastive learning, then what are the computational overheads of it (how much extra training time is needed)?\n4. I understand that the modality gap seems to be an issue in current multi-modal models. However, I am not sure forcing the data pair to have close representation is always a good thing, especially when the data has complex and mixed information. For example, videos contain rich information, even though some datasets have text labels to describe each video, the text labels may not cover all contents of the video. Then we will lose much information if we limit video representations close to their text label representations, and this might be harmful in some tasks. What is your thought on this?\n5. Why EM can group the representation of videos and their labels into the same clusters? From the paper, I figure EM is used as a dimension reduction method, which can remove some redundant information from the data, but I do not get why removing redundancy leads to reducing the modality gap and learning a strong shared representation space. I think they are two things. And also, does learning the compact representation guarantee reducing the modality gap?\n6. What are the model sizes of the models and pre-trained weights they use in Table 1? I think the information is necessary for a fair comparison.", "limitations": "The paper mentions that the initialization and hyper-parameter choices highly affect EM's performance. I also think the computational overhead (if EM trains with contrastive learning) and the strong assumption of pair data having similar representation might limit the proposed method to extend to other applications.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. Pros\n* The paper introduces an effective EM module that can help the model learn compact representations and reduce the modality gap.\n* The EM module can easily incorporate into existing contrastive learning frameworks.\n* The experiment session shows strong results, and the ablation studies discuss the design choices of the proposed method completely.\n\n2. Cons\n* I am a bit concern about the motivation of \"reducing modality gap\" (See Questions 4 and 5).\n* Some literature might be missing in related works. I think the problem that this paper is interested in (the modality gap) is also highly related to transfer learning and domain adaptation. However, there is no such discussion in the related works.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept