Video-Text Retrieval by Supervised Sparse Multi-Grained Learning
Authors: Yimu Wang
Peng Shi
Venue: EMNLP 2023 Findings (2023)
Keywords: Video-Text Retrieval
Multimodal Learning
TL;DR: We introduce S3MA for video-text retrieval, which utilizes a shared sparse space to align video and text and incorporates frame representations for multi-grained alignment with superior performance compared to existing methods.

Abstract: While recent progress in video-text retrieval has been advanced by the exploration of better representation learning, in this paper, we present a novel multi-grained sparse learning framework, S3MA, to learn an aligned sparse space shared between the video and the text for video-text retrieval. The shared sparse space is initialized with a finite number of sparse concepts, each of which refers to a number of words. With the text data at hand, we learn and update the shared sparse space in a supervised manner using the proposed similarity and alignment losses. Moreover, to enable multi-grained alignment, we incorporate frame representations for better modeling the video modality and calculating fine-grained and coarse-grained similarities. Benefiting from the learned shared sparse space and multi-grained similarities, extensive experiments on several video-text retrieval benchmarks demonstrate the superiority of S3MA over existing methods.

Tideo-Text Retrieval by Supervised Sparse Multi-Grained Learning
Yimu Wang
Peng Shi University of Waterloo peng. shi@uwaterroo.crmmmm ca
University of Waterloo yimu. www.wwaag@@wwtteloo.comm
Abstract
Video
Text
Original Dense Space
Shared Spurse Space
While recent progress in video-text retrieval has been advanced by the exploration of better rep- resentation learning, in this paper, we present novel muulti-gainedd sparse learning framework, S3MA, learn an aligned sparse space shared between the video and the text for video-text retrieval. The shared sparse space is initial- ized with finite number of sparse concepts, each of which refers to a number of words. With the text data at hand, we learn and up- date the shared sparse space in supervised manner using the proposed similarity and align- ment losses. Moreover, to enable multi-grained alignment, incorporate frame representa- tions for better modeling the video modality and calculating fine-grained and coarse-grained similarities. Benefiting from the learned shared sparse space and multi-grained similarities, ex- tensive experiments on several video-text re- trieval benchmarks demonstrate the superiority S3MA over existing methods. Our code is available at link.
Vidco Rep
Frame Rep
Sparse Video Rep
Sparse Frume Rep
Coarse-grained Similarity
Coarsa-grained Similasity
Fine-grained
Fine-grained
Similarity
Similarity
Seutence Rep
Sparse Sentence Rep
Supervised
Sparse Concepts
Figure 1: Our proposed supervised shared sparse multi- grained alignment framework for video-text retrieval maps sentence, video, and frame representations to shared sparse space to obtain sparse sentence, video, and frame representations. Then, calculates coarse- and fine- -reiggaaied similarities to fully explore the power of the sparse space, which is learned in a supervised fashion. "Original Dense Space" represents the space containing the representations generated from modality- dependent encoders. "Shared Sparse Space" represents the space containing the sparse concepts shared across two modalities. "Rep" refers to representation.
1 Introduction
As a fundamental task in visual-language under- standing (Wang et al., 2020b; Xu et al., 2021b; Park et al., 2022a; Miyawaki et al., 2022; Fang et al., b023a.b; Kim et al., 2023; Jian and Wang, 2023), video-text retrieval (VTR) (Luo et al., 2022; Gao al., 2021b; Ma et al., 2022a: Liu et al., 2022a; Zhao al., 2022: Gorti et al., 2022; Fang et al., 2022) has attracted interest from academia and in- dustry. Although recent years have witnessed the rapid development of VTR with the support from powerful pretraining models (Luo et al., 2022; Gao et al., b021b; Ma et al., 2022a; Liu et al., 2022a), improved retrieval methods (Bertasius et al., 2021; Dong et al., 2019; Jin et al., 2021), and video- language datasets construction (Xu al., 2016), it remains challenging to precisely match video and language due to the raw data being in hetero- geneous spaces with significant differences.
Current VTR research (Luo et al., 2022: Ma et al., 2022a; Liu et al., 2022b) mainly aims to learn joint feature space across modalities and then compares representations in this space. How- ever, with the huge discrepancy between different modalities and the design of modality-indeeendeer encoders, it challenging to directly compare and calculate the similarities between representations of different modalities generated from different
encoders (Liang et al., 2022). To alleviate the mismatch caused by heterogeneous encoders and data formats, Liu et al. (2022a); Cao et al. (2022) proposed to align different modalities in com- mon space without supervision from text or video. However, because of the unsupervised design, the shared spaces are either randomly initialized or up- dated in an unsupervised fashion, which blocks the power of that aligned space. We argue that learning

---

shared aligned space with supervision is promis- ing way improve video-text retrieval. Borrowing from text retrieval (Karpukhin et al., 2020; Zhao et al., 2021; Gao et al., 2021a), we represent the aligned space and the space containing represen- tations generated by modelity-dependent encoders sparse and dense spaces, respectively, the aligned space typically carries specific semantics.
grained (video and sentence) representations from the sparse space and the dense space, enabling us calculate multi-space multi-grained similarity for exploring the potential of supervised sparse space. Finally, to evaluate the effectiveness of our pro- posed S3MA, we conducted experiments on three video-text benchmarks (Chen and Dolan, 2011; Fabian Caba Heilbron and Niebles, 2015; Xu al., 2016). Benefiting from multi-grained and multi- space similarity, our proposed S3MA outperforms previous methods on all the benchmarks without requiring any additional data during training. In summary, our contributions are as follows!:
In this work, we propose a Supervised Shared Sparse Multi-grained Alignment framework for VTR, namely S3MA, in which the aligned sparse space is updated under the supervision of the video- text data at hand. Specifically. we initialize a finite number of sparse concepts by clustering a large number of basic concepts (words) to form the fine- grained aligned sparse space. In return, each sparse concept is composed of several words, which im- proves the interpretability of our model. Then, we match the sparse text and video representations effectively by projecting the video representation generated by the video encoder to this fine-ge grained sparse space. The sparse sentence (text) represen- tations can be obtained by looking up the sparse concepts. To obtain sparse video representations. we first calculate the cosine similarity between the video representations and the sparse concepts. Next, by summing up all the sparse concepts with the weight of the cosine similarity between video representation and sparse concepts, we obtain the sparse video representations. Furthermore, to better match these two sparse representations, we design two loss functions to update sparse concepts, push- ing the sparse representations of text and video as close as possible in the shared sparse space. This shared sparse space design not only improves the performance on VTR, but also allows us interpret what the models have learned. The sparse aligned space, as shown in Figure 5, enables the model to accurately capture the key concepts, resulting in improved alignment within the sparse space.
We propose the shared sparse space to al- leviate the problem of mismatched repre- sentations from different modalities, which arises from the raw data being in heteroge- neous spaces and the heterogeneous design of modality-dependent encoders.
Our proposed S3MA achieves SOTA perfor- mance on several metries across three VTR benchmarks.
Detailed analysis reveals the importance of shared sparse space and multi-grained simi- larity. Besides, we demonstrate that the de- sign of shared sparse space and multi-grained similarity significantly impacts retrieval per- formance.
2 Related Works
Video-Text Retrieval (VTR), which involves cross- modal alignment and abstract understanding of temporal images (videos), has been popular and fundamental task of language manguagg-grounding prob- lems (Wang al., 2020a.c. 2021: Yu et al., 2023). Most existing conventional video-text retrieval frameworks (Yu al., 2017; Dong et al., 2019; Zhu and Yang. 2020; Miech et al., 2020; Gabeur et al., 2020; Dzabraev et al., 2021; Croitoru et al., 2021) focus on learning powerful representations for video and text and extracting separated represen- tations. Inspired by the success of self-supervised pretraining methods (Devlin et al., 2019; Rad- ford al., 2019; Brown et al., 2020) and vision- language pretraining (Li et al., 2020b; Gan et al., 2020; Singh et al., 2022) on large-scale unlabeled cross cross-modal data, recent works (Lei et al., 2021; Cheng et al., 2021; Gao et al., 202 bb; Ma et al.,
Recently, Ma et (2022a) demonstrated that incorporating fine-grained video representa- tions (such as frame or segment representations) with high-level video features can further improve retrieval performance. Inspired by their work, we further project frame representations into our de- signed aligned sparse space. Compared high- level video representations, frame representations can be mapped to more detailed concepts, which enriches the overall video representations. In this way, we have ime-grained (frame) and coarse-
TTee code is released at link.

---

Frame Rep rd
Video Rep
Dense Space
Frame Uncoder
Temporal Finedder
Video
Sparse Space
Sparse Video Rep r?
Sparse Concepts
Sparse Frame Rep r!
Text
Look Up
Sentence Rep r
Text
Sparse Sentence Rep
Encoder
The illustration of representation generation in our proposed Supervised Shared Sparse Multi-gruinet Aligument framework, namely S3MA. Specifically, for multi-space alignment, we employ shared sparse space which is consisted of a number of sparse concepts. The shared sparse space is updated in a supervised manner during the training procedure, leading to the construction a fine-grained sparse space. "ø" refers to the calculation Eqs. (I), (2), and (3).
Figure 2:
2022a; Park et al., 2022a; Wang et al., 2022b,e; Zhao et al., 2022; Gorti et al.. 2022) have attempted to pretrain or fine-tune video-text retrieval models in an end-to-end manner. Frozen in time (Bain et al., 2021) uses end-to-end training both image- text and video-text pairs data by uniformly sam- pling video frames. CLIP4Clip (Luo al., 2022) finetunes models and investigates three similar- ity calculation approaches for video-sentence con- trastive learning on CLIP (Radford et al., 2021). Later, to enable unsupervised sparse learning in VTR, DiscretCodebook (Liu et al., 2022a) aligns modalities in shared space filled with concepts, which randomly initialized and unsupervisedly updated, while VCM (Cao al., 2022) constructs a sparse space with unsupervisedly clustered visual concepts. At the same time, OA-Trans (Wang et al., 2022a) and TABLE (Chen et al., 2023) both em- ploy small number of semantic tags as the input to the text encoder to improve alignment between modalities.
for multi-space multi- gratii- alignment.
Methods
In this section, we introduce our proposed frame- work for vidco-text retrieval, which aligns language and video in a shared sparse space. Typically, video-text retrieval, we have a set of examples {vv,, t:) t=)] where N is the number of exam- ples that are of video and language.
3.1 General Video-Text Retrieval Paradigm
In this part, we present general video-text retrieval framework widely used by previous methods (Luo al., 2022: Liu al., 2022a). With this paradigm, can obtain three representations for different modalities from the dense space, i.e., frame repre- sentation r', video representation r", and sentence representation r" by
encoders.
Frame and video representations: Given a video v, several video frames are first sampled as the inputs of the frame encoder to obtain the frame features rf E pet/rrme xd where frrame is the number of frames and d is the dimension of fea-
However, due the unsupervised design, con- cepts in DiscretCodebook and VCM are either ran- domly initialized or updated unsupervisedly, which limits the potential of aligned sparse space. On the other hand, OA-Trans and TABLE only em- ploy a limited number of concepts to serve as the input of the text encoder to encourage align- ment. Meanwhile, these methods only perform the hearse-grainee vidco-text similarity, lacking the fine-grained contrast between different modali- ties. In comparison, our proposed S3MA learn the aligned sparse space containing a large number of words in a supervised manner, under the supervi- sion of text, and calculate frame-sentence similarity
tures. As the frame representations r' are extracted through sampling, to explore the temporal correla- tion among different frames, we employ a temporal encoder to aggregate frame representations. With the temporal encoder and the frame representations rí. we obtain the video representations r" [R 1xd
Sentence representation: Given a sentence t, we use text encoder to obtain the text representation rd E 1xd

---

3.2 F'ieeGGriind Aligned Sparse Space
Video Rep r
Dense
The key to the video-text retrieval task to pre- cisely align representations from different modal- ities. However, due to the heterogencous encoder architectures and data formats of different modal- ities, it is difficult to align directly (Liang al., 2022). Therefore, instead of directly enforcing the representations to be aligned, we propose aligning them an aligned sparse constructed by ne sparse concepts E - Each sparse concept c repre- sents several basic concepts (words). Moreover, to supervise the updates of sparse concepts, we utilize the human-annotacee knowledge at hand, i..., text annotations in the paired video-text data.
Rep rf
Sptee
Space
Sentence Rep pr
Mulli-space Multi-grineed Similarily
Sporse Video
Rep rz Sparse Frame
Sup-1
Rep r Sparse Sentence Rep r?
Sparse
Space
Figure 3: The illustration of similarity calculation. To enable multi- multi -ultti-rained alignment, we calcu- late fine-graines (frame-sentence) and coarse-grained (video-sentence) similarity. Our preliminary experi- ments showed that the text encoder has good ability to capture semantics, tions for the text modality.
only use sentence representa-
Initialization. First, map all the words into beddings by the embedding layer femb of the text encoder. But as the number words is relatively large (for example, in Clip (Radford et al., 2021), 1), the number of sub-words is approximately 30k), we cluster embeddings into ne clusters using KNN (Gi-
Ohtaining the sparse frame representation. Sim- ilarly, the cosine similarity sime frmme Xn. be- tween the frame representations and sparse con- cepts calculated as simg, = cos(1, cÿ), Vi E [r framee Vj [me], where sim is the (i,j)-th element of sime and r/ is the i-th row of r'. Next, sparse frame representations are obtained as,
anfelici, 2008) to form the sparse concepts C' and represent all the words by their cluster's 's centers Consequently, each sparse concept represents bunch of words that similar the embedding space, enabling fine-grained alignment. The map- ping from words to sparse concepts is denoted by twwe E [noords]] + {0, 1 ,x Now, ne sparse concepts have been initialized.
Σ fElmframel
sim? C//lmmm/1 (3)
C
Finally, we have the sparse frame, video, and sentence representations r! E Ra xd r e I rr- E R with the frame and video sparse space similarity sim/ E R Xne and sim" mh along with the sentence sparse space similarity (supervision) sim..
Obtaining the sparse sentence representation. For text, as the caption is at hand, we can directly tokenize the sentences into words and look up the corresponding sparse concepts in C. The sparse sentence representation r R is obtained by averaging all the representations of concepts that are fetched with the surface form of the sentence, as follows,
3.3 Multi-Space Multi-Grained Similarity In this part, we will demonstrate our method for calculating the similarities between data from two different modalities, as shown in Figure 3, includ- ing the similarities in the dense space and in shared sparse space, inspired by Ma et al. (2022a). We can now compute multi-space (sparse and dense spaces) multi galti-grained (fine- -rai-gra and coarse-graie -grinedg similarity for precise alignment.
C/(t), C
T
(1)
where |t] is the number of words in t and sim Ewer (w)) is a vector with the length of ne- Obtaining the sparse video representation. We first calculate the cosine similarity sim" E Rxxn, between the video representations and sparse con- cepts C' as simy cos(r", c,), Vj e [nc]. where simy the j-th element of simY and cos(..-) is the cosine similarity. Next, sparse video representa- tions are obtained by weighted summing the sparse concepts as,
3.3.1 Dense Space Similarity
Video-Sentence similarity Sp--r. To obtain a fine-grained similarity, we use learnable matrix A E Rdddd to focus the discriminative fea- tures of video and sentence representations as,
th = sim s/immC/|ssimm"
Spr = r" Ar rer

---

Frame-Sentence similarity Spr-r-. To obtain fine-grained similarity, we first calculate an instance-aware weight using the softmax function applied to the dot product of r and then use a learnable matrix Ar E IR. frame frame to focus on discrininative frames. In this way, the similarity is calculated as,
representations (ré and r²)) with the sparse sentence representations r in the Eg distance, as,
latign
Σ
(1r²
råll2
sc[M]
1rl
+
firamm
Sp-r softmax(r'r (A DA
where is the vector only containing 1. In addition, match the frame and video repre- sentations with the corresponding sparse concepts, we minimize the sparse similarity loss as.
3.3.2 Sparse Space Similarity
Video-Sentence shared sparse space similarity Srg rs- Similarly, to obtain a fine-grained similar- ity on the shared sparse space, we use learnable matrix Arg r E RRddd to focus on the discrimina- tive features of sparse vidco and sentence represen- tations. Now, the similarity is calculated as,
l Esparse se
Σ EE(N
(I|sim"
<eim"=simmll,
1siml
1.)
sim*
+
Aframe
Seg-rg = A Arg-ra
The overall objective is the linear combination of the above three losses as,
Frame-Sentence shared sparse space simi- larity $i With instance-aware weights softmax(rgs) (r and learnable matrix A E R frame frame we get the similarity between the sparse frame and sentence representations as,
t losm al + Blsparse
where a and 3 are hyperparameters controlling the trade-off between three losses. We set a 0.02 and 3 0.01 for all the experiments.
S r softmax(r"r) (T) ( A
rer
4 Experiments
3.3.3 Overall Similarity
4.1 Datasets and Baselines
The overall video-text similarity is defined as,
To show the empirical efficiency our S3MA, we train it on MSR-VTT (Xu et al., 2016), MSVD (Chen and Dolan, 2011), and Activi- tyNet (Fabian Caba Heilbron and Niebles, 2015). We compare with VLM (Xu et al., 2021a),
Spi-r= + S +++5. S - + Sig Seg-ra
4
3.4 Objective
The objective consists of three different losses. The first component is contrastive loss. Follow- ing Clip4Clip (Luo et al., 2022), we employ the symmetric InfoNCE loss over the similarity matrix to optimize the retrieval model as,
HERO (Li et al., 2020a), VideoCLIP (Xu et al., 2021b), EvO (Shvetsova
al., 2022), OA- Trans (Wang et al., 2022a), RaP (Wu et al., 2022). Li (Chen et al., 2022), NCL (Park et al.. 2022b), TABLE (Chen et al., 2023), VOP (Huang et al., 2023), Clip4Clip (Luo et al., 2022), X- CLIP (Ma et al., 2022a), DiscreteCodeebook (Liu et al., 2022a), TS2-Net (Liu et al., 2022b), VCM (Cao al., 2022), HiSE (Wang et al.. 2022b), Align&Telll (Wang al., 2022c), Center- CLIP (Zhao al., 2022), and X-Pool (Gorti et al., 2022). Implementation details and evaluation pro- tocols are deferred to the Appendix.
Loim = + L,20
1
log exp(Sis)
N
jc[M] exp(S:s)
KE(N
1
(Sp()ia)
Σ RE(N
log
Σ IjE(M exp(S,,)
where St.j similarity between i-th video and j-th text and N the number of paired data.
4.2 Quantitative Results
The second loss we minimize is the alignment loss, which matches the sparse frame and video
MSR-VTT. As shown in Table 1, S3MA achieves the best R on the text-to-video retrieval results

---

Text-to-Vidde Retrieval
Video-to-Tese Retrieval
Methods
Venue
R@11
R@5t
R 101
MdR,
MnR,
R@11
R@5t
R@ 101
MdR1
MnR,
VLM
ACT.21
28.1
55.5
67.4
4.0
HERO
E
43.3
57.7
VideoCLIP
EMNLP'21
30.9
55.4
66.8
EvO
CVPR'22 22
23.7
52.1
63.7
4.0
OA-Trans
CVPR'22
35.8
63.4
76.5
3.0
RaP
EMNLP'22
67.2
76.9
BLIP-based
LiteVL-S
EMNLP'22
46.7
71.8
81.7
2.0
WT-R/22-hased
Teign&Tell
TMM
45.2
73.0
82.9
43.4
70.9
81.8
2.0
X-Pool
CVPR'22
46.9
72.8
82.2
2.0
14.3
CenterCLIP
SIGIR'22
44.2
71.6
82.1
2.0
15.1
42.8
71.7
82.2
2.0
10.9
TS2-Net
ECCV"22
47.0
74.5
83.8
13.0
45.3
74.1
83.7
2.0
9.2
X-CLIP
MM'22
74.3
83.1
13.2
73.3
9.1
NCL
EMNLP'22
43.9
71.2
81.5
2.0
15.5
44.9
71.8
80.7
2.0
12,8
TABLE
AAAT'23
47.1
74.3
82.9
2.0
13.4
47.2
74.2
84.2
2.0
11.0
VOP
CVPR'23
44.6
69.9
80.3
2.0
16.3
44.5
70.7
80.6
2.0
11.5
CLIP4Clip
NC
44.5
71.4
81.6
2.0
15.3
DiscreieCudebet
ACL'22
43.4
72.3
81.2
14.8
42,5
71.2
81.1
12.0
VCM
AAAI'22
43.8
71.0
14.3
45.1
72.3
82.3
2.0
10.7
S3MA
73.9
82.8
13.5
73.8
82.1
9.3
S3MA
$1.7
75.9
85.4
1.0
11.1
51.6
76.8
85.0
1.0
8.4
WT WTT-e/I6-ased
Tell
TMM
47.4
74.3
84,1
2.0
45.3
73.5
83.7
2.0
CenterCLIP
SIGIR 22
48.4
73.8
82.0
2.0
13.8
47.7
75.0
83.3
2.0
10.2
HISE
ACM MM'22
45.0
72.7
81.3
46.6
73.3
82.3
2.0
TS2-Net
ECCV'22
49.4
75.6
85.3
13.5
46.6
75.9
84.9
2.0
8.9
CLIP4Clip
NC
45.8*
74.3*
84.1*
2.0*
43.2*
71.3*
82.0*
2.0*
S3MA
75.1
83.9
12.2
47.3
76.0
84.3
2.0
8.9
S3MA
53.1
78.2
86.2
1.0
10.5
52.7
79.2
86.3
1.0
8.2
Table I: Video- Text retrieval results on MSR-VTT. represents data copied from Align&Tell. The best results are marked in bold. The second best results are underlined. "NC" refers Neurocomputing. t refers to the results with the inverted softmax.
Text-to- Vidco Retrieval R@5t
The performance of S3MA on the video-to-text re- trieval task is also comparable with previous meth- ods, achieving the best and second-best results on R@1 and R @@5 using ViT-B/32. Moreover, we no- tice that only previous method using ViT-B/16 outperforms S3MA with ViT-B/32 on the text-to- video retrieval, demonstrating the effectiveness of S3MA. Compared to DiscreteCodebook (Liu al., 2022a), which aligns modalities in an unsupervised manner, S3MA outperforms DiscreteCodebook on every metric. Mcanwhile, S3MA also outper- forms VCM (Cao et al., 2022), which constructs an aligned space with unsupervisedly clustered vi- sual concepts, demonstrating the importance of supervising alignment in the sparse space. This suggests that aligning modalities with fine-grained supervision is promising approach to improving video-to-text retrieval performance.
Methods
Venue
R@I1
R@10"
MnR.
MSVD
X-CLIP
ACM MM
47.1
77.8
9.5
HISE
ACM MM'22
45.9
76,2
84.6
X-Pool
CVPR'22
47.2
77.4
86.0
9.3
CLIP4Clip
NC
45.2
75.5
84.3
10.3
S3MA
47.3
78.8
85.7
9.3
ActivityNer
Tel
TMM
73.8
X-CLIP
MM/22
74.1
7.9
TS2-Net
BCCV'22
73.6
8.4
CLIP4Clip
40.5
72.4
7.5
VCM
AAAP22
40.8
72.8
2.3
S3MA
45.0
75.5
85.7
6.3
Table 2: TextoVideo retrieval results on MSVD and ActivityNet. The best results are marked in bold. The second best results are underlined.
using ViT- B/3- and ViT-B/16, outperforming the second-bes best method by 2.1 and 0.4, respectively.
MSVD and ActivityNet. The results on MSVD

---

Bext-to-Vides Retrieval
Video-to- Retrieval
R@ 11
R@51
R@ 101
MdR.
MnR
R@ If
R@5t
R@101
MdR,
MnR,
S3MA (VTT-B/32) w. SE
47.3
73.5
82.0
2.0
13.3
45.6
73.4
82.4
2.0
9.1
S3MA (VIT-B/32) Emb
49.1
73.9
82.8
2.0
13.5
46.9
73.8
82.1
2.0
9.3
Table Comparing the power of different sparse spaces on MSR-VTT. "Emb" and "SE" refers to the embedding space and semantic embedding space.
Text-to-Video Retrieval
Video-to- Retrieval
R@11
R@st
R@101
MdRL
MnR
R@ 11
R@51
R 101
MdR]
MnR
S3MA (ViT-B/32) w/o clustering S3MA (ViT-B/32)
48.7
74.4
83,0
2.0
13.4
46.7
73.3
82.6
2.0
9.2
49.1
73.9
2.0
13.5
46.9
73.8
82.1
9.3
Table 4: Ablation study on the effect of clustering when constructing the shared sparse space.
Text-to- Retrieval MnR
Video-to- Text Retrieval
while the semantic embedding space, is initialized by outputs of the last layer in the text encoder, with input consisting of word and two [SEP) tokens. By replacing the embedding initialization with the semantic embedding, the retrieval performance of S3MA decreases, proving the superiority of em- bedding space over the semantic embedding space. Size sparse space. Another important factor to consider is the size of the sparse space. When we have unlimited data to train models, a large sparse space is ideal. However, when the data is limited, large sparse space can lead to sparse gradients, resulting in most of the concepts not being able to be updated, while small sparse space will restrict the retrieval ability as becomes more challenging to distinguish between numerous data points. The results of these experiments can be found in Table 5. We see that halving and doubling the size of the sparse space slightly decreases performance.
Size
R@D
R@5
R@1 l
R@5
MoR
48.7
73.0
12,9
46.4
72.8
1024
49.1
73.9
13.5
46.9
73.8
9.3
2048
48.3
73.9
13.5
47.0
72.7
9.1
4096
47.6
73.6
13.6
46.8
73.4
9.3
DC (1024)
43.4
72.3
14.8
42.5
71.2
12.0
VCM
43.8
71.0
14.3
45.1
72.3
10.7
Table 5: Retrieval performance with different sizes of sparse space the MSR-VTT dataset using S3MA with ViT/B-32. "DC" represents DiscreteCode- book (Liu et al., 2022a), which also aligns modalities in a sparse space whose size is 1024 with the base model of ViT/B-32. The best results are marked in bold. The second best results underlined.
and ActicityNet are shown in Table 2. S3MA achieves the best R@I on text-to-video retrieval on two datasets compared to the previous meth- ods. Besides, with the shared sparse space and multi-grained alignment, S3MA also has the low- est MnR.
Impact of clustering. As S3MA clusters all the embeddings to initialize concept clusters, it is un- certain whether clustering will hinder the power of the shared sparse space. Clustering can be use- ful to extract high-level abstract concepts and duce noise. However, it may also lead loss of information, which is important for fine- mra-grained aligument. Specifically, we compare the perfor- mance of S3MA to that of a modified version, S3MA w/o clustering concepts, which directly uses over 30k basic concepts to form the shared sparse space. Quantitative results can be found in Table 4. The results show that without clustering, R @ 5, @ 10, and MnR on text-to-video retrieval and R @ 10 and MnR on video-to-text retrieval are improved. On one hand, similar basic concepts can be better separated, which leads to more precise alignment. On the other hand, that may lead to
4.3 Ablation Studies
In this part, we present a series of ablation experi- ments on MSR-VTT to demonstrate the effective- ness of different components of S3MA. The evalu- ation of two proposed losses, similarity calculation, and the importance of word-level features are de- ferred to the Appendix.
4.3.1 Efficiency of Sparse Space
The choice of different initialization of sparse spaces. To choose the best initialization method for the sparse space, we conduct experiments using two different initializations, i.e., , the embedding and semantic embedding spaces, as shown in Table 3. The embedding space is the one we use in S3MA,

---

Dense Space S-V
Sparse Space S-V
Text-to- Video Retrieval
Video-to -oxx Retrieval
S-F
S-F
R@ 1
R@51
R@101
MdR.
MnR.
R 11
R@51
R 101
MdR
MnR +
42.8
72.0
82.3
2.0
15.0
41.9
71.1
81.5
2.0
11.1
43.3
70.5
81.4
2.0
15.6
42.5
71.0
80.9
2.0
11.9
81.8
2.0
14.5
44.1
71.8
81.7
10.4
d
44.8
72.1
81.7
2.0
13.9
41.7
70.2
79.6
2.0
10.8
81.6
2.0
15.2
42.0
70.9
2.0
11.0
43.8
72.1
82.3
2.0
14.7
41.5
70.6
80.3
2.0
9.8
44.0
71.3
80.9
2.0
14.8
43.6
69.5
80.1
2.0
10.4
47.4
73.3
82.4
2.0
12,9
46.4
73.0
82.2
2.0
8,9
47.4
73.6
82.5
2.0
13.2
47.3
72.3
81.7
2.0
8,9
82.8
2.0
13.5
46.9
73.8
Table 6: Retrieval performance with different similarities on MSR-VE VTT using S3MA with the base model of ViT- B/32. "S-V" and "S-F" represent Sentence- Video (coarse-grained) and Sentence-Framme Frame (fine-grained) similarities.
Text-to- Video
Video-to-
retrieval from 42.8 to 43.3 compared to only performing coarse-grained similarity in the dense space, respectively.
Base Model
TE
R@I
R@5
MnR
R@1
R@5
MnR
47.0
73.9
14.5
45.7
72.3
9.6
VIT-B/32
49.1
73.9
13.5
46.9
73.8
9.3
47.3
74.9
46.1
75.1
Using multi-space and multi-grained similar- ities simultaneously achieves the best perfor- mance. R on text-to-video and vidco-to- text retrieval significantly improved from 42.8 and 41.9 to 49.1 and 46.9, respectively.
VIT-B/16
49.8
75.1
12.2
47.3
76.0
8,9
Table 7: Retrieval performance with or without the tem- poral encoder ("TE") and with different base models.
sparse gradients, resulting in some concepts not being fully updated while others are over-updated. This might cause concepts to be under or over-represented, which might negatively impact the performance (Radovanovic et al., 2010). There- fore, it's important to find the balance clustering to achieve the best performance.
4.3.3 Temporal Encoder and Larger Model
We also investigate the effect of the temporal en- coder (TE, small sequence transformer) and dif- ferent base models. The results are shown in Ta- ble 7. S3MA with TE outperforms S3MA without TE, because it is able to better model the temporal relation among different frames in video. Be- sides, using larger base model, such as ViT-B/16, further improves the performance of S3MA, as a larger base model typically has better representa- tion learning abilities benefiting this retrieval task as well. Similar conclusions can be found in previ- ous works (Luo et al., 2022; Ma et al., 2022a).
4.3.2 Efficiency of Multi-Grained Similarities In order to fully evaluate the impact of multi- grained similarities, compare different variants of S3MA and the results are shown in Table 6. From these results, we can draw three conclusions,
Multi-grained similarities are crucial for re- trieval. Using both coarse- and fine-grained alignments in the dense space improved R @ from 42.8 and 41.9 to 44.0 and 43.6 on text- retrieval compared with only using coarse granied alignment in the dense space, respectively. The same obser- vation can be observed in the sparse space.
4.4 Qualitative Results
To qualitatively validate the effectiveness of S3MA and the alignment in the sparse space, we present examples of video-to-text and text-to-video re- in Figures 4, 6 and 7, and the
to-video and
trieval on alignment in sparse space in Figure 5, respectively. The retrieval results show the satisfactory per- formance of S3MA, benefiting from multi-space multi-grained similarity. Notably, S3MA demon- strates precise identification of the color (green), objects (bicycle), and humans (a man), indicating its proficiency in capturing intricate details. In Fig-
Sparse space plays crucial role in improving the alignment of modalities. We observe that incorporating coarse- ssarse-grained the dense and sparse spaces improves R for text-to-video

---

Limitations
Query: a man discusses spongebob
In the future, it would be promising to seek more fine-grained alignment, such as instance (object)- level or word-level alignment, for aligning differ- ent modalities. Moreover, our experiment focused solely the application of sparse retrieval in video- text retrieval. It would be great to see whether sparse retrieval can help other moosll retrieval tasks, e.g. audio-text, image-text, audio-video, and audio-image retrieval. Additionally, incorporating more detailed information such as the relationship between different objects and frames would be ben- eficial for the video-text retrieval problem.
V
Topl
Top2
Top3
Query:
Topl: man talks in front of green bicycle
Top2: man talks about cars
Top3: people talking about a fight
Regarding the sparse space, we notice that some sparse concepts are retrieved a lot during the train- ing procedure which might lead to the emergence of hubness (Radovanovic et al., 2010). Investigat- ing improved clustering methods to mitigate hub- ness would be an interesting direction for future research. That might be due to the KNN clustering strategy and in the future and introducing better clustering strategies might be able to reduce the hubness issue, such weighted KNN, semantic- based KNN, or part-of-specch tagging- based KNN,
Top4: two people are preparing for sports
Top5: guys holding cups and talking
Figure 4: Video- retrieval examples.
Query: a(519) movie(947)) directort (694) (2lking(248) lo(1017) (he(519) 154) (en((8) (0(1017) (1es (on3ereacx(133
reganding(827) his(384) movia(947) und(522) hem(2133 also(41)
Video Sparse Similarity
Top10 Indices and Similarities
Ind
519
41
28
213
5522
140
248
827
578
Sim
0.91
0.80
0.70
0.63
0.57
0.53
0.50
0.44
0.42
Frame Sparse Simitarity TTpp0 Indlees and Similarities
Ind
519
213
1017
248
827
522
124
28
140
Sim
0.90
0.85
0.83
0.82
0.82
0.76
0.75
0.71
0.68
0.68
References
Figure 5: An example of alignment on the sparse space.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zis- serman. 2021. Frozen in time: joint video and image encoder for end-to-end retrieval. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 1708-1718, IERE
The index of the concepts is shown the brackets.
ure 5, we notice that, the video and frame features are perfectly aligned with the corresponding sparse concepts as exhibiting high similarities.
Gedas Bertasius, Heng Wang. and Lorenzo Torresani. 2021. Is space-time attention all you need for video understanding? In Proceedings of the 38th Inter- national Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 813 824. PMLR.
5 Conclusion
In this paper, to better align video and text modal- ities, we proposed a multi-space, multi-grained video-text retrieval framework, S3MA. Specifi- cally, S3MA aligned different modalities a fine- grained shared sparse space, which is initialized with a finite number of concept clusters consist- ing of a number of basic concepts (words) and updated in a supervised fashion with the guide of text. Besides, S3MA employed frame (finc- grained) and video (coarse-grained) features en- courage models perform multi-grained similarity alignment. Finally, we conducted extensive experi- ments on three representative video-text retrieval benchmarks, showing the superiority of S3MA.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah. Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voon Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12, 2020, viriual.

---

Shuqiang Cao, Bairui Wang, Wei Zhang, and Lin Ma. 2022. Visual consensus modeling for video-text re- Artificial
Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
trieval. In Thirty-Sixth AAAI Conference Intelligence, AAAI 2022, Thirty-Fourth Fourth Conference on Innovative Applications of Artificial Intelligence. TAAT 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, FAAT 2022 Virtual Event, February 22 March 1, 2022, pages 167-175. AAAI Press.
Maksim Dzabraev, Maksim Kalashnikov. Stepan Komkov, and Aleksandr Petiushko. 2021. MDMMT: multidomain multimodal transformer for video re- trieval. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2021, virtual, June 19-25, 2021, pages 3354-3363. Computer Vision Foundation / IEEE.
David Chen and William Dolan. 2011. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th Annual Meeting of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 190-200, Portland, Ore- gon, USA. Association for Computational Linguis- tics.
Bernard Ghanem Fabian Caba Heilbron, Victor Escor- cia and Juan Carlos Niebles. 2015. Activitynet: A large-scale video benchmark for human activity un- derstanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 961-99 161-
Dongsheng Chen, Chaofan Tao, Lu Hou, Lifeng Shang, Xin Jiang, and Qun Liu. 2022. VL: Effi- cient video-language learning with enhanced spatial- temporal modeling. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 7985-7997, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Xiang Fang, Daizong Liu, Pan Zhou, and Yuchong Hu, 2022.
cross-domain alignment network for video moment retrieval. IEEE Transactions on Multimedia.
Yizhen Chen, Jie Wang. Lijian Lin, Zhongang Qi. Jin Ma, and Ying Shan. 2023. Tagging before Align- ment: Integrating Multi-Modal Tags for Video-Text Retrieval. In AAAI Conference on Artificial Intelli gence. arXiv. AtXiv:2301.1266 [cs].
Xiang Fang, Daizong Liu, Pan Zhou, and Guoshun Nan. 2023a. You can ground earlier than see: An effective and efficient pipeline for temporal sentence ground- ing in compressed videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Put- tern Recognition, pages 2448-2460.
Xing Cheng, Hezheng Lin, Xiangyu Wu, Fan Yang, and Dong Shen. 2021. Improving video-text retrieval by multi-stream corpus alignment and dual softmax loss. CoRR, abs/2109.04209
Xiang Fang, Daizong Liu, Pan Zhou, Zichuan Xu, and Ruixuan Li. 2023b. Hierarchical local-global trans- former for temporal sentence grounding. IEEE Trans- actions on Multimedia.
Ioana Croitoru, Simion- -lad Bogolin, Marius Leordeanu, Hailin Jin, Andrew Zisserman, Samuel Albanie, and Yang Liu. 2021. Teachtext: Crossmodal generalized distillation for text-video retrieval. In 2021 IEEF/CVF International Conference on Com- puter Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 11563-11573. IEEE
Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. 2020. Multi-modal transformer for video retrieval. In Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV. volume 12349 of Lecture Notes in Computer Science, pages 214 214-229. Springer.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng. and Jingjing Liu. 2020. Large-scale adver- sarial training for lision-add-annuuaee representation learning. In Advances in Neural Information Pro- cessing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Jianfeng Dong, Xirong Li, Chaoxi Xu, Shouling Ji, Yuan He, Gang Yang, and Xun Wang. 2019. Dual encoding for zero-example video retrieval. In IEEE Conference on Computer Vision and Pattern Recogni- tion, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 9346-9355. Computer Vision Founda- tion/ / IEEE.
Luyu Gao, Zhuyun Dai, and Jamie Callan. 2021a. COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 3030-3042 Online. Association for Computational Linguistics.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai,

---

Zijian Gao, Jingyu Liu, Sheng Chen, Dedan Chang, Hao Zhang, and Jinwei Yuan. 2021b. CLIP2TV: an empirical study transformer-based methods for video-text retrieval. CoRR, abs/2111 11.05610.
Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. 2020a. HERO: Hierarchical encoder for Video+ anguuag Video+Languuae renresspresnntaioonn pre-training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP),, pages 2046-2065, Online. Association for Computational Linguistics.
F. Gianfelici. 2008, Nearest-neighbor methods in learn ing and vision. IEEE Transactions on Neural Net- works, 19(2)337-377
Xinjun Li. Xi Yin, Chunyuan Li, Pengchuan Zhang. Xiaowei Hu, Lei Zhang. Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. 2020b. Oscar: semantese aligned pre- training for vision-language tasks. In Computer Vi- sion ECCV 2020- 16th European Conference, Glas- gow; UK, August 23-28, 2020. Proceedings, Part XXX, volume 12375 of Lecture Notes in Computer Science, pages 121-137. Springer.
Satya Krishna Gorti, Noël Vouitsis. Junwei Ma. Key- van Golestan, Maksims Volkovs, Animesh Garg, and Guangwei Yu. 2022. X-pool: Cross-modal language- video attention for tixt-video retrieval. IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 4996-5000 5005. IEEE.
Siteng Huang, Biao Gong, Yulin Pan, Jianwen Jiang, Yiliang Lv, Yuyuan Li, and Donglin Wang. 2023. VoP: Text-Video Co-Operative Prompt Tuning for Cross-Modal Retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- lern Recognition, pages 6574.
Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. 2022. Mind the gap: Under- standing the modality gap in multi-mode modal contrastive representation learning. In Advances in neural infor- mation processing systems.
Xiangru Jian and Yimu Wang. 2023, Invge: Robust cross-modal retrieval by inverse graph convolution.
Alexander Liu, Sou Young Jin, Cheng-I Lai, Andrew Rouditchenko, Aude Oliva, and James Glass, 2022a. Cross-modal discrete representation learning. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume Long Papers), pages 3013-3035, Dublin, Ireland. Associa- tion for Computational Linguistics.
Weike Jin, Zhou Zhao, Pengcheng Zhang, Jieming Zhu, Xiuqiang He, and Yueting Zhuang. 2021. Hierar- chical cross-modal graph consistency learning for video-text retrieval. In SIGIR 21: The 44th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pages 1114-1124, ACM.
Yuqi Liu, Pengfei Xiong, Luhui Xu, Shengming Cao. and Qin Jin. 2022b. Ts2-net: Token shift and selec- tion transformer for text-video retrieval. In Computer Vision - FCCC 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XIV, volume 13674 of Lecture Notes in Computer Science, pages 319-335. Springer.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)), pages 6769-6781. Online. Association for Computational Linguisties.
Nyy Loshchilov and Frank Hutter. 2017. SGDR:
stochastic gradient descent with warm restarts. In 5th International Conference
Tachoon Kim, Pyunghwan Ahn, Sangyun Kim, Sihaeng Lee, Mark Marsden, Alessandra Sala, Seung Hwan Kim, Bohyung Han. Kyoung Mu Lee, Honglak Lee, Kyounghoon Bac, Xiangyu Wu, Yi Gao, Hailiang Zhang, Yang Yang. Weili Guo, Jianfeng Lu, Young- taek Oh, Jae Won Cho. Dong jin Kim, In So Kweon, Junmo Kim, Wooyoung Kang, Won Young Jhoo, Byungscok Roh, Jonghwan Mun, Solgil Oh, Ke- nan Emir Ak, Gwang-Govd Lee, Yan Xu, Mingwei Shen, Kyomin Hwang, Wonsik Shin, Kamin Lee, Wonhark Park, Dongkwan Lee. Nojun Kwak, Yujin Wang, Yimu Wang, Tiancheng Gu, Xingchang Lv, and Mingmao Sun. 2023. Nice: Cvpr 2023 challenge on shoo-shot image captioning.
Learning Representa- tions, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Ilya Loshchilov and Frank Hutter, 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net.
Huaishao Luo, Lei Ji, Ming Zhong. Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. 2022. Clip4elip: An empirical study of CLIP for end to end video clip retrieval and captioning. Neurocomputing, 508:293- 304.
Jie Lei, Linjie Li. Luowei Zhou, Zhe Gan, Tamara L. Berg, Mohit Bansal, and Jingiing Liu. 2021. Less is more: Clipbert for video-and- languge learning via sparse sampling. In IEFE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 7331-7341. Computer Vi- sion Foundation/ / IEBE.
Yiwei Ma. Guohai Xu. Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji. 2022a. X-CLIP: end-to- end multi-grained contrastive learning for videa-text retrieval. In MM '22: The 30th ACM International Conference on Multimedia, Lisbou, Portugal, Octo- ber 10 14, 2022, pages 638-647. ACM.

---

Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan. Ji Zhang, and Rongrong Ji. 2022b. X-CLIP: End-to- to- end multi-grained contrastive learning for video-text retrieval. In ACM international conference on mul- limedia, MM '22, pages 638-647, New York, NY, USA. Association for Computing Machinery. Num- ber of pages: 10 Place: Lishoa, Portugal.
Milos Radovanovic, Alexandros Nanopoulos, and Mir- jana Ivanovic. 2010. Hubs in space: Popular nearest neighbors in high-dimensional data, J. Mach. Learn. Res., 11:2487-2531.
Nina Shvetsova, Brian Chen, Andrew Rouditchenko, Samuel Thomas, Brian Kingsbury, Rogerio Feris, David Harwath, James Glass, and Hilde Kuehne. 2022. Everything at Once - modal Fusion Transformer for Video Retrieval. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 19988-19997, New Orleans, LA, USA. IEEE
Antoine Miech, Jean-Baptiste Alayrac. Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. 2020. End-to-end learning of visual representa- tions from uncurated instructional videos. In 2020 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 9876-9886. Computer Vi- sion Foundation/ / IEEE.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. 2022. FLAVA: A foun- dational language and vision alignment model. In IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 15617-15629, IEEE
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivie. 2019. Howto100m: Learning a text-video embed- ding by watching hundred million narrated video clips. In 2019 IEEF/CVF International Confer- ence on Computer Vision ICCV 2019, Seoul, Ko- rea (South), October 27 - November 2, 2019, pages 2630-2640. IEEE.
Alex Jinpeng Wang. Yixiao Ge. Guanyu Cai, Rui Yan, Xudong Lin, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. 2022a. Object-aware Video- language Pre-training for Retrieval.
2022
Shumpei Miyawaki. Taku Hasegawa, Kyosuke Nishida, Takuma Kato, and Jun Suzuki. 2022. Scene-text aware image and text retrieval with dual-encoder, In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties: Student Research Workshop, pages 422-433, Dublin, Ireland. Association for Computational Linguistics.
IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 3303-3312, New Orleans, LA. USA. IEEE
Haoran Wang. Di Xu, Dongliang He,, Fu Li, Zhong Ji. Jungong Han, and Errui Ding. 2022b. Boosting video-text retrieval with explicit high-level semantics. In MM '22: The 30th ACM International Conference on Multimedia, Lisboa, Portugal, October 10 14, 2022, pages 4887-4898. ACM.
Jae Sung Park, Sheng Shen, Ali Farhadi, Trevor Darrell, Yejin Choi, and Anna Rohrbach. 2022a. Exposing the limits of video-text models through contrast sets. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics; Human Language Technologies, pages 3574-3586, Seattle, United States. Association for Computational Linguistics.
Xiaohan Wang. Linchao Zhu, Zhedong Zheng, Min- gliang Xu, and Yi Yang. 2022c. Align and tell: Boosting rixt- retrieval with local alignment and fine-grained supervision. IEEE Transactions on Multimedia, pages 1-11.
Yookoon Park, Mahmoud Azab, Seungwhan Moon, Bo Xiong, Florian Metze, Gourab Kundu, and Kir- mani Ahmed. 2022b. Normalized contrastive learn- ing for text-video retrieval. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 248-260, Abu Dhabi, United Arab Emirates. Association for Computa- tional Linguistics.
Yimu Wang, Shiyin Lu, and Lijun Zhang. 2020a. Searching privately by imperceptible lying: novel private hashing method with differential privacy. In Proceedings of the 28th ACM International Confer- ence on Multimedia, page 2700-2709
Yimu Wang, Ren-Jie Song, Xiu-Shen Wei, and Lijun Zhang. 2020b. An adversarial domain adaptation network for cross-domain fine-grained recognition. In 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1217-1225.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 3748-8763 PMLR.
Yimu Wang. Xiu-Shen Wei, Bo Xue, and Lijun Zhang. 2020c. Piecewise hashing: A deep hashing method for large-scale fine-grained search. In Pattern Recog- nition and Computer Vision Third Chinese Confer- ence, PRCV 2020, Nanjing, China, October 16-18, 2020, Proceedings, Part II, pages 432-444.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Yimu Wang, Bo Xue, Quan Cheng, Yuhui Chen, and Lijun Zhang. 2021. Deep unified cross-modality hashing by pairwise data alignment, In Proceedings

---

of the Thirtieth International Joint Conference on Artificial Intelligncee IJCAI-21, pages 1129-11 1129-1135.
Tiancheng Zhao, Xiaopeng Lu, and Kyusong Lee, 2021. SPARTA: Efficient Open-Domale Domain Question Answer- ing via Sparse Transformer Matching Retrieval. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguisties: Human Language Technologies, pages 565-575, Online. Association for Computa- tional Linguistics.
Xing Wu, Chaochen Gao, Zijia Lin, Zhongyuan Wang, Jizhong Han, and Songlin Hu. 2022. RaP: Redundancy- video-language pre-training for text-video retrieval. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3036-3047, Abu Dhabi, United Arab Emirates, As- sociation for Computational Linguistics.
Linchao Zhu and Yi Yang. 2020. Actbert: Learn- ing global-local video-text representations. In 2020 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 8743-8752. Computer Vi- sion Foundation IEEE.
Hu Xu, Gargi Ghosh, Po- Yao Huang, Prahal Arora, Ma- soumeh Aminzadeh, Christoph Feichtenhofer, Flo- rian Metze, and Luke Zettlemoyer. 2021a. VLM: Task-agnostic video-language model pre-training for video understanding. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. pages 4227-4239, Online. Association for Computa- tional Linguisties.
Hu Xu. Gargi Ghosh, Yo- Huang, Dmytro Okhonko. Armen Aghajanyan, Florian Metze, Luke Zettle- moyer, and Christoph Feichtenhofer. 2021b. Video- CLIP: Contrastive pre-training for zero-shot video- text understanding. In Proceedings of the 2021 Con- ference Empirical Methods in Natural Language Processing, pages 6787-6800, Online and Punta Cana, Dominican Republic. Association for Com- putational Linguistics.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016. MSR- VTT: A large video description dataset for bridging video and language.
2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 5288-5296. IEEE Computer Society.
Qiying Yu. Yang Liu, Yimu Wang, Ke Xu, and Jingjing Liu. 2023. Multimodal federated learning via con- trastive representation ensemble. In The Eleventh International Conference Learning Representa- tions.
Youngjac Yu, Jongseok Kim, and Gunhee Kim. 2018. joint sequence fusion model for video question answering and retrieval. In Computer Vision ECCV 2018 15th Earopean Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part VII, volume 11211 of Lecture Notes in Computer Science, pages 487-503. Springer.
Youngjae Yu, Hyungjin Ko, Jongwook Choi, and Gun- hee Kim. 2017. End-to-end concept word detection for video captioning, retrieval, and question answer- ing. In 2017 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 5261-3266 IEEE Com- puter Society.
Shuai Zhao, Linchao Zhu, Xiaohan Wang, and Yi Yang. 2022. Centerclip: Token clustering for efficient text- video retrieval. In Proceedings of the 45th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '22, page 970-981, New York, NY, USA. Association for Computing Machinery.

---

A Experiments
(R@@ @ K, higher is better), median rank (MdR, lower is better), and mean rank (MnR, lower is better) retrieval metrics, which are widely used in previ- ous retrieval works (Radford et al,, 2021; Luo et al., 2022; Ma et al., 2022a).
A.1 Datasets Details
MSR-VTT (Xu et al., 2016) contains 10,000 videos with length varying from 10 to 32 seconds, each paired with about 20 human-labeled captions. Following the evaluation protocol from previous works Yu et al. (2018); Miech et al. (2019), we use the training-9K / test Ik-A splits for training and testing respectively.
A.3 Ablation Studies
Evaluating the calculation of similarity between video and frame representations and cluster con- cepts in S3MA. In S3MA, we use cosine similar- ity to calculate sim! and sim". Another way of calculating sime and sim' might be using multi- label classification. To compare the effect of multi- label classification and cosine similarity, we con- duct experiments using two mullii-laeer perceptrons (MLPs) with two layers and the ReLU activation to predict the similarity between video and frame representations and cluster concepts. Two MLPs are also trainable. Quantitative results are shown in Table 8. Our quantitative results, shown in Ta- ble indicate that the use of MLPs decreases R @!) on text-to-vide video and video-to- toetottteexttt retrieval. This suggests that cosine similarity is more suitable for VTR.
MSVD (Chen and Dolan, 2011) contains 1,970 videos with a split of 1200, 100, and 670 as the train, validation, and test set, respectively. The duration of videos varies from 1 to 62 seconds. Each video is paired with 40 English captions. AetivityNet (Fabian Caba Heilbron and Niebles, 2015) is consisted of 20,000 Youtube videos with 100,000 densely annotated descriptions. For a fair comparison, following the previous setting (Luo et al., 2022; Gabeur et al., 2020), we concatenate all captions together as paragraph to perform video-paragraph retrieval task by concatenating all the descriptions of video. Performances are reported on the "vall" split of the ActivityNet.
Evaluating the importance of supervised align- ment in S3MA. In S3MA, the aligned sentence representation r is obtained from the text as in Eq. (1). This process aligns the sentence represen- tation based on the instruction of the text. By doing so, the aligned sentence representation r can serve as the supervision (an anchor) for aligning video and frame features, providing a reference point for the alignment of different modalities. To investi- gate the importance of placing an anchor r² for bet- ter alignment, we compare it to obtaining aligned sentence representation through the similarity be- tween concept clusters C and sentence feature r². This altemattve approach allows to evaluate the effectiveness of using anchor for alignment and to understand how it improves the performance of the model. To investigate the alternative approach of obtaining aligned sentence representation with- out an anchor, we calculate the sentence sparse space similarity súm' E R xxne by calculating the cosine similarity between sentence representations and concepts sim; cos(r", Cj), where sim, is the j-th element of sim". C, is the j-th row of C, and cos is the cosine similarity. The aligned sen- tence representation I$ without the instruction of text is obtained by matrix multiplication as follows:
A.2 Implementation Details and Evaluation Protocols
Following Luo et al. (2022); Ma et al. (2022a), we use a standard vision transformer (Dosovitskiy et al., 2021) with 12 layers which are initialized with the public CLIP (Radford et al., 2021) check- points. We directly use the text encoder of CLIP as our text encoder which is also initialized with the public CLIP checkpoints.
We set the query, key, and value projection di- mension size as 512 to match CLIP's output di- mension and we initialize our logit scaling param- eter À with the value from the pre-trained CLIP model. All models optimized for 5 epochs on VST and MSVD, and for ActivityNet, the models are trained for 20 epochs. We use AdamW (Loshchilov and Hutter, 2019) with a weight decay of 0.2 and decay the learning rate using a cosine schedule (Loshchilov and Hutter, 2017), following the method used in CLIP (Rad- ford et al., 2021). For all experiments, we uni- formly sample 12 frames from every video, re- sizing each frame to 224x224 as per previous works (Luo al., 2022; Ma et al., 2022a). we set Noods = 1024 following DiscrekeCodebook (Liu et al., 2022a). To evaluate the retrieval performance of our proposed model, we use recall at Rank K
r² = -sim'C siimC/||ssimill
(4)

---

Text-to- Text-to-Video Retrieval
Vidco-to- Retrieval
R@11
R
R 107
MdR
MnR.)
R@ 11
R@55
R 101
MdR,
MnR
S3MA (iT-B/322)
multi-label classification
47.0
73.6
82.9
2.0
12.5
45.5
73.8
82.8
2.0
S3MA (VIT-B/32)
cosine
49.1
73,9
82.8
2.0
13.5
46.9
73.8
82.1
2.0
9.3
Table 8: Ablation study the calculation of similarity between video and frame representations and cluster concepts.
Text-to- Retrieval
Video-to- Retrieval
R@
R@5t
101
MdR.
MnR
R@
R@51
101
MdR.
MnR,
S3MA (VIT-B/32) w/o anchor
47.8
72.9
82.3
2.0
13.4
46.4
74.9
82.1
2.0
9.1
S3MA (VIT-B/32) w. anchor
49.1
73,9
82.8
2.0
13.5
46.9
73.8
82.1
2.0
9.3
Table 9: Ablation study on the instruction of text, i.e., generating r using the similarity or the text. "w. anchor" refers obtain r. by text Eq. (I). "w/o anchor" refers obtain r. by the similarity between sentence representations and concepts C' as Eq. (4)
Text-to- Video Retrieval
Video-to-Text Retrieval
Ealiyn
L Latignaio
@@ It
R@51
R@ 101
MnR
M
R@ 11
R@51
R@ 101
MnR
MeanR
48,0
72.9
82.4
2.0
13.5
45,4
73.2
82.1
2.0
9.3
48.0
73.5
82.7
2.0
13.4
47.1
74.2
82.9
2.0
9.1
47.4
73.5
82.7
2.0
13.5
46.8
73.2
82.2
2.0
9.2
49.1
73.9
82.8
2.0
13.5
46.9
73.8
82.1
2.0
9.3
Table 10: Ablation study of latign and falignsim MSR-VTT based S3MA (VIT-B/32).
Text-to-Vides Retrieval
Video-to-Text Retrieval R@ 101
8
@@ 11
R@51
R@ 101
MnR
MeanR.
R@ 11
R@51
MnR
MeanR.
0.02
0.01
49.1
73.9
82.8
2.0
13.5
46.9
73.8
82.1
2.0
9.3
0.02
0.02
48.5
73.8
83.2
2.0
14.0
46.3
73.1
82.1
2.0
9.4
0.02
0.05
47.6
72.7
82.4
2.0
14.0
45.8
74.0
82.2
2.0
9.2
0.02
0.1
47.7
72.3
82.9
2.0
13.4
45.3
73.6
83.3
2.0
9.0
0.01
0.01
47.6
74.0
82.7
2.0
13.8
46.7
73.5
82.2
2.0
9.5
0.05
0.01
48.1
73.6
83.1
2.0
13.2
46.3
72.9
82.7
2.0
9.1
0.1
0.01
47.9
74.2
82.3
2.0
13.3
46.3
73.4
82.5
2.0
9.1
Table 11: Ablation study and /3 on MSR-VTT based on S3MA (ViT-B/32).
Deuse Space
Sparse Space S-F
Text-to- Text-to-Video Retreval
Video-to -oxt Retieval
S.V
S.F
W.V
W-F
S.V
W.V
W-F
R@11
R
R 10)
MdRJ
MnR.
R@11
R@:51
R 101
MdRj
MnR
/
49.1
73.9
82.8
2.0
13.5
46.9
73.8
82.1
2.0
9.3
13.0
74.1
82.1
C-PI
13.2
Table 12: Retrieval performance with different similarities MSR-VTT using S3MA with the base model of ViT- B/32. "S-V", "S-F", "W-V", and "W-F" represent Sentence-Vicee (coarse-grained), Sentence-Frames (fine-graine)), Word-Vide (fine-grained), and Word-Frame (fine- (ine-grained) similarities.
where sim! is the similarity between sentence rep- resentations and concepts. The results of this com- parison can be found in Table 9. The experimental results show that with the "anchor". S3MA can better align different modalities as R@ R@5, and R @ 10 on text-to-video retrieval and R @ on video-
to-text retrieval have greatly improved, indicating that the supervised (anchor-based) alignment is cru- cial for better performance of the model.
Effect of losses and hyperparameter sensitiv- ity. To further demonstrate the effectiveness of the

---

Topl: child in pink watches w white bird in open box
Topl: man tulks in fiont of'a green bicycle
Top2: an animal is throwing piece junk
Top2: man talks about cars
Top3: puppy is crawling down some stairs
Top3: people talking about a fight
Topd: two parrots bird cage one white chick on green adult
Top4: two people are preparinn for sports
Topd: the house has at least three small pets
Top3: guys bolding cups and talking
SER N SER SERN SERIUS
Topl: sports vine clips foothall
Topl: yelling the phune
Top2: vine sports compilation
Top2: music video shut up bunch of times
Top3: vines of sports are being played
Top3: man with very red nose
Top4: it vine compilation
Top4: bunch of cartoon faces are chomping their teeth and making eating gestures
Tops: compilation vine videns shown
Top5: differrent sound
coming out sounding the way they
Figure 6: Top-5 video-to-text retrieval results on MSR-VTT.
while other friends bitting basket moother cager
basketball players making seven
achieve successful baskethall
Topl
Topl
Top2
Top2
Top3
Top3
interv about her happening brazil
discusses spongeboh
Top1
Topl
Top2
Top2
Top3
Figure 7: Top-3 text-to-video retrieval results on MSR-VTI -TT..
Figure 8: The activation of 20 sparse concepts by 100 randomly selected videos.
two proposed losses designed for aligning different modalities in the shared sparse space, conduct experiments compare the performance of these losses. The quantitative results of these experi- ments are shown Table 10. The results indicate that adding both losses simultancously achieves the best performance on the MSR-VTV dataset. When using only one loss, the performance on text-to-
video retrieval is comparable to the method with- out using both losses text-to-video retrieval, but outperforms the method without the two losses on video-to-text retrieval. Specifically, when us- ing two losses, R@ I on text-to-video retrieval and vidco-to-text retrieval is improved by 1.1 and 1.5, respectively. Additionally, all the other metrics, such as R and R@ 10, are also improved, demon-

---

strating the power of the two proposed losses in aligning different modalities in the shared sparse space. To gain a better understanding of the sensi- tivity of S3MA with respect to the two hyperpa- rameters, a and B, we conduct series of experi- ments with different settings of a and 3 as shown in Table 10. The results of these experiments demon- strate that, even with varying settings of a and B, the video-text retrieval performance remains con- sistent, indicating that the model robust and not highly sensitive to these hyperparameters. This suggests that S3MA is able to achieve good per- formance across a wide range of settings for these hyperparameters, making it easy to adjust and op- timize for specific use cases. Additionally, this also suggests that S3MA is not overly dependent precise values of these hyperparameters, and is instead able to leverage the more important under- lying features and patterns in the data.
S3MA performs well in visual and textual con- tent understanding, benefiting from multi-space and multi-grained similarity.
Moreover, we visualize the activation of sparse concepts by videos in Figure 8. We notice that, some hub sparse concepts are frequently retrieved while some are not retrieved lot, which might be due to the KNN clustering. Moreover, we notice that the difference between activations from videos are separable.
Are word-level features necessary? To investi- gale the necessity of word-level features, we intro- duce word-level dense and sparse representations, along with word-frame and word-video similari- ties, into the dense and sparse spaces. The results are presented in Table 12. Notably, we observe a decrease in performance when incorporating word- level contrast in both dense and sparse spaces, in- dicating possible feature redundancy. Moreover, approach, which incorporates word-level con- trast, can be viewed an extension of X-CLIP (Ma al., 2022b) with the shared sparse space. We no- tice that contrasting representations the aligned sparse space enhances the retrieval performance of X-CLIP.
A.4 Aligning Examples
To show the effectiveness of S3MA, we illustrate some examples of video-to-text and text-to-video retrieval examples in Figures 4, and 7. We notice that S3MA is able align some important con- cepts between video and text for precise retrieval. For example, in the bottom-left iideo-to-text result (Figure 6). the biggest difference between the top retrieved texts is "football". By precisely capturing "football" in the video, S3MA is able to give higher logits to the sentences that contain "football". Ad- ditionally, in the last (bottom-right) text video result (Figure 7), we notice that, by understanding "man" and "discuss". S3MA is able to distinguish the top 3 retrieved videos and select the one in which a man appears. This empirically shows that


Reviews:
[{"review_id": "LohakslYF7", "reviewer": "Reviewer_Vvmb", "summary": "", "questions": "A. Table 6 showcases that the most optimal performance is attained by employing the multi-space multi-grained similarity computation. Nonetheless, it is important to underscore that the analysis regarding the influence of the introduced sparse space does not encompass its individual performance outcomes.\n\nB. Referring to your assertion of achieving the state-of-the-art (SOTA) status, it might be appropriate to reconsider. To the best of my knowledge, several other published methods have surpassed your results, including CLIP-ViP [1], Cap4Video [2], DRL [3], and TemPVL [4].\n\n[1] Xue, Hongwei, et al. \"Clip-vip: Adapting pre-trained image-text model to video-language representation alignment.\"\n[2] Wu, Wenhao, et al. \"Cap4Video: What Can Auxiliary Captions Do for Text-Video Retrieval?\"\n[3] Wang, Qiang, et al. \"Disentangled representation learning for text-video retrieval.\"\n[4] Ma, Fan, et al. \"Temporal perceiving video-language pre-training.\"", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time, efforts, and your detailed and positive comments. We have carefully considered your comments and have provided responses to specific comments.\n\n**Questions A, Reasons To Reject [Performance of only using sparse space]:**\n\n> Thank you for your valuable question! We agree that the performance comes from **multi-grained multi-space similarity** (Sec. 3.3 in the main paper). We did run the experiments only using sparse space and the Text-Video R@1 on MSR-VTT is only 38.4 which is due to the random initialization of the sparse space and by the small number of learned concepts. However, the sparse space does have a great explainability as each concept has meaning. \n\n> The dense space (vanilla CLIP4Clip) excels in representation learning but is limited in terms of explainability. Conversely, the sparse space offers excellent explainability due to its concept-based nature, albeit with limitations in representation learning as the number of concepts is limited. And similar conclusion can be obtained from dense and sparse spaces [5,6,7]. By integrating both spaces, we aim to leverage the best of both spaces, achieving superior retrieval performance while retaining explainability.\n\n> We will include this analysis and the corresponding results in the final version of our paper. Moreover, we will consider revising the title of our paper. Thank you for your suggestion again! That is valuable to us.\n\n**Questions B [Related works]:**\n\n> Thank you for your detailed comments! We will add detailed comparisons with these papers in related works and experiments in the final version of our paper. But note that, our proposed method is the **SOTA without any auxiliary data, models, or post-processing methods**. Our S3MA achieves **51.7 R@1 on Text-to-Video (ViT-B/32)** on MSR-VTT with DSL [8]. We will carefully revise our paper to avoid any confusion and add related experimental results in our final version of the paper. \n\n> We acknowledge the improved performance achieved by models [1,2,3] leveraging auxiliary data, models, or post-processing methods achieves better retrieval performance, while TemPVL[4] is a pre-training task with lower retrieval performance on MSR-VTT. However, we will revise our paper and **carefully compare with these papers [1,2,3,4] for fair comparison** in our final version of the paper. \n\nWe sincerely thank you for your thoughtful comments and are open to further discussion on these matters.\n\n[1] Xue, Hongwei, et al. \"Clip-vip: Adapting pre-trained image-text model to -language representation alignment.\"\n\n[2] Wu, Wenhao, et al. \"Cap4: What Can Auxiliary Captions Do for Text- ?\" \n\n[3] Wang, Qiang, et al. \"Disentangled representation learning for text- .\" \n\n[4] Ma, Fan, et al. \"Temporal perceiving -language pre-training.”\n\n[5] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. [Dense Passage Retrieval for Open-Domain Question Answering](https://aclanthology.org/2020.emnlp-main.550). In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*.\n\n[6] Kyoung-Rok Jang, Junmo Kang, Giwon Hong, Sung-Hyon Myaeng, Joohee Park, Taewon Yoon, and Heecheol Seo. 2021. [Ultra-High Dimensional Sparse Representations with Binarization for Efficient Text ](https://aclanthology.org/2021.emnlp-main.78). In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP).*\n\n[7] Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. 2021. [Sparse, Dense, and Attentional Representations for Text ](https://aclanthology.org/2021.tacl-1.20). *Transactions of the Association for Computational Linguistics*, 9:329–345.\n\n[8] Xing Cheng, Hezheng Lin, Xiangyu Wu, Fan Yang, and Dong Shen. 2021. Improving video-text retrieval by multi-stream corpus alignment and dual softmax loss. CoRR, abs/2109.04290.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The conducted experiments aptly showcase the effectiveness of the proposed method. Notably, this paper stands out for its comprehensive ablation study and meticulous analysis of each individual module.\n", "reasons_to_reject": "In essence, this paper presents an integration of global video-text and local frame-text elements within both the introduced sparse space and the original dense space, all aimed at enhancing video retrieval. However, it's worth noting that the impact of the introduced sparse space alone is not thoroughly elucidated in the analysis. The method itself should be multi-space multi-grained learning framework for video retrieval.\n", "questions_to_authors": "A. Table 6 showcases that the most optimal performance is attained by employing the multi-space multi-grained similarity computation. Nonetheless, it is important to underscore that the analysis regarding the influence of the introduced sparse space does not encompass its individual performance outcomes.\n\nB. Referring to your assertion of achieving the state-of-the-art (SOTA) status, it might be appropriate to reconsider. To the best of my knowledge, several other published methods have surpassed your results, including CLIP-ViP [1], Cap4Video [2], DRL [3], and TemPVL [4].\n\n[1] Xue, Hongwei, et al. \"Clip-vip: Adapting pre-trained image-text model to video-language representation alignment.\"\n[2] Wu, Wenhao, et al. \"Cap4Video: What Can Auxiliary Captions Do for Text-Video Retrieval?\"\n[3] Wang, Qiang, et al. \"Disentangled representation learning for text-video retrieval.\"\n[4] Ma, Fan, et al. \"Temporal perceiving video-language pre-training.\"", "paper_topic_and_main_contributions": "This paper introduces a multi-grained sparse learning framework designed to acquire a shared, aligned sparse space for the purpose of video-text retrieval tasks.\n\nThe authors adopt a supervised approach to learning and continuously updating the shared sparse space for text and video representations. This is achieved through the incorporation of the proposed similarity and alignment losses. Furthermore, the paper suggests incorporating a multi-grained similarity approach in the context of video retrieval tasks.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zV1gFrvzZ5", "reviewer": "Reviewer_VnYa", "summary": "", "questions": "1- How this method can handle rare and frequent concepts?\n2- Is the cluster center selected as the representative?\n3- Is there a way to take advantage of the cooccurrence of cluster words?\n4- Why the Dense space similarity needed?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time, efforts, and your detailed and positive comments. We have carefully considered your comments and have provided responses to specific comments.\n\n**Questions 1 [Rare and frequent concepts]:**\n\n> Thank you for your valuable questions. S3MA does not consider the frequency of words or concepts and treats both rare and frequent words or concepts equally throughout the initialization and updates. We believe that a fair or weighted strategy towards different concepts would indeed yield enhanced retrieval performance. We acknowledge the potential benefits of such an approach and have included this aspect as part of our future work. This discussion can be found in the limitations section (Line 543 to Line 553).\n\n**Questions 2 [Is the cluster center selected as the representative?]:**\n\n> Correct. During training and inferencing, the cluster centers effectively represent the words that fall within their corresponding clusters.\n\n**Questions 3 [Take advantage of the cooccurrence of cluster words?]:**\n\n> Thank you for your valuable question! We share your view that the clustering procedure could be refined by using better advanced strategies such as (cooccurrence-)weighted KNN, semantic-based KNN, or part-of-speech tagging-based KNN. We have discussed this in the section of Limitations (Line 543 to Line 553). Moreover, we are actively working on experimenting with these enhanced KNN strategies, and we will present the corresponding results in the final version of our paper. \n\n**Questions 4 [Why the Dense space similarity needed?]:**\n\n>Thank you for your valuable question! The reason why we are incorporating the dense and sparse spaces is that they excel in different aspects. The dense space (vanilla CLIP4Clip) excels in representation learning but is limited in terms of explainability. Conversely, the sparse space offers excellent explainability due to its concept-based nature, albeit with limitations in representation learning as the number of concepts are limited. And similar conclusion can be obtained from dense and sparse retrieval [1,2,3]. By integrating both spaces, we aim to leverage the best of both spaces, achieving superior retrieval performance while retaining explainability.\n\n>Moreover, we did run the experiments only using sparse space and we noticed that the Text-Video R@1 on MSR-VTT is only 38.4 which is due to the random initialization of the sparse space and the small number of learnt concepts. But it provides meaningful explanations for each concept. \n> We will include this analysis and the corresponding results in the final version of our paper. \n\n**Reasons To Reject [Success of the networks might be mostly achieved by embeddings without clustering.]:**\n\n> Thank you for your valuable comment! We reported the performance without clustering in Table 4 (Page 7) in our main paper. We notice that the **performance without clustering (48.7 on Text-to-Video R@1) is slightly lower that the performance with clustering (49.1 on Text-to-Video R@1)**. Nevertheless, clustering brings superior performance and faster inference times as the number of concepts (1024 in our experiments) is much small than the number of sub-words (approximately 30k). We agree that clustering will benefit training by acceleration and concentration of gradients while the improvements brought by clustering are limited as shown in Table 4.\n\nWe sincerely thank you for your thoughtful comments and are open to further discussion on these matters.\n\nReferences:\n\n[1] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. [Dense Passage Retrieval for Open-Domain Question Answering](https://aclanthology.org/2020.emnlp-main.550). In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*.\n\n[2] Kyoung-Rok Jang, Junmo Kang, Giwon Hong, Sung-Hyon Myaeng, Joohee Park, Taewon Yoon, and Heecheol Seo. 2021. [Ultra-High Dimensional Sparse Representations with Binarization for Efficient Text Retrieval](https://aclanthology.org/2021.emnlp-main.78). In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP).*\n\n[3] Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. 2021. [Sparse, Dense, and Attentional Representations for Text Retrieval](https://aclanthology.org/2021.tacl-1.20). *Transactions of the Association for Computational Linguistics*, 9:329–345.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Interesting idea.\nImportant problem.\nThe solution can be applied to different types of visual-textual applications, for example phishing detection on social media, recommendation systems, and etc.\nSuperior results.\nWell written.", "reasons_to_reject": "KNN clustering to find concepts given words, makes the approach biased towards the word embeddings of he up-stream models. It can be claimed that the success of the networks is mostly achieved by the initial words embeddings, before clustering.", "questions_to_authors": "1- How this method can handle rare and frequent concepts?\n2- Is the cluster center selected as the representative?\n3- Is there a way to take advantage of the cooccurrence of cluster words?\n4- Why the Dense space similarity needed?", "paper_topic_and_main_contributions": "The authors have developed an efficient, sparse method for correlating textual and visual data within a unified conceptual framework. The most important novelty of the paper, in my opinion, is to cluster the textual tokens into concepts, and align the visual and textual representations in that space.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bjd85sbyKN", "reviewer": "Reviewer_UH8H", "summary": "", "questions": "see above", "limitations": "", "rating": 2, "confidence": 3, "soundness": 1, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time, efforts, and your detailed comments. We have carefully considered your comments and have provided responses to specific comments.\n\n**Reasons To Reject [1, What is sparse space?]:**\n\n>Your question is highly valuable, and we're grateful for the opportunity to clarify this aspect. As stated in the Abstract, ``The shared sparse space is initialized with a finite number of sparse concepts, each of which refers to a number of words`` (Line 7 to Line 9). Additionally, the definition of sparse space in our paper closely aligns with that commonly employed in text retrieval (Line 58 to Line 64 in the introduction). Recognizing the importance of conceptual clarity, we will present a more detailed and comprehensive explanation of the sparse space in our main paper, particularly within the introduction to facilitate readers' understanding of our models and the essence of our work.\n\n**Reasons To Reject [2, Why not verify the effect on other multi-modal video language tasks such as video question answering, video-grounded reasoning/dialogue.]:**\n\n>Thank you for your question. Choosing video-text retrieval is that during the inferencing, we can **directly and explicitly understand how the sparse space aids in aligning diverse modalities**, as evident in Figures 3 to 7 within our paper. These figures indicate that incorporating the sparse space does help with better aligning different modalities thus yielding better retrieval performance. As for other tasks, we have **acknowledged this potential within the limitations** section. We would like to explore the potential of the sparse space in other tasks as future work.\n\n**Reasons To Reject [3, Visualization of the effectiveness]:**\n\n>Thank you for your question, and we are delighted to direct your attention to our presented **retrieval examples showcased in Figures 3 to 7** within our main paper, appendix, and **Section 4.4 Qualitative results**. Through these visualizations, our proposed S3MA exhibits its exceptional capability in accurately identifying attributes such as color (e.g., green), objects (e.g., bicycle), and humans (e.g., a man), indicating its proficiency in capturing intricate details. In Figure 4, we notice that, the video and frame features are perfectly aligned with the corresponding sparse concepts as exhibiting high similarities.\n\nWe sincerely thank you for your thoughtful comments and are open to further discussion on these matters.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The best performances in three benchmarks in video-text retrieval\nSufficient experiments on the retrieval dataset", "reasons_to_reject": "[-] Poor writing. To use the word 'Sparse', it is necessary to first specify the sparsity on what (e.g., quantities, distribution?). When reading the Introduction, I didn't know the sparsity of what and also didn't know why the concept of sparsity is needed or what it does.\n\n[-] If the contributions of this paper are based on the representation enhancement of shared spaces about heterogeneous modalities, it is more convincing to validate the approach in several multi-modal video language tasks such as video question answering, video-grounded reasoning/dialogue. Why only experiments on text-video retrieval? Haven't you verified the effect on several other multi modal tasks?\n\n[-] Does the author can visualize the effectiveness of the proposed method?", "questions_to_authors": "see above", "paper_topic_and_main_contributions": "This paper proposes the shared space to alleviate the problem by the representation mismatches among the modalities (i.e., video and text).", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This aper presents a framework in which sparsification is employed to efficiently correlate and align textual and visual data for improved video-text retrieval. The most novel contribution of the paper is to cluster the textual tokens into concepts, and align the visual and textual representations in that space. Reviewers agree this presents an interesting idea, and is being applied to an important problem, while being applicable to further types of visual-textual applications, for example social media, recommendation systems, and etc. Authors achieve good results and discuss relationship with related work in-depth during the rebuttal period. Overall the paper is well written, and would be further strengthened by the incorporation of the feedback and the discussion. 

One reviewer mentions that a non-anonymized version of the paper has been published *before* the anonymization period, although I was not able to find it - but this would be fine in any case.
While reviewers are split in their evaluation, the AC agrees with the reviewers that explicitly state that they find this paper "worthy of acceptance" and that evaluation on additional datasets/ tasks does not seem warranted. Given the strength of the results, the AC's recommendation is for acceptance to Findings, while acceptance to the main conference would require stronger results (either better on-task performance, or other clear advantages of the approach such as efficiency).
Decision: Accept-Findings