Text-Adaptive Multiple Visual Prototype Matching for Video-Text Retrieval
Authors: Chengzhi Lin
Ancong Wu
Junwei Liang
Jun Zhang
Wenhang Ge
Wei-Shi Zheng
Chunhua Shen
Venue: NeurIPS 2022 Accept (2022)
Keywords: Video-Text Retrieval

Abstract: Cross-modal retrieval between videos and texts has gained increasing interest because of the rapid emergence of videos on the web. 
Generally, a video contains rich instance and event information and the query text  only describes a part of the information. Thus, a video can have multiple different text descriptions and queries. We call it the Video-Text Correspondence Ambiguity problem. Current techniques mostly concentrate on mining local or multi-level alignment between contents of video and text (e.g., object to entity and action to verb). It is difficult for these methods to alleviate video-text correspondence ambiguity by describing a video using only one feature, which is required to be matched with multiple different text features at the same time. To address this problem, we propose a Text-Adaptive Multiple Visual Prototype Matching Model. It automatically captures multiple prototypes to describe a video by adaptive aggregation on video token features. Given a query text, the similarity is determined by the most similar prototype to find correspondence in the video, which is called text-adaptive matching.  To learn diverse prototypes for representing the rich information in videos, we propose a variance loss to encourage different prototypes to attend to different contents of the video.  Our method outperforms the state-of-the-art methods on four public video retrieval datasets.

Text-Adaptive Multiple Visual Prototype Matching for Video-Text Retrieval
Chengzhi Lin', Ancong Wu' Junwei Liang²
Jun Zhang', Wenhang Get, Wei-Shi Zheng Chunhua Shen School of Computer Science and Engineering, Sun Yat-sen University, China AAI Thrust, Hong Kong University of Science and Technology (Guangzhou)
Tenneett Youtu Lab
*Guangdong Province Key Laboratory of Information Security, PR China KKey Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China
'Zhejiang University
Email: linchzh3 mattiihhh@mail.syyyy... myi.l. enu wuanc @ mail.sysu.eduu edu.cn, junweiliang hkust-gz. edu.cm
bobbyjzhang wew.eettttcnccccm gewh @ maa.2..yysu.du..a.m wszheng @ ieee.org, chunhua me.com
Abstract
Cross-modal retrieval between videos and texts has gained increasing research in- terest due the rapid emergence of videos on the web. Generally, a video contains rich instance and event information and the query text only describes a part of the information. Thus, a video can correspond multiple different text descriptions and queries. We call this phenomenon the "Vidco-Tor Text Correspondence Ambiguity" problem. Current techniques mostly concentrate on mining local or multi-level alignment between contents of a video and text (e.g., object to entity and action to verb). It is difficult for these methods alleviate the video-text correspondence ambiguity by describing video using only one single feature, which is required to matched with multiple different text features at the same time. To address this problem, we propose Texi-Adaptive Multiple Visual Prototype Matching model, which automatically captures multiple prototypes to describe a video by adaptive aggregation of video token features. Given a query text, the similarity is deter- mined by the most similar prototype to find correspondence in the video, which is termed text-adaptive matching. To learn diverse prototypes for representing the rich information in videos, we propose a variance loss to encourage different prototypes attend to different contents of the video. Our method outperforms the state-of- siie-oofthee f--th--an art methods on four public video retrieval datasets.
Introduction
Cross-modal retrieval has gained popularity as a means of matching semantically similar samples across multiple modalities [12 3 4. With the rapid increase in the number of videos, Text-Video Retrieval (TVR) has garnered a lot of interest. Given a text description/video, we aim to retrieve the corresponding video/text description in the gallery. TVR remains a challenging task since the video and text have different semantical descriptions which results in modality gap.
To overcome the modality gap issue, most existing methods generally follow
dual encoding network paradigm which encodes video and text through a visual encoder and a language encoder into the same embedding space, respectively [ 2
3 5 6). And then they focus on learning similarity function between the representations of text and video, which provides the basis for ranking the gallery video/text. However, the modality gap issue is still not well solved due to the misalignment.
"Correspondigg author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Some Frames of a Vidco
Some Corresponding Text Descriptions
a clip showing a woman riding a horse on a beach
man speaks to crowd about
group of people yelling
man who buys gifts for his girlfriend
Figure example text-to-video retrieval, video can described by different texts, because different persons focus on different characters when describing the videos. In this paper, multiple video prototypes matching is utilized to help mitigate this issue.
Some methods aim extract additional information from the video get more fine-grained video features to be aligned with corresponding text features 7 8]. They focus on properly combining different video modalities, such as appearance, audio, speech, etc. Some methods pay attention to multi-level alignment, including text entity to video object, text verb to video action, and text phrase video event [6 99. However, most of them ignore the fact that vidco contains rich instance and event information, while the query text only describes a part of the information, which leads to video-text correspondence ambiguity problem. Concretely, as shown in Figure I due to the multiple characters in the video, the video has different text descriptions: "a woman does something". "a man does something" or "a group does something". The text descriptions are very diverse and scattered in the feature space. With diverse video contents, it is infeasible to align a single video feature with the diverse text features. Neither additional information-bant methods nor multi-level alignment- methods consider such correspondence ambiguity since both of them match a single video feature with multiple text features.
In this work, we focus on video-text retrieval problem and aim to mine fine-grained and diverse event descriptions in a video and adaptively match them with multiple text descriptions mitigate correspondence ambiguity issue for better alignment. Specifically, we automatically capture multiple prototypes for each video to account for its rich instance and event information by aggregating on token features. Based on multiple prototypes of video, we aim at finding many- souny-to-man to-many correspondence between prototypes and text representations reduce ambiguity. To this end, we propose Aext--daptive Multiple Visual Prototype Matching Model. We follow the dual encoding network paradigm as previous methods 3 First, we input video into visual encoder to get the video token features. Then we employ loken-adaptive aggregation method to generate several visual prototypes from the final video token features. As shown in Figure 2 one prototype is the class token and the others are the linear combinations of all the token features. The text feature encoded by the language encoder matched with the most similar video prototype. This text-adaptive matching can help find text-to-prototype correspondence in video and alleviate the correspondence ambiguity. To further force the prototypes to represent rich information in the video, we propose a variance loss to enhance the diversity of prototypes to attend to different contents of the video.
Extensive experiments were conducted on the MSR-VTT T0], DIDEMO 11. MSVD 12, and LSMDC [13 to evaluate video-text bi-directional retrieval performance. Experimental results show that our proposed model can achieve state-of-the-an results on all the above datasets. Ablation studies were carried out to evaluate the effectiveness of each single part of our model.
The main contributions of our work are three folds. We propose automatically generate multiple prototypes for video to account for its rich information and introduce a text-adaptive matching strategy to adaptively find correspondence between texts and prototypes. 2) We further design variance loss to effectively encode diverse information into prototypes to attend to different contents in the video and boost alignment. 3) Our proposed model can achieve new state-of-the-ar results on four benchmark datasets. Extensive experiments demonstrate its effectiveness and generalization.
2 Related Work
Text- Video Retrieval. The methods for text- video retrieval can classified into three categories: 7
pre-extracted feature based [8]
6
14 15
[6), end-to-end training [3] 2 and adaptive matching

---

methods. Given the small size of most video-text retrieval datasets, the dominant paradigm for video retrieaal has been to combine pre-extracted features from "expert" models, such as models trained for a variety of tasks and on multiple modalities such as face, scene, and object recognition, action classification, and sound classification. CE [ and MMT [ compute the overall similarity for a video-text pair obtained as a weighted sum of each expert's is similarity with the text. DMM 7 uses hierarchical video-text alignment to explore the diverse properties in multi-modal cues for fine- grained aligmment. Another important paradigm is to first train on a large-scale vision-and-language dataset and then finetune on downstream dataset. ClipBERT [2] proposes an efficient end-to-end technique based sparse sampling and demonstrates that the pretrained model by image-langen langugl dataset enables superior video-text retrieval initialization. Frozen 31 proposes an end-to-end trainable model that can benefit from large-scale image and video captioning datasets. Some works focus on the matching of different modalities. PCME 4 represents the samples from different modalities as probabilistic distributions in the common embedding space to capture one-to-many correspondences. SSVR I7 highlights the issue of one-to-one assumption in evaluation of video retrieval methods and proposes new evaluation metric address this. MQVR [18] tackles the issue of imperfect annotations in existing video retrieval datasets by focusing on the less-studied setting of multi-query video retrieval. Most of them, however, ignore the fact that a video contains rich instance and event information, whereas the query text only describes a portion of the information, resulting in video-text correspondence ambiguity. We propose text-Adaptive Multiple Visual Prototype Matching Model to help mitigate this issue.
Vision Transformer. Now vision transformer becomes the visual basic backbone in most of the state-of-the-are methods for vision-language tasks [9
2 3 [. BLIP [T] makes effective use of noisy web data by bootstrapping captions, in which captioner generates synthetic captions and filter removes the noisy ones. The work of Frozen []] makes a minor modification to the Divided
Space-Time attention introduced by 200] Our model is based on the dual framework of Frozen.
3
Our Method
video, in general, contains a lot of rich instance and event information, and the text description only describes a portion of it. A video can have many charaatorss/eeenissaaiion,, whereas a text typically only describes one character, action, or event. So mining representation of multiple essential events facilitates the text-video matching by reducing the modality gap properly. As a result, a model that generates multiple visual prototypes is better than that generates one feature. With this rationale in mind, we propose our Adxttide Multiple Visual Prototype Matching Model. To generate several visual prototypes from video, we use a token-adaptive aggregation method on the final video token features. Given a query text, the most similar prototype is used to find correspondence in the video, process called as text-adaptive matching. We also propose variance loss to encourage different prototypes to attend to different contents of the video in order to learn diverse prototypes for representing the rich information in videos. We define the problem in Section 3.1, present Texp-Adaptive Multiple Visual Prototype Matching Model in Section 3.2, describe our variance loss in Section 3.3, and explain our dual encoder in Section 3.4.
3.1 Problem Definition
For the video-text retrieval task, we are given M videos V -V and N text descriptions T = (T. The goal of this task is to learn similarity function S : (T + R. It should maximize positive cross-modal sample similarity while minimizing negative cross- sample similarity. In general, we fed video and text to into an vision encoder and language encoder to get normalized feature f", Then the similarity function S is defined by
St, fr).
The similarity of text-video is calculated as the inner production of f"
3.2 Text-Adaptive Multiple Visual Prototype Matching Model
The similarity function of Equation D ignnrres the Video-Text Correspondence Ambiguity problem: some videos have multiple different text descriptions and queries. To furthermore illustrate this problem, we show the text descriptions diversity of videos by showing the inter-text similarity and
3

---

Token Feature Prototypes
cls
0.5
Video
Similarity
Video
VIT
Proj
0.9
MAX
0.9
0.1
Text
Text
BERT
cls
Proj
Inner Production
Figure 2: Workflow of method. We feed video into vision transformer get the token features. Then employ token-adaptive aggregation method to generate several visual prototypes from the final video token features. The text feature from the language model is matched with the video prototype which most similar to text after the projection layer.
the minimum of intra-simin similarity distribution in 3 We define inter-text similarity as the similarity of different text descriptions for different videos, intra-text similarity as the similarity of different text descriptions for same video. The less minimum of intra-text similarity, the diversity of text descriptions for a video. As shown in Figure 3 at least one-third of the minimum of intra-text similarity less than the mean of inter-text similarity on features from both models. It demonstrates that many videos have diverse captions and contents. So single feature in some videos cannot be aligned with multiple text features at the same time. As a result, using multiple visual prototypes to align with different text is critical.
intuitive idea for producing multiple visual prototypes to generate a feature for each frame/clip of video as a prototype. However, one frame/clip can contain multiple characters and events at the same time. So better way to generate many potential prototypes for each frame, then we aggregate them together into multiple visual prototypes. Vision Transformer [3] [] have shown it's effectiveness on learning multi-modality shared representation. So we take the final layer token features E B of vision transformer as the potential prototypes. B, D represents the number of tokens, the dimension of token feature.
There are many methods to aggregate the token features to produce multiple visual prototypes, such as transformer decoder, graph neural network, LSTM etc. However, we find that the mask-based production of the prototypes is good enough. The kth prototype
B
Pk Σ 2 pmash (2)) j=1
where 2², fmask represents the jth token feature of mask-generating function. fmask is imple- mented by a linear layer, followed by relu function. In general, the class token feature is regarded the final output of vision transformer 3 2] So expand pk+1 als which means K + visual prototypes. For comparing visual prototypes and text feature on same embedding space, imput p projection linear layer, do normalization and get multiple vision features f E (K+ Matching text to video becomes matter of matching text feature to multiple visual prototypes. The text is aligned with the video if one the prototypes is very close to it. Based on this text-adaptive matching, we have new similarity function
have
TTMVM 1,0
K+1 max k=
m
fFe)
(3)
3.3 Variance Loss
Define mask m = fmask ).k which represents the mask weight on jth token from kth prototype of ith sample. : It's intuitive idea to different prototypes of same video to attend different contents and the masks have different attention different tokens. However, shown in Figure in some

---

similarity
intereed similntty
Danslly
moan of intorted simiarit
Deasiity
022
29
2.4
-04
82 Similariy
Similarity
Figure 3: Inter-text similarity and minimum of intra-text similarity distribution on MSRVTT sample features. The features of the the left and right graph come from CLIP [99] and SentenccBertFornr p Intra-text similarity defined the similarity of different text descriptions for video. Inter-text similarity defined as similarity of different text descriptions contents can be represented by the minimum of intra-text similarity. It can be seen that at least one-third of the minimum of intra-text similarity is less than the mean of inter-text similarity on both graphs It demonstrates that many videos have diverse captions and contents and it necessary to multiple visual prototypes to align with different text descriptions.
respectively.
different videos.
result, diversity of video
videos, the text descriptions very similar. The videos may be too short and only have character, event or action. The prototypes and masks are similar. Instead, we'd like all masks all videos to pay attention to different token positions. Inspired by the variance loss feature of [21], design the variance loss on mask as
Lear Σ max(0, D(me, c));
(4)
1=1
where D(:-)) is the regularized standard deviation defined by
exx,e) /Var(z) +
(5)
where y is constant target value for standard deviation; is small scalar to prevent numerical instabilities. The variance loss forces the variance of all prototype mask values for each token to be large. It also encourages diverse prototypes for representing the rich information in videos.
3.4 The Dual-Encoder Framework
We use end-to-end training dual-encoder framework. Frozen B. basic and successful dual-encoder architecture (separate visual encoder and text encoder), serves as our foundation in this study. It's designed for video-language pretraining to match raw-pixel video and raw text with contrastive loss dual-encoder framework. The language and vision encoder is multi-layer bidirectional transformer and Timesformer [20] with some changes.
To train this encldrr framework, the matched text-video pairings in the batch are regarded as positive, while all other pairwise combinations in the batch are deemed negative. Each video appears most once each batch to prevent duplicates. With text-video pairs a batch, the symmetrical contrastive loss is calculated as follows
exp(sn/T) log
exp(8i/T)
Lr, = Σ log Σ exp(81//7)
(6)
Σ exp($ja/7)
where 7 is the temperature and 8;s, Siy is the similarity of ith paired text-video, ith text to jth video. Then the ranking of videos/texts is determined by this similarity caleulation for single text/video query. So overall loss is defined by
L - + a Lvar
where a controls the weight of Lear.
5

---

Table 1: Retrieval performance the MSR-VTT dataset
Text-lo- Video
Video-to-Text
Method
SumR
R@1 1
R@5
@@ 10
MedR
R@1
R@s
R@10
MedR
ActBERT
8.6
23.4
33.1
36.0
HowTo100M
14
14.9
40.2
52.8
9.0
16.8
41.7
55.1
8.0
221.5
NoiseEstimation
24
17.4
41.6
53.6
8.0
UviVL
25
21.2
49.6
63.1
6.0
AVLnet
27.1
55.6
28.5
54.6
4.0
297.5
HGR
27
21.7
47.4
61.1
6.0
20.4
47.9
60.6
6.0
259.1
TT-CE+ 22
29.6
61.6
74.2
3,0
Dual Encoding 29
21.1
48.7
60.2
6.0
21.7
49.4
61.6
6.0
262.7
TACO
9
24.5
52.8
65.5
5.0
SUPPORT_SEH
(30
30.1
58.5
69.3
3.0
28.5
58.6
71.6
3.0
316.6
HiT 6
30.7
60.9
73.2
2.6
32.1
62.7
74.1
3.0
333.7
CE 8
20.9
48.8
62.4
6.0
20.6
50.3
64.0
5.3
267.0
MMT
26.6
57.1
69.6
4.0
27.0
57.5
69.7
3.7
307.5
DMM
7
31.2
62.8
76.4
3.0
29.4
63.4
75.0
3.0
338.2
ClipBERT
22.0
46.8
59.9
6,0
Frozen
31.0
59.5
70.5
3,0
Ours
36.2
64.2
75.7
34.8
63.8
73.7
3.0
348.4
Table 2: Retrieval performance on the LSMDC dataset
Text-to- Video
Video-to Text
Method
SumR
R@
R@5
R@ 10
MedR
R@
R@5
RØ 10
MedR
JSFusion
31
9.1
21.2
34.1
27.0
HiT 6
14.0
31.2
41.6
18.5
TT-CE+
28
17.2
36.5
46.3
MEE 32
9.3
25.1
33.4
25.3
MEE-COCO 32
10.1
25.6
34.6
21.0
CE 8
11.2
26.9
34.8
25.3
MMT [5]
13.2
29.2
12.1
29.3
22.5
160.5
DMM
7
15.8
34.1
43.6
14.3
14.3
33.7
43.6
15.5
185.1
Frozen 3
15.0
30.8
39.8
20.0
Ours
17.8
37.1
45.9
13.5
16.5
34.3
44.6
14.0
196.2
Experiment
We evaluate our model's performance on several public datasets. In the sections that follow, we first introduce the datasets and metrics for comparing results, as well as implementation details. Then we demonstrating the overall performance of model on the datasets. Finally, present ablation studies to evaluate the effectiveness of each component of our model.
4.1 Datasets and Metrics
MSR-VTT
MSR-VT VTT [10] contains 10K YouTube videos, each of which is accompanied by natural sentences that describe the video content. Following [3], we train on 9K train+val videos and report results on 1K- test set.
MSVD
MSVD 12] consists of 80K English descriptions for 1,970 videos from YouTube, with each video containing 40 sentences each. Following [3], we use the standard split 1200, 100, and 670 videos for training, validation, and testing.
DIDEMO
DIDEMO [T] contains 10K Flickr videos annotated with 40K sentences. Following B.. we evaluate paragraph- to-video retrieval, where all sentence descriptions for video concatenated into single query.
LSMDC

---

Table 4: Retrieval performance on MSVD
Table 3: Retrieval performance on DIDEMO
Text- to Video
Text-to- Video
Method
Method
R@1
R@5
R@ 10
MedR
R@T
R@S
R@I
MedR
VSE
36
12.3
30.1
42.3
14.0
S2VT
34
11.9
33.6
13.0
VSE++
37
15.4
39.6
53.0
9.0
FSE 35
13.9
36.0
11.0
TT-CE:
28
25.4
56.9
71.3
4.0
TT-CE+
28
21.6
48.6
62.9
6.0
SUPP-SEB SET
[30]
28.4
60.0
72.9
4.0
CT: [8]
16.1
41.1
8.3
MC
[38
20.3
47.8
61.1
6.0
ClipBERT [2
20.4
48.0
60.8
6.0
CE
[8
19.8
49.0
63.8
6.0
Frozen 3
31.0
59.8
72.4
3.0
Frozen
3
33.7
64.7
76.3
3.0
Ours
36.5
64.9
75.4
Ours
36.7
67.4
81.3
2,5
LSMDC [13 consists of 1 118,081 video clips sourced from 202 movies. The validation set contains
7,408 clips and evaluation is done on a test set of 1, 1,000 videos from movies disjoint from the train and val sets. This follows the protocol outlined in 33
Metries
We evaluate retrieval performance using common information retrieval metrics such as Recall at (R@ K and K=1, 5, 10) and Median Rank (MedR). R@ K is the proportion of test queries for which at least one relevant item is found among the top-K, K retrieved results. The MedR calculates the median rank of correct items in the returned ranking list, with lower score indicating a better model. To reflect overall retrieval performance, we also take the sum of all R @ K as SumR.
4.2 Implementation Details
We fintune the pretrained model from Frozen 3 on text-video retrieval dataset. The visual encoder have 12 attention blocks, patch size P 16, sequence dimension
768, and 12 heads. The text encoder takes the architecture of DistilBERT base-uncased [39]. The dimensionality of the common text-video space is set to 256. We take some training settings same with Frozen [3]: we use 8 frames per video when training model; The temperature hyperparameter for the loss defined in Equation (6) is set to 0.05; We randomly crop and horizontally flip during training and center crop the maximal square crop at test time for visual augmentation; All videos resized to x 224 as input; settings, randomly sampling and
We use text augmentation during training for paragraph-retriee concatenating variable number of corresponding captions per video;The optimizer is Adam. We use a different learning rate scheduler: warmup for epochs with a linearly growing learning rate from 0.0 to 0.00003, then cosine decay scheduler to guide the learning rate from 0.00003 0.0 for other 45 epochs. The batch size set as 64. The number extra prototypes besides class token K is set as 3. The variance loss parameter 7, are set 0.75, 1e-4. Variance loss weight is set as 5.0.
4.3 Comparison to the State of the Art
Tables 4
present the retrieval results of HiT on MSR- VTT, LSMDC, DIDEMO and MSVD.
We also compare TMVM with other state-of-the-art methods: improving the robustness of pretrained
model 23 14 24 25 26]; designing better architecture or loss 27 28
29
9 30
34
35
on pre-extracted features; focusing on mining information of different visual modalities end-to-end trainable model 3].
5
As demonstrated by the results, TMVM outperforms all comparison methods clearly. We present video-to-text and text-to-video retrieval results for MSR-VT VTT and LSMDC. For MSR-VTT, our retrieval performance at SumR, in particular, is 348.4, outperforming recent state-of-the-aa methods 7 by a margin of 10.2. For LSMDC, our retrieval performance SumR also outperforms other methods by a clear margin. It accurately reflects TMVM's overall retrieval quality. We report retrieval performance terms of text-to-video retrieval for DIDEMO and MSVD. TVMM continues to outperform comparison methods. Our method doesn't outperform R 10 of some other methods in Tables T 2 possibly because the methods use multi modality information of video, such as speech, audio and appearance. Our method is based Frozen B and only uses the frames.
7

---

Table 5: Ablation study MSR-VTT investigate the contributions of multiple visual prototypes matching
variance loss.
Multiple Prototypes
Text-to- Video
Method
Variance Loss
K
R@1
R@5
R@ 10
SumR
Baseline
0
X
34.1
61.0
72.8
167.9
TVMM-PP
3
X
34.7
61.7
72.8
169.2
TVMM-Decoder
3
X
35.1
61.8
74.0
170.9
TVMM-Mask
3
35.9
63.3
74.5
173.2
TVMM-Mask
3
36.2
64.2
75.7
176.1
TVMM-Mask
1
35.4
63.1
73.5
172.0
TVMM-Mask
2
36.4
64.1
75.1
175.6
TVMM-Mask
4
34.8
63.5
73.4
173.1
TVMM-Mask
10
33.7
62.7
72.5
168.9
TVMM-Mas
1569
#tokens
32.3
59.8
71.7
163.8
Table 6: Ablation study on MSR-VTT to investigate the effect of the number of input video frames.
Pext-to-Vider
Number Video Frames
Method
R@1
R@5
R@ 10
SumR
Baseline
24.6
48.5
60.2
133.3
Ours
25.3
50.8
61.2
137.3
4
Baseline
33.6
59.1
71.0
163.7
Ours
34.6
61.9
73.9
170.4
Baseline
34.1
61.0
72.8
167.9
8
Ours
36.4
64.2
75.7
176.1
4.4 Ablation Study
We discover that the proposed components, such as Multiple Visual Prototypes and Variance Loss, help to improve retrieval performance. We exhaustively and comprehensively ablate two components to demonstrate their effectiveness and robustness.
Production of Multiple Visual Prototypes. As mentioned above, we use mask-based method to aggregate token features from the final layer. In this section, we design two other variants study whether this production is good enough.
TVMM-Part. We divide the token features into several parts according to the frame order. The part's feature centers are then treated as prototypes.
WVMM-Decodee We use several embeddings as queries, and the token features as memory, then we throw query and memory into two-layer decoder to process the queries visual prototypes.
Baseline is Frozen 31 with our training parameters. TVMM-Mask is our method which is introduced in Section 2. As shown in Table TVMM-Mask outperforms TVMM-Part and TVMM-Decoder with a clear margin. TVMM-Part improved little compared with the baseline, only 1.3 on SumR. shows that TVMM-Part is very limited in its ability model multiple prototypes.
The Number of Visual Prototypes. As shown in Table 5 multiple prototypes is better than single prototype. With K and its variance loss, TVMM-Mas. outperforms the bascline with 5.3 on RSum. This reflects video can contain rich instance and event information and the positive text only describes part of it. TVMM finds correspondence in the video with the help of multiple visual prototypes matching. The choice of number of prototypes is critical. K=3 is the best among 1,2,3,4 and 1569. When K is greater than equal appropriate number prototypes can assist the model in learning representative prototypes. When there many prototypes, the learned prototypes become noise rather than representative.
K gets larger, the performance continues degrade.
The Number of Input Visual Frames. In end-to-end training framework, the number of video frames is parameter that has big impact performance. As shown in Table 6 as the number

---

Text Query
Retrieval Result of Baseline
Retrieval Result of Ours
car driving down road goes from black and white vidco to colored video
man talking about old car
Figure 4: Some examples of text-video retrieval to compare the baseline with our method. The first video in the middle shows a guy talking about the transition from old one. The second video in the middle shows a man describing various parts car to another person.
Original Video
Prototype I: "a woman looking sad"
Prototype 2: "a and talking"
Prototype 3:
"a couple are having issues in music video"
Figure 5: Visualization multi prototypes. select a video and three corresponding captions MSR-VTT. The captions are matched with the prototype that was the most similar. Then, to visualize the prototypes, we plot the heatmaps based on the mask values the four sampled frames. The four frames depict serious sad woman, serious woman, and pair talking each other.
of video frames increases, so does the performance. It's worth noting that even on single- frame input, our method also improves the baseline with 4.0 on SumR. proves the importance of multiple prototype matching.
Some Retrieval Examples. Each test video in the original MSR-VTT split has only one text description. In this experiment, we take some samples from the training data set and add them to the test set demonstrate the superiority of the method. When compared to the bascline, our method provides correct retrieval results for the two test text queries, as shown in Figure 4 is because our TVMM model can alleviate the ambiguity problem video-text correspondence.
Visualization Multi Prototypes. We choose a video and its three captions from MSR-VTT. The captions matched with the most similar prototype. The heatmaps based the mask values on the four sampled frames are then plotted to visualize the prototypes. The four frames show serious man, sad woman, a serious woman, and pair conversing. As you see in Figure the three captions correspond well with the prototypes, which attend to different contents of the video.
Conclusion
In this paper, we have proposed a Text-Adaptive Multiple Visual Prototype Matching Model. generates multiple prototypes for video automatically to account for its rich information, and we present a text-adaptive matching strategy find correspondence between texts and videos. Then, we propose variance loss to encourage different prototypes to pay attention to different video contents.

---

Experimental results on four public video retrieval datasets demonstrate the advantages of our model. Ablation studies are also carried out to verify the effectiveness of each part of our model. Our study has one drawback in that the number of prototypes is specified manually rather than automatically. Typically, the number of prototypes in each video varies. That is our future work.
6 Acknowledgement
This work was supported by the National Science Foundation for Young Scientists of China (62106288). And it was in part supported by Foshan HKUST Projects (FSUST2I-FYTRIO1A A, FSUST21-FYTRRIZA), C. Shen's participation was in part supported by major grant from Zhejiang Provincial Government.
References
[1] Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping tanguage-image pre-training for unified vision-language understanding and generation. arXiv preprint, abs/2201.12085 2022.
[2] Jie Lei. Linjie Li, Luowei Zhou, Zhe Gan, Tamara L. Berg, Mohit Bansal, and Jingjing Liu. Less is ClipBERT for hideo-and-langure learning via sparse sampling. Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[3] Max Bain, Arsha Nagrani, GUI Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proc. IEEE/CVF International Conference on Computer Vision, 2021.
[4] Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio De Rezende, Yannis Kalantidis, and Diane Larlus. Probabilistic embeddings for cross-moda. retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8415 8424, 2021.
[5] Valentin Gabeur. Chen Sun. Karteek Alahari, and Cordelia Schmid. Multi-modal transformer for video retrieval. In Andrea Vedaldi. Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Proc. Eur. Conference on Computer Vision, pages 214-229, 2020.
[6] Song Liu, Haoqi Fan, Shengsheng Qian, Yiru Chen, Wenkui Ding, and Zhongyuan Wang. HiT: Hierarchical transformer with momentum contrast for video-text retrieval. In Proc. IEEE/CVF International Conference on Computer Vision, 2021.
[7] Wenzhe Wang. Mengdan Zhang, Runnan Chen, Guanyu Cai, Penghao Zhou, Pai Peng. Xiaowei Guo, Jian Wu, and Xing Sun. Dig into multi-modal cues for video retrieval with hierarchical alignment. In Proc. International Joint Conference on Artificial Intelligence, 2021.
[8] Yang Liu, Samuel Albanie, Arsha Nagrani, and Andrew Zisserman. Use what you have: Video retrieval using representations from collaborative experts. In Proc. British Machine Vision Conference, 2019.
[9] Jianwei Yang, Yonatan Bisk, and Jianfeng Gao. TACo: Token-aware cascade contrastive learning for video- text alignment. Proc. IEEF/CVF International Conference on Computer Vision, pages 11542- 1552, 2021.
[10] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. MSR-VTT: A large video description dataset for bridging video and language. In Proc. JEEE Conference Computer Vision and Puttern Recognition, 2016.
[11] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan C. Russell. Localizing moments in vidco with natural language. Proc. IEEE International Conference on Computer Vision, 2017.
[12] David L. Chen and William B. Dolan. Collecting highly parallel data for paraphrase evaluation. In Dekang Lin, Yuji Matsumoto, and Rada Mihalcea, editors, Proc. Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 190-200, 2011.
[13] Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schiele. A dataset for movie description. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 3202-3212. IEEE Computer Society, 2015.
[14] Antoine Micch, Dimitri Zhukov, Jean- Ban-BBttstt Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. HowTo100M: Learning a text-video embodding by watching hundred million narrated video clips. In Proc. IEEE/CVF International Conference Computer Vision, pages 2630 2640, 2019.
[15] Po- Yao Huang. Junwei Liang, Vaibhav Vaibhav, Xiaojun Chang, and Alexander Hauptmann. Informe- dia@ treevid 2018: Ad-hoc video search with discrete and continuous representations. In In TRECVID Proceedings (Vol. 70), 2018.
[16] Xiaojun Chang. Wenhe Liu, Po- Yao Huang, Changlin Li, Fengda Zhu, Mingfei Han, Mingjie Li, Mengyuan Ma. Siyi Hu, Guoliang Kang, et al. Mmvg-inf ent- @ trecvid 2019: Activities in extended video. In TRECVID, 2019.
10

---

[17] Michael Wray, Hazel Doughty, and Dima Damen. On semantic similarity in video retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3650-3660, 2021.
[18] Zeyu Wang. Yu Wu, Karthik Narasimhan, and Olga Russakovsky. Multi-query video retrieval. arXiv preprint arXiv:2201.00 033639, 2022.
[19] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proc. International Conference on Machine Learning. pages 8748 8763. PMLR, 2021.
[20] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? Marina Mcila and Tong Zhang, editors, Proc. International Conference on Machine Learning, 2021.
[21] Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Varianc-iivvarinne--vvaaaa regularization for self-supervised learning. arXiv preprint, abs/2105. 049066 2021.
[22] Nils Reimers and Iryna Gurevych. Bentence-BERTT: Sentence embeddings using siamese bert- networks.
arXiv preprint, arXiv:abs/1908 908. 10084, abs/1908. 10084, 2019.
[23] Linchao Zhu and Yi Yang. ActBERT: Learning global-local video-text representations. In Proc. IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 9743-8752 2020.
[24] Elad Amrani, Rami Ben-Ari, Daniel Rotman, and Alex M. Bronstein. Noise estimation using density estimation for self-super multimodal learning. In Proc, AAAI Conference on Artificial Intelligence,
pages 6644-6652 6614, 2021.
[25] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-langune tasks. In Proc. Advances in Neural Information Processing Systems, pages 13-23, 2019.
[26] Andrew Rouditchenko, Angie W. Boggust, David Harwath, Brian Chen, Dhiraj Joshi, Samuel Thomas, Kartik Audhkhasi, Hilde Kuehne, Rameswar Panda, Rogério Schmidt Feris, Brian Kingsbury, Michael Picheny, Antonio Torralba, and James R. Glass. AVLnet: Learning audio-visual language representations from instructional videos. In Proc. Annual Conference of the International Speech Communication Association, pages 1584-1588, 2021.
[27] Shizhe Chen, Yida Zhao, Qin Jin, and Qi Wu. Fine-grained video-text retrieval with hierarchical graph reasoning. In Proc. IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10635- 10644, 2020.
[28] Ioana Croitoru, Simion-Vlad Bogolin, Marius Leordeanu, Hailin Jin, Andrew Zisserman, Samuel Albanie. and Yang Liu. Teachtext: Crossmodal generalized distillation for text-video retrieval. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 11563-11573. IEEE, 2021.
[29] Jianfeng Dong, Xirong Li, Chaoxi Xu, Xun Yang, Gang Yang. Xun Wang, and Meng Wang. Dual encoding for vidco retricval by text. IEFE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[30] Mandela Patrick, Po- Yao Huang. Yuki Markus Asano, Florian Metze, Alexander G. Hauptmann, João F. Henriques, and Andrea Vedaldi. Support- set bottlenecks for video-text representation learning. In Proc. International Conference on Learning Representations, 2021.
[31] Youngjac Yu, Jongscok Kim, and Gunhee Kim. A joint sequence fusion model for video question answering and retrieval. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Proc. Eur. Conference on Computer Vision, 2018
[32] Antoine Miech, Ivan Laptev, and Josef Sivic. Learning a text-video embedding from incomplete and heterogeneous data. arXiv preprint, abs/1800 1018000146266 2018.
[33] Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Joseph Pal, Hugo Larochelle, Aaron C. Courville, and Bernt Schiele. Movie description. Int. J. Comput. Vis., 123(1):94-120, 2017.
[34] Subhashini Venugopalan, Huijuan Xu, Jeff Donahue, Marcus Rohrbach, Raymond J. Mooney, and Kate Saenko. Translating videos natural language using deep recurrent neural networks. In Rada Mihalcea, Joyce Yue Chai, and Anoop Sarkar, editors, NAACL Conf. North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1494-1504, 2015.
[35] Bowen Zhang, Hexiang Hu, and Fei Sha. Cross-modal and hierarchical modeling of video and text. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XIII. volume 1 11217 of Lecture Notes in Computer Science, pages 385-401. Springer, 2018
[36] Ryan Kiros, Ruslan Salakhutdinov, and Richard S. Zemel. Unifying visual-semante embeddings with multimodal neural language models. arXiv preprint, arXiv:abs/1411.555 abs/1411 2014.
[37] Fartash Faghri, David J. Fleet, Jamie Ryan Kiros, and Sanja Fidler. VSE++: improving semanicc embeddings with hard negatives. In British Machine Vision Conference 2018, BMVC 2018, Newcastle, UK September 3-6, 2018 page 12. BMVA Press, 2018

---

[38] Niluthpol Chowdhury Mithun, Juncheng Li. Florian Metze, and Amit K. Roy-Chowdhury. Learning joint embedding with multimodal cues for moos- video-text retrieval. In Kiyoharu Aizawa, Michael S. Lew, and Shin' ichi Satoh, editors, Proceedings of the 2018 ACM International Conference on Multimedia Retrieval, ICMR 2018, Yokohama, Japan, June 11-14, 2018, pages 19- 27. ACM, 2018.
[39] Victor Sanh, Lysandre Dehut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of BERT:
smaller, faster, cheaper and lighter. arXiv preprint, abs/1910.01 108, 2019.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(C) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [No]
(b) Did you include complete xproofs of all theoretical results? [No]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're cou're using/curating? [No] The method of acquisition is public.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] The data is publicly available.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [No] We didn't 't use crowdsoureing and conducted research with human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals. applicable? [No] We didn't use crowdsourcing and conducted research with human subjects.
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [No] We didn't use crowdsourcing and conducted research with human subjects.
12


Reviews:
[{"review_id": "sOvoQIDXZ9K", "reviewer": "Reviewer_nbMi", "summary": "The paper studies the video and text retrieval task. As the clip/video contains rich information in a sequence of frames, while the text contains limited information about one clip/video. There is a semantic gap between the video and text matching. To bridge the semantic gap, the author proposed to learn prototypes from the video and match the text with the video via those learnt prototypes. The prototypes should capture different semantic level information about one video. To learn these prototypes, the author proposed text-adaptive multiple visual prototype matching model. Specifically, it maps a video into a set of visual representation. It then learns an aggregation function to aggregate those visual representation to form the prototypes. Then the prototypes are used to match the text for video and text retrieval. The proposed approach achieves better performance on several video and text retrieval datasets.", "questions": "Please address the Q1, Q2, Q3, and Q4. My main concern is that the prototype might just be an interesting story for the proposed approach. It might not be the main reason for the improved performance.", "limitations": "There is no negative societal impact.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\nThe main argument that there exists a semantic gap between the text and video is important and interesting. The proposed model that using the prototypes to represent video and matching the prototypes with text is also interesting and promising. Although how to define the prototype is an open question, this approach provides one of the definition of the prototype and also shows one implementation of this idea.\n\nCons:\n1. Prototypes: I wonder whether the prototypes really capture the semantic meaning of the video. Concretely, what is the performance of setting the # of prototypes as the same as the # of tokens (B as in L128)? I wonder the improved performance is due to the similarity function (Eq. 3) or due to the proposed prototypes?\n\n2. I didn't fully understand the variance loss. Specifically, L145-L146 'we'd like all masks in a batch of videos ...' is pretty confusing.\n\n3. What is the meaning of the Figure 3? Specifically, what is the definition of the 'minimum of intra-text similarity'? And how to interpret the Figure 3? Although L112-L121 tries to explain this figure, I still didn't understand the aim and the meaning of this analysis.\n\n4. As the aims of prototypes is to capture different semantic meanings of video, I wonder does the proposed prototypes really achieve this goal? Specifically, is there any qualitative or quantitative results/experiments to verify this argument?\n\n5. My main concern is the improved performance might not due to the proposed prototypes. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "rbtWjKJpmVR", "reviewer": "Reviewer_UJze", "summary": "When performing video retrieval from a text query (or vice versa), representing each video with a single embedding can be problematic because a video can depict many things, and potentially match multiple different search queries. One solution is to represent the video by multiple embedding \"prototypes\", and define video-text similarity to be the similarity between a text query and the nearest of the video's prototypes.\n\nThe authors propose to create K different prototypes for each video by computing ViT tokens for the ~8 input frames, and applying K-different learned functions, each of which masks out a subset of the tokens and sums the remaining ones. These functions are learned as part of dual text-video encoder setup using symmetrical contrastive loss.\n\nOf course if the learning model produces several nearly-identical prototypes for a video, then there's no gain. To prevent this, a training loss term is introduced to maximize the variance of the aforementioned mask functions, encouraging diversity between prototypes.\n\nStrong recall numbers are demonstrated on a variety of video-text retrieval datasets. The components of the model are validated with ablation studies.\n", "questions": "Line 38: Why exactly is it infeasible to align a single video feature with multiple text features? It might be worth elaborating here.\n\nLine 158: \"all other pairwise combinations in the batch are deemed negative\". Of course, this means that even if a video has 20 potential ground truth queries (e.g. MSR-VTT) it'll only appear in a batch once, correct? It might be worth clarifying this in the paper.\n\nAre the mask values interpretable? For example, for a given video, could we see that mask#1 focuses on ViT tokens representing the woman in the video, mask#2 focuses on the man, and mask#3 on the crowd? This could make a compelling visualization.\n", "limitations": "The authors did not describe any shortcomings. I would appreciate some mention of where the model fails or could be improved. For example, the premise that a query need only match a single prototype might lead to low-precision results (think about matching based on an OR vs. AND operator). Is this observed? Are there any other areas for future work?\n\nI don't see any significant potential negative societal impact.\n\n", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality:** Although the idea of representing a complex item with multiple prototypes for the purposes of retrieval is not novel, this particular instantiation -- learning K different views that aggregate ViT tokens across all the input frames -- is novel and very interesting, particularly because it's learned rather than ad-hoc, and can potentially attend to one element across many frames of the video.\n\nI would like to have seen a little more investigation as to what the model is learning, e.g. a visualization of what each mask is attending to, or a demonstration that all of the prototypes for a video are useful -- matching different ground truth queries.\n\n**Quality:** The method and evaluation seem sound and convincing. It might have been nice to see a comparison to a baseline with one prototype per frame, as proposed on lines 122-123.\n\n**Clarity:**\n\nLine 130: What does \"we find that the mask-based production of kth is enough\" mean?\n\nWhat baseline method is used in Tables 5 & 6? I couldn't find it explained anywhere.\n\nEquation (2). What is the dimensionality of the output of f^mask? Is it 1-dimensional or D-dimensional? \n\nThe paper would benefit from additional editing for grammar.\nSome references are incorrect, e.g. \"Figure 3.3\" on line 118.\nThere are duplicate sentences on lines 105 and 107.\n\n**Significance:** I think the idea is technically novel, and the performance on standard benchmarks is strong enough that this paper would be of interest to the video search/retrieval community, and possibly to the video-text representation learning community as well.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Mf9Lyq8hW3", "reviewer": "Reviewer_Xjvw", "summary": "This paper tackles the task of video-text retrieval. Motivated by the fact that multiple different text descriptions can apply to the same video the paper proposes a new model where the video is represented by multiple prototypes in the video-text embedding space. This allows multiple different texts to have a high similarity score with the video. To encourage these different texts to be diverse the paper also proposes a variance loss. The proposed method is tested on 4 video-text retrieval datasets: MSR-VTT, MSVD, DIDEMO and LSMDC.\n", "questions": "- What are the advantages of the method proposed in this paper over the methods proposed in [A] and [C]?\n- Does this paper still achieve state-of-the-art results when compared to the results published in [C]?\n- Why were Didemo and LSMDC chosen as datasets to evaluate on over VATEX or ActivityNet Captions?\n- Why does the proposed method improve on R@1 and R@5 but not on R@10 in Tables 1 and 2?\n- Does not having exhaustive annotations of video-text relevancy cause any issues with the evaluation?", "limitations": "Limitations or ethical concerns are not mentioned which is a shame considering there was a half page of space left which could have been used for this.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n***\n- The observation that multiple different text descriptions can apply to the same video is interesting.\n***\n- The analysis on inter-text and intra-text similarity of captions for different videos is valuable.\n***\n- The proposed method is simple and the ablations should a clear improvement over the baseline approach.\n***\n\n## Weaknesses\n***\n- the main issue I have with this paper is that it ignores prior works which consider multiple relevant captions for images/videos \n- [A] considers one-to-many correspondences between images and text and proposes a probabilistic embedding method so that multiple texts can be matched with a single image. The related work of this paper also highlights previous methods in images considering many-to-many correspondences which are also relevant.\n- In video [B] highlighted issues with the one-to-one assumption in evaluation of video retrieval methods and proposes a new evaluation metric to address this. It also performs some additional experiments where videos are trained with other relevant captions from the training set in addition to their annotated captions.\n- [C] considers multi-query video retrieval, where multiple queries are used in both training and testing. [C] also has an experiment where they show multi-query training can improve on single-query testing (Figure 6) and the results seem to out-perform the ones in this paper\n- Without these works being acknowledged in related work and the differences betweeen these and the newly proposed method being explained I don't think this paper can be accepted.\n- Where appropriate prior works which consider multiple relevancies should also be compared to. This paper claims state of the art but it appears [C] may actually outperform it. I would like clarification from the authors in the rebuttal on this point. \n- I also think [A] should be compared to. It was created for image-text retrieval but since the motivation and problem it tackles are so similar to this paper it seems like a valuable comparison, particularly as the proposed method in this paper doesn't seem to be specific to video other than the backbone network the features come from. \n***\n- the choice of datasets is strange\n- MSR-VTT and MSVD make sense as each video is annotated with multiple captions\n- However, Didemo and LSMDC don't seem to fit with the problem tackled in the paper. For Didemo the paper states 'all sentence descriptions for a video are concatenated into a single query' and for LSMDC there is no mention of there being multiple captions per video.\n- VATEX and ActivityNet Captions would be suitable as they both have multiple captions per video\n***\n- Little insight is iven about the results\n- for instance, the proposed method consistently improves on Recall@1 and Recall@5 but doesn't on Recall@10 (Tables 1 and 2). This is not explained.\n***\n- There is a potential issue with the annotations for the datasets used as the datasets are not exhaustively annotated with possible captions\n- This means it is unknown whether captions from another video may correctly describe part of the video in question. Therefore, some correct retrievals may be penalized by the evaluation metric.\n- This not reason to reject this paper as this is potentially also a problem with prior works in video-text retrieval. I'm just interested in whether the authors think this is an issue and how much it would effect the results.\n***\n[A] Probabilistic Embeddings for Cross-Modal Retrieval. Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio de Rezende, Yannis Kalantidis, Diane Larlus. CVPR 2021.\n[B] On Semantic Similarity in Video Retrieval. Michael Wray, Hazel Doughty, Dima Damen. CVPR 2021.\n[C] Multi-query Video Retrieval. Zeyu Wang, Yu Wu, Karthik Narasimhan, Olga Russakovsky. ArXiv, Jan 2022.\n\n## Final Justification\nMy main concern prior to the rebuttal was that the paper ignore prior works which shared the motivation of multiple different text descriptions being applicable to a video or image. The rebuttal has addressed this concern by providing comparison to [A], an image-focused method which uses a probabilistic embedding, and [C] a video retrieval method which uses multiple relevant captions in training. The revised version of the paper also doe include a little more insight into the results, although this could have been better. I do appreciate the new Figure 5 visualizing the mask values and prototypes. With the new results comparing the paper to similar prior works I recommend accepting this paper, however I think it is important that in the final version of the paper these prior works are also included in the related work and the difference with this work is clearly highlighted.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8c6mf17sPo6", "reviewer": "Reviewer_DonZ", "summary": "The paper tackles the problem of text-video retrieval. The authors argue that current models concentrate on mapping one video to one sentence using only one feature that needs to be matched with multiple texts is hard to achieve and instead propose a text adaptive multiple visual prototype matching where the model captures multiple prototypes that describe a video. In this way, the query is matched with the most similar prototype instead of having only one video level representation. Finally, the authors test their approach on four retrieval benchmarks.", "questions": "1. Is the model initialized with Frozen?\n\n2. Have you tried the idea in conjunction with other methods? \n\n3. What do the groups correspond to in the Tables?\n\n4. What happens if K is larger? Any intuition on why there seems to be a decrease in performance with a higher K.\n\n5. Does the Baseline in Table 5 correspond to Frozen?", "limitations": "The limitations are somehow discussed through the ablations. The potential societal impact is not discussed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper tackles an important problem text-video retrieval and has a interesting idea at its basis. It reports results on four datasets and has good results.\n\nMy main concern is related to the novelty. While I like the idea and find it interesting, I fell like it is quite closely related to the idea of probabilistic embeddings that was introduced for text-image retrieval. While there are definitely differences, I feel like a proper comparison and discussion between the two needs to be added.\n\n* Chun, Sanghyuk, et al. \"Probabilistic embeddings for cross-modal retrieval.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\nFigure 4 - maybe make the picture larger (there still is space on the sides, so the frames can be made larger)\n\nMinor:\nlines 169-170 look weird - can they be moved to the next page?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept