An Empirical Study of Frame Selection for Text-to-Video Retrieval
Authors: Mengxia Wu
Min Cao
Yang Bai
Ziyin Zeng
Chen Chen
Liqiang Nie
Min Zhang
Venue: EMNLP 2023 Findings (2023)
Keywords: text-to-video retrival
frame selection

Abstract: Text-to-video retrieval (TVR) aims to find the most relevant video in a large video gallery given a query text. The intricate and abundant context of the video challenges the performance and efficiency of TVR. To handle the serialized video contexts, existing methods typically select a subset of frames within a video to represent the video content for TVR. How to select the most representative frames is a crucial issue, whereby the selected frames are required to not only retain the semantic information of the video but also promote retrieval
efficiency by excluding temporally redundant frames. In this paper, we make the first empirical study of frame selection for TVR. We
systemically classify existing frame selection methods into text-free and text-guided ones, under which we detailedly analyze six different frame selections in terms of effectiveness and efficiency. Among them, two frame selections are first developed in this paper. According to
the comprehensive analysis on multiple TVR benchmarks, we empirically conclude that the TVR with proper frame selections can significantly improve the retrieval efficiency without sacrificing the retrieval performance.

An Empirical Study of Frame Selection for Text-to- Video Retrieval
Mengxia Wu', Min Cao'; Yang Bai Ziyin Zeng', Chen Chen?, Liqiang Nie Min Zhang'
Soochow University, Institute of Automation, Chinese Academy of Sciences,
Harbin Institute of Technology, Shenzhen
mrxxwwwux)@sttu suda. eud..edu.a...mmeooomooo cn, {mcao Jesuda. edu. cn
Abstract
can benefit higher computational efficiency, while leading to the loss of visual details and thus impair- ing retrieval performance. Considering both effee- tiveness and efficiency, it is imperative to adopt an optimal frame selection method for TVR.
Text-to-video retrieval (TVR) aims to find the most relevant video in large video gallery given a query text. The intricate and abun- dant context of the video challenges the per- formance and efficiency of TVR. To handle the serialized video contexts, existing methods typ- ically select subset of frames within a video to represent the video content for TVR. How select the most representative frames is crucial issue, whereby the selected frames are required to not only retain the semantic infor- mation of the video but also promote retrieval efficiency by excluding temporally redundant frames. this paper, we make the first em- pirical study of frame selection for TVR. We systemically classify existing frame selection methods into text-free and text-guided ones, un- der which we detailedly analyze six different frame selections in terms of effectiveness and efficiency. Among them, two frame selections are first developed in this paper. According to the comprehensive analysis on multiple TVR benchmarks, empirically conclude that the TVR with proper frame selections can signifi- cantly improve the retrieval efficiency without sacrificing the retrieval performance.
Current frame selection methods can be divided into text-free frame selection and text-guided frame selection. The text-free frame selection only ex- plores the video contents themselves while ignor- ing their relevancy the texts in selecting frames. Specifically, the conventional techniques (Chang et al., 1999; Divakaran et al., 2002; Chang, 2003) for frame selection primarily rely on the color fea- tures of frames. However, these approaches fall short of comprehending the semantic content of videos and are not readily applicable to end-to-end training in various video-related tasks. For simplic- ity, existing works (Lei et al., 2021; Luo et al., 2022; Li et al., 2022a) typically employ uniform frame selection (Uni) or random frame selection (Rand) to eliminate the temporally redundant frames that contribute little to the retrieval. In contrast, the text-guided frame selection is aimed at utilizing textual information to filter the inessential frames. We partition the text-guided frame selection into two categories: non-interactive frame selection (N- In'T) (Gorti et al., 2022; Han al., 2022) and in- teractive frame selection (InT) (Buch al., 2022: Yang et al., 2023; Wang et al., 2022; Lin et al., 2022). The former directly calculates the similari- ties between the representations of frames and the text from two unimodal encoders without interac- tion, and then selects the frames with high similari- ties the representatives of the video. The latter constructs cross modal interactive module with extra parameters assess the relevancy between frames and the text.
1 Introduction
Text-to-video retrieval (TVR) aims to search for the most relevant video given query text, in- volving intra-modal understanding and inter-modal alignment. Videos composed of a sequence of frames, each of which can be considered as the pri- mary processing unit for video understanding and alignment with the text. However, due to the con- straints of computing resources, process all frames of one video for TVR. Therefore, existing methods (Liu et al., 2019: Gabeur et al., 2020; Lei et al., 2021; Luo et al., 2022) select a sub- set of frames from the original videos to perform TVR. Generally speaking. selecting fewer frames
impractical to
Although persistent efforts have been made in frame selection, there still exist three main prob- lems. 1) The critical clues to the retrieval a vidco typically exhibit a non-uniform distribution. The
"Corresponding author

---

text-free frame selection with the Uni/Rand does not follow this distribution, which may filter out the key frames that carry the critical clues and eventu- ally hamper the performance of TVR. And the text- guided frame selection circumvents this problem with the aid of text information, but at the expense of simpleness. Nevertheless, both of them have not considered the distribution of the video content itself. 2) The videos usually contain low-quality frames that are out-of-interest and less meaningful to TVR, leading to increased noise and decreased efficiency. These frames could caused by transi- tions, out-of-focus shots, inappropriate angles, etc. Yet, such frames are overlooked by the current text- free frame selection methods. 3) Currently, there is a noticeable absence of a comprehensive compar- ison and analysis of frame selection methods that take into account the aforementioned two factors. As a result, the optimal frame selection method for TVR still remains elusive, with no definitive answer yet.
frame selection methods. Through com- prehensive analysis, we aim to explore the optimal frame selection policy that balances effectiveness and efficiency.
We introduce two text-free frame selection methods, i.e., the redundancy-awar and low- quality-aware frame selection, thus providing broader views for the frame selection.
We conduct a comprehensive evaluation of the frame selection methods on the three common benchmarks, i.e., MSR-VTT (Xu et al., 2016), DiDemo (Anne Hendricks et al., 2017) and ActivityNet Captions (Caba Heilbron et al., 2015). A significant trade-off between perfor- mance and efficiency on the benchmarks be achieved by the explored optimal frame selection policy.
Related Work
2.1 Vixt-to-Video Retrieval
To this end, we extend the current frame selec- tion methods and conduct thorough empirical study of the frame selection methods to explore the optimal one in this paper. Specifically, be- yond the existing frame selections, we first develop a redundancy-aware frame selection (Redun-A) method for the first problem and a low-quality- aware frame selection method (L-QA)) for the sec- ond problem. Redun-A selects the most represen- tative and diverse frames by clustering the video frames, and LQ-A chooses the frames of high qual- ity by designing scorer network and retaining frames with high scores, both of which belong to the text-free frame selection. Then, based on the existing frame selections and the proposed ones, we investigate the effect of each frame selection method on TVR, as well as the potential combina- tions of them. Through extensive study, we con- clude that: (i) giving priority to the text-free frame selection, combining the proposed Redun-A and LQ-A is optimal policy; (ii) by further consid- ering the text-guided frame selection, the better trade-off between effectiveness and efficiency is achieved by the combination of Redun-A and N- InT.
The early TVR methods (Yu et al., 2018; Liu al., 2019; Gabeur et al., 2020) focused on designing a fusion module for cross-modal learning from the offline extracted video and text features. In re- cent years, visual-language pre-training technolo- gies have begun to push the development of TVR dominantly. CLIP4Clip (Luo et al., 2022) con- ducted empirical study video aggregation schemes for transferring the CLIP (Radford et al., 2021), which has been pre-trained on large-scale text-image pairs, TVR. This study demonstrated impressive performance in TVR with the aid of CLIP. As a result, a series of works have focused on excavating the power of CLIP by designing var- ious alignment strategies (Ma et al., 2022; Fang et al., 2023; Gorti et al., 2022; Chen et al., 2023) for performance improvement plugging token selec- tion strategies (Zhao et al., 2022: Liu al., 2022) for efficiency improvement. CenterCLIP (Zhao et al., 2022) performed token selection by cluster- ing numerous video patch tokens within each short video segment. However, this method limited the selection to the local content of the video and re- sulted in inefficient information selection through token clustering. Beyond adopting CLIP, two- stream network without the fusion module, TVR backbone, Lite VL (Chen et al., 2022) constructed baseed (Li al., 2022b) model, which in- corporates a pre-trained deep fusion module into TVR, and exhibits better performance but lower
Our main contributions can be summarized as follows:
We present an empirical study of frame se- lection for TVR, encompassing four text-free frame selection methods and two text-banc based

---

efficiency than the CLIP-based works. Differently, this paper makes an empirical study of frame selec- tion and explores an optimal frame selection policy for TVR, aiming at achieving the trade-off between retrieval performance and efficiency.
methods (i.c., non-interactive and interactive frame selections). Finally, we present the training and inference processes.
3.1 Model Architecture
Following BLIP (Li et al., 2022b), the model ar- chitecture comprises three main components: vision encoder, a text encoder, and multimodal encoder, as shown Figure 1 (a). Details for each component are as follows.
2.2 Frame Selection for TVR
Frame selection is an important process for TVR. It filter out the inessential frames in the video for a high-efffectiveness and high-performant
TVR. Uniform and random frame selections are two com- mon frame selection methods, by which some works (Luo et al., 2022; Lei et al., 2021) empir- ically explored the proper frame rate for TVR to balance efficiency and effectiveness. However, due to no reasonable guidance for frame selection, both methods prone to lose some key frames bene- ficial to TVR, resulting in limited trade-off be- tween efficiency and effectiveness. Actually, texts are the natural guidance for frame selection in TVR. Directly calculating the similarities between rep- resentations of the frame and text is one of the ways (Gorti et al., 2022; Han et al., 2022). Be- yond that, TW-BERT and FinCo (Yang et al., 2023; Wang al., 2022) proposed to simply concatenate the representations of frames with the text, and the concatenated representations were then input scorer network, which comprised a fully con- nected layer followed by a softmax layer, to assign a score to each frame. The resulting scores are adopted to guide the frame selection. To better fuse the motel- information during frame selection, some works (Buch et al., 2022; Lin et al., 2022) constructed self-attention- based encoder to fuse the frames and text representations. Different from the above text-guided frame selection methods, we employ pross-attention-basse encoder to fuse the frames and text representations, and then select the textual-relevant frames based on the attention weights in InT frame selection. Moreover, this pa- per provides comprehensive analysis of the frame selections to figure out the optimal one for TVR.
Vision Encoder. We employ the ViT-B/16 (Dosovitskiy et al., 2020) our vision encoder. Given video V, following the general practice in TVR methods (Lei et al., 2021; Luo al., 2022), we first uniformly pre-sample N frames V = (Fi, F2, Fx} from the video to repre- sent video contents. Then we conduct frame se- lection based on V to improve the retrieval effi- ciency. For the uniform and random frame se- lections, we directly operate on V and obtain K (K < N) frames V = Fea, Fex h. where 8k E (1,2....,N) N) denotes the index of the selected frame. Then, each frame Fsx is fed into ViT-B/16 and encoded as a sequence of rep- resentations faL = { fay. . f. . where P is the number of the frame patches and sel de- notes the representation of [CLS] token and can be viewed as the global representation of Fs.. For the other four frame selections (i.e., redundancy- low-quality-aware, non-interactive, and interactive frame selections), we first encode the frames y using ViT-B/16, based on which we select key frames. To sum up, the uniform and random frame selections are performed in image frame space, while the redundancy-awaid aware, non-interactive, and interactive frame selec- tions are carried out in feature representation space. These frame selection methods will be detailed in Section 3.2.
low-quality-
To this end, we can obtain the representations of the selected K frames. Based on this, we compute the global representation of the video fv. We first map the [CLS] representations fols of all K frames to a lower dimension (256-d) by a linear transfor- mation, denoted as felar For the uniform, random and redundancy-awart frame selections, we con- duct mean-pooling on the falst for generating the global representation of the video:
3 Methods
In this section, we first introduce the model ar- chitecture used for the empirical study of frame selection. Then, we elaborate on the core con- tent of our empirical study, including four text-free frame selection methods (i.e., uniform, random, Shundancyy-ware and lww-quality-wwar frame selections) and two text- guided frame selection
fy
celat
(1)

---

Unifeer Random frnnc selectied
Non intornetiiee frame selection
Text: young buly riling hiykl
on not milk mad.
Top X
[CLS] [[allic] (sodd]
(1D Redendanc--anee frame selection
(V1 Inturadire frams slection
Vision
Text Encuder
Crowo peweaatention
hasod enender
Closerine
Attn
Redus-A/ LQ-A
[soma) (Ifffie] [naa]
(VV) Low quality anarc fromse selection
Irares [LSS] fazes
Jorf (CLS] fetuee
Tap
frane's pash fecture
izets kakee
apprognted
VIM
pee-ompled tome
Model
(b)
Figure 1: An illustration of the model architecture and frame selection methods. (a) The model architecture consists three encoders, i.c. vision, text, and multimodal encoders. The uniform and random frame selections work in image frame space, while the other four in feature space. (b) We group all the frame selection methods into two types, i.c., text-free frame selection (I (IV and text-guided frame selection (V -VD). {si, sK) denotes the sequence of key frames. The now-quality-awee aware and interactive frame selection methods bring extra parameters. while the others are parameter-fres free.
While for the other four frame selection methods, we obtain fv by weighted average of polor.
the text-free and text-guided frame selections, shown in Figure 1 (b). In the following, we detail each frame selection method.
fv softmar(ass) ) Mf
(2)
3.2.1
Text-Free Frame Selection
where d'ss is the score of the frame Fsx computed by the corresponding frame selection method.
The text-free frame selection chooses frames only from the perspective of video contents themselves, including the uniform, random, redundancy-avee aware, and low-quality-wward frame selection methods. The former two frame selection methods are per- formed on the image frame level, while the latter two are on the feature representation level.
Text Encoder. We apply BERT-base (Devlin al., 2018) as the text encoder that transforms an input text T with M tokens into a sequence of representations {tel ,t",, ... ... ..), where the yala is the representation of the [CLS] token. We also map gals to the lower dimension (256-d) as the global representation of the text, denoted fr.
Uniform/Random Frame Selection. The uni- form and random frame selections are the most straightforward frame selection methods. Given the pre-sa frames V, the uniform frame selection evenly seleets K frames with the same intervals from V, while the random sampling selects with random intervals. The selected frames are then sent to the vision encoder for frame representation. Redundancy-aware Frame Selection. The uniform or random frame selection neglects the irregular distribution of crucial information within video contents, leading to the tendency to lose the crucial frames. For this, we develop redundancy-awar frame selection. which is dedicated reserving the most representative and diverse frames from each video. Specifically, given sequence of lower-dimensional [CL.S] representations of each frame in V, denoted edst we utilize as peds' = {ffel' .f ".."f
Multimodal Encoder. We adopt the BERT-base together with cross-attention layer inserted in each transformer block as the multimodal encoder. The BERT-base shares the parameters with the text encoder. The representations of the selected frames from the vision encoder act as the visual inputs of the multimodal encoder. For the textual in- puts, following BLIP (Li et al., 2022b), we append an [Encode] token to T, and its encoded represen- tation genc is denoted as the multimodal represen- tation of the video-text pair. In the cross-attention layer, we take textual inputs as the query and visual inputs as the key and value.
3.2 Frame Selection
According to the participation of text, we catego- rize the frame selection methods into two groups:

---

k-medoids++ clustering algorithm (Zhao et al., 2022) to partition them into K groups. The frames in the same group typically exhibit similar video contents, indicating redundancy within the video. The center frame each group is considered the most representative one. Naturally, we select the frame whose [CLS] representation corresponds to the center of each group.
where s(5-) denotes the cosine similarity. The frames with the highest K similarities to the given text are selected.
Interactive Frame Selection. The interactive frame selection constructs a specific multimodal- based selection module to model the relevancy be- (ween frames and the given text. In our implementa- tion, we design a cross-attention-biin frame selection, which consists of single trans- former layer. Specifically, in the antssnattention module, the text global representation fr pro- jected into a query Qt. and the frames' [CLS] repre- sentations frlle (i 1,... ..) are projected into key K and value Vy. We then adopt the scaled dot product attention to aggregate these representa- tions, denoted as STV
encoder for
Frame Selection. The pres- ence of low-quality frames videos, which are generally out of interest, has negative impact on both retrieval performance and efficiency. Nonethe- less, the low-quality frames are ignored by the ex- isting text-free frame selection methods. In this situation, we propose a low-quality-awarr frame selection filter them. In detail, we append scorer network after the vision encoder to assess the quality score a; of each frame:
Low-quality-
frr Attn(Qu. KJ) Attn(Q(, softmar
(5)
KT
02 Cenftmar(FCC(("
(6)
Vd
where (.(.) denotes the fully connected layer that map the folna (i = 1, 2, ..... .N) to one dimension. Given the absence of explicit annotations indicating frame quality, we instead leverage the paired text weak supervision to optimize the scorer network. This strategy is based on the assumption that frames highly relevant to the text are less likely to be of low quality. Specifically, during the training process, we employ the scores of selected frames as weights to aggregate the frames features as Eq. 2, and then utilize the VTC loss (detailed in section 3. 1) to optimize the scorer network. Finally, we retain the frames with the top K scores. During inference, we score each video frame by the trained scorer network to filter out the low-quality frames.
where d is the dimension of Ky, The dot product attention (ttn(Qu, (Q,, Ky) RIxN provides rele- vancy weights from a text to each frame. Through the attention mechanism, the fused video feature fry is encouraged to incorporate the text semantics. To optimize the selection module, we employ the Contrastive Loss (CL) to pull closer the fri and the paired fr in each batch B, denoted Ca:
B D log
UA-AA
Ld = B
(7)
where À is learnable temperature parameter. Fi- nally, select the frames with the top K attention scores to the text.
3.3 Training and Inference
3.2.2 Text-guided Frame Selection
3.3.1 Training
contrast to the text-free frame selection, the text- guided frame selection considers the query text information to help filter the inessential frames, which contain the non-interactive and interactive frame selections.
During training. we optimize the model mainly with two losses, i.e., video-text contrastive loss (VTC) and video-text matching loss (VTM). In par- ticular, we also append the Contrastive Loss (CL. to optimize the selection encoder of the interactive frame selection.
Non-interactive Frame Selection. It is efficient to directly calculate the similarities between the representations of frames and the given text for the frame selection. Specifically, we compute the cosine distances, followed by softmax layer, as the relevancy score of frame F; and the given text T, denoted a,:
Video- Tideo-Text Contrastive (VTC) pulls the repre- sentations of paired video and text close and pushes away the unpaired ones. Specially, given the [CLS] representation of video fv and text frs we calcu- late in-batch symmetric VTC loss as follows:
1 [ + Lvz0).
0; = softmar(s((FF fr)), (())),
(4)
Lute
(8)

---

Table 1: Performance comparison different frame selection methods on MSR- VTT and DiDeMo. We gray out the best performance in each row. The 'base" in the Frame column represents the video processing only with the pre-sampling and without further using the frame selection.
Text-Free
Text-Guided
Uni
Rand
Redun-A
LQ-A
N-InT
InT
Dataset
Frame
R@ 11
R sumt
R@ I1
R
sumt R@ If R sumt
R@ If R
sumt
R@ If
R
susu
R
If R@sumf
16(base)
52.9
212.9
52.9
212.9
52.9
212.9
52.9
212.9
52.9
212.9
52.9
212.9
16=+1 12
51.7
212.6
51.8
212.8
52.3
213.0
51.1
209.6
50.4
210.6
50.8
207.7
16
50,9
209.9
52.4
211.1
50.8
210.5
49.6
206.5
47.3
202.1
48.9
204.3
MSR-VTT
16=66
51.4
209
49.7
206.9
52.2
211.4
47.8
202.5
46.3
200.3
48.5
204.8
16=44
47.9
202.0
47.8
201.9
49.0
205.3
45.2
196.6
44
193.8
46.0
197.4
16=+1
24.3
125.6
26.5
136.3
39.4
176.1
33.8
158
41.7
185.4
32.6
156.3
32(hase)
58.4
229.2
58.4
229.2
58.4
229.2
58.4
229.2
58.4
229.2
58.4
229.2
32->24
57.8
228.1
58.7
228.6
58.3
229.7
59.2
230.7
58.6
228.6
58.6
226.4
32-+16
58.1
227.4
58.0
227.6
59.3
231.1
57.7
228.7
56.3
223.9
57.6
225.6
DiDeMo
32=112
56.9
225.7
57.6
226.5
58.3
231.3
56.6
226.7
55.2
224.2
55.7
222.8
32-+8
54.6
223.4
53.8
222.9
57.7
230.7
55,9
224.5
54.0
221.2
55.2
220.9
32-+4
52
215
52.5
215.4
57.0
225.8
51.2
213.5
51.6
217.2
49.8
215.5
32=1
33.2
166.3
34.2
164.5
38.5
180.6
36.8
173.5
47.2
203.6
41.7
191.8
where Livv and L.21 denotes text-to-video and video-to-text loss, respectively, formulated as:
Experiment
4.1
Experimental Settings
Datasets and Implementation. We conduct ex- periments on three common TVR benchmarks, in- cluding MSR-VTT (Xu et al., 2016), DiDeMo (Anne Hendricks et al., 2017) and ActivityNet Cap- tions (Caba Heilbron et al., 2015). The details of each benchmark and implementation are intro- duced in Appendix A and B, respectively.
Lzv
log
B
B
j=1
B
L,21 = B
1
Σ
log
(10)
where F is learnable temperature parameter and is the batch size.
Video-Text Matching (VTM) determines whether a video-text pair is matched or not. We feed the multimodal encoder's s output gone into binary classifier to predict two-class probability Potm- The VTM loss is formulated as:
Evaluation Metric. To evaluate the retrieval per- formance, employ three retrieval metries: recall at rank M (R @M, M=1, 10), median rank (MdR) and the sum of the recall (R@ @um)) @@M calcu- lates the percentage of the successful retrieval which the paired sample to the query is found in the top-M retrieved results. MdR measures the median rank of correct items in the retrieved ranking list. R @ sum is calculated by summing R@ 1, R R@5, and R @ 10, reflecting the overall retrieval performance.
Cutm E(y.yy- (yotm: Potm), (11)
where Yetm is a a2 2-dimensional one-hot vector rep- resenting the ground-truth label. In conclusion, the full training objective is:
(12) where | = when we use the interactive frame selection, otherwise 3 - 0.
L = Lootm Lute + Bhat,
4.2 Analysis of Frame Selection
In this section, we first compare the frame selection methods on retrieval performance. Subsequently, we investigate the impact of each frame selection method on retrieval efficiency. At last, based on the explore
3.3.2 Inference
To speed up the inference, as BLIP (Li et al., 2022b), we first compute the feature similarities S/20 for all text-video pairs by the vision and text encoders, and then choose the first top-128 video candidates according to Si2e for further ranking by the multimodal encoder. In addition, with our frame selections, the inference more efficient.
performance and efficiency analysis, combining multiple frame selection methods to de- termine the optimal method. All the analyses are based on the experimental results on MSR-VTT and DiDeMo, representing two canonical TVR types, i.e., standard sentence-to-viee retrieval and

---

Table 2: Efficiency comparison of frame selection methods under the best performance. McM is the max GPU memory cost during training. Time the interaction duration per query with all videos during inference.
MSRVTT
DiDeMo McM(GB)
Method
Frame
R@1
McM(GB)
GLOPs
Time(ms)
Frame
R@1
GLOPs
Time(ms)
Base
16
52.9
19.2
1107.6
245.8
32
58.4
33.4
1997.1
462.9
Uni
16=>12
15.8
836.7
191.9
32->24
57.8
26.4
1345.1
359.4
Rand
16=+12
51.8
15.8
836.7
191.9
32->24
58.7
26.4
1345.1
359.4
Redun-A
16=26
52.2
14.9
861.7
112.1
32=>16
59.3
26.4
1564.1
241.9
LQ-A
16=>11
17.6
1042.2
189.8
32=>24
59.2
30.0
1842.8
359.9
N-InT
16=+12
50.4
17.6
1070.1
197.2
32-*24
58.6
30.0
1854.6
365,7
InT
16=+1
50.8
17.6
1098.6
210.6
32==44
58.6
30.0
1895.7
410.1
Table 3: Performance of combinations of frame selec- tion methods. The best results are bold. The second best results are underline.
tion performs better than the text-guided frame lection. We conjecture that the text-guided frame selection introduces the textual bias for TVR. This means that by emphasizing the text-related frames, the similarities videos partially related to the given text may be improved, potentially resulting in extra disturbance for TVR.
NTT
DiDeMo
Redun-A LQQA N-InT
Frame R 1 R R@sum1]
Frame
1811R R sumt
16
52.9
212.9
32
58.4
229.2
16=s 12
52.3
213.0
12==16
59.3
231.1
16=+
51.1
209.6
32=>24
59.2
230.7
16=+12
50.4
210.6
32=>24
58.6
228.6
4.2.2
Efficiency of Each Frame Selection
16=>12 52.7
213.2
32=>16
59.5
232.1
16=+12
213.9
32=16
232.2
In this subsection, we explore the retrieval effi- ciency of different frame selection methods under the best performance in Table 2. Uni and Rand determine the K key frames prior to the vision en- coder, which means only fewer K frames need to be encoded, resulting in the lowest memory cost and GLOPs. The Redun-A filters out most of the re- dundant frames through clustering and only retains 16 most representative frames, which gives rise to the fastest inference speed, simultaneously accom- panied with remarkable performance of 59.3% at R @ 1. Although the LQ-A achieves competitive performance compared to Redun-A, it comes at the cost of increased frame count, resulting in lower efficiency. The N-In'T simply uses dot product between the whole frames and the given query select the key frames, bringing marginal addi- tional time consumption and comparable efficiency. Since the InT requires an additional fusion for the selection, it eventually exhibits an inferiority of re- trieval efficiency. In conclusion, the Redun-A and N-InT are two reasonable individual frame selec- tion methods for text-free and text-guided frame selection, respectively.
/
16=+12 52.6
213.1
32=
59.2
232.3
paragraph-to-vides retrieval, respectively.
4.2.1 Effectiveness of Each Frame Selection
Frame selection aims to select fewer K frames to represent video for retrieval. Table shows the re- trieval performance under different frame selection methods with the setting of K {12,8, 8, 6, 1, 1) for the MSR-VTT and K (24, 16, 12, 8, 1, for the DiDeMo. We can see that: (1) Redun-A cxhibits an overall superiority compared to the ex- isting methods (i.e., Uni and Rand) in reducing temporal redundancy in videos. Compared to the widely-used Uni and Rand which have not consid- ered the distribution of crucial information in the video contents, Redun-A clusters the frames with similar contents to select the most representative ones and discard other redundant frames. As result, Redun-A can effectively mitigate the perfor- mance decrease caused by the reduction of frame number. (2) When only selecting a single frame (i.e., K 1) for TVR to maximize the retrieval effi- ciency, the text-guided frame selections, especially the N-InT, present significant advantage com-- pared to the text-free frame selections. The text- guided frame selection can retain the most query- relevant frame in video and facilitate the retrieval.
4.2.3 Combinations of Frame Selections
We combine various frame selections explore better trade-off between efficiency and perfor- mance. According to the aforementioned analy- sis cach individual frame selection method, we choose Redun-A for the text-free frame selection
(3) When considering more frames (i.e., > 1) for higher performance, the text-free frame selec-

---

Table 4: Results of text-video retrieval on MSR- VTT and DiDeMo datasets. The numbers on the left and right of + respectively denote the number of frames sampled from raw videos and frames used for inter-modal interaction after frame selection. denotes our implementation of baselines.
MSR-VTT
DiDeMo
Method
Frames R@ 1f R@51 R 10 sum MdR, Frames R@ R@ R@ 101 R@sumf MdR,
(Liu et al., 2019)
20.9
48.8
62.4
132.1
16.1
82.7
139.9
8.3
ClipBERT (Lei et al., 2021)
22.0
46.8
59.9
128.7
16
20.4
48.0
60.8
129.2
6
Frozen (Bain al., 2021)
4
32.5
61.5
71.2
165.2
4
34.6
65.0
74.7
174.3
3
TW-BERT (Yang al., 2023)
20
38.4
65.1
76.6
180.1
20=
41.8
81.2
194.1
CLIP4Clip (Luo et al., 2022)
12
44.5
71.4
81.6
197.5
64
43.4
69.9
80.2
193.5
2
MOF (Han et al., 2022)
12=24
40.5
68.2
79.5
188.2
12=4
41.3
68.5
79.4
189.2
2
CAMoE (Cheng
2021)
47.3
74.2
84.5
206
CenterCLIP (Zhao et al., 2022)
12
48.4
73.8
82.0
204.2
X-CLIP (Ma et al., 2022)
12
49.3
75.8
84.8
209.9
64
47.8
79.3
Ts2net (Liu et al., 2022)
75.6
85.3
210.3
41.8
71.6
82.0
195.4
TABLE (Chen et al., 2023)
12
47.1
74.3
82.9
204.3
32
47.9
74.0
82.1
204
2
IBBI (Jin et al., 2023)
12
48.6
74.6
83,4
206.6
64
46,9
74.9
82.7
204.5
2
Cap4Video (Wu et al., 2022)
51.4
75.7
83.9
211
52.0
79.4
87.5
218.9
X-pool' (Gorti al., 2022)
46.6
202.5
44.7
72.4
80.5
197.6
X-poo((ecdunAAALO0- X pool(RedunAA-N-In
12-38
46.2
72.7
82.5
201.4
32-+
16
44,9
73.1
82.0
200.0
2
12=88
46.7
72.6
83.2
202.5
32 16
45.2
73.1
82.5
200.8
2
BLIP' (Li et 2022b)
52.9
75.9
84.1
212.9
32
58.4
82.8
88.0
229.2
1
SerRRedunAA+lQQ--. SLIP(RedunAANNAIIT
16=112
52.7
75.9
84.6
213.2
32=
59.5
83.5
89.1
232.1
16:>12
52.6
76.5
84.8
213.9
32-> 16
59.7
83.7
88.8
232.2
1
Table 5: Results of text-to-video retrieval ActivityNet Captions dataset.
selection.
For example, compared with the individual Redun-A, the combination of Redun-A and LQ- can achieve an R sum improvement of 0.9% on DiDeMo due to the elimination of the low-quality frames. When taking the text information into con- sideration for frame selection, the combination of Redun-A and N-InT gets the best performance R @ sum, consistently surpassing all of the single frame selection methods. We also observe that combining of the three frame selections could not bring additional gains compared with the com- binations of the two methods. We hypothesize that the N-InT could cover the ability to filter the low-quality frames to some extent. In addition, provide the performance comparison between the combination of text-free frame selection(i.e.. the combination of Redun-A and LQ-A) and the traditional key frame selection technology the Appendix C.
Method
Frames Ke 11R88)555R Re sum? MdR
2019)
ClipBERT (Lei al. 2021)
21.3
49.0
63.5
133.8
Prozen (Rain al,, 2021)
28.8
60.9
CLIP4Clip (Luo al... 2022
Ts2nel (Liu et 2022)
64
41.0
73.6
84.5
199.1
X-CLIP (Ma ot 2022)
46.2
75.5
IIBI (Jin
BLIP" (Li al., 20226) 20226)
32
54.3
80.6
55.8
223.7
BLIPMRedon-AALLQ-A
32=+24
54.6
80.0
88.8
223,4
BUJPRRedun-ANN-InT 62-524
24
54.8
80.4
89.0
224.2
and N-InT for the text-guided frame selection. In addition, considering the competitive performance of the LQ-A, we also combine it with the Redun-A further towards better performance of text-free frame selection. the combinations, we calculate the cumulative scores by summing the frame scores obtained from each single frame selection method and select the top-K frames with the highest scores. In particular, when combining the Redun-A, we fix the clustering number (denoted as Z) referred to the retrieval performance of the single Redun-A method, that is Z 6 for MSR-VTI and 16 for DiDeMo. We assign score of 1/Z each selected frame, while for the others. it is set 0. Table presents the results of the various combinations. By combining multiple frame selection methods, we can get better performance than a single frame
4.3 Comparison to State-of-the-Art
To further demonstrate the effectiveness of the frame selection in TVR, we compare our method with the state-of-the-art methods in Table 4 and Table 5. worth noting that the pure BL.I has outperformed the current SOTA methods, which in- dicates that a pretrained model with a deeper fusion

---

module benefits the TVR. However, the accompa- nying drawback of the deeper fusion lies in the low efficiency. Through our frame selection, we get a better trade-off between performance and effi- ciency. Specifically, with proper frame selections (c.g., Redun-A + N-InT), the performance of BLIP could have a 1.0%. 3.0%, and 0.5% improvement at R sum, while the frame number used for deeper fusion has a considerable decrease to 75%, 50% and 75% on MSR-VTT, VTT, DiDeMo and ActivityNet Captions, respectively. To verify the generalization ability of our method, we also conduct the frame selection methods on other baseline X- X-pool (Gorti et al., 2022) on MSR-VTT and DiDeMo datasets, as shown in Table 4. The table also clearly indi- cates that frame selection (e.g., Redun-A+ N-In'T) effectively mitigate performance degradation despite using fewer frames, which typically results in higher retrieval efficiency.
investigations to future work.
References
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. 2017. Localizing moments in video with natural language. In Proceedings of the IEEE international conference on computer vision, pages 5803-5812.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zis- serman. 2021. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1728-1738.
Shyamal Buch, Cristóbal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. 2022. Revisiting the" video" in video-language understand- ing. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2917-2927.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. 2015. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference computer vision and pattern recognition, pages 961-970.
5 Conclusion
this paper, we make an empirical study to ex- plore the optimal frame selection methods for bal- ancing the retrieval performance and efficiency in TVR. Beyond the existing four frame selection methods, we additionally develop two text-free paradigms, i.e., redundancy aware frame selection. We conduct abundant exper- iments to analyze the impact on TVR of the above six frame selections under the condition of various frame number, and make a comprehensive com- parison from the perspective of retrieval efficiency and resource consumption. Extended experimental results on multiple TVR benchmarks effectively demonstrate that with proper frame selection, we not only get a competitive performance but also realize higher retrieval efficiency under lower re- source consumption. Hopefully, this study could spur further study on deeper interaction between text and video in TVR.
Hyun Sung Chang, Sanghoo Sull, and Sang Uk Lee. 1999. Efficient video indexing scheme for content- based retrieval. IEEE Transactions on Circuits and Systems for Video Technology, 18):1269-1
and www-quality-
Shih-Fu Chang. 2003. Content-based video summa- rization and adaptation for ubiquitous media access. In 12th International Conference on Image Analysis and Processing, 2003. Proceedings., pages 494-496. IEEE.
Dongsheng Chen, Chaofan Tao, Lu Hou, Lifeng Shang, Xin Jiang, and Qun Liu. 2022. Litevl: Efficient video- language learning with enhanced eatial-temporal modeling. arXiv preprint arXiv:2210. 17929.
Yizhen Chen, Jie Wang, Lijian Lin, Zhongang Qi, Jin Ma, and Ying Shan. 2023. Tagging before alignment: Integrating multi modtl tags for video-text retrieval. arXiv preprint 2311v22301.12644.
Xing Cheng, Hezheng Lin. Xiangyu Wu, Fan Yang, and Dong Shen. 2021. Improving video-text retrieval by multi-stream corpus alignment and dual softmax loss. arXiv preprint arXiv: 049.0990
Limitations
This empirical study provides insight into the opti- mal frame selections for TVR. The major limitation may be manually searching for shared number of key frames in a dataset. However, video typ- ically contains a various number of key frames. Therefore, a more desirable frame selection method would be able to automatically determine an appro- priate number of key frames based on the content of each video individually. We leave the related
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre- Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint 1010.v::110048000
Ajay Divakaran, Regunathan Radhakrishnan, and Kadir A Peker. 2002. Motion activity-based extrac- tion of key-frames from video shots. In Proceedings. International Conference on Image Processing, vol- ume I, pages 1-1. IEEE

---

Alexey Dosovitskiy, Lucas Beyer. Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 11929.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer.
Bo Fang, Chang Liu, Yu Zhou, Min Yang, Yuxin Song. Fu Li, Weiping Wang. Xiangyang Ji, Wanli Ouyang, et al. 2023. Uatvr:
Yuanze Lin, Chen Wei, Huiyu Wang, Alan Yuille, and Cihang Xie. 2022. Smaug: Sparse masked autoen- coder for efficient video-language pre-training. arXiv preprint arXiv:2211. 1114..
adaptive text-video retrieval. arXiv preprint arXiv:2301.06300
Valentin Gabeur, Chen Sun, Karteck Alahari, and Cordelia Schmid. 2020. Multi-modal transformer for video retrieval. In Computer Visi 2020: 16th European Conference, Glasgow, UK, August 23- 28, 2020, Proceedings, Part IV 16. pages 214-229. Springer.
Yang Liu, Samuel Albanie, Arsha Nagrani, and An- drew Zisserman. 2019. Use what you have: Video retrieval using representations from collaborative ex- perts. arXiv preprint arXiv: 13487
Yuqi Liu, Pengfei Xiong, Luhui Xu, Shengming Cao, and Qin Jin. 2022. Ts2-net: Token shift and selection transformer for text-video retrieval. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XIV, pages 319-335. Springer.
Satya Krishna Gorti, Noël Vouitsis, Junwei Ma, Key- Golestan, Maksims Volkovs, Animesh Garg, and Guangwei Yu, 2022. X-pool: Cross-modal language- video attention for text-video retrieval. In Proceed- ings of the LEEE/CVF Conference on Computer Vi- sion and Puttern Recognition, pages 50155
Ning Han, Xun Yang. Be-Peng Lim, Hao Chen, and Qianru Sun. 2022. Efficient cross-modal video re- trieval with meta-optimized frames. arXiv preprint arXiv:2210088452.
Ilya Loshchilov and Frank Hutter. 2017. pled weight decay regularization. arXiv preprint 171.0:1711.05101.
Decou-
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. 2022. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing, 508:293- 304.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034.
Yiwei Ma. Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji. 2022. X-clip: End-to- end multi-grained contrastive learning for video-text retrieval. In Proceedings of the 30th ACM Interna- tional Conference on Multimedia, pages 638-64) 338-647.
Peng Jin, Jinfa Huang. Pengfei Xiong. Shangxuan Tian, Chang Liu, Xiangyang Ji, Li Yuan, and Jie Chen. 2023. Video-text as game players: Hierarchical banzhaf interaction for cross- cross-modal representation learning. arXiv preprint arXiv:2303. 74369.
KeplerLab. 2019. Katna: Tool for automating video keyframe extraction, video compression, image au- tocrop and smart image resize tasks.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International confer- ence on machine learning, pages 8748-8763. PMLR.
Jie Lei, Linjie Li. Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. 2021. Less is more: Clipbert for video-anddlanguage learning via sparse sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 7331-7341.
Zixu Wang, Yujic Zhong, Yishu Miao, Lin Ma, and Lu- cia Specia. 2022. Contrastive video- language leamn ing with fine-grained frame sampling. arXiv preprint 2rXiv:2210.050:9.
Dongxu Li. Junnan Li, Hongdong Li. Juan Carlos Niebles, and Steven CH Hoi. 2022a. Align and prompt: Aideo-and-languee prompts. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 4953-4963.
Wenhao Wu, Haipeng Luo, Bo Fang, Jingdong Wang, and Wanli Ouyang, 2022. Cap4video: What can auxiliary captions do for text-video retrieval? arXiv preprint arXiv:2301.001
pre-training with entity
Junnan Li. Dongxu Li. Caiming Xiong, and Steven Hoi. 2022b. Blip: Bootstrapping language-image pre-training for unified vision-language understand- ing and generation. International Conference on Machine Learning, pages 12888-12900. PMLR.
Jun Xu, Tao Mei. Ting Yao, and Yong Rui. 2016. Msr- vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE con- ference on computer vision and pattern recognition. pages 5288-5296.

---

Xu Yang. Zhangzikang Li, Haiyang Xu, Hanwang Zhang. Qinghao Ye, Chenliang Li, Ming Yan, Yu Zhang, Fei Huang, and Songfang Huang. 2023. Learning trajectory-word alignments for video- language tasks. arXiv preprint FrXXi:2301.019950
Table 6: Performance comparison between ours Redun-
A+LQ-A and Katna DiDeMo. Both methods are designed to eliminate redundant and low-quality frames.
Method
Frames R It RØ55 RG-101 R@sum? MdR
52.3
77.8
85.7
215.8
Youngjae Yu, Jongseok Kim, and Gunhee Kim. 2018. A joint sequence fusion model for video question answering and retrieval. In Proceedings of the Euro- pean Conference on Computer Vision (ECCV), pages 471-487.
Katna (KeplerLab, 2019)
16
56.0
81.0
86.6
223.6
1
8
$7.9
82.3
88.3
228.5
I
Redun A-LQA
16
59.5
83.5
89.1
252.1
1
that has been fine-tuned on the COCO (Lin et al., 2014). As for the extra parameters of the frame selection module of the LQ-A and InT, we use Kaiming Normal (He et al., 2015) initialization. To convert video into a frame sequence, we uni- formly pre-sample 16, 32, and 32 frames from each video of MIRRVVT,, DiDeMo, and Activi- tyNet Captions, respectively, and then resize each frame to 224 The maximum textual length is set as 35 for MSR-VTV and 64 for the other two datasets. We set the batch size to 4 and set the learning rate for BL.P-initialized weights to 5e-6 and for other parameters to 5c-4. We optimize our model for 5 epochs using the Adam W optimizer (Loshchilov and Hutter, 2017) with weight decay of 0.05 and decay the learning rate using cosine schedule.
Shuai Zhao, Linchao Zhu, Xiaohan Wang, and Yi Yang. 2022. Centerclip: Token clustering for efficient text- video retrieval. In Proceedings of the 45th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval, pages 970- 981.
A Datasets
We introduce the three involving benchmarks for TVR as follows:
MSR-VTT contains 10K videos, each of which ranges from 10 to 32 seconds and is cap- tioned with 20 descriptions. Each description provides a brief overview of the corresponding video for standard sentence-to-video retrieval. We follow the widely-used data split in previ- ous methods (Gabeur et al., 2020; Luo et al., 2022) to train the model on the training-9K set and test on the test 1k-A set.
Except for the BLIP, we also employ another baseline X-pool (Gorti et al., 2022) to verify the effectiveness of the optimal frame selection meth- ods in Section 4.3. We follow the default settings of the X-pool for MSR- VTT. For the DiDeMO, we set the batch size to 16 and the pre-sampled frames to 32. For the scorer network of LQ-A, set its learning rate to 5e-4.
DiDeMo includes 10K unedited videos paired with 40K descriptions. Each video divided into multiple segments, and each segment is accompanied by a corresponding description. Following previous works (Lei et al., 2021; Luo et al., 2022), we conduct paragraph-to- video retrieval by concatenating all descrip- tions of each video into single query.
C Comparison to Traditional Key Frame Selection Method
Key frame extraction techniques (Chang et al.. 1999; Divakaran et al., 2002; Chang, 2003) have been developed in the past decades. However, these techniques are generally based on the low-level pixel-level understandings of videos (e.g. frames' color feature) for the key frame selection. For instance, Katna (KeplerLab, 2019) identifies re- dundant and blurry frames by initially applying K-means Clustering to frames using image his- tograms. It subsequently selects the best frames from clusters based on the variance of the Laplacian and the overall sharpness of the frames. In Table 6, we compare the retrieval performance achieved by applying Redun-A+LQ-A A+LQ-A and Katna for frame selection, respectively. Both methods are designed
ActivityNet Captions contains 20K videos annotated with 100K descriptions. The train- ing set contains 10K videos, and we use the vall set with 4.9K videos to report results. Following (Gabeur et al., 2020; Luo et al., 2022), all descriptions of a vidco are also con- catenated together for paragraph-to--iddeo re- trieval.
B Implementation Details.
All experiments are conducted on NYIDIA A40 GPUs. We initialize the model parameters used for empirical study from the BLIP (Li et al., 2022b)

---

Query: A car is in wreek.
Redundancy-aware frame selection
Score
Frame
Low quality-aware frame selection
Nowr
Frame
Non-interactive frame selection
Scure
Frame
Interactive frame selection
Figure 2: Visualization of the selected frames under different frame selection methods. We select 6 key frames with the highest scores in the raw pre-sampled frames. Bar plots show the corresponding scores the displayed frames.
to eliminate redundant and low qua-qualty frames. Our Redun-A+LQ-A exhibits significant advantage in TVR compared to Katna, with performance im- provements of 3.5% and 5.6% at R@ when the frame number is set to 16 and 8, respectively. Our Redun-A+L.Q-A select frames using frame features computed by the video encoder, which encapsulate the understanding of video contents. In compari- son, the Katna pré-processes videos based on color features. On hand, Katna fails prioritize the primary area of interest in frames during the selection process. On the other hand, the combina- tion of the Katna and deep retrieval model could only done by video pre- prepprocessing. Therefore, our frame selection methods are more friendly to the deep retrieval model and thus get better perfor- mance.
D Visualization of Frame Selection
We display visualization results under various frame selection methods in Figure 2. We can see that the selected frames could effectively reflect the purpose of each selection method. Specifi- cally, the redundancy-aware frame selection retains the most diverse information of the pre-sampled frames, while the www-quality-aware frame selec- tion effcctively recognizes the blurry frames (i.c., the 2nd thth frames). In contrast, the text-guidee guided frame selection, including the non-interactive and the interactive frame selection, can retain almost all the frames that are related to the query text. The vi- sualization illustrates that the involved frame selec- tion methods could effectively filter the inessential frames based the respective criteria.


Reviews:
[{"review_id": "N9WedrHFP1", "reviewer": "Reviewer_ELKD", "summary": "", "questions": "Have you considered developing intuitive explanations for why your proposed techniques work better than the state of the art?\nWhat is the novelty of your redundancy aware approach over Zhao et al (whom you cite) given that you seem to be doing exactly what they did?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments and suggestions, which are exceedingly helpful for us to improve our manuscript. Our point-to-point responses to your comments are given below.\n\nIn the following, your comments are first stated and then followed by our point-to-point responses.\n\n\n\n>Q1:An inspection of the bibliographies of these papers will reveal that there have been a number of keyframe extraction techniques developed in the past. Would be nice of the authors to address why their technique establishes novelty over those methods.  \n\nThanks for your insightful comments and constructive suggestions. The techniques have shown charmful performance in the multimedia community in the past decades and could be one of the reasonable methods for frame selection nowadays. However, the past techniques[1,2,3] are generally based on the low-level pixel-level understandings of videos (e.g. frames' color feature) for the keyframe selection. In comparison, our frame selection methods enable the selection of frames grounded in **high-level semantic comprehension of videos**, and can be naturally integrated into the deep retrieval model to empower an **end-to-end** learning.\n\nIn addition, to further show the effect of the semantic-level understanding of videos for frame selection in TVR, we apply the existing keyframe extraction technology[4] (called *Katna*) for TVR. Initially, it employs K-means Clustering on frames using image histograms to identify **redundant frames**. Subsequently, from the optimal frame within these clusters, Katna detects **image blur**. This detection is rooted in the analysis of both the variance of Laplacian and the frame's overall sharpness. The results are shown below. \n\n|    Method    | Frame | R@1  | R@5  | R@10 | R@sum | MdR  |\n| :----------: | :---: | :--: | :--: | :--: | :---: | ---- |\n|    Katna     |  16   | 56.0 | 81.0 | 86.6 | 223.6 | 1    |\n| Redun-A+LQ-A |  16   | 59.5 | 83.5 | 89.1 | 232.1 | 1    |\n|    Katna     |   8   | 52.3 | 77.8 | 85.7 | 215.8 | 1    |\n| Redun-A+LQ-A |   8   | 57.9 | 82.3 | 88.3 | 228.5 | 1    |\n\n\\*  Comparison between retrieval with frames extract by the Katna and ours on **DiDeMo**.\n\nOur Redun-A+LQ-A surpass the retrieval performance of Katna significantly. In comparison, our Redun-A and LQ-A select frames using frame features computed by the video encoder. These features encapsulate the comprehension of video content. However, the Katna pre-processes videos based on color features. On one hand, the combination of the Katna and deep retrieval model could only be done by video pre-processing. On the other hand, the method fails to prioritize the primary area of interest during the selection process. Therefore, our frame selection methods are more friendly to the deep retrieval model and thus get better performance.\n\nWe will add bibliographies of past video summarization technologies and clarify the difference of our deep frame selection compared with these technologies in the revised version. The experimental comparison of ours with Katna will also be appended in the revised version.\n\n[1] Chang H S, Sull S, Lee S U. Efficient video indexing scheme for content-based retrieval[J]. IEEE Transactions on Circuits and Systems for Video Technology, 1999, 9(8): 1269-1279.\n\n[2] Chang S F. Content-based video summarization and adaptation for ubiquitous media access[C]//12th International Conference on Image Analysis and Processing, 2003. Proceedings. IEEE, 2003: 494-496.\n\n[3] Divakaran A, Radhakrishnan R, Peker K A. Motion activity-based extraction of key-frames from video shots[C]//Proceedings. International Conference on Image Processing. IEEE, 2002, 1: I-I.\n\n[4] https://github.com/keplerlab/katna\n\n> Q2: 1. Have you considered developing intuitive explanations for why your proposed techniques work better than the state of the art? \n\nThanks for your question. Our techniques are designed for detecting **redundant and noisy frames** within videos for TVR. On the one hand, the retrieval with redundant frames leads to unnecessary computational load and time consumption. On the other hand, noise frames, such as those of low-quality and text-irrelated frames, not only bring unnecessary resource expenditure but also disrupt the alignment between texts and their associated videos. We **simultaneously** address both scenarios by empirically combining multiple frame selection methods to refine video frames. We thus achieve both satisfying effectiveness and efficient retrieval outcomes. \n\nWe will provide more intuitive explanations for the effectiveness of our methods in the revised version.\n\n\n\n> Q2: 2. What is the novelty of your redundancy aware approach over Zhao et al (whom you cite) given that you seem to be doing exactly what they did?\n\nThanks for your question. Compared to the CenterCLIP[5], we propose a new strategy, **frame-level frame selection**, for solving video redundancy. On the one hand, CenterCLIP operates at the patch token-level features to identify video redundancy, whereas our method operates at the frame-level features, which contributes to more efficient information selection. On the other hand, CenterCLIP focuses on constraint redundancy within consecutive frames, whereas our frame-level frame selection approach reduces redundancy across the entirety of the video contents. By the frame-level frame selection, our proposed selection strategy gains both higher efficiency and performance. \n\nFor the retrieval performance, we conduct the token-level information selection (CenterCLIP) in our framework. We ensure the number of selected tokens in CenterCLIP corresponds to the same informational quantity as the number of frames in the Redun-A approach. The results are shown in the table below.\n\n\n|   Method   | R@1  | R@5  | R@10 | R@sum | MdR  |\n| :--------: | :--: | :--: | :--: | :---: | :--: |\n| CenterCLIP | 59.7 | 81.6 | 87.3 | 228.6 |  1   |\n|  Redun-A   | 59.3 | 83.3 | 88.5 | 231.1 |  1   |\n\n\\*  Comparision between CenterCLIP with Redun-A based $K=16$ on **DiDeMo**.\n\nOur frame selection achieves better performance than CenterCLIP overall. Our Redun-A **retains the integrity of the spatial information**, which is benifit to the learning of frame-level representation and further video representation, thus benefiting the TVR.\n\nFor the selection efficiency, the CenterCLIP clusters abundant video patch tokens (**N*P, P>>N**), while our Redun-A clusters among the frame-level tokens (**N**) in each video. As a result, in the information selection process, our Redun-A greatly saves computational resources and time consumption and thus exhibits a more efficient way for information selection.\n\nWe will append the explanations of the novelty and advantages of Redun-A compared to the CenterCLIP in the revised version.\n\n[5] Zhao S, Zhu L, Wang X, et al. Centerclip: Token clustering for efficient text-video retrieval[C]//Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2022: 970-981.\n\n> Q3. Line 092 please replace \"there lacks\" with \" the state of the art lacks\" ; Line 568 please replace \"State of the arts\" with \"State of the art\"\n\nThanks for your comments. We will correct the two minor mistakes and carefully check other possible errors in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Thorough review of the state of the art of post-deep learning key frame extraction methods.\n2. Clear improvement over the state of the art.\n3. Intuititively satisfying results.\n4. Well written paper.", "reasons_to_reject": "1, This is not a weakness peculiar to these authors. In general the video summarization literature seems to show ignorance of video summarization techniques developed in the multimedia community more than 20 years ago. Those were techniques that did not rely on the elaborate machine learning techniques on display at this moment but worked quite well. Here are three examples of such papers:\n\nhttps://www.semanticscholar.org/paper/Motion-activity-based-extraction-of-key-frames-from-Divakaran-Peker/515f899f61c33782ec78045fa849e27e607d9985\n\nhttps://www.ee.columbia.edu/~sfchang/papers/talk-iciap-0903.pdf\n\nhttps://ieeexplore.ieee.org/document/809161\n\nAn inspection of the bibliographies of these papers will reveal that there have been a number of keyframe extraction techniques developed in the past. Would be nice of the authors to address why their technique establishes novelty over those methods. \nNote that I don't want to single these authors out since the entire vision community seems to be reinventing past techniques while also genuinely exploiting the power of deep learning techniques that did not exist in the times of the three papers cited above.", "questions_to_authors": "Have you considered developing intuitive explanations for why your proposed techniques work better than the state of the art?\nWhat is the novelty of your redundancy aware approach over Zhao et al (whom you cite) given that you seem to be doing exactly what they did?", "paper_topic_and_main_contributions": "This paper presents two new video keyframe extraction techniques, the first based on redundancy reduction and the second based on quality detection, which are complementary approaches to keyframe extraction. The paper presents a thorough evaluation of state of the art techniques for key frame extraction on standard video retrieval datasets. The authors demonstrate clear advancement of the state of the art using their proposed novel keyframe extraction techniques.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Line 092 please replace \"there lacks\" with \" the state of the art lacks\"\n2. Line 568 please replace \"State of the arts\" with \"State of the art\""}, "extra_text": {"Missing_References": "https://www.semanticscholar.org/paper/Motion-activity-based-extraction-of-key-frames-from-Divakaran-Peker/515f899f61c33782ec78045fa849e27e607d9985\n\nhttps://www.ee.columbia.edu/~sfchang/papers/talk-iciap-0903.pdf\n\nhttps://ieeexplore.ieee.org/document/809161\n\nPlease see comments above to respond to these citations."}}, {"review_id": "thyr5WdQ8G", "reviewer": "Reviewer_FXRQ", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate your thoughtful comments in reviewing our manuscript. Our point-to-point responses to your comments are given below.\n\n\n\n> Q1: 1. in the training of the LQ-A module, is the text paired with a single frame or the whole video clip.\n\nIn the datasets of TVR, each text is paired with a video clip. Therefore, in the training of the LQ-A module, we constrain the text paired with the whole video clip by the VTM loss, formulated in Equal (8) ~ (10) in our manuscript.\n\n> Q1: 2. How to guarantee that the similarity can reflect the quality of a frame if the text is paired with a video clip? \n\nThe quality scores of the frames calculated by the LQ-A module are not in line with the similarity with the paired texts. The LQ-A evaluates the quality of the frames from the perspective of the video content itself rather than relying on the similarity to the particular text. During training, we first score each frame by the scorer network (formulated in Equal (3) in our manuscript) and then conduct a weighted mean based on the score of each frame in a video (formulated in Equal (2) in our manuscript). Finally, by the VTM loss, we optimize the scorer network of LQ-A. \n\nWhat's more, we provide more experiments to validate that the LQ-A's frame scores could reflect the frames' quality after training with texts. The principle is that **retrieval with low-quality frames performs worse**. We select high-quality frames with the highest K scores (denoted as top-K ) and low-quality frames with the lowest K scores (denoted as last-K) during inference. Subsequently, we proceed to compare the retrieval performance between these two sets. As shown in the tables below, the retrieval performance of high-quality frames exhibits an improvement of over 20% in terms of R@sum when contrasted with the performance of the low-quality frames. And with the frame number decreasing, the advantage of retrieval with the high-quality frame is more obvious. These imply that the discarded frames are of low quality and harmful for retrieval.\n\n| Frame  | R@1  | R@5  | R@10 | R@sum | MdR  |\n| :----: | :--: | :--: | :--: | :---: | :--: |\n| top-8  | 54.5 | 80.5 | 87.5 | 222.5 |  1   |\n| last-8 | 39.6 | 66.6 | 77.1 | 183.3 |  2   |\n| top-4  | 51.7 | 76.8 | 85.0 | 213.5 |  1   |\n| last-4 | 34.2 | 59.7 | 71.0 | 164.9 |  3   |\n| top-1  | 36.3 | 63.5 | 73.4 | 173.2 |  3   |\n| last-1 | 23.6 | 45.5 | 57.2 | 123.6 |  7   |\n\n\\*  Comparisions of retrieval with high-quality frames and low-quality frames on **DiDeMo**.\n\n> Q1: 3.  A partially matched frame could also play an important role in a video clip. \n\nThanks for your insightful comments. We agree that a partially matched frame could also play an important role in a video clip. However, the partially matched frames may contribute less to the correct retrieval. We could get a better trade-off between effectiveness and efficiency by discarding these frames with proper frame selection.  \n\n> Q1: 4.  How about considering the explicit lens movement, transition, and blurred frames, which are easy to detect, making the system interpretable? \n\nWe agree with the review that using video quality detection technology could make the system more interpretable. However, the final goal of the frame selection is to construct a **highly effective and efficient** retrieval system. \n\nFrom the efficiency perspective, our LQ-A offers an **end-to-end** video information processing method and could be integrated into the deep retrieval model conveniently. It incorporates a single multilayer perceptron for frame scoring, ensuring comparable efficiency to Katna. \n\nIn terms of effectiveness, we evaluate the retrieval performance of LQ-A compared to Katna[1], a method that offers video blur detection. As demonstrated in the table below, LQ-A outperforms Katna, potentially due to its ability to comprehend video content and make informed assessments.\n\n| Method | Frame | R@1  | R@5  | R@10 | R@sum | MdR  |\n| :----: | :---: | :--: | :--: | :--: | :---: | ---- |\n| Katna  |  16   | 56.0 | 81.0 | 86.6 | 223.6 | 1    |\n|  LQ-A  |  16   | 57.7 | 83.0 | 88.0 | 228.7 | 1    |\n| Katna  |   8   | 52.3 | 77.8 | 85.7 | 215.8 | 1    |\n|  LQ-A  |   8   | 55.9 | 81.2 | 87.4 | 224.5 | 1    |\n\n\\*  Comparison between retrieval with frames extract by the Katna and ours on **DiDeMo**.\n\nTherefore, our LQ-A is more reasonable for TVR from both perspectives of effectiveness and efficiency.\n\n[1] https://github.com/keplerlab/katna\n\n> Q2: How the LQ-A and Redun-A are combined? The calculation of the score of Redun-A is not clear.\n\nThe combination of the LQ-A and Redun-A is described in Lines 546 to 549 in the manuscript. We calculate the frame scores by LQ-A and Redun-A, respectively, and then sum the scores of corresponding frames. Finally, we select top-K frames with high scores. For the Redun-A, we fix the clustering number (denoted as $Z$ ) referred to the retrieval performance of the single Redun-A method. We assign a score of $1/Z$ to each selected frame, while for the others, it is 0. Thanks to the reviewer for reminding us of the important details of the score calculation of Redun-A, and we will append it in the revised version.\n\n> Q3: The overall performance of text-free methods is better than the text-guided methods which involve additional query information and higher computational cost. If it's properly implemented, could you please provide more analysis about it?\n\nThanks for your question. By our analysis, we conjecture that the text-guided frame selection introduces the **text bias** for TVR (described in Lines 503~510 in our manuscript). The video content encompasses a wide range of diversity. **Selecting frames according to their similarities with the given text may improve the similarities of the videos partially related to the given text, leading to extra disturbance for TVR.** We show the results of conducting N-InT during inference for a clearer understanding. The results are shown in the table below.\n\n| Training + Inference | R@1  | R@5  | R@10 | R@sum | MdR  |\n| :------------------: | :--: | :--: | :--: | :---: | :--: |\n|      Uni + Uni       | 51.7 | 76.1 | 84.8 | 212.6 |  1   |\n|     Uni + N-InT      | 50.4 | 75.5 | 84.4 | 210.3 |  1   |\n\n\\*  Comparision between inference with Uni and N-InT based $K=12$ on **MSRVTT**.\n\nFrom the table, the inference with N-InT leads to a decrease in performance, which verifies the text bias indeed destroys the performance of TVR. We will append more analysis and visualization in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors proposed two simple yet effective text-free methods to improve the frame selection of text-to-video retrieval\n- Experimental results show that the combination of two proposed text-free methods can outperform other text-free baselines. In the scenario of text-guided methods, the proposed redundant-aware method can also outperform the baselines.", "reasons_to_reject": "I have some concerns about the following points:\n- in the training of the LQ-A module, is the text paired with a single frame or the whole video clip? How to guarantee that the similarity can reflect the quality of a frame if the text is paired with a video clip? A partially matched frame could also play an important role in a video clip. How about considering the explicit lens movement, transition, and blurred frames, which are easy to detect, making the system interpretable?\n-  How the LQ-A and Redun-A are combined? The calculation of the score of Redun-A is not clear.\n- The overall performance of text-free methods is better than the text-guided methods which involve additional query information and higher computational cost. If it's properly implemented, could you please provide more analysis about it?   ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposed two simple approaches to sample frames for efficient text-to-video retrieval while remaining performance. They also compare the performance to existing baselines and demonstrate that the proposed method can achieve satisfactory performance.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "00cuwBppbl", "reviewer": "Reviewer_tb4G", "summary": "", "questions": "There are some writing errors in the paper as follows:\n1. Line 240:  F1 type error \n2. Figure 1(b): The index after frame selection should range from 1 to K. Only (I & II) meet this requirement.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments and suggestions, and they are exceedingly helpful for us to improve our paper. In the following, the comments are first stated and then followed by our point-to-point responses.\n\n\n\n> Q1: The evaluation indicators are not very reasonable. R@sum is rarely seen in articles in TVR.\n\nThank you for pointing out the problem of using the uncommon evaluation indicator R@sum. As the R@sum could reflect the overall performance of R@1, R@5 and R@10, we only display the R@1 and R@sum in Table 1 and Table 3 in the main paper in the condition of space limitation. We will add the results of all metrics(R@1, R@5, R@10 and MdR) in the Appendix of the revised version.\n\n> Q2: The experiment part uses too much space on the influence of different frame selections on the retrieval performance, which has little improvement. However, it spends little space on the effect of different frame selections on the retrieval efficiency, whose improvement is very obvious. \n\nThanks for your positive comments on the effect of our frame selection for efficient TVR. \n\nConsidering that the retrieval efficiency is explicitly related to the frame number (more frames mean lower efficiency and vice versa), we thus only present the efficiency comparison under **the optimal setting of frame number** for each method in Table 2 in our manuscript, i.e., comparing the efficiency of the methods with their optimal performance. The prerequisite for it is the detailed analysis of the influence of different frame selections on the retrieval performance in Table 1 in our manuscript.\n\nNevertheless, we also add the analysis of the retrieval efficiency on MSR-VTT datasets, as shown in the table below. \n\n| Method  | Frame | R@1  | MeM(GB) | GLOPs  | Time(ms) |\n| :-----: | :---: | :--: | :-----: | :----: | :------: |\n|  Base   |  16   | 52.9 |  19.2   | 1107.6 |  245.8   |\n|   Uni   |  12   | 51.7 |  15.8   | 836.7  |  191.9   |\n|  Rand   |  12   | 51.8 |  15.8   | 836.7  |  191.9   |\n| Redun-A |   6   | 52.2 |  14.9   | 861.7  |  112.1   |\n|  LQ-A   |  12   | 51.1 |  17.6   | 1042.2 |  189.8   |\n|  N-InT  |  12   | 50.4 |  17.6   | 1070.1 |  197.2   |\n|   InT   |  12   | 50.8 |  17.6   | 1098.6 |  210.6   |\n\n\\*  Efficiency comparison of frame selection method under the best performance on **MSRVTT**.\n\nWe speed up the inference by the frame selection, as shown in the last column. The Uni and Rand determine the K keyframe before the vision encoder, resulting in the lowest MeM and GLOPs. The Redun-A retains the least keyframes and thus the lowest MeM. Compared to the text-free methods, the text-guided methods are less efficient. We will add the experimental results on MSRVTT in the revised paper.\n\n> Q3: 1. In experiments, when combining different frame selections, the paper does not show how the hyper parameter K affects the results, which is similar to the analyses in Table 4.\n\nWe appreciate that you point out that we ignore the more detailed discussion about the influence of the hyperparameter K in the combination of frame selection methods. We thus append the experiments with different hyperparameters of K, shown in the below tables.\n\n| Frame |   R@1    |   R@5    |   R@10   |   R@sum   | MdR  |\n| :---: | :------: | :------: | :------: | :-------: | :--: |\n|  24   |   59.4   | **83.7** |   88.9   |   232.0   |  1   |\n|  16   | **59.5** |   83.5   | **89.1** | **232.1** |  1   |\n|  12   |   57.6   |   83.7   |   89.3   |   230.6   |  1   |\n|   8   |   57.9   |   82.3   |   88.3   |   228.5   |  1   |\n\n\\*  Retrieval performance of the combination of **Redun-A and LQ-A** under the different setting of $K$ on **DiDeMo**.\n\n| Frame |   R@1    |   R@5    |   R@10   |   R@sum   | MdR  |\n| :---: | :------: | :------: | :------: | :-------: | :--: |\n|  24   |   59.3   |   82.6   |   87.3   |   229.2   |  1   |\n|  16   | **59.7** | **83.7** | **88.8** | **232.2** |  1   |\n|  12   |   59.0   |   83.2   |   88.4   |   230.6   |  1   |\n|   8   |   55.8   |   82.0   |   87.6   |   225.4   |  1   |\n\n\\*  Retrieval performance of the combination of **Redun-A and N-InT** under the different setting of $K$ on **DiDeMo**.\n\n| Frame |   R@1    |   R@5    |   R@10   |   R@sum   | MdR  |\n| :---: | :------: | :------: | :------: | :-------: | :--: |\n|  24   | **60.1** |   83.6   |   88.9   | **232.6** |  1   |\n|  16   |   59.2   | **83.9** | **89.2** |   232.3   |  1   |\n|  12   |   57.7   |   83.9   |   89.1   |   230.7   |  1   |\n|   8   |   58.4   |   83.8   |   88.9   |   231.1   |  1   |\n\n\\*  Retrieval performance of the combination of **Redun-A, LQ-A and N-InT** under the different setting of $K$ on **DiDeMo**.\n\nFor DiDeMo, the optimal values of K that lead to the highest performance are 16, 16, and 24 for the three different cases, respectively. For the case of Redun-A+LQ-A+N-InT on DiDeMo, when K is set to 16, the achieved R@sum remains comparable to the results obtained with K=24 while with higher retrieval efficiency. Consequently, we opt to utilize K=16 for the comparison between single frame selection and the combinations of frame selection, as presented in Table 3 of our manuscript. Except for the shown results of DiDeMo, We will append all the experiments MSRVTT in the revised Appendix.\n\n> Q3: 2.  Drawing the conclusion that proper frame selections improve the retrieval efficiency without sacrificing the retrieval performance of TVR, should be more rigorous.\n\nThanks for your constructive comment. We agree that the expression of \"proper\" frame selection is not rigorous enough. We have concluded the exact optimal frame selection methods in Lines 115~122 of our manuscript, that is, the combination of Redun-A and LQ-A and the combination of Redun-A and N-InT. According to the results in Table 2 of our manuscript, the combination of Redun-A and N-InT performs slightly better than the combination of Redun-A and LQ-A, thanks to the text guidance. However, as shown in Table 3 of our manuscript, LQ-A has slightly higher efficiency than N-Int owing to the text-free frame selection. Overall, both combinations are simple but effective. We will modify the proper frame selection to the exact frame selection methods in the revised version.\n\n> Q4:  There are some writing errors in the paper as follows.\n>\n> 1. Line 240: F1 type error\n> 2. Figure 1(b): The index after frame selection should range from 1 to K. Only (I & II) meet this requirement.\n\nThanks for your careful comments. We will correct the mistakes and carefully check other possible errors in the revised vision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The Redundancy-Aware frame selection utilizes the k-medoids++ clustering algorithm to partition the representations of each frame into K groups. Choose the representation of the center frame of each group on behalf of the whole group. This eliminates redundant frames.\n2. The Low Quality-Aware frame selection appends a score network after the vision encoder to assess the quality score of each frame. Leverage the paired text as weak supervision to optimize the score network. This eliminates the low-quality frames.\n3. The six frame selection methods are verified in more detailed experiments, and the two proposed frame selection methods improve the efficiency of the TVR task without sacrificing the retrieval performance.", "reasons_to_reject": "1.The evaluation indicators are not very reasonable. R@sum is rarely seen in articles in TVR.\n2.The experiment part uses too much space on the influence of different frame selections on the retrieval performance, which has little improvement. However, it spends little space on the effect of different frame selections on the retrieval efficiency, whose improvement is very obvious.\n3.In experiments, when combining different frame selections, the paper does not show how the hyper parameter K affects the results, which is similar to the analyses in Table 4. Drawing the conclusion that proper frame selections improve the retrieval efficiency without sacrificing the retrieval performance of TVR,  should be more rigorous.", "questions_to_authors": "There are some writing errors in the paper as follows:\n1. Line 240:  F1 type error \n2. Figure 1(b): The index after frame selection should range from 1 to K. Only (I & II) meet this requirement.", "paper_topic_and_main_contributions": "The paper made an empirical study of frame selection for Text-To-Video Retrieval, and systemically classified existing frame selection methods into text-free and text-guided ones. The paper developed two frame selection methods, i.e. Redundancy-Aware frame selection and Low Quality-Aware frame selection. The paper conducted a number of experiments to verify the impact of frame selection methods on the efficiency and retrieval performance of TVR tasks. The experiments show that proper frame selections can improve retrieval efficiency without sacrificing the retrieval performance of TVR.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents two new methods for frame selection for Text-to-Video Retrieval (TVR). The paper has some merits. For example, reviewers agreed the proposed methods are effective as they show gains over baselines, and some reviewers enjoyed the clarity of the paper. However, reviewers also raised some concerns on the experimental settings and incremental novelty. Overall, AC agrees that the paper is a good source of experimental findings and methods of improvements in TVR.
Decision: Accept-Findings