T2VIndexer: A Generative Video Indexer for Efficient Text-Video Retrieval

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-01 04:38:33

Abstract: Current text-video retrieval methods mainly rely on cross-modal matching between queries and videos to calculate their similarity scores, which are then sorted to obtain retrieval results. This method considers the matching between each candidate video and the query, but it incurs a significant time cost and will increase notably with the increase of candidates. Generative models are common in natural language processing and computer vision, and have been successfully applied in document retrieval, but their application in multimodal retrieval remains unexplored. To enhance retrieval efficiency, in this paper, we introduce a model-based video indexer named T2VIndexer, which is a sequence-to-sequence generative model directly generating video identifiers and retrieving candidate videos with constant time complexity. T2VIndexer aims to reduce retrieval time while maintaining high accuracy. To achieve this goal, we propose video identifier encoding and query-identifier augmentation approaches to represent videos as short sequences while preserving their semantic information. Our method consistently enhances the retrieval efficiency of current state-of-the-art models on four standard datasets. It enables baselines with only 30%-50% of the original retrieval time to achieve better retrieval performance on MSR-VTT (+1.0%), MSVD (+1.8%), ActivityNet (+1.5%), and DiDeMo (+0.2%). The code is available at https://anonymous.4open.science/r/T2VIndexer-40BE.

Main paper: 
 introduction: 
 Given a query text description, text-video retrieval [26] aims to retrieve videos that are semantically relevant to the query. Text-video retrieval is flexible to express the user's intent and brings emerging attention for web search with the dramatic increasing of videos uploaded online every day. For a standard web search engine [20] , video retrieval and ranking are two core stages. The retrieval stage first retrieves limited number of candidate videos from massive online videos, and the following ranking stage predicts accurate ranking scores between per query and the candidate videos. Since videos have much richer and more diverse visual content compared with the query text, precise video ranking is costly for fine-grained text-video matching. Therefore, the efficiency and recall performance of video retrieval stage is essential to the fast and accurate text-video search.Existing text-video retrieval methods can be divided into two categories, namely one-stream and two-stream approaches. One-stream approaches [32] [12] [16] adopts deep models for feature-level interactions between each text-video pair to predict its similarity score, which require online feature extraction and fail to be applied for the time-sensitive retrieval stage. Thus, the efficient two-stream approaches [9] [17] [23] are widely applied. As shown in Figure 1 (a), they encode each video and text independently into dense embeddings and then adopt simple matching functions to measure their similarity. Since there are no text-video interactions in the encoding stage, two-stream approaches allows offline data embedding extraction and alleviating online computation. Some recent works begin to focus on the issues of reducing the high computational overload of dense video embedding by sparsely sampling a few clips [12] (see Figure 1 (b) ). However, all the existing solutions require to measure the query-video similarities and then rank videos for the entire video set (i.e., one-to-all retrieval framework). Thus, their online retrieval time grows linearly with the increase of retrieved videos, which limits their scalability on large-scale scenarios.To address the above issue, we explore to fundamentally change the traditional one-to-all embedding retrieval framework by a generative deep model that directly generates video identifiers and retrieves video candidates with constant time complexity. As illustrated in Figure 1 (c), our target of this work is not to propose a new model on text-video retrieval. We mainly investigate how to design a model-based indexer that effectively retrieves queryrelevant video candidates, which shortens the overall retrieval time while maintaining the retrieval accuracy of state-of-the-art ranking models. To this end, we propose a sequence-to-sequence generative network that supports Text query to Video candidate Index, named as T2VIndexer. The model is based on the encoder-decoder that feeds the query into the encoder and generates the identifier of the video candidate through the decoder. It is trained by queryidentifier pairs that supports controllable video recall at different semantic grained. During inference, the top 𝐾 videos are directly retrieved by beam search and identifier constrain.To guarantee the effectiveness of T2VIndexer, we have proposed several methods to tackle the key challenges. First, to get the semantic representations and coarse-to-fine identifiers of videos for controllable video recall, we utilize CLIP [19] to embed each video, and then cluster and encode the semantic embeddings in a hierarchical mode. Second, we propose to leverage the pre-trained multi-modal large language model [28] to generate new queries with diverse views of the video content, which augments the queryidentifier pairs during training for stronger generalization ability during inference. Third, we propose to train a generative network based on T5 [4] architecture to enable the deep interactions between the query and video identifier, which enhances the cross-modal correlation learning for precise identifier prediction.The main contributions are summarized as follows: (1) We propose a novel sequence-to-sequence generative framework as a video indexer for efficient video candidate retrieval. Our approach directly predicts the candidate videos with constant time complexity that significantly outperforms existing one-to-all embedding retrieval solutions with linear time complexity. It demonstrates the effectiveness of generative video index and sheds new light on the research on generation-based text-video retrieval mechanism. (2) We novelly propose the video identifier encoding and query-identifier augmentation approaches for learning T2VIndexer with strong generalization ability. T2VIndexer is model-agnostic and universal to cooperate with various independent-embedding approaches, which remarkably improves their retrieval efficiency with even better retrieval performance. (3) Our T2VIndexer approach is consistently effective for diverse text-video retrieval tasks. By cooperated with T2VIndex, the state-of-the-art models cost merely 30% to 50% of original retrieval time across four typical tasks. The time cost will be further reduced with the increase of retrieved videos, which impacts a broader range of text-video applications. 
related work: 
 Text-video retrieval. Existing methods can be divided into two categories, called one-stream and two-stream approaches. One-stream approaches are characterized by token-level interactions based on cross-modal attention mechanisms, which are used for fine-grained video-text matching [32] [12] [16] [24] . Two-stream approaches aim to coordinate videos and text in a unified semantic space and perform direct comparisons through distance metrics [9] [17] [23] . With the success of pre-trained image-text alignment model such as CLIP [19] , this method not only surpasses interactive embedding methods in efficiency but also has significant advantages in accuracy. In addition, efficiency enhancements have focused on video sampling strategies. Some methods choose to sample the frame sparsely [12] [17] . Besides, redundancy persists within the vision tokens of each frame, diminishing the prowess of CLIP-style retrieval. CenterCLIP [31] addressed this by refining patch subdivision and selection via clustering. These innovations enhance preprocessing efficiency but do not alleviate the inherent online retrieval latency due to similarity computations and ranking cost. Generative Model in Retrieval. In unimodal retrieval tasks, the same efficiency issues are faced. With the success of generative models in various visual and language tasks, they have demonstrated powerful capabilities. In document retrieval, models like DSI [21] demonstrate the ability to generate identifiers using Transformer architectures, while approaches like SEAL [3] innovate by substituting string identifiers with document n-grams. The NCI [22] further refines this approach by integrating positional information into the decoding process. Image-to-image retrieval tasks have transformed these methods into visual modality. For example, IRGen [30] tokenizes images to identifiers and uses a generative model to map queries to these identifiers for direct localization, thereby improving retrieval efficiency. These methods have demonstrated powerful capabilities in unimodal retrieval. However, videos contain rich target and event. There is an obvious many-to-many problem, which means one video corresponds to multiple different descriptions from different perspectives, and a summary description corresponds to multiple different videos. 
methodology: 
 The text-video retrieval involves a text query 𝑡 and a gallery of videos 𝑉 . The objective is to retrieve videos {𝑣 𝑗 } ∈ 𝑉 that are semantically relevant to the query. As shown in Figure 2 , our goal is to directly retrieve targeted videos by generating the video identifiers based on natural language queries. To this end, we design a sequence-to-sequence generative model that takes query 𝑡 as input and outputs the video identifier for video index. We first propose a semantic-aware tree structure to encode video identifiers, called SemID, which encodes the multi-grained semantics of videos by a sequence for controllable recall while maintaining the sequence length as short as possible for fast encoding. To augment the semantic expression of queries for more generalized model learning, we propose to utilize a Multi-modal Large Language Model (MLLM) [28] to generate a set of multi-view queries for each video, thereby enriching the contextual semantics encapsulated by the SemIDs for diverse queries. The model architecture, training and inference strategies are introduced in the end.The purpose of our work is to locate videos by taking query 𝑡 as input and outputting the most relevant video identifier. Therefore, finding a suitable identifier as the basis for video location is crucial. The identifier needs to have semantic prior information so that it For the inference stage, the new video is first inserted into the semantic tree and assigned a SemID, and the baseline model provides the precise retrieval results.can reflect the content of the video, and similar semantic videos are also similar in the identifier. Moreover, the sequence length should be short enough to reduce the difficulty and complexity of model generation. Based on this consideration, we first extract the representation of each video, and construct a Video Semantic Tree (Vi-SemTree) based on the semantics of the video, and provide a SemID as an identifier for each video based on the tree structure. This approach ensures the semantic consistency of locating videos and guarantees the recall rate of the generation phase. Video Semantic Representation. To construct Vi-SemTree and obtain a sequence representation of SemID, we first extract the representation of the video. Compared with pixel-level information, Vi-SemTree requires the integration of semantic-level information, which is more seamlessly integrated with the structure of natural language. To meet this requirement, we chose the image encoder of CLIP [19] , which is famous for its multimodal pretraining ability, as the basic tool for our video encoding. Given a video's sequence of frames 𝑣 𝑓 = {𝑓 1 , 𝑓 2 , ..., 𝑓 𝑁 }, where 𝑁 is the number of frames. We derive the corresponding frame representation as f = { f 1 , f 2 , ..., f 𝑁 }, culminating in the overall video representation F , obtained through mean pooling of the individual frame representations. Conforming to the methodologies laid out by ViT [7] and CLIP [19] , the output gleaned from the [class] token is utilized to represent each frame. Hierarchical Vi-SemTree Building. We use a tree structure to encode videos, which helps preserve semantic information and ensures that similar semantic videos are also similar in the identifier.Moreover, by controlling the depth 𝑑 of the tree, we can achieve different levels of granularity and control the sequence length.Following NCI [22] , we use hierarchical 𝑘-means method for video feature F , as shown in the training stage (a) of Figure 2 . First, we use the 𝑘-means algorithm to divide the training set videos into 𝑘 clusters based on their representation similarity. Each cluster is a tree node and contains a group of semantically similar videos, which serves as a layer of the tree structure. For each cluster, if the number of videos is greater than 𝑐, we use the 𝑘-means algorithm to further divide the cluster, generating the next layer of the tree, which is more granular at the semantic level. We repeat this process until we obtain a tree structure 𝑇 with the root 𝑟 , where semantically similar videos are located in the same path.Vi-SemTree for SemID Encoding. To directly retrieve a video set, we introduce SemID, a unique identifier derived from the Vi-SemTree, which serves as a pivotal reference for our generative model. We define the SemID as the path 𝐿 path = {𝑙 0 , 𝑙 1 , ..., 𝑙 𝑑 }, traversing from the root node 𝑟 down to a leaf node with 𝑑 marking tree depth. Specifically, the root node is represented as 0, serving as the start symbol. Each edge branching out from each node is numbered starting from 0, with values ranging from 0 to 𝑘. From top to bottom, this sequence serves as the SemID 𝐿 𝑝𝑎𝑡ℎ .Due to the characteristics of the video modality, it has rich semantic information, and different descriptions can be generated from different perspectives based on different events. Therefore, a video can To comprehensively capture the different elements such as targets, events, and themes that appear in the video and generate natural language descriptions that correspond to the relevant information as additional queries, Multimodal Large Language Model (MLLM) is an important tool. Unlike the single and coarse descriptions provided by dense video caption models, MLLM can generate multi-angle descriptions according to different prompts, capturing various information that appears in the video. Based on MLLM, we generated 50 different queries from different perspectives for each video, achieving coverage of semantic information.Based on the query 𝑡 provided in the dataset and the extended query t, the query set 𝑄 𝑖 = {𝑡 𝑖 , t𝑖 } is associated with each video 𝑖 in the training set 𝐷. We can represent the training data pairs for the generation model as{(𝑞, 𝑆𝑒𝑚𝐼𝐷 𝑖 ), 𝑞 ∈ 𝑄 𝑖 , 𝑖 ∈ 𝐷 }, where 𝑆𝑒𝑚𝐼𝐷 𝑖 is the 𝐿 𝑝𝑎𝑡ℎ = {𝑙 0 , 𝑙 1 , ..., 𝑙 𝑑 } of the node where video 𝑖 is located. However, as the tree structure deepens, the semantic information of videos in different nodes becomes increasingly similar. This may lead to a decrease in the recall effect of relatively coarse-grained Queries. To merge more videos with similar semantics and simplify the generation process, we choose the truncated version of the path as SemID, represented as 𝐿 𝑡 𝑝𝑎𝑡ℎ = {𝑙 0 , 𝑙 1 , ..., 𝑙 𝑑 -𝑡 }, to ensure efficient and semantically consistent grouping.To achieve direct positioning of the target video through SemID, we chose to use a sequence-to-sequence generative model to directly generate the corresponding SemID based on the input query, as shown in Figure 3 . First, the input query is added with position embedding and input into the transformer encoder to obtain the representation 𝑓 𝑡 . The probability of generating the SemID sequence as follows to construct,EQUATIONwhich means the next token is generated according to the sequence of previously generated tokens. This probability problem can be solved by the traditional transformer encoder-decoder structure.The input of the encoder and the output of the decoder can be regarded as two different semantic spaces, corresponding to the natural language space and the video semantic tree space. However, unlike standard decoding tasks, the same token appearing at different positions in the video semantic tree space has different meanings because they are in different layers of the Vi-SemTree. For example, as shown in the framework Figure 2 , for SemID '0 0 -0 1 -0 2 ', the token '0 1 ' in the first layer represents the semantic of the category "sports", while '0 2 ' in the second layer only represents "basketball". In addition, even if they are in the same layer, the same token '0 2 ' in SemID '0 0 -0 1 -0 2 ' and '0 0 -2 1 -0 2 ' expresses different concept due to their different prefixes. In order to identify different representations at different positions during decoding, we were inspired by NCI [22] in document retrieval and used the Prefix-Aware Weight-Adaptor (PAWA) decoder, as shown in Figure 3 .Unlike the standard transformer decoder, the PAWA decoder uses different parameters when generating tokens at different positions, and the parameters between different steps are not shared. The specific settings are as follows. First, the encoder encodes the query to obtain the encoding 𝑥, and decoder output 𝐸 as follows,EQUATIONwhere 𝜃 𝑖 represents the parameters of each decoding step, and the parameters of each step are different, distinguishing the semantic of different position tokens. In addition, to further enhance the prefix information as the basis for generation, the PAWA decoder further modifies the linear classification layers based on the prefix sequence. Specifically, instead of using the same projection weight 𝑊 in the linear classification layer, the PAWA decoder uses an additional decoder to generate different weights for each position,EQUATIONEQUATION) where 𝜃 ′ 𝑖 represents the parameters of the decoder that generates the weights, and 𝑊 𝑖 is the generated weight matrix for the corresponding classifier. Finally, the i-th token is represented as 𝑙 𝑖 , calculated by softmax(𝐸 𝑖 𝑊 𝑖 ).Training Loss. The loss function for a set of training examples D = {(𝑞, 𝐿 𝑝𝑎𝑡ℎ )}, consisting of queries (training queries and expansion queries) and video SemID, can be expressed as follows,EQUATIONwhere log(𝐿 𝑝𝑎𝑡ℎ |𝑞, 𝜃 ) represents the probability of generating SemID based on 𝑞, and is a standard sequence-to-sequence cross-entropy loss with teacher forcing.Coarse-to-fine Inference. During the training stage, since the videos in the test set are not visible, this part of the videos is not assigned a SemID. In order to cover test set, a new video needs to be assigned a SemID as its identifier as shown in Figure 2 Inference Stage. The new video will get the representation in the same way as the training set using CLIP encoder, and then the video similarity with the leaf nodes in the tree will be calculated. The video will be inserted into the leaf node with the highest similarity and inherit the SemID of the leaf node. For the input query 𝑞, the probability 𝑝 (𝐿 𝑡 𝑝𝑎𝑡ℎ |𝑞, 𝜃 ) is calculated by the generative model trained, and the target SemID is obtained to achieve direct positioning of a group of target videos. In the decoding process, considering the semantic summarization ability of natural language query, which makes videos that meet the description may not be in the same path, we use the Beam Search algorithm for decoding. This enables us to retrieve the top k SemIDs that meet the description, to adapt to this one-to-many issue. Based on the generated SemIDs, we obtain a small candidate set 𝑉 𝑐𝑎𝑛𝑑 = {𝑉 𝑆𝑒𝑚𝐼 𝐷 1 , 𝑉 𝑆𝑒𝑚𝐼 𝐷 2 , ..., 𝑉 𝑆𝑒𝑚𝐼 𝐷 𝑘 } containing the target videos, where 𝑆𝑒𝑚𝐼𝐷 𝑖 represents a group of videos corresponding to the i-th top SemID generated.Due to the limitations of the first stage, it is currently not feasible to achieve precise retrieval based on the query. In order to meet the requirements of the task and achieve precise retrieval, fine-tuning of 𝑉 𝑐𝑎𝑛𝑑 is required. For this purpose, we introduce a two-stage retrieval architecture and use the existing text-video retrieval model for precise retrieval in 𝑉 𝑐𝑎𝑛𝑑 . T2VIndexer can be integrated with the existing retrieval model without adjusting the training parameters. For a text-video retrieval model 𝑀 𝑏𝑎𝑠𝑒 , the similarity 𝑠 between 𝑞 and each video 𝑣 in 𝑉 𝑐𝑎𝑛𝑑 is calculated, and the final retrieval result is obtained by ranking 𝑠. Overall, the entire retrieval process can be divided into two main stages: Pre-select based on generative models and Precise Recall based on contrastive learning models. 
experiments: 
 Datasets and Evaluation Metrics. We validate our model on four dataset: MSR-VTT, MSVD, DiDeMo, and ActivityNet Caption. MSR-VTT [26] encompasses 10,000 videos, paired with 200,000 captions. We employ the Training-9k variant, following the data splits proposed by [10] . MSVD [5] contains 1,970 videos, and a wealth of approximately 40 associated English sentences per video. Train, validation and test splits contain 1,200, 100, and 670 videos, respectively. DiDeMo [1] contains 10,000 videos annotated with 40,000 sentences. We evaluate video-paragraph retrieval following [15] , [13] and [2] , where all sentence descriptions for a video are concatenated into a single query. ActivityNet [11] consists of 20,000 YouTube video. We follow the setting from [29] to concatenate all the descriptions of a video to form a paragraph and evaluate the model with video-paragraph retrieval.We adopt standard retrieval metrics, namely recall at rank K (R@K), calculates the percentage of instances where the correct result is successfully retrieved within top K. Implementation Details. We utilized the image encoder from the pre-trained CLIP (Vit B/32) model. For constructing the Vi-SemTree, we opted for 𝑘-means algorithm [18] , setting both k and c to 30. SemID truncation length 𝑡 set to 2 and select top 11 beam search results from generative model. The encoder parameters were initialized using the T5 pre-trained model [4] , while the decoder parameters were randomly initialized. During training, the learning rate was set to 2 × 10 -4 for the encoder and 1 × 10 -4 for the decoder. We utilized 8 NVIDIA V100-32GB GPUs, with a batch size of 16 per GPU and a dropout ratio of 0.1.The purpose of T2VIndexer is to improve retrieval efficiency while maintaining accuracy. In Table 1 , we analyzed the accuracy and efficiency of T2VIndexer under different candidate sets with a single RTX3090 GPU and 8255C CPU. In the efficiency analysis, we imposed some restrictions to simulate real application scenarios. First, the time cost from receiving the query was calculated, without considering the offline phase, such as the construction of Vi-SemTree and the allocation of SemID. Second, each query in the Test set was retrieved one by one to return the target video, instead of obtaining all Queries and returning the overall results at once. From the Table 1 , it can be seen that T2VIndexer significantly reduces inference time while maintaining the baseline effect. For example, compared with the traditional method under 1000 candidate videos, T2VIndexer reduces the time cost by 50%. The efficiency improvement increases gradually with the size of the candidate set. Under 10,000 candidate videos, the time compression reaches 30%. In addition, we also analyzed the performance of three other datasets, as shown in Table 2 .Further analysis shows the efficiency improvement is due to the structure of Vi-SemTree. When the candidate set expands 10,000, the traditional method needs to process an additional 9000 data, perform a large number of similarity calculations and sorting. For T2VIndexer, the added 9000 videos will be distributed to each leaf node, and cost of generative model generating SemID remains unchanged, which ultimately improves efficiency. For example, if Vi-SemTree has 100 leaf nodes, for 10,000 candidate videos, there are an average of 100 target videos per leaf node, which significantly reduces the retrieval pressure of the baseline. This means that the T2VIndexer does not suffer from the same scalability issues as traditional methods, and allows for a more distribution of data.To further investigate the effectiveness of T2VIndexer in real retrieval scenarios, we evaluate on larger scale data. Due to the limited size of the existing dataset test sets, which mostly consist of 1000 candidate videos, we decided to redivide the TGIF dataset [14] into 50,000 training data and 50,000 testing data in a 5:5 ratio. Both the baseline and the T2VIndexer generative model will be trained solely on the 50,000 training data. Table 3 displays our test results, with each block representing a set of test results. It is evident from the results that in large-scale retrieval scenarios, T2VIndexer demonstrates more significant improvements in both efficiency and accuracy compared to smaller-scale data.The improvements in both efficiency and accuracy primarily stem from the pre-select mechanism employed by T2VIndexer. T2VIndexer operates with constant-time complexity to generate an ID sequence, corresponding to a subset of videos on a small scale. And the base model conducts similarity calculations and sorting solely within this small-scale video subset, T2VIndexer significantly reduces computational overhead by avoiding individual matching, thus enhancing retrieval efficiency. Furthermore, the pre-select mechanism aids in pinpointing the target video, eliminating a majority of irrelevant videos in advance, thereby reducing noise-induced interference on the baseline and effectively acting as a filter to enhance retrieval accuracy.Table 1 : Evaluation of the inference costs on MSR-VTT dataset. We report the R@1 metric for the text-to-video task. Blue means stage 1 generative time cost, and red means stage 2 time cost. The improvement of inference time refers to the compression ratio of T2VIndexer, which is obtained by dividing the inference time with T2VIndexer by the inference time of the baseline. Table 4 is segmented into three blocks, each representing a distinct category of methods: Interactive embedding methods, Independent embedding method, and ours T2VIndexer, which is implemented based on different baselines. Observing the first two blocks, two-stream approaches usually exhibit excellent performance, which is mainly attributed to the powerful pretraining ability of CLIP [19] . In the third block, we constructed T2VIndexer based on CLIP4CLIP [17] , mPLUG [25] , and CLIP-VIP [27] . The original performance of these three models improved in turn, providing a basis for testing the effectiveness of T2VIndexer on different baselines with different degrees of effectiveness. As shown in Table 4 , T2VIndexer achieved significant improvement with the relatively low-performance CLIP4CLIP model, increasing R@1 by 3.3% on the MSR-VTT dataset. However, as the baseline model improved, the accuracy gain of T2VIndexer decreased, and the maximum improvement of the CLIP-VIP model was 1.0%. This can be attributed to the candidates provided by T2VIndexer to baseline models. T2VIndexer provides the same set of candidates to the baseline models for a given query, effectively eliminating a considerable number of irrelevant videos. For lower-performing models, this removal of irrelevant videos is useful, significantly reducing the input noise and enhancing accuracy. However, higherperforming models are less sensitive to noise and can distinguish relevant content more effectively, resulting in relatively lower improvements when assisted by T2VIndexer.To further investigate the impact of different components on the model's performance, we report the ablation results on the MSR-VTT dataset in Table 5 . (1) Without query expansion (w/o query expansion) This part has the most significant impact on the model's results. During the SemID generation process by T2VIndexer, the original video is not seen. If the semantics of the original video are not injected into SemID through queries during the training phase, the model will fail to establish a relationship between the text and SemID and will not be able to correctly generate SemID for queries not seen during training. (2) Without Multi-view query expansion (w/o Multi-view query expansion) This indicates not using an MLLM (Multilingual Language Models) to generate multi-view Utilizing Multi-Modal Large Language Models (MLLMs) for query expansion effectively enhances the model's generalization capabilities. Various MLLMs show little difference in the quality of query expansion, so the model's effectiveness does not rely on a specific MLLM. Apart from mPLUG-owl tested in the paper, we have also conducted supplementary tests with Minigpt-4 and LLaVA. As evident from the results in Table 6 on the MSR-VTT dataset, different MLLMs have a minor impact on retrieval accuracy. Different methods of video feature extraction will affect the structure of the Vi-Sem tree, thereby influencing the overall model's effectiveness. In Table7, we tested three different video feature extraction methods: S3D based on pixel features, CLIP image encoder based on semantic features of frames, and VideoMAE pre-trained for video recognition. From the results comparison, it can be observed that the method based on pixel features performs the worst, even falling below the baseline model CLIP-VIP. This is mainly because in this case, the tree structure reflects pixel information rather than the more relevant semantic information associated with natural language, resulting in poorer performance in the video preselect stage, failing to return clusters containing the correct videos to the precise recall stage.We further demonstrated the ability of T2VIndexer to locate target videos through visualization. Three examples are shown in Figure 4 , where the SemID generated by T2VIndexer has a high semantic similarity with the target SemID, even in wrong mapping cases. Specifically, for the query "video game clip showing here different characters", the SemID generated by T2VIndexer, 0-9-21, has a stronger matching relationship with the query than ground-truth video. This indicates that the model has effectively learned the mapping between natural language space and SemID space, achieving retrieve target videos directly.  
limitation and future work: 
 Although T2VIndexer has achieved certain results in efficient textvideo retrieval, there still exists performance limitations. T2VIndexer consists of two stages: pre-select and precise retrieval, which involve different models. This approach is similar to the ensemble architecture. To further analyze the effectiveness of T2VIndexer, we built a two-stage ensemble architecture based on the existing sota models mPLUG and CLIP-VIP for comparative analysis, as shown in Table 8 . It can be seen that for existing ranking-based models, the ensemble form can improve the accuracy of retrieval at the expense of efficiency, which is superior to the generative method of T2VIndexer in terms of effect. However, existing models need to process all candidate sets when performing Pre-select, while T2VIndexer can directly locate the candidate set, which has a significant advantage in efficiency. In addition, the existing pipline cannot efficiently retrieve new videos that have not been seen when constructing Vi-SemTree. For each new video, it is necessary to insert it into a leaf node to obtain the corresponding SemID. This operation involves a large number of similarity calculations and sorting, which has a significant time cost. For example, inserting videos in the MSR-VTT test set into the tree takes an average of 200 ms per video to assign SemID, which reduce the flexibility.To further improve the reliability, our future work will focus on improving the accuracy of the generative stage to achieve more precise localization. At the same time, we aim to reduce the time cost of new data and improve flexibility. 
conclusion: 
 In this paper, we propose T2VIndexer, a model-based video indexer that generates video identifiers directly and retrieves candidate videos with constant time complexity, in order to shorten the overall retrieval time while maintaining the retrieval accuracy of the base model. We use hierarchical clustering to organize videos into a tree structure called Vi-SemTree, which contains multiple layers corresponding to relationships from coarse to fine. We specifically trained a generative model for Vi-SemTree paths, correctly mapping natural language space and video semantic space. T2VIndexer is model-independent and can be seamlessly integrated with existing methods. However, the retrieval effect of the generative model is currently limited, and our future work will focus on improving the accuracy of the generative retrieval to achieve precise retrieval, and further improve the flexibility of the model when receiving new videos and reduce preprocessing time.  
 
 Appendix:  


Decision: This paper introduces a model-based video indexer named T2VIndexer, aiming to reduce retrieval time while maintaining high accuracy. It is a sequence-to-sequence generative model which can directly generate video identifiers and retrieve candidate videos with constant time complexity. It can enhance the retrieval efficiency of current state-of-the-art models on several datasets.

(+) On the positive side, the reviewers found the method to be interesting and effective and appreciate the clear motivation and good presentation . 

(-) On the negative side, there are still some concerns on the pre-processing costs, insufficient discussion, generalization ability, and clarity of the writing.

Several technical questions and suggestions were raised by the reviewers. The authors have taken these into consideration. Overall, all reviewers agreed to accept the paper after the rebuttal. Therefore, the AC recommends accepting the paper.

Reviews: review: paper_summary: paper_summary: The paper designs a Video Semantic tree to identify a video named Vi-SemTree to improve the efficiency of text-video retrieval independent of the number of candidate videos. It proposes a novel video indexer named T2VIndexer. For constructing Vi-SemTree and obtaining SemID sequence, the work extracts frame representation of the video by CLIP encoding, then integrates the representations into semantic-level information, and forms tree structures for identifying the videos and semID Encoding. The work introduces a sequence-to-sequence generative model to embed the input query into the transform encoder and output SemID sequences as tokens generated to generate the corresponding SemID based on the input query. Furthermore, the paper discusses the strategy of model training and inference. Experiments show that the retrieval time is sped, and the R@1 metric is improved. The idea is interesting and practical. Of course, the accuracy and robustness deserve further discussion.
strength_weakness: strength_weakness: (1) The paper is well-written and easy to follow;
(2) The paper introduces details around T2VIndexer construction, generation, inference, assignment, pre-select, and matching;
(3) The paper proposes a Vi-SemTree structure for Video Identification based on Video semantic representation and encoding. Video indexing does speed up text video retrieval efficiency in principle.
(4) The paper exploits Multi-modal Large Language Model(MLLM) tools to generate multi-view descriptions from rich semantic information in Video for multiple different.
(5) The paper introduces a Prefix-Aware Weight-Adaptor (PAWA) decoder to generate tokens at different positions for expressing different concepts due to their different prefixes and identify different representations at different positions.
limitations: limitations: (1)If the order of Vi-SemTree structure impacts indexing efficiency and text-video retrieval accuracy, please provide a detailed analysis and presentation.
(2)The paper mentions that both hierarchical Vi-SemTree construction and Vi-SemTree leaf node insertion depend on video semantic similarity. Please clarify the definition of semantic similarity metrics and their impact on accuracy and ranking results。
(3)In the context of the video semantic tree construction mechanism, if there are multiple videos in the leaf node corresponding to the same SemID, how should they be processed and matched? Could you provide some insights on this issue?
paper_summary: paper_summary: This work studies text-based video retrieval tasks and particularly focuses on boosting cross-modal retrieval efficiency. Specifically, the authors propose a novel model-agnostic module based on a multimodal representation foundation model (CLIP), multimodal LLM, and hierarchical k-means algorithm to improve the inference performance and efficiency of VTR models. By building a tree (named Video Semantic Tree in the manuscript) with CLIP and clustering algorithm, the video gallery is turned into a set of small semantic-related groups and each group can be encoded as a specific index, thus effectively mitigating the burdens in dual stream VTR paradigms. The authors evaluate the proposed T2VIndexer at four benchmark datasets combined with different baseline models. Experimental results demonstrate the effectiveness of both efficiency and performance.
strength_weakness: strength_weakness: Strength:
1. The motivation is interesting and practical, which can be applied to real application scenarios. 
2. The writing is generally clean and most ideas have been clearly presented. The figures are also informative. 
3. Extensive experiments demonstrate the effectiveness of the proposed work.
limitations: limitations: Weakness:
1. The authors may want to present their code but it seems that the repo had been expired. No code was found there. Please consider providing the correct URL. 

2. The authors are encouraged to add more discussions on generative retrieval models or efficiency-sensitive video content retrieval models, such as [1-4], that share similar ideas with this work, including building trees or turning online computation into an offline phase. 

[1] Generative Retrieval with Semantic Tree-Structured Item Identifiers via Contrastive Learning. ArXiv'23

[2] Learning Commonsense-aware Moment-Text Alignment for Fast Video Temporal Grounding. ACM TOMM'24

[3] Faster Video Moment Retrieval with Point-Level Supervision. ACM MM'23

[4] CenterCLIP: Token Clustering for Efficient Text-Video Retrieval. SIGIR'22
paper_summary: paper_summary: This paper introduces a novel approach named T2VIndexer which aims to improve the efficiency of existing text-to-video retrieval methods and maintain their performance simultaneously. It proposes a encoder-decoder framework which build a video semantic tree first and allocate the semantically similar videos to the same leaf. Then the decoder will predict the path to the leaf through the given query and obtain candidate videos and the number of it is far less than the origin candidates. To this end, the final retrieval process on the remain videos is remarkable faster than before. Besides, extensive experiments on four benchmarks shows the effectiveness on retrieval efficiency enhancement and performance retention of the method.
strength_weakness: strength_weakness: 1.The motivation of this paper is well-founded, the idea of build video semantic tree to cluster similar videos and training a network to filter candidate videos by path searching is novel and convincing .
2.The experiments are sufficient and have demonstrated the effectiveness of the proposed method. It also greatly boosting the retrieval efficiency and outperforms previous works.
3. Moreover, the paper is well-written and presents its concepts clearly and comprehensibly which makes it easy to follow.
limitations: limitations: 1. This paper tends to employ parameter-efficient learning strategies for text-video retrieval, however, the parameter-efficient learning methods in recent years are not mentioned in the related work.
2. Constructing video semantic tree by k-means is poor in interpretability, how about using concept-based methods?
3. Since the Vi-Sem Tree is built based on an individual dataset, I think generalization ability of it is weak.
4. In Sec 4.4, how to construct model without query expansion and  Without Vi-SemTree and SemID is not mentioned.
paper_summary: paper_summary: In this paper, the authors highlight a problem inherent to current text-video retrieval solutions, that is the cost of retrieval and ranking. Current solutions (be it single-stream or two-stream) follow a one-against-all framework, meaning they encode the query and compare it to the encoding of each video in the gallery, and then rank according to a similarity metric. The authors propose a pipeline composed of three main steps. First, they extract CLIP-based features of the videos in the training set, and create a hierarchical representation using k-means. The resulting structure is a tree, where leaf nodes represent specific topics (e.g. basketball or volleyball) and parent nodes represent higher level topics (e.g. sports), and so on. The path to the leaf (called SemID) is used as an encoding of the group of videos with that topic. Second, they expand the training set with artificially generated captions obtained through a multimodal LLM. Finally, a T5 is used to generate the path along the tree given the input query. At inference time, the process starts with assigning the SemID to the video based on its CLIP-based representation. Then, the query given to the trained generative model which predicts top-k SemIDs through beam search. The videos in that leaf are then collected and sorted based on existing text-to-video retrieval models. The experimental setup involves four well-known datasets and three recent text-video retrieval methods. The results under different amount of videos in the test gallery confirm that the proposed approach leads to great time reduction (with favorable scaling) while maintaining high performance. State-of-the-art comparison also shows their approach leads to improvements in the three selected methods even in the full-test scenario. Ablation and qualitative studies also show robustness in their design choices. The authors then conclude highlighting a limitation of their work.
strength_weakness: strength_weakness: Strengths:

- The problem is relatively under-explored, since recent works on video retrieval mostly focus on improving performance, without analyzing the effect on retrieval efficiency. Moreover, the proposed approach is very interesting and aligned with recent trends in content generation, although used in a somehow unexpected way.

- The text is clear enough and easy to follow, although there are some parts which are not clearly explained (more in the limitation/weaknesses). The idea is very interesting and I believe can be applied to other multimedia domains, meaning that it can have impact on the community as a whole.

- The results are not only interesting but also extensive, in-depth, and well-presented.
limitations: limitations: Weaknesses:

- The main limitation I see is that of additional "pre-processing costs". As the authors point out in the limitations, if a new video is tested, their approach requires the addition of the video to the tree structure, which is quite costly. Whereas in existing approaches there is no such cost and more flexbility. Moreover, while in existing approaches the video encoding is a single step (extract features), here there are potentially three steps (extract CLIP features, add to the tree, extract features with the model used for retrieval/ranking). While they still achieve their goal (that is, improving online retrieval efficiency in a predefined gallery), these limitations are important in real scenarios where new videos may appear at any time.

- Few things which are not clear to me:

* At lines 387-390, the authors mention using the truncated version of the path as SemID. How is it truncated? Is it truncated at a fixed length, as in max depth of the tree? Or is it truncated with respect to some other information? Currently, this aspect is not clear. Moreover, when defining the truncated path L^t_path, t is used although it is not the same as query t, creating some confusion.

* The sentence at lines 470-473, "In the decoding process, considering the semantic summarization ability of natural language query, which makes videos that meet the description may not be in the same path, we use the Beam Search algorithm for decoding", makes little sense to me.