TVPR: Text-to-Video Person Retrieval and a New Benchmark

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-03 17:51:27

Abstract: Most existing methods for text-based person retrieval focus on text-to-image person retrieval. Nevertheless, due to the lack of dynamic information provided by isolated frames, the performance is hampered when the person is obscured or variable motion details are missed in isolated frames. To overcome this, we propose a novel Text-to-Video Person Retrieval (TVPR) task. Since there is no dataset or benchmark that describes person videos with natural language, we construct a large-scale cross-modal person video dataset containing detailed natural language annotations, such as person’s appearance, actions and interactions with environment, etc., termed as Text-to-Video Person Re-identification (TVPReid) dataset.  In this paper, we introduce a Multielement Feature Guided Fragments Learning (MFGF) strategy, which leverages the cross-modal text-video representations to provide strong text-visual and text-motion matching information to tackle uncertain occlusion conflicting and variable motion details. Specifically, we establish two potential cross-modal spaces for text and video feature collaborative learning to progressively reduce the semantic difference between text and video. To evaluate the effectiveness of the proposed MFGF, extensive experiments have been conducted on TVPReid dataset. To the best of our knowledge, MFGF is the first successful attempt to use video for text-based person retrieval task and has achieved state-of-the-art performance on TVPReid dataset. The TVPReid dataset will be publicly available to benefit future research.

Main paper: 
 introduction: 
 With increasing concerns about public security, text-based person retrieval has drawn great attention from the multimedia community. It aims to retrieve images of a specific person from a database, which leverages natural language queries to describe the visual appearance of the target person. Previous works [19, 21, 22, 24] focus on analyzing static pedestrians' appearance by isolated frame. However, these isolated frames in text-to-image person retrieval cannot provide dynamic information, resulting in inferior performance, especially in ambiguous situations such as obscured persons or missing motion details. For example, as shown in Figure 1 , the text provides a precise description of the person's appearance, while the person in Figure 1(b ) is heavily obscured by other people who hold an umbrella, leading to failed matching. Moreover, as illustrated in Figure 2 , the text contains the details of the pedestrian motion such as the movements with change-over-time and interaction with surroundings, but the image-based methods fail to fully explore available dynamic contextual clues, which neglects crucial internal correlations among successive frames, resulting in low capability of retrieving.To address the mentioned problems, it is a remarkable solution that uses videos instead of isolated frames to offer these contextual clues. Specifically, compared to isolated frames, video provides a series of continuous frames that contain rich spatio-temporal information, revealing pedestrians' appearance changes as well as their critical motion information. As shown in Figure 1 , it can be noted that although there is a person momentarily obscured by objects, it can still obtain the complete appearance features using contextual information, due to uniform clothing style. Furthermore, the video (Figure 2 ) also shows the pedestrian's sequential motion even in interactions with the environment via continuous frames. Thus, aggregated spatio-temporal information provided by videos can excavate discriminative and dynamic information effectively, which auxiliary calibrates missing or misaligned body parts and naturally break through the issues of temporary occlusion in isolated frames. To break the limitations that appeared in the current dataset, we introduce the Text-to-Video Person Re-identification (TVPReid) dataset. This novel dataset is comprised of 6559 videos, with a total size of 19.2G, which is collected from three established videobased person re-identification datasets: PRID-2011 [10] , iLIDS-VID [20] , and DukeMTMC-VideoReID [23] . Our dataset covers different scenes, views, and camera specifications, which increases the diversity of the video content. Mainly, each video in the dataset corresponds to two distinct text descriptions, eventually a total of 13118 video pedestrian description sentences are created. This dataset offers an essential benchmark for the advancement of textto-video person retrieval. Figure 6 shows the thumbnail, which collects a large number of high-frequency words and video samples appearing in the TVPReid dataset.To further solve the mentioned issues in previous works, we propose a new task called Text-to-Video Person Retrieval(TVPR), and present an effective Multielement Feature Guided Fragments Learning (MFGF) strategy for robust person retrieval. In this model, we present fragments learning to mine high-quality representations, which use text and video fragments (such as high-frequency words and video frames) instead of the whole text and video. Firstly, we employ BERT [5] to extract text information. In terms of video, we leverage the S3D model [25] and ViT [6] to learn motion representation and visual representation within multiple successive frames. Subsequently, we integrate the knowledge acquired by S3D and ViT to generate a motion-enhanced representation. However, combining visual and motion features directly leads to information redundancy, at the same time there is a great semantic gap between text and video, thereby making accurate matching between text and video difficult.Considering these challenges, we introduce a Multielement Feature Guidance Learning strategy comprising two potential crossmodal spaces: Common space and Dual-Distilled space (𝐷 2 space). Specifically, we extract multielement features such as tips and dynamic movement sequences from distilled text and video information. Moreover, these multielement features are mapped into potential spaces for inherent essence correlations across different modalities and filter out the redundant information through contrastive calibration learning to guide purity feature matching between text and video. The proposed MFGF effectively enhances the reliability and accuracy of text-based person retrieval.Our main contributions can be summarized as follows:• We build a large-scale benchmark for text-to-video person retrieval, termed as Text-to-Video Person Re-identification (TVPReid) dataset. To the best of our knowledge, it is the first public dataset describing personal videos in natural language.• A novel task Text-to-Video Person Retrieval (TVPR) is proposed. Moreover, MFGF is designed to address occlusion and missing action details from isolated frames by purity crossmodal information and dynamic sequences, at the same time gradually reducing the semantic difference between text and video.• Extensive experiments are carried out to evaluate the proposed MFGF on the new person retrieval benchmark. MFGF demonstrates its superior performance and makes a successful attempt to employ video for text-based person retrieval tasks. 
related work 2.1 joint text-video understanding: 
 The joint text-video understanding model is devoted to comprehending both textual and visual inputs, condensing their representations, and subsequently establishing connections between text and video through joint embedding learning or alternative methodologies. In previous research, several text-learning models have exhibited outstanding performance in text understanding. Notably, Transformer [18] and the Transformer-based pre-trained language representation models like BERT have achieved notable success in this field. In video understanding, the most classic approach involves utilizing CNN models to generate frame-level representations for video. Subsequently, these frame-level representations are aggregated into video-level representations through mean pooling or LSTM [11] . While CNN-based models have shown promising results in obtaining video representations, they are constrained by their high computational complexity. In contrast, Transformerbased models offer high computational efficiency and are better suited for video retrieval tasks. For instance, Zhang et al. [29] introduce the Spatio-Temporal Transformer (STT), which leverages spatio-temporal information to address the issue of computational complexity.To bridge the two modalities, a common approach is to devise a joint embedding space to capture the shared semantic information, as well as compute the similarity between text and video representations. Luo et al. [15] devise three categories of similarity calculators for CLIP4Clip, namely parameter-free, sequential, and tight types, to assess the correlation between video features extracted by ViT and text features extracted by Transformer. Calculating the similarity between two distinct modalities directly poses considerable challenges. To address this, X-pool [8] introduces a scaled dot product attention mechanism to compute the final similarity score. The attention weights of the text across frames are employed to create an aggregated video representation, thereby bolstering the association between text and video.Text-based Person Retrieval entails retrieving pedestrian images based on provided textual queries. Li et al. [13] introduce a methodology that employs a CNN-RNN architecture to extract cross-modal features at a global scale. Initially, VGG [17] and LSTM architectures are utilized to learn visual-textual representations. These representations are subsequently aligned using a matching loss function. Expanding upon this foundation, ResNet [9] and BERT architectures have been incorporated for feature extraction. Additionally, a groundbreaking cross-modal matching loss is employed to reconcile global image-textual features by a unified embedding space. With the increasing interest in CLIP [16] , a surge in efforts is witnessed towards its advancement. Specifically, Yan et al. [27] , Jiang et al. [12] , Cao et al. [4] , and Bai et al. [1] are steadfastly devoted to harnessing the robust capabilities of CLIP to forge efficacious mapping relationships between images and textual descriptions. 
methodology: 
 In this section, we provide a detailed overview to the proposed Multielement Feature Guided Fragments Learning (MFGF) strategy, shown in Figure 3 . The network comprises three primary components: fragments learning of text, fragments learning of video, and multielement feature guidance learning.The function of Text Prompter is to extract keywords from the text, eventually composes 𝑇𝑖𝑝𝑠 (preparing for 𝐷 2 space in Section 3.3.2). We first perform distillation operations on all texts in the training data to obtain all keywords, including verbs, nouns, and adjectives, and then form into a set:K = {𝐾 1 , 𝐾 2 , • • • , 𝐾 ℎ },ℎ is the index of a sample of the dataset. Subsequently, we take the union of all subsets in K to get a Vocab Bank (VB), which is formulated as:EQUATIONAssume that given a video, we integrate all its sentence descriptions and then use the NLTK [14] tool to extract a set of keywords indicated as each video to obtain the initial 𝑇𝑖𝑝𝑠, which can be defined as :𝐾 ℎ = {𝑘 1 , 𝑘 2 , • • • , 𝑘 𝑛 }, 𝑘 𝑛 denotes 𝑛-𝑓 𝑂 𝑡𝑖𝑝𝑠 ∈ R 𝐷 𝑡𝑖𝑝𝑠 (2)where, the dimensions 𝐷 𝑡𝑖𝑝𝑠 depends on the size of the Vocab Bank. Significantly, these keywords often reveal a series of key movements and the appearance of pedestrians. In order to obtain more credible 𝑇 𝑖𝑝𝑠, the importance of each keyword in the given video also needs to be measured:EQUATIONwhere, 𝑤 𝑛 denotes the importance of 𝑛-th keyword; 𝑐 𝑛 means the number of 𝑛-th keyword appeared; and 𝐶 formulates the total number of all keywords appeared. It should be noted that 𝑤 𝑛 is not changed during the training.For the importance of all keywords in a given video, we set a semi-defined matrix:𝑊 ℎ = {𝑤 1 , 𝑤 2 , • • • , 𝑤 𝑛 } (4)where, the dimension of 𝑊 ℎ depends on the size of 𝐷 𝑡𝑖𝑝𝑠 . For all videos in dateset, the importance can be defined as:EQUATIONThe final 𝑇 𝑖𝑝𝑠 𝑓 𝑡𝑖𝑝𝑠 ∈ R 1×𝐷 𝑡𝑖𝑝𝑠 of given video is obtained by multiplying the weight 𝑊 ℎ and the initial 𝑇𝑖𝑝𝑠 𝑓 𝑂 𝑡𝑖𝑝𝑠 :EQUATIONFor all initial 𝑇 𝑖𝑝𝑠 F 𝑂 𝑡𝑖𝑝𝑠 in dataset, the final 𝑇𝑖𝑝𝑠 F 𝑡𝑖𝑝𝑠 can be formulated as a set: EQUATIONFor a video understanding model, processing all frames at the same time will cause heavy computational costs, so we intend to learn video representations from partial frames. These video frames serve as fragments of the video and have visual and motion information consistent with the whole video.The excellent performance and strong scalability of Vision Transformer (ViT) prompt us to study its application on video data. In this paper, we adopt the ViT with an improved version [2] to make it more suitable for processing video data. The subtle modifications are reflected in the residual connection part of the block input and the spatio-temporal attention output in the space-time transformer blocks. Given a video, 𝐿 1 frames are randomly selected to form a video sequence𝑉 𝑉 𝐸 = {𝑓 𝑟𝑎𝑚𝑒 1 , 𝑓 𝑟𝑎𝑚𝑒 2 , • • • , 𝑓 𝑟𝑎𝑚𝑒 𝐿 1 } as input, 𝑓 𝑟𝑎𝑚𝑒 𝐿 1 ∈ R 𝐻 ×𝑊 ×3. We divide each input video frame into 𝑁 patches of size 𝑃 × 𝑃, where 𝑁 = 𝐻𝑊 /𝑃 2 . Subsequently, we use convolutional layers to process video patches𝑋 = {𝑥 1,1 , 𝑥 1,2 , • • • , 𝑥 𝐿 1 ,𝑁 }, 𝑥 𝑙,𝑛 ∈ R 3×𝑃 ×𝑃denotes the 𝑛-th patch in 𝑙-th frame. And flatten the output to obtain the embedding sequence𝐸 𝑂 𝑣𝑖 = {𝑒 𝑂 1,1 , 𝑒 𝑂 1,2 , • • • , 𝑒 𝑂 𝐿 1 ,𝑁 }, 𝑒 𝑂 𝑙,𝑛 ∈ R 𝐷 𝑣𝑖means the embedding corresponding to 𝑥 𝑙,𝑛 , where 𝐷 𝑣𝑖 depends on the number of kernels in the convolutional layer. To enable the visual encoder to learn spatial and temporal knowledge, temporal and spatial embeddings are added to each embedding of patch to obtain input token 𝑒 𝑙,𝑛 ∈ R 𝐷 𝑣𝑖 :EQUATIONIt should be noted that patches at different spatial positions in the same frame have the same temporal embedding 𝐸 𝑇 𝑙 ∈ R 𝐷 𝑣𝑖 , while the patches at the same spatial position in different frames have the same spatial embedding 𝐸 𝑆 𝑛 ∈ R 𝐷 𝑣𝑖 . In this way, the visual encoder can more accurately understand the contextual connection between each block.Furthermore, similar to the text encoder, a learnable [CLS] token is added to the head of the token sequence and used for aggregating to generate the final visual feature 𝑓 𝑣𝑖𝑠𝑢𝑎𝑙 .Although transformer architectures such as the Visual Encoder described above can extract spatio-temporal features, which destroy the integrality of dynamic information. In order to ensure the continuity and integrity of dynamic information, we still need to design a scheme to make up for the lack of dynamic information.We adopt Spatio-temporal 3D Convolutional Neural Network (S3D) to learn stronger correlations in consecutive video frames and capture motion details. Different from Visual Encoder, Motion Encoder extracts 𝐿 2 continuous video frames 𝑉 𝑀𝐸 ∈ R 𝐿 2 ×𝐻 ×𝑊 ×3 as input. In the pre-processing part, we use 3D convolutional layers to learn the dynamic information in the video frames and the dynamic connection between adjacent frames to initially obtain the more conspicuous motion features 𝑓 𝑂 𝑚𝑜𝑡𝑖𝑜𝑛 in video frames:EQUATIONIn order to improve computational efficiency and obtain better accuracy, the S3D model uses a 3D convolution version that separates temporal and spatial convolutions.Specifically, it replaces the original standard [𝑘, 𝑘, 𝑘] 3D convolution with two consecutive convolutional layers, a [1, 𝑘, 𝑘] 2D convolutional layer, and a [𝑘, 1, 1] 1D convolutional layer, respectively. Where the 2D convolutional layer is used to learn spatial knowledge, and the 1D convolutional layer is used to learn temporal knowledge.Separate spatio-temporal Inception blocks are stacked to deeply learn detailed dynamic features. An Inception block receives the output 𝑦 𝑖 -1 of the previous layer as input, and then needs to be processed by spatial convolution and temporal convolution in order to extract dynamic spatio-temporal features:EQUATIONEQUATIONHere, ⊕ means the concatenate operation; Conv denotes the convolution with shape [1, 1, 1]; SpatConv and TempConv refer to the spatial convolution operation and the temporal convolution operation respectively; and MaxPool formulates the maximum pooling operation. The concatenate operation is introduced here to increase the width of the feature and the scale adaptability of the network.In order to further sharpen the details of dynamic features, a selfattention module is added after temporal convolution. First, we perform an average pooling operation on the temporal and spatial dimensions of the input features to obtain 𝑦 𝑝 𝑖 = 𝑝𝑜𝑜𝑙 (𝑦 ′′ 𝑖 ) and then perform the following operations:𝑦 𝑖 = sigmoid(𝜎𝑦 𝑝 𝑖 + 𝑏)𝑦 𝑝 𝑖 (12)where, 𝜎 and 𝑏 are the initial parameters of sigmoid, 𝑦 𝑖 is the output of the Inception block of the current layer, which is the input of the next layer. We take the output of the final layer as the motion feature 𝑓 𝑚𝑜𝑡𝑖𝑜𝑛 .We use two powerful video understanding models, ViT and S3D, to analyze visual and motion information in pedestrian videos. If only the features extracted in the visual encoder are used for retrieval, key motion details cannot be described due to weak dynamic information, which makes motion matching between text and video difficult. So we need S3D to make up for this shortcoming. We first concatenate the features extracted by the two encoders:EQUATIONHere, ⊕ also refers to the concatenate operation. Then the motion features are integrated into the visual features through the fully connected layer to enhance the motion information in the visual features and highlight the dynamic details:EQUATIONHere, 𝑓 𝑀𝐸 means motion-enhanced features, BN denotes the normalization operation, and FC refers to the fully connected layer.Figure 4 : ViT and S3D will repeatedly extract some visual and motion features, which will cause redundant information.There is a semantic difference between text and video, a gap that is difficult to bridge. In this paper, we design two potential spaces for text and video feature collaborative learning to progressively reduce the semantic difference between text and video by text-based multiple-frame visual and motion interaction.In order to ensure effective interaction between text and video features, 𝑓 𝑀𝐸 and 𝑓 𝑇 𝑒𝑥𝑡 will be projected into the same dimensional space, to learn a common feature distribution. Specifically, we continuously adjust text and video features through contrastive calibration learning, and close the distance between matched text and video pairs by calculating similarity scores. The similarity score between the video and the text is measured by calculating the cosine similarity between the two features.EQUATIONWe follow the training strategy in [28] and calculate the softmax loss, where matched text-video pairs in the same batch are regarded as positive samples and other text-video pairs are regarded as negative samples. We calculate the text-to-video loss as follows : ) refers to the method of calculating the similarity score between 𝑖-th text and 𝑗-th video, and 𝜃 refers to the temperature coefficient. Through this loss function, our purpose is to learn the unified feature distribution between text and video, thereby providing a more accurate basis for text-tovideo matching.L 𝑐𝑜𝑚𝑚𝑜𝑛 = -Common space initially narrows the distance between matched text and video pairs by learning the common feature distributions between text and video. However, there are still interfering redundant information and impurities in text and video features. Redundant information in text features comes from words that do not have the ability to describe appearance and actions, such as prepositions. In terms of video, one part of redundant information comes from the feature fusion process (as shown in Figure 4 ), because S3D and ViT will capture repeated visual and motion feature. The other part of the redundant information and impurities come from the invalid information contained in video that is irrelevant to the retrieval target. As shown in Figure 5 , the appearance information of the main target in the video is interfered by other pedestrians. When this redundant information appears in the video, it will be more difficult for the model to match text-video pairs. In the Dul-Distilled space, our purpose is to filter out invalid information in the features. The 𝑇 𝑖𝑝𝑠 we obtained in section 3.1.2 will be used as an indicator to measure the quality of text and video features. First, the text and video features need to be preprocessed. In the feature convertor, we first use linear layers to adjust the dimensions of text and video features to be consistent with the dimensions of 𝑇𝑖𝑝𝑠. Then, since 𝑇𝑖𝑝𝑠 uses a special one-hot encoding method and its content reveals the importance of each keyword, we also need to adjust the text and video features to represent the importance of every visual and motion detail. The converted text and video features are formulated as follows:EQUATION)EQUATION)Given a batch of samples, to prevent the converted text and video features from losing their original semantic commonalities, we calculate the similarity between text and video and optimize the following loss: and V 𝐷 2 represent a batch of converted text and video features. By calculating the above loss, our purpose is to optimize and constrain the feature convertor to avoid losing key features during the conversion process.L (T 𝐷 2 ,V 𝐷 2 ) = -The effective key information in the text and video features are able to be identified and distilled after the features are converted. Since 𝑇 𝑖𝑝𝑠 only contains keywords needed for retrieval, converted text and video features need to be closer to 𝑇𝑖𝑝𝑠 to achieve the purpose of distillation. We optimize the sum of two cross-entropy losses to constrain the distillation process of text and video features:EQUATION)Here, BCE means the binary cross entropy loss function, and Tips denotes a batch of 𝑓 𝑡𝑖𝑝𝑠 correspond to the current samples. The final loss L 𝐷 2 in Dul-Distilled space is formulated as:EQUATIONOur overall training goal is weighted the sum of 𝐶𝑜𝑚𝑚𝑜𝑛 loss and 𝐷 2 loss:EQUATION)where 𝛼 is a learnable weight parameter. To this end, we have obtained the final training objective of MFGF. Our goal is to learn a unified distribution of text and video features and then mine the internal correlation between text and video. 
experiments: 
 We first introduce our TVPReid dataset, followed by the implementation of the model. Then, we compare our results with classic video retrieval algorithms, and finally, we conduct ablation experiments to analyze the impact of different parts of the model on the experimental results.We build a large-scale labeled video dataset for the text-to-video person retrieval task and name it Text-to-Video Person Re-identification (TVPReid) dataset. Our dataset includes a total of 6559 pedestrian videos from three existing person re-identification datasets: PRID-2011 [10] , iLIDS-VID [20] , and DukeMTMC-VideoReID [23] . These source data contain image data of different pedestrians from surveillance videos. We first integrate them into the video data we need.For image data from the same pedestrian in the same time period from the same perspective, OpenCV [3] is used to integrate them into a video. After integrating three complete person reidentification datasets and eliminating highly similar data, we obtained a total of 6559 unique pedestrian videos. Then we annotate each pedestrian video with two different sentence descriptions, for a total of 13118 sentences. The sentence description adopts a natural language style and contains rich details about the pedestrian's appearance, actions, and environmental elements that interact with the pedestrian. The average sentence length of the TVPReid dataset is 30 words, and the longest sentence contains 83 words. In comparison, the average sentence length of RSTPReid [30] is 23 words, and that of CUHK-PEDES [13] is 23.5 words. The TVPReid dataset is divided into a training set, a validation set and a test set with a ratio of 0.8125 : 0.0625 : 0.125, which is based on the division method of the MSRVTT [26] dataset. Details of each sub-dataset can be seen in Table 2 . Among them, TVPReid-PRID has 2268 sentence descriptions, TVPReid-iLIDs has 1200 sentence descriptions, and the largest sub-dataset TVPReid-Duke has 9650 sentence descriptions.We randomly select 4 discrete video frames as the input of the Visual Encoder, 16 consecutive video frames as the input of the Motion Encoder, and each input video frame is adjusted to 224×224. The size of each patch in Visual Encoder is set to 16 × 16. During the training process, both sentence descriptions of each pedestrian video are used, but only one of the descriptions is extracted during the validation and testing sessions. The model uses Adam as optimization and sets the learning rate to 1 × 10 -4 . The learning rate setting here is slightly larger because we add learning rate Table 1 : Results on the proposed MFGF and compared methods to the proposed dataset, while recall at R@N and median rank.TVPReid-PRID TVPReid-iLIDs TVPReid-Duke R@1 R@5 R@10 R@50 MdR R@1 R@5 R@10 R@50 MdR R@1 R@5 R@10 R@50 MdR Frozen-in-time [2] We conduct a series of comparative experiments on the new benchmark dataset TVPReid to evaluate the effectiveness of MFGF. We compare our method with several existing methods dedicated to text-to-video retrieval. The aim is to understand the strengths and weaknesses of our approach. The performance results of several different methods on TVPReid are shown in Table 1 . We use standard retrieval metrics, including recall of rank N (R@N) and median ranking (MdR), where higher R@N values and lower MdR values both mean the great and stable ability of retrieval.It should be explained that the MMT [7] method can process visual and audio information, but the videos in the TVPReid dataset do not have audio, so we abandon the audio branch of MMT. From the data in the table, we can see that our method achieves the best performance in the recall metric, which shows that MFGF can more accurately match the correct text-video pairs in text-to-video person retrieval. The reason why our proposed method achieves powerful performance is that our method considers the effect of dynamic information, which can emphasize the difference of motion details between adjacent frames. While the mentioned models ignore the motion details, leading to poor performance.We conduct experiments to analyze the effectiveness of different components in our architecture. Our aim is to determine the contributions of each component to the overall network. The capacity of a model to acquire valuable representations from video data significantly influences retrieval performance. Therefore, we implement ablation experiments on Visual Encoder to evaluate its contribution to MFGF. We compare the results of three pairs of experiments in Table 3 : No.3 and No.7, No.4 and No.6, and No.8 and No.9, and find that the retrieval performance of MFGF without Visual Encoder is heavily reduced. This is because Visual Encoder provides most of the pedestrian's appearance features. For pedestrians whose clothing colors are the same, MFGF needs more detailed appearance features. So losing the learning ability provided by Visual Encoder, MFGF will be unable to accurately distinguish pedestrians with similar clothing, resulting in a decrease in retrieval accuracy.In order to evaluate the effectiveness of the Motion Encoder, we conduct experiments using the Motion Encoder and without using the Motion Encoder. We also compare three pairs of experimental results in Table 3 : No.1 and No.3, No.2 and No.9, and No.5 and No.6. And it can be seen from the data that after removing the Motion Encoder, the performance of the model is also reduced. The reason is that without the dynamic details provided by the Motion Encoder, the model's understanding of motion information becomes blurred. It is difficult to learn sufficient motion details by Visual Encoder, which will cause some key yet discriminative features missed in the final video features. In order to make up for this missed knowledge, we use the S3D model in Motion Encoder to capture the details of pedestrian movements in the video. With the support of Motion Encoder, the model greatly enhances its ability to identify the correct pedestrian from similar pedestrians through motion differences.In order to evaluate the effectiveness of 𝐷 2 space, we need to retain Common space to ensure the normal operation of MFGF. It can be seen from the results of No.3 and No.9 in Table 3 that the addition of the 𝐷 2 space improves the test results on the same sub-dataset. This is because in 𝐷 2 space, unmatched impurities in text features and video features are gradually filtered out, and they continue to move closer to purity features under the guidance of Tips. It can also be seen that poor text and video features will increase the difficulty of cross-modal matching. Therefore, it can be seen that 𝐷 2 space can not only filter out redundant information in text and video features but also learn key internal relationships in text and videos.In the previous ablation experiment, we keep the Common space R@1 R@5 R@10 R@50 R@1 R@5 R@10 R@50 R@1 R@5 R@10 R@50No.  
conclusion: 
 In this article, we propose a new task, Text-to-Video Person Retrieval. Our purpose is to make up for the lack of dynamic features and occasionally occluded details in isolated images. Since there is no dataset or benchmark that describes person videos with natural language, we construct a large-scale cross-modal person video dataset, termed as Text-to-Video Person Re-identification (TVPReid) dataset, which contains 6559 person videos, and each video has two natural language descriptions, with a total of 13118 description sentences. The dataset will be made publicly available to contribute to future research in this area. On this basis, we proposed Multielement Feature Guided Fragments Learning (MFGF) strategy to tackle uncertain occlusion conflicting and variable motion details. Specifically, we establish two potential cross-modal spaces for text and video feature collaborative learning to progressively reduce the semantic difference between text and video. Experimental results show that MFGF achieves state-of-the-art performance on the TVPReid dataset, and it is able to effectively retrieve relevant personal videos through natural language descriptions. These results highlight the potential of our approach in practical applications such as video surveillance and content-based video retrieval.  
 
 Appendix:  


Decision: In summary, the reviewers highlight several strengths of the paper, including its introduction of the novel Text-to-Video Person Retrieval (TVPR) task and dataset, the effective Multielement Feature Guided Fragments strategy, and detailed experimental results. Minor concerns are raised regarding the clarity and organization of the writing, the need for more statistical representation of experimental results, and the inclusion of specific implementation details such as video parameters and data management plans.

Overall, the ratings range from Borderline Accept to Borderline Reject. Most reviewers appreciate the authors' responses and vote towards acceptance. Considering this, I recommend accepting it for publication. However, regarding the limitations, please revise your manuscript further based on the reviewers' comments.

Reviews: review: paper_summary: paper_summary: In this paper, the authors propose a new task called Text-to-Video Person Retrieval, aimed at addressing the limitations of dynamic features and occasionally occluded details in isolated images. To support this, they created the Text-to-Video Person Re-identification (TVPReid) dataset, which is reconstructed based on the existing datasets. Furthermore, they introduce the Multielement Feature Guided Fragments Learning (MFGF) strategy to handle challenges such as uncertain occlusion conflicts and variable motion details.
strength_weakness: strength_weakness: - This study presents an approach to addressing the problem of dynamic features and occlusion in isolated images.
- The proposed dataset looks highly valuable.
- The description of the proposed method is well written and detailed.
limitations: limitations: (1) It is difficult to discern the ablation study results for the proposed method. The results shown in Table 3 are not organized according to specific rules, and the results for individual modules are absent. If the results are well-aligned for all combinations, it will be easier to understand the effectiveness of the proposed method.

(2) When I check the implementation details, it appears that randomness may be involved in the experiment described in the text, yet all experiments are presented with only one result. It seems necessary to verify whether the randomness is controlled, and even if it is, it would be more justified to express the experiment's results as statistical figures (e.g., means and standard deviations) based on executions with different seeds.

(3) The limitations of this study have not been disclosed. While this is not mandatory and thus has a minor impact on the review, including it would be beneficial. Moreover, for a paper that proposes a new dataset, I believe it is more important to address how the dataset will be managed in the future (e.g., corrections, supplements) rather than focusing solely on its initial construction. I am curious to know if there are any plans for this.


The credit can change depending on the response.
paper_summary: paper_summary: The paper proposes a new task named Text-to-Video Person Retrieval, along with the Text-to-Video Person Re-identification dataset. To address the multi-modal semantic gap, the paper proposes the Multielement Feature Guided Fragments Learning strategy for text-visual and text-motion matching, which collaboratively reduces the semantic difference between text and video, leading to robust person retrieval as demonstrated in the experimental section.
strength_weakness: strength_weakness: The paper introduces the first attempt at Text-to-Video Person Retrieval, which tackles the inherent occlusion problem that commonly occurs in the typical image-based task. The idea is simple, interesting, and intuitive. 

The paper introduces a novel Text-to-Video Person Re-identification dataset and corresponding benchmarks, which provide solid foundations for follow-up research.

The Multielement Feature Guided Fragments strategy effectively addresses the concerns proposed in the introduction, which bridges the feature representation gap between multiple modalities.

The experimental results are detailed and promising.
limitations: limitations: The structure in the introduction can be rephrased for better clarity, especially the last two paragraphs before the contribution summarization mentioning MFGF.

Redundancy in the methodology part. For example, the authors spend too much effort in explaining others' work, such as S3D.

The construction of tips is the weighted summarization of keywords. However, for a sentence containing "black coat", "black trousers", and "black hair", where "black" is associated with different objects, the model simply counts the number of "black" and adds them up, without considering the relations between keywords.

The authors present the ViT and S3D for capturing visual and motion representation separately and remove the redundancy. However, is it possible to apply the one-step video representation learning backbone, such as video swin transformer, etc?
paper_summary: paper_summary: This paper proposes a new Text-to-Video Person Retrieval (TVPR) task, addressing the person obfuscation problem and the missed motion details in typical text-to-image person retrieval task. To conduct TVPR, this paper constructs a Text-to-Video Person Re-identification (TVPReid) dataset and introduces a Multielement Feature Guided Fragments learning strategy (MFGF) for model optimization. Experimental results demonstrate the effectiveness of MFGF on the TVPReid dataset.
strength_weakness: strength_weakness: 1.	This paper proposes a novel Text-to-Video Person Retrieval (TVPR) task, providing more diverse person search capabilities in practical applications. The redundant and complex video data poses a challenge to the TVPR task, which is worthy of exploration in the future.

2.	This paper constructs a new benchmark, TVPReid, facilitating future research on TVPR.

3.	The designed MFGF strategy simultaneously captures motion and fine-grained key information using a Motion encoder and D^2 Space, respectively. Experiments demonstrate the effectiveness of both components.
limitations: limitations: 1.	The writing needs improvement. For example, there appear to be spaces within words in Figures 1, 3, and 5. In Figure 5, what the term “miss matched” referred to is unclearly. Symbols are missing at the end of equations, and there are grammatical mistakes in the methodology section. Furthermore, it would be advisable to consider introducing the text-video retrieval task, highlighting the unique differences and challenges it presents compared to the image-text retrieval task.

2.	There should be more examples of the text-video pairs from the TVPReid benchmark in the Appendix. From Figures 6 and 8, it appears that there is no significant difference in text descriptions between TVPReid and existing benchmarks for text-to-image person search. It seems the extra motion details involve mainly walking.

3.	It would be better to use SOTA methods for text-video retrieval in Table 1. As far as I am concerned, the video-text pretraining method UMT[1] shows excellent performance in text-video retrieval, or the X-CLIP[2], which emphasizes fine-grained alignment between video and text compared to the used X-pool. I wonder how these SOTA methods perform on the TVPReid benchmark.

4.	The main drawback is the lack of information on the data concatenation process, raising doubts about video quality. The authors should provide explicit details on video parameters (duration, frame rate) to avoid issues like short clips or noise interference. Additionally, the authors' video synthesis method differs significantly from real-world scenarios, reducing the dataset's practical relevance. Importantly, the publication of the datasets should be ensured.

5.	It would be advisable to cite more references. 

[1] Li, Kunchang, et al. "Unmasked teacher: Towards training-efficient video foundation models." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[2] Ma, Yiwei, et al. "X-clip: End-to-end multi-grained contrastive learning for video-text retrieval." Proceedings of the 30th ACM International Conference on Multimedia. 2022.