Not All Pairs are Equal: Hierarchical Learning for Average-Precision-Oriented Video Retrieval

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-06 11:31:32

Abstract: The rapid growth of online video resources has significantly promoted the development of video retrieval methods. As a standard evaluation metric for video retrieval, Average Precision (AP) assesses the overall rankings of relevant videos at the top list, making the predicted scores a reliable reference for users. However, recent video retrieval methods utilize pair-wise losses that treat all sample pairs equally, leading to an evident gap between the training objective and evaluation metric. To effectively bridge this gap, in this work, we aim to address two primary challenges: a) The current similarity measure and AP-based loss are suboptimal for video retrieval; b) The noticeable noise from frame-to-frame matching introduces ambiguity in estimating the AP loss. In response to these challenges, we propose the Hierarchical learning framework for Average-Precision-oriented Video Retrieval (HAP-VR). For the former challenge, we develop the TopK-Chamfer Similarity and QuadLinear-AP loss to measure and optimize video-level similarities in terms of AP. For the latter challenge, we suggest constraining the frame-level similarities to achieve an accurate AP loss estimation. Experimental results present that HAP-VR outperforms existing methods on several benchmark datasets, providing a feasible solution for video retrieval tasks and thus offering potential benefits for the multi-media application.

Main paper: 
 related work: 
 In this section, we will introduce several previous works that contribute to video retrieval and Average Precision optimization.Based on the granularity of similarity processing, video retrieval methods can be generally classified into two schemes, i.e. coarsegrained method and fine-grained method.2.1.1 Coarse-grained Method. This kind of method focuses on extracting and aggregating features into a vector space, representing each video by a single vector to compute similarity at the video level. In the early stage, methods such as Bag-of-Words [5, 50] , code books [30, 36] encode videos into a single vector by summarizing the extracted features through statistical aggregation, which neglect the temporal and spatial structures of the video. With the advent of deep learning in the video field, later approaches have started to train deep neural networks with metric learning [31, 35] , promoting the transition from coarse-grained methods to fine-grained methods in the subsequent research.2.1.2 Fine-grained Method. This kind of method typically extracts features from frames and thus generates multiple vectors to represent a video. Due to the utilization of a more enriched feature representation with spatial and temporal structures, fine-grained methods typically outperform coarse-grained methods within the same period. Early fine-grained methods focus on designing video temporal alignment solutions, e.g. temporal Hough Voting [15, 27] , graph-based Temporal Network [55, 58] and Dynamic Programming [11] , to match similar segments within the videos through hand-craft algorithms. Following the development of methods like TMK [44] and LAMV [2] , which use Fourier transform and kernel tricks for spatio-temporal representation learning, there has been a shift towards transformer-based architectures [16, 22, 23, 44, 52] . Among these methods, TCA [52] adopts a self-attention mechanism to capture temporal relationships among fine-grained features and utilizes a contrastive learning strategy for training, VRL [23] combines CNN with a transformer structure to train a model without labels. Recent methods concentrate on designing neural networks to learn similarity functions for calculating video-level similarities from original video representations. ViSiL [29] provides a supervised learning method that designs a 4-layer CNN to obtain video-level similarities from frame-level similarities. Additionally, Figure 3 : The architecture of our proposed framework. The data batch is processed through a feature extractor to obtain patch-level embeddings. Afterward, we compute frame-level and video-level similarity matrices utilizing spatial and temporal correlation aggregation modules in sequence. Simultaneously, the batch is fed into a pre-trained self-supervised model to generate pseudo labels that indicate frame-level relevance. Ultimately, we apply the QuadLinear-AP to both the frame-level and video-level similarity matrices and backpropagate the loss to optimize the model's parameters.DnS [33] employs knowledge distillation to train the student networks with ViSiL serving as a teacher network. More recently, S 2 VS [32] proposes a self-supervised learning approach built on the foundations of an improved structure of ViSiL. Despite previous studies developing increasingly complex models, their reliance on training with pair-wise objectives has led to a misalignment with the evaluation metric. Furthermore, these efforts typically focus solely on optimizing video-level similarity, neglecting the importance of frame-level similarity learning. In our work, we propose a method that employs an AP-based objective to bridge this gap, hierarchically optimizing AP for both frame-level and video-level similarities during the learning process.Traditional metric learning methods provide a learning paradigm for retrieval tasks, mapping instances into an embedding space and employing distance metrics to design pair-wise objective functions such as contrastive loss [10] or triplet loss [59] . However, these methods merely focus on increasing the distance between positive and negative instances within pairs or tuples, neglecting to improve the overall ranking of positives more comprehensively. This narrow focus can lead to overfitting, particularly in the face of imbalanced data distribution. A promising method is ranking-based metric learning with AP as the target. However, the non-differentiability of ranking terms in AP poses a challenge, obstructing the update of model parameters during backpropagation. To address this issue, numerous AP optimization methods have been developed. Listwise approaches [6, 7, 46, 57] utilize differentiable histogram binning to optimize loss functions based on ranking lists. Others provide structured learning frameworks based on SVM [39, 60] or conduct direct loss minimization [19, 53] to optimize AP. Moreover, Rolinek et al. [48] introduce BlackBox combinatorial solvers [43] to differentiate the ranking terms in AP. More recently, Brown et al. [4] propose Smooth-AP to use the Sigmoid function for approximating the indicator function, offering a simple and efficient way to differentiate AP. However, approximation methods like Smooth-AP neglect the gradient vanishing in the low AP area. To this end, we propose QuadLinear-AP, a novel loss for AP optimization, to designate appropriate gradients to the improperly ranked positivenegative pairs, ensuring the efficiency of the optimization process. 
methodology 3.1 task definition: 
 In the video space X, each video can be seen as a tensor 𝑽 = {𝒗 𝒋 ∈ R 𝐻 ×𝑊 ×𝐶 } 𝑇 𝑗=1 where 𝑇 , 𝐻 , 𝑊 , and 𝐶 represent the dimension of time, height, width, and channel, respectively. Given a pair of videos 𝑽 1 , 𝑽 2 ∈ X, video similarity learning aims to learn a similarity function 𝑓 : X×X → R, such that 𝑓 (𝑽 1 , 𝑽 2 ) represents the relevance between 𝑽 1 , 𝑽 2 . During the training stage, at each step, we sample a batch of videos 𝑩 = {𝑽 𝑖 ∈ X} 𝑁 𝑖=1 where the length of 𝑽 𝑖 is 𝑇 𝑖 . Let 𝒀 ∈ {0, 1} 𝑁 ×𝑁 be the video-level relevance matrix, where 𝒀 𝑖 𝑗 = 1 if 𝑽 𝑖 and 𝑽 𝑗 are relevant or 𝒀 𝑖 𝑗 = 0 otherwise. For the sake of presentation, we denote the similarity score as 𝑠 𝑖 𝑗 = 𝑓 (𝑽 𝑖 , 𝑽 𝑗 ), and denote the rankings among positive/negative subsets as 𝑺 𝑘+ = {𝑠 𝑘𝑖 = 𝑓 (𝑽 𝑘 , 𝑽 𝑖 )|𝑽 𝑖 ∈ 𝑩, 𝒀 𝑘𝑖 = 1, 𝑘 ≠ 𝑖}, 𝑺 𝑘 -= {𝑠 𝑘𝑖 = 𝑓 (𝑽 𝑘 , 𝑽 𝑖 )|𝑽 𝑖 ∈ 𝑩, 𝒀 𝑘𝑖 = 0}.According to the above definition, we aim to optimize 𝑓 such that 𝑓 (𝑽 𝑘 , 𝑽 𝑖 ) > 𝑓 (𝑽 𝑘 , 𝑽 𝑗 ) if 𝒀 𝑘𝑖 = 1 and 𝒀 𝑘 𝑗 = 0, such that it achieves a higher AP score:EQUATIONwhereR(𝑠, 𝑺) = 1 + 𝑠 ′ ∈𝑺 H (𝑠 ′ -𝑠) is the descending ranking of 𝑠 in 𝑺, H (•) is the Heaviside function [45], i.e., H (𝑥) = 1 if 𝑥 > 0 otherwise H (𝑥) = 0.We aim to design an AP-oriented framework for video similarity learning to align the training objective with the evaluation metric of video retrieval. As illustrated in fig. 3 , given two videos 𝑽, 𝑽 ′ ∈ X, we first utilize a feature extractor 𝑔(•) to extract patchlevel embeddings 𝑔(𝑽 ), 𝑔(𝑽 ′ ) ∈ R 𝑇 ×𝑅×𝐷 , where 𝑇 , 𝑅, 𝐷 are the number of frames, patches, and the embedding dimension, respectively. Afterward, the patch-to-patch similarities are measured with the cosine similarity, resulting in a patch-level similarity matrix 𝑆(𝑽, 𝑽 ′ ) ∈ R 𝑇 ×𝑅×𝑅×𝑇 ′ . Next, we optimize the similarity measure in a hierarchical strategy. At the video level, we aggregate the spatial and temporal correlation to video-level similarities via the proposed TopK-Chamfer Similarity (detailed in section 3.3). Following ViSiL [29] , we also apply a CNN to propagate the inter-frame similarities. Afterward, the video-level similarities are input into the proposed QuadLinear-AP loss. As outlined in section 3.4, for the frame-level constraint, we leverage a pre-trained vision model to generate pseudo labels and distill the frame-to-frame similarities to our feature extractor with the QuadLinear-AP loss.In this subsection, we first implement the similarity function 𝑓 through a bottom-up video similarity measure to map patch-level embeddings into similarities. Following this, we propose an AP surrogate loss with appropriate gradients for optimization, instructing 𝑓 to rank the similarities accurately.In this subsection, we present the detailed process of feature aggregation. Specifically, given a pair of videos, we first aggregate the patch-level similarities 𝑆(𝑽, 𝑽 ′ ) ∈ R 𝑇 ×𝑅×𝑅×𝑇 ′ along the spatial dimension, leading to a frame-level similarity matrix 𝑚 𝑠 (𝑽, 𝑽 ′ ) ∈ R 𝑇 ×𝑇 ′ . Afterward, we aggregate the temporal dimension as the video-level similarity 𝑓 (𝑽, 𝑽 ′ ) = 𝑚 𝑡 (𝑽, 𝑽 ′ ). Consider a batch of videos 𝑩 = {𝑽 𝑖 } 𝑁 𝑖=1 , similarities of all pairs form an 𝑁 × 𝑁 video-level similarity matrix.Early work utilizes a maximum/average operator to gather the fine-grained features. Kordopatis-Zilos et al. [29] suggest that two relevant frames/videos might be similar only in a part of region/period. From this perspective, to gather the spatial features, they propose to focus on the most similar region in 𝑔(𝑽 ′ ) for each query patch in 𝑔(𝑽 ), leading to the Chamfer-Similarity-based aggregation [3] :EQUATIONThe above operator identifies the maximum score for each query patch and averages these scores of all query patches in a frame to reflect the similarity between two frames. A similar operation is performed to gather the temporal features. However, focusing on the maximum score makes the similarity measure sensitive to spatial noises caused by distractors. Besides, different from the patch-to-patch similarity matrix with a fixed shape, the temporal dimension in videos is flexible and varies greatly. Furthermore, given a query video 𝑽 𝑘 and two relevant candidate videos 𝑽 1 , 𝑽 2 , the Chamfer Similarity might assign equal similarities for both 𝑽 1 and 𝑽 2 , even if 𝑽 2 contains more relevant frames. Such a phenomenon reduces the distinguishability of positive samples, leading to an ambiguous ranking estimation.Therefore, we seek a fine-grained similarity measure to estimate a precise AP loss. Specifically, rather than taking the maximum value, we jointly consider the top K scores:EQUATIONwhere 𝐾 = 𝑘 𝑠 ×𝑅 and 𝑆(𝑽, 𝑽 ′ ) 𝑥,𝑖,[𝑗],𝑦 refers to the 𝑗-th largest value, or formally:𝑆(𝑽, 𝑽 ′ ) 𝑥,𝑖,[1],𝑦 ≥ • • • ≥ 𝑆(𝑽, 𝑽 ′ ) 𝑥,𝑖,[𝑅],𝑦 .On top of the frame-level similarities, following ViSiL [29] , we utilize a CNN block 𝜓 to fuse the frame-to-frame similarities:EQUATIONwhere 𝑠 > 1 is the downsampling factor of 𝜓 . In this way, the framelevel similarity is mapped into a learnable measure space. Additionally, it downscales the similarity matrix to reduce the computational burden. Afterward, we utilize the proposed TopK-Chamfer Similarity in the temporal dimension, leading to the video-level similarity:EQUATIONwhere 𝐾 = 𝑘 𝑡 ×𝑇 ′ and m𝑠 (𝑽,𝑽 ′ ) 𝑖,[1] ≥ • • • ≥ m𝑠 (𝑽, 𝑽 ′ ) 𝑖,[𝑇 ′ ] .On one hand, compared with the original Chamfer Similarity, the TopK-Chamfer Similarity maintains fine-grained information; on the other hand, compared with the average operator, it avoids the disturbing noise introduced by the irrelevant frames. 𝐴𝑃 ↓ 𝑘 (𝑓 ) = 1 -𝐴𝑃 𝑘 (𝑓 ) = 1 |𝑺 𝑘+ | ∑︁ 𝑠 𝑘𝑖 ∈𝑺 𝑘+ 𝑠 𝑘 𝑗 ∈𝑺 𝑘 -H (𝑑 𝑘 𝑗𝑖 ) 1 + 𝑠 𝑘 𝑗 ∈𝑺 𝑘+ ∪𝑺 𝑘 -H (𝑑 𝑘 𝑗𝑖 ).(6) This AP risk is not differentiable due to the discontinuous function H (•). To this end, previous methods such as Smooth-AP [4] employ the Sigmoid function as a surrogate function:EQUATIONwhich results in an approximation risk, i.e., › 𝐴𝑃 Although Smooth-AP provides a straightforward solution to address the non-differentiable problem of AP, it might suffer from a gradient vanishing issue. Specifically, as shown in fig. 4 , when the score of a negative instance 𝑠 𝑘 𝑗 significantly exceeds that of a positive instance 𝑠 𝑘𝑖 , i.e. 𝑑 𝑘 𝑗𝑖 ≫ 0, the corresponding gradient is expected to be large such that the similarity function 𝑓 can be corrected. However, as depicted in fig. 4a , the gradient magnitude tends to 0, leading to slow convergence and sub-optimal solutions. This phenomenon is more evident in video similarity learning since the partial matching property (see section 3.3.1) makes 𝑑 𝑘 𝑗𝑖 more likely to fall into the gradient-vanishing area. To avoid this issue, we aim to propose a novel AP loss. To begin with, we argue that it is unnecessary to replace all H (•). Notice that the original AP risk in eq. ( 6) can be reformulated as:EQUATIONwhere ℎ(𝑥) = 𝑥 1+𝑥 is a monotonically increasing function. Then, the non-differentiable terms H (𝑑 𝑘 𝑗𝑖 ) can be divided into two types: 1) The positive-negative pair (𝑠 𝑘 𝑗 ∈ 𝑺 𝑘 -) in the numerator, which should be minimized to ensure the correct ranking; 2) The positivepositive pair (𝑠 𝑘 𝑗 ∈ 𝑺 𝑘+ ) in the denominator, which plays a role of weights. From this perspective, we only need to ensure that the surrogate loss of the former has an appropriate gradient, while for the latter we can directly use the original rankings such that the importance of each term can be precisely measured.Motivated by the above observation, for positive-positive pairs we still utilize the Heaviside function:EQUATION)As for the positive-negative pairs, the derivative of the surrogate loss should be large for the wrongly ranked pairs, i.e. 𝑑 𝑘 𝑗𝑖 + 𝛿 ≥ 0 for a given margin 𝛿 > 0. Besides, the surrogate loss should be convex such that the derivative is (non-strictly) monotonically increasing. Therefore, we design the following surrogate loss for positive-negative pairs:EQUATIONThe curves of R -(𝑥; 𝛿) and its derivative are visualized in fig. 4c and fig. 4d . Obviously, the above surrogate loss satisfies our design principles. Furthermore, by introducing an extra parameter 𝜌 to adjust the weight of positive-positive pairs, the score distribution between positive and negative instances can be balanced well. The analysis above induces the formulation of the following AP loss, namely QuadLinear-AP:EQUATIONwhich enjoys the following attractive properties:• Differentiable AP optimization QuadLinear-AP is differentiable for AP term, making it possible to backpropagate gradients in the learning process. • Suitable gradients for low AP area. Persistent and suitable gradients in the loss function force model to correct wrongly ranked positive-negative pairs, avoiding gradient vanishing in the low AP area. • Favorable mathematical properties. QuadLinear-AP is continuous, convex, and (non-strictly) monotonically increasing, ensuring a stable convergence during optimization. As formulated in eq. ( 12), the final AP loss is calculated by averaging QuadLinear-AP across all query videos, which is then applied to the video-level similarity learning process. Clearly, this objective is aligned with the evaluation metric.EQUATION)As discussed in section 1, two relevant video instances may not be completely relevant at the frame level due to the noticeable variation in the temporal dimension, i.e., only several frames are highly relevant with a query frame while the others are relatively low in actual. Therefore, solely optimizing 𝑓 on video-level instances proves inadequate. Next, we dive into the frame-level learning. Given a query frame, it is hard to locate the relevant frames from another video without fine-grained annotations. A possible route is leveraging self-distillation methods [8, 56] , which refines image features by distilling ensemble information from a mean teacher to the target model in a self-supervised manner. Unfortunately, since our feature extractor 𝑔 is trained with video data, it might ignore some image-level information. In this case, the pseudo labels generated by 𝑔 cannot provide more informative supervision.Consequently, we introduce another feature extractor𝑔 ′ : X ↦ → R 𝑇 ×𝐷 ′, where 𝐷 ′ is the embedding dimension. The feature extractor is pre-trained on image data with a self-supervised learning algorithm DINO [8] , and the parameters are frozen. Given a video pair 𝑽, 𝑽 ′ , we use 𝑔 ′ to extract features for all frames and compute the following frame-level similarities, where 𝑽 𝑥 and 𝑽 ′ 𝑦 are the 𝑥-th frame of 𝑽 and 𝑦-th frame of 𝑽 ′ , respectively.EQUATION)As shown in previous study [18] , the similarities are highly correlated to the relevance. However, for different queries, the similarity distributions of its relevant/irrelevant frames are various, hence discretizing them into binary pseudo labels with fixed thresholds is impractical. Instead, we identify the frames with the highest/lowest similarities as positive/negative, leading to the following pseudo labels:EQUATIONwhere 𝑟 𝑡 , 𝑟 𝑏 < 1 are tunable hyperparameters, 𝑆 ′ (𝑽, 𝑽 ′ ) 𝑥,[𝑘] refers to the 𝑘-th largest value in {𝑆 ′ (𝑽, 𝑽 ′ ) 𝑥,𝑦 } 𝑇 ′ 𝑦=1 . Notice the varying similarity distributions across different video types, it's suboptimal to set a fixed threshold for positive or negative frames to exceed during the training phase. A feasible solution is training the model to learn to rank positive frames above the negative ones. Resembling the method in video-level learning, we optimize the frame-level similarities by L 𝐹 𝑄𝐿𝐴𝑃 , which can be implemented by substituting the video instances with frame instances.Following previous methods on ranking optimization [13, 51] , we combine a basic loss L 𝑏𝑎𝑠𝑒 with the AP losses to promote collaborative optimization between ranking and representation learning. The basic loss comprises the InfoNCE loss [40] to support representation learning and an SSHN loss [32] for hard negative mining. Please refer to supplementary material for details.Ultimately, the total loss for hierarchical similarity learning is formulated in eq. ( 15), where 𝜆 𝑓 and 𝜆 𝑣 are hyperparameters for the trade-off between components, leading to the final optimization algorithm as summarized in algorithm 1. Sample a batch of videos {𝑽 𝑖 } 𝑁 𝑖=1 form 𝑺.EQUATIONExtract video embeddings 𝑔(𝑽 𝑖 ) and 𝑔 ′ (𝑽 𝑖 ).Generate pseudo labels Ŷ based on 𝑟 𝑡 and 𝑟 𝑏 .6:Calculate similarities with function 𝑓 in eq. ( 5).Compute " 𝐴𝑃 ↓ 𝑘 (𝑓 ) with eq. ( 11) to form L 𝑉 𝑄𝐿𝐴𝑃 and L 𝐹 𝑄𝐿𝐴𝑃 .8:Compute the total loss L by eq. ( 15). 9:Update parameters: 𝚯 𝑙+1 = 𝚯 𝑙 -𝜂 𝑙 ∇L. 10: end for 
experiments: 
 In this section, we begin with a brief overview of the basic settings, including the datasets, evaluation metrics, and implementation details. Next, we compare our proposed learning framework with several previous methods on three benchmark datasets. Finally, we conduct an ablation study to evaluate the performance of different modules. For further details, please see the supplementary material.Datasets. Our model is trained on the unlabeled subset of VCDB dataset [27] (we denote the core data and distractors as C and D, respectively) and evaluated on EVVE [47] , SVD [26] , and FIVR-5K/FIVR-200K [28] . For the FIVR dataset, we report the results of three specific subtasks: DSVR/DSVD, CSVR/CSVD, and ISVR/ISVD. Evaluation Metrics. For retrieval tasks, we adopt Mean Average Precision (mAP) as the evaluation metric. Specifically, mAP calculates the average AP scores for each query independently and then averages these scores to reflect the model's overall ranking performance. For detection tasks, we employ Micro Average Precision (𝝁AP), a metric widely used in previous studies [32, 34, 41, 42] . The 𝜇AP calculates the AP across all queries simultaneously, demonstrating the model's capability to consistently apply a uniform threshold across various queries to detect relevant instances.Implementation Details. Given an input video, we generate two video clips by applying random augmentations that include temporal manipulations [29, 32] , spatial transformations [12, 42] , and other basic augmentations. For the feature extractor, we adopt ResNet50 [21] following [29, 32, 33] , and for the pseudo label generator, we utilize DINO [8] pretrained ViT-small [14] with a patch size of 16. Our model is trained for 30,000 iterations with a batch size of 64. We use AdamW [38] with the Cosine Annealing scheduler for parameters optimization. The learning rate is set to 4 × 10 -5 with a warm-up period [37] , and weight decay is set to 1 × 10 -2 .Competitors. We evaluate HAP-VR against various leading video retrieval methods, categorized into two types. 1) Supervised methods include DML [31] , TMK [44] , TCA [52] , ViSiL [29] , DnS [33] with an attention student network (𝑆 𝑎 ) and with a binarization student network (𝑆 𝑏 ). 2) Unsupervised methods include LAMV [2] , The overall performance on video retrieval and detection tasks above is reported in table 1, leading to several key conclusions: 1) HAP-VR stands out among other unsupervised or self-supervised methods in both mAP and 𝜇AP metrics, with an average improvement of 0.71% and 2.25%, respectively. These outcomes underscore the effectiveness of aligning the training objectives with the evaluation metrics, directly enhancing the average precision. 2) Detection tasks enjoy larger performance gains than retrieval tasks. This is primarily due to the more pronounced imbalance between instances in detection tasks. By emphasizing the overall rankings of the positive instances, HAP-VR achieves a more optimal similarity distribution across all queries, resulting in a notable increase in 𝜇AP. 3) Compared with supervised methods, HAP-VR achieves a better overall performance. To investigate the underlying reason, we visualize the video similarity distributions in fig. 5 . Compared with the supervised methods, HAP-VR establishes a clearer margin between scores of relevant and irrelevant pairs. Since annotations are based on the video categories, the supervised model tends to distinguish the pre-defined categories but not video instances. Accordingly, when encountering videos beyond these pre-defined categories, the model is prone to overfit the categories, which hinders discriminating between negative instances, thereby reducing the model's transferability.Ablation results on proposed QuadLinear-AP loss. To validate the effectiveness of the proposed QuadLinear-AP loss, we make a comparison with other commonly used losses, which can be categorized into three types: 1) Point-wise losses, include Mean Absolute Error (MAE) and Mean Squared Error (MSE). These losses measure the discrepancy between predicted scores and actual labels for each item independently. 2) Pair-wise losses, include Contrastive loss [17] , Triplet loss [49] and Circle loss [54] . These losses focus on distinguishing between the positive and negative instances in pairs. 3) List-wise losses, include FastAP [6] , DIR [46] , BlackBox [43] , and Smooth-AP [4] . These approaches optimize the model directly based on ranking metrics such as AP.For a straightforward comparison, we only combine these losses with L 𝑏𝑎𝑠𝑒 and train the models using 10% of the VCDB (D) for 10,000 iterations. Except for the specific hyperparameters associated with each loss, all other settings remain constant to ensure a fair comparison.The comparison results are presented in table 2. From these results, we can draw the following conclusions: 1) In general, list-wise losses outperform point-wise and pair-wise losses, supporting our motivation to develop an AP-oriented method for video retrieval tasks. 2) QuadLinear-AP achieves an average improvement of about Ablation results on proposed modules. Comparing Line 1 with Line 2 in table 3, the application of the TopK-Chamfer Similarity measure yields average boosts of 0.59% on mAP and 1.13% on 𝜇AP based on the baseline model. This suggests the efficacy of the TopK-Chamfer Similarity measure, which will be further discussed in the ablation study on similarity measures. Comparing Line 2 with Line 3 shows that incorporating video-level AP optimization further enhances performance in retrieval and detection tasks, with increases of 0.69% and 2.13% respectively. Such improvements reveal the necessity of aligning training objectives with evaluation metrics. Moreover, implementing a frame-level learning process further improves the overall outcomes, emphasizing the value of learning the internal similarity within the video precisely.Ablation results on similarity measure. To validate the effectiveness of the proposed TopK-Chamfer Similarity measure, we evaluate the model performances varying the top-k rate 𝑘 𝑡 . Note that when 𝑘 𝑡 = 0.0, the measure can be seen as the original Chamfer Similarity; when 𝑘 𝑡 = 1.0, the measure is equal to average pooling. As indicated by the results in table 3, the optimal performance is achieved neither at 𝑘 𝑡 = 0.0 nor at 𝑘 𝑡 = 1.0. This outcome supports the utility of selecting top-K values. From another perspective, the best performance is obtained when 𝑘 𝑡 is small, demonstrating the capability of the TopK-Chamfer Similarity in diminishing redundancy and reducing the influence of noise, thereby ensuring robustness in similarity calculation. 
conclusion: 
 In this paper, we design a self-supervised framework for video retrieval, which features a video-oriented similarity measure to gather fine-grained features and a novel AP-based loss with reasonable gradients to correct mis-ranked instance pairs efficiently, filling the gap between the training objective and evaluation metric. Within the framework, we propose a hierarchical learning strategy to conduct AP optimization both on video and frame levels, which generates precise estimations of the AP loss, thus enhancing the accuracy of the similarity learning process. Experimental results demonstrate that our framework often surpasses previous works in several benchmark datasets, making it a feasible solution for video retrieval tasks. In future work, we plan to extend our framework to other applications, which we hope could support subsequent research to further contribute to the multimedia community.  
: 
 Figure 1 : Average Precision (AP) measures the average ranking of positive instances within a list, providing a comprehensive evaluation of the overall performance of the retrieval results. Pair-wise training objectives focus solely on pulling the positive instances closer while repelling the negative ones, failing to align with the AP metric. In contrast, AP-based objectives ensure this alignment by rectifying the rankings of mis-ranked positive-negative pairs in the list. embeddings of video frames, which have achieved a remarkable success compared with the early hand-craft approaches [11, 15, 27, 55, 58] . Nevertheless, these models are typically optimized by pair-wise training objectives such as triplet loss, which are inconsistent with the evaluation metric. Concretely, as a standard evaluation metric, Average Precision (AP) focuses on the top list by assigning larger weights to the top-ranked positive videos. Since the retrieval results are commonly processed sequentially in downstream tasks, the performance of the top list becomes crucial, thus making AP a more comprehensive metric as it better reflects this practical requirement. However, as shown in fig. 1 , pair-wise losses treat all mis-ranked video pairs equally, ignoring the relative rankings among the instance list. This leads to an evident gap between the training objective and the evaluation metric, calling for an effective AP-based objective function to bridge this gap.To solve a similar problem in image retrieval, a promising method is to optimize AP directly. Due to the non-differentiability of AP, existing methods concentrate on developing differential approximations of the AP [4, 6, 7, 46, 48, 57] . Although these AP optimization methods have achieved notable success in the image field, they cannot be directly applied to videos due to the complexity arising from the additional temporal dimension. Generally, the challenges are two-fold:a) The current similarity measure and the surrogate AP loss are suboptimal for AP-oriented video retrieval. Typically, in mainstream frameworks, image similarities are measured with the cosine similarity of an embedding pair, while video similarities are aggregated from the redundant temporal-spatial features. Since AP jointly considers the rankings of all instances rather than merely distinguishing whether a pair of videos matches, it is necessary to design a fine-grained similarity measure function for videos. Additionally, existing surrogate AP losses like Smooth-AP [4] suffer from a vanishing gradient when a sample pair is seriously misranked, leading to inefficient optimization. This phenomenon is more obvious for videos since the various video similarities fall into the gradient vanishing area more frequently.b) The noisy frame-to-frame matching leads to a biased AP estimation. As illustrated in fig. 2 , two relevant videos may not exhibit uniform relevance across all frames. Without fine-grained annotations, this ambiguity leads to false positive matching. In this case, the weights of the top-ranked videos might be reduced, hindering the AP loss from concentrating on the top list.Based on the above considerations, we propose the Hierarchical learning framework for Average-Precision-oriented Video Retrieval (HAP-VR), which contains video-level and frame-level constraints as detailed following:To tackle challenge a), we propose a topK-based similarity measure and a variant of AP loss with proper gradients. As a core component of our framework, the proposed TopK-Chamfer Similarity aggregates video-level similarities from frame-level similarities. Compared with previous maximum/average aggregations, the TopK-Chamfer Similarity retains fine-grained information while filtering out false correlations, providing refined video similarity for the following AP loss estimation. Another core component is a new surrogate AP loss, namely QuadLinear-AP, which enjoys a more reasonable distribution of gradients to rectify mis-ranked positive-negative pairs efficiently.In search of a solution to challenge b), we propose to correct the frame-level similarities without requiring fine-grained annotations. Motivated by the recent advance in self-supervised learning [9, 20, 24] , we leverage the pre-trained vision model [8] to extract frame-level representations. Subsequently, we generate pseudo labels indicating the matched frames from the gap between these representations and distill the frame-level information to avoid ambiguity, leading to a more precise estimation of AP loss.To summarize, the contributions of this work are three-fold:• We develop a self-supervised hierarchical learning framework for Average-Precision-oriented video retrieval, named HAP-VR, to fill the gap between training objectives and evaluation metrics that the previous work has overlooked. • Within HAP-VR, we propose the TopK-Chamfer Similarity and QuadLinear-AP loss to measure and optimize video-level similarities of the AP metric, alongside constraining framelevel similarities to produce a precise estimation of AP loss. • Our experimental evaluation of HAP-VR across several largescale benchmark datasets often presents a superior performance in terms of AP, ensuring its effectiveness for contentbased video retrieval tasks. 
 
 Appendix:  


Decision: This paper proposes a framework for assessing video retrieval results by considering the similarity of results, and therefore an alternative to the typical average precision metric. As confirmed by all reviewers, this is a novel approach and the paper not only provides a good motivation and is written well and clear, but also the claims of the work are confirmed by a sound and exhaustive evaluation with good results. Moreover, the authors provide the source code of their implementation. Overall, this is an interesting and novel contribution to the multimedia community.

Reviews: review: paper_summary: paper_summary: The authors introduce a novel method for improving video retrieval based on the self-supervised Hierarchical learning framework considering the Average precision during training. To this end, they aim to tackle the challenges for calculating the similarity by taking into account all the frames/features of the videos and addressing the vanish gradient problem that emerged in the cases where the samples are strongly mis-ranked. Specifically, they propose a novel loss for AP named, TopK-Chamfer Similarity and QuadLinear-AP loss to measure and optimize video-level similarities, alongside constraining frame-level similarities to produce a precise estimation of AP loss.
strength_weakness: strength_weakness: The paper's strengths are the novelty that is fully described and explained in the introduction section.  Specifically, the authors, present the significance of the video retrieval systems and the way these systems are evaluated (Average precision), and then they clearly illustrate their contributions based on a metric-learning-based approach. The work is comprehensibly presented and illustrated. The authors provide adequate state of the art of the related methods. They explain the state-of-the-art focus on fine-grained methods and detail illustrate the methods related to video retrieval as the main objective of the proposed work. In the same section, they also report the state of the art of the methods aiming to optimize metrics for improving their retrieval precision. In the methodology section the authors, provide a detailed explanation of the problem, making clear to the reader what are the objectives of the proposed work. They prove and explain both textually and mathematically, which helps the reader follow such a complex pipeline. Finally, the experimental setup section includes all the experiments to establish a fair comparison of the proposed method compared to the state-of-the-art.  The authors followed a common approach to prove that the proposed method outperforms the state-of-the-art, based on retrieval and detection evaluation. They also provide a necessary ablation study part that covers the unevaluated aspects during the main evaluation.
limitations: limitations: The proposed work reports a novel approach followed by a detailed technical justification. Although the work is well-written and comprehensively illustrated there are lacks in a few points. Please take care of the reported comments to improve the presentation of the work.
•	In line 12, “for the users” and in line 43, please remove “which”.
•	In Figure 2, there is no explanation of the TR and TQ axis, both in the text and in the caption. 
•	In lines 141 to 143, more details and an explanation of the phrase “fall into the gradient vanishing..” e.g. when aiming to maximize AP or minimize 1-AP similar to Smooth-AP losses. 
•	In line 155 the word “proper” gradients need further clarification. 
•	Section 2.1.2 consists of a very long paragraph that should be split into two paragraphs. 
•	In lines 322-327, there is a different formation of illustrating the methods. Update the formation of illustrating the methods to improve consistency. It would be better to keep reporting on the method names/cross-references and not provide author names.
•	Figure 3, the colour of the input batch sampling does not follow a consistent sequence in both streams, this should be explained in the text why inputting batches in a sequence of orange, red, blue, and green we extract embeddings of orange, red, green, and blue.
•	In line 362, please check the reference of the Heaviside function if it is the most representative of the definition.
•	In lines 368-369, there is confusion about what is illustrated in Fig 3 and whether there are input two videos or batches. Please rephrase and make it more clear either to the text or to the figure.
•	Lines 395-397, need further clarification when using the term of frame-level similarity and the term patch-level similarity.
•	In line 402 please update similarly to the above comment related to the author's name.
•	In line 467, this is the justification related to the vanish of gradient, minimizing the 1-AP. This is related to the comment for lines In lines 141 to 143.
•	In line 482, there is no specific reference to a specific tile of Fig 4, e.g. (a). Moreover, all the tiles belonging to Fig 4 should also be explained in the text.
paper_summary: paper_summary: The paper introduces several improvements in the methodology for video retrieval. The three main contributions include a hierarchical video retrieval architecture, a surrogate loss function that approximates AP loss with better differentiability and a different similarity measure. The authors provide theoretical grounding for their contributions and evaluate against a number of competitor systems.
strength_weakness: strength_weakness: The theoretical justifications provide a strong argument for their methodology. In particular: vanishing gradients can be problematic and the derivative of their QuadLinear-AP is more favorable to optimization than the Smooth-AP they compare themselves to. Their proposed TopK-Chamfer similarity can provide more fine-grained information since it focuses on the most relevant instances. The proposed architecture seems theoretically sound and is explained well. 

All claims in the paper are supported through both theory and experiments. The source is code is provided in a well-organized folder. At a glance the pytorch implementation appears reasonable. Both theory and results are further supported by supplementary material, keeping the main paper concise and providing additional insight.
limitations: limitations: It seems like the paper is an incremental improvement over previous methods. Many of the reference systems are developed by Giorgos Kordopatis-Zilos and do not seem to be state of the art on most video-retrieval datasets. His systems do show strong results on the FIVR dataset, which was created by the same author. There are comparisons to other unrelated systems, it just seems to be biased toward this particular researcher’s work. That said, there are comparisons on the EVVE and SVD datasets as well and this might not be a problem.
paper_summary: paper_summary: This paper proposes to tackle the gap between the training and  evaluation stages in the domain of video retrieval.
They propose a new methodology HAP-VR with a dedicated hierarchical similarity metric and learning framework.
Exhaustive experiments are presented to validate the proposal.
strength_weakness: strength_weakness: Major strength.
This paper is very well-written and easy to read. The authors clearly articulate their challenges with pedagogical figures, which is commendable.

Each step of their framework is justified by a thorough analysis of state-of-the-art references.

This paper makes a highly interesting contribution to the scientific community.

Exhaustive experiments and their performance evaluations are also among the strengths of this paper.
limitations: limitations: Minor limitations.
The paper is clearly structured and written similarly to the S2VS paper; however, its originality is distinctly different. The comparison result table evaluates the proposed method against other recent methodologies, with half of them developed by the same researchers. The other half of the comparisons include several instances of missing performance data due to unavailability - which is regretful.

At the end, a small typo, Table 3 should be Table 4. And all figures and tables should start with a capital letter. 

I think the ablation study analysis if limited to the content of the article is not detailed enough to explain clearly the given tables and figures. 

However, the supplementary material file completes greatly this article.
paper_summary: paper_summary: The paper presents a fine-grained video retrieval approach with an average-precision-based loss function.
strength_weakness: strength_weakness: The paper is generally well-written and describes the method in detail. While it is somewhat math-heavy, one can see that the authors tried not to complicate matters needlessly. The supplementary material contains further details and even the code of the implementation. While the authors do not explicitly mention it, this leads one to hope that they intend to publish their implementation as open-source software.
limitations: limitations: The paper has some minor linguistic and formatting issues. Specifically, it neglects to capitalize references to figures and sections. For a final version, it should undergo some additional proofreading.