Hierarchical Debiasing and Noisy Correction for Cross-domain Video Tube Retrieval

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 03:56:30

Abstract: Video Tube Retrieval (VTR) has attracted wide attention in the multi-modal domain, aiming to accurately localize the spatial-temporal tube in videos based on the natural language description. Despite the remarkable progress, existing VTR models trained on a specific domain (source domain) often perform unsatisfactory in another domain (target domain), due to the domain gap. Toward this issue, we introduce the learning strategy, Unsupervised Domain Adaptation, into the VTR task (UDA-VTR), which enables the knowledge transfer from the labeled source domain to the unlabeled target domain without additional manual annotations. An intuitive solution is generating the pseudo labels for the target-domain samples with the fully trained source model and fine-tuning the source model on the target domain with pseudo labels. However, the existing domain gap gives rise to two problems for this process: (1) The transfer of model parameters across domains may introduce source domain bias into target-domain features, significantly impacting the feature-based prediction for target domain samples. (2) The pseudo labels often tend to identify video tubes that are widely present in the source domain, rather than accurately localizing the correct video tubes specific to the target domain samples. To address the above issues, we propose the unsupervised domain adaptation model via Hierarchical dEbiAsing and noisy correction for cRoss-domain video Tube retrieval (HEART), which contains two characteristic modules: Layered Feature Debiasing (including the adversarial feature alignment and the graph-based alignment) and Pseudo Label Refinement. Extensive experiments prove the effectiveness of our HEART model by significantly surpassing the state-of-the-arts. The code is available (https://anonymous.4open.science/r/HEART).

Main paper: 
 related works: 
 Video Tube Retrieval. In the field of video tube retrieval, early techniques [25, 36] typically embraced a two-stage approach. They first necessitated the proposal of potential interest regions through a pre-trained object detector, followed by a custom-designed network to accurately select the relevant regions. The significant limitation inherent in these approaches is their heavy dependence on the capabilities of pre-trained detectors, which inherently constraint their performance. Contrastingly, a paradigm shift is observed in more recent research, [14, 15, 19, 24, 29, 31, 34] have pivoted towards a unified-stage methodology that bypasses the reliance on pre-trained object detectors entirely. For instance, [24] marked a significant milestone by employing a visual-linguistic transformer to achieve simultaneous spatial and temporal localization corresponding to the given textual query. Drawing inspiration from the success of text-driven object detection in [15] , [34] unveiled a combined video-text encoder and spatial-temporal transformer decoder to effectively bridge temporal, spatial, and multimodal interactions. In a similar vein, [14] introduced a multi-modal template approach to directly address the issues of alignment and consistency in feature prediction. Furthermore, [19] explores the integration of static and dynamic cues to refine target localization. Despite these advancements, the exploration into how these methods perform when applied to samples that deviate from their initial training set remains largely underexplored. Unsupervised Domain Adaption. Unsupervised domain adaptation, originally applied to image classification, has been broadly adapted for other tasks, among which object detection and video tasks are most closely related to our work. In recent years, cross-Domain object detection has attracted significant attention from researchers. Adversarial feature learning techniques [7, 23] utilize a domain discriminator to train the feature encoder, facilitating the extraction of domain-invariant features. Meanwhile, image-toimage translation approaches [4, 13] employ generative models like CycleGAN [38] to transform target-domain images into the style of the source domain, thereby bridging the domain gap. Recent efforts in domain adaptation [3, 5, 8, 9, 18, 35, 37] have focused on employing the Mean Teacher [26] . [8] tackles domain shift by leveraging [38] to augment training data, while [18] introduces a combination of weak and strong augmentations alongside adversarial training. Video unsupervised domain adaptation methods typically fall within a few common categories. Adversarial methods [6, 17] align representations across source and target domains through a domain discriminator network. Techniques like DANN [11] , originally designed for images, are also applicable to video architectures. [6] is a video-specific approach that uses separate adversaries for the spatial and temporal dimensions. Another class of methods [22] use contrastive learning and exploit the intrinsic structure of video. Despite rapid advancements, to the best of our knowledge, there has not yet been success in applying unsupervised domain adaptation to complex multimodal spatial-temporal video understanding tasks. 
preliminaries 3.1 adversarial learning: 
 Adversarial domain adaptation techniques are predicated on the notion of explicitly diminishing the discrepancy between domains by learning transferable features. These methods contrast with traditional supervised learning techniques by integrating a domain discriminator 𝐷 to differentiate between source and target domain features. Concurrently, the objective of the feature extractor 𝐹 is to generate features that confuse 𝐷, thereby promoting learning domain-invariant features. The loss function for the domain discriminator, 𝐿 dis , is expressed as follows:EQUATIONwhere 𝐷 𝑠 and 𝐷 𝑡 represent the features distributions within the source and target domains, respectively. The domain adversarial loss 𝐿 adv is formulated using a minimax optimization strategy:EQUATIONTo facilitate the minimax optimization, Gradient Reversal Layers (GRL) [10] are inserted to reverse the gradient during backpropagation from 𝐷.In graph theory, the distance between graph spectra quantifies the divergence in spectral characteristics of graphs with an equal number of vertices, simplifying the comparison of their discrepancies [1, 28] . Drawing from [33] , we present the definitions of graph Laplacians and spectral distances.Laplacians. Consider a finite graph 𝐺 = (𝑉 , 𝐸), comprising a set of vertices 𝑉 and weighted edges 𝐸. Define a vertex function 𝜑 : 𝑉 → R, mapping each vertex to a real value, and an edge weighting function 𝛾 : 𝐸 → R, assigning weights to edges. The graph Laplacian Δ, operating on 𝜑 for a vertex 𝑣, is given byEQUATIONwhere 𝑑 (𝑢, 𝑣) denotes the distance between vertices 𝑢 and 𝑣, and 𝛾 𝑢𝑣 represents the weight of the edge connecting 𝑢 and 𝑣.3.2.2 Spectral Distances. Consider two distinct, nonisomorphic simple graphs 𝐺 𝑠 and 𝐺 𝑡 , each with 𝑛 vertices. Let their Laplacian spectra be denoted asEQUATION)This metric quantifies the dissimilarity between the graphs based on their Laplacian spectra, facilitating a nuanced comparison of their structural properties. 
methodology: 
 Before delving into the details of our proposed methodology, we first outline the problem formulation and revisit the base video tube retrieval model and the teacher-student framework in Section 4.1, which forms the basis of our study. Subsequently, we introduce the Layered Feature Debiasing (LFD) module in Section 4.2 and the Pseudo Label Refinement (PLR) module in Section 4.3. Finally, we describe the overall training pipeline in Section 4.4.Given an untrimmed video 𝑉 = {𝑣 𝑡 } 𝑇 𝑡 =1 with 𝑇 frames and a textual query 𝑆 = {𝑠 𝑛 } 𝑁 𝑛=1 , the objective of the Video Tube Retrieval (VTR) task is to localize a spatial-temporal video tube 𝐵 = {𝑏 𝑡 } 𝑡 𝑒 𝑡 =𝑡 𝑠 described by the textual query 𝑆. Here, 𝑏 𝑡 denotes the bounding box in the 𝑡-th frame, and 𝑡 𝑠 and 𝑡 𝑒 are the start and end frames of the retrieved tube, respectively. In the unsupervised domain adaptation context, we are given a set of untrimmed videos 𝑉 𝑆 and language queries 𝑄 𝑆 in the source domain, and another set of untrimmed videos 𝑉 𝑇 and language queries 𝑄 𝑇 in the target domain. The videoquery pairs are labeled with the spatial-temporal tube 𝐵 𝑆 in the source domain but not in the target one. Under such circumstances, the main objective is to derive an effective VTR model on the target domain by fully exploiting labeled data in the source domain.State-of-the-art models for video tube retrieval are primarily based on the encoder-decoder paradigm. For example, STCAT [14] firstly utilizes two feature extractors to obtain both visual and textual features from the video frames and query sentence, respectively. It then models the video-text interactions through a spatial-temporal cross-modal encoder, which introduces a videolevel learnable token for the whole video to encode target object semantics, and a frame-level learnable token for each individual frame to represent the frame specific appearance. These tokens are further leveraged to produce a template for the target object by a template generator. Finally, the yielded template is treated as a query and fed into a decoder to aggregate features and predict the spatial-temporal tube. In this paper, we employ STCAT as the base video tube retrieval model. However, it is important to note that our module is not specifically designed for STACT, but can be universally applied to models based on the encoder-decoder architecture.Our teacher-student framework is composed of two architecturally identical models: target-only teacher model and cross-domain student model. The teacher model only takes the weakly-augmented video-query pairs from target domain while the student model takes strongly-augmented video-query pairs from both source and target domains. The student model is learned by standard gradient descent, and the teacher model is updated with the exponential moving average (EMA) of the weights from the student model. Thus, the teacher model can be considered as a temporal ensemble of multiple student models: the weights 𝜃 ′ 𝑡 of the teacher model at time step 𝑡 are derived from the EMA of the student model's successive weights 𝜃 𝑡 :EQUATIONwhere 𝛼 is a smoothing coefficient hyperparameter. Then we use the temporally ensembled teacher model to guide the student model's training in the target domain via pseudo labels.To generate precise and accurate pseudo labels for target domain video-query pairs, we feed the video-query pairs with weak augmentation to the teacher model and pairs with strong augmentation the student model. Specifically, weak augmentations include random horizontal flipping and cropping, while strong augmentations additionally comprise random color jittering, grayscaling, and Gaussian blurring. We further utilize the random temporal shift perturbation [32] on the encoded features within the student branch. Concretely, a subset of feature channels is selected at random; half of these are shifted forward in time, while the remaining half are shifted backward in time. In such ways, the teacher model's predictions can be more accurate than those of the student model, allowing the student model to learn from pseudo labels generated by the teacher model. graph that preserves the local feature variance among different frames and projects these graphs from both the source and target domains into a shared eigenspace for alignment according to their respective eigenvalues. where 𝐿 adv is defined in Equation (2), 𝜆 vis , 𝜆 txt , 𝜆 frame and 𝜆 video are hyper-parameters that weight the importance of each loss component in the overall adversarial alignment objective.Alignment. We meticulously construct selfcorrelation graphs for the source and target videos independently to model the relations between different frames in each video. Specifically, given the cross-modal decoder outputs of the source video and target video, denoted as 𝑂 𝑠 and 𝑂 𝑡 respectively, our objective is to construct undirected and weighted graphs 𝐺 𝑠 = (𝑉 𝑠 , 𝐸 𝑠 ) for the source domain videos and 𝐺 𝑡 = (𝑉 𝑡 , 𝐸 𝑡 ) for the target domain videos. In these graphs, each vertex 𝑣 𝑖 belonging to 𝑉 𝑠 or 𝑉 𝑡 is represented by a corresponding output vector 𝑂 𝑖 in the output sets 𝑂 𝑠 or 𝑂 𝑡 , respectively. The weighted edge 𝑒 𝑖,𝑗 is formulated as the relation between vertices 𝑣 𝑖 and 𝑣 𝑗 , quantified using a metric function 𝑠 between their feature representations 𝑂 𝑖 and 𝑂 𝑗 , based on the Gaussian similarity metric:EQUATION)where 𝜎 is the standard deviation of the Gaussian distribution and ∥ • ∥ represents the Euclidean norm. For a simple undirected graph with a finite number of vertices and edges, the definition of graph laplacians (Section 3.2.1) is just identical to the Laplacian matrix. We opt for the random walk Laplacian matrix 𝑀 using the formula:EQUATION)where 𝐴 is the adjacency matrix of the graph, 𝐼 is the identity matrix and 𝐷 is degree matrix of 𝐴. With the adjacency matrices 𝐴 𝑠 and 𝐴 𝑡 of the source and target domain graphs, we can obtain the Laplacian matrices 𝑀 𝑠 and 𝑀 𝑡 , and their eigenvalues Λ 𝑠 and Λ 𝑡 , respectively. Following the definition of spectral distances (Section 3.2.2), we can calculate the spectral distances between source domain graph 𝐺 𝑠 and target domain graph 𝐺 𝑡 , and thus the graph based alignment loss 𝐿 gba is defined using the function 𝜎 (•) detailed in Equation ( 4):EQUATIONIn this section, we first describe the Threshold Filtering (Section 4.3.1) which calculates the confidence of each pseudo label for the target domain samples and filters out uncertain items. Then, we design the Temporal Extension (Section 4.3.2) to extend the grounded temporal boundary in the pseudo labels and introduce the training weights for the bounding boxes.Filtering. The pseudo labels generated by the sourcedomain teacher model tend to ground the video clips/video objects widely existing in the source domain samples. However, these pseudo labels may not be suitable for the target domain samples due to the domain gap and even fail to capture the essential video semantics corresponding to the natural language query. To address this issue, we filter out a portion of the low-quality pseudo labels based on the confidence threshold. Specifically, for each frame, while predicting a bounding box, we also predict an "actionness" score, which indicates the confidence level of the match between the bounding box area in that frame and the associated text. We calculate the average actionness score for all frames within the predicted temporal boundaries, and only videos with an average score exceeding the threshold are included in the training with their pseudo labels. Formally, we define the filter function 𝐹 (•) as:𝐹 (𝐴, 𝑠, 𝑒) = 1 if 𝑒 𝑡 =𝑠 𝐴 𝑡𝑒 -𝑠+1 > 𝜏 0 otherwise (10) where 𝐴 𝑡 represents the actionness score at frame 𝑡, 𝑠 and 𝑒 are the predicted start and end of the temporal boundaries, and 𝜏 is the predefined confidence level above which the video is considered suitable for training inclusion.The VTR model predicts the probability of each frame being the start and end boundaries of the pseudo tube (i.e., the tube-shaped pseudo labels predicted by the teacher model), denoted as 𝜏 𝑠 and 𝜏 𝑒 , respectively. The start and end times of the pseudo tube, 𝑡 𝑠 and 𝑡 𝑒 , are determined by the formula:EQUATION)This formula selects the maximum from the joint start and end probability distribution (𝜏 𝑠 , 𝜏 𝑒 ), excluding invalid combinations where 𝑡 𝑒 ≤ 𝑡 𝑠 . The pseudo tube {𝑏 𝑡 } 𝑡 𝑒 𝑡 =𝑡 𝑠 is formed from bounding boxes 𝑏 𝑡 within the selected start and end times 𝑡 𝑠 and 𝑡 𝑒 . For target domain, even when the essential video semantics are grounded, there may still be discrepancies in the temporal boundaries of the pseudo labels. As a result, the pseudo labels may exclude the correct bounding boxes of frames within the correct temporal boundaries. This leads to a decrease in the availability of accurate training labels for the student model, which in turn negatively impacts its accuracy. In addition, the pseudo labels may include nonsensical bounding boxes of frames that fall outside the correct temporal boundaries. These nonsensical bounding boxes in the pseudo labels often encompass visual objects not aligned with the natural language queries.We introduce two strategies to extend the predicted temporal interval and refine the calculation of spatial loss via a frame weighting scheme throughout the entire expanded interval, based on the previously obtained actionness scores {𝐴 𝑡 } 𝑇 𝑡 =1 . First, we employ temporal non-maximum suppression (NMS) on (𝜏 𝑠 , 𝜏 𝑒 ) to find the second-best starting and ending frame pair 𝑡 ′ 𝑠 and 𝑡 ′ 𝑒 , which satisfies the condition that the temporal Intersection over Union (IoU) with (𝑡 𝑠 , 𝑡 𝑒 ) is below threshold 𝜏 𝑖𝑜𝑢 , and the average actionness score exceeds threshold 𝜏 𝑎𝑐𝑡 . If such a pair does not exist, the attempt is abandoned. Moreover, we bidirectionally expand a new interval (𝑡 ′′ 𝑠 , 𝑡 ′′ 𝑒 ) around the frame 𝑡 𝑐 with the highest confidence score (opting for the most central one in cases of multiple candidates) such that the difference between the mean confidence score inside the new interval and outside is maximized:EQUATIONThe final temporal boundary of pseudo tube, (𝑡 ext 𝑠 , 𝑡 ext 𝑒 ), is derived by unifying the original interval with these expanded ones:EQUATIONThe extension allows more spatial bounding boxes to participate in training. We use the normalized actionness score as a weighting factor for the spatial loss across the entire extended interval.The overall objective of our model is formulated as follows:EQUATIONwhere 𝐿 sup is the supervised loss in the base VTR model, 𝐿 adv is the adversarial feature alignment loss defined in Equation ( 6), 𝐿 gba is the graph based alignment loss defined in Equation ( 9) and 𝐿 unsup is the unsupervised loss that computed on pseudo labels that passed through Equation ( 10) and were processed by Equation (13) . 𝜆 gba and 𝜆 unsup serve as the weighting hyper-parameters.Prior to the domain adaptation process, the model undergoes initial training with 𝐿 sup using the annotated source domain data. This preliminary phase is designed to equip the model with the source domain knowledge, thereby facilitating the generation of informative pseudo labels. Both the teacher and student models are initialized with parameters acquired from this initial training stage. During the domain adaptation stage, we periodically reset the training of the student model's visual encoder, textual encoder and cross-modal encoder to their states trained only on the source domain, as proposed by [37] . After re-initialization, these retrained modules contain no knowledge of the target domain. Subsequently, the enhanced cross-modal decoder component of the student model, along with the teacher model, promotes convergence towards a more favorable optimum. 
experiments 5.1 datasets: 
 We evaluated our model's performance using two domain pairs: Indoor-to-Outdoor Video Tube Retrieval (I2O-VTR) and Real-to-Movie Video Tube Retrieval (R2M-VTR). The I2O-VTR dataset is derived from the comprehensive VidSTG dataset [36] , which consists of 6,924 videos annotated for video tube retrieval task, partitioned into training, validation, and test sets with 5,563, 618, and 743 videos, respectively. To facilitate indoor-to-outdoor domain adaptation studies, the VidSTG dataset was manually segmented into two subsets: indoor and outdoor scenes [2] . The segmentation resulted in 3,386 videos designated as indoor (with a split of 3,386/358/429 for training, validation, and test sets, respectively) and 2,177 videos categorized as outdoor (with a split of 2,177/260/314). The indoor domain encapsulates environments defined by limited spatial extents, while the outdoor domain encompasses scenes characterized by unbounded spatial areas. The R2M-VTR dataset incorporates the HC-STVG dataset [25] , consisting of 5,660 videos sourced from cinematic productions, along with the entire VidSTG dataset, which encompasses a broad spectrum of real-life scenarios. The HC-STVG dataset is segmented into training and test sets, containing 4,500 and 1,160 videos, respectively. This combination of datasets from cinematic and real-life sources establishes the groundwork for realto-movie domain adaptation analysis. We adhere to the original dataset splits provided for both the VidSTG and HC-STVG datasets to maintain consistency and integrity in our evaluation.To evaluate video tube retrieval, we follow the previous work [36] and define 𝑣𝐼𝑜𝑈 = 1|𝑆 𝑢 | 𝑡 ∈𝑆 𝑖 𝐼𝑜𝑈 (𝑏 𝑡 , 𝑏 ′ 𝑡 ), where 𝑆 𝑢 and 𝑆 𝑖 denote the sets of frames at the union and the intersection of the ground truth and the predicted timestamps, respectively. The terms 𝑏 𝑡 and 𝑏 ′ 𝑡 represent the ground truth and the predicted bounding boxes at time 𝑡, respectively. We use 𝑚_𝑣𝐼𝑜𝑈 as the metric, which is the average of 𝑣𝐼𝑜𝑈 . We also use 𝑣𝐼𝑜𝑈 @𝑅, which is the proportion of samples whose 𝑣𝐼𝑜𝑈 > 𝑅. To isolate the evaluation of temporal localization, we employ the metric 𝑚_𝑡𝐼𝑜𝑈 (mean of 𝑡𝐼𝑜𝑈 ), which is defined as 𝑡𝐼𝑜𝑈 =|𝑆 𝑖 | |𝑆 𝑢 | .We configure the batch size at 8 and the learning rates for the visual encoder ResNet-101 [12] and the textual encoder RoBERTa [21] at 10 -5 , while setting the learning rate for other modules at 10 -4 . The AdamW optimizer is employed for training with a weight decay of 10 -4 . Our training spans 10 epochs on the I2O-VTR dataset and drops the learning rate by 0.1 after 8 epochs. For the R2M-VTR dataset, the training duration extends to 90 epochs, with a similar learning rate reduction enacted after 50 epochs. To enhance computational efficiency, the input video was uniformly down-sampled to 64 frames and a resolution of 224. The final object tubes are generated through linear interpolation of the predicted bounding boxes in the sampled frames. We empirically determine the hyper-parameters, setting 𝜆 unsup = 1, 𝜆 vis = 𝜆 txt = 0.3, 𝜆 frame = 𝜆 video = 1 and 𝜆 gba = 100 in our experiments.Table 1 : Comparison with baselines on the I2O-VTR dataset. A higher score denotes superior performance, and we highlight the best and second best scores.Declarative Sentences Interrogative Sentences m_tIoU m_vIoU vIoU@0.3 vIoU@0.5 m_vIoU m_tIoU vIoU@0.3 vIoU@0. To enable a thorough evaluation of model efficacy across domains, we augment the capabilities of the state-of-the-art video tube retrieval model, TubeDETR [34] and STCAT [14] . Specifically, we first train the model on the source domain dataset and generate pseudo labels for the target domain dataset, which then serve as supervision for training the model within the target domain dataset.For the remaining experiments, we leverage the STCAT model as our base VTR model. Furthermore, we incorporate advanced domain adaptation strategies into the STCAT model, including MDD [16] and CST [20] , in an attempt to address the inherent challenges associated with UDA-VTR task. The comparison results between our model and baselines are shown in Table 1 and Table 3 . Beyond components specific to unsupervised domain adaptation, the configuration across the five models is identical.Our model significantly outperforms all baselines across two datasets. When examining four methods utilizing the same base video tube retrieval model, our approach boosts the m_vIoU metric from 14.27/14.83/16.77 to 23.22 in the I2O-VTR dataset, and from 12.57/12.94/13.91 to 25.72 in the R2M-VTR dataset. The limitations of the state-of-the-art STCAT model stem from its design, which is not tailored for the unsupervised domain adaptation scenario, leading to a lack of nuanced understanding and adaptation to domainspecific challenges. Traditional domain adaptation strategies, such as MDD and CST, fall short of achieving high performance due to the absence of targeted adaptations for the complex UDA-VTR task. Our approach leverages the teacher-student framework, incorporating specific module enhancements to tackle the challenges of unsupervised domain adaptation in the VTR task, achieving substantial improvements.Our model comprises of two core modules: the Layered Feature Debiasing (LFD) and the Pseudo Label Refinement (PLR). Ablation studies conducted on both LFD and PLR, with results detailed in Table 2 and Table 4 for the I2O-VTR and R2M-VTR datasets respectively, demonstrate that the inclusion of either module significantly improves model performance, highlighting their individual effectiveness. When both modules are employed together, the observed synergistic effect surpasses the performance gains achieved by each module in isolation, emphasizing the indispensability and combined efficiency of the LFD and PLR. This synergy underscores the complementary nature of our modules, which can function together and be progressively optimized in a mutual manner. We randomly select several samples from the target domain dataset for case studies and visualize the processes of handling pseudo labels, as shown in Figure 3 . From these case studies, we can observe:In case (a), the model fails to capture the essential video semantics that correspond to the natural language query. It predicts bounding boxes that frequently switch among multiple humans, with a lower average confidence level throughout the predicted temporal interval. These predictions will be filtered out by our module and no longer participate in the calculation of unsupervised loss in the current iteration. In cases (b) and (c), the model shows better learning of spatial objects, but its understanding of temporal actions is weaker. This leads to predictions with a higher average confidence level but with temporal intervals that are either too short or significantly shifted. Our module will temporally extend the pseudo labels in these cases, allowing more spatial boxes to participate in the calculation of unsupervised loss. It is observed that the temporally extended intervals are closer to the ground truth, which aids the model in better learning action information.We also show the t-SNE [27] visualizations of the features before and after the application of our model in Figure 4 . It is evident that there is a significant distribution gap between the source and target domains in Figure 4 (a). Our model narrows the domain shift, leading to closely intertwined feature distributions from both domains in Figure 4 (b), demonstrating the effectiveness of our approach in mitigating domain discrepancies. 
conclusion: 
 In conclusion, the HEART model significantly advances the field of Video Tube Retrieval (VTR) with its approach to Unsupervised Domain Adaptation (UDA). It effectively resolves major domain adaptation challenges, including source domain bias and inaccuracies in pseudo labels for target domain samples. By integrating the Layered Feature Debiasing module, HEART substantially reduces source domain bias, which enhances the accuracy of target domain feature representations. Concurrently, the Pseudo Label Refinement module elevates the quality and relevance of pseudo labels, ensuring more precise localization of video tubes in the target domain. Our extensive experiments validate that HEART not only outperforms existing state-of-the-art methods and establishes a new benchmark for future research in domain-adaptive VTR, but also highlights the potential of hierarchical debiasing and label correction strategies to effectively tackle the cross-domain challenges.  
: 
 sentences, is a fundamental and vital task in the visual-language understanding field [24, 25, 36] . There are mainly two stages in the feature-extracted process for most existing models [24, 36] : (1) Global Feature Extraction. Given the video-language pair, the VTR model extracts the cross-modal fusion features (global features) from the multi-modal input. (2) Local Feature Retrieval. With the global features, the VTR model further retrieves the local features of the spatial-temporal semantics described by the natural language query for each video frame. Subsequently, the extracted local features undergo further processing through simple linear layers to predict the final spatial-temporal tube. Despite the significant progress, the performance of existing VTR models trained in the specific domain (source domain) drops sharply in another different domain (target domain) due to the domain gap. Intuitively, fine-tuning the VTR models in the target domain with manual annotations may facilitate the model's acquisition of domain-specific knowledge. Unfortunately, it is tremendously time-consuming and labor-intensive to annotate the dense labels of bounding boxes frame-by-frame and temporal boundaries for each video-language sample of the target domain.To alleviate the unbearable demand for dense annotations of the target domain, we introduce the learning strategy, Unsupervised Domain Adaptation (UDA), to the VTR task, which transfers the source domain knowledge to the target domain without the target domain manual annotations. An intuitive solution for the UDA-VTR task is applying the source-domain VTR model (teacher model) to initialize the parameters of the target-domain VTR model, and then generate the pseudo labels of the target domain to fine-tune the target-domain VTR model (student model). However, the presence of the domain gap gives rise to two critical issues that significantly impair the performance of the student model: (i) Multi-level Feature Bias. Initializing the student model with the teacher model parameters may introduce the source domain bias from the teacher model into the feature-extraction module of the student model, resulting in the student model producing global and local features influenced by the source domain bias. Due to the domain gap, the source domain bias is not applicable to the target domain, which reduces the precision of the features extracted by the student model, thereby impacting the final prediction accuracy based on those features. Thus, how to alleviate the source domain bias in both the global and local features extracted by the student model is an essential problem. Furthermore, the local features capture the visual characteristics of each video frame, which is vital for generating distinct predictions for individual frames. However, such visual characteristics are intertwined with the information of the source domain bias. Therefore, in addition to eliminating the source domain bias of the global and local features, how to protect each frame's visual characteristics in the local features from being impacted in this elimination process is another problem worth considering. (ii) Pseudo Label Noise. The pseudo labels generated by the source-domain teacher model tend to ground the video clips/video objects widely existing in the source domain samples, which may not be suitable for the target domain samples due to the domain gap. There are mainly two types of failed grounding in pseudo labels: 1) Some pseudo labels fail to capture the essential video semantics corresponding to the natural language query. Such pseudo labels with excessive deviation may not be suitable for the training of the student model and should be filtered out. 2) Even if grounding the essential video semantics, there may still be discrepancies in the grounded temporal boundaries of the pseudo labels. As a result, the pseudo labels may exclude the correct bounding boxes of frames within the correct temporal boundaries. This leads to a decrease in the availability of accurate training labels for the student model, which in turn negatively impacts its accuracy. In addition, the pseudo labels may include nonsensical bounding boxes of frames that fall outside the correct temporal boundaries. These nonsensical bounding boxes in the pseudo labels often encompass visual objects not aligned with the natural language queries, and in some cases, even ground background information. Thus, such pseudo labels with relatively minor discrepancies in the grounded temporal boundaries may significantly disrupt the spatial grounding learning process of the target-domain student model.To address the aforementioned issues, we propose the unsupervised domain adaptation model via Hierarchical dEbiAsing and noisy correction for cRoss-domain video Tube retrieval (HEART). Based on the teacher-student framework, our HEART model contains two carefully designed modules: (i) Layered Feature Debiasing. To enhance the student model's extraction of global features, we introduce adversarial learning as a training mechanism, which enables the student model to capture invariant information across different domains. In addition, for the extracted local features, we establish the relation graph to preserve their variance among different frames. Subsequently, we project the local feature graphs of both the source domain and the target domain into the shared eigenspace and align them according to their respective eigenvalues. (ii) Pseudo Label Refinement. Firstly, we calculate the confidence of each pseudo label for the target domain samples and filter out uncertain items. Secondly, we extend the grounded temporal boundary in the pseudo labels. Specifically, we identify the frames near the grounded temporal boundary and incorporate the bounding boxes predicted by the source-domain teacher model for these frames into the pseudo labels. Then, we introduce the training weights for the bounding boxes, mitigating the adverse effects of low-quality ones.Our contributions can be summarized as:• To the best of our knowledge, we take the early exploration of the unsupervised domain adaptation for the video tube retrieval task. Toward this issue, we propose a novel model, HEART, based on the teacher-student framework. • We utilize adversarial learning and graph-based alignment to alleviate the source domain bias in the target domain features while retaining the crucial visual characteristics.In addition, we refine the pseudo labels generated for the student model with excessive or minor deviation, respectively.• Extensive experiments prove the effectiveness of our HEART model by surpassing the state-of-the-arts by a large margin. 
 
 Appendix:  


Decision: This paper proposes a teacher-student framework to address the unsupervised domain adaptation problem in the Video Tube Retrieval task, which contains a layered feature debiasing module and a pseudo label refinement module. Extensive experiments show the effectiveness of the proposed method. In the rebuttal, the questions and concerns of reviewers have been addressed and all reviewers reach a consensus on the acceptance of this submission. 

Quality: It is interesting to explore the unsupervised domain adaptation for the video tube retrieval task. The motivation of this paper is clear and reasonable. 

Clarity: The paper is well-written and easy to follow.

Originality: The novelty of this work is sufficient for acceptance.

Significance: The proposed framework has wide applications, especially in scenarios that suffer from the domain gap.

Reviews: review: paper_summary: paper_summary: This paper tackles the UDA problem in the VTR task, which enables the knowledge transfer from the labeled source domain to the unlabeled target domain. The porposed method Hierarchical dEbiAsing and noisy correction for cRoss-domain video Tube retrieval (HEART) resolves source domain bias and inaccuracies in pseudo labels for target domain samples.
strength_weakness: strength_weakness: 1. The proposed method HEART is well designed and can alleviate the problems in VTR tasks with the UDA settings.
2. The experiments are abundant to support the contributions of this work.
limitations: limitations: 1. What was the basis for determining the hyperparameters in Section 5.5, and does it apply to other datasets or scenarios? The discussion about the impact of these hyperparameters would be helpful.
2. The color in Table 1 and 3 can be switched to another format(like texts in bold).
paper_summary: paper_summary: This paper addresses cross-domain video tube retrieval and develops a HEART approach. The proposed HEART method draws inspiration from the unsupervised domain adaptation and includes two sophisticated modules: (i) Layered Feature Debiasing; and (ii) Pseudo Label Refinement. Furthermore, the HEART is a general strategy that can be incorporated into various encoder-decoder architectures. Extensive experiments verify the effectiveness of the HEART.
strength_weakness: strength_weakness: 1. This work takes the early exploration of the unsupervised domain adaptation for the video tube retrieval task, which is an interesting and promising direction.

2. HEART is a general strategy that can be incorporated into various encoder-decoder architectures.

3. The authors open-source their codes, which is good.
limitations: limitations: 1. In Figure 2, the abbreviation EMA could be written out as Exponential Moving Average for the sake of readability.

2. In Equation (14), should the term $L\_\{adv\}$ be $L\_\{afa\}$?

3. In the main text, the term $L\_\{unsup\}$  lacks a specific formula, which makes it not easy to understand.

4. The method is constrained by the requirement to configure numerous hyper-parameters, thereby reducing its flexibility. Additionally, the authors have omitted conducting experiments to assess the sensitivity of the parameters.

5. In Equation (10) of the appendix, what are the settings for the additional five hyperparameters introduced here? If $L\_\{sup\}$ is the loss function of STACT [1], I think it is better to cite [1] in the appendix.

6. HEART is a general strategy that can be incorporated into various encoder-decoder architectures. However, in the experiments, the authors solely adopt the STACT architecture. To demonstrate the effectiveness and generality of HEART, it would be beneficial to include experimentation with another architecture.

 [1] Yang Jin, Zehuan Yuan, Yadong Mu, et al. 2022. Embracing consistency: A one-stage approach for spatio-temporal video grounding. Advances in Neural Information Processing Systems 35 (2022), 29192–29204.
paper_summary: paper_summary: This paper introduce the unsupervised domain adaptation into video tube retrieval, enabling the knowledge transfer from the labeled source domain to the unlabeled target domain.
strength_weakness: strength_weakness: 1. This paper proposed HEART method to solve domain shift problem including layered feature debiasing, pseudo label refinement. 
2. This paper exploit UDA for video tube retrieval task via teacher-student framework
3. Adversarial learning and graph alignment are utilized to alleviate the source domain bias
limitations: limitations: 1. In table1 and table 3, the author only compare their method with TubeDETR, have they tried other popular method?

2. In Sec.5.3, the author use ResNet-101 and RoBERTa as image and text encoder, respectively, have they tried multi-modal model like CLIP?

3. In the graph establishment, it seems that one frame is treated as a node, so the node number in graph is fixed or dynamic? How does the author deal with this problem.

4. In Eq.(7), the author use Gaussian kernel based distance to generate correlation, have they tried any other distance, e.g. cosine similarity?
paper_summary: paper_summary: The paper introduces the HEART model to address the domain adaptation challenge in Video Tube Retrieval (VTR) tasks. HEART uses a combination of Layered Feature Debiasing and Pseudo Label Refinement to mitigate source domain bias and enhance the accuracy of pseudo labels used for training in target domains. The results show that HEART outperforms some existing methods, demonstrating its effectiveness in providing more domain-invariant features and reliable pseudo labels.
strength_weakness: strength_weakness: The proposed method utilises adversarial learning to improve the robustness of the model and learn domain-invariant features.

The components used in the model are well-researched and have proven their effectiveness, lending credibility to the proposed method.
limitations: limitations: 1. The domain gap between the source and target domains is not significant, raising concerns about the importance of this research topic. 

2. The model training involves numerous hyperparameters (e.g., for loss contribution, confidence threshold, etc.) without providing justifications or conducting ablation studies to determine optimal settings. This lack of detailed parameter analysis could hinder the application of this method to other datasets or tasks.

3. The proposed method is complex without an analysis of model size and running time compared with other benchmarks or the baseline method (STCAT).