Improving Online Source-Free Domain Adaptation for Object Detection by Unsupervised Data Acquisition

Authors: xiangyu  shi
yanyuan  qiao
qi  wu
lingqiao  liu
feras  dayoub
Venue: thecvf_com_ECCV_2024_Workshop_ROAM
Published: 2024-07-23 17:00:05

Abstract: Effective object detection in autonomous vehicles is challenged by deployment in diverse and unfamiliar environments. Online Source-Free Domain Adaptation (O-SFDA) offers model adaptation using a stream of unlabeled data from a target domain in an online manner. However, not all captured frames contain information beneficial for adaptation, especially in the presence of redundant data and class imbalance issues. This paper introduces a novel approach to enhance O-SFDA for adaptive object detection through unsupervised data acquisition. Our methodology prioritizes the most informative unlabeled frames for inclusion in the online training process. Empirical evaluation on a real-world dataset reveals that our method outperforms existing state-of-the-art O-SFDA techniques, demonstrating the viability of unsupervised data acquisition for improving the adaptive object detector.

Main paper: 
 introduction: 
 Autonomous vehicles operating in diverse and constantly changing environments require robust object detection systems. However, the performance of trained detectors significantly drops when they encounter unfamiliar scenarios. The real world is rarely predictable, with environments varying widely in their characteristics, leading to domain shifts that can severely compromise the effectiveness of pre-trained object detection models. This is a challenging task and many existing approaches focus on addressing this issue [1, 3, 15, 18, 23, 29] .The challenge lies in equipping vehicles with the ability to adapt to new domains "on-the-fly", without the need for manually labelling new data or undergoing an exhaustive re-training process. Online source-free domain adaptation (O-SFDA) [20, 23, 27] offers a promising avenue to address this challenge by transferring knowledge from a source domain to a target domain in an online manner. This means that the adaptation occurs concurrently with deployment in the target domain, contrasting with traditional methods that may require source data batches or offline processing [8, 14, 24, 28] . Current O-SFDA methods for object detection, like those described by [23] , utilize a transformer-style memory bank to store and use information from observed frames for future adaptation. However, these methods face limitations due to high computational costs and overreliance on redundant data. Adapting using every frame can result in severe class imbalances, such as more frequent detection of common objects like cars and people, and less frequent detection of rare objects like motorbikes, leading to poorer detection performance for rare classes.To address these issues, we employ an unsupervised data acquisition strategy based on incremental online clustering to dynamically identify and integrate both the most informative unlabeled frames and those containing rare categories. We augment this strategy by optimizing a dual architecture consisting of a teacher model and a student model based on the Mean Teacher framework [21] . Our method aims to reduce computational resource usage while enhancing object detection adaptivity in previously unencountered deployment environments.We conducted extensive experiments across four datasets, Cityscapes [2] , Sim10k [6] , Cityscapes-Foggy [16] , and SHIFT [19] with three domain adaptation processes that include real-world scenarios with varying complexities. Our results demonstrate that our method consistently outperforms existing state-of-the-art O-SFDA techniques in object detection [23] for 2.3 mAP under Cityscapes to Cityscapes-Foggy, 9.0 mAP under Sim10k to Cityscapes, and 9.3 mAP under SHIFT to Cityscapes.In summary, we propose a novel approach to improve O-SFDA for object detection through an unsupervised data acquisition methodology based on incremental online clustering. This data acquisition strategy selects the most informative and category-rare frames for online training. To solve the overlapping frame and class imbalance issues, our proposed data acquisition contains two stages, Acquisition Unsim Frame (AUF) and Acquisition Rare Category (ARC) where AUF aims at acquiring the dissimilar frame, and ARC aims to acquire frame that contains the rare category. We evaluate our model based on three domain adaptation processes, and the result surpasses the current state-of-the-art (SOTA) methods. 1  
related work: 
 Unsupervised Domain Adaptation for Object Detection. Unsupervised Domain Adaptation (UDA) [11, 30] aims to mitigate the domain-shift problem without requiring labeled target data, by aligning source and target distributions. The field has seen significant contributions specifically in object detection [1, 15, 24, 26, 31, 32] . For instance, Chen et al. [1] pioneered this space by proposing domain adaptation via both image-level and instance-level alignments. Later, Zhu et al. [31] focused on the alignment of congruent regions between source and target domains. Xu et al. [26] presented a framework that involves categorical regularization at the image level and consistency regularization for categories to address complex challenges in this domain. While these approaches have yielded impressive results, they inherently rely on the availability of source data during adaptation, limiting their applicability in real-world scenarios such as autonomous driving where source data may not always be accessible.Source-free Domain Adaptation for Object Detection. In environments where source data is not readily available, such as in autonomous vehicles, the specialized domain of Source-Free Unsupervised Domain Adaptation (SFDA) has garnered interest [12, 17] . SFDA uniquely operates without the source data during the adaptation phase, and typically focusing on offline model adaptation [5, 7, 10, 21, 22] . The significance of online adaptation in such settings is crucial. Vibashan et al. [23] ventured into this area by proposing a cross-attention transformer-based model for both online and offline SFDA in object detection. They extended the Mean Teacher framework with a MemXformer that pairs features between the teacher and student models [21] . However, despite these advancements, existing methods have not addressed the importance of keyframe selection for adaptation. Our work fills this gap by emphasizing online adaptation in SFDA for object detection, with a particular focus on selective adaptation through keyframe selection. This approach enhances both the efficiency of the adaptation process and the overall adaptation performance. 
problem formulation: 
 In the domain of O-SFDA for object detection, we consider an online scenario where access to the original source domain dataset as D s = {x s , y s }, is restricted, and only a pre-trained model Θ s on D s is available. The aim is to adapt Θ s to a target domain D t , which is represented by a stream of unlabeled data {x t }. The model is updated using only frames from D t that are deemed adaptation-worthy.Our method is built upon the framework of online adaptation of the Mean Teacher model [21] , specialized for the Faster-RCNN object detector [13] . The Mean Teacher framework employs two models: a student model (Θ S ) and a teacher model (Θ T ), both initialized using the pre-trained source model Θ s . For each incoming frame f t , we employ two forms of data augmentation: the weak augmentation A w (•) and the strong augmentation A s (•). The weakly augmented frame f w t is fed through the teacher model to generate pseudo-labels, denoted as ŷT , while the strongly augmented frame f s t is processed by the student model to predict labels y S . To improve the model's reliability, we consider only pseudolabels ŷT that possess a confidence score greater than 0.9, as supported by prior Fig. 1 : For streaming frames, ft, the data acquisition system continuously evaluates whether to retain the current frame using a similarity measure. Once ft is detected as "key frame", ft will be used to update the model. The condition x refers to the if statement used to determine whether the current frame contains the rare category after the warm-up stage.research [23] . Given this criterion, the loss function within the Faster-RCNN framework can be defined as:EQUATIONWe present our clustering method to choose frames for adapting the model, especially when data from the source domain is not available. Our data acquisition approach targets two types of data: dissimilar frames and frames containing the rare category. To do this, we divide the process into two stages: the first stage focuses on identifying dissimilar frames by Acquisition Unsim Frame (AUF), and the second stage targets the rare category by Acquisition Rare Category (ARC). Our overall framework is illustrated in Figure 1 .Acquisition Unsim Frame (AUF) Leveraging the image encoder E from the static teacher model Θ T from the initial teacher parameter, the clustering starts by initializing the first frame f 0 as the centroid p 11 of the first cluster m 1 . Subsequent frames produce embeddings e t , which are compared against existing centroids {p 11 , . . . , p 1n } using cosine similarity. Based on a predefined similarity threshold γ, a frame is identified as a keyframe to start a new cluster or is added to the closest existing cluster, m c , which then updates its centroid by ave(m c ).The AUF similarity score, Score max1 is calculated by:EQUATIONFrames that meet Score max1 < γ are considered as keyframes for model updates.Acquisition Rare Category (ARC) In an online setting, where the model adapts to each current frame, it tends to improve at recognizing frequent objects while performing worse on rarer ones. This issue is known as class imbalance. We addressed this by using the ARC approach, which employs an additional clustering mechanism to give frames with the rare category, c r a second opportunity for data acquisition. We aggregate the ŷT{1,...,t} and identify the category with the smallest aggregated sum as the rare category:c r = arg min c ( t i=0ŷTi,c for each category c)Specifically, if Score max1 > γ and c r ∈ ŷTt , e t will be processed into the ARC for a second selection. Similar to the operation in AUF, e t is compared against existing ARC centroids {p 21 , . . . , p 2n } using cosine similarity:EQUATIONFrames meeting the criterion Score max2 < γ are considered as keyframes and trigger model updates. This scheme ensures efficient and effective model adaptation by focusing on frames significantly differ from previous instances. To ensure the stability of the rare category identified, we propose the implementation of a warm-up phase. During the warm-up stage, we only activate AUF and collect the distribution of pseudo labels, ŷT{1,...,t} to identify which category is rare for ARC. This phase is designed to stabilise the rare category prior to activating the ARC mechanism. Notably, the ARC cluster operates independently of the AUF cluster.For the optimization of the teacher model and student model, we first use the ŷT to supervise the student model for parameter update. After the student model is updated, we do an Exponential Moving Average (EMA) to update the teacher model, which transfers the knowledge from the student model to the teacher model. Specifically, to update the student model, we first apply strong data augmentation to produce f s t upon identifying a current frame f t as a keyframe. This augmented frame is then processed by the student model to predict labels y S . Given the absence of true labels in the target domain, we employ pseudolabels ŷT generated by the teacher model to adapt the student model parameters Θ S . A Kullback-Leibler divergence loss [23] L S-T is also applied to align the Θ S classification results, E CLS S with the Θ T classification results, E CLS T , based on teacher model's proposal, as follows:EQUATIONThe final loss function L for the student model's update is as follows:EQUATIONAfter the student model has been updated by minimizing L, we do EMA to update the teacher model. To update the teacher model's weights (Θ T ) using Θ S , we use two key parameters: α 1 for the adaptation phase and α 2 for the final update. Specifically, α 1 is used in the iterative update during the online process, while α 2 is employed for the final model update:EQUATIONEQUATION4 Experiments 
datasets: 
 We evaluate our model using four rigorously selected datasets: Cityscapes [2] , Cityscapes-Foggy [16] , Sim10k [6] , and SHIFT [19] .-Cityscapes: Geared towards urban street environments, Cityscapes consists of 2,975 training images and 500 validation images. An additional sequence of 106,917 images from Frankfurt allows for comprehensive evaluations in online sequential adaptation scenarios. We excluded the labelled 267 frames from the Frankfurt sequence to evaluate our model's performance in the target domain. The performance on these 267 frames is reported as "mAP Frankfurt" in the results section. -Cityscapes-Foggy: Cityscapes-Foggy augments the original Cityscapes set with simulated fog, making it an excellent benchmark for evaluating adaptation to adverse weather conditions. -Sim10k: Originating from the video game Grand Theft Auto V, this dataset comprises 10,000 training frames and tests our model's ability to adapt to substantially different source domains. -SHIFT: Created in a simulator, SHIFT encompasses a variety of weather conditions, providing an ideal landscape for assessing the robustness of our model across diverse environments. Our implementation builds upon the Faster-RCNN framework [13] with a ResNet50 backbone [4] , following the baseline established by MemCLR [23] . The ResNet50 model is pre-trained on ImageNet [9] . We employ a batch size of 1 and run the experiment for 1 epoch to ensure online adaptation capabilities. The student model relies heavily on the teacher's predictions for adaptation. To guarantee robustness and keep the same setting with Baseline [23] , we only consider teachergenerated pseudo labels with a confidence score exceeding 0.9 as Ground Truth for guiding the student model. The Exponential Moving Average (EMA) parameter is set to 0.996 for α 1 and 0.9 for α 2 . The learning rate is set to 0.001, and the warm-up learning rate is set to 0.0001. The optimization is performed using the SGD optimizer. Our method for acquiring data involves extracting features from the Fix-Teacher model, using a similarity threshold, γ set at 0.975 for both AUF and ARC. The warm-up period was adjusted to suit different environments. To build a category occurrence distribution, we run the warm-up until the ratio of the rarest category to the most popular reaches 0.3% when the total pseudo label counts more than 10000.Online Sequential Adaptation: Our primary objective is an online adaptation for autonomous vehicles navigating through sequences of overlapping frames. We employ datasets SHIFT, Cityscapes and Sim10k to assess this feature. Detailed evaluation setups include: (1) Sim10k to Seq-Cityscapes with car category only: Utilizing Sim10k as the source domain, we evaluate our model's adaptability using the sequential Frankfurt set of the Cityscapes dataset. Since there is only one category, we only activate AUF. (2) SHIFT to Seq-Cityscapes with shared categories: In this experiment, SHIFT serves as the source domain to evaluate the model on the sequential Cityscapes dataset. The shared categories for evaluation are person, car, motorcycle, and bicycle.Weather Adaptation: This scenario scrutinizes the model's resilience to sudden environmental changes, particularly shifts in weather conditions. Cityscapes and Cityscapes-Foggy serve as key evaluation grounds. In addition, we introduce an extra evaluation set from the Cityscapes validation set to assess the model's robustness in unseen urban settings. Another reason for this experiment is that Memclr did not mention the sequential scenario. This table aims X. Shi et al. for a fair comparison with the existing dataset in MemCLR. We use these results to demonstrate that data acquisition is also effective in weather adaptation.Weather adaptation is non-sequential and involves only 500 frames, making it challenging to stabilize the rare category initially for ARC. Thus, we only consider using AUF.We evaluate our model's performance with two SOTA methods Tent [25] and MemCLR [23] in three specific settings: two types of Online Sequential Adaptation and one Weather Adaptation. Notably, Tent employs entropy minimization to boost model performance in classification and semantic segmentation tasks. We deploy the entropy minimization in object detection by disambiguating the Θ S classification result based on the teacher model's proposal. For each experiment, we used random seeds run five times, calculating the mean with standard deviation. We use mAP calculated via the AP50 as the evaluation metric.As shown in Table 1 , when testing the model that was trained on the source data directly, the average precision (AP) for car detection was 31.9, which was the lowest score, as expected. Our model demonstrates a marked improvement over the state-of-the-art method, MemCLR [23] , in car detection, achieving an AP score of 46.0±0.2. This is achieved using 345 frames for adaptation, accounting for only about 0.32% of the total available frames. As illustrated in Table 2 , we focus on four share categories: person, car, motorcycle, and bicycle, reporting results in terms of mAP. Our model surpasses MemCLR in performance by 9.3 mAP while utilizing only approximately 384 frames. Significantly, our method greatly improved the detection of motorcycles, a rare category, by achieving 12 AP higher than MemCLR [23] . As shown in Table 3 , our model outperforms MemCLR with an mAP score of 32.1, improving by 2.3 points. Notably, the AP score for the car category increased by 7 points compared to MemCLR's performance.Our study places significant emphasis on sequential datasets, particularly the transition from the SHIFT to the Cityscapes dataset, to simulate real-world conditions more effectively. This section elucidates the effects of various experimental components on the final performance metrics of our model. Table 4 illustrates the effects of various configurations on the overall model performance. The term "No acquire" refers to our method when using all the frames for adaptation. Resulting in a severe collapse of the performance, likely due to frame overlaps and redundant information. The configuration labelled "AUF" involves the use of dissimilar frame acquisition alone and achieved a mean Average Precision (mAP) of 22.7, which is an improvement of 21.4 mAP over the "No acquire" setup. The "AUF+ARC" configuration, which incorporates both dissimilar frame acquisition and rare category acquisition, demonstrated superior performance compared to "AUF". Specifically, it showed notable improvement in the rare category, with motorcycles achieving an AP of 5.5. Moreover, the overall mAP increased by 3.1 compared to the "AUF" configuration.Data acquisition not only enhances model performance in the target domain but also reduces the time needed for adaptation. We evaluated the performance speed of MemCLR and our method on the sequential Cityscapes dataset, measuring the average processing time per frame in milliseconds (ms). Our method demonstrated an efficiency of 12.7 ms per frame, compared to MemCLR's 21.3 ms, with both methods tested on RTX 4090 GPU. This is because MemCLR requires both inference and adaptation for each frame, while our method only performs inference and selects key frames, avoiding constant adaptation. 
conclusion: 
 In this work, we introduce an unsupervised data acquisition method in O-SFDA object detection tailored to autonomous vehicles in diverse and dynamic settings. This approach operates in two stages: initially, it selects highly informative frames, and then it gives extra attention to those frames that also include the rare category to reduce class imbalance and frame overlap. Experimental results on multiple datasets show the effectiveness of our method and outperform existing methods in terms of object detection performance.  
 
 Appendix:  


Decision: The average score of the paper given all the reviews received before the deadline was higher than 5.5 (1 is lowest, 10 is highest), therefore the paper is accepted. The Authors are encouraged to consider feedback for the camera ready version of the paper due on August 31st.

Reviews: review: title: title: Review of submission 3
main_review: main_review: This paper proposes an online source-free domain adaptation method for object detection in the context of autonomous vehicles. Overall, the main idea consists of integrating two ‘decision’ branches to find ‘key frames’ for updating models within a teacher/student framework.

Pros:

The idea is interesting (however I’m  not certain about its novelty) and it seems relevant for the context of autonomous vehicles.

The proposed method has been evaluated with multiple datasets, which contributes towards its applicability. 

Cons:

I strongly believe that this paper needs to include a related work section. It is not clear for me how their proposal differs from existing works. What is the real contribution here? Some parts of the system seem very similar to [17]. Where are the ideas from ARC and AUF clusters coming from? Are ARC and AUF something that all readers should know about? Or are these components firstly introduced in this paper? If yes, more details about these would be needed.

I have some concerns regarding the fairness of evaluation. It does not seem right to me. Why should one expect inferior performance if the model is strategically selecting frames with mostly correct detections? Why has only one ‘rare’ class been included? How is standard evaluation performed on this setting?

Would be good to include arrows on table headers to better understand performance comparisons. What does the symbol on the Tent method mean? Captions for all tables could be more descriptive.

I believe ablation studies should be extended. Since it seems that ARC and AUf are the main components of the proposal. What would happen when including those to other methods, e.g. [17]? 

Why is the reason to set gamma to 0.975? This leads to another concern, why there are not experiments with different hyperparameters?

Overall, the paper reads well, but there are some parts that could be improved. For example:

Line 006: Not sure about this statement. Why deploying object detection methods in unfamiliar environments is an issue? I think the issue here is not with the deployment itself, if not with the behaviour/performance of models under environments that differ a lot from the data distribution of training.

Line 021: First 2 sentences of this paragraph do not follow. I don’t see what is the logical relationship between them. I’d recommend rewriting it.

Line 043: Why are only motorbikes mentioned as ‘rare classes’? I believe there are many other examples.

Line 054: Here it is mentioned that the model will be evaluated with a ‘three domain adaptation process’. Later in line 067, these are referred to as benchmarks. It would be beneficial for readers to have some kind of consistency for this terminology.

Caption of figure 1 is unclear. Please have a look at the part that describes x, it is hard to understand what this means.
title: title: Review of submission 3
main_review: main_review: This paper explore Online Source-Free Domain Adaptation for object detection. The main contribution is that an unsupervised data acquisition method is designed. The data acquisition process includes two metrics, AUF and ARC. These two metrics are shown effective in the experiments.

The paper is hard to follow and many details in model and experiment setting are missing. Why is the result of "No acquire" setup near to 0? The authors explain that it is "due to frame overlaps and redundant information". However, it cannot convince me. Besides, no ablation study for hyper-parameter is conducted, which is importance for such a heuristic method.
title: title: Review of submission 3
main_review: main_review: The authors propose an online source-free domain adaptation methodology designed to enhance the training process of object detection models through unsupervised learning by discriminating between key frames. The goal of developing strategies to adapt models to new domains without relying on labeled data is both relevant and clearly motivated, and the proposed approach appears to be novel and interesting. However, I have a few concerns that need to be addressed:

1. Terms such as “weak adaptation” and “strong adaptation” need a more thorough explanation. Additionally, the concept of “rare category” should be defined more precisely. Specifically, it should be clarified whether “rare category” refers to pre-existing categories with low detection scores or to new, additional categories in the dataset.

2. An analysis of the computational complexity related to the number of centroids as the duration of the video increases would provide insight into the efficiency and scalability of the proposed method.

3. Authors could elaborate on how this methodology relates to existing literature on anomaly detection. Understanding the connection could provide valuable context for the proposed approach.

4. It would be helpful to discuss how the percentage of key frames is adapted based on desired latency and available computational resources.

Addressing these points will enhance the clarity and comprehensiveness of the proposed methodology.
title: title: Review of submission 3
main_review: main_review: The paper addresses the challenge of Online Source-Free Domain Adaptation for object detection in autonomous vehicles. The authors propose a novel two-stage unsupervised data acquisition method to enhance adaptive object detection. The method involves selecting the most informative unlabeled frames, with a focus on rare categories, to mitigate class imbalance and reduce computational costs. Empirical evaluations across multiple datasets demonstrate the performance. 

Thank you for submitting your paper. I hope my comments are helpful and contribute to further enhancing your work.

# Following my comments:
The dynamic selection of informative frames is an interesting approach. This is practical, especially in scenarios where computational resources are limited, and class imbalance is a concern. The paper is also well structured, and the results are comprehensive.

Despite the strengths, the paper has some weaknesses in the presentation of the method. First, the concept of "informative frames" is inadequately explained, leaving readers unclear on what constitutes an informative frame. The explanation of keyframes and the criteria for selecting them, particularly in relation to handling domain shift, lacks clarity. A literature review could have been beneficial, highlighting the differences with existing works. 

The empirical results, although promising, do not fully explore the generalizability of the method across different scenarios or environments, leaving gaps in understanding the broader applicability of the approach. For example, would have been interesting to show results in the dataset with higher domain gap.

The paper mentions the challenge of adapting to new domains "on-the-fly" but does not really discuss about it.

Clarifications are also necessary regarding the results in Tables 1-3: it should be specified whether these are based on the authors' implementations of Tent and MemCLR or if they are results directly cited from existing literature. Additionally, the significant difference in gains observed in Table 2 as compared to other datasets needs explanations to help in understanding these discrepancies. why the gains are much more different than from the other results?
title: title: Official Review for Submission 3
main_review: main_review: The paper proposes a domain adaptation method that dynamically selects frames on-the-fly to improve the performance of a network trained on an inaccessible source domain. I found the overall pipeline to be reasonable, and it appears that the network's performance has improved on the test domain.

However, I believe the paper would benefit from a clearer definition of some important processes, such as what constitutes weak and strong augmentation, as well as a detailed explanation of the losses defined in Eq. (1). Providing these clarifications would make the model more understandable and the paper more self-contained.