DPO: Dual-Perturbation Optimization for Test-time Adaptation in 3D Object Detection

Authors: zhuoxiao  chen
zixin  wang
yadan  luo
sen  wang
zi 2024 huang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-07 14:00:18

Abstract: LiDAR-based 3D object detection has seen impressive advances in recent times. However, deploying trained 3D detectors in the real world often yields unsatisfactory performance when the distribution of the test data significantly deviates from the training data due to different weather conditions, object sizes, etc. A key factor in this performance degradation is the diminished generalizability of pre-trained models, which creates a sharp loss landscape during training. Such sharpness, when encountered during testing, can precipitate significant performance declines, even with minor data variations. To address the aforementioned challenges, we propose dual-perturbation optimization (DPO) for Test-time Adaptation in 3D Object Detection (TTA-3OD). We minimize the sharpness to cultivate a flat loss landscape to ensure model resiliency to minor data variations, thereby enhancing the generalization of the adaptation process. To fully capture the inherent variability of the test point clouds, we further introduce adversarial perturbation to the input BEV features to better simulate the noisy test environment. As the dual perturbation strategy relies on trustworthy supervision signals, we utilize a reliable Hungarian matcher to filter out pseudo-labels sensitive to perturbations. Additionally, we introduce early Hungarian cutoff to avoid error accumulation from incorrect pseudo-labels by halting the adaptation process. Extensive experiments across three types of transfer tasks demonstrate that the proposed DPO significantly surpasses previous state-of-the-art approaches, specifically on Waymo $\rightarrow$ KITTI, outperforming the most competitive baseline by 57.72\% in $\text{AP}_\text{3D}$ and reaching 91% of the fully supervised upper bound. Our code is available in the supplementary materials.

Main paper: 
 introduction: 
 LiDAR-based 3D object detection has gained significant attention with the rapid advancements in autonomous driving [10, 25, 31, 37, 38, 48] and robotics [42, 84] , where mainstream 3D detectors are developed to interpret pure point clouds or fuse multimodal knowledge, commonly incorporating camera images [8, 36, 64] . However, deploying either point clouds-based or multimodal 3D detection models in real-world scenarios often leads to performance degradation due to distribution shifts between the training data and the encountered real-world data. For instance, a 3D detector trained on the nuScenes dataset [2] might suffer a performance drop when applied to the KITTI dataset [14] due to variations in object sizes and the number of beams. This is known as cross-dataset shift. Additionally, the shift can arise from real-world disturbances, termed as corruption-based shift [11, 20, 28] , which includes challenges like diverse weather conditions and sensor malfunctions. Moreover, multiple factors are likely to be concurrent, for instance, deploying a 3D detector in a different city while suffering severe snow. This scenario is termed as composite domain shift.Domain adaption has been discovered [5, 39, 68, 69] to mitigate the performance gap brought by various domain shifts. In 3D object detection, this involves aligning features between the labeled training data and the shifted test data to learn a domain-invariant representation [40, 79, 81] or conducting self-training with the aid of selected pseudo-labels [6, 32, 47, 73, 74] . However, these approaches necessitate extensive training over multiple epochs on both training and test sets, rendering them impractical for adaptation to the streaming data. Moreover, the exposure of the training data can significantly compromise its privacy, especially when it contains sensitive user information (e.g., user vehicle trajectories and individuals).To bridge the performance gap induced by domain shifts, while safeguarding the training data privacy and enabling swift adaptation, test-time adaptation (TTA) emerges as an ideal solution. Prior research on TTA typically adapts a source pre-trained model to the unlabeled test data, either through updating a selected subset of parameters (e.g., BatchNorm layers) [46, 55, 63] , or employing the mean-teacher model [58, 65, 66, 77] within a single epoch. However, these TTA works currently applied in image classification are inadequate for addressing the dual demands (i.e., object localization and classification) for supervision signals inherent in detection tasks. Within this context, MemCLR [62] stands out by refining the Region of Interest (RoI) features of detected objects through a transformerbased memory module for 2D object detection. Nevertheless, the stored target representations derived from the source pre-trained model cause performance degradation due to distribution shifts. These limitations pose significant challenges in utilizing previous TTA techniques for 3D object detection.To tackle these challenges, our goal is to devise an effective strategy for adapting the 3D detection model to various data shifts. We observe a common performance decline when the model encounters unfamiliar scenes. This degradation primarily occurs as the model tends to converge to sharp minima in the loss landscape during training [13] . Such convergence makes the model vulnerable to slight deviations in the test data, leading to a performance drop. Furthermore, high variability and limited availability of the test data significantly increase the vulnerability of the pre-trained source model. In response, we propose DPO to secure adaptation generalizability and robustness through a worst-case Dual-Perturbation Optimization in both model weight and input spaces. Specifically, at the model level, we apply a perturbation in the weight space [13] to the model's parameters to maximize loss within a predefined range, thereby optimizing the model toward noise-tolerant flat minima. However, due to the notable discrepancies between the training and testing scenes, merely weight perturbation is insufficient to fully address the extensive variability and complexity encountered in the 3D testing scenes. To overcome this, we augment our approach by incorporating an adversarial perturbation on the BEV feature of the test sample via element-wise addition. Once the model is adapted to maintain stability despite perturbed inputs, it becomes more resilient to noisy data, thereby enhancing its robustness. The generalization and robustness of the adaptation model heavily rely on accurate supervision-that is, adapting the detection model based on reliable pseudo-labeled 3D boxes. The supervision signals offered in previous works are either too weak for 3D detection tasks [46, 63] or excessively dependent on pre-trained source models [62] , which might be compromised by domain shifts. To this end, we introduce a reliable Hungarian matcher to ensure trustworthy pseudo-labels by filtering out 3D boxes that exhibit high matching costs before and after perturbations. The underlying assumption is that, given arbitrary perturbations, the prediction is more trustworthy if the model can still produce consistent box predictions. A consistently low Hungarian cost for pseudo-labels across recent test batches indicates the model has been sufficiently robust to shifts/noise in the test domain. Hence, to preserve generalization and minimize unnecessary computational expenses, we propose an Early Hungarian Cutoff strategy based on the Hungarian costs. We apply a moving average of the cost values from the current and all previous batches to determine when to cease the adaptation. Our approach exhibits state-of-the-art results surpassing previous TTA methods. We summarize our key contributions as follows:• We introduce TTA in LiDAR-based 3D object detection (TTA-3OD). To the best of our knowledge, this is the first work to adapt the 3D object detector during test time. To tackle the challenge in TTA-3OD, we prioritize the importance of model generalizability and reliable supervision. [61, 67] , weather conditions [20, 72] , sensor differences [19, 49, 71] , sensor failures [28] , and the synthetic-to-real gap [9, 30, 50] . Strategies to overcome these challenges include adversarial feature alignment [81] , 3D pseudo-labels [6, 7, 23, 33, 47, 51, 60, [73] [74] [75] , the mean-teacher model [21, 40] for prediction consistency, and contrastive learning [79] . Nonetheless, these cross-domain adaptation methods typically necessitate adaptation over multiple epochs, making them less suited for real-time test scenarios. 
test-time adaptation in 2d vision tasks: 
 Test-time adaptation (TTA) [34, 70] is designed to address domain shifts between the training and testing data [70] during inference time. As a representative, Tent [63] leverages entropy minimization for BatchNorm adaptation. Subsequent works [16, 22, 43, 44, 53] such as EATA [45] , identifies reliable and nonredundant samples to optimize. DUA [41] introduces adaptive momentum in a new normalization layer whereas RoTTA [77] and DELTA [83] leverage global statistics for batch norm updates. Furthermore, SoTTA [17] and SAR [46] improve BatchNorm optimization by minimizing the loss sharpness. Alternatively, some approaches optimize the entire network through the mean-teacher framework for stable supervision [59, 65] , generate reliable pseudo-labels for self-training [18, 78] , employ feature clustering [4, 26, 66] , and utilizing augmentations to enhance model robustness [80] . However, these TTA methods are developed for general image classification. Additionally, MemCLR [62] applies TTA for image-based 2D object detection, using a mean-teacher approach to align instance-level features. Nevertheless, the applicability of these image-based TTA methods to object detection from 3D point clouds remains unexplored.The concept of flat minima has been demonstrated to enhance model generalization. A prime example is SAM [13] , which improves generalization by simultaneously optimizing the original objective (e.g., cross-entropy loss) and the flatness of the loss surface. Besides, ASAM [29] aligns the sharpness with the generalization gap by re-weighting the perturbation according to the normalization operator. To deal with the presence of multiple minima within the perturbation's reach, GSAM [85] minimizes the surrogate gap between the perturbed and the original loss to avoid sharp minima with low perturbed loss. Moreover, GAM [82] introduces first-order flatness, which controls the maximum gradient norm in the neighborhood of minima. Current research on flat minima focuses mainly on supervised learning. While in TTA, the effectiveness of these strategies significantly relies on supervision signals and the shift severity of the test data, which suggests that the anticipated advantages of flat minima might not consistently materialize as expected. Challenges in TTA-3OD. The primary challenges of TTA-3OD lie in two aspects: (1) adapting the 3D detection model to unfamiliar test scenes often generates large and noisy gradients, leading to an unstable adaptation process. This instability hampers the model's ability to generalize effectively to the target domain; (2) uncontrollable variations in the testing scenes, such as environmental changes or sensor inaccuracies, can significantly compromise the quality and integrity of 3D scenes. Consequently, models trained on clean datasets struggle to maintain effectiveness and robustness when facing such distorted data, drastically diminishing their adaptation performance.To address the above two challenges, our method fundamentally enhances 1) the model's generalization and stability when adapting to new domains and 2) its robustness against noisy/corrupted data, by optimizing the sharpness of the loss landscape during model adaptation with the proposed dual-perturbation applied to both the model's weights and input data. 
minimizing sharpness in the weight space: 
 The sharpness of the training loss, is the rate of change in the surrounding region of the loss landscape. It has been identified to be empirically correlated with the generalization error [15, 24, 27] . Motivated by this, recent works propose to reduce the loss sharpness during the training phase, aiming to improve the generalization capabilities of the model. One notable example is Sharpness-Aware Minimization (SAM), which enhances model training by integrating and optimizing the worst-case perturbations in model weights. The fundamental principle of SAM is that by minimizing the loss with respect to maximally perturbed weights within a vicinity, the entire vicinity (i.e., all losses within it) is minimized. This directs the optimization trajectory toward a flat minima in the loss landscape. A flat minima is indicative of superior generalization capabilities, as the loss over it is less sensitive to large perturbations and/or noise in the model weights, unlike sharp minima.In the context of TTA-3OD, the loss sharpness [1] during the adaptation can be formally defined as follows:EQUATIONHere, 𝜖 is a perturbation vector in the weight space such that its Euclidean norm is bounded by 𝜌.Previous literature [12, 13, 29, 35, 82, 85] calculates the sharpness by the loss between model predictions 𝑓 𝑡 (𝑋 𝑡 ) and its ground truth labels 𝑌 𝑡 . While no supervision is available during test time, a soft loss [17, 46, 63] is commonly employed with selective supervision. Next, the optimization of the detection loss and its sharpness is defined as: minEQUATIONThe inner optimization aims to find a perturbation 𝜖 𝑤 on model weights Θ 𝑡 within a Euclidean ball of radius 𝜌 to maximize the detection loss L det , which is calculated based on the generated pseudo-labels Ŷ𝑡 . To obtain the worst-case 𝜖 𝑤 , we draw inspiration from [13] to approximate the inner optimization by the first-order Taylor expansion:EQUATIONThen ε𝑤 (Θ 𝑡 ), which satisfies this approximation, is derived by resolving a dual norm problem:EQUATION)To expedite computation, we omit the second-order term. While this improves model generalization by targeting non-sharp minima within the loss landscape, the optimized perturbations to weights do not directly deal with variations and/or noise of the input test scenes. When facing the test data, the detection performance is substantially degraded due to the severe data-level corruptions in the test point cloud. For example, when suffering heavy snow and shifted object scales, simultaneously, the performance drops from 73.45% to 3.84% in AP 3D . The empirical evidence suggests that augmenting the baseline model with SAM [46] results in a marginal improvement of only 0.9% in AP 3D , indicating its ineffectiveness in bridging the domain gap in the test 3D scenes.To surmount the above challenge, we strengthen the model's resilience against shifted input point clouds by optimizing the model with perturbed input. Rather than randomly mimicking test perturbations, our approach focuses on learning an adversarial perturbation that represents the worst-case corruption to the input data. The underlying rationale is that optimizing the detection model with maximal perturbed data within a given vicinity induces robustness to any perturbations encountered within that vicinity. As shown in Figure 1 , we simultaneously guide the detection model toward the flat minima in both weight and input space, such that the model can stably generalize to the test data with any potential noises.To introduce perturbations into the input batch, we incorporate an adversarial perturbation mask 𝜖 𝑧 into the bird's eye view (BEV) feature map 𝑍 𝑡 through element-wise addition to each grid of the BEV map. This is because the 3D detector primarily localizes object proposals from the BEV map, which contains rich spatial information about 3D instances. Thus, even minimal perturbations to the feature map can cause significant spatial shifts in the instances, leading to misalignment in the final predicted 3D bounding boxes. To seek the worst-case perturbation 𝜖 𝑧 within the input space that maximizes detection loss, we formulate the optimization problem as follows:EQUATIONSimilar to approximating ε𝑤 (Θ 𝑡 ), we derive the approximated ε𝑧 (𝑍 𝑡 ) within the input space. This resulting perturbation mask ε𝑧 shares the same dimension as the latent feature map 𝑍 𝑡 and is applied to 𝑍 𝑡 via element-wise addition, yielding the perturbed feature map 𝑍 𝑡 + ε𝑧 .The final objective is to train the detection model with the optimal dual-perturbation in both model ( ε𝑤 ) and input space ( ε𝑧 ). To this end, we approximate the gradient by substituting ε𝑤 and 𝑍 𝑡 + ε𝑧 into Eqn. (2), then performing differentiation to calculate the gradient 𝑔:EQUATIONFinally, the detection loss and its sharpness, calculated with the perturbed test batch, can be jointly minimized by: minEQUATIONwhere the inner optimization is solved through approximation (i.e., Eqn. ( 3)-( 5)) and the outer optimization goal is achieved by stochastic gradient descent (SGD) with the gradient 𝑔 calculated in Eqn. ( 6). The step-by-step workflow of the proposed DPO is introduced in Algorithm 1.However, the pursuit of flat minima in both the input and weight spaces depends on the gradients guided by high-quality supervision. Previous SAM-based TTA methods selectively adapt highconfidence samples [17, 46, 76] , as they assume that confidence reflects prediction reliability. Nevertheless, acquiring effective supervision in the TTA-3OD task is challenging due to the low-quality pseudo-labeled boxes, Ŷ𝑡 = { b1 , • • • , b𝑁 𝑡 }, used for calculating the detection loss L det , where 𝑁 𝑡 represents the number of predicted boxes in the current batch 𝑡. This issue arises from the difficulties the source-trained model 𝑓 𝑆 (•; Θ S ) faces in accurately predicting 3D boxes around objects in the test point clouds, which subjects to significant shifts or corruptions.To obtain reliable pseudo-labeled boxes that are robust to the test data noise, we aim to select those 3D boxes unaffected by optimized perturbations (Sect. 3.3). The rationale is that consistency in box predictions between clean inputs and perturbed input features 𝑍 𝑡 + ε𝑧 from the model before and after perturbation demonstrates resilience to noises ε𝑧 . The box prediction from the 𝑡-th perturbed input batch is defined as:EQUATIONwhere 𝑀 𝑡 is the number of predicted boxes at batch 𝑡 after perturbation. To measure the consistency between Ŷ𝑡 and Ỹ𝑡 , we adopt Hungarian matching [3, 54] , an effective bipartite matching technique that guarantees optimal one-to-one alignment between two sets of box predictions. Specifically, We ensure both sets are of equal size by augmenting the smaller set (assuming 𝑀 𝑡 < 𝑁 𝑡 ) with ∅ until it matches 𝑁 𝑡 in size. To achieve optimal bipartite matching between the equal-sized sets, the Hungarian algorithm is applied to find a permutation of 𝑁 𝑡 elements 𝑝 ∈ P 𝑁 𝑡 that minimizes the matching cost:EQUATIONThe cost C box (•; •) integrates intersection-over-union (IoU) and L1 distance to account for the central coordinates, dimensions, and orientations of a pair of boxes: b𝑛 and its corresponding matched box b𝑝 (𝑛) indexed by 𝑝 (𝑛).Utilizing the derived optimal assignment p, each pseudo-labeled box b𝑛 is associated with its corresponding minimal-cost match, denoted as C box ( b𝑛 ; b p (𝑛) ). Note that when b𝑛 pairs with the empty set ∅, we assign the cost to be infinite, indicating that the corresponding box is too noise-sensitive to be accurately localized within the perturbed point clouds.We categorize the reliability of each pseudo-label b𝑛 into three distinct levels by the thresholds C 1 and C 2 :EQUATION)To guide the model towards flat minima with trustworthy supervision, we selectively adapt the model with high-quality bounding boxes and treat those of low quality as background, as shown in C 1 = 𝐴 costs [⌈𝛼 × 𝑛⌉], C 2 = 𝐴 costs [⌈(1 -𝛼) × 𝑛⌉],whereEQUATIONThe ceiling function ⌈•⌉ ensures that the index for 𝐴 costs is always an integer. Adopting global thresholds C 1 and C 2 derived from all historical costs facilitates more precise categorization of pseudolabeled boxes into high and low-quality categories.While the Hungarian matcher significantly enhances the quality of pseudo-labels, the correctness of the selected pseudo-labels cannot be entirely guaranteed. Even a small number of incorrect pseudolabels once learned and accumulated, can lead to substantial performance degradation. Furthermore, updating the 3D detector demands significant computational resources and time. Identifying an optimal stopping point for the adaptation process is thus crucial.In this regard, we suggest using the Hungarian cost as a criterion to halt the adaptation process. The rationale is that a lower Hungarian cost for a given batch indicates the pseudo-labels are more accurate, thereby making the update process more reliable. Additionally, a consistently low Hungarian cost of pseudo-labels is crucial. Therefore, we introduce the use of a moving average to balance the current and all previous costs:𝐶 𝑡 ema = 𝛾𝐶 𝑡 box + (1 -𝛾) ∑︁ 𝐶 𝑡 -1 box ,whereEQUATIONis the average Hungarian cost of all boxes in the current batch 𝑡. 𝐶 𝑡 ema represents the moving average of the Hungarian cost, and 𝛾 denotes the decay rate. A threshold 𝐶 stop is further set for the moving average cost. When it falls below the threshold, the adaptation process is halted, the model thus transitions to the inference mode for all subsequent batches. [14] , Waymo [56] , and nuScenes [2] . Additionally, the recently released KITTI-C dataset [28] , which simulates real-world corruptions, is incorporated to pose the TTA-3OD challenge. Following prior works [6, 73, 74] , we address cross-dataset test-time adaptation tasks (e.g., Waymo → KITTI and nuScenes → KITTI), involving adaptation across (i) object shifts (e.g., scale and point density variations), and (ii) environmental shifts (e.g., changes in deployment locations and LiDAR configurations). Furthermore, we evaluate adaptation performance against real-world corruptions (e.g., KITTI → KITTI-C), including conditions such as fog, wet conditions (Wet.), snow, motion blur (Moti.), missing beams (Beam.), crosstalk (Cross.T), incomplete echoes (Inc.), and cross-sensor interference (Cross.S). Experiments also extend to the challenging scenarios of composite domain shifts (e.g., Waymo → KITTI-C), where inconsistencies across datasets and corruptions coexist within the test 3D scenes.4.1.2 Implementation Details. We leverage the OpenPCDet framework [57] . Experiments are conducted on a single NVIDIA RTX A6000 GPU with 48 GB of memory. We opt for a batch size of 8 and fix the hyperparameters 𝜌 = 1𝑒 -4, 𝛼 = 0.08, 𝛾 = 0.5, 𝜂 = 10 -3 . For evaluation purposes, we adhere to the official metrics of the KITTI benchmark, reporting the average precision for the car class in both 3D (i.e., AP 3D ) and bird's eye view (i.e., AP BEV ) perspectives, calculated over 40 recall positions and applying a 0.7 IoU threshold. The closed gap [73] is calculated as:AP method -AP No Adapt.AP Oracle -AP No Adapt. × 100%. 
baseline methods.: 
 We integrate a voxel-based backbone (i.e., SECOND) into our proposed method and evaluate it against a comprehensive array of baseline approaches:• No Adapt.: directly inferring the test data with a model pretrained on the source domain, without any adaptation. • SN [67] : weakly supervised domain adaptive 3D detection that adjusts source object sizes using target domain statistics. • ST3D [73] : an unsupervised domain adaptation method for 3D detection, utilizing multi-epoch pseudo-labeling for self-training. • Tent [63] : a fully TTA method that optimizes BatchNorm layers by minimizing the entropy of predictions.• CoTTA [65] : a TTA strategy that leverages mean-teacher framework to provide supervisory signals through augmentations and employs random neuron restoration to retain source knowledge.• SAR [46] : an advancement beyond Tent, employing sharpnessaware minimization for selected supervision. • MemCLR [62] : TTA for image-based object detection using meanteacher to align the instance-level features by a memory module. • Oracle: a fully supervised model trained on the test scenes.Cross-dataset Shifts. We conducted extensive experiments on two cross-dataset TTA-3OD tasks, evaluating AP BEV , AP 3D , and closed gap, as presented in Table 1 . Compared to direct inference (i.e. No Adapt.), our experiments revealed that existing TTA baselines might negatively impact adaptation in 3D object detection especially on AP BEV for the Waymo → KITTI task, indicating the importance of tailoring a TTA method specifically for 3D detection tasks. Additionally, compared to the most competitive baseline, CoTTA, DPO achieves significant improvements in AP 3D , with increases of 57.7% and 14.2% for the Waymo → KITTI and nuScenes → KITTI tasks, respectively. Similarly, DPO significantly outperforms CoTTA in AP BEV , demonstrating a considerable margin. Besides, DPO effectively reduces the closed gap, demonstrating a closure of about 61.47% and 65.66% for the Waymo → KITTI and nuScenes → KITTI tasks, correspondingly, in AP 3D . Moreover, it achieves up to 91% and 87.5% of the fully supervised Oracle's performance in AP BEV for the respective tasks. Overall, our proposed DPO not only surpasses all TTA baselines but also delivers performances competitive with those of Unsupervised Domain Adaptation (UDA) and fully supervised learning, highlighting its effectiveness in bridging domain gaps in 3D object detection.We evaluated DPO's efficacy against corruption-induced shifts on KITTI → KITTI-C with heavy severity of eight real-world corruption by AP 3D in hard difficulty scenarios. As indicated in Table 2 , DPO outperforms all TTA baselines in terms of Mean AP 3D , exceeding the performance of the closest competitive baseline, Tent, by 1.2%. DPO consistently achieves top performance across most corruption types, demonstrating the enhanced robustness of DPO and its effectiveness in adapting 3D models to a wide array of corrupted environments. In particular, the performance without any adaptation (column 1) significantly declines, illustrating the compounded challenges of composite shifts. For example, only 8.38% AP 3D is recorded for Motion Blur and 3.84% AP 3D for Incomplete Echoes at the moderate level. Conversely, against the most competitive baseline (column 3), our approach notably enhances adaptation performance for these challenging corruptions by 75.13% and 61.36%, respectively. Direct inference for Incomplete Echoes at hard difficulty yields only a 3.51% in AP 3D , whereas our method markedly increases this by more than 231.62%, achieving a 53.97% improvement over the highest baseline. In summary, existing TTA methods fall short in navigating significant domain shifts (i.e., composite domain shifts) in 3D scenes, while DPO could effectively tackle these challenges. that variations in 𝜌 exhibit a minimal influence on AP BEV , contrasting with AP 3D , which demonstrates significant performance variability when the perturbation radius is adjusted to 0.1. This discrepancy can be attributed to two primary factors. Firstly, an increase in perturbation radius adversely affects adaptation performance. Secondly, a larger perturbation radius results in the selection of a reduced number of pseudo-labeled 3D boxes for self-training due to the increased divergence in model predictions. However, when employing a perturbation radius within a smaller range (e.g., 10 -4 -10 -2 ), the stability of AP 3D is notably enhanced. 4 , updating BN (only 2% of the total parameters) results in a slight decrease of 0.41% in AP 3D and a slight increase in speed by 0.02s per frame. This demonstrates that our method remains effective even when only a small fraction of the parameters are updated. 4.4.4 Sensitivity to 3D Backbone Detector. To validate the effectiveness of DPO, we assess the performance sensitivity when coupled with a two-stage, point-and voxel-based backbone detector: PVRCNN [52] . The results of TTA baselines and our approach from Waymo to KITTI are summarized in Table 5 . Our observations indicate that changing the backbone has a significant impact on the performance of baseline TTA methods. Conversely, our proposed method not only exhibits stability but also achieves a remarkable performance enhancement (33.83% in AP 3D ) over the leading baseline. Besides, DPO also achieves state-of-the-art AP BEV performance compared to all baseline methods, emphasizing the consistent efficacy of our approach across different backbones.To assess the efficiency of DPO, we conducted a comparative analysis for adaptation speed (i.e., seconds per frame) and GPU memory usage, as illustrated in Figure 6 . CoTTA is identified as the most efficient TTA baseline for 3D object detection, demonstrating rapid adaptation capabilities (under 0.15 seconds per frame). Conversely, other baselines, notably SAR, required significantly more adaptation time and yet underperformed, achieving AP BEV of less than 66%. Despite a slightly longer processing time, DPO markedly surpassed all TTA baselines, showcasing its superior performance. In terms of GPU memory consumption, CoTTA reported moderate usage, whereas MemCLR exceeded 18,000 MiB but fell short in performance. The proposed DPO, in contrast, not only required less GPU memory than both MemCLR and CoTTA but also achieved dominating adaptation performance, highlighting the efficiency and effectiveness of our method. 
conclusion: 
 In this work, we present a novel framework for Test-Time Adaptation in 3D Object Detection (TTA-3OD) aimed at adapting detectors to new unlabeled scenes with a single pass. Our approach incorporates worst-case perturbations at both model and input levels to enhance robustness and generalization, thereby enabling 3D detectors to stably adapt to any test scenes with corruptions. We employ reliable Hungarian matching for trustworthy pseudo-label selection, with an early cutoff to avoid computation burden and error accumulation. Beyond point-and voxel-representation-based 3D detectors used in this paper, our future work will further validate multimodal detectors with different input modalities, such as BEVfusion [36] to verify shifts across modalities. 
 
 Appendix:  


Decision: The initial ratings of this paper were mixed. However, after the rebuttal, the authors effectively addressed the concerns of most reviewers, resulting in all positive ratings. Therefore, this paper is recommend acceptance.

Reviews: review: paper_summary: paper_summary: The paper introduces a new task, namely test-time Adaptation in 3D object detection and proposes dual-perturbation optimization (DPO). DPO add perturbation in weight and input space and ensure the model resiliency to minor data variations. Further, to obtain accurate pseudo labels, the authors propose a early Hungarian cutoff strategy. Experiments show that DPO can largely outperform previous 2D TTA works in 3D object detection task in different settings.
strength_weakness: strength_weakness: -	The method that add perturbation and ensure the stability of model is reasonable.

-	The paper is easy to follow.
limitations: limitations: -	The authors should report the number of updated parameters since the efficiency is important in TTA tasks. Besides, the authors add perturbation in weight and bev space, does it mean that a single frame will be calculated three times during adaptation? If so, I think the efficiency of the method need to be improved.

-	I wonder the result of combining DPO and more SOTA detectors such as DSVT. I notice that the results of using PV-RCNN are worse than use SECOND. However, PV-RCNN is a more powerful detector, I think the authors should give some explanation.

-	The authors should also conduct experiments under waymo-to-nuScenes settings.

-	More comparison should be included such as TTA in 3D [A][B].

[A] Mirza M J, Shin I, Lin W, et al. Mate: Masked autoencoders are online 3d test-time learners[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 16709-16718.

[B] Wang Y, Cheraghian A, Hayder Z, et al. Backpropagation-free Network for 3D Test-time Adaptation[J]. arXiv preprint arXiv:2403.18442, 2024.
paper_summary: paper_summary: This paper proposes a approach to improve the performance of 3D object detectors during test-time adaptation (TTA). The method addresses the challenges posed by distribution shifts between training and test data. The proposed Dual-Perturbation Optimization (DPO) strategy aims to create a flat loss landscape to enhance the model's generalization and robustness to data variations. The experimental results demonstrate significant improvements over state-of-the-art approaches in various transfer tasks, including cross-dataset, corruption-based, and composite domain shifts.
strength_weakness: strength_weakness: 1.The dual-perturbation approach enhances model robustness by addressing both weight and input feature spaces, ensuring improved generalization and stability.
2.The use of a Hungarian matcher to filter unreliable pseudo-labels ensures high-quality supervision. The early Hungarian cutoff strategy minimizes computational overhead while maintaining high adaptation performance.
limitations: limitations: 1.This paper is well-structured, and the method is easy to follow, but some of the wording is obscure and needs further polish.
2.While this paper is the first to use test-time adaptation for 3D object detection, adding perturbation/data augmentation is already common in the UDA field. Therefore, the innovation is not very clear, and the contribution should be highlighted with more analysis and comparison.
3.Section 3.2 repeats too much of [11] and should highlight this paper's unique aspects. Also, the technical ideas in sections 3.2 and 3.3 are very similar to [11]. Is there any unique design specifically for the 3D object detection task?
4.This paper has too many hyperparameters and thresholds, like Sharpness Radius, Pseudo-label Threshold, C_1, C_2, and C_stop. This problematic for DA method, as it requires careful tuning for different tasks and datasets.
paper_summary: paper_summary: This paper introduces the method to enhance the adaptability and robustness of LiDAR-based 3D object detection models in real-world scenarios. This method addresses performance degradation due to distribution shifts between training and test data by implementing a dual-perturbation strategy. The dual-perturbation optimization (DPO) method introduces perturbations in both the model weights and input data to achieve a flatter loss landscape, enhancing the model's resilience to data variations. Additionally, the method employs a reliable Hungarian matcher to filter out pseudo-labels sensitive to perturbations, ensuring the use of high-quality supervision signals during adaptation. Extensive experiments demonstrate that DPO significantly outperforms existing state-of-the-art methods in various transfer tasks.
strength_weakness: strength_weakness: Innovative Dual-Perturbation Strategy: The introduction of dual-perturbation optimization, which applies perturbations to both model weights and input data, is an innovative approach that significantly enhances the model's robustness and adaptability to data variations and noise.

High-Quality Supervision Mechanism: The use of a reliable Hungarian matcher to filter pseudo-labels ensures high-quality supervision during the adaptation process. This mechanism effectively mitigates the impact of noisy or incorrect labels, leading to more stable and accurate model updates.

Comprehensive Evaluation: The paper provides extensive experimental evaluations across various scenarios, including cross-dataset, corruption-based, and composite domain shifts. The results highlight the superior performance of DPO compared to existing methods, demonstrating its effectiveness in real-world applications.
limitations: limitations: Computational Overhead: The dual-perturbation strategy and Hungarian matching mechanism, while effective, may incur significant computational overhead. This could limit the practical applicability of the method in real-time or resource-limited settings.

Sensitivity to Hyperparameters: The method's performance is sensitive to hyperparameters, such as the perturbation radius and pseudo-label thresholds. Fine-tuning these parameters requires careful consideration and may not be straightforward, potentially hindering the method's ease of use and adaptability.
paper_summary: paper_summary: This paper proposes a dual-perturbation optimization method, called TTA-3OD, to improve 3D object detection models' test-time adaptation ability. On the one hand, TTA-3OD proposes to utilize weight and input perturbation to minimize the sharpness of data. This could improve the generalization and robustness of the model. Then, a Hungarian Matcher and early cutoff are adopted to predict precise pseudo-labels. The experiment result is promising.
strength_weakness: strength_weakness: 1. The paper is well-written and easy to follow.
2. The problem is practical and the motivation is strong;
3. The experiment is well-designed and extensive.
limitations: limitations: 1. In Table 3, there is a decrease in AP_{BEV} in terms of Easy difficulty from row 2 to row 3, could you please analyze the potential reason?
2. Despite the high test-time efficiency of TTA-3OD, could you please also analyze the training efficiency?