FARFusion V2: A Geometry-based Radar-Camera Fusion Method on the Ground for Roadside Far-Range 3D Object Detection

Authors: yanyong  zhang
yao  li
jiajun  deng
yuxuan  xiao
yingjie  wang
xiaomeng  chu
jianmin  ji
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-06 14:54:01

Abstract: Fusing the data of millimeter-wave Radar sensors and high-definition cameras has emerged as a viable approach to achieving precise 3D object detection for roadside traffic surveillance. For roadside perception systems, earlier studies have pointed out that it is better to perform the fusion on the 2D image plane than on the BEV plane (which is popular for on-car perception systems), especially when the perception range is large (e.g., > 150𝑚). Image-plane fusion requires critical transformations, like perspective projection from the Radar’s BEV to the camera’s 2D plane and reverse IPM. However, real-world issues like uneven terrain and sensor movement degrade these transformations’ precision, impacting fusion effectiveness. To alleviate these issues, we propose a geometry-based Radar-camera fusion method on the ground, namely FARFusion V2. Specifically, we extend the ground-plane assumption in FARFusion [20] to support arbitrary shapes by formulating the ground height as an implicit representation based on geometric transformations. By incorporating the ground information, we can enhance Radar data with target height measurements. Consequently, we can thus project the enhanced Radar data onto the 2D plane to obtain more accurate depth information, thereby assisting the IPM process. A real-time parameterized transformation parameters estimation module is further introduced to refine the view transformation processes. Moreover, considering various measurement noises across these two sensors, we introduce an uncertainty-based depth fusion strategy into the 2D fusion process to maximize the probability of obtaining the optimal depth value. Extensive experiments are conducted on our collected roadside OWL benchmark, demonstrating the excellent localization capacity of FARFusion V2 in far-range scenarios. Our method achieves an average location accuracy of 0.771m when we extend the detection range up to 500m.

Main paper: 
 introduction: 
 Far-range 3D object detection and tracking are crucial for roadside perception in intelligent transportation systems, as highlighted in recent studies [4, 7, 33] . These technologies can significantly enhance roadside-assisted autonomous driving and provide early warnings of potential traffic incidents [2, 13, 29, 41, 46] . To ensure precise vehicle location at large distances, deploying both millimeter wave radar (referred to as Radar in this paper) and high-definition (HD) cameras for joint perception has become an emerging solution in roadside scenes [8] . Feature-level Radar and camera fusion 𝑟𝑒𝑎𝑙 to the 𝑃 2𝐷 𝑟𝑒𝑎𝑙 and also can inversely map the 2D point 𝑃 2𝐷 𝑟𝑒𝑎𝑙 to the 𝑃 3𝐷 𝑟𝑒𝑎𝑙 via IPM. Differently, with the flat ground plane assumption, we can only project the 𝑃 3𝐷 𝑏𝑒𝑣 to the deviated position 𝑃 2𝐷 𝑑𝑒𝑣 on the 2D plane and inversely map the 𝑃 2𝐷 𝑟𝑒𝑎𝑙 to the deviated position 𝑃 3𝐷 𝑖𝑝𝑚 in the 3D space via IPM. (b) If the camera has an unexpected movement with a pitch angle 𝜃 , there will be a larger deviation 𝑑 ′ at far ranges.approaches [17, 34, 35, 43, 44] have achieved high performance. However, they usually require extensive annotated data [32] that is hard to obtain in far-range scenes. Therefore, target-level fusion methods are primarily used in practical systems.Fusion on the image plane first is more reliable for target-level fusion as illustrated in FARFusion [20] , especially for far-range roadside scenes (e.g., > 150m). Specifically, lifting 2D targets into 3D space directly suffers from larger location errors, causing incorrect association with Radar targets on the BEV plane. Thus, FARFusion first projects Radar targets to the 2D plane to associate with 2D targets, then maps 2D targets to the BEV plane using inverse perspective mapping (IPM) with Radar depth information. FARFusion also performs the fusion on the ground. This fusion strategy requires critical view transformations between Radar targets in 3D space and camera targets on the 2D plane. However, mapping from 2D to 3D space is an ill-posed problem [5, 21, 26] since a point on the image plane corresponds to all collinear 3D points on the ray from the optical center. Inspired by Monoground [26] , FARFusion represents vehicles as Radar points or camera 2D points on the ground. On the BEV plane, the Radar coordinate system is parallel to the ground plane because it is aligned to the UTM coordinate system in initial calibration [20] . On the 2D image plane, FARFusion models each object as a point at the contact area between the vehicle bottom and the ground, creating a one-to-one correspondence for the 2D-3D mapping. These strategies enable FARFusion to achieve accurate localization in far-range scenarios.However, during the fusion process, FARFusion assumes that the road is a flat plane, which may not hold in real far-range scenarios. The uneven road in the real world will affect the precision of view transformations of both perspective projection and IPM processes shown in Fig. 1(a) . Specifically, the commonly used Radar sensors lack height measurement, so we usually assign a fixed height to Radar data when projecting Radar points from the BEV plane to the 2D image plane. This will result in inaccurate projection depth values and positions on uneven roads. Meanwhile, IPM also relies on the planar assumption [40, 42] , which causes deviation between IPM points and actual positions on uneven roads. Additionally, FARFusion also assumes the roadside sensors are fixed in a time interval. Unfortunately, real-time unexpected sensor motions still exist due to the windy weather, causing inaccurate transformation between 2D and 3D spaces especially at larger distances shown in Fig. 1(b ). Here we only show the motion in the pitch angle, as this direction has the most significant impact on depth estimation.In this work, we propose FARFusion V2 to mitigate the above issues for practical far-range roadside perception. Our framework has two key designs -implicit ground height learning and realtime transformation parameters estimation. Firstly, considering the ground may have arbitrary shapes, we represent ground height as a learnable function. We also assume there is a virtual ground plane. Then we utilize a customized loss function to train the function on both 2D and BEV planes. Specifically, we integrate prior geometric relations [10, 20] between real and virtual ground planes into the training process. With the ground function, we can query the ground height at any location, which allows us to attach height information to the Radar data. Consequently, we can thus project the enhanced Radar points onto the 2D plane to obtain more accurate depth information, thereby assisting the IPM process. Secondly, roadside camera images contain rich background information, including the camera's relative pose to the ground [18, 23] . Therefore, we directly regress the transformation parameters in real time to refine the view transformations. To facilitate the learning of transformation parameters, we enable the network to regress the residuals of the parameterized homography matrix between 2D and BEV planes. These key designs in FARFusion V2 can boost the imageplane fusion pipeline proposed by FARFusion for far-range scenes. Moreover, considering various observation noises across these two sensors, we introduce an uncertainty-based depth fusion strategy into the image-plane fusion. This strategy aims to maximize the probability of obtaining the optimal depth value. Meanwhile, due to the absence of direct labels for uncertainty, we utilize a surrogate loss function to train the network for uncertainty estimation.In summary, we make the following contributions:• We propose a geometry-based Radar-camera fusion method for far-range object detection on the ground, namely FAR-Fusion V2. Our approach achieves excellent localization by integrating the ground information and a real-time transformation parameters estimation module in target-level fusion. • We also introduce an uncertainty-based depth fusion strategy into the fusion process to maximize the probability of obtaining the optimal depth value.• Extensive experiments are conducted on our roadside OWL testbed on an urban expressway. Results show that our method can improve AP BEV by absolutely 5.4%. Significantly, the approach can achieve 0.771𝑚 average location accuracy when the detection range is extended up to 500𝑚.Radar and camera fusion for autonomous driving. Featurelevel [22] and target-level fusion methods are the primary methods for integrating Radar and camera data in autonomous driving. We first introduce the feature-level methods, CenterFusion [24] first predicts 2D candidate positions of the targets and then utilizes the roi-based fusion strategy to associate them with Radar pillar features in 3D space via inverse perspective transformation. Finally, it refines the associated features using a refinement module. Cramnet [14] also applies such an association strategy to map the foreground pixels to 3D for point-wise fusion with foreground Radar points. Different from the roi-based fusion strategy, BEV-based fusion methods [43, 45] first lift the 2D image features to BEV features in 3D space and then fuse them with the Radar BEV features. In the target-level fusion methods [28, 31] , the fusion is performed on the target-level sensor outputs. Most methods [3, 6, 9, 16, 25] are based on Bayesian theory, Kalman Filtering, Dempster-Shafer theory, etc. Target-level fusion has been widely employed in practical scenarios thanks to the low computing cost.Radar and camera fusion for roadside perception. Nowadays, more research is dedicated to roadside dataset building [11, [37] [38] [39] , sensor calibration [8, 27] and object location [1] . However, there has been relatively little work on combining roadside radar and camera data. [1] utilizes a BEV-based method to fuse Radar and camera targets, it first maps the 2D points to BEV and associates two targets on the BEV plane, then uses a Gaussian mixture probability hypothesis density (GM-PHD) filter to integrate these two modalities. [8] also utilizes the BEV-based method to achieve spatiotemporal synchronization between Radar and camera coordinate systems. It first maps the 2D points to BEV and then employs the matched trajectory points to optimize the calibration parameter. [30] fuses roadside Radar and camera targets using the Kalman filter in the tracking stage. These studies primarily focus on near-range perception (less than 250 meters). Differently, FARFusion [20] proposes a far-range Radar and camera fusion method for 3D object detection, which first associates Radar-camera targets on the 2D plane and refines the transformation parameter on both 2D and BEV planes. In summary, most Radar and camera fusion methods for AD primarily use feature-level fusion strategies. Roadside Radar and camera fusion methods mainly utilize the target-level fusion strategy due to the lack of accessible open datasets in near range (<250m). By contrast, our FARFusion V2 focuses on roadside far-range (150 -500𝑚) perception, taking into account the key impacts of rough ground and sensor movements on the high-accuracy location.3 The FARFusion V2 Design 
preliminary and overview: 
 FARFusion V2 aims to enhance the precision of fusing Radar and camera targets in far-range roadside scenes. We first introduce the basic fusion pipeline designed for far-range scenarios in FARFusion [20] . FARFusion first projects the Radar-based target points (on the BEV plane) to the 2D image plane and then associates them with the camera-based object locations that are modeled as a CBM 1 point on each object. Subsequently, it maps the camera-based object locations to the BEV plane through IPM with the corresponding 1 CBM represents the bottom midpoint of the 2D bounding box detected by a pre-trained 2D detector, which is a good proxy of the car's location. Radar depth information. Finally, it engages a BEV tracking module to combine the outputs of the above 2D fusion module and generate target trajectories for traffic monitoring. Meanwhile, the fusion pipeline requires accurate transformations between Radar and pixel coordinate systems (CS). We follow FARFusion to represent the transformation parameters using a plane homography matrix M between Radar BEV and image planes. The coordinate systems are depicted in Fig. 2 on the OWL testbed deployed on an urban expressway with a detection area of 𝑌 = 150 -500𝑚 range.In the FARFusion, the BEV plane (Radar CS) corresponds to the ground surface, which assumes the ground is a plane without relief. This assumption may be inconsistent with actual circumstances, especially in far-range scenes, which has adverse effects on the projection process of Radar points and IPM process of camera CBM points. Therefore, we have revised this assumption to allow the ground to have arbitrary shapes in FARFusion V2. Moreover, FARFusion assumes that the camera is mounted in a pole without movements in a time window. However, we have observed the camera has little shake due to the windy weather, which influences the accuracy of the pre-calibrated M. Considering roadside camera images include the inherent pose information relative to the ground, we refine the M from each image in real time in FARFusion V2.Specifically, in our FARFusion V2, given a pre-calibrated M and a set 𝑋 = {𝑥 𝑖 𝑏𝑒𝑣 , 𝑥 𝑖 2𝐷 |𝑖 = 1...𝑁 } of 𝑁 pair detection points of vehicles for every frame, with BEV object coordinates 𝑥 𝑖 𝑏𝑒𝑣 ∈ R 2 in Radar CS and 2D pixel coordinates 𝑥 𝑖 2𝐷 ∈ R 2 , our goal is to estimate ground height and refine M in real time, then enhance the target-level Radar and camera fusion for far-range scenes. Here, considering it's usually hard to obtain the 3D ground truth of targets in the whole range of 𝑌 = 150 -550𝑚 in the Radar CS, we take fusion points (𝑋 𝑓 , 𝑌 𝑓 ) as 𝑥 𝑖 𝑏𝑒𝑣 and 2D CBM points (𝑈 𝑐𝑎𝑚 , 𝑉 𝑐𝑎𝑚 ) as 𝑥 𝑖 2𝐷 . The fusion pipeline in FARFusion provides these two types of points and their matching relations. Note that our FARFusion V2 attempts to construct data-driven surrogate training pipelines using prior geometric transformations and the outputs of FARFusion, instead of taking the outputs as the direct supervision. To this end, with the 𝑋 and M, our FARFusion V2 proposes three key designs: (1) A ground height network that uses a geometry-based training pipeline. ( 2 (1)We represent the ground height implicitly and construct a geometrybased training pipeline for ground height learning. We first assume there is a virtual ground plane shown in Fig. 2 , and then we utilize the differences between predicted positions based on the hypothesis and real positions to train the ground height network. Meanwhile, M denotes the transformation between 2D and virtual BEV planes.During the initial calibration [20] , the ground is assumed as a plane with a fixed ground height value to calibrate the M. Specifically, considering the requirements of practical deployment with arbitrary ground shapes, we first represent the ground height as follows:EQUATIONwhere in our case the height function is implicitly parameterized as a Multi-Layer Perceptron (MLP).Then given the pair fusion points (𝑋 𝑓 , 𝑌 𝑓 ) and 2D CBM points (𝑈 𝑐𝑎𝑚 , 𝑉 𝑐𝑎𝑚 ), we construct the loss function on both 2D and BEV virtual planes depicted in Fig. 3 .P 2 . Firstly, we start from the 2D CBM points (𝑈 𝑐𝑎𝑚 , 𝑉 𝑐𝑎𝑚 ). Based on the virtual ground-plane assumption, we map the 2D CBM points to the BEV plane via the IPM process:EQUATIONEQUATION)where H is the inverse of M, H 𝑖 denotes the i-th row of H, and ) is one-to-one as the intersection points of the single ray from camera optical center (with a height ℎ 𝑐 ) with the real ground and the virtual plane should be unique in the roadside scene. These points lie on a single ray shown in Fig. 2 . Inspired by the Gen-LaneNet [10] , we transfer the (𝑋 ′ 𝑐 , 𝑌 ′ 𝑐 ) on the virtual plane to the real position (𝑋 𝑐 , 𝑌 𝑐 ) as follows:𝑍 𝑐 = 1/(HEQUATIONEQUATIONwhere Δ𝑋 and Δ𝑌 are the translation deviations between Radar CS and ground CS shown in Fig. 2 , ℎ 𝑐 is the perpendicular distance from the camera's optical center to the ground. The variables Δ𝑋 , Δ𝑌 and ℎ 𝑐 are considered as learnable parameters. We consider the translation deviations because the geometry-based similar triangle relation (𝑋 𝑐 +Δ𝑋 𝑋 ′ 𝑐 +Δ𝑋 = ℎ 𝑐 -𝑍 𝑔 ℎ 𝑐) is only established with the ground CS while our BEV plane is in the Radar CS. Therefore we re-derive the Equ. 5 and Equ. 6. Then we take the fusion points (𝑋 𝑓 , 𝑌 𝑓 ) as the reference and calculate the first loss function on the BEV plane as:EQUATIONSecondly, we start from the fusion points (𝑋 𝑓 , 𝑌 𝑓 ) to calculate the re-projection errors on the 2D plane. In the training process, the matching relation of fusion points (𝑋 𝑓 , 𝑌 𝑓 ) and 2D CBM points (𝑈 𝑐𝑎𝑚 , 𝑉 𝑐𝑎𝑚 ) are known, therefore we can take the ground height 𝑍 𝑔 obtained in the calculation process of 𝐿 𝐵𝐸𝑉 to transfer the (𝑋 𝑓 , 𝑌 𝑓 ) from real position to the position on the virtual ground plane. The transformation process is formulated as follows:EQUATIONEQUATIONThen we can project the (𝑋 ′ 𝑓 , 𝑌 ′ 𝑓 ) on the virtual plane to the 2D plane with the homography matrix M as:EQUATION)EQUATION)Now we take the 2D CBM points (𝑈 𝑐𝑎𝑚 , 𝑉 𝑐𝑎𝑚 ) as reference and calculate the second loss function on the 2D plane as follows:EQUATION)The final geometric loss function can be calculated with all 𝑁 pair fusion and 2D CBM points by:EQUATION)Inspired by the FARFusion [20] , we apply the depth scaling strategy on the final geometric loss function due to the far-range detection requirements of 𝑌 = 150 -500𝑚 range. We also use the smooth 𝐿 1 loss on the 𝐿 2𝐷 and 𝐿 𝐵𝐸𝑉 , which is robust to outliers and easier for ground height training.Our camera data processing module takes high-resolution images as inputs. As mentioned before, far-range object detection highly relies on the homography matrix M for view transformations between 2D and virtual BEV planes. In practical deployment, the sensors may have unexpected movements due to windy weather. Therefore we first exploit a parameterized homography matrix estimation method in our camera data processing module depicted in Fig. 3(a) . Parameterized homography matrix estimation. The roadside sensors are fixed and installed on the pole, any deviations in unexpected movements should be measured relative to a fixed reference parameter, i.e., M. We directly regress the residuals relative to the initial M for every image. The image includes rich background information, which is helpful for estimating the camera pose relative to the ground. Specifically, we employ a 2D image backbone to extract the down-sampled image feature map from every image. Then the image feature map is flattened to a 1-D global feature 𝑀 𝐼 ∈ R 1×𝐶 . We transform the 𝑀 𝐼 to a feature embedding 𝑀 ′ 𝐼 ∈ R 1×𝐶 ′ for calibration using a MLP. The 𝑀 ′ 𝐼 is fed into different MLP layers to regress the residuals in parameterized representation relative to the M. We regress the scaling parameter (Δ𝑠 𝑥 , Δ𝑠 𝑦 ), rotation parameter Δ𝜃 and translation parameter (Δ𝑡 𝑥 , Δ𝑡 𝑦 ) separately. Finally, we derive the residual matrix ΔM as follows: 2D CBM points detection. We follow FARFusion [20] to detect the 2D CBM points using a pre-trained 2D detector.EQUATIONRadar can provide sparse point clouds and target-level results after clustering. We utilize target-level results that have more true positives as the inputs of the Radar data processing module, with each target represented by a point on the BEV plane. We first integrate height information into the Radar data. Given the Radar points (𝑋 𝑟 , 𝑌 𝑟 ) on the BEV plane, we approximately calculate the corresponding ground height 𝑍 𝑔 using height function 𝑓 in Equ. 2. This height value represents the Radar targets' height measurement, as our fusion process is performed on the ground. Then as shown in Fig. 2 , we normalize 𝑍 𝑔 as 𝑍 𝑛𝑜𝑟𝑚 𝑔 = (ℎ 𝑐 -𝑍 𝑔 )/ℎ 𝑐 . The normalization is necessary because we employ the homogeneous coordinate representation in the transformation between 2D and BEV planes. We next combine the Radar points and camera 2D CBM points through a 2-stage fusion pipeline on both 2D and BEV planes.On the 2D plane, we perform an uncertainty-based depth fusion between 2D CBM points and Radar points depicted in Fig. 3(c) . We first associate the Radar projection points with CBM points via the bipartite graph matching algorithm. After association, we have both matched point pairs and unmatched points. If a pair of points are matched, (𝑈 𝑐𝑎𝑚 , 𝑉 𝑐𝑎𝑚 ) and (𝑈 𝑟𝑎𝑑 , 𝑉 𝑟𝑎𝑑 ), we now regard the two points as a dual-modal point and map this point to the 3D space. For the matched CBM point (𝑈 𝑐𝑎𝑚 , 𝑉 𝑐𝑎𝑚 ), we have two types of depth values, i.e., Radar projection depth 𝑍 𝑟𝑎𝑑 and image depth 𝑍 𝑐 . Here, we calculate 𝑍 𝑐 by the IPM process𝑍 𝑐 = 1/H ′ 3 • (𝑈 𝑐𝑎𝑚 , 𝑉 𝑐𝑎𝑚 , 1), where H ′ is the inverse of M ′ . We assume that the two depths 𝑍 𝑟𝑎𝑑 and 𝑍 𝑐 are independent because they are obtained from different methods. According to Bayes' theorem, we can derive the posterior probability of this 2D CBM point's real depth 𝐷 as below:EQUATIONFurthermore, we assume the observation noises of 𝑍 𝑟𝑎𝑑 and 𝑍 𝑐 are subject to Gaussian distributions 𝑁 (0, 𝜎 2 𝑟 ) and 𝑁 (0, 𝜎 2 𝑐 ). According to the maximum of the posterior probability 𝑃 (𝐷 |𝑍 𝑐 , 𝑍 𝑟𝑎𝑑 ), we can obtain the following mathematical relation:EQUATION)Thus, we can calculate the derivative of the final expression (denoted by 𝐿(𝐷)) in Equ. 17 concerning 𝐷. By making 𝐿(𝐷) ′ = 0, we can calculate the optimal 𝐷 as follows:EQUATION)Through the mathematical derivation, we thus obtain the optimal fusion depth value in Equ. EQUATION)EQUATION)As for the training process, we employ a surrogate loss function.According to the Equ. 17, we construct the loss function as follows:EQUATIONwhere 𝛼 and 𝛽 are the hyperparameters and we take the fusion points as the reference to calculate the 𝐷 by 𝐷 = M ′ 3 •(𝑋 𝑓 , 𝑌 𝑓 , 𝑍 𝑛𝑜𝑟𝑚). Note that the calculation of 𝐷, 𝑍 𝑟𝑎𝑑 and 𝑍 𝑐 are detached from the computational graph in Sec. 3.2 and Sec. 3.3. 𝐿 𝜎 is only employed to train the two uncertainty estimation MLP networks. This training process is depicted in Fig. 3 .P 1 .Additionally, for unmatched 2D CBM points, we utilize the IPM process to map the CBM points to the virtual BEV plane. Then we calculate the corresponding ground heights of the real positions via the height function in Equ. 2, and transfer the IPM points to the real positions using the Equ. 5 and Equ. 6. We also reserve the unmatched Radar points. We then feed all these outputs to the subsequent BEV target tracking module.Finally, on the BEV plane, we adopt the common target-level fusion methods [20, 37] , utilizing a Kalman-based tracking module to combine the outputs from the 2D depth fusion module. 
implementation and evaluation: 
 In this section, we introduce our implementation details and evaluation results.OWL testbed and our benchmark. As shown in Fig. 2 , we have developed a testbed named OWL by deploying smart poles on an LiDAR provides the ground truth data of multiple cars in this range. *We follow [1] to exploit the same fusion method of the BEV association and "prediction-update" tracking pipeline as a comparison. Meanwhile, we use the Kalman filter and filter out the unmatched camera targets due to the large location errors of the IPM points in the tracking pipeline.urban expressway with an 80𝑘𝑚/ℎ speed limit. Specifically, we deploy a long-range millimeter wave Radar, a HD monocular camera, a spinning 80-beam LiDAR on each pole, and two vehicle-mounted RTK-GPS systems. Here, LiDAR and RTK-GPS are deployed to provide ground truth data only for evaluation. LiDAR generates the data for vehicles in 𝑌 = 150 -330𝑚 range, while GPS provides the data for the two experimental vehicles in 𝑌 = 330 -500𝑚 range.On this testbed, we establish a roadside benchmark for far-range scenarios. We follow [20] to perform spatiotemporal synchronization on our roadside dataset. Then We split the dataset into training (65%, first 65% frames per split) and validation splits (35%, other 35% frames per split) across three splits in different time intervals. Every split contains over 8000 frames and 39000 car samples. Metrics. We use average precision (AP) and mean absolute error (MAE) of location to evaluate the final detection results in our benchmark. We assign the detection results to ground truths of vehicles based on the center distances between them, using different distance thresholds. We set the (0.5m, 1.0m) (i.e., 𝑑𝑖𝑠 0.5 ), (1.0m, 2.0m) (i.e., 𝑑𝑖𝑠 1.0 ), (1.5m, 3.0m) (i.e., 𝑑𝑖𝑠 1.5 ) and (2.0m, 4.0m) (i.e., 𝑑𝑖𝑠 2.0 ) thresholds in (𝑋, 𝑌 ) axes (refer to Fig. 2 ) in the BEV's Radar coordinate system, because the length-width ratio of the vehicle is approximately 2 : 1. We report location MAEs in the lateral (Lat.) 𝑋 axis, longitudinal (Long.) 𝑌 axis and Euclidean distance (Euc.) from the bird's eye view with the threshold 𝑑𝑖𝑠 1.5 . We calculate the AP with location threshold values of {𝑑𝑖𝑠 0.5 , 𝑑𝑖𝑠 1.0 , 𝑑𝑖𝑠 1.5 , 𝑑𝑖𝑠 2.0 } and then average the values across different thresholds. Network structure. The ground height and uncertainty estimation networks are parameterized as different MLPs with three layers and 32 channels of the hidden layer. In the parameterized homography matrix estimation module, we take the image with a resolution of 3840 × 2160 as input. Then we utilize the ResNet-18 [12] as the image backbone to extract the image feature for transformation parameters estimation. The hidden layers of different parameterized homography matrix estimation MLPs are configured with 256 channels each. We employ the pre-trained YOLOE [36] to detect the 2D bounding boxes of vehicles. We don't use the same image backbone for both calibration and object detection tasks because they separately focus on static and dynamic targets. Training details. We decouple the training processes of the ground height network and the parameterized M estimation module in two steps. Because we find that it's hard for the network to converge when optimizing them simultaneously. In the first step, we train the ground height network and the learnable parameters of the camera/Radar height and coordinate deviations for 100 epochs. Then we use pre-trained weights and freeze the weights of the ground height network to train the parameterized M estimation We report the performance of our FARFusion V2 on the validation set for comparison with other methods and carefully investigate the effect of different modules. The results are evaluated using the average precision (AP) and mean absolute error (MAE) of location as illustrated in Sec. 4.1.Comparison results on the validation set. We first compare the performance of different methods in 𝑌 = 150 -330𝑚 range in Tab. 1: (1) Radar Only (i.e., obtaining a target's position using the Radar point location on the BEV plane), (2) Image Only (i.e., obtaining a target's position by mapping its CBM point to BEV using the plane-assumed IPM), (3) robustBEVfus [1] (i.e., the fusion of the previous two locations in BEV), (4) FARFusion [20] (i.e., the target-level fusion of the Radar and CBM points on both 2D and BEV planes), and (5) our FARFusion V2. Among them, the first two are single modal while the latter three are fusion-based. The results show our FARFusion V2 has the highest AP BEV among all methods.In addition, we have the following observations. Our FARFusion V2 has the highest AP BEV on the validation set, improving the average detection performance AP BEV by 5.4% than FARFusion and 28% than robustBEVfus. This demonstrates that 2D association is more effective in far-range scenes than BEV association used in robustBEVfus [1] . Moreover, our FARFusion V2 considers the impact of ground and sensor motions, achieving an improvement of 0.23𝑚 in location accuracy compared to FARFusion.Next, we perform a set of ablation studies in both 𝑌 = 150-330𝑚 and 𝑌 = 330 -500𝑚 ranges and summarize the results in Tab. 2 and Tab. 3. We take the FARFusion as our baseline for comparison. Impact of the ground height network. We first investigate the impact of the ground height network. As shown in Tab. 2, we obtain a 2.4% improvement in average AP BEV and a 0.194𝑚 improvement in location accuracy when adding the ground height network (GH) in the target-level fusion. The ground height network can assign height values to the Radar data, thereby assisting the IPM process Impact of the parameterized homography matrix estimation module. We also show the results with the parameterized homography matrix estimation module (PE) in the third row in Tab. 2. However, we observe that the effect of this module in the 𝑌 = 150 -330𝑚 range is minimal with an improvement of only 0.2% in AP BEV . This is because the sensor's motion has a greater influence at longer distances as illustrated in the introduction. Therefore, we also evaluate the effect of this module in the 𝑌 = 330 -500𝑚 range in Tab. 3. The results show adding this module yields a 1.5% improvement in AP BEV . Impact of the uncertainty estimation network. We finally evaluate the effect of the uncertainty estimation network (UE). In the last row in Tab. 2, we obtain a 2.0% improvement in AP BEV . This demonstrates the depth values of the CBM points are more accurate when using the uncertainty-based weighting between the depths obtained via the Radar projection and those calculated by IPM than using only the depth obtained via the Radar projection directly. However, we also observe that the improvement of this module is minimal in 𝑌 = 330 -500𝑚 range in Tab. 3. This is because the IPM points have much higher location errors in this range. The uncertainty of the depth calculated by IPM is greater than the uncertainty of depth obtained by Radar projection. Therefore, the final weighted depth is highly dependent on the depth obtained through Radar projection, resulting in the performance being similar to that achieved without this uncertainty estimation network. We will analyse the uncertainty network in more detail in Sec. 4.3.Longer ranges with GPS as ground truth. Finally, we extend the detection range to 𝑌 = 330 -500𝑚 to compare the performances of different methods in Tab. 4. Our FARFusion V2 still has the highest performance. We obtain a location accuracy of 0.771𝑚 for the experimentation vehicles. This range has a high requirement for accurate view transformations. The ground height network and homography matrix estimation module play important roles in this range, as demonstrated in the preceding ablation studies.In this subsection, we delve into FARFusion V2 to study the ground height network and uncertainty estimation network qualitatively.What did the ground height network learn? To investigate the outputs of the ground height network, we divide this area of (-15𝑚 ≤ 𝑋 ≤ 10𝑚, 200𝑚 ≤ 𝑌 ≤ 700𝑚) range in Radar CS into grids of uniform size of 2𝑚 × 20𝑚. Then we take the grid points as . We observe that in the 𝑌 = 300 -600𝑚 range, there is an upward trend, while in the 𝑌 > 600𝑚 range, it shows a downward trend in ground height. This trend is consistent with the ground's fluctuations in Fig. 4 (a). However, we also observe that there is an unexpected sudden drop in the ground's height in the position of 𝑌 = 200𝑚. We believe this is because we use fusion points, not ground truths, to train the network for ground height estimation. In practical scenarios, the fusion points are subject to inevitable detection noise with outliers, which causes the trained network to be imperfect. Even so, the overall surface contour of the output of the ground height network closely matches the actual ground.The analysis of the uncertainty estimation network. The measurement accuracy of the sensor is highly relative to the detection range. The measurement noise usually becomes larger as the distance increases. To analyse the uncertainty predicted by the uncertainty estimation networks of Radar projection depth and IPM-calculated depth concerning the distance, we select matched Radar and 2D CBM points from 1000 frames randomly as the input of the network. Then we plot the uncertainty values 𝜎 𝑟 and 𝜎 𝑐 concerning the 𝑌 coordinate of the BEV's Radar CS and mark the mean values in several locations in Fig. 5 . We observe that the Radar projection depth 2 is more reliable than the IPM-calculated depth in the whole range. Meanwhile, the magnitude of 𝜎 𝑐 is significantly greater than that of 𝜎 𝑟 in the 𝑌 > 300𝑚 range. This indicates that the camera's IPM performs poorly in 𝑌 > 300𝑚 range, where the 2 There is a downward trend of 𝜎 𝑟 in 𝑌 > 350𝑚 range due to the use of fusion points as the ground truth for network training. ) is only established with the ground CS while our BEV plane is in the Radar CS. We conduct an ablation study to analyse the effect of deviations in Tab. 5. The "base" represents we do not use translation deviations. The results show the performance is better with the translation deviations.The qualitative results in Radar CS of FARFusion and FARFusion V2. Finally, we show the qualitative results on the BEV plane of FARFusion and FARFusion V2 in Fig. 6 . We see that the FAR-Fusion V2 has higher location accuracy, especially the IPM points highlighted in blue are adjusted to be closer to the accurate fusion points at larger ranges. 
conclusion: 
 In this paper, to achieve high-accuracy location in far-range scenes, we exploit a geometry-based Radar and camera fusion method FARFusion V2 in target level on the ground for roadside 3D object detection. We find that ground height and sensor motion are important factors influencing the performance of roadside Radar and camera fusion. Therefore, we represent the ground height implicitly and employ a geometry-based transformation function for training. We also engage a parameterized homography matrix estimation module to refine the transformation parameters in real time. Additionally, we further improve the accuracy of depth values using an uncertainty-based depth fusion approach, supported by a mathematical theory derivation. With these designs, we achieve high-accuracy location performance for far-range object detection. 
: 
 module is further introduced to refine the view transformation processes. Moreover, considering various measurement noises across these two sensors, we introduce an uncertainty-based depth fusion strategy into the 2D fusion process to maximize the probability of obtaining the optimal depth value. Extensive experiments are conducted on our collected roadside OWL benchmark, demonstrating the excellent localization capacity of FARFusion V2 in far-range scenarios. Our method achieves an average location accuracy of 0.771m when we extend the detection range up to 500m.• Computing methodologies → Object detection. 
 
 Appendix:  


Decision: The paper presents FARFusion V2, a geometry-based radar-camera fusion method designed to enhance far-range 3D object detection for roadside traffic surveillance. The proposed method addresses real-world challenges such as uneven terrain and sensor movement, demonstrating significant improvements in localization accuracy. The reviewers acknowledged the practicality and robustness of the approach, along with the extensive experiments validating its effectiveness. Concerns regarding the explanation of certain aspects and the need for additional evaluations were addressed in the authors' rebuttal. Therefore, I recommend accepting this submission.

Reviews: review: paper_summary: paper_summary: This paper proposes FARFusion V2, which solves one problem of the existing FARFusion.
This method estimates the position and speed of a moving vehicle using fixed cameras and millimeter-wave radar, such as those used in road surveillance systems.
The proposed method achieves higher accuracy than when the shape and relative position of the road surface are fixed by geometrically considering the shift of the viewpoint due to the vibration of the system.
strength_weakness: strength_weakness: This is a practical method that addresses problems that arise in real-world environments, whereas many studies only pursue behavior in an ideal environment.
Furthermore, rather than leaving everything to machine learning, the proposed method attempts to solve the problem by considering geometric constraints.
limitations: limitations: -3D Object Detection-
Although it is described as an "object," it is thought to be specialized for "vehicles."

-machine learning-
Even though geometric constraints are taken into account, it is still insufficient to explain the advantages of going through machine learning.
It is necessary to explain whether the proposed method is effective in various environments, since the generalization performance problem arises as soon as ML is used.
(For example, does it work well when the road environment is not a wide road like the one shown in Figure 4, or when it is not a normal day but in rough weather with severe shaking?)

-OWL testbed-
This appears to be the authors' own dataset.
Will it not be made public?
If not, it cannot be used as a benchmark, so it is necessary to mention the behavior of the proposed method in areas other than a small number of ``urban expressways''.

-Figure 1(b)-
"Unexpected motion" means that only the angle from the sensor mount has changed, but in "real-time unexpected sensor motions still exist due to the windy weather," the rotation from the base of the pole is expected to be greater, so this explanation seems strange.
paper_summary: paper_summary: This paper aims to address the impact of uneven terrain and sensor movement on the Radar-camera fusion performance in the far-range 3D object detection task.
The main contributions include a geometric-based ground height learning method, a real-time transformation parameter estimation module, and an uncertainty-based depth fusion strategy.
Experiments on the roadside OWL dataset shows that it improves the object detection accuracy and the location accuracy in far-range.
strength_weakness: strength_weakness: The proposed method uses a geometric-based ground height learning network to reduce the influence of uneven terrain.
It also considers the sensor movement in windy weather and uses a real-time transformation parameter estimation module to improve the homography matrix. 
Considering the observation noises from two sensors, the uncertainty-based depth fusion strategy is applied.
limitations: limitations: 1. According to Figure 1, why does the sensor only move in one direction instead of moving in all three directions? Is it related to the fixing method of the sensor?
2. The pair fusion points (Xf, Yf) and 2D CBM points (Ucam, Vcam) are obtained based on the pre-calibrated M. What is the impact of the uneven terrain and sensor movement on the pair fusion points and the subsequent experiment?
3. Can the height change in the X direction also be reflected correctly as in the Y direction. The Figure 4 only reflects the height change in the Y direction.
4. For Table 2, it is better to add the results of PE, UE, GH+UE and PE+UE, to more clearly demonstrate the effectiveness and necessity of each module.
paper_summary: paper_summary: This paper proposes FARFusion V2 to estimate ground height with arbitrary shapes and regress the transformation parameters to further refine the view transformations. It also includes an uncertainty-based depth fusion strategy to maximize the probability of obtaining the optimal depth value.
strength_weakness: strength_weakness: - The paper is largely clearly written and makes an easy read.
- The authors solve practical issues on top of FARFusion and show improvements over baselines.
limitations: limitations: - In both the abstract and the introduction, the authors do not explain the extent to which uneven terrain and sensor movement affect the precision of object detection. Without this information, we cannot determine if addressing these issues is necessary.

- Similarly, I am curious whether the 5.4% improvement in AP over FARFusion is substantial enough to justify this approach.

- Since the authors propose a real-time parameterized transformation parameters estimation module, it is essential to report its latency and the required latency. I’m wondering If the latency is small enough to process the real-time sensor motions.

- The proposed approach must demonstrate adaptability to various scenarios and conditions. Since the testbed only includes one type of scenario (urban expressway), it is necessary to apply the approach to other scenarios, such as roads with continuous changes in ground height. Additionally, how large can sensor motions be while still allowing the approach to provide accurate estimations?