CalibRBEV: Multi-Camera Calibration via Reversed Bird's-eye-view Representations for Autonomous Driving

Authors: cal  params
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-27 03:10:17

Abstract: Camera calibration consists of determining the intrinsic and extrinsic parameters of an imaging system, which forms the fundamental basis for various computer vision tasks and applications, e.g., robotics and autonomous driving (AD). However, prevailing camera calibration models pose a time-consuming and labor-intensive off-board process particularly in mass production settings, while simultaneously lacking exploration of real-world autonomous driving scenarios. To this end, in this paper, inspired by recent advancements in bird's-eye-view (BEV) perception models, we proposes a novel automatic multi-camera Calibration method via Reversed BEV representations for autonomous driving, termed CalibRBEV. Specifically, the proposed CalibRBEV model primarily comprises two stages. Initially, we innovatively reverse the BEV perception pipeline, reconstructing bounding boxes through an attention auto-encoder module to fully extract the latent reversed BEV representations. Subsequently, the obtained representations from encoder are interacted with the surrounding multi-view image features for further refinement and calibration parameters prediction. Extensive experimental results on nuScenes and Waymo datasets validate the effectiveness of our proposed model.

Main paper: 
 related work: 
 In this section, we offer an extensive overview of related work concerning our proposed model, primarily focusing on three aspects: camera calibration, bird's-eye-view (BEV) perception models, and autonomous driving scenarios.Camera calibration can be broadly categorized into two main directions [36, 44] : traditional hand-crafted methods and deep learningbased models. This subsection will delve into each direction separately.2.1.1 Traditional Models. In traditional camera calibration models, calibration generally involves using established mathematical models and algorithms [1, 7, 12, 16, 19, 20, 41, 45, 61] to estimate camera parameters, which can be further categorized into three main categories [36] , namely, calibration-plane-based models [2, 19, 45, 61] , geometric-prior-based models [1, 7] , and selfcalibration models [15, 16, 20, 59 ].(i) The calibration-plane-based models typically necessitate the use of standard references (e.g., calibration target: checkerboard) in the world coordinate system to estimate parameters. A simple calibration pattern comprising equally spaced dots is proposed [45] for fish-eye lens camera. Zhang et al. [61] further introduced a simple and effective camera calibration model, wherein parameters are estimated through the utilization of key points from different viewpoints on the calibration plane. (ii) The geometric-prior-based models usually depend on geometric priors and properties for the prediction of calibration parameters. For example, the projective invariant properties [1] is employed in central catadioptric systems, while user-specified salient scene regions [7] are preserved according to human vision tendencies. (iii) The self-calibration models mainly directly take multiple images captured from different camera orientations to estimate the parameters. The epipolar structure of image pairs is leveraged in [16] and image correspondences is utilized in [20] . LiDAR edges information [59] is further used for automatic extrinsic calibration.In general, these traditional calibration methods heavily rely on known scenes and mathematical models, which greatly limit their ability to generalize in complex scenarios. Furthermore, they necessitate the design of specific calibration plane and extensive manual intervention, inevitably adding to costs and proving unsuitable for mass production scenarios.Recently, with the advancement of deep learning [14, 21, 32] , a novel learning-based calibration method has emerged, demonstrating promising potential as it directly utilizes deep neural networks (DNNs) to fit the calibration targets. Following the [36] and considering the camera model, deep learningbased calibration methods are categorized into four groups: standard model [9, 31, 50, 54] , distortion model [3, 8, 17, 60] , cross-view model [13, 37, 57] , and cross-sensor model [18, 46, 62, 64] .(i) The standard model aims to estimate the intrinsic and extrinsic parameters of the camera. Initially, DeepFocal [54] presented pioneering learning-based work to predict the focal length of cameras. The extrinsic parameters are similarly obtained through regression and are commonly regarded as camera pose estimation [9, 31] . Further techniques [4, 22, 50] have been suggested for the simultaneous estimation of both intrinsic and extrinsic parameters. (ii) The distortion model primarily focuses on calibrating radial distortion and roll shutter distortion. DeepCalib [3] firstly employed DNNs to regressively predict the camera distortion parameters. Later, more robust networks and training strategies are utilized to improve calibration performance [8, 17] . (iii) The cross-view model extends the single-camera settings to the more intricate multi-camera calibration scenarios, such as homography matrix prediction [13] and multi-calibration parameters prediction [37, 57] . (iv) The crosssensor model aims to achieve multi-sensor calibration, primarily aligning camera/LiDAR coordinates [18, 42, 62, 64] .However, despite the potential exhibited by the aforementioned learning-based camera calibration methods across diverse scenarios, further exploration is required in the challenging scenarios of autonomous driving. In this paper, we innovatively introduce the concept of BEV perception and incorporate reversed BEV features to directly predict the multi-view camera parameters in real-world autonomous driving scenarios.The BEV perception models [33] have garnered widespread attention in the research community for its benefits derived from a unified perspective and integration of multi-view or multi-sensor information, particularly in the context of AD scenarios. It is advisable to refer to the comprehensive review [33] to gain a comprehensive understanding of BEV perception models. Among these methods, perception strategies based on transformers have demonstrated outstanding performance [14, 49] .Initially, DETR [6] pioneered the utilization of the transformer architecture for the 2D object detection task. Given that crossattention entails prolonged training periods, Deformable DETR [63] further enhances model efficiency by integrating deformable attention mechanisms. Inspired by the aforementioned 2D detection models, DETR3D [52] is a classic model that generalizes BEV method from 2D to 3D detection based on DETR and Deformable DETR.Following the DETR3D, BEV models [25, 35, 38, 39] have experienced rapid development. PETR [38] and BEVDet [25] further improve the performance based on DETR3D. Then, BEVFormer [35] , PETRv2 [39] , BEVDet4D [24] , and BEVDepth [34] exploit temporal features in multi-camera 3D object detection, and achieve significant improvements over single-frame methods. Moreover, BEVFusion [40] and FUTR3D [11] further fuse multi-modal features of various sensors such as camera and LiDAR into BEV features to boost the performance.Our primary focus lies within methods pertaining to multi-view images BEV perception models, such as BEVFormer [35] . To ensure accurate detection, the utilization of multiple camera calibration parameters is imperative for multi-image inputs. In this paper, unlike above mentioned forward BEV perception models, we innovatively reverse the BEV pipeline to accomplish the camera calibration task.Modern autonomous driving (AD) systems are usually divided into several modules including perception [10, 33] , prediction [27] [28] [29] , planning [48] , and etc. Though they have the advantages of decoupled developments and explainability, module-based systems suffer from cumulative errors and inconsistent optimization objectives. Recently, end-to-end AD [23, 26, 30, 56 ] becomes a hot topic, which aims to build a fully differentiable systems for AD so as to enjoy the benefits of data-driven scalable paradigm [55] . Besides, the recent success of large language models (LLM) has inspired the community to explore the possibility of utilizing the strong common sense within LLM for AD [58] .In this paper, camera calibration emerges as a fundamental task in AD scenarios aimed at ensuring accurate interpretation and understanding of multi-view multimedia images captured by cameras, thereby achieving precise environmental perception, which is critical for ensuring the safety and optimal performance of AD systems.  
method 3.1 preliminaries: 
 EQUATIONwhere 𝑓 𝑥 , 𝑓 𝑦 denote the focal lengths, and 𝑐 𝑥 , 𝑐 𝑦 denote the coordinates of the principal point. The extrinsic parameter E is used to transform images from the world coordinate system to the camera coordinate system,EQUATIONwhere R denotes the rotation matrix and t denotes the translation vector.Parameters. Here, we firstly present an overview of the BEV representations [35, 53] and then discuss their close relationship with camera calibration parameters.The BEV features are defined as a pre-defined grid anchor, portraying the environment as a bird's-eye view, and are represented by a parameter matrix Q ∈ R 𝐻 ×𝑊 ×𝐶 . The height 𝐻 and width 𝑊 denote the region centered around the ego vehicle, and 𝐶 denote the number of channels in the coordinate region.To obtain the BEV features of each position, it is necessary to map the multi-view image features to the corresponding anchor positions through the intrinsic and extrinsic calibration parameters [35] . Specifically, suppose that the real-world position (𝑥 ′ , 𝑦 ′ ) corresponding to the queryEQUATIONwhere 𝑠 denotes the grid size in the real-world corresponding to the BEV features. In addition, it is essential to pre-define a set of anchors {𝑧 ′ 𝑗 } 𝑁 ref 𝑗=1 to describe the heights along the 𝑧-axis in the real environment. Finally, we obtain a series of 3D reference points (𝑥 ′ , 𝑦 ′ , 𝑧 ′ 𝑗 ) corresponding to the BEV features Q 𝑝 . According to the calibration parameters in Eqs. 1 and 2, we have the following projection mapping relationships,EQUATIONwhere (𝑥 𝑖 𝑗 , 𝑦 𝑖 𝑗 ) denotes the pixel coordinates of the 3D reference points projected on the 𝑖-th camera. I 𝑖 and E 𝑖 are the calibration parameters of the 𝑖-th camera. Note that an additional group of zerocolumn vectors needs to be included in I 𝑖 to ensure compliance with matrix dimensions. For simplicity, this mapping process is reformulated as follows:EQUATION3.2.1 Overview Architecture. The overall architecture of our proposed CalibRBEV model is shown in Fig. 2 , comprises three main components: the backbone network, the predictive calibration parameter module, and auto-encoder reconstruction module. Specifically, initially, the multi-view images captured from multiple cameras are processed through a backbone network to obtain latent feature representations of the surrounding environment. Then, the auto-encoder module is employed to reconstruct 3D bounding boxes, yielding a robust reversed BEV representations,and subsequently preparing for the calibration module. Ultimately, we reverse the entire pipeline of the BEV perception model, which accepts 3D bounding boxes and above multi-view image features as input for attention interaction to predict the camera calibration parameters.Module. Initially, we need to extract features from the surround view images. As show in 2(a), given a set of multi-view images 𝑥 img = {𝑥 𝑖 img } 𝑁 view 𝑖=1 captured by a multi-camera system on a vehicle, a backbone network (e.g. ResNet [21] ) is employed to extract a set of multi-view lower-resolution image features 𝐹 = {𝐹 𝑖 } 𝑁 view 𝑖=1 , where 𝑁 view denotes the total number of cameras.The calibration encoder module is utilized simultaneously for predicting calibration parameters and auto-encoder reconstruction. Structurally equivalent, the sole distinction lies in the initialization frozen learnable parameters of the former, derived from pre-training via autoencoder reconstruction.In order to extract the information from 3D bounding boxes comprehensively, we introduce a bounding box coder initially to enable feature mapping. Formally, we initialize a code map 𝑪 akin to BEV features as discussed in Sec. 3.1.2 to assimilate information pertaining to the 3D bounding boxes. Suppose that the groundtruth bounding box labels are denoted by 𝑏 bbox = {𝑏 𝑖 } 𝑀 𝑖=1 ∈ R 9 and the categorical labels are represented by 𝑐 bbox = {𝑐 𝑖 } 𝑀 𝑖=1 ∈ Z, where 𝑀 denotes the number of objects. Then, we directly concatenate them and pass them through a linear layer to obtain the feature representation, which is then mapped onto the corresponding code map.EQUATIONwhere [; ] denotes the concatenate operation, and 𝑝 𝑖 refers to the location of the 𝑖-th object on the code map. Then, a set of 𝑁 self-attention blocks is employed to obtain preliminary reversed BEV features 𝑩. Obviously, as discussed in Sec. 3.1.2, both the BEV features and the multi-view image features entail a considerable number of parameters, leading to substantial computational overhead when directly computing crossmodal attention. In this paper, we build upon the deformable crossattention [63] instead, which is defined as follows:EQUATIONwhere 𝒒, 𝒑, and 𝒙 denote the query element, 2D reference points, and the input feature map, respectively. 𝑁 head and 𝑁 key are the total number of attention heads and the sample points. Δ𝒑 𝑖 𝑗 is the sample offset of the 𝑗-th sample point in the 𝑖-th attention head. 𝐴 𝑖 𝑗 is the attention weight of the 𝑗-th sample point in the 𝑖-th head. 𝑾 𝑖 and 𝑾 ′ 𝑖 are learnable weights. The self-attention mechanism is constructed upon the deformable attention mechanism outlined in Eq. 7, thereby facilitating the generation of BEV features through enhanced interaction with adjacent local information. Moreover, the mechanism entails querying code map features and the reversed BEV query itself.SelfAttn(𝑸, 𝑝, 𝑪) = DeformAttn(𝑸 𝑝 , 𝑝, 𝑸)+DeformAttn(𝑸 𝑝 , 𝑝, 𝑪), (8) where 𝑸 denotes the learnable query, 𝑪 denotes the code map from bounding box as discussed in Eq. 6. The full self-attention block also encompasses both a feed-forward network (FFN) and normalization procedures, ultimately yielding reversed BEV features 𝑩 after stacking 𝑁 times.Particularly, as discussed at the beginning of this subsection, such calibration encoder module undergoes pre-training within the auto-encoder, followed by the seamless transfer of its parameters to the calibration prediction module, where the inherited parameters are entirely frozen to preserve the capability for reversed BEV feature extraction. We finally summarize the above reversed BEV feature extraction process as follows:EQUATIONOnce the reversed BEV features are obtained, the calibration decoder module takes in a novel learnable BEV queries 𝑸 ′ and the reversed BEV features 𝑩, then merges with multi-view image features {𝐹 𝑖 } 𝑁 view 𝑖=1 from backbone network, as discussed in Sec. 3.2.2, ultimately yielding the enhanced reversed BEV feature 𝑩 ′ to predict the calibration parameters.In detail, the space surrounding the vehicle is divided into 𝐻 ×𝑊 cells, with each cell corresponding to a BEV feature. Each BEV feature only utilizes the approximate position of the corresponding 3D space projected onto a 2D image as a reference point. Concurrently, we also require interaction with multi-view image features through multi-view cameras. Finally, the cross-attention mechanism is formulated as follows:EQUATION) where V hit denotes the set of image features mapped correspondingly to the BEV features P (𝑝, 𝑖, 𝑗) according to Eqs. 4 and 5. 𝐹 𝑖 denotes the 𝑖-th camera image feature as discussed in Sec. 3.2.2. Here, the mapping relationship in attention mechanism is established directly using uniform average calibration parameters. Overall, in the calibration decoder module, there are concurrent implementations of self-attention in Eq. 8 and cross-attention in Eq. 10, alongside a FFN. This whole attention module undergoes 𝑀 iterations of stacking, eventually yielding refined reversed BEV feature 𝑩 ′ .Following this, the refined BEV feature 𝑩 ′ is further subjected to a conventional transfromer decoder process within which the target feature 𝑻 is generated. The target feature 𝑻 is utilized across three distinct fully connection branches, each serving to predict the calibration parameters.EQUATION)We summarize above calibration decoder and final prediction header as follows.𝑩 ′ = CalibDecoder(𝑸 ′ , 𝑩),EQUATIONEQUATION3.2.5 Auto-encoder Module. In order to proactively obtain robust reversed BEV representations, we propose employing an attentionbased auto-encoder's reconstruction strategy for pre-training, as shown in Fig. 2(c ). For the encoder component within the autoencoder architecture, it adopts the same architecture as calibration encoder in Eq. 9.𝑩 = AEEncoder(𝑸, 𝑏 bbox , 𝑐 bbox ) = CalibEncoder(𝑸, 𝑏 bbox , 𝑐 bbox ). ( 14) For the decoder component, the reversed BEV features 𝑩 are supposed to output to a regular attention object detection decoder to generate 3D object bounding boxes again. We apply a similar self-attention mechanism in Eq. 8 and stack 𝐾 layers to obtain the final output of the 3D bounding boxes.EQUATION)3.2.6 Loss Functions. Our training procedure consists of two stages as shown in Fig. 2(d) . ( 1) reconstruction pre-training of the autoencoder module and (2) train the calibration module while freezing the encoder module. For the former, focal loss and L1 loss are used for classification and bounding box regression, respectively.EQUATIONEQUATIONwhere 𝑝 𝑐 ′ bbox denotes the estimated probability of category 𝑐 ′ bbox . 𝛼 and 𝛾 are the hyper-parameters. For the latter, all the calibration Table 1 : Comparison results of parameters prediction errors on nuScenes dataset. CalibRBEV-𝑁 refers to our model trained with 𝑁 cameras dataset. Here, t, R, and I denote the distance error of translation vector, angle error of rotation matrix, and error percentage of intrinsic parameters, respectively. f and c are the average error percentage of the focal length and the principal point. The symbol ↓ indicates that lower values represent better performance. Method t (𝑚) ↓ R ( • ) ↓ I (%) ↓ f (%) ↓ c (%) ↓ DeepPTZ [EQUATIONwhere ∥ • ∥ 1 here denotes the matrix element-wise norm, and the rotation matrix is represented using quaternions.Once the model training is complete, given a single-frame multi-view images and on-board bounding box from LiDAR sensor input, we can predict the intrinsic and extrinsic parameters using the calibration prediction module as shown in Fig. 2(e ). This process eliminates the need for additional complex manual procedures or multi-frame inputs, significantly alleviating the cost burden of calibration tasks. Furthermore, a potential avenue for extension lies in the real-time implementation of our approach directly within on-board vehicles, thereby significantly enhancing its applicability and utility in AD scenarios. 
dataset: 
 We conduct experiments on nuScenes [5] and Waymo [47] datasets in this paper for calibration parameters evaluation.The nuScenes Dataset. The nuScenes dataset consists of 1, 000 sequences, each spanning approximately 20 seconds and captured at a sampling rate of 20 frames per second, where the key samples are annotated at 2Hz. In total, there exist 28𝑘, 6𝑘, and 6𝑘 annotated instances for training, validation, and testing, respectively. In addition, each sample contains RGB images from 6 cameras: front_left, front, front_right, back_left, back, back_right.Hence, the nuScenes dataset encompasses a total of 6000 calibrated cameras. Among these, the subset of 900 cameras is withheld from public access, designated as the challenge test dataset, while an additional 900 cameras are specifically used for validation. And the remaining 4200 cameras are used for training. The error in calibration parameters is utilized as the primary evaluation metric. Moreover, the 3D object detection metric [35] , such as mAP, mATE, mASE, mAOE, etc, are also used for comprehensive evaluation.The ResNet101-DCN [21] pre-trained with FCOS3D [51] is implemented as the backbone module. For the reversed BEV representations, we encapsulate the real environment within a 100 meter radius around the vehicle to generate features with dimensions (height 𝐻 , width𝑊 , number of channels 𝐶) of (150, 150, 256). For the training of auto-encoder module, the AdamW optimizer is adopted with a weight decay of 0.01, the learning rate is initially set to 0.0002 and decayed with a cosine annealing. To train the calibration module, the same optimizer schedule as that of the auto-encoder module is employed, though with an adjustment in the learning rate specifically for the image backbone, which is lowered to 0.00002. It is noteworthy that the two modules are trained independently, as depicted in Figure 2(d) . The parameters of the calibration encoder module are inherited from the auto-encoder encoder, and they remain frozen during the overall calibration module training process.Table 1 presents the comparative experimental results on nuScenes dataset, including errors in translation vector t, rotation matrix R, intrinsic parameters I, focal length 𝑓 , and principal point 𝑐. The findings reveal that our model consistently exhibits outstanding prediction performance. As shown in Table 1 , when model is trained with total 4200 cameras data, our model achieves 0.0614 and 0.0188 error on rotation matrix and intrinsic parameters, respectively. Furthermore, as the calibration training data is reduced, it still achieves a translation vector error of 0.0102, demonstrating generalization capabilities of our model.As for the comparison results on the Waymo dataset in Table 2 , we attained errors of 0.0016, 1.276, and 1.2630 in translation vector, rotation matrix, and intrinsic parameters, respectively, serve as evidence validating the effectiveness of the model. Subsequently, further comparison is conducted regarding the errors in translation vectors along the XYZ axes. The performance of our model in predicting translation vectors is significantly improved, with errors reduced to 0.0031, 0.0026, and 0.0010, outperforming previous results of 0.0480, 0.0290, and 0.0410, respectively. Performance is further enhanced upon employing the original Waymo dataset, resulting in errors reduced to 0.001, 0.0009, and 0.0003.Table 3 and 4 present the results of our ablation results on the nuScenes dataset. The former is based on a 4200 cameras dataset, while the latter experimented solely with 300 cameras dataset. The "Reversed Pipe" is an abbreviation for our overall reversed BEV pipeline, indicating the direct training of the calibration module to achieve the prediction of calibration parameters. "AE-Finetune" and "AE-Frozen" signify the incorporation of auto-encoder pretraining, representing parameter fine-tuning and parameter freezing, respectively. Clearly, when utilizing the reversed BEV pipeline along with a frozen AE, we observe a notable overall performance improvement on both different dataset size, validating the effectiveness of our architectural design. Additionally, an increase in data volume contributes to more accurate predictive outcomes. Finally, particularly concerning intrinsic parameters, the performance of the frozen AE improved from 0.0207 to 0.0188 (Δ = 0.0019) in the case of the 4200 dataset. And in the scenario of only 300 cameras dataset, the performance improved significantly from 0.0455 to 0.0287 (Δ = 0.0168), providing further validation of the AE's performance on small calibration dataset. We additionally assesses the calibration parameters obtained by our CalibRBEV model in other object detection models, replacing the traditional calibration's intrinsic and extrinsic parameters [5, 61] with the parameters we predicted. We adopt BEVFormer-S [35] and PETRv2-vov [39] as our baseline methods. Specifically, the BEVFormer-S model directly utilizes weights trained on the original dataset, while PETRv2-vov is trained using our predicted calibrated intrinsic and extrinsic parameters. As shown in Table 5 , compared to traditional methods that are complex and time-consuming, utilizing the predicted calibration parameters for detection yields very close results, indicating that the model can be integrated into other BEV methods without performance degradation. Particularly, using only 300 cameras dataset, applying our model's prediction parameters to the object detection models, still yielding approximate results, confirming the efficacy of predicting calibration parameters with limited data.Moreover, we also assesses the detection performance results of the auto-encoder module within our CalibRBEV model in Table 6 . Obviously, the auto-encoder module demonstrates outstanding detection capabilities, suggesting that the generated BEV features encompass sufficient 3D object bounding box information and satisfy the requirements of the calibration module. 
discussion: 
 In this section, we discuss the limitations of our work and prospective directions for future research. We hope that our work to serve as a catalyst inspiring further future endeavors.(1) Firstly, in this paper, we extract multi-view features from multi-camera images and directly obtain 3D bounding boxes from LiDAR to achieve a reversed BEV pipeline for calibration parameters prediction. However, the information relationships between multiple sensors and fusion strategies merit further exploration to enhance the performance of parameter prediction.(2) Then, our current model still relies on predicting calibration parameters corresponding to single-frame data. However, in the context of autonomous driving scenarios, spatiotemporal information is accessible. Learning the correlation of the overall spatiotemporal structure can endow networks with structural knowledge derived from motion, thereby potentially enhancing the performance of calibration tasks.(3) Furthermore, the mapping relationship between the attention interaction mechanism of image features and BEV (Bird's Eye View) features, which is pre-determined using average parameters, may possess potential inaccuracies. Thus, there could be a necessity to devise a new attention interaction architecture in the future.(4) Finally, our model falls within the realm of learning-based calibration methods, substantially reducing traditional calibration costs. However, akin to other deep learning models, it necessitates pre-existing datasets for training to enable parameter prediction. Although this work introduces an auto-encoder module to alleviate the demand for extensive datasets, a potential future direction is to explore methods for calibrating parameter prediction using fewer training data.CONCLUSIONIn this paper, we introduce a novel multi-camera calibration method for calibrating intrinsic and extrinsic parameters, termed CalibR-BEV, which innovatively proposes the use of a reverse pipeline based on the BEV model. Our model mainly consists of two components: the calibration module and the auto-encoder module. The auto-encoder module fully explores latent environmental information to achieve the extraction of reversed BEV features, and the calibration module then predicts calibration parameters leveraging multi-view and reversed BEV features. Experimental results demonstrate superior performance compared to previous calibration methods. Furthermore, the model can be extended to scenarios with small data volumes, simplifying the requirements of extensive calibration datasets and thereby optimizing calibration efforts. 
: 
 such as hand-eye calibration is a time-consuming and laborious offboard procedure during mass production, let alone re-calibration after product delivery. For example, for the extrinsic parameter matrix, it usually requires to be executed within a specific calibration room. On the other hand, some recent deep learning-based calibration models achieve performance potential by directly regressing calibration parameter targets [31, 54] . However, there is a notable dearth of comprehensive exploration within the realm of multi-view cameras complex scenes in autonomous driving.Recent bird's-eye-view (BEV) models [34, 35] achieve notable progress in 3D perception using multi-camera imagery and calibration parameters within the context of AD scenarios. As shown in Fig. 1 (a), the encoder utilizes multi-view images and calibration parameters to produce BEV representations, which are subsequently fed into the decoder to generate predictive bounding boxes, enabling perception in the real world. On one hand, such a perceptual paradigm necessitates accurate calibration parameters to achieve precise detection, highlighting the crucial role of camera calibration in AD. On the other hand, the close relationship between BEV models and calibration parameters inevitably raises a question: can the outstanding BEV model concept be leveraged to accomplish the task of camera calibration in AD?Motivated by this observation, we rethink the camera calibration from a new perspective, i.e., a reversed BEV perception pipeline. In other words, reversely, our objective is centered on the optimization of calibration parameters utilizing bounding boxes data, rather than the prediction of bounding boxes, as shown in Fig. 1(b) .During model training, the multi-view images and bounding boxes can be readily acquired using alternate sensors from calibration datasets, such as cameras and high-precision LiDAR. Thus, the key challenges are (i) The training of the whole model still demands a substantial pre-collect calibration dataset, which goes against our initial intention of cost reduction. In the meanwhile, (ii) it is still unexplored in the community that how high-quality and comprehensive BEV features can be extracted from bounding boxes, which is a reversal of existing literatures [34, 35] .Regarding the above two issues, we employ an auto-encoder reconstruction strategy to pre-acquire the reversed BEV representations. Consequently, the representations derived from the encoding of 3D object bounding boxes should be able to capture comprehensive environmental information, facilitating accurate prediction of camera calibration parameters. Additionally, pre-training of the auto-encoder module for reconstruction enhances the robust generalization of the encoder, further diminish the requirement for extensive calibration datasets, as shown in Table 4 .To this end, this paper proposes the CalibRBEV model, which achieves automatic multi-camera calibration via reversed BEV representations for autonomous driving. Specifically, our model begins by reverse BEV perception pipeline and utilizes attention autoencoder modules to reconstruct bounding boxes for obtaining the reversed BEV representations. Subsequently, these features interact with multi-view image features for further refinement to predict the final calibration parameters. Experimental results on nuScenes and Waymo datasets validate the effectiveness of our method. Particularly, on nuScenes dataset, we achieve 0.0102, 0.1637, and 0.0287 errors on translation vector, rotation matrix, and intrinsic parameters, respectively.The main contributions of our work are summarized as:• This work introduces the concept of BEV perception models into calibration tasks, achieving the camera calibration in autonomous driving scenarios through the reversed BEV pipeline. • This paper proposes an auto-encoder-based strategy for reconstructing bounding box data to achieve high-quality reversed BEV feature extraction, thereby further enhancing the prediction of calibration parameters. • Experimental results on nuScenes and Waymo datasets validate the effectiveness of our model. The ablation results further substantiate the performance of module design and its efficacy across varying data size. 
 
 Appendix:  


Decision: The paper receives mixed ratings from three reviewers after the rebuttal. Two reviewers recommend borderline accept and one weak reject.  The two reviewers with positive ratings both like the idea and are convinced by the extensive experimental evaluation. The last reviewer feels that the setting used in the paper is questionable as it requires both bounding boxes and multi-view images as input for the camera calibration, however, this is indeed a novelty of the method itself, as acknowledged by the other two reviewers. AC also thinks that the proposed method is interesting and shows promising results, and thus recommends acceptance for this time.

Reviews: review: paper_summary: paper_summary: This submission has a very novel idea: predicting the camera intrinsic and extrinsic parameters from input RGB images and ground truth bounding boxes. The ground truth boxes are encoded by an auto-encoder into features, and cross-attended with input RGB image features and finally used to predict intrinsic and extrinsic camera parameters. Quantitative experiments are presented and nuScenes and Waymo, showing improved camera parameter prediction results compared with baselines. The authors also show good box auto-encoding results in table.6.
strength_weakness: strength_weakness: + The idea is definitely novel and I don't remember anyone doing this before: predicting camera intrinsic and extrinsic camera parameters from the cross-attened features between input RGB images and ground truth bounding boxes. 
+ Doing bounding box auto-encoding boils down to a generative model that captures the joint distribution of outdoor scene layout. While this methodology has been used for indoor scene layout synthesis for a long time, this is the first time I see this for autonomous driving. According to the results in Table.6, the results are quite strong so this module may be useful in related downstream tasks, not specifically this one.
limitations: limitations: - The biggest issue is the setting. While what is done here is novel, I cannot agree this is a calibration process. The authors exploit representations extracted from ground truth bounding boxes as input, but this is not doable actually in the real world. When we are trully faced with a calibration challenge, how could we get these bounding boxes? By contrast, I think a V2X setting would be more interesting. The authors could argue that the road side box detections are accurate thus can be used as input into the parameter prediction network.
- Scene layout autoencoding can be used as a generative prior for detection tasks, instead of this parameter prediction task. There are other interesting downstream task considering the calibration setting does not stand actually, IMHO.
paper_summary: paper_summary: This manuscript proposes CalibRBEV, a multi-camera calibration method designed for autonomous driving. The calibration module takes the multi-view image features and bounding boxes as input. The method utilizes an attention autoencoder module to reconstruct bounding boxes in order to extract the reversed BEV representations. Then, the obtained representations are combined with multi-view image features to facilitate the prediction of camera calibration parameters. According to the manuscript, the CalibRBEV method can achieve efficient and robust multi-camera calibration in autonomous driving scenarios, demonstrated in nuScenes and Waymo datasets.
strength_weakness: strength_weakness: 1.	By introducing the concept of BEV perception into the calibration task, the proposed CalibRBEV provides a new idea to solve the problem of multi-camera camera calibration in autonomous driving.

2.	This manuscript innovatively proposes a pre-training strategy based on attention autoencoder module. It aims to extract high-quality BEV representations by reconstructing 3D object bounding box, thereby enhancing the prediction of calibration parameters.

3.	During training pipeline, it has a two-stage strategy. The autoencoder module is optimized through reconstruction pre-training; then, the calibration module is trained while freezing the encoder module. This facilitates the model to learn feature representation and subsequently improve prediction of calibration parameters.

4.	The experimental design is adequate. It demonstrates that CalibRBEV achieves good performance compared to other methods on both the nuScenes and Waymo datasets. Ablation studies also provide evidence that the autoencoder pre-training contributes to the performance improvement. Furthermore, the authors apply the predicted calibration parameters to BEVFormer-S and PETRv2-vov, validating the effectiveness and compatibility of CalibRBEV with existing BEV perception frameworks. Overall, the experimental results show the effectiveness of the proposed CalibRBEV method in multi-camera calibration for autonomous driving scenarios.
limitations: limitations: This manuscript can be further enhanced by carefully considering the following comments:
1.	There are duplicate and redundant contents in the article, making it appear overly lengthy. For instance, the first sentence of the Abstract:" Camera calibration consists of determining the intrinsic and extrinsic parameters of an imaging system, which forms the fundamental basis for various computer vision tasks and applications." And the first sentence of the Introduction:" Camera calibration is a fundamental procedure in computer vision tasks [43, 52], mainly involving the determination of intrinsic and extrinsic parameters of an imaging system.". I think the manuscript needs to be revised, especially Introduction section.

2.	There are some grammatical mistakes in the manuscript. To enhance the readability and professionalism of the paper, the authors should conduct a further examination. Here are the grammar problems I noticed:
	In the Abstract, 'we proposes' should be corrected to 'we propose'.
	At line 393-395, “…model is shown in Fig. 2, comprises three main components:” contains a grammatical error.
	At line 827 and line 843, “We additionally assesses the calibration…” and “…we also assesses the detection…” both contain grammatical errors.

3.	At line 131 of the manuscript, the "auto-encoder reconstruction strategy" is mentioned as being able to capture environmental information and contribute to accurate prediction of camera calibration parameters. However, there is no specific explanation provided on why this strategy are adopted. The authors need to supplement relevant information.

4.	In the Abstract and Introduction, authors point out the time-consuming problem of traditional/prevailing camera calibration methods. However, experimental results do not contain the runtime performance and cost of the proposed CalibRBEV. It would be better if the authors could supplement this information.
paper_summary: paper_summary: This paper reverses the BEV 3D object detection process to predict the calibration parameters with multi-camera images and Lidar 3D object bounding boxes as input. The proposed CalibRBEV model comprises two key modules: the calibration module and the auto-encoder module. The auto-encoder module explores latent environmental information to achieve the extraction of reversed BEV features, and the calibration module predicts calibration parameters by leveraging multi-view and reversed BEV features.
strength_weakness: strength_weakness: The idea and explored task in this paper is quite new.
The paper is well-written and easy to follow.
limitations: limitations: Although the task is quite new, the approach of using both multi-camera images and LiDAR 3D object bounding boxes as input makes it less practical. The effort required to obtain 3D object bounding boxes is significantly higher than that needed for camera calibration, making the task setting questionable.

Has the author compared the time and computational costs with traditional calibration methods? Efficiency and cost are crucial factors in calibration.

Has the author considered using only image data as input? Achieving calibration with image-only data would be a more reasonable and practical approach.