Learning Markerless Robot-Depth Camera Calibration and End-Effector Pose Estimation
Authors: Bugra Can Sefercik
Baris Akgun
Venue: CoRL 2022 Poster (2022)
Keywords: Camera Calibration
Pose Estimation
Perception

Abstract: Traditional approaches to extrinsic calibration use fiducial markers and learning-based approaches rely heavily on simulation data. In this work, we present a learning-based markerless extrinsic calibration system that uses a depth camera and does not rely on simulation data. We learn models for end-effector (EE) segmentation, single-frame rotation prediction and keypoint detection, from automatically generated real-world data. We use a transformation trick to get EE pose estimates from rotation predictions and a matching algorithm to get EE pose estimates from keypoint predictions. We further utilize the iterative closest point algorithm, multiple-frames, filtering and outlier detection to increase calibration robustness. Our evaluations with training data from multiple camera poses and test data from previously unseen poses give sub-centimeter and sub-deciradian average calibration and pose estimation errors. We also show that a carefully selected single training pose gives comparable results.

Learning Markerless Robot-Depth Camera Calibration and End-Effector Pose Estimation
Bugra C. Sefercik
Baris Akgun CE Department KUIS Center Koc University, Turkiye baakgun@ku, edu.tr tr
CE Department KUIS AI Center
Koc University, Turkiye bsefercik0ku, edu.
Abstract: Traditional approaches extrinsic calibration use fiducial markers and learning-baas -baned approaches rely heavily on simulation data. In this work, we present a learning-basee markerless extrinsic calibration system that uses a depth camera and does not rely on simulation data. We learn models for end-effector (EE) segmentation, single-frame rotation prediction and keypoint detection, from automatically generated real-world data. We use transformation trick to get EE pose estimates from rotation predictions and matching algorithm to get EE pose estimates from keypoint predictions. We further utilize the iterative closest point algorithm, multiple-frames filtering and outlier detection to increase calibration robustness. Our evaluations with training data from multiple camera poses and test data from previously unseen poses give sub-centimeter and sub-deciradian aver- age calibration and pose estimation errors. We also show that carefully selected single training pose gives comparable results.
Keywords: Camera Calibration, Pose Estimation, Perception
Introduction
Camera to robot calibration is an important step in many robotic applications. Keeping the correct cal- ibration is a challenge for robots in dynamic and uncontrolled environments, even if the robot and the camera meant to be static. Multiple fac- tors alter this calibration such as re-positioning the robot/camera for better workspace coverage for different applications (e.g. research projects), inad- vertently moving them for cleaning, people bumping into them, wear-and fixtures, etc. As such, calibration is needed but time-consuming process. At the very least, calibra- tion needs be checked before application. This an all-too-real issue for robotics researchers and getting more widespread as cage-free robot arms and mobile manipulators become more common.
and backlash on low-cost
Robot
Buse
Traditional extrinsic calibration approaches rely on fiducial markers. Checkerboard patterns and aug- mented reality (AR) tags are commonly used in search settings. Adding markers is eiror- prone due intrinsic calibration errors, sensor noise, robot-to- marker fixture quality (which introduces transforma- tion errors), High precision systems employ precision machined fixtures and active markers (e.g. LEDs, reflectors+light-sourcee
Figure 1: Top-left: experimental setup, top- right: sample frame from Kinect VI, bottom: segmentation, keypoints and pose prediction
which are costly. Thus, using markers is either noisy or expensive.
In this work, we develop system that can handle the extrinsic calibration between depth camera and robot arm without additional hardware, markers or simulation. We utilize recent developments
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

in deep learning models for point clouds in combination with the ICP algorithm. Our system collects its own data, negating the need for manual labeling, only requiring an initial calibration effort. Two side benefits of our system are that it provides high-quality semantic segmentation information about the background, end-effector (EE) and the rest of the robot, and estimates the EE pose with low error.
There are other learning-basee based approaches for markerless calibration for both eye-to-hand calibra- tion [1, 2, 3] and eye-in-hand calibration, [4] that utilize colored images and synthetic/simulation data. Our work differs by exclusively using real-world data, negating the need for simulation, and depth data which readily enables the usage of the ICP algorithm for high performance.
The main contributions of our system are; (1) high performance (average test errors of 0.74cm and 1.699) markerless extrinsic calibration, (2) high performance (average test errors of 1.0cm and 2.749) EE pose estimation wrt. camera, and (3) automatic real-life data collection methodology.
2 Related Work
Calculating the transformation between a robot's base and camera, usually called extrinsic cali- bration, is decades old challenge [5]. Conventional techniques use fiducial markers to estimate camera to robot transformations. Most popular approaches stick a marker on the robot's EE [6. 7, 8] or other relevant parts [9, 10, 11] and collect data from multiple robot joint angle configurations to be used in offline calibration calculation. Major disadvantages of using such markers include wear- and-tear, distance dependent camera noise [12]. fixture backlash and build quality, etc. There are also single frame methods that use fiducial markers to estimate the EE pose [13, 14] in real-time. These methods start to fail when the reference objects are not as visible enough as anticipated. In this work, we aim develop markerless approach to handle extrinsic calibration.
The advent of deep learning brought developments in areas that are closely related to EE pose es- timation such 6-DoF object pose estimation [15, 16] and keypoint detection [17, 18). Several methods have produced successful results on robot pose estimation from a single frame. Lambrecht [1] and Lee et al. [2] learn to estimate keypoints of the entire robot arm from RGB image trained with newly created real world data and synthetic data, respectively, then to predict the robot pose combining keypoint data with forward kinematics information. Similarly. Labbé et al. [3] lears to predict robot pose and joint angles from RGB images via iterative CAD to image matching us- ing synthetic dataset [2]. Our system diverges from these methods in the following areas: (1) they require majority of the robot arm to be visible while our only requirement is to see the EE in the frame (2) our method uses depth data whereas others rely on Perspective-n-Poinn algorithm to compute affine transformations since they utilize 2D images, (3) we can readily apply the ICP algorithm to refine our pose estimates due to the depth data usage, and (4) do not require any synthetic/simulation data and only use real-world data. It can be argued that we are solving simpler problem (only EE and depth data) in exchange for higher calibration quality.
Hand-in-eye calibration, where the camera is mounted to the EE, is another example of camera to robot calibration problem to which conventional solutions that explained earlier are also appli- cable. Partially similar to our system, a recent method works on hand-in-eye calibration via deep learning [4]. This method uses synthetic data and 2D images, and different robot-camert setup.
3 Method
Our system consists of two main stages; (1) single frame EE 6-DoF pose estimation and (2) multi- frame extrinsic calibration. The output the the former along with forward kinematics is enough to get a calibration estimate but multiple frames increases robustness. Fig. 2 shows our single frame EE pose estimation and Fig. shows our calibration workflows.
For EE pose estimation, we first segment the EE from a point cloud using semantic segmentation approach. Then we utilize two approaches to estimate the EE pose. The first one does EE rotation prediction followed by point cloud transformation step to get the EE translation. The second one extracts keypoints from the EE and matches these with reference points to calculate the EE pose. Both approaches are followed by an ICP step initialized with the EE pose estimates to match the EE points and the EE CAD model. These steps are summarized in Fig. 2.

---

Rotation
Prediction
Eneoder
Rotation
Pase wrt. Camera
EE Point Clond
Initial EK Pose
MinkUnet
Clustering
Translation Calculation
Raw Point Cloud
Semantlic Segmentttion
3D Rigid Transform between
PointNet+ Enceder
ICP
EE 6D Pase wet. Camera
KE Pose
Key Points Prediction
Model Point
Matching
Figure 2: Single frame prediction architecture.
Sunity Chock
Intra-coniiguration Outlier Rejection Averaging
Inter-conßiguration Outlier Rejection Averaging
Rohot Base's 6D Pose Camera
Configuration
Averuged Group
Group
Predictions
Poses
Figure 3: Calibration architecture.
For calibration, the system collects multiple frames and the corresponding EE pose estimates described above. To increase robustness, sanity check and outlier detection steps are employed. The remaining poses then averaged in the rigid-body transformation space (SO(3) for rotation and R for position), to get the final extrinsic calibration result. These steps are summarized in Fig. 3.
3.1 Data Collection
Our EE segmentation and pose estimation rely learning and require robot specific data. We collect ground truth semantic segmentation labels, EE poses wrt. the depth camera and keypoint information. We collect this data from real world setup eliminate the need for simulation.
The first step to perform initial calibration. This can be done by existing approaches. We use a combination of markers, manual tuning on RViz [19] and ICP [20] towards this end. This is the only manual step that is needed and only required for data collection. This is only done once per camera pose want to collect data for. Once system is up and running, no manual tuning is needed. This calibration information gives us the robot poses with respect to the camera frame.
The second step is to collect semantic segmentation data. We label points background, EE and the of the robot automatically. Background points obtained by utilizing frames without the robot. Then, the robot is moved to multiple visible positions to collect more frames. Semantic segmentation data for the robot arm generated by subtracting the background points from every frame [21]. The EE points are further obtained by transforming the EE bounding box using the calibration information and taking the points inside this box!
The third step is to generate keypoint information. We extract a total of six keypoints; four are located at each corner of the EE and two are located the tip of the gripper fingers as shown in Fig. 4. During training, we have access to the calibration. As such, we know the pose of the EE in the camera frame. We also know the reference keypoint locations with respect to the EE as they are
When subtracting the background points is not possible, the CAD model of the robot, joint positions and the calibration information be used get the arm and EE points, and consequently background points.

---

fixed and use this information to get reference points in the camera frame. The EE points closest to these reference points are selected keypoints, as long as their distances are below threshold.
3.2 End-Effector Segmentation
The EE pose estimation starts by extracting the semantic segmentation information from an input point cloud. The outputs of this step are the EE, rest of the arm and the background labels for the points as shown in Fig. 1. For this step, we utilize a sparse version of the Unet architecture named the Minkowski Unet network [22, 23, 24], specifically the MinkUNet1 [22] architecture trained with the raw point clouds as the input and the collected semantic segmentation data as the target. We then apply linkage clustering [25] to points classified as EE order to reject superfuoos predictions.
3.3 Pose Estimation with Rotation Prediction and Transform Calculation: RPT
We first predict the EE rotation from EE points by utiliz- ing the encoder part of another MinkUNet18D backbone with the same configuration. We train this with the seg- mented EE points as input and the EE rotation wrt. cam- target, using the loss of the PoseCNN model [15].
To calculate the EE translation, we use the output of the learned rotation prediction model and the geometric rela- tionship between the EE frame and certain EE points on the bounding box, assuming that the EE is not occluded. We use the predicted rotation to rotate the EE points back to their non-rotated pose wrt. the camera. This is done by applying the Eq. 1 on each EE point, where REE is the rotation prediction of the model, PεE is the translation of an EE point and the PEE is the resulting rotated position. Both PEE and PEE are in the camera frame.
X
Z
Figure 4: End-effector segmentation (yellow), end-effector frame, and key- points (colored hexagons).
FEF (REB) x P
(1)
The EE points are along the surface of the EE. If we assume that the EE is fully visible, we can use the geometric relationship between the EE surface and the EE frame get the EE translation. In the non-rotated pose wrt. the camera frame, the EE frame is 0.015m inside the surface along the r-axis, centered along the y-axis and elosest the camera along the z-axis (see Fig, 4). We the
following equations to capture this relationship where [EE and P&E denote the axes positions of the EE frame and EE points in the rotated frame, all in the camera frame.
EE max (PEB)= - 0.015
(2)
t max (PEE) min (PFE)
(3)
2
F. min (PEE)
(4)
We finally rotate fEE back to get the EE translation, FFF wrt the camera using Eq.
E = REE x EE
(5)
Calculating the EE translation with such an approach is efficient and accurate given that the rotation predictions are accurate and that the EE is visible. However, full EE visibility is not a given for all the frames and as such, this method cannot be trusted for single frame pose estimation by itself.
3.4 End-Effector Pose Estimation via Keypoint Matching: KPM
In this approach, we first predict keypoints among the EE points. We use a dense point cloud encoder, PointNet++ [26] trained with segmented EE points as input and the collected ground truth keypoints as target. We then use version of the least-squares fitting [27] algorithm to find the rigid transformation between predicted keypoints and the reference points, which gives us the EE pose wrt. the camera. This is only possible when there at least four high quality keypoint predictions.

---

3.5 Pose Estimation Refinement via the Iterative Closest Point Algorithm
We use the ICP algorithm [20, 28] to improve the pose estimation performance. This algorithm requires three inputs; the initial pose estimate (needs to be fairly accurate), a source point cloud, and a target point cloud. The initial pose estimate comes from either the RPT or the KPM methods, and the target point cloud comes from the EE segmentation. To generate the source points, we convert the CAD model of the robot's s HE, provided by the manufacturer [29]. to a point cloud, and transform the points based on the current EE configuration. The ICP algorithm outputs the transformation needed match the source and target points which we use to refine the pose estimates.
3.6 Camera to Robot Base Calibration
A single EE pose with respect to the camera frame can be used to calculate the transformation between the camera and the robot using Eq. 6, where TX represents the homogeneous transformation between frames X and Y. and the letters B, C, and EE correspond to the robot base, camera and the EE respectively. The EE in the base frame, T obtained by forward kinematics and the EE in the camera frame, TEE is estimated by either the RPT or the KPM methods.
TO TE (TE
However, a single frame is not robust enough to get a good calibration. We move the robot around accurate estimates. We first run the
and capture multiple frames at each robot pose get semantic segmentation model and perform a sanity check on the output. If the number of EE points or the size of the EE bounding box below their respective thresholds, we remove the frame. This is done to eliminate frames where the EE is not visible enough for accurate pose estimation.
After the sanity check. group the predictions based the robot configuration (we capture mul- tiple frames per robot pose). We use a Z-score outlier detection algorithm using the absolute devi- ations about the median [30] for each translation axis of the camera to robot base predictions, per group. For rotation prediction outlier detection, we apply outlier detection on the rotational distances of every prediction
reference unit quaternion. We remove predictions that are marked out- liers. Then, the remaining camera to base transformations are averaged by computing the arithmetic average for translation predictions and utilizing quaternion averaging [31] for rotation predictions. This gives us individual calibration estimations for each robot configuration.
Lastly, employ the same outlier detection and averaging steps the calibration estimates of each robot configuration to get the final estimate.
Evaluation
We evaluate the semantic segmentation. single pose estimation and camera calibration performances of our system when trained with three camera poses. We test our approach with ground truth seg- mentation labels well to gauge the effects of the accuracy of the segmentation model the pose and calibration estimation. We also provide results without the ICP step to highlight its benefits and compare to AR-tag based classical baseline. Lastly, we perform evaluations with models trained on data from individual camera poses.
We use the translation error (Ez), rotation error (ER) and the average distance (ADD) metrics to measure the pose estimation and calibration performances. The e, is the Euclidean distance between the ground truth translation and the predicted translation. The e is the minimum rotation between the ground truth rotation and the predicted rotation. ADD is the average of point-to-poin Euclidean distances between the EE points transformed with the ground truth pose and the predicted pose [32]. To measure the accuracy of our semantic segmentation model, we use accuracy, precision and recall.
4.1 Collected Data
Our hardware setup only includes Kinect V1 and a Franka Emika Panda robot arm as shown in Fig. 1. We collect around 7000 frames from different camera poses. The cameras are placed on three sides of the robot setup, as shown in Fig. 5. We reserve 1000 frames for validation to control overfitting. To increase generalization, we apply 3D data augmentation methods such as clastic distortion, noise injection and point dropouts during training. We also collect test data from
5

---

P1
Training
Figure Visualization of Kinect camera positions during training and test data collection.
P1 Test
P2 Test
P3 Test
Ac [cm]
Ar
Ar [cm]
An
A. [cm]
An
P1 Training
5.15
14
95.12
71.33
173.49
119.07
P2 Training
102.7
96.32
5.88
18.31
88.12
95.37
P3 Training
175.35
170.16
86.44
112,66
6.2
42.5
Table I: Relative translation and rotation differences between camera pose pairs.
additional 3 cameras poses (see Fig. 5), 6 locations per pose and 10 frames per location for total of 180 frames. The translation and rotation differences between training and test camera poses are given in Tab. 1. We additionally collect test data using ArUco tag attached to the EE in the same test poses. We chose relatively large marker to be robust to camera distance as shown Fig.
The training data is collected with an initial calibration. As we do not have access to high-qualin markers (e.g. a motion capture system), this may suffer from all the issues laid out in this paper. The ADD of the training data calculated as 0.75cm, which implies a relatively high quality data set.
Semantic segmentation performance is important it is the first step and its EE segmentation output is used as in- put to the following steps. We use precision, recall and ac- curacy to assess the performance of segmentation. Tab. 2 presents these results. Overall, the segmentation model performs well. The recall ratio for EE segmentation is 100%, i.e., the model is able to catch all the EE points.
Class
Precision
Recall
Accuracy
EE
0.96
0.99
1.00
Arm
0.87
0.99
0.99
BG
1.00
0.99
0.99
All
0.94
0.99
0.99
Table 2: Semantic segmentation res.
The pose estimation and extrinsic calibration performances are calculated with both the model and ground truth segmentation labels (Table 3). Tab. shows that there is around 0.05cm and 0.5* differences between the average errors. These are negligible when the camera noise is taken into ac- count [12]. Combining this with the segmentation performance, we conclude that the segmentation model performs well enough to generate reliable inputs for the subsequent steps.
4.2 Single Frame Pose Prediction and Multi-Frame Calibration
Tab. shows the pose estimation performances of the RPT and KPM approaches with and without ICP post-processing step and with ground truth and predicted semantic segmentation labels.
ICP step provides considerable improvement for pose estimation: The addition of the ICP step improves both algorithms in all metries in both segmentation input cases. In addition, the absolute performance is very impressive with about 1cm translation and less than 3" rotation errors.
[em]
ER
ADD [em]
Method
GT
Model
GT
Model
GT
Model
RPT
2.36=10.36
2,491:0.7
7.75+55005
7.18:4.62
2.47+0.52
2.58+10.90
RPT+ICP
0.87+0.20
1.014:032
3.20+:2.42
3.47+22.7
0.97+0.30
1.074:0.35
KPM
1.35
1.43:10.48
7.46111.54
7.4411.44
1.53+1.33
1.60:10.47
KPM+ICP
0.96+0.27
1.00:10.27
2.42+:0.98
2.74+1.59
0.99+0.26
1.04++0.27
Table 3: EE pose estimation results with ground truth (GT) and predicted (Model) semantic labels.

---

High Performance Multi-Frame Calibration: The calibration results are given in Tab. The average translation and rotation er- ror between the predicted calibration and the ground truth calibra- tion are et 0.74cm and 6 = 1.69", respectively. As expected, the ICP step significantly improves performance, and semantic seg- mentation and ground truth inputs perform similarly. This absolute performance would allow for most manipulation applications out- side of assembly or high precision tasks. These results are affected by the sensor noise, and poten- tially the intrinsic calibration and the initial extrinsic calibration errors. We argue that a less noisy camera and/or an active marker system for initial calibration is needed to go beyond these values.
<: [em]
en [
AR-Tag
1.83
3.26
Ours
2.35
3.37
Ours ICP
0.74
1.69
GT ICP
0.77
1.19
Table 4: Calibration results.
We also compare our system with an AR- based baseline. The AR-tag results in Tab. are with out- lier detection but without the ICP step. These results show that the AR-tag baseline holds 0.5cm advantage in translation and is on par in rotation compared to our base method. We chose a relatively large marker to be robust to distance and large orientation changes. This alters the EE shape considerably (see Fig. 6), and makes ICP infeasible. A smaller tag degrades the performance considerably with distance to camera. If the ICP algorithm was feasible with the large marker, the final calibration results would be on-par. Thus, method can achieve similar results to classical marker based method while being easier to use and re- quiring much less manual effort down the line (a large marker needs to be removed for work and re-attached for calibration). Our system would have better performance than an AR-tag based approach with a smaller marker that is more practical to use.
Figure EE with the AR-tag attached
Discussion Pertaining to Existing Work: is difficult to conduct an apples-to-apples comparison between work and the existing learning- based methods since the setups significantly differ- ent. This is due to simulation requirements, lack of point cloud data in existing datasets and the difference in predicted outputs (e.g. full arm EE). However, qualitative discussion be made over the common ADD object pose estimation metric.
RPT and KPM approaches outperform state-of-the-art object pose detection methods [15, 16]. The performance gets much better when the ICP step is included. We also outperform a similar single get 100% pose estimation accuracy whereas they get less than 40% for Kinect V1. However, the generic object pose detection methods deal with multiple objects [33] and Lee et al. [2]'s result is for the entire robot arm, whereas we are only looking at the EE.
frame pose estimation method [2]. With ADD threshold of 2.0cm,
4.3 Training with Data from Individual Camera Poses
The results we presented so far were obtained from data collected from each training camera location (see Fig. 5). These locations cover either side and the front of the robot. They also cover range of distances to robot base where P1 is closest and P3 is the farthest. However, this raises the question of calibration performance without such coverage.
To answer this, we train different models using data from each training camera location and test it on all the test data (e.g. train with P2-training, test on P1-test, P2-test and P3-test). As given in Table. there are significant translation and rotation differences between the poses with the only similarity being the pitch axis. Tab. 5 shows the calibration results with the full system.
The results show that, the closer the train-test locations, the better the performance with exception for Pl-train and P1-test. The reason is that the P1-train pose is very close to the setup, and as such, its training set lacks data from farther EE poses. System trained with P2 data achieves the best overall results, as expected, since it is in the middle both translation- and rotation -ise.. Furthermore, its performance is too far from the system trained with data from three camera poses.
The results imply that, a careful selection of the training camera pose important but single camera is enough with a slight sacrifice in performance while reducing the initial calibration effort.

---

Pl Test
P2 Test
P3 Test
Average
< [em]
eR []]
< [em]
eR [*]
<, [cm]
<R []]
< [cm]
eR [*]
Training
1.32
0.64
1.64
2.81
4.21
5.63
2.39
3.03
P2 Training
0.84
0.60
0.70
1.82
1.61
2.32
1.05
1.58
P3 Training
1.82
1.20
2.28
2.12
0.93
1.89
1.68
1.74
Table 5: Calibration prediction results for respective training and test sets.
5
Limitations and Future Work
The quality of the training data depends on the initial calibration, RGBD sensor noise and forward kinematics error (e.g. due to gear backlash, joint sensor calibration, etc.). A careful calibration is needed for any robotics task so there no extra manual effort use our system. However, this should be performed carefully. We used relatively noisy RGBD camera [12]. We expect better camera to yield better results. The user should be aware of the forward kinematics errors and, if they are severe, should take this into consideration. In such cases, training data should be processed with ICP and only single EE frame pose estimations should be used. This affects any type of calibration.
Our system requires that the EE is not occluded during calibration. The RPT method would not be able calculate accurate translations otherwise. The KPM method more robust against occlusions as it can work with four keypoints. If occlusions are expected, more keypoints should be selected and RPT should be disabled.
We collect data with the robot arm in different joint configurations for the calibration operation. These joint configurations should be diverse for better calibration estimate. The test set in this work is collected from manually selected joint configurations. A fully automated system should choose these by itself while keeping the EE in the camera frame.
Our system assumes that there are no objects in the camera view during calibration since we do not collect object data. We use relatively high-capacity model, Mink 1ne [22], for the semantic segmentation task involving only three classes. As such, it is prone to overfitting. An out of sample input (i.e. point cloud with objects) would be difficult to handle. Using lower capacity models such as MinkUNet101 and MinkUNet14A deteriorated our semantic segmentation performance. Superimposing object point clouds as data augmentation approach is possible in to circumvent this limitation. However, this is mild assumption to handle extrinsic calibration prior to any application.
Currently, we provide the desired keypoints by hand. Even though it is enough to do this once for each EE, is not the most ideal approach since the user may not be familiar enough with robotics to do so the selected keypoints may not be easily distinguishable. An automated keypoint selection algorithm can be added in the future to remove this burden from the user.
We are not combining the outputs of the RPT and KPM methods to get single frame EE pose estimation and only average them during calibration. A future work is to investigate the potential of using them together, for example to have them share a backbone, feed the output of the rotation network into the KP network, etc.
Conclusion
We presented a learning- extrinsic calibration system that does not require fiducial markers, additional hardware or simulation. Our system collects and labels its own data. The only manual steps are initial calibration for data collection and keypoint selection. Our system performs extrinsic calibration from multiple frames and employs steps to increase its robustness. In addition to extrinsic calibration, system outputs high quality arm and EE segmentation information and camera-to- EE pose estimation. The extrinsic calibration challenge is never-ending, especially for low cost or multi-user setups and with system, the user needs to manually calibrate only once.
We tested our approach in different camera configurations then the ones used for training. Our re- sults showed that the ICP algorithm significantly improves estimation performance and the absolute calibration results are close to the limit of what possible with the used depth camera. We also showed that carefully selected single camera training pose is enough. The main limitations of our approach are the lack of object consideration, which is mild if the only aim is extrinsic calibration, and manual keypoint selection.
8

---

Acknowledgments
This work was supported by KUIS AI Center computational resources. The authors would also like thank Onur Berk Töre and Farzin Negahbani for their infrastructure support and work on an earlier version of the system.
References
[1] J. Lambrecht. Robust few-shot pose estimation of articulated robots using monocular cameras and deep-learning based keypoint detection. In 2019 7th International Conference on Robot Intelligence Technology and Applications (RiTA), pages 136-141. IEEE, 2019.
[2] T. E. Lee, J. Tremblay, T. To, J. Cheng. T. Mosier, O. Kroemer, D. Fox, and S. Birchfield. comera-to-robo pose estimation from a single image. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9426-9432. IEEE, 2020.
[3] Y. Labbé, Carpentier, M. Aubry, and Sivic. Single-view robot pose and joint angle estima- tion via render & compare. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1654 1663, 2021.
[4] E. Valassakis, K. Dreczkowski, and E. Johns. Learning eye-in-hand camera calibration from a single image. In Conference Robot Learning, pages 1336-1344 1346. PMLR, 2022.
[5] R. Y. Tsai and R. K. Lenz. A new technique for fully autonomous and efficient 3d robotics hand-eye calibration. In Proceedings of the 4th international symposium on Robotics Research, pages 287-297, 1988.
[6] S. Garrido-Jurado, R. Muñoz-Salinas, F. J. Madrid- Cuevas, and M. J. Marín-Jiménez. Auto- matic generation and detection of highly reliable fiducial markers under occlusion. Pattern Recognition, 47(6))2280-22 2292, 2014.
[7] M. Fiala. Artag, a fiducial marker system using digital techniques. In 2005 IEEE Computer Society Conference Computer Vision and Pattern Recognition (CVPR'05) volume 2, pages 590-596. IEEE, 2005.
[8] E. Olson. Apriltag: A robust and flexible visual fiducial system. In 1 IEEE international conference on robotics and automation, pages 3400-3407. IEEE, 2011.
[9] Q. Zhan and X. Wang. Hand-ey eye calibration and positioning for a robot drilling system. The International Journal of Advanced Manufacturing Technology, 615::9177 2012.
[10] X. Liu, Madhusudanan, W. Chen, Li, J. Ge, C. Ru, and Sun. Fast eye-in-hann in-hand 3-d scanner-robot calibration for low stitching errors. JEEE Transactions on Industrial Electronics, 68(9):8422-844) 8432, 2020.
[11] O. Kroeger, J. Huegle, and C. A. Niebuhr. An automatic calibration approach for a multi- camera-robot system. In 2019 24th IEEE International Conference Emerging Technologies and Factory Automation (ETPA), pages 1515-1518. IEEE, 2019.
[12] O. Wasenmüller and D. Stricker. Comparison of kinect v1 and v2 depth images in terms of accuracy and precision. In Asian Conference on Computer Vision, pages 34-45. Springer, 2016.
[13] H. Peng, X. Yang, Y.-.. Su, and B. Hannaford. Real-time data driven precision estimator for raven-ii surgical robot end effector position. 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 350-356. IEEE, 2020.
[14] K. Pauwels and D. Kragic. Integrated on-line robot-camera calibration and object pose esti- mation. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 2332-2339. IEEE, 2016.
[15] Y. Xiang, Schmidt, V. Narayanan, and D. Fox. Posecnn: convolutional neural network for 6d object pose estimation in cluttered scenes. 06 2018, 15500/5. 0055567//............. 2018.XIV.019.

---

[16] Y. Li, G. Wang, X. Ji. Y. Xiang, and D. Fox. Deepim: Deep iterative matching for 6d pose estimation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 683-698, 2018.
[17] Y. You, Y. Lou, Ci, Z. Cheng, L. Li, L. Ma, C. Lu, and W. Wang. Keypointnet: A large- scale 3d keypoint dataset aggregated from numerous human annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13647-13656, 2020.
[18] T. Jakab, R. Tucker, A. Makadia, J. Wu, N. Snavely, and A. Kanazawa. Keypointdeformer: Unsupervised 3d keypoint discovery for shape control. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 27783-12799 2021.
[19] H. R. Kam, S.-H. Lee, Park, and C.-H. Kim. Rviz: a toolkit for real domain data visualiza- tion. Telecommunication Systems, 60(2):337-345, 2015.
[20] Z. Zhang. Iterative point matching for registration of free-form curves and surfaces. Interna- lional journal of computer vision, 13(2):119-152, 1994.
[21] H. Sankoh, A. Ishikawa, S. Naito, and S. Sakazawa. Robust background subtraction method based on 3d model projections with likelihood. In 2010 IEEE International Workshop on Multimedia Signal Processing, pages 171-176. IEEE, 2010.
[22] C. Choy, J. Gwak, and S. Savarese. 4d spatio-temppral convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3075-3084, 2019.
[23] B. Liu, M. Wang, H. Foroosh, M. Tappen, and M. Pensky. Sparse convolutional neural net- works. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 806-814, 2015.
[24] O. Ronneberger, age segmentation. In International Conference on Medical image computing and computer- assisted intervention. pages 234-241. Springer, 2015.
P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical im-
[25] J. H. Ward Jr. Hierarchical grouping to optimize an objective function. Journal of the American statistical association, 58(301):236-244 1963.
[26] C. R. Qi, L. Yi, H. Su, and J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. Advances in neural information processing systems, 30, 2017.
[27] K. S. Arun, T. S. Huang, and S. D. Blostein. Least-squares fitting of two 3-d point sets. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMMI-95):698-7000 1987. doi: 1987.
[28] Q--Y. Zhou, J. Park, and V. Koltun. Open3d: A modern library for 3d data processing. arXiv preprint arXiv: 1801. 09847, 2018
[29] Franka Emika. Franka Emika ROS. http:://ggtaabccc////raaaaeallla 2022. [Online; accessed 9-June-2022].
[30] B. Iglewicz and D. Hoaglin. Volume 16: how to detect and handle outliers, The ASQC basic references in quality control: statistical techniques, Edward F. Mylyyka. PhD thesis. Ph. D., Editor, 1993.
[31] L. Markley, Y. Cheng, L. Crassidis, and Y. Oshman. Averaging quaternions. Journal of Guidance, Control, and Dynamics, 30(4):1193-1197, 2007.
[32] S. Hinterstoisser, V. Lepetit, S. Ilie, S. Holzer, G. Bradski, K. Konolige, and N. Navab. Model based training. detection and pose estimation of texture-less 3d objects in heavily cluttered scenes. In Asian conference computer vision, pages 548-562. Springer, 2012.
[33] B. Calli, A. Singh, A. Walsman, S. Srinivasa, P. Abbeel, and A. M. Dollar. The ycb object and model set: Towards common benchmarks for manipulation research. 2015 international conference on advanced robotics (ICAR), pages 510-517. IEFF, 2015.
10


Reviews:
[{"review_id": "e8P2xbpVfB", "reviewer": "Reviewer_HmKT", "summary": "The authors are tackling the problem of calibrating an external camera looking at a robot arm. They use learning based methods along with geometric considerations to perform the calibration.  Their method allows calibration to be performed quickly, without the need of specific calibration objects such as calibration tags.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Summary of Strengths**\n- The paper tackles an interesting problem that can have a large impact in robotics.\n- An interesting and informative method is presented, combining learning components and geometric considerations.\n\n**Summary of Weaknesses**\n- It would have been better to do an actual comparison/benchmarking against papers performing the same task, namely Labbe et al, and Lee et al.\n- The evaluation results seem to be dependent on the quality of the “ground truth” predictions, which can also be prone to errors, and there is no provided estimate of how large these errors may be.\n- Although one can follow what is being done, some points on the presentation can be improved.\n\n\n**Comments**\n\n- The extrinsic calibration and EE pose estimation are very closely related (by equation 6) , yet they are contrasted in the introduction without further explanation: This is confusing when reading it the first time and can be made clearer. Similarly, having the pose estimation and the extrinsic calibration in the same table later on compounds on the confusion. What is especially unclear is which rows of the table are used to perform the calibration. From  my understanding, multiple estimates of the ICP variants of both pose estimation heads are used independently to obtain the calibration through averaging and outlier rejection, but this is not very clear overall.  \n\n- Both the method and evaluation seem to be dependent on a “ground truth labelling” but we have no information on the accuracy of those ground truth values. In the experiment section in lines 214-216 it is unclear what the “ground truths” are: Is it the EE pose obtained through the robot kinematics? Is the calibration label obtained as described in line 101? This would need to be made clearer to better understand the results. The authors recognise this issue in lines 266- 269, but I believe this point needs to be expanded on and talked about earlier on, and it should also be added in the limitations of the paper.\n\n- Lines 247 - 249 : It would be good to provide some insight as to why this may be the case, either experimentally, or by conjecture.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The method presented by the paper is interesting and informative, and the problem tackled is important. However, there are some important points that are currently unclear in the paper, such as the possible reliance of the presented results to labelling that may also be error prone, without an estimate of what that error may be. \n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "It would be great if the authors could address the following:\n\n- Line 112 : is that alternative used? Or is simply a proposed alternative in case the previous steps cannot be performed. This could be made clearer.\n-It seems only point clouds are used throughout the paper as input modalities, this should be stated explicitly somewhere.\n\n- In line 130 I’m not sure that custom is the correct word? Perhaps modified? If not then custom to what? This paper?\n\n- Why is the accuracy not done per class in Table 1?\n\n- How prone is the errors in this method to the errors in the initial labelling? Has there been any evaluation of the calibration  performed for the labels? This also seems to be an issue with the evaluation of the method : Is the reported  accuracy relative to that label calibration?\n\n- What frames of reference are used in needs to be made a bit clearer throughout the paper. For example, in line 136, is the rotation in the base frame or the camera frame? It is the same for the equations, I suggest adding the frame explicitly in the notations of the variables (while explaining the notation), so that everything is clearer.\n\n- Lines 143-148 are not entirely clear, what is the canonical location? “The centre of the EE frame”  is not clear, a frame is simply a point with three axis, how can it have a center?  Similarly, how  are the rules  lines 153-154 obtained?\n"}}, {"review_id": "abGUyVT1Uyi", "reviewer": "Reviewer_2Nyt", "summary": "This paper presents a learning-based system for calibrating the extrinsic pose of a robot arm and its end effector by using a depth sensor. The approach collects depth segmentation labels in order for it to identify the robot arm and end effector, and uses these to estimate the end effectors pose. This pose estimate is used in combination with a CAD model of the robot to estimate the \"calibration\" between the camera and the robot, which is the rigid body transformation between the camera and the robot arm's base. Evaluation results on the semantic segmentation component, end effector prediction component, and the final robot transformation. These results show that the calibration errors are relatively small.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- Attempting to solve an important problem in robotics using learning\n- Results appear quite good for the particular environment test scenarios\n\n## Weaknesses\n- W1: The use of learning for this problem is dubious. L294 The paper states \"the system collects and labels its own data\". Because the labels here are automatically collected, why is learning needed? If indeed the ground truth targets can be computed automatically, then why learn anything? Isn't this automatic labeling process already solving the problem? It's possible I'm missing some motivation or key understanding of the method. Can we consider a hypothetical experiment in which we replaced the learner's predictions with the process for automatically collecting the labels, and compare the performance, and explain why learning is useful to this setting?\n\n- W2: The evaluation protocol is unclear. L294 The paper states \"the system collects and labels its own data\". Yet it's not clear how the labels for performance evaluation are collected. I think there needs to be a clear section on the difference between the labels used for training the system vs. the labels used for evaluating the system. Presumably the labels used for evaluating the system are not collected automatically (or at least they must be verified as correct by people in some way)? I think a simulation environment of the sensors would actually be beneficial here, because then ground truth performance metrics could be computed in a way that doesn't rely on human labels.\n\n- W3: The evaluation is quite limited. My understanding is that it is evaluated on a single robot arm in a single lab, at different camera positions. To have strong evidence that the method is robust, more results either with a different robot arm or in a different lab setup location (or both) are needed.\n\n- W4: The data collection protocol is unclear. L110 \"The EE points are further obtained by transforming the EE bounding box using the calibration information and taking the points inside this box.\" I don't follow this explanation. What is the calibration information? Isn't the calibration information the final output (and therefore not yet available)? This prevents me from understanding how the EE semantic segmentation labels are computed. Also, \"The CAD model of the robot along with joint positions and the calibration information can be used to get the arm and EE points, and consequently background points, as well. This would be needed when the robot arm is visible in the frame no matter what, to get the background points.\" Using the word \"can\" here (at the beginning of the sentence) is confusing, because it leads to multiple interpretations (1) One could devise a method to use the CAD model of the robot, *but we did not* (2) One could devise a method to use the CAD model of the robot, *and we did*. Please clarify.\n\n- W5: The method is unclear. Figs 2 and 3 could be significantly improved by including visualizations of the output of each module. Then the reader won't have to mentally visualize each component. Furthermore, it's confusing to separate Fig 2 and Fig 4 across so many pages. I can't determine how the outputs of the module in Fig 2 are used as inputs somewhere in Fig 4. Is it the \"Frame Predictions\" box in Fig 4? This is ambiguous (they should be called the same thing in both figures)?. The box \"Calibration Results\" in Fig 4 is also ambiguous. What exactly are the calibration results? The \"type\" of the results needs to be stated. Is it the 3D rigid transformation from the a robot’s base and the camera? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I have significant doubts that may stem from misunderstanding. Overall, I found the method fairly difficult to understand.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Poor: Difficult to follow and understand.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "Please address my weakness concerns. ", "reviewer_expertise": "2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper"}, "extra_text": {}}, {"review_id": "WQyCnxyUj2T", "reviewer": "Reviewer_5rEJ", "summary": "The paper presents a markerless method to perform hand-eye calibration through learning and estimating the end-effector pose. The idea is not new, which was previously researched in literature but has not applied to the extent presented in this paper. The main contribution is the realization of hand-eye calibration through direct learning of the pose of physical parts through CAD models and depth sensing. The paper is well-organized with a direct result and explanation, although the thoroughness of the results needs further explanation to consolidate the contribution of learning in such scenarios. ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n- Using the proposed method, if the robot’s CAD models are known (which is the case for most robots), no additional markers are needed to perform the calibration.\n\nWeakness: \n- Lack the in-depth analysis of the results is the main problem. For calibration, accuracy is the key here. For the proposed method, the system error is limited by the pose estimation of certain parts of the robot body, especially the end-effector. However, since only a Kinect V1 is used, it is difficult to evaluate the usefulness of the calibration accuracy. Does the error mainly caused by the sensor performance or caused by the algorithm’s performance in pose estimation? The estimated pose is also influenced by the sensor's data quality. \n- Further comparison of the calibrated results against the typical/traditional calibration method using markers is not well presented. Considering this is a reasonably classical problem, this part of the comparison seems to be critically essential to justify the contribution from learning, which is not well-argued based on the current content.\n- Further implementation of the method when applied to other robots. Although this is not that important for the paper's sake, it would be essential to see robot-specific characteristics in terms of design and occlusion, and how to make the process autonomous without interference by human operators. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The major problem is the thoroughness of the results, which is not well presented in depth to demonstrate or prove the contribution of learning in performing this calibration task. It is unclear if learning contributes significantly to the calibration accuracy or if superior calibration accuracy can be achieved using the proposed method even using a consumer-grade camera released years ago. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "- Revision in Paragraph 2, if “using markers is not an ideal solution for most applications,” then what solution is mainly used for robot hand-eye calibration? The logic here is misleading and confusing. \n- Further analysis of the proposed method applied with cameras of higher accuracy as the baseline of comparison. \n- Further analysis of calibration results compared with classical marker/checkboard-based method.\n- Further analysis of the proposed method in dealing with occlusion and if learning can be used to resolve the problem in ways that classical methods cannot.\n- What if simulation data is used, and what would the result look like using the proposed method?\n"}}, {"review_id": "BaTSaOwFEtu", "reviewer": "Reviewer_LhSL", "summary": "The paper tries to address the issue of Robot-to-Camera calibration for a Robot manipulator to a third-person Depth Camera using point cloud-based segmentation of the robot end-effector (EE). Initially, robot-specific data is collected to train the segmentation models to segment and predict specific key points on the robot EE. Using these outputs, additional networks are used to predict the transformation from the EE to the camera, which is refined using ICP with a CAD model of the EE. Using this and the forward kinematics of the robot, the transformation from the robot base to the camera is derived. Finally outlier rejection and averaging is used for multiple frames and robot poses to get the final calibration. The approach shows good performance (within cm range for translation and 3 degrees for rotation calibration) on a Franka robot arm.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- Present a good pipeline that can be trained end-to-end for robot-to-camera calibration. Once trained for a given robot and camera, the approach can, in theory, be applied as and when needed for different experimental settings without the need for re-calibration each time.\n- Overall good performance: \n    * Shows good segmentation between end-effector, robot and background points.\n    * Achieves low calibration error on their specific setting.\n    * Works well even with some occlusions (as seen in the video). \n- Limitations are considered and well addressed at the end of the paper, with respect to overfitting, occlusions etc.\n- Overall writing and organisation of the paper are fairly well done and easy to follow.\n- **The source code is also provided**\n\n(Post-Rebuttal Update)\n- The comparison with the AR Tags shows good performance especially after using ICP which is similar to using Ground Truth segmentation with ICP. \n- The application scenario is better highlighted in Section 4.4 where the approach is trained from one view point and tested on other view points showing the relative generalization capabilities.\n\n### Weaknesses\n- ~~The current approach needs extensive training data (10k robot frames) along with the initial ground truth generated for each robot. This is done using Markers, ICP and Manual tuning.~~\n- ~~There is no comparison with existing works, or even with some baselines like plain ICP, (AR) markers or a combination of these two or a  naive segmentation+ICP, etc. If the approaches are used to just initialize ICP, how much of an advantage does this give compared to say manually or semi-autonomously initializing it? This makes it difficult to gauge the effectiveness of the proposed approach.~~\n- ~~The paper proposes two approaches for predicting the transformations, one using segmented EE points from the point cloud and the other by using the predicted key points of the EE. However, I feel that the paper would benefit from a more principled coupling of the two approaches since they are interdependent, for example, either by jointly training them for a unified pose estimate or by using the segmentation-based prediction to initialise the key point-based prediction which is then subsequently refined. Currently, they are just used to generate pose estimates of multiple frames which are then fed to an outlier rejection and post-processing. While this process works, exploring these kinds of additional approaches for improving the pose prediction could make the approach more principled and novel.~~\n\n(Post-rebuttal Update)\nThe above weaknesses were taken care of in some form (either by experiments or by including them in the limitations section). \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper solves a relatively important problem used in the pipeline of robotic experiments. Having gone through such struggles myself in setting up and properly calibrating a robot w.r.t. an RGBD camera, I understand the relevance of the problem and the impressiveness of the results. ~However, the lack of evaluation with baseline methods (like the ones mentioned above) and the extensive requirement of calibrated data do not fully convince me to argue for a strong acceptance. These are important in order to gauge the effectiveness of the method in making it a fast reliable approach for calibration.~\n\n(Post-Rebuttal Update)\nSome of the concerns have been addressed and the comparison and the accuracy on individual camera poses shows the effectiveness of the proposed approach.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- The description of the data collection on how the key points for the EE are obtained was not clear to me. Is it a 3D bounding box fitted over the points from which fixed points relative to the bounding box frame are used? \n- The translation prediction seems quite ad-hoc. Were these values something that was chosen based on the ideal locations of the EE CAD model points or something like that? The paper would benefit from a description of this process and a also few lines on how this could be extended to other robotic EEs, which can help make the approach more generalizable.\n- The approach could be made more principled by maybe using the outputs of the Key Point Matching (KPM) in some form to possibly improve the prediction of the Rotation Prediction and Transform Calculation (RPT) since I would imagine that using particular key points of interest could help the RPT to predict more accurate poses. Or potentially using an estimate of the transformation to improve the key point detection for further refinement of the predicted transformation. \n- From the code, it seems like the ICP is not used for training the networks. I would imagine that the networks could benefit by using the output after ICP for training."}}]

Author rebuttal:

Decision: Accept (poster)