Learning Eye-in-Hand Camera Calibration from a Single Image
Authors: Eugene Valassakis
Kamil Dreczkowski
Edward Johns
Venue: CoRL2021 Poster (2021)
Keywords: Camera Calibration
Robot Manipulation
Sim-to-Real

Abstract: Eye-in-hand camera calibration is a fundamental and long-studied problem in robotics. We present a study on using learning-based methods for solving this problem online from a single RGB image, whilst training our models with entirely synthetic data. We study three main approaches: one direct regression model that directly predicts the extrinsic matrix from an image, one sparse correspondence model that regresses 2D keypoints and then uses PnP, and one dense correspondence model that uses regressed depth and segmentation maps to enable ICP pose estimation. In our experiments, we benchmark these methods against each other and against well-established classical methods, to find the surprising result that direct regression outperforms other approaches, and we perform noise-sensitivity analysis to gain further insights into these results.

Learning Eye-in-Hand Camera Calibration from a Single Image
Eugene Valassakis'
Kamil Drezckowski" The Robot Learning Lab Imperial College London United Kingdom krd1 ardi1 ac. uk
Edward Johns
The Robot Learning Lab Imperial College London United Kingdom pevi15@ic.o ac. uk
The Robot Learning Lab
Imperial College London
United Kingdom j.johs@immeeiill ac. uk
Abstract: Eye-in-hand camera calibration is a fundamental and long-studied problem in robotics. We present a study on using learning-based methods for solving this problem online from a single RGB image, whilst training models with entirely synthetic data. We study three main approaches: one direct regres- sion model that directly predicts the extrinsic matrix from image, sparse correspondence model that regresses 2D keypoints and then uses PnP, and one dense correspondence model that regressed depth and segmentation maps enable ICP pose estimation. In our experiments, we benchmark these methods against each other and against well-established classical methods, to find the prising result that direct regression outperforms other approaches, and we perform noise-sensitivity analysis gain further insights into these results.
Keywords: Camera Calibration, Robot Manipulation, Sim-to-Real
Introduction
Simulation environment
Real environment
Eye-in-hand setups, where images from a cam- era rigidly mounted to the end-effector are used control robot, are a popular choice for pre- cise robot manipulation. Compared to con- trol using a fixed external camera, the eye-in- hand setup enables precise control because im-
Bim fom-looee roal cxtrinsic cammean ca Eriiiin
ages can be captured very close to an object, as well as generalisation across the workspace when actions are defined relative to the camera end-effector [1, 2, 3]. To apply precise manipulation tasks, these methods require ac- ensure that actions poses inferred in the camera's
Camera view
Cámara siow
Figure 1: Illustration of our task and setup.
curate sye-in-hand extrinsic calibration. frame can be transformed to the robot's frame. Whilst calibration free approaches exist, such as end-to-end control [4, 5], these only suitable for model-free policy learning and cannot be used for methods that involve explicit planning or reasoning about 3D space.
Classically, solving the eye-in-hand camera calibration problem is performed off-line [6, 7, 8, 9] and involves the use specific external calibration object, such as a checkerboard or an April Poses of the calibration object can be estimated by the camera, and by recording them from several robot configurations, the rigid-body transformation between the camera and end-effector can be es- timated analytically. Although with perfect information these methods they are in practice limited by the quality of the data they use, including how precisely the pose of the calibration object can be estimated the accuracy of the recorded end-effector poses. Fur- thermore, these methods are unsuited for use by non-experts since they require dedicated external calibration objects and Finally, they are impractical as re-calibration needs to be performed every time the camera position changes due (I) re-mounting, (2) wear and tear, and (3) collisions between the camera and the environment. Deep learning methods can alleviate these drawbacks. Given the initial overhead re- quired to generate training data and to train model, a deep learning-based method can be readily
yield exact calibrations,
calibration dataset be carefully collected for each re-calibration.
"Joint First Author Contribution
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

used to re-calibrate a camera online from a single image using parts of the robot as the calibration object. This potential has been demonstrated for eye-to-hand camera calibration, where an external camera observes the robot from a distance [10, 11].
To the best of our knowledge, there are currently studies on using deep learning methods for eye-in-hand calibration, which is the focus of this paper. We propose that this calibration can be estimated directly from the camera's image itself, without requiring any external apparatus, as long as the image can observe part of the end-effector, such as the gripper's fingers (see fig, 1). Based on this, we identify and benchmark three natural approaches for leveraging the power of deep learn- ing in this way: two based on classical, geometric pose estimation where deep learning is used in place of components that are typically manually engineered, and one which uses deep learning to directly regress the camera calibration matrix. We show in our real-world experiments that the two approaches that rely on geometric analysis perform poorly, and we analyse empirically the structural limitations and challenges that cause this. We also show that, perhaps surprisingly, direct regression outperforms all alternatives, including classical methods based on fiducial markers with automatic data collection. Finally, given that direct regression can estimate the calibration matrix from single image online, we propose and evaluate a method for fusing multiple estimates of the camera's 's pose to increase the accuracy of the aggregated estimate.
As such, our work has three key contributions: (1) we investigate the suitability of three natural alternatives for using deep-learning for enabling eye-in-hand camera calibration from a single image, (2) we show the potential of end-to-end deep learning for online eye-in-hand camera calibration in everyday environments, and (3) we provide an analysis for the two learning-basee based methods that rely on geometric approaches, through which we discover why these methods may not be suitable for eye-in-hand camera calibration. An accompanying video can be found at https://wwws wobot//twwwooootoo/ooo/oo/oooo.. robot- -erooningennnnnnn..
learning-eye in-iiin-------e candd-callbbae calibration.
2 Related Work
Eye-in-hand camera calibration is a long-studied problem, with significant advances introduced in the 1990's, with what are now well-established solutions. The problem consists of inferring the camera to end-effector pose for a camera mounted on the wrist of robotic manipulator. To do so, several end-effector robot base poses are recorded, along with corresponding estimates of calibration object to camera poses. The problem is then reduced to solving the AX- = X equation through formalism introduced by Shiu and Ahmad in [12]. Several works then followed that mainly differ in their strategy for solving the AX- = X equation. For instance, Tsai and Lenz [7] improve on the efficiency of Shiu and Ahmad's s solution by proposing a closed-form solution. Park and Martin [8] and Dornaika and Harod [9] also consider such closed-form solution while relying on Lie theory and unit quaternions respectively.
While the above approaches solve for rotation and translation separately. Dornaika and Harod [9] also propose non-linear technique for solving for both simultancously. Daniilidis [6] proposes a solution to the problem using dual quaternions. Finally, more recent works study various extensions for particular settings such as using structure for motion for calibration [13], using model- based pose estimation and tracking for online calibration [14], simultaneously considering the data time syn- chronisation problem [15], calibrating depth sensors with non- overlapping views [16], automatically doing both calibration pattern localisation and eye-in-hand camera calibration [17]. and optimising the robot kinematics parameters and the camera to end-effector pose simultaneously [18].
Recently, deep learning methods have had great successes on closely related tasks such as pose esti- mation [19, 20] and end-to-end robotics manipulation from wrist-mounted cameras [1. 4]. Moreover, sim-to-real transfer has shown promise in alleviating the large data requirements that hinder the scal- ability of such methods [4. 21, 22, 23]. In this work, we investigate the naturally emerging question of how one might use such methods for eye-in-hand camera calibration, which is fundamental to a large number of robotics pipelines. Closest to our work, Lee et al. [11] and Labbe et al. [10] investigate two different deep learning approaches for eye-to-hand calibration of a camera looking straight at a robot. [11] is based on keypoint regression followed by Perspective-n-Poinn (PnP) [24] approach. In our work, we consider the eye-in-hand setup that is more common when precise manipulation needs to be achieved by looking closely at the end- effector of the robot [4, 25]. This problem also differs in its properties from the eye-to-hand setup,
while [10] adopts a render-and-compe

---

Regression
Sparse Correspondence
Correspondence
Niggoker location
Image enoader
Daod Agrondaa
laage
THI RETERAAA
Rully
Ioago
Irape
Inaupe
Transbeiio
Rocation unording on
SputialSof-Argmin
Depth prodiction
Segmentation map
20 Keypste botter
ICP iniioliioionn estiaaee
Sermental depth imure
Corropondatces
decoder
Endsefliatoo model
PrP RANSAC
ICP pose estimation
Comara afeere waddimottio
Carara
Carara esf-efhcaer trweffomnace
vradferrtte
Figure 2: From left to right: Illustration of our Direct Regression, Sparse Correspondence and Dense Correspondence models.
since in typical image there is much more constrained view that only allows for the tip of the end-effector to visible, and used as an anchor for pose estimation.
Methods
In this section, we present three methods that are conceptually promising for eye-in-hand camera calibration from single RGB image, and which also raised interesting and novel scientific questions that we discuss in sections and 5. The first method is an end-to-end direct regression method that estimates the camera's pose from RGB image. In contrast, the other two methods deep learn- ing to provide key missing components for well-established geometric pose estimation approaches. The latter have been shown work well in other problem settings [11, 26, 27, 28], but eye-in-hand camera calibration has the particularity that not only the camera is too close to the end-effector for typical vision sensors to capture a depth image of it, but also the visible geometry varies widely within our inference space. To test whether it is still feasible to use such approaches, our second method uses deep learning to regress the 2D locations of predefined keypoints on the end-effector, and passes them to the PnP [24] algorithm to solve for the camera's 's pose. The main challenge with to
this method is that the large variations in the visible geometry lead to keypoint occlusions some keypoints being often outside of the image frame. This is illustrated in fig. 3. Our final method tests whether we can alleviate this issue by using deep learning to regress segmented depth image of the end-effector and an initial guess of the camera's 's pose, and the Iterative Closest Point (ICP) [29, 30, 31] algorithm to refine this initial estimate. All three methods are illustrated in fig. 2.
3.1 Problem Setting
Our problem setting consists of the typical eye-in-hand camera calibration problem, where the aim is to estimate the camera's extrinsic matrix, which is the camera to end-effector pose, TEC [Rg trc] E SE(3), where Rgc E SO(3) and fgc R3 is the orientation and position of the camera the end-effector frame respectively. We also define the end-effector to robot base pose TBE. the calibration object to camera pose as Tco. and an image captured by the wrist-mounted camera as Throughout this paper, we use to denote an estimated quantity.
As opposed to classical approaches, we constrain ourselves to ehn-in-hand camera calibration from a single RGB image without any external apparatus, enabling our methods be readily deployed in the wild. We further constrain ourselves using only synthetic data in order to alleviate the costly data requirements of deep learning approaches, and obtain ground truth labels that facilitate effective learning. Finally, we assume that we have an accurate estimate of the camera intrinsic

---

Simulation Testing Environment
Real Testing Environment
Gripper Model
Keypoint Visibility
Examples of Randomised Images
Figure 3: Left to right: Model of the gripper. Illustration of keypoint visibility from different view- points. Examples of randomised simulation images. Simulation and real world testing environments.
matrix, which the manufacturer typically provides, and access to the CAD model of our robot's end-effector.
3.2 Direct Regression Model
Direct regression is the end-to-end deep learning method we consider. As illustrated in the left diagram of fig. 2, neural network (NN) tasked with regressing the camera's pose from a single RGB image and is trained in simulation using ground truth labels in a supervised setting. The only architectural constraint we introduce comes from the parameterisation of the orientation, where we use the 6D rotation encoding introduced in [32].
Overall, our network outputs a dimensional tensor, with 3 dimensions representing position and 6 the orientation. It consists of a convolutional encoder followed by linear layers that regress the camera pose. We trained with the ground truth labels using the Mean Squared Error (MSE) loss, and we fully breakdown the architecture and training details in our supplementary material.
3.3 Sparse Correspondence Model
Our sparse correspondence model is illustrated in the middle of fig. 2. Starting from an RGB image, uses U-Net [33] type architecture with a spatial-soft-argmaa [34] output activation in order to regress of 2D keypoints in image space. We use the sum of an L1 and MSE losses train network on the ground truth labels, such that each output dimension predicts the 2D location of predefined 3D keypoint. A detailed breakdown of our architecture and training parameters can be found in our supplementary material.
At test time, use the network's output, the corresponding 3D kcypoint locations defined in the object's 's model, and the camera's 's intrinsic matrix to estimate the camera's 's pose with PnP and RANSAC [24], which are popular pose estimation and outlier rejection algorithms.
Finally, in order to alleviate the issue of some keypoints being out of frame depending on the view- point, we (1) consider 38 keypoints that are present in at least 70% of the training images, and (2) for any keypoint that remains out frame, we train the model to predict its projection the im- age plane. This way, at test time, we can infer which keypoints are out of frame by looking at the predictions at the rim, and ignoring these predictions for PnP pose estimation.
3.4 Dense Correspondence Model
Our Dense Correspondence model is illustrated on the right of fig. 2. Starting from an RGB image, uses three independently trained NNs estimate the camera's pose two stages. In the first stage, the three NNs are used to predict (1) depth map, (2) a segmentation mask of the gripper, and (3) an ICP initialisation. the second stage, (1) the segmentation mask used segment the depth map to only return depth values on gripper pixels, (2) the camera intrinsic matrix is used to project the segmented depth map to point cloud, and (3) ICP is used establish dense correspondences between the estimated point cloud and the model of the end-effector and to refine the camera's 's pose estimate from the first stage. The network used for ICP initialisation the same as the one used for direct regression. The same U-Net architecture for the sparse correspondence model is used to predict the depth and segmentation maps. See our supplementary material for further architecture design choices and training details.

---

3.5 Fusing Multiple Estimates
Compared to classical eye-in-hand camera calibration methods, which require calibration to be done offline and with special apparatus, the proposed deep learning alternatives can estimate the camera's ' pose online unstructured environments. This creates the possibility fusing multiple estimates to increase the overall calibration accuracy. In order to explore this, we implemented simple aggregation algorithm that rejects 20% of the least likely samples from a set of candidate estimates under Gaussian data model. It then averages the remaining samples together to yield the final estimate. Our algorithm for this procedure is available in our supplementary material.
3.6 Dataset Generation and Sim-to-Real Transfer
We train our networks entirely with simulated data using the Coppelia [35] simulator. In order to generate the dataset, we randomise the pose of the camera relative to the end-effector at each simulated timestep and record (I) the current RGB, depth and end-effector segmentation images, (2) the ground truth extrinsic matrix, and (3) the 2D keypoint locations in image space.
In order to overcome the "reality gap". we apply visual domain randomisation [21, 22, 23]. Specif- ically, we (1) randomise the colours/textures of the simulated gripper and the light sources on the simulator, (2) replace the background of our images with random images of textures and indoor scenes [36, 37], and (3) apply a post-processing colour jitter operation [38], further randomly per- turbing the brightness, contrast, saturation and hue of the whole image. In total, we generated 10 000 labelled images in approximately 30 minutes, examples of which can be seen in fig. 3.
4 Experiments
In our experiments, we evaluate how the different proposed deep learning approaches compare to each other and to various standard off-the-shelf calibration methods both in simulation and in the real world. From them emerges the surprising result that simple direct regression using end-to-end deep
deep learning outperforms both the classical approaches tested and the correspondence- learning methods. As such, we perform a series of analysis experiments, which we describe in section 5, that are aimed at gaining insights into this surprising observation.
In our comparisons, we include all our deep learning-based methods and the following established methods readily available on OpenvCV [39]: Dainiilidis et al. [6]. Tsai and Lenz [7], Doraika and Harod [9], and Park and Martin [8]. For all experiments, we use the Realsense D435 (or its simulated counterpart) 480 848 resolution, and for the deep learning methods we downsample these
images to resolution of 144 x 256.
Comparing classical methods to deep learning ones fairly be somewhat challenging since classical methods require a training set of several end-effector to robot and calibration object to camera poses perform a single calibration, while our methods only require single image. In sections 4.1 and 4.2 we describe our procedures and what we did to medicate this issue, and in our supplementary material provide our algorithm for each of these procedures.
4.1 Simulation Experiment
Our simulated environment consists of a wrist-mounted camera attached to Sawyer robot's ' gripper that free to move around an April'Tag (see fig. 3). benchmark all methods, we (1) collect a dataset {T,, Tee H5 155 of images and corresponding AprilTag and end-effector poses, with the
tag poses Teo estimated using the AprilTags3 library [40, 41, 42], (2) use all of the 15 datapoints in this dataset estimate the extrinsic matrix using each classical method, and evaluate these estimates, (3) use each of the 15 datapoints in this dataset independently to estimate the extrinsic matrix using each of the proposed learned methods, and evaluate each of the single image estimates, and (4) for each of the learned methods, we fuse together all 15 independent predictions using our fusion procedure described in 3.5 and evaluate the fused estimate.
We repeat this procedure for 100 different ground truth extrinsic matrices, take the average and stan- dard deviation, and display the results in table 1. We evaluate translation and rotation independently: Given an estimate of the extrinsics TEC = [RE with ground truth T [Rel. we
5

---

Simulation
Real World
Method
Ex [mm]
ER [degrees]
Eatd [mm]
TSAI [7]
(171.4 271.2)
(15.9 20.6)
(17.5 18.9)
PARK [8]
(68.7=70.1)
4.73
(15.3 14.3)
HORAUD [9]
(6.3
(16.3
16.6) 16.6)
DANILIDDI [6]
+110.81122.
(11.5+
(14.7
DR
(13.4
#4.4
(10.6+
DR (fusion)
(13.4
(4.4+
(10.4+4.0)
SC
(1.74448.2)
(616.4-338.1)
SC (fusion)
(338.5 301.0)
120.2
(291.2
DC
[93.5
(33.9
+
(21.0+
DC (fuuionn)
(73.4 21.2)
(23.1 13.9)
(15.0=1.7)
Table I: Evaluation of the classical methods, our Direct Regression (DR), Sparse Correspondence
(SC) and Dense Correspondence (DC) methods, and their fusion variants with aggregated estimates.
define the position error as & ||ı /Bcll2. where || lla the 1.2 norm, and the rotational emror as en 0, the angle from the axis-angle representation of the rotation matrix R. = (w, 0) that satisfies the relationship Rgc Ra REE, where and 0 are the axis and angle of rotation.
4.2 Real World Experiment
The real-world evaluation environment is analogous to the simulated environment and consists of Sawyer robot with wrist-mounted camera moving around an AprilTag (see fig. 3). In order to eval- uate our methods in the real world, we (1) collect a training data bank Dtrain = { The Too) 10 and an evaluation dataset Deval = (Ti Too 1 automatically by scripting trajectory around the AprilTag, where AprilTag poses are estimated using the AprilTags3 library [40, 41, 42), (2) we sample 40 training datasets of 15 datapoints each from our training data bank, (3) we use all 15 datapoints in each training set to get an estimate of the camera extrinsic matrix using each classical method, and evaluate that estimate, (4) we use each one of the 15 datapoints in each dataset to get an estimate of the extrinsic matrix using our learned methods, and evaluate each single estimate, and (5) for each of the learned methods, we fuse together all 15 independent predictions using the procedure described 3.5 and evaluate the fused estimate.
We repeat this procedure for two different ground truth extrinsics, calculate the average and stan- dard deviation, and display the results in table 1. Since ground truth extrinsic parameters are not available in the real world, we take inspiration from [6. 7] and use an indirect eтror metric: For each corresponding end-effector robot pose and calibration object to camera pose from the evaluation dataset, (T Tcol E Deval, we estimate the pose of the calibration object in the robot's frame, = [ivo [T ho), where TEC is the estimate of the camera to end-effector pose that we are evaluating. We then define the error metric Estd (1/60 60
Tho = The
1/2
|1:B0 14|
as the standard deviation of the estimated calibration object position, where 1/60 >in 2 is the estimated mean object position.
We emphasise that although the AprilTag is visible in all images in both the simulated and real environments, it is never used to help the calibration of our deep learning methods. We these images simply to ensure that we have the exact same calibration and evaluation datasets for both classical and our learned methods.
4.3 Results
All benchmarking results are shown in table 1. Our main observation is that the direct regression method outperforms all others, which is surprising result. First, it was not expected to outperform classical methods since they use analytical solutions to estimate the extrinsics, which in the absence of noise in the system should give perfect calibration. We believe this stems from using automatic data gathering for the classical methods, which does not allow for carefully curating the calibration dataset and the distribution of poses within it, a manual trial-and-error process that is

---

generally required to bolster calibration accuracy. This also raises the question about the sensitivity of these methods said noise, which we investigate in section 5.
Second, it was also not expected to outperform the correspondence-aasee baadd learned methods, since those introduce strong geometric constraints that intuitively should help guide the model good solutions. We thoroughly investigate the reasons behind this observation in sections 5.2 and 5.3.
Analysis
In this section, we aim to build deeper understanding of the surprising results that we observed in our experiments, which showed that a direct regression of calibration parameters outperforms methods that incorporate well-understood geometric modelling. We split this section into three parts, each analysing the performance of one particular approach.
5.1 Classical Methods
Classical Methods
To better understand the results from our experiments in section 4, we first aim to assess how sensitive classical methods are to sources of noise in the system. Assuming the main source of error stems from the calibration ob-
1.00
Iassuu
Colbration Roditon Emor (m)
ject pose estimation, we fix every other source of noise in simulation to their ground truth values. We then vary the noise injected to the calibration object's 's poses by con- trolled amounts and observe how this affects the quality of the calibration result from classical methods.
0.10
1:00
23.0
200
1000
Precisely, we vary the noise levels from Orm to 10mm and 0" to 10" in increments of 0.5mm and 0.5%, and for each noise tier, we perform calibration with each of the
Postton and Oriemaatio Nokio Magnitude
Sparse Correspondence Model
framre
classical methods considered. We then compare the result to the ground truth extrinsics and obtain position errors.
Collraroon Pottion Emer
100
We repeat this procedure for 100 camera extrinsics, with the average errors obtained and their standard deviations illustrated in the top graph of fig. 4.
30'
We can that, as expected, with perfect information, the classical methods return an exact solution. How- ever, we also see that the calibration quality rapidly de- teriorates with increased noise in the calibration object's poses. This illustrates our motivation for using a deep learning- based approach, since our methods are indepen- dent of any test-time data gathering or calibration object pose estimation quality.
30
Notoc Magntude (pinds))
Figure 4: Top: Analysis of sensitivity to noise in the estimated tag positions of classical methods. Bottom: Analysis of sensitivity to noise in the estimated keypoint positions of PnP+RANSAC
5.2 Sparse Correspondences
In order to better understand why our sparse correspondence model did not perform strongly we set up another controlled experiment. Starting from ground truth 2D-3D correspondences, we add fixed amounts of noise to the 2D keypoint locations and observe the effect of this the quality of PnP RANSAC pose estimation. Specifically, we iterate through 200 random extrinsic matrices, and for each (2D keypoints, 3D keypoints, extrinsic} tuple, we perform PnP + RANSAC with the noise-injected 2D keypoints and compare the result to the ground truth. We repeat this for different magnitudes of noise, ranging from 0 to 9 pixels in increments of 1 pixel, and averaged over all the datapoints considered. We plot the resulting errors in the "70% in frame" curve in the bottom graph of fig. 4.
During our investigation we also considered using a smaller number of keypoints, but ones that always remain in frame. With those we observed that even though this is an easier task for the network prediction, our final extrinsics estimation did not improve. In order to understand this behaviour, we repeat the controlled noise experiment but only considering the keypoints that always appear in frame in the images, which resulted in 12 remaining keypoints clustered on the gripper's fingers. This is plotted in the "100% in frame" curve in the bottom graph of fig. 4.
7

---

Image
Depth Maps
Projected Depth Maps
Predictions
Ground Truth
Figure 5: Left: RGB image of the gripper rendered in simulation. Top centre: Depth image predicted by model from the simulated RGB image. Top right: point cloud projected from the predicted depth image. Bottom centre: Depth image rendered in simulation from the same viewpoint. Bottom right, point cloud projected from the ground truth depth image.
From fig. 4, we clearly see that there is very high sensitivity of sparse correspondence pose estima- tion to errors in 2D keypoint pixel locations. For the keypoints our training set, to get an extrinsic calibration position error of less than lcm, networks' average keypoint prediction error would need to be less than 4 to pixels. This is with the additional challenge of the network having to keep consistent predictions even though sometimes the keypoints appear out of frame. On the other hand, we consider the easier to learn problem of only predicting keypoints that always appear in-frame, the sensitivity skyrockets after single-pixel average error in 2D keypoint locations. Overall, be- lieve that this sensitivity to noise is the primary reason for why the sparse correspondence model did give strong performance.
5.3 Dense Correspondences
Although conceptually using dense geometric correspondence approach should alleviate some of the drawbacks of using sparse correspondences, found that in practice model that uses ICP to refine estimates made by the direct regression method decreases their accuracy.
The reason for this becomes apparent when considering the illustrations shown in fig. 5. Although depth values obtained by the neural networks look appealing when projected onto image plane, they fall short when considering the full 3D structure. We conjecture that the main reason for structure from such close distances, depth values need
this is that in order recover a correct to have strong discontinuous jumps that typical neural networks have difficulty modelling. This is supported by the observations fig. 5: We can see that the overall depth increase seems to match the ground truth values, but there is averaging effect that tends make the depth values increase smoothly, which does not allow us to recover the correct shape. As such, we can conclude that using simple depth regression technique in this setting inappropriate for this task's requirements, with further research needed to overcome the discontinuity problem.
Conclusion
In this work, we presented and evaluated three deep learning-basse methods for online eye-in-hand camera calibration from single image. We trained all our models entirely on synthetic images, and evaluated them against each other and long-established calibration methods in simulation and the real world. Surprisingly, experiments indicated that direct regression method from images to camera extrinsics outperformed other alternatives. order to better understand this result, we then conducted series of introspection experiments, which indicated that strong shortcoming of geometry-basee based methods seems to be their sensitivity to noise in their input data. Finally, while depth map regression for dense correspondences conceptually could have provided the answer to this, our experiments indicated that due to pronounced discontinuities in the depth profiles of such close-up viewpoints, further research is needed, specifically in learning depth prediction for eye-in- hand robot manipulation.

---

Acknowledgments
This work was supported by the Royal Academy of Engineering under the Research Fellowship scheme.
References
[1] E. Johns. Coarse-to-Fing Imitation Learning: Robot Manipulation from Single Demonstra- tion. In IEEE International Conference Robotics and Automation (ICRA), 2021.
[2] Y. Litvak, A. Biess, and A. Bar-Hillel. Learning Pose Estimation for Pigh-Prrcision Robotic Assembly Using Simulated Depth Images. In IEEE International Conference on Robotics and Automation (ICRA), 2019.
[3] E. Johns, Leutenegger, and A. Davison. Deep Learning a Grasp Function for Grasping under Gripper Pose Uncertainty. In IEEE International Conference Intelligent Robots and Systems (IROS), 2016.
[4] E. Valassakis, N. Di Palo, and E. Johns. Coarse-to-Fine for Sim-to-Real: Sub-Millimetre Precision Across Wide Task Spaces. In IEEE International Conference on Intelligent Robots and Systems (IROS), 2021.
[5] E. Y. Puang, K. Peng Tee, and W. Jing. KOVIS: Keypoint-based Visual Servoing with Zero- Shot Sim-to-Real Transfer for Robotics Manipulation. In IEEE International Conference on Intelligent Robots and Systems (IROS), 2020.
[6] K. Daniilidis. Hand-Eye Calibration Using Dual Quaternions. The International Journal of Robotics Research (IJRR), 1999,
[7] R. Y. Tsai, R. K. Lenz, et al. A new technique for fully autonomous and efficient 3D robotics hand/eye calibration. IEEE Transactions on Robotics and Automation, 1989.
[8] F. C. Park and B. J. Martin. Robot sensor calibration: solving AX= XB on the Euclidean group. IEEE Transactions on Roboties and Automation, 1994.
[9] R. Horaud and F. Domaika. Hand-eye Calibration. The International Journal of Robotics Research (IJRR), 1995.
[10] Y. Labbé, J. Carpentier, M. Aubry, and J. Sivic. Single-view robot pose and joint angle es- timation via render & compare. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[11] T. E. Lee, J. Tremblay, T. To, J. Cheng, T. Mosier, O. Kroemer, D. Fox, and S. Birchfield. Homeeratto-obolt Pose Estimation from Single Image. In IEEE International Conference Robotics and Automation (ICRA), 2020.
[12] Y. C. Shiu and S. Ahmad. Calibration of wrist-mounted robotic sensors by solving homoge- neous transform equations of the form AX=XB. IEEE Transactions on Roboties and Automa- tion, 1989.
[13] N. Andreff, R. Horaud, and B. Espiau. Robot Hand-Eye Calibration using Structure from Motion. The International Journal of Robotics Research (IJRR), 2001.
[14] K. Pauwels and D. Kragic. Integrated on-line robot-camera calibration and object pose esti- mation. IEEE International Conference on Robotics and Automation (ICRA), 2016.
[15) Furrer, M. Fehr, T. Novkovic, H. Sommer, I. Gilitschenski, and R. Siegwart. Evaluation of Estimation and Hand-Fye Calibration Robotic Datasets. In Field and Service Robotics (FSR), 2018.
Combined
[16] F. Faion, P. Ruoff, A. Zea, and U. D. Hanebeck. Recursive Bayesian calibration of depth sensors with non-overlapping views. In International Conference on Information Fusion (FU- SION), 2012.

---

[17] M. Antonello, A. Gobbi, S. Michieletto, S. Ghidoni, and E. Menegatti. A fully automatic hand-eye calibration system. In European Conference on Mobile Robots (ECMR), 2017.
[18] X. Liu, H. Madhusudanan, W. Chen, D. Li, Ge, C. Ru, and Y. Sun. Fast Eye-in- -and 3-D Scanner-Robo Calibration for Low Stitching Errors. IEEE Transactions on Industrial Electronics, 2020.
[19] Y. Li, G. Wang. X. Ji, Y. Xiang, and D. Fox. DeepIM: Deep Iterative Matching for 6D Pose Estimation. In European Conference on Computer Vision (ECCV), 2018.
[20] Y. Labbé, Carpentier, M. Aubry, and Sivic. CosyPose: Consistent multi-view, multi-object 6D pose estimation. In European Conference Computer Vision (ECCV), 2020.
[21] E. Valassakis, Z. Ding, and E. Johns. Crossing the Gap: A Deep Dive into Zero-Shot Sim- to-Real Transfer for Dynamics. In IEEE International Conference on Intelligent Robots and Systems (IROS), 2020.
[22] Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World. In IEEE Interna- tional Conference on Intelligent Robots and Systems (IROS), 2017.
[23] R. Alghonaim and E. Johns. Benchmarking Domain Randomisation for Visual Sim-to-Real Transfer. In IEEE International Conference Robotics and Automation (ICRA), 2021.
[24] M. A. Fischler and R. C. Bolles. Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography. Association for Computing Machinery (ACM). 1981.
[25] R. L. Haugaard, J. Langaa, C. Sloth, and A. G. Buch. Fast robust peg-in-hole insertion with continuous visual servoing. arXiv e-prints, 2020.
[26] S. Zakharov, I. Shugurov, and S. llie. DPOD: 6D Pose Object Detector and Refiner. In IEEE International Conference on Computer Vision (ICCV), 2019.
[27] R. Sheffer and A. Wiesel. PnP-Net: A hybrid Perspective-n-Poin Network, arXiv e-prints, 2020.
[28] Xiong, C. Liu, and Q. Chen. Region Pixel Voting Network (RPVNet) for 6D Pose Estimation from Monocular Image. Applied Sciences, 2021.
[29] Y. Chen and G. Medioni. Object modeling by registration of multiple range images. In IEEE International Conference on Robotics and Automation (ICRA), 1991.
[30] P. Besl and N. D. McKay. a method for registration of 3-D shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 1992.
[31] Z. Zhang. Iterative point matching for registration of free-form curves and surfaces. Interna- tional Journal of Computer Vision (IJCV), 1994.
[32] Y. Zhou, C. Barnes, J. Lu, J. Yang. and H. Li. On the Continuity of Rotation Representations in Neural Networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[33] O. Ronneberger, P. Fischer, and T. Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. In International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), 2015.
[34] C. Finn, X. Y. Tan, Y. Duan, T. Darrell, S. Levine, and P. Abbeel. Deep Spatial Autoencoders for Visuomotor Learning. In IEEE International Conference on Robotics and Automation (ICRA), 2016.
[35] E. Rohmer, S. P. N. Singh, and M. Freese. CoppeliaSim (formerly V-REP): Versatile and Scalable Robot Simulation Framework. In IEEE International Conference on Intelligent Robots and Systems (IROS), 2013.
10

---

[36] A. Quattoni and A. Torralba. Recognizing indoor scenes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2009.
[37] C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine. One-Shot Visual Imitation Learning via Meta-Leaming, In Conference on Robot Learning (CoRL), 2017.
[38] A. Paszke et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Conference on Neural Information Processing Systems (NeurIPS). 2019.
[39] G. Bradski. The OpenCV Library. Dr. Dobb's s Journal of Software Tools, 2000.
[40] Pupil-AprilTags. URL https: http:ppp....ppppppoopppppppppppp
[41] J. Wang and E. Olson. AprilTag 2: Efficient and robust fiducial detection. In IEEE Interna- tional Conference on Intelligent Robots and Systems (IROS), 2016.
[42] D. Malyuta. Guidance, Navigation, Control and Mission Logic for Quadrotor Full-cycle Au- tonomy. Master thesis, Jet Propulsion Laboratory, 2017.


Reviews:
[{"review_id": "mXAimWHee5y", "reviewer": "Reviewer_mbiM", "summary": "The present work considers the eye-in-hand camera calibration problem, i.e. the problem of estimating the relative pose between a camera's coordinate frame mounted on an end-effector and the end-effector's coordinate frame. The authors evaluate three different learning-based models and compare them against numerous classical baselines. All training is done on synthetic data and evaluated on a real-world platform. The work reports on the surprising finding that direct regression exhibits the best performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Strengths__\n\n- _Motivation:_ The problem considered in this work is highly relevant and well-motivated. It may be worth discussing why non deep-learning methods should fail here.\n- _Results_: The results compared to state-of-the-art seem convincing.\n- _Video:_ While the video would benefit from some narration and some more details, it still gives a nice first (teaser-style) overview.\n\n__Weaknesses__\n\n- _Literature:_ Numerous general and several specific related works are missing (e.g.  _\"Fast Eye-in-Hand 3-D Scanner-Robot Calibration for Low Stitching Errors\"_). Some of the works consider even non-overlapping views, e.g. for a depth camera (e.g.  _\"Recursive Bayesian Calibration of Depth Sensors with Non-Overlapping Views\"_). While I understand, that it is impossible to discuss all of the related work in the calibration space, CoRL does not impose limitations on reference pages and thus, the related work discussion could be much more extensive at least with regards to closely related works.\n- _Ablation:_ The work is missing several potential ablations. For example, the use of different orientation representations might impact performance. Similarly, it is not clear whether hyperparameter tuning was performed and to what extent. The work seems to suggest that this was not the case.\n- _Baselines:_ Why using OpenCV's baselines is meaningful, I am not sure if they provide the most recent results and I wonder if newer works already outperform these baselines.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, I believe this work to be interesting and worthy of publication. The method is useful for numerous real-world problems and opens up a relevant research direction of using deep-learning in eye-in-hand calibration. The work could benefit from providing more ablations to better understand the surprising phenomena of successful direct regression but all in all I tend to support its publication.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {"issues": "__General__\n\n- It is claimed (in. l.42) that  _\"this calibration can be estimated directly from the camera’s image itself, without requiring any external apparatus, as long as the image can observe part of the end-effector, such as the gripper’s fingers.\"_  Do I understand correctly that this ends up memorizing a specific end-effector and would not generalize to using a different end-effector without retraining?\n- The fact that geometric approaches perform worse is surprising to me as well. I wonder whether this is architecture related and experimenting with different types of architectures would have resulted in a better performance. The work mentions the sensitivity to outliers as a potential reason. Maybe using alignment methods that are robust to outliers (such as the methods developed Luca Carlone's group) could help?\n- How were the classical geometric methods integrated into the learning pipeline? Did you use differentiable variants of these methods?\n\n__Appendix__\n\n- It might be easier understandable to present the network architecture parameters in a table as, e.g., on p.2 of [this supplementary](https://www.robots.ox.ac.uk/~namhoon/doc/DESIRE-supp.pdf).\n\n__Code__\n\n- The instructions in the code repository seem to be (at least slightly) incorrect or incomplete. E.g. they say to execute `python direct_regression_model.py` but the file `direct_regression_model.py` does not exist. I assume you meant `example_direct_regression_model.py`? \n- Consider using [this codebase template](https://github.com/paperswithcode/releasing-research-code) for code publishing."}}, {"review_id": "Y1i7S1u3WJj", "reviewer": "Reviewer_VdxP", "summary": "The paper evaluates different methods for eye-in-hand calibration for the special case that a major part of the gripper is visible. Three alternative deep learning approaches are trained on synthetic data to solve this problem from a single RGB image: (1) Direct regression of the camera to end-effector pose (2) Depth prediction and ICP refinement with the gripper model (3) 2D Keypoint Detection + PnP + RANSAC. Also some classical baselines that detect a single April Tag from different robot poses are compared. In their experiments the direct regression method merged over some frames wins and the reason is found to be noise in the pose predictions of the gripper or marker.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The problem of single-image eye-in-hand calibration without external markers has not been tackled much, yet (maybe because the assumption that the gripper is always visible is too strong?)\n- The approach compares several standard learning approaches that estimate the pose of the gripper in the camera frame and surprisingly shows that some are more accurate than external marker detections\n- The simulation experiments on how the different calibration methods react to different types of noise are interesting\n- The problem that current depth-sensors are often unable to sense at very close proximity is identified and therefore RGB is used \n- It allows online calibration\n\nWeaknesses:\n\n- The resolution of the RGB sensor is not mentioned even though it has quite some impact on calibration results\n- A single April Tag is not really adequate to get a precise calibration. In my experience calibration boards, e.g. the Charuco board lead to higher accuracies than reported in the paper with classical methods\n- There is only one robotic experiment using a single camera-robot combination. More evidence is needed that this method consistently works well\n- The paper mentions the Render & Compare approach from Labbe et al. but even though it is SoTA for hand-eye calibration, it is not tried here (I expect this to work better than directly regressing the pose).\n- The assumption that the gripper is always in view is quite strong. In this case we could also just attach a marker to the gripper.\n- For the constrained and un-occluded pose space of the gripper I would also expect classical edge matching techniques to work well\n- The approach of pushing keypoints that are out of the image to the image border is questionable \n- Creating the synthetic training data might be more tedious than doing the external calibration\n- The achieved standard deviation of >1cm does not seem good enough to me ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While the tackled problem could be interesting, the proposed evaluation is insufficient to draw any conclusions. The baselines and proposed approaches seem a bit too weak. I would recommend the authors to extend and resubmit their study to more setups, different markers and the render&compare approach or maybe some classical edge matching. For me the approach is also too dependent to have a sufficient part of the gripper in the image but in some specific cases it might make sense.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "- Figure 4 top doesn't show green/blue lines\nl325 \"sythetic\"", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "FA-U20AP_CO", "reviewer": "Reviewer_NfZb", "summary": "The paper analyzes three methods of performing eye-in-hand calibration: 1) directly training a neural network to regress to the camera pose, 2) using a neural network to generate sparse keypoints and use Perspective-n-Points + RANSAC, and 3) using a neural network to generate dense correspondences and use ICP. Surprisingly, the authors found that direct regression, which achieves estimation error of 13.4mm and 4.4 degrees, significantly outperformed the other methods that have more informative geometric priors built in. The paper proceeds to analyze why this is the case and offers corresponding explanations – matching sparse keypoints seems too sensitive to noise, and predicted dense correspondences are unrealistically smooth, owing to neural networks’ interpolation bias.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n-\tThe paper conducts a well-focused study.\n-\tThe paper offers surprising and practical insights.\n-\tThe paper is clearly written.\n-\tImplementation and experiments seem well executed.\n\nWeaknesses:\n-\tWhile the authors acknowledge that doing “excessive hyperparameter tuning” for the geometric methods may make the comparison unfair, I would still encourage the authors to try to tune their hyperparameters to see what their performance ceilings are.\n-\tSpecifically, the authors may want to tune not just the network hyperparameters, but the ones for RANSAC and ICP.\n-\tThe authors should discuss how much time it takes to generate the data and train the network. \n- The authors should visualize more specific failure cases for each of the method to better illustrate their differences. This can be done in the supplementary materials.\n-\tSpacing and font of texts in Figure 2 could be improved.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "My recommendation of weak accept is due to the clean and focused insight the paper provides, even though it does not introduce new algorithmic advances (e.g. proposing ways to mitigate the issues found in the correspondence-based approaches).", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please see strengths and weaknesses.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "7xmNCFjJHLX", "reviewer": "Reviewer_SvQJ", "summary": "Eye-in-hand calibration estimates the pose of a camera rigidly mounted to some part of a robot, w.r.t. robot base. This paper explores 3 learning-based approaches for eye-in-hand calibration from a single RGB image. All training is done with synthetic images. Experiments compare the proposed approaches with each other and with traditional external marker-based optimization approaches, both in simulation and sim2real.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- paper is well written and addresses a well-motivated problem\n- experiments are thorough, and metrics are clearly described\n- Figure 2 is particularly helpful to understand the paper\n\n# Clarification\n- Does the sparse correspondence model also output the identity of keypoints in addition to their location? If now, how are 2D-3D correspondences decided?\n\n# Weaknesses\n- It makes intuitive sense that the direct regression model works best among the 3 proposals in the regime of small viewpoint changes. The sparse correspondence system could have been better designed by taking inspiration from parallel literature in keypoint matching. For example, researchers are having much more success with detecting and matching keypoints between a pair of images, rather than directly regressing keypoints. The problem can be cast as detecting and matching keypoints from query image to an image taken from a canonical known pose with good visibility. Many references can be found in Section 2 of \"COTR: Correspondence Transformer for Matching Across Images\" (that particular paper is quite new, so I don't expect the authors of this paper to have used it, I am just providing it for is Related Works section.)\n- A simulation-only analysis of the performance of these methods on different grippers would have verified whether the conclusions drawn here apply to all grippers, or just the Sawyer gripper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am in favour of accepting this paper because it introduces and thoroughly examines the idea of learning eye-in-hand calibration. However, I am a little skeptical of the conclusion regarding direct regression being significantly better than sparse correspondence **when** the sparse correspondence pipeline is more carefully designed (see above for a suggestion on how to do that).", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "- Sim-only experiments with some other gripper designs would make this paper stronger, but it is not necessary just for acceptance, in my opinion\n- I do not expect the sparse correspondence pipeline suggestion to be followed up in this paper. That can be explored in another paper"}}]

Author rebuttal:

Meta-review: The proposed direct regression method for eye-in-hand calibration is clearly interesting, particularly because it seems to give very good results and even outperforms classical geometrical methods. All reviewers agree in that this is an approach that is worth considering, but they also state that the comparison to the correspondence-based methods could be more elaborated. As it is, this comparison seems a bit unfair, both in terms of exploiting more the potential of the competing methods (e.g. hyperparameter tuning), and also regarding the actual overall effort required to get the aimed results. For the latter, it clearly needs to be taken into account that training data set generation is also an important step, which seems to be necessary here everytime the setup changes. It is definitely nice to see that the purely learning-based approach outperforms the more classical methods, but it is hard to draw a general conclusion out of this from the given experimental evaluations. The fact that only one specific gripper is used in the simulation and that it is required to always see the gripper in the images also limits this attempt of drawing a general conclusion. In their answer, the authors might consider whether a somewhat weaker claim still highlights the benefit of their proposed approach, but also mention in more detail relations to the recent Render & Compare method by Labbé et al.  

Post-rebuttal:
In their answers to the reviewers, the authors could clarify most of the raised questions. In the updated version of the paper, the work clearly represents a valuable contribution, meeting the criteria for CoRL. 


Decision: Accept (Poster)