Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The present work considers the eye-in-hand camera calibration problem, i.e. the problem of estimating the relative pose between a camera's coordinate frame mounted on an end-effector and the end-effector's coordinate frame. The authors evaluate three different learning-based models and compare them against numerous classical baselines. All training is done on synthetic data and evaluated on a real-world platform. The work reports on the surprising finding that direct regression exhibits the best performance.
strengths_and_weaknesses: __Strengths__

- _Motivation:_ The problem considered in this work is highly relevant and well-motivated. It may be worth discussing why non deep-learning methods should fail here.
- _Results_: The results compared to state-of-the-art seem convincing.
- _Video:_ While the video would benefit from some narration and some more details, it still gives a nice first (teaser-style) overview.

__Weaknesses__

- _Literature:_ Numerous general and several specific related works are missing (e.g.  _"Fast Eye-in-Hand 3-D Scanner-Robot Calibration for Low Stitching Errors"_). Some of the works consider even non-overlapping views, e.g. for a depth camera (e.g.  _"Recursive Bayesian Calibration of Depth Sensors with Non-Overlapping Views"_). While I understand, that it is impossible to discuss all of the related work in the calibration space, CoRL does not impose limitations on reference pages and thus, the related work discussion could be much more extensive at least with regards to closely related works.
- _Ablation:_ The work is missing several potential ablations. For example, the use of different orientation representations might impact performance. Similarly, it is not clear whether hyperparameter tuning was performed and to what extent. The work seems to suggest that this was not the case.
- _Baselines:_ Why using OpenCV's baselines is meaningful, I am not sure if they provide the most recent results and I wonder if newer works already outperform these baselines.
summary_of_recommendation: Overall, I believe this work to be interesting and worthy of publication. The method is useful for numerous real-world problems and opens up a relevant research direction of using deep-learning in eye-in-hand calibration. The work could benefit from providing more ablations to better understand the surprising phenomena of successful direct regression but all in all I tend to support its publication.
issues: __General__

- It is claimed (in. l.42) that  _"this calibration can be estimated directly from the camera’s image itself, without requiring any external apparatus, as long as the image can observe part of the end-effector, such as the gripper’s fingers."_  Do I understand correctly that this ends up memorizing a specific end-effector and would not generalize to using a different end-effector without retraining?
- The fact that geometric approaches perform worse is surprising to me as well. I wonder whether this is architecture related and experimenting with different types of architectures would have resulted in a better performance. The work mentions the sensitivity to outliers as a potential reason. Maybe using alignment methods that are robust to outliers (such as the methods developed Luca Carlone's group) could help?
- How were the classical geometric methods integrated into the learning pipeline? Did you use differentiable variants of these methods?

__Appendix__

- It might be easier understandable to present the network architecture parameters in a table as, e.g., on p.2 of [this supplementary](https://www.robots.ox.ac.uk/~namhoon/doc/DESIRE-supp.pdf).

__Code__

- The instructions in the code repository seem to be (at least slightly) incorrect or incomplete. E.g. they say to execute `python direct_regression_model.py` but the file `direct_regression_model.py` does not exist. I assume you meant `example_direct_regression_model.py`? 
- Consider using [this codebase template](https://github.com/paperswithcode/releasing-research-code) for code publishing.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper evaluates different methods for eye-in-hand calibration for the special case that a major part of the gripper is visible. Three alternative deep learning approaches are trained on synthetic data to solve this problem from a single RGB image: (1) Direct regression of the camera to end-effector pose (2) Depth prediction and ICP refinement with the gripper model (3) 2D Keypoint Detection + PnP + RANSAC. Also some classical baselines that detect a single April Tag from different robot poses are compared. In their experiments the direct regression method merged over some frames wins and the reason is found to be noise in the pose predictions of the gripper or marker.
strengths_and_weaknesses: Strengths:

- The problem of single-image eye-in-hand calibration without external markers has not been tackled much, yet (maybe because the assumption that the gripper is always visible is too strong?)
- The approach compares several standard learning approaches that estimate the pose of the gripper in the camera frame and surprisingly shows that some are more accurate than external marker detections
- The simulation experiments on how the different calibration methods react to different types of noise are interesting
- The problem that current depth-sensors are often unable to sense at very close proximity is identified and therefore RGB is used 
- It allows online calibration

Weaknesses:

- The resolution of the RGB sensor is not mentioned even though it has quite some impact on calibration results
- A single April Tag is not really adequate to get a precise calibration. In my experience calibration boards, e.g. the Charuco board lead to higher accuracies than reported in the paper with classical methods
- There is only one robotic experiment using a single camera-robot combination. More evidence is needed that this method consistently works well
- The paper mentions the Render & Compare approach from Labbe et al. but even though it is SoTA for hand-eye calibration, it is not tried here (I expect this to work better than directly regressing the pose).
- The assumption that the gripper is always in view is quite strong. In this case we could also just attach a marker to the gripper.
- For the constrained and un-occluded pose space of the gripper I would also expect classical edge matching techniques to work well
- The approach of pushing keypoints that are out of the image to the image border is questionable 
- Creating the synthetic training data might be more tedious than doing the external calibration
- The achieved standard deviation of >1cm does not seem good enough to me 
summary_of_recommendation: While the tackled problem could be interesting, the proposed evaluation is insufficient to draw any conclusions. The baselines and proposed approaches seem a bit too weak. I would recommend the authors to extend and resubmit their study to more setups, different markers and the render&compare approach or maybe some classical edge matching. For me the approach is also too dependent to have a sufficient part of the gripper in the image but in some specific cases it might make sense.
issues: - Figure 4 top doesn't show green/blue lines
l325 "sythetic"
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: Eye-in-hand calibration estimates the pose of a camera rigidly mounted to some part of a robot, w.r.t. robot base. This paper explores 3 learning-based approaches for eye-in-hand calibration from a single RGB image. All training is done with synthetic images. Experiments compare the proposed approaches with each other and with traditional external marker-based optimization approaches, both in simulation and sim2real.
strengths_and_weaknesses: # Strengths
- paper is well written and addresses a well-motivated problem
- experiments are thorough, and metrics are clearly described
- Figure 2 is particularly helpful to understand the paper

# Clarification
- Does the sparse correspondence model also output the identity of keypoints in addition to their location? If now, how are 2D-3D correspondences decided?

# Weaknesses
- It makes intuitive sense that the direct regression model works best among the 3 proposals in the regime of small viewpoint changes. The sparse correspondence system could have been better designed by taking inspiration from parallel literature in keypoint matching. For example, researchers are having much more success with detecting and matching keypoints between a pair of images, rather than directly regressing keypoints. The problem can be cast as detecting and matching keypoints from query image to an image taken from a canonical known pose with good visibility. Many references can be found in Section 2 of "COTR: Correspondence Transformer for Matching Across Images" (that particular paper is quite new, so I don't expect the authors of this paper to have used it, I am just providing it for is Related Works section.)
- A simulation-only analysis of the performance of these methods on different grippers would have verified whether the conclusions drawn here apply to all grippers, or just the Sawyer gripper.
summary_of_recommendation: I am in favour of accepting this paper because it introduces and thoroughly examines the idea of learning eye-in-hand calibration. However, I am a little skeptical of the conclusion regarding direct regression being significantly better than sparse correspondence **when** the sparse correspondence pipeline is more carefully designed (see above for a suggestion on how to do that).
issues: - Sim-only experiments with some other gripper designs would make this paper stronger, but it is not necessary just for acceptance, in my opinion
- I do not expect the sparse correspondence pipeline suggestion to be followed up in this paper. That can be explored in another paper
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper analyzes three methods of performing eye-in-hand calibration: 1) directly training a neural network to regress to the camera pose, 2) using a neural network to generate sparse keypoints and use Perspective-n-Points + RANSAC, and 3) using a neural network to generate dense correspondences and use ICP. Surprisingly, the authors found that direct regression, which achieves estimation error of 13.4mm and 4.4 degrees, significantly outperformed the other methods that have more informative geometric priors built in. The paper proceeds to analyze why this is the case and offers corresponding explanations – matching sparse keypoints seems too sensitive to noise, and predicted dense correspondences are unrealistically smooth, owing to neural networks’ interpolation bias.
strengths_and_weaknesses: Strengths:
-	The paper conducts a well-focused study.
-	The paper offers surprising and practical insights.
-	The paper is clearly written.
-	Implementation and experiments seem well executed.

Weaknesses:
-	While the authors acknowledge that doing “excessive hyperparameter tuning” for the geometric methods may make the comparison unfair, I would still encourage the authors to try to tune their hyperparameters to see what their performance ceilings are.
-	Specifically, the authors may want to tune not just the network hyperparameters, but the ones for RANSAC and ICP.
-	The authors should discuss how much time it takes to generate the data and train the network. 
- The authors should visualize more specific failure cases for each of the method to better illustrate their differences. This can be done in the supplementary materials.
-	Spacing and font of texts in Figure 2 could be improved.

summary_of_recommendation: My recommendation of weak accept is due to the clean and focused insight the paper provides, even though it does not introduce new algorithmic advances (e.g. proposing ways to mitigate the issues found in the correspondence-based approaches).
issues: Please see strengths and weaknesses.
review_title: null

Meta-review: The proposed direct regression method for eye-in-hand calibration is clearly interesting, particularly because it seems to give very good results and even outperforms classical geometrical methods. All reviewers agree in that this is an approach that is worth considering, but they also state that the comparison to the correspondence-based methods could be more elaborated. As it is, this comparison seems a bit unfair, both in terms of exploiting more the potential of the competing methods (e.g. hyperparameter tuning), and also regarding the actual overall effort required to get the aimed results. For the latter, it clearly needs to be taken into account that training data set generation is also an important step, which seems to be necessary here everytime the setup changes. It is definitely nice to see that the purely learning-based approach outperforms the more classical methods, but it is hard to draw a general conclusion out of this from the given experimental evaluations. The fact that only one specific gripper is used in the simulation and that it is required to always see the gripper in the images also limits this attempt of drawing a general conclusion. In their answer, the authors might consider whether a somewhat weaker claim still highlights the benefit of their proposed approach, but also mention in more detail relations to the recent Render & Compare method by Labbé et al.  

Post-rebuttal:
In their answers to the reviewers, the authors could clarify most of the raised questions. In the updated version of the paper, the work clearly represents a valuable contribution, meeting the criteria for CoRL. 


Decision: Accept (Poster)