Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a learning-based system for calibrating the extrinsic pose of a robot arm and its end effector by using a depth sensor. The approach collects depth segmentation labels in order for it to identify the robot arm and end effector, and uses these to estimate the end effectors pose. This pose estimate is used in combination with a CAD model of the robot to estimate the "calibration" between the camera and the robot, which is the rigid body transformation between the camera and the robot arm's base. Evaluation results on the semantic segmentation component, end effector prediction component, and the final robot transformation. These results show that the calibration errors are relatively small.
strengths_and_weaknesses: ## Strengths
- Attempting to solve an important problem in robotics using learning
- Results appear quite good for the particular environment test scenarios

## Weaknesses
- W1: The use of learning for this problem is dubious. L294 The paper states "the system collects and labels its own data". Because the labels here are automatically collected, why is learning needed? If indeed the ground truth targets can be computed automatically, then why learn anything? Isn't this automatic labeling process already solving the problem? It's possible I'm missing some motivation or key understanding of the method. Can we consider a hypothetical experiment in which we replaced the learner's predictions with the process for automatically collecting the labels, and compare the performance, and explain why learning is useful to this setting?

- W2: The evaluation protocol is unclear. L294 The paper states "the system collects and labels its own data". Yet it's not clear how the labels for performance evaluation are collected. I think there needs to be a clear section on the difference between the labels used for training the system vs. the labels used for evaluating the system. Presumably the labels used for evaluating the system are not collected automatically (or at least they must be verified as correct by people in some way)? I think a simulation environment of the sensors would actually be beneficial here, because then ground truth performance metrics could be computed in a way that doesn't rely on human labels.

- W3: The evaluation is quite limited. My understanding is that it is evaluated on a single robot arm in a single lab, at different camera positions. To have strong evidence that the method is robust, more results either with a different robot arm or in a different lab setup location (or both) are needed.

- W4: The data collection protocol is unclear. L110 "The EE points are further obtained by transforming the EE bounding box using the calibration information and taking the points inside this box." I don't follow this explanation. What is the calibration information? Isn't the calibration information the final output (and therefore not yet available)? This prevents me from understanding how the EE semantic segmentation labels are computed. Also, "The CAD model of the robot along with joint positions and the calibration information can be used to get the arm and EE points, and consequently background points, as well. This would be needed when the robot arm is visible in the frame no matter what, to get the background points." Using the word "can" here (at the beginning of the sentence) is confusing, because it leads to multiple interpretations (1) One could devise a method to use the CAD model of the robot, *but we did not* (2) One could devise a method to use the CAD model of the robot, *and we did*. Please clarify.

- W5: The method is unclear. Figs 2 and 3 could be significantly improved by including visualizations of the output of each module. Then the reader won't have to mentally visualize each component. Furthermore, it's confusing to separate Fig 2 and Fig 4 across so many pages. I can't determine how the outputs of the module in Fig 2 are used as inputs somewhere in Fig 4. Is it the "Frame Predictions" box in Fig 4? This is ambiguous (they should be called the same thing in both figures)?. The box "Calibration Results" in Fig 4 is also ambiguous. What exactly are the calibration results? The "type" of the results needs to be stated. Is it the 3D rigid transformation from the a robot’s base and the camera? 
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I have significant doubts that may stem from misunderstanding. Overall, I found the method fairly difficult to understand.
issues: Please address my weakness concerns.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents a markerless method to perform hand-eye calibration through learning and estimating the end-effector pose. The idea is not new, which was previously researched in literature but has not applied to the extent presented in this paper. The main contribution is the realization of hand-eye calibration through direct learning of the pose of physical parts through CAD models and depth sensing. The paper is well-organized with a direct result and explanation, although the thoroughness of the results needs further explanation to consolidate the contribution of learning in such scenarios. 
strengths_and_weaknesses: Strength: 
- Using the proposed method, if the robot’s CAD models are known (which is the case for most robots), no additional markers are needed to perform the calibration.

Weakness: 
- Lack the in-depth analysis of the results is the main problem. For calibration, accuracy is the key here. For the proposed method, the system error is limited by the pose estimation of certain parts of the robot body, especially the end-effector. However, since only a Kinect V1 is used, it is difficult to evaluate the usefulness of the calibration accuracy. Does the error mainly caused by the sensor performance or caused by the algorithm’s performance in pose estimation? The estimated pose is also influenced by the sensor's data quality. 
- Further comparison of the calibrated results against the typical/traditional calibration method using markers is not well presented. Considering this is a reasonably classical problem, this part of the comparison seems to be critically essential to justify the contribution from learning, which is not well-argued based on the current content.
- Further implementation of the method when applied to other robots. Although this is not that important for the paper's sake, it would be essential to see robot-specific characteristics in terms of design and occlusion, and how to make the process autonomous without interference by human operators. 

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: The major problem is the thoroughness of the results, which is not well presented in depth to demonstrate or prove the contribution of learning in performing this calibration task. It is unclear if learning contributes significantly to the calibration accuracy or if superior calibration accuracy can be achieved using the proposed method even using a consumer-grade camera released years ago. 
issues: - Revision in Paragraph 2, if “using markers is not an ideal solution for most applications,” then what solution is mainly used for robot hand-eye calibration? The logic here is misleading and confusing. 
- Further analysis of the proposed method applied with cameras of higher accuracy as the baseline of comparison. 
- Further analysis of calibration results compared with classical marker/checkboard-based method.
- Further analysis of the proposed method in dealing with occlusion and if learning can be used to resolve the problem in ways that classical methods cannot.
- What if simulation data is used, and what would the result look like using the proposed method?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper tries to address the issue of Robot-to-Camera calibration for a Robot manipulator to a third-person Depth Camera using point cloud-based segmentation of the robot end-effector (EE). Initially, robot-specific data is collected to train the segmentation models to segment and predict specific key points on the robot EE. Using these outputs, additional networks are used to predict the transformation from the EE to the camera, which is refined using ICP with a CAD model of the EE. Using this and the forward kinematics of the robot, the transformation from the robot base to the camera is derived. Finally outlier rejection and averaging is used for multiple frames and robot poses to get the final calibration. The approach shows good performance (within cm range for translation and 3 degrees for rotation calibration) on a Franka robot arm.
strengths_and_weaknesses: ### Strengths
- Present a good pipeline that can be trained end-to-end for robot-to-camera calibration. Once trained for a given robot and camera, the approach can, in theory, be applied as and when needed for different experimental settings without the need for re-calibration each time.
- Overall good performance: 
    * Shows good segmentation between end-effector, robot and background points.
    * Achieves low calibration error on their specific setting.
    * Works well even with some occlusions (as seen in the video). 
- Limitations are considered and well addressed at the end of the paper, with respect to overfitting, occlusions etc.
- Overall writing and organisation of the paper are fairly well done and easy to follow.
- **The source code is also provided**

(Post-Rebuttal Update)
- The comparison with the AR Tags shows good performance especially after using ICP which is similar to using Ground Truth segmentation with ICP. 
- The application scenario is better highlighted in Section 4.4 where the approach is trained from one view point and tested on other view points showing the relative generalization capabilities.

### Weaknesses
- ~~The current approach needs extensive training data (10k robot frames) along with the initial ground truth generated for each robot. This is done using Markers, ICP and Manual tuning.~~
- ~~There is no comparison with existing works, or even with some baselines like plain ICP, (AR) markers or a combination of these two or a  naive segmentation+ICP, etc. If the approaches are used to just initialize ICP, how much of an advantage does this give compared to say manually or semi-autonomously initializing it? This makes it difficult to gauge the effectiveness of the proposed approach.~~
- ~~The paper proposes two approaches for predicting the transformations, one using segmented EE points from the point cloud and the other by using the predicted key points of the EE. However, I feel that the paper would benefit from a more principled coupling of the two approaches since they are interdependent, for example, either by jointly training them for a unified pose estimate or by using the segmentation-based prediction to initialise the key point-based prediction which is then subsequently refined. Currently, they are just used to generate pose estimates of multiple frames which are then fed to an outlier rejection and post-processing. While this process works, exploring these kinds of additional approaches for improving the pose prediction could make the approach more principled and novel.~~

(Post-rebuttal Update)
The above weaknesses were taken care of in some form (either by experiments or by including them in the limitations section). 

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper solves a relatively important problem used in the pipeline of robotic experiments. Having gone through such struggles myself in setting up and properly calibrating a robot w.r.t. an RGBD camera, I understand the relevance of the problem and the impressiveness of the results. ~However, the lack of evaluation with baseline methods (like the ones mentioned above) and the extensive requirement of calibrated data do not fully convince me to argue for a strong acceptance. These are important in order to gauge the effectiveness of the method in making it a fast reliable approach for calibration.~

(Post-Rebuttal Update)
Some of the concerns have been addressed and the comparison and the accuracy on individual camera poses shows the effectiveness of the proposed approach.
issues: - The description of the data collection on how the key points for the EE are obtained was not clear to me. Is it a 3D bounding box fitted over the points from which fixed points relative to the bounding box frame are used? 
- The translation prediction seems quite ad-hoc. Were these values something that was chosen based on the ideal locations of the EE CAD model points or something like that? The paper would benefit from a description of this process and a also few lines on how this could be extended to other robotic EEs, which can help make the approach more generalizable.
- The approach could be made more principled by maybe using the outputs of the Key Point Matching (KPM) in some form to possibly improve the prediction of the Rotation Prediction and Transform Calculation (RPT) since I would imagine that using particular key points of interest could help the RPT to predict more accurate poses. Or potentially using an estimate of the transformation to improve the key point detection for further refinement of the predicted transformation. 
- From the code, it seems like the ICP is not used for training the networks. I would imagine that the networks could benefit by using the output after ICP for training.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors are tackling the problem of calibrating an external camera looking at a robot arm. They use learning based methods along with geometric considerations to perform the calibration.  Their method allows calibration to be performed quickly, without the need of specific calibration objects such as calibration tags.

strengths_and_weaknesses: **Summary of Strengths**
- The paper tackles an interesting problem that can have a large impact in robotics.
- An interesting and informative method is presented, combining learning components and geometric considerations.

**Summary of Weaknesses**
- It would have been better to do an actual comparison/benchmarking against papers performing the same task, namely Labbe et al, and Lee et al.
- The evaluation results seem to be dependent on the quality of the “ground truth” predictions, which can also be prone to errors, and there is no provided estimate of how large these errors may be.
- Although one can follow what is being done, some points on the presentation can be improved.


**Comments**

- The extrinsic calibration and EE pose estimation are very closely related (by equation 6) , yet they are contrasted in the introduction without further explanation: This is confusing when reading it the first time and can be made clearer. Similarly, having the pose estimation and the extrinsic calibration in the same table later on compounds on the confusion. What is especially unclear is which rows of the table are used to perform the calibration. From  my understanding, multiple estimates of the ICP variants of both pose estimation heads are used independently to obtain the calibration through averaging and outlier rejection, but this is not very clear overall.  

- Both the method and evaluation seem to be dependent on a “ground truth labelling” but we have no information on the accuracy of those ground truth values. In the experiment section in lines 214-216 it is unclear what the “ground truths” are: Is it the EE pose obtained through the robot kinematics? Is the calibration label obtained as described in line 101? This would need to be made clearer to better understand the results. The authors recognise this issue in lines 266- 269, but I believe this point needs to be expanded on and talked about earlier on, and it should also be added in the limitations of the paper.

- Lines 247 - 249 : It would be good to provide some insight as to why this may be the case, either experimentally, or by conjecture.



robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The method presented by the paper is interesting and informative, and the problem tackled is important. However, there are some important points that are currently unclear in the paper, such as the possible reliance of the presented results to labelling that may also be error prone, without an estimate of what that error may be. 

issues: It would be great if the authors could address the following:

- Line 112 : is that alternative used? Or is simply a proposed alternative in case the previous steps cannot be performed. This could be made clearer.
-It seems only point clouds are used throughout the paper as input modalities, this should be stated explicitly somewhere.

- In line 130 I’m not sure that custom is the correct word? Perhaps modified? If not then custom to what? This paper?

- Why is the accuracy not done per class in Table 1?

- How prone is the errors in this method to the errors in the initial labelling? Has there been any evaluation of the calibration  performed for the labels? This also seems to be an issue with the evaluation of the method : Is the reported  accuracy relative to that label calibration?

- What frames of reference are used in needs to be made a bit clearer throughout the paper. For example, in line 136, is the rotation in the base frame or the camera frame? It is the same for the equations, I suggest adding the frame explicitly in the notations of the variables (while explaining the notation), so that everything is clearer.

- Lines 143-148 are not entirely clear, what is the canonical location? “The centre of the EE frame”  is not clear, a frame is simply a point with three axis, how can it have a center?  Similarly, how  are the rules  lines 153-154 obtained?
review_title: null

Meta-review: metareview: Dear Authors,

Thank you for submitting your manuscript to CoRL. I'm happy to inform you that your paper has been accepted. We have completed the review of your manuscript and a summary is appended below. 
- The responses from authors have addressed concerns of most reviewers.
- The reviewers have advised accepting your manuscript as a poster after improvement of the quality of the manuscript based on the comments. 
- Please note it is crucial to incorporate all provided explanations by authors and recommended editing by reviewers into the final manuscript.

Regards,

confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)