Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper presents a training data augmentation technique to learn local keypoint descriptors which are invariant to certain 2D image transformations. The required setup is simpler than that based on 3D geometric transformations. The authors also present analyses that show performance variations with respect to different 2D transformations and the extent of it. Real world robot experiments are also presented.
strengths_and_weaknesses: **Strengths**

The approach section builds on existing works. The key contribution is in results and analyses which shows how 2D geometric transformations of RGB images can lead to competitive keypoint matching and tracking performance as compared to 3D geometric transformation based training. This is relevant to the field as the approach is simple and leverages the benefits of machine learning where one can use more training data (in a somewhat systematic manner) to achieve certain invariances.
It would have been amazing though to be able to combine the two compares techniques to achieve state-of-the-art performance.

Limitations section is clear and will be helpful in further advances.

Section 4.2 and 4.3 are insightful and clearly show the characteristics of the proposed approach.

Real robot experiments are a plus.

**Weaknesses**

In L114, the authors refer to “synthetic correspondences” in contrast with geometric correspondences. The operations they are performing for synthetic correspondences (described in L125-130) are also geometric as they transform the underlying 2D image pixel grid. The authors need to improve the description of their approach.

L113-114 need clarification: “origin of pixels” and “same origin”

L182 - “unprojects”?

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper clearly conveys its approach and results, which can act as an alternative or complementary way of achieving better results. The novelty lies in results and analyses, not in the approach section, and could still be relevant to the community. 

Post-rebuttal:
The authors have done an excellent job in addressing reviewers' concerns. Additional experiments and ablation studies conducted by the authors during the rebuttal are appreciated.

An important point was novelty to which the authors agree that their pipeline uses existing works but the paper mainly argues for simplification of data generation pipeline for training and thus improving performance in the robotics context. As an application, this work seems relevant for the community and is a good validation of 'what one thinks should work does actually work'.

The overall idea and the results are not surprisingly amazing although the execution is great, thus an improved rating.
issues: There are no major issues. The presented analysis adds value to the literature. For being more impactful, the paper would need to show how it can combine both 2D and 3D geometric transformation based supervision to achieve state-of-the-art performance.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes self-supervised learning of dense visual descriptors from unordered RGB images using data augmentation and contrastive losses. This is unlike prior works that require obtaining pixel-level correspondences from camera poses. Training works by sampling pairs of augmentations (affine, perspective, scale, etc) on the same input image, using a ResNet-34 to produce per-pixel descriptors, then using a contrastive loss to align descriptors of corresponding pixels. While this method loses the semantic and category-level information from priors works (e.g. Dense Object Nets), it is much simpler and offers competitive keypoint tracking performance. The authors use the trained descriptors in a downstream bin-picking task, where humans label suction grasp points that are then used to find new grasp points via descriptor matching. The proposed method achieves a 98.9% grasp success rate, higher than the 79.9% achieved by a baseline that does not use these pretrained descriptors.
strengths_and_weaknesses: Strengths

The paper is well-written. The method is simple to implement and the experiments are clear. I appreciate the authors’ ablations on the effects of different image augmentation types. Discussion of the failure cases on the model’s relatively poor performance on novel camera perspectives is also appreciated.

Weaknesses

My main complaint about the work is on generalization of the trained descriptor networks on novel objects, and this manifests in a couple different ways. As the authors note, the proposed method works best on known objects. While this method would still be useful (since the network clearly generalizes in other ways, like clutter configuration, camera views, lighting, etc), additional experiments and/or discussions on object-wise generalization would be helpful. In particular, it would be interesting to do the keypoint tracking experiments on completely novel objects, just to see what those numbers are. 

For the bin picking experiments, the authors describe the scene as consisting of “ten known objects, but are not contained in the training set.” I’m not sure what this means. One guess is that this means the 10 objects have human-labeled grasps, but they are not in the dataset used to train the descriptor network. If this is true, then does this mean the network does indeed have some generalization capabilities to novel objects?

Lastly, I think the paper would benefit from a quick comparison against training a descriptor network on a much larger but more generic dataset. This would be similar to other works that learn self-supervised image embeddings, except the output is per-pixel instead of per-image. This comparison can help further illustrate the extent of which domain-specific or even instance-specific training data is required.

Perhaps for future work (this would be a stretch for rebuttal, but it’d be impressive), the authors could use an equivariant CNN architecture that is equivariant to translations and rotations for better sample complexity and avoiding certain image augmentation modes. Here’s an example of using equivariant networks in robot manipulation: https://proceedings.mlr.press/v164/wang22j.html

One small comment: Figure 4b should just have 2 colors and 2 line types, instead of 2 colors and 4 line types. 

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I recommend weak accept because while the proposed method could be useful, I can see how generalization may be a contentious point for other reviewers or the community at large.
issues: Please see strengths/weaknesses section.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a data augmentation strategy of RGB images for self-supervised visual descriptor learning. Previous approaches require geometric correspondences derived from image registration via e.g., reconstructions of video sequences, to compute a contrastive (or variant of) loss during training. The presented approach simply augments a single RGB input image and learns an expressive pixel feature representation through the proposed "synthetic correspondences", i.e., same pixel after undergoing image augmentation. Experiments show that learning from synthetic correspondences achieves similar performance to learning from geometric correspondences in terms of keypoint matching. The method is used to demonstrate a robotic grasping task in which hand labelled grasp locations on objects can be identified in new scenes through pixel descriptor matching.
strengths_and_weaknesses: ## Strengths

The problem of learning visual descriptors is important and highly relevant for many applications such as robot manipulation as demonstrated by the authors. The authors have also covered the relevant literature sufficiently.

The technical approach is sound with a mostly good description of the approach (although there are some details missing as discussed later).

The experiments demonstrate that the proposed approach achieves good success in comparison to alternatives. They especially highlight the capability of good keypoint tracking without the requirement of generating geometric correspondences. Lastly, the grasping experiments make this paper highly relevant to the themes of CoRL.

## Weaknesses

The main weakness is that the authors have not strongly asserted their technical innovation in light of the SotA. The use of augmentation has already been shown in various papers, e.g., [4, 9, 10]; furthermore, the importance of the different augmentations, namely the affine transformation, have been demonstrated to achieve more robustness. It is argued that other work, e.g. [14], learn image embeddings instead of pixel descriptors but I do not see this as a major technical contribution as it translates to an adjustment of the loss, which is ultimately borrowed from [10], as well as task and metric for evaluation.

I find the grasping experiments poorly explained. What is the model-free grasp pose generation method? There is no explanation (not even in the supplementary) and I can only glean that it detects uniformly flat surfaces from Fig. 7b. Also, why does the final grasp need to be the intersection between the graspable areas and the grasp preference heat map? Who couldn't the raw user-selected pixels from one labelled image be used without having to first compute a heatmap? The original work in [2] shows this capability precisely. The descriptor success and hits metrics are not entirely fair for the baseline since it is completely agnostic to the descriptors. I am actually rather curious why the baseline achieves decent scores but it is hard to reflect on this without knowing more about the underlying method.

I have a minor issue with the metrics used in the experiments due to a lack of clarity. Many of the results are only expressed in terms of the 75% quantile without justification of why this is the best metric. Why was the median dropped after results in Fig. 4? Furthermore, most related work report PCK, as such, I think the authors should argue why their choice of metric is more appropriate than what appears to be the standard.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: While the paper presents an interesting and simple idea to learn visual descriptors, as well as demonstrate this for a robotic application, the technical contribution is relatively low as similar ideas have already been presented in previous work. The main difference appears to be the application of a pixel-wise contrastive loss from [10] for the setting proposed in [14] in order to learn pixel descriptors rather than image embeddings. This is only incremental in terms of contribution. Furthermore, it is difficult to assess the significance of the robotic grasping experiments without more information regarding the model-free grasping method (the performance comparison is unclear without a better understanding of the baseline). Lastly, the metrics used for evaluation are unjustified and differ from prior work. As such, I recommend a weak reject but am open to see a revision that potentially remedies these flaws.
issues: As discussed in "weaknesses", please clarify the main technical contribution in light of existing work on the same topic as well as provide much more detail about the robot grasping experiments. Other minor issues that should be addressed are as follows:

* In Sec. 4 (lines 170-171), it is explained that for geometric correspondence training, the augmentations are chosen with 50% (instead of 100% for synthetic correspondence training). There is no explanation, conjecture or quantitative results to support the statement in the text or supplementary. It would be good to add an analysis not only of this aspect but that 100% augmentation is the best for learning from synthetic correspondences. Interestingly, [14] report that augmenting only one of the images leads to better performance, so it seems that the "right" amount of augmentation is inconclusive.

* Sec. 4.2. could be made more clear with a diagram to explain the camera movements as it is not immediately clear from the text - we have to make a strong assumption about the camera coordinate system. The diagram can go in the supplementary and complement Fig. 8.

* The work does not explicitly address generalisation as the dataset considers the same objects in both testing and training. I was surprised that the robot manipulation experiments include unseen objects in the evaluation and that the descriptor matching appears good (in terms of descriptor success). Could the authors comment on why this is the case? Are the test objects similar to those seen during training?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors propose  a way for dense correspondence descriptor learning that can learn these descriptors from an unordered set of RGB images which can come from a single stationary camera.  They achieve this by transforming the input image with various data augmentations, keeping track of the ground truth correspondences. They then study the performance of this training regime compared to more standard dense descriptor learning that requires self-supervised data gathering with  registered cameras from different viewpoints.
strengths_and_weaknesses: **Summary of Strengths**
- Fair Literature review.
- Approach is clear and well presented.
- Good experimental procedures and analysis.

**Summary of Weaknesses**
- Novelty in the approach somewhat limited.
- Analysis suffers from having a single baseline.
- It is unclear why the grasping experiment is necessary.

**Comments**
- The idea of using synthetic data to train dense descriptors does not seem new (for example Learning Multi-object dense descriptor for autonomous goal-conditioned grasping by Yang et al.), nor does the specific loss used  (coming from Chen et al. A simple framework for contrastive learning of visual representations ). Moreover, the synthetic training from Yang et al. seems like it would provide the same benefits over vanilla dense object nets as the proposed approach, and the proposed approach generally does not perform better than the geometric baseline.It seems to be this work would benefit from expanding on the analysis side. That is, benchmarking more approaches  that are interesting and appear in different previous works, with synthetic or geometric training, and analyse the pros and cons of each approach to build a more wholistic study. This would then result in a very interesting analysis paper, as opposed to proposing a specific approach.

- Using dense descriptor representations for finding human-defined grasp locations in different scene configurations has been shown to work well in the papers from the related work. Showing that using dense descriptors to guide a model-free grasping method seems to have an incremental benefit over this, which distracts from what otherwise seems to be the main point of the paper which is a comparison between the ``standard’’, geometric , way of training dense correspondence descriptors and the synthetic way of training them.


robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper has limited novelty in the method but seems well executed and presented. I believe a much stronger paper could arise from it by further developing the analysis side,  by comparing over multiple baselines and approaches, rather than favouring any particular method as the "proposed approach". That is casting the narrative as "providing insights and finding the pros and cons of different approaches for learning dense correspondence descriptors", and comparing various interesting approaches (including those already analysed) to provide those insights. 
issues: 
- Perhaps the authors can better justify why the specific approach/design choices of synthetic training they used in that paper are interesting in their own right, compared to the other approaches using synthetic training for dense correspondences (for example in Learning Multi-object dense descriptor for autonomous goal-conditioned grasping by Yang et al.)?
- Similarly perhaps the authors could expand on what additional insights do the particular robotics experiments conducted in this paper bring, given that previous works have already demonstrated multiple times that dense descriptors are well suited for finding picking locations?
- line 79 - Title of this section seems a bit odd  with the phrase that follows: If you use a video and optical flow between subsequent images, then it is not descriptors from ``single images’’?
- line 132 - It is unclear what ``geometric datasets’’ refers to?
- Fig. 5 : median is more intuitive, if 75% quantile is needed instead, then it would be good to mention why?
- In section 4.2 it is unclear whether the same datasets where used as in section  4.1?
- In related work it would be nice to situate more explicitly this work with the previous works in the field, where does it draw inspiration and how, and where is it different. This information is currently scattered across the paper, it would be nice it if was made clearer in the related work.
- The loss in eq. (1) is quite central,  could be nicer to have a more complete explanation of it rather that simply refer to previous work : What is M and tau in the loss? What is the intuition behind it?
- Line 154 is a bit unclear:  The phrase as written reads as if the cosine similarity loss was doing the normalisation of the vectors. I believe this is not what was the intended meaning, descriptors can be normalised because the cosine similarity doesn’t impose anything  on the norm of the descriptors, but the normalisation is done independently from the cosine similarity?
- line 170 : It is not clear what the augmentations are: are those the same augmentations as with the synthetic correspondences?
- Are the training and test scenes constrained to have the same objects present? This should be made clearer.
review_title: null

Meta-review: metareview: Reviewers felt positively that the proposed task of learning visual descriptors is important to robotics.  The proposed approach is simple ,reasonable, and seems to work well.  The results and analysis are insightful, and reviewers also appreciated the real-robot experiments.  The reviewers also felt that the paper is well-written

Reviewers felt that the rebuttal was very convincing and addressed most of the original concerns with the paper. The main remaining reviewer concern is that the proposed method is somewhat incremental, with limited technical novelty compared to prior work. Nonetheless, reviewers felt that the paper is interesting and that members of the robot learning community would benefit from reading it.

confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)