Learning Dense Visual Descriptors using Image Augmentations for Robot Manipulation Tasks
Authors: Christian Graf
David B. Adrian
Joshua Weil
Miroslav Gabriel
Philipp Schillinger
Markus Spies
Heiko Neumann
Andras Gabor Kupcsik
Venue: CoRL 2022 Poster (2022)
Keywords: self-supervised learning
computer vision
representation learning
bin-picking

Abstract: We propose a self-supervised training approach for learning view-invariant
dense visual descriptors using image augmentations. Unlike existing
works, which often require complex datasets, such as registered RGBD sequences,
we train on an unordered set of RGB images. This allows for learning from a single 
camera view, e.g., in an existing robotic cell with a fix-mounted camera. We
create synthetic views and dense pixel correspondences using data augmentations.
We find our descriptors are competitive to the existing methods, despite the simpler
data recording and setup requirements. We show that training on synthetic
correspondences provides descriptor consistency across a broad range of camera
views. We compare against training with geometric correspondence from multiple
views and provide ablation studies. We also show a robotic bin-picking experiment 
using descriptors learned from a fix-mounted camera for defining grasp
preferences.

Learning Dense Visual Descriptors using Image Augmentations for Robot Manipulation Tasks
Christian Graft.1, David B. Adrian 1.122 Joshua Weil!.3, Miroslav Gabriel',
Philipp Schillinger', Markus Spies', Heiko Neumann', Andras Kupesik
Equal contribution, Bossh Center for Artifical Intelligence, Ulm University, 3KTH Royal Institute of Technology
Abstract: We propose a self-supervised training approach for learning view- invariant dense visual descriptors using image augmentations. Unlike existing works, which often require complex datasets, such as registered RGBD sequences, we train on an unordered set of RGB images. This allows for learning from sin- gle camera view, e.g., in an existing robotic cell with fix-mounted camera. We create synthetic views and dense pixel correspondences using data augmentations. We find our approach to be competitive compared to existing methods, despite the simpler data recording and setup requirements. We show that training on synthetic correspondences provides descriptor consistency across broad range of camera views. We compare against training with geometric correspondence from multi- ple views and provide ablation studies. We also show robotic bin-picking periment using descriptors learned from fix-mounted camera for defining grasp preferences.
Keywords: self-supervised learning, computer vision, representation learning, bin-picking
1 Introduction
Scene and object understanding is essential for robot manipulation tasks, including assembly or bin picking. Often, the representation of choice is task-specific segmentation supervised manner with labeled data. Labeling, however, is expen- sive and time-consuming, which is why self-supervised learning of dense visual descriptors has recently gained substantial attention in the robotics community, inspired by the works of [1] and [2].
pose estimation, trained in
Dense Object Nets (DONs) learn dense visual descriptors of objects fully self-supervised in robotic environment [2]. The learned de- scriptors are view- invaiintriant, show potential for category level gen- eralization and they naturally apply to non-rigid objects. The dense descriptor representation can be flexibly used for various downstream robotic tasks, such as, grasping [2, 3, 4], rope manipulation [5] and learning control [6, 7].
Self-supervised training of DONs, however, relies pixel correspon- dences across multiple camera views provided by registered RGBD image sequence, which requires accurate camera calibration and pose recording. Furthermore, pixel correspondence tends to be inaccurate with inexpensive depth cameras, even with data preprocessing. Fi- nally, data collection is constrained by robot kinematics and the need for expert setting up and supervising the procedure.
Figure 1: Bin Bin-picking setup with single, fix-mounted overhead camera.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

In this paper we relax these assumptions fundamentally and instead of complex setup with a single, robot-mounted moving camera, or multiple static ones, we solely rely on an unordered set of RGB images to learn object descriptors, for example, recorded by single fixed camera. In our work, instead of relying on multi-view, geometric correspondence, we use augmentations of single images to obtain alternative views and synthetic correspondence. This idea was already explored in computer vision [8, 9], in the context of learning geometrically consistent pixel-level descriptors across multiple object classes. In this paper we show that relying on synthetic image augmentations achieves competitive performance in terms of keypoint tracking accuracy compared to a network trained with geometric correspondence. Importantly, this approach can easily be adopted to existing industrial setups with fix-mounted cameras, or with cameras too heavy to be mounted on a robotic arm, without additional engineering effort. We show such robotic bin tin-picking setup in Fig. 1, with an overhead, fix-mounted camera.
Our contributions are as follows: (i) we adapt existing work on training self-supervised pixel embed- dings [9, 10] to robotic grasping downstream tasks. (ii) We show that for robotic grasping tasks our approach is par with state-of-the-art [4] in terms of keypoint tracking accuracy while drastically simplifying the data collection. Finally, (iii) we show a real-world robotic bin-picking experiment where human preference on grasp configuration is encoded with dense visual descriptors, with the constraint of using a single, sred-mouuted camera.
2 Related work
In the following, we review recent work self-supervised dense visual descriptor learning for robotic manipulation in more detail. We also discuss related work on self-supervised representation learning and learning from set of single images, which are core concepts in work.
Dense visual descriptors in robotic manipulation. Inspired by [1], Florence et al. [2] proposed self-supervised training of dense visual descriptors by and for robotic manipulation. Their approach was later adopted to learn from multi-view correspondence in dynamic scenes with an application for policy learning [7]. In [5] the descriptor space representation was applied to learn challenging rope manipulation in simulations. Applying dense descriptors for state representation in model predictive control learning was shown in [6]. In [11] multi- consistency for keypoint learning was proposed with an application for state representation in reinforcement learning.
Another line of work investigated improved training strategies of dense visual descriptors. There are multiple contributions focusing on learning multi-object and multi-class descriptors [12, 13, 4]. The work in [3] exploits known object geometry to compute optimal descriptor embeddings. In [14] NeRF is utilized to generate dense correspondence datasets from RGB images. This alleviates problems with noisy depth data and proves especially helpful for thin and reflective objects.
There are several works that proposed to learn dense visual object representations directly from syn- thetic images composed of random backgrounds and randomly sampled, masked objects distributed over the image, see [2, 15, 12]. Learning from such synthetic images can be more efficient due to higher object density and ground truth correspondence, however, they rely on labeled datasets with object masks. Masking is either achieved by 3D reconstruction with robot wrist-mounted cam- era 12, 15], or labelled RGBD dataset [12]. As opposed to image composition via datasets, the image augmentation technique, as in this paper, only requires an unordered, unlabelled RGB dataset. This significantly simplifies data collection and opens up the possibilities to learn dense visual descriptors where no 3D reconstruction is possible, or where object masks are not available.
Self-supervised descriptor learning from RGB images. An intuitive way generate geometric correspondence is to estimate the optical flow of subsequent frames from a video. In [16] this technique was adapted to train DONs using contrastive learning. [8] proposed use optical flow from videos, or image augmentations to embed pixels of objects in view-invariant coordinate frames. This method was adopted in [9] for pretraining of geometry oriented tasks, such as object specific part detection in images. The work in [17] learns pixel-wise descriptors from single images with augmentations by using hierarchical visual grouping of image patches based on contour. Our work follows the image augmentation technique of single RGB images to generate alternative views and synthetic correspondence. Equivariant network architectures (e.g., [18, 19I) could replace certain augmentations (e.g. rotation) during training and improve sample efficiency. In our setup we can
2

---

apply affine transformation on training data and we rely on vanilla ResNet architecture for our experiments, which achieves good S2( equivariance, as shown in our experiments.
Self-supervised visual representations learning. Instead of training on large supervised datasets, self-supervised methods have become a popular way obtain visual representations, which can be fine-tuned to specific downstream tasks. A recent and very successful approach using contrastive learning is SimCLR [10]. It aims maximize agreement between two augmented versions of the same image, while considering all other images in the batch as negative samples. BYOL (Boot- strap your own latent) [20]. in comparison to contrastive methods, does not rely on the sampling of negatives. Barlow Twins [21] also forgo negative samples by optimizing the cross-cove cros-coorrllttion matrix between embeddings from two augmented versions of the same image be close to identity. Our approach is most similar to SimCLR as we employ the same loss formulation, but with the im- portant difference that our batch is constituted by individual pixel descriptors instead of full image embeddings.
3 Method
In this section we discuss our proposed training approach using image augmentations. We first give overview of the whole training pipeline, then discuss image augmentation techniques and finally present the loss formulation and dataset requirements. For an illustration of the training pipeline we refer to Fig. 2.
1/
Robot cell with camera
8())
((':0)
Synthetic correspondence
Shared weights
Contrastive loss
Backprop.
())
I" ~ 9()
Figure 2: Illustration of the training pipeline. For every RGB image a batch, we sample a pair of augmented images I' and I" while keeping track of the pixel correspondences. We evaluate the RGB images with the trained fully-convolutional network f(; 0) with trainable parameters 8. Finally, compute the contrastive loss and backpropagate the error using the embedded images I' and 1 along with the correspondence information.
3.1 Dense Descriptor Training with Synthetic Correspondence
Inspired by the work of [9] we rely on training on an unordered set of images and use image augmen- tations to arrive at alternative views of each image. First, we sample minibatch of N RGB images from the training data set consisting of independent RGB images. For every image in the minibatch sample two augmented views I' ~ g(1) and I" g(I) by applying randomized augmentations g: H H x3 + H XXx x3 (described more detail below). A learned fully-convolutionaal network [22] model ((:0), f IR Hx x3 IR HxV D maps the augmented images I' and I" to their de- scriptor space embeddings I' and Iz The user defined parameter E N+ controls the resolution of the descriptor space.
By keeping track of the position of each pixel in the original image during the augmentations, we sample pairs of pixel locations between I' and /" that share the same position in 7. We refer to these as synthetic correspondences, emphasizing the use of synthetic image augmentations as opposed to geometric correspondences coming from the 3D geometry of multiple camera views, as

---

(a) Original
(b) Affine
(c) Perspective
(d) Resi
(c) Color Jitter
(f) Combined
Figure 3: Visualization of the augmentations utilized for synthetic view training. Figure 3f shows the combination of all augmentations as used in practice for training a synthetic view deseriptor model. While each augmentation is guaranteed to be applied, the individual parameters, e.g., scale of distortion, angle of rotation, crop size and location, etc. are still randomly selected each iteration.
in [1, 2]. The descriptor values at the sampled pixel correspondence locations serve as positive pairs for contrastive learning.
3.2 Image Augmentations
For robotic applications we require descriptors that are invariant to translations, rotations and per- spective changes of the objects, as well as changes of lighting conditions. In vanilla DONs training, cf. [2], this is achieved by recording a diverse training set of registered RGBD image sequences, which contain sufficient variance in camera and object poses, and to some extent in lighting con- ditions. In our work, we achieve a similar effect purely by imposing data augmentations on single RGB images from an unordered set.
We carefully select augmentations, which reflect the desired invariance properties stated above, as follows: affine transformation induce rotations and scale changes (zoom out), perspective distortions mimic view changes, resize&crop implies scale changes (zoom in), and lastly, color jitter affects brightness, contrast, hue, and saturation of the image, hence the lighting conditions of the scene. We show an illustration of these augmentations in Fig. 3. We utilize torchvision library [23] of Pytorch reference implementations.
The positive impact of image augmentations on the training dense visual descriptors was already demonstrated in [4] for datasets utilizing geometric correspondences. In work, each augmen- tation is not only helpful, but relevant to the ability of the model to successfully learn an invariant descriptor space. See Appendix F.1 for an ablation the respective impact of each augmentation on the overall performance.
3.3 Loss Function
Following [10] we adopt the NT-Xeet loss, for learning the dense descriptor representations. For pair of corresponding descriptors {d,, d, obtained from images I' and 1" in a minibatch of size M. we compare their distance to the distance of d, to all other sampled descriptors in the given minibatch arriving at the following individual loss term:
exp(dist (di, d,) /T)
bij = - -log
(1)
Σ
2-2M
Loyy exp(dist (di, dx. /T)
with temperature parameter which we fix to 0.07 throughout this paper following [4]. We choose the metric dist (-) to be the cosine similarity between descriptors d, and dy. Due to the cosine similarity normalize the descriptors d. The total loss is given the mean over all individual loss terms, cf. [10].
Note that this loss, together with our correspondence sampling, does not distinguish background from objects, nor does it explicitly address multiple object classes and instances, as opposed [2]. Instead, we follow [4] and sample correspondences uniformly in image plane assuming that every pixel is unique. This method tailored to datasets depicting densely packed scenes with single ob- ject instances, for example, a heap of objects in a bin-picking scenario. The learned descriptor space does not imply semantie information on object classes or background, but still provides consistent keypoint detection and robust tracking performance, which is essential for downstream tasks.

---

Comparison of Training with Synthetic and Geometric Correspondence
In this section show in-depth comparison between training with geometric and synthetic correspondence. We also investigate the invariance of descriptors obtained from a network trained with synthetic correspondence with respect to object-camera relative transformations.
Similar to [2] we use a pretrained ResNet-34 with 8-stride output for all experiments. The upsampled output matches the resolution of the input. In Appendix B we give more details on the baseline training method using geometric correspondence.
We recorded a dataset of registered RGBD sequences. Despite the availability for registered image pairs, the synthetic training only uses single RGB images for both training and validation. However, the camera poses help with the evaluation as they allow us to generate ground truth pixel matches across any two images of the same static scene without the need for manual labeling. The dataset consists of eight scenes with various object configurations. Every scene contains only one instance per object. The scenes are recorded with a robot wrist-mounted camera while the robot arm follows predefined trajectory keeping the camera view on the objects.
Both approaches are evaluated the same ground-truth image pairs and correspondences. For bustness, we perform a k-fold cross-validation, that is, each scene from our total dataset was once used
One scene is chosen as validation set, with the remaining six scenes used for training- We report the averaged results. We use the same loss function, training parameters and augmenta- tions for both approaches, with the exception that augmentations are chosen with 50% probability for the geometric training, as it yields better results. For synthetic training, each augmentation is ap- plied. An ablation study of using different augmentation probabilities is given in Appendix F.2. For brevity, we report the median and 75% quantile pixel errors. Extended results, including the mean, 90%, and 95% quantile pixel errors, as well as the percentage-or wwaggg--oooorreeee keypoints @k (PCK @k) for k E {3, 5, 10, 25, 50}, are reported in the respective Appendix sections.
4.1 Keypoint Tracking Performance
Given a common dataset, measuring keypoint tracking wery descriptor da in image B and record the pixel eπтоr || - 122 The network parameters 0 are either obtained by geometric [4] or synthetic correspondence training proposed Sec. 3.1.
Fig. 4a shows the distribution of pixel errors for the two training approaches. The median of the corresponding distribution is highlighted dashed line. With difference of only 1.9px in median pixel error, the synthetic correspondence performs competitively to the geometric training. The percentage of pixel errors that larger than 50 pixels 10.1% and 5.4% respectively, as indicated in the bottom right part of Fig. 4a.
Results.
0.200
greometric
geametric, median
0.175
median: 3.2
synthetic
Pixel [pixe]]
355
geometric, 75% quantile
synthetic, median
Donsity [a.u.]
0.150
median: 5.1 px
30
synthetic,75% quantile
0.125
25
0.100
20:
0.075
0.050
5.4%
50
10-
0.025
10.1%
50 px
10
20
30
40
50
13
32
64
256
Pixel Error [pixel]
Descriptor Dimensions
(a)
Figure 4: (a) Pixel emror distributions obtained using synthetic (orange) and geometric (blue) cor- respondence training. (b) Median and 75% quantile of the pixel error distribution for both training approaches and different descriptor dimensions.
5

---

In Fig. 4b we compare the median and the 75% quantile of the two training methods with respect to the descriptor dimensions D. These results were obtained without k-fold cross-validation. For both training approaches the median pixel error decreases for larger descriptor dimensions. For dimensions larger than the median pixel error decreases only marginally. In contrast, the 75% quantile error still increases further until D 64, before improvements saturate. For additional insights into these results we refer to the supplementary material.
4.2 Invariance Tests
In the following. we discuss the question: how well does training with image augmentations pro- posed in Sec. 3.2 generalize to physical camera transformations? We recorded different test scenes with a wrist-mounted camera and the following specific camera movements: (i) changing camera perspective (camera tilting), (ii) translation along the camera z-axis (zooming in and out), and (iii) camera rotation along the camera z-axis (see Appendix D.2 for details). We compare the perfor- mance of a network trained with synthetic and one with geometric correspondence both trained on same data as described in Sec. 4.1. We use descriptor dimensions and affine, perspective and resize&crop augmentations. As in the previous section we compute the 75% quantile pixel error and use 1 1000 keypoints per image pair, fix the base image A and only vary image B, which shows the changing camera views. The results for three different types of transformations are compiled in Fig.
It can be seen that training with synthetic correspondence generalizes well to large range of camera transformations, especially to those parallel to the object plane (Fig. 5a, Fig. 5b). These physical camera transformations are very similar to the affine and resize&crop augmentations used during training. For generalizing the perspective transformations with angles above 45 degrees, as shown in Fig. 5c, the synthetic correspondence training shows a clear deterioration in performance. As the perspective changes, occluded parts become visible and vice versa. This physical transfor- mation is not well captured by the synthetic augmentations for larger angles.
Camera Rotation (z-axis)
Camera Moving In/Out (2-axis)
Camera Perspective
60
75% quant. p. 8. [px]
geomerric
75% quant. p.o. [px]
geomeiric
75% quant. D.0. [px]
geomelric
50
synthetic
500
syuthetic
60
synthetic
40
40
30-
30
40
20
20
10-
10
50
100
150
200
-10
10
20
40
60
Rotation Angle [*]
Translation [cm]
Rotation Angle [*]
(a)
(c)
Figure 5: 75% quantile pixel error between two different images for the geometric (blue) and syn- thetic (orange) correspondence training approach for three different isolated camera movements between those images in varying magnitude: (a) camera rotates around z-axis, (b) camera moving closer and further away from the objects, (c) camera moving on a sphere in x-direction around the objects, facing the objects. Note the scale of the y-axes.
5 Grasping Experiment with Fix-mounted Camera
We demonstrate robotic bin din-picking experiment that relies dense visual descriptors for defining grasp preferences. We use a 7-DoF Franka Emika Panda arm with a suction gripper mounted on the end-effector, see Fig. 1. Our setup uses fix-mounted Zivid One+ camera above the bin in robotic cell. Training a descriptor network using this setup prevents the use of geometric correspon- dences. Instead, show that proposed method can be trained setup it is often present in real world applications and prove that the keypoint tracking accuracy good enough for guiding generic grasping method by human annotated grasp preferences.
In the experiment, we consider picking ten different types of objects from one bin. However, using suction gripper the objects are difficult to grasp certain locations: some have cutouts the

---

packaging, transparent or foldable parts, and uneven surfaces (see Appendix G for higher resolution images). In our experiments, a purely model-free grasp pose generator often predicts poses at these locations, ultimately reducing picking performance.
YueR
00
00
(a)
(c)
(d)
Figure 6: Illustration of the grasping experiment. (a) The RGB images with the predicted grasp
poses. (b) The graspable areas generated using RGB and depth images. (c) The predicted keypoint
heatmaps on graspable areas. (d) The predicted grasp poses overlayed on the heatmaps.
We want utilize human domain knowledge aid grasp pose selection using dense visual de- scriptors to increase grasp success rates for these objects. To do so, we show a small number of RGB images depicting the objects in different configurations in the bin and ask the human to click on pixel locations corresponding to preferred grasp location. We track these descriptors to gen- erate preference heatmap, as shown in Fig. 6c. The heatmap, in contrast to discrete single-pixel correspondences, enables intuitive consideration of matching uncertainty in the form of distance in descriptor space. Furthermore, it provides a more flexible and quantifiable basis for combination with grasp detectors.
To generate the final set of grasp pose candidates intersect the grasp preference heatmap with the detected graspable areas identified by the model-free grasp detector (Fig. 6b). While we make no specific assumption regarding which grasp detection method to use, we employ dense pixel-wise graspability estimation in this experiment. is based on a fully convolutional neural network with RGB-D input, specifically UNet [24] trained on annotated pixel-wise labels of expected graspability for a wide range of different bin picking scenes, For the example image, the resulting poses are shown in Fig. 6a and Fig. 6d.
Note that the descriptor network is trained purely from the set of RGB images showing the bin with objects in random configurations, recorded with the overhead camera. See Sec. G for more details on the experimental setup and heatmap generation.
5.1 Quantitative Evaluation
We evaluate the benefit of the proposed method for robotic bin picking based set of sixteen manually annotated scenes. The scene images include the ten known objects, but in different con- figurations compared to the training set. For each evaluation image, manually annotated object instances, pixel-wise graspable areas, and those areas that correspond to selected descriptors. In total, the evaluation dataset contains 131 graspable objects of which 103 have visible descriptors.
We compare our descriptor- based grasping approach with a purely model-free approach that directly uses the graspable areas without accounting for descriptors. With this study, we investigate two questions. First, how effective our approach to re-identify descriptor spots compared to grasping at these spots by chance? And second, is there a considerable negative effect on the amount of objects that can be grasped, e.g., due to non-detected grasp pose annotations?
Table shows summary of the results. Success Rate denotes the number of successful grasps compared to all grasps attempted. Descriptor Success denotes the number grasps at those spots marked as desired grasping points compared to all grasps attempted. Object Hits denotes the per- centage of all graspable objects for which at least one feasible grasp pose has been found irrespective of descriptors, including objects without visible descriptor spots. Descriptor Hits denotes the per-
7

---

centage of objects for which a grasping pose has been found at the respective descriptor spot, only including objects with visible descriptors. We consider descriptor spots with tolerance of around lcm, which corresponds to the radius of the suction gripper.
Table Evaluation results on individual, annotated scene images similar Fig. 6.
Success Rate Descriptor Success
Object Hits Descriptor Hits
Preference (ours) Baseline
98.9%
91.1%
63.44%
78.6%
79.9%
50.4%
78.6%
68.0%
can be concluded from Table 1 that using the proposed method to encode grasp preferences is effective as it significantly increases the amount of grasps at desired spots from 50.4% to 91.1%. Due to the challenging object geometries, this helps to raise the overall grasp success rate from 79.9% to 98.9%. The expected downside, however, is that the proposed method finds grasp poses only for a smaller amount of objects, 63.4% instead of 78.6%. Still, this includes objects that have no desired grasping spot visible. In some applications it can be the desired behavior to not propose a grasp pose for objects if they cannot be grasped at the preferred location. When only considering which preferred grasping spots have been covered by grasp poses, our method manages to find grasps for 78.6% instead of 68.0% of the visible spots. In consequence, we conclude that there is only a moderate negative effect due to limiting grasping to descriptor spots which can indeed be beneficial for some applications.
6 Limitations
Variance in camera poses. As seen in Fig. 5, the synthetic training ensures that descriptors are sta- ble within limited margin of camera transformations relative to the object. For example, in Fig. Sc, for viewing angles steeper than 45", accuracy deteriorates. This imposes a limit on descriptor con- sistency across images with large differences in object poses.
Changing Environment. A change in environment (background, lighting) between training and inference time may have negative influence keypoint tracking performance. It is expected that augmentations such as color jitter and, masks are available, background randomization can miti- gate these effects. We will investigate these aspects in future research.
Generalization to unseen objects. Although [2] demonstrated capabilities to generalize intra-class instances, our work focuses on instance specific descriptors. Given our outlined training setup, works best on known objects from the training set.
Object Edges. We observe worse descriptor consistency across images for keypoints located the edges of objects or close to parts that are occluded by other objects. In setups where object masks are present background randomization could reduce this effect.
7 Conclusion
In this paper we proposed novel training method for learning dense visual descriptors based on image augmentations for robotic manipulation. The evaluation shows that overall our proposed method is competitive to the existing geometric training approach. For physical transformations like changing the camera perspective on the scene, which are harder to mimic by augmentations, the training method using geometric correspondences shows superior performance. Being aware of these limitations our proposed method is expected to perform well for setups where objects are mostly altered by translations, rotations parallel to the camera plane, often the case for random heaps of objects a bin, our method is especially suitable for such setups that are constrained by fix-mounted camera. Finally, demonstrated the of our method in realistic grasping experiment to increase grasp success rates by human annotated grasp preferences.
slightly tilted. As this is
8

---

References
[1] T. Schmidt, R. Newcombe, and D. Fox. Self-Supervised Visual Descriptor Learning for Dense Correspondence. IEEE Robotics and Automation Letters, 2(2):420 427, 2017.
[2] Florence, L.. Manuelli, and R. Tedrake. Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation. Conference Robot Learning, 2018.
[3] A. G. Kupesik, M. Spies, A. Klein, M. Todescato, N. Waniek, P. Schillinger, and M. Bürger. Supervised training of dense object nets using optimal descriptors for industrial robotic appli- cations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1993-6100, 2021.
[4] D. B. Adrian, A. G. Kupesik, M. Spies, and H. Neumann. Efficent and robust training of dense object nets for multi-object robot manipulation. In JEEE International Conference on Robotics and Automation (ICRA), 2022.
[5] Sundaresan, J. Grannen, B. Thananjeyan, A. Balakrishna, M. Laskey, K. Stone, J. E. Gon- zalez, and K. Goldberg. Learning rope manipulation policies using dense object descriptors trained on synthetic depth data. IEEE International Conference on Robotics and Automation, pages 9411 1 1-9418, 2020.
[6] L. Manuelli, Y. Li, Florence, and R. Tedrake. Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learning. Conference on Robot Learning, 2020.
[7] P. Florence, L. Manuelli, and R. Tedrake. Self-Supervised Correspondence in Visuomotor Policy Learning. IEEE Robotics and Automation Letters, 5(2))492-499, 2020.
[8] J. Thewlis, H. Bilen, and A. Vedaldi. Unsupervised learning of object frames by dense equiv- ariant image labelling. Advances in Neural Information Processing Systems, pages 845-856, 2017.
[9] D. Novotny, S. Albanie, D. Larlus, and A. Vedaldi. Self-Supervised Learning of Geometri- cally Stable Features Through Probabilistic Introspection. Proceedings of the IEEE Computer Society Conference Computer Vision and Pattern Recognition, pages 3637-3645, 2018.
[10] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[11] M. Vecerik, J.-B. Regli, O. Sushkov, D. Barker, R. Pevceviciute, T. Rothörl, C. Schuster, R. Hadsell, L. Agapito, and J. Scholz. S3K: Self-Supervised Semantic Keypoints for Robotic Manipulation via Multi-View Consistency. Conference on Robot Learning, 2020.
[12] S. Yang. W. Zhang, R. Song. J. Cheng, and Y. Li. Learning multi-object dense descriptor for autonomous goal-conditioned grasping. IEEE Robotics and Automation Letters, 6(2):4109- 4116, 2021.
[13] D. Hadjivelichkov and D. Kanoulas. Fully Self-Supervised Class Awareness in Dense Object Descriptors. Conference on Robot Learning, pages 1-10, 2021.
[14] L. Yen-Chen, P. Florence, T. Barron, T.-Y. Lin, A. Rodriguez, and P. Isola. Nerf-supervision: Learning dense object descriptors from neural radiance fields. IEEE International Conference on Robotics and Automation, 2022.
[15] C.-Y. Chai, K.-F. Hsu, and S.-L. Tsao. Multi-step pick-and -llcee tasks using object-centric dense correspondences. In IEEE/RS/ International Conference Intelligent Robots and Sys- tems (IROS), pages 4004-4011, 2019. doi: 100110900/0000897.20000000 2019. 8968294.
[16] U. Deekshith, N. Gajjar, M. Schwarz, and S. Behnke. Visual descriptor learning from monocu- lar video. VISIGRAPP 2020- Proceedings of the 15th International Joint Conference on Com- puter Vision, Imaging and Computer Graphics Theory and Applications, 5:444-451, 2020.

---

[17] X. Zhang and M. Maire. Self-supervised visual representation learning from hierarchical grouping. Advances in Neural Information Processing Systems, (NeurIPS):1-12, 2020.
[18] T. Cohen and M. Welling. Group equivariant convolutional networks. 2016. doi: 10.48550/ ARXIV.160200757. URL https:///aaaat wwtp://////////////eeeae 016602007766
[19] D. Wang, R. Walters, X. Zhu, and R. Platt. Equivariant q learning in spatial action spaces, 2021. URL https: htthtts:://aar //arxiv....i///// wwtps//i.orr//eoooooee
[20] J. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. Á. Pires, Z. Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. Munos, and M. Valko. Bootstrap your own latent A new approach self-supervised learning. Advances in Neural Information Processing Systems, 2020.
[21] J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny. Barlow twins: Self-supervised learning via redundancy reduction. Proceedings of the 38th International Conference on Machine Learning, volume 139, pages 12310- 22310-12320. PMLR, 2021.
[22] E. S.elhhhmerr J. Long, and T. Darrell. Fully convolutional networks for semantic segmentatio....... IEEE Trans. Pattern Anal. Mach. Intell., 39(4))640-65) 651, 2017.
[23] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury. G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. Veto,,, M. Raison, A. Te- jani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, pages 8025-8035, 2019.
[24] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical im- age segmentation. In International Conference on Medical image computing and computer- assisted intervention, pages 234-241. Springer, 2015.
[25] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[26] S. Cho, S. Hong. S. Jeon, Y. Lee, K. Sohn, and S. Kim. Cats: Cost aggregation transformers for visual correspondence. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, I, NeurlPS 2021, December 14, 2021, virtual, pages 901 1-9023, 2021.
10


Reviews:
[{"review_id": "t19OnzNKJiO", "reviewer": "Reviewer_Wx5g", "summary": "The paper proposes self-supervised learning of dense visual descriptors from unordered RGB images using data augmentation and contrastive losses. This is unlike prior works that require obtaining pixel-level correspondences from camera poses. Training works by sampling pairs of augmentations (affine, perspective, scale, etc) on the same input image, using a ResNet-34 to produce per-pixel descriptors, then using a contrastive loss to align descriptors of corresponding pixels. While this method loses the semantic and category-level information from priors works (e.g. Dense Object Nets), it is much simpler and offers competitive keypoint tracking performance. The authors use the trained descriptors in a downstream bin-picking task, where humans label suction grasp points that are then used to find new grasp points via descriptor matching. The proposed method achieves a 98.9% grasp success rate, higher than the 79.9% achieved by a baseline that does not use these pretrained descriptors.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThe paper is well-written. The method is simple to implement and the experiments are clear. I appreciate the authors’ ablations on the effects of different image augmentation types. Discussion of the failure cases on the model’s relatively poor performance on novel camera perspectives is also appreciated.\n\nWeaknesses\n\nMy main complaint about the work is on generalization of the trained descriptor networks on novel objects, and this manifests in a couple different ways. As the authors note, the proposed method works best on known objects. While this method would still be useful (since the network clearly generalizes in other ways, like clutter configuration, camera views, lighting, etc), additional experiments and/or discussions on object-wise generalization would be helpful. In particular, it would be interesting to do the keypoint tracking experiments on completely novel objects, just to see what those numbers are. \n\nFor the bin picking experiments, the authors describe the scene as consisting of “ten known objects, but are not contained in the training set.” I’m not sure what this means. One guess is that this means the 10 objects have human-labeled grasps, but they are not in the dataset used to train the descriptor network. If this is true, then does this mean the network does indeed have some generalization capabilities to novel objects?\n\nLastly, I think the paper would benefit from a quick comparison against training a descriptor network on a much larger but more generic dataset. This would be similar to other works that learn self-supervised image embeddings, except the output is per-pixel instead of per-image. This comparison can help further illustrate the extent of which domain-specific or even instance-specific training data is required.\n\nPerhaps for future work (this would be a stretch for rebuttal, but it’d be impressive), the authors could use an equivariant CNN architecture that is equivariant to translations and rotations for better sample complexity and avoiding certain image augmentation modes. Here’s an example of using equivariant networks in robot manipulation: https://proceedings.mlr.press/v164/wang22j.html\n\nOne small comment: Figure 4b should just have 2 colors and 2 line types, instead of 2 colors and 4 line types. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I recommend weak accept because while the proposed method could be useful, I can see how generalization may be a contentious point for other reviewers or the community at large.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please see strengths/weaknesses section.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "sQ3SDnzwn8u", "reviewer": "Reviewer_etoZ", "summary": "This paper proposes a data augmentation strategy of RGB images for self-supervised visual descriptor learning. Previous approaches require geometric correspondences derived from image registration via e.g., reconstructions of video sequences, to compute a contrastive (or variant of) loss during training. The presented approach simply augments a single RGB input image and learns an expressive pixel feature representation through the proposed \"synthetic correspondences\", i.e., same pixel after undergoing image augmentation. Experiments show that learning from synthetic correspondences achieves similar performance to learning from geometric correspondences in terms of keypoint matching. The method is used to demonstrate a robotic grasping task in which hand labelled grasp locations on objects can be identified in new scenes through pixel descriptor matching.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThe problem of learning visual descriptors is important and highly relevant for many applications such as robot manipulation as demonstrated by the authors. The authors have also covered the relevant literature sufficiently.\n\nThe technical approach is sound with a mostly good description of the approach (although there are some details missing as discussed later).\n\nThe experiments demonstrate that the proposed approach achieves good success in comparison to alternatives. They especially highlight the capability of good keypoint tracking without the requirement of generating geometric correspondences. Lastly, the grasping experiments make this paper highly relevant to the themes of CoRL.\n\n## Weaknesses\n\nThe main weakness is that the authors have not strongly asserted their technical innovation in light of the SotA. The use of augmentation has already been shown in various papers, e.g., [4, 9, 10]; furthermore, the importance of the different augmentations, namely the affine transformation, have been demonstrated to achieve more robustness. It is argued that other work, e.g. [14], learn image embeddings instead of pixel descriptors but I do not see this as a major technical contribution as it translates to an adjustment of the loss, which is ultimately borrowed from [10], as well as task and metric for evaluation.\n\nI find the grasping experiments poorly explained. What is the model-free grasp pose generation method? There is no explanation (not even in the supplementary) and I can only glean that it detects uniformly flat surfaces from Fig. 7b. Also, why does the final grasp need to be the intersection between the graspable areas and the grasp preference heat map? Who couldn't the raw user-selected pixels from one labelled image be used without having to first compute a heatmap? The original work in [2] shows this capability precisely. The descriptor success and hits metrics are not entirely fair for the baseline since it is completely agnostic to the descriptors. I am actually rather curious why the baseline achieves decent scores but it is hard to reflect on this without knowing more about the underlying method.\n\nI have a minor issue with the metrics used in the experiments due to a lack of clarity. Many of the results are only expressed in terms of the 75% quantile without justification of why this is the best metric. Why was the median dropped after results in Fig. 4? Furthermore, most related work report PCK, as such, I think the authors should argue why their choice of metric is more appropriate than what appears to be the standard.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While the paper presents an interesting and simple idea to learn visual descriptors, as well as demonstrate this for a robotic application, the technical contribution is relatively low as similar ideas have already been presented in previous work. The main difference appears to be the application of a pixel-wise contrastive loss from [10] for the setting proposed in [14] in order to learn pixel descriptors rather than image embeddings. This is only incremental in terms of contribution. Furthermore, it is difficult to assess the significance of the robotic grasping experiments without more information regarding the model-free grasping method (the performance comparison is unclear without a better understanding of the baseline). Lastly, the metrics used for evaluation are unjustified and differ from prior work. As such, I recommend a weak reject but am open to see a revision that potentially remedies these flaws.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "As discussed in \"weaknesses\", please clarify the main technical contribution in light of existing work on the same topic as well as provide much more detail about the robot grasping experiments. Other minor issues that should be addressed are as follows:\n\n* In Sec. 4 (lines 170-171), it is explained that for geometric correspondence training, the augmentations are chosen with 50% (instead of 100% for synthetic correspondence training). There is no explanation, conjecture or quantitative results to support the statement in the text or supplementary. It would be good to add an analysis not only of this aspect but that 100% augmentation is the best for learning from synthetic correspondences. Interestingly, [14] report that augmenting only one of the images leads to better performance, so it seems that the \"right\" amount of augmentation is inconclusive.\n\n* Sec. 4.2. could be made more clear with a diagram to explain the camera movements as it is not immediately clear from the text - we have to make a strong assumption about the camera coordinate system. The diagram can go in the supplementary and complement Fig. 8.\n\n* The work does not explicitly address generalisation as the dataset considers the same objects in both testing and training. I was surprised that the robot manipulation experiments include unseen objects in the evaluation and that the descriptor matching appears good (in terms of descriptor success). Could the authors comment on why this is the case? Are the test objects similar to those seen during training?"}}, {"review_id": "nNZB4Wgw1Di", "reviewer": "Reviewer_M14W", "summary": "The paper presents a training data augmentation technique to learn local keypoint descriptors which are invariant to certain 2D image transformations. The required setup is simpler than that based on 3D geometric transformations. The authors also present analyses that show performance variations with respect to different 2D transformations and the extent of it. Real world robot experiments are also presented.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe approach section builds on existing works. The key contribution is in results and analyses which shows how 2D geometric transformations of RGB images can lead to competitive keypoint matching and tracking performance as compared to 3D geometric transformation based training. This is relevant to the field as the approach is simple and leverages the benefits of machine learning where one can use more training data (in a somewhat systematic manner) to achieve certain invariances.\nIt would have been amazing though to be able to combine the two compares techniques to achieve state-of-the-art performance.\n\nLimitations section is clear and will be helpful in further advances.\n\nSection 4.2 and 4.3 are insightful and clearly show the characteristics of the proposed approach.\n\nReal robot experiments are a plus.\n\n**Weaknesses**\n\nIn L114, the authors refer to “synthetic correspondences” in contrast with geometric correspondences. The operations they are performing for synthetic correspondences (described in L125-130) are also geometric as they transform the underlying 2D image pixel grid. The authors need to improve the description of their approach.\n\nL113-114 need clarification: “origin of pixels” and “same origin”\n\nL182 - “unprojects”?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper clearly conveys its approach and results, which can act as an alternative or complementary way of achieving better results. The novelty lies in results and analyses, not in the approach section, and could still be relevant to the community. \n\nPost-rebuttal:\nThe authors have done an excellent job in addressing reviewers' concerns. Additional experiments and ablation studies conducted by the authors during the rebuttal are appreciated.\n\nAn important point was novelty to which the authors agree that their pipeline uses existing works but the paper mainly argues for simplification of data generation pipeline for training and thus improving performance in the robotics context. As an application, this work seems relevant for the community and is a good validation of 'what one thinks should work does actually work'.\n\nThe overall idea and the results are not surprisingly amazing although the execution is great, thus an improved rating.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "There are no major issues. The presented analysis adds value to the literature. For being more impactful, the paper would need to show how it can combine both 2D and 3D geometric transformation based supervision to achieve state-of-the-art performance."}}, {"review_id": "HGGnTyCJ9rD", "reviewer": "Reviewer_aGSf", "summary": "The authors propose  a way for dense correspondence descriptor learning that can learn these descriptors from an unordered set of RGB images which can come from a single stationary camera.  They achieve this by transforming the input image with various data augmentations, keeping track of the ground truth correspondences. They then study the performance of this training regime compared to more standard dense descriptor learning that requires self-supervised data gathering with  registered cameras from different viewpoints.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Summary of Strengths**\n- Fair Literature review.\n- Approach is clear and well presented.\n- Good experimental procedures and analysis.\n\n**Summary of Weaknesses**\n- Novelty in the approach somewhat limited.\n- Analysis suffers from having a single baseline.\n- It is unclear why the grasping experiment is necessary.\n\n**Comments**\n- The idea of using synthetic data to train dense descriptors does not seem new (for example Learning Multi-object dense descriptor for autonomous goal-conditioned grasping by Yang et al.), nor does the specific loss used  (coming from Chen et al. A simple framework for contrastive learning of visual representations ). Moreover, the synthetic training from Yang et al. seems like it would provide the same benefits over vanilla dense object nets as the proposed approach, and the proposed approach generally does not perform better than the geometric baseline.It seems to be this work would benefit from expanding on the analysis side. That is, benchmarking more approaches  that are interesting and appear in different previous works, with synthetic or geometric training, and analyse the pros and cons of each approach to build a more wholistic study. This would then result in a very interesting analysis paper, as opposed to proposing a specific approach.\n\n- Using dense descriptor representations for finding human-defined grasp locations in different scene configurations has been shown to work well in the papers from the related work. Showing that using dense descriptors to guide a model-free grasping method seems to have an incremental benefit over this, which distracts from what otherwise seems to be the main point of the paper which is a comparison between the ``standard’’, geometric , way of training dense correspondence descriptors and the synthetic way of training them.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper has limited novelty in the method but seems well executed and presented. I believe a much stronger paper could arise from it by further developing the analysis side,  by comparing over multiple baselines and approaches, rather than favouring any particular method as the \"proposed approach\". That is casting the narrative as \"providing insights and finding the pros and cons of different approaches for learning dense correspondence descriptors\", and comparing various interesting approaches (including those already analysed) to provide those insights. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "\n- Perhaps the authors can better justify why the specific approach/design choices of synthetic training they used in that paper are interesting in their own right, compared to the other approaches using synthetic training for dense correspondences (for example in Learning Multi-object dense descriptor for autonomous goal-conditioned grasping by Yang et al.)?\n- Similarly perhaps the authors could expand on what additional insights do the particular robotics experiments conducted in this paper bring, given that previous works have already demonstrated multiple times that dense descriptors are well suited for finding picking locations?\n- line 79 - Title of this section seems a bit odd  with the phrase that follows: If you use a video and optical flow between subsequent images, then it is not descriptors from ``single images’’?\n- line 132 - It is unclear what ``geometric datasets’’ refers to?\n- Fig. 5 : median is more intuitive, if 75% quantile is needed instead, then it would be good to mention why?\n- In section 4.2 it is unclear whether the same datasets where used as in section  4.1?\n- In related work it would be nice to situate more explicitly this work with the previous works in the field, where does it draw inspiration and how, and where is it different. This information is currently scattered across the paper, it would be nice it if was made clearer in the related work.\n- The loss in eq. (1) is quite central,  could be nicer to have a more complete explanation of it rather that simply refer to previous work : What is M and tau in the loss? What is the intuition behind it?\n- Line 154 is a bit unclear:  The phrase as written reads as if the cosine similarity loss was doing the normalisation of the vectors. I believe this is not what was the intended meaning, descriptors can be normalised because the cosine similarity doesn’t impose anything  on the norm of the descriptors, but the normalisation is done independently from the cosine similarity?\n- line 170 : It is not clear what the augmentations are: are those the same augmentations as with the synthetic correspondences?\n- Are the training and test scenes constrained to have the same objects present? This should be made clearer."}}]

Author rebuttal:

Decision: Accept (poster)