Fully Self-Supervised Class Awareness in Dense Object Descriptors
Authors: Denis Hadjivelichkov
Dimitrios Kanoulas
Venue: CoRL2021 Poster (2021)
Keywords: Self-Supervision
Descriptor Learning
Object Correspondence

Abstract: We address the problem of inferring self-supervised dense semantic correspondences between objects
in multi-object scenes. The method introduces learning of class-aware dense object descriptors by providing either unsupervised discrete labels or confidence in object similarities. We quantitatively and qualitatively show that the introduced method outperforms previous techniques with more robust pixel-to-pixel matches. An example robotic application is also shown~- grasping of objects in clutter based on corresponding points. 

Fully Self-Supervised Class Awareness in Dense Object Descriptors
Denis Hadjivelichkov Department of Computer Science University College London
Dimitrios Kanoulas Department Computer Science
University College London
United Kingdom dennis. hemiis.hafjiilllicccooohhhh ac.ul. .k
United Kingdom d. kanoulas@ucl. ac. uk
Abstract: We address the problem of inferring self-supervised dense seman- tie correspondences between objects in multi-object scenes. The method intro- duces learning of class-aware dense object descriptors by providing either unsu- pervised discrete labels confidence in object similarities. We quantitatively and qualitatively show that the introduced method outperforms previous techniques with more robust pixel-to-pixel matches. An example robotic application also shown grasping of objects in clutter based on corresponding points.
Keywords: Self-Supervision, Descriptor Learning, Object Correspondence
Introduction
self-supervised methods allow models to train with no manual labelling, thus improving the ability to scale and learn from more data. In robotic applications, they could potentially allow robot to learn continuously 'in the wild'. Robots of the future would henefit from the ability to learn how to interact with many objects, including unseen novel ones. One particular problem is finding accurate pixel correspondences between similar objects in such a way that a robot agent recognises the corresponding parts between items and manipulates them in similar ways.
1. Source point
2. Query Cou eopendddeees
Grasp Correspondett Point
Figure 1: Correspondence (red circle) produced by our DON+Soft method used for robot grasping.
A recent work producing self-supervised object descriptors, namely Dense Object Nets [1] (DONs), has been applied as a representation learner visuomotor policies [2, 3] improving onto previous end-to-end methods [4]. While DONs produce excellent point correspondences between similar ob- jects, without access to ground truth classes they fail to find robust matches in multi-object scenes and produce false positives. In this work, we propose method for class-aware Dense Object Net- work and show that it leads to more informative descriptors that are better suited for correspondence generation in multi-object scenes. We present two variants exploring different ways of representing the object classes: "hard" discrete labels and "soft" labels, reflecting our confidence the similarity between the objects. We demonstrate that our method is better than the state-of state-of-the-ar -ft in finding correspondences in multi-object images, while simultaneously improving performance of on single- object images. A real example application shown using mobile manipulator This method could enable future work on more complex tasks such as unsupervised learning of category-level affordances and manipulation of objects in clutter.
See https://sites ottpa///eoooaaa/////aaaaaaaaaaaa
for video.
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

2
Related Work
In this section we briefly overview recent advances in descriptor learning. We then review meth- ods for unsupervised classification and class disentanglement. Our work focuses on self-supervised learning of descriptor representations, such that the descriptors of similar points belonging to similar object are consistent across time, view, and object instance. Our method builds upon Dense Object Nets (DONs) [1] which makes use of 2D pixel to 3D point projections as supervisory signal. The method is described in more detail in Sec. 3. Most other works on descriptor learning such as [5, 6. 7] learning requiring large amounts of human-annotated data and do not make efficient use of depth data, which robots usually have access to. Several methods re- lieve the need for annotations by using 3D CAD models of the observed objects [8. 9], however this is not applicable when encountering novel objects. Many works make use of spatio-temporal con- sistency across video frames, using tracking trajectories or multiple modalities: [10] learns unique descriptors via optical flow contrasive learning from dynamic data. Grasp2 Vec [11] learn similar features through autonomous robot intereaction. The semi-supervised S3K [12] uses multi-view consistency to produce keypoints with high accuracy. With only few ground truth examples, it is learns 3D keypoints by estimating the projections of 2D keypoints. It proves to be useful for appli- cations such cable plug-in, where keypoints do not have to be bound object, as long as they are consistent across views. However, it does not produce dense correspon- dences. We are not aware of any work that produces pixel-pixel correspondences across between multi-object scenes, based on unsupervised or self-supervised training without ground truth models or per-image labels, which is introduced in here.
make use of fully- or weakly-supervisee
physical location on the
For our method to work, we require a way of assessing the similarity or dissimilarity between ob- jects and disentangling their representations. This can be done by assigning some discrete class labels to each object or measuring our confidence in the similarity of object pairs. Most recent works unsupervised clustering techniques such as mean-shift and K-Means in combination with embeddings [13, 14, 15]. Patten et al. [16] improve on previous approaches by considering geomet- ric similarity and experience queries. Object-Contrastiv Network [17] finds object candidates via Faster-RCNN, embeds them and then samples triplets based on nearest neighbours in embedding space. Recently, Tan et al. [18] showed that clustering of object embeddings alone is not enough. Instead, they propose method for similarity measurement and triplet sampling via random walker graphs. Their method demonstrates state-of-the-ar results on various benchmarks. We use it as an inspiration to our method and provide further details in Sec. 4.2.1.
3 Dense Object Nets
Dense Object Nets [1] which are largely based on Schmidt et al. [19] are the foundation of our method. In this section we explain how they work and identify their current limitations. DONs learn dense descriptor representation of RGB inputs that is applicable to both rigid and non-rigid objects. It uses the strong prior of depth information taken from an RGB-D camera, and knowledge of robot manipulator's pose to project 2D image pixels onto a reconstructed 3D model, thus learn- ing correspondences between different views of the same 3D points. In particular, this is done by having a robot point a camera towards an object and move around to capture it from different views. In Siamese fashion, the model is fed pairs of augmented images from which pixels are randomly sampled. Augmentations consist of random background, color jitter, crops, and scale. Pixels corre- sponding to the same 3D point are considered 'matches' while any other random pair of pixels are considered 'non- mon-metthsss'. The loss is the sum of match and non-match losses defined as follows:
L Lmmacc-
EIId Id. db113
N_
I Luun-match
Σ ma, M - -Ilda - db 1
(2)
Nm Nm
where d, and dp are the output descriptors at the given pixel pair and N_. is the total number of match pairs. Similarly, N_ is the total number of mon-matches. M is margin hyperparameter determining the minimum distance between mon-match pairs if a non-match pair closer than M, the loss is
increased. Nn is the number of these matches that is below the margin. While Lmatch encourages

---

true pairs to be similar, Lnon-mun match ensures the pixels don't it all converge to the same descriptor. Since the robot has no knowledge of the object's class, the learned descriptors tend share the same descriptor space for all objects. While the original work shows that when the ground truth class is known, the descriptor subsets can be distinetly separated, does not propose how to implement this and remove the use of ground truth.
4 Method
The method presented in this section tries to solve the problem of disentangling the descriptor spaces shared by DON descriptors by including class awareness.
4.1
Problem Setup
Data Organization: each data-point consist of an RGB, Depth, and Mask image and camera pose. Sequences are comprised of data-points, while the dataset is composed of sequences. A 3D TSDF reconstruction is auto- matically generated by combining the RGB-D views of sequence during data collection.
Figure 2:
Pose
Sx). The inputs used for training are set of n sequences S = St. where each sequence The data 0 for timepoint and
contains timed data for single object S = (o 0thh object (z), consists of camera pose, full RGB and Depth image and object mask. A 3D TSDF reconstruction is also generated for the whole sequence. See Fig. In this paper, each training sequence strictly contains single instance of single object, while during inference with both single and multi-object sequences. We discuss how the method could be adapted to train from multi-object sequences in Sec. 6. We refer to each distinct object grouping as a category, e.g. different hats would belong to different instance classes but the same category. Given we want to learn visual descriptors that are robust over different objects.
,..., 0.") .
work
instance class and its
4.2 Class-Aware Dense Object Nets
Hard Jabel
Soft label
Confidence
Teannin
O
BesNetau
Gruph
Projection
Class
Traming
Jata
O
Bredkittio
Simbarity Gragh
Traming Traiing sbuse
Tratting Phono Trriting Phase
Selt experriaten
Sell-aupervisice
Doca Pair
Clusserr Label (citurett)
Deta
Traplet
DINV
Textning
Sampling
DON--Sot
O
Teaning
Seg. ID
Negathvw Sing,
Latest
(fromn)
(froor))
teferench
Choete
Graph
Figure 3: The simplified data flow for the hard (left) and soft (right) label variants of our method.
Blue arrows signify self-supervisory signals, grey the main data flow, green and red arrows denote positive and negative triplet pairings. (best viewed in colour)
To disentangle the descriptor subsets used by different objects, most straightforward is find non- match pairs between them and learn to repel them. We considered two variants: (i) DON+Hard producing discrete "hard" labels by assigning each instance to cluster or (ii) DON+Sofi using similarity-baad triplet sampling and continuous "soft" confidence scores. Both variants use the same DON model for dense correspondence, and similarity graph for object disentanglement, dif- fering mainly in data flow, sampling and loss generation, described below (see Fig. 3).

---

4.2.1 Similarity Graph
Since we assume access object sequences but not their class labels, the method requires way of understanding each object's identity or class grouping in a different way. Here we propose object disentangiement based on recent work [18] which consists of a similarity graph (SG) and random walker sampling (RWS). SG is an undirected graph with each node representing a training sequence and each edge the similarity between the node pair. Thus, given any node in the SG, we find which other nodes are most probably similar to it. RWS is the sampling of node pairs by "walking" on edges with transitional probability proportional to the edge weights.
The similarity graph is undirected, with each training sequence representing node S,. The edges between each pair of nodes is determined by similarity weights W S,) which are calculated using a similarity and dissimilarity measure, W+ and W respectively. Each RGB frame from each sequence is pre-ppee pre-processed via pre-trained ResNet before being used for edge weight calculation. Other inputs were also considered in Sec. 5.
The frames of each sequence clustered into N₁ local intra-sequence groupings via K-Means. Since the sequence contains only single object, these clusters tend to contain different view points of that object. This solves the over-representation issue that some sequences may suffer from, e.g. if one the views is prevalent over others. Each sequence S; then has corresponding cluster centroids Cti- The centroids have the same dimensions as the input features. Through minimum linear sum assignment [20] with assignment matrix X, the minimal cluster distance sum is found and used as dissimilarity measure (See Eq. 3). In our version of this weight, it normalized by the number of pairs Nd and the number of dimensions Na.
WT (S, minx Es Em -
Np.Na
(3)
s.t. Each cluster being assigned to at most 1 pair
Similarly, all frames from all sequences are clustered together in Ng global inter-sequence clusters via K-Means. By using large Ng. the frequency at which two objects share the same clusters is indicative of how similar those objects are, shaping the similarity measure W+,, in [18].
Finally, the two measures are combined along with a tuning parameter which determines the importance of each measure over the other (see Eq. 4). The final weight is strictly negative
W S,,,,,) max (AW+ (,) w (,,)))
(4)
Given a starting node in the graph, a transition probability distribution is formed proportional to the connected edge weights. By sampling a transition from Pt, a 'random walk' is done node that is more likely similar. Similarly, sampling from (1 - -P:) is used to reach more probably dissimilar nodes. We propose a dissimilarity confidence e in the pairing equal to the min-max normalized w- This is replacement of the confidence proposed in [18] which was found to be very sensitive to 2.
4.2.2 Dense Object Net
For DONs we use a 34 layer, 8-stride ResNet architecture. Its output descriptor image is upsampled to the original input size and has shape (H, W. (.,.WD where H and W are the input image height and width, and Daese is the number of descriptor dimensions.
Hard Classification: As in [18], for DON+Hard the triplets (anchor A, positive P. and negative N) of sample frames are generated using the similarity graph and fed into a projection network comprised of two fully connected layers with ReLU activations. The projection network is trained via triplet loss scaled by our confidence c and margin M. The loss is made non -egaatit as shown:
L ReLU (I - Pl12 - ||A -N1 + c.M)
(5)
The resulting projection output is a continuous latent class representation. Finally, the representa- tions of the training data are grouped into K clusters, determining the discrete classes. In inference time, assuming the input belongs to one of the existing classes, can be projected and matched to the closest cluster. We denote this as training phase one. In the second phase, the DON iteratively

---

accepts two kinds of image pairs as inputs with equal probability: (i) same sequence pairs that con- sist of two different frames of the same object from the same sequence and (ii) different sequence pairs that consist of two frames from random different sequences. In the different sequence scenario, the objects' discrete classes are inferred by assigning them to one of the clusters if their classes are the same, the pair is ignored, otherwise non-match pixel pairs are generated and used for DON loss. In the same sequence, both match and non-match pairs are generated.
Soft Classification: For the DON+Soft variant, the similarity graph is generated with all available sequences (training phase one). A random sequence is sampled and negative sequence pairing is randomly chosen based on the transition probabilities of the similarity graph. Two frame samples are taken from the initial sequence (anchor A and positive P) and one from the negative sequence, N, forming a triplet. The triplet is passed through the DON. Only non non-matches are generated for the negative pair. combination of the DON pixelwise match and non-match losses used (See definitions in Eq. and Eq. 2). The negative pair does not have any matches, instead its Lnon match scaled by the confidence e in the negative pairing, thus if there strong belief that the pair from the same class, the loss is smaller. The total loss is defined as follows:
L L pon match L.pos + C. non match
Experiments
With our experiments we want answer how effective our method is at separating object descriptors based on object categories. In this section, we investigate this both qualitatively and qualitatively.
5.1 Experimental Setup
(i) real data from [1] and (ii) newly produced simulated data (see Fig. 4). For the simulated data we placed a Gazebo-simulated Franka Emika Panda manipu- lator equipped with RealSense RS200 RGB-D camera, and captured objects from the 3D GEM dataset [21] in randomised poses. A summary of data is presented in Table 1. Samples of the datasets shown in Fig. 4. Note, that because of the use of background augmentation in DON, the unrealistic scene in the simulation does not affect training.
Training Data: For training
Name
Scenes
Images
Categories
Instance Classes
Real Shoes and Hats (S&H)
53
25754
(shoes and hats)
8
Real Robots (Bots)
10
6143
(3x robot toys)
3
Simulated Cans and Books (C&B)
30
22155
2 (books and cans)
8
Table 1: Summary of data
Figure 4: Samples of RGB object images used as training data, real (left) and simulated (right).
Hyperparameters: Unless otherwise specified, all trainings are done for 3,500 iterations with D 5 descriptor dimensions. The learning rate is 10-4, with a decay of 0.9. DON margin pa- rameter M is set to 0.5. For the similarity graph, we used N; 5 and N₂- 300 clusters, and 2 = 0.1. For Hord, the projection network is trained for 500 steps (phase one) before proceeding.
5.2 Object Disentanglement Experiments
Setup: For our method to work, object representations need to be effectively disentangled. We test the ability of the full classification model (i.e., consisting of the Similarity Graph and Projection Network, as in DON+Hard) to work with DON-compatible object sequences. We consider three input types: (i) raw RGB; (ii) output of pre-trained frozen ResNet50; and (iii) masked Descriptor Image produced by a pre-trained DON. All inputs have been cropped to fit only the object and interpolated to a fixed size. We test both on simulated (C&B) and real (H&S) objects, where each object category contains 4 separate instance classes. We take 600 random images for training of the
5

---

model and then test its cluster assignment on 200 random test images of known object instances from unseen sequences. Inference into clusters done via K-Means with K = 2 and K 8 clusters corresponding to the number of object categories and instance classes respectively. Note that K does not affect training. Cluster assignment to ground truth classes done by minimum linear sum assignment [20]. The model accuracy is then the fraction of images that are correctly assigned.
We also aim to show whether the descriptors produced by our methods are effective in separating the classes and how they compare to DONs trained with ground truth labels and without any labels. The models are trained on C&B. The descriptors of 50 random images are produced and 400 on-object pixels are sampled from each. These pixels are then projected via TSNE. For DON+Hard, K is set to class clusters to highlight the difficulty of not knowing the exact number of classes.
Results: As can be seen in Table 2, the ResNet features appear far superior over other inputs. Not only are they effective in separating the features into categories but also into object instance classes.
Moreover, as it can be seen in Fig. 5, even without any parameter informing the DON+ model that there are two object categories, we observe clear separation of two clusters containing only objects from that category. Our qualitative evaluation demonstrates that the method is effective in separating objects, having similar objects share similar descriptor spaces. Comparing it with the DON+Soff, there is no clear discrete separation between the classes, but there's s a gradual transition between them. We want to note that this is closer to how we interpret objects ourselves if two objects have some similar features such as a colour, a handle, or other, we notice that similarity, while still detecting the overall difference between the objects.
Data
Raw Image
ResNet Features
Masked Descriptor
C&B @K 2
52.93%
99.20%
96.31%
C&B @K= = 8 22.13%
95.25%
76.23%
H&S @K 2 51.00%
97.80%
85.26%
H&S @K = 8 16.07%
93.12%
71.09%
Table 2: Comparison of classification accuracy for different input types.
Vithou lobel
With ground truth
With hare lebel tours
With sot istels tours)
Figure 5: TSNE projection of randomly picked pixels from descriptor images produced by DONs trained without classification, with discrete ground truth labels, and our HON+Hard and DON+ methods. Fight classes are used from two categories (books and cans).
5.3 Deseriptor Match Accuracy Experiments
Setup: Given the descriptor scriptor image is found as the one with minimal 1.2-norm. When working in single object scenarios, the original DON finds good semantic correspondences between semantically corresponding ob- jects. However, as soon as there are multiple objects in the target image, the quality of the matches falls. To test this, we create new images, with unseen object instances in visually cluttered scenes, either through image collages or real photos (see Fig. 6). We manually label points on a set of 40 single-object images out of H&S (not used for training) and our 20 multi-object images, each having labelled semantically significant locations (when not occluded). For example, this includes distinctive points such as tip, top and back of a hat. This results 432 pixel pairs for multi-object matches and 1218 for single-object matches. The closest descriptors of the labelled points in the single-object images are found in query single- or multi-object images. The distance between the best match and the labelled point is the error. We plot the normalised error versus the fraction of
random point from query image, its best match in target de-

---

images as cumulative distribution (CDF). Comparison is done between our models and DON. As upper baseline models trained with ground truth DON+GT (as per [I]) and DON+Soft+GT, while DAISY [22] provide non-semantic lower baseline. All trained on H&S. A DON+Soft trained on three extra classes (on H&S+Bots) is also compared with DON+Soft.
The quality of matches is evaluated by picking random points on an image and comparing the cor- respondences on test image with novel instances of known classes. Models trained on H&S.
Figure 6: Examples of multi-object test images in cluttered scenes: shoe, hat, and both.
The CDFs shown in Fig. 7. can be seen that the change in performance with increase of classes is minimal, while there is an even smaller error in single object matches. As it can also be seen in Fig. 8, Soft outperforms the other methods.
Results:
(a] Strgle oblect motches
Multi sbject matches
matches
Multi matches
Froccuo imagus
Fractron lsbals"
Thaied on classes
12 pbel orror distancs, tormalizzed by diagonal
distance. normalized diagonal
Figure 7: CDF of L2 pixel error distance normalised by the image diagonal for single and multi- object queries. Left: comparing our methods (denoted by ") with DON. DON+Soft labels (purple) most effectively retain good match correspondences in multi-object queries with undeterred perfor- mance for 67% matches. Right: comparing DON+ DSft performance when trained on shoes and hats only vs. trained with three extra classes. (best viewed in colour)
Source
Target IDON
Target (DCN- Hardi
Target (DON, no SSCI
Figure 8: Random points from query images (left column) matched the closest descriptor points in the target images with DON+Soft, DON+ and no self-supervised classification (original).
Discussion: As it can be seen both qualitatively and quantitatively, DON+Soft produces more ro- bust correspondences in multi-ohject scenarios. Surprisingly there was slight improvement in single-object matches. We believe this could be due to the contrast between objects encouraging the descriptors to be more specific and informative.
5.4 Real Robot Experiment
Setup: To showcase the applicability of the model real robot we set up the following exper- iment. We use the mobile manipulator MPPL's [23] Franka Emika Panda arm equipped with RealSense D435i RGB-D camera. Firstly, point is selected on image of object from the
7

---

training set, in this case, a shoe or hat. Then the robot finds the best match within its environment, by querying a DON+Soft trained on H&S. The correspondent pixel projected into 3D space via the depth image and camera-to-won woold transformation. Finally, the robot produces a grasp for said point. Note that the objects used in the experiment are not part of the training set, and the camera used by the robot is different from the ones used for the data gathering. The clutter observed by the robot contains both known objects on which the model is trained and novel ones to check that the model does not confuse the correspondences with either. The source object, source point on object as well as the clutter configurations were varied to ensure robustness. A total of 20 trials were done.
Figure 9: Grasping given a point: the model finds the best match to selected point (shown in corne) within the robot's 'ss view, the pixel is projected into 3D space, the robot produces grasps.
Results: See Fig. 9. The correspondences are correctly placed on the queried object during all trials. One failure occurred when the point found was at the edge of the object couldn't 't be picked.
6 Conclusions
Limitations: Our method shows more robust matches than its predecessor in multi-object scenarios. One failure case we observed how our method deals with occlusion. Matches to occluded points are found in completely wrong locations which indicates that although the descriptor subsets for similar object are similar, the individual distances between descriptors within the subsets are big- Querying for best-matches does not consider neighbouring descriptors, geometric consistency or other metrics which likely causes some of the other errors our best-match correspondences, The DON in its current form is unable to work with dynamic training scenes. This could potentially be alleviated with the use of multiple viewpoints or tracking pixels with optical flow. Moreover, since the current system relies masks of the input sequence objects, it could be extended to multi-objec training scenes by adding a semantic segmentation network. The use of K-Means leads computational complexity scaling with number of sequences, as well as dealing with constant number of clusters when the real number of classes is unknown. In our exploratory work, to alleviate these issues we tried Mean Shift Clustering, Bayesian Gaussian Mixture Models and DBSCAN [24]. However, it was found that the classification accuracy on training data dropped significantly, likely because the latent classes would be unaffected by this since it purely relies on similarity measures.
DON+Hard
described by normal distributions. Doft
Applications: This work enables applications that might treat different classes of objects differently. Here we provide three pick-and-place examples: (i) given some knowledge base different objects and graspable points them, the method can be used to automatically detect the closest match points even in cluttered multi-object scenes; (ii) or the opposite way, the model could see an object, infer its latent class and match it to the closest of its class clusters, thus generate the best grasping points for that class; (iii) alternatively, the model could be used to observe how people interact with objects, and learn which descriptor-points In terms of more complex applications, we believe this representation could pave the way toward methods such self-supervised robot affordance learning and action imitation.
suitable grasping points for each self-supervised class.
In this paper, we extended an existing method for self-supervised object descriptors to perform more accurately in multi-object queries and cluttered scenes. We showcase its improve- ment over its predecessor both qualitatively and quantitatively. Further analysis of the method could explore how having novel objects vs known objects in the background of a scene affects the perfor- mance, ie. would it be better for the model to be trained on everything it sees, or does performance degrade once an object has been 'learned'. Future work will be focused making the descriptors more useful by grounding them to actionable representations, and extending the training toward training in the wild from dynamic unstructured scenes.
Future Work:
8

---

References
[1] Florence, L. Manuelli, and R. Tedrake. Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation. Conference Robot Learning (CoRL), 6 2018.
[2] Florence, L. Manuelli, and R. Tedrake. Self-Supervised Correspondence in Visuomotor Policy Learning. IEEE Conference on Robotics and Automoation (ICRA), 2020.
[3] L. Manuelli, Y. Li, P. Florence, and R. Tedrake. Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learning. Conference on Robot Learning (CoRL.), 2020.
[4] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-End Training of Deep Visuomotor Poli- cies. Journal of Machine Learning Research, 17(39):1-40, 2016.
[5] C. B. Choy, J. Y. Gwak, S. Savarese, and M. Chandraker. Universal Correspondence Network. Advances in Neural Information Processing Systems, pages 2414-2422, 2016.
[6] S. Kim, Min, B. Ham, Jeon, S. Lin, and K. Sohn. FCSS: Fully Convolutional Self- Similarity for Dense Semantic Correspondence. 30th IEEE Conference on Computer Vision and Pattern Recognition, 2017-Jamarry:6666655
2017.
[7] J. Lee, D. Kim, J. Ponce, and B. Ham. SFNet: Learning Object-aware Semantic Correspon- dence. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2019-Jume22273-2220 4 2019.
[8] A. Kupcsik, M. Spies, A. Klein, M. Todescato, N. Wanick, P. Schillinger, and M. Buerger. Supervised Training of Dense Object Nets using Optimal Descriptors for Industrial Robotic Applications. Association for the Advancement of Artificial Intelligence (AAAI), 2 2021.
[9] T. Zhou, P. Krähenbühl, M. Aubry, Q. Huang, and A. A. Efros. Learning Dense Correspon- dence via 3D-guided Cycle Consistency. IEEE Computer Society Conference on Computer Vision and Pattern Recognition,
2016.
[10] U. Deekshith, N. Gajjar, M. Schwarz, and S. Behnke. Visual Descriptor Learning from Monoc- ular Video. 15th International Joint Conference on Computer Vision, Imaging and Computer Graphies Theory and Applications Volume 5: VISAPP, pages 444 451, 2020. ISSN 2184-
4321. doi: 10.100.2220 10,5220000009933344444455
[11] E. Jang, C. Devin, V. Vanhoucke, and S. Levine. Grasp2Vec: Learning Object Representations from Self-Super Grasping. In Conference on Robot Learning (CoRL.), 2018.
[12] M. Vecerik, J.-B. Regli, O. Sushkov, D. Barker, R. Peyceviciute, T. Rothörl, C. Schuster, R. Hadsell, L. Agapito, and J. Scholz. S3K: Self- SuperSupe Semantic Keypoints for Robotic Manipulation via Multi-View Consistency. Conference on Robot Learning (CoRL), 9 2020.
[13] X. Wang, K. He, and A. Gupta. Transitive Invariance for Self-Supervised Visual Representa- tion Learning. In IEEE International Conference on Computer Vision (ICCV), pages 1338- 1347, 2017. doi: 10. 10.1101////22211....
[14] P. Huang, Y. Huang, W. Wang, and L. Wang. Deep Embedding Network for Clustering. In 2014 22nd International Conference on Pattern Recognition, pages 1532-1537, 2014. doi: N0.1109/CPR.2014.222
[15] M. Caron, P. Bojanowski, A. Joulin, and M. Douze. Deep Clustering for Unsupervised Learn- ing of Visual Features. In European Conference on Computer Vision ECCV 2018, pages 139-156, Cham, 2018. Springer International Publishing. ISBN 978-3-030-0126499
[16] T. Patten, K. Park, and M. Vineze. DGCM-Net: Dense Geometrical Correspondence Matching Network for Incremental Experieene-BBasee Robotic Grasping. Frontiers in Robotics and Al, 2020. 10010003/9//rrbb.200000
[17] S. Pirk, M. Khansari, Y. Bai, C. Lynch, and P. Sermanet. Online Object Representations with Contrastive Learning. 2019.

---

[18] J. Tan, C. Song, and A. Boularias. A supervupe Learning System for Object Detection Videos Using Random Walks on Graphs. IEEE International Conference on Roboties and Automation (ICRA), 11 2021.
[19] T. Schmidt, R. A. Newcombe, and D. Fox. Self-Supervised Visual Descriptor Learning for Dense Correspondence. IEEE Robotics and Automation Letters, 2:420-427, 2017.
[20] H. W. Kuhn. The Hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(1-2):83 97, 3 1955. doi: 10..10002222a.80000000000
[21] A. Rasouli and J. Tsotsos. The Effect of Color Space Selection on Detectability and Dis- criminability of Colored Objects. CoRR, 2017.
[22] E. Tola, V. Lepetit, and P. Fua. Daisy: An Efficient Dense Descriptor Applied to Wide Baseline Stereo. IEEE transactions on pattern analysis and machine intelligence, 32:815-30, 2010.
[23] Liu, P. Balatti, K. Ellis, D. Hadjivelichkov, D. Stoyanov, A. Ajoudani, and D. Kanoulas. Garbage Collection and Sorting with Mobile Manipulator using Deep Learning and Whole- Body Control. In 2020 IEEE-RAS International Conference on Humanoid Robots (Hu- manoids), 2021.
[24] M. Ester, H.-P. Kriegel, Sander, and X. Xu. A Density- Algorithm for Discover- ing Clusters in Large Spatial Databases with Noise. In Second International Conference on Knowledge Discovery and Data Mining, KDD'96, page 226-231, AAAI Press, 1996.
10


Reviews:
[{"review_id": "ur1tIJWKxAK", "reviewer": "Reviewer_ZCDs", "summary": "This paper presented a new method to extend the Dense Object Nets (DON) to multi-object scenes. It presents two ways to learn class-aware dense object descriptors to be robust to multi-object scenes.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main and the only baseline, which this paper has compared to, is the Dense Object Nets (DON). Compared to DON, this paper shows the robust feature matching in the multi-object case. This innovation in this paper is a bit incremental but it shows a step further to the real-world applications.\n\nMy main concern is that the testing case in this paper is a bit constrained.  \nAlthough it targets multi-object scenes, the test scenes actually only contain one instance per class. It would be interesting to test how the method will work in an environment where multiple shoes or hats exist. This would be a more realistic scenario.  \nSecondly, it seems that the reference image in the test has been seen in the training (L209, L228). This makes the test cases weak. It will be more convincing if the method can use an unseen object (in the same category) for the reference image.  \nThirdly, the method only trained for two-class. It will be important to see if such binary classification can be generalized to more classes.    \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The innovation in this paper is limited. It also requires some further improvements in the writing and experiments, even if it shows some interesting experimental results. My recommendation opinion is based on these considerations.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "In addition to the issues mentioned above, I will also be interested to see if the following concerns can be addressed.\n\nThis paper shows that the trained feature descriptor can separate different object classes. It would be interesting to see the feature descriptor visualizations and also compare them with DON's feature descriptor visualization. It can bring more insights, as DON did in their paper. \n\nIn the TSNE visualization for the soft labelling, it seems that the sprite can and pepsi can feature descriptor embeddings have mixed with the book embeddings. It means the book and can categories have mixed. I am not very convinced these two instances share similarities with the books unless some other (e.g. visual or qualitative) evidence can be shown. I am also wondering if this issue can be mitigated by increasing the feature descriptor dimensions. Actually, the authors did not mention the feature dimension number they use for the model. It is an important hyperparameter and can influence performance a lot.   \n\nThe only baseline method this paper compared to is DON. However, there have been many papers coming out in the field of learning feature descriptors, such as Self-supervised visual descriptor learning for dense correspondence, sosnet, d2net, or HyNet. Most of those approaches also learn the feature descriptor based on triplet loss or contrastive loss. It would be interesting to see some comparisons to these generic feature descriptor methods. To obtain the per-pixel correspondence from these descriptor approaches, one can simply run an argmin of l2 feature descriptor distance to the target image, as done in the DON and its following works.\n\nAs the authors have talked about in the limitations section, the usage of K-means has constrained the application of this approach. Knowing the number of clusters in advance is a bit contradictory to the target usage case of multi-object scenes.\n\nIn terms of the writing part:\n* equation 5: What does c.M mean?\n* L120: object view variance: has it already been addressed by training on the images captured from different viewpoints?"}}, {"review_id": "g4Wuw7Mlsf3", "reviewer": "Reviewer_2SMq", "summary": "The paper extends the Dense Object Network (DON) method by incorporating object category awareness. The idea of DON is to learn pixel-wise object descriptors in order to detect pixel-to-pixel correspondences between two objects of the same category appearing in different images. While DON focuses on finding correspondences between objects in single-object scenes, this paper extends DON to cope with multi-object scenes. The idea is to generate object category labels for each object in an unsupervised fashion (through clustering in a feature space), and use these labels to inform the object descriptor learning. The experiments show the promising results for disentangling two object categories in a multi-object scene and also improves the single-object case.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThe paper addresses the important problem of finding dense correspondences between objects and object parts without CAD models nor explicit object category labels. The paper shows promising results for disentangling two object categories (cans vs. books and hats vs. shoes) in a multi-object scene. The technical approach is sound and the presentation good. The example visualizations given in the paper are illustrative of the task.\n\n## Weaknesses\n\nThe main weakness of the paper are that the method is only evaluated on a low number of object classes. It is difficult to judge how generalizable the method is given that descriptors are only learned on two / five classes of objects respectively. I would expect the approach to be evaluated on more objects, including such object with similar shapes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper tackles an important problem, provides a good presentation and a technically sound approach. One weakness is that few object classes are used for evaluation which renders the experiments somewhat preliminary and limits the contribution of the paper. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "The paper would make a stronger point if more object classes in particular with similar shapes would be considered during evaluation and training.\n\n\n\n[The following issues concern an earlier version and have been addressed in the paper revision]\n\nTechnical approach: \n- It is a bit hard to follow the distinction between instance classes, categories, sequences, random walks on the graph etc. A figure giving an overview of all these entities and how they fit together would be extremely helpful.\n- The paper combines DONs with similarity graphs, and as both are critical components, similarity graphs should be introduced at the same level as DONs. Furthermore, it would be better to first explain the general idea of these similarity graphs (L137-L140) before delving into the details. Finally, a figure explaining how random walks on these graphs look like, how the matrices W+ and W- come into play etc. would be helpful.\n- The unsupervised generation of labels seems quite complicated, in particular since only two object classes are present. The paper doesn't fully convince me that such complex label generation machinery is necessary.\n- I can think of one simple lower baseline, for example obtaining labels by classifying the images with ImageNet und using ImageNet labels instead of the unsupervised approach, but I believe there are even easier unsupervised ones.\n- Similarly, it would be interesting to consider an upper baseline, by providing the ground truth label of the item class explicitly to the network. \n\nExperimental evaluation: \n- It is stated on the side that only two types of objects categories per experiment are considered. The description of the dataset, incl. how many instances are used, etc. should be best summarized in an experimental setup section and a table.\n- Entanglement of hypotheses and result discussion: How the experimental conditions differ between experiments 5.1, 5.2, 5.3? Are the same objects, trained networks etc used? This is not fully clear to me\n- L184: Why are K=2 and K=8 chosen? Why not K=3, 4, 20 ...?\n- L179 states a detail about 500 training steps - what is that mentioned here, why does it matter? \n- L172-173 states that the objective of the experiment is to “answer effective our method is at separating object descriptors” - but isn’t the scope of the paper broader, namely to present a method for class-aware dense object matching?\n- L178: “consider three input types” - input to what? Do you mean this is the feature space used for k-means?\n\nTechnical questions:\n- How is the depth/3D image used? The paper doesn’t mention any details about why a 3D sensor is required at all for descriptor learning.\n- L140: “Complicated” per se does not mean that something is bad. Maybe it performs better than what is proposed here? Why was the original approach discarded?\n- L165: How are these sub-loss functions L_pos,non-match, etc. defined?\n- L158: “a negative sequence pairing is randomly chosen” - how exactly? By starting from the randomly sampled sequence and then…?\n\nMinor issues:\n- In the introduction, I would suggest to explain on simple terms and concisely the high-level intuition of the method. E.g. the term “class awareness” is used in the title yet “unsupervised classification” is introduced in L21 without explaining how this unsupervised classification is going to work.\n- L90 “whereas” doesn’t fit here\n- L104 mentions “background augmentation” is mentioned but not explained before\n- Figure 3: I would suggest to add the inputs and outputs as well, I was confused by what the output of “Projection” is\n"}}, {"review_id": "256Kvfe7PhA", "reviewer": "Reviewer_RTPV", "summary": "This paper resolves a limitation of Dense Object Nets in scaling to multiple objects in a self-supervised fashion. The authors introduce a fully self-supervised module which does need the label of which object is in the camera view.  Specifically, in the original Dense Object Nets, in the multi-different-object case, a small amount of human supervision was needed in order to apply \"cross-object loss\" as stated in the original paper.  With this limitation resolved, this in theory aids the ability of Dense Object Nets to potentially larger scale than before, since the method is more fully self-supervised.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think this is good work.  That said, there are a few areas for improvement.\n\nStrengths:\n- The contribution is clean. (Although it could use some revision in making it more clearly stated. Please see my Summary for an example of how the main contribution could be more specifically stated.)\n- Figure 3 is very good at explaining the main idea.\n- Figure 6 represents a substantial improvement relative to the original Dense Object Nets paper, particularly for multi object matches.\n- The real robot experiment does not really add that much to the core academic argument, but still, it is appreciated.\n- Figure 4 shows compelling clustering.\n\nWeaknesses\n- Section 4.3.2, which is maybe the most critical section, is not very clear.  How is each C computed?  This section should be improved for clarity if the paper is accepted.  Please just run it by multiple rounds of somebody who is not familiar with the work, and see if they can read it and understand exactly all of the symbols and how they are computed.  \n- The cost of labeling which object is present in each scene, as was assumed in the original Dense Object Nets paper, is not actually that large of a cost.  Yet still, it is nonzero cost.  The missed opportunity though is in showing that, without this small cost, maybe this method can be scaled to a much larger scale than before. Especially since the authors set up a simulated pipeline, maybe they could have scaled this to 1000s of objects.  This would be interesting but wasn't done.\n\nI would recommend the paper title be slightly revised to \"Fully Self-Supervised Class Awareness in Dense Object Descriptors\".  The current title, \"Self-Supervised Dense Object Descriptors with Class Awareness\" is not as precise, since in the previous Dense Object Nets paper, there was the ability to do distinct multi-object descriptors.  The difference though is that in order to do this before, the images needed to be labeled with which object they corresponded to.  \"Fully Self-Supervised Class Awareness in Dense Object Descriptors\" is accordingly more precise. \n\nAlso a related work comment is incorrect.  It says: \"We are not aware of any work that produces pixel-pixel correspondences in multi-object scenes, based on unsupervised or self-supervised training without ground truth models, which is introduced in this paper.\"  This should be revised to include a couple key words, called out with asterisks. \"We are not aware of any work that produces pixel-pixel correspondences *across or between different* multi-object scenes, based on unsupervised or self-supervised training *without per-image labels of which object is present*.\"  Note that many SfM methods would be able to find dense correspondences in a multi-object scene if all the images are from the same scene, so the *across or between different* scenes is critical.  Also, the original Dense Object Nets work required a label of which object is present in order to scale to multiple distinct objects.\n\nPlease also cite Schmidt et al, \"Self-supervised visual descriptor learning for dense correspondence\", which Dense Object Nets is heavily based on, but is missing from the current citations.\n\nMinor note:\n- The related work, referring to S3K, says: \"it does not produce correspondences\".  But keypoints are correspondences.  They are just sparse correspondences, instead of dense.  So this statement should be revised to say that S3K does not produce dense correspondences.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think this paper provides a clean contribution.  The experiments are sufficient to show the method works.  And getting rid of the \"which object is present\" label needed from DON before could unlock additional scaling of the method.\n\nI put \"Strong Accept\" because I am willing to argue for this paper's acceptance.  That said, it would be a valid view that the contribution is incremental and of a small enough increment that it may not need to be accepted.  I'm willing to argue the contrary based on any specifics that other reviews point out, but I could ultimately support either decision for this paper.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "This is a list referring to things mentioned previously:\n1. Paper's key contribution could be more precisely stated.  For inspiration please see my Summary.\n2. I would recommend the paper title be slightly revised to \"Fully Self-Supervised Class Awareness in Dense Object Descriptors\". Please see discussion above.\n3. Please also cite Schmidt et al (see above)\n4. Please see comment above regarding comment on S3K."}}]

Author rebuttal:

Meta-review: This paper extends DON by making the object labels self-supervised and found via cluster assignment in pretrained visual feature space. This removes the prior limitation of DONs where the object-in-view needed to be labeled. The authors also show that augmenting the training set with cluttered images also improves generalization to multi-object scenes.  

This paper currently has 2 weak rejects and 1 strong accept, with reviewer RTPV willing to go to bat for this paper. 

*Strengths*: Reviewers appreciate the focus on scalable data for representation learning. Figure 6 represents an improvement over DON for multi-object matches, real robot results 

*Weaknesses*: Reviewers 2SMq, RTPV say that the motivations and technical implementation details of Similarity Graphs (Sec 4.3.2) are unclear. 2SMq and ZCDs also recommend that the overall method be improved for clarity. Both 2SMq and ZCDs take issue with the test scenes only involve two objects classes and only contain one instance per class. I think this is a valid criticism, because it is quite possible that a cluster-based assignment for auto-labeling data may scale poorly as the number of classes is increased. 

I have some questions about Figure 6 that I would like the authors to respond to in their rebuttal, and potentially RTPV as well.

It seems to me from reading the results in Figure 6 that the improved generalization to multi-object scenes is not a definitive property of the method, but rather due to the use of incorporating multi-object scenes into the training data (L208), and the use of soft labels. The benefit of training on multi-object scenes is obvious, so the remaining question is whether the proposed method contributes anything new to better descriptors in multi-object scenes beyond the data. Figure 6 seems to suggest this, but I noticed that the DON baseline performs about as well as the hard variant of the method on multi-object matches (hard to really understand the relative difference between blue/orange in Fig 6). This leads me to suspect that the improved performance of the soft labels is not due to the clustered label assignments, but rather the fact that the labels are just soft. In other words, a null hypothesis is that "one does not require a self-supervised method to recover soft labels; one could imagine a simple label smoothing trick applied to the hard labels would work as well as the DON + Soft Labels". 

I would appreciate if the authors could check this hypothesis. Another experiment  to check the null hypothesis would be to train on self-supervised labels from single-object scenes only, and then test on multi-object scenes. If my suspicions are correct, both DON and DON + self-supervised labels will do equally poorly.

If the null hypothesis that "soft labels improve multi-object generalization for DON" is a property of the label softness and not the self-supervised clustering method itself, I think that does diminish the novelty of the paper, especially considering that the method is simply using clustering to do label propagation. Many prior works such as Caron et al. '18 have utilized this trick to provide labels for some other downstream learning task. Connecting unsupervised label propagation to a relatively new descriptor learning method is not guaranteed to "just work", so I appreciate the research being done here, but then I think some more compelling evidence around the promised scalability of the method needs to be shown. See RTPV's point about "across or between different scenes is critical" or handling more than 2 object classes as the other reviewers have pointed out. One might argue that it would not be hard to extend the existing DON collection protocol (e.g. a camera orbiting a fixed object for which you know the class) by tossing in some distractor objects to make the data multi-view without additional labeling cost.

Some relevant work the authors may want to look at:
- There have been some prior works like Grasp2Vec (Jang and Devin et al. 2018) that show dense, self-supervised object features learned in multi-object scenes (though not via pixel-to-pixel correspondences) and Object Contrastive Nets (Pirk et al. 2019) which also propose self-supervised object correspondences in multi-object scenes. Pirk et al. uses a single scene to learn object representations, but conceptually may not be limited by that. 


---- update 20210904

I thank the authors for their detailed rebuttal and for the reviewers in engaging in discussion. One of the reviewers have updated their rating to weak accept, and given that the authors have addressed all my concerns as well, I will recommend an accept. 
Decision: Accept (Poster)