Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper extends the Dense Object Network (DON) method by incorporating object category awareness. The idea of DON is to learn pixel-wise object descriptors in order to detect pixel-to-pixel correspondences between two objects of the same category appearing in different images. While DON focuses on finding correspondences between objects in single-object scenes, this paper extends DON to cope with multi-object scenes. The idea is to generate object category labels for each object in an unsupervised fashion (through clustering in a feature space), and use these labels to inform the object descriptor learning. The experiments show the promising results for disentangling two object categories in a multi-object scene and also improves the single-object case.

strengths_and_weaknesses: ## Strengths

The paper addresses the important problem of finding dense correspondences between objects and object parts without CAD models nor explicit object category labels. The paper shows promising results for disentangling two object categories (cans vs. books and hats vs. shoes) in a multi-object scene. The technical approach is sound and the presentation good. The example visualizations given in the paper are illustrative of the task.

## Weaknesses

The main weakness of the paper are that the method is only evaluated on a low number of object classes. It is difficult to judge how generalizable the method is given that descriptors are only learned on two / five classes of objects respectively. I would expect the approach to be evaluated on more objects, including such object with similar shapes.
summary_of_recommendation: The paper tackles an important problem, provides a good presentation and a technically sound approach. One weakness is that few object classes are used for evaluation which renders the experiments somewhat preliminary and limits the contribution of the paper. 
issues: The paper would make a stronger point if more object classes in particular with similar shapes would be considered during evaluation and training.



[The following issues concern an earlier version and have been addressed in the paper revision]

Technical approach: 
- It is a bit hard to follow the distinction between instance classes, categories, sequences, random walks on the graph etc. A figure giving an overview of all these entities and how they fit together would be extremely helpful.
- The paper combines DONs with similarity graphs, and as both are critical components, similarity graphs should be introduced at the same level as DONs. Furthermore, it would be better to first explain the general idea of these similarity graphs (L137-L140) before delving into the details. Finally, a figure explaining how random walks on these graphs look like, how the matrices W+ and W- come into play etc. would be helpful.
- The unsupervised generation of labels seems quite complicated, in particular since only two object classes are present. The paper doesn't fully convince me that such complex label generation machinery is necessary.
- I can think of one simple lower baseline, for example obtaining labels by classifying the images with ImageNet und using ImageNet labels instead of the unsupervised approach, but I believe there are even easier unsupervised ones.
- Similarly, it would be interesting to consider an upper baseline, by providing the ground truth label of the item class explicitly to the network. 

Experimental evaluation: 
- It is stated on the side that only two types of objects categories per experiment are considered. The description of the dataset, incl. how many instances are used, etc. should be best summarized in an experimental setup section and a table.
- Entanglement of hypotheses and result discussion: How the experimental conditions differ between experiments 5.1, 5.2, 5.3? Are the same objects, trained networks etc used? This is not fully clear to me
- L184: Why are K=2 and K=8 chosen? Why not K=3, 4, 20 ...?
- L179 states a detail about 500 training steps - what is that mentioned here, why does it matter? 
- L172-173 states that the objective of the experiment is to “answer effective our method is at separating object descriptors” - but isn’t the scope of the paper broader, namely to present a method for class-aware dense object matching?
- L178: “consider three input types” - input to what? Do you mean this is the feature space used for k-means?

Technical questions:
- How is the depth/3D image used? The paper doesn’t mention any details about why a 3D sensor is required at all for descriptor learning.
- L140: “Complicated” per se does not mean that something is bad. Maybe it performs better than what is proposed here? Why was the original approach discarded?
- L165: How are these sub-loss functions L_pos,non-match, etc. defined?
- L158: “a negative sequence pairing is randomly chosen” - how exactly? By starting from the randomly sampled sequence and then…?

Minor issues:
- In the introduction, I would suggest to explain on simple terms and concisely the high-level intuition of the method. E.g. the term “class awareness” is used in the title yet “unsupervised classification” is introduced in L21 without explaining how this unsupervised classification is going to work.
- L90 “whereas” doesn’t fit here
- L104 mentions “background augmentation” is mentioned but not explained before
- Figure 3: I would suggest to add the inputs and outputs as well, I was confused by what the output of “Projection” is
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presented a new method to extend the Dense Object Nets (DON) to multi-object scenes. It presents two ways to learn class-aware dense object descriptors to be robust to multi-object scenes.
strengths_and_weaknesses: The main and the only baseline, which this paper has compared to, is the Dense Object Nets (DON). Compared to DON, this paper shows the robust feature matching in the multi-object case. This innovation in this paper is a bit incremental but it shows a step further to the real-world applications.

My main concern is that the testing case in this paper is a bit constrained.  
Although it targets multi-object scenes, the test scenes actually only contain one instance per class. It would be interesting to test how the method will work in an environment where multiple shoes or hats exist. This would be a more realistic scenario.  
Secondly, it seems that the reference image in the test has been seen in the training (L209, L228). This makes the test cases weak. It will be more convincing if the method can use an unseen object (in the same category) for the reference image.  
Thirdly, the method only trained for two-class. It will be important to see if such binary classification can be generalized to more classes.    

summary_of_recommendation: The innovation in this paper is limited. It also requires some further improvements in the writing and experiments, even if it shows some interesting experimental results. My recommendation opinion is based on these considerations.
issues: In addition to the issues mentioned above, I will also be interested to see if the following concerns can be addressed.

This paper shows that the trained feature descriptor can separate different object classes. It would be interesting to see the feature descriptor visualizations and also compare them with DON's feature descriptor visualization. It can bring more insights, as DON did in their paper. 

In the TSNE visualization for the soft labelling, it seems that the sprite can and pepsi can feature descriptor embeddings have mixed with the book embeddings. It means the book and can categories have mixed. I am not very convinced these two instances share similarities with the books unless some other (e.g. visual or qualitative) evidence can be shown. I am also wondering if this issue can be mitigated by increasing the feature descriptor dimensions. Actually, the authors did not mention the feature dimension number they use for the model. It is an important hyperparameter and can influence performance a lot.   

The only baseline method this paper compared to is DON. However, there have been many papers coming out in the field of learning feature descriptors, such as Self-supervised visual descriptor learning for dense correspondence, sosnet, d2net, or HyNet. Most of those approaches also learn the feature descriptor based on triplet loss or contrastive loss. It would be interesting to see some comparisons to these generic feature descriptor methods. To obtain the per-pixel correspondence from these descriptor approaches, one can simply run an argmin of l2 feature descriptor distance to the target image, as done in the DON and its following works.

As the authors have talked about in the limitations section, the usage of K-means has constrained the application of this approach. Knowing the number of clusters in advance is a bit contradictory to the target usage case of multi-object scenes.

In terms of the writing part:
* equation 5: What does c.M mean?
* L120: object view variance: has it already been addressed by training on the images captured from different viewpoints?
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper resolves a limitation of Dense Object Nets in scaling to multiple objects in a self-supervised fashion. The authors introduce a fully self-supervised module which does need the label of which object is in the camera view.  Specifically, in the original Dense Object Nets, in the multi-different-object case, a small amount of human supervision was needed in order to apply "cross-object loss" as stated in the original paper.  With this limitation resolved, this in theory aids the ability of Dense Object Nets to potentially larger scale than before, since the method is more fully self-supervised.
strengths_and_weaknesses: I think this is good work.  That said, there are a few areas for improvement.

Strengths:
- The contribution is clean. (Although it could use some revision in making it more clearly stated. Please see my Summary for an example of how the main contribution could be more specifically stated.)
- Figure 3 is very good at explaining the main idea.
- Figure 6 represents a substantial improvement relative to the original Dense Object Nets paper, particularly for multi object matches.
- The real robot experiment does not really add that much to the core academic argument, but still, it is appreciated.
- Figure 4 shows compelling clustering.

Weaknesses
- Section 4.3.2, which is maybe the most critical section, is not very clear.  How is each C computed?  This section should be improved for clarity if the paper is accepted.  Please just run it by multiple rounds of somebody who is not familiar with the work, and see if they can read it and understand exactly all of the symbols and how they are computed.  
- The cost of labeling which object is present in each scene, as was assumed in the original Dense Object Nets paper, is not actually that large of a cost.  Yet still, it is nonzero cost.  The missed opportunity though is in showing that, without this small cost, maybe this method can be scaled to a much larger scale than before. Especially since the authors set up a simulated pipeline, maybe they could have scaled this to 1000s of objects.  This would be interesting but wasn't done.

I would recommend the paper title be slightly revised to "Fully Self-Supervised Class Awareness in Dense Object Descriptors".  The current title, "Self-Supervised Dense Object Descriptors with Class Awareness" is not as precise, since in the previous Dense Object Nets paper, there was the ability to do distinct multi-object descriptors.  The difference though is that in order to do this before, the images needed to be labeled with which object they corresponded to.  "Fully Self-Supervised Class Awareness in Dense Object Descriptors" is accordingly more precise. 

Also a related work comment is incorrect.  It says: "We are not aware of any work that produces pixel-pixel correspondences in multi-object scenes, based on unsupervised or self-supervised training without ground truth models, which is introduced in this paper."  This should be revised to include a couple key words, called out with asterisks. "We are not aware of any work that produces pixel-pixel correspondences *across or between different* multi-object scenes, based on unsupervised or self-supervised training *without per-image labels of which object is present*."  Note that many SfM methods would be able to find dense correspondences in a multi-object scene if all the images are from the same scene, so the *across or between different* scenes is critical.  Also, the original Dense Object Nets work required a label of which object is present in order to scale to multiple distinct objects.

Please also cite Schmidt et al, "Self-supervised visual descriptor learning for dense correspondence", which Dense Object Nets is heavily based on, but is missing from the current citations.

Minor note:
- The related work, referring to S3K, says: "it does not produce correspondences".  But keypoints are correspondences.  They are just sparse correspondences, instead of dense.  So this statement should be revised to say that S3K does not produce dense correspondences.
summary_of_recommendation: I think this paper provides a clean contribution.  The experiments are sufficient to show the method works.  And getting rid of the "which object is present" label needed from DON before could unlock additional scaling of the method.

I put "Strong Accept" because I am willing to argue for this paper's acceptance.  That said, it would be a valid view that the contribution is incremental and of a small enough increment that it may not need to be accepted.  I'm willing to argue the contrary based on any specifics that other reviews point out, but I could ultimately support either decision for this paper.
issues: This is a list referring to things mentioned previously:
1. Paper's key contribution could be more precisely stated.  For inspiration please see my Summary.
2. I would recommend the paper title be slightly revised to "Fully Self-Supervised Class Awareness in Dense Object Descriptors". Please see discussion above.
3. Please also cite Schmidt et al (see above)
4. Please see comment above regarding comment on S3K.
review_title: null

Meta-review: This paper extends DON by making the object labels self-supervised and found via cluster assignment in pretrained visual feature space. This removes the prior limitation of DONs where the object-in-view needed to be labeled. The authors also show that augmenting the training set with cluttered images also improves generalization to multi-object scenes.  

This paper currently has 2 weak rejects and 1 strong accept, with reviewer RTPV willing to go to bat for this paper. 

*Strengths*: Reviewers appreciate the focus on scalable data for representation learning. Figure 6 represents an improvement over DON for multi-object matches, real robot results 

*Weaknesses*: Reviewers 2SMq, RTPV say that the motivations and technical implementation details of Similarity Graphs (Sec 4.3.2) are unclear. 2SMq and ZCDs also recommend that the overall method be improved for clarity. Both 2SMq and ZCDs take issue with the test scenes only involve two objects classes and only contain one instance per class. I think this is a valid criticism, because it is quite possible that a cluster-based assignment for auto-labeling data may scale poorly as the number of classes is increased. 

I have some questions about Figure 6 that I would like the authors to respond to in their rebuttal, and potentially RTPV as well.

It seems to me from reading the results in Figure 6 that the improved generalization to multi-object scenes is not a definitive property of the method, but rather due to the use of incorporating multi-object scenes into the training data (L208), and the use of soft labels. The benefit of training on multi-object scenes is obvious, so the remaining question is whether the proposed method contributes anything new to better descriptors in multi-object scenes beyond the data. Figure 6 seems to suggest this, but I noticed that the DON baseline performs about as well as the hard variant of the method on multi-object matches (hard to really understand the relative difference between blue/orange in Fig 6). This leads me to suspect that the improved performance of the soft labels is not due to the clustered label assignments, but rather the fact that the labels are just soft. In other words, a null hypothesis is that "one does not require a self-supervised method to recover soft labels; one could imagine a simple label smoothing trick applied to the hard labels would work as well as the DON + Soft Labels". 

I would appreciate if the authors could check this hypothesis. Another experiment  to check the null hypothesis would be to train on self-supervised labels from single-object scenes only, and then test on multi-object scenes. If my suspicions are correct, both DON and DON + self-supervised labels will do equally poorly.

If the null hypothesis that "soft labels improve multi-object generalization for DON" is a property of the label softness and not the self-supervised clustering method itself, I think that does diminish the novelty of the paper, especially considering that the method is simply using clustering to do label propagation. Many prior works such as Caron et al. '18 have utilized this trick to provide labels for some other downstream learning task. Connecting unsupervised label propagation to a relatively new descriptor learning method is not guaranteed to "just work", so I appreciate the research being done here, but then I think some more compelling evidence around the promised scalability of the method needs to be shown. See RTPV's point about "across or between different scenes is critical" or handling more than 2 object classes as the other reviewers have pointed out. One might argue that it would not be hard to extend the existing DON collection protocol (e.g. a camera orbiting a fixed object for which you know the class) by tossing in some distractor objects to make the data multi-view without additional labeling cost.

Some relevant work the authors may want to look at:
- There have been some prior works like Grasp2Vec (Jang and Devin et al. 2018) that show dense, self-supervised object features learned in multi-object scenes (though not via pixel-to-pixel correspondences) and Object Contrastive Nets (Pirk et al. 2019) which also propose self-supervised object correspondences in multi-object scenes. Pirk et al. uses a single scene to learn object representations, but conceptually may not be limited by that. 


---- update 20210904

I thank the authors for their detailed rebuttal and for the reviewers in engaging in discussion. One of the reviewers have updated their rating to weak accept, and given that the authors have addressed all my concerns as well, I will recommend an accept. 
Decision: Accept (Poster)