Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper aims at segmenting hand and object regions given an input image, optical flow, and location of hand (for some pixels), using a contrastive approach and measurement of features called "Responsibility".  
main_review: The paper is not well-written and the content do not follow naturally. It looks like a draft at this point. There are many ambiguities, grammatical errors, and lack of explanations throughout the paper (see details below). I read the paper twice to make sure I am not missing something and also checked the appendix. It is still not very clear to me many details of the paper.

**Technical / Clarity:**

- Regarding Eq. (1), which is measuring "Responsibility", what is the motivation and what is it computing? There is not much explanation on what is the goal of it. The first question is what is h_k? It is only at Eq. (2) that this becomes somewhat clear. Is h_k indeed proj(M_k)? if so, it should be clarified in Eq. (1). Now assuming proj(M_k) is indeed the answer, if M_k is the output of a model, the best predicted responsibility is to predicts the optical flow per-pixel. The question is then how this corresponds to a particular hand and why this is useful? In the denominator of Eq. (1) there are multiple hands for different h_k. How this equation differentiates different hands, if each hand output is predicting the optical flow?


- On a related note, in lines 128 and 133 M_k is defined as both homography and model. Which one is it? In line 129 the M_k is defined as optical flow plus current index location, while in Eq. (2) the same M_k is compared only to the optical flow (and not the sum of optical flow and index). These two definitions are not the same, so which one is correct? Also it is not clear how M_k and h_k related to the Z (common embedding), Q (hand features), and K (object features) given by the model.  In architecture section, this is not clarified. how M is obtained? Is it an output of the model or estimated differently?


- Authors mention in lines 158 to 160 that at test time given a query q, a score q^T \times K_{i,j} can be measured. What does this score between hand and object feature maps indicate and how it is used for hand region or object region segmentation? Q^T \times K should be mostly zero, if the two heads truly separates the activation for hand and object, otherwise if the multiplication is not-zero it indicates the model is not correctly separating them. The description at lines 176 to 179 also does not clarify it. A lot of such details are missing  in the paper.


- In Section 3.3 In "Attention/Direct Query Prediction Loss" section, a L1 loss is measured between Q^T \times K and the responsibility map. Again why this product should equal the responsibility? Assuming responsibility map is mostly showing the optical flow (given Eq. (1)), why this loss makes sense? again not much motivation is provided.


- In "Contrastive Loss" in Section 3.3, it is not clear how the mask of the object is measured? It is not clear how much annotation is required here. again many important details are missing. 


- Authors mention, they use model [38] to get bounding box and hand joints, together with weak perspective values. However, it is not clear how and where these annotations are used. Do you use just some hand joint locations or entire segmentation? In both losses, how annotations given by different models or ground truth (if anything used) are leveraged?


- In Section 4.1 and 4.2 authors talk about attention-only and embedding-only, while these terms and what they mean are not defined in the text. While the authors mention their architecture has attention head and embedding, it is not clear for attention-only or embedding-only which loss is applied and how they are measured.


- Section Metric: it is not clear what is the difference between All and pair. It seems in both cases the union of hand and object is considered as positive.


- For K-means clustering, authors mention there are 3 clusters, but images show 4 colors (yellow, red, blue, and green). Why there is such a difference?


- In the appendix it is not clarified how responsibility maps are measured.  How the annotations or other info required for measuring the responsibility map are acquired? When using multiple frames authors mention "We then compute the necessary features between these +/- 3 frames and the center frame to produce a single responsibility map". how these "necessary features" are computed? 



- In the appendix it is explained for detecting object parts, a hand query (provided as input) and a background query is used and then subtracted from the responsibility map. Since no background is provided as input, how this is computed? No visualised result of this process is shown. If this approach works better than using PCA or clustering (shown in images 2 and 3), why are these results not shown. On the other hand if PCA or clustering work better, why are the results in tables not computed that way?



- The usage of term responsibility throughout the whole paper is very unconventional. This term has not been used in similar literature. I recommend replacing it with something that better conveys what the method is actually doing.


**Technical:**

- The major drawback of the proposed approach is that it requires hands and objects to be moving and background to be almost stationary, which is due to usage of optical flow. If the hand and object are not moving, there is no optical-flow signal. Also, if the background is moving rapidly together with hand (e.g. when camera is moving) it can produce a high optical flow which interferes with the optical flow of hand and object, so detection of them becomes erroneous. This limits the application of the proposed approach.


- In the contrastive section, it seems the labels for hand and object parts are already provided using [21] and union of responsibility. If such annotation is available one can directly train on labels and no contrastive or semi-supervised training is needed. On a related note, in Table 2 better results are obtained with attention-only. Authors do not explain why this is happening in lines 292 to 295. Is it because you have enough annotations that embedding is not needed anymore?


- Are the test sets composed of new/unseen actions interacting with new objects? It is important to have unseen objects and actions in the test set, otherwise it is not clear if the model really generalises to new actions and unseen objects that exist in the real world. 



- In Figures 3, 4, and 5 in appendix (also images 2 and 3 in the main paper), it is not clear what are the outputs of the model for each one of all, pair, object, and hand given the process described in D.1. In particular, the question is how the model separates hand, object, and background in these cases.


- The responsibility map shows the model can detect the hand and object regions together, but the separation of these two parts is not accurate. It seems the contrastive loss that should separate them is not helping much in separating the two.




**Grammatical/writing Errors:**

A lot of grammatical errors are seen in the paper. Such errors are also seen commonly in Appendix.  					
- line 34, 35 :  needs rephrasing, the phrase reads as if you are segmenting the motion, while you are segmenting the 3D object and not the motion. 

- lines 56, 57: The first paragraph of related work is referring to itself and different parts of proposed work, while it is still not clear at this point what authors are particularly doing. Instead, authors need to talk about more general concepts in the literature and then relate them to their approach. The whole paragraph needs rephrasing. 	
															
- lines 69, 70: the example is not clear enough. The monitor is then used or unused in their case and how does it impact the results? Is it a problem of annotation, or the way the models are trained?

- line 78: weird positioning of "our work falls into this category. "

- line 82 : "Rather than learn" -> Rather than learning

- lines 118, 119: following phrase should be corrected "Formally, we the  responsibility as a temperature-softened softmax per-pixel"

- lines 162, 163: Three models are mentioned that predict embedding Z and it is not clear which one is actually used.

- line 164: "In one cases" -> In one case

- line 164: it is not clear how the phrase in parenthesis relates to the previous phrase.

- lines 168 and 169: the phrases in parentheses are not connected to the main sentence and it interferes with the flow of the text.

- line 184: "these each require " -> each of these require

- Section 3.3.The usage of the term "direct" for the loss is very uncommon. If the loss is supervised, the term supervised should be used.

- line 192: 'after' can be replaced by 'once'. It is more academic.

- The usage of punctuations and other grammatical symbols have been casual throughout the text. The dot is sometimes missed.:
   - line 227: before "We also generate 30K " 
   -line 237: before " This data is sufficiently "
   
- lines 243 to 245: the phrase is erroneous and should be rephrased.

- line 280: "adjacent pairs of video in an image " I guess authors mean "adjacent frames of a video". such writing is very ambiguous and is seen often in the text.

- line 339 "with their hands can have of course be used for surveillance purposes. " -> remove "have"



limitations_and_societal_impact: Authors mention limitations and potential negative societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper tackles the problem of hand and object segmentation. At inference, the model takes a single image and the 2D location of the hand and outputs a hand-object segment. The key idea is to extract supervision from optical flow by contrastive learning. 
main_review: 1. Most hand and object interaction work requires strong supervision, this work however is able to derive supervision from optical flow in videos which is advantageous and extendable to in-the-wild datasets.

2. Both quantitative and ablation studies demonstrate promising results from their approach.
limitations_and_societal_impact: 1. Although the method is tested on two large datasets (100DOH and Epic-Kitchens), the performance on Table 1 is not convincing. Only full comparison is available for Bbox [38] which the paper is not able to match the performance. 

2. Manual labeling for both 100DOH and Epic-Kitchen datasets prevents readers from reproducing the results as they are not publicly available. Also, HO3D (Hampali et al., CVPR 2020) is another recent dataset which contains hand-object segmentation ground truth. I would suggest evaluation for readers to improve understanding the performance of the proposed method.

3. The model requires both a single image and a query point at the hand. I am interested in how this point is obtained and failure cases when the query point is far away from the hand. Otherwise, I found it counterintuitive if the model is limited by the query point.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper addresses a novel problem of segmenting the object which is contacted by hand from a singe image. Since there are no annotations for this task, the authors present an approach to mine pseudo labels from videos based on optical flow. In particular, the authors assume that the contacted object has similar motion fields as the hand. The method works well for many cases except when objects are not moving or not moving in planar planes as also discussed in the paper. The experimental results demonstrate the effectiveness of the approach.
main_review: The addressed problem is interesting and novel. In particular, the self-supervised learning style helps to generalize to a large number of contact objects which is expensive to annotate in supervised learning.

But the method is very simple. Using optical flow to find objects has been studied previously although not for contact object segmentation [1,2,3]. In addition, the method has limitations when objects are not moving as is also discussed in Figure 4. I believe handling static objects will make the paper stronger.

In general, the paper is well written and reads easily. Some minor issues:
(1) In line104, what does "responsibility minus people" mean?
(2) In training, is the number of hands N for different images fixed? Based on my understanding, the responsibility maps should have fixed shapes. But in practice, the number of objects can be very different from image to image.
(3) In line152, you mentioned to select data in which hands are contacting objects. How did you get this information?
(4) Reference [21] is incomplete.

One interesting baseline is missing. In inference, if we are given a pair of consecutive images, we can segment the contacted object using optical flow (as in training). How is the accuracy for that?

Ablation study with the contrastive loss should be provided.

[1]Romero, Javier, Matthew Loper, and Michael J. Black. "FlowCap: 2D human pose from optical flow." In German conference on pattern recognition, pp. 412-423. Springer, Cham, 2015.
[2]Fragkiadaki, Katerina, Han Hu, and Jianbo Shi. "Pose from flow and flow from pose." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2059-2066. 2013.
[3]Wang, Chunyu, Yizhou Wang, and Alan L. Yuille. "An approach to pose-based action recognition." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 915-922. 2013.
limitations_and_societal_impact: Yes
review_title: null

Meta-review: The reviewer ratings of this paper were significantly lower prior to the author response.
Reviewers were concerned about the quality of the writing.
Additional experiments, clarifications and even some simplifications of the proposed method were provided during the author response.
As a result of these actions all reviewers increased their scores after reading the author response.
After accounting for the author response and discussions with reviewers, all reviewers recommend accepting this paper.
It will be important for the authors to address the points and issues raised during the review and author discussion in the final version of this paper. 

The AC recommends accepting this paper.

Decision: Accept (Poster)