COHESIV: Contrastive Object and Hand Embedding Segmentation In Video
Authors: Dandan Shan
Richard Ely Locke Higgins
David Fouhey
Venue: NeurIPS 2021 Poster (2021)
Keywords: Hand Object Interaction
Object Segmentation
Contrastive Learning
Attention
Embeddings

Abstract: In this paper we learn to segment hands and hand-held objects from motion. Our system takes a single RGB image and hand location as input to segment the hand and hand-held object. For learning, we generate responsibility maps that show how well a hand's motion explains other pixels' motion in video. We use these responsibility maps as pseudo-labels to train a weakly-supervised neural network using an attention-based similarity loss and contrastive loss. Our system outperforms alternate methods, achieving good performance on the 100DOH, EPIC-KITCHENS, and HO3D datasets.

COHESIV: Contrastive Object and Hand Embeddings for Segmentation In Video
Dandan Shan" University of Michigan dandans@umich ch edu
Richard E.L. Higgins" University of Michigan relh@umi edu
David F. Fouhey University of Michigan fouhey@umi ch. edu
Abstract
In this paper we learn to segment hands and hand-held objects from motion. Our system takes a single RGB image and hand location as input to segment the hand and hand-held object. For learning, we generate responsibility maps that show how well hand's motion explains other pixels' motion in video. We use these responsibility maps as pseudo-labels to train a weakly-supervised neural network using attention-based similarity loss and contrastive loss. Our system outperforms alternate methods, achieving good performance on the 100DOH, EPIC-KITCHENS, and HO3D datasets.
1 Introduction
We invite you to pick up an object your vicinity and bring it towards you. As you hold the object and move your hand, the object moves coherently, and is roughly rigidly attached to coordinate frame in your hand. While your adult brain does not need this signal and can readily differentiate the object in your hand from both your hand and the background even if you hold your hand still, how did you learn to do this? One answer 42 is that as a human you have time locked modalities of your own hand's configuration via proprioception and vision, and your hand and the object share "common fate" 47). The goal of this paper is to operationalize this idea by learning to segment hand and ind objects (irrespective of name) from single image by learning from video data.
This goal poses many challenges for current computer vision. While there been a longstanding interest in learning segmentation and object individuation from motion cues in vision 41 11. the general case of being broadly able to segment everything has not made substantial progress. While there has, of course, been substantial progress on instance segmentation for particular categories (for instance MaskRCNN (21)) powered by large segmentation datasets [28), the space of categories that one can pick up is vast. Indeed, recent work that aims to reconstruct in-hand objects via segmentation losses (7] reconstructs a few objects by finding category correspondences (e.g., tennis racket masks used for knives). At the same time, while there is work understanding hands, including contact state [32], and detecting boxes around held generic objects [40], there is no work on segmenting objects, much less work that learns from video data. In sum segmenting lots of generic objects is still challenging, and segmenting generic objects in contact no exception.
Our approach makes progress on this challenging problem by assuming a small amount of knowledge about humans. We assume we can estimate landmarks on hands (37), identify whether they are holding something [40]. as well as identify which pixels belong to people (23). This small amount of information guides understanding of the the deluge of information from optical flow [3]. Rather than segmenting the full 3D motions of 3D objects from the flow, instead only have to identify whether the flow is better explained by a moving hand or a background. Moreover, by knowing about an ubiquitous object (the hand), we show we can learn about many less ubiquitous objects.
Equal contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

We implement our solution with a network, named COHESIV, that takes an RGB image and one 2D location on a hand as inputs to segment the hand and the object that the hand is interacting with (Section 3). At test time, COHESIV maps the input image to an embedding space with CNN [38]; this embedding is processed with lightweight heads to produce per-pixel detectors at each hand, and feature maps for objects that can be queried to produce a heatmap. At training time, we use the motion with nearby frames to derive signal. We define a responsibilityy map for a moving hand as the relative goodness of fit on optical flow for planar motion model in comparison to background model. These responsibilities power two losses: a similarity loss that directly supervises per-hand segments; a contrastive loss [3] that encourages in grouppp affinity as well as separates embeddings among people, objects, and background via pseudo-labelt labels.
We train and validate on video data of humans engaged in complex behaviors using subsets of the 100 Days of Hands (100DOH) [40], EPIC-KITCHENS
(EPICK) (9 TQ, and HO3D [8] datasets.
We compare with alternate methods that range from fully-supervised bounding boxes 40 to basic motion eues from optical flow 4 to saliency (51]. We show that our weakly-supervised method is
comparable to the supervised bounding box detector method, while outperforming flow and saliency methods.
2 Related Work
Our work aims to segment hands and in-hand objects from a single image and thus interacts with a variety of related areas, ranging from: the domain we work on (understanding hand-object contact), the signals we use (common fate with optical flow), and the methods we use to extract meaning from this signal (contrastive learning). Our approach differs from each by using contrastive learning to extract supervision from optical flow in order to segment out objects people are holding,
Understanding hands in contact with objects has long been of interest. In addition work focused on finding hands [3 31 44). there has been considerall work on reconstructing hands (for instance with known shape model like (66) along with outputs such as poses 12 or 3D meshes 20 19 7
While these approaches can often produce meshes, they usually rely on strong supervision. In contrast, our approach derives its signal from using a small amount of information to make sense of optical flow in videos. While our approach resembles work on human-object relationship detection [5]] 17 our relationship is specifically physical contact (e.g., in HOI 45 a person may be using a monitor they are holding a mouse while we would only segment the monitor if it were physically grasped). Robots accomplish similar object-in-contact tasks [14), but can be challenged by the need to collect large scale interactions. The most related work is understanding objects in contact, ranging from detailed contact models like (5) to bounding boxes like [40) Our work produces a richer per-pixel segmentation compared to [40] while also requiring less supervision.
We extract this information from optical flow, which has been known as a signal for perceptual organization since the Gestaltists [47] and Gibson [16]. Optical flow at inference time has been used, for instance for doing motion segmentation ( ([4 6 35| 26 255 among many others). When coupled with learning machinery, the optical flow provides a supervisory signal that can be absent at inference time; our work falls into this category. This signal has been used to learn representations e.g., for pure recognition 34. but our approach is most related to its use for perceptual organization, for instance learning correspondence [46] 24 48 or boundaries 50 27]. Our work most similar to works learning boundaries but focuses on a specific type of boundary those of hand-held objects. Rather than learning to segment everything, we learn to group hands and the objects they hold. To the best of our knowledge, there is no work that does this; the closest is 4 that finds important objects for an egocentric user based on gaze (as opposed to what object is in a person's 's hand).
Our approach for learning uses attention for prediction and contrastive learning for grouping. We use attention as a means of relating query points against an entire image. This mechanism is common in methods that solve visual question answering [ 30] though our use of attention aims to instead discover unknown object extents. In this regard, our work is loosely related to 49) an attention- based visual question answering method. Contrastive learning has become standard formalism for self-supervised learning, including remarkable performance g 33
We use a formulation inspired by [44], which uses saliency as grouping signal on Internet image data. In our method, we use optical flow, hands and people instead. Our approach also combines self-supervvied learning for per-pixel embeddings with supervision that aims to directly predict which object goes with the hand.
2

---

COHESIV Architecture
Responsibility Generation
Location
Hand/Q
Latent
M
M.lij.JJ
image
Eq. (
Prediction
Responsibility.,
Flow
Object
Total Loss
Embeddding Loss
Respansibiily Loss
Figure I: COHESIV architecture (left) and responsibility generation (right). First, homography capturing hand movement is between hand vertices in frame and hand vertices transformed by flow from to t+i Then, responsibility for each hand and the background is computed by Equation [] COHESIV takes an RGB image and hand location as input, uses a UNet to get embeddings for each pixel, conducts contrastive losses between objects, people, and background the embeddings, and finally inputs embeddings to hand and object branches to relate query points to held-objects in producing a predicted responsibility mask.
Method
Our approach learns to map the image and an on-hand query point a prediction of what parts of the scene move with the hand at the query point. While the system requires single image at test time, makes use the rich temporal signal found video for training. At the core of our approach is the notion of responsibilities for hands, inspired by the notion of responsibilities when fitting GMM [15]: the responsibility of pixel for hand is how well that hand explains the pixel's motion compared to other hands and the background (Section 3.1
At test time, we aim to predict the hand and hand-held object masks for the queried hand. Our network (Section 3.2) predicts this by mapping the image to shared embedding, which is in turn converted into per-pixel object features that are attended to by per-pixel hand detectors. At training time, the responsibilities provide the basis (along with the pixel locations of humans) for set of losses (Section 3.3) that we use to train our network, using both direct supervision on the outputs with responsibility maps as well as a constrastive loss on the embeddings.
Throughout, assume access systems that estimate between-fiee frame optical flow, landmarks on the hand, and which pixels belong to people. We use optical flow from RAFT [43], projected hand joints from FrankMocap [37 (to get set of points the hands), and people masks from an off-the-shelf people segmentation system 23
3.1 Responsibility
formalize the notion of synchronous motion (or common fate [47] for hand and in-hand object via the notion of responsibility. Given an optical flow map O E H x2 and set of N hands H. aim produce N responsibility maps R TR x (NN1) how well each pixel is explained by each hand's motion model or the background. For the kth hand, we formally model the responsibility as a temperature--sftennd softmax per-pixel, or:
with D R..j that explain
exp; (sp;(-dsp (-da(O
Rijik=
exp, (-dBG(OO...) + Σ exp, (-dx((Oi..))
where (x),(x)) cxp(-x/t) is an exponential with tunable temperature (set to based on a small held-out set) and dya and de compute distances between an optical flow vector and a model (namely a background model and model for the kth hand respectively). Equation DaatiionnTrrre requires building and evaluating distances between the movement seen in flow vector E R²2 We treat the background as static and thus the model (o) simply ||o||2- This not the best model for egocentric data, but we find it to be effective, potentially due to the high frame rate of the dataset 99 we use.

---

RGB
Responsibility
RGB
Responsibility
RGB
Responsibility
Figure 2: Responsibility maps. Images and responsibility maps from 100DOH (top row) and EPICK (bottom row).
Our hand models (o) assume a planar motion because it is simple solve for, can handle out of image-plane rotation, and is reasonable approximation at the distances and timescales the data shows. This entails fitting homography M2 R3x3 on hand landmarks. Supposing proj(-) converts homogeneous to normal coordinates ([z, y, 2] + [z//,, z. v/2D, then the landmarks should satisfy (i + J + 02) 1e]T" proj MM j. 1]²)) for a pixel i with flow o. We tried with a variety of options, but found simply fitting M, to the estimated flow at FrankMocap [37] vertices to be most effective. Using (37 to provide correspondence proved worse, likely since the resulting hand motion model also included disagreement about the landmark locations in each frame. Then, given model M, and pixel i with flow o, the models' prediction is proj [i, IJ²).. We define the distance as the difference between the actual flow and the predicted flow,
dx (O,,) = 1|| [|:"]" + O., - proj(M, 112)
While simple, the planar model has a counter-intuitive catch. By handling out of plane rotation, the planar model will produce many points that match optical flow by accident specially for static & [i. 1)²T) often has zeros in the image. While a learning system could learn to
cameras: pM ignore these accidental matches by training on large dataset, accelerate this by averaging it out: we calculate responsibility maps to the next and previous frames, and then average them.
Techniques: We estimate optical flow with RAFT 43 and hand vertices with FrankMocap [37 FrankMocap fuses hand detector [4) with network that predicts 2D and 3D hand joints. This information is used to compute homographies and then responsibility maps. Once responsibilities are computed, they serve as pseudo-labels at training time and are then discarded. At test time, we aim to predict hands and held-objects for new images.
Data Selection and Implementation Details: The data we use challenging non-scripted videos that include vast amounts of non-interaction. We select our training and validation data so that the network primarily sees hands that are visible and in contact. We select clips where hands appear in 10 consecutive frames as well as hands are moving within the clip. In 100DOH, the clips are detected as visible hands holding reasonably sized objects using 40 We note that we only have to do this selection since the data used is large and non-scripted. In EPICK, we use their action segments. (see supplemental for more details)
3.2 Architecture
Our approach aims to take an image E Hx XXxWxx, query point p [z, y]. and produce a segmenta- tion of the hand and object that the hand at p is in contact with. Our architecture produces query feature map Q E HW XA representing per-hand detectors and a key feature map K E R x x representing object features. At test time, one can extract Qæ.g.: and produce detection score at any i by q" KR,,;- Thus, a forward pass can be used to parse multiple hands by extracting different locations.
We produce Q and K from common backbone that produces embedding Z R XX We predict the embedding Z with a standard U-Net-style 38) network with a SE-Net 22 see-eesne--50- 4d) backbone and ImageNet [39] pretrained encoder weights. Then, use two lightweight paths (two 3 x Conv layers) from to heads Q and K (see the supplemental for full description).
The use of an intermediate embedding plus the attention heads enables the heads to handle integration of hand-specific information (e.g., given this hand, this particular object goes with it) while the embedding captures how people, object, and background pixels are distinct. The asymmetry via two

---

heads also enables producing the object the hand is holding rather than only the hand: for a single embedding, the hand must have maximum self-similarity if the data normalized.
Hand Detector: One can obtain P via a hand detector, but this detector can also be built into COHESIV. We have trained hand detector on its own. We trained another head (similar to head Q and K, see the supplemental for details) to instead map the latent space to a ID heatmap for hand center location. Like pose estimation techniques, this model is trained to minimize the MSE to a Gaussian placed at the center of the hand box reported by 40).
frozen backhone up to Z, enabling it to run entirely
Inference:
At inference time, the model processes a single image to return embeddings that separate people, objects, and background. When optionally given query point the hand, it also predicts the hand and object segmentation masks for that query, enabling an agent to investigate any potential point of interest. For objects, first average 500 embeddings produced from training images wherever our pseudo-labels are "object". We then calculate a cosine similarity between this "average object embedding" and the predicted embedding for the inference image. We take this similarity map and the predicted responsibility attention map (i.e., q" K) and fit two thresholds on validation data, such that object pixels need to both have "objectness" above a threshold and predicted responsibility above another threshold. Hand mask predictions are created analogously, except an "average person embedding" is created from "person" pseudo-labell and used in combination with an embedding and responsibility threshold.
3.3 Losses
We train with two losses that encourage different behaviors for the embedding and attention heads: the responsibility loss direct optimizes the output while the embedding loss encourages pixels on hand-held objects to group together. Both require dot products between a full embedding space and handful of vectors opposed to expensive all-pair 2²22 tensor. While progress on both losses is often in the same direction, optimally minimizing the proposed losses on the same feature is impossible with multiple hands the embedding loss is satisfied if all hand-held objects are identical while the responsibility loss is not satisfied until each hand-held object is separated. We therefore jointly minimize these losses on different features.
Responsibility Loss: Our first loss directly supervises responsibility: given a point p = [x. y] on a hand (blue dot in Figure 3). we compute Q Ki for all i and train the network to match the responsibility. After computing the prediction, we clamp to [0,1], meaning that the system can make an object not match query point by making the vectors orthogonal (a large and vast space) rather than polar opposites (a specific location). We minimize a per-pixel Smooth-L1 loss (with 0.5) between the predicted value and the computed responsibility map.
Embedding Loss: We also use a contrastive loss, based on [4], that encourages coherent embeddings regardless of hands. These use a set of masks for people, objects, and background via mask operations on 23 and the union of responsibilities. Suppose p,, zo, and d represent the average embeddings from Z for the people, objects, and background masks. Then the contrastive loss aims to ensure that
image embeddings within these masks are all similar to one another, and that the average embedding across each of these masks is far from the average embedding of the other two groupings. The contrastive loss for hand-held objects at pixel i, j is the negative log-likelihood
exp (z. +e
logg exp (z.) + exp (i.) + exp Zx))) +
where 10-3 aids numerical stability. This loss is evaluated only for pixels i, that are members of the hand-held object pseudo-label. The same loss is analogously defined for people and background, respectively replacing z in the numerator for zp z and evaluated only for pixels i. in the person or background pseudo-label.
Implementation: We weight the attention loss and embedding loss as 10:1. People masks are outputs from 23]; object masks are responsibility masks that have had the person segmentation subtracted from them, and thresholded; background masks the remaining pixels.
5

---

Image
Embeddings
Attention
Hands + Objects
All Masks
Figure 3: Visualizing COHESIV on 100DOH (rows 1-3), EPICK (rows 4-6), and HO3D (row 7).
Given the RGB image and on an- query point (blue dot) input (col 1). the embeddings column (col 2) shows the contrastive visual embeddings visualized using PCA dimensionality reduction (d = 3); the attention column (col 3, viridis colormap) shows the predicted responsibility of each pixel's movement related to the query point; the hands and objects column (col 4) shows the predicted segmentation masks for the given pair (green for hands, blue for objects, and cyan for hand object overlap); and the masks column (col 5) shows all predicted segmentation masks in red.
Experiments
We evaluate our approach on contact segmentation. Given an image and query point (ær, y) on a hand, goal is find the extent of the hand and potential in-contact object. We test 100DOH (40), EPICK (9), and HO3D 18]. We first compare against baselines and alternative methods and then ablate model components understand how design decisions impact performance. We qualitatively evaluate our methods and show how learning per-pixel embedding helps segmentation. Finally, we evaluate hand detector branch that makes the model usable without pre-detected hand location.
4.1 Experimental Setup
Datasets: We conduct experiments on 100DOH 40 EPICK 9 and HO3D 18 We first generate 10-frame clips from these video datasets following the original dataset splits. Given each 10-frame clip, we compute and select responsibility maps from the middle (4th, 5th, 6th and 7th) frames

---

Handj
Object,
Pairy
Pair;
All
Pi
GT
GT
GT
GT
GT
+
ookdre
Prod
Pred
Pred
Prod
Prod
+
P
Figure Four evaluation tasks. The first row illustrates that a hand and object annotation are added to form pair annotation. Then all pairs in an image are added to form all annotation. COHES/V inputs the RGB image on the left and query point pi. producing outputs shown the bottom row for Hand, and Object,, which are then combined into Pairj before finally all pairs are combined into All.
for use as training data (see supplemental for details). For each hand, we choose a query point by averaging the vertices produced by the 3D hand mesh prediction in [37].
In the 100DOH 40 dataset, we generate 80.4K 10-frame training clips from the "make drinks" and "make food" genres. 100DOH [40] is collected from large variety of public YouTube videos containing hand and object interaction from everyday life with offensive content intentionally removed. Specifically, we select from the "make drinks" and "make food" videos due to the variety of objects and the tendency of these objects to be roughly rigid. In EPICK (9), we generate 30K 10-frame training clips from action segments. EPICK [9] is an egocentric video dataset that captures daily activities the kitchen and only shows peoples' hands and home environments (also without offensive content). The HO3D [8] dataset contains sequences of people manipulating different objects in laboratory settings, and has ground truth hand and object segmentation masks for evaluation. We generated 8K clips from HO3D, using 5K of these clips for training.
Annotations for Evaluation: To evaluate our approach on the 100DOH and EPICK datasets, we annotate hand and held object masks with a 2-step manual labeling task via a labeling platform. Given a red dot on each hand, we annotators label the hand mask; then, given the hand mask, we ask annotators to label the in-contact object mask if it exists. This gives 1, 1,123 test and 482 validation images for 100DOH
and 1, 1,169 test and 437 validation images for [9]
Training Details: We trained the COHESIV model and all ablations on GTX 1080 GPUs. We use the AdamW (29] optimizer, with initial learning rate of 10-2 and batch size of 10. We used learning rate scheduler that cut the learning rate in half after 5 epochs without validation loss decrease. We used early stopping and halted training if the validation loss did not decrease after 10 epochs.
We evaluate using Intersection over Union (IoU) on four different tasks that change the definition of ground-trute positive: (1) All: any hand held object is positive, which evaluates how well method can identify all hands and hand-held objects (akin to semantic segmentation); (2) Pair: the hand in question and any object it holds are positive, which evaluates how well a system can individuate hands and their held object (akin to instance segmentation): (3) Hand: only the hand in question is considered positive; (4) Object: only the object the hand is holding (if any) is considered positive. COHESIV can solve all four tasks. Other methods are sometimes unable produce all tasks, which denote with "_" We calibrate thresholds for binarization using grid search on the validation set.
Evaluation Metrics:
4.2 Contact Segmentation
We use this setup to compare our approach qualitatively and then against baselines. We then ablate the model and test the automatic detection system built on COHESIV.
Qualitative Results: Figure 3 visualizes COHESIV outputs. Our system has the ability to distinguish hands via the query point. The queried hand of high probability while the unrelated hand of low probability. At times, large objects are poorly predicted, with predicted responsibility decreasing for pixels at points distant from the hand. In such instances, using the embedding to separate objects from people and the background is helpful.
7

---

Table 1: Comparing COHESIV against baselines and supervised method on the 100DOH and EPICK datasets using mloU (%). Our proposed model outperforms the baselines and is comparable to the supervised bounding box method.
100DOH
[40]
EPICK [9]
All
Pair
Hand
Obj
All
Pair
Hand
Obj
COHESIV
51.9
46.5
53.6
29.3
43.2
42.1
60.7
19.5
Saliency [51
25.2
20.1
8.6
17.0
21.6
15.9
6.0
11.7
Flow 43
29.3
21.5
12.9
12.1
15.4
11.9
6.2
6.6
Thresholded Responsibility
44.5
37.0
42.9
30.0
Supervised Bounding Box 40
56.9
47.0
56.5
34.9
54.3
44.8
53.8
34.4
Table 2: Ablations of COHESIV, evaluated on the 100DOH and EPICK datasets using mloU(%). It worth noting that the solely attention based model cannot readily separate the co-occurring motion of hand and objects.
100DOH
40
EPICK 9
All
Pair
Hand
Obj
All
Pair
Hand
Obj
COHESIV Attention-Omi Only Embeddings-Only
51.9
46.5
53.6
29.3
43.2
42.1
60.7
19.5
42.8
40.0
38.1
37.8
25.7
18.3
13.2
22.9
30.0
20.8
24.6
14.4
COHESIV w/ ResNet Backbone
45.8
41.2
48.1
25.2
39.8
39.1
55.2
17.9
COHESIV w/ Predicted Query
47.7
42.8
47.8
28.1
40.0
38.6
55.1
19.4
Table 3: Comparing COHESIV against baselines and a supervised method on the 100DOH and EPICK datasets using mloU (%). Our proposed model outperforms the baselines and is comparable to the supervised bounding box method.
100DOH
EPICK
All
Pair
Hand
Obj
All
Pair
Hand
Obj
COHESIV
51.9
46.5
53.6
29.3
43.2
42.1
60.7
19.5
Saliency
25.2
20.1
8.6
17.0
21.6
15.9
6.0
11.7
Flow
29.3
21.5
12.9
12.1
15.4
11.9
6.2
6.6
Thresholded Responsibility
44.5
37.0
42.9
30.0
Supervised Bounding Box
56.9
47.0
56.5
34.9
54.3
44.8
53.8
34.4
Comparisons: To understand the capabilities of COHESIV, we compare it against baselines and supervised method.
(1) Saliency: We use recent saliency detection system 51 which aims to identify regions of interest in an image, and serves as a benchmark for how brgground/backrrommg segmentation approaches might perform on our task. (2) Flow: We use RAFT 43 to estimate the flow between adjacent pairs of frames in the video, then fit a threshold across validation annotations and treat all flow above this threshold as positive prediction. (3) BarehholdddRResonnibbnn
We directly binarize the responsibility pscudo-labels using a threshold chosen from the validation set. (4) Supervised Bounding Box: Rectangular areas from the supervised hand-object bounding box detector in [40]. As trained on over 132K annotated samples from the training sets of both 100DOH [40] and a variety of egocentric datasets, including [9] we do not expect to match its performance. We report it as a comparison, however, for context. Of course, by virtue of our system producing segmentations, can in principle get refined segmentations (although box is often good approximation for the object).
The Saliency and Flow baselines aim to serve as alternate explanations for the results seen (e.g., that the approach just picking up on a salient object or moving pixels), both of which are generated for the entire image and then evaluated directly against all four targets (All/aai/Hand/Obeecc)
8

---

Table 4: Ablations of COHESIV, evaluated on the 100DOH and EPICK datasets using mloU(%)) It
worth noting that the solely attention based model cannot readily separate the co-occurring motion of hand and objects.
100DOH
EPICK
All
Pair
Hand
Obj
All
Pair
Hand
Obj
COHESIV Attention-Oni Only Embeddings-Only
51.9
46.5
53.6
29.3
43.2
42.1
60.7
19.5
42.8
40.0
38.1
37.8
25.7
18.3
13.2
22.9
30.0
20.8
24.6
14.4
COHESIV w/ ResNet Backbone
45.8
41.2
48.1
25.2
39.8
39.1
55.2
17.9
COHESIV w/ Predicted Query
47.7
42.8
47.8
28.1
40.0
38.6
55.1
19.4
Image
Embeddings
Attention
Hands + Objects
All Masks
Image
Responsibility
Image
Responsibility
Image
Responsibility
Figure 5: Two types of failures. The first shows the prediction of an amorphous halo around the hand rather than the grasped bottle. The second row shows missing interior to the grasped tinfoil tray. The last shows three failures of computed responsibility maps related to distance the hand, hand occlusion, and accidentally similar motion between adjacent objects.
Ablations and Extensions: We include number of modifications to COHESIV to understand the importance of various components in Table We test models that train with only attention loss embedding loss, followed by one that uses a ResNet backbone rather than SE-NET (22] Finally, test extension that uses predicted hand point from system rather than locations from existing hand detector 37 as mentioned in Section 3.2
Quantitative Results: In Table we present how our model compares to similar baseline methods. Our method approaches the performance the supervised approach in 40) and surpasses on the EPICK Hand task. Our systems outperform simpler approaches which find salient objects use optical flow estimations above a threshold. We conducted a limited evaluation on the HO3D dataset (where All is the same Pair due to the absence of multiple hands). Our approach obtains good performance: 49.1 (Pair), 49.6 (Hand), 30.6 (Obj), just 8.4 (Pair), 4.2 (Hand), and 8.5 (Obj) points short of Supervised Bounding Box [40].
In Table 4 we demonstrate the value of our hybrid attention and embedding model, by comparing against ablated versions. Jointly training with both generally improves results on 100DOH and EPICK. Our extended hand-detection branch works slightly worse than using an existing hand detector (37 but shows comparable performance and enables single forward pass.
4.3 Analysis and Limitations
While we cannot directly evaluate the learned embeddings, we show results throughout the paper using PCA. We fit PCA on the embeddings, H W x (dimension 64 here) per-pixel feature vectors, and convert the resulting dimensions into RGB space after z-scoring.

---

Our system is by no means perfect and we show some failure modes in Figure 5 One is in our system. As shown in the first row, our model find larger and amorphous blob but instead of the detailed grasped bottle. The second row shows that for large non-rigid objects, it is hard to capture the whole area. The other one is in the responsibility. Generally, when the motion very small and affected by shadow and occlusion, it is hard to get a good responsibility map. The third row shows the responsibility map quality is affected when the object part is distant from the hand (less similar motion with the hand), when the hand is heavily occluded, and when there are other objects in the image move similarly with the hand.
In addition to standard mispredictions by deep learning systems, some failures are intrinsic to the signal we use: if the object is unlikely to move, then our system is unlikely to see it as a hand-hell object. We see this as a challenging next step, and problem of great importance.
5 Discussion
We present a system for segmenting hands and hand-held objects by learning from motion in videos through weak supervision. We generate responsibility maps as pseudo-labels for learning and incorporate contrastive learning to distinguish held objects from people and background in images. Our work and possible extensions, such as work that separates the embeddings of objects classes, make possible research that can readily address the long-tail of object classes that exist in the real world, outside of prescribed datasets. Our method an attempt to learn about objects from hands using common motion signals. We hope it serves useful next step in developing a deeper understanding of the interrelationship between hands and objects.
Systems that learn from signals or correspondence within videos (rather than manual labels from human) have the potential to see and discover portions of the visual world that have been overlooked by researchers. Using hands as a medium to explore adjacent objects in the natural world a promising way to use manipulation to learn physical relations. Starting from hands and generalizing to the unlabelled object segments of the world can positively impact many downstream applications that need a more real world understanding of object classes than is provided by a fixed number of classes.
Broader Impact:
On the other hand, our system can have negative downstream consequences. First, while the data in public videos is often closer to reality than Flickr images, deep learning systems can replicate and amplify the biases observed in these videos. While there has been lots of effort towards making datasets more reflective of everybody, there is still lot of progress to be made. This requires both careful reflection during data curation and monitoring during deployment. Second, premature deployment of these systems without proper oversight can have negative affects, especially for people not rell-sepresented.
Acknowledgments: We thank Mohamed El Banani, Karan Desai, Sarah Jabbour. Nilesh Kulkarni, Max Smith, and Shengyi Qian for the feedback and support. This material is based upon work supported by the National Science Foundation under Grant No. 2006619.
References
[1] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433, 2015.
[2] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. arXiv preprint arXiv: /906./990.00999 009/0, 2019.
[3] Sven Bambach. Stefan Lee, David Crandall, and Chen Yu. Lending hand: Detecting hands and recognizing activities in complex egocentric interactions. In ICCV, 2015.
[4] Gedas Bertasius, Hyun Soo Park, Stella X Yu, and Jianbo Shi. Unsupervised learning of important objects from first-person videos. In Proceedings of the IEEE International Conference on Computer Vision, pages 1956-1964, 2017.
[5] Samarth Brahmbhatt, Chengeheng Tang. Christopher D. Twigg, Charles C. Kemp, and James Hays. ContactPose: A dataset of grasps with object contact and hand pose. In The European Conference on Computer Vision (ECCV), August 2020.
[6] Thomas Brox and Jitendra Malik. Object segmentation by long term analysis of point trajectories. In European conference computer vision, pages 282-295. Springer. 2010.
[7] Zhe Cao, Iija Radosavovic, Angjoo Kanazawa, and Jitendra Malik. Reconstructing hand-object interactions in the wild. arXiv preprint 201.:2201..09856, 2020.
10

---

[8]
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. arXiv preprint
arXiv:2011.108 1.Q01.11566.. 2020.
[9] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling egocentric vision: The epic-kitchens dataset. European Conference on Computer Vision (ECCV), 2018. [10] Dima Damen, Hazel Doughty. Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos. Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. The epic- kitchens dataset: Collection, challenges and baselines. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2020.
[11] Achal Dave, Pavel Tokmakov, and Deva Ramanan. Towards segmenting anything that moves. In Proceed-
ings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0-0, 2019.
[12] Bardia Doosti, Shujon Naha, Majid Mirbagheri, and David Crandall. Hope-net: A graph-based based model for hand-object pose estimation, 2020.
[13] Alexey Dosovitskiy, Philipp Fischer, Jost Tobias Springenberg. Martin Riedmiller, and Thomas Brox. Dis-
criminative unsupervised feature learning with exemplar convolutional neural networks. IEEE transactions on pattern analysis and machine intelligence.
(119):734-1727 2015.
[14] Victoria Florence, Jason Corso, and Brent Griffin. Robot-supervised learning for object segmentation. In
2020 IEEE International Conference on Robotics and Automation (ICRA), pages 1343-1349. IEEE, 2020. [15] Jerome Friedman, Trevor Hastie, Robert Tibshirani, et al. The elements of statistical learning, volume 1. Springer series in statistics New York. 2001.
[16] J. Gibson. The ecological approach to visual perception, Boston: Houghton Mifflin, 1979.
[17] Georgia Gkioxari, Ross Girshick, Piotr Dollar, and Kaiming He. Detecting and recognizing human-object
interactions. In CVPR, 2018,
[18] Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. Honnotate: A method for 3d annotation of hand and object poses. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3196-3206, 2020.
[19] Yana Hasson, Bugra Tekin, Federica Bogo, Ivan Laptev, Mare Pollefeys, and Cordelia Schmid. Leveraging photometric consistency over time for sparsely supervised hand-object reconstruction. In CVPR, 2020.
[20] Yana Hasson, Güi Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J. Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. In CVPR, 2019.
[21] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE
international conference on computer vision, pages
2017.
[22] Hu, Li Shen, and Gang Sun. Squeeze-and-exciin -ecittextttaiion networks. In Proceedings of the IEEE conference
on computer vision and puttern recognition, pages 7132-7141, 2018.
[23] Vladimir Iglovikov and Alexey Shvets. Ternausnet: U-net with vggll I encoder pre- pre-trained on imagenet for
image segmentation. arXiv preprint
2018.
[24] Allan Jabri, Andrew Owens, and Alexei A Efros. Space-time correspondence as a contrastive random walk.
arXiv preprint ar arXiv:2006.1 14613. 2420.
[25] Zihang Lai, Erika Lu, and Weidi Xic. Mast: A
self-supervised tracker. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6479-6488, 2020. [26] Siyang Li. Bryan Seybold, Alexey Vorobyov, Xuejing Lei, and C-C Jay Kuo. Unsupervised video object segmentation segmentation with moteon-based bilateral networks. In proceedings of the European Conference on
Computer Vision (ECCV), pages 207-223, 2018
[27] Yin Li, Manohar Paluri, James M Rehg, and Piotr Dollár. Unsupervised learning of edges. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 1619-1627. 2016.
[28] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco; Common objects in context. In European conference on computer vision, pages 740-755, Springer, 2014.
[29] Ilya Loshchilov and Frank Hutter.
Decoupled weight decay regularization. arXiv preprint
arXiv:1711.05 2017.
[30] Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Hicrarchical question-image co-attention for visual question answering. arXiv preprint arXiv: 1606.6060000 00061, 2016
[31] A. Mittal, A. Zisserman, and P. H. S. Torr. Hand detection using multiple proposals. In BMVC, 2011.
[32] Supreeth Narasimhaswamy. Trung Nguyen, and Minh Hoai. Detecting hands and recognizing physical
contact in the wild. 2020.
[33] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive
coding, arXiv preprint arXiv:1807.00 800.034888 2018
[34] Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Darrell, and Bharath Hariharan. Learning features by watching objects move. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2701-2710, 2017.
[35] Jordi Pont- Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine- Morkine-Hornung... and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:1704.00677 2017.
[36] J. Romero, D. Tzionas, and M. J. Black. Embodied hands: Modeling and capturing hands and bodies
together. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia). 36(6), 2017.
[37] Yu Rong, Takaaki Shiratori, and Hanbyul Joo. Frankmocap: Fast monocular 3d hand and body motion capture by regression and integration. arXiv preprint arXin:2008.08322, 2020.

---

[38]
Olaf Ronneberger, Philipp Fischer, and Thomas Brox, U-net: Convolutional networks for biomedical image segmentation. In International Conference Medical image computing and computer-assisted intervention. pages 234-241. Springer, 2015.
[39] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge.
International journal of computer vision, 115(3):21 1-21221122222 2015.
[40] Dandan Shan, Jiaqi Geng, Michelle Shu, and David F Fouhey. Understanding human hands in contact at internet scale. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9869-9878, 2020.
[41] Jianbo Shi and Jitendra Malik. Motion segmentation and tracking using normalized cuts. In Sixth International Conference on Computer Vision (IEEE Cat. No. 98CH13277), pages 1154-1160. IEEE,
1998.
[42] Linda Smith and Michael Gasser. The development of embodied cognition: Six lessons from babies. Artif.
Life, 11(12)):3-30, Jan. 2005.
[43] Zachary Teed and Jia Deng. Raft: Recurrent all-pairss field transforms for optical flow. European Conference on Computer Vision, pages 402-419. Springer, 2020.
[44] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Lue Van Gool. Unsupervised semantic segmentation by contrasting object mask proposals. International Conference on Computer Vision, 2021.
[45] Tiancai Wang, Tong Yang. Martin Danelljan, Fahad Shahbaz Khan, Xiangyu Zhang, and Jian Sun. Learning human-object interaction detection using interaction points. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 41 16-4125, 2020.
[46] Xiaolong Wang, Allan Jabri, and Alexei A Efros. Learning correspondence from the cycle-consistency of time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2566 2576, 2019.
[47]
Max Wertheimer. Laws organization perceptual forms. 1938.
[48]
Jiarui Xu and Xiaolong Wang. Rethinking sulf-supervised correspondence learning: A video frame-level similarity perspective. arXiv preprint arXiv:2103.17266
2021.
[49] Zichao Yang. Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for image question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition,
pages 21-29, 2016.
[50]
Zhenheng Yang. Peng Wang. Yang Wang, Wei Xu, and Ram Nevatia. Lego: Learning edge with geometry all once by watching videos. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 225-234, 2018.
[51] Ting Zhao and Xiangqian Wu. Pyramid feature attention network for saliency detection. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2019.
June
12


Reviews:
[{"review_id": "rammwVi5JBw", "reviewer": "Reviewer_BKyv", "summary": "This paper tackles the problem of hand and object segmentation. At inference, the model takes a single image and the 2D location of the hand and outputs a hand-object segment. The key idea is to extract supervision from optical flow by contrastive learning. ", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Most hand and object interaction work requires strong supervision, this work however is able to derive supervision from optical flow in videos which is advantageous and extendable to in-the-wild datasets.\n\n2. Both quantitative and ablation studies demonstrate promising results from their approach.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. Although the method is tested on two large datasets (100DOH and Epic-Kitchens), the performance on Table 1 is not convincing. Only full comparison is available for Bbox [38] which the paper is not able to match the performance. \n\n2. Manual labeling for both 100DOH and Epic-Kitchen datasets prevents readers from reproducing the results as they are not publicly available. Also, HO3D (Hampali et al., CVPR 2020) is another recent dataset which contains hand-object segmentation ground truth. I would suggest evaluation for readers to improve understanding the performance of the proposed method.\n\n3. The model requires both a single image and a query point at the hand. I am interested in how this point is obtained and failure cases when the query point is far away from the hand. Otherwise, I found it counterintuitive if the model is limited by the query point.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "7G1yTwumTDM", "reviewer": "Reviewer_mYxT", "summary": "This paper aims at segmenting hand and object regions given an input image, optical flow, and location of hand (for some pixels), using a contrastive approach and measurement of features called \"Responsibility\".  ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is not well-written and the content do not follow naturally. It looks like a draft at this point. There are many ambiguities, grammatical errors, and lack of explanations throughout the paper (see details below). I read the paper twice to make sure I am not missing something and also checked the appendix. It is still not very clear to me many details of the paper.\n\n**Technical / Clarity:**\n\n- Regarding Eq. (1), which is measuring \"Responsibility\", what is the motivation and what is it computing? There is not much explanation on what is the goal of it. The first question is what is h_k? It is only at Eq. (2) that this becomes somewhat clear. Is h_k indeed proj(M_k)? if so, it should be clarified in Eq. (1). Now assuming proj(M_k) is indeed the answer, if M_k is the output of a model, the best predicted responsibility is to predicts the optical flow per-pixel. The question is then how this corresponds to a particular hand and why this is useful? In the denominator of Eq. (1) there are multiple hands for different h_k. How this equation differentiates different hands, if each hand output is predicting the optical flow?\n\n\n- On a related note, in lines 128 and 133 M_k is defined as both homography and model. Which one is it? In line 129 the M_k is defined as optical flow plus current index location, while in Eq. (2) the same M_k is compared only to the optical flow (and not the sum of optical flow and index). These two definitions are not the same, so which one is correct? Also it is not clear how M_k and h_k related to the Z (common embedding), Q (hand features), and K (object features) given by the model.  In architecture section, this is not clarified. how M is obtained? Is it an output of the model or estimated differently?\n\n\n- Authors mention in lines 158 to 160 that at test time given a query q, a score q^T \\times K_{i,j} can be measured. What does this score between hand and object feature maps indicate and how it is used for hand region or object region segmentation? Q^T \\times K should be mostly zero, if the two heads truly separates the activation for hand and object, otherwise if the multiplication is not-zero it indicates the model is not correctly separating them. The description at lines 176 to 179 also does not clarify it. A lot of such details are missing  in the paper.\n\n\n- In Section 3.3 In \"Attention/Direct Query Prediction Loss\" section, a L1 loss is measured between Q^T \\times K and the responsibility map. Again why this product should equal the responsibility? Assuming responsibility map is mostly showing the optical flow (given Eq. (1)), why this loss makes sense? again not much motivation is provided.\n\n\n- In \"Contrastive Loss\" in Section 3.3, it is not clear how the mask of the object is measured? It is not clear how much annotation is required here. again many important details are missing. \n\n\n- Authors mention, they use model [38] to get bounding box and hand joints, together with weak perspective values. However, it is not clear how and where these annotations are used. Do you use just some hand joint locations or entire segmentation? In both losses, how annotations given by different models or ground truth (if anything used) are leveraged?\n\n\n- In Section 4.1 and 4.2 authors talk about attention-only and embedding-only, while these terms and what they mean are not defined in the text. While the authors mention their architecture has attention head and embedding, it is not clear for attention-only or embedding-only which loss is applied and how they are measured.\n\n\n- Section Metric: it is not clear what is the difference between All and pair. It seems in both cases the union of hand and object is considered as positive.\n\n\n- For K-means clustering, authors mention there are 3 clusters, but images show 4 colors (yellow, red, blue, and green). Why there is such a difference?\n\n\n- In the appendix it is not clarified how responsibility maps are measured.  How the annotations or other info required for measuring the responsibility map are acquired? When using multiple frames authors mention \"We then compute the necessary features between these +/- 3 frames and the center frame to produce a single responsibility map\". how these \"necessary features\" are computed? \n\n\n\n- In the appendix it is explained for detecting object parts, a hand query (provided as input) and a background query is used and then subtracted from the responsibility map. Since no background is provided as input, how this is computed? No visualised result of this process is shown. If this approach works better than using PCA or clustering (shown in images 2 and 3), why are these results not shown. On the other hand if PCA or clustering work better, why are the results in tables not computed that way?\n\n\n\n- The usage of term responsibility throughout the whole paper is very unconventional. This term has not been used in similar literature. I recommend replacing it with something that better conveys what the method is actually doing.\n\n\n**Technical:**\n\n- The major drawback of the proposed approach is that it requires hands and objects to be moving and background to be almost stationary, which is due to usage of optical flow. If the hand and object are not moving, there is no optical-flow signal. Also, if the background is moving rapidly together with hand (e.g. when camera is moving) it can produce a high optical flow which interferes with the optical flow of hand and object, so detection of them becomes erroneous. This limits the application of the proposed approach.\n\n\n- In the contrastive section, it seems the labels for hand and object parts are already provided using [21] and union of responsibility. If such annotation is available one can directly train on labels and no contrastive or semi-supervised training is needed. On a related note, in Table 2 better results are obtained with attention-only. Authors do not explain why this is happening in lines 292 to 295. Is it because you have enough annotations that embedding is not needed anymore?\n\n\n- Are the test sets composed of new/unseen actions interacting with new objects? It is important to have unseen objects and actions in the test set, otherwise it is not clear if the model really generalises to new actions and unseen objects that exist in the real world. \n\n\n\n- In Figures 3, 4, and 5 in appendix (also images 2 and 3 in the main paper), it is not clear what are the outputs of the model for each one of all, pair, object, and hand given the process described in D.1. In particular, the question is how the model separates hand, object, and background in these cases.\n\n\n- The responsibility map shows the model can detect the hand and object regions together, but the separation of these two parts is not accurate. It seems the contrastive loss that should separate them is not helping much in separating the two.\n\n\n\n\n**Grammatical/writing Errors:**\n\nA lot of grammatical errors are seen in the paper. Such errors are also seen commonly in Appendix.  \t\t\t\t\t\n- line 34, 35 :  needs rephrasing, the phrase reads as if you are segmenting the motion, while you are segmenting the 3D object and not the motion. \n\n- lines 56, 57: The first paragraph of related work is referring to itself and different parts of proposed work, while it is still not clear at this point what authors are particularly doing. Instead, authors need to talk about more general concepts in the literature and then relate them to their approach. The whole paragraph needs rephrasing. \t\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\n- lines 69, 70: the example is not clear enough. The monitor is then used or unused in their case and how does it impact the results? Is it a problem of annotation, or the way the models are trained?\n\n- line 78: weird positioning of \"our work falls into this category. \"\n\n- line 82 : \"Rather than learn\" -> Rather than learning\n\n- lines 118, 119: following phrase should be corrected \"Formally, we the  responsibility as a temperature-softened softmax per-pixel\"\n\n- lines 162, 163: Three models are mentioned that predict embedding Z and it is not clear which one is actually used.\n\n- line 164: \"In one cases\" -> In one case\n\n- line 164: it is not clear how the phrase in parenthesis relates to the previous phrase.\n\n- lines 168 and 169: the phrases in parentheses are not connected to the main sentence and it interferes with the flow of the text.\n\n- line 184: \"these each require \" -> each of these require\n\n- Section 3.3.The usage of the term \"direct\" for the loss is very uncommon. If the loss is supervised, the term supervised should be used.\n\n- line 192: 'after' can be replaced by 'once'. It is more academic.\n\n- The usage of punctuations and other grammatical symbols have been casual throughout the text. The dot is sometimes missed.:\n   - line 227: before \"We also generate 30K \" \n   -line 237: before \" This data is sufficiently \"\n   \n- lines 243 to 245: the phrase is erroneous and should be rephrased.\n\n- line 280: \"adjacent pairs of video in an image \" I guess authors mean \"adjacent frames of a video\". such writing is very ambiguous and is seen often in the text.\n\n- line 339 \"with their hands can have of course be used for surveillance purposes. \" -> remove \"have\"\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors mention limitations and potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12 hours"}, "extra_text": {}}, {"review_id": "1YGwyvayrS", "reviewer": "Reviewer_obE8", "summary": "This paper addresses a novel problem of segmenting the object which is contacted by hand from a singe image. Since there are no annotations for this task, the authors present an approach to mine pseudo labels from videos based on optical flow. In particular, the authors assume that the contacted object has similar motion fields as the hand. The method works well for many cases except when objects are not moving or not moving in planar planes as also discussed in the paper. The experimental results demonstrate the effectiveness of the approach.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The addressed problem is interesting and novel. In particular, the self-supervised learning style helps to generalize to a large number of contact objects which is expensive to annotate in supervised learning.\n\nBut the method is very simple. Using optical flow to find objects has been studied previously although not for contact object segmentation [1,2,3]. In addition, the method has limitations when objects are not moving as is also discussed in Figure 4. I believe handling static objects will make the paper stronger.\n\nIn general, the paper is well written and reads easily. Some minor issues:\n(1) In line104, what does \"responsibility minus people\" mean?\n(2) In training, is the number of hands N for different images fixed? Based on my understanding, the responsibility maps should have fixed shapes. But in practice, the number of objects can be very different from image to image.\n(3) In line152, you mentioned to select data in which hands are contacting objects. How did you get this information?\n(4) Reference [21] is incomplete.\n\nOne interesting baseline is missing. In inference, if we are given a pair of consecutive images, we can segment the contacted object using optical flow (as in training). How is the accuracy for that?\n\nAblation study with the contrastive loss should be provided.\n\n[1]Romero, Javier, Matthew Loper, and Michael J. Black. \"FlowCap: 2D human pose from optical flow.\" In German conference on pattern recognition, pp. 412-423. Springer, Cham, 2015.\n[2]Fragkiadaki, Katerina, Han Hu, and Jianbo Shi. \"Pose from flow and flow from pose.\" In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2059-2066. 2013.\n[3]Wang, Chunyu, Yizhou Wang, and Alan L. Yuille. \"An approach to pose-based action recognition.\" In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 915-922. 2013.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewer ratings of this paper were significantly lower prior to the author response.
Reviewers were concerned about the quality of the writing.
Additional experiments, clarifications and even some simplifications of the proposed method were provided during the author response.
As a result of these actions all reviewers increased their scores after reading the author response.
After accounting for the author response and discussions with reviewers, all reviewers recommend accepting this paper.
It will be important for the authors to address the points and issues raised during the review and author discussion in the final version of this paper. 

The AC recommends accepting this paper.

Decision: Accept (Poster)