COHESIV: Contrastive Object and Hand Embedding Segmentation In Video

1 Introduction . We invite you to pick up an object in your vicinity and bring it towards you . As you hold the object and move your hand , the object moves coherently , and is roughly rigidly attached to a coordinate frame in your hand . While your adult brain does not need this signal and can readily differentiate the object in your hand from both your hand and the background even if you hold your hand still , how did you learn to do this ? One answer [ 42 ] is that as a human you have time locked modalities of your own hand ’ s configuration via proprioception and vision , and your hand and the object share a “ common fate ” [ 47 ] . The goal of this paper is to operationalize this idea by learning to segment hand and in-hand objects ( irrespective of name ) from a single image by learning from video data . This goal poses many challenges for current computer vision . While there has been a longstanding interest in learning segmentation and object individuation from motion cues in vision [ 41 , 34 , 11 ] , the general case of being broadly able to segment everything has not made substantial progress . While there has , of course , been substantial progress on instance segmentation for particular categories ( for instance MaskRCNN [ 21 ] ) powered by large segmentation datasets [ 28 ] , the space of categories that one can pick up is vast . Indeed , recent work that aims to reconstruct in-hand objects via segmentation losses [ 7 ] reconstructs a few objects by finding category correspondences ( e.g. , tennis racket masks are used for knives ) . At the same time , while there is work on understanding hands , including contact state [ 32 ] , and detecting boxes around held generic objects [ 40 ] , there is no work on segmenting objects , much less work that learns from video data . In sum – segmenting lots of generic objects is still challenging , and segmenting generic objects in contact is no exception . Our approach makes progress on this challenging problem by assuming a small amount of knowledge about humans . We assume we can estimate landmarks on hands [ 37 ] , identify whether they are holding something [ 40 ] , as well as identify which pixels belong to people [ 23 ] . This small amount of information guides understanding of the the deluge of information from optical flow [ 43 ] . Rather than segmenting the full 3D motions of 3D objects from the flow , we instead only have to identify whether the flow is better explained by a moving hand or a background . Moreover , by knowing about an ubiquitous object ( the hand ) , we show we can learn about many less ubiquitous objects . ∗Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We implement our solution with a network , named COHESIV , that takes an RGB image and one 2D location on a hand as inputs to segment the hand and the object that the hand is interacting with ( Section 3 ) . At test time , COHESIV maps the input image to an embedding space with a CNN [ 38 ] ; this embedding is processed with lightweight heads to produce per-pixel detectors at each hand , and feature maps for objects that can be queried to produce a heatmap . At training time , we use the motion with nearby frames to derive signal . We define a responsibility map for a moving hand as the relative goodness of fit on optical flow for a planar motion model in comparison to a background model . These responsibilities power two losses : a similarity loss that directly supervises per-hand segments ; a contrastive loss [ 13 ] that encourages in-group affinity as well as separates embeddings among people , objects , and background via pseudo-labels . We train and validate on video data of humans engaged in complex behaviors using subsets of the 100 Days of Hands ( 100DOH ) [ 40 ] , EPIC-KITCHENS-55 ( EPICK ) [ 9 , 10 ] , and HO3D [ 18 ] datasets . We compare with alternate methods that range from fully-supervised bounding boxes [ 40 ] to basic motion cues from optical flow [ 43 ] , to saliency [ 51 ] . We show that our weakly-supervised method is comparable to the supervised bounding box detector method , while outperforming flow and saliency methods . 2 Related Work . Our work aims to segment hands and in-hand objects from a single image and thus interacts with a variety of related areas , ranging from : the domain we work on ( understanding hand-object contact ) , the signals we use ( common fate with optical flow ) , and the methods we use to extract meaning from this signal ( contrastive learning ) . Our approach differs from each by using contrastive learning to extract supervision from optical flow in order to segment out objects people are holding . Understanding hands in contact with objects has long been of interest . In addition to work focused on finding hands [ 3 , 31 , 40 ] , there has been considerable work on reconstructing hands ( for instance with a known shape model like [ 36 ] ) along with outputs such as poses [ 12 ] or 3D meshes [ 20 , 19 , 7 ] . While these approaches can often produce meshes , they usually rely on strong supervision . In contrast , our approach derives its signal from using a small amount of information to make sense of optical flow in videos . While our approach resembles work on human-object relationship detection [ 45 , 17 ] , our relationship is specifically physical contact ( e.g. , in HOI [ 45 ] a person may be using a monitor if they are holding a mouse while we would only segment the monitor if it were physically grasped ) . Robots accomplish similar object-in-contact tasks [ 14 ] , but can be challenged by the need to collect large scale interactions . The most related work is understanding objects in contact , ranging from detailed contact models like [ 5 ] to bounding boxes like [ 40 ] . Our work produces a richer per-pixel segmentation compared to [ 40 ] while also requiring less supervision . We extract this information from optical flow , which has been known as a signal for perceptual organization since the Gestaltists [ 47 ] and Gibson [ 16 ] . Optical flow at inference time has been used , for instance for doing motion segmentation ( [ 41 , 6 , 35 , 26 , 11 , 25 ] among many others ) . When coupled with learning machinery , the optical flow provides a supervisory signal that can be absent at inference time ; our work falls into this category . This signal has been used to learn representations e.g. , for pure recognition [ 34 ] , but our approach is most related to its use for perceptual organization , for instance learning correspondence [ 46 , 24 , 48 ] or boundaries [ 50 , 27 ] . Our work is most similar to works learning boundaries but focuses on a specific type of boundary – those of hand-held objects . Rather than learning to segment everything , we learn to group hands and the objects they hold . To the best of our knowledge , there is no work that does this ; the closest is [ 4 ] , that finds important objects for an egocentric user based on gaze ( as opposed to what object is in a person ’ s hand ) . Our approach for learning uses attention for prediction and contrastive learning for grouping . We use attention as a means of relating query points against an entire image . This mechanism is common in methods that solve visual question answering [ 1 , 30 ] , though our use of attention aims to instead discover unknown object extents . In this regard , our work is loosely related to [ 49 ] , an attentionbased visual question answering method . Contrastive learning has become a standard formalism for self-supervised learning , including remarkable performance [ 8 , 33 , 2 ] . We use a formulation inspired by [ 44 ] , which uses saliency as a grouping signal on Internet image data . In our method , we use optical flow , hands and people instead . Our approach also combines self-supervised learning for per-pixel embeddings with supervision that aims to directly predict which object goes with the hand . 3 Method . Our approach learns to map the image and an on-hand query point to a prediction of what parts of the scene move with the hand at the query point . While the system requires a single image at test time , it makes use of the rich temporal signal found in video for training . At the core of our approach is the notion of responsibilities for hands , inspired by the notion of responsibilities when fitting a GMM [ 15 ] : the responsibility of a pixel for a hand is how well that hand explains the pixel ’ s motion compared to other hands and the background ( Section 3.1 ) . At test time , we aim to predict the hand and hand-held object masks for the queried hand . Our network ( Section 3.2 ) predicts this by mapping the image to a shared embedding , which is in turn converted into per-pixel object features that are attended to by per-pixel hand detectors . At training time , the responsibilities provide the basis ( along with the pixel locations of humans ) for a set of losses ( Section 3.3 ) that we use to train our network , using both direct supervision on the outputs with responsibility maps as well as a constrastive loss on the embeddings . Throughout , we assume access to systems that can estimate between-frame optical flow , landmarks on the hand , and which pixels belong to people . We use optical flow from RAFT [ 43 ] , projected hand joints from FrankMocap [ 37 ] ( to get a set of points on the hands ) , and people masks from an off-the-shelf people segmentation system [ 23 ] . 3.1 Responsibility . We formalize the notion of synchronous motion ( or common fate [ 47 ] ) for hand and in-hand object via the notion of responsibility . Given an optical flow map O ∈ RH×W×2 and a set of N handsH , we aim to produce N responsibility maps R ∈ RH×W× ( N+1 ) with ∑N+1 k=1 Ri , j , k=1 that explain how well each pixel is explained by each hand ’ s motion model or the background . For the kth hand , we formally model the responsibility as a temperature-softened softmax per-pixel , or : Ri , j , k = expt ( −dk ( Oi , j , : ) ) expt ( −dBG ( Oi , j , : ) ) + ∑N k′=1 expt ( −dk′ ( Oi , j , : ) ) ( 1 ) where expt ( x ) = exp ( −x/t ) is an exponential with a tunable temperature ( set to 2 based on a small held-out set ) and dBG and dk compute distances between an optical flow vector and a model ( namely a background model and a model for the kth hand respectively ) . Equation 1 requires building and evaluating distances between the movement seen in a flow vector o ∈ R2 . We treat the background as static and thus the model dBG ( o ) is simply ||o||2 . This is not the best model for egocentric data , but we find it to be effective , potentially due to the high frame rate of the dataset [ 9 ] we use . Our hand models dk ( o ) assume a planar motion because it is simple to solve for , can handle out of image-plane rotation , and is a reasonable approximation at the distances and timescales the data shows . This entails fitting a homography Mk ∈ R3×3 on hand landmarks . Supposing proj ( · ) converts homogeneous to normal coordinates ( [ x , y , z ] → [ x/z , y/z ] ) , then the landmarks should satisfy [ i+ o1 , j + o2 ] T = proj ( Mk [ i , j , 1 ] T ) for a pixel i , j with flow o . We tried with a variety of options , but found simply fitting Mk to the estimated flow at FrankMocap [ 37 ] vertices to be most effective . Using [ 37 ] to provide correspondence proved worse , likely since the resulting hand motion model also included disagreement about the landmark locations in each frame . Then , given a model Mk and pixel i , j with flow o , the models ’ prediction is proj ( Mk [ i , j , 1 ] T ) . We define the distance as the difference between the actual flow and the predicted flow , dk ( Oi , j ) = ∣∣∣∣ [ i , j ] T +Oi , j − proj ( Mk [ i , j , 1 ] T ) ∣∣∣∣2 . ( 2 ) While simple , the planar model has a counter-intuitive catch . By handling out of plane rotation , the planar model will produce many points that match optical flow by accident specially for static cameras : proj ( Mk [ i , j , 1 ] T ) often has zeros in the image . While a learning system could learn to ignore these accidental matches by training on a large dataset , we accelerate this by averaging it out : we calculate responsibility maps to the next and previous 3 frames , and then average them . Techniques : We estimate optical flow with RAFT [ 43 ] and hand vertices with FrankMocap [ 37 ] . FrankMocap fuses a hand detector [ 40 ] with a network that predicts 2D and 3D hand joints . This information is used to compute homographies and then responsibility maps . Once responsibilities are computed , they serve as pseudo-labels at training time and are then discarded . At test time , we aim to predict hands and held-objects for new images . Data Selection and Implementation Details : The data we use is challenging non-scripted videos that include vast amounts of non-interaction . We select our training and validation data so that the network primarily sees hands that are visible and in contact . We select clips where hands appear in 10 consecutive frames as well as hands are moving within the clip . In 100DOH , the clips are detected as visible hands holding reasonably sized objects using [ 40 ] . We note that we only have to do this selection since the data used is large and non-scripted . In EPICK , we use their action segments . ( see the supplemental for more details )

Summary: This paper tackles the problem of hand and object segmentation. At inference, the model takes a single image and the 2D location of the hand and outputs a hand-object segment. The key idea is to extract supervision from optical flow by contrastive learning.