What Can You Learn From Your Muscles? Learning Visual Representation from Human Interactions

1 INTRODUCTION . Encoding visual information from pixel space to a lower-dimensional vector is the core element of most modern deep learning-based solutions to computer vision . A rich set of algorithms and architectures have been developed to enable learning these encodings . A common practice in computer vision is to explicitly train the networks to map visual inputs to a curated label space . For example , a neural network is pre-trained using a large-scale annotated classification dataset ( Deng et al. , 2009 ; Krasin et al. , 2017 ) and the entire network or part of it is fine-tuned to a new target task ( Goyal et al. , 2019 ; Zamir et al. , 2018 ) . In recent years , weakly supervised and self-supervised representation learning approaches ( e.g. , Mahajan et al . ( 2018 ) ; He et al . ( 2020 ) ; Chen et al . ( 2020a ) ) have been proposed to mitigate the need for supervision . The most successful ones are contrastive learning-based approaches such as ( Chen et al. , 2020c ; b ) and they have shown remarkable results on target tasks such as image classification and object detection . Despite their success , there are two primary caveats : ( 1 ) These self-supervised methods are still trained on ImageNet or similar datasets , which are fairly cleaned up and/or include a pre-specified set of object categories . ( 2 ) This method of training is a passive approach in that it does not encode interactions . On the contrary , for humans , a vast majority of our visual understanding is shaped by our interactions and our observations of others interacting with their environments . We are not limited to learning from visual cues alone , and there are various other supervisory signals such as body movements and attention cues available to us . It is shown that by learning how to move the joints to walk and crawl , infants can significantly enhance their perception and cognition ( Adolph & Robinson , 2015 ) . Moreover , by observing another person interact with the environment humans obtain a visual and physical perception of the world ( Bandura , 1977 ) . The question we investigate in this paper is , “ can we learn a rich generalizable visual representation by encoding human interactions into our visual features ? ” . In this work , we consider the movement of human body parts and the center of attention ( gaze ) as an indicator of their interactions with the environment and propose an approach for incorporating interaction information into the musclysupervisedrepresentation learning process . To study what we can learn from interaction , we attach sensors to humans ’ limbs and see how they react to visual events in their daily lives . More specifically , we record the movements of the body parts by Inertial Movement Units ( IMUs ) and also the gaze to monitor the center of attention . We introduce a new dataset of more than 4,500 minutes of interaction by 35 participants engaging in everyday scenarios with their corresponding body part movements and center of attention . There are no constraints on the actions , and no manual annotations or labels are provided . Our experiments show that the representation we learn by predicting gaze and body movements in addition to the visual cues outperforms the visual-only baseline on a diverse set of target tasks ( Figure 1 ) : semantic ( scene classification ) , temporal ( action recognition ) , geometric ( depth estimation ) , physics ( dynamics prediction ) and affordance-based ( walkable surface estimation ) . This shows that movement and gaze information can help to learn a more informative representation compared to a visual-only model . 2 RELATED WORK . Visual representations can be learned using many different techniques from full supervision to no supervision at all . We outline the most common paradigms of representation learning , namely supervised , self-supervised , and interaction-based representation learning . Supervised Representation Learning . Supervised representation learning in computer vision is typically performed by pre-training neural networks on large-scale datasets with full supervision ( e.g. , ImageNet ( Deng et al. , 2009 ) ) or weak supervision ( e.g. , Instagram-1B ( Mahajan et al. , 2018 ) ) . These models are fine-tuned for a variety of tasks including object detection ( Girshick et al. , 2014 ; Ren et al. , 2015 ) , semantic segmentation ( Shelhamer et al. , 2015 ; Chen et al. , 2017 ) , and visual question answering ( Agrawal et al. , 2015a ; Hudson & Manning , 2019 ) . However , collecting a manually annotated large-scale dataset such as ImageNet requires extensive resources in terms of cost and time . In contrast , in this paper , we only use human interaction data , which does not require any manual annotation . Self-supervised Representation Learning . There has been a wide range of research on selfsupervised learning of visual representations in which properties of the images themselves act as supervision . The objectives for these methods cover a variety of tasks such as solving jigsaw puzzles ( Noroozi & Favaro , 2016 ) , colorizing grayscale images ( Zhang et al. , 2016 ) , learning to count ( Noroozi et al. , 2017 ) , predicting context ( Doersch et al. , 2015 ) , inpainting ( Pathak et al. , 2016 ) , adversarial training ( Donahue et al. , 2017 ) and predicting image rotations ( Gidaris et al. , 2018 ) . This type of representation learning is not limited to learning from single frames . Agrawal et al . ( 2015b ) and Jayaraman & Grauman ( 2015 ) both use egomotion , Wang & Gupta ( 2015 ) cyclically track patches in videos , Pathak et al . ( 2017 ) use low-level non-semantic motion-based cues , and Vondrick et al . ( 2016 ) predict the representation of future frames . Inspired by contrastive learning ( Hadsell et al. , 2006 ) , recent methods have used “ instance discrimination ” in which the network uniquely identifies each image . A network is trained to produce a non-linear mapping that projects multiple variations of an image closer to each other than to all other images . Using Noise Contrastive Estimation ( Gutmann & Hyvärinen , 2010 ) , networks are trained to differentiate between similar images under complex noise models ( such as non-overlapping crops and heavy color jittering ) and dissimilar images . Oord et al . ( 2018 ) and Hénaff et al . ( 2019 ) introduce and investigate the Contrastive Predictive Coding ( CPC ) method , which encodes the shared information between different crops of an image to predict the features from masked regions of the image . Wu et al . ( 2018 ) ; Misra & van der Maaten ( 2020 ) use a memory bank , which enables contrasting features of the current image against a large set of negative samples , increasing the likelihood of finding a nearby negative . The MoCo technique ( He et al. , 2020 ; Chen et al. , 2020c ) encodes the positive samples with a momentum encoder to avoid the rapid changes in the original feature extractor . They achieve comparable results with supervised learning representations . Chen et al . ( 2020a ; b ) show that by using a trainable non-linear transformation between the representation and contrastive latent space and larger batch sizes , they can omit memory banks entirely , allowing for full backpropagation through both positive and negative samples , and achieve better results . Bachman et al . ( 2019 ) ; Tian et al . ( 2019 ) maximize the mutual information between different extracted features of the same image from multiple views . Zhuang et al . ( 2019 ) enforce the extracted features of similar images to move towards the same part of the embedding space . Gordon et al . ( 2020 ) , Yao et al . ( 2020 ) , and Devon Hjelm & Bachman ( 2020 ) apply contrastive method to videos and leverage spatio-temporal cues to learn visual representations . Similar to these approaches , we do not rely on any manual annotation , but in contrast to them , we utilize human interactions along with their visual observation for representation learning . Interaction-Based Representation Learning . The third class of learning representations relies on cues obtained by interacting with a dynamic environment . Pinto et al . ( 2016 ) learn a representation from interactions of a robotic arm ( e.g. , grasping and pushing ) with different objects . Chen et al . ( 2019 ) and Weihs et al . ( 2019 ) both tackle the representation learning problem by training an agent to play a game in an interactive environment . Ehsani et al . ( 2018 ) learn a representation by modeling the non-semantic movements of a dog . Our work falls in this category since we use human interactions for learning the representation . We differ from these approaches in that we use lowlevel observations of human interaction such as body part movements and gaze to show significant improvement over a state-of-the-art baseline across multiple low-level and high-level target tasks . 3 HUMAN INTERACTION DATASET . We introduce a new dataset of human interactions for our representation learning framework . In this section , we describe the data collection . Our goal is to capture how humans react to the visual world by recording their movements and focus of attention . Previous datasets of human actions and gaze include only gaze information ( Fathi et al. , 2012 ; Xu et al. , 2018 ) , part movements from a third-person view ( Ionescu et al. , 2014 ; Hassan et al. , 2019 ) , or only action or hand labels in an egocentric setting ( Damen et al. , 2018 ; Sigurdsson et al. , 2018 ) . In contrast , our new dataset includes ego-centric observations along with the corresponding gaze and body movement information during their daily activities ranging from walking and cycling to driving and shopping . To collect the dataset , we record egocentric videos from a GoPro camera attached to the subjects ’ forehead . We simultaneously capture body movements , as well as the gaze . We use Tobii Pro2 eyetracking to track the center of the gaze in the camera frame . We record the body part movements using BNO055 Inertial Measurement Units ( IMUs ) in 10 different locations ( torso , neck , 2 triceps , 2 forearms , 2 thighs , and 2 legs ) . Figure 2 shows the data collection setup along with two clips of the captured sequences . In total , we collected 4,260 minutes of videos with their corresponding body part movement and gaze from 35 people . Unlike the common large-scale datasets used for representation learning such as ImageNet , there is no restriction on the categories observed in the images , and no manual annotation is provided . Although the supervision from movement and gaze does not come for free and still requires the participants wearing the sensors , no manual or semantic annotation is needed for acquiring this dataset . Statistical analysis of the dataset is provided in Appendix A.3 . Moreover , we provide details of aligning the video with the motion sensors and synchronization of the sensors in Appendix A.2 . The supplementary video shows a few examples of the video clips . 4 INTERACTION-BASED REPRESENTATION LEARNING . Visual representation learning is typically performed using visual cues from single images or videos ( He et al. , 2020 ; Gordon et al. , 2020 ) . Our goal in this paper is to incorporate human interactions into our representations to move beyond a purely visually-trained feature representation . Below , we describe our approach for integrating movement and gaze information in the representation learning pipeline . Intuitively , body part movements should encode the temporal changes in the image based on the underlying cause of those changes ( e.g. , moving legs results in walking which makes distant objects move closer ) . Additionally , gaze grounds the visual features with the location in the image where the person pays the most attention . This should correlate well with semantic concepts such as objects , or affordances such as walkable surfaces .

Summary: The paper uses a combination of visual, human gaze and human motion sensors to build representations that perform better on downstream tasks such as action recognition, physics prediction and depth estimation than representations extracted from solely visual input. The paper announces the release a new data set of aligned visual images, eye gaze fixations and IMU motion readings from test subjects walking around an environment. Representations are computed using three different forms of information simultaneously. Given a visual input, the system tries to predict the location of eye gaze in image frame coordinates, whether each of 6 groups of motion detectors are active or not (head, torso, legs, etc.) and the result of a more traditional auxiliary visual pretext task. In this work, the paper uses “instance discrimination” where representations of augmented versions of a specific image are pushed close together in latent space and far away from augmentations of other images. Tests on diverse benchmarks show that the gaze and motion prediction improve over visual pretext tasks alone and that there is a small benefit to using both together, but it is not additive. The paper also shows the benefit of gaze and motion is present for two different visual auxiliary tasks.