What Can You Learn From Your Muscles? Learning Visual Representation from Human Interactions
Authors: Kiana Ehsani
Daniel Gordon
Thomas Hai Dang Nguyen
Roozbeh Mottaghi
Ali Farhadi
Venue: ICLR 2021 Poster (2021)
Keywords: representation learning
computer vision

Abstract: Learning effective representations of visual data that generalize to a variety of downstream tasks has been a long quest for computer vision. Most representation learning approaches rely solely on visual data such as images or videos. In this paper, we explore a novel approach, where we use human interaction and attention cues to investigate whether we can learn better representations compared to visual-only representations. For this study, we collect a dataset of human interactions capturing body part movements and gaze in their daily lives. Our experiments show that our ``"muscly-supervised" representation that encodes interaction and attention cues outperforms a visual-only state-of-the-art method MoCo (He et al.,2020), on a variety of target tasks: scene classification (semantic), action recognition (temporal), depth estimation (geometric), dynamics prediction (physics) and walkable surface estimation (affordance). Our code and dataset are available at: https://github.com/ehsanik/muscleTorch.

Published as a conference paper at ICLR 2021
WHAT CAN YOU LEARN FROM YOUR MUSCLES? LEARNING VISUAL REPRESENTATION FROM HUMAN INTERACTIONS
Kiana Ehsani Daniel Gordon'
Thomas Nguyen Roozbeh Mottaghi
Farhadi
University of Washington, Allen Institute for Al
ABSTRACT
Learning effective representations of visual data that generalize to variety of downstream tasks has been a long quest for computer vision. Most representation learning approaches rely solely visual data such images videos. In this paper, we explore a novel approach, where we use human interaction and attention cues investigate whether only representations. For this study, we collect a dataset human interactions capturing body part movements and gaze in their daily lives. Our experiments show that our "muscly- tention cues outperforms visual-only state-of-the-ar method MoCo (He al., 2020), on a variety of target tasks: scene classification (semantic), action recogni- tion (temporal), depth estimation (geometric), dynamics prediction (physics) and walkable surface estimation (affordance). Our code and dataset are available at: https///iiubb.comm//ismk/mussei
learn better representations compared to visual-
representation that encodes interaction and at-
Recognition
Physies Prediction
Learned Representation
Scene Classification
C
Walkable Surface Estimation
Depth Estlination
Figure I: We propose to use human's 's interactions with their visual surrounding as a training signal for representation learning. We record first person observations well the movements and gaze of people living their daily routines and use these cues to learn a visual embedding. We use the learned representation on variety of diverse tasks and show consistent improvements compared to state-of-the-ar self-supervised vision-only techniques.
INTRODUCTION
Encoding visual information from pixel space to a diver-dimensional vector is the core element of modern deep learning- based solutions to computer vision. A rich of algorithms and archi- tectures have been developed enable learning these encodings. A common practice in computer vision is to explicitly train the networks to map visual inputs to a curated label space. For example, neural network is pre-trained using large-scale annotated classification dataset (Deng et al.) 2009 Krasin et al. (0077) and the entire network or part is fine-tuned to new target task (Goyal al. 2019 Zamir al. 2018).
In recent years, weakly supervised and self- suprvvuper representation learning approaches (e.g., Mahajan et al. (2018); ; He et al. (2020); Chen et al.] (2020a)) have been proposed mitigate the need for supervision. The most successtul ones are contrastive souming-bassee approaches such
(Chen al. 2020c b) and they have shown remarkable results target tasks such as image

---

Published as a conference paper at ICLR 2021
classification and object detection. Despite their success, there are two primary caveats: (1) These self-supervised methods are still trained on ImageNet or similar datasets, which are fairly cleaned up and/or include a pre-specified set of object categories. (2) This method of training is passive approach in that it does not encode interactions. On the contrary, for humans, a vast majority of our visual understanding is shaped by our interactions and our observations of others interacting with their environments. We are not limited to learning from visual cues alone, and there are various other supervisory signals such as body movements and attention cues available us. is shown that by learning how to move the joints to walk and crawl, infants can significantly enhance their perception and cognition (Adolph Robinson 2013). Moreover, by observing another person interact with the environment humans obtain a visual and physical perception of the world (Bandura 1977..
The question we investigate in this paper is, "can we learn rich generalizable visual representation by encoding human interactions into our visual features?". In this work, we consider the movement of human body parts and the center of attention (gaze) as an indicator of their interactions with the environment and propose approach for incorporating interaction information into the muscly- supervissedrepresenttiiin learning process.
To study what can learn from interaction, we attach sensors to humans" limbs and see how they react to visual events in their daily lives. More specifically, we record the movements of the body parts by Inertial Movement Units (IMUs) and also the gaze to monitor the center of attention. We introduce a new dataset of more than 4,500 minutes of interaction by 35 participants engaging in everyday scenarios with their corresponding body part movements and center of attention. There are no constraints on the actions, and no manual annotations or labels are provided.
Our experiments show that the representation we learn by predicting gaze and body movements in addition to the visual cues outperforms the visual-only baseline on a diverse set of target tasks (Fig- ure D: semantic (scene classification), temporal (action recognition), geometric (depth estimation), physics (dynamics prediction) and affordance-baser (walkable surface estimation). This shows that movement and gaze information can help to learn a more informative representation compared to a visual-only model.
RELATED WORK
Visual representations can be learned using many different techniques from full supervision to no supervision at all. We outline the most common paradigms of representation learning, namely su- pervised, self-supervised, and interaction-based representation learning.
Supervised Representation Learning. Supervised representation learning in computer vision is typically performed by pre-training neural networks on large-scale datasets with full supervision (e.g., ImageNet (Deng et al. 2009)) or weak supervision (e.g.. Instagram- 1B (Mahajan al,[[0]1 2018). These models are fine-tuned for a variety of tasks including object detection (Girshick et al. 2014 Ren et al.] 2015), semantic segmentation (Shelhamer et al.) 2015)] Chen et al.] 2017), and visual question answering (Agrawal et al. 2015a Hudson & [aaniing,2019) However, collecting a man- ually annotated large-scale dataset such ImageNet requires extensive resources in terms of cost and time. In contrast, in this paper, we only use human interaction data, which does not require any manual annotation.
Self-supervised Representation Learning. There has been a wide range of research on self- supervised learning of visual representations in which properties of the images themselves act as supervision. The objectives for these methods cover a variety of tasks such as solving jigsaw puz- zles (Noroozi & Favaro 2016). colorizing grayscale images (Zhang et al. (016). learning to count (Norozzi et al. (017). predicting context (Doersch et al. 2015). inpainting (Pathak et al. 2016), adversarial training (Donahue et al.] (017)) and predicting image rotations (Gidaris et al. 2018).
This type of representation learning not limited to learning from single frames. Agrawal et al. (0155) and Jayaraman & Grauman (2015) both use egomotion, Wang & Guptal (2015) cyclically track patches in videos, Pathak et al. (2017) use low-level non-semantic motion- motion-based cues, and Vondrick et al. (2016) predict the representation of future frames.
Inspired by contrastive learning (Hadsell et al. 2006), recent methods have used "instance discrim- ination" " in which the network uniquely identifies each image. A network is trained to produce a non-linear mapping that projects multiple variations of image closer each other than to all other
2

---

Published as a conference paper at ICLR 2021
Camera
Gaze Tracker
IMUS
Figure 2: Dataset examples. Two sequences from our dataset are shown on the left. The first row shows the sequence of the images and the second row shows the movements of the body parts cording to the IMU readings. We visualize the gaze using the red circle. This is just for visualization purposes and does not exist in the image. On the right, show the data collection setup.
images. Using Noise Contrastive Estimation (Gutmann & Hyvärinen 2010), networks are trained to differentiate between similar images under complex noise models (such as non non nverllaegg crops and heavy color jittering) and dissimilar images. Oord et al. (2018) and Hénaff et al. (2019) intro- duce and investigate the Contrastive Predictive Coding (CPC) method, which encodes the shared information between different crops of image predict the features from masked regions of the image. Wu et al. (2018); Misra & van der Maaten (2020) use a memory bank, which enables contrasting features of the current image against large set of negative samples, increasing the like- lihood of finding a nearby negative. The MoCo technique (He et al., 2020 Chen et al.) 2020c) encodes the positive samples with momentum encoder to avoid the rapid changes in the original feature extractor. They achieve comparable results with supervised learning representations. Chen ctt al. 2020a b) show that by using trainable non-linear transformation between the representation and contrastive latent space and larger batch sizes, they can omit memory banks entirely, allow- ing for full backpropagation through both positive and negative samples, and achieve better results. Bachman et al. (2019); Tian et al. (2019) maximize the mutual information between different tracted features of the same image from multiple views. ZZhuang et al. (2019) enforce the extracted features of similar images to move towards the same part of the embedding space. Gordon et al.) (2020). Yao et al. (2020). and Devon Hjelm & Bachman (2020) apply contrastive method to videos and leverage spatio-temporal cues to learn visual representations. Similar these approaches, we do not rely on any manual annotation, but in contrast to them, we utilize human interactions along with their visual observation for representation learning.
Ierraction-Basee Representation Learning. The third class of learning representations relies on cues obtained by interacting with a dynamic environment. Pinto al. (2016) learn representation from interactions of robotic arm (e.g., grasping and pushing) with different objects. Chen et al. (2019) and Weihs et al. (2019) both tackle the representation learning problem by training an agent to play game in an interactive environment. [Ehaan et al. (2018) learn representation by mod- eling the non-semantic movements
dog. Our work falls in this category since we use human interactions for learning the representation. We differ from these approaches in that level observations of human interaction such as body part movements and gaze to show significant improvement over state-of-the-ee -rr baseline across multiple low-level and high-level target tasks.
low-
3 HUMAN INTERACTION DATASET
We introduce a new dataset of human interactions for our representation learning framework. In this section, describe the data collection. Our goal is capture how humans react to the visual
3

---

Published as a conference paper at ICLR 2021
Previous datasets of human actions and gaze include only gaze information (Fathi et al. 2012 Xu et al. 2018), part movements from a
world by recording their movements and focus of attention.
third-person view (lonescu et al. 22144; Hassan et al. 2019), or only action or hand labels in ego- centric setting (Damen et al. 2018 Sigurdsson et al. 2018). In contrast, our new dataset includes ego-centric observations along with the corresponding gaze and body movement information during their daily activities ranging from walking and cycling to driving and shopping.
To collect the dataset, we record egocentrie videos from GoPro camera attached to the subjects' forehead. We simultaneously capture body movements, as well as the gaze. We use Tobii Pro2 eye- tracking to track the center of the gaze in the camera frame. We record the body part movements using BNO055 Inertial Measurement Units (IMUs) in 10 different locations (torso, neck, 2 triceps, forearms, thighs, and 2 legs). Figure 2 shows the data collection setup along with two clips of the captured sequences. In total, we collected 4,260 minutes of videos with their corresponding body part movement and gaze from 35 people. Unlike the common large-scale datasets used for representation learning such ImageNet, there is restriction on the categories observed in the images, and no manual annotation is provided. Although the supervision from movement and gaze does not come for free and still requires the participants wearing the sensors, no manual or semantic annotation is needed for acquiring this dataset. Statistical analysis of the dataset is provided in Appendix A.3 Moreover, synchronization of the sensors in Appendix A.2 The supplementary video shows a few examples of the video clips.
provide details of aligning the video with the motion sensors and
INTERACIION-BSSER REPRESENTATION LEARNING
Visual representation learning is typically performed using visual cues from single images or videos (He et al. 2020 Gordon et al., 2020 Our goal in this paper is to incorporate human in- teractions into our representations
move beyond a purely visually-traines feature representation. Below, we describe our approach for integrating movement and gaze information in the representa- tion learning pipeline. Intuitively, body part movements should encode the temporal changes in the image based on the underlying cause of those changes (e.g.. moving legs results in walking which makes distant objects move closer). Additionally, gaze grounds the visual features with the location in the image where the person pays the most attention. This should correlate well with semantic concepts such as objects, or affordances such as walkable surfaces.
4.1 LEARNING FEATURES
Our goal is learn visual representations by simultaneous learning of visual encoding for each frame and predicting body part movements and gaze attention from the sequence of observations. Formally, given an ego-centric video as sequence of images V (Tr,.... Je+k), the goal is to 1) estimate the gaze G (G,, ..))), where G, is the person's 's center of focus in 2D camera coordinates, and 2) predict the body part movements P = (P....., Pt++). where Pt is a binary vector of length equal to the number of body parts, indicating whether part is moved at time t.
We optimize three objectives: (1) gaze prediction, (2) body part movement prediction, and (3) aux- iliary visual prediction. The visual features obtained from CNN backbone combined with sequence-to- sequence-to-sequencc to-sequeeee model in order to predict gaze and movement. Note that the weights of the backbone randomly initialized, i.e. train the model from scratch. Figure 3 shows overview of the architecture. The objectives are jointly optimized. In the following, we explain each of them in more detail.
Gaze: We predict the person's 's focus of attention by modeling their gaze in the camera reference frame. We use the Huber loss to train the center of attention Lattention (oaooomCh,h, (G, G),,
& G
G G-G <
Lattention IiatenttionGGGO (G.G) =
(1)
G G
otherwise
Movement: We find the task of predicting body part movement direction and magnitude to be highly ambiguous. For example, when walking, the visual information may not show the legs, so we cannot know how high the legs were lifted. Instead, for each body part, predict whether it is moving
4

---

Published as a conference paper at ICLR 2021
Legend
Loss Functions
Memory Bank
Luisual
Leamed weights
Attention Decoder LSTM
Luttention
Encoder LSTM
ResNet
1x1 conv
Movement Decoder LSTM
Loovement
3x224x224
512x7x7
64x767
Backbone
Figure Model Overview. We learn muselyssuperviedippessetaaaoo by jointly optimizing visual, movement and center of focus (gaze) objectives. The portion outlined with rectangle is the backbone that is used to evaluate the representation for target tasks. All parts the network are initialized randomly and trained from seratch.
at all which is less ambiguous and reduces the problem binary classification task. Rather than predicting the movements for lower and upper parts of the joint separately (leg and thigh, forearm and tricep), we combine the movements into 6 categories of torso, neck, right arm, left arm, right leg, and left leg. To estimate this movement, we use binary cross-entropy loss and denote as Emovement (P,P).
Auxiliary Visual Prediction: We also use a visual objective, Lvisuat [Le, 4t). For this objective, similar (He al. 2020), we instance discrimination. Any alternative visual encoding objective can be used instead. In section I.3.I we provide results for another type of visual encoding as well.
Instance discrimination's 's objective to force the visual features of different augmentations of the same image to be as close as possible in the latent space (Wu et al. 2018 Chen et al. 2020al Zhhang al. 2019) while pushing apart all other image embeddings. By learning extract what makes each image unique, the network focuses on semantically meaningful features of the image. This enables the feature extractor embed more invariant representation of the image, which is especially important when transferring to different tasks and domains. To contrast the positive samples (the augmentations of the image), with a large set negative samples, we maintain a memory bank of embedded features from different images in the data. The final objective can be formalized as Lvisual (which is also known as the InfoNCE (Oord et al. 2018) loss),
Cvisual (fc, log exp(f(I) f(f./T)
Ee (II(f(I) M,/7)*
where 4,, le are two different random augmentations of the first image of the sequence V, is the image feature extractor (ResNet backbone), .M (Mo,..., Mx) is the bank of negative samples and 7 is a parameter that controls the concentration level the distribution (Hinton et al., 2015). We also
commentum-uudattee encoder in (He et al. (020). We apply the visual loss only the first image of the sequence, as images within a sequence tend to be visually similar to each other.
The overall objective is weighted of the described loss functions. More details the archi- tecture are provided in Appendix 1 1.4.1
Linteraction = Laattention (G, G) + m.mommmee (P, + Criemot Lt)
(3)
4.2 ADAPTING THE REPRESENTATION TO NEW TASKS
After training the model using Linteraction objective, use the trained weights of our feature traction network (i.e., only the ResNet part) as the initialization for our target tasks. Our goal in this paper is evaluate the visual representation its own rather than using it initialization for end-to-end training. Hence, during training for the target tasks, the weights of the feature extraction backbone frozen. We have diverse set of target tasks, where each requires specific network
5

---

Published as a conference paper at ICLR 2021
Datasets
SUN397
Epic Kitchen
VIND
NYUv2
Xiso et al
2010
Damen al
2018
Mottighi
2016
Nathon Sitherman & Forgu 2012
Method
Training Objective
(a) Scene
(b) Action
(c) Dynamics
(d) Walkable
(e) Depth (RMSElog }
(Top-1 1)
(Top-1 f)
(Top-1 f)
(IOU t)
MoCo
He a 2020
vis
15.80
24.45
13.18
58.97
0.148
Ours
vis/attn
21.27
26.80
13.71
59.65
0.145
Ours
vis/move 21.08
26.71
13.22
58.42
0.144
Ours
vis/move/attn
22.82
27.95
14.44
58.38
0.146
Table 1: Target task results. We compare the performance of our learned representation from movement and gaze cues with recent self-supervised baseline MoCo (He et al. (2020) (which is trained on our data). Note that the MoCo baseline trained using only visual data ('vis'). We evaluate the performance on variety of different target tasks.
architecture (for example, depth estimation requires up-convolutional layers, while action recogni- tion requires a temporal architecture). Below, we describe the result of the transfer to the target tasks. We explain the details of the architectures for each target task in Appendix A.4.2
EXPERIMENTS
To evaluate our representation learning approach, we consider five different types of target tasks. The tasks are chosen such that they cover wide range of domains: semantic (scene classification), temporal (action recognition), geometric (depth estimation), physical (dynamics prediction), and affordance (walkable surface estimation). We show that our learned representation, which encodes body part movement and gaze and does not rely any manual annotation, outperforms strong self-supervised baseline which relies on purely visual cues. Furthermore, we provide ablations of our model by using an alternative visual loss and using subset of body parts for representation learning. For implementation details, refer to Appendix A.4 A.4
5.1 SELF-SUPERVISED BASELINE
We compare our method with the recently introduced self-supervised representation learning tech- nique, Momentum Contrast network (MoCo) (He et al.] (020), which is a state-of-the-ar represen- tation learning approach and achieves strong performance on a variety of target tasks such as image classification and object detection. The original work was trained on images from the ImageNet dataset. To ensure the comparison between our method and the baseline fair, we train MoCo on the images from show whether we can learn better representations when we use movement and gaze information in addition to the visual information.
dataset. Note that this baseline relies visual cues only. Our goal is to
5.2 EVALUATION OF THE LEARNED REPRESENTATION
We evaluate the learned representation on five different target tasks. The weights for feature ex- traction backbone are frozen, and only the task-specific layers are trained. We show that the repre- sentation trained using the movement and attention (gaze) supervision in addition to the visual cues outperforms MoCo (He et al., 2020) baseline (trained on our data) across the board. For each target task, we report the results in four settings, each using a different combination of visual, movement, and gaze (attention) cues for representation learning.
Scene Classification. For the task of scene classification, network receives single image input and predicts the scene category of the image. We use SUN397 (Xiao al.) 2010 dataset for this task, it provides large-scale dataset of 130k images of 397 different scene categories (c.g., park, restaurant, kitchen). The results are shown in Table column (a). The representation that encodes both movement and attention cues performs the best the semantic task of scene classification. We achieve nearly a 7% improvement compared to fine-tuning the MoCo (He et al.) 2020 baseline.
Action Recognition. The task is to predict the category of action from ego-centric videos. We use the EPIC- KITCHENS dataset (Damen et al., 2018) for this task, which is a large-scale dataset of 1 11M images from different action categories that performed in various kitchens.

---

Published as a conference paper at ICLR 2021
Scene Classification Top-1
Action Recognition Top-1 f
Dynamics Prediction Top-11
Walkable Estimation IoU t
Depth Estimation RMSElog
Training Objective
Lue
11.59
23.84
9.62
43.64
0.175
Lae + Latt + Lmove 15.08
25.69
10.78
47.20
0. 0.169
Enoe + Latt + Lmove 22.82
27.95
14.44
58.38
0.146
Table 2: Ablation of the visual loss. The result of using an autoencoder for the visual loss. We
re-train the models for the five target tasks. Latt, Lmooe and Lace are the ones used in Eq. 3
As shown in Table column (b), our method outperforms the strong baseline representation learning method by 3.5%. This again shows that incorporating additional cues such as part movements and gaze the representation learning is beneficial for downstream tasks. seems that both movement and attention cues are helpful for action recognition. This is aligned with our intuition that predicting the gaze of a person and how they move their body parts may be beneficial to recognizing the actions they perform.
Future Prediction of Dynamics. The goal of this task is to predict the future dynamics of an object in image. We use the VIND (Mottaghi et al. 2016a) dataset for this task. It includes 150K images with corresponding object bounding boxes. The dataset categorizes physical dynamics into Newtonian scenarios such sliding, projectile motion, and bouncing. The goal is to predict these Newtonian scenarios and the camera viewpoint for a query object that is specified by a bounding box and physical motion labels. There are 66 classes in total. The input to the network is single RGB image and the bounding box for the query object. Table column (c) includes the results for this task. We outperform the baseline by 1.3%. The representation that learned by using both attention and movement provides the best performance for this task, which involves predicting the future trajectory of objects.
Walkable Surface Estimation. The goal of this task is to segment the pixels in an image that a person can walk on. We use the data from (Mottaghi et al.) 2016b), which provides annotation for 1449 images of the NYU D2 (Nathan Silberman & Fergus) 2012) dataset. The results are shown in Tablum (d). The variation of our method that uses only the gaze information achieves the highest accuracy. This might be due to the fact that, during walking, human attention is focused on the places that they can walk on. Therefore, the gaze provides sufficient information to perform this task.
Depth Estimation. For depth estimation, the task is to regress the values of the depth for a single monocular RGB image. We use NYU D2 (Nathan Silberman & Fergus] (012) dataset for this task, which provides 1449 densely labeled pairs of RGB and depth images. The results are shown in Table collumm (e). Our learned representation outperforms the baseline for this task as well. Movement cues seem more aligned with the task of depth estimation, and the representation embedding with this information performs better. Note that the metrics for depth and walkable surface estimation are global metrics i.e. they are computed for the entire image. Therefore, typically small improvement those metries has significant effect on the qualitative results.
5.3 ABLATIVE ANALYSES
We ablate results by replacing the InfoNCE visual loss with autoencoder loss. Additionally, we show how gaze information affects the prediction of the body part movements. Finally, we evaluate which movements serve as an important supervision by masking out subsets of the body parts and retraining the representation.
5.3.1 VISUAL LOSS
As discussed in Section 4.1 we use the InfoNCE loss (Oord et al. 2018) while learning the rep- resentation. order to investigate the impact of using this objective, we learn the representation using an autoencoder loss for our visual objective and evaluate the learned representation five target tasks after re-training using the new backbone. The autoencoder loss, Lae is defined as Lae (d)f 1)) [[)(f(I)) Iellg. where f is the feature extractor backbone and d is de- coder network five up-convolution layers, which receives the 512 x 7 x feature as input and reconstructs 3 56 x 56 image.
all
7

---

Published as a conference paper at ICLR 2021
Prediction
Avg. Accuracy
Visual + Part Movement
79.19
Visual + Gaze + Part Movement
81.01
Table 3: Body part movement prediction. We investigate the correlation of the movement and attention by using the human gaze additional input to predict the body part movements.
Scene Classification Top-1 f
Action Recognition Top-1 f
Dynamics Prediction Top-1
Walkable Estimation IoU
Depth Estimation RMSElog +
Masked Parts
w/o Torso
21.56
25.42
13.47
57.50
0.143
w/o Neck
21.50
26.25
13.54
56.76
0.148
w/o Arms
20.72
24.97
13.79
58.08
0.148
w/o Legs
21.38
25.65
12.62
57.16
0.147
w/ all
22.82
27.95
14.44
58.38
0.146
Table 4: Ablation of body parts. We show how the performance on the target tasks changes when we ignore body part during representation learning.
Table 2 shows that gaze and movement information still provide a strong signal compared to the visual-only case. However, the results worse than the that the InfoNCE loss for learning the representation.
5.3.2 MOVEMENT ESTIMATION
To better understand the effect of gaze. we predict body part movements with and without gaze information. This experiment is not part of the representation learning experiments. It is just to evaluate whether using gaze provides any additional cue for prediction of the movements.
In this experiment, we predict which subset of the six groups of body parts (neck, torso, left arm, right arm, left leg, right leg) have moved. The overall architecture for this experiment is the same as our representation learning model, except for the inputs to the LSTM modules, which instead concatenated image features from ResNet and the embedded input gaze. The input gaze embedding a two-layer network encoding the gaze into a feature vector of size 512.
Table 3 shows the results for this experiment. The network achieves improvement in predicting the body parts movements by having the additional information of the person's 's center of attention, which can intuitively serve proper indicator of their "intentions".
5.3.3 EFFECTS OF THE BODY PARTS
To evaluate how each body part affects the learned representation, perform experiment where we ignore a subset of body parts, re-train the representation learning (from scratch) and evaluate the features target tasks. Table 4 summarizes the results. The performance on the target tasks (except depth estimation) drops when we ignore body part during representation learning, For depth estimation, removing torso results in slightly lower error, which might indicate that the torso movement is not as helpful for estimating the depth.
CONCLUSION
Representations that encode movements and actions become necessity as we move deeper towards embodied visual understanding. In this paper, we investigate the idea of using human interactions to learn visual representations. To enable this research, we introduce a new dataset human in- teractions which includes hours of synchronized streams of image frames, body part movements, and gaze information across different subjects and activities. We show that representations trained to predict body movements and gaze encode additional information compared to their purely visual counterparts. More specifically, we show our representation outperforms a state-of the-an art self- supervised representation learning baseline for variety of target tasks. Our main point in this paper
8

---

Published as a conference paper at ICLR 2021
to show there signals other than visual information that can be used for improving visual rep- resentation. However, we require additional sensors to capture data from other modalities and we acknowledge that scalability of our dataset requires additional data collection.
ACKNOWLEDGMENTS
This work is in part supported by NSF IIS 1652052, IIS 17303166, DARPA N66001-19-2-4031. DARPA W91 INF-15-1-0545 and gifts from Allen Institute for Artificial Intelligence. We thank Winson Han for helping with making the figures for this paper.
REFERENCES
Karen Adolph and Scott Robinson. Motor Development, volume 2, pp. 113-157. John Wiley & Sons, 2015.
Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Devi Parikh, and Dhruv Batra. Vqa: Visual question answering. IJV,, 2015a.
Pulkit Agrawal, Joao Carreira, and Jitendra Malik. Learning to see by moving. In ICCV, 2015b.
Philip Bachman, Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In NewrlPS, 2019.
Albert Bandura. Social learning theory. Prentice-hall, 1977.
Boyuan Chen, Shuran Song, Hod Lipson, and Carl Vondrick. Visual hide and seek. arXiv, 2019.
Liang-Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI, 2017.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006.10029 2020b.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv, 2020c.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella. Sanja Fidler, Antonino Furnari, Evange- los Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling egocentric vision: The epic-kitchens dataset. In ECCV, 2018.
Dima Damen, Will Price, Evangelos Kazakos, Antonino Furnari, and Giovanni Maria Farinella. Epic-kitchens 2019 challenges report. Technical report, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
R Devon Hjelm and Philip Bachman. Representation learning with video deep infomax. arXiv, 2020.
Carl Doersch, Abhinav Gupta, and Alexei A. Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. ICLR, 2017.
Kiana Ehsani, Hessam Bagherinezhad, Joseph Redmon, Roozbeh Mottaghi, and Ali Farhadi. Who let the dogs out? modeling dog behavior from visual data. CVPR, 2018.
Alireza Fathi, Yin Li, and James M Rehg. Learning to recognize daily actions using gaze. In ECCV, 2012.

---

Published as a conference paper at ICLR 2021
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accu- rate object detection and semantic segmentation. CVPR, 2014.
Daniel Gordon, Kiana Ehsani, Dieter Fox, and Ali Farhadi. Watching the world go by: Representa- tion learning from unlabeled videos. arXiv, 2020.
Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra. Scaling and benchmarking self- supervised visual representation learning. ICCV, 2019.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In AISTATS, 2010.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In CVPR, 2006.
Mohamed Hassan, Vasileios Choutas, Dimitrios Tzionas, and Michael J. Black. Resolving 3d human pose ambiguities with 3d scene constraints. In ICCV, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Olivier Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint arXiv:/905.000 09272,9222,1 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXivv, 2015.
Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019.
Catalin lonescu, Dragos Papava, V. Olaru, and C. Sminchisescu. Human3.6m Large scale datasets and predictive methods for 3d human sensing in natural environments. TPAMI, 2014.
Dinesh Jayaraman and Kristen Grauman. Learning image representations tied to ego-motion. In ICCV, 2015.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Ha--El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Andreas Veit, Serge Belongie, Victor Gomes, Abhi- nav Gupta, Chen Sun, Gal Chechik, David Cai, Zheyun Feng, Dhyanesh Narayanan, and Kevin Murphy. Openimages: public dataset for large-scale multi-label and multi-class image classifi- cation. Dataset available from http:///gihuub.ooooopeinet
2017.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.
Feature pyramid networks for object detection. In CVPR, 2017.
David G Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60(2):91-110, 2004.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits weakly supervised pretraining. In ECCV, 2018.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of tions. In CVPR, 2020.
pretext-invane invariant representa-
10

---

Published as a conference paper at ICLR 2021
Roozbeh Mottaghi, Hessam Bagherinezhad, Mohammad Rastegari, and Ali Farhadi. Newtonian scene understanding: Unfolding the dynamics of objects in static images. In CVPR, 2016a.
Roozbeh Mottaghi, Hannaneh Hajishirzi, and Ali Farhadi. A task-oriented approach for cost- sensitive recognition. In CVPR, 2016b.
Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. ECCV, 2012.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. ECCV, 2016.
Mehdi Noroozi, Hamed Pirsiavash, and Paolo Favaro. Representation learning by learning to count. In ICCV, 2017.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv, 2018.
Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei Efros. Context en- coders: Feature learning by inpainting. In CVPR, 2016.
Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Darrell, and Bharath Hariharan. Learning fea- tures by watching objects move. CVPR, 2017.
Lerrel Pinto, Dhiraj Gandhi, Yuanfeng Han, Yong-Lae Park, and Abhinav Gupta. The curious robot: Learning visual representations via physical interactions. In ECCV, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NeurlPS, 2015.
Evan Shelhamer, Jonathan Long, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
Wenzhe Shi, Jose Caballero, Ferene Huszár, Johannes Totz, Andrew Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using efficient sub-pixel convolutional neural network. In CVPR, 2016.
Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteck Alahari. Charades-cgo: A large-scale dataset of paired third and first person videos. arXiv, 2018
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv, 2019.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Anticipating visual representations from unlabeled video. In CVPR, 2016.
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In ICCV, 2015.
Luca Weihs, Aniruddha Kembhavi, Winson Han, Alvaro Herrasti. Eric Kolve, Dustin Schwenk, Roozbeh Mottaghi, and Ali Farhadi. Artificial agents learn flexible visual representations by playing a hiding game. arXiv, 2019.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In CVPR, 2018.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba, Sun database: Large-scale scene recognition from abbey zoo. CVPR, 2010.
Yanyu Xu, Yanbing Dong, Junru Wu, Zhengzhong Sun, Zhiru Shi, Jingyi Yu, and Shenghua Gao. Gaze prediction in dynamic 360 immersive videos. In CVPR, 2018.
Ting Yao, Yiheng Zhang, Zhaofan Qiu, Yingwei Pan, and Tao Mei. Seco: Exploring sequence supervision for unsupervised representation learning- arXiv, 2020.

---

Published as a conference paper at ICLR 2021
Amir R. Zamir, Alexander Sax. William Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In CVPR, 2018.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV, 2016.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. 7PAMI, 2017.
Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. In
12

---

Published as a conference paper at ICLR 2021
APPENDIX
A.1 DATASET EXAMPLES
The supplementary video provides examples of our dataset and also qualitative results our repre- sentation learning model.
A.2
DATA COLLECTION DETAILS
We describe the details of our hardware setup and the alignment method used synchronize the recordings between our camera, gaze tracker, and movement sensors.
A.2.1 ALIGNMENT AND SYNCHRONIZATION OF DEVICES
There are three different devices in setup, 1) Tobii Pro eye-tracking glasses record gaze, 2) BNO055 IMU sensors to record movements, and 3) GoPro camera attached to the forehead to cap- ture ego-centric videos. These different devices record data independently. Therefore, it is necessary to synchronize all recordings.
Gaze Tracker and GoPro Alignment. Tobii Pro2 eye-tracking captures a video and the center of the gaze in the camera frame. Due the low quality of the video captured by the eye-tracking glasses, we use an additional high-quality GoPro hero 6 camera (with resolution 1920 x 1080 and 60 frames per second). To synchronize the videos from the gaze tracker and GoPro, we extract SIFT (Lowe) 2004) features and use a brute force algorithm for feature matching and the RANSAC method to find homography that maps the gaze from Tobii's camera frame GoPro's camera coordinate. Note that the gaze might be missing for some frames due to the device noise.
IMU Sensors and GoPro Synchronization. The outputs of the IMU sensors are recorded on Rasp- berry Pi board. There is also a microphone on the Raspberry Pi that records the audio. We synchro- nize the IMU and video recordings using two methods, 1) synchronize the audio from the GoPro video and the voice recording on the Raspberry Pi board, and 2) repeat a specific pattern of body movements in front of mirror, it can be uniquely identified in both the movements depicted by the IMU sensors and the GoPro camera (which recorded the participant's body pose in the mirror).
A.2.2 MOVEMENT CALCULATIONS
We record the body part movements using BNO055 Inertial Measurement Units (IMUs) in 10 differ- ent locations (torso, neck, 2 triceps, 2 forearms, thighs, and legs). The body parts may not appear in ego-centric video frames, therefore, the task of predicting the exact orientation and location of body part (c.g., arm) using ego-centric videos can be very challenging. We train the model using the simpler task of predicting whether part has moved or not. This still contains rich information about the action that is happening in the video, for example, walking can be defined as periodic movements of the left and right legs.
To compute the loss function, we need to distinguish between movement and no movement in the dataset. One way is finding a threshold in the domain of the angles of the part movements, and label all the moves smaller than the threshold as по movement and the rest as movement. However, this might result in ambiguities for the movements close to the threshold, and the network might over penalize the wrong predictions in the neighborhood close to the threshold. Therefore, we add a third gray area label, where the network is not penalized for wrong predictions. We divide the range of movements for each sensor into three equal ranges, the first 33% is labeled as no movement, the last 33% is labeled as movement and the remaining interval is the gray area, for which there is no penalty for misprediction.
A.3 DATASET ANALYSIS
To ensure that the videos in the dataset consist of wide range of activities, we do not provide any specific instructions to the subjects, and we ask them to perform their daily routine activities. Hence, the dataset includes variety of different situations including but not limited to driving.
13

---

Published as a conference paper at ICLR 2021
Approrimate Scone Distribution
Gaze Distribution in Frame
Changes in Rotation Between Consecutive frames
www. Tr
dogress
Phyte Jiberiayy
raral irt,
modeess, siggooriac
beany p0bs
Wigh
un
bovll
Figure 4: Dataset Statistics. Left: The approximate distribution top 20 scene classes according to scene classifier trained on the Places (Zhou et al. (0177) dataset. We show how often each scene category is predicted as top-1. Middle: The distribution of gaze across the dataset. Right: The average magnitude of the change in the orientation of body parts between consecutive frames.
cycling, playing pool, cooking, cleaning, walking in the streets, and shoveling snow. Purely for dataset analysis purposes, we gather proxy scene labels for each image. We use an off-the-shelf scene classification model trained on the Places dataset (Zhou et al. 2017) and record the top-1 prediction for each frame of our dataset. Many scene categories in our data are not present in Places, so we see a moderate amount of misclassification. However, the classifier confidently (more than 70%) predicts 101 of the 365 classes exist somewhere within our dataset, showing the diversity of our data. Figure Left shows the 20 most frequent predictions. Even though there are some mispredictions among them (such as jail cell which is frequently mistaken for dark rooms), observe that our data is fairly diverse.
Figure Middle shows the distribution of the gaze in the images. As expected, the focus is mostly in the center of the image. In Figure[Riigtt we show the change in the orientation of the body parts between consecutive frames. We observe more movements in the limbs compared to the torso and neck. We additionally notice more right arm movement than the left which is likely caused by more of participants being right-handed.
A.4 ARCHITECTURE & TRAINING DETAILS
In this section, describe the details of network architectures well the hyperparameters and optimization methods that we used, for reproducibility purposes. Also, the code and data will be made publicly available for further research.
A.4.1 BACKBONE NETWORK
We train the feature extractor network by using sequence of images of length k: 5, which th of a second apart, as input. We use the ResNet18 (He et al. (016)) convolution layers as the feature extraction backbone. To preserve spatial information, which essential for gaze prediction, we use the 512 x 7 x 7 features before average pooling. We then add 1 X 1 convolutional layer on top reduce the feature size to 64 x 7 x The flattened feature is then input to 3 layer LSTM with hidden size 512, which encodes the input video into hidden feature vector. Next, the embedded video feature vector is decoded using 3 layer LSTM, to predict the binary movement vector and gaze. For Luiswat, we use the feature size 128 (obtained by a fully connected layer on top of the ResNet18 features) and a memory bank of size 16384. We choose å = 1 in Equation [ 0.09, 3 0.01, 0.9 in Equation 3 and - 0.07 in Equation 2 We use dropout of 0.5 and weight decay 0.1. For data augmentation, we only use color jitter and random flip. We flip the entire sequence of images, swap the part movements for right and left arms and legs, and calculate the updated gaze in the flipped images.
A.4.2 TARGET TASK NETWORKS
Shared Implementation Details. During the target task training, the weights for the backbone are frozen. For all of experiments, use the Adam optimizer (Kingma & Ba 2015) and images
14

---

Published as a conference paper at ICLR 2021
reshaped to 224 224. The size of the hidden layer in our LSTM in all temporal experiments is 512. We use leaky-Redl ReLU non-linearities between all network layers except for LSTMs.
Self-supervised Baseline Details. When training the MoCo encoder network, optimizer with 0.03 learning rate for the MoCo baseline since it performs best in this setting. For data augmentation,
the SGD
random cropping, horizontal flipping, gray scaling, and color jittering with the same parameters used in that work. We train the baseline with batch size 256 on 8 GPUs for 200 epochs with the same training regime as the original work.
Scene Classification, We use a decoder network of a single x convolution layer that reduces the feature size from 512 7 x 7 to 64 7 x 7, followed by two fully connected layers, that convert these features to a vector of size 512 and then 397. We use the cross-entropy loss for training, and evaluate using mean per class top-1 accuracy.
Action Recognition. For this task, we train a single 1 X 1 convolution layer that reduces the feature size from 512 x x 64 7 x 7, followed by an LSTM to embed the video in one hidden vector of size 512, and two fully connected layers that convert the features to a vector of size 200 and then to the number of actions. As before, we use the cross-entropy loss as the objective and evaluate with mean per class top-1 I accuracy. Since some of the action classes in this dataset appear in limited set of videos, following one of the EPIC challenge finalists (Damen et al. 2019), we choose verbs that result in state transition, namely, take, put, open, close, wash, cut, mix. pour and peel and ignore verbs that do not cause state transition (c.g., check).
Dynamic Prediction. We create binary rectangular mask using the object bounding box. We use this mask image as the input to a two-layer convolutional network. As the result, we obtain a feature of size 64 x x 7. These two convolutional layers are trained for both method and the baseline. We concatenate the mask feature with the feature vector obtained from the image and add two fully convolutional layers on top, obtain the class labels. Again, optimize the network using the cross-entropy loss and use mean per class top-1 accuracy as the evaluation metric.
Depth Estimation. The ResNet backbone connected to Feature Pyramid Network (FPN) (Lin) iet al. 2017). We use Pixel Shuffle layers (Shi et al. 2016)) for up-scaling the lower level features. The learned ResNet backbone frozen; the 5 up-convolution layers are the only layers trained for the target task. We use the Huber loss as the objective.
Walkable Surface Estimation. The architecture this network is the same as the depth estimation network. The ResNet backbone is frozen and five up-convolution layers are the only layers that are trained for the target task. We use the binary cross-entropy loss the objective, where the goal is segmenting walkable and non-walkable pixels. For evaluation, we use the standard Intersection over Union (IOU) metric for segmentation tasks.
A.5 RESULT OF FULL SUPERVISION
As point of reference, also provide the results using fully supervised backbone that is trained using ImageNet. Neither our method nor the purely visual baselines use any supervision for repre- sentation learning. Therefore, direct comparison is not fair. The results are shown in Table 5 The
corresponding self-supervised results are shown in Table I
Datasets
SUN397
Epic Kitchen
VIND
NYUv2
Nia all 2010
Damen et 2018
Monaghi et
2016
Nothon Silbemon Forgun 2012
Method
Training
(a) Scene
(b) Action
(c) Dynamics
(d) Walkable
(c) Depth (RMSElog J)
Objective
(Top-1 t)
(Top-1 1))
(Top-1 f)
(IOU 1>
MoCo (e 2020
InfoNCE
18 47.27
30.28
16.37
63.95
0.135
Supervised
Classification
32.09
20.01
65.80
0.132
Table 5: Results of pre-training with ImageNet. We provide the results of full supervision for pre-training using ImageNet and also the MoCo model trained on ImageNet data.
15


Reviews:
[{"review_id": "nKbMstJujAI", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an interesting video dataset with body-part movement signal, gaze signal. And they also treat these signals together with infoNCE/or AE as self-supervised signals. With random initialization on ResNet18, their model is better than the model only with infoNCE/AE. \n\nI agree with the author that their supervision is working and working together with infoNCE. However, this supervision does not come for free like Rotation/Speed/Contrastive learning. For example, how do you apply your signal on the same 'egocentric' video like 'Epic-kitchen' dataset?\n\nPros: a new dataset contributing to the 'affordance' society, a novel and useful supervision signal.\n\nCons: Even though the signal is useful, it does not come for free like SpeedNet, Rotationnet or infoNCE, they still need to capture the information when recording the data.\n\nIf the author can make the supervision more free?(self-supervision), I can change my rating.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "U_W5QKLhvhB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to improve upon unsupervised representation learning for various downstream vision tasks by leveraging human motion and attention (gaze) information. The authors collect a large spatio-temporal dataset with gaze and body motion labels for this task. They train a network to jointly predict the visual focus of attention in scenes and body motion besides visual instance recognition via an NCE loss to learn good visual representations. They show large improvements in accuracy of many different visual recognition downstream tasks with their approach versus the SOTA MOCO approach, which uses visual information only.\n\nPros:\nThe work is novel and considers a new dimension to solving the problem of representation learning, which hasn't been explored before. It explores supervising neural networks to predict humans' motion and visual focus of attention. This is biological motivated by human beings' similar learning strategies. The novel datatset containing both gaze and body motion labels can be useful to the research community for other tasks beyond visual representation learning. The authors show consistent improvements with their proposed method of incorporating knowledge of gaze and body motion versus using visual information only for many downstream tasks they consider. The experimental section is fairly thorough (except for an important missing experiment as explained below). \n\nCons:\nThe authors argue that one of the disadvantages of the current approaches for unsupervised representation learning is that they use ImageNet-type datasets that require significant effort for curation and cleanup. In contrast to this, the authors' proposed approach of requiring large amounts of data with expensive TOBII eye-tracking glasses and body-IMUs from multiple subjects and with special calibration and syntonization procedures seems even more cumbersome and less accessible to ordinary practitioners of AI in the real world. How do the authors justify this? To clearly show the superiority of their dataset versus ImageNet, the authors should also include the results of MOCO trained on ImageNet for each of downstream tasks shown in their paper.\n------\n\nPost Rebuttal:\nI thank the authors for their response and additional experiments to show the performance of MOCO trained on ImageNet for the various downstream tasks considered in this paper. It is evident from the results that the authors presented in Table 5 that their best method (using their multimodal data) performs worse than MOCO trained only on ImageNet with InfoNCE. Hence, while this current work has some interesting novel insights of theoretical value, I don't think the complete proposed method of data collection and training is very practical or broadly scalable. It is likely to be of limited practical applicability.  In contrast to the authors' proposed cumbersome method of collecting annotated data using expensive gaze and motion sensors, cell phones and cameras are nowadays ubiquitous and image and video data is routinely uploaded to the internet by users all over the world. Using such abundantly available existing data on the web, which can often times simply be downloaded for free without any annotations, is what I believe is likely to be a much more practical and broadly applicable approach to solving the problem of representation learning via self-supervision. This concern is also shared by Reviewer 3. \n\nOn weighing the various pros and cons of the proposed approach, I will maintain my previous rating.", "extra_scores": {}, "extra_text": {}}, {"review_id": "HfT4A1qCpH9", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The main aim of the paper is to make use of human interaction/motion to learn a visual  representation that can be re-used for classic visual tasks such as depth estimation. The authors claim that by encoding interaction and attention cues in the self-supervised representation, the method can outperform visual-only state-of-the-art methods. To study the interaction element, the authors attach sensors like Inertial Movement Units (IMUs) to the limbs of subjects and monitor their reaction to visual events in daily life. The paper also introduces a new dataset of 4260 minutes of human interactions by 35 participants which include synchronized streams of images, body part movements, and gaze information.\n\nPaper Strengths:\nThis is a simply awesome paper. Idea is novel, well-validated, and well-written. The result is strong.\n+ Novel intuition: The idea of the paper is intuitive, where it proposes to incorporate body part movements and gaze information in learning visual representations. Attention does play an impact in many tasks like action recognition and scene classification, which might benefit from the proposed representation learning. Also, in case of tasks like depth estimation and future prediction of dynamics, it is insightful to use body movement since it encodes temporal changes.\n+ Experimental setup and ablation studies: The intuition of the authors to incorporate body part movements and gaze information in their representation has been well justified by the experimental setups and ablation studies. The importance of using each objective in the representation learning has been effectively demonstrated by showing its impact on various target tasks.\n+ Performance: The authors have shown that the representation, trained using movement and attention supervision, outperforms the visual-only representations in all the tasks mentioned in the paper by a range of 1.3% to 7%\n+ Dataset: The paper also introduces a new dataset of 4260 minutes of human interactions by 35 participants. This is a novel dataset with synchronized streams of images, body part movements and gaze information.\n\nPaper Weaknesses:\n- Objective function selection: In the ablation studies of body parts, we see how the removal of a body part can affect the performance on the target tasks. However, it is still not clear how each individual part may fare since the performances do not vary greatly from each other. The exclusion of the torso results in a lower error for the depth estimation, but an explanation for why exactly would that be the case may be beneficial, since intuitively it may seem that the complete movement of the body should result in better performance.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "3yD8e2U9rcU", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n# Paper Summary \n\nThe paper uses a combination of visual, human gaze and human motion sensors to build representations that perform better on downstream tasks such as action recognition, physics prediction and depth estimation than representations extracted from solely visual input. The paper announces the release a new data set of aligned visual images, eye gaze fixations and IMU motion readings from test subjects walking around an environment. Representations are computed using three different forms of information simultaneously. Given a visual input, the system tries to predict the location of eye gaze in image frame coordinates, whether each of 6 groups of motion detectors are active or not (head, torso, legs, etc.) and the result of a more traditional auxiliary visual pretext task. In this work, the paper uses “instance discrimination” where representations of augmented versions of a specific image are pushed close together in latent space and far away from augmentations of other images. Tests on diverse benchmarks show that the gaze and motion prediction improve over visual pretext tasks alone and that there is a small benefit to using both together, but it is not additive. The paper also shows the benefit of gaze and motion is present for two different visual auxiliary tasks.\n\n\n# Pros and Cons \n\nThe paper defines a new way of obtaining self-supervised representations for a variety of core computer vision tasks which is important and highly relevant to the ICML community. \n\nThe paper clearly describes prior work and situates its contributions well within this space. \n\nThe paper evaluates the proposed augmented loss function on a diverse collection of standard benchmarks to test their hypothesis including scene classification, action recognition, future motion, walkable surfaces and depth estimation. \n\nTable 1 provides a clear ablation study showing that predicting gaze and motion improve downstream performance on diverse benchmarks by non-trivial amounts:  ~7% , 3.5%, 1%, 1% and reduce RMSE  on depth estimation by 2% or so.  \n\nIn the first benchmark, adding either attention or movement increases results by 6%, but adding both only improves results by 7%, not 14%! This suggests that there is a lot of redundant information between gaze and motion sensors which is a surprising as superficially they seem like very different modalities. \n\nIt is interesting and surprising that eye-gaze, where a person is looking in an image, is a useful feature for tasks such as depth prediction. This suggests there are features that are easy to compute (compared to interactions between robots and physical world)  that significantly improve network intuitions about what is important in images.  \n\nFrom table 4, it seems like Torso movement is more informative for scene classification, neck more informative for action recognition, arms for dynamics and arms for walkability. It is interesting that different parts are more or less informative for different tasks. \n \nI am a little unsure what inference to draw about the results in table 2 (experiments on Lnce vs. Lae). I guess the main point is that the addition of gaze and motion also improves Lae results, so the effect is not specific to Lnce? It is interesting that gaze and motion are not as helpful with Lae ( 3% gain vs 7% with Lnce) but it is not clear why. Why would there be better complementarity between gaze, motion and NCE? Some thoughts on this would be interesting. \n\n\n# Recommendation  \n\nAccept - it is an interesting and surprising result that adding self-supervised tasks for other modalities significantly improves self-supervised representations and that gaze and motion are redundant with each other.  \n\n \n# Questions  \n\nHow are alpha, beta and gamma set? Hmm, see from appendix A.4.1 these are 0.09, 0.01 and 0.9 respectively indicating that visual loss is the primary driver here and the others are acting more like regularizations on the representation. It is a bit surprising that the other modalities have so large an effect given their small weights in the loss function. Were any other forms of regularization used such as dropout or weight decay?  Didn’t see anything about this in the appendix. \n\nSection 5.3.2 It seems surprising that gaze, a 2 dimensional quality needs a 512 D embedding. Is this some sort of one-hot encoding over a matrix of locations? \n\n \n# Feedback  \n\nSection 3: “feature extractor to embed a more detailed representation” … I don’t know if it is more detailed, but hopefully, it is more invariant, at least to the augmentations applied, ideally to all non-semantic attributes of the image. \n\nI guess it is implied that the baseline ‘vis’ approach in the paper  is identical to the ‘vis’ in He et al 2020, but it would be nice if this was explicitly in the caption for Table 1.  \n\n ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an attempt to learn interaction-based representations by taking advantage of body part movements and gaze attention. Video representations are learned by benefiting from additional supervisory signals, which are not the ones commonly used, making the paper more interesting.

R3 expresses a concern that the supervisory signal does not come "for free" and that the paper is misleading. The ACs do agree with R3 that the paper benefits from additional signals and is not a pure self-supervised learning paper, strictly speaking. The authors also agreed to this in their response to the R3’s comment. R1 also mentioned (after the rebuttal phase) that the proposed approach is not a practical self-supervised learning solution and that it does not perform as effectively as conventional self-supervised learning methods like InfoNCE on Moco.

Simultaneously, the AC and the majority of the reviewers believe that the paper itself has a value as a multi-modal learning paper. We strongly suggest the authors revise the paper to remove the 'self-supervision' claim. As mentioned above, the paper is not a self-supervised learning paper and the authors are asked to correct the details of the paper to reflect this. We also recommend adding analysis on each body signal qualitatively in the final manuscript, as suggested by R4.

It will be great if the authors can consider this as a "conditional accept". In particular, the 'self-supervision' claim in the current version of the paper is misleading and this must be corrected in the final version. Note that this was also pointed out by the Program Chairs.
Decision: Accept (Poster)