EgoDistill: Egocentric Head Motion Distillation for Efficient Video Understanding
Authors: Shuhan Tan
Tushar Nagarajan
Kristen Grauman
Venue: NeurIPS 2023 poster (2023)
Keywords: Egocentric Video; IMU; Efficient Video Understanding

Abstract: Recent advances in egocentric video understanding models are promising, but their heavy computational expense is a barrier for many real-world applications. To address this challenge, we propose EgoDistill, a distillation-based approach that learns to reconstruct heavy ego-centric video clip features by combining the semantics from a sparse set of video frames with head motion from lightweight IMU readings. We further devise a novel IMU-based self-supervised pretraining strategy. Our method leads to significant improvements in efficiency, requiring 200× fewer GFLOPs than equivalent video models. We demonstrate its effectiveness on the Ego4D and EPIC- Kitchens datasets, where our method outperforms state-of-the-art efficient video understanding methods.

EgoDistill: Egocentric Head Motion Distillation for Efficient Video Understanding
Shuhan Tan', Tushar Nagarajan', Kristen Grauman 1.2 University of Texas at Austin, FFAIR, Meta
Abstract
Recent advances in egocentric video understanding models promising, but their heavy computational expense a barrier for many real-world applications. To address this challenge, we propose EgoDistill, distillation-based approach that learns to reconstruct heavy egocentric video clip features by combining the semanties from sparse set of video frames with the head motion from lightweight IMU readings. We further devise a novel self-supervised training strategy for IMU feature learning. Our method leads to significant improvements in efficiency, requiring 200x fewer GFLOPs than equivalent video models. We demonstrate its effectiveness on the Ego4D and outperforms tfte-offthhe-rr efficient video understanding methods. Project page: https://vist cs. utexas. edu/projjects/egddist::l
datasets, where our method
Introduction
Recent advances in augmented and virtual reality (AR/VR) technology have the potential change the way people interact with the digital world, much like the smartphone did in the previous decade. A fundamental requirement for AR/VR systems the ability to recognize video captured from a head-mounted camera. Towards this goal, several egocentric video datasets have been proposed in recent years, spurring increasing attention of the research community [26, 56. 11].
behavior from egocentric
Current egocentric video understanding models use powerful clip-based video backbones that operate video clips of a few seconds time [18, 54, 25, 16, 12, 55, 43, 44]. Despite encouraging performance, these models typically process densely sampled frames with temporally-awe tions, making them computationally heavy. This makes them impractical for AR/VR devices with constrained resources, or for real-time video applications that require low latency. How to efficiently perform egocentric video understanding is therefore important, yet unsolved problem.
opera-
To address this issue, we take inspiration from how animals perceive the world with ego-motion. Neuroscience research has found that during active movement, the animal visual cortex receives and encodes head motion signals from the motor cortex for visual processing [27, 52, 53], highlighting its role in the efficient understanding of the animal's 's visual stream. Inspired by this phenomenon, explore the relationship between human head motion and ego-video for efficient video understanding.
In practice, we consider head motion signals captured by the inertial measurement unit (IMU) of head- mounted camera. IMU measures motion from an accelerometer and gyroscope and is widely available popular wearable devices. Prior work leverages IMU as extra modality for human action recognition [68, 69, 13] (e.g., jumping, walking), or as geometric cues for visual-inertial odometry [7, 20, 71]. In contrast, we propose to drive efficient video understanding by drawing on IMU as a substitute for dense video frame observations. The intuition as follows. A video clip contains two things: semantic content (appearance of objects, places, people) and dynamics (how the scene and the camera move). While densely sampled frames are sure to capture both the above-as as done by current clip models [54, 16, 17] hypothesize they are sometimes overkill. For short
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Video Frame
Camera motion (IMU)
EgoDistil
22
Figure 1: Illustration of EgoDistill. Given single video frame and camera motion from IMU, EgoDistill learns to reconstruct the more expensive dense video clip feature. With its lightweight input, EgoDistill significantly improves efficiency.
video clip, much of the semantic content is intelligible from even single frame; meanwhile, the head motion provides good portion of the dynamics, implicitly revealing how the visual appearance changes across neighboring frames.
Building on this insight, we introduce EgoDistill, an approach that learns to reconstruct dense egocentric video clip features using temporally sparse visual observations (as few one RGB frame) together with the head motion from IMU. Specifically, EgoDistill employs a new form knowledge distillation from video models. During training, we train lightweight model that takes sparsely sampled image(s) and IMU to approximate video features extracted by a powerful but expensive video model. We further improve the model with novel IMU-guided self-supervised training stage. During inference, we directly utilize the lightweight model for egocentric video recognition, leading much higher efficiency. Our model is flexible the target heavy video feature, as we demonstrate with multiple current leading egocentric video models [54, 16, 18, 17J. See Figure 1.
Importantly, EgoDistill offers major upgrade in efficiency. Low-dimensional IMU and few frames are much more efficient to process than a dense stack of frames. In practice, EgoDistill uses 200x approach is economical
fewer GFLOPs than the original video model. Furthermore, GoPro IMU sensor our experiments (BOSCH BMI260) costs only $3 and it is practical most AR/VR devices and phones have in-built IMU sensors, making our method widely adoptable.
the
We experiment on the two largest egocentric action recognition datasets: Ego4D [26] and EPIC- Kitchens- 100 [11]. We show that IMU coupled with image offers better cross-modality knowledge distillation performance than images alone or images with audio. For a typical 50-minute egocentric video, EisDistill reduces model inference time from 25 minutes to 36 seconds. Moreover, with only 1-4 frames, our lightweight distillation model achieves a better accuracy-efficienn than state-of -frthe models for adaptively sampling video content [50, 65]. Notably, surpass the accuracy of even these fast approaches by a large margin while requiring 4-8x less computation.
trade-off
Related Work
IMU for activity recognition. Recent work explores using the IMU sensor on mobile devices for human activity recognition of actions like walking, jumping, or sitting [47, 59, 60, 3, 61]. Normally, these models use IMU sensors mounted on human body joints [60, 9, 42], waist-mounted [41] or in-pocket smartphones [33]. See [64] for survey. Abundant work in video recognition explores ways to learn from RGB coupled with other modalitics-and -adiio [1, 22, 38], optical flow [57, 58, 19] or both [51, 36, but much fewer use IMU [48], and unlike our work, they focus third-person video [68, 69, 13] and do not target model efficiency. Our idea is for IMU to help reconstruct expensive video features, rather than simply fuse IMU with RGB for modal recognition.
IMU for odometry. Inertial odometry aims to estimate position and orientation of the camera- wearer with readings from the IMU sensor. Traditionally, methods rely on IMU double integration [4] or enhancements thereof [40, 5, 35]. Recent data-driven methods automatically learn to perform inertial odometry with supervised [30, 70] self- self-supervised learning [7], combine IMU and

---

visual input for more robust estimates with visual-inertial odometry [20, 71J. While IMU can convey geometric ego-motion our learned model, our goal is produce efficient egocentric video features rather than to output odometry.
Visual feature learning with IMU. IMU is also used to learn better vision features [34, 14, 15, 63], e.g., to encourage image features that are equivariant with ego-motion [34], to predict an IMU- captured body part (leg, hand) [14, 15], or to predict video-IMU correspondence [63], for applications like action recognition [15, 63] and scene understanding [14, 34]. While these results reinforce that IMU can inject embodied motion into visual features, our idea to use head motion infer pretrained video features for speedy video understanding is distinct.
Efficient video recognition. Being crucial for mobile applications, efficient video recognition has received increasing attention in recent years. Several studies focus on designing lightweight architectures [30, 17, 37, 72, 62] by reducing 3D CNN operations across densely-samplle frames. In contrast, our model focuses on inputs with sparsely-sampled frames. As we show in experiments, our method is compatible with different video architectures.
Another line of research achieves efficiency by adaptively selecting video content to process. Some reduce temporal redundancy by adaptively selecting which video clip [39], frames [49, 24], and/or feature channel [50] to process and which to skip, while others reduce spatial redundancy, selecting for each frame smaller but important region to process [66, 67]. Other work dynamically selects tokens in video transformers among both the spatial and temporal dimensions [65]. Our idea is complementary: rather than dynamically subsample the available video content, we show how to infer "full" video features for every clip using static image(s) and motion data. Our results outperform state-of-the -ftt-offthheart sampling models (ef. Sec. 4). In addition, focus egocentric video, where head motion is particularly meaningful for inferring unobserved visual content. To our knowledge, ours is the first technique specifically aimed at accelerating egocentric video processing.
Multimodal distillation. Knowledge distillation aims to transfer knowledge learned by an expensive model to lightweight model [3 Recent work explores multimodal distillation, e.g., transferring from a RGB model to a flow or depth model [23, 28]. from a 3D model to a 2D model [45], or from visual model to audio model [2, 211. More related to work, the Listen ToLook model [22] incorporates both clip subsampling and video-to-audio distillation for fast activity recognition in third-person video. While share similar motivation, our contribution is distinct: explore the relationship between the camera-wearer's ' head motion and RGB signals for egocentric video, and we introduce novel IMU- pretraining strategy. Our experiments show EgoDistill's advantage over Listen ToLook in terms of the speed-accuracy tradeoff on egocentric video datasets. Furthermore, whereas [22] attempts only single video model (R(2+1)D), we show our idea generalizes well to multiple video models, meaning it can be dropped in to benefit multiple popular frameworks.
3 Approach
We introduce EgoDistill, which uses sparsely-sampled frames and head motion from IMU to ap- proximate the features of heavy video models for efficient egocentric video understanding. We first introduce the egocentric action recognition task (Sec. 3.1). Then, we introduce our pipeline (Sec. 3.2), our distillation model and training objective (Sec. 3.3), and our self-supervised IMU feature learning (Sec. 3.4). Figure 2 overviews our approach.
3.1 Egocentric action recognition
Given fixed-length video clip V Tx consisting of T RGB frames of size H x W and a set of C action classes, the task of action recognition is to output a score for each action class, representing its likelihood. Typically, this is done with powerful but expensive video model L that directly operates on all the available frames to output the C' class logits ?(V) E IRC. Ω is trained with standard classification loss:
LACT Σ Cce((c,
(I)
where V; is the i-th video clip in the dataset, o; is the corresponding ground-truth action label, or is the softmax function, and LeE is cross-entropy loss. Popular video recognition models use clips that are typically- -2 seconds long [54, 16, 18]. For longer videos, scores are averaged across all clips contains to infer the video action label.
3

---

RAC)
(I,,
Image Encodor
Image Enooder
Frame
Frame
sutrux
Image Enooder
PUU Enooderr
Cyere
E.
Video model
Sohres
IMU Ar
dlip
mV
SX(V)
Figure 2: EgoDistill architecture. Left: Self-supervised IMU feature learning. Given start and end frames of a clip, we train the IMU encoder to anticipate visual changes. Right: Video feature distillation with IMU. Given image frame(s) and IMU, along with our pre-trained IMU encoder, our method trains a lightweight model with knowledge distillation to reconstruct the features from heavier video model. When the input includes more than one image frame, the image encoder aggregates frame features temporally with a GRU.
3.2 Efficient video inference with head motion
Processing the video clip V for action recognition is computationally intensive; however, the com- putation cost can be modulated depending on how frames from the clip are used. On the one hand, clip-based models [18, 54, 16, 17] process most (or all) frames in video clip V to achieve strong recognition performance, but come at a high computational cost. On the other hand, frame-leve models [49, 24, 51, 67] only process but suffer a drop in performance as a result. Our goal is to train a frame- based model that can approximate heavy clip-based model performance while maintaining high efficiency.
(or small number) of frames from V and are more efficient,
For this, we turn to head motion captured by IMU. Along with RGB frames, each video clip paired with IMU measurements M that record the camera (head) motion during the video. Specifically, the IMU readings are composed of 6-dimensional accelcrometer and gyroscope mcasurements in the ryz axes, which encode strong temporal motion information about camera pose changes (both translation and rotation) across frames.
For short video clips, a set of sparsely sampled frames I often already captures most appearance information. Complementary to this, the IMU readings capture camera motion information (see below for discussion on scene motion). Moreover, IMU is very efficient to process due to its low dimensionality. By processing inputs from these two sources with lightweight frame-based model, we can infer the semantic and dynamic features of heavier clip-based video model.
Given I and M, train an efficient lightweight model & to approximate the output of video model Ω. Specifically, we train our EgoDistill model d that achieves
4 M) ~ (2V)..
(2)
Such lightweight model will be able to approximate the result of the heavy video model, while being much more efficient. Our approach is agnostic the specific video model Q; in experiments, we demonstrate its versatility for MotionFormer [54], MVIT [16], SlowFast [18] and X3D [17].
In practice, uniformly sample N frames! from V to obtain I. We achieve trade-off between efficiency and performance by changing the number of frames N. In our experiments we use very low values of N to 4 frames), In the next section, we discuss how we train d.
3.3 Video feature distillation with IMU
We address Equation 2 via knowledge distillation [31], where transfer knowledge learned by the expensive teacher model to lightweight student model $. Next we present the design of & and the training objectives, followed by our self-supervised IMU feature pretraining stage in Sec. 3.4.
Otter frame sampling heuristics (e.g., selecting from the start or center of the video) performed equivalently worse than uniform sampling.

---

We design & to be two-stream model. For a video clip and associated IMU signal (I.M), we extract image features Zz = fI( and IMU features ZM (M(M) using lightweight feature encoders fz. JM respectively. Then, we fuse 2g and 2M with fusion network IT to obtain the fused VisIMU feature 2 = [Tz, 7.M). Finally, a fully-connected layer uses the fused feature to predict class logits ((I, e R. The fused feature 2 contains semantic information from the image frame coupled with complementary motion information from IMU, allowing us to accurately reconstruct the video clip feature. See Figure
We train & with a combination of three losses, as follows. First, we train o to approximate the original video feature Zy from the video model ?::
L1
Σ (Ry, 25,
1zy, Zo
This cross-modal loss encourages the fused feature 2o to match the video feature, i.e., the combined features from the different modalities should match the feature space,
Training with L1 alone does not fully capture the classification output of Ω. Therefore, we also train with knowledge distillation loss:
CKD
Σ De..o((0033/)),0000..MM (IIM.))))
where (V:, I, M,)) represents the i-th clip in the dataset, DxL measures Kl.-divergence between the class logits from the teacher model Q and student model Φ, and is a temperature parameter. Intuitively, LKD casts the output of the video teacher model as a soft target for training the student model. this way, the student model learns to better generalize by mimicking the output distribution of the heavy video model.
Finally, to further encourage the features to preserve elements useful for activity understanding, we also compute an action classification loss:
Ler
Σ o (I,.M.)
where O is the ground-trute truth action label, following Equation 1. The final training loss is combination of these three loss functions:
L= aLDD (1 at) + BL1,
(6)
where a controls the balance between knowledge distillation and activity training [31], and controls the weight for feature space matching.
Critically, processing few image frame(s) and the low-dimensional IMU readings is substantially faster than processing the entire video. Once trained, our model approximates the behavior of the source video model for recognition tasks, with the key benefit of efficient egocentric recognition.
What kind of motion does our model preserve? Video motion decomposes into scene motion (e.g.. how the objects and the camera wearer's hands moving their own), and camera motion (i.e., how the camera wearer is moving their head). By itself, IMU would directly account only for camera motion, scene motion. However, by learning to map from the RGB frame and IMU to the full video feature, we are able to encode predictable scene motions tied to scene content, e.g., how does hand and object movement in subsequent frames relate to the camera wearer's 's head motion. Moreover, our model is applied to relatively short clips (1-2 seconds) in sequence, which means the appearance content is regularly refreshed as we slide down to process the longer video.
3.4 Self-supervised IMU feature learning
The success of EgoDistill depends on how well the IMU feature encoder f. extracts useful camera motion information and associates it with the visual appearance change in the video clip. In this way EgoDistill can learn to anticipate unseen visual changes in the video with I and M. We design a self-supervised pretraining task to initialize the weights of fM to achieve this.
Specifically, for each clip V. we obtain its first and last frames (I as well as the IMU M. We first extract visual features zZ, z and IMU feature ZM with feature extractors fi and fM
5

---

mentioned above. Then, we train feature predictor h to predict the IMU feature 2M = h(zg. z By connecting a- a function of image features only-with Z,M. we encourage fM extract useful camera motion features specifically associated with the visual appearance changes. Note that those appearance changes may include scene motion. Therefore, we include an L1 loss to train SM. which encourages f' to extract motion features accounting for scene motion in the full video.
We train fM, h, and the fusion network II using Li and NCE loss [29]: pretrain
LNCE + L1, where
Sim(2M,.
LNCE Σ
Log
log Σj sim(a..: RM,)
We sample negative examples aM; from other instances in the same mini-batch for o i, and sim(g. kt) exp( 9-k
|all / with temperature 0.13.
To summarize, prior the main training stage of Equation 6, we pretrain the IMU feature extractor fM and fusion network II. As we will show below, both pretraining losses result in IMU features that are consistent with visual changes and lead better finetuning performance.
4 Experiments
We evaluate our approach for resource-efficien action recognition.
4.1
Experimental setup
Datasets. We experiment on two large-scale egocentric action recognition datasets. (1) Ego4D [26]
contains 3,670 hours of egocentric videos of people performing diverse tasks (from cooking to farming) across the globe. As action recognition is not part of the original Ego4D benchmark, we construct this task with annotations from the Hands+Objects temporal localization benchmark [26] (see Supp. for details). We include clips with paired IMU and audio", and consider classes with at least 2 labeled instances. This results in 94-class action recognition dataset with 8.5k training vidcos and 3.6k evaluation videos. (2) EPIC-Kitchens [11] contains 100 hours of egocentric videos capturing daily activities in kitchen environments. We use annotations from the action recognition benchmark. Similar to Ego4D, we select videos that have paired IMU and audio data, and split the resulting data by camera-wearer. This results in 62-class action dataset with 29k training videos and 6.2k evaluation videos. For both datasets, we use "verb" labels as the target for action recognition as they are well aligned activity motions.
Evaluation metrics. To measure action recognition performance, we report the per-video top-1 accuracy on the validation set. We densely sample clips from each vidco and average their predictions to compute accuracy. To benchmark efficiency, we measure computational cost with FLOPs (floating- point operations) during inference.
Implementation details. In our main experiments, we use MotionFormer [54] as the video teacher model Ω due to its strong performance for egocentric video. For EPIC-Kitchens, we use the authors' provided checkpoint. For Ego4D, we finetune the above model for 50 epochs with I learning rate and 64 batch size on the training set. We use 16-frame input with sample rate 4. For the student model & we use ResNet-18 as the image backbone /z and backbone JM- The feature fusion module II uses concatenation operation following two-layer fully-com mully-connected layer with hidden dimension 1024. For each video clip, the input image(s) is resized to 224 x 224, and the IMU is a 422> x 6 matrix (around 2 seconds with 198Hz frequency), representing the accelerometer and gyroscope readings along the zyz axes. For the image input, we uniformly sample N frames from the video clip. If N 1, frame and aggregate them with a GRU module [10]. For both datasets, we first pretrain the model with the self superviedd objective (Sec. 3.4) for 50 epochs with Adam [46] using batch size 64 and learning rate -4 Then, we finetune all the models with the same setting (Equation 6). We set 0.95 and 3 1.0 based validation data. For Ego4D, we 7 10.0 and train the model for 150 epochs. For EPIC-Kitchens, we set r = 1.0 and train for 50 epochs.
Dilated CNN [6] for the IMU
fz sequentially generate features for each
2 We keep the Irattnnet-reetraineed fz model frozen, as finetuning leads to mode collapse.
We require audio compare with the audio-based baseline [22].

---

Ego4D
EPIC-Kitchena
38
TiC-ktcchnn accurs-cy (84)
50
37
acounay
45
36
35
34
35
33
30
8
10
12
14
16
2
8
10
12
14
16
Inforence cast per video clip (GFLOPs)
Inference cost per video clip (GFLOPs)
EgoDDssiill (ours)
AdaFuse 150]
STIS (65)
LitenTolook 1221
Vistnly-Distil
ViIMU
VisDnly
Figure 3: Accuracy vs. efficiency for action recognition on Ego4D (left) and EPIC-
Kitchens (right). EgoDistill outperforms state-of-the-art efficient video recognition methods that adaptively sample video content, while using 4x 8x fewer GFLOPs.
4.2 Baselines
We compare with the following methods: (1) AdaFuse [50] trains a lightweight policy network to adaptively compute (or skip) feature map channels for each frame during inference. We use the AdaFuseR50 TSN model with the provided syper-parameters. (2) STTS [65] trains module to rank spatio-temporal tokens from videos in transformer-basee model, and selects only the top-K tokens to speed up inference. We use MVīT-B16 backbone [17]. (3) Listen'ToLook [22] uses the audio-based distillation module from [22] following the same audio processing and model architecture.
These methods represent recent advances in efficient video recogmition models. AdaFuse represents sfateoof-the-ar art approaches that achieve efficiency by reducing temporal redundancy in CNN models, STTS is one of the most recent approaches that efficiently reduces both spatial and temporal redun- dancy in VIT models, which achieves the state-of-the-an on Kinectics-400 [8]. Listen'Tol also relies on distillation, but using audio rather than head motion. For each model we generate multiple versions with different computation budgets to plot accuracy vs. GFLOPs. We train all AdaFuse and STTS models with 4 input frames to align with the maximum frames used by our model. For AdaFuse, we use the only provided pppee-paraetter in the paper.* 4 For STTS, we use three provided variants: To TS.. To. Sp.9 and the full model without token selection. For LostenToLook we adopt the same afficiency-accuracy trade-off as our method, i.e., varying the number of input frames.
We also test variants of method: (1) VisOnly-Distill is model without the IMU branch and fusion layer, but trained with the same loss function. This model reveals the importance IMU for distillation. (2) VisIMU is our model trained with only Ler in Equation 5. It shows the effectiveness of distillation from the video model compared with directly training the features with action labels. (3) VisOnly is an image-only model trained with LGr, which serves as the baseline.
4.3
Main Results
Importance of IMU-guided distillation. Figure shows the accuracy vs. efficiency curves. Methods towards the top-left of the plot represent those with both high accuracy and efficiency. Our method achieves good accuracy with low computational cost. Specifically, on EPIC-Kitchens, when = 1, EgoDistill improves over VisOnly-Distill by 8.4% with only small increase in computation. This shows the effectiveness of IMU for reconstructing egocentric video features. Compared to VisIMU, EgoDistill improves by 9.9%. showing the effectiveness of knowledge distillation from the video model. Importantly, this reveals that EgoDistill does not simply benefit from the extra IMU context; our idea to approximate video features is necessary for best results. We see similar results on Ego4D.
Comparison with the state of the art Figure also shows that EgoDistill achieves better accuracy with less computation than existing efficient video recognition models AdaFuse [50]. STTS [65], and ListenTooood [22]. With N 4 frames, EgoDistill surpasses STTS by 7.4% and AdaFuse by 4.2%
with 2x fewer GFLOPs, and surpasses both methods by 2.1% Ego4D. When we use fewer frames, Eististill can still outperform STTS and AdaFuse using 4x to 8x fewer
Modifyyng parametrass to control the -afuaccyefffciencyy trade-of off results in unstable training and unreliable performance.
7

---

LxD
L1
Lat
L pretrain
Excr-preitain preirain
Ego4D
EPIC FitchKitheens
34.15
35.04
35.51
37.71
42.20
37.46
36.99
41.21
37.26
/
37.49
43.51
4
37.95
44.95
Table 1: Ablation study of EgoDistill'ss model components. Accuracy with N
Ego4D EgoDistill
EPIC- Kitches EisDistill
Source Model
Vidco
VisOnly-D
Video
VisOnly-D
MFarmer [54]
46.38
37.95
34.32
77.28
44.95
37,20
MVFT [16]
40.32
36.46
33.40
53.38
36.90
31.22
[18]
40.52
33.47
X3D [17] 37,56
33.57
32.90
52.28
36.34
31.71
Table 2: Versatility to model architectures. EgoDistill outperforms the baselines for multiple common architectures, showing the generality of our idea. "Video" refers to the more expensive source model, essentially the upper bound for accuracy. We show the model accuracy under N = 1.
GFLOPs. In addition, EgoDistill surpasses ListenToLook by 7.4% and 2.9% on EPIC-Kitchens and Ego4D respectively, which suggests that head motion more informative than audio for feature reconstruction in egocentric video.
4.4 Analysis
Model component ablations. Table ablates different design choices in our model, setting N = 1 for all experiments. We observe that training EgoDistill without L1, LKD or LGT deteriorates performance. Specifically, training without LKD Icads the largest performance drop, which indicates that knowledge distillation is an essential component in our approach. Training without Li also degrades performance, which shows the importance of our idea align features from the different modalities. Further, our self-supervised pretraining stage is very effective at training the IMU extractor to encode useful motion information that is consistent with visual feature change. We find an alternative IMU extractor pretraining strategy supervised action recognition task inferior; we outperform pretaining with action recognition by 3.74% EPIC-Kitchens and 0.96% on Ego4D, Finally, we compare with a model that simply does multi-modal recognition with IMU (top row). The IMU predict video model features, as opposed to simply adding IMU as an additional input modality.
strong contrast here indicates the importance of our idea
Impact of teacher video model architecture. In our main experiments as the teacher video model due to its strong performance on egocentric video tasks. To emphasize the generality of our idea, show the performance of EgoDistill with other video teacher architectures in Table 2. Similar to the MotionFormer model, we train these models on each of the labeled datasets, and then train model using the resulting video models the teacher. As expected, better video teacher models lead to better student model performance. More importantly, we observe consistent the VisOnly-Distill baseline both datasets and with different
MotionFormer [54]
improvement by EgoDistill video teacher models, highlighting our idea's generality and versatility.
Efficiency analysis. To compare the efficiency of different models, aside from GFLOPs, also compare their inference run-time and number of parameters. For run-time, we record the time spent to infer single video clip's s label with a single A40 GPU, and take the average time over the full validation datasets of Ego4D and EPIC-Kitchen with batch-size of 32. Table shows the results. EgoDistill runs much faster than the other methods. Notably, it reduces the GFLOPs of MotionFormer by nearly x00 Furthermore, it runs 6.5x faster than STTS [65] while achieving 4.4% higher accuracy EPIC-Kitchens.
Effect of fusing IMU data. EgoDistill achieves superior performance by combing IMU information when distilling features from video models. Here explore how helpful it is to combine IMU

---

GFLOPs
Runtime
Parameterss (M)
Ego4D
EPIC-Kitchens
Method
w/ IMU
w/o IMU
w/ IMU
w/o IMU
Video [54]
369.51
108.9)
AdaPuse [50]
15.20
2.04
38.85
STTS [65]
34.93
34,54
38,38
32.24
STIS [65]
7.19
1.63
36.63
EgoDistill
37.95
34.32
44,95
37.20
ListenToLook [22]
3.10
0.43
25.53
EgoDistill
1.91
0.25
20.56
Table 4: Effect of combining IMU with vi- sion as input. In both EgoDistill and STTS, method is able to combine vision and IMU information more effectively.
Table 3: Efficiency analysis. Our approach is the most efficient. "Video" refers to the original (full-clip) feature. Lower better.
C'EE:C DJD0
doae D:18
Spert DUO
opert 0.34
EgoDistil
VisOnlyDD
Pue 0.60
PUE:014 taloc 0.64
EgoDistil
VisOnly-D
Figure 4: Anticipating scene motion with EgoDistill. For each clip, we show the head motion and video frames. Note, only the center frame (red border) is observed by the model. Action classification scores are shown on the right. EgoDistill can successfully anticipate scene motion and disambiguate the action semantics in the input frame. For example, in the top center frame. the image alone cannot reveal if the door is being opened or closed, whereas our feature, learned with head motion, recovers correlations with the scene motion (i.e., hand motion and door motion) to disambiguate "close" from "open". A similar effect for "put" vs. "take" is seen in the second example.
information with the most recent efficient video understanding method, iee. STTS [65]. S To this end, for STTS, we add an IMU encoding branch identical to the one used in EgoDistill, and concatenate the IMU feature with video features before the last FC layer. We train both the video and IMU branches simultaneously for the action recognition task. Results in Table show that combining IMU features indeed helps STTS, yet EgoDistill's performance is significantly better than STTS's. These results again indicate that our advantage is not simply access to IMU; rather it is the proposed way we leverage vision and IMU. The reason that EgoDistill is able to outperform STTS+IMU is mainly that EgoDistill learns to jointly distill information from a video feature with image and IMU inputs. This distillation process will encourage the model to extract the interplay between static semantic information in the image and the dynamic information in IMU, as it trained to reconstruct the video model feature. On the other hand, simple concatenation of the IMU feature to the image feature for action classification will not exploit the relationship between the image and IMU-it lacks the training signal to target the dynamic video clip Ceature-leading to inferior performance.
Efffet of using partial IMU. Is the net displacement of the camera what matters, or does the motion in between give cues? To answer this, experiment with feeding EgoDistill with partial IMU readings of the full video clip. Specifically, we train and evaluate our model with only the first K% of IMU readings of the full clip and pad the rest with O. We show the results in Table 5. This ablation study shows that more complete IMU readings yield the best results, while using 25% and 50% IMU data achieves the majority of the gain over the VisOnly-D (no IMU) baseline.
Effect of number of input frames. To investigate the effect of using different numbers input frames, we conduct an ablation study with N. Specifically, we use three settings: N = (use the center frame),
2 (use the start frame and end frame), N = 4 (uniformly sample frames). We compare EgoDistill and VisOnly-D under different settings in Table 6. We observe that only using the overall displacement between the end frame and the starting frame (NV = 2) leads to better
We tried augment AdaFuse [50] the same way. but adding IMU to its codebase gave unreliable results.

---

Sph
Figure 5: Retrieving video clips with EgoDistill. Given query frame (bottom left) and paired IMU segment (red camera frustums), retrieve the nearest clip in the video dataset according to EgoDistill and visualize its (unobserved) frames (strip to the right). Compared to VisOnly-Distill, which outputs single feature for given input frame (bottom row), EgoDistill outputs a distinct feature by conditioning on IMU, showing its ability to preserve both semantic and motion during reconstruction. For instance, in the top-right example, EgoDistill retains the cabinet interaction semantics in the frame as well as the upward camera-motion in the IMU. Zoom in to view best.
Input IMU
Ego4D
EPIC Kitchens
Ego4D
100% (igoDistlll)
37.95
44.95
Inpot Frame
EgoDistill
VisOnly D
EgoDistill
VisOnly
75%
37.88
43.43
50%
37.65
43.69
N=1 (center)
37.95
34.32
44.95
37.2
25%
37.21
42.68
last)
38.29
50.07
46.71
36.60
41.45
No4 (uniform)
38,46
34.84
52.43
48.96
0% (VisOaly-D)
34.32
37,20
Table 5: Effect of partial IMU. We observe using partial IMU input al- ready achieves significant gain.
Table 6: Effect of number of input frames. Using more input frames leads to larger improvement of EgoDistill over VisOnly-D baseline.
performance by VisOnly-D than using only one frame (N 1). However, the motion trajectory still helps EgoDistill obtain significantly better performance in either case. This result indicates the information in the motion trajectory captured by IMU. Note that while N = 2, 4 leads to better performance, it requires higher computational cost.
What do EioDistill features capture? To explore this, we pair a single input frame with different IMU clips as inputs to EgoDistill. then retrieve the nearest video clip for each resulting anticipated video feature. We also compare with the VisOnly-Diste EgoDistill outputs video features that all involve interaction with the cabinet (right panel), and is able to use different IMU inputs to retrieve different video clips that show consistent camera motion. In contrast, different input IMUs lead to corresponding camera motion in the frames. The result shows that EgoDistill approximates video features that capture both semantic and motion information, whereas VisOnly- -iittil only retains the semantic context to retrieve a single clip.
result. Figure illustrates this. We see that
Is there evidence EgoDistill captures scene motion? Figure shows how our features learned with head motion can nonetheless expose certain scene motion cues. EgoDistill improves the accuracy over VisOnly-Diss -Dsstill on ambiguous categories (like close and put) by a large margin (20.3% and 10.4% on Kitchess 8.5% and 3.9% on Ego4D). See caption and Supp. video for more details.
L Limitations of EgoDistill. Although EgoDistill is useful for many action categories in egocentric videos, for some other classes it is less effective. We observe that video clips that EgoDistill does not perform well tend to contain little head motion -in which case IMU is redundant to the RGB frame or drastic head motion that is weakly correlated with the camera wearer's activity and introduces blur the frame. Additionally, EgoDistill might not be useful when we want to focus on the "noun" part of the action (c.g., differentiating watching TV and using the cell phone), as IMU information does not contain information about the object semanties in the scene.
Conclusion
We present EgoDistill, the first model to explore egocentric video feature approximation for fast recognition. Experiments on action recognition on Ego4D and @PIC-Kitchens demonstrate that our model achieves a good balance between accuracy and efficiency, outperforming state-of-the art efficient video understanding methods. Our approach has great potential to accelerate video understanding for egocentric videos using a data stream that is already ubiquitous in egocentric cameras. In the future, we plan to investigate how to use head motion for long-term human activity understanding with room context and visual correspondence learning for multi-view videos.
10

---

Acknowledgements
We thank the UT Austin vision group for helpful discussions. We also thank the anonymous reviewers for their insightful suggestions. UT Austin is supported in part by IFML NSF Al Institute. KG is paid as research scientist at Meta.
References
[1] Relja Arandjetović and Andrew Zisserman. Look, listen and learn. 2017 IEEE International Conference on Computer Vision (ICCV), pages
2017. 2
[2] Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS 16, page 892- 900, Red Hook, NY, USA, 2016. Curran Associates Inc. 3
[3] Anna Borowska-Terrs parametrised and raw signals recorded from inertial measurement unit. Applied Sciences, 10(12) 2020. 2
and Pawel Strumillo. Person independent recognition of head gestures from
[4] John E. Bortz. A new mathematical formulation for strapdown inertial navigation. IEEE Transactions on Aerospace and Electronic Systems, AES-7:61-85 1971.
[5] Agata Brajdic and Robert K. Harle. Walk detection and step counting on unconstrained smartphones. Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing, 2013.
[6] M. Brossard, S. Bonnabel, and A. Barrau, Denoising imu gyroscopes with deep learning for open-loop attitude estimation. IEEE Robotics and Automation Letters, 5(3) -8003, 2020. 6
[7] Xiya Cao, Caifa Zhou, Dandan Zeng, and Yongliang Wang. Rio: Rotation-equivarrance supervised learning of robust inertial odometry. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition (C (CVPR), pages 6614-6625 June 2022. 1,2
[8] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299-6308,
2017. 7
[9] Chen Chen, Roozbeh Jafari, and Nasser Kehtarnavaz. Utd-mhad: A multimodal dataset for human action recognition utilizing depth camera and a wearable inertial sensor. In 2015 IEEE International Conference on Image Processing (ICIP), pages 168-172, 2015. 2
[10] Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülgehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using mn machine translation.
for statistical
EMNLP, 2014. 6
[11] Dima Damen, Hazel Doughty, Giovanni Maria Farinella. Antonino Furnari, Jian Ma. Evangelos Kazakos. Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michacl Wray. Rescaling egocentric vision:
Collection, pipeline and challenges for spie-kitchens-100 International Journal of Computer
Vision (IJCV), 130:33-55 55, 2022. 1, 2, 6
[12] Samyak Datta, Sameer Dharur, Vincent Cartillier, Ruta Desai, Mukul Khanna, Dhruv Batra, and Devi Parikh. Episodic memory question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19119-19128, June 2022.
[13] Alexander Diete, Timo Sztyler, and Heiner Stuckenschmidt. Vision and acceleration modalities: Partners for recognizing complex activities. In 2019 IEEE International Conference on Pervaxive Computing and
Communicutions Workshops (PerCom Workshops), pages 101-106, 2019. 1,2
[14] Kiana Ehsani, Hessam Bagherinezhad, Joseph Redmon, Roozbch Mottaghi, and Ali Farhadi. Who let the dogs out? modeling dog behavior from visual data. In CVPR, 2018. 3
[15] Kiana Ehsani, Daniel Gordon, Thomas Nguyen. Roozbeh Mottaghi, and Ali Farhadi. Learning visual representation from human interactions. International Conference on Learning Representations, 2021.
[16] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In ICCV, 2021. 1, 3.
[17] Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. 2020 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 200-210, 2020. 1, 3,4,7, 7,8
[18] Christoph Feichtenhofer. Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6201-6210 2019. 1, 4, [19] Christoph Feichtenhofer, Axel Pinz, and Andrew 7isserman. Convolutional two-stream network fusion for video action recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pages 1133-1941,200 2016. 2
[20] Christian Forster, Luca Carlone, Frank Dellacrt, and Davide Scaramuzza. On-manifold preintegration for real-time visual-inertial odometry. IEFF Transactions on Rohotics, 33(1):1-21, 2017. 1,
[21] C. Gan, H. Zhao, P. Chen, D. Cox, and A. Torralba. supervied moving vehicle tracking with stereo sound. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 7052-7061. Los Alamitos, CA. USA, nov 2019. IEEE Computer Society. 3
11

---

[22] Ruohan Gao. Tae-Hyun Oh, Kristen Grauman, and Lorenzo Torresani. Listen to look: Action recognition by previewing audio. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10457-10467, 2020. 2, 3. 6, 7.9
[23] Nuno C. Garcia, Pietro Morerio, and Vittorio Murino. Modality distillation with multiple stream networks for action recognition. In The European Conference on Computer Vision (ECCV), September 2018. 3 Amir Ghodrati, Babak Ehteshami Bejnordi, and Amirhossein Habibian. Framcexit: Conditional carly exiting for efficient video recognition. In Proceedings of the IEEE Conference on Computer Vision and
[24]
Pattern Recognition, 2021. 3,4
[25]
Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra. Omnivore: A Single Model for Many Visual Modalities. In CVPR, 2022.
[26]
Kristen Grauman, Andrew Westbury, Eugene Byme, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhong- cong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane, Tien Do, Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer. Adriano Fragomeni, Qichen Fu, Christian Fuegen, Abrham Gebreselasie, Cristina Gonzalez, James Hillis, Xuhua Huang, Yifei Huang. Wenqi Jia, Weslie Khoo, Jachym Kolar, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li. Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang, Xindi Wu, Takuma Yagi. Yunyi Zhu, Pablo Arbelaez, David Crandall, Dima Damen, Giovanni Maria Farinella, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Jon, Kris Kitami, Haizhou Li, Richard Newcomhe, Aude Oliva, Hyun Soo Park, James M. Rehg. Yoichi Sato, Jianho Shi, Mike Zheng Shou, Antonio Torralha, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. Ego4d: Around the World in 3,000 Hours of Egocentric Video. IEEE/CVF Computer Vision and Pattern Recognition (CVPR). 2022. 1,2, 6
[27] Grigori Guitchounts, Javier Masís, Steffen B.F Wolff, and David Cox. Encoding of 3d head orienting movements in the primary visual cortex. Neuron, 108(3):512-525 e4, 2020.
[28] Saurabh Gupta, Judy Hoffman, and Jitendra Malik. Cross modal distillation for supervision transfer. In
2016 TEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2827-2836, 2016.
[29] Michael Gutmann and Aapo Hyvärinen. Noise conts-contreetive estimation: A new estimation principle for unnormalized statistical models. In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intellignnce and Statisties, volume 9 of Proceedings of Machine Learning Research, pages 297-304, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010.
PMLR. 6
[30] Sachini Herath, Hang Yan, and Yasutaka Furukawa. Ronin: Robust neural inertial navigation in the wild: Benchmark, evaluations, & new methods. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 3146-3152, 2020. 2,
[31] Geoffrey F.. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. NIPS
Deep Learning Workshop, abs/1 003312231, 2014, 3, 4,5
[32] Yu Huang. Chenzhuang Du Zibui Xue, Xuanyao Chen. Hang Zhao, and Longbo Huang. What makes multi- modal learning better than single (provably). In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. 2
[33] Andrey Ignatov. Real- -ime human activity recognition from accelerometer data using convolutional neural networks. Applied Soft Computing. 02:915-9222 2018. 2
[34] D. Jayaraman and K. Grauman. Learning image representations tied to egomotion. In ICCV, 2015. 3
[35] A.R. Jimenez, F. Seco, C. Prieto, and Guevara. A comparison of pedestrian decd-nec algorithms using a low-cast mems imu. In 2009 IEFF International Symposium on Intelligent Signal Processing,
pages 37-42, 2009. 2
[36] Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen. Epic-fusion. Audio- temporal binding for egocentric action recognition. In IEEF/CVF International Conference on Computer
Vision (ICCV), 2019. 2
[37] Dan Kondratyuk, Liangzhe Yuan, Yandong Li, Li Zhang, Mingxing Tan, Matthew Brown, and Boqing Gong. Movinets: Mobile video networks for efficient video recognition. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 16020-16030, 2021. 3
[38] Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from self- supervised synchronization. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, page 7774-7785, Red Hook, NY, USA, 2018. Curran Associates Inc. 2
[39] Bruno Korbar, Du Tran, and Lorenzo Torresani. Sesampler: Sampling salient clips from video for efficient action recognition. 2019 JEEE/CVF International Conference on Computer Vision (ICCV), pages 6231-6241 2019. 3
[40] Masakatsu Kourogi and Takeshi Kurata. A method of pedestrian dead reckoning for smartphones using frequency domain analysis on patterns of acceleration and angular velocity. In Record- - IEEE PLANS, Position Location and Navigation Symposium. pages 164- 168, 05 2014. 2
12

---

[41] Jennifer R. Kwapisz, Gary M. Weiss, and Samuel A. Moore. Activity recognition using cell phone accelerometers. SIGKDD Explor. Newsl., 12(2):74-82, mar 2011. 2
[42] Heike Leutheuser, Dominik Schuldhaus, and Bjoern M. Eskofier. Hierarchical, multi-sensor based classification of daily life activities: Comparison with state-of-the-arl algorithms using benchmark dataset. PLOS ONE, 8(10):111, 10 2013. 2
[43] Yiming Li, Ziang Cao. Andrew Liang, Benjamin Liang, Luoyao Chen, Hang Zhao, and Chen Feng. Egocentric prediction of action target in 3d. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), June 2022.
[44] Shaowei Liu, Subarna Tripathi, Somdeb Majumdar, and Xiaolong Wang. Joint hand motion and interaction hotspots prediction from egocentric vidcos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3282-3292, June 2022.
[45] Zhengzhe Liu, Xiaojuan Qi, and Thi-Wing Fu. 3d-to-2d distillation for indoor scene parsing. 2021 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 4462-4472, 2021. 3
[46] Nya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam, 2018.
[47] Niall Lyons. Avik Santra, and Ashutosh Pandey. Improved deep representation learning for human activity recognition using imu sensors. In 2021 20th IEEE International Conference on Machine Leurning and Applications (ICMLA), pages 326-332, 2021. 2
[48] Sharmin Majumder and Nasser Kehtarnavaz. Vision and inertial sensing fusion for human action recogni-
tion: A review. IEEE Sensors Journal, 2 213):2255-22677 2021. 2
[49] Yue Meng, Shung-Chingg Lin, Rameswar Panda, Prasanna Sattigeri, Leonid Karlinsky, Aude Oliva, Kate Saenko, and Rogerio Feris. Ar-net: Adaptive frame resolution for efficient action recognition. arXiv
preprint Xix:2000..1579, 2020. 3,4
[50] Yue Meng. Rameswar Panda, Chung-Ching Lin, Prasanna Sattigeri, Leonid Karlinsky, Kate Saenko, Aude Oliva, and Rogerio Feris. Adafuse: Adaptive temporal fusion network for efficient action recognition. In International Conference on Learning Representations, 2021. 2, 3, 7.9
[51] Rameswar Panda, Chun-Fu Chen, Quanfu Fan, Ximeng Sun, Kate Sacnko, Aude Oliva, and Rogerio Feris. AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition. In International Conference
on Computer Vision (ICCV), 2021. 2, 4
[52] Philip R.L. Parker, Morgan A. Brown, Matthew C. Smear, and Cristopher M. Niell. Movement-related
signals in sensory areas: Roles in natural behavior. Trends in Neurosciences, 43(5):881-555, 2020,
[53] Philip R. L. Parker, Elliou T. T. Abe, Emmalyn S. P. Leonard, Dylan M. Martins, and Cristopher M. Niell.
Joint coding of visual input and eye/head position in v of freely moving mice. bioRxiv, 2022. 1
[54] Mandela Patrick, Dylan Campbell, Yuki M. Asano, Ishan Misra Florian Metze, Christoph Feichtenhofer, Andrea Vedaldi, and João F. Henriques. Keeping your eye on the ball: Trajectory attention in vidco transformers. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 1, 2. 3, 4, 6, 8, 9 [55] Chiara Plizzari, Mirco Planamente, Gabriele Goletto, Marco Cannici, Emanuele Gusso, Mattco Matteucci, and Barbara Caputo. E2(go)motion: Motion augmented event stream for egocentric action recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 19935-19947, June 2022.
[56] Gunnar A Sigurdsson, Abhinay Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. Charades-ego:
A large-scale dataset of paired third and first person videos. arXiv preprint arXiv: (004,80049000 2018. 1 [S7] Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Newral Information Processing Systems, volume 27. Curran Associates, Inc., 2014. 2
[S8] Shuyang Sun, Zhanghui Kuang, Lu Sheng. Wanli Ouyang, and Wei Zhang. Optical flow guided feature: fast and robust motion representation for video action recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018. 2
[59] Yin Tang. Lei Zhang, Fuhong Min, and Jun He. Multi scale deep feature learning for human activity recognition using wearable sensors. IEEE Transactions on Industrial Electronics, pages 1-1, 2022. [60] Wenjin Tao, Haodong Chen, Md. Moniruzzaman, Ming C. Leu, Zhaozheng Yi, and Ruwen Qin. Attention- based sensor fusion for human activity recognition using imu signals. ArXiv, abs/21 1224, 2021.
[61] Catherine Tong, Jinchen Ge, and Nicholas D Lane. Zero-shot learning for imu-based activity recognition using video embeddings. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 5(4):1-23, 2021.
[62] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray. Yann LeCun, and Manohar Paluri. A closer look spatiotemporal convolutions for action recognition. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6450-6453 2018.
[63] Satoshi Tsutsui, Ruta Desai, and Karl Ridgeway. How you move your head tells what you do: Self- supervised video representation learning with egocentric cameras and imu sensors. ICCV Workshop on Egocentric Perception, Interaction and Computing (EPIC), 2021.
[64] Jindong Wang, Yiqiang Chen, Shuji Hao, Xiaohui Peng. and Lisha Hu. Deep learning for sensor- sensor-based activity recognition: A survey, Patter Recognit. Lett., 119:3 11, 2019. 2
13

---

[65] Junke Wang, Xitong Yang. Hengduo Li. Liu Li, Zuxuan Wu, and Yu-Gang Jiang. Efficient video transformers with spatial-temporal token selection. In ECCV, 2022. 8,
[66] Yulin Wang, Zhaoxi Chen, Haojun Jiang, Shiji Song, Yizeng Han, and Gao Huang. Adaptive focus for efficient video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision
(ICCV), October 2021. 3
[67] Yulin Wang, Yang Yue, Yuanze Lin, Haojun Jiang. Zihang Lai, Victor Kulikov, Nikita Orlov, Humphrey Shi, and Gao Huang. Adafocus v2: End-to-end training of spatial dynamic networks for video recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
3,
[68] Haoran Wei, Pranav Chopada, and Nasser Kehtarnavaz. C-mhad: Continuous multimodal human action
dataset of simultancous video and inertial sensing. Sensors, 20(10), 2020. 1,2
[69] Haoran Wei and Nasser Kehtarnavaz. Simultaneous utilization of inertial and video sensing for action detection and recognition in continuous action streams. IEEE Sensors Journal, 20(1 176055-6066 2020.
1,2
[70] Hang Yan, Qi Shan, and Yasutaka Furukawa. Ridi: Robust imu double integration. In ECCV, 2018. 2
[71]
Mingyu Yang. Yu Chen, and Hun-Seok Kim. Efficient deep visual and inertial odometry with adaptive visual modality selection. ArXiv, abs/2205.06187, 2022. 1,
[72] Mohammadreza Zolfaghari, Kamaljeet Singh, and Thomas Brox. ECO: efficient convolutional network for online video understanding. In ECCV, 2018.
14


Reviews:
[{"review_id": "ugfak9jRQR", "reviewer": "Reviewer_M5sy", "summary": "The work studies efficient video recognition via distilling IMU signals. Instead of heavily processing each frame, the authors propose to leverage very sparse frames (1-4) for visual features and IMU data. It leads to much more efficient inference cost. The proposed method has shown improvement over other multi-modality fusion techniques like AdaFuse, etc. They also show IMU trajectory provides complementary information to sparse frames. ", "questions": "See weakness above", "limitations": "Limitation is discussed ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The idea to leverage IMU signals for efficiency is novel. And the empirical results support the idea. The improvement with IMU is consistent across different architectures. \n+ The two stage training (first aligns IMU features then fusing them) is shown effective over alternative multi-modality fusion baselines. \n+ The claims are well supported by ablation studies like two-stage training, loss term, usage of modality, etc.\n\n", "weaknesses": "1. Since IMU only represents head motions, it is sensible to disambiguate activity relevant to scene motion (open / close, put /take), but it may not contain enough signals about finer grained activity temporally or spatially. But it is the problem of IMU signal, not the paper’s. \n2. I wonder how much does the motion _trajectory_ help, compared to the overall displacement between the end frame and the starting frame, i.e X_t - X_0. More specifically, in table 2, how does EgoDistill with less IMU steps work and how does feeding start frame __and__ end frame works? \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments! We will address your concerns point-by-point:\n\n---\n\n> 1. Since IMU only represents head motions, it is sensible to disambiguate activity relevant to scene motion (open / close, put /take), but it may not contain enough signals about finer-grained activity temporally or spatially. But it is the problem of IMU signal, not the paper’s.\n\nYes, although IMU is useful for many action categories in egocentric videos, it cannot provide much information about some other categories. \n\nTo understand how information in IMU contributes to the performance boost, we analyze when IMU is useful and its limitations:\n\n(Please refer to the attached **PDF** file in the overall author rebuttal for the mentioned figures.)\n\nWe saw in our paper that using IMU leads to an overall performance improvement in action recognition, indicating better video feature prediction capability (Lines 265-272 and Figure 3). Here, we explore what kinds of clips are better reconstructed using EgoDistill. Figure.A1 in the uploaded pdf file shows the improvement of EgoDistill over the VisOnly-Distill model on Ego4D and EPIC-Kitchens split by action class.  We observe that IMU is more useful for actions with predictable head motion (e.g., *break*, *cut*, *close*), and is less helpful for actions where head motion may be small or unrelated (e.g., *empty*, *fill*, *press*).\n\nFigure.A2 in the uploaded pdf file shows clip examples whose video features are best and worst reconstructed.We observe that the best-reconstructed clips (top) contain moderate head motion that is predictive of scene motion and action semantics. For example, the camera wearer's head moves slightly backward while opening the cabinet.On the other hand, more poorly reconstructed clips tend to contain little head motion (third row)---in which case IMU is redundant to the RGB frame---or drastic head motion that is weakly correlated with the camera wearer's activity and introduces blur to the frame (last row). \n\n---\n\n> 2. I wonder how much does the motion trajectory help, compared to the overall displacement between the end frame and the starting frame. More specifically, in table 2, how does EgoDistill with less IMU steps work and how does feeding start frame and end frame works?\n\nThis is a good point! To address your questions, we provide two sets of experiments. \n\n**Firstly, we conduct an ablation study of feeding different input frames**:\n\n* N=1 - use the center frame\n* N=2 - **use the start frame and end frame**\n* N=4 - uniformly sample 4 frames\n\n\nWe compare the performance of EgoDistill and VisOnly-Distill under different settings:\n\n| Input Frame | Ego4D |   | EPIC-Kitchens |   |   |\n| :--: | :--: | :--: | :--: | :--: | :--: |\n|   | EgoDistill | VisOnly-D | EgoDistill | VisOnly-D | GFLOPs |\n| N=1 (center) | 37.95 | 34.32 | 44.95 | 37.2 | 1.9 |\n| N=2(first and last) | 38.29 | 34.48 | 50.07 | 46.71 | 3.8 |\n| N=4 (uniform) | 38.46 | 34.84 | 52.43 | 48.96 | 7.6 |\n\nFrom the results above, we observe that only using the overall displacement between the end frame and the starting frame (N=2) leads to indeed better performance of VisOnly-D than using only one frame (N=1). However, the motion trajectory still helps EgoDistill obtain significantly better performance in either case. This result indicates the information contained in the motion trajectory captured by IMU.  N=2, 4 leads to better performance, but requires higher computational cost as noted in L236.\n\n**Secondly, we conduct an ablation study about using fewer IMU steps**.\n\nAccording to the suggestion, we experiment with feeding EgoDistill with partial IMU readings of the full video clip. Specifically, we train and evaluate our model with only the first N% IMU reading data of the full clip and pad the rest with 0. Results are below\n\n| IMU Usage | Ego4D | EPIC-Kitchens |\n| :--: | :--: | :--: |\n| 100% IMU | 37.95 | 44.95 |\n| 75% IMU | 37.88 | 43.43 |\n| 50% IMU | 37.65 | 43.69 |\n| 25% IMU | 37.21 | 42.68 |\n| 10% IMU | 36.6 | 41.45 |\n| No IMU (VisOnly-D) | 34.32 | 37.2 |\n\nThis ablation study shows that more complete IMU readings yield the best results, while using 25% and 50% IMU data achieves the majority of the gain over the “No IMU” baseline.\n"}, {"review_id": "W7aIKFuKyj", "reviewer": "Reviewer_xst2", "summary": "This paper for on Ego-centric video understanding, a promising and popular topic recently.\nDifferent from previous works that focus on improve heavy video encoder, this work try to cut the computation cost down and focus on lightweight temporal reasoning.\n\nPlease reply the questions and weakness point-by-point.", "questions": "- Why the first row of Fig.6 sometimes seems not well matched with the action, can you give detail explanation about how to get this figure and why always no changed? The head motion is from AR glasses or?\n-  How to define ambiguous categories  in Ego4D?\n", "limitations": "- The performance is hard to compare with video encoder based method.\n- The proposed work also suitable for ego-centric video.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation is quite good and make sense. I like it. Actually ego-centric video have fixed background in general and learn the motion pattern based on the first frame is more easily than third-view video. Considering this area is  new and not well studied, I think it's valuable.\n- The method is also simple but works.\n- The presentation is good and I can understand it very well.\n- The computation cost is cut a lot and provide possibly for small and mobile device.\n", "weaknesses": "- The method section especially the IMU predictor is not solid. Actually, the idea of bring GRU for temporal modeling is quite a old technique. There also exist a lot of works that predict the temporal sequence on feature level like MemDPC two years ago.\n- Although the performance is not a very important measurement to me, the compared method is somewhat old. For example, Line243-248.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments and positive feedback! We will answer your questions point-by-point:\n\n---\n\n\n> The method section especially the IMU predictor is not solid. Actually, the idea of bring GRU for temporal modeling is quite a old technique. There also exist a lot of works that predict the temporal sequence on feature level like MemDPC two years ago. Although the performance is not a very important measurement to me, the compared method is somewhat old.\n\nWe regard the GRU as a simple and effective architecture for aggregating frame information in our setting, as demonstrated in the results (see Figure 3 where we observe significant performance gain with more frames).\n\nWe discuss our choice of baselines in L249. To summarize, these methods represent recent, leading approaches in the area of efficient video recognition (e.g., STTS, ECCV22). Recent architectures are promising for performance but are far less efficient, making them unsuitable given our focus on efficiency.\n\n---\n\n\n> 1. Why the first row of Fig.6 sometimes seems not well matched with the action, can you give detail explanation about how to get this figure and why always no changed? The head motion is from AR glasses or?\n\nWe obtain the head motion from the IMU sensor of a GoPro camera, which is the device used to collect data in these egocentric video datasets. To obtain the figure, we double integrate the IMU readings (which are accelerations) to obtain the motion trajectory of the head. Then, by setting the initial head position at the origin and facing the x-axis, we plot the head trajectory across time.\n\nWith only static figures it can be hard to match the head motion plot in Fig.6 with the action shown in the video frames. Therefore, we refer the reviewer to our **supplementary video** to watch an animated version of Fig.6 (from 03:00 to 04:11). We hope the animated version makes the correspondence clearer.\n\n---\n\n\n> 2. How to define ambiguous categories in Ego4D?\n\nWe define ambiguous categories in Ego4D as the ones that cannot be determined by only a few frames (L318-320). For example, only from a single frame of the video clip, one cannot distinguish between “close” and “open”. By using IMU signals to obtain head motions, our model is able to classify these two categories despite having only a few (1-4) visual frames.\n\n\nTo have a clearer understanding of the ambiguous categories, we analyze when IMU is useful and its limitations:\n\n(Please refer to the attached **PDF** file for the mentioned figures.)\n\nWe saw in our paper that using IMU leads to an overall performance improvement in action recognition, indicating better video feature prediction capability  (Lines 265-272 and Figure 3). \n\nHere, we explore what kinds of clips are better reconstructed using EgoDistill. Figure.A1 in the uploaded pdf file shows the improvement of EgoDistill over the VisOnly-Distill model on Ego4D and EPIC-Kitchens split by action class.  We observe that IMU is more useful for actions with predictable head motion (e.g., *break*, *cut*, *close*), and is less helpful for actions where head motion may be small or unrelated (e.g., *empty*, *fill*, *press*).\n\nFigure.A2 in the uploaded pdf file shows clip examples whose video features are best and worst reconstructed.  We observe that the best-reconstructed clips (top) contain moderate head motion that is predictive of scene motion and action semantics. For example, the camera wearer's head moves slightly backward while opening the cabinet.  On the other hand, more poorly reconstructed clips tend to contain little head motion (third row)---in which case IMU is redundant to the RGB frame---or drastic head motion that is weakly correlated with the camera wearer's activity and introduces blur to the frame (last row). \n\n\n---\n\n\n> 3. The performance is hard to compare with video encoder based method.\n\nThe purpose of EgoDistill is to obtain an efficient video understanding method for egocentric videos. In this regard, EgoDistill reaches a significantly better performance-efficiency balance than the compared methods. Compared with video-based methods (e.g., MotionFormer), our frame-based model requires significantly less computation resources, which is vital for mobile devices.\n"}, {"review_id": "SUqQ2QkwUy", "reviewer": "Reviewer_gfRs", "summary": "This article presents an innovative approach called EgoDistill for egocentric video understanding. The authors highlight the computational challenges faced by existing models in this field and propose EgoDistill as a solution to overcome these limitations. EgoDistill aims to distill crucial information from egocentric video frames and IMU readings, resulting in a more efficient representation. The article provides detailed technical explanations of EgoDistill, illustrating how it effectively combines semantics from video frames with IMU readings. Additionally, the authors introduce a novel self-supervised training strategy specifically designed for IMU feature learning. To validate the effectiveness of EgoDistill, the authors conduct experiments and present compelling results. Their findings demonstrate that EgoDistill surpasses state-of-the-art efficient video understanding techniques when evaluated on the Ego4D and EPIC-Kitchens datasets. Overall, this article contributes significant advancements to the field of egocentric video understanding by introducing EgoDistill and showcasing its superior performance over existing methods.", "questions": "See Weaknesses", "limitations": "None", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. EgoDistill offers several advantages over equivalent video models, including significantly reduced computational requirements in terms of GFLOPs, parameters, and inference time. This enhanced efficiency makes it a more practical option for real-world applications. Furthermore, the authors provide experimental evidence illustrating that EgoDistill surpasses state-of-the-art efficient video understanding methods when evaluated on the Ego4D and EPIC-Kitchens datasets.\n2. The authors have devised a concise and effective self-supervised IMU feature learning architecture. This architecture enables the extraction of valuable camera motion features that are specifically associated with visual appearance changes. As a result, EgoDistill demonstrates improved performance in capturing and understanding these subtle visual cues, leading to enhanced overall performance.", "weaknesses": "1. The main text contains some basic errors, such as confusion between the order of figures and tables, as well as incorrect citations. It is crucial to carefully review and correct these issues.\n2. Undoubtedly, EgoDistill exhibits commendable performance in terms of generalization and versatility. However, the explanation provided in the paper for why EgoDistill performs better than STTS when both methods incorporate IMU features (as mentioned in the \"Effect of fusing IMU data\" subsection) is too superficial and lacks a profound theoretical explanation for EgoDistill. Additionally, there is limited discussion on how the balance between different components of the model is controlled. It would be beneficial to address these points and provide more insights into the underlying mechanisms and strategies employed by EgoDistill.\n3. EgoDistill relies on head motion data to capture camera pose changes. It is important to explore and discuss the limitations and potential mitigation strategies for scenarios where reliable head motion data may not be accurate.\n4. The authors have not thoroughly discussed any potential shortcomings or limitations of EgoDistill. It would be valuable to identify specific scenarios or use cases where EgoDistill might not be as effective or efficient. Understanding these limitations would provide a more comprehensive understanding of the applicability and scope of EgoDistill and could open avenues for future research to address these areas of improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and constructive comments! We will address your comments point-by-point:\n\n---\n\n\n> 1. The main text contains some basic errors, such as confusion between the order of figures and tables, as well as incorrect citations. It is crucial to carefully review and correct these issues.\n\nThank you; we will correct them in our final version.\n\n---\n\n\n> 2. Undoubtedly, EgoDistill exhibits commendable performance in terms of generalization and versatility. However, the explanation provided in the paper for why EgoDistill performs better than STTS when both methods incorporate IMU features (as mentioned in the \"Effect of fusing IMU data\" subsection) is too superficial and lacks a profound theoretical explanation for EgoDistill. Additionally, there is limited discussion on how the balance between different components of the model is controlled. It would be beneficial to address these points and provide more insights into the underlying mechanisms and strategies employed by EgoDistill.\n\nPlease see L185-190 in the paper for discussion.  The reason that EgoDistill is able to outperform STTS+IMU is mainly that EgoDistill learns to jointly distill information from a video feature with *image and IMU* inputs. This distillation process will encourage the model to extract the interplay between static semantic information in the image and the dynamic information in IMU, as it is trained to reconstruct the video model feature. On the other hand, simple concatenation of the IMU feature to the image feature for action classification will not exploit the relationship between the image and IMU—it lacks the training signal to target the *dynamic video clip* feature—leading to inferior performance.\n\nTo balance the different components of the model, we tune the weights in the loss function on the validation dataset for higher classification accuracy (L239-240).\n\nWe will add the above notes to our final version.\n\n\n---\n\n\n> 3. EgoDistill relies on head motion data to capture camera pose changes. It is important to explore and discuss the limitations and potential mitigation strategies for scenarios where reliable head motion data may not be accurate.\n\nAs mentioned in L51-53, we use an inexpensive and widely available sensor for head tracking, which could already provide strong performance as we show.  Specifically,  the IMU sensor used to collect the datasets (Ego4D and EPIC-Kitchens) we use is pretty cheap ($3 per unit) and outputs reasonable but not perfect motion readings.\n\nSpecifically, we observe that the raw IMU input has significant drifting and bias issues. This induces inconsistent correspondence between camera motion and IMU reading across different clips and videos. Therefore, for the IMU reading of each clip, on each dimension, we separately subtract raw readings by the mean values on the corresponding dimension. This operation normalizes IMU readings in each dimension to have zero average values. In this way, our model can only focus on the temporal motion patterns in each clip.  Learning-based methods used in inertial navigation and odometry works [1][2] may benefit any IMU model faced with low-quality IMU inputs. However, we find our simple mechanism already works well in our case.\n\nWe will add this discussion to our final version of the paper.\n\n[1] RIO: Rotation-equivariance supervised learning of robust inertial odometry. Zhou et al.\n[2] RoNIN: Robust Neural Inertial Navigation in the Wild: Benchmark, Evaluations, and New Methods. Yan et al.\n\n---\n\n\n> 4. The authors have not thoroughly discussed any potential shortcomings or limitations of EgoDistill. It would be valuable to identify specific scenarios or use cases where EgoDistill might not be as effective or efficient. Understanding these limitations would provide a more comprehensive understanding of the applicability and scope of EgoDistill and could open avenues for future research to address these areas of improvement.\n\nWe agree that discussion of limitations of EgoDistill is important. \n\nAlthough EgoDistill is useful for many action categories in egocentric videos, for some other classes it is less effective. To have a clearer understanding EgoDistill’s limitations, we analyze when IMU is useful and its limitations:\n\n(Please refer to the attached **PDF** file in the overall author rebuttal for the mentioned figures.)\n\nHere, we explore what kinds of clips are better reconstructed using EgoDistill. Figure.A1 in the uploaded pdf file shows the improvement of EgoDistill over the VisOnly-Distill model on Ego4D and EPIC-Kitchens split by action class.  We observe that IMU is more useful for actions with predictable head motion (e.g., *break*, *cut*, *close*), and is less helpful for actions where head motion may be small or unrelated (e.g., *empty*, *fill*, *press*).\n\nFigure.A2 in the uploaded pdf file shows clip examples whose video features are best and worst reconstructed.  We observe that the best-reconstructed clips (top) contain moderate head motion that is predictive of scene motion and action semantics. For example, the camera wearer's head moves slightly backward while opening the cabinet.  On the other hand, more poorly reconstructed clips tend to contain little head motion (third row)---in which case IMU is redundant to the RGB frame---or drastic head motion that is weakly correlated with the camera wearer's activity and introduces blur to the frame (last row). \n\nAdditionally, EgoDistill might not be useful when we want to focus on the “noun” part of the action (e.g., differentiating watching TV and using the cell phone), as IMU information does not contain information about the object semantics in the scene.\n\nWe will add this discussion to our final version of the paper about the scope and applicability of EgoDistill.\n"}, {"review_id": "sbyPfyQvR3", "reviewer": "Reviewer_K65f", "summary": "The work presents a new approach and architecture, EgoDistill, which could efficiently perform action recognition through sparse video frames augmented with head motion data. The model goes through a two-stage training. A pre-training phase, which uses a contrastive loss that enables the motion encoder to learn semantic visual features, and a distillation phase, which makes the motion-visual fusion model jointly mimics the more stronger visual-only model. The results are impressive, supported by a series of ablation study. \n\nThere is no major flaws in the paper, except a minor confusion on the argument on the fact that the pre-training of the model forces the motion encoder learning semantic features.", "questions": "During the pre-training phase, L1 would train the fusion network as well as the IMU encoder, while NCE loss would mix and match samples to train both the IMU predictor and IMU encoder. How is the schedule of training the IMU predictor \\hat{z} match with the optimization of the IMU encoder? How to prevent IMU encoder from being affected by under-trained IMU predictor?", "limitations": "I am not super convinced on the argument that the IMU encoder learns scene motion cues (line 318). The action confusion (between close/open and put/take) is a good example, but the two action pairs seem to be more relevant to time sequence rather than scene motion.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper is very well written and easy to follow. The figure showcases the method very well.\n\nThe authors made a keen observation in using a separate modality (head motion) jointly with egocentric frames to efficiently perform action recognition task. The proposed architecture is novel and yields impressive results.", "weaknesses": "No major weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and constructive comments! We will address your comments point-by-point:\n\n---\n\n\n> 1. During the pre-training phase, L1 would train the fusion network as well as the IMU encoder, while NCE loss would mix and match samples to train both the IMU predictor and IMU encoder. How is the schedule of training the IMU predictor \\hat{z} match with the optimization of the IMU encoder? How to prevent IMU encoder from being affected by under-trained IMU predictor?\n\nThat’s a good point!  We just initialize the IMU encoder and IMU predictor from scratch and train them directly with NCE loss and L1 loss.\n\nWhen both L1 loss and NCE loss is used, this will not be an issue as the IMU encoder will be trained by L1 loss, which encourages it to reconstruct the video model feature.\n\nWhen only NCE loss is used, we found that if we also finetune the image encoders, we would encounter the mode collapse problem, leading to a less useful IMU encoder. However, if we fix the image encoder and then train the IMU encoder and IMU predictor together, we observe that we are still able to achieve a useful IMU encoder (as shown in the 6th row of Table.1). \n\nWe conjecture this is due the fact that the correlation between the starting and end image features (pretrained) and the IMU features are high enough such that the IMU predictor can be easily trained with NCE loss without mode collapsing. \n\nNevertheless, your suggestion of using a special scheduling algorithm is very valuable. We will explore such mechanisms to further stabilize our self-supervised training stage.\n\n---\n\n\n> 2. I am not super convinced by the argument that the IMU encoder learns scene motion cues (line 318). The action confusion (between close/open and put/take) is a good example, but the two action pairs seem to be more relevant to time sequence rather than scene motion.\n\n\nActually, in this example, we think the time sequence and scene motion is the same: moving the door forward vs. pulling it back gives both different sequences of states as well as two different scene motions.\n\nAlso, we note that it might be hard to intuitively understand from static figures in our main paper. Please refer to our **supplementary video** (04:50-05:55) for animated results. We hope this experiment could help explain what information does EgoDistill’s IMU encoder learn.\n"}, {"review_id": "AOlhjkir5E", "reviewer": "Reviewer_F46j", "summary": "To improve the efficiency of ego-centric video representation learning, this paper presents a distillation-based approach that learns to reconstruct heavy egocentric video clip features from the combination of sparsely-sampled video frames and the camera motion from IMU.", "questions": "Please refer to the weakness.", "limitations": "The authors have discussed the potential limitations in the paper.", "rating": 4, "confidence": 3, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and the easy to follow.\n\n2. Improving the efficiency of ego-centric video representation learning is indeed valuable.", "weaknesses": "1. As mentioned in the paper, the dynamics in a video include both the movement of the scene and the camera (Line 34-35). I agree that IMU could provide some cues for the motion of the camera, however, I'm afraid it is difficult to deduce the motion of the scenes, which is more important for the video understanding (although some analysis is presented in Figure 6, it is still contradictory to the intuition). \n\n2. From Table 1, we can see that the performance improvements mainly come from L_{KD}, other components like L1, L1-pretrain, and L1-NCE, bring subtle improvements.\n\n3. It could be better to compare the model performance in Table 4 (which is mismarked as Figure 4). ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and constructive comments! We’ll address your questions point-by-point:\n\n---\n\n\n> 1. As mentioned in the paper, the dynamics in a video include both the movement of the scene and the camera (Line 34-35). I agree that IMU could provide some cues for the motion of the camera, however, I'm afraid it is difficult to deduce the motion of the scenes, which is more important for the video understanding (although some analysis is presented in Figure 6, it is still contradictory to the intuition).\n\n\nIn L182-189, we discussed why we think EgoDistill could learn scene motion cues from IMU. In short, we believe this is because EgoDistill learns to map from the RGB frame (**scene context**) **and** IMU (**head motion**) to full video feature (scene motion + head motion).  In this way, it learns to encode predictable scene motions tied to scene context. Please refer to the paper for more details.\n\nIn addition to the discussion in the paper, we conduct two more additional analyses:\n\n(Please refer to the attached **PDF** file in the overall author rebuttal for the mentioned figures.)\n\n\n**To understand how information in IMU contributes to the performance boost, we analyze when IMU is useful and its limitations:**\n\nWe saw in our paper that using IMU leads to an overall performance improvement in action recognition, indicating better video feature prediction capability (Lines 265-272 and Figure 3).  \n\nHere, we explore what kinds of clips are better reconstructed using EgoDistill. Figure.A1 in the uploaded pdf file shows the improvement of EgoDistill over the VisOnly-Distill model on Ego4D and EPIC-Kitchens split by action class.  We observe that IMU is more useful for actions with predictable head motion (e.g., *break*, *cut*, *close*), and is less helpful for actions where head motion may be small or unrelated (e.g., *empty*, *fill*, *press*).\n\nFigure.A2 in the uploaded pdf file shows clip examples whose video features are best and worst reconstructed.  We observe that the best-reconstructed clips (top) contain moderate head motion that is predictive of scene motion and action semantics. For example, the camera wearer's head moves slightly backward while opening the cabinet.  On the other hand, more poorly reconstructed clips tend to contain little head motion (third row)---in which case IMU is redundant to the RGB frame---or drastic head motion that is weakly correlated with the camera wearer's activity and introduces blur to the frame (last row). \n\n\n**To understand what information does EgoDistill extract from IMU data, we conduct an experiment to explore the feature extracted from EgoDistill:**\n\nTo explore what our model learns, we pair a single input frame with different IMU clips as inputs to EgoDistill, then retrieve the nearest video clip for each resulting anticipated video feature.  We also compare with VisOnly-Distill result.\n\nFigure.A3 in the uploaded pdf illustrates this.  We see that EgoDistill outputs video features that all involve interaction with the cabinet (right panel), and is able to use different IMU inputs to retrieve different video clips that show consistent camera motion. In contrast, different input IMUs lead to corresponding camera motion in the frames.  The result shows that EgoDistill is able to approximate video features that capture both semantic and motion information, VisOnly-Distill only retains the semantic context to retrieve a single clip. These results indicate that EgoDistill is able to approximate video features that capture both semantic and motion information.\n\nWe thank you for pointing out this question and will add this additional analysis in our final version.\n\n---\n\n\n> 2. From Table 1, we can see that the performance improvements mainly come from L_{KD}, other components like L1, L1-pretrain, and L1-NCE, bring subtle improvements.\n\nActually, the other components also contribute significantly to our model’s final performance. For example, the IMU feature learning leads to a performance boost of 0.96% on Ego4D and 3.74% on EPIC-Kitchen, which is a good boost. \n\nOn the other hand, as our method is, in essence, a distilled student model from the video teacher model with IMU information, it is natural that L_{KD} brings the largest performance boost.\n\n---\n\n\n> 3. It could be better to compare the model performance in Table 4 (which is mismarked as Figure 4).\n\nThanks for the suggestion! We have included a performance comparison in Table 4 as shown below:\n\n|   | GFLOPs | Runtime (ms) | Parameters (M) | Ego4D - Acc (%) | EPK - Acc (%) |\n| :--: | :--: | :--: | :--: | :--: | :--: |\n| Video | 369.51 | 10.7 | 108.91 | 46.38 | 77.28 |\n| AdaFuse | 15.2 | 2.04 | 38.85 | 36.35 | 48.15 |\n| STTS | 7.19 | 1.63 | 36.63 | 34.54 | 32.24 |\n| ListenToLook | 3.1 | 0.43 | 25.53 | 34.4 | 39.25 |\n| EgoDistill | 1.91 | 0.25 | 20.56 | 37.95 | 44.95 |\n\nWe can observe from the table that EgoDistill requires significantly less computation while achieving good performance. Note that all the accuracy numbers here can be found in Figure 3 and Table 2 of our main paper; this table is simply to put all the information together in one place per the reviewer’s suggestion\n"}]

Author rebuttal:
We sincerely thank all the reviewers for their hard work and constructive feedback! 

Four reviewers recommend accepting (one as a Strong Accept). 

Overall the reviewers give very positive comments on our work, e.g., “contributes significant advancements to the field of egocentric video understanding”, “The motivation is quite good and make sense. I like it.”, “The authors made a keen observation in using a separate modality.”,  “The proposed architecture is novel and yields impressive results.” “Claims are well supported”, “novel and yields impressive results”, “much more efficient”, “presentation is good and I can understand it very well”, “very well written and easy to follow”, “innovative approach”, “surpasses state-of-the-art efficient video understanding”, “no major weaknesses”, We thank them for their appreciation.

We address questions from different reviewers separately. We upload a **pdf** file in the overall rebuttal for figures used to reply to some of the reviewers’ questions.


Meta-review: Four of the five reviewers recommend acceptance; the authors reasonably address the fifth reviewer's concerns, although the reviewer did not respond.  Based on the reviews and responses, the AC recommends accepting the paper.  Authors are recommended to revise the work based on the reviewers' suggestions.
Decision: Accept (poster)