Title: EgoDistill: Egocentric Head Motion Distillation for Efficient Video Understanding
Venue: NeurIPS 2023
Accepted: yes
Keywords: Egocentric Video; IMU; Efficient Video Understanding
Abstract: Recent advances in egocentric video understanding models are promising, but their heavy computational expense is a barrier for many real-world applications. To address this challenge, we propose EgoDistill, a distillation-based approach that learns to reconstruct heavy ego-centric video clip features by combining the semantics from a sparse set of video frames with head motion from lightweight IMU readings. We further devise a novel IMU-based self-supervised pretraining strategy. Our method leads to significant improvements in efficiency, requiring 200× fewer GFLOPs than equivalent video models. We demonstrate its effectiveness on the Ego4D and EPIC- Kitchens datasets, where our method outperforms state-of-the-art efficient video understanding methods.

EgoDistill: Egocentric Head Motion Distillation for Efficient Video Understanding

Abstract

Recent advances in egocentric video understanding models are promising, but their heavy computational expense is a barrier for many real-world applications. To address this challenge, we propose EgoDistill, a distillation-based approach that learns to reconstruct heavy egocentric video clip features by combining the semantics from a sparse set of video frames with the head motion from lightweight IMU readings. We further devise a novel self-supervised training strategy for IMU feature learning. Our method leads to significant improvements in efficiency, requiring 200 × fewer GFLOPs than equivalent video models. We demonstrate its effectiveness on the Ego4D and EPIC-Kitchens datasets, where our method outperforms state-of-the-art efficient video understanding methods. Project page: https://vision.cs.utexas.edu/projects/egodistill/

1 Introduction

Recent advances in augmented and virtual reality (AR/VR) technology have the potential to change the way people interact with the digital world, much like the smartphone did in the previous decade. A fundamental requirement for AR/VR systems is the ability to recognize user behavior from egocentric video captured from a head-mounted camera. Towards this goal, several egocentric video datasets have been proposed in recent years, spurring increasing attention of the research community [26, 56, 11].

Current egocentric video understanding models use powerful clip-based video backbones that operate on video clips of a few seconds at a time [18, 54, 25, 16, 12, 55, 43, 44]. Despite encouraging performance, these models typically process densely sampled frames with temporally-aware operations, making them computationally heavy. This makes them impractical for AR/VR devices with constrained resources, or for real-time video applications that require low latency. How to efficiently perform egocentric video understanding is therefore an important, yet unsolved problem.

To address this issue, we take inspiration from how animals perceive the world with ego-motion. Neuroscience research has found that during active movement, the animal visual cortex receives and encodes head motion signals from the motor cortex for visual processing [27, 52, 53], highlighting its role in the efficient understanding of the animal's visual stream. Inspired by this phenomenon, we explore the relationship between human head motion and ego-video for efficient video understanding.

In practice, we consider head motion signals captured by the inertial measurement unit (IMU) of a head-mounted camera. IMU measures motion from an accelerometer and gyroscope and is widely available on popular wearable devices. Prior work leverages IMU as an extra modality for human action recognition [68, 69, 13] (e.g., jumping, walking), or as geometric cues for visual-inertial odometry [7, 20, 71]. In contrast, we propose to drive efficient video understanding by drawing on IMU as a substitute for dense video frame observations. The intuition is as follows. A video clip contains two things: semantic content (appearance of objects, places, people) and dynamics (how the scene and the camera move). While densely sampled frames are sure to capture both of the above-as done by current clip models [54, 16, 17]-we hypothesize they are sometimes overkill. For a short

+

Camera motion (IMU)

EgoDistill

video clip, much of the semantic content is intelligible from even a single frame; meanwhile, the head motion provides a good portion of the dynamics, implicitly revealing how the visual appearance changes across neighboring frames.

Building on this insight, we introduce EgoDistill, an approach that learns to reconstruct dense egocentric video clip features using temporally sparse visual observations (as few as one RGB frame) together with the head motion from IMU. Specifically, EgoDistill employs a new form of knowledge distillation from video models. During training, we train a lightweight model that takes sparsely sampled image(s) and IMU to approximate video features extracted by a powerful but expensive video model. We further improve the model with a novel IMU-guided self-supervised training stage. During inference, we directly utilize the lightweight model for egocentric video recognition, leading to much higher efficiency. Our model is flexible to the target heavy video feature, as we demonstrate with multiple current leading egocentric video models [54, 16, 18, 17]. See Figure 1.

Importantly, EgoDistill offers a major upgrade in efficiency. Low-dimensional IMU and a few frames are much more efficient to process than a dense stack of frames. In practice, EgoDistill uses 200 × fewer GFLOPs than the original video model. Furthermore, our approach is economical - the GoPro IMU sensor in our experiments (BOSCH BMI260) costs only $3 -and it is practical - most AR/VR devices and phones have in-built IMU sensors, making our method widely adoptable.

We experiment on the two largest egocentric action recognition datasets: Ego4D [26] and EPICKitchens-100 [11]. We show that IMU coupled with an image offers better cross-modality knowledge distillation performance than images alone or images with audio. For a typical 50-minute egocentric video, EgoDistill reduces model inference time from 25 minutes to 36 seconds . Moreover, with only 1-4 frames, our lightweight distillation model achieves a better accuracy-efficiency trade-off than state-of-the-art models for adaptively sampling video content [50, 65]. Notably, we surpass the accuracy of even these fast approaches by a large margin while requiring 4-8 × less computation.

2 Related Work

IMU for activity recognition. Recent work explores using the IMU sensor on mobile devices for human activity recognition of actions like walking, jumping, or sitting [47, 59, 60, 3, 61]. Normally, these models use IMU sensors mounted on human body joints [60, 9, 42], waist-mounted [41] or in-pocket smartphones [33]. See [64] for a survey. Abundant work in video recognition explores ways to learn from RGB coupled with other modalities-audio [1, 22, 38], optical flow [57, 58, 19] or both [51, 36, 32]-but much fewer use IMU [48], and unlike our work, they focus on third-person video [68, 69, 13] and do not target model efficiency. Our idea is for IMU to help reconstruct expensive video features, rather than simply fuse IMU with RGB for multi-modal recognition.

IMU for odometry. Inertial odometry aims to estimate the position and orientation of the camerawearer with readings from the IMU sensor. Traditionally, methods rely on IMU double integration [4] or enhancements thereof [40, 5, 35]. Recent data-driven methods automatically learn to perform inertial odometry with supervised [30, 70] or self-supervised learning [7], or combine IMU and

visual input for more robust estimates with visual-inertial odometry [20, 71]. While IMU can convey geometric ego-motion to our learned model, our goal is to produce efficient egocentric video features rather than to output odometry.

Visual feature learning with IMU. IMU is also used to learn better vision features [34, 14, 15, 63], e.g., to encourage image features that are equivariant with ego-motion [34], to predict an IMUcaptured body part (leg, hand) [14, 15], or to predict video-IMU correspondence [63], for applications like action recognition [15, 63] and scene understanding [14, 34]. While these results reinforce that IMU can inject embodied motion into visual features, our idea to use head motion to infer pretrained video features for speedy video understanding is distinct.

Efficient video recognition. Being crucial for mobile applications, efficient video recognition has received increasing attention in recent years. Several studies focus on designing lightweight architectures [30, 17, 37, 72, 62] by reducing 3D CNN operations across densely-sampled frames. In contrast, our model focuses on inputs with sparsely-sampled frames. As we show in experiments, our method is compatible with different video architectures.

Another line of research achieves efficiency by adaptively selecting video content to process. Some reduce temporal redundancy by adaptively selecting which video clip [39], frames [49, 24], and/or feature channel [50] to process and which to skip, while others reduce spatial redundancy , selecting for each frame a smaller but important region to process [66, 67]. Other work dynamically selects tokens in video transformers among both the spatial and temporal dimensions [65]. Our idea is complementary: rather than dynamically subsample the available video content, we show how to infer 'full" video features for every clip using static image(s) and motion data. Our results outperform state-of-the-art sampling models (cf. Sec. 4). In addition, we focus on egocentric video, where head motion is particularly meaningful for inferring unobserved visual content. To our knowledge, ours is the first technique specifically aimed at accelerating egocentric video processing.

Multimodal distillation. Knowledge distillation aims to transfer knowledge learned by an expensive model to a lightweight model [31]. Recent work explores multimodal distillation, e.g., transferring from a RGB model to a flow or depth model [23, 28], from a 3D model to a 2D model [45], or from a visual model to audio model [2, 21]. More related to our work, the ListenToLook model [22] incorporates both clip subsampling and video-to-audio distillation for fast activity recognition in third-person video. While we share a similar motivation, our contribution is distinct: we explore the relationship between the camera-wearer's head motion and RGB signals for egocentric video, and we introduce a novel IMU-based pretraining strategy. Our experiments show EgoDistill's advantage over ListenToLook in terms of the speed-accuracy tradeoff on egocentric video datasets. Furthermore, whereas [22] attempts only a single video model (R(2+1)D), we show our idea generalizes well to multiple video models, meaning it can be dropped in to benefit multiple popular frameworks.

3 Approach

We introduce EgoDistill, which uses sparsely-sampled frames and head motion from IMU to approximate the features of heavy video models for efficient egocentric video understanding. We first introduce the egocentric action recognition task (Sec. 3.1). Then, we introduce our pipeline (Sec. 3.2), our distillation model and training objective (Sec. 3.3), and our self-supervised IMU feature learning (Sec. 3.4). Figure 2 overviews our approach.

3.1 Egocentric action recognition

Given a fixed-length video clip V ∈ R T × H × W × 3 consisting of T RGB frames of size H × W and a set of C action classes, the task of action recognition is to output a score for each action class, representing its likelihood. Typically, this is done with a powerful but expensive video model Ω , that directly operates on all the available frames to output the C class logits Ω( V ) ∈ R C . Ω is trained with standard classification loss:

where V i is the i -th video clip in the dataset, c i is the corresponding ground-truth action label, σ is the softmax function, and L CE is cross-entropy loss. Popular video recognition models use clips that are typically /tildelow 2 seconds long [54, 16, 18]. For longer videos, scores are averaged across all clips it contains to infer the video action label.

Video clip

Video model

Frame

IMU

Image Encoder

IMU Encoder

Fusion Layer

FC-Softmax

FC-Softmax

Frame

IMU

Image Encoder

IMU Encoder

Frame

Image Encoder

IMU  Predictor

3.2 Efficient video inference with head motion

Processing the video clip V for action recognition is computationally intensive; however, the computation cost can be modulated depending on how frames from the clip are used. On the one hand, clip-based models [18, 54, 16, 17] process most (or all) frames in a video clip V to achieve strong recognition performance, but come at a high computational cost. On the other hand, frame-level models [49, 24, 51, 67] only process one (or a small number) of frames from V and are more efficient, but suffer a drop in performance as a result. Our goal is to train a frame-based model that can approximate heavy clip-based model performance while maintaining high efficiency.

For this, we turn to head motion captured by IMU. Along with RGB frames, each video clip is paired with IMU measurements M that record the camera (head) motion during the video. Specifically, the IMU readings are composed of 6-dimensional accelerometer and gyroscope measurements in the xyz axes, which encode strong temporal motion information about camera pose changes (both translation and rotation) across frames.

For short video clips, a set of sparsely sampled frames I often already captures most appearance information. Complementary to this, the IMU readings capture camera motion information (see below for discussion on scene motion). Moreover, IMU is very efficient to process due to its low dimensionality. By processing inputs from these two sources with a lightweight frame-based model, we can infer the semantic and dynamic features of a heavier clip-based video model.

Given I and M , we train an efficient lightweight model Φ to approximate the output of video model Ω . Specifically, we train our EgoDistill model Φ that achieves

Such a lightweight model will be able to approximate the result of the heavy video model, while being much more efficient. Our approach is agnostic to the specific video model Ω ; in experiments, we demonstrate its versatility for MotionFormer [54], MViT [16], SlowFast [18] and X3D [17].

In practice, we uniformly sample N frames 1 from V to obtain I . We can achieve a trade-off between efficiency and performance by changing the number of frames N . In our experiments we use very low values of N (1 to 4 frames). In the next section, we discuss how we train Φ .

3.3 Video feature distillation with IMU

We address Equation 2 via knowledge distillation [31], where we transfer knowledge learned by the expensive teacher model Ω to a lightweight student model Φ . Next we present the design of Φ and the training objectives, followed by our self-supervised IMU feature pretraining stage in Sec. 3.4.

We design Φ to be a two-stream model. For a video clip and associated IMU signal ( I , M ) , we extract image features z I = f I ( I ) and IMU features z M = f M ( M ) using lightweight feature encoders f I , f M respectively. Then, we fuse z I and z M with a fusion network Π to obtain the fused VisIMU feature z ϕ = Π( z I , z M ) . Finally, a fully-connected layer uses the fused feature to predict class logits Φ( I , M ) ∈ R C . The fused feature z ϕ contains semantic information from the image frame coupled with complementary motion information from IMU, allowing us to accurately reconstruct the video clip feature. See Figure 2.

We train Φ with a combination of three losses, as follows. First, we train Φ to approximate the original video feature z V from the video model Ω :

This cross-modal loss encourages the fused feature z ϕ to match the video feature, i.e., the combined features from the different modalities should match in the feature space.

Training with L 1 alone does not fully capture the classification output of Ω . Therefore, we also train Φ with a knowledge distillation loss:

where ( V i , I i , M i ) represents the i -th clip in the dataset, D KL measures KL-divergence between the class logits from the teacher model Ω and student model Φ , and τ is a temperature parameter. Intuitively, L KD casts the output of the video teacher model as a soft target for training the student model. In this way, the student model learns to better generalize by mimicking the output distribution of the heavy video model.

Finally, to further encourage the features to preserve elements useful for activity understanding, we also compute an action classification loss:

where c i is the ground-truth action label, following Equation 1. The final training loss is a combination of these three loss functions:

where α controls the balance between knowledge distillation and activity training [31], and β controls the weight for feature space matching.

Critically, processing a few image frame(s) and the low-dimensional IMU readings is substantially faster than processing the entire video. Once trained, our model approximates the behavior of the source video model for recognition tasks, with the key benefit of efficient egocentric recognition.

What kind of motion does our model preserve? Video motion decomposes into scene motion (e.g., how the objects and the camera wearer's hands are moving on their own), and camera motion (i.e., how the camera wearer is moving their head). By itself, IMU would directly account only for camera motion, not scene motion. However, by learning to map from the RGB frame and IMU to the full video feature, we are able to encode predictable scene motions tied to scene content, e.g., how does hand and object movement in subsequent frames relate to the camera wearer's head motion. Moreover, our model is applied to relatively short clips (1-2 seconds) in sequence, which means the appearance content is regularly refreshed as we slide down to process the longer video.

3.4 Self-supervised IMU feature learning

The success of EgoDistill depends on how well the IMU feature encoder f M extracts useful camera motion information and associates it with the visual appearance change in the video clip. In this way EgoDistill can learn to anticipate unseen visual changes in the video with I and M . We design a self-supervised pretraining task to initialize the weights of f M to achieve this.

Specifically, for each clip V , we obtain its first and last frames ( I 0 , I T ) as well as the IMU M . We first extract visual features z 0 I , z T I and IMU feature z M with feature extractors f I and f M

mentioned above. Then, we train a feature predictor h to predict the IMU feature ˆ z M = h ( z 0 I , z T I ) . By connecting ˆ z M -afunction of image features only-with z M , we encourage f M to extract useful camera motion features specifically associated with the visual appearance changes. Note that those appearance changes may include scene motion. Therefore, we include an L 1 loss to train f M , which encourages f M to extract motion features accounting for scene motion in the full video.

We train f M , h , and the fusion network Π using L 1 and NCE loss [29]: L pretrain = L NCE + L 1 , where

̸

We sample negative examples z M j from other instances in the same mini-batch for j = i , and sim ( q, k ) = exp( q · k | q || k | 1 τ ′ ) with temperature τ ′ = 0 . 1 2 .

To summarize, prior to the main training stage of Equation 6, we pretrain the IMU feature extractor f M and fusion network Π . As we will show below, both pretraining losses result in IMU features that are consistent with visual changes and lead to better finetuning performance.

4 Experiments

We evaluate our approach for resource-efficient action recognition.

4.1 Experimental setup

Datasets. We experiment on two large-scale egocentric action recognition datasets. (1) Ego4D [26] contains 3,670 hours of egocentric videos of people performing diverse tasks (from cooking to farming) across the globe. As action recognition is not part of the original Ego4D benchmark, we construct this task with annotations from the Hands+Objects temporal localization benchmark [26] (see Supp. for details). We include clips with paired IMU and audio 3 , and consider classes with at least 2 labeled instances. This results in a 94-class action recognition dataset with 8.5k training videos and 3.6k evaluation videos. (2) EPIC-Kitchens [11] contains 100 hours of egocentric videos capturing daily activities in kitchen environments. We use annotations from the action recognition benchmark. Similar to Ego4D, we select videos that have paired IMU and audio data, and split the resulting data by camera-wearer. This results in a 62-class action dataset with 29k training videos and 6.2k evaluation videos. For both datasets, we use 'verb' labels as the target for action recognition as they are well aligned to activity motions.

Evaluation metrics. To measure action recognition performance, we report the per-video top-1 accuracy on the validation set. We densely sample clips from each video and average their predictions to compute accuracy. To benchmark efficiency, we measure computational cost with FLOPs (floatingpoint operations) during inference.

Implementation details. In our main experiments, we use MotionFormer [54] as the video teacher model Ω due to its strong performance for egocentric video. For EPIC-Kitchens, we use the authors' provided checkpoint. For Ego4D, we finetune the above model for 50 epochs with 1 e -4 learning rate and 64 batch size on the training set. We use 16-frame input with sample rate 4. For the student model Φ , we use a ResNet-18 as the image backbone f I and a 1D Dilated CNN [6] for the IMU backbone f M . The feature fusion module Π uses a concatenation operation following a two-layer fully-connected layer with hidden dimension 1024. For each video clip, the input image(s) is resized to 224 × 224 , and the IMU is a 422 × 6 matrix (around 2 seconds with 198Hz frequency), representing the accelerometer and gyroscope readings along the xyz axes. For the image input, we uniformly sample N frames from the video clip. If N > 1 , we use f I to sequentially generate features for each frame and aggregate them with a GRU module [10]. For both datasets, we first pretrain the model with the self-supervised objective (Sec. 3.4) for 50 epochs with AdamW [46] using batch size 64 and learning rate 1 e -4 . Then, we finetune all the models with the same setting (Equation 6). We set α = 0 . 95 and β = 1 . 0 based on validation data. For Ego4D, we set τ = 10 . 0 and train the model for 150 epochs. For EPIC-Kitchens, we set τ = 1 . 0 and train for 50 epochs.

2

4

6

8

10

12

14

16

Inference cost per video clip (GFLOPs)

33

34

35

36

37

38

Ego4D

2

4

6

8

10

12

14

16

Inference cost per video clip (GFLOPs)

30

35

40

45

50

EPIC-Kitchens accuracy (%)

EPIC-Kitchens

EgoDistill (ours)

AdaFuse [50]

STTS [65]

ListenToLook [22]

VisOnly-Distill

VisIMU

VisOnly

4.2 Baselines

We compare with the following methods: (1) AdaFuse [50] trains a lightweight policy network to adaptively compute (or skip) feature map channels for each frame during inference. We use the AdaFuse TSN R50 model with the provided hyper-parameters. (2) STTS [65] trains a module to rank spatio-temporal tokens from videos in a transformer-based model, and selects only the top-K tokens to speed up inference. We use a MViT-B16 backbone [17]. (3) ListenToLook [22] uses the audio-based distillation module from [22] following the same audio processing and model architecture.

These methods represent recent advances in efficient video recognition models. AdaFuse represents state-of-the-art approaches that achieve efficiency by reducing temporal redundancy in CNN models. STTS is one of the most recent approaches that efficiently reduces both spatial and temporal redundancy in ViT models, which achieves the state-of-the-art on Kinectics-400 [8]. ListenToLook also relies on distillation, but using audio rather than head motion. For each model we generate multiple versions with different computation budgets to plot accuracy vs. GFLOPs. We train all AdaFuse and STTS models with 4 input frames to align with the maximum frames used by our model. For AdaFuse, we use the only provided hyper-parameter in the paper. 4 For STTS, we use three provided variants: T 0 0 . 5 -S 4 0 . 7 , T 0 0 . 8 -S 4 0 . 9 and the full model without token selection. For ListenToLook we adopt the same efficiency-accuracy trade-off as our method, i.e., varying the number of input frames.

We also test variants of our method: (1) VisOnly-Distill is our model without the IMU branch and fusion layer, but trained with the same loss function. This model reveals the importance of IMU for distillation. (2) VisIMU is our model trained with only L GT in Equation 5. It shows the effectiveness of distillation from the video model compared with directly training the features with action labels. (3) VisOnly is an image-only model trained with L GT, which serves as the baseline.

4.3 Main Results

Importance of IMU-guided distillation. Figure 3 shows the accuracy vs. efficiency curves. Methods towards the top-left of the plot represent those with both high accuracy and efficiency. Our method achieves good accuracy with low computational cost. Specifically, on EPIC-Kitchens, when N = 1 , EgoDistill improves over VisOnly-Distill by 8 . 4% with only a small increase in computation. This shows the effectiveness of IMU for reconstructing egocentric video features. Compared to VisIMU, EgoDistill improves by 9 . 9% , showing the effectiveness of knowledge distillation from the video model. Importantly, this reveals that EgoDistill does not simply benefit from the extra IMU context; our idea to approximate video features is necessary for best results. We see similar results on Ego4D.

Comparison with the state of the art. Figure 3 also shows that EgoDistill achieves better accuracy with less computation than existing efficient video recognition models AdaFuse [50], STTS [65], and ListenToLook [22]. With N = 4 frames, EgoDistill surpasses STTS by 7 . 4% and AdaFuse by 4 . 2% on EPIC-Kitchens, with 2 × fewer GFLOPs, and surpasses both methods by 2 . 1% on Ego4D. When we use fewer frames, EgoDistill can still outperform STTS and AdaFuse using 4 × to 8 × fewer

GFLOPs. In addition, EgoDistill surpasses ListenToLook by 7 . 4% and 2 . 9% on EPIC-Kitchens and Ego4D respectively, which suggests that head motion is more informative than audio for feature reconstruction in egocentric video.

4.4 Analysis

Model component ablations. Table 1 ablates different design choices in our model, setting N = 1 for all experiments. We observe that training EgoDistill without L 1 , L KD or L GT deteriorates performance. Specifically, training without L KD leads to the largest performance drop, which indicates that knowledge distillation is an essential component in our approach. Training without L 1 also degrades performance, which shows the importance of our idea to align features from the different modalities. Further, our self-supervised pretraining stage is very effective at training the IMU extractor to encode useful motion information that is consistent with visual feature change. We find an alternative IMU extractor pretraining strategy of supervised action recognition task is inferior; we outperform pretaining with action recognition by 3 . 74% on EPIC-Kitchens and 0 . 96% on Ego4D. Finally, we compare with a model that simply does multi-modal recognition with IMU (top row). The strong contrast here indicates the importance of our idea to use IMU to predict video model features, as opposed to simply adding IMU as an additional input modality.

Impact of teacher video model architecture. In our main experiments we use MotionFormer [54] as the teacher video model due to its strong performance on egocentric video tasks. To emphasize the generality of our idea, we show the performance of EgoDistill with other video teacher architectures in Table 2. Similar to the MotionFormer model, we train these models on each of the labeled datasets, and then train our model using the resulting video models as the teacher. As expected, better video teacher models lead to better student model performance. More importantly, we observe consistent improvement by EgoDistill over the VisOnly-Distill baseline on both datasets and with different video teacher models, highlighting our idea's generality and versatility.

Efficiency analysis. To compare the efficiency of different models, aside from GFLOPs, we also compare their inference run-time and number of parameters. For run-time, we record the time spent to infer a single video clip's label with a single A40 GPU, and take the average time over the full validation datasets of Ego4D and EPIC-Kitchens with batch-size of 32. Table 3 shows the results. EgoDistill runs much faster than the other methods. Notably, it reduces the GFLOPs of MotionFormer by nearly 200 × . Furthermore, it runs 6 . 5 × faster than STTS [65] while achieving 4 . 4% higher accuracy on EPIC-Kitchens.

Effect of fusing IMU data. EgoDistill achieves superior performance by combing IMU information when distilling features from video models. Here we explore how helpful it is to combine IMU

close: 0.88

open: 0.01

close: 0.18

open: 0.34

GT: close

EgoDistill

VisOnly-D

put: 0.40

take: 0.10

put: 0.14

take: 0.44

GT: put

EgoDistill

VisOnly-D

information with the most recent efficient video understanding method, i.e. , STTS [65]. 5 To this end, for STTS, we add an IMU encoding branch identical to the one used in EgoDistill, and concatenate the IMU feature with video features before the last FC layer. We train both the video and IMU branches simultaneously for the action recognition task. Results in Table 4 show that combining IMU features indeed helps STTS, yet EgoDistill's performance is significantly better than STTS's. These results again indicate that our advantage is not simply access to IMU; rather it is the proposed way we leverage vision and IMU. The reason that EgoDistill is able to outperform STTS+IMU is mainly that EgoDistill learns to jointly distill information from a video feature with image and IMU inputs. This distillation process will encourage the model to extract the interplay between static semantic information in the image and the dynamic information in IMU, as it is trained to reconstruct the video model feature. On the other hand, simple concatenation of the IMU feature to the image feature for action classification will not exploit the relationship between the image and IMU-it lacks the training signal to target the dynamic video clip feature-leading to inferior performance.

Effect of using partial IMU . Is the net displacement of the camera what matters, or does the motion in between give cues? To answer this, we experiment with feeding EgoDistill with partial IMU readings of the full video clip. Specifically, we train and evaluate our model with only the first K% of IMU readings of the full clip and pad the rest with 0. We show the results in Table 5. This ablation study shows that more complete IMU readings yield the best results, while using 25% and 50% IMU data achieves the majority of the gain over the VisOnly-D (no IMU) baseline.

Effect of number of input frames . To investigate the effect of using different numbers of input frames, we conduct an ablation study with N . Specifically, we use three settings: N = 1 (use the center frame), N = 2 (use the start frame and end frame), N = 4 (uniformly sample 4 frames). We compare EgoDistill and VisOnly-D under different settings in Table 6. We observe that only using the overall displacement between the end frame and the starting frame ( N = 2 ) leads to better

EgoDistill

EgoDistill

VisOnly

EgoDistill

EgoDistill

EgoDistill

VisOnly-D

EgoDistill

performance by VisOnly-D than using only one frame ( N = 1 ). However, the motion trajectory still helps EgoDistill obtain significantly better performance in either case. This result indicates the information in the motion trajectory captured by IMU. Note that while N = 2 , 4 leads to better performance, it requires higher computational cost.

What do EgoDistill features capture? To explore this, we pair a single input frame with different IMU clips as inputs to EgoDistill, then retrieve the nearest video clip for each resulting anticipated video feature. We also compare with the VisOnly-Distill result. Figure 5 illustrates this. We see that EgoDistill outputs video features that all involve interaction with the cabinet (right panel), and is able to use different IMU inputs to retrieve different video clips that show consistent camera motion. In contrast, different input IMUs lead to corresponding camera motion in the frames. The result shows that EgoDistill approximates video features that capture both semantic and motion information, whereas VisOnly-Distill only retains the semantic context to retrieve a single clip.

Is there evidence EgoDistill captures scene motion? Figure 4 shows how our features learned with head motion can nonetheless expose certain scene motion cues. EgoDistill improves the accuracy over VisOnly-Distill on ambiguous categories (like close and put ) by a large margin ( 20 . 3% and 10 . 4% on EPIC-Kitchens, 8 . 5% and 3 . 9% on Ego4D). See caption and Supp. video for more details.

Limitations of EgoDistill . Although EgoDistill is useful for many action categories in egocentric videos, for some other classes it is less effective. We observe that video clips that EgoDistill does not perform well tend to contain little head motion -in which case IMU is redundant to the RGB frameor drastic head motion that is weakly correlated with the camera wearer's activity and introduces blur to the frame. Additionally, EgoDistill might not be useful when we want to focus on the 'noun' part of the action (e.g., differentiating watching TV and using the cell phone), as IMU information does not contain information about the object semantics in the scene.

5 Conclusion

We present EgoDistill, the first model to explore egocentric video feature approximation for fast recognition. Experiments on action recognition on Ego4D and EPIC-Kitchens demonstrate that our model achieves a good balance between accuracy and efficiency, outperforming state-of-theart efficient video understanding methods. Our approach has great potential to accelerate video understanding for egocentric videos using a data stream that is already ubiquitous in egocentric cameras. In the future, we plan to investigate how to use head motion for long-term human activity understanding with room context and visual correspondence learning for multi-view videos.

Acknowledgements We thank the UT Austin vision group for helpful discussions. We also thank the anonymous reviewers for their insightful suggestions. UT Austin is supported in part by IFML NSF AI Institute. KG is paid as a research scientist at Meta.

References

Relja Arandjelovi´ c and Andrew Zisserman. Look, listen and learn. 2017 IEEE International Conference on Computer Vision (ICCV) , pages 609-617, 2017. 2

Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. In Proceedings of the 30th International Conference on Neural Information Processing Systems , NIPS'16, page 892-900, Red Hook, NY, USA, 2016. Curran Associates Inc. 3

Anna Borowska-Terka and Pawel Strumillo. Person independent recognition of head gestures from parametrised and raw signals recorded from inertial measurement unit. Applied Sciences , 10(12), 2020. 2

John E. Bortz. A new mathematical formulation for strapdown inertial navigation. IEEE Transactions on Aerospace and Electronic Systems , AES-7:61-66, 1971. 2

Agata Brajdic and Robert K. Harle. Walk detection and step counting on unconstrained smartphones. Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing , 2013. 2

M. Brossard, S. Bonnabel, and A. Barrau. Denoising imu gyroscopes with deep learning for open-loop attitude estimation. IEEE Robotics and Automation Letters , 5(3):4796-4803, 2020. 6

Xiya Cao, Caifa Zhou, Dandan Zeng, and Yongliang Wang. Rio: Rotation-equivariance supervised learning of robust inertial odometry. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6614-6623, June 2022. 1, 2

Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 6299-6308, 2017. 7

Chen Chen, Roozbeh Jafari, and Nasser Kehtarnavaz. Utd-mhad: A multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor. In 2015 IEEE International Conference on Image Processing (ICIP) , pages 168-172, 2015. 2

Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP , 2014. 6

Dima Damen, Hazel Doughty, Giovanni Maria Farinella, , Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100. International Journal of Computer Vision (IJCV) , 130:33-55, 2022. 1, 2, 6

Samyak Datta, Sameer Dharur, Vincent Cartillier, Ruta Desai, Mukul Khanna, Dhruv Batra, and Devi Parikh. Episodic memory question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 19119-19128, June 2022. 1

Alexander Diete, Timo Sztyler, and Heiner Stuckenschmidt. Vision and acceleration modalities: Partners for recognizing complex activities. In 2019 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops) , pages 101-106, 2019. 1, 2

Kiana Ehsani, Hessam Bagherinezhad, Joseph Redmon, Roozbeh Mottaghi, and Ali Farhadi. Who let the dogs out? modeling dog behavior from visual data. In CVPR , 2018. 3

Kiana Ehsani, Daniel Gordon, Thomas Nguyen, Roozbeh Mottaghi, and Ali Farhadi. Learning visual representation from human interactions. International Conference on Learning Representations , 2021. 3

Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In ICCV , 2021. 1, 2, 3, 4, 8

Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 200-210, 2020. 1, 2, 3, 4, 7, 8

Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 6201-6210, 2019. 1, 2, 3, 4, 8

Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. Convolutional two-stream network fusion for video action recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1933-1941, 2016. 2

Christian Forster, Luca Carlone, Frank Dellaert, and Davide Scaramuzza. On-manifold preintegration for real-time visual-inertial odometry. IEEE Transactions on Robotics , 33(1):1-21, 2017. 1, 3

C. Gan, H. Zhao, P. Chen, D. Cox, and A. Torralba. Self-supervised moving vehicle tracking with stereo sound. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 7052-7061, Los Alamitos, CA, USA, nov 2019. IEEE Computer Society. 3

Ruohan Gao, Tae-Hyun Oh, Kristen Grauman, and Lorenzo Torresani. Listen to look: Action recognition by previewing audio. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10457-10467, 2020. 2, 3, 6, 7, 9

Nuno C. Garcia, Pietro Morerio, and Vittorio Murino. Modality distillation with multiple stream networks for action recognition. In The European Conference on Computer Vision (ECCV) , September 2018. 3

Amir Ghodrati, Babak Ehteshami Bejnordi, and Amirhossein Habibian. Frameexit: Conditional early exiting for efficient video recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2021. 3, 4

Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra. Omnivore: A Single Model for Many Visual Modalities. In CVPR , 2022. 1

Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane, Tien Do, Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Christian Fuegen, Abrham Gebreselasie, Cristina Gonzalez, James Hillis, Xuhua Huang, Yifei Huang, Wenqi Jia, Weslie Khoo, Jachym Kolar, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang, Xindi Wu, Takuma Yagi, Yunyi Zhu, Pablo Arbelaez, David Crandall, Dima Damen, Giovanni Maria Farinella, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. Ego4d: Around the World in 3,000 Hours of Egocentric Video. In IEEE/CVF Computer Vision and Pattern Recognition (CVPR) , 2022. 1, 2, 6

Grigori Guitchounts, Javier Masís, Steffen B.E. Wolff, and David Cox. Encoding of 3d head orienting movements in the primary visual cortex. Neuron , 108(3):512-525.e4, 2020. 1

Saurabh Gupta, Judy Hoffman, and Jitendra Malik. Cross modal distillation for supervision transfer. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2827-2836, 2016. 3

Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , volume 9 of Proceedings of Machine Learning Research , pages 297-304, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. 6

Sachini Herath, Hang Yan, and Yasutaka Furukawa. Ronin: Robust neural inertial navigation in the wild: Benchmark, evaluations, & new methods. In 2020 IEEE International Conference on Robotics and Automation (ICRA) , pages 3146-3152, 2020. 2, 3

Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. NIPS Deep Learning Workshop , abs/1503.02531, 2014. 3, 4, 5

Yu Huang, Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. What makes multimodal learning better than single (provably). In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021. 2

Andrey Ignatov. Real-time human activity recognition from accelerometer data using convolutional neural networks. Applied Soft Computing , 62:915-922, 2018. 2

D. Jayaraman and K. Grauman. Learning image representations tied to egomotion. In ICCV , 2015. 3

A.R. Jimenez, F. Seco, C. Prieto, and J. Guevara. A comparison of pedestrian dead-reckoning algorithms using a low-cost mems imu. In 2009 IEEE International Symposium on Intelligent Signal Processing , pages 37-42, 2009. 2

Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen. Epic-fusion: Audio-visual temporal binding for egocentric action recognition. In IEEE/CVF International Conference on Computer Vision (ICCV) , 2019. 2

Dan Kondratyuk, Liangzhe Yuan, Yandong Li, Li Zhang, Mingxing Tan, Matthew Brown, and Boqing Gong. Movinets: Mobile video networks for efficient video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16020-16030, 2021. 3

Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from selfsupervised synchronization. In Proceedings of the 32nd International Conference on Neural Information Processing Systems , NIPS'18, page 7774-7785, Red Hook, NY, USA, 2018. Curran Associates Inc. 2

Bruno Korbar, Du Tran, and Lorenzo Torresani. Scsampler: Sampling salient clips from video for efficient action recognition. 2019 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 6231-6241, 2019. 3

Masakatsu Kourogi and Takeshi Kurata. A method of pedestrian dead reckoning for smartphones using frequency domain analysis on patterns of acceleration and angular velocity. In Record - IEEE PLANS, Position Location and Navigation Symposium , pages 164-168, 05 2014. 2

Jennifer R. Kwapisz, Gary M. Weiss, and Samuel A. Moore. Activity recognition using cell phone accelerometers. SIGKDD Explor. Newsl. , 12(2):74-82, mar 2011. 2

Heike Leutheuser, Dominik Schuldhaus, and Bjoern M. Eskofier. Hierarchical, multi-sensor based classification of daily life activities: Comparison with state-of-the-art algorithms using a benchmark dataset. PLOS ONE , 8(10):1-11, 10 2013. 2

Yiming Li, Ziang Cao, Andrew Liang, Benjamin Liang, Luoyao Chen, Hang Zhao, and Chen Feng. Egocentric prediction of action target in 3d. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2022. 1

Shaowei Liu, Subarna Tripathi, Somdeb Majumdar, and Xiaolong Wang. Joint hand motion and interaction hotspots prediction from egocentric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3282-3292, June 2022. 1

Zhengzhe Liu, Xiaojuan Qi, and Chi-Wing Fu. 3d-to-2d distillation for indoor scene parsing. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4462-4472, 2021. 3

Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam, 2018. 6

Niall Lyons, Avik Santra, and Ashutosh Pandey. Improved deep representation learning for human activity recognition using imu sensors. In 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA) , pages 326-332, 2021. 2

Sharmin Majumder and Nasser Kehtarnavaz. Vision and inertial sensing fusion for human action recognition: A review. IEEE Sensors Journal , 21(3):2454-2467, 2021. 2

Yue Meng, Chung-Ching Lin, Rameswar Panda, Prasanna Sattigeri, Leonid Karlinsky, Aude Oliva, Kate Saenko, and Rogerio Feris. Ar-net: Adaptive frame resolution for efficient action recognition. arXiv preprint arXiv:2007.15796 , 2020. 3, 4

Yue Meng, Rameswar Panda, Chung-Ching Lin, Prasanna Sattigeri, Leonid Karlinsky, Kate Saenko, Aude Oliva, and Rogerio Feris. Adafuse: Adaptive temporal fusion network for efficient action recognition. In International Conference on Learning Representations , 2021. 2, 3, 7, 9

Rameswar Panda, Chun-Fu Chen, Quanfu Fan, Ximeng Sun, Kate Saenko, Aude Oliva, and Rogerio Feris. AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition. In International Conference on Computer Vision (ICCV) , 2021. 2, 4

Philip R.L. Parker, Morgan A. Brown, Matthew C. Smear, and Cristopher M. Niell. Movement-related signals in sensory areas: Roles in natural behavior. Trends in Neurosciences , 43(8):581-595, 2020. 1

Philip R. L. Parker, Elliott T. T. Abe, Emmalyn S. P. Leonard, Dylan M. Martins, and Cristopher M. Niell. Joint coding of visual input and eye/head position in v1 of freely moving mice. bioRxiv , 2022. 1

Mandela Patrick, Dylan Campbell, Yuki M. Asano, Ishan Misra Florian Metze, Christoph Feichtenhofer, Andrea Vedaldi, and João F. Henriques. Keeping your eye on the ball: Trajectory attention in video transformers. In Advances in Neural Information Processing Systems (NeurIPS) , 2021. 1, 2, 3, 4, 6, 8, 9

Chiara Plizzari, Mirco Planamente, Gabriele Goletto, Marco Cannici, Emanuele Gusso, Matteo Matteucci, and Barbara Caputo. E2(go)motion: Motion augmented event stream for egocentric action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 19935-19947, June 2022. 1

Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. Charades-ego: A large-scale dataset of paired third and first person videos. arXiv preprint arXiv:1804.09626 , 2018. 1

Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 27. Curran Associates, Inc., 2014. 2

Shuyang Sun, Zhanghui Kuang, Lu Sheng, Wanli Ouyang, and Wei Zhang. Optical flow guided feature: A fast and robust motion representation for video action recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2018. 2

Yin Tang, Lei Zhang, Fuhong Min, and Jun He. Multi-scale deep feature learning for human activity recognition using wearable sensors. IEEE Transactions on Industrial Electronics , pages 1-1, 2022. 2

Wenjin Tao, Haodong Chen, Md. Moniruzzaman, Ming C. Leu, Zhaozheng Yi, and Ruwen Qin. Attentionbased sensor fusion for human activity recognition using imu signals. ArXiv , abs/2112.11224, 2021. 2

Catherine Tong, Jinchen Ge, and Nicholas D Lane. Zero-shot learning for imu-based activity recognition using video embeddings. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies , 5(4):1-23, 2021. 2

Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6450-6459, 2018. 3

Satoshi Tsutsui, Ruta Desai, and Karl Ridgeway. How you move your head tells what you do: Selfsupervised video representation learning with egocentric cameras and imu sensors. ICCV Workshop on Egocentric Perception, Interaction and Computing (EPIC) , 2021. 3

Jindong Wang, Yiqiang Chen, Shuji Hao, Xiaohui Peng, and Lisha Hu. Deep learning for sensor-based activity recognition: A survey. Pattern Recognit. Lett. , 119:3-11, 2019. 2

Junke Wang, Xitong Yang, Hengduo Li, Liu Li, Zuxuan Wu, and Yu-Gang Jiang. Efficient video transformers with spatial-temporal token selection. In ECCV , 2022. 2, 3, 7, 8, 9

Yulin Wang, Zhaoxi Chen, Haojun Jiang, Shiji Song, Yizeng Han, and Gao Huang. Adaptive focus for efficient video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2021. 3

Yulin Wang, Yang Yue, Yuanze Lin, Haojun Jiang, Zihang Lai, Victor Kulikov, Nikita Orlov, Humphrey Shi, and Gao Huang. Adafocus v2: End-to-end training of spatial dynamic networks for video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022. 3, 4

Haoran Wei, Pranav Chopada, and Nasser Kehtarnavaz. C-mhad: Continuous multimodal human action dataset of simultaneous video and inertial sensing. Sensors , 20(10), 2020. 1, 2

Haoran Wei and Nasser Kehtarnavaz. Simultaneous utilization of inertial and video sensing for action detection and recognition in continuous action streams. IEEE Sensors Journal , 20(11):6055-6063, 2020. 1, 2

Hang Yan, Qi Shan, and Yasutaka Furukawa. Ridi: Robust imu double integration. In ECCV , 2018. 2

Mingyu Yang, Yu Chen, and Hun-Seok Kim. Efficient deep visual and inertial odometry with adaptive visual modality selection. ArXiv , abs/2205.06187, 2022. 1, 3

Mohammadreza Zolfaghari, Kamaljeet Singh, and Thomas Brox. ECO: efficient convolutional network for online video understanding. In ECCV , 2018. 3

Peer reviews:
comments: SUMMARY:
To improve the efficiency of ego-centric video representation learning, this paper presents a distillation-based approach that learns to reconstruct heavy egocentric video clip features from the combination of sparsely-sampled video frames and the camera motion from IMU.

STRENGTHS:
1. The paper is well-written and the easy to follow.

2. Improving the efficiency of ego-centric video representation learning is indeed valuable.

WEAKNESSES:
1. As mentioned in the paper, the dynamics in a video include both the movement of the scene and the camera (Line 34-35). I agree that IMU could provide some cues for the motion of the camera, however, I'm afraid it is difficult to deduce the motion of the scenes, which is more important for the video understanding (although some analysis is presented in Figure 6, it is still contradictory to the intuition). 

2. From Table 1, we can see that the performance improvements mainly come from L_{KD}, other components like L1, L1-pretrain, and L1-NCE, bring subtle improvements.

3. It could be better to compare the model performance in Table 4 (which is mismarked as Figure 4).

QUESTIONS:
Please refer to the weakness.

LIMITATIONS:
The authors have discussed the potential limitations in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 1
DATE: 2023-07-01

comments: SUMMARY:
The work presents a new approach and architecture, EgoDistill, which could efficiently perform action recognition through sparse video frames augmented with head motion data. The model goes through a two-stage training. A pre-training phase, which uses a contrastive loss that enables the motion encoder to learn semantic visual features, and a distillation phase, which makes the motion-visual fusion model jointly mimics the more stronger visual-only model. The results are impressive, supported by a series of ablation study. 

There is no major flaws in the paper, except a minor confusion on the argument on the fact that the pre-training of the model forces the motion encoder learning semantic features.

STRENGTHS:
The paper is very well written and easy to follow. The figure showcases the method very well.

The authors made a keen observation in using a separate modality (head motion) jointly with egocentric frames to efficiently perform action recognition task. The proposed architecture is novel and yields impressive results.

WEAKNESSES:
No major weaknesses.

QUESTIONS:
During the pre-training phase, L1 would train the fusion network as well as the IMU encoder, while NCE loss would mix and match samples to train both the IMU predictor and IMU encoder. How is the schedule of training the IMU predictor \hat{z} match with the optimization of the IMU encoder? How to prevent IMU encoder from being affected by under-trained IMU predictor?

LIMITATIONS:
I am not super convinced on the argument that the IMU encoder learns scene motion cues (line 318). The action confusion (between close/open and put/take) is a good example, but the two action pairs seem to be more relevant to time sequence rather than scene motion.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article presents an innovative approach called EgoDistill for egocentric video understanding. The authors highlight the computational challenges faced by existing models in this field and propose EgoDistill as a solution to overcome these limitations. EgoDistill aims to distill crucial information from egocentric video frames and IMU readings, resulting in a more efficient representation. The article provides detailed technical explanations of EgoDistill, illustrating how it effectively combines semantics from video frames with IMU readings. Additionally, the authors introduce a novel self-supervised training strategy specifically designed for IMU feature learning. To validate the effectiveness of EgoDistill, the authors conduct experiments and present compelling results. Their findings demonstrate that EgoDistill surpasses state-of-the-art efficient video understanding techniques when evaluated on the Ego4D and EPIC-Kitchens datasets. Overall, this article contributes significant advancements to the field of egocentric video understanding by introducing EgoDistill and showcasing its superior performance over existing methods.\n\nSTRENGTHS:\n1. EgoDistill offers several advantages over equivalent video models, including significantly reduced computational requirements in terms of GFLOPs, parameters, and inference time. This enhanced efficiency makes it a more practical option for real-world applications. Furthermore, the authors provide experimental evidence illustrating that EgoDistill surpasses state-of-the-art efficient video understanding methods when evaluated on the Ego4D and EPIC-Kitchens datasets.\n2. The authors have devised a concise and effective self-supervised IMU feature learning architecture. This architecture enables the extraction of valuable camera motion features that are specifically associated with visual appearance changes. As a result, EgoDistill demonstrates improved performance in capturing and understanding these subtle visual cues, leading to enhanced overall performance.\n\nWEAKNESSES:\n1. The main text contains some basic errors, such as confusion between the order of figures and tables, as well as incorrect citations. It is crucial to carefully review and correct these issues.\n2. Undoubtedly, EgoDistill exhibits commendable performance in terms of generalization and versatility. However, the explanation provided in the paper for why EgoDistill performs better than STTS when both methods incorporate IMU features (as mentioned in the \"Effect of fusing IMU data\" subsection) is too superficial and lacks a profound theoretical explanation for EgoDistill. Additionally, there is limited discussion on how the balance between different components of the model is controlled. It would be beneficial to address these points and provide more insights into the underlying mechanisms and strategies employed by EgoDistill.\n3. EgoDistill relies on head motion data to capture camera pose changes. It is important to explore and discuss the limitations and potential mitigation strategies for scenarios where reliable head motion data may not be accurate.\n4. The authors have not thoroughly discussed any potential shortcomings or limitations of EgoDistill. It would be valuable to identify specific scenarios or use cases where EgoDistill might not be as effective or efficient. Understanding these limitations would provide a more comprehensive understanding of the applicability and scope of EgoDistill and could open avenues for future research to address these areas of improvement.\n\nQUESTIONS:\nSee Weaknesses\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper for on Ego-centric video understanding, a promising and popular topic recently.
Different from previous works that focus on improve heavy video encoder, this work try to cut the computation cost down and focus on lightweight temporal reasoning.

Please reply the questions and weakness point-by-point.

STRENGTHS:
- The motivation is quite good and make sense. I like it. Actually ego-centric video have fixed background in general and learn the motion pattern based on the first frame is more easily than third-view video. Considering this area is  new and not well studied, I think it's valuable.
- The method is also simple but works.
- The presentation is good and I can understand it very well.
- The computation cost is cut a lot and provide possibly for small and mobile device.

WEAKNESSES:
- The method section especially the IMU predictor is not solid. Actually, the idea of bring GRU for temporal modeling is quite a old technique. There also exist a lot of works that predict the temporal sequence on feature level like MemDPC two years ago.
- Although the performance is not a very important measurement to me, the compared method is somewhat old. For example, Line243-248.

QUESTIONS:
- Why the first row of Fig.6 sometimes seems not well matched with the action, can you give detail explanation about how to get this figure and why always no changed? The head motion is from AR glasses or?
-  How to define ambiguous categories  in Ego4D?

LIMITATIONS:
- The performance is hard to compare with video encoder based method.
- The proposed work also suitable for ego-centric video.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
The work studies efficient video recognition via distilling IMU signals. Instead of heavily processing each frame, the authors propose to leverage very sparse frames (1-4) for visual features and IMU data. It leads to much more efficient inference cost. The proposed method has shown improvement over other multi-modality fusion techniques like AdaFuse, etc. They also show IMU trajectory provides complementary information to sparse frames.

STRENGTHS:
+ The idea to leverage IMU signals for efficiency is novel. And the empirical results support the idea. The improvement with IMU is consistent across different architectures. 
+ The two stage training (first aligns IMU features then fusing them) is shown effective over alternative multi-modality fusion baselines. 
+ The claims are well supported by ablation studies like two-stage training, loss term, usage of modality, etc.

WEAKNESSES:
1. Since IMU only represents head motions, it is sensible to disambiguate activity relevant to scene motion (open / close, put /take), but it may not contain enough signals about finer grained activity temporally or spatially. But it is the problem of IMU signal, not the paper’s. 
2. I wonder how much does the motion _trajectory_ help, compared to the overall displacement between the end frame and the starting frame, i.e X_t - X_0. More specifically, in table 2, how does EgoDistill with less IMU steps work and how does feeding start frame __and__ end frame works?

QUESTIONS:
See weakness above

LIMITATIONS:
Limitation is discussed
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07