SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shuhan Tan $^{1}$ , Tushar Nagarajan $^{2}$ , Kristen Grauman $^{1,2}$ $^{1}$ University of Texas at Austin,  $^{2}$ FAIR, Meta
Recent advances in egocentric video understanding models are promising, but their heavy computational expense is a barrier for many real-world applications. To address this challenge, we propose EgoDistill, a distillation-based approach that learns to reconstruct heavy egocentric video clip features by combining the semantics from a sparse set of video frames with the head motion from lightweight IMU readings. We further devise a novel self-supervised training strategy for IMU feature learning. Our method leads to significant improvements in efficiency, requiring  $200 \times$  fewer GFLOPs than equivalent video models. We demonstrate its effectiveness on the Ego4D and EPIC-Kitchens datasets, where our method outperforms state-of-the-art efficient video understanding methods. Project page: https://vision.cs.utexas.edu/projects/egodistill/
Recent advances in augmented and virtual reality (AR/VR) technology have the potential to change the way people interact with the digital world, much like the smartphone did in the previous decade. A fundamental requirement for AR/VR systems is the ability to recognize user behavior from egocentric video captured from a head-mounted camera. Towards this goal, several egocentric video datasets have been proposed in recent years, spurring increasing attention of the research community [26, 56, 11].
Current egocentric video understanding models use powerful clip-based video backbones that operate on video clips of a few seconds at a time [18, 54, 25, 16, 12, 55, 43, 44]. Despite encouraging performance, these models typically process densely sampled frames with temporally-aware operations, making them computationally heavy. This makes them impractical for AR/VR devices with constrained resources, or for real-time video applications that require low latency. How to efficiently perform egocentric video understanding is therefore an important, yet unsolved problem.
To address this issue, we take inspiration from how animals perceive the world with ego-motion. Neuroscience research has found that during active movement, the animal visual cortex receives and encodes head motion signals from the motor cortex for visual processing [27, 52, 53], highlighting its role in the efficient understanding of the animal's visual stream. Inspired by this phenomenon, we explore the relationship between human head motion and ego-video for efficient video understanding.
In practice, we consider head motion signals captured by the inertial measurement unit (IMU) of a head-mounted camera. IMU measures motion from an accelerometer and gyroscope and is widely available on popular wearable devices. Prior work leverages IMU as an extra modality for human action recognition [68, 69, 13] (e.g., jumping, walking), or as geometric cues for visual-inertial odometry [7, 20, 71]. In contrast, we propose to drive efficient video understanding by drawing on IMU as a substitute for dense video frame observations. The intuition is as follows. A video clip contains two things: semantic content (appearance of objects, places, people) and dynamics (how the scene and the camera move). While densely sampled frames are sure to capture both of the above—as done by current clip models [54, 16, 17]—we hypothesize they are sometimes overkill. For a short
Figure 1: Illustration of EgoDistill. Given a single video frame and camera motion from IMU, EgoDistill learns to reconstruct the more expensive dense video clip feature. With its lightweight input, EgoDistill significantly improves efficiency.
video clip, much of the semantic content is intelligible from even a single frame; meanwhile, the head motion provides a good portion of the dynamics, implicitly revealing how the visual appearance changes across neighboring frames.
Building on this insight, we introduce EgoDistill, an approach that learns to reconstruct dense egocentric video clip features using temporally sparse visual observations (as few as one RGB frame) together with the head motion from IMU. Specifically, EgoDistill employs a new form of knowledge distillation from video models. During training, we train a lightweight model that takes sparsely sampled image(s) and IMU to approximate video features extracted by a powerful but expensive video model. We further improve the model with a novel IMU-guided self-supervised training stage. During inference, we directly utilize the lightweight model for egocentric video recognition, leading to much higher efficiency. Our model is flexible to the target heavy video feature, as we demonstrate with multiple current leading egocentric video models [54, 16, 18, 17]. See Figure 1.
Importantly, EgoDistill offers a major upgrade in efficiency. Low-dimensional IMU and a few frames are much more efficient to process than a dense stack of frames. In practice, EgoDistill uses  $200 \times$  fewer GFLOPs than the original video model. Furthermore, our approach is economical — the GoPro IMU sensor in our experiments (BOSCH BMI260) costs only $3 — and it is practical — most AR/VR devices and phones have in-built IMU sensors, making our method widely adoptable.
We experiment on the two largest egocentric action recognition datasets: Ego4D [26] and EPIC-Kitchens-100 [11]. We show that IMU coupled with an image offers better cross-modality knowledge distillation performance than images alone or images with audio. For a typical 50-minute egocentric video, EgoDistill reduces model inference time from 25 minutes to 36 seconds. Moreover, with only 1-4 frames, our lightweight distillation model achieves a better accuracy-efficiency trade-off than state-of-the-art models for adaptively sampling video content [50, 65]. Notably, we surpass the accuracy of even these fast approaches by a large margin while requiring  $4 - 8\times$  less computation.
IMU for activity recognition. Recent work explores using the IMU sensor on mobile devices for human activity recognition of actions like walking, jumping, or sitting [47, 59, 60, 3, 61]. Normally, these models use IMU sensors mounted on human body joints [60, 9, 42], waist-mounted [41] or in-pocket smartphones [33]. See [64] for a survey. Abundant work in video recognition explores ways to learn from RGB coupled with other modalities—audio [1, 22, 38], optical flow [57, 58, 19] or both [51, 36, 32]—but much fewer use IMU [48], and unlike our work, they focus on third-person video [68, 69, 13] and do not target model efficiency. Our idea is for IMU to help reconstruct expensive video features, rather than simply fuse IMU with RGB for multi-modal recognition.
IMU for odometry. Inertial odometry aims to estimate the position and orientation of the camerawearer with readings from the IMU sensor. Traditionally, methods rely on IMU double integration [4] or enhancements thereof [40, 5, 35]. Recent data-driven methods automatically learn to perform inertial odometry with supervised [30, 70] or self-supervised learning [7], or combine IMU and
visual input for more robust estimates with visual-inertial odometry [20, 71]. While IMU can convey geometric ego-motion to our learned model, our goal is to produce efficient egocentric video features rather than to output odometry.
Visual feature learning with IMU. IMU is also used to learn better vision features [34, 14, 15, 63], e.g., to encourage image features that are equivariant with ego-motion [34], to predict an IMU-captured body part (leg, hand) [14, 15], or to predict video-IMU correspondence [63], for applications like action recognition [15, 63] and scene understanding [14, 34]. While these results reinforce that IMU can inject embodied motion into visual features, our idea to use head motion to infer pretrained video features for speedy video understanding is distinct.
Efficient video recognition. Being crucial for mobile applications, efficient video recognition has received increasing attention in recent years. Several studies focus on designing lightweight architectures [30, 17, 37, 72, 62] by reducing 3D CNN operations across densely-sampled frames. In contrast, our model focuses on inputs with sparsely-sampled frames. As we show in experiments, our method is compatible with different video architectures.
Another line of research achieves efficiency by adaptively selecting video content to process. Some reduce temporal redundancy by adaptively selecting which video clip [39], frames [49, 24], and/or feature channel [50] to process and which to skip, while others reduce spatial redundancy, selecting for each frame a smaller but important region to process [66, 67]. Other work dynamically selects tokens in video transformers among both the spatial and temporal dimensions [65]. Our idea is complementary: rather than dynamically subsample the available video content, we show how to infer "full" video features for every clip using static image(s) and motion data. Our results outperform state-of-the-art sampling models (cf. Sec. 4). In addition, we focus on egocentric video, where head motion is particularly meaningful for inferring unobserved visual content. To our knowledge, ours is the first technique specifically aimed at accelerating egocentric video processing.
Multimodal distillation. Knowledge distillation aims to transfer knowledge learned by an expensive model to a lightweight model [31]. Recent work explores multimodal distillation, e.g., transferring from a RGB model to a flow or depth model [23, 28], from a 3D model to a 2D model [45], or from a visual model to audio model [2, 21]. More related to our work, the ListenToLook model [22] incorporates both clip subsampling and video-to-audio distillation for fast activity recognition in third-person video. While we share a similar motivation, our contribution is distinct: we explore the relationship between the camera-wearer's head motion and RGB signals for egocentric video, and we introduce a novel IMU-based pretraining strategy. Our experiments show EgoDistill's advantage over ListenToLook in terms of the speed-accuracy tradeoff on egocentric video datasets. Furthermore, whereas [22] attempts only a single video model  $(\mathrm{R}(2 + 1)\mathrm{D})$ , we show our idea generalizes well to multiple video models, meaning it can be dropped in to benefit multiple popular frameworks.
We introduce EgoDistill, which uses sparsely-sampled frames and head motion from IMU to approximate the features of heavy video models for efficient egocentric video understanding. We first introduce the egocentric action recognition task (Sec. 3.1). Then, we introduce our pipeline (Sec. 3.2), our distillation model and training objective (Sec. 3.3), and our self-supervised IMU feature learning (Sec. 3.4). Figure 2 overviews our approach.
3.1 Egocentric action recognition
Given a fixed-length video clip  $\mathcal{V} \in \mathbb{R}^{T \times H \times W \times 3}$  consisting of  $T$  RGB frames of size  $H \times W$  and a set of  $C$  action classes, the task of action recognition is to output a score for each action class, representing its likelihood. Typically, this is done with a powerful but expensive video model  $\Omega$ , that directly operates on all the available frames to output the  $C$  class logits  $\Omega(\mathcal{V}) \in \mathbb{R}^C$ .  $\Omega$  is trained with standard classification loss:
$$
\mathcal {L} _ {\mathrm {A C T}} = \sum_ {\mathcal {V} _ {i}} \mathcal {L} _ {\mathrm {C E}} \left(c _ {i}, \sigma \left(\Omega \left(\mathcal {V} _ {i}\right)\right)\right), \tag {1}
$$
where  $\mathcal{V}_i$  is the  $i$ -th video clip in the dataset,  $c_i$  is the corresponding ground-truth action label,  $\sigma$  is the softmax function, and  $\mathcal{L}_{\mathrm{CE}}$  is cross-entropy loss. Popular video recognition models use clips that are typically  $\sim 2$  seconds long [54, 16, 18]. For longer videos, scores are averaged across all clips it contains to infer the video action label.
Figure 2: EgoDistill architecture. Left: Self-supervised IMU feature learning. Given start and end frames of a clip, we train the IMU encoder to anticipate visual changes. Right: Video feature distillation with IMU. Given image frame(s) and IMU, along with our pre-trained IMU encoder, our method trains a lightweight model with knowledge distillation to reconstruct the features from a heavier video model. When the input includes more than one image frame, the image encoder aggregates frame features temporally with a GRU.
3.2 Efficient video inference with head motion
Processing the video clip  $\mathcal{V}$  for action recognition is computationally intensive; however, the computation cost can be modulated depending on how frames from the clip are used. On the one hand, clip-based models [18, 54, 16, 17] process most (or all) frames in a video clip  $\mathcal{V}$  to achieve strong recognition performance, but come at a high computational cost. On the other hand, frame-level models [49, 24, 51, 67] only process one (or a small number) of frames from  $\mathcal{V}$  and are more efficient, but suffer a drop in performance as a result. Our goal is to train a frame-based model that can approximate heavy clip-based model performance while maintaining high efficiency.
For this, we turn to head motion captured by IMU. Along with RGB frames, each video clip is paired with IMU measurements  $\mathcal{M}$  that record the camera (head) motion during the video. Specifically, the IMU readings are composed of 6-dimensional accelerometer and gyroscope measurements in the  $xyz$  axes, which encode strong temporal motion information about camera pose changes (both translation and rotation) across frames.
For short video clips, a set of sparsely sampled frames  $\mathcal{I}$  often already captures most appearance information. Complementary to this, the IMU readings capture camera motion information (see below for discussion on scene motion). Moreover, IMU is very efficient to process due to its low dimensionality. By processing inputs from these two sources with a lightweight frame-based model, we can infer the semantic and dynamic features of a heavier clip-based video model.
Given  $\mathcal{I}$  and  $\mathcal{M}$ , we train an efficient lightweight model  $\Phi$  to approximate the output of video model  $\Omega$ . Specifically, we train our EgoDistill model  $\Phi$  that achieves
$$
\Phi (\mathcal {I}, \mathcal {M}) \approx \Omega (\mathcal {V}). \tag {2}
$$
Such a lightweight model will be able to approximate the result of the heavy video model, while being much more efficient. Our approach is agnostic to the specific video model  $\Omega$ ; in experiments, we demonstrate its versatility for MotionFormer [54], MViT [16], SlowFast [18] and X3D [17].
In practice, we uniformly sample  $N$  frames from  $\mathcal{V}$  to obtain  $\mathcal{I}$ . We can achieve a trade-off between efficiency and performance by changing the number of frames  $N$ . In our experiments we use very low values of  $N$  (1 to 4 frames). In the next section, we discuss how we train  $\Phi$ .
3.3 Video feature distillation with IMU
We address Equation 2 via knowledge distillation [31], where we transfer knowledge learned by the expensive teacher model  $\Omega$  to a lightweight student model  $\Phi$ . Next we present the design of  $\Phi$  and the training objectives, followed by our self-supervised IMU feature pretraining stage in Sec. 3.4.
<sup>1</sup>Other frame sampling heuristics (e.g., selecting from the start or center of the video) performed equivalently or worse than uniform sampling.
We design  $\Phi$  to be a two-stream model. For a video clip and associated IMU signal  $(\mathcal{I},\mathcal{M})$ , we extract image features  $\mathbf{z}_{\mathcal{I}} = f_{\mathcal{I}}(\mathcal{I})$  and IMU features  $\mathbf{z}_{\mathcal{M}} = f_{\mathcal{M}}(\mathcal{M})$  using lightweight feature encoders  $f_{\mathcal{I}},f_{\mathcal{M}}$  respectively. Then, we fuse  $\mathbf{z}_{\mathcal{I}}$  and  $\mathbf{z}_{\mathcal{M}}$  with a fusion network  $\Pi$  to obtain the fused VisIMU feature  $\mathbf{z}_{\phi} = \Pi (\mathbf{z}_{\mathcal{I}},\mathbf{z}_{\mathcal{M}})$ . Finally, a fully-connected layer uses the fused feature to predict class logits  $\Phi (\mathcal{I},\mathcal{M})\in \mathbb{R}^{C}$ . The fused feature  $\mathbf{z}_{\phi}$  contains semantic information from the image frame coupled with complementary motion information from IMU, allowing us to accurately reconstruct the video clip feature. See Figure 2.
We train  $\Phi$  with a combination of three losses, as follows. First, we train  $\Phi$  to approximate the original video feature  $\mathbf{z}_{\nu}$  from the video model  $\Omega$ :
$$
\mathcal {L} _ {1} = \sum_ {\left(\mathbf {z} \nu_ {i}, \mathbf {z} _ {\phi_ {i}}\right)} \left\| \mathbf {z} _ {\mathcal {V} _ {i}} - \mathbf {z} _ {\phi_ {i}} \right\| _ {1}. \tag {3}
$$
This cross-modal loss encourages the fused feature  $\mathbf{z}_{\phi}$  to match the video feature, i.e., the combined features from the different modalities should match in the feature space.
Training with  $\mathcal{L}_1$  alone does not fully capture the classification output of  $\Omega$ . Therefore, we also train  $\Phi$  with a knowledge distillation loss:
$$
\mathcal {L} _ {\mathrm {K D}} = \sum_ {\left(\mathcal {V} _ {i}, \mathcal {I} _ {i}, \mathcal {M} _ {i}\right)} \mathcal {D} _ {\mathrm {K L}} \left(\sigma \left(\Omega \left(\mathcal {V} _ {i}\right) / \tau\right), \sigma \left(\Phi \left(\mathcal {I} _ {i}, \mathcal {M} _ {i}\right) / \tau\right)\right), \tag {4}
$$
where  $(\mathcal{V}_i,\mathcal{I}_i,\mathcal{M}_i)$  represents the  $i$ -th clip in the dataset,  $\mathcal{D}_{\mathrm{KL}}$  measures KL-divergence between the class logits from the teacher model  $\Omega$  and student model  $\Phi$ , and  $\tau$  is a temperature parameter. Intuitively,  $\mathcal{L}_{\mathrm{KD}}$  casts the output of the video teacher model as a soft target for training the student model. In this way, the student model learns to better generalize by mimicking the output distribution of the heavy video model.
Finally, to further encourage the features to preserve elements useful for activity understanding, we also compute an action classification loss:
$$
\mathcal {L} _ {\mathrm {G T}} = \sum_ {\left(\mathcal {I} _ {i}, \mathcal {M} _ {i}\right)} \mathcal {L} _ {\mathrm {C E}} \left(c _ {i}, \sigma \left(\Phi \left(\mathcal {I} _ {i}, \mathcal {M} _ {i}\right)\right)\right), \tag {5}
$$
where  $c_{i}$  is the ground-truth action label, following Equation 1. The final training loss is a combination of these three loss functions:
$$
\mathcal {L} = \alpha \mathcal {L} _ {\mathrm {K D}} + (1 - \alpha) \mathcal {L} _ {\mathrm {G T}} + \beta \mathcal {L} _ {1}, \tag {6}
$$
where  $\alpha$  controls the balance between knowledge distillation and activity training [31], and  $\beta$  controls the weight for feature space matching.
Critically, processing a few image frame(s) and the low-dimensional IMU readings is substantially faster than processing the entire video. Once trained, our model approximates the behavior of the source video model for recognition tasks, with the key benefit of efficient egocentric recognition.
What kind of motion does our model preserve? Video motion decomposes into scene motion (e.g., how the objects and the camera wearer's hands are moving on their own), and camera motion (i.e., how the camera wearer is moving their head). By itself, IMU would directly account only for camera motion, not scene motion. However, by learning to map from the RGB frame and IMU to the full video feature, we are able to encode predictable scene motions tied to scene content, e.g., how does hand and object movement in subsequent frames relate to the camera wearer's head motion. Moreover, our model is applied to relatively short clips (1-2 seconds) in sequence, which means the appearance content is regularly refreshed as we slide down to process the longer video.
3.4 Self-supervised IMU feature learning
The success of EgoDistill depends on how well the IMU feature encoder  $f_{\mathcal{M}}$  extracts useful camera motion information and associates it with the visual appearance change in the video clip. In this way EgoDistill can learn to anticipate unseen visual changes in the video with  $\mathcal{I}$  and  $\mathcal{M}$ . We design a self-supervised pretraining task to initialize the weights of  $f_{\mathcal{M}}$  to achieve this.
Specifically, for each clip  $\mathcal{V}$ , we obtain its first and last frames  $(\mathcal{I}^0,\mathcal{I}^T)$  as well as the IMU  $\mathcal{M}$ . We first extract visual features  $\mathbf{z}_{\mathcal{I}}^{0},\mathbf{z}_{\mathcal{I}}^{T}$  and IMU feature  $\mathbf{z}_{\mathcal{M}}$  with feature extractors  $f_{\mathcal{I}}$  and  $f_{\mathcal{M}}$
mentioned above. Then, we train a feature predictor  $h$  to predict the IMU feature  $\hat{\mathbf{z}}_{\mathcal{M}} = h(\mathbf{z}_{\mathcal{I}}^{0},\mathbf{z}_{\mathcal{I}}^{T})$ . By connecting  $\hat{\mathbf{z}}_{\mathcal{M}}$  — a function of image features only—with  $\mathbf{z}_{\mathcal{M}}$ , we encourage  $f_{\mathcal{M}}$  to extract useful camera motion features specifically associated with the visual appearance changes. Note that those appearance changes may include scene motion. Therefore, we include an  $\mathcal{L}_1$  loss to train  $f_{\mathcal{M}}$ , which encourages  $f_{\mathcal{M}}$  to extract motion features accounting for scene motion in the full video.
We train  $f_{\mathcal{M}},h,$  and the fusion network  $\Pi$  using  $\mathcal{L}_1$  and NCE loss [29]:  $\mathcal{L}_{\mathrm{pretrain}} = \mathcal{L}_{\mathrm{NCE}} + \mathcal{L}_1$  , where
$$
\mathcal {L} _ {\mathrm {N C E}} = \sum_ {i} - \log \frac {\operatorname {s i m} \left(\hat {\mathbf {z}} _ {\mathcal {M} _ {i}} , \mathbf {z} _ {\mathcal {M} _ {i}}\right)}{\sum_ {j} \operatorname {s i m} \left(\hat {\mathbf {z}} _ {\mathcal {M} _ {i}} , \mathbf {z} _ {\mathcal {M} _ {j}}\right)}. \tag {7}
$$
We sample negative examples  $\mathbf{z}_{\mathcal{M}_j}$  from other instances in the same mini-batch for  $j \neq i$ , and  $\sin(q, k) = \exp \left( \frac{q \cdot k}{|q||k|} \frac{1}{\tau'} \right)$  with temperature  $\tau' = 0.1^2$ .
To summarize, prior to the main training stage of Equation 6, we pretrain the IMU feature extractor  $f_{\mathcal{M}}$  and fusion network II. As we will show below, both pretraining losses result in IMU features that are consistent with visual changes and lead to better finetuning performance.
We evaluate our approach for resource-efficient action recognition.
Datasets. We experiment on two large-scale egocentric action recognition datasets. (1) Ego4D [26] contains 3,670 hours of egocentric videos of people performing diverse tasks (from cooking to farming) across the globe. As action recognition is not part of the original Ego4D benchmark, we construct this task with annotations from the Hands+Objects temporal localization benchmark [26] (see Supp. for details). We include clips with paired IMU and audio $^3$ , and consider classes with at least 2 labeled instances. This results in a 94-class action recognition dataset with 8.5k training videos and 3.6k evaluation videos. (2) EPIC-Kitchens [11] contains 100 hours of egocentric videos capturing daily activities in kitchen environments. We use annotations from the action recognition benchmark. Similar to Ego4D, we select videos that have paired IMU and audio data, and split the resulting data by camera-wearer. This results in a 62-class action dataset with 29k training videos and 6.2k evaluation videos. For both datasets, we use "verb" labels as the target for action recognition as they are well aligned to activity motions.
Evaluation metrics. To measure action recognition performance, we report the per-video top-1 accuracy on the validation set. We densely sample clips from each video and average their predictions to compute accuracy. To benchmark efficiency, we measure computational cost with FLOPs (floating-point operations) during inference.