SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Implementation details. In our main experiments, we use MotionFormer [54] as the video teacher model  $\Omega$  due to its strong performance for egocentric video. For EPIC-Kitchens, we use the authors' provided checkpoint. For Ego4D, we finetune the above model for 50 epochs with  $1e^{-4}$  learning rate and 64 batch size on the training set. We use 16-frame input with sample rate 4. For the student model  $\Phi$ , we use a ResNet-18 as the image backbone  $f_{\mathcal{I}}$  and a 1D Dilated CNN [6] for the IMU backbone  $f_{\mathcal{M}}$ . The feature fusion module  $\Pi$  uses a concatenation operation following a two-layer fully-connected layer with hidden dimension 1024. For each video clip, the input image(s) is resized to  $224 \times 224$ , and the IMU is a  $422 \times 6$  matrix (around 2 seconds with  $198\mathrm{Hz}$  frequency), representing the accelerometer and gyroscope readings along the xyz axes. For the image input, we uniformly sample  $N$  frames from the video clip. If  $N > 1$ , we use  $f_{\mathcal{I}}$  to sequentially generate features for each frame and aggregate them with a GRU module [10]. For both datasets, we first pretrain the model with the self-supervised objective (Sec. 3.4) for 50 epochs with AdamW [46] using batch size 64 and learning rate  $1e^{-4}$ . Then, we finetune all the models with the same setting (Equation 6). We set  $\alpha = 0.95$  and  $\beta = 1.0$  based on validation data. For Ego4D, we set  $\tau = 10.0$  and train the model for 150 epochs. For EPIC-Kitchens, we set  $\tau = 1.0$  and train for 50 epochs.
2We keep the ImageNet-pretrained  $f_{\mathcal{I}}$  model frozen, as finetuning it leads to mode collapse.
3We require audio to compare with the audio-based baseline [22].
Figure 3: Accuracy vs. efficiency for action recognition on Ego4D (left) and EPIC-Kitchens (right). EgoDistill outperforms state-of-the-art efficient video recognition methods that adaptively sample video content, while using  $4 \times$  to  $8 \times$  fewer GFLOPs.
We compare with the following methods: (1) AdaFuse [50] trains a lightweight policy network to adaptively compute (or skip) feature map channels for each frame during inference. We use the  $\mathrm{AdaFuse}_{\mathrm{R50}}^{\mathrm{TSN}}$  model with the provided hyper-parameters. (2) STTS [65] trains a module to rank spatio-temporal tokens from videos in a transformer-based model, and selects only the top-K tokens to speed up inference. We use a MViT-B16 backbone [17]. (3) ListenToLook [22] uses the audio-based distillation module from [22] following the same audio processing and model architecture.
These methods represent recent advances in efficient video recognition models. AdaFuse represents state-of-the-art approaches that achieve efficiency by reducing temporal redundancy in CNN models. STTS is one of the most recent approaches that efficiently reduces both spatial and temporal redundancy in ViT models, which achieves the state-of-the-art on Kinetics-400 [8]. ListenToLook also relies on distillation, but using audio rather than head motion. For each model we generate multiple versions with different computation budgets to plot accuracy vs. GFLOPs. We train all AdaFuse and STTS models with 4 input frames to align with the maximum frames used by our model. For AdaFuse, we use the only provided hyper-parameter in the paper. For STTS, we use three provided variants:  $\mathrm{T}_{0.5}^{0}-\mathrm{S}_{0.7}^{4}$ ,  $\mathrm{T}_{0.8}^{0}-\mathrm{S}_{0.9}^{4}$  and the full model without token selection. For ListenToLook we adopt the same efficiency-accuracy trade-off as our method, i.e., varying the number of input frames.
We also test variants of our method: (1) VisOnly-Distill is our model without the IMU branch and fusion layer, but trained with the same loss function. This model reveals the importance of IMU for distillation. (2) VisIMU is our model trained with only  $\mathcal{L}_{\mathrm{GT}}$  in Equation 5. It shows the effectiveness of distillation from the video model compared with directly training the features with action labels. (3) VisOnly is an image-only model trained with  $\mathcal{L}_{\mathrm{GT}}$ , which serves as the baseline.
Importance of IMU-guided distillation. Figure 3 shows the accuracy vs. efficiency curves. Methods towards the top-left of the plot represent those with both high accuracy and efficiency. Our method achieves good accuracy with low computational cost. Specifically, on EPIC-Kitchens, when  $N = 1$ , EgoDistill improves over VisOnly-Distill by  $8.4\%$  with only a small increase in computation. This shows the effectiveness of IMU for reconstructing egocentric video features. Compared to VisIMU, EgoDistill improves by  $9.9\%$ , showing the effectiveness of knowledge distillation from the video model. Importantly, this reveals that EgoDistill does not simply benefit from the extra IMU context; our idea to approximate video features is necessary for best results. We see similar results on Ego4D.
Comparison with the state of the art. Figure 3 also shows that EgoDistill achieves better accuracy with less computation than existing efficient video recognition models AdaFuse [50], STTS [65], and ListenToLook [22]. With  $N = 4$  frames, EgoDistill surpasses STTS by  $7.4\%$  and AdaFuse by  $4.2\%$  on EPIC-Kitchens, with  $2\times$  fewer GFLOPs, and surpasses both methods by  $2.1\%$  on Ego4D. When we use fewer frames, EgoDistill can still outperform STTS and AdaFuse using  $4\times$  to  $8\times$  fewer
4Modifying hyper-parameters to control the accuracy-efficiency trade-off results in unstable training and unreliable performance.
\( \mathcal{L}_{\mathrm{KD}} \)	\( \mathcal{L}_{1} \)	\( \mathcal{L}_{\mathrm{{GT}}} \)	\( \mathcal{L}_{1} \)-pretrain	\( \mathcal{L}_{\mathrm{{NCE}}} \)-pretrain	Ego4D	EPIC-Kitchens
✓	34.15	35.04
✓	✓	✓	✓	35.51	39.33
✓	✓	✓	✓	37.71	42.20
✓	✓	✓	✓	37.46	43.17
✓	✓	✓	36.99	41.21
✓	✓	✓	✓	37.26	42.30
✓	✓	✓	✓	37.49	43.51
✓	✓	✓	✓	✓	37.95	44.95
Table 1: Ablation study of EgoDistill's model components. Accuracy with  $N = 1$  .
Source Model	Ego4D	EPIC-Kitchens
Video	EgoDistill	VisOnly-D	Video	EgoDistill	VisOnly-D
MFormer [54]	46.38	37.95	34.32	77.28	44.95	37.20
MViT [16]	40.32	36.46	33.40	53.38	36.90	31.22
SlowFast [18]	40.52	33.29	33.04	58.34	39.42	33.47
X3D [17]	37.56	33.57	32.90	52.28	36.34	31.71
Table 2: Versatility to model architectures. EgoDistill outperforms the baselines for multiple common architectures, showing the generality of our idea. "Video" refers to the more expensive source model, essentially the upper bound for accuracy. We show the model accuracy under  $N = 1$ .
GFLOPs. In addition, EgoDistill surpasses ListenToLook by  $7.4\%$  and  $2.9\%$  on EPIC-Kitchens and Ego4D respectively, which suggests that head motion is more informative than audio for feature reconstruction in egocentric video.
Model component ablations. Table 1 ablates different design choices in our model, setting  $N = 1$  for all experiments. We observe that training EgoDistill without  $\mathcal{L}_1$ ,  $\mathcal{L}_{\mathrm{KD}}$  or  $\mathcal{L}_{\mathrm{GT}}$  deteriorates performance. Specifically, training without  $\mathcal{L}_{\mathrm{KD}}$  leads to the largest performance drop, which indicates that knowledge distillation is an essential component in our approach. Training without  $\mathcal{L}_1$  also degrades performance, which shows the importance of our idea to align features from the different modalities. Further, our self-supervised pretraining stage is very effective at training the IMU extractor to encode useful motion information that is consistent with visual feature change. We find an alternative IMU extractor pretraining strategy of supervised action recognition task is inferior; we outperform pretaining with action recognition by  $3.74\%$  on EPIC-Kitchens and  $0.96\%$  on Ego4D. Finally, we compare with a model that simply does multi-modal recognition with IMU (top row). The strong contrast here indicates the importance of our idea to use IMU to predict video model features, as opposed to simply adding IMU as an additional input modality.
Impact of teacher video model architecture. In our main experiments we use MotionFormer [54] as the teacher video model due to its strong performance on egocentric video tasks. To emphasize the generality of our idea, we show the performance of EgoDistill with other video teacher architectures in Table 2. Similar to the MotionFormer model, we train these models on each of the labeled datasets, and then train our model using the resulting video models as the teacher. As expected, better video teacher models lead to better student model performance. More importantly, we observe consistent improvement by EgoDistill over the VisOnly-Distill baseline on both datasets and with different video teacher models, highlighting our idea's generality and versatility.
Efficiency analysis. To compare the efficiency of different models, aside from GFLOPs, we also compare their inference run-time and number of parameters. For run-time, we record the time spent to infer a single video clip's label with a single A40 GPU, and take the average time over the full validation datasets of Ego4D and EPIC-Kitchens with batch-size of 32. Table 3 shows the results. EgoDistill runs much faster than the other methods. Notably, it reduces the GFLOPs of MotionFormer by nearly  $200\times$ . Furthermore, it runs  $6.5\times$  faster than STTS [65] while achieving  $4.4\%$  higher accuracy on EPIC-Kitchens.
Effect of fusing IMU data. EgoDistill achieves superior performance by combing IMU information when distilling features from video models. Here we explore how helpful it is to combine IMU
GFLOPs	Runtime (ms)	Parameters (M)
Video [54]	369.51	10.70	108.91
AdaFuse [50]	15.20	2.04	38.85
STTS [65]	7.19	1.63	36.63
ListenToLook [22]	3.10	0.43	25.53
EgoDistill	1.91	0.25	20.56
Table 3: Efficiency analysis. Our approach is the most efficient. "Video" refers to the original (full-clip) feature. Lower is better.
Method	Ego4D	EPIC-Kitchens
w/ IMU	w/o IMU	w/ IMU	w/o IMU
STTS [65]	34.93	34.54	38.38	32.24
EgoDistill	37.95	34.32	44.95	37.20
Table 4: Effect of combining IMU with vision as input. In both EgoDistill and STTS, our method is able to combine vision and IMU information more effectively.
Figure 4: Anticipating scene motion with EgoDistill. For each clip, we show the head motion and video frames. Note, only the center frame (red border) is observed by the model. Action classification scores are shown on the right. EgoDistill can successfully anticipate scene motion and disambiguate the action semantics in the input frame. For example, in the top center frame, the image alone cannot reveal if the door is being opened or closed, whereas our feature, learned with head motion, recovers correlations with the scene motion (i.e., hand motion and door motion) to disambiguate "close" from "open". A similar effect for "put" vs. "take" is seen in the second example.
information with the most recent efficient video understanding method, i.e., STTS [65].<sup>5</sup> To this end, for STTS, we add an IMU encoding branch identical to the one used in EgoDistill, and concatenate the IMU feature with video features before the last FC layer. We train both the video and IMU branches simultaneously for the action recognition task. Results in Table 4 show that combining IMU features indeed helps STTS, yet EgoDistill's performance is significantly better than STTS's. These results again indicate that our advantage is not simply access to IMU; rather it is the proposed way we leverage vision and IMU. The reason that EgoDistill is able to outperform STTS+IMU is mainly that EgoDistill learns to jointly distill information from a video feature with image and IMU inputs. This distillation process will encourage the model to extract the interplay between static semantic information in the image and the dynamic information in IMU, as it is trained to reconstruct the video model feature. On the other hand, simple concatenation of the IMU feature to the image feature for action classification will not exploit the relationship between the image and IMU—it lacks the training signal to target the dynamic video clip feature—leading to inferior performance.
Effect of using partial IMU. Is the net displacement of the camera what matters, or does the motion in between give cues? To answer this, we experiment with feeding EgoDistill with partial IMU readings of the full video clip. Specifically, we train and evaluate our model with only the first  $\mathrm{K\%}$  of IMU readings of the full clip and pad the rest with 0. We show the results in Table 5. This ablation study shows that more complete IMU readings yield the best results, while using  $25\%$  and  $50\%$  IMU data achieves the majority of the gain over the VisOnly-D (no IMU) baseline.
Effect of number of input frames. To investigate the effect of using different numbers of input frames, we conduct an ablation study with  $N$ . Specifically, we use three settings:  $N = 1$  (use the center frame),  $N = 2$  (use the start frame and end frame),  $N = 4$  (uniformly sample 4 frames). We compare EgoDistill and VisOnly-D under different settings in Table 6. We observe that only using the overall displacement between the end frame and the starting frame ( $N = 2$ ) leads to better
<sup>5</sup>We tried to augment AdaFuse [50] the same way, but adding IMU to its codebase gave unreliable results.
Figure 5: Retrieving video clips with EgoDistill. Given a query frame (bottom left) and a paired IMU segment (red camera frustums), we retrieve the nearest clip in the video dataset according to EgoDistill and visualize its (unobserved) frames (strip to the right). Compared to VisOnly-Distill, which outputs a single feature for a given input frame (bottom row), EgoDistill outputs a distinct feature by conditioning on IMU, showing its ability to preserve both semantic and motion during reconstruction. For instance, in the top-right example, EgoDistill retains the cabinet interaction semantics in the frame as well as the upward camera-motion in the IMU. Zoom in to view best.
Input IMU	Ego4D	EPIC-Kitchens
100% (EgoDistill)	37.95	44.95
75%	37.88	43.43
50%	37.65	43.69
25%	37.21	42.68
10%	36.60	41.45
0% (VisOnly-D)	34.32	37.20
Input Frame	Ego4D	EPIC-Kitchens
EgoDistill	VisOnly-D	EgoDistill	VisOnly-D
N=1 (center)	37.95	34.32	44.95	37.2
N=2 (first and last)	38.29	34.48	50.07	46.71
N=4 (uniform)	38.46	34.84	52.43	48.96
Table 5: Effect of partial IMU. We observe using partial IMU input already achieves significant gain.
Table 6: Effect of number of input frames. Using more input frames leads to larger improvement of EgoDistill over VisOnly-D baseline.
performance by VisOnly-D than using only one frame ( $N = 1$ ). However, the motion trajectory still helps EgoDistill obtain significantly better performance in either case. This result indicates the information in the motion trajectory captured by IMU. Note that while  $N = 2,4$  leads to better performance, it requires higher computational cost.
What do EgoDistill features capture? To explore this, we pair a single input frame with different IMU clips as inputs to EgoDistill, then retrieve the nearest video clip for each resulting anticipated video feature. We also compare with the VisOnly-Distill result. Figure 5 illustrates this. We see that EgoDistill outputs video features that all involve interaction with the cabinet (right panel), and is able to use different IMU inputs to retrieve different video clips that show consistent camera motion. In contrast, different input IMUs lead to corresponding camera motion in the frames. The result shows that EgoDistill approximates video features that capture both semantic and motion information, whereas VisOnly-Distill only retains the semantic context to retrieve a single clip.
Is there evidence EgoDistill captures scene motion? Figure 4 shows how our features learned with head motion can nonetheless expose certain scene motion cues. EgoDistill improves the accuracy over VisOnly-Distill on ambiguous categories (like close and put) by a large margin (20.3% and 10.4% on EPIC-Kitchens, 8.5% and 3.9% on Ego4D). See caption and Supp. video for more details.
Limitations of EgoDistill. Although EgoDistill is useful for many action categories in egocentric videos, for some other classes it is less effective. We observe that video clips that EgoDistill does not perform well tend to contain little head motion—in which case IMU is redundant to the RGB frame—or drastic head motion that is weakly correlated with the camera wearer's activity and introduces blur to the frame. Additionally, EgoDistill might not be useful when we want to focus on the "noun" part of the action (e.g., differentiating watching TV and using the cell phone), as IMU information does not contain information about the object semantics in the scene.
We present EgoDistill, the first model to explore egocentric video feature approximation for fast recognition. Experiments on action recognition on Ego4D and EPIC-Kitchens demonstrate that our model achieves a good balance between accuracy and efficiency, outperforming state-of-the-art efficient video understanding methods. Our approach has great potential to accelerate video understanding for egocentric videos using a data stream that is already ubiquitous in egocentric cameras. In the future, we plan to investigate how to use head motion for long-term human activity understanding with room context and visual correspondence learning for multi-view videos.
Acknowledgements We thank the UT Austin vision group for helpful discussions. We also thank the anonymous reviewers for their insightful suggestions. UT Austin is supported in part by IFML NSF AI Institute. KG is paid as a research scientist at Meta.
[65] Junke Wang, Xitong Yang, Hengduo Li, Liu Li, Zuxuan Wu, and Yu-Gang Jiang. Efficient video transformers with spatial-temporal token selection. In ECCV, 2022. 2, 3, 7, 8, 9
[66] Yulin Wang, Zhaoxi Chen, Haojun Jiang, Shiji Song, Yizeng Han, and Gao Huang. Adaptive focus for efficient video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2021. 3
[67] Yulin Wang, Yang Yue, Yuanze Lin, Haojun Jiang, Zihang Lai, Victor Kulikov, Nikita Orlov, Humphrey Shi, and Gao Huang. Adafocus v2: End-to-end training of spatial dynamic networks for video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 3, 4
[68] Haoran Wei, Pranav Chopada, and Nasser Kehtarnavaz. C-mhad: Continuous multimodal human action dataset of simultaneous video and inertial sensing. Sensors, 20(10), 2020. 1, 2
[69] Haoran Wei and Nasser Kehtarnavaz. Simultaneous utilization of inertial and video sensing for action detection and recognition in continuous action streams. IEEE Sensors Journal, 20(11):6055-6063, 2020. 1, 2
[70] Hang Yan, Qi Shan, and Yasutaka Furukawa. Ridi: Robust imu double integration. In ECCV, 2018. 2
[71] Mingyu Yang, Yu Chen, and Hun-Seok Kim. Efficient deep visual and inertial odometry with adaptive visual modality selection. ArXiv, abs/2205.06187, 2022. 1, 3
[72] Mohammadreza Zolfaghari, Kamaljeet Singh, and Thomas Brox. ECO: efficient convolutional network for online video understanding. In ECCV, 2018.3