Modeling Fine-Grained Hand-Object Dynamics for Egocentric Video Representation Learning
Authors: Baoqi Pei
Yifei Huang
Jilan Xu
Guo Chen
Yuping He
Lijin Yang
Yali Wang
Weidi Xie
Yu Qiao
Fei Wu
Limin Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Video representation learning
Egocentric video
Action recognition
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: In egocentric video understanding, the motion of hands and objects as well as their interactions play a significant role by nature.
However, existing egocentric video representation learning methods mainly focus on aligning video representation with high-level narrations, overlooking the intricate dynamics between hands and objects.
In this work, we aim to integrate the modeling of fine-grained hand-object dynamics into the video representation learning process.
Since no suitable data is available, we introduce HOD, a novel pipeline employing a hand-object detector and a large language model to generate high-quality narrations with detailed descriptions of hand-object dynamics. 
To learn these fine-grained dynamics, we propose EgoVideo, a model with a new lightweight motion adapter to capture fine-grained hand-object motion information. 
Through our co-training strategy, EgoVideo effectively and efficiently leverages the fine-grained hand-object dynamics in the HOD data. 
Extensive experiments demonstrate that our method achieves state-of-the-art performance across multiple egocentric downstream tasks, including improvements of 6.3% in EK-100 multi-instance retrieval, 5.7% in EK-100 classification, and 16.3% in EGTEA classification in zero-shot settings. Furthermore, our model exhibits robust generalization capabilities in hand-object interaction and robot manipulation tasks.

Published as a conference paper at ICLR 2025
MODELING FINE-GRAINED HAND-OBJECT DYNAMICS FOR EGOCENTRIC VIDEO REPRESENTATION LEARNING
Baoqi Peit.2x Yifei Huang Jilan Xu Guo Chen', Yuping He,, Lijin Yang", Yali Wang5.2 Weidi Xie Yu Qiao², Fei Wu', Limin Wang 'Zhejiang University, Shanghai Artificial Intelligence Laboratory, The University of Tokyo, 'Fudan University, 'Nanjing University, SIAT, 'Shanghai Jiao Tong University, peibaoqi gmail com; hyfeiis. u- tokkookkoccooooo ac. jp
ABSTRACT
In egocentric video understanding, the motion of hands and objects as well as their interactions play a significant role by nature. However, existing egocentric video representation learning methods mainly focus on aligning video representa- tion with high-level narrations, overlooking the intricate dynamics between hands and objects. In this work, we aim integrate the modeling of fine- fine-grained hand- object dynamics into the video representation learning process. Since no suitable data is available, we introduce HOD, a novel pipeline employing a hand-object detector and large language model to generate high-quality narrations with de- tailed descriptions of hand-object dynamics. To learn these fine-grained dynam- ics, we propose Video, a model with a new lightweight motion adapter to capture fine-graine fine-grained hand-object motion information. Through our co-training strategy. EgoVideo effectively and efficiently leverages the fine-grained hand- object dynamics in the HOD data. Extensive experiments demonstrate that our method achieves state-of-thes performance across multiple egocentric down- stream tasks, including improvements of 6.3% EK-100 multi-instance retrieval, 5.7% in EK-100 classification, and 16.3% in EGTEA classification in zero-shot settings. Furthermore, our model exhibits robust generalization capabilities hand-object interaction and robot manipulation tasks. Code and data available at http:///hhhhacccoo////aaaaaaaaaa
INTRODUCTION
Egocentric video understanding has recently garnered increasing attention due to its crucial role in areas such as augmented reality (Pan et al. 2023), embodied AI (Srivastava et al., 2022 Huang et al. 2024b), and personalized assistants (Huang et al., 2018). With the collection of large-scale egocentric video datasets (Damen et al.) 2020: Grauman et al. 2022), researchers hegin to adopt video-language pretraining (Lin et al. (022) based on these annotations to learn egocentric video representations. Since the original annotations tend to be highly template-driven and lack diversity, previous works explore using Large Language Models (LLM) to rephrase the narration (Zhao al., 2023) or introducing new rideo-languae pairs from exocentric datasets (Dou et al. 2024). This scheme has shown success wide range of downstream tasks (Plizzari et al.) 2024).
However, as can be seen from the example in Figure I right, the original annotations in egocentric video datasets are typically highly condensed, describing only overall actions like "C draws on book or "C moves both hands". Since no additional information is provided, previous works like La (Zhao et al.) 2023) can only rephrase the same level of abstraction as the original anno- tations, neglecting a crucial aspect of egocentric videos the fine-grained dynamics of hands and objects. Most egocentric videos contain a large portion of hand-object interactions, which reflects the camera wearer's behavior and intentions. As will be seen, integrating this information in vision-
Equal contribution. Yifei Huang is the corresponding author.

---

Published as a conference paper at ICLR 2025
EX-Ine VER-25 (BAF))
RIC-00 CLS-25
(Tgp1AAe)
BaARO
Original Datast Annotation:
171
22.
15.
#C both
(TQp)AA0)
EpeNT. (.))
Lattl Norriter Aevsstation:
left hand V.cC lifts the right hand
lifts right NNC around
Latiaa Rephraur Arnstatior:
66.7
*C moves both hands *a Both hands are used by CCC both hands, *C Both hends moved by C'
45%
RaACQ WA.1
91.9
EXTERA (Tephinre)
56.5
The left hand moves upward and then to the right, manipulating abject that is initially above and to the right of the left hand and then moves it to a position upward and to left of the right hand, while the right hend moves then
Presiou SOTA
88.3
(MAR)
[lsp--AA))
Figure I: Left: Our Ego model achieves state-of-the-art performance across multiple video benchmarks by learning fine-grained hand-object dynamics from videos. Right: Annotations from different sources: original Ego4D annotation (Grauman al.J 2022). LaViLa (Zhao et al. [023), and our HOD. Our HOD annotations provide a detailed description of hand movements and object manipulation, demonstrating a higher level of detail and context.
language pretraining significantly enhances egocentric video representation learning, resulting in state-of-the-ar performance across various benchmarks.
Firstly, to incorporate hand-object dynamics into vision-language pretaining, it is essential struct data that accurately captures the detailed motion of hands and objects videos. A recent work directly uses the output of off-the-shelf hand-object detectors (Shan et al.] 2020) the ground truth of auxiliary targets in the pretraining (Zhang et al. (203)) However, this approach only models the appearance of hands and objects without considering their dynamics. It also fails learn the semantic connections between hand-object interactions and the original narration. To address this, we introduce HOD, a novel framework for generating descriptions with fine- mme-grained Hand-Ou Abje-Object Dynamics for a given video clip. We begin with using hand-object detectors to obtain bounding boxes of hands and contact objects. Then we design prompts based on these bounding boxes to generate descriptions of the trajectories of the hand and object, as well as their contact states and positions. Finally, using the new prompts and original annotations, we leverage large language model (LLM) to generate semantically rich captions that encompass the motion states of hands and objects. By utilizing high framerate inputs, we ensure the capture of more detailed motions.
Secondly, to efficiently and effectively exploit the fine-grained spatiotemporal information in HOD, we propose Video, a novel ViT-based model with a lightweight motion adapter. Cooperating design and co- training strategy. The backbone with the HOD data, Video employs a video-language alignment, while the adapter branch branch is trained normally to learn fundamental is trained with higher framerate to capture detailed hand-object dynamics. The motion adapter has a separable convolution design, allowing for information aggregation from both adjacent frames temporally and from hands and objects at different locations. This design enables Ego Video to model detailed hand-object dynamics while maintaining low computational costs. This also allows us to scale the model size to 1B to fully unlock its potential to comprehend egocentric videos.
We extensively evaluate Ego Video across multiple pretraining data sources and various egocentric downstream tasks. Experimental results show that our model sets new state-of-the-at on 9 tasks as partially shown in Figure [ same model size in both zero-shot and fine-tuning settings. Further experiments demonstrate that our HOD data is also beneficial for robot manipulation tasks.
Notably, our model also achieves the best performance under the
Our main contributions are as follows: (1) We develop a HOD data pipeline to generate captions that describe fine-grained hand-object dynamics, which are crucial for egocentric video understand- ing; (2) We propose Video, a dual- branch model with a novel lightweight motion adapter and co-training strategy to leverage the HOD data efficiently and effectively; (3) We demonstrate state-of-tte the performance on 12 downstream tasks, and our approach generalizes well to robot manipulation tasks. All code and data will be made publicly available.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Egocentric Video Understanding is receiving increasing research attention. Previous works focus
on diverse tasks such as action recognition (Plizzari et al. 2022 Huang et al. 2222aa). action antic-
ipation (Girdhar& & Grauman 2021), and cross-view understanding (Xue et al. 2022 Huang et al.,
2024a Luo al. 2024). Recent methods begin to work egocentric representation learning (Lin) et al. 2022, Pei et al., 2024) using the large-scale data from Ego4D (Grauman et al. 2022). or re- fining the Ego4D narrations by LLM rephrasing (Zhao & Krähenbühl] 2023). A recent work also searches for additional data from exocentric datasets to improve the pretraining (Dou et al. 2024). However, since the Ego4D narrations are highly abstract, these methods fail to learn one critical aspect of egocentric videos - fine-graines hand-object dynamics. Recently. Helping Hands (Zhang al., 2023) utilizes hand and object coordinates as auxiliary targets during pretraining. However, it only focuses on the spatial information of hands and objects, neglecting their motion dynamics. Additionally, the provided supervision does not integrate the states of hands and objects with the video descriptions, limiting the model's ability to comprehend fine-grained details.
Unlike previous works, we propose the first method to integrate the hand-object dynamics into egocentric representation learning. On the data side, we propose the HOD pipeline, which generates high-quality video-language pairs. The language in these pairs explicitly represents the complex states and motions of hands and objects in the vidcos, enabling the model to learn detailed information about these dynamics. On the model side, we introduce Video, a model equipped with lightweight motion adapter. This adapter is designed to effectively capture the intricate hand and object dynamics provided by the HOD data, enhancing the model's ability to understand and interpret fine-grained dynamics in egocentric videos.
Dynamics)
Video-Language Representation Learning has also attracted researchers after the success of CLIP (Radford et al. 2021), due to the need for generating robust video representations. Several large-scale video-language datasets (Kay et al. 2017 Miech al. 2019 Caba Heilbron al. 2015) further fueled the research in this area. However, generating high-quality video-text pairs remains a challenging task, prompting researchers to develop innovative solutions. LaViLa (Zhao et al.) 2023) leverages Large Language Models (LLMs) to generate dense narrations for videos. Video Recap (Islam et al. 2024) utilizes a curriculum learning training scheme to generate summaries for long videos. EMBED (Dou et al.) (024) and Egolnstructor (Xu et al., (024) use rules retrieval models to add additional training data. However, the previous methods can only pretrain their mod- els the same abstraction level as the original annotation. In contrast, our approach integrates finer-level details into the representation learning process.
Hand-Object Interaction Understanding has long been a key research topic within the field of egocentric vision. recent years, several works have made significant strides modeling estimate 3D hand joints (Brahmbhatt et al., 2020)] Cai et al.] 2018 Yang & Yao. 2019; Yuan et al. 2018
Ohkawa et al. (023) and reconstructing hand-object shape (Cao et al. 2021 Doosti et al. 2020
Hasson et al. 2019 2020 Liu et al., ). EgoHOS (Zhang et al. 2022) provides a labeled dataset with fine- gane-graaedd per-pixel labels of hand and objects and a reliable foundational tool for 2D hand- object segmentation, 100DOH (Shan et al. 2020) introduces large-scale video dataset containing hands and hand-object interactions, providing a rich resource for hand object detector training. In work, we utilize existing hand and object detectors in HOD pipeline to convert information related to hand/object motion and contact details into natural language descriptions. By integrating these detailed descriptions with our Ego model, we can integrate this finer level of detail into the video representation learning process.
3 METHOD
3.1 DATA GENERATION PIPELINE: HOD
The fine-grained dynamics of hands and objects play pivotal role in egocentric video understand- ing (Fathi et al. 201 Iay. To effectively integrate this information in the video-language pretraining process, propose HOD, novel data generation pipeline to transform hand-object dynamics into natural languages. An overview of HOD is illustrated in Figure top. First, we utilize an off-the- shelf hand object detector (Shan al. 2020) generate bounding boxes for hands and objects in
3

---

Published as a conference paper at ICLR 2025
Original eapiion: tokes sriswwy
1100
New captioa:: caccee actturr right morn shew right of Messwhiis
MGI Dettecton
Hend cbject coorchancs
Text Enooder
Prompl
pigure
1xt Inmee
Parameter Weight
LAYLE
Nation
Prpecaon
Concre FC
Barns
VioVideo
Figure 2: Illustration of our HOD pipeline and Ego Video model. In our Hand-Object Dynamics data generation pipeline (top), we first use hand object detector to obtain the spatial coordinates of hands and objects in the clip. then we combine the motion information of hands and objects with the original narrations to generate semantically richer narrations. In our Ego Video model (bottom), the backbone is trained with lower framerate. We design a lightweight motion adapter to learn fine-grained dynamics efficiently with higher framerate inputs.
each frame of the video clips. Next, employ large language model (AI al. 2024) to enrich the original video captions. The model is prompted to generate new narrations that integrate the original captions with hand-object dynamies information, enhancing the semantic richness of the annotations. Below, go into the details of the HOD data generation process.
3.1.1 DATA SELECTION
Before going into the generation process, is essential to select appropriate source data. The basic data component which has been proven useful in egocentric video-language pretraining (Pramanick et al. 2023)
from the 4M subset (Lin al. 2022) of Ego4D (Grauman al. 2022)
Additionally, we curate data from the large-scale HowTol00M dataset (Miech et al.) (2199) since it contains rich hand-object interactions. We specifically choose Now2-Interlink7MM (Wang et al.) 2024a) which contains 7M clips with high-quality GPT-4 (OpenAT 2023) refined caption. Since the videos are from diverse sources that may include portion that impedes egocentric representation learning, we employ filtering technique to retain only clips with egocentric style. train a style classifier P by manually annotating 10,000 clips as "ego-like" or "non-ego-like" With this classifier, we obtain an additional 3.4M egocentric-style clips. More details can be found in Appendix
3.1.2 GENERATING CAPTIONS WITH HAND OBJECT DYNAMICS
In this section, introduce HOD framework. Existing methods of refining descriptions in video-language pretraining(Z Zhao et al. 2023 Dou et al. 2024) focus on high-level abstracts but overlook the fine-grained details hand-object dynamics. This oversight detrimental in egocen- tric representation learning, where understanding these interactions forms considerable proportion of egocentric videos by nature. To address this gap, our HOD framework, we first detect the positions of hands and objects using hand object detector. With this information, we prompt large language model to augment the original annotation with detailed descriptions hand and ob- ject movements. Followed by the subsequent video-language pretraining, our Ego Video model can understand videos at finer- finer-grained level.
Thanks the rapid advancement in the field of hand-object interaction (Jiang et al. 2022 Ohkawa et al. (003), off-the-shelf hand-object detectors can provide robust hand and object positions. In our framework, employ 100DOH (Shan et al. 2020) the detector & Pe for bounding boxes extraction.
Hand Object Dynamics Detector.
For video clip (I1, T2, ...T., uniformly sample 16 frames within the clip to obtain fine-grained motion information. Then we use &it to acquire the bounding boxes of hands and objects these frames, which can be represented as
LH, RH,, LO,, RO, (xx(2x)
4

---

Published as a conference paper at ICLR 2025
where LH,, R,,,,O,,, LO,, RO, denotes the bounding box of the left hand, right hand, objects in contact with the left hand, and objects in contact with the right hand in the i-th frame. We use linear interpolation to compensate for missing hand boxes of frame if the corresponding hand boxes can be detected for both frames 1t-1 and T++1-
Hand Object Dynamics Rephraser. Current pretraining methods only high-level language descriptions (e.g.. "C takes the scissors" in Figure which lacks important egocentric details like hand and object interaction. In this work, incorporate these details into the video-language pretraining process. Hand-object dynamics encompass a variety of information, including bounding boxes of hands and objects, hand and object movement directions and trajectories. as well as their contact conditions. To integrate all this information into the video-language pretraining process, we use a L.LM as rephraser to express these dynamics natural language.
Specifically, we employ Yi-34B (Al et al. 2024) as our LLM. To capture the nuances of hand and object movements, we extract the central points of bounding boxes to derive trajectories for hands and objects. This process yields six essential categories of information: spatial-temporal data for 1) the left hand, 2) the right hand, 3) objects contacted by the left hand, 4) objects contacted by the right hand, 5) objects contacted by both hands, and 6) the original narration. We then prompt the LLM to amalgamate this detailed information, enabling the generation of rich narratives that intricately describe hand-object dynamics. Further details on prompting can be found in Appendix A
Analysis of HOD Data. We conduct additional analyses on our HOD data to evaluate its quality. First, we identify the top 30 most frequent words in HOD captions and the original EgoClip nar- rations and plot their normalized frequencies in Figure 3 The EgoClip narrations exhibit a more pronounced long-tail distribution, while our HOD captions display more balanced distribution. Notably, HOD captions include many "dynamic" words, such as "up" and "downwards," which aligns with the rationale behind our data generation process. To further verify the quality of our HOD, we employ GPT-4o [OpenAl (202) for quality assessment. We randomly select 1000 clips and let GPT evaluate the score of the caption data for the video clip in a range from 0 to 10. To ensure GPT does not simply assign high scores based on the length of the captions, we also conduct random gerund replacements on our data for comparison. The results, summarized Table show that our HOD data have a significantly better GPT-Score. Additional details on the scoring process and evaluations using other metries are provided in Appendix
HOD(OUn))
EgoClip
Results of narration quality, where HOD-random represents the narration after re- placing keywords. The GPT-Score ranges from 0 to 10, with higher values indicating higher quality of narration.
Table 1:
Data
GPT-Score
EgoClip HOD-random
5.53
3.70
Figure 3: Normalized frequency of the Top- 30 word in EgoClip (green) and our HOD (blue). Our HOD data has a less long-tail dis- tribution, showing its word diversity.
HOD
7.71
3.2 EGOCENTRIC REPRESENTATION LEARNING MODEL: EGOVIDEO
The narrations generated by our HOD pipeline are highly detailed. As result, the previous pre- training scheme struggles to capture the corresponding visual information at this level of detail. In response, we introduce Video (Figure bottom)) a model comprising a backbone and a motion adapter. The motion adapter aids in learning fine-grained hand-object dynamics from densely sam- pled video frames. Cooperating with a co-training strategy, our Ego Video model can obtain richer video representations while maintaining computational efficiency.
Visual and Text Encoder. Following the standard lideo-language pretraining setting (Lin et al. 2022), our model includes visual encoder F, (including motion adapter) and text encoder
5

---

Published as a conference paper at ICLR 2025
Fi. In the visual encoder, for a clip Tx with a learnable class token. The output of our visual encoder is Ey E RR.. For the text encoder, employ a 12-layer GPT-like Transformer (Radford et al.) 2019) that input tokens after BPE tokenization (Sennrich, 2015). The output of our text encoder is E, E R.P.
concatenate image tokens in T frames
Motion Adapter. Intuitively, to encode visual representations at the same level of detail as the languages, it is essential to utilize a greater number of frames as input. Since increasing the number of frames in training will result in unacceptable computational overhead, inspired by the PEFT technique in LLMs (Ding et al., 2023), we propose to use a lightweight motion adapter. The motion adapter is injected between the layers of the visual backbone, and tailored to learn the finer- grained details with high framerate. Since the hand and object motion forms spatiotemporal pattern, unlike previous methods (Pan et al., 2022. Xing et al., 2024 that only focus on learning temporal information, our module is designed to learn both spatial and temporal information.
Our motion adapter is attached to the top of each of the transformer layers. Without loss of generality, here we illustrate the motion adapter for one transformer layer and illustrate it in Figure Denote Y E IR Lx as the output transformer layer in F, where L is the number tokens, we first forward Y to a down-projection layer Wdewn down with ratio and followed with a GELU activation function o. Then, we use a 2D convolution layer Conv2D with kernel size (k,,k) to aggregate spatial information from each frame, followed by ID temporal convolution layer TConviD 1D and Linear layer W W to model the dynamics between adjacent frames. Finally, an up-projection layer Wap is used restore the dimension. Formally, the structure can be described as:
Y' (YYWowww)) Y,- = ReLU((BN(Con<2D(Y"))
(2)
Y = (TConniD(YY,))W: MotionAdapterr(Y Y) = Y + Y Wap:
where Wdown& E DX Wm E DD and Wap E D BN denotes BatchNorm2D.
Co-training Strategy. In EgoVideo, the motion adapter re- ceives imput at a higher framerate to capture the fine- fine-graines grained information. Additionally, the backbone must be trained to fully adapt to the egocentric domain. Thus, different from previous PEFT methods that freeze the backhone and only train the adapter part, we need to train both the backbone and adapter parameters. Motivated by the architecture of (Feicht- enhoter et al. 2019), we employ a co-training strategy to train the backbone and the motion adapter jointly.
Upsample
TConv1D
Conv2D
Specifically, we use an upsampling parameter À to sample the input using two sampling rates. For the input Φ, e xTxxxxxxx xC' with a low sampling rate, we pass it through the backbone and unfreeze all parameters. As a result, we get the output E.1- E DD_ For the input with a higher sampling rate In E R2 H XC we pass the input through both the back- bone and adapter parameters and get the output Evh R., during which we freeze the parameters of the backbone and only train the adapter. Finally, for the outputs of the two path- ways, we concatenate them and pass them through a fully con- nected layer to obtain the final output Ey D.D.
Downsample
Figure 4: Architecture of our mo- tion adapter. We use a 2D convo- lution layer and 1D temporal con- volution layer to capture the spatial and temporal dynamics efficiently.
Ev1 = Fbackbone (z1), Evh = Fo(
(3)
E,- = Ev1; Evn.W..
Where "[:]" denotes the concatenation operation, Fracktome denotes the visual backbone, F, denotes Fbockbone with motion adapter and W. E 2D With this strategy, we integrate the training of the backbone and adapter into single stage, reducing the cost of data and computation.
Texi Alignment. We follow the standard InfoNCE (Oord et al. 2018) loss as the objective for alignment between visual embedding E, and text embedding E. For sampled batch B, have:
(BL 1/TTT
(BL D/T
log ΣEJEB E/)/ + log
(4)
B
ΣBSEB CLEE).E//T
(E), E

---

Published as a conference paper at ICLR 2025
Table Zero-shot performance comparison on 4 tasks between methods with different model sizes ('B' for base, 'L' for large, and 'G' for our IB parameter backbone). Our Ego Video outperforms pre- vious methods with less but higher-quality pretraining data. Specifically, our Ego Video-G achieves significant performance improvements across all datasets.
EK-
MIR
EK-100 CLS
EGTEA
EgoMCQ
Method (ZS)
Data Size
mAP
nDCG
Topl-ace.
TopS-acc.
Mean-acc.
Topl-acc.
Intra
Inter
EgoVLPv2
26.7
60.9
LaViLa-B
35M
30.9
32.0
16.4
34.4
28.9
35.4
59.9
93.8
AVION-B
35M
32,9
32.7
EMBED-B
38.3M
36.0
34,9
19.0
39.0
37.0
42.7
61.3
94.5
EgoVideo-B
7.4M
36.5
34.5
22.4
43.3
43.6
51.0
64.6
95.0
LaViLa-L
35M
36.1
34.6
20.8
41.4
34.1
40.1
63.1
94.5
AVION-I
35M
37.6
35.3
Helping Hands
4M
37,5
37.8
39.1
46.6
63.0
94.5
EMBED-L
38.3M
40.8
37.5
22.8
45.0
40.3
46.7
64.7
95.6
VgoVodo-
7.4M
41.8
37.0
24.0
46.8
47.1
51.7
65.5
95.9
Videod
7.4M
47.1
39.0
28.5
54.3
58.0
63.0
69.1
96.6
where s s(E), E,)) denotes dot product operation between the i-th sample in the batch of Ey and E,, and T a temperature parameter that scales the similarity scores.
4 EXPERIMENTS
4.1 DATASETS AND EVALUATION PROTOCOLS
Pretraining Dataset. As stated in the previous section, the source of our pretraining data comes from Ego4D (Grauman al. 2022) and Hfw2--nterlink--7M (Wang al. 2024a). After processing by our HOD pipeline, the total amount of data is 7.4M clips.
Evaluation Protocols. We follow previous works ZZhao et al. 2023 Pramanick al. 2023) and use the following evaluation protocols. (1) Zero-shot (ZS): the pretrained video-text encoders are directly applied to the downstream datasets to perform video-text retrieval tasks without any addi- tional tuning. For classification, we compute the similarity score between the video clip and the textual descriptions of all possible classes. (2) Finetuned (FT): This approach involves taking pretrained video-text model and performing end-to-end finetuning the training split of the target downstream dataset. (3) ceature-based We extract video features using a frozen encoder and only train task-specific head the downstream dataset.
Model Architecture and Hyperparameters. Our vision- lisgun-language model follows the initialization of CLIP (Radford al. 2021), which is composed of vision encoder and text encoder. For the base and large models, ViT is used as our vision encoder, and we use temporal position embedding learn temporal information, which is randomly initialized. For our giant size model, we use Internvide02 (Wang et al., 2024b). For hyperparameters, we use T = 4 and X = 4 for frame inputs, and we use downsample ratio y 0.5 for the motion adapter. During pretraining, we freeze the temperature parameter 7 = 0.07. More details are placed in Appendix
Downstream Tasks. We evaluate models on several egocentric downstream tasks: (1) Epic- Kitchens- 100 (Damen et al. 2020) (EK-100) tasks. For this dataset, we evaluate our method on retrieval (EK- MIR) and action recognition (EK-100 CLS) tasks; 2) Ego4D (Grau-) man et al. 2022) tasks. For Ego4D, we evaluate our model on multiple choice questions (EgoMCQ) Li et al. 2021). and natural language query (EgoNL.Q) and moment query (EgoMQ) tasks; 3) EGTEA (Li et al.) 2018) tasks. We evaluate our model on the action recognition task that is focused on fine-grained cooking activities and hand-object interaction. 4) Other tasks. We also evaluate model on GTEA (Fathi et al. (011b) and HOI4D (Liu et al., (202)) datasets for the action segmentation task. Meanwhile, to show the generalization ability of our learned video representation, evaluate the task success rate Franka Kitchen dataset (Gupta al.) 2019), simulation environment for embodied AI.

---

Published as a conference paper at ICLR 2025
Table Fine-tuning performance of models of different sizes on tasks. Compared the previous
SOTA EMBED-L, our EgoVideo achieves new state-of- sate-of-the--aee thet performances on all datasets.
EK-100 MIR
10- CLS
EGTEA
EgoNLQ
EgoMQ
Method (FT)
mAP
nDCG
Topl-acc.
Topl-acc.
RI@0.5
RI@0.5
mAP
EgoVLPv2-B
47.3
61.9
7.9
31.1
12.2
EgoVideo-B
52.7
65.3
49.8
74.6
8.1
34.7
14.7
Helping Hands-L
7.9
33.4
16.0
La
50.9
66.5
51.0
76.0
7.3
32.5
13.4
EMBED-L
56.0
67.9
51.9
76.1
8.5
33,9
15.1
EgoVid
56.6
69.0
53.7
79.2
8.9
36.6
17.1
Ego VideoO
60.3
70.0
56.0
80.0
10.0
38.7
19.6
4.2 COMPARISON TO STATE-OF-THE-AR
Zero-shot Evaluation. Table shows the results on 4 tasks in the zero-shot setting. We compare our method against previous egocentric representation learning methods including EgoVLPv2 (Pra- manick et al. 2023 LaViLa (Zhao et al. 2023), AVION (Zhao & Krähenbühl 2023). Helping Hands (Zhang et al. 2023) and EMBED (Dou et al. 2024). Notably, despite the use of refined cap- tions and significantly larger training datasets, La ViL. L, AVION, and EMBED fail to achieve results our Ego Video. In the following experiments will demonstrate that both our high-quality HOD data and our design of the Video model play important roles in achieving good performance. stronger backbone TimeSformer (Bertasius al. (021) and adds additional
Helping Hands decoders for auxiliary object-oriented tasks. However, our method can still outperform Helping Hands, demonstrating the superiority of our representation learning scheme.
Specifically, in the EK-100 MIR task, our EgoVideo outperforms EMBED by 0.5%, 1.0%, 6.3% in mAp and significantly outperforms the LaViLa the model size. In the EK-100 CLS task, our Ego Video-B B model demonstrates superior performance with top-1 accuracy of 22.4% and a top-5 accuracy of 43.3%. significantly outperforming -a and EMBED-B.
On the EGTEA dataset known for its focus on hand-object interactions, our Ego achieves mean accuracy 43.6% and top-1 accuracy of 5 51.0% surpassing EMBED-B and EMBED-L. This underscores the importance of learning hand object dynamics and shows the strong general- ization capability of our model. The EgoMCQ task further highlights the efficacy of our method, with Ego Vi outperforming LaViLa-B by 4.7%, 1.2% and EMBED-B's by 3.3% and 0.5% on inter-class and intra-class accuracy, respectively. Our Ego Video-L model also shows significant improvements with an inter-class accuracy of 65.5% and an intra-class accuracy of 95.9%. These results demonstrate the superior performance and generalization capability of our method without any additional supervision. We take step forward explore the scaling law in egocentric repre- sentation learning, finding that Ego Video-G has elevated performance to the next level.
Fine-tuning Evaluation. Table shows the result of the fine-tuning evaluation. Our Ego Video method outperforms previous approaches across all tasks and datasets. Our Ego Video-B demon- strates significant performance enhancements compared EgoVIPvV2-B, with improvements of 5.4% and 2.5% in mAP for the EK-100 MIR and FgoMCQ tasks, respectively. This performance even comparable the larger LaV-LL For our Ego Video-L, we observe consistent improvements across all tasks, including substantial enhancement by 1.8% and 3. 3.1% in EK-100 CLS and EGTEA action recognition tasks, highlighting the superior performance understanding. Moreover, we achieve improvements of 0.4% in RI @0.5 in the EgoNLQ task and 2.7% and 2.0% in @0.5 and mAP the EgoMQ task, confirming the richness of representations learned by our model and its capacity to capture intricate hand-object interaction information.
our model in fine fine-graine sreeggraindd action
4.3 ABLATION STUDIES
Pretraining Data. We first conduct experiments by fixing the models and varying the pretraining data. Here we choose to use AVION for fair comparison since both AVION and Ego Video use ViT the backbone. As shown in 4 both our Ego Video and AVION achieve the best performance
8

---

Published as a conference paper at ICLR 2025
when the combination of Ego4D-HOD HOD data and How2-HOD data is used, and Ego Video consistently outperforms AVION when trained on the same data, emphasizing the effectiveness of the model design. Comparing models trained with EgoClip and Ego4D-HOD (rows 1,2 and 5,6), it is clear that significant improvements can be observed in the EK-100 MIR and EGTEA tasks. Adding additional data from How2-HOD can improve both models substantially (rows 1,3 and 5,7). Furthermore, when using only Ego4D-HOD, the performance on EGTEA surpasses EgoClip and How2-HOD together, indicating the beneficial impact of data on fine-grained dynamics understanding.
Table 4: Ablations on different pretrain datasets, include original EgoClip (Zhao & Krähenbühl 2023
Ego4D-HOD and How2-HOD selected by our classifer from How2Interllnk--MM
Ego4D-
Ego4D-
How2-
EK-100 MIR
EGTEA
ID
Model
EgoClip
HOD
HOD
MAP
#DCG
Mean-acc.
Topl-acc.
27.3
29.3
26.2
30.5
2
(+.0
(12.0+
(+..
37.0(+6.5)
AVION-B
3
33.2(45.9)
(2.5(+3.2)
31.6(+5.4)
35.6(+5.1)
4
34.4 (47.))
33.7
39.4 (413.2)
46.4 (6.4(416.9)
31.1
32.0
30.8
36.0
6
34.4(43.8)
(3.9(4
41. (1.1(410.3)
(7.9(+
EgoViden-B
7
(4.5
(4.1()2.)
40.8(+10.0)
47.1(+11.1)
8
36.5 (45.4)
34.5 (42.8)
(
(11.0)
Table 6: The computational cost during infer- ence. Views #frames #spatial crops x #temporal clips. "Extra GFLOPs" means extra computation compared ViT.
Table 5: Comparison of the number of pa-
rameters.
EK100
Method
Backhone
Params
mAP
Extra
LaViLa-B
TSF-B
121M
30.9
Method
Views
GFLOPs
GFLOPs
AVION-B
ViT-B
86M
32.9
VIT-B
4 1 3
201
EMBED-B
TSF-B
121M
36.0
ViT-B
16 x x
EgoVide
VIT-B
112M
36.5
LaViLa-B
16 x x
1432
LaViLa-L.
TSF-
438M
36.1
Videoo
16 x 1 x
1092
288
AVION-L
VIT-L
307M
37.6
ViT-L
4 1 1
1047
EMBED-L
TSF-L
438M
40.8
VIT-L
16
4188
EgoVidco-L
VIT-L
375M
41.8
LaViLa-L
16 x x
4956
EgoVideo-G
VIT-G
1050M
47.1
Vgo
16 x x
5350
1162
Model Size and Inference Computational Cost. Table Table compares the number of parameters. Our Ego Video model maintains relatively small parameter count, and even with the addition of the mo- tion adapter, the total remains lower than that of La and EMBED, highlighting the efficiency of our approach. Meanwhile. Table we compare the inference computational cost our Ego Video with ViT and ViLa. Thanks to our MotionAdapter the increase in inference time for our model compared to VIT 16 frames, only similar to ViT's inference time frames.
Training Efficiency. In Table 7 we compare the performance and computational speed of our Video-B with AVION-B, where AVION-B is trained under two different parameter settings: pre-training with 16 frames and pre-training with 4 frames. Our Video is trained in mixed 16 and frame fashion, thus being faster than directly using all 16 frames to train the whole backbone. Mcanwhile, Video achieves the best performance the EK-100 MIR and EGTEA datasets. These results strongly demonstrate the effectiveness of our training strategy and motion adapter design in the Ego Video model.
Motion Adapter vs Other Adapters. We compare our motion adapter with the standard Adapter (Houlsby al. 2019) and the ST-adapter (Pan et al., 2022). In the standard adapter, only use downsample MLP and upsample MLP, while in the ST-adapter, we perform convolution op- erations solely along the temporal dimension. As shown in Table 8 the results on the EK-100 MIR

---

Published as a conference paper at ICLR 2025
task demonstrate that both the ST-adapter and our motion adapter outperform the standard adapter. This improvement can be attributed to the limited parameters of the standard adapter, which restrict its abilitt to capture complex, fine-gratd aneeggained information. Compared the ST-adapter, Motion Adapter achieves the best performance by adding a spatial convolution operation, suggesting that both spatial and temporal information crucial for egocentric video representation learning.
Table Ablations on training strategy. Mod- els are all trained on our Ego4D-HOD dataset with 10 epochs.
Table 8: Experiment with different adapters. Our motion adapter achieves the best perfor- mance with small increase in parameters.
GPU
EK-100
MIR
EGTEA
EK-100 MIR
Method
Design
Param Size
Hours
mAP
nDCG
Topl-acc.
mAP
nDCG
AVION-4F
95.5
34,4
33.7
46.4
Adapter
8.28M
34.7
33.0
AVION-16f
395.5
36,2
34.3
47.4
ST-adapter
10.08M
35,9
34.1
Video
180.6
36.5
34.5
51.0
Motion Adapter
26.01M
36.5
34.5
4.4
FEATURE-BASED EVALUATION OTHER TASKS
With the knowledge of hand-object dynamics, EgoVideo features can well generalize to other hu- man behavior understanding tasks and robot manipulation tasks. 9 shows the results of Action Segmentation the HO14D (Liu et al.) 2022) and GTEA (Fathi et al.) 201 datasets, using fea- tures extracted from I3D (Carreira & Zisserman 2017), AVION, and our EgoVideo. The results demonstrate our Ego Video is also effective in the action segmentation task, especially for HOI4D which requires differentiating fine-grained hand-object interaction.
Also, we test the generalization capability of Ego on the robot manipulation task the Franka Kitchen dataset (Gupta et al. 2019). We follow the same setting and compare with previous robotic representation learning works MVP (Radosavovic et al. 2023 Voltron (Karamcheti al.) 2023) and MPI (Zeng et al. 2024). For MPI we compare both MPI with and without additional detection supervision. From Table TO our Ego can consistently surpass MVP and Voltron the "Turn knob (TK)", "Open Microwave (OM)" and "Open door (ODY* tasks. While MPI uses additional detection and prediction transformers and performs better than Ego on two tasks, Ego Video still performs comparably in overall success rate. Complete results with more details and analyses can be seen in Appendix E The results strongly prove the delicacy and generalization ability of our Ego Video learned representations.
Table Experiment on the action segmentation task. We report results of ASFormer (Yi et al.) with different input features.
Table 10: Results on Franka Kitchen. We report the success rate (%) on 50 sampled trajectories.
202T
Method
TK
OM
OD
Avg.
HOI4D
GTEA
MVP
79.0
41.0
48.0
56.0
Feature
F/
Edit
Fl 50
Edit
Voltron
76.0
41.0
45.3
54.1
13D
35.0
80.3
79,2
84.6
MPI
85.5
49.0
52.5
62.3
AVION
70.2
89.1
84.5
89.4
Ego
80.1
65.0
52.7
66.0
EgoVideo
74.8
90.1
87.1
90.1
MPI+Det
89.0
54.0
57.7
66.9
5 CONCLUSION
In this work, we inject fine-nrained hand-object dynamics into egocentric video representation learn- ing. Our method addresses the drawbacks of the existing method from two perspectives. On the data side, we propose HOD, a novel framework to generate new paired video-language data, where the language contains intricately depicted hand-object dynamics. On the model side, we propose Ego Video, where we use a model with a motion adapter combined with a co- co-training technique, to fully exploit the fine- mne-grained dynamics provided by HOD data in the representation learning pro- cess. Experimental results demonstrate that our method achieves state-of-the-aan performance across multiple downstream tasks, and can generalize in the embodied manipulation environment.
10

---

Published as a conference paper at ICLR 2025
Acknowledgement This work is funded in part by the National Key R&D Program of China (2022D01160201), and Shanghai Artificial Intelligence Laboratory, and JSPS KAKENHI Grant Number JP22KF0119.
REFERENCES
01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang. Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang. Shiming Yang. Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Yi: Open foundation models by 01.ai, 2024.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, volume 2, pp. 4, 2021.
Samarth Brahmbhatt, Chengcheng Tang, Christopher D Twigg, Charles C Kemp, and James Hays. Contactpose: A dataset of grasps with object contact and hand pose. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow; UK, August 23-28, 2020, Proceedings, Part XIII 16, pp. 361-378. Springer, 2020.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 961-970, 2015.
Yujun Cai, Liuhao Ge, Jianfei Cai, and Junsong Yuan. Weakly-supervised 3d hand pose estimation from monocular rgb images. In Proceedings of the European conference on computer vision (ECCV), pp. 666-682, 2018.
Zhe Cao, Ilija Radosavovic, Angjoo Kanazawa, and Jitendra Malik. Reconstructing hand-object interactions in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12417-12426, 2021.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 0299-6308, 2017.
Guo Chen, Yifei Huang, Jilan Xu, Baoqi Pei. Zhe Chen, Zhiqi Li, Jiahao Wang, Kunchang Li, Tong Lu, and Limin Wang. Video mamba suite: State space model as a versatile alternative for video understanding. arXiv preprint arXiv:2403.0960
2024.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European conference computer vision (ECCV), pp. 720-736, 2018
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, al. The epic-kitchens dataset: Collection, challenges and baselines. IEEE Transactions on Puttern Analysis and Ma- chine Intelligence, 43(11):4412) 4141, 2020.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efffcient fine-tuning of large-scale pre-traines language models. Nature Machine Intelligence, 5(3):220-22 235, 2023.
Bardia Doosti, Shujon Naha, Majid Mirbagheri, and David Crandall. Hope-net: graph-based model for hand-object pose estimation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 6608-666 1008-661,.00 2020.
Zi-Yi Yi Dou, Xitong Yang. Tushar Nagarajan, Huiyu Wang, Jing Huang, Nanyun Peng, Kris Kitani, and Fu-Jen Chu. Unlocking exocentric language data for egocentric video representation learning. arXiv preprint arXiv:2408.08:: 03566, 2024.

---

Published as a conference paper at ICLR 2025
Yazan Abu Farha and Jurgen Gall. Ms-ten: Mtlti-stage temporal convolutional network for tion segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3575-3584 2019.
Alireza Fathi, Ali Farhadi, and James M Rehg. Understanding egocentric activities. In 2011 inter- national conference computer vision, pp. 407-414. IEEE, 201
Alireza Fathi, Xiaofeng Ren, and James M Rehg. Learning to recognize objects in egocentric activ- ities. In CVPR 2011, pp. 3281-3288, IEEE, 2011b.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference computer vision, pp- 6202-6211, 2019. 2019.
Rohit Girdhar and Kristen Grauman. Anticipative video transformer. In Proceedings of the JEEE/CVF international conference on computer vision, pp. 13505-13515, 2021.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18995-19012. 2022.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long- horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910. 11956, 2019.
Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11807-11816, 2019.
Yana Hasson, Bugra Tekin, Federica Bogo, Ivan Laptev, Mare Pollefeys, and Cordelia Schmid. Leveraging photometric consistency over time for sparsely supervised hand-object reconstruction. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 571- 580, 2020.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference machine learning, pp. 2790-2799 PMLR, 2019.
Yifei Huang, Minjie Cai, Zhenqiang Li, and Yoichi Sato. Predicting gaze in egocentric video by learning task-dependent attention transition. In European Conference Computer Vision, 2018.
Yifei Huang, Minjie Cai, Zhenqiang Li, Feng Lu, and Yoichi Sato. Mutual context network for jointly estimating egocentric gaze and action. IEEE Transactions Image Processing, 29:7795- 7806, 2020a.
Yifei Huang, Yusuke Sugano, and Yoichi Sato. Improving action segmentation via graph-based temporal reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14024-1444 14034, 2020b.
Yifei Huang, Guo Chen, Jilan Xu, Mingfang Zhang, Lijin Yang, Baoqi Pei, Hongjic Zhang, Lu Dong, Yali Wang. Limin Wang, al. Egoexolearn: A dataset for bridging asynchronous ego-and exo-centric view procedural activities in real world. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22072-22086. 2024a.
Yifei Huang, Jilan Xu, Baoqi Pei, Yuping He, Guo Chen, Lijin Yang, Xinyuan Chen, Yaohui Wang, Zheng Nie, Jinyao Liu, et al. Vinci: A real-time embodied smart assistant based egocentric vision-language model. arXiv preprint 12. 2024b.
Md Mohaiminul Islam, Ngan Ho, Xitong Yang. Tushar Nagarajan, Lorenzo Torresani, and Gedas Bertasius. Video recap: Recursive captioning of hour-long videos. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 18198-18208, 2024.
12

---

Published as a conference paper at ICLR 2025
Hanwen Jiang, Shaowei Liu, Jiashun Wang, and Xiaolong Wang. Hand-object contact consistency reasoning for human grasps generation. In Proceedings the IEEE/CVF international conference computer vision, pp. 11107-11116, 2021.
Siddharth Karamcheti, Suraj Nair, Annie S Chen, Thomas Kollar, Chelsea Finn, Dorsa Sadigh, and Percy Liang. Language-driven representation learning for robotics. arXiv preprint arXiv:2302. 12766, 2023.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint
2017.
Shijie Li, Yazan Abu Farha, Yun Liu, Ming-Ming Cheng, and Juergen Gall. Ms-ten++: Multi-stage temporal convolutional network for action segmentation. IEEE transactions on paltern analysis and machine intelligence, 45(6):6647-6658 2020.
Yanghao Li, Tushar Nagarajan, Bo Xiong, and Kristen Grauman. Ego-exo: Transferring visual rep- resentations from third- horrd-peroontee to first-person videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6943-6953, 2021.
Yin Li, Miao Liu, and James M Rehg. the eye of beholder: Joint learning of gaze and actions in first person video. In Proceedings of the European conference computer vision (ECCV), pp. 619-635, 2018.
Kevin Qinghong Lin, Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Xu, Difei Gao, Rong-Cheng Tu, Wenzhe Zhao, Weijie Kong, et al. Egocentric video-language pretraining. Advances in Neural Information Processing Systems, 35:7575-7586, 2022.
Shaowei Liu, Hanwen Jiang, Jiarui Xu, Sifei Liu, and Xiaolong Wang. Semi-supervised 3d hand- object poses estimation with interactions in time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14687-14697, 2021.
Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. Hoi4d: A 4d egocentric dataset for category-level human-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21013-21022, 2022.
Mi Luo. Zihui Xue, Alex Dimakis, and Kristen Grauman. Put myself in your shoes: Lifting the egocentric perspective from exocentric videos. arXiv preprint arXiv:2403.06331, 2024.
Antoine Miech, Dimitri Zhukov, Jean- Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howtol00m: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEFF/CVF international conference on computer vision, pp-
2630-2640, 2019.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A univer- sal visual representation for robot manipulation. arXiv preprint arXiv:2203.12601 2022.
Takehiko Ohkawa, Kun He, Fadime Sener, Tomas Hodan, Luan Tran, and Cem Keskin. Assembly- hands: Towards egocentric activity understanding via 3d hand pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12999-13008, 2023.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, arXiv preprint arXiv: 877807787778. 2018.
R OpenAL. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2(5), 2023.
Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li. St-adapter: Parameter-efficient image-to-video transfer learning. Advances in Neural Information Processing Systems, 35:26462- 26477, 2022.
Xiaqing Pan, Nicholas Charron, Yongqian Yang, Scott Peters, Thomas Whelan, Chen Kong, Omkar Parkhi, Richard Newcombe, and Yuheng Carl Ren. Aria digital twin: A new benchmark dataset for egocentric 3d machine perception. In Proceedings of the IEEE/CVF International Conference Computer Vision pp. 20133-20143 2023.
13

---

Published as a conference paper at ICLR 2025
Baoqi Pei, Guo Chen, Jilan Xu, Yuping He, Yicheng Liu, Kanghua Pan, Yifei Huang, Yali Wang, Tong Lu, Limin Wang, et al. Egovideo: Exploring egocentric foundation model and downstream adaptation. arXiv preprint arXiv:2406.18070 2024.
Chiara Plizzari, Mirco Planamente, Gabriele Goletto, Marco Cannici, Emanuele Gusso, Matteo Matteucci, and Barbara Caputo. E2 (go) motion: Motion augmented event stream for egocentric action recognition. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 19935-199477 2022.
Chiara Plizzari, Gabriele Goletto, Antonino Furnari, Siddhant Bansal, Francesco Ragusa, Gio- vanni Maria Farinella, Dima Damen, and Tatiana Tommasi. An outlook into the future of egocen- tric vision. International Journal of Computer Vision, pp. 1-57, 2024.
Shraman Pramanick, Yale Song, Sayan Nag. Kevin Qinghong Lin, Hardik Shah, Mike Zheng Shou, Rama Chellappa, and Pengchuan Zhang. Egovipv2: Egocentric video-language pre-training with fusion in the backbone. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5285-52852 5297, 2023.
Alec Radford, Jeffiey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763, PMLR. 2021.
Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training. In Conference on Robot Learning, pp- 416-426. PMLR, 2023.
Snehanshu Saha, Archana Mathur, Aditya Pandey, and Harshith Arun Kumar. Diffact: A unifying framework for activation functions. In 202/ International Joint Conference on Neural Networks (IJCNN) pp. 1-8. IEEE, 2021.
Rico Sennrich. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.00 07909, 2015.
Dandan Shan, Jiaqi Geng, Michelle Shu, and David F Fouhey, Understanding human hands in contact at internet scale. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1869-98775 2020.
Sanjana Srivastava, Chengshu Li, Michael Lingelbach, Roberto Martin-Martin, Fei Xia, Kent Elliott Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, Karen Liu, et al. Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments. Conference on robot learning, pp. 477-490 PMLR, 2022.
Alex Jinpeng Wang, Linjie Li, Kevin Qinghong Lin, Jianfeng Wang, Kevin Lin, Zhengyuan Yang, Lijuan Wang, and Mike Zheng Shou. Cosmo: Contrastive streamlined multimodal model with interleaved pre-training. arXiv preprint arXiv:2401.00849 2024a.
Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, al. Internvideo2: Scaling video foundation models for multimodal video understanding, arXiv preprint arXiv:2403. 15377, 2024b.
Zhen Xing, Qi Dai, Zejia Weng, Zuxuan Wu, and Yu-Gang Jiang. Aid: Adapting image2video diffusion models for instruction-guided video prediction. arXiv preprint 46644.240.006665, 2024.
Jilan Xu, Yifei Huang, Junlin Hou, Guo Chen, Yuejie Zhang, Rui Feng, and Weidi Xie. Retrieval- augmented egocentric video captioning. In Proceedings of the IEEE/CVF Conference on Com- puler Vision and Pattern Recognition, pp. 13525-13536, 2024.
14

---

Published as a conference paper at ICLR 2025
Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo. Advancing high-resolution video-language representation with large-scale video transcriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5036-5045, 2022.
Linlin Yang and Angela Yao. Disentangling latent hands for image synthesis and pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9877-9886, 2019.
Fangqiu Yi, Hongyu Wen, and Tingting Jiang. Asformer: Transformer for action segmentation.
arXiv preprint 1rXiv:2110008550 2021.
Shanxin Yuan, Guillermo Garcia-HHerando, Björ Stenger, Gyeongsik Moon, Ju Yong Chang, Ky- oung Mu Lee, Pavlo Molchanov, Jan Kautz, Sina Honari, Liuhao Ge, et al. Depth-based 3d hand pose estimation: From current achievements to future goals. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2636-2645, 2018.
Jia Zeng, Qingwen Bu, Bangjun Wang, Wenke Xia, Li Chen, Hao Dong, Haoming Song. Dong Wang, Di Hu, Ping Luo, et al. Learning manipulation by predicting interaction. arXiv preprint arXiv:2406.00439 2024.
Chuhan Zhang, Ankush Gupta, and Andrew Zisserman. Helping hands: An object-aware ego- centric video recognition model. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 13901-13912, 2023.
Hao Zhang, Aixin Sun, Wei Jing, and Joey Tianyi Zhou. Span-based localizing network for natural language video localization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6543 6554, Online, July 2020. Association for Computational Linguistics. Uttttts:::/www wwwweb.b.......................l http://wwww.......t.o...oo/..... act.mmmm.....mma........... 585
Lingzhi Zhang, Shenghao Zhou, Simon Stent, and Jianbo Shi. Fine-grained egocentric hand-object segmentation: Dataset, model, and applications. In European Conference on Computer Vision, pp. 127-145. Springer, 2022.
Chen Zhao, Ali K Thabet, and Bernard Ghanem. Video self-stitching graph network for temporal action localization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13658-13667, 2021.
Yue Zhao and Philipp Krähenbühl. Training large video model on a single machine in day. arXiv preprint ar0iv:2309. 16669, 2023.
Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar. Learning video representations from large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6586-6597, 2023.
15

---

Published as a conference paper at ICLR 2025
A
DETAILS ABOUT HOD
Positive
Negative
Egocentric Video
Ego-like Video
Non-ego Video
Figure 5: Examples of egocentric video/ego-like video and non-ego video.
Data Selection Since our HOD involves data from not only Ego4D but also How2link-7M, we use a style classifier P filter egocentric style videos from the How2link-7W4 7M dataset. Specifically, our style classifier employs a simple two-layer MLP architecture. We utilize InternVideo2 (Wang al, 2024b) to extract video features from all videos of the How2-Interlink7M dataset. After that, we manually annotate 10,000 clips with positive and negative labels, where the positive label indicates this video is an egocentric video (or ego-like video). Examples of positive and negative labeled videos can be found in Figure 5 We randomly select 10% of these clips form the validation set. After training our classifier on the train set, we get 89% accuracy on the validation set.
HOD Rephraser We Yi-34B model to generate hand object dynamics narrations. The Yi-34B model is trained on a corpus of over I 150,000 high-quality texts and its model weights are open- source, which has high ranking among all existing open-source Large Language Models. We directly use the model without finetuning.
To generate reliable narrations, we need to convert the obtained hand-object information into appropriate texts. For the movement trajectories of hands and objects, we directly calculate the center points of the bounding boxes and perform normalization (w)), (wi, du) ..., (wis, ). To determine whether the object is contacted by the left hand or right hand separately, or contacted by both hands simultaneously, we use generalized loU function on the left-contact object and right-contact object. For loU value greater than 0.9, classify the object as being contacted by both hands.
get sequence
Subsequently, we prompt the LLM with system prompt of:
# System Prompt
you captioning assistant, you need generate hand object interaction caption and combine them with the crigin narration. Given the origin narration of video elip spatial localization ([x, yl) hands and objects in the elip, please help me describe the direction of motion of the left and right hands, their relative relationship to objects and whether they are teuching or not. not
mention the pixel info. Two.hand.object means objects with two hands in contact, left.hand.ob ject means objects with left hand in centact, right jet means objects with right hand in contact.
If Rand Object Dynamics es left_hand: h_0,,, (w_1, h_1,,,11), ...,, (x_15)
right.hand: ((w_0, h_0), (w.1, h.11,,1)).... ...,, (w.15, b.15)
laft.hand.object: ((w.0, h_0), (k.1,h.1), right.hand.ebjeed (w.0, h20), (w.1, b..15,
(w.15,
two.hand.object: [w_0,, h20), (.11,,11),. h.115,
origin narration: C takea a scisscrs.
16

---

Published as a conference paper at ICLR 2025
44 System Prompt
P Please help me sumarrize the direction of movement of the left hand, right hand, and objects, and generate new caption based or the original caption. is strictly forbidden to mention the frame number and spatial position coordi nates in the description.
For the computational cost, takes around 2 days to extract bounding boxes from all vision-language clips and days generate narrations with LLM using 32 A100 GPUs, resulting total of around 4000 GPU-hours.
Data Evaluation. Here, provide detailed explanation of our evaluation process. First, prompted the LLM to generate new narrations with different verbs/nouns using the prompt:
Please help me modify the key verbs and nouns in this sentence to slightly alter its meaning while keeping the sentence structure largely unchanged. Just return the modified sentence to me. Ensure the semantic shift minimal, such as changing one or two verbs and nouns.
Then we utilized GPT-4o as a judge determine the quality of the narrations with the prompt:
You are judge. There are 16 frames in the video, have three captions and need your help to score the three captions based on three criteria: relevance, accuracy, and level of detail. The score ranges from to 10, with higher score indicating batter quality of the caption. You can just answer me in the following format: First: scorel, Second: score2, Third: score3. First caption: textl tex:3
Second caption: text2 Third caption:
As mentioned in the main manuscript, to further validate the quality of our HOD dataset, we utilize two standard unsupervised automatic metries to evaluate the quality narrations. We use the human narration the ground truth and compare HOD data with LaViLa-Narrator on METEOR and CIDEr scores. The results in Table II reveal that while our HOD data achieves a slightly lower METEOR score, it outperforms
in CIDEr. This discrepancy arises because many LaViLa narrations closely mirror the original text, whereas our narrations incorporate additional dynamic information. Although our performance does drastically exceed that of La ViLa, the results demonstrate that our narrations successfully retain the original semantic content.
Table 11: Comparison with La LaViLa-Narrin on the narration quality.
Text
METEOR
CIDEr
LaViLa-Narrate
0.45
0.34
HOD
0.39
0.40
Limitations and future work. Our model relies the quality of hand-object detection and the rephrasing of LLM, which may include error accumulation. In addition to reducing the error in the data construction, exploring how the hand-object dynamics can be better involved into language or in other formats is promising direction for our future work.
B DATASET DETAILS
Ego4D Ego4D Grauman et al., 2022) contains 3,670 hours of egocentric videos with temporally dense narrations. Each narration has timestamp and an associated free-form sentence. We follow previous works ((Zhao et al. ) (Lin et al., 2022)) to prepare the Ego4D dataset for vision- language pretraining. Specifically, we drop the narrations that either contain "#unsure"/*4Unsu" tags or are shorter than words. This results in 4M video-text clip pairs.
Howto-Interlinke Howto-Interlin77M WWag et al. 2024a) contains 1M videos and 7M clips, which is part of the broader Howtol00m dataset. Diverging from the original dataset, clips in Inttoo-nterlink7M have concise descriptions, and dense region captions and leverage GPT-4 to generate comprehensive summaries from detailed annotation. We use a classifier to select 3.3M vision-text pairs from the dataset.
17

---

Published as a conference paper at ICLR 2025
EpicKitchens-100 The SpiccKitchenn- 100 (EK-100) dataset (Damen al.) 2020: 2018 contains 100 hours of egocentric cooking videos. Each clip is annotated with a start and end timestamp, a short textual narration, and verb and noun class that the narration belongs to. The action class can also be uniquely determined by combining the verb and the noun. In EpicKitchens-Mi use Mean Average Precision and normalized Discounted Cumulative Gain (nDCG) evaluation metrics. In EpicKitchens-CCe we use top-1 action accuracy and top-5 action accuracy as evaluation metrics.
we
EGTEA EGTEA et 2018) contains 28 hours of cooking activities from 86 unique sessions of 32 subjects. In zero-shot evaluation, we compute the similarity score between every video embed- ding and the 106 text embeddings, and take the text embedding with the highest similarity score as the predicted class. In fine-tuning evaluation, we finetune the video encoder for action classification. using the linear probing protocol.
GTEA The Georgia Tech Egocentric Activities (GTEA) dataset (Fathi et al., 201 consists of seven distinct types of everyday activities, including making a sandwich, preparing tea, and brewing coffee. Each of these activities is demonstrated by four different individuals, resulting a total of 28 unique video recordings. Each video captures around 20 fine-grained action instances, such as "take bread" or "pour ketchup." all occurring within approximately one minute. This dataset provides comprehensive look at egocentric perspectives, making it invaluable resource for research in activity recognition and human-computer interaction.
HOI4D The HOI4D dataset (Liu et al. 2022) represents significant advancement in the study of -ategory-level human-object interaction, offering a large-scale 4D egocentric resource enriched with detailed annotations. Comprising 2.4 million RGB-D egocentric video frames across more than 4,000 sequences, the dataset captures interactions performed by nine participants with 800 unique object instances spanning 16 categories within 610 diverse indoor environments. To foster advancements in category-level human-objec interaction, HOI4D introduces three benchmarking tasks: semantic segmentation of 4D dynamic point cloud sequences, category-level object pose tracking, and egocentric action segmentation involving variety of interaction targets.
Franka Kitchen The Franka Kitchen dataset (Gupta et al. 2019) is comprehensive resource de- signed to facilitate research in robotic manipulation and human-robor interaction within a kitchen environment. This dataset comprises a diverse collection of videos showcasing a humanoid robot, Franka Emika Panda, performing various cooking tasks. The setup features a Franka robot with 9 degrees of freedom positioned within a kitchen environment equipped with various common house- hold items, including microwave, kettle, overhead light, cabinets, and an oven. This environment is designed for multitask objectives, requiring the robot to interact with these items to achieve specific goal configurations.
C IMPLEMENTATION DETAILS
Pretraining Details We pre-train on the video-narration pairs generated by our HOD from Ego4D and Hlow-InterLink7M. We use AdamW optimizer with betas (0.9,0.999) for 15 epochs. We use different settings for different size models. For Ego Video-B, we adopt a batch size of 128 over 16 GPUs with fixed learning rate of 5c-5, For Ego Video-L, we use a batch size of 32 over 16 GPUs with a fixed learning rate of 3c-5. For Ego we choose to use a batch size of 16 over 16 GPUs with a fixed learning rate of 1e-5. For input frames, we preprocess the frames by resizing the shorter side to 320 pixels, which accelerates the data loading speed. Subsequently, we applied standard RandomResizedCrrg function (Zhao & Krähenbühl 2023) with scale parameter of (0.5, 1.0) to obtain the corresponding input frames.
Finetuning Details We finetune the downstream tasks using Adam W with (81, (2) (0.9, 0.999) and weight decay of 0.05 with cosine annealing. Table shows the hyperparameters details and in all tasks we use GPUs for finetuning. During the training phase, we resize the shorter side of the video 256 pixels and subsequently extract 224x224 crop. During the testing phase, scale the shorter side to 224 pixels and take a central 224x224 crop.
For the EgoNLQ task Grauman et al. 2022),
build on the methodologies introduced by Ego
(Lin et al., 2022) and LAVILA (Zhao et al. 2023) for faimess. We adopt VSLNet (Zhang et al.) 2020) as the task head. We train the task head for 50 epochs, using learning rate of 3e-3, dropout
18

---

Published as a conference paper at ICLR 2025
Table 12: Hyperparameters for Different Downstream Tasks
Task
Model Size
Epochs
LR_start
LR_end
Batch Size
EgoVideo-B
100
le-6
le-5
256
EK100-MIR
EgoVideo-L
70
5c-7
5e-6
64
Ego EgdVoG
50
4c-7
4c-6
32
EgoVideo-BB
100
le-6
le-5
256
EKI00-CLS
EgoVideo-LL
70
5e-7
5e-6
64
EgoVideo-G
60
4e-7
4e-6
32
EgoVideo-B
100
le-6
le-5
256
EGTEA
EgoVideo-L
70
7e-7
7e-6
64
EgoVideo-G
50
4e-7
4c-6
32
0.3, batch size 32 on single A GPU. For the EgoMQ task, we use VSGN (Zhao et al. 202] as our task head for training. We set batch size as 16, learning rate as 2e-4, gamma as 0.6, and train the task head on single A100 GPU.
Table 13: Comparison of performance across different model sizes and vision-language datasets.
EK-100 MIR
Model
Pretrain Data
Data Size
MAP
nDCG
EgoVideo
EgoClip
4M
31.1
32.0
EgoVideo-B
Ego4D-HOD
4M
34.4
33,9
EgoVideo-L
EgoClip
4M
35.3
34.6
EgoVide
Ego4D-HOD
4M
38.3
35,9
EgoVideo-G
EgoClip
4M
42.1
37.5
Videod
Ego4D-HOD
4M
44.8
38.2
D ADDITIONAL ABLATIONS
Pretraining Data To further demonstrate the effectiveness of our HOD, we fix the amount of data and conduct experiments using different model sizes. As shown in Table [3 with the same model size and the same data size, using our Ego4D-HOD data can consistently achieve improvement. Since one sample in EgoClip corresponds strictly on sample in Ego4D-HOD, this table strongly demonstrates the high quality of our HOD data.
Adapter Downsampling Ratio We test the design of our motion adapter by studying the effect of adapter downsampling ratio and show the result in Figure It can be observed that as the value of y increases. the model's performance continues improve. This indicates that our generated narrations contain rich semantic information and further validates the effectiveness of our motion adapter. To reduce computational overhead, we ultimately decide set 7 = 0.5.
Frame number We further study the effect of the number of sampled frames as input. We consis- tently use frames as the sampling rate for the backbone part. The results Table 14 indicate that as the number of frames increases from to 16, the model's performance improves continuously from 34.2% to 36.5%. However, when the frame count reaches to 32, the performance plateaus, showing significant improvement with further increases in frame count. Besides, increasing the number frames beyond this point incurs substantial computational cost. As result, we choose to the default value in Video balancing the speed and accuracy.
19

---

Published as a conference paper at ICLR 2025
37.0
Frame count
EK MIR
À
mAP
36.6
36.5
4
1
34.2
J6.2
36,2
8
2
35.3
35,9
35.8
12
3
35,9
35.8
35.6
16
4
36.5
35.4
32
8
36.5
35.0
Table 14: Comparison of perfor- mance across different frame up- sampling rate X.
Dowmsampling
Figure 6: Ablations on Downsampling Ratio y.
E ADDITIONAL RESULTS
Table 15: Comparison with the state-of shate-of-the-ee methods on 50Salads, GTEA and HOI4D dataset.
GTEA
HOI4D
Feature
Method
FI@ 10, 25, 50
Edit
Acc
FI@ 10, 25, 50
Edit
Acc
IBD
MS-TCN
85.8 83.4 69.8
79.0
76.3
35.6/77.8/311
74.7
44.2
I3D
MS-TCN++
88.8 85.7 76.0
83.5
80.1
54.7 46.5 30.3
75.2
42.2
I3D
ASFormer
88.8 90.1 88.8 79.2
84.6
79.7
I3D
DiffAct
92.5/91.5 84.7
89.6
82.2
AVION
ASFormer
92.5/91.0 84.5
89.4
81.4
84.4
81.1
70.2
89.2
74.2
EgoVideo
ASFormer
92.7/92.22 87.1
90.1
82.7
88.9 85.3 74.8
90.1
76.2
E. DETAILS ON ACTION SEGMENTATION TASKS.
Action segmentation tests the representation on its understanding of the temporal dependencies of the video Huang al. (2020b): Yi al. (202T). We evaluate model two benchmark datasets: GTEA (Fathi et al. 2011b), and HOI4D (Liu et al. 2022). We follow the previous work to use four-fold cross-validations
both datasets. We accuracy (Acc), the edit distance (Bdit), and the Fl scores at overlap thresholds 10%, 25%, 50% (F1 @ 10, 25, 50) as metrics for evaluation.
We ASFormer Yi et al. 2021 the task head, with input features extracted by our Egovideo, I3D (Carreira & Zisserman, 2017). and AVION (Zhao & Krähenbühl 2023). We follow (Chnn
al. 2024), using learning rate 5c-4, drop rate = 0.3, epoch 100 for training. Table 15 presents the experimental results of our method and other recent approaches, including MS-TCN (Farha & Gall 2019), MS-TCN+ (Li al., 2020), ASFormer (Yl et al. 2021), and DiffAct (Saha et al.
2021 The results clearly show the high quality of our Ego feature. With the same task head ASFormer, the EgoVideo feature can outperform the AVION feature consistently. Eid can even help ASFormer to beat the stronger task head DiffAct.
F.2 DETAILS ON FRANKA KITCHEN DATASET.
Here we introduce the details of experiments on the Franka Kitchen Gataset(Gupta et al.) 2019). In this dataset, we adopt 3 tasks, including "Turn the stove top knob (TK)", "Open the microwave (OM)" and "Open the left door (OD)". The goal is to predict 9-DoF joint velocities (7 joints and 2 grippers) based on the visual representations and proprioceptive states (i.e., joint velocities). We follow the MPI mode (Zeng et al. 2024), which trains shallow MLP policy network. For evalua- tion. we follow the R3M method (air et al. 2022) and (Karamcheti et al, 2023) and calculate the average success rates for each setting across the tasks.
20

---

Published as a conference paper at ICLR 2025
Tum on the knob
Open microwave
Open door
Figure 7: Qualitative results the Franka Kitchen dataset. We show the tasks of turning the stovetop knob, opening the microwave and opening the door. All tasks are trained with 25 demonstrations.
We compare our Ego Video with MVP (Radosavovic et al.) 2023). Voltron (Karamcheti et al. 2023) and MPI (Zeng et al. 2024). MVP learns representation for robot manipulation by masked image modeling. Voltron takes step forward combine masked image modeling with vision text align- ment. MPI designs detection and prediction transformers to use object detection signals additional guidance. Notably, these works also use Ego4D as training data.
The experimental results in Table TO indicate that our model significantly outperforms both the MVP and Voltron models by more than 10% in average success rate, and exhibits performance comparable to the more advanced MPI model, which integrates multiple pre-training tasks related to robot learning. When the MPI model is solely trained using contrastive learning and masked signal modeling supervision,
achieve 3.7% improvements in average success rate than MPI model. When MPI incorporates the video prediction task, which has been proven crucial for robot learning, average success rate is only 0.9% lower. This demonstrates the robust generalization capabilities of our model and highlights the contribution of our hand-object dynamics learning scheme to fine- grained hand operations. Figure shows the qualitative results on turning on the knob, and opening the microwave and opening the door tasks.
F QUALITATIVE RESULTS
In Figure we show more examples to compare narrations generated by our HOD with LaViLa Rephraser and the original narrations. We can observe that the narrations generated by our HOD model can well describe the hand-object dynamics (e.g., 'The left hand moves downwards to touch the bicycle tire"). Moreover, compared to the LaViLa rephraser, which often merely changes word order modifies nouns/verbs, our model can combine original actions generate more semanti- cally rich descriptions of actions and scenes, resulting in significantly higher quality narrations. (See the first example: HOD generates 'Person C picks card with their right hand, which is then handed to their left hand.' while LaVa yields '#C C chooses a card/#C selects a card/#C C picks card').
21

---

Published as a conference paper at ICLR 2025
ORIGIN:
#C C picks card.
Our HOD:
LAVILA REPHRA SER: #C
Person C picks a card with their right hand, which is then handed to their left hand.
picks card. #C C chooses card.
#C selects card.
chooses card from the deck.
ORIGIN:
#c presses piano while tapping table.
Our HOD:
LAVILA REPHRASER SER:
The left hand moves up and down on the piano while the right hand taps the table.
#c presses piano while tapping table.
W When tapping the table c presses piano.
ORIGIN: #C C touches the bicycle tire.
Our HOD:
LAVILA REPHRA SER:
The left hand moves downwards to touch the bicycle tire, while the right hand remains still.
#C C is referring to C bicycle tire.
#C C addresses the bicycle tire.
#C mentions the bicycle tire
ORIGIN: #C carries mold. LAVILA REPHRA SER: is the product mold. #C is fan of the mold.
Our HOD:
A person is carrying a mold with their right hand, while theirleft hand is empty.
ORIGIN:
Our HOD:
#C gathers the short bamboo with both hands.
The left hand moves from the left to the top center, gathering short bamboo. The right band assists from the right, helping to collect the bamboo. Together, the hands work to gather the bamboo into compact bundle.
LAVILA REPHRASER
#C C gathers the short. bamboo with both hands.
#C holds the short. bamboo with both hands.
ORIGIN: #C C takes piece of cloth.
Our HOD:
LAVILA REPHRASER:
The left hand moves from the left to the center, while the right hand remains still. The object, piece cloth, is taken by the left hand and brought towards the center of the frame.
#C takes piece of cloth.
is given piece cloth.
piece of cloth taken from bag.
Figure 8: Comparison between: 1) the original Ego4D narrations; 2) LaViLa narrations; and 3) Narrations generated by our HOD. Our narration can describe the dynamic motion information of hands and objects, enhancing the semantic richness of the original narration.
22


Reviews:
[{"review_id": "KoDfuqarqF", "reviewer": "Reviewer_bLce", "summary": "This paper aims to enhance the understanding of complex hand-object interactions in egocentric video-text learning. To accomplish this, the authors introduce a pipeline called HOD, which combines a hand-object detector with a large language model (LLM) to generate high-quality, detailed descriptions of hand-object dynamics. Additionally, they propose a model named EgoVideo, which utilizes a lightweight motion adapter to capture fine-grained hand-object motion information. Through a co-training strategy, EgoVideo effectively and efficiently leverages the detailed hand-object dynamics provided by HOD data. Extensive experiments demonstrate that this approach achieves state-of-the-art performance across various egocentric downstream tasks.", "questions": "Please refer to my weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper has good writing and is easy-to-follow, the idea is straightforward and the experiments shows remarkable improvement. Unlike some previous works who simply rely on extracting additional data of hand-object interaction, this paper provide an effective component (motion adapter) to utilize the data.", "weaknesses": "(1) While the motion adapter proves useful, it is not a novel approach, as similar techniques have been explored in numerous prior works. This does not lower my current rating, but it is the primary reason I have not assigned a score of 8.\n(2) The experiments could be more comprehensive. For instance, it would be insightful to see performance results without any adapter, using only a single branch. Additionally, I am interested in the model's performance on more challenging tasks, such as robotic manipulation. Providing such additional results could positively impact my rating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4aLB3aarPP", "reviewer": "Reviewer_Fnhi", "summary": "The paper introduces a new method for learning from egocentric videos by focusing on hand-object interactions, which are vital for understanding these videos. The authors propose HOD (Hand-Object Dynamics), a system that generates detailed video captions using hand-object detectors and large language models. To leverage this enriched data, they present EgoVideo, a model with a lightweight motion adapter and a co-training strategy that effectively captures hand-object dynamics. EgoVideo outperforms other models in various egocentric video tasks and adapts well to applications like robot manipulation. Because my research field is somewhat distant from this article, I can only give a relatively general judgment. For more detailed judgments, please refer to the analysis of other reviewers.", "questions": "Please see weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The data generation pipeline creates high-quality video-language pairs, enhanced with detailed descriptions of hand-object interactions, enabling effective pretraining for video-language tasks.\n2. The proposed methods demonstrate promise for real-world applications in augmented reality, robotic manipulation, and personalized assistants by achieving exceptional results in egocentric tasks.", "weaknesses": "1. The author should be more careful in writing. One of the most obvious mistakes is that the title of Figure 3 is wrong. Here, the author uses the title in Figure 6.\n2. When the author obtains a better language description, they should also present the underlying semantic rules to clarify its rationale. Furthermore, evaluating the description's quality is essential to ensure standardization and scientific rigor.\n3. I think the author could do some further qualitative analysis, starting with the good cases and bad cases. These examples could be considered to be put in the main text instead of the appendix to help readers better understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RAd4oFmeKy", "reviewer": "Reviewer_xrMB", "summary": "The paper presents a new method for egocentric video representation learning that integrates fine-grained hand-object dynamics. It introduces the HOD data pipeline for generating detailed captions and the EgoVideo model, which uses a lightweight motion adapter for efficient learning. The approach achieves state-of-the-art performance across various tasks.", "questions": "1. What is the inference process of the proposed method? If it still relies on both the hand-object detector and LLM as in training, will it significantly increase inference time?\n2. The HOD framework uses the original caption as input; what should be done if the dataset lacks corresponding annotation information?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The HOD pipeline effectively enriches video annotations with hand-object dynamics, capturing interactions more realistically than prior approaches.\n2. By using a lightweight motion adapter and co-training strategy, EgoVideo is effective in capturing detailed movements.", "weaknesses": "1. Adding a figure to introduce the “Motion Adapter” would make it more intuitive.\n2. The proposed method lacks comparison with other methods in terms of parameter and computational complexity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7r7G7hxiDR", "reviewer": "Reviewer_PuQN", "summary": "The authors introduce a new video-language model that learns video representations from fine-grained hand-object interaction descriptions. Unlike previous works that learn from abstract short texts with a fixed number of frames, they propose HOD and EgoVideo. HOD generates detailed descriptions by prompting a large language model (LLM) using noisy HOI bounding boxes, while EgoVideo incorporates a dual-branch video architecture that adopts a backbone and a lightweight adapter to handle inputs with different frame sampling rates. The authors conduct comprehensive experiments across various benchmarks to demonstrate the strength of their pretrained model.", "questions": "- **Lack of Citation**: The key idea in EgoVideo is closely related to the well-known SlowFast Network [SlowFastNetworks for Video Recognition](https://arxiv.org/abs/1812.03982) (ICCV 2019), but the authors do not cite it. Although their implementation ways are different, I recommend including this citation and adding a proper discuss.\n- **Lack of Parameter Comparisons in Table 1**: In addition to dataset sizes, I suggest adding information on trainable parameters, as models like Helping Hands have significantly fewer trainable parameters than other methods.\n- **Improper Comparison of EgoVideo-G**: One problem is the unfair comparisons involving EgoVideo-G with other models. While the Giant model demonstrates impressive performance, it lacks a fair comparison with other models that primarily use smaller variants. Besides, other backbones mainly leverage CLIP for initialization, whereas EgoVideo-G is initialized with the state-of-the-art InterVideo2 (L313). This not only means a larger model size but also a different initialization method. There is a lack of a straightforward ablation study or adequate explanation the choices.\n- **Improper Comparison in Robot Manipulation**: While I appreciate the emphasis on robot manipulation, the comparison is unfair, as EgoVideo-G (2B) is significantly larger than other methods (e.g., MPI-Base, which is only 86M). The authors should include control speed and parameter sizes, as EgoVideo-G is likely slower and larger than the other methods, which impacts the evaluation.\n- **Input to the Manipulation Model**: During the manipulation process, EgoVideo relies on multiple frames as input, whereas methods like MPI use only a single observed frame. I'm curious about how the authors address this difference.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Good Paper Writing**: The paper is well-organized, clear, and easy to follow.\n- **Useful Architecture for Egocentric Video Understanding**: The architecture effectively improves performance in the egocentric domain, though the core idea of learning from varying frame sampling rates originates from SlowFast. The proposal of using the lightweight adapter increase only a few parameters.\n- **Extensive Experiments and SoTA Performance**: The proposed EgoVideo-G (their largest model) achieves state-of-the-art performance across all benchmarks. The experiments cover a wide range of datasets and domains, including commonly used evaluation datasets in egocentric contexts like Epic-Kitchens-MIR and EgoMCQ, as well as evaluations in robotic manipulation. I appreciate the additional focus on robotic applications.", "weaknesses": "- **Lack of Analyses on HOD Data Corpus or Quality**: This paper lacks thorough analyses of the data corpus, especially considering the data generation pipeline is simple and naive. From the motivation of HOD, the authors claim to create 'HOI dynamics' using an HOI detector. However, since the HOI detector is image-based, there is no guarantee that the generated texts will capture temporal information. While I acknowledge the robustness of the HOI detector, I am unsure if this robustness ensures consistent temporal information and how much noise would impact the final model. From the data corpus, the authors do not analyze the data statistics of the generated texts at any perspective except for the experimental results. As a result, I have limited insight into the quality of the pretraining dataset. To illustrate the significance, take what LaViLa has done as an example. To demonstrate the effectiveness of generated sentences, LaViLa's Narrator evaluates their model on the validation set of Ego4D. I think similar experiments are essential to validate the usefulness of HOD. For instance, it is important to determine whether the original information is remained in the generated text and whether there are unreliable hallucinations present from Yi-34B.\n- **Inadequate Analyses at Inference Time**: While mainstream video-language models use 4 frames during inference, EgoVideo requires sampling both 4 and 16 frames, leading to slower inference speeds and also potential unfair comparisons. Although Table 4 demonstrates the effectiveness of training EgoVideo jointly with 4 and 16 frames, another straightforward baseline of other models incorporating test-time augmentation, e.g., simply aggregating outputs of AVION-4f from both 4 and 16 frames, would enhance the evaluation. Besides, I think the comparison on GFLOPS during inference between models should also be provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)