Do Egocentric Video-Language Models Truly Understand Hand-Object Interactions?
Authors: Boshen Xu
Ziheng Wang
Yang Du
Zhinan Song
Sipeng Zheng
Qin Jin
Venue: ICLR 2025 Poster (2025)
Keywords: egocentric
hand-object interaction
video-language model
contrastive learning
multimodal representation learning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We construct EgoHOIBench to reveal that EgoVLMs struggle to distinguish HOI combinations with word variations. To address this, we introduce a new contrastive loss, EgoNCE++, which improves the performance of EgoVLMs across multiple downstream tasks.

Abstract: Egocentric video-language pretraining is a crucial step in advancing the understanding of hand-object interactions in first-person scenarios. Despite successes on existing testbeds, we find that current EgoVLMs can be easily misled by simple modifications, such as changing the verbs or nouns in interaction descriptions, with models struggling to distinguish between these changes. This raises the question: "Do EgoVLMs truly understand hand-object interactions?'' To address this question, we introduce a benchmark called $\textbf{EgoHOIBench}$, revealing the performance limitation of current egocentric models when confronted with such challenges. We attribute this performance gap to insufficient fine-grained supervision and the greater difficulty EgoVLMs experience in recognizing verbs compared to nouns. To tackle these issues, we propose a novel asymmetric contrastive objective named $\textbf{EgoNCE++}$. For the video-to-text objective, we enhance text supervision by generating negative captions using large language models or leveraging pretrained vocabulary for HOI-related word substitutions. For the text-to-video objective, we focus on preserving an object-centric feature space that clusters video representations based on shared nouns. Extensive experiments demonstrate that EgoNCE++ significantly enhances EgoHOI understanding, leading to improved performance across various EgoVLMs in tasks such as multi-instance retrieval, action recognition, and temporal understanding. Our code is available at https://github.com/xuboshen/EgoNCEpp.

Published as a conference paper at ICLR 2025
Do EGOCENTRIC VIDEO-LANGUAG MODELS TRULY UNDERSTAND HAND-OBJECT INTERACTIONS?
Boshen Xu'
Ziheng Wang
Yang Du
Zhinan Song'
Sipeng Zheng² Qin Jin
*Renmin University of China
2 Beijing Academy of Artificial Intelligence
ABSTRACT
Egocentric video-language pretraining is a crucial step in advancing the under- standing of hand-object interactions in first-person scenarios. Despite successes existing testbeds, we find that current EL can be easily misled by simple modifications, such changing the verbs nouns in interaction descriptions, with models struggling to distinguish between these changes. This raises the question: "Do VLS truly understand hand-object interactions?" To address this ques- tion, we introduce a benchmark called EgoHOlBench, revealing the performance limitation of current egocentric models when confronted with such challenges. We attribute this performance gap to insufficient fine-grained supervision and the greater difficulty EgoVLMs experience recognizing verbs compared to nouns. To tackle these issues, we propose novel asymmetric contrastive objective named EgoNCE++. For the video-to-text objective, we enhance text supervi- sion by generating negative captions using large language models leveraging pretrained vocabulary for HOI-related word substitutions. For the text-to-vides objective, we focus on preserving an object-centric feature space that clusters video representations based on shared nouns. Extensive experiments demonstrate that EgoNCE++ significantly enhances EgoHOI understanding, leading to im- proved performance across various Ego in tasks such as multi-instance retrieval, action recognition, and temporal understanding. Our code available https: : / thub. com http///xoooooe/oo....ep..
INTRODUCTION
Humans have long envisioned embodied agents that can perform various societal roles. promising approach to realizing this vision involves leveraging knowledge from egocentric demonstrations to train agents to imitate human actions during daily activities. Egocentric videos, captured from first-person view using wearable devices, effectively showcase how individuals interact with nearby objects using their hands. This has sparked significant interest understanding egocentric vidco. particularly hand-object interactions, due to its potential applications in VR/AR GGauman et al.
2024a Plizzari et al. 2024) and embodied agents (Zeng et al. 2023 Zheng et al. 2023).
Recent works (Lin et al. 2022 have utilized the large-scale dataset Ego4D Grauman al. 2022
to pretrain egocentric languaae models (LgoVLMs), enhancing performance in tasks, such as egocentric video-text retrieval (Lin et al.] 2022. Sigurdsson al. (018b) and action recogni- tion (Sigurdsson et al.] 2018a). However, despite the impressive capabilities of these models and their benefit from large-scale pretraining critical issue in video-text matching: when tasked with selecting the correct sentence for a video from the sentences where the verb or noun varies significantly in meaning, Ego VLMs often fail make accurate distinctions, as shown in Figure [ This raises an important question: Do existing ELo truly understand egocentric hand-object interactions?
hand-object interaction (HOI) data, we have identified
To delve deeper into this question, we introduce EgoHOlBench, novel multi-choice testbed derived from Ego4D. This benchmark specifically designed to assess the ability of VLM to comprehend
"Equal contribution. "orresponning author.

---

Published as a conference paper at ICLR 2025
Natural Language Query
Simple Multi-choice Test
Similarity
hangs the frying pan
Where was doing sth?
C whispers the frying
C writes the frying pan
Action Recognition
the frying pan
5. paints the frying pan
GT: C shakes the frying pan
EgoVLM
HOI-Noun
What am doing?
Video-Text Similarity
drops paper
C. drops the cork
Video Text Retrieval
3.C drops the dough
grab plates
drops spice
C. drops the pan
Which are the most relevant?
GT: drops the dough
A Although EgoVLMS have been pretrained on millions of worldwide egocentric videos and applied to challenging downstream tasks like video-text retrieval, we observe that they often fail to select the matched sentence from the simplest sun-ttubttt candidates for videos.
Figure I:
HOI combinations with verbs nouns varies through video-text matching. After evaluating state- of-the-art VLMS on EgoHOlBench, we were surprised to observe a substantial decline in performance. Despite being trained extensive EgoHOI data, these models still exhibit difficulty in accurately recognizing HOIs when confronted with even the most basic word substitutions.
We attribute this suboptimal performance of these models primarily to lack of fine- negative supervision during the pretraining process. In egocentric lideu-language pretraining, which employs video-to-text and text-to-video contrastive losses (i.e., EgoNCE (Lin al. 2022) and InfoNCE (Zhao et al. 2023)). training batches often contain many easy negative samples (e.g. "cut grass" vs. "pick apple") While these negatives facilitate model generalization across HOI sentences with simultaneous verb-noun changes, they fail to provide effective supervision for understanding the nuances of HOI combinations, such as distinguishing between "shakes the frying pan" and "hangs the frying pan". Consequently, the models exhibit fragile robustness when evaluated on EgoHOlBench. One potential solution is to enhance fine- grainee supervision through hard negative mining (Yuksekgonul et al. 2023). We therefore propose generating hard negatives that differ by single word, HOl-related noun or verb. However, this approach carries risks: it may disrupt the understanding of other unchanged words, potentially reducing performance and limiting model generalization other tasks (Momeni et al. 2023). Thus, a carefully designed training strategy is essential to address these challenges.
Furthermore, EgoVL.Ms VLMS demonstrate stronger robustness towards recognizing nouns through analysis of EgoHOlBench performance on HOI-verbs and HOI-nouns. By visualizing video repre- sentations in a oow-dinensional space, we reveal that these VLMs develop object-centric feature spaces, where representations with the same nouns are more robustly encoded and clustered than HOI-nouns,
those with the same verbs. This phenomenon, leading improved performance can be viewed as advantageous, as previous studies have shown the effectiveness of establishing object-centric features through additional structures trained on object images (Escorcia al. 2022
or supervision from HOI detection tasks (Zhang et al. 2023 Li et al. 2021b).
In this work, we aim to preserve the object-centric nature of the feature space, without requiring additional visual data or architectural changes, while simultaneously enhancing HOIw comprehension, from a contrastive learning perspective. To this end, we introduce EgoNCE++, a novel contrastive learning objective that incorporates asymmetric video-to-text and texi-lo-video losses. Specifically, the video-to-text loss enables the model to capture both word- and sentence-level semantics for each video through hard negative supervision, enhanced by generating HOI-related negative captions using large language models (LLMs) or leveraging vocabulary prior knowledge from the pretraining dataset. Conversely, the text-to-video loss preserves the established object-centric feature space by clustering
2

---

Published as a conference paper at ICLR 2025
video representations with similar nouns in their captions. We conduct extensive experiments across various EgoHOI downstream benchmarks, demonstrating that EgoNCE++ significantly improves the generalization of VLs to other tasks in zero-shot manner.
Our contributions in this work are threefold: (1) We develop EgoHOlBench, a novel benchmark specifically designed to evaluate EgoVLMs' capabilities in understanding variations of HOI com- bination. (2) We propose EgoNCE++, an innovative HOl-aware asymmetric contrastive learning objective for egocentric video-language pretraining. (3) Our experimental results demonstrate the versatility and efficacy of EgoNCE++, notably enhancing performance across three EgoVLMs and improving generalization on seven downstream EgoHOI tasks.
EGOHOIBENCH: Do EGOVLMS TRULY UNDERSTAND HOIS?
Existing benchmarks in egocentric vision have primarily focused on EgoHOI. For instance, Damen al. (202T) emphasize action recognition kitchen scenarios. which limits its ability evaluate the broader knowledge embedded in VLMs. Wang et al. (2023b) propose assessing model's 's temporal distinguish between actions with similar semantics. [Lin al. (2022) suggest querying the correct video from multiple options across various scenarios based provided text descriptions. contrast to these testbeds, EgoHOlBench introduces a straightforward multi- choice test for video-to-text matching, featuring comprehensive real-world scenarios and a rich, diverse vocabulary centered on hand-object interactions (HOIs). This benchmark is designed to evaluate the ability of EgoVLMs to select the correct sentence from multiple HOI-related options using video-text matching.
understanding by requiring
2.1 NEW BENCHMARK FOR NUANCED EGOHOI DISTINCTION
To clarify the task definition, we design each EgoHOI multi-choice trial as follows: given video segment x, the model is required to distinguish the correct caption S* from N verb-focused hard negative cap- tions {S: where S; and S* differ only in the verb (HOI-verb task). Similarly, the model must iden- tify S" from N noun-focused hard negative captions 1.llj-=Wherr S, is generated by replacing the noun in S* with alternative nouns (HOI-noun task), A trial considered successful only when the model accu- rately identifies the correct caption S for both the HOI-verb and HOI-noun tasks.
serlect
EgovLP
EgoVLPv2
0.8
LaViLa
Accuracy
0.4
We contend that ideal EgoVLM should excel solving these relatively straightforward tasks, pro- vided that the choice options are not deliberately made excessively difficult. Therefore, leveraging LLMs generate such less challenging options is a practical approach, given their robust world knowl- edge and cerruccioo-folloiinn capabilities. This method provides a more efficient and scalable alternative to relying on human labor. The LLM is prompted to perform word substitutions, randomly replacing HOI-related verbs or nouns with alternatives of different meanings. This process transforms the multi-choice candidates into alternative sentences, significantly altering their semanties. Ulti mately, EgoHOIBench provides a comprehensive evaluation of models' understanding of EgoHOIs across 29K test trials. More details the data construction process available in Appendix B
HOl-action
HOIverb
HOI-noun
Figure 2: EgoHOI performance of VLMs EgoHOlBench.
We evaluate three state-of -hee---ftfhhh-rr-- 2023), and LaViLa (Zhao et al. 2002). Surprisingly, all models perform poorly on EgoHOlBench, as illustrated in Fig. 2 To better understand the underlying reasons for this suboptimal performance, we focus on addressing two key questions:
the-art art VLMs: Ego (Lin et al. 2022), EgoVLPy2 [Pramanick et al.
Why EgoVLMs struggle with the seemingly simple multi-choice test? (Section 2.2
Why performance HOI-noun is better than that HOI-verb? (Section 2.3
3

---

Published as a conference paper at ICLR 2025
2.2 LIMITATIONS OF EXISTING EGOVLP OBJECTIVE
Egocentric video-language pretraining (VLP) follows the standard VLP paradigm, which utilizes a dual-encoder architecture to perform contrastive learning between video and text modalities. Current EgoVLPs consider two symmetric contrastive learning objectives: InfoNCE (Zhao et al. 2023) and
EgoNCE (Iin et al. 2022 Pramanick et al., 2023; Zhang et al. 2023
Phan et al. 2024).
InfoNCE (Radford et al. 2021). InfoNCE is widely used contrastive learning objective that encourages positive video-text pairs closer while pushing negative pairs further apart through an online cross-entropy loss. The symmetric InfoNCE loss, applied to batch of (video, caption)
samples, can be formulated as:
log E E.) exp(1; tz/T) + Σ log exp(l: + w2/T) E.yeB()) exp(t; U//T)
exp(e: 14/T)
Cinfo
(1)
B
v)EB(v)
(EB(()
where (Rt (4) denotes the L2 normalized feature vectors of the i-th (video, caption) sample within
a batch. ((x) 1) = {w and B(1) = (4:) refer to the videos and captions of the batch
{(U,, (= respectvely
EgoNCE (Lin et al. 2022). It is specifically tailored for egocentric scenarios. As shown in the following video-to-text loss, EgoNCE enhances the learning of subtle differences in scenes by enlarging the batch to include additional video clips with visually similar backgrounds. It also expands positive video-text pairs by including texts that depict similar HOIs occurring in different contexts. The video-to-text loss is defined as:
1
EnEP(L) exp(e; tx)
C
Σ
log ELEBO exp(e; b2) E, Et,,E8t)) exp(1; tp)
2B
B(o)UBBB)(()
where B = {(vy, tv ) B(o) = {uy ),=_ and B(t) = {tv W represent the enlarged batch samples. Each (uy, ) corresponds to the (video, caption) pair sourced from the same recording environments as the i-th video clip in the original batch. Furthermore, P(t,) S B(t) U B(t) defines the set of captions, each containing at least one verb or noun that matches those in 4.- To save space, we omit displaying the text-to-video loss as it is symmetrically formulated.
Lack of Fine-Grained Text Supervision for HOI-action. While EgoVLMs pretrained with InfoNCE and EgoNCE have gained substantial knowledge about EgoHOI, they lack fine-grained text supervision. Specifically, InfoNCE often samples easy negative pairs (e.g., 'opens drawer" vs. "picks an egg") without employing effective hard negative mining for text. Additionally, EgoNCE's positive sampling expansion strategy treats pairs like "opens a drawer" and "closes drawer". or "opens a drawer" and "opens a bottle". as positive pairs, which weakens the model's understanding of fine-grained HOIs. As result, these objectives often distinguish EgoHOIs based simultaneous verb-noun variation, ignoring the need to learn the true semantics of HOI combinations. While these pretraining objectives are proven be effective, we believe that more generalizable Ego should be capable of recognizing word-level variations in sentences.
2.3
OBSERVATION OF OBJECT-CENTRO FEATURE SPACE IN EGOVLMS
VLMs Establish an Object-Centric Feature Space for HOI-noun. While performance on the HOI-noun task shows room for improvement, results are even lower on the HOI-verb task. Recognizing complex actions and temporal dynamics is generally more challenging than identifying static objects (Damen et al. 202T). We hypothesize that this pattern extends to video-text matching. where matching video to the correct verb is more difficult than matching it to the appropriate noun.
To test this hypothesis, we visualize egocentric video and text embeddings a low-dimensional space to examine their distribution. Specifically, for visualizing verb-anchored text and video embeddings. we select several HOI-related verbs as anchors (e.g.. "pick"). For each anchor verb, we gather 150 text captions that share the same verb but feature different nouns (e.g.. "... pick apples ... pick clothes ..." pick books... ..."). We then visualize the embeddings of these verb-anchored texts and the embeddings of their paired videos. Similarly, we create visualizations for noun anchhordd text and video embeddings. Using t-SNE (van der Maaten & Hinton 2008) for dimensionality reduction, we generate visualizations, shown in Figure 3 focusing on I. ViLa's feature space. After egocentric
4

---

Published as a conference paper at ICLR 2025
preen (reun] Ilonn adpusti [rean]
poed Inount
this [neun]
piaca (neun)
Ivsiol drower
boars
[veet] board
(valifonte
sducts
Iverol kioll
verb-anchored
noun-anchored
verb-anchored
noun-anchored
Text
Video
Figure 3: Visualization of VaViLa's feature space. Both video and text feature space exhibits the object-centric property. Apparently, the videos/texts more separable by nouns, indicating video is more easily matched with the correct noun on HOI-noun tests rather than verbs.
pretraining, the feature space reveals that noun-anchored embeddings form tighter clusters, while verb-anchored embeddings are more dispersed. This suggests that video-noun matching is easier than video-verb matching, which explains the poorer performance on HOI-verb tasks compared to HOI-noun tasks. A similar pattern observed in EgoVLPPs feature space, as detailed Figure T2]
Current Objectives Are Not Tailored for Learning Object-Centree Features. Upon revisiting the InfoNCE and EgoNCE, we observe that the video and text embeddings generated during training tend to cluster around nouns. However, these objectives are not explicitly designed to learn an object-centric feature space. On the other hand, prior research (Zhang et al. 2023 Escorcia et al., 2022, [Zhou et al., (023) has demonstrated the benefits of enhancing object-centric features. Building on this insight, we aim to further strengthen these features through a contrastive learning perspective on the text-to-video side, thereby retaining the advantage for the HOI-noun task.
3 EGONCE++: HOI-AWARE ASYMMETRIC PRETRAINING OBJECTIVE
Building upon the analyses above, our primary goal is to enhance the model's 's sensitivity to word variations that benefits HOI-action recognition, while also reinforeing the object-centric feature space in Ego new contrastive learning objective called EgoNCE++, which incorporates asymmetric video-to-text (Section 3.1) and text-to-video losses (Section 3.2p. The video-to-text loss enables the model to better understand HOI combinations by generating negatives through HOI-related word changes, while the text-to-vided loss preserves object-centric feature properties by clustering video representations based on similar nouns. Figure fillustrates an overview our method.
maintain their advantage in HOI-noun tasks. To achieve this, we introduce
3.1 V2T: HOI-AWARE NEGATIVE GENERATION BY LLM OR VOCABULARY
To build more robust ELM that is sensitive to variations in HOI combinations. focus enhancing the negative text supervision by generating fine-grained hard negatives NN()) through targeted word changes specific verbs nouns. This approach ensures that each video is paired with a fixed set of nuanced hard negatives, while retaining easier negatives in B(t). The generated false HOI combinations (tt) provide more stable and high-quality supervision for understanding true HOI combinations compared to easy negatives in B(t).
We propose two methods for generating these hard negatives: (1) utilizing the vocabulary from the pretraining dataset, or (2) leveraging an LLM when the vocabulary is unavailable. The simplest approach involves substituting HOI-related words from the pretraining vocabulary, encouraging VLM to better capture HOIs within the pretraining data. Specifically, we use spaCy (Honnibal al. 2020) to extract all verbs and nouns in the pretraining dataset. Then, we replace HOI-nouns or HOI-verbs by extracted words randomly. When the pretraining vocabulary is insufficient or unavailable, an LLM can be employed to generate negatives. We prompt an LLM to use the JSON format for response and an in-context learning example to generate sentences that differ semantically from the original text. LLM- -e.--eeeerttedd negatives are not only more fluent and diverse but also more aligned with real- world contexts, owing the LLM's extensive world knowledge. Comparing these two strategies, negatives generated by the vocabulary help develop a better understanding within the pretraining dataset, while the -eneerree negatives may generalize to unseen HOI combinations.
5

---

Published as a conference paper at ICLR 2025
(b) Video-to-Text
EgoNCE++
similarity
paper
C cuts grass
#C burns grass
#C cuts dloth
#C grinds the grain
video encoder
text encoder
grass
closes the pan
video
M HOI Negative Generation O
minimize
Word embedding
(c) Text-to-Video
maximize
similarity
video
C cuts grass
C cuts grass
(a) Architecture
NC waters gresses
#C cuts onion
MC C opens an oven
Figure 4: Illustration of our pretraining framework. (a) VLs trained with EgoNCE++ where the visual encoder is trained using LoRA (Hu et al. 2022) to enhance video representation, while the text encoder remains frozen. Specifically, Ego CE++ consists of (b) V2T: generating HOI-related negative captions for fine-grained supervision, and (c) T2V: strengthening the strong ability of Ego VLMs to recognize nouns by aggregating video features associated with similar nouns.
After generating negative captions with plausible semantics for videos, apply the following supervision loss to improve HOI understanding from the video-to-text perspective:
exp(U; (4)
Lezt B
Σ
log E)ye8(e) exp(e. ty) ENNowa (N) exp(u; tx)
(3)
v(EB(e)
where Neverb (tz) and (t.) denote the verb negatives and noun negatives, respectively. Our video-to-text loss guides videos to correct HOI meanings with supervision from both the coarse- grained easy negatives and fine- Einegrained word substituted negatives.
3.2 T2V: OBJECT-CENTRE POSITIVE VIDEO SAMPLING
Since our V2T negative mining on HOI-verbs might damage the strong recognition of nouns, we aim to maintain the noun clustering nature by T2V positive sampling nouns. The text-to-video loss is designed to preserve the object-centric video features that enhance video-text matching. As discussed in Section 2.3 it is natural reach the solution that can continue to group video representations with the same nouns for given narrations.
To this end, devise object-centric text-to-video loss, where Pnoun (v:) denotes the videos that feature similar nouns in their captions:
Lizv
1 B
log (.E(()
Σ
I exp(le vk) ExjeB(o) exp(li Uz)
Different from EgoNCE which considers videos with either similar verbs nouns positives, argue that only grouping videos with the same nouns is more suitable for learning the object-centric nature of EgoVLMs' feature spaces.
3.3 TRAINING STRATEGY
To refine the video representation of Ego for better generalization, we freeze the text encoder, except for the word embedding to adapt for novel negative sentence distribution, while fine-tuning the visual encoder using LoRA (Hu al. 2022). The dual encoders are trained with both video-to-text loss by fine-grained negative text supervision and text-to-video loss with object-centric positive video sampling. Our final objective comprises the sum of text-to-video and video-to-text losses:
LBgoNCE++
Lizv + Luzs
(5)

---

Published as a conference paper at ICLR 2025
LaViLa
EgoNCE++
EK100-MIR (nDCG)
w/ EgoNCE++
EK100-OV
38-1
EK10D-MIR
EgoVLPv2
EgOVLP
(acc)
320
(mAP)
41.6
8.5
30.8
32-0
EGTEA
EgoHOlBench
(mean acc.)
34.0
30.9
36.9
53.2
(acc)
10.14
79.9
14.67
Bench (acc)
20.9
20.6
(top-1 acc)
EK100-CLS
63.2
CharadesEgo (mAP)
LaViLa
(a) Across Benchmarks
(b) Across EgoVLMs
Figure 5: Overview of experimental results. (a) LaViLa++ that is pretrained on LaViLa using
EgoNCE+ achieves remarkable improvements across benchmarks under zero-shot settings, mean- while (b) EgoNCE++ universally enhances HOI comprehension on EgoHOlBench across ELoSLMs.
EXPERIMENTS
4.1 EXPERIMENTAL. SETTINGS
To ensure the robustness of our approach, we evaluate a range of well-known EgoVLMs including EgoVLP (Lin al. 2022), EgoVLPv2 (Pramanick al. 2023) and LaViLa (Zhao al., 2023).
Details of these models can be found in our Appendix C.I In this paper, we continue to pretrain these models instead of training them from scratch due to computational resource constraints.
Pretraining Dataset and Details. Our pretraining video clips are sourced from EgoClip-3.8N, (in al. 2022), ensuring no overlap with the clips used EgoHOlBench. The dataset focuses on EgoHOls, excluding videos that primarily capture the activities of other persons, resulting in a dataset of 2.5 million entries. The videos are typically about second long, accompanied by captions describing verbs and nouns relevant to hand-object interactions. During pretraining, frames from each video. We employ LoRA tuning with both rank and alpha set to 16. The models continually pretrained for 10 epochs over a period of 12 hours using 8x A800 GPUs, with total batch size 576. We utilize LLaMA3-SB (Al 2024) to generate negative captions for the videos.
sample
Downstream Benchmark and Evaluation Setups. We evaluate model three types of tasks across seven benchmarks in a zero-shot setting: (1) Open-vocabulary recognition: tasks that test video-text matching for andeo-and-langunee
models. We evaluate EgoHOlBench, EK-100- OV (Chatterjee et al. 2024), and ActionBench (Wang et al. 2023b). EgoHOlBench assesses models' sensitivity HOI word changes, EK-100-OV evaluates the recognition of unseen object categories in kitchen scenarios, and ActionBench focuses on temporal understanding in open-wer open-world scenarios. (2)
retrieval: conducted on SpiccKKicchens 100 (Damen et al.) (202)), ditchen-oriente retrieval benchmark where multiple video clips can correspond to the same narration. (3) Action recognition: tested on CharadesFgo (Sigurdsson al., 2018a). FK-110-CILS (Damen al. 2021),
and EGTEA (Li et al.) 2018). CharadesEigo presents an -f--man challenge (I.in et al. 2022
Zhao et al. 2023) for models trained on Ego4D with 157 indoor activity classes. EGTEA requires classifying 106 cooking activities, while EK-100-CLS evaluates 97 verbs and 300 nouns in kitchens.
4.2
MAIN RESULTS
EgoNCE++ Enhances VLo Sensitivity on HOI-related Word Changes. On EgoHOlBench, models are required to comprehend HOI combinations and distinguish specific word changes in sentence candidates. As shown Figure (b), all VLo pretrained with EgoNCE++ exhibit significant improvements, showcasing the versatility of our method across various architectures, training strategies, and loss functions. The notable enhancements primarily arise from improved verb
7

---

Published as a conference paper at ICLR 2025
Table
Comparison with state-of-the-art methods HelpingHands Zhang et al. 2023 and
HENASY (Phan et al. 2024) on zero-shot EK 100-MIR and EGTEA. All these models are built upon La model. The numbers of the method with * are sourced from Phan al. 20244)
Opic-Kitchens- 100-MIR
EGTEA
mAP (%)
nDCG (%)
METHOD
Extra Param
V-+ T
T- V
Avg.
V- T
T-
Avg.
mean-ace
topl-ace
LaViLa
35.1
26.6
30.8
33.7
30.4
32.0
30,9
35,1
HaldingHands*
37M
35.6
26.8
31.2
34.7
31.7
33.2
29.4
35.3
HENASY
112M
35.5
27.1
31.3
34.6
31.7
33.2
29.6
35,9
LaViLa++
35.8
27.9
32.0
31.4
33.1
34.0
Table 2: Comparison of temporal understanding action recognition on ActionBench, where * denotes the fine-tuned model by Wang et al. 2023b).
MODEL
Intern Video
Clip-Vip*
Singularity"
Human
LaViLa
LaViLa++
ACTION ACCURACY
90.1
89.3
83.8
92.0
79.89
91.18
understanding, e.g. +34.02% increase in verb accuracy leading to +26.32% improvement in action accuracy for LaViLa++. Detailed numbers are provided in the Appendix C..3
EgoNCE++ Consistently Benefits EgoVLMs Across Multiple Benchmarks. Taking state- of-the-art EgoVLM LaViLa as an example as shown in Figure 5 (a), EgoNCE++ demonstrates consistent improvements across all benchmarks. From the perspective of video-text aligmment during pretraining, VLs clearly benefit from EgoNCE++ leading to significant gains HOI comprehension, especially reflected on Ego4D benchmarks including EgoHOlBench (+26.32 %), ActionBench (+11.3%). Moreover, EgoNCE++ exhibits strong generalizations across other datasets, showing improvements on EKI00-CLS (+4,53%) EK100-OV (+0.9%), and EGTEA (+3.1 %)
Retrieval and Action Recognition. As shown in Table LaViLa++ remains competitive with state-of-the-art models built upon LaViLa across all metrics for retrieval and action recognition tasks. Specifically, it achieves a notable +1.2% increase in average mAP over ViLa, surpassing models that incorporate additional HOI detection supervision (Zhang et al. 2023) or hierarchical architecture (Phan et al. 2024). It also improves nDCG by +1.1 over la Vil aa, achieving competitive results compared to other models. Furthermore, LaViLa++ demonstrates significant +4.4% boost than other methods in mean accuracy on EGTEA, indicating that EgoNCE++ serves as a promising pretraining objective for ELMS.
La ViLa++ Competes with SoTA Models in Zero-Shot
Ngo Boosts Model Temporal Understanding Capability. ActionBench (Chatterjee et al.) 2024) focuses on temporal understanding tasks, such as distinguishing between "pick up" and "put down". In Table 2 evaluate both La ViLa and LaViLa++ in zero-shot setting. Although do not specifically create negatives for temporal understanding, the results indicate that La LaViLa++ accurately classify verbs by distinguishing them from their antonyms. Our model surpasses the previous best models reported Wang et al. (2023b) and even approaches human-level performance.
4.3 ABLATION STUDY
All ablation studies are conducted by pretraining the EgoVLP model (Lin et al. 2022). More detailed ablation studies can be found in Appendix C.4
Our Asymmetric V2T and T2V Losses Bring Collaborative Enhancement in Performance. As shown in Table 3] our video-to-text supervision ("ours") significantly enhances the Vgo'LM's ability to capture tinengraieed details, achieving +9.8% improvement HOI-action and +0.3% increase in mAP for generalization to EK- 100-000MMI, outperforming both InfoNCE and EgoNCE. Comparing the 3rd and 4th rows, we observe that combining our asymmetric video-to-text loss with text-to-video loss further strengthens the model's 's HOI comprehension, resulting in additional +1.32% improvement in HOI-action on EgoHOlBench and enhanced generalization on EK 100-MIR, with additional +0.3% increase in mAP.
8

---

Published as a conference paper at ICLR 2025
Table 3: Ablation of V2T and T2V losses.
Table 4: Ablation of the negative generator.
V2r
F2V
EgoliOIBench
EK: 100 MIR
reganec
GENERATOR
verb
EgolIOlBench noun
action
aygmAA
100
aggaDCG
InfoNCE
40.70
nome
40.27
68.60
30.16
22.2
26.7
InfoNCE
EgoNCE++
40.60
09:15
30.62
22.3
25-7
rulc-based
43.52
68.94
32.63
22.1
26.7
EgoNCE-4
InfoNCR
54.56
68.96
40:31
22.4
26.9
hased
68.56
BguNCE++
RgoNCE++
56.11
69.05
41.63
22.7
27.1
LLM--assd
56.11
69.05
41.63
22.7
27.1
133-
v_neg
runeg
600
nuneg
nuneg
nunog
frequenccy
543
Frequency
203
24 video text similanty value
video-teck similenty value value
video text similanty value
video text similarity value
LaViLa
LaviLat+
EgovLP
EgoVLP++
Figure 6: Histogram of video-text similarities for EgoVLP and LaViLa on EgoHOlBench. After applying EgoNCE++, the video-verb negatives are especially suppressed and thus the video-positives are more distinguished. LaViLa that is pretrained with InfoNCE benefits more from EgoNCE++ than Ego that is pretrained with EgoNCE.
Our Hard Negative Generation Performs Better than Rule-Based Generation. We compare LLM-based and socab-based hard negative generation methods with the rule-based method: (1) LLM- where LLM performs word substitutions through in-context learning; (2) besab, where HOI verbs are replaced with bitrary verbs from the predefined Ego4D vocabulary con- taining thousands of words. (3) rule-based, where hard negatives are selected by choosing captions with the high- est BLEU (Papineni et al., 2002) scores from the sentences; As shown in Table both the LLM-based and vocab-basee methods result in significant improvements, surpassing the rule-based method by at least +7.44% HOI-action and +0.4% on EK 100-MIR in mAP. The rule-based method provides negatives that have already been encountered during pretraining, making less effective. While the vocab-based method occasionally generates meaningless HOI combinations, the LLLM-based method produces more effective hard negatives, resulting in slightly better performance. The trade-off between the efficiency and efficacy of the vocab-based and LLM-based methods is demonstrated in Appendix C.5
23.6
EgoVLP
LaViLa
EgoVLP mAP
23.1
LaViLa mAp
22.6
31.1
22.1
30.4
10
64
Number of negatives
Figure 7: Scaling effect of negative num- ber on EK-I00-MIR (mAP).
Performance Improves the Negative Number Increases. Figure 7 illustrates the trend in mAP for 1K-1000MIR the number of negative samples increases. A clear correlation is observed: with more negatives leading improved performance across various VgM, such EgoVLP and La More negatives during pretraining significantly enhance the distinguishability of true video-HOI matches from other false HOIs, contributing to better performance.
4.4 FURTHER ANALYSIS
Histogram of Video- Tixe Similarity on EgoHh.eench. To examine how video-text similarities are changed by EgoNCE++, visualize histograms of video-text similarities Ego and LaLiLa in Figure 6 Video-positives roughly remain a high range of similarity, while the negativess suppressed lower after applying EgoNCE++. We note that the LaViLa pretrained on InfoNCE benefits more from EgoNCE++ than EgoVLP pretrained on EgoNCE.
Qualitative results. For detailed qualitative results, please refer to Appendix

---

Published as a conference paper at ICLR 2025
5 RELATED WORK
Egocentric Hand-Object Interaction. Captured by head-mounted cameras, egocentric hand-
object interaction (EgoHOI) (Grauman et al. 2022 Chatterjee et al. 2024 Xue & Grauman 2024 Plizzari et al. 2023 Zhang et al. 2022 Mangalam et al., 2024) provides insight into how humans interact with objects from a first- person view. To address this task, Huang et al. (2018) and Kazakos et al. (2021) focus on recognizing close-set EgoHOls using additional mu Itimodal cues (e.g., gaze, sound), while Wang et al. 2023a) adopt a self-supervised approach (He et al. 2022) to exploit visual information. Considering the abundant resources of third-person data, some works (Li et al. 202 Xu et al., 2023) aim to transfer view-agnostic knowledge from third-person videos to egocentric viewpoints. However, the unpredictable nature of open-world environments poses new challenges, requiring models to handle a variety of unseen concepts. Recent studies (Chatterjee et al. 2024 Wang al.) 2023b) seek to improve the understanding of open- vocabulary EgoHOI, but these efforts are either limited to specific domains like kitchens (Damen et al.) (202T) or laboratories (Sener et al., 2022), or involving easy EgoHOI recognition that is well-solved by current egocentric models. A promising strategy to address these limitations involves egocentric video-language pretraining (Lin)
al.. 2022 Pramanick al. 2023 Zhao al. 2023 Zhang et al. 2023), which learns generalizable
representations by leveraging the Ego4D (Grauman et al. 2022) dataset with over 3,000 hours of footage of daily human interactions. As a pioncering work, VLo (Lin et al. 2022 uses the
EgoNCE loss to treat video-text samples with similar HOIs as positives and visually similar videos as negatives during pretraining. Another method, La ViLa, enhances text supervision by generating diverse positive captions for videos to foster robust contrastive learning through visual-conditioned GPT-2 (Radford et al. 2019) and a T5 (Raffel et al. 2020) rephraser. In this work, we expose the limitation of these Ego VLMs on recognizing HOl-related word variations and address the issue by improving the contrastive loss, which also benefits other downstream tasks.
Hard Negative Mining. Hard negative mining is a pivotal technique (Robinson et al. 2021 Zolfaghari et al. 202T) for refining representations within the visual-language metric space during
contrastive learning. Traditionally, this process pairs positive samples with hard negatives that exhibit
high feature similarity within pretraining datasets (Pramanick et al.] 2023] Bao et al.] [022] Li et al.
20 Xu et al. (021), selecting hard negative from clips recorded in similar environment (Lin ct al. 2022). However, despite current EgoVLMs have adopted either visual- or caature-based nega- tive mining to enhance learning EgoHOI knowledge, they fall short in addressing the straightforward understanding task in EgoHOlBench. Recent innovations have introduced the generative negative sampling strategy using L.M,s, aiming to enhance improve compositional understanding (Yuksek- gonul et al. 2023) in image-VLMss VLMS (Radford et al. 2021 Li et al. 2022 Singh et al., 2022 Zeng et al. 2022), and action comprehension (Momeni et al. [023] Bansal al. 2024) in video- VLMS (Luo al. (2022. For instance, ViA (Momeni et al. 2023) proposes verb-focused pretraining framework that creates negative captions of sentences and verb phrases using an LLM. However, most existing approaches are tailored for pretraining third-person datasets like MiT (Monfort et al. (2199) and applied to simple scenarios like Kinetics-400 (Carreira & Zisserman 2017), leaving the complex egocentric domain underexplored. In this paper, our proposed learning objective, EgoNCE++, incor- porates generative negative mining into the pretraining process, using either the powerful LLM or the more efficient vocabulary from the pretraining set facilitate more robust VLO..
CONCLUSION
In this work, we introduce EgoHOlBench, straightforward test designed assess Ego comprehension of HOI combinations, highlighting the current limitations of these models in under- standing hand-object activities. We identify the underlying issues, including lack of fine-grained negative text supervision and the object-centric feature space that favors HOI-noun recognition but adversely impacts HOI-verb recognition. Building upon these analyses, we propose an asymmetric learning objective called EgoNCE++ which enhances the video-to-text loss by incorporating gen- erated dense hard negatives, and a text-to-video loss that focuses on grouping videos with similar nouns. Through extensive experimental analyses across diverse benchmarks, we demonstrate that our proposed VLP training framework can effectively equip different Ego VLMs with greater robustness to HOI combinations and benefit various downstream EgoHOI tasks.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was partially supported by the Beijing Natural Science Foundation (No. L.233008) and the Outstanding Innovative Talents Cultivation Funded Programs 2024 of Renmin University of China.
REFERENCES
Meta AL Meta Ilama 3. http:://eeittu...oonnntteeaaaaaa
2024.
Hritik Bansal, Yonatan Bitton, Idan Szpektor, Kai-Wei Chang, and Aditya Grover. Videocon: Robust video-language alignment via contrast captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Hangbo Bao, Wenhui Wang. Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. VLMo: Unified lision-language pre-trat pre-training with mixture-of- modality-expertts In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL /estps://openreveee net. d=bydks
João Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4724- 4733, 2017. hnttps/:://
semant ccscholaa ar hwwptticcccccoo.ooooooooooooooo Cwwppoo////oo::: 2D:35991121
Dibyadip Chatterjee, Fadime Sener, Shugao Ma, and Angela Yao. Opening the vocabulary of egocentric actions. Advances in Neural Information Processing Systems, 36, 2024.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. The epic-kitchens dataset: Collection, challenges and baselines. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 43 1):412555244 4141, 2021. doi: 10.1 2..1109//PMM..22202222 1965.
DeepSeck-AL. Deepseek Ilm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024. URL/attss:// http:/gttuhbbb com deepseek ai/ ai/DeepSeek LLM
Zi-Yi Dou, Xitong Yang. Tushar Nagarajan, Huiyu Wang, Jing Huang, Nanyun Peng, Kris Kitani, and Fu-Jen Chu. Unlocking exocentric video- languae data for egocentric video representation learning. arXiv preprint arXiv:2408.00 03567,, 2024.
Victor Escorcia, Ricardo Guerrero, Xiatian Zhu, and Brais Martínez. Sos! self-supervised learning over sets of handled objects in egocentric action recognition. In European Conference on Computer Vision, 2022. WtRL:ttts::///
/etttps://api... www.sseeaiiiioom www.ttccccooollrr.....occccc.aae httpiooll...rgg//oorss..-.-.-.:.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18995-19012, 2022.
Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024a.
Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19383-19400, 2024b.
Kaiming Hc, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spacy: Industrial- strength natural language processing in python. 2020. doi: 10.5281/zenodo. 1212303.
11

---

Published as a conference paper at ICLR 2025
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. International Conference Learning Representations, 2022. BtLL:tts:://// http://://.eeeeeeeeeeteeeeeeeeee
iddnZeeKeeFFYf)
Yifei Huang, Minjie Cai, Zhenqiang Li, and Yoichi Sato. Predicting gaze in egocentric video by learning daskddependent attention transition. In Proceedings of the European conference on computer vision (ECCV). pp. 754-769, 2018.
Evangelos Kazakos, Jaesung Huh, Arsha Nagrani, Andrew Zisserman, and Dima Damen. With a little help from my temporal context: Multimodal egocentric action recognition. In British Machine Vision Conference (BMVC), 2021.
Jie Lei, Tamara Berg, and Mohit Bansal. Revealing single frame bias for video-and-language learning. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 6 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 487-507, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10. 10.18653/v1/2000 acl-long.29 29, URL htt.ttts//aeeattooooooo wttp:///w.aaaa......ooooro////// 202.:/222.... ac1-loncclonngn 29
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694 99:55,4-7705, 2021a.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International conference on machine learming, pp. 12888-12900 PMLR, 2022.
Yanghao Li, Tushar Nagarajan, Bo Xiong, and Kristen Grauman. Ego-exo: Transferring visual repre- sentations from third-person to first-person videos. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 6943-6953, 2021b.
Yin Li, Miao Liu, and James M Rehg. In the eye of beholder: Joint learning of gaze and actions in first person video. In Proceedings of the European conference on computer vision (ECCV), pp. 619-635, 2018.
Kevin Qinghong Lin, Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Xu, Difei Gao, Kong-Cheng Tu, Wenzhe Zhao, Weijie Kong, et al. Egocentric rideo-language pretraining. Advances in Neural Information Processing Systems, 35:7575 77866, 2022.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1997..11999 2019. URL htt.h::////// /app. semant aeeeannicschooi iescholar wwwp:erroooooooooooooo 198953378
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing, 508: 293-304, 2022.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36, 2024.
Liliane Momeni, Mathilde Caron, Arsha Nagrani, Andrew Zisserman, and Cordelia Schmid. Verbs in action: Improving verb understanding in video-language models. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 15579-15591, 2023.
Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfreund, Carl Vondrick, et al. Moments in time dataset: one million videos for event understanding. IEEE transactions pattern analysis and machine intelligence, 5222):502-508, 2019.
OpenAl. Gpt-4o: A specialized variant of gpt-4, 2024. URL htttt:t//pppeeaaaee com com Accessed November, 2024.
12

---

Published as a conference paper at ICLR 2025
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evalu- ation of machine translation. In Anmual Meeting of the Association for Computational Linguistics, 2002.
ozg/CorpusI::1100773
Baoqi Pei, Guo Chen, Jilan Xu, Yuping He, Yicheng Liu, Kanghua Pan, Yifei Huang, Yali Wang, Tong Lu, Limin Wang, et al. Egovideo: Exploring egocentric foundation model and downstream 2024.
adaptation. arXiv preprint arXiv:2406.1880
Khoa Vo Thinh Phan, Kashu Yamazaki Minh Tran, and Ngan Le. Henasy: Learning to assemble scene- entities for interpretable egocentric video-language model. In Advances in Neural Information Processing Systems, 2024.
Chiara Plizzari, Toby Perrett, Barbara Caputo, and Dima Damen. What can cook in italy teach mechanic in india? action recognition generalisation over scenarios and locations. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 13656-13666, 2023.
Chiara Plizzari. Gabriele Goletto, Antonino Fumaar,, Siddhant Bansal, Francesco Ragusa, Gio- vanni Maria Farinella, Dima Damen, and Tatiana Tommasi. An outlook into the future of egocentric vision. International Journal of Computer Vixion, pp. 1-57, 2024.
Shraman Pramanick, Yale Song, Sayan Nag. Kevin Qinghong Lin, Hardik Shah, Mike Zheng Shou, Rama Chellappa, and Pengchuan Zhang. Egovlpv2: Egocentric video-language pre-training with fusion in the backbone. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5285-5297, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140): 2020.
Joshua David Robinson, Ching- Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. In International Conference on Learning Representations, 2021. URL
net forum de: XOQOUTTT
Fiona Ryan, Hao Jiang, Abhinav Shukla, James M. Rehg, and Vamsi Krishna Ithapu. Egocen- tric auditory attention localization in conversations. 2023 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pp. 14663-14674, 2023. URL https://appi
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108. 2019. URL https::/ //api
Semanticscholle
/targ//rruuusc ID: : 203626972
Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun Hc, Dipika Singhania, Robert Wang. and Angela Yao. Assembly 101: A large-scale multi- view video dataset for understanding proce- dural activities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21096-21 106, 2022.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 10.18653/11/P11--166 URL https: //ww www. . ac lebb
org/ wwthhlogg/Pl66--ooooo 1162
Dandan Shan, Jiaqi Geng, Michelle Shu. and David F Fouhey. Understanding human hands in contact at internet scale. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9869-9889988 9878.9878. 2020.
13

---

Published as a conference paper at ICLR 2025
Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. Actor and observer: Joint modeling of first and hirr--ersson videos. In proceedings of the IEEE conference computer vision and pattern recognition, pp. 7396-7404, 2018a.
Gunnar A. Sigurdsson, Abhinav Kumar Gupta, Cordelia Schmid, Ali Farhadi, and Alahari Karteek. Actor and observer: Joint modeling of first and third-person videos. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7396-7404, 2018b. URL https:/ /api Semanticschola:s org wttpg///oopsss::::.oo 45621677
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition, pp. 15638-15650, 2022.
Swathikiran Sudhakaran, Sergio Escalera, and Oswald Lanz. Lsta: Long short-term attention for egocentric action recognition. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp.
2019.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Ma- chine Learning Research, 9(86):257--2655 2008. StRp:ttp://// /mmrr htrggpapers/v9/
wandermaatenn0Be htm
Huiyu Wang, Mitesh Kumar Singh, and Lorenzo Torresani. Ego-only: Egocentric action detection without exocentric transferring. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9250-5261, 2023a.
Xiaohan Wang, Linchao Zhu, Heng Wang, and Yi Yang. Interactive prototype learning for egocentric action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8168-8177, 2021.
Zhenhailong Wang. Ansel Blume, Sha Li, Genglin Liu, Jaemin Cho, Zineng Tang. Mohit Bansal, and Heng Ji. Paxion: Patching action knowledge in video-language foundation models. In Thirty-seventh Conference Neural Information Processing Systems, 2023b. URL https //openreview. net. htte///oommmmdmmmmmmeee/eoe
Michael Wray, Diane Larlus, Gabriela Csurka, and Dima Damen. Fine-grained action retrieval through multiple parts-of-speech embeddings. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 450-459, 2019.
Boshen Xu, Sipeng Zheng, and Qin Jin. Pov: Prompt-oriented aiew-agnostic learning for egocentric hand-object interaction in the multi-view world. Proceedings of the 31st ACM International Conference on Multimedia, pp. 2807-2816, 2023.
Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. VideoCLIP: Contrastive pre-training for zero-shot video-text understanding. In Marie- Francine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen- tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6787-6800, Online and Punta Cana, Dominican Republic, November 2021. https
Association for Computational Linguistics. doi: 10.18633/112021, 10111866////2211.mmmmmm........
//aclanthology. wwwoorgg//... 202l. emnlp main. 544
Zihui Sherry Xue and Kristen Grauman. Learning fine-graine tne-grained view-invariant representations from unpaired ego-exo videos via temporal alignment. Advances in Neural Information Processing Systems, 36, 2024.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words and what to do about it? In International Conference on Learning Representations, 2023. URL
net forum?
lid KRLUUxUxx.auaX
Andy Zeng, Maria Attarian, brian ichter, Krzysztof Marcin Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveck Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence. Socratic models: Composing zero-shot multimodal reasoning with
14

---

Published as a conference paper at ICLR 2025
language. In The Eleventh International Conference on Learning Representations, 2023. URL
htttps://ppeneeviee net/ 2
Yan Zeng, Xinsong Zhang, and Hang Li. Multi-grained vision language pre-training: Aligning texts with visual concepts. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 25999.26009. PMLR, 2022.
URL /ttps::/ppoceeedngg. wwlpsssmmm...... press/
v1 62 zeng 22c html
Chuhan Zhang, Ankush Gupta, and Andrew Zisserman. Helping hands: An object-aware ego-centric video recognition model. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 13901-13912, 2023.
Lingzhi Zhang, Shenghao Zhou, Simon Stent, and Jianbo Shi. Fine-grained egocentric hand-object segmentation: Dataset, model, and applications. In European Conference Computer Vision, pp. 127-145. Springer, 2022.
Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar. Learning video representations from large language models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 6586-6597, 2023.
Sipeng Zheng, Yicheng Feng, Zongqing Lu, et al. Steve-eye: Equipping Ilm-based embodied agents with visual perception open worlds. In The Twelfth International Conference on Learning Representations, 2023.
Xingyi Zhou, Anurag Arnab, Chen Sun, and Cordelia Schmid. How can objects help action recogni- tion? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2353-2362, 2023.
Mohammadreza Zolfaghari, Yi Zhu, Peter Gehler, and Thomas Brox. Crossclr: Cross-modal contrastive learning for multi-modal video representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4450-1459, 2021.
15

---

Published as a conference paper at ICLR 2025
Discussions
16
More Details of the EgoHOlBench
16
Construction Process
16
B.2
Vocabulary Statistics
17
More Experimental Analysis
17
Implementation Details
17
Benchmark Details
18
Main Results
19
C.3.1
Zero-Shot Setup Evaluation
19
C.3.2
Fine- Tuning Setup Evaluation
20
4
More Ablation Studies
20
More Analyses
22
Qualitative Results
23
A
DISCUSSIONS
Limitations. While EgoNCE++ delivers significant improvements across various fine-grained HOI benchmarks for multiple EgoVLMs, it does have some limitations. First, although hand-object interactions constitute significant portion of egocentric activities, egocentric scenarios encompass broader range of actions, such as simple observation of the environments or VR/AR activities like dancing or playing sports (Grauman et al. 2024b). promising solution to this challenge is to incorporate third-person videos into the pretraining corpus (Dou et al.) 2024) or leverage pretrained models based on third-person video data (Pei et al. 2024). Second, we find challenging to enhance the EgoVLMM object recognition capabilities solely through text supervision. This difficulty likely stems from the broader diversity of object categories compared to action types in the real world, making effective capture challenging with limited number of negative samples. Introducing visual
supervision signals such as bounding boxes may be beneficial (Zhang et al.) 2023 Phan et al. 2024). We plan to address these challenges in future works.
Social Impact. The knowledge of EgoHOls acquired by Ego holds great potential for real- world applications, including embodied agents and VR/AR systems. However, the of egocentric videos raises privacy concerns, as they often capture personal and sensitive information. If not carefully managed, these privacy issues could lead negative consequences. Furthermore, EgoVLMMs are particularly relevant in contexts like kitchen environments, where recognizing dangerous activities critical. Misinterpreting EgoHOIs could result harmful outcomes, such as failing recognize unsafe actions during tasks involving sharp objects. Our research addresses some of these challenges by demonstrating a more robust understanding of HOI actions, providing improved generalization and potentially mitigating these risks.
B MORE DETAILS OF THE EGOHOIBENCH
B. CONSTRUCTION PROCESS
develop EgoHOlBench based on EgoMCQ (Lin et al., 2022), which sources a diverse collection of 39,000 video clips from the validation set of the Ego4D dataset. In our curation process, only keep those EgoHOI clips performed by the camera wearer, excluding clips that record other people's activities, such multi-person interactions (Ryan al. 2023). We achieve this by keeping the captions that begin with '#C' (denoting the wearer) and are followed by HOI-related verbs and nouns, while filtering out any notations related other individuals, such as ##O'. To construct the
16

---

Published as a conference paper at ICLR 2025
verb/noun answer
verb/noun in negative
verb in answer: 366 (44.15%)
noun answer: 1935 (42.47%)
(55.85%)
negative: 2621 (57.53%)
Figure 8: Illustration of the vocabulary statistics of EgoHOlBench.
HOI recognition trials as defined in our task definition above, given video and its ground truth caption, we prompt an LLM to create candidate captions that contain semantically different HOIs from the ground truth. Specifically, candidates through in-context learning. We provide the specific prompts and two exemplary tasks used in this process, along with examples of the final cases in Figure [4] which target generating words with different meanings to make the choices easier. To avoid semantic redundancies and ensure the uniqueness of the hard negative candidates, use the Ego4D dictionary eliminate possible synonyms from the generated captions. Ultimately, EgoHOlBench comprises 29,651 video clips, each accompanied by one ground truth caption, 10 negative captions with verb changes, and 10 negative captions with noun changes. Setting the number of negatives to 10 (i.e. 10 noun negatives, 10 verb negatives) forms 100 HOI negatives, which aligns with the typical action recognition setting.
employ the LLaMA-3-SB (Al 2024) model to generate HOI
B.2 VOCABULARY STATISTICS
The statistics of the vocabulary information are presented in Figure This dataset features rich and diverse vocabulary, including approximately 800 verbs and 4,000 nouns. The options generated by LLMs effectively double the vocabulary size compared to the original correct answers, resulting in extensive combinations of verbs and nouns.
C
MORE EXPERIMENTAL ANALYSIS
For fair comparisons, we have re-implemented experiments in the same environment and under identical settings, without any adjustments to the hyperparameters.
C.1 IMPLEMENTATION DETAILS
As introduced in the main paper, we validate our approach on three EgoVLMs including EgoVLP, its advanced version EgoVLPP2. and LaViLaa EgoVIP is pretrained the EgoCLIP-3388M dataset and employs the EgoNCE loss for optimization. EgoVLPP22 enhances the original model by incorporating cross-attention mechanism between dual encoders and by pretraining additional proxy tasks. LaViLa, on the other hand, is trained on vast dataset of million videos, with 56 million captions generated by visual-conditioned GPT-2 (Radford al. 2019) and further refined using sentence rephraser T5 (Raffel et al. 2020). This extensive training regimen enables LaViLa to improve the generalization of VLMs We present summary of well-nnoown VLP methods in Table 5 Our proposed EgoNCE++ further enhances the EgoHOI understanding capabilities of pretrained Ego models, utilizing only few trainable parameters and novel pretraining objective.
For all models, we adopt the Adam optimizer with parameters 31 0.9 and ,3 0.999. The learning rate follows a cosine annealing schedule, starting at 3e-5 and gradually reducing to 3e-7. During training, we apply standard RandomResiizedCron for data augmentation and employ LoRA tuning to continuously pretrain our VL.. The text encoder for EL is DistilBERT (Sanh et al. 2019), while La ViLa uses CLIP (Radford et al. (021). l). In the case of Ego VLPv2. implement a dual encoder architecture without cross- aooss-tttnttinn fusion, training it exclusively with EgoNCE++ The text encoder for VL. is RoBERTa (Liu et al. 2019). It important note that in the
17

---

Published as a conference paper at ICLR 2025
Table 5: Summary of existing egocentric video-language pretraining methods compared with ours.
METHOO
Pretrain Data
Objective
Negative Mining
Visual Encoder
Text Encoder Train Param
EggVLPP
3.8M
EgoNCE:
video sim
ImageNet ImageNet
BistillBert
172M
Eg0VLPv2
3.8M
EgoNCE-MLM-VTT
feature sim
Roberta
364M
LaViLa
4M
InfoNCE
none
CLIP
CLIP
180M
Ours 2.5M
EgoNCE++
text sim
EgoVIMM
EgOVIM
+3M-43M
text encoder, for VaVaaa we fine-tune the word embeddings, while for the other two Ego the entire text encoder remains frozen. Our experiments show that fine-tuning the word embeddings in VLP and VLPV results in reduced generalization performance across public benchmarks such as EK-100-MIR. The difference may result from the tokenizers, where only La ViLa uses the BPE tokenizer (Sennrich et al. 2016).
We implement all models using their original codebases, with one exception: videos are loaded using the Decord library, as recommended by LaVaLa, instead of using pre-extracted frames via FFmpeg. This may lead to slight numerical differences in the results for pretrained EgoVLP and EgoVLPv2 compared to the figures reported in their original papers.
C.2 BENCHMARK DETAILS
Multi-Instance Retrieval in EK-100-MIR. For the zero-shot setting, we conduct video-text match- ing for retrieval tasks, using 16 frames for evaluation. For the fine-tune setting, we finetune the VLM using the AdamW optimizer. The learning rate is dynamically adjusted from 3e-3 to le-5 using a cosine annealing scheduler that incorporates a linear warmup, starting at 1e-6 for the first epoch. We deploy a total batch size of 128 across 8 GPUs. During both training and inference, 16 frames are sampled from each video.
Action Recognition in EGTEA. For the zero-shot setup, we evaluate mean results across all evaluation splits, as suggested by Li al. (2018), by conducting a video-text retrieval task between video clips and their corresponding action text labels. We prepend the text labels with the prompt "#C C... to standardize the input format. For the fine-tuning setup, leverage the visual encoder and attach an additional linear projection head for the classification purpose, following Kazakos et al. (202T). The models are trained and evaluated on the first split of the validation set. We employ the same optimizer, scheduler, batch size, and frame sampling rate as used in EK- 1K--000MIR. At inference time, we perform three spatial crops of size 224 224 from each 256 256 frame of the video clip, averaging their predictions to form the final prediction.
Action Recognition in CharadesEgo. We treat action recognition as a video-text retrieval task, where video clips are matched with their corresponding action text labels in zero-shot evaluation setting. During inference, we sample 16 frames from each video. Notably, previous works evaluate their models on CharadesFggo using the initial checkpoint due to the domain gap problem, where continued training often leads to performance drops (Lin et al. 2022). In contrast, these studies use their best checkpoint for evaluation on other datasets, such as 10. In our experiments, EgoNCE++ continues pretraining from the best checkpoints instead of starting from the initial checkpoint of Ego to ensure fair comparison and consistent pretraining setting between Ego VLM and Ego LLM+.. Consequently, it is common to observe lower numbers in our paper than those reported in their original papers. Given the consistent improvements offered by EgoNCE+, we were to pretrain from their initial checkpoints, VLMS would still benefit from EgoNCE++
believe that if
Action Recognition in EK E0000-CLS. For the zero-shot setting, organize the task similarly to EgoHOIBench. For verb classification, we append the ground truth noun, while for noun classification, we prepend the ground truth verb. In the linear probing setting, we freeze the visual encoder and add a linear layer to map the feature embeddings to the predefined classes.
18

---

Published as a conference paper at ICLR 2025
Table 6: Comparison of open-vocabulary action recognition on the EK-100-OV dataset.
IIOI DETECTOR
OPEN-SET
CLOSB-SET
METHOD
TYPE
top-1 action (%)
top-5 action (%) top-1 action (%) top-5 action (%)
S3D
/
fine-tune
0
37.6
2xS3D
/
fine-tune
0.1
36.7
OAP+AOP
V
fine-tune
11.2
35.9
LaViLa
X
zero-shol
7.57
22.78
16.59
34.88
L
zero-shot
8.48
21.36
17.34
36.96
Table 7: Comparison downstream benchmarks under the zero-shot setup, where "MODEL+4" denotes using EgoNCE++ to continue to pretrain the original MODEL.
EgoHOIBenchh
Epic Keeee-itchen--e 1aaesss--0---0 100-MIR
CharadesEgo
mAP (%)
nDCG (%)
METHOD
verb (%)
noun (%)
action (%)
mAP
V-+
T-
Avg.
V-+
T- V
Avg.
EgoVLP
25.2
19.2
22.2
28.1
25.4
26.7
EgoVLP++
56.11
69.05
41,63
25.6
19.7
22.7
28.6
25.7
27.1
19.7
EgoVLPv2
36.10
63.40
26.40
26.9
19.9
23.4
28.8
26.8
27.8
17.2
EgoVLPv244
64.10
32.40
28.0
19,9
23.9
29.8
26.8
28.3
17.5
LaViLa
46.61
74.33
36.85
35.1
26.6
30.8
33.7
30.4
32.0
20.6
LaViLatt
80.63
75.30
63.17
35.8
27.5
31.7
33,9
30.7
32.3
20.9
C.3
MAIN RESULTS
C.3.1 ZERO-SHOT SETUP EVALUATION
EgoNCE++ Consistently Improves Generalization Across All Vg. As shown in Table 7 all VLM benefit from pretraining with EgoNCE++, resulting in consistent improvements across EgoHOIBench, EK100-MIR and CharadesEgo.
Open-Set EgoHOI Recognition on EK-100-
OV. The EK-100-OV (Chatterjee et al., (024) aims recognize unseen categories, especially novel objects, at inference time. We evaluate both LaViLa and LaViLa+ on this benchmark in the zero-shot setup, with results presented in Table 6 Although our model does not out- perform those specifically designed models that extract object region features using an HOI de- tector (Shan et al., 2020) at inference time. it demonstrates strong generalization capabilities and competitive results on top-5 actions, consid- ering 2,639 candidate HOI combinations at inference time. Compared to LaViLa. our enhanced model La ViLa++ shows clear improvement across most key metries (e.g.. +0.91 in open-set top-1 action accuracy), highlighting its effectiveness in adapting to open-set conditions.
Table 8: Comparison of action recognition tasks Epic-Kitchens-In 100, where the zero-shot setting organized the same way as EgoHOlBench.
SETTING
METHOD
verb
nown
action
LaViLa
11.65
39.78
10.14
Zero-Shot
LaVilat
16.10
43.35
14.67
LaViLa
$9.06
41.53
26.29
Linear Probing
LaViLa+
59.43
42.41
26.86
ViLa++ Exhibits Better Linear Probing Property. We conduct zero-shot classification and shown in Table which highlights the improvement in the video
linear probing on EK100-CLS feature space and generalization capabilities of LaViLt++. We achieve steady improvement of +4.53% on action accuracy under zero-shot settings and +0.57 % under linear probing settings.
Larger Model Sizes Still Benefit from EgoNCE++. EgoNCE++ can also be applied to the LaViLa-Large model, as shown in T First, VaVaLa++ enhances the model's robustness to HOI- related word variations, demonstrated by significant improvement of +27.01 on EgoHOIBench. Additionally, our model achieves notable gains of +1.0% in mAP and +1.1 % in nDCG on other datasets. While our model surpasses the HelpingHands model (without multitask from the HOI detection) by average of +0.8% in mAP, its overall performance still falls short. The HelpingHands model freezes La LareeLare and adds a transformer decoder to learn an object-aware feature space through multitask learning, combining enhanced video-language pretraining with video-noun match-
19

---

Published as a conference paper at ICLR 2025
Table Comparison of models built upon VaVa on zero-shot EK-100-MIR and EGTEA.
Epic- Epic-Kitchens- 100-heens-O0-MRRR
BGTEA
mAP (%)
nDCG (%)
METHOD
action
V-T
T- V
Avg.
V-+ T
T- V
Avg.
mean-ace
topl-acc
LaViLa
38.69
40.0
32.2
36.1
36.1
33.2
34.6
34.1
40.1
La LaViL+++
65.70
41.3
32.8
37.1
37.8
33.6
35.7
37.5
38.6
Helpingglands
42.3
32.7
37.5
39.3
36.2
37.8
39.1
46.6
HelpingHands wo obj
40.7
31.1
35,9
38.3
35.0
36.6
44,9
40.1
Table 10: Comparison with state-of of-th-of-hhe-rtss on EK-100-MIR and EGTEA under the fine-tune setup.
Epic-Kitchens- 100-MIR
EGTEA
METHOD
MAP (%)
nDCG (%)
top-1 1 acc
mean acc
MME
Wray al.
2019
38.5
48.5
JPOSE
Wrav al.
2019
44.0
53.5
I.STA
Sudhhakrrm el al
2019
61.86
53.00
IPL
Wang cl aln 202T
60.15
MTCN
Kazzaos cf al.
2021
73.59
65.87
LaViLa Zhao et al.
2023
50.4
64.8
78.04
70.56
Vi
50.1
65.1
78.33
71.20
ing, video-sentence matching via EgoNCE, and HOI detection using pseudo-labels. Although the HelpingHands model is generally stronger, the frozen features from LaViLa would still struggle EgoHOlBench. We believe our method is orthogonal and compatible with models like HelpingHands. By replacing EgoNCE with EgoNCE++, we propose that these objectives could collaboratively strengthen EgoVLMs by providing both better visual supervision and text supervision.
C.3.2 FINE-TUNING SETUP EVALUATION
In this setup, we further finetune the model on the training and validation splits of downstream tasks.
Multi-Instance Retrieval on EK-100-MIR. As illustrated in Table TO La outperforms its original version in terms of nDCG but shows a decrease in mAP. These results suggest that while our approach enhances the ranking of candidates, it does retrieve data with similar HOIs effectively, highlighting trade-off between fine tne-grained HOI recognition and the diversity of retrieved outcomes. Despite this, LaViLa++ still serves as strong zero-shot learner for EgoHOI actions.
Action Recognition on EGTEA. This benchmark specifically focuses on cooking activities. Notably, LaViLa++ achieves state-of-the-att performance on EGTEA, showcasing its ability leverage the robust generalization capabilities of LaViLa. The improvement observed on EGTEA demonstrates that our proposed approach remains effective even when evaluated on out-of-domain benchmarks.
C.4 MORE ABLATION STUDIES
Ablation studies are conducted by pretraining the EgoVLP model (Lin et al. 2022 using EgoNCE++
Type of Negatives in V2T. The impact of verb ("VERB") or noun ("NOUN") negatives gener-
Table 13: Ablation of different training strategies.
ated by the LLM is detailed in Table [T] Neg- ative verb samples effectively enhance model training, improving verb accuracy by +14.89% %. In contrast, noun negatives yield a modest im- pact with an accuracy improvement of +0.17% This discrepancy could be attributed to the noun vocabulary size of approximately 7k words, which considerably larger than the verb vo- cabulary of about 2k words, making it more challenging to acquire visual knowledge from text supervision with limited data.
EgoHOlBench
EK- 100-
VIS
TEXT PARAM
action
mAP
nDCG
frozen
finzen
30.16
22.2
26.7
LoRA frozen
3.1M
41.63
22.7
27.1
full
frozen
109M
44.39
22.4
27.0
frozen
full
63.5M
60.18
9.6
16.8
LoRA full
66.7M
60.01
9.8
16.9
full full
172.5M
59.82
12.5
19.2
20

---

Published as a conference paper at ICLR 2025
FgOVEP
LaVila
0.6
action
28.6
33.5
EgoVLP nDCG
LaviLs nDC3
28.1
33.0
Accurads
0.5
27.6
32.5
0.4
27.1
32.0
26.6
31.5
0.3
64
Number negatives
100%
Percentage pretraining data
Figure 9: Scaling effect of negative number on EK-100-MIR (avg. nDCG).
Figure 10: Impact of varying data vol- ume used in pretraining.
Table 11: Ablation of types of negatives.
Table 12: Ablation of types of positives.
VERB
NOUN
verb (%)
(%)
action (%)
VERB
NOUN
verb (%)
noun (%)
action (%)
40.70
68.86
30.51
$5.29
69.03
40.88
41.47
69.06
31.29
55.34
68.89
41.00
x
55.16
69.03
40.81
x
55.93
68.80
41.26
55.29
69.03
40.88
x
56.11
69.05
41.63
Type of Positives in T2V. Table shows different positive sampling strategies in T2V loss. These strategies aggregate video representations based on verbs (VERB) or nouns (NOUN) in their captions. Due the strong bias towards nouns, the results show that aggregating nouns alone yields the largest improvements, whereas pulling videos with similar verbs slightly damages noun recognition.
Training Strategy for Dual Encoder. We further investigate the impact of various training strategies for dual encoders, as shown in Table [3] Comparing row 2 and row 3, we observe that full tuning outperforms LoRA tuning by +2.76% EgoHOlBench but underperforms by an average of -0.3% on EK-100-MIR. These results indicate that while using additional parameters during full tuning can improve performance, may also lead to decreased generalization on out-of-domain benchmarks. Given the importance of generalization in the real world. opt for LoRA tuning for the visual encoder while keeping the text encoder frozen. When the text encoder is trainable, as shown in rows EgoHOlBench,
4-6, there is boost in performance LaViLat+. However, the lack of generalization to EK- 100-MIR suggests significant overfitting to the pretraining dataset. Therefore, choose to freeze the text encoder ensure generalization.
approaching the results achieved by
Volume of Used Pretraining Data. Results on the pretraining data size are presented in Figure TO The findings highlight significant increase when only 10% of the data (250K) used, with action accuracy rising from 30.3% to 39.2%. In contrast, using the remaining data only results in an improvement of +2.43%.
LoRA Rank. We conduct another study to inves- tigate the impact of rank configurations for LoRA. As detailed in Table 14 our findings reveal that a LoRA rank of 16 enhances generalization capabil- ities, while even minimal rank of I can signifi- cantly improve EgoHOI recognition performance. This trend suggests that relatively small training adjustments can significantly enhance the visual feature space, leading improved performance with minimal computational cost.
Table 14: Ablation of LoRA rank.
LoRA
PARAMS
(%)
noun (%)
action (%)
0.24M
54.76
68.86
40.52
0.82M
55.11
3.14M
55.29
69.03
40.89
32
6.24M
55.01
68.80
40.64
Scaling Effect of Negative Numbers nDCG. Figure 9 illustrates the trend in nDCG for 100- MIR as the number of negative samples increases. Similar to mAP, there is clear correlation where using more negatives leads better performance. While mAP focuses identifying the single
21

---

Published as a conference paper at ICLR 2025
Table 15:
Comparison of results on EgoHOlBench generated from different LLMs, where
"MODEL++" denotes using EgoNCE++ to continue to pretrain the original MODEL.
EgoHOBennet
UPT4o-EgoHOIBBen
DeepSeck- EgoHOIBench
METHOD
verb (9)
noun (8)
action (9)
verb (9t)
noun (96)
action (%)
verh (%)
noun (%)
action (5))
EgoVLP
40.27
68.60
30.16
41.44
66.55
33.96
40.75
56.42
30.56
EgoVLP++
69.05
50.89
41.78
LaViLa
46.61
74.33
36.85
45.18
82
38.20
44.16
62.28
34.09
LaViLa++
80.63
75,30
63.17
53.95
73.65
44.63
52.73
63.48
39.15
LLaHA EgcHClBench
CPP-40 EgeHHiBanch
Deeptionk- DeeppSeee-ppprraaaaa EgaiHGiBench
LLaHt EgeHOBunch
GFT 4F7-4D-B EgaHOIBench
DerpsSiaat Egett3lBacch
100
LLaMA Ege.Hreenne
100.0%
37.9%
41.5%
LLMMAEgaHOOBBncC
100.0M
67.5%
51.3%
70
GPT-4o EgoHorBench-
37.9%
200.0%
58.0%
GPT-4o ETatHOBegoOOBecch
47.9%
100.0%
69.4%
DeepSeek- EigHHCilencch-hh
41.5%
58.00 0%
100.0%
DespSeei EgoHolBench
Verb Vocabulary Overlapping
Noun Vocabulary Overlapping
Figure 11:
Vocabulary overlap ratio of EgoHOlBench generated by different LLMs. Word frequencies in the vocabulary are normalized, resulting in symmetric matrix.
Table 16:
Analysis of the efficiency and effectiveness of different negative generators. The vocabulary- based method offers efficient online generation, while the LLM-basee approach provides greater generalization at a higher computational cost. The results are derived from ELo+LP++
EgolCOlBench noun (%)
EK- 100-MIR
METHOD
SPEED (ms/neg)
verb (%)
action (%)
mAP
nDCG
none vocab-based
40.27
68.60 68.60
30.16
22.2
26.7
0.005616 (on CPU)
54.46
68.56
40.07
22.5
27.1
LLM-based
27. 27.648 (on GPU)
56.11
69.05
41.63
22.7
27.1
correct answer, nDCG emphasizes the overall quality of the ranking. The improved nDCG indicates that increasing the number of negatives helps refine the ranking, elevating more relevant HOIs and relegating irrelevant ones, thereby enhancing HOI understanding.
C.5 MORE ANALYSES
Bias of LLMS in EgoHOlBench. To reveal bias on the negative generation in LLMs, we create Ego- HOIBench similarly using other LLMs: DeepSeek-200B (DeepSeek-Al (022) and GPT-4o (OpenAl 2024). In Figure II we calculate the vocabulary overlapping ratio, where the value in position (t, j) represents the proportion of the vocabulary from the i-th set that overlaps with the j-th set. The GPT- 4o and DeepSeek tend to generate different words compared with LLaMA. The evaluation results are shown in Table 15 The vocabulary bias affects the results some extent, but our EgoNCE++ steadily improves the performance on all benchmarks.
Efficiency Analyses of Vocab-Based and LLM-Based Negative Generators. We analyze the trade-off between the efficiency and efficacy of our designed vocab-based and LLM-based negative generator in Table [6 Our LLM-based approach, which uses LLaMA-3-SB to generate HOI candidates, operates offline and requires higher computational cost. Specifically, generating training data for 2.5M samples required approximately 192 GPU hours A6000 hardware. In comparison, vocab-based negative sampling method operates far more efficiently, requiring only 0.039 hours on 16 CPU threads and enabling online updates. Since both methods demonstrate notable improvements
22

---

Published as a conference paper at ICLR 2025
p0tk (noun)
(nount
Evertl dea
pata trunt (houn) (houn))
hent bern
picks (roge)
Evera) drancs
Dvers] board
Evartll ko
criee
verb-anchored
noun-anchored
verb-anchored
noun-anchored
Text
Video
Figure Visualization of Ego feature space. Both video and text feature space keep exhibiting an object-centric feature space.
Owidl com berti
hootl fncort
(erg) Eveint power
toathes joard
board (wrrd p:t
(ian))
adaot (oom)
verb-anchored
noun-anchored
verb-anchored
noun-anchored
Text
LaViLa
Video
P20 (oOH)) inta
(Verto cath
cuft Droun) It's Irxant
(eers) och fearsl
bokihes (ncord aure Troun)
towati Everbologt lapte
bouches (Rail) aduss treunt
twertl beand teerss bn
verb-anchored
noun-anchored
verb-anchored
noun-anchored
Text
Video
LaViLa++
Figure 13: Comparison of LaViLa and LaViLa++'s feature spaces. Although they keep exhibiting object-centric feature spaces, the video-text matching is greatly improved by EgoNCE++
on EgoHOIBench and EK-100-MIR, recommend selecting the appropriate method based on the size of the pretraining vocabulary and the available computational resources.
D QUALITATIVE RESULTS
VLP'LP's Also Exhibits Object-Centrie Feature Space. Since EgoVLPy2 used our paper (i.e.. dual encoder without fusion in the backbone) shares similar structure with Ego visualize the EgoVLP's feature space in the same way as LaViLa, which also exhibits an object- centric characteristic. We suspect that most of the egocentric video-language models pretrained with contrastive learning will be object-centric, regardless of their detailed architecture. To illustrate why representations tend cluster by nouns when verbs vary, we consider both the pretraining data and the video encoding architecture. From a data perspective, for example, a video of someone "cutting grass" is more visually similar to one of "watering grass" in the same environment (Grauman 2022), whereas "cutting onion" in kitchen would appear quite different from "cutting grass" due the change in both verb semantics and visual content. From an architectural perspective, current vision models primarily encode videos based on single-frame visual information (Lei et al. (023). focusing on objects rather than actions. As result, the model tends group representations by nouns (visual similarity) rather than verbs (temporal information). To create more verb-friendly feature space, a potential solution could incorporating multi- modality data that captures motion and temporal dynamics, such as optical flow event cameras.
Negatives Sampled from Different Generators. We provide several examples of negative samples Notably, LLM- based captions tend to be more
produced by different generators in Figure [5] semantically plausible than those generated by vocab-based or mule-based methods, which may include words not found in the Ego4D dictionary.
23

---

Published as a conference paper at ICLR 2025
Comparison before/after Using EgoNCE++. As previously discussed, EgoVLP++ significantly outperforms Ego after pretraining with EgoNCE++. To illustrate this, we provide examples of both improved cases and bad cases in Figure 16 and in Figure I7 respectively. Figure [6 shows that EgoNCE++ enhances the model's ability to learn more robust video-text alignments, enabling our refined model to identify fine-grained EgoHOls. In contrast, Figure [ highlights some extreme cases where our model struggles. In these cases, the background tends to be more complex, and the differences among actions are subtle, making them difficult to differentiate. We also visualize the feature space of ViLa and LaViLa++ in Figure [3] These visualizations reveal that the feature space preserves its object-centric nature. However, the inherent structure of the video or text representation space does not necessarily impact the video-text matching results Instead, our approach enhances video-text matching by making the correct video-text pair more distinguishable from negative video-text ones.
24

---

Published as a conference paper at ICLR 2025
SYSTEM PROMPT: Respond the user's input helpful assistant and human-like
the please change word, which a verb, make the action the text great different, and keep the other unchanged.
Please meaning by repeating what said before.
output toxts with different
Output:
You should use ISON formet for both input and output.
{
"AnswerO": C grabs the laundry bag from ground",
Input:
"Answerr": "ec C picks the laundry bag from
"Text": "deC ground"
ground".
"Answor2": "#C throws the laundry bag on the ground".
"Answer3": "NN pushs the loundry bag on the ground".
"Answerd"" pulls laundry bag the ground".
"Answer5":
the ground".
"Answer"": "NC C kicks the laundry bag
the ground".
Annswer?":"Ae destroys the laundry bag
ground",
"Answere": "eC C folds the laundry bag on
"AnswerS" "AC C litts the laundry bag from the ground"
Input:
"teat": "HC sits a chair"
Output:
"Anwwer":
"Answer1": "NC C jumps on chair"
"#C reppairs chair"
"Answer3" *#C lios on chair'
"Answerd"" "#C raises chair"
"Answer5":
chair"
"Answere": "#C thrrews chair"
"Anwwer?": removes chair"
"Answer8": "AC steps on chair",
"Answer9": "NC draws chair"
Input:
Toxt": 'aC drops the grass"
Ousput:
"Answard": "#C plants the grass"
'Answer1": '#C Catter the
'Answer2": "#C Ctrims gross".
"Anuwer3": 'HC digs the
Im
"Answerd": mows grass",
"Answer5": 'HC C Crabes the gress".
"Answer6": "HC
"Anowar7": "AC prunes the grass"
'Answerd": '#C Ceertiizes the
"Answer9: 'HC Caerates the prass"
Figure 14: Examples of options EgoHOlBench generated by LLM's in-context learning.
25

---

Published as a conference paper at ICLR 2025
LLM-based
laundry bag on #C C drags the laundry bag on the floor
creeps the laundry bag the floor
places laundry bag beside
#C C pushes the laundry bag off the floor
IC C. the laundry bag the ground
Rule-based
Vocab-based
holds the shopping
the ground
laundry bag on
looks at the bag on the ground
#C demolishes the laundry bag on the floor
#C C picks the bag on the ground
#C fries the loundry bag the floor
touches the bag the ground
paves loundry bag on
moves laundry bag on
#C hoes the laundry bag on the floor
LLM-based
#C mends gress cutting scissor
sharpens grass cutting
#C paints grass cutting scissar
#C cleans cutting scissor
grass cutting
#C picks grass cutting scissor
Rule-based
Vocab-based
#C down scissor
#C picks building scissor
grips scissor
grass cutting
#C carries scissor
#C downs grass cutting scissor
NC touches scissor
#C returns cutting scissor
wipes scissor
grass cutting
LLM-based
electronic
#C pushes the bowi off the electronic scale
#C weighs the bowl on the electronic scale
against electronic #C spins around the bowl on the electronic scale
measures
puts the bowl electronic scale
Rule-based
Nocab-based
adjusts bowl
mingles
electronic
adjusts the bowl on the bowl
#C C speeds the bowl on the electronic scale
#C C turns the bowl on her laps
#C C displaces the bowl the electronic scale
adjusts bowl lotchen
dogs
electronic
adjusts
her laps
IC C inherits the bowl on the electronic scale
Figure 15: Examples of options generated by LLM in the pretraining set. We provide five candidates for simplicity. The green words denote the word to be replaced while the red ones denote words generated by different strategies.
26

---

Published as a conference paper at ICLR 2025
EgoVLP
verb
1. #C balances a plate on top something sink
1. puts bowl sink
2. puts plate in the sink
2. RC C puts plate in the sink
3. hangs plate
3. puts lid sink
4. #C glues plate together in the sink
puts sponge
5. #C takes plate out of the sink
5. AC C puts cup in the sink
EgoVLP++ (Ours)
puts
the sink
1. BC puts plate in the sink
2. takes a plate out the sink
puts bowl sink
3. MC balances a plate on top of something in the sink
3. puts lid on sink
4, hangs
the sink 5. glues plate together
4. HC puts sponge in the sink
5. puts the sink
EgoVLP
verb
noun
#C cleans bowl stored in the cabinet with right hand
1. BC C picks up pot from stair with right hand
bowl from the cabinet with right hand
2. BC C picks bowl from the cabinet with right hand
IC inspects bowl stored in the cabinet with right hand
bowl from the dishwasher with hand 4. #C C picks up sponge with right hand
4, IC moves bowl around in the cabinet with right hand pushes bowl of the cabinet with right hand
5. HC C picks spoon from the spoon rack with right hand
EgoVLP++ (Ours)
1. MC C picks up bowl from the cabinet with right hand
1. #C C picks up bowl from the cabinet with right
2. IC cleans a bowl stored in the cabinet with right hand
2. IC C picks pot from stair with right hand
bowl around
cabinet with hand
3. HC C picks bowi from the dishwasher with hand
4. IC C inspects bowl stored the cabinet with right hand
4. C picks up sponge with right hand
5. IC C pushes bowl out of the cabinet with right hand
5. #c c picks up spoon from the spoon rack with right hand
EgoVLP
verb
noun
wraps clath across sewing machine
puts
sewing
#C tangles the cloth around the sewing machine
2. #C C puts the cloth on the sewing machine
3. #C C rolls the cloth over the sewing machine
3. BC C puts the
the sewing machine
drapes
above sewing machine
puts the thread the sewing
#C puts the dlath on the sowing machine
5. #c C puts the yarn halder on the sewing machine
EgoVLP++ +++ (Ours)
1. AC C puts the cloth on the sewing machine
puts the cloth
machine
aC tangles the cloth around
machine
2. RC puts the material sewing machine
sewing
3. ac C puts the yarn on the sewing machine
AC wraps the cloth across the sewing machine
BC the yarn holder the sewing machine
AC drapes the cloth above the sewing machine
5. #C C puts the thread on the sewing machine
Figure 16: Improved cases on EgoHOlBenchh after using EgoNCE++ Five candidates are provided for simplicity. The green sentences denote ground-truth caption that is correctly classified by EgoVLP++ while the red ones are false positives predicted by EgoVLP.
27

---

Published as a conference paper at ICLR 2025
#C takes out the knife from the dish rack
1. #C places the knife the sink table
wraps
knife dish
knife
verb NC buries the knife the dish rack
noun 3. puts the chopstick the dish rack
4. #C cleans the knife in the dish rack
4. #C places the knife the orange
RC places the knife dish rack
places the spoon the dish
picks
#C cuts cutter
2. C Cpicks box
verb #C paints cutter
noun
3. #C picks cutter
picks
picks
HC glues a cutter
5. picks a jacket
cutting
picks
floor
AC moves the cutting board across the floor
2. picks the shoe from cutting board
verb 3. AC drags the cutting board along the floor
noun
3. #C C picks the clip from the cutting board
#C glues the cutting board the floar
4, picks the sausage from cutting board
HC picks the cutting board from the floor
5. C picks the cutting board from floor
AC shakes the acrylic paint tube lid squeezes acrylic paint
the paint tin
opens acrylic paint
verb 3. MC punctures the acrylic paint tube lid
noun 3. #C C opens the can lid
8C crushes the acrylic paint tube
4,
the Japanese vinegar lid
opens acrylic paint
opens
1. AC C unfolds the cloth
1. #C C turns the lid
cloth
verb AC C scrunches cloth
noun 3. #C C turns the pillow
4. AC straightens the cloth
4. #C C turns the carrot
Figure 17: Bad cases on EgoHOlBench where EgoVLP++ struggles. The green sentences are ground truth and the red ones are mistakenly predicted by Ego Others the remaining candidates. Five candidates are provided for simplicity.
28


Reviews:
[{"review_id": "z90gYPWgEK", "reviewer": "Reviewer_Ns3f", "summary": "This paper focuses on understanding hand-object interactions using egocentric video-language models (EgoVLMs). The authors introduce a new benchmark called EgoHOIBench to evaluate EgoVLMs' ability to distinguish between similar HOI descriptions by changing verbs or nouns. They then propose EgoNCE++, an asymmetric contrastive learning objective to improve the models' sensitivity to nuanced changes in HOI-related language by using LLM-generated negative samples to enhance text supervision and preserve object-centric video representations. The paper shows that EgoVLMs generally behave better at object recognition while struggling with action. The experiments show that EgoNCE++ enhances performance across three EgoVLMs and improves generalization on seven downstream EgoHOI tasks.", "questions": "Please see weakness section for the questions.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is clear and well-motivated. The authors first diagnose current EgoVLMs' capability on the proposed EgoHOIBench benchmark and then propose a method to address the problem. This makes the paper easy to follow and provides insights to readers.\n2. The proposed EgoHOIBench provides a targeted evaluation for current egocentric video-language models regarding the capability of understanding hand-object interactions with variations in verbs and nouns. The authors also provide an inspiring analysis of the current EgoVLM and find their common failure to understand actions.\n3.  The paper conducts extensive experiments and shows the proposed EgoNCE++ consistently improves model performance across various state-of-the-art EgoVLMs, highlighting its versatility and effectiveness.\n4. The authors release the codes that promote reproducibility.", "weaknesses": "The major concern about this paper is its novelty. The idea of strengthening fine-grained compositional understanding by constructing hard-negative examples is not novel e.g. (Yuksekgonul et al., 2023). What are the major differences between the proposed method and previous works?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jRwqBOkt8f", "reviewer": "Reviewer_DGPT", "summary": "This paper introduces a novel way to identify HOIs in EgoVLMs, which addresses the limitation of current egocentric models regarding verb recognition. They propose EgoHOIBench, a new benchmark designed to evaluate understanding in EgoVLMs, that can evaluate EgoVLM’s capabilities in understanding variations of HOI combinations. In addition, their experiment demonstrates a stronger robustness towards recognizing nouns through their analysis of EgoHOIBench performance on HOI-verbs and HOI-nouns. Furthermore, they propose EgoNCE++, an asymmetric contrastive learning objective to address these limitations by enhancing model robustness in handling fine-grained verb and noun variations within HOIs for egocentric video-language pretraining, which successfully fulfills their aim to preserve the object-centric nature of the feature space without additional visual data usage or architectural changes.", "questions": "Questions are written with weakness.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "•\tThe paper pinpoints the current weakness that existing benchmarks in egocentric vision with EgoHOI are limited. While some of them are only emphasized in kitchen scenarios, the others fail to provide effective supervision for understanding the nuances of HOI combinations that make our current model “a lack of fine-grained negative supervision during pretrained process”. \n\n•\tThe author discovers this critical gap and gives out their own solution in making EgoHOIBench, a benchmark specifically designed to test HOI comprehension in egocentric contexts, which is designed to more effectively evaluate the ability of EgoVLMs to select the correct sentence from multiple HOI-related options using video-text matching.\n\n•\tCompared with InfoNCE and EgoNCE, which often sample easy negative pairs without employing hard negative mining for text, and distinguish EgoHOIs based on verb-noun variation, EgoNCE++ incorporates asymmetric video-to-text and text-to-video losses, which enables the model to better understand HOI combinations by generating negatives through HOI-related word changes and preserves object-centric feature properties by clustering video representations based on similar nouns.\n\n•\tThe paper is well-written and also the experiments are well-structured by covering a large range of benchmarks in Egocentric Vision tasks across commonly used datasets: Open-vocabulary recognition, multi-instance retrieval, and action recognition.", "weaknesses": "•\tThe paper introduces EgoNCE++ as an asymmetric contrastive learning objective. Figure 3 shows the visualization of LaViLa’s feature space indicating both video and text feature space exhibit the object-centric property and suggests that video-noun matching is easier than video-verb matching since noun-anchored embeddings form tighter clusters, while verb-anchored embeddings are more dispersed. Can we make a feature space comparison after applying EgoNCE++ to see if the verb-anchored changed?\n\n•\tThe paper mentions that it utilizes the LLaMA-3-8B model to generate HOI candidates through in-context learning. It would be better if we could also conduct additional studies on the performance gains versus computational expense(FLOPs) in order to better know the trade-off for the approach.\n\n•\tEgoNCE++ works significantly well on the author’s self-designed benchmarks EgoHOIBench, but it seems that for other datasets the improvement is marginal: EK100-MIR, CharadesEgo shown in Figure.5; EK-100-MIR and EGTEA in Table 9 for zero-shot setting; LaViLa and LaViLa++ in Table 10 and Table 7. Will there be a possible overfitting or lack of variety since the primary performance measure is based on EgoHOIBench, which specifically focuses on nuanced HOI comprehension?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mCodOclIug", "reviewer": "Reviewer_7uyU", "summary": "In this paper, the authors propose a new dataset, named EgoHOIBench, which consists of multiple choice questions for video clips in which there are two settings for verb and noun understanding. There are 10 choices for each case in which the verb or the noun respectively have been changed to create a hard negative. The paper finds that current Egocentric VLMs are unable to handle this change and so a new loss, named EgoNCE++ in which an asymmetric objective forces the model to understand minor differences in text yet for video groups the representations via their noun representations. The results show that for the EgoHOIBench, training current Egocentric VLMs with the new objective leads to an improvement in performance as well as for other downstream tasks.", "questions": "1. Has an investigation/analysis been carried out regarding the fact that the loss is designed in the same way as the answers within the dataset? An outcome of this can be seen within Table 4 perhaps, where the choice of generator leads to a large increase in results for HOIBench, but is marginal for the EK-100-MIR task.\n2. What does it mean by the sentence not made excessively difficult? Is this because false negatives could be introduced via semantic matching? If so, what is used to prevent this?\n3. Why cluster videos based on similar noun representations only, instead of a mix of verb and noun representations? \n4. How are negatives introduced into the video-to-text loss to ensure that false negatives are not included? It is mentioned that either vocabulary from the dataset or an LLM is used to generate the negatives, but there is no information on how this is done. Is this the same as the collection information for HOI-Bench\n5. Were all videos chosen for HOI-Bench from the validation set in a similar fashion to EgoMCQ? Or were some videos excluded? How was 10 settled on as the number of captions?\n6. When constructing the dataset, has any care been taken to ensure that the LLM used to generate the questions isn't hallucinating? Were any measures put in place to remove noise and ensure a cleaner outcome for the dataset? Also, has any human checking/evaluation been carried out to get a sense of how clean the data is?\n7. Has an ablation been carried out in which EgoNCE++ (ours in Table 3) is used for the loss and InfoNCE is used for the V2T loss?\n8. For figure 6, has this been evaluated with how the number of negatives scales for the base models without the EgoNCE++ loss? It would be interesting to see how this compares for both models.\n9. Has a histogram of positive/negative similarities been created instead of using the new PND metric in Figure 7?\n10. It would be interesting to see Figure 3 after the EgoNCE++ objective has been applied if this has been created to see how this differs.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The results of the different Egocentric VLMs on downstream tasks after being trained with the new EgoNCE++ objective are good with nice increases in performance.\n* Creating and proposing an asymmetric loss with EgoNCE++ is interesting and makes a lot of sense in how these two settings need to be treated differently whereas in the past this has not necessarily been true.\n* There are a lot of results (and qualitative figures within the appendix) to showcase the method and struggles of the current methods without the EgoNCE++ objective.", "weaknesses": "# Weaknesses\n* One potential reason for the new loss doing so well on the constructed HOI-Bench is as the benchmark has been designed in the same way as the loss function. A model that has been trained using a loss with negatives that represent the same style of negatives that are in the ground truth answers is certainly going to do better.\n* Models seem to already do well on noun focused tasks, so it isn't clear to me why there is a large focus on still clustering videos based on similar noun representations only.\n* Currently, the method section (Section 3) is lacking some important information regarding how the negatives are generated beyond an LLM/using the vocabulary of the dataset. Also, if there is any checking that is done to reduce/remove false negatives.\n* The details of how HoI-bench is collected within the main paper is very scarce. The appendix includes some more information, but is still quite lacking. Details of why the number of videos were chosen, number of captions, any choice of category within Ego4D etc. is missing\n* Instead of defining a new metric to look at the positive/negative similarities within equation 6/figure 7. A histogram could have been used instead which might have given a clearer picture (again using the max negative). It's less intuitively clear from the figure what the numbers represent, especially as these have been multiplied by 100, emphasising the small differences even further.\n\n# Additional Comments\nTable 3 is inconsistent with how it uses lower case and upper case compared to the rest of the paper. Additionally, 'ours' is used here instead of EgoNCE++.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eP9zuRs8WK", "reviewer": "Reviewer_3po1", "summary": "* Current EgoVLMs, used to understand hand-object interactions in first-person views, can be easily misled by minor changes in interaction descriptions (e.g., changing verbs or nouns).\n* A benchmark was created to highlight the limitations of EgoVLMs, particularly their struggle with recognizing verbs compared to nouns, often due to insufficient fine-grained supervision.\n* An asymmetric contrastive objective was introduced to improve video-language alignment:\n Video-to-Text Objective utilizes enhanced text supervision with negative captions generated by large language models or HOI-related vocabulary substitutions.\nText-to-Video Objective focuses on an object-centric feature space that clusters video representations based on shared nouns.\n* The proposed approach, EgoNCE++, significantly improves EgoVLM performance, especially in tasks like multi-instance retrieval, action recognition, and temporal understanding.", "questions": "1. How does the proposed asymmetric contrastive objective compare with other contrastive techniques in video-language modeling? Could you clarify what specific elements make it novel in this context?\n\n2. Can you elaborate on the robustness of the generated negative captions? Have you evaluated how biases in large language models might affect the model's performance?\n\n3. How does the proposed benchmark compare to existing ones since the improvement is not significant on the existing dataset, and do you anticipate any limitations when applying it to real-world scenarios or other egocentric datasets?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The study tackles a key shortcoming in egocentric video-language models (EgoVLMs), which is their limited ability to distinguish subtle changes in interaction descriptions. This issue has significant implications for understanding hand-object interactions, a critical area in ego-centric vision applications.\n\n2. The work introducing a specialized benchmark to evaluate EgoVLMs under challenging scenarios is a valuable contribution. This benchmark exposes performance gaps that were previously under-explored, providing a robust foundation for future research and model improvements.", "weaknesses": "1. The proposed asymmetric contrastive objective may lack the degree of methodological novelty expected. Similar objectives and contrastive learning techniques have been explored in other contexts as follows:\n * (a) The use of negative mining in contrastive learning is well explored in [1], where it aims to find better hard negatives to benefit contrastive learning.\n * (b) In the field of using augmented captions in multimodal contrastive learning is explored in [2].\n * (c) The use of LLM to generate more text for learning is explored in [3].\n\n\n2. Since the model is trained and evaluated on a benchmark specifically designed for the study, it’s unclear if these gains will translate to other existing benchmarks or real-world applications. As shown in Tables 3 and 4, the gain over the existing dataset using the proposed dataset is limited. This creates the risk of overfitting to a controlled benchmark, which could raise questions about generalizability.\n\n[1] Robinson, Joshua, et al. \"Contrastive learning with hard negative samples.\" ICLR 2021.\n\n[2] Yuan, Xin, et al. \"Multimodal contrastive training for visual representation learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\n[3] Fan, Lijie, et al. \"Improving clip training with language rewrites.\" Advances in Neural Information Processing Systems 36 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)