Do Egocentric Video-Language Models Truly Understand Hand-Object Interactions?

## Abstract

Abstract Egocentric video-language pretraining is a crucial paradigm to advance the learning of egocentric hand-object interactions (EgoHOI).
Despite the great success on existing testbeds, these benchmarks focus more on closed-set visual concepts or limited scenarios.
Due to the occurrence of diverse EgoHOIs in the real world, we propose an open-vocabulary benchmark named EgoHOIBench to reveal the diminished performance of current egocentric video-language models (EgoVLM) on fined-grained concepts, indicating that these models still lack a full spectrum of egocentric understanding.
We attribute this performance gap to insufficient fine-grained supervision and strong bias towards understanding objects rather than temporal dynamics in current methods.
To tackle these issues, we introduce a novel asymmetric contrastive objective for EgoHOI named EgoNCE++ .
For video-to-text loss, we enhance text supervision through the generation of negative captions by leveraging the in-context learning of large language models to perform HOI-related word substitution.
For text-to-video loss, we propose an object-centric positive video sampling strategy that aggregates video representations by the same nouns.
Our extensive experiments demonstrate that EgoNCE++ significantly boosts open-vocabulary HOI recognition, multi-instance retrieval, and action recognition tasks across various egocentric models, with improvements of up to +26.55% .
Our code is available at https://github.com/xuboshen/EgoNCEpp .

## 1 Introduction

Humans have long dreamed of embodied agents replacing human labor in various societal roles.
One effective strategy to achieve this involves leveraging transferable knowledge from diverse egocentric demonstrations to train these agents to imitate human actions in daily activities.
Egocentric videos, captured from first-person views using wearable devices, typically demonstrate how humans interact with nearby objects using their hands.
Given its potential application in embodied agents [, , ] and augmented/virtual reality [, ], the understanding of egocentric video, especially hand-object interaction, has seen a surge of interest in recent years.
Based on the large-scale dataset Ego4D [], recent works [] have proposed pretraining an egocentric video-language model (EgoVLM) to enhance downstream egocentric tasks, such as video-text retrieval [, ] and action recognition [].

Figure: Figure 2: Illustration of EgoHOIBench and the diminished performance on our benchmark compared to previous EgoHOI benchmarks in diverse scenarios. An EgoHOI recognition trial is considered correct only when both the verb and noun are accurately classified. We obtain these results across various benchmarks using a state-of-the-art EgoVLM [].
Refer to caption: x2.png

Despite notable advancements in these tasks, we observe that current egocentric hand-object interaction (EgoHOI) benchmarks are either limited to closed-set categories in specific domains [, ] or consist of easy open-vocabulary tasks in the open world, which are already well-solved by current EgoVLMs.
Therefore, existing benchmarks do not sufficiently evaluate EgoVLMs’ capability to recognize EgoHOIs in real-world applications.
This leads us to a question: Do existing EgoVLMs truly understand egocentric hand-object interactions?
To address this, we develop EgoHOIBench, a benchmark designed to challenge well-trained EgoVLMs with fine-grained and open-vocabulary EgoHOI understanding trials.
Specifically, EgoHOIBench requires models to recognize the correct caption for a given video from a group of caption candidates that differ by only one feasible verb or noun.
In Figure 2, we evaluate a state-of-the-art EgoVLM on open-vocabulary benchmarks and observe diminished performance on our EgoHOIBench compared to previous efforts.
Despite being trained on extensive EgoHOI data, the EgoVLM still struggles to accurately recognize HOIs in our new benchmark.
We attribute their suboptimal performance to two main factors:
lack of fine-grained supervision and strong bias towards understanding nouns in HOI.

Specifically, when examining the video-language pretraining that employs video-to-text and text-to-video contrastive losses, it is evident that training batches typically contain numerous easy negative samples (e.g. “cut grass” vs. “pick apple”).
While these samples help the model differentiate between coarse-grained HOIs, they fail to provide effective supervision for comprehending finer distinctions in actions such as “pick apple” versus “move apple”, which can be considered as hard negatives for each other.
Such hard negatives rarely appear within the same batch.
To enhance learning to discern such fine-grained differences, previous works have adopted hard negative mining strategies, which involve sampling data with high feature similarities [] or selecting visually similar videos recorded in identical environments [, ].
However, these heuristic approaches fall short of addressing the intricate challenges posed by EgoHOIBench.
Furthermore, since it is much simpler to understand static visual content than to interpret temporal dynamics in a video,
EgoVLMs often resort to object recognition as a shortcut for the EgoHOI task.
Consequently, these models have developed object-centric feature spaces with a strong bias towards recognizing nouns rather than verbs.
Recent studies [, ] have explored enhancing VLMs’ verb understanding through hard negative sampling by LLMs.
However, these initiatives primarily focus on third-person datasets [, , ], which overlooks the nuanced challenges in understanding HOI combinations from egocentric views.

To address the aforementioned issues, we introduce EgoNCE++, a new contrastive learning objective that comprises asymmetric video-to-text and text-to-video losses.
Specifically, the video-to-text loss enables the model to discern subtle differences in captions for a given video through extensive hard negative supervision, which is achieved by instructing a large language model (LLM) to generate more challenging HOI-related negative captions via in-context learning [].
Conversely, the text-to-video loss is designed to preserve the well-established object-centric feature space by aggregating video representations whose captions share similar objects.
We conduct extensive experimental analyses on various EgoHOI downstream benchmarks and demonstrate that EgoNCE++ can more effectively handle challenges in EgoHOI understanding.

Our contributions can be summarized threefold:
(1) We developed EgoHOIBench, a benchmark specifically designed to evaluate models’ capabilities in fine-grained and open-vocabulary EgoHOI comprehension.
(2) We propose EgoNCE++, an innovative HOI-aware asymmetric contrastive learning objective for egocentric video-language pretraining.
(3) As illustrated in Figure 1, our experimental results demonstrate that EgoNCE++ can be applied to various EgoVLMs, notably improving generalization across six downstream EgoHOI tasks, especially in zero-shot settings.

## 2 Related Work

Egocentric Hand-Object Interaction.
Captured by head-mounted cameras, egocentric hand-object interaction (EgoHOI) [, , , , , ] provides insight into how humans interact with objects from a first-person view.
To address this task, Huang *et al.* [] and Kazakos *et al.* [] focus on recognizing close-set EgoHOIs using additional multimodal cues (e.g., gaze, sound), while Wang *et al.* [] adopt a self-supervised approach [] to exploit visual information.
Considering the abundant resources of third-person data, some works [, ] aim to transfer view-agnostic knowledge from third-person videos to egocentric viewpoints.
However, the unpredictable nature of open-world environments poses new challenges, requiring models to handle a variety of unseen concepts.
Recent studies [, ] seek to improve the understanding of open-vocabulary EgoHOI, but these efforts are either limited to specific domains like kitchens [] or laboratories [], or involving easy EgoHOI recognition that is well-solved by current egocentric models [, ] as shown in Figure 2.
A promising strategy to address these limitations involves egocentric video-language pretraining [, , , ], which learns generalizable representations by leveraging the Ego4D [] dataset with over 3,000 hours of footage of daily human interactions.
As a pioneering work, EgoVLP [] uses the EgoNCE loss to treat video-text samples with similar HOIs as positives and visually similar videos as negatives during pretraining.
Another method, LaViLa, enhances text supervision by generating diverse positive captions for videos to foster robust contrastive learning through a visual-conditioned GPT-2 [] and a T5 [] rephraser.
In this work, we assess and address the limitations of these models on our developed EgoHOIBench, with a particular focus on advancing open-vocabulary EgoHOI comprehension.

Hard Negative Mining.
Hard negative mining is a pivotal technique [, ] used to refine representations within the visual-language metric space during contrastive learning.
Traditionally, this process involves pairing positive samples with hard negatives that exhibit high feature similarity in pretraining datasets [, , , ], or selecting hard negative clips recorded in the same environment [].
Recent innovations have introduced the use of LLMs to generate hard negative captions, aiming to enhance action comprehension [, ] in video-VLMs [], and improve compositional understanding [] in image-VLMs [, , , ].
For instance, ViA [] proposes a verb-focused framework that creates negative captions of sentences and verb phrases using an LLM.
However, most of these works adapt their methods to third-person datasets [, ] such as Kinetics-400 [], or to close-set recognition tasks [, ] like SSv2 [, ], which may not fully capture the nuanced challenges of open-world EgoHOIs.
In our paper, we demonstrate that employing LLMs to expand hard negative captions during pretraining can significantly enhance fine-grained EgoHOI understanding for EgoVLMs.

## 3 EgoHOIBench: Open-Vocabulary Egocentric HOI Benchmark

Existing HOI benchmarks in egocentric vision primarily focus on closed-set visual concepts [, ] or are limited to simpler open-vocabulary tasks [], leading to an inadequate capture of complex human activities in the real world.
Since the remarkable performance achieved by existing models on these testbeds, as illustrated in Figure 2, the need for a more challenging open-vocabulary benchmark becomes evident.
Therefore, we develop EgoHOIBench, which features an expansive vocabulary and includes thousands of verbs and nouns from rich real-life scenarios.

To evaluate the model’s EgoHOI understanding ability, we design each EgoHOI recognition trial as follows:
given a video segment $x$, the model is required to distinguish the correct caption $S^{*}$ from $N$ verb-focused hard negative captions $\{S_{i}\}_{i=1}^{{N}}$, where $S_{i}$ and $S^{*}$ only differ in the verb, and similarly, to identify $S^{*}$ from $N$ noun-focused hard negative captions $\{S_{j}\}_{j=1}^{{N}}$, where $S_{j}$ is generated by replacing the noun in $S^{*}$ with alternative nouns.
A recognition trial is regarded successful only when the model accurately identifies the correct caption $S^{*}$ from both verb-focused and noun-focused hard negatives.
EgoHOIBench measures the models’ recognition accuracy across all 29K test trials.
Details of data construction by leveraging LLM are available in Appendix B.

## 4 EgoNCE++: HOI-Aware Asymmetric Pretraining Objective

Our goal is to enhance the capability of pretrained EgoVLMs to recognize fine-grained and open-vocabulary EgoHOIs.
Firstly, we briefly recap and diagnose the symmetric contrastive learning objectives used by previous EgoVLMs in Section 4.1.
We identify that one bottleneck of current objectives is the lack of fine-grained text supervision for videos.
Furthermore, EgoVLMs trained with these objectives exhibit a strong bias towards recognizing nouns in HOI understanding.
To enhance EgoVLM pretraining by enriched text supervision while maintaining strong noun recognition capabilities, we devise a new contrastive learning objective featuring asymmetric video-to-text and text-to-video losses in Section 4.2.
Figure 3 illustrates an overview of our method.

Figure: Figure 3: Illustration of our pretraining framework. (a) EgoVLMs are trained with EgoNCE++, where the visual encoder is trained using LoRA [] to enhance video representation, while the text encoder remains frozen. Specifically, EgoNCE++ consists of (b) V2T: generating HOI-related negative captions for fine-grained supervision, and (c) T2V: leveraging the strong ability of EgoVLMs to recognize nouns by aggregating video features with similar nouns.
Refer to caption: x3.png

### 4.1 A Recap of Egocentric Video-Language Pretraining Objectives

Egocentric video-language pretraining (VLP) adheres to the typical VLP paradigm, which utilizes a dual-encoder architecture to perform contrastive learning between video and text modalities.
Current EgoVLPs consider two symmetric contrastive learning objectives: InfoNCE [] and EgoNCE [, ].
Both of them employ video-to-text and text-to-video losses in symmetric forms.

InfoNCE [].
It is a widely used contrastive learning objective, which brings positive samples closer while pushing negative samples further apart through an online cross-entropy loss among video-text pairs.
The symmetric InfoNCE loss applied to a batch of (video, caption) samples is formulated as:

$$ $\mathcal{L}^{\rm info}=-\frac{1}{B}(\sum_{v_{i}\in\mathcal{B}(v)}\log\frac{ \exp(v_{i}\cdot t_{i}/\tau)}{\Sigma_{t_{j}\in\mathcal{B}(t)}\exp(v_{i}\cdot t_ {j}/\tau)}+\sum_{t_{i}\in\mathcal{B}(t)}\log\frac{\exp(t_{i}\cdot v_{i}/\tau)} {\Sigma_{v_{j}\in\mathcal{B}(v)}\exp(t_{i}\cdot v_{j}/\tau)})$ (1) $$

where $(v_{i},t_{i})$ denotes the $L_{2}$ normalized feature vectors of the $i$-th (video, caption) sample within a batch. $\mathcal{B}(v)=\{v_{i}\}_{i=1}^{B}$ and $\mathcal{B}(t)=\{t_{i}\}_{i=1}^{B}$ refer to the videos and captions of the batch $\mathcal{B}=\{(v_{i},t_{i})\}_{i=1}^{B}$, respectively. $\tau$ is the temperature parameter.

EgoNCE [, ].
It is specifically tailored for egocentric scenarios.
As shown in the following video-to-text loss, EgoNCE enhances the learning of subtle differences in scenes by enlarging the batch to include additional video clips with visually similar backgrounds.
It also expands positive video-text pairs by including texts that depict similar HOIs occurring in different contexts.

$$ $\mathcal{L}^{\rm ego}_{v2t}=-\frac{1}{2B}\sum_{v_{i}\in\mathcal{\widetilde{B}} (v)\cup\mathcal{B}(v)}\log\frac{\Sigma_{t_{k}\in\mathcal{P}(t_{i})}\exp(v_{i} \cdot t_{k}/\tau)}{\Sigma_{t_{j}\in\mathcal{B}(t)}\exp(v_{i}\cdot t_{j}/\tau)+ \Sigma_{t_{j^{\prime}}\in\mathcal{\widetilde{B}}(t)}\exp(v_{i}\cdot t_{j^{ \prime}}/\tau)}$ (2) $$

where $\mathcal{\widetilde{B}}=\{(v_{i^{\prime}},t_{i^{\prime}})\}_{i^{\prime}=1}^{B}$, $\mathcal{\widetilde{B}}(v)=\{v_{i^{\prime}}\}_{i^{\prime}=1}^{B}$ and $\mathcal{\widetilde{B}}(t)=\{t_{i^{\prime}}\}_{i^{\prime}=1}^{B}$ represent the enlarged batch samples. Each $(v_{i^{\prime}},t_{i^{\prime}})$ corresponds to the (video, caption) pair sourced from the same recording environments as the $i$-th video clip in the original batch.
Furthermore, $\mathcal{P}(t_{i})\subseteq\mathcal{\widetilde{B}}(t)\cup\mathcal{B}(t)$ defines the set of captions, each containing at least one verb or noun that matches those in $t_{i}$.
To save space, we omit displaying the text-to-video loss as it is symmetrically formulated.

Deficiencies of Current Objectives.
Although EgoVLMs pretrained with InfoNCE and EgoNCE have acquired substantial knowledge about EgoHOI, they lack fine-grained supervision and exhibit a strong bias towards understanding nouns rather than dynamic visual content.
Specifically, InfoNCE tends to sample easy contrastive pairs (e.g., ‘opens a drawer” vs. “picks an egg”), which do not differ significantly by HOI-related words.
In contrast, EgoNCE adds more challenging negative samples by enriching its pretraining batches with similar environments.
However, EgoNCE treats data with similar HOIs as positives, and the additional samples may appear as positives due to the repeated appearance of the same objects.
Therefore, it may inadvertently group similar but distinct HOIs together, such as mistakenly considering “opens a drawer” and “closes a drawer” as a positive pair.
As a result, both objectives tend to differentiate between coarse-grained EgoHOIs, not nuanced ones.
Moreover, recognizing the objects in frames instead of complex temporal dynamics is a well-known shortcut solution for current models [].
The strong bias towards nouns diminishes their ability to interpret EgoHOI which requires both verb and noun comprehension.
Consequently, both InfoNCE and EgoNCE exhibit considerable limitations in comprehending the EgoHOIs.

### 4.2 An Asymmetric Egocentric Pretraining Objective

To overcome the limitations of InfoNCE and EgoNCE, we propose an asymmetric objective named EgoNCE++.
The term “asymmetric” signifies that we craft the video-to-text and text-to-video losses to meet different needs —  specifically addressing the issue of inadequate fine-grained supervision and leveraging the strong ability of noun recognition.
EgoNCE++ enhances the fine-grained text supervision in the video-to-text loss by utilizing an LLM for hard negative generation, while it maintains the capability to recognize nouns through the text-to-video loss that aggregates video representations with similar nouns.

#### 4.2.1 V2T: HOI-Aware Negative Generation by LLM

We devise the video-to-text loss by integrating a greater number of hard negative samples that are essential for improving EgoHOI recognition.
These samples are generated using the comprehensive world knowledge and advanced reasoning capabilities of LLMs, which provide viable alternatives for HOI-related terms.
Unlike random sampling of verbs and nouns from a large vocabulary, LLMs ensure that the generated sentences are not only fluent and diverse but also closely aligned with real-world contexts.
For instance, an LLM might suggest “C waters grass” as an alternative to “C cuts grass”, avoiding nonsensical outputs like “C drinks grass” that might arise from random sampling.

Note that the LLM may not follow the instructions and thus produces hallucinations or generates synonyms of the original HOIs.
To address this, we utilize in-context learning [] to guide the LLM with specific task examples.
Additionally, we employ the Ego4D dictionary to filter out possible synonyms, ensuring that the generated captions maintain their distinctiveness.
Each ground truth video caption $t_{i}$ is then paired with LLM-generated, HOI-aware negative captions, which include verb negatives where only the verb is altered or noun negatives where only the HOI-related noun changes.
After generating negative captions with plausible semantics for videos, we implement the following supervision loss to enhance EgoHOI understanding from the video-to-text perspective:

$$ $\mathcal{L}_{v2t}=\frac{1}{B}\sum_{v_{i}\in\mathcal{B}(v)}\log\frac{\exp(v_{i} \cdot t_{i}/\tau)}{\Sigma_{t_{j}\in\mathcal{B}(t)}\exp(v_{i}\cdot t_{j}/\tau)+ {\color[rgb]{.75,0,.25}\Sigma_{t_{k}\in\mathcal{N}_{\rm noun}(t_{i})\cup \mathcal{N}_{\rm verb}(t_{i})}\exp(v_{i}\cdot t_{k}/\tau)}}$ (3) $$

where $\mathcal{N}_{\rm verb}(t_{i})$ and $\mathcal{N}_{\rm noun}(t_{i})$ denote the verb negatives and noun negatives, respectively.

#### 4.2.2 T2V: Object-Centric Positive Video Sampling

Although the text-to-video process does not directly impact EgoHOI recognition, the text features produced by the frozen text encoder effectively act as anchors to guide the optimization of video features.
Specifically, we aim to preserve the capability to recognize coarse-grained nouns by aggregating video representations that depict similar objects and separating those that differ in nouns.
To this end, we devise an object-centric text-to-video loss, where $\mathcal{P}_{n}{(v_{i})}$ denotes the videos that feature similar nouns in their captions:

$$ $\mathcal{L}_{t2v}=\frac{1}{B}\sum_{t_{i}\in\mathcal{B}(t)}\log\frac{{\color[ rgb]{0,0,1}\Sigma_{k\in\mathcal{P}_{n}(v_{i})}}\exp(t_{i}\cdot v_{k}/\tau)}{ \Sigma_{v_{j}\in\mathcal{B}(v)}\exp(t_{i}\cdot v_{j}/\tau)}$ (4) $$

#### 4.2.3 Training Strategy

To refine the video representation of EgoVLMs for better generalization, we freeze the text encoder while fine-tuning the visual encoder using LoRA [].
We train the dual encoders with both negative caption mining for videos via an LLM and object-centric positive video sampling for texts.
Our final pretraining objective comprises the sum of text-to-video and video-to-text losses.

$$ $\mathcal{L}_{\rm EgoNCE++}=\mathcal{L}_{t2v}+\mathcal{L}_{v2t}$ (5) $$

## 5 Experiments

### 5.1 Experimental Settings

To ensure the robustness of our approach, we evaluate a variety of well-known EgoVLMs including EgoVLP [], EgoVLPv2 [] and LaViLa [].
Details of these models can be found in our Appendix C.1.
In this paper, we continue to pretrain these models instead of training them from scratch due to computational resource constraints.

Pretraining Dataset and Details.
Our pretraining video clips are sourced from EgoClip-3.8M [], ensuring no overlap with the clips used in EgoHOIBench.
The dataset is curated to focus on EgoHOIs and excludes any video that primarily captures activities of other persons, resulting in a dataset of 2.5 million entries.
The videos in our dataset are typically about 1 second long, with captions that include verbs and nouns relevant to hand-object interactions.
During pretraining, we sample 4 frames from each video.
We employ LoRA tuning with both rank and alpha set to 16.
The models continue to be pretrained by 10 epochs in 12 hours using 8$\times$ A800 GPUs, with a total batch size of 576.
We utilize LLaMA3-8B [] to generate hard negative captions for the videos.

Downstream Benchmark and Evaluation Setups.
We evaluate the model on three types of tasks across six benchmarks under a zero-shot setting, where the pretrained model is directly evaluated on downstream tasks without additional fine-tuning:
(1) Open-vocabulary EgoHOI recognition on EgoHOIBench, EK-100-OV [] and ActionBench []. EK-100-OV assesses unseen categories in kitchen scenarios, while EgoHOIBench and ActionBench focus on open-world HOI recognition.
Unlike the challenging task in our EgoHOIBench, ActionBench only contains a binary classification of semantically reversed sentences.
(2) Multi-instance retrieval on Epic-Kitchens-100 [], a kitchen-oriented retrieval task where multiple video clips may correspond to the same narration.
(3) Action recognition on CharadesEgo [] and EGTEA [].
CharadesEgo involves 157 indoor activity classes, presenting out-of-domain challenges when compared to Ego4D or Epic-Kitchens as it utilizes different wearable devices, while EGTEA focuses on classifying 106 cooking activities.

**Table 1: Comparison on downstream benchmarks under the zero-shot setup, where “MODEL++” denotes using EgoNCE++ to continue to pretrain the original MODEL.**
|  | EgoHOIBench |  | Epic-Kitchens-100-MIR |  | CharadesEgo |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| METHOD | verb (%) | noun (%) | action (%) |  | mAP (%) |  | nDCG (%) |  | mAP |  |  |  |  |
|  | V$\rightarrow$ T | T$\rightarrow$ V | Avg. |  | V$\rightarrow$ T | T$\rightarrow$ V | Avg. |  |  |  |  |  |  |
| EgoVLP [] | 40.27 | 68.60 | 30.16 |  | 25.2 | 19.2 | 22.2 |  | 28.1 | 25.4 | 26.7 |  | 19.3 |
| EgoVLP++ | 56.11 | 69.05 | 41.63 |  | 25.6 | 19.7 | 22.7 |  | 28.6 | 25.7 | 27.1 |  | 19.7 |
|  | (+15.84) | (+0.45) | (+11.45) |  | (+0.4) | (+0.5) | (+0.5) |  | (+0.5) | (+0.3) | (+0.4) |  | (+0.4) |
| EgoVLPv2 [] | 36.10 | 63.40 | 26.40 |  | 26.9 | 19.9 | 23.4 |  | 28.8 | 26.8 | 27.8 |  | 17.2 |
| EgoVLPv2++ | 44.41 | 64.10 | 32.40 |  | 28.0 | 19.9 | 23.9 |  | 29.8 | 26.8 | 28.3 |  | 17.5 |
|  | (+8.31) | (+0.70) | (+6.0) |  | (+1.1) | (+0.0) | (+0.5) |  | (+1.0) | (+0.0) | (+0.5) |  | (+0.3) |
| LaViLa [] | 46.61 | 74.33 | 36.85 |  | 35.1 | 26.6 | 30.8 |  | 33.7 | 30.4 | 32.0 |  | 20.6 |
| LaViLa++ | 80.63 | 75.30 | 63.17 |  | 35.8 | 27.5 | 31.7 |  | 33.9 | 30.7 | 32.3 |  | 20.9 |
|  | (+34.02) | (+0.97) | (+26.32) |  | (+0.7) | (+0.9) | (+0.9) |  | (+0.2) | (+0.3) | (+0.3) |  | (+0.3) |

### 5.2 Main Results

Open-Vocabulary EgoHOI Recognition.
For this task, models must comprehend targets expressed in free-form language and accurately discriminate between answers given video inputs.
As shown in Table 1, all EgoVLMs pretrained using EgoNCE++ demonstrate significant improvements on EgoHOIBench, demonstrating the versatility of our method across EgoVLMs with varying architectures, training strategies, and losses.
The notable improvements primarily arise from enhanced verb understanding, e.g. a +34.02% increase in verb accuracy leading to a +26.32% improvement in action accuracy for LaViLa++.
In contrast, noun classification has been effectively achieved using previous pretraining objectives, resulting in only minor improvements [].
Importantly, the pretraining of LaViLa utilizes 56 million captions, while our approach reduces the data volume by 55%, indicating that enriching negative supervision could be more crucial for efficient HOI learning than merely increasing the volume of data.
Surprisingly, EgoNCE++ achieves more than double the improvement on LaViLa (+26.32%) than on EgoVLP (+11.45%) and EgoVLPv2 (+6.0%).
We consider that the degree of improvement is significantly dependent on the quality of the learned features produced by the frozen text encoder, which act as supervisory signals for EgoNCE++.
As the quality of text supervision improves, the visual encoder is tuned to generate more advanced visual representations.
Consequently, the notable difference in improvement suggests that the InfoNCE loss adopted by LaViLa may provide superior text features, being more effective than the EgoNCE objective.
More results on the other two benchmarks are presented in Section C.3.

Multi-Instance Retrieval.
As shown in Table 1, we observe consistent benefits from EgoNCE++ across all metrics, validating the effectiveness of asymmetric objective in enhancing the model generalization.
Here, mAP is a binary metric that regards a retried case with similar HOI as correct and incorrect otherwise, while nDCG is non-binary, which measures the quality of ranked retrieved cases based on their HOI similarity.
The results reveal that our approach effectively improves the ranking of candidates while retrieving data with similar HOIs, demonstrating more promising retrieval properties in terms of fine-grained HOI recognition and the diversity of retrieved outcomes.

Action Recognition.
This task focuses on human activities in CharadesEgo and kitchen activities in EGTEA.
As demonstrated in Table 1, EgoVLMs pretrained using EgoNCE++ exhibit modest improvements over their baselines.
The slight enhancements in CharadesEgo stem from the differences between Ego4D and CharadesEgo, where videos are recorded by crowdsourced workers using mobile cameras instead of wearable glasses.
More results on EGTEA are available in Section C.3.

**Table 2: Ablation of types of negatives.**
| VERB | NOUN | verb (%) | noun (%) | action (%) |
| --- | --- | --- | --- | --- |
| ✗ | ✗ | 40.70 | 68.86 | 30.51 |
| ✗ | ✓ | 41.47 | 69.06 | 31.29 |
| ✓ | ✗ | 55.16 | 69.03 | 40.81 |
| ✓ | ✓ | 55.29 | 69.03 | 40.88 |

### 5.3 Ablation Study

All ablation studies are conducted by continuing to pretrain the EgoVLP model [].

Type of Negatives in V2T.
The impact of verb negatives (“VERB”) or noun negatives (“NOUN”) generated by the LLM is detailed in Table 3.
Negative verb samples effectively enhance the model training, improving verb accuracy by +14.89%.
In contrast, noun negatives yield a modest impact with an accuracy improvement of +0.17%.
This discrepancy could be attributed to the size of the noun vocabulary of about 7k words, which is far larger than the verb vocabulary of about 2k words, making it more challenging to acquire visual knowledge from text supervision with limited data.

Figure: Figure 4: Impact of varying data volume used in pretraining.
Refer to caption: extracted/5639241/file/pretrain.png

Type of Positives in T2V.
Table 3 showcases different positive sampling strategies used in T2V loss.
The positive sampling aggregates video representations that share similar HOI-related words.
Due to the strong bias towards nouns, the results show that only aggregating the nouns brings the largest improvements, while pulling videos with similar verbs slightly damages the recognition of nouns.

**Table 4: Ablation of V2T and T2V losses.**
| V2T | T2V | verb (%) | noun (%) | action (%) |
| --- | --- | --- | --- | --- |
| Info | Info | 40.70 | 68.86 | 30.51 |
| ours | Info | 55.29 | 69.03 | 40.88 |
| ours | ours | 56.11 | 69.05 | 41.63 |

Figure: Figure 5: Ablation of the negative number.
Refer to caption: extracted/5639241/file/fig_num_negativesv2.png

Video-to-Text and Text-to-Video Loss.
As shown in Table 5, our video-to-text supervision generated by the LLM (“ours”), clearly enhances the EgoVLM’s ability to distinguish fine-grained details compared to InfoNCE loss (“Info”).
Furthermore, optimizing over the text-to-video loss can maintain the strong bias towards nouns, where the EgoHOI recognition also benefits from the aggregation of video representations with similar nouns.

Hard Negative Generator.
We compare three hard negative generators:
(1) LLM-based, where an LLM performs word substitution tasks through in-context learning;
(2) rule-based, where hard negatives are identified by selecting captions with the highest BLEU [] scores from the pool;
(3) vocab-based, where HOI verbs are substituted with arbitrary verbs from the predefined Ego4D vocabulary containing 2,653 verbs.
As shown in Table 5, the rule-based approach tends to generate negatives already encountered during contrastive learning, thus offering less fine-grained supervision.
However, the vocab-based approach may yield meaningless HOI combinations, acting as easy negatives rather than the more effective hard negatives provided by the LLM, and thus the LLM-based method performs better.

Volume of Used Pretraining Data.
Discussion regarding the scale of pretraining data is provided in Figure 4.
The results highlight a surge at the point where only 10% of the data (250K) is used, with action accuracy increasing from 30.3% to 39.2%, while using the remaining data only improves by +2.43%.

Number of Negatives.
Figure 5 illustrates the trend of HOI recognition accuracy as the number of negative samples increases.
There is a clear correlation where using more negatives leads to improved performance.
Incorporating hard negative captions for videos not only expands the density of V2T supervision but also broadens the vocabulary associated with the videos.

Training Strategy for Dual Encoder.
We further investigate the impact of various training strategies for dual encoders as shown in Table 6.
Comparing row 2 and row 3, we observe full tuning outperforms LoRA tuning by +2.76% on EgoHOIBench but underperforms by an average of -0.3% on EK-100-MIR.
The results indicate that for full tuning, using additional parameters improves performance but may also lead to decreased generalization ability on out-of-domain benchmarks.
Given the importance of generalization in the real world, we opt for LoRA tuning for the visual encoder while keeping the text encoder frozen.
When the text encoder is trainable, as shown in rows 4-6, there is a substantial boost in performance on EgoHOIBench, even approaching the results achieved by LaViLa++.
However, the lack of generalization to EK-100-MIR suggests severe overfitting to our pretraining dataset.

**Table 6: Ablation of different training strategies, where “EgoHOI-B” denotes EgoHOIBench.**
|  |  |  | EgoHOI-B | EK-100-MIR |  |
| --- | --- | --- | --- | --- | --- |
| VIS | TEXT | PARAM | action | mAP | nDCG |
| frozen | frozen | 0M | 30.16 | 22.2 | 26.7 |
| LoRA | frozen | 3.1M | 41.63 | 22.7 | 27.1 |
| full | frozen | 109M | 44.39 | 22.4 | 27.0 |
| frozen | full | 63.5M | 60.18 | 9.6 | 16.8 |
| LoRA | full | 66.7M | 60.01 | 9.8 | 16.9 |
| full | full | 172.5M | 59.82 | 12.5 | 19.2 |

### 5.4 Visualizations

In Figure 6, we visualize the changes in contrastive similarities from EgoVLP to EgoVLP++ on EgoHOIBench.
The values in matrix grid represent the cosine similarity of features between the video in that row and caption in that column.
First, the (video, caption) similarities tend to have less variance based on the same nouns than on the same verbs in sentences.
For instance, the similarity between video1 and text1 “#c c opens the tap water” is 0.49, whereas its similarity to text2 “#c c operates the tap” is 0.50.
These texts share the same noun but different verbs, yet their similarities are nearly identical, leading to potential misclassification.
In contrast, the similarity between video1 and text3 “#c c opens the pan” is much lower at -0.01, despite sharing the same verb.
After pretraining with EgoNCE++, which offers more text-negative supervision for videos, the diagonal elements, which represent positive video-text pairs, exhibit higher similarities compared to other entries (e.g., video1 is now correctly classified to text1).
These results demonstrate that EgoVLP++ achieves better video-text alignments by leveraging EgoNCE++, effectively reducing the excessive focus on nouns over verbs.

Figure: Figure 6: Qualitative examples of the EgoVLP’s optimization process on EgoHOIBench.
Refer to caption: x4.png

## 6 Conclusion

We construct EgoHOIBench, an open-vocabulary, fine-grained EgoHOI recognition benchmark that highlights the current limitations of EgoVLMs in comprehending hand-object activities.
To address these limitations, we propose an asymmetric learning objective called EgoNCE++, which enhances the video-to-text loss using dense hard negatives generated by an LLM, and a text-to-video loss focusing on video-positive sampling.
Through extensive experimental analyses of diverse benchmarks, especially our EgoHOIBench, we demonstrate that the proposed EgoNCE++ is effective across different EgoVLMs, regardless of their varying architectures and learning objectives.

## Appendix A Discussions

Limitations.
Although our EgoNCE++ yields significant improvements across various fine-grained HOI benchmarks for various EgoVLMs, it is not without its limitations.
First, there is potential for further enhancement through the use of more advanced language models, and by increasing the number of negative samples.
Second, due to computational resource constraints, we do not pretrain our model from scratch.
We anticipate that utilizing a more diverse and extensive dataset with EgoNCE++ could lead to even better performance outcomes.

Social Impact.
The knowledge of EgoHOI acquired by EgoVLMs can significantly benefit real-world applications such as embodied agents and augmented reality systems.
However, the use of egocentric videos raises privacy concerns, as these recordings often capture personal and sensitive information.
Such privacy issues could potentially lead to negative impacts if not properly managed.

## Appendix B More Details of the EgoHOIBench

### B.1 Construction Process

The overall data construction pipeline can be seen in Figure 7.
We develop EgoHOIBench based on EgoMCQ [], which sources a diverse collection of 39,000 video clips from the validation set of the Ego4D dataset.
In our curation process, we only keep those EgoHOI clips performed by the camera wearer, excluding clips that record other people’s activities, such as multi-person interactions [].
To construct the HOI recognition trials as defined in our task definition above, given a video and its ground truth caption, we prompt an LLM to create candidate captions that contain semantically plausible HOIs differing from the ground truth.
Specifically, we employ the LLaMA-3-8B [] model to generate HOI candidates through in-context learning.
We show the specific prompts and two exemplary tasks used in this process, and examples of the final cases in Figure 10.
To avoid semantic redundancies and ensure the uniqueness of the hard negative candidates, we use the Ego4D dictionary to eliminate possible synonyms from the generated captions.
Furthermore, we also select hard negative candidates from the Ego4D captions based on their BLEU scores [] relative to the ground truth caption, which allows us to choose captions that are structurally similar to the ground truth but differ in semantics from the original pretraining dataset.
By doing this, we ensure the trials are challenging in this task.
Finally, EgoHOIBench comprises 29,651 video clips, each accompanied by one ground truth caption, 10 verb-focused negative captions, and 10 noun-focused negative captions.

Figure: Figure 7: Illustration of the automatic construction pipeline of EgoHOIBench.
Refer to caption: x5.png

### B.2 Vocabulary Statistics

The statistics of the vocabulary information are provided in Figure 8.
This dataset features a rich and diverse vocabulary, including approximately 800 verbs and 4,000 nouns.
The options generated by LLMs expand the vocabulary by two times more than that of the original correct answers, leading to massive combinations of verbs and nouns.

Figure: Figure 8: Illustration of the vocabulary statistics of EgoHOIBench.
Refer to caption: x6.png

## Appendix C More Experimental Analysis

For fair comparisons, we have re-implemented all experiments in the same environment and under identical settings, without any adjustments to the hyperparameters.

### C.1 Implementation Details

As introduced in the main paper, we validate our approach on three EgoVLMs including EgoVLP, its advanced version EgoVLPv2, and LaViLa.
EgoVLP is pretrained on the EgoCLIP-3.8M dataset and employs the EgoNCE loss for optimization.
EgoVLPv2 enhances the original model by incorporating a cross-attention mechanism between dual encoders and by pretraining on additional proxy tasks.
LaViLa, on the other hand, is trained on a vast dataset of 4 million videos, with 56 million captions generated by a visual-conditioned GPT-2 [] and further refined using a sentence rephraser T5 [].
This extensive training regimen enables LaViLa to improve the generalization of EgoVLMs.

We provide a summary of existing well-known EgoVLP methods in Table 7.
Our proposed EgoNCE++ continues to enhance the capabilities of pretrained EgoVLMs while using significantly fewer parameters.
For all models, we employ the AdamW optimizer with parameters ($\beta_{1}$, $\beta_{2}$) = (0.9, 0.999).
The learning rate is managed via a cosine annealing scheduler, initiating at 3e-5 and gradually decreasing to 3e-7.
During the training phase, we incorporate standard RandomResizedCrop for data augmentation and use LoRA tuning to continually train our EgoVLM.
For EgoVLPv2, we deploy a dual encoder architecture without cross-attention fusion.

**Table 7: Summary of existing egocentric video-language pretraining methods compared with ours.**
| METHOD | Pretrain Data | Objective | Negative Mining | Visual Encoder | Text Encoder | Param |
| --- | --- | --- | --- | --- | --- | --- |
| EgoVLP | 3.8M | EgoNCE | video sim | ImageNet [] | DistillBert [] | 172M |
| EgoVLPv2 | 3.8M | EgoNCE+MLM+VTM | feature sim | ImageNet [] | Roberta [] | 364M |
| LaViLa | 4M | InfoNCE | none | CLIP [] | CLIP [] | 180M |
| Ours | 2.5M | EgoNCE++ | text sim | EgoVLM | EgoVLM | 3M-43M |

### C.2 Benchmark Details

Multi-Instance Retrieval in EK-100-MIR.
In this benchmark, we finetune the EgoVLMs using the AdamW optimizer.
The learning rate is dynamically adjusted from 3e-3 to 1e-5 using a cosine annealing scheduler that incorporates a linear warmup, starting at 1e-6 for the first epoch.
We deploy a total batch size of 128 across 8 GPUs.
During both training and inference, 16 frames are sampled from each video.

Action Recognition in EGTEA.
For the zero-shot setup, we evaluate mean results across all evaluation splits by conducting a video-text retrieval task between video clips and their corresponding action text labels.
We prepend the text labels with the prompt “#C C …” to standardize the input format.
For the fine-tuning setup, we leverage the visual encoder and attach an additional linear projection head for the classification purpose.
The models are trained and evaluated on the first split of the validation set.
We employ the same optimizer, scheduler, batch size, and frame sampling rate as used in EK-100-MIR.
At inference time, we perform three spatial crops of size $224\times 224$ from each $256\times 256$ frame of the video clip, averaging their predictions to form the final prediction.

Action Recognition in CharadesEgo.
We perform a video-text retrieval task to achieve action recognition by matching video clips with their corresponding action text labels during zero-shot evaluation.
During inference, we also sample 16 frames from each video.

**Table 8: Comparison of open-vocabulary action recognition on the EK-100-OV dataset [].**
| METHOD | HOI<br>DETECTOR | TYPE | OPEN-SET | CLOSE-SET |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| top-1 action (%) | top-5 action (%) | top-1 action (%) | top-5 action (%) |  |  |  |
| S3D [] | ✓ | fine-tune | 0 | - | 37.6 | - |
| 2$\times$S3D [] | ✓ | fine-tune | 0.1 | - | 36.7 | - |
| OAP+AOP [] | ✓ | fine-tune | 11.2 | - | 35.9 | - |
| LaViLa | ✗ | zero-shot | 7.57 | 22.78 | 16.59 | 34.88 |
| LaViLa++ | ✗ | zero-shot | 8.48 | 21.36 | 17.34 | 36.96 |
|  |  |  | (+0.91) | (-1.42) | (+0.75) | (+2.08) |

### C.3 Main Results

#### C.3.1 Zero-Shot Setup Evaluation

Open-Vocabulary EgoHOI Recognition on EK-100-OV.
The EK-100-OV [] aims to recognize unseen categories, especially novel objects, at inference time.
We evaluate both LaViLa and LaViLa++ on this benchmark in the zero-shot setup, with results illustrated in Table 8.
Although our model does not outperform those specifically designed to extract object region features using an HOI detector [], it demonstrates strong generalization capabilities and competitive results on top-5 actions, with 2,639 candidate actions considered at inference time.
When compared to LaViLa, our enhanced model LaViLa++ shows clear improvement across most key metrics (e.g., +0.91% in open-set top-1 action accuracy), highlighting its effectiveness in adapting to open-set conditions.

Open-Vocabulary Action Recognition on ActionBench.
The ActionBench [] is a binary classification task focused on distinguishing the correct sentence answer from its antonyms.
In Table 9, we evaluate both LaViLa and LaViLa++ in the zero-shot setup.
The results indicate that our LaViLa++ can almost accurately classify the correct semantics of verbs from its antonyms, surpassing the state-of-the-art work proposed in [] and even approaching the human-level recognition ability.

**Table 9: Comparison of binary open-vocabulary action recognition on ActionBench, where * denotes the fine-tuned model by [].**
| MODEL | InternVideo* | Clip-Vip* | Singularity* | Human | LaViLa | LaViLa++ |
| --- | --- | --- | --- | --- | --- | --- |
| ACTION ACCURACY | 90.1 | 89.3 | 83.8 | 92.0 | 79.89 | 91.18 |

**Table 10: Comparison of zero-shot action recognition on EGTEA.**
| METHOD | mean-acc | top1-acc |
| --- | --- | --- |
| LaViLa | 30.92 | 35.08 |
| LaViLa++ | 31.43 | 33.02 |

Action Recognition on EGTEA.
As shown in Table 10, we evaluate LaViLa and LaViLa++ on the EGTEA dataset under zero-shot setup.
Here, the metric “top1 acc” does not account for the class imbalance whereas the metric “mean acc” eliminates long-tail imbalance.
LaViLa++ performs clearly better results on the mean-acc metric, indicating that our model effectively generalizes to the long-tail classes.
Given that the EGTEA dataset is confined to cooking activities, which may introduce strong biases that LaViLa++ does not fully capture, it is reasonable that LaViLa++ falls short in the top1-acc metric.

#### C.3.2 Fine-Tuning Setup Evaluation

In this setup, we further finetune the model on the training and validation splits of the downstream datasets after pretraining.

**Table 11: Comparison with state-of-the-arts on EK-100-MIR and EGTEA under the fine-tune setup.**
| METHOD | Epic-Kitchens-100-MIR | EGTEA |  |  |
| --- | --- | --- | --- | --- |
| mAP (%) | nDCG (%) | top-1 acc | mean acc |  |
| MME [] | 38.5 | 48.5 | - | - |
| JPoSE [] | 44.0 | 53.5 | - | - |
| LSTA [] | - | - | 61.86 | 53.00 |
| IPL [] | - | - | - | 60.15 |
| MTCN [] | - | - | 73.59 | 65.87 |
| LaViLa [] | 50.4 | 64.8 | 78.04 | 70.56 |
| LaViLa++ | 50.1 | 65.1 | 78.33 | 71.20 |
|  | (-0.3) | (+0.3) | (+0.29) | (+0.64) |

Multi-Instance Retrieval on EK-100-MIR.
As illustrated in Table 11, LaViLa++ outperforms its original version in terms of nDCG but shows a decrease in mAP.
These results suggest that while our approach enhances the ranking of candidates, it does not as effectively retrieve data with similar HOIs, underlining a trade-off between fine-grained HOI recognition and the diversity of retrieved outcomes.
Despite these, LaViLa++ leverages the robust pretrained knowledge in LaViLa, delivering competitive performance on the EK-100-MIR dataset.

Action Recognition on EGTEA.
This benchmark specifically focuses on cooking activities.
Notably, LaViLa++ achieves state-of-the-art performance on EGTEA, showcasing its ability to leverage the robust generalization capabilities from LaViLa.
The improvement observed on EGTEA demonstrates that our proposed approach is robust even when evaluated on out-of-domain benchmarks.

### C.4 Further Analysis

Distance between Positive Similarity and Negative Similarities.
In order to examine how a correct video-text pair is distinguished from negative video-text pairs, we define a novel metric PND to measure the distance between the correct option and wrong options:

$$ ${\rm PND}=\frac{1}{N}\sum_{i=1}^{N}({\rm sim}(v_{i},t_{i})-{\rm max}(\{{\rm sim }(v_{i},t_{j})\}_{j=1,j\neq i}^{M}))$ (6) $$

**Table 12: PND results ($\uparrow$) comparing positive samples with negative samples on EgoHOIBench.**
|  | EgoVLP | EgoVLP++ | LaViLa | LaViLa++ |
| --- | --- | --- | --- | --- |
| VERB | -1.36 | 1.68 | -0.84 | 11.08 |
| NOUN | 8.68 | 9.02 | 14.58 | 14.24 |

where $N$, $M$ denote the scale of the dataset and the number of wrong options, respectively.
The calculated results on EgoHOIBench are multiplied by 100 and shown in Table 12.
As can be seen, our pretrained models improve the discrimination ability, e.g. increasing the PND of verbs from -1.36 to 1.68 and from -0.84 to 11.08 on EgoVLP and LaViLa, respectively.
We observe that LaViLa achieves a larger verb improvement than EgoVLP, while LaViLa’s PND of nouns is already at a high value of 14.58, surpassing EgoVLP++’s 9.02 by a large margin.
We suggest that LaViLa has learned a well-established feature space that enables nouns to be well recognized compared to EgoVLP.
Therefore, even if the PND of nouns decreases slightly, the noun recognition still remains robust.

**Table 13: Ablation of LoRA rank.**
| LoRA | PARAMS | verb (%) | noun (%) | action (%) |
| --- | --- | --- | --- | --- |
| 1 | 0.24M | 54.76 | 68.86 | 40.52 |
| 4 | 0.82M | 55.11 | 68.89 | 40.95 |
| 16 | 3.14M | 55.29 | 69.03 | 40.89 |
| 32 | 6.24M | 55.01 | 68.80 | 40.64 |

LoRA Rank.
We conduct another study to investigate the impact of rank configurations for LoRA.
As detailed in Table 13, our findings reveal that a LoRA rank of 16 enhances generalization capabilities, while even a minimal rank of 1 can significantly improve EgoHOI recognition performance.
This trend indicates that relatively small training adjustments can significantly enhance the visual feature space, leading to better performance with minimal computational cost.

## Appendix D Qualitative Results

Object-Centric Feature Space.
We illustrate the EgoVLMs’ feature space in Figure 9, including LaViLa and EgoVLP.
We observe that after egocentric pretraining, the noun-focused features are well clustered, while the verb-focused features tend to be more scattered, proving again that understanding nouns is much easier than verbs.

Figure: Figure 9: Illustrations of the object-centric feature space for EgoVLP and LaViLa.
Refer to caption: x7.png

Negatives Sampled Different Generators.
We provide a few examples to show negative samples produced by different generators in Figure 11.
It’s important to note that LLM-based captions appear to be more semantically plausible than captions from vocab-based or rule-based generators, which may include words that are never seen in the Ego4D dictionary.

Comparison before/after Using EgoNCE++.
As we discussed before, EgoVLP++, EgoVLP++ significantly outperforms EgoVLP after pretraining using EgoNCE++.
To visualize this, we provide a few examples to show both improved cases and bad cases in Figure 12 and in Figure 13, respectively.
As can be seen in Figure 12, EgoNCE++ helps the model to learn more robust video-text alignments, enabling our refined model to identify fine-grained EgoHOIs.
On the other hand,  Figure 13 showcases some extreme cases that our model struggles to handle.
In these cases, the background tends to be more complex while the differences among actions are subtle to differentiate.

Figure: Figure 10: Examples of options on EgoHOIBench generated by LLM’s in-context learning.
Refer to caption: x8.png

Figure: Figure 11: Examples of options generated by LLM in the pretraining set. We provide five candidates for simplicity. The green words denote the word to be replaced while the red ones denote words generated by different strategies.
Refer to caption: x9.png

Figure: Figure 12: Improved cases on EgoHOIBench after using EgoNCE++. Five candidates are provided for simplicity. The green sentences denote groundtruth caption that is correctly classified by EgoVLP++ while the red ones are false positive predicted by EgoVLP.
Refer to caption: x10.png

Figure: Figure 13: Bad cases on EgoHOIBench after using EgoVLP++. The green sentences are groundtruth and the red ones are mistakenly predicted by EgoVLP++. Others are the remaining candidates.
Refer to caption: x11.png

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see weakness section for the questions.
soundness: 3
strengths: 1. The paper is clear and well-motivated. The authors first diagnose current EgoVLMs' capability on the proposed EgoHOIBench benchmark and then propose a method to address the problem. This makes the paper easy to follow and provides insights to readers.
2. The proposed EgoHOIBench provides a targeted evaluation for current egocentric video-language models regarding the capability of understanding hand-object interactions with variations in verbs and nouns. The authors also provide an inspiring analysis of the current EgoVLM and find their common failure to understand actions.
3.  The paper conducts extensive experiments and shows the proposed EgoNCE++ consistently improves model performance across various state-of-the-art EgoVLMs, highlighting its versatility and effectiveness.
4. The authors release the codes that promote reproducibility.
summary: This paper focuses on understanding hand-object interactions using egocentric video-language models (EgoVLMs). The authors introduce a new benchmark called EgoHOIBench to evaluate EgoVLMs' ability to distinguish between similar HOI descriptions by changing verbs or nouns. They then propose EgoNCE++, an asymmetric contrastive learning objective to improve the models' sensitivity to nuanced changes in HOI-related language by using LLM-generated negative samples to enhance text supervision and preserve object-centric video representations. The paper shows that EgoVLMs generally behave better at object recognition while struggling with action. The experiments show that EgoNCE++ enhances performance across three EgoVLMs and improves generalization on seven downstream EgoHOI tasks.
weaknesses: The major concern about this paper is its novelty. The idea of strengthening fine-grained compositional understanding by constructing hard-negative examples is not novel e.g. (Yuksekgonul et al., 2023). What are the major differences between the proposed method and previous works?

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: Questions are written with weakness.
soundness: 4
strengths: •	The paper pinpoints the current weakness that existing benchmarks in egocentric vision with EgoHOI are limited. While some of them are only emphasized in kitchen scenarios, the others fail to provide effective supervision for understanding the nuances of HOI combinations that make our current model “a lack of fine-grained negative supervision during pretrained process”. 

•	The author discovers this critical gap and gives out their own solution in making EgoHOIBench, a benchmark specifically designed to test HOI comprehension in egocentric contexts, which is designed to more effectively evaluate the ability of EgoVLMs to select the correct sentence from multiple HOI-related options using video-text matching.

•	Compared with InfoNCE and EgoNCE, which often sample easy negative pairs without employing hard negative mining for text, and distinguish EgoHOIs based on verb-noun variation, EgoNCE++ incorporates asymmetric video-to-text and text-to-video losses, which enables the model to better understand HOI combinations by generating negatives through HOI-related word changes and preserves object-centric feature properties by clustering video representations based on similar nouns.

•	The paper is well-written and also the experiments are well-structured by covering a large range of benchmarks in Egocentric Vision tasks across commonly used datasets: Open-vocabulary recognition, multi-instance retrieval, and action recognition.
summary: This paper introduces a novel way to identify HOIs in EgoVLMs, which addresses the limitation of current egocentric models regarding verb recognition. They propose EgoHOIBench, a new benchmark designed to evaluate understanding in EgoVLMs, that can evaluate EgoVLM’s capabilities in understanding variations of HOI combinations. In addition, their experiment demonstrates a stronger robustness towards recognizing nouns through their analysis of EgoHOIBench performance on HOI-verbs and HOI-nouns. Furthermore, they propose EgoNCE++, an asymmetric contrastive learning objective to address these limitations by enhancing model robustness in handling fine-grained verb and noun variations within HOIs for egocentric video-language pretraining, which successfully fulfills their aim to preserve the object-centric nature of the feature space without additional visual data usage or architectural changes.
weaknesses: •	The paper introduces EgoNCE++ as an asymmetric contrastive learning objective. Figure 3 shows the visualization of LaViLa’s feature space indicating both video and text feature space exhibit the object-centric property and suggests that video-noun matching is easier than video-verb matching since noun-anchored embeddings form tighter clusters, while verb-anchored embeddings are more dispersed. Can we make a feature space comparison after applying EgoNCE++ to see if the verb-anchored changed?

•	The paper mentions that it utilizes the LLaMA-3-8B model to generate HOI candidates through in-context learning. It would be better if we could also conduct additional studies on the performance gains versus computational expense(FLOPs) in order to better know the trade-off for the approach.

•	EgoNCE++ works significantly well on the author’s self-designed benchmarks EgoHOIBench, but it seems that for other datasets the improvement is marginal: EK100-MIR, CharadesEgo shown in Figure.5; EK-100-MIR and EGTEA in Table 9 for zero-shot setting; LaViLa and LaViLa++ in Table 10 and Table 7. Will there be a possible overfitting or lack of variety since the primary performance measure is based on EgoHOIBench, which specifically focuses on nuanced HOI comprehension?

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Has an investigation/analysis been carried out regarding the fact that the loss is designed in the same way as the answers within the dataset? An outcome of this can be seen within Table 4 perhaps, where the choice of generator leads to a large increase in results for HOIBench, but is marginal for the EK-100-MIR task.
2. What does it mean by the sentence not made excessively difficult? Is this because false negatives could be introduced via semantic matching? If so, what is used to prevent this?
3. Why cluster videos based on similar noun representations only, instead of a mix of verb and noun representations? 
4. How are negatives introduced into the video-to-text loss to ensure that false negatives are not included? It is mentioned that either vocabulary from the dataset or an LLM is used to generate the negatives, but there is no information on how this is done. Is this the same as the collection information for HOI-Bench
5. Were all videos chosen for HOI-Bench from the validation set in a similar fashion to EgoMCQ? Or were some videos excluded? How was 10 settled on as the number of captions?
6. When constructing the dataset, has any care been taken to ensure that the LLM used to generate the questions isn't hallucinating? Were any measures put in place to remove noise and ensure a cleaner outcome for the dataset? Also, has any human checking/evaluation been carried out to get a sense of how clean the data is?
7. Has an ablation been carried out in which EgoNCE++ (ours in Table 3) is used for the loss and InfoNCE is used for the V2T loss?
8. For figure 6, has this been evaluated with how the number of negatives scales for the base models without the EgoNCE++ loss? It would be interesting to see how this compares for both models.
9. Has a histogram of positive/negative similarities been created instead of using the new PND metric in Figure 7?
10. It would be interesting to see Figure 3 after the EgoNCE++ objective has been applied if this has been created to see how this differs.
soundness: 2
strengths: * The results of the different Egocentric VLMs on downstream tasks after being trained with the new EgoNCE++ objective are good with nice increases in performance.
* Creating and proposing an asymmetric loss with EgoNCE++ is interesting and makes a lot of sense in how these two settings need to be treated differently whereas in the past this has not necessarily been true.
* There are a lot of results (and qualitative figures within the appendix) to showcase the method and struggles of the current methods without the EgoNCE++ objective.
summary: In this paper, the authors propose a new dataset, named EgoHOIBench, which consists of multiple choice questions for video clips in which there are two settings for verb and noun understanding. There are 10 choices for each case in which the verb or the noun respectively have been changed to create a hard negative. The paper finds that current Egocentric VLMs are unable to handle this change and so a new loss, named EgoNCE++ in which an asymmetric objective forces the model to understand minor differences in text yet for video groups the representations via their noun representations. The results show that for the EgoHOIBench, training current Egocentric VLMs with the new objective leads to an improvement in performance as well as for other downstream tasks.
weaknesses: # Weaknesses
* One potential reason for the new loss doing so well on the constructed HOI-Bench is as the benchmark has been designed in the same way as the loss function. A model that has been trained using a loss with negatives that represent the same style of negatives that are in the ground truth answers is certainly going to do better.
* Models seem to already do well on noun focused tasks, so it isn't clear to me why there is a large focus on still clustering videos based on similar noun representations only.
* Currently, the method section (Section 3) is lacking some important information regarding how the negatives are generated beyond an LLM/using the vocabulary of the dataset. Also, if there is any checking that is done to reduce/remove false negatives.
* The details of how HoI-bench is collected within the main paper is very scarce. The appendix includes some more information, but is still quite lacking. Details of why the number of videos were chosen, number of captions, any choice of category within Ego4D etc. is missing
* Instead of defining a new metric to look at the positive/negative similarities within equation 6/figure 7. A histogram could have been used instead which might have given a clearer picture (again using the max negative). It's less intuitively clear from the figure what the numbers represent, especially as these have been multiplied by 100, emphasising the small differences even further.

# Additional Comments
Table 3 is inconsistent with how it uses lower case and upper case compared to the rest of the paper. Additionally, 'ours' is used here instead of EgoNCE++.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. How does the proposed asymmetric contrastive objective compare with other contrastive techniques in video-language modeling? Could you clarify what specific elements make it novel in this context?

2. Can you elaborate on the robustness of the generated negative captions? Have you evaluated how biases in large language models might affect the model's performance?

3. How does the proposed benchmark compare to existing ones since the improvement is not significant on the existing dataset, and do you anticipate any limitations when applying it to real-world scenarios or other egocentric datasets?
soundness: 3
strengths: 1. The study tackles a key shortcoming in egocentric video-language models (EgoVLMs), which is their limited ability to distinguish subtle changes in interaction descriptions. This issue has significant implications for understanding hand-object interactions, a critical area in ego-centric vision applications.

2. The work introducing a specialized benchmark to evaluate EgoVLMs under challenging scenarios is a valuable contribution. This benchmark exposes performance gaps that were previously under-explored, providing a robust foundation for future research and model improvements.
summary: * Current EgoVLMs, used to understand hand-object interactions in first-person views, can be easily misled by minor changes in interaction descriptions (e.g., changing verbs or nouns).
* A benchmark was created to highlight the limitations of EgoVLMs, particularly their struggle with recognizing verbs compared to nouns, often due to insufficient fine-grained supervision.
* An asymmetric contrastive objective was introduced to improve video-language alignment:
 Video-to-Text Objective utilizes enhanced text supervision with negative captions generated by large language models or HOI-related vocabulary substitutions.
Text-to-Video Objective focuses on an object-centric feature space that clusters video representations based on shared nouns.
* The proposed approach, EgoNCE++, significantly improves EgoVLM performance, especially in tasks like multi-instance retrieval, action recognition, and temporal understanding.
weaknesses: 1. The proposed asymmetric contrastive objective may lack the degree of methodological novelty expected. Similar objectives and contrastive learning techniques have been explored in other contexts as follows:
 * (a) The use of negative mining in contrastive learning is well explored in [1], where it aims to find better hard negatives to benefit contrastive learning.
 * (b) In the field of using augmented captions in multimodal contrastive learning is explored in [2].
 * (c) The use of LLM to generate more text for learning is explored in [3].


2. Since the model is trained and evaluated on a benchmark specifically designed for the study, it’s unclear if these gains will translate to other existing benchmarks or real-world applications. As shown in Tables 3 and 4, the gain over the existing dataset using the proposed dataset is limited. This creates the risk of overfitting to a controlled benchmark, which could raise questions about generalizability.

[1] Robinson, Joshua, et al. "Contrastive learning with hard negative samples." ICLR 2021.

[2] Yuan, Xin, et al. "Multimodal contrastive training for visual representation learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

[3] Fan, Lijie, et al. "Improving clip training with language rewrites." Advances in Neural Information Processing Systems 36 (2024).

Meta-review: The submission addresses the problem of understanding hand-object interactions in egocentric videos. It introduces an asymmetric contrastive learning objective whose goal is to handle fine-grained verb and noun variations when describing hand-object interactions in natural language. It also introduces a new benchmark "EgoHOIBench" to evaluate video-language models' understanding of fine-grained human-object interactions. The proposed approach achieves competitive performance on multiple benchmarks. After rebuttal, the submission received two accepts (8) and two borderline accepts (6). Most of the questions were addressed by the rebuttal. The AC finds no basis to overturn the consensus reached by all reviewers, and thus recommends acceptance.

Decision: Accept (Poster)