HENASY: Learning to Assemble Scene-Entities for Interpretable Egocentric Video-Language Model
Authors: Khoa Vo
Thinh Phan
Kashu Yamazaki
Minh Tran
Ngan Hoang Le
Venue: NeurIPS 2024 poster (2024)
Keywords: egocentric
video understanding
vision langauge models
Primary area: machine_vision

Abstract: Current video-language models (VLMs) rely extensively on instance-level alignment between video and language modalities, which presents two major limitations: (1) visual reasoning disobeys the natural perception that humans do in first-person perspective, leading to a lack of reasoning interpretation; and (2) learning is limited in capturing inherent fine-grained relationships between two modalities.

In this paper, we take an inspiration from human perception and explore a compositional approach for egocentric video representation. We introduce HENASY (Hierarchical ENtities ASsemblY), which includes a spatiotemporal token grouping mechanism to explicitly assemble dynamically evolving scene entities through time and model their relationship for video representation. By leveraging compositional structure understanding, HENASY possesses strong interpretability via visual grounding with free-form text queries. We further explore a suite of multi-grained contrastive losses to facilitate entity-centric understandings. This comprises three alignment types: video-narration, noun-entity, verb-entities alignments.

Our method demonstrates strong interpretability in both quantitative and qualitative experiments; while maintaining competitive performances on five downstream tasks via zero-shot transfer or as video/text representation, including video/text retrieval, action recognition, multi-choice query, natural language query, and moments query.

Project page: https://uark-aicv.github.io/HENASY

HENASY: Learning to Assemble Scene-Entities for Interpretable Egocentric Video-Language Model
Khoa Vo Thinh Phan Kashu Yamazaki Minh Tran Ngan Le
AICV Lab, University of Arkansas, Fayetteville, USA (khoavoho, thinhhp feiinh,,kymmae kyamazak, minht, thile)0uark edu
Abstract
Current video-language models (VLMs) rely extensively on instance-level align- ment between video and language modalities, which presents two major limitations: (I) visual reasoning disobeys the natural perception that humans do in first-person perspective, leading to a lack of reasoning interpretation; and (2) learning is limited in capturing inherent fine- ome-grained relationships between two modalities.
In this paper, we take an inspiration from human perception and explore a com- positional approach for egocentric video representation. We introduce HENASY (Hierarchical ENtities ASsemblY), which includes spatiotemporal token grouping mechanism explicitly assemble dynamically evolving scene entities through time and model their relationship for video representation. By leveraging compositional structure understanding, HENASY possesses strong interpretability via visual grounding with free-form text queries. We further explore suite of multi-grained contrastive losses facilitate entity-centen understandings. This comprises three alignment types: video-narration, noun-entity. verb-entities alignments.
Our method demonstrates strong interpretability both quantitative and qualitative experiments; while maintaining competitive performances on five downstream tasks via zero-shot transfer or video/text representation, including video/text retrieval, action recognition, multi-choice query, natural language query, and moments query.
Project page: https://uark: av...avvv.o...o..... github....... Lo/HENASY
Introduction
Recent advancements in technology and hardware devices for augmented reality (AR) have fueled hopes for virtual assistant applications that can provide users wide range of assistance, such as real-time procedural instructions, moments retrieval, and interactive learning experiences, all through egocentric video streams of similar perspective with user. Publicly available massive-scale egocentric datasets such as Ego4D [ and Epic Kitchens- 100 (2). providing suites of egocentric tasks, have further sparked even more interest within the research community.
Video-language models (VLMs) have currently become a de-facto approach to egocentric video understanding. By learning robust visual-language representations from video-caption pairs VLMs can be applied flexibly to a wide range of downstream tasks, either through zero-shot transfer modality encoders. Existing state-of-the-art (SOTA) VLMs for egocentric videos 4 6
exhibit remarkable performances by following CLIP-like 77] dual-encoder architecture. During training, these models generally lear through the instance-level alignment g between pairs of video and caption representations (Fig. [a)).
However, vidcos consist of complex and dynamic interactions among arbitrary entities, which cannot be effectively captured by simple instance- alignment alone. In fact, a caption contains textual elements that concisely capture video entities. For examples, nouns indicate entity occurrences 4
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

HelpingHands
Vice--lamto Aigmer
Input video:
Visual Grounding:
Cisect aeaac
CRlect gronict
Lear reble
Object
Norres
Enityy
Nouw-Foety Aignneet
Wr-Gr Aigonar
lnailo
GecuewooR
board
Metton
General MLMMs
Okobal Elicodor
Eritt Enteder
Algenani
"id Faccdd
Clip
Narration: et piaces the knite the choppiing
Enoodkrr
Text
Instance level
Video Clip
Narration
fiieggained lecel
(a.)
VLMs
(b) Our proposed framework
(c) Interpretation capscity
Figure 1: Problem Overview. (a) Current VLMs [5] rely on instance-level contrastive learning between video & narration. HelpingHands
implicitly induces object occurrence information into video features at final layer of video encoder. (b) Our proposed (HENASY) aims to assemble dynamic entities from video patches via local entity encoder, while entity-aware decoder captures interactions between entities and global context to form comprehensive video. HENASY is trained with suite of multi-go multi-grained contrastive alignments to enforce visual representations entity-level upto video-level. (c) By such compositional approach, HENASY is the first VLM that shows strong interpretability via visual grounding with both appearance/motion query types.
while verb phrases convey motion information [9] the video. To fully capture these fine-grained alignments, VLM will perform more effectively if it: (1) understand videos in a bottom-up manner, where semantically similar patches form entities, and relationships between entities construct the video representation; and (2) explicitly model fine-grained relationships between video entities and nouns/verbs to comprehensively capture appearance/motton information, respectively.
Human perception aligns closely with the above requirements. We perceive the dynamic surroundings in compositional manner [10), where distinct entities emerge from smaller parts that combine to form whole. Each entity maintains spatial and temporal coherence and interacts with others only when in close proximity. Understanding the compositional structure of the surroundings enables us to intrinsically comprehend and memorize information, while also allowing us to provide interpretations of our decision-making process, which is absent in current egocentric VLMs.
Inspired by such observation, we propose HENASY: Hierarchical ENtities ASsemblY framework (pronounced heh-nuh-see), which follows compositional understanding as in Fig [bb. Concretely, HENASY comprises three key components: (i) Local Entity Encoder, hierarchical transformer- based encoder that learns to assemble dynamic scene-entities from video patches via our proposed spatiotemporal token grouping mechanism, which is an enhanced version from slot-based groupings in stationary images 12: (ii) Global Encoder, a pre-trained video representation module that perceives the input video at global level; and (iii) Entily-Aware Decoder, which models the internal interactions among scene entities and their relationship with the global features, thereby enriching the entity-centric video representation extraction. Furthermore, HENASY is able to perform visual grounding to obtain dynamic segmentations corresponding to either entity or activity with the produced entity embeddings and their attention maps side product of its local entity encoder, showing promising interpretation via dynamic saliency maps across frames (Fig. [c)..
Developing an effective model necessitates strong network architecture and well-defined objectives. With the proposed HENASY architecture, lnstance-level contrastive loss only handles global align- ment, failing to address dynamic entity alignment. Hence, we introduce losses to optimize HENASY for both entity- and video-level representations using narration alone. Specifically, HENASY is trained with three types of alignment: verb-entities. While the first two employ instance-level contrastive loss and model object occurrences via narration's 's nouns, respectively, verb-entity alignment is newly introduced. It aims to incorporate activity/motion information from narration's verb phrases into entities using a one-to-many strategy, which emphasizes the alignment of verb phrase to the most semantically relevant entities. Addi- tionally, we propose a new projection loss that employs detected hand/object boxes [4] to ensure segmentation masks tightly cover respective entities, enhancing HENASY's interpretative robustness.
contrastive
noun-entity, and
We are the first to demonstrate the value of compositional perception approach for egocentric video understanding. Our experiments show that by tasking our proposed local entity encoder to assemble dynamic entities, video representations are effectively improved to outperform current VLMs in wide range of benchmarks, including video retrieval (EgoMCQ ( & Epic SpiicKitchnn-MI (2).
2

---

activity recognition (EpicKitchen-CLe CLS & EGTEA [33 via zero-shot transfer. Furthermore, temporal
localization models [14 [31 equipped with HENASY video/text features can achieve state-of-the-art
performances in episodic memory tasks of EgoNI.Q and EgoMQ [I. Finally, HENASY possesses strong interpretability that is quantitatively and qualitatively superior to current VLMss
Related Works
Video- Video-Language Pre-Trained Models. Pre-training VLMS on a large-scale dataset of video-text pairs and deploying them in down-stream tasks has now become a standard practice. Transformer- powered pre-trained VLMs [6] 17 [8
9 have accomplished superior results wide range of tasks, i.e., text-to-video retrieval, action recognition, or events localization. VLMs can be divided into two common categories, i.e., unified- and dual-encoder. The former models [16 [8 20 fuse multimodal via cross-attention and can be trained with proxy tasks of masked language modeling
19 3
21 22
or masked frame modeling [8 [6]. The latter models [23 5
9 employ separate
encoders for video and text, trained jointly via contrastive learning 8 3
Recently, several VLMs 4 17 [ employ fine- mnee-graied information from captions by decomposing them to capture object/activity through nouns/verbs, respectively. However, these models do not fully exploit fine-grained learning within the video encoder itself, and true granular-level alignment between modalities remains unexplored. In our work, we explicitly model visual content as dynamic entities, capturing their interactions to form a comprehensive video representation. Additionally, our proposed method is trained with multi-grained objectives, ranging from video-text, noun-entity, to verb-entities pairs.
Interpretable Video Representation. There have been efforts to enhance the interpretability of video representations [24 25 26 [7] these typically involved factorizing videos into entities and environmental contexts, and utilizing adaptive attention mechanisms 244 to selectively focus on relevant entities. Such mechanisms enable interpretation through their selection module, highlighting only the primary entities contributing to the final predictions. However, these approaches often require entities to be pre-detected by external models, which is restricted a predefined set of objects. Additionally, these works have focused on models specifically designed for individual tasks, e.g., temporal action detection, video dense captioning. contrast, our work introduces an end-to-end method capable of learning to form entities without an off-the-shel detector. Moreover, we aim to develop a robust aadeo-language model (VLM) that is versatile across a variety of lasks.
Interpretation via Object Discovery. Recent years have seen a growing body of research in end- to-end learning of object discovery, which learns decompose an image
video into distinct objects without direct supervision. Slot-based methods such as IODINE (28] and Slot Attention [T] utilize modell likelihoods [29 to tackle this challenge, demonstrating promising performance through evaluations on synthetic images with simple objects. Subsequently, GroupViT 12] and ODIN [30] incorporate slot attention with contrastive learning and achieved notable success in identifying semantic groupings on natural in-the-wild images. However, these models are not capable of modeling dynamic objects in videos domain. To mitigate this problem, SaVI++ [31 proposes workaround technique, which requires groundtruth depth information in a reconstruction objective to bootstrap object discovery training. In our work, we enhance slot-based grouping mechanisms introduced in ViT [12] to model temporal coherency of dynamic objects in vidcos. Different from [3]], HENASY does not require any extra data further than color video sequences. Instead, HENASY utilizes learned patch features of pre-trained global encoder to bootstrap several early layers of its local entity encoder for entities grouping via a cross-attention mechanism.
3 Preliminaries
Video-language representation learning aims to learn a common latent space to represent video and text. A training dataset for this task comprises of N tuples T where V/ denotes a short sequence of RGB frames, and T; is a free-form text sentence that describes visual content.
Dual-encoder architecture a common paradigm that current SOTA VLMS 4 5
employ for the above task, which consists of (a) visual encoder f mapping the input video V; to visual embedding feature v, = f(V,), and (b) a language encoder g mapping the text Tj to a linguistic embedding feature t; g(T).
3

---

Contrastive-based losses are common objective for langu-language representation. Given a batch of B normalized video-text embedding pairs i = {V, = v.//x.1, L = 1/It.),. a contrastive-base loss pulls embeddings of aligned (positive) pairs close in feature space, while pushing embeddings of misaligned (negative) pairs away. We adopt EgoNCE variation [ of contrastive loss one of our training objectives because of its effective approach in identifying positive and negative pairs. Specifically, each sample i E B is associated to set of positives P. constructed by comparing nouns and verbs across all texts. Additionally, for each sample i, a hard negative i' sampled from a temporally adjacent segment within the same vidco, expanding our batch to B. For a more in-depth discussion the strategy used for sample selection, refer to [3] Herein, the video-to-text objective is succinctly expressed as:
EpEp, exp((' (p/T) EnEB exp(V in/T) + exp((' L. (T)
Σ BEB
log
where T denotes a temperature
B
The objective of text-to-video Cizo is derived from Eq. by inverting and t, and EgoNCE loss is summation of both directions Lego L
4 HENASY
We present the HENASY framework (Fig. 2 for egocentric video-language modeling. HENASY compositional video understanding approach featuring dual-encoder architecture, designed to explore an interpretable, entity-based visual representation. Specifically, besides typically capturing global features via global encoder (Sec. (.I). our video encoder also assembles dynamic scene entities from video patches via local entity encoder (Sec. 4.2). then entity-aware decoder (Sec. 4.3) models their ontra-connections as well inter-connections with global features to form a comprehensive video representation. Our objective is to develop an interpretable reasoning process that robustly supports decision- decision-making, while allowing visual grounding with text queries. To achieve this target, it requires not only an effective network design, but also suite of multi- tittii-rrined contrastive learning to enforce both entity-level and video-level representations.
(Sec. 4.4 4)
For readability, we denote five types of tokens as follows: z for video patch tokens, e for learnable video tokens, g for learnable group tokens, for segment tokens, and for entity tokens.
4.1 Global Encoder
Global encoder provides global visual information of the entire input video. We adopt the pre-trained TimeSFormer (32 to capture the global visual context of the entire input video. Particularly, we follow the protocol of B. 5 4 to decompose the given input video sequence V E Rx* XXW with T RGB frames
resolution H x W into T x K non-overlapping patches of resolution P x P. where K = HW/P². Then, every patch is linearly projected by a 2D convolutional layer, forming video patch tokens ding features at every temporal and spatial location, where D is hidden dimension. TimeS- Former processes the video patch tokens with additional learnable video tokens с E RIXD through a stack of divided space-time attention (DST) blocks, which is described as follows: |c++1,//t = (ST((c';z') where | denotes concatenation operator. Please see Appendix A for more details on the computational process of a DST block.
T RRXx (with TK T K) representing embed-
4.2 Local Entity Encoder
Local entity encoder models fine- fine-grained information of the input video by consistently capturing dynamic scene-entities. We adopt a hierarchical bottom-up architecture, consisting of attention-baat layers that are divided into multiple stages. Each stage progressively groups small video patch tokens z from the previous stage into larger segments. As a result, local entity encoder forms scene entity tokens that depiet individual entities, which dynamically evolve across video frames.
While following GrT [12] to directly process input video patch tokens could be an option, we find doing that diminishes performance. Additionally, the tokens grouping mechanism from 12 not capable of modeling dynamic entities videos domain. To address these challenges, we introduce a bootstrapping stage, which couples itself with early layers of the global encoder through

---

Entity Awaar Decoder
Video Embedding
Aug Pooking
X.
Uatded ST
Atantion S
Terporri-oose Grouping
Divided ST Attention
Global Encoder
Dirited ET Allention X.S.
Lyzz
Hestty
X S
ppogect
A.
Altuntion
Text Encoder
Local Encoder
Scone Elements Extractor
box.
box
#C opens box of saladi
Verb
Nouns
Phrases
Narrations
Figure 2: Overview of the HENASY framework for video-language modeling. Left: HENASY features a dual-encoder architecture with a compositional video understanding approach. The local entity encoder assembles dynamic scene entities from video patches, while the global encoder provides contextual features. These are combined in the entity-aware decoder to create an interpretable video representation. Right: HENASY is supported by suite of multi-grained contrastive learning enforce both entity-level and video-level representations.
cross-attention to group video patches into initial entities' segments. Furthermore, to capture dynamic entities in video, introduce temporal-aware grouping mechanism.
Bootstrapping Stage. Bootstrapping stage consists of S consecutive cross-attention layers, which takes a set of learnable group tokens gooot E R as initial queries (G' is the initial number of tokens). each cross-attention layer starting from O, the queries aggregate information from the patch tokens z at corresponding layer of global encoder:
1+1 Sboot
CrossAtt' (gioott z'), for 0,..,
At the final layer of bootstrapping stage, we obtain ghoot This is then associated with patch tokens from the corresponding layer of global encoder within temporal-aware grouping block (TAG) to group patches into larger segment tokens:
TAG ( gooots z') where I = S1
(3)
Herein, TAG(q, k) merges key tokens k together based on their similaritics with query q. while preserving the temporal dimension of key tokens (discussed in detail later). As result, obtain new segment tokens s' E RTGD which then utilized as inputs to the entity grouping stage.
Entity Grouping Stage. From this point, local entity encoder is decoupled from the global encoder and is trained to merge these input segments into complete scene entities e. At this stage, a new set of learnable group tokens Sentity E Fx is introduced, which aims to relate segment tokens with similar semantics into an individual scene entity. It is important to note that maintaining consistent temporal dynamics is required at each stage. Therefore, we adopt S₃ DST blocks 32 to propagate information mutually between learnable group tokens and segment tokens: gentity
+1
DST( B'attity s']), for S1. ........ S1 +
After the final layer. segment tokens s' are grouped to generate intermediate entity tokens, i.c.. èl -AG ( s') E RTXx where / S Sz. Then, enable interactions between scene entities and across temporal dimension, we apply a stack of S, DST blocks to all entity tokens: a!+1 DST(é'), for S + S2. S1 S2 S3 1.
We observed that segment tokens and entity tokens e facilitate temporal consistencies within the temporal attention of DST block. Unlike in TSF 32), where tokens are spatially limited within patch, these tokens can evolve freely across frames, enhancing the flexibility of the time-attention mechanism in DST block. To obtain spatio-temporal entity embeddings, apply temporally
5

---

average pooling entity tokens of the final layer: e AvgPool(é'). where E IR RxD and = S + +2 S3.
Tewporal-Aware Grouping (TAG). As aforementioned, TAG is employed at the final layers of bootstrapping stage and entity grouping stage, merge semantically similar tokens (i.e., a larger segment while preserving the temporal dimension. Generally, this mechanism takes a set tokens i R (i can be either or s) as inputs and a of learnable group tokens gy gQxD as queries. We re-shape into 3-dimension RTxIxD and evaluate the similarity between B and i.
into
firstly evaluates similarity between each group token and every input token, forming 3D similarity matrix E _T'xxx Then, an assignment matrix A E {0, 1j7xxxi is computed, assigning each input token to the most relevant group based on similarity scores. Finally, it performs per- frame groupings, merging the input tokens of the same group together, forming set of new tokens R TxQxD representing larger segments. We formalize the computation of every new group as
follows:
O - T= O1, (B,, = (Ba): Σj=1 Ar..
Σj=1 ALi
Afterwards, we re-shape RTQxD output of TAG.
The computation of similarity matrix A and assignment matrix Á are detailed in Appendix B. Additionally, the derivation of saliency maps for interpreting assignments between video patches and entities is explained Appendix B.2
4.3 Entity-Aware Decoder
We seek to propagate entity-level features e! from the local entity encoder to the final video embedding for a comprehensive video representation. For this purpose, we introduce entity-aware decoder, which is illustrated in Fig. Entity-aware decoder includes stack of hybrid- refine the interactions between entities and video patches, and render the video embedding. At a block bdecs it first performs noss-attention with entity tokens as queries and patch tokens keys, values. Then, self-attention followed by a multi-layer perceptron (MLP) is applicd over the output:
blocks to
Entity-Aware Embaading
biec CrossAtt(e"dee z!)
Query Entities
(5)
bdact1 MLP(SelfAtt(@
Cross Attention
Salf Attention
Eventually, we obtain the video representation, dubbed as entity-aware video embedding v. by averaging the final outputs of entity tokens:
Embbecalngs
Entity-Aware Decoder
AvgPool(e
(6)
Figure 3: Illustration of entity-aware decoder.
4.4 Multi-grained Contrastive Learning
Beside video-narration contrastive loss B 3]. which captures coarse-grained semantic alignment between the video and narration, we introduce two finer-grained contrastive losses: noun-entity contrastive loss (NEC) and verb-entities contrastive loss (VEC), which focuses on inducing visual appearance and motion cues directly to the composed entities. We also utilize projection loss, leverag- ing object boxes from an off-the-shelf detector 35 as weak supervision to encourage the generated entity masks tightly conforming to the corresponding entity, promoting robust interpretability of our proposed model.
Noun-Entity Contrastive Loss (NEC). From the groundtruth narration, we obtain N,, nouns and their embeddings E RR-,. via the text encoder. Following (4. compute similarity matrix between noun embeddings and entity embeddings. Every noun is matched with an entity token having highest similarity score via Hungarian matching. Following this, we construct a noun-entity contrastive loss using the InfoNCE (8), where positive pairs consist of the matched noun embedding n, and entity embedding ep. The contrast defined over the embeddings n', of all nouns in the dataset taxonomy dictionary D [I:
1
exple np/T) log
CEEC
N
ΣJED exp(efnn//7)

---

Verb-Entities Contrastive Loss is new loss term that instills motion information directly into entity tokens from narration's 's verb phrases. As suggested in 9 that LLMs are superior classical methods such as part-of-speech tagging in retrieving verb phrases, we use Llama-2 134 to obtain N₂ verb phrases from an input narration. Given that a verb phrase describes an activity involving several scene entities, we introduce weighted many- oone- alignment strategy to prioritize the most relevant entity-verb alignments. Firstly, let a; E P be one of the embedded verb phrases, we obtain
a.ee; a normalized similarity scores between a, and every entity ey: s(a., eÿ) " ". Then, re-weight entities by the computed weight and obtain weighted average of entities representation: E s(a,, ey),, Here, cn re-weights each entity based on its relevancy with verb phrase ay. Finally, we compute contrastive loss between this paired representations:
carg Ej
1
N.
exp
((og00 aapp
LVEC
log
(8)
N
ΣJEB exp ((020))T (ayy7)
where we utilize batch formation technique from egocentric contrastive loss in LvEc.
3 to form negatives set
Projection Loss operates on each individual frame of the input video, utilizing an external object detector [33] to identify bounding boxes b (b, E T of scene entities. Let m = (m, E (0, (0,1) 1 /i=1 be the predicted saliency maps of scene entities (see Appendix [B.2 for saliency maps formulation). Hungarian matching pairs each detected box b; with the predicted mask m, having the highest IoU.
Designing differentiable loss function that guides the predicted mask mj by groundtruth box by quite challenging. To address this, we utilize an axis projection function (35] to minimize the discrepancy of vertical and horizontal projections of by and m, on two axes. This ensures that the smallest box encompassing m, matches with by. Concretely, by is firstly converted binary mask format b, E {0, 1] [[H where pixels inside by is assigned by and 0 otherwise. Then, a projection loss is defined as follows:
No.
Gproj
(Laiee max(m;); max(b;)) Latice (max(my), max(b,)) x
No
where Ldice is Dice loss function 36), max,(+) and max, (.) are max- max- max-project operators along y-axis and z-axis of the frame, respectively.
Total Optimization. Overall, our model is optimized with weighted sum of EgoNCE loss over video-text pairs and three objectives stated above:
L = Luze L + Ci + A2L + Aapproj
(10)
where \1, X2, and 0 balance contributions of different loss terms.
Experiments
5.1 Training and Implementation Details
Architecture. We use video clip inputs of size 224 x 224, text inputs are tokenized and processed by
12-layer Transformer following [5). We employ Former 32 Base (TSF-B) for the global encoder. In the local entity encoder, all layers share a hidden dimension
= 512. Bootstrapping stage includes S = 6 cross-attention layers with 64 group tokens. Entity grouping stage consists of
S2 = 3 DST blocks with group tokens, followed by S3 = 3 DST blocks. Entity-aware decoder is a stack of 3 byyrrd-attention blocks.
Training. HENASY is trained on EgoClip which contains 3.8M clip-narration pairs covering sub-set of 2.927 video hours from Ego4D For each video clip, we uniformly sample 4 frames. We employ the pre-extracted narration's 's nouns and pre-detected hand and object bounding boxes
from 4 for NEC loss and projection loss, respectively. For verb phrases, we employ Llama-2 (34
with a prompt as discussed in Appendix The loss weights in Eq. [0 are set as: À = 0.5, A 0.5, >a 1.0. We train HENASY on two A6000 GPUs, in 5 epochs with AdamW optimizer [37 fixed learning rate of 3e 5, and with batch size of 128. We initialize global encoder and text encoder with pretrained model provided from [5], but freeze them in the entire training process.
7

---

Table 1: Comparison on the zero-shot transfer over EgoMCQ. EKI00-MIR, EKI00-CLS, and EGTEA. HelpingHands* refers our re-produeed results with TSF-B backbone using provided
source code
EgoMCQ
EK100-MIR
EK100-CLS
FGTEA
Methods
mAP
nDCG T-V
Top-1 Acc
Top-5
Top-1 Acc
Mean
Inter
Intra
V-T
T-V
Avg
V-T
Avg
Acc
Acc
Ego [
90.6
57.2
26.0
20.6
23.3
28.8
27.0
27.9
17.6
EgoVI.Py2 6
91.0
60.9
26.7
29.1
LaViLa
93.8
59.9
35.1
26.6
30.9
33.7
30.4
32.0
16.4
34.4
35.5
28.9
Handss
93.2
58.8
35.6
26.8
31.2
34.7
31.7
33.2
35.3
29.4
Ours
94.1
61.3
35.5
27.1
31.3
34.6
31.7
33.2
19.5
38.2
35.9
29.6
Table 2: Comparison on the visual & textual representation over EgoNLQ and EgoMQ. Grey indicates result we obtained using provided pre-trained checkpoint that.
EgoNLQ
EgoMQ
Methods
mloU@e
@0.3
mloU@U,
RI @0.5
R5@0.5
mAP
R1
R5
RI
RS
SlowFast
38
5.5
10.7
3.1
6.6
25.2
46.2
EgoVLP
3
10.8
18.8
6.8
13.5
30.1
52.0
11.4
LaViLa(B) S
10.5
19.1
6.7
13.6
27.4
49.0
11.3
HelpingHlands*
11.2
20.4
6.9
14.7
27.5
49.0
11.7
Ours
11.5
21.5
7.0
14.7
28.3
51.0
12.4
5.2 Benchmarks and Evaluation Protocols
Ego4D benchmarks [1] Ego4D is the largest publicly available egocentric video dataset, featuring 3,670 hours of daily-life activity video for wide range of benchmarks. We evaluate three tasks:
EgoMCQ 33: A multi-choice questions task to select the correct video clip from candidates for
each query. Accuracy is evaluated in intra-/inter-video (candidates from the same/different video). EgoNI.Q: sub-task in episodic memory involving localizing video intervals that answer a given a free-form text query. Evaluation metrics include Recall K for mloU thresholds 0, where
K E 5} and 0 E {0.3, 0.5).
EgoMQ: Also sub-task of episodic memory, involves identifying and categorizing action instances from 110 activity classes. Evaluation metrics are recalls (at mloU=0.5) and mean Average Precision (mAP).
EpicKitchens 100 benchmarks [2]: This dataset focuses on indoor and kitchen activities with 100
hours of video. We evaluate tasks:
FK100-MIR: multi-instance retrieval task evaluating video and narration matching both T-VV and V-+T. Metrics mAP and normalized Discounted Cumulative Gain (nDCG).
100-CLS: action recognition task classifying videos into 300 noun classes or 97 verb classes.
Metrics are Top-1 and Top-5 accuracy.
EGTEA benchmark [3]: This dataset contains 28 hours of video with 106 classes. We evaluate
fine-grained cooking action recognition EGTEA and report Top-1 and mean accuracies.
Evaluation Protocols. We evaluate our model using three protocols:
Zero-Shot Transfer assesses generalization on unseen data and tasks without extra tuning. We conduct zero-shot evaluation EgoMCQ, EK 100-MIR, EK 100-CLS, and EGTEA.
Visual & Textual Representation is evaluated through EgoNL.Q and EgoMC, where we use our pre-trained model
visual/textual feature extractor. Following 4 ()) train downstream models (VSLNet [15] for EgoNLQ, VSGN 14) for EgoMC) with pre-computed features.
Vision-Language Grounding. We evaluate local entity understanding and interpretation via qualita- tive results on EgoCLIP 3 We illustrate the saliency maps produced by our model and compare
with bounding boxes from

---

Table 3: Ablation results on multi-grained losses.
Loss Settings
EgoMCQ Inter
EK100-MIR
EKI00-CLS
Avg mAP
Avg nDCG
Top-1
Top-5
Lega
LN
EVEC
Lyros
Acc
Acc
x
x
x
93.4
58.4
30.8
32.7
18.2
36.8
?
X
X
93.6
59.9
30.9
32.8
19.1
37.5
x
93.7
32.9
18.9
37.3
x
X
93.2
58.5
30.8
32.6
18.5
37.0
X
94.0
61.1
31.3
33.0
19.3
37.7
94.1
61.3
31.3
33.1
19.3
38.2
5.3 Main Results
Comparison in Zero-shot Transfer. In Table to ensure fairness, we re-train HelpingHands 4 using their official codebase with TSF-B backbone and the same pre-trained weights ours, provided
by LaVila a [5] Our model consistently outperforms previous SOTA, achieving 3. 5.1% improvement in top-1 accuracy
EK EK100-CLS, 0.5% and 0.3% increase in intra- and inter-video accuracy EgoMCQ, and 0.5% improvement in mean accuracy on EGTEA. It also competes competitively with HelpingHands in the video/text retrieval EKI00-MIR, Overall, our method demonstrates strong performance for zero-shot transfer across multiple benchmarks.
Comparison in Visual & Textual Representation. In Table 2 method outperforms prior SOTA models across all metrics in EgoNLQ by adequate gaps. In EgoMQ. HENASY shows comparable performance, particularly excelling in mAP where it surpasses SOTA by 1%. This highlights HENASY's effectiveness when being applied to downstream models for features extraction.
Vision-L.angun Languae Grounding We include qualitative experiment (Fig. to compare with Help- ingHands [4], which, our knowledge, the only VLM including weak visual grounding capacity via bounding boxes. As we can HENASY provides stronger interpretation with saliency maps reflecting dynamically evolving regions that most related to both appearance and motion queries. Furthermore, HelpingHands cannot correctly perform grounding with verb phrases (e.g., "scrolling the phone"), therefore, we show the bounding box of noun instead (e.g., "phone").
5.4 Ablation Studies
Imput video pasucting mundhruth
Trus sidee
We investigate various com- binations of multi-grained loss ponents and report results in Table 3 We found that HENASY trained only with instance-level loss Lego yields 1- 2% lower across all benchmarks com- pared to full loss setting. Besides, Lv contributes slightly more to performance gains in EgoMCQ and EK100-MIR, compared LNEC. Fi- nally, Lproj shows a slight improve- ment of the overall performance.
Losses:
Bustban
Namuton: scrolling phone
Helpingllande:
Helpingllande
Impact of Entity-Aware Decoder: We evaluate the influence of the entity- aware (EA) decoder on zero-shot tasks in the first rows of Table 4 In the first experiment (labeled as "w/ avg. pool'), the proposed EA de- coder is omitted, and entity tokens are merely processed using average pool- ing to generate the video representa-
Figure 4: Vision- Grounding, Qualitative com- parisons with HelpingHands [4] on EgoCLIP [3]. Left: com- parison with query obtained from narration and the sccudo-groundtruth boxes detected by [33] for reference. Right: verb phrase in the narration is used for comparison, as verb phrase cannot be captured by 33 we do not include pseudo boxes.
tion. This configuration results
notable decline in performance across all benchmarks. In the second experiment (labeled as "w/

---

Table 4: Ablation results on model design.
Table 5: Comparison computational com- plexity and memory cost.
Model designs
BBOMCQIEK000-MI EKI00-CLS
Avg
Avg
Top-1
Top-5
HelpingHlands Ours
Inter Intra
mAP
nDCG
Acc
Acc
Autoregressive
w/ avg. pool
87.6 47.9
18.8
25.5
6.7
18.1
GFLOPss video chip (Million)
530
599
Number Parameters (Million)
216
291
w/ SA dec.
93.3 59.1
30.4
32.8
18.0
36.3
Train GPU Memory (GB)
w/o bootstrap
192.6
59.2
31.1
32.6
19.2
37.9
Inference GPU Memory (CB)
4.4
4.8
Inferensee Time (seconds)
2.87
1.02
complete settings 94.1 61.3
31.3
33.2
19.5
38.2
SA Dec.'), video patch tokens are combined with entity tokens and then input into self-attention decoder, which has the same dimensions as our EA decoder. This setup leads to a performance decrease of approximately 2% across benchmarks compared our proposed decoder ('complete settings').
These ablation studies show that modeling interactions between global features and entity embeddings plays a critical role, and the proposed design of entity-aware decoder significantly enhances overall model performance.
Impact of Bootstrapping Stage: We report the effect of bootstrapping stage in the third row of Table (labeled w/o bootstrap), where remove bootstrapping stage by directly processing video patch tokens. The performance degrades by 1 % across all benchmarks, showing the effectiveness of this design choice.
Table Our model is slightly more expensive but quite competitive in terms of memory requirements, the number of parameters and GFLOPs. Importantly, our inference time is 3 times faster than that the HelpingHands. This superior running time of HENASY compared to HelpingHands can be attributed to HelpingHands' utilization of an autoregressive decoder, which reduces parallel computations and makes it less efficient despite its lower computational cost.
Computational and Memory Costs: We compare our method with HelpingHands
6 Conclusions
In this work, we explored the Hierarchical Entities Assembly framework, dubbed HENASY, which is designed improve video representation of previous vision-language models by addressing their limitations in fine-grained modeling. Our model explicitly captures the dynamic interactions between visual entities to form comprehensive video representation. Our experiments showed that HENASY outperforms existing SOTA methods across challenging egocentric video understanding benchmarks like EgoMCQ. EK 100-MIR, EK 100-CLS, EgoNLQ, and EgoMQ in both zero-shot transfer and feature extraction settings, while also demonstrating strong interpretation capabilities. Despite these strengths, there are several opportunities for future work to improve our model further.
Limitations and Future Works Although our focus has been on tasks utilizing VIT encoders for variety of benchmarks, believe it is important extend HENASY to generative tasks such as video generation (e.g., stable diffusion) or to handle long-form videos. While HENASY can provide interpretability by focusing relevant scene entities for both objects and actions, it is still limited in explicitly showing the interactions between scene entities. This necessitates the development of a dynamic scene graph, which remains an open question due the unavailability of data.
Acknowledgements
The authors gratefully acknowledge funding supports from U.S. National Science Foundation (NSF) under Award No. OIA-1946391 and NSF EFRI BRAID 2223793.
10

---

References
[1] Kristen Grauman, Andrew Westhury, Fugene Byme, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. EgoAd: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 18995-19012. June 2022.
[2] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma. Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchen--00. International Journal of Computer Vision (WCV), 130:33 55, 2022.
[3] Kevin Qinghong Lin, Jinpeng Wang. et al. Egocentric video-language pretraining. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[4] Chuhan Zhang, Ankush Gputa, and Andrew Zisserman. Helping hands: An object-aware ego-centric video recognition model. In International Conference on Computer Vision (ICCV), 2023.
[5] Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar, Learning video representations from large language models. In CVPR, 2023.
[6] Shraman Pramanick, Yale Song, Sayan Nag, Kevin Qinghong Lin, Hardik Shah, Mike Zheng Shou, Rama Chellappa, and Pengehuan Zhang. Egovlpv2: Egocentric video-language pre- pre-tranning with fusion in the backbone. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 5285-5297
October 2023.
[7] Alec Radford, Jong Wook Kim, et al. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings the 38th International Conference on Machine Learning, ICML 2021, volume 139 of Proceedings of Machine Learning Research, pages 8748 8763, 2021.
[8] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2019.
[9] Liliane Momeni, Mathilde Caron, Arsha Nagrani, Andrew Zisserman, and Cordelia Schmid. Verbs in action: Improving verb understanding in video-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15579-15591, October 2023.
[10] Irving Biederman. Recognition-by-compooeee review, 94 2:115 2:115-147, 1987.
theory of human image understanding. Psychological
[11] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 11525 11538. Curran Associates, Inc., 2020.
[12] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 181 11134-88444 June 2022.
[13] Yin Li, Miao Liu, and James M. Rehg. the eye of beholder: Joint learning of gaze and actions in first person video. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision ECCV 2018, pages 639 655, Cham, 2018. Springer International Publishing.
[14] Chen Zhao, Ali K. Thabet, and Bernard Ghanem. Video self-stitching graph network for temporal action localization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 13658-1366 October 2021.
[15] Hao Zhang, Aixin Sun. Wei Jing, Liangli Zhen, Joey Tianyi Zhou, and Rick Siow Mong Goh. Natural language video localization: A revisit in span- based question answering framework. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(8):422 4266, 2022.
[16] Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L. Berg. Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for sideo-and-land langugge learning via sparse sampling. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 7331-7341 June 2021.
11

---

[17] Yuying Ge. Yixiao Ge, Xihui Liu, Dian Li. Ying Shan, Xiaohu Qie, and Ping Luo. Bridging video-text retrieval with multiple choice questions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16167-16176, June 2022.
[18] Jinpeng Wang, Yixiao Ge. Rui Yan, Yuying Ge, Kevin Qinghong Lin, Satoshi Tsutsui, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. All in one: Exploring unified video-language pre- ve-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 6598-6608, June 2023.
[19] Thinh Phan, Khoa Vo, Duy Lc, Gianfranco Doretto, Donald Adjeroh, and Ngan Lc. Zeetad: Adapting pretrained vision-language model for zero-shot end-to-end temporal action detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 7046-7055, January 2024.
[20] Trong Thang Pham, Jacob Brecheisen, Anh Nguyen, Hien Nguyen, and Ngan Le. I-ai: A controllable & interpretable ai system for decoding radiologists' intense focus for accurate cxr diagnoses. In Pruceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 7850-7859. January 2024.
[21] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and AunMichaall Frahm, editors, Computer Vision ECCV 2020, pages 104-120, Cham, 2020. Springer International Publishing.
[22] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsich, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language, 2019.
[23] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In IEEE International Conference on Computer Vision, 2021.
[24] Khoa Vo, Hyekang Joo, Kashu Yamazaki, Sang Truong, Kris Kitani, Minh-Triet Tran, and Ngan Lc. Aci: Actors-environment interaction with adaptive attention for temporal action proposals generation. In 32nd British Machine Vision Conference 2021, BMVC 2021, Viriual Event, UK, November 22-25, 2021, 2021.
[25] Khoa Vo, Sang Truong, Kashu Yamazaki, Bhiksha Raj. Minh-Triet Tran, and Ngan Le. Aoc-net: Enti- ties interactions modeling with adaptive attention mechanism for temporal action proposals generation. International Journal of Computer Vision, Oct 2022.
[26] Kashu Yamazaki, Sang Truong. Khoa Vo, Michael Kidd, Chase Rainwater, Khoa Luu, and Ngan Le. Vlcap: Vision-language with contrastive learning for coherent video paragraph captioning. In 2022 IEEE International Conference on Image Processing (ICIP), pages 3656-3661, 2022.
[27] Kashu Yamazaki, Khoa Vo. Quang Sang Truong, Bhiksha Raj. and Ngan Le. Vltint: Visual-linguistt transformer-in in-transformer for coherent video paragraph captioning. Proceedings of the AAAI Conference on Artificial Intelligence, 37(3):388130999 Jun. 2023.
[28] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerehner. Multi-object representation learning with iterative variational inference. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2424-2433. PMLR, 09-15 Jun 2019.
[29] Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuher. Neural expectation maximization. In
I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[30] Olivier J. Hénaff, Skanda Koppula, Evan Shelhamer, Daniel Zoran, Andrew Jaegle, Andrew Zisserman, João Carreira, and Relja Arandjelovié. Object discovery and representation networks. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella. and Tal Hassner, editors, Computer Vision ECCV 2022, pages 123 113,1,,,000 Cham, 2022. Springer Nature Switzerland.
[31] Gamaleldin F. Elsayed, Aravindh Mahendran. Sjoerd van Steenkiste, Klaus Greff, Michael C. Mozer, and Thomas Kipf. SAVi++: Towards end-to-ena object-centric learning from real-world vidcos. In Advances in Neural Information Processing Systems, 2022.
[32] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In Proceedings of the International Conference Machine Learning (ICML), July 2021.
12

---

[33] Dandan Shan, Jiaqi Geng, Michelle Shu, and David F, Foubey. Understanding human hands in contact at internet scale. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
[34] Hugo Touvron, Louis Martin, et al. Llama 2: Open foundation and fine-tuned chat models, 2023.
[35] Zhi Tian, Chunhua Shen, Xinlong Wang, and Hao Chen. Boxinst: High-performance instance segmentation with box annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 5443 5452, June 2021.
[36] Carole H. Sudre, Wengi Li. Tom Vereauteren, al. Generalised dice overlap deep learning loss function highly unbalanced segmentations. Deep Learning in Medicul Image Analyxis and Maltinodal Learning for Clinicul Decision Support, pages 240-248, Cham, 2017. Springer International Publishing.
[37] Ilya Loshchilov Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[38] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, Kaiming He. recognition. In ICCV, October 2019.
Slowfast networks for video
[39] Eric Jang. Shixiang Gu, and Poole. Categorical reparameterization with gumbel-softmax. In International Conference on Learning Representations, 2017.
[40] Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learning.
I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
13

---

A Divided Space- Time Block
Divided Space-Time (DST) block [32] is mainly utilized in the global encoder and entity grouping stage of the local entity encoder in our HENASY framework.
In global encoder, DST typically takes a concatenation learnable CLS token and video patch tokens, i... [c': as inputs. While in the local entity encoder, inputs to DST comprises segment tokens Sentity and segment tokens s'.
A DST block reduces computational cost of full space-time attention by factorizing it into time and space attention, consecutively:
8/,4- = p=1 Softmax {(th: k// ss/vall dn
v
K yi,E Σx k== Softmax {(ak. 1/ /dn/ Vi.
where gi kl vh, E Rd are query, key, and value vectors, respectively, which are lin- early projected from the input of DST block after being split by number of heads. Likewise, L. k V2 E RRdh are query, key, and value vectors derived from Vi,.-
B
Temporal-Aware Grouping
B.1 Details of Tokens Assignment and Grouping
Similarity Computation.
Given learnable group tokens 8g E RQxD and input tokens to be grouped E TxxXxx, we follow 12 to compute the 3D similarity array A xTxQx// between each video-level group token gi E ga and every segment token it,j k, where t and j are temporal and spatial indices, respectively. Gumbel-Sofftax
339 is then applied to rescale similarity matrices over group tokens:
cxp (Wyg) Wic +%)
AI,.J
(11)
Ee exp (Waq: Wiie + +7k)
where W, and W; are learned linear projections for group and segment tokens, respectively, and % sampled from Gumbel(O,) 1) distribution.
Group Assignment. Afterwards, segment token hardly assigned to group token via arg max operation over group tokens (non-diiferreniaabl) with the straight-through trick [40] to allow end-to- end training:
Á one-hot arg max A) + -sg(A)
(12)
where sg(.) is a stop-gradient function, and one-hot(-) operator converts the assigned group indices into one-hot vectors.
B.2 Saliency Map Generation
Saliency maps of each dynamic entity that evolving across frames of input video can be constructed from similarity arrays produced in temporal-aware grouping layers bootstrapping and entity grouping stage. Let denote them as A boot and A entity respectively. We first compute the assignment probability array between video patches at each frame and final entity tokens by the following equation:
M, = A boot (A, (Antsit
(13)
where i is frame index in T, and M RTxXxE (K is the number of patches per frame). Then, saliency maps can be obtained via softmax activation function over the patches M softmax (M). Splitting the saliency array M over entity dimension, we can obtain saliency maps all frames, each of which highlights the spatial location and shapes of the corresponding entity.
14

---

C Verb Phrase Generation
We utilize Llama-2 3
generate verb phrases from narration due its superior performance in processing free-form texts. Below is prompt we design to capture verb phrases:
System: "Act as if you are robot that only outputs python list of strings."
User: "Task: You are given an input sentence. Your job to output the action verb phrases,
which always starting by verb-ing."
D Quantitative evaluation on visual grounding
We conducted a rigorous quantitative analysis on Ego4D dataset to compare with the SOTA model of HelpingHands, they only provide visual grounding results this datasets. We create semantic segmentation labels by ourselves for 200 videos. The results are reported under mloU metrics between visual grounding prediction with corresponding groundtruth, showing our model's superior visual grounding capability compared to HelpingHands:
Model
mloU
HandsngHands
22.73%
Ours
41.06%
Table 6: Comparison with HelpingHands visual grounding task.
Despite being the SOTA in egocentric tasks and visual grounding task, HelpingHands 4 only provides coarse bounding boxes of objects, leading to much lower mloU scores due to inadequate coverage of the target masks. In contrast, our model employs segmentation masks that closely align with the ground truth, resulting in higher mloU scores, demonstrating superior visual grounding capabilities.
Discussion:
15

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We provided experiments to support our claims.
2. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Our paper follows the NeurIPS Code of Ethics.
3. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our work helps to reduce the carbon footprint when training large models using VIT. There is no negative societal impact.
4. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We included it at the end of the paper.
5. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: All assumptions are already stated.
6. Experiments
Question: Does the paper provide appropriate and extensive experiments prove the significance the their method
Answer: [Yes]
Justification: We provided experiment descriptions in the main paper and appendix.
7. Training Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We provided details for experiments in the paper.
8. Error Bars
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We conduct experiments on diverse datasets and follow the protocol used by previous works for fair comparisons.
9. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed reproduce the experiments?
16

---

Answer: [Yes]
Justification: This information is included in our results.
10. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provided experiment descriptions in the main paper and appendix. Further- more, we will release our GitHub implementation soon.
11. Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: Our work does not pose such risks.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have properly cited papers and resources used in experiment.
13. Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We don't have experiments involving crowdsourcing research with human subjects.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We do not involve crowdsourcing research with human subjects.
17


Reviews:
[{"review_id": "vVTB9OEaCT", "reviewer": "Reviewer_EqYJ", "summary": "This paper presents HENASY (Hierarchical ENtities ASsemblY), a pretraining framework to learn scene-entities representations for egocentric videos. The authors proposed to learn compositional and hierarchal video representations by three levels: 1) global video features from a global video encoder. 2) entity features by assembling dynamic entities from video patches via local entity encoder, 3) interactions between entities and global context by an entity-aware decoder. With the designed model structure and losses, the authors conducted experiments on several egocentric tasks on two datasets. The results show the effectiveness of the proposed method.", "questions": "See weakness section.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The idea is well-motivated. The analysis in the introduction part that current simple instance-level video-caption alignment being hard to capture the complex and dynamic interactions among arbitrary entities are inspiring. The proposed method of utilizing both global information and local entitiy-level info is novel.\n2. The authors conducted extensive experimets across multiple tasks including zero-shot transfer, down-stream finetuning and vision-language grounding. The experiments result show the effectiveness of the proposed method.\n3. The ablation study (table 3 and 4) show the contribution of each proposed components clearly, providing intuiation and experience for future model design.", "weaknesses": "1. My major concern is the performance shown in the downstream task. In table 1, though the proposed methods generally achieved better performance compared with previous sota models (HelpingHands), the improvement seems to be minor. Besides, the paper didn't report the error analysis to dimish the influcen of randomness, which makes the contribution and effectiveness of the proposed method less convincing.\n2. In Projection loss calculation, how is the mask predicted is not clearly presented.\n3. The writing could be further improved. In line 127 it should be $L_{ego} = L_{ego}^{v2t} + L_{ego}^{t2v}$. In section 4, too many variables are used including z, c, g, s, e, making it hard to follow the context while reading.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful evaluation and acknowledgment that our approach is well-motivated, novel, and effective under extensive experiments, providing intuition and experience for future model design. We will fix all noticed typos in our final version. Below, we would like to address the concerns raised:\n\n## **1: Clarify the improvement over most recent SOTA model (HelpingHands)**\n\nWe would like to recap the differences between our work and HelpingHands by the following table:\n\n| Property | Our Work | HelpingHands |\n| --- | --- | --- |\n| Method | Explicitly models video as dynamic entities (object-centric), capturing their interactions (action-centric) to form spatiotemporal interpretable, object/action-aware video representation | Implicitly induces object occurrence (nouns) on top of pre-extracted feature map via auxiliary task-specific (detection) head to form an object-aware video representation |\n| Fine-grained alignment | Spatiotemporal fine-grained alignments. Object appearance: Noun-entity alignment. Activity motion: verb-entities alignments | No alignment at granular level |\n| Interpretability | Strong, both spatial and temporal dimensions with object-centric and action-centric via dynamic entity assembly mechanism | None. |\n| Visual grounding | Strong, with spatiotemporal saliency maps of scene entities and relationships | Weak, only provide predicted bounding boxes |\n| Efficiency | faster than HelpingHands x3 times in inference | slower x3 times in inference due to autoregressive decoder |\n\nThis discussion is included in Section 2, lines 87-93 of the submission.\n\nIn addition, we highlight our improvements over HelpingHands across various metrics as below (Tables 1 and 2 in the paper):\n\n| Benchmark | Metric | HelpingHands | Our Method | Improvement |\n| --- | --- | --- | --- | --- |\n| EgoMCQ | Intra Acc | 93.2 | 94.1 | +0.9 |\n| EgoMCQ | Inter Acc | 58.8 | 61.3 | +2.5 |\n| EGTEA | Top-1 Acc | 35.3 | 35.9 | +0.6 |\n| EgoNLQ | mIoU @0.3 R5 | 20.4 | 21.5 | +1.1 |\n| EgoMQ | mAP | 11.7 | 12.4 | +0.7 |\n\n**Discussions:** We emphasize that the primary goal of our proposed method is to advance interpretable video representation by leveraging a compositional video understanding approach. This would create a new vibe to our research community from solely competing on benchmarks to prioritizing trustworthiness, transparency, and reliability. Although the improvements in some metrics may appear incremental, they are substantial within the context of interpretability and compositional analysis, as it pursuits of building transparent and accountable AI systems. Equipping the model with interpretability to ensure trustworthiness while enhancing performance is crucial for practical applications, especially in sensitive areas such as healthcare and autonomous driving.\n\n## **2: Error Analysis to diminish the influence of randomness**\n\nReproducibility is highly prioritized in our work, and as such, we have implemented fixed seeds in all random-related functions for every experiment to ensure that the community can reliably reproduce the performance reported in our paper.\n\nIn response to this concern, we initiated evaluations of our model trained with different random seeds. However, due to the limited time available for the rebuttal and resource constraints, we can only train two additional models. We report the performance in terms of mean and std across several zero-shot benchmarks as below:\n\n| Benchmark | Metric | Our performance (mean±std) |\n| --- | --- | --- |\n| EK100-CLS | Top-1 Acc | 19.20 ± 0.51 |\n| EK100-MIR | Avg mAP | 31.07 ± 0.25 |\n| EGTEA | Top-1 Acc | 35.5 ± 1.1 |\n| EgoMCQ | Inter Acc | 59.74 ± 0.21 |\n\nThe above table with high mean and low std indicates that our method is effectiveness, stable, robust and converges well under randomness.\n\n## **3: Formulation of masks prediction for Projection loss**\n\nIn Appendix Section B, we provided a comprehensive description of three steps involving the formulation of the masks prediction. Specifically:\n\n1. **Similarity Computation**: We calculate a similarity array between the learnable group tokens and the input tokens.\n2. **Group Assignment**: Each input token is assigned to a group token based on the highest similarity score. To retain differentiability essentially for training, we employ the straight-through trick during the assignment step. Obtaining assignment array.\n3. **Saliency Map Generation**: Saliency maps are then generated using the assignment. These maps effectively highlight the spatial locations and dynamic contours of entities across different frames of the video, providing a visual representation of the model’s focus and understanding of the scene.\n\nThe complete mathematical formulation and additional details are thoroughly explained in Appendix Section B, ensuring transparency and reproducibility of our methods. We will add reference to corresponding appendix in the projection loss description to make it clearer.\n\n## **4: Clarification on the use of multiple variables**\n\nWe use abbreviation to name variables for clarity and succinctness in mathematical formulations.\n\nTo recap, there are five types of tokens as follows:\n\n- z: video patch tokens, which capture local features from video frames.\n- c: learnable <CLS> token, which is used to aggregate and represent global video features.\n- g: learnable group tokens, which are pivotal in organizing video information into meaningful clusters.\n- s: segment tokens, which are derived from segmenting video patch tokens after the bootstrapping stage.\n- e: entity tokens, which represent distinct entities clustered at the final stage of the process.\n\nTo aid readability, we plan to include a notation table in the appendix, summarizing all variables and functions with clear descriptions and references to their respective sections. This will facilitate easier navigation and understanding of the methodology for all readers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a1DI6mzPIl", "reviewer": "Reviewer_xaMC", "summary": "The paper introduced HENASY, a novel framework for enhancing video-language understanding in egocentric videos. It utilized multi-grained contrastive losses from alignments of  video-narration, noun-entity, and verb-entity, to improve interpretability and performance. The method showed competitive results across various downstream tasks such as video-text retrieval, action recognition and visual question answering.", "questions": "My questions are listed in the part of weaknesses above.", "limitations": "They had a separate \"Limitations\" section and elaborated on future works that could improve the model.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The paper was well organized and easy to follow.\n* This paper leveraged more fine-grained information on egocentric video-language models apart from the original instance-level alignment. In detail, it included noun-entity and verb-entity alignment as a bonus.\n* The framework achieved competitive results in various downstream tasks, such as video/text retrieval and action recognition, showcasing its effectiveness in real-world applications compared with other pre-trained models.", "weaknesses": "* Such an idea of leveraging fine-grained information, especially alignment of nouns/verbs and the video contents, has been intensively studies in non-egocentric scenarios and there are a number of related papers [1, 2]. It is not clear whether there is a significant different in egocentric videos. Otherwise, it is expected to improve the model performance by incorporating those fine-grained alignments in the pre-training stage.\n* Even though the structure to extract noun/verb entities was a little different, I feel like the essential idea was still similar to GroupViT. The authors mentioned that if directly using GroupViT, the performance would drop. Was the comparison in Table 3, w/o bootstrapping? Actually, in GroupViT there was also a hierarchical design to use different query tokens at different stages. Then what's the main contribution of the design in the paper? I am wondering if simply adding the number of trainable parameters of GroupViT to the same number in the paper would lead to comparable performance.\n* I am wondering if co-training with data mixes from both egocentric and non-egocentric videos would boost performance for both domains. In addition, can the model pre-trained on the egocentric scenario be transferred to the non-egocentric tasks?\n* The evaluation on visual grounding was not sufficient. Currently there were only some qualitative results in Figure 4. It would be better to report some quantitative results for more concrete tasks.\n\n[1] Ge, Yuying, et al. \"Bridging video-text retrieval with multiple choice questions.\" CVPR 2022.\n\n[2] Xiong, Yuanhao, et al. \"Structured Video-Language Modeling with Temporal Grouping and Spatial Grounding.\" ICLR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing that our paper is well-organized, easy to follow, competitive results in real-world applications.\n\n## **1: Comparison with [1, 2]**\n\nWe provide the comparison as below:\n\n| Property | Ours | [1] | [2] |\n| --- | --- | --- | --- |\n| Method | Explicitly models video as dynamic entities (object aware), capturing their interactions (action aware)  | Employs a proxy module (BridgeFormer) to implicitly inject fine-grained information of text into video representation during training | Explicitly models video via group tokens (object aware). It is unable to capture the  interactions between objects (action aware) |\n| Fine-grained alignment | Spatiotemporal fine-grained alignments. Object appearance: Noun-entity alignment. Activity motion: verb-entities alignments | No granular level alignment | Only alignment on object appearance: Noun-entity alignment |\n| Interpretability | Strong, both spatial and temporal dimensions with object-centric and action-centric via dynamic entity assembly mechanism | None, the proxy module is removed during inference, rendering the model incapable of interpretation | Medium, only object-centric |\n| Visual grounding | Strong, with spatiotemporal saliency maps of scene entities and relationships | None | Weaker, only saliency map related to a scene entities |\n| Efficiency | 2x A6000 GPU for training | 40x A6000 GPUs for training | No source code nor report |\n\nWe note that [2] is published at ICLR 2024 (2 weeks before NeurIPS submission).\n\n## **2: Comparison with GroupViT**\n\nTo illustrate the challenges with merely applying GroupViT to video, how our method addresses these issues, and what set us stood out from GroupViT, we provide the following table:\n\n| Limitation in GroupViT | Our Innovative Solution |\n| --- | --- |\n| Domain: image | Domain: video |\n| Direct Processing of Patch Tokens: GroupViT learns to process input patches from scratch, which prevents it from utilizing the powerful representation of existing pre-trained models | Bootstrapping Stage: incorporates early layers of global encoder (using a frozen model from LaViLa) to leverage its powerful video representation. The effectiveness of this design choice is reported in our ablation (Table 3, row 3) |\n| Inability to Model Dynamic Entities: GroupViT is primarily proposed to process static images and is prohibitively unable to group video patches into dynamic entities | Temporal-Aware Grouping: a novel mechanism that merges  semantically similar tokens into larger entities while preserving temporal dimension of the video |\n| Absence of Object-Environment Interactions: GroupViT solely focuses on object level and lacks a mechanism to model relationships between objects and environment, which is crucial for spatiotemporal tasks in video domain. | Entity-Aware Decoder: a pivotal component that propagates entity-level features from local entity encoder to enrich video-level embedding in global encoder. It captures the object-environment interactions, enhancing the spatiotemporal representation. The effectiveness of entity-aware decoder is shown in our ablation (Table 3, rows 1-2) |\n\n## **3: Co-training from both egocentric (ego) and exocentric (exo) and transferability from ego to exo tasks**\n\nOn one hand, it is obvious that incorporating ego and exo is essential for many real applications in robotics and augmented reality. However, the dramatically different viewpoints pose significant challenges. Most ego datasets just recently become available, and very few ego videos are synchronized with corresponding exo videos. Additionally, ego datasets are much smaller in scale compared to exo ones. Therefore, leveraging exo to improve model performance on ego is a viable approach [A]. Early work [B] focused on representation learning using paired ego-exo videos. However, paired videos are much harder to obtain compared to unpaired ones. Recent studies [C] made notable progress with unpaired videos.\n\nOn the other hand, transferability from ego to exo tasks presents a significant challenge and opportunity. The primary challenge is the alignment of different viewpoints, particularly between unpaired multi-view [D]. The ongoing research on joint view-invariant learning [E], domain adaptation [F], knowledge distillation [G] offer promising solutions.\n\nDespite co-training with mixed ego-exo data and transferability between ego-exo are exciting directions, our current objectives are solely focused on ego tasks, with an emphasis on providing trustworthiness and transparency to the model at a fine-grained level of alignment. Furthermore, we strongly believe that this capability can help bridge the gap between different viewpoints at fine-grained alignment, and this should be a focus of future research.\n\n[A] Weinland, D., etc \"Making action recognition ...\", ECCV2010\n\n[B] Yu, H., etc “What i see is what you see …”, ICM2019\n\n[C] Huang, Y., etc “EgoExoLearn …”, CVPR2024\n\n[D] Wang, Q., etc. “Learning from semantic alignment…”, ICCV2023\n\n[E] Xue, Z.S., etc. “Learning fine-grained …”, NeurIPS2023\n\n[F] Choi, J.,  etc, “Unsupervised and semi-supervised …”, WACV2020\n\n[G] Li, Y., etc. “Ego-exo…”, CVPR2021\n\n## **4: Visual grounding**\n\nWe conduct a quantitative experiment on Ego4D, as HelpingHands provides visual grounding results exclusively on this dataset. In the absence of ground truth segmentation masks for Ego4D, we generated these masks ourselves. Due to time constraints, we managed to obtain labels for only 200 videos. Below, we present the performance comparison in terms of mIoU scores:\n\n| Model | mIoU |\n| --- | --- |\n| HelpingHands | 22.73% |\n| Ours | 41.06% |\n\n**Discussion:** HelpingHands uses coarse bounding boxes for visual grounding, leading to lower mIoU scores due to inadequate coverage of the target masks. In contrast, our model employs segmentation masks that closely align with the ground truth, resulting in higher mIoU scores,  demonstrating superior visual grounding capabilities.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kVS7lzkAhL", "reviewer": "Reviewer_cNiH", "summary": "The paper introduces a novel framework for improving interpretability and performance in video-language models, specifically for egocentric videos. The framework, Hierarchical ENtities ASsemblY, employs a spatiotemporal token grouping mechanism that assembles and models relationships between dynamically evolving scene entities. This method aims to mimic human perceptual abilities by focusing on a compositional understanding of scene dynamics. The training of HENASY incorporates multi-grained contrastive losses, enhancing the model's ability to produce robust entity-level and video-level representations. Extensive experimental results demonstrate that HENASY significantly outperforms existing benchmarks on a range of egocentric video tasks.", "questions": "1. Could the authors provide insights into HENASY's performance on diverse real-world datasets and its computational efficiency and scalability compared to existing methods?\n2. Can the authors elaborate on the spatiotemporal token grouping mechanism’s impact on training dynamics and model convergence, and provide a comparative analysis of interpretability with other state-of-the-art methods?", "limitations": "The paper should explicitly address potential limitations related to the scalability of HENASY and its performance on varied real-world datasets. Additionally, the authors should discuss any potential negative societal impacts, such as privacy concerns in the context of egocentric video analysis, and suggest possible mitigation strategies for such issues to ensure ethical deployment.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. HENASY introduces a groundbreaking approach to VLMs by focusing on dynamic scene entity assembly, which significantly diverges from traditional methods that generally emphasize static frame analysis.\n2. The paper provides comprehensive experimental evidence showing that HENASY achieves superior performance on multiple egocentric video benchmarks, validating the effectiveness of its novel methodologies.\n3. By leveraging visual grounding with free-form text queries, HENASY enhances the interpretability of VLMs, which is critical for applications requiring transparent decision-making processes.\n4. The use of multi-grained contrastive losses to optimize the model at both the entity and video levels is a well-defined objective that contributes to the model's strong performance.", "weaknesses": "1. The paper lacks a detailed discussion on the scalability and computational demands of HENASY, which are crucial for its application in real-world settings, especially on resource-constrained devices.\n2. While HENASY shows impressive results in controlled experiments, its ability to generalize across diverse real-world scenarios and different video domains remains underexplored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer of their acknowledgment that our approach is groundbreaking with strong interpretability, comprehensive experiments. We would like to address concerns and questions raised by the reviewer as follows:\n\n## **1: Scalability and computation**\n\n**Scalability**: In our submission, we primarily reported results using TSF-B architecture as the backbone. As suggested by the reviewer, we have initiated training with a larger version of the backbone, TSF-L, to further investigate the scalability of our method. However, due to the constrained time frame of the rebuttal period, this model has not finished its training process. Nevertheless, we are pleased to report the current best results and comparison with a baseline (LaViLa) using similar backbones:\n\n| Benchmark | EK100-MIR | EK100-CLS | EGTEA | EgoMCQ |\n| --- | --- | --- | --- | --- |\n| Metric | Avg mAP | Top-1 Acc | Top-1 Acc | Intra Acc |\n| LaViLa w/ TSF-B | 30.9% | 16.4% | 35.5% | 59.9% |\n| LaViLa w/ TSF-L | +5.2% | +4.5% | +4.6% | +3.2% |\n| Our w/ TSF-B | 31.3% | 19.5% | 35.9% | 61.3% |\n| Ours w/ TSF-L | +5.1% | +5.3% | +5.3% | +2.7% |\n\nThis table illustrates the performance improvements of TSF-L over TSF-B for both LaViLa and our method. The results demonstrate that our model scales effectively with the larger backbone, showing consistent or greater gains across all benchmarks compared to LaViLa. These evidences underscore the scalability of our approach when equipped with more powerful backbones. We will add our final result to camera-ready version.\n\n**Computation**: We have conducted a study to evaluate the computational cost (Table 5, Page 9 of the submission). In this study, we also compared with the SOTA HelpingHands. It is summarized as follows.\n\n|  | HelpingHands | Ours |\n| --- | --- | --- |\n| Autoregressive  | YES | NO |\n| GFLOPs (per clip) | 530M | 599M |\n| Number of Parameters | 216M | 291M |\n| GPU Memory (train) | 38GB | 42GB |\n| GPU Memory (inference) | 4.4GB | 4.8GB |\n| Inference Time (seconds) | 2.87 | 1.02 |\n\n## **2: Resource-constrained devices**\n\nIn real-world applications, computational resource constraints can vary significantly, leading us to categorize environments into two primary types: cloud computation and standalone systems. For cloud computation, HENASY is particularly well-suited as it can leverage extensive computational resources similar to those used in deploying large language models (LLMs). For standalone systems, we are investigating several optimization techniques to ensure efficient deployment. These techniques include model pruning, quantization, and knowledge distillation, which notably reduce model size and computational load while maintaining competitive performance. The original HENASY model requires 4.8GB of GPU memory, making it compatible with modern devices like the Jetson AGX Xavier (up to 32GB GPU memory), Jetson Xavier NX (8GB GPU memory), and Jetson TX2 (8GB GPU memory). However, deployment in resource-constrained environments is beyond the scope of this paper.\n\n## **3: Generalize across diverse real-world scenarios and different video domain**\n\nAs clarified in the session “Benchmarks and Evaluation Protocols” on Pages 7, 8 of the submission, we evaluated the proposed HENASY using different protocols, including  zero-shot transfer protocol and visual & textual representation protocol. We conducted evaluation on various datasets, including  EgoMCQ, EgoNLQ, EgoMQ, EK100-MIR, EK100-CLS, and EGTEA. It is important to note that  those datasets have been acquired by different devices under different real-world scenarios settings and across various video domains as follows:\n\n- EK-100: kitchen activities, collected by GoPro.\n- EGTEA: indoor activities, collected by SMI eye tracking glasses.\n- EgoMCQ, EgoNLQ, EgoMQ: indoor and outdoor daily activities (cooking, sport, shopping, lawn mowing, etc.), collected by multiple cameras such as GoPro, Vuzix Blade, Pupil Lab, etc.\n\nThe experimental results in Table 1, Page 8 on the zero-shot transfer protocol and Table 2, Page 9 on the visual and textual representation protocol demonstrate the effectiveness of our proposed method across various video domains in diverse real-world scenarios. Notably, our model consistently outperforms previous SOTA methods with significant margins.\n\n## **4: Impact of spatiotemporal token grouping mechanism**\n\nOur Temporal-aware grouping (TAG) mechanism has a foundational role in our model, which leverages slot-based methods in images into modeling spatiotemporal token grouping. Concretely, it helps preserve the temporal dimension during our hierarchical dynamic entities assembly process. Given the essential role of TAG in our model, we are unable to conduct a separate ablation study for it, as its removal would undermine the functionality of our model.\n\n## **5: Comparative analysis of interpretability**\n\nWe conduct a quantitative experiment on Ego4D for visual grounding task. In the absence of ground truth segmentation masks for Ego4D, we generated these masks ourselves. Due to time constraints, we managed to obtain labels for 200 videos. Below, we present the performance comparison in terms of mIoU scores with HelpingHands, our most related work that supports visual grounding:\n\n| Model | mIoU  |\n| --- | --- |\n| HelpingHands | 22.73% |\n| Ours | 41.06% |\n\n**Discussion:** although being the SOTA in egocentric tasks and visual grounding task, HelpingHands only provides coarse bounding boxes of objects, leading to lower mIoU scores due to inadequate coverage of the target masks. In contrast, our model employs segmentation masks that closely align with the ground truth, resulting in higher mIoU scores,  demonstrating superior visual grounding capabilities.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0CY0fdaBtC", "reviewer": "Reviewer_4RbK", "summary": "This paper proposed HENASY, a novel framework for learning egocentric video-language models where texts are grounded to visual scene entities. Its main idea is to use both global and local visual encoder to encode video features so that nouns and verb phrases in the paired text could be matched individually. The overall framework is optimized through contrastive losses while a projection loss is further introduced to improve the grounding quality. Experiments show that the proposed the method could achieve good performance on a wide range of egocentric video tasks including video/text retrieval, action recognition, multi-choice query, natural language query, and moments query.", "questions": "Please refer to the Weakness section.", "limitations": "The authors have widely discussed the limitation of the proposed work and also its challenges in modeling the interactions between complex scene entities.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is overall well-written and easy to follow.\n\n2. The problem addressed by this paper is important to the field. Learning grounded and interpretable egocentric video-language models has been seldomly studied in previous works, which might raise new research interests in this field.\n\n3. The experiments are extensive where the proposed method has been evaluated on a wide range of egocentric video-text tasks through zero-shot transfer or learned video-text representations.", "weaknesses": "1. The performance improvement of the proposed method on most tasks are very marginal, compared to HelpingHands, which is the most related work to the paper. Given the fact that the design of  HelpingHands is much simpler than the proposed methods, I have concern on whether the complex components introduced in this paper is efficient.\n\n2. Given the huge complexity of the proposed method, some important studies on the proposed design choices are missing. E.g., (1) the authors mentioned that using GroupViT to directly process input video patch tokens will diminishe performance but the results are not reported in the paper; (2) rather than using [5] to initialize the global encoder, have the authors tried other options, and what are the results? These help the reader to gain a better understanding of the robustness of the proposed method.\n\n3. What about the scaling properties of the proposed method? Does it scale well with larger model sizes or more training data?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your acknowledgment of our well-written paper, the importance and novelty of our approach, and the extensive experiments.\n\n## **1: Comparison with HelpingHands**\n\nWe recap the key improvements of our work over HelpingHands as below\n\n| Property | Our Work | HelpingHands |\n| --- | --- | --- |\n| Method | Explicitly models video as dynamic entities (object-centric), capturing their interactions (action-centric) to form spatiotemporal interpretable, object/action-aware video representation | Implicitly induces object occurrence (nouns) on top of pre-extracted feature map via auxiliary task-specific (detection) head to form an object-aware video representation |\n| Fine-grained alignment | Spatiotemporal fine-grained alignments. Object appearance: Noun-entity alignment. Activity motion: verb-entities alignments | No alignment at granular level |\n| Interpretability | Strong, both spatial and temporal dimensions with object-centric and action-centric via dynamic entity assembly mechanism | None. |\n| Visual grounding | Strong, with spatiotemporal saliency maps of scene entities and relationships | Weak, only provide predicted bounding boxes |\n| Efficiency | faster than HelpingHands x3 times in inference | slower x3 times in inference due to autoregressive decoder |\n\nThis discussion is included in Section 2, lines 87-93 of the submission.\n\nWe highlight our improvements over HelpingHands across various metrics as below (Tables 1 and 2 in the paper):\n\n| Benchmark | Metric | HelpingHands | Our Method | Improvement |\n| --- | --- | --- | --- | --- |\n| EgoMCQ | Inter Acc | 93.2 | 94.1 | +0.9 |\n| EgoMCQ | Intra Acc | 58.8 | 61.3 | +2.5 |\n| EGTEA | Top-1 Acc | 35.3 | 35.9 | +0.6 |\n| EgoNLQ | mIoU @0.3 R5 | 20.4 | 21.5 | +1.1 |\n| EgoMQ | mAP | 11.7 | 12.4 | +0.7 |\n\n**Discussion:** Our primary objective extends beyond achieving high benchmark scores to advancing interpretability within video-language models. Our model significantly diverges from HelpingHands by emphasizing compositional video understanding. Additionally, it improves upon HelpingHands by focusing on interpretable representations. This shift towards interpretability represents a strategic choice aimed at impacting areas where understanding model decisions is critical.\n\n## **2: Ablation studies**\n\nIn Lines 159-161, we identified 3 major limitations in slot-attention (GroupViT) if we merely employ it:\n(i) direct process of video patch tokens from scratch, which neglects powerful video representations already learned by pre-trained models.\n\n(ii) originally proposed in image domain, making it unable to model dynamic entities with temporal information in videos.\n\n(iii) unable to model object-environment interactions, which is essential to capture spatiotemporal dynamics of the videos.\n\nWe address these limitations through:\n(i) A novel bootstrapping stage: This aims to leverage the powerful pre-trained model for video patches encoding. The corresponding ablation study was in Table 3, row 3.\n\n(ii) Temporal-aware grouping (TAG): This targets preserving temporal dimensions, crucial for our hierarchical dynamic entity assembly approach. Given the foundational role of TAG in our model, we did not conduct a separate ablation study for it, as removing it would undermine the model’s functionality.\n\n(iii) Entity-aware decoder: This decoder plays a crucial role in ****modeling entities-environment interactions**,** enhancing our model’s overall performance. It propagates entity-level features from local entity encoder to enrich video-level embeddings from global encoder. It significantly contributes for the effective performance of our model. If we excluded this decoder, the model would rely solely on entity features from our slot-based local entity encoder and would end up with a substantial performance drop (Table 3, rows 1-2).\n\n## **3: Initialization with other pre-trained models**\n\nDue to time constraint of the rebuttal period, we were able to conduct experiments on an additional pre-trained model, EgoVLP, besides the pre-trained model LaViLa reported in the submission. Below is a performance comparison when initializing our model with different pre-trained models, against the baseline results from LaViLa and EgoVLP:\n\n| Benchmark | Metric | LaViLa | Ours (initialized with LaViLa) | EgoVLP | Ours (initialized with EgoVLP) |\n| --- | --- | --- | --- | --- | --- |\n| EgoMCQ | Intra Acc | 59.9 | 61.3 | 57.2 | 59.7 |\n| EK100-MIR | Avg mAP | 30.9 | 31.3 | 23.3 | 30.9 |\n| EGTEA | Top-1 Acc | 35.5 | 35.9 | 17.6 | 34.0 |\n\nThis table demonstrates that our model, when initialized with EgoVLP, not only performs competitively with its initialization using LaViLa, but also consistently outperforms the original EgoVLP across various benchmarks. This highlights the versatility and robustness of our model. We will add this result in our final version with an extra page.\n\n## **4: Scalability**\n\nIn our submission, we primarily used TSF-B as the backbone. As suggested by the reviewer, we have trained with a larger version, TSF-L, to further investigate the scalability. Due to time constraints, this model has not finished its training process. Nevertheless, we are pleased to report the current best results and comparison as below:\n\n|Benchmark|EK100-MIR|EK100-CLS| EGTEA | EgoMCQ |\n|---|---|---|---|---|\n| Metric | Avg mAP | Top-1 Acc | Top-1 Acc | Intra Acc |\n| LaViLa w/ TSF-B | 30.9 | 16.4 | 35.5 | 59.9 |\n| LaViLa w/ TSF-L | +5.2 | +4.5 | +4.6 | +3.2 |\n| Our w/ TSF-B | 31.3 | 19.5 | 35.9 | 61.3 |\n| Ours w/ TSF-L | +5.1 | +5.3 | +5.3 | +2.7 |\n\nThis table illustrates the performance improvements of TSF-L over TSF-B for both LaViLa and our method. The results demonstrate that our model scales effectively with the larger backbone, showing consistent or greater gains across all benchmarks compared to LaViLa. These evidences underscore the scalability of our approach when equipped with more powerful backbones. We will add our final result to camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate all reviewers for their valuable time and constructive feedback. We are grateful for their recognition about the **significance, inspiration and impact** of our grounded and **interpretable** work (Reviewers EqYJ, cNiH, 4RbK), the **novelty** of our method (Reviewers EqYJ, cNiH), **extensive results** (Reviewers EqYJ, xaMC, cNiH), and **easy to follow presentation** (Reviewers xaMC, 4RbK).

The reviewers raised several concerns, which we addressed in the individual response. We summarize the highlights of our responses as follows:

**1. Architectural comparisons to related works [1, 2] and HelpingHands [3]:** We discuss the distinctions and advantages of our approach over [1, 2, 3], emphasizing the superior interpretability and dynamic entity modeling of our model. These properties are achieved via our compositional approach, which explicitly models video as dynamic entities (object-centric) and capturing their interactions (action-centric) to form spatiotemporal interpretable, object/action-aware video representation. Moreover, our approach is **the first to** explicitly integrate **spatiotemporal fine-grained alignments**, such as *object appearance (n*oun-entity alignment) and *activity motion* (verb-entities alignments).

**2. Performance Comparisons to HelpingHands [3]:** In Tables 1 and 2 of our submission, we reported significant improvements over HelpingHands, the most recent SOTA method, across various benchmarks and metrics. We would like to highlight again the such improvements that our model achieves against HelpingHands:

| Benchmark | Metric | HelpingHands | Our Method | Improvement |
| --- | --- | --- | --- | --- |
| EgoMCQ | Intra Acc | 93.2 | 94.1 | +0.9 |
| EgoMCQ | Inter Acc | 58.8 | 61.3 | +2.5 |
| EGTEA | Top-1 Acc | 35.3 | 35.9 | +0.6 |
| EgoNLQ | mIoU @0.3 R5 | 20.4 | 21.5 | +1.1 |
| EgoMQ | mAP | 11.7 | 12.4 | +0.7 |

**3. Comparison with GroupViT:** We clearly demonstrate three critical limitations inherent to GroupViT’s architecture:

(i) direct process of video patch tokens from scratch, which neglects powerful video representations already learned by pre-trained models.

(ii) originally proposed in image domain, making it unable to model dynamic entities with temporal information in videos.

(iii) unable to model object-environment interactions, which is essential to capture spatiotemporal dynamics of the videos.

In our local entity encoder, we address these limitations through:

(i) A novel bootstrapping stage: This aims to **leverage the powerful pre-trained model** for video patches encoding at early layers. The corresponding ablation study was in Table 3, row 3.

(ii) Temporal-aware grouping (TAG): This targets **preserving temporal dimensions**, crucial for our hierarchical dynamic entity assembly approach. Given the foundational role of TAG in our model, we did not conduct a separate ablation study for it, as removing it would undermine the model’s functionality.

(iii) Entity-aware decoder: This decoder plays a crucial role in **modeling entities-environment interactions,** enhancing our model’s overall performance. It propagates entity-level features from local entity encoder to enrich video-level embeddings from global encoder. It significantly contributes for the effective performance of our model. If we excluded this decoder, the model would rely solely on entity features from our slot-based local entity encoder and would end up with a substantial performance drop (Table 3, rows 1-2).

**4. Quantitative evaluation on visual grounding:** We conduct a rigorous quantitative analysis on Ego4D dataset to compare with the SOTA model of HelpingHands, as they only provide visual grounding results on this datasets. We create semantic segmentation labels by ourselves for 200 videos. The results are reported under mIoU metric between visual grounding prediction with corresponding groundtruth, showing our model’s superior visual grounding capability compared to HelpingHands:

| Model | mIoU |
| --- | --- |
| HelpingHands | 22.73% |
| Ours | 41.06% |

Our model outperforms HelpingHands thanks to its strong interpretability with dynamic segmentation masks associated with every entity in the video, while HelpingHands is only able to obtain bounding boxes of objects. 

**5. Scalability:** We show that scaling of the network from the base model (TSF-B) to a large model (TSF-L) leads to the consistent gains in all benchmarks.

| Benchmark | EK100-MIR | EK100-CLS | EGTEA | EgoMCQ |
| --- | --- | --- | --- | --- |
| Metric | Avg mAP | Top-1 Acc | Top-1 Acc | Intra Acc |
| LaViLa w/ TSF-B | 30.9% | 16.4% | 35.5% | 59.9% |
| LaViLa w/ TSF-L | +5.2% | +4.5% | +4.6% | +3.2% |
| Our w/ TSF-B | 31.3% | 19.5% | 35.9% | 61.3% |
| Ours w/ TSF-L | +5.1% | +5.3% | +5.3% | +2.7% |

Finally, we have carefully addressed all the reviewers' comments and questions. We will revise and update the final version based on all suggestions using the allowed extra page.

[1] Ge, Yuying, et al. "Bridging video-text retrieval with multiple choice questions." CVPR 2022. 

[2] Xiong, Yuanhao, et al. "Structured Video-Language Modeling with Temporal Grouping and Spatial Grounding." ICLR 2024.

[3] Zhang, Chuhan, et al. "Helping Hands: An Object-Aware Ego-Centric Video Recognition Model.” ICCV 2023.

Meta-review: This paper introduces a novel framework for egocentric video language understanding. The authors proposed to learn compositional and hierarchal video representations by three levels: 1) global video features from a global video encoder. 2) entity features by assembling dynamic entities from video patches via local entity encoder, 3) interactions between entities and global context by an entity-aware decoder. The paper received mixed reviews with 2 borderline accept, 1 weak accept and 1 borderline reject. Most reviewers see the merits of the paper: 1) clear motivation 2) extensive experiments on multiple downstream tasks 3) well-written. There are a few concerns raised by reviewers as well. Reviewer EqYJ and 4RbK think the improvement over Helping hands is marginal; Reviewer xaMC and 4RbK want to understand the delta between this work and group ViT; Reviewer cNiH and Reviewer cNiH question about scalability. Authors have provided detailed answers to reviewers' questions in their rebuttal. In specific, authors provided clear architectural comparison, more experiment results based on different base models, and comparison with GroupViT. Two reviewers acknowledged that their questions are resolved and increased their final scores. Considering all reviews and discussion, the AC recommends to accept this paper. The AC also encourages the authors to include the additional experiment results in their final version.
Decision: Accept (poster)