HENASY: Learning to Assemble Scene-Entities for Interpretable Egocentric Video-Language Model

Authors: khoa  vo
thinh  phan
kashu  yamazaki
minh  tran
ngan  le
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 08:04:48

Abstract: Current video-language models (VLMs) rely extensively on instance-level alignment between video and language modalities, which presents two major limitations: (1) visual reasoning disobeys the natural perception that humans do in first-person perspective, leading to a lack of reasoning interpretation; and (2) learning is limited in capturing inherent fine-grained relationships between two modalities.

In this paper, we take an inspiration from human perception and explore a compositional approach for egocentric video representation. We introduce HENASY (Hierarchical ENtities ASsemblY), which includes a spatiotemporal token grouping mechanism to explicitly assemble dynamically evolving scene entities through time and model their relationship for video representation. By leveraging compositional structure understanding, HENASY possesses strong interpretability via visual grounding with free-form text queries. We further explore a suite of multi-grained contrastive losses to facilitate entity-centric understandings. This comprises three alignment types: video-narration, noun-entity, verb-entities alignments.

Our method demonstrates strong interpretability in both quantitative and qualitative experiments; while maintaining competitive performances on five downstream tasks via zero-shot transfer or as video/text representation, including video/text retrieval, action recognition, multi-choice query, natural language query, and moments query.

Project page: https://uark-aicv.github.io/HENASY

Main paper: 
 introduction: 
 Recent advancements in technology and hardware devices for augmented reality (AR) have fueled hopes for virtual assistant applications that can provide users a wide range of assistance, such as real-time procedural instructions, moments retrieval, and interactive learning experiences, all through egocentric video streams of similar perspective with user. Publicly available massive-scale egocentric datasets such as Ego4D [1] and Epic Kitchens-100 [2] , providing suites of egocentric tasks, have further sparked even more interest within the research community.Video-language models (VLMs) have currently become a de-facto approach to egocentric video understanding. By learning robust visual-language representations from video-caption pairs [3] , VLMs can be applied flexibly to a wide range of downstream tasks, either through zero-shot transfer or as modality encoders. Existing state-of-the-art (SOTA) VLMs for egocentric videos [4, 5, 6, 3] exhibit remarkable performances by following CLIP-like [7] dual-encoder architecture. During training, these models generally learn through the instance-level alignment [8, 3] between pairs of video and caption representations (Fig. 1(a) ). However, videos consist of complex and dynamic interactions among arbitrary entities, which cannot be effectively captured by simple instance-level alignment alone. In fact, a caption contains textual elements that concisely capture video entities. For examples, nouns indicate entity occurrences [4] ,38th Conference on Neural Information Processing Systems (NeurIPS 2024). [5] rely on instance-level contrastive learning between video & narration. HelpingHands [4] implicitly induces object occurrence information into video features at final layer of video encoder. (b) Our proposed (HENASY) aims to assemble dynamic entities from video patches via local entity encoder, while entity-aware decoder captures interactions between entities and global context to form comprehensive video. HENASY is trained with suite of multi-grained contrastive alignments to enforce visual representations entity-level upto video-level. (c) By such compositional approach, HENASY is the first VLM that shows strong interpretability via visual grounding with both appearance/motion query types.while verb phrases convey motion information [9] in the video. To fully capture these fine-grained alignments, a VLM will perform more effectively if it: (1) understand videos in a bottom-up manner, where semantically similar patches form entities, and relationships between entities construct the video representation; and (2) explicitly model fine-grained relationships between video entities and nouns/verbs to comprehensively capture appearance/motion information, respectively.Human perception aligns closely with the above requirements. We perceive the dynamic surroundings in a compositional manner [10] , where distinct entities emerge from smaller parts that combine to form a whole. Each entity maintains spatial and temporal coherence and interacts with others only when in close proximity. Understanding the compositional structure of the surroundings enables us to intrinsically comprehend and memorize information, while also allowing us to provide interpretations of our decision-making process, which is absent in current egocentric VLMs.Inspired by such observation, we propose HENASY: Hierarchical ENtities ASsemblY framework (pronounced heh-nuh-see), which follows compositional understanding as in Fig 1(b) . Concretely, HENASY comprises three key components: (i) Local Entity Encoder, a hierarchical transformerbased encoder that learns to assemble dynamic scene-entities from video patches via our proposed spatiotemporal token grouping mechanism, which is an enhanced version from slot-based groupings in stationary images [11, 12] ; (ii) Global Encoder, a pre-trained video representation module that perceives the input video at a global level; and (iii) Entity-Aware Decoder, which models the internal interactions among scene entities and their relationship with the global features, thereby enriching the entity-centric video representation extraction. Furthermore, HENASY is able to perform visual grounding to obtain dynamic segmentations corresponding to either entity or activity with the produced entity embeddings and their attention maps as a side product of its local entity encoder, showing promising interpretation via dynamic saliency maps across frames (Fig. 1(c)) .Developing an effective model necessitates a strong network architecture and well-defined objectives. With the proposed HENASY architecture, instance-level contrastive loss only handles global alignment, failing to address dynamic entity alignment. Hence, we introduce multi-grained contrastive losses to optimize HENASY for both entity-and video-level representations using narration alone. Specifically, HENASY is trained with three types of alignment: video-narration, noun-entity, and verb-entities. While the first two employ instance-level contrastive loss and model object occurrences via narration's nouns, respectively, verb-entity alignment is newly introduced. It aims to incorporate activity/motion information from narration's verb phrases into entities using a one-to-many strategy, which emphasizes the alignment of a verb phrase to the most semantically relevant entities. Additionally, we propose a new projection loss that employs detected hand/object boxes [4] to ensure segmentation masks tightly cover respective entities, enhancing HENASY's interpretative robustness.We are the first to demonstrate the value of compositional perception approach for egocentric video understanding. Our experiments show that by tasking our proposed local entity encoder to assemble dynamic entities, video representations are effectively improved to outperform current VLMs in a wide range of benchmarks, including video retrieval (EgoMCQ [3] & EpicKitchen-MIR [2] ), activity recognition (EpicKitchen-CLS & EGTEA [13] ) via zero-shot transfer. Furthermore, temporal localization models [14, 15] equipped with HENASY video/text features can achieve state-of-the-art performances in episodic memory tasks of EgoNLQ and EgoMQ [1] . Finally, HENASY possesses strong interpretability that is quantitatively and qualitatively superior to current VLMs. 
related works: 
 Video-Language Pre-Trained Models. Pre-training VLMs on a large-scale dataset of video-text pairs and deploying them in down-stream tasks has now become a standard practice. Transformerpowered pre-trained VLMs [16, 17, 18, 19, 3, 6, 5, 4, 9] have accomplished superior results on a wide range of tasks, i.e., text-to-video retrieval, action recognition, or events localization. VLMs can be divided into two common categories, i.e., unified-and dual-encoder. The former models [16, 18, 20] fuse multimodal via cross-attention and can be trained with proxy tasks of masked language modeling [21, 22] or masked frame modeling [18, 16] . The latter models [23, 3, 5, 6, 9] employ separate encoders for video and text, trained jointly via contrastive learning [8, 3] .Recently, several VLMs [4, 17, 9 ] employ fine-grained information from captions by decomposing them to capture object/activity through nouns/verbs, respectively. However, these models do not fully exploit fine-grained learning within the video encoder itself, and true granular-level alignment between modalities remains unexplored. In our work, we explicitly model visual content as dynamic entities, capturing their interactions to form a comprehensive video representation. Additionally, our proposed method is trained with multi-grained objectives, ranging from video-text, noun-entity, to verb-entities pairs. Interpretable Video Representation. There have been efforts to enhance the interpretability of video representations [24, 25, 26, 27] , these typically involved factorizing videos into entities and environmental contexts, and utilizing adaptive attention mechanisms [24] to selectively focus on relevant entities. Such mechanisms enable interpretation through their selection module, highlighting only the primary entities contributing to the final predictions. However, these approaches often require entities to be pre-detected by external models, which is restricted to a predefined set of objects. Additionally, these works have focused on models specifically designed for individual tasks, e.g., temporal action detection, video dense captioning. In contrast, our work introduces an end-to-end method capable of learning to form entities without an off-the-shelf detector. Moreover, we aim to develop a robust video-language model (VLM) that is versatile across a variety of tasks.Interpretation via Object Discovery. Recent years have seen a growing body of research in endto-end learning of object discovery, which learns to decompose an image or a video into distinct objects without direct supervision. Slot-based methods such as IODINE [28] and Slot Attention [11] utilize mixture-model likelihoods [29] to tackle this challenge, demonstrating promising performance through evaluations on synthetic images with simple objects. Subsequently, GroupViT [12] and ODIN [30] incorporate slot attention with contrastive learning and achieved notable success in identifying semantic groupings on natural in-the-wild images. However, these models are not capable of modeling dynamic objects in videos domain. To mitigate this problem, SaVI++ [31] proposes a workaround technique, which requires groundtruth depth information in a reconstruction objective to bootstrap object discovery training. In our work, we enhance slot-based grouping mechanisms introduced in GroupViT [12] to model temporal coherency of dynamic objects in videos. Different from [31] , HENASY does not require any extra data further than color video sequences. Instead, HENASY utilizes learned patch features of pre-trained global encoder to bootstrap several early layers of its local entity encoder for entities grouping via a cross-attention mechanism. 
preliminaries: 
 Video-language representation learning aims to learn a common latent space to represent video and text. A training dataset for this task comprises of N tuples {V i , T i } N i=1 , where V i denotes a short sequence of RGB frames, and T i is a free-form text sentence that describes visual content.Dual-encoder architecture is a common paradigm that current SOTA VLMs [4, 5, 3] employ for the above task, which consists of (a) a visual encoder f mapping the input video V i to a visual embedding feature v i = f (V i ), and (b) a language encoder g mapping the text T i to a linguistic embedding feature t i = g(T i ).Contrastive-based losses are common objective for video-language representation. Given a batch of B normalized video-text embedding pairs i = {v i = vi /|vi|, ti = ti /|ti|}, a contrastive-based loss pulls embeddings of aligned (positive) pairs close in feature space, while pushing embeddings of misaligned (negative) pairs away. We adopt EgoNCE variation [3] of contrastive loss as one of our training objectives because of its effective approach in identifying positive and negative pairs. Specifically, each sample i ∈ B is associated to a set of positives P i constructed by comparing nouns and verbs across all texts. Additionally, for each sample i, a hard negative i ′ is sampled from a temporally adjacent segment within the same video, expanding our batch to B. For a more in-depth discussion on the strategy used for sample selection, refer to [3] . Herein, the video-to-text objective is succinctly expressed as:L v2t ego = 1 B i∈ B log p∈Pi exp(v T tp /τ ) n∈B exp(v T tn /τ ) + exp(v T tn ′ /τ )where τ denotes a temperature (1)The objective of text-to-video L t2v ego is derived from Eq. 1 by inverting v and t, and EgoNCE loss is a summation of both directions L ego = L v2t ego + L t2v ego . 
henasy: 
 We present the HENASY framework (Fig. 2 ) for egocentric video-language modeling. HENASY is a compositional video understanding approach featuring a dual-encoder architecture, designed to explore an interpretable, entity-based visual representation. Specifically, besides typically capturing global features via global encoder (Sec. 4.1), our video encoder also assembles dynamic scene entities from video patches via local entity encoder (Sec. 4.2), then entity-aware decoder (Sec. 4.3) models their intra-connections as well as inter-connections with global features to form a comprehensive video representation. Our objective is to develop an interpretable reasoning process that robustly supports decision-making, while allowing visual grounding with text queries. To achieve this target, it requires not only an effective network design, but also a suite of multi-grained contrastive learning (Sec. 4.4) to enforce both entity-level and video-level representations.For readability, we denote five types of tokens as follows: z for video patch tokens, c for learnable video tokens, g for learnable group tokens, s for segment tokens, and e for entity tokens.Global encoder provides global visual information of the entire input video. We adopt the pre-trained TimeSFormer [32] to capture the global visual context of the entire input video. Particularly, we follow the protocol of [3, 5, 4] to decompose the given input video sequence V i ∈ R T ×3×H×W with T RGB frames or resolution H × W into T × K non-overlapping patches of resolution P × P , where K = HW/P 2 . Then, every patch is linearly projected by a 2D convolutional layer, forming video patch tokens z ∈ R T K×D (with T K = T × K) representing embedding features at every temporal and spatial location, where D is hidden dimension. TimeS-Former processes the video patch tokens z with an additional learnable video tokens c ∈ R 1×D through a stack of divided space-time attention (DST) blocks, which is described as follows:[c l+1 ; z l+1 ] = DST([c l ; z l ]) where [•;•] denotes concatenation operator. Please see Appendix A for more details on the computational process of a DST block.Local entity encoder models fine-grained information of the input video by consistently capturing dynamic scene-entities. We adopt a hierarchical bottom-up architecture, consisting of attention-based layers that are divided into multiple stages. Each stage progressively groups small video patch tokens z from the previous stage into larger segments. As a result, local entity encoder forms scene entity tokens that depict individual entities, which dynamically evolve across video frames.While following GroupViT [12] to directly process input video patch tokens could be an option, we find doing that diminishes performance. Additionally, the tokens grouping mechanism from [12, 11] is not capable of modeling dynamic entities in videos domain. To address these challenges, we introduce a bootstrapping stage, which couples itself with early layers of the global encoder through cross-attention to group video patches into initial entities' segments. Furthermore, to capture dynamic entities in video, we introduce temporal-aware grouping mechanism.Bootstrapping Stage. Bootstrapping stage consists of S 1 consecutive cross-attention layers, which takes a set of learnable group tokens g l boot ∈ R G×D as initial queries (G is the initial number of tokens). At each cross-attention layer l starting from 0, the queries aggregate information from the patch tokens z l at corresponding layer of global encoder:EQUATIONAt the final layer of bootstrapping stage, we obtain g l boot . This is then associated with patch tokens z l from the corresponding layer of global encoder within temporal-aware grouping block (TAG) to group patches into larger segment tokens:EQUATIONHerein, TAG(q, k) merges key tokens k together based on their similarities with query q, while preserving the temporal dimension of key tokens (discussed in detail later). As a result, we obtain new segment tokens s l ∈ R T G×D , which is then utilized as inputs to the entity grouping stage.Entity Grouping Stage. From this point, local entity encoder is decoupled from the global encoder and is trained to merge these input segments s into complete scene entities e. At this stage, a new set of learnable group tokens g l entity ∈ R E×D is introduced, which aims to relate segment tokens with similar semantics into an individual scene entity. It is important to note that maintaining consistent temporal dynamics is required at each stage. Therefore, we adopt S 2 DST blocks [32] to propagate information mutually between learnable group tokens and segment tokens: [g l+1 entity ; s l+1 ] = DST([g l entity ; s l ]), for l = S 1 , .., S 1 + S 2 -1. After the final layer, segment tokens s l are grouped to generate intermediate entity tokens, i.e., êl = TAG(g l entity , s l ) ∈ R T E×D , where l = S 1 + S 2 . Then, to enable interactions between scene entities and across temporal dimension, we apply a stack of S 3 DST blocks to all entity tokens: êl+1 = DST(ê l ), for l = S 1 + S 2 , .., S 1 + S 2 + S 3 -1.We observed that segment tokens s and entity tokens e facilitate temporal consistencies within the temporal attention of a DST block. Unlike in TSF [32] , where tokens are spatially limited within a patch, these tokens can evolve freely across frames, enhancing the flexibility of the time-attention mechanism in a DST block. To obtain spatio-temporal entity embeddings, we apply temporally average pooling on entity tokens of the final layer: e = AvgPool(ê l ), where e ∈ R E×D and l = S 1 + S 2 + S 3 .Temporal-Aware Grouping (TAG). As aforementioned, TAG is employed at the final layers of bootstrapping stage and entity grouping stage, to merge semantically similar tokens (i.e., z or s) into a larger segment while preserving the temporal dimension. Generally, this mechanism takes a set tokens i ∈ R T I×D ( i can be either z or s) as inputs and a set of learnable group tokens g q ∈ R Q×D as queries. We re-shape i into 3-dimension R T ×I×D and evaluate the similarity between g q and i.It firstly evaluates similarity between each group token and every input token, forming a 3D similarity matrix A ∈ R T ×Q×I . Then, an assignment matrix Ã ∈ {0, 1} T ×Q×I is computed, assigning each input token to the most relevant group based on similarity scores. Finally, it performs perframe groupings, merging the input tokens of the same group together, forming a set of new tokens o ∈ R T ×Q×D representing larger segments. We formalize the computation of every new group as follows:EQUATIONAfterwards, we re-shape o to R T Q×D as output of TAG.The computation of similarity matrix A and assignment matrix Ã are detailed in Appendix B.1.Additionally, the derivation of saliency maps for interpreting assignments between video patches and entities is explained Appendix B.2.We seek to propagate entity-level features e l from the local entity encoder to the final video embedding for a comprehensive video representation. For this purpose, we introduce entity-aware decoder, which is illustrated in Fig. 3 . Entity-aware decoder includes a stack of hybrid-attention blocks to refine the interactions between entities and video patches, and render the video embedding. At a block b dec , it first performs cross-attention with entity tokens as queries and patch tokens as keys, values. Then, self-attention followed by a multi-layer perceptron (MLP) is applied over the output: Eventually, we obtain the video representation, dubbed as entity-aware video embedding v, by averaging the final outputs of entity tokens:EQUATIONBeside video-narration contrastive loss [8, 3] , which captures coarse-grained semantic alignment between the video and narration, we introduce two finer-grained contrastive losses: noun-entity contrastive loss (NEC) and verb-entities contrastive loss (VEC), which focuses on inducing visual appearance and motion cues directly to the composed entities. We also utilize projection loss, leveraging object boxes from an off-the-shelf detector [33] as a weak supervision to encourage the generated entity masks tightly conforming to the corresponding entity, promoting robust interpretability of our proposed model.From the groundtruth narration, we obtain N n nouns and their embeddings n ∈ R Nn×D via the text encoder. Following [4] , we compute a similarity matrix between noun embeddings and entity embeddings. Every noun is matched with an entity token having highest similarity score via Hungarian matching. Following this, we construct a noun-entity contrastive loss using the InfoNCE [8] , where positive pairs consist of the matched noun embedding n p and entity embedding e p . The contrast is defined over the embeddings n ′ j of all nouns in the dataset taxonomy dictionary D [1]:EQUATIONVerb-Entities Contrastive Loss is a new loss term that instills motion information directly into entity tokens from narration's verb phrases. As suggested in [9] that LLMs are superior to classical methods such as part-of-speech tagging in retrieving verb phrases, we use a Llama-2 [34] to obtain N v verb phrases from an input narration. Given that a verb phrase describes an activity involving several scene entities, we introduce weighted many-to-one alignment strategy to prioritize the most relevant entity-verb alignments. Firstly, let a i ∈ R D be one of the embedded verb phrases, we obtain a Softmax-normalized similarity scores between a i and every entity e j : s(a i , e j ) =ai•e T j E k ai•e T k . Then, we re-weight entities by the computed weight and obtain a weighted average of entities representation: e avg = E j s(a i , e j )e j . Here, e avg re-weights each entity based on its relevancy with verb phrase a i . Finally, we compute contrastive loss between this paired representations:EQUATIONwhere we utilize batch formation technique from egocentric contrastive loss [3] to form negatives set in L V EC .Projection Loss operates on each individual frame of the input video, utilizing an external object detector [33] to identify bounding boxes b = {b i ∈ R 4 } N b i=1 of scene entities. Let m = {m i ∈ (0, 1) H×W } E i=1 be the predicted saliency maps of scene entities (see Appendix B.2 for saliency maps formulation), Hungarian matching pairs each detected box b i with the predicted mask m i having the highest IoU.Designing a differentiable loss function that guides the predicted mask m j by groundtruth box b j is quite challenging. To address this, we utilize an axis projection function [35] to minimize the discrepancy of vertical and horizontal projections of b j and m j on two axes. This ensures that the smallest box encompassing m j matches with b j . Concretely, b j is firstly converted to binary mask format bj ∈ {0, 1} H×W where pixels inside b j is assigned by 1 and 0 otherwise. Then, a projection loss is defined as follows:EQUATIONwhere L dice is a Dice loss function [36] , max y (•) and max x (•) are max-project operators along y-axis and x-axis of the frame, respectively.Total Optimization. Overall, our model is optimized with a weighted sum of EgoNCE loss over video-text pairs and three objectives stated above:EQUATION) where λ 1 , λ 2 , and λ 3 balance contributions of different loss terms. 
training and implementation details: 
 Architecture. We use video clip inputs of size 224 × 224, text inputs are tokenized and processed by a 12-layer Transformer following [5] . We employ TimeSFormer [32] Base (TSF-B) for the global encoder. In the local entity encoder, all layers share a hidden dimension D = 512. Bootstrapping stage includes S 1 = 6 cross-attention layers with 64 group tokens. Entity grouping stage consists of S 2 = 3 DST blocks with 8 group tokens, followed by S 3 = 3 DST blocks. Entity-aware decoder is a stack of 3 hybrid-attention blocks.Training. HENASY is trained on EgoClip [3] , which contains 3.8M clip-narration pairs covering a sub-set of 2,927 video hours from Ego4D [1] . For each video clip, we uniformly sample 4 frames. We employ the pre-extracted narration's nouns and pre-detected hand and object bounding boxes from [4] for NEC loss and projection loss, respectively. For verb phrases, we employ Llama-2 [34] with a prompt as discussed in Appendix C. The loss weights in Eq. 10 are set as: λ 1 = 0.5, λ 2 = 0.5, λ 3 = 1.0. We train HENASY on two A6000 GPUs, in 5 epochs with AdamW optimizer [37] at fixed learning rate of 3e -5, and with batch size of 128. We initialize global encoder and text encoder with pretrained model provided from [5] , but freeze them in the entire training process.Table 1 : Comparison on the zero-shot transfer over EgoMCQ, EK100-MIR, EK100-CLS, and EGTEA. HelpingHands* refers to our re-produced results with TSF-B backbone using provided source code [4] . Methods EgoNLQ EgoMQ mIoU@0.3 mIoU@0.5 R1@0.5 R5@0.5 mAP R1 R5 R1 R5 SlowFast [38] 5.5 10. 7 Methods EgoMCQ EK100-MIR EK100-CLS EGTEA Inter Intra mAP nDCG Top-1 Top-5 Top-1 Mean V-T T-V Avg V-T T-VEgo4D benchmarks [1] . Ego4D is the largest publicly available egocentric video dataset, featuring 3,670 hours of daily-life activity video for a wide range of benchmarks. We evaluate on three tasks:• EgoMCQ [3]: A multi-choice questions task to select the correct video clip from 5 candidates for each query. Accuracy is evaluated in intra-/inter-video (candidates from the same/different video). • EgoNLQ: A sub-task in episodic memory involving localizing video intervals that answer a given a free-form text query. Evaluation metrics include Recall@K for mIoU thresholds θ, where K ∈ {1, 5} and θ ∈ {0.3, 0.5}. • EgoMQ: Also a sub-task of episodic memory, it involves identifying and categorizing action instances from 110 activity classes. Evaluation metrics are recalls (at mIoU=0.5) and mean Average Precision (mAP).EpicKitchens 100 benchmarks [2] : This dataset focuses on indoor and kitchen activities with 100 hours of video. We evaluate two tasks:• EK100-MIR: A multi-instance retrieval task evaluating video and narration matching in both T→V and V→T. Metrics are mAP and normalized Discounted Cumulative Gain (nDCG). • EK100-CLS: A action recognition task classifying videos into 300 noun classes or 97 verb classes.Metrics are Top-1 and Top-5 accuracy.EGTEA benchmark [13] : This dataset contains 28 hours of video with 106 classes. We evaluate fine-grained cooking action recognition EGTEA and report Top-1 and mean accuracies.Evaluation Protocols. We evaluate our model using three protocols:• Zero-Shot Transfer assesses generalization on unseen data and tasks without extra tuning. We conduct zero-shot evaluation EgoMCQ, EK100-MIR, EK100-CLS, and EGTEA. • Visual & Textual Representation is evaluated through EgoNLQ and EgoMC, where we use our pre-trained model as a visual/textual feature extractor. Following [4, 5] , we train downstream models (VSLNet [15] for EgoNLQ, VSGN [14] for EgoMC) with pre-computed features. • Vision-Language Grounding. We evaluate local entity understanding and interpretation via qualitative results on EgoCLIP [3] . We illustrate the saliency maps produced by our model and compare it with bounding boxes from [4] . Comparison in Zero-shot Transfer. In Table 1 , to ensure fairness, we re-train HelpingHands [4] using their official codebase with TSF-B backbone and the same pre-trained weights as ours, provided by LaViLa [5] . Our model consistently outperforms previous SOTA, achieving 3.1% improvement in top-1 accuracy on EK100-CLS, 0.5% and 0.3% increase in intra-and inter-video accuracy on EgoMCQ, and 0.5% improvement in mean accuracy on EGTEA. It also competes competitively with HelpingHands in the video/text retrieval EK100-MIR. Overall, our method demonstrates strong performance for zero-shot transfer across multiple benchmarks.Comparison in Visual & Textual Representation. In Table 2 , our method outperforms prior SOTA models across all metrics in EgoNLQ by adequate gaps. In EgoMQ, HENASY shows comparable performance, particularly excelling in mAP where it surpasses SOTA by 1%. This highlights HENASY's effectiveness when being applied to downstream models for features extraction.Vision-Language Grounding We include qualitative experiment (Fig. 4 ) to compare with Help-ingHands [4] , which, in our knowledge, is the only VLM including weak visual grounding capacity via bounding boxes. As we can see, HENASY provides stronger interpretation with saliency maps reflecting dynamically evolving regions that most related to both appearance and motion queries. Furthermore, HelpingHands cannot correctly perform grounding with verb phrases (e.g., "scrolling the phone"), therefore, we show the bounding box of noun instead (e.g., "phone"). [4] on EgoCLIP [3] . Left: comparison with a noun query obtained from narration and the pseudo-groundtruth boxes detected by [33] for reference. Right: verb phrase in the narration is used for comparison, as verb phrase cannot be captured by [33] , we do not include pseudo boxes.Losses: We investigate various combinations of multi-grained loss components and report results in Table 3 . We found that HENASY trained only with instance-level loss L ego yields 1-2% lower across all benchmarks compared to full loss setting. Besides, L V EC contributes slightly more to performance gains in EgoMCQ and EK100-MIR, compared to L N EC . Finally, L proj shows a slight improvement of the overall performance.We evaluate the influence of the entityaware (EA) decoder on zero-shot tasks in the first two rows of Table 4 . In the first experiment (labeled as 'w/ avg. pool'), the proposed EA decoder is omitted, and entity tokens are merely processed using average pooling to generate the video representation. This configuration results in a notable decline in performance across all benchmarks. In the second experiment (labeled as 'w/ SA Dec.'), video patch tokens are combined with entity tokens and then input into a self-attention decoder, which has the same dimensions as our EA decoder. This setup leads to a performance decrease of approximately 2% across benchmarks compared to our proposed decoder ('complete settings').These ablation studies show that modeling interactions between global features and entity embeddings plays a critical role, and the proposed design of entity-aware decoder significantly enhances overall model performance.We report the effect of bootstrapping stage in the third row of Table 4 (labeled as w/o bootstrap), where we remove bootstrapping stage by directly processing video patch tokens. The performance degrades by 1% across all benchmarks, showing the effectiveness of this design choice.Computational and Memory Costs: We compare our method with HelpingHands [4] in Table 5 .Our model is slightly more expensive but quite competitive in terms of memory requirements, the number of parameters and GFLOPs. Importantly, our inference time is 3 times faster than that of the HelpingHands. This superior running time of HENASY compared to HelpingHands can be attributed to HelpingHands' utilization of an autoregressive decoder, which reduces parallel computations and makes it less efficient despite its lower computational cost. 
conclusions: 
 In this work, we explored the Hierarchical Entities Assembly framework, dubbed HENASY, which is designed to improve video representation of previous vision-language models by addressing their limitations in fine-grained modeling. Our model explicitly captures the dynamic interactions between visual entities to form a comprehensive video representation. Our experiments showed that HENASY outperforms existing SOTA methods across challenging egocentric video understanding benchmarks like EgoMCQ, EK100-MIR, EK100-CLS, EgoNLQ, and EgoMQ in both zero-shot transfer and feature extraction settings, while also demonstrating strong interpretation capabilities. Despite these strengths, there are several opportunities for future work to improve our model further.Limitations and Future Works Although our focus has been on tasks utilizing ViT encoders for a variety of benchmarks, we believe it is important to extend HENASY to generative tasks such as video generation (e.g., stable diffusion) or to handle long-form videos. While HENASY can provide interpretability by focusing on relevant scene entities for both objects and actions, it is still limited in explicitly showing the interactions between scene entities. This necessitates the development of a dynamic scene graph, which remains an open question due to the unavailability of data. 
 
 Appendix: Divided Space-Time (DST) block [32] is mainly utilized in the global encoder and entity grouping stage of the local entity encoder in our HENASY framework.In global encoder, DST typically takes a concatenation of learnable CLS token and video patch tokens, i.e., [c l ; z l ] as inputs. While in the local entity encoder, inputs to DST comprises segment tokens g l entity and segment tokens s l . A DST block reduces computational cost of a full space-time attention by factorizing it into time and space attention, consecutively:ỹl t,k = T t ′ =1 Softmax (q l t,k • k l t ′ ,k )/ d h v l t ′ ,k y l t,k = K k ′ =1Softmax (q l t,k Given learnable group tokens g q ∈ R Q×D and input tokens to be grouped i ∈ R T ×I×D , we follow [12] to compute the 3D similarity array A ∈ R T ×Q×I between each video-level group token g i ∈ g q and every segment token i t,j ∈ k, where t and j are temporal and spatial indices, respectively. Gumbel-Softmax [39] is then applied to rescale similarity matrices over group tokens:A t,i,j = exp (W q g l i • W i i t,j + γ i ) Q k=1 exp (W q q k • W i i t,j + γ k )where W q and W i are learned linear projections for group and segment tokens, respectively, and γ i is sampled from Gumbel(0, 1) distribution.Group Assignment. Afterwards, a segment token is hardly assigned to a group token via arg max operation over group tokens (non-differentiable) with the straight-through trick [40] to allow end-toend training: Ã = one-hot arg maxEQUATIONwhere sg(•) is a stop-gradient function, and one-hot(•) operator converts the assigned group indices into one-hot vectors.Saliency maps of each dynamic entity that evolving across frames of input video can be constructed from similarity arrays produced in temporal-aware grouping layers at bootstrapping and entity grouping stage. Let denote them as A boot and A entity , respectively. We first compute the assignment probability array between video patches at each frame t and final entity tokens by the following equation:EQUATION)where t is a frame index in T , and M ∈ R T ×K×E (K is the number of patches per frame). Then, saliency maps can be obtained via softmax activation function over the patches M = softmax K (M).Splitting the saliency array M over entity dimension, we can obtain saliency maps of all frames, each of which highlights the spatial location and shapes of the corresponding entity. 


Decision: This paper introduces a novel framework for egocentric video language understanding. The authors proposed to learn compositional and hierarchal video representations by three levels: 1) global video features from a global video encoder. 2) entity features by assembling dynamic entities from video patches via local entity encoder, 3) interactions between entities and global context by an entity-aware decoder. The paper received mixed reviews with 2 borderline accept, 1 weak accept and 1 borderline reject. Most reviewers see the merits of the paper: 1) clear motivation 2) extensive experiments on multiple downstream tasks 3) well-written. There are a few concerns raised by reviewers as well. Reviewer EqYJ and 4RbK think the improvement over Helping hands is marginal; Reviewer xaMC and 4RbK want to understand the delta between this work and group ViT; Reviewer cNiH and Reviewer cNiH question about scalability. Authors have provided detailed answers to reviewers' questions in their rebuttal. In specific, authors provided clear architectural comparison, more experiment results based on different base models, and comparison with GroupViT. Two reviewers acknowledged that their questions are resolved and increased their final scores. Considering all reviews and discussion, the AC recommends to accept this paper. The AC also encourages the authors to include the additional experiment results in their final version.

Reviews: review: paper_summary: paper_summary: This paper presents HENASY (Hierarchical ENtities ASsemblY), a pretraining framework to learn scene-entities representations for egocentric videos. The authors proposed to learn compositional and hierarchal video representations by three levels: 1) global video features from a global video encoder. 2) entity features by assembling dynamic entities from video patches via local entity encoder, 3) interactions between entities and global context by an entity-aware decoder. With the designed model structure and losses, the authors conducted experiments on several egocentric tasks on two datasets. The results show the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. My major concern is the performance shown in the downstream task. In table 1, though the proposed methods generally achieved better performance compared with previous sota models (HelpingHands), the improvement seems to be minor. Besides, the paper didn't report the error analysis to dimish the influcen of randomness, which makes the contribution and effectiveness of the proposed method less convincing.
2. In Projection loss calculation, how is the mask predicted is not clearly presented.
3. The writing could be further improved. In line 127 it should be $L_{ego} = L_{ego}^{v2t} + L_{ego}^{t2v}$. In section 4, too many variables are used including z, c, g, s, e, making it hard to follow the context while reading.
questions: questions: See weakness section.
limitations: limitations: Yes
paper_summary: paper_summary: The paper introduced HENASY, a novel framework for enhancing video-language understanding in egocentric videos. It utilized multi-grained contrastive losses from alignments of  video-narration, noun-entity, and verb-entity, to improve interpretability and performance. The method showed competitive results across various downstream tasks such as video-text retrieval, action recognition and visual question answering.
strength_weakness: strength_weakness: * Such an idea of leveraging fine-grained information, especially alignment of nouns/verbs and the video contents, has been intensively studies in non-egocentric scenarios and there are a number of related papers [1, 2]. It is not clear whether there is a significant different in egocentric videos. Otherwise, it is expected to improve the model performance by incorporating those fine-grained alignments in the pre-training stage.
* Even though the structure to extract noun/verb entities was a little different, I feel like the essential idea was still similar to GroupViT. The authors mentioned that if directly using GroupViT, the performance would drop. Was the comparison in Table 3, w/o bootstrapping? Actually, in GroupViT there was also a hierarchical design to use different query tokens at different stages. Then what's the main contribution of the design in the paper? I am wondering if simply adding the number of trainable parameters of GroupViT to the same number in the paper would lead to comparable performance.
* I am wondering if co-training with data mixes from both egocentric and non-egocentric videos would boost performance for both domains. In addition, can the model pre-trained on the egocentric scenario be transferred to the non-egocentric tasks?
* The evaluation on visual grounding was not sufficient. Currently there were only some qualitative results in Figure 4. It would be better to report some quantitative results for more concrete tasks.

[1] Ge, Yuying, et al. "Bridging video-text retrieval with multiple choice questions." CVPR 2022.

[2] Xiong, Yuanhao, et al. "Structured Video-Language Modeling with Temporal Grouping and Spatial Grounding." ICLR 2024
questions: questions: My questions are listed in the part of weaknesses above.
limitations: limitations: They had a separate "Limitations" section and elaborated on future works that could improve the model.
paper_summary: paper_summary: The paper introduces a novel framework for improving interpretability and performance in video-language models, specifically for egocentric videos. The framework, Hierarchical ENtities ASsemblY, employs a spatiotemporal token grouping mechanism that assembles and models relationships between dynamically evolving scene entities. This method aims to mimic human perceptual abilities by focusing on a compositional understanding of scene dynamics. The training of HENASY incorporates multi-grained contrastive losses, enhancing the model's ability to produce robust entity-level and video-level representations. Extensive experimental results demonstrate that HENASY significantly outperforms existing benchmarks on a range of egocentric video tasks.
strength_weakness: strength_weakness: 1. The paper lacks a detailed discussion on the scalability and computational demands of HENASY, which are crucial for its application in real-world settings, especially on resource-constrained devices.
2. While HENASY shows impressive results in controlled experiments, its ability to generalize across diverse real-world scenarios and different video domains remains underexplored.
questions: questions: 1. Could the authors provide insights into HENASY's performance on diverse real-world datasets and its computational efficiency and scalability compared to existing methods?
2. Can the authors elaborate on the spatiotemporal token grouping mechanism’s impact on training dynamics and model convergence, and provide a comparative analysis of interpretability with other state-of-the-art methods?
limitations: limitations: The paper should explicitly address potential limitations related to the scalability of HENASY and its performance on varied real-world datasets. Additionally, the authors should discuss any potential negative societal impacts, such as privacy concerns in the context of egocentric video analysis, and suggest possible mitigation strategies for such issues to ensure ethical deployment.
paper_summary: paper_summary: This paper proposed HENASY, a novel framework for learning egocentric video-language models where texts are grounded to visual scene entities. Its main idea is to use both global and local visual encoder to encode video features so that nouns and verb phrases in the paired text could be matched individually. The overall framework is optimized through contrastive losses while a projection loss is further introduced to improve the grounding quality. Experiments show that the proposed the method could achieve good performance on a wide range of egocentric video tasks including video/text retrieval, action recognition, multi-choice query, natural language query, and moments query.
strength_weakness: strength_weakness: 1. The performance improvement of the proposed method on most tasks are very marginal, compared to HelpingHands, which is the most related work to the paper. Given the fact that the design of  HelpingHands is much simpler than the proposed methods, I have concern on whether the complex components introduced in this paper is efficient.

2. Given the huge complexity of the proposed method, some important studies on the proposed design choices are missing. E.g., (1) the authors mentioned that using GroupViT to directly process input video patch tokens will diminishe performance but the results are not reported in the paper; (2) rather than using [5] to initialize the global encoder, have the authors tried other options, and what are the results? These help the reader to gain a better understanding of the robustness of the proposed method.

3. What about the scaling properties of the proposed method? Does it scale well with larger model sizes or more training data?
questions: questions: Please refer to the Weakness section.
limitations: limitations: The authors have widely discussed the limitation of the proposed work and also its challenges in modeling the interactions between complex scene entities.

Comments: title: Reminder: Feedback Request Before Discussion Period Closes
Reminder: Feedback Request Before Discussion Period Closes
Kindly Reminder before Discussion Phase Ends
Kindly Follow-up on Review Rating Update
Kindly Follow-up on Rebuttal
Kindly Follow-up Our Rebuttal
Appreciation for Your Revised Feedback and Acceptance
Follow Up Our Rebuttal
Follow Up Our Rebuttal
Appreciation
Follow Up Our Rebuttal
Thank you for your valuable comments !
comment: Dear Reviewer xaMC,

As we are nearing the end of the discussion phase, we kindly seek your feedback to ensure that our responses have addressed your concerns. If our rebuttal satisfactorily addressed your concerns and questions, we would be very grateful if you would consider to raise your rating.

Thank you very much for your attention.
Dear Reviewer cNiH,

As we are nearing the end of the discussion phase, we kindly seek your feedback to ensure that our responses have addressed your initial concerns. If our rebuttal satisfactorily addressed your concerns and questions, we would be very grateful if you would consider to raise your rating.

Thank you very much for your attention.
Dear Reviewer EqYJ,

We apologize for the urgency of this message and the multiple reminders. As we approach the final hours of the discussion phase, we kindly request your assistance in updating the rating to reflect your previously indicated preference for a "weak accept". We appreciate your efforts and understanding in resolving this matter swiftly.

We would like to thank you again for your invaluable feedback and consideration to raise your rating.
Dear Reviewer EqYJ,

Thank you once again for your constructive feedback and for the time you have invested in reviewing our submission. We greatly appreciate your decision to raise your rating to “*weak accept*”.

We noticed that the updated rating appears as “*borderline accept*” in the system. Therefore, we would like to kindly follow up with you regarding the rating update. We apologize for any inconvenience this request may cause and truly value your support and understanding. Thank you for your attention to this matter.
Dear Reviewer cNiH,

As we approach the end of the discussion period, we wanted to kindly remind you of our rebuttal. We are keen to ensure that our responses have thoroughly addressed your concerns and would appreciate any additional feedback you may have.  

Thank you once again for your invaluable insights and the time you have invested in reviewing our work. We look forward to hearing from you soon.
Dear Reviewer xaMC,

As we approach the end of the discussion period, we wanted to kindly remind you of our rebuttal. We are keen to ensure that our responses have thoroughly addressed your concerns and would appreciate any additional feedback you may have.  

Thank you once again for your invaluable insights and the time you have invested in reviewing our work. We look forward to hearing from you soon.
Thank you very much for your encouraging feedback and for taking the time to re-evaluate our work. We are delighted to hear that our responses and the new experiment have addressed your concerns effectively.

We will certainly incorporate the new results into the final version of the paper when one additional page is allowed, as you recommended.

We are grateful for your updated rating and truly appreciate your support for our work. Thank you once again for your thoughtful and thorough review.
Thanks for your detailed reply. I think the provided new results make the model design more convincing and encourage the authors to add them to the revised version of the paper. Given all of my concerns have been addressed, I would like to increase my rating to weak accept.
Dear reviewer 4RbK, 

Thank you for taking the time to review our response. We value your follow-up question regarding our use GroupViT. We appreciate your insights and the opportunity to further clarify our contributions and novelty over GroupViT.

Initially, it is important to note that GroupViT was originally designed for static images and does not inherently support video processing. To address this limitation, we proposed a temporal-aware grouping (TAG) mechanism specifically adapted for video patches grouping. This mechanism is critical for our hierarchical dynamic entities grouping approach, hence, it cannot be removed.

With such mechanism in place, we tried to leverage the original architecture of GroupViT as much as possible, including number of groups and stages (which are fixed in all reported experiments). Unfortunately, the experiment resulted in significantly poor performance. Let’s call this version as GroupViT+TAG, below are its performances on EK-MIR and EK-CLS datasets and comparisons to other ablation studies (Table 3):

| Model | GroupViT | TAG | Bootstrapping stage | Entity-aware decoder | EK-MIR (Avg. mAP) | EK-MIR (Avg. nDCG) | EK-CLS (Top-1 Acc.) | EK-CLS (Top-5 Acc.) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GroupViT+TAG | ✓ | ✓ |  |  | 17.2 | 25.2 | 6.2 | 17.8 |
| w/o entity-aware decoder | ✓ | ✓ | ✓ |  | 18.8 | 25.5 | 6.7 | 18.1 |
| w/o bootstrapping stage | ✓ | ✓ |  | ✓ | 31.1 | 32.6 | 19.2 | 37.9 |
| complete setting | ✓ | ✓ | ✓ | ✓ | 31.3 | 33.2 | 19.5 | 38.2 |

We would like to note that the ablation study of GroupViT+TAG was conducted at the initial phase of our work.

This empirical finding of the GroupViT+TAG led us to propose two critical enhancements: the bootstrapping stage and the entity-aware decoder. By integrating these modules, our model extends beyond the original capabilities of GroupViT.

We hope this response would answer your concern and we welcome any further discussion you may have.
I thank the authors for their responses. Most of my concerns have been addressed. I just have one followup question on "2. Ablation studies" (on GourpViT): I acknowledge the proposed improvements on GroupViT, but I wonder did you have a chance to get the numbers if GroupViT is used. I am asking because my major concern about the work is still the additional complexity introduced by the method, therefore presenting the performance gain brought by the model itself should make the proposed method more convincing.
Dear Reviewer 4RbK

We sincerely appreciate the time and effort you have dedicated to providing feedback on our work. Your insights are invaluable in helping us improve its clarity and overall quality. We want to follow up to check if our response fully addressed your concerns/questions before ending the discussion period.

Thank you once again, and we look forward to your feedback.
Dear Reviewer cNiH

We sincerely appreciate the time and effort you have dedicated to providing feedback on our work. Your insights are invaluable in helping us improve its clarity and overall quality. We want to follow up to check if our response fully addressed your concerns/questions before ending the discussion period.

Thank you once again, and we look forward to your feedback.
Dear Reviewer EqYJ,

Thank you for taking the time to read our rebuttal and for providing a positive rating. We greatly appreciate the valuable feedback you have given us, and we will incorporate your suggestions in the final revision of our paper.

Thank you once again.
Dear Reviewer xaMC,

We sincerely appreciate the time and effort you have dedicated to providing feedback on our work. Your insights are invaluable in helping us improve its clarity and overall quality. We want to follow up to check if our response fully addressed your concerns/questions before ending the discussion period.

Thank you once again, and we look forward to your feedback.
Thank you for the authors' responses. I think the responses generally resolve my concerns and I decide to raise my score to weak accept.
We would like to extend our gratitute once again to the reviewers for their constructive feedback and insightful questions, which have significantly helped us enhance the quality of our paper. We have diligently addressed all questions and suggestions from the reviewers in our rebuttal, including conducting new experiments as recommended.

We welcome any additional comments or insights the reviewers may have and are committed to promptly addressing further questions that arise.

In light of the rebuttal and additional data presented, we respectfully request that the reviewers reconsider the evaluation score for our paper. Thank you very much for your thoughtful consideration !