EgoDTM: Towards 3D-Aware Egocentric Video-Language Pretraining
Authors: Boshen Xu
Yuting Mei
liu xinbi
Sipeng Zheng
Qin Jin
Venue: NeurIPS 2025 poster ()
Keywords: egocentric vision
video-language pretraining
multi-task learning
3D-aware video representation
Primary area: applications

Abstract: Egocentric video-language pretraining has significantly advanced video representation learning. Humans perceive and interact with a fully 3D world, developing spatial awareness that extends beyond text-based understanding. However, most previous works learn from 1D text or 2D visual cues, such as bounding boxes, which inherently lack 3D understanding.  To bridge this gap, we introduce EgoDTM, an Egocentric Depth- and Text-aware Model, jointly trained through large-scale 3D-aware video pretraining and video-text contrastive learning. EgoDTM incorporates a lightweight 3D-aware decoder to efficiently learn 3D-awareness from pseudo depth maps generated by depth estimation models. To further facilitate 3D-aware video pretraining, we enrich the original brief captions with hand-object visual cues by organically combining several foundation models. Extensive experiments demonstrate EgoDTM's superior performance across diverse downstream tasks, highlighting its superior 3D-aware visual understanding. Code: \url{https://github.com/xuboshen/EgoDTM}.

EgoDTM: Towards 3D-Aware Egocentric Video-Language Pretraining
Boshen Xu Yuting Mei!
Xinbi Liu' AIM3 Lab, Renmin University of China "BeingBeyond
Sipeng Zheng² Qin Jin
Abstract
Egocentric video-language pretraining has significantly advanced video represen- tation learning. Humans perceive and interact with fully 3D world, develop- ing spatial awareness that extends beyond text-based understanding. However, most previous works learn from ID text or 2D visual cues, such as bounding boxes, which inherently lack 3D understanding. To bridge this gap, we intro- duce EgoDTM, an Egocentric Depth- and Text-aware Model, jointly trained through large-scale 3D-aware video pretraining and video-text contrastive learn- ing. EgoDTM incorporates a lightweight 3D-aware decoder to efficiently learn 3D-awareness from pseudo depth maps generated by depth estimation models. To further facilitate 3D-aware video pretraining, we enrich the original brief captions with hand-object visual cues by organically combining several foundation mod- els. Extensive experiments demonstrate EgoDTM's superior performance across diverse downstream tasks, highlighting its superior 3D-aware visual understand- ing. Code: https: /tte:://ghhthtttt http///booeennngDDD..
Introduction
The development of embodied Al capable of fulfilling diverse societal roles has long been goal in artificial intelligence [25, 49]. A promising path to achieving this vision in- volves developing egocentric AL, which can comprehend human activities by analyzing large-scale egocentric videos captured from first-person perspective using wearable devices. These videos provide rich insights into hand- object interactions (HOI) [19, 79, 11, 39], of- fering window into how individuals inter- act with nearby objects through their hands. With the emergence of large-scale egocentric datasets [22], video-language pretraining [36, 87] has become a dominant paradigm for leamn- ing egocentric video representations, signifi- cantly improving performance on downstream tasks such video-text retrieval [11, 67] and action recognition [66, 34].
Auxillary Supervision
Alignment
[a] None (EgoVLP, LaViLa)
[b) Region-Aware (Hands)
Text Encoder
Video Encoder
(c) Depth-Aware (ours)
#C C pick up bag of cloth
Figure Comparison of egocentric pretraining paradigms. While previous paradigms focus on text-based [36, 51, 87] or 2D spatial region-aware learning [80], EgoDTM incorporates 3D spatial information enhance video representations.
Humans possess an innate ability to perceive and reason about 3D spatial relationships, effortlessly
perceiving relative distances and spatial arrangements from visual cues alone [68, 28, 65]. While there is no universal definition of model's 3D awareness, we define it as the ability to infer 3D
Qin Jin is the corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

information from 2D images. However, models pretrained solely on 2D frames and general- purpose text (e.g,, CLIP [52]) often struggle to develop robust 3D awareness [18, 42, 56]. To equip ego- centric models with such 3D awareness, we wonder: Can video-language models better understand egocentric contexts by incorporating 3D-aware perception?
Achieving this goal presents two key challenges. First, 3D representations are diverse, and obtaining corresponding labels is often costly. Recent works [78. 84] explore using multi-view images for 3D reconstruction to distill 3D awareness into 2D models. However, in egocentric scenarios, depth maps are more practical and easier to obtain. Depth maps provide direct 3D distance information and help distinguish salient objects from the background, offering crucial cues for spatial understanding. Unfortunately, existing egocentric datasets with depth maps [39, 55, 29] remain limited in both scale and diversity for large-scale pretraining. Recent foundation models for depth estimation [76, 77, 5] demonstrate strong out-of-domain generalization, enabling us to construct a large-scale depth- augmented egocentric dataset at a low cost.
maps
Second, effective 3D-aware pretraining requires bridging the modality gap between depth and text. As illustrated in Figure 1, common video-language pretraining [36, 51, 87] primarily relies on tex- tual supervision, thus avoiding this challenge. Recent works [80, 47] explore region-aware video- language pretraining using non-pixel-level cues, such as text and sparse object bounding boxes, which face only minor modality gaps. However, unlike these non-pixel-level cues, regressing the pixel-level outputs (e.g., depth maps) requires fundamentally different capabilities compared to pre- dicting texts or bounding boxes, as suggested by previous studies [30, 31j. Additionally, depth es- timation typically requires high-resolution inputs and multi-scale features, whereas video-language models rely on large batch sizes for contrastive learning, making direct integration nontrivial. Thus, designing an effective learning approach and enriching textual supervision with spatial information are crucial for successful 3D-aware pretraining.
To address these aforementioned challenges, we introduce EgoDTM, a novel 3D-aware egocen- tric video-language model that learns video representation from depth maps and spatially informed captions. In addition to dual transformer encoders for video-text alignment, EgoDTM adopts a 3D- aware module for video pretraining and data construction pipeline enrich captions with spatial information. To adapt the 3D-aware module for the vidco-language framework, we propose uni- fied visual representation with a lightweight depth decoder, supervised by depth predictions from foundation models [76]. Specifically, the lightweight depth decoder takes as input the visual repre- sentation from the last layer of the video encoder and predicts the composition of discrete adaptive bins [3, 4, 64, 35] to estimate low-resolution depth maps. Moreover, we adopt off-the-shelf visual foundation models [63, 59] to create high-quality HOI masks by first detecting HOIs, then select- ing key frames, and finally tracking bidirectionally. To enrich the original captions with spatial information, we leverage a large language model (LLM) guided by the temporally consistent HOI masks. Through 3D-aware video-language pretraining, EgoDTM improves visual generalization downstream tasks involving egocentric HOls.
Our contributions are threefold: (1) We introduce EgoDTMM 3D-aware egocentric video-language model learned from 3D-aware video-language pretraining. (2) We develop a lightweight 3D-aware decoder for depth estimation and a data construction pipeline to enrich captions with spatial infor- mation. As a byproduct, we generate millions of egocentric data, including captions, HOI boxes, HOI masks, and depth maps. (3) Extensive experiments demonstrate that EgoDTM significantly en- hances performance on video understanding tasks like video-text matching, and 3D understanding tasks like robot manipulation.
2 Related Works
Egocentric Video-Language Pretraining. Egocentric video understanding [22, 8, 50, 83, 43, 74, 11, 73, 75, 33, 72] typically involves human daily interaction between hands and objects from a first-person perspective. Inspired by visual-language pretraining paradigms in third- person datasets [2, 41, 52, 47, 80]. EL [36] firstly proposes to conduct egocentric video- language pretraining on the large-seale egocentric dataset Ego4D [22], which aims to learn video representations from massive video-text data via contrastive learning. Since EgoVLP, similar paradigms [36, 51, 87, 1] have gained large success in EgoHOI understanding. For example, LaV. iLa [87] employs video-conditioned GPT-2 [53] and a T5 [54] rephraser to generate text descrip-
2

---

Pouudo-Lablls Dg.
Predictions
Eapph
video V
Teacher Model Depthlnything)
dapts
Lightwiight
Teacher Model
3D Aware Decoder
Plain Pyramid
Depth-Aosare Transforner
Hlead
Eaggete
Video Encoder
Text
deoth-bnccamtee
depth queries
(1) 3D-Aware Vidco Pretraining
Caption w/ HOI Spatial prior yrar
(2) Spatial-Aware Text Enrichment
IC cpriica medium stood bag clstren
sideo
Spatial
Deteett (HOIB))
TRaCE [AMI)
Generate (LUM)
Injection
fromith foor with both hands
(1]
(2)
Original Caption
pidic bay
temporal consistent masks M
Only active during training
cluthes
Figure 2: EgoDTM learns 3D-aware representations from depth and text. Our dual encoders are constructed using only transformers [16, 70, 52] with flash attention [12]. During pretraining,
conduct (1) 3D-aware video pretraining: we design lightweight 3D-aware decoder to predict depth using visual feature maps, supervised by a teacher foundation model [77]. The decoder contains a plain feature pyramid to get multi-scale features, a depth-aware transformer decoder to process depth queries with video features, and the heads predict depth maps; (2) Spatial-aware textual enrichment: we enhance captions with spatial information by organically combining foundation pipeline. Different green markers denote inconsistency HOI
models in the predictions; identical ones indicate consistency.
tions for egocentric videos to expand video-text data. Despite the progress, pretraining with text alone often lacks precision in target localization. In response, some studies have focused on de- veloping region-aware representations for EgoVLMs. For example, HelpingHands [80] proposes learning from noisy hand-object detection results, while HENASY [47] ensembles an additional hi- erarchical encoder to learn HOI region-ave aware representation. However, these methods are limited 2D reasoning and lack an understanding of real-world 3D context. this work, we take step towards 3D-aware egocentric video-language pretraining.
3D-Awareness of 2D Vision Models. As suggested by research in developmental psychology and psychophysics, we humans are capable understanding 3D information, like depth and orienta- tions, from only 2D visual signals [68, 28, 65]. With the growing interest in embodied intelli- gence [25, 44] and personal AI assistants [49], the demand for 3D-aware 2D visual models has become increasingly important. Although there is no widely accepted consensus define and build effective 3D-aware models, some studies have started to assess the 3D awareness of visual founda- tion models. For example, Probing3D [18] demonstrated that CLIP, which is pretrained exclusively with text, is significantly limited in predicting depth and 3D surface normals. EysWide revealed that CLIP struggles with 3D-related tasks, such as recognizing object orientation and un- derstanding compositional contexts. More recently, relevant works have made endeavors to enable visual foundation models (e.g., CLIP) with 3D awareness by fine-tuning on 3D-aware data. For instance, FIT3D [78] finetunes visual foundation models with 3D Gaussian features reconstructed from multi-view images. SpatialVI.M [9] conducts visual instruction tuning on metric depth-aware QA pairs for multi-modal large language models. Different from these learning paradigms, define 3D-awareness as the model's latent ability to estimate depth and pretrain an egocentric video- language model with a lightweight 3D-aware decoder.
3 Method
Our objective is to develop 3D-aware egocentric lideo-language model, where 3D-awareness is defined as the model's ability to infer depth information from its representations. EgoDTM achieves this by pretraining depth-aware and text-aligned video representations. Figure 2 illustrates the architecture of our proposed EgoDTM. We first describe the video-language model architecture in Section 3.1. Next, we introduce the 3D-aware video pretraining in Section 3.2 and then present

---

our approach to generate aware captions in Section 3.3. The details of training and inference are presented Section 3.4.
3.1 Video-Language Model Architecture
The video-language model basically consists of a video encoder and a text encoder.
Video and text encoders. Our video encoder employs plain vision transformer [16] backbone to process video tokens. The video input E RF = W x3 is divided into N = Fo
F x Ho and
H x Wo
W
non-overlapping cubes of dimension F6 x Ho x Wo x 3. where F is the number of frames, W denote height and width. These cubes are combined with positional embeddings and processed by the video encoder to produce a feature map Vz E IN N Note that a [CLS] token added to represent the global video embedding Vels- For the text encoder, captions T are tokenized via Byte Pair Encoding (BPE) tokenizer [62] and encoded with a transformer initialized from CLIP [52] to output the sentence embedding tels- Both encoders adopt the flash-attention [12] to mitigate the memory bottleneck of the attention mechanism.
Video-text alignment loss. We use the InfoNCE loss to align video and text embeddings. For simplicity, we define the video-to-text loss within a batch B ((v,, t. E as follows:
cxp(v; tz/r) Cyre -
1 Log log E t//T)
(1)
where (v,, tú) represent the video and text embeddings (Vela. tels) for the i-th (video, caption) pair within the batch. The text-to-video loss is defined symmetrically.
3.2 3D-Aware Video Pretraining
A critical issue in leveraging depth maps for 3D-aware pretraining is that depth maps typically high-resolution images (e.g.. 1024p). However, most low-level details in these maps are redundant, as video-language models primarily benefit from the relative depth information between patches. Therefore, high-resolution depth maps may be unnecessary and inefficient for pretraining. Addi- tionally, video-text pretraining often requires a large batch size to ensure good performance, further emphasizing the need for lightweight and memory-efficient processing. To address these concerns, design a lightweight 3D-aware decoder estimate low-resolution depth maps from video repre- sentations. Specifically, the decoder consists of plain feature pyramid, a depth-aware transformer decoder, and two specialized heads, described below.
Plain feature pyramid. Depth estimation methods [48, 3, 57] often utilize scale features from intermediate backbone layers. However, video-language pretraining typically uses only the final-layer representations. To reconcile these differences, we adopt a simplified FPN [32] that projects the video feature map Vz into multi-scale outputs (Vzix> V V ). where Vz denotes the feature map of size (N X m²) X C, and the size of the predicted depth maps equals N x m² Moreover, inspired by the adaptive bins [3, 4, 64, 35] in depth estimation, we segment the depth range into intervals and predict distributions each pixel over the interval centers.
Depth-aware transformer decoder. This module processes multi-scale features to generate depth- aware representations capable of estimating distinct distance ranges. The transformer decoder fol- lows standard architecture as [7, 10), transforming T. learnable depth queries Q E L.C into depth- aware representations Z E IR LXC using self-attention and cross-attention mechanisms. The cross attention processes Q as queries and concatenates (V2ix² V V V2m=1x)
as key-value pairs (k, v). The learned depth queries estimate depths at different distances, as illustrated by varied gray colors in Figure 2.
Bin head. We partition the depth range of [0, 1] into L intervals. The bin head F(.): : RC + [0.1]
processes depth-aware representations {z, V. to predict widths of depth interval bins satisfying El w;- 1 where we denote wo = 0. The i-th depth interval is defined by boundaries [ 17, Σl=1 wy). with its center computed as:
W4
C
Wi +
(2)
2
1=1

---

E (in din-oonin, unseen)
MECCANO (out- domainn, unseen)
Video
Video
Ours
Ours
DAv2
DAv2
H20 (out-of-domain, unseen)
Epic (out-of-domain, unseen)
Video
Video
DAv2
DAv2
Figure 3: Qualitative results of depth estimation from EgoDTM and DepthAnythingy2 [77](Dvv2)
on datasets including in-domain but unseen Ego4D validation set [22], out-of-domain and unseen data of EK100 [11], MECCAÑO [55], and H2O [29]. Note that DAv2 operates with high resolu- tion of 512p. while EgoDTM uses lower resolution input of 224p and generates a depth map at a resolution of 56p. Despite the lower resolution input. EgoDTM demonstrates intuitive generaliza- tion across unseen egocentric datasets with diverse environments, illuminations, backgrounds, and varying HOI object sizes.
Mask head. Then, we aim to predict distributions over the depth centers at each pixel. Specifically, the mask head g(.) RC EC takes z; imput, and the output m, multiplied by the largest feature map m, VVZ.. to acquire unnormalized probabilistic maps d; E R x where & denotes the composition of element-wise multiplication and sum along the feature dimension. After normal- izing {d+ by the Softmax operation along the query dimension L, we get the probabilistic map
D D Dr] E _ The process is described as:
D = Softmax (Z( & V
(3)
Finally, depth prediction δ E x is obtained by the linear combination of the depth-bin-centers and the probabilistic maps:
D
Ck
Dk
3D-aware pretraining loss. Given the lack of depth annotations in large-scale egocentric videos [22], we generate pseudo-depth labels using monocular depth estimation foundation mod-
els [57, 76, 77]. The pretraining loss is defined as:
Eddpth | Darl22
where and Drt are the predicted and ground-truths inverse depth maps within the range [0, 1].
3.3 Spatial-Aware Textual Enrichment
Previous egocentric video-language pretraining frameworks typically learn from short descriptions composed verbs and nouns. To foster 3D-aware pretraining, we argue that captions should contain more spatial-temporal information like HOI positions, object shapes, and movements. We therefore enrich the captions with visual cues by employing a detect-track. -enerrte pipeline tailored for ego- centric videos using foundation models [63, 59, 14].
Detect. We use the detector [63], HOID, finetuned on egocentric data to detect hands, objects, and their contact states for given frame. The detector f(-) receives V4 as input and outputs diverse
5

---

Epic Kitchens- Maaeeenss-OO--MM- 100-MIR
EGTEA
EgoMCQ
MAP (%)
nDCG (%)
Method
V-+ T
T- V
Avg.
V
T
T- V
Avg.
mean
topl
Inter
Intra
EgoVLP [36]
26.0
20.6
23.3
28.8
27.0
27.9
90.6
57.2
EgoVLPv2 [51]
35.1
26.6
30.8
33.7
30.4
32.0
30.9
35.1
91.0
60.9
LaViLa [87]
35.1
26.6
30.8
33.7
30.4
32.0
30.9
35.1
93.6
59.1
AVION [86]
37.1
28.7
32.9
34.4
31.0
32.7
38.6
42.3
94.4
62.1
HelpingHands' [80]
35.6
26.8
31.2
34.7
31.7
33.2
29.4
35.3
93.2
58.8
HENASY" [47]
35.5
27.1
31,3
34.6
31.7
33.2
29.6
35.9
94.1
61.3
EgoDTM (ours)
37.9
29.1
33.5
34.8
31.9
33.4
40.2
43.2
94.6
63.6
Table I: Comparison with state-of-the-art methods zero-shot video-text retrieval and action recognition. All methods use the same 4M videos from Ego4D for pretraining and have similar hidden dimension sizes. The numbers of the method with are sourced from [47].
information {(by, Cjs L, 8)};, where bj the bounding boxes with confidence scores cj. t object type, and is contact state of hand.
Track. Since predictions of HOID lack temporal consistency, we address this issue by tracking across the video. We utilize SAM2 [59] g(-), ), foundation model for promptable segmentation and tracking. The initial tracking frame V* is selected when the hands make contact with objects. The HOI masks of this frame obtained by prompting SAM2 with HOI boxes {m;); g(V*, {b; Iy), where the bounding boxes serve as negative prompts for each other. Finally, the model tracks HOI masks throughout all frames acquire the temporal consistent masks M ({m1,) ) j...... (m.......................... hy)-
Generate. As suggested by previous works [37, 20), LLMs are capable understanding visual cues like bounding boxes. We use an LLM h(-) to comprehend the HOI spatial information and original captions, then generate spatial-aware captions T (TT, M).
Equipped with the enriched video-text pairs, our new video-text contrastive learning objective is:
Laugyte
1
Σ
log exp(v; t2/T))
B
Exexpy; t2/T)
where t² rand(t,, t, is randomly sampled from the original and enriched text embedding t.
3.4 Training Strategy
Training and evaluation. We employ both the augmented video-text alignment loss and the 3D- aware pretraining loss to pretrain our model:
C Laugeie + Edepth
Our pretraining improves video-language representations for downstream egocentric tasks without modifying the dual-encoders themselves. As result, only the dual-encoders are used for down- stream tasks, introducing no additional computational costs at inference time.
Pretraining data. Our pretraining data consists of four million (video, text) pairs, with each video approximately 1 second long. We generate enhanced text descriptions and depth maps specifically for videos featuring hand-object interactions, resulting in two million (video, enriched texts, depths) triplets. Specifically, for each video, we extract eight depth maps using the DAv2-Large [77] model. The HOI boxes are detected by HOID [63], which is finetuned on egocentric data. The HOI masks are segmented and tracked using the same unified SAM2-Laas Large model [59]. The final enriched text descriptions are generated using the MaNSSeck-LLM-2000 model [14].
4 Experiments
To evaluate our EgoDTM, we conduct experiments from three perspectives: short video understand- ing (video-text retrieval and action recognition). 3D space comprehension (depth estimation and robot manipulation), and long video understanding (natural language query and moment query). These evaluations cover seven benchmarks across five datasets. In the following sections, detail

---

Scale- Aware Metries
Scale-Invarian Metrics
Method
61
d2t
dst
RMSEL
61
6u1
RMSEL
ConvNext [40]
0.721
0.965
0.991
0.644
0.727
0.969
0.996
0.593
CLIP [52]
0.795
0.966
0.988
0.624
0.811
0.976
0.994
0.559
EgP [36]
0.778
0.954
0,989
0.610
0.853
0.977
0.996
0.497
LaViLa [87]
0.801
0.954
0,987
0.598
0.811
0.964
0.993
0.552
AVION [86]
0.786
0.960
0.991
0.606
0.812
0.969
0.996
0.543
EgoDTM (ours)
0.826
0.964
0.993
0,539
0.848
0.977
0.998
0.481
Table 2: Comparisons of depth estimation task on H2O dataset.
the experimental setups (Section 4.1), main results (Section 4.2), ablation studies (Section 4.3), and further analyses (Section 4.4).
4.1 Benchmarks and Settings
Zero-shot video-text retrieval (ZS-VTR). To assess video-text alignment, we conduct zero-shot retrieval evaluations on text-to-video multi-choice retrieval on Ego4D [22] (EgoMCQ) and multi- instance video-text retrieval on Epic-Kitchens- 100 [11] (EKI00MIR). Following [36, 11], we use mean Average Precision (mAP) and the normalized Discounted Cumulative Gain (nDCG) as metrics for EKI00MIR and accuracy for EgoMCQ.
Zero-shot action recognition (ZS-AR). Action recognition is conducted in a video-to-text retrieval manner. We experiment on the EGTEA [34] and Epic- Epic-Kitchens- 100 [11] (EK 1 10CCLS). The task on EGTEA requires models to recognize 106 classes of cooking activities, and the EK100CLS includes evaluation on 97 verbs and 300 nouns in kitchens. The metrics include mean accuracy, top-1 and top-5 accuracy across test splits.
Depth estimation (DE). To assess the model's 3D awareness, we conduct depth estimation by fine- tuning our model on the H2O [29] dataset. Following [18], we add a prediction head composed of a linear layer and an upsampling convolution layer on top of the frozen video encoder. Sim- ilar to [18, 17]. the metrics are threshold accuracy (6:): percentage of pixels D, in ground-truth depth maps that satisfy max( Puye Day = 6, < thr* for thr = 1.25, i E {1,2,3), and RMSE:
D
|| - Dall2. The scale- invariant metrics are shifted and normalized from scale-aware metrics.
Robot Manipulation (RM). The robot must learn accurately perceive and interact in 3D spaces to accomplish daily tasks. Following previous works [45, 26], we evaluate the visual representations frozen perception modules for downstream policy learning within the Franka Kitchens simulation environment [23]. Five tasks are adopted: turn knob (TK), open door (OD), flip switch (FS), open microwave (OM), and slide door (SD). We use success rate as the metric.
Natural language queries (NLQ). The NLQ task localizes time intervals in a long video given a language query. We experiment the EgoNLQ task on Ego4D [22]. To fairly evaluate different pretrained visual representations, we extract the visual features by video encoders and text features by the same pretrained BERT [15], then train VSLNet [81] to solve this task. The evaluation metrics are "Rn@m" m"" where n E (1,5) and m E (0.3, 0.5). presenting the percentage of at least one of the top-n predicted intervals having loU greater than m.
Moment Queries (MQ). The MQ task is a video-only problem that aims to detect all temporal activity intervals in long video given specified activity category. We conduct experiments on the EgoMQ benchmark from Ego4D [22]. Following the setup in [36], we extract visual features using video encoders and then train VSGN model [85] to perform this task. The evaluation metrics are "R @n, m", consistent with those used the NLQ task.
Pretraining details. The dual-encodens are initialized by the checkpoint pretrained on the original four million video-text pairs [36] from Ego4D. EgoDTM is then trained for two epochs on A800 GPUs, which requires approximately 10 hours and a learning rate of 3e-5. The depth-aware trans- former decoder comprises six layers. The hidden dimension of the dual encoders is 768, while the 3D-aware decoder uses dimension of 256 for efficient design. We use frames with 224p as input
7

---

Method
R5@0..3
RL@0.5
R50005
Method
R1
R58003
R100
R5800.5
Eg [36]
6.32
13.84
3.41
8.80
EgoVLP [36]
30.44
46.06
22.41
35.75
LaviLa [87]
7.12
14.82
3.87
9.55
LaViLa [87]
32.9
48.68
24.12
37,59
AVION [86]
14.89
4.31
[86]
32.17
47.3
23.11
PM (ours)
8,13
16.11
4.83
10.30
FM (ouns)
32.92
50.08
23.94
39.15
Table 3: Comparisons of NLQ task.
Table 4: Comparisons of MQ task.
and 56p as output of the depth maps. Consequently, our 3D-aware decoder only has 9M parameters, and the batch size is set to 4096.
4.2 Main Results
Zero-shot video-text retrieval. In Table 1, EgoDTM outperforms models like LaViLa [87] and AVION [86]. For example, EK100-MIR, EgoDTM outperforms the state-of-the-ar AVION by +0.9% in mAP and +0.3% in nDCG, while also achieving a +1.5% improvement in intra accuracy on EgoMCQ. This indicates that captions generated based on visual cues from our data curation pipeline are more accurate and informative than captions generated from visual-conditions LLMs or simply text rewriting. Besides, our model surpasses HelpingHands [80] and HENASY [47], which rely on noisy HOI detection supervision with extra parameters in the visual encoder, indicating that depth modality can potentially offer larger merits for video-language models.
Zero-shot action recognition Table illustrates the superior performance of EgoDTM on EGTEA. While previous works have demonstrated that depth modality enhances traditional action recognition tasks [82, 38], their applicability has been largely constrained to specific scenarios and closed-set datasets. Our results extend this understanding, showing that video-language models can effectively leverage depth modality to learn more generalizable video representations.
Depth estimation. We evaluate the ability of visual-language models to infer depth from images in Table 2. Our model surpasses previous video-language models across most metrics by a large margin. In particular, our method improves the scale-aware RMSE by approximately 9.8% over the second-bes best result achieved by LaViLa. Interestingly, Ego performs better in scale-invariant òi metric, suggesting that it latently captures finer- finer-grained pixel-level details. We hypothesize that this advantage stems from EgoVLP's scene-aware negative sampling strategy, which samples video-text pairs captured from the same environments into the same batch, thereby learning relevant details.
Robot manipulation. As shown in Table 5, EgoDTM consistently outperforms pretrained visual-language models such as CLIP [52] and
Mathod
TK
0D
GM
FS
SD
Averrge
RIM [45)
5% 3.6
50.75
50.35 30.3%%
86. 35
97.76
69.45
LaViLa [87] by +20.2% and +6.6%, respec- tively, demonstrating stronger spatial percep- tion in visual representations. Additionally, EgoDTM performs competitively with special- ized robot learning methods on certain tasks, such as "turn knob" but underperforms on oth- ers, like "open microwave". A possible reason is that our model, pretained on depth and text, may overfit to real-worold scenarios, whereas methods like R3M [45] leverage self-supervised pretraining, which could provide better generaliza- tion diverse manipulation tasks.
MPI (26)
83.3%
34%
44.5%
93.5%
100%
75%
ResNet [24]
25%
18%
26.7%
5016
75.5%
39.7%
CLIP [52]
26 358
13%
24.7%
41.7%
86.39%
38.4%
LaVila (87)
48%
265
22.5%
69%
94.5%
52%
EP (ours)
56%
28%
35.5%
81%
92.5%
58.6%
Table 5: Comparison of robot manipulation tasks in Franka Kitchen simulation to assess model's 3D-awareness.
Natural language queries. As shown in Table 3, EgoDTM outperforms other egocentric video- language models, e.g., +1.22 R5@0.3 over AVION. In long video localization tasks, depth aware- ness enhances spatial understanding, enabling EgoDTM to better comprehend visual content.
Moment queries. As shown in Table 4, EgoDTM achieves competitive performance among all methods. Specifically, surpasses AVION by +2.78 on R5@0.3 and +2.85 on R500.5, and slightly improves over La by +1.4 on R5@0.3 and +1 +1.56 on R5@0.5. These results demonstrate that depth-aware representation learning in EgoDTM effectively enhances spatial reasoning and temporal localization in long egocentric videos, leading to more accurate moment predictions.
8

---

MAP
EXTOOMIR
#DCG
EgaMCQ
eDCG
Inter
EgaMCO
EXTOOMIR
Inter
EgoMCC
Lyte
Loupst Cayes
fere Casph
Gaggse
33.1
31.3
31.3
33.1
31.2
32.3
94.6
94.0
02.6
61.6
16
32.6
33.1
32.3
32.5
33.1
32.4
94.72
94.61
91.6
6261
02.46
50.0
94.28
94.37
94.54
(a) EgoDTMMs components. Both the 3D-aware video pretraining and textual enhancement enhance video-language
(b) Depth query numbers. We
(c) Resolution of depth. The
set 8 queries the default, which empirically generalizes hest on EK EK000MIR.
batch size is set to 512 to accom- modate the high GPU memory demands of large depth maps.
representations.
EKI00MIR
EgaMCQ
EKI00MIR
EgoMCQ Inrra
EKI0MMIR
EgoMCQ
mAP
nDCG
Inter
Intra
mAP
aDCG
Inter
mAP
nDCG
Inter
Intra
Large
Base
33.1
36.4
33.1
34.2
95.13
94.6
66.04
62.64
Rand
32,8
33.1
33.1
32.7
94.46
94.6
62.64
61.92
2048
1024
32.4
33.1
32.5
33.1
94.55
94.6
62.64
61.48
94.54
(d) Model size. Large model
Text Sampling Strategy. Random substitution is better than thorough replacement.
(c)
(f) Batch Size. Increasing the batch size steadily yields larger improvements.
brings more performance gain intra accuracy of EgoMCQ.
Table 6: Zero-shot ablation studies. Models are pretrained on 4M video-text pairs from Ego4D and evaluated with zero-shot video-text retrieval on Kitchens--00--- Unless specified. the default setting includes: ViT-B/16 backbone, batch size of 2048, random sub- stitution of enriched texts, both Leepth and Laugrte as losses, 8 depth queries, and 56p depth maps.
[11] and EgoMCQ [36].
EKI00MIR mAP/ #DCG
EgoMCQ inter/ / intra acc
EKIOOCLS
EgaNLQ
EgoMQ mAP f
DE
Metrics
top-1 / 10p-5 acc
mloU 1
scale-aware RMSE/ scale-inaarinn RMSE
Ex:
29.7/30.7
94.2 60.2
12.847/ 22.077 30.037
6.14
6.97
0.572/0.495
31.2/31.2
94.2/ 62.6
15.412 / 32 995
5.98
7.52
0.5637 0.464
31.2
94/61.6
16.508 32.851
6.53
6.14
0.550/ 0.489
Expe Capphi
33.1/33.1
94.6 62.6
15,898 / 33.895
6.17
8.87
0.339/ D.481
Table 7: Ablation studies on downstream tasks.
4.3 Ablation Study
EgoDTM's components. Combining video-text matching with depth estimation enhances each learning objective. EgoDTM leverages depth maps to capture object relations, while textual de- scriptions enriched with shape and movement details can benefit more from depth information. As shown in the table Table 7, progressively adding our modules leads consistent improvements across the majority of tasks, and our model outperforms all baselines in the main experiments. One potential limitation is that depth pretraining may negatively impact the performance on EgoNL to some extent. Nevertheless, our proposed AugVTC module is able to mitigate this effect and yields improvements that surpass those achieved by Avion, ViLa, and the EgP encoder shown in Table4 of our main paper. Besides, comparing row 2 and row 4 when adding Langvre. we observe an increase in the scale-invariant RMSE and a decrease in the scale-aware RMSE. We hypothesize that this may result from the proxy task bias introduced by multi-task pretraining.
Depth query numbers. The number of depth queries affects depth granularity. We find that setting the query number to is optimal, with each query covering a moderate depth range within [0, 1].
Method
(0005:0.95)
mAP
map #0.50
(5005:00.55)
(.0.5:0..55)
abjecs-AP
Upperhound [13]
60.70
73.31
50.87
30.53
Our Pipeline
43.02
$4.58
61.01
25.03
Resolution of the predicted depth map. The reso- lution of depth maps greatly impacts GPU memory usage. We find that higher-resolution depth maps slightly improve the video-text alignment, but this comes at the cost of reducing the maximum batch size. Therefore, we choose a resolution to 56p as a balanced trade-off between high resolution and the ability to maintain large batch size.
Table 8: Evaluation of our HOI mask gener- ation pipeline on HOI segmentation bench- mark VISOR [13] in kitchen environment.
Model size. larger model size enhances performance, especially on the EgoMCQ intra task, which requires selecting the correct video from visually similar alternatives.

---

Original Video
Inconsistent HOT Boxes
detggnall Coprbon Coljanss weighing plite
Sparid Aware Capiiens: Ic aajuuss soall mug seighinng
fowrroonn whill right no neaire contait with the soutinnury oljacc
Temporaa Consistent Macks
Spatial Awwrar--Awwreeeee
Figure 4: Comparisons of the noisy HOI bound- ing boxes (left) and the spatial-temporal consis- tent HOI masks (right).
Figure 5: Example of generalizable data con- struction. For better visualizations, we blur the background to highlight HOI regions.
Text sampling strategy. We examine the impact of text sampling strategy on model performance. A mixed sampling strategy that includes both enriched and original texts for pretraining yields stronger results by leveraging both detailed and simplified textual information.
Batch size. The experiment results show that increasing batch size steadily improves performance.
4.4 More Analysis
Quantitative analyses of the HOI mask generation pipeline. To evaluate the reliability of the HOI detector and SAM2 model in the egocentric domain, particularly for hand-object interactions, we evaluate the image-based HOI segmentation on the VISOR dataset [13]. The results are presented in Table 8. The supervised training model on VISOR serves the upper bound for performance comparison. While our generation pipeline does not fully reach the upper bound, demonstrates high-quality segmentation results. Notably, hand segmentation is significantly lower than the upper bound, likely due to inaccuracies in HOI detection prompts and challenges segmenting hand- object interactions with SAM2 in cluttered egocentric backgrounds.
Qualitative analyses of the HOI mask generation pipeline. We firstly compare inconsistent HOI bounding boxes from HOID [63] (frame-by-fraate detection) with spatial-tempor HOI masks gen- erated by combining HOID [63] and SAM2 [59] in Figure 4. The HOI boxes detected by HOI detector often lose tracks, since they are detected by an image-based model. Our pipeline achieves consistent HOI tracking across frames, offering more precise HOI labels. Figure presents that our data construction pipeline is useful even when HOI regions are small and the background is noisy.
5 Conclusion
In this work, we present EgoDTM, novel egocentric 3D-aware video-language model. EgoDTM integrates dual transformer encoders with lightweight 3D-aware depth decoder, trained using video-text contrastive learning and depth estimation objectives. To enable large-scale pretraining, generate millions of depth maps and spatially enriched captions by leveraging foundation mod- els. The captions are enhanced through a detect-track -eneertt pipeline specifically tailored for egocentric videos. EgoDTTM demonstrates intuitive generalization in estimating depths in unseen environments. Extensive experiments across diverse benchmarks, spanning short video understand- ing, 3D understanding, and long video understanding, validate the effectiveness of our approach.
Discussions. While EgoDTM demonstrates strong performance in egocentric hand-object interac- tion scenarios, its generalization to broader indoor scenarios remains limited. Further exploration may include integrating 3D-aware visual encoders into multimodal large language models to en- hance spatial awareness. Moreover, pretraining large-scale spatial-aware egocentric models with richer 3D signals, as explored in VGGT [71], remains a promising yet challenging direction.
10

---

Acknowledgments
This work was partially supported by the Beijing Natural Science Foundation (No. L.230088) and the Outstanding Innovative Talents Cultivation Funded Programs 2024 of Renmin University of China.
References
[1] Kumar Ashutosh, Rohit Girdhar, Lorenzo Torresani, and Kristen Grauman. Hiervl: Learning hicrarchi- video-language embeddings. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23066-23078, 2023.
[2] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1728 1738, 2021.
[3] Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Adabins: Depth estimation using adaptive bins. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1009-4018, 2021. 2,4
[4] Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Localbins: Improving depth estimation by learning local distributions. In European Conference on Computer Vision, pages 480-4940999 19000-99..00 Springer, 2022.
2, 4
[5] Alexcy Bochkovskiy, Amaël Delaunoy, Hugo Germain, Mareel Santos, Yichao Zhou, Stephan Richter, and Vladlen Koltun. Depth pro: Sharp monocular metric depth in less than second. In The Thirteenth International Conference on Learning Representations, 2025. 2
[6] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large- scale video benchmark for human activity understanding. In Proceedings of the ieee conference on com- puter vision and pattern recognition, pages 961-970, 2015. 25
[7] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.
[8] Dibyadip Chatterjee, Fadime Sener. Shugao Ma. and Angela Yao. Opening the vocabulary egocentric actions. Advances in Neural Information Processing Systems, 36, 2024. 2
[9] Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spa- Galvlm: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14455-14455, 2024.
[10] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1290-1299 2022.
[11] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kaza- kos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. The epic- shic-kitchens dataset: Collection, challenges and baselines. IEEE Transactions Pattern Analysis and Machine Intel- ligence (TPAMI), 43(1 1311554115441 2021. 1, 5, 7,9,25 25
[12] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory- efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 3.4
[13] Ahmad Darkhalil, Dandan Shan, Bin Zhu, Jian Ma, Amlan Kar, Richard Higgins, Sanja Fidler, David Fouhey, and Dima Damen. Epic-kitchens visor benchmark: Video segmentations and object relations. Advances in Neural Information Processing Systems, 35:13745 -37558, 2022. 9, 10
[14] DeepSeek-AL Deepseek I1m: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.0295 2024. 5, 6,
[15] Jacob Devlin, Ming Wei Chang, Kenton Lee, Kristina Toutanova. Bert: Pre-training deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pages 4171-4186, 2019.

---

[16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. 3, 4
[17] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. Advances in neural information processing systems, 27, 2014. 7
[18] Mohamed El Banani, Amit Raj, Kevis-Kokitsi Maninis, Abhishek Kar, Yuanzhen Li, Michael Rubin- stein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. Probing the 3d awareness of visual foundation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21795 21806, 2024, 2, 2,3,7 25
[19] Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael Black, and Otmar Hilliges. Arctic: A dataset for dexterous himamual hand-ohject manipulation. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 12943-12954, 2023.
[20] Weixi Feng, Wanrong Zhu, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Xuchai He, S Basu, Xin Eric Wang, and William Yang Wang. LayoutGPT: Compositional visual planning and generation with large language models. seirty-seventh Conference on Neural Information Processing Systems, 2023, 6
[21] Rohit Girdhar, Alaacddin Fll-Nobby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One emhedding space to bind them all. In CVPR, 2023. 26
[22] Kristen Grauman, Andrew Westbury, Bugene Byme, Zachary Chavis, Antomino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 19012, 2022. 1, 2, 5,
[23] Abhishek Gupta, Vikash Kumar, Corey I. ynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long horizon tasks via imitation and reinforcement learning. Conference on Robot Learning (CoRL), 2019. 7
[24] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[25] Yafei Hu, Quanting Xie, Vidhi Jain, Jonathan Francis, Jay Patrikar, Nikhil Keetha, Seungehan Kim, Yaqi Xie, Tianyi Zhang, Hao-Shu Fang, et al. Toward general-purpose robots via foundation models: A survey and meta-analysis. arXiv preprint arXiv:2312.08755 2023. 1,3
[26] Zeng Jia, Bu Qingwen, Wang Bangjun, Xia Wenke, Chen Li, Dong Hao, Song Haoming, Wang Dong, Hu Di, Luo Ping, Cui Heming, Zhao Bin, Li Xuelong, Qiao Yu, and Li Hongyang. Learning manipulation by predicting interaction. Proceedings of Robotics: Science and Systems (RSS), 2024. 7, 2,
[27] Evangelos Kazakos, Jaesung Huh, Arsha Nagrani, Andrew Zisserman, and Dima Damen. With little help from my temporal context: Multimodal egocentric action recognition. In British Machine Vision Conference (BMVC), 2021. 25
[28] Jan J Koenderink and Andrea Van Doorn. Surface shape and curvature scales. Image and vision computing, 10(8):557-566 564, 1992. 1,3
[29] Taein Kwon, Bugra Tekin, Jan Stühmer, Federica Bogo, and Mare Pollefeys. H22: Two hands manip- ulating objects for first person interaction recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10138-10148, October 2021. 2, 7.
[30] Feng Li, Hao Zhang, Huaizhe Xu, Shilong Liu, Lei Zhang, Lionel M Ni, and Heung-Yeur Yeung Shum. Mask dino: Towards a unified transformer- based framework for object detection and segmentation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3041-3050, 2023.
[31] Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Sealing language- image pre-training via masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23400, 2023.
[32] Yanghao Li, Hanzi Mao. Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In European conference on computer vision, pages 280 280-296. Springer, 2022. 4
12

---

[33] Yanghao Li, Tushar Nagarajan, Bo Xiong, and Kristen Grauman. Ego-exo: Transferring visual rep- resentations from third-person first-person videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6943-6953, 2021. 2
[34] Yin Li, Miao Liu, and James M Rehg. In the eye of beholder: Joint learning of gaze and actions in first person video. In Proceedings of the European conference on computer vision (ECCV), pages 619-635, 2018. 1, 1,7,2 25
[35] Zhenyu Li, Xuyang Wang. Xianming Liu, and Junjun Jiang. Binsformer: Revisiting adaptive bins for monocular depth estimation. IEEE Transactions on Image Processing, 2024. 2.4
[36] Kevin Qinghong Lin, Jinpeng Wang, Mattia Soldan, Michacl Wray, Rui Yan, Eric Z Xu, Difei Gao, Rong- Cheng Tu, Wenzhe Zhao, Weijie Kong, et al. Egocentric jideo-language pretraining. Advances Neural Information Processing Systems, 35:7575 7586,, 2022. 1, 2, 6, 7, 8, 9, 25
[37] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 16:34892-3491 2023.
[38] Jun Liu, Amir Shahroudy. Mauricio Perez, Gang Wang. Ling- Yu Duan, and Alex C Kot. Ntu rgb+ d 120: A large-scale benchmark for 3d human activity understanding. IEEE transactions on pattern analysis and machine intelligence, (1110))6684-77011 2019.
[39] Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. Hoi4d: A 4d egocentric dataset for category-level human-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21013- 21022, 2022. 1.2
[40] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976 1997--19966, 11986, 2022.
[41] Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. ClipAclip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing, 508:293-304, 2022. 2
[42] Yunze Man. Shuhong Zheng, Zhipeng Bao, Martial Hebert, Liang- Yan Gui, and Yu- Xiong Wang, Lex- icon3d: Prohing visual foundation models for complex 3d scene understanding. In Advances in Neural Information Processing Systems, 2024. 2
[43] Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances Neural Information Processing Systems, 36, 2024.
[44] Robert McCarthy, Daniel CH Tan, Dominik Schmidt, Fernando Acero, Nathan Herr, Yilun Du, Thomas G Thuruthel, and Zhibin Li. Towards generalist robot learning from internet video: A survey, arXiv preprint arXiv:2404. 19664, 2024, 3
[45] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. In 6th Annual Conference on Robot Learning, 2022. 7,8
[46] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199, 2023. 24
[47] Khoa Vo Thinh Phan, Kashu Yamazaki Minh Tran, and Ngan Le. Henasy: Learning assemble scene- entities for interpretable egocentric video-language model. In Advances in Neural Information Processing Systems, 2024. 2, 3, 6,
[48] Luigi Piccinelli, Yung- -ssu Yang. Christos Sakaridis, Mattia Segu. Siyuan Li, Luc Van Gool, and Fisher Yu. Unidepth: Universal monocular metric depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10106-10116, 2024. 4
[49] Chiara Plizzari, Gabriele Goletto, Antonino Furnari, Siddhant Bansal, Francesco Ragusa, Giovanni Maria Farinella, Dima Damen, and Tatiana Tommasi. An outlook into the future of egacentric vision. Interna- tional Journal of Computer Vision, pages 1 1-57, 2024. 1,3
[50] Chiara Plizzari. Toby Perrett, Barbara Caputo, and Dima Damen. What can a cook in italy teach a mechanic in india? action recognition generalisation over scenarios and locations. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 13656-13666 2023. 2
13

---

[51] Shraman Pramanick, Yale Song. Sayan Nag. Kevin Qinghong Lin, Hardik Shah, Mike Zheng Shou, Rama Chellappa, and Pengchuan Zhang. Egovipy2: Egocentric video-language pre-training with fusiin in the backbone. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5285- 5297, 2023. 1, 2, 6
[52] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. 2021. 2, 3, 4, 7.8
[53] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8)9, 2019. 2
[54] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text to-text trans- former. Journal of machine learning research, 21(140):1 67, 2020. 2
[55] Francesco Ragusa, Antonino Furnari, Salvatore Livatino, and Giovanni Maria Farinella. The meccano dataset: Understanding human-object interactions from egocentric videos in an industrial-like domain. In IEEE Winter Conference on Application of Computer Vision (WACV), 2021. 2,5
[56] Santhosh Kumar Ramakrishnan, Erik Wijmans, Philipp Krachenbuchl, and Vladien Koltun. Does spatial cognition emerge in frontier models?, 2024. 2
[57] René Ranftl, Katrin Lasinger. David Hafner, Konrad Schindler. and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions on pattern analysis and machine intelligence, 44(3):1623 1637, 2020. 4.5
[58] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). 2020. 24
[59] Nikhila Ravi, Valentin Gabeur, Maan-Ting&& Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Al- wala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollar, and Christoph Feichtenhofer. SAM 2: Segment anything in images and videos. In The Thirteenth International Conference on Learning Representations, 2025. 2, 5, 6, 10, 24
[60] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object de- tection with region proposal networks. IEEE transactions on pattern analysis and machine intelligence, 39(6): (366)1117-1114 1149, 2016. 24
[61] Fiona Ryan, Hao Jiang, Abhinav Shukla, James M. Rehg. and Vamsi Krishna Ithapu. Egocentric auditory attention localization in conversations. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14663-14674, 2023. 24
[62] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with sub- word units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany, August 2016. Association for Computa- tional Linguistics.
[63] Dandan Shan, Jiaqi Geng, Michelle Shu, and David Fouhey. Understanding human hands in contact at internet scale. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 9869-9878, 2020. 2, 5, 6, 10, 24
[64] Shuwei Shao, Zhongcai Pei, Xingming Wu, Zhong Liu, Weihai Chen, and Zhengguo Li. lebins: Itera- tive clastic bins for monocular depth estimation. Advances in Neural Information Processing Systems, 36:53025 53037, 2023. 2,
[65] Roger N Shepard and Jacqueline Metzler. Mental rotation of three-dimensional objects. Science,
17133972)701-703 1971. 1,3
[66] Gunnar Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteck Alahari. Actor and observer: Joint modeling of first and third-person vidcos. In proceedings of the IEFE conference on computer vision and pattern recognition, pages 7396-7404, 2018. 1
[67] Gunnar A. Sigurdsson, Abhinav Kumar Gupta, Cordelia Schmid, Ali Farhadi, and Alahari Karteek. Actor and observer: Joint modeling of first and third-person videos. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7396-7404, 2018.
14

---

[68] Elizabeth Spelke, Sang Ah Lee, and Véronique Izard. Beyond core knowledge: Natural geometry. Cog- mitive science, 3465)863-8884 2010. 1,3
[69] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llss. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 9568-9578, 2024. 3
[70] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 3
[71] Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In Proceedings of the Computer Vision and Puttern Recog- nition Conference, pages 5294-5306, 2025. 10
[72] Boshen Xu, Ziheng Wang, Yang Du, Zhinan Song, Sipeng Zheng, and Qin Jin. Do egocentric video- language models truly understand hand-object interactions? In The Thirteenth International Conference on Learning Representations, 2025.
[73] Boshen Xu, Sipeng Zheng, and Qin Jin. Pov: Prompt-oriented view-agnostic learning for egocentric hand- object interaction in the multi-view world. In Proceedings of the 31 ACM International Conference on Multimedia, pages 2807-2816 16, 2023. 2
[74] Yue Xu, Yong-Lu Li. Zhemin Huang. Michael Xu Liu, Cewu Lu, Yu- Ku-Wing Tai, and Chi-Keung Tang. Egopca: A new framework for egocentric hand-object interaction understanding. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 5273-5284, 2023. 2
[75] Zihui Sherry Xue and Kristen Grauman. Learning fine-grained view-invariant representations from un- paired ego-eXo videos via temporal alignment. Advances in Neural Information Processing Systems, 36:53688 53710, 2023.
[76] Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth any- thing: Unleashing the power of large-scale unlabeled data. In CVPR, 2024. 2.5
[77] Lihe Yang, Bingyi Kang. Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao, Depth anything v2. In Advances in Neural Information Processing Systems (NeurlPS), 2024. 2, 3, 5, 6, 24
[78] Yuanwen Yue, Anurag Das, Francis Engelmann, Siyu Tang, and Jan Eric Lenssen. Improving 2D Feature Representations by 3D-Aware Fine-Tuuing. In European Conference on Computer Vision (ECCV), 2024.
2.3
[79] Xinyu Zhan, Lixin Yang, Yifei Zhao, Kangrui Mao, Hanlin Xu, Zenan Lin, Kailin Li, and Cewu Lu. Oakink2: A dataset of bimanual hands-object manipulation in complex task completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 445-456, 2024.
[80] Chuhan Zhang, Ankush Gupta, and Andrew Zisserman. Helping hands: An object-aware ego-centric video recognition model. In Proceedings of the IEEFJCVF International Conference on Computer Vision, pages 13901-13912, 2023. 2, 3, 6,
[81] Hao Zhang, Aixin Sun, Wei Jing, and Jocy Tianyi Zhou. Span- localizing network for natural language video localization. arXiv preprint arXiv:2004. 13931, 2020." 7
[82] Jing Zhang, Wanqing Li, Philip O Ogunbona, Pichao Wang, and Chang Tang. Rgh-d-based action recog- nition datasets: A survey. Pattern Recognition, 60:86-105, 2016. 8
[83] Lingzhi Zhang, Shenghao Zhou, Simon Stent, and Jianbo Shi. Fine-grained egocentric hand-object seg- mentation: Dataset, model, and applications. In European Conference Computer Vision, pages 127- 145. Springer, 2022. 2
[84] Xiaoshuai Zhang, Zhicheng Wang. Howard Zhou, Soham Ghosh, Danushen Gnanapragasam, Varun Jam- pani, Hao Su, and Leonidas Guibas. Condense: Consistent 2d/3d pre-training for dense and sparse fea- tures from multi-view images. In European Conference on Computer Vision, pages 19-38. Springer, 2024. 2
[85] Chen Zhao, Ali K. Thabet, and Bernard Ghanem. Video self-stitching graph network for temporal ac- tion localization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13658- 13667,166677000 2021.
15

---

[86] Yue Zhao and Philipp Krähenbühl. Training large video model on single machine in day. arXiv preprini arXiv:2309. 16669, 2023. 6, 25
[87] Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar. Learning video representations from large language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5586-6597, 2023. 1, 2. 24. 25
[88] Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, WANG HongFa, Yatian Pang, Wenhao Jiang. Junwu Zhang, Zongwei Li, Cai Wan Zhang. Zhifeng Wei Liu, Yuan. Languagebind: Extending video- language pretraining to n-modality by language-basee semantic alignment. In The Twelfth International Conference on Learning Representations, 2024. 26
16

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency. research ethics, and societal impact. Do not re- move the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] [NA]-
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatiin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evalu- ation. While "[Yes] " generally preferable "[No] ", perfectly acceptable answer "[No] provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable find the license for the dataset general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased binary way. acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
used"). In
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Check- list",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: Abstract and introduction explains contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yss]
17

---

Justification:
Limitations are discussed in Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and faireess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof'
Question: For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[NA]
Justification:
No theory is included in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Ys]
Justification:
We explain all details in experimental setups.
Guidelines:
18

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language modell, releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recogmize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide open access to the data and code, with sufficient instructions to reproduce our main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
seriides/CodSSubiisse
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.c//ppll gttic//ggge///eeeeeoeooocc httgg//idd///deuuuaaaeeeee CooeeSbmieiieeeicceeee ssi seisssinoooooo onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
19

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide all details in experimental setups and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
Justification:
To ensure fairness, all settings are kept the same.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK report I-sigma error bars, but should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We explain the computational resources in experimental setups.
Guidelines:
The answer NA means that the paper does not include experiments.
20

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g.. preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics tttp:://eaarrpp..cppllll/// cecciihscccellleeee ines??
Answer: [Yes]
Justification:
Our research adheres to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our research aims to promote the development of related fields. We are unable predict the impact this research will have society.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [No]
21

---

Justification: We employed open-source models and source datasets. They are rea- sonably safe.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators or original owners of assets used in the paper are credited, and the license and terms of use are mentioned.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswithcode thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer:
[Yes]
Justification:
The documentation will be provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
22

---

Question: For crowdsoureing experiments and research with human subjects, does the pa- per include the full text of instructions given participants and screenshots, applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects. and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: httpas////nurdtte //neunrrps... cc//oonferencees//22///1. for what should or should not be described.
23

---

We provide additional details, extended experimental results, and further discussion in this supple- mentary material, including:
background of foundation models (Appendix A. dataset details (Ap- pendix A.2), and experimental details (Appendix A.3).
Implementation details:
Additional quantitative results: analyses and evaluations (Appendix B).
Additional qualitative results: Examples and visualizations to complement the main results (Ap-
pendix C).
Discussion on related works: Insights and comparisons with prior research (Appendix D).
A
Implementation Details
A.1 Background of Foundation Models
HOI Detector [63]. HOID a robust system for detecting human hands and their interacting objects in images. It is built upon Faster-RCNN [60], pretrained on 100K image dataset with hand-object interaction annotations, including hand bounding box, interacting object bounding box, hand side (left or right), hand contact state (c.g., no contact, self-contact, other person contact. contact with portable object, or contact with non-portable object). To enhance its capabilities, HOID is further trained with an additional 42K egocentric data samples, enabling improved understanding of HOI from egocentric view. We leverage HOID to generate spatial HOI boxes, hand side, hand contact state for each video, from 12 uniformly sampled frames per video. However, HOID often produces temporally inconsistent box predictions across adjacent frames. To address this, we apply a robust image and video segmentation model to refine the detection results, ensuring greater consistency and accuracy.
Segment Anything 2 [59]. SAM2 is a versatile segmentation model capable of segmenting ob- jects in both images and videos according to given prompt. such as a point, box or mask, with remarkable efficiency. It is trained on a large-scale SA-V dataset, comprising 50.9K videos and 35.5M high-quality masks. SAM2 employs a hierarchical image encoder and a memory mechanism to handle streaming frame input. In our approach, SAM2 is utilized to generate the spatial-temporal consistent HOI masks by leveraging prompts derived from HOID outputs.
Depth Anything 2 [77]. DAv2 excels in monocular depth estimation, offering fine-grained details, strong generalization and efficient inference. is built upon the pretrained visual foundation model DINOv2 [46] and depth decoder DPT [58]. Pretrained on 595K synthetic images and 62M pseudo- labeled real images, DAv2 exhibits strong out-of-domain generalization. In our work, we use DAv2 to generate depth maps for eight frames sampled from egocentric videos, serving as supervision sig- nals. Following the recommendations of DAv2, we employ the DAv2-Large variant, which produces more spatialllemporal consistent depth maps.
DeepSeek-LLM [14]. We employ the LLM to interpret HOI information and enrich textual descrip- tions with shape and movement details. DeepSeek-LLM demonstrates exceptional ability to follow instructions and comprehend HOI mask prompts. Specifically, we use the DeepSeek API to access their 2000-0000aa -aramerrre LLM to facilitate these tasks.
A.2 Dataset Details
Ego4D [22]. Ego4D contains 3,670 hours of egocentric videos with dense narrations, covering diverse scenarios and activities from worldwide. Each narration is timestamped and paired with free-form sentence. Following the approach in [87], we construct 4M video-text clip pairs for pre- training, with an average clip length of 1 second (+0.9). In our text enrichment process, we only keep those hand-object interaction clips performed by the camera wearer, where the text begins with '#C' (denoting the wearer) and then follows HOI-related verbs and nouns. This strategy excludes clips that record other people's activities, such as multi-person interactions [61] where the text begins with '#O', and the videos like '#C C walks away'. For the natural language query task, comprises 1,659 untrimmed videos, each averaging 500 seconds in duration. On average, each video contains 12 clip-query pairs. Following the official split from [22], we use 11, 11,291 queries for training and 3,874 for validation.
24

---

Epic-Kitchens [11]. Spic-Kitchensee 100 (EK-100) consists of 100 hours of egocentric cooking videos divided into training (67,217 clips), validation (9.668 clips), and testing (13,092 clips) splits. Each clip includes start and end timestamps, short textual narration, and a verb and noun class that correspond to the narration. There are 3805 action classes, 97 verb classes, and 300 noun classes. We evaluate our pre-trained model on the validation split.
EGTEA [34]. EGTEA comprises 28 hours of egocentric cooking videos, annotated with 10,321 instances of fine-grained actions across 106 classes. The average action duration is 3.2 seconds. For our experiments, we use only the visual frames as input. We follow prior works [27, 87] and report top-1 accuracy and mean class accuracy on all three test splits, including 2,022 testing instances for each split.
H20 [29]. H2O is a dataset capturing egocentric hand-object interactions in a laboratory, including 36 action classes. The egocentric data is captured from an Azure Kinect camera mounted egocentri- cally for recordings. Since our primary target is to evaluate the transfer learning capability of visual representation, the train/val splits have 7862/11633 frames.
A.3 Experimental Details
Zero-Shot Video-Text Retrieval in EK100MIR and EgoMCQ. We perform video-text matching with 16 frames as input for IOOMIR and 4 frames for EgoMCQ, following [86].
Zero-Shot Action Recognition in EGTEA, We follow the evaluation protocol proposed by [34]
to compute the mean performance across all evaluation splits. This involves performing video-text retrieval between video clips and the action text labels, which are prompted by prepending "#C C
... During inference, we apply three spatial crops of size 224 x 224 from each 256 x 256 frame of 10 video clip, averaging predictions across these crops to produce the final results.
Depth Estimation in H2O. The frozen visual encoder produces feature maps of dimension 768, which are passed to a linear decoder to estimate depths with resolution of 720 x 720. The model is trained for 10 epochs with batch size of 64 and learning rate of 0.0005, where the first 1.5 epochs serve as a warm-up phase. Our evaluation code is built upon Probing3D [18].
Robot Manipulation in Franka Kitchen. In Franka Kitchen environment, all baselines apply imi- tation learning for visuomotor control. A policy network is trained for each task using observations from the environment and video representation from EgoDTM. To adapt EgoDTM and LaViLa, repeat the image observation times as video input. For each task, the experiment conducted from two different camera viewpoints for two random seeds using 50 randomly sampled trajectories. The final result is the average of the success rate. Our codebase is built upon MPI [26].
Natural Language Queries Ego4D. The task typically operates on a 6-minute video. Using the video compression technique from [6], we compress the original video frames by times to save storage. Then we extract the features with the fps of 1.87 and sampling frame number 4. We takes 256 dimension global video features and 768 dimension BERT features as input. Our codebase is built upon VLP [36].
B Additional Quantitative Results
Predicted depth contains valuable informa- tion for action recognition. Since ground- truth data is unavailable for directly evaluating the depth decoder, we demonstrate the utility of our predicted depth maps for multimodal ac- tion recognition, shown in ??. We simply encode the depth map using an MLP at 56p resolution. The action recognition accuracy im- proved by +1,4%, confirming that our predicted depths contain meaningful information for un- seen egocentric data.
25

---

Task: Turn Knob
EgoDTM
LaViLa x
Figure 6: Qualitative results of robot manipulations.
C Additional Qualitative Results
LLM Prompts. We the LLM prompts in Figure 7 to enrich the texts with HOI shape and movement information. Specifically, the shape information provided by the HOI mask areas, where the large, medium, small object occupies [0.1.1], [0.01, 0.1] and [0, 0.01] areas, respectively.
Generated Data. Examples of generated HOI boxes, HOI masks, depths, and enriched texts are illustrated in Figure 8.
Case Study on Robot Manipulation. In Figure the learned policy based on EgoDTM visual representation enables the robot to approach the switch and turn it, while LaViLa successfully ap- proaches but misses the switch.
D Discussions
Comparison with Related Works that Pretrained with Depth. ImageBind [21] and Language- Bind [88] are the most relevant depth-based lasion-language pretraining works. These methods em- ploy multiple encoders to align various modalities within unified feature space through contrastive learning. While both methods utilize depth for pretraining, their application may be less impactful when applied conventional third-person datasets. In contrast, depth is essential for egocentric perception, where spatial awareness is critical for understanding human indoor activities. Further- more, their pretraining processes treat depth depth features that lack pixel-level 3D information and video representations that remain unaware of 3D structure. In work, aim to enable video representations predict depth maps, thereby emhedding 3D awareness directly into the representations.
input rather than prediction target, resulting in
Potential for Real-World Applications. While model demonstrates improvements over text- pretrained models both video understanding and robotic manipulation tasks, it falls short state- of-the-art performance of manipulation models. However, our model predicts more meaningful depth maps in real-world settings than in simulations, offering promising potential for real-world deployment.
26

---

Egocentric Video with HOI Masks
System Prompt
Background
1. The user will provide information about a short egocentric video (12 frames] captured by one penson using VR/AR, approximately one second with continunus frame box annotations. The annotations represent the center point size of objects using the format:
<length x, width area contact state (optional)>, where values range from [0, inclcating percentages of the length or width, The
the product of length and width. The annotations may include to four elements: left hand, the right hand, object related the left hand, and objects being manipulated by the hands.
2. About the directions, smallerx means more left, largerx x means more right, smaller means more higher, larger moans more lower. object large object;
the area larger than
area larger
but smaller 0.1, the object
medium sice; if the area is smeller than 0.01, then the object is small object.
hands), with portable object (egg, an apple), or contact with stationary object (n-g,, fumntuure.. types left/rghtt
five possible
self-contact (between
person, contact
not be guaranteed: Typically, there exists hands, the hand with lower x the left hand, the hand with larger is the right hand. Another you should notice is that, If there are mootly left hand but exists few right hand data, you should at Response Requirements
Note
only
the hands,
the right hand data, vice
1. You should interpret the hand-object interaction (HOI) the video: Use the given text information to describe the interaction process, such relevant relations, human actions objects. Describe hand positions, movement directions, and speed, well the sizes and positionss of any objects.
Remember the information in original must contained your response.
3. Your resporse should be precise, fluent and unified netural language summary, restrictly using less than two sentences. I denote C your response with
4. Avoid using pronouns like 'their', his, 'hor'. Never mention 'In the video', just express what happens.
parentheses Le. (I, to explain your meaning. will provide you the above mentioned information. The informatio will keep empty if the video does not have that type of object.
Don't express the same thing twice.
Input Prompt
Given the HOI-related information below, respond me with a new sentence following the above requirements. chopping board knife.
Original text.
Possible left hand: [].
Possible right hand: [[0.59, 0.92, 0.01, 'portable object contact'), 10.96, 0.96, 'portable object contact' [0.98, 0.88, 0.0, 'oortable
object contact'], (0.59, 0.92, 0.01, 'portable oloject contact'), (. 90,96, 0.96, 0.0, 'portable object contact"), [0.98, 0.87, 0.0, pootabll object
contact"] [0.59, 0.93, 0.01, portable object contact"), (0.97, 0.95, 0.0, 'portable object contact)) [0.98, 0.86, 0.0, "portable object
contact'), [0.59, 0.93, 0.01,
portable object contact"), [0.97, 0.94, 0.0, 'portable object contact') [0.98, 0.84, 'portable object
contact']. [0.58, 0.93, 0.01,
portable object contact'). (0.98, 0.93, 0.0. 'portable object contact'), [0.se, 0.82, 0.0, 'portable object
contact']. [0.58, 0.01,
portable object contact'). [0.98, 0.92, 0.01. 'portable object contact], [0.98,
'portable object
contact'). [0.58, 0.93, 0.01,
portable object contact"), [0.98, 0.91, 0.01, 'portable object contact]] J. [0:98, 0.8, 0.0, 'portable object
contact'). [0.58,
portable object contact"), [0.98, 0.91,
'portable object contact
n.8, 'portable abject
contact']. [0.58, 0:93, 0.01,
portable object contact"], [0.98, 0.9, 0.01, 'portable object contact') [0:98, 0.78, 0.0,
'portable object
contact') [0.59, 0.00,
portable object contact"), [0.98, 0.89, 'portable object
'portable object
contact') [0.59, 0.93, 0.01,
portable object contact"], [0.98, 0.88, 0.01, "portable object contact [0.95, 0.77, o.0, 'portable object
contact'l [0.59, 0.01,
portable object contact"), [0.99, 0.88, 0.01, 'portable object contact [0.97, 0.76, 'portable objest
contact']]
Hand (not sure which hand]:
Object1: [[0.45, 0.77, 0.0), [0.46, D.O), (0.46, 0.78, 0.0], [0.46, 0.78, 0.0), Jo.46, 0.79, n.o), [0.47, 0.79, o.o), (0.48, D.8, n.0), [0.48, 0.8, 0.0], [0.49, 0.8, 0.0), (0.45, 0.8, 0.0%, (0.5, 0.8, 0.01, (0.51, 0.79, 0.0]]. Object2:
Your response:
LLM Response
#C C holds knife with the right hand, moving it downward to cut small cucumber on chopping board.
Figure 7: LLM prompt strategy for generating enriched text from HOI masks and the original text.
27

---

Original Text: puts some glue wooden piece from bottle of glue his right hand.
Enriched Text: ac appliess glue wooden piece bottle held the right hand, while the left hand remains stationary.
Original Text: #C picks the dough Enriched with small object
up dough stationary right
Original TextraC C adjusts weighing plate Enriched Cadjusts mug on weighing plate hand, slightl to right downward, the right hand maintains contact with the stationary object
Original Text: #C picks the bawl on the countertop Enriched Cuses right steadlly pick up countertop.
Original Text: #C C holds the egg crate with his right hand,
egg crate right hand, moving it slightly right downward.
Figure 8: Illustration of data generated by our data generation pipelines, including intermediate HOI boxes, masks, and the enriched texts and depth maps used as supervision signals. The text that includes HOI movements is marked blue, while the contents that include HOI spatial information marked red.
28


Reviews:
[{"review_id": "gcKawb2DWx", "reviewer": "Reviewer_kLWJ", "summary": "The research paper introduces EgoDTM, an Egocentric Depth- and Text-aware Model designed to address the lack of 3D understanding in existing video-language pretraining frameworks. Current models often rely on 1D text or 2D visual cues, but humans perceive a 3D world. To bridge this gap, EgoDTM jointly trains a model using both video-text contrastive learning and a novel 3D-aware video pretraining task. This task incorporates a lightweight 3D-aware decoder that efficiently learns to predict depth from pseudo-depth maps generated by powerful depth estimation models. To further enhance spatial reasoning, the system enriches brief, original captions with detailed hand-object interaction (HOI) information through a \"detect-track-generate\" pipeline that leverages a combination of foundation models. Extensive experiments show that EgoDTM achieves superior performance across diverse downstream tasks, including video-text retrieval, action recognition, and robot manipulation, highlighting its improved 3D-aware visual understanding.", "questions": "The questions are listed in the weaknesses section. If the author could solve the concerns, I would consider raising my score. Besides, I want to know whether the paper is not written in the full 9 pages.", "limitations": "The model and its entire pretraining data pipeline are highly specialized for egocentric video, with a strong emphasis on hand-object interactions (HOIs). While this leads to state-of-the-art performance in this domain, it raises concerns about the generalizability of the learned representations to third-person video or other video domains that do not feature prominent hands and close-up object manipulation. The model may be overfitting to the specific biases of the egocentric perspective, potentially limiting its utility as a general-purpose video representation model. The authors acknowledge this as a limitation, but its significance as a trade-off for specialization warrants emphasis.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1.The paper addresses a well-motivated and significant limitation in video-language understanding—the absence of 3D spatial reasoning. \nThe core idea of integrating a depth-prediction task as an auxiliary objective for pretraining is effective, as demonstrated by strong empirical results.\n\n2.The design of the lightweight 3D-aware decoder is a key strength. It resolves the tension between the high-resolution requirements of dense prediction tasks (like depth estimation) and the large-batch requirements of contrastive learning. \n\n3.The empirical evaluation is thorough and convincing. The authors test their model on a wide array of downstream tasks, including zero-shot retrieval, action recognition, depth estimation, and temporal localization. Consistent and significant performance gains over strong baselines across all these tasks strongly support the paper's claims.\n\n4.The paper is well-written, clearly structured, and easy to follow. Figure 2, in particular, provides an excellent and comprehensive illustration of the entire pretraining framework.\n\nWeakness\n1. The paper defines 3D awareness as the ability to infer depth from its representations. While practical, this is a narrow definition of 3D understanding. The model is trained to regress a 2.5D depth map provided by a teacher model. This raises the question of whether EgoDTM is learning fundamental geometric principles or simply becoming very good at mimicking the output of the teacher model (DepthAnythingv2). True 3D reasoning involves understanding object permanence, volume, relative pose, and interaction affordances, which are not directly measured or enforced by the current depth-prediction task. The improvements on downstream tasks could stem from better foreground-background segmentation (a side-effect of depth prediction) rather than a deeper, more holistic 3D scene understanding.\n\n2. The enriched captions generated by the pipeline are not only more spatially aware but also significantly longer and more descriptive than the original captions. Could the performance improvement in retrieval tasks be partially attributed to this increased textual detail, rather than purely to the improved spatial accuracy? An analysis that attempts to disentangle the effects of caption length/detail from spatial correctness would strengthen this contribution.\n\n3. The paper suggests that its contributions are beneficial for downstream tasks like \"robot manipulation\" in line 82. However, the experiments do not include any tasks on a physical robot or in a simulated robotic environment. There remains a significant sim-to-real (or in this case, perception-to-action) gap, and therefore, the claim of improving robot manipulation is speculative and not directly obtained by the provided results.\n\n4. The overall setting utilized in the ablation study should be further clarified for easier performance comparison. Besides, it would be better if the author could conduct the ablation study on other tasks (not merely the short video understanding ( video-text retrieval and action recognition).", "was_revised": true, "final_justification": "Depth estimation as proxy task may not be very strong technical contribution. But the result of the robot manipulation experiments is a strong support. I maintain my original score.", "rebuttal": "We greatly appreciate the time and effort you invested in providing these detailed observations and questions. We have carefully considered your comments and outlined our responses and proposed revisions below. We hope these clarifications and revisions address your concerns and further enhance the manuscript.\n\n**W1：The proxy task of depth estimation may not lead the model to truly understand 3D information. And downstream tasks do not include such true 3D tasks.** \n\n-  **On the choice of proxy task**: We acknowledge that depth estimation may not fully capture all aspects of true 3D understanding. Richer 3D supervision (such as from multi-view geometry or dynamic 3D correspondences in videos) could provide stronger inductive signals. However, our work represents an initial attempt to incorporate 3D supervision into large-scale egocentric video-language pretraining. Introducing 3D signals and training procedures—whether through data collection or model design—is highly non-trivial at the million-scale pretraining level. Despite these challenges, our results show that depth estimation serves as a practical and effective proxy, enabling the encoder to become more 3D-aware and motivating further research in this direction.\n- **On downstream task evaluation**: We agree that evaluating a visual encoder's physical 3D reasoning capability is essential. However, we argue that there are currently few suitable benchmarks to directly evaluate the encoder's capability. Instead, we choose to demonstrate that 3D-aware encoders have potential for solving 2D perception tasks like short video understanding on EK100MIR, EgoMCQ, **EK100CLS [added during rebuttal period, see response to W3 below]**, long video understanding on EgoNLQ, **EgoMQ [action detection task, added during rebuttal period, see table below]**, and 3D-aware tasks on H2O, FrankaKitchen [in supplementary].\n\n|          |        |        |  EgoMQ |        |        |        |         |\n|:--------:|:------:|:------:|:------:|:------:|:------:|:------:|:-------:|\n|          | R1@0.3 | R5@0.3 | R1@0.5 | R5@0.5 | R1@0.7 | R5@0.7 | avg-mAP |\n|  EgoVLP  |  30.44 |  46.66 |  22.41 |  35.75 |  13.05 |  20.21 |   6.11  |\n|  LaViLa  |  32.9  |  48.68 |  **24.12** |  37.59 |  **15.23** |  22.47 |   8.12  |\n| AVION |  32.17 |  47.3  |  23.11 |  36.3  |  13.45 |  19.6  |   6.23  |\n|  EgoDTM  |  **32.92** | **50.08** |  23.94 |  **39.15** |  14.81 | **22.65** |   **8.87**  |\n\n\n**W2：Is the performance gain in the downstream retrieval task mainly attributed to longer captions?.** \n\nNo.\n- First, for downstream vision-language tasks such as retrieval, **we only utilize the encoder during inference**. As shown in Figure 2, no caption augmentation is performed at inference time. Therefore, the improved retrieval performance reflects the model’s enhanced video-text alignment, which is acquired during training with our enriched captions. \n- Furthermore, we highlight that **both baseline methods, LaViLa and AVION, leverage LLMs and MLLMs to expand and rewrite the original captions**, resulting in 35 million enriched descriptive captions. In contrast, we only have 2.5 million newly generated captions informed by HOI masks. Despite using significantly fewer enriched captions, our method outperforms both baselines.\n- To follow your advice, we promise to conduct a pretraining ablation as follows: generate enriched longer and descriptive captions using LLMs without spatial information, to eliminate the impact of spatial-awareness in texts. Due to space and resource constraints, we have not included such pretraining ablation in this version, but we promise to add them to our revised version.\n\n**W3: Do you have experimental support for robot manipulation?**\n\nYes, we apologize for the confusion. In Appendix Table 1, we present results from robot manipulation experiments conducted in the Franka environment. EgoDTM outperforms baselines including CLIP, ConvNext and LaViLa, demonstrating the potential value of EgoDTM encoder in robotic applications.Some beneficial results are presented below:\n\n|        |   TK   |  OD |   OM   |   FS   |   SD   | Average |\n|--------|:------:|:---:|:------:|:------:|:------:|:-------:|\n| ResNet |   28%  | 18% | 26.70% |   50%  | 75.50% |  39.70% |\n|  CLIP  | 26.30% | 13% | 24.70% | 41.70% | 86.30% |  38.40% |\n| LaViLa |   48%  | 26% | 22.50% |   69%  | **94.50%** |   52%   |\n| EgoDTM |   **56%**  | **28%** | **35.50%** |   **81%**  | 92.50% |  **58.60%** |\n\n**W4: Could you conduct ablation studies on more tasks?** \n\nYes. We have conducted additional ablation studies on almost all tasks to further validate the effectiveness of our model. During rebuttal, we add experiments including action recognition task on Epic-Kitchens-100 (EK-100-CLS) and action detection task on Ego4D (EgoMQ). The results are presented below: \n\n|                                              | EK100MIR     | EgoMCQ          | EK100CLS         | EgoNLQ | EgoMQ |Depth                                  | \n|----------------------------------------------|---------------|-----------------|------------------|--------|-------|---------------------------------------|\n| metrics                                      | mAP / nDCG↑   | inter / intra acc ↑| top-1 / top-5 acc ↑| mIoU   ↑| mAP   ↑|scale-aware RMSE / scale-invariant RMSE ↓ | \n| $\\mathcal{L}_{\\rm{vtc}}$                          | 29.7 / 30.7  | 94.2 / 60.2     | 12.847 / 30.037  | 6.14   | 6.97  |0.572 / 0.495                          | \n| $\\mathcal{L}\\_{\\rm{vtc}}+\\mathcal{L}_{\\rm{depth}}$      | *31.3* / 31.2| *94.2* / **62.6**| 15.412 / *32.995* | 5.98   | *7.52* |*0.5637* / **0.464**                   | \n| $\\mathcal{L}_{\\rm{augvtc}}$                       | *31.3* / *32.2*| 94 / 61.6       | **16.508** / 32.851| **6.53**| 6.14  |0.550 / 0.489                          | \n| $\\mathcal{L}\\_{\\rm{augvtc}}+\\mathcal{L}_{\\rm{depth}}$   |  **33.1** / **33.1** | **94.6** / **62.6** | *15.898* / **33.895** | *6.17* | **8.87** |**0.539** / *0.481*                    |\n\nAs shown in the table, progressively adding our modules leads to consistent improvements across the majority of tasks, and our model outperforms all baselines in the main experiments. One potential limitation is that depth pretraining may negatively impact the performance on EgoNLQ to some extent. Nevertheless, our proposed AugVTC module is able to mitigate this effect and yields improvements that surpass those achieved by Avion, LaViLa, and the EgoVLP encoder as shown in Table4 of our main paper. Besides, comparing row 2 and row 4 when adding $\\mathcal{L}\\_{\\rm{augvtc}}$, we observe an increase in the scale-invariant RMSE and a decrease in the scale-aware RMSE. We hypothesize that this may result from the proxy task bias introduced by multi-task pretraining.\n\n**Q: The paper is not written in the full 9 pages.** \n\nThanks for pointing this out. In the current submission, we aimed to balance content between the main paper and the supplementary material and some detailed results and analyses were moved to the supplementary to avoid exceeding the page limit. However, this led to underutilization of the main paper space. In the revised version, we will reorganize the content to make full use of the 9-page limit, including adding the comprehensive ablation study across downstream tasks as well as results on the EK-100-CLS and EgoMQ benchmark."}, {"review_id": "S9a3g2Z5nl", "reviewer": "Reviewer_YGLr", "summary": "The paper proposes an egocentric video pretraining model that enhances traditional video-language models with 3D awareness. The pretraining process introduces a lightweight 3D-aware decoder, supervised by incorporating 3D depth information. Additionally, the authors enrich traditional textual descriptions with spatial awareness. Experimental results demonstrate that the proposed pretraining approach yields improved performance across several downstream tasks, including depth estimation, language-based video queries, video-text retrieval, and action recognition.", "questions": "1. How much does it cost to generate the enhanced dataset using all these foundation models? How does the author envision generalizing this process to web-scale video data or general video pretraining?\n\n2. Given the limited improvement in the numerical results, can the author provide an in-depth analysis of common failure cases where the 3D-aware video features still fail to resolve the issues?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\nThe paper addresses a significant limitation in egocentric video understanding by incorporating 3D depth information. The proposed method effectively integrates off-the-shelf depth estimation models for supervision and enriches video captions with spatially-aware information through a detect-track-generate pipeline.\n\n**Weakness**\n\nOne concern is that while the improvements across benchmarks are consistent, they are often modest (typically around 1–2%). Another concern lies in the resource requirements for generating the pretraining data. The method heavily depends on multiple foundation models (e.g., DepthAnything, SAM2, HOI-D, and large language models), which may limit scalability for broader or more general video pretraining applications.", "was_revised": true, "final_justification": "Thank you for the response. All of my concerns have been addressed with the additional results. In the rebuttal, the authors have shown promising potential to extend the training data generation pipeline at an acceptable cost, and they plan to include more failure case analyses as suggested. The proposed method, EgoDTM, makes solid contributions by enhancing video pretraining with both enriched captions and depth information. While using only depth has limitations in fully representing the 3D world—including aspects like object location and orientation—the method still represents a meaningful step forward. I will keep my positive rating.", "rebuttal": "We greatly appreciate your thoughtful assessment and comments on our paper. Below, we address the concerns in detail:\n\n**W1: The performance improvements across benchmarks seem marginal.**\n\nWe argue that our improvements are non-trivial. As shown in the ablation study in Table 3(a), EgoDTM achieves a +3.4 mAP gain over the vanilla baseline on the EK-100-MIR benchmark, improving from 29.7 to 33.1. Furthermore, on the depth estimation task, our model achieves a 9.8% improvement in Scale-aware RMSE over the best-performing baseline (LaViLa), as shown in Table 2. We enumerate the additional supervision signals employed by the baselines and present comparisons to demonstrate that the improvements achieved by our method are substantial rather than modest.\n\n**Q1: How long does it take to generate the training data? And how do you consider the scalability of the approach?**\n- At inference time, depth map generation takes approximately 0.45 seconds per 4-frame 720p video, using 4 GB of GPU memory. Generating HOI bounding boxes and masks takes around 4.96 seconds per 4-frame 720p input, with a memory footprint of 3.8 GB. Therefore, generating 2.5 million training samples requires approximately 159 A100 GPU hours for HOI bounding box and mask generation, and 15 A100 GPU hours for depth map generation. For LLM text generation, we spent $418 USD and approximately 8 hours of API usage to generate 2.5 million captions.\n\n|           Evaluated on 1 A100 card           |    Depth    | HOI Box+HOI Mask |\n|:--------------------------------------------:|:-----------:|:----------------:|\n|          Time per 4-frame video (s)          |    0.45s    |       4.96s      |\n|                  GPU Mem (G)                 |      4G     |       3.8G       |\n|    Total A100 GPU Hours for 2M data (hour)   |     15h     |       159h       |\n\n- From a scalability perspective, we believe this is a highly cost-effective and easily extensible paradigm. The cost of pseudo-label generation is relatively low, and our 3D-aware decoder is lightweight. We have already demonstrated the feasibility of this approach on the million-sample scale, and scaling it to tens of millions is within reach.\n\n**Q2: Common failure cases of EgoDTM.** \n\nBased on our observations, EgoDTM achieves notable improvements over the base model in action recognition and is better able to attend to HOI-relevant regions in complex scenes. Nonetheless, it still suffers from a relatively high error rate in challenging scenarios, such as those involving extreme viewpoints or occlusions of human-object interactions. We plan to include the corresponding qualitative examples in the appendix of the revised version."}, {"review_id": "6IFy6SR62e", "reviewer": "Reviewer_pB7n", "summary": "To enhance the understanding of egocentric scenarios, the authors propose EgoDTM, a video-language model equipped with 3D-aware capabilities. The model incorporates a lightweight 3D-aware decoder, which learns spatial structure information from pseudo-depth maps generated by a depth estimation algorithm, under the supervision of pseudo-depth labels provided by a teacher model. Additionally, EgoDTM integrates several foundational modules—such as hand detection and object recognition—to enrich the original concise textual descriptions with semantic cues of hand-object interactions. The effectiveness of EgoDTM is demonstrated across various tasks, including video-text retrieval, action recognition, depth estimation, robotic manipulation, and long video understanding.", "questions": "1) Lack of in-depth ablation studies on the 3D-aware video pretraining architecture. The current evaluation only includes an ablation on 𝐿_depth , which is insufficient to thoroughly and clearly demonstrate the advantages and underlying mechanism of the proposed 3D-aware decoder.\n\n2) Concerns about the impact of the teacher model on performance. Since the enhancement relies on incorporating DepthAnything as a teacher model, it remains unclear how much of the performance gain is attributable to this component. This raises the question of whether a simpler knowledge distillation approach could achieve similar improvements in 3D perception without introducing complex structures.\n\n3) More detailed ablation on the depth estimation task of the H2O dataset would provide stronger evidence. As this task directly reflects the model's 3D perception capability, conducting thorough ablation on key components (e.g., enhanced captions, 3D-aware decoder) on the H2O dataset would offer clearer validation of each module’s contribution to 3D understanding.", "limitations": "The limitation mentioned by the authors is solely that the method has not yet been explored in broader domains.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1) Clear motivation and sound methodology. The authors aim to bridge the modality gap in video-language pretraining by introducing depth supervision, incorporating HOI detection, keyframe selection, and bidirectional tracking, and generating spatially-grounded captions using large language models. This motivation is well-justified and practically relevant.\n\n2) Well-structured and clearly written. The paper is logically organized, and the methods and experiments are presented in a complete and comprehensible manner.\n\nWeaknesses:\n1) Limited ablation study. The paper only conducts a coarse ablation on 𝐿_vtc, 𝐿_depth, and 𝐿_augvtc. While the difference between 𝐿_vtc and 𝐿_augvtc clearly demonstrates the benefit of spatially-enriched text, the actual impact of the 3D-aware decoder is unclear. Its effect is only reflected by whether 𝐿_depth is applied or not, which fails to provide insightful evidence for the value and effectiveness of this module.\n\n2) Potential fairness issue with the teacher model. To generate pseudo-depth labels, the authors incorporate DepthAnything as a teacher model during 3D-aware video pretraining. This could lead to unfair comparisons with other methods. Moreover, since the role of the 3D-aware decoder is not sufficiently validated, it raises the question of whether the gains are primarily due to the teacher model itself. For instance, replacing the current pretraining strategy with a simpler knowledge distillation approach might yield similar results—this possibility is not ruled out.\n\n3) Lack of direct evaluation of 3D perception enhancement. Both the enhanced textual descriptions and the proposed 3D-aware decoder are designed to improve 3D perception. However, no ablation is conducted on the depth estimation task of the H2O dataset, which would have been crucial to explicitly identify which components most effectively contribute to 3D understanding.", "was_revised": true, "final_justification": "After reading the authors' rebuttal, the authors have addressed my major concerns. I will raise my score.", "rebuttal": "We greatly appreciate the time and effort you invested in providing these detailed observations and questions. We have carefully considered your comments and outlined our responses and proposed revisions below. We hope these clarifications and revisions address your concerns and further enhance the manuscript.\n\n**W1 & Q1：The role of the 3D-aware decoder is unclear.**\n\n$L\\_{\\rm{depth}}$ refers to the combination of the 3D-aware decoder and the associated depth loss ($L\\_{\\rm{depth}}$); both components are either jointly added or removed. Therefore, the ablation of $L\\_{\\rm{depth}}$ illustrates the effectiveness of the 3D-aware video pretraining process. \n\nDuring rebuttal, we sincerely apologize that we might not be able to conduct more pretraining experiments, but In the revised version, we promise to include additional ablations by replacing the 3D-aware decoder with alternatives such as a lightweight MLP decoder or a ConvNet decoder to further validate the effectiveness of our 3D-aware decoder. \n\nNonetheless, In this work, our primary contribution is the introduction of 3D-awareness into EgoVLP, demonstrating its general applicability across a wide range of tasks and achieving performance that surpasses previous methods.\n\n**W2 & Q2: Potential Unfair Comparison of Using Depth Anything as Teacher Model**\n\nWe first provide a comparison of different methods' pretraining process as follows:\n\n|              | EgoVLP | EgoVLPv2 | AVION | LaViLa | HelpingHands |  HENASY  |   EgoDTM   |\n|:------------:|:------:|:--------:|:-----:|:------:|:------------:|:--------:|:----------:|\n|  Supervision |  Text  |   Text   |  Text |  Text  |   Text+Box   | Text+Box | Text+Depth |\n|   Text Num   |   4M   |    4M    |  35M  |   35M  |      35M     |    35M   |     6M     |\n|    Params    |  172M  |   363M   |  169M |  142M  |     206M     |   277M   |    151M    |\n| EK100MIR-mAP |  23.3  |   30.8   |  32.9 |  30.8  |     31.2     |   31.3   |    33.5    |\n\n- **First, all methods are evaluated fairly during inference.** We ensure fair comparisons by using the same original video-text pairs and maintaining comparable model sizes. However, it is both common and inevitable for different approaches to incorporate distinct foundation models for pseudo-label generation during pretraining. For example, LaViLa utilizes T5-Large for text rewriting, while HelpingHands employs an HOI detector to obtain HOI bounding boxes.\n- Moreover, **different methods aim to prove the effectiveness of different types of supervision signals**, as illustrated in Figure 1 of our paper, often relying on powerful teacher models tailored to their objectives. For instance, LaViLa and AVION leverage a multimodal large language model and LLM to synthesize ten times more text data, i.e., 35M text data for supervision, while we use only 6M text data in total for pretraining. HelpingHands[ICCV'23] and HENASY[NeurIPS'24] not only generates HOI boxes using an HOI detector but also adopts HOI detection as an auxiliary pretraining task.\n- **Finally, our objective is not to distill knowledge from a specific foundation model, but rather to learn from noisy 3D supervision signals directly**. As such, knowledge distillation is not the focus of this work. Depth supervision derived from any depth estimation foundation model, DepthPro [ICLR'25], can serve the same purpose. But we admit that there exists bias from the DepthAnything model. In future work, we plan to include additional experiments using depth maps generated by DepthPro to further validate our approach.\n\n**W3 & Q3: Lack of ablation studies for depth estimation on H2O.**\n\nThank you very much for your suggestion. Following your advice, we have conducted ablation studies on almost all downstream tasks including H2O. The results are as follows:\n\n|                                              | EK100MIR     | EgoMCQ          | EK100CLS         | EgoNLQ | EgoMQ |Depth                                  | \n|----------------------------------------------|---------------|-----------------|------------------|--------|-------|---------------------------------------|\n| metrics                                      | mAP / nDCG↑   | inter / intra acc ↑| top-1 / top-5 acc ↑| mIoU   ↑| mAP   ↑|scale-aware RMSE / scale-invariant RMSE ↓ | \n| $\\mathcal{L}_{\\rm{vtc}}$                          | 29.7 / 30.7  | 94.2 / 60.2     | 12.847 / 30.037  | 6.14   | 6.97  |0.572 / 0.495                          | \n| $\\mathcal{L}\\_{\\rm{vtc}}+\\mathcal{L}_{\\rm{depth}}$      | *31.3* / 31.2| *94.2* / **62.6**| 15.412 / *32.995* | 5.98   | *7.52* |*0.5637* / **0.464**                   | \n| $\\mathcal{L}_{\\rm{augvtc}}$                       | *31.3* / *32.2*| 94 / 61.6       | **16.508** / 32.851| **6.53**| 6.14  |0.550 / 0.489                          | \n| $\\mathcal{L}\\_{\\rm{augvtc}}+\\mathcal{L}_{\\rm{depth}}$   |  **33.1** / **33.1** | **94.6** / **62.6** | *15.898* / **33.895** | *6.17* | **8.87** |**0.539** / *0.481*                    |\n\nAs shown, incorporating our components consistently improves the scale-aware RMSE metric, indicating that our method enhances the understanding of the true depth. Notably, when comparing row 2 and row 4, where enriched text is added, we observe an increase in the scale-invariant RMSE, which indicates impairing the understanding of relative depth structures.\nGenerally, the two components proposed in this work generally lead to improvements on the majority of 2D vision tasks, indirectly demonstrating the effectiveness of 3D-awareness."}, {"review_id": "kWPs4wRZED", "reviewer": "Reviewer_LJgN", "summary": "This paper proposes a novel egocentric video-language pretraining method, which aims to inject 3D-awareness into video-language foundation models. The key design of the proposed EgoDTM is incorporating a lightweight 3D-aware decoder to effciently learn 3D-awareness from pseudo depth maps generated by DepthAnythingV2. Besides, EgoDTM enriches the original short video captions with hand-object visual cues by organically combining several foundation models. Experiments on Epic-Kitchens-100-MIR, EGTEA and EgoMCQ demonstrate the effectiveness of the proposed method.", "questions": "- The experiments are conducted on mainly three benchmarks, which is less than previous works like HENASY. HENASY has conducted more experiments on EK100-CLS, EgoNLQ and EgoMQ, thus I think the authors can consider adding more experiments on these benchmarks. \n- The proposed method uses DeepSeek-LLM-200B model to enrich video captions. However, DeepSeek-LLM-200B should be an LLM without involving video inputs, how do you generate rich video descriptions? If no video inputs are involved, I think there may be some hallucination. \n- The key contribution of the paper is introducing 3D-awareness into egocentric video-language pretraining. However, the paper does not provide experiments specifically designed to show models' 3D-awareness. I suggest the authors to add some experiments, e.g., can the proposed design lead to better understanding of the relationships of objects in terms of front and back.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper is well written and easy to follow.\n- To the best of my knowledge, this paper is the first to introduce 3D-awareness in egocentric video-language pretraining. \n- The proposed design based on depth is reasonable, and the experiments demonstrate the effectiveness. \n\nWeaknesses:\n- The experiments are limited on mainly three benchmarks, which is less than previous works like HENASY. \n- I think the authors should provide some experiments to demonstrate their superiority in understanding 3D scenes, as their main contribution is about 3D-awareness. \n- Please refer to the Questions part for more details. If the authors can address my concerns, I am very willing to increase my rating.", "was_revised": true, "final_justification": "My concerns have been addressed after rebuttal. Most importantly, the authors provide experiments results on EgoMQ and EK100-CLS to further demonstrate the effectiveness of the proposed model, and they also present a detailed explanation to show that why their experiments can verify the 3D-awareness of their model. I really appreciate their effort in addressing these concerns. \n\nMy only concern is similar to that of Reviewer YGLr, i.e., introducing only depth as 3D guidance has limitations. However, I still believe this work represents **a meaningful step forward**. \n\nOverall, I recommend accepting this paper due to its good contributions.", "rebuttal": "We greatly appreciate the time and effort you invested in providing these detailed observations and questions. We have carefully considered your comments and outlined our responses and proposed revisions below. We hope these clarifications and revisions address your concerns and further enhance the manuscript.\n\n**W1 & Q1: The paper lacks sufficient downstream task evaluations.**\n\nThank you for your valuable feedback. First, we sincerely apologize if the formatting caused you to miss our EgoNLQ experiments in Table 4. We also include robot manipulation experiments in Appendix Table 1. Furthermore, we have added new experiments on EgoMQ, as presented below: \n\n|          |        |        |  EgoMQ |        |        |        |         |\n|:--------:|:------:|:------:|:------:|:------:|:------:|:------:|:-------:|\n|          | R1@0.3 | R5@0.3 | R1@0.5 | R5@0.5 | R1@0.7 | R5@0.7 | avg-mAP |\n| AVION35M |  32.17 |  47.3  |  23.11 |  36.3  |  13.45 |  19.6  |   6.23  |\n|  LaViLa  |  32.9  |  48.68 |  **24.12** |  37.59 |  **15.23** |  22.47 |   8.12  |\n|  EgoVLP  |  30.44 |  46.66 |  22.41 |  35.75 |  13.05 |  20.21 |   6.11  |\n|  EgoDTM  |  **32.92** |  **50.08** |  23.94 |  **39.15** |  14.81 |  **22.65** |   **8.87**  |\n\nBesides, we add ablation on action recognition task on Epic-Kitchens-100, i.e., EK100CLS below:\n|          |  EK100-CLS |         |\n|:---------:|:------:|:-------:|\n|          | top-1 acc | top-5 acc|\n|$\\mathcal{L}_{\\rm{vtc}}$    | 12.84 | 30.03 |\n|$\\mathcal{L}\\_{\\rm{vtc}}+\\mathcal{L}_{\\rm{depth}}$        | 15.41|32.99  |\n|$\\mathcal{L}_{\\rm{augvtc}}$    | 16.50 | 32.85 |\n|$\\mathcal{L}\\_{\\rm{augvtc}}+\\mathcal{L}_{\\rm{depth}}$     | 15.89 | 33.89 |\n\nIn our updated version, we will revise the layout to present EgoNLQ and EgoMQ side by side in two columns.\n\n**W2 & Q3：This paper should provide more experiments related to 3D scene understanding.**\n\nThis work currently focuses on 3D-awareness centered around human actions from an egocentric view instead of indoor scene understanding, as illustrated in Figure 3. Although our model is capable of providing some depth understanding of surrounding objects / humans (as illustrated in the first Ego4D example in Figure3), it is overall constrained within the scope of egocentric hand-object interactions. \n\nWe argue that the benefits of 3D-awareness can be reflected through improved performance on general downstream tasks like short video understanding on EK100MIR, EgoMCQ, EK100CLS [added during rebuttal period], long video understanding on EgoNLQ, EgoMQ [added during rebuttal period], and 3D-aware tasks on H2O, FrankaKitchen [in supplementary]. We plan to further explore more general-purpose 3D-aware visual encoders in future work.\n\n**Q2: How are the captions generated using LLMs? How do you handle hallucinations?**\n\nWe carefully prompt an LLM to understand the change of HOI masks to ensure faithfulness compared with original captions. To prove the faithfulness, we evaluate the enriched captions and original captions by calculating the BERTScore[ICLR'20], the results are presented below, which indicates the good quality of our captions.\n\n| Recall             | Precision         | F1                 |\n|--------------------|-------------------|--------------------|\n| 0.818 | 0.546 | 0.652 |\n\n**Regarding the hallucination issue, our perspective is as follows:**\n- First, through careful prompt engineering and manual verification, we found that the LLM can effectively capture changes in bounding boxes and generate corresponding descriptive modifications, while largely preserving the original semantic content.\n- Second, although some hallucinations are present, training with these captions consistently improves performance across most downstream benchmarks, as shown in our ablation studies. This suggests that the benefits outweigh the potential drawbacks.\n- Finally, prior work such as LLaVA [NeurIPS’23] has demonstrated that LLMs are capable of understanding bounding box information and producing fine-grained descriptions, which leads to better generalization. Inspired by this, we believe it is feasible to generate high-quality captions by prompting LLMs to understand HOI mask information. Moreover, we intuitively believe that generating captions using video-based MLLMs (e.g., Gemini-2.5-Pro, QwenVL2.5, InternVL3) is more prone to hallucination compared to our proposed pipeline."}]

Author rebuttal:

Meta-review: In their initial ratings, the reviewers were split: two leaned toward rejection and two toward acceptance.

Reviewers agreed that the paper addresses an important limitation of egovideo understanding models, that is the lack of 3D awareness. The authors propose a reasonable pretraining method that uses an off-the-shelf depth estimator to supervise a lightweight 3D encoder. The submission is well written, and the experiments across multiple benchmarks show the effectiveness of the approach. However, reviewers raised several concerns, including: The need for ablations to separate the effects of generated captions versus spatial awareness; missing evidence of improved performance in robot manipulation tasks; potential scalability issues; missing evaluation of improved 3D perception.

In the rebuttal, the authors provided new experiments and results that effectively addressed these concerns. After rebuttal, all reviewers are leaning toward acceptance, with reviewers LJgN and YGLr noting that while relying only on 3D awareness has limitations, the work still represents a meaningful step forward.

The AC agrees with the reviewers consensus and recommends acceptance. The AC also asks the authors to include the new ablations and results from the rebuttal in the final version for completeness.
Decision: Accept (poster)