Title: EgoDTM: Towards 3D-Aware Egocentric Video-Language Pretraining
Venue: NeurIPS 2025
Accepted: yes
Keywords: egocentric vision

video-language pretraining

multi-task learning

3D-aware video representation
Abstract: Egocentric video-language pretraining has significantly advanced video representation learning. Humans perceive and interact with a fully 3D world, developing spatial awareness that extends beyond text-based understanding. However, most previous works learn from 1D text or 2D visual cues, such as bounding boxes, which inherently lack 3D understanding.  To bridge this gap, we introduce EgoDTM, an Egocentric Depth- and Text-aware Model, jointly trained through large-scale 3D-aware video pretraining and video-text contrastive learning. EgoDTM incorporates a lightweight 3D-aware decoder to efficiently learn 3D-awareness from pseudo depth maps generated by depth estimation models. To further facilitate 3D-aware video pretraining, we enrich the original brief captions with hand-object visual cues by organically combining several foundation models. Extensive experiments demonstrate EgoDTM's superior performance across diverse downstream tasks, highlighting its superior 3D-aware visual understanding. Code: \url{https://github.com/xuboshen/EgoDTM}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel egocentric video-language pretraining method, which aims to inject 3D-awareness into video-language foundation models. The key design of the proposed EgoDTM is incorporating a lightweight 3D-aware decoder to effciently learn 3D-awareness from pseudo depth maps generated by DepthAnythingV2. Besides, EgoDTM enriches the original short video captions with hand-object visual cues by organically combining several foundation models. Experiments on Epic-Kitchens-100-MIR, EGTEA and EgoMCQ demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is well written and easy to follow.\n- To the best of my knowledge, this paper is the first to introduce 3D-awareness in egocentric video-language pretraining. \n- The proposed design based on depth is reasonable, and the experiments demonstrate the effectiveness. \n\nWeaknesses:\n- The experiments are limited on mainly three benchmarks, which is less than previous works like HENASY. \n- I think the authors should provide some experiments to demonstrate their superiority in understanding 3D scenes, as their main contribution is about 3D-awareness. \n- Please refer to the Questions part for more details. If the authors can address my concerns, I am very willing to increase my rating.\n\nQUESTIONS:\n- The experiments are conducted on mainly three benchmarks, which is less than previous works like HENASY. HENASY has conducted more experiments on EK100-CLS, EgoNLQ and EgoMQ, thus I think the authors can consider adding more experiments on these benchmarks. \n- The proposed method uses DeepSeek-LLM-200B model to enrich video captions. However, DeepSeek-LLM-200B should be an LLM without involving video inputs, how do you generate rich video descriptions? If no video inputs are involved, I think there may be some hallucination. \n- The key contribution of the paper is introducing 3D-awareness into egocentric video-language pretraining. However, the paper does not provide experiments specifically designed to show models' 3D-awareness. I suggest the authors to add some experiments, e.g., can the proposed design lead to better understanding of the relationships of objects in terms of front and back.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo enhance the understanding of egocentric scenarios, the authors propose EgoDTM, a video-language model equipped with 3D-aware capabilities. The model incorporates a lightweight 3D-aware decoder, which learns spatial structure information from pseudo-depth maps generated by a depth estimation algorithm, under the supervision of pseudo-depth labels provided by a teacher model. Additionally, EgoDTM integrates several foundational modules—such as hand detection and object recognition—to enrich the original concise textual descriptions with semantic cues of hand-object interactions. The effectiveness of EgoDTM is demonstrated across various tasks, including video-text retrieval, action recognition, depth estimation, robotic manipulation, and long video understanding.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1) Clear motivation and sound methodology. The authors aim to bridge the modality gap in video-language pretraining by introducing depth supervision, incorporating HOI detection, keyframe selection, and bidirectional tracking, and generating spatially-grounded captions using large language models. This motivation is well-justified and practically relevant.\n\n2) Well-structured and clearly written. The paper is logically organized, and the methods and experiments are presented in a complete and comprehensible manner.\n\nWeaknesses:\n1) Limited ablation study. The paper only conducts a coarse ablation on 𝐿_vtc, 𝐿_depth, and 𝐿_augvtc. While the difference between 𝐿_vtc and 𝐿_augvtc clearly demonstrates the benefit of spatially-enriched text, the actual impact of the 3D-aware decoder is unclear. Its effect is only reflected by whether 𝐿_depth is applied or not, which fails to provide insightful evidence for the value and effectiveness of this module.\n\n2) Potential fairness issue with the teacher model. To generate pseudo-depth labels, the authors incorporate DepthAnything as a teacher model during 3D-aware video pretraining. This could lead to unfair comparisons with other methods. Moreover, since the role of the 3D-aware decoder is not sufficiently validated, it raises the question of whether the gains are primarily due to the teacher model itself. For instance, replacing the current pretraining strategy with a simpler knowledge distillation approach might yield similar results—this possibility is not ruled out.\n\n3) Lack of direct evaluation of 3D perception enhancement. Both the enhanced textual descriptions and the proposed 3D-aware decoder are designed to improve 3D perception. However, no ablation is conducted on the depth estimation task of the H2O dataset, which would have been crucial to explicitly identify which components most effectively contribute to 3D understanding.\n\nQUESTIONS:\n1) Lack of in-depth ablation studies on the 3D-aware video pretraining architecture. The current evaluation only includes an ablation on 𝐿_depth , which is insufficient to thoroughly and clearly demonstrate the advantages and underlying mechanism of the proposed 3D-aware decoder.\n\n2) Concerns about the impact of the teacher model on performance. Since the enhancement relies on incorporating DepthAnything as a teacher model, it remains unclear how much of the performance gain is attributable to this component. This raises the question of whether a simpler knowledge distillation approach could achieve similar improvements in 3D perception without introducing complex structures.\n\n3) More detailed ablation on the depth estimation task of the H2O dataset would provide stronger evidence. As this task directly reflects the model's 3D perception capability, conducting thorough ablation on key components (e.g., enhanced captions, 3D-aware decoder) on the H2O dataset would offer clearer validation of each module’s contribution to 3D understanding.\n\nLIMITATIONS:\nThe limitation mentioned by the authors is solely that the method has not yet been explored in broader domains.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an egocentric video pretraining model that enhances traditional video-language models with 3D awareness. The pretraining process introduces a lightweight 3D-aware decoder, supervised by incorporating 3D depth information. Additionally, the authors enrich traditional textual descriptions with spatial awareness. Experimental results demonstrate that the proposed pretraining approach yields improved performance across several downstream tasks, including depth estimation, language-based video queries, video-text retrieval, and action recognition.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThe paper addresses a significant limitation in egocentric video understanding by incorporating 3D depth information. The proposed method effectively integrates off-the-shelf depth estimation models for supervision and enriches video captions with spatially-aware information through a detect-track-generate pipeline.\n\n**Weakness**\n\nOne concern is that while the improvements across benchmarks are consistent, they are often modest (typically around 1–2%). Another concern lies in the resource requirements for generating the pretraining data. The method heavily depends on multiple foundation models (e.g., DepthAnything, SAM2, HOI-D, and large language models), which may limit scalability for broader or more general video pretraining applications.\n\nQUESTIONS:\n1. How much does it cost to generate the enhanced dataset using all these foundation models? How does the author envision generalizing this process to web-scale video data or general video pretraining?\n\n2. Given the limited improvement in the numerical results, can the author provide an in-depth analysis of common failure cases where the 3D-aware video features still fail to resolve the issues?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe research paper introduces EgoDTM, an Egocentric Depth- and Text-aware Model designed to address the lack of 3D understanding in existing video-language pretraining frameworks. Current models often rely on 1D text or 2D visual cues, but humans perceive a 3D world. To bridge this gap, EgoDTM jointly trains a model using both video-text contrastive learning and a novel 3D-aware video pretraining task. This task incorporates a lightweight 3D-aware decoder that efficiently learns to predict depth from pseudo-depth maps generated by powerful depth estimation models. To further enhance spatial reasoning, the system enriches brief, original captions with detailed hand-object interaction (HOI) information through a \"detect-track-generate\" pipeline that leverages a combination of foundation models. Extensive experiments show that EgoDTM achieves superior performance across diverse downstream tasks, including video-text retrieval, action recognition, and robot manipulation, highlighting its improved 3D-aware visual understanding.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1.The paper addresses a well-motivated and significant limitation in video-language understanding—the absence of 3D spatial reasoning. \nThe core idea of integrating a depth-prediction task as an auxiliary objective for pretraining is effective, as demonstrated by strong empirical results.\n\n2.The design of the lightweight 3D-aware decoder is a key strength. It resolves the tension between the high-resolution requirements of dense prediction tasks (like depth estimation) and the large-batch requirements of contrastive learning. \n\n3.The empirical evaluation is thorough and convincing. The authors test their model on a wide array of downstream tasks, including zero-shot retrieval, action recognition, depth estimation, and temporal localization. Consistent and significant performance gains over strong baselines across all these tasks strongly support the paper's claims.\n\n4.The paper is well-written, clearly structured, and easy to follow. Figure 2, in particular, provides an excellent and comprehensive illustration of the entire pretraining framework.\n\nWeakness\n1. The paper defines 3D awareness as the ability to infer depth from its representations. While practical, this is a narrow definition of 3D understanding. The model is trained to regress a 2.5D depth map provided by a teacher model. This raises the question of whether EgoDTM is learning fundamental geometric principles or simply becoming very good at mimicking the output of the teacher model (DepthAnythingv2). True 3D reasoning involves understanding object permanence, volume, relative pose, and interaction affordances, which are not directly measured or enforced by the current depth-prediction task. The improvements on downstream tasks could stem from better foreground-background segmentation (a side-effect of depth prediction) rather than a deeper, more holistic 3D scene understanding.\n\n2. The enriched captions generated by the pipeline are not only more spatially aware but also significantly longer and more descriptive than the original captions. Could the performance improvement in retrieval tasks be partially attributed to this increased textual detail, rather than purely to the improved spatial accuracy? An analysis that attempts to disentangle the effects of caption length/detail from spatial correctness would strengthen this contribution.\n\n3. The paper suggests that its contributions are beneficial for downstream tasks like \"robot manipulation\" in line 82. However, the experiments do not include any tasks on a physical robot or in a simulated robotic environment. There remains a significant sim-to-real (or in this case, perception-to-action) gap, and therefore, the claim of improving robot manipulation is speculative and not directly obtained by the provided results.\n\n4. The overall setting utilized in the ablation study should be further clarified for easier performance comparison. Besides, it would be better if the author could conduct the ablation study on other tasks (not merely the short video understanding ( video-text retrieval and action recognition).\n\nQUESTIONS:\nThe questions are listed in the weaknesses section. If the author could solve the concerns, I would consider raising my score. Besides, I want to know whether the paper is not written in the full 9 pages.\n\nLIMITATIONS:\nThe model and its entire pretraining data pipeline are highly specialized for egocentric video, with a strong emphasis on hand-object interactions (HOIs). While this leads to state-of-the-art performance in this domain, it raises concerns about the generalizability of the learned representations to third-person video or other video domains that do not feature prominent hands and close-up object manipulation. The model may be overfitting to the specific biases of the egocentric perspective, potentially limiting its utility as a general-purpose video representation model. The authors acknowledge this as a limitation, but its significance as a trade-off for specialization warrants emphasis.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}