Title: CausalVTG: Towards Robust Video Temporal Grounding via Causal Inference
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video Temporal Grounding

Causal Inference

Vision-Language Understanding
Abstract: Video Temporal Grounding (VTG) aims to localize relevant segments in untrimmed videos based on natural language queries and has seen notable progress in recent years.
However, most existing methods suffer from two critical limitations. 
First, they are prone to learning superficial co-occurrence patterns—such as associating specific objects or phrases with certain events—induced by dataset biases, which ultimately degrades their semantic understanding abilities.
Second, they typically assume that relevant segments always exist in the video, an assumption misaligned with real-world scenarios where queried content may be absent.
Fortunately, causal inference offers a natural solution to the above-mentioned issues by disentangling dataset-induced biases and enabling counterfactual reasoning about query relevance. 
To this end, we propose CausalVTG, a novel framework that explicitly integrates causal reasoning into VTG. 
Specifically, we introduce a causality-aware disentangled encoder (CADE) based on front-door adjustment to mitigate confounding biases in visual and textual modalities. To better capture temporal granularity, we design a multi-scale temporal perception module (MSTP) that reconstructs query-conditioned video features at multiple resolutions. 
Additionally, a counterfactual contrastive learning objective is employed to help the model discern whether a query is truly grounded in a video.
Extensive experiments on five widely-used benchmarks demonstrate that CausalVTG outperforms state-of-the-art methods, achieving higher localization precision under stricter IoU thresholds and more accurately identifying whether a query is truly grounded in the video. 
These results demonstrate both the effectiveness and generalizability of proposed CausalVTG.
The code is available at https://github.com/MxLearner/CausalVTG.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes CausalVTG, a novel framework for Video Temporal Grounding that explicitly incorporates causal inference into the model design. By constructing a Structural Causal Model and employing front-door adjustment via a Causality-Aware Disentangled Encoder, the method aims to eliminate dataset-induced confounding biases. Additionally, a Multi-Scale Temporal Perception Module is used to enhance temporal reasoning at different granularities, and a Counterfactual Contrastive Learning objective is introduced to distinguish relevant from irrelevant queries. Extensive experiments on five widely-used VTG benchmarks demonstrate that CausalVTG achieves state-of-the-art performance in both localization precision and query relevance detection tasks. The paper is well-structured and the proposed ideas are clear and well-motivated.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The paper addresses two fundamental but underexplored challenges in VTG — spurious correlations and grounding absence — through a principled causal framework. The introduction of causal inference, particularly front-door adjustment, into the VTG task is novel and fills a critical gap in the literature. The proposed components, including the SCM-based encoder, multi-scale temporal module, and counterfactual learning objective, are coherently designed and theoretically justified. The experimental evaluation is comprehensive, covering multiple benchmarks and metrics, and effectively validates the proposed contributions. The paper is clearly written, well-organized, and presents figures and tables that are informative and supportive of the narrative.\n\nWeaknesses: The ablation study shows that removing the causal module (CADE) causes smaller performance drops compared to other modules, raising concerns about the direct impact of the causal design.The counterfactual learning strategy relies on negative sampling rather than generating challenging counterfactuals, potentially underutilizing the full potential of causal reasoning.The method depends heavily on large-scale pre-trained models, but there is no detailed discussion on computational cost or efficiency compared to existing baselines.\n\nQUESTIONS:\n1. While CADE is conceptually important, its removal causes relatively minor degradation in performance. Could the authors provide more targeted experiments (e.g., under domain shifts or style variations) to better demonstrate its effect?\n2. Have the authors considered generating more challenging or realistic counterfactual samples instead of random negative sampling? Would this further improve the robustness?\n3. The method depends heavily on large-scale pre-trained models, but there is no detailed discussion on computational cost or efficiency compared to existing baselines.\n\nLIMITATIONS:\nYES", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, authors firstly formulate a structural causal model for visual temporal grounding task. They define stylistic variations in visual and textual modalities as confounders. Building on top of the proposed structural causal graph, authors implement CausalVTG and verify its effectiveness on five benchmarks. As a technical summary, authors integrate causality-aware disentangled encoder, multi-scale temporal perception module, and counterfactual contrastive learning objective in the proposed CausalVTG framework, which composes a solid contribution to the community.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The clarity of this paper is good and the paper is easy to follow. This paper is well motivated by two challenges: 1) superficial co-occurrence patterns and 2) assumption that relevant segments always exist.\n\n2. Authors also consider a scenario when the given query is not relevant to the video content, the model should be able to reject grounding any segment. This makes the proposed paper more thorough and suitable for real world application.\n\nWeaknesses\n\n1. No thorough ablation study is provided in main paper. Since the proposed CausalVTG is a combination of CADE, MSTP, QRM and QGR as shown in Table 4 in supplementary material, a detailed ablation study is needed in main paper. A base version of CausalVTG can be base model + CADE, then the integration of any one or two modules/methods given MSTP, QRM, and QGR should be included.\n\n2. Performance gain from CausalVTG is mainly from MSTP, which is not related to causal reasoning.\n\nQUESTIONS:\n1. The first claimed contribution of this paper is causal graph for video temporal grounding task. It would be great to straightforwardly show what is the confounder exactly in main paper. From common understanding of our community, confounder can be something like video background information or some template or pattern in language query. Explicitly show it in paper can enhance the paper quality.\n\n2. Referring to the weakness part, can you include a detailed and thorough ablation study in main paper? You can save much space by making Figure 2 as single-column\n\nLIMITATIONS:\nAuthors discuss limitations of proposed method on page 14 and 15. Other limitations please refer to weakness part.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes CausalVTG, a framework designed to address two major limitations in Video Temporal Grounding (VTG): (1) reliance on superficial co-occurrence patterns due to dataset biases, and (2) inability to handle scenarios where the query content might not be present in the video. The authors employ causal inference, specifically front-door adjustment, to mitigate confounding biases. They introduce a Causality-Aware Disentangled Encoder (CADE) to obtain unbiased modality-specific representations, and a Multi-Scale Temporal Perception (MSTP) module to capture temporal dynamics at multiple granularities. Additionally, the authors incorporate counterfactual contrastive learning to improve the model’s capability to distinguish between grounded and ungrounded queries. Experimental evaluations on **five** established benchmarks demonstrate state-of-the-art results across various settings.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n\n- The paper clearly identifies important, practical limitations in existing VTG methods, and convincingly motivates the application of causal inference.\n\n- The introduction of the causal inference framework, specifically using the front-door adjustment through the proposed CADE, is novel and technically sound. The MSTP module enhances temporal modeling effectively, complementing the causal design.\n\n-  Extensive experiments across five widely-used benchmarks (QVHighlights, Charades-STA, ActivityNet Caption, Charades-RF, ActivityNet-RF) clearly demonstrate improvements, especially under strict IoU thresholds and challenging ungrounded-query scenarios. I would recommend to add YouCookII and the partictions for Charades-STA and ActivityNet from https://arxiv.org/abs/2101.09028\n\n- The paper is very clearly written, methodically structured, and provides comprehensive implementation details and hyperparameters, aiding reproducibility.\n\n### **Weaknesses**\n\n- The combination of multiple intricate modules (CADE, MSTP, counterfactual training) introduces considerable complexity. This complexity may hinder interpretation of performance gains, particularly the exact contribution of each individual component.\n\n- The complexity of multi-scale temporal processing and causal disentanglement is computationally demanding. While runtime details are briefly mentioned, a deeper analysis of inference efficiency or scalability considerations would be valuable.\n\nQUESTIONS:\n## Questions\n- Could you elaborate further on the assumptions underpinning the structural causal model and clarify their appropriateness and limitations specifically for VTG tasks?\n- How sensitive is your framework to the design of mediators? Have alternative mediator designs been explored, and how might different mediator definitions affect grounding performance?\n- Given multiple complex components (CADE, MSTP, counterfactual contrastive learning), could you further clarify which components are most critical? Additional ablations or sensitivity analyses would strengthen the paper significantly.\n- Could you provide more details on computational overhead during inference (e.g., runtime and memory consumption), especially compared to simpler baselines?\n- Can you provide analyses of your predictions similar to https://openaccess.thecvf.com/content/ICCV2023W/CLVL/papers/De_la_Jara_An_Empirical_Study_of_the_Effect_of_Video_Encoders_on_ICCVW_2023_paper.pdf Figure 1? such that we can evaluate potential  bias of the predictions of the model.\n\n## Suggestions\nI have some suggestions and comments that might help further strengthen your paper:\n\n1. **Related Work & Citations (Lines 42-50)**:  \n   It's great to see that you've cited Escorcia et al. (2019). For completeness and richer context, the paper would strongly benefit from citing two additional relevant works:\n   - [DORi (Rodriguez-Opazo et al., WACV 2021)](https://openaccess.thecvf.com/content/WACV2021/html/Rodriguez-Opazo_DORi_Discovering_Object_Relationships_for_Moment_Localization_of_a_Natural_WACV_2021_paper.html) introduces a spatio-temporal graph explicitly designed to reduce correlations between background elements and directly model interactions between objects, humans, and actions, addressing similar concerns about spurious correlations that you highlight.\n   - [\"A closer look at Temporal Sentence Grounding in Videos:\nDataset and Metric\" (ACM MM 2021)](https://dl.acm.org/doi/abs/10.1145/3475723.3484247) systematically analyzes temporal bias in moment localization tasks and provides dataset splits (Charades-STA CD and ActivityNet CD) designed explicitly to measure robustness against these temporal biases.\n\n   Including these references would enhance your motivation and clarify the novelty and contributions of your approach relative to existing work.\n\n\n2. **Multiple Occurrences and Temporal Causality**:  \n   An interesting scenario that your method might help analyze further relates to queries that describe events occurring multiple times within the same video.  \n   - For instance, as is mentioned in https://arxiv.org/abs/1809.01337 there are queries that has temporal dependency and would be great to explore and quantify\n- Also,**Figure 8** of DORi’s supplemental material, the Charades-STA query *\"person walks over to the refrigerator open it up\"* occurs twice. This raises the important practical scenario where a query might occur **multiple times**, **one**, or **not at all** in a video.  \n   - Similarly, the query examples from ActivityNet illustrated in **Figure 3** of the supplemental material from [\"Memory-efficient Temporal Moment Localization in Long Videos\" (EACL 2023)](https://aclanthology.org/2023.eacl-main.140/) highlight temporal causality and continuity, as in *\"The man continues...\"*. These are examples that explicitly involve temporal causal reasoning.\n\n   Your causal grounding method could show excellent performance for that kind of challenging queries to better showcase its strengths and contributions. An explicit analysis or discussion would significantly strengthen the practical motivation of your work.\n\n3. **Generalization and Causal Dependencies (YouCook2 dataset)**:  \n   It would be highly valuable and insightful to see your causal inference-based grounding applied to a dataset like **YouCook2**, where the causality of an action depends directly on previously executed actions.  \n   - For instance, Figures 13 and 14 of the supplemental material from DORi indicate specific cases where models incorrectly learn spurious correlations (e.g., associating certain ingredients or actions like *\"pouring dressing\"* or *\"adding oil\"* with incorrect moments due to biases in object presence or appearance).  \n   \n   Demonstrating your method’s robustness against such spurious correlations and temporal dependencies using YouCook2 would substantially strengthen the empirical validation and show clearer evidence of the claimed causal benefits.\n\nI encourage you to address these suggestion, as they would greatly enhance the quality, depth, and impact of your contribution.\n\nLIMITATIONS:\nYes, the paper discusses limitations clearly. However, additional reflections on the complexity, computational efficiency, and assumptions in causal modeling would further strengthen this section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper targets the task of Video Temporal Grounding (VTG) and proposes the causal-inference framework CausalVTG. The method has three key components:\n1.Causality-Aware Disentangled Encoder (CADE) based on front-door adjustment to eliminate confounding bias between visual and textual modalities;\n2.Multi-Scale Temporal Perception (MSTP) to adaptively capture actions of different temporal lengths;\n3.Counterfactual Contrastive Learning to judge whether a query can truly be grounded.\nThe unified model simultaneously addresses Moment Retrieval, Highlight Detection, and Query Relevance. It achieves state-of-the-art results on five benchmarks—QVHighlights, Charades-STA, ActivityNet-Caption, Charades-RF, and ActivityNet-RF—showing especially strong performance under strict IoU thresholds and in “false-query” scenarios.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1.First to systematically introduce structured causal modeling, front-door adjustment, and counterfactual contrastive learning into VTG, effectively mitigating vision/text confounding and improving interpretability and generalization.\n2.Builds a single framework that performs Moment Retrieval, Highlight Detection, and Query Relevance in one pass, avoiding error accumulation in pipeline systems and keeping the overall design concise.\n3.Achieves top accuracy of 84.78 / 89.20 Acc on Charades-RF and ActivityNet-RF “false-query” tests, outperforming existing methods.\n\nWeaknesses\n1. The experimental section does not include comparisons with other causal-inference-based VTG methods, making it difficult to judge whether the proposed front-door adjustment combined with counterfactual contrast truly offers an advantage over alternative causal strategies.\n2.The model assumes that a K-means–derived mediator satisfies the front-door criterion, yet provides no evidence on how the choice of cluster number K or clustering stability affects performance. Without such sensitivity analysis, the claimed causal robustness remains speculative.\n3. The paper does not report mean ± std or statistical significance tests (e.g., p-values) for the main metrics, so it is difficult to gauge the reliability of the reported performance gains.\n\nQUESTIONS:\n1. Could you include at least one representative causal baseline in the comparison tables (or explain why this is not feasible) to clarify the benefit of your causal strategy?\n\n2. How sensitive is CADE to the K-means cluster number K and clustering randomness? An ablation or plot would help justify the causal robustness claim.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}