CausalVTG: Towards Robust Video Temporal Grounding via Causal Inference
Authors: Qiyi Wang
Senda Chen
Ying Shen
Venue: NeurIPS 2025 poster ()
Keywords: Video Temporal Grounding
Causal Inference
Vision-Language Understanding
Primary area: applications
TL;DR: We propose a causal framework for video temporal grounding that mitigates confounding biases and improves robustness to linguistic variations and irrelevant queries.

Abstract: Video Temporal Grounding (VTG) aims to localize relevant segments in untrimmed videos based on natural language queries and has seen notable progress in recent years.
However, most existing methods suffer from two critical limitations. 
First, they are prone to learning superficial co-occurrence patterns—such as associating specific objects or phrases with certain events—induced by dataset biases, which ultimately degrades their semantic understanding abilities.
Second, they typically assume that relevant segments always exist in the video, an assumption misaligned with real-world scenarios where queried content may be absent.
Fortunately, causal inference offers a natural solution to the above-mentioned issues by disentangling dataset-induced biases and enabling counterfactual reasoning about query relevance. 
To this end, we propose CausalVTG, a novel framework that explicitly integrates causal reasoning into VTG. 
Specifically, we introduce a causality-aware disentangled encoder (CADE) based on front-door adjustment to mitigate confounding biases in visual and textual modalities. To better capture temporal granularity, we design a multi-scale temporal perception module (MSTP) that reconstructs query-conditioned video features at multiple resolutions. 
Additionally, a counterfactual contrastive learning objective is employed to help the model discern whether a query is truly grounded in a video.
Extensive experiments on five widely-used benchmarks demonstrate that CausalVTG outperforms state-of-the-art methods, achieving higher localization precision under stricter IoU thresholds and more accurately identifying whether a query is truly grounded in the video. 
These results demonstrate both the effectiveness and generalizability of proposed CausalVTG.
The code is available at https://github.com/MxLearner/CausalVTG.

CausalVTG: Towards Robust Video Temporal Grounding via Causal Inference
Qiyi Wang
Senda Chen Tongji University 2411498@tongji. edu. en
Ying Shen' Tongji University
Tongji University wgy126179@tonngi. edu.cn
yingshen@tonugi edu. cn
Abstract
Video Temporal Grounding (VTG) aims to localize relevant segments in untrimmed videos based on natural language queries and has seen notable progress in recent years. However, most existing methods suffer from two critical limitations. First, they are prone to learning superficial co-occurrence patterns- such as associating specific objects or phrases with certain events-induced by dataset biases, which ultimately degrades their semantic understanding abilities. Second, they typically assume that relevant segments always exist the video, assumption misaligned with real-world scenarios where queried content may be absent. Fortunately, causal inference offers natural solution the above-mentioned issues by disentangling dataset-induced biases and enabling counterfactual reasoning about query relevance. To this end, we propose CausalVTG, novel framework that explicitly integrates causal reasoning into VTG. Specifically, we introduce a causality gled encoder (CADE) based on front-door adjustment to mitigate confounding biases in visual and textual modalities. To better capture temporal granularity, we design a multi-scale temporal perception module (MSTP) that reconstructs query- conditioned video features at multiple resolutions. Additionally, counterfactual contrastive learning objective is employed to help the model discern whether a query is truly grounded in video. Extensive experiments on five widely-used benchmarks demonstrate that CausalVTG outperforms state-of-the-ee methods, achieving higher localization precision under stricter IoU thresholds and more accurately identifying whether a query is truly grounded in the vidco. These results demonstrate both the effectiveness and generalizability of proposed CausalVTG. The code is available at https: //github......t com wom//MsLeaanner/aaaa Caa.aaalreraaaaa....a
disentan-
Introduction
With the development of modern Internet and the popularity of video-sharing platforms, videos have surged exponentially, providing vital medium for information exchange in entertainment, education, and news. This rapid expansion has given rise challenges in efficient video browsing and retrieval. Video Temporal Grounding (VTG), which aims localize video segments that semantically correspond to natural language queries, has become a core task in vision- language understanding [1, 2]. It encompasses two representative sub-tasks: Moment Retrieval (MR) [3. 4, 5], which focuses predicting precise temporal boundaries the target segment, and Highlight Detection (HD) [4, 6]. which estimates frame- wise saliency scores to identify the most informative and representative vidco portions. The applications of VTG enhance video content utilization and satisfy the escalating demands for efficient video content acquisition [4, 7, 8].
Moment-DETR the pioneer work proposed for jointly solving MR and HD tasks along with unified evaluation benchmark QVHighlights constructed by the authors [4]. Building upon these
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Queryl: A person if washing their hands in the sink.
07 Howel
Query2: A person washes their hands in the kitchen sink
Coaetrit Prefictonn
Query3: person Eurns light.
Leds
Ceaetiitt Preditton
Sunting Protition
Figure 1: Qualitative examples illustrating two key limitations of existing temporal grounding models. Queryl: I: successful video grounding example for both VGT models. Query2: a minor rephrasing of the query leads incorrect localization of R2-Tuning due to reliance on superficial co-occurrence cues (e.g., "kitchen"). Query3: the of -2-Tunnng- Tuning model erroneously predicts segment for a query describing an unseen action, reflecting its inability to assess query relevance. The proposed CausalVTG, better captures temporal causal structures and abstains when appropriate.
efforts, a number of subsequent studies have further advanced the field, notably by leveraging lcage-scale pre-trained models [4, 5, 9] and incorporating adapter-basee techniques [10] to enhance cross-modal alignment. However, the majority of efforts remain centered on improving video- text alignment through increasingly elaborate fusion strategies [11, 12. 13], often overlooking inherent biases embedded in training datasets [4, 14]. These biases include stylistic variations, such as visual decoration patterns and linguistic phrasings [15]. Such factors can lead models to overfit to dataset-specific cues instance, correlating background styles or frequent wordings with particular evther- than learning genuine semantic associations. This spurious reliance ultimately compromises the robustness and generalization ability of temporal grounding systems. For example in Figure 1, given Query1 "A person is washing their hands in the sink," the existing model successfully retrieves the correct moment. Yet, minor rephrasing as "A person washes their hands in the kitchen sink" leads incorrect localization of R -unnnng [10] model on later segment where the person is drying their hands. This failure illustrates how the model can be misled by superficial visual-textual cues (e.g., "kitchen") instead of reasoning about the actual temporal causal structure of events. Moreover, most existing models operate under the assumption that relevant segments always exist in the video, making them incapable of identifying when query is actually ungrounded. Given Query3 "A person turns on a light," the R--Tuning model still returns the above segment based on the presence of light source, even though the described action does not occur. These limitations reflect common underlying issue: current approaches are fundamentally correlation-driven and lack the ability to reason about causality, which essential for robust video temporal grounding.
To address these limitations, we resort to causal inference to tackle these challenges. By explicitly modeling the underlying data-generating process, causal inference methods can disentangle spurious correlations and enable counterfactual reasoning- wllowinng models to focus on genuine semantic causality and assess whether a query is truly grounded in a video. Causal inference [16, 17] has shown promise across machine learning tasks, including image recognition [18, 19], visual question answering [20, 21], and vision-and-lane language navigation [22, 15). These methods typically construct structural causal model [23] to explicitly represent the relationships among variables, and apply causal tools such as the do-operator, which simulates interventions to estimate causal effects beyond observational correlations. In addition, techniques like front-door and buck-door adjustments are used to control for latent confondders-eilee influence, or by blocking spurious dependencies introduced by common causes. These mechanisms enable the model to better capture true causal relationships, rather than being misled by stylistic artifacts or co-occurrence patterns. Motivated by these successful applications, we investigate how causal principles can be leveraged in VTG, particularly under stylistic variation and uncertain query relevance, two scenarios where conventional methods struggle.
by leveraging mediating variables that transmit causal
To accomplish these goals, we firstly design structural causal model (SCM) specifically tailored to the VTG task to reveal in-depth causal relations, as illustrated in Figure 2. In this model, we explicitly identify stylistic variations, such as visual decoration patterns or linguistic phrasing, as confounding factors that simultancously influence both visual and textual modalities. To mitigate their

---

impact, we introduce latent mediators that encode core semantic information from both modalities. These mediators enable the application of front-door adjustment, which reduces the influence of confounding biases by leveraging the causal pathway from inputs to grounding outcomes through the mediators. Secondly, building on SCM, we design a causality-aware disentangled encoder (CADE) that performs modality-specific front-door adjustments [16] on video and query features. CADE employs mediator-guided attention mechanisms to extract causally disentangled and unbiased representations for each modality, encouraging the model to find out true semantic causalities rather than superficial co-occurrence patterns. Thirdly, to further enhance temporal reasoning, incorporate multi-scale temporal perception module (MSTP), which reconstructs video representations conditioned on the query and projects them across multiple temporal resolutions. This design enables the model to capture both fine- ain-igrieed and long-range event dynamics, improving localization performance for events of varying durations. Lastly, we introduce a counterfactual contrastive learning objective [16] to complement the standard VTG training paradigm. This objective enforces discrimination between observed and counterfactual representations at the video level, allowing the model to determine with greater reliability whether a query is semantically grounded in the video. Taken together, these components form a principled and interpretable framework for moment retrieval and highlight detection, offering enhanced robustness against confounding biases and linguistic variations.
Our main contributions are summarized as follows:
We formulate a structural causal model for VTG that identifies stylistic variations in visual and textual modalities as confounders. By introducing latent mediators and applying front- door adjustment to block spurious dependencies between video/query input and grounding outcomes, our approach establishes a principled causal foundation that improves model robustness and generalization.
Building on this causal formulation, we design CVu, a unified framework that jointly addresses moment retrieval and highlight detection. Central to our architecture is causality- aware disentangled encoder which applies front-door adjustment to learn unbiased and semantically meaningful representations for video and query modalities independently. Additionally, we propose a multi-scale temporal perception module to capture hierarchical temporal dynamics, and incorporate counterfactual contrastive learning objective that enables the model to reliably determine query relevance, improving its ability to reject ungrounded queries.
We conduct extensive experiments on five widely-used benchmark datasets, demonstrating that CausalVTG consistently outperforms state-of-the-art baselines. Our method achieves significantly higher localization precision under strict IoU thresholds and exhibits superior robustness in discerning whether queries are truly grounded in videos. validating both the effectiveness and generalizability of our causal modeling approach.
2
Related Work
Video Temporal Grounding VTG focuses on identifying video segments corresponding to nat-
ural language queries, encompassing two primary sub-tasks: Moment Retrieval [3, 4, 24, 14, 5] and Highlight Detection [4, 6, 9, 5J. The introduction of the QVHigglights benchmark [4] unified the evaluation of MR and HD tasks and provided baseline method, Moment-DETR. Subsequent approaches, such as UniVTG [9] and UMT [5], leveraged large-scale vision-language pre-training to enhance model performance. Methods like QD-DETR [11] and CG-DETR [13] emphasized sophisticated cross-modal fusion strategies to improve semantic alignment between video and textual inputs. Approaches including UnLoc [25] and R [10] utilized
and transfer
learning to adapt CLIP-based models specifically for VTG tasks, whereas LLMEPET [26] integrated large language model encoders into traditional VTG architectures. Recognizing the unrealistic as- sumption of -rouuddd queries, the Charades-RF and ActivityNet-RF datasets [27] introduced scenarios with potentially irrelevant queries. A robust transformer-based framework RaTSG [27] is proposed to explicitly address these false-query situations. However, these models remain fundamen- tally correlation-driven, vulnerable to confounding biases arising from subtle linguistic and stylistic variations.
3

---

Causal Inference in Video Understanding Causal inference has emerged as an influential framework for improving generalization visual understanding by explicitly modeling the underlying data- generating processes and mitigating spurious correlations [16, 23]. Compared to traditional debiasing methods [28, 29], causal approaches leverage counterfactual reasoning to robustly handle domain shifts and biases. In recent computer vision research, causal inference techniques have been applied across tasks such as image classification [18. 19, 30, 31, 32), visual question answering [20, 21, 33), and vision-language navigation [22, 15]. Specifically within the VTG context, DCM [14] utilized back-door adjustments to mitigate temporal biases video data, while IVG [24] employed both back-door intervention and dual contrastive learning strategies to disentangle misleading visual- textual associations. Beyond VTG, causal inference has also demonstrated effectiveness in video summarization [34] and video captioning [35]. These causal modeling strategies underpin the development of structured causal frameworks for robustly addressing confounding factors such as stylistic variations and irrelevant query relevance in video grounding tasks.
3 Preliminary
3.1 Task Formulation
Given an input video y= (v composed Lg clips, where each clip v; E Po represented by De-dimensional feature vector, and natural language query tokens, each with dimension Dg. the VTG task comprises three sub-tasks:
Ty consisting of Tug (q
Moment Retrieval The model predicts temporal segments in the video that semantically align with the query. Since multiple segments may match single query, the output is set of predicted moments represented as {bb,,., be.i, C;) (:)]² N=z where b,.i and be.i denote the start and end timestamps of the i-th moment, respectively, and o is the associated confidence score for ranking.
Highlight Detection The model estimates the semantic relevance between each clip v, and the query Q by assigning saliency score s; [0, 1]. Higher values indicate greater relevance.
Query Relevance Prediction (QR) To handle scenarios where the query may not correspond to any content within the video, the model further predicts a query-level relevance score E 0, 1). This score quantifies the confidence that the query is grounded in the video, with higher scores indicating higher certainty of relevance, and lower scores reflecting ambiguity or irrelevance.
3.2 Structural Causal Model of VTG
As illustrated in Figure 2, we formulate a structural causal model [16] tailored for the VTG task. Specifically, we denote the input video as V, the natural language query as Q, and the grounding outcome as V, collectively defining inputs as X' = (V. Q}. In this directed acyclic graph, edges represent causal relationships, with input variables X' causing grounding predictions y. Traditional VTG methods predominantly model observational distributions P()")X") neglecting confounding biases introduced via back-door paths X + Z + y. Confounders (Z), such as subtle stylistic variations (c.g., visual decoration patterns in videos and linguistic phrasing variations in queries), simultaneously influence both the input data distribution and grounding outcomes. The presence of these confounding factors inevitably creates spurious correlations, adversely impacting model robustness and generalization.
Video
Query
y
Grounding
Confounder
M
Mediator
Figure 2: Illustration of the structural causal model of VTG.
Since confounders typically comprise intricate and difficult-to-explicttyy characterize patterns, we introduce intermediate mediator variables M, establishing explicit front-door causal pathways X' - M + y. This enables effective causal intervention and mitigates the influence of unobserved confounders. The detailed mechanisms for causal adjustment are described in subsequent sections.

---

Cousality-Awrre Disentangled
Multi-Scale Temporal Perception
In-Semplin
Sampling
Refiament
Rondon Complng
Video
K-
Pongrrund
A person washing their hands the sink.
Langaga
Global Dicrionary
Salieny Scores
Query
Sompling
Query
In-Sompling
Prediction Heads
Figure 3: Overview of the CausalVTG framework. Given a video and natural language query. sppelit encoders extract features and perform front-door adjustment a causality- disentangled encoder to mitigate confounding bias. A query-guided refinement module fuses con- textual signals, followed by a multi-scale temporal perception module that captures hierarchical dynamics. Prediction heads jointly perform highlight detection, moment retrieval, and query rele- vance estimation under a counterfactual training paradigm.
4 Methodology
4.1 -ausality-Aware Disentangled Encoder
Traditional VTG models typically learn the observational distribution P x) by directly modeling the correlation between input X' = [[V. 2) and output V. This process can be expressed as:
P(J = z)
Σ
ml = z),
(1)
where M denotes set of latent mediators that encode high-level semantic information [36] from X. However, such observational modeling fails to climinate the influence of confounders Z, which introduce spurious correlations via back-door paths X' Z y.
To mitigate this, we adopt front-door adjustment based on causal intervention using the do- operator[16]. Specifically, we introduce a causality- mediators through modality-specific encoders. For each modality (video or text), the encoder computes the mediator representation high-level semantic features. Subsequently, the front-door adjustment leverages these mediators to isolate the influence of confounders, enabling the model to focus genuine causal effects. This constructs causal path + M + y that permits front-door estimation of the interventional distribution:
disentangled encoder that generates
f(x) using self-attention module [37], which extracts
P(y|do(X' = =x))= = Σ P mm)) = m//o(X' x))
M=m
Σ P(z')
Σ P(M= x)) =
M=
Ex-=x BM P()
(2)
where x' denotes inputs sampled from the empirical training distribution via K-means [38], and m is the mediator representation computed from the current input
To approximate the front-door adjustment, we define a causal feature function modeled linear combination of sampled inputs and mediators [15, 39];
Fl),mm) = Ex Meei m), f(z,m) = h(x') g(m),
(3)
which yields the separable form:
F(r', m) Ex=x(h<x')) E EM
(4)
5

---

We estimate the two expected terms using attention based mechanisms [40, 14]. Let 2 and m, be sampled input and mediator representation from imput and mediator distributions, respectively. The expectations are computed as:
exp(p" T xx))
Ex=r [h(x')] = Σ ash(1a).
Σn exp(p' T x//)) a
(5)
exply T m2) Σ 3eg(m))) Bj Σn explq ma)
E [M
(6)
where p p(x) and g ((x) are attention queries derived from the current input guide the aggregation of sampled features and mediators.
Through this modality-specific front-door adjustment, obtain causally disentangled representations for both video and language inputs. These representations are explicitly purified from dataset-induced confounding effects, enabling the model focus genuine semantic structures rather than superficial correlations. This serves as a robust foundation for downstream grounding tasks.
4.2 Multi-Scale Temporal Perception Module
To enhance the temporal modeling capacity of video features, design two-branch perception module comprising: (1) a guery-guided refinement and (2) a multi-scale temporal feature pyramid.
Query-Guided Refinement Given an input video embedding V = [Vi...., V E R with T
clips and a query embedding Q = [q,,.... "ll.....9p 4r.] RLLD with I. tokens, we first compute a similarity matrix S E R RTL between video and query features using a trilinear attention function [41]:
S.j v, we + 9, wQ + (v; Ww) qj.
where Wa, WQ. WM E D are learnable parameters and denotes element multiplication.
Based on the similarity matrix S, we compute context-to context-to-query and
attention as:
a; (Q)
softmax,(S..) qi a (V)
Σ (Sft Vj-
8
We then enhance each video clip v, by concatenating its interactions with the attended query and video representations:
V. = Conv ID [v.. (00) vi v woal"])
where v; E IPP the refined feature of the i-th video clip.
This query-guided refinement enables the model to highlight semantically relevant clips and suppress irrelevant noise by embedding the query context directly into the video representation, thereby laying a solid foundation for subsequent temporal reasoning.
Multi-Scale Temporal Feature Pyramid To capture hierarchical temporal patterns across different time granularities, we adopt a multi-scale feature pyramid to refine video representations. Given the query-guided video embedding V = [ r E RTxD, where T is the number of video clips we construct of temporal feature sequences varying resolutions.
Formally, let S = {81, 82, .2....,,88 {x} .2. denote a set of temporal strides, where each 88 represents the downsampling rate for the k-th scale. For each scale 8 S. we apply series of temporal convolution layers to obtain:
(k) Conv sx (V) E T//
(10)
where (k) is the video representation at the k-th temporal scale.
Each temporal convolution operates along the time dimension and reduces the resolution of the video sequence by factor of sş- The resulting set of multi-resolution video features Vpyr (V(1), ( V(2)..., ...,, V(K) forms a hierarchical representation that captures both fine-grained actions and long-tern temporal dependencies.

---

4.3
Prediction Heads
We design three task-specific prediction heads perform highlight detection, moment retrieval, and query relevance prediction. These modules operate on either the refined video features V or the multi-scale temporal representations Ppr.
Highlight Detection To measure the semantic alignment between video clips and the query, we first apply adaptive average pooling to the query sequence to obtain global vector: quda = AdaPool( Q) e RD. Then, for each video clip V. E RD,, we compute its relevance score via cosine similarity:
V. Gada
i
i..... T.
(11)
[ell
The saliency scores {si} trained using the SampledNCE loss [42], where positives correspond informative clips and negatives to unrelated clips.
Moment Retrieval We leverage the multi-scale video features Vpyr (Va),... V(K) to localize temporal moments. For each scale V(k) E R2xxD, two parallel ID convolutional heads are applied: one for span regression and the other for foreground classification. Specifically, the Conv span
head predicts the relative temporal offsets (R) [dittrr, d! for each clip i, and the Conv DDr head
estimates its foreground confidence score s!AA) E
b Conv D
E IR",
((()) = Conv Dig
E.R. E..
(12)
The predicted span boundaries bi (k) are supervised using L1 loss, while the foreground scores l!a)
are optimized using focal loss [43] to distinguish query-relevant (foreground) clips from background.
Query Relevance via Counterfactual Reasoning To enable query-aware abstention when no grounding evidence exists, we adopt a counterfactual learning strategy. During training, we construct negative samples by pairing video with query that is not semantically grounded in it. For both valid and invalid pairs, we compute a joint representation by concatenating the adaptively pooled query and video features:
Vada AdaPool( V), r Sigmod ([Vaa qodul).
(13)
The predicted score E J0, 1] indicates whether the query is relevant to the video, and is optimized using a Dynamic Binary Cross cooss-Entropy loss [44].
4.4 Training & Inference
The model is jointly trained with three task-specific objectives. For the HD task, we supervise the clip-wise saliency scores {s:} using the SampledNCE loss [42] Las- For the MR task, the predicted boundary offsets (b. (b())) are optimized with L1 loss Lmrs while the foreground confidence scores (f!(() are trained with focal loss [43] Lig- For the QR task, loss [44] fr to supervise the scalar prediction r that indicates whether the query is grounded the
a Dynamic Binary Cross-Entropy
video. The overall loss is: = Apahha + AmchCm + >rgLig + Agr
(14)
During inference, the saliency scores {si} directly indicate clip-level highlight relevance. For MR, each predicted span biah (k) is converted into temporal interval (b,,;, be.)) with confidence score C, = o we perform Non-Maximum Suppression (NMS) [45] to obtain final ranked results.
The query relevance score E [0, 1] determines whether to output any grounding result, enabling the model to abstain in irrelevant cases.
Experiments
5.1 Datasets & Evaluation Metrics
Experiments are conducted on five benchmarks: QVHighlights [4] annotated for both MR and HD, serving as comprehensive benchmark for multi-task evaluation; Charades-STA [3] and ActivityNet-
7

---

Table 1: Performance comparison the QVHighlights test set for joint MR and HD tasks.
MR
HD
Method
RI @0.5
@1@0.7
mAP Avg.
mAP@0.5
mAP@0.75
mAP
HIT@
MCN [48]
11.41
2.72
10.67
24.94
8.22
CAL [49]
25.49
11.54
9.89
23.40
7.65
XML [50]
41.83
30.35
32.14
44.63
31.73
34.49
55.25
XML+ [4]
46.69
33.46
34.90
47.89
34.67
35.38
55.06
Moment-DETR [4]
52.89
33.02
30.73
54.82
29.40
35.69
55.60
UMT [5]
56.23
41.18
36.12
53.83
37.01
38.18
59.99
MomentDiff [51]
58.21
41.48
36.84
54.57
37.21
QD-DETR [11]
62.40
44.98
39.86
62.52
39.88
38.94
62.40
MH-DETR [12]
60.05
42.48
38.38
60.75
38.13
38.22
60.51
UniVTG [9]
58.86
40.86
35,47
57.60
35.59
38.20
60.96
TR-DETR [52]
64.66
48.96
42.62
63.98
43.73
39.91
63.42
CG-DETR [13]
65.43
48.38
42.86
64.51
45.77
40.33
66.21
Rº_ R--Tuning [10]
68.03
49.35
46.17
69.04
47.56
40.75
64.20
CausalVTG (Ours)
68.87
52.53
49.63
70.70
51.77
40.66
65.63
Caption [46] annotated with precise temporal segments for MR; and Charades-RE RF and ActivityNet- RF [27] extend their original datasets by introducing false-query scenarios for evaluating whether the query is grounded.
For evaluation metrics, we follow previous work [10, 9]. On QVHighlights, we report performance on both MR and HD. For the MR task, we use mean Average Precision (mAP) at Intersection- over-Union (IoU) thresholds OIo {0.5, 0.75), mAP averaged over thresholds from 0.5 to 0.95 (with step size of 0.05), and Recall at Oja E {0.3, 0.5, 0.7]. For the HD task, we adopt mAP and HIT @ 1, considering a clip as positive if labeled as "Very Good", On Charades-STA and SetivityNet-Caption
evaluate MR performance using Recall mean IoU (mloU). On Charades-Ro RF and ActivityNet-RE we follow the dataset splits defined in [27] and use Recall @ at Plo {0.3, 0.5, 0.7), mloU, and grounding accuracy (Acc) to measure both localization quality and the model's 's ability to identify whether a query is truly grounded the video.
t E {0.3, 0.5, 0.7) and
Implementation Details
All models are implemented using PyTorch and trained for 50 epochs single NVIDIA RTX 4070 SUPER GPU (12GB VRAM, 32GB RAM), Intern Video2-CLIP [47] serves as the unified backbone for both video and text encoding. with all hidden dimensions set 256. The temporal stride set for the multi-scale fcature pyramid is S {1. 2, 4, 8} for standard-lengut videos, and extended to {1,2 4, 8, 16) for longer videos in ActivityNet-Cappe -aptton and ActivityNet-RE The training objective a weighted combination of four losses, with coefficients Am 0.1, Amr = 0.2, X 1.0, and Aye 0.1. Training the QVHighlihhts dataset takes approximately 50 minutes.
5.3 Comparison with State-of-the-Arts
Joint Moment Retrieval and Highlight Detection on QVHighlights We evaluate our model on the QVHighlights test set for the joint tasks of MR and HD, with results summarized in Table 1. CausalVTG achieves state-of saae--of-th-aaee performance across all MR metrics, including notable improve- ments in RI @0.7 and mAP 0.75, indicating stronger robustness under strict temporal alignment. We attribute these gains to our causal modeling design, which mitigates the influence of spurious correlations. Our method also performs competitively on HD, demonstrating the effectivenesss of unified, causally grounded temporal grounding.
Moment Retrieval on Charades-STA and ActivityNet-Capp Caption We conduct moment retrieval experiments on Charades-STA and ActivityNet-Caation with results shown Table 2. VTG achieves the best performance across all metrics on both datasets. Notably, on Charades-STA, where each video segment annotated with multiple queries exhibiting diverse linguistic expressions-aee model demonstrates substantial improvements. We attribute this to the causal modeling framework.

---

Table 2: Moment retrieval results Charades-STA. and ActivityNet-Caption test sets.
Charades-STi STA
ActivityNet Captions
Method
RI @0.3
@0.55
@0.7
mloU
@0.3
RI @00.5
@0.7
mloU
VSLNet [53]
67.47
54.62
35.43
49.37
62.12
43.76
25.64
44.54
SeqPAN [54]
70.70
59.14
41.02
52.32
63.71
45.31
26.69
45.73
TCN+DCM [14]
55.8
34.4
48.7
44.9
27.7
43.3
DORi [55]
72.72
59.65
40.56
53.28
57.89
41.35
26.41
42.79
EAMAT [56]
74.25
61.18
41.72
54.53
62.20
41.60
24.14
44.15
ADPN [57]
71.24
56.88
39.73
51.96
61.46
41.49
24.78
44.12
QD-DETR [11]
70.32
58.92
38.54
50.62
62.20
41.60
24.14
44.15
UniVTG [9]
71.62
60.06
33.34
49.92
61.78
43.34
22.59
42.71
Re-Tuning [10]
70.91
59.78
37.02
50.86
RaTSG [27]
74.19
56.61
37.47
53.02
61.46
42.36
24.74
43.72
CTu (Ours)
81.37
70.89
49.25
59.96
64.44
45.62
26.28
45.74
Table 3: Performance comparison on Charades-RF and ActivityNet-RF datasets.
Charades-RR RF
ActivityNet RF
Method
Acc
RI @0.3
R005
Ri @0.7
mloU
Acc
RI 0.33
RI R055
RI @0.7
mloU
VSLNet [53]
50.00
33.74
27.31
17.72
24.69
50.00
31.06
21.88
12.82
22.27
UniVTG [9]
50.00
35.81
30.03
16.67
24.96
50.00
30.89
21.67
11.29
21.35
QD-DETR [11]
50.00
35.16
29.46
19.27
25.31
50.00
26.50
19.15
11.07
18.99
ADPN [57]
50.00
35.62
28.44
19.87
25.98
50.00
30.72
20.74
12.38
22.05
SeqPAN [54]
50.00
35.35
29.57
20.51
26.14
50.00
31.85
22.65
13.34
22.86
EAMAT [56]
50.00
37.12
30.59
20.86
27.27
50.00
31.10
20.80
12.07
22.07
VSLNet+
71.94
61.40
56.77
49.65
54.67
81.60
66.15
58.37
50.64
58.65
UniVTG+
71.94
62.58
58.55
48.79
54.65
81.60
66.15
58.36
49.46
58.00
QD-DETR+
71.94
62.18
58.20
50.96
55.13
81.60
62.43
56.13
49.27
55.97
+
71.94
62.26
$7.23
51.16
55.41
81.60
65.85
57.41
50.28
58.47
S
71.94
62.12
58.01
51.61
55.49
81.60
66.77
58.98
51.11
59.11
EAMAT
71.94
63.55
59.17
51.96
56.23
81.60
66.13
57.36
49.93
58.45
RaTSG [27]
76.85
68.17
61.91
54.19
59.93
84.27
69.02
60.68
52.88
61.15
CausalVTG (Ours)
84.78
76.22
71.07
61.03
67.86
89.20
72.70
63.62
54.64
63.82
which effectively mitigates the impact of stylistic variations and enhances temporal grounding accuracy.
Moment Retrieval with Query Relevance on Charades-RF and ActivityNet-RF We evaluate our model on Charades-RF and ActivityNet-RF which introduce ungrounded queries to test model robustness. As prior methods assume all queries are grounded, they yield a chance-level accuracy of 50% under the balanced (1:1) positive-negative split. For fair comparison, we implement the +- versions baselines by equipping them with trivial relevance discriminator following previous studies [27]. RaTSG represents the current state-of-ther th-trr--tt. As shown in Table 3, CausalVTG outperforms all baselines by large margin across both accuracy and localization metrics. These improvements highlight the effectivvness of our causal modeling in identifying true grounding conditions and rejecting spurious matches.
5.4 Ablation Study
To thoroughly evaluate the contribution of each component in CausalVTG, we conduct comprehen- sive ablation study on the QVHighlights validation set. The analysis decomposes CausalVTG into its four core modules: the Causality- Aware Disentangled Encoder (CADE), Query-Guided Refinement (QGR), Multi-Scale Temporal Perception (MSTP), and Query Relevance Module (QRM). Starting from a baseline model without these modules, progressively integrate each component individually and jointly. As shown Table 4, CADE and MSTP yield the most significant gains, demonstrating the effectiveness of causal disentanglement and temporal multi-scale modeling. The addition of

---

QGR further enhances rideo-languas alignment, while QRM improves the model's ability to handle irrelevant queries. These results validate the complementary benefits of each proposed module.
Table 4: Comprehensive ablation study on the QVHighlights validation set. Each configuration shows
the incremental integration of CADE (Causality-Awe) Disentangled Encoder), QGR QQuery-Guides Refinement), MSTP (Multi-Scale Temporal Perception), and QRM (Query Relevance Module).
Modules
Performance Metrics
Model Variant
CADE QGR MSTP
QRM
RI @10.5.
RI @07.77
mAP@0.5
mAP@0.75
Avg. mAP
57.74
36.52
58.96
35.36
35, 19
(b)
59.94
39.55
60.22
37.43
36.47
(c)
60.39
38.52
60.39
37.05
36,61
(d)
67.68
51.68
69.48
51.33
47.85
60.77
39.16
61.32
36,98
36.87
(
61.32
38.58
61.65
37.22
37.11
g
68.58
52.71
69.69
50.89
48.99
(h)
62.19
40.05
61.66
37.86
37.42
1
68.13
52.90
69.95
52.00
49.54
70.26
54.32
71.34
52.67
50.15
(k)
70.84
56.00
72.17
53.79
50.98
6
Conclusion
This paper introduces CausalVTG, a causal framework for Video Temporal Grounding that addresses the limitations of existing correlation-driven methods. By modeling structural causal relationships among video, query, and grounding outcomes, the framework effectively mitigates spurious correla- tions caused by dataset biases. Key components include causality- disentangled encoder for front-door adjustment, a multi-scale temporal perception module for capturing hierarchical dynamics, and counterfactual contrastive learning objective for reliable query relevance estimation. Experi- mental results across five benchmarks confirm the effectiveness and generalizability of the proposed approach in both standard and challenging grounding scenarios.
Acknowledgments
This work was supported in part by the National Natural Science Foundation of China under Grant 62476202 and 62272343, in part by Shanghai Pujiang Program (No. 23PJ1412700), in part by the Fundamental Research Funds for the Central Universities.
References
[1] Max Bain, Arsha Nagrani, Güi Varol, and Andrew Zisserman. clip-hitchhiker's guide to long video retrieval. arXiv preprint arXiv:2205.0888 2022.
[2] Siteng Huang, Biao Gong, Yulin Pan, Jianwen Jiang, Yiliang Lv, Yuyuan Li, and Donglin Wang. Vop: Text-video co-operative prompt tuning for cross-mood moss-modal retrieval. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 6565-6574 2023.
[3] Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In Proceedings of the IEEE international conference computer vision, pages 5267-5275, 2017.
[4] Jie Lei, Tamara Berg, and Mohit Bansal. Detecting moments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems, 34:1 14:11886-11888 2021.
[5] Ye Liu, Siyuan Li, Yang Wu, Chang-Wae Wen Chen, Ying Shan, and Xiaohu Qie. Umt: Unified multi- modal transformers for joint video moment retrieval and highlight detection. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 3042-3051,
2022.
10

---

[6] Min Sun, Ali Farhadi, and Steve Seitz. Ranking domain- specific highlights by analyzing edited videos. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland. September 6-12, 2014, Proceedings, Part I 13, pages 787-802. Springer, 2014.
[7] Kristen Grauman, Andrew Westbury, Eugene Byme, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang. Miao Liu, Xingyu Liu, et al. EgoAd: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18995-19012, 2022.
[8] Haoran Chen, Dong Yi, Moyan Cao, Chensen Huang, Guibo Zhu, and Jinqiao Wang. A bench- mark for crime surveillance video analysis with large models. arXiv preprint arXiv:2502.09325 2025.
[9] Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shraman Pramanick, Difei Gao, Alex Jin- peng Wang, Rui Yan, and Mike Zheng Shou. Univtg: Towards unified video-language temporal grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2794-2804, 2023.
[10] Ye Liu, Jixuan He, Wanhua Li, Junsik Kim, Donglai Wei, Hanspeter Pfister, and Chang Wen Chen. R 2-tuning: Efficient image-to-video transfer learning for video temporal grounding. In European Conference on Computer Vision, pages 421-438. Springer. 2024.
[1]] WonJun Moon, Sangeek Hyun, Skng Park, Dongchan Park, and Jae-Pil Heo. Query- dependent video representation for moment retrieval and highlight detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 23023-23033 2023.
[12] Yifang Xu, Yunzhuo Sun, Benxiang Zhai, Youyao Jia, and Sidan Du. Mh-detr: Video moment and highlight detection with cross-modal transformer. In 2024 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE, 2024.
[13] WonJun Moon, Sangeek Hyun, SuBeen Lee, and Jae-Pil Heo. Correlation-guided query- dependency calibration for video temporal grounding, arXiv preprint arXiv:2311.08885, 2023.
[14] Xun Yang, Fuli Feng. Wei Ji, Meng Wang, and Tat- Tat-Seng Chua. Deconfounded video moment re- trieval with causal intervention. In Proceedings the 44th international ACM SIGIR conference on research and development in information retrieval, pages 1-10, 2021.
the
[15] Liuyi Wang. Zongtao He, Ronghao Dang, Mengjiao Shen, Chengju Liu, and Qijun Chen. Vision- and-language navigation via causal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13139-131550 2024.
[16] Judea Pearl and Dana Mackenzie. The book of why: the new science of cause and effect. Basic books, 2018
[17] Leland Gerson Neuberg. Causality: models, reasoning, and inference, by judea pearl, cambridge university press, 2000. Econometric Theory, 19(4):675-688 2003.
[18) Zitan Chen, Zhuang Qi, Xiao Cao, Xiangxian Li, Xiangxu Meng, and Lei Meng. Class-level structural relation modeling and smoothing for visual representation learning. In Proceedings of the 31st ACM International Conference on Multimedia, pages 2964 2972, 2023.
[19] Xiangxian Li, Yuze Zheng, Haokai Ma, Zhuang Qi, Xiangxu Meng, and Lei Meng. Cross-modal learning using privileged information for long-tailed image classification. Computational Visual Media, 10(5):981-99) 2024.
[20] Weixing Chen, Yang Liu, Binglin Chen, Jiandong Su, Yongsen Zheng, and Liang Lin. Cross- modal causal relation alignment for video question grounding. arXiv preprint 53X 2025.
[21] Yang Liu, Guanbin Li, and Liang Lin. Cross-modal causal relational reasoning for event-level visual question answering. IEEE Transactions on Pottern Analysis and Machine Intelligence, 16240))11624-1166 2023.
11

---

[22] Liuyi Wang, Zongtao He. Ronghao Dang, Huiyi Chen, Chengju Liu, and Qijun Chen. Causality- based cross-modal representation learning for vision-and-lane and-language navigation. arXiv preprint arXiv:2403.03405. 2024.
[23] Judea Pearl. Causal inference in statistics: An overview. 2009.
[24] Guoshun Nan, Rui Qiao, Yao Xiao, Jun Liu, Sicong Leng, Hao Zhang, and Wei Lu. Inter- ventional video grounding with dual contrastive learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2765-2775, 2021.
[25] Shen Yan, Xuehan Xiong, Arsha Nagrani, Anurag Arna,, Zhonghao Wang, Weina Ge, David Ross, and Cordelia Schmid. Unloc: unified framework for video localization tasks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13623- 13633, 2023.
[26] Yiyang Jiang, Wengyu Zhang, Xulu Zhang, Xiao Yong Wei, Chang Wen Chen, and Qing Li. Prior knowledge integration via Ilm encoding and pseudo event regulation for video moment retrieval. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 7249-7258, 2024.
[27] Jianfeng Dong, Xiaoman Peng, Daizong Liu, Xiaoye Qu, Xun Yang. Cuizhu Bao, and Meng Wang. Temporal sentence grounding with relevance feedback in videos. Advances in Neural Information Processing Systems, 37:43107-433133 2024.
(28) Tao Wang, Yu Li, Bingyi Kang, Junnan Li, Junhao Liew, Sheng Tang, Steven Hoi, and Jiashi Feng. The devil classification: A simple framework for long-tail instance segmentation. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pages 728-744. Springer, 2020.
[29] Yuqing Wang, Xiangxian Li, Zhuang Qi, Jingyu Li, Xuelong Li, Xiangxu Meng, and Lei Meng. Meta-causal feature learning for out-of-distribution generalization. In European Conference on Computer Vision, pages 530-545. Springer. 2022.
[30] Yuze Zheng. Zixuan Li, Xiangxian Li, Jinxing Liu, Yuqing Wang, Xiangxu Meng, and Lei Meng. Unifying visual and semantic feature spaces with diffusion models for enhanced cross-modal alignment. In International Conference on Artificial Neural Networks, pages 110-125. Springer, 2024.
[31] Wei Qin, Hanwang Zhang, Richang Hong, Ee-Peng Lim, and Qianru Sun. Causal interventional training for image recognition. IEEF Transactions on Multimedia, 25: 25:1033-1044, 2021.
[32] Xiheng Zhang, Yongkang Wong, Xiaofci Wu, Juwei Lu, Mohan Kankanhalli, Xiangdong Li, and Weidong Geng. Learning causal representation for training cross-domain pose estimator via generative interventions. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11270-11280 2021.
[33] Yushen Wei, Yang Liu, Hong Yan, Guanbin Li, and Liang Lin. Visual causal scene refinement for video question answering. In Proceedings of the 31st ACM International Conference on Multimedia, pages 377-386, 2023.
[34] Jia-Hong Huang, Chao-Han Huck Yang, Pin- Yu Chen, Andrew Brown, and Marcel Worring. Causal video summarizer for video exploration. In 2022 IEEE International Conference Multimedia and Expo (ICME), pages 6. IEEE, 2022.
[35] Asmar Nadeem, Faegheh Sardari, Robert Dawes, Syed Sameed Husain, Adrian Hilton, and Armin Mustafa. Narrativebridge: Enhancing video captioning with causal-temporal narrative. arXiv preprint arXiv:2406. 06499, 2024.
[36] Xu Yang, Hanwang Zhang, Guojun Qi, and Jianfei Cai. Causal attention for vision-language tasks. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 9847-9857, 2021.
12

---

[37] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[38]
John A Hartigan and Manchek A Wong. Algorithm as 136: A k-means clustering algorithm. Journal of the royal statistical society. series e (applied statistics), 28(1):100-008, 1979.
[39] Xiao Lin, Yun Peng. Liuyi Wang, Xianyou Zhong, Minghao Zhu, Jingwei Yang. Yi Feng. Chengju Liu, and Qijun Chen. Cleanpose: Category-level object pose estimation via causal learning and knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025.
[40] Bing Liu, Dong Wang, Xu Yang, Yong Zhou, Rui Yao, Zhiwen Shao, and Jiaqi Zhao. Show, deconfound and tell: Image captioning with causal inference. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 18041-18050, 2022.
[41] Heliang Zheng, Jianlong Fu, Zheng-Jun Jun Zha, and Jiebo Luo. Looking for the devil in the details: Learning trilinear attention sampling network for fine-grained image recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5012-5021, 2019.
[42] Zhen Yang, Ming Ding, Chang Zhou, Hongxia Yang, Jingren Zhou, and Jie Tang. Understanding negative sampling in graph representation learning. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1666-1676, 2020.
[43] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[44] Chinmay Kulkarni, Mohith Rajesh, et al. Dynamic binary cross entropy: An effective and quick method for model convergence. In 2022 s/ IEEE International Conference on Machine Learning and Applications (ICMLA), pages 814-818. IEEE, 2022.
[45] Alexander Neubeck and Lue Van Gool. Efficient non-maximum suppression. In 18th in- ternational conference on pattern recognition (ICPR 066), volume 3, pages 850-855, IEEE, 2006.
[46] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense- captioning events in videos. In Proceedings of the IEEE international conference on computer vision, pages 706-715, 2017.
[47] Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng. Zun Wang. Yansong Shi, al. Internvide02: Scaling foundation models for multimodal video understanding, In European Conference on Computer Vision, pages 396-416. Springer, 2024.
[48] Lisa Anne Hendricks, Oliver Wang. Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In Proceedings of the IEEE international conference on computer vision, pages 5803-5812. 2017.
[49] Victor Escorcia, Mattia Soldan, Josef Sivic, Bernard Ghanem, and Bryan Russell, Temporal localization of moments in video collections with natural language. 2019.
[50] Jie Lei, Licheng Yu, Tamara L. Berg, and Mohit Bansal. Tvr: A large-scale dataset for video-subtitle moment retrieval. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXI 16, pages 447-463. Springer, 2020.
[51] Pandeng Li, Chen-Wei Xie, Hongtao Xie, Liming Zhao, Lei Zhang, Yun Zheng, Deli Zhao, and Yongdong Zhang. Momentdiff: Generative video moment retrieval from random to real. Advances in neural information processing systems, (1665988659666 2023.
[52] Hao Sun, Mingyao Zhou, Wenjing Chen, and Wei Xie. Tr-detr: Tesk-reciprocal transformer for joint moment retrieval and highlight detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4998-5007, 2024.
13

---

[53] Hao Zhang, Aixin Sun, Wei Jing, and Joey Tianyi Zhou. Span-based based localizing network for natural language video localization. arXiv preprint vrXiv:2004.1393/1 2020.
[54] Hao Zhang, Aixin Sun, Wei Jing, Liangl Zhen, Joey Tianyi Zhou, and Rick Siow Mong Goh. Parallel attention network with sequence matching for video grounding. arXiv preprint arXiv:2105.08481, 2021.
[55] Cristian Rodriguez-Opazo. Edison Marrese- Marrese-Taylor, Basura Fernando, Hongdong Li, and Stephen Gould. Dori:
Discovering object relationships for moment localization of a natural language query in a video. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1079-1088, 2021.
[56] Shuo Yang and Xinxiao Wu. Eatity-aware and motion-aware transformers for language-driven action localization in videos. arXiv preprint arXiv:2205.05884, 2022.
[57] Houlun Chen, Xin Wang, Xiaohan Lan, Hong Chen, Xuguang Duan, Jia Jia, and Wenwu Zhu. Gurriculum-listenes Consistency-and complementarity-wwan audio-enhanced temporal sentence grounding. In Proceedings of the 31st ACM International Conference Multimedia, pages 3117-3, 2023.
14

---

A Technical Appendices and Supplementary Material
A.1 Details of the Datasets
QVHighlights [4] the only dataset supporting both moment retrieval and highlight detection. contains 10,148 trimmed videos spanning diverse domains such as daily vlogs, travel, and news. total of 10,310 queries are annotated with 18,367 disjoint temporal moments. also includes private test split for benchmarking.
Charades-STA [3] augments the Charades dataset with temporal annotations. consists of 9,848 indoor videos averaging 30.6 seconds in duration, annotated with 16, 16,128 query-moment pairs.
ActivityNet-Captions [46] is built upon ActivityNet v1.3 and contains over 20,000 untrimmed videos with 100,000 sentence-level annotations. Each video averages 120 seconds in length and includes multiple annotated events. The dataset is split into 10,024 training, 4.926 validation, and 5,044 test videos.
Charades-RF and Aetivity.ett-RF [27] are extensions of Charades- STA and SetivityeetCaattiinn that introduce ungrounded (false) queries to evaluate query relevance prediction. Each query is either grounded or intentionally mismatched test the model's 's abstention capability.
A.2 Sensitivity to the Number of K-means Clusters and Clustering Randomness
To investigate the robustness of the proposed CADE framework with respect to the number of K-means clusters and clustering randomness, we conduct a systematic ablation on the Charades- RF [27] dataset by varying the number of clusters K E (16, 32, 64, 128, 256, 512, 1024, 2048). For each K, multiple random seeds are used to initialize the K-means algorithm, and we report the mean standard deviation of Recall@ IoU = 0.7 (RI @0.7), mean IoU (mloU), and grounding accuracy (Acc).
Table 5: Ablation on the number of K-meang clusters K on Charades-RF Mean = std over multiple seeds.
Metrie /
16
32
64
128
256
512
1024
2048
RI @00.7
58.93
+ 1.71
60:39
0.66
61.25-00.11
60.29 + 0.40
58
0.41
59.12
+ 0.89
59.25
0.53
59.06 4 0.45
mloU
65.58
+ 1.30
66,18
+ 0.68
67.16=0.007
66.11 = 0.37
61.36
0.54
65.61
+ 0.33
65.56
0.16
65.15
0.41
Ace
84.25 + 85.11
0.14
84.96 0.51 83.50
0.15
84.37 +
0.51
84.11 4
As shown in Table 5, the model achieves the best overall performance when K = 64, where all three metrics peak with low standard deviation, indicating stable behavior across different random seeds. Using too few clusters (K < 32) leads to under- sedeeesegmentttionn of the latent semantic space, resulting in insufficient modeling of confounders and a notable drop in grounding performance. In contrast, using too many clusters (K > 512) introduces noisy and overly fine-grained mediator representations, increasing computational cost and reducing performance stability. The standard deviation remains relatively small for K values in the range of [32, 128), which demonstrates that CADE maintains robustness to K-means initialization randomness within this moderate range. This analysis confirms that a balanced cluster granularity (e.g.. K 64) provides the best trade-off between semantic coverage and stability.
A.3 Sensitivity to Temporal Strides in MSTP
We evaluate the effect of multi-scale temporal proposals (MSTP) on the QVHighlights validation set by varying the temporal stride set used generate proposals. We compare model without MSTP to configurations using strides {1), (1,2), {1, 2, 2]. {1, 2, 4, 8). and {1, 2,,2,4, 4,8, 8, 16). As summarized Table 6, introducing multi-scale temporal modeling substantially improves recall and average precision; performance increases monotonically from single- to four-scale settings, indicating that capturing range of action durations is critical for precise event localization, with an additional coarse scale (16) providing complementary gains.
15

---

Table 6: Sensitivity analysis of temporal strides used in MSTP on the QVHighlights validation
Temporal strides
R1@0.5
R1@0.7
mAP@0.5
mAP@0.75
Avg.mAP
w/o MSTP
61.35
38.00
61.40
36.75
36.29
[1]
60.84
40.58
62.15
37.81
37.56
[1,2]
67.10
48.45
67.69
45.16
43.35
(1,2,4)
70.39
53.10
70.91
50.00
48.63
[1,2,4,8)
70.84
56.00
72.17
53.79
50,98
(1,2,4.8,16)
70.19
54.97
72.33
53.76
51.88
A.4
Computational Cost and Efficiency
We evaluate computational overhead on the QVHighlights dataset using an NVIDIA A800 GPU (80GB) with batch size 64 for 50 epochs. As summarized in Table 7, CausalVTG's inference runtime slightly longer than simpler baselines yet remains competitive. Notably, prior methods such Moment-DETR and QD-DETR originally required up 200 epochs, and R2-Tuning utilized extensive GPU memory in training due to the reversed recurrent block.
Table 7: Computational cost comparison on QVHighlights.
Method
GPU Memory
#Parameters
Training Time
Inference Time
Moment-DETTE [4]
1.41 GB
4.82 M
9.71 min
31
QD-DETR[11]
1.89 GB
7.58
13.15 min
37 8
CG-DETR [13]
3.09 GB
12.61 M
40.05 min
45 8
R - [10]
37.24 GB
2.7 M
544.33 min
69
CausalVTG
2.31 GB
7.86 M
43.21 min
53
A.5 More Visualizations
To further assess model behavior, qualitative comparisons are conducted on the QVHighlights [4] validation set against the strong baseline R²- Tunnngg [10]. illustrated Figure 4, CausalVTG yields more precise and consistent results in both moment retrieval and highlight detection across various scenarios, reflecting better temporal alignment and semantic understanding. To analyze failure cases, Figure 5 presents examples where both R²-Tuning and CausalVTG do not produce accurate predictions. These often involve fine-grained visual attributes such as subtle differences in clothing color object type, which remain challenging due to limited perceptual cues.
A.6 Limitations & Future Work
While CausalVTG demonstrates strong performance in temporal grounding, it currently leverages only visual and textual modalities. In certain scenarios, especially those involving speech-centric queries or audio-specific events, incorporating audio cues could provide critical complementary information. Moreover, the model still struggles with fine-grained distinctions such as subtle visual attributes or small object interactions, as shown the failure cases. Future work could explore multimodal extensions with audio and depth signals, as well as finer perceptual modeling to improve grounding in visually ambiguous or low-resolution settings.
16

---

QUERY: guy with grey showing box filled with rubbish. wreeeee Peecevaeeee
Constrtic Prockcion
280
20.0
600
500
50.3
323
10.0
2040
123.
135
LBC.8
1400
QUERY: tourist couple the mango smoothie ball
Bricklin Cofe Molaysia
ViChantng Fradittio
20.0
50.0
1353
1900
headphones having interview
K3 Pradktos
380
1353
155
1900
QUERY: curly hair speaks directly
133
Cocoe//t Proeletio
52 tureing Profuciie
Womon puts items into plastic bags.
1000
Figure 4: Qualitative results showing successful predictions by CausalVTG compared R2-Tuning. VTG more accurately captures the target moments and saliency.
17

---

QUERY: A woman in blue and white striped wiggling her body around doing dance.
114
280
20.0
A00
200
20.3
72.0
62.0
90.0
3350
120.3
130
11C.0
1400
2A0
QUERY: Mon wearing sunglasses in interviewed
114
280
20.0
400
200
20.3
70.0
62.0
50.0
3300
120.3
120.0
120.0
1000
2000
Figure 5: Failure cases where CausalVTG struggles with fine-grained visual distinctions, such as clothing color or object details.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction of this paper accurately reflect its contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, the paper discusses the limitations of the work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
19

---

Justification:
The paper provide the full set of assumptions and a complete (and correct)
proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper fully discloses all the necessary information reproduce the main experimental results, ensuring that the main claims and conclusions can be independently verified.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
20

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The code and data have been submitted supplementary material in ZIP format.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
cnnolicyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /ttpps.ccpppliiiii/mmm/tao./nttc wwtppccppcc////ccccccccccccccclt htpollc/ggiieee http//uuddeee/eee https//ooSSSuuiisseeeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: This paper provides comprehensive details on the training and test settings, including data splits, hyperparameters.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The paper does not provide error bars, confidence intervals, statistical significance tests for the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the type of compute workers (GPUs), the size of memory and the time of execution.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //tprr:////uuiss....
Answer: [Yes]
Justification:
The research fully adheres the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The proposed method enhances robustness in video-language understanding, which can benefit accessibility and content summarization.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
22

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
23

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The paper does not involve LLMs as any important, original, or non- non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy pattp://neurigge http:///aarrppppccooooeeeeeeeeee / for what should or should not be described.
25


Reviews:
[{"review_id": "Ro87SWhtgB", "reviewer": "Reviewer_JXSa", "summary": "This paper targets the task of Video Temporal Grounding (VTG) and proposes the causal-inference framework CausalVTG. The method has three key components:\n1.Causality-Aware Disentangled Encoder (CADE) based on front-door adjustment to eliminate confounding bias between visual and textual modalities;\n2.Multi-Scale Temporal Perception (MSTP) to adaptively capture actions of different temporal lengths;\n3.Counterfactual Contrastive Learning to judge whether a query can truly be grounded.\nThe unified model simultaneously addresses Moment Retrieval, Highlight Detection, and Query Relevance. It achieves state-of-the-art results on five benchmarks—QVHighlights, Charades-STA, ActivityNet-Caption, Charades-RF, and ActivityNet-RF—showing especially strong performance under strict IoU thresholds and in “false-query” scenarios.", "questions": "1. Could you include at least one representative causal baseline in the comparison tables (or explain why this is not feasible) to clarify the benefit of your causal strategy?\n\n2. How sensitive is CADE to the K-means cluster number K and clustering randomness? An ablation or plot would help justify the causal robustness claim.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1.First to systematically introduce structured causal modeling, front-door adjustment, and counterfactual contrastive learning into VTG, effectively mitigating vision/text confounding and improving interpretability and generalization.\n2.Builds a single framework that performs Moment Retrieval, Highlight Detection, and Query Relevance in one pass, avoiding error accumulation in pipeline systems and keeping the overall design concise.\n3.Achieves top accuracy of 84.78 / 89.20 Acc on Charades-RF and ActivityNet-RF “false-query” tests, outperforming existing methods.\n\nWeaknesses\n1. The experimental section does not include comparisons with other causal-inference-based VTG methods, making it difficult to judge whether the proposed front-door adjustment combined with counterfactual contrast truly offers an advantage over alternative causal strategies.\n2.The model assumes that a K-means–derived mediator satisfies the front-door criterion, yet provides no evidence on how the choice of cluster number K or clustering stability affects performance. Without such sensitivity analysis, the claimed causal robustness remains speculative.\n3. The paper does not report mean ± std or statistical significance tests (e.g., p-values) for the main metrics, so it is difficult to gauge the reliability of the reported performance gains.", "was_revised": true, "final_justification": "During the rebuttal process, most of my concerns were adequately addressed. After reviewing the feedback from other reviewers, I have decided to maintain my positive score and increase the confidence score to 3.", "rebuttal": "We thank the reviewer for the thoughtful and constructive feedback, which helped us strengthen the paper with deeper analyses and clarifications. In response, we:\n\n- Added causal baselines (DCM) and the other state‑of‑the‑art methods (DoRi, CG‑DETR) to our comparison tables, clearly demonstrating that our front‑door adjustment with counterfactual contrast achieves consistent and substantial gains over alternative causal strategies (Q1).\n- Provided a sensitivity analysis of the K‑means cluster number (K) in CADE, reporting mean ± std metrics across multiple runs, which confirms the stability of our mediator design and supports the claimed causal robustness (Q2).\n\nWe believe these additions address the reviewer’s concerns and further highlight the robustness and contribution of our approach.\n\n\n## Response to Q1\n> Could you include at least one representative causal baseline in the comparison tables (or explain why this is not feasible) to clarify the benefit of your causal strategy?\n\nYes. In our revision, we have added DCM (Deconfounded Cross‑modal Matching) ​[1]– a causal method addressing dataset temporal‑annotation bias via back‑door adjustment . \nTo further strengthen our empirical comparison, we also included DoRi ​[2]\n and CG‑DETR [3], which are state‑of‑the‑art methods that mitigate superficial co-occurrence patterns by reducing correlations between background and foreground.\n\n\n| Method          |       |   Charades-STA        |             | |   ActivityNet-Caption      |            |    |    QVHighlights       |            |\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|                 | R1\\@0.5                | R1\\@0.7   | mIoU     | R1\\@0.5              | R1\\@0.7   | mIoU     | R1\\@0.5            | R1\\@0.7   | mAP Avg. |\n| TCN+DCM [1]         | 55.8                  | 34.4     | 48.7     | 44.9                | 27.7     | 43.3     | -                 | -        | -        |\n| DORi [2]            | 59.65                 | 40.56    | 53.28    | 41.49               | **26.41**    | 42.78    | -                 | -        | -        |\n| CG-DETR [3]         | 58.4                  | 36.3     | 50.1     | -                   | -        | -        | 65.43             | 48.38    | 42.86    |\n| CausalVTG | **70.89**                 | **49.25**    | **59.96**    | **45.62**               | 26.28    | **45.74**    | **68.87**             | **52.53**    | **49.63**    |\n\n\nThe updated comparison demonstrates that our method significantly outperforms all baselines, particularly on the Charades-STA dataset, where the presence of multiple stylistically diverse annotations per video segment renders models more susceptible to superficial co-occurrence patterns.\n\n\n## Response to Q2\n> How sensitive is CADE to the K-means cluster number K and clustering randomness? An ablation or plot would help justify the causal robustness claim.\n\nWe performed a comprehensive sensitivity analysis on the Charades‑RF dataset by varying the number of K‑means clusters in {16, 32, 64, 128, 256, 512, 1024, 2048}. \nFor each value, we ran multiple random seeds to compute mean ± standard deviation of metrics like R1\\@0.7, mIoU, and classification accuracy. \nDue to rebuttal constraints, only tabular results are presented here, but we commit to including detailed plots in the final version of the paper.\n\n|Metric/n_cluster|16|32|64|128|256|512|1024|2048|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|R1\\@0.7|58.93±1.71|60.39±0.66|**61.25±0.31**|60.29±0.40|58.20±0.41|59.12±0.89|59.25±0.53|59.06±0.45|\n|mIoU|65.58±1.30|66.18±0.68|**67.16±0.07**|66.44±0.37|64.36±0.54|65.64±0.33|65.56±0.16|65.15±0.43|\n|Acc|84.25±0.57|85.11±0.14|**85.76±0.69**|84.96±0.51|83.50±0.15|84.37±0.28| 84.00±0.51|84.11±0.11|\n\n\nThe experimental results indicate that a cluster number of 64 yields optimal performance. This choice provides comprehensive coverage of underlying semantic categories, effectively capturing confounders. In contrast, using too few clusters risks missing critical semantic distinctions, while employing too many introduces computational overhead and noisy representations, negatively impacting model performance.\n\n\n[1] \"Deconfounded video moment retrieval with causal intervention.\" SIGIR 2021.\n\n[2] \"DORi: Discovering object relationships for moment localization of a natural language query in a video.\" WACV 2021.\n\n[3] \"Correlation-guided query-dependency calibration for video temporal grounding.\" arXiv preprint arXiv:2311.08835 (2023)."}, {"review_id": "Ski4cnzLzJ", "reviewer": "Reviewer_Cyaw", "summary": "This paper proposes CausalVTG, a framework designed to address two major limitations in Video Temporal Grounding (VTG): (1) reliance on superficial co-occurrence patterns due to dataset biases, and (2) inability to handle scenarios where the query content might not be present in the video. The authors employ causal inference, specifically front-door adjustment, to mitigate confounding biases. They introduce a Causality-Aware Disentangled Encoder (CADE) to obtain unbiased modality-specific representations, and a Multi-Scale Temporal Perception (MSTP) module to capture temporal dynamics at multiple granularities. Additionally, the authors incorporate counterfactual contrastive learning to improve the model’s capability to distinguish between grounded and ungrounded queries. Experimental evaluations on **five** established benchmarks demonstrate state-of-the-art results across various settings.", "questions": "## Questions\n- Could you elaborate further on the assumptions underpinning the structural causal model and clarify their appropriateness and limitations specifically for VTG tasks?\n- How sensitive is your framework to the design of mediators? Have alternative mediator designs been explored, and how might different mediator definitions affect grounding performance?\n- Given multiple complex components (CADE, MSTP, counterfactual contrastive learning), could you further clarify which components are most critical? Additional ablations or sensitivity analyses would strengthen the paper significantly.\n- Could you provide more details on computational overhead during inference (e.g., runtime and memory consumption), especially compared to simpler baselines?\n- Can you provide analyses of your predictions similar to https://openaccess.thecvf.com/content/ICCV2023W/CLVL/papers/De_la_Jara_An_Empirical_Study_of_the_Effect_of_Video_Encoders_on_ICCVW_2023_paper.pdf Figure 1? such that we can evaluate potential  bias of the predictions of the model.\n\n## Suggestions\nI have some suggestions and comments that might help further strengthen your paper:\n\n1. **Related Work & Citations (Lines 42-50)**:  \n   It's great to see that you've cited Escorcia et al. (2019). For completeness and richer context, the paper would strongly benefit from citing two additional relevant works:\n   - [DORi (Rodriguez-Opazo et al., WACV 2021)](https://openaccess.thecvf.com/content/WACV2021/html/Rodriguez-Opazo_DORi_Discovering_Object_Relationships_for_Moment_Localization_of_a_Natural_WACV_2021_paper.html) introduces a spatio-temporal graph explicitly designed to reduce correlations between background elements and directly model interactions between objects, humans, and actions, addressing similar concerns about spurious correlations that you highlight.\n   - [\"A closer look at Temporal Sentence Grounding in Videos:\nDataset and Metric\" (ACM MM 2021)](https://dl.acm.org/doi/abs/10.1145/3475723.3484247) systematically analyzes temporal bias in moment localization tasks and provides dataset splits (Charades-STA CD and ActivityNet CD) designed explicitly to measure robustness against these temporal biases.\n\n   Including these references would enhance your motivation and clarify the novelty and contributions of your approach relative to existing work.\n\n\n2. **Multiple Occurrences and Temporal Causality**:  \n   An interesting scenario that your method might help analyze further relates to queries that describe events occurring multiple times within the same video.  \n   - For instance, as is mentioned in https://arxiv.org/abs/1809.01337 there are queries that has temporal dependency and would be great to explore and quantify\n- Also,**Figure 8** of DORi’s supplemental material, the Charades-STA query *\"person walks over to the refrigerator open it up\"* occurs twice. This raises the important practical scenario where a query might occur **multiple times**, **one**, or **not at all** in a video.  \n   - Similarly, the query examples from ActivityNet illustrated in **Figure 3** of the supplemental material from [\"Memory-efficient Temporal Moment Localization in Long Videos\" (EACL 2023)](https://aclanthology.org/2023.eacl-main.140/) highlight temporal causality and continuity, as in *\"The man continues...\"*. These are examples that explicitly involve temporal causal reasoning.\n\n   Your causal grounding method could show excellent performance for that kind of challenging queries to better showcase its strengths and contributions. An explicit analysis or discussion would significantly strengthen the practical motivation of your work.\n\n3. **Generalization and Causal Dependencies (YouCook2 dataset)**:  \n   It would be highly valuable and insightful to see your causal inference-based grounding applied to a dataset like **YouCook2**, where the causality of an action depends directly on previously executed actions.  \n   - For instance, Figures 13 and 14 of the supplemental material from DORi indicate specific cases where models incorrectly learn spurious correlations (e.g., associating certain ingredients or actions like *\"pouring dressing\"* or *\"adding oil\"* with incorrect moments due to biases in object presence or appearance).  \n   \n   Demonstrating your method’s robustness against such spurious correlations and temporal dependencies using YouCook2 would substantially strengthen the empirical validation and show clearer evidence of the claimed causal benefits.\n\nI encourage you to address these suggestion, as they would greatly enhance the quality, depth, and impact of your contribution.", "limitations": "Yes, the paper discusses limitations clearly. However, additional reflections on the complexity, computational efficiency, and assumptions in causal modeling would further strengthen this section.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### **Strengths**\n\n- The paper clearly identifies important, practical limitations in existing VTG methods, and convincingly motivates the application of causal inference.\n\n- The introduction of the causal inference framework, specifically using the front-door adjustment through the proposed CADE, is novel and technically sound. The MSTP module enhances temporal modeling effectively, complementing the causal design.\n\n-  Extensive experiments across five widely-used benchmarks (QVHighlights, Charades-STA, ActivityNet Caption, Charades-RF, ActivityNet-RF) clearly demonstrate improvements, especially under strict IoU thresholds and challenging ungrounded-query scenarios. I would recommend to add YouCookII and the partictions for Charades-STA and ActivityNet from https://arxiv.org/abs/2101.09028\n\n- The paper is very clearly written, methodically structured, and provides comprehensive implementation details and hyperparameters, aiding reproducibility.\n\n### **Weaknesses**\n\n- The combination of multiple intricate modules (CADE, MSTP, counterfactual training) introduces considerable complexity. This complexity may hinder interpretation of performance gains, particularly the exact contribution of each individual component.\n\n- The complexity of multi-scale temporal processing and causal disentanglement is computationally demanding. While runtime details are briefly mentioned, a deeper analysis of inference efficiency or scalability considerations would be valuable.", "was_revised": true, "final_justification": "Thank you to the authors for their thoughtful and well-articulated response. I remain positive about the paper and will maintain my original score", "rebuttal": "We appreciate the reviewer’s constructive feedback, which helped us strengthen the paper. In response, we:\n\n- Clarified the core causal assumptions and their appropriateness and limitations for VTG (Q1).\n- Analyzed mediator design sensitivity and discussed alternative definitions (Q2).\n- Expanded ablations to show the distinct and complementary roles of CADE, MSTP, and counterfactual learning (Q3).\n- Detailed computational overhead with runtime and memory comparisons to simpler baselines (Q4).\n- Provided prediction distribution analyses and committed to adding more visualizations to assess bias (Q5).\n- Incorporated suggested references and scenarios, including temporal causality, multiple occurrences, and new YouCook2 results (S1,S2,S3).\n\nWe believe these additions meaningfully address the reviewer’s concerns and further highlight the novelty, rigor, and practical impact of our work.\n\n## Response to Q1\n\n### Key Causal Assumptions:\n- Assumption 1 (Existence of Confounders): There exist latent confounders $Z$, such as visual stylistic cues (e.g., background contexts) and linguistic variations (e.g., phrasing habits), which simultaneously influence both the inputs \n$X=\\\\{V,Q\\\\}$ and the grounding outcome $Y$.\n- Assumption 2 (Front-door Identifiability): There exists a set of mediator variables \n$M$ (semantic representations) that fully mediate the causal relationship $X→Y$. These mediators are assumed to be independent of direct influences from the latent confounders $Z$, thus satisfying the conditions required for the front-door adjustment.\n### Appropriateness for VTG tasks:\n- Prior studies [1,2] clearly document the presence of stylistic biases within existing VTG datasets, highlighting the necessity of explicitly modeling such confounders.\n- Employing semantic mediators aligns with recent causal inference methods that effectively disentangle spurious correlations in related multimodal grounding tasks[3,4]​.\n### Limitations and Constraints:\nOur current SCM explicitly addresses stylistic variations but does not directly account for temporal location biases, which are also prevalent in VTG tasks [5,6]. In contrast, the DCM framework [5] explicitly employs back-door adjustments to mitigate temporal biases. \n\n\n## Response to Q2\n\nThe sensitivity of our framework to mediator design primarily stems from the choice of cluster number $K$ used in the K-means clustering of semantic mediators. To evaluate this sensitivity, we conducted experiments varying $K$, summarized in the table below.\n\n|Metric/n_cluster|16|32|64|128|256|512|1024|2048|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|R1\\@0.7|58.93±1.71|60.39±0.66|**61.25±0.31**|60.29±0.40|58.20±0.41|59.12±0.89|59.25±0.53|59.06±0.45|\n|mIoU|65.58±1.30|66.18±0.68|**67.16±0.07**|66.44±0.37|64.36±0.54|65.64±0.33|65.56±0.16|65.15±0.43|\n|Acc|84.25±0.57|85.11±0.14|**85.76±0.69**|84.96±0.51|83.50±0.15|84.37±0.28| 84.00±0.51|84.11±0.11|\n\nExperimental results show that setting the number of clusters to 64 achieves the best performance, effectively balancing semantic granularity and noise. Fewer clusters may miss key distinctions, while too many introduce redundancy and degrade model stability.\n\nRecent work, such as FDVAE (Front-Door Variational Autoencoder) [7], explores an alternative approach for front-door adjustment by learning latent mediators through variational inference instead of explicit clustering. Due to time constraints, we have not explored alternative mediator designs, but we will investigate them in future work to further improve performance and robustness.\n\n## Response to Q3\n\nTo clarify the contribution of each component, we provide detailed ablation results in Table 4 of our paper, which show that MSTP plays a crucial role in standard VTG tasks by generating proposals across multiple scales and granularities. To further support this, we include a new sensitivity analysis on the choice of temporal strides used in MSTP. This demonstrates that multi-scale temporal modeling is key to capturing varied action durations and enhances the model’s ability to localize events more precisely.\n\n|temporal strides|R1\\@0.5|R1\\@0.7|mAP\\@0.5|mAP\\@0.75|Avg.mAP|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|w/o MSTP|61.35|38.00|61.40|36.75|36.29|\n|{1}|60.84|40.58|62.15|37.81|37.56|\n|{1,2}|67.10|48.45|67.69|45.16|43.35|\n|{1,2,4}|70.39|53.10|70.91|50.00|48.63|\n|{1,2,4,8}|**70.84**|**56.00**|72.17|**53.79**|50.98|\n|{1,2,4,8,16}|70.19|54.97|**72.33**|53.76|**51.88**|\n\nFurthermore, we evaluated the impact of Counterfactual Contrastive Learning (QRM) on the Charades-RF dataset, which includes queries not grounded in videos. Results confirm that QRM substantially improves the model's ability to discern and reject ungrounded queries.\n\n||Acc|R\\@0.3|R1\\@0.5|R1\\@0.7|mIoU|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|CausalVTG|**84.78**|**76.22**|**71.07**|**61.03**|**67.86**|\n|w/o QRM|47.82|38.84|34.14|25.11|30.61|\n\nFinally, we validated the effectiveness of CADE on the Charades-CG dataset [8], which includes novel phrase compositions and unseen words in the test split. The observed performance gains highlight CADE’s crucial role in mitigating stylistic biases and enhancing the model’s generalization to unseen linguistic patterns.While MSTP indeed contributes the largest raw performance gain, CADE plays a distinct and critical role in addressing superficial co-occurrence patterns through front-door adjustment, which MSTP alone cannot resolve.\n\n|||Novel-Composition|||Novel-Word||\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n||R1\\@0.5|R1\\@0.7|mIoU|R1\\@0.5|R1\\@0.7|mIoU|\n|CausalVTG|**56.68**|**32.71**|**49.59**|**59.28**|**34.96**|**51.12**|\n|w/o CADE|52.15|29.05|46.15|54.68|30.36|47.03|\n\n\n## Response to Q4\n\nTo evaluate the computational overhead, we trained and evaluated all models on the QVHighlights dataset using an NVIDIA A800 GPU (80GB memory) with a batch size of 64 over 50 epochs. As summarized below, CausalVTG's inference runtime is slightly longer than simpler baselines, yet remains highly competitive. \n\n| Method||Training||Inference|\n|:-:|:-:|:-:|:-:|:-:|\n||GPU Memory|#Parameters|Time| Time|\n|Moment-DETR [1]|1.41 GB|4.82 M|9.71 min|31 s|\n|QD-DETR [11]|1.89 GB|7.58 M| 13.15 min| 37 s|\n|CG-DETR [12]|3.09 GB|12.61 M| 40.05 min|45 s|\n|R2-Tuning [13]|37.24 GB|2.7 M| 544.33 min|69 s|\n|CausalVTG|2.31 GB|7.86 M|43.21 min|53 s|\n\n## Response to Q5\n\nDue to rebuttal constraints, we cannot include full visualizations. As a substitute, we provide below the normalized distributions of predicted temporal intervals on the Charades-STA test set. The first table corresponds to successful cases (IoU > 0.7) and  the second table corresponds to failed cases (IoU < 0.7).We commit to including full visualizations and deeper analysis of temporal prediction distributions and feature biases in the final version.\n\n|start/end|0-0.2|0.2-0.4|0.4-0.6|0.6-0.8|0.8-1.0|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|**0-0.2**| 0.072|0.317|0.135|0.007|0|\n|**0.2-0.4**|0|0.004|0.094|0.035|0.003|\n|**0.4-0.6**|0|0|0.004|0.066|0.078|\n|**0.6-0.8**|0|0|0|0.001|0.151|\n|**0.8-1.0**|0|0|0|0|0.027|\n\n|start/end|0-0.2|0.2-0.4|0.4-0.6| 0.6-0.8 | 0.8-1.0 |\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|**0-0.2**|0.067|0.286|0.111|0.023|0.002|\n|**0.2-0.4**|0|0.003|0.085|0.073|0.018|\n|**0.4-0.6**|0|0|0.003|0.065|0.092|\n|**0.6-0.8**|0|0|0|0.003|0.145|\n|**0.8-1.0**|0|0|0|0|0.023|\n\n\n## Response to S1\nWe thank the reviewer for pointing out these two key references, both are highly relevant to the challenges addressed in our work. We will include and discuss them in the final version.\n\n## Response to S2\n\nFor temporal causality, we thank the reviewer for highlighting these important aspects. Regarding temporal causality, the referenced works introduce queries requiring reasoning over event dependencies rather than isolated segments. We agree this is a challenging and meaningful scenario, and we plan to explore temporal causal reasoning in future work.\n\nFor multiple occurrences, the QVHighlights dataset [1] includes annotations where queries may appear multiple times within a video. Our framework can handle such cases, as illustrated by the example visualization in the third subgraph of Figure 4 in our paper, which shows correct localization of repeated events. We will clarify this capability more explicitly in the final version.\n\n## Response to S3\n\nWe appreciate the reviewer’s suggestion to evaluate causal grounding on YouCook2.Our results demonstrate that CausalVTG outperforms prior methods by a clear margin, indicating improved modeling of causal relations. \n\n|Method|R1\\@0.3|R1\\@0.5|R1\\@0.7|mIoU|\n|:-:|:-:|:-:|:-:|:-:|\n|DORi [9]|43.36|30.47|18.24|30.46|\n|LOCFORMER [10]|46.76|31.33|15.81|30.92|\n|CausalVTG|**52.38**|**38.80**|**23.54**|**37.09**|\n\n[1] \"Detecting moments and highlights in videos via natural language queries.\" NeurIPS 2021.\n\n[2] \"Compositional temporal grounding with structured variational cross-graph correspondence learning.\" CVPR 2022.\n\n[3] \"Cross-modal causal relation alignment for video question grounding.\" CVPR 2025.\n\n[4] \"Vision-and-language navigation via causal learning.\" CVPR 2024.\n\n[5] \"Deconfounded video moment retrieval with causal intervention.\"SIGIR 2021.\n\n[6] \"A closer look at temporal sentence grounding in videos: Dataset and metric.\" ACM MM 2021.\n\n[7] \"Causal inference with conditional front-door adjustment and identifiable variational autoencoder.\" ICLR 2024.\n\n[8] \"Compositional temporal grounding with structured variational cross-graph correspondence learning.\" CVPR 2022.\n\n[9] \"DORi: Discovering object relationships for moment localization of a natural language query in a video.\" WACV 2021.\n\n[10] \"Memory-efficient temporal moment localization in long videos.\" EACL 2023.\n\n[11] \"Query-dependent video representation for moment retrieval and highlight detection.\" CVPR 2023.\n\n[12] \"Correlation-guided query-dependency calibration for video temporal grounding.\" arXiv preprint arXiv:2311.08835 (2023).\n\n[13] \"R2-tuning: Efficient image-to-video transfer learning for video temporal grounding.\" ECCV 2024."}, {"review_id": "pvvmjz9xgQ", "reviewer": "Reviewer_U4c9", "summary": "In this paper, authors firstly formulate a structural causal model for visual temporal grounding task. They define stylistic variations in visual and textual modalities as confounders. Building on top of the proposed structural causal graph, authors implement CausalVTG and verify its effectiveness on five benchmarks. As a technical summary, authors integrate causality-aware disentangled encoder, multi-scale temporal perception module, and counterfactual contrastive learning objective in the proposed CausalVTG framework, which composes a solid contribution to the community.", "questions": "1. The first claimed contribution of this paper is causal graph for video temporal grounding task. It would be great to straightforwardly show what is the confounder exactly in main paper. From common understanding of our community, confounder can be something like video background information or some template or pattern in language query. Explicitly show it in paper can enhance the paper quality.\n\n2. Referring to the weakness part, can you include a detailed and thorough ablation study in main paper? You can save much space by making Figure 2 as single-column", "limitations": "Authors discuss limitations of proposed method on page 14 and 15. Other limitations please refer to weakness part.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. The clarity of this paper is good and the paper is easy to follow. This paper is well motivated by two challenges: 1) superficial co-occurrence patterns and 2) assumption that relevant segments always exist.\n\n2. Authors also consider a scenario when the given query is not relevant to the video content, the model should be able to reject grounding any segment. This makes the proposed paper more thorough and suitable for real world application.\n\nWeaknesses\n\n1. No thorough ablation study is provided in main paper. Since the proposed CausalVTG is a combination of CADE, MSTP, QRM and QGR as shown in Table 4 in supplementary material, a detailed ablation study is needed in main paper. A base version of CausalVTG can be base model + CADE, then the integration of any one or two modules/methods given MSTP, QRM, and QGR should be included.\n\n2. Performance gain from CausalVTG is mainly from MSTP, which is not related to causal reasoning.", "was_revised": true, "final_justification": "Thank you for the detailed reply. Additional results on Charades-CG make the method more convincing. I've changed my rating from borderline reject to borderline accept.", "rebuttal": "We appreciate the reviewer’s critical feedback and agree that additional targeted analysis is essential for strengthening the paper. In response, we conducted new experiments and provided clarifications.\n- To address the need for a thorough ablation study, we added detailed experiments showing the individual and combined contributions of CADE, MSTP, QRM, and QGR, confirming that each module is complementary (W1, Q2).\n- To clarify the reviewer’s concern that performance gains mainly come from MSTP, we demonstrate that MSTP improves proposal generation, while CADE uniquely mitigates spurious correlations via causal adjustment, playing an irreplaceable role (W2).\n- To respond to the reviewer’s question about confounders, we explicitly define and illustrate them in the VTG context (e.g., spurious query–visual co-occurrences) and commit to clarifying these points in the main paper (Q1).\n\nWe believe these additions resolve the reviewer’s key concerns and strengthen the overall contribution of our work. If any issues remain unclear, we would be happy to provide further discussion.\n\n## Response to W1\n> No thorough ablation study is provided in main paper. Since the proposed CausalVTG is a combination of CADE, MSTP, QRM and QGR as shown in Table 4 in supplementary material, a detailed ablation study is needed in main paper. A base version of CausalVTG can be base model + CADE, then the integration of any one or two modules/methods given MSTP, QRM, and QGR should be included.\n\n\nWe thank the reviewer for highlighting this important point. In response, we have conducted a comprehensive ablation study following the reviewer’s suggestion. The results demonstrate the individual and joint contributions of CADE, MSTP, QRM, and QGR, validating their effectiveness. The ablation results demonstrate that each module is complementary and their integration is crucial to the overall effectiveness of CausalVTG. We will incorporate a detailed version of this analysis into the main paper.\n\n| | CADE | QGR | MSTP | QRM | R1\\@0.5 | R1\\@0.7 | mAP\\@0.5 | mAP\\@0.75 | Avg. mAP |\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n| (a)  |      |     |      |     | 57.74  | 36.52  | 58.96   | 35.36    | 35.19    |\n| (b)  | ✓    |     |      |     | 59.94  | 39.55  | 60.22   | 37.43    | 36.47    |\n| (c)  |      | ✓   |      |     | 60.39  | 38.52  | 60.39   | 37.05    | 36.61    |\n| (d)  |      |     | ✓    |     | 67.68  | 51.68  | 69.48   | 51.33    | 47.85    |\n| (e)  |      |     |      | ✓   | 60.77  | 39.16  | 61.32   | 36.98    | 36.87    |\n| (f)  | ✓    | ✓   |      |     | 61.32  | 38.58  | 61.65   | 37.22    | 37.11    |\n| (g)  | ✓    |     | ✓    |     | 68.58  | 52.71  | 69.69   | 50.89    | 48.99    |\n| (h)  | ✓    |     |      | ✓   | 62.19  | 40.05  | 61.66   | 37.86    | 37.42    |\n| (i)  | ✓    | ✓   | ✓    |     | 68.13  | 52.9   | 69.95   | 52       | 49.54    |\n| (j)  | ✓    |     | ✓    | ✓   | 70.26  | 54.32  | 71.34   | 52.67    | 50.15    |\n| (k)  | ✓    | ✓   | ✓    | ✓   | **70.84**  | **56**     | **72.17**   | **53.79**    | **50.98**    |\n\n## Response to W2\n> Performance gain from CausalVTG is mainly from MSTP, which is not related to causal reasoning.\n\nWe thank the reviewer for this important point. While MSTP indeed contributes the largest raw performance gain by improving multi-scale proposal generation, CADE plays a distinct and critical role in addressing superficial co-occurrence patterns through front-door adjustment, which MSTP alone cannot resolve. To evaluate CADE’s effect, we evaluated it on the Charades-CG dataset [1], which stresses compositional generalization (novel phrase compositions and unseen words). Results show that removing CADE leads to clear drops in performance on these challenging settings, confirming that CADE is essential for enabling the model to generalize beyond spurious correlations, complementing MSTP rather than being redundant.\n\n\n|||Novel-Composition|||Novel-Word||\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n||R1\\@0.5|R1\\@0.7|mIoU|R1\\@0.5|R1\\@0.7|mIoU|\n|CausalVTG|**56.68**|**32.71**|**49.59**|**59.28**|**34.96**|**51.12**|\n|w/o CADE|52.15|29.05|46.15|54.68|30.36|47.03|\n\n\n\n\n## Response to Q1\n> The first claimed contribution of this paper is causal graph for video temporal grounding task. It would be great to straight forwardly show what is the confounder exactly in main paper. From common understanding of our community, confounder can be something like video background information or some template or pattern in language query. Explicitly show it in paper can enhance the paper quality.\n\nThank you for your valuable feedback. In our work, the primary confounder is the presence of superficial co-occurrence patterns between query phrases and visual contexts that spuriously influence grounding outcomes.\n\nFor example, in the Charades-STA training set, queries containing the word “sink” are commonly paired with actions like “wash” and “put”. However, when the query also includes “kitchen”, the dominant associated actions shift to “put” and “run”. This distributional shift—caused by an irrelevant contextual word—demonstrates a classic confounding effect: the added word jointly affects both the input representation and the prediction target, while being independent of the actual intended action.\n\nThis explains the failure case in Figure 1 of our paper, where R2-Tuning [2] succeeds on Query 1 but fails on Query 2. The presence of “kitchen” introduces a confounder that changes the model’s grounding behavior despite both queries referring to the same event.\n\nIn addition, we also consider unobservable confounders, such as visual decoration styles or recurring sentence templates, which may similarly bias grounding decisions. Our method addresses both types via a front-door adjustment mechanism, as supported by prior causal grounding work [3].\n\nTo address the reviewer’s suggestion, we will revise the main paper to more directly and concretely present these confounders.\n\n\n\n## Response to Q2\n> Referring to the weakness part, can you include a detailed and thorough ablation study in main paper? You can save much space by making Figure 2 as single-column\n\nFollowing the reviewer's suggestion, we will resize Figure 2 to a single-column format in the main paper and incorporate the detailed ablation study from the supplemental material into the updated version.\n\n\n[1] \"Compositional temporal grounding with structured variational cross-graph correspondence learning.\" CVPR 2022.\n\n[2] \"R2-tuning: Efficient image-to-video transfer learning for video temporal grounding.\" ECCV 2024.\n\n[3] \"Cross-modal Causal Relation Alignment for Video Question Grounding.\" CVPR 2025."}, {"review_id": "aGAwB9HkuR", "reviewer": "Reviewer_P72B", "summary": "This paper proposes CausalVTG, a novel framework for Video Temporal Grounding that explicitly incorporates causal inference into the model design. By constructing a Structural Causal Model and employing front-door adjustment via a Causality-Aware Disentangled Encoder, the method aims to eliminate dataset-induced confounding biases. Additionally, a Multi-Scale Temporal Perception Module is used to enhance temporal reasoning at different granularities, and a Counterfactual Contrastive Learning objective is introduced to distinguish relevant from irrelevant queries. Extensive experiments on five widely-used VTG benchmarks demonstrate that CausalVTG achieves state-of-the-art performance in both localization precision and query relevance detection tasks. The paper is well-structured and the proposed ideas are clear and well-motivated.", "questions": "1. While CADE is conceptually important, its removal causes relatively minor degradation in performance. Could the authors provide more targeted experiments (e.g., under domain shifts or style variations) to better demonstrate its effect?\n2. Have the authors considered generating more challenging or realistic counterfactual samples instead of random negative sampling? Would this further improve the robustness?\n3. The method depends heavily on large-scale pre-trained models, but there is no detailed discussion on computational cost or efficiency compared to existing baselines.", "limitations": "YES", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: The paper addresses two fundamental but underexplored challenges in VTG — spurious correlations and grounding absence — through a principled causal framework. The introduction of causal inference, particularly front-door adjustment, into the VTG task is novel and fills a critical gap in the literature. The proposed components, including the SCM-based encoder, multi-scale temporal module, and counterfactual learning objective, are coherently designed and theoretically justified. The experimental evaluation is comprehensive, covering multiple benchmarks and metrics, and effectively validates the proposed contributions. The paper is clearly written, well-organized, and presents figures and tables that are informative and supportive of the narrative.\n\nWeaknesses: The ablation study shows that removing the causal module (CADE) causes smaller performance drops compared to other modules, raising concerns about the direct impact of the causal design.The counterfactual learning strategy relies on negative sampling rather than generating challenging counterfactuals, potentially underutilizing the full potential of causal reasoning.The method depends heavily on large-scale pre-trained models, but there is no detailed discussion on computational cost or efficiency compared to existing baselines.", "was_revised": true, "final_justification": "With all queries resolved through the authors' cogent responses and the study's inherent strengths maintained, I retain my positive rating as originally assigned.", "rebuttal": "We sincerely thank the reviewer for the thoughtful and detailed feedback. In response to the concerns raised, we have made the following clarifications and additions:\n\n- We evaluated CADE on the Charades-CG dataset to show its benefit under compositional generalization and domain shift scenarios (Q1).\n- We introduced two more challenging negative sampling strategies — GT-excluded and Semantic Hard negatives — and demonstrated their effectiveness on Charades-RF (Q2).\n- We reported training and inference cost comparisons with major baselines, confirming CausalVTG’s computational efficiency (Q3).\n  \nWe believe these clarifications and additional results address the reviewer’s concerns. If any issues remain unclear, we would be happy to provide further discussion.\n\n## Response to Q1\n> While CADE is conceptually important, its removal causes relatively minor degradation in performance. Could the authors provide more targeted experiments (e.g., under domain shifts or style variations) to better demonstrate its effect?\n\nYes, we further evaluated CADE separately on the Charades-CG dataset [1], which specifically tests novel phrase compositions and unseen words in temporal grounding tasks. These experiments demonstrate CADE's strong capability in addressing superficial co-occurrence patterns by enabling the model to generalize beyond previously observed linguistic combinations, an advantage particularly evident in compositional generalization scenarios​​.\n\n|||Novel-Composition|||Novel-Word||\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n||R1\\@0.5|R1\\@0.7|mIoU|R1\\@0.5|R1\\@0.7|mIoU|\n|CausalVTG|**56.68**|**32.71**|**49.59**|**59.28**|**34.96**|**51.12**|\n|w/o CADE|52.15|29.05|46.15|54.68|30.36|47.03|\n\n\n\n## Response to Q2\n> Have the authors considered generating more challenging or realistic counterfactual samples instead of random negative sampling? Would this further improve the robustness?\n\nWe thank the reviewer for the constructive suggestion. In addition to the original random negative sampling, we investigated two more challenging strategies to improve robustness: (1) GT-excluded negatives, where ground-truth segments are masked to force the model to reason over visually similar but irrelevant content; and (2) Semantic Hard negatives, where each query is paired with the most semantically similar but non-matching video in the batch.\n\nExperiments on Charades-RF, which contains naturally ungrounded queries, confirm the benefit of both strategies: GT-excluded negatives improve performance under stricter IoU settings, while Semantic Hard negatives further strengthen overall discriminative ability by compelling the model to distinguish fine-grained semantic differences.\n\n\n| Model          | Acc  | R1\\@0.3 | R1\\@0.5 | R1\\@0.7 | mIoU |\n|----------------|------|--------|--------|--------|------|\n| CausalVTG      | 84.78 | 76.22  | 71.07  | 61.03  | 67.86 |\n| + GT-excluded  | 83.17 | 76.29  | **72.35**  | **62.36**  | **67.88** |\n| + Semantic Hard | **86.24** | **77.05**  | 71.90  | 60.58  | 67.11 |\n\n\n\n## Response to Q3\n> The method depends heavily on large-scale pre-trained models, but there is no detailed discussion on computational cost or efficiency compared to existing baselines.\n\nTo evaluate the computational overhead, we trained and evaluated all models on the QVHighlights dataset using an NVIDIA A800 GPU (80GB memory) with a batch size of 64 over 50 epochs. As summarized below, CausalVTG's inference runtime is slightly longer than simpler baselines, yet remains highly competitive. Notably, prior methods such as Moment-DETR and QD-DETR originally required up to 200 epochs, and R2-Tuning utilized extensive GPU memory in training due to reversed recurrent block.\n\n| Method||Training||Inference|\n|:-:|:-:|:-:|:-:|:-:|\n||GPU Memory|#Parameters|Time| Time|\n|Moment-DETR [2]|1.41 GB|4.82 M|9.71 min|31 s|\n|QD-DETR [3]|1.89 GB|7.58 M| 13.15 min| 37 s|\n|CG-DETR [4]|3.09 GB|12.61 M| 40.05 min|45 s|\n|R2-Tuning [5]|37.24 GB|2.7 M| 544.33 min|69 s|\n|CausalVTG|2.31 GB|7.86 M|43.21 min|53 s|\n\n## Response to Paper Formatting Concerns\n> The paper uses bolded phrases at the beginning of paragraphs without any punctuation; The reference formatting is inconsistent.\n\nWe thank the reviewer for carefully pointing out these formatting issues, and we will correct the bolded paragraph headers and ensure consistent reference formatting in the final version.\n\n[1] \"Compositional temporal grounding with structured variational cross-graph correspondence learning.\" CVPR 2022\n\n[2] \"Detecting moments and highlights in videos via natural language queries.\" NeurIPS 2021.\n\n[3] \"Query-dependent video representation for moment retrieval and highlight detection.\" CVPR 2023.\n\n[4] \"Correlation-guided query-dependency calibration for video temporal grounding.\" arXiv preprint arXiv:2311.08835 (2023).\n\n[5] \"R2-tuning: Efficient image-to-video transfer learning for video temporal grounding.\" ECCV 2024."}]

Author rebuttal:

Meta-review: This paper proposes a novel framework for video temporal grounding that explicitly incorporates causal inference into the model design, consisting of structural causal model, modality-specific front door adjustment, and counterfactual reasoning. Extensive experiments on five widely-used benchmarks demonstrate that it achieves state-of-the-art performance in both localization precision and query relevance detection tasks. This paper received positive ratings from all reviewers; all reviewers agree that the paper tackles an important problem, proposes an interesting approach, and shows significant improvement over previous methods. The rebuttal also addresses most of the reviewer’s concerns, e.g., missing comparisons and analyses. The AC also appreciates the scientific contribution of the novel causal inference framework for video temporal grounding, and thus recommends accepting this work.
Decision: Accept (poster)