Counterfactually Augmented Event Matching for De-biased Temporal Sentence Grounding

Authors: xun  jiang
zhuoyuan  wei
shenshen  li
xing  xu
jingkuan  song
tao  heng
  shen
heng  tao
  counterfactually
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-03 12:56:51

Abstract: Temporal Sentence Grounding (TSG), which aims to localize events in untrimmed videos with a given language query, has been widely
studied in the last decades. However, recently researchers have demonstrated that previous approaches are severely limited in out-
of-distribution generalization, thus proposing the De-biased TSG challenge which requires models to overcome weakness towards
outlier test samples. In this paper, we design a novel framework, termed Counterfactually-Augmented Event Matching (CAEM),
which incorporates counterfactual data augmentation to learn event-query joint representations to resist the training bias. Specifically, it
consists of three components: (1) A Temporal Counterfactual Augmentation module that generates counterfactual video-text pairs by
temporally delaying events in the untrimmed video, enhancing the model’s capacity for counterfactual thinking. (2) An Event-Query Matching model that is used to learn joint representations and predict corresponding matching scores for each event candidate. (3) A Counterfact-Adaptive Framework (CAF) that incorporates the counterfactual consistency rules on the matching process of the same event-query pairs, furtherly mitigating the bias learned from training sets. We conduct thorough experiments on two widely used DTSG datasets, i.e., Charades-CD and ActivityNet-CD, to evaluate our proposed CAEM method. Extensive experimental results show our proposed CAEM method outperforms recent state-of-the-art methods on all datasets. Our implementation code is available at https://github.com/CFM-MSG/CAEM_Code.

Main paper: 
 introduction: 
 Temporal Sentence Grounding (TSG) task [1] [2] [3] [4] aims at localizing video events from untrimmed long videos in terms of given text queries. In previous works, the TSG models followed a prior assumption that the training and testing samples are under Independent-Identical-Distribution (IID) settings. However, test samples from the real-world environment are unpredictable, which could be outlier samples and lead to Out-Of-Distribution (OOD) problems. Recently, researchers [5, 6] demonstrated that the conventional TSG methods suffer from heavily biased training data and show limited generalization on outlier data. As is illustrated in Fig. 1 (a), the target moments in IID data share a similar statistical distribution with the training data, while the target moments in OOD data show a shifted distribution, simulating the open-world outlier test samples. According to the initial works [5, 6] on exploring the bias in TSG task, the previous TSG methods [1, 3, 4, 7, 8] heavily overfit spurious correlations between video-text pairs and manual temporal annotations, leading to dramatically dropped performance on OOD test samples. To this end, exploring De-biased TSG (DTSG) methods, which could overcome the weakness of model generalization, is becoming an emerging task, raising wide attraction to the community of multimodal learning and multimedia applications.Following the mentioned initial works [5, 6] , a group of works [5, 6, [9] [10] [11] are proposed to tackle the DTSG task. Most of these methods incorporated causal reasoning [5, 6, 12] or data augmentation [9, 10] into the TSG task and achieved promising de-biased ability in the challenging DTSG task. Although these methods have made promising progress in avoiding the direct influence of annotation bias on prediction results, they ignore the indirect impact of the local location of visual content on the learned event-query joint representations, which is an essential intermediate process for conventional TSG paradigm. As illustrated in Fig. 1 (b), the widely adopted TSG paradigm, i.e., proposal-based pipeline [1, 3, 4, 7] , firstly generates video candidates and then assigns the matching score for ranking. In such a paradigm, the TSG models are trained to discriminate well-matched event-query pairs, instead of predicting the temporal location directly. Hence, we consider the causal graph in Fig. 1(c) , where the joint representations have three independent parent nodes: text query, video event and event location.In this paper, we propose a novel DTSG approach for resisting training bias in learning event-query matching, termed Counterfactually Augmented Event Matching (CAEM). Concretely, we first devise a Temporal Counterfact Augmentation (TCA) module to facilitate TSG models with counterfactual thinking ability. We observed that the process of annotating moments, which is actually matching event-query pairs manually, depends on the currently observed video content sequence. However, such a process forgo teaching the model the ability of counterfactual thinking, i.e., What if the target moment does not happen at this time but in the rest of the given videos? To this end, we propose the TCA module to generate counterfactuals by delaying the observed events to later moments. The counterfactuals are used to train our models with the observed factual data jointly. Secondly, we design an Event-Query Matching (EQM) model that follows the proposal-based TSG pipeline illustrated in Fig. 1 (b) to predict event-query matching scores. Finally, considering the causal graph depicted in Fig. 1 (c), we introduce the Counterfactual Adaptive Framework (CAF), which learns event-query joint representations and integrates counterfactual consistency rules [13, 14] to ensure consistent semantics for identical events occurring in different contexts.We evaluate our proposed CAEM method on two widely used DTSG benchmark datasets, i.e., Charades-CD and ActivityNet-CD. Additionally, we also generalize it on two benchmark datasets with novel text queries. Extensive experimental results prove our method outperforms recent state-of-the-art methods on the DTSG task.Overall, our contributions can be summarized as follows:•  
related works: 
 Out-Of-Distribution Generalization. The OOD generalization endeavors to train a model using data from the training environments so that it can effectively generalize to unfamiliar environments. As a challenging but practical problem, the OOD generalization, numerous algorithms [15] [16] [17] have been devised to enhance OOD generalization. One set of methodologies concentrates on reducing the disparities among the training environments [15, 18] . The meta-learning domain generalization method [19] utilizes a meta-learning approach and introduces simulated training and testing distributional shifts during training. In [16, 17, 20] , robust optimization is employed to train models for minimizing the worst-case training loss across a predetermined set of groups, while several approaches [18, 21] incorporate adversarial training to enhance OOD generalization performance. Additionally, the OOD generalization has been also widely explored in multimodal learning tasks, such as multimodal fusion analysis [22] [23] [24] [25] , vision-language understanding [26] [27] [28] , or multimedia content retrieval [5, 6, [29] [30] [31] , etc.. Particularly, Yuan et al. [5] and Yang et al. [6] raised a discussion on the temporal out-of-distribution problems in the temporal sentence grounding task, which motivated us to develop this work.De-biased Temporal Sentence Grounding. As a fundamental problem in multimodal video understanding, the temporal sentence grounding task [3, 4, 32, 33] has been studied for several yearsThe man then runs around being <MASK> by the bull.The man then runs around being <MASK> by the bull. and achieves promising performance via manual annotations with sufficient temporal cues. However, recent studies [5, 6] empirically demonstrated that the previous methods have defects in outlier test samples, showing limited generalization on unpredictable samples from the real world. To this end, more researchers focus on a more challenging task, i.e., De-biased Temporal Sentence Grounding (DTSG) [9] [10] [11] 34] , which aims at resisting training bias and improving model generalization on outlier test data. Specifically, Yuan et al. [5] proposed the initial DTSG work on exploring the performance of TSG models on IID and OOD test samples and proposed a corresponding solution for this challenge. Yang et al. [6] proposed a causality-inspired TSG framework that builds a structural causal model to capture the true effect of query and video content on the prediction. Nevertheless, all these DTSG methods studied the direct impacts on temporal predictions caused by biased training data, but ignore the truth that TSG methods select predictions by evaluating the matching scores of event-query joint representations. Inspired by this observation, we aim at developing a more effective DTSG method against training bias. Counterfactual Data Augmentation. The concept of counterfactuals originates from causal inference [35] [36] [37] , where it pertains to hypothetical situations contrary to observed facts. In the context of data augmentation, counterfactuals represent plausible variations of existing data points that maintain their fundamental characteristics while introducing changes that are consistent with the underlying data distribution. In recent years, CDA has garnered significant attention across various domains, including recommendation systems [38] [39] [40] , natural language processing [41] [42] [43] , and multimodal learning [44] [45] [46] [47] . For example, Wang et al. [38] proposed a CDA-based framework to mitigate the impact of imperfect training data and empower sequential recommendation models. Dixit et al. [41] devised a two-stage counterfactual data augmentation for large-scale pre-trained language models, achieving better generalization to OOD data. Overall, by exploring counterfactual spaces, CDA enables models to learn from a more diverse range of scenarios, thereby improving model robustness to handle unseen instances and adapt to real-world outlier data. To this end, we incorporate the CDA into DTSG task for improving the resistency against training bias caused by manual annotations. 
proposal method 3.1 preliminary: 
 Problem Formulation. We represent the untrimmed video set as 𝑉 , while the text query set is represented as 𝑄. For each case, the input can be represented as a pair consisting of untrimmed video 𝑣 and a sentence query 𝑞. Let 𝑡 s and 𝑡 e be the start and end time of one target video event respectively, and let M 𝜃 represent the TSG model, the goal of TSG task can be formulated as M 𝜃 (𝑣, 𝑞 ; 𝑉 , 𝑄, 𝐿) → {(𝑡 𝑘 s , 𝑡 𝑘 e )} 𝐾 𝑘=1 , 𝑡 𝑘 s < 𝑡 𝑘 e . (𝑡 s , 𝑡 e ) 𝑘 is the 𝑘-th retrieved moment in the ranked list of candidates. 𝐾 is the number of predicted candidates. 𝐿 represents the manual temporal annotations of training sets. Particularly, let 𝐿 iid and 𝐿 ood represent the ground truth of two test datasets sharing identical distribution and outlier distribution with the training set 𝐿, M 𝜃 is expected to achieve promising performance on both test datasets synchronously, which requires better generalization ability. Causal Graph in Our CAEM Method. The causal graph is a directed acyclic graph that reveals the causal relationships among variables. The nodes denote the variables and the edges represent the causal relationships among variables. As the proposal-based pipeline of TSG methods firstly generates video event candidates and then measures matching scores between each event and text queries, we describe causalities among five variables: text query 𝑄, video event 𝐸, corresponding temporal location in the video 𝐿, the joint event-query representations 𝑀 and predicted matching score 𝑆. Concretely, we illustrate the causal graph in Fig. 1(c Augmentation: at the annotated moment, we delay the matched video events to those video parts that have not been observed, thus generating a counterfactual that the eventquery matching does not happen currently but later.causal links, where the direct link denotes the causality between two nodes: cause → effect. The temporal location in the video 𝐿 will influence indirectly the predicted matching scores 𝑆 between text queries 𝑄 and video events 𝐸 via the joint representations 𝑀.Discussions on Factual Event-Query Matching Annotations.Given a video that contains multiple events, it can be regarded as a time series of visual signals, and its semantics are aligned with the textual modalities along the temporal dimension. Such a priori condition leads to directed contextual information, thus we humans tend to discriminate if an event happens by considering current situations and previous memories. For example, we illustrate a typical case in Fig. 3 . For any annotator, who is asked to determine the starting and ending timestamp with the given text query, it firstly watches the video until the described event begins at this time, and then keeps recording in mind until it ends. In this way, an annotated moment is generated, which represents an observed fact that the required event happened at this moment and the query is expected to build a strong correlation with this video content. Counterfactual Data Generation. In the context of the discussions above, we find that humans don't have to observe the rest of this video to discriminate the matching relationships between video events and textual queries. Therefore, we argue that the event-query matching should be context-irrelevant since the semantics video event is complete enough to build flawless matching pairs with text queries. As illustrated in Fig. 4 , we raise counterfactual thinking in determining if the event happens at a particular moment: What if the event is not observed now but later? Under the counterfactual consistency rule [13, 14] , if the event does not happen at the current moment but still happens in the unobserved rest part of the video, the matching correlations should be established consistently.To this end, we incorporate counterfactual thinking in data augmentation and propose a novel strategy for counterfactual data generation. Specifically, we denote any annotated video-text pair selected from the training set as (𝑣, 𝑞, 𝑙), where 𝑙 = (𝑡 𝑠 , 𝑡 𝑒 ) is the normalized moment, and denote the lengths of video as 𝑙 v . For short events which satisfy 𝑙/𝑙 v < 0.5, we move the event frames entirely and insert them at 𝑡 𝑠 + 𝜌 with a random ratio 𝜙, delaying the annotated event to (𝑡 𝑠 + 𝜌 1 , 𝑡 𝑒 + 𝜌 1 ). Here 𝜌 1 ∈ [0, 1 -𝑡 𝑒 ] is a random relative temporal distance. Moreover, for those events that satisfy 𝑙/𝑙 v ≥ 0.5, we pad a group of video frames with complete event semantics at the beginning of the video sample, extending the video length into (1 + 𝜌 2 ) * 𝑙 v , where 𝜌 2 ∈ [0, 3 * (𝑡 𝑒 -𝑡 𝑠 )] is a random relative temporal distance similarly. Here we also conduct the extending operation with a random ratio 𝜙. Particularly, the padded video frames are selected from the training set, which is a complete event from another video. In this way, we can get a group of counterfactually augmented training pairs, which can be used to train our models with original factual data jointly.Feature Extraction. Following the previous methods [5, 10, 34] , we extract the video and text features in an offline manner. Firstly, given an untrimmed video 𝑉 , we adopt the pre-trained visual backbones [48, 49] to extract the visual features. Moreover, we employ the GloVe [50] to extract the word-level embeddings of each sentence query 𝑄. Specifically, we apply mean pooling on the framelevel representations within each clip to obtain video feature sequence, represented asF V = f 𝑖 V |𝑖 = 1, 2, ..., 𝑙 V , where f 𝑖 V ∈ R 𝑑 𝑉 denote the 𝑖-th clip feature and 𝑙 V is the video feature length. For the query sentence 𝑄, the feature sequence consists of multiple word-level embeddings, denoted asF Q = f 𝑗 Q |𝑖 = 1, 2, ..., 𝑙 Q , where f 𝑗 Q ∈ R 𝑑 𝑄and 𝑙 Q are the 𝑗-th word feature and the number of words respectively. Event-Query Joint Representation. To learn an effective joint representation for event-query pairs, we incorporate vision-language transformers [51] into temporal adjacent networks [3] . Concretely, the vision-language transformers are deployed to conduct crossmodal interactions thus learning fine-grained multimodal representations that contain both visual and textual information. Furthermore, we leverage the temporal adjacent map [3] to generate abundant candidates, where the representation of each candidate is composed of multiple multimodal tokens sampled from the output of vision-language transformers. Let the VLTrans (•) denote vision-language transformers, we formulate this process as follows:EQUATIONwhere Mask (•) is a random mask operation: we adopt masked language modeling [51] here to learn more semantics from textual queries. h cls is the <CLS> token and [; ] is the concatenation opera-tion. E V = e 𝑖 V |𝑖 = 1, 2, ..., 𝑙 V and E Q = e cls , e 𝑖 Q |𝑖 = 1, 2, ..., 𝑙 Q are vision-aware and text-aware multimodal token sequence respectively, e cls , e 𝑖V , e 𝑖 Q ∈ R 𝑑 h . Following that, we generate a temporal adjacent map [3] illustrated in Fig. 2 , and fill each grid with composed features sampled from the vision-aware multimodal tokens. In this temporal adjacent map, the grid in (𝑡 𝑖 , 𝑡 𝑗 ) represents a candidate event starting at 𝑡 𝑖 and ending at 𝑡 𝑗 timestamps in the untrimmed videos, where 0 <= 𝑡 𝑖 <= 𝑡 𝑗 <= 𝑙 V . Take the grid (𝑡 𝑖 , 𝑡 𝑗 ) as an example, we denote the process of filling multimodal features as follows:EQUATIONwhere W 1,2,3 are learnable parameter matrics and ⌊•⌉ represents the rounding operation. b is the bias matrix. e(𝑡 𝑖 ,𝑡 𝑗 ) Mis the event-query joint representation for a candidate event at (𝑡 𝑖 , 𝑡 𝑗 ).Semantic Consistency Learning. As the matched event-query pairs share the same semantic in both vision and text modalities, we firstly leverage cross-modal contrastive learning to guide our model to learn semantic consistency. Specifically, given the feature of the target event e + M , and the global text-aware token e cls , we project these features into a joint embedding space and let the model distinguish between anchor sample and negative samples. Let 𝑁 𝑣 represent a set of features containing e + M and the negative samples of mis-matched event-query pairs from the same and other videos, the contrastive learning loss can be formulated as follows:EQUATIONwhere S (•) is the cosine similarity calculation. 𝑝 (Σ) represents the expected distribution depicting the correlations of {𝑉 , 𝑄, 𝐿}. 𝜏 is a temperature factor. 𝑓 𝑣 and 𝑓 𝑞 are project layers with normalizing operations for visual and textual modalities respectively. Additionally, we also employ the widely used Masked Language Modeling (MLM) to exploit the semantics hidden in textual queries.Given the text-aware multimodal token sequence E Q , we predict the masked words in Mask(F Q ). Take the 𝑖-th word masked as an example, the training loss for MLM is denoted as follows:EQUATIONwhere 𝑤 𝑖 represents 𝑖-th word at the original textual query sentence. Overall, the loss function of the Semantic Consistency Learning can be summarized as 𝐿 𝑆𝐶𝐿 = 𝐿 𝑐𝑙 + 𝐿 𝑚𝑙𝑚 . Temporal Supervision Perception. We devise two parts of loss functions to perceive direct supervision signals from the manual annotations, i.e., matching confidence and relative location modification. The first part is used to guide our model to discriminate whether an event-query pair, which has been represented as composed multimodal features, is well-matched. Moreover, as the temporal adjacent map is pre-defined manually, we also incorporate the second part, i.e., relative location modification to enable our model to predict a subtle relative location for each grid. Specifically, the matching confidence can be calculated as follows:𝐿 𝑚𝑠 = E ∼𝑝 (Σ) [E ∼𝑝 (𝑆 |Σ) [(-ŝ𝑖 log 𝑠 𝑖 -(1 -ŝ𝑖 ) log (1 -𝑠 𝑖 ))]],(5) where 𝑠 𝑖 = 𝑓 𝑠 (e 𝑖 M ) is a predicted matching score of the 𝑖-th candidate event in the temporal adjacent map, and 𝑓 𝑠 is a classifier. Meanwhile, ŝ𝑖 is a ground truth matching score of the 𝑖-th candidate event, which is generated by calculating the temporal IoU score between the pre-defined temporal adjacent map and manual annotations, such as:EQUATIONwhere (𝑡 𝑖 𝑠 , 𝑡 𝑖 𝑒 ) represents normalized timestamps in the 𝑖-th grid at the temporal adjacent map and ground truth, while ( t𝑠 , t𝑒 ) is the normalized ground truth. Similarly, the second part loss used for relative location modification can be formulated as:EQUATIONwhere (𝑟 𝑖 𝑠 , 𝑟 𝑖 𝑒 ) = 𝑓 𝑟 (e 𝑖 M ), which is the margin distance between the pre-defined location (𝑡 𝑖 𝑠 , 𝑡 𝑖 𝑒 ) and ground truth location ( t𝑠 , t𝑒 ).Here 𝑓 𝑟 is a regression head for predicting margin distance. Overall, the training objective for perceiving temporal supervision can be summarized as: 𝐿 𝑇 𝑆𝑃 = 𝐿 𝑚𝑠 + 𝐿 𝑟𝑙𝑚 .Training the Event-Query Matching Model with our counterfactually augmented video-text pairs could enhance the counterfactual thinking ability, but the intervented video frame order may also show influence in vision-language transformers as the changed token sequences. To this end, we furtherly propose the Counterfact-Adaptive Framework to overcome this defect.Pseudo-Siamese Transformers. As the goal of our Counterfact-Adaptive Framework is to mitigate the influence caused by the changed token sequences in transformers, we firstly extend the Vision-Language Transformers into a pseudo-siamese architecture [52] . As illustrated in Fig. 2 , we employ two groups of visionlanguage transformers with the same architecture but unshared parameters. They are designed to handle different inputs: one is asked to process the original training data while the other one aims to process the counterfactually augmented data. Specifically, let F * V represent the counterfactual augmented video features, we denote the pseudo-siamese vision-language transformers as follows:EQUATIONwhere " * " denotes the pipeline for counterfactual augmented data. We employ two group of unshared learnable matrices, i.e., W 1,2,3 and W * 1,2,3 to fill multimodal features for each candidate in two pipeline respectively. Similarly, we also calculate the Semantic Consistency Learning loss for two pipelines separately, i.e., 𝐿 SCL and 𝐿 * SCL . Finally, we employ a shared classifier to predict matching score for each candidate for both two vision-language transformers and combine them to learn final matching scores. Similarly, take the 𝑖-th candidate event as an example, the process of predicting the final matching score can be denoted as: 𝑠 ′ 𝑖 = 𝑓 𝑠 (e 𝑖 M ) + 𝜆𝑓 𝑠 (e 𝑖 * M ), where e 𝑖 M and e 𝑖 * M are the multimodal features of 𝑖-th candidate event in the two predicted temporal adjacent map. 𝑠 ′ 𝑖 and 𝑓 𝑠 (•) are predicted final matching score and shared 𝑓 𝑠 (•) classifier. Counterfactual Consistency Learning. Following the counterfactual consistency rule [13, 14] , if the event does happen in the rest of the videos, the event-query pairs should also be well-matched. In other words, the completeness of semantics in the same video events should be maintained. To this end, we leverage contrastive learning to align the multimodal representations of the same event from both two pipelines. Specifically, let êM and ê * M represent the multimodal representations for ground truth events in factual and counterfactual pipelines, the loss function of counterfactual consistency learning can be represented as:EQUATION)M and ê𝑛 M represent multimodal representations of the same and different events respectively. 𝑁 𝑒 represent a set of features containing ê+ M and negative samples. 𝜏 is the temperature factor. 
experiments 4.1 experimental settings: 
 Datasets. We evaluate our proposed CAEM models on the Charades-CD and ActivityNet-CD [5] datasets, which are repartitioned to evaluate the performance and generalizing ability of TSG models.In the two datasets, the target moments of samples in the training, val, and test-iid sets are independent and identically distributed (denoted as IID), while the test-ood set contains out-of-distribution (denoted as OOD) samples. Moreover, we also generalize our proposed CAEM method on the Charades-CG and ActivityNet-CG [53] datasets, which contain normal test queries where all words are seen in the training set (denoted as Trivial) and generalized test queries with unseen words in the training set (denoted as Novel).Implementation Details and Metrics. Following the previous methods [3, 5, 11, 34] , we adopt the off-the-shelf video features that are extracted by pre-trained 3D CNN backbones [48, 49] . The visual features and textual embeddings are projected into 256 dimensions before sending to vision-language transformers, and the hidden dimension of transformers is also set to 256. As for the hyperparameters, we set the random ratio 𝜙 in the Temporal Counterfactual Augmentation to 0.5. The balance factor 𝜆 for calculating the final matching score in the Counterfact-Adaptive Framework is set to 0.6. The temporal factor in 𝐿 SCL and 𝐿 CCL is set to 0.1. the training stage, we employ the Adam optimizer [54] is employed to update the parameters with the learning rate set to 4 × 10 -4 on a single Nvidia A6000 with 64 batch size. During the evaluation and testing stages, we adopt the widely used de-biased recall metrics proposed by [5] , which evaluate the model generalization on OOD samples by discounting the normal recall metrics for suppressing the performance of speculation methods that over-rely on moments annotation biases. Moreover, as a number of D-TSG methods [9, 11, 12, 34] employed conventional recall metrics to evaluate the model performance on the D-TSG task, we also evaluate our model under the same metrics for fair comparisons.We compare our proposed CAEM method with recent state-of-theart methods [5, 6, 10, 11, 34, 55] . For fairness, we adopt two types of metrics, i.e., De-Biased Metrics [5] and Normal Metrics [1, 7] , covering all previous methods evaluated with different metrics. Specifically, we summarize the counterpart methods as following two groups: (1) De-Biased Metrics: TCN-DCM [6] , MDD [5] , Multi-NA [10] , and DFM [11] . (2) Normal Metrics: D-TSG [12] , SVTG [9] , MomentDiff [55] , and BSSARD [34] . To further show the effectiveness of our method, we also make fair comparisons with the latest methods that show generalization on text modality on the Charades-CG and ActivityNet-CG [53] , including VISA [53] , VDI [56] , SSL2CG [57] , and MESM [58] .Comparisons with Recent DTSG Methods. We report the experimental results on Charades-CD and ActivityNet-CD [5] in Table 1 and Table 2 . By comparing our proposed CAEM method and the recent state-of-the-art methods, we list the following observations:(1) Our method consistently outperforms these counterpart methods on both benchmark datasets. Particularly, on the OOD test of the Charades-CD dataset, our method achieves 4% and 7% absolute improvements on the de-biased discounted metric dR@1, IoU=0.5, and conventional metric R@1, IoU=0.5 compared with existing state-of-the-arts. It demonstrates that our method shows better generalization towards OOD samples. This is because our proposed CAEM method leverages the TCA and CAF designs, which enhance counterfactual thinking in learning joint representations of matched query-event pairs, thus showing better generalization compared with others. (2) By comparing the improvements achieved by our method on the Charades-CD and ActivityNet-CD datasets, it can be observed that our proposed CAEM method shows more superiority over the former benchmark dataset. One probable reason is the described events in the ActivityNet-CD dataset contain more complicated semantics. These complex queries raise more challenges in learning joint representations for matched event-query pairs. Model Generalization on Novel Queries. We generalize our proposed CAEM method on two additional TSG methods, i.e., Charades-CG and ActivityNet-CG, which contain novel words in test queries that are unseen in the training set. According to the experimental results in Table 3 , we can observe that our proposed CAEM methods also achieve superior results and outperform existing methods with remarkable margins. Specifically, compared with the latest state-of-the-art method SSL2CG [57] , which aims to improve the generalization of models towards novel word or word-level compositions via self-supervised contrastive learning, our method achieves approximate 2% absolute improvements at least on all the metrics. Such results demonstrate that our proposed CAEM methods also show superior generalization toward text queries with unseen words. This is because our method focuses on the causal structure of learning joint representations with textual queries, events, and temporal locations, instead of processing a single modality or predicting grounding results directly. Ablation Studies. To explore the impact of different model components in our proposed CAEM method, we conduct ablation studies on the three key modules. Specifically, the experimental results are illustrated in Table 4 , and we can observe that: (1) The SCL performs a crucial role for IID test samples. By comparing the second and third lines in the upper subtable, we find that the ablated model SCL + TCA achieves remarkable improvements over the ablated model TCA. Moreover, the comparisons between TCA + CAF and the complete model SCL + TCA + CAF also illustrate the effectiveness of SCL. This is because the SCL is used to guide our model to learn hidden semantics in text queries and video events, and further align them within a common space. (2) The TCA significantly mitigates the bias in the training set and enhances the generalization ability of our models toward out-of-distribution test samples. We can see that the ablated models with TCA consistently achieve remarkable improvements on the OOD test results, as the TCA is used to teach models counterfactual thinking. (3) Moreover, we can also note that the CAF effectively boosts the comprehensive performance on both IID and OOD test samples. Facilitated with the CAF module, the models achieve the best performance on most metrics and outperform other ablated models with remarkable margins. Analysis on Temporal Counterfactual Augmentation. To furtherly demonstrate the effectiveness of our proposed TCA module, we conduct more analysis on the random ratio 𝜙 which controls if a sample will be transformed to a counterfactual case. Specifically, we adjust 𝜙 from low to high and observe how the model performance on OOD test samples fluctuates. Moreover, here we also adopt a discount ratio between the test results of IID and OOD sets to quantify the general performance, which could be calculated as: Δ = (𝑅 IID -𝑅 OOD )/𝑅 IID . Here 𝑅 IID and 𝑅 OOD represent model performance on IID and OOD test samples respectively. Note that lower Δ means better general performance of a DTSG model. By observing the experimental results illustrated in Fig. 4 , we can see that our model achieves the best performance on OOD test samples when the random ratio 𝜙 is near 0.5. Moreover, with a low or over-high random ratio 𝜙, the performance drops dramatically. One probable reason is the delicate balance between factuals and counterfactuals: A low random ratio undermines the effectiveness of TCA while an over-high random ratio increases the difficulty of exploiting the counterfactual consistency rule in CAF, thus both lead to poor generalization toward OOD samples. Analysis on Counterfact-Adaptive Framework. We also conduct further analysis of the CAF module to show its superiority. Specifically, we follow the experiments adopted in the analysis of TCA to explore how the performance of models facilitated by CAF fluctuates with the random ratio 𝜙. Moreover, we also observe the influence of the balance factor 𝜆 by evaluating the improvements of dR@1, IoU=0.5 achieved by our complete CAEM over CAEM w/o CAF. The experimental results above are illustrated in Fig. 4 and Fig. 5 respectively. In the context of the illustrations, we can observe that: (1) The CAF significantly boosts the effectiveness of our proposed TCA module. By comparing the ablated model CAEM w/o CAF and the complete model CAEM, we find the CAF consistently brings better improvements under different random ratio conditions. It demonstrates the superiority of introducing the counterfactual consistency rule into our models for learning semantic-invariable matching between video events and textual queries. (2) Higher balance factor 𝜆 can lead to a degeneration of general performance. With 𝜆 increasing, the IID performance is decreased while the OOD performance is improving gradually. We speculate that this is due to models increasingly emphasizing counterfactual consistency at the expense of overall model flexibility. When 𝜆 is set too high, the model becomes overly focused on imaging counterfactual instances, at the cost of learning meaningful semantic representations and capturing diverse patterns in the data. Analysis on Event-Query Matching Model. We also take a closer look at the Event-Query Matching (EQM) model by conducting further ablation studies on the training objectives, including relative location modification loss 𝐿 rlm , masked language modeling loss 𝐿 mlm , and cross-modal contrastive learning loss 𝐿 cl . According to the experimental results in Table 4 , we list the following observations: (1) 𝐿 rlm performs an essential role in our EQM model for both IID and OOD performance. We can see that the model performance on both IID and OOD drops dramatically without 𝐿 rlm . This is because the representations of event candidates in our EQM are learned with the compositions of vision-aware tokens, which have fixed mapping temporal locations determined manually. 𝐿 rlm aims to teach our model to learn relative temporal locations between these fixed temporal locations and ground truth. (2) Both 𝐿 cl and 𝐿 mlm show effectiveness in model performance. This is because the two training objectives are designed to exploit the hidden semantics and learn cross-modal alignment for better joint representations. Qualitative Analysis. We illustrate two typical test cases from the Charades-CD and ActivityNet-CD datasets in Fig. 6 . Moreover, to show the generalization of DTSG models, we also visualize the distributions of the training set following [10, 11] . By observing the visualization results, we can see that our proposed CAEM method precisely localizes the target event that is the most relevant to the given text query. Specifically, compared with the state-of-the-art DTSG method BSSARD [34] , our CAEM method achieves more accurate grounding results. Moreover, by observing the distributions illustrated in Fig. 6 (a) and (b), we can also find that our CAEM method shows better generalization on OOD test samples. It proves the effectiveness of our solution again, which introduces counterfactual data augmentation and consistency rule into event-query matching to achieve better generalization. 
conclusion: 
 In this paper, we proposed a novel method, termed Counterfactually Augmented Event Matching (CAEM), for the De-biased Temporal Sentence Grounding (DTSG) task. It aims to improve OOD generalization in localizing video events with given text queries by introducing counterfactual data augmentation and consistency rule into event-query matching. Thorough experiments on three benchmark datasets demonstrated our proposed method established new state-of-the-art performance on the DTSG task. For future work, we will study the bias in multimodal learning, and explore corresponding solutions for causal reasoning. 
 
 Appendix:  


Decision: This paper is well-written and easy to follow. The task is to localize events in untrimmed videos which is very challenging. The paper proposes a new method called Counterfactually-Augmented Event Matching (CAEM) to solve the training bias problem in the dataset. Its validation is mainly on biased datasets -- which the AC thinks is reasonable. For this exact point, the reviewer ub5m has a different view and his/her final review rating is rejection. The rebuttal has resolved his/her questions (from the AC's understanding), but seems not very convincing to him/her, while he/she did not reply specifically why not convincing. 

Given the high quality and novelty of this paper, the AC would like to accept it for publication. Authors are strongly suggested to make revisions according to the reviews for the camera-ready.

Reviews: review: paper_summary: paper_summary: The paper proposes a novel framework for addressing bias in temporal sentence grounding tasks, named Counterfactually Augmented Event Matching (CAEM). This framework enhances the model's counterfactual thinking by generating counterfactual video-text pairs through temporal delays of events in untrimmed videos. Additionally, an Event-Query Matching model is designed to learn joint representations and predict matching scores for each event candidate. The experimental results demonstrate that this method surpasses existing state-of-the-art methods on the Charades-CD and ActivityNet-CD datasets.
strength_weakness: strength_weakness: 1. The CAEM method introduced in the paper is innovative, incorporating the concept of counterfactual data augmentation into the temporal sentence grounding task, which is a novel attempt in this field.

2. Extensive experiments are conducted on two widely used datasets, and comparisons are made with several state-of-the-art methods.

3. The paper is well-structured, with clear methodology, detailed experimental design, and thorough analysis of results, making it accessible for readers to understand and replicate.

4. Addressing bias in video understanding has significant potential value for practical multimedia applications.
limitations: limitations: 1. The paper could benefit from a deeper theoretical analysis of why counterfactual augmentation helps in this specific task. Including a theoretical justification or a model that explains the observed improvements could strengthen the paper's contributions.

2. The proposed method involves generating counterfactual data, which could potentially increase the computational overhead. The paper does not discuss the scalability of the method or its performance implications on larger datasets. Add a discussion about the scalability of the proposed method, possibly supplemented with experiments on larger or more complex datasets. Evaluating the method's performance in terms of computational resources and time would be valuable for practical applications.

3. To enhance clarity, the paper should detail the relationship between the causal graph and the proposed model. For instance:

    **Define variables clearly** within the causal graph, such as video events, text queries, and their interactions.
    
    **Map model components** to elements in the causal graph, explaining how each part of the model corresponds to and enforces the causal relationships.
    
    **Demonstrate causal reasoning** through experiments showing how the model leverages the graph to improve accuracy and bias mitigation.

    **Visualize the causal graph** to aid understanding and provide interpretative insights into the model’s predictions.
paper_summary: paper_summary: This paper tackles the task of debiased video grounding. It proposes a counter-factual augmentation module, an event-query matching model, and a counter-factual adaptive framework. Experiments on Charades-STA and ActivityNet-Captions demonstrate the effectiveness of the proposed methods.
strength_weakness: strength_weakness: 1. The paper is well-organized, particularly the presentation of experiments, which is structured effectively and ensures a smooth reading experience.
2. The experiments conducted on two datasets demonstrate the effectiveness of the proposed methods.
limitations: limitations: **Major concerns:**

1. The motivation of proposed method still presents remains unclear.
    
    i) Why does the independence of the event, queries, and location contribute to debiasing grounding?
    
    ii) In my understanding, bias in grounding is mainly caused by the bias in annotated location. Given this, a debiased grounding model should: 1) be based solely on the matching of video content and text semantics, and 2) eliminate the interference of location when determining the matching score. According to Figure 1c, location information is explicitly injected into the multimodal representation. How does this result in debiased grounding?
    
2. Why does the author claim that the event, queries, and location are independent in their method? Conversely, why are these three entities dependent in conventional methods? In conventional grounding, the multimodal representation is also determined by these three entities independently (e.g., the previous method 2D-TAN).
3. How are Figures 1c and 2 related? Figure 1c is the causal graph of the proposed method, while Figure 2 shows the details of the proposed method. How is the idea in Figure 1c instantiated in Figure 2?  Additionally, Figure 2 indicates that the main framework of the proposed method is still based on a 2D Map, a framework from the previous 2D-TAN (which is designed for common grounding). Given this, why does the author claim their method is fundamentally different as claimed in Figure 1?

**Other concerns:**

- According to Line 658, dR@1, IoU=0.5 is used to denote the debiased discounted metric, while R@1, IoU=0.5 represents the conventional metric. Why are the results for conventional  2D-TAN [3] reported under dR@1, IoU=0.5 instead of R@1, IoU=0.5? 

**Note:** While I have concerns about the ambiguity of the motivation, I find the overall structureof the paper to be good and clear. I would raise my final rating if the author's response addresses my major concerns.
paper_summary: paper_summary: The paper introduces a novel framework called Counterfactually Augmented Event Matching (CAEM) to address training bias through counterfactual data augmentation.The framework includes a Temporal Counterfactual Augmentation module, an Event-Query Matching mode and a Counterfact-Adaptive Framework (CAF) that applies counterfactual consistency rules to the matching process, further mitigating training set biases. The experimental results demonstrate CAEM's potential in improving event-query representation learning and reducing bias.
strength_weakness: strength_weakness: 1. This paper addresses the issue of dataset bias in TSG, presenting an intuitive and effective solution.
2. The proposed method's efficacy is validated through experiments on two types of debiased datasets.
limitations: limitations: 1. This paper only performs comparisons on debiased datasets. While ActivityNet-CD and Charades-CD contain IID parts, it is recommended that the authors also conduct experiments on general datasets to compare performance in IID scenarios.
2. The overall framework of this paper is based on a proposal method. Given the prevalence of proposal-free methods, it would be beneficial to explore the adaptation of this approach to a proposal-free framework.
paper_summary: paper_summary: This paper addresses the out-of-distribution problem between train and test set in temporal sentence grounding. In this paper, they design a novel framework, termed Counterfactually-Augmented Event Matching. They devise 3 novel components to help the model to mitigate the distribution bias in datasets. The results on benchmarks shows the effecience.
strength_weakness: strength_weakness: 1. It introduces a novel debiased network for TSG. It devises a new counterfactual augmentation method by delaying the target moment and proposes counterfact-adaptive framework to learn event-query joint representations. The modules they proposed appear to be reasonable and complementary. Experiments on benchmarks seem to show the effectiveness.
2. Extensive experiments is sufficient and convincing.
Overall, this is a complete and novelty works.
limitations: limitations: 1.  Temporal Counterfactual Augmentation generates the counterfactual samples by delaying the target video segment and insert a random event into the previous location. However, this approach may result in moving backwards of distribution of target moment , which seems to not satisfy the independence of variable in counterfactual reasoning. It needs some extension experiments or at least a disscussion about it.
2. There are many losses in training, sensitive analysis of their weight balance is needed.