Efficient Dual-Confounding Eliminating for Weakly-supervised Temporal Action Localization

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 16:16:55

Abstract: Weakly-supervised Temporal Action Localization (WTAL) following a localization-by-classification paradigm has achieved significant results, yet still grapples with confounding arising from ambiguous snippets. Previous works have attempted to distinguish these ambiguous snippets from action snippets without investigating the underlying causes of their formation, thus failing to effectively eliminate the bias on both action-context and action-content. In this paper, we revisit WTAL from the perspective of structural causal model to identify the true origins of confounding, and propose an efficient dual-confounding eliminating framework to alleviate these biases. Specifically, we construct a Substituted Confounder Set (SCS) to eliminate the confounding bias on action-content by leveraging the modal disparity between RGB and FLOW. Then, a Multi-level Consistency Mining (MCM) method is designed to mitigate the confounding bias on action-content by utilizing the consistency between discriminative snippets and corresponding proposals at both the feature and label levels. Notably, SCS and MCM could be seamlessly integrated into any two-stream models without additional parameters by Expectation-Maximization (EM) algorithm. Extensive experiments on two challenging benchmarks including THUMOS14 and ActivityNet-1.2 demonstrate the superior performance of our method.

Main paper: 
 related work: 
 Weakly-supervised Temporal Action Localization. To solve the issue of fine-grained annotations, weakly-supervised techniques [59, 7, 35, 54, 52, 37, 36, 21] have emerged, enabling the localization of action instances with only video-level labels. Among these, UntrimmedNet [47] stands out as a pioneering effort, employing a Multiple Instance Learning (MIL) framework [29] to tackle the weakly-supervised temporal action localization task. Attentionbased methods [10, 31] have also gained traction, offering enhanced performance and architectural flexibility. These methods, including BaS-Net [17] and WSAL-BM [32] , could be used to suppress activations from background frames. More recently, pseudo labels are used to supervise the model to generate more accurate attention scores for video snippets [53] . TSCN [59] fuses RGB and FLOW modalities to generate better pseudo labels, while work [54] considers the complementarity. Traditionally, RGB and FLOW modalities have been treated uniformly, with their concatenated features at the feature level providing a straightforward yet limited approach to exploiting modality distinctions, such as ASM-Loc [8] , RSKP [12] , GauFuse [65] , TFE-DCN [64] , P-MIL [36] and so on, but it is not sufficient to capture the subtle differences between RGB and FLOW modalities. In contrast, another research direction focuses on calculating separate attention scores for each modality before combining them, aiming to better harness their complementary aspects, liking CO 2 -Net [10] , DELU [3] , DDG-Net [43] and so on. However, these methods are not fully utilize collaborations between modalities. In this paper, we mitigate confounding bias by exploiting the unique differences and synergies between RGB and FLOW modalities. Causal Intervention and Deconfounder. Causal intervention has been recently introduced to address the confounding bias in computer vision tasks, including image semantic segmentation [60] , image captioning [55] , image classification [42, 58] , object detection [38] and so on. This bias presents challenges not just in image-based tasks but also in video analyses, particularly spatial-temporally complex videos, where video temporal grounding [30, 48] , video action localization [26] , video action anticipation [61] , moment retrieval [56] etc. are explored. For example, work [60] propose a structural causal model that analyzes the causal relationships among images, contexts, and class labels. Based on this model, they develop a novel method called Context Adjustment (CONTA). DCM [56] introduces a causal model to discern the direct influence of queries and video content on outcomes, sidestepping the confounding effects of moment locations by isolating the core visual content features. For WTAL, Liu et al [26] point out that the "background" issue is impossible to be fundamentally resolved due to weak supervision, but they simply attribute every spurious association to an unobserved confounding and propose to learn a substitute by adopting PCA on temporal dimension. However, this approach rests on stringent assumptions, limiting its applicability. Different from previous works that consider the confounding bias on whole video level and model the confounding with additional parameters or extra storage costs, we seek to efficient eliminate the confounding bias on both actioncontext and action-content separately with the consideration of individualized differences among specific action instances. 
the proposed method: 
 In this section, we detail our efficient dual-confounding elimination framework. The overview of the proposed methods is shown in Fig. 3 , the framework follows the EM algorithm to iteratively mitigate confounding bias. Initially, we give the task formation (Section 3.1). Subsequently, we revisit WTAL from the perspective of structural causal model (Section 3.2). Following this, we construct the Substituted Confounder Set (SCS) to approximate action-context confounder (Section 3.3). To eliminate action-content confounding bias, we propose the Multi-level Consistency Mining (MCM) method in Section 3.4. Notably, our method can be applied to any two-stream models [10, 3, 43] without any additional learnable parameters by the EM algorithm. Finally, we introduce the training and inference details in Section 3.5.WTAL aims to identify action instances within untrimmed videos under the guidance of only video-level labels. Suppose we have a dataset of training videos represented as {F 𝑖 } 𝑁 𝑖=1 is given, where 𝑁 is the number of training videos and F 𝑖 ∈ R 𝑇 ×𝐷 is the extracted 𝐷dimensional video features of 𝑇 segments. Only the corresponding video-level label y 𝑖 ∈ {0, 1} 𝑀 is available to train models for each video V 𝑖 , where 𝑀 is the number of classes. During testing, the goal is to get a set of action proposals 𝑆 = (𝑠 𝑖 , 𝑒 𝑖 , 𝑐 𝑗 , 𝜑 𝑖 ) for each testing video, where 𝑠 𝑖 and 𝑒 𝑖 are the start and the end times of an action instance respectively, 𝑐 𝑗 indicates the predicted action class, and 𝜑 𝑖 denotes the related confidence score.In this section, we revisit WTAL with the perspective of structural causal model [4] . As illustrated in Fig. 4 (b), our structural causal model encapsulates four key variables: action prior 𝐶, video 𝑉 , action label 𝑌 and the diverse representations of action instances 𝐼 . The model establishes direct causal links, indicating cause-andeffect relationships between pairs of variables: cause → effect. Conventional WTAL methods (as depicted in Fig. 4(a) ) typically focus on learning from two main causal links: 𝑉 → 𝑌 and 𝐶 → 𝑌 to improve the model's generalization, but they overlook the variability among different action instances within the same action class. While our methodology scrutinizes the confounding bias at the instance-level, which is crucial for uncovering the underlying causes of the confounding bias.𝐶 → 𝑉 . Action prior 𝐶 derived from the entire dataset and considered as a universal action templates, influences the action depicted in video 𝑉 . 𝐶 not only tells the model what to expect in video 𝑉 and how to act in the video, which is specifically reflected in the RGB modality and FLOW modality, but also contains the action context that may lead the model to mistakenly identify snippets with similar visual appearances or motion patterns as actual actions. It is a significant challenge to accurately represent the 𝐶 → 𝑉 relationship in complex action scenarios. Fortunately, we can avoid the action-context bias through SCC, which will be introduced in Section 3.3.𝑉 → 𝐼 ← 𝐶. 𝐼 represents the distinct action instance, derived by the action templates from 𝐶. 𝑉 → 𝐼 indicates the causal relationship between video 𝑉 and action instances 𝐼 , while 𝐶 → 𝐼 depicts that instances are also determined by action prior 𝐶. For example, the action "Diving" may occur many times in a video 𝑉 𝑖 , but not all instances follow the same procedure, any "Diving" action instance 𝐼 𝑖 𝑗 can be descried by "Diving" template 𝐶 𝐷𝑖𝑣𝑖𝑛𝑔 . Importantly, there exists both commonalities and individual characteristics within different action instances of the same action category. Previous methods in Fig. 4 (a) usually focus on the commonalities, the overlook of instance-specific individualities makes models not confident for the less discriminative content snippets, resulting in action-content bias. Our perspective in Fig. 4(b ) is to balance the commonalities and individual characteristics.𝑉 → 𝑌 ← 𝐼 . In WTAL task, video class labels 𝑌 typically come from the conventional classification model 𝑉 → 𝑌 (𝑃 (𝑌 |𝑉 )). In fact, 𝑌 is also influenced by the mediation 𝐼 . 𝐼 → 𝑌 denotes an obvious causality that the action instances within a video determine the video-level class label. It is worth noting that the action instances 𝐼 are not used to supervise the model in WTAL task. The relationship 𝐶 → 𝑌 is not shown in the structural causal model, because the action prior template 𝐶 indirectly affects 𝑌 through the action instances 𝐼 . If 𝐼 → 𝑌 does not exist, there is an only path 𝐶 → 𝑉 → 𝑌 left from 𝐶 to 𝑌 , which is impossible for action prior 𝐶 to make effect to 𝑌 , and we would never uncover the confounding bias in WTAL. Multi action categories and multi action instances within a video prove the necessity of 𝐼 → 𝑌 .So far, we have established the structural causal model for WTAL task, showcasing it as an instrumental framework for uncovering the genuine causal connections among variables. Thanks to the graphical representation, we can clearly see how 𝐶 confounds 𝑉 and 𝑌 via the backdoor path 𝑉 ← 𝐶 → 𝐼 → 𝑌 , even if some video snippets in 𝑉 have nothing to do with 𝑌 , the backdoor path can still help to correlate them with 𝑌 , resulting the confounding bias. However, 𝐶 and 𝐼 remain unobservable and immeasurable in practice, posing challenges in directly intervening the backdoor path. Nonetheless, constructing a substitute for the unobserved confounder on 𝐶 is feasible by leveraging the modal disparity between RGB and FLOW modalities, and the substitute could be used to intervene the video snippets 𝑉 to eliminate action-context confounding. Similarly, the confounding bias related to the action-content instances 𝐼 could be mitigated by enforcing model to learn the consistency across each action instance. Ultimately, 𝑃 (𝑌 |𝑑𝑜 (𝑉 )) could be approximated.The existing of backdoor path 𝐶 → 𝑉 in the structural causal model leads to the confounding bias. As "physical" intervention is impossible, we apply the backdoor adjustment procedure [34] category 𝑖. The causal intervention can be formulated as:EQUATIONwhere 𝐼 = 𝑓 (𝑉 , 𝑐 𝑗 ) signifies the action instance within the localization task, 𝑓 (•) is the function that generates instances 𝐼 from𝑉 and 𝐶. 𝑃 (𝑌 |𝑉 , 𝐼 ) represents the conditional probability distribution that WTAL models need to learn. According to Eq. (1), 𝐶 is no longer correlated with 𝑉 , the causal intervention makes 𝑉 have a fair opportunity to incorporate every context 𝑐 𝑗 into the prediction of 𝑌 through 𝑃 (𝑐 𝑗 ). Now the challenge is how to obtain the unobserved and unmeasurable confounder on 𝐶 in practice. Fortunately, based on the discussion in Section 1 that visual ambiguity and motion ambiguity tend not to occur simultaneously [43] , the modal disparity between RGB and FLOW could be utilized to construct a substitute for the confounder on 𝐶. For a video 𝑉 𝑖 , we can get A 𝑅𝐺𝐵 𝑖 and A 𝐹 𝐿𝑂𝑊 𝑖 respectively through existing WTAL model. Then the substitute confounder 𝛹 𝑖 could be constructed byEQUATIONin which 𝐼𝑛𝑑𝑒𝑥 is indicator function to get the index of the condition, 𝛿 is a threshold to measure the gap between RGB and FLOW modalities. After getting 𝛹 𝑖 , we can get the representation of video 𝑉 𝑖 with causal intervention byEQUATIONwhere F 𝑖 is the representation of 𝑉 𝑖 , 𝜆 is the weight to control the strength of the intervention.The whole process of our causal intervention for video 𝑉 𝑖 can be formulated as:EQUATION)where Θ is the parameters of the WTAL model, and 𝑃 (𝑌 | F𝑖 , 𝐼 𝑖 ; Θ) is the prediction of the WTAL model after the causal intervention.By constructing the substitute confounder 𝛹 𝑖 and intervening the video feature F 𝑖 , we could effectively eliminate the confounding bias on action-context.Regarding the action-content bias, it arises because the classification loss steers the model towards concentrating on the most discriminative snippets. This focus may lead to the potential neglect of less discriminative action-content snippets, which are also crucial for accurate model performance. To address this issue, we propose a Multi-level Consistency Mining method to leverage the consistency between discriminative snippets and corresponding proposals on feature-level and label-level to effectively mitigate the action-content confounding.For feature-level consistency mining, instead of previous works [12] that select the representative action embeddings across the entire dataset while disregarding the individuality among same category action instances, we propose to pick up the representative snippets from corresponding proposals for each action instance, our approach then minimizes the distance between the representative action embeddings and the corresponding proposals. Specifically, following the previous work [10, 3, 43] , we can get the final attention scores A 𝑖 through existing WTAL model for video 𝑉 𝑖 , and then proposal set 𝑆 𝑖 = (𝑠 𝑗 , 𝑒 𝑗 , 𝑐 𝑗 , 𝜑 𝑗 ) can be generated by threshold 𝜃 𝑝𝑟𝑜 . For each proposal 𝑠 𝑗 , 𝑒 𝑗 in 𝑆 𝑖 , we first select the most representative snippets P 𝑖 𝑗 by:EQUATIONwhere 𝜃 𝑟𝑒𝑝 is the threshold to measure the representative snippets. Then, corresponding representative snippets feature can be calculated by:EQUATIONThe feature of corresponding proposal 𝑠 𝑗 , 𝑒 𝑗 can be calculated by:EQUATIONWe aim to minimize the distance between the representative action embeddings and the corresponding proposals, and the feature-level consistency loss can be formulated as follows:EQUATION)where 𝑁 is the number of training videos, 𝐾 is the number of proposals in video 𝑉 𝑖 .The consistency in features for each action instance should accompany the consistency in labels [49] , we consider that the consistency constraints of attention scores for proposals enable the model to treat the action instances within the same category more equitable and could further eliminate the action-content confounding bias. More specifically, after getting proposal set 𝑆 𝑖 = (𝑠 𝑗 , 𝑒 𝑗 , 𝑐 𝑗 , 𝜑 𝑗 ) and representative snippets P 𝑖 𝑗 for video 𝑉 𝑖 , for each proposal (𝑠 𝑗 , 𝑒 𝑗 , 𝑐 𝑗 , 𝜑 𝑗 ), the attention score sequences of (𝑠 𝑗 , 𝑒 𝑗 , 𝑐 𝑗 , 𝜑 𝑗 ) can be got by:EQUATIONThe average attention score of the representative snippets P 𝑖 𝑗 can be calculated by:EQUATIONThen, we use Ā𝑟𝑒𝑝 𝑖 𝑗 to supervise all snippets in the proposal (𝑠 𝑗 , 𝑒 𝑗 , 𝑐 𝑗 , 𝜑 𝑗 ). The label-level consistency loss can be formulated as:EQUATIONOur final multi-level consistency mining loss function is expressed by integrating all the above constraints.L 𝑐𝑜𝑛𝑠 = 𝛽L 𝑓 𝑒𝑎𝑡 𝑐𝑜𝑛𝑠 + 𝛾 L 𝑙𝑎𝑏𝑒𝑙 𝑐𝑜𝑛𝑠 (12)where 𝛽 and 𝛾 are the weights to control the strength of the featurelevel and label-level consistency mining respectively.Optimizing Process. The overall training objective of the proposed method can be formulated as:L 𝑡𝑜𝑡𝑎𝑙 = L 𝑏𝑎𝑠𝑒 + L 𝑐𝑜𝑛𝑠 = L 𝑏𝑎𝑠𝑒 + 𝛽L 𝑓 𝑒𝑎 𝑐𝑜𝑛𝑠 + 𝛾 L 𝑙𝑎𝑏 𝑐𝑜𝑛𝑠 (13)where L 𝑏𝑎𝑠𝑒 is the train objectives of the baseline WTAL models, which including CO 2 -Net [10] , DELU [3] , DDG-Net [43] . More details about the experiments and results will be introduced in the following section. Model Inference. In the testing phase, we adhere to the same pipeline as the baseline WTAL models [10, 3, 43] to generate action proposals, and we employ the same test settings and postprocessing methods. 
experiments 4.1 datasets: 
 THUMOS14. THUMOS14 [13] dataset comprises 200 validation videos and 213 test videos that offer frame-wise annotations for 20 classes, and with an average of 15.5 actions per video. We train the model on the untrimmed videos in its validation set and evaluate it on the untrimmed videos from the test set following the same setting as [10, 3, 43] . ActivityNet-1.2: ActivityNet-1.2 [9] dataset covers 200 daily activities and provides 10,024 videos for training, 4,926 for validation and 5,044 for testing offering a larger benchmark for temporal action localization. We use all the training set to train model and all the validation set to evaluate our methodologies. Evaluation Metrics. We follow the standard evaluation protocol by reporting mean average precision (mAP) values. A prediction proposal is considered correct if the Intersection over Union (IoU) between the predicted proposal and the ground truth is deemed satisfactory. For evaluation purposes, we employ the code directly provided by the baseline WTAL models [10, 3, 43] . In subsequent experiments, † denotes results obtained on our platform using public code provided by the authors. Feature Extractor. Following previous work [33, 5, 10, 3, 43] , the optical flow maps are generated using the TV-L1 algorithm [50] , and we employ I3D network [1] pre-trained on the Kinetics dataset [15] to extract both RGB and FLOW features without fine-turning. Each video is segmented into 16-frame intervals following the settings in [10, 3, 43] .Training Settings. For a fair comparison, we utilize the same training hyperparameters as those used by the baseline WTAL models [10, 3, 43] . For THUMOS14 dataset, we set 𝛿 = 0.7, 𝜃 𝑟𝑒𝑝 = 0.8, 𝜃 𝑝𝑟𝑜 = 0.55, 𝛽 = 𝛾 = 0.5 when using CO 2 -Net [10] as baseline model, 𝛿 = 0.8, 𝜃 𝑟𝑒𝑝 = 0.9, 𝜃 𝑝𝑟𝑜 = 0.6, 𝛽 = 𝛾 = 0.5 when using DELU [3] as baseline model, 𝛿 = 0.9, 𝜃 𝑟𝑒𝑝 = 0.75, 𝜃 𝑝𝑟𝑜 = 0.86, 𝛽 = 𝛾 = 0.6 when using DDG-Net [43] as baseline model. Our method is implemented using PyTorch 2.2 and trained under Ubuntu Server 22.04 platform with single NVIDIA RTX 4070Ti GPU.To demonstrate the effectiveness and generalizability of our proposed methods, we deploy our methods on three different WTAL models on THUMOS14 dataset, which are CO 2 -Net [10] , DELU [3] and DDG-Net [43] . The results are shown in Table 1 , where we can see that our methods consistently outperform the baseline methods across average evaluation metrics. Specifically, the proposed method achieves an absolute improvement of 2.2%, 1.2% and 1.5% in terms of average mAP@IoU(0.1:0.7) over CO 2 -Net, DELU and DDG-Net respectively. Notably, the improvement at IoU = 0.7 for DDG-Net is minimal, potentially due to DDG-Net's focus on modeling ambiguous snippets, whereas our methods effectively reduce the impact of confounding from such snippets, which may slightly decrease performance. Table 2 presents the results on ActivityNet-1.2 Datasets for different WTAL methods. Compared with the baseline methods, our method still maintains a certain performance advantage, achieving an absolute improvement of 0.1% and 0.5% in average mAP@IoU(0.5:0.95) over CO 2 -Net and DDG-Net respectively.Experiments on both THUMOS14 and ActivityNet datasets confirm that our dual-confounding eliminating framework could realize 𝑃 (𝑌 |𝑑𝑜 (𝑉 )) estimation and improve the performance of WTAL methods. The improvement is achieved on different WTAL models, illustrating the effectiveness and generalizability of our method.We choose DDG-Net [43] as the baseline method to compare with state-of-the-art methods on THUMOS14 dataset. The comparison results are shown in Table 3 . This table shows that our method outperforms the baseline method and achieves competitive performance compared with the state-of-the-art methods. Specifically, Weakly our method achieves an absolute improvement of 1.5% in terms of average mAP@IoU(0.1:0.7) over the baseline method, and mAP has a significant improvement of 2.9% at the IoU of 0.2 without introducing additional learnable parameters and model re-designs. What's more, compared with some fully supervised methods, our method also achieves competitive performance, which demonstrates the advantages of our method in weakly supervised temporal action localization. Table 4 presents the comparison results on ActivityNet dataset. It can be observed that our method achieves an absolute improvement of 0.5% in terms of average mAP@IoU(0.5:0.95) over the baseline method, and mAP has a significant improvement of 0.8% at the IoU of 0.75. The results demonstrate the effectiveness and generalizability of our method on different datasets and WTAL methods.Effects of components. To investigate the effectiveness of each component in the proposed method, we conduct ablation studies on the THUMOS14 dataset with different WTAL methods. The results are shown in Fig. 5 . For different baseline methods, SCM has a significant improvement in AVG mAP@IoU(0.1:0.7). Specifically, an absolute improvement of 1.7%, 0.8% and 1.1% in terms of AVG mAP@IoU(0.1:0.7) is achieved over CO 2 -Net, DELU and DDG-Net respectively. These improvements demonstrate the SCM is effective in reducing the impact of confounding bias and improving the performance of WTAL methods. While only using MCM does not have a significant improvement in AVG mAP@IoU(0.1:0.7) over the baseline methods. However, when combining SCS and MCM, the proposed method achieves the best performance. Analysis of insight. Reviewing our methods, the dualconfounding eliminating framework is proposed to approximate the 𝑃 (𝑌 |𝑑𝑜 (𝑉 )) estimation, including SCS and MCM. SCS is designed to eliminate the confounding bias on action-context, and MCM is designed to reduce the impact of action-content bias. The prerequisite for MCM is the accurate identification of action regions without action-context confounding. Utilizing MCM alone does not yield significant performance improvements, likely due to the persistent confounding bias in action-context. If this bias is not effectively eliminated, MCM's effectiveness is compromised, resulting in negligible performance gains. However, with SCS effectively eliminating action-context confounding, the more accurate action regions are obtained. This improvement is beneficial for MCM, as it enables the enforcement of consistency between the discriminative snippets and their corresponding action instances. The synergistic combination of SCS and MCM achieves the optimal performance, demonstrating the effectiveness of our dual-confounding eliminating framework. Construction strategy of 𝛹. 𝛹 is constructed based on the modal gap between RGB and FLOW features. Despite Eq. 2, we also try others construction strategies, such asEQUATIONwhere 𝜉 is a threshold, the "+" represents the concatenation of two index lists. Also, we try to use the absolute difference and the ratio of the two modalities, which are defined asEQUATIONBesides, we also try to use the ratio of the two modalities, which is formulated asEQUATIONWe conduct experiments to analyze the effects of different construction strategies of 𝛹. The results are shown in Table 5 . It can be observed that the construction strategy of 𝛹 has a significant impact on the performance of our method. The proposed construction strategy in Eq. 2 achieves the best performance, which may be due to the fact that Eq. 2 could limit the values of 𝛹 to [0, 1], which is beneficial for the subsequent causal intervention. Strength of Causal Intervention. 𝜆 represents the strength of causal intervention. We conduct experiments to analyze the effects of 𝜆 on the performance of our method. The results are shown in Fig. 6 . It can be observed that the performance of our method is sensitive to 𝜆. Low 𝜆 may lead to a weak causal intervention, which may not eliminate the confounding bias effectively. Overall, with the increase of 𝜆, we observe an enhancement in overall performance, indicating that our SCS serves as an effective substitute confounder. 
conclusion: 
 In this paper, we revisit WTAL from the perspective of structural causal model, and propose a dual-confounding eliminating framework. Specifically, we construct a substitute confounder set to eliminate the action-context confounding bias by leveraging the modal gap between RGB and FLOW features, and enforce the consistency between the discriminative snippets and the corresponding action instances to reduce the impact of action-content bias. Our methods can be seamlessly integrated into any two-stream model without the addition of parameters, using the EM algorithm, at a low computational cost. Extensive experiments on the THUMOS14 and ActivityNet-1.2 datasets demonstrate the effectiveness and generalizability of our methods. Through detailed ablation studies and analysis, the critical role of each component within our framework is highlighted, underscoring their collective contribution to overall performance.  
: 
 annotations. To address this issue, weakly-supervised temporal action localization (WTAL) [59, 35, 7, 52, 21, 54, 37, 36] with only video-level annotations has recently gained intensive attention.Most WTAL approaches tackle the annotation sparsity challenge by reconceptualizing localization as a classification task to identify temporal regions most relevant to video-level classification [59, 35, 7, 52] . While these methods have achieved certain performances, they still suffer from confounding [26] arising from ambiguous snippets, leading to suboptimal results. Existing approaches usually categorize these ambiguous snippets under a newly devised "action-context" label, striving to segregate them from definitive action snippets. For instance, ACSNet [27] first brings in the assistant category "context" to assist in separating foreground action from context, while FACNet [11] utilizes class-wise foreground classification branch to regularize the relation between actions and foreground. Similarly, DDG-Net [43] designs a graph network to explicitly model ambiguous snippets. However, these methods often oversimplify ambiguous snippets into a singular, predefined category without investigating the underlying origins of their formation, consequently failing to eliminate the confounding bias efficiently.In light of these challenges, it is crucial to identify the underlying causes of confounding bias and mitigate them. Inspired by [60, 38] , we revisit WTAL from the perspective of structural causal model [4] . Specifically, existing WTAL models predict action label 𝑦 𝑖 𝑗 for every video snippet 𝑣 𝑖 𝑗 in video 𝑉 𝑖 via a classification model 𝑃 (𝑌 |𝑉 ). The scarcity of snippet-specific annotations causes models to excessively focus on most discriminative action snippets, thereby introducing biases on both action-context and action-content. More specifically, action-context owning visual or motion similarities with discriminative action snippets(as illustrated in Fig. 1 (a) and Fig. 1(b )) may erroneously be classified as action instances. When actioncontent exhibit some uniqueness diverging from the majority of discriminative action snippets, 𝑃 (𝑌 |𝑉 ) is inclined to assign them ambiguous confidence values, as shown in Fig. 2 . Generally, these bias on action-context and action-content is a significant challenge that has yet to be effectively addressed.Fortunately, structural causal model offers a solution through 𝑃 (𝑌 |𝑑𝑜 (𝑉 )) instead of 𝑃 (𝑌 |𝑉 ) to eliminate the confounding bias. The 𝑑𝑜 (•) operation [4] denotes the causality between the cause 𝑉 and the effect 𝑌 without confounders [34] . The ideal way to calculate 𝑃 (𝑌 |𝑑𝑜 (𝑉 )) is to "physically" intervene 𝑉 , such as removing all ambiguous snippets in every action instance. Obviously, it is impossible to intervene the video snippets in practice.To address these challenges, we propose an efficient dualconfounding eliminating framework to mitigate the confounding bias on action-context and action-content through two core components, respectively. The first component, termed Substituted Confounder Set (SCS) is constructed by leveraging the modal disparity between RGB and optical FLOW. Then, SCS is applied in a backdoor adjustment procedure [34] to address context-related bias. Unlike previous methods [43, 26] , the construction of SCS does not require an elaborate network architecture or additional learnable parameters, making it more efficient and effective. The second component, named Multi-level Consistency Mining (MCM), is designed to tackle action-content bias by ensuring consistency between discriminative snippets and respective proposals on both feature and label levels. Different from previous works [12, 65, 14, 19] selecting the representative action embeddings across the entire dataset, yet disregarding the individualized differences among specific action instances, MCM selects the representative snippets from corresponding proposals for each action instance, to mitigate the action-content confounding on these less discriminative snippets by enforcing the model to learn the consistency across each action instance. Ultimately, by integrating the current WTAL models with the SCS and MCM components, we can effectively approximate 𝑃 (𝑌 |𝑑𝑜 (𝑉 )) using the Expectation-Maximization (EM) algorithm, while maintaining low computational costs.In summary, our main contributions are summarized as follows: (1) We revisit WTAL from the perspective of structural causal model, and propose an efficient dual-confounding eliminating framework to mitigate the confounding bias. (2) We construct a Substituted Confounder Set (SCS) to specifically eliminate the confounding bias on action-context by exploiting the modal disparity between RGB and FLOW. (3) We present the Multi-level Consistency Mining (MCM) method to address the action-content bias, utilizing consistency between discriminative snippets and associated proposals on both feature and label levels. (4)The SCS and MCM could be applied to any two-stream model without additional learnable parameters using EM algorithm. Extensive experiments on THUMOS14 [13] and ActivityNet-1.2 [9] demonstrate the superior performance of our methods. 
 
 Appendix:  


Decision: This paper revisits WTAL from the perspective of structural causal model to identify the true origins of confounding, and propose an efficient dual-confounding eliminating framework to alleviate these biases. All reviewers are satisfied with the rebuttal and unanimously recommend acceptance of the paper.

Reviews: review: paper_summary: paper_summary: The authors revisit Weakly-supervised Temporal Action Localization (WTAL) from the perspective of the structural causal model to identify the true origins of confounding and propose an efficient dual-confounding eliminating framework to alleviate these biases. This framework consists of Substituted Confounder Set (SCS) and Multi-level Consistency Mining (MCM). The former is designed to eliminate the confounding bias on action-content by leveraging the modal disparity between RGB and FLOW. The latter is responsible for mitigating the confounding bias on action-content by utilizing the consistency between discriminative snippets and corresponding proposals at both the feature and label levels.
strength_weakness: strength_weakness: 1. The authors revisit WTAL from the perspective of the structural causal model to identify the true origins of confounding.
2. The motivation of this paper is clear and novel.
3. Extensive experiments on THUMOS14 and ActivityNet-1.2 demonstrate the superior performance of the proposed method.
limitations: limitations: 1. The term 'dual-confounding' is ambiguous; it is unclear whether it denotes two distinct solutions or refers to two cofounders.
2. Could you please specify which section of Figure 3 corresponds to the I element depicted in Figure 4?
3. For enhanced comprehension, it would be beneficial if the concept of 'cutting off C->V' is clearly illustrated in Figure 3.
paper_summary: paper_summary: Guided by Causal Intervention, this paper propose an efficient dualconfounding eliminating framework to mitigate the confounder on action-context and action-content.
strength_weakness: strength_weakness: This work revisit WTAL from the perspective of structural causal model to identify the true origins of confounding, and propose an efficient dual-confounding eliminating framework to alleviate these biases. Experiments on 2 challenging benchmarks including THUMOS14 and ActivityNet-1.2 demonstrate the superior performance of our method.
limitations: limitations: 1. Is Eq.3 preprocessed or used at each epoch.

2. The proposed context regularization method is limited by dual streams.

3. In my views，the confounding action context and content are also accumulated by different action instances. Therefore, the action instance also seems to determine the action priors. 

4. In Eq.3, the differences between RGB and OF are used to measure the confounder.  However, when the samples of both modalities are simultaneously confounding？ The difference doesn't seem to exceed the threshold either.  

In summary, the method in this paper looks like a simple regularization of context and content. But the author utilizes causal intervention to endow regularity with interpretability. And this manuscript is reasonable and well-organized.
paper_summary: paper_summary: In this paper, the authors revisit Weakly-supervised Temporal Action Localization~(WTAL) from the perspective of the structural causal model to identify the true origins of confounding and propose an efficient dual-confounding eliminating framework to alleviate these biases. Specifically, a Substituted Confounder Set (SCS) is constructed to eliminate the confounding bias on action-content by leveraging the modal disparity between RGB and FLOW. A Multi-level Consistency Mining (MCM) method is designed to mitigate the confounding bias on action-content by utilizing the consistency between discriminative snippets and corresponding proposals at both the feature and label levels. Extensive experiments demonstrate the superior performance of the method.
strength_weakness: strength_weakness: 1. Well written;
2. Detailed ablation studies.
limitations: limitations: 1. The authors claim in Fig. 1~(a) and~(b) that the previous method exists Visual confounding and Motion confounding, how can the above problems be seen from the visual similarity? In addition, why Fig. 1. does not provide the results of this paper's method to show effectiveness. The same doubt exists for Fig. 2.
2. Why didn't this work compare Sota results on other dataset benchmarks, e.g. FinaAction, FineGym?
3. High-quality visualization results are missing in the experimental part to further demonstrate the advantages of the method.
4. What is the relation between SCS & MCM and the structural causal model?
5. In the SOTA comparison, do all the methods use both RGB and optical flow modalities?
paper_summary: paper_summary: This paper revisits WTAL from the perspective of structural causal model to identify the true origins of confounding, and propose an efficient dual-confounding eliminating framework to alleviate these biases. Specifically, a Substituted Confounder Set (SCS) is developed to eliminate confounding bias in action-content by exploiting the modal disparity between RGB and FLOW. Additionally, a Multi-level Consistency Mining (MCM) method is designed to reduce confounding bias in action-content by leveraging the consistency between discriminative snippets and their corresponding proposals at both the feature and label levels. Extensive experiments on two popular benchmarks demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. As far as I'm concerned, few existing methods have considered to adopt structural causal models to identify the origins of confounding. This should be a valuable direction for future research.

2. The paper is well-written and easy to understand. The model performance is strong. Extensive experiments demonstrate the effectiveness of the method. I like this paper generally.

3. I appreciate that the authors include the source code in the supplementary material.
limitations: limitations: 1. I am curious that whether the model performance is sensitive to the hyperparameters $\beta$ and $\gamma$ in Eq.(13). An experiment of parameter analysis may help readers better understand the method.