Modeling Event-level Causal Representation for  Video Classification

Authors: yuqing  wang
lei  meng
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 14:29:36

Abstract: Classifying videos differs from that of images in the need to capture the information on what has happened, instead of what is in the frames. Conventional methods typically follow the data-driven approach, which uses transformer-based attention models to extract and aggregate the features of video frames as the representation of the entire video. However, this approach tends to extract the object information of frames and may face difficulties in classifying the classes talking about events, such as "fixing bicycle". To address this issue, This paper presents an Event-level Causal Representation Learning (ECRL) model for the spatio-temporal modeling of both the in-frame object interactions and their cross-frame temporal correlations. Specifically, ECRL first employs a Frame-to-Video Causal Modeling (F2VCM) module, which simultaneously builds the in-frame causal graph with the background and foreground information and models their cross-frame correlations to construct a video-level causal graph. Subsequently, a Causality-aware Event-level Representation Inference (CERI) module is introduced to eliminate the spurious correlations in contexts and objects via the back- and front-door interventions, respectively. The former involves visual context de-biasing to filter out background confounders, while the latter employs global-local causal attention to capture event-level visual information. Experimental results on two benchmarking datasets verified that ECRL may better capture the cross-frame correlations to describe videos in event-level features. The source code is provided in the supplementary material.

Main paper: 
 introduction: 
 Video classification aims to automatically identify and categorize video content into predefined categories. This enables the efficient retrieval of video from the extensive corpus of the internet. Videos are not just a collection of static images, but a comprehensive concept characterized by the cross-frame interconnections among scenes, events, and objects. In video classification, the modeling of any specific entity is invariably affected by the presence of the other two types of entity, thereby impeding its process of representation learning. Therefore, how to learn the discriminate representations of the predicted targets from the complex spatio-temporal dynamics is a permanent challenge in video classification tasks.To address the spatio-temporal dynamics issue in video classification, existing studies attempt to achieve the video understanding by aggregating all the frame representations via the spatio-temporal convolution [11, 28] and the self-attention mechanism [2, 54, 64] . The first type of these studies conducts the sparse sampling on video sequences to achieve the local feature aggregation [43] . The second type fully leverages the unsaturated discriminative power of residual attention to understand the long-term temporal dependencies in video sequences [17] . As stated in the second line of Fig. 1 , these aforementioned methods are profoundly influenced by the configurations of their visual encoders, which results in an excessive concentration on the object knowledge within each frame. This leads to a mechanical aggregation of object-level features from each frame, thereby impeding their in-depth exploration of cross-frame event correlations. In contrast, the event-level representation in the third line of Fig. 1 can precisely highlight the position of "hand" and "whole wheel" during the video sequences modeling, thereby correctly understanding the category of "fixing bicycle".Causal representation learning (CRL), benefiting from its interpretability, has achieved outstanding performance in video understanding, including Video Moment Retrieval [61] and Video Question Answering [26, 27] . This arises from the capacity of CRL to disentangle video representations in a frame-by-frame manner through the logical reasoning process. It effectively eliminates the inherent confounding factors that precipitate errors in model recognition, thereby facilitating the mining of event-level knowledge from the temporal correlations among objects within the video sequences. These methods typically focus on leveraging the additional textual knowledge to construct the Structural Causal Model (SCM), thereby guiding the visual representation modeling by the multi-model representation alignment [25] . However, these above CRL-based methods struggle to be directly applied in video classification tasks where semantic information is unavailable. Therefore, how to utilize CRL to model the event-level knowledge from the video sequences remains an urgent problem to be addressed.To address this issue, we propose a Event-Level Causal Representation Learning (ECRL) model for video classification, which leverages the causal theory to discover the association among inter-frame objects in video sequences and guide the event-level representations modeling in current video classification algorithms. As illustrated in Fig. 2 , ECRL is comprised of two principal components: the Frame-to-Video Causal Modeling (F2VCM) module and the Causality-aware Event-level Representation Inference (CERI) module. Specifically, F2VCM first systematically extracts the background and foreground content from the raw video sequences to highlight the indispensable knowledge in each frame. To discover the event-lavel causal correlation, it simultaneously models the in-frame object interactions and the cross-frame object correlations on these continuous back-and fore-ground frames to construct an event-level causal graph. Recognizing the spurious associations between the ground-truth category with the background elements and the intra-frame knowledge, we argue that video classification is subject to background biases and data biases in video sequences. Therefore, CERI is introduced to alleviate these inherent biases via the Background Debiasing (BD) component and the Global-Local Causal Attention (GLCA) component. Firstly, BD conducts backdoor interventions to reduce the spurious associations such as "beach, sky -> playing volleyball", thereby alleviating the background biases. In contrast, GLCA implements front-door interventions to remove data biases within the inter-frame object correlations, highlighting the indispensable frames and filter the dispensable frames. With F2VCM and CERI, ECRL is able to effectively constructs the video-level causal graph with the intra-frame information and interframe correlations, thereby enabling the structured interventions to model the informative event-level causal representations.We have conducted extensive experiments on two benchmark datasets to verify the effectiveness of ECRL, including performance comparison, ablation study, in-depth analyses, and case studies. These discoveries demonstrate the efficacious mechanism of ECRL, which models video-level causal structures with in-frame object interactions and cross-frame correlations. Additionally, it refines event-level causal representations through multi-perspective causal interventions, thereby enhancing the accuracy of video classification tasks. To summarize, this paper includes three contributions:• We propose an Event-level Causal Representation Learning method to model the event-level spatio-temporal representations from the causal perspective. To the best of our knowledge, this is the pioneering solution that models the event-level representation via causal theory in video classification. • We introduce two effective F2VCM and CERI modules to construct the video-level causal graph with the intra-and inter-frame correlations and model the event-level causal representation via the multi-perspective causal interventions. • We conduct the visualization analysis to highlight the ability of ECRL to mitigate the erroneous predictions stemming from the inherent background bias and data bias, which enables the precise event-level video understanding of ECRL. operation. S3D [57] introduces temporal separable convolution and spatio-temporal feature gating to improve the performance of I3D. TSM [21] introduces temporal shift operations. SlowFast [12] consists of a slow path and a fast path. TEA [19] adjusts spatio-temporal features with motion features to enhance motion patterns. X3D [11] expands 2D networks across dimensions including time, space, depth, and width. The emergence of Transformers has advanced video classification. ViViT [2] introduces a transformer model for video classification, offering variants that consider both temporal and spatial dimensions. TimeSformer [5] uses individual attention mechanisms within each patch block, aiding in modeling long sequence videos. Swin Transformer [27] merges CNNs and transformers, converting spatio-temporal attention using localized induction bias and non-overlapping windows.Existing methods explore spatio-temporal video dynamics but overlook video and background biases, limiting event-related representation and performance. Thus, learning causal event representations via cross-frame correlations is crucial for video classification. 
causal inference in video understanding: 
 Causal inference is an analytical tool designed to infer the dynamics of events under changing conditions (e.g. different treatments or external interventions) [36] . Compared to conventional debiasing techniques [49, 53] , causal inference [37, 51, 61] shows its potential in reducing spurious correlations [4] and disentangling model effects [6] , thereby achieving better generalisation performance. Causal inference and counterfactual reasoning have received increasing attention in a variety of tasks in computer vision [18, 22, 38] , including visual explanation [20, 24, 25, 39, 52] , scene graph generation [55] , image recognition [9, 18, 33, 40, 65, 66] , video analysis, and visual-language tasks [1, 62] . Existing causal theories are relatively mature in the field of images and are also widely applied in the domain of videos. Causal graphs are used to establish causal-effect inference for multimodal video summarization [16] . Dataset selection bias can result in spurious correlations between speech and video in video grounding tasks [35] . Causal inference is used to address imbalances in the distribution of the datasets. The counterfactual-based model ensemble in video anomaly detection integrates long-term and local image contexts for effective anomaly detection [56] . Causal methods in video moment detection are used to address spurious correlations between long-tail annotations, user queries, and moment positions. [61] . Causal methods are widely used in Video Question Answering (VQA) [55] due to biases in datasets related to visual, linguistic, and annotation aspects. These methods enhance the robustness of visual question answering by exploring potential causal relationships in complex spatio-temporal scenarios.However, existing causal methods in the video domain focus on finding correlations between visual and textual data, require information other than the video data (e.g., textual information ), and thus cannot be directly applied to video classification tasks. The challenge of video classification is the representation of events. It is important to design a causal model that is suitable for the video classification task and to find the directly related to the event information representation via the causal mechanism.  
problem formulation: 
 F 𝑣 = M v (V), where M v (•) denotes the feature extractor. Then, the category mapping P (•) predicts the category of the sample, i.e. P (F 𝑣 ) → ȳ.Different from the conventional approaches, the proposed video classification method ECRL, based on event-level causal representation learning, first utilizes obtaining the foreground and background information V 𝑠 = {𝑠 𝑖 | 𝑖 = 1, . . . , 𝑡 }, and V 𝑐 = {𝑐 𝑖 | 𝑖 = 1, . . . , 𝑡 }. Then, the original frame-level of visual features F 𝑣 = M v (V), foreground features F 𝑠 = M s (V 𝑠 ) and background feature F 𝑐 = M c (V 𝑐 ), where M s (•) and M c (•) denote the visual feature extraction network. To construct the causal graph of the video, we further model the relation between F 𝑠 and F 𝑟𝑠 = R s (F 𝑠 ) , F 𝑟𝑐 = R 𝑐 (F 𝑐 ) to get content-level representations of F 𝑟𝑠 and background F 𝑟𝑐 , where, R s (•) and R c (•) denote the networks for cross-frame correlation modeling. To address background bias and data bias in videos, we process F 𝑟𝑠 and F 𝑟𝑐 via BD module and GLCA module, obtain the background debiased feature F cc and the causal representation F gg . Finally, we calculate the predictive classification results by mapping the two features P (F cc ) + P F gg → ȳ. 
methods: 
 ECRL introduces an Event-level Causal Representation Learning method to enhance the model's causal awareness of event information. The architecture of ECRL is illustrated in Fig. 3 . An event-level causal graph is constructed using the Frame-to-Video Causal Modeling (F2VCM) module, which explores the event correlations by finding the correlation between the foreground and background of the video frames; the Causality-aware Event-level Representation Inference (CERI) module eliminates background and data bias in video data by implementing causal intervention on the causal graph. This enables the model to find information directly related to event representation and the causal structure in video classification tasks.The Frame-to-Video Causal Modeling (F2VCM) module aims to build the in-frame causal graph and learn cross-frame causal correlations to create a Structural Causal Model (SCM) suitable for video classification to assist causal inference. It consists of two main processes: extracting related visual elements and constructing content-level representations by modeling inter-frame correlations between the foreground and background visual elements.Learning. This module aims to extract three types of visual information: foreground, background, and original video frame features. Given a data set of videos D = {V 𝑖 | 𝑖 = 1, . . . , 𝑁 }, we distinguish the foreground S 𝑖 from V 𝑖 using a pre-trained Saliency model [23] and then implement a masking process to obtain the corresponding contextual background C 𝑖 , where V 𝑖 is a video sample. Next, we extract features using the specified encoder.EQUATIONwhere M 𝑣 (•), M 𝑠 (•), and M 𝑐 (•) represent the encoders for the original video, foreground, and background frames respectively. 𝐹 𝑣𝑖 , 𝐹 𝑠𝑖 , 𝐹 𝑐𝑖 ∈ R 𝑡 ×𝑑 , t denotes the number of frames, and 𝑑 denotes the dimension of the extracted features. EQUATIONwhere H (., .) denotes the Hadamard inner product operation, 𝜙 denotes the softmax function, ⊙ represents the dot product operation and 𝜌 means concat and linear operation. We obtain the representations for video, foreground, and background 𝐹 𝐿 , 𝐹 𝑟𝑠 , 𝐹 𝑟𝑐 ∈ R 𝑑 . 𝑽 → 𝑪 → 𝒀 , 𝑽 → 𝑺 → 𝒀 : 𝑆 represents the foreground feature. These two causal pathways reflect the estimation of 𝑌 based on unbiased contextual features 𝐶 and content of video frames 𝑉 .𝑽 ← 𝒁 1 → 𝑪 → 𝒀 : We use the backdoor intervention to cut off the adverse backdoor path and eliminate the impact of 𝑍 1 , which misleads the spurious "context->category" correlation (Fig. 4 (b) ).𝑴 ← 𝑽 ← 𝒁 2 → 𝒀 : We use a mediator M to construct the front door path and adopt front door intervention to mitigate the spatio-temporal data bias in the video. (Fig. 4 (c)) .Training a video classification model focuses on learning true causal effects. It aims to discern category 𝑌 based on key events in the video, avoiding spurious correlations from confounders 𝑍 1 and 𝑍 2 . By cutting off these two paths, the causal structure in video classification is discovered (Fig. 4 (d) ).Through the implementation of the F2VCM module, we have established an event-level SCM and conducted targeted causal interventions to derive precise event causal representations. The where 𝑓 𝑟𝑠 (•) and 𝑓 𝑟𝑐 (•) represent CCM module encoding functions.To mitigate this observation bias caused by 𝑍 1 , an intuitive idea is to intervene on 𝑉 to ensure that each contextual semantics contributes equally to the video classification. To address this issue, we stratify 𝑍 1 based on backdoor adjustments [37] , and then estimate the average causal effect based on the context proportion of training samples and implement intervention on 𝑉 :EQUATIONwhere 𝑑𝑜 (𝑉 ) denote intervene on 𝑉 . By implementing do-calculus on 𝑉 , the path in Fig. 4 (b) from 𝑍 1 to 𝑉 is cut-off, and the model will approximate causal intervention 𝑃 (𝑌 | 𝑑𝑜 (𝑉 )), rather than a spurious association 𝑃 (𝑌 | 𝑉 ).Confounder Dictionary 𝑍 1 . Given the difficulty of capturing all real-world confounders, we use a stratified confounder dictionary 𝑍 1 = [𝑧 1 , 𝑧 2 , . . . , 𝑧 𝐿 ] structured as an 𝐿 × 𝑑 matrix, where 𝐿 is the context category size and 𝑑 is the background feature dimension, pre-trained using the UniformerV2 model. For practical applications, we define the confounder pool as 𝑀 𝑏 = {𝒎 𝑘 |𝑘 = 1, . . . , 𝑁 }, with 𝑁 representing the number of training samples in the dataset. We apply k-means++ [3] and principal component analysis [41] to derive 𝑍 1 from the confounder pool, ensuring each 𝑧 𝑖 reflects contextual clustering. Specifically, each 𝑧 𝑖 is the average feature of its cluster, calculated as𝑧 𝑖 = 1 𝜎 𝑖 𝜎 𝑖 𝑗=1 𝑚 𝑖 𝑗 ,where 𝜎 is the count of contextual features in the 𝑖-th cluster.Instantiation of the Proposed BDM. To implement the theoretical and imaginative interventions in Eq 4. The BDM is described as follows. As the computation of 𝑃 (𝑌 | 𝑑𝑜 (𝑉 )) requires multiple pass backs for all z, to reduce computation, we use Normalised Weighted Geometric Mean (NWGM) [59] to approximate the results expected from the above feature layers: (5) We parameterize the network model to approximate the conditional probability of Eq. 5, inspired by [48] , as follows:EQUATION)where 𝑊 𝑎 ∈ R 𝑑 𝑚 ×𝑑 𝑎 and 𝑊 𝑏 ∈ R 𝑑 𝑚 ×𝑑 are learnable parameters. Additionally, 𝐽 = 𝜓 (𝑠, 𝑐) ∈ R 𝑑 𝑎 ×1 , where 𝜓 (•) represents a fusion strategy, integrating 𝑠 and 𝑐 into the joint representation 𝐽 . The above approximation is justified as the influence on 𝑌 comes from 𝑣, 𝑐, and the confounding factor 𝑧. Therefore, we approximate E 𝑧 [𝑔 1 (𝑧)] as a weighted integration of all background prototypes.EQUATIONwhere 𝜇 𝑖 represents the important weight coefficient measuring the interaction between each 𝑧 𝑖 and the original feature 𝐽 , with 𝑃 (𝑧 𝑖 ) = 𝑁 𝑖 /𝑁 𝑚 . In practice, we employ two implementation approaches for𝜇 𝑖 : Dot Product (softmax (𝑊 𝑐 𝐽 ) 𝑇 (𝑊 𝑑 𝑧 𝑖 )/ √ 𝑑 ) and Additive At- tention (softmax 𝑊 𝑇 𝑒 • Tanh (𝑊 𝑐 𝐽 + 𝑊 𝑑 𝑧 𝑖 ) ), where 𝑊 𝑒 ∈ R 𝑑 𝑛 ×1 , 𝑊 𝑐 ∈ R 𝑑 𝑛 ×𝑑 𝑎 , and 𝑊 𝑑 ∈ R 𝑑 𝑛 ×𝑑 are mapping matrices.By using the contextual debiasing module, we end up with a debiased visual feature of 𝐹 𝑐𝑐 . We then input this feature into the classifier to get the predicted score 𝑆 1 , where 𝑆 1 = classifier (𝐹 𝑐𝑐 ).Causal Intervention. It is very complex and difficult to observe this data bias 𝑍 2 , where not all content in the video is assigned to a category. Fortunately, the front-door adjustment is a feasible way to compute 𝑃 (𝑌 | 𝑉 ) when 𝑍 2 is unobservable (Fig. 4 (c) ). Insert a mediator M between V and Y, we can construct a front door path 𝑉 → 𝑀 → 𝑌 to transmit the knowledge, where 𝑚 denotes the knowledge selected from M:EQUATIONThe classification predictor can be represented by 𝑉 → 𝑀 and 𝑀 → 𝑌 . The intervention probability 𝑃 (𝑌 | do(𝑉 )) as follows:EQUATIONIn Fig. 4 (c), for the causal link 𝑉 → 𝑀, the backdoor path between V and M is blocked by 𝑀 ← 𝑉 ← 𝑍 2 → 𝑌 . Therefore, the intervention probability is equal to the conditional probability.EQUATION𝑀 → 𝑌 : To achieve 𝑃 (𝑌 | do(𝑀 = 𝑚)), we can indirectly cut off the link 𝑀 ← 𝑉 to block the backdoor path 𝑀 ← 𝑉 ← 𝑍 2 → 𝑌 .EQUATIONIn summary, by applying Eq.9 and Eq.10 into Eq.11, we can compute the true causal effect between V and Y.EQUATIONWe parameterize 𝑃 (𝑌 | 𝑉 , 𝑀) as 𝑔 2 (•) followed by a softmax layer, as most visual tasks are formulated similarly.EQUATIONwhere 𝜙 denotes the softmax function. To address the cost of forwarding for all samples is prohibitively high, we still adopt the NWGM approximation, as shown in the formula.EQUATIONwhere M and V stand for the estimates of 𝑀 and 𝑉 , while 𝑓 (•) and ℎ(•) denote the network mapping functions.Global Clustering Dictionary 𝐺: To improve representation of causally perceived visual features, the GLCA module estimates M and V within a unified attention framework. Local features 𝐹 𝐿 come from the input, while global features 𝐹 𝐺 are sampled from a K-means initialized and training-updated dictionary 𝐺. M and V are derived from local-local 𝐹 𝐿𝐿 and local-global 𝐹 𝐿𝐺 features, respectively. We illustrate using 𝐹 𝐿𝐺 . First, the global-local fusion obtains the fusion feature ℎ = 𝑊 𝑣 𝐹 𝐺 ,𝑊 𝑞 𝐹 𝐿 ⊙ 𝑊 𝑘 𝐹 𝐺 , where [., .] denotes the join operation, ⊙ denotes the Hadamard inner product, 𝑊 𝑞 ,𝑊 𝑘 ,𝑊 𝑣 ,𝑊 ℎ ′ denote the linear layer weights. Then it goes to the attention unit attention weights 𝛼, and weights the local features to obtain the global-local feature 𝐹 𝐿𝐺 , the process is as follows:EQUATIONwhere 𝑏 ℎ and 𝑏 ℎ ′ denote the linear layer bias. The local-local visual feature 𝐹 𝐿𝐿 can be obtained and combined with 𝐹 𝐿𝐿 and 𝐹 𝐿𝐺 to obtain the causal representation after the visual front door intervention as 𝐹 𝑔𝑔 . We then input this feature into the classifier to get the predicted score 𝑆2, where 𝑆 2 = classifier 𝐹 𝑔𝑔 . The bias 𝐵 1 and 𝐵 2 in the video are eliminated using the Eventdetected Causal Graph Construction Module. This results in the causal-aware structure for video classification shown in Fig. 4(d) .In this module, the main objective is to integrate the predicted scores 𝑆 1 , 𝑆 2 , obtained from the BD and GLCA modules respectively, to derive the final video classification category. we fusion the scores 𝑆 1 , 𝑆 2 , obtaining the 𝑆, where 𝑆 = 𝑆 1 + 𝑆 2 . The prediction 𝑝 = 𝑒 𝑆 𝑗 𝑒 𝑆 , where 𝑝 represents the predicted final outcome. A single-stage training model is used to update the model by minimizing the prediction loss L 𝑐𝑒 = 𝐶𝐸 (𝑝, 𝑝 ′ ), where CE is the cross-entropy loss, 𝑝 is the predicted result, and 𝑝 ′ is the ground truth label.Training ECRL from scratch is challenging due to uneven feature distributions. To address this, we use a two-step strategy: pre-train on raw video frames to extract features and calculate losses, then [58] and ActivityNet [7] to verify the effectiveness of ECRL. Their statistics are shown in Table 1 . MSR-VTT dataset comprises 10,000 video clips. We split it into 9,000 and 1,000 videos for training and testing. ActivityNet consists of 20,000 videos. We clipped the video durations to 0-30 seconds. We remove video samples that lack labels. We split the data into 1,009 and 4,515 videos for training and testing, respectively.5.1.2 Evaluation Criteria. Following conventional video classification methods [64] , we leverage the Accuracy@k with 𝑘 = {1, 5} to evaluate the video classification performance. 
implementation details.: 
 Following the pre-training of the large model of UniformerV2 [17] , we set 768 as the feature dimension. We use the Adam optimizer with a learning rate chosen from 1e-6 to 1e-4. The decay rate of the learning rate parameter is selected from 0.1 and 0.5, and the decay interval is 4 epochs. The batch size is set to 8, and the UniformerV2-based model requires 4-6 hours for training. For confounder dictionary construction, the backdoor confounder dictionary is sized at 256 for MSR-VTT and 512 for ActivityNet, while the fron-tdoor dictionary 𝑍 1 is uniformly set at 512 with a dimension of 768 across all datasets.We compare ECRL with 12 video classification methods, including X3D [11] , TANet [28] , TDN [46] , GC-TDN [15] , H 2 CN [44] , LAPS [63] ,ViT [10] , ViViT [2] , TokShift [64] , TimeSformer [5] , Video-Swin [27] , and UniformerV2 [17] . Note that UniformerV2 is one of the SOTA methods, and leverage it as the base model. Only the ViT model leverages pre-trained ImageNet weights, while the remaining models utilize weights pre-trained on Kinetics400. Specifically, for ViViT and X3D, we sample 16 video frames for input. For the other models, we select 8 video frames as input. The best performance is marked in bold. The following observations can be drawn from • ECRL demonstrates superior performance on two distinct datasets due to its meticulous modeling of category-related eventlevel information in video classification, enhancing the model's causal perception and overall efficacy. • The ECRL achieves its most notable performance enhancement on the MSR-VTT. This improvement is primarily due to the dataset's wide variety of events, highlighting our model's ability to accurately capture event information in videos.In this section, we further studied the working mechanism of each module of ECRL, as shown in Table 3 . The best performance is marked in bold. The following findings could be observed:• Using only foreground (+FCM) or background information (+BCM) significantly reduces model performance, highlighting that background information is crucial for complete video representation. However, this reliance can also cause confusion. 5.4.1 Effectiveness Analyses of the background confounders dictionary 𝑍 1 . As delineated in Table 5 , we scrutinized the efficacy of the background confounder dictionary 𝑍 1 within the BD module. Experimental results show that replacing 𝑍 1 with a random dictionary significantly deteriorates performance, validating our contextual prototypes. Using average video features as a confounder dictionary is less effective than class-average background features, indicating that random dictionaries and class averages are insufficient context confounders and highlight the impact of inherent background biases on model generalization. these biases, making feature distributions distinct and enabling the learning of causal representations of events.To evaluate ECRL's efficacy, we compared it with Uni-formerV2 on the ActivityNet validation dataset, examining their attention to causally relevant visual information in videos. Using GradCAM [42] to generate heatmaps, we observed significant differences. ECRL mitigates erroneous predictions caused by background biases, as seen in Fig. 6  
object-level features: 
 ( UniformerV2 )time Figure 1 : Object-level features v.s. Event-level features. In the case of "fixing bicycle" from ActivityNet, UniformerV2 misclassifies it as "assembling bicycle". It stems from overly focusing on the object-level feature of the "whole wheel". 
 
 Appendix:  


Decision: This paper proposes a novel approach for video classification that learns causal representations by integrating temporal and contextual cues from multiple modalities, to enhance video understanding and classification. 
The paper is well-structured, and clearly written.  It has a high degree of novelty and it is technically sound.
All reviewers recommended to accept the paper and I recommend it for oral presentation.

Reviews: review: paper_summary: paper_summary: The manuscript proposes an Event-level Causal Representation Learning (ECRL) model for video classification, which leverages the causal theory to discover the association among inter-frame objects in video sequences and guide the event-level representations modeling. The ECRL manuscript model consists of two principal components: the Frame-to-Video Causal Modeling (F2VCM) module and the Causality-aware Event-level Representation Inference (CERI) module. The F2VCM module builds the in-frame causal graph and learns cross-frame causal correlations to create a Structural Causal Model (SCM) suitable for video classification to assist causal inference. The CERI module eliminates background and data bias in video data by implementing causal intervention on the causal graph. The ECRL model is able to effectively construct the video-level causal graph with the intra-frame information and inter-frame correlations, thereby enabling the structured interventions to model the informative event-level causal representations.
strength_weakness: strength_weakness: The paper is technically sound, presenting a clear methodology for constructing and utilizing causal graphs. The use of back-door and front-door interventions to eliminate spurious correlations showcases a deep understanding of causal inference techniques. The integration of these techniques within a video classification framework is executed with precision, ensuring the model's reliability and correctness.
limitations: limitations: 1. The author seems to have not provided an analysis of the computational complexity of the proposed model. The evaluation focuses primarily on accuracy improvements. However, other important metrics such as computational efficiency, memory usage, and latency are not thoroughly discussed. Including these metrics would provide a more comprehensive understanding of the model's performance and practicality.
2. It is necessary to distinguish vectors and scalars in the formula. Please use bold to indicate vectors.
3. The interpretability of the ECRL model is poor, making it difficult to understand the decision-making process of the model. Although causal graphs are inherently more interpretable than purely data-driven approaches, the complexity and high-dimensional nature of video data can still make the resulting graphs difficult to interpret.
4. How to integrate the results of backdoor and front door interventions. What impact will different fusion methods have on the results?
paper_summary: paper_summary: The submission introduces a novel model, ECRL, which focuses on event-level causal representation for video classification. The ECRL consists of two modules i.e., F2VCM and CERI. The F2VCM module constructs an in-frame causal graph and models cross-frame correlations to build a video-level causal graph. Another one is the CERI module which uses backdoor interventions for visual context de-biasing and front-door interventions through global-local causal attention to focus on event-level visual information. The classification results on MSR-VTT and ActivityNet verify the effectiveness of ECRL.
strength_weakness: strength_weakness: - The proposed ECRL with its F2VCM and CERI modules, offers a well-structured methodology for discovering causal correlations and reducing biases in video classification tasks.
- The submission is well-structured, with a clear presentation of the problem, methodology, experiments, and results.
limitations: limitations: Cons:
- The difference between CRL-based Video QA and ECRL should be clarified. What are the major differences between CRL-base Video QA and the proposed ECRL? It seems that it is simple to adopt a video CRL-based video QA model for video classification since it also has the ability for video understanding. More specifically, what are the differences between CMCIR [1] and the proposed ECRL, except for the missing text modality?
- The reasons for evaluating the proposed method on MSR-VTT and ActivityNet should be clarified. Do the categories of them contain any event-level information? These two datasets seem not to be used often for video classification.

Typos:
 - Line 115, 'multi-model' -> 'multi-modal'

Ref:
[1] Cross-modal causal relational reasoning for event-level visual question answering.
paper_summary: paper_summary: This paper proposes to classify actions with causal methods. Specifically, it extracts three representations from foreground (objects), background (scene), and full videos. With these three representations, it builds in-frame and cross-frame causal graphs. Experiments on several video understanding benchmarks show the effectiveness of the methods.
strength_weakness: strength_weakness: 1. Unlike current methods that use neural networks for video classification, this paper uses causal graphs for video content analysis, which provides better explainability.
2. Experimental results on several video classification benchmarks show the effectiveness of these methods.
limitations: limitations: 1. The causal representation is quite complex for video classification. Does it have limitations in applying to different benchmarks？(robustness)
2. The presentations of method could be further improved, current version is kind of complex and not easy to follow.
paper_summary: paper_summary: The paper introduces an Event-level Causal Representation Learning (ECRL) model for video classification, addressing the limitations of conventional methods that struggle with event-level classification. ECRL incorporates two main components: the Frame-to-Video Causal Modeling (F2VCM) module and the Causality-aware Event-level Representation Inference (CERI) module. F2VCM builds a video-level causal graph by extracting and modeling foreground and background information. CERI eliminates spurious correlations via backdoor and front-door interventions. Experimental results on benchmark datasets demonstrate that ECRL effectively captures cross-frame correlations and reduces biases, enhancing event-level video classification accuracy.
strength_weakness: strength_weakness: Innovative Approach: The ECRL model introduces a novel way to handle video classification by focusing on event-level causal relationships, which is a significant improvement over traditional object-level methods. This is also the first time I have seen problem-solving from a causal reasoning perspective in video classification tasks, the paper's approach is relatively innovative.

Effective implementation: The combination of F2VCM and CERI modules offers a robust framework that captures both intra-frame and inter-frame correlations, addressing both background and data biases. The paper effectively combines ideas with relevant implementation techniques of deep learning.

Empirical Validation: The experimental results on multiple benchmark datasets (MSR-VTT and ActivityNet) show that ECRL outperforms state-of-the-art methods, validating the effectiveness of the proposed approach.

Detailed Analysis: The paper provides extensive ablation studies, in-depth analyses, and visualizations to explain the contributions of different components of ECRL. These experiments further confirm that the various components proposed by the author are not redundant. For the visualization of the heat map in Figure 6, it is impressive to see the advantages of the proposed framework compared to traditional frameworks.
limitations: limitations: Complexity: The proposed model's complexity, involving multiple modules and interventions, may pose challenges in terms of computational efficiency . Therefore, you should add a comparison term for the number of model parameters or computational complexity in Table 2.

Paper expression: There are certain problems in the writing and image expression of the article. 

(1) Firstly, the lack of a period in line 191 is an issue that cannot be ignored.

(2) Although the paper drew many figure, the mismatch between the explanations and the figure actually caused greater reading confusion. 

(a). Like line144-line147, the modules involved in this explanation (at least the abbreviations of the modules) should be indicated in Figure.
 
(b). What are the full names of BCM and FCM in Figure 3? I did not see a clear full name in the article. From the content, should lines 385-397 describe these two modules? In addition, the title CCM module of 4.1.2 is not indicated in Figure 3. 

(c). Line 497，What is the full name of BDM?

(d). 4.2.2，Which part of the figure 3 does GLCA correspond to? It should be indicated in the figure.

(3)  For computer vision papers, the formulas in the article are biased towards theoretical descriptions, which is not a conference manuscript in the field of machine learning. Formulas that lean towards mathematical descriptions should not be used.

A far fetched explanation: Why does the model only focus on the hand in the first frame of the visualization in Figure 1?

Note: The readability of a manuscript largely determines its quality! I hope the author can correct the expression issues in the manuscript, I still tend to improve my score.