Probabilistic Vision-Language Representation for Weakly Supervised Temporal Action Localization

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 14:08:10

Abstract: Weakly supervised temporal action localization (WTAL) aims to detect action instances in untrimmed videos with only video-level annotations. As many existing works optimize WTAL models based on action classification labels, they encounter the task discrepancy problem (i.e., localization-by-classification). To tackle this issue, recent studies have attempted to utilize action category names as auxiliary semantic knowledge with vision-language pre-training (VLP). However, there are still areas where existing research falls short. Previous approaches primarily focused on leveraging textual information from language models but overlooked the alignment of dynamic human action and VLP knowledge in joint space. Furthermore, the deterministic representation employed in previous studies struggles to capture fine-grained human motion. To address these problems, we propose a novel framework that aligns human action knowledge and VLP knowledge in the probabilistic embedding space. Moreover, we propose intra- and inter-distribution contrastive learning to enhance the probabilistic embedding space based on statistical similarities. Extensive experiments and ablation studies reveal that our method significantly outperforms all previous state-of-the-art methods. Our code will be available after publication.

Main paper: 
 introduction: 
 The development of multimedia services, such as YouTube and Netflix, has led to increasing interest in the field of computer vision for analyzing long-form videos. Temporal Action Localization (TAL) refers to the problem of precisely determining the time intervals in a lengthy, untrimmed video when human activities occur, which is of fundamental significance in video understanding [15, 30, 31] .Fully supervised TAL [3, 4, 26, 27, 49, 58, 59 ] handles this task based on frame-level, rich annotations. Despite its great success, training a TAL model in dense frame-level annotation poses challenges in that the cost of annotation is extremely high and its generality is limited. To address these challenges, weakly supervised temporal action localization (WTAL) [9, 11, 18, 21, 24, 33, 45] , which only requires video-level categorical labels, has received a lot of attention. In scenarios where only video-level category supervision is accessible, existing WTAL methods solve the localization problem as a classification problem that selects discriminative snippets 1contributing mainly to video-level classification. However, these classification-based approaches mainly suffer from the task discrepancy problem, which results from a localization-by-classification framework. To deal with this problem, a lot of research has been done on snippet-level pseudo label approaches [12, 25, 42, 65 ]. Yet, these pseudo labels, constrained by video-level annotations, inherently carry noisy proposals and fall short of achieving the desired accuracy.Recently, to tackle these problems, some approaches [17, 24] leveraging action category text information have been proposed to guide powerful semantic knowledge without incurring extra annotation costs. These approaches establish additional learning cues by exploiting category text embedding vectors instead of merely utilizing category information as one-hot vectors. While significant improvements were achieved through additional semantic information, some factors were overlooked in earlier research. First, Li et al. [24] adopted a language model (e.g., GloVe [38] ) pre-trained on only text modality, which results in an inadequate initialization state with respect to the alignment with the human action pre-trained visual feature. Second, Chen et al. [17] proposed an alternate optimization strategy to introduce an effective distillation framework. However, the proposed alternate optimization scheme necessitates manual identification of the optimal settings in accordance with the dataset. Third, most importantly, we observe that the utilization of deterministic representation in previous studies for incorporating text information is not suitable for human action understanding.To confirm this, we conducted an analysis of the zero-shot classification with CLIP [39] , a prominent study in the domain of VLP. As shown in Figure 1 (a), we compared the similarity response between the text prompt (i.e., "a frame of [CLS]") representation and the corresponding frame visual representation. It reveals a high level of activation even when actual human actions do not occur, as long as there is visual relevance to the action text category. This is because CLIP was pretrained, considering only one-to-one matching between a single image and its caption. The previous research depicted in Figure 1 (b) cannot address the aforementioned issue as it solely relies on deterministic representation via one-to-one matching, making it challenging to capture fine-grained human motion. Furthermore, the lack of consideration for direct alignment with pre-trained human action knowledge results in insufficient temporal dynamics modeling.To overcome this issue, we introduce a novel framework, PVLR, Probabilistic Vision Language Representation for Weakly supervised Temporal Action Localization, which integrates VLP knowledge and human action knowledge within the probabilistic embedding space, as shown in Figure 1 (c). To begin with, pre-trained human action knowledge, such as Kinetics [2] , is utilized to construct a probabilistic embedding space. In this step, probabilistic adapters are introduced to estimate parameters for the snippet-level probability distribution. Subsequently, we transfer the large-scale VLP knowledge to the estimated probability distribution to create a joint probabilistic embedding space. To capture the temporal dynamics of action, we obtain samples from the estimated probability distribution to offer diverse perspectives, many-to-one matching, and then evaluate their similarity with category text embedding via Monte-Carlo estimation.Furthermore, to learn distinctive embedding space, we propose a distribution contrastive learning scheme to capture the statistical similarity between distributions. We enhance intra-class compactness by learning the similarity of content (action or background) within videos and maximize inter-class separability by leveraging action category information across videos. To enhance the intraclass compactness, we draw inspiration from snippet mining in prior work [57] to differentiate a similar snippet distribution among related content. For inter-class separability, we build a video-level probabilistic distribution based on Gaussian mixture model (GMM) and make the mixture distribution separable between different action classes. To the best of our knowledge, this is the first attempt to investigate multimodal probabilistic representations for weakly supervised temporal action localization.Our main contributions to this work are summarized as follows:(1) We introduce a novel framework that aligns VLP knowledge and action knowledge within a probabilistic space to make full consideration of temporal dynamics for fine-grained motion modeling. (2) We introduce an intra-and inter-distribution contrastive strategy to construct distinctive probabilistic embedding space.(3) We conduct extensive experiments and ablation studies to reveal the significance of probabilistic embedding and the proposed method, and our superior performance on two public benchmarks (THUMOS14 and ActivityNet v1.3). 
related work 2.1 weakly supervised temporal action localization: 
 Weakly supervised temporal action localization (WTAL) is proposed to alleviate the laborious annotation procedure for Temporal Action Localization, training with only video-level labels. In the early stages of research, Multiple Instance Learning (MIL)-based approaches [10, 14, 21, 22, 33, 37, 46] were proposed, which pretended a video as a bag of multiple action and background instances. Zhang et al. [57] introduced a snippet contrast loss, refining the representation of ambiguous instances in the feature space through snippet mining and contrastive learning. Afterward, several approaches [12, 25, 42, 65] generate snippet-level pseudo labels to explicitly guide the model as a localization-by-localization framework. However, pseudo labels generated based on video-level supervision were inaccurate and noisy, making it challenging to achieve the desired performance.Recently, approaches utilizing the semantic information of action category names have emerged to address the fundamental absence of temporal annotation in WTAL [17, 24] . Li et al. [24] designed a novel framework with a discriminative objective to enlarge inter-class differences and a generative objective to enhance intra-class integrity via text information. Chen et al. [17] proposed a novel distillation and collaboration framework with complementary Classification Based Pre-training (CBP) and Vision-Language Pre-training (VLP) branches. While these works distinguish themselves with promising performances without additional annotation costs, there is yet potential for further development. In our framework, we integrate VLP knowledge and human action knowledge within the probabilistic space previously unexplored in existing literature, enabling a better understanding of human action.Vision language pre-training (VLP) learns a joint representation through large-scale image-text pair datasets with consistent contextual information. A representative work is CLIP [39] , mapping image-text pairs with consistent contextual information into the visual and textual encoders separately and facilitating the learning of a joint embedding space through aligned representations. CLIP has shown great success in many image understanding tasks, including image classification [6, 34] , semantic segmentation [20, 40] , image generation [7, 43] , and visual question answering [35] . Building upon the success of CLIP in the image domain, some research efforts [29] [30] [31] 54] aiming to leverage the vision-language representation of CLIP in the video domain have emerged. Our work is also a contribution to the research aimed at extending VLP knowledge into the realm of untrimmed video and human action understanding.The main idea of probabilistic embedding is to map inputs to probability distributions in the embedding space. To achieve this objective, the desired distributions are estimated by a deep neural network and optimized to maximize their likelihood. PCME [5] represents one-to-many relationships in the joint embedding space with uncertainty estimation and introduces a soft cross-modal contrastive loss. Park et al. [36] proposed self-supervised video representation learning that bridges contrastive learning with probabilistic embedding with Gaussian mixture model. ProbVLM [50] utilizes a probabilistic adapter that, without the requirement of extensive datasets or intensive computing, estimates probability distributions for the embeddings of a vision-language pre-trained model through inter-and intra-modal alignment in a post-hoc manner. The objective of this study is to transfer the knowledge of a pre-trained vision-language model into the probabilistic embedding space, with an explicit objective of strengthening human action understanding. 
method: 
 In this section, we provide a detailed explanation of the proposed approach. We first describe our baseline approach in Sec 3.1. We then introduce the probabilistic embedding space with VLP knowledge for WTAL problem in Sec 3.2 and discuss the proposed intraand inter-distribution contrastive learning in Sec 3.3. An illustration of the overall framework is presented in Figure 2 .In the WTAL problem setting, a set of untrimmed videos {𝑉 𝑖 } 𝑁 𝑖=1 and their corresponding video-level category labels {y 𝑖 } 𝑁 𝑖=1 are given. Specifically, the video-level category label is a multi-hot vector y 𝑖 ∈ {0, 1} 𝐶+1 , where 𝐶 is the number of action classes. Since all untrimmed videos have various background regions, we add an auxiliary class to model the background. Due to the computational memory limit, we split each video into multi-frame, non-overlapping snippets and sample a fixed number 𝑇 of snippets to address the large variations in the video length. WTAL aims to predict a sequence of actions {𝑐 𝑖 , 𝑠 𝑖 , 𝑒 𝑖 , 𝑝 𝑖 } 𝑀 𝑖=1 for an input video 𝑉 𝑛 , where 𝑀 is the number of proposals, 𝑐 𝑖 is the action category, 𝑠 𝑖 and 𝑒 𝑖 are the start and end time of each proposal, and 𝑝 𝑖 is confidence score.Following prior works [10, 21, 57] , it is common practice to use a pre-trained extractor for snippet representation. We extract the RGB features X 𝑅 = x 𝑟 𝑡 𝑇 𝑡 =1 and Optical Flow features X 𝑂 = x 𝑜 𝑡 𝑇 𝑡 =1 , which allows us to model temporal dynamics efficiently when embedding 𝐷-dimensional features x 𝑟 𝑡 ∈ R 𝐷 and x 𝑜 𝑡 ∈ R 𝐷 for each snippet. Afterwards, we concatenate features from each modality [X 𝑅 ; X 𝑂 ] ∈ R 𝑇 ×2𝐷 and feed into base WTAL head 𝑓 𝑏𝑎𝑠𝑒 to generate base feature X 𝐵 ∈ R 𝑇 ×2𝐷 , written as:EQUATIONwhere 𝑓 𝑏𝑎𝑠𝑒 is mainly implemented with a series of temporal convolution with ReLU activation. In addition, an attention weight a ∈ R 𝑇 ×1 is generated to differentiate between the foreground and the background region:EQUATIONwhere A (•) is an attention branch consisting of several temporal convolutional layers. Following the MIL framework, we feed the base feature X 𝐵 to classification head 𝑓 𝑐𝑙𝑠 to generate the base class activation sequence (CAS) is defined as:EQUATIONThen we aggregate snippet-level activation scores, to obtain video-level class prediction p 𝑏𝑎𝑠𝑒 = K (S 𝑏𝑎𝑠𝑒 ) ∈ R 𝐶+1 , where K (•) denotes the top-k average pooling along the temporal axis. After obtaining the video-level category prediction, we can build a loss function L 𝑏𝑎𝑠𝑒 with cross-entropy loss as follows:EQUATIONwherey 𝑏𝑎𝑠𝑒 = [𝑦 1 , • • • , 𝑦 𝐶 , 1] ∈ R 𝐶+1is the video-level label with an auxiliary background class. Meanwhile, background suppressed CAS is acquired by multiplying the attention weight S 𝑠𝑢𝑝𝑝 = a ⊗ S 𝑏𝑎𝑠𝑒 . We can also build a loss function L 𝑠𝑢𝑝𝑝 with background suppressed video-level class score p 𝑠𝑢𝑝𝑝 = K (S 𝑠𝑢𝑝𝑝 ) ∈ R 𝐶+1 as follows:EQUATIONwherey 𝑠𝑢𝑝𝑝 = [𝑦 1 , • • • , 𝑦 𝐶 , 0] ∈ R 𝐶+1is the video-level label without a background class. By optimizing L 𝑐𝑙𝑠 = L 𝑏𝑎𝑠𝑒 + L 𝑠𝑢𝑝𝑝 , the model learns to distinguish snippets that contribute significantly to video-level action classification. Moreover, we also utilized L 𝑜𝑝𝑝𝑜 , L 𝑛𝑜𝑟𝑚 , L 𝑔𝑢𝑖𝑑𝑒 . Since these losses have been proposed in previous works [14, 21, 22, 32, 37] , we do not claim our contribution in this part. The overall objective of baseline approach L 𝑣𝑖𝑑 is as follows:EQUATIONIn this section, we reformulate CAS as a probabilistic class activation sequence (P-CAS) designed to effectively leverage VLP knowledge within a probabilistic embedding space. To achieve this, we model the probabilistic distribution 𝑝 z|x (z|𝜃 ) and estimate the parameters 𝜃 , optimizing neural networks via human action and VLP knowledge.Embedding. Initially, we establish probabilistic embedding space by leveraging the human action pre-trained knowledge on Kinetics [2] . From the base feature X 𝐵 = {x 𝑡 } 𝑇 𝑡 =1 ∈ R 𝑇 ×2𝐷 , we formulate a snippet-level probability distribution 𝑝 (z|x 𝑡 ) as a multivariate Gaussian distribution with a mean vector and a diagonal covariance matrix to model the probabilistic embedding space:EQUATIONwhere 𝑔 𝜇 is an embedding layer that estimates the mean vector 𝑔 𝜇 (x 𝑡 ) ∈ R 𝐷 and 𝑔 Σ is an embedding layer that estimates covariance matrix 𝑔 Σ (x 𝑡 ) ∈ R 𝐷 of targeted Gaussian. With the estimated 𝑝 (z|x 𝑡 ), we can sample 𝐾 random embeddings z (𝑘 ) ∈ R 𝐷 that can represent the estimated distribution following [19] :EQUATIONwhere 𝜖 (𝑘 ) ∈ R 𝐷 are independently and identically sampled from a 𝐷-dimensional unit Gaussian. Our goal is to utilize 𝐾 embeddings sampled from the estimated probability distribution for each snippet to capture the human action from a more diverse range of perspectives. Also, for textual information, we transform action category names into pre-trained embeddings. For this, we freeze the CLIP text transformer Ψ C (•) and extract the embeddings X C = {x 𝑐 } 𝐶 𝑐=1 :EQUATIONwhere L 𝑠 , L 𝑒 are learnable tokens, 𝑡 𝑐 refers to action category, and Ψ 𝑒𝑚𝑏 is word embedding layer. Then, P-CAS can be defined by determining the action confidence score along the temporal axis with the estimated probability distribution and action category representation. Specifically, to measure the confidence score between the estimated distribution and category representation, we formulate P-CAS as S 𝑝𝑟𝑜𝑏 ∈ R 𝑇 × (𝐶+1) via Monte-Carlo estimation:EQUATIONwhere sim(•) means cosine similarity and 𝜏 is temperature parameter. Besides, to avoid ambiguity among action categories, we design an orthogonal loss L 𝑜𝑟𝑡ℎ𝑜 ensuring the uniqueness of each category representation as:EQUATION)where I is the identity matrix and ∥•∥ 2 F is the Frobenius norm of a matrix.Nevertheless, during the estimation of the present probability distribution, only the textual information from VLP is utilized, overlooking the alignment between human action knowledge and the visual representation provided by VLP. Therefore, we aim to integrate VLP visual knowledge into the estimation process of the probability distribution.Due to deterministic pre-training of CLIP, estimating the entire distribution can be challenging, but large-scale pre-trained representation can offer a generalized point approximation (e.g., mean vector) for the desired distribution. To achieve this, our probabilistic embedding utilizes the CLIP's deterministic representations as estimates for the mean, 𝑔 𝜇 (x 𝑡 ) of the targeted distribution. To transfer VLP knowledge into probabilistic embedding space, we first sample a set of frames {𝑓 𝑡 } 𝑇 𝑡 =1 within a fixed temporal stride from the video. Next, we freeze the CLIP Image Encoder Ψ I (•) and extract the embeddings X I = {Ψ I (𝑓 𝑡 )} 𝑇 𝑡 =1 ∈ R 𝑇 ×𝐷 . For a pair of snippet feature and CLIP image feature (x 𝑏 𝑡 , x 𝑖 𝑡 ), the distillation loss L 𝑘𝑑 is defined as:EQUATION)We utilize the rescaled cosine similarity between the estimated mean 𝑔 𝜇 (x 𝑏 𝑡 ) and CLIP image representation x 𝑖 𝑡 as the matching score. The objective of L 𝑘𝑑 is to align the estimated mean 𝑔 𝜇 (x 𝑏 𝑡 ) with the generalized fixed point x 𝑖 𝑡 of CLIP embedding for transferring pre-trained CLIP knowledge into the desired probability distribution.We defined a probabilistic embedding space by aligning human action knowledge and VLP knowledge. However the crucial factor of distributional similarity remained unexplored. The distributions corresponding to human actions should exhibit similarities amongst themselves while contrasting with background distributions. To address these objectives, we aim to enhance the completeness of the probabilistic embedding space through probabilistic representation learning based on statistical distances.Learning. We begin by considering contrastive learning between distributions within the video. In the video, the distributions of actions are expected to share distributional similarities internally while being separate from the background. To achieve these objectives, we adopt the snippet mining algorithm of zhang et al. [57] , which uses attention weight a ∈ R 𝑇 ×1 to differentiate action and background snippet within the video. Firstly, to mine the action and background snippet, we threshold the attention weight (1 or 0 indicates the action or background, respectively):EQUATIONwhere 𝜃 𝑏 is the threshold value. We utilize the same strategy performing, cascaded dilation or erosion operations to identify challenging samples (hard to differentiate) at the action/background boundaries.EQUATIONEQUATIONwhere (•) -and (•) + represent the binary erosion and dilation operations with mask respectively. Following earlier work [57] , we consider the inner regions as hard action snippet sets, and the outer regions are considered as hard background snippet sets. Here, we define hard actions as having a positive relation P 𝑎𝑐𝑡 with easy actions (top-k). Similarly, we define hard backgrounds as having a positive relation P 𝑏𝑘𝑔 with easy backgrounds (bottom-k). Note that we do not claim technical contribution over snippet mining. Instead, our main contribution is to develop effective intra-distribution contrastive learning for probabilistic embedding space. After snippet mining, we utilize KL divergence as a statistical metric for measuring the similarity of snippet distributions. The KL divergence between multivariate Gaussian is defined as:EQUATION)For intra-class compactness of embedding space, we propose an intra-contrastive loss L 𝑖𝑛𝑡𝑟𝑎 to refine the snippet-level distribution similarity. Finally, the intra-contrastive loss L 𝑖𝑛𝑡𝑟𝑎 is formulated as:EQUATION)For the matching probability 𝑝 (N ), we simply formulated as the Jensen-Shannon divergence JSD(N 𝑃 , N 𝑄 ) to decrease the divergence of distributions corresponding for the positive pairs and maximize the divergence for the negative pairs.Learning. We further introduce inter-distribution contrastive learning utilizing action category labels to ensure inter-class separability. Here, we represent the whole video distribution 𝑝 (z|𝑉 ) as a Gaussian mixture model (GMM) to measure video-level similarity,EQUATIONFor estimating 𝑝 (z|𝑉 ), we adopt the attention weight a ∈ R 𝑇 as a mixing coefficient to appropriately mix distributions based on the actionness score. Given video-level category labels, we formulate a self-similarity map H ∈ R 𝑁 ×𝑁 (1 for the same class, 0 for different) to characterize relationships between videos. Similar to intra-contrastive learning, we compute the matching probabilities between 𝑁 videos within a mini-batch across mixture models, then enhance inter-video representation by comparing them with the self-similarity map. Finally, the inter-contrastive loss L 𝑖𝑛𝑡𝑒𝑟 is formulated as:EQUATIONwhere L BCE is a binary cross entroy loss.Beyond aligning with VLP knowledge and the probabilistic embedding space, our proposed contrastive learning framework poses constraints on the suggested probabilistic representation to ensure compliance with both intra-compactness and inter-separability.Objectives. Given all the previously mentioned objectives, the total objective L 𝑡𝑜𝑡𝑎𝑙 of the entire framework is determined as:EQUATIONwhere 𝛼, 𝛽, 𝛾 are the hyper-parameters to balance these loss terms and L 𝑝𝑟𝑜𝑏 = L 𝑖𝑛𝑡𝑟𝑎 + L 𝑖𝑛𝑡𝑒𝑟 .  
experiments 4.1 experimental settings: 
 We conduct experiments on two popular WTAL benchmarks: THU-MOS14 [13] and ActivityNet v1.3 [1] . THUMOS14 is a widely used benchmark for the WTAL problem. It contains 200 validation videos and 213 test videos for 20 sports categories. Following previous works [10, 42, 57] , we use 200 validation videos to train our framework and use 213 test videos for evaluation. In WTAL, THUMOS14 is the most challenging dataset because of the motion blur, significant intra-class varieties, and extremely short action instances. ActivityNet v1.3 has 10,024 training videos, 4,926 validation videos, and 5,044 testing videos from 200 action categories. Since annotations for the testing set are not released, we train on the training set and test on the validation set. Challenges in ActivityNet usually lie in numerous action categories. Following the standard evaluation metrics, we evaluate our method with mean Average Precision (mAP) under different Intersection over Union (IoU) thresholds on the temporal axis. We adopt the same evaluation code from previous works [10, 14] for fair comparisons. Furthermore, in the inference phase, we computed the action confidence score using the estimated mean without utilizing the reparameterization strategy.To conduct experiments on the THUMOS14 dataset and Activi-tyNet v1.3 dataset, we first divide each video into non-overlapping segments consisting of 16 frames. Subsequently, we extract the 1024-dimensional RGB and optical flow features from the I3D network [2] pre-trained on the Kinetics400 dataset. We utilized the TV-L1 algorithm to extract the flow features. The fixed number of segments 𝑇 is set to 320 and 60 for THUMOS14 and ActivityNet v1.3, respectively. For the base WTAL model, it can be any existing WTAL method and we use the CO 2 -Net [10] as the base head for its simple framework. We adopt ResNet-50 as a backbone network for the CLIP image encoder Ψ I . It is worth noting that the I3D network and the CLIP encoders are not fine-tuned during training.For CLIP image feature, we divide the video as described above and the middle frame of each snippet is fed into Ψ I . For the probabilistic adapter, 𝑔 𝜇 indicates a single linear layer, while 𝑔 Σ is a separate network with a linear layer followed by the ReLU function, to ensure the Σ remains positive definite. Similar to previous work [16] , we prepend and append 4 prompt vectors to word embedding Ψ 𝑒𝑚𝑏 (𝑡 𝑐 ), which is initialized with N (0, 0.01). In P-CAS, we used the learnable vector to model the background class, which is hard to characterize. Our experiments are conducted on an NVIDIA Tesla V100 GPU. In this section, we compare our proposed PVLR with previous state-of-the-art methods. For the THUMOS14 [13] , it is evident that the proposed PVLR outperforms the performance of all previous state-of-the-art methods as shown in Table 1 . Especially in the WTAL scenarios, where performance under high IoU(0.5-0.7) is of particular importance, our proposed method distinguishes itself by surpassing the performance of all existing methodologies. In direct comparison to prior studies [17, 24] that also incorporate textual information, our approach exhibits superior performance, with a margin ranging from 0.9% to 4.3% when it comes to the important criterion of average mAP (0.1:0.7). Additionally, our approach was shown to either outperform or reach similar performance levels as recent fully supervised methods. In Table 2 , results for the larger dataset ActivityNet v1.3 are presented. Similarly, our proposed PVLR demonstrates superior performance compared to existing weakly supervised state-of-the-art methods.To demonstrate the effectiveness of our model components, we analyze the impact of each component in this section with THU-MOS14 [13] . Table 3 presents the effect of the proposed components in comparison to the baseline approach [10] . The VLP knowledge distillation module serves as a pivotal step within our framework, marking the inception of our approach. By conducting feature alignment in a probabilistic space, PVLR introduces a fundamental basis that was previously overlooked in earlier literature. By integrating VLP knowledge, we realized a performance boost of 2.4% through the implementation of a probabilistic class activation sequence (P-CAS). Also, refinement of our probabilistic embedding space, which benefited from the introduction of distribution contrastive learning, led to a 2.0% improvement in efficacy. Finally, an evident gain of 1.3% can be obtained by introducing the orthogonalization of text embedding which enhances the discriminative capacity between text category embeddings. We have ultimately demonstrated the effectiveness of our proposed module by achieving a performance improvement of 5.7%, a level that has been difficult to find in previous research.To provide deeper insights into the design aspect of our proposed framework, we conducted several experiments in this section.As the initiative procedure in our framework, the probabilistic representation is of great importance. However, to compare its significance, we developed a simple baseline using deterministic representation. For the deterministic baseline, we conducted experiments utilizing one-to-one matching between human action knowledge and text embedding without probabilistic adapter. In Table 4 , the first row "Deterministic CAS" indicates the deterministic baseline. Quantitatively comparing, there is a performance decrease of about 3.1% compared to the proposed probabilistic approach in Table 4 . Also, we compare qualitative visualization results of selected videos from the THU-MOS14 dataset. Figure 3 illustrates that the deterministic approach frequently produces predictions beyond the ground truth boundaries, struggling to capture subtle variations in human action. In contrast, the probabilistic method effectively models temporal dynamics, focusing its predictions on the segments where real actions unfold. In Figure 3 (a), the probabilistic approach appears to struggle with completely filling the GT segment, yet this specific area, characterized by an absence of motion change, is designated for future exploration. From Table 4 , besides the KL divergence, other statistical metrics are also suitable for our contrastive learning. Table 4 reveals that metrics capable of assessing inter-distributional similarity exhibit relatively consistent performance with minimal variation. By not relying on a specific distance metric, it can be considered that a well-generalized probability distribution has been estimated, leading to the successful modeling of a probabilistic embedding space. Finally, the marginal superiority of KL divergence led to its utilization for the proposed distribution contrastive learning.4.5.2 Number of 𝐾 samples. During the generation of P-CAS, we reparameterized 𝐾 samples of the estimated snippet distribution to compute the similarity between the estimated distribution and the action category text embedding. To analyze the impact of the number of samples, we compared the performances under different values of 𝐾, as shown in Table 5 . As observed, a small value for 𝐾 leads to suboptimal performance, resulting in a lack of representation of the estimated distribution. Here, we denoted the previously described deterministic baseline as 𝐾 = 0. Considering that larger values of 𝐾 capture the entire distribution of the snippet through Monte-Carlo estimation, performance improves with an increase in 𝐾. Nevertheless, an increased value for 𝐾 results in higher computational demands. Calculating the confidence score for composing a P-CAS requires computations on the order of O (𝐾) for each snippet and action category. Considering the computational overhead, we decided on 𝐾 = 20.In Table 6 , we demonstrate the generality of our contributions by integrating them into previous works in a plug-and-play manner. To achieve this, we conducted comparative experiments by substituting the base WTAL head with previous works [21, 57] . The additional training modules exclusively considered the proposed probabilistic adapter for probabilistic embedding. Furthermore, we reformulated the classification objective using probabilistic class activation sequences (P-CAS). The results show that our framework enhances their performance by an average mAP increase ranging from 2% to 3%, indicating robust generalization across various methods and model architecture designs. 
conclusion and future works: 
 In this work, we present a novel framework that leverages largescale vision-language pre-training (VLP) to address weakly supervised temporal action localization (WTAL). Our motivation stemmed from the observation that the deterministic representation with VLP may not be optimal for WTAL. Furthermore, we noticed that previous studies did not consider the alignment between human action and VLP knowledge. To address these concerns, we introduce a probabilistic embedding framework aligned with human action and VLP knowledge, enhanced by distribution contrastive learning. Our method significantly outperforms previous approaches on two prominent datasets, revealing the efficacy of probabilistic embedding within the VLP representation. However, the exploration of probabilistic embedding for text data solely represented by action category names remains unexplored. For future work, we will explore leveraging the recently acclaimed large-language model (LLM) to generate attributes for each action category and subsequently integrate them with probabilistic embeddings. We believe that the probabilistic embedding with a vision-language pretrained model will be a promising direction for various weakly supervised and unsupervised learning tasks.  
 
 Appendix: In our field, a snippet refers to a set of consecutive frames composed of 16 frames. 


Decision: This paper proposes a probabilistic vision-language representation learning method for Weakly Supervised Temporal Action Localization, which uses the CLIP and distribution contrastive learning to achieve good alignments between human action knowledge and VLP knowledge in the probabilistic embedding space. In the rebuttal, the questions and concerns of reviewers have been addressed and most reviewers reach a consensus on the acceptance of this submission.  

Quality: The idea of probabilistic representation learning on Temporal Action Localization is interesting and effective in weakly supervised settings. 

Clarity: The paper is well-written, but the presentation about method details and illustration should be improved.

Originality: The novelty of this work is sufficient for acceptance.

Significance： The proposed framework has wide applications based on probabilistic vision-language representation learning and has the potential usage on exiting VLM downstream tasks.

Reviews: review: paper_summary: paper_summary: This paper tackles the task of weakly supervised temporal action localization. The authors propose a probabilistic modeling framework that utilizes CLIP. Experiments conducted on the Thumos and ActivityNet datasets demonstrate that their methods achieve superior results compared to existing approaches.
strength_weakness: strength_weakness: 1. The idea of replacing deterministic modeling with probabilistic approaches is interesting.

2. Experiments on the Thumos and ActivityNet datasets show improvement with the proposed methods.
limitations: limitations: **Major concerns**:

1. The final training objective actually contains an excessive number of loss terms (more than 9 items), including 
$$
L_{kd}, L_{norm}, L_{ortho}, L_{oppo}, L_{guide}, L_{intra}, L_{inter}, L_{base}, L_{supp}
$$
and so forth. This multitude of terms complicates implementation, particularly in determining hyperparameters such as the loss weights for each term.  Are all nine of these items truly valuable? The author should consider eliminating redundant terms and simplifying the training objectives to enhance clarity.

2. In Table 3, the author reference [10] as their baseline. But a disparity emerges between [10] and the loss function delineated in Equation 6. Notably, the main contribution of [10] is to introduce a consensus loss, which is absent from this paper's loss function. Why do the metric values in Table 3 remain unchanged compared to those in reference [10]?

**Other concerns**:
1. Section 3.2 illustrates how to model $p(z \mid \theta)$. However, the variable $z$ is introduced here without any prior explanation. What does $z$ denotes here?

2. The paper uses the term "vision-language pretraining" (VLP) in a misleading way, indicating that VLP is part of their proposed methods. However, there is actually no VLP in their proposed method.  For instance, the following sentences inaccurately suggest that vision-language pretraining is part of their methods:
i) In the "Conclusion" section, the author claims they "leverage large-scale vision-language pre-training to address the WAL task.”
ii) In Figure 2 of the "Overview of Proposed Methods," the author labels part b of their method as "vision-language pretraining."  
The author's intended meaning should be that they utilize "pretrained vision-language models" rather than conducting "vision-language pretraining" themselves.

3. The implementation details for hyperparameter configuration are lacking, specifically for parameters such as $\alpha$, $\beta$, and $\gamma$ in Eq. 20, and $\lambda_1$, $\lambda_2$, $\lambda_3$, $\lambda_4$ in Eq. 6.  
Additionally, details for other hyperparameters like batch size, and learning rate are missing.

**Note**: I would raise my rating If my major concerns are addressed.
paper_summary: paper_summary: The paper proposes a VLM-enhanced probabilistic representation learning method for Weakly Supervised Temporal Action Localization (WSTAL).  
The text-encoder of CLIP is used to encoder the semantics of categories labels and an orthogonal loss is proposed to enhance the uniqueness representations for each category. 
The visual-encoder of CLIP is used to encode visual input to ensure a good mean representation of the learned probabilistic embedding.  
Distribution Contrastive Learning is proposed to ensure a well learned probabilistic embedding.
strength_weakness: strength_weakness: 1. Probabilistic representations and corresponding Distribution Contrastive Learning is a novel idea to me for the Weakly Supervised Temporal Action Localization (WSTAL).  
2. Experiments in commonly used datasets demonstrated the effectiveness of the proposed method, achieving the best performance.
limitations: limitations: 1. The probabilistic representations are implemented by learning a mean and variances, which is not an inspired technology. And there are methods have exploration the uncertainty of representations, such as [1], which is also a probabilistic representation and using a distribution-based distance metric. 
2. The method present is hard to read, and there are some missing explanations for symbol in the equation, such as $m, \mathcal{M}$ in Eq.(14) and Eq.(15), $\mathcal{P}, \mathcal{N}_p, \mathcal{N}_Q$ in Eq.(17).
3. The results of the paper achieve best compared with existing methods, but the paper additionally used a pre-trained CLIP, which is not fair for the existing methods that only use the I3D features. I am wondering the results of adding the proposed orthogonal loss of category representations and distillation loss of visual representations to existing method.
4. The results of table 6 can be integrated into table 3 that changing the backbone for ablation study to make the proposed method more convincing. 

[1] Chang J, Lan Z, Cheng C, et al. Data uncertainty learning in face recognition[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 5710-5719.
paper_summary: paper_summary: This paper introduces a novel framework for Weakly Supervised Temporal Action Localization (WTAL), which integrates Vision-Language Pre-training (VLP) knowledge and human action knowledge within a probabilistic embedding space to better capture temporal dynamics and fine-grained human motion. Extensive experiments and ablation studies demonstrate the effectiveness of the proposed probabilistic embedding and the overall method, showing significant improvements over state-of-the-art methods.
strength_weakness: strength_weakness: 1. The idea of constructing a probabilistic space using VLP knowledge and leveraging uncertainty information to achieve many-to-one alignment between textual information and fine-grained actions is novel.
2. Achieved new state-of-the-art (SOTA) results on multiple public datasets.
limitations: limitations: 1. In addition to the I3D features, this method requires extracting CLIP features for each snippet during runtime, which significantly increases computational complexity, memory usage, and runtime.
2. The necessary of VLP konwledge Distillation: The visual representations of the VLP model already reflect action semantics to some extent. In the mean estimation of snippet-level distributions, why not directly use the mean of CLIP's visual representations, but instead distill CLIP knowledge into the mean estimator defined by the authors? What would be the impact of directly using CLIP visual feature representations as the mean?
3. The necessary of L𝑜𝑟𝑡ℎo: The purpose of leveraging textual information is to utilize additional textual data. When all category label features are orthogonal, it seems to be no different from using one-hot representations, as one-hot labels are a case where all category label representations are orthogonal. Could you conduct a comparative experiment to see what impact directly using one-hot representations instead of  X^C?
paper_summary: paper_summary: This work propose a novel framework that aligns human action knowledge and VLP knowledge in the probabilistic embedding space. Meawhile, it leverage intra- and inter-distribution contrastive learning to enhance the probabilistic embedding space based on statistical similarities. Extensive experiments prove that it significantly outperforms other works.
strength_weakness: strength_weakness: 1. This work aligns VLP knowledge and action knowledge within a probabilistic space, which is technically-sound and novel.

2. This work  introduce an intra- and inter-distribution contrastive strategy to construct distinctive probabilistic embedding space.

3. This work  performs extensive experiments on two public datasets, demonstrating the outstanding performance of the model on various evaluation metrics, and contrasting and analyzing it with existing methods.
limitations: limitations: 1.The Section 3.2 is include many loss function. The description of their interaction is not very obvious, it looks somewhat cumbersome and intricate, in the revision, the authors need to polish this part to make it clearer.

2. There are severa hyper-parameters like $\lambda$ in Eq.6, $\alpha$, $\beta$, $\gamma$ in Eq.20. What are the effects of using different values? Ablations on these hyper-parameters are missing.

3.The paper does not provide an analysis of the complexity and running time of the model, nor does it discuss the efficiency and scalability of the model, which makes the application performance of the model on other datasets unclear.

4. Compared with other results on THUMOS (table1), the performance is not well . What is the possible reason for this?