Revisiting Unsupervised Temporal Action Localization: The Primacy of High-Quality Actionness and Pseudolabels

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 07:34:32

Abstract: Recently, temporal action localization (TAL) methods, especially the weakly-supervised and unsupervised ones, have become a hot research topic. Existing unsupervised methods follow an iterative "clustering and training" strategy with diverse model designs during training stage, while they often overlook maintaining consistency between these stages, which is crucial: more accurate clustering results can reduce the noises of pseudolabels and thus enhance model training, while more robust training can in turn enrich clustering feature representation. We identify two critical challenges in unsupervised scenarios: 1. What features should the model generate for clustering? 2. Which pseudolabeled instances from clustering should be chosen for model training? After extensive explorations, we proposed a novel yet simple framework  called Consistency-Oriented Progressive high actionness Learning to address these issues. For feature generation, our framework adopts a High Actionness snippet Selection (HAS) module to generate more discriminative global video features for clustering from the enhanced actionness features obtained from a designed Inner-Outer Consistency Network (IOCNet). For pseudolabel selection, we introduces a Progressive Learning With Representative Instances (PLRI) strategy to identify the most reliable and informative instances within each cluster for model training. These three modules, HAS, IOCNet, and PLRI, synergistically improve consistency in model training and clustering performance. Extensive experiments on THUMOS’14 and ActivityNet v1.2 datasets under both unsupervised and weakly-supervised settings demonstrate that our framework achieves the state-of-the-art results.

Main paper: 
 introduction: 
 The proliferation of intelligent surveillance devices in recent years has necessitated the development of efficient video processing techniques in the multimedia understanding field [19, 40, 41] . Among them, temporal action localization (TAL) [16, 38] , which aims to accurately localize the temporal boundaries of actions in untrimmed videos and identify their categories from a pre-defined action list, is one of the most important areas.In initial research stages, temporal action localization methods predominantly follow a fully supervised setting [5, 27, 56] , which requires the annotator to perform segment-level annotations for each video in the training dataset, i.e., annotate all action instances with the start and end timestamps and the corresponding action categories. This process requires a lot of manual operations and is prone to the problems of costly annotation and subjectivity of the annotation results. To address those issues, weakly supervised temporal action localization (WTAL) [14, 39, 53] has gradually emerged as a new research focus, which only requires the video-level annotation, i.e., labeling all the pre-defined action categories contained in each video. However, compared to simply collecting unlabeled videos, even such video-level annotations require a significant cost.As a result, the focus has shifted towards training video action localization networks on unlabeled videos, a task known as Unsupervised Temporal Action Localization (UTAL) [10] . This method only requires annotating the total number of action categories for the entire dataset, thereby further reducing annotation costs. As illustrated in Fig 1(a) , all existing UTAL methods follow an iterative "clustering-training" pipeline [10, 48] . During each iteration, the video-level pseudolabels for each video are generated based on the clustering results of the global video features, which are derived from class-agnostic attention obtained from a localization model M. Subsequently, the entire pseudolabeled video set is adopted to train the localization model M.While the above unsupervised methods have garnered notable success, their primary emphasis lies in the "training" stage, i.e., designing various localization models based on uncertainty [48] or co-attention mechanisms [10, 26] , while ignoring the high consistency between the "clustering" and "training" stages. This consistency is crucial for improving clustering accuracy and enhancing model training robustness during iterations, because accurate clustering reduces pseudolabel noise while robust training enriches clustering feature representation. This raises two key challenges: 1) What features should the model generate for clustering? 2) Which pseudolabeled instances from clustering should be chosen for model training? After extensive investigations, we argue that global video features with high actionness across two-branch and high-quality pseudolabels matter for these two challenges. For feature generation, since existing methods aggregate global video features using only the class-agnostic attention of the entire video, the importance of high actionness snippets and the inconsistency of action positioning between class-agnostic and class-specific branches are often ignored. For pseudolabel selection, adopting all pseudolabeled videos for training is not advisable since the clustering results are unreliable in the initial iteration. To address the above issues, we propose a novel yet simple Consistency-Oriented Progressive Learning (COPL) framework for UTAL. Specifically, the High Actionness snippet Selection (HAS) clustering module utilizes a two-branch consistency criterion to generate filtered HAS attention, which eliminates numerous background segments while retaining only snippets with discriminative action-related information for global video feature generation. Subsequently, to generate the high-quality pseudolabeled video sets, we introduce a progressive learning with representative instances strategy based on intra-cluster cohesion and inter-cluster separation, aiming to identify and exclude video instances exhibiting comparatively lower discriminative video global features. During the training phase, utilizing both the refined pseudolabeled video set and the unlabeled video set, we design an Inner-Outer Consistency Network (IOCNet) based on the Teacher-Student architecture [42] . This network enforces consistency constraints jointly from modalities, branches, and data augmentation, thereby enhancing discriminative learning capabilities without relying on explicit labels. The entire framework's workflow is illustrated in Fig 1(b) . As depicted in the figure, the three crucial modules progress sequentially through iterative optimization, reinforcing the discriminative features and effectively improving model performance.The contributions of this paper are as follows:• We propose a conceptually novel yet simple framework termed Consistency-Oriented Progressive Learning, to tackle two crucial challenges, i.e., feature generation and pseudolabel selection, of unsupervised temporal action localization.  
related work 2.1 weakly-supervised temporal action localization: 
 Weakly supervised Temporal Action Localization (WTAL) has raised great research attention recently, owing to its relatively lower annotation costs. Existing methods can be categorized into two groups: representation learning based methods and temporal attention modeling methods. The first group designs various loss functions for better feature representations [8, 10] , such as the multi-label center loss [30] that penalizes the distance between the features and the corresponding action class centers and the snippet-level contrastive loss [12, 53] that refines the representations of the action snippets. The second group aims to enrich the class-agnostic attention weights to separate the action and the background parts [15, 29] . For example, Lee et al. [20] added a suppression branch to the network, effectively modeling background class as an auxiliary component for WTAL. Besides, Li et al. [24] identified the actionness inconsistency in a classical two-branch network and then introduce an action consistency loss to mitigate this issue.As previously discussed, the UTAL method iteratively employs pseudolabels obtained through clustering for training WTAL models. However, current WTAL models face two challenges if adopted in unsupervised settings: handling noisy instances and effectively utilizing both pseudolabeled and unlabeled video datasets. In contrast, our IOCNet tackles these challenges by introducing a Teacher-Student network and employing several inner-outer consistency losses, thereby enhancing the model's ability to mine high actionness snippets and resist noisy instances.Recent advances have shifted into unsupervised temporal action localization task. The prevailing UTAL approaches adopt an iterative pipeline, i.e., first adopting spectral clustering [37] to generate action pseudolabel for each video, and then training a localization model with pseudolabels, which is adopted to generate the pseudolabels for next iteration. The principal difference among these methods lies in the second phase, namely the design of the localization model. Some methods leveraged representation learning techniques to distinguish between action and background in the video [26] or manipulate the distribution within the embedding space [10] . In contrast, other approaches incorporated uncertainty awareness to enhance the learning process for both RGB and optical flow features [48] .In conclusion, these methods uniformly generate the global feature only with the class-agnostic branch and utilize all pseudolabeled instances for training. These operations, however, do not filter out noisy instances during clustering, nor do they enrich the high actionness expression of video features. In contrast, our COPL effectively addresses these issues with three coupled modules.Progressive learning (PL) aims to train models incrementally, guiding them from simpler instances to more complex ones. This approach has found widespread use in representation learning [6, 55] , image classification [7, 57] , and ReID [9, 13, 47] . There are two primary types of PL methods. The first one concentrates on the network level [11, 57] . These methods aid in mitigating the challenges of training deep networks by progressively enhancing network capacity (both width and depth) [21, 54] . This enhancement is crucial for improving the abilities of the network. The second type focuses on the data level [43, 47] . These methods leverage PL to gradually integrate a substantial amount of unlabeled data into the training process within an incomplete-supervised setting, thus addressing the scalability challenge of incomplete labeled datasets [2, 9] .In this paper, we draw inspiration from these methods and incorporate the concept of progressive learning into UTAL. The distinctions are as follows: 1) we design a novel confidence measurement criterion based on the fully connected graph employed in spectral clustering, which is effective in assessing the reliability of the video instances; 2) we train pseudolabeled video set and unlabeled video set with IOCNet jointly. This approach maximally utilizes information from all instances. 
methodology 3.1 preliminaries: 
 We first detail the denotations in this section. Under the UTAL settings, given the unlabeled videos set 𝑉 = {𝑋 1 , 𝑋 2 , . . . , 𝑋 𝑁 } that contains 𝑁 videos from 𝐶 distinct action categories, the goal is to train a localization model 𝜙 (𝜃 ; •) parameterized with 𝜃 to directly classify and localize the corresponding action segments in an untrimmed video. For a video 𝑋 of length 𝑇 in 𝑉 , we first adopt the pre-trained network to extract its RGB features 𝑋 𝑅 = {𝑥 𝑅 𝑖 } 𝑇 𝑖=1 ∈ 𝑅 𝑇 * 𝑑 and optical flow features 𝑋 𝐹 = {𝑥 𝐹 𝑖 } 𝑇 𝑖=1 ∈ 𝑅 𝑇 * 𝑑 , which are also concatenated together to obtain 𝑋 = {𝑥 𝑖 } 𝑇 𝑖=1 ∈ 𝑅 𝑇 * 2𝑑 . As illustrated in Fig 2 , taking the above features of each video in 𝑉 as input, our COPL contains the following stages during each iteration: 1) The HAS module generates the high-actionness attention 𝐴 𝐻𝐴𝑆 that exhibit consistency across both the class-specific branch 𝐴 𝑐𝑠 and class-agnostic branch 𝐴 𝑐𝑎 of IOCNet to obtain the discriminative global feature 𝐹 for each video; 2) The PLRI module performs clustering on all video features, and generates the pseudolabeled set 𝑃 and unlabeled video set 𝑈 , based on the intra-cluster cohesion and inter-cluster separation; 3) The IOCNet is trained with the 𝑃 and 𝑈 sets, which facilitates the generation of 𝐴 𝐻𝐴𝑆 in next iteration. We will introduce these three stages in detail.During the clustering stage, existing UTAL methods typically employ the snippet-wise attention of an entire untrimmed video to generate its global video feature. This attention is often derived from a single branch, such as the class-agnostic branch or the classspecific branch aggregated along the action category dimension. However, there are two serious problems with obtaining high actionness features for clustering: 1) Using complete attention is not advisable. Untrimmed videos typically contain substantial background snippets. Including these snippets in the attention may decrease the action-specific expression of the global feature. 2) Results obtained from a single branch may lack reliability. As shown in Fig 3,  we visualize the activation zones of high and low activation in two branches. It can be observed that class-specific branch is prone to interference from contextual information, leading to erroneously high activation zones when significant background related to the action exists. Conversely, the class-agnostic branch is susceptible to interference from actions unrelated to the target, resulting in erroneously high activation zones. Both scenarios can lead to inconsistent and unexpected localization results [24, 28] .To this end, we propose the HAS module based on the principle of two-branch consistency, which enriches the reliable actionness of global feature extracted from a video. This module selects highactionness snippets that exhibit consistency across both branches to overcome the limitations of relying on the complete attention of a single branch. With the filtered discriminative snippets, the quality of global video feature and the accuracy of later clustering can be enhanced.Specifically, as shown in . Thereafter, an action filter is performed to obtain high-actionness attention across two branch. Firstly, a binarization operation on 𝐴 𝑐𝑠 and 𝐴 𝑐𝑎 is conducted to generate high activation zones and low activation zones [52] on two branches as:EQUATIONwhere 𝑚𝑒𝑑𝑖𝑎𝑛(𝐴) is the median value of actionness sequence 𝐴.To identify and filter out the snippets with low actionness across two branches, a consistency comparison between 𝐴 𝑏𝑖𝑛 𝑐𝑠 and 𝐴 𝑏𝑖𝑛 𝑐𝑎 is conducted as:EQUATIONThereafter, we obtain the actionness-aware attention𝐴 𝑓 𝑐 = {𝑎 𝑓 𝑐,𝑖 } 𝑇 𝑖=1 . Subsequently, the top-𝑘 snippets of 𝐴 𝑓 𝑐 = {𝑎 𝑓 𝑐,𝑖 } 𝑇𝑖=1 is selected to generate the high-actionness attention as:EQUATIONwhere 𝐴represents the index of 𝐴 𝑓 𝑐 when sorted in descending order. 𝑘 = 𝑚𝑎𝑥 (1, ⌊ 𝑇 𝛾 ⌋). 𝛾 controls the proportion of selected snippets. The obtained attention 𝐴 𝐻𝐴𝑆 = {𝑎 𝐻𝐴𝑆 𝑖 } 𝑇 𝑖=1 combines key snippets of the video from both the class-specific and class-agnostic branches, emphasizing action-discriminative snippets. However, this action filter may be overly aggressive, leading to the removal of crucial action snippets. To address this issue, we propose an effective voting function to smooth the HAS attention across epochs as follows:EQUATIONwhere 𝛼 ∈ (0, 1) denotes the decay rate. 𝑎 𝐻𝐴𝑆 𝑖,𝑘 is the HAS attention score for snippet 𝑖 obtained at the 𝑘-th training epoch. This function select snippets that consistently exhibit high actionness EQUATIONwhere 𝐿 2 𝑁𝑜𝑟𝑚 denotes 𝐿 2 normalization.Given the global features 𝐹 of all videos in 𝑉 , we construct a fullyconnected graph G = {V, E} and perform the spectral clustering [37] to assign 𝑉 into 𝐶 clusters as in [10] . The edge weight between video 𝑋 𝑖 and video 𝑋 𝑗 is computed as 𝑤 𝑖 𝑗 = exp(-𝑑𝑖𝑠𝑡 (𝐹 𝑖 ,𝐹 𝑗 ) 2 2𝜎 2),where 𝜎 = 1 Considering the inherent differences between features of different actions, we craft a metric based on intra-cluster cohesion and inter-cluster separation within the fully connected graph 𝐺 [34] to filter out less discriminative instances, thereby retaining reliable ones in 𝑃 𝑜 . More Formally, the intra-cluster cohesion is measured as the average distance of video 𝑋 𝑖 to other videos within their cluster 𝑌 𝐼 , which is computed as:EQUATIONwhere |𝑌 𝐼 | represents the cluster size. Similarly, the inter-cluster separation is measured as the average distance between 𝑋 𝑖 and videos affiliated with its nearest neighboring cluster 𝑌 𝐽 :EQUATIONThe final confidence score 𝑠 𝑖 is formulated by incorporating the metrics 𝑎 𝑖 and 𝑏 𝑖 :𝑠 𝑖 = 𝑎 𝑖 -𝑏 𝑖 𝑚𝑎𝑥 (𝑎 𝑖 , 𝑏 𝑖 ) (8)where a higher 𝑠 𝑖 indicates a stronger intra-cluster cohesion coupled with a larger inter-cluster separation. Essentially, the reliable video with higher confidence of pseudolabel should exhibit a global representation that is more tightly clustered within their respective action classes, while also demonstrating distinct separation from representations of other actions. Therefore, this metric not only signifies higher confidence in the assigned labels but also reflects the strong discriminative nature of the action-related features.With this metric, videos in 𝑃 𝑜 with lower confidence will be removed into an unlabeled set 𝑈 as follows:EQUATIONFurthermore, the refined pseudolabeled set 𝑃 is constructed as:EQUATION)where 𝑦 𝑖 is the action pseudolabel of 𝑋 𝑖 obtained from clustering.With the refined pseudolabeled video set 𝑃, the second stage, i.e., training the localization model with 𝑃, can be conducted. To further facilitate the awareness of high-actionness snippets while taking advantage of unlabeled set 𝑈 , we introduce an IOCNet based on the Teacher-Student architecture [42] . Specifically, since the teacher and student networks have an identical structure, we detail the structure of teacher as an example. For the class-specific branch, 𝑋 , the concatenated feature of 𝑋 𝑅 and 𝑋 𝐹 , is fed into a convolution layer and a followed attention layer to derive class-specific attention as 𝐶𝐴𝑆 = 𝐴𝑡𝑡 (𝐶𝑜𝑛𝑣 (𝑋 )), where 𝐴𝑡𝑡 comprises several convolution layers and the sigmoid function [22, 53] . Thereafter, 𝐶𝐴𝑆 is aggregated along the action categories dimension as follows:EQUATIONFor class-agnostic branch, we separately input 𝑋 𝑅 and 𝑋 𝐹 into a temporal convolution layer to obtain class-agnostic attention 𝐴 𝑅 𝑐𝑎 and 𝐴 𝐹 𝑐𝑎 for each respective modality as:EQUATIONAfterwards, the results from both modalities, 𝐴 𝑅 𝑐𝑎 and 𝐴 𝐹 𝑐𝑎 , are combined together to generate the class-agnostic attention 𝐴 𝑐𝑎 as:EQUATIONAfter the training of IOCNet, 𝐴 𝑐𝑎 and 𝐴 𝑐𝑠 of each video are fed into HAS module for the generation of the global feature in the next iteration.Training. The optimization of IOCNet incorporates the following loss: a) Weakly-Supervised loss. With the pseudolabeled set 𝑃, the common action classification loss can be adopted. Specifically, we first calculate the action selection function [28] as:EQUATIONwhere 𝛽 balances those two branches. We select top-𝑙 action instances with the highest ℎ 𝑐𝑠 values and generate their video-level predictions 𝑝 ∈ R 𝐶 based on the action categories with the maximum values. The action classification loss is computed as:EQUATIONTo optimize the class-agnostic branch, we partition the selected instances into a positive set 𝑆 𝑝 , encompassing instances that match the groundtruth classes and a negative set 𝑆 𝑛 , which includes all other instances. The action selection loss is denoted as:𝐿 𝑎𝑠𝑙 = 1 |𝑆 𝑝 | ∑︁ 𝑡 ∈𝑆 𝑝 1 -(𝐴 𝑚 𝑐𝑎;𝑡 ) 𝑞 𝑞 + 1 |𝑆 𝑛 | ∑︁ 𝑡 ∈𝑆 𝑛 1 -(1 -𝐴 𝑚 𝑐𝑎;𝑡 ) 𝑞 𝑞 (16)where 𝑚 ∈ {𝑅, 𝐹 }. The weakly-supervised loss is obtained as:EQUATIONb) Unsupervised loss. To fully leverage the information in 𝑈 , we introduce unsupervised consistency constraints from both innerand outer-sources to enhance the discrimination of our network. Specifically, the inner consistency loss include the following two aspects:Modality. The two modalities, RGB and optical flow, are both distinctive and complementary [17] . Within the class-agnostic branch, a late-fusion strategy is employed to adjust the prediction outcomes of both modalities. The consistency constraints can achieve the information supplementation between the two modalities as follows:EQUATIONBranch. As discussed in Section 3.2, the actionness generated by the two branches faces distinct zones of erroneous activation. During the training phase, we impose constraints on both branches, compelling them to produce similar actionness sequences as:EQUATION)where 𝑀𝑆𝐸 is the mean square error. The inner consistency constraints are as follows:EQUATION)Afterwards, we build the outer consistency loss that aims to maintain the output consistency between the teacher and student network. Leveraging temporal feature shift [44] , we randomly shift half of 𝑐 selected channels forward and the other half backward in the student network to create Data-Augmented (DA) features X = [ X 𝑅 , X 𝐹 ], contrasting with the teacher network's use of original features. Thereafter, we impose constraints on the outputs of the class-specific and class-agnostic branches of the teacher and student, respectively, which further enhance the temporal action modeling capability of the network as:EQUATIONwhere ℎ 𝑐𝑠 and 𝐴 𝑐𝑎 are from the student network, h 𝑐𝑠 and Ã 𝑐𝑎 are from the teacher network. Besides, we also employ contrastive loss 𝐿 𝐶 to enhance snippets features with significant actionness differences between branches [24] . Our unsupervised loss is:EQUATION)The student network is trained jointly with the sets 𝑃 and 𝑈 . For the set 𝑃, all losses are adopted. As for the unlabeled set 𝑈 , only the unsupervised loss is applied. Note that the weights 𝜙 of the teacher model are updated via Exponential Moving Average (EMA) [42] from the corresponding weights of the student 𝜙 ′ . The entire algorithm is provided in Appendix.Inference. Following [10, 26] , we map the cluster to a specific action category for evaluations. The teacher model is adopted for inference. Firstly, we calculate ℎ 𝑐𝑠 to obtain action instance positions. Subsequently, we adopt 𝐶𝐴𝑆 to gauge the likelihood of each position corresponding to actions in various categories. After that, we utilize several thresholds to filter out snippets with probabilities greater than 𝜃 to form candidate proposals. Finally, we apply NMS to eliminate redundant proposals. More details are in Appendix. [20, 29] . 
evaluation metrics.: 
 The localization performance is evaluated under varying Intersection over Union (IoU) thresholds using the mean Average Precision (mAP) metric, following the standard protocols [1, 16] . As for the evaluation of the clustering results, three widely-used metrics: purity, Normalized Mutual Information score (NMI), and Adjusted Rand Index (ARI) are adopted, on which the better performance is indicated by higher values.Following the common settings [28, 53] , each video stream is first divided into 16-frame non-overlapping snippets, and then the TV-L1 [50] algorithm and the pre-trained I3D network [4] are employed to extract the 1024-dimension RGB and optical flow features. During both training and testing, 750 and 50 snippets for each video in THUMOS'14 and ActivityNet v1.2 are sampled, respectively. For HAS module, we set 𝛾 = 20, 𝛼 = 0.9 for THUMOS'14 and 𝛾 = 2, 𝛼 = 0.5 for ActivityNet v1.2. Additionally, we set 𝛿 = 0 in Section 3.3 to create refined pseudolabeled and unlabeled video sets. In training, we use 𝑞 = 0.7 for Eq.( 16), and instance selection parameters 𝑙 = 𝑇 /8, 𝛽 = 1/3 for THUMOS'14, and 𝑙 = 𝑇 /2, 𝛽 = 1/2 for ActivityNet v1.2. We maintain 𝜇 1 = 5, 𝜇 2 = 0.1, 𝜆 1 = 0.01 across both datasets, while adjusting 𝜆 2 = 10 for THUMOS'14 and 𝜆 2 = 1 for ActivityNet v1.2. Data augmentation employs 𝑐 = 512, with batch sizes of 16 for THUMOS'14 and 128 for ActivityNet v1.2. In addition, we adopt Adam [18] with a learning rate of 1e-4 and a weight decay of 5e-4 to optimize our model over 300 epochs for THUMOS'14 and 30 epochs for ActivityNet v1.2. The total iteration number is set to 3. During testing, the thresholds 𝜃 ∈ [0, 1.0 : 0.1] are applied to generate proposals. All experiments are run on a Nvidia Tesla A100 GPU.We compare our method with the several state-of-the-art (SOTA) UTAL and WTAL methods across various IoU thresholds in this section. For the weakly supervised setting, COPL solely utilized the IOCNet, trained once on the entire dataset with the video-level groundtruth labels.Results on THUMOS'14 Dataset. Table 1 illustrates the mAP results of different methods on THUMOS'14 dataset with IoU thresholds ranging from 0.1 to 0.7, with an interval of 0.1. The "Avg" column represents the average mAP results. Our method achieves significant performance in the UTAL scenario, achieving 41.7% on Avg, which marks a notable 1.5% and 6.5% improvement compared with UGCT and APSL. In addition, for mAP@0.5, our COPL method outperforms all other SOTA unsupervised methods by 1.1%, attaining 33.9%.The localization performance on the ActivityNet v1.2 is presented in Table 2 . The "Avg" column represents the average mAP results over the IoU intervals from 0.5 to 0.95, with an interval of 0.05. As depicted in Table 2 , even under the UTAL setting, our COPL achieves the best results over all baselines, showing competitive performance compared to several recent weakly supervised methods.In this section, we analyze the roles of different modules in COPL under UTAL conditions. If not specified, the experiments were conducted on THUMOS'14. More results are provided in Appendix. Furthermore, the integration of both HAS and PLRI can significantly enhance the performance within the iterations.4.3.2 Analysis of HAS module. This module involves two parts: the action filter of two-branch consistency and the voting function. For the former, we compare the clustering results under the full HAS attention with those under various alternative attention scores, as shown in table 3. For the results of line #2 and #3, the attention filter is conducted on single branch to select high-actionness snippets [26] . Obviously, employing a single branch leads to inferior clustering results. Besides, the clustering advantage of the HAS module 48.4 28.9 6.5 29.9over the unfiltered attention on the class-agnostic branch (line #1) also verifies the importance of high actionness for clustering. Moreover, we also analyze the impact of the 𝛾 in Eq.3, i.e., the selected proportion of high-actionness snippets, on the localization results, as shown in Table 5 . A larger 𝛾 represents less selected snippets. Too small 𝛾 will cause HAS to introduce too many low-action fragments, which are often associated with background information, consequently leading to a decrease in overall performance. Conversely, when 𝛾 is large, the total number of selected snippets is too small, which will also hurt the video representation. In our experiment, the best results are obtained when 𝛾 is set to 20.For the voting function, as shown in Table 4 , we calculate the mean and variance of the respective NMI for COPL and COPL (w/o Vote) variants over their last 10 and 50 epochs. It can be seen that the voting function significantly improves the clustering performance and stability of the model by integrating action information obtained from different epochs.In Table 6 , we analye the impact of threshold 𝛿 in our PLRI strategy, which balances the quantity and quality of the pseudolabeled videos in 𝑃: the greater the 𝛿, the more strictly the pseudolabeled set 𝑃 is built. It is evident that when 𝛿 approaches 0, COPL consistently outperforms COPL (w/o PLRI) variant. However, a decrease in localization results is observed when 𝛿 = 0.10. This observation highlights that while high-quality labels are beneficial, an insufficient number of instances hinders model learning.We incrementally incorporate various components based on supervised loss to assess their impact on IOC-Net's performance, as shown in Table 7 . The incorporation of outer consistency constraints 𝐿 𝐶𝑂 enables us to leverage more temporal information, Omitting 𝐿 𝐶𝑂 simplifies the teacher-student network to a single-model structure with all weakly-supervised loss solely on unaugmented data. The setup of using only supervised loss (line #1) yields inferior results, because the low accuracy in class-specific branch activationa will lead to suboptimal clustering performance. Moreover, compared to the variant in line #1, introducing the inner consistency loss 𝐿 𝐶𝐼 enhances the HAS module and joint training on unlabeled video sets, resulting in significant improvements. Further enhancement in discriminative power is achieved through the contrastive learning loss 𝐿 𝐶 . Overall, we can see that all loss functions in IOCNet contributes to the final localization results.We present two qualitative results in Fig 5 , where the activation scores of our COPL and the Baseline variant are also provided. It is evident that the Baseline variant generates inaccurate predictions in both the action and background segments as in the dashed rectangle. Conversely, our COPL framework demonstrates enhanced feature discrimination, yielding more precise predictions.  
conclusion: 
 In this paper, we propose a novel consistency-oriented progressive learning framework for UTAL task. Specifically, a high actionness snippet selection module is designed to select most discriminative snippets for better generation of the global video features. To tackle the issue of noisy pseudolabels during training, we propose a PLRI strategy to choose the most reliable pseudolabeled instances based on intra-cluster cohesion and inter-cluster separation. Finally, we propose the teacher-student structured IOCNet that leverages various consistency constraints to improve the temporal action modeling ablities. We also conduct extensive experiments on two public datasets to demonstrate that our COPL framework achieves SOTA UTAL performance and competing weakly-supervised performance. Ablation studies verify the effectiveness of all proposed modules.Limitations and Future Work. While our paper conducts analysis experiments with empirically determined fixed thresholds across various values, our future research will focus on exploring self-adaptive thresholds [45] . Additionally, we are interested in investigating the integration of a deep clustering algorithm [3] , moving away from the conventional approach of clustering with original features.  
 
 Appendix:  


Decision: The submission received mixed scores (borderline reject, borderline accept, accept). All reviewers have acknowledge the work addressed two critical challenges in the task of unsupervised action localization. Two reviewers regarded the technical novelty to be clear and sound. Some of the raised concerns (e.g., missing ablation analysis) have been addressed in the rebuttal. The ACs would recommend to accept it and suggest the authors to revise the work according to the suggestions.

Reviews: review: paper_summary: paper_summary: In this paper, the authors identify two critical challenges in unsupervised temporal action localization task, and propose a novel yet simple framework named consistency-orientated progressive high actionness learning to generate discriminative global video features for clustering and identify the most reliable and informative instances.
strength_weakness: strength_weakness: (1) The identified two challenges are very critical and deserve research further.
(2) The paper is well-written and good organized.
limitations: limitations: 1.Why is the student network not utilized during the inference phase? Does its usage impact performance? 
2.What are the differences between the inputs of the teacher network and the student network? 
3.Do the authors consider using other clustering methods?
4.The inference efficiency of the proposed method is not compared with the state-of-the-art unsupervised methods.
5.Can you provide more insight into the impact of the HAS module, such as providing visual explanations?
6. In the proposed unsupervised action localization method, there are notable complexities. The reliance on a pre-trained I3D network to distinguish pseudo-labeled or unlabeled instances, along with the introduction of a custom IOCNet, raises concerns about computational overhead and resource utilization. Without empirical data on computational requirements, it is unclear how practical and efficient this approach will be in real-world applications.
7. In terms of the design of IOCNet, the lack of interpretability is noticeable in the placement of the attention structure and certain loss functions, as well as the utilization of some parameters. The complexity in setting hyperparameters also raises questions about the inspiration behind these choices. Were there specific network structures or references that guided the design of IOCNet?
8. The introduction of the dataset is scanned. We only know that these are two action video datasets, using 16 frames. We are not sure if there are any specific downsampling data operations, including the data augmentation mentioned in the article.
9. Despite the adoption of new methods and improvements in effectiveness, the experimental results did not show significant improvement compared to unsupervised and weakly supervised methods in the past two years.
paper_summary: paper_summary: This paper tackles unsupervised and weakly supervised temporal action localization. It introduces a module for selecting high-actionness snippets to capture discriminative action-related information effectively. Furthermore, a teacher-student architecture is proposed to enhance discriminative learning. Experimental validation on Thumos and ACNet datasets demonstrates the efficacy of the proposed method.
strength_weakness: strength_weakness: 1. The rationale behind selecting discriminative action-related information to improve clustering is sound.
2. The introduction of a teacher-student architecture for enhancing discriminative learning is novel.
3. The effectiveness of the high-actionness snippet selection module is clearly demonstrated through experiments.
limitations: limitations: **Major concerns:**

1. There is a lack of explanation regarding why the proposed method outperforms previous approaches in weakly supervised scenarios. Further analysis is needed to elucidate this superiority.

2. Ablation studies focusing on weakly supervised settings are absent, particularly in explaining why the proposed method exhibits significant improvements in ActivityNet. A detailed analysis of different modules through ablation studies is necessary.

3. Ablation analysis of the teacher-student model is missing. What is the performance in both unsupervised/weakly-supervised settings when either dropping the student network and solely using the teacher model, or dropping the teacher network and solely using the student model? 

**Other concerns:**  
1. The terms "pseudolabels" and "pseudolabeled" should be replaced with the standard usage "pseudo-labels" and "pseudo-labeled" to align with the existing literature.
paper_summary: paper_summary: This paper addresses two critical questions in the literature of unsupervised temporal action localization (UTAL):

- What features should the model generate for clustering?
- How to select reliable pseudo-labeled instances?

To answer the first question, this paper introduces a high actionness snippet selection (HAS) module that considers the consistency of class-specific and class-agnostic branches. To address the second question, this paper proposes an IOCNet and a progressive learning paradigm (PLRI), which selects reliable labeled instances for training. Generally, the proposed method can be extended to both weakly supervised temporal action localization (WTAL) and UTAL tasks. Extensive experiments have verified the superiority of the proposed approach, COPL.
strength_weakness: strength_weakness: - This paper exhibits good novelty by addressing two critical issues in the pipeline of UTAL methods.
- The use of case studies to clarify its motivation (e.g., Figure 3) makes the paper easy to understand. Moreover, this paper is well-written and easy to follow.
- To generate high-quality global features and select reliable instances, this paper introduces specific algorithms, which are technically sound.
- The proposed COPL (UTAL) has achieved comparative results compared to advanced WTAL methods. The proposed unsupervised method shows promise for future applications with larger-scale video data.
limitations: limitations: Although this paper has good novelty and is well-written, there are still some drawbacks:

- Mistake in Eq. (6): Since the summation is over both $i$ and $j$, the result $a_i$ should not contain subscript $i$. Otherwise, the summation should not be over $I$. In addition, the $C_I$ in Eq. (6) should be $Y_I$.

- In Eq. (16), this paper incorporates the action selection loss for training, which was proposed by [1] in the WTAL task. It would be better to add this reference.

- The hyper-parameters $\mu_1, \mu_2, \lambda_1, \lambda_2$ in Eqs. (20), (22) should be explained, similar to $\beta$ in Eq. (14). 

- Some grammar mistakes: “high-actionness attention that exhibit consistency” in line 266; “a Nvidia” in line 632

[1] Ma, Junwei, Satya Krishna Gorti, Maksims Volkovs, and Guangwei Yu. "Weakly supervised action selection learning in video." In the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7587-7596. 2021.