Class‐wise boundary regression by uncertainty in temporal action detection

IET Image ProcessingVolume 16, Issue 14 p. 3854-3862 ORIGINAL RESEARCHOpen Access Class-wise boundary regression by uncertainty in temporal action detection Yunze Chen, Yunze Chen orcid.org/0000-0002-6043-5028 Center of Precision Sensing and Control, Institute of Automation, Chinese Academy of Sciences, Beijing, China School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, ChinaSearch for more papers by this authorMengjuan Chen, Mengjuan Chen Center of Precision Sensing and Control, Institute of Automation, Chinese Academy of Sciences, Beijing, ChinaSearch for more papers by this authorQingyi Gu, Corresponding Author Qingyi Gu [email protected] orcid.org/0000-0001-8332-5350 Center of Precision Sensing and Control, Institute of Automation, Chinese Academy of Sciences, Beijing, China Correspondence Qingyi Gu, Center of Precision Sensing and Control, Institute of Automation, Chinese Academy of Sciences, Beijing, China. Email: [email protected]Search for more papers by this author Yunze Chen, Yunze Chen orcid.org/0000-0002-6043-5028 Center of Precision Sensing and Control, Institute of Automation, Chinese Academy of Sciences, Beijing, China School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, ChinaSearch for more papers by this authorMengjuan Chen, Mengjuan Chen Center of Precision Sensing and Control, Institute of Automation, Chinese Academy of Sciences, Beijing, ChinaSearch for more papers by this authorQingyi Gu, Corresponding Author Qingyi Gu [email protected] orcid.org/0000-0001-8332-5350 Center of Precision Sensing and Control, Institute of Automation, Chinese Academy of Sciences, Beijing, China Correspondence Qingyi Gu, Center of Precision Sensing and Control, Institute of Automation, Chinese Academy of Sciences, Beijing, China. Email: [email protected]Search for more papers by this author First published: 04 August 2022 https://doi.org/10.1049/ipr2.12599AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Temporal action detection is a crucial aspect of video understanding. It aims to classify the action as well as locate the start and end boundaries of the action in the untrimmed videos. As deep learning is frequently utilized, the accuracy of annotation is crucial to boundary localization. However, it is observed that some annotation instances are ambiguous and the ambiguity varies between categories. To solve the problem above, a Gaussian model is built to estimate the boundary uncertainty for each instance. Based on instance uncertainty, category uncertainty is applied to describe the uncertainty of each category. By combining instance and category uncertainty, the boundaries of the selected proposals are refined and the ranking of candidate proposals is adjusted. Furthermore, overcorrection is avoided for categories with a high level of uncertainty. With the uncertainty approach, state-of-the-art performance is achieved: 57.5% on THUMOS14 ([email protected]) and 35.4% on ActivityNet ([email protected]). 1 INTRODUCTION With the progress of technology, an ever-increasing number of videos are saved and made available for a variety of daily activities. As a fundamental aspect of video understanding, temporal action detection can be applied to a variety of fields, including video content analysis and video recommendation. This task is not only to determine which actions are depicted in the video, but also to find their temporal boundaries, that is, their start and end times. Because most temporal action detection algorithms rely significantly on deep learning, the quality of the predicted proposals is dependent on the precision of the boundary annotations. To limit the impact of unclear annotations, the ambiguity of annotation boundaries are represented by instance uncertainty in [1]. Based on instance uncertainty, we observe that the difficulty of locating the boundaries of different categories varies. The labeling of some action types is ambiguous. These ambiguities include ambiguity in action definitions, ambiguity in action boundaries, and ambiguity in video content, as shown in Figure 1. This labeling uncertainty leads to prediction bias, which is difficult to correct. FIGURE 1Open in figure viewerPowerPoint (a) Ambiguity of action definitions. The upper part of (a) is the complete Hammer Throw. The bottom half of (a) is a partial replay of the complete action. However, the partial replay is still labelled as a complete Hammer Throw. (b) Ambiguity in the action's boundaries. There is only a fraction of a second between the begin and end of Billiards. As a result, determining the end boundary of Billiards is difficult. For labeling convenience, the white ball touched by the club, the white ball touched by the target ball (yellow ball in (b)), and the target ball into the hole may all be labeled as the end boundary for labeling purposes. (c) Ambiguity in video content. In this instance, the start and end boundaries of volleyball spiking are accurately labeled. However, there are other actions of other people in this video, making it difficult for the model to learn volleyball spiking To describe the uncertainty of various categories, we create a metric called category uncertainty. With these criteria, we aim to avoid over-refinement of proposals that fall into high-uncertainty categories. Specifically, we model the uncertainty of each instance in the training process. To describe the uncertainty of various categories, we create a metric called category uncertainty. Throughout the testing phase, we apply the reliability score module to select reliable proposals based on the instance uncertainty score. By combining the instance uncertainty and the category uncertainty, we refine the scores as well as the boundary of the predicted proposal simultaneously in uncertainty voting. Compared to [1], we introduce category uncertainty to describe the uncertainty of each category. Based on this metric, we propose the uncertainty voting module. In this module, we refine the boundaries and adjust the ranking of candidate proposals. In addition to the one-stage and two-stage methods, we deploy our methodology in the anchor-free method AFSD [2], and achieve state-of-the-art performance. Our contribution can be summarized as: 1. We introduce category uncertainty to describe the annotation accuracy of each category. 2. According to category uncertainty, the uncertainty voting module is designed to refine proposals belonging to categories with low uncertainty. 3. We deploy our uncertainty method on a one-stage, two-stage and anchor-free framework to demonstrate the effectiveness of our method. On the anchor-free method AFSD [2], we achieve state-of-the-art performance. The performance is 57.5% on THUOMS14 ([email protected]) and 35.4% on ActivityNet ([email protected]). The remainder of this work is arranged in the following manner. Section 2 provides background on temporal action localization and uncertainty learning in deep neural networks. Section 3 describes the proposed temporal action detection framework. In Section 4, the proposed method is validated by experimental results and compared to earlier research. And Section 5 concludes the paper and provides an overview of future directions. 2 RELATED WORKS Temporal action localization Recently, great progress has been made in deep learning, which facilitates the development of temporal action localization. The methodology for this task can be divided into two categories: single-stage and two-stage. The one-stage approach combines proposal generation and classification into a single end-to-end framework for high efficiency. Inspired by SSD [3], [4] designs a one-dimensional temporal convolution to generate multiple temporal action anchors for proposal generation. Furthermore, SSTAD [5] employs a recurrent neural network (RNN) architecture to perform proposal generation and classification simultaneously. To provide valid proposal boundaries and classification results, Decouple-SSAD [6] employs two branches: one for regression and the other for classification. GTAN [7] introduces a Gaussian kernel to dynamically optimize the time scale of each action proposal. On the other hand, the two-stage strategy first generates action proposals and then classifies them to achieve high performance. By incorporating 3D regional convolutional networks (C3D [8]), R-C3D [9] optimizes localization and classification loss in an end-to-end framework. Inspired by the end-to-end framework Faster R-CNN [10], TAL-Net [11] adds context information into proposals. And to make better use of contextual information, PGCN [12] considers the proposal-proposal interactions as well as leverages proposal relationships. Furthermore, AFSD [2] presents an efficient and effective anchor-free temporal localization strategy that yields state-of-the-art results. However, each of these methods suffers from boundary ambiguity, resulting in unreliable proposals or inaccurate boundaries. To reduce the impacts of unreliability boundary, we introduce uncertainty metrics to describe the accuracy of annotations. Uncertainty learning in deep neural networks (DNNs) To improve the robustness and interpretability of discriminative deep neural networks (DNNs), researchers are increasingly turning to the uncertainty approach. There are two types of uncertainty approaches: model uncertainty and data uncertainty. The model uncertainty captures the noise in the deep neural network's parameters, which can be minimized by increasing the training data amount. Data uncertainty, on the other hand, captures the noise in the given training data. As a result, data uncertainty does not alter as the amount of training data grows. This study is based on the uncertainty of the data. Previous methods primarily focused on the image task's uncertainty. For example, ambiguity in bounding box labeling in object detection [13], and ambiguity caused by blurry pictures in face recognition [14]. In the action localization task, the ambiguity of boundary labeling, as well as the ambiguity of action definition, are both present. Furthermore, the level of uncertainty differs for different categories of actions. Therefore, we build a Gaussian model to estimate the boundary uncertainty for each instance. Based on instance uncertainty, we apply category uncertainty to describe the uncertainty of each category. By combining instance and category uncertainty, we refine the boundaries of the selected proposals and adjust the ranking of candidate proposals. 3 METHOD Baseline architecture 3.1.1 Base feature extraction Temporal action detection aims to discover precise temporal boundaries and classes of action instances in the untrimmed video. Due to the limitation of computational resources, it is impossible to feed the untrimmed video straight into the visual coder for feature extraction (untrimmed videos are often very long, up to several minutes). A typical strategy is to divide the video into different segments at equally sized intervals. Formally, the input video with frame l can be separated into l s $l_s$ segments by dividing into time intervals σ. S = s n n = 1 l s , l s = l σ . \begin{equation} S={\left\lbrace s_{n}\right\rbrace} _{n=1}^{l_{s}}, \quad l_{s}=\frac{l}{\sigma }. \end{equation} (1)Then, each segment is input into a pre-trained visual coding system, such as two-stream [15] or I3D [16] to extract spatial and temporal feature vectors, respectively. Finally, these two features are linked together for further processing (Figure 2). FIGURE 2Open in figure viewerPowerPoint Given an untrimmed video, we encode the RGB frames and optical flow into the video feature sequence. With the feature sequence, we acquire a series of candidate proposals using uncertainty regression and classification branch, respectively. The candidate proposals include classification score and uncertainty score of the boundary. In post-processing, the reliability score combines the uncertainty score of the boundary with the classification score to choose reliable proposals. Furthermore, uncertainty voting sorts the candidate proposals and revises the boundaries of the selected proposals by incorporating category uncertainty 3.1.2 Proposal generation Equipped with the extracted features, we forecast the location of the instance and the action score using the proposal and classification branches, respectively. For the proposal branch, it's usual to build a set of anchors in different scales to determine the default proposal location ( p c , p w ) $(p_c,p_w)$ , where p c $p_c$ and p w $p_w$ are the default center and width, respectively. To locate the temporal boundaries of action instances accurately, each proposal outputs two predictions by 1D convolution: 1) Regression parameters of the proposal ( Δ p c , Δ p w ) $(\Delta p_c ,\Delta p_w)$ , indicating the offset of the default temporal center and width. 2) The overlap score p o v $p_{ov}$ , which indicates the intersection-over-union (IoU) score between the proposal and its closest ground-truth segment. Finally, the regressed boundaries are: x s = p c + β 1 p w Δ p c − 1 2 p w e β 2 Δ p w x e = p c + β 1 p w Δ p c + 1 2 p w e β 2 Δ p w , \begin{equation} \def\eqcellsep{&}\begin{array}{l} x_s=p_c+\beta _1 p_w\Delta p_c - \dfrac{1}{2}p_w e^{\beta _2\Delta p_w} \\[11pt] x_e=p_c+\beta _1 p_w\Delta p_c +\dfrac{1}{2}p_w e^{\beta _2 \Delta p_w} \end{array} , \end{equation} (2)where x s $x_s$ and x e $x_e$ denote the start and end position of the action, respectively. β1 and β2 are hyper-parameters. The classification branch, on the other hand, is responsible for determining which category the action instance belongs to. For each instance in proposal branch, the classification branch outputs the classification score p = [ p 0 , p 1 , ⋯ , p C ] $\mathbf {p}=[p_{0}, p_{1}, \dots , p_{C}]$ , indicating the probability that the action instance belongs to C categories and one background. 3.1.3 Post-processing After predicting the location and action score, each proposal can be represented as: ψ = B , S B = x s , x e S = p o v × argmax p r r = 1 C . \begin{equation} \def\eqcellsep{&}\begin{array}{l}\psi ={\left\lbrace \mathcal {B}, S\right\rbrace} \\[3pt] \mathcal {B}= {\left\lbrace x_s, x_e\right\rbrace} \\[3pt] S = p_{ov} \times \text{argmax}{\left({\left\lbrace p_{r}\right\rbrace} _{r=1}^{C}\right)} \end{array} . \end{equation} (3) B $\mathcal {B}$ is the detection boundary of the proposal, where x s $x_s$ and x e $x_e$ represent the predicted start and end boundary of action, respectively. S represents the final score of the proposal, which is obtained by multiplying the overlap score p o v $p_{ov}$ and the maximum classification score p r $p_{r}$ . For these proposals, we perform NMS [17] to remove any redundant predictions (in Algorithm 1). Standard multi-class classification and boundary regression As mentioned above, the baseline architecture generates three attributes for each proposal: (1) Start and end boundaries of action instances; (2) IoU between the proposal and the ground truth that is closest to it. (3) Scores of the proposal belonging to C categories. Therefore, we supervise each of these three attributes through location regression loss, overlap loss, and classification loss. Specifically, we use the Smooth L1 loss ( S L 1 $S_{L1}$ ) [18] in location regression to shift the proposal ( x s , x e ) $(x_s,x_e)$ closer the ground truth ( g s , g e ) $(g_s,g_e)$ that is closest to it, which is determined as: L loc = S L 1 x s − g s + S L 1 x e − g e . \begin{equation} L_{\text{loc}}=S_{L 1}{\left(x_s-g_s\right)}+S_{L 1}{\left(x_e-g_e\right)}. \end{equation} (4)And the overlap loss is calculated by Mean Square Error (MSE) loss: L o v = p o v − g i o u 2 , \begin{equation} L_{o v}={\left(p_{o v}-g_{i o u}\right)}^{2} , \end{equation} (5)where g i o u $g_{iou}$ is the IoU value between the proposal and the ground truth closest to it. For classification, we use the standard softmax loss: L c l s = − ∑ i = 0 C I i = t log P t and P t = exp p t ∑ j = 0 C exp p j , \begin{equation} L_{c l s}=-\sum _{i=0}^{C} I_{i=t} \log {\left(P_{t}\right)} \text{ and } P_{t}=\frac{\exp {\left(p_{t}\right)}}{\sum _{j=0}^{C} \exp {\left(p_{j}\right)}}, \end{equation} (6)where I i = t $I_{i=t}$ is an indicator function that equals 1 if i is the ground truth class label t, otherwise 0. Finally, we accumulate the losses of the three attributes. The overall training objective is defined as follows: L = L cls + β L loc + γ L ov , \begin{equation} \mathcal {L}=\mathcal {L}_{\mathrm{cls}}+\beta \mathcal {L}_{\mathrm{loc}}+\gamma \mathcal {L}_{\mathrm{ov}}, \end{equation} (7)where β and γ are the hyperparameters that need to be adjusted. Uncertainty modeling However, inaccurate regression boundaries and unreliability of final scores may be caused by the ambiguity of action instance labeling. To mitigate the impact of this ambiguity, we estimate the uncertainty of the boundary in addition to the standard regression loss (uncertainty-aware boundary regression). This uncertainty indicator is further used to adjust the final score of the proposal (Reliability Score) and the regression boundary (Uncertainty Voting). 3.3.1 Uncertainty-aware boundary regression As mentioned above, we build a Gaussian model to reduce the impact of labeling ambiguity. This model locates the action instances' locations while simultaneously estimating the regression boundary's uncertainty: P ( g i | x i ) = 1 2 π σ i e − g i − x i 2 2 σ i 2 , \begin{equation} P({g^i}| x^i)=\frac{1}{\sqrt {2 \pi }\sigma ^{i}} e^{-\frac{{\left(g^i-x^i\right)}^{2}}{2 {\left({\sigma ^i}\right)} ^{2}}}, \end{equation} (8)where σ denotes the boundary uncertainty variance and g i $g^i$ is the corresponding ground truth of the location prediction x i $x^i$ . Equipped with this model, The learnable parameters Θ ̂ $\hat{\Theta }$ that maximize the probability P ( y i | x i , Θ ) $P({y^i}| x^i,\Theta )$ over N samples are then estimated. Θ ̂ = arg max Θ ∏ i = 1 N 1 2 π σ i e − g i − x i 2 2 σ i 2 . \begin{equation} \hat{\Theta }=\underset{\Theta }{\arg \max }\prod _{i=1}^{N} \frac{1}{\sqrt {2 \pi } \sigma ^i} e^{-\frac{{\left(g^{i}-x^i\right)}^{2}}{2 {\left(\sigma ^{i}\right)}^2}}. \end{equation} (9)The logarithm of Θ ̂ $\hat{\Theta }$ is: arg max Θ − N 2 ln 2 π − N 2 ln σ i 2 − 1 2 σ i 2 ∑ i = 1 N x i − y i 2 . {\fontsize{9}{11}{\selectfont{ \begin{eqnarray} \underset{\Theta }{\arg \max }{\left\lbrace -\frac{N}{2} \ln 2 \pi -\frac{N}{2} \ln {\left({\left(\sigma ^{i}\right)}^2 \right)} -\frac{1}{2 {\left(\sigma ^{i}\right)}^2} \sum _{i=1}^{N}{\left(x^{i}-y^i\right)}^{2}\right\rbrace}.\nonumber\\ \end{eqnarray}}}} (10)The Gaussian likelihood estimation can be treated as a regression loss by omitting the correlation term (i.e. N 2 ln 2 π $\frac{N}{2} \ln 2 \pi$ ) that does not depend on the estimated parameter Θ: L l o c ′ ∝ g − x 2 2 σ 2 + 1 2 ln σ 2 . \begin{equation} L^{\prime }_{loc} \propto \frac{{\left(g-x\right)}^{2}}{2 \sigma ^{2}}+\frac{1}{2} \ln {\left(\sigma ^{2}\right)}. \end{equation} (11)For some tiny values of σ2, we notice that L l o c ′ $L^{\prime }_{loc}$ can be negative. Because the duration of actions fluctuates greatly, this condition occurs from time to time, resulting in training errors. As a result, to avoid a negative loss, we adjust the regression loss as follows: L l o c ′ = g − x 2 2 σ 2 + 1 2 ln σ 2 + 1 . \begin{equation} L^{\prime }_{loc} = \frac{{\left(g-x\right)}^{2}}{2 \sigma ^{2}}+\frac{1}{2} \ln {\left(\sigma ^{2}+1\right)}. \end{equation} (12) Uncertainty post-processing In post-processing, the traditional localization process multiplies the classification and overlap scores as a detection criterion score for non-maximum suppress (NMS). However, unreliable proposals with high classification or overlap scores may be selected. To limit the impact of unreliable proposals, we employ the uncertainty score in conjunction with classification and overlap scores to pick more reliable proposals (i.e. Reliability score) before the NMS. And in the NMS process, the proposal score, as well as the related boundaries are regressed in uncertainty voting. In this regression approach, It is worth noting that we incorporate category uncertainty to avoid over-refinement of categories with high uncertainty. And the details are described below. 3.4.1 Reliability score We employ ( σ s , σ e ) $(\sigma _s,\sigma _e)$ in conjunction with classification and overlap scores to pick more reliable proposals. The modified proposal score is: S = p o v × max p r r = 1 C × S Reliability , S Reliability = ( 1 − f ( σ s 2 ) ) × ( 1 − f ( σ e 2 ) ) , f ( σ 2 ) = e σ 2 ∑ e σ 2 , \begin{equation} \begin{aligned} &S=p_{o v} \times \text{max}{\left({\left\lbrace p_{r}\right\rbrace} _{r=1}^{C}\right)} \times S_{\text{Reliability }}, \\ &S_{\text{Reliability}} = (1-f(\sigma ^2_s)) \times (1-f(\sigma ^2_e)), {f}(\sigma ^2)=\frac{e^{\sigma ^2}}{\sum {e^{\sigma ^2}}}, \end{aligned} \end{equation} (13)where f ( · ) $f(\cdot )$ is a softmax function to normalize the reliability score. ∑ e σ 2 ${\sum {e^{\sigma ^2}}}$ denotes the index summation of all input proposals' uncertainty variance. 3.4.2 Uncertainty voting In uncertainty voting, we refer to Soft-NMS [19] that update the candidate proposal's confidence score based on the overlap between the selected proposal and the candidate proposals. Specifically, we aim to award low ratings to candidate proposals that correspond to large boundary uncertainties. The formula for softnms is: S i = S i , I o U m , b i < N t S i · σ b σ c i 1 − f σ b 1 − I o U m , b i , I o U m , b i ≥ N t , {\fontsize{9.5}{11.5}{\selectfont{ \begin{eqnarray} \begin{aligned} S_{i}= {\begin{cases} S_{i}, &\quad IoU {\left(m, b_{i}\right)}<N_{t} \\[11pt] S_{i} \cdot \dfrac{\sigma _{b}}{\sigma _{c}^{i}} {\left(1-f{\left(\sigma _{b}\right)}\right)} {\left(1-IoU{\left(m, b_{i}\right)}\right)} , &\quad IoU{\left({m}, b_{i}\right)} \ge N_{t} \end{cases}}, \end{aligned}\nonumber\\ \end{eqnarray}}}} (14)where s i $s_i$ is the candidate proposal i's original score, f ( · ) $f(\cdot )$ is the sigmoid function, σ b = ( σ s + σ e ) / 2 $\sigma _{b}=(\sigma _s + \sigma _e)/2$ is the average value of uncertainty boundary ( σ s , σ e ) $(\sigma _s,\sigma _e)$ , σ c j $\sigma _{c}^{j}$ represents the category uncertainty of the proposal, which is derived by averaging the mean value of all proposals in the same category j. And I o U ( m , b i ) $IoU(m,b_i)$ denotes the overlapping proportion of the selected proposal m and the candidate proposal b i $b_i$ . On the other hand, uncertainty voting aims to regress the selected proposal's boundaries through uncertainty: b m ′ = f σ c j p m b m / σ b , m 2 + 1 − f σ c j ∑ i ≠ m p i b / σ b , i 2 f σ c j p m / σ b , m 2 + 1 − f σ c j ∑ i ≠ m p i / σ b , i 2 p i = e − 1 − I o U m , b i 2 / σ t , I o U m , b i > 0 , \begin{equation} \begin{aligned} &b_m^{\prime } =\frac{f{\left(\sigma _{c}^{j}\right)} p_{m} b_{m}/ \sigma _{b,{m}}^{2}+{\left(1-f{\left(\sigma _{c}^{j}\right)}\right)}\sum _{i \ne m} p_{i} b / \sigma _{b, i}^{2}}{f{\left(\sigma _{c}^{j}\right)} p_{m}/ \sigma _{b,{m}}^{2}+{\left(1-f{\left(\sigma _{c}^{j}\right)}\right)}\sum _{i \ne m} p_{i} / \sigma _{b,{i}}^{2}} \\[10pt] &p_{i} =e^{-{\left(1-IoU{\left(m,b_{i}\right)}\right)}^{2} / \sigma _{t}}, IoU {\left({m}, b_{i}\right)}>0 \end{aligned}, \end{equation} (15)where b m ′ $b_m^{\prime }$ is the temporal boundary of the selected proposal m after regression, and σ t $\sigma _t$ is the hyperparameter. In this way, we aim to avoid over-regression on classes with high uncertainty. Specifically, for the category s i g m a c j $sigma_c^j$ with large uncertainty, we reduce the weight of the boundary adjustment by 1 − f ( σ c j ) $1- f(\sigma _{c}^{j})$ and increase the weight of the original boundary of the chosen proposal by f ( σ c j ) $f(\sigma _{c}^{j})$ . ALGORITHM 1. Uncertainty Post-processing B $\mathcal {B}$ is N × 2 $N\times 2$ matrix of initial proposal boundaries. S $\mathcal {S}$ contains corresponding final proposal scores. U denotes the boundary uncertainty variance. B = { b 1 , . . , b N } $\mathcal {B} = \lbrace b_1 , ..,b_N\rbrace $ , C = { U 1 , . . , U N } $\mathcal {C} = \lbrace U_1, ..,U_N\rbrace$ b = ( x s , x e ) $b=(x_s,x_e)$ , U = ( σ s 2 , σ e 2 ) $U=(\sigma ^2_s,\sigma ^2_e)$ D ← { } $\mathcal {D} \leftarrow \lbrace \rbrace$ T ← B $\mathcal {T} \leftarrow \mathcal {B}$ Estimate S $\mathcal {S}$ by Equation (13) while T ≠ $\mathcal {T} \ne$ empty do m ← $m \leftarrow$ argmax S $\mathcal {S}$ T ← T − b m $ \mathcal {T} \leftarrow \mathcal {T} - b_m$ Refine S $\mathcal {S}$ by Equation (14) Refine b m $b_m$ by Equation (15) D ← D ⋃ b m $\mathcal {D} \leftarrow \mathcal {D} \bigcup b_m$ end while return D , S $\mathcal {D}, \mathcal {S}$ 4 EXPERIMENTS Experimental settings 4.1.1 Datasets We conduct experiments on two commonly used datasets THUMOS14 [20] and ActivityNet v1.3 [21]. For THUMOS14, it is standard procedure to train on a validation set containing 200 temporal annotations in 20 categories and to evaluate on 213 test videos. In this dataset, each video has an average of 15 action clips, and some videos have numerous action clips. Furthermore, the video length varies widely, ranging from a few seconds to over an hour. For the reasons stated above, it's challenging to perform temporal action detection on THUMOS14. ActivityNet v1.3 is a widely used benchmark for temporal action detection. It contains 10024 training, 4926 validation, 5044 test videos, and 200 action categories. Most of the videos in this dataset have instances of activity in a single category. This dataset is much larger than THUMOS14 in terms of the number of activity categories and the number of videos. We use a training subset for training and a validation subset for testing, as is standard practice [2, 12]. TABLE 1. Effectiveness of each component in uncertainty method based on [2] in THUMOS14. And the mAP of tIoU is calculated by using 0.5 threshold ([email protected]). For [email protected], the basic framework performs at 55.5%. The performance is enhanced to 57.5% after incorporating three proposed approaches Method Performance Uncertainty modeling × ✓ $\checkmark$ ✓ $\checkmark$ ✓ $\checkmark$ ✓ $\checkmark$ Reliability score × × ✓ $\checkmark$ × ✓ $\checkmark$ Uncertainty voting × × × ✓ $\checkmark$ ✓ $\checkmark$ [email protected](%) 55.5 56.1 56.8 57.0 57.5 4.1.2 Evaluation metrics Implementation details We use the uncertainty method on one-stage Tb-SSAD [1], two-stage P-GCN [12], and anchor-free approach AFSD [2], respectively. For a fair comparison, we stick to their initial setup in our studies. To extract the video's features, we fine-tune the I3D [16] model that was pre-trained on Kinetics. In the training process, Our model is trained by Adam [22] with a learning rate of 10−5 and weight decay of 10−3. We apply random crop as well as horizontal flipping for data augmentation. The results of RGB and optical flow frames are averaged to determine final locations and class scores. Ablation study To verify the validity of the individual modules of uncertainty, we employ our algorithm on the state-of-the-art method [2]. As shown in Table 1, the model's performance improves from 55.5% to 56.1% by integrating uncertainty modeling in the training phase. With uncertainty modeling, the model's performance improves from 56.1% to 56.8% and 57.0%, respectively, when the reliability score and uncertainty voting are combined throughout the testing process. Finally, by combining the three modules above, our method achieves a 2.0% improvement on [email protected] when compared to [2]. In post-processing, we visualize the reliability score and uncertainty voting in Figures 3a and 3b, respectively. The reliability score is calculated by multiplying the uncertainty score to the final proposal score, resulting in the se