Feature Integration and Group Transformers for Action Proposal Generation
Authors: He-Yen Hsieh
Ding-Jie Chen
Tung-Ying Lee
Tyng-Luh Liu
Venue:  (2021)
Keywords: temporal action proposal
transformer
video analysis

Abstract: The task of temporal action proposal generation (TAPG) aims to provide high-quality video segments, i.e., proposals that potentially contain action events. The performance of tackling the TAPG task heavily depends on two key issues, feature representation and scoring mechanism. To simultaneously take account of both aspects, we introduce an attention-based model, termed as FITS, to address the issues for retrieving high-quality proposals. We first propose a novel Feature-Integration (FI) module to seamlessly fuse two-stream features concerning their interaction to yield a robust video segment representation. We then design a group of Transformer-driven Scorers (TS) to gain the temporal contextual supports over the representations for estimating the starting or ending boundary of an action event. Unlike most previous work to estimate action boundaries without considering the long-range temporal neighborhood, the proposed action-boundary co-estimation mechanism in TS leverages the bi-directional contextual supports for such boundary estimation, which shows the advantage of removing several false-positive boundary predictions. We conduct experiments on two challenging datasets, ActivityNet-1.3 and THUMOS-14. The experimental results demonstrate that the proposed FITS model consistently outperforms state-of-the-art TAPG methods.

Under review as a conference paper at ICLR 2021
FEATURE INTEGRATION AND GROUP TRANSFORMERS FOR ACTION PROPOSAL GENERATION
Paper ID: 2544
Paper under double-blind review
ABSTRACT
The task of temporal action proposal generation (TAPG) aims to provide high- quality video segments, i.e., proposals that potentially contain action events. The performance of tackling the TAPG task heavily depends on two key issues, representation and scoring mechanism. To simultaneously take account of both aspects, we introduce an attention based model, termed as FITS, to address the issues for retrieving high-quality proposals. We first propose a novel Feature- Integration (FI) module to seamlessly fuse two-stream features concerning their interaction to yield robust video segment representation. We then design a group of Transformer-driven Scorers (TS) to gain the temporal contextual supports over the representations for estimating the starting or ending boundary of an action event. Unlike most previous work to estimate action boundaries without considering the long-range temporal neighborhood, the proposed action- boundayyy co-estimation mechanism in TS leverages the bi-directional contextual supports for such boundary estimation, which shows the advantage of removing several false-positive boundary predictions. We conduct experiments and THUMOS-14 The experimental results demonstrate that the proposed FITS model consistently outperforms state-of-the-art TAPG methods.
challenging datasets, ActivityNet-1.
INTRODUCTION
Owing to the fast development of digital cameras and online video services, the rapid growth of video sequences encourages the research of video content analysis. The applications of interest include video summarization Yao et al. 2015 2016), captioning (Chen et al. 2019a Chen & Jiang 2019), grounding (Chen et al. 2019b), and temporal action detection (Gao et al.) 2019 Zhang et al., 2019). The temporal action detection task is an important topic related to several video content analysis methods, and aims to detect the human-action instances within the untrimmed long video sequences. Like the image object detection task, the temporal action detection can be separated into a temporal action proposal generation (TAPG) stage and an action classification stage. Recent studies (Escorcia al. 2016 Buch et al.) 2017b Lin al. 2018 Liu et al.) 2019 Lin al. 2019 2020) demonstrate that the way to pursue the proposal quality clearly improves the performance of two-stage temporal action detectors. To this end, temporal action proposal generator is demanded to use a limited number of proposals for capturing the ground-truth action instances in a high recall rate, hence reducing the burden of the succeeding action classification stage.
One popular way to tackle the TAPG task is to generate the proposals via the estimations of boundary and actioness probabilities. The boundary probability usually factorized as the starting and ending for an action instance. Rather than directly estimating the actioness boundary as the existing methods, we leverage the actioness estimation and the additional background estimation in a bi-directional temporal manner to co-estimate the action boundaries. The background means existing no actions. This sort of boundary estimation derived from the observation that the features for describing the long-time actioness/backgroumm are more consistent along the temporal dimension than the short-time starting/ending. Therefore, estimating the boundary with the actioness and background features allows us to estimate the proposal boundaries of much less false-positive, hence obtaining the high-qualit proposal candidates for further scoring. In practice, we estimate an action starting boundary as the time of descending background with simultaneous ascending actioness. In contrast, action ending boundary occurs with the ascending background with descending actioness. Figure | Iillustrates our action-boundary co-estimation mechanism.

---

Under review as a conference paper at ICLR 2021
Foowers
Video
Action Boundary Co Esiimttio
Action Boundaries
Starting
Deckvess
ya
Decepsed
allonded
Transtormer-driven Scorers
(TS)
Action-
Eniling
The proposed module estimates four boundary estimations of forward-actioness, background, and bockwardd-ackgrowne estimate the action starting, and the forward-actioness and backward- wward-background co-estimate the action ending. The transformer-style units enable these temporal estimations to collect the temporal contextual supports over each inputted representation. The right-most figures show the estimations of action starting and action ending without (top) and with (bottom) the TS co-estimation. The proposed action- bounddy co-estimation within our TS module is able to reduce more false- dalle--ositive predictions.
Figure 1:
co-estimation mechanism. Our transformer-driven scorers
forward- The backward-actions and forward -wwaudd-bbbooounddd co-
This paper introduces an effective temporal action proposal generator, i.e., FITS, which aims to provide the action proposals, that precisely and exhaustively cover the numan-action instances. By considering the two essential TAPG issues, namely the feature representation and the scoring mechanism, and the above-mentioned action- couioorb co-estimation, our attention- based FITS model comprises Feature Integration (FI) module and Transformer-drite
Scorers (TS) module for dealing with these considerations. Precisely, our module enhances the common TAPG two-stream features (Simonyan & Zisserman 2014 Wang et al., 2016; Xiong et al., 2016) by concerning the feature interaction. The previous TAPG methods usually directly concatenate the appearance stream and motion stream features for usage. In contrast,
were inspired by the non-local attention mechanisms (Wang et al. 2018 Hsich et al. 2019) to extend such a long-range attention mechanism for integrating the two-stream features. As result, experiments show the robustness of the integrated features by reducing their mutual feature discrepancies. More importantly, to score the temporal action proposals for discriminating high-quality ones, we devise novel transformer-driven scoring mechanism. The TS mechanism leverages the temporal contextual supports over the feature representations to obtain the self-attended representations and then associates these self-attended representations to co-estimate the action boundaries. The experiments show the retrieved action proposals containing much less false-positive ones. Figure overviews our temporal action proposal model, termed as FITS network.
To sum up, our main contributions include i) We introduce the novel feature integration module to integrate the two-stream features by reducing their feature discrepancies via non-local-style attention and obtaining robust representation. iī) We devise the novel transformer-driver scorers module to co-estimate the transformer-driven self-attended representations, which leverage long-range temporal contextual supports. Hence, we are able to retrieve high-qualit, temporal action proposals. iii) The extensive experiments demonstrate that the proposed FITS model achieves significantly better performance than current state-of- offte--f-thearrr the-art TAPG methods.
RELATED WORK
Feature Representation. As a de facto trend, instead of using the handcrafted features, the neural- network-based features are widely employed for addressing the action classification task. These popular neural network approaches include the two-stream networks (Simonyan Zisserman, 2014] Feichtenhofer et al.) 2016: Wang et al. 2016), which separately represent the appearance feature and the motion feature, and 3D networks (Tran et al. 2015] Carreira & Zisserman 2017: 1 Qiu et al.
2017 Xu et al. 2017), which directly represent video sequence as the spatio- spatio-temporal feature. In
this paper, we use the action recognition model (Wang et al.] 2016)) Xiong et al.] 2016) to extract two-stream features for representing each untrimmed video sequence.
2

---

Under review as a conference paper at ICLR 2021
Attention Mechanism. The attention mechanism is the process of selectively focusing on a few relevant things in comparison with everything. Fields like natural language processing and computer vision, broadly leverage such attention mechanism. For instances, Bahdanau et al. (2015) enable their model to focus on searching a group of related words from the input sentences for predicting the target words, Xu et al. (2013) introduce the soft and hard attention to generate image captions, and LFB (Wu et al. 2019) introduces long-term feature banks to analyze videos, Hsich et al. (2019)
mastances,Bahdanau
employ non-local attention (Wang et al., 2018) capture long-range dependencies and channel attention (Hu et al. 2018) to re-weight the channel-level feature maps. Since Vaswani et al. (011)) introduce a self-attention mechanism, called Transformer, for exploring the intra- attra-attention and inter- attention to address the machine translation task, the Traasfoomer-basee models show their advantages to tackle various tasks, such as object detection (Carion et al.) 2020), action recognition (Girdhar et al. 2019). image generation (Parmar et al., 2018), and image captioning (Cornia et al. 2020). We
notice that the work of Girdhar et al. (2019) that classifies the human actions by first use the RPN (Ren et al. 2015) to localize the human body parts as several frame crops, and then encodes these crops via Transformers for the subsequent classification. In contrast, a TAPG task discriminates whether given video segment, composed of several video frames, covering any action event. Our model extends the merits of these attention efforts for constructing a robust action proposal generator.
Temporal Action Proposal Generation. We categorize the TAPG methods into Anchor-based (Gao et al., 2017, Heilbron et al. 2016, et al. 2016) and Probability-bar -based (Lin et al., 2020 2019 2018 Zhao et al. (017). The former focuses designing several multi-scall anchor boxes to cover action instances, while the latter estimates the temporal location probabilities of the action instances. Besides, some methods (Gao et al. 2018. Liu al. 2019: Gao et al. (2020) also explore the way to integrate the above-mentioned two categories for precisely localizing the temporal boundaries. In anchor-based methods, the S-CNN (Shou et al. 2016) and Heilbron et al. (2016) respectively evaluate anchors via C3D network and sparse learning, and TURN (Gao et al. (017)) suggests regressing the temporal boundaries of action instances. The probability- work, TAG (Zhao et al., 2017p, generates action proposals via temporal watershed algorithm to merge contiguous temporal locations of high actioness probabilities. BSN (Lin et al. 2018) generates proposals as well as their confidence by formulating the probabilities of boundaries and actioness. BMN (Lin et al.) 2019) proposes a houndary-matching matching mechanism to evaluate the confidence among densely distributed proposals. DBG (Lin et al. 2020) uses the maps of dense boundary confidence and completeness to further score boundaries for all action proposals. BC-GNN (Bai et al.) (2020) employs a Graph Neural Network model the relationship between the actioness and boundaries. Zhao et al.) (2020) propose loss terms to regularize the feature consistency between the actioness and boundaries.
In sum, the anchor- based methods focus the anchor-box design and usually lack the flexible temporal boundaries for various action instances. The existing probability- -based methods may generate the action proposals, but merely rely on the actioness (Zhao al. 2017) or the boundary information (Lin et al., 2019), or lack the association between the actioness and the boundaries of action instances (Lin et al., 2020 2018). By contrast, we introduce the Transformer-drive scorers module to score proposals by explicitly associating the self-attended representations during scorers training. Combining with our feature integration module for retrieving the robust feature representation, the experiments show that our model outperforms the existing leading TAPG methods.
3 METHOD
TAPG Formulation. Given video sequence X' = { of N frames, we assume it includes action instances. The TAPG task aims at generating a proper set of action proposals that can be used to detect the underlying human actions in X. We denote an action proposal as indicate that the proposal p starts at the sth frame and ends at the eth frame of X. Analogous to the object proposals for detection, action proposals are generic and class agnostic. Let the I ground-truth actions of X' be y = {(s", ee), = (sf, (rtee)) e)) An action proposal (s. is said to be matched to some ground- -oound-truthhao action (s* ee) if their time-interee IoU (in terms of frames) is greater than specified threshold r. Considering a set P of proposals, its goodness to N' can be explicitly measured by the number of matched action proposals.
es to
3

---

Under review as a conference paper at ICLR 2021
Feature Integration
Transformer-drivee Scorers
Encoder
Time
Pusiional Encoding
Enoodor
Docodar
Hoad Solectin A&N
FFN
MHA
Ian
jp'e
ASN
Enoodor
Decodar
heads
|
Enoodor
Decoder
pl
Decoder
Pookional Ensod ng
M
Head Selecton
Hoad Selectin
masked
Eriodde
Dacoder
AAN
A&N
hiosss
Video
A
Acte
Dackgroand
AMN
And Faam
FFN
Figure 2: The architecture of our FITS temporal action proposal generation model. Our model is composed of Feature Integration module, in charge of integrating the two-stream representations, and Transformer-dring Scorers module, in charge of extracting the temporal contextual supports within representation and then associating these representations to co-estimate the action boundaries.
3.1 FEATURE INTEGRATION
We decompose each video sequence X' into a set of, say, consecutive video segments, denoted V [v.)ī-1. where is called snippet. Each snippet is then represented with two-stream features. To account for videos of various lengths, we adopt the setting in BSN (Lin et al. 2018) to sample single-stream features the temporal (frame) dimension to consistently obtain 7 snippets per video sequence. Precisely, each video sequence X' is represented by an appearance feature tensor A E CCT and motion feature tensor M E RCxT, To enhance the representation power of each snippet, we enrich the representation by three-step operations: co-attention, cnntuallxxcitaiion, and aggregation. Before we describe the steps, we first define a basic convolutional layer & by
o(X; f. o) = ReLU(W * x+b). b).
where X, f. respectively denotes the input feature, the filter size, and the number of the output filters. ReLU, W, b means the activation, convolution operation, the weights and bias of Φ.
Co-attention. We consider the non-local function (Hsieh et al. 2019 Wang et al. 2018) to explore the snippet-level correlations between the two-stream cues. Hence we enable our model to learn to emphasize the temporal correlations of human-action descriptions between the two-stream cues. Given a video sequence X, its appearance tensor A E BCxT and motion tensor M E CCTT the proposed two-stream co-attention achieved by carrying out the following feature re-adjustment:
A= (A: A, M ((M; @M,
(2)
where Å, M E RCT, the non-local function ((.) yields T snippet-level feature correlations for feature adjustment that conditioning on the other feature, and denotes the element-wise addition.
Mutual-excitation. We further enhance the two-stream features by re-weighting over the channel dimension C. To this end, we introduce a mutual-excitation mechanism re-weight each single- stream feature concerning the chamell-ttention from the other stream. A convolution layer ø of filter size 1 over the channel dimension is applied to re-organize the two-stream features. With the adjusted features, we construct a two-stream -uxital-xcitatioo function Y to extract the channel attention vector from one stream excite the feature of the other stream. That is, we have
Ā (o(M; I)) o(A: 1,C), M ( ; 1 1,C) O (MM; x 1,C), (3)
where Á E RXT, the mutual-excitation function v(()) is generalized from munezze-and-exciation Huu et al. 2018 for describing thannel-attention. and @ symbolizes the element-wise multiplication.
Aggregation. Inspired by the inception block (Szegedy et al. 2015 2016), we further enrich the feature representation concerning the multi-scale temporal contexts before integrating the two-stream features. Given the two-stream features À and M, we employ the convolution layer and max-pooling layer to respectively map each of the two-stream features in two different temporal contexts by
A (A;1133 C'), À b(p(A)) x3. C"), M1 = o(M; 3, M₂ ((p(M) C'), (4)
4

---

Under review as a conference paper at ICLR 2021
where A1, A2, M2 RC'' the max- max-pooling function p()) using the filter of size 1 3. We then concatenate all features followed by another convolution layer to unify them. Formally, the aggregated feature, say F, is generated by
(AA A2 M1 || M2; x x C").
(5)
where the notation symbolizes the concatenation over an augmented last dimension, namely, (Ai A2 || Mi || M2) E R xT'x, and F E IRC x1', is squeezed over the augmented last dimension.
3.2 TRANSFORMER DRRIVEN SCORERS
Our transformer-driven scorers comprises four encoder-decoder pairs. Both encoder and decoder made of stacks of attention layers. Each encoder-decoder pair, i.e., a Transformer unit, in charge of extracting the inputted representation F's temporal contextual supports, in which the temporal contextual supports consider one single temporal direction with respect to the actioness or back- ground. Hence, our encoder-decoder pairs generate the attended representations F Fba, F Fbb for
and backward bww.ardbbckgouudd. respec- tively. Figure 2 the left part sketches our FITS model, and the right part details the architecture of one transformer units, ie., encoder-decoder. Precisely, we employ a one-layer eight-head Transformer with additional head-selection operation as one single encoder-decoder pair within our FITS model.
backward-actioness, Borward-backgroud
Basic Units Since our model shares the same Transformer's basic units, i.e, positional encoding, multi-head attention, add & norm, feed forward network, and masked multi-head attention, we refer the readers to Vaswani et al. (2017) for more details about these units. Here we briefly review the original Transformer architecture. The Transformer a self-attention mechanism to compare each feature with all other features in the sequence. This paper employs such self-attention mechanism to gain the temporal contextual supports from all other snippets for the snippet-level representation. The mechanism embeds the inputted feature to query Q, key K, and value V via linear projections. The product of Q and K formulates the attention weights for adjusting the value V as the output.
Transformer Unit. Each of our encoder-decoder pair takes representation of F as input, and map it into query, key, and value. For a snippet of one action event as the query, the encoder-decoder pair then compares the relationship between the query and the keys of all other snippets to update the snippet's value. Intuitively, the self-attention process collects the contextual information from all other snippets to adjust the queried snippet's representation. Therefore, each snippet-level prediction actually considers the other snippets and hence smooths the prediction results, as shown in Figure [] We now show the method employ our encoder-decoder pairs, An attention function of a head; (Vaswani et al. 2017) is defined as:
head, (Q,K,V) V) = softmax( QKT
V
(6)
Vd
where d denotes the normalization term. The head attention function is the main component of the multi-head attention module and the masked multi- attention module in the Transformer. Yet use the additional head selection to fuse all heads within these modules. Our encoder and decoder empirically respectively take the representations of Á || M and F as inputs. We use four encoder-decoder pairs to generate the attended representations Fla pba F pbb E XO"" Note that the inputs for Fha and Fhb are reversed over the temporal dimension concerning the backward temporal relation. For an attended representations F' E {Fit, Fba, Fth, pbb), J. we use (1) to predict its probability p (FF;; x 3, 1) E RIXT Hence, we can obtain the probabilities pla pbs, plb pbb,
Head Selection. Unlike Vaswani et al. (2017) to directly concatenate all heads' results with a subsequent linear projection, we present the head selection to make each neuron adaptive select its representation from the multiple heads. The head selection & is defined as
{(hhead,) }) o( Σ (scale; head;);; x 1, 1,
(8) where head, denotes the ith head result, Σ denotes the element wise summation, FC and GAP are fully-comnection and global average pooling, respectively. Note that the softmax in (8) operates over the head dimension rather than the channel or temporal dimension. The head selection is designed to adjust the channel scales for enhance the representation power as the SE-Net (Hu et al. 2018).
scale; Softmax(FC(GAA Σ hcad,))),
5

---

Under review as a conference paper at ICLR 2021
Scorers. Intuitively, snippet v, could be action starting boundary if its previous snippet v,-1 is considered as a background, and its subsequent snippet Vi+1 is likely as an action instance. On the other hand, snippet v; could be action ending boundary if V.-1 is also an action instance, and v++1 likely as a background. Bearing these observations in mind, we predict the probabilities of action starting ps and action ending pe over all snippets with convolution layers by
p ( Fbaa pa
Fh;; 1 3, C); 1,1),
(9)
p (F
Pbb, 1 3, 1x
(10)
where P*, Pe E RIxT, the notation symbolizes the concatenation over the channel-dimension Besides the snippet-level probabilities Pa and P we further consider the proposal probabilities. Given a proposal starting from the ith snippet to the jth snippet, we predict the action-covering probability pe and boundary-relation probability pse as
p = ((H 1 x 3 x 3), pat = - 6H 3 3)
(11)
where p ps 1I Each element H E H indicates the truncated probability vector from the ith bin to the jth bin, followed by ROI-Align (Ren et al. 2015), where the probability vector is 6 | 1 3, 1). Each element H : E H indicates two truncated intermediate representations of (9) and (T)) from the ith snippet the jth snippet, followed by element-wise addition, where the intermediate representations are results of the first convolution layer of (9) and (TO). Note that we calculate Pe and ps by using the filter of size 3 x 3 to consider the neighboring proposals.
Given an action proposal starting from the ith snippet to the jth snippet, we empirically define its score Siz with the probabilities mentioned above as:
Si cxp(P) PJ) x x (Pi))
(12)
where exp PJ), Ph and (P:;) 11/4 are introduced for respectively preferring high boundary confidence, high boundary correlation for starting-ending pair, and high overlapping ratio between proposal and action-instance In our implementation, we follow BSN (Lin et al. 2018) to collect candidate proposals by employing the probabilities of Pa and Pe, Thus, each proposal is scored via
(12) and followed by the soft non-maximum suppression for retrieving the top-scored proposals.
3.3
OPTIMIZATION
The overall loss function for training is formulated as multi-task objective that comprises encoding loss (Cenc) and scoring loss (Ceer):
L Lene + X Lser,
(13)
where Lene is used for training the four probabilities pfa Nir,pba,pp ,,,, pbb, and Leer is designed for learning the remaining probabilities ps, p p and p The encoding loss Lene encourages each Transformer to encode all its representations for predicting the probabilities of actioness or background and the scoring loss Leer encourages all Transformers to correlate their representations for ranking the proposals. Both the loss terms in (13) employ the binary logistic regression loss.
4 EXPERIMENTS
Datasets and Metrics. We conduct experiments on ActivityNet-L.3 (Heilbron et al. 2015) dataset and THUMOS-14 14 (Jiang et al. 2014) dataset. The ActivityNet-1.3 is large-scale action understand- ing dataset, which available for evaluating the tasks of proposal generation, action recognition, temporal detection, and dense captioning. There are 19,994 temporal annotated untrimmed videos comprising 200 action categories. The THUMOS- 14 dataset contains 1,010 validation videos and 1,574 testing videos of 20 action categories. There are 200 validation videos, and 212 testing videos contain temporal action annotations. We use the validation set for training and use the testing set for evaluating. We use three metrics to evaluate the proposal quality. i) AR @ AN, which evaluates the relation between Average Recall (AR). that calculated with multiple specified IoU thresholds, and Average Number of proposals (AN). AUC, which denotes the area under the AR vs. AN curve.

---

Under review as a conference paper at ICLR 2021
Table I: Comparison of the state-of-the-art methods on ActivityNet-1.3 1.3 validation and testing split and on THUMOS- 14 testing split. Notation "*" indicates the model using non-two-stream features.
ActivityNet-113
THUMOS-14
Method
AR 100 (val)
AUC (val)
AUC (test)
AR
AR#100
AR&200
AR
AR 1000
GTAN
Long et al.
2019
74.80
67.10
67.40
54.30
BMN
Lin
9019
75.01
67.10
67.19
39.36
47.72
54.70
62.07
65.49
RupNet
Gao ct al.
2020
76.71
67.63
67.72
40.35
48.23
$4.92
61.41
64.47
DBG Lin et al.
2020
76.65
68.23
68.57
37.32
46.67
$4.50
62.21
66.40
Zhao et
75.27
66.51
44.23
50.67
55.74
BC-GNN Bai et al.
2020
76.73
68.05
40,50
49.60
56.33
62,80
66.57
FITS
77.59
69.36
70.00)
40.32
49.53
$7.55
65.34
69.56
Label Assignment. We adopt actioness Pa, action starting Ps, and action ending P to estimate each frame's probabilities being an actioness, action- startinn point, and action-ending point, re- spectively. Given video sequence X' = { of N frames, we assume it includes I action instances. Hence the ground-truth actions of X are y = {(s", ee);; = e We then define the ground-truth probabilities of ps, ps, and pe within time span of äth action instance
(14) where we shift the time span by 6; [e" -=)//n and we set r to 40 empirically. Given the wihh frame, we represent the corresponding time span as T, = [n, n + 1]. We then use T, to calculate the overlap ratio with respect to T,", TV and TV. for actioness, action starting, and action ending, respectively. The overlap ratio serves as the ground-truth probability. The probabilities of actioness and background complementary with summation of 1. Note that the ground-truth actioness probabilities for both directions, i.e., Pla and pba, are the same. Analogously, the ground-truth background probabilities of pbb and pbb are the same.
The = [s]
= 6;, + T |eq 5,,e7 + 6:]
We take two aspects to define proposal-level confidence scores. First, we determine the action- covering ratio the probability pe for each action proposal e) by calculating the time-interval IoU with all action instances I. The action instance of the highest IoU value is regarded as the ground- truth action-covering score of the proposal. That is, a time-interval IoU determines the coverage between an action proposal and action instances within time interval of [s. e]. We then employ the aspect of the proposal's 's boundary to define the confidence score. Higher boundary probability potentially denotes action starting at the sth frame and ending at the eth frame. Specifically, we probabilities of pse for all proposals by applying the outer product between ground-truth probabilities of ps and Pe.
obtain the ground-truth boundary-
4.1 COMPARISON WITH STATE-OP-TIIB-ARTS
Table summarizes the comparison of our approach against state-of-the-art TAPG methods. Our model Significantly outperforms other TAPG methods all metrics of both datasets. Specially, improve AR @ 100 and AUC of ActivityNet-1.3 validation split by 0.86% and 1.13%, respectively. On the ActivityNet-1.3 testing split evaluating on the official server, we further improve AUC by 1.43%. On the THUMOS- 14 testing splits, though the very recent method Zhao et al. (2020) shows the outstanding ranking results and hence surpasses FITS AR@50 and AR@100, our FITS model surpasses all other methods with sufficient number of available proposals. This comparison also shows that our FITS is able retain more high-quality proposals compared to other TAPG methods.
4.2 ABLATION STUDY
We conduct detailed ablation study on ActivityNet- 1.3 validation split to realize the importance per component of our FITS model. Table summarizes the ablation study of the feature integration module and transformer-driven scorers module, and please notice assume single module in this ablation equips with all the components of the other module. Table summarizes the ablation study of the configurations in the transformerddriven scorers module.
The left part in Table 2 compares the components within the feature integra- tion module. The first row, i.e., baseline-FI, serves the baseline that concatenates the two-stream
Feature Integration.
7

---

Under review as a conference paper at ICLR 2021
Table Ablation study on ActivityNet-I.3 validation split. The meanings of abbreviations are CA: co-attention; ME: mutual-excitation; AG: aggregation; PG: performance gain on AUC; baseline-Fl Fl:: concatenating the two-stream features; baseline- TS: estimating action boundaries as BSN and BMN.
Componene
Integration
Compooeent pe pr
Transfortee: driven Scorees
CA
ME
AG
AUC
PG
ARR850
ARG91
ARS 100
P
AUC
PG
AB930
ARUF3
AlE 10)
bascline-Fi
67.51
66.54
71,02
14.47
75.85
bascline-TS
65.63
64.74
09.13
72.50
73.85
-0.40
67.11
71.40
14.81
+2.58
67.58
74:01
68.42
-0.91
67.52
71.92
73.23
76.55
V
68.50
+2.67
67.87
12.06
75.48
76,93
67.64
72.03
75.37
+2.97
67.37
72,34
75.37
69.36
-1.85
68.79
73/08
76.24
77.59
V
69.36
43.73
68.79
73.08
76.34
77.39
xelf-aticaaium
60.16
-1.65
63.65
72.81
16.00
wclf excitation
69.10
-1.59
68.35
72.77
76.12
77.29
Table 3: Ablation study of transformer-drivee scorers module on ActivityNet- 1.3 validation split.
Inpot
Tranefomer- driven Scorers
Input
Transformer driven Sconens
Frocsdar
Decaddrr
AUCC
PG
A0
AR3
Alceso
AR39 doo
Faender
Docadlr
AUC
PG
AR
Alleroo
ARR500
Alle-100
Al X7
A tr
67.49
66.62
70.25
74.35
75:00
A Jr
68 94
-0.85
67.49
71.88
75.00
76,
69.19
+1.70
68.59
72,77
76.11
77.42
A A2
69.36
+1.87
68.79
76.34
77.59
Ilead Selection
Transformer driven Sconers
Iead Selection
Transformer driven Scorens
heads
Opezation
AUC
PG
Alce
ARS99)
AReen
AR3e 1o0
houls
Operation
PG
AReeN)
ARre0
ARR/0
Concitenation
09.07
08.53
72,59
75.90
77.22
Selection
69:36
+0.29
68.79
73.08
76.24
77,59
features over the channel dimension for the subsequent proposal generation. The comparisons show that all three components, i.e., CA, ME, AG, contribute positively, Precisely, the CA & ME improve AUC by 0.91 % and the AG module improves AUC by 0.96%, which respectively shows that concern- ing the two stream's interaction and the multi-scale temporal contexts is beneficial for obtaining the robust video representation. With the complete FI module, we can further improve AUC by 1.85%. Furthermore, the bottom two rows show the comparison results once we employ self-adjussment on each single-stream feature. The results show that the two-stream interaction is indeed advantaged. Precisely, compared with our full model, i) when replacing co-attention with self-attention, the AUC declines to 69.16 by 0.20%; ti) when replacing nutual-excitation with self-excitation, the AUC also declines to 69.10 by 0.26%.
Transeormer-driven Scorers. The right part in Table compares the probability setting within the TS module. The first row, i.e., baseline- TS, serves the baseline that only employs the probabilities P & P to generate proposals. The comparisons show that all these scoring probabilities contribute positively. Precisely, the ps & P using our boundary co-estimation has significantly improved the AUC by 2.58%. With the aids of the proposal-level scoring probabilities pse and P we can improve more AUC by 0.29% and 0.39%. As result, our complete TS module remarkably improves AUC by 3.73%. The ablation results demonstrate that it is beneficial to retrieve action proposals concerning the boundary co-estimation and the association of the self-attended representations. Our TS-based approach fulfills the goal with noticeable performance gain.
Table 3 ablates the configurations of the TS module. The top part shows the various inputs for the encoder-decoded pair. The ablation shows that though the aggregated representation F helps use in the encoder-decoder pair, simultaneous employing the Á || M & F gains more performance improvement. The bottom part shows the advantage of our head selection. Comparing to concatenating multiple heads directly, our head selection shows higher performance improvement.
Visualization. Figure 3 visualizes the effects of employing our co-attention and -exettttixiiitiionn and transformer-driven scorers. The left figures select the two-stream features of the 70th snippet locating at an action instance for comparison. The result shows that our co-attention and mutual- excitation modules obviously reduces the feature discrepancies between two-stream features. As a result, our proposal generation can employ the two-stream features without noticeable preferences. The right four figures show the estimated action- boundary probabilities between "baseline- TS" and our "full model." In comparison with baseline-TS. -S, the full model estimates the boundary probabilities by using probabilities ps, P ps, and p The results show that our model contributes to estimating less false positive estimations, which again demonstrate the effectiveness of our FITS model.
8

---

Under review as a conference paper at ICLR 2021
Sacigrand
Actoness
Actovvetsiing
andding
Feature difiennc
AWil At witeet Co Jtsnion Mutic PectIaoo
Satc SHANC
prosatts
proshblly
Teature
Kejh Mi ddticttin Putto Nette'-crctata CAcRal01
proboblity
probusiity
Teature channel
2nippet
Snippet
Figure 3: Effect visualization of our model video id "plk9qMEEEd4." The top row shows the sampled frames of the video. The left two figures in the bottom image set, which depict the absolute differences (y-axis) between motion cue and appearance cue over feature channels (X-axis), show our co-attention and mutual-excitation mechanism can reduce the two-stream feature discrepancies. The right four figures in the bottom image set, which plot the predicted boundary probabilities (y-axis) P and P over the snippet dimension (x-axis), show our TS module's 's effect. In this four- four-image set, the top row shows the original results, and the bottom row shows the results using our modules.
CONCLUSIONS
We have shown that the proposed FITS model, which is composed of the Feature- Inteura-nntegotiionn module and the Transformer-driven Scorers module, better addresses the temporal action proposal generation task and achieves state-of-the-art performance. Specifically, the proposed action-boundary co- estimation mechanism shows its advantages in retrieving proposals of less false-positive predictions and, hence, helps retain high-qualit proposals. The extensive experiments show that the performance gain is derived from not only the Ceature-Integratee module, which enriches and integrates the two-stream features as a robust snippet-level representation, but also the Transformer-driver Scorers module, which first generates the self-attended representations concerning the temporal contextual supports and then associates these self-attended representations for proposal scoring. As a result, our FITS model effectively utilizes the standard two-stream features and explores the self-attended vidco representations, therefore, lead to sate-of-the-ane TAPG performances on two challenging datasets.
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. In ICLR, 2015.
Yueran Bai, Yingying Wang, Yunhai Tong, Yang Yang, Qiyue Liu, and Junhui Liu. Boundary content graph neural network for temporal action proposal generation. In ECCV, 2020.
Shyamal Buch, Victor Escorcia, Bernard Ghanem, Fei-Fei, and Juan Carlos Niebles. End-to-end, single-stream temporal action detection in untrimmed videos. In BMVC, 2017a.
Shyamal Buch, Victor Escorcia, Chuanqi Shen, Bernard Ghanem, and Juan Carlos Niebles. SST: single-stream temporal action proposals. In CVPR, pp. 6373 -7722,63,,200 2017b.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformer. In ECCV, 2020.
João Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In CVPR, pp. 4733- 2017.

---

Under review as a conference paper at ICLR 2021
Yu- Wei Chao, Sudheendra Vijayanarasinhan, Bryan Seybold, David A. Ross, Jia Deng, and Rahul Sukthankar. Rethinking the faster R-CNN architecture for temporal action localization. In CVPR, pp. 1130-1139, 2018.
Jingwen Chen, Yingwei Pan, Yehao Li, Ting Yao, Hongyang Chao, and Tao Mei. Temporal deformable convolutional encoder-decoder networks for video captioning. In AAAI, pp. 8167- 8174, 2019a.
Jingyuan Chen, Lin Ma, Xinpeng Chen, Zequn Jie, and Jiebo Luo. Localizing natural language in 2019b.
videos. In AAAI, pp.
Shaoxiang Chen and Yu-Gang Jiang. Motion guided spatial attention for video captioning. In AAAI, pp. 8191-8198, 2019.
Marcella Coria,, Mattco Stefanini, Lorenzo Baraldi, and Rita Cucchiara. Meshed former for image captioning. In CVPR, pp. 10575-10584, 2020.
Meshed-memory memory trans-
Xiyang Dai, Bharat Singh, Guyue Zhang, Larry S. Davis, and Yan Qiu Chen. Temporal context network for activity localization in videos. In ICCV, pp. 5727-5736, 2017.
Victor Escorcia, Fabian Caba Heilbron, Juan Carlos Niebles, and Bernard Ghanem. Daps: Deep action proposals for action understanding. In ECCV, pp. 768-784, 2016.
Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. Convolutional two-stream network fusion for video action recognition. In CVPR, pp. 1933-1941,22 2016.
Jialin Gao, Zhixiang Shi, Jiani Li, Guanshuo Wang, Yufeng Yuan, Shiming Ge, and Xi Zhou. Accurate temporal action proposal generation with relation-aware pyramid network. In AAAI, 2020.
Jiyang Gao, Zhenheng Yang, Chen Sun, Kan Chen, and Ram Nevatia. TURN TAP: temporal unit
regression network for temporal action proposals. In ICCV, pp. 1664--6556 2017.
Jiyang Gao, Kan Chen, and Ram Nevatia. CTAP: complementary temporal action proposal generation. In ECCV, pp. 70-85, 2018
Zhanning Gao, Le Wang, Qilin Zhang, Zhenxing Niu, Nanning Zheng, and Gang Hua. Video imprint segmentation for temporal action detection in untrimmed videos. AAAI, pp. 8328-8335, 2019.
Rohit Girdhar. João Carreira. Carl Doersch, and Andrew Zisserman. Video action transformer network. In CVPR, pp. 244-253. Computer Vision Foundation / IEEE, 2019.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. CVPR, pp. 961-970, 2015.
Fabian Caba Heilbron, Juan Carlos Niebles, and Bernard Ghanem. Fast temporal activity proposals for efficient detection of human actions in untrimmed videos. In CVPR, pp. 1914-1923, 2016.
Ting-I Hsich, Yi-Chen Lo, Hwann-Tzong Chen, and Tyng- Liu. One-shot object detection with co-attention and co-excitation. In NeurlPS, pp. 2721 2019.
Jie Hu, Li Shen, and Gang Sun. Squeezze-nnd-exciiation networks. In CVPR, pp. 7132-7141, 2018.
Yu-Gang Jiang, Jingen Liu, Amir Roshan Zamir, George Toderici, Ivan Laptev, Mubarak Shah, and Rahul Sukthankar. Thumos challenge: Action recognition with large number of classes, 2014. URL http:///crvv.cf..ccd/HHUUOCCC
Jun Li, Xianglong Liu, Zhuofan Zong, Wanru Zhao, Mingyuan Zhang, and Jingkuan Song. Graph attention based proposal 3d convnets for action detection. 2020.
Chuming Lin, Jian Li, Yabiao Wang, Ying Tai, Donghao Luo, Zhipeng Cui, Chengjie Wang, Jilin Li, Feiyue Huang, and Rongrong Ji. Fast learning of temporal action proposal via dense boundary generator. In AAAI, pp. 11499-11506, 2020.
Tianwei Lin, Xu Zhao, and Zheng Shou. Single shot temporal action detection. In ACM MM, pp. 988-996, 2017a.
10

---

Under review as a conference paper at ICLR 2021
Tianwei Lin, Xu Zhao, and Zheng Shou. Temporal convolution based action proposal: Submission to activitynet 2017. CoRR, abs/1707.06650, 2017b.
Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang, and Ming Yang. BSN: boundary sensitive network for temporal action proposal generation. In ECCV, pp. 3-21, 2018.
Tianwei Lin, Xiao Liu, Xin Li, Errui Ding, and Shilei Wen. BMN: boundary-matching network for temporal action proposal generation. ICCV, 2019.
Qinying Liu and Zilei Wang. Progressive boundary refinement network for temporal action detection. In AAAI, 2020.
Yuan Liu, Lin Ma, Yifeng Zhang, Wei Liu, and Shih-Fu Chang. Multi-granularity generator for temporal action proposal. In CVPR, pp. 3604-3613, 2019.
Fuchen Long, Ting Yao, Zhaofan Qiu, Xinmei Tian, Jiebo Luo, and Tao Mei. Gaussian temporal awareness networks for action localization. In CVPR, pp. 344-353, 2019.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In ICML, volume 80, pp. 4052 4061. PMLR, 2018
Haonan Qiu, Yinghin Zheng, Hao Ye, Yao Lu, Feng Wang, and Liang He. Precise temporal action localization by evolving temporal proposals. In ICMR, pp. 388-396, 2018.
Zhaofan Qiu, Ting Yao, and Tao Mei. Learning spatio-temporal representation with pseudo-3d residual networks. In ICCV, pp. 3534-5542, 2017.
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. In NIPS, pp. 91-99, 2015.
Zheng Shou, Dongang Wang, and Shih-Fu Chang. Temporal action localization in untrimmed videos via multi-stage cnns. In CVPR, pp. 1049-1058, 2016.
Zheng Shou, Jonathan Chan, Alireza Zareian, Kazuyuki Miyazawa, and Shih-Fu Chang. Cdc: Convolutionalldecccovoooiii networks for precise temporal action localization in untrimmed videos. In CVPR, 2017.
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In NIPS, pp. 568-576, 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions, In CVPR, pp. 1-9, 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In CVPR, pp. 2818-2826, 2016.
Yao Ting, Li Yehao, Qiu Zhaofan, Long Fuchen, Pan Yingwei, Li Dong, and Mei Tao. Msr asia msm at activitynet challenge 2017: Trimmed action recognition, temporal action proposals and densecaptioning events in videos. In CVPR ActivityNet Challenge Workshop, 2017.
Du Tran, Lubomir D. Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In ICCV, pp. 4489-4497, 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pp. 5998-6008, 2017.
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In ECCV, pp. 20-36, 2016.
Limin Wang, Yuanjun Xiong, Dahua Lin, and Luc Van Gool. Untrimmednets for weakly supervised action recognition and detection. In CVPR, 2017.

---

Under review as a conference paper at ICLR 2021
Xiaolong Wang. Ross B. Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks, In CVPR, pp. 7794-7803, 2018.
Chao-Yuaa Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krähenbühl, and Ross B. Girshick. Long-term feature banks for detailed video understanding. In CVPR, pp. 284-293, 2019.
Yuanjun Xiong, Limin Wang. Zhe Wang, Bowen Zhang, Hang Song, Wei Li, Dahua Lin, Yu Qiao, Luc Van Gool, and Xiaoou Tang. CUHK & ETHZ & SIAT submission to activitynet challenge 2016. In CVPR ActivityNer Workshop, 2016.
Huijuan Xu, Abir Das, and Kate Saenko. R-C3D: region convolutional 3d network for temporal activity detection. In ICCV, pp.
2017.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, pp. 4048-2057, 2015.
Mengmeng Xu, Chen Zhao, David S. Rojas, Thabet, and Bernard Ghanem. G-tad: Sub-graph localization for temporal action detection. In CVPR, 2020.
Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher J. Pal, Hugo Larochelle, and Aaron C. Courville. Describing videos by exploiting temporal structure. In ICCV, pp. 4507-4515, 2015.
Ting Yao, Tao Mei, and Yong Rui. Highlight detection with pairwise deep ranking for first-person video summarization. In CVPR, pp. 982-990, 2016.
Runhao Zeng, Wenbing Huang, Mingkui Tan, Yu Rong, Peilin Zhao, Junzhou Huang, and Chuang Gan. Graph convolutional networks for temporal action localization. ICCV, 2019.
Tao Zhang, Shan Liu, Thomas H. Li, and Ge Li. Boundary information matters more: Accurate temporal action detection with temporal boundary network. In ICASSP, pp. 1642-1646, 2019.
Peisen Zhao, Lingxi Xie, Chen Ju, Ya Zhang, Yanfeng Wan, and Qi Tian. Bottom-up temporal action localization with mutual regularization. In ECCV, 2020.
Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, and Dahua Lin. Temporal action detection with structured segment networks. In ICCV, pp. 2933-2942, 2017.
12

---

Under review as a conference paper at ICLR 2021
6 APPENDIX
(0016) pre-trained on the Implementation Details. We use the two-stream features (Xiong et al. |
training set of ActivityNet-I.33 with the same parameter settings as (Gao al. 2017 Lin et 2018). We set 16 frames per snippet in ActivityNet-13 and 5 frames per snippet in THUMOS- 14. Further, we sample the snippets with T 100 via linear interpolation ActivityNet- and with T 128 via truncation and overlapped sliding windows in THUMOS-14, All the snippet manipulations are the same as (Lin et al. 2019 2018). For the setting of soft-NMS, we respectively use threshold and 0.65 for ActivityNet-I. and THUMOS- 14, and the same decay parameter 0.85 for both datasets. We train our model using Adam optimizer with batch size 24 ActivityNet-1. and batch size in THUMOS-14 14. The initial learning rate is 10 with decayed by 10 10-3 for every 10 epochs. The channel numbers = 200, C' = 1600, and C" 400. The weighting factor 13) is set to 0.5.
Comparison against SOTA proposal generators Table 4 and Table show the complete marization of our proposal generator, i.e., FITS, against state-of-the-ard proposal generators on ActivityNet-1.33 and THUMOS-I4 14, respectively. both tables, the notation "*" indicates the method using non-two-stream features, and the notation "+" indicates the supplementary results. Our results consistently outperform all other stateoof-the-ae proposal generators on two datasets. Please note that the results of the ActivityNet- 1.3 testing split are evaluated using the official server.
Table 4: Comparison of the state-of-the-art methods on ActivityNet-1.3 validation and testing splits.
Method
+TCN Dai et
2017
+MSRA Ting ot al.
2017
+Prop-SSAD Lin et al. 2017b
+CTAP Gio ct
2018
FITS
100 (val)
73.17
AUC (val)
59.58
63.12
65.72
AUC (test)
61.56
64.18
64.80
70.00
Method
+BSN
(Ln al.
2018
+MGG Liu ct al.
2019
BMN Lin et al.
2019
"GTAN (oog et 2019
FITS
@ 100 (val)
74,16
74.54
75.01
74.80
77.59
AUC
AUC (test)
66.26
66.47
67.19
67.40
70.00
Method
RapNet
Gao
2020
DBG Lin
2020
Zhao
2020
BC-GNN Bai
2020
FITS
@ 100 (val)
76.71
76.65
75.27
76.73
77.59
AUC (val)
67.63
68.23
66.51
68.05
69.36
AUC (lest)
67.72
68.57
70.00
Table 5: Comparison of the state-of-the-art methods on THUMOS-14 14 testing split.
Method
AR 550
AR@ 100
AR@200
AR@500
AR 1000
SCNN-prop Shou et al.
2016
17.22
26.17
37.01
51.57
58.20
+ **STT
BBchh et al.
2017b
19,90
28.36
37.90
51.58
60.27
+TAG
Zhao et al.
2017
18.55
29.00
39.61
+TURN
[Go al.
2017
21.86
31.89
43.02
57.63
64.17
+CTAP
Gao et al.
2018
32.49
42,61
51.97
+BSN
Lin et al.
2018
37.46
46.06
53.21
60.64
64.52
+MGG
Liu al. 2019
39.93
47.75
$4.65
61.36
64.06
GTAN
Long et al. 2019
54.30
BMN
Lin et al.
2019
39.36
47.72
54.70
62.07
65.49
RapNet
(Go al. 2020
40.35
48.23
54.92
61.41
64.47
DBG
Lin et al. 2020
37.32
46.67
54.50
62.21
66.40
Zhao et al.
2020
44,23
50.67
55.74
BC-GNN Bai al.
2020
40.50
49.60
56.33
62.80
66.57
FITS
40.32
49.53
57.55
65.34
69.56
13

---

Under review as a conference paper at ICLR 2021
Table 6: Temporal action detection results ActivityNet- 1.3 validation split. Notation indicates the model leveraging the results from UntrimmedNet-bast action classifier (Wang et al. 2017)
Method
mAP@0.5
mAP@0.75
mAP@0.95
Average
CDC Shou al.
2017
43.83
25.88
0.21
22.77
SSN Zhao et al.
2017
39.12
23.48
5.49
23.98
*BSN Lin et
2018
46.45
29.96
8.02
30.03
*P-GCN Leng
2019
48.26
33.16
3.27
31.11
*BMN Lin ct al.
2019)
50.07
34.78
8.29
33.85
*G-TAD Xu ct al.
2020
50.36
34.60
9.02
34.09
*BC-GNN (Bai 2020
50.56
34.75
9.37
34.26
FITS
51.89
35.05
10.16
34.75
Action detection with FITS action proposals Due to the limited space allowed in the main paper, we provide the experimental results of temporal action detection task in the supplementary material.
The methods (Buch et al. 2017b] Escorcia et al.] (2016] Gao et al., 2018 2020 [017 2019 Lin
al. 2020 2019 2018; Liu et al. helping video action classifier, we first feed FITS proposals into the state-of- lfte-offtheear temporal action classifier, i.e., P-GCN (Zeng et al. 2019 in which the P-GCN classifier infers the action probability distribution of each proposal.
2019; et al. 2019 Shou et al. 2016 Xu et al. 2017 Zhao
2017 have been mentioned in the main paper. For assessing the quality of our proposals for
then re-score each proposal by multiplying its action score, the highest specific-action probability derived from the P-GCN classifier, with the FITS proposal score, the equation (12) in the main paper. Finally, we re-rank the proposals according to their multiplied scores for evaluation.
Thble the detection performance with top-100 proposals in metric mAP@IoUE which evaluates the relation between mean Average Precision (mAP) and specified IoU thresholds, compared to the state-of-the- art methods on ActivityNet-11.3 validation split. Following the methods (Lin et al. 2018 Zeng et al., 2019; Xu et al., 2020), we adopt the video-level action probability distribution derived from the UntrimmedNet-basee action classifier (Wang et al. 2017). We then multiply the action probability our FITS proposal score, i.e., the equation 112) defined in the main paper.
Table T shows the detection performance metric mAP@ IoU, which evaluates the relation between mean Average Precision (mAP) and specified IoU thresholds, compared to the state-of-the-art methods on THUMOS-14 testing split. Note that the original P-GCN method (Zeng et al.) 2011 adopts the proposals generated by BSN Lin et al. (2018). When replacing the BSN proposals of P-GCN with our FITS proposals, our detection result FFITS+P-GNN achieves 5% performance gain compared to P-GCN, 4% performance gain compared to "Zhao al. GCN, and 2.5% performance gain compared to "G-TAD+P-GCN: respectively. The experiment shows the advantage of our proposals improve an action classifier achieving state-of-the- performance while addressing the action detection task.
Lin
PP-GCN implementation: http://giitaatcmmm///ii//eengggn
14

---

Under review as a conference paper at ICLR 2021
Table 7: Temporal action detection results on THUMOS-14 14 testing split.
Method
mAP@0.1
mAP@0.2
mAP@0.3
mAP@0.4
mAP@0.5
mAP@06
mAP@0.7
DAP
Escorcia et al.
2016
13.9
S-CNN
Shou et
2016
47.7
43.5
36.3
28.7
19.0
SSAD
2017a
47.8
43.0
35.0
24.6
SS-TAD
Buch ct
2017a
45.7
29.2
9.6
SST
Buch al.
20
37.8
23.0
CDC
Shou
2017
40.1
29.4
23.3
7.9
TURN
Gao et al.
2017
54.0
50,9
44.1
34.9
25.6
R-C3D
al.
2017
54.5
51.5
44.8
35.6
28.9
SSN
al.
2017
66.0
59.4
51.9
41.0
29.8
ETP
Qiu al
2018
48.2
42.4
34.2
23.4
13.9
CTAP
Gao et al
201
29.9
al.
45.0
TAL-Net
Chao al
201
59.8
57.1
53.2
48.5
42.8
33.8
20.8
MGG
Liu ct al
2019
53.9
46.8
37.4
29.5
21.3
GTAN
Long al.
2019
63.7
47.2
38.8
BMN
Lin et
2019
56.0
47.4
38.8
29.7
20.5
P-GCN
Zong ct al
2019
69.5
67.8
63.6
57.8
49.1
et 2019
56.7
54.7
50.6
43.1
34.3
24.4
14.7
DBG
2020
57.8
49.4
39,8
30.2
21.7
FC-AGCN- P-GGNN---CDD [1. et al.
2020
59.3
59.6
57.1
51.6
38.6
28.9
17.0
PBRNet (Liu Wang
2020
58.5
54.6
51.3
29.5
G-TAD
2020
54.5
47.6
40.2
30.8
23.4
Zhao et al. 2020
53.9
50.7
45,4
38.0
28.5
BC- BC-GNN Bai et
2020
57.1
49.1
40.4
31.2
23.1
G-TAD Xu 2020
+P-GCN
60.4
$1.6
37.6
22.9
Zhao et al. - P--CN
71.8
70.3
66.3
61.0
50.1
FITS+P-GCN
75.0
72.7
69.1
62.7
54.1
41.1
27.8
15


Reviews:
[{"review_id": "oGDECSBRvyv", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposed two modules for temporal action proposal generation in videos: feature integration and group transformers. Both modules rely on self-attention and transformer architectures to improve the feature representation and the scoring component. Experiments on two well-known temporal action datasets shows the potential of the proposed approach.\n\nStrengths:\n\nThe paper is well written and nicely visualized. The novelties of the paper are clearly outlined, while the modules are carefully presented in Section 3 and in Figures 1 and 2. This presentation makes the paper easy to understand and to build upon.\n\nThe experiments are accompanied with multiple ablation studies detailing the effect of different components. The studies show which parts are most effective and how they interact.\n\nWeaknesses:\n\n[Side note] While the paper does directly deal with learning representations, the topic of temporal action proposal generation seems like a much better fit for a computer vision conference. The relevant papers cited are also from CVPR/ICCV/ECCV/etc. While no grounds for a negative rating here, it will likely negatively affect the impact of the paper to not submit it to a computer vision conference.\n\nOn the paper itself, a core limitation is that the innovation of the paper is on the marginal side. The two novelties: feature integration and transformer-driven scorers, consist of existing components added to the temporal action proposal pipeline. The feature integration makes use of attention (through co-attention), akin to the encoder. The transformer-driven are deemed the most innovate part of the paper, although at its core, the transformers mainly attend for representations. Overall, it is interesting to see that attention and transformers can be applied to the problem of temporal action proposal generation, but the insights gained from the approach are modest.\n\nThe experimentation contains a number of limitations to be addressed:\n- The qualitative visualization in its current form provides no insight. First, it is unclear to the reader what the video is, only the id \"pIk9qMEyEd4\" is provided. Furthermore, the graphs are not clear. Why is reducing the two-stream feature discrepancy important for the mysterious video? What is action-instance? Is green the start and yellow the end?\n\n- The reviewer appreciates that the state-of-the-art comparisons include references to ECCV 2020, which took place only a little over a month before the ICLR deadline. The comparison to (Bai et al., 2020) does raise a number of questions. First, it seems that the submission performs worse for low average numbers of proposals, and better for high average numbers of proposals. In general, action proposal generation is not the end goal, but the first step for the task of temporal action detection. The question is: what is more effective for action detection? Better AR@50 or better AR@500? (Bai et al., 2020) include action detection results for their approach and it would be good to have the same for the proposal approach.\n\n- What is missing in the paper is insights into what makes the two modules effective for the task at hand. The ablation studies show that the components help to improve the metric, but why are the modules effective? What problem do they solve? Perhaps success and failure cases can help with understanding the benefit of the modules.\n\nConclusion:\n\nOverall, the paper is well written and clearly presented. The ability to integrate attention and transformers is an interesting but modest addition to the problem of temporal action proposal generation. The experiments contain good ablation studies, but also have a number of open questions. For these reasons, a current rating hovering borderline is given. For the rebuttal, it would be interesting to see discussions regarding the novelty of the approach and the questions raised in the experiments.\n\nOpinion after rebuttal:\n\nOriginally, my rating was a 5. The authors have rebutted my concerns regarding the experimentation, which have gained in insight. The limited novelty and fit to the conference do remain pressing issues and it seems that this is partially shared with the other reviewers. My recommendation is therefore to extend and resubmit.", "extra_scores": {}, "extra_text": {}}, {"review_id": "cj3Fb5tkvmC", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- The paper proposed a feature integration (FI) module and utilized transformers to capture long-range temporal dependencies, which is reasonable and interesting. However, these two modules focus on different aspects for action proposal generation. Is there a connection between these two modules?\n\n- The proposed method was evaluated on two popular benchmarks, and achieved convincing results. Besides this, a comprehensive ablation study is provided, which is helpful to understand the contribution of each part.\n\n- For the transformer, each of them uses the A||M as input for encoder, but F as input for decoder. F is a feature vector computed from A and M, is there any intuition about why A||M are used for encoder and F for decoder?\n\n- More qualitative results and analysis should be provided. For example, the proposed method cannot beat the performance of Zhao et al (2020) on THUMOS-14, and is there any result or analysis could explain this?\n\n- For implementation details, details about transformer, such as number encoder and decoder layers, are not provided, which could make the re-implementation of the paper as a concern.\n\nOverall, it is an interesting paper, and I lean to borderline accept before the rebuttal.", "extra_scores": {}, "extra_text": {}}, {"review_id": "PzAPzZloP4p", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In general, it is an interesting paper to utilize multiple techniques to enhance two-stream features and transformer to improve proposal scores, though all the techniques are not first proposed in this paper. But some technical details are not clearly presented, so the solidarity cannot be evaluated. Furturemore, more ablation study is needed to verify the contribution of the paper. If all the the concerns are properly addressed in the rebuttal, this paper can be accepted. \n\n- Quality: The overall quality is ok but not good enough considering that the idea is interesting and the performance is promising, but the presentation is not quite clear.\n- Clarity: This needs improvement. Please see the following detailed comments. \n- Originality: All techniques used in this paper are not originally proposed, but the ways they are utilized for this specific problem have novelty. \n- Significance: This work has promising performance on action proposals, but its significance can only be evaluated after the detection performance is provided. \n\nDetailed comments:\n1) It is not mentioned how the groundtruth labels for the 8 different probabilities in Eq. (13) are generated, especially for p^c and p^{se}. [Addressed]\n2)  It is not clear how the forward and backward transformers are computed. It is mentioned at Page 5 that the input for backward are reversed. Does it mean that the first snippet becomes the last one in the backward transformer? If so, how can F^{ba} and F^{fb} be directly concatenated considering that their snippets indices do not match?  How does a backward transformer differ from a forward transformer considering that self-attention is not uni-directional? [Addressed]\n3) The feature integration part is claimed to utilize the interaction of the two streams to fuse them with co-attention and mutual-excitation. Does co-attention work better than self-attention on each stream? Does mutual-excitation work bettern than self-excitation on each stream? Ablation experiments should be provided to verify that the two stream interaction is indeed important.  [According to the response, the advantage of co-attention and mutual-excitation seems marginal. That makes the claimed novelty of proposing these modules trivial because attention and self-excitation are existing works. ]\n4) Though the paper claims to solve the problem of action proposal generation, the purpose of generating action proposals is to do action detection. So action detection performance is expected on the two datasets as well.  [Addressed]\n5) Minor issues. \n- In the 'Mutual-excitation' part on Page 4, it says that 1*1 convolutional filter is used on the C*T feature. What does that mean? How could you apply 1*1 convolution on a two-dimensional feature? Does it actually compute anything?  \n- In the 'Aggeration' part, the convolutional kernel for the temporal convolution is written as '1*3', which should be '3' because you only have 1 dimention apart from the feature dimension. \n- In the 'Aggeration' part, the purpose is claimed to take advantage of 'multi-scale temporal contexts'. From Eq. (5), all the 4 features can be directly concatenated, which means that they have the same temporal size. So in Eq. (4), there is no stride for the max-pooling operation. Then why is it multi-scale?\n\n--------------------------------\nAfter rebuttal:\nThough the paper is interesting in showing improved proposal and detection performance by using attention, excitation, and transformers, the novelty is not significant, especially when it is shown that the claimed cross-modality attention and excitation do not help much. The other reviewers also show the same concerns. I incline to degrade my score to 5. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "DDt6tNPAed4", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles the problem of temporal action proposal generation (TAPG). The authors address the problem from two perspectives: features wise and score fusion wise. They use non-local blocks to integrate appearance features and motion features together. For score fusion, they propose transformer based module to incorporate long range temporal information. The proposed method is evaluated on two benchmark datasets and achieved state-of-the art performance.\n\nStrength:\n+ The paper is well written and easy to follow.\n+ The analysis and ablation studies are thorough and adequate.\n\nWeakness:\n+ The novelty of the propose method is incremental, specifically: \n(1). The Feature Integration module is standard  non-local blocks which have been used in many video related applications. Such feature integration module does not specific designed for action proposal generation. Even though the final performance benefits from the feature integration module, it's because of the improvement of feature representations. If you replace the two stream feature with more advanced video features, the performance will likely improve. \n(2). Compared to previous scorer, the Transformer-driven Scorer uses the transformer backbone and outputs four additional scores for actioness and background. The difference between these scorers are not significant. \n\n+ State-of-the-art performance is not necessary. These two papers that seems to have higher numbers though:\n[a] Accurate Temporal Action Proposal Generation with Relation-Aware Pyramid Network, AAAI 2020.\n[b] Sequence-to-Segments Networks for Detecting Segments in Videos, PAMI 2019.\n\nQuestions:\n1. Some details regarding how to train P^{fa}, P^{ba}, P^{fb}, P^{bb} are missing. What is the ground truth for these scores? My understanding is that the ground truth for P^{fa} and P^{ba} are mostly the same except for the boundary?\n2. Why separate transformer modules are needed for backward and forward? This increased a lot of extra parameters. Unlike RNNs, either feeding the stream forward or backward doesn't matter for transformers. Why not share the same transformer for both forward and backward? Only changes the final FC layers for backward and forward?\n3. The conclusion from Table 3 top part is confusing. Why using integrated feature F for both encoders and decoders does not perform the best? \n\n\nSummary:\nMy major concern is about novelty. Although it has good performance, this paper seems kinda tricky and engineering.\n\n\n---------\nUpdate after rebuttal:\n\nThe updated paper addressed my concerns regarding missing details. I appreciated the efforts of comparing with more sophisticated backbone features. However, as pointed by other reviewers, the novelty of the paper is marginal. I keep my original rating as 5.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper focuses on the task of finding higher fidelity action proposals for temporal action proposal detection. As the reviewers mentioned, this task is a pre-task to temporal activity localization/detection in video, which is the main task to be solved. The paper may be perceived differently if it were presented as a detection method instead. Apart from the scope of the paper, the reviewers also unanimously agree on the limited technical novelty of the proposed methodology in the paper. The proposed method can be seen as an application of self-attention and transformer techniques on the problem of activity detection. The goal of these techniques is feature enrichment that serves to incorporate information across long-term context, a concept that has appeared previously in other work but not necessarily with the same machinery (e.g. G-TAD). 

Despite its shortcomings and since it presents promising experimental results on well-known proposal/detection benchmarks, the authors can benefit from considering the reviewers' comments and suggestions to produce a stronger and more compelling future submission.
Decision: Reject