PointTAD: Multi-Label Temporal Action Detection with Learnable Query Points
Authors: Jing Tan
Xiaotong Zhao
Xintian Shi
Bin Kang
Limin Wang
Venue: NeurIPS 2022 Accept (2022)
Keywords: multi-label temporal action detection
keyframe-based detection
query-based detection
temporal action detection

Abstract: Traditional temporal action detection (TAD) usually handles untrimmed videos with small number of action instances from a single label (e.g., ActivityNet, THUMOS). However, this setting might be unrealistic as different classes of actions often co-occur in practice. In this paper, we focus on the task of multi-label temporal action detection that aims to localize all action instances from a multi-label untrimmed video. Multi-label TAD is more challenging as it requires for fine-grained class discrimination within a single video and precise localization of the co-occurring instances. To mitigate this issue, we extend the sparse query-based detection paradigm from the traditional TAD and propose the multi-label TAD framework of PointTAD. Specifically, our PointTAD introduces a small set of learnable query points to represent the important frames of each action instance. This point-based representation provides a flexible mechanism to localize the discriminative frames at boundaries and as well the important frames inside the action. Moreover, we perform the action decoding process with the Multi-level Interactive Module to capture both point-level and instance-level action semantics. Finally, our PointTAD employs an end-to-end trainable framework simply based on RGB input for easy deployment. We evaluate our proposed method on two popular benchmarks and introduce the new metric of detection-mAP for multi-label TAD. Our model outperforms all previous methods by a large margin under the detection-mAP metric, and also achieves promising results under the segmentation-mAP metric.

PointTAD: Multi-Label Temporal Action Detection with Learnable Query Points
Jing Tan Xiaotong Zhao Xintian Shi Bin Kang
Limin Wang 1.3f
State Key Laboratory for Novel Software Technology, Nanjing University
Platform and Content Group (PCG), Tencent
Shanghai Al Lab
jtan@smail. aanimmaalnjjuu.u njju edu. {davidxtzhao, tinaxtshi, binkangg)@aee (tankkanngteenenttccom com, Imwangōnju.cm w.aggnnju...u.... edu. cn.
Abstract
Traditional temporal action detection (TAD) usually handles untrimmed videos with small number of action instances from a single label (e.g., ActivityNet, THU- MOS). However, this setting might be unrealistic different classes of actions often co-occur in practice. In this paper, we focus on the task of multi-label tem- poral action detection that aims localize all action instances from a multi-label untrimmed video. Multi-label TAD is more challenging as it requires for fine- grained class discrimination within single video and precise localization of the co-occuring instances. To mitigate this issue, we extend the sparse query- query-based detection paradigm from the traditional TAD and propose the multi-label TAD framework of PointTAD. Specifically, our PointTAD introduces small set of learnable query points to represent the important frames of each action instance. This point-based representation provides a flexible mechanism to localize the dis- criminative frames boundaries and as well the important frames inside the action. Moreover, perform the action decoding process with the Multi-level Interactive Module to capture both point-level and instance-level action semantics. Finally, our PointTAD employs end-to-end trainable framework simply based on RGB input for easy deployment. We evaluate our proposed method on two popular benchmarks and introduce the new metric of detection maP for multi-label TAD. Our model outperforms all previous methods by a large margin under the detection- mAP metric, and also achieves promising results under the segmentation-mAA metric. Code is available at hhttps//gttubbccom/M///UUUUUUPUO
1
Introduction
With the increasing amount of video resources on the Internet, video understanding becoming one of the most important topics in computer vision. Temporal action detection (TAD) 50 23
21
[Q
19
36 has been formally studied on traditional benchmarks such as THUMOS (55,, Activity Net
[14,
and HACS [48]. However, the task seems impractical because their videos almost contain non- overlapping actions from single category: 85% videos THUMOS are annotated with single action category. As result, most TAD methods 22 21
simply cast this TAD problem into sub-problems action proposal generation and global video classification [40]. In this paper, we shift our playground to the more complex setup of multi-label temporal action detection, which aims to detect all action instances from multi-labeled untrimmed videos. Existing works 9 16
43
5 34
38 g in
this field formulate the problem as dense prediction task and perform multi-labe classification in frame-wise wise manner. Consequently, these methods are weak in localization and fail to provide the instance-level detection results (i.e., the starting time and ending time of each instance). In analogy to image instance segmentation [22), we argue that is necessary to redefine multi-label TAD as
Work is done during internship at Tencent PCG. CCorespponiig author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Groundtruths
39.0s
Run 43.4s
34.6s BodyContact 35.4s
31.55
Jump
36.25
24.05
Sit
39.0s
Predictions
[39.05, 43.85, Walk]
(38.8s, 43.95, Run)
S2 (32.25, 36.3s, Sit)
[32.1s, Jump)
(25.75, 32.45, Run)
124,0s, 25.0s, Sit)
Mistake S1
Mistake
Other Mistakes
Figure 1: Illustration of action predictions by segment- based action detectors in multi-label TAD.
instance-level detection problem rather than frame-wise segmentation task. In this sense, multi-label TAD results not only provide the action labels, but also the exact temporal extent of each instance.
Direct adaptation of action detectors is insufficient to deal with the challenges of concurrent instances and complex action relations multi-label TAD. The convention of extracting action features from action segments 4 31 47 lacks the flexibility of handling both important semantic frames inside the instance as well as discriminative boundary frames. Consider
groundtruth action of "Jump" " in Fig. segment- based action detectors mainly produce two kinds of error predictions, in type S₁ and type S2. S₁ successfully predicts the correct action category with an incomplete segment of action highlights, whereas S2 does better job in locating the boundaries yet get misclassified as "Sir" due to the inclusion of confusing frames. In addition, most action detectors [47 31 25 inadequate in processing sampled frames and classifying fine-grained action classes. They often exploit temporal context modeling at single level and ignore the exploration of channel semantics.
To address the above issues, we present PointTAD, sparse query-based action detector that lever- ages learnable query points to flexibly attend important frames for action instances. Inspired by RepPoints [5), the query points are directly supervised by regression loss. Given specific regression targets, the query points learn to locate at discriminative frames at action boundaries as well as semantic key frames within actions. Hence, concurrent actions of different categories can yield dis- tinctive features through the specific arrangement of query points. Morcover, we improve the action localization and semantics decoding by proposing the Multi-level Interactive Module with dynamic kernels based on query vector. Direct interpolation or pooling at each query point lacks temporal reasoning over consecutive frames. Following deformable DETR [53]. extract point-level features with deformable convolution [6] [52] from a local snippet to capture the temporal cues of action
change important movement. At instance-level, both temporal context and channel semantics captured with frame-wise and channel-wise dynamic mixing [35 to further decode the distinctive features of simultaneous actions.
PoinTTAD streamlines end-to-end TAD with joint optimization of the backbone network and action decoder without any post-processing technique. We validate our model two challenging multi- label TAD benchmarks. Our model achieves the state-of-the the-art detection-mAs MAPP performance and competitive segmentation-mAP performance previous methods with RGB input.
2 Related Work
Multi-label temporal action detection. Multi-label temporal action detection has been studied as a multi-label frame-wise classification problem in the previous literature. Early methods [29 30 paid a lot of attention on modeling the temporal relations between frames with the help of Gaussian filters in temporal dimension. Other works integrated features at different temporal scales with dilated temporal kernels [9] or iterative cnnvolution-atention pairs [8) Recently, attention has shifted beyond temporal modeling. Coarse-Fint -Fne [16] handled different temporal resolutions in the slow-fas fashion and performed blocks at both spatial and class dimension to model class relations at each timestamp. In our proposed method, we view the task as a instance-level detection problem and employ query-based framework with sparse temporal points for accurate action detection. In addition, we study the temporal context at different semantic levels, including inter-propenn proposal, intra-proposal and point-level of modeling.
attention during fusion. MLAD B8 used multi-head

---

Segment-based representation. Following the prevailing practice of bounding boxes 20
13 32
in object detection, existing temporal action detectors incorporated action segments heavily with three kinds of usage: as anchors, as intermediate proposals, and as final predictions. Segments as anchors are explored mainly in anchor-based frameworks. These methods 12X
used 50 49
sliding windows or pre-computed proposals as anchors. Most TAD methods 47 19
use segments as intermediate proposals. Uniform sampling or pooling are commonly used to extract features from these segments. P-GCN [47] applied max-pooling within local segments for proposal features. G-TAD 43 uniformly divided segments into bins and peeaageppooled each bin to obtain proposal features. AFSD [91 proposed boundary pooling in boundary region to refine action feature. Segments as final predictions are employed among all TAD frameworks, because segments generally facilitate the computation of action overlaps and loss functions. Instead, in this paper, we do not need segments as anchors and directly employ learnable query points as intermediate proposals with iterative refinement. The learnable query points represent the important frames within action and action feature is extracted only from these keyframes rather than using Rol pooling.
based representation. Several existing works have used point representations to describe keyframes 12 37 objects [45 [I]. tracks (51), and actions [8] 12 37 tackled keyframe selection by operating greedy algorithm on spatial SIFT keypointss 2 clustering on local extremes of image color/intensity (37. These methods followed a bottom-up strategy to choose keyframes based on local cues. In contrast, Poin(TAD represents action as a set of temporal points (keyframes). We follow RepPoints [45] to handle the important frames of actions with point representations and refine these points by action feature iteratively. Our method directly regresses keyframes from query vectors in top-down manner for more flexible temporal action detection. Note that PointTAD tackles different tasks from RepPoints 45). We also built PointTAD upon a query-based detector, where small set of action queries is employed to sparsely attend the frame sequence for potential actions, resulting in an efficient detection framework.
Temporal context in videos. Context aggregation at different levels of semantics is crucial for temporal action modeling (39] and has been discussed in previous TAD methods. G-TAD 43 treated each snippet input as graph node and applied graph convolution networks to enhance snippet-leved
features with global context. ContextLoc (54] handled action semanties in hierarchy: it updated snippet features with global context, obtained proposal features with frame-wise dynamic modeling within each proposal and modeled the inter- inter-proposal relations with GCNs. Although we considered the same levels of semantic modeling, our method is different from ContextLoc. PoinITAD focuses on aggregating temporal cues at multiple levels, with deformable convolution at point-level as well as frume and channel attentions at intra-proposal level. We also apply multi-head self-attention for inter-propone relation modeling.
PointTAD
We formulate the task of multi-label temporal action detection (TAD) as a set prediction problem. Formally, given a video clip with T consecutive frames, we predict a set of action instances =
{ve, = (ti, th { Ng is the number of learnable queries, t t are the starting and ending timestamp of the n-th detected instance, on is its action category. The groundtruth action set to detect
are denoted vi = (en, (1e fe, (x)) n- where ia fe, are the starting and ending timestamp of the n-th action, é, is the groundtruth action category, Ng is the number of groundtruth actions.
The overall architecture of PoDntTAD is depicted in Fig. 2 PoinTTAD consists of a video encoder and an action decoder. The model takes three inputs for each sample: RGB frame sequence of length
T. set of learnable query points P- = ( and query vectors q E I ND Learnable query points explicitly describe the action locations by positioning themselves around action boundaries and semantic key frames, and the query vectors decode action semanties and locations from the sampled features. In the model, the video encoder extracts video features X E R from RGB frames. The action decoder contains L stacked decoder layers and takes query points P. query vectors q and video features X as input. Each decoder layer contains two parts: 1) the multi-head self- ceff-attention block models the pair-wise relationship of query vectors and establishes inter-proposal modeling for action detection; 2) the Multi-level Interactive Module models the point-level and instance-level semantics with dynamic weights based on query vector. Overall, the action decoder aggregates the temporal context at peint-level. intra-proposal level and inter- proposal level. Finally, we use two
3

---

Class Scorest T
X
Multi-level Interactive Module
Video Encoder
FFN
Offsets:
Video features: T
Proposals:: N₂ x
Query Vectors:
Updated Query Points:
Transform
MHSA
Updated Query Vectors:
N2
FFN
Class: x
REB frame
Action Decoder
Figure 2: Pipeline of PointTAD. It consists of backbone network that extracts video features from consecutive RGB frames and an action decoder of L layers that directly decodes actions from video features. PointTAD enables end-to-end training of backbone and action decoder without any post-processing of predictions.
linear projection heads to decode action labels from query vectors, and transform query points to detection outputs.
3.1 Video Encoder
We use the I3D backbone 3 as the video encoder in our framework. The video encoder is trained end-to-end with the action decoder and optimized by action detection loss bridge the domain gap between action recognition and detection. For easy deployment avoid the usage of optical flow due to its cumbersome preparation procedure. In order to achieve good performance on par with two-stream features by only using RGB input, we follow [44) to remove the temporal pooling at Mixed_ 5c and fuse the features from Mixed_ 5c with features from Mixed 4f as in [24]. As result, the temporal stride of encoded video features is 4. Spatial average pooling is performed to squeeze the spatiotemporal representations from backbone temporal features.
our framework in practice, we
3.2 Learnable Query Points
Segment- action representation (i.e., representing each action instance simply with a starting and ending time) is limited in describing its boundary and content at the same time. To increase the representation flexibility, we present novel point-based representation to automatically learn the positions of action boundary as well as its semantic key frames inside the instance. Specifically, the point- based representation is denoted by P {t for each query, where t, is the temporal location of jth query point, and the point quantity per query is N, and set 21 empirically. We explain the updating strategy and the learning of query points below.
Iterative point refinement. During training, the query points are initially placed the midpoint of the input video clip. Then, they are refined by query vectors through iterations of decoder layers to reach final positions. To be specific, at each decoder layer, the query point offsets are predicted from updated query vector (see Sec. 3.3 by linear projection. We design a self-paced updating strategy with adaptive scaling for each query at each layer to stabilize the training process. At decoder layer I, the query points for one query are represented by p' 1 The N, offsets are denoted ( j,=,, The refinement can be summarized as:
pt+1 = {(t) + Ar/ 0.5)
where s' max((t)))) min((t)))) is the scaling parameter and describes the span of query points at layer I. As a result, the updated step size gets smaller for shorter action, which helps with the localization of short actions. Updated query points from previous layer inputs to the next layer.
Learning query points. The training of query points directly supervised by regression loss at both intermediate and final stages. We follow (4 to transform query points to pseudo segments for regression loss calculation. The resulted pseudo segments participate in the calculation of L1-loss and tloU loss with groundtruth action segments in both label assignment and loss computation.
The transformation function denoted by T of functions: Min-max Ti and Partial min-mas Tz. Min-max is take the minimum and maximum
= (12, (e) We experiment with two kinds

---

Lnner
Query Vector:
Deformable Conv.
N, points
Residual connection
Query Points: N2 X
Linear
Limear
ReLU
W/ params
Frame Mixing
Fatton
chousels
Linear
Liner
LN ReLU
w/ parama
Mixing
Point-Level
Instance-Level
Figure 3: Multi-level Interactive Module aggregates action semanties at point-level and instance-
level with dynamic parameters.
location from the set of query points the starting and ending timestamp of the pseudo segment, Ti P - min( {ty hep)), ((t With Min-max transformation, the query points are strictly bounded within the local segment of the target action instance. Partial min-max function is to select a subset of query points Piocal and perform the min-max function on them to determine a pseudo segment, T2 aggregate information from outside the action. Empirically, we choose partial min-max by default. For each query, we randomly take N, points from the query point set form Plocal.
max ((1) jeProcat It allows several query points
3.3 Multi-level Interactive Module
Apart from the limitation in the segment- segment-based representation, previous temporal action detectors are also insufficient in decoding the sampled frames. These methods seldom consider semantic aggregation from different levels and at multiple aspects. Accordingly, interactive module that consider both local temporal cues at point-level and intra-proposal relation modeling at instance-level, depicted in Fig. To achieve distinct representations for each query, these parameters at both levels are dynamically generated based on the query vector.
present multi-level
Point-level local deformation. Paired with the refined point representation, employ deformable convolutions to extract point features within the local neighborhood. For the jth query point, we predict four temporal offsets fox ) from the point location and corresponding weights {wx) [= The query point at frame 1, acts as the center point and is added with temporal offsets to achieve four deformable sub-points. These sub-points characterize the local neighborhood of center points. The features at sub-points are extracted by bilinear interpolation, then multiplied with weights and combined together get point-level feature x(j).. This process be expressed
x(i) = Σ X(1 + 0k) We-
(2)
k=1
Both the offsets and weights are generated from query vectors q by linear projection,
Linear(q) e N.x1 w Softmax(Linear(g) N2x4
(3)
where the weights are additionally normalized by softmax for each query.
Instance-level adaptive mixing. Faced with the challenge of simultaneous actions, with only temporal modeling, actions with large overlap may result in similar representation and harms the classification. To tackle this problem, we propose adaptive mixing at both frames and channels by using dynamic convolutions. Specifically, the stacked features of query points denoted by E N. xD Given the query vector q, we generate dynamic parameters for frame mixing and channel mixing:
M = Linear(q) E N.N,N
(4)
M Linear(q) R2 DxD'
(5)
M = Linear(g) E I D'DD
(6)
5

---

Frame mixing is carried out with dynamic projection followed by LayerNorm and ReLU activation on N, points to explore intra-pree proposal frame relations:
f MelLU(LaayNNNommm"MMM M3) RDN..
Similar to frame mixing, channel mixing uses two bottle-necked layers of dynamic projection on the channel dimension to enhance action semanties:
Z = ReL ReLLU(Laae (LayerNoo [ReLUG (LLLyyaaooo((MMM
E DNDD
(8)
These two mixed features concatenated along channel and squeezed by linear operations to the size of query vector. The query vector q' at the ( layer is then updated with residual connection:
q q q' Linear CannarCConcat(z3/
Finally, the query point offsets and the action labels are decoded from the updated vector by two linear projection heads to produce detection results at each layer.
3.4 Training and Inference
Label assignment. Similar to all query-based detectors 2
53 II
36
we apply hungarian matcher
on (detected) pseudo segments to search for the optimal permutation (-)) for label assignment. The groundtruth set V on each video clip is extended with no action Ø to the size of Ng- The matching cost is formulated as:
C= Σ an (L)(((PP), Vo(n)) asson tloUU(((P)))) ( - Octss Cn- (
(10)
By minimizing the matching cost, the bipartite matching algorithm finds the optimal permutation
(-)-) that assigns each prediction with target. aL1. Dioun Oels set to 5, 5, respectively.
Loss functions. PoinTTAD is jointly optimized by localization loss and classification loss. We use L loss and iou loss localization loss:
Lioc
Σ
LI, (T(Pn). + (1 (T (o.
(11)
(1)
The cross-entropy loss between query labels and target labels is used as classification loss. In addition, improve the performance under segmentation-mAP we generate dense classification scores S E T by linear projection from video features X. Cross-entropy loss is enforced on S with dense groundtruth $ T'xC" Therefore, the classification loss is composed of the set prediction loss and the dense per-frame loss:
Lets = Leellen + Lce $).
(12)
The overall loss function is formulated follows:
= Atoc Lioe + Adle Lets-
(13)
Ads: Atoe and Asng hyper-parameters that are 10, 5, respectively.
Inference. During inference, our PoinTTAD uses a single linear projection layer followed by LayerNorm and ReLU activation predict class labels from query vectors. As for localization, use pseudo segments transformed from query points as the final predictions. These sparse predictions then evaluated under the detection-mAP metric. Additional dense scores S could be generated calculation. The sparse predictions are filtered with
at video features for the segmentation-mal threshold 7 and processed with Gaussian kernels to approximate dense scores each frame. Then, the approximated scores are added with weight 3 to the predicted dense scores for Segmentation-mAP calculation. The final dense scores are generated as:
N₂
Sinal 3
Σ
Gaussian (v/.) ()-5))-5
(14)
n=1
Note that the sparse predictions adjusted by dense scores only for segmentation-

---

Table I: Comparison with the state of the art on the MuliTHUMMS test set and Charades test set, under detection-mAP (%) and segmentation-mAAP())
MuliTHUMOS
Charades
Methods
Modality
Det-mAP
Seg-mAP
Det-mAP
Seg-mAP
R-C3D
42
RGB
17.6
Super-event
29
RGB
36.4
18.6
TGM 30
RGB
37.2
20.6
TGM 30
RGB+OF
44.3
21.5
PDAN
RGB
17.3/17.1
40.2
23.7
Coarse- Fine
16
RGB
25.1
MLAD
38
RGB
14.2/13.9/
42,2
18.4
MLAD 88
RGB+OF
51.5
23.7
CTRN
RGB
44.0
25.3
CTRN
RGB+OF
51.2
27.8
AGT
RGB+OF
28.6
MS-TCT S
RGB
16.2/16.0
43.1
7.9
25.4
Ours
RGB
21.5/21.4* 23.5/23.4
39.8
11.1
21.0
Ours
RGB
41.2
12.1
22.1
indicates detection results excluding NoHuman class. inndicates results trained with stronger image augmen- tation as (24).
4
Experiments
4.1 Datasets and Setup
Datasets. We conduct experiments on two popular multi-label action detection benchmarks: Mul- tiTHUMOS [46] and Charades 33 MultiTHUMOS is densely labeled dataset extended from THUMOS14 14. It includes 413 sports videos of 65 classes. The average number of distinctive action categories per video MuliTHUUMSS 10.5, compared with 1.1 in THUMOS14. Charades is a large Multi-label TAD dataset that contains 9848 videos of daily in-door activities. The annotations are spread over 157 action classes, with an average of 6.8 instances per video.
Implementation details. With I3D backbone network, extract frames 10 fps for MultiHHU- MOS and 12 fps for Charades. The spatial resolution set 1922 for both datasets. report ablations with only Center _rrpp for training, and report comparison in Tab. with stronger image augmentation following [24). The video sequence is pre- pee-crscessd with sliding window mechanism. To accommodate most of the actions, the window size is set to 256 frames for MultiTHUMOS (99.1 % actions included), and 400 frames for Charades (97.3% actions included). The overlap ratio is 0.75 at training, and 0 inference. Ng is query N₂ is 21. The number of deformable sub-points is set to according to the number of sampling points in TadTR (25) The optimal is 0.01 for both datasets.
48 for both benchmarks. The number of query points per
Appropriate initialization required for backbone, query points and point-level deformable con- volutions for stable training. Following common practice, the 13D backbone are initialized with Kinetics400 [17] pre-trained weights for MultiTHUMOS and Charades pre-trained weights for Cha- rades. Query points are initialized with constant 0.5 in training and with learned weights inference. Other possible initializations are explored in ablations. The linear layer to produce deformable offsets initialized follows: zeroing for weights and [1, 3, for biases. The weights and biases to generate deformable weights are initialized as zero.
We adopt AdamW W as optimizer with le-4 weight decay. The network is trained on server with 8 V100 GPUs. The batch size per GPU for MultiHHUMOS and 2 per GPU for Charades. The learning rate is set to 2e-4 and drops by half at every 10 epochs. Backbone learning rate is additionally multiplied with 0.1 for stable training.
Evaluation metrics. The default evaluation metric for multi-label TAD is segmentation-m.ll which the frame-wise mAP. addition, we extend the detection-mAP metric from traditional TAD to further evaluate the completeness of predicted action instances. The detection-mAP is the instance- wise mAP of action predictions under different tloU thresholds. We report the average mAP as well as mAPs at tloU threshold set (0.2, 0.5, 0.7} for both datasets. The average detection-mAi
7

---

Table 2: PointTAD Ablation experiments on MuliTHUMOS, Default setting is colored gray
(a) Segments Query Points in query-hased action detectors
(b) Initialization of query points training.
(c) Point2Segment Transforma- tion : P +
0.2 0.5 0.7 Avg.
Init.
0.2
0.5 0.7 Avg
T 0.2 0.5 0.7 Avg.
Segment 20.1 9.8 19.4
(0.1) 36.9 22.6 10.0 21.3
Min max 36.5 22.9 10.9 21.6
Point 36.6 22.8 10.6 21.5
Norm(0.5,0k 37.0 21.9 9,4 21.2
Partial
36.6 22.8 10.6 21.5 Min-max
Const.(O.5) 36.6 22.8 10.6 21.5
(d) Number of Query Points N,
(c) Loint-Level: Deformable Con- volution.
(f) Instance-Level. different vari- of mixing strategy.
per query.
N.
0.2
0.5
0.7 Avg
Deform.
Mixing
0.2 0.5 0.7 Avg
0.2
0.5
0.7
Avg
9
36.2
22.3
10.3 21.0
Conv.
Frame Only 35.7 22.3 10.2 20.9
15
36.6
22.8
10.4 21.4
36.6
22.8
10.6
21.5
Channel Only
34,2 9.8 20.1
36.6
22.8
10.6 21.5
35.7
22.1
9.9
20.8
Trame-+Channe, 34.3 21.4 9.2 19.9
27
36.6
22.6
10.6 21.4
30.7 17.6 6.7 17.1
Parallel Mixing 36.6 10.6 21.5
calculated with tloU thresholds set [0.1 0.1 0.9). We argue that detection -erpettooommPPa10 more reasonable for an instance detection task.
4.2 Comparison with the State-of-the-Art Methods
Tab. []] we compare the performance of PointTAD with previous multi-label TAD methods under both segmentation-mA. and detection-mAP. The sparse prediction tuples of PointTAD are converted to dense segmentation scores with Eq. (T4) for segmentation-mAA In order to compare the detection- mAP, reproduce several previous multi-label temporal action localization methods and convert their dense segmentation results to sparse predictions following [46]. The prediction confidence score of an action with / consecutive frames is calculated as:
L
0..1 0.01(Ly-C))* -H
score(C.pI...pp
pi) x exp(
(15)
o
where p; is the probability for class frame i, MC and oc is the mean and standard derivation of action duration of class in the training set.
Our PointTAD surpasses all previous multi- TAD methods by large margin under detection- mAP, indicating our ability predict complete actions is beyond previous dense segmentation models. As for segmentation-mal we achieve encouraging and comparable results to the previous methods on both benchmarks with sparse detection framework.
Spurstruth
Geamerr lanted taeeiie
In Fig. we show the qualitative results of PoinTTAD on the MuliTHUMOS test set compared to the segment- based baseline, PDAN [9] and MS-TCT [8]. PoinTTAD detects more instances at harder categories, such as "Fall" and "SoccerPenalty".. MS-TCT and PDAN perform better at "NoHuman" category. We argue that this is because "NoHuman" " class is not well-defined action category with precise paired action boundaries, whereas PointTAD have to leverage pair relations of boundaries for localization.
Walk
NoHuman
Soccer Fenalty
Run
Figure 4: Qualitative results on the Multi- THUMOS test set.
4.3 Ablation Study
Segment-based representation vs point-based representation. In Tab, 2a we compare the perfor- mance between segment-basse based representation and point-based representation. In the segment-based baseline, actions are represented by segments of paired start-end timestamps. We apply temporal Rol align to the segments as in 41 49 to retrieve action features. Temporal Rol align divides segments into uniform bins and apply average pooling on each bin to transform frame features for query vector mixing, For fair comparison, the number of bins in Rol align is set to N, and the parallel dynamic

---

Table 3:
Ablation study w.r.t. fusion parameter 33 and scaling parameter & on MuliiTHUMOS.
(a) Result fusion parameter B: 1 indicates full sparse detection and 0 indicates full dense results. The results are reported under segmentation-mAAR
(b) Offsee scaling parameter scale to win- dow size vs, scale to action duration. The re- sults are reported under deteectionmAA@uutt
B
0 0.2
0.4
0.6
0.8 0.96 1
0.2
0.5
0.7
Avg.
MuliTHHUMSS 33.0 39.8
39.2
38,1
37.3
36.8
35.9
scale to clip duration
36.4
21.9
9.8
21.0
Charades
13.8
14.3
15.1
16.6
19.2
21.0 18.7
scale to action duration 36.6
22.8
10.6
21.5
mixing is applied to both implementations. Results show that point-based representation significantly outperforms segment-based representation, demonstrating the advantage of adaptive sampling based on temporal points over grid alignment based on segments.
Study on query point initialization. We consider three different initialization for query points in training: uniform distribution in [0, IJ. normal distribution with mean=0.5 and std 0.3, and initialization with constant 0.5. Results Tab. 2b indicates that all three initialization methods are beneficial for the training, with little performance gap in avg- -.A. In addition, normal distribution and uniform distribution achieve higher detection-mAP with lower tloU threshold, which shows these two initialization methods are weaker accurate localization under higher tloU threshold. Empirically, we set constant initialization by default.
Study pseudo segment transformation functions. Tab. 2c shows two alternatives of trans- formation function T. Min-max and Partial min-max achieve similar performance under average detection-mAP. We choose Partial min-max as the default function because it offers a more relaxed constraint for query points with higher flexibility.
Study the number of query points. We ablate with different numbers of query points N, in Tab. 2d In general, the performance is in proportion to the point quantity, although the increase of N, only benefits the performance little. N, reaches the sweet spot 21, where the performance no longer increases with larger N.. To balance the model complexity and performance, we set N, to 21.
Study on point-level deformation. In Tab. 2e, compare point-level deformable convolution and direct interpolation for point-level feature extraction. Using deformable operator achieves higher performance than direct interpolation query point location, demonstrating that it is beneficial to consider local context with dynamic modeling in action detection.
Study instance-level mixing. The effect of Ctame-wise and channel-wise mixing is studied in Tab, 21] We first ablate with the single application of each mixing, i.e. frame-only or channel- only. Results show that the performance degrades more without frame mixing, demonstrating the importance of frame mixing over channel mixing in multi-label TAD. Furthermore, we explore different combinations of frame and channel mixing with two cascaded alternatives, i.e. frame mixing- + channel mixing and channel mixing- + frame mixing. Tab. shows that compared with frame-only and channel-only performances respectively, cascaded designs backfire at the performance with a decrease of 1% avg-mAP at frame - channel and 3% avg-mAP at channel + frame. To tackle action detection video domain, intra-proposal channel mixing does not help with subsequent frame mixing and vice versa.
Study on the result fusion parameter B. Combining sparse detection results with dense segmen- tation scores provides smoother frame-level scores for segmentation-mAN We ablate with choices of 3 on both datasets in Tab. 3a | is set to 0.2 for MultiTHUMOS and 0.96 for Charades based on empirical results.
Study on the offset scaling parameter This scaling parameter is conventional in box- box-based object detectors [32 I 35), which is to scale regression offsets with respect to the box size instead of the image size. We extend this design to our PointTAD. Tab. 3b] we compare the regression offsets predicted with respect to action duration (a.k.a offset scaled by duration) and with respect to clip duration (offset without scaling) on MultiTHUMOS. The result demonstrates the effectiveness of this scaling strategy on based detectors.

---

Groundtrütt
Pants
Gobal Paitls
Layer
Layer
Layer
Layer
00:00 00:02
00:04 00:06 00:08
00:10
00:12
00:14 00:16 00:18 00:20
00:22 00:24
Time
Figure 5: Visualization of learned query points and the corresponding action groundtruth on a sample from MuliTHUMOS..
4.4 Query Points Visualization
Fig. [5] fillustrates the learned query points and the corresponding action target a sample from MultiTHUMOS. Partial mix-max transformation divides the query points into confined local points (blue) and open-ended global points (orange). Local points are learned to attend action boundaries with two extreme points and semantic key frames with interior points. Global points sparsely distribute over the entire video for global context. Specifically, we observe that the interior local points effectively capture the most distinguished characteristic of target action "Run" ", by capturing rapid forward movement of both legs and neglecting frames of similar yet disruptive movements: hopping and scoring. Moreover, the iterative refining process query points are depicted from initial positions the final positions in the last decoder layer. Query points are learned to kick off from the median position in the window. After the first decoder layer, the query points are already able to coarsely locate the action target. Then, these query points automatically converge to finer localization along the decoder layers.
5 Limitations and Future Work
PointTAD is proposed to solve the complex problem of multi-label TAD, by leveraging learnable query points for flexible and distinct action representation. Currently, we validate our model on two popular multi-label TAD benchmarks which include sport events and daily indoor activities. PointTAD achieves superior performance to all previous multi-label TAD methods as well as the state-of-the-art single-label TAD methods under detection-mAP metric. We have not demonstrated our model's ' ability in general action detection in more diverse scenarios, including important tasks such action spotting, sentence grounding, and so on. In the future, would continue explore the advantages of point-based action representation in broader scope for video understanding. Meanwhile, PointTAD training still relies the intermediate supervision and we hope design more effective training paradigm for the query-based detection pipeline.
6 Conclusion
In this paper, have studied the complex multi-label TAD that aims detect all actions from multi-label untrimmed video. We formulate this problem as a sparse detection task and extend the traditional query-based action detection framework from single-label TAD. Faced with the challenge of concurrent instances and complex action relations in multi-label TAD, we introduce a set of learnable query points to effectively capture action boundaries and characterize action semantics for fine-grained action modeling Moreover, to facilitate the decoding process, we propose the Multi- level Interactive Module that integrates action semanties both point level and instance level, by using dynamic kernels based on query vector. Finally, PointTAD yields an end-to-end trainable architecture by using only RGB inputs for easy deployment practice. Our PointTAD surpasses all previous methods by a large margin under the detection- -AP and achieves promising results under the segmentation-min -AA on two popular multi-label TAD benchmarks.
Acknowledgments and Disclosure of Funding
This work is supported by National Natural Science Foundation of China (No. 62076119, No. 61921006), the Fundamental Research Funds for the Central Universities (No. 020214380091), and Collaborative Innovation Center of Novel Software Technology and Industrialization.
10

---

References
[1] Zhaowei Cai and Nuno Vasconcelos, Cascade R- R-CNN: delving into high quality object detection. In CVPR, pages 6154 6162, 2018.
[2] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, pages 213-229, 2020.
[3] João Carreira and Andrew Zisserman. Quo vadis, action recognition? A new model and the kinetics datasct.
In CVPR, pages 4724-4733, 2017.
[4] Yu-Wei Chao, Sudheendra Vjjaaanarasimhan, Bryan Scybold, David A. Ross, Jia Deng, and Rahul Sukthankar. Rethinking the faster R-CNN architecture for temporal action localization. In CVPR, pages
1130-1138 2018
[5] Guo Chen, Yin-Dong Zheng, Limin Wang, and Tong Lu. DCAN: improving temporal action detection via dual context aggregation. In AAAI, pages 248 257, 2022.
[6] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In JCCV, pages 764-773,200 2017.
[7] Rui Dai, Srijan Das, and François Brémond. CTRN: class-temporal relational network for action detection. In BMVC. page 224, 2021.
[8] Rui Dai, Srijan Das, Kumara Kahatapitiya, Michael S. Ryoo, and François Brémond. MS-TCT: multi-scale
temporal convtransformrr for action detection. In CVPR, pages 20009-20011 2022.
[9] Rui Dai, Srijan Das, Luca Minciullo, Lorenzo Garattoni, Gianpiero Francesca, and François Brémond.
PDAN: pyramid dilated attention network for action detection. In WACV, pages 2969-2978, 2021. [10] Jialin Gao, Zhixiang Shi, Guanshuo Wang, Jiani Li, Yufeng Yuan, Shiming Ge, and Zhou. Accurate temporal action proposal generation with relation-aware pyramid network. In AAAI, 2020.
[11] Ziteng Gao, Limin Wang. Bing Han, and Sheng Guo. AdaMixer: A fast-converging query-hased object detector. In CVPR, pages 5374-5373, 2022.
[12] Genliang Guan, Zhiyong Wang. Shiyang Lu, Jeremiah Da Deng. and David Dagan Feng. Keypoint-based
keyframe selection. IEEE Trans. Circuits Syst. Video Technol., 23(4):729 -744. 2013.
[13] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R R-CNN. JEEE Trans. Pattern Anal. Mach. Intell., 02(2):386-397 2020.
[14] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR, pages 961-970, 2015.
[15] Haroon Idrees, Amir Roshan Zamir, Yu- la-G Jiang, Alex Gorban, Ivan Laptev, Rahul Sukthankar, and Mubarak Shah. The THUMOS challenge on action recognition for videos "in the wild". Comput. Vis. Image Underst., 155:: 1-23, 2017.
[16] Kumara Kahatapitiya and Michael S. Ryoo. Coarse-fing fine networks for temporal activity detection in videos. In CVPR, pages 8385 8394, 2021.
[17] Will Kay. João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijyanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset. CoRR, bos/1705,06950 2017.
[18] Yixuan Li, Zixu Wang, Limin Wang, and Gangshan Wu. Actions as moving points. In ECCV, 2020.
[19]
Chuming Lin, Chengming Xu, Donghao Luo, Yabiao Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, and Yanwei Fu. Learning salient boundary feature for anchor-free temporal action localization. In CVPR, pages 3320-3329, 2021.
[20] Tsung-Yi Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie.
Feature pyramid networks for object detection. CVPR, pages 936 944, 2017.
[21] Tianwei Lin. Xiao Liu, Xin Li. Errui Ding, and Shilei Wen. BMN: boundary-matching network for temporal action proposal generation. In ICCV, pages 3888 3897, 2019.
[22] Tsung-Yr Yi Lin, Michael Maire. Serge J. Belongie. James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV, pages 740-755, 2014.
[23] Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang, and Ming Yang. BSN: boundary sensitive network for temporal action proposal generation. In ECCV, pages 3-21, 2018.
[24] Xiaolong Liu. Song Bai, and Xiang Bai. An empirical study of end-to-end temporal action detection. In CVPR, pages 20010-20019, 2022.
[25] Xiaolong Liu. Qimeng Wang. Yao Hu, Xu Tang, Song Bai, and Xiang Bai. End-to-end temporal action detection with transformer. IEEE Transactions on Image Processing. 31:5427-5441. 2022.
[26] Pascal Mettes, Jan C. van Gemert, Spencer Cappallo, Thomas Mensink, and Cees G. M. Snoek. Bag-of- fragments: Selecting and encoding video fragments for event detection and recounting. In ICMR, pages 427 2015.
[27] Megha Nawhal and Greg Mori Activity graph transformer for temporal action localization. CoRR, abs/2101.085400 2021.
[28] Dan Oneata, Jakob Verbeek, and Cordelia Schmid. Action and event recognition with fisher vectors on compact feature set. In ICCV, pages 1817-1824, 2013.

---

[29] A. J. Piergiovanni and Michael S. Ryoo. Learning latent super-events to detect multiple activities in videos.
In CVPR. pages 5304-5313, 2018.
[30] A. J. Piergiovanni and Michael S. Ryoo. Temporal gaussian mixture layer for videos. In JCML, pages
5152-5161, 2019.
[31] Zhiwu Qing, Haisheng Su, Weihao Gan, Dongliang Wang, Wei Wu, Xiang Wang. Yu Qiao. Junjie Yan. Changxin Gao, and Nong Sang. Temporal context aggregation network for temporal action proposal refinement. In CVPR, pages 485-494, 2021.
[32] Shaoqing Ren, Kaiming He. Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. In NIPS, pages 91-99, 2015.
[33] Gunnar A. Sigurdsson, Gül Varol. Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Holly-
wood in homes:
Crowdsourcing data collection for activity understanding. In ECCV, 2016.
[34] Haisheng Su, Weihao Gan, Wei Wu, Yu Qiao, and Junjie Yan. BSN++ complementary boundary regressor with scale-balanced relation modeling for temporal action proposal generation. In AAAI, 2021.
[35] Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan. Changhu Wang. and Ping Luo. Sparse R-CNN: end-to-end object detection with learnable
proposals. In CVPR, pages 14454-14463 2021.
[36] Jing Tan, Jiaqi Tang, Limin Wang, and Gangshan Wu. Relaxed transformer decoders for direct action proposal generation. In ICCV, pages 13506-13515, 2021.
[37] Hao Tang, Hong Liu, Wei Xiao, and Nicu Sebe. Fast and robust dynamic hand gesture recognition via key frames extraction and feature fusion. Neurocompating, 331:424-433, 2019.
[38] Praveen Tirupattur, Kevin Duarte, Yogesh S. Rawat, and Mubarak Shah. Modeling multi-label action
dependencies for temporal action localization. CVPR. pages 1460-1470, 2021.
[39] Limin Wang. Zhan Tong. Bin Ji, and Gangshan Wu. TDN: temporal difference networks for efficient
action recognition. In CVPR, pages 1895-1904, 2021.
[40] Limin Wang, Yuanjun Xiong. Dahua Lin, and Luc Van Gool. UntrimmedNets for weakly supervised action recognition and detection. In CVPR, pages 1402-6411, 2017.
[41] Jiannan Wu, Peize Sun, Shoufa Chen, Jiewen Yang. Zihao Qi, Lan Ma, and Ping Luo. Towards high-quality temporal action detection with sparse proposals. CoRR, 0bs/2209.08844 2021.
[42] Huijuan Xu, Abir Das, and Kate Saenko. R-C3D; region convolutional 3d network for temporal activity detection. ICCV, pages 5794-5803, 2017.
[43] Mengmeng Xu, Chen Zhao, David S. Rojas, Ali K. Thabet, and Bernard Ghanem. G-TAD: sub-graph localization for temporal action detection. In CVPR, pages 10153-10162, 2020.
[44] Min Yang. Guo Chen, Yin-Dong Zheng, Tong Lu, and Limin Wang. BasicTAD: an astounding rgb-only baseline for temporal action detection. CoRR, 1w/2205502217 2022.
[45] Yang, Shaohui Liu, Han Hu, Liwei Wang, and Stephen Lin. Reppoints: Point set representation for
object detection. In ICCV, pages 9656-9665, 2019.
[46] Serena Yeung, Olga Russakovsky, Ning Jin, Mykhaylo Andriluka, Greg Mori, and Li Fei-Fei. Every moment counts: Dense detailed laheling of actions in complex videos. Int. J. Comput. Vis., 126(2-4):375- 389, 2018.
[47] Runhao Zeng, Wenbing Huang, Chuang Gan, Mingkui Tan, Yu Rong. Peilin Zhao, and Junzhou Huang.
Graph convolutional networks for temporal action localization. In ICCV, pages 7093-7102, 2019.
[48] Hang Zhao, Antonio Torralba, Lorenzo Torresani, and Zhicheng Yan. HACS: human action clips and segments dataset for recognition and temporal localization. In ICCV, pages 8667-8677, 2019.
[49] Peisen Zhao, Lingxi Xie, Chen Ju, Ya Zhang, Yanfeng Wang, and Qi Tian. Bottom-up temporal action localization with mutual regularization. In ECCV, pages 539-555, 2020.
[50] Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, and Dahua Lin. Temporal action detection with structured segment networks. Int. J. Comput. Vis., 128(1):74-95, 2020.
[51] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Tracking objects as points. In ECCV, pages 474-490, 2020.
[52] Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. Deformable convnets V2: more deformable, better results. CVPR, pages 13388-9366 2019.
[53] Xizhou Zhu, Woijic Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: deformable transformers for end-to-end object detection. In ICLR, 2021.
[54] Zixin Zhu, Wei Tang, Le Wang, Nanning Zheng, and Gang Hua, Enriching local and global contexts for temporal action localization. In ICCV, pages 14496-1550, 2021.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Sec. 5
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See supplementary material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] No theoretical results.
(b) Did you include complete proofs of all theoretical results? [N/A] No theoretical results.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Sec. 4.1 and the supplementary material.
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] See supplementary material.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Sec. 4.1
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes] See Sec. [.]
(b) Did you mention the license of the assets? [Yes] See the supplementary material.
(c) Did you include any new assets either the supplemental material or as URL? [Yes]
See the supplementary material for details.
(d) Did you discuss whether and how consent
obtained from people whose data you're using/curating? [N/A] The datasets we use are open datasets.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "qwDbSbC0buz", "reviewer": "Reviewer_tdcx", "summary": "\nThis paper introduces PointTAD, an architecture for temporally detecting activities in videos that contain multiple co-occurring activities of different labels. \nThe key idea is to use a sequence of query vectors, where each vector aims to predict an activity instance (or background) (similar to DETR). A key contribution of this work is that it associates a set of learnable points to each query vector, which points try to cover the duration of the activity instance and are iteratively refined. It also models intra-proposal relationships among the query-points as well as inter-instance relationships among the query vectors. The method is evaluated on MultiTHUMOS and Charades, where it leads to improved performance under the newly proposed detection-mAP metric, and competitive performance under the classic per frame mAP metric.\n\n", "questions": "Questions for rebuttal\n==========\n1. How was the threshold chosen for the post-processing of other methods’ results in Table 1? \n2. Ideally it would be good to evaluate the method on Thumos or ActivityNet or explain why this is not possible.\n3. Adding ablations for $beta$ and $s$, as well as a DETR-based baseline would strengthen the paper.\n\nSuggestions\n1. What is the motivation of 4 deformable sub-points? (instead of 2 etc)\n2. State-of-the-art table is missing a lot of methods, I would suggest taking a look at the tables from [Nagwal, Activity Graph Transformer for Temporal Action Localization, arxiv21] and [Dai, CTRN: Class Temporal Relational Network for Action Detection, BMVC 2021].\n3. How did you obtain the segments in the ablation study (Table 2a)? Is it from the query points (partial min-max)?\n4. How did you choose the hyperparameters, e.g., for the loss weighting?\n", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n===========\n1. This paper addresses an important problem, i.e. temporally detecting activities in videos that contain multiple co-occurring activities of different label. It also introduces a metric for evaluating the segment-level prediction of activity instances (instead of frame-based).\n2. Ablations show that the proposed deformable convolution and the mixing strategy improve performance.\n3. The method leads to improved detection-mAP on Charades and MultiThumos.\n4. The idea of using query points alongside the query vectors is interesting and seems to be working nicely for pooling features to describe temporal segments.\n\nWeaknesses\n===========\n1. Unfair comparison with existing per-frame labeling approaches with the detection-mAP metric: Although the addition of the detection-mAP metric is an important contribution of this work, comparison with prior work under this metric seems not to be fair for the compared approaches, since they do not directly predict segments. This paper post-processes the per-frame action predictions of these works based on thresholding in order to generate segments (from consecutive frames with predictions above a threshold). However, this is a very naive post-processing approach, which is also very sensitive to the choice of threshold (other options would be to detect peaks/blobs in the score time-series, which also involve hyperparams). Therefore, the big improvements of the proposed approach under this metric could be because of the sub-optimal choice of post-processing of the results of competing methods. Under the segmentation-mAP metric, the proposed approach is lagging behind SOTA. (Also, state-of-the-art numbers are missing. Even if other methods use optical flow, results should be reported and discussed).\n2. Missing baseline: it seems like a baseline would be to use just query vectors, the Multi-Head Self-Attention head and predict proposals and class. This would be very similar to DETR and improvements over it would motivate the need for query points/for the multi-level interactive module.\n3. Missing important ablations: What is the benefit of combining sparse predictions with dense scores instead of using just the dense scores (\\beta=0)? What is the importance of the scaling parameter $s$ which differentiates the current approach from RepPoints[38]? \n4. Choice of datasets: ActivityNet and THUMOS indeed don’t have many co-occurring activities like Charades/MultiThumos, but they are the standard benchmarks for activity detection methods that predict instances (with start/end times) instead of per-frame activity predictions. Evaluating on either one of them would allow comparison with stronger methods for detection (instead of applying a naive post-processing step on frame labeling methods as done now on Charades/MultiThumos).\n5. Relation with existing modules is not explicitly described: The query points module adapts RepPoints[8] to the activity detection task,  the Point-level Locality Preservation uses ideas from deformable DETR, etc.\n6. Some equations/statements need clarification: For example, a) how does the query vector predict $N_s$ offsets (ln 140?), b) $\\mathbf{q} is defined to be NqxD in ln 119, but in later equations it seems to refer to one of the N_q vectors., c) The Linear() functions in eq 3, 4, 6, 7 could be explained in more detail.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "o8vOZSqT5D", "reviewer": "Reviewer_eH4Z", "summary": "This paper identifies the impractical setup of classic single-label TAD and solves a more complex problem of multi-label TAD. It presents a novel query-based action detector with action point representation and multi-level interactive module to handle the co-occurring actions and fine-grained discrimination between categories. Extensive experiments on MultiTHUMOS and Charades demonstrate the effectiveness of the proposed method under detection metrics.", "questions": "Please refer to the 'Weaknesses' for the detailed questions.", "limitations": "Yes, discussed in supplements.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:  \n+ This paper is the first to discuss the reason behind wide usage of video-level classifiers in traditional TAD, which attributes to the label-deficient traditional TAD benchmarks. It provides the first instance detection baseline for the more complex multi-label TAD with detection metrics and is very likely to encourage future TAD works to validate on these challenging multi-label benchmarks.\n+ The proposed method tackles concurrent instances and fine-grained classification by introducing query points to replace action segments, for flexible capture of boundary and semantic information at the same time. To my knowledge, PointTAD is the first to integrate point detector with query-based detector in the field of temporal action detection. \n+ The proposed method improves action decoding by designing a comprehensive local-to-global temporal modeling module at pointwise, intra-proposal (Instance-level) and inter-proposal (MHSA).\n+ Experimental results under detection-mAP are good and surpass previous single-label and multi-label methods. \n+ Ablations are solid. The performance improvement by newly designed modules, i.e., query points and multi-level interactive module are verified in ablations. Other important parameter choices are also discussed in ablations.\n+ Detailed visualizations (fig. 5, fig. A) demonstrate the ability of query points to capture different essential motion cues for fine-grained actions. \n+ The paper is well written and easy to follow, with motivation highlighted and major contributions well organized. The difference of this work and other query-based detectors is clearly addressed in the related work. \n \nWeaknesses：\n- The segmentation-mAP is weaker than previous SOTA. However, it’s understandable as instance detectors commonly get outperformed by segmentation methods under segmentation metrics. I would suggest the authors to try using sparse detections to refine SOTA segmentation results to improve segmentation-mAP (could use the practice in [M.1] but reversed).\n- Do the Nq action queries share the same set of query points, or each query has its own query points?\n- Qualitative results are suggested to add the comparison with segment-based variant to show the improvement of query points over action segments. \n- The wording in Line 42 is inaccurate. Segment-based detectors can at least detect some of the ground truth actions, it should be “segment-based action detectors mainly predict two kinds of error predictions”.\n- The captions of S1 and S2 in fig1 should specify the basic unit for the timestamps, e.g., second. \n\n[M.1] \"Cdc: Convolutional-de-convolutional networks for precise temporal action localization in untrimmed videos.\", Zheng et al., ICCV 2017.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Zf9ih2zCrer", "reviewer": "Reviewer_QSXY", "summary": "The paper proposes a learnable query points-based method for multi-label temporal action detection, a more challenging and realistic task compared to single-level temporal action detection, aka. temporal action localization. Instead of uniform sampling on pre-defined segments, PointTAD uses a set of learnable query points to indicate the frames to attend to for each segments. Further, it applies frame mixing and channel mixing on the segment-level feature to integrate instance-level semantics.  The entire model takes RGB frames only as input and achieves significant improvement on detection-level mAP on MultiTHUMOS and Charades datasets. ", "questions": "1. In L153-154, \"*partial Min-Max* function is to select a subset of query points\". How is the subset of local points determined?\n\n\n==== Post-rebuttal Revision\nMy questions have been well addressed by the authors. My previous concerns are mostly on the presentation and I've lifted the quality of the presentation from \"2 fair\" to \"3 good\" and the overall rating from \"6 weak accept\" to \"7 accept\".", "limitations": "The authors haven't discussed the limitations and potential negative societal impact of their work.\nSome potential one might include:\n+ High computation cost of training video models end-to-end.\n+ Potential biases in existing video datasets.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*** Strengths\n\n+ Using learnable query points to select representative frames for segment-level video representation is a novel idea, and proves to be more effective than uniform sampling +  Temporal RoIAlign (Table 2a). This might potentially bring some insights to the broader community of general video understanding since the video clips/segments are non-uniform by nature, especially in untrimmed videos.\n\n+ The method achieves significant improvement on detection-level mAP on two datasets. Particularly, the detection mAP is more than doubled on Charades.\n\n+ The paper conducts extensive ablations on the design choices. Most of improvements can be clearly explained by (1) query point-based representation v.s. segment-based representation and (2) the parallel application of frame mixing and channel mixing.\n\n\n*** Weaknesses\n\n- Some technical details are unclear. Some questions may appear in the next section.\n  - How doe \"partial Min-Max\" select the subset query points?\n  - What's the choice of the query numbers?\n  - Eq (9) and Eq (10) look similar. Do you first do label assignment (Eq. 9) off the shelf and then optimize the network based on Eq. 10, or do you do them simultaneously?\n\n- In L271-272, the authors argue that \"*NoHuman* class is not a well-defined action category that has paired action boundaries\". You can also report the detection-mAP by excluding NoHuman class to show it quantitatively.\n\n- From the introduction, \"the action decoder accomplishes context modeling at **point-wise**, **intra-proposal** and **inter-proposal** levels\". How is the inter-proposal level modeled? The Multi-level Interactive Module is operated within individual proposal, isn't it? Any further operation on top?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BzOSwbiteML", "reviewer": "Reviewer_ZWpN", "summary": "This paper focuses on the complex multi-label temporal action detection that aims to localize all action instances from a multi-label untrimmed video. Existing query-based action detectors employ a segment to represent an action instance, which is insufficient to handle the concurrent instances and their richer relations. To mitigate this issue, this paper introduces a small set of learnable query points to represent important frames of each action instance. PointTAD provides a flexible mechanism to localize the discriminative frames at boundaries and as well the important frames inside the action.", "questions": "See Weaknesses.", "limitations": "The author did not provide the limitations and potential negative societal impact of their work.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: This paper is well written and easy to understand. \n\nWeaknesses: \n1. The idea of using points to represent keyframes or objects is not very new, as it was already mentioned in [1,2,3,4]. The authors should add a subsection to the related work section summarizing the current related work and explaining the differences from these approaches.\n2. From Table 1, the method proposed in this paper does not have a significant performance improvement over the SOTA methods such as MS-TCT.\n3. More related methods such as [1,2,3,4] should be compared and discussed in the experimental part.\n\n[1]Guan, Genliang, Zhiyong Wang, Shiyang Lu, Jeremiah Da Deng, and David Dagan Feng. \"Keypoint-based keyframe selection.\" IEEE Transactions on circuits and systems for video technology 23, no. 4 (2012): 729-734.\n[2]Zhou, Xingyi, Vladlen Koltun, and Philipp Krähenbühl. \"Tracking objects as points.\" In European Conference on Computer Vision, pp. 474-490. Springer, Cham, 2020.\n[3]Tang, Hao, Hong Liu, Wei Xiao, and Nicu Sebe. \"Fast and robust dynamic hand gesture recognition via key frames extraction and feature fusion.\" Neurocomputing 331 (2019): 424-433.\n[4]Li, Yixuan, Zixu Wang, Limin Wang, and Gangshan Wu. \"Actions as moving points.\" In European Conference on Computer Vision, pp. 68-84. Springer, Cham, 2020.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept