Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper focuses on the complex multi-label temporal action detection that aims to localize all action instances from a multi-label untrimmed video. Existing query-based action detectors employ a segment to represent an action instance, which is insufficient to handle the concurrent instances and their richer relations. To mitigate this issue, this paper introduces a small set of learnable query points to represent important frames of each action instance. PointTAD provides a flexible mechanism to localize the discriminative frames at boundaries and as well the important frames inside the action.
strengths_and_weaknesses: Strengths: This paper is well written and easy to understand. 

Weaknesses: 
1. The idea of using points to represent keyframes or objects is not very new, as it was already mentioned in [1,2,3,4]. The authors should add a subsection to the related work section summarizing the current related work and explaining the differences from these approaches.
2. From Table 1, the method proposed in this paper does not have a significant performance improvement over the SOTA methods such as MS-TCT.
3. More related methods such as [1,2,3,4] should be compared and discussed in the experimental part.

[1]Guan, Genliang, Zhiyong Wang, Shiyang Lu, Jeremiah Da Deng, and David Dagan Feng. "Keypoint-based keyframe selection." IEEE Transactions on circuits and systems for video technology 23, no. 4 (2012): 729-734.
[2]Zhou, Xingyi, Vladlen Koltun, and Philipp Krähenbühl. "Tracking objects as points." In European Conference on Computer Vision, pp. 474-490. Springer, Cham, 2020.
[3]Tang, Hao, Hong Liu, Wei Xiao, and Nicu Sebe. "Fast and robust dynamic hand gesture recognition via key frames extraction and feature fusion." Neurocomputing 331 (2019): 424-433.
[4]Li, Yixuan, Zixu Wang, Limin Wang, and Gangshan Wu. "Actions as moving points." In European Conference on Computer Vision, pp. 68-84. Springer, Cham, 2020.
questions: See Weaknesses.
limitations: The author did not provide the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: 
This paper introduces PointTAD, an architecture for temporally detecting activities in videos that contain multiple co-occurring activities of different labels. 
The key idea is to use a sequence of query vectors, where each vector aims to predict an activity instance (or background) (similar to DETR). A key contribution of this work is that it associates a set of learnable points to each query vector, which points try to cover the duration of the activity instance and are iteratively refined. It also models intra-proposal relationships among the query-points as well as inter-instance relationships among the query vectors. The method is evaluated on MultiTHUMOS and Charades, where it leads to improved performance under the newly proposed detection-mAP metric, and competitive performance under the classic per frame mAP metric.


strengths_and_weaknesses: Strengths
===========
1. This paper addresses an important problem, i.e. temporally detecting activities in videos that contain multiple co-occurring activities of different label. It also introduces a metric for evaluating the segment-level prediction of activity instances (instead of frame-based).
2. Ablations show that the proposed deformable convolution and the mixing strategy improve performance.
3. The method leads to improved detection-mAP on Charades and MultiThumos.
4. The idea of using query points alongside the query vectors is interesting and seems to be working nicely for pooling features to describe temporal segments.

Weaknesses
===========
1. Unfair comparison with existing per-frame labeling approaches with the detection-mAP metric: Although the addition of the detection-mAP metric is an important contribution of this work, comparison with prior work under this metric seems not to be fair for the compared approaches, since they do not directly predict segments. This paper post-processes the per-frame action predictions of these works based on thresholding in order to generate segments (from consecutive frames with predictions above a threshold). However, this is a very naive post-processing approach, which is also very sensitive to the choice of threshold (other options would be to detect peaks/blobs in the score time-series, which also involve hyperparams). Therefore, the big improvements of the proposed approach under this metric could be because of the sub-optimal choice of post-processing of the results of competing methods. Under the segmentation-mAP metric, the proposed approach is lagging behind SOTA. (Also, state-of-the-art numbers are missing. Even if other methods use optical flow, results should be reported and discussed).
2. Missing baseline: it seems like a baseline would be to use just query vectors, the Multi-Head Self-Attention head and predict proposals and class. This would be very similar to DETR and improvements over it would motivate the need for query points/for the multi-level interactive module.
3. Missing important ablations: What is the benefit of combining sparse predictions with dense scores instead of using just the dense scores (\beta=0)? What is the importance of the scaling parameter $s$ which differentiates the current approach from RepPoints[38]? 
4. Choice of datasets: ActivityNet and THUMOS indeed don’t have many co-occurring activities like Charades/MultiThumos, but they are the standard benchmarks for activity detection methods that predict instances (with start/end times) instead of per-frame activity predictions. Evaluating on either one of them would allow comparison with stronger methods for detection (instead of applying a naive post-processing step on frame labeling methods as done now on Charades/MultiThumos).
5. Relation with existing modules is not explicitly described: The query points module adapts RepPoints[8] to the activity detection task,  the Point-level Locality Preservation uses ideas from deformable DETR, etc.
6. Some equations/statements need clarification: For example, a) how does the query vector predict $N_s$ offsets (ln 140?), b) $\mathbf{q} is defined to be NqxD in ln 119, but in later equations it seems to refer to one of the N_q vectors., c) The Linear() functions in eq 3, 4, 6, 7 could be explained in more detail.

questions: Questions for rebuttal
==========
1. How was the threshold chosen for the post-processing of other methods’ results in Table 1? 
2. Ideally it would be good to evaluate the method on Thumos or ActivityNet or explain why this is not possible.
3. Adding ablations for $beta$ and $s$, as well as a DETR-based baseline would strengthen the paper.

Suggestions
1. What is the motivation of 4 deformable sub-points? (instead of 2 etc)
2. State-of-the-art table is missing a lot of methods, I would suggest taking a look at the tables from [Nagwal, Activity Graph Transformer for Temporal Action Localization, arxiv21] and [Dai, CTRN: Class Temporal Relational Network for Action Detection, BMVC 2021].
3. How did you obtain the segments in the ablation study (Table 2a)? Is it from the query points (partial min-max)?
4. How did you choose the hyperparameters, e.g., for the loss weighting?

limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a learnable query points-based method for multi-label temporal action detection, a more challenging and realistic task compared to single-level temporal action detection, aka. temporal action localization. Instead of uniform sampling on pre-defined segments, PointTAD uses a set of learnable query points to indicate the frames to attend to for each segments. Further, it applies frame mixing and channel mixing on the segment-level feature to integrate instance-level semantics.  The entire model takes RGB frames only as input and achieves significant improvement on detection-level mAP on MultiTHUMOS and Charades datasets. 
strengths_and_weaknesses: *** Strengths

+ Using learnable query points to select representative frames for segment-level video representation is a novel idea, and proves to be more effective than uniform sampling +  Temporal RoIAlign (Table 2a). This might potentially bring some insights to the broader community of general video understanding since the video clips/segments are non-uniform by nature, especially in untrimmed videos.

+ The method achieves significant improvement on detection-level mAP on two datasets. Particularly, the detection mAP is more than doubled on Charades.

+ The paper conducts extensive ablations on the design choices. Most of improvements can be clearly explained by (1) query point-based representation v.s. segment-based representation and (2) the parallel application of frame mixing and channel mixing.


*** Weaknesses

- Some technical details are unclear. Some questions may appear in the next section.
  - How doe "partial Min-Max" select the subset query points?
  - What's the choice of the query numbers?
  - Eq (9) and Eq (10) look similar. Do you first do label assignment (Eq. 9) off the shelf and then optimize the network based on Eq. 10, or do you do them simultaneously?

- In L271-272, the authors argue that "*NoHuman* class is not a well-defined action category that has paired action boundaries". You can also report the detection-mAP by excluding NoHuman class to show it quantitatively.

- From the introduction, "the action decoder accomplishes context modeling at **point-wise**, **intra-proposal** and **inter-proposal** levels". How is the inter-proposal level modeled? The Multi-level Interactive Module is operated within individual proposal, isn't it? Any further operation on top?
questions: 1. In L153-154, "*partial Min-Max* function is to select a subset of query points". How is the subset of local points determined?


==== Post-rebuttal Revision
My questions have been well addressed by the authors. My previous concerns are mostly on the presentation and I've lifted the quality of the presentation from "2 fair" to "3 good" and the overall rating from "6 weak accept" to "7 accept".
limitations: The authors haven't discussed the limitations and potential negative societal impact of their work.
Some potential one might include:
+ High computation cost of training video models end-to-end.
+ Potential biases in existing video datasets.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper identifies the impractical setup of classic single-label TAD and solves a more complex problem of multi-label TAD. It presents a novel query-based action detector with action point representation and multi-level interactive module to handle the co-occurring actions and fine-grained discrimination between categories. Extensive experiments on MultiTHUMOS and Charades demonstrate the effectiveness of the proposed method under detection metrics.
strengths_and_weaknesses: Strengths:  
+ This paper is the first to discuss the reason behind wide usage of video-level classifiers in traditional TAD, which attributes to the label-deficient traditional TAD benchmarks. It provides the first instance detection baseline for the more complex multi-label TAD with detection metrics and is very likely to encourage future TAD works to validate on these challenging multi-label benchmarks.
+ The proposed method tackles concurrent instances and fine-grained classification by introducing query points to replace action segments, for flexible capture of boundary and semantic information at the same time. To my knowledge, PointTAD is the first to integrate point detector with query-based detector in the field of temporal action detection. 
+ The proposed method improves action decoding by designing a comprehensive local-to-global temporal modeling module at pointwise, intra-proposal (Instance-level) and inter-proposal (MHSA).
+ Experimental results under detection-mAP are good and surpass previous single-label and multi-label methods. 
+ Ablations are solid. The performance improvement by newly designed modules, i.e., query points and multi-level interactive module are verified in ablations. Other important parameter choices are also discussed in ablations.
+ Detailed visualizations (fig. 5, fig. A) demonstrate the ability of query points to capture different essential motion cues for fine-grained actions. 
+ The paper is well written and easy to follow, with motivation highlighted and major contributions well organized. The difference of this work and other query-based detectors is clearly addressed in the related work. 
 
Weaknesses：
- The segmentation-mAP is weaker than previous SOTA. However, it’s understandable as instance detectors commonly get outperformed by segmentation methods under segmentation metrics. I would suggest the authors to try using sparse detections to refine SOTA segmentation results to improve segmentation-mAP (could use the practice in [M.1] but reversed).
- Do the Nq action queries share the same set of query points, or each query has its own query points?
- Qualitative results are suggested to add the comparison with segment-based variant to show the improvement of query points over action segments. 
- The wording in Line 42 is inaccurate. Segment-based detectors can at least detect some of the ground truth actions, it should be “segment-based action detectors mainly predict two kinds of error predictions”.
- The captions of S1 and S2 in fig1 should specify the basic unit for the timestamps, e.g., second. 

[M.1] "Cdc: Convolutional-de-convolutional networks for precise temporal action localization in untrimmed videos.", Zheng et al., ICCV 2017.

questions: Please refer to the 'Weaknesses' for the detailed questions.
limitations: Yes, discussed in supplements.
review_title: null

Meta-review: metareview: This paper considers the problem of detecting temporal activities in videos which contain multiple co-occurring activities of different labels. It is an important problem that arises in many computer vision tasks. The paper is generally well written. Specifically, using learnable query points to select representative frames for segment-level video representation seems to be a novel idea. The experiment results also show promises of the proposed method. Nevertheless, a number of comments and questions were raised by the reviewers. We thank the authors for responding to them in detail and even revising their paper accordingly, which includes providing more experiment results to support their claims. The authors are recommended to further revise their paper by addressing the remaining comments raised.

confidence: Certain
Decision: Accept