Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper focuses on using human movements for action proposal generation. The proposed MAT (end2end Movement to action transformer networks) use a range of low to high level human movements for temporal action proposal generation. 
strength_and_weaknesses: - The method is robust to occlusions 
- the boundary and proposal networks can be trained end2end jointly. 
Weaknesses:
- this idea is heavily based on human body movements, while a lot of actions might include minimal body movements, or the person acting could be heavily covered and therefore the body movement analysis and pose estimation could include a lot of error. 

clarity,_quality,_novelty_and_reproducibility: The paper is clear and understandable. The idea is new and seems to be outperforming SOTA. 
summary_of_the_review: I think the proposed idea has some novelty in it where there are two movement descriptors (atomic and semantic). I would be interested in seeing the computational complexity of this algorithm as well as its performance on newer datasets. 
I would also include comparisons with more SOTA methods and cite them, some examples here: 
- V. Choutas, P. Weinzaepfel, J. Revaud, and C. Schmid. Potion: Pose motion representation for action recognition. In CVPR 2018, 2018.
- C. Gu, C. Sun, S. Vijayanarasimhan, C. Pantofaru, D. A. Ross, G. Toderici, Y. Li, S. Ricco, R. Sukthankar, C. Schmid, et al. Ava: A video dataset of spatio-temporally localized atomic visual actions. arXiv preprint arXiv:1705.08421, 3(4):6, 2017.
- G.Che ́ron,I.Laptev,andC.Schmid.P-cnn:Pose-basedcnn features for action recognition. In Proceedings of the IEEE international conference on computer vision, pages 3218– 3226, 2015.
- Asghari-Esfeden, S., Sznaier, M. and Camps, O., 2020. Dynamic motion representation for human action recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 557-566).
- D. Zhang, G. Guo, D. Huang, and J. Han. Poseflow: A deep motion representation for understanding human behaviors in videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6762–6770, 2018.
- M. Zolfaghari, G. L. Oliveira, N. Sedaghat, and T. Brox. Chained multi-stream networks exploiting pose, motion, and appearance for action classification and detection. In Com- puter Vision (ICCV), 2017 IEEE International Conference on, pages 2923–2932. IEEE, 2017.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Paper proposed a novel approach of using end-to-end Transformer network to compute 8 semantic measures based on Laban Movement Analysis (LMA) of the input video to solve the Temporal Action Proposal Generation problem.

The proposed network shows comparable results against SOTA methods on 3 datasets, THUMBOS14, ActivityNet and PKU-MMD. 
strength_and_weaknesses: Strengths
1. Paper is well-written and has good theorical motivation.
2. The experimental results somewhat support the main claims of the paper: LBM framework is useful for solving the TAPG problem.
3. Sufficient ablation experimental results were done to demonstrate the contributions of the different modules/subnets.

Weaknesses
1. The direct link from LBM to the TAPG is weak. While LBM can be used to describe an action, it is not clear how the measures are directly translated to solving the TAPG problem.
2. While the experimental results are promising, they are quite mixed. The exceptional results for PKU-MMD is somewhat weakened by the fact that only 3 other methods were compared.
clarity,_quality,_novelty_and_reproducibility: 1. Clarity is good, but link between LBM and TAPG is not clearly established.
2. Quality is excellent as the paper attempts to demonstrate the usefulness of an established framework for understanding human's actions, especially dance to solve a challenging CV problem: TAPG.
3. Novelty is medium-high. While the network design is somewhat incremental, taking inspiration from LBM is novel and highly commendable.
4. Reproducibility is high. While paper is quite concise in describing the implementation details, the source code will be provided if accepted by ICLR.
summary_of_the_review: This is a very interesting and somewhat novel paper. It draws from established framework to solve a challenging CV problem. However, the link between the two were not clearly explained and established in the paper. Furthermore, the mixed experimental results also throw the question wide-open if the proposed approach is indeed as useful as the paper is claiming.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors propose a transformer-based method for the temporal action proposal generation task. Compared with existing methods, the main difference lies in the choice of visual features, which are obtained from the Laban Movement Analysis (LMA) method that are originally designed for action analysis. The proposed method achieves comparable results compared to SOTA methods.
strength_and_weaknesses: Strengths:
1.	It is new to see that Laban Movement Analysis (LMA) can be used as a new action descriptor for representing actions and can be exploited to generate action proposals. Such a design is novel in the TAPG task.
2.	The proposed method achieves comparable results compared to SOTA methods.
Weaknesses&Questions:
Though the overall paper is interesting, I have the following concerns:
1.	The technical contribution is not very significant. To my understanding, the transformer-based TAPG method has been studied in previous works such as RTD-Net (ICCV2021). The main difference between RTD-Net and the proposed method lies in the visual features. This paper additionally considers 3D pose and the action features obtained by Laban Movement Analysis (LMA). However, LMA has been proven to be effective in dance action recognition, it is not that surprising to see that the use of additional features could improve the performance. Thus, such a choice may not bring enough insights to the community.
2.	Tied with the above question, if LMA features are indeed useful for representing action, why not use LMA features to replace the action features in other TAPG or TAD methods? How will the performance change when only LMA features are used in the proposed framework?
3.	Could the LMA features (compared with I3D features) make the proposed more generalizable?
4.	From Figure1, it seems that the proposed method could be affected by the occlusion problem since the detector may fail to detect precise pose. Is the proposed method sensitive to the pose detectors? 
5.	From Section 4.1, it seems that the proposed method requires manual annotations for the timestamps.
6.	As for the proposal transformer, the authors claim that they do not need an additional backbone to generate and save boundary scores (compared with RTD-Net). Does this design bring improvements? 
7.	From Table2, it is confusing that the proposed method achieves good scores in terms of AR@AN (with more high-quality proposals) but obtains worse results in terms of mAP (worse detection performance).
clarity,_quality,_novelty_and_reproducibility: Using LMA as a new action descriptor to generate action proposals is novel but the overall technical contribution is not very significant.
summary_of_the_review: To my understanding, the idea of using LMA to represent actions for the TAPG task is new and interesting. However, considering the limited technical contribution (see detailed comments in the Weakness Section), I vote for rejecting this paper for its current shape. One direction that may improve the paper is testing LMA features for other action recognition or detection methods.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the author proposes a new method, which utilizes RGB/Flow/3D poses to compute LMA-based low-to-high-level movement features, and uses Transformer to generate action proposals. The proposed method achieves "state-of-the-art" performance on major benchmark datasets including THUMOS14, ActivityNet and the newly PKU-MMD dataset.
strength_and_weaknesses: This paper proposes a new action proposal generation method based on Laban Movement Analysis. Based on LMA, the author proposes Transformer based methods to generate action proposals.

I think this paper is somehow novel, which includes new action representations. However, I think this paper has following drawbacks:

1. The author misses recent state-of-the-art literature and comparisons. The author claimed that they achieves state-of-the-art performance over three benchmark datasets: THUMOS14, ActivityNet v1.3 and PKU-MMD. However, the most recent literature in the Table is TSP from ICCV 2021. In AAAI/CVPR 2022/recent journals, there are several paper including TadTR [1], E2E-TAD [2], ActionFormer [3] and CPNet [4]. Also, TadTr, E2E-TAD and ActionFormer are Transformer based methods for temporal action localization. Missing discussion with recent state-of-the-art models are a fatal drawback of this paper.

2. The author claims to use Laban Movement Analysis to generate action proposals. However, I'm a little bit confused: LMA needs extra 3D poses (though they directly extract the pose from the original video using LCRNet). Extra information does not bring much performance gain to the model compared to existing baselines. However, existing baselines does not use extra 3D poses, they only use RGB/Flow features. Thus the importance of LMA can be neglected.

1: Liu X, Wang Q, Hu Y, et al. End-to-end temporal action detection with transformer[J]. IEEE Transactions on Image Processing, 2022, 31: 5427-5441.

2: Liu X, Bai S, Bai X. An Empirical Study of End-to-End Temporal Action Detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 20010-20019.

3. Zhang C, Wu J, Li Y. Actionformer: Localizing moments of actions with transformers[J]. arXiv preprint arXiv:2202.07925, 2022.

4: Hsieh H Y, Chen D J, Liu T L. Contextual Proposal Network for Action Localization[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022: 2129-2138.
clarity,_quality,_novelty_and_reproducibility: Please mainly see the weaknesses sections for details, my concern mains lies in related works and the neccessity of LMA.
summary_of_the_review: In general, my review lies in the reject side. Cause the paper does not comparison to SoTA and the LMA related/performance issue, I would give this paper a reject rating.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper was reviewed by four experts in the field and received a mixed score. The main concerns are unconvincing experiments and lack of clarity. The authors did a good job of rebuttal and addressed many of the concerns. However, AC still feels that more work is needed to get it to the best version. AC also agrees that this work can be much stronger with additional experiments. While this paper clearly has merit, the decision is not to recommend acceptance. The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.


justification_for_why_not_higher_score: 1. Unconving experiments
2.  Limited technical contribution 
justification_for_why_not_lower_score: NA
Decision: Reject