Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: In general, it is an interesting paper to utilize multiple techniques to enhance two-stream features and transformer to improve proposal scores, though all the techniques are not first proposed in this paper. But some technical details are not clearly presented, so the solidarity cannot be evaluated. Furturemore, more ablation study is needed to verify the contribution of the paper. If all the the concerns are properly addressed in the rebuttal, this paper can be accepted. 

- Quality: The overall quality is ok but not good enough considering that the idea is interesting and the performance is promising, but the presentation is not quite clear.
- Clarity: This needs improvement. Please see the following detailed comments. 
- Originality: All techniques used in this paper are not originally proposed, but the ways they are utilized for this specific problem have novelty. 
- Significance: This work has promising performance on action proposals, but its significance can only be evaluated after the detection performance is provided. 

Detailed comments:
1) It is not mentioned how the groundtruth labels for the 8 different probabilities in Eq. (13) are generated, especially for p^c and p^{se}. [Addressed]
2)  It is not clear how the forward and backward transformers are computed. It is mentioned at Page 5 that the input for backward are reversed. Does it mean that the first snippet becomes the last one in the backward transformer? If so, how can F^{ba} and F^{fb} be directly concatenated considering that their snippets indices do not match?  How does a backward transformer differ from a forward transformer considering that self-attention is not uni-directional? [Addressed]
3) The feature integration part is claimed to utilize the interaction of the two streams to fuse them with co-attention and mutual-excitation. Does co-attention work better than self-attention on each stream? Does mutual-excitation work bettern than self-excitation on each stream? Ablation experiments should be provided to verify that the two stream interaction is indeed important.  [According to the response, the advantage of co-attention and mutual-excitation seems marginal. That makes the claimed novelty of proposing these modules trivial because attention and self-excitation are existing works. ]
4) Though the paper claims to solve the problem of action proposal generation, the purpose of generating action proposals is to do action detection. So action detection performance is expected on the two datasets as well.  [Addressed]
5) Minor issues. 
- In the 'Mutual-excitation' part on Page 4, it says that 1*1 convolutional filter is used on the C*T feature. What does that mean? How could you apply 1*1 convolution on a two-dimensional feature? Does it actually compute anything?  
- In the 'Aggeration' part, the convolutional kernel for the temporal convolution is written as '1*3', which should be '3' because you only have 1 dimention apart from the feature dimension. 
- In the 'Aggeration' part, the purpose is claimed to take advantage of 'multi-scale temporal contexts'. From Eq. (5), all the 4 features can be directly concatenated, which means that they have the same temporal size. So in Eq. (4), there is no stride for the max-pooling operation. Then why is it multi-scale?

--------------------------------
After rebuttal:
Though the paper is interesting in showing improved proposal and detection performance by using attention, excitation, and transformers, the novelty is not significant, especially when it is shown that the claimed cross-modality attention and excitation do not help much. The other reviewers also show the same concerns. I incline to degrade my score to 5. 
review_title: This paper targets two issues in action proposal generation: feature representation and scoring, and provide a feature integration module and transformer-driven scorer respectively. The performance on two datasets is promising. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper tackles the problem of temporal action proposal generation (TAPG). The authors address the problem from two perspectives: features wise and score fusion wise. They use non-local blocks to integrate appearance features and motion features together. For score fusion, they propose transformer based module to incorporate long range temporal information. The proposed method is evaluated on two benchmark datasets and achieved state-of-the art performance.

Strength:
+ The paper is well written and easy to follow.
+ The analysis and ablation studies are thorough and adequate.

Weakness:
+ The novelty of the propose method is incremental, specifically: 
(1). The Feature Integration module is standard  non-local blocks which have been used in many video related applications. Such feature integration module does not specific designed for action proposal generation. Even though the final performance benefits from the feature integration module, it's because of the improvement of feature representations. If you replace the two stream feature with more advanced video features, the performance will likely improve. 
(2). Compared to previous scorer, the Transformer-driven Scorer uses the transformer backbone and outputs four additional scores for actioness and background. The difference between these scorers are not significant. 

+ State-of-the-art performance is not necessary. These two papers that seems to have higher numbers though:
[a] Accurate Temporal Action Proposal Generation with Relation-Aware Pyramid Network, AAAI 2020.
[b] Sequence-to-Segments Networks for Detecting Segments in Videos, PAMI 2019.

Questions:
1. Some details regarding how to train P^{fa}, P^{ba}, P^{fb}, P^{bb} are missing. What is the ground truth for these scores? My understanding is that the ground truth for P^{fa} and P^{ba} are mostly the same except for the boundary?
2. Why separate transformer modules are needed for backward and forward? This increased a lot of extra parameters. Unlike RNNs, either feeding the stream forward or backward doesn't matter for transformers. Why not share the same transformer for both forward and backward? Only changes the final FC layers for backward and forward?
3. The conclusion from Table 3 top part is confusing. Why using integrated feature F for both encoders and decoders does not perform the best? 


Summary:
My major concern is about novelty. Although it has good performance, this paper seems kinda tricky and engineering.


---------
Update after rebuttal:

The updated paper addressed my concerns regarding missing details. I appreciated the efforts of comparing with more sophisticated backbone features. However, as pointed by other reviewers, the novelty of the paper is marginal. I keep my original rating as 5.
review_title: Good motivation but kind of tricky and engineering

final_score: rating: 6: Marginally above acceptance threshold
review_content: - The paper proposed a feature integration (FI) module and utilized transformers to capture long-range temporal dependencies, which is reasonable and interesting. However, these two modules focus on different aspects for action proposal generation. Is there a connection between these two modules?

- The proposed method was evaluated on two popular benchmarks, and achieved convincing results. Besides this, a comprehensive ablation study is provided, which is helpful to understand the contribution of each part.

- For the transformer, each of them uses the A||M as input for encoder, but F as input for decoder. F is a feature vector computed from A and M, is there any intuition about why A||M are used for encoder and F for decoder?

- More qualitative results and analysis should be provided. For example, the proposed method cannot beat the performance of Zhao et al (2020) on THUMOS-14, and is there any result or analysis could explain this?

- For implementation details, details about transformer, such as number encoder and decoder layers, are not provided, which could make the re-implementation of the paper as a concern.

Overall, it is an interesting paper, and I lean to borderline accept before the rebuttal.
review_title: An interesting application paper that models long-term temporal feature for action proposal generation

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:

This paper proposed two modules for temporal action proposal generation in videos: feature integration and group transformers. Both modules rely on self-attention and transformer architectures to improve the feature representation and the scoring component. Experiments on two well-known temporal action datasets shows the potential of the proposed approach.

Strengths:

The paper is well written and nicely visualized. The novelties of the paper are clearly outlined, while the modules are carefully presented in Section 3 and in Figures 1 and 2. This presentation makes the paper easy to understand and to build upon.

The experiments are accompanied with multiple ablation studies detailing the effect of different components. The studies show which parts are most effective and how they interact.

Weaknesses:

[Side note] While the paper does directly deal with learning representations, the topic of temporal action proposal generation seems like a much better fit for a computer vision conference. The relevant papers cited are also from CVPR/ICCV/ECCV/etc. While no grounds for a negative rating here, it will likely negatively affect the impact of the paper to not submit it to a computer vision conference.

On the paper itself, a core limitation is that the innovation of the paper is on the marginal side. The two novelties: feature integration and transformer-driven scorers, consist of existing components added to the temporal action proposal pipeline. The feature integration makes use of attention (through co-attention), akin to the encoder. The transformer-driven are deemed the most innovate part of the paper, although at its core, the transformers mainly attend for representations. Overall, it is interesting to see that attention and transformers can be applied to the problem of temporal action proposal generation, but the insights gained from the approach are modest.

The experimentation contains a number of limitations to be addressed:
- The qualitative visualization in its current form provides no insight. First, it is unclear to the reader what the video is, only the id "pIk9qMEyEd4" is provided. Furthermore, the graphs are not clear. Why is reducing the two-stream feature discrepancy important for the mysterious video? What is action-instance? Is green the start and yellow the end?

- The reviewer appreciates that the state-of-the-art comparisons include references to ECCV 2020, which took place only a little over a month before the ICLR deadline. The comparison to (Bai et al., 2020) does raise a number of questions. First, it seems that the submission performs worse for low average numbers of proposals, and better for high average numbers of proposals. In general, action proposal generation is not the end goal, but the first step for the task of temporal action detection. The question is: what is more effective for action detection? Better AR@50 or better AR@500? (Bai et al., 2020) include action detection results for their approach and it would be good to have the same for the proposal approach.

- What is missing in the paper is insights into what makes the two modules effective for the task at hand. The ablation studies show that the components help to improve the metric, but why are the modules effective? What problem do they solve? Perhaps success and failure cases can help with understanding the benefit of the modules.

Conclusion:

Overall, the paper is well written and clearly presented. The ability to integrate attention and transformers is an interesting but modest addition to the problem of temporal action proposal generation. The experiments contain good ablation studies, but also have a number of open questions. For these reasons, a current rating hovering borderline is given. For the rebuttal, it would be interesting to see discussions regarding the novelty of the approach and the questions raised in the experiments.

Opinion after rebuttal:

Originally, my rating was a 5. The authors have rebutted my concerns regarding the experimentation, which have gained in insight. The limited novelty and fit to the conference do remain pressing issues and it seems that this is partially shared with the other reviewers. My recommendation is therefore to extend and resubmit.
review_title: Review

Meta-review: The paper focuses on the task of finding higher fidelity action proposals for temporal action proposal detection. As the reviewers mentioned, this task is a pre-task to temporal activity localization/detection in video, which is the main task to be solved. The paper may be perceived differently if it were presented as a detection method instead. Apart from the scope of the paper, the reviewers also unanimously agree on the limited technical novelty of the proposed methodology in the paper. The proposed method can be seen as an application of self-attention and transformer techniques on the problem of activity detection. The goal of these techniques is feature enrichment that serves to incorporate information across long-term context, a concept that has appeared previously in other work but not necessarily with the same machinery (e.g. G-TAD). 

Despite its shortcomings and since it presents promising experimental results on well-known proposal/detection benchmarks, the authors can benefit from considering the reviewers' comments and suggestions to produce a stronger and more compelling future submission.
Decision: Reject