Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors proposed to approach the problem of spatiotemporal video grounding by a one stage end-to-end framework. The model adopts an encoder-decoder strategy and generates the video tube directly from output. Comparisons are made against other approaches to demonstrate the performance of the proposed approach.
strengths_and_weaknesses: The paper is generally well written, the model architecture is intuitive and the results are reasonably well.
questions: 1) In terms of the model architecture contributions, if I understood correctly, the analogy is to add the decoder part similar to the one in traditional transformer on top of the encoder in ViT-based models and use it across different modalities, right? If that's the case, could you elaborate what specific modifications need to be done that can impact the performance of the model?

2) For results in Table 1, when compared with OMRN, the other metrics are clearly outperforming, but it's not the case for m_tIoU. It was mentioned that "OMRN leverages an off-the-shelf detector", so can you explain why the detector mainly helps with one specific metric, or does this indicate that STCAT and OMRN are actually performing differently on different types of test data? If so, some qualitative comparisons might be helpful for error analysis and intuitive understanding.

3) For the generated tube from the decoder, is there any kind of constraint applied so that the output is more consistent across frames, or is it simply enforced by attention? In other words, for other frame-based models, if some simple heuristic constraints can be added to output frames, would that also solve the inconsistency issue, at least to some extent?
limitations: The authors addressed the limitations in Supplementary Section 5.2.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents an end-to-end one-stage Transformer-based framework for spatio-temporal video grounding (STVG). In particular, the authors introduced a novel multi-modal template to alleviate the alignment inconsistency and prediction inconsistency issues in the task above. The model followed an encoder-decoder design paradigm without any pre-trained object detectors, which reached resonable performance on both VidSTG and HC-STVG datasets.
strengths_and_weaknesses: **Strengths**
- The main contribution is the semantic-consistency flow pattern between the encoder and decoder for the proposed Spatio-Temporal Consistency-Aware Transformer (STCAT) architecture, which mainly consists of an encoder, a template generator, and a corresponding decoder with couples of prediction heads.
- The encoder utilized two specifically-designed interaction layers (spatial interaction layer and temporal interaction layer) to enhance the visual-linguistic feature fusion and the semantic consistency intra- and inter-frames. The template generator produced two terms for both global query representation and frame-level prediction constraint, which benefitted the decoder for spatial-temporal grounding.
- STCAT sets new state-of-the-art performance on the two aforementioned datasets under most evaluation metrics.
- The paper is well-written and easy to follow. The ablation study is also reasonable.

**Weaknesses**
- I am confused about the selection of ResNet instead of ViT as the visual feature extractor. ViT can naturally provide well-organized and adequately-interacted tokens for each frame, which can promote the feature quality for encoder layers. So why not use a transformer-fashion backbone for visual features?
- The naming of the “Spatial Interaction Layer” seems a bit imprecise to me. It seems this module is mainly designed for visual and linguistic semantic fusion instead of boosting spatial information interaction in each frame. If the spatial information of visual tokens needs to be enriched before inter-frame interaction, I sincerely suggest you consider ViT-fashion models.
- Detailed information about the input for the cross-attention layer in each decoder block might need to be supplied.
- The usage of certain loss functions also seems unreasonable. What’s the aim of combining $L_1$loss with GIoU loss for bbox prediction? If the $L_1$loss is complementary to the GIoU loss for eliminating certain special cases when supervising via IoU,  the CIoU or DIoU might be a better metric than the combination, which also avoids tolerating the slow converge speed of GIoU.
- The comparison between STCAT with other one-stage attention-based models seems insufficient. For example, what’s the throughput and parameter amount for STCAT to reach better performance than TubeDETR and STVGBert?
- Could the authors provide an efficiency analysis such as FLOPs, latency and running time on the proposed method and the baselines to provide prove the effectiveness of the proposed method.
questions: Please see the *Strengths And Weaknesses* section.
limitations: Yes,  the authors adequately addressed the limitations.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces a new algorithm, STCAT, to solve the Spatio-Temporal video grounding problem. The new approach is based on the DETR implementation and achieves state-of-the-art performance. By considering the video's global information, it is expected to solve the problem of Prediction Inconsistency, i.e., prediction is performed only at the frame-level, ignoring the consistency between frames.
strengths_and_weaknesses: Strengths:
1. The newly proposed method achieves state-of-the-art performance. 
2. The article is very clearly written and easy to understand.
Weaknesses:
1. The ablation study is insufficient, and we do not know the effect of the prediction head.
2. The method is constructed on the top of DETR, so the paper contributes less to the network structure.
questions: 1. I find that there is no explicit loss to enforce the feature consistency, I wonder how your designed module can make the feature “consistency”. I think more visualization and analyze deserve to be done for this part. 
2. In the ablation study, the effect of the temporal interaction layer section, what is the global embedding pg if the model is trained without the temporal interaction layer.
limitations: None
review_title: null

Meta-review: metareview: This paper presents a novel end-to-end framework for Spatio-Temporal video grounding where the feature alignment and prediction inconsistency can be jointly addressed. All of our reviewers believe this paper is well-presented with a novel idea and SOTA results. Overall, I would like to recommend this paper to be accepted with a poster presentation.
confidence: Certain
Decision: Accept