Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies the problem of multi-modal video grounding. The authors exploit multiple modalities including RGB, depth, optical flow to extract complementary information for improving video grouding. A dynamic fusion scheme with transformer is proposed to better learn the interaction and integration of multiple modalities. Moreover, a self-supervised intrao-modal module is proposed to obtain better feature representation. The extensive experiments support that the proposed method achieves new state-of-the-art performance on multiple bechmark datasets.
main_review: Strengths:
1. Satisfactory paper writing. Clear technical presentation.
2. Good performances and thorough empirical studies.

Weaknesses:
1. My main concern is the novelty. All the proposed modules look natural but kind of straighforward to me. They indeed contribute to the performance, but there is no inspiring techniques or insightful technical conclusion. For example, co-attentional transformer layers are adopted from [17] to model the interaction among different modalities; contrastive learning is adopted to model the intra-modal feature learning. 

Questions:
1. Why actions like "sitting in a bed" require depth information? Can they be recognized by RGB information?
2. In the intra-modal feature learning, given only the sentence query, how to judge whether two videos are from the same action category?

Rebuttal summary:
1. I agree with other reviewers that depth information is interesting and properly leveraged for the multi-modality task. From non-technical aspect, it is a novel idea.
2. The technical novelty might be limited but acceptable. I increase my rating to "Marginally above acceptence threshold".
limitations_and_societal_impact: No limitations provided. No more suggestions.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Paper addresses the problem of text-guided video temporal grounding, which aims to localize the starting and ending time of a segment corresponding to a text query. The key contribution is integration of three modalities of data in this context: video, motion (flow), and depth; along with the textural query. To integrate the three modalities, paper proposes a dynamic fusion scheme with transformers, which takes the form of co-attention (similar to ViLBERT). Further, to improve the performance within each modality contrastive learning is applied to enhance the feature discriminability. In this contrastive learning formulation positive pairs come from instance of the same action class, while negative pairings are formed from videos coming from two different action classes. Competitive (state-of-the-art) performance is illustrated on Charades-STA and ActivityNet Captions benchmark datasets.
main_review: The paper is well written and the approach is intuitive and easy to understand and follow. Performance is also competitive and improves on state-of-the-art.  The technical novelty is somewhat increment, with components effectively borrowed from other recent works. That being said the choices are well motivated and work well together for the task at hand. Overall, I feel the novelty is sufficient for a poster publication in NeurIPS. 

A few minor comments

- The supervised loss (L_{and}) is not defined. It should be defined in the paper, if for no other reason than completeness.
- The extraction of video segment features (Lines 188-190) is unclear and should be clarified. 
- In addition, one of the more surprising aspects of the paper, for me, is that depth was helpful for the task. I am not aware of any other works that use estimated depth for video tasks. As such, I would be interested in seeing more analysis on the depth modality. For example, how well would depth features work on their own (as one stream DRFT)? How well would they perform in combination with RGB (as a two stream DRFT)? etc. 
limitations_and_societal_impact: Yes. I do not believe there are deep societal impacts to discuss for this work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper addresses text-guided video grounding, which identifies the time interval of an event according to the query text. In contrast to prior work that only leverages RGB information, the authors propose DRFT to combine RGB, depth, and flow maps to boost the performance. To facilitate representation learning, the authors adopt a co-attentional transformer to fuse multi-modal information and contrastive learning to enhance the features across different videos. Experimental results shows that the proposed DRFT outperforms state-of-the-art on Charades-STA and ActivityNet.
main_review: Overall, the paper is well-written and easy to follow. Experimental results also justify the effectiveness of the proposed method. However, the authors are expected to justify their design choices in the proposed method rather than just give a combination of existing techniques. It would help further extension if the authors could elaborate more on the current design. More details can be found below.

1.	In DRFT, the authors only consider two kinds of fusion, namely RGB-depth and RGB-flow. Is there any specific reason not to use other combinations, e.g., depth-flow?
2.	The baseline models in Sec. 3.1 do not make sense to me. It seems to me that the most straightforward would be to concatenate different modalities and learn a 1x1 conv to fuse those features. Is there any reason that the authors adopt an average strategy? What’s the gap between the concatenation and other models mentioned in the paper?
3.	I am curious about the necessity of using a co-attentional transformer. There have been a few lightweight attention modules, such as CBAM [1]. Can the authors briefly elaborate on this design choice?
4.	Although the experiments suggest multi-modality does help, it is unclear how they help. Maybe the authors could provide visualization showing what kind of information helps which activities more, by showing the attention weights and the corresponding activities.
5.	The reason why intra-model feature learning works remains unclear to me. It might be a novel point of view, but the authors are encouraged to provide more insights in the paper.

[1] Woo, Sanghyun, et al. "Cbam: Convolutional block attention module." Proceedings of the European conference on computer vision (ECCV). 2018.

limitations_and_societal_impact: The authors are encouraged to discuss potential abuse and misuse.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: A method for the video temporal grounding in the multi-modal domain (RGB, flow, depth). To this end, the transformer-based co-attention (+ adaptive fusion) and contrastive learning techniques are developed.

main_review: The proposed modules are straightforward use of existing techniques (transformers, contrastive learning). Although the co-attention scheme is developed by modifying the self-attention scheme of the transformer, it has already been tried in various multi-modal methods. Comprehensively, the novelty is weak. Also, enough analysis and studies are not contained, and several definitions/descriptions (loss term and compared method in analysis) are unclear. 

Not enough analysis and studies and several unclear definitions/descriptions, although there is room for the amount of pages.
1) In table 1, does the single-stream DRFT means replacing the inter-modal feature learning module with self-attention? If not, the single-stream DRFT is just the method w/o any attention mechanism, and then it is not proper to verify the effect of the proposed co-attention. 

2) Unclear description for REG module. For example, how the start and end timing is obtained if there is multiple start-end in a video?

3) why the definition/formulation of L_grn is not described?

4) in line 205-206, the authors mentioned that their LGI module is similar to LGI model [19]. Then, is the proposed method built on top of [19] by adding the proposed modules?

5) There is no analysis or studies for if the weight of dynamic fusion differs depending on videos or frames. Without showing this point, the reviewer thinks that the gain of dynamic fusion results from merely using more computational cost.

6) Is there no previous multi-modal methods for this task? As the multi-modal baseline, the authors use the simple combination of independently learned uni-modal models. But, the lower performance of the baseline is not surprising at all. To show the effectiveness of the proposed multi-modal fusion, it is required to compare the proposed fusion with other (even basic) intermediate fusion schemes.

limitations_and_societal_impact: The authors don't address limitations
review_title: null

Meta-review: This paper introduces an approach for multi-modal video temporal grounding. The paper received diverging review scores (4, 6, 6, 7),
with the positive reviewer championing the paper. All reviewers agree that the approach is somewhat incremental, but it introduces as one of the first the use of depth information. The rebuttal addressed a significant part of the concerns of the reviewers.

The recommendation is accept as a poster, with the expectation that the authors include the information provided in the rebuttal. 
Decision: Accept (Poster)