Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: In this paper, the authors proposed to pretrain an end-to-end vision-language model especially on long videos. A hierarchical temporal window based attention mechanism is proposed to deal with long temporal dependencies. The effectiveness of the proposed model is validated on extensive empirical evaluations.


strengths_and_weaknesses: Pros:
1. The idea of vision-language modeling on long-form videos is very interesting.
2. The paper is well presented.
3. Some strong empirical results are obtained.

Cons:
1. Important ablations are missing in the current evaluations.
a. The most comparable baseline HD-VILA is unfortunately not based on the same backbone model, which makes it hard to understand the source of the performance gain.
b. The authors entangled number of frames sampled and the temporal length in the current evaluation. What if only short video segments are used but the frames are densely sampled, e.g., using the HD-VILA-100M video segments but sampling 32 frames from each ~13-sec video segment.
2. Important existing benchmarks on long-form video understanding should have been also evaluated, including [41,5] and [r1, r2, r3]. These well-established benchmarks are important as they require even longer-term video inputs, which could be good test-bed to understand the generalization of the proposed model on longer videos, e.g., 3-4 minutes. More justification on the selection of downstream datasets should have been provided, including the length and complexity analysis of these datasets.


[r1] COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis, CVPR 2019
[r2] Learning To Recognize Procedural Activities with Distant Supervision, CVPR 22
[r3] Timeception for Complex Action Recognition, CVPR 19
questions: Please see the cons for details.
limitations: Yes
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a method for Long-Form Video-Language Pre-Training, in which the author introduce 1) multimodal temporal contrastive loss for aligning multiple modalities and 2) hierarchical temporal window attention for capturing spatiotemporal dependencies at lower cost. Experimental results demonstrate improvement over the previous state-of-the-art.
strengths_and_weaknesses: Strengths:

1.	This paper is well written, and the core idea is easy to understand. The proposed method and formulate is clean, straightforward, and easy to re-implement. 

2.	Long-term video modelling and multi-model alignment are unsolved problems in the community. They are challenging due to the video redundancy/complexity and memory burden. It is great to see this work trying to investigate these problems. 

3.	The proposed hierarchical temporal window attention is an interesting direction for reducing the computation cost of the video transformer, which build local temporal attention window in early stage and global temporal dependencies in later stages when the feature map is smaller.

4.	The experimental result looks good, which outperform previous state-of-the-art works.

Weaknesses:

1.	In multimodal temporal contrastive loss, is there an assumption that the clip content and sentence will not repeat in the video? Otherwise, the line 50-51 will not make sense.

2.	Line 157-158, what is the sampling strategy here for K and V. If these two sets are far away from each other, samples in the positive pair may not match each other.

3.	The effectiveness of each component is not well analysed. There are some ablation studies in the Table 5. However, it is not clear to see the effectiveness of L_time alone, L_mlm, L_vtm, \lambda_1 and \lambda_2. It will be great to show how each part contribute to the final performance.

4.	Missing details in the comparison: Table 1-3 demonstrate the performance of this paper and previous works. In addition to the pretraining dataset, these details are also important: a). Computational cost. 2) Running Time. 3) Additional dataset (used in pretraining and pretrained model). 4) Input resolution. Combined with the weakness No.3, it is hard to distinguish which part of this paper is the key for the superior performance.

questions: 1.	The multi-model in this paper refer to the vision and language. Is audio data useful or complementary in the proposed method?
limitations: 1.	The large-scaled self-supervised pre-training may generate more carbon emission.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper focuses on pre-training a long-form video-language deep learning model. To handle the challenges in modeling long-form data, the authors propose 1) a large-scale dataset, 2) temporal contrastive loss, and 3) a hierarchical attention strategy. The proposed approach achieves favorable results on the retrieval and VQ tasks.
strengths_and_weaknesses: *Strengths*
1. Except for some indexing issues, the paper is well-written and easy to follow.
2. The authors build a large-scale long-form video-paragraph dataset and promise to release it to the community. As the field is not well-explored, the dataset could be a critical factor in stimulating further research in this field.
3. The proposed pre-training algorithm improves the performance on several downstream tasks.
*Weakness*
1. The indexing $v_{i, k}$ and $t_{i, k}$ in Section 3.2 are a bit confusing. The authors could consider to change the index to $v^i_k$ and $t^i_k$.
2. It is unclear how the authors sample the set $\mathcal{V}$ for the positive instance $t_{i, q^+}$. Specifically, if the set $\mathcal{V}$ only contains frames that are all far away from the anchor along the time axis, then the instance with index $q^+$ is also far away from the anchor. How does the proposed method handle this situation?
3. Recently, there have been some papers showing the benefit of leveraging a pre-trained text encoder for visual-language tasks, e.g., Imagen. Can the authors discuss/comment on the potential results of leveraging a well-pre-trained text encoder on tasks this paper focuses on?
Saharia el at., "Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding."
questions: Please see Weakness
limitations: Ths authors addressed the limitations and potential negative social impact well.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a method for long-form video-language pre-training by a multimodal temporal contrastive learning that leverages temporal continuity of clips within videos and their corresponding textual descriptions, and a hierarchical temporal window attention for encoding long videos using transformers. In addition, this paper proposes a new dataset with long-form video and paragraphs useful for pre-training video-language models. In experiments, the authors show that their method outperforms baselines in seven downstream tasks.
strengths_and_weaknesses: Strengths:
+ New dataset for long-form video-language modeling that will be released for research
+ New neural network architecture and contrastive pre-training for long-form video-language models
+ Outperforms baselines.


Weaknesses:

- Baselines not pre-trained with proposed dataset
-- In Table 1, the authors compare the proposed method trained with the proposed dataset outperforming the baselines on ActivityNet Captions. However, the baselines are not pre-trained with the same dataset as the proposed method. This makes the result inconclusive as to whether the performance boost comes from the proposed LF-VILA-8M dataset or the LF-VLP method or both. I suggest the authors to add results from the baselines pre-trained using LF-VILA-8M so that readers can make a stronger conclusion about the proposed method. Also, are all other results using baselines not pre-trained using LF-VILA-8M?
 

- Importance of HTWA.
-- In Table 5.b, the authors show a comparison between not using HTWA and using fixed size windows of 4 and 32. The difference between fixed 4 and HTWA performance seems marginal. Can the authors clarify how this result validates the effect of HTWA?

- Without contrastive loss on global representation?
-- In Table 5.a, the authors clearly show that pre-training is important. They also show the difference between using the proposed contrastive learning that uses time continuity to select the negatives and positives. However, I think this table is missing an experiment where the global contrastive learning is not used. This would make it clear if global contrastive learning is an important piece of the equation or if the proposed contrastive learning is good enough on its own.

- Use stage two for retrieval to compare the performance of the cross modal encoder?
-- The authors mention the representations learned from the model in stage one are widely used and efficient for retrieval. However, it would be good to report whether the cross modal encoder performs as well or better than the stage one model, especially since the cls token of this encoder outputs a score denoting whether the input paragraph and video are matched. 
questions: Please see weaknesses.
limitations: Yes
review_title: null

Meta-review: metareview: After a discussion, the reviewers reach a consensus towards the acceptance. The author rebuttal resolved most of the concerns that the reviewers raised and the reviewers find its value to the community. Together with some of the reviewers, I also appreciate that the paper explores long-term video modelling which is an under-explored field. In addition to the long-form video pretraining technique, the paper also introduces a pretraining dataset with long-form videos showing the pretrained model achieves competitive performances on multiple retrieval and VideoQA benchmarks.
confidence: Certain
Decision: Accept