Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper tackles one important issue in video-language modeling, that is,  previous works made assumptions that video frames and the text descriptions for the video are semantically correlated. However, this assumption is not valid for the real world <video,text> pairs, since first,  the video-level text descriptions may not cover all information in the video, and second, a raw video often contains noisy or meaningless information that does not appear in the text descriptions. The previous correlation assumption did not take account of these noises, and the self-attention mechanism still leaves mis-aligned frames which harm cross-modal alignment. 

This paper proposed an E2E language-guided denoising network, LGDN, for video-language modeling. LGDN includes (1) a salient frame proposal (SFP) mechanism to dynamically filter out irrelevant or redundant frames and sparsely sample salient frames to improve video-language modeling,   (2) cross-modal interactions at three levels, i.e., salient frame matching at the token-level guided by language, momentum frame-level MSL-contrastive learning (MFCL), and momentum video-level contrastive learning (MVCL). Experimental results showed that LGDN outperforms the current SOTA. Ablation studies further demonstrated the importance of tackling the noise issue.

strengths_and_weaknesses: Strengths:

1. This paper addressed an important limitation in previous works of video-language modeling, that is, the impractical assumption that video frames and the text descriptions for the video are semantically correlated. There is no sufficient tackling of noises in raw videos. The existing employment of self-attention mechanism cannot fully address this problem. These irrelevant or redundant information still harms cross-modal alignment.  Hence, this work is valuable to the research community.

2. The SFP mechanism and the three-level cross-modal interactions are sound, well motivated, and have good novelty.  SFP is achieved through exploring MVCL and MFCL. After obtaining the language-guided salient frames, LGDN also has a language-guided salient frame matching (LSFM) module to conduct token-level semantic alignment between visual patches and words for improved performance. The final loss of LGDN is the combined loss of MVCL, MFCL, and LSFM.

3. The Related Work section is clearly written. It summarizes significant works in the past and their limitations, and clearly explained the choice of momentum frame-level MSL-contrastive learning to help address the mis-aligned frames.

4. The paper also conduct the interesting investigations of the four strategies for estimating relevance scores.

5. The evaluations are comprehensive, on four public text-video retrieval datasets and one VQA dataset. The evaluation settings support a fair comparison to previous models.  Performance gains from LGDN over existing approaches, including SOTA, are quite strong.  Experimental results showed that LGDN outperforms previous methods, including the current SOTA,  by a large margin on text<->video retrieval on MSR-VTT. LGDN also outperforms methods exploiting extra modalities or those pre-trained on very large video data.  LGDN also shows promising model capacity, as its performance is significantly improved when trained on much larger pre-training data. The extensive evaluations on other text-video retrieval datasets and VQA also demonstrated that LGDN outperforms existing approaches including the current SOTA.  

6. Ablation studies verified the contributions from the proposed SFP, MVCL, MFCL, and LSFM. Last but not least, the visualization results are interesting and helpful for directly illustrating the importance of addressing the noisy frame issue and showing that the SFP mechanism helps LGDN to improve video-language modeling.

7. Although the code is not released, the main body and Appendix provide enough details to help reproducibility. The Appendix also includes more detailed experimental results on analyzing effect of SFP mechanism, different relevance score estimations, and memory bank sizes, and useful additional visualization results.

Weaknesses:

1. The paper missed the important section of Limitations (and potential negative societal impacts). It is highly desired that the authors add discussions on limitations and potential negative societal impacts of this work.

questions: The paper is clear written. However, the authors mentioned the pre-training is set up as described in the paper due to restricted computation resources. It would be useful to explain the experimental setup alternative if more computation resources are available, that is, how to scale up the pre-training setup.
limitations: The paper missed the important section of Limitations (and potential negative societal impacts). It is highly desired that the authors add discussions on limitations and potential negative societal impacts of this work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces LGDN, which utilizes a self-taught mechanism to select salient frames to train the fusion layers. The vision and language encoder still use all frames and texts to train, and their goal is to learn representations while computing relevance scores for the frame selection. The whole mechanism by contrastive learning helps to filter out noisy information in the last few layers, boosting performance while reducing computation costs. They also demonstrate better empirical results over the baselines.
strengths_and_weaknesses: 1. Pros:

* The paper proposes an effective method to select frames from noisy video, leading to an improvement in both fine-tuned accuracy and computation efficiency.

* To address the noisy problem in video is novel to me, it might become a new approach to constructing multi-modal models. There is no perfect and exact matching between any kinds of pair data so that the impact can be extended to other domains.

2. Cons:

* The empirical results are not fully convincing to me. I think it would be better if the authors can report the model capacity of each method (see Questions 3).

* The central claim of this paper is that noisy video might harm the training, but the model still highly relies on learning on noisy data as the vision and language encoder still use full video to learn how to match the relevance between language and frames.

* It seems to me that the redundancy issue has not been addressed by LGDN (see Question 2), and the sampling might introduce another problem (see Question 3).
questions: 1. ClipBERT samples frames before feeding them into the model while your approach does the sampling only before fusion layers, so I wonder about the memory efficiency and speed of LGDN compared to ClipBERT, Frozen in Time, and other methods.
2. I understand that the SFP might address the misalignment issue, but how it can help to ease the redundancy in the video. If you compute the relevance score between text and frames, it's likely that similar frames have similar scores, and I am not sure what can prevent LGDN to select the other frames that are similar to the salient frame. I feel like some diversity loss might be necessary to address this problem.
3. What's the model size of other baselines? LGDN is composed of vision, language, and fusion Transformers while Frozen in Time doesn't have a fusion Transformer, making me concerned if the performance gain somewhat comes from the bigger models. For better comparison, you can try to use lesser layers in vision and language encoders.
4. What are global and local alignments used in the inference phase? I didn't find an explanation in the paper. Why does using global inference generates such low performance?
5. The salience sampling breaks the temporal dependency of the video (compared to uniform sampling), which is an important characteristic to solve most video tasks. Do you think this sampling will prevent LGDN from applying to other video tasks that highly depend on temporal information?
limitations: The paper's writing can be improved. The method has many components, and I was confused about which part addresses what problem at the beginning. But in general, I still understand the idea and contribution of the paper. My main concern is about the fairness of the experiments and some questions about the technical approach (refer to Questions). I will raise the score if those problems are addressed.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a Language-Guided Denoising Network (LGDN) for video-language modeling, which can deal with the noisy information in video frames. LGDN dynamically filters out the misaligned or redundant frames under the language supervision and obtains only 2–4 salient frames per video for cross-modal token-level alignment. Extensive experiments on five public datasets show that LGDN outperforms the state-of-the-arts by large margins.
strengths_and_weaknesses: Strengths: 
The paper is well organized and easy to read. The noise issue in video-text retrieval task studied by this work is practical, and the proposed salient frame proposal mechanism seems simple and effective. Extensive experiments are performed on video-text datasets and the experimental results are promising.

Weaknesses:
The paper needs more analysis and ablation studies to support its network architecture and proposals. For example, in-depth comparison between salient frame proposal (SFP) mechanism and sparse sampling or other frame sampling techniques. Also, some experimental results (e.g., Effect of SFP mechanism) in supplementary material can be included in the paper.

questions: 1. Table 2 shows the improvement of $\mathcal L_{MVCL}$ is marginal. Is it necessary to the LGDN model? Can MIL-NCE be used to compute the $\mathcal L_{MVCL}$?
2. The authors claim that the proposed LGDN outperforms the state-of-the-arts by large margins. However, as shown in Table 8, LGDN performs worse than Clip4CLIP. This point needs further discussion. 
3. Since the pre-training with CC12M brings significant improvements, it would be nice to compare some SOTA methods under the same pre-training settings.

limitations: The authors did not discuss the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: This paper proposes Language-Guided Denoising Network (LGDN) with the goal of addressing the redundancy and noisy alignment issues in video-language modeling, by dynamically filtering out misaligned or redundant frames under the language supervision and obtains only 2-4 salient frames per video for cross-modal token-level alignment. Experiments on multiple benchmarks obtain good results. Reviewers appreciated the intuitive salient frame proposal mechanism and well-written paper but also had several questions on more ablations and efficiency details and fair comparisons, some of which were answered by the authors but some remaining were left unanswered for some reviewers and they were promised in the final version (e.g., diversity loss, SOTA methods with CC12M, etc.) so it will be good to add these.

confidence: Certain
Decision: Accept