Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposed to distill the large-scale pre-trained video-language multi-modal knowledge into the language modeling. Different knowledge distillation objectives are applied. In the downstream task, the distilled language model shows performance improvement over the conventional language model. 
main_review: Strength:
+ The novelty comes from dIstilling the video-language multimodal knowledge learned by video-text pre-training into the language modeling. That could enable the language model the ability to deal with more visual-relevant commonsense such as action.
+ Multiple knowledge distillation methods are well studied and compared. 
+ Proposed method outperforms the conventional language model pre-trained only by text in quite a few downstream tasks.
+ The fair comparison against the Vokenization method further validates the effectiveness of the proposed framework. 

Weakness:
- When comparing with Vokenization, 30K videos are pre-selected according to L205. A question about the detail is how to select the videos?
- In the downstream tasks, it is interesting to see in what samples your model is generally better than text-pretrained language model.  Does that correspond to your motivation that your model should help in activity-based and physical commonsense knowledge?

Clarity:
The paper is clearly illustrated and well written. 
limitations_and_societal_impact: There is no potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper studies the problem of how to use visual information to help NLP models. It argues that videos contain rich visual grounding information and could potentially benefit pure-text NLP tasks. It proposes to distill such visual information into a language model during pre-training. Experiments on GLUE benchmark suggest the efficacy of the method.

Overall, I applaud the novelty and problem the paper tries the address. However, the experiments could be strengthened by better control of the baselines and more analysis (see cons). I lean towards acceptance but I would suggest adding the suggested experiments later if possible.
main_review: Pro:
The paper tackles a long-standing problem: grounding NLP models. The paper takes a practical yet interesting angle: distill grounding information in the pre-trained language models so it could benefit various NLP tasks.

The distill approaches make sense and works well in experiments.



Con:
	
1. Potentially important baselines missing. To verify that the gain is from injected visual information, the authors introduces a student model that is pre-trained on HowTo100M. However, another possible contributing factor is the distillation process, as the distillation process itself might bring performance improvement, as long as the teacher is a reasonable model. 
Thus, I think a baseline where the teacher is trained only on the text of HowTo100M/Wiki and distilled to a student should be included to disentangle the effect of the visual information and the effect of the distillation process.

2. Concern over the setting. Is the teacher model initialized from BERT? If so, then a baseline where the teacher is simply the original BERT should be included, to investigate if some of the useful information distilled into the student is from BERT.

3. Lack of analysis. While showing improvement on GLUE benchmarks is a good sign, it is still hard to see how the grounding information helps a concrete NLP task (e.g., it might be hard to imagine how exactly videos help SST-2). Some examples of how the grounding helps PIQA or Knowledge test in GLUE could be helpful.
Another potential aspect I could think of is to test whether the grounded pre-training relieves reporting bias [1,2].
I won’t count this as a reason for rejection but would appreciate a better analysis.
[1] Reporting bias and knowledge acquisition. AKBC 2013.
[2] Do Neural Language Models Overcome Reporting Bias? COLING 2020.
	
limitations_and_societal_impact: Nothing I could think of.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes to improve language understanding using a video-language knowledge distillation method. Compared to the previous state of the art approach, “Vokenization”, they use a large multi-modal video dataset for video-language pre-training to accommodate larger vocabulary needed for better language understanding. For their knowledge distillation based approach, the teacher is the pre-trained model on the multi-modal dataset whose weights are distilled to the student language model trained only on a text dataset. This also avoids the approximation error in the vokenization approach. They perform extensive experiments and improve performance on several downstream language understanding benchmarks (GLUE, SQuAD and SWAG) and also show the commonsense knowledge and reasoning learned from videos. 

main_review: The paper is marginally novel and looks at an interesting side of improving language understanding models using video-language based pre-training. It is well written and comprehensive. They perform extensive experiments to show the effectiveness of their method on downstream tasks and several ablation studies to compare different alternatives for the loss functions and pre-training datasets.

1. In line 104, the authors discuss one of the limitations of the vokenization paper - the approximation error and that their method addresses this. It is though not very clear what this error is and how the method takes care of it? 

2. Vokenization trains on MSCOCO which is a relatively small image-text dataset, but combining it with Conceptual Captions [1] or SBU captions [2] datasets wouldn’t be enough? What is training on video datasets bringing extra than just on the combination of the above image datasets?

3. What happens if we take the teacher language model after cross-modal pretraining and fine-tune on the text pre-training dataset without knowledge distillation? I believe this is different from the baseline mentioned in Table 1., first row.

4. Why are the ablation experiments done on BERT 6L/512H whereas the main model is trained on BERT 12L/768H ?

5. Although the models show effectiveness over different tasks and datasets, the novelty of the method is limited as it uses state of the art knowledge distillation and pretext self-supervised objectives for training the model. 

[1]Sharma, P., N. Ding, S. Goodman, et al. Conceptual captions: A cleaned, hypernym, image-text dataset for automatic image captioning, ACL, pages 2556–2565. 2018
[2] Ordonez, V., G. Kulkarni, T. L. Berg. Im2text: Describing images using 1 million captioned photographs, NIPS 2011 



##Post Rebuttal

I have carefully read the authors' response and they address my concerns. I appreciate the new experimental results provided by the authors and they give a better idea of the issue raised. I maintain my original rating.


limitations_and_societal_impact: I recommend the authors to add the broader impact and limitations of the paper from the supplementary to the main paper. The main limitation discussed by the authors is the bias in the video datasets (gender, racial, etc.) and this bias getting amplified during training. 
Another potential limitation is the carbon footprint and cost effectiveness of training on these large-scale video datasets. Is there any alternative where you could still use visual information for improving language understanding in a more environmentally friendly way ? An ideal proposition but worth thinking about.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes a new method to improve language understanding with the help of the vision modality. In the proposed method VidLanKD, a teacher model is firstly pre-trained with video-language contrastive loss on Howto100M data. Then the LM part of the teacher model is used to distill knowledge to a student LM model.
main_review: Using vision modality to help language understanding definitely has huge research value. However, I find the paper lacks detailed explanations on some design choices and also on the reasons why it is more preferable compared with Vokenizer.

1. Why do we need a distillation process from the teacher LM to another model? What will happen if we directly pre-train more with only text on the teacher LM pre-trained with image-caption pairs, and fine-tune it on downstream tasks?

2. The paper mentions one limitation of vokenization, images cannot represent words well that require more activity-based commonsense, which is valid. However, how can the proposed method address this problem?

3. Both the teacher model pre-training and the distillation have multiple losses. I feel that the ratio of those losses is a very important hyper-parameter that can significantly affect the performance. For example, for the teacher LM pre-training, since the captions are simply too short and few compared with the unsupervised text, if we don’t set a high coefficient for the contrastive side,  the MLM loss will dominate, which would make the whole learning-from-vision idea pointless. While the paper lacks thorough discussions on this.

limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact of their work
review_title: null

Meta-review: All but one reviewer recommend the acceptance of this paper.

The reviewer not recommending acceptance has remaining concerns about positioning this work with respect to vtokenizer and the AC encourages the authors to address this. Please also be careful about explaining additional details in this submission as requested by this reviewer. However other reviewers were quite positive about this work and one reviewers noted that they felt that this paper does indeed make a more convincing case for the general topic of creating visually-grounded NLP models than Vokenization.

The AC recommends acceptance.

Decision: Accept (Poster)