Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper concerns the problem of video-text representation learning and its application to video-text retrieval. The source of pre-training data comes from YouTube video-ASR pairs. The main novelty is the adoption of a generative objective (i.e., video captioning) to refine the video encoder ($\Psi''$) and the text encoder, given the paired data input. Further, the paper observes that sometimes videos belong to the same batch could share similar characteristics/semantics, and therefore traditional contrastive learning objectives that repel positive samples from these hard negative samples could potentially hurt the learned representation. To this end, this paper introduces an idea called support-set bottlenecks, which effectively alleviate this conflict through batch-wise attention.
 
Like the review title indicated, the paper has demonstrated strong empirical results on some common benchmarks. The related work is mostly comprehensive. The method section is also described with good clarity. However, the main concerns include the somewhat weak motivation of the method and some overstated claims. The reasons are detailed as follows.

i) The motivation behind using a support set is still unclear, besides the obvious reason of empirical gain. Compare to related techniques such as semi-hard negative mining, what are their relations and the advantage of the proposed approach? Also, since the idea is generic to the training objective, what is the rationale for not using it for the contrastive objective (but instead on the more complicated captioning objective)? Or if you have made attempts, what are the observations?

ii) The overall scope of the proposed method is somewhat overstated. Despite that the method is generic to learn video-text representation, the downstream tasks only involve a single type of problem (i.e., video-text retrieval), which raises concern on its effectiveness across a broader range of tasks. Besides, even for video-text retrieval, notable benchmarks such as [YouCook2](https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/viewFile/17344/16367) and [MSVD](https://www.aclweb.org/anthology/P11-1020.pdf) are missing, not even in the related work section. Consider acknowledging these works.

iii) From Tab. 2, we know that the model variant "Cross" works the best. But chances are there, other videos from the same batch might be totally irrelevant to the video-of-interest (more pronounced when the batch is small). Could this potentially hurt the learned representation? Also, any insight on what intuitively makes "Cross" better than other model variants?

iv) The conclusion drawn on Contrastive Loss from this work (page six bottom) is opposite to existing works, that is, triplet loss works much better InfoNCE. Any insight or observation besides these mentioned in the paper would be appreciated.

vi) Consider the relevance of [Prototypical Contrastive Learning](https://arxiv.org/pdf/2005.04966.pdf) to this paper (also alleviate the over-discrimination/"false negative" issue on instance discrimination), please acknowledge the work in the related work section.

Minor comments:

i) Please check if the font style follows ICLR'21 guideline, for example, for the title and references.

ii) The green arrow between the positive pair is missing in Fig. 1b.

=========================== Post-Rebuttal ===========================

My questions are well addressed in the rebuttal. I acknowledge the novelty in the paper and the convincing empirical results. The authors have committed to include more datasets to enhance the result completeness and add relevant but missing related work. Taking all these into account, I am adjusting my final rating to Accept.
review_title: Strong empirical results; motivation needs clarification

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: Summary: 
This paper focuses on better learning of video-text representations. To this end, the paper introduces a new generative task of cross-captioning which alleviates the typical issue of contrastive learning by learning to reconstruct a sample’s text representation as a weighted combination of a video support set. The proposed approach performs better than previous work on various datasets. 

Pros: 
1) The paper is well written and easy to follow. 

2) The proposed method sounds novel and interesting. The empirical evaluations on various datasets suggest that the proposed method is better. 

3) The ablations on various modules of the proposed method is very interesting and thorough. 


Cons:
1) The current approach limits to using only the videos in the current batch for the support set. One could also try retrieving the support set from the full dataset in an online way. It would be interesting to see this ablation. 

Overall:
The proposed method of cross-captioning is novel and the thorough empirical evaluations/ablations further show the superiority of the proposed method as well as the usefulness of each component.  

Questions: 
1) Please provide statistical significance scores wherever necessary, e.g., Table-4 ours vs ours-pretrained difference is statistically significant? 

2) Is it possible to ablate on the choice of support set from within a batch vs. full dataset?



review_title: Better learning of video-text representations via cross-captioning loss

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
The paper introduces an interesting problem when doing noise contrastive learning. In particular, the model will push away the representations of all negative pairs although these samples sometimes are semantically related. To alleviate this problem, the authors develop a generative model to push these related samples together by reconstructing the caption from the weighted feature of support samples. Overall, the insight of semantically related pairs in negative sampling is interesting for me. My major concern is about the architecture of the proposed model, the performance on other tasks, and some additional ablation studies (see below).

Pros: 
1. The paper takes an important issue in contractive learning: semantically related samples in negative sets.
2. The paper proposes the cross-instance captioning to alleviate the discouraging concept sharing among samples. The comprehensive experiments on retrieval tasks show SOTA results with a margin.

Cons: 
1. A pretrained T5 with 12-layer encoder and 12-layer decoder is used in the model. It is doubt that why the cross-instance captioning needs such a heavy decoder. How about a shallow decoder affect the performance? 
2. The caption results shown in the appendix demonstrate a weak advantage in generation tasks. Are there some reasons for the opposite performance compared with the retrieval tasks?
3. The model places the pretrained architectures after the initialized Transformer pooling head, which will affect the pretrained weights. What is the training strategy and learning rate to deal with such a problem?
4. What is the possible reason for the reconstruction bottleneck that both smaller and very large sizes degrading the performance? Besides, the performance is sensitive and high-impact with different sizes.
5. There is a doubt that the MdR of T5-Small in Table 1 (c) is the same as that of T5-Base but R@1 and R@5 have a margin. 
6. What is the meaning of `None` in Table 2? Does it mean training without cross-instance captioning?
7. This paper sets the $\alpha$ to 0.2 in Eq. (1). How about other numbers affect the results, e.g., 0.1 and 0.3? What is the value of temperature T in Eq. (3)?
8. Are there some zero-shot results for the pretrained model?
9. How about the performance of the video representation on other tasks, e.g., action recognition?
10. The support-set is the samples of the mini-batch. How to know which one semantically relates to the anchor? 

11. Missing references:
1). Linchao Zhu and Yi Yang. 2020. Actbert: Learning global-local video-text representations. In CVPR.
2). Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. 2020. UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation. arXiv:2002.06353.
3). Bruno Korbar, Fabio Petroni, Rohit Girdhar, and Lorenzo Torresani. 2020. Video understanding as machine translation. arXiv:2006.07203.
review_title: contrastive learning for video-text representation learning with a new proposed generative objective

final_score: rating: 7: Good paper, accept
review_content: ## Summary

The paper proposes a new way to train joint representation of video and text. In particular, it argues that the contrastive based approach is too strict in the sense that it forces pairs of video and text to be far away only because they are not from the same instance without considering potential shared semantic. To address this, they propose to combine the contrastive loss with a generative loss that tries to generate the text from one video based only on *different* videos within the batch. Interestingly, adding this objective improves performance of retrieval.

## Strengths

* The paper is clearly written which makes it easy to understand
* To me the fact that the Cross approach works best is not completely intuitive which makes the result quite interesting. This approach is novel to the best of my knowledge.
* The ablation study is thorough and overall well conducted.
* The final performance are quite impressive (albeit some difficulty to fully compare to previous work as explained in the weaknesses section). 

## Weaknesses

There are a few questions/concerns about the work that I hope can be answered in the rebuttal.

**Is the support set idea always good?**: While this idea seems to work well on MSRVTT I wonder whether or not the same loss would work well on much larger scale and more diverse datasets. In particular the method seems to rely strongly on the fact that there are other semantically related videos/text in the batch. While this may be true for small scale datasets it might not always be the case. I see a few options to try to illustrate a bit more that point:

* Is the support set idea also important when training on HowTo100M? Does it matter there? Does it hurt?
* I would expect the support set size to depend on the dataset? Was this observed in practice? 

**What are zero shot performance when training on HowTo?** I would be curious to know what would be the ZS performance when just training on HowTo. In particular (related to the previous question) is the generation loss also important there at all?

**Comparison to the state-of-the-art is not totally fair**: even though less features are used, the pretrained models are trained on different data (IG65M dataset), therefore it's not clear that the comparison is fair compared to MMT. Would it be possible to do the comparison using the same set of features? Currently its not clear if the best performance comes from the fact that the method is better or if the input features are better.

**Technical questions about aggregation**:

* For the text encoder, it does not seem to be standard to combine RNN with a Transformer. Was it really important? Similarly for the vision model, how important was it to add the convolution in the transformer?

* Have you tried other options for $c_v$ (e.g. by averaging all the outputs instead of just taking the first? Asking since in that case it seems that no special [CLS] token was added so taking the first token looked arbitrary.

* When averaging temporal representations of the different videos in equation (3), this somewhat assume that all videos are temporally aligned. However this is often not true. Is this a problem that the authors have considered?


**Some details about the tables**:

* In Table 1. e) why the numbers with batch size 64 do not match the other best numbers in the table (55.7 vs 55.2).
* Table 1. d). What does inter+intra means? In particular for the intra does it mean you do a contrastive loss between video and video? Are there any augmentations that are used for that case? I might have missed this but I could not find the details of that in the paper.

**Comment about the title + question**: when I first read the title I thought that visual features would also be learned end to end (due to the term representation learning). I wonder if representation learning is the best term to put in the title due to the potential confusion. Related to that, have you tried to also finetune the video networks? If yes, were there benefits? 

## Justification of rating

The paper presents a simple but effective idea for training joint text and video models. The paper is clearly written and the results are compelling and do support the claim of the paper. There are still some things that can be improved about the paper and I am expecting the authors to reply to my questions and concerns. 

## Post rebuttal comment

The authors have successfully answered my concerns and added the necessary experiments. I think the paper should be accepted as the support set idea is an important one to improve upon the simple contrastive idea (in particular to soften the set of positives). 

review_title: An interesting objective for video-text models

Meta-review: This paper studies the problem of learning better video-text representation learning with an application to video-text retrieval. It proposes a key innovation: it uses a new generative task of cross-captioning that addresses issues with contrastive learning by learning to reconstruct a sample’s text representation as a weighted combination of a video support set, using a novel objective function using video-set bottlenecks. It uses pre-training based on YouTube video-ASR pairs, and shows empirical results where the proposed method outperforms multiple SOTA methods.

The authors have addressed the feedback of the reviewers, especially with the following improvements:

- Experiments were run on more datasets
- Relevant work pointed out by the reviewers were added
- Concerns regarding technical details were clarified

Decision: Accept (Spotlight)