Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper is about a self-supervised video representation with a multi-modal learning process that the authors then use for performance on a variety of tasks. The main contribution of the paper is a successful effort to incorporate BERT-like models into vision tasks. As is detailed in the related work, the field has been inching towards this but without as much success as this paper has.

My main criticism of the paper is that it feels like there is everything and a bag of chips happening; It's exceptionally hard to tease apart what is the main contribution to its success. I mostly came away from the paper thinking that it was good to see an existence proof of successfully incorporating the result, but not having really understood anything more wrt why or how this works. Other than it being a good idea to have a bigger model and more varied types of gradients, it's unclear what this model does that distinguishes it from other approaches.

On a more specific critique level, why use COIN? And why compare on a frame accuracy metric? The comparison to Ding & Xu seems a bit odd given that they don't assume access to annotations but rather to video transcripts. There are other datasets that you could make use of here that are more applicable, like Thumos14 or ActivityNet. I understand that this is a small section, but arguably the paper would be stronger if more time was spent on the main result than on this sidebar.

Overall, I'm giving it a weak accept because I do think that the community should be aware of this paper's result.
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This is one of those papers where the number of experiments conducted to produce the results is beyond the capabilities of "almost all" research groups. From the paper: "we use 32 Cloud TPUs. The model is trained for 2 million iterations, which takes around 2 days." However, with that being said, it's a good paper of general interest to the community.

The paper focuses on self-supervised learning in video, and combines two contributions. The first is using a noise contrastive estimation loss (2016) which can be used for any visual dataset. The second is a cross-modal (BERT) model that requires language and vision. A few modifications over other BERT flavours are introduced. The cross-modal BERT is not tested alone, however when added to the NCE loss function, seems to suit a range of downstream tasks from classification to anticipation and captioning. NCE alone seems to clearly produce better results over published results, however these are not compared like-to-like, as published results are used for this comparison.

The paper is full of technical details to reproduce the results. This makes the main novelty is actually in showing that this approach works. However, the approach is technically sound and up to my knowledge has not been attempted before.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper presents a novel method to extract cross-modal text-visual embeddings on the HowTo 100M corpus. The core idea is to extend previous work on clip-level embeddings (e.g. the max-margin ranking loss proposed for HowTo 100M) to a transformer architecture which takes into account the entire context of a video, which should lead to better learned representations and improved performance in downstream tasks. In addition, the max-margin loss is replaced by noise contrastive estimation. 

The paper is well written and explains the main problem well, however I do have a few questions:
- I do not understand the sentence "However, for images and videos, the inputs are real-valued vectors." (Section 3.2) - Transformers are being used for speech recognition or speech translation - the input features are not the problem. The outputs are assumed to be discrete (in the original formulation)
- Why not directly compare your approach to the approach presented in (Miech, 2019c) - it would be interesting to see a direct comparison, but as far as I can tell, there is no overlap in tasks?
- What is the influence of adding punctuation to the ASR output, how good is it, and how good is the underlying ASR? Why did you not use the original text annotations provided by HowTo 100M, but run the audio through Google ASR (again?) It would be good to know how good the ASR is, and if adding in punctuation post-hoc works well, and how this influences your use with a pre-trained BERT model. My guess is that the BERT model will be happy as long as it sees a "." at the end?
- Also, would it be possible to compare the results of your work with some of the work in (Miech, 2019c) - it almost seems that your work avoids comparing your results to this previous work.

review_title: Official Blind Review #3

Meta-review: This paper studies self-supervised video representations with a multi-modal learning process that the authors then use for performance on a variety of tasks. The main contribution of the paper is a successful effort to incorporate BERT-like models into vision tasks.

Reviewers acknowledged the extensive empirical evaluation and the good performance of the approach. However, they raised some concerns about the lack of clarity and the absence of analysis and interpretation of the results. The AC shares this view, and recommends rejection at this time, encouraging the authors to revise their work addressing these analysis and clarity questions.  
Decision: Reject