Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper looks into video-text retrieval problem.  The claim is that existing works mostly rely on titles and captions.  The paper argues for using user comments.  The challenge is that not all user comments are meaningful or relevant.  Therefore, the paper looks into attention mechanism to filter out the irrelevant content. The main contribution is a context adapter module based on transformer that allows relating visual input and textual input. The base architecture seems to have been derived from CLIP.
main_review: Strengths of the paper:

S1. The idea is intuitive.  Indeed a lot of videos now have user comments.  This is especially true for popular videos.  Having said that, popular videos probably are already getting a lot of signals from the popularity alone (likes, votes, etc.).  I'd be curious how the method could help retrieve less popular but relevant videos, where there may be some but not a lot of user comments in the first place (as it's a less popular video).

S2. The method itself is straightforward.

S3. Experiments are conducted on real datasets.  There is also ablation type of experiments looking into various components of the model. However, it is not immediately clear if there's a baseline that just throws all the comments as a single piece of text (just like a title would be), without attention to filter them out.  If so, that would be a more comparable baseline as it has the same information, but with a simpler processing.

Weaknesses of the paper:

W1. The novelty is rather incremental.  It's an extension of CLIP that already has a notion of video and text (e.g., title). The extension is in terms of user comments, which is another form of the text.  Though seemingly useful, the step to include comments as another form of text is not that surprising technically.

W2. Though the paper emphasises heavily on the video element, page 6 states that experiments are conducted on a single still frame.  There needs to be more discussion on the generalisability of the result.  I wonder if the choice of frame may affect the overall results.

W3.  The treatment of user comments as another piece of text is rather simplistic.  There could be richer structures to exploit, for instance the response/reply or discussion thread nature of some of these comments, which would be more interesting given the nature of this data.

summary_of_the_review: The paper proposes an intuitive idea of leveraging on user comments to improve video-text retrieval, but the technical novelty is rather incremental.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes the use of user comments in addition to videos and titles to learn better representations for retrieval. Since user comments may be loosely related to the video, they use an attention-based mechanism to ignore irrelevant user comments. Their experiments show that using comments using this mechanism leads to better contextualized representations, which lead to competitive results on standard benchmarks.
main_review: # Strengths

1. The paper is well written, and the problem is well motivated as well as clearly formulated.  The authors have made a strong case for using comments to learn better representations. They have adequately explained the challenges of using this modality, which explains why it was not used in earlier work. Although I was not aware of research in this area, I was able to follow the paper clearly. 
2. As far as I can tell, the idea of using comments while ignoring irrelevant comments has not been tried before. Hence, having an attention mechanism that can actually ignore irrelevant comments can aid many machine learning tasks, including representation learning.
3. The experiments are carried out in diverse settings, which is impressive to carry out. However, I would expect this method to be tried on a public video dataset that already has comments. The dataset released by [1] can be a good fit.
4. I appreciate that the limitations of this work are properly addressed. However, as shown in Figure 4, it concerns me that most real-world videos will have “distractor” comments which will be relevant to some part of the images in the video and may result in noisy representations. 

# Weaknesses

1. It is not clear if something complicated like the Context Adapter Module is required to discard v_i and t_i pairs, where v_i and t_i are not equally informative of each other. Can a simple heuristic be used instead? For instance, if the initial value of A_{ij} does not qualify a certain threshold, then they are probably NOT equally informative of each other and are worth discarding.
2. Similarly, can a heuristic be used to discount off-topic comments that have a low similarity score with the title? It would be great if the authors can compare their method with a heuristic like this.
3. In the experimental section under 4.3, the authors show that varying the number of comments leads to improvement in performance. However, to simulate a realistic scenario, a subset of these comments should be unrelated or ambiguous. As an ablation study, the ratio of these unrelated comments should also be varied. It is difficult to conclude if the attention mechanism is working in absence of these experiments.
4. The authors mention that they collected comments from the Kinetics dataset. For realistic use cases, it would be great to see experiments on a dataset which has video lengths larger than 10 seconds. Moreover, it would be important to see some statistics on the Kinetics dataset (particularly a histogram for number of comments on different videos). The authors mention that there are at least 3 comments, but more statistics would be appreciated. This is important because this is the only video dataset with comments that will be publicly released by the authors.

# Other Concerns
1. The majority of the experiments involve using the RedditVC dataset. The authors mention that they do not plan to release this dataset due to dataset bias issues. This makes reproducibility a major concern. It would be ideal if the authors can evaluate their method on a publicly available video dataset with comments [1].

2. It would be important to add a significance test for these experiments. There are no error bars reported in the experiments. Were the experiments run multiple times to estimate errors in retrieval performance (particularly in Table 1)?

3. While referring to the Appendix, it would be great if the authors can link to a particular section. For example, in Sec 4.5, Limitations, the reader would want to know where exactly they can look for additional examples, rather than simply mentioning appendix. I spent quite some time scrolling through the appendix to find more examples of limitations but could not find any.

# References
[1] Ma, Shuming, et al. "Livebot: Generating live video comments based on visual and textual contexts." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. No. 01. 2019.

Edit after response:
Thank you for your detailed response.
I would like to agree again that the proposed method has some merit. However, my main concern about the applicability of this module for realistic scenarios where the number of comments is very large still remains. I can see that this method will work when there are only a handful of diverse comments. It is unclear if it would work in a setting with large number of comments, where the number of distractor comments is larger than the non-distractor ones.
summary_of_the_review: Please refer to the discussion in the Main Review.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors proposed a multi-modal video representation learning and retrieval method based on it, taking advantage of visual and text embeddings, particularly users' comments on video sharing platforms. For this, the authors present a new component called Context Adapter Module (CAM), applying skip connections to selectively reflect user comments in the model. According to their experiments, the authors claim that the proposed method outperforms baseline models, by providing more contextualized representations of videos.

main_review: 1. The main idea of this paper is taking users' comments as a noisy signal to train multimodal (visual-text) video representation learning. Since the comments are not always strongly correlated to the contents that we are trying to learn, they proposed the CAM to selectively reflect the patterns discovered from the comments. Specifically, they utilize skip connections that are used in ResNet, so the model is forced to reflect patterns only if they are really necessary. This is an interesting idea and seems to work well in practice, according to the experiments.

2. This paper focuses mainly on the effect of "comments", but I think the proposed idea may be applicable more broadly. For instance, multimodal video models using audio and visual features might take advantage of similar approaches. Some videos convey more information with visual signals (e.g., the sound is edited with a background music) while some other videos convey more information with audio (e.g., a music video with the visual is the album jacket all the time). Selectively reflecting signals from each mode may be applicable to any mode, even including visual or audio, not just text (comments).

3. In this sense, this paper uses an idea which can be more generalized only within the narrow scope of "comments", and the experiments are verifying the efficacy of comments, not that of the model. For instance, the conclusion we can learn from Table 1 is that training and testing with comments help visual-text retrieval. However, this is already known. I believe the real value of this paper is coming from the model design, but that part is not well reflected in this table.

4. The authors constructed two new datasets instead of using existing publicly available ones. This significantly limits opportunities of replication by others, so I am curious if the authors are willing to publicly share the dataset. Given the scale of the two datasets, they should be valuable sources to advance this field.

5. Regarding comparison with SOTA models (in Section 4.4): The authors chose two datasets, MSVD and MSR-VTT. Although these are widely used datasets, there are many other datasets for this retrieval task. As it is known that retrieval performance highly depends on the training datasets, I'd recommend the authors to compare on other datasets like YouCook2, TVR, ActivityNet Captions, and more.

6. In Table 6, the authors are comparing the proposed models trained on CLIP + RedditVC (+K700) against baselines trained on other pre-training datasets, mostly on HowTo100M. Related to my comments #5 above, the final performance can be significantly dependent on the pre-training datasets. For instance, we observe CLIP4Clip performs comparably with the proposed model, and it looks reasonable to guess it is because all 5 rows (CLIP4Clip and 4 Ours) were pre-trained on CLIP dataset. To be comparable, pre-training datasets should be the same for all models, either on HowTo100M or CLIP, or both. It is unclear now if the improvement is coming from the novel model or from more informative pre-training datasets. (Note that the number of samples is not the only factor determining the quality of datasets. HowTo100M is relatively weaker, for example, as most clips are short and the associated text sentences are very short and less informative for many clips.) As it is hard to control all conditions across datasets, it is recommended to (pre-)train on the same datasets.
summary_of_the_review: Overall, the paper proposes an interesting idea, which I believe to have potentially broader impact than what the current manuscript claims. I encourage the authors to conduct additional experiments to apply the idea to other modes, as well as comparing with other methods on more datasets in a fairer way.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a text-based video retrieval method based on context from weakly related user comments. Evaluation shows that considering user comments improves the retrieval performance. Compared to conventional methods, the proposed method shows similar performance with CLIP4Cilip. However, CLIP4clip needs a much larger-scale training, so the superiority of the proposed method is claimed.
main_review: [Writing]
- First of all, what is "video-text retrieval"? I think this could be comprehended as a video OCR method, which apparently is not what the paper is about. According to what I have understood from the paper, I think the authors are performing "text-based video retrieval". Please reconsider the vocabulary, and also neatly define the task at the beginning of the paper.

- Table 1: What datasets do CLIP and CLIP + Our CAM use?

- References: Most references miss detailed bibliographic information. Please provide complete bibliographic information for all references.

[Evaluation]
- I am not convinced that the proposed method is superior to CLIP4Clip. Although CLIP4Clip seems to require larger-scale training, since it has already been trained and published, there should be no demerit for that. Please explain in more detail the merit of the proposed method compared to CLIP4Clip. In addition, the proposed method requires user comments, which can not always be expected in large quantity. This could actually be a demerit in some situations.

[Presentation]
- Section 4.1: Please illustrate the structure of the network.
- Fig. 3: The background color makes it difficult to see the lines, so please remove the background color, and also broaden the line widths.
summary_of_the_review: The proposed method works well in case user comments are available, but even so, compared to a conventional method; CLIP4Clip, it performs similarly, so the technical merit is limited.
review_title: null

Meta-review: This paper focuses on how to improve video-text retrieval via using additional user comments, and uses an attention mechanism to filter out the irrelevant comments. The main contribution is a context adapter module that allows learning from the auxiliary modality through an attention mechanism. The reviewers appreciated the overall idea's intuition and well-written paper, but they also felt that the technical novelty is incremental, and that the treatment of user comments should be more intuitive via the dialogue thread structure. There were also concerns about the applicability of the context adapter module to more realistic scenarios with much longer videos, where the number of comments is very large, and where number of distractor comments is larger than the non-distractor ones.
Decision: Reject