Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a contrastive learning-based method for learning video representation. The core idea is to use the temporal order of video segments and sentences to align the temporal-sensitive representation. Meanwhile, a negative sampling strategy based on temporal granularity and shuffling the units in the positive sequence is proposed to obtain positive and negative samples. Experimental results on three tasks show the effectiveness of the proposed method. 
strength_and_weaknesses: Strengths:
1. The overall idea is simple and proved to be effective empirically.
2. The core idea of exploiting temporal dynamics to align videos and paragraphs through global and local granularities provides insights for studying the temporal characteristics of videos.
3. Extensive experiments have been conducted to verify the effectiveness of the proposed sampling strategy and DTW-based metric.
4. The proposed method achieves good results on three tasks.

Weaknesses&Questions:

Though the overall paper is interesting, I have the following concerns:

1. The authors claimed that the motivation is to address the case where two clips are visually similar but are from different segments. However, it seems this problem has not been well addressed through the proposed scheme.

2. I suggest the authors provide more discussions on the difference between the proposed method and other works that align video and text, such as [a].

3. The performance of the zero-shot action step localization task is not that good compared with other methods (see Table4).

4. The loss L_{seq} brings improvements on most of the settings while leading to worse results on Action Step Localisation (compared with Video Clip in Table 6(b)). Could the authors provide some discussions?


[a] Temporal Alignment Networks for Long-term Video. CVPR2022

clarity,_quality,_novelty_and_reproducibility: Exploiting temporal dynamics to align videos and paragraphs through global and local granularities is novel. This may provides insights for studying the temporal characteristics of videos.
summary_of_the_review: The overall idea is simple but effective and achieves consistent improvements on three tasks, including video retrieval, action step localization, and few-shot action recognition.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work studies video-text contrastive learning with the focus on learning the global temporal context over long videos. The authors propose a new contrastive learning method, TempCLR, which poses the pairwise distance as matching cost between text (sentences) and visual (clip) sequences, computed by dynamic time warping. Negative samples are constructed by shuffling the clip order of a positive video, as well as frame order within each clip. Experiments are conducted on zero-shot video retrieval, action step localization, and few-shot action classification tasks, where TempCLR improves the results over VideoCLIP and other baselines, especially when the task requires global temporal information.
strength_and_weaknesses: ## Pros
- The proposed method of matching sequences for contrastive learning is instinct, and the idea of shuffling sequence order to create negatives also arises naturally from the motivation of using global context of long videos.  
- Experiments are conducted on three different downstream tasks, under zero-shot, few-shot and fine-tuning settings, with significant improvements over the VideoCLIP baseline.  
- I find the ablation studies and visualizations quite helpful in understanding the mechanism of individual components in the approach.  
- The paper is very well written, with concise problem formulation, and intuitive figures to illustrate the idea. 

## Cons
- While not directly comparable, the paper can benefit from a discussion of prior work using shuffled clips for video unsupervised learning ([Shuffle & learn](https://arxiv.org/abs/1603.08561,), [OPN](https://arxiv.org/abs/1708.01246), [VCP](https://arxiv.org/abs/2001.00294) etc.).  
- It is interesting that incorporating negative samples from other instances ("unpaired" in table 7) performs the worst, as instance discrimination is typically helpful in video contrastive learning. I wonder if it is possible to use both unpaired videos and shuffled videos as negatives in contrastive learning?  
- From supplemental table 8 using VideoCLIP with `DTW + Cap. Avg.` metric seems to already produce strong results. Does this mean the TempCLR pretraining is not essential to achieving good retrieval performance? Is the same observation true for other tasks too?  
- For text-to-video retrieval, including results on more datasets (DiDeMo, ActivityNet Captions etc.) should make the comparisons more convincing.  

## Questions
- In table 1, is there a particular reason why cap. avg. measure is not reported, or cannot be used for full video retrieval with background?  
- Can the authors comment on the number of negatives per video used in TempCLR, as well as the robustness of the method to the size of negative set?  
clarity,_quality,_novelty_and_reproducibility: The writing quality and clarity of the paper is good, though there are a few implementation details I might have missed (see questions above). I think the method is sufficiently novel, as it differs in may ways from prior works that inspired this work.
summary_of_the_review: Given the simplicity and effective of the proposed method, I lean towards accepting the paper, though there are a few questions on the constructions on the negative samples, among other minor concerns listed above. I look forward to the authors' response to further clarify these issues.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper addresses the problem of video and text matching and pre-training in a contrastive learning contrast. Specially, the paper argues that previous methods considered the "unit-level" context, i.e. : the similarity of one given clip and sentence pair, but did not consider the global context of the clip as part of a longer video or the sentence as part of a longer paragraph. Thus, the paper addresses video and paragraph matching and explicitly considers the temporal alignment of clips and sentences. 


To this end, the paper presents a method of using dynamic time warping to match a sequence of sentences (i.e. a paragraph) with a sequence of clips (i.e. a video) along with a temporal contrastive learning framework where negative samples are obtained by shuffling the temporal order of the anchor samples. Thus, the learned representation can facilitate better temporal paragraph/video alignment. The method can also be used for learning video representation without considering video-text pairs. The paper presents experiments on video retrieval, action step localisation and few-shot action recognition. 


strength_and_weaknesses: Strengths:
1. The paper is well written. 
2. Considering the clip-level temporal context as aligned to sentence level description of the video is a interesting and novel motivation for learning representation by contrastive learning. 

Weaknesses: 
EDIT: The rebuttal have addressed the limitations I have written below.

Generally, I agree with the approach of considering a video as a sequence of clips and matching it with a paragraph when it's considered as a sequence of sentences. However, I have a few concerns regarding the way that it's done in the paper and a few limitations/considerations that were not discussed:

1. **Video and clip length**: it is worthwhile to discuss what is the length of a video and a clip in the video in the paper and how that affects the method and the performance and how does the method perform/consider untrimmed vs trimmed videos. For example, the Large Scale Movie Description and Understanding Challenge [1] could fit perfectly in this framework as it is composed of untrimmed Hollywood movies (sometimes an hour plus long) divided into clips, each one associated with a caption. Thumos14 [2] could also fit perfectly as it composed of untrimmed action localization videos. The paper considers benchmarks with much shorter and constrained video, such as YouCook2 and CrossTask where videos are only a few minutes long and the context remains the same and something-something dataset where videos are a few second in length. In my opinions, the generalisation ability of the method to longer and untrimmed videos should be discussed. 

2. **Change of context**: There is no discussion on how the change of context affects the method and the learned representation. For example, say that we take a video composed of 2 completely different scenes (as depicted in Thumos and LSMDC for example), how would that affect the performance, given that the method tries to find a matching that produces consistent temporal dynamics where in such example, there is an abrupt change in the temporal dynamics? I want to be clear that I'm not arguing that the method would not work, I argue that given the emphasis on temporal consistent, a case where there is a clear change in the temporal context in a video should be discussed. 

3. **Dividing a video into clips**: does the method assumes that the video is already divided into clips? how would the method perform if such a division is not given or it can be easily divided into clips by running shot detection? 


[1] https://sites.google.com/site/describingmovies
[2] https://www.crcv.ucf.edu/THUMOS14/home.html
clarity,_quality,_novelty_and_reproducibility: In my opinion, the motivation of this paper and the problem it addresses are interesting and novel. However, I'm not sure about the generalisation ability of the method as I wrote above and specifically there are a few limitations or aspects of it that were not discussed. Regarding originality, considering temporal information as a pre-text signal for pre-training has been discussed and considered before (in several papers), but AFAIK not in this specific context and setting. 

[3] Lee, Hsin-Ying, et al. "Unsupervised representation learning by sorting sequences." Proceedings of the IEEE international conference on computer vision. 2017.
[4] Misra, Ishan, C. Lawrence Zitnick, and Martial Hebert. "Shuffle and learn: unsupervised learning using temporal order verification." European conference on computer vision. Springer, Cham, 2016.
[5] Xu, Dejing, et al. "Self-supervised spatiotemporal learning via video clip order prediction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.

summary_of_the_review: In my opinion, there are a few aspects and limitations of the method which are not discussed in the paper and affect the applicability of the method to other, less constrained, datasets. Also, the technical novelty is a bit limited as using temporal ordering as a self-supervision signal has been proposed and discussed in various papers before.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The proposes a novel contrastive learning pipeline for text-video correspondence. The method adopts dynamic time warping to align the sequence-level consistency and takes shuffled clips as the negative pairs. The experiments on video retrieval, action step localization, and few-shot action recognition have shown the method achieves effective representation. In addition, the ablation study also dissects the core strategies in the paper like DTW and negative sample selection. 
strength_and_weaknesses: Strength:
+ The empirical results achieve SOTA performance across three tasks under six different setups. 
+ The methodology is straightforward and makes sense. Unit-level alignment is important for downstream tasks, e.g., text retrieval. 

Weakness:
+ Compared to VideoCLIP[1], the technique contribution seems weak. I wonder about the actual effect of DTW compared to the temporally overlapped pairs strategy. Besides, the random shuffling trick is not novel in self-supervised video representation learning [2].
+ The writing is hard to follow due to typos and needs to be polished. For example, the caption of Table 2. 'Mesure' --> 'Measure'.


[1] VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding. EMNLP 2021.

[2] Self-supervised Spatiotemporal Learning via Video Clip Order Prediction. CVPR 2019.
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper writing is hard to understand. For example, in table 4, '$^*$' of VideoCLIP$^*$ misses notation. The grammar and the usage of the vocabulary also need to be improved. 

Reproducibility: The implementation details are listed in the supplementary appendix thoroughly.


summary_of_the_review: Overall, the contribution of the paper is incremental and lacks novelty. Thus, I recommend rejecting the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Paper proposes a contrastive learning-based method to learn video representations.  The contrastive learning scheme is built on top of the segment temporal ordering and and sentences for positive pairs and shuffled frames and clips for negative pairs.

strengths:
+ good clarity and presentation
+ strong experimentation, analysis and results

weakness:
- some open points and clarifications from the reviewers.  Most seems to have been addressed in the rebuttal.  The authors are strongly recommended to revise their manuscripts accordingly
justification_for_why_not_higher_score: issues of clarity
justification_for_why_not_lower_score: all four reviewers are borderline positive but appreciate the ideas presented
Decision: Accept: poster