Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a self-supervised video representation learning that would take advantage of artificial cascades and fixation points mimicking human eye behavior.  Using the generated fixation and saccades, they could generate  positive and negative samples for each video frame that is used later in a contrastive learning approach to bring similar examples closer and negative ones further away. 


strength_and_weaknesses: The idea of using gaze data to improve the SSL is interesting and previously also has been shown promising results. The paper covered related work in depth and gave good explanation upon each work. 

However I have a hard time reading and following the paper, I find that the paper has some vague sentences or references to some papers with not clear justifications. Fro example on page 4 "To
this end, we treat such representations as negative pairs and otherwise positive pairs and optimize
the latent feature space by minimizing the contrastive loss." I really have a hard time understanding such vague sentences.. 

I am not sure why presence of saccade should indicates a semantic change in the video, in real human gaze data we performs saccades to cover different parts of stimuli and it does not necessarily means a semantic change in the stimuli or the scene, as simply human eye fovea can not cover the whole scene in details as once.  Hence comparing the work to a biological inspired approach is over reaching as it has not been tested using real gaze data. 

I could not find the details how the fixation locations was generated, why 5 fixations were selected or what is the size of the mask, are these locations generated randomly? or a saliency based approach is used to generate fixations data? 

How is the v_ik ' generated? What does it mean it is an augmented version of vik? 

Overall, I was not able to understand the paper fully as it was not written clearly and I could not really make sense of the results. 
It is also not clear if the overall performance is due to the using more informative labels to generate positive and negative pairs? 

clarity,_quality,_novelty_and_reproducibility: I find the paper not clear and not easy to read and follow, given lots of vague sentences and definitions, I find it hard to reproduce. 
As I understand the main novelty of paper is to use artificial fixation and saccades to generate negative and positive samples, which I find then the novelty being very marginal. 






summary_of_the_review: I think using human gaze data could be a potential use case in SSL approaches, and the idea of the paper is interesting and promising for future research. However, this paper is not written clearly, and I personally find it very hard to read and follow.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a new self-supervised learning model for video representation by introducing the idea of saccades/fixations. More specifically, a semantic change aware contrastive learning is proposed such that a positive pair is from the same fixation region of the same video, followed by reorganization of prototypical contrastive learning. Further more,  semantic consistency learning module is also included to encourage the consistency between the states of two time points within the same fixation, by minimizing the prediction error of using earlier state to predict later state.

Experiments are conducted for both video retrieval and video action recognition, on UCF101 and HMDB51 data sets. 
strength_and_weaknesses: Strengths:

--The idea of in-cooperating saccades/fixation into SSL video representation learning is very interesting. 

--And the proposed model contain some novelty, e.g, contrastive learning with fixations.

--There are not only experiments for classification, but also retrieval, showing the advantages of the learned representations. And the experiments are conducted on two data sets, with comparison to multiple existing video SSL methods (including contrastive ones and non-contrastive ones), making it quite convincing.


Weakness

--Some technical part is not clear. For example, how are the 5 fixations created ? The authors only mention "simulate the pupil function of an ideal system" without any explanation in details. 

--Further more, for two different frames in the same video, how do we judge whether two fixations are the same, e.g., when two fixation masks are at the same location (top right corner), or two fixation masks contain the same object, or some other way?
clarity,_quality,_novelty_and_reproducibility: The idea is quite interesting and the proposed model has some novel ideas. However, the technical part is not very clearly presented.
summary_of_the_review: Overall, the proposed idea is interesting and novel to some extent. The experiment results are also encouraging. However, some key technical aspect is unclear, making it unconvincing, so I can not give high rating for now.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Inspired by the human visual perception that saccades usually indicate semantic changes in the visual receptive field, this paper proposes a self-supervised learning method by artificially generating masks that represent saccades, and learning through Semantic-change-aware contrastive learning, semantic consistency, and representation reorganization. The proposed self-supervised learning method shows
good performance on both video retrieval and action recognition tasks.
strength_and_weaknesses: The strengths of this work are as follows:
1. The bio-inspired self-supervised learning scheme is interesting and reasonable. Constructing artificial saccades rather than collecting data by eye-trackers is a clever and effective approach. This scheme is also proved by the experimental results.

2. The three parts of the proposed method are designed with reasonable motivation. There are also corresponding experiments in the next section to show the effect of each part.

3. The experiments are well-designed and enough to show the effectiveness of the proposed method.

I list several weaknesses of this paper, if revised, could make this work better:
1. Conducting experiments on larger-scale datasets can strengthen this work.

2. There is no experiment nor explanation of the spatial size and temporal length of the fixation mask. This size may have a great impact on the quality of self-supervised learning.

3. It is also good to discuss the use of pre-trained saliency/gaze prediction models in this self-supervised learning scheme, rather than the artificially generated saccades. Based on my knowledge, [A] is a related work that provides a gaze prediction model on videos consisting of the modeling of attention transition, which can be used as a simulation saccades.

[A] Huang et al, "Predicting Gaze in Egocentric Video by Learning Task-dependent Attention Transition", ECCV 2018.
clarity,_quality,_novelty_and_reproducibility: This paper is understandable and of good quality technically. 

There is also enough novelty in this work. 

I believe the paper does not contain enough details for reproduction. Especially, the generation of fixation maps and the simulation of saccades are not clear.
summary_of_the_review: I think this paper proposes a novel and effective self-supervised learning scheme for videos. The method is reasonable and experiments are well conducted. I don't see major weaknesses in this paper.

After the revision, I believe this work becomes clear and the experiments become more solid. Thus I am raising my recommendation to accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a scheme to improve self-supervised learning for videos. This is done by randomly generating synthetic “saccades” which are expressed as binary masks applied to the input images. This masking and a modified loss allows to better capture semantic changes. Furthermore, a loss for semantic consistency and a prototypical clustering method is proposed to further improve the method. The authors show consistent performance improvements for linear and fine-tuning evaluations of retrieval and action recognition, on the UCF101 and HMDB51 datasets.
strength_and_weaknesses: The proposed modules are relatively intuitive and as can be seen from the ablation studies (Tab. 3 and Tab. 4), the individual contributions seem to consistently improve performance as measured via top-1 retrieval acc. on UCF101.

Once it is understood, the idea of using “saccades” via masked inputs sounds reasonably sensible. However, the details of how it is done is not immediately apparent and the mentions of actual human saccades mis-leads the reader (as well as Fig. 1b). Further clarity in writing would be beneficial. No study of the no. of synthetic fixations is done - despite this procedure being introduced for the first time in this paper. The authors do not discuss whether real eye-tracking data could be used here, or whether pre-trained saliency models could be applied to the training process.

While the prototypical clustering method is reasonably well motivated, and Fig. 4 tells a meaningful story, one wonders why prototypical clustering was not built in as a more key component. That is, prototypes could have been updated every epoch, as is commonly done in methods such as DeepCluster and others.
clarity,_quality,_novelty_and_reproducibility: While the paper is reasonably easy to read, the detail regarding the simulation of saccades is not explained well despite it being a key part of the proposed idea.
summary_of_the_review: The paper proposes contributions that can be understood fairly well, and backs up these proposals with strong experimental results. While the individual contributions may have been well-inspired, the written description on “saccades” and “prototypical contrastive learning” could be improved somewhat.

Edit post rebuttal: I appreciate the authors' efforts to respond to all concerns raised by the reviewers. My questions are partly addressed, and the paper's consistent performance improvements warrant a recommendation for acceptance. However, after carefully reviewing the other review texts and the authors' responses, I will not change my rating from 6. This is because the following changes would be better incorporated by a thorough re-write of the submission: (a) factors such as no. of fixations, spatial size of the fixation mask were analyzed only during the rebuttal phase, resulting in a different set of optimal parameters to the ones used in the paper, (b) the UniSAL approach show improvements over the arbitrarily designed "synthetic saccades" and probably should have been used as the main method of guiding positive/negative generation.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors propose a new self-supervised learning model for video representation learning based on the idea of saccades with the goal of better capturing semantic changes across frames ("saccades usually indicate semantic changes in the visual receptive field", https://en.wikipedia.org/wiki/Saccade). To implement this idea, the authors express synthetic saccades as binary masks applied to the input images and apply prototypical contrastive learning coupled with a semantic consistency module. The authors evaluate the proposed method on several video retrieval and action recognition tasks where it achieves competitive performance both for fine-tuning and linear probes.

The reviewers appreciated the idea of using gaze data to inform a SSL method and the placement of this paper relative to existing work. Reviewers found that the submission was lacking: (1) A thorough empirical evaluation, both in terms of datasets and missing ablations (spatial size and temporal length of the fixation mask), comparison to pre-trained saliency/gaze prediction models, (2) Clarity and presentation of the technical aspects of the work. (3) Unclear, or at least, non-verifiable intuition that saccades actually imply semantic changes. (4) Novelty, if the method is primarily about a new way to generate positive and negative pairs -- this depends on the thorough empirical evaluation in (1). 

This is a borderline submission. Taking into account the rebuttal and the discussion, I feel that the empirical evaluation needs to be more through and a major revision is necessary for acceptance to ICLR. The authors should focus on resolving (1) given that it's the main contribution of the paper and the novelty (beyond the connection to biological mechanisms) is limited.
justification_for_why_not_higher_score: Major:
- Lacking a thorough empirical evaluation, both in terms of datasets and missing ablations (spatial size and temporal length of the fixation mask), comparison to pre-trained saliency/gaze prediction models, 
- Clarity and presentation of the technical aspects of the work. 
- Novelty, if the method is primarily about a new way to generate positive and negative pairs -- this depends on the thorough empirical evaluation.

Minor:
- Unclear, or at least, non-verifiable intuition that saccades actually imply semantic changes.
Decision: Reject