Bringing Saccades and Fixations into Self-supervised Video Representation Learning
Authors: Qiuxia LAI
Ailing Zeng
Ye Wang
Lihong Cao
Qiang Xu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Self-supervised learning
video self-supervised learning
bio-inspired

Abstract: In this paper, we propose a self-supervised video representation learning (video SSL) method by taking inspiration from cognitive science and neuroscience on human visual perception. Different from previous methods that mainly start from the inherent properties of videos, we argue that humans learn to perceive the world through the self-awareness of the semantic change or consistency in the input stimuli in the absence of labels, accompanied by representation reorganization during the post-learning rest periods. To this end, we first exploit the presence of saccades as an indicator of semantic change in a contrastive learning framework to mimic the self-awareness in human representation learning, where the saccades are generated without eye-tracking data. Second, we model the semantic consistency by minimizing the prediction error between the predicted and the true state of another time point during a fixation. Third, we later incorporate prototypical contrastive learning to reorganize the learned representations such that perceptually similar representations would be associated closer. Compared to previous counterparts, our method can capture finer-grained semantics from video instances, and the associations among similar ones are further strengthened. Experiments show that the proposed bio-inspired video SSL method significantly improves the Top-1 video retrieval accuracy on UCF101 and achieves superior performance on downstream tasks such as action recognition under comparable settings.

Under review as a conference paper at ICLR 2023
BRINGING SACCADES AND FIXATIONS INTO SELF- SUPERVISED VIDEO REPRESENTATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper. we propose a self-supervised video representation learning (video SSL) method by taking inspiration from cognitive science and neuroscience on human visual perception. Different from previous methods that focus on the in- herent properties of videos, we argue that humans learn to perceive the world through the self-awareness of the semantic changes consistency in the input stimuli the absence of labels, accompanied by representation reorganization during the post-learning rest periods. To this end, first exploit the presence of saccades as an indicator of semantic changes in a contrastive learning framework, mimicking the self-awarenes, in human representation learning. The saccades are generated artificially without eye-tracking data. Second, we model the semantic consistency in eye fixation by minimizing the prediction error between the pre- dicted and the true state of another time point. Finally, we incorporate prototyp- ical contrastive learning to reorganize the learned representations to enhance the associations among perceptually similar ones. Compared to previous video SSL solutions, our method can capture finer-grained semantics from video instances, and the associations among similar ones are further strengthened. Experiments show that the proposed bio-inspired video SSL method significantly improves the Top-1 video retrieval accuracy on UCF101 and achieves superior performance on downstream tasks such as action recognition under comparable settings.
INTRODUCTION
Learning without labels the most common way for humans to get to know the world (DiCarlo al. 2012), and it has also been widely studied in machine learning for developing intelligent agents. In particular, many researchers focus on self-supervised learning (SSL) from dynamic visual input data, i.e., videos (Hurri & Hyvärinen 2003 Mobahi et al. 2009 Srivastava et al., 2015), which comes closest to the natural data perceived by humans. Recently, deep learning based video SSL methods have also shown superior performance over traditional non-deep learning methods (Wang et al., 2021a; Duan et al.] (0022.. However, there is still a large room for improvement considering the gap between the unsupervised learning abilities of deep learning models and humans.
One notable difference between deep video SSL methods and human unsupervised learning is that the former typically learn discriminative representations by considering the inherent data properties, such as the clip order (Misra et al., 2016)) the spatiotemporal coherence (Vondrick et al. 2018) the transformations exerted (Jenni al. 2020), etc., and propose various pretext tasks accordingly. While for humans, the self-awareness of the semantic change or consistency in the input stimuli is essential for learning without labels (Melcher Colby 2008). Besides, the encoded representations in the brain are not left unchanged but kept being reorganized to yield a representation structure with strengthened associations among perceptually similar representations (Diekelmann & Born, video
2010). Fig. [ shows overall comparison. This discrepancy inspired to propose SSL method by taking inspiration from cognitive science and neuroscience on human visual percep- tion. Recently, Illing et al. (202T) proposed a bio-inspired unsupervised learning rule that treats the presence of saccades as a global synaptic modulator. However, is less powerful due to the inherent difficulty in optimizing deep networks with layer-wist wise optimization.
Human visual perception is mainly accomplished by alternating saccade and fixation when the heads relatively still. The former is the rapid foveal motion from one target of interest another,

---

Under review as a conference paper at ICLR 2023
Spatiotemporal coherence
Predicting future Trames
Order
Speed
Spatiotemporal
puzzles
Foreground
Transformations
Spatiotemporal
Background
statistics
Saccade
Motion
Tracking
Semantic change
(a)
Video
(D)
Figure I: Overall comparison. (a) Previous video SSL methods design pretext tasks based on the inherent properties videos, while (b) our method explores the presence of saccades as an indicator of semantic change mimic the role of self-awareness in human perception. Since it is relatively expensive to collect real gaze data, we construct artificial saccades for training ($3.1)
while the latter is the period where the eye is kept aligned with one target for processing visual details. To capture the semantic change in the video, we propose exploit the presence of saccades as an indicator of the semantic change and propose a semantic-change-avee framework. This inspired by the fact that the human would perform a saccade when semantic change occurs in the fixation area. Specifically, the positive pairs are formed by features of the same fixation location in a video, and the negative pairs are formed by features different fixation locations in the same video or features from different videos. Compared to previous contrastive- based video SSL methods, our method captures finer- greiied semantics within the same video. Note that we manually construct saccades by exerting different fixation masks the input without using real gaze data, making our method general one for any video data without extra supervision.
contrastive learning
We further encourage the semantic consistency within fixation duration by minimizing the predic- tion error (PE) when using the current state to predict that of another time point in the same fixation duration. In this way, PE can serve as an extra supervision signal to avoid semantic discrepancy during semanti--chang--aaar contrastive learning. This is also biorational, as PE is known as an important modulator in perception, attention, and motivation control (Den Ouden et al. 2012).
To enhance the association among the previously learned finer mear-erained semantics, inspired by the reorganization human representation learning (Diekelmann Born 2010), we incorporate pro- totypical contrastive learning (Li et al. 2020) to gradually redistribute the representations. The learned representations are pulled towards their corresponding prototypes and pushed away from other prototypes. Such post-learning reorganization facilitates grouping unseen input stimuli into meaningful categories based on similarity, which leads to improved Top-I retrieval accuracy com- pared with previous contrastive-based video SSL methods.
In summary, we propose a video SSL framework by taking inspiration from cognitive science and neuroscience human visual perception. We first exploit the presence of saccades of semantic change in a contrastive learning framework for modeling the role of self-awareness in human representation learning. Then, model the semantic consistency in the input by minimiz- ing PE between a predicted and the true states of different time points during a fixation. Third, we incorporate prototypical contrastive learning reorganize the learned representations such that the associations among perceptually similar ones would be strengthened after redistribution. Ex- periments show that the proposed bio-inspired video SSL method significantly improves the Top-1 video retrieval accuracy on UCF101, and achieves superior performance on downstream tasks such as action recognition. The code and the pre-trained models will be released.
indicator
2 RELATED WORK
Self-supervised learning has been studied with various data formats, including image (Wu et al.
2018 Grill ct al. 2020 He et al. 2020; Li et al. (2020), video (Xu et al. 2019 Benaim et al. 22200]
Qian et al. 202 Duan al. 2022). and multi-modal data (Alayrac et al. 2020 Patrick et al.
2021 In this section, we focus on self-supervised representation learning on videos.
Non-contrastive video SSL methods. Previous video SSL methods mainly learn discrimina- tive representations by designing various pretext tasks based on the analysis of the inherent spa- tiotemporal properties of video data. The pretext tasks include figuring out the correct order of
2

---

Under review as a conference paper at ICLR 2023
the clips (Misra al.,
2016 Fernando et al.
2017 Lee et al.
2017 7 Wei et al.; 2018 Xu et al.
2019). tracking contents across ad ad aceett frames
Wang & Gupta
2015 Vondrick et (001 Pathak
et al.
2017| 2 Vondrick et al. 2018 Wang et al.
2019b), studying foreground and background robust-
ness
[uu et al. 2017. Wang et al. [202 e,
Ding et al. 2022), predicting future frames (Vondrick)
et al.
20T6 Luo al. 2017] Villegas et al.
2017
Han et al. 2020a Behrmann al.] 2021). solving
spatiotemporal puzzles (Kim et al. (2019) or video cloze (Luo et al.
(020), learning the spatiotem- 202 recognizing transformations exerted on
poral statistics of the videos
Wang et al.
2019a
the video (Jenni et al. 2020; Duan et al. 2022) or determining whether video is played at the
intrinsic speeds (Benaim et al. [020 Wang al. 2020 Yao al. 2020 Chen et al. 2021).
Contrastive video SSL methods. Another line of research adapts spatiotemporal properties into the contrastive learning framework (Hadsell et al. (006) by constructing the positive and negative pairs based on various spatiotemporal cues. More specifically, some methods extend the instance discrim- ination methods in image SSL (Wu et al.] 2018), and directly use clips randomly sampled from the
same video as positive pair (Han et al. 2020b Lin et al. 2021 Pan et al. 2021 Qian et al. 202 Ib Yao et al. 1). Some methods consider the spatiotemporal consistency of video and treat the predicted and the anoundd-ruthh features at the same spatiotemporal location
positive pair (Han
et al., 2019] 2020a). Some methods relate video understanding to pace reasoning ability, and con- struct positive pairs by sampling clips with different sampling rates from the same video sampling clips with the same pace from different videos (Huang et al.] (021). Some methods construct the positive pair from both frame-level and video-level representations (Kong et al. 2020 Kuang al. 2021). Some others construct positive pairs through spatiotemporal data augmentations (Qian ct al.
2021b Sun et al. 202T), or exploit motion information for data augmentation (Dwibedi et al. 2019
Li et al. 2021] || Wang et al. 2021b). Our method also belongs to this line of research. However, we
construct positive and negative pairs based on the semantic change indicated by saccades, resulting in finer-grained distinction of the semantics from the same video instance.
Comparison with CLAPP. Both the CLAPP model (Ilingg et al., 202T) and our work exploit the self-awareness of saccades for self-supervised representation learning, but we differ significantly in several aspects. First, CLAPP aims to propose a local learning rule for building deep representations without back-propagation, where each layer is trained independently to predict whether saccade happens. While ours utilizes the presence of saccades to construct the positive and negative pairs in contrastive learning framework for end-to-end learning. Second, for each layer, CLAPP restricts the predictions to be similar to its responses to future inputs and as different as possible from its responses to fake inputs. However, our method minimizes the discrepancies between the predictions and the future responses in the absence of a saccade for semantic consistency modeling. Third, besides constructing a saccade by switching the network inputs from one video to another as in CLAPP, we also consider inter-video saccades, which are realized by intentionally changing the fixation area on the same video to capture finer-gran iner-grained semantics. As shown in U improve over CLAPP for the video recognition task on UCF101.
we significantly
3 METHOD
As shown in Fig. 2 our method consists of three parts. First, explore saccades as the indicator of semantic change in a contrastive learning process (3..1). where the negative pairs consist of features before and after a saccade occurs. Second, we model the semantic consistency during fixation by minimizing the prediction eтror (PE) between the predicted and the true features of different time points ( (.2).. Third, we perform post-learning reorganization to strengthen the associations among perceptually similar representations ($ 3.3
3.1 CAPTURING SEMANTIC CHANGE VIA CONTRASTIVE LEARNING
Preparing saccades. The ground grouh gaze data are generally collected using eye- eye-tracking devices, which requires a lot of manual effort, and usually has personal heterogeneity considering the exact fixation locations. To mitigate these problems, we propose to construct artificial saccades, and only consider a representative set of coarse fixation locations. We simulate the receptive field of the fovea by exerting fixation masks ) on the input stimuli. The artificial saccades are constructed by intentionally alternating the fixation masks. To balance the performance and efficiency, we set N2 = 5. The spatial size of fixation area is 1/4 of the whole input as shown in Fig. (a), and the temporal length is the same as the clip length. More details are presented in A.
N₂
3

---

Under review as a conference paper at ICLR 2023
Sapcade
Fixation l
Fixation
Semantic Consistency Learning
Input (shifted)
Backbone
Eypel
X
Faxation mask
Predictor
m2g
Cell (itt pred
(c Any
Push
Input (current)
Backbone
Video dip features
Semantic Change Learning
Reorganization
Prakkepestaamnii
Figure 2: Framework of the proposed bio-inspired video SSL method. Given video x,, the positive and negative pairs for caaaaatii-ccane-aaaaaa contrastive learning can be constructed by exerting the same different fixation masks m different clips of the video ($3.I). respectively. Further, the semantic consistency is modeled by minimizing the prediction error between the pre- dicted state ( and the true state of another time point c ce++Ar ($3.2). ). Finally, the learned representations reorganized through prototypical contrastive learning process strengthen the association among perceptually similar ones (3.3
Semantic-change-aware contrastive learning. To capture the semantic change the presence of a saccade, we try to make the representations before and after saccade distinct from each other. To this end, we treat the representations before and after a saccade as negative pairs, and otherwise positive pairs. The latent feature space is thus optimized by minimizing the contrastive loss.
Formally, given training set X (x1, X xN) of N videos without category labels, we aim .....
to learn embedding function fe to map X to features V (V1, V....., VN h. where v, = fe (x,) and v, E R is expected to capture the semantics of x, The semantic-change-awarr contrastive loss inspired by InfoNCE (Oord et al. 2018 He al. 2020), and is calculated as follows:
exp(vix V/R/T)
Let
log
I N Noeg
exp(va V/H/T)+ + Ejez_ exp(vi V/n/T))
Here, fo(x m2) positive sample for VsA: fo(x; me) (O is Hadamard product), x/ obtained by applying commonly-used data augmentations on x;, V/a 7 VV' is negative sample, I_ is the set of indices for Nung selected negative samples, and k. h E (1, N......,,,,p are indices of the mask m. Note that a sample considered positive and only if j j= and h h= i.e., they are from the same fixation region of the same video. By minimizing Eq. [ the embedding function fo is trained to distinguish hetween finer-grained semantics within video. Perceptually similar finer-grained semantics will be further associated together through reorganization process.
Memory bank. As previously revealed in (Oord al. 2018 Wu al. 2018), a large number of negative pairs is essential for training InfoNCE loss, which is typically restricted by the batch size.
To alleviate this issue, we follow (Wu et al. 20118) and maintain a memory bank (=(v N+N, N+N, for different fixation locations of all the videos in the training dataset. Here, N is the number of training videos, and Np is the number of masks. Similar to WWu et al. 2018), we initialize with random D-dimensional unit vectors and update the slot V. with the latest feature v, follows:
V
m)V; + mV;
where E [0, is momentum value. With V, we can rewrite the contrastive learning procedure and Eq. II) by replacing the negative samples v' by their memory bank representations v.
3.2 MODELING SEMANTIC CONSISTENCY VIA MINIMIZING PE
To encourage the semantic consistency between the states of time points within the a fixation, given video X,, minimize the PE when using of to predict eh- where of E RCI((HHW.)
4

---

Under review as a conference paper at ICLR 2023
E
x
attention weight computation a(+,
Rred
value transform
O
g(+)
Figure 3: Method details. (a) Exemplar fixation masks for N, 5, which the major region in the visual field. See 13.1) for more details. (b) Prediction module introduced in $3.2), which is designed as a variant the self-attention module to capture the spatiotemporal correlations.
reshaped from the I-th level feature from the backbone fo whose size is Cj C H, x Wi, and Ci, Ti, H1, W are the feature channel number, the temporal resolution, the height, and the width, respectively. Note that is the time point of clip from x,, and AL can be either positive or negative.
The predictor module Pe is designed as variant of the self-attention module (Bahdanau et al, 2015 Xie al., [202]) capture the spatiotemporal correlations. It first obtains the spatiotemporal weights based on the element-wise cosine similarity, and then calculates each predicted element as the weighted sum of the transformed version of all other spatiotemporal elements. An illustration
shown Fig. (b). The u-th element of the predicted feature (c² At Ar is calculated as:
THHW Wi
(c Atypedd
Σ
a(ciuu clu) g(clu),
(3)
e=1
where a(:,) calculates the attention weights as: a(c/u. c(..) ReLU(cos ci,)),
(4)
and the transform function 9(.) is a linear layer that maps Ci-dim input to a Ci-dim output. The semantic consistency optimized by minimizing the prediction loss defined as follows: TiH:W;
Lyred
T.H.W, Σ
1+At 50
(c
1+ (1(+5t)pod)) prod
(5)
3.3 REORGANIZING VIA PROTOTYPICAL CONTRASTIVE LEARNING
Inspired by the gradual redistribution and reorganization of memory representations during the post- learning rest periods (Diekelmann & Born. 2010), after training the whole framework with La (Eq. and Cpedd (Eq. (5) to converge, further perform reorganization through prototypical contrastive learning (Li et al. 2020) to strengthen the associations among similar representations.
Specifically, we first cluster the representations {va) for R times to obtain R distinct clustering sults (G(r) (. where (r) contains ((r) clusters. Then, randomly pick Q = =min(Q(r), N
clusters from each (r)) to form G(r)"= = [G fg=1- where the centroid of G is (r)) The reorga-
(
nization loss calculated follows:
N
R
exp(vi* o6 /
Greeeg
log
6
N
R
Ey=o exp(V o /0/"//6/11) /c)(r)))
k=
Here, ((r) is the cluster to which Vik is assigned, !! is the centroid of G(") and d!d denotes
(r)
the concentration estimation of the cluster G(T)) as (Li et al. 2020). this way, the associa- tions among previously learned finer-grained semantics are further strengthened, which can facilitate
similarity- based categorization for unknown input stimuli.
3.4
LOSS FUNCTION
The overall loss function is combination of the three loss terms introduced above:
C La + Lped + Froog.
(7)
where the third term only takes effect the late stage of the training process.
5

---

Under review as a conference paper at ICLR 2023
Table I:
Video retrieval results on UCF101 and HMDB51. All the models are pre-trained on
UCF101 Larger values are better. See 442 for details.
UCFI01
HMDB51
Method
Backbone
Top-1
Top-5
Top-10
Top-20
Top-1
Top-5
Top-10
Top-20
VCOP
R(2+1)D
25.9
35.4
47.3
5.7
19.5
30.7
45.8
VCP
R(2+1)D
19.9
33.7
42.0
50.5
6.7
21.3
32.7
49.2
PRP
R(2+1)D
20.3
34.0
41.9
51.7
8.2
25.3
36.2
51.0
Pace
R(2+1)D
25.6
42.7
51.3
61.3
12.9
31.6
43.2
58.0
STS
R(2+1)D
38.1
58.9
68.1
77.0
16.4
36.9
50.5
65.4
Non-contrastive
VCOP
R3D-18
14.1
30.3
40.0
51.1
7.6
22.9
34.4
48.8
video SSL methods
VCP
R3D-18
18.6
33.6
42.5
53.3
7.6
24.4
36.3
53.6
PRP
R3D-18
22.8
38.5
46.7
55.2
8.2
25.8
38.5
53.3
Pace
R3D-18
23.8
38.1
46.4
56.6
9.6
26.9
41.1
56.1
STS
R3D-18
38.3
59.9
68.9
77.2
18.0
37.2
50.7
64.8
TransRank
R3D-18
46.5
63.7
72.8
19.4
45.4
59.1
MemDPC
R18
20.2
40.4
52.4
64.7
7.7
25.7
40.6
57.5
Contrastive video
MLRep
R3D-18
39.6
57.6
69.2
78.0
18.8
39.2
51.0
63.7
SSL methods
R3D-18
48.5
58.6
65.3
72.1
17.6
35.7
51.4
65.5
Ours
R(2+1)D
47.6
58.8
66.2
74.6
19.0
39.7
54.2
82.8
EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
Datasets. We conduct experiments on two representative video datasets, namely UCF101 (Soomro
et al. 2012) and HMDB51 (Kuehne et al. 2011). UCF101 consists of 13K videos of 101 action classes. HMDB51 contains 7K videos from 51 action classes. Both UCF101 and HMDB51 have three official train/test splits. We pre-trained on the first train split of UCF101 and used the first train/test split of UCF101 and HMDB-51 for evaluation following (Wang al., 2020 Qian et al, 202 For the ablation study, use the first train/test split of UCFI01.
Network architectures. We experiment on two backbones that are commonly used in previous video SSL methods, namely R3D-18 (Hara et al. 2018) and R(2+1)D-18 (Tran al. 2018). Given Np randomly sampled 16-frame clips of resolution 112 112 from a video, the backbone outputs Np D-dim feature vectors, where D = 512, and Np 5 is the number of types of fixation masks. We sample Np clips for video and ensure that all the N₂ memory slots for the video can be updated.
Pre-training. The batch size for R3D is 16, and the batch size for R(2+1)D is 14. The two backbones are trained for 300 epochs on the UCF101 training set using SGD with a momentum of 0.9 and weight decay of 10-4 The initial learning rate is 0.1 and is decayed by 5 at epoch 90, 180, and 240. The number of negative samples Neeg 1024. For PE minimization, we set /= 5, i.e., the state oa from the 5-th level of the backbone network, which shows the best balance between accuracy and efficiency. For reorganization, we use an unsupervised clustering algorithm Faiss (Johnson et al., 2019), and set R R=3, ((+) 1500 for r= R. and Q = 1024 based on ablation study. We train for another 60 epochs after incorporating Crong using SGD with learning rate of 8 8-4
Action recognition. We initialize the backbone using the model parameters obtained in the pre- training stage except for the last linear layer. We consider
settings: linear probe, where only the last linear layer is trained with cross-entropy loss, and ii) finetune, where the entire network finetuned with cross-entropy loss. For linear probe. when training on UCF101, we use SGD optimizer and trained for 200 epochs with an initial learning rate of 0.1, which is further decayed by 10 epoch 60, 120 and 180. For HMDB51, we use Adam with a learning rate of 0.001 and trained to converge. We use batch size 32 for both R3D and R(2+1) 1)+ backbones with input resolution 1 112x 112. For finetune, we use an SGD optimizer and trained for 200 epochs with a large initial learning rate of 0.1 following (Duan et al. 2022),, which is further decayed by 10 epoch 60, 120, and 180. The batch size is 32 for both backbones. During training, we apply the same data augmentation uniformly sample 10 clips from testing video,
in (Han et al., 2020b). For evaluation, perform the center crop, resize them to 112x 112, and average the predicted probabilities as the final prediction, following (Wang et al. 2020 Qian et al. 2021a).

---

Under review as a conference paper at ICLR 2023
0.3
02
0.1
Figure 4: Top-1 retrieval accuracy comparison before and after reorganization UCF101. The reorganization process can improve the Top-1 retrieval performance by 57.4% out of all the classes, leading comparative performance of 22.8% out of all. See for for more discussions.
Query
Top-1 NN Before Rearg.
Top-1 NN After Reang.
Jugginguslls
Juggingusile
Plyyingiielo
PlayngSiar
Playingoolo
Portmaloorse
Parte
Figure 5: Qualitative results for video retrieval before and after reorganization. The first two examples are improved by reorganization, and the third example a failure case, where the retrieval result after the reorganization is more similar in appearance while different in semanties. See $4
4.2 COMPARISON WITH THE STATE-OF TRE
To evaluate the representations learned in the self-supervised pre-training stage, we perform video retrieval and action recognition and compare them with other state-of-the-ee methods.
Video retrieval. We query the k-nearest neighbors of the testing set video clips from the training set. The pre-trained backbone model (R3D or R(2+1)D) directly used feature extractor without further finetuning. For each query video, we obtain one 512-d feature vector by extracting and averaging the features of 10 uniformly sampled clips. We use cosine similarity to measure the distance of the features for determining the k-nearest neighbors. A correct retrieval is counted when the & nearest neighbors contain at least one video of the same class with the query video. We report Top-k retrieval accuracy in Table [ where & 1,5, 10, 20, and compare with other video SSL methods pre-trained on the RGB modality of UCF101, including VCOP (Xu et al. 2019). VCP (Luo) al. 2020 PRP Yao al. 2020), Pace (Wang et al.] 2020), STS (Wang et al.] 2021a), and TransRank Duunn et al. 2022). ) As can be observed, our method achieves the best Top- I retrieval accuracy on UCFI01, and comparable Top-1 performance on HMDB51. Though our method is slightly inferior to other Top-k values, we argue that the Top-1 metric matters the most in security- demanded real-world applications such as autonomous driving. Primates including humans are also more good at giving out one promising guess than several candidates (Freedman et al. 2001).
Compared to previous methods, our method captures finer-granee semantics, which are reorganized to yield a better representation structure. To see the change of the learned finer-grained semantics before and after reorganization, we make statistics on the Top-1 retrieval accuracy before and after reorganization UCF101 regarding the 101 categories. As shown in Fig. 4 the post-learning reorganization improves the Top-1 retrieval accuracy for 57.4% of the 101 action categories. For 22.8% categories, the Top-1 retrieval accuracy remains the same. This shows that the redistributed finer-grained semantics can facilitate similarity based categorization.
We further visualize video retrieval examples that corrected by reorganization conversely in query clip,
F The first two examples are rectified by reorganization. For the it retrieves BoxPunch ngBag clip with similar foreground and background color, iee, person
7

---

Under review as a conference paper at ICLR 2023
Table 2: Action recognition results on UCF101 and HMDB51. The models are pre-trained on
UCF101 with RGB only. means learning without back-propagation. See $4.2 for details.
Method
Buckbone
Input Size
Frozen
UCFI01
HMDB51
CLAPP [iling al.
2021
VGG-5
41.6
Object Patch (Hudsell
2006
AlexNet
42.7
15.6
Sey Ver. MMisra ct al. 2016
CaffeNet
50.9
19.8
OPN (Lee 2017
CaffeNet
xx
56.3
22.1
Geumetry Gan
2018
FlowNet
54.1
22.6
Bi-Geometry (Gan ct al.
2018
FlowNet
55.1
23.3
Ours
R3D-18
16x
59.2
32.0
VCOP (Xu
2019
R3D-18
16x112
64.9
29.5
PRP
Yão et al.
2020
R3D-18
16x
X
66.5
29.7
STS
Wang
202 la
R3D-18
16x1122
70.4
34.9
VCP
Luo al.
2020
R3D-18
16x1122
66.0
31.5
Pace
Wang et al,
2020
R3D-18
16x
X
77.1
36.6
MIRep Qian
2021a
R3D-18
16x
76.2
41.1
Ours
R3D-18
16x1122
76.6
VCOP Xu al. 2019
R(2+1)D
16x112
72.4
30.9
PRP
Yau al. 2020
R(2+1)D
16x1122
72.1
35.0
STS
Wang et al. 2021a
R(2+1)
16x1122
77.8
40.7
VCP VCPoLuu ot al.
2020
R(2+1)D
16x
66.3
32.2
Pace Wang
2020
R(2+1)D
16x1122
75.9
35,9
Ours
R(++1)
16x
78.1
44.8
black and red background. For the PlayingCell query clip, retrieves P Playts payinggi ioggitan clip that resembles in both appearance and motion. By representation reorganization, such ambigu- ousness are removed, and clips with the correct action labels are found. However, we also notice that, for clips with complicated backgrounds, such PommeiHorse in the third row of Fig. reorganization tends to make easier to be mixed up with clips having similar appearance.
Action recognition. We report the Top-1 action recognition accuracy of linear probe (Frozen V) and finetune (Frozen X) in Table 2] For a fair comparison, we exclude methods based on much deeper backbones, with larger input resolution, using multi-model data, pre-trained much larger video datasets such as Kinetics (Carreira Zisserman, 2017). For linear probe experiments, method outperforms all previous video SSL methods pre-trained UCF101, especially for CLAPP (Iling et al.) (021). a bio-inspired unsupervised representation learning method without back-propagation. This clearly demonstrates that our method is a competitive practice of exploring cognitive inspirations in deep self-supervised representation learning. For the finetune setting, our method achieves the highest Top-1 accuracy compared with other video SSL methods pre- pre-trained on the RGB modality of UCF101, showing a good generalization ability of the learned representations.
4.3 ABLATION STUDY
In this section, assess the effectiveness of the framework design regarding three components: the cemantic-change-avee contrastive learning that utilizes saccade as an indicator ($3.1). the seman- tic consistency learning by minimizing the prediction error (PE) 3.2). and the reorganization via prototypical contrastive learning ($3.3) We report Top-1 video retrieval accuracy on UCF101 to evaluate the learned video representations without further finetuning.
Overall framework design. As shown in Table although increasing the size of the memory bank benefits the retrieval performance, major improvements are brought by incorporating the three com- ponents. Specifically, incorporating saccades for constructing negative pairs achieves an absolute improvement of point, and further including PE minimization or post-learning reorganization leads to an absolute improvement of 0.8 and 2.4, respectively. The full model is 4.3 point higher than the baseline with a memory bank the same size. The results clearly demonstrate the effec- tiveness of each framework design.
Alternatives of artificial saccades. Besides constructing artificial saccades for training, we tried to incorporate real gaze data in scheme by resorting current video saliency prediction datasets such as DHFIK [Wang et al. 2018), Hollywood-2, and UCF sports (Mathe & Sminchisescul (2015). However, those datasets typically contain no more than 1.5K training videos, which much
8

---

Under review as a conference paper at ICLR 2023
Ablation study on framework design. We report Top-1 video retrieval re- sults on UCF101 split Here, N is #train- ing samples and N₂ is #fixation locations considered. See 4.3 for details.
Table 3:
Table 4: Ablation study on reorganization pa- rameters assessed by Top-1 retrieval accuracy on UCFI01 split 1. Here, R is #cluster results, and Q is #clusters in the r-th result. denotes updat- ing prototypes every epoch. See $4.3
No. Succude Pred. ErT. Reorg. Mem. Size
Top-1
No.
R Q
Top-1
No.
R
Qiii
Top-1
N
39.4
1
0
40.8
500
1000 1500
41.7
2
N N₂
42.5
2
1
500
41.5
8
1000
1000 1000
42.2
/
N Np
44.2
3
1
1000
42.1
1500
1500 1500
42.3
4
/
/
N No
45.0
4
1
1500
42.0
10
3
1000
1500 2000
41.8
5
/
N Np
46,5
5
1
2000
41.9
11
3 1500
1500 1500
41.0"
6
/
N Np
48.5
1
2500
41.3
12
1500 1500 1500
41.4"
smaller than video SSL datasets such as HMDB51 and UCFI01. Besides, the distribution of the videos in the saliency prediction datasets are typically not the same that of the videos used for SSL training. Thus, it is hard for our SSL method trained on video saliency prediction datasets to achieve competitive performance when evaluated on downstream tasks.
To mitigate the above-mentioned problems, we resort to saliency models pre-trained on real gaze data for egocentric (Huang et al. 2018) or third-person videos (Droste et al.) 2020), which are promising to provide an approximation of real gaze data. Considering that UCF101 contains third- person videos, we utilize UNISAL (Droste et al., 2020) to predict visual saliency maps for UCF101, and then use these maps to guide the construction of saccades during training. The fixation mask of a video clip is determined by the majority of the corresponding visual saliency maps. Since more than 90% of the resulted fixation masks are center masks, mitigate such distribution center bias, we randomly perturb the fixation mask labels with a probability of 0.5. To assess the effectiveness of the saccades, we train the baseline with saccades for 300 epochs on UCF101. The model achieves 45.1% 15. Top-1 retrieval accuracy on UCF101, which is slightly better than 44.2%, the Top-I accuracy of the one trained with artificial saccades as reported in Table 3 It is promising that a comparable amount of real gaze data would also bring in such benefits in our video SSL framework.
Reorganization parameters. To better reorganize the learned finer-grained semantics, we experi- ment on two key parameters introduced in $3.3] namely the number of cluster results Ił, the number of clusters in each result Q("), as well as the clustering frequency and the number of warmup epochs. All the baselines are trained on UCF101 for 100 epochs using SGD. The learning rate is 0.1 and is decayed by at epoch 30, 60 and 80. The results are shown in Table For the first ten baselines, the reorganization loss Crowgg is incorporated at epoch 61,, and the prototypes are updated every 5 epochs. For baseline 11 and 12, Lroong is introduces at epoch 2 and 61, respectively, and the prototypes are updated every epoch. As can be observed, reorganization can consistently improve the Top-1 re- trieval accuracy for a wide range of R and Q!+). However, it is recommended to start prototypical learning later when the finer- grained semantics are relatively better captured, with less frequent pro- totype updates. In our full experiments where the models are trained for 300 epochs, we set R = and Qir) 1500 for r 1l ... R. and update prototypes every 5 epochs since epoch 181.
CONCLUSION
In this work, we propose a video SSL method by taking inspiration from cognitive science and neuroscience on human visual perception. Instead of designing pretext tasks based on the inherent properties of videos, we explore the presence of saccades as an indicator of semantic change in contrastive learning framework to mimic the role of self-awareness in human perception. To achieve semantic consistency in the absence of a saccade, we minimize the prediction error when using the state of a time point to predict that of another time point during fixation. Finally, we strengthen the associations among similar representations through a post-learning reorganization process. Com- pared to previous contrastive learning based video SSL methods, our method learns more powerful representations by first making finer-grained distinctions for semantics in a video instance, and then associating similar semantics across different video instances through reorganization process. Se- mantic consistency between the states of two time points within the same fixation is encouraged by minimizing the prediction error of using the earlier state predict the later state. The proposed bio-inspired video SSL method achieves superior Top-1 1 video retrieval accuracy on UCF101 and outperforms other methods on the action recognition tasks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Lean-Baptiste Alayrac, Adria Recasens, Rosalia Schneider, Relja Arandjelovié, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Self-supervised mul- timodal versatile networks, NeurIPS, 33:25-37, 2020.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.
Nadine Behrmann, Jurgen Gall, and Mehdi Noroozi. Unsupervised video representation learning by bidirectional feature prediction. In WACV, pp. 1670-1679, 2021.
Sagie Benaim, Ariel Ephrat, Oran Lang, Inbar Mosseri, William T Freeman, Michael Rubinstein, Michal Irani, and Tali Dekel. Speednet: Learning the speediness videos. CVPR, pp. 9922- 9931, 2020.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In CVPR, pp. 6299-6308, 2017.
Peihao Chen, Deng Huang, Dongliang He, Xiang Long, Runhao Zeng, Shilei Wen, Mingkui Tan, and Chuang Gan. Rspnet: Relative speed perception for unsupervised video representation learn- ing. In AAAI, volume 1, 2021.
Hanneke EM Den Ouden, Peter Kok, and Floris De Lange. How prediction errors shape percep- tion, attention, and motivation. Frontiers in Psychology, 3:548, 2012.
James J DiCarlo, Davide Zoccolan, and Nicole C Rust. How does the brain solve visual object recognition? Neuron, 73(3):415 434, 2012.
Susanne Diekelmann and Jan Born. The memory function of sleep. Nature Reviews Neuroscience, 11(2):1141266, 2010.
Shuangrui Ding, Maomao Li, Tianyu Yang, Rui Qian, Haohang Xu, Qingyi Chen, and Jue Wang. merging.
Motion-aware self-supervised video representation learning via In CVPR, 2022.
Richard Droste, Jianbo Jiao, and Alison Noble. Unified image and video saliency modeling. In ECCV, pp. 419-435, 2020.
Haodong Duan, Nanxuan Zhao, Kai Chen, and Dahua Lin. Transrank: Self-supervised video repre- sentation learning via ranking- based transformation recognition. In CVPR, pp. 3000-3010, 2022.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Tem-
poral cycle-consistency learning. In CVPR, pp. 1801-1810, 2019.
Basura Fernando, Hakan Bilen, Efstratios Gavves, and Stephen Gould. Self-supervised video rep- resentation learning with odd-one-out networks. In CVPR, pp. 3636-3645, 2017.
David Freedman, Maximilian Riesenhuber, Tomaso Poggio, and Earl K Miller. Categorical repre- sentation of visual stimuli in the primate prefrontal cortex. Science, 29((5522)31122 2001.
Chuang Gan, Boqing Gong, Kun Liu, Hao Su, and Leonidas Guibas. Geometry guided convo- lutional neural networks for self-supervised video representation learning. In CVPR, pp. 5589- 5597, 2018
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernaro Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. NeurIPS, 33:21271- 21284, 2020.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. In CVPR, volume 2, pp. 1735-1742, 2006.
Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding. In ICCVW, pp. 0-0, 2019.
10

---

Under review as a conference paper at ICLR 2023
Tengda Han, Weidi Xie, and Andrew Zisserman. Memory-augmenteed dense predictive coding for video representation learning. ECCV, pp. 312-329, 2020a.
Tengda Han, Weidi Xie, and Andrew Zisserman. Self-supervised co-training for video representa- tion learning. In NeurIPS, 2020b.
Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet? In CVPR, pp. 6546-6555, 2018.
Kaiming He. Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pp. 9729-99999999 9738, 2020.
Deng Huang, Wenhao Wu, Weiwen Hu, Xu Liu, Dongliang He, Zhihua Wu, Xiangmiao Wu, Mingkui Tan, and Errui Ding. Ascnet: Self-supervised video representation learning with appearance-specd consistency. In ICCV, 2021.
Yifei Huang, Minjie Cai, Zhenqiang Li, and Yoichi Sato. Predicting gaze in egocentric video by learning task-dependent attention transition. In ECCV, pp. 754-769, 2018.
Jarmo Hurri and Aapo Hyvärinen. Simple-cell-like receptive fields maximize temporal coherence in natural video. Neural Computation, 15(3)6633-611, 2003.
Bernd Illing, Jean Ventura, Guillaume Bellec, and Wulfram Gerstner. Local plasticity rules can learn deep representations using supe-super contrastive predictions. NeurIPS, 34, 2021.
Simon Jenni, Givi Meishvili, and Paolo Favaro. Video representation learning by recognizing tem- poral transformations. In ECCV, pp. 425-442, 2020.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Trans. on Big Data, 7(3):535-547, 2019.
Dahun Kim, Donghyeon Cho, and So Kweon. Self-supervised video representation learning with space-time cubic puzzles. In AAAI, volume 33, pp. 8545 8552, 2019.
Quan Kong, Wenpeng Wei, Ziwei Deng, Tomoaki Yoshinaga, and Tomokazu Murakami. Cycle- contrast for self-supervised video representation learning. NeurlPS, 33:8089-8100 2020.
Haofei Kuang, Yi Zhu, Zhi Zhang, Xinyu Li, Joseph Tighe, Sören Schwertfeger, Cyrill Stachniss, and Mu Li. Video contrastive learning with global context. In ICCVW, pp. 3195-3204, 2021.
Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: 2011.
large video database for human motion recognition. In CVPR, pp.
Hsin-Ying Lee, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang. Unsupervised representa- tion learning by sorting sequences. In CVPR, pp. 667-676, 2017.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi. Prototypical contrastive learning of unsuper- vised representations. In ICLR, 2020.
Rui Li, Yiheng Zhang, Zhaofan Qiu, Ting Yao, Dong Liu, and Tao Mei. Motion-focused contrastive learning of video representations. In ICCV, pp. 2105-21 14, 2021.
Yuanze Lin, Xun Guo, and Yan Lu. Self-supervised video representation learning with meta- contrastive network. In ICCV, pp. 8239-8249, 2021.
Dezhao Luo, Chang Liu, Yu Zhou, Dongbao Yang, Can Ma, Qixiang Ye, and Weiping Wang. Video cloze procedure for self-supervised spatio-temporal learning. In AAAI, volume 34, pp. 11701- 11708, 2020.
Zelun Luo, Boya Peng, De-An Huang, Alexandre Alahi, and Li Fei-Fei. Unsupervised learning of long-term motion dynamics for videos. In CVPR, pp. 2203-2212, 2017.
Stefan Mathe and Cristian Sminchisescu. saliency models for visual recognition. Intelligence, 37(7):1408-11 1424, 2015.
Actions in the eye: dynamic gaze datasets and learnt IEEE Transactions on Pattern Analysis and Machine
11

---

Under review as a conference paper at ICLR 2023
David Melcher and Carol L Colby. Trans-saccadic perception. Trends in Cognitive Sciences, 12 (12):466 473, 2008.
Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In ECCV, pp. 527-544, 2016.
Hossein Mobahi, Ronan Collobert, and Jason Weston. Deep learning from temporal coherence in video. In ICML, pp. 737-744, 2009.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.037/444 2018
Tian Pan, Yibing Song. Tianyu Yang, Wenhao Jiang, and Wei Liu. Videomoco: Contrastive video representation learning with temporally adversarial examples. In CVPR, pp. 11205-11214, 2021.
Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Darrell, and Bharath Hariharan. Learning fea- 2017.
tures by watching objects move. In CVPR, pp.
Mandela Patrick, Yuki M Asano, Polina Kuznetsova, Ruth Fong, João F Henriques, Geoffrey Zweig, and Andrea Vedaldi. On compositions of transformations in contrastive self-supervised learning. In ICCV, pp. 9577-9587, 2021.
Rui Qian, Yuxi Li, Huabin Liu, John See, Shuangrui Ding, Xian Liu, Dian Li, and Weiyao Lin. Enhancing self-supervised video representation learning via multi-level feature optimization. In ICCV, pp.
2021a.
Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang, Huisheng Wang, Serge Belongie, and Yin Cui. Spatiotemporal contrastive video representation learning. In CVPR, pp. 6964-6974, 2021b.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint 121.:0222. 2012.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using Istms. In /CML, pp. 843 843-852, 2015.
Chen Sun, Arsha Nagrani, Yongiong Tian, and Cordelia Schmid. Composable augmentation encod- ing for video representation learning. In ICCV, pp. 8834-8844, 2021.
Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. closer look spatiotemporal convolutions for action recognition. In CVPR, pp. 6450-6459, 2018.
Po-He Tseng, Ran Carmi, Ian GM Cameron, Douglas Munoz, and Laurent Itti. Quantifying center bias of observers in free viewing of dynamic natural scenes. Journal of Vision, 9(7):444 2009.
Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lec. Decomposing motion and content for natural video sequence prediction. In ICLR, 2017.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Anticipating visual representations from unlabeled video. In CVPR, pp. 98-106, 2016.
Carl Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy. Track- ing emerges by colorizing videos. In ECCV, pp. 391-408 2018.
Jiangliu Wang, Jianbo Jiao, Linchao Bao, Shengfeng He, Yunhui Liu, and Wei Liu. Self-supervised spatio-temporey representation learning for videos by predicting motion and appearance statistics. In CVPR, pp. 4006-4015, 2019a.
Jiangliu Wang, Jianbo Jiao, and Yun-Hui Liu. Self-supervised video representation learning by pace prediction. In ECCV, pp. 504-521, 2020.
Jiangliu Wang. Jianbo Jiao, Linchao Bao, Shengfeng He. Wei Liu, and Yun-Hui Liu. Self-supervised video representation learning by uncovering spatio-temporal statistics. IEEE TPAMI, 2021a.
12

---

Under review as a conference paper at ICLR 2023
Jinpeng Wang, Yuting Gao, Ke Li, Xinyang Jiang, Xiaowei Guo, Rongrong Ji, and Xing Sun. En- hancing unsupervised video representation learning by decoupling the scene and the motion. In AAAI, 2021b.
Jinpeng Wang, Yuting Gao, Ke Li, Yiqi Lin, Andy J Ma, Hao Cheng, Pai Peng, Feiyue Huang, Rongrong Ji, and Xing Sun. Removing the background by adding the background: Towards background robust self-supervised video representation learning. In CVPR, pp. 11804-11813, 2021c.
Wenguan Wang, Jianbing Shen, Fang Guo, Ming-Ming Cheng, and Ali Borji. Revisiting video saliency: A large-scale benchmark and new model. In CVPR, 2018.
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In ICCV, pp. 2794-2802, 2015.
Xiaolong Wang, Allan Jabri, and Alexei A Efros. Learning correspondence from the cycle- consistency of time. In CVPR, pp. 3566-2576, 2019b.
Donglai Wei, Joseph J Lim, Andrew Zisserman, and William T Freeman. Learning and using the arrow of time. In CVPR, pp. 8052-8060, 2018.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. CVPR, pp. 3733-3742, 2018.
Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. In CVPR, pp- 16684-16693, 2021.
Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang. Self-supervised spa- tiotemporal learning via video clip order prediction. In CVPR, pp. 10334-10343, 2019.
Ting Yao, Yiheng Zhang, Zhaofan Qiu, Yingwei Pan, and Tao Mei. Seco: Exploring sequence supervision for unsupervised representation learning. In AAAI, volume 35, pp. 10656-10664, 2021.
Yuan Yao, Chang Liu. Dezhao Luo, Yu Zhou, and Qixiang Ye. Video playback rate perception for self-supervised spatio-temporal representation learning. In CVPR, pp. 6548-6557 2020.
A APPENDIX
In this section, we present more details of saccades preparation in SA.] and study more configura- tions on fixation masks in A.2
A.1 DETAILS OF PREPARING SACCADES
The presence of a saccade indicates semantic change in the stimuli that fall in the receptive field of the fovea (denoted as "fixation area" in our paper). In view of this, we proposed to manually determine such receptive fields using fixation masks. Thus, the change of the fixation mask, i.e., a "saccade" in our design, reasonably indicates semantic change of the stimuli in the fixation area, which inspires the design of our semantic-chane seaiii-hhhnneaawrrrr contrastive learning framework. The procedure for generating artificial saccades is detailed as follows.
We first determine the fixation locations by considering two aspects: i) the fixation locations are better to be evenly distributed since humans may attend to anywhere the scene, and ii) the central region of the input shall be covered considering the center bias in free-viewing visual saliency Teeng ct al. 2009). To balance the performance and efficiency, in our experiments, we divide the mage into 2! x grids, and take their centroids and the centroid of the entire image, which gives N₂ = locations in total. We also experiment on fixation masks which corresponds to fixation locations centered in the 3 3 grids, which is shown in A.2
Then, since the human eye can be viewed as an optical imaging system, we consider the generalized pupil function of ideal imaging system and design the fixation mask binary mask where the
13

---

Under review as a conference paper at ICLR 2023
675
10%
20%
2%
10%
13%
20%
25%
30%
35%
40%
45%
50%
207
(a)
Figure 6: Exemplar fixation masks of various spatial sizes for (a) N2 = and (b) N, = 9, respec- tively. See AA.I for more details.
Table 5: Ablation study on fixation mask configurations. We report Top-1 video retrieval results on UCF101 split 1. The best values boldfaced. The value of default settings underlined. See A.2 for more discussions.
Spatial size of fixation area
5%
11%
15%
20%
25%
30%
35%
40%
45%
50%
N, 55
43.3
44.7
44.3
44.4
44.2
45.0
45,9
44.8
44.9
44.8
N, 99
43.5
44.3
44.7
45.2
43.9
value is 1 at every point within the fixation area and 0 otherwise. The fixation area, i.e., the receptive field of the fovea, represented using a rectangle centered aspect ratio the input. In our experiments, we set the spatial size of the fixation area 25% of entire image size. We also explore the impact of different spatial sizes and numbers of fixations in A.2 Exemplar fixation masks are shown in Fig.
fixation location that has the same
Finally, we construct an artificial saccade by manually assigning two different fixation masks to two video clips, which guarantees that the two fixation masks before and after artificial saccade capture different visual receptive fields the input scene. The two different fixation masks are randomly picked from Ny pre-defined fixation masks.
A.2 FURTHER STUDY ON FIXATION MASKS
In this section, we study the effect of the spatial size of the fixation areas and the number of fixations Np used for constructing artificial saccades. We train the baseline with saccades only instead of the full model on UCF101 split 1 for 100 epochs. The Top-1 retrieval accuracy on UCF101 of N₂ = and N,=9 with various spatial sizes are show in Table
As can be observed, the Top-1 I retrieval performance first increases and then decreases as the spatial sizes of the fixation area becomes larger. The optimal spatial sizes are 35% and 20% for - and N, =9 respectively. This is because that the information fall in the visual receptive field increases with the fixation area, and more information is beneficial for representation learning. However, when the overlap among different fixation areas enlarges, unintentional perturbations would be in- troduced, which impedes finer graiied semantic learning. Thus it is crucial to determine the optimal overlapping for different N, values.
14


Reviews:
[{"review_id": "m1cAAfReFJ", "reviewer": "Reviewer_ucm9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a new self-supervised learning model for video representation by introducing the idea of saccades/fixations. More specifically, a semantic change aware contrastive learning is proposed such that a positive pair is from the same fixation region of the same video, followed by reorganization of prototypical contrastive learning. Further more,  semantic consistency learning module is also included to encourage the consistency between the states of two time points within the same fixation, by minimizing the prediction error of using earlier state to predict later state.\n\nExperiments are conducted for both video retrieval and video action recognition, on UCF101 and HMDB51 data sets. ", "summary_of_the_review": "Overall, the proposed idea is interesting and novel to some extent. The experiment results are also encouraging. However, some key technical aspect is unclear, making it unconvincing, so I can not give high rating for now. ", "strength_and_weaknesses": "Strengths:\n\n--The idea of in-cooperating saccades/fixation into SSL video representation learning is very interesting. \n\n--And the proposed model contain some novelty, e.g, contrastive learning with fixations.\n\n--There are not only experiments for classification, but also retrieval, showing the advantages of the learned representations. And the experiments are conducted on two data sets, with comparison to multiple existing video SSL methods (including contrastive ones and non-contrastive ones), making it quite convincing.\n\n\nWeakness\n\n--Some technical part is not clear. For example, how are the 5 fixations created ? The authors only mention \"simulate the pupil function of an ideal system\" without any explanation in details. \n\n--Further more, for two different frames in the same video, how do we judge whether two fixations are the same, e.g., when two fixation masks are at the same location (top right corner), or two fixation masks contain the same object, or some other way?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "by3Zuz5F6_O", "reviewer": "Reviewer_zkjU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a scheme to improve self-supervised learning for videos. This is done by randomly generating synthetic “saccades” which are expressed as binary masks applied to the input images. This masking and a modified loss allows to better capture semantic changes. Furthermore, a loss for semantic consistency and a prototypical clustering method is proposed to further improve the method. The authors show consistent performance improvements for linear and fine-tuning evaluations of retrieval and action recognition, on the UCF101 and HMDB51 datasets.", "summary_of_the_review": "The paper proposes contributions that can be understood fairly well, and backs up these proposals with strong experimental results. While the individual contributions may have been well-inspired, the written description on “saccades” and “prototypical contrastive learning” could be improved somewhat.\n\nEdit post rebuttal: I appreciate the authors' efforts to respond to all concerns raised by the reviewers. My questions are partly addressed, and the paper's consistent performance improvements warrant a recommendation for acceptance. However, after carefully reviewing the other review texts and the authors' responses, I will not change my rating from 6. This is because the following changes would be better incorporated by a thorough re-write of the submission: (a) factors such as no. of fixations, spatial size of the fixation mask were analyzed only during the rebuttal phase, resulting in a different set of optimal parameters to the ones used in the paper, (b) the UniSAL approach show improvements over the arbitrarily designed \"synthetic saccades\" and probably should have been used as the main method of guiding positive/negative generation.", "strength_and_weaknesses": "The proposed modules are relatively intuitive and as can be seen from the ablation studies (Tab. 3 and Tab. 4), the individual contributions seem to consistently improve performance as measured via top-1 retrieval acc. on UCF101.\n\nOnce it is understood, the idea of using “saccades” via masked inputs sounds reasonably sensible. However, the details of how it is done is not immediately apparent and the mentions of actual human saccades mis-leads the reader (as well as Fig. 1b). Further clarity in writing would be beneficial. No study of the no. of synthetic fixations is done - despite this procedure being introduced for the first time in this paper. The authors do not discuss whether real eye-tracking data could be used here, or whether pre-trained saliency models could be applied to the training process.\n\nWhile the prototypical clustering method is reasonably well motivated, and Fig. 4 tells a meaningful story, one wonders why prototypical clustering was not built in as a more key component. That is, prototypes could have been updated every epoch, as is commonly done in methods such as DeepCluster and others.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XrK-8HppSZ", "reviewer": "Reviewer_aMU3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Inspired by the human visual perception that saccades usually indicate semantic changes in the visual receptive field, this paper proposes a self-supervised learning method by artificially generating masks that represent saccades, and learning through Semantic-change-aware contrastive learning, semantic consistency, and representation reorganization. The proposed self-supervised learning method shows\ngood performance on both video retrieval and action recognition tasks.", "summary_of_the_review": "I think this paper proposes a novel and effective self-supervised learning scheme for videos. The method is reasonable and experiments are well conducted. I don't see major weaknesses in this paper.\n\nAfter the revision, I believe this work becomes clear and the experiments become more solid. Thus I am raising my recommendation to accept.", "strength_and_weaknesses": "The strengths of this work are as follows:\n1. The bio-inspired self-supervised learning scheme is interesting and reasonable. Constructing artificial saccades rather than collecting data by eye-trackers is a clever and effective approach. This scheme is also proved by the experimental results.\n\n2. The three parts of the proposed method are designed with reasonable motivation. There are also corresponding experiments in the next section to show the effect of each part.\n\n3. The experiments are well-designed and enough to show the effectiveness of the proposed method.\n\nI list several weaknesses of this paper, if revised, could make this work better:\n1. Conducting experiments on larger-scale datasets can strengthen this work.\n\n2. There is no experiment nor explanation of the spatial size and temporal length of the fixation mask. This size may have a great impact on the quality of self-supervised learning.\n\n3. It is also good to discuss the use of pre-trained saliency/gaze prediction models in this self-supervised learning scheme, rather than the artificially generated saccades. Based on my knowledge, [A] is a related work that provides a gaze prediction model on videos consisting of the modeling of attention transition, which can be used as a simulation saccades.\n\n[A] Huang et al, \"Predicting Gaze in Egocentric Video by Learning Task-dependent Attention Transition\", ECCV 2018.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "9Fa4nQCe1J", "reviewer": "Reviewer_MEj8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a self-supervised video representation learning that would take advantage of artificial cascades and fixation points mimicking human eye behavior.  Using the generated fixation and saccades, they could generate  positive and negative samples for each video frame that is used later in a contrastive learning approach to bring similar examples closer and negative ones further away. \n\n", "summary_of_the_review": "I think using human gaze data could be a potential use case in SSL approaches, and the idea of the paper is interesting and promising for future research. However, this paper is not written clearly, and I personally find it very hard to read and follow. \n\n\n\n", "strength_and_weaknesses": "The idea of using gaze data to improve the SSL is interesting and previously also has been shown promising results. The paper covered related work in depth and gave good explanation upon each work. \n\nHowever I have a hard time reading and following the paper, I find that the paper has some vague sentences or references to some papers with not clear justifications. Fro example on page 4 \"To\nthis end, we treat such representations as negative pairs and otherwise positive pairs and optimize\nthe latent feature space by minimizing the contrastive loss.\" I really have a hard time understanding such vague sentences.. \n\nI am not sure why presence of saccade should indicates a semantic change in the video, in real human gaze data we performs saccades to cover different parts of stimuli and it does not necessarily means a semantic change in the stimuli or the scene, as simply human eye fovea can not cover the whole scene in details as once.  Hence comparing the work to a biological inspired approach is over reaching as it has not been tested using real gaze data. \n\nI could not find the details how the fixation locations was generated, why 5 fixations were selected or what is the size of the mask, are these locations generated randomly? or a saliency based approach is used to generate fixations data? \n\nHow is the v_ik ' generated? What does it mean it is an augmented version of vik? \n\nOverall, I was not able to understand the paper fully as it was not written clearly and I could not really make sense of the results. \nIt is also not clear if the overall performance is due to the using more informative labels to generate positive and negative pairs? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject