Self-Supervised Video Representation Learning with Constrained Spatiotemporal Jigsaw
Authors: Yuqi Huo
Mingyu Ding
Haoyu Lu
Zhiwu Lu
Tao Xiang
Ji-Rong Wen
Ziyuan Huang
Jianwen Jiang
Shiwei Zhang
Mingqian Tang
Songfang Huang
Ping Luo
Venue:  (2021)
Keywords: self-supervised learning
video representation learning
spatiotemporal jigsaw

Abstract: This paper proposes a novel pretext task for self-supervised video representation learning by exploiting spatiotemporal continuity in videos. It is motivated by the fact that videos are spatiotemporal by nature and a representation learned to detect spatiotemporal continuity/discontinuity is thus beneficial for downstream video content analysis tasks. A natural choice of such a pretext task is to construct spatiotemporal (3D) jigsaw puzzles and learn to solve them. However, this task turns out to be intractable. We thus propose Constrained Spatiotemporal Jigsaw (CSJ) whereby the 3D jigsaws are formed in a constrained manner to ensure that large continuous spatiotemporal cuboids exist in a shuffled clip to provide sufficient cues for the model to reason about the continuity. With the constrained jigsaw puzzles, instead of solving them directly, which could still be extremely hard, we carefully design four surrogate tasks that are more solvable but meanwhile still ensure that the learned representation is sensitive to spatiotemporal continuity at both the local and global levels. Extensive experiments show that our CSJ achieves state-of-the-art on two downstream tasks across various benchmarks.

Under review as a conference paper at ICLR 2021
SELF-SUPERVISED VIDEO REPRESENTATION LEARN- ING WITH CONSTRAINED SPATIOTEMPORAL JIGSAW
Anonymous authors Paper under double-blind review
ABSTRACT
This paper proposes novel pretext task for self-supervised video representation learning by exploiting spatiotemporal continuity in videos. It is motivated by the fact that videos are spatiotemporal by nature and representation learned to detect spatiotemporal contiuity/disconiniity is thus beneficial for downstream video content analysis tasks. A natural choice of such a pretext task construct spa- tiotemporal (3D) jigsaw puzzles and learn to solve them. However, this task turns out to be intractable. We thus propose Constrained Spatiotemporal Jigsaw (CSJ) wherehy the 3D jigsaws are formed in a constrained manner to ensure that large continuous spatiotemporal cuboids exist a shuffled clip to provide sufficient cues for the model to reason about the continuity. With the constrained jigsaw puzzles, instead of solving them directly, which could still be extremely hard, we carefully design four surrogate tasks that are more solvable but meanwhile still ensure that the learned representation is sensitive to spatiotemporal continuity at both the local and global levels. Extensive experiments show that CSJ achieves state-of-the-ae on two downstream tasks across various benchmarks.
INTRODUCTION
Self-supervised learning (SSL) has achieved tremendous successes recently for static images (He et al., (2000; Chen et al. 2020) and shown to be able to outperform supervised learning on wide range of downstream image understanding tasks. However, such successes have not yet been re- produced for videos. Since different SSL models differ mostly on the pretext tasks employed on the unlabeled training data, designing pretext tasks more suitable for videos is the current focus for self-supervised video representation learning (Han al. 2020 Wang et al. 2020).
Videos are spatiotemporal data and spatiotemporal analysis is the key to many vidco content un- derstanding tasks. A good video representation learned from the self-supervised pretext task should therefore capture discriminative information jointly along both spatial and temporal dimensions. is thus somewhat explicitly require joint spatiotemporal video understanding. For example, some spatial pretext tasks have been borrowed from images without any modification (Jing et al., 2018)) ignoring the temporal dimension. On the other hand, many recent vidco-specific pretext tasks typically involve speed or temporal order prediction (Lee al., 2017 Wei i.e., operating predominately along the temporal axis.
inntuttrr-itutivve to note that most existing SSL pretext tasks for videos do not
2018 Benaim al. 2020 Wang al. 2020).
A natural choice for a spatiotemporal pretext task is to solve 3D jigsaw puzzles, whose 2D counter- part has been successfully used for images (Noroozi & Favaro 2016). Indeed, solving 3D puzzles requires the learned model understand spatiotemporal continuity, a key step towards video con- tent understanding. However, directly solving a 3D puzzle turns out to be intractable: puzzle of 3x3x3 pieces (the same size as a Rubik's s cube) can have 27! possible permutations. Video volume in short clip is much larger than that. Nevertheless, the latest neural sorting models (Paumard ctt al.) 2020; Du et al. 2020) can only handle permutations a few orders of magnitude less, so offer solution. This is hardly surprising because such a task is daunting even for humans: Most people would struggle with a standard Rubik's 's cube, let alone a much larger one.
In this paper, we propose novel Constrained Spatiotemporal Jigsaw (CSJ) pretext task for self- supervised video representation learning. The key idea is to form 3D jigsaw puzzles a constrained manner that it becomes solvable. This is achieved by factorizing the permutations (shuffling)

---

Under review as a conference paper at ICLR 2021
6
4
9
8
2
1
Raw Image
Unconstrained Jigsaw
Constrained J Jigsaw
LCC's in Correct Order (d)
LCC's in Reverse Order
(b)
(c)
Figure/ 1: Illustration of our constrained jigsaw and the surrogate pretext tasks using an image ex- ample (only spatial for clarity). Our constrained jigsaw can easily extended to the spatiotemporal domain done in this work. (a): The raw image. (b),(c): Comparing an unconstrained puzzle (b) and our constrained one (c), is clear that ours is much more continuous (hence interpretable) reflected by the size of the largest continuous cuboids (LCCs, rectangles in images here) shown in red. (d), Illustration of the importance of the relative order of the top-2 LCCs for determining the global continuity level of the shuffled image. (d) and (e) have the same top-2 LCCs, but only (d) keeps the correct relative order between them. Locating these LCCs and predicting their relative order are thus the key objectives of our surrogate tasks.
into the three spatiotemporal dimensions and then applying them sequentially, This ensures that for a given video clip, large continuous spatiotemporal cuboids exist after the constrained shuffling to provide sufficient cues for the model to reason about spatiotemporal continuity (see Fig. [c)).. Such large continuous cuboids are also vital for human understanding of video as revealed in neu- roscience and visual studies (Stringer et al., 2006; Chen al.) 2019p. Even with the constrained puzzles, solving them directly could still be extremely hard. Consequently, instead of directly solv- ing the puzzles (i.e., recovering the permutation matrix so that each piece can be put back), four surrogate tasks are carefully designed. They are more solvable but meanwhile still ensure that the learned representation is sensitive to spatiotemporal continuity at both the local and global levels. Concretely, given video clip shuffled with constrained permutations, top-2 largest continuous cuboids (LCCs) dominate the clip volume. The level of continuity in the shuffle clip as whole is thus determined mainly by the volumes of these LCCCs, and whether they are at the right order (see Fig. [d)(e)) both spatially and temporally. Our surrogate tasks are thus designed to locate these LCCs and predict their order so that the model learned with these tasks can be sensitive to spatiotemporal continuity both locally and globally.
make sure that the
Our main contributions are three-fold: (1) We introduce new pretext task for self-supervised video representation learning called Constrained Spatiotemporal Jigsaw (CSJ). To our best knowledge, this is the first work on sule-vupe video representation learning that leverages spatiotemporal jigsaw understanding. (2) We propose novel constrained shuffling method to construct easy 3D jigsaws containing large LCCs. Four surrogate tasks are then formulated in place of the original jigsaw solving tasks. They are much more solvable yet remain effective in learning spatiotemporal discriminative representations. (3) Extensive experiments show that our approach achieves state-of- the-art on two downstream tasks across various benchmarks.
2 RELATED WORK
Self-supervised Learning with Pretext Tasks Self-supervised learning (SSL) typically employs a pretext task to generate pseudo-labels for unlabeled data via some forms of data transformation. Ac- cording to the transformations used by the pretext task, existing methods for video presentation learning can be divided into three categories: (1) Spatial-Only Transformations: Derived from the original image domain (Gidaris et al., 2018), (20g).0inng et al. (2118) leveraged the spatial-only transforma-
tions for self-supervieed video presentation learning. (2) Temporal-Only Transformations: MMsra et al.) (2016): Fernando et al. (2017); Lee et al.) (2017): Wei et al. (2018) obtained shuffled video frames with the temporal-only transformations and then distinguished whether the shuffled frames are in chronological order. Xu et al. (2019) chose to shuffle video clips instead of frames. [enaam] jtt al. (2020); Yao et al. (2020): Jenni et al. (202)) exploited the speed transformation via determin- ing whether one video clip is accelerated. (3) Spatiotemporal Transformations: There only few recent approaches (Ahsan et al.) 2019: Kim et al.) 2019) that leveraged both spatial and tempo- ral transformations by permuting 3D spatiotemporal cuboids. However, due to the aforementioned
2

---

Under review as a conference paper at ICLR 2021
intractabilityy of solving the spatiotemporal jigsaw puzzles, they only leveraged either temporal or spatial permutations as training signals, ie., they exploited the two domains independently. There- fore, no true spatiotemporal permutations have been considered in Ahsan et al. (2019); [Kim et al. (2019). In contrast, given that both spatial appearances and temporal relations are important cues for video representation learning, the focus of this work is investigating how to exploit the spa- tial and temporal continuity jointly for self-supervised video presentation learning. To that end, our Constrained Spatiotemporal Jigsaw (CSJ) presents the first spatiotemporal continuity based pretext task for video SSL, thanks to a novel constrained 3D jigsaw and four surrogate tasks to reason about the continuity in the 3D jigsaw puzzles without solving them directly.
Self-supervised Learning with Contrastive Learning Contrastive learning is another self- supervised learning approach that has become increasingly popular in the image domain (Misra) & Maaten 2020, He et al.) 2020 Chen et al. 2020). Recently, it has been incorporated into video SSL as well. Contrastive learning and transformation based pretext tasks are orthogonal to each other and often combined in that different transformed versions of data sample form the positive set used in contrastive learning. In Enl-Noby et al.] (2019); Knights et al. (2020); Qian et al. (2020); Wang et al. (2020) Yang et al.) (2020), the nesitive/negative samples were generated based on tem- poral transformations only. In contrast, some recent works (Han et al.) 2019 2020) Zhuang et al.
2020 leveraged features from the future frame embeddings or with the memory bank (Wu et al.,
2018 They modeled spatiotemporal representations using only contrastive learning without trans- formations. Contrastive learning is also exploited in one of our surrogate pretext tasks. Different from existing works, we explore the spatiotemporal transformations in the form of CSJ and employ contrastive learning to distinguish different levels of spatiotemporal continuity in shuffled jigsaws. This enables us to learn more discriminative spatiotemporal representations.
3 CONSTRAINED SPATIOTEMPORAL JIGSAW JIGSAW
3.1 PROBLEM DEFINITION
The main goal of self-supervised video representation learning is to learn a video feature represen- tation function f()) without using any human annotations. general approach to achieving this goal is to generate supervisory signal y from an unlabeled video clip æ and construct pretext task P to predict y from (x)). The process of solving the pretext task P encourages f(-) to learn discriminative spatiotemporal representations.
The pretext task P is constructed typically by applying to a video clip a transformation function t(: 0) parameterized by 0 and then automatically deriving y from 0, e.g., y can be the type of the transformation. Based on this premise, P is defined as the prediction of y using the feature map of the transformed video clip f()), i.e., P f()) y. where ã f(x; 9). For example, in Lee et al. (2017). {(:; 0) denotes temporal transformation that permutes the four frames of video clip temporal order 0, क = f( 0) is the shuffled clip, and the pseudo-label y is defined as the permutation order 0 (e.g.. 1324, 4312, etc.). The pretext task P is then a classification problem of 24 categories because there are = 24 possible orders.
3.2 CONSTRAINED PERMUTATIONS
Solving spatiotemporal video jigsaw puzzles seems to be an ideal pretext task for learning discrimi- native representation as it requires an understanding of spatiotemporal continuity, After shuffling the pixels in a video clip using 3D permutation matrix, the pretext task is to recover the permutation matrix. However, as explained earlier, this task intractable given even moderate video clip sizes. Our solution is to introduce constraints on the permutations. As a result, new pretext task Pesi based on Constrained Spatiotemporal Jigsaw (see Fig. ((a)) is formulated, which is much easier to solve than random//nconstrainee jigsaw.
Specifically, our goal is to introduce constraints to the permutations so that the resultant shuffled video clip is guaranteed to have large continuous cuboids (see Fig aa) Similar to humans (Stringer et al. 2006), having large continuous cuboids is key for a model to understand a 3D jigsaw and therefore to have any chance to solve it. Formally, the volume of shuffled video clip H are denoted as (T, H, W) ). measuring its sizes along the temporal, height, and width dimensions, respectively. A cuboid is defined crop of T: Wiys tng mggg where 4,, l2 E {1,2,... T), hy, h2
3

---

Under review as a conference paper at ICLR 2021
FPN
LCCD
LIcCD
X
NL (
Raw Video
MICCD
1(30gs)
CCMR
LECMR
the Phe Foo Inl
Conventional
Each clip
3D J Jissaw
CSPC
CScC
+
Share Weight
[()
3D CNN
CLSC
CLSC
f(o)
Embeldlogs
X
Raw Video Clip
f()
Constrained
g Shuttled Clips
X
f(x)
3D J Igsaw
M:
Spatinterngoral
Mini-batch
(a)
(b)
Figure 2: (a) Illustration of Constrained Spatiotemporal Jigsaw (CSJ) (see Sec. 3.2
(b) The
pipeline of our proposed framework for self- video representation learning (see Sec. 3.3 raw video clip transformed into 8 shuffled clips with our Constrained Spatiotemporal Jig- saw (CSJ), and 3D CNN sharing weights extracts the feature representations from them. The model then trained by solving four self-supervised tasks jointly.
{1, 2.. H). w1, W2 2.. W). If all the jigsaw pieces (smallest video clip unit, e.g. pixel or a 3D pixel block) in keep the same relative order as they were in æ (before being shuffled), we call the cuboid e as continuous cuboid e The cuboid's volume equals (12 (1) (hz - h1) x (w2 w,), and the largest continuous cuboid (LCCC) Cmax is the c c with the largest volume.
We introduce two permutation strategies ensure that the volumes of LCCs are large in relation to the whole video clip volume after our shuffling transformation t-; OCsi). First, instead of shuffling in three spatiotemporal dimensions simultaneously, ((: OcsI) factorizes the permutations into the three spatiotemporal dimensions and then utilizes them sequentially to generate shuffled clips, e.g., in the order of T, W, H and only once. Note that the volume of the generated E stays the same with different permutation orders (e.g.. TW H and HTW). Second, we shuffle group of jigsaw pieces together instead of each piece individually along each dimension. Taking spatial shuffling as an example, if there are 8 pieces per frame (along each of the two spatial dimensions), Ocsı could be represented as the permutation from (12345678) } to (84567123) The longest and the second- longest index ranges are: 5] for coordinates (4567), and [6, 8] for coordinates {123}. With these two permutation strategies, not only do we have large LCCs, but also they are guaranteed to have clearly separable boundaries (see Fig. (()) with surrounding pieces due to the factorized and grouped permutation design. This means that they are easily detectable.
3.3 SURROGATE TASKS
Having permutation constraints preserves more spatiotemporal continuity in the shuffled clip and re- duces the amount of possible permutations. But exploiting these constraints to make neural sorting model tractable is still far from trivial. Instead of solving the jigsaw directly, our Pesi thus formu- lated as four surrogate tasks: Largest Continuous Cuboid Detection (LCCD), Clip Shuffling Pattern Classification (CSPC), Contrastive Learning over Shuffled Clips (CLSC), and Clip Continuity Mea- sure Regression (CCMR). As illustrated in Fig. 2b). given unlabeled clip mini-batch of clips (ø1, 22, T3} by shuffling æ with different but related constrained permuta- tions (to be detailed later). These shuffled clips and the clip æ then fed into 3D CNN model f(-) for spatiotemporal representation learning with non-local operation (Wang et al. 2018):
first construet
(w.) (w.) fx)),
where NL(:,)) denotes the non-local operator, and f(,,) and f(x) denote the feature map of ã; and æ from the last convolutional layer of f(),, respectively. The resultant feature map fNI. (ā;)) is further passed through spatial pooling layer followed by separately fully-connected layer for
4

---

Under review as a conference paper at ICLR 2021
each surrogate task. Note that the raw video feature map f (ax) ) is used guidance through the non- local based attention mechanism to help fulfill the tasks. This is similar to humans needing to see the completed jigsaw picture to help solve the puzzle.
Before we detail the four tasks, we first explain how the eight permutations from the same raw clip generated. First, the factorized and grouped permutations are applied to æ to create one shuffled clip. By examining the largest and the second-largest continuous puzzle piece numbers of each dimension ((T, H, W)), we can easily identify the top-2 largest continuous cuboids (LCCs). Next, by varying the relative order of the top-2 LCCs either in the correct (original) order or the reverse order in each dimension, 2x2x2= permutations are obtained. By controlling the group size in permutation, we can make sure that the top-2 LCCs account for a large proportion, saying 80% of the total clip volume. Our four tasks are thus centered around these two L.CCs as they largely determine the overall spatiotemporal continuity of the shuffled clip.
The first task LCCD is to locate the top-2 LCCs {come ()) j = 1, and formulated as a regression problem. Given ground-truth LCC C (j), Gaussian kernel is applied to its center to depict the possibility of each pixel in ãेt belonging to the LCC. This Ieads to soft mask MiCCD with the same
size of w: MiceD is all 0 outside the region of m (j), and exp( ||a |aa aell3 la-acll inside the region, 20
where a, ag denote any pixel and the center point, respectively. 0g is the hyper-parameter which is set as 1 empirically. In the training stage, FPN (Lin et al.) 2017) is used for multi-level feature fusion. LCCD is optimized using the MSE loss in each point:
LLCCD =
Σ Σ
MSE(M (M) Micco(a))),
JE(1.2) AE
where MSE(.,.) denotes the MSE loss function, and MiccD is the prediction of each pixel a.
CSPC is designed to recognize the shuffling pattem of a shuffled clip. As mentioned early, the eight shuffled clips in each mini- batch are created from the same raw clip and differ only in the relative order of the top-2 LCCs along each of the three dimensions. There are thus eight permutations depending on the order (correct or reverse) in each dimension. Based on this understanding, CSPC formulated as multi-class classification task to recognize each shuffled clip into one of these eight classes, which is optimized using the Cross-Entropp (CE) loss:
LCSPC
Σ
(3)
(E(0.......7
where CE(:,) denotes the CE loss function and [<spe[i] is the predicted class label of i-th sample
(shuffled clip) in each mini-batch.
The two tasks above emphasize on local spatiotemporal continuity understanding. In contrast, CLSC leverages the contrastive loss to encourage global continuity understanding. In particular, since the top-2 LCCs dominate the volume of clip, it is safe to assume that if their relative order is correct in all three dimensions, the shuffled clip largely preserve continuity compared to the original clip, while all other 7 permutations feature large discontinuity in at least one dimension. We thus form contrastive learning task with the original video æ and the most continuous shuffled video T; as a positive pair, and æ and the rest 2; (j * i) as negative pairs. CLSC is optimized using the Noise Contrastive Estimation (NCE) (Tian et al. 2020) loss:
f
Lasc = llog (xppsim(((x)))
(4)
+ E, exp ( ((;))//T)
where sim(...) ( is defined by the dot product: f(x) T f(:),, and 7 is the temperature pyreee-ar Note that the non-local operator is not used in CLSC.
CCMR similar to CLSC in that it also enforces global continuity understanding, but differs in that it is a regression task aimed at predicting a global continuity measure. We consider two such measures. Since the total size of the top-2 LCCs { ( (j) = 1,2} is a good indicator of how continuous shuffle video clip is, the first measure 4u directly measures the relative total size of
v v(cemmt 1)) + v( c
the t2p-2 LCCs: 44- =
where v(-) represents the volume of a clip/cuboid.
v
5

---

Under review as a conference paper at ICLR 2021
The second measure hd
gh/w examines the shuffling degree of Iे' in each dimension, computed as the hamming(( at)
normalized hamming distance:
NN 1)/2' where hamming(-) denotes the hamming distance
in each dimension between the original piece sequence and the permuted one, and Ne represents the number of pieces in each dimension so that (NVe 1) indicates the maximum possible hamming distance in the dimension. CCMR is optimized using the Mean Squared Error (MSE) loss:
LCCMR MSE([ha, laa thss Gal Mua thas tha (dl))
(5)
where has thas 1h did are the prediction of the model.
3.4 OVERALL LEARNING OBJECTIVE
Our entire CSJ framework is optimized end-to-end with the learning objective defined as:
L = LLCCD + LzLcsre + d'LLuse + ogLLCRR.
where 01, 0g. Tyy denote the weights for the four losses. We deploy the adaptive weighting mech- anism (Kendall et al. 2018) to weight these tasks, and thus there is no free prrer--prameterss to tune. We also adopt curriculum learning (Bengio et al. 2009 Korbar et al. 2018) to train our network by shuffing clips from easy to hard. More details are presented in Appendix, A.1 and(..
4 EXPERIMENTS
4.1 DATASETS AND SETTINGS
We select three benchmark datasets for performance evaluation: UCF101 (Somro et al, 2012). HMDB51 (Kuehne et al., 2011). and Kinetics-400 (K400) (Kay et al. 2017), containing 13K/7K/306K video clips from 101/5T/400 action classes, respectively.
the self-supervised pre- training stage, utilize the first training split of UCFI01/HMDB51 and the training split of K400 without using their labels. As in Han et al. (2020), we adopt R2D3D as the backbone network, which is modified from R3D (Hara et al.) 2018) with fewer parameters. By fine-tuning the pre-trained model. we can evaluate the SSL performance on a downstream task (i.e., action classification). Fol- lowing Han al. (2019): He et al. (2020), two evaluation protocols are used: comparisons against -tate-of-the-arss follow the more popular fully fine-tuning evaluation protocol, but ablation analysis takes both the linear evaluation and fully fine-tuning protocols. For the experiments on supervised learning, we report top-1 accuracy on the first test split of UCFI0//HMDBS! I as the standard (Han al. 2020). More details of the datasets are provided in Appendix
4.2 IMPLEMENTATION DETAILS
Raw videos in these datasets are decoded at frame rate of 24-30 fps. From each raw video, we start from randomly selected frame index and sample consecutive 16-frame video clip with temporal stride of 4. For data augmentation, we first resize the video frames to 128x171 pixels, from which extract random crops of size 112x 1 12 pixels. We also apply random horizontal flipping and random color jittering to the video frames during training. We exploit only the raw RGB video frames input, and do not leverage optical flow or other auxiliary signals for self-supervised pre- training. We adopt the Adam optimizer with a weight decay of 10-3 and a batch size of 8 per GPU (with total of 32 GPUs). We deploy cosine annealing learning rate with an initial value of 10-4 and 100 epochs. The jigsaw puzzle piece sizes of {T. H, W} dimensions are set as 1, 4, 4. respectively. 16x x 112 112 video clip thus contains 16 x 28x 28 pieces. We the temperature hyper- parameert T to 0.07. A dropout of 0.5 is applied to the final layer of each task. More implementation details of the fine-tuning and test evaluation stages can be found in Appendix B
4.3 MAIN RESULTS
Comparison Action Recognition A standard way to evaluate a self-supervised video represen- tation learning model is to use it to initialize action recognition model on small dataset. Specif- ically, after self-supervised pre- pre-training on UCFI01/HMDB5 1/K400, we exploit the learned back- bone for fully fine-tuning on UCF101 and HMDB51, following Han al. (2020); WWang al. (2020)

---

Under review as a conference paper at ICLR 2021
Table 1: Comparison the state-of-the-art UCFI0I(U) and HMDB51(H). All models pre- trained with the RGB modality only. t: Methods with temporal-only transformations. Σ: Methods with both spatial and temporal transformations. Methods that leverage spatiotemporal represen- tations. HT: How To100M, The underline represents the second-best result.
Methods
Backbone
Pre-trained
Input size (T*H)
UCFI01
HMDB51
Datasets
CMC (ECCV'20) (Tian 2020
C2D
55.3
Skip-Clipl (EI-Nouby
2019
R3D-18
U/H
16*112
64.4
VCOPi (CVPR 19] (Xu al. 2019
R3D-18
U/H
16*112
64.9
29.5
VCPÍ (AAAI220) Lao ct al. 2020b
R3D-18
16*112
66.0
31.5
PRP (CVPR 20) Yao et al
2020
R3D-18
U/H
16*112
66.5
29.7
MemDPC (ECCVT) 20) Han et al. 2020
R2D3D-18
40*128
69.2
CSJ (ours)
R2D3D-18
16*112
70.4
36.0
Video-Jigsawf (WACV"19) (Alssn et al. Statisics" (CVPR'1) 19) (Wan cl al. 2019
2019
C2D
K400
25*224
55.4
27.0
C3D
K400
16*112
61.2
33.4
ST-Puzzle (AAAI'19) [Kim et al. 2019
R3D-18
K400
16*112
63.9
33.7
DPC (ICCVW' 19) (Hañ el al. 2019
R2D3D-18
K400
40a128
68.2
34.5
SpeedNet (CVPR 20) (Benarm al.
2020
13D
K400
16*224
66.7
43.7
VIE (CVPR 20) Zhuang et al 2020
R3D-18
K400
16*112
75.5
44,6
Pacel (ECCV'20) Wang et al. 2020
R(2+1)D-18
K400
16*112
77.1
36.6
CSJ (ours)
R2D3D-18
K400
16*112
76.2
46.7
MemDPC (ECCV'20) (Han al. 2020
R2D3D-34
K400
40*224
78.1
41.2
CBT Sun et al. 2019
S3D
К600=HI
79.5
44,6
CSJ (ours)
R2D3D-34
K400
16*224
79.5
50,9
Upper Bound: Super-sude
R3D-34
K4400
16*224
87.7
59.1
Table 2: Comparison with state-of-the-art self-supervised learning methods for nearest neighbor video retrieval (top-k recall) on UCFI01. The underline represents the second-best result.
Methods
Backbone
Top1
Top5
Top10
Top20
Top50
VCOP (CVPR'19 Xu et al.
2019
R3D-18
14.1
30.3
40.0
51.1
66.5
VCP (AAAI'20) Luo et al. 2020b
R3D-18
18.6
33.6
42.5
53.5
68.1
SpeedNet (CVPR 20) (Benamm et al.
2020
S3D-G
13.1
28.1
37.5
49.5
65.0
PRP (CVPR' 20)
Yao et al.. 2020
R3D-18
22.8
38.5
46.7
55.2
69.1
Pace (ECCV'20) (Wang et al. 2020
R3D-18
19.9
36.2
55.6
ERUV (Luo et al.
2020a
R3D-18
21.4
35.2
43.8
53.1
68.3
RCMDPC 20) (Han al. 2020
R2D3D-18
20.2
40.4
52.4
64.7
CSJ (ours)
R2D3D-18
21.5
40.5
53.2
64.9
70.0
We consider baseline: fully-supervised learning with pre-training K400. Note that this base- line is commonly regarded as the upper bound of self-supervised representation learning (Alwassel al. 2019). From Table T we have the following observations: (1) Our CSJ achieves state-of-the- art performance on both UCF101 and HMDB51. Particularly, with the backbone R2D3D-18 18 that weaker than 1(2+1)D-18, our CSJ performs comparably w.r.t. Pace on UCF101 but achieves 10% improvement over Pace on HMDB51. (2) By exploiting spatiotemporal transformations for self-supervised representation learning, our CSJ beats either methods with only temporal transfor- mations (f) methods with both spatial and temporal transformations (f), well as those learning spatiotemporal representations (x) via only contrastive learning (w./o. spatiotemporal transforma- tions). (3) Our CSJ also outperforms CBT (Sun al, (201)), which used ten-times more massive datasets (K600 (Carreira et al.) 2018) + Howtol00M (Miech et al.) 2019)) and multiple modalities AAGB+Audio). (4) Our CSJ is the closest to the fully-supervised one (upper bound), validating its effectiveness in self-supervised video representation learning.
Comparison in Video Retrieval We evaluate CSJ method in the video retrieval task. Fol- lowing Xu et al. (2019), we extract each video clips' embeddings with the pre- pre-training model and each clip in the test set query the k nearest clips in the training set. The comparative results Table show that our method outperforms all other self-supervised methods and achieves new state-of-the-an afe-ttr-rttt in video retrieval UCF101. Particularly, method beats the latest competitor

---

Under review as a conference paper at ICLR 2021
Table 3: Evaluation of pre-training tasks with the backhone R2D3D-18 under linear probe and fully
fine-tuning protocols on UCFIOI, AW: Adaptive Weighting. CL: Curriculum Learning.
Tasks
Linear Probe
Fully Fine-tuning
Random Initialization
LCCD
21.8
67.8
CSPC
22.6
68.1
CLSC
18.9
68.1
CCMR
22.7
68.1
CCMR+CSPC
24.7
69.2
CCMR+CSPC+CLSC
25.5
69.3
CCMR+CSPC4CLSCHLLCO
27.9
69.5
CCAR-CSPCHCLSCALCCDANN
28.2
70.0
CCMB-CSRC--ISSCHLCCD-AAW
28.5
70.4
Skiing
Playing Flute
Hammering
Golf Swing
Brushing Teeth Boxing Speed Bag
Raw frames
w./o, SSL
SSL
Figure 3: Attention visualization of the last feature maps from the fine-tuned models on UCF101. The first row denotes the raw frames from videos, and the last two rows correspond to fine-tuning from random initialization and our self-supervised pre-traine pre-trained model, respectively.
PRP Yao al. 2020) four of five metries. This indicates that proposed CSJ is also effective for video representation learning in video retrieval.
4.4 FURTHER EVALUATIONS
Ablation Study We conduct ablative experiments to validate the effectiveness of four CSJ sur- rogate tasks and two additional learning strategies. From Table 3 we can observe that: (1) Self- supervised learning with each of the four tasks shows better generalization than fine-tuning the network from scratch (random initialization). (2) By training over all the four tasks jointly, we can achieve large performance gains (see '+LCCD' vs. 'CCMR") ). (3) Each additional learning strategy (i.e., adaptive weighting or curriculum learning) leads to a small boost to the performance by 0.3- 0.5%. (4) Our full model achieves remarkable classification accuracy of 70.4%. demonstrating the effectiveness of our proposed CSJ with only the RGB video stream (without additional optical flow, audio, text modalities). More ablative analysis be found in Appendix D
Visualization of Attention Maps Fig. visualizes the attention map of the last feature maps from two models fine-tuned on UCF101 with or without adopting our self-supervised pre-training. Since each frame's attention map involves four adjacent frames, it actually contains spatiotemporal seman- tie features. We can see that our self-supervised pre- pra-training with CSJ indeed helps to better capture meaningful spatiotemporal information and thus recognize the action categories more correctly.
8

---

Under review as a conference paper at ICLR 2021
t
0
-=4 4
t=8
t=12
t =0
t=4
t=8
t=
200
100
100
100
100
100
100
100
303
100
100
100
303
100
100
100
100
300
100
1.00
100
100
100
300
(a)
(b)
20
20
20
20
20
20
20
20
20
20
20
20
20
0.
20
20
20
23
0
0.
0
(C)
(d)
Figure 4: Visualization of the LCCD predictions from the pre-trained models. Each row denotes the frames at time stamp (0, 4, 8, 12) from one video clip. (a) raw frames (with color jittering); (b) shuffled frames; (c) the ground truth of LCCD; (d) network's 's prediction.
Visualization of LCCD Predictions We also demonstrate the visualization of the LCCD predic- tions from the pre-trained models in Fig. 4 We can observe that solving the LCCD task indeed enables the model to learn the locations of LCCs and understand spatiotemporal continuity, which a key step towards video content understanding.
5 CONCLUSION
We have introduced a novel self-supervised video representation learning method named Con- strained Spatiotemporal Jigsaw (CSJ). By introducing constrained permutations, our proposed CSJ is the first to leverage spatiotemporal jigsaw in self-supervised video representation learning. We also propose four surrogate tasks based on our constrained spatiotemporal jigsaws. They are designed to encourage a video representation model to understand the spatiotemporal continuity, key building block towards video content analysis. Extensive experiments were carried out to val- idate the effectiveness of each of the four CSJ tasks and also show that our approach achieves the of-t-offthe-art on two downstream tasks across various benchmarks.
REFERENCES
Yazan Abu Farha and Juergen Gall. Ms-ten: Multi-stage temporal convolutional network for action segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, pp. 3575-3584, 2019.

---

Under review as a conference paper at ICLR 2021
Unaiza Ahsan, Rishi Madhok, and Irfan Essa. Video jigsaw: Unsupervised learning of spatiotem- poral context for video action recognition. WACV, pp. 179-189, 2019.
Humam Alwassel, Dhruv Mahajan, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self- supervised learning by cross-modal audio-video clustering. arXiv preprint arXiv: 126677, 2019.
Sagie Benaim, Ariel Ephrat, Oran Lang, Inbar Mosseri, William T. Freeman, Michael Rubinstein, Michal Irani, and Tali Dekel. Speednet: Learning the speediness videos. CVPR, pp. 9922 9931, 2020.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston, Curriculum learning. In ICMLE, pp. 41 48, 2009.
João Carreira, Eric Noland, Andras Kanki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. arXiv preprint 1rX..v/808.01344 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, pp. 10709-10719, 2020.
Yue Chen, Yalong Bai, Wei Zhang, and Tao Mei. Destruction and construction learning for fine- grained image recognition. In CVPR, pp. 5157-5166, 2019.
Ruoyi Du, Dongliang Chang, Ayan Kumar Bhunia, Jiyang Xie, Yi-Zhe Song, Zhanyu Ma, and Jun Guo. Fine-graines visual classification via progressive multi muati-granullrity training of jigsaw patches. arXiv preprint arXiv:2003.03836 2020.
Alaaeldin El-Nouby, Shuangfei Zhai, Graham W. Taylor, and Joshua M. Susskind. Skip-clip: Self- supervised spatiotemporal representation learning by future clip order ranking. arXiv preprint 1rXiv:1910.12770, 2019.
Basura Fernando, Hakan Bilen, Efstratios Gavves, and Stephen Gould. Self-supervised video rep- resentation learning with odd-one-out networks. In CVPR, pp. 3636-3645, 2017.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. ICLR, 2018.
Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding. In ICCV Workshop, pp. 1483-1492, 2019.
Tengda Han, Weidi Xie, and Andrew Zisserman. Memory-augmented dense predictive coding for video representation learning. ECCV, 2020.
Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet? In CVPR, pp. 6546-6555, 2018.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pp. 9739-9738, 2020.
Simon Jenni, Givi Meishvili, and Paolo Favaro. Learning video representations by transforming time. In ECCV, 2020.
Longlong Jing, Xiaodong Yang, Jingen Liu, and Yingli Tian. Self-supervised spatiotemporal feature learning via video rotation prediction. arXiv preprint arXiv:/811.11388 2018
Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset. arXiv preprint arXiv:1705.06955 2017.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In CVPR, pp. 7482-7491, 2018.
Dahun Kim, Donghyeon Cho, and So Kweon. Self-supervised video representation learning with space-time cubic puzzles. In AAAI, pp. 8555-8552, 2019.
10

---

Under review as a conference paper at ICLR 2021
Joshua Knights, Anthony Vanderkop, Daniel Ward, Olivia Mackenzie-Rane Ross, and Peyman Moghadam. Temporally coherent embeddings for sulf--uppr video representation learning. arXiv preprint arXiv:2004.02753, 2020.
Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from self-supervised synchronization. In NeurIPS, pp. 7763-7774, 2018.
Hilde Kuehne, Ali Arslan, and Thomas Serre.
The language of actions: Recovering the syntax and semantics of goal-directed human activities. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 780-787, 2014.
Hildegard Kuehne, Hueihan Jhuang, Estibaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: A large video database for human motion recognition. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2556-2566 2011.
Hsin-Ying Ying Lee, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang. Unsupervised representa- tion learning by sorting sequences. In ICCV, pp. 667-676, 2017.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, pp. 2117-2125, 2017.
Dezhao Luo, Bo Fang, Yu Zhou, Yucan Zhou, Dayan Wu, and Weiping Wang. Exploring relations in untrimmed videos for self-supervised learning. arXiv preprint arXiv:2008.027111 2020a.
Dezhao Luo, Chang Liu, Yu Zhou, Donghao Yang, Can Ma, Qixiang Ye, and Weiping Wang. Video cloze procedure for self-supervised spatio-temporal learning. In AAAI, 2020b.
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. 100m: Learning a text-video embedding by watching hundred million narrated video clips. In ICCV, pp. 2630-2640, 2019.
Ishan Misra and Laurens der Maaten. Self-supervised learning of pretext-invariant representa- tions. In CVPR, pp. 6707-6717, 2020.
Ishan Misra, C. Lawrence Zitnick, and Martial Hebert. Shuffle and learn: Unsupervised learning using temporal order verification. In ECCV, pp. 527-544, 2016.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, pp. 69-84, 2016.
Marie-Morgane Paumard, David Picard, and Hedi Tabia. Deepzzle: Solving visual jigsaw puzzles with deep learning and shortest path optimization. TIP, 29:3569-3581. 2020.
Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang. Huisheng Wang, Serge Belongie, and Yin Cui. Spatiotemporal contrastive video representation learning. arXiv preprint arXiv:2008. 03800, 2020.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: /2 12.04402 2012.
Simon M. Stringer, Gavin Perry, Edmund T. Rolls, and J. H. Proske. Learning invariant object recognition in the visual system with continuous transformations, Biological Cybernetics, 94(2): 128-142, 2006.
Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. Learning video representations using contrastive bidirectional transformer. arXiv preprint arXiv:1906.05743. 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV, 2020.
Jiangliu Wang, Jianbo Jiao, Linchao Bao, Shengfeng Hc, Yunhui Liu, and Wei Liu. Self-supervised spatio-temporal representation learning for vidcos by predicting motion and appearance statistics. In CVPR, pp. 4015, 2019.
Jiangliu Wang, Jianbo Jiao, and Yun-Hui Liu. Self-supervised video representation learning by pace prediction. In ECCV, 2020.
11

---

Under review as a conference paper at ICLR 2021
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR. pp. 7794-7803, 2018.
Donglai Wei, Joseph Lim, Andrew Zisserman, and William T. Freeman. Learning and using the arrow of time. In CVPR, pp. 8052-8060, 2018.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In CVPR, pp. 3733-3742, 2018.
Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang. Self-supervised spa- tiotemporal learning via video clip order prediction. In CVPR, pp. 10334-10343, 2019.
Ceyuan Yang, Yinghao Xu, Bo Dai, and Bolei Zhou. Video representation learning with visual tempo consistency. arXiv preprint arXiv:2006. 15489, 2020.
Yuan Yao, Chang Liu, Dezhao Luo, Yu Zhou, and Qixiang Ye. Video playback rate perception for self-supervised
representation learning. In CVPR, pp. 3348-6557, 2020.
Chengxu Zhuang, Tianwei She, Alex Andonian, Max Sobol Mark, and Daniel Yamins. Unsuper- vised learning from video with deep neural embeddings, CVPR, pp. 9563-9572, 2020.
12

---

Under review as a conference paper at ICLR 2021
A ADDITIONAL LEARNING STRATEGIES
A.1 ADAPTIVE WEIGHT
Formally, CSJ has two continuous outputs Y4, 4 from LCCD and CCMR, and two discrete outputs Y2.Ys Ys from CSPC and CLSC, modeled with Gaussian likelihoods and softmax likelihoods, respectively. The joint loss for these four tasks L(W, oi 02, 03, 04) is:
L(W, Ø1, 02, 03. 04)
lo.... (x), log (y4;; f f (x), oî)
softmax(yy= (x). 02) softmax(y==c)) (x), 0z)
207 20pllyi p (x)]l log
201 yy f' (x)ll log
logplys P. (x). (x)))
- logplys (x),
= Li ( Lz (W) + o3 (W)
20 LW)
1
20?
o3
+ lo + log log log
where a is the weight factor that can be automatically learned from the network, and the log likeli- hood for the output y defined
1 log p(y = cf (x),0)= (x) fe (x)) (x)-log Σ exp(
0 fe (x)).
(8)
A.2 CURRICULUM LEARNING
We adopt curriculum learning (Korbar et al. 2018) to train our network by shuffling clips from easy to hard. Let d be the shuffle degree of shuffled clip æ, representing the number of continuous cuboids in each dimension. We gradually increase dl from 3 to 5 during the training phase to produce more permuted clips. Note that when the video content is ambiguous in one dimension, e.g., a static video clip inflated from image, there is temporal variance learn the transformation. Kim ict al. (2019); Noroozi & Favaro (2016) also mentioned this problem as similar-looking ambiguity. To solve this problem, calculate the variance each dimension and set threshold. If the variance is lower than the threshold, we decrease d from 3 to so that the pieces are not shuffled in the corresponding dimension.
B DATASETS AND IMPLEMENTATION
B.1 DETAILS OF DATASETS
UCF101 (Soomro al. 2012 is widely-used dataset in the action recognition task, which contains 13,320 videos with 101 action classes. The dataset is divided into three training/testing splits. In this paper, following prior works (Wang et al. 2020 Han et al.) 2020). we use the first training split as the pre-training dataset and the first testing split for evaluation.
HMDB51 (Kuehne al. 2011) is relatively small action recognition dataset. consisting of 6,766 videos with 51 categories. It is also divided into three training/testing splits. Following Wang et al. ;2000): [an et al. (2020), we use the first training split as the pre-training dataset and the first testing split for evaluation.
Kinetics-400 (K400) (Kay et al. human action classes and around 306k videos. In this work, we use the training split of K400 as the pre-training dataset.
2017) a very large action recognition dataset consisting of 400
B.2 IMPLEMENTATION DETAILS
In the fine-tuning stage, weights of convolutional layers initialized with self-supervised pre- training. but weights of fully-com colly-connected layers are randomly initialized. The whole network is then trained with the cross-entropy loss. The pre-processing and training strategies are the same as in the
13

---

Under review as a conference paper at ICLR 2021
Table 4: The structure of the encoding function f()) R2D3D-18 is used as an example.
output size
stage
detail
HW
input data
16 112 x
1 1 7"; 64
conv
stride 1,
16 562 x 64
1 x 3 64
pool
16 x 28 x 64
stride 1,2
x 3 64
re82
1 x 32, 64
16 28 x 64
x 32.128 128
re5g
1 x 1x33.128 128
14 x 128
3 3 256 256
IeS4
x 25.
x
8 x 72 256
3 31,
Tes;
x 31, 512
x 4 512
x 4 512
Avgpool
1 x 1 512
stride 1.
self-super pre- pre-training stage, except that the total epochs are 300 and the initial learning rate is 10-3 We use batch size of 64 per GPU and total of 8 GPUs for fine-tuning.
We follow the standard evaluation protocol (Han et al., 2020) during inference and use ten-crop to take the same sequence length as training from the video. The predicted label of each video is calculated by averaging the softmax probabilities of all clips in the video.
C NETWORK ARCHITECTURE
We deploy the network backbone R2D3D Han al. (209 (R3D) similar to Hara et al.] (2018). The only difference between R2D3D and R3D lies in that: R2D3D keeps the first two residual blocks as 2D convolutional blocks while R3D uses 3D blocks. Therefore, the modified R2D3D has fewer parameters (only the last two blocks are 3D convolutions). We present the CNN structure of R2D3D Table
2020). which 3D-ResNet
D ADDITIONAL ABLATION STUDIES
Table 5: Evaluation of pre-training tasks under different designs of LCCD on UCF101.
Methods
LCCS
1.CD+ I.CCD +L1 I.CCD + MSE
Top-1 Acc
66.5
64.0
66.5
67.8
D.1 LCCD
Instead of predicting center points using the detection method, we also design a segmentation method - largest continuous cuboid segmentation (LCCS) to predicts the location of top-2 LCCs ( (j) j = 1, The difference between LCCD and LCCS lies in that: LCCS is formulated as segmentation task to discriminate whether pixel is in the region of c (j). Concretely, LCCS predicts binary mask Mices where only points in the region of {c (j) set to be 1, otherwise 0. As a result, LCCS is optimized using the Cross Entropy (CE) loss at each point:
LLCCS Σ CB(M (o)MLoss()) Ma)cs(a)),
JE(1.2 aE
where CE(..) denotes the CE loss function, and (fccs(a) is the predicted class of pixel a.
We report the performance of four different designs of LCCD in Table 5 (1) LCCS: LCCS is used instead of LCCD. (2) LCCD+Miccs: The Gaussian mask MLCCD is substituted by the binary mask Muccs, but the LCCD task is optimized using the MSE loss. (3) LCCD L1: The LCCD task is
14

---

Under review as a conference paper at ICLR 2021
optimized by the LI loss. (4) LCCD MSE: The LCCD task is optimized by the MSE loss. From 5 can be seen that the segmentation task also helps self-supervised representation learning but doesn't 't perform well as LCCD. Also, under the three different settings of LCCD, the MSE loss with the Gaussian map performs the best.
Table 6: Evaluation of different temperature 7 for CLSC on UCF101
Methods Top-1 Ace
==1
T = 0.1
0.07
60.9
66.3
68.1
D.2 CLSC
Tabboe shows the accuracies obtained with different temperatures used in contrastive learn-
ing. We can observe that: (1) When is in the range ~ 0.07, the accuracy increases with smaller
T, (2) When 7 is large (e.g., 1), the accuracy drops considerably. this work, T is set to 0.0.
Table 7: Evaluation of different designs of CSPC on UCF101.
Methods
2 Categories
4 Categories
8 Categories
Top-1 Acc
67.0
68.0
68.1
D.3 CSPC
In addition our CSPC with pattern categories (see Sec. 3.3), we consider another designs:
(1) 2 Categories: the shuffled clip is discriminated by whether it has the same relative order of the top-2 LCCs as the raw clip. It is almost the same as CLSC but is optimized by the CE loss. (2) Categories: the shuffled clip is discriminated by how it differs from the raw clip: non-difference, spatial-only difference, temporal-only difference, spatiotemporal difference. From Table we can see that CSPC with 8 categories outperforms the other two designs. These results support our moti- vation for leveraging spatiotemporal transformations.
Table 8: Evaluation of different designs of CCMR UCF101.
Methods
ld
hd
Id+hd
Top-I Acc
66.9
67.2
68.1
D.4 CCMR
We report the performance of three different designs of CCMR: (1) Id: the learning degree hi used as supervision, which only contains volume information. (2) hd: the hamming distances lha tha 4 are used, which contain only the relative order information. (3) ld + hd: both ld and hd are used as supervision. From Table [ we can see that: First, both ld and hd help the model to learn continuous characteristics during pre-training, and hd outperforms Id by small margin. Second, our CCMR learns the best representation by combining and hd.
D.5 RESULTS OF DIRECTLY SOLVING CSJ
We also demonstrate the results of solving the CSJ task directly in Table g We randomly shuffle video clips into 4 x 4 x 4 jigsaw puzzles. To recognize the correct permutation, the model solve a (4! x 4! x 4!)-way classification task in the pre-training stage. We compare the CSJ task with the joint LCCD+CCMR task under the same setting for fair comparison. Linear evaluation is adopted to show the effectiveness of different tasks. We observe from the table that solving LCCD+CCMR jointly is more effective than solving CSJ directly.
E TEMPORAL ACTION SEGMENTATION
To show the effectiveness of our CSJ for solving new downstream tasks, we apply the pretrained model obtained by CSJ to temporal action segmentation, which is more challenging than the
15

---

Under review as a conference paper at ICLR 2021
Table 9: Evaluation of pre-training tasks with the backbone R2D3D-18 under the linear evaluation protocol on UCFI01. For computation efficiency, CSJ is only defined on x 4 x 4 cells.
Random Initialization
CSJ
LCCD+CCMR (4 x 4) LCCD+CCMR (16 28 x 28)
8.3
13.8
18.2
23.1
Raw
frames
Pour Milk
Spoon Powder
Stir Milk
GT
CSJ
MemDPC
Raw frames
Take Cup
Pour Coffee
Pour Milk
GT
CSJ
MemDPC
(b)
Stir Coffee
Figure 5: Qualitative results for the temporal action segmentation task the Breakfast dataset. Note that the notation () denotes an unannotated segment in the ground truth.
conventional action recognition and retrieval tasks. Specifically, choose compare CSJ model with the latest competitor MemDPC (Han et al.] 2020) on the Breakfast dataset (Kuehne al. 2014). For fair comparison, our CSJ model and the McmDPC model adopt the same R2D3D- 34 backbone. Due the time constraint, from the original Breakfast dataset, we only use small subset of 200 long videos as the training for fine-tuning, and select few long videos for the test. For temporal action segmentation, we follow the overall framework of MS-TCN (Abu Farba & Gall 2019), but changes its backbone R2D3D-34 pretrained by our CSJ or MemDPC.
We present the qualitative results on two test videos in Fig. 5 We can clearly observe that our CSJ outperforms MemDPC on both test videos. Particularly, the predictions of our CSJ are much closer to the ground truth, but MemDPC tends to produce unwanted segments for temporal action segmen- tation: it wrongly recognizes the segment (color in yellow) in the middle part the first video as 'Pour Milk', and the segment (color in black) in the last part of the second video 'Stir Coffee'. In conclusion, as compared to the latest SSVRL method MemDPC, our CSJ can learn more robust features for temporal action segmentation due its 'true' spatiotemporal jigsaw understanding.
16


Reviews:
[{"review_id": "mTIm92kcKn_", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a novel pretext task for self-supervised video representation learning (SSVRL). The authors design several surrogate tasks for tackling intentionally constructed constrained spatiotemporal jigsaw puzzles.  The learned representations during training to solve the surrogate tasks can be transferred to other video tasks. The proposed method shows superior performances than state-of-the-art SSVRL approaches on action recognition and video retrieval benchmarks. \n\n## Strengths: \n\n+. Good performances on two benchmarks. \n\n+. Carefully designed surrogate tasks. \n\n## Weaknesses:\n\n-. Lack insightful analysis of how the idea is inspired, why it works. It seems the intuition of the paper is to make the 3D jigsaw problem easier to solve and it will just work. But why the easier problem could help learn better representations? Each of the two steps making the problem easier need to be analyzed more thoroughly: first, making the unconstrained jigsaw problem constrained; second, solving the surrogate tasks instead of solving the constrained jigsaw problem. Actually, the carefully designed surrogate tasks are quite different from the constrained jigsaw problem. They seems more ad-hoc but not a principled way to tackle the jigsaw problem. All these questions need more indepth clarification. \n\n-. Experimental analysis is not thorough. In case the proposed method is not a principled method, but a carefully designed method. Extensive experiments of different variations of the proposed method could help better understand why the method works. A good performance on well-established benchmarks might be impressive, but analysis of why the performance can be achieved is more important. \n\n-. Writing needs improvements. In the exposition of the proposed method section, some sentences are casual and misleading. For example, the third paragraph of sec 3.2. Besides, section 3.3 is a little bit difficult to follow. It could be possibly revised more concisely. \n\n## Summary\n\nOverall, the paper presents yet another method to design the pretext task for SSVRL. But my major concern is it lacks enough insights for inspiring future research for this topic. It might not be good enough for ICLR. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ihYiW7QaKd", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\nThis paper proposes a new way of formulating and solving \"spatiotemporal jigsaw puzzles\", as a self-supervised pretext task for learning useful video representations. Positive results on two downstream tasks, action recognition and video retrieval, are shown.\nThe main contribution of the paper is a novel way of constraining the space of possible spatiotemporal permutations, in order to increase the tractability of the problem, and proposal of surrogate tasks that require learning and understanding of spatiotemporal continuity and correlations in order to solve them.\n\n**Strengths**:\n+ The pretext task of solving spatiotemporal jigsaw puzzles in order to learn meaningful video representations is well motivated, as has also been demonstrated in the literature.\n+ Constraining the number of permutation to make the problem tractable indeed seems necessary, and the proposed method of doing so, along with the proposed surrogate tasks, are indeed effective, as indicated by the positive results on downstream tasks.\n+ The ablation analysis performed shows a positive contribution of each of the described surrogate tasks and training scheme.\n\n**Weaknesses**:\n\n- I found some of the major statements in this work to be over-claimed:\n\n  - \" To our best knowledge, this is the first work on self-supervised video representation learning that leverages spatiotemporal jigsaw understanding\". As you mention in Sec. 2,  Ahsan et al. (2019); Kim et al. (2019) both attempted to solve spatiotemporal jigsaw puzzles as a self-supervised pretexts task for learning spatiotemporal representation. This paper claims that the particular constraints imposed on the permutations used in those papers in order to increase the tractability of the problem, are not \"true\" spatiotemporal permutations. I believe this work at most relaxes some of those constraints, albeit in creative ways, but is not solving a fundamentally different problem.\n - Table 1, which demonstrates \"state-of-the-art performance\" on action recognition, is incomplete. Some stronger, not-included results of methods you did include in the table, and which, to the best of my knowledge, use only the RGB modality:  \n Pace           |  S3D-G  |  87.1  |  52.6  \n SpeedNet  |  S3D-G  |  81.1  |  48.8  \nfor UCF101 (left) and HMDB51 (right).\n  - For Table 2, which shows \"new state-of-the-art in video retrieval\", additional, stronger, \"Pace\" results exist :  \nPace | C3D | 31.9 | 49.7 | 59.2 | 68.9 | 80.2  \nPace | R(2+1)D | 25.6 |  42.7 |  51.3 |  61.3 |  74.0  \nfor (L-R) top 1, 5, 10, 20, 50.\n\n- For the visualization (Sec. 4.4), it would be nice to see what the network attends to in order to solve the pretext task, before fine-tuning on UCF101 to solve action recognition.\n\n\nOut of curiosity -- often in self-supervised learning the network tends to learn \"artificial cues\" (such as boundary or compression artifacts) which help it solve the pretext task, without really learning anything meaningful. Significant work is usually required to mitigate such trivial learning. Did you have a similar problem? I can't seem to find any documentation of such a phenomenon in your work.\n\nIn general, I find new SSL work especially interesting if it (A) enables solving new tasks that were unfeasible before, and/or (B) pushes the boundary of SSL results on interesting/important tasks. Since (A) is, according to my understanding outlined above, not accurately demonstrated, and (B) is not shown, I vote for rejecting this paper in its current form. I would gladly reconsider given stronger results or the demonstration of newly enabled tasks based on the proposed method.\n\n\n**Post-rebuttal**\n\nI'd like to thank the authors for addressing my comments. I've read through the other reviews and responses, as well as the revised paper. The presented method for learning \"true spatiotemporal permutations\" is novel, and does indeed seem to learn effective representations.\n\nWhat I'm not entirely sure about is how much this method manages to push the boundary of SSL. Comparing methods with different backbones is indeed tricky, and my intention was definitely not to discourage SSL works from academia. But the burden of proof should be on the new method to perform as close to an apples-to-apples comparison (in terms of backbone) to existing methods as possible. In the end, there are many many potential pretext tasks for SSL of video representations, and I do feel that in order to be publishable at a top-tier venue, they should either enable new tasks, or show clear superiority over existing methods.\n\nRegarding temporal action segmentation as a newly enabled task -- I honestly missed this section, since it's in the appendix. This should be moved to the main paper.\n\nIf I could, I would be borderline on this paper. But since I can't, I'll give the authors the benefit of the doubt, and raise my rating to 6 (marginally above).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "UY-n9WDgNZr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary and contributions**:\nThe authors propose an approach for solving a constraint version of the 3D (space+time) jigsaw puzzle over a video, using four easier surrogate tasks. The four surrogate tasks are diverse, having different formulations: regression (LCCD, CCMR), classification (CSPC), and noise contrasting (CLSC) problems. All of them are learned together, using a joint learning objective.\n\nThe authors show the value of the newly learned representations in the self-supervised scenario on two downstream tasks (video action recognition and video retrieval), achieving state-of-the-art results compared with new methods.\n\n**Strengths**:\n- The paper comes with a solution for expanding a classical self-supervised 2D problem formulation to 3D, proposing a tractable approach by breaking it into four tasks and imposing constraints through them. \n- A good amount of details on the method and on how the permutations and the surrogate tasks were chosen.\n- The fact that even though the permutations are heavily constrained, they proved to be useful. \n- The newly learned representations, that embed the temporal and spatial continuity aspects, achieve state-of-the-art results on two video tasks.\n\n**Weaknesses**:\n- The temporal aspect is not sufficiently highlighted in the experiments:\n- An ablation study to better distinguish between the spatial and spatiotemporal representations, with both quantitative and qualitative experiments would strengthen the submission.\n- It would be interesting to see how much the quantity of temporal information reflects in the performance (ablation on the number of used frames)\n\n**Quality**:\nThe idea is simple but complex enough to generate valuable representations, having the temporal aspect integrated. The paper is technically sound.\n\n**Clarity**: \nThe paper is clearly written and easy to read and follow.\n\n**Novelty**:\nThe overall idea is not novel, but the way it is implemented and the proposed constraints are novel.\n\n**Significance of this work**:\nThis work is relevant for the field, it incrementally advances the current integration of the temporal and spatial aspects in video.\n\n**Typos**:\nThe 3rd vertical line in Table 3 should be shifted with 1 column.", "extra_scores": {}, "extra_text": {}}, {"review_id": "9yrhPdDp1d-", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary\nIn this paper, the authors extend the self-supervised 2D jigsaw puzzle solving idea to 3D for self-supervised video representation learning. To make the 3D jigsaw puzzle problem tractable, they propose a two-fold idea. First, they constrain the 3D jigsaw puzzle solution space by factorizing the permutations into time, x, and y dimensions and by grouping pieces. Second, since the constrained 3D jigsaw is still intractable, they propose four surrogate tasks of the 3D jigsaw: 1) LLCD (detecting largest continuous cuboid), 2) CSPC (3D permutation pattern classification), 3) CLSC (contrastive learning over permuted clips), 4) CCMR (measuring the global continuity of the permuted clips)\nThey evaluate their method's efficacy on the public benchmarks by following linear/finetuning self-supervised learning evaluation protocols.\n \n#### Strengths\nI like the idea of solving a 3D jigsaw puzzle as a pretext spatio-temporal learning task. By learning to solve the 3D jigsaw puzzle, the learned representations could be discriminative for the downstream tasks. Solving the jigsaw puzzle as a pretext task for representation learning is already explored and shown to be effective both in 2D spatial for images [Noroozi & Favaro, ECCV 2016] and 1D temporal dimension for videos [Xu et al., CVPR2019]. Nevertheless, due to the problem's intractability, there is no such prior work on solving a jigsaw puzzle for video representation learning. Therefore, I think this work is valuable as the authors make the problem tractable, and they show the efficacy of the 3D jigsaw puzzle solving.\n \n#### Weaknesses and suggestions\nHowever, I have several concerns about the work. \n* I do not understand how the puzzle pieces are grouped exactly. The authors show an example of grouped permutation: {12345678} -> {84567123}. It is confusing to me. It seems that the groups are {123}, {4567}, {8} from the original sequence. However, how do we make these groups? Is the group sizes always 1,3,4 for length-8 sequences? I suggest the authors provide more details on how they group the pieces.  \n* Artificial patterns in the shuffled clips might be problematic. In contrast to the 2D jigsaw [Noroozi & Favaro, ECCV 2016] and 1D jigsaw [Xu et al., CVPR2019], the backbone encoder in this work takes the shuffled clips with artificial patterns (see the Fig. 1(c). There are vertical and horizontal lines). It is unlikely to see these artificial patterns in the downstream tasks. There is a training-testing mismatch. Finetuning might fix the problem, but it is not guaranteed. I want to listen to the authors' opinions on this issue.\n* Missing ablation experiments and analysis. I list the missing analyses below.\n  1. Why the number of largest continuous cuboids are two? What happens if it is one, three, or four? \n  2. They use non-local operation between the permuted and the original features to guide the surrogate tasks. It would be informative to show the performance when we remove this part. Also, I am not quite sure why FPN is used only for LCCD. What happens if we do not use FPN for LCCD?\n  3. Analysis of the correlation between surrogate task performance and the downstream task performance.\n\n I will increase my rating if the majority of concerns are resolved.\n  \n#### Minor comments\nFor me, Table 3 is a bit hard to parse. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new idea and approach for self-supervised video representation learning.

The reviewers' opinions diverge. R1 suggests that the paper is (marginally) above the threshold. R2 supports the paper, saying that he/she likes the idea behind the paper. R3 explicitly mentioned that he/she would like to provide a borderline rating (but cannot due to the system). R4 is not in favor of the paper, even after the rebuttal.

The AC’s opinion is more aligned with R3 and R4, who are the more senior reviewers among the four. There are two main concerns with the paper: technical contribution and experimental comparison. In terms of the contribution, both the reviewers find that the paper is lacking: "What I'm not entirely sure about is how much this method manages to push the boundary of SSL." (by R3). "Overall, the paper presents yet another method to design the pretext task for SSVRL. But my major concern is it lacks enough insights for inspiring future research for this topic." (by R4). The authors argue the difficulties of being in academia in doing this research with limited computation resources and argue that the reviewers should focus more on novelty and contribution, but even after the rebuttal, R4 is not convinced and R3 is still mildly concerned whether the proposed approach really brings something new to the field as the paper fails to show "clear superiority over existing methods".

In addition, as pointed out by the reviewers, there are several state-of-the-art self-supervised video representation learning works that the paper misses to cite, or compare against. In addition to Pace and SpeedNet R3 mentioned, below are approaches reporting results on UCF101 and HMDB with the standard self-supervised classification task setting (Table 1):

AVTS 89.0, 61.6
CVRL 92.1, 65.4
ELo 93.8, 67.4
XDC 94.2, 67.4
GDT 95.2, 72.8

We note that all these results are much superior to the best results reported by the proposed approach, 79.5 and 50.9 on UCF101 and HMDB. The authors mention in the rebuttal that these superior approaches not included in the paper use stronger backbones (and are thus omitted), but we believe a more academically proper attitude is to include all these numbers and explicitly describe why the proposed approach is not performing better, instead of completely omitting their results.

The AC also questions whether the R2D3D-34 backbone used in this paper really is computationally lighter compared to the backbones used in previous approaches like R(2+1)D-18, which alternates 2D residual modules and 2+1D residual modules (using much fewer parameters and compute than 3D modules) and also has fewer layers. XDC using R(2+1)D-18 backbone reports 86.8/52.6 (UCF/HMDB) accuracies with Kinetics-400 unlabeled data. AVTS also reports 84.1/52.5 (UCF/HMDB) accuracies using MC3-18 backbone. Similarly, GDT uses R(2+1)D-18 backbone and reports 89.3/60.0 (UCF/HMDB) accuracies using unlabeled Kinetics-400.  Even MemDPC reports 86.1/54.5 using R-2D3D backbone when optical flow feature is added. All these are far superior to the results being reported in the paper.

Overall, we view the experimental section of this paper as incomplete, and we cannot convince ourselves that the paper reaches the quality of ICLR.


[AVTS] Korbar, B., Tran, D., Torresani, L.: Cooperative learning of audio and video models
from self-supervised synchronization. In: NeurIPS (2018)

[ELo] A. Piergiovanni, A. Angelova, and M. S. Ryoo. Evolving losses for unsupervised video representation learning. In Proc. CVPR, 2020

[XDC] H. Alwassel, D. Mahajan, L. Torresani, B. Ghanem, and D. Tran. Self-supervised learning by cross-modal audio-video clustering. arXiv preprint arXiv:1911.12667, 2019

[GDT] M. Patrick, Y. M. Asano, R. Fong, J. F. Henriques, G. Zweig, and A. Vedaldi. Multi-modal self-supervision from generalized data transformations. arXiv preprint arXiv:2003.04298, 2020

[CVRL] R. Qian, T. Meng, B. Gong, M.-H. Yang, H. Wang, S. Belongie, and Y. Cui. Spatiotemporal contrastive video representation learning. arXiv preprint arXiv:2008.03800, 2020
Decision: Reject