Self-Supervised Video Representation Learning with Constrained Spatiotemporal Jigsaw

1 INTRODUCTION . Self-supervised learning ( SSL ) has achieved tremendous successes recently for static images ( He et al. , 2020 ; Chen et al. , 2020 ) and shown to be able to outperform supervised learning on a wide range of downstream image understanding tasks . However , such successes have not yet been reproduced for videos . Since different SSL models differ mostly on the pretext tasks employed on the unlabeled training data , designing pretext tasks more suitable for videos is the current focus for self-supervised video representation learning ( Han et al. , 2020 ; Wang et al. , 2020 ) . Videos are spatiotemporal data and spatiotemporal analysis is the key to many video content understanding tasks . A good video representation learned from the self-supervised pretext task should therefore capture discriminative information jointly along both spatial and temporal dimensions . It is thus somewhat counter-intuitive to note that most existing SSL pretext tasks for videos do not explicitly require joint spatiotemporal video understanding . For example , some spatial pretext tasks have been borrowed from images without any modification ( Jing et al. , 2018 ) , ignoring the temporal dimension . On the other hand , many recent video-specific pretext tasks typically involve speed or temporal order prediction ( Lee et al. , 2017 ; Wei et al. , 2018 ; Benaim et al. , 2020 ; Wang et al. , 2020 ) , i.e. , operating predominately along the temporal axis . A natural choice for a spatiotemporal pretext task is to solve 3D jigsaw puzzles , whose 2D counterpart has been successfully used for images ( Noroozi & Favaro , 2016 ) . Indeed , solving 3D puzzles requires the learned model to understand spatiotemporal continuity , a key step towards video content understanding . However , directly solving a 3D puzzle turns out to be intractable : a puzzle of 3×3×3 pieces ( the same size as a Rubik ’ s cube ) can have 27 ! possible permutations . Video volume even in a short clip is much larger than that . Nevertheless , the latest neural sorting models ( Paumard et al. , 2020 ; Du et al. , 2020 ) can only handle permutations a few orders of magnitude less , so offer no solution . This is hardly surprising because such a task is daunting even for humans : Most people would struggle with a standard Rubik ’ s cube , let alone a much larger one . In this paper , we propose a novel Constrained Spatiotemporal Jigsaw ( CSJ ) pretext task for selfsupervised video representation learning . The key idea is to form 3D jigsaw puzzles in a constrained manner so that it becomes solvable . This is achieved by factorizing the permutations ( shuffling ) into the three spatiotemporal dimensions and then applying them sequentially . This ensures that for a given video clip , large continuous spatiotemporal cuboids exist after the constrained shuffling to provide sufficient cues for the model to reason about spatiotemporal continuity ( see Fig . 1 ( b ) ( c ) ) . Such large continuous cuboids are also vital for human understanding of video as revealed in neuroscience and visual studies ( Stringer et al. , 2006 ; Chen et al. , 2019 ) . Even with the constrained puzzles , solving them directly could still be extremely hard . Consequently , instead of directly solving the puzzles ( i.e. , recovering the permutation matrix so that each piece can be put back ) , four surrogate tasks are carefully designed . They are more solvable but meanwhile still ensure that the learned representation is sensitive to spatiotemporal continuity at both the local and global levels . Concretely , given a video clip shuffled with our constrained permutations , we make sure that the top-2 largest continuous cuboids ( LCCs ) dominate the clip volume . The level of continuity in the shuffle clip as a whole is thus determined mainly by the volumes of these LCCs , and whether they are at the right order ( see Fig . 1 ( d ) ( e ) ) both spatially and temporally . Our surrogate tasks are thus designed to locate these LCCs and predict their order so that the model learned with these tasks can be sensitive to spatiotemporal continuity both locally and globally . Our main contributions are three-fold : ( 1 ) We introduce a new pretext task for self-supervised video representation learning called Constrained Spatiotemporal Jigsaw ( CSJ ) . To our best knowledge , this is the first work on self-supervised video representation learning that leverages spatiotemporal jigsaw understanding . ( 2 ) We propose a novel constrained shuffling method to construct easy 3D jigsaws containing large LCCs . Four surrogate tasks are then formulated in place of the original jigsaw solving tasks . They are much more solvable yet remain effective in learning spatiotemporal discriminative representations . ( 3 ) Extensive experiments show that our approach achieves state-ofthe-art on two downstream tasks across various benchmarks . 2 RELATED WORK . Self-supervised Learning with Pretext Tasks Self-supervised learning ( SSL ) typically employs a pretext task to generate pseudo-labels for unlabeled data via some forms of data transformation . According to the transformations used by the pretext task , existing SSL methods for video presentation learning can be divided into three categories : ( 1 ) Spatial-Only Transformations : Derived from the original image domain ( Gidaris et al. , 2018 ) , Jing et al . ( 2018 ) leveraged the spatial-only transformations for self-supervised video presentation learning . ( 2 ) Temporal-Only Transformations : Misra et al . ( 2016 ) ; Fernando et al . ( 2017 ) ; Lee et al . ( 2017 ) ; Wei et al . ( 2018 ) obtained shuffled video frames with the temporal-only transformations and then distinguished whether the shuffled frames are in chronological order . Xu et al . ( 2019 ) chose to shuffle video clips instead of frames . Benaim et al . ( 2020 ) ; Yao et al . ( 2020 ) ; Jenni et al . ( 2020 ) exploited the speed transformation via determining whether one video clip is accelerated . ( 3 ) Spatiotemporal Transformations : There are only a few recent approaches ( Ahsan et al. , 2019 ; Kim et al. , 2019 ) that leveraged both spatial and temporal transformations by permuting 3D spatiotemporal cuboids . However , due to the aforementioned intractability of solving the spatiotemporal jigsaw puzzles , they only leveraged either temporal or spatial permutations as training signals , i.e. , they exploited the two domains independently . Therefore , no true spatiotemporal permutations have been considered in Ahsan et al . ( 2019 ) ; Kim et al . ( 2019 ) . In contrast , given that both spatial appearances and temporal relations are important cues for video representation learning , the focus of this work is on investigating how to exploit the spatial and temporal continuity jointly for self-supervised video presentation learning . To that end , our Constrained Spatiotemporal Jigsaw ( CSJ ) presents the first spatiotemporal continuity based pretext task for video SSL , thanks to a novel constrained 3D jigsaw and four surrogate tasks to reason about the continuity in the 3D jigsaw puzzles without solving them directly . Self-supervised Learning with Contrastive Learning Contrastive learning is another selfsupervised learning approach that has become increasingly popular in the image domain ( Misra & Maaten , 2020 ; He et al. , 2020 ; Chen et al. , 2020 ) . Recently , it has been incorporated into video SSL as well . Contrastive learning and transformation based pretext tasks are orthogonal to each other and often combined in that different transformed versions of a data sample form the positive set used in contrastive learning . In El-Nouby et al . ( 2019 ) ; Knights et al . ( 2020 ) ; Qian et al . ( 2020 ) ; Wang et al . ( 2020 ) ; Yang et al . ( 2020 ) , the positive/negative samples were generated based on temporal transformations only . In contrast , some recent works ( Han et al. , 2019 ; 2020 ; Zhuang et al. , 2020 ) leveraged features from the future frame embeddings or with the memory bank ( Wu et al. , 2018 ) . They modeled spatiotemporal representations using only contrastive learning without transformations . Contrastive learning is also exploited in one of our surrogate pretext tasks . Different from existing works , we explore the spatiotemporal transformations in the form of CSJ and employ contrastive learning to distinguish different levels of spatiotemporal continuity in shuffled jigsaws . This enables us to learn more discriminative spatiotemporal representations . 3 CONSTRAINED SPATIOTEMPORAL JIGSAW . 3.1 PROBLEM DEFINITION . The main goal of self-supervised video representation learning is to learn a video feature representation function f ( · ) without using any human annotations . A general approach to achieving this goal is to generate a supervisory signal y from an unlabeled video clip x and construct a pretext task P to predict y from f ( x ) . The process of solving the pretext task P encourages f ( · ) to learn discriminative spatiotemporal representations . The pretext task P is constructed typically by applying to a video clip a transformation function t ( · ; θ ) parameterized by θ and then automatically deriving y from θ , e.g. , y can be the type of the transformation . Based on this premise , P is defined as the prediction of y using the feature map of the transformed video clip f ( x̃ ) , i.e. , P : f ( x̃ ) → y , where x̃ = t ( x ; θ ) . For example , in Lee et al . ( 2017 ) , t ( · ; θ ) denotes a temporal transformation that permutes the four frames of video clip x in a temporal order θ , x̃ = t ( x ; θ ) is the shuffled clip , and the pseudo-label y is defined as the permutation order θ ( e.g. , 1324 , 4312 , etc. ) . The pretext task P is then a classification problem of 24 categories because there are 4 ! = 24 possible orders . 3.2 CONSTRAINED PERMUTATIONS . Solving spatiotemporal video jigsaw puzzles seems to be an ideal pretext task for learning discriminative representation as it requires an understanding of spatiotemporal continuity . After shuffling the pixels in a video clip using a 3D permutation matrix , the pretext task is to recover the permutation matrix . However , as explained earlier , this task is intractable given even moderate video clip sizes . Our solution is to introduce constraints on the permutations . As a result , a new pretext task PCSJ based on Constrained Spatiotemporal Jigsaw ( see Fig . 2 ( a ) ) is formulated , which is much easier to solve than a random/unconstrained jigsaw . Specifically , our goal is to introduce constraints to the permutations so that the resultant shuffled video clip is guaranteed to have large continuous cuboids ( see Fig . 2 ( a ) ) . Similar to humans ( Stringer et al. , 2006 ) , having large continuous cuboids is key for a model to understand a 3D jigsaw and therefore to have any chance to solve it . Formally , the volume of a shuffled video clip x̃ are denoted as { T , H , W } , measuring its sizes along the temporal , height , and width dimensions , respectively . A cuboid is defined as a crop of x̃ : c = x̃t1 : t2 , h1 : h2 , w1 : w2 , where t1 , t2 ∈ { 1 , 2 , . . . , T } , h1 , h2 ∈ { 1 , 2 , . . . , H } , w1 , w2 ∈ { 1 , 2 , . . . , W } . If all the jigsaw pieces ( smallest video clip unit , e.g . a pixel or a 3D pixel block ) in c keep the same relative order as they were in x ( before being shuffled ) , we call the cuboid c as a continuous cuboid ccont . The cuboid ’ s volume equals ( t2 − t1 ) × ( h2 − h1 ) × ( w2 − w1 ) , and the largest continuous cuboid ( LCC ) ccontmax is the ccont with the largest volume . We introduce two permutation strategies to ensure that the volumes of LCCs are large in relation to the whole video clip volume after our shuffling transformation t ( · ; θCSJ ) . First , instead of shuffling x in three spatiotemporal dimensions simultaneously , t ( · ; θCSJ ) factorizes the permutations into the three spatiotemporal dimensions and then utilizes them sequentially to generate shuffled clips , e.g. , in the order of T , W , H and only once . Note that the volume of the generated x̃ stays the same with different permutation orders ( e.g. , TWH and HTW ) . Second , we shuffle a group of jigsaw pieces together instead of each piece individually along each dimension . Taking spatial shuffling as an example , if there are 8 pieces per frame ( along each of the two spatial dimensions ) , θCSJ could be represented as the permutation from { 12345678 } to { 84567123 } . The longest and the secondlongest index ranges are : [ 2 , 5 ] for coordinates { 4567 } , and [ 6 , 8 ] for coordinates { 123 } . With these two permutation strategies , not only do we have large LCCs , but also they are guaranteed to have clearly separable boundaries ( see Fig . 2 ( b ) ) with surrounding pieces due to the factorized and grouped permutation design . This means that they are easily detectable .

Summary: In this paper, the authors extend the self-supervised 2D jigsaw puzzle solving idea to 3D for self-supervised video representation learning. To make the 3D jigsaw puzzle problem tractable, they propose a two-fold idea. First, they constrain the 3D jigsaw puzzle solution space by factorizing the permutations into time, x, and y dimensions and by grouping pieces. Second, since the constrained 3D jigsaw is still intractable, they propose four surrogate tasks of the 3D jigsaw: 1) LLCD (detecting largest continuous cuboid), 2) CSPC (3D permutation pattern classification), 3) CLSC (contrastive learning over permuted clips), 4) CCMR (measuring the global continuity of the permuted clips)