Learning Self-Similarity in Space and Time as a Generalized Motion for Action Recognition

1 INTRODUCTION . Learning spatio-temporal dynamics is the key to video understanding . To this end , extending convolutional neural networks ( CNNs ) with spatio-temporal convolution has been actively investigated in recent years ( Tran et al. , 2015 ; Carreira & Zisserman , 2017 ; Tran et al. , 2018 ) . The empirical results so far indicate that spatio-temporal convolution alone is not sufficient for grasping the whole picture ; it often learns irrelevant context bias rather than motion information ( Materzynska et al. , 2020 ) and thus the additional use of optical flow turns out to boost the performance in most cases ( Carreira & Zisserman , 2017 ; Lin et al. , 2019 ) . Motivated by this , recent action recognition methods learn to extract explicit motion , i.e. , flow or correspondence , between feature maps of adjacent frames and they improve the performance indeed ( Li et al. , 2020c ; Kwon et al. , 2020 ) . But , is it essential to extract such an explicit form of flows or correspondences ? How can we learn a richer and more robust form of motion information for videos in the wild ? In this paper , we propose to learn spatio-temporal self-similarity ( STSS ) representation for video understanding . Self-similarity is a relational descriptor for an image that effectively captures intrastructures by representing each local region as similarities to its spatial neighbors ( Shechtman & Irani , 2007 ) . Given a sequence of frames , i.e. , a video , it extends along the temporal dimension and thus represents each local region as similarities to its neighbors in space and time . By converting appearance features into relational values , STSS enables a learner to better recognize structural patterns in space and time . For neighbors at the same frame it computes a spatial self-similarity map , while for neighbors at a different frame it extracts a motion likelihood map . If we fix our attention to the similarity map to the very next frame within STSS and attempt to extract a single displacement vector to the most likely position at the frame , the problem reduces to optical flow , which is a particular type of motion information . In contrast , we leverage the whole volume of STSS and let our model learn to extract an effective motion representation from it in an end-toend manner . With a sufficient volume of neighborhood in space and time , it effectively captures long-term interaction and fast motion in the video , leading to robust action recognition . We introduce a neural block for STSS representation , dubbed SELFY , that can be easily inserted into neural architectures and learned end-to-end without additional supervision . Our experimental analysis demonstrates its superiority over previous methods for motion modeling as well as its complementarity to spatio-temporal features from direct convolutions . On the standard benchmarks for action recognition , Something-Something V1 & V2 , Diving-48 , and FineGym , the proposed method achieves the state-of-the-art results . 2 RELATED WORK . Video action recognition . Video action recognition is a task to categorize videos into pre-defined action classes . One of the conventional topics in action recognition is to capture temporal dynamics in videos . In deep learning , many approaches attempt to learn temporal dynamics in different ways : Two-stream networks with external optical flows ( Simonyan & Zisserman , 2014 ; Wang et al. , 2016 ) , recurrent networks ( Donahue et al. , 2015 ) , and 3D CNNs ( Tran et al. , 2015 ; Carreira & Zisserman , 2017 ) . Recent approaches have introduced the advanced 3D CNNs ( Tran et al. , 2018 ; 2019 ; Feichtenhofer , 2020 ; Lin et al. , 2019 ; Fan et al. , 2020 ) and show the effectiveness of capturing spatio-temporal features , so that 3D CNNs now become a de facto approach to learn temporal dynamics in the video . However , spatio-temporal convolution is vulnerable unless relevant features are well aligned across frames within the fixed-sized kernel . To address this issue , a few methods adaptively translate the kernel offsets with deformable convolutions ( Zhao et al. , 2018 ; Li et al. , 2020a ) , while several methods ( Feichtenhofer et al. , 2019 ; Li et al. , 2020b ) modulate the other hyper-parameters , e.g. , higher frame rate or larger spatial receptive fields . Unlike these methods , we address the problem of the spatio-temporal convolution by a sufficient volume of STSS , capturing far-sighted spatio-temporal relations . Learning motion features . Since using the external optical flow benefits 3D CNNs to improve the action recognition accuracy ( Carreira & Zisserman , 2017 ; Zolfaghari et al. , 2018 ; Tran et al. , 2018 ) , several approaches try to learn frame-by-frame motion features from RGB sequences inside neural architectures . Fan et al . ( 2018 ) ; Piergiovanni & Ryoo ( 2019 ) internalize TV-L1 ( Zach et al. , 2007 ) optical flows into the CNN . Frame-wise feature differences ( Sun et al. , 2018b ; Lee et al. , 2018 ; Jiang et al. , 2019 ; Li et al. , 2020c ) are also utilized as the motion features . Recent correlation-based methods ( Wang et al. , 2020 ; Kwon et al. , 2020 ) adopt a correlation operator ( Dosovitskiy et al. , 2015 ; Sun et al. , 2018a ; Yang & Ramanan , 2019 ) to learn motion features between adjacent frames . However , these methods compute frame-by-frame motion features between two adjacent frames and then rely on stacked spatio-temporal convolutions for capturing long-range motion dynamics . We propose to learn STSS features , as generalized motion features , that enable to capture both shortterm and long-term interactions in the video . Self-similarity . Self-similarity represents an internal geometric layout of images . It is widely used in many computer vision tasks , such as object detection ( Shechtman & Irani , 2007 ) , image retrieval ( Hörster & Lienhart , 2008 ) , and semantic correspondence matching ( Kim et al. , 2015 ; 2017 ) . In the video domain , Shechtman & Irani ( 2007 ) firstly introduce the concept of STSS and transforms the STSS to a hand-crafted local descriptor for action detection . Inspired from this work , early methods adopt self-similarities for capturing view-invariant temporal patterns ( Junejo et al. , 2008 ; 2010 ; Körner & Denzler , 2013 ) , but they use temporal self-similarities only due to computational costs . Recently , there are several non-local approaches ( Wang et al. , 2018 ; Liu et al. , 2019 ) that utilize STSS for capturing long-range dynamics of videos . However , they use STSS for reweighting or aligning visual features , which is an indirect way of using STSS . Different from these methods , our method leverages full STSS directly as generalized motion information and learns an effective representation for action recognition within a video-processing architecture . To the best of our knowledge , our work is the first in learning STSS representation using modern CNNs . The contribution of our paper can be summarized as follows . First , we revisit the notion of selfsimilarity and propose to learn a generalized , far-sighted motion representations from STSS . Second , we implement STSS representation learning as a neural block , dubbed SELFY , that can be integrated into existing neural architectures . Third , we provide comprehensive evaluations on SELFY , achieving the state-of-the-art on benchmarks : Something-Something V1 & V2 , Diving-48 , and FineGym . 3 OUR APPROACH . In this section , we first revisit the notions of self-similarity and discuss its relation to motion . We then introduce our method for learning effective spatio-temporal self-similarity representation , which can be easily integrated into video-processing architectures and learned end-to-end . 3.1 SELF-SIMILARITY REVISITED . Self-similarity is a relational descriptor that suppresses variations in appearance and reveals structural patterns in images or videos ( Shechtman & Irani , 2007 ) . Given an image feature map I ∈ RX×Y×C , self-similarity transformation of I results in a 4D tensor S ∈ RX×Y×U×V , whose elements are defined as Sx , y , u , v = sim ( Ix , y , Ix+u , y+v ) , where sim ( · , · ) is a similarity function , e.g. , cosine similarity . Here , ( x , y ) is a query coordinate while ( u , v ) is a spatial offset from it . To impose a locality , the offset is restricted to its neighborhood : ( u , v ) ∈ [ −dU , dU ] × [ −dV , dV ] , so that U = 2dU + 1 and V = 2dV + 1 , respectively . By converting C-dimensional appearance feature Ix , y into UV -dimensional relational feature Sx , y , it suppresses variations in appearance and reveals spatial structures in the image . Note that the selfsimilarity transformation closely relates to conventional cross-similarity ( or correlation ) across two different feature maps ( I , I′ ∈ RX×Y×C ) , which can be defined as Sx , y , u , v = sim ( Ix , y , I ′ x+u , y+v ) . Given two images of a moving object , the cross-similarity transformation effectively captures motion information and thus is commonly used in optical flow and correspondence estimation ( Dosovitskiy et al. , 2015 ; Sun et al. , 2018a ; Yang & Ramanan , 2019 ) . For a sequence of frames , i.e. , a video , one can naturally extend the spatial self-similarity along the temporal axis . Let V ∈ RT×X×Y×C denote a feature map of the video with T frames . Spatiotemporal self-similarity ( STSS ) transformation of V results in a 6D tensor S ∈ RT×X×Y×L×U×V , whose elements are defined as St , x , y , l , u , v = sim ( Vt , x , y , Vt+l , x+u , y+v ) , ( 1 ) where ( t , x , y ) is the spatio-temporal coordinate and ( l , u , v ) is a spatio-temporal offset from it . In addition to the locality of spatial offsets above , the temporal offset l is also restricted to its temporal neighborhood : l ∈ [ −dL , dL ] , so that L = 2dL + 1 . What types of information does STSS describe ? Interestingly , for each time t , the STSS tensor S can be decomposed along temporal offset l into a single spatial self-similarity tensor ( when l = 0 ) and 2dL spatial cross-similarity tensors ( when l 6= 0 ) ; the partial tensors with a small offset ( e.g. , l = −1 or +1 ) collect motion information from adjacent frames and those with larger offsets capture it from further frames both forward and backward in time . Unlike previous approaches to learn motion ( Dosovitskiy et al. , 2015 ; Wang et al. , 2020 ; Kwon et al. , 2020 ) , which rely on crosssimilarity between adjacent frames , STSS allows to take a generalized , far-sighted view on motion , i.e. , both short-term and long-term , both forward and backward , as well as spatial self-motion . 3.2 SPATIO-TEMPORAL SELF-SIMILARITY REPRESENTATION LEARNING . By leveraging the rich information in STSS , we propose to learn a generalized motion representation for video understanding . To achieve this goal without additional supervision , we design a neural block , dubbed SELFY , which can be inserted into a video-processing architectures and learned end-to-end . The overall structure is illustrated in Fig . 2 . It consists of three steps : self-similarity transformation , feature extraction , and feature integration . Given the input video feature tensor V , the self-similarity transformation step converts it to the STSS tensor S as in Eq. ( 1 ) . In the following , we describe feature extraction and integration steps .

Summary: This submission proposed a motion representation method based on spatio-temporal self-similarity (STSS), which represents each local region as similarities to its neighbors in both spatial and temporal dimension. There are previous works (e.g., Ref[1] , [2], [5] listed here) which utilize STSS for feature extractions, authors claim that this work is the first one to learn STSS representation based on modern CNN architecture. The proposed method is implemented as a neural block, i.e., SELFY, which can be applied into neural architectures and learned end-to-end without additional supervision. On 3 standard human action recognition data sets, Something-Something-V1 & V2, Diving-48, and FineGym, the proposed method achieves quite good empirical results.