Can an Image Classifier Suffice For Action Recognition?

1 INTRODUCTION The recent advances in convolutional neural networks ( CNNs ) ( He et al. , 2016 ; Tan & Le , 2019 ) , along with the availability of large-scale video benchmark datasets ( Kay et al. , 2017 ; Monfort et al. , 2019 ; Damen et al. , 2020 ) , have significantly improved action recognition , one of the fundamental problems of video understanding . Many existing approaches for action recognition naturally extend or borrow ideas from image recognition . At the core of these approaches is spatio-temporal modeling , which regards time as an additional dimension and jointly models it with space by extending image models ( i.e. , 3D CNNs ) ( Tran et al. , 2015 ; Carreira et al. , 2017 ; Feichtenhofer , 2020 ) or fuses temporal information with spatial information processed separately by 2D CNN models ( Lin et al. , 2019 ; Fan et al. , 2019 ) . CNN-based approaches demonstrate strong capabilities in learning saptio-temporal feature representations from video data . Videos present long-range pixel interactions in both space and time . It ’ s known in approaches like non-local networks ( Wang et al. , 2018 ) that modeling such relationships helps action recognition . The recently emerging Vision Transformers naturally own the strength of capturing long-range dependencies in data , making them very suitable for video understanding . Several approaches ( Bertasius et al. , 2021a ; Li et al. , 2021 ; Arnab et al. , 2021 ) have applied ViTs for action recognition and shown better performance than their CNN counterparts . However , these approaches are still following the conventional paradigm of video action recognition , and perform temporal modeling in a similar way to CNN-based approaches using dedicated self-attention modules . In this work , we explore a different perspective for action recognition by casting the problem as an image recognition task . We ask if it is possible to model temporal information with ViT directly without using dedicated temporal modules . In other words , can an image classifier alone suffice for action recognition ? To this end , we first propose a simple idea to turn a 3D video into a 2D image . Given a sequence of input video frames , we rearrange them into a super image according to a pre-defined spatial layout , as illustrated in Fig . 2 . The super image encodes 3D spatio-temporal patterns in a video into 2D spatial image patterns . We then train an image classifier to fulfill the task of action recognition , in exactly the same way as image classification . Without surprise , based on the concept of super images , any image classifier can be re-purposed for action recognition . For convenience , we dub our approach SIFAR , short for Super Image for Action Recognition . We validate our idea by using Swin Transformer ( Liu et al. , 2021 ) , a recently developed vision transformer that has demonstrated good performance on both image classification and object detection . Since a super image has a larger size than an input frame , we modify Swin Transformer to allow for full self-attention in the last layer of the model , which further strengthens the model ’ s ability in capturing long-range temporal relations across frames in the super image . With such a change , we show that SIFAR produces strong performance against the existing SOTA approaches ( Fig . 1 ) on several benchmarks including Kinetics400 ( Kay et al. , 2017 ) , Moments in Time ( Monfort et al. , 2019 ) , Something-Something V2 ( SSV2 ) Goyal et al . ( 2017 ) , Jester ( Materzynska et al. , 2019 ) and Diving48 ( Li et al. , 2018 ) . SIFAR also enjoys efficiency in computation as well as in parameters . We further study the potential of CNN-based classifiers directly used for action recognition under the proposed SIFAR framework . Surprisingly , they achieve very competitive results on Kinetics400 against existing CNN-based approaches that rely on much more sophisticated spatio-temporal modeling . Since 3 × 3 convolutions focus on local pixels only , CNN-based SIFAR handles temporal actions on Something-Something less effectively . We experiment with larger kernel sizes to expand the temporal receptive field of CNNs , which substantially improves CNN-based SIFAR by 4 % -6.8 % with ResNet50 . SIFAR brings several advantages compared to the traditional spatio-temporal action modeling . Firstly , it is simple but effective . With one single line of code change in pytorch , SIFAR can use any image classifier for action recognition . We expect that similar ideas can also work well with other video tasks such as video object segmentation ( Duke et al. , 2021 ) . Secondly , SIFAR makes action modeling easier and more computationally efficient as it doesn ’ t require dedicated modules for temporal modeling . Nevertheless , we do not tend to underestimate the significance of temporal modeling for action recognition . Quite opposite , SIFAR highly relies on the ability of its backbone network to model long-range temporal dependencies in super images for more efficacy . Lastly , but not the least , the perspective of treating action recognition the same as image recognition unleashes many possibilities of reusing existing techniques in a more mature image field to improve video understanding from various aspects . For example , better model architectures ( Tan & Le , 2019 ) , model pruning ( Liu et al. , 2017 ) and interpretability ( Desai & Ramaswamy , 2020 ) , to name a few . 2 RELATED WORK . Action Recognition from a Single Image . One direction for video action recognition is purely based on a single image ( Davis & Bobick , 1997 ; Zhao et al. , 2017 ; Safaei & Foroosh , 2019 ; Bilen et al. , 2016 ) . In ( Davis & Bobick , 1997 ) , multiple small objects are first identified in a still image and then the target action is inferred from the relationship among the objects . Other approaches such as ( Safaei & Foroosh , 2019 ) propose to predict the missing temporal information in still images and then combine it with spatial information for action classification . There are also approaches that attempt to summarize RGB or motion information in a video into a representation image for action recognition . For instance , motion-energy image ( MEI ) ( Davis & Bobick , 1997 ) , Dynamic Image Network ( Bilen et al. , 2016 ) , Informative Frame Synthesis ( IFS ) ( Qiu et al. , 2021 ) , Adaptive Weighted Spatio-temporal Distillation ( AWSD ) ( Tavakolian et al. , 2019b ) and Adversarial Video Distillation ( AVD ) ( Tavakolian et al. , 2019a ) . Nonetheless , our approach does not attempt to understand a video from a single input image or a summarization image . Instead the approach composites the video into a super image , and then classifies the image with an image classifier directly . Action Recognition with CNNs . Action recognition is dominated by CNN-based models recently ( Feichtenhofer et al. , 2018 ; Carreira et al. , 2017 ; Fan et al. , 2019 ; Feichtenhofer , 2020 ; Chen et al. , 2021 ; Lin et al. , 2019 ; Wang et al. , 2016 ; Zhou et al. , 2018 ; Liu et al. , 2020 ; Jiang et al. , 2019a ; Tran et al. , 2019 ) . These models process the video as a cube to extract spatial-temporal features via the proposed temporal modeling methods . E.g. , SlowFast ( Feichtenhofer et al. , 2018 ) proposes two pathways whose speed is different to capture short-range and long-range time dependencies . TSM ( Lin et al. , 2019 ) applies a temporal shifting module to exchange information between neighboring frames and TAM ( Fan et al. , 2019 ) further enhances TSM by determining the amount of information to be shifted and blended . On the other hand , another thread of work attempts to select the key frame of an activity for faster recognition ( Wu et al. , 2019 ; 2020 ; Meng et al. , 2020 ; 2021 ) . E.g. , Adaframe ( Wu et al. , 2019 ) employs a policy network to determine whether or not this is a key frame , and the main network only processes the key frames . ARNet ( Meng et al. , 2020 ) determines what the image resolution should be used to save computations based on the importance of input frame images . Nonetheless , our approach is fundamentally different from conventional action recognition . It simply uses an image classifier as a video classifier by laying out a video to a super image without explicitly modeling temporal information . Action Recognition with Transformer . Following the vision transformer ( ViT ) ( Dosovitskiy et al. , 2021 ) , which demonstrates competitive performance against CNN models on image classification , many recent works attempt to extend the vision transformer for action recognition ( Neimark et al. , 2021 ; Li et al. , 2021 ; Bertasius et al. , 2021b ; Arnab et al. , 2021 ; Fan et al. , 2021 ) . VTN ( Neimark et al. , 2021 ) , VidTr ( Li et al. , 2021 ) , TimeSformer ( Bertasius et al. , 2021b ) and ViViT ( Arnab et al. , 2021 ) share the same concept that inserts a temporal modeling module into the existing ViT to enhance the features from the temporal direction . E.g. , VTN ( Neimark et al. , 2021 ) processes each frame independently and then uses a longformer to aggregate the features across frames . On the other hand , divided-space-time modeling in TimeSformer ( Bertasius et al. , 2021a ) inserts a temporal attention module into each transformer encoder for more fine-grained temporal interaction . MViT ( Fan et al. , 2021 ) develops a compact architecture based on the pyramid structure for action recognition . It further proposes a pooling-based attention to mix the tokens before computing the attention map so that the model can focus more on neighboring information . Nonetheless , our method is straightforward and applies the Swin ( Liu et al. , 2021 ) model to classify super images composed from input frames . Note that the joint-space-time attention in TimeSformer ( Bertasius et al. , 2021a ) is a special case of our approach since their method can be considered as flattening all tokens into one plane and then performing self-attention over all tokens . However , the memory complexity of such an approach is prohibitively high , and it is only applicable to the vanilla ViT ( Dosovitskiy et al. , 2021 ) without inductive bias . On the other hand , our SIFAR is general and applicable to any image classifiers . 3 APPROACH . 3.1 OVERVIEW OF OUR APPROACH . The key insight of SIFAR is to turn spatio-temporal patterns in video data into purely 2D spatial patterns in images . Like their 3D counterparts , these 2D patterns may not be visible and recognizable by human . However , we expect they are characteristic of actions and thus identifiable by powerful neural network models . To that end , we make a sequence of input frame images from a video into a super image , as illustrated in Fig . 2 , and then apply an image classifier to predict the label of the video . Note that the action patterns embedded in a super image can be complex and may involve both local ( i.e. , spatial information in a video frame ) and global contexts ( i.e. , temporal dependencies across frames ) . It is thus understandable that effective learning can only be ensured by image classifiers with strong capabilities in modeling short-range and long-range spatial dependencies in super images . For this reason , we explore the recently developed vision transformers based on self-attention to validate our proposed idea . These methods come naturally with the ability to model global image contexts and have demonstrated competitive performance against the best-performed CNN-based approaches on image classification as well as action recognition . Next we briefly describe Swin Transformer ( Liu et al. , 2021 ) , an efficient approach that we choose to implement our main idea in this work . Preliminary . The Vision Transformer ( ViT ) [ 13 ] is a purely attention-based classifier borrowed from NLP . It consists of stacked transformer encoders , each of which is featured with a multi-head self-attention module ( MSA ) and a feed-forward network ( FFN ) . While demonstrating promising results on image classification , ViT uses an isotropic structure and has a quadruple complexity w.r.t image resolution in terms of memory and computation . This significantly limits the application of ViT to many vision applications that requires high-resolution features such as object detection and segmentation . In light of this issue , several approaches ( Liu et al. , 2021 ; Chu et al. , 2021 ; Zhang et al. , 2021 ) have been proposed to perform region-level local self-attention to reduce memory usage and computation , and Swin Transformer is one of such improved vision transformers . Swin Transformer ( Liu et al. , 2021 ) first adopts a pyramid structure widely used in CNNs to reduce computation and memory . At the earlier layers , the network keeps high image resolution with fewer feature channels to learn fine-grained information . As the network goes deeper , it gradually reduces spatial resolution while expanding feature channels to model coarse-grained information . To further save memory , Swin Transformer limits self-attention to non-overlapping local windows ( W-MSA ) only.The communications between W-MSA blocks is achieved through shifting them in the succeeding transformer encoder . The shifted W-MSA is named as SW-MSA . Mathematically , the two consecutive blocks can be expressed as : yk = W-MSA ( LN ( xk−1 ) ) + xk−1 , xk = FFN ( LN ( yk ) ) + yk , yk+1 = SW-MSA ( LN ( xk ) ) + xk , xk+1 = FFN ( LN ( yk+1 ) ) + yk+1 , ( 1 ) where xl is the features at the lth layer and FFN and LN are feed-forward network and layer normalization , respectively . SIFAR . In our case , SIFAR learns action patterns by sliding window , as illustrated in Fig 3 . When the sliding window ( blue box ) is within a frame , spatial dependencies are learned . On the other hand , when the window ( red box ) spans across frames , temporal dependencies between them are effectively captured . The spatial pooling further ensures longer-range dependencies across frames captured . Creation of Super Image . Given a set of video frames , we order them by a given layout ( Fig . 4 ) to form a large super image . Different layouts give different spatial patterns for an action class . We hypothesize that a more compact structure such as a square grid may facilitate a model to learn temporal dependencies across frames as such a shape provides the shortest maximum distance between any two images . Given n input frames , we create a super image by placing all the frames in order onto a grid of size ( m− 1 ) ×m when n < ( m− 1 ) ×m or m×m when n ≥ ( m− 1 ) ×m where m = d √ ne . Empty images are padded at the end if the grid is not full . With this method , for example , 12 frames will be fit into a 3×4 grid while 14 frames into a 4×4 grid . In the default setting , we use a 3× 3 layout for 8 images and a 4× 4 one for 16 images , respectively . There are other spatial arrangements as well ( see Fig . 4 for more examples ) . However our experiments empirically show that a square grid performs the best . our approach has linear computational complexity w.r.t the number of input frames . As described above , the size of a super image is m ( m = d √ ne ) times as large as the size of a frame image , suggesting that the total number of tokens ( or image patches ) in Swin grows linearly by n. Sliding Window . As previously mentioned , Swin Transformer performs self-attention within a small local window to save memory . It uses a uniform window size across all layers , and the default window size is 7 in the original paper . Since a larger window leads to more interactions across frames , which is beneficial for SIFAR to learn long-range temporal dependencies in super images , we slightly modify the architecture of Swin Transformer ( Liu et al. , 2021 ) for it to take different window sizes flexibly in self-attention . In particular , we keep the same window size for all the layers except the last one , whose window is as large as its image resolution , implying a global self-attention including all the tokens . Since the last layer has only two transformer encoders , the computational overhead imposed by an increased window size is quite small , as indicated in Table 1 . The change of window size may result in adjustment of the input image size as the image resolution at each layer must be divisible by the window size in Swin Transformer . As noted in Table 1 , SIFAR-B-7 keeps the vanilla architecture of Swin-B . SIFAR-B-12 is more efficient than SIFAR-B-7 because SIFAR-B-12 takes smaller images ( 1922 ) as input . We demonstrate later in Sec . 4 that a larger window is critical for SIFAR to achieve good performance on more temporal datasets such as SSV2 . Implementation . Once the spatial layout for the input frames is determined , implementing our idea in pytorch is as simple as inserting into an image classifier the following line of code , which changes the input of a video to a super image . # create a super image with a layout ( sh , sw ) pre-specified by the user . x = rearrange ( x , ’ b c ( sh sw ) h w - > b c ( sh h ) ( sw w ) ’ , sh=sh , sw=sw ) The trivial code change described above transforms an image classifier into an video action classifier . Our experiments show that the same training and evaluation protocols for action models can be still applied to the repurposed image classifier .

Summary: The paper proposes to perform action recognition by first rearranging the frames from a video into a 3x3 or 4x4 grid to form a "super image", and then giving the super image to a standard image classifier to perform action recognition. Given that this super image will be a larger image, the paper leverages the more memory efficient Swin Transformer [1] as an image classifier to perform action recognition. Experiments on Kinetics400, Moments In Time, Something-Something V2 (SSV2), Jester and Diving48 show that the proposed method is on par or exceeds SOTA in terms of accuracy. On Kinetics400, the method not only is SOTA in terms of accuracy, but also is the most FLOPs-efficient method given a specific accuracy. The strong performance suggests that a deep network's ability to model spatial relationships could also be applied to model temporal relationships across frames in a video, which is an orthogonal direction to having explicit components in the network modeling temporal relationships. Furthermore, being able to connect action recognition with image classification enables existing image classification techniques to be applied to action recognition, which could potentially accelerate the field.  [1]: Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. arXiv.org, March 2021.