Year: 2021

Abstract:
We explore a new perspective on video understanding by casting the video recognition problem as an image recognition task. Our approach rearranges input video frames into super images, which allow for training an image classifier directly to fulfill the task of action recognition, in exactly the same way as image classification. With such a simple idea, we show that transformer-based image classifiers alone can suffice for action recognition. In particular, our approach demonstrates strong and promising performance against SOTA methods on several public datasets including Kinetics400, Moments In Time, Something-Something V2 (SSV2), Jester and Diving48. We also experiment with the prevalent ResNet image classifiers in computer vision to further validate our idea. The results on both Kinetics400 and SSV2 are comparable to some of the best-performed CNN approaches based on spatio-temporal modeling. Our codes and models will be publicly available.

text: 1 INTRODUCTION
The recent advances in convolutional neural networks (CNNs) (He et al., 2016; Tan & Le, 2019), along with the availability of large-scale video benchmark datasets (Kay et al., 2017; Monfort et al., 2019; Damen et al., 2020), have significantly improved action recognition, one of the fundamental problems of video understanding. Many existing approaches for action recognition naturally extend or borrow ideas from image recognition. At the core of these approaches is spatio-temporal modeling, which regards time as an additional dimension and jointly models it with space by extending image models (i.e., 3D CNNs) (Tran et al., 2015; Carreira et al., 2017; Feichtenhofer, 2020) or fuses temporal information with spatial information processed separately by 2D CNN models (Lin et al., 2019; Fan et al., 2019). CNN-based approaches demonstrate strong capabilities in learning saptio-temporal feature representations from video data.
Videos present long-range pixel interactions in both space and time. It’s known in approaches like non-local networks (Wang et al., 2018) that modeling such relationships helps action recognition. The recently emerging Vision Transformers naturally own the strength of capturing long-range dependencies in data, making them very suitable for video understanding. Several approaches (Bertasius et al., 2021a; Li et al., 2021; Arnab et al., 2021) have applied ViTs for action recognition and shown better performance than their CNN counterparts. However, these approaches are still following the conventional paradigm of video action recognition, and perform temporal modeling in a similar way to CNN-based approaches using dedicated self-attention modules.
In this work, we explore a different perspective for action recognition by casting the problem as an image recognition task. We ask if it is possible to model temporal information with ViT directly without using dedicated temporal modules. In other words, can an image classifier alone suffice for action recognition? To this end, we first propose a simple idea to turn a 3D video into a 2D image. Given a sequence of input video frames, we rearrange them into a super image according to
a pre-defined spatial layout, as illustrated in Fig. 2. The super image encodes 3D spatio-temporal patterns in a video into 2D spatial image patterns. We then train an image classifier to fulfill the task of action recognition, in exactly the same way as image classification. Without surprise, based on the concept of super images, any image classifier can be re-purposed for action recognition. For convenience, we dub our approach SIFAR, short for Super Image for Action Recognition.
We validate our idea by using Swin Transformer (Liu et al., 2021), a recently developed vision transformer that has demonstrated good performance on both image classification and object detection. Since a super image has a larger size than an input frame, we modify Swin Transformer to allow for full self-attention in the last layer of the model, which further strengthens the model’s ability in capturing long-range temporal relations across frames in the super image. With such a change, we show that SIFAR produces strong performance against the existing SOTA approaches (Fig. 1) on several benchmarks including Kinetics400 (Kay et al., 2017), Moments in Time (Monfort et al., 2019) , Something-Something V2 (SSV2) Goyal et al. (2017), Jester (Materzynska et al., 2019) and Diving48 (Li et al., 2018). SIFAR also enjoys efficiency in computation as well as in parameters. We further study the potential of CNN-based classifiers directly used for action recognition under the proposed SIFAR framework. Surprisingly, they achieve very competitive results on Kinetics400 against existing CNN-based approaches that rely on much more sophisticated spatio-temporal modeling. Since 3 × 3 convolutions focus on local pixels only, CNN-based SIFAR handles temporal actions on Something-Something less effectively. We experiment with larger kernel sizes to expand the temporal receptive field of CNNs, which substantially improves CNN-based SIFAR by 4%-6.8% with ResNet50.
SIFAR brings several advantages compared to the traditional spatio-temporal action modeling. Firstly, it is simple but effective. With one single line of code change in pytorch, SIFAR can use any image classifier for action recognition. We expect that similar ideas can also work well with other video tasks such as video object segmentation (Duke et al., 2021). Secondly, SIFAR makes action modeling easier and more computationally efficient as it doesn’t require dedicated modules for temporal modeling. Nevertheless, we do not tend to underestimate the significance of temporal modeling for action recognition. Quite opposite, SIFAR highly relies on the ability of its backbone network to model long-range temporal dependencies in super images for more efficacy. Lastly, but not the least, the perspective of treating action recognition the same as image recognition unleashes many possibilities of reusing existing techniques in a more mature image field to improve video understanding from various aspects. For example, better model architectures (Tan & Le, 2019), model pruning (Liu et al., 2017) and interpretability (Desai & Ramaswamy, 2020), to name a few.

heading: 2 RELATED WORK
text: Action Recognition from a Single Image. One direction for video action recognition is purely based on a single image (Davis & Bobick, 1997; Zhao et al., 2017; Safaei & Foroosh, 2019; Bilen et al., 2016). In (Davis & Bobick, 1997), multiple small objects are first identified in a still image and then the target action is inferred from the relationship among the objects. Other approaches such as (Safaei & Foroosh, 2019) propose to predict the missing temporal information in still images and then combine it with spatial information for action classification. There are also approaches that attempt to summarize RGB or motion information in a video into a representation image for action recognition. For instance, motion-energy image (MEI) (Davis & Bobick, 1997), Dynamic Image Network (Bilen et al., 2016), Informative Frame Synthesis (IFS) (Qiu et al., 2021), Adaptive Weighted Spatio-temporal Distillation (AWSD) (Tavakolian et al., 2019b) and Adversarial Video Distillation (AVD) (Tavakolian et al., 2019a). Nonetheless, our approach does not attempt to understand a video from a single input image or a summarization image. Instead the approach composites the video into a super image, and then classifies the image with an image classifier directly.
Action Recognition with CNNs. Action recognition is dominated by CNN-based models recently (Feichtenhofer et al., 2018; Carreira et al., 2017; Fan et al., 2019; Feichtenhofer, 2020; Chen et al., 2021; Lin et al., 2019; Wang et al., 2016; Zhou et al., 2018; Liu et al., 2020; Jiang et al., 2019a; Tran et al., 2019). These models process the video as a cube to extract spatial-temporal features via the proposed temporal modeling methods. E.g., SlowFast (Feichtenhofer et al., 2018) proposes two pathways whose speed is different to capture short-range and long-range time dependencies. TSM (Lin et al., 2019) applies a temporal shifting module to exchange information between
neighboring frames and TAM (Fan et al., 2019) further enhances TSM by determining the amount of information to be shifted and blended. On the other hand, another thread of work attempts to select the key frame of an activity for faster recognition (Wu et al., 2019; 2020; Meng et al., 2020; 2021). E.g., Adaframe (Wu et al., 2019) employs a policy network to determine whether or not this is a key frame, and the main network only processes the key frames. ARNet (Meng et al., 2020) determines what the image resolution should be used to save computations based on the importance of input frame images. Nonetheless, our approach is fundamentally different from conventional action recognition. It simply uses an image classifier as a video classifier by laying out a video to a super image without explicitly modeling temporal information.
Action Recognition with Transformer. Following the vision transformer (ViT) (Dosovitskiy et al., 2021), which demonstrates competitive performance against CNN models on image classification, many recent works attempt to extend the vision transformer for action recognition (Neimark et al., 2021; Li et al., 2021; Bertasius et al., 2021b; Arnab et al., 2021; Fan et al., 2021). VTN (Neimark et al., 2021), VidTr (Li et al., 2021), TimeSformer (Bertasius et al., 2021b) and ViViT (Arnab et al., 2021) share the same concept that inserts a temporal modeling module into the existing ViT to enhance the features from the temporal direction. E.g., VTN (Neimark et al., 2021) processes each frame independently and then uses a longformer to aggregate the features across frames. On the other hand, divided-space-time modeling in TimeSformer (Bertasius et al., 2021a) inserts a temporal attention module into each transformer encoder for more fine-grained temporal interaction. MViT (Fan et al., 2021) develops a compact architecture based on the pyramid structure for action recognition. It further proposes a pooling-based attention to mix the tokens before computing the attention map so that the model can focus more on neighboring information. Nonetheless, our method is straightforward and applies the Swin (Liu et al., 2021) model to classify super images composed from input frames.
Note that the joint-space-time attention in TimeSformer (Bertasius et al., 2021a) is a special case of our approach since their method can be considered as flattening all tokens into one plane and then performing self-attention over all tokens. However, the memory complexity of such an approach is prohibitively high, and it is only applicable to the vanilla ViT (Dosovitskiy et al., 2021) without inductive bias. On the other hand, our SIFAR is general and applicable to any image classifiers.

heading: 3 APPROACH

heading: 3.1 OVERVIEW OF OUR APPROACH
text: The key insight of SIFAR is to turn spatio-temporal patterns in video data into purely 2D spatial patterns in images. Like their 3D counterparts, these 2D patterns may not be visible and recognizable by human. However, we expect they are characteristic of actions and thus identifiable by powerful neural network models. To that end, we make a sequence of input frame images from a video into a super image, as illustrated in Fig. 2, and then apply an image classifier to predict the label of the video. Note that the action patterns embedded in a super image can be complex and may involve both local (i.e., spatial information in a video frame) and global contexts (i.e., temporal dependencies across frames). It is thus understandable that effective learning can only be ensured by image classifiers with strong capabilities in modeling short-range and long-range spatial dependencies in super images. For this reason, we explore the recently developed vision transformers based on self-attention to validate our proposed idea. These methods come naturally with the ability to model global image contexts
and have demonstrated competitive performance against the best-performed CNN-based approaches on image classification as well as action recognition. Next we briefly describe Swin Transformer (Liu et al., 2021), an efficient approach that we choose to implement our main idea in this work.
Preliminary. The Vision Transformer (ViT) [13] is a purely attention-based classifier borrowed from NLP. It consists of stacked transformer encoders, each of which is featured with a multi-head self-attention module (MSA) and a feed-forward network (FFN). While demonstrating promising results on image classification, ViT uses an isotropic structure and has a quadruple complexity w.r.t image resolution in terms of memory and computation. This significantly limits the application of ViT to many vision applications that requires high-resolution features such as object detection and segmentation. In light of this issue, several approaches (Liu et al., 2021; Chu et al., 2021; Zhang et al., 2021) have been proposed to perform region-level local self-attention to reduce memory usage and computation, and Swin Transformer is one of such improved vision transformers.
Swin Transformer (Liu et al., 2021) first adopts a pyramid structure widely used in CNNs to reduce computation and memory. At the earlier layers, the network keeps high image resolution with fewer feature channels to learn fine-grained information. As the network goes deeper, it gradually reduces spatial resolution while expanding feature channels to model coarse-grained information. To further save memory, Swin Transformer limits self-attention to non-overlapping local windows (W-MSA) only.The communications between W-MSA blocks is achieved through shifting them in the succeeding transformer encoder. The shifted W-MSA is named as SW-MSA. Mathematically, the two consecutive blocks can be expressed as:
yk = W-MSA(LN(xk−1)) + xk−1, xk = FFN(LN(yk)) + yk,
yk+1 = SW-MSA(LN(xk)) + xk, xk+1 = FFN(LN(yk+1)) + yk+1,
(1)
where xl is the features at the lth layer and FFN and LN are feed-forward network and layer normalization, respectively.
SIFAR. In our case, SIFAR learns action patterns by sliding window, as illustrated in Fig 3. When the sliding window (blue box) is within a frame, spatial dependencies are learned. On the other hand, when the window (red box) spans across frames, temporal dependencies between them are effectively captured. The spatial pooling further ensures longer-range dependencies across frames captured.
Creation of Super Image. Given a set of video frames, we order them by a given layout (Fig. 4) to form a large super image. Different layouts give different spatial patterns for an action class. We hypothesize that a more compact structure such as a square grid may facilitate a model to learn temporal dependencies across frames as such a shape provides the shortest maximum distance between any two images. Given n input frames, we create a super image by placing all the frames in order onto a grid of size (m− 1)×m when n < (m− 1)×m or m×m when n ≥ (m− 1)×m where m = d √ ne. Empty images are padded at the end if the grid is not full. With this method, for example, 12 frames will be fit into a 3×4 grid while 14 frames into a 4×4 grid. In the default setting,
we use a 3× 3 layout for 8 images and a 4× 4 one for 16 images, respectively. There are other spatial arrangements as well (see Fig. 4 for more examples). However our experiments empirically show that a square grid performs the best.
our approach has linear computational complexity w.r.t the number of input frames. As described above, the size of a super image is m (m = d √ ne) times as large as the size of a frame image, suggesting that the total number of tokens (or image patches) in Swin grows linearly by n.
Sliding Window. As previously mentioned, Swin Transformer performs self-attention within a small local window to save memory. It uses a uniform window size across all layers, and the default window size is 7 in the original paper. Since a larger window leads to more interactions across frames, which is beneficial for SIFAR to learn long-range temporal dependencies in super images, we slightly modify the architecture of Swin Transformer (Liu et al., 2021) for it to take different window sizes flexibly in self-attention. In particular, we keep the same window size for all the layers except the last one, whose window is as large as its image resolution, implying a global self-attention including all the tokens.
Since the last layer has only two transformer encoders, the computational overhead imposed by an increased window size is quite small, as indicated in Table 1.
The change of window size may result in adjustment of the input image size as the image resolution at each layer must be divisible by the window size in Swin Transformer. As noted in Table 1, SIFAR-B-7 keeps the vanilla architecture of Swin-B. SIFAR-B-12 is more efficient than SIFAR-B-7 because SIFAR-B-12 takes smaller images (1922) as input. We demonstrate later in Sec. 4 that a larger window is critical for SIFAR to achieve good performance on more temporal datasets such as SSV2.
Implementation. Once the spatial layout for the input frames is determined, implementing our idea in pytorch is as simple as inserting into an image classifier the following line of code, which changes the input of a video to a super image.
# create a super image with a layout (sh, sw) pre-specified by the user. x = rearrange(x, ’b c (sh sw) h w -> b c (sh h) (sw w)’, sh=sh, sw=sw)
The trivial code change described above transforms an image classifier into an video action classifier. Our experiments show that the same training and evaluation protocols for action models can be still applied to the repurposed image classifier.

heading: 4 EXPERIMENTS

heading: 4.1 DATASETS AND EXPERIMENTAL SETUP
text: Datasets. We use Kinetics400 (K400) (Kay et al., 2017), Something-Something V2 (SSV2) (Goyal et al., 2017), Moments-in-time (MiT) (Monfort et al., 2019), Jester (Materzynska et al., 2019), and Diving48 (Li et al., 2018) in our evaluation. Kinetics400 is a widely-used benchmark for action recognition, which includes ∼240k training videos and 20k validation videos in 400 classes. SSV2 contains 220k videos of 174 types of predefined human-object interactions with everyday objects. This dataset is known for its high temporal dynamics. MiT is a fairly large collection of one million 3-second labeled video clips, involving actions not only from humans, but also from animals, objects and natural phenomena. The dataset includes around 800k training videos and 33,900 validation videos in 339 classes. Jester contains actions of predefined hand gestures, with 118,562 and 14,787 training and validation videos over 27 classes, respectively. Diving48 is action recognition dataset without representation bias, which includes 15,943 training videos and 2,096 validation videos over 48 classes.
Training. We employ uniform sampling to generate video input for our models. Such a sampling strategy divides a video into multiple segments of equal length. During training, one frame from each segment is randomly selected to form the input sequence to a model while in test, a frame at a fixed position of each segment (usually the center frame) is picked. Uniform sampling has shown to be effective on both Kinetics400 and SSV2. We train all our models by finetuning a Swin-B model (Liu et al., 2021) pretrained on ImageNet-21K (Deng et al.,
2009), except for those SSV2 models, which are fine tuned from the corresponding Kinetics400 models in Table 3.
Our training recipes and augmentations closely follow DeiT (Touvron et al., 2020). First, we apply multi-scale jitter to augment the input (Wang et al., 2016) with different scales and then randomly crop a target input size (e.g. 8×224×224 for SIFAR-B-7). We then use Mixup (Zhang et al., 2018) and CutMix (Yun et al., 2019) to augment the data further, with their values set to 0.8 and 1.0, respectively. After that, we rearrange the image crops as a super image. Furthermore, we apply drop path (Tan & Le, 2019) with a rate of 0.1, and enable label smoothing (Szegedy et al., 2016) at a rate of 0.1. All our models were trained using V100 GPU cards with 16G or 32G memory. Depending on the size of a model, we use a batch size of 96, 144 or 192 to train the model for 15 epochs on MiT or 30 epochs on other datasets, including 5 warming-up epochs. The optimizer used in our training is AdamW (Loshchilov & Hutter, 2019) with a weight decay of 0.05, and the scheduler is Cosine (Loshchilov & Hutter, 2017) with a base linear learning rate of 0.0001.
Inference. We first scale the shorter side of an image to the model input size and then take three crops (top-left, center and bottom-right) for evaluation. The average of the three predictions is used as the final prediction. We report results by top-1 and top-5 classification accuracy (%) on validation data, the total computational cost in FLOPs and the model size in number of parameters.
4.2 MAIN RESULTS
Table 2: Comparison with Baseline Methods. All models use 8 frames as input.
Model SSV2 Kinetics400 Top-1 Top-5 Top-1 Top-5
I3D-R50 61.1 86.5 72.6 90.6 TSM-R50 59.1 85.6 74.1 91.2 TAM-R50 62.0 87.3 72.2 90.4 TimeSformer∗ 35.9 71.1 77.5 92.5 TimeSformer∗∗ 58.7 85.9 80.1 94.4 SIFAR-B-7 59.0 86.0 79.6 94.4 SIFAR-B-12 60.1 86.8 80.0 94.5 SIFAR-B-14 60.6 86.7 80.4 94.4 ∗: Swin-B (space only); ∗∗: Swin-B (divided space-time).
Comparison with Baselines. We first compare our approach with several representative CNN-based methods including I3D (Carreira et al., 2017), TSM (Lin et al., 2019) and TAM (Fan et al., 2019). Also included in the comparison are two TimeSformer models (Bertasius et al., 2021a) based on the same backbone Swin-B (Liu et al., 2021) as used by our models. All the models considered take 8 frames as input. As can be seen from Table 2, our approach substantially outperforms the CNN baselines on Kinetics400 while achieving comparable results on SSV2. Our approach is also better than TimeSformer on both datasets. These results clearly demonstrate that a powerful image classifier like Swin Transformer can learn expressive
spatio-temporal patterns effectively from super images for action recognition. In other words, an image classifier can suffice video understanding without explicit temporal modeling.
The results also confirm that a larger sliding window is more helpful in capturing temporal dependencies on temporal datasets like SSV2. Our approach performs global self-attention in the last layer of a model only (see Table 1). This substantially mitigates the memory issue in training SIFAR models.
Kinetics400. We report our Kinetics400 results in Table 3 and compare them with SOTA approaches. Our 8-frame models (SIFAR-12 and SIFAR-14) achieve 80.0% and 80.4% top-1 accuracies, outperforming all the CNN-based approaches while being more efficient than the majority of them. SIFAR-B-14† further gains ∼ 1.8% improvement, benefiting from more input frames. Especially, SIFAR-L-12‡ yields an accuracy of 84.2%, the best among all the very recently developed approaches based on vision transformers including TimeSformer (Bertasius et al., 2021b) and MViT-B (Fan et al., 2021). Our proposed approach also offers clear advantages in terms of FLOPs and model parameters compared to other approaches except MViT-B. For example, SIFAR-B-12‡ has 5× and 37× fewer FLOPs than TimeSformer-L and ViViT-L, respectively, while being 1.4× and 3.6× smaller in model size.
SSV2. Table 4 lists the results of our models and the SOTA approaches on SSV2. With the same number of input frames, our approach is 1 ∼ 2% worse than the best-performed CNN methods. However, our approach performs on par with other transformer-based method such as TimeSformer (Bertasius et al., 2021a) and VidTr-L (Li et al., 2021) under the similar setting. Note that ViViT-L (Arnab et al., 2021) achieves better results with a larger and stronger backbone ViT-L (Dosovitskiy et al., 2021). MViT-B (Fan et al., 2021) is an efficient multi-scale architecture, which can process much longer input sequences to capture fine-grained motion patterns in SSV2 data. Training SIFAR models with more than 16 frames still remains computationally challenging, especially for models like SIFAR-B-14 and SIFAR-L-14†, which need a larger sliding window size. Our results suggest that developing more efficient architectures of vision transformer be an area of improvement and future work for SIFAR to take advantage of more input frames on SSV2.
MiT. MiT is a large diverse dataset containing label noise. As seen from Table 5, with the same backbone ViT-L, SIFAR-L-12‡ is ∼4% better than ViViT-L (Arnab et al., 2021), and outperforms AssembelNet (Ryoo et al., 2020) based on neural architecture search by a considerable margin of 8%.
Jester and Diving48. We further evaluate our approach on two other popular benchmarks: Jester (Materzynska et al., 2019) and Diving48 (Li et al., 2018). Here we only consider the best single models from other approaches
for fair comparison. As shown in Table 6, SIFAR achieves competitive results again on both datasets, surpassing all other models in comparison. Note that the Diving48 benchmark contains videos with similar background and objects but different action categories, and is generally considered as an unbiased benchmark. Our model SIFAR-B-14† outperforms TimeSformer-L by a large margin of 6%.
Classification by CNNs. We also test our proposed approach using the ResNet image classifiers on both SSV2 and Kinetics400 datasets. For fairness, the ResNet models are pretrained on ImageNet-21K. Table 7 shows the results. Our models clearly outperform the traditional CNN-based models for action recognition on Kinetics400.
Especially, with a strong backbone R152x2 (a model 2× wider than Resnet152), SIFAR-R152x2 achieves a superior accuracy of 79.0%, which is surprisingly comparable to the best CNN results (SlowFast16×8+NL: 79.8%) reported in Table 3.
On SSV2, the results of CNN-based SIFAR are less
Table 7. As seen from the table, using larger kernel sizes consistently improves the results on both ResNet50 and ResNet101 models. For example, we obtain an absolute 5.0% improvement over original ResNet50 and 2.0% over original ResNet101 respectively, using one more block with a kernel size of 21x21. When adding another block with a kernel size of 11x11 (i.e. SIFA-R50-C21-11 and SIFA-R101-C21-11), it further boosts the performance up to 6.8% with ResNet50 and 2.7% with ResNet101. These results strongly suggest that expanding the receptive field of CNNs be a promising direction to design better CNN-based SIFAR models. We leave this as future work.

heading: 4.3 ABLATION STUDIES
text: In this section, we conduct ablation studies to provide more insights about our approach.
How does an image layout affect the performance? The layout of a super image determines how spatiotemporal patterns are embedded in it. We hypothesize that the layout could affect the learning effectiveness of a model. To analyze this, we trained a SIFAR model on SSV2 for each layout illustrated in Fig. 4. As shown in Table 8a, a strip layout performs the worst while a grid layout produces the best results, which confirms our hypothesis.
Does absolute positioning embedding help? Absolute Position Embedding (APE) assigns fixed or learnable position information to each token in a transformer model, and it has been proven helpful to vision transformers such as ViT (Dosovitskiy et al., 2021). However, the Swin paper (Liu et al., 2021) shows that when relative position bias are added, APE is only moderately beneficial for classification, but not for object detection and segmentation. They thus conclude that inductive bias that encourages certain translation invariance is still important for vision tasks. To find out whether or not APE is effective in our approach, we add APE to each frame rather than each token. The results in Table 8b indicate that APE slightly improves model accuracy on SSV2, but is harmful to Kinetics400. In our main results, we thus apply APE to SSV2 only.
Table 9: Effects of temporal order.
Order Kinetics400 SSV2
normal 80.0 60.1 reverse 79.8 23.9 random 79.7 39.4
Does the temporal order of input matter? We evaluate the SIFARB-12 model ( trained with input frames of normal order) by using three types of input with different temporal orders, i.e the reverse, random and normal. As can be seen from Table 9, SIFAR is not sensitive to the input order on Kinetics400, whereas on SSV2, changing the input order results in a significant performance drop. This is consistent with the finding in the S3D paper (Xie et al., 2018), indicating that for
datasets like SSV2 where there are visually similar action categories, the order of input frames matters in model learning.
What does SIFAR learn? One big advantage of our proposed approach is that many techniques developed in the image domain now can be directly used for video understanding without re-invention. Here we apply ablation cam (Desai & Ramaswamy, 2020), an image model interpretability technique, to understand what our models learn. Fig. 5 shows the Class Activation Maps (CAM) of 4 actions correctly predicted by SIFAR-B-12. Not surprisingly, the model learns to attend to objects relevant to the target action such as the hula hoop in a) and soccer ball in b). In c) and d), the model seems to correctly focus on where meaningful motion happens.

heading: 5 CONCLUSION
text: We have presented a new perspective for action recognition by casting the problem as an image recognition task. Our idea is simple but effective, and with one line of code to transform an sequence of input frames into a super image, it can re-purpose any image classifier for action recognition. We have implemented our idea with both CNN-based and transformer-based image classifiers, both of which show promising and competitive results on several popular public video benchmarks. Our experiments and results show that applying super images for video understanding is an interesting direction worth further exploration.

Review summaries:
The paper deals with action recognition in videos, i.e. detecting to which class a given sequence of frames belongs to. However, the paper proposes to explore whether an image classifier (instead of a video or spatiotemporal-based classifier) would already be enough to accomplish this task. In order to do so, the authors organize the frames from a video into a single image by organizing them into a grid, then proceed to learn them using Swin Transformers (Swin-B) image classification models.  The authors report surprising results which are indeed on-par or higher than the SotA in Kinetics400, MiT, Jester and Diving48 datasets.

The paper proposes to perform action recognition by first rearranging the frames from a video into a 3x3 or 4x4 grid to form a "super image", and then giving the super image to a standard image classifier to perform action recognition. Given that this super image will be a larger image, the paper leverages the more memory efficient Swin Transformer [1] as an image classifier to perform action recognition. Experiments on Kinetics400, Moments In Time, Something-Something V2 (SSV2), Jester and Diving48 show that the proposed method is on par or exceeds SOTA in terms of accuracy. On Kinetics400, the method not only is SOTA in terms of accuracy, but also is the most FLOPs-efficient method given a specific accuracy. The strong performance suggests that a deep network's ability to model spatial relationships could also be applied to model temporal relationships across frames in a video, which is an orthogonal direction to having explicit components in the network modeling temporal relationships. Furthermore, being able to connect action recognition with image classification enables existing image classification techniques to be applied to action recognition, which could potentially accelerate the field.  [1]: Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. arXiv.org, March 2021.