Can an Image Classifier Suffice For Action Recognition?
Authors: Quanfu Fan
Chun-Fu Chen
Rameswar Panda
Venue: ICLR 2022 Poster (2022)
Keywords: action recognition
image classifier
super image
vision transformer

Abstract: We explore a new perspective on video understanding by casting the video recognition problem as an image recognition task. Our approach rearranges input video frames into super images, which allow for training an image classifier directly to fulfill the task of action recognition, in exactly the same way as image classification. With such a simple idea, we show that transformer-based image classifiers alone can suffice for action recognition. In particular, our approach demonstrates strong and promising performance against SOTA methods on several public datasets including Kinetics400, Moments In Time, Something-Something V2 (SSV2), Jester and Diving48. We also experiment with the prevalent ResNet image classifiers in computer vision to further validate our idea. The results on both Kinetics400 and SSV2 are comparable to some of the best-performed CNN approaches based on spatio-temporal modeling. Our source codes and models are available at \url{https://github.com/IBM/sifar-pytorch}.

Published as a conference paper at ICLR 2022
CAN AN IMAGE CLASSIFIER SUFFICE FOR ACTION RECOGNITION?
Quanfu Fan; Chun-Fu (Richard) Chen," Rameswar Panda" MIT-IBM Watson AI Lab
gfandus com, chenrich@us. com.com, rpanda b... com
ABSTRACT
We explore a new perspective on video understanding by casting the video recog- nition problem
image recognition task. Our approach rearranges input video frames into super images, which allow for training an image classifier di- rectly fulfill the task of action recognition, in exactly the same way as image classification. With such simple idea, we show that transformer-based image classifiers alone can suffice for action recognition. demonstrates strong and promising performance against SOTA methods on several public datasets including Kinetics400, Moments In Time, Something-Somen V2 (SSV2), Jester and Diving48. We also experiment with the prevalent ResNet image classifiers in computer vision to further validate our idea. The results on both Kinetics400 and SSV2 are comparable to some of the best- rest-performed CNN approaches based on spatio-temporal modeling. Our source codes and models are available https: tttpttttttbutg/gto com/ TBM/ si far siifarrpttorre pytorch.
particular, our approach
INTRODUCTION
The recent advances in convolutional neural networks (CNNs) (He et al., 2016; Tan Le, 2019), along with the availability of large-scale video benchmark datasets (Kay et al., 2017; Monfort et al., 2019; Damen et al., 2020), have significantly improved action recognition, one of the fun- damental problems of video understanding. Many existing approaches for action recognition naturally extend bor- row ideas from image recognition. At the core of these approaches is spatio-temporal modeling, which regards time as an additional dimension and jointly models it with space by extending image models (i.e.. 3D CNNs) (Tran et al., 2015; Carreira et al., 2017; Feichtenhofer, 2020) or fuses temporal information with spatial information pro- cessed separately by 2D CNN models (Lin et al., 2019; Fan ct al., 2019). CNN-based approaches demonstrate strong capabilities in learning saptio-temporal feature rep- resentations from video data.
Top-1 Accuracy (9))
FLOPS (G)
Figure 1: Comparison of our proposed SI- FAR (red) with SOTA approaches for action recognition Kinetics400 dataset.
Videos present rongerange pixel interactions in both space and time. It's known in approaches like non-local networks (Wang et al., 2018) that modeling such relationships helps action recognition. The recently emerging Vision Transformers (ViTs) naturally dependencies data, making them very suitable for video understanding. Several approaches (Berta- sius al., 2021a; Li et al., 2021; Arnab al., 2021) have applied VIT's for action recognition and shown better performance than their CNN counterparts. However, these approaches are still following the conventional paradigm of video action recognition, and perform temporal modeling in a similar way to the CNN-based approaches using dedicated self-attention modules.
the strength of capturing long-range
In this work, we explore different perspective for action recognition by casting the problem as an image recognition task. We ask if it is possible to model temporal information with ViT directly
"qqual contribution.

---

Published as a conference paper at ICLR 2022
without using dedicated temporal modules. In other words, can image classifier alone suffice for action recognition? To this end. we first propose a simple idea to turn a 3D video into a 2D image. Given sequence of imput video frames, we rearrange them into super image according to a pre-defined spatial layout, as illustrated in Fig. 2. The super image encodes 3D spatio-temporal patterns in video into 21D spatial image patterns. We then train an image classifier to fulfill the task of action recognition, in exactly the same way as image classification. Without surprise, based on the concept of super images, any image classifier can be re-purposed for action recognition. For convenience, we dub our approach SIFAR, short for Super Image for Action Recognition.
We validate our proposed idea by using Swin Transformer (Liu et al., 2021), a recently developed vision transformer that has demonstrated good performance on both image classification and object detection. Since a super image has larger size than an input frame, we modify the Swin Transformer to allow for full self-attention in the last layer of the model, which further strengthens the model's s abil- ity in capturing long-range temporal relations across frames in the super image. With such change, we show that SIFAR produces strong performance against the existing SOTA approaches (Fig. I) on several benchmark datsets including Kinetics400 (Kay et al., 2017), Moments in Time (Monfort et al., 2019), Something Somettinng V2 (SSV2) Goyal et al. (2017), Jester (Materzynska et al., 2019) and Diving48 (Li et al., 2018). Our proposed SIFAR also enjoys efficiency in computation as well as in parameters. We further study the potential of CNN-based classifiers directly used for action recognition under the proposed SIFAR framework. Surprisingly, they achieve very competitive results on Kinetics400 dataset against existing CNN-based approaches that rely on much more sophisticated spatio- spatio-temporal modeling. Since 3 x 3 convolutions focus on local pixels only, CNN-based SIFAR handles temporal actions on Something-Somen sizes to expand the temporal receptive field of CNNs, which substantially improves the CNN-based SIFAR by 4% 6.8% with ResNet50.
less effectively. We experiment with larger kernel
SIFAR brings several advantages compared to the traditional spatio-temporal action modeling. Firstly. it is simple but effective. With one single line of code change in PyTorch, SIFAR can use any image classifier for action recognition. We expect that similar ideas can also work well with other video tasks such as video object segmentation (Duke et al., 2021). Secondly, SIFAR makes action modeling easier and more computationally efficient as it doesn't 't require dedicated modules for temporal modeling. Nevertheless, we do not tend to underestimate the significance of temporal modeling for action recognition. Quite opposite, SIFAR highly relies the ability of its backbone network to model long-range temporal dependencies in super images for more efficacy. Lastly, but not the least, the perspective of treating action recognition the same image recognition unleashes many possibilities of reusing existing techniques in a more mature image field to improve video understanding from various aspects. For example, better model architectures (Tan & Le, 2019), model pruning (Liu et al., 2017) and interpretability (Desai & Ramaswamy, 2020), to name a few.
2 RELATED WORK
Action Recognition from a Single Image. One direction for video action recognition is purely based on single image (Davis & Bobick, 1997; Zhao al., 2017: Safaei & Foroosh, 2019; Bilen et al., 2016). In (Davis& & Bobick, 1997), multiple small objects are first identified in a still image and then the target action is inferred from the relationship among the objects. Other approaches such as (Safaci & Foroosh, 2019) propose to predict the missing temporal information in still images and then combine it with spatial information for action classification. There are also approaches that attempt to summarize RGB or motion information in a video into representative image for action recognition, e.g., motion-energy image (MEI) (Davis & Bobick, 1997), Dynamic Image Network (Bilen et al., 2016), Informative Frame Synthesis (IFS) (Qiu et al., 2021). Adaptive Weighted Spatio-temporal Distillation (AWSD) (Tavakolian et al., 2019b) and Adversarial Video Distillation (AVD) (Tavakolian et al., 2019a). Nonetheless, our approach does not attempt to understand video from single input image or a summarized image. Instead our proposed approach composites the video into a super image, and then classifies the image with image classifier directly.
Action Recognition with CNNs. Action recognition is dominated by CNN-based models re- cently (Feichtenhofer et al., 2018; Carreira et al., 2017; Fan et al., 2019; Feichtenhofer, 2020; Chen et al., 2021; Lin et al., 2019; Wang et al., 2016; Zhou et al., 2018; Liu et al., 2020; Jiang et al., 201 9a; Tran et al., 2019). These models process the video cube to extract spatial-temporal
2

---

Published as a conference paper at ICLR 2022
Image Classifier
running jumping walking disc golfing
Figure 2: Overview of SIFAR. A sequence of input video frames are first rearranged into super image based on a 3 x 3 spatial layout, which is then fed into an image classifier for recognition.
features via the proposed temporal modeling methods. E.g., SlowFast (Feichtenhofer et al., 2018) proposes two pathways with different speed to capture short-range and long-range time dependen- cies. TSM (Lin et al., 2019) applies a temporal shifting module to exchange information between neighboring frames and TAM (Fan et al., 2019) further enhances TSM by determining the amount of information to be shifted and blended. On the other hand, another thread of work dynamically attempts to select the key frame of an activity for faster recognition (Wu et al., 2019; 2020; Meng al., 2020; 2021; Sun et al.. 2021). E.g., Adaframe (Wu al., 2019) employs a policy network to determine whether or not this is key frame, and the main network only processes the key frames. ARNet (Meng et al., 2020) determines what the image resolution should be used to save computations based on the importance of input frame images. Nonetheless, our approach is fundamentally different from conventional action recognition. It simply uses an image classifier as a video classifier by laying out a video to super image without explicitly modeling temporal information.
Action Recognition with Transformer. Following the vision transformer (ViT) (Dosovitskiy et al., 2021), which demonstrates competitive performance against CNN models image classification, many recent works attempt to extend the vision transformer for action recognition (Neimark et al., 2021; Li et al., 2021; Bertasius et al., 2021b; Arnab et al., 2021; Fan et al., 202 l). VTN (Neimark et al., 2021), VidTr (Li et al., 2021), TimeSformer (Bertasius et al., 2021b) and ViVIT (Arnab et al., 2021) share the same concept that inserts temporal modeling module into the existing VIT enhance the features from the temporal direction. E.g., VTN (Neimark et al., 2021) processes each frame independently and then uses longformer aggregate the features across frames. On the other hand, divided-space seeedd-pace-tnnne time modeling in TimeSformer (Bertasius et al., 2021a) inserts a temporal attention module into each transformer encoder for more fine-gr ane-grained temporal interaction. MViT (Fan et al., 2021) develops a compact architecture based on the pyramid structure for action recognition. It further proposes a pooling- pooling-based attention to mix the tokens before computing the attention map so that the model can focus more on neighboring information. Nonetheless, our method is straightforward and applies the Swin (Liu et al., 2021) model to classify super images composed from input frames.
Note that the joint-space-time attention in TimeSformer (Bertasius et al., 2021a) is special case of our approach since their method can be considered as flattening all tokens into one plane and then performing self- colf-attention over all tokens. However, the memory complexity of such an approach is prohibitively high, and it is only applicable to the vanilla ViT (Dosovitskiy et al., 2021) without inductive bias. On the other hand, our SIFAR is general and applicable any image classifiers.
3 APPROACH
3.1
OVERVIEW OF OUR APPROACH
The key insight of SIFAR to turn spatio-temporal patterns in video data into purely 2D spatial patterns in images. Like their 3D counterparts, these 2D patterns may not be visible and recognizable by human. However, we expect they are characteristic of actions and thus identifiable by powerful neural network models. To that end, we make sequence of input frame images from video into super image, as illustrated in Fig. 2, and then apply an image classifier to predict the label of the video. Note that the action patterns embedded in super image can be complex and may involve both local (i.e.. spatial information in a video frame) and global contexts (i.e., temporal dependencies across frames). It is thus understandable that effective learning can only be ensured by image classifiers with
3

---

Published as a conference paper at ICLR 2022
strong capabiliiiis in modeling short-range and long-range spatial dependencies in super images. For this reason, we explore the recently developed vision transformers based on self-attention to validate proposed idea. These methods come naturally with the ability model global image contexts and have demonstrated competitive performance against the best-performed CNN- approaches image classification well as action recognition. Next briefly describe Swin Transformer (Liu et al., 2021), an efficient approach that we choose to implement our main idea in this work.
Preliminary. The Vision Transformer (ViT) [13] is a purely attention-based classifier borrowed from NLP. It consists of stacked transformer encoders, each of which is featured with a multi-head self-attention module (MSA) and feed-forward network (FFN). While demonstrating promising results on image classification, VIT uses an isotropic structure and has quadruple complexity w.r.t image resolution terms of memory and computation. This significantly limits the application of ViT to many vision applications that requires high-resolution features such as object detection and segmentation. In light of this issue, several approaches (Liu al., 2021; Chu al., 2021; Zhang al., 2021) have been proposed to perform region-level local self-attention to reduce memory usage and computation, and Swin Transformer one of such improved vision transformers.
Swin Transformer (Liu et al., 2021) first adopts pyramid structure widely used in CNNs to reduce computation and memory. At the earlier layers, the network keeps high image resolution with fewer feature channels to learn fine- fine-grained information. As the network goes deeper, it gradually reduces spatial resolution while expanding feature channels to model coarse- caarse-graied information. To further memory, Swin Transformer limits self-attention non-overlapping local windows (W-MSA) only. The communications between W-MSA blocks is achieved through shifting them in the succeeding transformer encoder. The shifted W-MSA is named as SW-MSA Mathematically, the two consecutive blocks can be expressed as:
Yk W-MSA(LNN(X---)) +x-11 FFN(LN((H)) + Yk. Yk+11 SW-MSA(LN(X:)) + Xt.
X++1 = FFN(LN((R+1)) Yk+1,
where Xr is the features at the 1 layer and FFN and LN are feed-forward network and layer normalization, respectively.
SIFAR. In case, SIFAR learns action patterns by sliding window, as illustrated in Fig. 3. When the sliding window (blue box) within a frame, spatial dependencies are learned. On the other hand, when the window (red box) spans across frames, temporal dependencies between them effectively captured. The spatial pooling further ensures longer-range dependencies across frames captured.
Spertial Ponling
2
3
7
b)
2
3
2
5
6
7
4
5
Figure 3: Swin Transformer does self-attention in local window. In SIFAR, when the window (blue box) is within a frame, spatial dependen- cies are learned within a super image (4 frames here). When the window spans across differ- ent frames (red box), temporal dependencies be- tween them are effectively captured. The spatial pooling further ensures longer-range dependen- cies to be learnt. Best viewed in color.
c)
4
5
7
2
3
6
7
Figure Grid Layout. We apply grid to lay out the input frames. Illustrated here are several possible layouts for frames, i.e., a) 8, b) and c) 2 x 4, and d) x 3, respectively. Empty images are padded at the end if grid is not full.
Creation of Super Image. Given set of video frames, we order them by given layout (Fig. 4) to form large super image. Different layouts give different spatial patterns for an action class. We hypothesize that more compact structure such square grid may facilitate model learn temporal dependencies across frames as such shape provides the shortest maximum distance between any two images. Given input frames, create super image by placing all the frames in
4

---

Published as a conference paper at ICLR 2022
order onto grid of size (m 1) x when (m 1) x m x when 2 (m 1in x m where m = V Empty images are padded at the end if the grid is not full. With this method, for example, 12 frames will be fit into a 3 4 grid while 14 frames into 4 x 4 grid. In the default setting, we use a 3 x 3 layout for 8 images and a 4 x 4 one for 16 images, respectively. There are other well (see Fig. 4 for more examples). However our experiments empirically show that a square grid performs the best across different datasets. Note that our approach has linear computational complexity w.r.t the number of input frames. As described above, the size of a super image is m (m nz) times as large as the size of a frame image, suggesting that the total number of tokens (or image patches) in Swin grows linearly by n.
spatial arrangements
Sliding Window. As previously mentioned, Swin Transformer performs self-attention within small local window to save memory. It uses uniform window size across all layers, and the default window size is 7 in the original paper. Since a larger window leads to more interactions across frames, which is beneficial for SIFAR to learn long- Long-range temporal dependencies in super images, slightly modify the architecture of Swin Transformer (Liu et al., 2021) for it to take different window sizes flexibly in self-attention. In particular, we keep the same window size for all the layers except the last one, whose win- dow is as large as its image resolution, implying a global eeff-attention including all the tokens.
Table 1: Model architectures of SIFAR. The number in a model name indicates the window size used by the model before the last layer. "B" means Swin-B. f denotes the models using 16 frames as input and & indicates the models using a larger input image resolution.
Model
Frames
Image Size
FLOPs (G)
Window Size
SIFAR-B-7
224
138
[7.7.7.7]
SIFAR-B-I B-11
192
106
[12.12.12.18]
SIFAR-B-14
224
147
[14,14.1422]
SIFAR- B- 12F
16
192
189
[12,12,12,24)
SIFAR B- 14)
16
224
263
[14,14,14,28)
SIFAR-BI B- B-12;
384
423
(12,12,12,36)
Since the last layer has only two transformer encoders, the computational overhead imposed by an increased window size quite small, as indicated Table 1.
The change of window size may result in adjustment of the input image size as the image resolution at each layer must be divisible by the window size in Swin Transformer. As noted in Table 1, SIFAR-B-7 keeps the vanilla architecture of Swin-B. SIFAR-B-11 12 is more efficient than SIFAR-B-7 because SIFAR-B- B-RR 12 takes smaller images (192?) as input. We demonstrate later in Sec. 4 that a larger window is critical for SIFAR to achieve good performance more temporal datasets such SSV2.
Implementation, Once the spatial layout for the input frames is determined, implementing our idea in Py ToTrhh is as simple as inserting into an image classifier the following line of code, which changes the input of a video to a super image.
create a super image with a Layout (sh, Sw) pre-specitied by the user. = rearrange (x, "b (sh sw) h w -> b a (sh h) (sw w)', sh=sh, sw=sw)
The trivial code change described above transforms an image classifier into an video action classifier. Our experiments show that the same training and evaluation protocols for action models can be still applied to the repurposed image classifier.
4 EXPERIMENTS
4.1 DATASETS AND EXPERIMENTAL SETUP
Datasets. We use Kinetics400 (K400) (Kay et al., 2017). Something--Something V2 (SSV2) (Goyal et al., 2017), Moments-in-time (MiT) (Monfort et al., 2019), Jester (Materzynska et al., 2019), and Diving48 (Li et al., 2018) datasets in our evaluation. Kinetics400 is a widely-used benchmark for action recognition. which includes ~40k training vidcos and 20k validation videos in 400 classes. SSV2 contains 220k videos of 174 types of predefined human-object interactions with everyday objects. This dataset is known for its high temporal dynamics. MiT is fairly large collection of one million 3-second labeled video clips, involving actions not only from humans, but also from animals, objects and natural phenomena. The dataset includes around 800k training videos and 33,900 validation videos in 339 classes. Jester contains actions of predefined hand gestures, with 118,562 and 14,787 training and validation videos over 27 classes, respectively. Diving48 is action recognition dataset without representation bias, which includes 15,943 training videos and 2,096 validation videos over 48 action classes.
5

---

Published as a conference paper at ICLR 2022
Training. We employ uniform sampling generate video input for our models. Such sampling strategy divides a video into multiple segments of equal length, and has shown to be effective on both Kinetics400 and SSV2 Chen et al. (2021). We train all our models by finetuning Swin-B model (Liu et al., 2021) ) pretrained on ImageNet-21K (Deng et al., 2009), except for those SSV2 models, which fine tuned from the corresponding Kinetics400 models in Table 3.
Our training recipes and augmentations closely follow DeiT (Touvron et al., 2020). First, we apply multi-scale jitter to augment the input (Wang al., 2016) with different scales and then randomly crop a target input size (e.g. 8x224x22 x 224 for SIFAR-B-7). We then use Mixup (Zhang et al., 2018) and CutMix (Yun et al., 2019) to augment the data further. with their values set 0.8 and 1.0, respectively. After that, we rearrange the image crops as a super image. Furthermore, we apply drop path (TTn & Le, 2019) with a rate of 0.1, and enable label smoothing (Szegedy et al., 2016) at a rate of 0.1. All our models were trained using V100 GPUs with 16G or 32G memory. Depending on the size of a model, we use a batch size of 96, 144 or 192 to train the model for 15 epochs on MiT or 30 epochs on other datasets, including 5 warming-up epochs. The optimizer used in our training is A (Loshchilov & Hutter, 2019) with a weight decay of 0.05, and the scheduler is Cosine (Loshchilov & Hutter, 2017) with base linear learning rate of 0.0001.
Inference. We first scale the shorter side of an image to the model input size and then take three crops (top-left, center and bottom-right) for evaluation. The average of the three predictions is used as the final prediction. We report results by top-1 and top-5 classification accuracy (%) on validation data, the total computational cost in FLOPs and the model size in number of parameters.
4.2
MAIN RESULTS
Comparison with Baselines. We first compare
Table 2: Comparison with Baseline Meth-
approach with several representative CNN-
ods. All models use 8 frames as input.
based methods including I3D (Carreira et al., 2017).
TSM (Lin et al., 2019) and TAM (Fan et al., 2019). Also included in the comparison are two TimeS- former models (Bertasius et al., 2021a) based on the same backbone Swin-B (Liu et al., 2021) as used by our models. All the models considered take 8 frames as input. As can be seen from Table 2, our ap- proach substantially outperforms the CNN baselines on Kinetics400 while achieving comparable results on SSV2. Our approach is also better than TimeS- former on both datasets. These results clearly demon-
Model
SSV2
Kinetics400
Top-1
Top-5
Top-1
Top-5
13D-R50
61.1
86.5
72.6
90.6
TSM-R50
59.1
85.6
74.1
91.2
TAM-R50
62.0
87.3
72.2
90.4
TimeSformer*
35.9
71.1
77.5
92.5
TimeSformer"*
58.7
85.9
80.1
94.4
SIFAR-BE B-7
59.0
86.0
79.6
94.4
SIFAR-B-12
60.8
87.3
80.0
94.5
SIFAR-B- B-14
61.6
87.9
80.2
94.4
Swin-B (space only). "": Swin-B (divided space-time).
strate that a powerful image classifier like Swin Transformer can learn expressive spatio-temporal patterns effectively from super images for action recognition. In other words, image classifier can suffice video understanding without explicit temporal modeling. The results also confirm that a larger sliding window is more helpful in capturing temporal dependencies on temporal datasets like SSV2. Our approach performs global self-attention in the last layer of a model only (see Table 1). This substantially mitigates the memory issue in training SIFAR models.
Kinetics400. Table 3 shows the results on Kinetics400. Our 8-frame models (SIFAR-13 12 and SIFAR- 14) achieve 80.0% and 80.2% top-1 accuracies, outperforming all the bNNe approaches while being more efficient than the majority of them. SIFAR-B-141 further gains ~ 1.8% improvement, benefiting from more input frames. Especially, SIFAR-L-I2t yields an accuracy of 84.2%, the best among all the very recently developed approaches based on vision transformers including TimeSformer (Bertasius et al., 202 and MViT-B (Fan al., 2021). Our proposed approach also offers clear advantages in terms of FLOOPs and model parameters compared to other approaches except MVīT-B. For example, SIFAR-B-12H has 5x and 37x fewer FL.PPs than TimeSformer-L. and ViViT-L, respectivly, while being 1.4x and 3.6x smaller in model size.
SSV2. Table lists the results of our models and the SOTA approaches on SSV2. With the same number of input frames, our approach is 1 ~ 2% worse than the best-performed CNN methods. However, our approach performs on par with other transformer- TimeSformer (Bertasius et al., 2021a) and VidTr-L (Li et al., 2021) under the similar setting. Note that ViViT-L (Amab et al., 2021) achieves better results with a larger and stronger backbone ViT- L (Dosovitskiy et al., 2021). MVTT-B (Fan et al., 2021) is an efficient multi-scale architecture,
method such as
6

---

Published as a conference paper at ICLR 2022
Table 3: Comparison with Other Approaches on Kinetics400.
Model
#Frames
Pretrain
Purams(M)
FLOPs(G)
Top-1
Top-5
TSN-R50 (Wang et al., 2016)
32
IN-IK
24.3
170.8x
69.8
89.1
TAM-R50 (Fan al., 2019)
IN-IK
171.5x30
76.2
I3D-R50 (Carreira et al., 2017)
32
IN-IK
47.0
335.3x30
76.6
92.7
13D-R50+NL (Wang et al., 2018)
32
IN-IK
282x30
76.5
92.6
13D--R01++NL (Wang
2018)
IN-1K
71.7
ip-CSN-152 (Tran al. 2019)
32
32.8
109x30
77.8
92.8
SlowFust8x8 (Feichtenhofer et al., 2018)
32
27.8
63.7x
77.0
92.6
Slow (Feichtenhooer
2018)
78.7
Flow (Feichtenhofer al., 2018)
64
59.9
234x30
79.8
93.9
X3D-M (Feichtenhofer, 2020)
16
3.8
6,2x30
76.0
92.3
X3D-XL (Feichtenhofer, 2020)
38
79.1
TPN101 (Yang al., 2020)
32
374x30
78.9
93.9
VTN-VIT-B (Neimark et al., 2021)
250
IN-2IK
114.0
4218x1
78.6
93.7
VidTr-L (Li et ul. 2021)
32
IN-2IK
x3
79.1
93.9
TimeSformer (Bertasius al,, 2021b)
IN-21K
196x3
78.0
TimeSformer-HR (Bertasius al., 202
16
IN-2IK
121.4
1703
79.7
TimeSformer-L (Bertasiuus et al., 2021b)
96
IN-2IK
121.4
2380x3
N0.7
VIVIT-L (Amab al., 2021)
IN-2IK
310.8
3992x
81.3
94.7
MVTT-B (Fan al., 2021)
36.6
70.5x
78.4
93.5
MVTE-B (Fam et al., 2021)
64
36.6
455x9
81,2
95.1
SIFAR B--1
8
IN-21K
87
106x3
80.0
94.5
SIFAR-B-12f
IN-21K
189x3
80.4
94.4
SIFAR-B-14
IN-21K
147x3
80.2
94.4
SIFAR-B-14
16
IN-21K
87
263x3
81.8
93.2
SIFAR-L-I4f
IN-21K
576x3
82.2
SIFAR-B-12E
IN-2IK
423x3
81.6
95.2
SIFAR-L.-12
IN-21K
196
944x3
84.2
96.0
Table 4: Comparison with Other Approaches on SSV2.
Model
#Frames
Params(M)
FLOPs(G)
Top-1
Top-5
TAM-R50 (Fan al., 2019)
8
24.4
42.9x6
62.8
87.4
TAM-R50 (Fan et al., 2019)
32
24.4
171.5x6
63.8
88.3
13D-R50 (Carreira et al., 2017)
47.0
83.8x6
61.1
86.5
I3D-R50 (Carreira al. 2017)
336.3x6
62.8
88.0
TSM-R50 (Lin et al., 2019)
8
24.3
32x6
59.1
85.6
TSM-R50 (1inn et al., 2019)
16
24.3
65x6
63.4
88.5
TPN-R50 (Yang et al., 2020)
8
62.0
TAM-bLR101
2019)
96.4x
65.2
90.3
MSNet wK al., 2020)
16
24.6
67x
64.7
89.4
STM (Jiang ct al., 2019b)
16
24.0
67x30
64,2
89,8
TEA (Liu al., 2020)
16
70x30
89.9
Tioeerormer (Bertasius et al., 2021b)
8
1214
196x3
59.5
TimeSformer-H HR (Bertasius al., 2021b)
16
121.4
1703x3
62.5
VIVIT-I (Armah et al., 2021)
32
100.7
65.4
89.8
VidTr- (Li al., 2021)
32
60.2
MVIT-B (Fan al., 2021)
16
36.6
70.5x
64.7
89.2
MVGT-B (Fan al., 2021)
36.6
455x3
67.7
90.9
SIFAR-B-12
106x3
60.8
87.3
SIFAR-B-122
16
87
189x3
61.4
87.4
SIPAR-B-14
8
87
147x3
61.6
87.9
SIFAR-B-14 B-14
16
263x3
62.6
88.5
SIFAR-L-147
64.2
88.4
which can process much longer input sequences to capture fine-grained motion patterns in SSV2 data. Training SIFAR models with more than 16 frames still remains computationally challenging, especially for models like SIFAR-B-14 14 and SIFAR-L-144, which need larger sliding window size. Our results suggest that developing more efficient architectures of vision transformer be an area of improvement and future work for SIFAR to take advantage of more input frames SSV2.
MIT. MIT is a large diverse dataset containing label noise. As seen from Table 5, with the same backbone ViT-L, SIFAR-L-121 is ~4% better than ViVIT-L (Arnab et al., 2021), and outperforms AssembelNet (Ryoo et al., 2020) based on neural architecture search by considerable margin of 8%.
Jester and Diving48. We further evaluate proposed approach two other popular benchmarks: Jester (Materzynska et al., 2019) and Diving48 (Li et al., 2018). Here we only consider the best single models from other approaches for fair comparison. As shown in Table 6, SIFAR achieves competitive
7

---

Published as a conference paper at ICLR 2022
Table 5: Comparison with Other Methods on MiT.
Model
Top-1
Top-5
TRN-Incpetion (Zhou et al., 2018)
28.3
53.9
TAM-R50 (Fan et al., 2019)
30.8
58.2
13D-R50 (Chen et al., 2021)
31.2
58.9
Faot (Feichtenhofer et al., 2018)
31.2
58.7
CoST-R 101 (Li et al., 2019)
32,4
60.0
SRTG-R3D--001 (Stergiou & Poppe, 2020)
33.6
58.5
AssembleNet (Ryoo et al., 2019)
33.9
60,9
ViViT-L (Amab et al., 2021)
38.0
64.9
SIFAR-B-12g
39,9
69.2
SIFAR-L-12
41.9
70.3
Table 6: Comparison with Other Approaches on Jester and Diving48.
(a) Jester
(b) Diving48
Model
Top-1
Top-5
JSN--nception (Wang et al., 2016)
95.0
99.9
Model
Top-1
Top-5
IRN-Inception (Zhou al. 2018)
95.3
TSM-R50 (Lin et al., 2019)
95.0
99.9
Timmer (Bertasius 2021b)
74.9
PAN-R50 (Zhang
99.6
99.8
TimeSformer-tH HR (Bertasius et al., 2021a)
78.0
STM-RS0(Jiag et al., 2019n)
96.7
99.9
TimeSformer-L (Bertasius et al., 202
81.0
13D BDD-R50 (Cameira et 2017)
96.4
SlowFast (Feichtenhofer et al., 2018)
77.6
TAM-R50 R50 (Fan et al., 2019)
96.4
SIFAR-B-121
85.3
98.3
SlowFast R50-8x (Feichtenhofer al., 2018)
96.8
SIFAR B-144
87.3
98.8
BIFAR-B-12 SIFAR-B-141
97.2
99.9
97.2
99.9
results again both datasets, surpassing all other models in comparison. Note that the Diving48 benchmark contains videos with similar background and objects but different action categories, and is generally considered as unbiased benchmark. Our model SIFAR-B-147 outperforms TimeSformer-L by a large margin of 6% on this challenging Diving48 dataset.
Classification by CNNs. We also test our proposed approach using the ResNet image classifiers on both SSV2 and Kinetics400 datasets. For fairness, the ResNet models are pretrained on ImageNet- 21K. Table shows the results. Our models clearly outperform the traditional CNN-base models for action recognition on Kinetics400. Especially, with strong backbone R152x2 (a model 2x wider than 152), SIFAR-R1522 achieves a superior accuracy of 79.0%, which surprisingly comparable to the best CNN results (SlowFast 66X8+NL 79.8%) reported in Table 3.
On SSV2, the results of CNN-based SIFAR are less satisfactory but reasonable. This is because 3x3 convolutions are local with a small recep- tive field, thus failing to capturing long-range temporal dependencies in super images. We hy- pothesize that larger kernel size with wider receptive field may address this limitation and potentially improve the performance of CNN- based SIFAR models. To validate this, we per- form additional experiments by adding one or two more residual blocks to the end of ResNet models with larger kernel sizes, i.c. replacing the second convolution in those new blocks by 7x7, 11x11 or 21x21 kernel. These models are indicated by names ending with "C7" (7x7), "C11" (11x11) or "C21" (21x21) in Table 7. As seen from the table, using larger kernel sizes consistently improves the results on both ResNet50 and ResNet101 models. For example, we obtain an absolute 5.0% improvement over original ResNet50 and 2.0% over original ResNet101 respectively, using one more block with a kernel size of 21x21. When adding another block with a kernel size 11x1l and LIFA-R101-C21-1
Table 7: CNN-based SIFAR Results
Model
# Frames
SSV2
Kinetiest00
13D-R50 (Cameim
72.6
TSM-R50 (Wang et al., 2016)
59.1
74.1
TAM R50 (Fan et al., 2019)
62.0
72.2
SIHAR-R50
73.2
SIFAR-RIO]
56.3
76.6
SIFAR R152xx2
58.2
79.0
R50-C7
54.4
(+1.)
SHARRRS5--C
58.2 (14.2)
74.5 (+1.3)
DIFAR-R5--CO
55.8 (+5.0)
74,8 (+1.6)
SIFAR R50-C21-11
SIFAR-RIO1 -21
58.1 (+1.8)
71.7 (+1.1)
SIFAR-RI0] R101-C21-11
59.6 (+3.3)
77.5
modell two times Wider than R152
(i.e. SIFA-R50-C21- ResNet50 and 2.7% with ResNet101. These results strongly suggest that expanding the receptive field of CNNs be promising direction design better CNN-based SIFAR models.
it further boosts the performance up to 6.8% with
8

---

Published as a conference paper at ICLR 2022
Table 8: Ablation Study. The effects of each component on model accuracy.
(a) Super Image Layout. (SIFAR-B B- 12 on SSV2)
(b) Absolute Positioning Embedding.
Layout
Top-1
Top-5
Model
SSV2
Kinetics400
1 x (Fig. 4a)
44.4
74.4
APE
w/o APE
APE
w/o APE
2x4 (Fig. 4b)
58.6
85.5
SIFAR-B-7
56.6
56.4
79.7
79.6
2x4 (Fig. 4c)
58.1
85.1
SIFAR-B-12
60.8
59,5
79.7
80.0
SIFAR-B-14
61.6
60.1
80.0
80.2
3x3 (Fig. 4d)
60.8
87.3
4.3 ABLATION STUDIES
How does an image layout affect the performance? The layout of a super image determines how spatio-temporal patterns are embedded in it. To analyze this, we trained a SIFAR model on SSV2 for each layout illustrated in Fig. 4. As shown Table 8a, a strip layout performs the worst while a grid layout produces the best results, which confirms hypothesis.
Does absolute positioning embedding help? Swin paper (Liu et al., 2021) shows that when relative position bias added, Absolute Position Embedding (APE) is only moderately beneficial for classification, but not for object detection and segmentation. They thus conclude that inductive bias that encourages certain translation invariance still important for vision tasks. To find out whether or not APE is effective in our proposed approach, we add APE to each frame rather than each token. The results in Table 8b indicate that APE slightly improves model accuracy on SSV2, but is harmful to Kinetics400. In our main results, we thus apply APE to SSV2 only.
Does the temporal order of input matter? We evaluate the SIFAR-B-12 model (trained with input frames of normal order) by using three types of input with different temporal orders, i.e the reverse, random and normal. As shown in Table 9, SIFAR not sensitive to the input order on Kinetics400, whereas on SSV2, changing the input order results in a significant perfor- mance drop. This is consistent with the finding in the S3D paper (Xie et al., 2018), indicating that for datasets like SSV2 where there visually similar action categories, the order of input frames matters in model learning.
Table 9: Effects of temporal order.
Order
Kinetics400
SSV2
normal
80.0
60.8
reverse
79.8
23.9
random
79.7
39.4
Hula hooping
Kicking
Lifting up something lesting it down
Turning something upside down
Figure Visualization by Ablation CAM (Desai & Ramaswamy, 2020)
What does SIFAR learn? We apply ablation CAM (Desai Ramaswamy, 2020), an image model interpretability technique, to understand what our models learn. Fig. shows the Class Activation Maps (CAM) of 4 actions correctly predicted by SIFAR-B. 1-2 Not surprisingly. the model learns to attend to objects relevant to the target action such as the hula hoop in a) and soccer ball in b). In c) and d), the model seems to correctly focus on where meaningful motion happens.
5 CONCLUSION
In this paper, we have presented new perspective for action recognition by casting the problem as an image recognition task. Our idea is simple but effective, and with one line of code to transform an sequence of input frames into super image, it can re-purpose any image classifier for action recognition. We have implemented our idea with both CNN-based and transformer based image classifiers, both of which show promising and competitive results several popular publicly available video benchmarks. Our extensive experiments and results show that applying super images for video understanding an interesting direction worth further exploration.

---

Published as a conference paper at ICLR 2022
REFERENCES
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: A video vision transformer, 2021.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is Space- Time Attention All You Need for Video Understanding? arXiv.org, February 2021a.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. space-time attention all you need for video understanding?, 2021b.
Hakan Bilen, Basura Fernando, Efstratios Gavves, Andrea Vedaldi, and Stephen Gould. Dynamic image networks for action recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3034-3042, 2016.
Joao Carreira, Andrew Zisserman, and xxx. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, pp. (299-6308, 2017.
Chun-Fu Chen, Rameswar Panda, Kandan Ramakrishnan, Rogerio Feris, John Cohn, Aude Oliva, and Quanfu Fan. Deep analysis of enn-based spatio-temporal representations for action recognition, June 2021.
Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting Spatial Attention Design in Vision Transformers. arXiv.org, April 2021.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Mumro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision. CoRR, abs/2006. 2020. URL https: //aa:///aaivva org/ abs hasgg//bb//200 13256.
James Davis and Aaron Bobick. The representation and recognition of action using temporal templates. In Proceedings of the IEEE International Conference Computer Vision, pp. 2736-2744, 1997.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-24882 255. leee, 2009.
Saurabh Desai and Harish G. Ramaswamy, Ablation-cam: Visual explanations for deep convolutional network via gradient-free localization. In 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 972-980, 2020. doi: 1100/ACY4557222200099933
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021. URL https: /ttpp::/opeee https://openreviii net/ / forum? i dNTTy.
Brendan Duke, Abdalla Ahmed, Christian Wolf, Parham Aarabi, and Graham W. Taylor. SSTVOS: sparse spatiotemporal transformers for video object segmentation. CoRR, abs/2101.08833, 2021. URL httpt:tt/////c http::///xxvvvvo http://v..rra/b////////// /ass/2111 08833.
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li. Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers, 2021.
Quanfu Fan, Chun-Fu (Ricarhd) Chen, Hilde Kuehne, Marco Pistoia, and David Cox. More Is Less: Learning Efficient Video Representations by Temporal Aggregation Modules. In NewrIPS, 2019.
Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In CVPR. June 2020.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. arXiv: /8/2,0998 2018.
10

---

Published as a conference paper at ICLR 2022
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Sueller-Freettge et al. The" something something" video database for learning and evaluating visual common sense. In ICCV, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Boyuan Jiang, Meng Wang, Weihao Gan, Wei Wu, and Junjie Yan. Stm: Spatiotemporal and motion encoding for action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019a.
Boyuan Jiang. Mengmeng Wang, Weihao Gan, Wei Wu, and Junjie Yan. Stm: Spatiotemporal and motion encoding for action recognition. In 2019 IEEE/CVF International Conference Computer Vision (ICCV), pp. 2000-2009, 2019b. doi: 10. 109/ICCV.2019.000000
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, al. The kinetics human action video dataset. vrXiv:/705.06900 2017.
Heeseung Kwon, Manjin Kim, Suha Kwak, and Minsu Cho. Motionsqueeze: Neural motion feature learning for video understanding. In ECCV, 2020.
Chao Li, Qiaoyong Zhong, Di Xie, and Shiliang Pu. Collaborative Spatiotemporal Feature Learning for Video Action Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Xinyu Li, Yanyi Zhang, Chunhui Liu, Bing Shuai, Yi Zhu, Biagio Brattoli, Hao Chen, Ivan Marsic, and Joseph Tighe. VidTr: Video Transformer Without Convolutions. arXiv, 2021.
Yingwei Li, Yi Li, and Nuno Vasconcelos. Resound: Towards action recognition without representa- tion bias. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
Lin, Chuang Gan, and Song Han. Temporal Shift Module for Efficient Video Understanding. In ICCV, 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, arXiv.org, March 2021.
Zhaoyang Liu, Donghao Luo, Yabiao Wang. Limin Wang. Ying Tai, Chengjic Wang. Jilin Li, Feiyue Huang, and Tong Lu. TEINet: Towards Efficient Architecture for Video Recognition. Proceedings of the AAA/ Conference on Artificial Intelligence, 34(07): 1669-11676 April 2020.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2736 2744, 2017.
Nya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In Interna- tional Conference on Learning Representations, 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL https: http::///peeeeeeeeee net wwt.:neentttatteee forum?ide Bkg6RiCqY7.
Joanna Materzynska, Guillaume Berger, Ingo Bax, and Roland Memisevic. The jester dataset: A large-scale video dataset of human gestures. In ICCV Workshops, Oct 2019.
Yue Meng, Chung- Ching Lin, Rameswar Panda, Prasanna Sattigeri, Leonid Karlinsky, Aude Oliva, Kate Saenko, and Rogerio Feris. Ar-net: Adaptive frame resolution for efficient action recognition. arXiv preprint 2077 2020.
11

---

Published as a conference paper at ICLR 2022
Yue Meng. Rameswar Panda, Chung- Chung-Ching Ching Lin, Prasanna Sattigeri, Leonid Karlinsky, Kate Saenko, Aude Oliva, and Rogerio Feris. Adafuse: Adaptive temporal fusion network for efficient action recognition. arXiv preprint arXiv:2102.05775, 2021.
Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Yan Yan, Lisa Brown, Quanfu Fan, Dan Gutfreund, Carl Vondrick, et al. Moments in time dataset: one million videos for event understanding. JEEE TPAMI, 2019.
Daniel Neimark, Omri Bar, Maya Zohar, and Dotan Asselmann. Video transformer network, 2021.
Zhaofan Qiu, Ting Yao, Yan Shu, Chong Mhong-Wahh Wah Ngo, and Tao Mei. Condensing a sequence to one informative frame for video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 16311-16320 October 2021.
Michael S. Ryoo, A. J. Piergiovanni, Mingxing Tan, and Anelia Angelova. Assemblenet: Searching for multi-stream neural connectivity in video architectures. CoRR, abs/1905.... 13209, 2019. URL http:// /ttp:////xxivi wwwoorrg/ss..... 905. 13209.
Michael S. Ryoo, AJ Piergiovanni, Mingxing Tan, and Anelia Angelova. Assemblenet: Searching for multi-stream neural connectivity video architectures. In International Conference on Learning Representations, 2020. URL https: wwwpoppeeeeyyeeee wee.riiwwweeeeeeeee..eee
net forum idd SJgKK Ywr.
Marjaneh Safaci and Hassan Foroosh. Still image action recognition by predicting spatial- -ential-emmpoal pixel evolution. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 111-120, 2019, doi: 10. ww..109//AC..201.000000
Alexandros Stergiou and Ronald Poppe. Learn to cycle: Time-consistent feature discovery for action recognition. arXiv, 2020. doi: 10.1016/j, pttp:p/atrc..02....... 11.012.
Ximeng Sun, Rameswar Panda, Chun-Fu Richard Chen, Aude Oliva, Rogerio Feris, and Kate Saenko. Dynamic network quantization for efficient video inference. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7375-7385, 2021.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Mingxing Tan and Quoc Le. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, pp. 6105-6114, Long Beach, California, USA, June 2019, PMLR.
Mohammad Tavakolian, Mohammad Sabokrou, and Abdenour Hadid. AVD: adversarial video distillation. CoRR, abs/1907.005400 2019a. URL http: /ttppi:////xvv..//p org/ abs 1907. 05640.
Mohammad Tavakolian, Hamed R. Tavakoli, and Abdenour Hadid. Awsd: Adaptive weighted spatiotemporal distillation for video representation. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), October 2019b.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012.12871 2020.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning Spa- tiotemporal Features With 3D Convolutional Networks. In ICCV, 2015.
Du Tran, Heng Wang, Lorenzo Torresani, and Matt Feiszli. Video classification with channel- separated convolutional networks. In ICCV, October 2019.
Limin Wang, Yuanjun Xiong, Zhe Wang. Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In ECCV. Springer, 2016.
12

---

Published as a conference paper at ICLR 2022
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, June 2018.
Zuxuan Wu, Caiming Xiong, Chih- Ma, Richard Socher, and Larry S. Davis, Adaframe: Adaptive frame selection for fast video recognition. In 2019 IEEE/CVE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 1278-1287, 2019. doi: 10.1 21109//VR.2019.0000
Zuxuan Wu, Caiming Xiong, Yu-Gang Jiang, and Larry S Davis. LiteEval: A Coarse-to-Fine Frame- work for Resource Efficient Video Recognition. In Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2020. URL https: httpp//pceeiinn..neeeee neurips. 5d5 90259922el11 1ete2e11003300 00d3d2 (0t.0022220007777772220-220
oc httppapprr/2111
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification. In ECCV, September 2018.
Ceyuan Yang. Yinghao Xu, Jianping Shi, Bo Dai, and Bolei Zhou. Temporal pyramid network for action recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Can Zhang, Yuexian Zou, Guang Chen, and Lei Gan. PAN: Towards Fast Action Recognition via Learning Persistence of Appearance. arXiv, 2020.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018. URL https:
h //openreview.nee net forum? i dd==r1dp1-rddd
Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi-Scale Vision Longformer: A New Vision Transformer for Biigh-Resolution Image Encoding. arXiv:org, March 2021.
Zhichen Zhao, Huimin Ma, and Shaodi You. Single image action recognition using semantic body part actions. In The IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In ECCV, pp. 803-818, 2018.
13


Reviews:
[{"review_id": "iK2zDfLA81n", "reviewer": "Reviewer_7Wf5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n\nThe paper reads well and has almost no typos. The method described is simple and achieves surprising results. The authors have provided extra ablation experiments to evaluate the importance of the grid layout, APE and temporal order of the frames in the grid, as well as activation visualizations using CAM. The experimental setup is clear and results seem convincing.\n\n# Weaknesses\n\na) It is not entirely clear how the frames are sampled before they are organized in a grid. For example, in other methods such as I3D, consecutive frames are sampled 10 frames apart [A, sections 2.3 and 2.5]. However, the current paper states uniform sampling is used to generate the video input for the models [p.5]. Does it mean the frames are uniformly sampled from the start and ending frames of the entire video, or are they sampled considering a fixed skip with a random starting frame as in I3D?\n\nb) It would seem to me that both the super image representation and the use of a transformer-based architecture are essential in order to achieve such good results due the self-attention mechanism. Have the authors experimented with non-transformer architectures to evaluate the efficacity of the super image representation by itself?\n\nminor) \"Our\" should be capitalized in second paragraph of page 5, or merged with the previous paragraph.\n\n[A]: Carreira et al, Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset, CVPR 2017.", "summary_of_the_paper": "The paper deals with action recognition in videos, i.e. detecting to which class a given sequence of frames belongs to. However, the paper proposes to explore whether an image classifier (instead of a video or spatiotemporal-based classifier) would already be enough to accomplish this task. In order to do so, the authors organize the frames from a video into a single image by organizing them into a grid, then proceed to learn them using Swin Transformers (Swin-B) image classification models.\n\nThe authors report surprising results which are indeed on-par or higher than the SotA in Kinetics400, MiT, Jester and Diving48 datasets.", "summary_of_the_review": "The paper presents a simple yet effective idea to transform simpler image classification models into video classification models. Even being simple, the approach manages to achieve surprisingly good results when compared to SotA video-classification models which explicitly handle the temporal dimension. The paper may thus contain findings that should be of interest for the ICLR community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cGE4HvsZ6j2", "reviewer": "Reviewer_JuKq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In this paper, the authors propose a simple but effective approach for video action recognition by casting the problem as an image recognition task. Different from modeling temporal information, it provides a different perspective to think about the action recognition task. \n\nIt provides solid experiments and ablation studies to evaluate the effectiveness of the proposed approach. Transformer-based and CNN-based models are both tested on public benchmarks and good experimental results are reported.\n\n", "summary_of_the_paper": "This paper presents a new approach for video action recognition by casting the problem as an image recognition task. The video clips are rearranged into a super image according to a pre-defined spatial layout. ", "summary_of_the_review": "This paper provides a novel idea for video action recognition. Their claims are well supported by solid experiments and ablation studies. I believe it would inspire others in this research field.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RaOjVhd1Dsy", "reviewer": "Reviewer_K68r", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n- The proposed method (Super Image for Action Recognition, SIFAR) is extremely simple and easy to implement (one liner in pytorch). Its accuracy is on par with SOTA, and its speed is also SOTA given a fixed accuracy threshold.\n- Taking deep networks' ability to model spatial relationships and then using it to model temporal relationships is a very interesting direction. This is an orthogonal approach than having explicit components for modeling temporal relationships. I was surprised that it worked so well.\n- Being able to connect image classification and action recognition can enable image classification techniques to be applied to action recognition, which could potentially accelerate the field.\n- The ablation studies on layout and ordering are interesting.\n- The writing is very clear. The writing also clearly mentions the limitations, such as \"*Training SIFAR models with more than 16 frames still remains computationally challenging, especially for models like SIFAR-B-14 and SIFAR-L-14†, which need a larger sliding window size.*\"\n\n### Weakness\n- From the analysis done on SSV2, it seems like the bigger limitations of SIFAR are (1) it has difficulty taking in more than 16 images at once, and (2) it is unclear what is the limit of SIFAR in terms of doing fine-grained temporal modeling. However, I don't think these limitations prevent SIFAR from being an interesting idea.\n- \"*Note that the joint-space-time attention in TimeSformer (Bertasius et al., 2021a) is a special case of our approach since their method can be considered as flattening all tokens into one plane and then performing self-attention over all tokens. However, the memory complexity of such an approach is prohibitively high...*\" I am not sure I agree with this claim. In figure 3 of (Bertasius et al., 2021a, https://arxiv.org/pdf/2102.05095.pdf), the method can use up to 96 frames with the \"Divided Space-time\" approach, which is less memory intensive than SIFAR, which has difficulty going more than 16 frames at once. \n- In Table 9, SSV2 in reverse order leads to the worst performance, which is surprising. I would expect that normal and reverse ordering should lead to the same performance, as whether the first image is at the top left or bottom right should not affect the network’s ability to learn as long as if the network is given consistently ordered input. Can the authors elaborate on this? Thank you!\n\n\n### Typos\n- P5: “o”ur approach has linear computational complexity… (capitalize)\n- P5: We demonstrate later in Sec. 4 “thata” larger window", "summary_of_the_paper": "The paper proposes to perform action recognition by first rearranging the frames from a video into a 3x3 or 4x4 grid to form a \"super image\", and then giving the super image to a standard image classifier to perform action recognition. Given that this super image will be a larger image, the paper leverages the more memory efficient Swin Transformer [1] as an image classifier to perform action recognition. Experiments on Kinetics400, Moments In Time, Something-Something V2 (SSV2), Jester and Diving48 show that the proposed method is on par or exceeds SOTA in terms of accuracy. On Kinetics400, the method not only is SOTA in terms of accuracy, but also is the most FLOPs-efficient method given a specific accuracy. The strong performance suggests that a deep network's ability to model spatial relationships could also be applied to model temporal relationships across frames in a video, which is an orthogonal direction to having explicit components in the network modeling temporal relationships. Furthermore, being able to connect action recognition with image classification enables existing image classification techniques to be applied to action recognition, which could potentially accelerate the field.\n\n[1]: Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. arXiv.org, March 2021.", "summary_of_the_review": "Overall, I think the proposed method, though simple, leads to surprisingly good results. It not only provides a new way of thinking about modeling temporal relationships, but also better connects action recognition and image classification. Therefore, even though there may not be as much technical novelty in the paper, I still vote for acceptance of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "8ylf25Sp3Vc", "reviewer": "Reviewer_utfq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1) Transforming a video to an image for video recognition is a good direction to explore and has potential application field in the future;\n2) This paper is well written and the experiment is extensive;\n3) The visualization provides interesting insight of this task;\n\nWeakness:\n1) The novelty of the proposed method is limitted. The main contribution of this paper is frame re-arrangement strategy, which provides little inspiration to the community;\n2) The size of the introduced super image is larger than the video frame. There are already some works focusing on transforming a video to an image, like [1]. Their generated images bear the same resolution as the video frame；\n3) The involvement of the Swin-Transformer seems unreasonable. The local operation can only extract the boundary information of the consecutive frames in the super image. For example the information from the right boundary of frame 1 and the left boundary of frame 2, which can not be stated as the temporal dependency. In my understanding, modeling the temporal dependency is to caputure the variation of the similar region (for example the same object or human) across frames.The only part in the modified Swin-Transformer that is able to model temporal dependncy is the kernel with the same size of the image in the last layer;\n4) The reference is not complete. The works like AWSD [2], AVD [3] are missing, which also considering to treat a video clip as an image;\n\nReference:\n1) Qiu, Zhaofan, et al. \"Condensing a Sequence to One Informative Frame for Video Recognition.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n2) Tavakolian, Mohammad, Hamed R. Tavakoli, and Abdenour Hadid. \"Awsd: Adaptive weighted spatiotemporal distillation for video representation.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.\n3) Tavakolian, Mohammad, Mohammad Sabokrou, and Abdenour Hadid. \"AVD: Adversarial Video Distillation.\" arXiv preprint arXiv:1907.05640 (2019).", "summary_of_the_paper": "This paper includes two parts. 1) A video frame re-arrangement strategy that transforms a video clip to an super image such that the video can be processed by image model like 2D-CNN; 2) A slightly modified Swin-Transformer that is more suitable to the proposed super image; The proposed method is evaluated on five benchmark datasets to show its effectiveness and efficiency.", "summary_of_the_review": "Based on the comments in the Main Review part, I tend to reject this paper. The main reasons are:\n1) Limittd novelty;\n2) Lack of considering the efficiency of the proposed method, i.e. the input image size it too large;\n3) Unreasonable model design;", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper regards video understanding as an image classification task, and reports promising performance against state of the arts on several standard benchmarks. Though the method is quite simple, it achieves good results. The visualization in this paper also provides good insight. All reviewers give positive recommendations for this paper.
Decision: Accept (Poster)