Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a new approach for video action recognition by casting the problem as an image recognition task. The video clips are rearranged into a super image according to a pre-defined spatial layout. 
main_review: In this paper, the authors propose a simple but effective approach for video action recognition by casting the problem as an image recognition task. Different from modeling temporal information, it provides a different perspective to think about the action recognition task. 

It provides solid experiments and ablation studies to evaluate the effectiveness of the proposed approach. Transformer-based and CNN-based models are both tested on public benchmarks and good experimental results are reported.


summary_of_the_review: This paper provides a novel idea for video action recognition. Their claims are well supported by solid experiments and ablation studies. I believe it would inspire others in this research field.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper deals with action recognition in videos, i.e. detecting to which class a given sequence of frames belongs to. However, the paper proposes to explore whether an image classifier (instead of a video or spatiotemporal-based classifier) would already be enough to accomplish this task. In order to do so, the authors organize the frames from a video into a single image by organizing them into a grid, then proceed to learn them using Swin Transformers (Swin-B) image classification models.

The authors report surprising results which are indeed on-par or higher than the SotA in Kinetics400, MiT, Jester and Diving48 datasets.
main_review: # Strengths

The paper reads well and has almost no typos. The method described is simple and achieves surprising results. The authors have provided extra ablation experiments to evaluate the importance of the grid layout, APE and temporal order of the frames in the grid, as well as activation visualizations using CAM. The experimental setup is clear and results seem convincing.

# Weaknesses

a) It is not entirely clear how the frames are sampled before they are organized in a grid. For example, in other methods such as I3D, consecutive frames are sampled 10 frames apart [A, sections 2.3 and 2.5]. However, the current paper states uniform sampling is used to generate the video input for the models [p.5]. Does it mean the frames are uniformly sampled from the start and ending frames of the entire video, or are they sampled considering a fixed skip with a random starting frame as in I3D?

b) It would seem to me that both the super image representation and the use of a transformer-based architecture are essential in order to achieve such good results due the self-attention mechanism. Have the authors experimented with non-transformer architectures to evaluate the efficacity of the super image representation by itself?

minor) "Our" should be capitalized in second paragraph of page 5, or merged with the previous paragraph.

[A]: Carreira et al, Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset, CVPR 2017.
summary_of_the_review: The paper presents a simple yet effective idea to transform simpler image classification models into video classification models. Even being simple, the approach manages to achieve surprisingly good results when compared to SotA video-classification models which explicitly handle the temporal dimension. The paper may thus contain findings that should be of interest for the ICLR community.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper includes two parts. 1) A video frame re-arrangement strategy that transforms a video clip to an super image such that the video can be processed by image model like 2D-CNN; 2) A slightly modified Swin-Transformer that is more suitable to the proposed super image; The proposed method is evaluated on five benchmark datasets to show its effectiveness and efficiency.
main_review: Strength:
1) Transforming a video to an image for video recognition is a good direction to explore and has potential application field in the future;
2) This paper is well written and the experiment is extensive;
3) The visualization provides interesting insight of this task;

Weakness:
1) The novelty of the proposed method is limitted. The main contribution of this paper is frame re-arrangement strategy, which provides little inspiration to the community;
2) The size of the introduced super image is larger than the video frame. There are already some works focusing on transforming a video to an image, like [1]. Their generated images bear the same resolution as the video frame；
3) The involvement of the Swin-Transformer seems unreasonable. The local operation can only extract the boundary information of the consecutive frames in the super image. For example the information from the right boundary of frame 1 and the left boundary of frame 2, which can not be stated as the temporal dependency. In my understanding, modeling the temporal dependency is to caputure the variation of the similar region (for example the same object or human) across frames.The only part in the modified Swin-Transformer that is able to model temporal dependncy is the kernel with the same size of the image in the last layer;
4) The reference is not complete. The works like AWSD [2], AVD [3] are missing, which also considering to treat a video clip as an image;

Reference:
1) Qiu, Zhaofan, et al. "Condensing a Sequence to One Informative Frame for Video Recognition." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.
2) Tavakolian, Mohammad, Hamed R. Tavakoli, and Abdenour Hadid. "Awsd: Adaptive weighted spatiotemporal distillation for video representation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.
3) Tavakolian, Mohammad, Mohammad Sabokrou, and Abdenour Hadid. "AVD: Adversarial Video Distillation." arXiv preprint arXiv:1907.05640 (2019).
summary_of_the_review: Based on the comments in the Main Review part, I tend to reject this paper. The main reasons are:
1) Limittd novelty;
2) Lack of considering the efficiency of the proposed method, i.e. the input image size it too large;
3) Unreasonable model design;
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes to perform action recognition by first rearranging the frames from a video into a 3x3 or 4x4 grid to form a "super image", and then giving the super image to a standard image classifier to perform action recognition. Given that this super image will be a larger image, the paper leverages the more memory efficient Swin Transformer [1] as an image classifier to perform action recognition. Experiments on Kinetics400, Moments In Time, Something-Something V2 (SSV2), Jester and Diving48 show that the proposed method is on par or exceeds SOTA in terms of accuracy. On Kinetics400, the method not only is SOTA in terms of accuracy, but also is the most FLOPs-efficient method given a specific accuracy. The strong performance suggests that a deep network's ability to model spatial relationships could also be applied to model temporal relationships across frames in a video, which is an orthogonal direction to having explicit components in the network modeling temporal relationships. Furthermore, being able to connect action recognition with image classification enables existing image classification techniques to be applied to action recognition, which could potentially accelerate the field.

[1]: Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. arXiv.org, March 2021.
main_review: ### Strengths
- The proposed method (Super Image for Action Recognition, SIFAR) is extremely simple and easy to implement (one liner in pytorch). Its accuracy is on par with SOTA, and its speed is also SOTA given a fixed accuracy threshold.
- Taking deep networks' ability to model spatial relationships and then using it to model temporal relationships is a very interesting direction. This is an orthogonal approach than having explicit components for modeling temporal relationships. I was surprised that it worked so well.
- Being able to connect image classification and action recognition can enable image classification techniques to be applied to action recognition, which could potentially accelerate the field.
- The ablation studies on layout and ordering are interesting.
- The writing is very clear. The writing also clearly mentions the limitations, such as "*Training SIFAR models with more than 16 frames still remains computationally challenging, especially for models like SIFAR-B-14 and SIFAR-L-14†, which need a larger sliding window size.*"

### Weakness
- From the analysis done on SSV2, it seems like the bigger limitations of SIFAR are (1) it has difficulty taking in more than 16 images at once, and (2) it is unclear what is the limit of SIFAR in terms of doing fine-grained temporal modeling. However, I don't think these limitations prevent SIFAR from being an interesting idea.
- "*Note that the joint-space-time attention in TimeSformer (Bertasius et al., 2021a) is a special case of our approach since their method can be considered as flattening all tokens into one plane and then performing self-attention over all tokens. However, the memory complexity of such an approach is prohibitively high...*" I am not sure I agree with this claim. In figure 3 of (Bertasius et al., 2021a, https://arxiv.org/pdf/2102.05095.pdf), the method can use up to 96 frames with the "Divided Space-time" approach, which is less memory intensive than SIFAR, which has difficulty going more than 16 frames at once. 
- In Table 9, SSV2 in reverse order leads to the worst performance, which is surprising. I would expect that normal and reverse ordering should lead to the same performance, as whether the first image is at the top left or bottom right should not affect the network’s ability to learn as long as if the network is given consistently ordered input. Can the authors elaborate on this? Thank you!


### Typos
- P5: “o”ur approach has linear computational complexity… (capitalize)
- P5: We demonstrate later in Sec. 4 “thata” larger window
summary_of_the_review: Overall, I think the proposed method, though simple, leads to surprisingly good results. It not only provides a new way of thinking about modeling temporal relationships, but also better connects action recognition and image classification. Therefore, even though there may not be as much technical novelty in the paper, I still vote for acceptance of the paper.
review_title: null

Meta-review: This paper regards video understanding as an image classification task, and reports promising performance against state of the arts on several standard benchmarks. Though the method is quite simple, it achieves good results. The visualization in this paper also provides good insight. All reviewers give positive recommendations for this paper.
Decision: Accept (Poster)