Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper considers the action recognition problem from 3D video input. The core idea of the manuscript is to propose a triple 2D decomposition (T2D) for 3D vision transformer, which groups 3D video tokens along XY,XT and TY axis for self-attention computation. To address the memory cost issue in stander transformer models, the authors propose to extract spatial and spatiotemporal features from the 2D slices and fuse the attention feature for the final prediction. Extensive experiments on multiple datasets demonstrate that the proposed method is able to improve the recognition accuracy while greatly reducing the computation complexity. 
strength_and_weaknesses: What's good:
1) First, the motivation of this paper is clear and intuitively effective. The idea of learning attention features by tri-plane modeling is novel and interesting.
2) The idea of triple 2D decomposition seems general and can be extended to related tasks.
3) Fig. 3 is really helpful for understanding the decomposition.
4) The proposed method shows better quantitative scores than the previous approach.

To be improved:
1) For Eq. 1, what does "+" means? Sum or concatenate?
2）Lacking the reference, discussion on previous triplane representation, and tensor decomposition methods. For example EG3D, TensoRF.
clarity,_quality,_novelty_and_reproducibility: The paper writing should be improved
Novelty: Yes as commented above
Reproducibility: not sure
summary_of_the_review: The proposed triple 2D decomposition for action recognition from videos is novel and interesting to me, but the paper lacks a detailed discussion on the branch of decomposition work.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposed to decompose video data (3D, space 2D + time 1D) into 3 data planes. Each of the data plane is only 2D. 

This allows the T2D model to create tokens in 3 2D planes separately. ViT 3D self-attention is decomposed along each plane. Compared with a single 3D self-attention, using the three 2D self-attention is less computationally expensive. The weights of the ViT's self-attentions are tied for the two 2D planes w/ time dimension.




strength_and_weaknesses: Strength
- Decomposing ViT's 3D self-attention along three 2D plane is novel. It reduced computation complexity. This exact decomposition has not been explored before (see Fig 3), though similar to the "2D + 1D attention", and the "Axis attention".
- The authors show this simple model design can achieve competitive evaluation accuracy on video action recognition datasest, if not better.
    -- higher or competitive performance on Kinetics-400 and Something-Something-v2 benchmarks 
    -- significantly improves SOTA accuracy on Diving-48, Gym99, and Gym288
- comprehensive ablation studies

Weakness
- It's not entirely clear by reading the abstract, the author decomposes the self-attention in 3D ViT. I suggest revise the abstract to make it clearer. Or consider move the Fig 2 to the first page.
- The improvement from 2D + 1D to T2D is not that significant (Table 2). Which is as expected - somewhat incremental.
clarity,_quality,_novelty_and_reproducibility: Paper is well written and easy to follow. The method is novel and the experiments are comprehensive. Results are good -- improved upon SoTA.
summary_of_the_review: The authors proposed an atomic and novel decomposition of ViT 3D self-attention along each 2D planes (XY, XT, and YT). The authors conducted extensive experiments to show this approach has advantage over existing approaches. The authors tuned the method well, so that it improved upon SoTA accuracy on a few popular action recognition datasets.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes triple 2D decomposition (T2D) of a 3D vision transformer for spatiotemporal feature learning. It is achieved by decomposing the 3D representation into three 2D representations, e.g. XY, YT, XT. The isolated self-attention operation on three 2D representations improves the model performance on widely-used datasets. 

Despite the performance improvements, this paper has the following weakness and shortcomings, which lead me to give a weak rejection.
1.	The decomposition operation is a bit too simple, and the improvement is not obvious from the results of the ablation study in Table2 compared with 2D+1D.
2.	I doubt the core difference between XT+YT in T2D and +1D in 2D+1D. The former will introduce more computation but bring little benefit.
3.	The decomposition operation does not bring efficient inference speed, nor does it greatly reduce the complexity. 
4.	The ablation study on the combination of XY+XT, XY+YT, XY+XT+YT, etc. is missing. I think XY+XT or XY+YT contains enough temporal information for feature extraction. 
5.	I think this decomposition operation applies to any vision network, either CNN or transformer. As the results in Table5, the performance improvement comes more from the network structure and pre-training, rather than the proposed method.

strength_and_weaknesses: Please see summary
clarity,_quality,_novelty_and_reproducibility: Please see summary
summary_of_the_review: Please see summary
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This submission proposed a new plug-in module which decomposes the 3D vision attention in video modeling. The proposed module, T2D, decomposes a self-attention among 3D spaces (spatial: XY and temporal: T) into three 2D space attention: XY, XT and YT. The proposed idea is very similar to what has been shown in R(2+1)D model and TimesFormer. The performance has been validated on different datasets including Kinetics 400, SomethingSomething v2, etc.
strength_and_weaknesses: Strength:
1. The proposed method has been validated on different datasets and the experiments are very comprehensive to validate the performance of T2D

Weakness
1. The idea of decomposing 3D attention into different planes of 2D attention is not novel, it has been proposed and experimented in R(2+1)D (cnn based architecture) and TimeSformer (transformer based architecture). The decomposition of T2D is very similar to axial decomposition in TimeSformer, which undermines the novelty of this proposed module

2. The performance on different datasets shows the introduction of T2D only leads to very marginal gain. For example, on Kinetics 400 and SSv2 datasets, T2D compared to regular 2D+1D, the performance gain is 0.4 and 0.3 respectively. Comparing proposed method with current SoTA method, when using pretrained CLIP-L ViT, the proposed method only achieved 0.0 and 0.3 in top1 and top3 accuracy, under similar GFLOPS cost.
clarity,_quality,_novelty_and_reproducibility: This proposed method original and writing is good. However, the novelty is very marginal.
summary_of_the_review: Based on the reviews in weakness part, I am prone to reject the submission.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: The paper proposes T2D, a new decomposition of video frames into three planes (a tri-plane representation).  The decomposition allows for more efficient self-attention with transformer-based architectures. Experiments compare with existing decompositions (none, space-only, divided space-time, axial, local-global) studied in TimeSformer, and show the proposed decomposition (for models without pretrained CLIP) has slight accuracy improvements and is more efficient in memory and compute compared to prior work.

Strengths
- The paper proposes a tri-plane decomposition of video frames which as not be studied in that domain
- The proposed decomposition performs well compared with prior work and is more efficient 

Weaknesses
- The proposed decomposition is not that different from what was proposed in TimeSformer
- More careful comparison with 2D+1D (divided space-time) decomposition from TimeSformer show that the gain from the T2D decomposition is relatively small, and with similar computational cost. 

justification_for_why_not_higher_score: Reviewers who were part of the virtual meeting are just not that excited by this paper.  The AC feel that while the T2D decomposition is new, and could potentially be interesting, the experimental results (based on Tab 2 which was added in the rebuttal) do not show that the proposed decomposition is significant faster than the previously proposed 2D+1D (divided space time) decomposition from TimeSformer, or significant better performing.
Decision: Reject