Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work studies the problem of decomposing 3D convolutions. Previous works only consider the spatial and temporal information separately, e.g., E3D, P3D, without jointly considering the spatiotemporal information. Therefore, this work decompose the 3D convolution into a temporal atom layer, a spatial atom layer, and a joint coefficient layer. Besides, by the atom-swapping property, two training strategies are proposed to effectively train the proposed network. Experimental results show the proposed approach outperforms the baselines in terms of memory usage, while the performance remains comparable.
main_review: Several issues require addressing as listed below.

1) Although the idea is interesting and improves Rank-1 3D, the goal is relatively ambiguous in the paper presentation. For example, adding joint coefficient layer captures additional relationship between spatial and temporal dimension, which usually leads to a performance improvement. Nevertheless, according to the experimental results, the performance is not improved in action recognition, e.g., 75.6 top-1 accuracy of SlowFast but only 70.3 top-1 accuracy in STDCF-R50-s. The significant improvement is in memory. It is suggested to state the goal at the beginning and interpret the idea in terms of memory to better position this paper.

2) The experiments are relatively weak in terms of the accuracy. Moreover, Table 1 only shows STDCF-R50-t and STDCF-R50-s. It is suggested to list the whole model (STDCF-R50). Moreover, it is desirable to explain the memory usage difference between STDCF-R50-t and STDCF-R50-s.

3) Figures 7 and 8 visualize the attention using Grad-Cam and activation maps. Although the results demonstrate the difference, it does not show the advantage of the proposed method since spatial-temporal CNN models usually lead to a similar result. It is suggested to show the one related to the joint coefficient layer.
 
4) The numbering of tables and figures is messy. Line 254 mentions Table 5 but there is no Table 5. It probably refers to Figure 5. However, Line 249 mentions Figure 5. Is it a figure or a table? Moreover, Line 298 mentions Table 9, but there is no Table 9 (or Figure 9). Similarly, Line 237 mentions Table 1 for the 3D convolution but Table 1 in fact shows the results on Kinetics-400 dataset. When click the number, it refers to the abstract. The errors hinder the readability of this paper.
limitations_and_societal_impact: No. It is encouraged to discuss the inferior performance in terms of accuracy.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a spatiotemporal joint filter decomposition for 3D CNN. Compared with the common 2D+1D decomposition, which is referred to as rank-1 decomposition in the paper, the proposed decomposition has an additional joint coefficients module to encode the spatial and temporal dependency. The proposed STDCF has a larger capacity in theory than the rank-1 decomposition, and is shown to be more friendly to iterative spatial-temporal learning. Toy examples and evaluations on KTH and Kinetics dataset demonstrate the efficiency of STDCF. 
main_review: This paper addresses the 3D CNN decomposition problem, which is quite important as it is hard to fit a full-fledged 3D CNN into modern GPU when the spatial resolution, temporal sampling rate, and batch size are at a reasonable level. 

The proposed decomposition method has a larger capacity, or representation power, than the commonly used 2D+1D, or rank-1, decomposition. A toy example shown in Fig. 2 clearly demonstrates the difference. A nice feature of the proposed STDCF is that it is very friendly to two-stage spatial-temporal learning, especially when proper dilation is applied. This is due to the joint coefficient layer working as a bridge between spatial atoms and temporal atoms. The results presented in Fig.5 & Fig.6 are quite convincing. 

The problems or concerns this reviewer has are:
1. Why the authors set E=1 (one iteration) when evaluating on Kinectics 400? Fig.6 shows that STDCF has a huge performance boost after training for two or three iterations. There is even no sign of saturation in accuracy. This reviewer is curious about the performance on K400 when E=3. 
2. How does STDCF perform on Something-Something dataset? As many of us know, Kinetics dataset is biased towards spatial information than temporal information. This characteristic is reflected in the performance gap between STDCF-R50-t and STDCF-R50-s. But what is the case for SS dataset which relies more on temporal information? The experiments are not comprehensive. 

----after rebuttal----
The authors provide additional experimental results, which meet my expectation and alleviate my concerns. I therefore raised my rating by one point to Accept. 

limitations_and_societal_impact: No obvious negative societal impact is identified.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces spatiotemporal jointly decomposed convolutional ﬁlter (STDCF) to decouple spatial and temporal learning for traditional 3D convolution. The STDCF is composed of spatial atoms, joint coefficients, and temporal atoms. The STDCF parameters are optimized by the Iterative Two-Stage Learning (ITSL). Experiments show that STDCD is more efficient compared to traditional 3D CNN. The effective decomposition of 3D convolution may be of interest to the community.
main_review: The idea of STDCF is decomposition for 3D convolution but the proposed operation is novel. The paper is well present and some experimental results show the effectiveness of the STDCF compared to normal rank 1 decomposition. However, I have some concerns about the potential application for STDCF and some questions on the paper as follows:

1) Algorithm 1. The proposed STDCF requires two-stage training compared to normal 3D and 2D approaches. Does this two-stage training cost more training time compared to normal methods?

2) Table 1 is somewhat difficult to highlight the useful information. 
a) It seems lack the performance of using ITSL setting;  
b)  Normally for Kinetics-400 dataset, 3D CNN approaches normally outperforms 2D CNN approaches. [Original R(2+1)D-ResNet34](https://arxiv.org/pdf/1711.11248v3.pdf) is able to achieve 72.0 on Kinetcs. So I suppose that R(2+1)D-ResNet50 here should get accuracy > 72.0. But here R(2+1)D is worse than TSM and TIN in which I am concerned that  R(2+1)D is not well trained in the experiment; 
c) Both STDCF-R50-t and STDCF-R50-s here perform similar as TSM and TIN but contain much FLOPs even with less parameters. During inference, I suppose inference speed of the proposed method is slower than TSM and TIN; 
d) Why do we need STDCF if it has the similar performance compared to 2D approaches and it might be slower than 2D CNN. Just save GPU memory?
e) As STDCF replaces 3D convolution, why not equip STDCF with SlowFast architecture to show if it is able to boost the performance or achieve similar performance but saves more GPU and requires less FLOPs.

3) Only two experiment datasets and only ResNet architecture are carried out in this study, which is not able to support the effectiveness of STDCF.


limitations_and_societal_impact: The authors have not stated the limitations and potential negative societal impact of their work.

The current limitation can be that the performance of STDCF has the similar performance compared to 2D CNN but more FLOPs. The inference speed test can be also added to the paper.
review_title: null

Meta-review: All three reviewers recommend acceptance (1 rating of 7, 2 ratings of 6). 

Reviewer mC1w raises the initial rating of 6 to 7, because of the convincing additional results included in the author response. However, the Reviewer still recommends revising a claim and clarifying the contribution in the abstract and title. The experimental setup should be corrected to make the objective of the work clearer.

Reviewer Taqs asked several clarifying questions, which were well addressed in the author response. The rating was thus increased to 6. 

Finally, based on author's feedback, Reviewer 3Etu acknowledges the memory reduction as an important improvement and raises the rating to 6.

The ACs concur with the acceptance recommendation made by the reviewers.
Decision: Accept (Poster)