Spatiotemporal Joint Filter Decomposition in 3D Convolutional Neural Networks
Authors: Zichen Miao
Ze Wang
Xiuyuan Cheng
Qiang Qiu
Venue: NeurIPS 2021 Poster (2021)
Keywords: 3D Filter Decomposition
Efficient Learning

Abstract: In this paper, we introduce spatiotemporal joint filter decomposition to decouple spatial and temporal learning, while preserving spatiotemporal dependency in a video. A 3D convolutional filter is now jointly decomposed over a set of spatial and temporal filter atoms respectively. In this way, a 3D convolutional layer becomes three: a temporal atom layer, a spatial atom layer, and a joint coefficient layer, all three remaining convolutional. One obvious arithmetic manipulation allowed in our joint decomposition is to swap spatial or temporal atoms with a set of atoms that have the same number but different sizes, while keeping the remaining unchanged. For example, as shown later, we can now achieve tempo-invariance by simply dilating temporal atoms only. To illustrate this useful atom-swapping property, we further demonstrate how such a decomposition permits the direct learning of 3D CNNs with full-size videos through iterations of two consecutive sub-stages of learning: In the temporal stage, full-temporal downsampled-spatial data are used to learn temporal atoms and joint coefficients while fixing spatial atoms. In the spatial stage, full-spatial downsampled-temporal data are used for spatial atoms and joint coefficients while fixing temporal atoms. We show empirically on multiple action recognition datasets that, the decoupled spatiotemporal learning significantly reduces the model memory footprints, and allows deep 3D CNNs to model high-spatial long-temporal dependency with limited computational resources while delivering comparable performance.

Spatiotemporal Joint Filter Decomposition in 3D Convolutional Neural Networks
Zichen Miao', Ze Wang', Xiuyuan Cheng", and Qiang Qiu
Purdue University?
Duke University?
(mi aoz, zewang, qqiu) @purdue edu crrpuaannchenggduiie
Abstract
In this paper, we introduce spatiotemporal joint filter decomposition to decouple spatial and temporal learning, while preserving spatiotemporal dependency in a video. A 3D convolutional filter is now jointly decomposed over of spatial and temporal filter atoms respectively. In this way, 3D convolutional layer becomes three: temporal atom layer, spatial atom layer, and a joint coefficient layer, all three remaining convolutional. One obvious arithmetic manipulation allowed in our joint decomposition is to swap spatial or temporal atoms with a set of atoms that have the same number but different sizes, while keeping the remaining unchanged. For example, as shown later, we can now achieve innpo-iivvriance by simply dilating temporal atoms only. To illustrate this useful flom-swapping property, we further demonstrate how such a decomposition permits the direct learning 3D CNNs with full-size videos through iterations of two consecutive sub-stages of learning: In the temporal stage, full-temporal downsampled-spattal data are used to learn temporal atoms and joint coefficients while fixing spatial atoms. In the spatial stage, full-spatial downsampled temploral data used for spatial atoms and joint coefficients while fixing temporal atoms. We show empirically on multiple action recognition datasets that, the decoupled spatiotemporal learning significantly reduces the model memory footprints, and allows deep 3D CNNs model high-spatial long-temm cong-temporal dependency with limited computational resources while delivering comparable performance.
Introduction
Convolutional Neural Networks (CNNs) have been used intensively in the field of video understanding. Particularly, networks with 3D convolutional layers capture spatiotemporal correlation and achieve great success in applications like action recogmition [I 30]. However, joint spatiotemporal modeling of videos usually requires significant training time, computation, and memory, which becomes one of the main obstacles video understanding. In practice, we almost always need to first downsample video data spatially temporally both to meet real-world constraints of computational resources and training time. For instance, a 64-frame video with the spatial resolution 224 224 downsampled 2 times temporally for training Non-Loccal Networks [30], and 8 times temporally for the R(2+1) D models (26). As in most cases, for spatiotemporal modeling of high-resolution videos, 3D CNNs cannot be fit into most modern GPUs due to the huge model memory footprints. However, it has been observed that the full spatial and temporal resolution are essential achieve superior performance in
many video understanding tasks (24 27 30).
In this paper, we propose spatiotemporal joint filter decomposition to decouple spatial and temporal learning while preserving spatiotemporal dependency in video. As shown in Figure [ 3D convolu- tional filter is jointly decomposed over a group of spatial atoms and temporal atoms. The two groups of atoms can reconstruct the 3D filter together with the joint coefficients through tensor multiplication.
35th Conference on Neural Information Processing Systems (NeurIPS 2021))

---

A single 3D convolutional layer is now divided into three convolutional layers: a temporal-aton -atom layer to focus on time, spatial-atom layer for space, and joint coefficient layer to model the spatiotemporal dependency. Thus, spatial and temporal atoms can now be optimized separately. Different from methods that decorrelate the spa- tial and temporal modeling, the proposed decom- position can still capture spatiotemporal corre- lations in the joint coefficients. Moreover, our approach can also significantly reduce model parameters and computation complexity by lim- iting the number of atoms.
Spatial atoms
Coefficients
Temporal atoms
w
N
Figure 1: Spatiotemporal jointly decomposed convo- lutional filters (STDCF) over spatial atoms (vx) M temporal atoms { and joint coefficients ax with M = 3, N = 2. It divides one spatiotemporal convo- lutional layer into three convolutional layers: spatial- atom layer, temporal-aton layer, and joint coeffi- cient layer which mixes information from spatial-atom and lemporal-atoni convolutional to capture joint depen- deney.
One obvious arithmetic property of our joint decomposition is to allow spatial or temporal atoms to be swapped with a set of atoms with the same number but different sizes while keeping the remaining unchanged. For example, we can now achieve tempo-invariance by simply dilating temporal atoms only. To exploit this useful atom-swapping property, further show how the proposed spatiotemporal jointly decomposed convolutional filter (STDCF) permits the direct learning of 3D CNNs with full-size videos.
We start with pedagogical learning decoupling to train a 3D CNN iteratively with full-temporal downsampled--ppee
data first, and then with full-spatial downsampled downsampled-temppoa data. Note that, such cample-minded decoupled learning is less capable of modeling spatiotemporal dependency. A regular 3D CNN model trained from the two-stage strategy above may capture rich temporal features in the first stage, however, such features will be severely degraded by the downsampled-tempp data in the second stage.
Exploiting the atom-swapping property of the proposed spatiotemporal joint decomposition, we can instead decouple spatial and temporal learning into iterations of two consecutive sub-stages of learning: temporal-focus stage (stage-f) and spatial-focus stage (stage-s). In stage-t, temporal atoms and joint coefficients are learned from full-temporal downsampled-spee spatial atoms fixed, while in stage-s, spatial atoms and coefficients gownsampled-temorre data by keeping temporal atoms fixed. Note that, in stage-s, spatial atoms are learned from full spatial-resolution. while temporal atoms are frozen to prevent degradation from temporal down- nown-sampling similar for atoms update in stage-t. The proposed two-stage 3D CNN training strategy can model full temporal and spatial resolution video only from downsampled data while preserving spatiotemporal dependency. Empirically, we show on multiple action recognition datasets, e.g., KTH 19), Kinetics-400 D and trained with the proposed strategy significantly reduces memory usage, while producing comparable results with the state-of-the-an models trained with full-size videos.
videos by keeping updated from full-spatial
[1O]. that, the STDCF model
2 Method
In this section, we start by introducing the proposed spatiotemporal jointly decomposed convolutional filter, STDCF, along with its two properties of capturing spatiotemporal dependency and enabling atom swapping. Then, we decouple spatiotemporal learning into iterative two sub-stage learning. Next, we introduce a simple but effective design, enabled by the atom swapping property, to allow the model to learn from videos with different spatial and temporal resolutions in different sub-stages.
2.1 Spatiotemporal Joint Filter Decomposition
A regular spatiotemporal convolutional filter in deep video models consists of group of 3D tensors with shape x k x & for capturing local joint dependency on space and time, where and are temporal and spatial filter sizes. The 3D structure not only leads to dramatic increase in parameters compared to 2D CNNs, but also couples the spatial and temporal learning as each tensor attends to both dimensions simultaneously. To decouple spatial and temporal learning, as well as to reduce model parameters, we propose to jointly decompose 3D filter over spatial and temporal atoms, as shown in Figure[[ Our idea originates from spatial filter decomposition as proposed in [17] and we extend it to 3D filters by taking the time dimension into consideration.

---

Inpot Sequence
Rank-1 Reconstructed
1\
STDCE Reconstructed
/7
X
7
Figure 2: Translate Mansslte-RRoate--te Ponate-MNIST reconstruction results.
As proposed in [17. 2D spatial filter W E Xkxk (Cout, Cin input, output channels, and spatial kernel size), can be decomposed as linear combination of M spatial atoms {1/ M 14 E R with coefficients R RCM.Cuu The spatial convolution then becomes two: A spatial-atom convolution with thi a x convolution with joint coefficients a. This decomposition not only reduces the parameters and computation cost but also imposes low-rank filter structures.
Then, for a 3DD spatiotemporal filter W E Rouu C.a xtxxxk (t is the temporal kernel size). as shown in Figure 1 decompose it spatial atoms T { Pill and temporal atoms $ {6, )-1
6 E R', with joint coefficients E Cou X as W EM Σ a
In this way, the 3D convolution becomes three,
Spatial-aton convolution:
Z,(X,)) 1) = Σ (tatt)
Temporal-aton convolution:
Yij(A) = Σ (,(())((X1++)))
(1)
X 1 joint coefficient convolution: J(A,t,)) u) = ΣΣ W
(2) (o
where J denotes the input and output, u E E X' are spatial and temporal indices, X, X are output, input channel indices, and Z;, Yi, are intermediate outputs of spatial-atom layer and temporal-atan atom layer.
2.1.1 Spatiotemporal Dependency
Note that a popular alternative way of 3D filter decomposition is to apply rank- decomposition to
obtain a 2D spatial filter and a temporal filter, as proposed 18 26 37]. Specifically, a 3D filter and W, Cout xtx1x as W W. x W.- In this way, 3D convolution becomes two: spatial sub-convolution with W,, and temporal sub-convolution with Wt-
W E C....Co.. xtxkxk is decomposed into W, C XCou
However, such rank-1 decomposition neglects joint spatiotemporal dependency, The spatial sub- convolution only focuses on spatial modeling, while the temporal sub-convolution merely attends to temporal modeling. The spatial and temporal features are extracted independently by omitting spatiotemporal correlation. Instead, in our STDCF, joint coefficients a encode spatiotemporal dependency while keeping learning in both dimensions independent. We derive below that STDCF has more capacity than rank-1 decomposition. Recall our joint decomposition of 3D filter W.
W(x)= = Σ
i.j (X.x)
RR
(2)
[C],'' E iC'),iE [M]. E [N]. and a (X.x) are freely trainable. For each fixed pair of (A, X'),
is (aiiis is M-by-N matrix, and generally is full rank.
In the rank-l decomposition, spatial filters W, w()".)()),, temporal filters are W w()x")(t) X" [C"]. We write W-(X" (X) and W
(A.2") combination of atoms ii and d;

---

Foram
Bluet No Icaupo diserepancee
Fracr
Greent Withor Atom Swapping
Redi Temporal Atom Steapinng
(a)
(b)
Figure 3: Tempo discrepancy experiments on KTH.
respectively,
A(
-E
Tracking the degree freedom reveals that rank-1 decomposition is more restrictive than our STDCE Let C' = C" = C", or in STDCF has N many variables if all free. In comparison, μ has MC² many variables, and has 2C² many, thus the rank-1 decomposed convolution formulation has only (M + C² many free variables in total. This quantifies the loss of expressiveness from STDCF to rank-1 decomposition. A more comprehensive analysis is provided in Appendix A
To empirically illustrate STDCF more expressive than rank-1 decomposition, we design toy example on the Prussate-rotate-MNNSS (TR-MNIST) dataset. It consists of short clips of moving digits, which contain complex spatiotemporal correlations. Adopting 3-layer 3D CNN, conduct video reconstruction experiments and compare the qualitative and quantitative results between STDCF and rank-1 decomposition. Details of experimental settings are provided in Appendix B As shown in Figure 2 given a sequence of movinggrrottting digits, STDCF reconstructs rank-1. 1. As for qualitative results, we measure mean PSNR on the test set. STDCF achieves a mean PSNR of 24.60, whereas the result of rank-1 decomposition is 21.50, indicating STDCF possesses more capacity to capture spatiotemporal correlations.
better quality than
2.1.2 Atom Swapping Property
The proposed joint decomposition further enables us to manipulate the atoms by changing their spatial or temporal sizes, while keeping their numbers and the rest of components fixed. In this way, we can handle variances in a specific domain by adapting only the corresponding atoms, while keeping the knowledge about spatiotemporal correlation and another domain intact. Rather, would have to modify the whole 3D filters unwieldy. We call this kind of atom manipulations Atom Swapping. We illustrate next its potential usage with a toy tempo-invariance example.
In the temporal domain of videos, a common variance tempo changes т, which characterizes the moving speed of objects in a video. We create a toy example based on KTH 19 to illustrate this in Figure 3 a). where the tempo discrepancy happens between training clips (Terain 1) and test clips (Teest = 2). This leads to test clips proceeding faster than training clips. Without atom swapping, the representation of the model can suffer from severe distortions on the testing set, as shown in Figure ()). To tackle this, we can swap temporal atoms with the ones having the corresponding dilation. As shown in Figure 3bb) the model with swapped temporal atoms almost restores the representation compared to the baseline. We provide quantitative results for dilated atoms in Section 3.1
2.2 Decoupled Spatiotemporal Learning
By exploiting the atom swapping property above, will illustrate next how the proposed STDCF enables direct learning of 3D CNNs with full-size videos. Training deep video models usually utilizes large-scale video datasets with samples in the size of T x H x W. When T or H, W are large enough, model training will require significantly large amount of memory. The proposed STDCF jointly decomposes 3D filters as well as 3D convolution, and thus decouples spatial and temporal learning, while preserving dependency across both dimensions. As shown in Figure 4 decoupled learning performed as iterations of two consecutive sub-stages of learning to model full-size videos but from down-sampled data only, to significantly reduce the memory usage in training.
Cemporal-Focur Learning (stage-f). In this stage, focus modeling the temporal dimension using full-temporal nownsampled-spptiaa vidcos by a factor 3. The original temporal sampling rate is adopted here to update temporal atoms {Ø, 1. best encode the temporal knowledge. The

---

Beraaiia Twu-Siag Spotial- Cempprll Learning
Spatial- Lauming (stapes)
Temporal Fercnr Learning (stge-))
Terpprnd
Fooo
Trainol
Trabad
Kaceladge
W
Cempial-eempralll Learning with Eull--patiial tempural videes
Vemperal
Keossingge
W
Figure 4: Iterative two-stage spatiotemporal learning. By decomposing 3D filter W. we convert spatiotemporal learning from full spatial and temporal resolution videos, which is usually infeasible due to memory issues, videos, and spatial-focus
to iterations of temporal-focus learning from full-temporal downsampled-spian learning from full-spatial downsampled-temppon temporal videos that dramatically reduces the memory footprints. The superscripts of atoms and coefficients denote the current iteration (e) and the sub-stage of learning (t for stage-f, and s for stage-s). Note that our joint decomposition allows swap atoms with different dilation handle resolution changes, e.g. [155.]] has dilation 2 to deal with full-temporal data, whereas (15)**) has dilation for 2x downsampled-tempon data.
joint coefficients a are also updated to capture the spatiotemporal dependency. As we downsample training video spatially by a factor /3 to meet the memory limit, we freeze spatial atoms ( 1 to preserve knowledge learned on full spatial-resolution videos in the spatial-focus learning stage next.
Spatial-Foee Learning (stage-s). Similarly, in this stage, focus modeling the spatial dimen- sion using full-spatial nownsampled-temporee videos by a factor of y. The full spatial resolution used to update spatial atoms { to best encode the spatial knowledge, while temporal atoms [ are frozen to preserve knowledge learned on full frame-rate videos in stage-I above. The joint coefficient a is updated about the spatiotemporal dependency.
Iterative Two-Stage Learning. The above two stages are iterated to model full temporal and spatial resolution videos from down-sampled ones, with far fewer memory footprints, elaborated in Algorithm]] [ Intuitively, more temporal knowledge gained in stage-t will further help spatial learning in the subsequent stage-s and refine spatiotemporal dependency in joint coefficients. Such iterative two-stage learning (ITSL) is empirically validated Section 3.1
We provide an interpretation our iterative two-stage learning from the view of alternating opti- mization. After being jointly decomposed over spatial and temporal atoms, 3D filter W becomes matrix notation here for simplicity, spatial atoms V T x and
W T where temporal atoms & E N and the columns of V (4) consist of the M (N) spatial (temporal) atoms. The atoms V and $. and the joint coefficients a are all trainable, and in our proposed algorithm, we train the triplet in an alternative fashion. the stage-s, we fix temporal atoms $ and optimize y and a jointly, which is equivalent to training in a spatial CNN; In the stage-t, we fix the spatial atoms V/ and optimize & and jointly, and this is equivalent to training in a temporal CNN. As result, the optimization in stage-s prepares an improved model for the stage-t, and vice versa, and the alternative optimization converges suppose each step can make progress. Combined with our dilated atoms discussed next, each step we adjust the sampling rate of the data accordingly so as to most efficiently use the memory. The alternative stages store the model refinement in the shared a. which passed the next stage, and is independent of geometrical dilation filter decomposition in space or time.
2.3 Dilated Atoms
In ITSL, a severe problem lies in the resolution variation in different sub-stages. Considering the spatial dimension as an example, videos with downsampled spatial resolution, e.g. 2 2 are used in stage-t, whereas in stage-s and final testing, full spatial size H x W videos are used; similar for different temporal resolution across sub-stages. We observe this resolution discrepancy in learning and testing will lead to significant performance degradation, which poses a great challenge to our proposed learning strategy. For instance, model learned on stage-1 will perform poorly on data in stage-s due to the mismatched resolutions, c.g. T v.s. T
H. W
5

---

Algorithm 1 Iterative Two-Stage Learning
Require:
Full-size video training dataset D {( uc) him. æ, of size T H x W
Initialize the STDCF model .M with spatial atoms [ex]] temporal atoms (dy [ joint coefficient cc Classification loss function L. Itcrations E.
W/ :99 "9; :0.0 0.* +- wi, $5. a
for = 1,2 do
stage-r Training:
Acquire full fell-temporal downsampled -ppttaa data: D. = {(0)) 34) W. of of size T W
0j a - VML(M(D.)v 1. oo a,11,)), 1/ah t 1/-1.1
stage-s Training:
Acquire full full-spatial downsampled-tempon data: D. = {(al' vi)) at of size x W
Vi a + VuLLM((DlV ,05t ax)),, 07 + og.
end for Return M( ,05 B.
As discussed in Section 2.1.2
we can handle this problem using dilated atoms. To be specific, model learned in stage-I from videos of size T x 4 W. T has its spatial atoms { dilated by factor when learned in stage-s and tested full spatial-resolution videos. Similarly, temporal atoms { will be dilated by y when the model is trained in stage-r and tested, and dilated by 1 in stage-s.
2.4 Parameter and Computation Reduction
Regular 3D filters have a significant increase in parameters and computation flops compared to 2D) filters, which contributes to the huge demand on memory and computation power. The proposed STDCF reduces the parameters and computation complexity of 3D filters significantly.
To be specific, a regular 2D filter is of size Cin x Cout x k x l has Couth" parameters, while regular 3D filter has Cin Couthh? parameters with the additional time dimension. In STDCE, the joint coefficient a has Cin Cott N parameters, and spatial and temporal atoms { {Ø, J has MK², Nt parameters, respectively. Thus, the reduction rate in number of parameters compared to the original 3D filter is Nt+ CCn In practice, C.n Cout MN is much larger than Nt MR22. Ci..ouuthh?
MN .M which gives the approximate reduction of parameters rate of F Frr than regular 3D filters, and MN compared to regular 2D filters. If k 3, and we can choose M 3, N 2, giving reduction rate compared to non-decomposed 3D filters, and 1/2 compared to non-decomposed 2D filters.
As for the computation cost, regular 2D convolution with input with shape of C₁p, X W x W needs Co, (1 + 21 Rs 2C Cin 22 FLOPS, and regular spatiotemporal convolution with inputs in size of Cin. x T x W x W costs Con W + 2k²tt ~ 2Co 2 FLOPs. In STDCF, the computation is split into three sub-layers: (1) spatial-atom layer with s/s costs 2M, FLOPs, (2) temporalatom layer with Φ, requires 2N C,, FLOPs, (3) coefficient layer with a on every spatiotemporal location costs Cin MN FLOPs totally. Together, the computation cost is 2C,,, MTW? NNt++ NCout) FLOPs. Since NCout is usually
Table 1: Dilated atoms experiment. D D." in-
Table 2: Accuracies and GPU memory footprints different 3D layers on KTH dataset under regular full- size videos learning and the proposed iterative two-stage learning.
dicate the dilation of spatial/temporal atoms in the testing/lcaming phase.
Train size
16x30x40
16x60x80 8x60x80
Test size
16x60x80
full-size leamning
ITSL (E 3)
Dilation
D
D!"
D.
Layer type
Memory
Aoc.
Memory
Acc.
53.08
66.49
77.11
Reg. 3D
459.7M
81.26
229 (50.07% ()
76.94 (1.32 ()
Rank-1 3D
582.5M
82.55
290.5M (50.13% ()
(3.15 ()
STDCF
459.4M
85.61
228.9M
(50.17% L)
84,93 (0.68

---

much larger than k² NNt, the cost is around 20 Cou MN FLOPs, showing that the reduction rate in computation FLOOPs is again N M
compared to regular 3D convolution.
k²
Experiments
In this section, we validate the proposed approach on multiple action recognition datasets, KTH [9] Kinetics-400 [1 and Something-Somettinge [10].
3.1 Illustrative Experiments
We illustrate the basic idea of the proposed decomposed learning process on the KTH action recog- nition dataset [19]. The KTH dataset consists of 600 videos 25 fps of 6 action classes from 25 people. The videos are further sampled as short clips of size 16 x 80 x 60, which are used in the full-size learning setting. The training set contains 4667 clips, and the validation set has 4551 clips. Models are learned on the training set and report accuracy on the validation set.
We adopt a simple three-layer 3D CNN model with regular 3D convolution (Reg. 3D) as the baseline, and further replace each convolutional layer with STDCF and Rank-1 1 3D decomposed convolutional layer (Rank-1 3D) in Section 2.I.I as shown in Table B For training models with full-size data, we set batch-size as 16, learning rate as 0.001, and train all models for 40 epochs, with learning rate reduced to 0.0001 at the 20th epoch. For ITSL. we set total iterations E' 3. B 7 = 2, i.c. we downsample clips spatially to the size of 16 x 30 x 40 in stage-t, and temporally to 8 x 60 x 80 in stage-s. To counter the discrepancy in resolution, we dilate temporal atoms by 2 in stage-l, spatial atoms by in stage-s, and both atoms by for testing on full-size clips. We also evaluate the Reg. 3D and Rank-1 3D model under ITSL data setting, For Reg. 3D, we train its 3D filters in both stage-t and stage-s with downsampled data, and for Rank-1 3D, we train its W, with full-temporal downsampled- paapsampled-temmpore
spatial videos, W. with full-spatial these two models with the same dilation setting as STDCF. The training setting aforementioned is adopted to each stage-t and stage-s, and all models trained with ITSL are tested with full-size clips.
videos. Dilated convolutions are applied
We illustrate the effectiveness of the proposed dilated atoms in Table A STDCF model is learned and tested on data with resolution discrepancy. The poor performance caused by the spatial discrepancy can be improved by dilating spatial atoms during testing. Same for the temporal dimension, dilating temporal atoms counter the degradation caused by variation in temporal resolution.
Then, as shown in Table we report accura- cies on validation clips and the maximum train- ing GPU memory usage with the above settings. Specifically, in ITSL experiments, we report the max memory usage in one iteration, which oc- curs in stage-s. Under the full-size learning set- ting, STDCF gives better performance over the Reg. 3D and the Rank-1 3D, showing its effec- tiveness in capturing spatiotemporal dependency. In the ITSL setting, the GPU memory used is sig- nificantly reduced for all models (around 50%). However, the performance of Reg. 3D model drops by a large margin because of the down- sampled training clips as well as the coupled spatial and temporal learning. The Rank-1 3D model decouples spatial and temporal learning but still shows large accuracy drop it can not fully leverage the iterative learning. On the other hand, STDCF with ITSL gives the closest accuracy to the one obtained in full-size learning by a small difference of 0.68%, validating that not only decouples spatial and temporal learning but also refines spatiotemporal modeling iteratively.
Rep, 33
Rank-3 STDCF
06 001 205
62
00
75.2115.255
78.13004.30
72.34
73.07/10.21
Merlollaee
Harz Two Imo-Stage Learring Iteration
Kers
Figure 5: Accuracies of different models after every iteration of two-stage spatiotemporal lcarning. The num- ber in the parenthesis the relative performance gain w.r.t. the model trained after stage-t in the first iteration.
We further plot accuracies of different models at each learning iteration, as shown in Figure 5 Simi- larly, due to coupled spatial and temporal learning, Reg. 3D model only gains 0.22 in performance in stage-s compared to stage-f in the first learning iteration, and its performance does not increase significantly (by 1.17, 0.7) in the two learning iterations afterward. As for the Rank-1 3D model, though it shows a large increase from Iter dier_stage-t to Iter stagesstgessss in Iterl, it does not produce a substantial gain in accuracy in Iter2, Iter3. In comparison, the proposed STDCF encodes spatial and
7

---

Table 3: Results on Kinetics-400. We only compare with methods with the same setting as ours.
Model
#Params
Memory
Top-1
Top-5
GFLOPs
1(2+))D
26
63.7M
5.0G
73.1
89.8
53.2
FlowFast
8
34.6M
9.1G
75.6
92.3
65.7
TSM
[1
24.3M
7.1G
74.1
90.8
33,0
TIN 21
24.3M
6.2G
70.9
89.8
34.0
STDCF-R50-t-3
1.9G
73.6
90.3
STDCF-R50-s-3
18.5M
3.8G
74.0
90.6
52.2
STDCF-R50
7.9G
74.5
91.2
temporal knowledge the corresponding atoms in stuge-f and stage-s separately, and thus shows significant leap in accuracy from Iter ser_stt to Iter stage-s. Besides, as the joint coefficient a captures richer spatiotemporal dependency with the model trained iteratively, the STDCF model shows consistent and substantial performance gain in the next two learning iterations (by 1.87, 1.85), and an accuracy comparable to full-size learning in the end.
3.2 Experiments on Kinetics-400
3.2.1 Experimental Setup
Kinetics-400 [ consists of around 240k training videos, 19k validation videos for total 400 action classes. All experiments are conducted with RGB modality and evaluated on validation sets. Follow- ing the settings (8), input frames are sampled temporally from 64 consecutive frames at interval of 8, which is the full-temporal size. Spatially, each frame is randomly rescaled so that its short side is in [256, 320], and then randomly flipped, cropped into 224 224, which is the full-spatial size in our ITSL setting. As for ITSL setting, we set E = 3, B 7 = 2, so STDCF-R50 is learned with videos of the size 8 X 112 X 112 in stage-t, and with videos of the size 4 x 224 x 224 in stage-s.
We adopt 3D ResNet-50 (3D-R50) the backbone, which inflates 3 kernels in 2D ResNet-50 MT to 3 3 x 3. We further substitutes all those spatiotemporal filters with our STDCF, where we set M 6, N = 2, and obtain STDCF-R50. The architecture of STDCF-R50 is shown in Table STDCF-RSU is learned from scratch for 256 epochs in stage-t of the first iteration (e 1) with the SGD optimizer (momentum of 0.9) of learning rate of 0.03, and weight decay of 0.0001. In stage-s and stage-/ of e 2,3. STDCF-R50 is learned with the same hyper-parameters in stage-/ of the first iteration except for learning rate reduced to 0.0005, and updated for 50 epochs. For inference. adopt the 30-crop setting in (8] to conduct the inference with full-size videos. Specifically, uniformly sample 10 clips from a video along its temporal dimension. Then, we downsample each frame spatially so that its short side size is 256, and take three crops of 256 x 256 covering all spatial information. We dilate both temporal atoms and spatial atoms by 2 in inference for STDCF-R50-t and STDCF-R50-s.
3.2.2 Results
We firstly compare STDCF-R50 learned in stage-t, STDCF-R50--33, with the one in stage-s that has updated spatial atoms. STDCF-R50-s-3 of the final iteration (E = 3). As shown in Table 3 the performance gain in STDCF-R50-s-3 w.r.t. STDCF-R50-t-3 indicates the temporal knowledge and spatial knowledge are learned separately without mutual interference, validating the effectiveness of the proposed ITSL. We further compare STDCT-R50-s-3 with the model learned from full-size data, STDCF-R50. The only 0.5% performance gap shows the proposed model learned with iterative learning strategy can achieve nearly the same performance of the full-size learned model with half GPU memory usage reduction. The accuracies of models in iterations are provided Table D Then, we compare our models with other state-of-the-arr models. Note that for fair comparison, we only compare with methods using networks of the same depth, and the same data setting as ours. STDCF-R50 learned with ITSL shows comparable results with other methods, with significantly fewer GPU memory footprints in the learning phase. Moreover, and computation FLOPs per clip in the inference phase. Compared to other methods, STDCF models show significant reduction in the number of parameters and comparable FLOPs.
evaluate the number of parameters

---

Table Results Something-Sometthngn
Method
Memory
Pretrain
frames
Val Top-1
Val Top-5
2D-R50
5.9G
ImageNet
8
22.3
48.2
TSN 29
ImageNet
16
19.7
46.6
TRN
ImageNet
13D
Kinetics
32
41.6
72.2
BD+GCN 31
Kinetics
32
43.3
75.1
TIN 121
7.5G
Kinetics
8
45.8
75.1
TSM
15
7.1G
ImageNet
8
45.6
74.2
STDCF-R50--22
1.9G
Kinetics
8
44.8
74.3
STDCF-R50-s-2
3.8G
4
45.1
74.7
STDCF-R50
7.8G
Kinetics
8
45.9
75.2
3.3 Experiments on Something-Somethhngell
Something--Somethnnge Somethingy1 [10] is a more challenging dataset, as the activity can only be inferred by modeling spatiotemporal joint dependency. It contains 86k training videos and 12k validation videos for 174 classes. We apply the same model STDCF-R50 with pretrained on Kinetics-400. The model learned directly from full-size videos is finetuned for 50 epochs with batchsize of 64, learning rate of 0.001 which drops by 0.1 the 30th epoch. For the ITSL setting, we learned with ITSI. are all finetuned for 20 epochs with learning rate of 0.0002 which drops by 0.1 at the 15th epoch.
2, B 7 Models
As shown Table our full-size learned model, STDCF-R50, achieves competitive results with other methods. Moreover, our model learned with ITSL can nearly recover the full model's performance, while reducing the GPU memory usage by half. The results of two iterations of ITSL are illustrated in Table
3.4 Visualization
KTH visualization. To better understand the proposed decoupled spatiotemporal learning, we visualize the feature maps of the last convolutional layer in KTH experiment, shown Figure Note that we sum all feature maps along the channel index, and adds it back to original frames. We compare the model from the first iteration stage-1 (Iter-I / stage-r) and the one from the second iteration stage-s (Iter-2 stage-s). both Figure 6a and Figure 6bb the Iter-1 stage-f model fail to
6.
Applsing Cremn
Mogg
bestierr
Stggon
(a)
Haahuning
Berbocoing
Blen Suger
Far Sage-
Stapp
(b)
(b)
Figure 6: Feature maps added back to original video frames to show where the network attends to. In (a), Iter-1 stage-1 model does not focus on the right arm region while Iter2 stage-s model does. In (b), Iter/ stage-1 model does not focus right and left arms regions while 2ter-2 stage-s model does.
Figure 7: Activation maps stage-1 and stage-s mod- that highlights region contributing to the prediction. (a), both stage-f and stage-s predict right, but model fine-tuned in stage-s focuses more on the finger with cream. (b), stage-s model corrects the wrong prediction in stage-f from applauding to harbecuing by attending to meats and sausages.

---

attend to the discriminative part of frames, i.e., right arm in Figure ba two arms in Figure 6b whereas the Iter-2 stage-s does by learning finer spatial knowledge.
Kinetics visualization. We further use Grad-CAM [20] to visualize the activation maps of STDCF- R50-1 and STDCF-R50-s on Kinetics that highlight region contribution to the prediction, as shown in Figure 7 The top row of frames in Figure 7a illustrates the activation maps of STDCF-R50-1, and the bottom row of frames are for STDCF-R50-s. The frames belong to class "applying the cream'. It shows that while the stage-r model captures moving regions and gets the correct prediction, the learning in stage-s refines the activations 7b activations for stage-t model on the top row attends to wrong regions and makes the wrong prediction. By fine-tuning in stage-s, the model focuses on the correct area and corrects its prediction 'barbecuing'.
space the most discriminative areas. In Figure
4 Related Works
3D CNN Models. In [33. 2D CNN models are directly extended to videos by temporal pooling all frames' spatial features extracted by 2D CNN per video. However, this method ignores the temporal relations, so 66] (39 replace the temporal pooling with the LSTM to capture high-level spatiotemporal
dependency. To model low-level spatiotemporal dependency, 3D CNNs 112 23
25 28 have
been proposed and improved the performance on action recognition significantly by capturing richer spatiotemporal correlation. Nevertheless. 3D CNNs achieve this with a dramatic increase in training
time, computation, and memory. [8] 26 37 propose to decompose the 3D filter into 2D filter to capture spatial dependency, and a ID filter for temporal dependency, so as to reduce the cost of regular 3D kernels. This decomposition have been applied to recent methods such as Non-local [30), SlowFast [8). and TPN [38]. However, this decomposition loses the ability to jointly model space and time as regular 3D filters, it performs spatial and temporal modeling sequentially. STDCF, on the other hand, reduces the parameter number and computation cost, while capturing spatiotemporal dependency.
Efficient Video Models Learning. Improvements in the efficiency of deep models learning can
originate from refined optimization algorithms 7 14 16 22), pre-training [5), and softwares for
acceleration [4. For deep video model's ' learning, pre- tre-training schemes [I, 9 have been widely adopted. Recently proposed Multigrid (36] has further reduced the amount of training time of deep video models. The above methods mainly focus on learning efficiency in terms of time. We propose complementary method that attends to the reduction of memory footprints, by avoiding the usage of full-size samples.
Decomposed Convolutional Filters. Decomposed filters is proposed in [7 for parameter and computation efficiency with provably representation stability. It is further extended to domain adaptation (34), generative modeling (32), graph neural network [2], equivariant networks 41]. and flexible convolution [3] 35). In this paper, extend (17) to 3D filter which is jointly decomposed over spatial and temporal atoms for decoupling spatiotemporal learning.
Conclusion
In this paper, we proposed to decouple spatiotemporal learning to significantly reduce the model size, computation, and memory usage in modeling videos. We adopted an iterative two-stage learning to model full-resolution videos with only downsampled data significantly reduce the memory footprints in the learning phase by around 50%, while producing comparable results with full-size data learning. The decoupled learning paves the way for more accessible and scalable video understanding.
Acknowledgement
The work is supported by the DARPA TAMI program under No. HR001 190038 and NSF (DMS- 1820827). XC is partially supported by NIH and the Alfred P. Sloan Foundation.
10

---

References
[1] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299-6308, 2017.
[2] Xiuyuan Cheng, Zichen Miao, and Qiang Qiu. Graph convolution with low-rank learnable local filters. In International Conference on Learning Representations, 2020.
[3] Xiuyuan Cheng, Qiang Qiu, Robert Calderbank, and Guillermo Sapiro. Rotdef: Decomposition of convolutional fil-ters for rotation-equivarianl deep networks. In International Conference Learning Representation, 2019.
[4] Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. cudnn: Efficient primitives for deep learning. arXiv preprint 110.0:4410.0759 2014.
[5] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4186, 2019.
[6] Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. Long-term recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2015.
[7] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
[8] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE international conference on computer vision, pages 6202-6211 2019
2019.
[9] Deepti Ghadiyaram, Du Tran, and Dhruv Mahajan. Large-scale weakly-supervise pre-training for video action recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 12046-12055, 2019.
[10] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, al. The" something something" " video database for learning and evaluating visual common sense. In ICCV, volume 1, page 5, 2017.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[12] Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 3d convolutional neural networks for human action recognition. IEEE transactions on pattern analysis and machine intelligence, 35(1):221- 231, 2012.
[13] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1725-1732, 2014.
[14] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference Learning Representations, 2015.
[15] Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video under- standing. In Proceedings of the IEEE International Conference on Computer Vision, pages 7083-7093, 2019.
[16] Siyuan Qiao, Zhe Lin, Jianming Zhang and Alan I. Yuille. Neural rejuvenation: Improving deep network training by enhancing computational resource utilization. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 61-71, 2019.
11

---

[17] Qiang Qiu, Xiuyuan Cheng, Guillermo Sapiro, et al. DCFNet: decomposed convolutional filters. International Conference on Machine Learning, pages 4198-4207, 2018.
Deep neural network with
[18] Zhaofan Qiu, Ting Yao, and Tao Mei. Learning spatio-temporal representation with pseudo-3d residual networks. In proceedings of the IEEE International Conference Computer Vision, pages 3533-5541, 2017.
[19] Christian Schuldt, Ivan Laptev, and Barbara Caputo. Recognizing human actions: local svm approach. In Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004., volume 3, pages 32-36. IEEE, 2004.
[20] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages
618 626, 2017.
[21] Hao Shao, Shengju Qian, and Yu Liu. Temporal interlacing network. In AAAI, 2020.
[22] Samuel L. Smith, Pieter-Jan Kindermans, and Quoc V. Le. Don't decay the learning rate, increase the batch size. International Conference on Learning Representations, 2018.
[23] Graham W Taylor, Rob Fergus, Yann LeCun, and Christoph Bregler. Convolutional learning of spatio-temporal features. In European conference on computer vision, pages 140-153. Springer, 2010.
[24] Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou. Fixing the train-test resolution discrepancy. In Advances in Neural Information Processing Systems, pages 8252- 8262, 2019.
[25] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 1489-4497, 2015.
[26] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1450-6459, 2018
[27] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 6450-6459, 2018
[28] Gül Varol, Ivan Laptev, and Cordelia Schmid. Long-term temporal convolutions for action recognition. IEEE transactions on pattern analysis and machine intelligence, 1006)1151- 2017.
[29] Limin Wang, Yuanjun Xiong. Zhe Wang. Yu Qiao, Dahua Lin, Xiaoou Tang. and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In European conference on computer vision, pages 20-36. Springer, 2016.
[30] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference computer vision and pullern recognition, pages 7794-7803, 2018
[31] Xiaolong Wang and Abhinav Gupta. Videos space-time region graphs In Proceedings of the European conference on computer vision (ECCV), pages 399 417, 2018.
[32] Ze Wang, Xiuyuan Cheng. Guillermo Sapiro, and Qiang Qiu. Stochastic conditional generative networks with basis decomposition. In International Conference Learning Representations, 2019.
[33] Ze Wang, Xiuyuan Cheng, Guillermo Sapiro, and Qiang Qiu. Acde: Weight sharing in atom- coefficient decomposed convolution. arXiv preprint wtX.::2009.0233000 2020.
[34] Ze Wang, Xiuyuan Cheng, Guillermo Sapiro, and Qiang Qiu. A dictionary approach to domain- invariant learning in deep networks. NeurIPS, 2020.
12

---

[35] Ze Wang, Zichen Miao, Jun Hu, and Qiang Qiu. Adaptive convolutions with per-pixel dynamic filter atom. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12302-12311, 2021.
[36] Chao-Yure Wu, Ross Girshick, Kaiming He, Christoph Feichtenhofer, and Philipp Krahenbuhl. A multigrid method for efficiently training video models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 133-162, 2020.
[37] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy, Rethinking spa- tiotemporal feature learning: Speed-accuracy trade-offs in video classification. In Proceedings of the European Conference on Computer Vision (ECCV), pages 305-321, 2018.
[38] Ceyuan Yang, Yinghao Xu, Jianping Shi, Bo Dai, and Bolei Zhou. Temporal pyramid network for action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 591-600) 2020.
[39] Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijayanarasimhan Oriol Vinyals, Rajat Monga, and George Toderici. Beyond short snippets: Deep networks for video classification. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4694-4702, 2015.
[40] Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In Proceedings of the European Conference on Computer Vision (ECCV), pages 803-818, 2018.
[41] Wei Zhu, Qiang Qiu, Robert Calderbank, Guillermo Sapiro, and Xiuyuan Cheng. Scale- equivariant neural networks with decomposed convolutional filters. In 2020 Fall Eastern Virtual Sectional Meeting. AMS, 2020.
13


Reviews:
[{"review_id": "SjiSd6A6rbP", "reviewer": "Reviewer_3Etu", "summary": "This paper introduces spatiotemporal jointly decomposed convolutional ﬁlter (STDCF) to decouple spatial and temporal learning for traditional 3D convolution. The STDCF is composed of spatial atoms, joint coefficients, and temporal atoms. The STDCF parameters are optimized by the Iterative Two-Stage Learning (ITSL). Experiments show that STDCD is more efficient compared to traditional 3D CNN. The effective decomposition of 3D convolution may be of interest to the community.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of STDCF is decomposition for 3D convolution but the proposed operation is novel. The paper is well present and some experimental results show the effectiveness of the STDCF compared to normal rank 1 decomposition. However, I have some concerns about the potential application for STDCF and some questions on the paper as follows:\n\n1) Algorithm 1. The proposed STDCF requires two-stage training compared to normal 3D and 2D approaches. Does this two-stage training cost more training time compared to normal methods?\n\n2) Table 1 is somewhat difficult to highlight the useful information. \na) It seems lack the performance of using ITSL setting;  \nb)  Normally for Kinetics-400 dataset, 3D CNN approaches normally outperforms 2D CNN approaches. [Original R(2+1)D-ResNet34](https://arxiv.org/pdf/1711.11248v3.pdf) is able to achieve 72.0 on Kinetcs. So I suppose that R(2+1)D-ResNet50 here should get accuracy > 72.0. But here R(2+1)D is worse than TSM and TIN in which I am concerned that  R(2+1)D is not well trained in the experiment; \nc) Both STDCF-R50-t and STDCF-R50-s here perform similar as TSM and TIN but contain much FLOPs even with less parameters. During inference, I suppose inference speed of the proposed method is slower than TSM and TIN; \nd) Why do we need STDCF if it has the similar performance compared to 2D approaches and it might be slower than 2D CNN. Just save GPU memory?\ne) As STDCF replaces 3D convolution, why not equip STDCF with SlowFast architecture to show if it is able to boost the performance or achieve similar performance but saves more GPU and requires less FLOPs.\n\n3) Only two experiment datasets and only ResNet architecture are carried out in this study, which is not able to support the effectiveness of STDCF.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have not stated the limitations and potential negative societal impact of their work.\n\nThe current limitation can be that the performance of STDCF has the similar performance compared to 2D CNN but more FLOPs. The inference speed test can be also added to the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Q-7B7UTl_C6", "reviewer": "Reviewer_Taqs", "summary": "This paper presents a spatiotemporal joint filter decomposition for 3D CNN. Compared with the common 2D+1D decomposition, which is referred to as rank-1 decomposition in the paper, the proposed decomposition has an additional joint coefficients module to encode the spatial and temporal dependency. The proposed STDCF has a larger capacity in theory than the rank-1 decomposition, and is shown to be more friendly to iterative spatial-temporal learning. Toy examples and evaluations on KTH and Kinetics dataset demonstrate the efficiency of STDCF. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper addresses the 3D CNN decomposition problem, which is quite important as it is hard to fit a full-fledged 3D CNN into modern GPU when the spatial resolution, temporal sampling rate, and batch size are at a reasonable level. \n\nThe proposed decomposition method has a larger capacity, or representation power, than the commonly used 2D+1D, or rank-1, decomposition. A toy example shown in Fig. 2 clearly demonstrates the difference. A nice feature of the proposed STDCF is that it is very friendly to two-stage spatial-temporal learning, especially when proper dilation is applied. This is due to the joint coefficient layer working as a bridge between spatial atoms and temporal atoms. The results presented in Fig.5 & Fig.6 are quite convincing. \n\nThe problems or concerns this reviewer has are:\n1. Why the authors set E=1 (one iteration) when evaluating on Kinectics 400? Fig.6 shows that STDCF has a huge performance boost after training for two or three iterations. There is even no sign of saturation in accuracy. This reviewer is curious about the performance on K400 when E=3. \n2. How does STDCF perform on Something-Something dataset? As many of us know, Kinetics dataset is biased towards spatial information than temporal information. This characteristic is reflected in the performance gap between STDCF-R50-t and STDCF-R50-s. But what is the case for SS dataset which relies more on temporal information? The experiments are not comprehensive. \n\n----after rebuttal----\nThe authors provide additional experimental results, which meet my expectation and alleviate my concerns. I therefore raised my rating by one point to Accept. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No obvious negative societal impact is identified. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "4fGzqGAKynE", "reviewer": "Reviewer_mC1w", "summary": "This work studies the problem of decomposing 3D convolutions. Previous works only consider the spatial and temporal information separately, e.g., E3D, P3D, without jointly considering the spatiotemporal information. Therefore, this work decompose the 3D convolution into a temporal atom layer, a spatial atom layer, and a joint coefficient layer. Besides, by the atom-swapping property, two training strategies are proposed to effectively train the proposed network. Experimental results show the proposed approach outperforms the baselines in terms of memory usage, while the performance remains comparable.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Several issues require addressing as listed below.\n\n1) Although the idea is interesting and improves Rank-1 3D, the goal is relatively ambiguous in the paper presentation. For example, adding joint coefficient layer captures additional relationship between spatial and temporal dimension, which usually leads to a performance improvement. Nevertheless, according to the experimental results, the performance is not improved in action recognition, e.g., 75.6 top-1 accuracy of SlowFast but only 70.3 top-1 accuracy in STDCF-R50-s. The significant improvement is in memory. It is suggested to state the goal at the beginning and interpret the idea in terms of memory to better position this paper.\n\n2) The experiments are relatively weak in terms of the accuracy. Moreover, Table 1 only shows STDCF-R50-t and STDCF-R50-s. It is suggested to list the whole model (STDCF-R50). Moreover, it is desirable to explain the memory usage difference between STDCF-R50-t and STDCF-R50-s.\n\n3) Figures 7 and 8 visualize the attention using Grad-Cam and activation maps. Although the results demonstrate the difference, it does not show the advantage of the proposed method since spatial-temporal CNN models usually lead to a similar result. It is suggested to show the one related to the joint coefficient layer.\n \n4) The numbering of tables and figures is messy. Line 254 mentions Table 5 but there is no Table 5. It probably refers to Figure 5. However, Line 249 mentions Figure 5. Is it a figure or a table? Moreover, Line 298 mentions Table 9, but there is no Table 9 (or Figure 9). Similarly, Line 237 mentions Table 1 for the 3D convolution but Table 1 in fact shows the results on Kinetics-400 dataset. When click the number, it refers to the abstract. The errors hinder the readability of this paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No. It is encouraged to discuss the inferior performance in terms of accuracy.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers recommend acceptance (1 rating of 7, 2 ratings of 6). 

Reviewer mC1w raises the initial rating of 6 to 7, because of the convincing additional results included in the author response. However, the Reviewer still recommends revising a claim and clarifying the contribution in the abstract and title. The experimental setup should be corrected to make the objective of the work clearer.

Reviewer Taqs asked several clarifying questions, which were well addressed in the author response. The rating was thus increased to 6. 

Finally, based on author's feedback, Reviewer 3Etu acknowledges the memory reduction as an important improvement and raises the rating to 6.

The ACs concur with the acceptance recommendation made by the reviewers.
Decision: Accept (Poster)