V4D: 4D Convolutional Neural Networks for Video-level Representation Learning
Authors: Shiwen Zhang
Sheng Guo
Weilin Huang
Matthew R. Scott
Limin Wang
Venue:  (2020)
Keywords: video-level representation learning
video action recognition
4D CNNs

Abstract: Most existing 3D CNN structures for video representation learning are clip-based methods, and do not consider video-level temporal evolution of spatio-temporal features. In this paper, we propose Video-level 4D Convolutional Neural Networks, namely V4D, to model the evolution of long-range spatio-temporal representation with 4D convolutions, as well as preserving 3D spatio-temporal representations with residual connections. We further introduce the training and inference methods for the proposed V4D. Extensive experiments are conducted on three video recognition benchmarks, where V4D achieves excellent results, surpassing recent 3D CNNs by a large margin.

Published as a conference paper at ICLR 2020
V4D:4D CONVOLUTIONAL NEURAL NETWORKS FOR VIDEO-LEVEL REPRESENTATION LEARNING
Shiwen Zhang, Sheng Guo, Weilin Huang" & Matthew R. Scott
Malong Technologies, Shenzhen, China
Shenzhen Malong Artificial Intelligence Research Center, Shenzhen, China {shizhang, sheng, whuang, mscott) @ma @ma long. com
Limin Wang
State Key Laboratory for Novel Software Technology, Nanjing University, China 1 1mwang@n ju edd.ccm
ABSTRACT
Most existing 3D CNNs for video representation learning are clip-based methods, and thus do not consider video-level temporal evolution of spatio-temporal features. In this paper, we propose Video-level 4D Convolutional Neural Networks, referred as V4D, to model the evolution of long-range spatio-temporal representation with 4D convolutions, and at the same time, to preserve strong 3D spatio-temporal representation with residual connections. Specifically, we design a new 4D residual block able to capture inter-clip interactions, which could enhance the representation power of the original clip-level 3D CNNs. The 4D residual blocks can be easily integrated into the existing 3D CNNs to perform long-range modeling hierarchically. We further introduce the training and inference methods for the proposed V4D. Extensive experiments are conducted on three video recognition benchmarks, where V4D achieves excellent results, surpassing recent 3D CNNs by large margin.
INTRODUCTION
3D convolutional neural networks (3D CNNs) and their variants (ji et al. 2010 Tran et al. 2015 Carreira & Zisserman 2017: Qiu et al. 2017: Wang et al. 2018b) provide simple extension from 2D counterparts for video representation learning. However, due to practical issues such as memory consumption and computational cost, these models mainly used for clip-level feature learning instead of learning from the whole video. The clip-based methods randomly sample a short clip (c.g.. frames) from video for representation learning, and calculate prediction scores for each clip independently. The prediction scores of all clips are simply averaged to yield the vidco-level prediction. These clip-based models often ignore the video-level structure and long-range spatio- temporal dependency during training, as they only sample small portion of the entire video. In fact, in some cases, it could be difficult to identify an action correctly by only using partial observation. Meanwhile, simply averaging the prediction scores of all clips could be sub-optimal during inference. To overcome this issue, Temporal Segment Network (TSN) (Wang et al., (016) was proposed. TSN uniformly samples multiple clips from the entire video, and the average scores used to guide back pack-propagation during training. Thus TSN is a video-level representation learning framework. However, the inter-clip interaction and video-level fusion in TSN is only performed at very late stage, which fails to capture finer temporal structures.
In this paper, we propose general and flexible framework for video-level representation learning, called V4D. As shown in Figure I to model long-range dependency in more efficient way. V4D is composed of two critical designs: (1) holistic sampling strategy, and (2) 4D convolutional interaction. We first introduce a video-level sampling strategy by uniformly sampling a sequence of short-term units covering the whole video. Then we model long-range spatio-temporal dependency by designing a unique 4D residual block. Specifically, we present a 4D convolutional operation to capture inter-clip
Thhe corresponding author.

---

Published as a conference paper at ICLR 2020
interaction, which could enhance the representation power of the original clip-level 3D CNNs. The 4D residual blocks could be easily integrated into the existing 3D CNNs to perform long-range modeling hierarchically, which is more efficient than TSN. We also design specific video-level inference algorithm for V4D. Finally, we verify the effectiveness of V4D on three video action recognition benchmarks, Mini-Kinetie (Xie al. 2018), Kinetics-400 (Carreira & Zisserman 2017) and Something Someehing-Someehiigeee on the three benchmarks, and obtains evident performance improvement over its 3D counterparts.
(Goyal et al.) 2017 Our V4D achieves very competitive performance
2 RELATED WORKS
Two-stream CNNs. Two-stream architecture was originally proposed by (Simonyan & Zisserman 2014), where one stream is used for learning from RGB images, and the other one is applied to model optical flow. The results produced by the two streams are then fused at later stages, yielding the final prediction. Two-stream CNNs have achieved impressive results on various video recognition tasks. However, the main limitation that the computation of optical flow highly expensive where parallel optimization is difficult to implment, with significant resource explored. Recent effort has been devoted to reducing the computational cost on modeling optical flow, such as (Dosovitskiy et al. 2013 Sun et al., 2018 Piergiovanni & Ryoo 2018 8, Zhang et al.) 2016). The two-stream design is a general framework to boost the performance of various CNN models, which is orthogonal to the proposed V4D.
3D CNNs. Recently, 3D CNNs have been proposed (Tran et al. 2015 Carreira & Zisserman 2017 7
Wang et al. 201 8a b Feichtenhofer et al. 2018). By considering video stack of frames, is natural to develop 3D convolutions applied directly on video sequence. However, 3D CNNs often introduce large number of model parameters, which inevitably require large amount of training data to achieve good performance. As reported in (Wang et al. 2018b) Feichtenhofer et al. 2018). recent experimental results on large-scale benchmark, likes Kinetics-400 (Carreira & Zisserman 2017), show that 3D CNNs can surpass their 2D counterparts in many cases,and even can be on par with or better than the two-stream 2D CNNs. is noteworthy that most of 3D CNNs are clip-based methods, which only explore certain part of the holistic video.
Long-term Modeling Framework. Various long- term modeling frameworks have been developed for capturing more complex temporal structure for video-level representation learning. In (Laptev ct al. 2008), video compositional models were proposed to jointly model local video events, where temporal pyramid matching was introduced with a www.ooof-llwwoooooooooo framework to compute long- term temporal structure. However, the rigid composition only works under defined constraints, c.g., prefixed duration and anchor points provided in time. A mainstream method is to process a continuous video sequence with recurrent neural networks Ng et al. (2015); Donahue et al. (2015), where 2D CNNs are used for frame-level feature extraction. Temporal Segment Network (TSN) (Wang et al. 2016) has been proposed to model video-level temporal information with a sparse sampling and aggregation strategy. TSN sparsely samples a set of frames from the whole video, and then the sampled frames are modelled by the same CNN backbone, which outputs a confident score for each frame. The output scores are averaged to generate final video-level prediction. TSN was originally designed for 2D CNNs, but it can be applied to 3D CNNs, which serves as one of the basclines in this paper. One of the main limitations of TSN is that it difficult to model finer temporal structure due to the average aggregation. Temporal Relational Reasoning Network (TRN) (Zhou et al.) 2018) was introduced to model temporal segment relation by encoding individual representation of each segment with relation networks. TRN is able to model video-level temporal order but lacks the capacity of capturing finer temporal structure. The proposed V4D can outperform these previous video-level learning methods on both appearance-des uppearance-dominated video recognition (e.g.. on Kineties) and motion-dominate video recognition (e.g., on Something-Something Something). It is able to model both short-term and long-term temporal structure with unique design of 4D residual blocks.
3 VIDEO-LEVEL 4D COVOLUTIONAL NEURAL NETWORKS
In this section, introduce new Video-level 4D Convolution Neural Networks, namely V4D, for video action recognition. This is the first attempt to design 4D convolutions for RGB-based video recognition. Previous methods, such You & Jiang (2018); Choy al. (2019), utilize 4D CNNs to
2

---

Published as a conference paper at ICLR 2020
CelleTatoo
30 Laysrs
3D Leyers
Reskive
Besitan
Layts
Vite-LLve
30
36
Leyet
Action Unis
Weight Sharing
Action UkDaTeeaa
Sharing
V4D Pipeline
Residual Block
Figure 1: Video-level 4D Convolutional Neural Networks for video recognition.
process videos of point cloud by using 4D data as input. Instead, our V4D processes videos of RGB frames with input of 3D data. Existing 3D CNNs often take a short-term snippet as input, without considering the evolution of 3D spatio-temporal features for video-level representation.
Wang al. 201 8b;; Yue et al. (2018); Liu et al. 2019), self-attention mechanism was developed to model features, but these methods were originally designed for clip-based 3D CNNs. It remains unclear how to incorporate such operations on holistic video representation, and whether such operations are useful for video-level representation learning. Our goal is to model 3D) spatio-temporal features globally, which can be implemented in higher dimension. In this work, we introduce new Residual 4D Blocks, which allow us to cast 3D CNNs into 4D CNNs for learning long-range interactions of the 3D features, resulting in a "time of time" video-level representation.
non- Non-local spatio-tempol
3.1 A VIDEO-LEVEL. SAMPLING STRATEGY
To model meaningful video-level representation for action recognition, the input to the networks has to cover the holistic duration of a given video, and at the same time preserve short-term action details. A straightforward approach is to implement per-frame training of the networks yet this is not practical by considering the limit of computation resource. In this work, uniformly divide the whole video into U sections, and select a snippet from each section to represent a short-term action pattern, called "action unit". Then we have Ú action units to represent the holistic action in video. Formally, we denote the video-level input V = {A, A2,.., Ay ). where A, E RCXT xxxTxxxxx H During training, each action unit A; is randomly selected from each of the U sections. During testing, the center of each A, locates exactly at the center of the corresponding section.
3.2 4D CONVOLUTIONS FOR LEARNING -PATIO-TEMPORAL. INTERACTIONS
3D Convolutional kernels have been proposed, and are powerful to model short-term spatio-temporal features. However, the receptive fields of 3D kernels are often limited due to the small sizes of kernels, and pooling operations are applied to enlarge the receptive fields, resulting in significant cost of information loss. This inspired us to develop new operations which are able to model both short- and long-term spatio-temports representations simultancously, with easy implementations and fast training. From this prospeetive, we propose 4D convolutions for better modeling the long-range spatio-temporal interactions.
Specifically, we denote the input to 4D convolutions as a tensor V of size (C,U.T,H,W) where C number of channel, U is the number of action units (the 4-th dimension in this paper). T, H. W are temporal length, height and width of an action unit. We omit the batch dimension for simplicity. By following the annotations provided in Ji et al. (2010), a pixel position (u, L, h, w) of the jth channel in the output is denoted as o oythw and a 4D convolution operation can be formulated as
Cin S-1P-1Q-1R-- oetho = ΣΣΣΣ
Wapee o of (uttt)((tpp))((--))())
where by is a bias term, e is one of the Cin input channels of the feature maps from input V. S X.Px2 Q x is the shape of 4D convolutional kernel, Weper is the weight the position (s, p, q, r) of the kernel, corresponding to the c-th channel of the input feature maps and j-th channel of the output feature maps.
3

---

Published as a conference paper at ICLR 2020
HxW
HXW
HkW
(a) 3D
(b)
3x3xixt--40
Figure 2: Implementation of 4D kemelss, compared to3D kernel s. denotes the number of action units, with shape of T, H, W. Channel and batch dimensions are omitted for clarity, The kernels are colored in Blue, with the center of each kernel colored Green.
Convolutional operation linear, and the sequential sum operations E.q. are exchangeable. Thus we can generate E.q. 2 where the expression in the parentheses can be implemented by 3D convolutions, allowing us to implement 4D convolutions using 3D convolutions, while most deep learning libraries do not directly provide 4D convolutional operations.
Cun P-1Q-1R- SEEED p=0 T=U
0 ouchw by +
W (u+ ((04o)((4())(++0))t)) ue
With the 4D convolutional kernel, the short-term 3D features of an individual action unit and long- term temporal evolution of multiple action units can be modeled simultaneously in the 4D space. Compared to 3D convolutions, the proposed 4D convolutions are able to model videos in a more meaningful 4D feature space that enables it to learn more complicated interactions of long-range 3D spatio-temporal representation. However, 4D convolutions inevitably introduce more parameters and computation cost. For example, 4D convolutional kernel of k: x k x k employs k: times more parameters than a 3D kernel of l x k x k. In practice, we explore k x k x 1 x 1 and & x 1 x 1 x 1 kernels, reduce the parameters and avoid the risk of overfitting. The implementation of different kernels is shown in Figure?) 2
3.3 VIDEO-LEVEL 4D CNN ARCHITECTURE
In this section, we demonstrate that our 4D convolutions can be integrated into existing CNN architecture for action recognition. To fully utilize current state-of-the-ard 3D CNNs, we propose new Residual 4D Convolution Block, by designing a 4D convolution in the residual structure introduced in (He et al. 2016). This allows it to aggregate both short-term 3D features and long-term evolution of the spatio-temporal representations for video-level action recognition. Specifically, we define permutation function (dd,dd)) d x... xd.x.x.... xd.xx..xd, Xd. + Md x... Xdy X... di X... Xdd
which permutes dimension d, and d, of a tensor M E RRdx.... The Residual 4D Convolution Block can be formulated as:
Yap = XaD (Fal Wan))
where (A:; W4D) is the introduced 4D convolution. Xap, V'D RUXCXTXHXW and U is merged into batch dimension so that Xap. can be directly processed by standard 3D CNNs. Note that we employ o permute the dimensions of X&D from U C H x W to CxX X x W so that it can be processed by 4D convolutions. Then the output of 4D convolution is permuted reversely 3D form so that the output dimensions are consistent with Xsp. Batch Normalization (Toffe & Szegedy 2015) and ReL.U activation (Nair & Hinton 2010) are then applied. The detailed structure described in Figure [
Theoretically, any 3D CNN structure can be cast to 4D CNNs by integrating our 4D Convolutional Blocks. As shown in previous works (Zolfaghari et al. 2018 Xie et al. 2018 Wang et al./ 2018b Feichtenhofer et al. 2018), higher performance can be obtained by applying 2D convolutions lower layers and 3D convolutions at higher layers of the 3D networks. In our framework, we utilize the "Slow-path" introduced in Feichtenhofer al. (2018) as backbone, denoted I3D-S. Although the original "Slowpath" is designed for ResNet50, we can extend it to I3D-S ResNet18 for further experiments. The detailed structure of our 3D backbone is shown in Table
4

---

Published as a conference paper at ICLR 2020
layer
13D-S ResNer18
13D-$ ResNet50
output size
conv
1x7x7, 64, stride 2,
1x7x7, 1,
4x112x112
1x3x3, 64
1xlxL, 64
1x3x3, x2
T952
1x3x3,
x3
4x56x56
1x1x1,
1x3x3, 128
1x1x1, 1, 128
x2
108g
1x3x3, 128
1x3x3, 128
x4
4x28x28
112
3x3x3, 256
3x1x1, 256
TeS4
3x3x3, 256
1x3x3, 256
x6
4x14x14
1x1x1, 1024
3x3x3, 512
1,
T085
512
1x3x3,
x3
4x7x7
1x1x1, 2048
global average pool. fc Table 1: We use 13D-Slowpath from (Feichtenhofer et al. 2018) as our backbone. The output size of an example is shown in the right column, where the input has size of xx224x224. No temporal degenerating is performed in this structure.
1xlx1
3.4 TRAINING AND INFERENCE
Training. As shown in Figure ! the convolutional part of the networks is composed of 3D convolution layers and the proposed Residual 4D Blocks. Each action unit is trained individually and in parallel in the 3D convolution layers, which share the same parameters. The 3D features computed from the action units are then fed to the Residual 4D Block, where the long-term temporal evolution of the consecutive action units can be modeled. Finally, global average pooling is computed on the sequence of all action units to form the final video-level representation.
Inference. Given U action units {A1, A2,... Au} of video, denote Utrain the number of action units for training and for as the number of action units for inference. Utrain and Un are usually different because computation resource limited training, but high accuracy is encouraged in inference. We develop a new video-level inference method, which is described in Algorithm The 3D convolutional layers are denote as N3D. followed by the proposed 4D Blocks, NAD-
Algorithm I: V4D Inference.
Networks
The structure of networks divided into two sub-networks by the first 4D Block, namely Nan and NAD-
Input
: Unffer action units from holistic video: {Ai, A2, AUinter
Output
: The video-level prediction.
V4D Inference:
1 A2, AUrnjar are fed into N&D. generating intermediate feature maps for each unit F.....,uu......... J.F. E RCXT*HXW.
2 For the Unfer intermediate features, we equally divide them into Urrain sections. Then we select one unit from each section ssne; and combine these Utvai units into video-level intermediate representation proideo = (Feeci: F Foeczi..., F Frecurra)) ). These video-level
representations form new set [Fvideo Fyideo .... U.rom 1. where
Ucombined (Uinfer/ Caafar//Urann Uerrain puideo E <TRuerai x xCxXx Evideo Ucomdines } are processed by NAD to form a set of prediction scores, {PI, P2. PUemmiinzd):
3 Each Frideo set (Ervideo, F
4 {P1, P2, PUcombinee! are averaged to give the final video-level prediction.
3.5 DISCUSSION
We further demonstrate that the proposed V4D can be considered 4D generalization of number of recent aipely-applied methods, which may partially explain why our V4D works practically well on learning meaningful video-level representation.
Temporal Segment Network. Our V4D is closely related to Temporal Segment Network (TSN). TSN was originally designed for 2D CNN, but it can be directly applied to 3D CNN to model
5

---

Published as a conference paper at ICLR 2020
video-level representation. TSN also employs a video-level sampling strategy with each action unit named "segment". During training, each segment is calculated individually and the prediction scores after the fully-connected layer are then averaged. Since the folne-con layer is linear classifier, is mathematically identical to calculating the average before the fully- fully-connected layer (similar to global average pooling) after the fully-connected layer (similar TSN). Thus our V4I) can be considered as 3D CNN + TSN when all parameters in the 4D Blocks are set to O.
Dilated Temporal Convolution. One special form of 4D convolution kernel, A 1 X x 1, is closely related to Temporal Dilated Convolution (Lea et al. 2016). The input tensor V can be considered as a (C, U T, H, W) tensor when all action units are concatenated along the temporal dimension. In this case, the l X 1 X 1 X 1 4D convolution can be considered as dilated 3D convolution kernel of x 1 x with dilation of T frames. Note that the k x x kernel is just the simplest form of our 4D convolutions, while our V4D architecture can utilize more complex kernels and thus can be more meaningful for learning stronger video representation. Furthermore, our 4D Blocks utilize residual connections, ensuring that both long-term and short-term representation can be learned jointly. Simply applying the dilated convolution might discard the short-term fine- ane-grained features.
4 EXPERIMENTS
4.1 DATASETS
We conduct experiments on three standard benchmarks: Mini- Mini-Kinetics (Xie et al. (2018), Kinetics-400 (Carreira & Zisserman, 2017), and Something-Somethingge dataset covers 200 action classes, and is a subset of Kinetics-400, Since some videos are no longer available for Kinetics dataset, our version of Kinetics-400 contains 240,436 and 19,796 videos in the training subset and validation subset, respectively. Our version of Mini-kinetics contains 78,422 videos for training, and 4,994 videos for validation. Each video has around 300 frames. Something- Something-v v contains 108,499 videos totally, with 86,017 for training, 11,522 for validation, and 10,960 for testing. Each video has 36 72 frames.
(Goyal al. 2017). Mini-kinetics
4.2 ABLATION STUDY ON MINI-KINETICS
We use pre-trained weights from ImageNet to initialize the model. For training, we adapt the holistic sampling strategy mentioned in section 3.1 We uniformly divide the whole video into U sections, and randomly select a clip of 32 frames from each section. For each clip, by following the sampling strategy in Feichtenhofer et al.) (2018), we uniformly sample frames with fixed stride of 8 to form an action unit. We will study the impact of U in the following experiments. We first resize each frame to 320 256, and then randomly cropping is applied as Wang et al. (20186). Then the cropped region is further resized to 224 x 224. We utilize a SGD optimizer with an initial learning rate of 0.01, weight decay is set 10 with momentum of 0.9. The learning rate drops by 10 epoch 35, 60, 80, and the model is trained for 100 epochs in total.
To make a fair comparison, we use spatial fully convolutional testing by following Wang al. (2018b); Yue et al. (2018); Feichtenhofer et al. (2018). We sample 10 action units evenly from full-length video, and crop 256 x 256 regions to spatially cover the whole frame for each action unit. Then we apply the proposed V4D inference. Note that, for the original TSN, 25 clips and 10-crop testing are used during inference. To make fair comparison between 13D and our V4D, we instead apply this 10 clips and 3-crop inference strategy for TSN.
To verify the effectiveness of V4D, we compare it with the clip-based method I3D-S, and video-based method TSN+3D CNN. To compensate the extra parameters introduced by 4D blocks, we add 3 3 x residual block at res4 for 13D-S for a fair comparison, denoted as 13D-S ResNet18++. As shown in Table 2a by using 4 times less frames than I3D-S during inference and with less parameters than I3D-S ResNetl8++, V4D still obtain a 2.0% higher top-1 accuracy than I3D-S. Comparing with the state-of-the-an video-level method TSN+3D CNN, V4D significantly outperforms it by 2.6% t accuracy, with the same protocols used in training and inference.
Results.
4D Convolution Kernels. As mentioned, our 4D convolution kernels can use 3 typical forms:
x 1 x x 1, k x x 1 X and l x & x k x k. In this experiment, we set = 3 for simplicity, and apply single 4D block at the end of res4 in I3D-S ResNet18. As shown in Table 2c V4D with

---

Published as a conference paper at ICLR 2020
model
Trrii X Drtain
Tinfer Xiinfer Rerop
top-1
tops
parameters
BDDS RosNeti8
X
1x 10 x
72.2
91.2
32.3M
13D-S ResNet18
13D ş ResNetl8+t
16 X
16 x 10 x
73.6
91.5
34.1M
TSN+13D- Re
x
91.3
V4D ResNet18
484
4 10 3
75.6
92.7
33.1M
(a) Effectiveness V4D. represents temporal length of each action unit. represents the number of action units.
kernel
top-1
lop5
model
Imput sizee
I3D-S R8
72.2
91.2
BDDS ResNetlS
16 x 256 x 256
55.1G
TSN+13D-S ResNet18
73.0
91.3
TSN+I3D ResNer18
4 256 256
55.1G
V4D ResNet18
73.8
92.0
V4D ResNet88
x 256 x 256
58.8G
V4D ResNet18
8 113x1x
74.5
92,4
(b) Forward flops of previous works and V4D. One
3x3x3x
4D block res3 and res4 for V4D.
(c) Different Forms of 4D Convolution Kernel.
model
4D kemel
top-1
top5
model
Utrasn
top-
lop5
IDD-S ResNet18
72.2
91.2
BD-S ResNeri8
12.2
91.2
TSN+13D-S ResNet18
73.0
91.3
TSN-E3D- ResNet18
73.0
91.3
V4D ResNet18
al res3
74.2
92.3
V4D ResNet18
74.3
92.2
8
ResNet18
V4D ResNet18
at res5
73.6
91.4
V4D ResNer18
74.5
92.3
ResNet18
at res3, res4
75.6
92.7
V4D ResNet18
74.6
92.5
(d) Position and Number of 4D Blocks.
(c) Effect of Utrain-
Table 2: Ablations on Mini-Kinetics with top-1 and top-5 classification accuracy (%).
x 3 kernel can achieve the highest performance. However, by considering the trade-off between model parameters and performance, use the kernel of 3 3 x 1 x in the following experiments.
On 4D Blocks. We evaluate the impact of position and number of 4D Blocks for our V4D, We investigate the performance of V4D by using one x x x 1 block res3, res4 or res5. As be obtained by applying the 4D block at res3 or res4,
shown in Table 2d] higher accuracy indicating that the merged long-short term features of the 4D block need to be further refined by 3D meaningful representation. Furthermore, inserting one 4D block at res3 and one at res4 can achieve a higher accuracy.
convolutions generate
Number of Action Units U. We further evaluate V4D by using different numbers of action units for training, with different values of hyperparameter U. In this experiment, one 3 x 3 x x Residual 4D block is added the end of res4 of ResNet18. As shown in Table 2e U does not have sigmificant impact to the performance, which suggests that: (1) V4D is video-level feature learning model, and robust against the number of short-term units; (2) an action generally does not contain many stages, and thus increasing U is not helpful. In addition, increasing the number of action units means that the 4-th dimension is increased, requiring a larger 4D kernel to cover the long-rang oangrangee evolution of apatio-tempone representation.
With stafe-of-the-art. We compare the results on Mini-Kinetics 4D Residual Blocks are added into every other 3D residual blocks in res3 and res4. With much fewer frames utilized during training and inference, our V4D ResNet50 achieves a higher accuracy than all reported results, which even higher than 3D 101Net101 having compact Generalized Non-local Blocks. Note that our V4D R8sNet18 can achieve a higher accuracy than 3D ResNet50, which further verify the effectiveness of V4D structure.
Model
Terain
Usrgin
Tinfer NAA
Unnfer X terop
top-
top3
S3D
Xie
2018
S3D Inception
64
78.9
13D
2013
3D ReaNet50
32 x
32 x
10 x
75.5
92.2
13D
ResNet101
32 x
13D+NI.
Yiz al
2018
3D ResNet50
32
32 x 10 x
77.5
94.0
BDD+CGN
Yuu et 2018
ResNet50
32
32 x x
78.8
13D+NL
You al
2013
3D ResNet101
32
32 x x
79.2
93.2
13D+CGNL Your et al
2018
3D ResNet101
32
32 x 10
79.9
93.4
ResNetTS
V4DXOuss)
V4D ResNet50
x
4 10 x
80.7
95.3
Table 3: Results on Mini-Kinetics. T temporal length of action unit. number action units.
7

---

Published as a conference paper at ICLR 2020
4.3 RESULTS ON KINETICS
We further conduct experiments large-scale video recognition benchmark, Kinetics-400, to evaluate the capability our V4D. To make a fair comparison, we utilize ResNet50 as backbone for V4D. The training and inference sampling strategy is identical to previous section, except that each action unit now contains frames instead of 4. We set U so that there are 8 x 4 frames in total for training. Due to the limit of computation resource, train the 3D ResNet50 backbone with 8-frame inputs. Then we load the 3D ResNet50 weights to V4D ResNet50, with all 4D Blocks fixed zero. The V4D ResNet50 is then fine-tuned with 8 x 4 input frames. Finally, we optimize all 4D Blocks, and train the V4D with 8 x 4 frames. As shown in Table
train the model in multiple stages. We first
V4D achieves competitive results on Kinetics-400 benchmark.
Model
Backbone
top-1
top-5
ARTNst with TSN (Wang
2018a
ARTNet ReaNet18
70.7
89.3
BCO (Zolfaghari et at 2018
BN-Inccptiin+33 ResNet18
70.0
89.4
S3D-G (ix el 2018)
Inception
Nonlocal Network Wang et al
20180
3D ResNet50
76.5
92.6
SlowFast (Feichenninler el 2018
SlowFast ResNet5U
77.0
92.6
13D Carreina
201
Inception
72.1
90.3
Two-stream BD Carreira & Ziserman 2017
BDD Inception
75.7
92.0
B3D-S Feichtehoor at 2018
Slow pathway ResNet50
74.9
V4D(Oir))
V4D ResNet50
77.4
93.1
Table 4: Comparison with statc-of-the-as on Kinetics.
4.4 RESULTS ON SOMETHING-SOMETHINGG
Something-Sonen Something dataset focuses on modeling temporal information and motion. The back- ground is much cleaner than Kinetics but the motions of action categories more complicated. Each video contains one single and continuous action with clear start and end on temporal dimension.
Model
Backbone
lop-
MultiScale TRN (Zoo al 2018
BN Inception
34.4
ECO (ooffghari al. 2013)
BN IN--Inepptionn3D ResNet18
46.4
S3D-G (xc et al 2018
S3D Inception
45 8
Noulocal FGCN (Wane Gupta
ReaNer50
46.1
TrajectoryNet (Zhao et al: 2018)
S3D ResNei18
47.8
V4D(O
V4D ResNet50
50.4
Table 5: Comparison with state-of-the-arr on Something- Soathiing-Something-v.....
V4D achieves competitive results the Gomething-Somethingg v1. We use V4D ResNet50 pre-trained on Kinetics for experiments. Temporal Order. As shown in Xie et al. 2018), the performance can drop considerably by reversing the temporal order of short-term 3D features, suggesting that 3D CNNs can learn strong temporal order information. We further conduct experiments by reversing the frames within each action unit or reversing the sequence of action units, where the top-1 accuracy drops considerably by 50.4%- 17.2% and 50.4%-220 respectively, indicating that our V4D can capture both long-term and short-term temporal order.
Results. As shown in Table 4.4
CONCLUSIONS
We have introduced new Video-level 4D Convolutional Neural Networks, namely V4D, to learn strong temporal evolution of long-range spatio-temporal representation, well as retaining 3D features with residual connections. In addition, we further introduce the training and inference methods for our V4D. Experiments were conducted on three video recognition benchmarks, where our V4D achieved the state-of-the-ar results.
REFERENCES
João Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In CVPR, 2017.
8

---

Published as a conference paper at ICLR 2020
Christopher Bongsoo Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. CoRR, abs/1904.08755, 2019.
Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venu- gopalan, Kate Saenko, and Trevor Darrell. Long-term recurrent convolutional networks for visual recognition and description. In CVPR, 2015.
Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Häusser, Caner Hazirbas, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In ICCV, 2015.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. CoRR, (tss/182.00392,, 2018.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fründ, Peter Yianilos, Moritz -reeitr-Freetag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The "something something" video database for learning and evaluating visual common sense. In ICCV, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR, 2015.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 3d convolutional neural networks for human action
recogmition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 15:2,1-231, 2010.
Ivan Laptev, Marcin Marszalek, Cordelia Schmid, and Benjamin Rozenfeld. Learning realistic human actions from movies. In CVPR, 2008
Colin Lea, René Vidal, Austin Reiter, and Gregory D. Hager. Temporal convolutional networks: A unified approach to action segmentation. In ECCV Workshops, 2016.
Xingyu Liu, foon-Young Lee, and Hailin Jin. Learning video representations from correspondence 2019.
proposals. CoRR,
Vinod Nair and Geoffrey F. Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, 2010.
Joe Yue-Hei Ng. Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, and George Toderici. Beyond short snippets: Deep networks for video classification. In CVPR, 2015.
A. J. Piergiovanni and Michael S. Ryoo. Representation flow for action recognition. CoRR, abs/1810.01455, 2018
Zhaofan Qiu, Ting Yao, and Tao Mei. Learning spatio-temporal representation with pseudo-3d residual networks. ICCV, 2017.
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition
in videos. In NIPS, 2014.
Shuyang Sun, Zhanghui Kuang, Lu Sheng, Wanli Ouyang, and Wei Zhang. Optical flow guided feature: A fast and robust motion representation for video action recognition. In CVPR, 2018.
Du Tran, Lubomir D. Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. ICCV, 2015.
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In ECCV, 2016.

---

Published as a conference paper at ICLR 2020
Limin Wang, Wei Li, Wen Li, and Luc Van Gool. Appearance Appearance-and--clahteee and-rolation networks for video
classification. In CVPR, 2018a.
Xiaolong Wang and Abhinav Gupta. Videos as space-time region graphs. In ECCV, 2018.
Xiaolong Wang, Ross B. Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks, In CVPR, 2018b.
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy, Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. In ECCV, 2018.
Quanzeng You and Hao Jiang. Action4d: Real-time action recognition in the crowd and clutter. CoRR, abs/1 00218806002244 2018.
Kaiyu Yue, Ming Sun, Yuchen Yuan, Feng Zhou, Errui Ding, and Fuxin Xu. Compact generalized non-local network. NeurlPS, 2018.
Bowen Zhang, Limin Wang, Zhe Wang. Yu Qiao, and Hanli Wang. Real-time action recognition with enhanced motion vector CNNs. In CVPR, 2016.
Yue Zhao, Yuanjun Xiong, and Dahua Lin. Trajectory convolution for action recognition. In NeurIPS, 2018.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. CVPR, 2016.
Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In ECCV, 2018.
Mohammadreza Zolfaghari, Kamaljeet Singh, and Thomas Brox. ECO: efficient convolutional network for online video understanding. In ECCV, 2018
A APPENDIX
A.1 EXTENDED EXPERIMENTS LARGE-SCALE UNTRIMMED VIDEO RECOGNITION
In order to check the generalization ability of our proposed V4D, we also conduct experiments for untrimmed video classification. To be specific, we choose ActivityNet v1.3 HHeilrro et al. (2015). which is a large-scale untrimmed video dataset, containing videos of to 10 minutes and typically large time lapses of the videos are not related with any activity of interest. We adopt V4D ResNer50 to compare with previous works. During inference, Multi-scale Temporal Window Integration is applied following (Wang et al. 2016). The evaluation metric is mean average precision (mAP) for action recognition. Note that only RGB modality is used input.
Model
Backbone
mAP
TSN Wang et
2016
BN-Inception
79.7
TSN Wing et
2016
Inception V3
83.3
TSN: Wang al, 2016
Inception V3
84.5
V4D(Ours)
V4D ResNet50
Table 6: Comparison with state-of-the-art ActivityNet v1.3.
A.2 VISUALIZATION
We implement 3D CAM based on Zhou et al. (2016), which was originally implemented for 2D cases. Generally, class activation maps (CAM) imply which areas most important for classification. Here we show some random visualization results from validation set of Mini Min--Kinetics, where TSN + 13D-S ResNet18 generates wrong prediction while V4D ResNet18 generates correct prediction. The original RGB frames are shown in the first row. The second row shows the CAMs of TSN + I3D-S ResNet18. 8. The third row shows the CAMs of V4D ResNet18.
10

---

Published as a conference paper at ICLR 2020
Figure
Figure 4:
11

---

Published as a conference paper at ICLR 2020
Figure
US SEAA
RESA BER
Figure 6:
12

---

Published as a conference paper at ICLR 2020
Figure 7:
U
NX
Figure
13

---

Published as a conference paper at ICLR 2020
Figure
Figure 10:
14

---

Published as a conference paper at ICLR 2020
Figure 11:
Figure 12:
15

---

Published as a conference paper at ICLR 2020
Figure 13:
16


Reviews:
[{"review_id": "rkgoA2UCKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents 4D convolutional neural networks for video-level representations. To learn long-range evolution of spatio-temporal representation of videos, the authors proposed V4D convolution layer. Benchmark on several video classification dataset shows improvement.\n\n1. In section 3.1, the authors selected a snippet from each section, but this was not rigorously defined. Same for action units. It does intuitively makes sense, but more mathematical definition (e.g., dimensionality) may be needed.\n\n2. In section 3.2, the authors argued that 3D kernel suffers from trade-off between receptive field and cost of computation. At the end of the subsection, the authors argue that 4D convolution is just k times larger than 3D kernels, which sounds like contradicting. 3D convolution is already expensive and not scalable, but 4D operation sounds even more expensive and more prohibitive.\n\n3. In the paper, the authors argued that clip-level feature learning is limited as it is hard to learn long-range spatio-temporal dependency. It makes sense, and I expect the proposed model may benefit from its design for long-range spatio-temporal feature learning. However, what I see in the experiments is on ~300 frames for Mini-Kinetics and 36-72 frames for Something-Something dataset. Assuming that a second is represented with 15-30 frames, this corresponds to 10-20 sec and 1-4 sec, respectively. I'd say these short videos are still clips.\n\nThe paper presents an interesting idea, but there are some issues that need to be addressed before published on ICLR.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "S1xUCqYpYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes 4d convolution and an enhanced inference strategy to improve the feature interaction for video classification.  State-of-the-art performance is achieved on several datasets.  However, there are still details and concerns.\n\n1. The paper should also talk about the details of ARNet and discuss the difference, as I assume they are the most related work \n2. during sampling, either training or testing, how do authors handle temporal overlap or make it overlap?\n3. can you provide the training memory, inference speed, and total training time?\n4. Most importantly, for table4, authors are comparing to the ResNet18 ARNet instead of ResNet50? which is better than the proposed method.\n5. lack of related work:  4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks, CVPR 2019.\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HyecCk_TFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Summary]\nThe paper presents a video classification framework that employs 4D convolution to capture longer term temporal structure than the popular 3D convolution schemes. This is achieved by treating the compositional space of local 3D video snippets as an individual dimension where an individual convolution is applied. The 4D convolution is integrated in resnet blocks and implemented via first applying 3D convolution to regular spatio-temporal video volumes and then the compositional space convolution, to leverage existing 3D operators. Empirical evaluation on three benchmarks against other baselines suggested the advantage of the proposed method.         \n\n[Decision]\nOverall, the paper addresses an important problem in computer vision (video action recognition) with an interesting. I found the motivation and solution are reasonable (despite some questions pending more elaboration), and results also look promising, thus give it a weak accept (conditional on the answers though).\n\n[Comments]\nAt the conceptual level, the idea of jointly modeling local video events is not novel, and can date back to at least ten years ago in the paper “Learning realistic human actions from movies”, where the temporal pyramid matching was combined with the bag-of-visual-words framework to capture long-term temporal structure. The problem with this strategy is that the rigid composition only works for actions that can be split into consecutive temporal parts with prefixed duration and anchor points in time, which is clearly challenged by many works later when more complicated video events are studied. It seems to me that the proposed framework also falls in this category, with a treatment from deep learning. It is definitely worth some discussion on this path.            \n\nThat said, I would like to see more analysis on the behavior of the proposed method under various interesting cases not tested yet. Despite the claim that the proposed method can capture long-term video patterns, the static compositional nature seems to work best for activities with well-defined local events and clear temporal boundaries. These assumptions hold mostly true for the three datasets used in the experiment, and also are suggested by results in table 2(e), where 3 parts are necessary to achieve optimal results. How does the proposed method perform in more complicated tasks such as        \n- action detection or localization (e.g., in benchmarks JHMDB or UCF101-24). \n- complex video event modeling (e.g., recognizing activities in extended video of TRECVID).\nWill it still be more favorable than other concerning baselines?  \n\nBesides, on the computation side, it would be complexity, an explicit comparison of complexity makes it easier to evaluate the performance when compared to other state-of-the-art methods. \n\n[Area to improve]\nBetter literature review to reflect the relevant previous video action recognitions, especially those on video compositional models.  \nProof reading - The word in the title should be “Convolutional”, right?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes video-level 4D CNNs and the corresponding training and inference methods for improved video representation learning. The proposed model achieves state-of-the-art performance on three action recognition tasks. 
Reviewers agree that the idea well motivated and interesting, but were initially concerned with positioning with respect to the related work, novelty, and computational tractability. As these issues were mostly resolved during the discussion phase, I will recommend the acceptance of this paper. We ask the authors to address the points raised during the discussion to the manuscript, with a focus on the tradeoff between the improved performance and computational cost.
Decision: Accept (Poster)