Dynamic Normalization and Relay for Video Action Recognition
Authors: Dongqi Cai
Anbang Yao
Yurong Chen
Venue: NeurIPS 2021 Poster (2021)
Keywords: Video Action Recognition
Batch Normormalization
Group Normalization
Convolutional Neural Networks
Recurrent Neural Networks

Abstract: Convolutional Neural Networks (CNNs) have been the dominant model for video action recognition. Due to the huge memory and compute demand, popular action recognition networks need to be trained with small batch sizes, which makes learning discriminative spatial-temporal representations for videos become a challenging problem. In this paper, we present Dynamic Normalization and Relay (DNR), an improved normalization design, to augment the spatial-temporal representation learning of any deep action recognition model, adapting to small batch size training settings. We observe that state-of-the-art action recognition networks usually apply the same normalization parameters to all video data, and ignore the dependencies of the estimated normalization parameters between neighboring frames (at the same layer) and between neighboring layers (with all frames of a video clip). Inspired by this, DNR introduces two dynamic normalization relay modules to explore the potentials of cross-temporal and cross-layer feature distribution dependencies for estimating accurate layer-wise normalization parameters. These two DNR modules are instantiated as a light-weight recurrent structure conditioned on the current input features, and the normalization parameters estimated from the neighboring frames based features at the same layer or from the whole video clip based features at the preceding layers. We first plug DNR into prevailing 2D CNN backbones and test its performance on public action recognition datasets including Kinetics and Something-Something. Experimental results show that DNR brings large performance improvements to the baselines, achieving over 4.4% absolute margins in top-1 accuracy without training bells and whistles. More experiments on 3D backbones and several latest 2D spatial-temporal networks further validate its effectiveness. Code will be available at https://github.com/caidonkey/dnr.

Dynamic Normalization and Relay for Video Action Recognition
Dongqi Cai* Intel Labs China .ongqi.cal waweintel....eeeeeaeeaaa.nanee.. com
Anbang Yao Intel Labs China
Yurong Chen
Intel Labs China yurong. yurong.chen@intel.. chen@intel.com
Abstract
Convolutional Neural Networks (CNNs) have been the dominant model for video action recognition. Due to the huge memory and compute demand, popular action recognition networks need to be trained with small batch sizes, which makes learn- ing discriminative spatial-temp pptiial-temporal representations for videos become challenging problem. In this paper, we present Dynamic Normalization and Relay (DNR), an improved normalization design, augment the spatial-temporaa representation learning of any deep action recognition model, adapting to small batch size training settings. We observe that state-of-the-me action recognition networks usually apply the same normalization parameters to all video data, and ignore the dependencies of the estimated normalization parameters between neighboring frames (at the same layer) and between neighboring layers (with all frames of video clip). Inspired by this, DNR introduces dynamic normalization relay modules explore the potentials of cross-tempora and cross-layer feature distribution dependencies for estimating accurate layer-wiss wise normalization parameters. These DNR modules are instantiated as light-weight recurrent structure conditioned on the current input features, and the normalization parameters estimated from the neighboring frames based features at the same layer from the whole video clip based features the preceding layers. We first plug DNR into prevailing 2D CNN backbones and test its performance public action recognition datasets including Kinetics and Something- Sooiiiiing. Experimental results show that DNR brings large performance improvements to the baselines, achieving over 4.4% absolute margins in top-1 accuracy without training bells and whistles. More experiments on 3D backbones and several latest 2D spatial-tempon temporal networks further validate its ef- fectiveness. Code will be available at fitttts:///. /ttes:///ttttttt com/ caidonkey/daae
Introduction
Human action recognition a fundamental problem in video understanding, which has been studied for decades. The performance of an intelligent action recognition system depends on how well it can extract compact and discriminative features to characterize temporal evolutions of human object appearance and its motion information in videos. Early seminal methods 29 ubiquitously use hand-crafted features to construct spatial-temporal descriptors. recent years, deep learning based models [70) have become the mainstream in action recognition research, mainly due their remarkably better representation and generalization abilities compared to conventional methods, especially model large amounts of training data.
46 28 45 58
Despite the prevalence of deep learning based methods for action recognition, learning efficient yet effective video representations is still challenging problem. To improve the spatial-temporal
The first two authors contributed equally to the writing of the paper. Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

representation learning of CNNs for action recognition, we investigate new technical perspective by presenting an improved nommalization design called Dynamic Normalization and Relay (DNR), learning to predict accurate layer-wise normalization parameters for any action recognition network in an input-dependent manner. We are motivated by two plain facts. On the one side, due to the problem of space-time cube data structure, the huge memory and compute demand in training popular network for action recognition usually restricts the batch size to a much smaller range compared to the settings for image classification tasks. For instance, on Kineties [5] the batch size is typically set to 32 video clips of length 8, while on ImageNet 4+I it is set to 256 images. On the other side, existing CNN models for video action recognition use Batch Normalization (BN) 24 as a standard component to normalize feature maps learnt at each layer. Although the significance of BN has been well demonstrated in many previous action recognition works, it will easily introduce noise during the estimation of layer-wise wise normalization parameters when the batch size is small, degenerating the accuracy of the trained model to some extent. This problem could be alleviated by using recent normalization methods 23 63 26 31 38, 677. However, the performance gain of doing so is limited as they are primarily proposed for image recognition tasks. For example, Kinetics with 32-frame video clips, it only brings -0.3% and 0.7% top-1 accuracy improvement to ResNet-50 C3D baseline (53) trained with batch size of and video clips per GPU by replacing BN with Group Normalization (GN), as reported in [63]. We argue this is mainly because direct extension of them from image to video domain lacks a proper mechanism to handle complicated apatial-tempore feature variations of video data. However, to the best of our knowledge, there is almost no research effort made to explore a better normalization mechanism for promoting the training of existing action recognition networks with video clip inputs.
As an improved video normalization design for action recognition networks, our DNR considers three questions at multiple representation learning scales to alleviate the aforementioned gap: (1) At individual frame scale, how to estimate input-dependent layer-wise normalization parameters? (2) At time scale, how to enhance the dependencies of the estimated normalization parameters for layer-specific feature representations between neighboring frames? (3) At network depth scale, how to enhance the clip-level dependencies of the estimated normalization parameters between neighboring layers? To the first question, we formulate our DNR as dynamic normalization predication scheme [ where normalization parameters for any layer of an action recognition network are learnt and generated dynamically both training and inference, making them to be nepu-ddeenneet. To the other two questions, we bridge the DNR formulation with two interdependent normalization relay modules called cross-temporal DNR and cross-layer DNR, which are encapsulated into light-weight recurrent structure [20]. For certain layer, the normalization parameters learnt by cross-temporal DNR module are conditioned on the input features as well as the normalization parameters estimated from the neighboring frames based features at the same layer, while the normalization parameters learnt by cross-layer DNR module are conditioned on the input features as well as the normalization parameters estimated from the whole video clip based features at the preceding layers. On the one side, the dynamic normalization relay along the temporal axis models the layer-specific frame-Ievel correlations of the side, the dynamic normalization relay along the sequential layers considers all the hidden layers as whole system, and estimates the video clip- feature dynamics of the current layer by jointly considering the feature distributions of its preceding layers. Benefiting from the above complementary designs, our DNR can significantly improve the performance of existing action recognition networks via replacing their normalization layers by two types of DNR modules.
feature distributions between neighboring frames. On the other
Experimental results on Kinetics and Something-Somethiin DNR to prevailing 2D CNN backbones such as ResNet [19], ResNeXt 64 and BNInception (25). brings large accuracy improvements to the baselines. We also apply DNR to 3D CNN backbones and several latest 2D CÑN designs constructed with sophisticated spatial-temporaa modules for further validation of its effectiveness. Thorough ablative experiments also conducted to have deep analysis of the proposed method.
datasets [88 40 show that by applying
2 Related Works
In this section, we summarize existing works that are related to our method, and discuss their connections and differences.
2

---

Modeling, Efficient spatial-tempora modeling is hot research topic in video action recognition. To strike a direct trade-off between network accuracy and efficiency. some 54, 37 use factorized convolutions to approximate 3D convolutions, and some works 665
Efficient
works 42
71
15
mix 2D and 3D convolutions into a single CNN architecture. Two-path networks [16 14
43
30
are another kind of efficient 3D designs. One representative is SlowFast [16] which uses
slow pathway operating at low frame rate of high image resolution to capture spatial semantics and a fast pathway operating at high frame rate of low image resolution to capture motion information. Compared to 2D CNN counterparts, the aforementioned networks mixing 2D and 3D convolutions as well as two-path networks with 3D convolutions their own motivations. In some recent works 34 51 47
not runtime efficient. They are efficient only to 1 39 59]. switchable channel grouping operations such as shift and shuffle along the temporal dimension are incorporated into either 2D or 3D CNN networks to promote the information exchange between neighboring frames. As our method improves the basic normalization strategy for training action recognition networks, it could be potentially combined with some of these efficient designs to get improved performance.
Action Recognition with Recurrent Models. Thanks to their natural abilities to model temporal relations within video data, Recurrent Neural Networks (RNNs) instantiated as Long Short Term Memory (LSTM) networks [20] are popularly used in action recognition. This line of research is originally explored by [ in which an LSTM network is used to classify actions in soccer
videos. taking
features the input. In the deep learning era, a lot of architectural 48 52
4 55] 33 have been presented, using LSTM networks to aggregate the frame-level CNN features for video clip-level action predications. These methods follow a basic CNN-LSTM framework, using LSTM networks as the fusion module to model temporal dynamics over the frame-level features extracted by one several pre-trained CNN models. Unlike them, we investigate a new perspective for the usage of LSTM: we bridge a light-weight LSTM structure with a dynamic normalization formulation, learning to generate frame-adaptive layer-wise normalization parameters for improving the performance of existing action recognition networks.
extensions [69 12 [50, 17
Action Recognition with Attentive Models. There are lot of existing works which use different attention blocks to boost the spatial-temporal representation learning of CNNs for action recogni- tion. [61] uses self-attention block with non-local operations to capture long-range dependencies. Several recent methods [55 32 9 extend the attention designs of the SENet family [2I] 62 to
construct attention blocks for video action recognition, among which (35 addresses the attention block design for 2D CNNs while 32 and [9] are for 3D CNNs. Besides these attention blocks, dynamic convolutions (27 68 6 are used in [10 36) to build their attention blocks. (36 presents a temporal adaptive module to generate video-specific temporal kernels. Compared to these action recognition methods inspired by SENet and dynamic convolutions, our method shares the similarity of modulating learnt features dynamically to improve the spatial-temport representation learning. However, they focus on designing powerful attention blocks replace existing blocks of an action recognition network, while our method introduces two drop-in normalization relay modules built with light-weight recurrent structure in a dynamic video normalization perspective.
Normalization Methods. For modern CNNs, Batch Normalization (BN) 24 is a standard compo- nent, improving the optimization stability and accelerating the training convergence. Despite its great success, BN is prone to making an inaccurate estimation of the batched statistics when training with a small batch size, exhibiting a noticeable accuracy drop. To alleviate this issue, lot of variant methods have been proposed. Batch Renormalization (BRN) 23 introduces two extra hyperparameters to limit the estimated batch statistics. Eval Normalization (EN) 49 presents an online strategy to estimate corrected statistics for BN. Moving Average Batch Normalization (MABN) (67) uses moving average statistics in both forward and backward propagation. These three methods concentrate on improving estimated statistics, particularly mean and standard deviation for each channel. Recurrent Batch Normalization 7 utilizes the benefits of BN to normalize the hidden states of LSTM networks. Switchable Normalization (SN) [57 learns to automatically select the best choice for each layer among a predefined set of normalizers including BN, Layer Normalization (LN) (2 and Instance Normalization (IN) [56]. Group Normalization (GN) (63) divides the channels into groups and computes the group-wise batched statistics. Adaptive Normalization (AdaNorm) (66) controls the
scaling weight of LN with simple linear transformation. Batch Kalman Normalization (BKN) (57 considers layer correlations when computing statistics of a certain BN layer. Following the basic principle of BKN to model layer correlations, DIANet (22) uses an LSTM network instead of Kalman Filtering process to improve the design of convolutional attention blocks for image classification
3

---

Layer
Layeri
Layer
Framered,
IAAP
aite
Frame;
Sana
R2
Go-di
C--
Frame;-de
Coos Scspodd DNB
A schematic overview of DNR. The left figure shows cross-temporal DNR (for neighboring frames at the same layer) and the right figure shows cross-layer DNR (for the whole video clip between neighboring layers). They are encapsulated into light-weight recurrent structure. SD/LT denotes standardization//ing section. Being a drop-in design, DNR could be used to replace existing normalization layers of any action recognition network, largely improving the model training shown in the experiments (where cross-temporal DNR and cross-temporal DNR are typically used an interleaving manner).
Figure 1:
transform, and the mathematical notations are clarified in the Method
tasks. However, DIANet leads noticeable top-1 accuracy drops (over 3% dataset) when using its attention blocks to replace batch normalization layers. Recently, there are also some works attempting make the normalization be cumple-adaptive. Instance-Level Meta Normalization (ILM) (26) learns to predict the normalization parameters adaptively with an encoder- decoder sub-network. Attentive Normalization (3I] makes an integration of feature normalization and channel-wise feature attention [21]. As discussed in the previous section, these normalization methods are mainly proposed for image recognition tasks, while this work explores the dynamic normalization mechanism to promote the training of action recognition networks at multiple scales.
CIFAR-100
Method
In this section, first review mainstream normalization methods for video action recognition in general formulation, and then present the insights of our method, its formulation and design details.
3.1 Normalization Formulation
Given a deep video model, let & E RNXCXTXXX be the feature tensor extracted at a certain layer, where N is the batch size, C is the number of channels, T the video clip length, H and W are the spatial feature height and width. In the subsequent normalization, mainstream methods such as BN, IN, LN and GN perform two-stage computation. The first stage is the standardization (SD):
1
where is the feature index, and denote the feature mean and standard deviation. For all methods of BN, IN, LN and GN, the input feature tensor is divided into K non-overlapping sub-sets. The main difference of them lies in the feature partition which is defined along different feature dimensions. Let (S1, Sk) denote the resulting feature pixel sub-sets, then their mean and standard deviation
statistics could be approximated by: P* where is small positive constant. After is standardized to have zero mean and unit variance, the following stage is a channel-wise linear transform (LT) to recover the feature representation ability:
[Sc] Σn,es, I4 and σ V S Ex,es, (#:- -u)² + e.
yi -.. 3,
(2)
where y and 3 are learnable scale and shift parameters.
3.2 Dynamic Normalization and Relay
As we discussed before, existing normalization methods primarily consider image recognition tasks. Directly using them to handle video action recognition task would not likely to get satisfied

---

performance as video data usually incorporate complex -emporrll variations. Motivated by this, we present Dynamic Normalization and Relay (DNR), an improved drop-in normalization design. to boost the training of action recognition networks. Intuitively, being a normalization solution for video action recognition, it necessary to think of feature dependencies among video frames. Accordingly, our DNR explores feature dependencies at multiple scales. On the one side, note that BN is used as the standard normalization layers in almost all high-performanne action recognition networks, however it is not sample-adaptive as the learnt normalization parameters are fixed during evaluation. Recent works [21 62 27 68
31 26 I show that CNNs could acquire stronger conditioned on the pertinent priors. Inspired by them, given a normalization operator (e.g.. BN and GN, which are considered in our experiments), we first formulate our DNR dynamic normalization prediction scheme, learning to generate input-dependent normalization parameters of the linear transform (i.e., the second stage defined in Eq. 2 while calculating the parameters of the standardization usual. One the other side, existing works on the fusion module designs for action recognition [69] 12 50, 17
representation abilities when their parameters
48 52
41
[55] 33] show
the effectiveness of LSTM networks in modeling motion dynamics, and recent works the feature calibration designs for image recognition [57 [22] indicate the importance of layer correlations in learning discriminative feature representations. Inspired by them, in our DNR, we further bridge the dynamic normalization prediction scheme with two complementary relay modules namely cross- temporal DNR and cross-layer DNR, which are the core contributions of our method. They are encapsulated into light-weight LSTM structure conditioned on the current input features, and the normalization parameters estimated from the neighboring frames based features the same layer or from the whole clip based features at the preceding layers. Cross-temporal DNR enhances the dependencies of the estimated normalization parameters of layer-specific feature representations between neighboring frames, and cross-layer DNR enhances the video clip-level dependencies of the estimated normalization parameters between neighboring layers. schematic overview of DNR is shown in Figure !
In the following, we detail the formulation of DNR, considering cross-temporal DNR first. For the tth layer, let 23 denote the feature maps extracted from the thhh frame group sampled from an input video clip and 3g be the input feature computed by applying channel-wise Global Average Pooling (GAP) operations to Hy. Let the scale Yt and the shift B, be the normalization parameters need to be estimated. Cross-temporal DNR then defined as a light-weight LSTM which models the recurrent transition of normalization parameters across neighboring frames by
(Jes ic, 9t, o.) (he d,,) + du (ā+) +
(3)
where (f-) is a bottleneck unit (we will detail its structure in the next sub-section) for processing the input feature Hn and the hidden state hy-d, (d, is the relay distance which will be clarified later), and b is the bias. fax de. It, or form a set of gates to regularize the update of the LSTM by
C = (((i ② dzdd. + (ix) @ tanh(g)),
(4)
h4 o o (ct).
(5)
where G is the cell state, o.)) is the sigmoid function, and @ is the Hadamard product operator. For simplicity, we set
h., B = ct.
(6)
That is, the scale Yt and the shift Sr are the hidden state h, and the cell state et that need to be learnt. This simple setting makes the normalization parameters )t and Se for the feature maps of tth frame group be conditioned on not only the current input feature Fe but also the previously estimated Nt-d, and 3:-d, at the preceding (t d frame group. This controlled recurrent transition across neighboring frames based feature maps provides a frame-adaptive prediction of normalization parameters and allows a feature statistics relay along the temporal axis at the same layer.
As cross-temporal DNR considers the feature distribution dependencies at the same layer between neighboring frames, cross-layer DNR further extends this formulation to consider the feature distribu- tion dependencies of the entire video clip between neighboring layers. Specifically, in cross-layer DNR, the LSTM is unrolled along the network depth and makes the following changes in notations: (I) 3e denotes the input feature computed by applying channel-wise GAP operations the feature maps 2, extracted from the whole video clip at the hth layer; (2) The scale and shift parameters 7t-d, and Bi-d, known priors estimated at the (I (t-66 dh layer; (3) 7/4 and S, are the scale and shift parameters need to be leant for the hth layer.
5

---

3.3 Implementation of DNR
Now we describe the LSTM structure used in DNR. To get improved feature learning ability and enjoy good efficiency, we modify the structure of conventional LSTM [20] leveraging the wisdom of recent attention block designs for CNN architecture research. First, following the design principle bottleneck unit to process the input feature H. and the hidden state hy.d,. Let Fe; 91 E RMMC be the input and the output where C is the number of channels and M is the number of frames (in either a frame group for cross-temporal DNR or the
of SENet [21], we adopt contraction-expanse
whole video clip for cross-layer DNR), then this bottleneck unit takes the form:
2t2 = W26(W1
where W1 E RC denotes the weight matrix of Fully Connected (FC) layer which maps the imput to low dimensional space with the reduction ratio r (typically set to 4 w.r.t. the input dimension, avoiding high model complexity). 6()) denotes the Rectified Linear Unit (ReL.U) activation function, and 22 E R S xC denotes the weight matrix of another FC layer which makes the output have the same number of channels as the input. Second, following the recent practice in attentive convolutional block designs [62] [2 (68), for the activation function to compute the hidden state h4, we replace the original (anh(c)) by the sigmoid function as shown in Equation to learn slightly better 7/2 and št with DNR.
We empirically find this can help
In implementation, we plug temporal axis and along the sequential layers interleavingly. On the one side, the DNR module along the temporal axis models the frame-level correlations of the mratial-tempordl feature distributions between neighboring frames at the same layer. Here, for the cross- trosstat DNR module, each video clip is sequentially divided into a number of non-overlapping groups with the same length first, then the relay distance dr is defined as the group length (i.e., the number of frames in each group, which is computed as video clip length divided by group number). On the other side, the DNR module along the sequential layers considers all the hidden layers as a whole system, and estimates the video clip-level feature dynamics of the current layer by jointly considering the feature distributions of its preceding layers. Here, for the cross-layer DNR module, the relay distance d, is defined as the distance between two neighboring layers added with a cross-layer DNR module. Generally speaking, our DNR could be used to replace any normalization layers in popular CNN architectures for video action recognition, such as ResNet 19. ResNeXt [64 and BNInception 251 as tested in our experiments. These networks are all constructed with a block-wise design principle, and each block usually includes two or three convolutional layers. For basic blocks with skip connections. we apply the cross-temporal DNR module to the first convolutional layer and the cross-layer DNR module to the second convolutional layers between two neighboring blocks. At the input side of euch DNR pair, we additionally insert simple channel interlacing operation [34] [7 47 [3) along the temporal axis to strengthen local short-term feature interactions during the model training process. It is noted that the parameters of DNR modules could be optimized simultaneously together with any action recognition network, since their computation flow is completely differentiable.
two DNR modules into action recognition networks along the
4 Experiments
In this section, we provide comprehensive experiments to study our method from a lot of aspects, validate its effectiveness on different video action recognition datasets, and compare its performance with other related methods.
4.1 Experimental Setup
Datasets. Four public datasets, including Kinetics-400 [5) Kinetics-200 665
Something-SSomttinng
(Sth-Sth) V1 [18] and V2 [40], are considered in the experiments. Specifically, we use Kinetics-200 200 to analyze the architectural design choices of two relay modules in DNR and perform ablative studies, and use the other three datasets to testify the generalization ability of DNR. Details of these four datasets are described in the supplemental material.
Implementation Details. For fair comparisons, we choose MMaction2] for implementing all methods. On Kineties-200/4400, we randomly sample an 8-frame clip with an interval of 8 from the

---

Table 1: Exploring different design choices for DNR. Experiments performed on Kinetics- 200 dataset using action recognition model TSN with the ResNet50 backbone. The baseline has top-5 accuracy of 72.80 12.80%191.599 Best results (in terms of top-1 accuracy) are bolded.
(a) Inserted locations.
(b) Relay distance.
Method
Location
Top-I((6)
Top 5(8)
Method
Relay Distance (ra)
Top-1(%)
Top-5(%)
CT
BN1 74.52
92.55
1
74.06
92.11
BN:
75.52
92.49
2
74.56
92.33
CT
BN:
75.70
92.63
75.52
92.49
CL
BNg
74,50
92.30
8
75.28
93.13
CT+CL CL+CT (DNR)
BN; + BN₁
77.28
93.81
1
75.70
92.63
BN: + BNo
78.39
94.08
2
75.44
92.85
Relay mechanism.
Method
Relay
Top-1
Top-5((8)
(c) LSTM structure.
72.96
91.55
CL
FC
72.28
90.83
Method
Settings
'Top-1
Top Top-5(%)
v
SE
75.20
92.93
LSTM
75.70
92.63
77.98
94,23
reduction ratio (d = 1)
77.72
94.39
73.40
91.21
DNR
4
78.39
94.08
CT
FC
72.78
91.29
r
78.29
94.35
SH
75.04
92.17
depth 4)
94.08
LSTM
75.52
92.49
d 2
78.00
94.13
76.90
93.71
DNR
FC
74.50
92.13
SE
77.78
93.97
LSTM
78.39
94.08
full-length video (unless otherwise stated) during training. The input clip is resized to 340 x 256 pixels and cropped to 224 x 224 using multi-scale cropping. Following the from-scratch training strategy in (16). we adopt cosine schedule of learning rate decaying and warm-up ratio of 0.01 in the first 60K/1 28K iterations. For evaluation, we follow common practice to uniformly sample 10 clips from a video and take crops of 256 256 pixels. The prediction score is averaged over all clips. On Sth-Sth V1&V2 (with shorter video durations compared to Kinetics), we follow the pre-training (on Kinetics-400) and fine-tuning strategy, and report 1 clip and center-crop testing accuracy on validation set. Detailed implementations are in the supplemental material.
linear warm-up strategy with
4.2 Optimal Settings of DNR
We first study the optimal design choice for two modules of DNR, i.c., cross-temporal DNR and cross-layer DNR (CT and CL for short), and the optimal relay design of the LSTM structure for DNR and its two core modules. Experiments are performed on Kinetics-200 dataset using the popular Temporal Segment Networks (TSN) (60) with the ResNet50 backbone as the test case to explore these questions in the following aspects.
Inserted Locations. We first consider where place CT and CL using our default settings of LSTM structure. Table Ta summarizes the results, where BN, and BN2 denote the first and second normalization layers in a bottleneck block of ResNet50, respectively. We can find that at an individual layer, CT at BN2 and CL at BN, perform better compared to the other choices. When combining CT and CL, this combination also performs the best. Besides, we also tried to place CT/CL at BN3 yet got marginal top-1 gain (less than 0.3 percent). According to these experimental results, place CL at the first BN layer and CT at the second BN layer in each block of all networks in an interleaving manner, and used it as the default settings, yielding good accuracy and efficiency tradeoff.
Relay Distance. Once the inserted locations of two typed DNR modules are determined, how to set relay distance for each of them is now critical. For CT, each video clip is sequentially divided into number non-overlapping groups with the same length first, then the relay distance d, defined as the number of frames in each group. Considering that the input video length is 8, we explore relay distance from to From Table [bb the best performance of CT achieved when the relay distance is 4. For CL, d, is defined as the distance between two neighboring layers added with a CL module.

---

Table 2: Main results of DNR on Kinetics-200 dataset using TSN with different backbones. Best results are bolded.
Table 3: Comparison of DNR with different nor- malization methods under small training batch sizes (BS) per GPU on Kinetics-200 dataset us- ing TSN with ResNer50. Best results are bolded.
Method
Backbone
Top- ((%)
Top-5(S))
ATop 1(8)
TSN-DNR
BNInception
69.13
74.52
92.17
+5.39
Method
BS/GPU
6
TSN TSN-DNR
ResNet50
78.39
72.80
94.09
91.59
+5.59
DNR
BN
12.80/9.599
78.39/94.08
77.66/94.11
70.93/91.00
67 59..9999..
75.48/93.57
TSN TSN-DNR
ResNett01
73.58
92.29
+5.79
ATop-1/-5(5))
+5.59/2.49
+6.73/3.08
+7.89/4.35
79.37
94.51
GN
67.27/88.54
67,35/88, 42
67 07.41/88849
TSN TSN-DNR
ResNeXt101
79.53
75.
91.55
94.19
+4.43
ATop-1/-5(6)
DNR
77.22/93.35
+9.95/4.81
76.24/93.15
+8,89/4,73
75.40/92.95
+7.99/4.46
Table 4: Main results of DNR different datasets. Best results are bolded.
Dataset
Method
Backbone
Pretrain
Top-1(%)
Top-5(9)
A (Tp)
TSN TSN+DNR
ResNet50
68.17
87.97
+5.58
73.75
91.44
Kinetics 400
TSN+DNR
TSN
ResNet101
74.80
69.50
91,99
88,90
+5.30
Sth-Sth VI
TSN TSN+DNR
ResNet50
Kinetics 400
17.81
75.24
44.62
+27.77
Sth-Sth
TSN TSN+DNR
ResNet50
Kinetica-4UM
58.60
31.35
86.21
62.66
+27.25
Since the most common number of bottleneck blocks within one stage in ResNets is 3, we explore relay distance of and 2 here. From Table 1b] relaying feature distribution statistics between nearby blocks performs better than that between blocks having a larger layer distance. Therefore, we set relay distance of CT and CL to 4 and 1 respectively as the default settings.
Architectural Choices of LSTM. As both CT and CL are encapsulated into an LSTM structure, it necessary to study its architectural choices. Our LSTM structure has two basic hyper-pare parametera:: reduction ratio r and the number of bottleneck units d. Table Tc provides the results of experiments in which we alter the settings of r and d separately. It can be seen that the LSTM structure reaches a compromise between accuracy and efficiency with settings. A somewhat surprising finding is that increasing the depth of LSTM structure does not brings extra gains to model accuracy compared to the default settings. We conjecture this maybe because the optimization has already saturated with the shallow design.
and d = 1 which are used as the default
Importance of Relay Mechanism. Since relay mechanism plays an important role in our DNR design, we conduct experiments show the advantage of relay and compare different relay designs. The core idea of relay is to build the connections between learnt features statistics across neighboring frames and across neighboring layers. the experiments, we first consider the case without relay design, only using dynamic normalization of our method at the current frame or layer. To some extent, this can be considered as a simplified implementation of dynamic normalization designs such as ILM (26] and AN 31 Next, consider relay with "FC" connected) or (squeeqe and excitation) 21, meaning that we use an FC or SE unit as shared attentive relay aggregating current features and previously estimated channel statistics to conduct dynamic normalization. Table [d shows the performance comparison. We can observe: (1) Both dynamic normalization and relay are helpful to improve the top-1 accuracy of the baseline in most cases; (2) Relay with LSTM brings over 2% extra top-1 accuracy gain to dynamic normalization; (3) DNR with LSTM structure performs the best among all designs. This comparison backs up the general dynamic relay philosophy of DNR.
4.3 Main Results
To validate the effectiveness of our DNR method with the default settings identified above, we apply it to train TSN with a spectrum popular 2D video action recognition backbones on different datasets. Table prooidde the results on Kinetics-200) Clearly, DNR brings large accuracy improvements to
all baseline models with BNInception 255 ResNet [19] and ResNeXt (64] backbones, under the from-scratch training setting, yielding at most 5.79% top-1 margin and at least 4.43% top-1 margin.
8

---

Table Performance comparison of applying DNR to 3D action recognition networks with different backbones on Kinetics-200 dataset. Best results are bolded.
Method
Backbone
Top-1($)
Top-5(%)
ATop-1
R(++)D
ResNer34
71.77
91.19
+2.45
R(2+I)D+DDR
74.22
92.25
SlowFast 8x8 SlowFast 8x8+DNR
ResNet18
77.37
93.48
+1.76
79.13
93.89
We can see the improvements of DNR on Kinetics-400 and Kinetics-200 are very similar, showing its good generalization ability to a larger dataset. Significant improvements are also achieved on Sth-Sth VI&V2, demonstrating the transfer learning ability of DNR to enhance the performance of action recognition networks.
Results on Kinetics-400 and Sth-Sth V1&V2 are given in Table 4
4.4 Ablative Studies
We further provide a number of ablative studies to have deep analysis of DNR method.
Impacts of Normalization Methods and Batch Size Settings, Being a drop-in design. DNR could be applied to other popular normalization methods besides BN. In image recognition field, GN [63 is known to have much better capability to combat small batch size training problem compared to BN. Accordingly, we also conduct experiments on Kinetics-200 to explore the effectiveness of applying DNR to GN besides BN, as well as its advantage in small batch size training. For a fair comparison, adopt the linear learning rate scaling rule commonly used in GN to adapt to batch size changes. Specifically, we make the learning rate be proportional to the batch size (0.1 x N/8, where is the batch size per GPU). For instance, under the batch size of 6/6/4, the learning rate is initialized to 0.1/0.055/0. 0..057500555 and decayed with a cosine scheduling. We also use a linear warm-up strategy with warm-up ratio of 0.01 in the first 60K iterations. T to BN and GN with different batch sizes. As can be seen, DNR outperforms the baseline method with consistently large margins (at least 5.59% gain and at most 9.95% gain top-1 accuracy). Moreover, to BN, DNR achieves larger margins when reducing mini- size per GPU from 8 to 4 video clips, showing its advantage in resource constrained training scenarios. Compared to the results on BN with DNR, reverse performance improvement trend can be observed on GN with DNR. This is because GN performs better in small batch size training scenarios while performing worse large batch size training scenarios. We also compare DNR with dynamic normalization (without relay) in Table [d]
the performance of applying DNR
Applying DNR to 3D CNNs. Although 3D CNNs generally have better spatial-temporal feature modeling capabilities than 2D CNNs, we also apply DNR to 3D CNN backbones to further evaluate its potential in enhancing apatial-temporaa representation learning. Restricted by our available computational resource, we choose R(2+1)D [54] with ResNet34 and SlowFast [16] with ResNet18 as the baseline models, which are relatively liggh- in 3D CNNs. As shown in Table DNR brings around 2% top-1 accuracy boosts for two baseline models, demonstrating its efficacy to promote the performance of 3D CNNs besides 2D ones.
Applying DNR to State-of-the-art Methods. In order to validate the capability of DNR to boost action recognition networks with advanced temporal modules, we particularly perform another set of experiments on Kinetics-400 dataset. In the experiments, besides TSN (60], we also choose TSM (34) TANet 36 and TDN [59] the baseline models, which three latest 2D attentive designs for spatial-temporal modeling. We apply DNR to these four baselines for performance evaluation under the exactly same training and test settings. Table 6hows the results, and also provides a comparison of DNR with some state-of thce-of-thhearr art methods including 3D and 2D CNNs. Comparatively, TSN+DNR 2.7% better than TSN (with 71.65% top-1 accuracy), TSM+DNR is 1.1% better than TSM (with 73.78% top-1 accuracy). TANet+DNR is 0.65% better than TANet (with 76.28% top-1 accuracy), and TDN+DNR is 0.71% better than TDN (with 76.34% top-1 accuracy). Note these four baseline models contain three types of advanced action recognition models with 2D CNNs. Specifically, TSN 660 the very basic temporal module, TSM (34 uses an improved temporal module with shift operations across neighboring frames, while more advanced attentive temporal modules are used in TANet 36 (built with dynamic convolutions) and TDN (59) (encoding frame differences into the attentive design). Particularly, TANet and TDN are two of the latest perforronnn 2D attentive

---

Table 6: Comparison of method with state-of-the-ar methods Kinetics-400 dataset. "o" and "*" refer to our reproduced models using MMaction2 and public code released by the authors, respectively. Best results are bolded.
Model
Backhone
Pretrain
Frames
GFL.PPsx Views
Top-1(%)
Top-5(%)
SlowFast
T6
ResNet50
None
(4+32)x 10x3
36.1x330
75.60
92.1
SlowFast
[6
ResNet101
None
(8+32)x10x4
106x30
77.90
93.20
NL SlowFast
16
ResNet101
None
(16+64)x10x3
234x30
79.80
93.90
X3D L5
X3D-XL
None
16x10x3
48.4x
79.10
93.90
SmallBigNet
30
ResNet50
None
8x10x3
57x30
76.30
92.50
TSN
ResNet50
ImageNet
x0
x.11x250
70.60
89.26
TSM 34
ResNet50
ImageNet
8x10x3
33x30
74.10
N/A
TEINet
35
ResNet50
ImageNet
8x10x3
33x30
74.90
91.80
TEA
32
ResNet50
ImageNet
16x 10x3
70x30
76.10
92.50
TAM
14
Re.ReNNe500
Kinetics
48x3x3
93.4x9
73.50
91.20
TANel
ResNet50
ImageNet
8x10x3
36x30
76.09
92.30
TDN
ResNet50
ImageNet
8x10x3
36x30
76.60
92.80
TSN
ResNet50
None
8x10x3
33x30
71.65
90.17
TSN+DNR
ResNet50
None
8x10x3
33x30
74.35
91.68
TSM
ResNet50
ImageNet
8x10x3
33x30
73.78
91.32
TSM+DNR
ResNet50
ImageNet
8x10x3
33x30
74.88
91.99
TANet
ResNet50
ImageNet
8x10x3
36x30
76.28
92.46
TAN
ResNet50
ImageNet
8x10x3
36x30
76.93
92.81
TDN'
ResNet50
ImageNet
8x10x3
36x30
76.34
92.63
TDN+DNR
ResNe150
ImageNet
8x10x3
36x30
77.05
92.98
temporal modules, in this point of view, the performance improvement of our DNR decent as DNR merely added to normalization layers. Generally, DNR performs the best among 2D CNNs under the same settings.
More Ablative Experiments. In the supplemental material, provide more ablative experiments, showing: (1) Bidirectional LSTM design can slightly improve cross-temporal DNR: (2) The number of extra FLOPss introduced by DNR is almost negligible, and the extra runtime latency is not negligible but is acceptable; (3) DNR learns better peatialttemorra features by visualization comparisons.
4.5 Limitations of DNR
Despite that DNR shows promising performance improvements to lot of action recognition net- works, its limitations in two aspects. Firstly, in the experiments, provide the comparison of computational cost (in terms of both FLOPs and runtime inference speed) of different baseline models without and with DNR, showing although DNR introduces almost negligible extra FLOPs different baselines, it introduces extra latency at inference stage. Secondly, DNR has three critical hyperparameters, namely the layer location, the relay distance and the L.STM structure for setting cross-temport and cross-layer DNR modules. Although we provide relatively thorough experiments on Kinetics-200 dataset with the TeN+Re baseline to study the setting for each of them, and apply the resulting settings to all baseline models tested in our experiments, it is not the optimal settings to different action recognition networks. Besides, the potential of applying DNR emerging new types of deep action recognition networks is reserved as a future extension.
Conclusions
In this paper, we present DNR, drop-in normalization method, to promote the ppatial-temporal representation learning of CNNs for video action recognition. The core contributions DNR are cross-temporal and cross-layer dynamic normalization and relay designs, learning to generate frame- adaptive wase normalization parameters at both training and runtime. The efficacy of DNR was validated by thorough experiments several public action recognition datasets.
10

---

References
[1] Marcin Andrychowicz, Misha Denil, Sergio Gómez, Matthew Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando de Freitas. Learning to learn by gradient descent by gradient descent. In NIPS, 2016.
[2] Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv preprint arXiv:1607.064500 2016.
[3] Moez Baccouche, Franck Mamalet, Christian Wolf, Christophe Garcia, and Atilla Baskurt. Action classification in soccer videos with long short-term memory recurrent neural networks. In ICANN, 2010.
[4] Nicolas Ballas, Li Yao, Chris Pal, and Aaron Courville. Delving deeper into convolutional networks for learning video representations. In ICLR, 2016.
[5] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In CVPR, 2017.
[6] Yinpeng Chen, Xiyang Dai, Mengehen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic convolution: Attention over convolution kernels. In CVPR, 2020.
[7] Tim Cooijmans, Nicolas Ballas, César Laurent, Çaglar Gülçehre, and Aaron Courville. Recur- rent batch normalization. In ICLR, 2017.
[8] Navneet Dalal, Bill Triggs, and Cordelia Schmid. Human detection using oriented histograms of flow and appearance. In ECCV, 2006.
[9] Ali Diba. Mohsen Fayyaz, Vivek Sharma, M Mahdi Arzani, Rahman Yousefzadch, Juergen Gall, and Luc Van Gool. Spatio-temporal channel correlation networks for action classification. In ECCV, 2018.
[10] Ali Diba, Vivek Sharma, Lue Van Gool, and Rainer Stiefelhagen. Dynamonet: Dynamic action and motion network. In ICCV, 2019.
[11] Piotr Dollár, Vincent Rabaud, Garrison Cottrell, and Serge Belongie. Behavior recognition via sparse spatio-temporal features. In IEFE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance, 2005.
[12] Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. Long-term recurrent convolutional networks for visual recognition and description. In CVPR, 2015.
[13] Linxi Fan, Shyamal Buch, Guanzhi Wang. Ryan Cao, Yuke Zhu, Juan Carlos Niebles, and Li Fei-Fei. Rubiksnet: Learnable 3d-shift for efficient video action recognition. In ECCV, 2020.
[14] Quanfu Fan, Chun-Fu Richard Chen, Hilde Kuehne, Marco Pistoia, and David Cox. More less: Learning efficient video representations by big-little network and depthwise temporal aggregation. In NewrIPS, 2019.
[15] Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In CVPR, 2020.
[16] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, 2019.
[17] Harshala Gammulle, Simon Denman, Sridha Sridharan, and Clinton Fookes. Two stream Istm
A deep fusion framework for human action recognition. In WACV, 2017.
[18] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, al. The" something something" video database for learning and evaluating visual common sense. In ICCV, 2017.
11

---

[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[20] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.
[21] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-exciaation networks. In CVPR, 2018.
[22] Zhongzhan Huang. Senwei Liang. Mingfu Liang, and Haizhao Yang. Dianet: Dense-and- implicit attention network. In AAAI, 2020.
[23] Sergey loffe. Batch renormalization: Towards reducing minibatch dependence in batch- normalized models. In NIPS, 2017.
[24] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
[25] Sergey Ioffe, Christian Szegedy, Francis R. Bach, and David M. Blei. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
[26] Songhao Jia, Ding-jie Chen, and Hwann-Tzong Chen. Instance-level meta normalization. In CVPR, 2019.
[27] Xu Jia, Bert De Brabandere. Tinne Tuytelaars, Lue Van Gool, Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Gamett. Dynamic filter networks. In NIPS, 2016.
[28] Alexander Klaser, Marcin Marszalek, and Cordelia Schmid. A spatio-temporal descriptor based on 3d-gradients. In BMVC, 2008
[29] Ivan Laptev and Tony Lindeberg. Space-time interest points. In ICCV, 2003.
[30] Xianhang Li, Yali Wang. Zhipeng Zhou, and Yu Qiao. Smallbignet: Integrating core and contextual views for video classification. In CVPR, 2020.
[31] Xilai Li, Wei Sun, and Tianfu Wu. Attentive normalization. In 2020
[32] Yan Li, Bin Ji, Xintian Shi, Jianguo Zhang, Bin Kang, and Limin Wang. Tea: Temporal excitation and aggregation for action recognition. In CVPR, 2020.
[33] Zhenyang Li, Efstratios Gavves, Mihir Jain, and Cees G. M. Snoek. Videolstm convolves, attends and flows for action recognition. CVIU, 2018.
[34] Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understand- ing. In ICCV, 2019.
[35] Zhaoyang Liu, Donghao Luo, Yabiao Wang, Limin Wang. Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, and Tong Lu. Teinet: Towards an efficient architecture for video recognition. In AAAI, 2020.
[36] Zhaoyang Liu, Limin Wang. Wayne Wu, Chen Qian, and Tong Lu. Tam: Temporal adaptive module for video recognition. arXiv preprint arXiv:2005.0688 2020.
[37] Chenxu Luo and Alan Yuille. Grouped spatial-temporal aggregation for efficient action recogni- tion. ICCV, 2019.
[38] Ping Luo, Zhanglin Peng, Wenqi Shao, Ruimao Zhang, Jiamin Ren, and Lingyun Wu. Differen- tiable dynamic normalization for learning deep representation. In JCML, 2019.
[39] Pingchuan Ma. Yao Zhou. Yu Lu, and Wei Zhang. Learning efficient video representation with video shuffle networks. arXiv preprint arXiv:1911.1131) 2019.
[40] Farzaneh Mahdisoltani, Guillaume Berger, Waseem Gharbieh, David Fleet, and Roland Memi- sevic. On the effectiveness of task granularity for transfer learning. arXiv preprint arX- 1v:1804.09235, 2018
12

---

[41] Li Qing, Zhaofan Qiu, Ting Yao, Tao Mei, Rong Rui, and Jiebo Luo. Action recognition by learning deep multi- granular spatio-temporal video representation. 2016.
[42] Zhaofan Qiu, Ting Yao, and Tao Mei. Learning residual networks. In ICCV, 2017.
representation with pseudo-3d
[43] Zhaofan Qiu, Ting Yao, Chong- Wah Ngo, Xinmei Tian, and Tao Mei. Learning spatio-temporal representation with local and global diffusion. In CVPR, 2019.
[44] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 2015.
[45] Sreemanananth Sadanand and Jason Corso. Action bank: A high-level representation of activity in video. In CVPR, 2012.
[46] Paul Scovanner, Saad Ali, and Mubarak Shah. A 3-dimensional sift descriptor and its application to action recognition. In MM, 2007.
[47] Hao Shao, Shengju Qian, and Yu Liu. Temporal interlacing network. In AAAI, 2020.
[48] Yemin Shi, Yonghong Tian, Yaowei Wang, Wei Zeng, and Tiejun Huang. Learning long- dependencies for action recognition with a pellogically-inspieee deep network. In ICCV, 2017.
[49] Saurabh Singh and Abhinav Shrivastava. Evalnorm: Estimating batch normalization statistics for evaluation. In ICCV, 2019.
[50] Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using lstms. ICML, 2015.
[51] Swathikiran Sudhakaran, Sergio Escalera, and Oswald Lanz. Gate-shift networks for video action recognition. In CVPR, 2020.
[52] Lin Sun, Kui Jia, Kevin Chen, Dit Yan Yeung, Bertram E. Shi, and Silvio Savarese. Lattice long short-term memory for human action recognition. 2017.
[53] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In ICCV, 2015.
[54] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray. Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In CVPR, 2018
[55] Amin Ullah, Jamil Ahmad, Khan Muhammad, Muhammad Sajjad, and Sung Wook Baik. Action recognition in video sequences using deep bi-directional Istm with enn feature. IEEE Access, 2017.
[56] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv: 1007.0000,,, 2016.
[57] Guangrun Wang, Ping Luo, Xinjiang Wang, and Liang Lin. Kalman normalization: Normalizing internal representations across network layers. NeurlPS, 2018
[58] Heng Wang and Cordelia Schmid. Action recognition with improved trajectories. In ICCV, 2013.
[59] Limin Wang, Zhan Tong, Bin Ji, and Gangshan Wu. Tdn: Temporal difference networks for efficient action recognition. In CVPR, 2021.
[60] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In ECCV, 2016.
[61] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018.
13

---

[62] Sanghyun Woo, Jongchan Park, Joon- Young Lee, and In So Kweon. Cbam: Convolutional block attention module. In ECCV, 2018.
[63] Yuxin Wu and Kaiming He. Group normalization. ECCV, 2018.
[64] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR, 2017.
[65] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy, Rethinking s- patiotemporal feature learning: Speed-accuracy trade-offs in video classification. In ECCV, 2018.
[66] Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. Understanding and improving layer normalization. In NeurlPS, 2019.
[67] Junjie Yan, Ruosi Wan, Xiangyu Zhang, Wei Zhang, Yichen Wei, and Jian Sun. Towards stabilizing batch statistics in backward propagation of batch normalization. In ICLR, 2020.
[68] Brandon Yang, Gabriel Bender, Quoe V Le. and Jiquan Ngiam. Condeonv: Conditionally parameterized convolutions for efficient inference. In NeurIPS, 2019.
[69] Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijaanarasimmann, Oriol Vinyals, Rajat Monga, and George Toderici. Beyond short snippets: Deep networks for video classification. In CVPR, 2015.
[70] Yi Zhu, Xinyu Li, Chunhui Liu, Mohammadreza Zolfaghari, Yuanjun Xiong, Chongruo Wu, Zhi Zhang, Joseph Tighe, R. Manmatha, and Mu Li. comprehensive study of deep video action recognition. arXiv preprint arXiv:2012.06567 2020.
[71] Mohammadreza Zolfaghari, Kamaljeet Singh, and Thomas Brox. Eco: Efficient convolutional network for online video understanding. In ECCV, 2018.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you descrihe the limitations of your work? [Yes] See Section 4 and supplemental
material.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL.)? [Yes] See Abstract, Section 4 and supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section and supplemental material.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [No] Error bars are not reported because it would too computationally expensive.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section and supplemental material.
4. If you using existing assets (c.g., code, data, models) or curating/releasing assets...
14

---

(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 4
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material URL? [Yes]
See Abstract.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals. if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "_BF2jfMmT6h", "reviewer": "Reviewer_zNtD", "summary": "This paper presents a Dynamic Normalization and Relay (DNR) module to augment the spatial-temporal representation learning for action recognition. The proposed DNR module can ease the effect of small batch size for video action recognition therefore improve the performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The authors proposed a DNR module to dynamic normalize each layer according to the input sample, which can ease the weakness of small batch size when training a video action recognition networks.\n- Extensive  ablation studies are conducted to show the effectiveness of their method.\n\nWeaknesses:\n- How the standardization parameters \\mu and \\sigma  are calculated is not very clear in the paper, is that the same as BN?\n- The runtime of their method compared with other methods should be given apart from FLOPs, because they use LSTM structures in their module, FLOPs cannot represent the true performance of the model", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "ZXv2VA8yMOs", "reviewer": "Reviewer_rsjC", "summary": "The method proposes a module to dynamically scale the tensors within a temporal action recognition model that can act as a drop-in addition to existing models. To achieve this, the per-frame input tensors are pooled into vectors, and a LSTM is used to condition the linear scaling of the tensor depending on the (avgpooled) representation of the current frame, but also previous predictions. This is also extended to include the predictions from the previous layer. There are also a couple of tweaks to the LSTM to improve performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper becomes somewhat clear as you read through, but it is not easy to get into. The terminology and the way the explanations go don't let the reader get an early understanding of what it's been tackled. For example, reading through the abstract I honestly had no idea what the paper was actually proposing. Then there are a number of explanations and terminology that are not easy to get through their meaning (I still don't understand what's \"meta\" about the method). \n\nThe paper is somewhat novel. The mechanism used is indeed novel, but similar self-referential scaling modules are common for images and there are definitely multiple variants for video (something for example noted in the literature review). While the authors propose an LSTM-based methodology, some other works have proposed convolution-based approaches, but the global aim looks similar to me. Speaking in an abstract (a bit hand-wavy), you have a function that takes a sequence of input tensors and predicts linear scaling. That function, in this case, is an LSTM variant. Despite this, the variant is a valid one methodologically speaking, and in my opinion the approach is correct.\n\nFrom the experimental point of view, experiments need to be more conclusive (and I know there are a lot of tables!). The main problems are a) On Kinetics400, the performance improvement seems to go down as the method is better (TSN+Ours is 2.7% better than TSN, but TSM+Ours is +1.1% better than TSM, and TDN+Ours is 0.71% better than TDN). b) I miss a more direct comparison with other methods with similar aim - I know the module improves over base architectures, but there are other alternatives to combine the base architectures with (e.g. section \"action recognition with attentive models\" lists a few). In order to be a useful module, you not only need to improve the base architecture, you also need to present an advantage over similar modules. c) FLOPs are ok (and the numbers look good), but timings would make a lot of sense as these modules are oftentimes quite sequential in nature and FLOPs do reflect particularly poorly what the impact is in practice (admittedly, this is not something common in the literature so it is hard to hold it against the authors).\n\nSmall details (no need to address them, just suggestions):\n\nThe ssv1/v2 table on the supplementary shoudl be upgraded into the main body, and maybe Table 4 is spurious and can be replaced instead?\n\nThe authors might want to revise a bit Efficient Spatial-Temporal Modelling. Some assertions do not sound great to me, e.g. l77 (mixing 2D and 3D convs) and l78 (two-paths networks are not designed for efficiency - slowfast adds the second path on top of a standard, relatively heavy backbone, so it is \"efficient\" in the sense that it doesn't add \"too much\" for the accuracy gain). \n\nWhy is the \"meta\" term used? wouldn't it make the paper easier to understand to drop it?\n\nThe paragraph starting on l209 is not particularly clear when explaining the positioning of the modules. Maybe the authors could revise this paragraph to make it easier to understand (Later the ablation study has a very good explanation, which is the one I used to understand it)\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not discuss the limitations despite claiming to do so in the introduction and supplementary. In terms of ethical impact, they reply N/A and might be correct.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "GGn1bWSInNG", "reviewer": "Reviewer_kHhy", "summary": "This paper proposes a novel dynamic batch normalization and relay (DNR) approach regarding the sequential information in videos. The authors have demonstrated extensive experiments and shown the effectiveness of the proposed DNR for a number of advanced backbones for video action recognition.\n\nApart from some technical confusions and questions, it is an interesting paper and the idea of dynamic batch normalization may be useful to the community.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a novel dynamic batch normalization and relay (DNR) approach regarding the sequential information in videos. The authors argue that traditional batch normalization is not frame-adaptive and ignores dependencies of the estimated normalization parameters between neighboring frames. Then the proposed DNR contains cross temporal DNR (CT) and cross layer DNR (CL) to handle the above problems. \n\nThe paper is well presented and the idea of DNR for video action recognition is novel. The authors have demonstrated extensive experiments and shown the effectiveness of the proposed DNR for a number of advanced backbones for video action recognition. Apart from good results in the paper, I have few technical questions on the paper below:\n\n1. P2 L54 mentions that the normalization parameters are generated dynamically both on training and inference. The authors also use the word “meta learning” in the abstract. Traditional meta learning requires extra training or fine-tuning procedure during the inference. However, I did not see that specific description on extra training on inference for specifying normalization parameters. I suppose inference stage directly uses the trained parameters and does not contain extra training during inference. If this is exactly what you are doing, I think the word “meta learning” and meta LSTM is somewhat misleading.\n\n2. P7 L256 “relay distance is the number of sampled frames”. What does this mean? Does this correspond to the num_goups in _TBatchHyperNorm class in the provided codes? \n\n3. P8 L305 “We compare DNR with dynamic normalization (without relay) in Table 1d.” Does no relay mean that (e.g., for CT) operate batch normalization simply on different frames independently? \n\n4. Why not also perform three crops evaluation on SthV1 & SthV2 ? I understand 1 centre-crop is more efficient but three crops evaluation is able to give more convincing results to the audience.\n\n5. It is good to see DNR introduces very limited extra FLOPs to the model in Table 6. How about the inference speed compared to no DNR? In some real applications, inference speed is more important than the introduced FLOPs. \n\n6. In table 3, the authors experiment with different batch size performances. Is the [linear scaling rule](https://research.fb.com/wp-content/uploads/2017/06/imagenet1kin1h5.pdf) used in this experiment since the different batch size requires different learning rates? The authors should clarify the learning rate used for each batch size experiment. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Have not addressed adequate limitations and potential negative societal impact.\n\nIt can be done such as by providing the inference speed of with and w/o DNR to demonstrate how much latency is introduced to the inference stage. Also, there are a number of hyperparameters in DNR which may cause differences on different backbones.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents work on action recognition.  The main contribution is a drop-in module that dynamically scales tensors.  The initial reviews pointed to concerns over clarity, novelty, and experimetns.  The reviewers engaged in discussion and were satisfied with the authors' responses to their initial concerns.  The reviewers unanimously recommended acceptance of the paper after the discussion period, based on the novel approach to dynamic scaling for videos.  The authors are encouraged to make changes to presentation to help in describing the contributions in a clearer fashion, especially earlier in the paper, to improve its readability and future impact.

Decision: Accept (Poster)