Dense Correlation Fields for Motion Modeling in Action Recognition
Authors: Yichen Qian
Junyan Wang
Xiuyu Sun
Venue: Submitted to ICLR 2023 (2023)
Keywords: Action recognition
Motion modeling
Video understanding

Abstract: The challenge of action recognition is to capture reasoning motion information. Compared to spatial convolution for appearance, the temporal component provides an additional (and important) clue for motion modeling, as a number of actions can be reliably recognized based on the motion information. In this paper, we present an effective and interpretable module, Dense Correlation Fields (DCF), which builds up dense visual correlation volumes at the feature level to model different motion patterns explicitly. To achieve this goal, we rely on a spatially hierarchical architecture that preserves both fine local information provided in the lower layer and the high-level semantic information from the deeper layer. Our method fuses spatial hierarchical correlation and temporal long-term correlation, which is better suited for small objects and large displacements. This module is extensible and can be plugged into many backbone architectures to accurately predict object interactions in the video. DCF shows consistent improvements over 2D CNNs and 3D CNNs baseline networks with 3.7% and 3.0% gains respectively on the standard video action benchmark of SSV1.

Under review as a conference paper at ICLR 2023
DENSE CORRELATION FIELDS FOR MOTION MODEL- ING IN ACTION RECOGNITION
Anonymous authors Paper under double-blind review
ABSTRACT
The challenge of action recognition is to capture reasoning motion information. Compared to spatial convolution for appearance. the temporal component pro- vides additional (and important) clue for motion modeling, as a number of actions can be reliably recognized based on the motion information. per, we present effective and interpretable module, Dense Correlation Fields (DCF), which builds up dense visual correlation volumes at the feature level to model different motion patterns explicitly. To achieve this goal, rely spa- tially hierarchical architecture that preserves both fine local information provided in the lower layer and the high-level semantic information from the deeper layer. Our method fuses spatial hierarchical correlation and temporal long-term correla- tion, which is better suited for small objects and large displacements. This module is extensible and can be plugged into many backbone architectures to accurately predict object interactions in the video. DCF shows consistent improvements over 2D CNNs and 3D CNNs baseline networks with 3.7% and 3.0% gains respectively on the standard video action benchmark of SSVI.
this pa-
INTRODUCTION
Action recognition is fundamental problem in video understanding (Karpathy al. 2014 Laptev ett al., (2008). Unlike image classification, action recognition should distinguish visual tempo vari- ation as well as its semantic appearance. Recently, great progress has been made by deep learning based models to improve the accuracy of video action recognition (Feichtenhofer et al. 2019 Jiang al., 2019 Yang et al. 2020a). CNNs for video understanding has been extended with the capa- bility of capturing not only appearance information contained in individual frames but also motion information extracted from the temporal dimension of the image sequence.
One common method for action recognition is to use two-stream network (Simonyan & Zisserman 2014 Crasto et al.) 2019; Feichtenhofer et al., 2016 Qiu et al., 2019), where one stream on raw frames to extract appearance information, and the other is to leverage optical flow to learn motion information. An alternative strategy implicitly uses 3D CNNs (Carreira Zisserman [017]] Tran eet al. 2015a Feichtenhofer al. 2019) temporal convolution (Tran a 2018 Xie al. 2018),
as these methods can jointly capture spatial and temporal information framework. Some other methods extend 2D CNN-based backbones with temporal modules (Lin)
unified spatiotemporal
cet al., 2019 Li et al.) 2020b] Meng et al. 2021 Liu et al.] (022b) to learn motion information. However, the pertormance
previous action recognition systems is limited by difficulties including small objects and large displacements (fast moving objects). One conundrum for these methods is that the high-level feature semantically strong but spatially coarse, as the spatial feature is crucial to capture motion information. As the cases shown in Figure [ strong semantic features and fine spatial features are both the keys distinguishing action classes.
This paper propose Dense Correlation Fields (DCF), new temporal module for motion modeling. The correlation operator captures the motion information by computing the alignment of visually similar image regions between frames. Visual correlation is highly relevant to optical flow, which the most important clue for capturing motion patterns. The correlation operator can be used as approximate motion information, which has shown effectiveness in action recognition in CorrNet (Wang et al. 2020). DCF combines low-resolution, semantically strong correlation features with high-resolution, semantically weak correlation features to recover different motion patterns. Our

---

Under review as a conference paper at ICLR 2023
(a) Small object (the pen on the box)
(b) Moving something at fast tempo
Figure 1: The action examples above show small object and large displacement (fast moving object) from SSVI valid videos. (a) This example is captioned with 'Moving something across surface until it falls down'. There is another similar type of action where the object does not fall down. The action of the pen on the box is crucial to distinguish these two classes. (b) This example shows fast-tempo action with large displacement across the frames.
DCF consists of two main contributions: (1) correlation aggregation over a spatial pyramidal hierar- chy; (2) fross-frame correlation volume with both short-term and long-term temporal information. Our DCF enables the efficient integration of spatial information and semantic information for motion modeling throughout the network.
The design of DCF draws inspiration from many existing works but is substantially novel. First, DCF builds up dense fields by combining features from spatial pyramidal correlation hierarchy. This is different from the individual temporal module applied over multi-stage in prior works (Wang et al. 2020 2018b Huang al. 2021). Temporal modeling normally presents short-term motion between adjacent frames at low-level feature and long-term temporal aggregation at high-level feature. These methods use motion modeling as different stages deal with different motions. In practice, this fails in cases where the determined motion
lower scale is too spatially coarse to be close to the correct motion of a higher scale. Our DCF uses spatial pyramid hierarchy to hallucinate spatially coarser but semantically stronger correlation feature by spatially finer correlation feature. The principle advantage of featuring each level of a correlation pyramid is that it produces a multi-scale motion feature representation in which all levels spatially fine, including the low-resolution levels.
Second, DCF maintains a cross-frame correlation with both short-term and long-term temporal in- formation. While CorNNet only uses the correlation between adjacent frames, compute the cor- relation between consecutive frames to form long-term temporal information, following the strategy of previous methods (Wang et al. 2021 2018b). DCF provides the network with long-term motion information by operating on cross- frame correlation volume.
DCF can be applied to different backbone architectures as plugin module. We construct DCF net- works with two backbone networks, (R(2+1)D (Tran et al. 2018) and X3D(Feiihhenhoofer 2020
In order to evaluate the proposed method in terms of modeling motion variations, we construct ex- periments on the Something Soahhinng-Smmhiingg dataset (Goyal et al. 2017) which has been well-known to be challenging to classify an action due to the temporal complexity. In addition, we validate various design choices of DCF through extensive ablation studies. Moreover, we show the performance on Kinetics-400 dataset (Kay al. 2017) to compare the proposed method to the many state-of-the- arts.
2 RELATED WORK
Action Recognition. Action recognition research has been largely driven by learned features and various learning models utilizing deep networks. Two-stream CNNs (Simonyan & Zisserman 2014) with one stream of static images and the other stream of optical flows are proposed to fuse the infor- mation of appearance and motion. Temporal Segment Networks (Wang et al. (016) sample frames and optical flow on different time segments to extract information for activity recognition. 3D-CNNs Jji et al. 2012 Tran et al. (2015b) proposed 3D convolution to directly learn spatiotemporal features from videos. Several variants decompose 3D convolution into a 2D convolution and ID temporal convolution, for example P3D Qiu et al. (2017), R(2+1)D (Tran et al. 2018), S3D (Xie et al. 2018), and CT-Net (Li al. 2021). Recently, the great success of image Transformers has led to investi-
2

---

Under review as a conference paper at ICLR 2023
gation of Transformer-basee architectures for video recognition tasks (Liu al. 2021a Bertasius et al. 2021 Arnab et al. 2021 Fan et al. [022 Neimark et al. 2021) Our work is motivated by the success in incorporating optical flow for action recognition. There is an important issue in existing methods with optical flow for action recognition: the dependency on beforehand extraction of opti- cal flow lowers the efficiency and effectiveness of the recognition system. Our DCF module inherits the effectiveness of optical flow and provides the video model with precise motion information.
Motion Modeling for Action Recognition. For action recognition, the appearance of still frames and motion information such as optical flow is the most important cues to identify the action. Many recent works (Choutas et al., 2018; Sun al., 2018 Jiang et al., 2018 Fan al. 2018 Asghari- Esfeden et al. [2000] Wu et al. 2018) design powerful temporal modules and insert them into 2D CNNs for efficient action recognition. TRN (Zhou et al. 2018) and TSM (Lin et al., 2019) cap- ture information along the temporal dimension with an interpretable relational module and a shift module respectively. Some methods learn motion information by incorporating optical flow or RGB difference for action recognition systems. TDN (Wang et al. 2021 generalizes temporal difference operator for capturing both short-term and long-term temporal. ActionFlow.Ne (Ng et al.) 2018) proposes to jointly estimate optical flow and recognize actions in one network. Several works pro- pose carefully designed modules to capture different types of temporal dependency, such as Slowfast (Feichtenhofer al. 2019), TPN (Yang et al. 2020a), Non-local Net (Wang al. 2018b).
2018a) introduce cost volume or multiplicative inter- actions without the reliance on optical flow. CorrNet Wang al.] (020) and MSNettKwoo
Some methods (Zhao et al. 2018 Wang et al.
al.) 2020) also compute frame-to-frant frafee correlation over feature maps for effective motion estimation. (Piergiovanni & Ryoo| 2019) propose flow layer that unrolls the iterations of the TV-LI algo- rithm with learned parameters. DynamoNet(Diba et al. (019)) propose dynamic motion filters by predicting the future frames enrich motion representation. However, these methods learn motion representation at single level, or multiple levels of the network individually, which may encounter a problem that local information disappears at spatially coarse levels. (Yang et al., (020b) also present hierarchical method to bridge different levels in network. This hierarchical design use discrimi- native contrastive loss to enforce the motion features at high-level to predict the ones at low-level. However, this contrastive learning encourages features to have more similar representations, leading to less diversity to cover motion patterns. In contrast, we combine low-level features with high-level features via top-down and skip connections. In this way, fine local information provided in the lower layer and the high-level semantic information from the deeper layer.
preserve the diversity from the
APPROACH
The Dense Correlation Fields are designed to model motion patterns explicitly. The proposed DCF can be applied to different backbone architectures a plugin module. An overview of our approach is given in Figure 2 Given a video clip, we sample T frames as input I with the shape of is [T, H, W, D). where D is the number of channels and H, W the spatial resolution. The backbone network is applied to and maps the input frames to feature maps at lower spatial resolution. This process is independent of the backbone architectures, and here we present a backbone example using ResNets(He et al. 2016). Typically, a ResNer-like backbone outputs feature pyramid at resolutions for multiple stages, and note that they have spatial strides of 4, 8, 16, 32. We construct Dense Correlation Fields by exploiting connections that associate correlation volume across multiple stages. DCF is built upon collection of spatial pyramid correlation features. We first detail the specific instantiation of the correlation operator, which computes frame-to-frane visual similarity. Afterwards, we show how our DCF aggregates motion information over the correlation features from spatial pyramid feature hierarchy. Finally, provide the implementation detail to instantiate DCF with backbone networks.
3.1 BASIC CORRELATION OPERATOR
The correlation operator explicitly captures the motion information by computing visual similarity between two frames. Given two frame features fi R #xxD and fitr E "xx X the corre- lation operator volume is formed by taking the dot product between feature vectors. For a feature vector (i. we compute the similarity of this vector with vectors in fetr, where (i.j) is the
3

---

Under review as a conference paper at ICLR 2023
Video Clip
Backbone
Dense Correlation Fields
Dosn Sumple
Down Sample
APA
W74
w/4
W//3
W256
Furam
Cample
Operstor
A/8
W233
Fuccon
Aya
Commaatinn Openinor
A/16
Fucen
The framework of Dense Correlation Fields with spatial pyramid feature hierarchy. A Backbone network extracts multi-level features for video clip. A correlation layer constructs a 4D correlation volume with shape x x Tx x (K x K 2T.) for each level feature. indicates the spatial region and I. indicates the temporal range. Correlation volumes in the low-level stages are down-sampled generate multi-scale volumes, which are connected high-level stages. The vol- umes from the current stage and the bottom-up pathway combined to learn motion information. learned block aggregates the motion information from the correlation hierarchy. Our dense corre- lation fields maintain coarse and fine resolution information, also long-term and short-term temporal information. helps to handle various object sizes and motion tempos in action recognition.
Figure 2:
spatial location of the vector. We consider pixel-to-pixel similarity between two frames, ft and fitrs
Cifa f(+t) RH*WKK* C.j = VD (flli.).
(1)
where J/b is for normalization. (i', is often limited in a K x K spacial neighborhood of (i.j) for computational reason. A correlation volume can be seen as a of visual similarities between the pairs of consecutive frames f and f +
3.2 DENSE CORRELATION FIELDS
In this subsection, we describe Dense Correlation Fields, which improve the action recognition system's robustness to capture motion variation. We first consider an extension of correlation from image-wise to video-wise. This aims to better capture long-term dependencies. Then we build spatial pyramid correlation features on top of it. Our goal preserve both fine spatial information provided in the lower layer and the high-level semantic information from the deeper layer.
Long-term Correlation. represent the motion across sequence of T frames, we first repeat the correlation operator in video frames by computing the correlation volume for every pair of adjacent frames the input sequence, obtain video-level correlation volume = [S] ..., Sr].
natural to consider extension to bi-directional correlation, which can be obtained by computing an additional set of displacements in the opposite direction. We stack the forward correlation between frames t and t + 1 and the backward correlation between frames t and t 1. To capture long- range dependencies, we construct long-term correlation across 2T. neighhorhood frames (T. forward frames and backward frames). We enlarge the matching region the temporal step increases dilation factor. In this long-range bi-directional case, we combine multiple displacement fields into single correlation volume by concatnation.
S {C(fi. fi-L), C(fi, fi-1), C(fe, f(41), C(f..
4

---

Under review as a conference paper at ICLR 2023
Video teature
Spatial Pynarid
f
Gonte
Gintr
frue
hame
4-2
fe
fuz
(a) Correlation Operator
(b) Long-term Correlation
(c) Block
Figure Dense Correlation Fields detail. (a). The correlation is formed by taking the inner product of feature vectors between patches from different frames. (b). In order to explore the long- term temporal dependency, we construct correlation volume between bidirectional and consecutive frames. (c). The feature of the current stage first undergoes 1 1xIx1 layer to reduce channel dimensions and then compute a correlation volume. The correlation volume current stage is combined with correlation volumes from the top-down pathway. These correlation volumes attached by another 1 Ix x Conv (we omit BN and ReLU for clarity) to produce the correlation feature. The video feature is merged by addition with the correlation feature. The current correlation volume is downsampled spatially and connected to the deeper stage for pyramid correlation.
Thus, the correlation volume S for T frames has dimensions T x H X W X (K X K x 2L) (last dimension can seen as correlation volume channels).
Spatial Correlation Hierarchy Aggregation. We employ the correlation block for multiple stages and obtain the correlation volumes as St several spatial scales with a scaling step of 2'. We further construct spatial correlation pyramid by collecting correlation volume hierarchy. For example, we have pd for the correlation pyramid in the stage 4,
p4 = {S*, Pool(S"), Pool(" (So)
where ()ot*() denote pooling at the spatial dimensions with kernal 2*. The pooled correlation volume Poot* has dimensions I' x H/2* x W/2k x (K x K x 2L).
We use a top-down pathway combine correlation volume from lower resolution with spatially stronger correlation volume from higher resolution. We wrap our DCF into residual connection block that can be incorporated into many existing architectures. The residual connection allows us to insert our DCF block into any pre-trained model, without breaking its initial behavior. Figure 3 (c) shows the building block that constructs our DCF. The video feature first undergoes a 1 layer to reduce channel dimensions and then is attached correlation layer. The correlation volume is then merged with the top-down correlation volumes by concatenation the channel dimension. Finally, append another 1 x layer (we omit BN and ReLU for clarity) the stacked correlation volumes to generate the final feature map, which is merged with the input video feature by element- wise addition. To generate spatial correlation hierarchy, pooling) the spatial resolution of the current correlation volume for deeper stage. The top-down pathway starts at early stage without correlation volume input. This process is iterated until the final dense correlation fields are generated.
simply downsample (using average
3.3
DCF NETWORKS
Our DCF presents a new motion modeling mechanism learn action explicitly in action recognition tasks. DCF is designed as a plug-in substitution for video models. We describe our baseline network architecture for this task, and then wrap them into our proposed DCF networks. In practice, the DCF only adds small overhead the computational cost of the backbone network.
We first use the R(2+1)D (Tran et al. 2018) with ResNet (He et al. 2016) pretrained on ImageNet (Deng et al. 2009) as our backbone model to instantiate our DCF networks. Following the practice in previous temporal module methods (Wang et al. 2021 Yang et al. 2020al Wang et al. 2020). we use the output features of stage2, stage3, stage4 to build DCF, where they spatially downsampled by respectively 4, 8 and 16 times, compared to the input frames. We omit stageō as its spatial resolution is too low. Compared with the original R(2+1)D, make few changes to further
5

---

Under review as a conference paper at ICLR 2023
simplify and improve its efficiency, e.g., using depthwise temporal ID-Conv, removing temporal downsampling. Alternatively, we also construct DCF with 3D CNNs based on the recent X3D model(Feichtenhoter 2020). For the X3D backbone, we build DCF on the collection of feature maps from the stages with the same resolutions as in R(2+1)D model. For the structure detail of the I(D+I)D ResNet and X3D, please refer Appendix.
The computation of DCF block is lightweight when is used in sub-sampled feature maps rather than high-resolution image. We set the channels of the first 1x x layer be half of the input channels in X3D and a quarter in R(2+1)D. Then, we perform the correlation operation with spatial region size K and temporal range L 2 layer is used to restore the original channels. We add BatchNorm (BN) (loffe & Szegedy (0135 and ReLU (Nair& & Hinton 2010) layer after the second x x layers.
for the last stage). After that, another 1x k
EXPERIMENTS ON ACTION RECOGNITION
In this section, we present the experiment results of our DCF on three large-scale public datasets. To understand the behavior of DCF, we first perform comprehensive studies the challenging temporal-related dataset Something- Soaetiingn V1 (Goyal et al.) 2017). Ablation studies present consistent improvements, which show the effectiveness and generality of our DCF, We also report results on the Something Sooiiining V2 (Goyal et al. (017) and Kinetics dataset (Kay et al. 2017) to show the generality of our method.
4.1 DATASET AND IMPLEMENTATION DETAILS
Datasets. SamnethingSSomethingg (Goyal
2017) is a large-scale dataset created by crowdsoure- ing. The videos are collected by performing the same action with different objects so that action recognition is expected to focus on the motion property instead of appearance. The first version consists of 86k training videos and 1 11 validation videos belonging to 174 action categories, whose durations vary from to 6 seconds. The second release (V2) of Sameethig-SSometiee
increases the number of videos to 220k. Kinetics-400 (Kay et al. 2017) is among the most popular datasets for video classification. It contains around 240k training You Tubeube videos and 19k validation videos that last for 10 seconds. It includes 400 action categories in total. The Kinetics dataset contains activities in daily life and some categories are highly correlated with interacting objects or scene context.
Training and evaluation. In R(2+1)D backbone experiments, we use ResNet50 and ResNet101 initialized using ImageNet (Deng et al. 2009) pre-trained weights (He et al. 2016) to implement DCF networks. We train the R(2+ 1)D model for 60 epochs Something-Sometting dataset and 100 epochs for Kinetis-400 dataset. As for X3D backbone experiments, we use X3D-S and X3D-M to train our DCF networks from scratch. We train the X3D model for 128 epochs on Something-Sometthngg Something dataset and 256 epochs for Kinetis-400 dataset. Following common practice (Feichtenhofer al., 2019 Wang et al. 202I). during training, each video frame is resized to have shorter side in [256, 320) and crop of 224 x 224 is randomly cropped. Temporally, we perform interval based sampling for Kinetics-400, with interval of 8 for frames, interval of for 13 frames and interval of 5 for 16 frames. On sampling with segments of or 16. We implement two kinds of testing scheme. When we consider the efficiency. 1-clip per video and the center crop of 224 x 224 is used for evaluation on SSVI and SSV2 dataset. When pursuing high accuracy, we followed the common setting in (Wang et al. 2018b) to sample multiple clips per video and use three spatial crops of 256 x 256 Kinetics-400 dataset. We include further training details in the appendix.
dataset, we perform segment based
4.2 ABALATION STUDY
This section provides comprehensive ablation studies to verify the effectiveness of the proposed DCF in terms of motion modeling. Something Something V1 dataset is used as it is widely acknowledged to focus on motion modeling. For these evaluations, we use the testing scheme of clip and center crop, and report the Topl accuracy. We first explore the effect of different components and the locations for DCF, using X3D-S as backbone network. Then we investigate the generality of DCF on different backbones.

---

Under review as a conference paper at ICLR 2023
Basic Corr
Long Term
Spatial Pyramid
Stage
Topl
A
FLOPs
Topl
1
2
4
44.6
2.0G
44.6
46.1
+1.5
2.4G
46.5
46.6
+2.0
/
/
2.1G
46.8
47.3
+2.7
2.5G
47.2
47.6
+3.0
2.2G
47.6
Table 1: Contributions of the proposed compo- nents in DCF, including basic correlation oper- ator, long-term temporal correlation and spatial pyramid aggregation.
Table Different stage groups employ DCF. The results imply that employing blocks at stages 2-4 obtains the best recognition accuracy and the computational cost is also reasonable.
Backbone
DCF
Frames
Params
FLOPs
Topl
A
X
13
3.3M
2.0G
44.6
X3D-S
/
13
3.4M
2.2G
47.6
+3.0
X
16
3.3M
4.7G
47.3
X3D-M
/
16
3.4M
5.2G
49.5
+2.2
23.9M
32.7G
46.8
R(2+1D) R50
8
24.6M
35.9G
50.5
+3.7
Table 3: Different backbones employing DCF.
Different components of DCF. We study the effect of the individual component of DCF and the results are shown Table [ particular, we measure the effect of: using basic correlation op- erator; long-term temporal correlation; spatial pyramid aggregation. We add one correlation block to stages of stage2, stage3, stage4 respectively. First of all, we can conclude that the correlation operator is beneficial for action recognition, provides the network with explicit motion informa- tion. Second, we find that long-term temporal correlation is also helpful, as it enhances the original representation with long-tem motion information. Finally, note that spatial pyramid correlation significantly boosts the performance by ~ 1,% which confirms the importance of associating low- level correlation across resolutions and semantic levels for motion modeling. It hallucinates lower resolution features by downsampling spatially finer, but semantically weaker, correlation feature maps from higher resolution features, In the rest of this paper, default.
the full version of DCF by
Different stages employing DCF. We perform the ablation study on which stage use the corre- lation block. The results are shown in Table First, we see that adding more correlation blocks into the backbone network will lead better results. More blocks increase the computational cost slightly. The setting of using blocks in stages 2-4 obtains the best recognition accuracy and the com- putational cost is also reasonable. Noting that DCF on stage3 contributes more to the performance, as this stage provides feature maps of high-level semantics well more accurately localized in- formation. the rest of this paper, we use three blocks (1 to stage2, to stage3, to stage4) to build DCF by default.
DCF with diffferent backbones. We compare the proposed DCF with other backbone architectures in Table 3 DCF improves the recognition performance with small computation overhead on dif- ferent backbones, including X3D (Feichtenhofer, 2020) and R(2+1)D (Tran et al., 2018) by 3.0% (2.2% for X3D-M) and 3.7% respectively is not just because they add depth to the baseline model. To see this, we note that in Table 3 the DCF with X3D-S model has 47.6% accuracy, higher than the deeper X3D-M baseline's 47.3%. This comparison shows that the improvement due to DCF is complementary to going deeper in standard ways.
SSVI. Is noteworthy that the improvement of DCF
4.3 COMPARISON WITH THE STATE OF THE ART
After the ablation study of DCF setting to the datasets of Someetinng V2 and Kinetics-400. Compared to Kinetics-400, Something-Somethinng
VI dataset, we directly transfer its optimal
V1&V2 require more temporal modeling ability than spatial appearance. In this subsection, we compare our DCF with other of-te-o-thhe-e methods on these benchmarks. As expected, sampling more frames further improve the accuracy, but also increase the FLOPS.

---

Under review as a conference paper at ICLR 2023
Sth-Sth V1
Method
Backbone
Pretrain
Frames
GFLOPs
Top1
Top5
TSN
BNInception
ImageNet
16
19.5
MiN--uutiecale
BNInception
ImageNet
33
34.4
1(2+1)D*
ResNet50
ImageNet
8
33
46.8
74.7
TSM
ResNet50
ImageNet
8+16
98
49.7
78.5
TANet
ResNet50
ImageNet
8+16
99
50.6
79.3
SmallBigNet
ResNet50
ImageNet
8+16
157
50.4
80.5
CorrNet
ResNet101
32
224x30
51.7
Yang el al.
ResNet101
32
150x30
52.8
TDN
ResNet101
ImageNet
16+64
132
55.3
83.3
X3D
X3D-S
13
2.0
44.6
74.4
X3D*
X3D-M
4.7
47.3
76.6
DCF (Ours)
X3D-S
13
2.2
47.6
76.3
DCF (Ours)
X3D-M
16
5.1
49.5
78.5
DCF (Ours)
ResNet50
ImageNet
35.9
50.5
79.4
DCF (Ours)
ResNet50
ImageNet
16
71.9
54.1
82.0
DCF (Ours)
ResNet101
ImageNet
16
131.3
55.8
84.3
Sth-Sth V2
Method
Backbone
Pretrain
Frames
GFLOPs
Top1
Top5
TSM
ResNet50
ImageNet
16
86x6
63.4
88.8
SmallBigNet
ResNet50
ImageNet
8+16
157
63.3
88.8
TANet
ResNet50
ImageNet
16
86x6
64.6
89.5
TAda2D
ResNet50
ImageNet
16
86x6
65.6
89.2
TDN
ResNet101
ImageNet
16+64
132
66.9
90.9
TimeSformer-HE
ViT-B
ImageNet
16
1703x3
62.2
MVTT
VīT-B
Kinetics400
16
70.5
64.7
89.2
X3D*
X3D-M
16
4.7
59.3
86.1
DCF (Ours)
X3D-M
16
5.1
63.4
88.7
DCF (Ours)
ResNet50
ImageNet
71.9
65.5
90.1
DCF (Ours)
ResNet101
ImageNet
16
131.3
67.5
91.1
Table 4: Comparison with the state-of-the-art methods over action recognition on Something- Something V1 and V2 validation set. The results of R(2+1)D ResNet and X3D baselines are trained with the same training protocols for a fair comparison, which are marked with *
Method
Backbone
Pretrain ImageNet
Frame
GFLOPs
Top1
Top5
TSN
Inception V3
25
3.2x250
72.5
90.2
TSM
ResNet50
ImageNet
16
86x30
74.7
91.4
CorrNet
ResNet50
32
115x10 10
77.2
TDN
ResNet50
ImageNet
16+64
94x30
77.5
93,2
TAda2D
ResNet50
ImageNet
16
86x30
77.4
93.1
X3D
X3D-M
16
6,2x30
76.0
92.3
SlowFast
ResNet50
8+32
65.7x30
77.0
92.6
MVIT
ViT-B
16
70.5x5
78.4
93.5
DCF (Ours)
X3D-M
16
6.8x30
77.2
92.7
DCF (Ours)
ResNet50
ImageNet
16
93.9x30
77.4
93.1
Table Comparison with the state-of-the-arr methods over action recognition Kinetics-400 val-
idation set. We instantiate our DCF with the backbones of R(2+1)D ResNet and X3D for evaluation.
For fair comparison with previous methods, we use clip and center crop testing scheme on the Something-Somethhngg dataset and 10 clips and crops for testing the Kinetics- 400 dataset.
Results on Saanething-Somethiine We first validate the efficiency and effectiveness the pro- V1&V2. Table shows the results and computation bud- gets (i.e., number GFLOPs) other methods: temporal modeling based on 2D CNN methods
posed DCF on Something-
TSM(Lin et al.] 2019). LuN & Yuille 2019). CorrNett Wang al. 2020), (Yang et al. 2020b) IDN(Wang et al. 2021). TAda2D(Huang et al. 202T)), 3D CNN architectures [arrCrrre &
Zisserman 2017), Sma al. 2020a and X3D(Feichtenhofer 2020)) and Transformer
8

---

Under review as a conference paper at ICLR 2023
Stage?
Stage3
direction
Staged
Figure 4: An example of the correlation pyramid on the SSV1 dataset. We provide color wheel for correlation direction tendency on the right. The tendency of the correlation feature is consistent with the correct motion pattern. Note that the correlation from the early stage preserves fine details. Best viewed in color.
et al., 2021 and M et al., 2021). Under the same input frames and the same backbone of ResNet50, DCF consistently outperforms other methods with a comparable computation budget on both Something V1&V2. When compared to 2D CNN with temporal modules for all stages, our DCF consistently outperforms them both datasets, demonstrating the effectiveness of DCF motion modeling for action recognition. When com- pared to recent 3D CNNs and Transformer methods, DCF can still obtain slightly better performance than those methods. CorrNet (Wang et al. (020) shares a similar with ours, namely the construction of correlation operator establish frame-to- frame matches over convolutional fea- ture maps to capture motion information. In particular, our DCF achieves better performance than CorreNet even with fewer input frames SSVI. This result demonstrates that the proposed dense correlation fields make it better to model coarse-to-fine motion information.
methods HommeSSfrmer-HRRBereeee
Results on Kinetics-400. We also compare DCF to other state-of-the-art methods on Kinetics-400. DCF with R(2+1)D achieves 77.4% Top-1 accuracy, and it shows better performance than the state- of-the-art temporal modeling methods. DCF with X3D also surpasses 3D CNNs methods, including Slowfast(Feichtenhooerr et al. 2019) and X3D(Feeictennhofer 2020). Our best result is on par with the previous best performance on the Kinetics dataset.
4.4 VISUALIZATION OF CORRELATION FEATURE
In this section, we visualize the correlation fields to better understand our method. Visualization results are shown Figure(5] A color wheel on the right indicates the correlation direction tendency. Note that the correlation is incomplete at image boundary because of the image padding before the correlation operator. The correlation exhibits reliable motion patterns for the correct actions. For example, the key moves right (correlation color is red -ed-ppppee) while the ruler moves left (correlation color is green-blue). The correlation pyramid shows that lower stage can preserve considerably more details and more localization information as it is downsampled fewer times. It motivates to combine resolutin,,,,, semantically strong correlations with high-resolution, semantically weak correlations.
5 CONCLUSION
This work proposes Dense Correlation Fields (DCF) for explicitly model motion information, which computes frame-to-frame similarity and builds up temporal-spatial correlation fields with coarse-to- fine strategy. DCF is unique in that it operates
spatial correlation hierarchy that associates motion feature across resolutions and semantic levels. DCF is complementary to increasing the model capacity for motion modeling stand-alone and plug-in in module. Our method outperforms the otate-of-the-r art temporal modeling 2D CNNs methods as well as 3D CNN methods terms of motion modeling. We hope this work facilitate further research in motion modeling.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Anurag Arna,, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer, 2021.
Sadjad Asghari-Esfeden Mario Sznaier, and Octavia Camps. Dynamic motion representation for human action recognition. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 557-566, 2020.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. space-time attention all you need for video understanding? In International Conference on Machine Learning, pp. 813-824. PMLR, 2021.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp- 6308, 2017.
Vasileios Choutas, Philippe Weinzaepfel, Jérôme Revaud, and Cordelia Schmid. Potion: Pose mo- tion representation for action recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7024-7033, 2018.
Nieves Crasto, Philippe Weinzaepfel, Karteek Alahari, and Cordelia Schmid. Mars: Motion- augmented rgb stream for action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7882-7891, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Ali Diba, Vivek Sharma, Lue Van Gool, and Rainer Stiefelhagen. Dynamonet: Dynamic action and motion network. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 6192-6201, 2019.
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In ICCV, 2021.
Lijie Fan, Wenbing Huang, Chuang Gan, Stefano Ermon, Boqing Gong, and Junzhou Huang. End- to-end learning of motion representation for video understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6016-6025, 2018.
Christoph Feichtenhofer. X3I): Expanding architectures for efficient video recognition. In CVPR, pp. 203-213, 2020.
2020.
Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. Convolutional two-stream network fusion for video action recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1933-1941, 2016.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pp- 2019. 2019.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag et al. The "Something Something" video database for learning and evaluating visual common sense. In ICCV, pp. 5842-5850, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdi- nov. Improving neural networks by preventing co-adaptation feature detectors. arXiv preprint arXiv:/207.0888 05800, 2012.
10

---

Under review as a conference paper at ICLR 2023
Ziyuan Huang, Shiwei Zhang, Liang Pan, Zhiwu Qing, Minggian Tang, Ziwei Liu, and Marcelo H Ang Jr. Tada! smpoorlly-adaptivy convolutions for video understanding. arXiv preprint arXiv:2110.06176, 2021.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 3d convolutional neural networks for human action recognition. JEEE transactions on pattern analysis and machine intelligence, 35(1):221-23)) 2012.
Boyuan Jiang, MengMeng Wang, Weihao Gan, Wei Wu, and Junjie Yan. STM: Spatiotemporal and motion encoding for action recognition. In ICCV, pp. 2000 2000-2009, 2019.
Jianwen Jiang, Yu Cao, Lin Song, Shiwei Zhang, Yunkai Li, Ziyao Xu, Qian Wu, Chuang Gan, Chi Zhang. and Gang Yu. Human centric spatio-temporal action localization. In ActivityNet Workshop on CVPR, 2018.
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei- Fei. Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Puttern Recognition, pp. 1725-1732, 2014.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06955, 2017.
Heeseung Kwon, Manjin Kim, Suha Kwak, and Minsu Cho. Motionsqueeze: Neural motion feature learning for video understanding. In European conference on computer vision, pp. 345-362. Springer, 2020.
Ivan Laptev, Marcin Marszalek, Cordelia Schmid, and Benjamin Rozenfeld. Learning realistic hu- man actions from movies. 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-8. IEEE, 2008.
Kunchang Li, Xianhang Li, Yali Wang, Jun Wang. and Yu Qiao. {CT)-net: Channel tensorization network for video classification. In International Conference on Learning Representations, 2021.
Xianhang Li, Yali Wang, Zhipeng Zhou, and Yu Qiao. Smallbignet: Integrating core and contextual views for video classification. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1092-1101, 2020a.
Yan Li, Bin Ji, Xintian Shi, Jianguo Zhang. Bin Kang, and Limin Wang. TEA: Temporal excitation and aggregation for action recognition. In CVPR, pp. 909-918, 2020b.
Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 7083-7093, 2019.
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. transformer. arXiv preprint arXiv:2106.1322 2021a.
Video swin
Zhaoyang Liu, Limin Wang, Wayne Wu, Chen Qian, and Tong Lu. TAM: Temporal adaptive module for video recognition. In ICCV, pp. 13708-13718, 2021b.
Chenxu Luo and Alan Yuille. Grouped spatial-temporal aggretation for efficient action recognition. In Proceedings of the IEEE International Conference on Computer Vision, 2019.
Yue Meng, Rameswar Panda, Chung-Ching Lin, Prasanna Sattigeri, Leonid Karlinsky, Kate Saenko, Aude Oliva, and Rogerio Feris. Adafuse: Adaptive temporal fusion network for efficient action recognition. 2021.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In leml, 2010.
11

---

Under review as a conference paper at ICLR 2023
Daniel Neimark, Omri Bar, Maya Zohar, and Dotan Asselmann. Video transformer network. In ICCV, pp. 3163-3172, 2021.
Joe Yue-Hei Ng. Jonghyun Choi, Jan Neumann, and Larry S Davis. Actionflownet: Learning mo- tion representation for action recognition. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1616-1624. IEEE, 2018.
AJ Piergiovanni and Michael S Ryoo. Representation flow for action recognition. In Proceedings of the IEEE/CVE conference on computer vision and pallem recognilion, pp. 9945-9953, 2019.
Zhaofan Qiu, Ting Yao, and Tao Mei. Learning spatio-temporal representation with pseudo-3d residual networks. In proceedings of the IEEE International Conference on Computer Vision, pp. 5533-5541, 2017.
Zhaofan Qiu, Ting Yao, Chong-Wai Ngo. Xinmei Tian, and Tao Mei. Learning spatio-temporal representation with local and global diffiusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12056-12065, 2019.
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. Advances in neural information processing systems, 27, 2014.
Shuyang Sun, Zhanghui Kuang, Sheng, Wanli Ouyang, and Wei Zhang. Optical flow guided feature: A fast and robust motion representation for video action recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1390-1399, 2018.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3D convolutional networks. In ICCV, pp. 4489-44977 2015a.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 4489-4497, 2015..
Du Tran, Heng Wang, Lorenzo Torresani. Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 6450-6459, 2018.
Heng Wang, Du Tran, Lorenzo Torresani, and Matt Feiszli Video modeling with correlation net- works. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 352-361, 2020.
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In European conference on computer vision, pp. 20-36. Springer, 2016.
Limin Wang, Wei Li, Wen Li, and Lue Van Gool. appearance-and-reladn networks for video classification. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1430-1439, 201 8a.
Limin Wang. Zhan Tong, Bin Ji, and Gangshan Wu. Tdn: Temporal difference networks for efficient action recognition. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1895-1904, June 2021.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference computer vision and pultern recognition, pp. 7794-7803, 2018b.
Chao-Yua Wu, Manzil Zaheer, Hexiang Hu, R Manmatha, Alexander J Smola, and Philipp Krähenbühl. Compressed video action recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6026-6035 2018
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotem- poral feature learning: -ecuraac trade-offs in video classification. Proceedings of the European conference computer vision (ECCV), pp. 305-321, 2018
12

---

Under review as a conference paper at ICLR 2023
Ceyuan Yang, Yinghao Xu, Jianping Shi, Bo Dai, and Bolei Zhou. Temporal pyramid network for action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 591-600, 2020a.
Xitong Yang, Xiaodong Yang. Sifei Liu, Deqing Sun, Larry Davis, and Jan Kautz. Hierarchical trastive motion learning for video action recognition. arXiv preprini arXiv:2007. 2020b.
Yue Zhao, Yuanjun Xiong, and Dahua Lin. Recognize actions by disentangling components of dynamies. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6566-6575, 2018.
Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In Proceedings of the European conference on computer vision (ECCV), pp. 803-818, 2018.
13

---

Under review as a conference paper at ICLR 2023
APPENDIX
A
MODEL STRUCTURE
Stage
R(2+1)D-50
Output size
data
T'x224222
1x7x7, 64, stride 1,2,2
stagel
Tx56x56
pool, 1x3x3, stride 1,2,2
x x 1,64
3 1 1, 64
stage2
x 3
Tx56x56
1 x 3, 64
1 x 256
256
Tx 128
3 1,8
stage3
Tx28x28
1 x 3, 128
1x1x 1,2
1x1x 1, 1,256
3 x x 256
stage4
X
Tx14x14
1 256
x 1 1, 1024
1, 512
3 X 1 1, 512
stage5
X 3
Tx7x7
1 x 3,212
x 1 X 1, 2048
global average pool, fc
# classes
Table 6: The R(2+1)D ResNet-50 backbone for building DCF networks.
Stage
X3D-S
Output size
data
Tx 160x 160
stagel
1x3x3, 3x1x1, 24, stride 1,2,2
8x80x80
X 5,7
stage2
3 3 3, 5,
x
4x40x40
1 1,24
1 1, 1087
stage3
3 X 3, 108
x
Tx20x20
1 1 1,
x 1, 2167
stage4
3 3 x 3, 216
11
Tx 1x10x10
1x 9,
4327
stage5
3 x 3 3, 432
x
Tx5x5
1x 1 1, 192
global average pool, fc
# classes
Table 7: The X3D-S backbone for building DCF networks.
Here we introduce the implementation of DCF for action recognition. We use the R(2+1)D(Tram al. 2018) as 2D backbone network, and X3D(FFichtenhofer 2020) as the 3D backbone network. Compared with the original R(2+1)D, we make few changes to further simplify and improve its efficiency, e.g., using depthwise temporal ID-Conv, removing temporal downsampling. We provide the structure of R(2+1)D ResNet-50 in Table and X3D in Table 7 for the reference. The dimen- sions of 3D output maps and filter kernels are in TxHx W. with the number of channels following. Residual blocks are shown in brackets. We use the output features of stage2, stage3, stage4 to build DCF, where they are spatially downsampled by respectively 4, and 16 times, compared to the input frames. We omit slage5 as its spatial resolution is too low.
14

---

Under review as a conference paper at ICLR 2023
Algorithm The DCF block
Parameters:
Pyramid output level temporal range 7, patch size p.
Input: Video feature F,, correlation volumes from the top-down pathway Cin.
I Convl(F);
2: C' corr_op(x, T. p);
3: if C.n not None then
4: corr.pyramid concat(C, Cin);
5: else
6:
corr-pyramid C:
7: end if
8: C" BN2 pyramid));
9: F' = CC::
F" ReLU(F)):
11: Cout = List(:
for 1 to do
13: C = Avg-pool(C, kernel=2, stride=2); % in spatial dimensions;
15: end for Return F', Cout
Output: Video feature F" enhanced by DCF, pyramid correlation volumes Cout.
B DCF BLOCK INSTANTIATION DETAILS
We wrap the DCF into a plugin module that can be incorporated into many existing architectures. DCF is based on several correlation blocks. The design of correlation block is similar to Residual block (He et al.) 2016). The current feature first undergoes 1x1x layer to reduce channel di- mensions. And then we compute correlation volume C' across frames. The correlation volume of current stage C combined with correlation volumes C.n. from the top-down pathway by concate- nation. These correlation volumes attached by 1x1x1 layer, a BN layer and ReLU layer to produce the correlation feature C². The video feature is merged by addition with the correlation feature. The current correlation volume is downsampled spatially and connected to the deeper stage for a pyramid correlation. The details presented in Algorithm [
C COMPITATIONAL ANALYSIS
The correlation operator intentionally designed to capture matching information between consec- utive frames. The correlation operator uses dot-product visual similarity and does not include learnable parameters. Consider the input feature with the shape of x H x W x Cin, where Cin denotes the number of input channels. The computation is:
FLOPs(Cor) = T' x x W Oi (K2 x 2L)
(4)
where K indicates the spatial region and L indicates the temporal range. In contrast to the 3D convolutions for spatiotemporal modelling, the computation is:
FL.OPs(DD Conv) = x
x Cin x Cont x (K, x K₂ Ky)
(5)
where K,, K Ky indicate the filter sizes for temporal, height and width respectively. For the FLOPs, the computation of the correlation operator is at least an order magnitude smaller than the standard 3D convolution term. A correlation layer constructs 4D correlation volume with shape x H x W x (K2 x 2T. for each level feature.
D IMPLEMENTATION DETAILS
Here, we further describe the implementation details for action recognition. For a fair comparison, keep all the training strategies the same for our baseline.
15

---

Under review as a conference paper at ICLR 2023
Our experiments action recognition conducted three large-scale datasets. In the training scheme, we train with synchronized SGD using 16 GPUs for all action recognition models. Fol- lowing common practice (Feichtenhoter et al. 2019 Wang et al. 202I). during training, each video frame is resized to have shorter side 2256, 320 and a crop of 224 x 224 is randomly cropped. For all models, Temporally. we perform interval based sampling for Kinetics-400, with interval of 8 for 8 frames, interval of 6 for 13 frames and interval of 5 for 16 frames. On Something-Somethinn perform segment based sampling with segments of 8 or 16. We implement two kinds of testing scheme. When we consider the efficiency, 1-clip per video and the center crop of 224 x 224 is used for evaluation on SSVI and SSV2 dataset. When pursuing high accuracy. we followed the com- mon setting in (Wang et al. 2018b) to sample multiple clips per video and use three spatial crops of 256 x 256 on Kinetics-4000 dataset. Note that X3D-S has different resolution for training and testing.
dropout ratio (Hinton et al. 2012) of 0.5 before the classification heads.
dataset, we
In R(2+1)D backbone experiments, we use ResNet50 and ResNet101 initialized using ImageNet (Deng et al. 2009) pre-trained weights (He et al. 2016) to implement our DCF networks. We train the R(2+1)D model for 60 epochs Something Something dataset and 100 epochs on Kinetis-400 dataset. The batch size is 64 and the initial learning rate 0.01 for Something-Sometting dataset. The batch size is 128 and the initial learning rate is 0.02 for Kinetis-400 dataset. We use half-period cosine schedule for decaying the learning rate. The weight decay is set to 5e-4 for Something- Something dataset and le-4 for Kinetis-400 dataset.
As for X3D backbone experiments, we use X3D-S and X3D-M to train our DCF networks from scratch. We train the X3D model for 128 epochs on Something-Somethin on Kinetis-400 dataset. The batch size is 128 and the initial learning rate is 0.4 for Something- Something dataset. The batch size is 256 and the initial learning rate is 0.4 for Kinetis-400 dataset. We use half-perioo cosine schedule for decaying the learning rate. The weight decay is set to 5e-5 for all datasets. For X3D-S models, we use a special resolution of 160 160 for training and testing.
dataset and 256 epochs
16

---

Under review as a conference paper at ICLR 2023
E
MORE VISUALIZATION EXAMPLES
Corr direction
Stage2
Stage3
Stage4
Stage2
Stage3
Stage4
Stage2
Stage3
Stage4
Figure 5: Examples of correlation pyramid the SSV1 dataset. Best viewed in color.
17


Reviews:
[{"review_id": "m922Yc891r", "reviewer": "Reviewer_hBxH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed dense correlation fields for action recognition. Based on CorrNet, the paper proposed to model temporal long-term correlation by computing correlation for frames that are far apart. The paper also designs a spatial hierarchical architecture to aggregate the information at different granularities. The paper presents results on the something-something V1 & V2, and Kinetics-400 dataset. The proposed method shows superior results than the original CorrNet.\n", "summary_of_the_review": "See above.", "strength_and_weaknesses": "The following are more detailed comments and questions regarding the paper:\n1, In Section 3.2, the paper suggests computing bi-directional correlation, L forward frames and L backward frames. This will lead to the correlation being computed multiple times for a given pair of frames. This may lead to a waste of computational cost. How does the paper handle this problem?\n \n2, How to do padding for frames at the boundary of the sampled clip? Since now we are considering L frames, the padding may cause issues for the computation of correlation.\n \n3, Figure 2 and Figure 3 (c) are not very illustrative.  I suggest explaining what exactly each operator is, and what are the sizes of input tensor and output tensor.\n \n4, In section 4.1, how to use R(2+1)D with ResNet pretrained with ImageNet? Since R(2+1)D has separate spatial and temporal filters, how to initialize the filters with ImageNet pretrained ResNet?\n \n5, In section 3.3, the paper suggests that L is set to L=2 and L=1 for the last stage. Will L=2 allow the model to long-term temporal information? It seems pretty short.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SI5qyS1E2d7", "reviewer": "Reviewer_bPjE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on recognizing actions based on the motion information in the videos. The proposed idea (Dense Correlation Fields, DCF) which can be applied to different CNN backbones, computes frame to frame similarity to find spatial-temporal correlation fields. ", "summary_of_the_review": "The paper is pretty easy to follow and approaches the video action recognition from motion modeling point of view (which is not new, a lot of papers have leveraged differently from motion, either using Optical Flow or a different representation of the motion). I like this paper because it proposes a module that helps the network learn from motion at different scales (compared to SOTA which uses most likely a fixed representation of motion). It is a simple idea that works well.\nMy only concern about this method would be its computational time complexity and I would like to see a section on that to see how it performs comparing other approaches (specially since it captures motion at different stages in the network, my guess is it might make it computationally expensive). \nAlso, it would be nice to cite more related papers that took into account the motion modeling and representation, there are a lot, but here are some examples: \n- V. Choutas, P. Weinzaepfel, J. Revaud, and C. Schmid. Potion: Pose motion representation for action recognition. In CVPR 2018, 2018.\n- S. Sun, Z. Kuang, L. Sheng, W. Ouyang, and W. Zhang. Optical flow guided feature: a fast and robust motion represen- tation for video action recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.\n- J. Jiang, Y. Cao, L. Song, S. Z. Y. Li, Z. Xu, Q. Wu, C. Gan,\nC. Zhang, and G. Yu. Human centric spatio-temporal action\nlocalization.\n- L.Fan,W.Huang,S.E.ChuangGan,B.Gong,andJ.Huang.\nEnd-to-end learning of motion representation for video understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6016– 6025, 2018.\n- Asghari-Esfeden, S., Sznaier, M. and Camps, O., 2020. Dynamic motion representation for human action recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 557-566).\n- C.-Y. Wu, M. Zaheer, H. Hu, R. Manmatha, A. J. Smola, andP.Kra ̈henbu ̈hl. Compressedvideoactionrecognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6026–6035, 2018", "strength_and_weaknesses": "- DCF is a simple and novel idea that outperforms SOTA by focusing on motion in videos.\n- Using spatial pyramid correlation feature \n- DCF is stand-alone and can be applied to different backbones. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "AFQLT4K5lSU", "reviewer": "Reviewer_SoQA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to use Dense Correlation Fields which is a pyramid of correlation volume of visual features. The correlation takes into account long-term correlation, and high-level semantically correlation feature. The method shows improvement over 2D CNN and 3D CNN baselines by a healthy margin.", "summary_of_the_review": "Considering the limited novelty and the fact that improvements are based on a somewhat weak baseline, I don't think the paper meets the bar of ICLR.\n\n\n===\n\nSee the latest comment for the final rating.", "strength_and_weaknesses": "### Pros\n1. The idea is simple and the paper describes it in an organized way.\n\n2. Nice visualization on correlation pyramid. It is an interesting finding that higher-level feature (low-resolution) has more semantically stong correlation.\n\n\n### Cons\n1. The idea of using cross-frame correlation is not new. The authors are suggested to discuss more papers which either construct a cost volume [a] or apply multiplicative interactions [b].\n\n    [a] Zhao, et al. Recognize Actions by Disentangling Components of Dynamics. CVPR 2018\n\n    [b] Wang, et al. Appearance-and-Relation Networks for Video Classification. CVPR 2018\n\n2. The spatial correlation hierarchy is realized by \"simply downsample\"-ing. Have you tried other ways such as (1) conducting correlation operation on higher-level (lower spatial resolution) appearance feature map, or (2) a downsample version of the same-level feature map (i.e. downsample -> correlation instead of correlation -> downsample), or (3) conducting correlation on feature map with larger stride?\n\n3. The baselines to compare with are weak and outdated. \n\n    (1) Worse tradeoff than SlowFast. DCF w/ ResNet-50 is higher than SlowFast (R-50) by 0.4% but with 30% more GFLOPs. If the authors show an accuracy-GFLOPs tradeoff curve, I don't think DCF would have a comparable trade-off on par with with SlowFast compared to Slow-only.\n\n    (2) There lacks a comparison to recent Transformer-based video models which remarkably boost the performance in the late few years. Also, I think the correlation operator is conceptually similar to the dot product operation in the self-attention. It makes me wonder if self-attention is a better way to capture such multiplicative relations and the gain of DCF compared to plain 2D/3D ConvNets might be simply diminishing.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6Bh0go2hRCu", "reviewer": "Reviewer_NeMJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new module, termed Dense Correlation Fields (DCF), to model motion information at feature level within neural networks. DCF involves a short-term module (i.e., correlation between adjacent frames), a long-term module (i.e., correlation between bidirectional and consecutive frames) and a spatial pyramid design (i.e. spatially downsampling correlation features for the next network stage). DCF can be applied to different backbone architectures and improves the baseline model on temporal modeling and action recognition. The paper conducts experiments on three common video action recognition datasets and provide ablation and visualization to analyze the impact of the DCF module.", "summary_of_the_review": "The paper proposes a new module, DCF, for modeling motion information at the feature level. However, the technical novelty and contribution of the proposed method is not sufficient, and the experiments in the paper is not convincing enough. Therefore, I'd suggest \"weak reject\" to the paper.", "strength_and_weaknesses": "Weakness\n1. My major concern is the limited technical novelty and contribution of the paper. Most designs in DCF have considerable overlap with existing works: For example, the idea of using correlation at the feature level has been extensively studied in [1,2,3] and the idea of exploiting long-term motion and motion hierarchy has been discussed in [3]. The paper also doesn't discuss and compare with some important recent work on motion modeling, such as [2, 3, 4, 5]. \n\n2. The experiments in the paper is not convincing and the overall performance of DCF is not strong enough. First of all, many related works on motion model [2,3,4,5] are not compared in the paper, and many recent works on action recognition are not included. The visualization of learned motion patterns in Fig. 4 seem less appealing as those in [2, 3] as well.\n\n\n[1] Wang, Heng, et al. \"Video modeling with correlation networks.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n[2] Kwon, Heeseung, et al. \"Motionsqueeze: Neural motion feature learning for video understanding.\" European conference on computer vision. Springer, Cham, 2020.\n[3] Yang, Xitong, et al. \"Hierarchical contrastive motion learning for video action recognition.\" BMVC 2021.\n[4] Piergiovanni, A. J., and Michael S. Ryoo. \"Representation flow for action recognition.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.\n[5] Diba, Ali, et al. \"Dynamonet: Dynamic action and motion network.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject