TAda! Temporally-Adaptive Convolutions for Video Understanding
Authors: Ziyuan Huang
Shiwei Zhang
Liang Pan
Zhiwu Qing
Mingqian Tang
Ziwei Liu
Marcelo H Ang Jr
Venue: ICLR 2022 Poster (2022)
Keywords: Video understanding
Action classification
Dynamic networks

Abstract: Spatial convolutions are widely used in numerous deep video models. It fundamentally assumes spatio-temporal invariance, i.e., using shared weights for every location in different frames. This work presents Temporally-Adaptive Convolutions (TAdaConv) for video understanding, which shows that adaptive weight calibration along the temporal dimension is an efficient way to facilitate modelling complex temporal dynamics in videos. Specifically, TAdaConv empowers the spatial convolutions with temporal modelling abilities by calibrating the convolution weights for each frame according to its local and global temporal context. Compared to previous temporal modelling operations, TAdaConv is more efficient as it operates over the convolution kernels instead of the features, whose dimension is an order of magnitude smaller than the spatial resolutions. Further, the kernel calibration brings an increased model capacity. We construct TAda2D and TAdaConvNeXt networks by replacing the 2D convolutions in ResNet and ConvNeXt with TAdaConv, which leads to at least on par or better performance compared to state-of-the-art approaches on multiple video action recognition and localization benchmarks. We also demonstrate that as a readily plug-in operation with negligible computation overhead, TAdaConv can effectively improve many existing video models with a convincing margin.

Published as a conference paper at ICLR 2022
TADA! TEMPORALLY-ADAPTIE CONVOLUTIONS FOR
VIDEO UNDERSTANDING
Ziyuan Huang', Shiwei Zhang*', Liang Pan', Zhiwu Qing",
Minggian Tang", Ziwei Liu', Marcelo H. Ang Jr
Advanced Robotics Centre, National University of Singapore
DAMO Academy, Alibaba Group $S-Lab, Nanyang Technological University
yuan. huanquu. mus. edda, {zhanggin zsw, mi mingqi tmg, gingehsww..gwweellleee.. www.wealbaabb-iieeeeoo. com, (1 lliang-pan cang-pan, ziwei. ftwei.liufe iuuuennu.ueeee. edu. sg, mpeanghhnus edu..gg
ABSTRACT
Spatial convolutions' are widely used in numerous deep video models. It funda- mentally assumes spatio-temporal invariance, i.e., using shared weights for every location in different frames. This work presents Temporally-Adaptiy Convo- lutions (aAdaCony) for video understanding', which shows that adaptive weight calibration along the temporal dimension is an efficient way to facilitate modelling complex temporal dynamics in videos. Specifically, TAdaConv empowers the spa- tial convolutions with temporal modelling abilities by calibrating the convolution weights for each frame according to its local and global temporal context. Com- pared to previous temporal modelling operations, TAdaConv is more efficient it operates over the convolution kernels instead of the features, whose dimension is order of magnitude smaller than the spatial resolutions. Further, the kernel calibration brings an increased model capacity. We construct TAda2D and TAda- Co networks by replacing the 2D convolutions in ResNet and ConvNeXt with TAdaConv, which leads to at least on par or better performance compared to state-of-the approaches on multiple video action recognition and localization benchmarks. We also demonstrate that as a readily plug-in operation with neg- ligible computation overhead, TAdaConv can effectively improve many existing video models with convincing margin.
INTRODUCTION
Convolutions are an indispensable operation in modern deep vision models (He et al., 2016; Szegedy et al., 2015; Krizhevsky et al., 2012), whose different variants have driven the state-of-the-art perfor- mances of convolutional neural networks (CNNs) in many visual tasks (Xie et al., 2017; Dai et al., 2017; Zhou et al., 2019) and application scenarios (Howard et al., 2017; Yang et al., 2019). In the video paradigm, compared to the 3D convolutions (Tran et al., 2015), the combination of 2D spatial convolutions and ID temporal convolutions are more widely preferred owing to its efficiency (Tran et al,, 2018; Qiu et al., 2017). Nevertheless, 1D temporal convolutions still introduce non-negli computation overhead on top of the spatial convolutions. Therefore, we seek to directly equip the spatial convolutions with temporal modelling abilities.
One essential property of the convolutions is the translation invariance (Ruderman & Bialek, 1994; Simoncelli & Olshausen, 2001), resulted from its local connectivity and shared weights. However, recent works in dynamic filtering have shown that strictly shard weights for all pixels may be sub- optimal for modelling various spatial contents (Zhou et al., 2021; Wu al., 2018).
Given the diverse nature of the temporal dynamics in videos, we hypothesize that the temporal mod- elling could also benefit from relaxed invariance along the temporal dimension. This means that the convolution weights for different time steps are no longer strictly shared. Existing dynamic filter networks can achieve this, but with two drawbacks, First, is difficult for most of them (Zhou al.,
In this work, we use spatial convolutions and 2D convolutions interchangeably. Projecct page: hetps: /ttp///adaa/ddaaaeaeaa/aaaaa lol +2022. thu.. 16/.

---

Published as a conference paper at ICLR 2022
0 Compolation
Convelution with specified weights
Frame descriptoes
Calibnatinn weiglts
11
11
W
(1) Spatial convolution in videos
TAdaConv videos
Figure 1: Comparisons between TAdaConv and the spatial convolutions in video models. (a) Standard spatial convolutions in videos share the kernel weights between different frames. (b) Our TAdaConv adaptively calibrates the kernel weights for each frame by its temporal context.
2021; Yang et al., 2019) to leverage the existing pre-trained models such as ResNet. This is critical in video applications, since training video models from scratch is highly resource demanding (Fe- ichtenhofer et al., 2019; Feichtenhofer, 2020) and prone to over-fitting on small datasets. Second, for most dynamic filters, the weights are generated with respect to its spatial context (Zhou et al., 2021: Jia et al., 2016) or the global descriptor (Chen al., 2020a; Yang al., 2019), which has difficulty in capturing the temporal variations between frames.
In this work, we present the caaporraly-Adaptiii Convolution (TAdaConv) for video understanding, where the convolution kernel weights are no longer fixed across different frames. Specifically, the convolution kernel for the 1-th frame W, is factorized to the multiplication of the base weight and a calibration weight: W, = a₁ W where the calibration weight a is adaptively generated from the input data for all channels in the base weight W.. For each frame, we generate the calibration weight based on the frame descriptors of its adjacent time steps as well as the global descriptor, which effectively encodes the local and global temporal dynamics in videos. The difference between TAdaConv and the spatial convolutions is visualized in Fig. 1.
The main advantages of this factorization are three-fold:
(i) TAdaConv can be easily plugged into any existing models to enhance temporal modelling, and their pre-trained weights can still be ex- ploited: (ii) the temporal modelling ability can be highly improved with the help the temporally- adaptive weight; (iii) in comparison with temporal convolutions that often operate on the learned 2D feature maps, TAdaConv more efficient by directly operating on the convolution kernels.
TAdaConv is proposed as drop-in replacement for the spatial convolutions in existing models. It can both serve as a stand-alone temporal modelling module for 2D networks, or inserted into existing convolutional video models to further enhance the ability to model temporal dynamics. For efficiency, we construct TAda2D) by replacing the spatial convolutions in ResNet (He et al., 2016), which leads to at least on par or better performance than the tate-off-hhe-ars. Used as an enhancement of existing video models, TAdaConv leads to notable improvements on multiple video datasets. The strong performance and the consistent improvements demonstrate that TAdaConv can be an important operation for modelling complex temporal dynamics in videos.
2 RELATED WORK
ConvNets for temporal modelling, A fundamental difference between videos and images lies in the temporal dimension, which makes temporal modeling an important research area for understanding videos. Recent deep CNNs for video understanding can be divided into two types. The first type jointly models spatio-temporal information by 3D convolutions (Carreira & Zisserman, 2017; Tran et al., 2015; Feichtenhofer, 2020; Tran et al., 2019). The second type builds upon 2D networks, where most approaches employ 2D convolutions that share weights among all the frames for spatial modelling, and design additional operations for temporal modelling, such as temporal shift (Lin et al., 2019a), temporal difference (Wang et al., 2021; Jiang et al., 2019), temporal convolution (Tran et al., 2018; Liu et al., 2021b) and correlation operation (Wang et al., 2020), etc. Our work directly empowers the spatial convolutions with temporal modelling abilities, which can be further coupled with other temporal modelling operations for stronger video recognition performances.
2

---

Published as a conference paper at ICLR 2022
Cx7
Globel Modeling
Dynemic paremeters
TAdaConv
Spat Pool
Tomp Pool Cx1
Learnable parameters
CxT
1D conv
cxI
TAdaCouv
CXTXHXW
conv
CrrxT
Cony
xx
AvgP
BN, ReLU
1XKxK
CxCX K?
ExT
BN
BN
3D Cony
X
w
Cxxxx
KxKxK
ReLU
CX
X
IxXX
TxcxxxK*
(a) Other convolutions
(b) Our TAdaConv
(c) Temporal feature aggregation
Figure 2: An instantiation of TAdaConv and the temporal feature aggregation used in TAda2D. (a) Standard convolutions used in video models.
(b) Our TAdaConv using non-linear weight calibra- tions with global temporal context. (c) The temporal feature aggregation scheme used in TAda2D.
Dynamic networks. Dynamic networks refer networks with content-adaptive weights mod- ules, such as dynamic filters/convolutions (Jia et al., 2016; Yang et al., 2019; et al., 202 I), dynamic activations (Li et al., 2020d; Chen al., 2020b), and dynamic routing (Wang al., 2018; Li al., 2020c), etc. Dynamic networks have demonstrated their exceeding network capacity and thus per- formance compared
the static ones. In video understanding, dynamic filter (Liu et al., 2021b) or temporal aggregation (Meng et al., 2021) also demonstrate strong capability of temporal modelling. Recently, some convolutions applies spatially-adaptive weights (Flsayed al., 2020; Chen et al., 2021), showing the benefit of relaxing the spatial invariance in modelling diverse visual contents. Similarly, our proposed TAdaConv enhances temporal modelling by relaxing the invariance along the temporal dimension. TAdaConv has two key differences from previous works: (i) the convolu- tion weight in TAdaConv is factorized into a base weight and a calibration weight, which enables TAdaConv exploit pre-trained weights; (ii) the calibration weights generated according the temporal contexts.
3 TADACONV: TEMPORALLY-ADAPTIVE CONVOLUTIONS
In this work, we seek to empower the spatial convolutions with temporal modelling abilities. In- spired by the calibration process of temporal convolutions (Sec. 3.1), TAdaConv dynamically cali- brates the convolution weights for each frame according to its temporal context (Sec. 3.2).
3.1 REVISITING TEMPORAL CONVOLUTIONS
We first revisit the temporal convolutions to showcase its underlying process and its relation to dynamic filters. We consider desth-wise temporal convolution for simplicity, which is more widely used because of its efficiency (Liu et al., 202 lb; Jiang et al., 2019). Formally, for a 3x1x temporal convolution filter parameterized by B = [B1, , .2 [3]] and placed (ignoring normalizations) after the 2D convolution parameterized by W, the output feature X, of the f-th frame be obtained by: (1)
X, = B1 5(W x,- + B₂ 5(W x,) + B3 5(W x(+1).
where the indicates the element-wise multiplication, * denotes the convolution over the spatial dimension and 8 denotes ReLU activation (Nair & Hinton, 2010). It can be rewritten follows:
N, W,-1 * Xx-1 + W
W Xx+1.
(2)
where W = M . W. w -M.. B2 W and w M -33 W are spatio- dynamic tensor, with its
temporal location adaptive convolution weights. M, E RCxH value dependent on the result of the spatial convolutions (see Appendix A for details). Hence, the temporal convolutions in the (2+1)D convolution essentially performs (i) weight calibration on the spatial convolutions and (ii) feature aggregation between adjacent frames. However, if the temporal modelling achieved by coupling temporal convolutions to spatial convolutions, a non-negligible computation overhead is still introduced (see Table
3

---

Published as a conference paper at ICLR 2022
3.2 FORMULATION OF TADACONV
For efficiency, we set out to directly empower the spatial convolutions with temporal modelling abilities. Inspired by the recent finding that the relaxation of spatial invariance strengthens spatial modelling (Zhou et al., 2021; Elsayed et al., 2020), we hypothesize that temporally adaptive weights can also help temporal modelling. Therefore, the convolution weights in a TAdaConv layer are varied on a frame- by-arem basis. Since we observe that previous dynamic filters can hardly utilize the pretrained weights, we take inspiration from our observation in the temporal convolutions and factorize the weights for the t-th frame We into the multiplication of base weight W shared for all frames, and a calibration weight at that are different for each time step:
X We x = (a: W)) * X
(3)
Calibration weight generation. To allow for the TAdaConv to model temporal dynamics, it is crucial that the calibration weight a for the t-th frame takes into account not only the current frame, but more importantly, its temporal context, i.e., TAdaConv would degenerate a set of unrelated spatial convolutions with different weights applied on different frames. We show an instantiation of the generation function G in Fig. 2(b).
((..,, xt-1, xt+1: Otherwise
In our design, we aim for efficiency and the ability to capture inter-frame temporal dynamics. For efficiency, we operate on the frame description vectors E _ obtained by the global average pooling over the spatial dimension GAP, for each frame, i.e. V. = GAP, (x.). For temporal mod- elling, apply stacked two-layer ID convolutions F with dimension reduction ratio of on the local temporal context vindi = (V--1, Vr, 1 obtained from x, {xx-1, Xx, Xr+1):
F(x:())) = Conv
(4)
where 8 and BN denote the ReLU (Nair & Hinton, 2010) and batchnorm (loffe & Szegedy, 2015).
In order for a larger inter-frame field of view in complement to the local ID convolution, we further incorporate global temporal information by adding a global descriptor g to the weight generation process F through a linear mapping function FC:
F(x; adj .g) = Conv 1 DDC/CCC (B(BN(Conv) CocCCoooi--Ciii CDC-C/1 (v. adj + FCC+0 (g))))
(5)
where g = (GAA. (x) with GAPst st being global average pooling on spatial and temporal dimensions.
Initialization. The TAdaConv is designed be readily inserted into existing models by simply replacing the 2D convolutions. For an effective use of the pre-trained weights, TAdaConv is initial-
ized to behave exactly the same as the standard convolution. This is achieved by zero-initializing the weight of the last convolution in F and adding constant vector to the formulation:
2₂ = Gx) = 1 + F(x
adj .g)
(6)
In this way, at initial state, W, = W.- = Wa, where we load W. with the pre-trained weights.
Calibration dimension. The base weight W, E RCax can be calibrated in different di- mensions. We instantiate the calibration on the Cn dimension (a, E R x x1), as the weight generation based on the input features yields a more precise estimation for the relation of the input channels than the output channels or spatial structures (empirical analysis in Table 7).
Comparison with other dynamic filters. Ta-
Table 1: Comparison with other dynamic filters.
ble I compares TAdaConv with existing dynamic filters. Mixtue-of-expert based dynamic filters such as CondCony dynamically aggregates multi- ple kernels to generate the weights that are shared for all locations. The weights in most other dy- namic filters are completely generated from the input, such as DynamicFilter (Jia et al., 2016) and DDF (Zhou et al., 2021) in images and TAM (Liu et al., 20216) in videos. Compared to image based ones, TAdaConv achieves temporal modelling by generating weights from the local and global tem- poral context. Compared to TANet (Liu et al., 2021b). TAdaConv is better at temopral modellligg because of temporally adaptive weights. Further, TAdaConv cal to the pre- treitra ones, while it is difficult for previous approaches to exploit pre-trained models. More detailed comparisons of dynamic filters are included in Appendix J.
Temporal Location Pretrained modelling adaptive weights
Operations CondCony
X
DynamicFilter
x
DDF
x
TAM
TAdaConv
effectively generate weights identi-
4

---

Published as a conference paper at ICLR 2022
Table 2: Comparison of (2+1)D convolution and TAdaConv in FLOPs and number of parameters.
Example setting for operation: C. = C; = 64, k = 3. T 8, H = W = 56 and r = 4. Example setting for network: ResNet-50 with input resolution 8 x 224 Colored numbers denote the extra FLOPs/Params, added to 2D convolutions or ResNet-50. Refer to Appendix D for model structures.
(2+1)D Cony
TAdaConv
C. x C, b2 x THW
C. C.xxx x THW + C, x (THW + T) -CXx (2
FLOPs
+C. C, x x THW
+ C C. x X 0.9268 (+0.002, 10.2%) 33.02 (+0.02, 10.06%)
E.G. Op
1.2331 (+0.308, 133%)
E.G. Net
37,94 (+4.94, 115%)
Params.
C. x C, X K² C C, X
Co x C. k- +2 X C. X C./rxk 43,008 (+6,144, 117%) 27.5M (43.2M, 11311%
E.G. Op.
49,152 (+12,288, 133%)
E.G. Net
18. (+3.8M, 115.6%)
Comparison with temporal convolutions. Table 2 compares the TAdaConv with R(2+1)D in pa- rameters and FLOOs, which shows most of our additional computation overhead on top of the spatial convolution is an order of magnitude less than the temporal convolution. For detailed computation analysis and comparison with other temporal modelling approaches, please refer to Appendix B.
TADA2D: TEMPORALLY ADAPTIVE 2D NETWORKS
We construct TAda2D networks by replacing the 2D convolutions in ResNet (R2D, see Appendix D) with our proposed TAdaConv. Additionally, based on strided average pooling, we propose a tem- poral feature aggregation module for the 2D networks, corresponding to the second essential step for the temporal convolutions. As illustrated in Fig. 2(c). the aggregation module is placed after TAdaConv. Formally, given the output of TAdaCony X. the aggregated feature can be obtained as follows:
X Xaggr 8(BN, (x) + BN2 (TempArgPool,, (8)));
where TempAvgPool. denotes strided temporal average pooling with kernel size of k. We use different batch normalization parameters for the features extracted by TAdaConv X and aggregated by strided average pooling TempAvvPool,,(3) as their distributions are essentially different. During initialization, we load pre-trained weights to BN1, and initialize the parameters of BN2 to zero. Coupled with the initialization of TAdaConv, the initial state of the TAda2D is exactly the same as the Temporal Segment Networks (Wang al., 2016), while the calibration and the aggregation notably increases the model capacity with training (See Appendix 1). In the experiments, we refer to this structure as the shortcut (Sc.) branch and the separate BN (SepBN.) branch.
EXPERIMENTS ON VIDEO CLASSIFICATION
To show the effectiveness and generality of the proposed approach, we present comprehensive eval- uation of TAdaConv and TAda2D on two video understanding tasks using four large-scale datasets.
Datasets. For video classification, we use Kinetics-400 (Kay al., 2017), Someehing--Something V2 (Goyal et al., 2017), and Epic-Kitchens- 100 (Damen et al., 2020). K400 is widely used action classification dataset with 400 categories covered by 300K videos. SSV2 includes 220K videos with challenging spatio-temporal interactions in 174 classes. EK100 includes 90K segments labelled by 97 verb and 300 noun classes with actions defined by the combination of nouns and verbs. For action localization, we use HACS (Zhao et al., 2019) and Epic- Kitchens- -000
100 (Damen et al., 2020).
Model. In our experiments, we mainly use ResNet (R2D) as our base model, and construct TAda2D by replacing the spatial convolutions with the TAda-structure in Fig. 2(c). Alternatively, we also construct NedtConvNeXt based on the recent ConvNeXt model (I.iu et al., 2022). For TAdaCon- tubelct embedding stem similar to (Arnab et al., 2021) and only use TAdaConv to replace the depth-wiss wise convolutions in the model. More details are included in Appendix D.
vNeXt,
Training and evaluation. During training, 8, 16 or 32 frames are sampled with temporal jittering, following convention (Lin et al., 2019a; Liu et al., 2021b; Feichtenhofer et al., 2019). We include further training details in the appendix C. For evaluation, clips (K400&EK 100), or 2 clips (SSV2) uniformly sampled along the temporal dimension. Each crop has the size of 256x256, which is obtained from video with its shorter side resized to 256.
use three spatial crops with 10
5

---

Published as a conference paper at ICLR 2022
Table 3: Plug-in evaluation of TAdaConv in existing video models on K400 and SSV2 datasets.
Base Model
TAdaConv
Frames
Params.
GFLOPs
K400
SSV2
X
8
32.5M
54.52
74.56
60.31
SlowOnly 8x8*
8
35.6M
54.53
75.85
+1.29
63.30
+2.99
4+32
34.5M
36.10
75.03
56.71
SlowFast 4x 16'
4+32
37.7M
36.11
76.47
+1.44
59.80
+3.09
X
8+32
34.5M
65.71
76.19
61.54
SlowFast 8
8+32
37.7M
65.73
77.43
+1.24
63.88
+2.34
8
28.1M
49.55
73.63
61.06
R(2+1)D*
(20)
8
31.2M
49.57
75.19
+1.56
62.86
+1.80
/ (2de1dd)
8
34.4M
49.58
75.36
+1.73
63.78
+2.72
8
47.0M
84.23
73.83
59.86
R3D
(34)
50.1M
84.24
74.91
+1.08
62.85
+2.99
Notation indicates implementation. See Appendix for details the model structure.
Table 4: Calibration weight generation.
Table 5: Feature aggregation scheme. FA: fea- ture aggregation: Sc: shortcut for convolution feature; SepBN: separate batch norm.
kernel size; Lin./Non-iin.: linear/non-linear weight generation; G: global information g.
Model TAdaCony K.
G.
Top-1
TAdaConv FA. Sc. SepBN. Top-1
A
TSN"
32.0
32.0
Lin.
37.5
59.2
+27.2
Lin.
3
x 56.5
Avg.
X
47.9
+15.9
Non-Lin.
(1, 1)
x
36.8
X
Avg. /
x
49.0
+17.0
Non-Lin.
(3, 1)
x
57.1
Avg. /
57.0
+25.0
Ours
Non-Lin,
(1,3)
57.3
Avg.
60.1
+28.I
Non-Lin.
(3,3)
57.8
Avg.
61.5
+29.5
Lin.
/
53.4
Avg.
/
/
63.8
+31.8
Non (1. 1)
54.4
Max.
63.5
+31.5
Non-Lin. (3,
/
59.2
Mix.
63.7
+31.7
5.1 TADACONY ON EXISTING VIDEO BACKBONES
TAdaConv is designed plug-in substitution for the spatial convolutions in the video models. Hence, we first present plug-in evaluations Table 3. TAdaConv improves the classification per- formance with negligible computation overhead on wide range of video models, including Slow- Fast (Feichtenhofer et al., 2019), R3D (Hara et al., 2018) and R(2+1)D (Tran et al., 2018), by an average of 1.3% and 2.8% respectively on K400 and SSV2 an extra computational cost of less than 0.02 GFlops. Further, not only can TAdaConv improve spatial convolutions, it also notably im- prove 3D and ID convolutions. For fair comparison, all models are trained using the same training strategy. Further plug-in evaluations for action classification is presented in Appendix G.
5.2 ABLATION STUDIES
We present thorough ablation studies for the justification of our design choices and the effectiveness of our TAdaConv in modelling temporal dynamics. SSV2 used the evaluation benchmark, it widely acknowledged to have more complex spatio-temporal interactions.
Dynamic learnable calibration. We first compare different source of calibration weights (with our initial- ization strategy) in Table 6. We compare our calibration with no calibration, calibration using learnable weights, and calibration using dynamic weights generated only from global descriptor (Cx1). Compared with the baseline (TSN (Wang al., 2016)) with no calibration, learnalle calibration with shared weights has limited improvement, while temporally varying learnable cal-
Table 6: Benefit of dynamic calibration. T.V.: temporally varying. *: w/o our init.
Calibration
T.V.
Top-1
Top-
None
32.0
34.3
32.6
Leamable
45.4
43.8
51.2
41.7
Dynamic
53.8
49.8
TAda
59.2
47.8
ibration (different calibration weights for different temporal locations) performs much stronger. A larger improvement is observed when we use dynamic calibration, where temporally varying cali- bration further raises the accuracy. The results also validate our hypothesis that temporal modelling can benefit from temporally adaptive weights. Further, TAdaConv generates calibration weight from both local (CxT) T) and global (Cx 1) contexts and achieves the highest performance.

---

Published as a conference paper at ICLR 2022
67
60
GT Burying sth. In sth.
55
65
1534
Top-1 8813
64
30-
Tull v27.33
63
ISN Pilling uth.up up
45-
40-
61
-- MM PL.20
35
60
RI2+DD Sprending siib, onto sth.
59
0
1/4
1/2
3/4
50
75
100
125
150
175
TAdaCann cnabled
spotial sixe 2567
TAda2D Burving stt. sth.
Figure 3: The classification performance of TAda2D
Figure 4: Grad-CAM visualization and pre- diction comparison between TSN, R(2+1)D
(a) with different channels (C.) and stages (S.) en- abled; (b) in comparison with other tfate-of- and TAda2D (more examples in Fig. A3).
Calibration weight initialization. Next, we show that our initialization strategy for the calibration weight generation plays a critical role for dynamic weight calibration. As in Table 6, randomly ini- tializing learnable weights slightly degrades the performance, while randomly initializing dynamic calibration weights (by randomly initializing the last layer of the weight generation function) no- tably degenerates the performance. It is likely that randomly initialized dynamic calibration weights purturb the pre-trained weights more severely than the learnable weights since it is dependent on the input. Further comparisons on the initialization are shown in the Table A7 in the Appendix.
Calibration weight generation function. Having established that the temporally adaptive dynamic calibration with appropriate initialization can be an ideal strategy for temporal modelling, we further ablate different ways for generating the calibration weight in Table 4. Linear weight generation function (Lin.) applies a single ID convolution generate the calibration weight, while non-linear one (Non-Lin.) uses two stacked ID convolutions with batch normalizations and ReLU activation in between. When no temporal context considered (K.=1 or (1,1)), TAdaConv can still improve the baseline but with a limited gap. Enlarging the kernel size to cover the temporal context (K.=3, (1,3), (3,1) or (3.3)) further yields a boost of over 20% on the accuracy, with K.=(333) having the strongest performance. This shows the importance of the local temporal context during calibration weight generation. Finally, for the scope of temporal context, introducing global context to frame descriptors performs similarly only generating temporally adaptive calibration weights solely on the global context (in Table 6). The combination of the global and temporal context yields a better performance for both variants. We further show in Appendix J that this function in TAdaConv yields a better calibration on the base weight than other existing dynamic filters.
Feature aggregation. The feature aggregation module used in the TAda2D network is ablated in Table 5. First, the performance is similar for plain aggregation x = Avg(x) and aggregation with shortcut (Sc.) branch x x + Avg(x), with Sc. being slightly better. Separating the batchnorm (Eq. 7) for the shortcut and the aggregation branch brings notable improvement. Strided max and mix (avg+max) pooling slightly underperform the average pooling variant. Overall, the combination of TAdaCony and our feature aggregation scheme has an advantage over the TSN bascline of 3..8%.
Calibration dimension. Multiple dimensions can be cali- brated in the base weight. Table 7 shows that calibrating the channel dimension more suitable than the spatial dimension,
Table 7: Calibration dimension.
Cal. dim. Arrmms
AGELON
Top-1
Cin
3.16M
0.016
63.8
which means that the spatial structure of the original convolu- tion kernel should be retained. Within channels, the calibration works better on Cnn than Cout both combined. This is prob-
Cou
3.16M
0.016
63.4
Oi A²
x Cou
4.10M
0.024
63.7
2.24M
0.009
62.7
ably because the calibration weight generated by the input feature can better adapt to itself.
Different stages employing TodaConv. The solid lines in Fig show the stage by stage replace-
ment of the spatial convolutions in a ResNet model. It has a minimum improvement of 17.55%,
when TAdaConv is employed in Res2. Compared to early stages, later stages contribute more to the final performance, as later stages provide more accurate calibration because of its high abstraction level. Overall, TAdaCony is used in all stages for the highest accuracy.
7

---

Published as a conference paper at ICLR 2022
Table Comparison with the top approaches on Samnething-Somethnngg (Goyal al., 2017).
Model
Backbone
Frames clipsxxroppc
GFLOPs
Top-1
Top-5
TDN (Wang ct al., 2021)
ResNet-50
(8f+32/)xIxI
47
64.0
88.8
TDN (Wang et al., 2021)
ResNet-50
(16f+64))XIX
94
65.3
89.5
TDN (Wang al., 2021)
ResNet-50
(8/432/416/4644xXXX
141
67.0
90.3
TSM (Lin et al., 2019a)
ResNet-50
8fx2x3
43
59.1
85.6
TSM (Lin et al., 2019a)
ResNet-50
16fx2x3
86
63.4
88.5
SmallBigNet (1.i ct al., 2020a)
ResNet-50
8fx2x3
57
61.6
87.7
SmallBigNet (1.i ct al., 2020a)
ResNet-50
16fx2x3
114
63.8
88.9
SmallBigNet (Li al., 2020a)
ResNet-50
(8f+16f)x2x3
171
64.5
89.1
TANet (Liu et al., 2021b)
ResNet-50
8fx2x3
43
62.7
88.0
TANet (Liu et al., 2021b)
ResNet-50
16/x2x3
86
64.6
89.5
TANet (Liu et al., 2021b)
ResNet-50
(8/+16/)x2x3
129
64.6
89.5
TAda2D (Ours)
ResNet-50
xfx2x3
43
64.0
88.0
TAda2D (Ours)
ResNet-50
16/x2xx
86
65.6
89.2
T. (Ours)
ResNet-50
(8/+16/)x2x3
129
67.2
89.8
TAdtConvNeXt-T (Ours)
ConvNeXt-T
16 x2x3
47
64.8
88.8
TAdaConvNeXI-T (Ours)
ConvNeett-T
x2x3
94
67.1
90.4
Gray font indicates models with different inputs. FLOPs are caleulated with 256 256 resolution in the evaluation.
Table Comparison with the state-of-the-art approaches over action classification on Epic- Kitchens-100 (Damen al., 2020). indicates our own implementation for fair comparison. indicates the main evaluation metric for the dataset.
Top-1
Top-5
Model
Frames
Act.1
Verb
Noun
Act.1
Verb
Noun
TSN (Wang et al., 2016)
8
33.19
60.18
46.03
55.13
89.59
72.90
TRN (Zhou et al., 2018)
8
35.34
65,88
45.43
56.74
90.42
71.88
TSM (Lin et al., 2019a)
8
38.27
67.86
49.01
60.41
90.98
74.97
SlowFast (Feichtenhofer et al., 2019)
8+32
38.54
65.56
50.02
58.60
90.00
75.62
TSN" (Our baseline)
8
30.15
51.89
45.77
53.00
87.51
72.16
TAda2D (Ours)
8
41.61
65.14
52.39
61.98
90.54
76.45
Different proportion of channels calibrated by TAdaConv. Here, calibrate only proportion of channels using TAdaConv and leave the other chamnels uncalibrated. The results are presented as dotted lines in Fig 3. We find TAdaConv improve the baseline by large margin even if only 1/64 channels are calibrated, with larger proportion yielding further larger improvements.
Visualizations. We qualitatively evaluate approach in comparison with the baseline approaches (TSN and R(2+1)D) by presenting the Grad-CAM (Selvaraju et al., 2017) visualizations of the last stage in Fig. 4. TAda2D can more completely spot the key information in videos, thanks to the temporal reasoning based on global spatial information and the global temporal information.
5.3 MAIN RESULTS
SSV2. As shown in Table 8, TAda2D outperforms previous approaches using the same number of frames. Compared TDN that uses more frames, TAda2D performs competitively. Visualization in Fig. 3(b) also demonstrates the superiority of our performannc/efficiennyy trade-off. An even stronger performance achieved with a similar amount of computation by TAdaConvNeXt, which provides an accuracy of 67.1% % with 94GFL.OPs.
Epic-Kitchens- Table 9 lists our results on EK100 in comparison with the previous ap- proaches". We calculate the final action prediction following the strategies in Huang et al. (2021). For fair comparison, we reimplemented our baseline TSN using the same training and evaluation strategies. TAda2D improves this baseline by 11.46% the action prediction. Over previous ap- proaches, TAda2D achieves a higher accuracy with a notable margin.
Kinetics-400. Comparison with the state-of ofute-of-the-an models on Kinetics-400 is presented in Table 10, where we show TAda2D) performs competitively in comparison with the models using the same
"hhe performances referenced from the official relcase of the EK 100 dataset (Damen et al., 2020).
8

---

Published as a conference paper at ICLR 2022
Table 10: Comparison with the state-of-the-art approaches on Kinetics 400 (Kay al., 2017).
Model
Pretrain
Frames
GFLOPs
Top-1
Top-5
TSM (Lin et al., 2019a)
IN-IK
8x3x10 10
43
74.1
N/A
Set (Li et al., 2020a)
IN-IK
8x3x10 10
57
76.3
92.5
TANet (Liu et al., 2021b)
IN-IK
8x3x10
43
76.3
92.6
TANet (Liu et 2021b)
IN-IK
16x3x10
86
76.9
92.9
SlowFast 4x (Feichtenhofer et al., 2019)
(4+32)x3x10
36.1
75.6
92.1
SlowFast 8x8 (Feichtenhofer et al., 2019)
(8+32)x3x110
65.77
77.0
92.6
TDN (Wang et al., 2021)
IN-1K
(8+332xx3x10
47
76.6
92.8
TDN (Wang et al., 2021)
IN-IK
(16+64)x3x10 10
94
77.5
93.2
CornNet (Wang al., 2020)
IN-IK
32x1x10
115
77.2
N/A
TAda2D (Ours)
IN-IK
8x3x10
43
76.7
92.6
TAda2D (Ours)
IN-1K
16x3x10
86
77.4
93.1
TAda2Dea (Ours)
IN-IK
(8+16)x3x10
129
78.2
93.5
MViT-B (Fan et al., 2021)
x6
70.5
78.4
93.5
MVTT-B (Fan et al., 2021)
32x1 x5
170
80.2
94.4
TimeSformer (Bertasius al., 2021)
IN-21K
8x3x1
196
78.0
93.7
ViViT-L (Amab et 2021)
IN-21K
16x3x4
1446
94.6
Swin-T (Liu et al., 2021a)
IN-IK
32x3x4
88
78.8
93.6
TAdaConnNNeX-TT (Ours)
IN-IK
16x3x4
47
78.4
93.5
TAdaConvNeXt-T (Ours)
IN-IK
32x3x4
94
79.1
93.7
Table 11: Action localization evaluation on HACS and Epic-Kitchens-100 100. t indicates the main
evaluation metric for the dataset, iee. average mAP for action localization.
HACS
Spic-Kittheen-Iee
Model
@0.5
@0.6
@0.77
@0.8 @0.9 Avg.
Task @0.1
@0.2
@0.3
@0.4
@0.5
Avg.t
Verb
15.98
15.01
14.09
12.25
10.01
13:47
TSN
43.6
37.7
31.9
24.6
15.0 28.6
Noun 15.11
14.15
12.78
10.94
8.89
12.37
Act.t 10.24
9.61
8.94
7.96
6.79
8.71
Verb
19.70
18.49
17.41
15.50
12.78
16.78
TAda2D
48.7
42.7
36.2
28.1
17.3
32.3
Noun
20.54
19.32
17.94
15.77
13.39
17.39
Act.t 15.15
14.32
13.59
12.18
10.65
13.18
backbone and the same number of frames. Compared with models with frames, e.g., TDN, TAda2D achieves a similar performance with less frames and computation. When compared to the more recent Transformer-base based models, our TAdaConvNeXt-T provides competitive accuracy with similar or less computation.
EXPERIMENTS ON TEMPORAL ACTION LOCALIZATION
Dataset, pipeline, and evaluation. Action localization is an essential task for understanding untrimmed videos, whose current pipeline makes it heavily dependent the quality of the video rep- resentations. We evaluate our TAda2D on two large-scale action localization datasets, HACS (Zhao al., 2019) and Opic-KKithenss100 (Damen al., 2020). The general pipeline follows (Damen et al., 2020; Qing et al., 2021 1açc). For evaluation, we follow the standard protocol for the respective dataset. We include the details on the training pipeline and the evaluation protocal the Appendix C.
Main results. Table 11 shows that, compared to the baseline, TAda2D provides stronger feature for temporal action localization, with an average improvement over 4% on the average mAP on both datasets. In Appendix H, we further demonstrate the TAdaConv can also improve action localization when used as a plug-in module for existing models.
CONCLUSIONS
This work proposes sagroalllyAdaptivee Convolutions (TAdaConv) for video understanding, which dynamically calibrates the convolution weights for each frame based on its local and global temporal context in video. TAdaConv shows superior temporal modelling abilities on both action classifica- tion and localization tasks, both as stand-alone and plug- in modules for existing models. We hope this work can facilitate further research in video understanding.

---

Published as a conference paper at ICLR 2022
Acknowledgement: This research is supported by the Agency for Science, Technology and Re- search (A*STAR, under its AME Programmatic Funding Scheme (Project #A (AA8A200046), by the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s), and by Alibaba Group through Alibaba Research Intern Program.
REFERENCES
Anurag Arnb, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6836-6846, 2021.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding. arXiv preprint arXi::2102.05005 2(3):4, 2021.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 2299-6308, 2017.
Jin Chen, Xijun Wang, Zichao Guo, Xiangyu Zhang, and Jian Sun. Dynamic region-aware convolu- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8064 8073, 2021.
Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic convolution: Attention convolution kernels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11030-11039, 2020a.
Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic relu. In European Conference Computer Vision, pp. 35 351-367. Springer, 2020b.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEEE international conference computer vision, pp. 764-773 2017.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocentric vision. arXiv preprint arXiv:2006. 13256, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scall scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Gamaleldin Elsayed, Prajit Ramachandran, Jonathon Shlens, and Simon Kornblith. Revisiting spa- tial invariance with low-rank local connectivity. In International Conference on Machine Learn- ing, pp. 2868 2879. PMLR, 2020.
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. Proceedings of the IEEE/CVF Inter- national Conference Computer Vision, pp. 6824-6835, 2021.
Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 203 203-213, 2020.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, pp. 6202-6211, 2019.
10

---

Published as a conference paper at ICLR 2022
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, volume 1, pp- 5, 2017.
Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet? In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 6546-6555, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi- nov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint 1207000007 2012.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: (el:::4044444// 2017.
Ziyuan Huang, Zhiwu Qing, Xiang Wang, Yutong Feng, Shiwei Zhang, Jianwen Jiang, Zhurong Xia, Mingqian Tang, Nong Sang, and Marcelo H Ang Jr. Towards training stronger video vision transformers for epic-kitchens-Iie action recognition. arXiv preprint arXiv: 05.005088 2021.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448 456. PMLR, 2015.
Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc V Gool. Dynamic filter networks. Advances in neural information processing systems, 29:667-675, 2016.
Boyuan Jiang, MengMeng Wang, Weihao Gan, Wei Wu, and Junjie Yan. Stm: Spatiotemporal and motion encoding for action recognition. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2000-2009, 2019.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint 1655000005.06950,, 2017.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. volutional neural networks. Advances in neural information processing systems, 15:1097- 2012.
Imagenet classification with deep con-
Xianhang Li, Yali Wang, Zhipeng Zhou, and Yu Qiao. Smallbignet: Integrating core and contextual views for video classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1092- 1101, 2020a.
Yan Li, Bin Ji, Xintian Shi, Jianguo Zhang, Bin Kang, and Limin Wang. Tea: Temporal excitation and aggregation for action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 918, 2020b.
Yanwei Li, Lin Song, Yukang Chen, Zeming Li, Xiangyu Zhang, Xingang Wang, and Jian Sun. Learning dynamic routing for semantic segmentation. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 8553-8562, 2020c.
Yunsheng Li, Yinpeng Chen, Xiyang Dai. Dongdong Chen, Mengchen Liu, Lu Yuan, Zicheng Liu, Lei Zhang, and Nuno Vasconcelos. Micronet: Towards image recognition with extremely low flops. arXiv preprini arXiv:2011 1. 2020d.
11

---

Published as a conference paper at ICLR 2022
Yunsheng Li, Yinpeng Chen, Xiyang Dai, Dongdong Chen, Ye Yu, Lu Yuan, Zicheng Liu, Mei Chen, Nuno Vasconcelos, et al. Revisiting dynamic convolution via matrix decomposition. In International Conference Learning Representations, 2021.
Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In Proceedings of the IEEE International Conference Computer Vision, pp. 7083-7093, 2019a.
Tianwei Lin, Xiao Liu, Xin Li, Errui Ding, and Shilei Wen. Bmn: Boundary-matching network for temporal action proposal generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3889-3898, 2019b.
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. transformer. arXiv preprint arXivv:2166 13230, 2021a.
Video swin
Zhaoyang Liu, Limin Wang, Wayne Wu, Chen Qian, and Tong Lu. Tam: Temporal adaptive module for video recognition. Proceedings of the IEEE International Conference on Computer Vision, 2021b.
Zhuang Liu, Hanzi Mao, Chao-Yual Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. arXiv preprint arXiv:2201.03545, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
11X10:1711.05101, 2017.
Yue Meng, Rameswar Panda, Chung-Ching Lin, Prasanna Sattigeri, Leonid Karlinsky, Kate Saenko, Aude Oliva, and Rogerio Feris. Adafuse: Adaptive temporal fusion network for efficient action recognition. In International Conference on Learning Representations, 2021.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Icml, 2010.
Zhiwu Qing. Ziyuan Huang, Xiang Wang, Yutong Feng, Shiwei Zhang, Jianwen Jiang. Mingqian Tang, Changxin Gao. Marcelo H Ang Jr. and Nong Sang. A stronger baseline for ego-centric action detection. arXiv preprint w0xiv:2106.06944 2021a.
Zhiwu Qing, Haisheng Su, Weihao Gan, Dongliang Wang, Wei Wu, Xiang Wang, Yu Qiao, Junjic Yan, Changxin Gao, and Nong Sang. Temporal context aggregation network for temporal ac- tion proposal refinement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 485 494, 2021b.
Zhiwu Qing, Xiang Wang, Ziyuan Huang, Yutong Feng, Shiwei Zhang, Mingqian Tang, Changxin Gao, Nong Sang, et al. Exploring stronger feature for temporal action localization. arXiv preprint arXiv:2106. 13014, 2021c.
Zhaofan Qiu, Ting Yao, and Tao Mei. Learning spatio-temporal representation with pseudo-3d residual networks. In proceedings of the IEEE International Conference on Computer Vision, pp. 5533-5541, 2017.
Daniel L Ruderman and William Bialek. Statistics of natural images: Scaling in the woods. Physical review letters, 73(6):814 84,, 1994.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Eero P Simoncelli and Bruno A Olshausen. Natural image statistics and neural representation. Anmual review of neuroscience, 1993:1113-1216, 2001.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015.
12

---

Published as a conference paper at ICLR 2022
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 4489 4497,
Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. Proceedings of the IEEE conference on Computer Vision and Puttern Recognition, pp. 6450-6459, 2018.
Du Tran, Heng Wang. Lorenzo Torresani, and Matt Feiszli. Video classification with channel- separated convolutional networks, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5552-5561, 2019.
Heng Wang, Du Tran, Lorenzo Torresani, and Matt Feiszli. Video modeling with correlation net- works. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 352-361, 2020.
Limin Wang, Yuanjun Xiong, Zhe Wang. Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In European conference on computer vision, pp. 20-36. Springer, 2016.
Limin Wang, Zhan Tong, Bin Ji, and Gangshan Wu. Tdn: Temporal difference networks for efficient action recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1895-1904, 2021.
Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph Gonzalez. Skipnet: Learning dy- namic routing in convolutional networks. In Proceedings of the European Conference on Com- puter Vision (ECCV), pp. 409-424, 2018
Jialin Wu, Dai Li, Yu Yang, Chandrajit Bajaj, and Xiangyang Ji. Dynamic filtering with large sampling field for convnets. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 185-200. 2018.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
Brandon Yang, Gabriel Bender, Quoe V Le, and Jiquan Ngiam. Condconv: Conditionally parame- terized convolutions for efficient inference. arXiv preprint arXiv:1904. 190.:9994. 2019.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceed- ings of the IEEE/CVF international conference on computer vision, pp. 6023-6032, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412. 2017.
Hang Zhao, Antonio Torralba, Lorenzo Torresani, and Zhicheng Yan. Hacs: Human action clips and segments dataset for recognition and temporal localization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6668-8678, 2019.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmen- tation. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 13001- 13008, 2020.
Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 803-818, 2018.
Daquan Zhou, Xiaojie Jin, Qibin Hou, Kaixin Wang, Jianchao Yang, and Jiashi Feng. Neural epit- ome search for ceattecctur-aannostt network compression. arXiv preprint arXiv:1907.05644 056442,,, 2019.
Jingkai Zhou, Varun Jampani, Zhixiong Pi, Qiong Liu, and Ming- Hsuan Yang. Decoupled dynamic filter networks. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattrnn Recognition, pp. 6647-6656, 2021.
Ming-Hsuan
13

---

Published as a conference paper at ICLR 2022
APPENDIX
In the appendix, we provide detailed analysis on the temporal convolutions (Appendix A), compu- tational analysis (Appendix B), further implementation details (Appendix C) on the action classi- fication and localization, model structures that we used for evaluation (Appendix D), per-category improvement analysis
(Appendix E), further plug-in evaluations on Epic- BptchKKs classification (Appendix G) plug-in evaluations on the temporal action localization task (Appendix H), the visualization of the training procedure of TSN and TAda2D (Appendix I), as well as detailed comparisons between TAdaConv and existing dynamic filters (Appendix J). Further, show additional qualitative analysis in Fig. A3.
A DETAILED ANALYSIS ON TEMPORAL CONVOLUTIONS
Here, we provide detailed analysis to showcase the underlying process of temporal modelling by temporal convolutions. As in Sec. 3. we use depth-wise temporal convolutions for simplicity and its wide application. We first analyze the case where temporal convolutions are directly placed after spatial convolutions without non-linear activation in between, before activation functions inserted in the second part of our analysis.
Without activation. We first consider simple case with no non-linear activation functions be- tween the temporal convolution and the spatial convolution. Given 3 x depth-wise temporal convolution parameterized by
[3, Bs], where B1, B2, B3 E RC,, a spatial convolution parameterized by W E CCCxC xx2 tte output feature Xe of the f-th frame can be obtained by:
X, 3 (W x) + B2 (W xx) B3 (W xx+1),
(8)
where- denotes element-wist multiplication with broadcasting, and denotes convolution over the spatial dimension. In this case, B could be grouped with the spatial convolution weight W and the combination of temporal and spatial convolution can be rewritten Eq. in the manuscript:
X = W,-1*X, + W. X2 + W X(+1
where W,- 31 W. W. B2 W and W++1 = = W. This equation share the same form with the Eq. 2 in the manucript. this case, the combination of temporal convolution with spatial convolution can be certainly viewed as the temporal convolution is simply performing calibration on spatial convolutions before aggregation, with different weights assigned to different time steps for the calibration.
With activation. Next, we consider a case where activation is in between the temporal convolution and spatial convolution. The output feature X, now obtained by Eq. in the manuscript:
X = 31 5 5(W x;-1) + B2 5(W x2) + B3 5(W x(+1).
Next, we show that this can be still rewritten the form of Eq. 2. Here, we consider the case where ReLU (Nair & Hinton, 2010) is used as the activation function, denoted as 6:
((x)
O
< 0
Hence, the term 5(W * xz) can be easily expressed as:
8(W * xx) = M, W * Xt,
(10)
where M, E CxH is binary map sharing the same shape xx, indicating whether the corresponding element in W x, is greater than or not. That is:
{
if (W x,) (c3r)) >
M
(o
(11)
if (W x,) (otd)) 50"
where c, j are the location index in the tensor. Hence, Eq. 1 can be expressed as:
x = G1 M W x X,-1 + B2 M, W * x, + By M W * Xx+11
(1)
In this case, we can set W(L)) = B M W. w((-)) = B2 M((.)). W, and ((.))
(+
Bg ((.)) W. where (i,)) indicate the spatial location index. In this case, each filter for a specific ++1
time step t is composed of H x W filters and Eq. I can be rewritten as Eq. 2. Interestingly. it can be observed that with ReLU activation function, the convolution weights are different for all spatio- temporal locations, since the binary map M depends the results of the spatial convolutions.
14

---

Published as a conference paper at ICLR 2022
Table Al: Comparison of different operations for spatial and temporal modeling (Lin et al., 2019a;
Tran et al., 2018; Wang et al., 2020; Hara et al., 2018). "T.' refers to the temporal modeling ability.
T. Operation
Parameters
FLOPs CxCCxFE X THW
X Spat. conv
C. x C. x k
/ Temp. conv
C. x C x
X THW
/ Temp. shift
C C. x
C, K THW C. X, (k² + THW
(2+1)D conv
C. x C (k² +k)
/ 3D conv
C, x C. k2
C. C, x & THW
Correlation
x x 22
x THW
C.x.x THW + (THW +T)
TAdaConv C. x C x 12
C
(xx
B COMPUTATIONAL ANALYSIS
Consider the input tensor with the shape of C2 x T x H x W. where C2 denotes the number of in- put channels, TAdaConv essentially performs 2D convolution, with weights dynamically generated. Hence, a good proportion of the computation is carried out by the 2D convolutions:
FLODssConn2D)- C, X
Params(Conv2D) = C. C k
where C. denote the number of output channels, and denotes the kernel size of the 2D) convolu- tions. For the weight generation, the features are first aggregated by average pooling over the spatial dimension and the temporal dimension, which contains no parameters:
FLOPs(GAP sponial) C. x T 11
FLOPs remporal) C2 x T
For the local information, layer ID convolution with kernel size of kw are applied, with reduction ratio of r in between. For global information, a one layer ID convolution with kernel size of is applied.
FLOPs(Gen) 2 x C₁ C C. C k Ct/r
Params(Gen)
Further, the calibration weight a E R is multiplied to the kernel weight of 2D convolutions W e CCxxx
FalossCal)ibratio) C. x C XT,
Hence, the overall computation and parameters are:
FLOPP:(AAdaComN) = =FI.OPs(Com2D)) +
+ FL.OPs(Calibratio))
=C, x C. 3 THW + +0xT
+2x C.
C + C x
Params(TAdaConv) -Params(Con22D) + Params(Gen)
=C, x 2 + x C. C k+ x Ci/r
For the FLOPs, despite the overwhelming number of terms, all the other terms at least order of magnitude smaller than the first term. This is in contrast to the (2+1)D convolutions for spatio- temporal modelling, where the FLOPs are C. C x (k2 + k) THW. The extra computation introduced by the temporal convolutions is only k times smaller than the 2D convolutions. Table Al shows the comparison of computation and parameters with other approaches.
C FURTHER IMPLEMENTATION DETAILS
Here, further describe the implementation details for the action classification and action localiza- tion experiments. For fair comparison, we keep all the training strategies the same for our baseline, the plug-in evaluations well our models.
15

---

Published as a conference paper at ICLR 2022
C.I ACTION CLASSIFICATION
Our experiments on the action classification are conducted on three large-scale datasets. For all action classification models, we train them with synchronized SGD using 16 GPUs. The batch size for each GPU is 16 and 8 respectively for 8-frame and 16-frame models. The weights in TAda21) are initialized using ImageNet (Deng et al., 2009) pre-trained weights (He et al., 2016), except for the calibration function 9 and the batchnorm statisties (BNg) in the average pooling branch. In the calibration function, we randomly initialize the first convolution layer (for non-linear weight generation) following He et al. (2015), and fill zero the weight of last convolution layer. The batchnorm statistics are initialized to be zero so that the initial state behaves the same as without the average pooling branch. For all models, we use a dropout ratio (Hinton et al., 2012) of 0.5 before the classification heads. Spatially, we randomly resize the short side of the video to [256, 320] and crop a region of 224 to the network in ablation studies, and set the scale to [224, 340] following TANet (Liu et al., 2021b) for comparison against the state-of-the-arts. Temporally, we perform interval based sampling for Kinetics-400 and Epic-Kiine Soiic-Ktthhens-i 100, with interval of 8 for 8 frames, interval of 5 for 16 frames and interval of 2 for 32 frames. On we perform segment based sampling.
On Kinetics-400, a half-period cosine schedule is applied for decaying the learning rate follow- ing Feichtenhofer et al. (2019), with the base learning rate set to 0.24 for ResNet-base models using SGD. For TAdaConNNeXt, the base learning rate is set 0.0001 for the backbone and 0.001 for the head, using adamw (Loshchilov & Hutter, 2017) as the optimizer. The models are trained for 100 epochs. In the first 8 epochs, we adopt a linear warm-up strategy starting from a learning rate of 0,01. The weight decay is set to le-4. The frames are sampled based on a fixed interval, which is 8 for 8-frame models, 5 for 16-frame models and 2 for 32-frame models. Additionally for TAdaConvNeXt-T, drop-patt path rate of 0.4 is employed.
On SpiicKiichhens-O0 the models are initialized with weights pre-trained on Kinetics-400, and are further fine-tuned following similar strategy as in kinetics. The training length is reduced to 50 epochs, with 10 epochs for warm-up. The base learning rate 0.48. Following Damen et al. (2020) and Huang et al. (2021), we connect two separate heads for predicting verbs and nouns. Action predictions are obtained according to the strategies in Huang et al. (2021), which is shown to have a higher accuracy over the original one in Damen et al. (2020), For fair comparison, we also trained and evaluated our baseline using the same strategy.
On Something- Something- V2, we initialize the model with ImageNet pretrained weights for ResNet- based models and Kinetics-400 pre-trained weights for TAdaConNNeXt.. A segment-based sampling strategy is adopted, where for T-frame models, the video is divided into T segments before one frame is sampled from each segment randomly for training or uniformly for evaluation. The models are trained for 64 epochs, with the first 4 being the warm-up epochs. The base learning rate is set to 0.48 in training TAda2D with SGD, and 0,001/0.0001 respectively for the head and the backbone in training NedaCon with adamw. Following Liu et al. (2021a), we use stronger augmentations such as mixup (Zhang et al., 2017), cutmix (Yun et al., 2019), random erasing (Zhong et al., 2020) and randaugment (Cubuk et al., 2020) with the same parameters in Liu al. (2022).
It is worth noting that, for SlowFast models (Feichtenhofer et al., 2019) in the plug-in evaluations, we do not apply precise batch normalization statistics in our implementation as in its open-sourced codes, which is possibly the reason why our re-implemented performance is slightly lower than the original published numbers.
C.2 ACTION LOCALIZATION
We evaluate our model on the action localization task using two large-scale datasets. The overall pipeline for our action localization evaluation is divided into finetuning the classification models, obtaining action proposals and classifying the proposals.
Finetuning. On KitchKischens, we simply use the evaluated action classification model. On HACS. following (Qing et al., 2021c), we initialize the model with Kinetics-400 pre-trained weights and train the model with adamW for 30 epochs (8 warmups) using 32 GPUs. The mini-batch size is 16 vidcos per GPU. The base learning rate is set to 0.0002, with cosine learning rate decay as in Kinetics. In our case, only the segments with action labels are used for training.
16

---

Published as a conference paper at ICLR 2022
Table A2: Model structure R3D, R(2+1)D and R2D that used in our experiments. Blue and green fonts indicate respectively the default convolution operation and optional operation that can be replaced by TAdaConv. (Better viewed color.)
Stage
R3D
R(2+1)D
R2D (default baseline)
output sizes
Sampling
interval 8,
interval 8, 1-
interval I'
8x224x224
3x7., 64
1x7,, 64
1x7 64
conv1
stride 2
stride 2
stride 1,
8x112x112
1x1². 64
1x1², 2 64 3x3², 64
1x12 64
1x3². 64
res2
x3
3x12.64
x3
1x32, 64
x
8x56x56
1x1*, 256
Ix1², 256
1x1,, 256
Ix1². 128
1x13, 128
Ix1². 128
1x32, 128 128
re83
3x3², 128
x4
3x12,128
x4
1x3², 128
x4
2x
1x12, 512
1x12, 512
1x12.512
1x1², 256
1x12, 256
Ix1², 256
1x32, 256
TCS4
3x3², 256
x6
3x12,256
x6
1x3², 256
x6
8x14x14
1x12, 1024
1x1,, 1024
Ix1², 1024
1x1;, 512
1x1², 512
Ix1², 512
1x32, 512
rc85
3x3², 512
x3
3x12,512
x3
1x3', 512
x3
8x7x7
1x1?, 2048
1x12, 2048
1x1² 2048
2048
global average pool,
1x1x1
Proposal generation. For the action proposals, boundary matching network (BMN) (Lin et al., 2019b) is trained over the extracted features on the two datasets. On Epic-Kitchens, we extract features with the videos uniformly decoded at 60 FPS. For each clip, use frames with an interval to be consistent with finetuning, which means a feature roughly covers video clip of one seconds. The interval between each clip for feature extraction is 8 frames (i.e., 0.133 sec) as well. The shorter side of the video resized to 224 and we feed the whole spatial region into the backbone to retain much information possible. Following Qing al. (2021a). we generate proposals using BMN based on sliding windows. The predictions on the overlapped region of different sliding windows simply averaged. On HACS, the videos decoded at 30 FPS, and extend the interval between clips to be 16 (i.e., 0.533 sec) because the actions in HACS last much longer than in Epic- Kitchens. The shorter side is resized 128 for efficient processing. For the settings in generating proposals, we mainly follow Qing et al. (2021c), except that the temporal resolution is resized to 100 in our case instead of 200.
Classification. On Epic-Kitchens. we classify the proposals with the fine-tuned model using 6 clips. Spatially, to comply with the feature extraction process, we resize the shorter side to 224 and feed the whole spatial region to the model for classification. On HACS, considering the property of the dataset that only one action category can exist video, we obtain the video level classification results by classifying the video level features, following Qing al. (2021c).
Evaluation. For evaluation, we follow the standard evaluation protocol used the respective datasets, i.e., the average mean Average Precision (average mAP) at IoU threshold [0.5:0.05:0.95] for HACS (Zhao et al., 2019) and [0.1:0.1:0.35 for Epic-Kine Kitchens- (Damen et al., 2020).
D MODEL STRUCTURES
The detailed model structures for R2D, R(2+1)D and R3D specified in Table A2, We highlight the convolutions that are replaced by TAdaConv by default optionally. For all of our models, small modification made that we remove the max pooling layer after the first convolution and set the spatial stride of the second stage be 2, following Wang al. (2020). Temporal resolution is kept unchanged following recent works (Feichtenhofer et al., 2019; Li et al., 2020b; Jiang et al., 2019). Our R3D is obtained by simply expanding the R2D baseline in the temporal dimension by factor
17

---

Published as a conference paper at ICLR 2022
TSN
Improvement of TAdx2D
80
60
Accuracy
80-
20
10
20
30
40
50
60
70
80
90
100
110
120
130
140
150
160
170
Class Index
Figure Al: Per-category performance comparison of TAda2D against the baseline TSN. We achieve average per-category performance improvement of 30.35%.
Table A3: Comparison with the state-of-the-art approaches over action classification on Epic- Kitchens- 100 (Damen et al., 2020). indicates the main evaluation metric for the dataset. For fair comparison, we implement all the baseline models using own training strategies.
Top-1
Top-5
Model
Frames
GFLOPs
Params.
Act.
Verb
Noun
Act.f
Verb
Noun
SlowFast 4x 16
4+32
36.10
34.5M
38.17
63.54
48.79
58.68
89.75
73.37
SlowFast 16 + TAdaConv
4+32
36.11
37.7M
39.14
64.50
49.59
59.21
89.67
73.88
SlowFast 8x8
8+32
65.71
34.5M
40.08
65.05
50.72
60.10
90.04
74.26
SlowFast 8x8 TAdaConv
8+32
65.73
37.7M
41.35
66.36
52.32
61.68
90.59
75.89
R(2+1)D
49.55
28.1M
37.45
62.92
48.27
58.02
89.75
73.60
R(2+1)D TAdaConva
49.57
31.3M
39.72
64.48
50.26
60.22
90.01
75.06
R(2+1)D TAdaConvaasid
49.58
34.4M
40.10
64.77
50.28
60.45
89.99
75.55
R3D
84.23
47.0M
36.67
61.92
47.87
57.47
89.02
73.05
R3D TAdaCCnvva
84.24
50.1M
39.30
64.03
49.94
59.67
89.84
74.56
of three. We initialize with weights reduced by times, which means the original weight is evenly distributed in adjacent time steps. We construct the R(2+/)D by adding a temporal convolution operation after the spatial convolution. The temporal convolution can also optionally replaced by TAdaConv, as shown in Table and Table A3. For its initialization, the temporal convolution weights are randomly initialized, while the others are initialized with the pre-trained weights on ImageNet. For SlowFast models, we keep all the model structures identical the original work (Feichtenhofer et al., 2019).
For TAdaConvNeXt. keep most of the model architectures in ConvNeXt (Liu et al., 2022), except that we use a tubelet embedding similar to (Amab et al., 2021), with size of 3x4x4 and stride of 2x4x4. Center initialization is used as in (Araab et al., 2021). Based on this, simply replace the depth-wise convolutions with TAdaConv to construct N'daConnNeeXt.
E PER-CATEGORY IMPROVEMENT ANALYSIS ON SSV2
This section provides a per-category improvement analysis on the Something-Somettie in Fig.Al. As shown Table our TAda2D achieves an overall improvement of 31.7%. Our per-category analysis shows an mean improvement of 30.35% over all the classes. The largest improvement observed in class 0 (78.5%, Approaching something with your camera), 32 (78.4) 78,4% Moving away from something with your camera). 30 (74.3%, Lifting up one end of something without letting drop down), 44 (66.2%, Moving something towards the camera) and something away from the camera). Most of these categories contain large movements across the whole video, whose improvement benefits from temporal reasoning over the global spatial context. For class 30, most of its actions lasts long time (as it needs to be determined whether the end of something is let down or not). The improvements over the baseline mostly benefits from the global temporal context that included in the weight generation process.
dataset
166,1%, Moving
18

---

Published as a conference paper at ICLR 2022
Table A4: Ablation studies. (b) Ablation studies kernel size with non linear calibration weight generation function.
(a) Ablation studies ker- nel size with linear calibration weight generation function.
(c) Ablation studies reduction ratio
1 K
Kernel size
Top-1
K,=1
K
K==5
K2=7
Ratio r
Top-1
37.5
1
36.8
57.1
57.8
57.9
57.79
3
56.5
K,=3
57.3
57.8
57.9
58.0
2
57.83
5
57.3
K,=5
57.6
57.9
58.2
57.9
4
57.78
7
56.5
K,=7
57.6
58.0
57.6
57.66
Table A5: Plug-in evaluation of TAdaConv the action localization HACS and SriicKKichenn... 1 indicates the main evaluation metric for the dataset. 'S.F' is SlowFast network.
HACS
Epie-Kitchen-
Model
@0.5 @0.6 @0.7 @0.8 @0.9 Avg.1
Task @0.1 @0.2
@0.3 @0.4 @0.5
Avg.t
Verb 19.93
18.92 17.90 16.08
13.24
17.21
S.F. 8x8
50.0 44.1 37.7 29.6 18.4 33.7 Noun 17.93
16.83 15.53
13.68
11.41
15.07
Act. 14.00 13.19 12.37
11.18
9.52
12.04
Verb 19.96 18.71 17.65
15.41
13.35
17.01
S.F. 8x8 + TAdaConv 51.7 45.7 39.3 31.0 19.5 35.1
Noun 20.17 18.90
17.58
15.83
13.18
17.13
Act.t 14.90
14.12
13.32
12.07
10.57
13.00
F FURTHER ABLATION STUDIES
Here provide further ablation studies on the kernel size in the calibration weight generation. As shown in Table A4a and Table A4b, kernel size does not affect the classification much, as long as the temporal context is considered. Further, Table A4c shows the sensitivity analysis the reduction ratio, which demonstrate the robustness of our approach against different set of typer-parametent
G FURTHER PLUG-IN EVALUATION FOR TADACONV ON CLASSIFICATION
In complement Table 3, we further show in Table A3 the plug-in evaluation the action clas- sification task on the Epic-Kitchens- dataset. As in the plug-in evaluation on Kinetics and Sometting-Sometting--Z. we compare performances with and without TAdaConv over three base- line models, SlowFast (Feichtenhofer et al., 2019), R(2+1)D (Tran et al., 2018) and R3D (Hara al., 2018) respectively representing three kinds of temporal modeling techniques. The results are in line with our observation in Table 3. Over all three kinds of temporal modelling strategies, adding TAdaConv further improves the recognition accuracy of the model.
H PLUG-IN EVALUATION FOR TADACONV ON ACTION LOCALIZATION
Here, we show the plug-in evaluation on the temporal action localization task. Specifically, we use SlowFast as our baseline, as is shown be superior in the localization performance in Qing et al. (2021b) compared to many early backbones. The result is presented in Table A5, With TAdaConv, the average mAP on HACS is improved by 1.4%, and the average mAP on Sppic-Kitchens-e action localization is improved by 1.0%.
I COMPARISON OF TRAINING PROCEDURE
In this section, we compare the training procedure of TSN and TAda2D on Kinetics-400 and SoarthinngSSmething---.e V2. The results presented in Fig. A2. TAda2D demonstrates stronger fitting ability and generality even from the early stages of the training, despite that the initial state of TAda2D is identical to that of TSN.
19

---

Published as a conference paper at ICLR 2022
90
Kinetics 400
Something Something V2
90
80-
80
70
TTpp Eroee (Ng
60-
Topl Iiroe (N6)
60
.50
50-
40
40-
TSN (training)
TSN (training)
TSN (validation)
30
TSN (validacion)
TAda2D (training)
TAda2D (raining)
20
TAda2D (validation)
TAde2D (validatio)
20
10
0
20
40
60
80
100
10
30
40
50
60
Epochs
Epochs
Figure A2: Training and validation on Kinetics-400 and Something SooethingSSmmthing-- V2. On both datasets, TAda2D shows a stronger capability of fitting the data and better generality to the valida- tion set. Further, TAda2D reduces the overfitting problem in
V2. Table A6: Approach comparison between different dynamic filters. The weights column denotes how weights in respective approaches are obtained. The pre-trained weights colmun shows whether the weight generation can exploit pre-trained models such as ResNet (He al., 2016).
Temporal Location Pretrained
Operations
Weights
Modelling Adaptive
weights
CondConv
Mixture of experts W = Σn f(x), W..
X
X
DynamicFilter
Completely generated W =g(x)
DDF
Completely generated W =g(x)
TAM
Completely generated W = =9(x)
X
X
TAdaConv
Calibrated from a base weight W l,
/
/
J COMPARISON WITH EXISTING DYNAMIC FILTERS
In this section, we compare our TAdaConv with previous dynamic filters in two perspectives, re- spectively the difference in the methodology and in the performance.
J.1 COMPARISON IN THE METHODOLOGY LEVEL
For the former comparison, we include Table A6 to show the differences in different approaches, which is a full version of Table 1. We compare TAdaConv with several representative approaches in image and in videos, respectively CondConv (Yang et al., 2019), DynamicFilter (Jia et al., 2016), DDF (Zhou al., 2021) and TAM (Liu et al., 2021b).
The first difference the methodology level lies in the source of weights, where previous approaches obtain weights by mixture of experts generation completely dependent the input. Mixture of experts denotes W = Σn, on W.. where on a scalar obtained by a function f. iee. W En, fxx,,, W., n- Completely generated means the weights are only dependent on the input, i.e. W = 9(x)), where generates complete kernel for the convolution. In comparison, the weights in TAdaConv are obtained by calibration, i.e, W a W,, where is vector calibration weight and h((x)) where h(.) generates the calibration vector for the convolutions. Hence, this fundamental difference in how obtain the convolution weights makes the previous approaches difficult to exploit pre-trained weights, while TAdaConv can easily load pre-trained weights in W,, This ability is essential for video models to speed up the convergence.
The second difference lies in the ability to perform temporal modelling. The ability to perform tem- poral modelling does not only mean the ability to generate weights according to the whole sequence in dynamic filters for videos, but it also requires the model to generate different weights for the same set of frames with different orders. For example, weights generated by the global descriptor obtained by global average pooling over the whole video GAP, st does not have the temporal mod- elling ability, since they can not generate different weights if the order of the frames the input sequence reversed randomized. Hence, most image based approaches based global de-
20

---

Published as a conference paper at ICLR 2022
Table A7: Performance comparison with other dynamic filters, Our Init. denotes initializing the cal- ibration weights to ones so that the initial calibrated weights identical to the pre-trained weights. Temp. Varying is short for temporally varying, which indicates different weights for different tem- poral locations (frames). * denotes that the branch was originally not designed for generating filter calibration weights, but slightly modified the structure that it be used for calibration weight generation. (Numbers in brackets) show the performance improvement brought by our initialization scheme for calibration weights.
Calibration Generation
Our Init.
Temp. Varying Generation source
Top-1
DynamicFilter
(A),, x 1)
41.7
DDF-like
GA),
49.8
TAM (global branch)
X
X
(x)
39.7
TAM +global branch)
X
GXP,
41.3
DynamicFilter
X
(C
51.2 (+9.5)
DDF-like
/
(X
53.8 (44.0)
TAM (global branch)
X
(x). (X
52,9 (+13,2)
TAM (local*+global branch)
J
(C
54.3 (+13.0)
TAdaConv w/o glohal info
(X
57.9
both (XPx( X
TAdaConv
59.2
and GAPs T)
scriptor vectors (such as CondConv and DynamicFilter) or based on adjacent spatial contents (DDF) can not achieve temporal modelling. TAM generates convolution weights for temporal convolutions based on temporally local descriptors obtained by the global average pooling over the spatial dimen- sion GAP a. which yields different weights if the sequence changes. Henee, in this sense, TAM has the temporal modelling abilities. In contrast, TAdaConv exploits both temporally local and global descriptors utilize not only local but also global temporal contexts. Details on the source of the weight generation process is also shown in Table A7.
The third difference lies in whether the weights generatd shared for different locations. For CondConv, DynamicFilter and TAM, their generated weights are shared for all locations, while for DDF, the weights are varied according spatial locations. In comparison, TAdaConv generate temporally adaptive weights.
J.2 COMPARISON IN THE PERFORMANCE LEVEL
Since TAdaConv is fundamentally different from previous approaches in the generation of calibra- tion weights, it is difficult to directly compare the performance on video modelling, especially for those that not designed for video modelling. However, since the calibration weight in TAdaConv a is completely generated, iee, a = f(xx)), we can use other dynamic filters to generate the cal- ibration weights for TAdaConv. Since MoE based approaches such CondConv were essentially designed for applications with less memory constraint but high computation requirements, it is not suitable for video applications since it would be too memory-heavy for video models. Hence, we ap- ply approaches that generate complete kernel weights to generate calibration weights, and compare them with TAdaConv. The performance listed Table A7.
It is worth noting that these approaches originally generate weights that are randomly initialized. However, as is shown in Table 6, our initialization strategy for the calibration weights are essential for yielding reasonable results, we further apply our initialization on these existing approaches to see whether their generation function better than the one in TAdaConv. In the following paragraphs, we provide details for applying representative previous dynamic filters in TAdaConv to generate the calibration weight.
For DynamicFilter (Jia al., 2016), the calibration weight is generated using MLP over the global descriptor that is obtained by performing global average pooling over the whole input GAP ats i.e, MLP(GAP,.< (x)). In this case, the calibration weights shared between different time steps.
For DDF (Zhou et al., 2021), only the channel branch since it is shown in Table 7 that it is better to leave the spatial structure unchanged for the base kernel. Similarly, the weights in DDF also generated by applying MI. over the global descriptor, iee, MIP(GAP (x)). The
21

---

Published as a conference paper at ICLR 2022
difference between DDF and DynamicFilter is that for different time step, DDF generates a different calibration weight.
The original structure of TAM (Liu et al., 2021b) only generates kernel weights with its global branch, and uses local branch to generate attention maps over different time steps. our experi- ments, modify the TAM little bit and further make the local branch to generate kernel calibra- tion weights as well. Hence, for only-global version of TAM, the calibration weights are calculated ((GAA,(x)), where GAP, denotes global average pooling over the spatial di- mension and G denotes the global branch in TAM. In this case, calibration weights are shared for all temporal locations. For local+global version of TAM, the calibration weight are calculated by combining the results of the local L and the global branch G. i.e. a G(GAP, ([GAP,xx)) (x)) L(GAP, (x)), where denotes element-wis wise multiplication with broadcasting. This means in this case, the calibra- tion weights are temporally adaptive. Note that this is our modified version of TAM. The original TAM does not have a temporally adaptive convolution weights.
follows:
The results in Table A7 show that (a) without our initialization strategy, previous approaches that generate random weights at initialization are not suitable for generating the calibration weights in TAdaConv; (b) our initialization strategy can conveniently change this and make previous ap- proaches yield reasonable performance when they are used for generating calibration weights; and (c) the calibration weight generation function in TAdaConv, which combines the local and global context, outperform all previous approaches for calibration.
Further, when we compare TAdaConv without global information with TAM (local"+global branch), can seen that although both approach generates temporally varying weights from the frame descriptors GAP,(x) (x) with shape C' x T, our TAdaCony achieves notably higer performance. Adding the global information enables TAdaCony to achieve a more notable lead in the comparison with previous dynamic filters.
22

---

Published as a conference paper at ICLR 2022
GT
Protending throw
TSN
Squoczing
R(2+rjn
Tumnng upside
TAda2D
threw
GT
Patting TSN
Wheeo flliiing sth
R(2+1)D
Putting sth. into sth.
Patting sth, underneath sth.
Pulling sth, cato sth.
Moving sth so that they pass each other
R(2+1JD
Putting
onto sth.
TAda2D
Pulling
cato
Putting sth, behind sth.
TSN
Pretending sth. behind
R12+11D
Putting bohhniiddd
TAdu2D
Puting
Figure A3: Further qualitative evaluations on the Something-Somettingg- V2 dataset. In most cases, TAda2D captures meaningful areas in the videos for the correct classification. Further, the activated region of TAda2D also lasts longer along the temporal dimension compared to other two models, thanks to the global temproal context in the weight generation function G.
23


Reviews:
[{"review_id": "gBaOIPZaHdF", "reviewer": "Reviewer_DgYX", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The main strength of the paper lies in the novelty of applying calibration weights directly to the spatial weights in the temporal dimension. Significant previous works have explored various approaches where excitation is applied to the output features in the temporal dimension, but this approach appears to have benefits in terms of computation.\n- The set of experiments is well laid out and there are sufficient ablation studies over the TAdaConv components. Each significant design decision appears to be tested in the experiments.\n- As a drop-in replacement for spatial convolution, it shows clear performance improvements for established architectures. It is clear that this dynamic temporal adaption is something which can be beneficial broadly in convolution-based video-understanding models.\n\nWeaknesses:\n- Computational comparison - it would have been useful to have a visualisation plot of performance vs GFLOPS to be able to compare the model against other state of the art models, given that this is one of the main benefits of the model.\n- The introduction to the paper is not precise on the motivation for the work. The 3rd paragraph introduces what is sought, (\"integrating the dynamic weight calibration into the spatial convolutions\"), but not why it is sought. It would be useful for the reader to understand why this would be a better approach, and what are the deficiencies of the existing approaches that necessitate this.\n- Performance comparisons to TDN are prefaced with the fact that TDN uses more frames. I am not sure if this is fair, given the GFLOPs of this model are in line despite the larger number of frames according to Table 7.\n- The language could do with a little bit of finessing in the paper. There are some instances of grammar mistakes and some unwieldy sentences that should be made more clear to improve readability. For examples the last sentences in the 'Calibration dimension' and 'Different proportion of channels employing TAdaConv' paras on page 7.\n- Some care could be made to clarify what is meant by a term before its use. For example, linear vs nonlinear weight generation, temporally sensitive on page 3.", "summary_of_the_paper": "This work seeks to improve performance of video understanding models though the use of spatial convolutions which are dynamically adapted in time when applied to a sequence of frames. The novelty of this work lies in the fact that the weights of the spatial convolutions are adapted, rather than applying a temporal excitation to feature maps to achieve the same purpose. To achieve this, a drop-in replacement for spatial convolutions is designed which uses pooled local and global frame descriptors to produce the temporal calibration weights. Furthermore, this operation is used as part of a temporal feature aggregation block that is used as part of 2D CNN architecture. The benefits of the approach include that it can make effective use of pre-trained weights and that it can improve model capacity for a marginal increase in computation. Experimentally, it is show large performance improvements over a baseline and match state of the art on significant datasets.", "summary_of_the_review": "I believe that this work presents a novel idea of weight calibration that has benefits in terms of performance and model efficiency that for video understanding tasks. Its use as a drop-in replacement shows that it improves performance for existing architectures. It also shows that there may still be unexplored aspects for improving these model types. Despite the issues that I see in the paper, which I believe could be straightforwardly improved, I think this paper is at an acceptable threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_4d3swuZYD", "reviewer": "Reviewer_4biu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I like the general idea of adaptive weight calibration and its simplicity. The paper was easy to read and understand, and the experimental results are extensive and show good performance. I also appreciate the detailed information presented in the supplementary material.\nThe following are some concerns that I would like to point out.\n\n1) There are no comparisons with any of the dynamic modules. The idea of content-adaptive weights (or modules) has been extensively explored in the area of dynamic networks. However, this paper lacks experiments that compare the performance of TAda2D and previous approaches. In its current form, the paper only shows that using TAda2D can bring some performance gain when compared to TSN (Table 3(a) and Figure 3). I believe that more experiments should be performed to justify the use of TAda2D. For example, is TAda2D the most effective way to calibrate the convolutioanl weights? I do not think that I could find the answer from the current version of the paper.\n\n2) I have another concern about the scalability of TAda2D. In Tables 5 and 7, using more frames brings only a small performance gain for TAda2D. For example, 16fx2x3 increases Top-1 score by 4.3% for TSM but 1.4$ for TAda2D. I am wondering if it implies that TAda2D works better on shorter input video clips.\n\n3) It looks like TAda2D and TANet are very similar in terms of their adaptive characteristics across the temporal dimension. I believe that TAda2D should be compared more deeply with TANet. It is also interesting that TANet and TAda2D have similar patterns in Table 7 (76.3% with 43x30 GFLOPS and 76.9% with 86x30 GFLOPS).\n\n4) I think it would be better to add a column for GFLOPS and #Params for Table 5 and A2 because the paper put weight on TAdaConv's efficiency. In addition, it is a bit hard to grasp Table 1. It would be better to add example values as already done in footnote 2.", "summary_of_the_paper": "This paper proposes TAdaConv that calibrates kernel weights of convolutional layers adaptively according to the temporal dynamics of the input tensor. It is designed to incorporate both the local and global temporal context by using stacked two-layer 1D convolutional operations and global average pooling. Since it works exactly the same as the original convolutional layer at the initial stage, it is easy to insert TAdaConv into existing ConvNet architectures. On top of this, the authors construct TAda2D networks by introducing the temporal feature aggregation module that is based on a strided temporal average pooling. Many experiments performed on video/action classification and localization demonstrate the effectiveness of the proposed module.", "summary_of_the_review": "The paper is well-motivated and easy to understand. There are a few things I would like to point out: 1) Comparison with dynamic modules and TANet, 2) Scalability for longer input clips, 3) Clearly show the efficiency of TAda2D.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YjNA9BaHgA", "reviewer": "Reviewer_jnMz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n- The paper is well written and easy to follow and understand. The approach is well described.\n- The idea of generating weights is interesting and has not been greatly studied in video recognition.\n- The paper conducts experiments on multiple standard datasets, so it can be compared to other approaches.\n\n\n**Weaknesses**\n\n- Eq. 1 and 2 are a bit misleading. The addition of the activation function (and normalization layers) leads to a non-linearity in the network, which has been shown to be the critical part of NNs. Omitting this part is leaving out a key piece of NNs. It makes it seem like (2+1)D convolution is just a multiplication of the weights, when in fact the non-linearity is important. This should be corrected to clarify the connection and differences between the proposed approach and convolutions.\n\n- Similarity to previous works. The approach is not especially novel, the idea of generating temporal weights from 2d weights has been shown before. For example, see \"3D Inflated TGM layer\" in \"Evolving Space-Time Neural Architectures for Videos\", ICCV'19, as well as all the (2+1)D CNN works (e.g., S3D, \" A Closer Look at Spatiotemporal Convolutions for Action Recognition\",CVPR'18, etc.). It seems the main difference is that the temporal weights are dynamically generated here. However, there are no experiments showing the difference between static, learned weights and dynamic weights. Adding this experiment would strengthen the paper.\n\n- Table 1 is quite hard to understand and see the difference between the approaches. It would be helpful to have a table comparing say a ResNet-50 with each of those configurations (or a variety of networks) to show how the flops and parameters varies. As is, it appears that this approach is less than 3D conv, but more that the others? (This seems to be part of table 2, but table 1 still is quite hard to follow).\n\n- The claims of state-of-the-art performance are wrong. Many results on Kinetics-400 are omitted (see https://paperswithcode.com/sota/action-classification-on-kinetics-400 have many over 82% while this paper reports a best of 77.9). The claims and/or table should be updated and corrected/clarified. Similarly, S-Sv2 (https://paperswithcode.com/sota/action-recognition-in-videos-on-something) has numbers over the best reported 66.7% in this paper and epic kitchens (https://paperswithcode.com/sota/action-recognition-on-epic-kitchens-100) well over the 40.89% reported.\n\n", "summary_of_the_paper": "This paper proposes a modified version of convolution for videos where a 2D conv kernel is converted to a 3D conv kernel by multiplying the 2d weights by a per-frame value. This creates a set of weights for 3D convolution that are dynamic for each frame. The approach is evaluated on multiple video datasets, and shows some benefit over their baseline.", "summary_of_the_review": "Overall, the paper makes an incremental contribution by essentially making (2+1)D convolution adaptive by generating the 1D conv weights within the network. The experimental claims need clarifications and revisions, as they are currently omitting stronger performing models. The ablation experiments could be strengthened by including experiments on the effect of dynamic vs. static weights. Overall the paper makes an incremental contribution and the experiments are not currently strong enough to show the value of it. With revisions, the paper could be much stronger.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "B4vPWr1V4LI", "reviewer": "Reviewer_6Gv3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The paper is well-organized and easy to follow.\n2. The proposed approach is simple and efficient. Temporal modeling by calibrating convolutional weights can achieve significant improvement while introducing less computation cost and additional parameters compared with other methods like (2+1)D.\n3. TAdaConv can be easily plugged into any convolutional network based deep models for sequential modeling.\n4. Adequate experiments on K400 and SSV2 illustrate the superiority of the proposed approach.\n\nWeakness:\n1. In Equation 7, the outputs of the first term and the second term are in different shapes. How do you add those features together? I suppose you may simply aggregate them together, but please describe the details clearly.\n2. Please add the descriptions of GAPs as global average pooling over spatial dimension and GAPst as global average pooling over spatial and temporal dimension to make the audience have a good understanding of Equation 5.\n3. According to Equation 4, v_t is the combination of features with temporal kernel size as 3. I would like to see more ablation study of kernel size over temporal dimension of TAdaConv (e.g., 3 vs 5 vs 7).\n4. According to the recent progress on neural architecture search for action recognition [1], I would like to suggest the authors adding TadaConv into search space in the future work. \n\n[1] Kondratyuk, Dan, et al. \"Movinets: Mobile video networks for efficient video recognition.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.", "summary_of_the_paper": "The authors proposed a novel approach for video understanding. They present temporally adaptive convolutions (TAdaConv) based on dynamic networks by adapting the convolutional weights on each frame with its temporal contexts. The experiment results on public benchmarks demonstrate the efficiency of the proposed approach.", "summary_of_the_review": "Overall, the paper is clear and the author propose an efficient and practical approach for video understanding. So I would like to provide a positive score in the initial stage of the review and keep tuning during the discussion period.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors study the problem of video classification and propose a new module which promises to increase accuracy while keeping the computational overhead low. The main idea is not to share the spatial convolution weights over different time steps, but allow some modulation based on pooled local and global frame descriptors. The resulting module can be used as a drop-in replacement for spatial convolutions in existing models and yields competitive performance on multiple video action recognition and localisation benchmarks.

The reviewers appreciated this challenging setting and the simplicity of the main idea. They found that the paper was clearly written, well organised, and easy to follow. The reviewers raised some issues in connections with related work and the empirical evaluation which were successfully resolved during the discussion phase.

Given that computational efficiency remains as one of the most challenging topics in video understanding, I believe that this technique will be relevant for the larger video understanding community. I strongly suggest that the authors incorporate the feedback received during the discussion, especially the GFLOPS vs accuracy plots, and further clarify the relationship to existing work.
Decision: Accept (Poster)