Year: 2021

Abstract:
Spatial convolutions1 are widely used in numerous deep video models. It fundamentally assumes spatio-temporal invariance, i.e., using shared weights for every location in different frames. This work presents Temporally-Adaptive Convolutions (TAdaConv) for video understanding, which shows that adaptive weight calibration along the temporal dimension is an efficient way to facilitate modelling complex temporal dynamics in videos. Specifically, TAdaConv empowers the spatial convolutions with temporal modelling abilities by calibrating the convolution weights for each frame according to its local and global temporal context. Compared to previous temporal modelling operations, TAdaConv is more efficient as it operates over the convolution kernels instead of the features, whose dimension is an order of magnitude smaller than the spatial resolutions. Further, the kernel calibration also brings an increased model capacity. We construct TAda2D networks by replacing the spatial convolutions in ResNet with TAdaConv, which leads to on par or better performance compared to state-of-the-art approaches on multiple video action recognition and localization benchmarks. We also demonstrate that as a readily plug-in operation with negligible computation overhead, TAdaConv can effectively improve many existing video models with a convincing margin.

heading: 1 INTRODUCTION
text: Convolutions are an indispensable operation in modern deep vision models (He et al., 2016; Szegedy et al., 2015; Krizhevsky et al., 2012), whose different variants have driven the state-of-the-art performances of convolutional neural networks (CNNs) in many visual tasks (Xie et al., 2017; Dai et al., 2017; Zhou et al., 2019) and application scenarios (Howard et al., 2017; Yang et al., 2019). In the video paradigm, compared to the 3D convolutions (Tran et al., 2015), the combination of 2D spatial convolutions and 1D temporal convolutions are more widely preferred owing to its efficiency (Wang et al., 2016; Lin et al., 2019a; Liu et al., 2021; Qiu et al., 2017). Nevertheless, 1D temporal convolutions still introduce non-negligible computation overhead on top of the spatial convolutions. Therefore, we seek to directly equip the spatial convolutions with temporal modelling abilities.
One essential property of the convolutions is the translation invariance (Ruderman & Bialek, 1994; Simoncelli & Olshausen, 2001), resulted from its local connectivity and shared weights. However, recent works in dynamic filtering have shown that strictly shard weights for all pixels may be suboptimal for modelling various spatial contents (Zhou et al., 2021; Wu et al., 2018).
Given the diverse nature of the temporal dynamics in videos, we hypothesize that the temporal modelling could also benefit from relaxed invariance along the temporal dimension. This means that the convolution weights for different time steps are no longer strictly shared. Existing dynamic filter networks can achive this, but with two drawbacks. First, it is difficult for most of them (Zhou et al., 2021; Yang et al., 2019) to leverage the existing pre-trained models such as ResNet. This is critical in video applications, since training video models from scratch is highly resource demanding (Feichtenhofer et al., 2019; Feichtenhofer, 2020) and prone to over-fitting on small datasets. Second, for most dynamic filters, the weights are generated with respect to its spatial context (Zhou et al., 2021; Jia et al., 2016) or the global descriptor (Chen et al., 2020a; Yang et al., 2019), which is difficult to capture the temporal variations between frames.
1In this work, we use spatial convolutions and 2D convolutions interchangeably.
In this work, we present the Temporally-Adaptive Convolution (TAdaConv) for video understanding, where the convolution kernel weights are no longer fixed across different frames. Specifically, the convolution kernel for the t-th frame Wt is factorized to the multiplication of the base weight and a calibration weight: Wt = αt ·Wb, where the calibration weight αt is adaptively generated from the input data for all channels in the base weight Wb. For each frame, we generate the calibration weight based on the frame descriptors of its adjacent time steps as well as the global descriptor, which effectively encodes the local and global temporal dynamics in videos. The difference between TAdaConv and the spatial convolutions is visualized in Fig. 1.
The main advantages of this factorization are three-fold: (i) TAdaConv can be easily plugged into any existing models to enhance temporal modelling, and their pre-trained weights can still be exploited; (ii) the temporal modelling ability can be highly promoted with the help of the temporallyadaptive weight; (iii) in comparison with temporal convolutions that often operate on the learned 2D feature maps, TAdaConv is more efficient by directly operating on the convolution kernels.
TAdaConv is proposed as a drop-in replacement for the spatial convolutions in existing models. It can both serve as a stand-alone temporal modelling module for 2D networks, or be inserted into existing convolutional video models to further enhance the ability to model temporal dynamics. For efficiency, we construct TAda2D by replacing the spatial convolutions in ResNet (He et al., 2016), which leads to on par or better performance than the state-of-the-arts. Used as an enhancement of existing video models, TAdaConv leads to notable improvements on multiple video datasets. The strong performance and the consistent improvements demonstrate that the temporally adaptive convolutions can be an important operation for modelling complex temporal dynamics in videos.

heading: 2 RELATED WORK
text: ConvNets for temporal modelling. A fundamental difference between videos and images lies in the temporal dimension, which makes temporal modeling an important research area for understanding videos. Recent deep CNNs for video understanding can be divided into two types. One models spatio-temporal information jointly by 3D convolutions (Carreira & Zisserman, 2017; Tran et al., 2015; Feichtenhofer, 2020; Tran et al., 2019). The other builds upon 2D networks, where most approaches employ 2D convolutions that share weights among all the frames for spatial modelling, and design additional operations for temporal modelling, such as temporal shift (Lin et al., 2019a), temporal difference (Wang et al., 2021; Jiang et al., 2019), temporal convolution (Jiang et al., 2019; Tran et al., 2018; Liu et al., 2021) and correlation operation (Wang et al., 2020), etc. This work directly empowers the spatial convolutions with temporal modelling abilities, which can be further coupled with other temporal modelling operations for stronger video recognition performances.
Dynamic networks. Dynamic networks refer to networks with content-adaptive weights or modules, such as dynamic filters/convolutions (Jia et al., 2016; Yang et al., 2019; Li et al., 2021), dynamic activations (Li et al., 2020d; Chen et al., 2020b), and dynamic routing (Wang et al., 2018b; Li et al., 2020c), etc. Dynamic networks have demonstrated their exceeding network capacity and thus performance compared to the static ones. In video understanding, dynamic filter (Liu et al., 2021) or temporal aggregation (Meng et al., 2021) also demonstrate strong capability of temporal modelling. Recently, some convolutions applies spatially-adaptive weights (Elsayed et al., 2020; Chen et al., 2021), showing the benefit of relaxing the spatial invariance in modelling diverse visual contents. Hence, the proposed TAdaConv enhances temporal modelling by relaxing the invariance along the temporal dimension. TAdaConv has two key differences from previous works: (i) the convolution weight in TAdaConv is factorized into a base weight and a calibration weight, being able to exploit pre-trained weights; (ii) the calibration weights are generated with respect to the temporal contexts.

heading: 3 TADACONV: TEMPORALLY-ADAPTIVE CONVOLUTIONS
text: In this work, we seek to empower the spatial convolutions with temporal modelling abilities. Inspired by the calibration process of temporal convolutions (Sec. 3.1), TAdaConv dynamically calibrates the convolution weights for each frame according to its temporal context (Sec. 3.2).

heading: 3.1 REVISITING TEMPORAL CONVOLUTIONS
text: We first revisit the temporal convolutions to showcase its underlying process and its relation to dynamic filters. We consider depth-wise temporal convolution for simplicity, which is more widely used because of its efficiency (Liu et al., 2021; Jiang et al., 2019). Formally, for a 3×1×1 temporal convolution filter parameterized by β = [β1,β2,β3] and placed (ignoring normalizations) after the 2D convolution parameterized by W, the output feature x̃t of the t-th frame can be obtained by:
x̃t = β1 · δ(W ∗ xt−1) + β2 · δ(W ∗ xt) + β3 · δ(W ∗ xt+1) , (1) where the · indicates the element-wise multiplication, ∗ denotes the convolution over the spatial dimension and δ denotes ReLU activation (Nair & Hinton, 2010). It can be rewritten as follows:
x̃t = Wt−1 ∗ xt−1 +Wt ∗ xt +Wt+1 ∗ xt+1 , (2)
where Wi,jt−1 = M i,j t−1 · β1 ·W,W i,j t = M i,j t · β2 ·W and W i,j t+1 = M i,j t+1 · β3 ·W are spatiotemporal location adaptive convolution weights. Mt ∈ RC×H×W is a dynamic tensor, with its value dependent on the result of the spatial convolutions (see Appendix A for details). Hence, the temporal convolutions in the (2+1)D convolution essentially performs (i) weight calibration on the spatial convolutions and (ii) feature aggregation between adjacent frames. However, if the temporal modelling is achieved by coupling temporal convolutions to spatial convolutions, a non-negligible computation overhead is still introduced (see Table 2).

heading: 3.2 FORMULATION OF TADACONV
text: For efficiency, we set out to directly empower the spatial convolutions with temporal modelling abilities. Inspired by the recent finding that the relaxation of spatial invariance strengthens spatial modelling (Zhou et al., 2021; Elsayed et al., 2020), we hypothesize that temporally adaptive weights can also help temporal modelling. Therefore, the convolution weights in a TAdaConv layer are varied on a frame-by-frame basis. Since we observe that previous dynamic filters can hardly utilize the pretrained weights, we take inspiration from our observation in the temporal convolutions and factorize the weights for the t-th frame Wt into the multiplication of a base weight Wb shared for all frames, and a calibration weight αt that are different for each time step:
x̃t = Wt ∗ xt = (αt ·Wb) ∗ xt , (3) Calibration weight generation. To allow for the TAdaConv to model temporal dynamics, it is crucial that the calibration weight αt for the t-th frame takes into account not only the current frame, but more importantly, its temporal context, i.e., αt = G(...,xt−1,xt,xt+1, ...). Otherwise TAdaConv would degenerate to a set of unrelated spatial convolutions with different weights applied on different frames. We show an instantiation of the generation function G in Fig. 2(b). In our design, we aim for efficiency and the ability to capture inter-frame temporal dynamics. For efficiency, we operate on the frame description vectors v ∈ RT×C obtained by the global average pooling over the spatial dimension GAPs for each frame, i.e., vt = GAPs(xt). For temporal modelling, we apply stacked two-layer 1D convolutions F with a dimension reduction ratio of r on the local temporal context vadjt = {vt−1,vt,vt+1} obtained from x adj t = {xt−1,xt,xt+1}:
F(xadjt ) = Conv1DC/r→C(δ(BN(Conv1DC→C/r(v adj t )))) . (4)
where δ and BN denote the ReLU (Nair & Hinton, 2010) and batchnorm (Ioffe & Szegedy, 2015).
In order for a larger inter-frame field of view in complement to the local 1D convolution, we further incorporate global temporal information by adding a global descriptor g to the weight generation process F through a linear mapping function FC:
F(xadjt ,g) = Conv1DC/r→C(δ(BN(Conv1DC→C/r(v adj t ) + FC C→C/r(g))) , (5)
where g = GAPst(x) with GAPst being global average pooling on spatial and temporal dimensions.
Initialization. The TAdaConv is designed to be readily inserted into existing models by simply replacing the 2D convolutions. For an effective use of the pre-trained weights, TAdaConv is initialized to behave exactly the same as the standard convolution. This is achieved by zero-initializing the weight of the last convolution in F and adding a constant vector 1 to the formulation:
αt = G(x) = 1+ F(xadjt ,g) . (6) In this way, at initial state, Wt = 1 ·Wb = Wb, where we load Wb with the pre-trained weights.
Calibration dimension. The base weight Wb ∈ RCo×Ci×k 2
can be calibrated in different dimensions. We instantiate the calibration on the Cin dimension (αt ∈ R1×Cin×1), as the weight generation based on the input features yields a more precise estimation for the relation of the input channels than the output channels or spatial structures (empirical analysis in Table 7).
Comparison with other dynamic filters. Table 1 compares TAdaConv with existing dynamic filters. Mixtue-of-experts based dynamic filters such as CondConv dynamically aggregates multiple kernels to generate the weights that are shared for all locations. The weights in most other dynamic filters are completely generated from the input, such as DynamicFilter (Jia et al., 2016) and
DDF (Zhou et al., 2021) in images and TAM (Liu et al., 2021) in videos. Compared to image based ones, TAdaConv achieves temporal modelling by generating weights from the local and global temporal context. Compared to TANet (Liu et al., 2021), TAdaConv is better at temopral modellling because of temporally adaptive weights. Further, TAdaConv can effectively generate weights identical to the pre-trained ones, while it is difficult for previous approaches to exploit pre-trained models. More detailed comparisons of dynamic filters are included in Appendix J.
Comparison with temporal convolutions. Table 2 compares the TAdaConv with R(2+1)D in parameters and FLOPs, which shows most of our additional computation overhead on top of the spatial convolution is an order of magnitude less than the temporal convolution. For detailed computation analysis and comparison with other temporal modelling approaches, please refer to Appendix B.

heading: 4 TADA2D: TEMPORALLY ADAPTIVE 2D NETWORKS
text: We construct TAda2D networks by replacing the 2D convolutions in ResNet (R2D, see Appendix D) with the proposed TAdaConv. Additionally, based on strided average pooling, we propose a temporal feature aggregation module for the 2D networks, corresponding to the second essential step for the temporal convolutions. As illustrated in Fig. 2(c), the aggregation module is placed after TAdaConv. Formally, given the output of TAdaConv x̃, the aggregated feature can be obtained as follows:
xaggr = δ(BN1(x̃) + BN2(TempAvgPoolk(x̃))) , (7)
where TempAvgPoolk denotes strided temporal average pooling with kernel size of k. We use different batch normalization parameters for the features extracted by TAdaConv x̃ and aggregated by strided average pooling TempAvgPoolk(x̃), as their distributions are essentially different. During initialization, we load pre-trained weights to BN1, and initialize the parameters of BN2 to zero. Coupled with the initialization of TAdaConv, the initial state of the TAda2D is exactly the same as the Temporal Segment Networks (Wang et al., 2016), while the calibration and the aggregation notably increases the model capacity with training (See Appendix I). In the experiments, we refer to this structure as the shortcut (Sc.) branch and the separate BN (SepBN.) branch.
This structure is also inspired by Li et al. (2020a), where individual batch normalization parameters for the small and big views are crucial for its valid improvements. Since our aggregation module is placed after TAdaConv, we are able to effectively aggregate information along the temporal dimension without doubling the computation for the preceding convolution as in (Li et al., 2020a).

heading: 5 EXPERIMENTS ON VIDEO CLASSIFICATION
text: To show the effectiveness and generality of the proposed approach, we present comprehensive evaluation of TAdaConv and TAda2D on two video understanding tasks using four large-scale datasets.
Datasets. For video classification, we use Kinetics-400 (Kay et al., 2017), Something-SomethingV2 (Goyal et al., 2017), and Epic-Kitchens-100 (Damen et al., 2020). K400 is a widely used action classification dataset with 400 categories covered by ∼300K videos. SSV2 includes 220K videos with challenging spatio-temporal interactions in 174 classes. EK100 includes 90K segments labelled by 97 verb and 300 noun classes with actions defined by the combination of nouns and verbs.
Training and evaluation. For training TAda2D, 8 or 16 frames are sampled with temporal jittering, following convention (Lin et al., 2019a; Liu et al., 2021; Feichtenhofer et al., 2019). We include further training details in the appendix C For evaluation, we use three spatial crops with 10 clips (K400&EK100) or 2 clips (SSV2) uniformly sampled along the temporal dimension. Each crop has the size of 256×256, which is obtained from a video with its shorter side resized to 256.

heading: 5.1 TADACONV ON EXISTING VIDEO BACKBONES
text: TAdaConv is designed as a plug-in substitution for the spatial convolutions in the video models. Hence, we first present plug-in evaluations in Table 3. TAdaConv improves the classification performance with negligible computation overhead on a wide range of video models, including SlowFast (Feichtenhofer et al., 2019), R3D (Hara et al., 2018) and R(2+1)D (Tran et al., 2018), by an average of 1.3% and 2.8% respectively on K400 and SSV2 at an extra computational cost of less than 0.02 GFlops. Further, not only can TAdaConv improve spatial convolutions, it also notably improve 3D and 1D convolutions. For fair comparison, all models are trained using the same training strategy. Further plug-in evaluations for action classification is presented in Appendix G.

heading: 5.2 ABLATION STUDIES
text: We present thorough ablation studies for the justification of our design choices and the effectiveness of our TAdaConv in modelling temporal dynamics. SSV2 is used as the evaluation benchmark, as it is widely acknowledged to have more complex spatio-temporal interactions.
Dynamic vs. learnable calibration. We first compare different source of calibration weights (with our initialization strategy) in Table 6. Compared with the baseline (TSN (Wang et al., 2016)) with no calibration, learnable calibration with shared weights has limited improvement, while temporally varying learnable calibration (different calibration weights for different temporal locations) performs much stronger. When we use the global descriptor (C×1) to generate dynamic weights,
we observe a larger improvement even if the weights are shared. Making the dynamic weights temporally varying further raises the accuracy. The results also validate our hypothesis that temporal modelling can benefit from temporally adaptive weights. Further, TAdaConv generates calibration weight from both local (C×T) and global (C×1) contexts and achieves the highest performance.
Calibration weight initialization. Next, we show that our initialization strategy for the calibration weight generation plays a critical role for dynamic weight calibration. As in Table 6, randomly initializing learnable weights slightly degrades the performance, while randomly initializing dynamic calibration weights (by randomly initializing the last layer of the weight generation function) notably degenerates the performance. It is likely that randomly initialized dynamic calibration weights purturb the pre-trained weights more severely than the learnable weights since it is dependent on the input. Further comparisons on the initialization are shown in the Table A7 in the Appendix.
Calibration weight generation function. Established that the temporally adaptive dynamic calibration with appropriate initialization can be an ideal strategy for temporal modelling, we further ablate different ways for generating the calibration weight in Table 4. Linear weight generation function (Lin.) applies a single 1D convolution to generate the calibration weight, while non-linear one (Non-Lin.) uses two stacked 1D convolutions with batch normalizations and ReLU activation in between. When no temporal context is considered (K.=1 or (1,1)), TAdaConv can still improve the baseline but with a limited gap. Enlarging the kernel size to cover the temporal context (K.=3, (1,3), (3,1) or (3,3)) further yields a boost of over 20% on the accuracy, with K.=(3,3) having the strongest performance. This shows the importance of the local temporal context during calibration weight generation. Finally, for the scope of temporal context, introducing global context to frame descriptors performs similarly to only generating temporally adaptive calibration weights solely on the global context (in Table 6). The combination of the global and temporal context yields a better performance for both variants. We further show in Appendix J that this function in our TAdaConv yields a better calibration on the base weight than other existing dynamic filters.
Feature aggregation. The feature aggregation module used in the TAda2D network is ablated in Table 5. First, the performance is similar for plain aggregation x = Avg(x) and aggregation with a shortcut (Sc.) branch x = x + Avg(x), with Sc. being slightly better. Separating the batchnorm (Eq. 7) for the shortcut and the aggregation branch brings notable improvement. Strided max and mix (avg+max) pooling slightly underperform the average pooling variant. Overall, the combination of TAdaConv and our feature aggregation scheme has an advantage over the TSN baseline of 31.8%.
Calibration dimension. Multiple dimensions can be calibrated in the base weight. Table 7 shows that calibrating the channel dimension more suitable than the spatial dimension, which means that the spatial structure of the original convolution kernel should be retained. Within channels, the calibration works better on Cin than Cout or both combined. This is prob-
ably because the calibration weight generated by the input feature can better adapt to itself.
Different stages employing TAdaConv. The solid lines in Fig 3 show the stage by stage replacement of the spatial convolutions in a ResNet model. It has a minimum improvement of 17.55%, when TAdaConv is employed in Res2. Compared to early stages, later stages contribute more to the final performance, as later stages provide more accurate calibration because of its high abstraction level. Overall, TAdaConv is used in all stages for the highest accuracy.
Different proportion of channels calibrated by TAdaConv. Here, we calibrate only a proportion of channels using TAdaConv and leave the other channels uncalibrated. The results are presented as dotted lines in Fig 3. We find TAdaConv can improve the baseline by a large margin even if only 1/64 channels are calibrated, with larger proportion yielding further larger improvements.
Visualizations. We qualitatively evaluate our approach in comparison with the baseline approaches (TSN and R(2+1)D) by presenting the Grad-CAM (Selvaraju et al., 2017) visualizations of the last stage in Fig. 4. TAda2D can more completely spot the key information in videos, thanks to the temporal reasoning based on global spatial information and the global temporal information.

heading: 5.3 MAIN RESULTS
text: SSV2. As shown in Table 8, TAda2D outperforms previous approaches using the same number of frames. Compared to TDN that uses more frames, TAda2D performs competitively. Visualization in Fig. 3(b) also demonstrates the superiority of our performance/efficiency trade-off.
Epic-Kitchens-100. Table 9 lists our results on EK100 in comparison with the previous approaches2. We calculate the final action prediction following the strategies in Huang et al. (2021). For fair comparison, we reimplemented our baseline TSN using the same training and evaluation strategies. TAda2D improves this baseline by 11.64% on the action prediction. Over previous approaches, TAda2D achieves a higher accuracy with a notable margin.
Kinetics-400. Comparison with the state-of-the-art models on Kinetics-400 is presented in Table 10, where we show TAda2D performs competitively against previous approaches. In comparison with the models using the similar backbone and the same number of frames, TAda2D achieves competitive accuracy. The ensemble of TAda2D even achieves a higher accuracy than some of the previous works using more frames, e.g., 77.5 for TDN using 80 frames vs 77.4 for ours using 16 frames.
2The performances are referenced from the official release of the EK100 dataset (Damen et al., 2020).

heading: 6 EXPERIMENTS ON TEMPORAL ACTION LOCALIZATION
text: Dataset, pipeline, and evaluation. Action localization is an essential task for understanding untrimmed videos, whose current pipeline makes it heavily dependent on the quality of the video representations. We evaluate our TAda2D on two large-scale action localization datasets, HACS (Zhao et al., 2019) and Epic-Kitchens-100 (Damen et al., 2020). The general pipeline follows (Damen et al., 2020; Qing et al., 2021a;c). For evaluation, we follow the standard protocol for the respective dataset. We include the details on the training pipeline and the evaluation protocal in the Appendix C.
Main results. Table 11 shows that, compared to the baseline, TAda2D provides a stronger feature for temporal action localization, with an average improvement of over 4% on the average mAP on both datasets. In Appendix H, we further demonstrate the TAdaConv can also improve action localization when used as a plug-in module for existing models.

heading: 7 CONCLUSIONS
text: This work proposes Temproally-Adaptive Convolutions (TAdaConv) for video understanding, which dynamically calibrates the convolution weights for each frame based on its local and global temporal context in a video. TAdaConv shows superior temporal modelling abilities on both action classification and localization tasks, both as stand-alone and plug-in modules for existing models. We hope this work can facilitate further research in video understanding.

heading: A DETAILED ANALYSIS ON TEMPORAL CONVOLUTIONS
text: Here, we provide detailed analysis to showcase the underlying process of temporal modelling by temporal convolutions. As in Sec. 3.1, we use depth-wise temporal convolutions for simplicity and its wide application. We first analyze the case where temporal convolutions are directly placed after spatial convolutions without non-linear activation in between, before activation functions is inserted in the second part of our analysis.
Without activation. We first consider a simple case with no non-linear activation functions between the temporal convolution and the spatial convolution. Given a 3×1×1 depth-wise temporal convolution parameterized by β = [β1,β2,β3], where β1,β2,β3 ∈ RCo , a spatial convolution parameterized by W ∈ RCo×Ci×k2 , the output feature x̃t of the t-th frame can be obtained by: x̃t = β1 · (W ∗ xt−1) + β2 · (W ∗ xt) + β3 · (W ∗ xt+1) , (8) where · denotes element-wise multiplication with broadcasting, and ∗ denotes convolution over the spatial dimension. In this case, β could be grouped with the spatial convolution weight W and the combination of temporal and spatial convolution can be rewritten as Eq. 2 in the manuscript:
x̃t = Wt−1 ∗ xt−1 +Wt ∗ xt +Wt+1 ∗ xt+1 , (2) where Wt−1 = β1 ·W, Wt = β2 ·W and Wt+1 = β3 ·W. This equation share the same form with the Eq. 2 in the manucript. In this case, the combination of temporal convolution with spatial convolution can be certainly viewed as the temporal convolution is simply performing calibration on spatial convolutions before aggregation, with different weights assigned to different time steps for the calibration.
With activation. Next, we consider a case where activation is in between the temporal convolution and spatial convolution. The output feature x̃t are now obtained by Eq. 1 in the manuscript:
x̃t = β1 · δ(W ∗ xt−1) + β2 · δ(W ∗ xt) + β3 · δ(W ∗ xt+1) . (1)
Next, we show that this can be still rewritten in the form of Eq. 2. Here, we consider the case where ReLU (Nair & Hinton, 2010) is used as the activation function, denoted as δ:
δ(x) =
{ x x > 0
0 x ≤ 0 . (9)
Hence, the term δ(W ∗ xt) can be easily expressed as: δ(W ∗ xt) = Mt ·W ∗ xt , (10) where Mt ∈ RC×H×W is a binary map sharing the same shape as xt, indicating whether the corresponding element in W ∗ xt is greater than 0 or not. That is:
M (c,i,j) t = { 1 if (W ∗ xt)(c,i,j) > 0 0 if (W ∗ xt)(c,i,j) ≤ 0 , (11)
where c, i, j are the location index in the tensor. Hence, Eq. 1 can be expressed as: x̃t = β1 ·Mt−1 ·W ∗ xt−1 + β2 ·Mt ·W ∗ xt + β3 ·Mt+1 ·W ∗ xt+1 . (1)
In this case, we can set W(i,j)t−1 = β1 ·M (i,j) t−1 ·W, W (i,j) t = β2 ·M (i,j) t ·W, and W (i,j) t+1 = β3 ·M(i,j)t+1 ·W, where (i, j) indicate the spatial location index. In this case, each filter for a specific time step t is composed of H ×W filters and Eq. 1 can be rewritten as Eq. 2. Interestingly, it can be observed that with ReLU activation function, the convolution weights are different for all spatiotemporal locations, since the binary map M depends on the results of the spatial convolutions.
Table A1: Comparison of different operations for spatial and temporal modeling (Lin et al., 2019a; Tran et al., 2018; Wang et al., 2020; Hara et al., 2018). ’T.’ refers to the temporal modeling ability.
T. Operation Parameters FLOPs 7 Spat. conv Co × Ci × k2 Co × Ci × k2 × THW 3 Temp. conv Co × Ci × k Co × Ci × k × THW 3 Temp. shift Co × Ci × k2 Co × Ci × k2 × THW 3 (2+1)D conv Co × Ci × (k2 + k) Co × Ci × (k2 + k)× THW 3 3D conv Co × Ci × k3 Co × Ci × k3 × THW 3 Correlation Ci × T × k2 Ci × k2 × THW 3 TAdaConv Co × Ci × k2 + 2× Ci × Ci/r × k Co × Ci × k2 × THW + Ci × (THW + T )
+Ci × Ci/r × (2× k × T + 1) + Co × Ci × k2 × T

heading: B COMPUTATIONAL ANALYSIS
text: Consider the input tensor with the shape of Ci × T ×H ×W , where Ci denotes the number of input channels, TAdaConv essentially performs 2D convolution, with weights dynamically generated. Hence, a good proportion of the computation is carried out by the 2D convolutions:
FLOPs(Conv2D) = Co × Ci × k2 × T ×H ×W Params(Conv2D) = Co × Ci × k2 ,
where Co denote the number of output channels, and k denotes the kernel size of the 2D convolutions. For the weight generation, the features are first aggregated by average pooling over the spatial dimension and the temporal dimension, which contains no parameters:
FLOPs(GAPspatial) = Ci × T ×H ×W FLOPs(GAPtemporal) = Ci × T .
For the local information, a two layer 1D convolution with kernel size of kw are applied, with reduction ratio of r in between. For global information, a one layer 1D convolution with kernel size of 1 is applied.
FLOPs(Gen) = 2× Ci × Ci/r × k × T + Ci × Ci/r Params(Gen) = 2× Ci × Ci/r × k + Ci × Ci/r ,
Further, the calibration weight α ∈ RCi×T is multiplied to the kernel weight of 2D convolutions W ∈ RCo×Ci×k2 : FLOPs(Calibration) = Co × Ci × k2 × T , Hence, the overall computation and parameters are:
FLOPs(TAdaConv) =FLOPs(Conv2D) + FLOPs(GAP) + FLOPs(Gen) + FLOPs(Calibration)
=Co × Ci × k2 × THW + Ci × THW + Ci × T + 2× Ci × Ci/r × k × T + Ci × Ci/r + Co × Ci × k2 × T
Params(TAdaConv) =Params(Conv2D) + Params(Gen)
=Co × Ci × k2 + 2× Ci × Ci/r × k + Ci × Ci/r
.
For the FLOPs, despite the overwhelming number of terms, all the other terms are at least an order of magnitude smaller than the first term. This is in contrast to the (2+1)D convolutions for spatiotemporal modelling, where the FLOPs are Co × Ci × (k2 + k) × THW . The extra computation introduced by the temporal convolutions is only k times smaller than the 2D convolutions. Table A1 shows the comparison of computation and parameters with other approaches.

heading: C FURTHER IMPLEMENTATION DETAILS
text: Here, we further describe the implementation details for the action classification and action localization experiments. For fair comparison, we keep all the training strategies the same for our baseline, the plug-in evaluations as well as our own models.
C.1 ACTION CLASSIFICATION
Our experiments on the action classification are conducted on three large-scale datasets. For all action classification models, we train them with synchronized SGD using 16 GPUs. The batch size for each GPU is 16 and 8 respectively for 8-frame and 16-frame models. The weights in TAda2D are initialized using ImageNet (Deng et al., 2009) pre-trained weights (He et al., 2016), except for the calibration function G and the batchnorm statistics (BN2) in the average pooling branch. In the calibration function, we randomly initialize the first convolution layer (for non-linear weight generation) following He et al. (2015), and fill zero to the weight of last convolution layer. The batchnorm statistics are initialized to be zero so that the initial state behaves the same as without the average pooling branch. For all models, we use a dropout ratio (Hinton et al., 2012) of 0.5 before the classification heads. Spatially, we randomly resize the short side of the video to [256, 320] and crop a region of 224×224 to the network in ablation studies, and set the scale to [224, 340] following TANet (Liu et al., 2021) for comparison against the state-of-the-arts. Temporally, we perform interval based sampling for Kinetics-400 and Epic-Kitchens-100, with interval of 8 for 8 frames and interval of 5 for 16 frames. On Something-Something-V2, we perform segment based sampling.
On Kinetics-400, a half-period cosine schedule is applied for decaying the learning rate following Feichtenhofer et al. (2019), with the base learning rate set to 0.24. The models are trained for 100 epochs. In the first 8 epochs, we adopt a linear warm-up strategy starting from a learning rate of 0.01. The weight decay is set to 1e-4. The frames are sampled based on a fixed interval, which is 8 for 8-frame models and 5 for 16-frame models.
On Epic-Kitchens-100, the models are initialized with weights pre-trained on Kinetics-400, and are further fine-tuned following a similar strategy as in kinetics. The training length is reduced to 50 epochs, with 10 epochs for warm-up. The base learning rate is 0.48. Following Damen et al. (2020) and Huang et al. (2021), we connect two separate heads for predicting verbs and nouns. Action predictions are obtained according to the strategies in Huang et al. (2021), which is shown to have a higher accuracy over the original one in Damen et al. (2020). For fair comparison, we also trained and evaluated our baseline using the same strategy.
On Something-Something-V2, we initialize the model with ImageNet pretrained weights. A segment-based sampling strategy is adopted, where for T -frame models, the video is divided into T segments before one frame is sampled from each segment randomly for training or uniformly for evaluation. The models are trained for 64 epochs, with the first 4 being the warm-up epochs. The base learning rate is set to 0.48.
It is worth noting that, for SlowFast models (Feichtenhofer et al., 2019) in the plug-in evaluations, we do not apply precise batch normalization statistics in our implementation as in its open-sourced codes, which is possibly the reason why our re-implemented performance is slightly lower than the original published numbers.
C.2 ACTION LOCALIZATION
We evaluate our model on the action localization task using two large-scale datasets. The overall pipeline for our action localization evaluation is divided into finetuning the classification models, obtaining action proposals and classifying the proposals.
Finetuning. On Epic-Kitchens, we simply use the evaluated action classification model. On HACS, following (Qing et al., 2021c), we initialize the model with Kinetics-400 pre-trained weights and train the model with adamW for 30 epochs (8 warmups) using 32 GPUs. The mini-batch size is 16 videos per GPU. The base learning rate is set to 0.0002, with cosine learning rate decay as in Kinetics. In our case, only the segments with action labels are used for training.
Proposal generation. For the action proposals, a boundary matching network (BMN) (Lin et al., 2019b) is trained over the extracted features on the two datasets. On Epic-Kitchens, we extract features with the videos uniformly decoded at 60 FPS. For each clip, we use 8 frames with an interval of 8 to be consistent with finetuning, which means a feature roughly covers a video clip of one seconds. The interval between each clip for feature extraction is 8 frames (i.e., 0.133 sec) as well. The shorter side of the video is resized to 224 and we feed the whole spatial region into the backbone to retain as much information as possible. Following Qing et al. (2021a), we generate proposals
Table A2: Model structure of R3D, R(2+1)D and R2D that we used in our experiments. Blue and green fonts indicate respectively the default convolution operation and optional operation that can be replaced by TAdaConv. (Better viewed in color.)
Stage R3D R(2+1)D R2D (default baseline) output sizes sampling interval 8, 12 interval 8, 12 interval 8, 12 8×224×224
conv1 3×72, 64 1×72, 64 1×72, 64 8×112×112
stride 1, 22 stride 1, 22 stride 1, 22
res2
 1×12, 643×32, 64 1×12, 256 ×3  1×12, 64 1×32, 64 3×12,64
1×12, 256
×3  1×12, 641×32, 64
1×12, 256
×3 8×56×56
res3
 1×12, 1283×32, 128 1×12, 512 ×4  1×12, 128 1×32, 128 3×12,128 1×12, 512 ×4  1×12, 1281×32, 128 1×12, 512 ×4 8×28×28
res4
 1×12, 2563×32, 256 1×12, 1024 ×6  1×12, 256 1×32, 256 3×12,256
1×12, 1024
×6  1×12, 2561×32, 256
1×12, 1024
×6 8×14×14
res5
 1×12, 5123×32, 512 1×12, 2048 ×3  1×12, 512 1×32, 512 3×12,512
1×12, 2048
×3  1×12, 5121×32, 512
1×12, 2048 ×3 8×7×7 global average pool, fc 1×1×1
using BMN based on sliding windows. The predictions on the overlapped region of different sliding windows are simply averaged. On HACS, the videos are decoded at 30 FPS, and extend the interval between clips to be 16 (i.e., 0.533 sec) because the actions in HACS last much longer than in EpicKitchens. The shorter side is resized to 128 for efficient processing. For the settings in generating proposals, we mainly follow Qing et al. (2021c), except that the temporal resolution is resized to 100 in our case instead of 200.
Classification. On Epic-Kitchens, we classify the proposals with the fine-tuned model using 6 clips. Spatially, to comply with the feature extraction process, we resize the shorter side to 224 and feed the whole spatial region to the model for classification. On HACS, considering the property of the dataset that only one action category can exist in a video, we obtain the video level classification results by classifying the video level features, following Qing et al. (2021c).
Evaluation. For evaluation, we follow the standard evaluation protocol used in the respective datasets, i.e., the average mean Average Precision (average mAP) at IoU threshold [0.5:0.05:0.95] for HACS (Zhao et al., 2019) and [0.1:0.1:0.5] for Epic-Kitchens-100 (Damen et al., 2020).

heading: D MODEL STRUCTURES
text: The detailed model structures for R2D, R(2+1)D and R3D is specified in Table A2. We highlight the convolutions that are replaced by TAdaConv by default or optionally. For all of our models, a small modification is made in that we remove the max pooling layer after the first convolution and set the spatial stride of the second stage to be 2, following Wang et al. (2020). Temporal resolution is kept unchanged following recent works (Feichtenhofer et al., 2019; Li et al., 2020b; Jiang et al., 2019). Our R3D is obtained by simply expanding the R2D baseline in the temporal dimension by a factor of three. We initialize with weights reduced by 3 times, which means the original weight is evenly distributed in adjacent time steps. We construct the R(2+1)D by adding a temporal convolution operation after the spatial convolution. The temporal convolution can also be optionally replaced by TAdaConv, as shown in Table 3 and Table A3. For its initialization, the temporal convolution weights are randomly initialized, while the others are initialized with the pre-trained weights on ImageNet. For SlowFast models, we keep all the model structures identical to the original work (Feichtenhofer et al., 2019).
Class Index 0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170
0
20
40
60
80
A cc
ur ac
y
TSN Improvement of TAda2D
Figure A1: Per-category performance comparison of TAda2D against the baseline TSN. We achieve an average per-category performance improvement of 30.35%.
Table A3: Comparison with the state-of-the-art approaches over action classification on EpicKitchens-100 (Damen et al., 2020). ↑ indicates the main evaluation metric for the dataset. For fair comparison, we implement all the baseline models using our own training strategies.
Top-1 Top-5
Model Frames GFLOPs Params. Act.↑ Verb Noun Act.↑ Verb Noun SlowFast 4×16 4+32 36.10 34.5M 38.17 63.54 48.79 58.68 89.75 73.37 SlowFast 4×16 + TAdaConv 4+32 36.11 37.7M 39.14 64.50 49.59 59.21 89.67 73.88 SlowFast 8×8 8+32 65.71 34.5M 40.08 65.05 50.72 60.10 90.04 74.26 SlowFast 8×8 + TAdaConv 8+32 65.73 37.7M 41.35 66.36 52.32 61.68 90.59 75.89 R(2+1)D 8 49.55 28.1M 37.45 62.92 48.27 58.02 89.75 73.60 R(2+1)D + TAdaConv2d 8 49.57 30.0M 39.72 64.48 50.26 60.22 90.01 75.06 R(2+1)D + TAdaConv2d+1d 8 49.58 31.9M 40.10 64.77 50.28 60.45 89.99 75.55 R3D 8 84.23 46.5M 36.67 61.92 47.87 57.47 89.02 73.05 R3D + TAdaConv3d 8 84.24 48.5M 39.30 64.03 49.94 59.67 89.84 74.56

heading: E PER-CATEGORY IMPROVEMENT ANALYSIS ON SSV2
text: This section provides a per-category improvement analysis on the Something-Something-V2 dataset in Fig.A1. As shown in Table 5, our TAda2D achieves an overall improvement of 31.7%. Our per-category analysis shows an mean improvement of 30.35% over all the classes. The largest improvement is observed in class 0 (78.5%, Approaching something with your camera), 32 (78.4%, Moving away from something with your camera), 30 (74.3%, Lifting up one end of something without letting it drop down), 44 (66.2%, Moving something towards the camera) and 41 (66.1%, Moving something away from the camera). Most of these categories contain large movements across the whole video, whose improvement benefits from temporal reasoning over the global spatial context. For class 30, most of its actions lasts a long time (as it needs to be determined whether the end of something is let down or not). The improvements over the baseline mostly benefits from the global temporal context that are included in the weight generation process.

heading: F FURTHER ABLATION STUDIES
text: Here we provide further ablation studies on the kernel size in the calibration weight generation. As shown in Table A4a and Table A4b, kernel size does not affect the classification much, as long as the temporal context is considered. Further, Table A4c shows the sensitivity analysis on the reduction ratio, which demonstrate the robustness of our approach against different set of hyper-parameters.

heading: G FURTHER PLUG-IN EVALUATION FOR TADACONV ON CLASSIFICATION
text: In complement to Table 3, we further show in Table A3 the plug-in evaluation on the action classification task on the Epic-Kitchens-100 dataset. As in the plug-in evaluation on Kinetics and Something-Something-V2, we compare performances with and without TAdaConv over three baseline models, SlowFast (Feichtenhofer et al., 2019), R(2+1)D (Tran et al., 2018) and R3D (Hara
Table A4: Ablation studies. (a) Ablation studies on kernel size with linear calibration weight generation function.
Kernel size Top-1 1 37.5 3 56.5 5 57.3 7 56.5
(b) Ablation studies on kernel size with non-linear calibration weight generation function.
K2=1 K2=3 K2=5 K2=7 K1=1 36.8 57.1 57.8 57.9 K1=3 57.3 57.8 57.9 58.0 K1=5 57.6 57.9 58.2 57.9 K1=7 57.4 57.6 58.0 57.6
(c) Ablation studies on reduction ratio r for K1 = K2 = 3.
Ratio r Top-1 1 57.79 2 57.83 4 57.78 8 57.66
Table A5: Plug-in evaluation of TAdaConv on the action localization on HACS and Epic-Kitchens. ↑ indicates the main evaluation metric for the dataset. ‘S.F.’ is SlowFast network.
HACS Epic-Kitchen-100
Model @0.5 @0.6 @0.7 @0.8 @0.9 Avg.↑ Task @0.1 @0.2 @0.3 @0.4 @0.5 Avg.↑ S.F. 8×8 50.0 44.1 37.7 29.6 18.4 33.7 Verb 19.93 18.92 17.90 16.08 13.24 17.21 Noun 17.93 16.83 15.53 13.68 11.41 15.07 Act.↑ 14.00 13.19 12.37 11.18 9.52 12.04 S.F. 8×8 + TAdaConv 51.7 45.7 39.3 31.0 19.5 35.1 Verb 19.96 18.71 17.65 15.41 13.35 17.01 Noun 20.17 18.90 17.58 15.83 13.18 17.13 Act.↑ 14.90 14.12 13.32 12.07 10.57 13.00
et al., 2018) respectively representing three kinds of temporal modeling techniques. The results are in line with our observation in Table 3. Over all three kinds of temporal modelling strategies, adding TAdaConv further improves the recognition accuracy of the model.

heading: H PLUG-IN EVALUATION FOR TADACONV ON ACTION LOCALIZATION
text: Here, we show the plug-in evaluation on the temporal action localization task. Specifically, we use SlowFast as our baseline, as it is shown to be superior in the localization performance in Qing et al. (2021b) compared to many early backbones. The result is presented in Table A5. With TAdaConv, the average mAP on HACS is improved by 1.4%, and the average mAP on Epic-Kitchens-100 action localization is improved by 1.0%.

heading: I COMPARISON OF TRAINING PROCEDURE
text: In this section, we compare the training procedure of TSN and TAda2D on Kinetics-400 and Something-Something-V2. The results are presented in Fig. A2. TAda2D demonstrates a stronger fitting ability and generality even from the early stages of the training, despite that the initial state of TAda2D is identical to that of TSN.

heading: J COMPARISON WITH EXISTING DYNAMIC FILTERS
text: In this section, we compare our TAdaConv with previous dynamic filters in two perspectives, respectively the difference in the methodology and in the performance.
J.1 COMPARISON IN THE METHODOLOGY LEVEL
For the former comparison, we include Table A6 to show the differences in different approaches, which is a full version of Table 1. We compare TAdaConv with several representative approaches in image and in videos, respectively CondConv (Yang et al., 2019), DynamicFilter (Jia et al., 2016), DDF (Zhou et al., 2021) and TAM (Liu et al., 2021).
The first difference in the methodology level lies in the source of weights, where previous approaches obtain weights by mixture of experts or generation completely dependent on the input. Mixture of
Epochs 0 20 40 60 80 100
90
80
70
60
50
40
30
20
0 10 20 30 40 50 60 Epochs
90
80
70
60
50
40
30
20
10
To p1
E rr
or (%
)
To p1
E rr
or (%
)
TSN (training) TSN (validation) TAda2D (training) TAda2D (validation) TSN (training) TSN (validation) TAda2D (training) TAda2D (validation)
Kinetics 400 Something-Something-V2
Figure A2: Training and validation on Kinetics-400 and Something-Something-V2. On both datasets, TAda2D shows a stronger capability of fitting the data and a better generality to the validation set. Further, TAda2D reduces the overfitting problem in Something-Something-V2.
Table A6: Approach comparison between different dynamic filters. The weights column denotes how weights in respective approaches are obtained. The pre-trained weights colmun shows whether the weight generation can exploit pre-trained models such as ResNet (He et al., 2016).
Temporal Location Pretrained Operations Weights Modelling Adaptive weights CondConv Mixture of experts W = ∑ n f(x)nWn 7 7 7 DynamicFilter Completely generated W = g(x) 7 7 7 DDF Completely generated W = g(x) 7 3 7 TAM Completely generated W = g(x) 3 7 7 TAdaConv Calibrated from a base weight W = h(x)Wb 3 3 3
experts denotes W = ∑
n αnWn, where αn is a scalar obtained by a function f , i.e., W =∑ n f(x)nWn. Completely generated means the weights are only dependent on the input, i.e., W = g((x)), where g generates complete kernel for the convolution. In comparison, the weights in TAdaConv are obtained by calibration, i.e,, W = αWb, where α is a vector calibration weight and α = h((x)) where h(.) generates the calibration vector for the convolutions. Hence, this fundamental difference in how to obtain the convolution weights makes the previous approaches difficult to exploit pre-trained weights, while TAdaConv can easily load pre-trained weights in Wb. This ability is essential for video models to speed up the convergence.
The second difference lies in the ability to perform temporal modelling. The ability to perform temporal modelling does not only mean the ability to generate weights according to the whole sequence in dynamic filters for videos, but it also requires the model to generate different weights for the same set of frames with different orders. For example, weights generated by the global descriptor obtained by global average pooling over the whole video GAPst does not have the temporal modelling ability, since they can not generate different weights if the order of the frames in the input sequence are reversed or randomized. Hence, most image based approaches based on global descriptor vectors (such as CondConv and DynamicFilter) or based on adjacent spatial contents (DDF) can not achieve temporal modelling. TAM generates convolution weights for temporal convolutions based on temporally local descriptors obtained by the global average pooling over the spatial dimension GAPs, which yields different weights if the sequence changes. Hence, in this sense, TAM has the temporal modelling abilities. In contrast, TAdaConv exploits both temporally local and global descriptors to utilize not only local but also global temporal contexts. Details on the source of the weight generation process is also shown in Table A7.
The third difference lies in whether the weights generatd are shared for different locations. For CondConv, DynamicFilter and TAM, their generated weights are shared for all locations, while for DDF, the weights are varied according to spatial locations. In comparison, TAdaConv generate temporally adaptive weights.
Table A7: Performance comparison with other dynamic filters. Our Init. denotes initializing the calibration weights to ones so that the initial calibrated weights is identical to the pre-trained weights. Temp. Varying is short for temporally varying, which indicates different weights for different temporal locations (frames). * denotes that the branch was originally not designed for generating filter or calibration weights, but we slightly modified the structure so that it can be used for calibration weight generation. (Numbers in brackets) show the performance improvement brought by our initialization scheme for calibration weights.
Calibration Generation Our Init. Temp. Varying Generation source Top-1 DynamicFilter 7 7 GAPst(x)(C × 1) 41.7 DDF 7 3 GAPst(x)(C × 1) 49.8 TAM (global branch) 7 7 GAPs(x)(C × T ) 39.7 TAM (local*+global branch) 7 3 GAPs(x)(C × T ) 41.3 DynamicFilter 3 7 GAPst(x)(C × 1) 51.2 (+9.5) DDF 3 3 GAPst(x)(C × 1) 53.8 (+4.0) TAM (global branch) 3 7 GAPs(x)(C × T ) 52.9 (+13.2) TAM (local*+global branch) 3 3 GAPs(x)(C × T ) 54.3 (+13.0) TAdaConv w/o global info g 3 3 GAPs(x)(C × T ) 57.9 TAdaConv 3 3 both GAPst(x)(C × 1) 59.2and GAPs(x)(C × T )
J.2 COMPARISON IN THE PERFORMANCE LEVEL
Since TAdaConv is fundamentally different from previous approaches in the generation of calibration weights, it is difficult to directly compare the performance on video modelling, especially for those that are not designed for video modelling. However, since the calibration weight in TAdaConv α is completely generated, i.e., α = f((x)), we can use other dynamic filters to generate the calibration weights for TAdaConv. Since MoE based approaches such as CondConv were essentially designed for applications with less memory constraint but high computation requirements, it is not suitable for video applications since it would be too memory-heavy for video models. Hence, we apply approaches that generate complete kernel weights to generate calibration weights, and compare them with TAdaConv. The performance is listed in Table A7.
It is worth noting that these approaches originally generate weights that are randomly initialized. However, as is shown in Table 6, our initialization strategy for the calibration weights are essential for yielding reasonable results, we further apply our initialization on these existing approaches to see whether their generation function is better than the one in TAdaConv. In the following paragraphs, we provide details for applying representative previous dynamic filters in TAdaConv to generate the calibration weight.
For DynamicFilter (Jia et al., 2016), the calibration weight α is generated using an MLP over the global descriptor that is obtained by performing global average pooling over the whole input GAPst, i.e., α = MLP(GAPst(x)). In this case, the calibration weights are shared between different time steps.
For DDF (Zhou et al., 2021), we only use the channel branch since it is shown in Table 7 that it is better to leave the spatial structure unchanged for the base kernel. Similarly, the weights in DDF are also generated by applying an MLP over the global descriptor, i.e., α = MLP(GAPst(x)). The difference between DDF and DynamicFilter is that for different time step, DDF generates a different calibration weight.
The original structure of TAM (Liu et al., 2021) only generates kernel weights with its global branch, and uses local branch to generate attention maps over different time steps. In our experiments, we modify the TAM a little bit and further make the local branch to generate kernel calibration weights as well. Hence, for only-global version of TAM, the calibration weights are calculated as follows: α = G(GAPs(x)), where GAPs denotes global average pooling over the spatial dimension and G denotes the global branch in TAM. In this case, calibration weights are shared for all temporal locations. For local+global version of TAM, the calibration weight are calculated by combining the results of the local L and the global branch G, i.e., α = G(GAPs(x)) ·L(GAPs(x)), where · denotes element-wise multiplication with broadcasting. This means in this case, the calibration weights are temporally adaptive. Note that this is our modified version of TAM. The original TAM does not have a temporally adaptive convolution weights.
The results in Table A7 show that (a) without our initialization strategy, previous approaches that generate random weights at initialization are not suitable for generating the calibration weights in TAdaConv; (b) our initialization strategy can conveniently change this and make previous approaches yield reasonable performance when they are used for generating calibration weights; and (c) the calibration weight generation function in TAdaConv, which combines the local and global context, outperform all previous approaches for calibration.
Further, when we compare TAdaConv without global information with TAM (local*+global branch), it can be seen that although both approach generates temporally varying weights from the frame descriptors GAPs(x) with shape C × T , our TAdaConv achieves a notably higer performance. Adding the global information enables TAdaConv to achieve a more notable lead in the comparison with previous dynamic filters.
GT
Pretending to throw sth.
TSN
Squeezing sth.
R(2+1)D
Turning sth. upside down
TAda2D
Pretending to throw sth.
GT
Putting sth. underneath sth.
TSN
Turning the camera downwards
when filming sth
R(2+1)D
Putting sth. into sth.
TAda2D
Putting sth. underneath sth.
GT
Pulling sth. onto sth.
TSN
Moving sth. and sth. so that they pass
each other
R(2+1)D
Putting sth. onto sth.
TAda2D
Pulling sth. onto sth.
GT
Putting sth. behind sth.
TSN
Pretending to put sth. behind sth.
R(2+1)D
Putting sth. behind sth.
TAda2D
Putting sth. behind sth.
Figure A3: Further qualitative evaluations on the Something-Something-V2 dataset. In most cases, TAda2D captures meaningful areas in the videos for the correct classification. Further, the activated region of TAda2D also lasts longer along the temporal dimension compared to other two models, thanks to the global temproal context in the weight generation function G.

Review summaries:
This paper proposes TAdaConv that calibrates kernel weights of convolutional layers adaptively according to the temporal dynamics of the input tensor. It is designed to incorporate both the local and global temporal context by using stacked two-layer 1D convolutional operations and global average pooling. Since it works exactly the same as the original convolutional layer at the initial stage, it is easy to insert TAdaConv into existing ConvNet architectures. On top of this, the authors construct TAda2D networks by introducing the temporal feature aggregation module that is based on a strided temporal average pooling. Many experiments performed on video/action classification and localization demonstrate the effectiveness of the proposed module.

This work seeks to improve performance of video understanding models though the use of spatial convolutions which are dynamically adapted in time when applied to a sequence of frames. The novelty of this work lies in the fact that the weights of the spatial convolutions are adapted, rather than applying a temporal excitation to feature maps to achieve the same purpose. To achieve this, a drop-in replacement for spatial convolutions is designed which uses pooled local and global frame descriptors to produce the temporal calibration weights. Furthermore, this operation is used as part of a temporal feature aggregation block that is used as part of 2D CNN architecture. The benefits of the approach include that it can make effective use of pre-trained weights and that it can improve model capacity for a marginal increase in computation. Experimentally, it is show large performance improvements over a baseline and match state of the art on significant datasets.

The authors proposed a novel approach for video understanding. They present temporally adaptive convolutions (TAdaConv) based on dynamic networks by adapting the convolutional weights on each frame with its temporal contexts. The experiment results on public benchmarks demonstrate the efficiency of the proposed approach.

This paper proposes a modified version of convolution for videos where a 2D conv kernel is converted to a 3D conv kernel by multiplying the 2d weights by a per-frame value. This creates a set of weights for 3D convolution that are dynamic for each frame. The approach is evaluated on multiple video datasets, and shows some benefit over their baseline.