Low-Fidelity Video Encoder Optimization for Temporal Action Localization
Authors: Mengmeng Xu
Juan-Manuel Perez-Rua
Xiatian Zhu
Bernard Ghanem
Brais Martinez
Venue: NeurIPS 2021 Poster (2021)
Keywords: End-to-End Pre-training
Temporal Action Localization

Abstract: Most existing temporal action localization (TAL) methods rely on a transfer learning pipeline: by first optimizing a video encoder on a large action classification dataset (i.e., source domain), followed by freezing the encoder and training a TAL head on the action localization dataset (i.e., target domain). This results in a task discrepancy problem for the video encoder – trained for action classification, but used for TAL. Intuitively, joint optimization with both the video encoder and TAL head is a strong baseline solution to this discrepancy. However, this is not operable for TAL subject to the GPU memory constraints, due to the prohibitive computational cost in processing long untrimmed videos. In this paper, we resolve this challenge by introducing a novel low-fidelity (LoFi) video encoder optimization method. Instead of always using the full training configurations in TAL learning, we propose to reduce the mini-batch composition in terms of temporal, spatial, or spatio-temporal resolution so that jointly optimizing the video encoder and TAL head becomes operable under the same memory conditions of a mid-range hardware budget. Crucially, this enables the gradients to flow backwards through the video encoder conditioned on a TAL supervision loss, favourably solving the task discrepancy problem and providing more effective feature representations. Extensive experiments show that the proposed LoFi optimization approach can significantly enhance the performance of existing TAL methods. Encouragingly, even with a lightweight ResNet18 based video encoder in a single RGB stream, our method surpasses two-stream (RGB + optical-flow) ResNet50 based alternatives, often by a good margin. Our code is publicly available at https://github.com/saic-fi/lofi_action_localization.

Low-Fidelity Video Encoder Optimization for Temporal Action Localization
Mengmeng Xu
Juan-Manuel Pérez-Rúa'
Xiatian Zhu'
mengmeng, xu@kaust. edu. sa
PerezRua. PerezRua.JNogmail JM@gmail com
xiatian zwu@samsung com
Bernard Ghanem
Brais Martinez! brais. a@samsung............... com
bernard, ghanem@kaust edu sa
Samsung Al Centre Cambridge, UK
King Abdullah University of Science and Technology, Saudi Arabia
Abstract
Most existing temporal action localization (TAL) methods rely transfer learn- ing pipeline, first optimizing a video encoder on a large action classification dataset (i.e., source domain), followed by freezing the encoder and training TAL head on the action localization dataset (i.e., target domain). This results in task dis- crepancy problem for the video encoder trained for action classification, but used for TAL. Intuitively, joint optimization with both the video encoder and TAL head is obvious solution this discrepancy. However, this is not operable for TAL subject to the GPU memory constraints, due to the prohibitive computational cost in processing long untrimmed videos. In this paper, we resolve this challenge by introducing a novel low-fidelity (LoFi) video encoder optimization method. Instead always using the full training configurations in TAL learning, we propose reduce the mini-batch composition in terms of temporal, spatial or spatio tetitoaateprraal resolution so that jointly optimizing the video encoder and TAL. head becomes oper- able under the same memory conditions of a mid-range hardware budget. Crucially, this enables the gradients flow backwards through the video encoder conditioned TAL supervision loss, favourably solving the task discrepancy problem and providing more effective feature representations. Extensive experiments show that the proposed LoF' optimization approach can significantly enhance the perfor- mance existing TAL methods. Encouragingly. even with a lightweight ResNet11 based video encoder in single RGB stream, our method surpasses two-stream (RGB optical flow) ResNet50 based alternatives, often by a good margin. Our code is publicly available at httpss//github.com/siic
action Jocalizatoniiinnnn
Introduction
Video analysis has become an increasingly important area of research, encompassing multiple relevant problems such action recognition 88
[3] temporal action localization [68] 7 12
22
59
60
55 67
[3], video grounding 36 65| 64
66
49 16 24), and video question answering 25
31] Among those, 22 is fundamental task, as natural videos not temporally
temporal action localization (TAL)
688
trimmed. Given an untrimmed video, TAL aims to identify the start and end points of all action instances and recognize their category labels simultaneously. A typical TAL model is based on deep convolutional neural networks (CNNs) composed of two modules: a video encoder and a TAL head.
The video encoder is often shared across different TAL methods (e.g., G-TAD 660 BC-GNN
Work done during intersship at Samsung Al Centre.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

by taking specific off shellf action classification model (e.g., C3D [51] 13D 8 TSM [33] with the differences residing only in the TAL head. However, instead of short (e.g. 10 seconds) trimmed video clips as in action recognition, the input videos to a TAL model are characterized by much longer temporal duration (e.g., 120 seconds). This causes unique computational challenges that remain unsolved, particularly in model optimization.
In standard optimization of a TAL model, a two-stage transfer learning pipeline often involved:
1. First, the video encoder is optimized a large source video classification dataset (e.g., Kinetics (28) and, optionally, finetunned on the trimmed version of the target dataset under action classification supervision;
2. Second, the video encoder is frozen and the TAL head is optimized on the target action localization dataset (e.g., ActivityNet [22 HACS (68) under TAL. task supervision.
With this widely-used TAL training pipeline, the video encoder is only optimal for action classification but not for the target TAL task. Specifically, the video encoder is trained so that different short segments within action sequence are mapped to similar outputs, thus encouraging insensitivity to the temporal boundaries of actions. This is not desirable for a TAL model. We identify this as a task discrepancy problem. Consequently, the final TAL model could suffer from suboptimal performance.
Indeed, jointly optimizing all components of CNN architecture end-to-end with the target task's supervision is a common practice, e-g., training models for object detection in static images [8] 46 38 Unfortunately, this turns out to be non-trivial for TAL.. As mentioned above, model training is severely restricted by the large input size of untrimmed videos and subject to the memory constraint of GPUs. This is why the two-stage optimization pipeline as described above becomes the most common and feasible choice in practice for optimizing a TAL model. On the other hand, existing transfer learning methods mostly focus on tackling the data distribution shift problem across different datasets 170 [5).. rather than the task shift problem we study here. Regardless, believe that solving this limitation of the TAL training design bears a great potential for improving model performance.
In this work, we present simple yet effective low-fieelity (LoFi) video encoder optimization method particularly designed for better TAL. model training. It designed to adapt the video encoder from action classification to TAL. whilst subject to the same hardware budget. This is achieved by introducing simple strategy characterized by a new intermediate training stage where both the video encoder and the TAL head are optimized end-to-end using a lower temporal and/or spatial resolution (i.e., fidelityy in the mini-batch construction. Compared to the standard training method, our proposed strategy does not increase the GPU memory standard (often hard constraint for many practitioners). Crucially, with our LoFi training the gradients back-propas propapre to the video encoder from temporal action localization loss whilst conditioned on the target TAL head, enabling the learning of a video encoder sensitive to the temporal localization objective.
We make the following contributions in this work. (1) We investigate the limitations of the standard optimization method for TAL. models, and consider that the task discrepancy problem hinders the performance of existing TAL. models. Despite it being a significant ingredient, video encoder optimization is largely ignored by existing TAL methods, left without systematic investigation. (2) To improve the training of TAL models, we present novel, simple, and effective low-fidelity (LoFi) video encoder optimization method. It is designed specifically to address the task discrepancy problem with the TAL model's video encoder. (3) Extensive experiments show that the proposed LoFi optimization method yields new state-of-the models (e.g., G-TAD (6)). Critically, method achieves superior efficiency/accuracy trade-off with clear inference cost advantage and good generalizability to auyyingccapalie video encoders.
performance when combined with off- she--f TAL
2 Related Work
Temporal action localization (TAL) models: TAL models can be grouped by architectural design pattern into two categories, one-stage and two-stage architectures. One-stage methods, either predict temporal action boundaries or generate proposals, and classify them within the same network [3] The latter type, two-stage models, generate sets of action proposals (e.g., and then an auxiliary head used for classification of each proposal
9. 21 40
58
60
62
segments) 6
23
39
into an action class
35
47
69]. In this work, rather than introducing novel model design, 48
63

---

we focus on the training of generic TAL models, with particular aim to improve the video encoder optimization. This is a relatively less investigated aspect in the TAL literature.
Video encoders in TAL: The video encoder is an indispensable part of a TAL model. Main design choices include the base architecture of video encoder and its optimization procedure. With regards to the architecture, the two-stream Temporal Segment Network (TSN) [54] is one of the most common Concretely, these works use two TSN backbone trained RGB video frames, and the other with BN-Inception backbone 26 trained on optical flow. Other alternatives used as a video encoder for
video encoders in existing TAL. methods 3
34
35
60)-
networks, with a ResNet50 20
TAL include two-stream I3D model
g
(see [19 63 and Pscudo-3D 45
(see (40)
In terms of optimization, a typical paradigm is two-staged: first pre-training the video encoder and then, in second stage, training the TAL head of the model with the video encoder fixed. This constrained by the inherent hardware budget derived from having a large per-video imput size. In particular, the video encoder is pre-trained using cross- cross-entropy entropy loss for action recognition on a large-scale video classification dataset such as Kinetics (28 pre-train it the foreground segments of the target TAL dataset 34
64]. An optional step is to further
60
44 This brings I1
mismatch between training and inference for the video encoder, which we call a task discrepancy problem. More specifically, although trained to distinguish the content of different action classes, the
video encoder is less sensitive to action temporal boundaries and thus less effective for the TAL task. In fact, due to their inherent design, CNNs have limited localization capabilities [4]], unless they are augmented with specialized localization-speciin task focuses only on the foreground content whilst ignoring per-class background segments, including the transition between foreground and background. In this paper we propose novel low-fidelity video encoder pre- pre-training method to solve this limitation with existing TAL methods.
layers (37). Additionally, the action classification
While current TAL literature mostly relies on pre-training through supervised learning, the rapid advancement of self-supervised learning makes it a promising alternative 2 42
43 [66.. Some
works have focused on finding effective temporal-related pretext tasks, from frame ordering learnt through triplets of frames < to sorting the frames of sequence (30, distinguishing whether sequences are played forward or backwards [56] or through playback speed-related pretext tasks [4] These methods exploit video-specific characteristics force the network to focus on some sort of semantic content within video, inducing representations capturing long-term temporal semantic relations, but force invariance to or ignore the relative positioning of the snippets within the action instances. They are thus not suited for pre-training the video encoder of a TAL model.
61 53 27.
Very recent works 1 59 have exploited some of the aforementioned techniques for better pre-training of action localization models. For example, localization-tailoee data augmentation and classification is adopted by [59]. However, these works introduce large amount of extra video data and additional stream networks, both of which are expensive in terms of memory and computation. In contrast, our method aims improve TAL modelling directly without the need for learning from extra training video data and using an expensive second network, nor relying on optical flow obtained at high computational cost.
Method
TAL model takes as input long untrimmed video with a varying number of frames. For design convenience, it is typical to represent varying-length video by decomposing it into a fixed-length sequence of snippets. The definition of a snippet is the same as in action recognition, where first a number of consecutive frames (e.g., 64) is selected and then sub-sampled with stride r (e.g., stride to obtain 8-frame snippets). To represent a snippet, one first applies a video encoder to extract fraee-level feature vectors and then averages them to obtain the snippet-level feature representation 6 12 17 35). The resulting smippet feature sequence is denoted as X E CxL, where C' is the feature dimension of each snippet, and L is the number of snippets.
In the training set, each video associated with its ground truth, consisting of set of action instance annotations V. In particular, each action instance is represented as segment, each including the start time, the end time, and the action class label. The objective to train TAL model that can accurately localize all the target action instances in given untrimmed video. To that end, the model predicts
3

---

LoFi TAL
fewer clips
VideO
fidelity
clip classification pre-tramning
T-LoFi TAL optimmzation
backgropagation
S-Lofi sample
smaller ciipe
Tewer,
shared backbone weights
smaller
fidelity training
Video
video
cleesifier
hoad
S-LoFi TAL optimization
ST-LoFi TAL opiiiizaiin
Schematic overview of the proposed TAL model training procedure. Three stages involved during model training: (1) Pre-training the video encoder under action classification task's supervision on an auxiliary video dataset (e.g., Kinetics (28)): (2) Low-fidelity (low mini-batch configuration in the spatial, temporal, spatio-temporal resolution of training videos) optimization of the video encoder together with the TAL head under TAL task's supervision on the target dataset; This is the key stage introduced in this paper for resolving the task discrepancy problem without memory overhead increase. (3) Training the TAL head in the full fidelity configuration under TAL task's 's supervision on the target dataset.
Figure 1:
varying number of action instances 4 each comprised of the predicted temporal boundaries, action class, and confidence score.
3.1 Model Training Procedure
Our training procedure consists of three stages as depicted Figure (1) First, we pre-train the video encoder by action classification supervision on large video dataset (e.g.. Kinetics (28) (2) Second, we conduct low-fidelity (LoFi) optimization the video encoder with TAL head the target dataser (Sec. 3.2). This is under TAL supervision with the objective loss function derived from the ground- truth and model prediction Φ. To this end, we propose to reduce the mini-batch configuration in terms of the spatial and temporal resolution of the input otherwise end-to-end optimization cannot satisfy the hardware constraints. Crucially, by training on the target task and target dataset, the task discrepancy gap be reduced. (3) Last, freeze the already end-to-end optimized video encoder and train the TAL head choice from scratch on the target dataset at full spatial and temporal resolutions. Note that in this setup, we cannot perform end-to-end optimization of the final TAL model, limited by the hardware memory constraint. Next, we will detail the proposed LoFi training.
3.2 Low-Fidelity Training Configurations
Formally, we define the full fidelity configuration mini-batch as
B2 L H W,
where L specifies the temporal resolution, and H W refers the 2D spatial resolution. Under the full fidelity regime and with a certain memory size constraint, only the TAL head can be trained whilst leaving the video encoder frozen. In order to enable the video encoder be optimized end-to-end together with the TAL head under the TAL task's supervision, we design four low-fidelity configurations for the mini-bate
(I) Spatial Low-Fidelity (S-LoFi) In the first configuration, we lower the spatial resolution the input videos by factor of r, in both spatial dimensions as:
Ω L x (H/r.) x (W/ra)
With smaller spatial feature maps, this could effectively reduce the memory consumption of the video encoder's 's feature maps which in turn creates space to enable the learning of the video encoder.

---

(II) Temporal Low-Fidelity (T-LoFi) In the second configuration, we instead consider temporal resolution reduction in the following form:
Ω - (L/ri) x H H W.
where r; > 1 is the scaling factor. This corresponds to a smaller number of snippets being taken as input by the TAL head, finally outputting less predictions in case the candidates per time location remains the same. This reduces the memory demands of both the video encoder and the temporal localization head.
(III) Temporal Low-Fidelity (ST-LoFi) In the third configuration, we apply a (typically smaller) reduction in both temporal and spatial resolutions concurrently, formulated as:
Ωgt = (I//ra) x (H/r.) x (W/rs)
In this setup, memory saving can be shared between the temporal and the spatial dimensions.
(TV) Cyclic Low-Fidelity (C-I.oFi) Each of the above three LoFi configurations is used in isolation. To further explore their complementary benefits, we propose apply all of them in structured fashion. To that end, we take inspiration from the recently proposed multi-grid training strategy (57], originally designed for speeding-up up action recognition training. Specifically, we form sampling grid with the three LoFi configurations and cycle through them repeatedly. We integrate both the short and long cycle strategies [57]. In particular, the long cycle changes the values of rtz and r, at the beginning of an epoch, and cycling through the configurations I to III after oj epochs. Instead, the short cycle also cycles through configurations I III. but changes r, and ra after e, batches instead.
is important to note some practical differences with the original multri-grin -rid training. First, contrary to the original multi-grid method, shift the reduction in input resolution between the temporal and the spatial dimensions so that the per-video memory usage remains (approximately) constant. Thus, the batch size also remains constant throughout, which simplifies training. Second, the full resolution setting (i.e. the full fidelity) is not used at any time during the video encoder training.
We compare the above four configurations in Section 4.3 By default, we use the C-LoFi configuration with the long cycle strategy (Long C-LoFi).
3.3 An Instantiation of LoFi
Off-the-shelf TAL model: Without loss of generality, in this study we adopt G-TAD 60) a state- of-the-art TAL method, as temporal localization module. However, any other standard alternative, egg. [34], can be adopted without any additional considerations (see Table A in supplementary material). Relying on graph convolutions (29). G-TAD is composed of stack of GCNeXt blocks to obtain context-aware features. In each GC block, there are two graph convolution streams to model two types of contextual information. One stream operates with temporal neighbors, and the other adaptively aggregates semantic neighbors in the snippet feature space. At the end of the last GCNeXt block, G-TAD extracts a set of sub- sub-graphs based on pre-defined temporal anchors. With the sub-graph of interest alignment layer. SGAlign, it represents each sub-graph using a feature vector, which is further used as input to multiple nullycconeected layers to predict the final action predictions.
To train G-TAD with our LoFi, we select one of the proposed low-fidelity variants and apply the original TAL loss function to optimize both the video encoder and the TAL head. We initialize G-TAD weights as in [60].
Implementation details: We use ResNet-based TSM 33 as the video encoder due to its good accuracy-cost trade-off and reasonable memory requirements compared to 3D-based alternatives. For the full fidelity setting (Eq. (I). we follow the standard G-TAD protocol and represent each video with L = 100 snippets. The full spatial resolution is x W = 224 x 224. We keep the other hyper-parameter
(e.g., the number of GCNeXt layers) the same as in the default G-TAD configuration. However, the number of anchor proposals can be reduced when L is less. Concretely, we enumerate all the possible combinations of start and end the anchors, e.g., {(ta, te)) 0 t, to < L; t,, to E A; t, - t, < L).
For LoFi training, we use an SGD optimizer. The batch size is 16 for all the training methods and input patterns. The weight decay is 10-4 and we set the momentum to O, which is standard for fine-tuning [32]. The learning rate is 0.1, and it is decayed by 0.5 after every 5 epochs.
5

---

When we train G-TAD using the full fidelity setting, we keep the same training strategy as described in the original paper [60]. except that we perform a learning rate search within the {0.0002. 0.0005, 0,001, 0.002, 0.005). We follow the rest of the common post-processing steps for TAI. as specified in the original paper, including the application of soft-NMS [5] with a threshold of 0.84. We select the top-100 predictions for the final evaluation. We provide our code and script as part of the supplementary material.
Hardware and software settings: We implemented our method using PyTorch 1.8 with CUDA 10.1. For LoFi training, we use 4 NVIDIA V100 GPUs, each with 32GB memory. Under this setting, the memory constraint is 128GB, which constitutes mid-range computational budget. In the supplementary material, we further test a low-budget setting with a single V100 GPU in Table B.
Experiments
4.1 Experimental setup
We use Kinetics400 28 as the auxiliary video classification dataset for initial pre- training of the video encoder. For model performance evaluation, use two popular temporal action localization benchmarks. (1) Activity.Net-vI.3 [22] contains 20K temporally annotated untrimmed videos with 200 action categories. In the standard evaluation protocol, these videos divided into the baining/validationneessii sets by the ratio of 2:1:1. (2) Human Action Clips and Segments (HACS--v.I) [68] is a recent temporal action localization dataset. It contains 140K complete action segments from 50K videos including 200 action categories (the same ones as ActivityNet-vl., 1.33).
Datasets:
specified IoU thresholds as the main evaluation metrics. Following the standard evaluation setting, mAP values at set of IoU thresholds, {0.5, 0.75, 0.95) are reported, well as the average mAp over 10 different IoU thresholds [0.5 0.05 0.95].
Evaluation metries: We adopt the mean Average Precision (mAP)
4.2 Pre-training Methodology Comparisons
Setting: In this set of experiments, directly compare different methodologies for pre-training the video encoder. To compare with our proposed LoFi method, we first consider the most widely used pre-training method that optimizes the video encoder through the action classification task on an auxiliary dataset (Kinetics400 in our case). We denote this method as Action Classification Pre-training (ACP). To demonstrate the effect of video encoder pre-training, we also take into account an ImageNet pre-trained video encoder, which uses no video data. We refer to this baseline as Image Classification Pre-training (ICP). To adapt the video encoder to the target dataset, it can be further trained through an action classification task on a clip version of the target TAL dataset, using each positive segment as a training instance. We denote this variant ACP+.
In this experiment, we train our model using the eyclic low-fidelity (C-LoFi) configuration with the long cycle strategy. Other configurations will be evaluated in Section 4.3
Table 1:
Comparing TAL results of different video encoder pre-training methods. ACP: Action Classification Pre-training, ICP: Image Classification Pre-training, ACP+: further fine-tuning ACP on the target dataset using classification task using positive action segments.
Metric
0.5
0.75
0.95
Average
0.5
0.75
0.95
Average
Dataset
ActivityNet-v1.3
HACS-vL.1
ACP
49.64
34.16
7.68
33.59
35.68
22.79
6.51
23.00
ICP
48.45
32,40
6.89
32.16
31.74
19.64
5.67
20.18
ACP+
49.87
34.58
7.85
33.84
36.31
22.96
6.62
23.31
LoFi (ours)
50.68
35.16
8.16
34.49
37.47
24.36
7.08
24.62
The results of the different video encoder pre-training methods are reported in Table We make the following observations. (1) Without pre-training related auxiliary dataset, the model
Results:

---

performance could be significantly degraded (see row 1 v.S. row 2). This indicates the significance of the video encoder and pre-training on large, relevant video data. (2) With action classification based pre-training on the target video data, the performance indeed improves to some degree (see row v.s. row 3). This means that addressing the data distribution shift between the auxiliary dataset and the target dataset is essential. (3) Importantly, the biggest performance gains come from the proposed LoFi method, which instead solves the task discrepancy issue by optimizing the video encoder with the TAL task (see the last row). This indicates that although auxiliary video data is similar to the target data distribution, the task-level differences would still pose obstacles harming the model performance. This confirms the motivation and hypothesis of this study. Once this obstacle is properly tackled with our low- sww-iidliity pre- tee-ttaiinng....i more significant performance gains can then be rewarded. Overall, this verifies the efficacy of the proposed method in pre-tratne pre-training the video encoder.
4.3 LoFi Configuration Comparisons
Setting: We investigate different LoFi configurations. We use the setting r, 4 for T-LoFi, y. 2 for S-LoFi (note that this is applied in both spatial dimensions, thus being comparable to T-LoFi), rs = v2,rt rt = 2 for ST-LoFi, and c, = 16, = for C-LoFi's short and long cycle strategies, respectively. We include the standard pre-training strategy (ie., ACP) provide baseline and facilitate comparisons.
Table 2: Comparing different low-fidelity configurations. Dataset: ActivityNet-v and HACS- v1.1. ACP: Action Classification Pre-training. TR: Temporal Resolution; SR: Spatial Resolution.
ActivityNet-v1.
HACS-v1.1
Configuration
TR
SR
0.5
0.75
0.95
Avg
0.5
0.75
0.95
Avg
ACP
224x224
49.64
34.16
7.68
33.59
35.68
22.79
6.51
23.00
S-LoFi
100
112x112
50.47
34.71
7.57
34.12
37.16
24.23
6.84
24.25
T-LoFi
25
224x224
50.28
35.21
8.09
34.32
37.30
24.18
7.07
24.36
ST-L.Fi
50
158x 158
50.36
34.79
7.59
34.12
37.13
24.27
7.08
24.36
Short C-LoFi
Batch-level Cycle
50.57
35.12
8.14
34.38
37.63
24.45
6.95
24.60
Long C-LoFi
Epoch-level Cycle
50.68
35.16
8.16
34.49
37.78
24.40
7.29
24.64
The results for ActivityNet-vl.3 and HACS-v1.1 are shown in Table 2 We provide the following observations. (1) Each of our proposed LoFi configurations can improve the video encoder, thus suggesting that spatial and temporal dimensions both good selections for low-fidelity manipulation. (2) Integrating our LoFi configurations into a more ad advanced cyclic training strategy produces some further, although moderate, improvement. In particular, the long cycle leads the best overall performance for both datasets.
Results:
Table 3: Comparing TAL results with state-of-the-art methods on ActivityNet-v11. validation set. *** indicates RGB-only Kinetics400 pre-trained TSM video encoder without fine-tuning. O.F.: Optical Flow. R18/50: MisNeet-8/500
Method
O.F.
Arch.
#Par
0.5
0.75
0.95
Average
SCC
X
C3D
79M
40.00
17.90
4.70
21.70
CDC 47
X
C3D
79M
45.30
26.00
0.20
23.80
R-C3D 558
X
C3D
79M
26.80
BSN 335
V
R50
23M
46.45
29.96
8.02
30.03
P-GCN
I3D
25M
48.26
33.16
3.27
31.11
BMN R5
R50
23M
50.07
34.78
8.29
33.85
BC-GNN 8
/
R50
23M
50.56
34.75
9.37
34.26
G-TAD 6
/
R50
23M
50.36
34.60
9.02
34.09
G-TAD baseline*
X
RI8
12M
49,64
34.16
7.68
33.59
G-TAD+ LoFi
X
R18
12M
50.68
35.16
8.16
34.49

---

Table 4: Comparing TAL results with state-of-the-art methods on HACS-v1.1 validation set. indicates RGB-only Kinetics400 pre-trained TSM video encoder without fine-tuning. O.F. Optical Flow. R R18/50: ResNet-18/50. 2S.: 2-stream.
Method
O.F.
Arch
#Par
0.5
0.75
0.95
Average
SSN 669
/
28
12M
28.82
18,80
5.32
18.97
G-TAD baseline*
X
R18
12M
35.68
22.79
6.51
23.00
G-TAD LoFi
x
R18
12M
37.78
24.40
7.29
24.64
4.4 Comparison with State-of-the-Art
Setting: Following video encoder pre-trate pre-training evaluation, we further conduct system-leve perfor- mance comparison with previous state-of ofth--oftthe-art methods. Whilst ResNet50- (R50) methods are common backhone choice for the video encoder, we still use more lightweight ResNet18 (R (R)8) for our method due its higher efficiency in computation and memory. Furthermore, compared to the popular architecture design that uses two streams (one for RGB and one for optical flow), our method only uses RGB frames, avoiding the excessive costs incurred from computing optical flow and running second forward pass.
Results: The results of our method are compared with existing alternatives in Table 3 for ActivityNet-v1.3 and Table for HACS-v1.1. It is evident that our method can achieve the best performance among all the competitors, despite the single stream input modality and much lighter video encoder backbone. This clearly demonstrates that tackling the pre-training of the video encoder is of particular importance for TAL, and that existing efforts towards improving the TAL head model have neglected as key model component. ActivityNet-vl..3 it is encouraging to see that with a stronger pre-trained video encoder using our method and a shallower architecture, optical flow can be favourably eliminated without performance sacrifice (actually even enjoying better performance). This result is substantial, since the study of means to get rid of optical flow for more efficient action analysis is itself an important research problem [10].
4.5
Using Different Video Encoders
Setting: Our LoFi method be used in combination with different video encoders, long as the backbone is end-to-end trainable. We compare the performance gains of our default encoder, i.e. ResNe11 8-based TSM, with a 3D-based alternative, i.e.an 18-layer R(2+1)D encoder [52], ActivityNet-v1.3 Note that the default spatial resolution for R(2+1)D, as defined by the authors, is 112 112 pixels. We thus T-LoFi and maintain the default spatial resolution for both networks.
Table 5: Ablation different video encoders. The performance improvement from using LoFi pre- tre-training does not depend on the video encoder's 's backbone. Dataset: ActivityNet-vl.3. TAL head: G-TAD. R18: ResNet18.
TSM-R18 backbone
R(2+1)D-RI8 backbone
Method
0.5
0.75
0.95
Avg
0.75
0.95
Avg
G-TAD ACP
49.64
34.16
7.68
33.59
49.65
34.11
8.66
33.55
G-TAD T-LoFi
50.28
35.21
8.09
34.32
49.84
34.73
8.64
34.21
Gain
+0.64
+1.05
+0.41
+0.73
+0.19
+0.62
-0.02
+0.66
The results are summarized Table We observe that with either TSM-R18 or R(2+1)D as video encoder, our method can similarly and consistently improve the performance of the state- of-the-art G-TAD method. This verifies that our LoFi method is generally effective and useful in training TAL models.
Results:

---

4.6 Scaling up to ResNet-34 and ResNet-50
Setting: To evaluate the generalizability our LoFi, we further evaluate two deeper video encoders: ResNet34/50-basee TSM. We use the long eyclic low-fidelity configuration. Limited by the GPU memory size, we reduce the batch size accordingly whilst keeping all other hyper-paramett and the training protocol unchanged. This experiment is conducted on ActivityNet-vl. in comparison with the standard action classification pre-training (ACP) method as baseline.
Table 6: Ablation on different depths of TSM video encoders. The performance improvement from using our LoFi pre-training does not depend the video encoder's backbone. Dataset: ActivityNet-vl.3. TAL head: G-TAD. R18: ResNet18; R34: ResNet34; R50: ResNet50.
Encoder
Method
0.5
0.75
0.95
Avg. (Gain)
TSM-R18
ACP
49.64
34.16
7.68
33.59
TSM-R18
Long C-LoFi
50.68
35.16
8.16
34.49 (+0.90)
TSM-R34
ACP
50.16
34.35
8.31
33,90
TSM-R34
Long C-LoFi
50.79
35.39
8.38
34.74 (+0.84)
TSM-R50
ACP Long C-L.FFi
50.32
35.07
8.02
34.26
TSM-R50
50.91
35.86
8.79
34.96 (+0.70)
Results: Table 6 shows that whilst the gains with deeper encoders are slightly reduced, LoFi can still consistently improve the results. This suggests good generalizability properties across varying- capacity networks.
4.7
Comparison to Self-Supervised Learning
We compare our method to two representative video self-supervised learning methods: Arrow of Time [56] and SpeedNet 44. For both competitors, TSM-R18 is used video encoder, and pre-trained on the Kinetics400 dataset. We use ActivityNet-vl. dataset for this experiment.
Setting:
From Table 7 we see that both SSL. methods without action classification based supervision are clearly inferior for the TAL task (see row and 3). When com- bined with the standard action classification pre-training (ACP), only slight performance gains can be achieved despite doubling the computational cost in video encoding (see row 4 and 5). This suggests marginal com- plementary benefit. Overall, this result in- dicates that the proposed LoFi optimization superior over recent SSL. alternatives in pre-training TAL's video encoder.
Results:
Table 7: Comparison to self-supervised learning methods. We use TSM-R18 video encoder. Dataset: ActivityNet.
Method
0.5
0.75
0.95
Average
ACP
49.64
34.16
7.68
33.59
Arrow 56
44.14
28.87
5.90
28.82
Speed 4
44.50
29.52
6.14
29.39
ACP+Arrow
49.79
34.48
7.70
33.72
ACP+Speed
49,84
34.11
7.50
33.75
LoFi
50.68
35.16
8.16
34.49
5 Conclusion
In this work have presented simple and effective low-fidelity (LoFi) video encoder optimization method for achieving more effective TAL models. This is motivated by an observation that in existing TAL methods, the video encoder is merely pre-trained by action classification supervision on short video clips, lacking desired optimization w.r.f. the temporal localization supervision on the target dataset. Indeed, joint optimization itself is not novel. However, this is non-trivial to conduct for training TAL model, due to large per-video size that would easily overwhelm the GPU memory, rendering it infeasible in practice. To overcome this obstacle, we propose to reduce the mini-bater construction configurations in the temporal and/or spatial dimensions of training videos so that end-to-end optimization becomes operable under the same memory condition. Extensive experiments demonstrate that our method can clearly improve the performance of existing off-the-shel TAL. models, yielding new state-of-the performance even with only RGB modality as input and more lightweight backbone based single-stream video encoder on two representative TAL benchmarks.

---

Ethical considerations and broader impact
Any inherent bias present in the training data is likely to be captured by the learning algorithm given its data-driven nature. Deploying of the model into real-world scenarios should thus take that aspect into account. Furthermore, the method discussed in this paper focuses on fundamental problem and as such its potential applications are hard to predict. However, to the best of our knowledge, there are currently no applications of this technology that raise ethical issues. Finally, biases based on gender, race or sexuality are unlikely, although not impossible.
7 Acknowledgements
This work was supported by the King Abdullah University of Science and Technology (KAUST) Office of Sponsored Research through the Visual Computing Center (VCC) funding.
References
[1] Humam Alwassel, Silvio Giancola, and Bernard Ghanem. TSP: Temporally--sesittiv pretraining
of video encoders for localization tasks. iccvw, 2021.
[2] Humam Alwassel, Dhruv Mahajan, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self- supervised learning by cross-modal audio-video clustering. In NeurIPS, 2020.
[3] Yueran Bai, Yingying Wang, Yunhai Tong, Yang Yang, Qiyue Liu, and Junhui Liu. Boundary Content Graph Neural Network for Temporal Action Proposal Generation. In ECCV, 2020.
[4] Sagie Benaim, Ariel Ephrat, Oran Lang, Inbar Mosseri, William T Freeman, Michael Rubinstein, Michal Irani, and Tali Dekel. SpeedNet: Learning the speediness in videos. In CVPR, 2020.
[5] Navaneeth Bodla, Bharat Singh, Rama Chellappa, and Larry S. Davis. Soft-NMS improving object detection with one line of code. In ICCV, 2017.
[6] Shyamal Buch, Victor Escorcia, Chuanqi Shen, Bernard Ghanem, and Juan Carlos Niebles. SST: single-stream temporal action proposals. In CVPR, 2017.
[7] Fabian Caba Heilbron, Juan Carlos Niebles, and Bernard Ghanem. Fast temporal activity proposals for efficient detection of human actions in untrimmed videos. In CVPR, 2016.
[8] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the Kinetics dataset. In CVPR, 2017.
[9] Yu-Wei Chao, Sudheendra Vijayanarasimhan, Bryan Seybold, David A. Ross, Jia Deng, and Rahul Sukthankar. Rethinking the faster R-CNN architecture for temporal action localization.
In CVPR, 2018.
[10] Nieves Crasto, Philippe Weinzaepfel, Karteek Alahari, and Cordelia Schmid. MARS: Motion- augmented RGB stream for action recognition. In CVPR, 2019.
[11] Cristian Sattiiuuz--pazzo Edison Marrese- Taylor, Fatemeh Saleh, Hongdong Li and Stephen Gould. Proposal-free temporal moment localization of natural- query video using guided attention. In WACV, 2020.
[12] Victor Escorcia, Fabian Caba Heilbron, Juan Carlos Niebles, and Bernard Ghanem. DAPs:
Deep action proposals for action understanding.
ECCV, 2016.
[13] Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. Convolutional two-stream network
fusion for video action recognition. In CVPR, 2016.
[14] Jiyang Gao, Kan Chen, and Ramakant Nevatia. CTAP: Complementary temporal action proposal generation. ECCV, 2018.
[15] Jialin Gao, Zhixiang Shi, Guanshuo Wang, Jiani Li, Yufeng Yuan, Shiming Ge, and Xi Zhou. Accurate temporal action proposal generation with relation-aware pyramid network. AAAI, Apr. 2020.
1407))10810-1088
[16] Jialin Gao, Xin Sun, Mengmeng Xu, Zhou, and Bernard Ghanem. Relation-aware video reading comprehension for temporal language grounding. EMNLP-IJCNLP, 2021.
[17] Jiyang Gao, Zhenheng Yang, Kan Chen, Chen Sun, and Ram Nevatia. TURN TAP: Temporal
unit regression network for temporal action proposals. ICCV, 2017.
[18] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, page 580-587, 2014.
[19] Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking ImageNet pre-training. CVPR, 2019.
10

---

[20]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[21] Fabian Caba Heilbron, Wayner Barrios, Victor Escorcia, and Bernard Ghanem. SCC: Semantic
context cascade for efficient action detection. In CVPR, 2017.
[22] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. ActivityNet:
A large-scale video benchmark for human activity understanding. In CVPR, 2015.
[23] Fabian Caba Heilbron, Juan Carlos Niebles, and Bernard Ghanem. Fast temporal activity proposals for efficient detection of human actions in untrimmed videos. In CVPR, 2016.
[24] Jiabo Huang, Yang Liu, Shaogang Gong, and Hailin Jin. Cross-sentence temporal and semantic relations in video activity localisation. In ICCV, 2021.
[25] Qingbao Huang, Jielong Wei, Yi Cai, Changmeng Zheng, Junying Chen, Ho-fung Leung, and Qing Li. Aligned dual channel graph convolutional network for visual question answering. In ACI., 2020.
[26] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In JCML, 2015.
[27] Simon Jenni, Givi Meishvili, and Paolo Favaro. Video representation learning by recognizing temporal transformations. In ECCV, 2020.
[28] Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The Kinetics human action video dataset. arXiv preprint, 2017.
[29] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional
networks. In ICLR, 2017.
[30] H. Lee, J. Huang, M. Singh, and M. Yang. Unsupervised representation learning by sorting sequences. In ICCV, 2017.
[31] Jie Lei, Licheng Yu, Mohit Bansal, and Tamara Berg. TVQA: Localized, Compositional
Video Question Answering. In EMNLP-IJCNLP, 2018.
[32] Hao Li, Pratik Chaudhari, Hao Yang, Michael Lam, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Rethinking the hyperparameters for fine-tuning. In ICLR, 2020.
[33] Ji Lin, Chuang Gan, and Song Han. TSM: Temporal shift module for efficient video understand- ing. In ICCV, 2019.
[34] Tianwei Lin, Xiao Liu, Xin Li, Errui Ding, and Shilei Wen. BMN: boundary network
for temporal action proposal generation. In ICCV, 2019.
[35] Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang, and Ming Yang. BSN: Boundary
sensitive network for temporal action proposal generation. In ECCV, 2018.
[36] Daizong Liu. Xiaoye Qu, Xiao- Yang Liu. Jianfeng Dong, Pan Zhou, and Zichuan Xu. Jointly cross-and self-modal graph attention network for query-based moment localization. In ACM MM, 2020.
[37] Rosanne Liu, Joel Lehman, Piero Molino, Felipe Petroski Such, Eric Frank, Alex Sergeev, and Jason Yosinski. An intriguing failing of convolutional neural networks and the coordcony solution. In NewrIPS, 2018.
[38] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C. Berg. SSD: Single shot multibox detector. In ECCV, 2016.
[39] Yuan Liu, Lin Ma, Yifeng Zhang, Wei Liu, and Shih-Fu Chang. Multi-granularity generator for temporal action proposal. In CVPR, 2019.
[40] Fuchen Long, Ting Yao, Zhaofan Qiu, Xinmei Tian, Jiebo Luo, and Tao Mei. Gaussian temporal awareness networks for action localization. CVPR, 2019.
[41] Stéphane Mallat. Understanding deep convolutional networks. Philosophical Transactions of
the Royal Society A: Mathematical, Physical and Engineering Sciences, 374(2065), 2016.
[42] Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In CVPR. 2020.
[43] Ishan Misra, C. Lawrence Zitnick, and Martial Hebert. Shuffle and learn: Unsupervised learning using temporal order verification. In ECCV, 2016 [44] Jonghwan Mun, Minsu Cho, and Bohyung Han. Local-global video-text interactions for temporal grounding. In CVPR, 2020.
[45] Zhaofan Qiu, Ting Yao, and Tao Mei. Learning spatio-temporal representation with pseudo-3D residual networks. In ICCV, 2017.
11

---

[46]
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In CVPR, 2016.
[47] Zheng Shou, Jonathan Chan, Alireza Zareian, Kazuyuki Miyazawa, and Shih-Fu Chang. CDC: networks for precise temporal action localization in untrimmed
cunvolutional-dc-covooouue
videos. In CVPR, 2017.
[48] Zheng Shou, Dongang Wang, and Shih-Fu Chang. Temporal action localization in untrimmed videos via multi-stage CNNs. In CVPR, 2016.
[49]
Mattia Soldan, Mengmeng Xu, Sisi Qu, Jesper Tegner, and Bernard Ghanem. VLG-Net:
Video-language graph matching network for video grounding. In ICCV workshops, 2021.
[50] Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu. A survey on deep transfer learning. In Véra Kúrková, Yannis Manolopoulos. Barbara Ham- mer, Lazaros Iliadis, and Ilias Maglogiannis, editors, International Conf. Artificial Neural Networks, 2018.
[51] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. Learning spatiotemporal features with 3D convolutional networks. In ICCV, 2015.
[52] Du Tran, Heng Wang. Lorenzo Torresani, Jamie Ray. Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In CVPR, 2018.
[53] Jiangliu Wang. Jianbo Jiao, and Yun-Hui Liu. Sclf-supervised video representation learning by pace prediction. In ECCV, 2020.
[54] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In ECCV, 2016.
[55] Xiang Wang, Zhiwu Qing, Ziyuan Huang, Yutong Feng, Shiwei Zhang, Jianwen Jiang, Mingqian Tang, Changxin Gao, and Nong Sang. Proposal relation network for temporal action detection. CoRR, abs/2106. 11812, 2021.
[56] D. Wei, J. Lim, A. Zisserman, and W. T. Freeman. Learning and using the arrow of time. In CVPR, 2018.
[57] Chao Yuan Wu, Ross Girshick, Kaiming He, Christoph Feichtenhofer, and Philipp Krahenbuhl. A multigrid method for efficiently training video models. In CVPR, 2020.
[58] Huijuan Xu, Abir Das, and Kate Saenko. R-C3D; Region convolutional 3D network for temporal activity detection. In ICCV, 2017.
[59] Mengmeng Xu, Juan-Manuce Pérez-Rúa, Victor Escorcia, Brais Martinez, Xiatian Zhu, Bernard Ghanem, and Tao Xiang. Boundary-sensitive pre-training for temporal localization in videos. In ICCV, 2021.
[60] Mengmeng Xu, Chen Zhao, David S Rojas, Ali Thabet, and Bernard Ghanem. G-TAD: Sub- graph localization for temporal action detection. In CVPR, 2020.
[61] Yuan Yao, Chang Liu, Dezhao Luo, Yu Zhou, and Qixiang Ye. Video playback rate perception for self-supervised spatio-temporal representation learning. In CVPR, 2020.
[62] Ze-Huan Yuan, Jonathan C. Stroud, Tong Lu, and Jia Deng. Temporal action localization by
structured maximal sums. In CVPR, 2017.
[63] Runhao Zeng. Wenbing Huang, Mingkui Tan, Yu Rong, Peilin Zhao, Junzhou Huang, and Chuang Gan. Graph convolutional networks for temporal action localization. In ICCV, 2019. Runhao Zeng, Haoming Xu, Wenbing Huang, Peihao Chen, Mingkui Tan, and Chuang Gan.
[64]
Dense regression network for video grounding. In CVPR, 2020.
[65] Hao Zhang, Aixin Sun, Wei Jing, and Joey Tianyi Zhou. Span-based localizing network for natural language video localization. In ACL, 2020.
[66]
Songyang Zhang, Houwen Peng, Jianlong Fu, and Jiebo Luo. Learning 2d temporal adjacent networks for moment localization with natural language. In AAAI, 2020.
[67] Chen Zhao, Ali K Thabet, and Bernard Ghanem. Video self-stitching graph network for temporal action localization. In ICCV, 2021.
[68] Hang Zhao, Zhicheng Yan, Lorenzo Torresani, and Antonio Torralba. HACS; Human action clips and segments dataset for recognition and temporal localization. icev, 2019.
[69] Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, and Dahua Lin. Temporal action detection with structured segment networks. In ICCV, 2017.
[70] Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning. Proceedings of the IEFE, 109(1):43-76, 2021.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Please see more discussion of our
LoFi method in the supplementary material Section 2.
(c) Did you discuss any potential negative societal impacts your work? [Yes] Please see Section 2 in the supplementary material for societal impact discussion.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] We provide our code and script as part of the supplementary material.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] Please see Section 3.3 for implementation details.
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? (Ye]] Although most of the TAL works do not report error bars, we include them in the supplementary material, Section 1.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please see Section 3.3 for hardware and software settings.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "TD4NdfTzib", "reviewer": "Reviewer_mQTb", "summary": "Updating after the rebuttal:\nThe rebuttal well addressed some of my concerns. I would raise the rating to 6: Marginally above the acceptance threshold\n\n---\n\nThe paper presents a low-fidelity video encoder optimization approach to relieve the large memory constraints in the temporal action localization problem. The proposed approach is easy to implement and reasonable results have been obtained on ActivityNet and HACS datasets. However, the proposed approach is more like an engineering solution. The technical novelty of the proposed approach is limited. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Originality\nThe originality is below the threshold. The main contribution of the paper is a new training pipeline which is based on the low-fidelity setting to enable joint ttraining of action classification and localization.  It is not sufficient to support a NIPS-level paper. \n\n## Quality\nThe proposed approach is technical sound. Based on the details in the paper, it should not be difficult to reproduce the results. Also, the proposed pipeline obtain good results on two benchmark datasets like ActivityNet and HACS. Ablation studies are sufficient to support the proposed training pipeline.\n\n## Clarity\nThe paper is well presented and sufficient details have been provided in the paper. \n\n## Significance\nThe proposed pipeline should be useful for as a module for other temporal localization algorithms. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "## Contribution is limited\nThe paper is mainly based on a new training pipeline with three stages. To releive the high memory cost issue, it proposes to use low-fidelity setting in order to train both video encoder and temporal action localization head. It is more like an engeering solution for this problem. The techinical contribution is not significant.\n\n## Experiments\nHow about the training time for the proposed approach? With the similar training, how about the performance of baseline. \nIn Table 6, it presents the results based on different backbone like Resnet 18/34/50. How about the detail setup for this experiment? For example, Res50 should be much larger than Res18, the memory cost should be larger. \n\nIn Table 2, the experiments provide several setting for the low-fidelity configurations. Is it possible to provide more results on different setups. \n\nBased on the setup,  4 V100 GPUs have been utilized. How about the performance with more GPUs. \n                                      ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "6oxgpO0gf6k", "reviewer": "Reviewer_krQv", "summary": "Models for temporal action localization (TAL) in long, untrimmed video are typically trained in two stages, a video encoder pretrained on a auxiliary video clip classification dataset and then frozen, followed by a TAL head. End-to-end training of the encoder and TAL head is infeasible because the encoder activations for an entire, long video will not fit in GPU RAM. This paper proposes a solution: use the ideas of mulgrid training [51] to do end-to-end training of a lower resolution video encoder and TAL head. They explore strategies for cycling through regimes of lower spatial, temporal, and spatio-temporal resolution training, and demonstrate performance gains on the ActivityNet and HACS datasets, using a number of backbone networks and TAL heads. \n", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Originality] This paper confronts a limitation--the inability to train the video encoder e2e on the TAL task--that is simply accepted in hundreds (?) of TAL papers, and proposes a workable solution. The multigrid methods lying at the heart of this were proposed in another paper about video action classification [51], but here they're applied in a very novel way to address a long-standing problem.\n\n[Clarity] The paper is very well written, and clear. I enjoyed reading it. However it is missing a clear explanation as to why training on lower spatial or temporal resolution inputs would be expected to work - both that the models would be compatible, and that the features learned when r_s = 2 would even make sense for r_1 = 1, for example. I did find this in the \"Multigrid Method\" paper [51]:\n\n>_\"Multigrid training is possible because video models are compatible with input data of variable space and time dimensions due to weight sharing operations (e.g., convolutions). In addition, CNNs are effective at learning patterns at multiple scales, e.g., as observed when training with data augmentation [18, 26, 38]. We observe similar multi-scale robustness and generalization with multigrid training\"_\n\nThis paper really needs to include a similar argument.\n\nSome additional minor points:\n- The \"Further Discussion\" points in the appends should be in the main paper\n- Line 164 confused me \"finally outputting less predictions in case the candidates per time location remains the same\"\n- Line 179 Typo: multi-grid\n\n[Quality] This is a solid, well-supported paper. I think there is a limitation to this approach that is not fully-addressed: in the case of high-parameter SOTA video feature encoder, or when input videos are long, the approach might not work at all because end-to-end training would require such drastically large values of r_s and r_t. I noticed that the authors focus on a very cheap model ResNet18, which seemed like a dodge to the limitation, but felt better when I saw the R(2+1)D 3DCNN experiments. Still, it might be nice to confront the problem directly, do the math about what is feasible, and say whether I the approach will work with SlowFast, a Non-local 3D ResNet-101s, etc.\n\n[Significance] I think this paper has a high degree of novelty, by addressing a long-standing problem in a unique way. The performance numbers are good, not amazing, but certainly enough to demonstrate that this new direction has promise and warrants more attention.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, mostly addressed. See discussion of scalability and surveillance in the \"Main Review\"", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "0dyOmRI1gkB", "reviewer": "Reviewer_6HSH", "summary": "** Update 8/16/21 ** \n\nThanks to the authors for their rebuttal in response to my questions/concerns. After reading the rebuttal and the other reviews, I am updating my score to 6, to better reflect the contributions made by the work.\n\n** \n\nThe paper aims to tackle the hardware constraint for end-to-end optimization of both video encoding and TAL head models. Different from prior works that freeze the video encoding network when training the TAL head, the proposed method optimizes both of them jointly on low-resolution (spatially and temporally) video frames and then finetunes the TAL head on full-resolution videos. The main contribution of this paper is resolving the task discrepancy problem for the video encoder when training TAL models. Extensive experiments are conducted in the paper to compare the lofi setting with several state-of-the-arts, as well as similar models under different pretraining conditions. Results indicate the proposed method outperforms the baselines.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors would like to tackle the task discrepancy problem and meanwhile bypass the GPU memory limitations for joint end-to-end optimization, so they train the model on low-fidelity data. It’s practical, but lacks sufficient novelty in my opinion. The solution proposed in the paper would generally be considered a data-augmentation technique that expands the dataset by sampling snippets/frames from full-resolution data and reduces GPU usage. The downsampling augmentation is common in image classification or localization tasks, but citations for this are not adequate. (ex. A survey on image data augmentation for deep learning.) Also, the C-LoFi method is too similar to [51]. The resampling method and the training schedule are, though used differently for the TAL task for GPU memory issue, the same as the literature.  \n\nMaybe it’s not appropriate to regard this work as a pretraining task and compare it with pretraining baselines. In reviewing previous works, perhaps it’s better to view this work from the angle of data augmentation. It’s also interesting to see if lofi works for other tasks. \n\nTechnical soundness:\n\nThe author presents in-depth experiments and discussions regarding the lofi setting and other related methods. The tradeoff between accuracy and efficiency can also be found in the paper. Indeed, the proposed method does outscore the baselines. However, it’s not clear what the reason is, whether because the task discrepancy problem is mitigated or just because the data is augmented and the model has more learnable parameters. The experiments are not fair enough because the lofi setup is accessible to the augmented data in the target domain. \n\nSolving the task discrepancy problem is crucial in transfer learning, as we hope the knowledge can be transferred to new domains but don’t want the discriminative features to be wiped out. More efforts need to be put into the investigations and experiments on it. Maybe one possible way is to compare the lofi setting with the model trained end-to-end on full-resolution data, compare with a ACP setting but with a comparable number of learnable weights, or visualize the features from the video encoder.\n\nClarity of exposition:\nThe exposition is clear and it’s not hard for readers to get to the point. The proposed method can be easily reproduced. \nPerhaps the word “low-fidelity” is a bit confusing because readers might associate it with the model, not the data.\n\nSignificance:\n\nThe results show that the lofi setup outperforms other pretraining procedures. However, the claim that the proposed method mitigates the task discrepancy problem is not well-supported. If lofi setting were comparable with high-fi training, or is shown relatively advantageous in terms of the accuracy-efficiency tradeoff in the experiments, practitioners can benefit and build on the proposed idea. \nOn the other hand, the results will be more important if not only TAL, but other related tasks are addressed using the same lofi setting. \n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of this work are stated above. The task discrepancy issue is worth investigating and the authors are encouraged to continue their efforts on the research problem.\n\nThe authors do a reasonable job to address the societal implications of the work. ", "review_text": "", "extra_scores": {"ethical_concerns": "None. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2-4 hours"}, "extra_text": {}}, {"review_id": "08ZjP7xDjf", "reviewer": "Reviewer_BMhR", "summary": "In this paper, the authors propose a new method to allow video encoder optimization in training temporal action localization (TAL) models, which is often ignored by existing TAL methods due to GPU memory constraints. The proposed method LoFi reduces the demanding GPU memory requirement by using a lower temporal and/or spatial resolution in the mini-batch construction. The proposed LoFi technique has shown to help enhance the performance of off-the-shelf TAL models on ActivityNet and HACS-v1.1. Also, the performance improvement is universal across different types of video encoder.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-motivated. \n\nSparse temporal sampling and different spatial scales may have been widely adopted as data augmentation techniques when training action classification models with relatively short videos. However, for temporal action localization tasks on long videos, the frozen video encoder can be a bottleneck to the model performance. LoFi is indeed a simple yet effective method to allow adaptation of the pre-trained video encoder to the target video domain of the downstream TAL task. \n\nTraining with long videos is always resource-demanding, for different kinds of video understanding or video+language understanding tasks.  I believe that the proposed method LoFi may have the potential to be extended to other tasks on long videos other than temporal action localization. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents work on temporal action localization.  The main idea is to use low fidelity (e.g. lower temporal resolution) to enable end-to-end training within the constraints imposed by large video batches / models in GPU memory.  The reviewers appreciate the simplicity and effectiveness of this idea.  While based on related efforts in training image and other video models, identifying this approach to enable end-to-end training is interesting to the researchers working in this field, challenging a standard assumption about the difficulties in end-to-end training.

Decision: Accept (Poster)