Title: Low-Fidelity Video Encoder Optimization for Temporal Action Localization
Authors: Mengmeng Xu
Juan-Manuel Perez-Rua
Xiatian Zhu
Bernard Ghanem
Brais Martinez
Venue: NeurIPS 2021
Keywords: End-to-End Pre-training
Temporal Action Localization

Abstract: Most existing temporal action localization (TAL) methods rely on a transfer learning pipeline: by first optimizing a video encoder on a large action classification dataset (i.e., source domain), followed by freezing the encoder and training a TAL head on the action localization dataset (i.e., target domain). This results in a task discrepancy problem for the video encoder – trained for action classification, but used for TAL. Intuitively, joint optimization with both the video encoder and TAL head is a strong baseline solution to this discrepancy. However, this is not operable for TAL subject to the GPU memory constraints, due to the prohibitive computational cost in processing long untrimmed videos. In this paper, we resolve this challenge by introducing a novel low-fidelity (LoFi) video encoder optimization method. Instead of always using the full training configurations in TAL learning, we propose to reduce the mini-batch composition in terms of temporal, spatial, or spatio-temporal resolution so that jointly optimizing the video encoder and TAL head becomes operable under the same memory conditions of a mid-range hardware budget. Crucially, this enables the gradients to flow backwards through the video encoder conditioned on a TAL supervision loss, favourably solving the task discrepancy problem and providing more effective feature representations. Extensive experiments show that the proposed LoFi optimization approach can significantly enhance the performance of existing TAL methods. Encouragingly, even with a lightweight ResNet18 based video encoder in a single RGB stream, our method surpasses two-stream (RGB + optical-flow) ResNet50 based alternatives, often by a good margin. Our code is publicly available at https://github.com/saic-fi/lofi_action_localization.

TL;DR: We tackle the problem of temporal action localization and propose an intermediate low-fidelity optimization stage to jointly optimize the video encoder and the localization head on the target dataset and task, outperforming SOTA performance.

Review discussion:
The paper is well writen. The graphs and equations are clear.
However, the idea of the paper is easy to get and seems trivial. It only decrease the spatial or temporal features in the feature maps to ensure the model can be trained under the hardware constraints.
Moreover, a sampling grid similar to the multi-grid training strategy is also proposed as short and long cycle strategies.  I believe that the innovativeness of the paper is not enough.
No
We thank the reviewer for the detailed review as well as the suggestions for improvement. Our response to the reviewer’s comments is as below:
Q1: _However, the idea of the paper is easy to get and seems trivial. It only decrease the spatial or temporal features in the feature maps._
R1: We have extensive experiments showing that our solution is effective, clearly surpassing more complex solutions. In our opinion, that the proposed solution is also intuitive and simple should be seen as a plus, not a minus.
Furthermore, a multitude of papers argue that end-to-end training is not possible for TAL. Challenging this widespread belief is, in our opinion, a contribution of its own.
Finally, we would like to stress again the contributions listed in L67-76 in our main paper. We also note that our LoFi reduces the spatial and temporal resolution in the video space (please see L12, 59-62, 143-145, and Figure 2’s caption), rather than in the feature space.
Q2: _A sampling grid similar to the multi-grid training strategy is also proposed as short and long cycle strategies._
R2: We would like to note that the novelty of our method does not lie on our modifications of the multi-grid strategy. Our main contribution is to shift the widespread view that the community has for the TAL problem: we show that, against the widespread belief, end-to-end training under specific memory budget is possible, propose a simple yet effective mechanism to explore this, and show that end-to-end training outweigh any potential issues arising from training with lower resolutions.
Furthermore, while most recent works keep designing new TAL heads as the main strategy, we instead show that improving the representation capability of the video encoder (a relatively neglected aspect) has high potential and practical impact.
In summary, the cyclic schedulers are a neat idea we adapt to fit our LoFi, not the main contribution of this work.
I still insist that the Rating would be 5: Marginally below the acceptance threshold. The idea is trival and lacks innovation for NeurIPS.
Pros:
-	The motivation is sound and makes sense. Pretraining the encoder under a separate image-/action-classification has become a common practice and been taken as granted, and few have considered the distribution-shift and task-shift between the pretraining and the main TAL task. This paper has noticed this and taken a further step towards the direction, which is a good complementation to the literature and contribution to the practice.
-	The method is simple and concise, and could be plugged in any main-stream TAL models, which would be good news for implementation in the future.
-	Experiments seem complete enough to fairly support the motivation and the algorithmic innovation. Table 1 justifies the motivation that task-shift could degrade the performance non-trivially. Other experiments show the advantage of plugging in the proposed pretraining step over multiple TAL baselines and several self-supervised pretraining methods, as well as the consistent improvement under different settings.
Cons:
-	In Table 1, the authors compare ACP/ACP+ with LoFi to show that the task-shift is more significant than the distribution shift.
However, here LoFi is an approximation of the full-fidelity counterpart which is an upper bound of the algorithm and can fully demonstrate how significant the task-shift is. It would be nice to see results of full-fidelity pretraining (maybe in a setting with smaller model to meet the GPU memory requirement) as a clue of how significant the task-shift is.
-	In comparison with other baselines, LoFi use smaller model (ResNet18 vs. Resnet50) and less modality (RGB+Flow vs. RGB). In the final version it would be nice to see the more fair comparison with the same backbone and same data modality used.
Overall, this is a concise and solid work which in my opinion may not seem quite “fancy” as other work but has significance both to the literature and to the practitioners. I’m inclined to accept the paper.
No
Thank you for your positive feedback regarding our work.
Q1: _It would be nice to see results of full-fidelity pre-training as a clue of how significant the task-shift is._
R1: Unfortunately producing a full fidelity model is not straightforward - that is indeed the main limitation our paper is trying to tackle. We tried training a full fidelity model, but it used a very low batch size so the results were not encouraging. To solve this issue, we would need to implement training distributed on multiple machines.
Thus we believe this might be out of the scope of the rebuttal. We nonetheless think it is an interesting proposal and we will run the experiments in the future for completeness.
Q2: _In the final version it would be nice to see the more fair comparison with the same backbone and same data modality used._
R2: Thanks for the constructive suggestion. The table below shows the LoFi’s results using ResNet50 w/ and w/o Optical Flow. As suggested, we will include this table in the revised version. These results show that when optical flow is used, LoFi can further improve performance while maintaining the performance gain. Note that LoFi strategy is not used for the optical flow stream. That would require extracting raw optical flow for all videos, which is not feasible within the time period of rebuttal.
Instead, we use the standard pre-computed optical flow features from BSN work in this experiment.
| Method |Optical Flow|0.5|0.75|0.95|Average|
| --- |--- |--- |--- |--- |--- |
| G-TAD [54]| Yes | 50.36| 34.60| 9.02| 34.09| 
| ACP| No | 50.32 | 35.07 | 8.02 | 34.26| 
| LoFi| No | 50.91 | 35.86 | 8.79 | 34.96| 
| ACP| Yes | 51.12| 36.22| 8.80| 35.20| 
| LoFi| Yes | 51.68| 36.93| 8.69| 35.76|
Advantages:
1. This paper is well motivated and is dedicated to solving the task discrepancy problem for the TAL task.
2. This paper is well written and easy to understand, both the figures and equations are clearly presented.
3. The ablation studies are well designed.
Weaknesses:
1. The experiment evaluation is not adequate. Since the contribution of this paper is a new training pipeline for TAL, it should be evaluated in more methods instead of just G-TAD. There are many other available open-source algorithms (BMN, BSN, P-GCN, AFSD and so on), the authors are encouraged to verify the effectiveness of the proposed method on other TAL methods.
2. The performance improvement is marginal. Compared with ACP+,
the performance improvement is rather limited, less than 1%.
However, the ACP+ is much simpler to implement, taking this into consideration, the proposed method in this paper is not so competitive.
3. The efficiency (time, memory etc) of different training pipelines should be reported.
No
We thank the reviewer for the detailed review as well as the suggestions for improvement. Our response to the reviewer’s comments is as below.
Q1: _It should be evaluated in more methods instead of just G-TAD._
R1: The supplementary material already includes an experiment using BMN (this is indicated in L188 of the main paper). We have now additionally tested our LoFi in combination with BSN. The result is shown in the table below, and yields a gain of 0.54 in terms of Area Under the AR vs AN curve, consistently verifying the effectiveness of our method. We cannot test PGCN as it provides no dictionary file for ActivityNet 1.3, which is necessary for experiments.
Finally, AFSD was published after our submission. We will investigate it with our LoFi for the revised version.
| Method      | Backbone | AR@1 | AR@5 | AR@10 | AR@100 | AUC |
| ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |
| BSN      | TSM-RN50 | 32.75 | 47.51 | 54.01 | 74.61 | 66.19 | 
| BSN + LoFi   | TSM-RN50        | 32.86 | 48.00 | 55.14 | 75.08 | 66.73 (+0.54) |
Q2: _(1) Compared with ACP+, the performance improvement is rather limited, less than 1%. (2) However, the ACP+ is much simpler._
R2: (1) We respectfully disagree with the reviewer. The evaluation metric on ActivityNet, average mAP (AmAP), is a very strict metric as it involves high tIoU thresholds (0.5 to 0.95). If we examine recent TAL methods, for example, we see an improvement of 0.24 by G-TAD (34.09, CVPR20) over BMN (33.85, ICCV19), and 0.17 for BC-GNN (34.26, ECCV20) over G-TAD.
Instead, LoFi’s improvement of 0.65 is very competitive compared to that of previous TAL methods.
(2) ACP+ also has 3 training stages, so it's complexity is comparable to LoFi.
Q3: _The efficiency (time, memory, etc) of different training pipelines should be reported._
R3: Thanks. As suggested, we will report these metrics in the revised paper as the Table below:
| Configuration | Backbone | TR | SR | Memory/video | Time/epoch | 
| ---- | ---- | ---- | ---- | ---- | ---- | 
| S-LoFi | ResNet18 | 100 | $112^2$  | 8.6 GB | 1284 sec 
| T-LoFi| ResNet18| 25| $224^2$ | 9.0 GB| 990 sec| 
| ST-LoFi|ResNet18| 50| $158^2$ | 8.9 GB| 1075 sec| 
| ST-LoFi| ResNet34| 50| $158^2$ | 11.5 GB| 1624 sec| 
| ST-LoFi| ResNet50| 50| $158^2$ | 29.0 GB| 2694 sec|
I have read the response and other reviews, while the response indeed addressed parts of my concerns, I think the paper still needs a large amount of revision and future improvements to meet the bar of NeurIPS. Especially, experimental results are not adequate and more baseline methods shold be evaluated to demonstrate the effectiveness of the proposed model.
Therefore I keep my previous rating "5: Marginally below the acceptance threshold".

Meta-review:
role: PC
data: comment: The paper initially received mixed reviews, and after the rebuttal the review consensus has slightly converged to the reject opinion. The reviews recommend that the following points should be addressed for the future stronger submission: (i) more thorough experimental evaluation with more baselines and (ii) better justification about the significance of the proposed idea.

Decision: Accept (Poster)