Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: In this paper, the authors investigate the limitations of the standard optimization method, and propose a new training pipeline to deal with this limitation.
main_review: Advantages:
1. This paper is well motivated and is dedicated to solving the task discrepancy problem for the TAL task.
2. This paper is well written and easy to understand, both the figures and equations are clearly presented.
3. The ablation studies are well designed.

Weaknesses:
1. The experiment evaluation is not adequate. Since the contribution of this paper is a new training pipeline for TAL, it should be evaluated in more methods instead of just G-TAD. There are many other available open-source algorithms (BMN, BSN, P-GCN, AFSD and so on), the authors are encouraged to verify the effectiveness of the proposed method on other TAL methods.
2. The performance improvement is marginal. Compared with ACP+,
the performance improvement is rather limited, less than 1%. However, the ACP+ is much simpler to implement, taking this into consideration, the proposed method in this paper is not so competitive.
3. The efficiency (time, memory etc) of different training pipelines should be reported. 

limitations_and_societal_impact: The authors have clarified the limitations and social impact well.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a simple yet effective method of pretraining the feature encoder in TAL models. The motivation comes from the distribution-shift as well as the task-shift between the classical image-/action-classification pretraining and the TAL training. To make the encoder also adapt to the TAL task, the authors propose to unfreeze the encoder and pretrain under the TAL task. Due to GPU memory limit, the authors use spatially/temporally low fidelity data to pretrain the encoder. Experiments are conducted to verify the authors’ motivation and show the advantage of such a pretraining routine.
main_review: Pros:
-	The motivation is sound and makes sense. Pretraining the encoder under a separate image-/action-classification has become a common practice and been taken as granted, and few have considered the distribution-shift and task-shift between the pretraining and the main TAL task. This paper has noticed this and taken a further step towards the direction, which is a good complementation to the literature and contribution to the practice.
-	The method is simple and concise, and could be plugged in any main-stream TAL models, which would be good news for implementation in the future.
-	Experiments seem complete enough to fairly support the motivation and the algorithmic innovation. Table 1 justifies the motivation that task-shift could degrade the performance non-trivially. Other experiments show the advantage of plugging in the proposed pretraining step over multiple TAL baselines and several self-supervised pretraining methods, as well as the consistent improvement under different settings.

Cons:
-	In Table 1, the authors compare ACP/ACP+ with LoFi to show that the task-shift is more significant than the distribution shift. However, here LoFi is an approximation of the full-fidelity counterpart which is an upper bound of the algorithm and can fully demonstrate how significant the task-shift is. It would be nice to see results of full-fidelity pretraining (maybe in a setting with smaller model to meet the GPU memory requirement) as a clue of how significant the task-shift is.
-	In comparison with other baselines, LoFi use smaller model (ResNet18 vs. Resnet50) and less modality (RGB+Flow vs. RGB). In the final version it would be nice to see the more fair comparison with the same backbone and same data modality used.


Overall, this is a concise and solid work which in my opinion may not seem quite “fancy” as other work but has significance both to the literature and to the practitioners. I’m inclined to accept the paper.

limitations_and_societal_impact: The limitations are properly discussed by the authors.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The low-fidelity video encoder optimization method is proposed for TAL.The proposed methed outperform the state-of-the-art methods in most cases. It proposes that the traditional two-stage transfer lrarning pipeline has a task discrepancy problem since  the video encoder from the first stage of action classification would be less sensitive to the temporal boundaries of actions.
main_review: The paper is well writen. The graphs and equations are clear. However, the idea of the paper is easy to get and seems trivial. It only decrease the spatial or temporal features in the feature maps to ensure the model can be trained under the hardware constraints. Moreover, a sampling grid similar to the multi-grid training strategy is also proposed as short and long cycle strategies.  I believe that the innovativeness of the paper is not enough. 
limitations_and_societal_impact: The authors addressed the problem to some extent. However, maybe more innovative method should be considered to cope with the proposed  task discrepancy problem.
review_title: null

Meta-review: The paper initially received mixed reviews, and after the rebuttal the review consensus has slightly converged to the reject opinion. The reviews recommend that the following points should be addressed for the future stronger submission: (i) more thorough experimental evaluation with more baselines and (ii) better justification about the significance of the proposed idea.
Decision: Accept (Poster)