Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposed a new loss function that combines KL loss and Huber loss and can be used in actor-critic-based teacher networks.
They designed a 3-phases algorithm to distill knowledge from a full-precision teacher network to a low-bit student network.
They discussed the influence of the choice of teacher algorithm.
strength_and_weaknesses: Strengths:
1. A loss function is proposed to distill a dual-head network, which is suitable for actor-critic-based teacher networks.

Weaknesses:
Major
1. This paper claims they proposed a new Distillation loss and detail it in Section 4.1. 
However, instead of using this proposed loss, the proposed QPD algorithm only uses the traditional distillation loss.
Do you have any reason for this?

2. The proposed loss simply combines two existing loss functions, and the novelty is limited.

3. The experiments can not evaluate the effectiveness of the proposed methodology.

4. The writing should be improved since some sentences and tables are confusing.
clarity,_quality,_novelty_and_reproducibility: The paper is poor in clarity, quality, and novelty. 
I can not foresee the reproducibility of this paper.
summary_of_the_review: This paper proposed a new loss function that can be used in the dual-head network.
But the writing is poor, and the novelty is limited.
So I recommend rejecting this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This submission proposed to generated a compact and quantized version for actor-critic type reinforcement learning model. Basically, it first distilled a full-precision student model by introducing an auxiliary loss for critic model. Then the learnt model is served as initializer for Quantization-Aware Training (QAT).
strength_and_weaknesses: Weaknesses:
1. The distillation is decoupled with QAT process: they are conducted in a pipe-line fashion. Though author claimed that the submission is a smoother transition, a more valuable work would be solving the instability of `Quantization-aware Distillation'. Directly applying distillation-quantization makes the work less innovative.
2. Quantization is decoupled with reinforcement learning: the quantization phrase is more like an application of quantization (that works in other domains such as computer vision and neural language processing) to reinforcement learning. DoReFa and STE does not consider any properties that inherits in actor-critic models.

Strength:
1. The distillation method and analysis somehow enlight reinforcement learning understanding. Author may pay more attention to distillation part, instead of adding quantization if it is decoupled with reinforcement learning.
clarity,_quality,_novelty_and_reproducibility: 1. Overall, it is a litte hard to follow as too many analysis is accompanied with the method description.
2. Novlety is limited in the application of quantization.
summary_of_the_review: Basically, author should 1) pay more attention to distillation, including analysis. 2) Get rid of quantization if it is just a plugin for the method. 3) make progress in integration of distillation and quantization in reinforcement learning scenario.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose to apply knowledge distillation and model quantization techniques to Deep Reinforcement Learning (DRL), which contributes to less memory footprint and space complexity during training. They show that the dual-head architecture leads to superior performance than the common actor smoothing and policy distillation. The experimental results reported in Figure 4 seem to further support the effectiveness of dual distillation. 
strength_and_weaknesses: # Strength:
- The results of "these combined methods can effectively compress a policy network down to 0.5% of its original size, without any loss in performance" seem encouraging. Though the authors mainly focus on the deployment, I personally think the proposed scheme may shed some light on the efficient training process of DRL, considering that the dual head architecture may serve as a self-distillation training pipeline to accelerate the convergence of both actor-network and critic-network. The idea of *AUXILIARY CRITIC LOSS* is interesting.
- The proposed method is quite simple and easy to follow, which may help to reproduce the results reported in this paper.
# Weaknesses:
- ### Major Comments:
  - As far as I can tell, the novelty of this paper can be limited given the current manuscript. The quantization and distillation methods have been proposed in previous model compression works [1], and are even widely discussed in the DRL community [2,3,4]. The only difference between QPD and pioneering approaches is the AUXILIARY CRITIC LOSS. However, the ablation study on the AUXILIARY CRITIC LOSS seems missing. It remains unclear whether the performance improvements come from the dual-head architecture or the CRITIC LOSS. 
  - The authors elaborate on the benefits of "Distillation" $\rightarrow$ "PTQ" $\rightarrow$ "QAT". However, it seems that the authors use the same quantization strategy for “PTQ" and "QAT", which makes the Algorithm degrade into "Distillation" $\rightarrow$ "QAT". From my point of view, the overall training pipeline of Algorithm 1 is not new. It is common knowledge to equip "Distillation" with "Quantization" simultaneously or successively.
- ### Minor Issues:
  - According to Figure 7, it is obvious that network quantization results in negligible performance drops but the effect of distillation is hard to be found. Though Figures 4-7 have illustrated that "an auxiliary loss noticeably impacts the internal representation", more quantitative analysis on the improvements over strong baseline methods are encouraged, e.g., the real-runtime speed, memory footprint, model size, training time cost, and average return. 
  - What if we further decrease the value of $\lambda$? Besides, it would be better to report the selection of $\tau$ in this paper.
  - I failed to find the descriptions of the experiment settings, such as the dataset, metrics, etc. Did I miss anything? 
  - It seems that the authors conduct all experiments under the same circumstance. I expect more results reported on popular benchmarks with mainstream DRL methods. 
  - "Our experiments show that these combined methods can effectively compress a policy network down to 0.5% of its original size, without any loss in performance" with a $200\times$ compression ratio seems inconsistent with Table 3.
  - It would be better to include a brief introduction to the "Atari Breakout environment" in Figure 1. 

# Reference:
- [1] QKD: quantization-aware knowledge distillation. arXiv2019
- [2] Policy distillation. ICLR2016
- [3] Distillation strategies for proximal policy optimization. arXiv2019
- [4] Quantized reinforcement learning (QUARL). arXiv2019
clarity,_quality,_novelty_and_reproducibility: Since the proposed methods are rather simple, the clarity and reproducibility seem ok. However, the quality and novelty are my major concerns. From the view of model compression, the real contribution of this paper can be quite limited. The technique used in the paper has been common knowledge in network quantization and distillation (i.e., Section 4.1.1 and 4.2.2). Though the AUXILIARY CRITIC LOSS seems interesting, the experiments are too weak to support the contribution of this paper. I am not sure whether the proposed scheme could be generally effective in DRL.
summary_of_the_review: The main track may not be the best venue for the current manuscript. The authors are encouraged to solve the weaknesses and re-submit this paper to the following conference.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper received controversial appreciations: it implements an interesting underlying idea, especially in the context of distillation and reinforcement learning.
However, there are common criticisms regarding the weak novelty, especially in relation to distillation (which is deemed standard), limited technical contributions, insufficient experimental analysis, including ablation study, and also about the clarity of the presentation. 
Authors provide a rebuttal, but reviewers are still considering the paper not ready for publication, all giving ratings below threshold.
For these reasons, this work cannot be accepted for publication to ICLR 23.
Decision: Reject