Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper studies sparse training with dynamic sparsity in the context of extremely sparse neural network models, trained from scratch, as function approximators for deep reinforcement learning. Consequently, the paper proposes a new sparse training method for deep reinforcement learning, named the Rigged Reinforcement Learning Lottery (RLx2). According with the experimental evaluation on four continuous control task environments, RLx2 seems to perform considerably better than the sparse baseline methods considered. Moreover, it can also outperform in a handful number of cases the dense equivalent.
strength_and_weaknesses: Strength:
* Timely, relevant, and under-studied topic, particularly for deep reinforcement learning. If enough efforts are put into it, and if it will become more mature, it has the potential of seriously decreasing the costs associated with deep reinforcement learning. Counterintuitively, as shown in the paper, it also has the potential of improving the state-of-the-art performance in deep reinforcement learning.
* The proposed method has a good level of novelty.
* The empirical validation is well-designed. The results show that the proposed method can outperform the baselines.
* Well written paper

Weaknesses – in my opinion, the paper doesn’t seem to have major flaws. Further, I would raise some points for discussion which are intriguing me:
* Why the Humanoid environment hasn’t been considered in the experiments? I assume that at this very high sparsity levels, given its large state space, the performance may be far from optimal. Can you consider adding an extra small experiment on Humanoid where you can study two network architectures (one like in the paper and one with more hidden neurons to compensate for the high sparsity regime) for RLx2 and the typical baselines used in the paper?
* From Table 2, I see that the proposed method (RLx2) consistently improves the dense SAC baseline in terms of performance even when over 90% sparsity is considered. At the same time, on TD3 the gain induced by RLx2 is considerably smaller. Do you have any idea why RLx2 impacts SAC in a different manner than TD3?

Other comments:
* In table 1, I believe that it is worth also adding the work of Graesser et al., 2022 for a more complete overview.
* page 4, I believe that “…links with the smallest weights.” is not accurate for negative weights. According with Alg 1, line 9, it is rather about … links with the smallest absolute value of the weights… 
* page 5, “…Here we adopt the methods in (Frankle & Carbin, 2019) for…”. The following procedure is not exactly the one from the mentioned paper. I believe that it would be informative to clarify exactly what was taken from that paper and what was added by you.
* page 6, Dynamic capacity buffer – According with the description, the buffer size (capacity) is always increasing? Is my understanding correct?
* page 7, “…SET (Bellec et al., 2017),…” is not correct. SET was proposed by Mocanu et al., July 2017, if referring to the first arxiv version, or 2018 if referring to the year of formal publication.
* page 8, “…which replace the topology evolution scheme in RLx2 with Tiny, SS and SET, while keeping…” may be ok, but it is not sufficiently clear to me what are the changes as some of those methods are static and some are dynamic (evolves). Perhaps you can clarify it better in the paper.
* page 9 – Conclusion – a fair paper, with a good level of self-criticism, discusses also a bit of the limitations of the proposed methodology, while possibly indicating future work directions. These are missing in this paper.
 * page 25 – Appendix C.5 – Figures 9 ad 10 – it may be useful for the reader to see also somehow the sparsity levels in those plots without the need of scrolling back to Section 5.1


clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is clear and reads well, in my opinion.

Quality: The paper has a good quality level, and it seems to be well executed.

Originality: Good level of novelty as reflected also by the proper related work discussion.

Reproducibility: I believe that it may be possible (but not straightforward) for a reader with a good level of experience on the topic to reproduce the proposed methodology given just the paper details, if enough time is put into it. Anyway, the authors promise the release of open-source code if accepted (but it is not presented in the supplementary material) and this shall increase seriously the reproducibility levels. 

summary_of_the_review: Overall, I believe that the paper has a good level of novelty, with good results, and a good presentation. I am curious to see the authors’ opinion about the comments raised above.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Sparse neural networks that have an order of magnitude fewer parameters than a dense network without a significant drop in performance can greatly reduce computational requirements of reinforcement learning (RL) agents, allowing their deployment on resource-limited devices. Different sparse network topologies can have drastic differences in performance. A method for the training of sparse networks from scratch, known as RigL, learns the topology throughout training by dropping low-magnitude weights and growing connections that have large gradients. This paper extends RigL to the deep RL domain by adding two techniques for making the value function estimation more robust. One technique is using multi-step temporal difference targets instead of one-step ones. The other is dynamically changing the size of the replay buffer if its stored actions get too far from the current policy. The paper demonstrates with examples that robust value learning is essential for improving the performance of RigL since, in part, the value function affects the gradient-based topology learning of RigL. Experiments with RLx2 using SAC and TD3 algorithms on four mujoco environments show significant model compression with minimal loss in performance.
strength_and_weaknesses: The RigL method learns a sparse topology from scratch. Unlike other model compression methods, there is no need for pre-training a dense network, which markedly reduces the computational requirements of both training and inference. RLx2 could therefore greatly benefit online learning problems on resource-constrained devices. I appreciate that the author has validated the importance of robust value estimation and its effect on the topology learning of RigL.

There are two main weaknesses in the paper. The first is the lack of proper sensitivity analysis for some of the new algorithm hyper-parameters. Although the hyper-parameters are the same for all four environments, it is unclear how sensitive the performance is to each hyper-parameter and how the sensitivity differs between environments. These hyper-parameters include the initial mask update fraction, mask update interval, buffer adjustment interval, buffer policy distance threshold, and multi-step delay.

The second weakness relates to the computational complexity of the algorithm. The topology learning of RigL consumes O(N logN) time with N as the number of weights in the corresponding dense network. Although the topology is updated every 10000 steps and the cost may be negligible compared to the main RL algorithm in these experiments, 10000 is still a constant. With the exponentially increasing number of parameters, in the future the computational requirement of the topology learning step might begin to dominate and grow faster than the main algorithm, making it less scalable. It would be great if the dropping and growing of connections could be done locally for each connection instead of requiring a global sort. This issue concerns the RigL method adopted by this paper and not the techniques introduced in this paper for making the value estimate robust.

Here are some more questions and comments:
* What do the shaded areas in the plots represent?
* Did you experiment with using multi-step TD targets from the beginning? What did you observe?
* What happens if you do not anneal the update fraction, but keep it constant at a smaller value?
* Should the two robustification techniques be applied to the original algorithms even when not using RigL?
* “We also show in Appendix C.2 that the performance of RLx2 is insensitive to the policy threshold.” I do not see any sensitivity plots in C.2.
* “Upon any update to \phi and appending new transitions to…” Is this sentence correct? Was the buffer adjustment not done every \Delta_b steps?
clarity,_quality,_novelty_and_reproducibility: The paper is clear, well-written, and makes novel scientific contributions. Pseudocodes of algorithms and their hyper-parameters are provided in the appendix.
summary_of_the_review: The paper establishes that having a robust value estimator is important when learning the sparse network topology using RigL in Deep RL. The method trains a sparse network from scratch and achieves significant model compression with minimal loss in performance. Claims are backed up with experiments. The two introduced techniques are simple yet effective for applying RigL to Deep RL. The paper can be improved by adding sensitivity analyses for the new hyper-parameters as discussed previously.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes to learn an efficient sparse network for reinforcement learning (RL). It learns the sparse network from scratch instead of relying on knowledge distillation or pruning. The proposed method is designed based on the gradient-based topology evolution criteria. Experiments show that the proposed method can learn very sparse networks with satisfying performance. 
strength_and_weaknesses: Strength
- The proposed method is carefully designed and well implemented. It can maintain good performance with very high-level sparsity on different datasets. The proposed method integrates the topology evolution method into the RL task very well. 
- The details of the algorithms and experiments are discussed and analyzed carefully, making the work easy to reproduce. 
- The paper is easy to follow. 

Weakness
- The proposed method is an integration of the RigL method into the RL task and framework. The major contribution is on how to integrate it into the RL framework. This limits the significance of the proposed method, especially the lottery ticket based method for sparse network learning has been studied wildly.
- Apart from RigL, the authors may consider studying how different sparse network learning methods can help RL in what kind of way. This may generalize the proposed techniques to a more general sparse network based RL framework. 

clarity,_quality,_novelty_and_reproducibility: The whole paper is written clearly. The work is with good reproducibility with a detailed introduction of the details. Considering that the core method is RigL, the novelty is limited at some level. But the studies on the RL are in detail. 

summary_of_the_review: The major contribution is about how to integrate the RigL based sparse network learning approach into the RL framework, which limits the novelty. But the work conducts very detailed design and studies and analyses of the sparse network learning for RL. Many learning strategies are investigated.  

The rebuttal addressed my concerns. I am happy for this paper to be accepted.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents the first work to train a sparse DRL agent from scratch. The major contributions are that the robust value estimation and effecient topology evolution are important to train a sparse DRL agent, and this paper proposes to leverage the gradient for network topology evolution and multi-step TD-target with dynamic buffer for robust value estimation. Experimental results are presented using two off-policy TD learning DRL models, and show the best performance over existing approaches.
strength_and_weaknesses: Strengths:
- Training ultra-sparse DRL model from scratch is important while underexplored topic in practice.
- The contributions of this paper are clearly summarized, and the motivations are grounded in empirical observations.
- Experimental results show significantly better performance than the listed baselines.
 
Weaknesses:
- Since the one-step and multi-step TD-targets are used in a hybrid way in training, it's not a proper claim that a multi-step TD target could overcome the issue of inaccurate value estimation from a sparse network. 
- Also, from the Eq. (4), it's unclear why the multi-step TD-targets could overcome the unreliable and inaccurate value estimations.
- The dynamic size of replay buffer is controlled by a pre-defined threshold on the policy distance, which is kind of heuristic in practice. 
- Moreover, transitions are actually **randomly** selected from the replay buffer (to keep samples iid) rather than using the oldest top-K transitions. Therefore, the policy distance measure (Eq. (6)) may not effectively evaluate the inconsistency between behavior policy and current target policy.
- From Table 3, it's curious to me that the performance degrades when more than 3 steps are used. On the Hal. and Ant. benchmarks, the best number of steps is 2. Since it's a consistent observation that the performance would degrade with too many steps, this paper should present more in-depth discussion to explain this phenomenon.

 
clarity,_quality,_novelty_and_reproducibility: - The organization is clear and the technical approaches are well motivated.
- A little bit overclaim. The scope of the method in this paper should be in the off-policy TD learning area rather than the entire DRL community. For other DRL methods, such as the on-policy methods, it's unclear if the method is applicable or not. 
- The novelty of method may be limited. Though I acknowledge the interesting empirical observations, the proposed technical solutions (gradient-based topology evolution, multi-step TD-targets, and thresholding the policy distance) are kind of trivial in practice.
summary_of_the_review: Beyond the technical side, this paper is of good readability (well organized, empirically grounded motivations, etc). However, in terms of the technical novelty, the proposed solutions are a little bit trivial in sense.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work addresses the important problem of learning a reinforcement learning (RL) agent from scratch with a sparse network throughout. Recent development in sparse connection learning, for example RigL, allowed the authors to aim for this ambition. The authors proposed network-topology evolution through RigL, but recognized that robust value learning is important for good performance under sparse networks, which they enabled through a multi-step TD target and a dynamic-capacity replay buffer. The authors show substantial model compression and acceleration of training and inference with only a slight performance loss. The work successfully and clearly makes a significant contribution to an important problem; likewise, I recommend accepting the paper. I suggest mentioning the super-linear computational complexity of the topology evolution in the paper.


justification_for_why_not_higher_score: The work benefits substantially due to a prior method, RigL, making the work an obvious win but not among the most excellent ones.

justification_for_why_not_lower_score: The work is an important contribution to learning sparse networks for RL agents.
Decision: Accept: notable-top-25%