Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: **Update after reading the other reviews and authors' responses:** Some valid criticism has been raised and addressed by the authors to a reasonable degree (given e.g. the limitations of a conference-format paper). Given all current information I remain in favor of accepting the paper (my score and confidence remains unchanged).

**Summary:**

The paper investigates the lottery-ticket effect in detail in reinforcement learning tasks. While it has been shown before that lottery-tickets also exist in deep reinforcement learning, this paper performs a thorough analysis using (i) both discrete and continuous action- and observation-spaces, (ii) an on- and an off-policy RL algorithm (as well as a behavioral cloning baseline), and (iii) careful ablations to distinguish the importance of the binary mask and the initial weight values which together form a winning ticket. Various controls and ablations are performed, leading to a plethora of experimental results. The main result is that winning tickets can be found efficiently via iterative magnitude-based pruning in RL tasks, and that in many tasks the binary mask contributes more to performance than the initial weight values. Compared to the supervised (behavioral cloning) baseline, RL performance degrades more rapidly with increasing sparsity levels which is attributed to the distributional shift introduced by exploration in RL. Finally, the paper also investigates the effect of pruning (in winning tickets) of the first-layer weights, and shows that in many cases multiple input dimensions (or regions, which are often semantically meaningful) can be pruned without loss in task performance. This observation could be an interesting approach to producing sparse input representations in RL.

**Main contributions**

1) Investigation of the lottery ticket effect in RL. Significance: though it has been reported before that winning tickets can be found in RL, the paper performs an impressive amount of empirical investigations, including many baselines and ablations across a range of continuous and discrete tasks and an on- and off-policy RL algorithm as well as a supervised baseline. Naturally, results vary somewhat between settings and variations, but the sheer number of results allows to extract trends that hold across many variations. This significantly increases the reliability of the findings.
2) Disentangling the lottery ticket effect through control experiments that allow to attribute the effect strength to (i) the binary mask, (ii) the initial weight-values, or (iii) the layer-specific pruning ratios, which are all combined in a winning ticket. Significance: these control experiments shed some important light on the role of the three components. Interestingly, the binary mask seems to play the most important role in RL tasks, whereas initial weight-values seem to matter more in the supervised setting. This is an interesting, and very consistent insight which might spawn further investigation into the topological structure induced by the winning mask.

3) Investigation of first-layer weights of winning tickets. In many cases whole input dimensions (in continuous tasks) or input regions (in discrete tasks) that are irrelevant to the problem are pruned. This reflects an important (implicit?) regularizing effect - the corresponding weights remain at low magnitudes and can be safely pruned. Investigation of the pruning patterns or pruned dimensions also provides semantic insight which allows for interesting hypotheses regarding how the network solves the task and which information it ignores. Significance: this is an interesting artifact of the analysis of winning tickets in RL. Results are promising and ask for further investigation and perhaps even formulation of a method for sparse/compressed input representations for RL (the identified irrelevant dimensions also transfer to non-pruned architectures).

main_review: **Quality, Clarity, Soundness, Correctness**

The paper is an impressive empirical study covering a vast range of variations, baselines and ablations. All experiments are repeated multiple times and std-deviations are reported which helps with judging the stat. significance of findings. The necessary background and related work is briefly, but well summarized. Experiments are and findings are well presented - though given the sheer number of experiments and results some parts of the paper appear a bit crammed (but there is no easy solution to this given the limitations of a conference-format paper). Experiments and the conclusions drawn appear to be correct to me - though, sometimes aggregate results like Fig 1 right, bottom do not strictly hold for all experiments that went into the aggregate result (Fig 4 bottom row, where there is no very pronounced difference between the red, green and blue lines). My one criticism is the hypothesis that attributes distributional shift due to exploration as the reason for why pruning rations (without performance degradation) in RL are typically lower than in supervised learning. While the hypothesis is sensible, it is never formally verified, and no alternative explanations are presented (one simple control could be to train the supervised baseline on data that includes the various stages of the RL agent, including early exploration phases, rather than just using “expert” trajectories).

**Verdict**

Overall I think this is a great empirical paper. Experiments are plentiful, well executed and with an impressive amount of relevant baselines and controls. While the lottery ticket effect has been investigated in great detail in non-RL domains before, and many ideas and inspiration for experiments could be transferred from previous work, I do not think this limits the merit of the current paper. Important empirical verification was carried out and results were presented in detail, and with some interesting observations. The paper lays the groundwork for now digging deeper into some of the findings. I have some small suggestions for improvements, but am currently in favor of accepting the paper. Looking forward to hearing the other reviewer’s opinion.

**Improvements**

1) As stated earlier, my main criticism is that the “distributional shift” hypothesis is never verified and no alternative explanations are provided as to why pruning rates in RL seem to be generally lower. At the current stage I would phrase the finding as a (sensible) hypothesis that needs further investigation rather than an established finding.
    * 1a: One control experiment that comes to my mind is to run the behavioral cloning baseline on data across the full “lifetime” of an RL agent, including the early exploration phase and thus exhibiting the same distributional shift throughout supervised training. 
    * 1b: One alternative explanation not related to distributional shift is that the supervised and RL learning signal might be qualitatively different and thus lead to the different pruning rates (not entirely sure how to test that though). 
    * 1c: If the hypothesis is true there should also be observable qualitative effects for tasks where this distributional shift is large compared to tasks where the distributional shift is small.
    * 1d: See 3 for more alternatives related to exploration.

2) Another limitation that should ideally be noted more explicitly is that most findings are based on PPO (an on-policy algorithm) with fewer experiments for DQN (off-policy). Large parts of the paper are written as if the findings were established in general about RL. While the paper certainly does a great job of supporting the main claims with broad sets of experiments, I think it would be nice to point out a small hint of caution (e.g. in the limitations section) regarding the generality of findings in the off-policy setting.

3) Clarification: how is exploration in the RL algorithms implemented? Is it baked in via a fixed epsilon-greedy action-selection? Is there some annealing of the exploration strength? 
    * 3a: If the policy has to account for exploration as well (e.g. through some entropy regularization) this might be another explanation why pruning rates in RL are lower - because the policy needs to be more complex than the supervised policy. 
    * 3b: If the RL algorithm is using fixed epsilon-greedy (no annealing), is this taken into account when generating the expert data for the behavioral cloning baseline (i.e. using an expert with the same epsilon)? If not, the data-distribution for RL (even at the end of training) is potentially much broader compared to the behavioral cloning baseline which could also be a reason for the lower pruning rates in RL.

summary_of_the_review: An impressive empirical paper that investigates many findings regarding the lottery ticket effect found in other domains to RL tasks and algorithms. The paper presents results for a large number of tasks (with continuous and discrete observation- and action-spaces), and loads of relevant controls, baselines and ablations. The main finding is that iterative magnitude-based pruning can effectively find winning tickets in RL, and that there are some qualitative differences between RL and supervised learning (masks seem to be more relevant than initial-weight values of winning tickets). I only have some small criticism regarding one hypothesis, and whether it can be stated as empirically verified given the current set of experiments. Currently I think the paper is interesting and relevant to a large part of the ICLR community and vote in favor of accepting the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the Lottery Ticket hypothesis in the context of deep RL for identification of sparse task representation in low-dimensional control tasks. This is primarily an empirical investigation where several experiments and consequent analysis reveal what a "winning ticket" means in case of policy models or Q function approximators given that Deep RL is prone to gradual distribution shift. The experiments also help to identify how standard magnitude pruning will behave under different environment updates, feedback schedules or initialization dynamics.  
main_review: Strength

1. The paper targets the important problem of dimensionality reduction of task representation which helps in improving performance of several applications where inference response plays a key role.

2. Such empirical studies are extremely important and it provides valuable insights to researchers interested in model optimization in the context of control problems. This study will definitely motivate further work into developing newer classes of sparsification/pruning algorithms in the context of deep RL. 


Weakness				

1. The intent of the paper is at times difficult to realize .. since the paper has not been structured well. The contributions listed in the introduction does not align well with the arrangement of the sections and respective discussions. 

2. Many of the experimental details are pushed to the appendix. This is an important study and researchers will want to try their hands on the the actual experiments and validating their results against the results presented here. So more details on experimental settings .. choice of exp env. ... system reqs etc. is needed.

3. Novelty over state of the art is not clear. While it is understandable that this is an empirical analysis, the reader will still expect at least one of 2 things 
       3a. One liner empirical outcome. As in what is right / wrong / broken/ beautiful  about the thing there is being studied
       3b. What can be done about it? A proposal of some sort at least. 

summary_of_the_review: This an important study and analysis that will motivate further research. However the current version has some major concerns that the authors need to address.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper empirically studies the lottery ticket hypothesis in deep reinforcement learning. It confirms the existence of lottery tickets in RL agents, and provides a detailed study of lottery tickets in both control problems and supervised behavioural cloning problems to identify the effect of RL-specific distribution shifts and to evaluate properties of the weight masks produced by iterative magnitude pruning.

main_review: ### Strengths

- The principal strength of the paper is in the thoroughness of its empirical evaluations. 
    - In particular, the paper identifies a key source of the distinction between RL and supervised learning relevant to lottery tickets: that of distribution shifts induced by the policy improvement operator over the course of training in RL, something absent in supervised learning.
    - The distinction between policy-based and value-based RL is important and I was happy to see both value-based and policy-based agents included in the experimental evaluations.
    - Similarly, I am aware of many papers which study the effect of variations on the IMP masking procedure (e.g. Frankle et al. 2019, 2020; Yu et al., 2020; Zhou et al., 2020), but the clarity and comprehensiveness of the ablation in this work makes it a worthwhile contribution to the literature. I also appreciated that the effect of convolutional layers was discussed in the main paper, and that late rewinding was also evaluated in the appendix.
- The connection between magnitude pruning and state abstraction, particularly in low-dimensional input settings, presents an interpretable and intuitive visualization of lottery tickets.
- The paper is clearly written and it is easy to see how the experimental results justify the claims made in the text.
- While prior works have studied lottery tickets in RL (Yu et al., 2020), this work is distinguished by its focus on the interaction between the non-stationarity inherent in RL and magnitude pruning methods. 
- The paper provides useful recommendations for training sparse networks in RL tasks, such as noting the importance of the weight initialization scheme in determining how aggressively inputs are pruned.

### Weaknesses

- The focus of the paper at the moment seems to be “about” the LTH, with RL as an experimental condition in which to study it. In contrast, I think the paper also has the potential to tell us a lot about properties of deep RL agents using the LTH as an experimental condition. The degree to which a network can be pruned seems to be telling us something really interesting about the complexity of a given task, and I think there’s potential to draw a deeper connection to prior work studying non-stationarity in RL. 
- The paper dives quite quickly into its findings, but leaves some gaps in the motivation of its investigation. At the moment, my interpretation of the motivational paragraph is that it essentially says “People are interested in lottery tickets but haven’t looked at them properly in deep RL yet, so that is what we will do”. A sentence or two justifying why looking at lottery tickets in RL is interesting and important, beyond simply an absence of existing literature, would greatly benefit the paper. For example, would lottery tickets enable more efficient control systems? Would this investigation tell us something important about deep neural networks? 
While it is interesting to look at “retraining” a pruned network from scratch on an RL task, it’s not clear that this is the best approach to sparsify RL agents. As the motivation of IMP is largely to find computationally efficient sparse subnetworks that perform well on the target task, finding a maximally sparse network which attains high return in the target task seems desirable. In this case, presumably a better approach to obtaining sparse agents would be to distill the sparsified network on the outputs of the dense network as a form of supervised training, rather than forcing the network to maintain the capacity necessary to fit all of the intermediate policies and value functions necessary to reach the final high-performing policy.  
- I am somewhat skeptical of the interpretation of the convolutional masks in Figure 6. While I can see how each of the labels might correspond to their respective mask, I think it might be reaching a bit to say that particular horizontal lines correspond to bullets as opposed to cannons. I agree with the overall take that magnitude pruning preserves semantically significant components of the filters, but in contrast to the thorough and careful analysis of the rest of the paper, the mapping from the mask on a particular filter into a specific interpretation seems less than rigorous.


### Potential Improvements/Questions
- While behaviour cloning is one way to introduce supervision into RL tasks, I would also be interested in seeing what happens when policy evaluation of an expert policy is used as the supervised task, to complement the comparison between policy-based vs value-based agents trained with policy improvement.
- A priori I would have thought that the value prediction networks would be more sensitive to pruning than policy networks, and the paragraph at the top of page 6 seems to support this intuition. I think it would be easier to parse the paper if it is made clearer in the figures which pruning scheme is used in which evaluations. 
summary_of_the_review: Overall, I think the careful analysis and clear presentation of the findings of this paper make it a valuable contribution to both the lottery ticket hypothesis literature as well as to our understanding of non-stationarity in deep RL. I therefore recommend that it be accepted to ICLR.
review_title: null

Meta-review: This paper studies the Lottery Ticket hypothesis in reinforcement learning for identifying good sparse representations for low-dimensional tasks. The paper received initial reviews tended towards acceptance. However, the reviewers had some clarification questions and concerns. The authors provided a thoughtful rebuttal. The paper was discussed and most reviewers updated their reviews in the post-rebuttal phase. Reviewers generally agree that the paper should be accepted but still have good feedback. AC agrees with the reviewers and suggests acceptance. However, the authors are urged to look at reviewers' feedback and incorporate their comments in the camera-ready.
Decision: Accept (Spotlight)