Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The main aim of this paper is to increase the efficiency (in terms of wall clock time) of Transformer based models for reinforcement learning.  

Previous works have shown Transformer-like models to be highly performant across a range of domains, including recent results on reinforcement learning (with gated Transformer-XLs). However a drawback of these models  (over say, LSTMs) is their relatively slower inference speeds. This is especially a problem in RL settings where Actors are typically run on CPUs (not GPUs) and send trajectories to a central learner. The problem here is two-fold: 1) slower overall training time due to high latency with the learner waiting on actors 2) slower inference post training in latency sensitive deployment settings like robotics/other control based settings. 

The solution proposed here (the method is referred to as "Actor-learner distillation") is to instead use LSTMs for acting and Transformers for learning. Typically the learner would send parameters to the actors every update -- as this is not possible in this hybrid approach, a distillation loss is instead suggested as a means of updating the actors (with a replay buffer). The authors successfully show that this transfers both "good policies" and the relevant inductive biases from the Transformer as well allowing fast inference as expected. Further, the "off policy"-ness of the model does not turn out to be an issue in the domains considered. The results are promising and the the writing very clear. A few points below:

* One question I have is whether the authors ran an experiment replacing the LSTMs in the actors simply with smaller transformer models? For example, 2 layers instead of 4 or smaller embedding sizes? This would also allow faster inference while inherently retaining some of the inductive biases. 

* As a second baseline: while this wouldn't solve the issue of slower training, one could achieve faster inference of the trained model simply by distilling the final Transformer into an LSTM (and then optionally fine-tuning). Are the authors able to report numbers for these?

* Further, in between the two extremes of only LSTMs acting or only Transformers one could also consider more hybrid approaches that start with a Transformer to learn a very good starting policy in a few steps and then distill this into an LSTM online similar to work here (https://arxiv.org/pdf/1806.01780.pdf). The worked linked here performs "online distillation" combined with a convex combination of model outputs which would then negate the need for a replay buffer (though would require running both models on the actor until fully switching over to the LSTM). 

* While the results are conclusive, the work could be improved by running on larger (or 3-dimensional) environments as considered in the original Gated Transformer paper this builds on. 

* Lastly, are the authors able to provide possible future directions to be considered that could boost this work further? 

Overall this is well written paper exploring a conceptually simple but well-implemented and important idea. As models grow in size, techniques like this will be very important to allow scaling up of RL capabilities. I would recommend accepting this paper. 
review_title: A well written paper exploring a simple but sensible idea

final_score: rating: 7: Good paper, accept
review_content: ##########################################################################

Summary:

The paper proposes a solution to actor-latency constrained settings in RL by using policy distillation to compress a large “learner model” towards a more tractable “actor model”. In particular, it proposes to exploit the superior sample efficiency of transformer models while utilising an LSTM-based actor during execution. The proposed procedure, called Actor-Learner Distillation (ALD), provides comparable performance to transformers in terms of sample efficiency, yet produces a wall-clock run-time that's on par with LSTM agents.

##########################################################################

Reasons for score:

I'm voting for accepting the paper. I like the idea of policy distillation between transformer-based learner and LSTM-based actor in the distributed RL framework. The paper is written with great clarity and is easy to follow. I don't have major concerns regarding this work but hope that the authors can address my minor concerns in the rebuttal period.

##########################################################################

Pros:

1. The work addresses an important problem for distributed RL: whether it is possible to utilise large model capacity while still acting within the computational limits imposed by the problem setting and do all of this during the training. The problem itself is real and practical, e.g., in robotic operations with limited computation resources.
2. The process of policy distillation between transformer-based learner and LSTM-based actors is described in great detail and provides additional insight into concrete decisions made for the method, such as the value distillation loss for the actor model and Distil/RL step ratio.
3. This manuscript includes thorough experimental of the proposed method that includes both qualitative analysis and quantitative results. The choices regarding the two environment tasks and baselines are reasonable.

##########################################################################

Cons:

1. Regarding the DpRL hyperparameter (actor SPS / learner SPS), it isn't very clear to me how this ratio has been manipulated for instance in Figure 3. Is this achieved using modifications of actor/learner size? I'd appreciate it if the authors could add clarification on this.
2. On the result figures where the X-axis corresponds to Wall-Clock Time, it isn't clear to me what was used as a basis for choosing the maximum range of the X-axis. For example, Figure 4 (Right) was cut after 5 minutes, where ALD is much higher than the baselines. However, I'd be interested to know when do ALDs and GTrXL (which achieves the same result in eventually) meet on the diagram.

##########################################################################

Questions during rebuttal period:

Please address and clarify the cons above
review_title: Actor-learner distillation for distributed RL

final_score: rating: 7: Good paper, accept
review_content: ### Summary

The paper proposes an original idea to use distillation to speed up modern distributed RL settings, when data collection is done on CPUs with the learning happening on accelerated hardware, e.g. GPU. More specifically, the authors propose to use a transformer for the learner and distil the policy into LSTM actors. With this, they achieve much faster wallclock time compared to the Transformer for Actors setup, however, losing in sample-efficiency.

### Pros

 - The paper tackles a practical problem arising when designing an RL learning pipeline.
 - The paper proposes an original idea of applying distillation in the RL setting to solve a problem above.
 - Implicitly (because there is no discussion of this), the paper raises an important question of a trade-off between sample efficiency and computational efficiency in RL.

### Cons

- Confusing positioning of the paper;
- Implicit assumptions not discussed/challenged in the paper;
- Confusing results presentation (plotting, result interpretation);

### Reasoning behind the score

I believe this paper has great potential and can be improved in two possible ways. First, which will make it more general and increase its impact, is to turn it into a discussion of the trade-off between sample efficiency and computation complexity. Second, to narrow the scope and improve the positioning/clarity to get an improved version of this submission. I would really like the authors to choose the first route, but it is their decision.  At this moment, due to the pros and cons I described above, I give it a 6, Marginally Above Acceptance Threshold.

### Questions to the authors

- The paper assumes that it's better to lose in sample efficiency while computing faster. Why do you think it should be like this? What is the motivation behind such an assumption? Can you discuss the sample/computation efficiency trade-off in the paper?
- The paper assumes that LSTMs are less computationally efficient than Transformers. What exactly do you mean by computational efficiency? Yes, your plots show that the particular model instances make LSTM to achieve better performance with smaller wall-clock time. But is it general? One of the reasons behind transformers' success is their ability to parallelise. Can you explicitly state the assumptions for which you are solving the posed problem?
- The storyline in the abstract and in the intro/conclusion do not quite fit. For example, the abstract goes from distributed RL setting → constrained actors → cannot train more complex models. But in the intro, you start with the 'Transformers are great' story. In addition, in the second paragraph of the intro, you mention 'actor-latency-constrained' settings when 'there exists a max latency constraint on the acting policy'. I don't think you consider this setting later in your experiments or before (in the abstract). What is the exact problem you are solving?
- "An agent designer has to make a hard decision between ... better sample efficiency ... or ... lower wall-clock time and reduced computational cost". Can you elaborate a bit more on the setting? Ideally, describe this trade-off in more detail. Sometimes, sample-efficiency and time are intertwined, if your data collection is slow, then sample inefficiency will bump the wall-clock time.
- I'm really confused by the presentation of the results. Can you, please, plot the curves till convergence of all the algorithms? For example, the left subplot in Figure 1 shows the transformer converged to 1.0, but not the LSTM. At the same time, the next subplot shows the LSTM converged, but not the transformer. I think this is of utmost importance for better interpretation of the results. Same holds for other figures.
- For the table on top of the page 3, how do you get the numbers? At which point in training did you do the measurements? Did you average?
- On page 6 you say "... we can observe that ALD achieves near sample complexity parity to the transformer reaching a success rate of 1 in less than twice as much steps." Why is it a near parity (2X!)?
- Intro says 'ALD provides sample efficiency on par with transformers'. This is not what the plots show (e.g. 2x difference on Fig 4, 12.5 vs 20 score on Fig 5 and 0.7 vs >0.8 in episode success on the 3rd subplot of Fig 5. What is your definition of 'to be on par'?

### Other comments

- Background
    - You cite Puterman for MDPs, however, Puterman does not include the discounting coefficient into the MDP tuple.
    - Can you give a page of Sutton&Barto for your reference for the existence of an optimal policy?
    - You say you focus on POMDPs, but later use $s$ as inputs to the policy. This feels weird.
    - End of the first paragraph of the background: 'recurrent LSTMs'. Are there non-recurrent LSTMs?
    - Great second paragraph describing the glossary, I really like it.
- Distributed Actor-Learner Distillation
    - I think, V-MPO comes too late here. I've had the question about the learning algorithm several times, before I found it here. The same hold for IMPALA. I think this should be mentioned in the introduction.
    - I like the description of the algorithms in the paragraph, however, I would love to have pseudocode in the appendix.
    - "which are uniformly sampled from the FIFO queue": How does FIFO affect sampling?
- Experiments
    - "For each environment, we run the actor and learner model architectures used in ALD as baselines, where each of these models are independently trained using standard RL". This sentence is a bit confusing. Also, can you explain what does 'standard RL mean?
    - "Comparison of ALD to Asymm." should be on the next line, I believe.
    - Why did you decide to do you present the per-seed curves on 15x15 maze while presenting the aggregated results for 9x9? How do these look on 9x9 maze?
- Related Work
    - It would be great to have a paragraph giving pointers to the existing distributed RL frameworks and some questions the papers introducing them investigate.

review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:

The paper proposes a method for "actor-latency constrained" settings: Recently, transformers have been shown to be powerful models in RL which, in particular, exhibited better sample complexity in settings in which long-term credit assignment in partial observability was required (e.g. the T-maze). However, they are computationally expensive. Consequently, the authors propose to train transformers on the learner, supported by hardware acceleration, but also train a smaller LSTM agent which can be efficiently executed on the actors. 

Positive points:
(+) I think this is a relevant problem setting
(+) Clear description of the algorithm which seems well thought-out 

Possible weakness:
(-) Experimental evaluation. 

I'm currently recommending a weak rejection. I believe the idea and execution of the paper (in particular the method part) is good. However, a stronger experimental section which also evaluates on more complex visual domains would greatly strengthen the paper. Doing so on tasks with an equal complexity on the required long-term memory might be computationally challenging, however, I personally would already be satisfied with showing that the proposed algorithm doesn't perform worse than the baselines, even on tasks which don't require (as much) memory. 

In other words: The current experiments clearly show why and when ALD can provide an advantage. What is missing for me is a visually more complex experiment that shows me that the actor/learner split and associated off-policy-ness doesn't create additional problems on (visually) more complex environments. If those environments don't require as much memory, there is no reason to expect that ALD can outperform the baseline, which would be ok for me, as long as it doesn't underperform it.

One additional question (unrelated to evaluation of the paper) I was wondering: The authors mention Teh et al. What I am wondering is why, when updating pi_A, you only use the distillation loss and not also the RL loss? Did you try?

Other minor point (no impact on evaluation): I would have found a brief description of HOGWILD, e.g. in the appendix, to be helpful. 
review_title: Interesting idea, more visually complex environments would strengthen the paper

Meta-review: I thank the authors for their submission and participation in the author response period. The reviewers unanimously agree that the papers proposes an interesting and original approach to using a costly model on a learner node, while distilling to a cheaper model run on actor nodes to gather experiences in a distributed RL framework. During discussion, R1 and myself emphasized the concern that the experiments in this paper leave open the question whether the approach will work beyond toy environments. However, I side with R2 and R3 in that the paper presents a valuable contribution to the community as it stands, and that the experiments proof the concept to the point that the paper should be accepted. I therefore recommend acceptance.
Decision: Accept (Poster)