Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper investigates off-policy actor critic (AC) learning with experience replay using V-trace. It shows that V-trace policy gradient is not guaranteed to converge to a local optimal solution. To mitigate the bias and variance problem of V-trace and importance sampling, a trust region approach is proposed to adaptively selects only suitable behavior distributions when estimating the state-value of a policy. To this end, a behavior relevance function (KL divergence) is introduced to classify behavior as relevant. The proposed learning method LASER demonstrates the state-of-the-art data efficiency in Atari among agents trained up until 200M frames. In all, this paper is well motivated and technically sound. The draft can be improved by making it more self-contained by providing a sketch of the proof rather than refer everything to the appendix. Also it might be helpful to provide a pseudocode of LASER to help readers better understand the technical details. 

Other comments and questions:

1) When talking about the selection process, z is treated as a random variable. What is its distribution?
2) what does “very off-policy learning” mean?
3) In figure 3(left), why “LASER: shared + trust region” performs worse than “LASER: not shared”? 
4) In proposition 3. Q^w should be explained in the main text.

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper aims to improve the efficiency of the actor-critic method. The authors first analyzed the cause of instability in the prior work, from the perspective of bias and variance. Two remedies were then presented: (i) mixing the experience replay with online learning; (ii) proposing a trust region scheme to select the behavior policies. The authors finally tested the proposed method on Atari games, and showed the better results, compared with the state-of-the-art methods.

In my opinion, the empirical results are impressive, and the authors also provided some insights for the motivation. Given the results on Atari games, this paper could be a great contribution on the actor critic methods. The propositions are presented to support relevant claims, while their significance seems a bit limited, and some further clarification is necessary. The authors also need to address a few confusing statements and missing details.

1. In Proposition 3, the authors claimed that mixing with on-policy data can reduce the bias. I checked the proof but did not find anything relevant. Also, what is the amount of bias reduced?
2. In Equation (1), could you provide a formal definition for "V"? 
3. The authors claimed at the beginning of Section 4 that the trust region method was proposed to mitigate the bias and variance problem of V-trace. However, I did not see how this is reflected in Propositions 4 and 5. Is this statement only based on empirical results?
4. It was mentioned right below Equation (4) that "Observe how this inner expectation ... matches the on-policy return...". Could you provide a formal proof?
5. What are the hyperparameters for the 9 agents used in Figure 1? Also, how did you choose "b" in trust region?
6. A few notation issues / typo:
(1) it's -> its
(2) In Equation (5), should "z \in M_{\beta, \pi} (s_t)" be "\mu_z \in M_{\beta, \pi} (s_t)"?
(3) At the 2nd line of Page 7, should the content for the indicator function be "\beta (\pi, \mu, s_t) < b"?

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The authors investigate off-policy actor-critic reinforcement learning where they want to make use of shared experience replay. Two approaches were suggested and compared. One was to mix replayed experience with on-policy data and the other was to create trust regions that only selects well-behaved behavioral distributions for state value estimation.
According to the authors the several experiments provide evidence that their algorithm achieves competitive or even state-of-the-art results in data efficiency. They underpin this with some theoretical analysis.

review_title: Official Blind Review #2

Meta-review: The paper presents an off-policy actor-critic scheme where i) a buffer storing the trajectories from several agents is used (off-policy replay) and mixed with the on-line data from the current agent; ii) a trust-region estimator is used to select trajectories that are sufficiently close to the current policy (e.g. in the sense of a KL divergence).

As noted by the reviews, the results are impressive. 

Quite a few concerns still remain:
* After Fig. 1 (revised version), what matters is the shared replay, where the agent actually benefits from the experience of 9 other different agents; this implies that the population based training observes 9x more frames than the no-shared version, and the question whether the comparison is fair is raised;
* the trust-region estimator might reduce the data seen by the agent, leading it to overfit the past (Fig. 3, left);
* the influence of the $b$ hyper-parameter (the trust threshold) is not discussed. In standard trust region-based optimization methods, the trust region is gradually narrowed, suggesting that parameter $b$ here should evolve along time. 


Decision: Reject