Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: SUMMARY
The major contribution of the paper is a generalization of lambda-returns called Confidence-based Autodidactic Returns (CAR), wherein the RL agent learns the weighting of the n-step returns in an end-to-end manner.   These CARs are used in the A3C algorithm.  The weights are based on the confidence of the value function of the n-step return.  Even though this idea in not new the authors propose a simple and robust approach for doing it by using the value function estimation network of A3C.

During experiments, the autodidactic returns perform better only half of the time as compared to lambda returns.


COMMENTS
The j-step returns TD error is not written correctly

In Figure 1 it is not obvious how the confidence of the values is estimated.
Figure 1 is unreadable.


A lighter version of Algorithm 1 in Appendix F should be moved in the text, since this is the novelty of the paper.

review_title: The paper extends the A3C algorithm with lambda returns.  In addition it proposes  an approach for learning the weights of the returns.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper revisits the idea of exponentially weighted lambda-returns at the heart of TD algorithms. The basic idea is that instead of geometrically weighting the n-step returns we should instead weight them according to the agent's own estimate of its confidence in it's learned value function. The paper empirically evaluates this idea on Atari games with deep non-linear state representations, compared to state-of-the-art baselines.

This paper is below the threshold because there are issues with the : 1) motivation, 2) the technical details, and (3) the empirical results.

The paper begins by stating that the exponential weighting of lambda returns is ad-hoc and unjustified. I would say the idea is well justified in several ways. First the lambda return definition lends itself to online approximations that achieve a fully incremental online form with linear computation and nearly as good performance of the off-line version. Second, decades of empirical results illustrating good performance of TD compared with MC methods. And an extensive literature of theoretical results. The paper claims that the exponential has been noted to be ad-hoc, please provide a reference for this.

There have been several works that have noted that lambda can and perhaps should be changed as a function of state (Sutton and Barto, White and White [1], TD-Gammon). In fact, such works even not that lambda should be related to confidence. The paper should work harder to motivate why adapting lambda as a function of state---which has been studied---is not sufficient.

I don't completely understand the objective. Returns with higher confidence should be weighted higher, according to the confidence estimate around the value function estimate as a function of state? With longer returns, n>>1, the role of the value function in the target is down-weighted by gamma^n---meaning its accuracy is of little relevance to the target. How does your formalism take this into account? The basic idea of the lambda return assumes TD targets are better than MC targets due to variance, which place more weight on shorter returns.

I addition I don't understand how learning confidence of the value function has a realizable target. We do not get supervised targets of the confidence of our value estimates. What is your network updating toward?

The work of Konidaris et al [1] is a more appropriate reference for this work (rather than the Thomas reference provided). Your paper does not very clearly different itself from Konidaris's work here. Please expand on this.

The experiments have some issues. One issue is that basic baselines could more clearly illustrate what is going on. There are two such baselines: random fixed weightings of the n-step returns, and persisting with the usual weighting but changing lambda on each time step (either randomly or according to some decay schedule). The first baseline is a sanity check to ensure that you are not observing some random effect. The second checks to see if your alternative weighting is simply approximating the benefits of changing lambda with time or state.

I would say the current results indicate the conventional approach to TD is working well if not better than the new one. Looking at fig 3, its clear the kangaroo is skewing the results, and that overall the new method is performing worse. This is further conflated by fig7 which attempts to illustrate the quality of the learned value functions. In Kangaroo, the domain where your method does best, the l2 error is worse. On the other hand in sea quest and space invaders, where your method does worse, the l2 error is better. These results seem conflicting, or at least raise more questions than they answer.

[1] A Greedy Approach to Adapting the Trace Parameter for Temporal Difference Learning . Adam White and Martha White. Autonomous Agents and Multi-agent Systems (AAMAS), 2016
[2] G. D. Konidaris, S. Niekum, and P. S. Thomas. TDγ: Re-evaluating complex backups in temporal difference learning. In Advances in Neural Information Processing Systems 24, pages 2402–2410. 2011. 
review_title: Alternative definitions for lambda returns

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors present confidence-based autodidactic returns, a Deep learning RL method to adjust the weights of an eligibility vector in TD(lambda)-like value estimation to favour more stable estimates of the state. The key to being able to learn these confidence values is to not allow the error of the confidence estimates propagate back though the deep learning architecture.

However, the method by which these confidence estimates are refined could be better described. The authors describe these confidences variously as: "some notion of confidence that the agent has in the value function estimate" and "weighing the returns based on a notion of confidence has been explored earlier (White & White, 2016; Thomas et al., 2015)". But the exact method is difficult to piece together from what is written. I believe that the confidence estimates are considered to be part of the critic and the w vector to be part of the theta_c parameters. This would then be captured by the critic gradient for the CAR method that appears towards the end of page 5. If so, this should be stated explicitly.

There is another theoretical point that could be clearer. The variation in an autodidactic update of a value function (Equation (4)) depends on a few things, the in variation future value function estimates themselves being just one factor. Another two sources of variation are: the uncertainty over how likely each path is to be taken, and the uncertainty in immediate rewards accumulated as part of some n-step return. In my opinion, the quality of the paper would be much improved by a brief discussion of this, and some reflection on what aspects of these variation contribute to the confidence vectors and what isn't captured.

Nonetheless, I believe that the paper represents an interesting and worthy submission to the conference. I would strongly urge the authors to improve the method description in the camera read version though. A few additional comments are as follows:

  • The plot in Figure 3 is the leading collection of results to demonstrate the dominance of the authors' adaptive weight approach (CAR) over the A3C (TD(0) estimates) and LRA3C (truncated TD(lambda) estimates) approaches. However, the way the results are presented/plotted, namely the linear plot of the (shifted) relative performance of CAR (and LRA3C) versus A3C, visually inflates the importance of tasks on which CAR (and LRA3C) perform better than A3C, and diminishes the importance of those tasks on which A3C performs better. It would be better kept as a relative value and plotted on a log-scale so that positive and negative improvements can be viewed on an equal setting.
  • On page 3, when Gt is first mentioned, Gt should really be described first, before the reader is told what it is often replaced with.
  • On page 3, where delta_t is defined (the j step return TD error, I think the middle term should be $gamma^j V(S_{t+j})$
  • On page 4 and 5, when describing the gradient for the actor and critic, it would be better if these were given their own terminology, but if not, then use of the word respectively in each case would help.
review_title: An interesting adaptive weight method for using eligibility vectors with deep RL. I am a little concerned that a key part of the method is imperfectly described.

Meta-review: This is an interesting paper, but was quite difficult to follow. As they stand, the empirical results are not altogether convincing nor warrant acceptance.
Decision: Reject