Guided Adaptive Credit Assignment for Sample Efficient Policy Optimization

Abstract: Policy gradient methods have achieved remarkable successes in solving challenging reinforcement learning problems. However, it still often suffers from sparse reward tasks, which leads to poor sample efficiency during training. In this work, we propose a guided adaptive credit assignment method to do effectively credit assignment for policy gradient methods. Motivated by entropy regularized policy optimization, our method extends the previous credit assignment methods by introducing more general guided adaptive credit assignment(GACA). The benefit of GACA is a principled way of utilizing off-policy samples. The effectiveness of proposed algorithm is demonstrated on the challenging \textsc{WikiTableQuestions} and \textsc{WikiSQL} benchmarks and an instruction following environment. The task is generating action sequences or program sequences from natural language questions or instructions, where only final binary success-failure execution feedback is available. Empirical studies show that our method significantly improves the sample efficiency of the state-of-the-art policy optimization approaches.

Reviews and author responses:

This paper proposes guided adaptive credit assignment (GACA) for policy gradient methods with sparse reward.

GACA attacks the credit assignment problem by
1) using entropy regularized RL objective (KL divergence), iteratively update prior \bar{\pi} and \pi_\theta;
2) generalizing KL to f-divergence to avoid mode seeking behaviour of KL;
3) using 2 tricks to estimate the gradient of f-divergence to update \pi_\theta, a) modified MAPO (Liang et al., 2018) estimator (using two buffers), b) replacing rho_f by the inverse of tail probability (Wang et al., 2018).

Experiments of program synthesis and instruction following are conducted, to show the proposed GACA outperform competitive baselines.

Although the experimental results look promising, I have many concerns with respect to this paper as follows.

1. The organization is bad. The main algorithm has been put into the appendix. It should appear in the paper.

2. There are too many typos and errors in the paper and derivations, which quite affected reading and understanding.
For example:
in Eq. (6), what is z \sim Z? Should be z \in Z? It also appears in many other places.
in Eq. (7), there should not be \sum_{z \in Z} here.
Proof for Prop. 1, I cannot really understand the notations here. Please rewrite and explain this proof. (I can see it follows Grau-Moya et al., 2019, but the notations here are not clear.)
in Eq. (11), \bar{\pi} / \pi_\theta is used, but in Eq. (12), \pi_\theta / \bar{\pi} appeared, which one is correct? While in the proof for Lemma 2, it is \bar{\pi} / \pi_\theta. And in Alg. 1 it is \pi_\theta / \bar{\pi}. Please make this consistent.
Typos, like "Combining Theorem 1 and Theorem 2 together, we summarize the main algorithm in Algorithm 1." in the last paragraph of p6. However, they appeared as Prop. 1 and Lemma 2. Please improve the writing.

3. The mutual information argument Eq. (9) seems irrelevant here. (It follows Grau-Moya et al., 2019, but the notations in the proof are bad and I cannot understand it). Whether the solution is mutual information or not seems not helpful for getting better credit assignment. I suggest remove/reduce related arguments around Eq. (9) and (10), and make space for the main algorithm.

4. The entropy regularized objective and the KL is kind of well known. Maybe reduce the description here. And the key point is Eq. (8), which lays the foundation of iteratively update \bar{\pi} and \pi_\theta. However, Eq. (8) is the optimal solution of KL Eq. (7). Is it also the optimal solution of f-divergence used in the algorithm? If it is, clearly show that. If not, then update \bar{\pi} in Alg. 1 is problematic. Please clarify this point.

5. The 2 tricks used here for estimating the gradient of f-divergence with respect to \pi_\theta, i.e., modified MAPO estimator in Prop. 2, and inverse tail probability in Wang et al., 2018, seems quite important for the empirical performance.
However, motivation is not clear enough. First, why using two replay buffers "leads to a better approximation"? Any theory/intuition or experiment to support this claim? Second, why using inverse tail probability "achieve a trade-off between exploration and exploitation". It seems not obvious to see that. And also, explain why using this trick makes "\pi_\theta adaptively coverage and approximate prior distribution \bar{\pi}".

6. The claim that GACA recovers all the mentioned methods as special cases are questionable. For example, as in E.1, "by simply choosing \rho_f as constant 1", comparing Eq. (12) with the gradient of REINFORCE, there is a difference that REINFORCE has a reward term, but GACA does not have. Then why GACA reduces to REINFORCE? Also in E.5, the RAML objective seems wrong. There is no reward term here. Please check them.

Overall, the proposed GACA method achieves promising results in program synthesis tasks. However, there are many concerns with respect to motivation and techniques that should be resolved.

=====Update=====
Thanks for the rebuttal. I keep my rating since some of my concerns are still not resolved. In particular, "Eq. (8) is the optimal solution of KL Eq. (7). Is it also the optimal solution of f-divergence used in the algorithm?" Eq. (8) looks not the same as the paragraph above Lemma 2 "\bar{\pi} = \pi_\theta" to me. If Eq. (8) is not the optimal solution of Eq. (11), the update in Alg. 1 is somewhat problematic and other better choices exist. Since Algorithm 1 explicitly uses f-divergence, I think at least this point should be clarified by the authors rather than my guess.

Thank you for your time, detailed feedback, and a clear summary of our contribution.
We are happy to hear you found this work valuable. 
We answer most of the said points below. It would be very helpful to hear your thoughts on our responses so that we can make the appropriate modifications to the paper. 


1. Is there an experiment demonstrate f-divergence performs better than KL-divergence?
    
    GACA w/o AG is equivalent to replacing f-divergence with KL-divergence in GACA. Table 1. shows that GACA > GACA w/o AG > MAPO on both program synthesis benchmarks, this result demonstrates that f-divergence(adaptive gradient estimation) is better KL-divergence. 



2. Does dropping out zero-reward trajectory buffer have an impact on the performance?
    
    Intuitively, dropping out zero-reward trajectory buffer reduces the number of samples used to estimate gradient, thus it hurts the minimization of f-divergence and downgrades the performance. Empirically, we found that dropping out zero-reward buffer downgrades the performance of GACA but still better than MAPO. This result further validates the value of this work — enable reusing off-policy samples by efficient credit assignment is important. 


3. Does separate buffer help baselines?
    
    Separate buffer doesn’t help baselines and is not necessary because mathematically baselines method can only utilize high-reward trajectories to compute gradient, as shown at the beginning of Section 4 and Appendix E. 
    We use separate buffer because GACA can use both high-reward and zero-reward trajectories, naturally, we can use stratified sampling to estimate unbiased and low variance gradient. 


4. Typos


    Thank you for pointing out typos, we have improved the presentation of the paper. The updated version of the paper is available now. Again, thank you for reading the paper. 

Thank you for your time and detailed feedback. 
We are happy to hear you found this work valuable, and understand your concerns. We are confident that, following the points of clarification highlighted by you, we can resolve any ambiguities in the paper, and thank you for helping make the paper stronger as a result. We answer most of the said points below. It would be very helpful to hear your thoughts on our responses so that we can make the appropriate modifications to the paper. 
We hope that you will be willing to consider revising your assessment in light of the clarifications. 


1. Typos in the paper, move algorithm to main paper, and other presentation suggestions


    Thanks for pointing out typos in the submission, we have moved the main algorithm from Appendix to main paper, adjust the organization, and improved the presentation of the paper a lot. The updated version of the paper is available now. Again, thank you for reading the paper. 


2. Why using two replay buffers "leads to a better approximation"


    We acknowledge that this sentence is a little bit confusing. What we mean is GACA enables using both high-reward and zero-reward trajectories, which leads to a higher sample efficiency. While previous methods either only reply on on-policy trajectories(e.g. REINFORCE, MML, etc), or use a buffer to save high-reward trajectories for replaying(e.g. MAPO). 


4. Motivations behind using inverse tail probability to approximate f-divergence 


    By using f-divergence, we reveal the connections between various previous credit assignment methods, for example, IML and RAML both minimize a reverse KL-divergence between policy and a target distribution, MAPO minimizes KL-divergence, etc. Different divergence measures lead to a different approximation of policy to target distribution[1], for example, KL-divergence often leads to mode-collapse. We want the policy distribution to approximate/cover the target distribution as good as possible, thus we utilize inverse tail probability technique which we found performs the best. 


    [1] Yingzhen Li, Richard E. Turner. Rényi Divergence Variational Inference.  Advances in Neural Information Processing Systems(NeurIPS), 2016.



5. The claim that GACA recovers all the mentioned methods as special cases are questionable


    Your concern about REINFORCE and RAML as special cases of GACA can be resolved if you recall that for simplicity and without loss of generality, we assumed that reward is 1 for successful task completion trajectories and 0 otherwise, this follows the notations convenience in [1, 2]. In the meanwhile, we do take your suggestions on improving the presentation of the paper that we have updated Appendix E with improved presentation of how does GACA recover existing credit assignment methods. Thank you for reading our Appendix.


    [1] Chen Liang, Mohammad Norouzi, Jonathan Berant, Quoc Le, and Ni Lao. Memory augmented policy optimization for program synthesis with generalization. Advances in Neural Information Processing Systems(NeurIPS), 2018.
    
    [2] Kelvin Guu, Panupong Pasupat, Evan Liu, and Percy Liang. From language to programs: Bridging reinforcement learning and maximum marginal likelihood. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1051–1062, Vancouver, Canada, July 2017.

Thank you for your time and detailed feedback. 
We are glad to hear you found this work interesting and valuable, and understand your concerns. We are confident that, following the points of clarification highlighted by you, we can resolve any ambiguities in the paper, and thank you for helping make the paper stronger as a result. 
We answer most of the said pints below. It would be very helpful to hear your thoughts on our responses so that we can make the appropriate modifications to the paper. 
We hope that you will be willing to consider revising your assessment in light of the clarifications.


1. Evaluation of the proposed method on other domains

      We would like to point out GACA can be useful in other challenging tasks such as combinational optimization and structured prediction where credit assignment from binary feedback remains a major challenge. We want emphasize that one of the main purposes of this paper is to introduce the method of guided adaptive credit assignment. As such, the purpose of this paper is not to beat every single benchmark but to show the benefit of this framework. In particular, we demonstrate the effectiveness of GACA on two challenging program synthesis benchmarks, to our best knowledge, this work is the current state-of-the-art method that uses only binary supervision and outperforms previous methods by a large margin. We leave the investigation of our method on other domains for future work. 


2. Results explanation/significance


    GACA outperforms recent state-of-the-art methods MeRL, BoRL, and MAPO by a large margin, on a variety of tasks—including the challenging WikiSQL and WikiTable,  as shown in Table 1, 2, 3. To our best knowledge, GACA is by far the state-of-the-art method on these benchmarks using only binary feedback. GACA is easy to implement and is a generalization of various credit assignment methods(MAPO MML, EML, RAML, and REINFORCE), we want to emphasize that GACA is general and can be further improved by combining it with techniques in other methods to further boost performance, such as meta-learning proposed in MeRL.



3. More related work on program synthesis


    We have included several program synthesis papers in related work.



4. Why use two-buffer estimation in Eq. 13


    Because GACA enables reusing all the past trajectories while previous methods only use high-reward trajectories, two-buffer estimation naturally arises here as a result of using stratified sampling to obtain unbiased and low variance gradient. w_b and w_c are calculated via stratified sampling, refer to Eq(13) for details. 


5. What’s the performance of GACA if f-divergence is replaced with KL-divergence?


    If KL-divergence is used, then GACA reduces to GACA w/o AG which means GACA without adaptive gradient estimation. From experimental results(e.g. Table 1), we can see that GACA w/o AG greatly outperforms baselines on both benchmarks.

Summary:
This work proposed an off-policy framework for policy gradient approach called
guided adaptive credit assignment (GACA) in a simplified setting of goal-oriented
entropy regularized RL.
GACA optimizes the policy via fitting a learnable prior distribution that using
both high reward trajectory and zero reward trajectory to improve the sample efficiency.  
The experiments on sparse reward tasks such as WikiTableQuestions and WikiSQL
demonstrate the effectiveness of the GACA, comparing with a set of advanced baselines.


Detailed comments:

Off-policy learning:
The Environment dynamic is not considered. The trajectory
reward is determined by the initial state, goal, and the sequence of actions taken thereafter. The off-policy learning can be applied since the distribution of
initial state, and goal is not affected by the policy. This reduces to a
weighted maximum likelihood problem. 

Resolving the sparse reward issue:
In sparse reward tasks, many of trajectories have zero rewards, in order to utilize
the zero reward trajectory (since in the weighted problem those samples have no
contribution to the gradient). This work proposed to store the trajectories
into two replay buffers and samples from both of them separately. 
Intuitively, it is not clear to me why minimizing mutual information between z and
reward would help the learning. I am suspecting the reason is that mutual information brings non-zero gradient for zero reward trajectories (given zero-reward trajectories indeed helps the learning). 
The authors also claimed that KL divergence performs worse than f-divergence due to the mode seeking issue. Do the experiments in GACA w/o AG support this claim?

Ablation study:
The authors claimed that using zero reward trajectory can help with sample efficiency.
I wonder what the performance would be if we drop the zero reward trajectory buffer if we have a reasonable high frequency to reach the high trajectory reward sequence. 
Is it necessary to incorporate the zero reward trajectory? 

What is the exact formula of GACA w/o GP and GACA w/o AG? 

The proposed method consists of three parts (GP, AG, and separate buffer. ) 
Two variants (w/o GP, w/o AG) of GACA is conducted in the ablation study. 
How does the GACA perform if we drop the separate buffer? What if we incorporate separate buffer for baselines. Does GP/AG play an essential role in performance improvement,
comparing to a separate buffer? 


Other questions:
Since the sequence of actions is considered as a group, the performance
may highly depend on the size of action space and horizon. 
What is the size of the horizon of the tested problems? 
What is the value of WB and WC in each experiment?


Minor:
There are many typos or grammar issues in this version. e.g.,
L 3, Page 4, learn-able prior
Last paragraph, page 3, " as as a combination of expectations", 
Page, 15 "is actually equals mutual"
Eq 23 -> 24

The authors formulate the credit assignment method as minimizing the divergence between policy function and a learned prior distribution. Then they apply f-divergence optimization to avoid the model collapse in this framework. Empirical experiments are conducted on the program synthesis benchmark with sparse rewards. 

The main contribution of this paper is applying f-divergence optimization on the program synthesis task for credit assignment. 

+ One of my concerns is that the experiment section is in a limited domain to argue it is a broad algorithm for credit assignment. The paper will be stronger if the comparison is applied in a distant domain like goal-based robot learning etc. With some experiments on a different domain, the paper will be more convincing. 

+ The improvement/margin in program synthesis task needed to be explained well, is the margin significant enough?  

+ The paper could discuss more on related papers on program synthesis in the related work section as the main experiment is in this work.

+ The authors claim that the two-buffer estimation is better and lead to better gradient estimation, but it is not demonstrated empirically or theoretically. It could be better if the ablation study is conducted in the experiment. Or the author could provide a theoretical analysis of why equation (13) is better. Moreover, the investigation of different choices of $w_b$ and $w_c$ is necessary.  

+ Another study needed is the investigation of different divergences; the work will be stronger if a KL divergence version is compared. Otherwise, it is not clear how much the f-divergence will contribute to the performance. 


Meta-review: The paper proposes a policy gradient algorithm related to entropy-regularized RL, that instead of the KL uses f-divergence to avoid mode collapse.

The reviewers found many technical issues with the presentation of the method, and the evaluation. In particular, the experiments are conducted on particular program synthesis tasks and show small margin improvements, while the algorithm is motivated by general sparse reward RL.

I recommend rejection at this time, but encourage the authors to take the feedback into account and resubmit an improved version elsewhere.

Decision: reject