Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The authors formulate the credit assignment method as minimizing the divergence between policy function and a learned prior distribution. Then they apply f-divergence optimization to avoid the model collapse in this framework. Empirical experiments are conducted on the program synthesis benchmark with sparse rewards. 

The main contribution of this paper is applying f-divergence optimization on the program synthesis task for credit assignment. 

+ One of my concerns is that the experiment section is in a limited domain to argue it is a broad algorithm for credit assignment. The paper will be stronger if the comparison is applied in a distant domain like goal-based robot learning etc. With some experiments on a different domain, the paper will be more convincing. 

+ The improvement/margin in program synthesis task needed to be explained well, is the margin significant enough?  

+ The paper could discuss more on related papers on program synthesis in the related work section as the main experiment is in this work.

+ The authors claim that the two-buffer estimation is better and lead to better gradient estimation, but it is not demonstrated empirically or theoretically. It could be better if the ablation study is conducted in the experiment. Or the author could provide a theoretical analysis of why equation (13) is better. Moreover, the investigation of different choices of $w_b$ and $w_c$ is necessary.  

+ Another study needed is the investigation of different divergences; the work will be stronger if a KL divergence version is compared. Otherwise, it is not clear how much the f-divergence will contribute to the performance. 

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper proposes guided adaptive credit assignment (GACA) for policy gradient methods with sparse reward.

GACA attacks the credit assignment problem by
1) using entropy regularized RL objective (KL divergence), iteratively update prior \bar{\pi} and \pi_\theta;
2) generalizing KL to f-divergence to avoid mode seeking behaviour of KL;
3) using 2 tricks to estimate the gradient of f-divergence to update \pi_\theta, a) modified MAPO (Liang et al., 2018) estimator (using two buffers), b) replacing rho_f by the inverse of tail probability (Wang et al., 2018).

Experiments of program synthesis and instruction following are conducted, to show the proposed GACA outperform competitive baselines.

Although the experimental results look promising, I have many concerns with respect to this paper as follows.

1. The organization is bad. The main algorithm has been put into the appendix. It should appear in the paper.

2. There are too many typos and errors in the paper and derivations, which quite affected reading and understanding.
For example:
in Eq. (6), what is z \sim Z? Should be z \in Z? It also appears in many other places.
in Eq. (7), there should not be \sum_{z \in Z} here.
Proof for Prop. 1, I cannot really understand the notations here. Please rewrite and explain this proof. (I can see it follows Grau-Moya et al., 2019, but the notations here are not clear.)
in Eq. (11), \bar{\pi} / \pi_\theta is used, but in Eq. (12), \pi_\theta / \bar{\pi} appeared, which one is correct? While in the proof for Lemma 2, it is \bar{\pi} / \pi_\theta. And in Alg. 1 it is \pi_\theta / \bar{\pi}. Please make this consistent.
Typos, like "Combining Theorem 1 and Theorem 2 together, we summarize the main algorithm in Algorithm 1." in the last paragraph of p6. However, they appeared as Prop. 1 and Lemma 2. Please improve the writing.

3. The mutual information argument Eq. (9) seems irrelevant here. (It follows Grau-Moya et al., 2019, but the notations in the proof are bad and I cannot understand it). Whether the solution is mutual information or not seems not helpful for getting better credit assignment. I suggest remove/reduce related arguments around Eq. (9) and (10), and make space for the main algorithm.

4. The entropy regularized objective and the KL is kind of well known. Maybe reduce the description here. And the key point is Eq. (8), which lays the foundation of iteratively update \bar{\pi} and \pi_\theta. However, Eq. (8) is the optimal solution of KL Eq. (7). Is it also the optimal solution of f-divergence used in the algorithm? If it is, clearly show that. If not, then update \bar{\pi} in Alg. 1 is problematic. Please clarify this point.

5. The 2 tricks used here for estimating the gradient of f-divergence with respect to \pi_\theta, i.e., modified MAPO estimator in Prop. 2, and inverse tail probability in Wang et al., 2018, seems quite important for the empirical performance.
However, motivation is not clear enough. First, why using two replay buffers "leads to a better approximation"? Any theory/intuition or experiment to support this claim? Second, why using inverse tail probability "achieve a trade-off between exploration and exploitation". It seems not obvious to see that. And also, explain why using this trick makes "\pi_\theta adaptively coverage and approximate prior distribution \bar{\pi}".

6. The claim that GACA recovers all the mentioned methods as special cases are questionable. For example, as in E.1, "by simply choosing \rho_f as constant 1", comparing Eq. (12) with the gradient of REINFORCE, there is a difference that REINFORCE has a reward term, but GACA does not have. Then why GACA reduces to REINFORCE? Also in E.5, the RAML objective seems wrong. There is no reward term here. Please check them.

Overall, the proposed GACA method achieves promising results in program synthesis tasks. However, there are many concerns with respect to motivation and techniques that should be resolved.

=====Update=====
Thanks for the rebuttal. I keep my rating since some of my concerns are still not resolved. In particular, "Eq. (8) is the optimal solution of KL Eq. (7). Is it also the optimal solution of f-divergence used in the algorithm?" Eq. (8) looks not the same as the paragraph above Lemma 2 "\bar{\pi} = \pi_\theta" to me. If Eq. (8) is not the optimal solution of Eq. (11), the update in Alg. 1 is somewhat problematic and other better choices exist. Since Algorithm 1 explicitly uses f-divergence, I think at least this point should be clarified by the authors rather than my guess.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: Summary:
This work proposed an off-policy framework for policy gradient approach called
guided adaptive credit assignment (GACA) in a simplified setting of goal-oriented
entropy regularized RL.
GACA optimizes the policy via fitting a learnable prior distribution that using
both high reward trajectory and zero reward trajectory to improve the sample efficiency.  
The experiments on sparse reward tasks such as WikiTableQuestions and WikiSQL
demonstrate the effectiveness of the GACA, comparing with a set of advanced baselines.


Detailed comments:

Off-policy learning:
The Environment dynamic is not considered. The trajectory
reward is determined by the initial state, goal, and the sequence of actions taken thereafter. The off-policy learning can be applied since the distribution of
initial state, and goal is not affected by the policy. This reduces to a
weighted maximum likelihood problem. 

Resolving the sparse reward issue:
In sparse reward tasks, many of trajectories have zero rewards, in order to utilize
the zero reward trajectory (since in the weighted problem those samples have no
contribution to the gradient). This work proposed to store the trajectories
into two replay buffers and samples from both of them separately. 
Intuitively, it is not clear to me why minimizing mutual information between z and
reward would help the learning. I am suspecting the reason is that mutual information brings non-zero gradient for zero reward trajectories (given zero-reward trajectories indeed helps the learning). 
The authors also claimed that KL divergence performs worse than f-divergence due to the mode seeking issue. Do the experiments in GACA w/o AG support this claim?

Ablation study:
The authors claimed that using zero reward trajectory can help with sample efficiency.
I wonder what the performance would be if we drop the zero reward trajectory buffer if we have a reasonable high frequency to reach the high trajectory reward sequence. 
Is it necessary to incorporate the zero reward trajectory? 

What is the exact formula of GACA w/o GP and GACA w/o AG? 

The proposed method consists of three parts (GP, AG, and separate buffer. ) 
Two variants (w/o GP, w/o AG) of GACA is conducted in the ablation study. 
How does the GACA perform if we drop the separate buffer? What if we incorporate separate buffer for baselines. Does GP/AG play an essential role in performance improvement,
comparing to a separate buffer? 


Other questions:
Since the sequence of actions is considered as a group, the performance
may highly depend on the size of action space and horizon. 
What is the size of the horizon of the tested problems? 
What is the value of WB and WC in each experiment?


Minor:
There are many typos or grammar issues in this version. e.g.,
L 3, Page 4, learn-able prior
Last paragraph, page 3, " as as a combination of expectations", 
Page, 15 "is actually equals mutual"
Eq 23 -> 24
review_title: Official Blind Review #3

Meta-review: The paper proposes a policy gradient algorithm related to entropy-regularized RL, that instead of the KL uses f-divergence to avoid mode collapse.

The reviewers found many technical issues with the presentation of the method, and the evaluation. In particular, the experiments are conducted on particular program synthesis tasks and show small margin improvements, while the algorithm is motivated by general sparse reward RL.

I recommend rejection at this time, but encourage the authors to take the feedback into account and resubmit an improved version elsewhere.
Decision: Reject