Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: In this paper, the authors develop a new policy gradient method to reduce the variance in the gradient estimations.
In the commonly used policy method, the bias is a function of the state. e.g., V(x_t). In this paper, the authors propose to use bias V(x_t,\phi_t) where \phi_t is a statistics of future events such that \phi_t is conditionally independent of the action at time t.

The authors show that using such statistics in V(x_t,\phi_t) results in a reduction in the gradient estimate used in policy gradient methods. 


Later, the authors also show that their method performs well in practice.


There is a set of problems with the paper's presentation, which resulted in the negative evaluation.
The analysis in the paper is straightforward and also easy to follow. However, I could not find how the proposed algorithm learns the \phi.

I encourage the authors to improve the clarity, presentation, and language in this paper. 

1) I did not get what the authors mean by luck or skill. These terms do not seem to be coherent terms in this paper. I highly encourage the authors to rethink such usage. Unless the authors mathematically define it in the paper. 

2) "Another issue of model-free methods is that counterfactual reasoning, i.e. reasoning about what would have happened had different actions been taken with everything else remaining the same, is not possible." 

Can the authors clarify it? Why is it not? When I learn a Q function, that tells me what would be the expected return if I choose other actions following the same policy, right? 
If you mean evaluating other policies is not possible, I still doubt the statement is true. 

3) "Given a trajectory, model-free methods can in fact only learn about the actions that were actually taken to produce the data, and this limits the ability of the agent to learn quickly."

Can you clarify this? I can use function approximation based methods, and then, the first part of the authors' statement is no longer true. The second statement is inaccurate since the author did not quantify with respect to what method the quickness in learning is compared to.

4) "actions taken by the agent will only affect a vanishing part of the outcome". What do the authors mean here? What the vanishing part of the outcome refers to?

5) "mak- ing it increasingly difficult to learn from classical reinforcement learning algorithms", what the authors mean by learning from classical RL algorithm? and why the authors think a better credit assessment is needed and is the way to go. What motivates the authors to state the issue is the credit assignment?

6) "Second, removing the value function V (Xt) from the return Gt does not bias the estimator and typically reduces variance". Would the author refer to a paper stating that removing the value function V (Xt) from the return Gt typically reduces variance?

7)"This estimator updates the policy through the score term; note however the learning signal only updates the policy πθ(a|Xt) at the value taken by action At = a "
I am not sure I understand this sentence. Is πθ(a|Xt) the policy, or it is πθ. Do authors have a different model for each state and action pair? Even in that case, since the need to normalize action probability, changing πθ(a|Xt) will affect other πθ(a|X) as well. Therefore, I am not sure what the authors mean here.

8) Distinction between single action and all actions.
In both propositions 1 and 2, it seems that the learning signal is provided for both actions. It is not clear to me how the authors make the distinction. Especially here 
"The policy gradient theorem from (Sutton et al., 2000), which we will also call all-action policy gradient, shows it is possible to provide learning signal to all actions,".
I am not sure what the authors mean. 

The authors state that"A particularity of the all-actions policy gradient estimator is that the term at time t for updating the policy ∇π(a|Xt)(Q(Xt, a) depends only on past information;" but it seems to me that Q is a function of the measure on the future. Isnt it the case?

9) To motivate the usage of phi, the authors talk about a scenario in a soccer game, which again I could not find useful, especially when they bring luck and skill. 
The authors state that "When using the single-action policy gradient estimate, the outcome of the game being a victory, and ,assuming a ±1 reward scheme, all her actions are made more likely". 
How is it possible that all actions become more likely? when their probabilities should be sum to one?
I am not sure again. Are the authors talking about using one trajectory for all the estimates? The update in proposition 1 shows that in the case the agent action does not change the outcome, then the gradient is zero.


10) The authors state that
"In contrast, if the agent could measure a quantity Φt which has a high impact on the return but is not correlated to the agent action At , it could be far easier to learn Q(Xt , Φt , a)."
It is not clear why learning Q(Xt, a) is harder than Q(Xt , Φt , a). So far, Q(Xt, a) seems an easier function to approximate and most likely needs a fewer sample to learn Q(x, a) than something presumably complicated like Q(x, \phi , a).

11) In section 3.1, I strongly encourage the authors to elaborate more clearly on what they do. Is W a scaler? if yes, then how F can be constructed?

Do you draw U,V,W each time step??



12) Aside from many unclear statements in this paper that the authors can easily address, I could not find how the authors find \phi. Since this is the main key component of the paper, it would be great if the authors could explain it in depth. I also could not find it clear in the appendix. 

13) I strongly encourage the authors to expand their study on plain MDP before getting to the POMDP complication. It is not clear where the performance gain comes from.


................................................................

Post rebuttal. The confidence rating is reduced.
I might have been mistaken, but the authors might find this paper useful. "Troubling Trends in Machine Learning Scholarship"
Again, I might be wrong, and the mentioned paper might be of no use here.



review_title: Unclear and vague, but can be improved. I could not find how the authors find the key component of the method, i.e., phi.

final_score: rating: 6: Marginally above acceptance threshold
review_content: #### Summary:
The paper explores a new approach to credit assignment that complements existing work. It focuses on model-free approaches to credit assignment using hindsight information. In contrast to some prior work on this topic, e.g., (Harutyunyan et al. 2019), the paper does not rely explicitly on hand-crafted information, but instead learns to extract useful hindsight information. The contributions of the paper are two-fold. First, the paper introduces two new policy gradient estimators, FC-PG and CCA-PG, and it proves that the novel gradient estimators are unbiased. Second, it provides experimental evidence that the novel estimators are beneficial compared to some prior work (in particular (Harutyunyan et al. 2019)).  


#### Comments:

Overall, I found the contributions of the paper interesting, but I'm somewhat on the fence about this paper due to the following pros and cons. 

Pros: The paper is clearly written, easy to follow, and interesting to read. It provides a good overview of the related work, and motivates well the problem at hand. Furthermore, the paper showcases that its algorithmic approach has theoretical grounding, and it experimentally verifies that it's beneficial compared to concurrent approach from (Harutyunyan et al. 2019). 

Cons: Given that a very similar type of counterfactual credit assignment approach has already been proposed in prior work, the technical contributions (theorems) of the paper seem somewhat incremental. The experiments, while indicating potential benefits of the proposed approach, utilize relatively simple environments compared to some of the recent papers on credit assignment (e.g. (Arjona-Medina et al. 2019), (Guez et al 2019)). Moreover, the experiments could include more state of the art baselines. 

Apart from these high level comments, the following comments include suggestions for improvements and questions.  

Related work: Since the hindsight credit assignment of (Harutyunyan et al. 2019) is a special case of FC-PG, this connection should be mentioned earlier in the paper, not just in the related work section. The flow of the paper is currently misleading, given that there is prior work that does propose quite similar ideas, e.g., the content between the title to section 2.4 does not seem to be reflect relevant prior work. Perhaps referencing relevant papers in earlier sections, or moving the related work section, would resolve this issue. 

Notation: Notation in the paper often omits important dependences, making some of the calculations confusing or not immediately clear. In the interest of making the claims more precise, it would be very useful to add important dependencies where needed. For example, in equation (1), does $P(a|X_t, \Phi_t)$ depend on policy $\pi$? Moreover, the notation does not seem to be consistent, e.g., policy $\pi$ sometimes has dependency on $\theta$ sometime not (in gradient calculations).  

Appendix: I think adding some parts from the appendix could improve the clarity of the content. In particular, the last paragraph on Page 3 that starts with 'We ensure that these statistics...' is not providing sufficient explanations regarding the technical content important for understanding the results. It is also not clear if all the content in the appendix is relevant for the results described in the main text. 

Minor typos: 
-removed from the advantage, resulting a significantly lower variance estimator. --- resulting in?
-$\lambda_{IM}$ does not seem to be defined before being used (in the paragraph before section 3.2)
-and the the benefits of the more general FC-PG and all-actions estimators. --- remove one 'the'?

#### Questions:

A) I'm a bit puzzled by the discussion regarding the conditional independence requirement in Section 2.5. Why is this an 'intuitive' requirement? How does it influence the interpretation in the  paragraph before Theorem 3? How does this compare to  (Harutyunyan et al. 2019) argument that '$h(.)$ quantifies the relevance of action a to the future state $X_k$'? 

B) The proof of Theorem 3 and Theorem 4 in Section D3 says that the theorems follows from Theorem 1 and Theorem 2 given the conditional independence assumption. Could you explain in more detail why the second statement (about variance) in Theorem 3 follows from Theorem 1 and 2? 

C) How does this approach compare to Ferret et al.: Self-Attentional Credit Assignment for Transfer in Reinforcement Learning?
review_title: Official Blind Review #3

final_score: rating: 5: Marginally below acceptance threshold
review_content: This work attempts to address the problem posed by high reward variance and low sample efficiency in model-free RL algorithms. The proposal is to use counterfactuals to do finer-grained credit-assignment and reasoning about alternative actions without having to learn a potentially difficult environment model. 

This is done by conditioning the value function on a random variable $\phi$ that attempts to capture everything else about the future trajectory not resulting from the current action. This is done by maximizing the independence between $\phi$ and $A$ given the current state. A classifier that predicts action based on $\phi$ is required to do the above. This is also learned from data.

Claimed contributions:
Proposing a set of environments with difficult credit assignment.
Novel algorithms that use counterfactuals that are unbiased and guarantee lower variance.

+ The approach seems novel and interesting. 
+ The claimed contributions are supported to a large extent by theory and experimentation.
+ The idea of constructing value functions conditioned on future trajectory information is not novel (Hindsight Credit Assignment does this), but the idea of learning the conditioning variable is (HCA uses states or returns).
+ The paper is clearly written. The illustrative example of counterfactuals in hindsight with Alice and Megan is helpful.
+ The approach is evaluated first on a bandit task and then on different versions of a partially observable gridworld environment and finally on a multi-task setting.
+ Comparison to vanilla policy gradient and a couple of versions of prior work (HCA) over a substantial number of random seeds. 
+ The task interleaving setting is an interesting benchmark for multi-task settings. 

This work builds off of HCA and mainly addresses the case of high variance in rewards where the prior work seems to fail. It performs similar to vanilla PG on environments with little randomness in reward for similar actions, but better than HCA. 

The authors claim that they do not require a model of the environment but a classifier $h(A_T|X_T, \phi)$ is learned which resembles an inverse model. Even though the approach does not require building a forward model, I am curious to know the performance of a model-based approach such as by Buesing et al. trained on the same data available for $h(A_T|X_T, \phi)$ in these environments. Is it difficult to learn a model for the proposed tasks?

I think the work contains enough novelty, the writing is clear and the experimentation is extensive. But, I am unsure whether to recommend acceptance without a model-based baseline trained on data available to the classifier used in this approach.


review_title: Novel and interesting work on hindsight counterfactuals with perhaps some missing evaluations

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
The message and paper propose a couple of environments where there is exogenous noise added to the reward function and the particular method in the paper specifically looks at this type of noise. While the method proposed may work in these types of environments it's not clear if more interesting environments do have these properties and we should be more concerned with this problem or that the environments used in the paper were specifically constructed to fit the use case of the algorithm.

The proposed method in the paper does offer interesting insight into how certain temporary consistent variables and the identification of such variables can help decrease the variance over policy estimates. However, the results in the paper are not overly convincing with respect to understanding the importance of this method on more realistic tasks that the community is generally interested in.

Some more detailed notes:
- The introduction does not state that the particular credit assignment problems being looked into is that of partially observed environments. Overall, I find the writing in the introduction to not motivate the problem well our lead the reader towards what to expect in the rest of the paper. This makes it very difficult to understand and appreciate the paper.
- If it's still not clear from the middle section to let the detail of the contribution is going to be period by this point it sounds like the method is just going to be a modification to a q function.
- There does not appear to be my significant information on how the mutual information metric is computed between the action space and latent variable space.
review_title: Interesting but confusing

Meta-review: In this paper, the authors aim to develop a new method for credit assignment, where certain types of future information is conditioned on.  The authors are well-aware that naive conditioning on future information introduces bias due to Berkson's paradox (explaining away), and introduce a number of corrections (described in section 2.4 and 2.5).

The authors illustrate their approach via a number of simulation studies and constructed problems.

I think it would be nice if the authors found a way of connecting their notion of counterfactual to one used in causal inference (for instance, I think there is a connection via e.g. importance correction terms).

Reviewers were worried about the contribution being incremental given existing work (from 2019), and relative simplicity of the evaluation of the approach, compared to existing similar work.

Decision: Reject