Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents Policy Gradients Incorporating the Future (PGIF), a novel approach to incorporate future information during training to improve performance of model-free RL agents that must overcome the challenges of planning in environments with sparse rewards. The approach involves relying on recent work in "Z-forcing" in which training incorporates information about the future, encoded as a learned latent state that depends on the full not-yet-executed trajectory and state information. The future information helps the agent to learn effectively and overcome challenges associated with credit assignment notorious in partially observed environments; additional losses are added to ensure that the latent sate contains sufficient task-relevant information and that the planner does not rely too heavily on future information. The authors show solid results in a number of challenging environments, outperforming competitive approaches (including PPO) in nearly all of these, showing the effectiveness of the technique in both Online and Offline RL experiments.
main_review: Overall, this is a solid paper, that combines an interesting and novel theoretical idea and convincing theoretical results. The clarity of the technical content is a particular strength, and the authors' clear prose helps to communicate non-trivial theoretical concepts and provides intuition for their design decisions. There are still a few places in which I believe the clarity of the paper could be enhanced that would serve to make the paper both easier to follow and the results more impactful.

The main change I would suggest is to rely less heavily on Appendices to communicate conclusions about the various other activities. For example, while the reference to Appendix.F in Sec. 3.2 is quite informative (and mentions that applying PGIF to the policy and value functions degrades performance), for other content, the reader *must* go to the appendices to  understand the conclusions of supplementary experiments: e.g., appendix G in which Transformers are used instead of an RNN; there should be couple sentences in the main text summarizing those results. While I do not think it is an issue that there is so much content in the appendices---and, in fact, it may be a benefit so that more time is devoted to the clear descriptions of the technical content of the paper---mentions of the appendices containing results (including both Appendix G and E) should be expanded on.

Some other smaller questions and suggestions are as follows:
- [Sec 3.2] It would be worth mentioning that the Force and VPN objectives are separate from one another. It was only by the time that section was over that it was clear that only one would be used at a time.
- [Sec 5.1] Some readers may not be as familiar with the Gym-MiniGrid environments. Mention that the agent is given only given local ego-centric view of its environment, which is what gives rise to the partial-observability.
- [Sec 5.2] While not essential, it would be helpful to understand why Fig. 2 includes results only from the State-based forcing of PGIF. Would the results from VPN look quite different? Its omission is especially odd considering that the VPN results are clearly better for the experiments from Table 2.
- [Sec 6] I appreciate the potential risks highlighted in the Discussion section. It might be helpful to include how much slower "much slower to train" implies. Could the authors include an order-of-magnitude number for the difference in wall-clock training time for one of the experimental environments?

summary_of_the_review: This seems to be a solid paper with an interesting and well-executed novel theoretical contribution tackling a difficult problem of relevance to many in the RL and planning-under-uncertainty communities. The experiments are also convincing, and compare against popular RL strategies. I believe that the clarity of the paper could be further enhanced, yet point out that the paper is already well-written and easy to follow.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method for incorporating future trajectory information when training model-free RL methods. Two technical challenges were reducing over-reliance on future information and allowing the policy to make predictions without future trajectory information at deployment time. These are addressed by introducing an information bottleneck. Optimization issues with latent variable models are tackled using two different Z-forcing approaches. The method shows favorable empirical performance in both online and offline settings. 
main_review: # Method
- I see why the proposed method would help in partially observed environments: conditioning on some representation of the future allows the policy to condition on a function of the hidden information and give a lower variance policy gradient. However, in fully observed online settings it is not obvious to me why this method improves performance, and I don't see much intuition provided in the paper. 
- Equations 4 and 5: why are $u$ and $z$ separate? Did you try learning a single latent variable for both policy and $Q$ function? 
- This latent variable $u$ seems to contain information about both the dynamics and logging policy since these interact to form the future. If we know the logging policy, the information bottleneck means we artificially throw away information that we know. This seems like an undesired side-effect of this information bottleneck. 
- 3.2: "Especially in settings with highly stochastic learning signals". What does this mean?
- I didn't follow how $p_{v^{(u)}}$ was updated. It is described as a "learned prior" but it only appears in the KL term in the loss. If the network for learning this has sufficiently many parameters will it not just overfit to tracking $q_{\phi}$? 

# Experiments
- It seems that Table 1 might address partial observability. If the first state and action are not maintained in the state for the rest of the task (if I have misunderstood this please clarify), we do not have an MDP. If this is true, I don't see a convincing case for this method improving performance much outside of partially observable environments (in online RL) besides the minor improvement in appendix E, and the great results in appendix H. If the focus of improvement is in partially observable or offline settings, this should be made clear in the motivation. 
- In Appendix H, I did not find the explanation for improvement convincing, even though the results themselves are impressive. "We hypothesize our algorithm has benefits in these environments since as soon as it obtains a reward signal it can adapt quickly and make use of the signal by incorporating it in both policy and value optimization, therefore accelerating learning." With enough Bellman updates, SAC should also be able to propagate this information back to states earlier in the trajectory. How many rounds of Bellman updates are being used in SAC? Is it possible that in fully observed environments, PGIF is giving an optimization benefit by skipping this reward information directly to early-trajectory states instead of indirectly through Bellman updates? 
- Figure 2: main text should highlight LavaGapS5 as a case where we don't see improvement. Is there some intuition as to why?
- Section 5.1 is supposed to evaluate the method on sparse rewards, testing credit assignment. But a partially observed environment is used for some of the experiments in this section. Partial observability is the focus of 5.2.
- Table 1 shows great results when using PGIF-SAC, but then in the next experiments in Figure 2 the PPO version is used? 
- Why is HIMO not compared to in Table 1? Same for Figure 3. It is explained in the appendix but could perhaps be a footnote in the main paper. 
- 5.2: "more realistic scenario" is probably not an accurate way to describe this particular modification to the environment. 
- Experiments in appendix E. The environment has no partial observability but we are in a finite horizon setting, meaning that the number of remaining steps in an episode is not known to the agent. It could be that the slight improvement in performance over SAC is because PGIF can provide the policy with the number of remaining steps. This can be tested by including the number of remaining episode steps into the state and rerunning the experiment. 
- How does the method perform with no Z-forcing? This seems like a straightforward ablation.
- Mesnard et al. (2021) is a very similar approach that is not compared to.  
Why is the improvement in offline continuous control much greater than the online equivalent? Is it because in the offline case the future contains information about the logging policy? 

# Minor points
- The proposed method is called PPO-PGIF in the text but in Figure 2 the legend uses PGIF-PPO. 
- Appendix B: " Attention augment agents utilize dynamics" Typo
- "PGIF agent is able to efficiently" typo
- Row 1 of Table 7 in the appendix: typo?
- The text on figures (eg legends) is too small. 

I'm not too familiar with the related work on hindsight, so my review takes the paper's comparison with this prior work at face value. 
summary_of_the_review: Here I review the key constructive points:
- Beyond the online ant setting in the appendix, I don't see much evidence that this method improves performance outside of partially observable environments in the online setting. The intuition behind why it would help in fully observed environments is also not clear from my reading of the paper. Given this, I think that the paper should focus the motivation more heavily on the POMDP and offline setting. This was the main reason for my score. I'm happy to discuss if I've misunderstood something about the utility of the method in the fully observed setting. 
- From my description above I think there are a few ways in which the experiments could be more convincing. 

Why I liked the paper and recommended to accept:
The core idea in this paper is very elegant and the variational inference + z forcing to implement this idea is well done. As far as I can tell the use of Z-forcing and the variational bottleneck is novel in this setting. The results on partially observable and offline settings are impressive.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to incorporate future information for more accurate policy and Q-function estimation, and consequently to derive variants of policy gradient based algorithms such as PPO, SAC, and BRAC. The authors argue that the ability to condition on the future information enables better credit assignment. The authors empirically demonstrate that their approach improves performance on a range of tasks that feature various challenges, including sparse rewards credit assignment, partial observability, and offline RL.
main_review: Strengths:
1) The paper attempts to address the problem of credit assignment in RL which is very important. The authors build off of the existing idea of learning future conditioned policy/Q function and introduce several tricks (informational bottleneck and z-forcing) to make it work.
2) The paper demonstrates that the proposed approach can be applied to several RL algorithms, such as PPO, SAC, and BRAC.
3) The paper features an extensive empirical study across several tasks that feature various challenge, such as sparse reward credit assignment, partial observability, and learning from offline datasets.

Weaknesses:
1) The novelty of the method is limited as it largely builds off of existing work from Harutyunyan et al, 2019 and Mesnard et al., 2021. The the authors attempt to differentiate their work from the prior methods, but the difference mostly comes from the fact that PGIF can be made off-policy, unlike the prior work (i.e. Hindsight Credit Assignment) which is on-policy.
2) It is not clear how much information about the future one can force into the prior distribution that is only conditioned on the current state.  I'm not convinced that this does any better than for example SAC (or DDPG/TD3) + n-step returns. In general, the paper omits comparison to any methods that uses n-step return.
3) The authors feature z-forcing as an advantage of their method, but I believe that the need for z-forcing stems from the fact that there is very little mutual information between the prior and posterior distributions, so the backward model without z-forcing will simply collapse to  a trivial solution in order to minimize KL-div to the prior distribution.
4) The final method ends up being extremely complicated featuring several auxiliary losses and the recurrent model. The empirical results are encouraging but not ground breaking. Thus I'm not sure the algorithm is practical and can enjoy wide adoption considering its complexity.

Questions:
1) Re the umbrella-length experiment: what episode length has been actually used? or this presents an average results across multiple horizons?
2) Re the umbrella-length experiment: it would be interesting to see a baseline that uses n-step returns, for example DDPG/TD3 + nstep. It would be much stronger baseline than SAC in these sparse reward/long credit assignment settings.
3) Re the MuJoCo partial-observability experiments: I don't think it is a fair comparison to compare SAC-based PGIF against vanilla SAC (that doesn't use recurrent policy/Q). Obviously PGIF, that has a way to deal with partial observability, will outperform SAC. Instead, the authors compare to PPO-LSTM, which can deal with partial observability, but not as strong baseline as SAC on MuJoCo. It would be interesting to also see performance of SAC-LSTM and PPO-based PGIF to make this experiment valid. 
summary_of_the_review: While the paper studies an interesting direction of learning future conditioned policy/Q-function, which is quite exciting, I have several issues with the paper, namely:
1) The incremental nature of the work.
2) The complexity of the method.
3) Several concerns regarding empirical evaluation.

Taking these into account, I'm not sure the paper contributes enough in its current state to justify an acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The work proposes to run two recurrent LSTM neural networks backwards from the end of an episode, giving the rewards $r$ and states $s$ as input to the networks, and producing two separate latent state ($z$, $u$) distributions for each time step. Such procedure can be done during training after having collected an episode. Once, the $z$ and $u$ are computed, these are given as an input to the Q-function $Q(s,a,u)$, and the policy network $\pi(a|s,z)$ (note that the action was already sampled during the episode, but here the probability is recomputed using the new $z$). And the policy gradient/critic updates are compute as usual together with this augmented state. One remaining question is how to pick the $z$ during the episode when the future states are not known yet. This is done by training a prior network $p(z|s)$ by minimizing the KL divergence to the latent state distribution obtained from the backwards network (then one can just sample a $z$ from this prior network during the episode). To aid the backwards network in learning meaningful latent representations, they use z-forcing, which is a technique where the variable $z$ is used directly to predict some quantity to ensure that it incorporates useful information. They consider two options: 1) predict $b$ the hidden state of the backward RNN, 2) predict the values, rewards, discounts.
(Note that this z-forcing is done for both $z$ and $u$).

They perform experiments on Bsuite umbrella-length, Gym-minigrid, custom partially observable MuJoCo environments, Offline RL on MuJoCo D4RL tasks. They compare with PPO, SAC, and other baselines. The experimental results improved in all cases, but their method also requires more computational time due to having to train the LSTMs.
main_review: Strengths
------------------------------------
- The clarity was mostly good.
- Experimental results appeared good (in terms of improving over the baselines, but I am not sure the baselines were well-implemented).
- The method is general and could be applied together with most RL algorithms.
- The method improved performance for both online and offline RL algorithms.

Weaknesses
------------------------------------
- It wasn't clear to me why the method should improve the performance.
- The experiments only looked at reward scores without examining any other statistic or property of the learning.
- The computational time is much larger compared to other methods (they said it's much slower, but unfortunately did not say exactly by how much).
- It was not clear to me that the baseline algorithms were solid, as it seems that the experimental results are new, and not reproductions of previous results in other publications.
- Some ablation studies are missing (in particular, adding a prior network to the models essentially increases the capacity of the Q and policy models, so I thought it should be checked whether the baseline algorithms may also benefit from using larger models).

Suggested decision
------------------------------------
Currently, I am leaning toward recommending to reject, as I cannot recommend accepting a paper unless I am sure the empirical results are solid. Potentially, this can be cleared up in the rebuttal.

Supporting arguments
------------------------------------
**Experiments**

I can't think of any good theoretical reason why one would want to do what the authors did in their paper, so I think that the experimental results are crucial to convey usefulness and validity of their approach. Currently, I was not convinced that the experiments are solid (see below), and I also think that reporting only reward scores is quite a limited method of experimentation as it does not explain why the performance may have improved. One of the clearest ways to show results that can be trusted is to reproduce existing results from another publication, then outperform those results, but it didn't seem like this was done.

The Bsuite umbrella-length experiment seems fine. The result of ~0.6 is higher than the result of ~0.5 for DQN reported in the bsuite paper. However, the new baselines in this paper (PPO and SAC) only had a score of ~0.4. Also, why were none of the other bsuite environments tested?

In minigrid there is also no reference to any previous publication showing the same results as produced in this paper.

The experiments with partial observability were based on the work of Yang & Nachum, (2021), but it appears executed in a different way, and it's not easy to see that the baselines reproduced any results from their work.

The offline RL experimental results also are not clear to me. For example, on halfcheetah your baselines achieve around 4.5k score, but in the BRAC paper, the algorithms achieve around 6k score. Where does the discrepancy come from?

Also, in the antmaze results in Table 10, your baselines achieve ~0, whereas the in the original paper
by Florensa et al. the baseline algorithms reached around 0.6 reward.

It is quite possible that the experiments are performed solidly; however, this is not explained sufficiently for me to understand this. I would appreciate if the authors can comment on the correctness of their implementations, and provide evidence that the results achieved by their baseline algorithms are competitive (e.g., by providing references to papers where the scores are similar to the baselines implemented in this paper).

**Concept of algorithm**

The algorithm was motivated from the point of view of allowing to "look into the future" to improve the optimization performance; however, typical policy gradient algorithms already do this when using Monte Carlo returns. Instead of using your approach, one can give perfect information of the future by just using the empirical return. Then what is the motivation of the method, and why should it help with learning?  One could think of some possible reasons such as variance reduction (perhaps it is doing something similar to variance elimination https://arxiv.org/abs/1811.06225 or http://proceedings.mlr.press/v100/cheng20a.html), or better representation learning; however, such points are not explored.

**Ablation studies**

The prior network $p(z|s)$ is essentially increasing the capacity of the policy network. I think it would be better to also compare to a version where the backward RNNs are not trained, but the $p(z|s)$ is just trained end-to-end as if it were a part of the policy with some additional latent noise $z$. It would also be interesting to know what happens if you increase the number of parameters of the critic (to account for the greater model capacity achieved by using the backward RNN).

Other notes
------------------------------------

Is the prior network also trained using the policy gradient loss (the action depends on the sampled $z$ from the prior, so it could be trained end-to-end)?

The discussion around equations 1 and 2 is not wrong, but
I think it would be better to explain that this is a surrogate loss
that allows easily implementing the policy gradient using
automatic differentiation, and that only the $\log \pi$ term will
be differentiated. Originally, the policy gradient theorem was not
presented in the way that you did.

On page 5 in the VPN section, what do you mean by predicting the discounts?
Isn't the discount factor just a constant? Also, is the encoding network
trained end-to-end?

"Our method is more versatile that previous works,"
-> "...versatile than..."

The figure axis labels are small and hard to read. Also, you used a
non-vector figure format, which makes them blurry. I prefer vector
format figures so that they look sharp when you zoom in. Moreover,
it would be better to use more than color to differentiate between
the lines.

"Furthermore, we find that our method is much slower to train than the
baselines we compare to, due to the fact that the architecture
requires training an LSTM, with at times, very long trajectories as
input."
It is good that you mentioned this, but can you roughly quantify it?
How much slower was it?

Table 7 top right, are you missing one digit in the PGIF score?
summary_of_the_review: The experimental results look impressive, but I was not convinced that the baselines were well-implemented. The experimentation also only looked at reward scores, so it is not clear why the performance improved. It was also not clear to me that there is any good theoretical reason why the method should improve the performance. Currently, I am recommending to reject. I hope the authors will provide strong arguments and evidence in their rebuttal.

---------------------------------------------------------------------------------------------------------------------
Update after author rebuttal
----------------------------------------
The authors have provided additional evidence on the correctness of their baselines.
They also performed additional ablations, such as checking the change in performance when increasing
the model capacity. I was satisfied with these changes, so I have increased the correctness to 3, and the score the 6.
My recommendation is a borderline accept, as I was satisfied that the method empirically improves the performance; however,
I think the authors could have done more to examine why their method improves the performance.

I still have some concerns with the paper:
- The experiments still only look at the reward curves. They could have explored points such as the prediction accuracy of the value function, the variance and accuracy of the gradients, or tried to do some experiment to examine the quality of the representations (although this one is more difficult, as I don't think there is a standard way to look into this).
- I am still not convinced by the authors' explanation of why "looking into the future" is useful. When the information bottleneck is removed, the value function would ideally just predict the Monte Carlo return. So the authors claim appears to be that "looking into the future" is good, but if it is done too much, then it stops being useful. I think this point could have been examined in more detail, e.g. by doing a study on how the information bottleneck coefficient affects the performance (admittedly, it would have been better for me to say this in my initial review). Previous works (e.g., https://arxiv.org/abs/2011.09464) doing related ideas provided longer more principled explanations of why their methods would be useful by arguing that it might reduce the policy gradient variance by removing the random fluctuations in the future.

Some more minor concerns:
- I still think the discount prediction requires more clarification in the appendix B.
- Regarding the discussion around equations 1 and 2. I think you still
need to give more clarification. If one differentiates this "loss"
twice, one does not obtain the correct Hessian of the RL objective
(e.g., https://arxiv.org/abs/1802.05098).  I think this section should
be explained as an automatic differentiation trick to obtain first
order policy gradients.
review_title: null

Meta-review: This paper presents a new reinforcement learning algorithm for POMDPs that specifically deals with the credit assignment problem. The proposed algorithm consists in using at each time-step t of a training trajectory the subsequent future trajectory that starts at time t+1 as additional inputs to the policy and value networks. Instead of using the trajectories directly, two RNNs are used to encode the trajectories into latent two variables that are then given as inputs to the policy and value networks. A key novel contribution of this work is the use of "Z-forcing" to help the RNNs learn the relevant information. Since future trajectories are not available during testing, a "prior" network is trained to predict the latent variable given a state. During testing, the latent variable is sampled from the network. Empirical experiments on simple simulated environments show that the proposed algorithm outperforms several baselines.

Key issues raised by the reviewers include the complexity of the proposed algorithm, the fact that several interesting results are in the appendix rather than the main paper, and the weakness of certain baselines. The authors responses helped clarify these issues, and additional experiments (such as a comparison to a DQN with n-step value updates) were performed and added to the paper. The reviews are updated accordingly.

In summary, the paper contains several novel ideas in the context of learning in partially observable environments. It is not entirely clear similar effects of the proposed algorithm can be obtained by using simpler tricks, but the evidence provided by the authors supports the claim that the algorithm outperforms several SOTA techniques in the context of POMDPS.
Decision: Accept (Poster)