Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: Summary:

The authors propose PeVFA: a value function able to evaluate the expected return of multiple policies. They do so by extending the conventional value function, allowing it to receive as input the parameter (or a representation) of the policy. The authors study the local generalization property of PeVFA, propose possible ways of encoding the policy parameters and compare traditional PPO with an extended version of PPO using PeVFA. While the idea of generalization among many policies is an interesting topic in RL, there are many theoretical and experimental issues that prevent acceptance. Moreover, the authors do not at all compare their approach to recent work which also uses value functions with policy parameters as input.

Review:

Below the major problems I found:

- State and action value functions receiving as input the parameters of a policy, which generalize to unseen policies were proposed in a work on Parameter-based Value Functions [1] (PVFs, see version v1 from 16 Jun 2020). Recent work on Policy Evaluation Networks (PENs) proposes policy embedding for a value function receiving as input the parameters of a policy [2]. These works introduce value functions which generalize to many policies. The authors must relate their Surface Policy Representation to the fingerprint mechanism in PENs and introduce their V and Q in the PVF framework.

- Since the state space is not finite, $f_{\theta}(\pi)$, the approximation loss of $V_{\theta}$ should be defined considering an expectation over the state space, which is weighted by a distribution over the states (e.g. the on-policy stationary distribution). Since the loss proposed does not include a distribution over the states, it is not clear if the contraction assumption should hold in expectation or for every possible state. The former would imply that all the results should be stated in a probabilistic framework; the latter would be false in a continuous state setting.

- The authors assume that the class of value function considered can achieve zero approximation error. Where is this assumption used in the proofs and why is it useful?

- Theorem 1 seems quite trivial. It is trivial that if the loss is decreasing in one point and it has some smooth properties, then there exists a close enough point such that the loss decreases also there. Assumption 2 should be stated in a clearer way, differentiating the cases of Lipschitz continuous function, Lipschitz continuous gradient and Lipschitz continuous Hessian more clearly.

- Assuming that $\gamma_g$ in Corollary 1 is lower than 1 during the training process is quite unreasonable, and the authors do not check if this holds in their experiments. I expect that for Corollary 1 to hold, the learning rate should be extremely small, thus preventing learning.

- In Appendix A.2, there is an additional proof assuming that also $f(\pi_2)$ is Lipschitz. The authors start from the bound in eq (8), which is strictly less tight than the assumption $f(\pi_1) \leq f(\pi_2)$. By assuming only $f(\pi_1) \leq f(\pi_2)$ one would get the same final condition for $f(\pi_1)$, except for the term L_0 which would be zero. Hence the condition could be even less restrictive.

- In Corollary 2 it is assumed that the sum of the losses over 2 consecutive policies is lower than the distance between the optimal value function of the two policies. Is there an interpretation of this assumption or is it just a technical requirement to complete the proof? Please discuss why this assumption should hold during the learning process.

- When the policy representation is learned, the problem of mapping the policy parameters to the expected return becomes nonstationary, i.e. the same policy representation over time would be optimally mapped onto different values. How is this addressed in the experiments and how does nonstationarity affect the theoretical claims?

- From the experiments it is not clear if a representation for the policy is necessary. I would have expected to see a comparison between PeVFA with raw policy representation (RPR) as input and PeVFA using the learned representations. The authors should provide strong evidence that RPR is not enough and the policy representation is needed. Furthermore, the policy used is very small (2 layers and 2 neurons per layer). I would expect to see benefits in using a policy representation when the policy is bigger (e.g. 2 hidden layers, 64 or 128 neurons per layer).

- The authors provide no details about the hyperparameters used and about how they tuned their methods. Policy representation learning methods are only in part explained. Without further details, it is difficult to assess if the experiments provided were fair, and it is not possible to reproduce the results.

- In appendix B.1. the transition function is not reported. The state includes sinusoidal terms that do not appear in the reward function. Why is it necessary for the agent to observe these terms? Are not just the x-y coordinate sufficient?

- In Figure 3 and 7 it is not clear why there is only one learning curve for the policy policy and 6 value functions losses in standard PPO.

- Algorithms 1 and 2 make almost no distinction in training when the update is on-policy or off-policy, because the authors consider PeVFA as just a replacement for standard value function when the algorithm is already derived. However, if PeVFA is introduced before deriving the algorithm, the derivation leads to different policy gradient theorems (see [1]). Please discuss this issue.

- Figure 8 represents $\pi(a|s)$ for each possible a, s. Therefore I would expect that for each s, the integral over A of the curve is 1. However, the area under the curve is much lower than 1. What is the explanation for this?

- Last page in the Appendix is truncated.

I think the most interesting contribution of this paper is the proposed policy representation. I would like to see a revised version published in the future. However, a lot of additional work is needed to address the aforementioned problems in the theory and the missing experimental evidence & implementation details & comparisons to very similar related work. 

Minor:
- The title is grammatically incorrect.
- "with a limited parameter space" -> what does limited mean? Finite? Or that $\Theta$ is only a subset of the space of value functions?
- same for "unlimited"
- Appendix C.1 "Advantage Acotor-Critic" -> Advantage Actor-Critic


[1] Francesco Faccio and Juergen Schmidhuber. Parameter-based Value Functions. arXiv preprint arXiv:2006.09226v1, 2020.

[2] Jean Harb, Tom Schaul, Doina Precup, and Pierre-Luc Bacon. Policy evaluation networks. arXiv preprint arXiv:2002.11833, 2020.

********************

Edit: responses after rebuttal:

> The class of PVFs are developed to be applied and evaluated in the online learning setting.

This is not true. See for instance off-line and zero-shot learning experiments with PVFs that can learn new policies which do not interact with the environment and still show generalization.

> We think this may also be a reason to the insignificant improvement compared with their baselines in their experiments (their Figure 2), even the inferior results in simple tasks like InvertedPendulum and CartPole. In contrast, our proposed policy representations enable PeVFA to be compatible with normal-scale policy networks and we then demonstrate the superiority of PPO-PeVFA against PPO in standard continuous control task of OpenAI Gym

I would strongly discourage you from claiming this. PPO-PeVFA is built on top of PPO, so it is very hard for your algorithm to perform worse than the baseline. On the other hand, PVFs and PENs are novel algorithms that rely COMPLETELY on the prediction of the value function. It is expected that they might outperform baselines in some environments and be comparable or worse in others. The PSSVF proposed in the work on PVFs is outperforming the baseline ARS (Mania et al. 2018) in all environments but Reacher, even when the policy is a neural net with 2 layers and 64 neurons per layer.

About Q2:

1. If the authors are using the $L_{\infty}$ norm, then the results practically apply only to finite state MDPs. Indeed, this is the case for the papers [2,3,4] cited by the authors in the comment above. However, the experiments proposed in the paper deal with continuous state spaces, so the theoretical results are disconnected.

It is still not clear what the assumption in Corollary 2 means and if that assumption is met in practice. If the authors cannot justify the assumption, the Corollary should be removed. The novelty of Theorem 1 is not obvious, since it seems to be a quite trivial result from Mathematical Analysis.

> The results show that PeVFA consistently shows lower losses (i.e., closer to approximation target) across all tasks than convention VFA before and after policy evaluation along policy improvement path, which demonstrates Corollary 2

Again, is the assumption of Corollary 2 met here? If you have a logical rule (or corollary) of the form: assumption -> consequence, and you show that the consequence holds, this does not imply that he assumption is true!

One main argument in favor of PVFs is that even with a much bigger policy, PVFs are able to outperform the results of PENs. This suggests that Raw Policy Representation (RPR) can be a strong baseline and any work trying to introduce a different policy representation should at least compare to RPR. Without such a baseline it is difficult to assess the benefits of the proposed policy representations. 

Note that PeVFAs are PVFs! The PVF formalism already accounts for all kinds of policy representations.

Of course, all of these relations should also be clarified in title and abstract.


(Mania et al. 2018): Horia Mania, Aurelia Guy, and Benjamin Recht (2018).  Simple random search of static linear policies is competitive for reinforcement learning.  In Advances in Neural Information Processing Systems,pp. 1800-1809, 2018.




review_title: Issues: theory / experiments / related work

final_score: rating: 5: Marginally below acceptance threshold
review_content: This work proposes an interesting idea of using the policy as an input to the value function. This is an interesting idea, moving away from the conventional approaches, and proposing to evaluate the value of a policy, by taking the policy as an input itself. Overall, it is an interesting direction, but there are few major concerns I have about the work, including some overstated claims which are not backed up properly. 

Comments : 
- The proposed PeVFA framework is interesting, since the value function can take as input many different policies, providing a way for generalization among policies. Based on this, the key algorithmic contribution is to propose generalized policy improvement (GPI) which can guarantee policy improvements among many different policies following the generalized value functions. Previous works have considered taking goal states or other heuristics as input, often for exploration purposes. In contrast, this work suggests that taking policy as input might lead to better generalization, especially in transfer learning situations, as originally proposed by Successor Features (Barreto et al) proposing the GPI algorithm. 
- I do not think, however, that this idea is completely novel. Previous work on policy evaluation networks (Harb et al; arxiv paper from earlier this year) also proposed a similar approach, and this paper does not properly compare or take account of similar ideas from literature. It would be useful if the authors can comment on how their approach is different from Harb et al., given that the framework is almost similar. 
- The key idea is to propose value generalization among policies, such that the value fn of a policy can generalize to a new policy, as outlined in figure 2 and section 3.1. I have a major concern about Assumption 1 being made - I am not fully convinced whether this is a valid assumption for the contraction property to hold in this. I do not think it is a major bottleneck, but to state this assumption makes me question whether the learnt PeVFAs can at all be useful. 
- Theorem 1 is well stated. However, the proof of theorem 1 is not properly justified, and makes the contribution less significant. It would be useful if the authors can expand more on this, especially how the Lipchitz assumptions from Nesterov et al.,  are valid in this case. Otherwise, theorem 1 itself does not add much value for the overall contribution. 
- My major concern with this work is how are PeVFas actually implemted in practice. The authors seem to build up from PPO, which would require the policies to be separately parameterized too. In this case, it would mean that the policy network is fed as input to the value function network itself. How does this approach scale up when the policy network is large? More importantly, how are gradients computed in this case? It is not clear whether this sort of hypernetwork idea is applicable for practical implementation of this approach, and this seems to be a major unavoidable issue with the work. Similar issues also arise in Harb et al., which they describe as the "fingerprinting" step. However, I think this is a major bottleneck for any of these works. Although the idea itself has potential, but taking policy as input, and considering parameterized policies might not be the way to go for this? I wonder if this kind of approach may be more preferrable in value based settings? Can the authors comment on this? Figure 4 seems to justify this sort of network architecture, but is not convincing. 
- The paper discusses on the representation learning aspect of the policy. I understand that this is useful, leading to the generalization aspect this work is trying to propose. However, the claims are not made clear and difficult to understand. Is this work suggesting that the policy representations learnt through this framework can help learn invariant representations, which are then further useful for generalization? Some of the experimental demonstrations seem to address on this - however, it is not clear to what extent the invariant representations are indeed meaningful? As a reader, I was expecting the authors to compare and discuss experimentally with successor features and variants, for the GPI algorithm? However, there seem to be no discussions on those?
- Empirically, it is not clear what the paper is trying to propose. The authors claim to build up from PPO, but the experiments are not well justified for the generalization or transfer learning aspect? What would be the key takeaway from the experimental results? Table 1 is almost meaningless, given that the plots for these results in figure 11 in appendix shows marginal performance improvements. Experiments are only compared with PPO on control tasks, which does not have any generalization aspect per say? What are the authors trying to demonstrate in this case? I would suggest the authors propsoe their approach on simpler tasks, e.g four rooms domains and variants, and show how the proposed framework can be useful to transfer across tasks? Would that be a doable experiment that might add value to the contributions?
- Another minor issue is that it is not clear whether the algorithm can be implemented online or offline? How are the value functions trained, when the policy is used for rollouts? For example, if policy 1 is used for rollouts to compute the value functions, and doing a policy improvement step, how are these samples re-used when policy 2 is used as input to the value function network? I assume this will make the approach off-policy, since the rollouts under one policy would induce distribution shift compared to the other policy? I am not sure I fully understand how this algorithm can be implemented in practice, and what are the major drawbacks? Can the authors comment on the major difficulties of practical implementation of this idea? It would perhaps be useful if the pseudo-code of the algorithm can be included?

Overall, I think it is an interesting idea; but there are major concerns I have about the work which seem to be unaddressable? My main concern is with the practical implementation of this algorithm (I think I understand the intuitive and overall motivation for this, and why doing this might be useful). The paper claims to take this approach for the first time. However, there seems to be a prior work on a similar idea, which is not addressed in this work either. It would be useful if the authors can clarify on some of these doubts, for me to fully understand the practical significance of this work. 
review_title: Interesting idea, but few overclaims; algorithmic implementation not clear, with key things not mentioned clearly. Difficult to understand the true significance of the work.

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary 

The paper proposes to learn a value function that takes as input both a state and a policy embedding (PeVFA), which is used to design a new version of a generalized policy iteration (GPI) algorithm, named PPO-PeVFA. The authors also introduce a new way of learning policy embeddings and show superior results relative to vanilla RL (PPO) on several MuJoCo tasks. 


Strengths 

I found the proposed idea to be novel and interesting. Considering the value of multiple policies is an interesting and neglected line of work, which could prove useful in many ways for RL, so I commend the authors for taking a step in this direction. I also particularly liked the theoretical analysis, as well as the experiments supporting the claim for local and global generalization. However, I believe there are a number of issues with the experiments and clarity of the paper, which I would like to see addressed.
 

Weaknesses

My main concern about this paper is the significance of the results and the empirical evaluation. The performance gain from using PeVFA does not seem very significant. After looking carefully at Table 1 and Figure 11 in the appendix, it seems like PPO-PeVFA is typically within a standard deviation of PPO. In addition, some of the reported results for PPO are much lower than the ones reported in the original PPO paper (Schulman et al. 2017). For example, on Hopper-v1 the reported numbers after 1M training sweets are 1600 and 2200, ,while for Walker2d-v1 they are 1500 and 3000 for the authors’ implementation and the original one, respectively. 

In addition, I found the description of PPO-PeVFA to be quite confusing, so I think it can benefit from more details about the training procedure. Does the policy network share any parameters with the PeVFA network? Usually, parameters are shared between the policy and value function to improve learning, but it doesn’t seem like this is the case with PPO-PeVFA. Typically the policy and value function are updated at the same time in PPO but from Algorithm 2, it seems like before a policy update, you first update the PeVFA network with data from the current policy and then use the updated value to update the policy using the PG loss. Is my understanding correct? If this is the case, then it seems like the method is using some kind of privileged information about the current policy so an ablation that uses the same training scheme of first updating the value function (without conditioning on a policy) using the Monte Carlo rollouts from the current policy and then updating the policy should be included. 

I also think it would be useful to include comparisons with other baselines that achieve strong results on MuJoCo such as SAC or TD3 or alternatively show that PeVFA can improve over them as well. 

Do you have an intuition for why the difference between PeVFA and VFA is more significant for InvertedPendulum compared to Ant (and the other MuJoCo tasks) as illustrated in Figure 3b? 
I’ve found these plots slightly worrying because it seems like the value approximation gains are rather small for most of these tasks. It would be interesting to better understand the relation between these value generalization errors and the performance improvement.

Have you looked into the sensitivity of PeVFA’s generalization with respect to the set (and number) of policies it is trained on? One can imagine training it on the entire history of policies during training or on a subset of these (perhaps the most recent ones). It would be useful to analyze how this choice affects the results. 

Have you tried combining the contrastive loss with the auxiliary loss for action action prediction? Is this better than either of them or not?

Can you provide some quantitative metrics for the results in Figure 3a such as the MSE and ranking loss for the train and test values?

Why doesn’t Table 1 include results for InvertedPendulum? Can you add standard deviation for PPO and Ran PR?


Minor Points

Typos: 
“which are beyond the scope of this paper” instead of “which beyond…” in section 3.3
“In the last section” instead of “in last…” in section 4
“To answer the above questions” instead of “...answer above...” in section 5


Recommendation

While the proposed idea is interesting and novel, I believe more work is needed for publication (particularly in the empirical evaluation and algorithm description), so I lean towards rejection at this stage. 

review_title: Interesting idea but the clarity and experiments can be improved

final_score: rating: 7: Good paper, accept
review_content: The paper conditions the value function on a representation of the policy. The representation can be based on a batch of state-action pairs or based on the policy filters. When conditioning on the representation of a new policy, the value function can better approximate the value of the new policy. Experiments show benefits on continuous control tasks.

As mentioned in the paper, conditioning the value function on the policy was considered by some people before. It was unclear how to represent the policy. This paper makes it work and demonstrates clear benefits.
The policy representation can be still improved, and this paper can encourage people to try better representations.

The paper explains the ideas well.
I would like to see a more detailed description of the Surface Policy Representation (SPR). The Appendix D.5 is missing the review of the related works.

Pros:
- The value conditioned on the policy makes sense.
- Proposed end-to-end training, contrastive learning and auxiliary distillation to train the policy embedding.
- Tested raw policy representation, Surface Policy Representation and Origin Policy Representation.
- Demonstrated benefits in experiments.
- Visualization of the learned embeddings.

Cons:
- It is unclear how to practically encode policy representations for policies operating on images.

Minor typos:
- String "which beyond" should probably be "which are beyond".
- String "together with PeVFA end-to-end" should be "together with PeVFA is end-to-end".
- String "policies along the policy improvement path naturally provides" should be "policies along the policy improvement path naturally provide".
- String "Use PPR" should probably be "Use SPR".
review_title: Towards value function conditioned on the policy

Meta-review: This paper proposes to consider value functions as explicit functions of policies, in order to allow generalization not only on the state(action) space, but also on the policy space.
The initial reviews assessed that the paper was dealing with an important RL topic, but also raised many concerns about the position to previous works (PVN and PVF), the theoretical contributions and the experiments. The authors provided a rebuttal and revision that only partly addressed the initial concerns (check also the review of R3, updated to provide additional feedback following the author’s rebuttal).
The final discussion led to the assessment that the paper is not ready for publication. Remaining concerns include clarity, claims being not fully supported by the experiments, theoretical aspects and missing baselines.

Decision: Reject