Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: Update: See the comments. Review score remains at 7.

The paper considers learning value function approximators that take a policy representation as an input to the value function to allow generalization across different policies. There are several other recent works of this type. The aspect differentiating the current work is the type of policy representation used that allowed the method to scale to policies with 10^5 parameters.

The paper considered two types of data sources for the policy representations:
1. State action pairs obtained by the policy.
2. The policy parameters.

Then, the policy representation is formed by passing the data through a multilayer neural network that embeds the data into a lower dimensional representation.
The embedding is fed into the value function as an input, and the network creating the embedding is trained end-to-end to minimize the value function prediction error. Optionally they also added auxiliary losses such as contrastive learning.

Experiments were performed using PPO combined with their new value function and compared to the standard PPO as well as variants taking a random policy representation and also a policy representation with no emebedding. They performed experiments on 6 MuJoCo tasks, and generally the new methods outperformed the baselines.

tSNE plots showed that the policies with similar representations had a similar performance at the task (when comparing the achieved rewards). Moreover, experiments showed that the value prediction error was smaller for the new method compared to a standard value function method, and also that the value function was accurate for a test set of new policies not encountered during training.
main_review: The paper was reasonably well written. The experimental work examined the performance from multiple aspects and showed that their method could deal with the issue of the large parameter space of the policy by embedding it to a smaller space.

I have a question about the state action pair embedding: When you are applying the value function to a new policy, do you first need to gather new data to perform the embedding to compute the value?

One down-side of the work is that it is not compared to the other recent works doing a similar thing, e.g. 
J. Harb, T. Schaul, D. Precup, and P. Bacon. "Policy evaluation networks"
or
F. Faccio and J. Schmidhuber. Parameter-based value functions.

Although, these works are quite recent, so maybe it's not expected.
The paper could also benefit from fixing some English language errors, but it's not a big issue.
I would also suggest changing the title: I'm particular confused by why the "Represent your own policies" is added in the beginning. Doesn't everyone decide their own representations?
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The authors propose PeVFA, an extension to value functions that include not only states (or state-action pairs), but a representation of the parameters of the policy. They apply their method to PPO and show experimentally that such policy representations help improving over standard PPO. Unfortunately, the theoretical results in the paper seem trivial and disconnected from the rest of the paper. The value functions presented in the paper were already introduced in previous work. The authors compare extensively these methods to their approach, but introduce their PeVFAs separately, creating some confusion on the novelty of the work.

main_review: 

  - I like the extensive comparison between PeVFAs and PBVFs[1]/PENs. I think this work can provide and important contribution in that line of research by providing useful ways of representing the policy parameters. This tackles the main limitations of PBVFs. However, I think the authors should present their work as an improvement over PBVFs (in particular the PSVF), rather than introducing PeVFAs from scratch and then claiming after that this was already done in PBVFs. Note that PBVFs are theoretically sound to receive any representation of the policy, although in their experiments no representation is used.
   - The theory proposed is a basic analysis result and should be omitted. In particular, if a function is L-continuous and it decreases in a point, then there exists another point close enough such that the function decreases also there. Moreover, the contraction property is never holding in the experiments performed, hence the theoretical results are completely disconnected from the paper.
  - The assumption in Theorem 1 seems to be a technical step that is needed to prove the theorem itself, but not a reasonable assumption. It states that the sum of the losses of two consecutive policies has to be lower than the distance between the value functions of the two policies. How can one interpret this? 
 - In line 229 the authors claim that Figure 3 demonstrates the consequences of Theorem 1. This is not possible, since the assumptions of Theorem 1 are never met.
- I think the term GPTI is never defined
- PBVFs are used for offline learning (not online)
- the citation on PBVFs refers to an old ArXiV submission and not to the version published [1]


Again, the contribution relative to policy representation seems interesting and useful and it should be the main focus, but the paper in its present form is not ready for acceptance. If the authors remove the theoretical part and address my concerns about how they introduce PeVFAs, I will increase the score accordingly.

[1] Francesco Faccio, Louis Kirsch, and J¨urgen Schmidhuber. Parameter-based value functions. In Int. Conf. on Learning Representations (ICLR), Virtual only, May 2021

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper studies a class of reinforcement learning methods which utilize a value function which in addition to the state is conditioned on some representation of the current policy. The idea being that this can allow the agent to learn a value function that explicitly generalizes among policies as the policy changes over the course of learning and ultimately improve performance. The approach is named Policy-extended Value Function Approximation (PeVFA).

They present some theoretical results with essentially show that if we assume that lowering the error in the value function for one policy results in sufficiently lower error for nearby policies, and the policy does not change too much between iterations, we can bound the value error for a policy dependent value function in terms of distance between policies at subsequent iterations. They also show that assuming the policy dependent value function generalizes well it can provide a benefit over a policy independent value function in terms of value error during a policy iteration procedure.

They further present empirical results using policy dependent value functions with a variety of different techniques for conditioning the value function on the policy. Policy representation techniques tested include: Raw Policy Representation (RPR) which uses all the parameters of the policy network as input to a value approximation, Surface Policy Representation (SPR) which uses a set of state action pairs gathered from running the policy, and Origin Policy Representation (OPR) which again uses the policy network parameters, but preprocessed by a permutation invariant transform. In addition, they test combining these representation strategies with auxiliary losses encouraging policy recovery from eh learned representation.

They study the use of policy dependent value functions in two settings, a supervised learning like setting where a number of fixed policies are initially evaluated to train the value function and a reinforcement learning setting where a policy is adapted online while learning a policy dependent value function for the policies the arise along the way. Results indicate that in the supervised case the learned value function can generalize to unseen policies.  In the reinforcement learning case, policy dependent value functions appear to confer some performance improvement when used with PPO compared to standard value functions.
main_review: The basic idea of somehow learning policy conditioned value functions as a way to more rapidly adapt to a changing policy is interesting and reasonable. A key challenge in implementing this is obviously how to encode a policy in a way that can be input to a neural network. My overall impression is that the empirical aspect of this work was reasonable, but could be improved, while the theory does not seem terribly useful.

Having checked over the theory in reasonable detail the results appear correct to me. However, the theory is arguably not very useful as it's intuitively obvious that if we assume the value function generalizes between policies meaningfully such that reducing the loss for one policy reduces the loss for nearby policies sufficiently, then such a policy dependent value function function will confer a benefit over a policy independent one. The hard part is how to actually achieve such generalization, and how to encode a policy into a form that is convenient to input into a neural network.

Also Theorem 1, does not actually use any of the lemmas, which makes it strange to call them lemmas, and seems trivial. Essentially it just applies definitions and the triangle inequality to conclude that if the sum of the value errors for two subsequent policies is smaller then the actual difference between the true values of the policies, then the value error for the second policy will be less than the error in approximating the second policy by the value of the first. The authors claim that this theorem shows that we can get a benefit from using value functions that depend directly on the policy. However, it's clear that such a benefit is possible if the policy dependent value function happens to generalize well even without this theorem, and it's not really clear to me what this theorem adds. Again the hard part is actually ensuring such favourable generalization occurs.

On the empirical side the results for learning the value of multiple pertrained policies are interesting and seem to demonstrate the possibility of improving generalization to new policies using PeVFA. The proposed algorithm for reinforcement learning seems to rely on growing a buffer containing trajectory data for all past policies and repeatedly retraining on it, which seems fairly restrictive in practice. Nonetheless, it is perhaps a reasonable way to start in order to highlight the potential utility of PeVFA. The results seem to demonstrate the PeVFA can provide an advantage over standard VFA in the settings explored, however aspects of the empirical evaluation could be improved and clarified to make the conclusion more clear.

According to the appendix, the added hyperparameters for PPO-PeVFA were tuned using a grid search, but the hyperparameters of the PPO baseline were not which perhaps confers an unfair advantage to PPO-PeVFA. Furthermore, I couldn't find any details on how the shared hyperparameters were selected this should be stated clearly to aid in interpreting the results.


Minor Corrections and Suggestions
=================================
Line 34 'efficiency value approximation'->efficiency of value approximation'

Lemma 1: Is the assumption $f(\pi)_1\leq f(\pi_2)$ really necessary here? It seems a little unnatural and it seems to me you could drop it and modify the result to $\hat{f}(\pi_2)\leq\gamma f(\pi_2)+\gamma \mathcal{M}(\pi_2,\pi_1,\hat{L})+\mathcal{M}(\pi_1,\pi_2,\hat{L})$

For proofs in the appendix: it would help readability if you rewrite the theorems/lemmas before the proof.


Questions for authors
=====================
Line 313: What is meant by Ran PR? I can't find any details on this and I don't really know what is meant by "randomly generated policy representation". Randomized how exactly?

It seems quite odd to me that OPR uses permutation invariant transformations. Surely the ordering of the weights and biases is important for a multilayer network, what is the justification for processing it in this way?
limitations_and_societal_impact: Limitations are reasonably well covered. I don't see significant potential for negative societal impact.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper studies value function approximators that explicitly take the policy as input, which the authors named policy-extended value function approximator (PeVFA). This is different from the common value function approximators which implicitly represent the value of the current policy. The authors discuss two scenarios where PeVFA could provide potential benefits via generalization: global generalization where the policies are sampled from a fixed distribution and local generalization where the policies are sampled from the policy improvement path. The latter is a typical scenario for policy improvement. The authors provide a theoretical analysis of the generalization properties of PeVFA. To apply PeVFA in practice, the authors also investigate several different ways of learning a compact embedding for policies: end-to-end learning, auxiliary loss of policy recovery, and contrastive learning. Empirical study shows that PeVFA improves over the conventional value function approximators on the Mujoco continuous control benchmark. The empirical results also provide some insight into the effectiveness of different policy embedding learning methods.
main_review: Things to like:
* 1. Overall the paper is written in a coherent manner. The motivation is clear. The discussion on how to learn compact embeddings of policies addresses a critical issue for applying PeVFA in practice.
* 2. The idea of taking the policy as an input to the value function approximator is interesting.

Things that can be improved:
* 1. I find Theorem 1 not particularly insightful - it basically says "when PeVFA generalizes well, it helps". Maybe the authors can make a more detailed comment on Theorem 1 in their response to help me understand its insight.
* 2. While the local generalization scenario is theoretically possible, it is unclear to me how it would work in practice when the PeVFA is implemented by standard deep neural networks. Intuitively, the policies along the policy improvement path have increasing values. Therefore the PeVFA must "extrapolate" to generalize well. However, it is known that standard deep neural networks are not very good at extrapolating without specialized architecture design. In the experiments, the PeVFAs were implemented by standard MLPs, which are not expected to generalize well. Yet they still demonstrated performance improvement. It is possible that my hypothetical argument is wrong, but I would find the empirical results more convincing if the authors can address this question.
* 3. I find the choice of implementing PeVFA with PPO rather intriguing. Based on Theorem 1, the larger the value difference between consecutive policies during policy iteration, the better the chance PeVFA could offer benefits via generalization. However, like other trust region policy optimization methods, PPO puts a constraint on consecutive policies so that the updated policy cannot deviate too far from the old policy. Intuitively PPO introduces an upper bound on the room for the improvement of PeVFA. Perhaps the authors could provide some explanation on why choosing PPO as the base agent.
* 4. It is unclear to me how the proposed policy embedding learning methods would scale up to larger policy networks. One possibility is to separate the policy network into a state embedding module and a policy head and only encoding the policy head which is usually smaller. Could the authors comment on how to scale up and perhaps provide some preliminary results?
* 5. L231-232, "... PeVFA can achieve lower approximation error than conventional VFA after the same number of training with the same on-policy samples". What does "the same number of training" mean here? Intuitively, PeVFA needs more training, though not necessarily more data, to outperform the conventional VFA because it is learning a more complicated function.

Minor errors and typos:
* 1. L301, delete "to"
limitations_and_societal_impact: This work is not directly related to any real-world applications thus is far from making any negative societal impact at the current stage.
review_title: null

Meta-review: The reviewers agree that the paper tackles an important problem and that the proposed approach has its merits. They also agree that the paper is clear and easy to follow for the most part. The main concern expressed in the reviews and discussions regards the theory, which was considered rather trivial and somewhat disconnected from the rest of the paper. The overall opinion is that the main result builds on fairly strong assumptions that render them somewhat vacuous:  it is intuitively obvious that if the proposed approximator generalizes well across policies this should be useful. The real challenge lies in how to enforce this property in the first place.
 
Given the above, the reviewers suggested during the discussion that the authors remove the theory and present the paper as essentially an empirical argument. Although the authors initially resisted the idea, eventually they agreed to adopt the suggestion. There was some discussion whether the paper without the theory should be accepted or not, as all the reviewers seem to agree that the empirical part of the paper is reasonably well executed and has some interesting insights in it. 
 
We eventually decided to reject the submission for two main reasons. First, the removal of the theory would be a fairly extensive modification to the paper. Second, and more important, it is unclear how much of a contribution the resulting empirical study would be with respect to the literature. The idea of adding an extra dimension to the value function corresponding to policies is quite straightforward and has appeared in the literature multiple times in different forms. Since the idea is already out there, the main challenge that remains to be resolved is how to represent these policies in a way that clearly promotes generalization across policies –especially those policies along the value-improvement path. 
 
This work can be seen as a promising first step towards the goal above, but more work must be done in this direction. We hope the feedback provided can help in defining the next steps of this research.

Decision: Reject