What About Taking Policy as Input of Value Function: Policy-extended Value Function Approximator
Authors: Hongyao Tang
Zhaopeng Meng
Jianye HAO
Chen Chen
Daniel Graves
Dong Li
Wulong Liu
Yaodong Yang
Venue:  (2021)
Keywords: Reinforcement Learning
Value Function Approximation
Representation Learning

Abstract: The value function lies in the heart of Reinforcement Learning (RL), which defines the long-term evaluation of a policy in a given state. In this paper, we propose Policy-extended Value Function Approximator (PeVFA) which extends the conventional value to be not only a function of state but also an explicit policy representation. Such an extension enables PeVFA to preserve values of multiple policies in contrast to a conventional one with limited capacity for only one policy, inducing the new characteristic of \emph{value generalization among policies}. From both the theoretical and empirical lens, we study value generalization along the policy improvement path (called local generalization), from which we derive a new form of Generalized Policy Iteration with PeVFA to improve the conventional learning process. Besides, we propose a framework to learn the representation of an RL policy,  studying several different approaches to learn an effective policy representation from policy network parameters and state-action pairs through contrastive learning and action prediction. In our experiments, Proximal Policy Optimization (PPO) with PeVFA significantly outperforms its vanilla counterpart in MuJoCo continuous control tasks, demonstrating the effectiveness of value generalization offered by PeVFA and policy representation learning. 

Under review as a conference paper at ICLR 2021
WHAT ABOUT TAKING POLICY AS INPUT OF VALUE FUNCTION: POLICY-EXTENDED VALUE FUNCTION APPROXIMATOR
Anonymous authors Paper under double-blind review
ABSTRACT
The value function lies the heart of Reinforcement Learning (RI.), which de- fines the long-term evaluation of policy in a given state. In this paper, we pro- pose Policy-extended Value Function Approximator (PeVFA) which extends the conventional value to be not only a function of state but also an explicit policy representation. Such an extension enables PeVFA to preserve values of multiple policies in contrast to a conventional one with limited capacity for only one policy, inducing the new characteristic of value generalization among policies. From both the theoretical and empirical lens, we study value generalization along the policy improvement path (called local generalization), from which we derive a new form of Generalized Policy Iteration with PeVFA to improve the conventional learning process. Besides, we propose framework to learn the representation of an RL policy, studying several different approaches to learn an effective policy represen- tation from policy network parameters and state-action pairs through contrastive learning and action prediction. In our experiments, Proximal Policy Optimization (PPO) with PeVFA significantly outperforms its vanilla counterpart in MuJoCo continuous control tasks, demonstrating the effectiveness value generalization offered by VeA and policy representation learning.
INTRODUCTION
Reinforcement learning (RI.) has been widely considered as promising way to learn optimal poli- cies many decision making problems (Mnih et al. 2015 Lillicra et al. 2015 Silver al., 2016
You et al. 2018 (chrrec et al., 2019: Vinyals et al. 2019 Hafner et al. 2020) Lying in the heart of RL the value function which defines the long-term evaluation of policy. With function approxi- mation (e.g., deep neural networks). a value function approximator (VFA) is able to approximate the values of policy under large and continuous state spaces. As commonly recognized. most RL algo- rithms can be described as Generalized Policy Iteration (GPI) (Sutton & Barto, 1998). As illustrated in the left of Figure [ at each iteration the VTA is trained to approximate the true values of current policy, regarding which the policy are improved. However, value approximation can never be per- fect and its quality influences the effectiveness of policy improvement, thus raising a requirement for better value approximation (v. Hasselt 2010: Bellemare et al. 2017 Fujimoto et al. 2018).
Since conventional VFA only approximates the values (i.c., knowledge (Sutton et al.) 2011 for one policy, the knowledge learned from previously encountered policies is not preserved and utilized for future learning in explicit way. For example in GPI, conventional VFA cannot track the val- ues of the changing policy by itself and has no idea of the direction of value generalization when approximating the values of new policy. In this paper, we propose Policy-extended Value Func- tion Approximator (Pe,FA), which additionally takes an explicit policy representation as input in contrast to conventional VFA. PeVFA is able to preserve values for multiple policies and induces an appealing characteristic, i.e., value generalization among policies. We study the formal generaliza- tion and contraction conditions on the value approximation error of PeVFA, focusing specifically on value generalization along the policy improvement path which we call local generalization. Based on both theoretical and empirical evidences, we propose a new form of GPI with PeVFA (the right of Figure I which can benefit from the closer approximation distance induced by local value general- ization under some conditions: thus, GPI with PeVFA is expected to be more efficient in consecutive value approximation along the policy improvement path.

---

Under review as a conference paper at ICLR 2021
R1
To
17
TT
No, (Xx,)
Vo. (Zs,)
14a
Vo, (r,,
Vo, (X..)
Vo. (Xn,,
Improvement
Evaluation
Approximation
Generalization
Figure Generalized Policy Iteration (GPI) with function approximation, Left: GPI with con- ventional value function approximator Vg- Right: GPI with PeVFA Vo(Xx) (Sec. where extra generalization steps exist. The subscripts of policy and value function parameters Ø, 0 denote the iteration number. The squiggle lines represent non-perfect approximation of true values.
Moreover, we propose framework to learn effective policy representation for an RL policy from policy network parameters and state-action pairs alternatively, through contrastive learning and an auxiliary loss of action prediction. Finally, based on Proximal Policy Optimization (PPO), we derive a practical RL algorithm PPO-PeVFA from the above methods. Our experimental results demon- strate the effectiveness of both value generalization offered by PeVFA and policy representation learning. Our main contributions summarized as follows:
We propose PeVFA which improves generalization of values among policies and provide theoretical analysis of generalization especially in local generalization scenario.
We propose a new form of GPI with PeVEA resulting in closer value approximation along the policy improvement path demonstrated through experiments.
To knowledge. are the first learn representation flwwddimensional embedding) for an RL policy from its network parameters (i.e., weights and biases).
BACKGROUND
2.1 REINFORCHMENT LEARNING
We consider Markov Decision Process (MDP) defined as AS,Arr, P,T) where S is the state space, A is the action space, r is the reward function, P the transition function and y E [0, is the discount factor. The goal of an RL agent is to learn policy π E II where (a)s)) is distribution of action given state that maximizes the expected long-term discounted return. The state-value function (s) is defined in terms of the expected discounted return obtained through following the policy from state v=(ss) E,, IEte ''
s] for all S where re+1 = r(st, az). We use v = (.) to denote the vector of values for all possible states. Value function is determined by policy and environment models (i.e., P and r). For conventional value function, policy is modeled implicitly within table or function approximator, i.e., a mapping from only state to value. One refer to Appendix E. to see more detailed description.
2.2 EXTENSIONS OF CONVENTIONAL VALUE FUNCTION
Schaul al. (2015) introduced Universal Value Function Approximators (UVFA) that generalize values over goals in goal-conditioned RL. Similar ideas are also adopted for low-level learning of Hicrarchical RL (Nachum et al., 2018). Such extensions
also studied in more challenging RL problems, c.g., opponent modeling (He Boyd-Graber 2016 Grover et al. 2018 Tacchetti et al.) 2019), and context-based Meta-RL. RI. (Rakelly et al. 2019 Lee et al. 2020). General value function (GVF) in (Sutton et al.] 2011] are proposed as form of general knowledge representation through cumulants instead of rewards. In an unified view, each approach generalizes different aspects of the conventional VFA, focusing on different components of the vector form Bellman equation (Sutton)
& Barto 1998) expanded on as discussed in Appendix E.2
work, several works also study take policy as an input of value functions. Harb et al. 2200) propose Policy Evaluation Networks (PVN) to approximate the objective func- Eagepo [V" (80)) of different policy æ, where Po is the initial state distribution. Later
Concurrnt
tion
2

---

Under review as a conference paper at ICLR 2021
in (Raileanu et al. 2020), Policy-Dynamicc Value Function (PDVF) is proposed which takes both policy and task context as additional inputs, for the purpose of value generalization among policies and tasks so that to adapt quickly to new tasks. PDVF can be viewed as an integration of PVN and task-specific context learning (Rakelly et al, a 2019 Zhou et al. 2019a Both PVN and PDVF con- duct value approximation for given collection of policies and then optimize policy with gradients through policy-specific inputs (shorted as GTP/ below) of well trained value function variants in a zero-shot manner. We view this as a typical case of global generalization discussed further in Sec. 3 In contrast, we focus more on a local generalization scenario and utilize value generalization to improve learning during standard GPI process with no prior policies given.
Closely related our work, Faccio & Schmidhuber 2020 propose class of Parameter-based Value Functions (PVFs) which take policy parameters as inputs. Based on PVFs, new policy gradients introduced in the form of a combination of conventional policy gradients plus GTPI (i.e., by backpropagating through policy parameters in PVFs). Beyond zero-shot policy optimization, PVFs also utilize value generalization of PVFs in the iterative learning process. Our work differs with PVFs at two aspects: first, we consider a general policy representation in our proposed PeVFA along with framework of policy representation learning, in contrast to parsing the policy parameters directly; second, we do not resort to GTPI for the policy update in our algorithms and only utilize value generalization for more efficient value estimation in GPI. Moreover, in these previous works, it is not clear how the value generalization among policies can be and whether it is beneficial for learning or not. In this paper, we study the condition of beneficial value generalization from both theoretical and empirical lens.
POLICY-EXTEND VALUE FUNCTION APPROXIMATOR
In this paper, we propose Policy-extended Value Function Approximator (PeVFA), an extension of the conventional value function that explicitly takes policy (representation) as input. Formally, consider a function g I1I + X I in general form that maps any policy π to a n-dimensional representation X. = g(T) E X. The policy-extended value function V= : S x X + R, defines the values over state and policy space:
V(s, = Ex
7 r 10 = for all E STTEL II.
(1)
1=0
When only one policy æ is considered, Vss, X=) is equivalent in definition to a conventional value function en (s). Note that if the policy The is explicitly parameterized and its parameters are used as the representation, i.e. Ns. = w. PeVFA is equivalent to Parameter-based State Value Function (PSVF) in PVF family (Faccio & Schmidhuber 2020). Similarly, we can define policy-extended action-value function (8,, a, X²). ). We use V(s, X=) for demonstration in this paper.
The key point is, V(s.-) is not defined for specific policy and is able to preserve the values of multiple policies. With function approximation, PeVFA is expected to approximate values among policy space, i.c., {o"(s) (s) Formally, with finite parameter space θ, we consider the optimal parameter 6* with minimum approximation error over all possible states and policies as:
0 -- =ag min F(O, where F(O, II) = max (Vo(T)) VLI BE0
max [Vo(. (T) (-)lo) TEll
TEll
where F(O. I1) is the overall approximation error of Vo which is defined as the maximum Loe norm of value vector distance among 11, as a typical metric commonly adopted in study on value estimation approximation and policy iteration (Kakade & Langford 2002 Munos] 2003 Lagoudakis
Parr 2003 Therefore, the learning process of a PeVFA Vo
8
As commonly recognized, a conventional value function approximator (VFA) can generalize the values to unseen states or state-action pairs after properly training on known states and state-action pairs under a specific policy. Beyond this, PeVFA provides an appealing characteristic that allows values be generalized among policies, evaluating new policies with the knowledge of old ones. In the following, we study the characteristic of value generalization theoretically (Sec. (.I and provide some empirical evidences (Sec. 3.2), from which we finally introduce new form of GPI (Sec. 3.3
3

---

Under review as a conference paper at ICLR 2021
Value Space
Value Space
Policy Improvement
jossume v (mt) m ehc
(s))
Vo-(Ben)
P
la G
La (Tee))
Vig
Vos
Vas
(a,)
Approsimation [1e,
Learned Cb Unicarned
Valee Gerareliatizn
Generalization
As. = (P(,(r) - Vp((+)
(a) Global generalization
(b) Local generalization
Figure 2: Illustrations of value generalization with PeVFA among policies. Each circle represents the value function of policy. (a) Global generalization: values can generalize to unlearned policies (n' E IT)) from already learned policies (œ E Il.). (b) Local generalization: values of previous policies (=m))=0)) can generalize to the successive policy (TT+1) along policy improvement path.
3.1 VALUE GENERALIZATION AMONG POLICIES
Concretely, we introduce scenarios of value generalization: global generalization and local gen- cralization, which are closely related to many RL problems. An illustration is show in Figure 2 In this section, first focus general two-policy case and study whether the learned value ap- proximation of one policy can generalize to that of the other one. We propose a common form of generalization approximation error (Theorem D and analyze the condition of generalization contrac- tion (Corollary [[]. Further, we go to the two generalization scenarios as illustrated in Figure 2 and especially introduce the closer approximation distance along the policy improvement path induced by PeVFA (Corollary
2). from which a more efficient form of GPI can be derived (Sec. [3.3].
Formally, consider two policies and T2 and PeVFA Vo to approximate the values of them. For convenience of demonstration, we consider an identical representation space (i.e.. X' II) and an infinite parameter space è where zero approximation error can be achieved, i.c., F(6" II) = 0. We then define the approximation loss of V for each policy π E II [a(x) [Vo(T) V (T) [[Vo(: T) - V (-, T]] 2 O, which measures the value distance of to the optimal parameter at some policy æ. In the following, we analyze the generalization performance value estimation of unlearned policy Ø2 when PeVFA Va is only trained to approximate the values of policy TT We first introduce the two assumptions below:
Assumption 1 (Contraction). The learning process 3 of value approximation for iı is y- contraction, that is fo(T)) S (m)) where (0,1] and 0 F(r.), Ô.
0 and i are the parameters of PeVFA before and after some learning of π 's values. We use f and f as abbreviations for fe and fa below for the ease of demonstration. Recent works (Keskar et al, 2017 Novak et al. 2018; Wang al. 2018) suggest that generalization performance Is related to local properties of the model. We then assume following smoothness property for following analysis:
Assumption (Smoothness). f is (or has) Lipschitz continuous (gradient/Hessiann) at कı with Lip- schitz constant Lo, e.g., If(=) f(T1)) < Lo -d(1, m1) π E II with some metric space (II, d).
With above two assumptions, next we derive general form of an upper bound of the generalized value approximation for unlearned policy #2 (i.e., ((T2))) as follows:
Theorem 1. Under Assumption [and when f(T) < f(TT), have the following bound:
f(T2) <
Tf(22)
M(
generalised contraction
locality margin
The form of M depends on the smoothness property, e.g., M(T1, n2, 1.) = -o -. TT) when f Lipschitz continuous. See Appendix A. for more instances of M when higher-order smoothness properties (Nesterov & Polyak 2006) are considered.
4

---

Under review as a conference paper at ICLR 2021
The main idea is to chain the Lipschitz continuity upper bound (Assump- tion 2) the contraction upper bound (Assumption D and the inequality of f.
Proof. See Appendix A.I
Remark 1. The case f(T1) S f(T2) considered in Theorem 7 can usualty exist since only æ 's values are learned. Under such circumstances, we suggest that the complementary case f(T1) > (T2) is acceptable since the approximation error of T2 is already lower than the trained one.
Theorem provides a generalization upper bound for (TT2), ). as generalized contraction on (T22 plus locality margin term .M which is related to TI, T2 and the smoothness property L. Further, we analyze the condition when a PeVFA can also obtain a contraction approximation for unlearned policy Ta though only trained on T1, as below:
Corollary 1. Followed by Theorem 4 consider f is Lipschitz continuous and f( * 0, we have,
((22) & (T2) where "/o = y + Le Fo. When f(T2) 2 La
also a 1-7 0 (51))
((x)
7g contracctin for value approximation of T2, with 7g E (0, 1].
Proof. Replace M with Lipschitz continuous upper bound and transform the RHS in Equation 2
then let RHS not greater than (T2). See Appendix A.2 for complete derivation.
Remark From the generalization contraction condition provided in Corollary L we can find that: as i. 7 + 0, or ii d( π2) - 0, or f.g + 0, the contraction condition is easier to achieve (or the contraction gets tighter), i.e., the generalization on unlearned policy T2 is better.
In the above, we discuss value generalization in a two-policy case, i.c. from one Iearned policy to another unlearned policy. Global generalization in Figure 2(a] is extension scenario of the two-policy case where values generalize from a known policy set (i E 1lg) of which the values are learned to unseen policies (r' E ). In this paper, we focus more on local generalization of PeVFA as shown in Figure 2(b) Recall the GPI form shown the left of Figure [T] at each iteration t, value generalization from current policy Ty to improved policy T++1 is exactly the two-policy case we discussed above. However, it is unclear how local generalization can impact the value estimation in GPI (i,e., policy evaluation). To this end, we propose the following Corollary to see connection between local generalization of PeVFA and a closer value approximation distance:
Corollary 2. At iteration t in local generalization scenario of PeVFA (Figure 2(b) if (TLe) + So, (T++1) < || (Tt) - V (T+41))l, I., then So, (T++1) S (V ) - Vo (T++1)
Proof. Proof can be obtained by applying Triangle Inequality. See Appendix
A.3
Corollary 2 indicates that local generalization of PeVFA can induce a more preferable start point (Vo, (T++1)) }) which is closer to the optimal approximation target (Vo. (m++1) than the conventional
one (V). f equivalent to Vo, (mx) definition) for policy evaluation process at iteration t &+1
Remark 3. With closer start points Vo, (T++1), policy evaluation (i.e., minimize fe.( can be more efficient with PeVFA. One assume an ideal case with perfect local generalization, where policy evaluation is no longer necessary and policy improvement can be performed consecutively.
3.2 EMPIRICAL. EVIDENCES FOR TWO KINDS OF GENERALIZATION
Beyond theoretical discussion, we empirically investigate whether two kinds of generalization can be achieved by a PeVFA neural network. First, we demonstrate global generalization in a continuous 2D Point Walker environment. We build the policy sets IT with synthetic policies, each of which is a randomly initialized 2-layer tanh-activated neural policy network with 2 units for each layer. Here the concatenation of all weights and biases of the policy network as representation X for each policy æ for demonstration, called Raw Policy Representation (RPR), while more advanced policy representation methods will be introduced in Sec. 4 We rollout the policies in the environ- ment to collect trajectories T = (r and then obtain the dataset {(x,; Tap >>> We separate the synthetic policy set into training set (i.e., known policies II,) and testing set (i.e., unseen poli- cies I1,). Figure ((a shows that a PeVFA trained with data collected by Hl achieves comparable testing approximation performance when evaluating values of policies in II; (the average MSE on training/testing set is 2.909 and 4.155), as well as almost maintains the order of optimality among policies. More experimental details are provided in Appendix B.1
5

---

Under review as a conference paper at ICLR 2021
The lod tiher brist
(a) Global generalization
(b) Local generalization
Figure 3: Empirical evidences of two kinds of generalization of PeVFA. (a) Global generalization: PeVFA shows comparable value estimation performance on the testing policy set (red) after learning on the training policy set (blue) while maintains reasonable order of optimality. (b) Local general- ization: PeVFA (V (Vo(x=) )) shows lower losses (i.c., closer distance to approximation target) than convention VFA (VT) before and after the training for values of successive policies along policy improvement path, which demonstrates Corollary 2 The left axis is for approximation loss (lower is better) and the right axis is for average return as a reference of the learning process of PPO policy.
We then demonstrate local generalization of PeVFA to examine the existence of Corollary 2 We also use 2-layer 8-unit policy network trained by PPO (Schulman et al.) 2017) algorithm in Ope- nAI MuJoCo continuous control tasks. Parallel to the conventional value network (s(s) (i.e., VFA) in PPO, we set a PeVFA network Vo(s, Xx) with RPR as input. Compared with Vg. the PeVEA network (e(s, Xar) ) is additionally trained with data from historical policies ({T.)=)) along the pol- icy improvement path. Note that (x(xx) does not interfere with PPO training here, and only referred as a comparison with V.- on the approximation error to the true values (i.e., collected re- turns) of successive policy Te+1- To see whether the local generalization exists and can be beneficial or not, in Figure ((b) we illustrate the value approximation losses for V (s) (red curves) and (x) (blue curves), before (solid) and after (dashed) updating with on-policy samples, each iteration during the learning process of PPO (green curves). The results demonstrate the existence and benefits of local generalization of Pe in InvertedPendulum-Vi and Ant-v1. By comparing approximation losses before updating (red and blue solid curves), we can observe that the good generalization of value approximation for successive policies is obtained with PeVFA Ve(s, Xa). The approximation loss of Vo Xx)) is almost consistently lower than that of V (s) before update, providing the evidence of Corollary P For the dashed curves, it indicates that PeVFA can achieve lower approximation loss than conventional VFA after updating with the same on on-policy samples and number of training. We emphasize the importance of the empirical evidence of beneficial lo- cal generalization, based on which we assume that VFA can improve conventional GPI and then propose our approaches in the following. See Appendix B.2 for complete results in MuJoCo tasks and more experimental details.
3.3 REINFORCEMENT LEARNING WITH PEVEA
Since PeVFA extends the capacity of conventional VFA and induces value generalization among policies, we expect to derive improved RL algorithms with PeVFA by utilizing the two kinds of generalization we discussed in previous sections. In this paper, we focus on local generalization and propose a new GPI form with VEA As illustrated in the right of Fi the key difference is that the generalized value approximation Vo, (Xx+4) of successive policy πe+1 is taken as the start point of policy evaluation at each iteration. One can see the difference by comparing Vo, (Xm.) with the conventional one V which is equivalent to Ve, (xx,) in definition. From Corollary 2 and the evidence in Figure (bb) such generalized starting points are closer to approximation target (i.e., true
values v² in some cases, thus we suggest that these local generalization steps (gray arrows) help to reduce approximation error with finite updates and improve the efficiency of the overall learning process. See a pseudo-code of GPI with PeVFA (Algorithm 2 and more discussion in Appendix C
For global generalization, we assume that one can obtain a high-quality evaluation of unseen policies by training a PeVFA only on known policies. This indicates an appealing potential to circumvent extra sample collection and training cost for policy evaluation in many problems, e.g., evolutionary

---

Under review as a conference paper at ICLR 2021
Policy Decoder
MLP
(MLE) á
MLP
Data Source/Input
Policy Encoder
Auxiliary Loss
Policy Params (OPR)
Contrastive Learning
Unsupervised Leorning
P1 Trans
MLP
Xit gradlents
Policy (5,d) Pairs (SPR)
(End to End)
MLP
MLP
v(8.x))
MLP
PeVFA
Figure 4: Framework of policy representation training. Policy network parameters (OPR) or policy state-action pairs (SPR) are fed in policy encoder with cormuutttioninnvarinn (PI) transformations, producing representation Xx- Xx can be trained by gradients from PeVFA value approximation (i.e., End-to-End), as well as with optional auxiliary action recovery loss and unsupervised contrastive learning, shown with separation by the gray dotted line.
strategy (Salimans al. 2017 opponent modeling (He Boyd-Graber, 2016) and policy adapta- tion (Ameekiss et al. 2019:; Raileanu et al. 2020), which are beyond the scope of this paper.
4 POLICY REPRESENTATION LEARNING
In the last section, we analyze value generalization among policies induced by PeVFA and propose a new GPI for more efficient learning process. To derive practical deep RL algorithms, one key problem is policy representation, i.e., low-dimensional embedding of RL policy. This is differ- ent from the notion of policy representation in some works (Zhou et al., 2019b 9b Ma et al.) 2020), which is related to parameterization of policy, usually advanced policy network models or architec- tures, A usual policy network may have large number parameters, thus making it inefficient and even impractical to use Raw Policy Representation (RPR) for optimization and generalization (Harb) al. 2020 To our knowledge, it remains unclear about how obtain an effective and compact representation for an RL policy.
We propose framework of policy representation learning shown in Figure Our intuitions come from answering the question: what makes good policy representation, particularly when the policy is neural network. Our proposed policy representations motivated by geometric perspective of the policy as curved surface of the policy distribution; details are provided in the Appendix D.I Concretely, we provide two policy representations below:
Surface Policy Representation (SPR): The first way is to extract policy representation from the state-action pairs (or trajectories). since they reflect the information about how policy may behave under such states. We view this as scattering sample points on policy curved surface, which should be able to capture the features the number of samples increases.
Origin Policy Representation (OPR): Moreover, we propose a novel way to learn a low- dimensional embedding from the policy network parameters directly. Generally, we con- sider policy network to be an MILP with well represented state features (e.g.. features extracted by CNN for pixels or by LSTM for sequences) as input and deterministic or stochastic policy output. We then define a lossy mapping compress all the weights and biases of the MLP to obtain the corresponding OPR.
For both SPR and OPR, we use permutation irmutation-invariint transformations, i.e., mainly use MLP then Mean-Reduce operation. The above two policy representations proposed are scalable to encode most RL policies from different stream of policy data (as illustrated in Figure +. In this paper, we use OPR and SPR to encode the typical MLP policy. Implementation details and a discussion on encoding sophisticated RL policy like that operates images can been seen in Appendix D.2
The related work PVN (Harb et al. 2020 proposes Network Fingerprint to circumvent the difficulty of learning policy representation from network parameters. However, it introduces additional non-
7

---

Under review as a conference paper at ICLR 2021
trivial optimization for set of probing states. For PVFs (Faccio & Schmidhuber 2020g. all the policy weights are simply parsed as inputs to the value function without embedding. even in the nonlinear case. Besides, a few works also involve representation or embedding learning for RL policy in Multiagent Learning (Grover et al. 2018), Hierarchical RL (Wang et al.] 2020). Policy Adaptation and Transfer (Hausman et al. 2018 Arnekvist et al.) 2019 Raileanu et al. 2020: Harb
et al. 2020). We found almost all of them belongs to the scope of SPR. detailed review for them is provided in Appendix D.5 For OPR, our goal is to learn a representation of the policy itself from its
parameters rather than of the architecture of a policy network as is common in Neural Architecture Search (NAS) (Zhou et al.) 2019b; Ma et al. (020).
The most straightforward way is train policy representation Xx together with PeVFA end-to- end as shown in Figure 4 by backpropagating the value approximation error through the policy representation. Intuitively, a good representation should be able to predict the policy's behavior. To this end, we resort to an auxiliary loss to recover the actions of x from X under different states (Grover et al., 2018 Raileanu et al.) 2020). Concretely, an auxiliary policy decoder (or a master Xx) is trained through behavioral cloning, i.c., to minimize cross-entropy objective ((..a)-T log T (a]ss, In addition, propose to improve the representation learning of Xn with unsupervised Contrastive Learning (Oord et al. 2018 [Srinivas et al.) 2020 Schwarzer et al. 2020j. In this way, policies are encouraged to be close to similar ones (i.e.. positive samples) and to be apart from different ones (i.e., negative samples) in policy representation space. For each policy, we construct positive samples by data augmentation on policy data, depending on SPR or OPR considered; and different policies along the policy improvement path naturally provide negative samples for each other. The policy representation network then trained with InfoNCE loss (Oord (He et al. 2020 Chen et al., 2020). More implementation details
policy) iT [s, (-||s,
et al.) 2018) as commonly done in
are Appendix D
5 EXPERIMENTS
We conduct several experiments with our proposed methods and try to answer the following ques- tions: i. How much does the local generalization property of PeVFA benefit deep RL learning algorithm? ii. Is our policy representation learning framework effective to learn good representation for a policy network? To answer the above questions, we derive a practical deep RL algorithm with PeVFA based on Proximal Policy Optimization (PPO) (Schulman et al., 2017) to demonstrate and evaluate GPI with PeVFA and our approaches of policy representation learning.
Setting. Due to the popularity and simplicity in implementation, we use PPO as the base algorithm for a representative implementation of GPI with PeVFA. PPO is policy optimization algorithm that learns both a policy network π and a value network V that approximates the values of m, typically following GPI in the left of Figure [ The policy æ is optimized with respect to a surrogate objective (Schulman
al. 2013) using advantages calculated by V and GAE (Schulman et al., 2016). We propose to replace the conventional value network (s(s) by a PeVFA network Va(s, X with parameters 0 as given in Figure 4 and perform GPI with PeVFA as in the right of Figure D Without modifying the policy optimization process of vanilla PPO, we then obtain new algorithm. called PPO-PeVFA. Note that what is actually changed here is simply drop-anddrepllacmment of the value function. The policy update scheme is the same for PPO and PPO-PeVFA, which means that the policy of PPO-PeVFA is not updated by gradients through policy representation feeded in Ve(s.X* Xx). For all the experiments in this section, we use a normal-scale policy network with 2 layers and 64 units for each layer, rather than the small policy networks used in Sec. [.2 for demonstration. As a result, the number of policy network parameters can be over 10k for example in Ant-v1, thus prohibiting the rationality of directly using RPR for PeVFA Vo(s, Xx).
Training. The only difference of PPO-PeVFA is the training of policy representation X= and PeVFA network Vo X=). For policy representation X7, it depends on policy data used and training losses adopted as introduced in Sec. See Algorithm 3 in Appendix D.4 for an overall pode of policy representation training. For PeVFA network Ve(s, Xx). it is trained to approximate the values of multiple policies along the policy improvement (i.e. optimization) path, as Algorithm in Appendix
C Both of above training use historical policies, whose data (i.e., network parameters and state-action pairs) are stored. Note that for PPO (and PPO-PeVFA), new policies are only produced after policy optimization process. We do not assume access to pre-collected policies and thus the size of policy set increases from 1 during the learning process. In our experiments, about 1 to 2k
8

---

Under review as a conference paper at ICLR 2021
Table 1: Max Average Return 10 trials of 2M time steps (4M for Ant-v1). First two maximum
values for each task are holded. + corresponds to half std over trials.
Benchmarks
Origin Policy Representation (Ours)
Palicy Representation (Ours)
Environments
PPO
Ran PR
E2E
CL
AUX
E2E
CL
AUX
HallCheetah-v1
2621 + 250 31
2470 - 29165
3171 + 437.63
3725 348.55
3175
517.52
2774 + 233.30
A349 341.42
1236 + 506.39
Hopperavi
1639
1226 348.10
2085
2351 231.31
2214
360.78
2227
297.35
23922
2577 217.73
Walkerld.vl
320.55
1269 209.01
305.51
2044
316.32
1930.57
456,02
381.95
1980 325.54
Ant-v1
2838 + 152.04
2742 71.11
3581
185.43
4019 - 162.47
3784
268.99
3173 + 184.75
3632 + 134.27
3397 + 200.03
InnDouPend-vi
9344 11.02
9355 + 0.40
9357 0.2.9
9355 + 0.64
9355 0.68
9355 = 0.89
9356 + 0.9%
9355 = 1.42
Lndert.ander-22
2.83
3,37
234
3.13
5.700
policies are produced in an experimental trial. We use all historical policies in our training and for the case with many more policies, other sophisticated training methods can be considered.
Results. We conduct the following experiments in OpenAl Gym MuJoCo continuous control tasks (Brockman et al. 2016 Todorov et al., 2012). Beside the comparison to PPO, we consider an- other benchmark, PPO-PeVFA with fixed randomly generated policy representation for each policy, namely Ran PR. The overall experimental results are in Table I More experimental details are in Appendix F. and complete learning curves are provided in Appendix F2
due space limitation.
Question 1. Can PPO-PeVFA with only End-to-End (E2E) policy representation outperform PPO?
From Table 1 we can find that both PPO-PeVFA with OPR and with SPR trained in E2E fashion outperforms PPO in all 6 tasks, especially in Ant-v1. This indicates that effective policy representa- tion can be learned from value approximation loss in our experiments, based on which PeVFA can benefit from local generalization thus improve the learning process. Additionally, Ran PR shows an overall degeneration compared to PPO, demonstrating the significance of OPR and SPR.
Can representation learned through contrastive learning (CL) and auxiliary loss (AUX) further benefu performance of PPO-PeVFA?
Question 2.
For both OPR and SPR, we observe consistent improvements in- duced by CL and AUX over E2F, which means that additional learning to maintain unsupervised consistency (CL.) and capture policy behavior (AUX) further benefits policy representation, and eventually results in substantial improvement over PPO in all tasks except for the first two simple tasks. In an overall view, OPR slightly outperforms SPR and as CL does over AUX. We hypothesize that it is due to the stochasticity of state-action pairs which serve as inputs of SPR and training samples for AUX. We suggest that this remains space for future improvement.
20M
155
Tra-5
Question 3. How is the learned representation like when view- ing in low-dimensional space?
Figure 5: I-SNE visualization for OPR (E2E) of 6k policies from trials (denoted by differ- ent markers) in Ant-vl. Policies are colored by average return.
We show a 2D t-SNE (Maaten & Hinton 2008 view in Figure 5 and more visualisations in Appendix F Policies from differ- ent trials are locally continuous (multimodality) while a global evolvement emerges with respect to policy performance.
CONCLUSION AND FUTURE WORK
In this paper, we propose Policy-extended Value Function Approximator (PeVFA) which induces value generalization among policies. We propose new form of GPI with PeVFA which can po- tentially benefit from local value generalization along the policy improvement path. Moreover, we propose several approaches to learn an effective low-dimensional representation of an RL policy. Our experiments demonstrate the effectiveness of the generalization properties of PeVFA and our proposed methods of representing the RL policy.
Our work opens up new ways to potentially improve the policy learning process from global or local perspectives using PeVFAs that receive an explicit policy representation to improve generalization of the value approximation. We believe our work can guide future research directions policy

---

Under review as a conference paper at ICLR 2021
representations and PeVFA in many RL problems. We plan to further study policy representation learning, as well as more complex value generalization scenarios, including TD learning with non- stationary approximation targets. In addition, optimizing policies in the representation space can also be an interesting direction.
REFERENCES
M. Andrychowicz, D. Crow, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba. Hindsight experience replay. In NeurlPS, pp. 5048-5058, 2017.
M. Andrychowicz, A. Raichuk, P. Stanczyk, M. Orsini, Girgin, R. Marinier, L. Hussenot, M. Geist, O. Pietquin, M. Michalski, S. Gelly, and O. Bachem. What matters in on-policy inforcement learning? A large-scale empirical study. CoRR, 0bss22006059900 2020.
I. Amekvist, D. Kragic, and A. Stork. VPE: variational policy embedding for transfer reinforce- ment learning. In ICRA, pp. 36-42, 2019.
M. G. Bellemare, W. Dabney, and R. Munos. distributional perspective on reinforcement learning. In ICML, volume 70, pp. 449 458, 2017.
G. Brockman, V. Cheung. L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. CoRR, abs/
2016.
Chen, Kornblith, M. Norouzi, and Hinton. A simple framework for contrastive learning of visual representations. CoRR, abs/2002.05709, 2020.
L. Engstrom, A. Ilyas, S. Santurkar, D. Tsipras, F. Janoos, L. Rudolph, and A. Madry. Implementa- tion matters in deep RL: A case study PPO and TRPO. In ICLR, 2020.
F. Faccio and J. Schmidhuber. Parameter-basee based value functions. CoRR, abs/2006.092266 2020.
H. Fu, H. Tang, J. Hao, C. Chen, X. Feng, D. Li, and W. Liu. Towards effective context for meta- reinforcement learning: an approach based on contrastive learning. CoRR, 138/2009.13891, 2020.
Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In ICML, 2018.
A. Grover, M. AL-Shedivat, J. K. Gupta, Y. Burda, and H. Edwards. Learning policy representations in multiagent systems. In ICML, volume 80, pp. 1797-1806, 2018.
D. Hafner, T. P. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. In ICLR, 2020.
Harb, T. Schaul, D. Precup, and Bacon. Policy evaluation networks. CoRR, abs/2002.11833, 2020.
K. Hausman, J. Tobias Springenberg, Z. Wang, N. Heess, and M. A. Riedmiller. Learning an em- bedding space for transferable robot skills. In ICLR, 2018.
H. He and J. L.. Boyd-Graber. Opponent modeling in deep reinforcement learning. In ICML, vol- ume 48, pp. 1804-1813, 2016.
K. He. H. Fan, Y. Wu, S. Xie, and Girshick. Momentum contrast for unsupervised visual 2020.
representation learning. In CVPR, pp.
M. Igl, G. Farquhar, J. Luketina, W. Boehmer, and S. Whiteson. The impact of non-stationarity on generalisation in deep reinforcement learning. CoRR, abs/2006.05826, 2020.
S. M. Kakade and J. Langford. Approximately optimal approximate reinforcement learning. In ICML, pp. 267-274, 2002.
N. S. Keskar, D. Mudigere, Nocedal, M. Smelyanskiy, and P. Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In ICLR, 2017.
I. Kostrikov, D. Yarats, and R. Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. CoRR, abs/2004. 13649, 2020.
10

---

Under review as a conference paper at ICLR 2021
M. G. Lagoudakis and R. Parr. Least-squares policy iteration. J. Mach. Learn. Res., 4:1107-1149, 2003.
M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas. Reinforcement learning with augmented data. CoRR, abs/2004. 14990, 2020.
K. Lee, Seo, S. Lee, Lee, and Shin. Context-aware dynamics model for generalization in model-based reinforcement learning. CoRR, abs/2000.06800 2020.
A. Levy, G. D. Konidaris, R. P. Jr., and K. Saenko. Learning multi-level hierarchies with hindsight. In ICLR. OpenReview.net, 2019.
T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Tassa, D. Silver, and D. Wierstra. Contin- uous control with deep reinforcement learning. In ICLR, 2015.
H. Liu, K. Simonyan, and Y. Yang. DARTS: differentiable architecture search. In ICLR, 2019.
R. Luo, F. Tian, T. Qin, Chen, and T. Liu. Neural architecture optimization. In NeurlPS, pp- 7827-7838, 2018.
X. Ma, K. Gupta, and M. Kochenderfer. Normalizing flow model for policy representation in continuous action multi-agent systems. In AAMAS, pp. 1916-1918, 2020.
L. V. D. Maaten and G. E. Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9:2579-2605, 2008
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. Bellemare, A. Graves, M. A. Riedmiller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, Wierstra, Legg, and Hassabis. Human-level control through deep reinforce- ment learning. Nature, 518(7540):529-33 533, 2015.
V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. P. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In JCML, 2016.
R. Munos. Error bounds for approximate policy iteration. In ICML, pp. 560-567, 2003.
O. Nachum, S. Gu, H. Lee, and S. Levine. Data-efficient hierarchical reinforcement learning, CoRR, 2018
O. Nachum, S. Gu, Lee, and S. Levine. Near-optimal representation learning for hierarchical reinforcement learning. In ICLR. OpenReview.net, 2019.
Y. E. Nesterov and B. T. Polyak. Cubic regularization of newton method and its global performance. Math. Program., (08(1):177-200 2006.
R. Novak, Y. Bahri, D. A. Abolafia, J. Pennington, and J. Sohl-Dickstein. Sensitivity and general- ization neural networks: empirical study. In ICLR, 2018.
A. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. CoRR, abs/1 80780077445 03748, 2018.
R. Raileanu, M. Goldstein, A. Szlam, and R. Fergus. Fast adaptation via policy-dynamin value functions. CoRR, hbb/2200.002879, 2020.
K. Rakelly, A. Zhou, C. Finn, S. Levine, and D. Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In ICML, volume 97, pp. 5331-53444 2019.
Salimans, J. Ho, X. Chen, and Sutskever. Evolution strategies as scalable alternative to reinforcement learning. CoRR, abs/1703.0330 703.03864, 2017.
T. Schaul, D. Horgan, K. Gregor, and D. Silver. Universal value function approximators. In ICML, volume 37, pp. 1312-1320, 2015.
S Schreck, C. W Coley, and K. JM Bishop. Learning retrosynthetic planning through simulated experience. ACS central science, 5(6):970-981, 2019.

---

Under review as a conference paper at ICLR 2021
J. Schulman, S. Levine, P. Abbeel, M. I. Jordan, and P. Moritz, Trust region policy optimization. In ICML, pp. 1889-1897, 2015.
J. Schulman, P. Moritz, S. Levine, M. I. Jordan, and P. Abbeel. High-dimensional continuuus control using generalized advantage estimation. In ICLR, 2016.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization
algorithms. CoRR, 00//17077063447 2017.
M. Schwarzer, A. Anand, R. Goel, R. D. Hjelm, A. C. Courville, and P. Bachman. Data-efficient reinforcement learning with momentum predictive representations. CoRR, abs/2007.05929, 2020.
D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. A. Riedmiller. Deterministic policy gradient algorithms. In ICML, pp. 387-395, 2014.
D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. den Driessche, J. Schrittwieser, Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbren- ner, I. Sutskever, T. P. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis. Mas- tering the game of go with deep neural networks and tree search. Nature, 529(7587):484-489 2016.
A. Srinivas, M. Laskin, and P. Abbeel. CURL: contrastive unsupervised representations for rein- forcement learning. CoRR, abs/2004.04136, 2020.
R. S. Sutton and A. G. Barto. Reinforcement learning an introduction. Adaptive computation and machine learning. MIT Press, 1998.
R. S. Sutton, J. Modayil, M. Delp, T. Degris, P. M. Pilarski, A. White, and D. Precup. Horde: a scal- able real-time architecture for learning knowledge from unsupervised sensorimotor interaction.
In AAMAS, pp. 761-768, 2011.
A. Tacchetti, H. F. Song, P. A. M. Mediano, V. Flores Zambaldi, J. Kramár, N. C. Rabinowitz, Th. Graepel, M. Botvinick, and P. W. Battaglia. Relational forward models for multi-agent learning. In ICLR, 2019.
E. Todorov, T. Frez, and Y. Tassa Mujoco: A physics engine for model-based control. 2012 JEEE/RSJ International Conference Intelligent Robots and Systems, pp. 5026-5033, 2012.
Y. H. Tsai, Y. Wu, R. Salakhutdinov, and L. Morency. Demystifying self-supervised learning: An cnammation-theoretiin framework. CoRR, abs/2006.05576.
T. Unterthiner, D. Keysers, S. Gelly, O. Bousquet, and I. O. Tolstikhin. Predicting neural network accuracy from weights. CoRR, abs/2002.1 1448, 2020.
H. v. Hasselt. Double q-learning. J. D. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. S. Zemel, and A. Culotta (eds.), NewrlPS, pp. 2613 2621, 2010.
O. Vinyals, Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Pow- ell, T. Ewalds, P. Georgiev, J. Oh, D. Horgan, M. Kroiss, I. Danihelka, A. Huang, L. Sifre, T. Cai, J. P. Agapiou, M. Jaderberg, A. S. Vezhnevets, R. Leblond, T. Pohlen, V. Dalibard, D. Bud- den, Y. Sulsky. J. Molloy, T. L. Paine, C. Gulcehre, Z. Wang. T. Pfaff. Y. Wu, R. Ring, D. Yo- gatama, D. Wünsch, K. McKinney, O. Smith, T. Schaul, T. Lillicrap, K. Kavukcuoglu, D. Has- sabis, C. Apps, and D. Silver. Grandmaster level in starcraft ii using multi-agent reinforcement 2019.
learning. Nature, 575(777)35-777
H. Wang, N. S. Keskar, C. Xiong, and R. Socher. Identifying generalization properties in neural networks. CoRR, abs//809.00. 07400, 2018
R. Wang, R. Yu, B. An, and Z. Rabinovich. 1'rrl: Interactive influence-bart based hierarchical reinforce- ment learning. In IJCAI, pp. 3131-3138, 2020.
J. You, B. Liu, Z. Ying, V. S. Pande, and J. Leskovec. Graph convolutional policy network for dire-dieected molecular graph generation. In NeurlPS 2018, pp. 6412-6422, 2018.
W. Zhou, L. Pinto, and A. Gupta. Environment probing interaction policies. In ICLR, 2019a.
W. Zhou, Y. Yu, Y. Chen, K. Guan, T. Lv, C. Fan, and Z. Zhou. Reinforcement learning experience reuse with policy residual representation. In IJCAI, pp. 4447-4453, 2019h.
12

---

Under review as a conference paper at ICLR 2021
A PROOFS
A.1 PROOF OF THEOREM 1
Proof.
Start from Assumption
derive the upper bound of fTT2):
f((2)) - f(mi) + M(TI L),
(3)
where M(TI, n2, B) can have different forms that depends on the specific smoothness property, for examples:
Lo x2)
1 با d(TTI n²)² (f(()), mT) +
M(T1,T222)
2
(2 mi) + 2 (m) - m1). -1) Le
d(T1, T2)
6
1 2. c correspond to Lipschitz Continuous, Lipschitz Gradients and Lipschitz Hessian Nesterov & Polyak 2006).
Combined with Assumption and consider the case f(x1) S f(T2). Equation transformed as follows:
can be further
f(T2) S f(x1) M(TI; n2, 1.)
S <f((m) + MMTI, T2, L)
(T(22)
M(TI, TT, L).
generalized contraction
locality margin
which yields the generalization upper bound in Theorem [ tion 4 as generalized contraction since is from the contraction on f(T)), and the second term as locality margin since it is determined by specific local property.
We note the first of RHS of Equa-
A.2 PROOF OF COROLLARY
Proof. Consider the concrete form of locality margin M(T T2, f.) in the case of Lipschitz contin- uous, i...
f(T²) S 7 + Lo d( and consider f(T2) F (note that f(T2) 2 0):
(5)
f(TT2 S Ng (M2). where Ng + Lo d(
(6)
f(x2)
Then we get the contraction condition of value generalization ma in Corollary [ by letting the RHS of Equation 5 not greater than f(T2), , i.e., Yy I 1 (note that 7 E (0, 1)):
(1(22) + Lo d(TIi T2) S ((T22
(1 - (f=))((T2) 2 Lo d(T1, R2)
f(T2) > im T2)
Intuitions observed from Equation 7 are discussed in Remark In another word, the tighter the contraction on learned policy क is, the closer the two policies are, the smoother the approximation loss function f is, the generalization on unlearned policy T2 is better.
Corollary provides the generalization contraction condition on f We also provide another view from the perspective of (TT1) by additionally considering the smoothness property of f-
Similar as in Assumption 2 we assume that Lipschitz continuous क, with Lipschitz constant
Lo. e.g.. [f(T) [(m)] Lo d(TT,, m)) with some metric space (II, d). Higher-order smoothness property also be considered here. We then obtain lower bound of (TT2 below:
[(m) Lo d(x,, m() {(()))
(8)
13

---

Under review as a conference paper at ICLR 2021
We then chain the second line of RHS in Equation 4 (M in the case of Lipschitz continuous) and the LHS Equation
(T22 (T())) + in d(T,, mi S (m) Lo d(T TI) ((T2),
re-arrange and yield the generalization contraction condition f(TT)):
(f(m)) + Lo d(x1, - f(m1) La
(1-)) (Lo Ey ( T2) f(ā1) 2
(10)
(Lo + Lo Lo)d(TT nT)
1 1-y
A.3 PROOF OF COROLLARY
Proof. Due to Triangle Inequality, we have:
S0. (Tx) + VV, (mx) - Vo- (1++)) 2 (T - (T++1)))
(1I)
i.e., VVo (Tx) Vo- (Tx) | (T, Vo- (Te+ 2 Vo (Te) Vo- (T++))
Combined with condition
( + fa, (T++1) 1 \/ (mT) Vo. (m++1ll
(12)
i... Ve. (T.) ( + Vo, (T++1) - Vo- (T(+1))) V (TT) V (T++))
Chain above two inequality,
(me) (mee1) (Te)) Vo- (Te+1)
fo, ( + | (TTe V (Te+1)) (TL)) (Te+1)l)
(13)
generalizated VAD with PeVEA
conventional VAD
then we have that with local generalization of values for successive policy T++1, the value approxi- mation distance (VAD) can be closer in contrast to the conventional (RHS of Equation [3).
In practice, it is also possible for farther distance to exist, e.g., the condition in above Corollary is not satisfied. Moreover, under nonlinear function approximation, it is not necessary that closer approximation distance (induced by Corollary 2 ensures easier approximation or optimization pro- cess. This is related to many things, e.g., the underlying function space, the optimization landscape, the learning algorithm used and etc. this paper, we provide a condition for potentially beneficial
local generalization and we resort to empirical examination shown in Sec. 3.2 Further study of
beneficial cases especially under nonlinear function approximation is planned for future work.
B DETAILS OF EMPIRICAL EVIDENCE OF Two KINDS OF GENERALIZATION
B.1 GLOBAL GENERALIZATION IN 2D POINT WALKER
Global generalization denotes the generalization scenario that values can generalize to unlearned policies (л' II,) from already learned policies (I Πlg). We conduct the following experiments to demonstrate global generalization in a 2D continuous Point Walker environment with synthetic simple policies.
Environment. consider a point walker on a continuous plane with
state: (z, y, sin (0), cos( cos(2), cos())), where 0 is the angle of the polar coordinates.
action: 2D displacement, a 187-1,111
deterministic transition function that describes the locomotion of the point walker, de-
pending on the current position and displacement issued by agent, i.e., (x', o/) (a, y) ta,
reward function: 7t
with utility ot = = y² illustrated in Figure 6(a)
10
14

---

Under review as a conference paper at ICLR 2021
Hest Map Udiliy Function
(a) Utility function heat map
(b) Examples of synthetic policy population
Figure 6: 2D Pointer Walker. (a) The heat map of the utility function of the 2D plane. The darker regions have higher utilities. (b) Demonstrative illustrations of trajectories generated by 30 synthetic policies, showing diverse behaviors and patterns. Each subplot illustrates the trajectories generated in 50 episodes by a randomly synthetic policy, with different colors as separation. For each trajectory (the same color in one subplot), transparency represents the dynamics along timesteps, i.e., fully transparent and non non-tronnee con-transparent denotes the positions at first and last timesteps.
Synthetic Policy. We build the policy sets IT Tl U IT and TT ITT, 0 with synthetic policies. Each synthetic policy is 2-layer tanh-activated neural policy network with random initialization. Each policy is deterministic, taking an environmental state as input and outputting a displacement in the plane. We find that the synthetic population generated by such simple way can show diverse behaviors. ( shows the motion patterns of example of such a synthetic population. Note that the synthetic policies are not trained in this experiment.
We rollout each policy in environment to collect trajectories T {Tr ] For such small synthetic policies, it is convenient to obtain policy representation. Here we use the concatenation of all weights and biases of the policy network (22 in total) representation Xx for each policy æ, called raw policy representation (RPR). Therefore, combined with the trajectories collected, obtain the policy dataset, i.c.. {(x,, Tay )) In total, 20k policies synthesized in our experiments and we collected 50 trajectories with horizon 10 for each policy.
Policy Dataset.
We separate the synthetic policies into training set (i.e., un- known policies Il) and testing set (i.e., unseen policies II1) in proportion of 8 2. We set PeVTA network Vo X²) to approximate the values of training policies (i.e., π E Io)) and then conduct evaluation use Monte Carlo return (Sutton & Barto 1998) of collected trajectories approximation target (true value of policies) in this experiment. The network architecture of Vo Xx) is il- lustrated in Figure 7 The learning rate is 0.005, batch size is 256. K-fold validation is performed through shuffling training and testing sets.
Xx
FC (64) ReLU
FC (64) - ReLU
testing policies (i.e., E II.). We
FC (128) ReLU
W (m.
Figure shows that PeVFA trained with data collected by training set Hh achieves reasonable value prediction of unseen testing policies II, as well as maintains the order of opti- mality among testing policies. This indicates that value gener- alization exist among policy space with a properly trained Pe FF. RPR can also be one alternative of policy representa- tion when policy network is of small scale.
Figure 7:
An illustration of ar- chitecture of PeVTA network. FC is abbreviation for Fully-connected layer.
15

---

Under review as a conference paper at ICLR 2021
bos otfer 'VITA bas Iafen)
2110
2007
Poant
L100
(a) InvertedPendulum-vl
(b)
(c)
Aatrabe
19t ldlent
VTA Iran Judtoo NesTAiisettatorn
FasTa roface
(d) HalfCheetah-vl
(e) Hopper-v1
(f) Walker2d-vl
EU orher PASTA 34000
300
(g) Ant-vl
Figure 8: Complete empirical evidence of local generalization of PeVFA across 7 MuJoCo tasks. The learning rate of policy and value function approximators are 0.0001 and 0.001 respectively. Each plot has two vertical axes, the left one for approximation error (red and blue curves) and the right one for average return (green curves). Red and blue denotes the approximation error of conventional VFA (VS (s)) and of PeVFA (Vo(s, X=)) respectively; solid and dashed curves denote the approximation error before and after the training for values of successive policy (i.e., policy evaluation) with conventional VFA and PeVFA, averaged over 6 trials. The shaded region denotes half a standard deviation of average evaluation. PeVFA consistently shows lower losses (i.e., closer to approximation target) across all tasks than convention VFA before and after policy evaluation along policy improvement path, which demonstrates Corollary
B.2 LOCAL GENERALIZATION MUJoCo CONTINUOUS CONTROL TASKS
We demonstrate local generalization of PeVFA, especially to examine the existence of Corollary 2 i.e. PeVFA can induce closer approximation distance (i.e., lower approximation error) than conven- tional VFA along the policy improvement path.
We use 2-layer 8-umit policy network trained by PPO (Schulman et al. 2017) algorithm in Ope- nAI MuJoCo continuous control tasks. As in previous section, using very small policy network is for the convenience of training and acquisition of policy representation in this demonstrative experiment. We use all weights and biases of the small policy network (also called raw policy rep- resentation, RPR), whose number is about 10 to 100 in our experiments, depending on the specific environment (i.e., the state and action dimensions). We train the small policy network commonly done with PPO (Schulman et al., (017) and GAE (Schulman et al. 2016). The convention value network V (s) (VFA), is 2-layer 128-um uniu ReLU-activated MLP with state input and value as output. Parallel to the conventional VFA in PPO, we VeVFA network (o Xx) with RPR as
16

---

Under review as a conference paper at ICLR 2021
ba lifer 'VIAA bu Iafan
Aog ROCEY
XTA oeo lafler FuSTA IN lafer
(a) InyPend-vl (Ir= le"
(b) InvPend-vl (Ir= 1e -3)
(c) InvPend-vl (Ir = 5e
Adaann
swrliinng trigh Lifeer
Jog hear
(d) Ant-vl (Ir= 1e
(c) Ant-vl (Ir=le
(f) Ant-vl (Ir= 5e
Figure 9: Empirical evidence of local generalization of PeVFA on InvertedPendulum-V1 and Ant- vI with different learning rates of policy, i.e., {0.0001, 0.001, 0.005). Results are averaged over trials.
additional input. The structure of PeVFA differs at the first hidden layer which has two input streams and each of them has 64 units, as illustrated in Figure 7 so that making VFA and PeVFA have simi- lar scales of parameter number. In contrast to convention VFA V which approximates the value of current policy (e.g.. Algorithm D. PeVFA Vo(s, X²) has the capacity to preserve values of multiple policies and thus is additionally trained to approximate the values of all historical policies ((T; =o)) along the policy improvement path (e.g., Algorithm 2 The learning rate of policy 0.0001 and the
learning rate of value function approximators (V7 (s) and Vo xx))
)) is 0.001. The training scheme
of PPO policy here is the same as that described in Appendix F. and Table 2
comparison with V on the approximation eтror to the true values of successive policy T We use the MC returns of on-policy data (i.e., trajectories) collected by current successive policy as unbiased esti- mates of true values, similarly done in (v. Hasseli 2010 Fujimoto et al.) 2018). Then we calculate the approximation error for VFA V and PeVFA (o(x=) to the approximation target before and af- ter value network training of current iteration. Finally, we compare the approximation error between VFA and Ve to approximately examine local generalization and closer approximation target in Corollary 2 Complete results of local generalization across all 7 MuJoCo tasks are show in Figure g The results show that VeV consistently shows lower losses (i.e., closer to approximation target) across all tasks than convention VFA before and after policy evaluation along policy improvement path, which demonstrates Corollary2 2 Moreover, we also provide similar empirical evidence when
Note that Vo(X=) does not interfere with PPO training here, and is only referred
policy is updated with larger learning rates, as in Figure
A common observation across almost all results in Figure g and in Figure B is that the larger the extent of policy change (see the regions with a sheer slope on green curves), the higher the losses of conventional VFA tend to be (see the peaks of red curves), where the generalization tends be bet- ter and more significant (see the blue curves). Since InvertedPendulumm-V is a simple task while the complexity of the solution for Ant-v1 1 is higher, the difference between value approximation losses of VeA and VFA is more significant at the regions with fast policy improvement. Besides, the Raw Policy Representation (RPR) we used here does not necessarily induce smooth and efficient policy representation space, among which policy values are easy to generalize and optimize. Thus, RPR may be sufficient for good generalization in InvertedPenduluu-v/ but may be not in Ant-v1. Overall, we think that the quantity of value approximation loss is related to several factors of the environment such the reward scale, the extent of policy change, the complexity of underlying so-
17

---

Under review as a conference paper at ICLR 2021
lution (e.g., value function space) and some others. A further investigation this can be interesting.
C GENERALIZED POLICY ITERATION WITH PEVFA
C.I COMPARISON BETWEEN CONVENTIONAL GPI AND GPI WITH PEVFA
graphical comparison of conventional GPI and GPI with PeVFA shown in Figure] Here we provide another comparison with pseudo-codes.
From the lens of Generalized Policy Iteration (Sutton & Barto 1998), for most model-free policy- based RL algorithms, the approximation of value function and the update of policy through pol- icy gradient theorem are usually conducted iteratively. Representative examples are REINFORCE (Sutton & Barto 1998), Advantage Actor-Critic (Mnih et al. 2016). Deterministic Policy Gradi- ent (DPG) (Silver et al.) 2014) and Proximal Policy Optimization (PPO) (Schulman et al. 2017). With conventional value function (approximator), policy evaluation is usually performed in an on- policy off- felpplliy fashion. We provide general GPI description of model-free policy-based RL. algorithm with conventional value functions in Algorithm
Algorithm Generalized policy iteration for model-free policy-based RL algorithm with conven- tional value functions (V* (s) ) or Q" (x, a))
Initialize policy тo and V (s) or (s,
2: Initialize experience buffer 2
3: for iteration 0, 1, 2, ... do
4: Rollout policy in the environment and obtain trajectories T, = (Ti)
5: Add experiences Ti in buffer ID
6: if on-policy update then
7:
Prepare training samples from rollout trajectories Te
8:
else off-policy update then
9:
Prepare training samples by sampling from buffer DD
10:
end
11:
Calculate approximation target {yr from training samples (e.g., with MC or TD)
12:
Generalized Policy Evaluation
13:
Update (s) Q7,(88, a) with ((8;,::)); or {(s,, oi w:)). ie., V f- V or
QT
QT-1
14:
# Generalized Policy Improvement
Update policy ni with regard (s) or (s, a) through policy gradient theorem, i.e.,
Te+11 4
end for
Note that we use subscript t correspond to the evaluated policy ny during policy evaluation process in current iteration.
(Line 13 in Algorithm I let the updated value functions to
As comparison, parameterization of value function, PeVFA can perform additionally training on historical policy experiences each iteration (Line 7-8). This is naturally compatible with PeVFA since it develops the capacity of conventional value function to preserve the values of multiple policies. Such a training to improve the value generalization of PeVFA among policy set policy space. Note that for value approximation of current policy n, (Line 10- (0)1 the start points are generalized values of from historical approximation, i.c., V,-1 Xa,) and Q;-1(s, Xm.). In another word, this the place where local generalization steps (illustrated in Figure 2(b) conventional start points (V,z, (s) and QT,((s,)) a), Line 13 in Algorithm 1 and see the difference.
form of GPI with PeVFA is shown in Algorithm 2 Except for the different
) are. One may compare with
e.g., (8) < V (8) + V Xm, ,) is different with V/-1 (8, Xx,), where < is used denote equivalence in definition. As discussed in Sec. 3.2 and 3.3 suggest that such
local generalization steps help to reduce approximation error and thus improve efficiency during the learning process.
18

---

Under review as a conference paper at ICLR 2021
Algorithm 2 Generalized policy iteration of model-free policy-based RL algorithm with PeVFAs (V(s, Xx) or Q(s, a, Xx))
1: Initialize policy To and PeVIA V-1 or (-1(3,
2: Initialize experience buffer D
3: for iteration = 0, 1, 2, do
4: Rollout policy मe in the environment and obtain trajectories = [Ti)
5: Get the policy representation Xx, for policy (from policy network parameters or policy rollout experiences)
6: Add experiences (X Tr) in buffer D
7:
# Value approximation training for historical policies {T:}
8:
Update PeVFA Vs-1(s,Xx.) or Qs-1 (x.) with all historical policy experiences
{(x=1) Ti)
9:
Conventional value approximation training for current policy T if con-policy update then
10:
11:
Update PeVFA Ve-1(8, or (8, for te with on-policy experiences
(XT.: TT)
12:
else off-policy update then
13:
Update PeVFA Vi-1(s, Xm.) ) Q4-1 a,ss,,,nnn Xmx) for ai with off-policy experiences Xa. and [Ti)i-o from experience buffer D end if
14:
15:
Ve Vr-1 or Qr - Qc-1
16:
Update policy na with regard to Ve(s, or (8, (8, a, Xo.) through policy gradient theorem, Tr.
i.e., Te+1
17:
end for
C.2 MORE DISCUSSIONS ON GPI WITH PEVEA
Off-Policy Learning. Off-policy Value Estimation (Sutton & Barto 1998) denotes to evaluate the values of some target policy from data collected by some behave policy. As commonly seen in RL (also shown in Line 6-10 in Algorithm| []), different algorithmm adopt on-policy off-policy methods. For GPI with PeVFA, especially for the value estimation of historical policies (Line 8 in Algorithm 2). on-policy and off-policy methods also be considered here. One interesting thing is, in off-policy case, one can use experiences from any policy for the learning of another one, which can be appealing since the high data efficiency of value estimation of each policy can strengthen value generalization among themselves with PeVFA, which further improve the value estimation process.
Convergence of GPI with PeVFA. Convergence of GPI usually discussed in some ideal cases, e.g., with small and finite state action spaces and with sufficient function approximation ability. In this paper, focus the comparison between conventional VTA and PeVTA in value estimation, i.e. Policy Evaluation, and we make no assumption on the Policy Improvement part. We conjecture that with the same policy improvement algorithm and sufficient function approximation ability, GPI with conventional VFA and GPI with PeVFA finally converge to the same policy. Moreover, based on Corollary 2 and
empirical evidence in Sec. 3.2 GPI with PeVFA be more efficient in some cases: with local generalization, it could take less experiences (training) for PeVFA to reach the same level of approximation error than conventional VFA, or with the same amount of experience (training). PeVFA could achieve lower approximation error than conventional VFA. We believe that deeper dive in convergence analysis is worth further investigation.
PeVFA with TD Value Estimation. this paper, we propose PPO-PeVFA as representative instance of re-implementing DRL algorithms with PeVFA. Our theoretical results and algorithm 2 proposed under the general policy iteration (GPI) paradigm are suitable for TD value estimation as well in principle. One potential thing that deserves further investigation is that, it can be a more complex generalization problem since the approximation target of TD learning is moving (in contrast to the stationary target when unbiased Monte Carlo estimates are used). The non-stationarity induced by TD is recognized to hamper the generalization performance in RL as pointed out in recent work (igl et al., 2020). Further study on PeVFA with TD learning (e.g.. TD3 and SAC) is planned in the future mentioned in Sec.
19

---

Under review as a conference paper at ICLR 2021
D POLICY REPRESENTATION LEARNING DETAILS
D.1 POLICY GEOMETRY
A policy x E 11 P(A)S, defines the behavior (action distribution) of the agent under each state. For more intuitive view, consider the geometrical shape of policy; all state E S and all action a E AA are arranged along the r-axis and y-axis of a 2-dimensional plane, and the probability (density) (a]s) is the value of 2-axis over the 2-dimensional plane. Note that for finite state space and finite action space (discrete action space), the policy can viewed as (S) |A table with each entry in it is the probability of the corresponding state-action case. Without loss of generality, we consider the continuous state and action space and the policy geometry here. Illustrations of policy geometry examples are shown in Figure 10
Figure Tora shows the policy geometry in general case, where the policy can be defined arbitrarily. Generally, the policy geometry can be any possible geometrical shape (s.t. Vs E S. EeeA π (a|s) 1). This means that the policy geometry case. Specially, one can imagine that the geometry of a deterministic policy consists of peak points (z 1) for each state and other flat regions (z 0). Figure [Txb] shows an example of synthetic continuous policy which can be viewed as a 3D curved surface. In Deep RL, a policy may usually be modeled as deep neural network. Assume that the neural policy is a function that is almost continuous and differentiable everywhere, the geometry of such a neural policy can also be contin- uous and differentiable almost everywhere. As shown in Figure [T(c) we provide demo of neural policy by smoothing an arbitrary policy along both state and action axes.
general necessarily continuous or differentiable
cows
(a) Arbitrary policy
(b) Continuous policy
(c) Demo of neural policy
Figure 10: Examples of policy geometry. (a) An arbitrary policy, where p(s, a) is sampled from N (0, 1) for a joint space of 40 states and 40 actions and then normalized along action axis. States are squeezed into the range of [-1, 1] for clarity.
(b) A synthetic continuous policy with p(s, a) (1 - α + 5²) exp(-s² - a²) ) for joint space of E [- and a E [-2, (each of which are discretized into 40 ones) and then normalized along action axis. (c) A general demo of neural network policy, generated from an arbitrary policy (as in (a)) over a joint space of 200 states and
100 actions with some smoothing skill. States are squeezed into the range of [-1, 1] for clarity and the probability masses of actions under each state are normalized to sum into
D.2 IMPLEMENTATION DETAILS OF SURFACE POLICY REPRESENTATION (SPR) AND ORIGIN POLICY REPRESENTATION (OPR)
Here provide detailed description of how to encode different policy data for Surface Policy Representation (SPR) and Origin Policy Representation (OPR) we introduced Sec.
Encoding of State-action Pairs for SPR. Given set of state-action pairs {s,, ai) (with size [n. (_.i_ + a_dim)) generated by policy π (i.e., as ~ T T(-(s))), we concatenate each state-action pair and obtain an embedding of it by feeding it into an MLP, resulting in stack of state-action embedding with size in, e_dim]. After this, we perform a mean-reduce operator on the stack and obtain an SPR with size [1, e_dim]. A similar permutation-invart adopted to encode trajectories in Grrver et 2018
transformation is previously
Encoding of Network Parameters for OPR. We propose a novel way to learn low-dimensional embedding from policy network parameters directly. To our knowledge, are the first learn
20

---

Under review as a conference paper at ICLR 2021
policy embedding from neural network parameters in RL. Note that latent space of neural networks are also studied in differentiable Network Architecture Search (NAS) (Liu al., 2019 Luo et al.) 2018), where architecture-level embedding are usually considered. In contrast, OPR cares about parameter-level embedding with given architecture.
Consider policy network to be MLP with well-represented state (e.g., CNN for pixels, LSTM for sequences) as input and deterministic or stochastic policy output. We compress all the weights and biases of the MLP to obtain an OPR that represents the decision function. The encoding process of an MLP with two hidden layers illustrated in Figure II The main idea perform permutation- invariant transformation for inner-layer weights and biases for each layer first. For each unit of some layer, view the unit non-linear function of all outputs, determined by weights, bias term and activation function. Thus, the whole layer can be viewed as a batch of operations of previous outputs, e.g., with the shape [h4, -
for 1 and t 0 is also for the imput layer. Note that we neglect activation function in the encoding since we consider the policy network structure is given. That is also why call OPR as parameter-level embedding in contrast to architecture- enbedding in NAS (mentioned in the last paragraph). We then feed the operation batch in an MLP and perform mean-reduce to outputs. Finally we concatenate encoding of layers and obtain the OPR.
We use permutation- invarrian transformation for OPR because that we suggest the operation batch introduced in the last paragraph can be coomutation-invariaan is not strict obtain permutation-invarian are not included during the encoding process. We also tried to incorporate the order information during OPR encoding and we found similar results with the way we present in Figure II which adopt in our experiments.
Actually, our encoding shown in representation since inter-layer dependencies
is
Input
Hidden
Output
Layer-wise Extraction
d.dim ht]
[hg,hg] h2]
[h2 dim)
hg]
[1,h2]
a1, dim]
Operations Preprocess
Concatenate & Transpose
(h,, 5_dim
[bg
[a_dim,hı
MLP Transformation & Mean-Reduce
Permutation-
[1,eg]
[1.ez]
[1, eg]
Invariant
Transform
Concatenate
[1,e, + te)
An illustration for policy encoder of Origin Policy Representation (OPR) for two-layer MLP. f1, h2 denotes the numbers of hidden units for the first and second hidden layers respectively. The main idea to perform permutation-invarian transformation for inner-layer weights and biases for each layer first and then concatenate encoding of layers.
Figure 11:
Towards more sophisticated RL policy that operates images. Our proposed two policy represen- tations (i.e., OPR and SPR) can basically be applied to encode policies that operate images, with the support of advanced image-based state representation. For OPR, a policy network with image in- put usually has pixel feature extractor like Convolutional Neural Networks (CNNs) followed by decision model (e.g., an MLP). With effective features extracted, the decision model can be mod- erate (or relatively small) scale. Recent works unsupervised representation learning like MoCo
21

---

Under review as a conference paper at ICLR 2021
(e et al. 2020), SimCLR (Chen all 2020 CURL [Srinivas al. 2020 also show that linear
classifier or a simple MLP which takes compact representation images learned an unsupervised fashion is capable of solving image classification and image-based continuous control tasks. In an- other direction, it promising develop more efficient even gradient-free OPR, for example using the statistics of network parameters in some way instead of all parameters as similarly considered in
Unterthiner et 2020
For SPR, to encode state-action pairs (or sequences) with image states can be converted to the encoding in the latent space. The construction of latent space usually involves self-supervised repre- sentation learning, e.g., image reconstruction, dynamics prediction. A similar scenario can be found in recent model-based RL like Dreamer (afner al. 2020 where the imagination is efficiently carried out in the latent state space rather than among original image observations.
Overall, we believe that there remain more effective approaches to represent RL policy be devel- oped in the future in a general direction of OPR and SPR, which are expected to induce better value generalization in different RL problems.
D.3 DATA AUGMENTATION FOR SPR AND OPR IN CONTRASTIVE LEARNING
Data augmentation is studied to be an important component in contrastive learning in deep RL re- cently (Kostrikov al. 2020 Laskin et al. 2020). Contrastive learning usually resorts data augmentation to build postive samples. Data augmentation is typically performed on pixel inputs (e.g., images) problems (He et al. 2020; Chen al. 2020). In our work, train policy repre- sentation with contrastive learning where data augmentation is performed on policy data. For SPR, i..., state-action pairs policy data, there is need to perform data augmentation since different batches randomly sampled state-action pairs naturally forms positive samples, since they all re- flect the behavior of the same policy. A similar idea can also be found in (Fu et al. 2020) when dealing with task context Meta-RL.
For OPR, i.e., policy network parameters as policy data, it is unclear how to perform data aug- mentation on them. In this work, we consider two kinds of data augmentation for policy network parameters as shown in Figure [2] We found similar results for both random mask and noise cor- ruption, and random mask as default data augmentation in our experiments.
Original
Random Mask
Noise Corruption
Input
Hidden
Output
Input
Hidden
Output
Input
Hidden
Output
Examples of data augmentation on policy network parameters for Origin Policy Repre- sentation (OPR). Left: an example of original policy network. Middle: dropout-like random masks performed original policy network, where gray dashed lines represent the weights masked out. Right: randomly selected weights are corrupted by random noises, denoted by orange lines.
Figure 12:
As unsupervised representation learning method, contrastive Learning encourages policies to be close to similar ones (i.e., positive samples π*) and to be apart from different ones (i.e.. negative samples π in policy representation space. The policy representation network is then trained with InfoNCE loss (Oord al. 2018), i.e., to minimize the cross-entropy loss below:
exp(x Wxm. X
La. = -E log
Σ,- exp X
D.4 PSEUDO-CODE OF POLICY REPRESENTATION LEARNING FRAMEWORK
The pseudo-code of the overall framework of policy representation learning is in Algorithm 3 Note that in Line 21, the positive samples X is obtained from momentum policy encoder (He et al.
22

---

Under review as a conference paper at ICLR 2021
2020) with another augmentation for corresponding policy data, while negative samples Xx, other policy embeddings in the same batch, i.c., Xn, E B....
Algorithm A Framework of Policy Representation Learning
Input: policy dataset DD {(T,, Wi, state-action pairs 7,, {(5j, ay)jy=
Dr,) ) consisting of policy Ta policy parameters w; and
1: Initialize the policy encoder Ja with parameters
2: Initialize the policy decoder (or master policy) (network) 7 (alss, Xx) for SPR and the weight matrix W for ORP respectively
3: for iteration i 0, 1...... do
4: Sample a mini-batch of policy data B from D
5: Encode and obtain the policy embedding Xn, with SPR OPR
6: if Use OPR then
7:
if Use Contrastive Learning then Perform data augmentation on each wy E B
8:
9:
end if
10:
Xm; OPR 1/oR((rr) (Wx;) for each (TT :) B
11:
else if Use SPR then
12:
Xn: = 0d SPR (B:) where B₁ a mini-batch of state-action pairs sampled from D,,, for
each (M. 1, Dx,)
13:
end
14:
Train policy encoder Ia in different ways (i.e., AUX or CL)
15:
if Use Auxiliary Loss then
Sample a mini-batch of state-action pairs B = (8i, ai) from Dr, for each mi Compute the auxiliary loss, LAux -- log Na (ails,, Xx,)
16:
17:
18:
Update parameters ax, 3 to minimize Aux
19:
end if
20:
if Use Contrastive Learning then
exp(x Wx,+) Calculate contrastive loss, La. = - Ex,, log exp(xt, wx,+)+ Ix, exp(x?, Wx,;)"
21:
where Xa, are positive and negative samples
22:
Update parameters a, W to minimize CCL
23:
end
24:
# Train policy encoder da with the PeVFA approximation loss (E2E)
25:
Calculate the value approximation loss of PeVFA, Lval
26:
Update parameters a to minimize Lval
end for
D.5 A REVIEW OF RELATED WORKS ON POLICY REPRESENTAATION/EEEDDD LEARNING
Recent years, few works involve representation or embedding learning for RL policy (Hausman et al. 2018 Grover et al. 2018 Arnekvist et al., 2019; Raileanu et al. 2020 Wang et al. 2020 Harb et al. 2020). We provide brief review and summary for above works below.
The most common way to learn a policy representation is to extract from interaction trajectorics through action recovery (i.c., behavioral cloning). For Multiagent Opponent Modeling (Grover) ctt al. 2018), a policy representation is learned from interaction episodes (i.e., state-action trajec- tories) through generative loss and discriminate loss. Generative loss is the same as the action prediction auxiliary loss; discriminate loss is triplet loss that minimize the representation distance of the same policy and maximize those of different ones, which has the similar idea of Contrastive Learning (Oord et al. 2018 Srinivas et al. (2020). Such opponent policy representations are used for prediction of interaction outcomes for ad-hoc teams and are taken in policy network for some learn- ing agent facilitate the learning when cooperating or competing with unknown opponents. More recently, in Hierarchical RL (Wang et al., 2020). a representation is learned to model the low-level policy through generative loss mentioned above. The low-level policy representation taken in high-level policy to counter the non-stationarity issue of co-learning of hierarchical policies. Later, Raileanu al. (2020) resort almost the same method and the learned policy representation is taken
23

---

Under review as a conference paper at ICLR 2021
in their proposed PDVF. Along with task context, the policy for specific task can be optimized in policy representation space, inducing a fast adaptation in new tasks. In summary, such a represen- tation learning paradigm can be considered as Surface Policy Representation (SPR) for policy data encoding (trajectories as a special form of state-action pairs) plus action prediction auxiliary loss (AUX) as we introduced in Sec.
A recent work (Harb et al. 2020 proposes Policy Evaluation Network (PVN) approximate objec- tive function J (T)) We consider PVN as an predecessor of PDVF we mentioned above since offline policy optimization is also conducted in learned representation space in a single task after similarly well training the PVN on many policies. The authors propose Network Fingerprint to represent policy network. To circumvent the difficulty of representing the parameters directly, policy network outputs (policy distribution) under a set of probing states are concatenated and then taken as pol- icy representation. Such probing states are randomly sampled for initialization and also optimized with gradients through PVN and policies, like a search in joint state space. In principle, we also consider this as special instance of SPR, because network fingerprint follows the idea of reflecting the information of how policy can behave under some states. Intuitively from a geometric view, this can be viewed as using the concatenation of several representative cross-scctions in policy surface (e.g., Figure TO to represent a policy. For another view, one can imagine an equivalent case between SPR and network fingerprint, when state-action pairs of a deterministic policy are processed in SPR and representation consisting of a number of actions under some key states or representative states is used in network fingerprint. Two potential issues may exist for network fingerprint. First, the dimensionality of representation is proportional to the number of probing states (i.e., n|[A),, where a dilemma exists: more probing states are more representative while dimensionality can increase correspondingly. Second, it can be non-trivial and even unpractical to optimize probing states in the case with relatively state space of high dimension, which induces additional computational com- plexity and optimization difficulty.
In another concurrent work (Faccio & Schmidhuber 2020 a class of Parameter-based Value Func- tions (PVFs) are proposed. Instead of learning designing representation of policy, PVFs simply parse all the policy weights as inputs to the value function (i.c., Raw Policy Representation as also mentioned in our paper), even in the nonlinear case. Apparently, this can result in a unnecessarily large representation space which increase the difficulty of approximation and generalization. The issues of naively flattening the policy into vector input are also pointed out in PVN Harb et al. 2020
Others, several works in Policy Adaptation and Transfer (Hausman et al. 2018 Amekvist et al., 2019) Gaussian policy embedding representations are construct through Variantional Inference.
D.6
CRITERIA OF A GOOD POLICY REPRESENTATION
To answer the question: what is a good representation for RL policy ought to be? We assume the following criteria:
Dynamics. Intuitively, a good policy representation should contain the information of how the policy influences the environment (dynamics and rewards).
Consistency. A good policy representation should keep the consistency among both pol- icy space and presentation space. Concretely, the policy representation should be distin- guishable, i.e., different policies also differ among their representation. In contrast, the representation of similar polices should lay on the close place in the representation space.
Geometry. Additionally, from the lens of policy geometry as shown in Appendix D.I] a good policy representation should be an reflection of policy geometry. It should show connection to the policy geometry or be interpretable from the geometric view.
From the perspective of above criteria, SPR follows Dynamics and Geometry while OPR may ren- der them in an implicit way since network parameters determine the nonlinear function of policy. Auxiliary loss for action prediction (AUX) a learning objective to acquire Dynamics; Contrastive Learning (CL) is used to impose Consistency.
Based on the above thoughts, we hypothesize the reasons of several findings as shown in the com- First, AUX naturally overlaps with SPR and OPR to some degree for Dynamics while CL is relatively complementary to SPR and OPR for Consistency. This may be the reason
parison in Table T
24

---

Recall the vector form of Bellman equation (Equation 14 indicates that value function a func- tion of policy π and environmental models (i.e., P and r). In conventional value functions and approximators, only state (and action) is usually taken as input while other components in Equation Under review as a conference paper at ICLR 2021
why CL improves the E2E representation more than AUX in overall view. Second, the noise of state-action samples for SPR may be the reason to OPR's ' slightly better overall performance than that of SPR (similar results are also found in our visualizations as in Figure 19
Moreover, the above criteria are mainly considered from an unsupervised or self-supervised perspec- tive. However, sufficiently good representation of all the above properties may not be necessary for a specific downstream generalization or learning problem which utilizes the policy representation. A mroblem-specinee learning signal, e.g., the value approximation loss in our paper (E2E represen- lation), can be efficient since it is to extract the most relevant information in policy representation for the problem. A recent work Tsai et al.] also studies the relation between self-supervised repre- sentation and downstream tasks from the lens of mutual information. Therefore, we suggest that trade-off between good unsupervised properties and efficient problem-specifi information of policy representation should be considered when using policy representation in specifie problem.
E COMPLETE BACKGROUND AND MORE
E REINFORCEMENT LEARNING
Markov Decision Process. We consider a Markov Decision Process (MDP) defined as (S, A, r, P. Po) with S the state space, A the action space, r the reward function, P the tran- sition function, Y 10,1) the discount factor and Po the initial state distribution. A policy
II P(A)(5), defines the distribution over all actions for each state. The agent interacts with the environment with its policy, generating the trajectory 80, Go 71, 81, 01 22, 8t: de, T++1s-.. where rz+1 r(st,a)) de). An RL agent seeks for an optimal policy that maximizes the expected long-term discounted return, J (TT) = Esu-
Ee 7rr+1.
Value Function. Almost all RL algorithms involve value functions (Sutton & Barto 1998). which estimate how good a state or a state-action pair is conditioned on a given policy. The state-value function v (s) is defined in terms of the expected return obtained through following the policy n from a state s:
o (*) = Ex TTe
= for all s E.
Similarly, action-value function is defined for all state-action pairs (s, En EE t y're+1 (su = 8, ag = a). Typically, value functions are learned through Monte Carlo (MC) or Temporal Difference (TD) algorithms (Sutton & Barto 1998).
Bellman equations defines the recursive relationships among value functions. The Bellman Expec-
tation equation of 6² (s) has matrix form below (Sutton & Barto 1998):
===r" + 7PVV = (I (P-
(14)
where // is |S)-dimensional vector, px is the state-to-state transition matrix (s'(ss) = EaeA T (P a) and rM is the vector of expected rewards (*()) = Eae (alss)r(s.a) a).
Equation 14 indicates that value function is determined by policy π and environment models (i.c., P and r. For conventional value function, all of them are modeled implicitly within table or function approximator, i.c., a mapping from only states (and actions) to values.
E.2 AN UNIFIED VIEW OF EXTENSIONS OF CONVENTIONAL VALUE FUNCTION FROM THE VECTOR FORM OF BELLMAN EQUATION
14 are modeled implicitly. Beyond state (and action), consider explicit representation of some of components in Equation| 14 during value estimation can develop the ability of conventional value functions in different ways, to solve challenging problems, e.g., goal-conditioned RL. JL.CShhul et al., 2015 Andrychowicz et al., 2017), Hierarchical RL (Nachum al. 2018, Wang al.) 2020). op- ponent modeling and ad-hoc team (He & Boyd-Grraber] 2016) Grover et al. 2018 Tacchetti et al.
2019) and context-based Meta-RI. (Rakelly al. 2019
Lee et al. 2020).
25

---

Under review as a conference paper at ICLR 2021
As discussed in Sec. 2.2 most extensions of conventional VFA mentioned above are proposed for the purpose value generalization (among different space). Therefore, we suggest such extensions are derived from the same start point (i.e., Equation [4] and differ at the objective to represent and take as additional input explicitly of conventional value functions. We provide unified view of such extensions below:
Goal-conditioned RL and context-based meta-RI. usually focus on series of tasks with similar goals and environment models (i.e.. P and r)) With goal representation as input,
usually a subspace of state space (Schaul et al. 2015 Andrychowicz et al. 2017), a value
function approximation (VFA) can generalize values
among goal space. Similarly, with
context representation (Rakelly et al. 2019 Fu et al.
2020 Raileanu et al.
2020 values
generalize in meta tasks.
Opponent modeling, ad-hoe team (He & et al., 2019) seek to generalize among different opponents or teammates opponents. This can be viewed as a special case
2016 Grover et al. 2018 Tacchetti Multiagent
System, with learned representation of value generalization among environment models since from one agent view, other op- ponents are part of the environment which also determines the dynamics and rewards. In multiagent case, one can expand and decompose the corresponded joint policy in Equation 14 to see this.
Hierarchical RL is also a special case of value generalization among environment mod- els. In goal-reaching fashioned Hierarchical HRL (Nachum et al. 2018 Levy et al. 2019 Nachum et al.] 2019), high-level controllers (policy) issue goals for low-level controls at abstract temporal scale, while low-level controls take goals also input and aim to reach the goals. For low-level policies. a VFA with a given or learned goal representa- tion space can generalize values among different goals, similar to the goal-conditioned RL case as discussed above. Another perspective is to view the separate learning process of hierarchical policies for different levels as a multiagent learning system. Recently, a work (Wang et al., (020) follows this view and extends high-level policy with representation of low-level learning.
The common thing of above is that, they learn representation of the environment (we call external variables). In contrast, we study value generalization among agent's own policies in this paper, which cares about internal variables, i.e., the learning dynamics inside of the agent itself.
Relation between PeVFA Value Approximation and Context-based Meta-RL. For a given MDP. performing policy in the MDP actually induces a Markov Reward Process (MRP) (Sutton & Barto 1998). One can view the policy and actions are absorbed in the transition function of MRP. value function defines the expected long-term returns starting from a state. Therefore, different policies induces different MRPs and PeVFA value approximation can be considered as meta prediction task. In analogy to context-bass based Meta-RL where a task context is learned to capture the underlying transition function and reward function of a MDP (i.c., task), one view policy representation as the context of corresponding MRP, since it is the underlying variable that determines the transition function of MRPs.
F EXPERIMENTAL DETAILS AND COMPLETE RESULTS
F.1 EXPERIMENTAL DETAILS
Environment. We conduct our experiments on commonly adopted OpenAl continuous con- trol tasks (Brockman et al. 2016 Todorov et al. 2012). We use the OpenAl Gym with version 0.9.1, the mujoco-py with version 0.5.4 and the MuloCo products with version MJPRO131. Our codes are implemented with Python 3.6 and Tensorflow.
Implementation. We use Proximal Policy Optimization (PPO) (Schulman et al.) 2017) with Gen- eralized Advantage Estimator (GAB) (Schulman et al.] 2016) as our baseline algorithm. Recent works (Engstrom et al., 2020 Andrychowiez al.] (020) point out code-level optimizations in- fluence the performance of PPO lot. For a fair comparison and clear evaluation, we perform no
http://gym. wptp:/ai.goo.pai....o/ com/
26

---

Under review as a conference paper at ICLR 2021
code-level optimization in our experiments, e.g., state standardization, reward scaling, gradient clip- ping, parameter sharing and etc. Our proposed algorithm PPO-PeVFA implemented based on PPO, which only differs at the replacement for conventional value function network with PeVFA network. Policy network a 2-layer MLP with 64 units per layer and ReLU activation, outputting Gaussian policy, i.e., tanh-activated mean along with a iate-indeeendent vector-parameteriiee log standard deviation. For PPO, the convention value network V (s) (VFA) is a 2-layer 128-unit ReLU-activated MLP with state input and value output. For PPO-PeVFA, the PeVFA network Vo(s, Xxr takes as input state and policy representation Xx which has the dimensionality of 64, with the structure illustrated in Figure 7 We do not use parameter sharing between policy and value function approximators for more clear evaluation.
Training and Evaluation. We use Monte Carlo returns for value approximation. In contrast to convention VFA V which approximates the value of current policy (c.g., Algorithm T. PeVFA
Vo(s, Xx) is additionally trained to approximate the values of all historical policies ({T.)) ==)) along the policy improvement path (e.g.. Algorithm 2) The policy network parameterized by is then updated with following loss function:
(PPO(W)) = -En [min (prA(ss, at), clip(p,,
e, + 1 cllpsss
c))
(15)
where A(st, an) is advantage estimation of old policy Tu-, which is calculated by GAE based on conventional VFA V or PeVFA (a X=) respectively, and pt = (ar.st))))
[ulae.8s) the importance
sampling ratio. Note that both PPO and PPO-PeVFA update the policy according to Equation 15 and only differ advantage estimation based on conventional VFA V or PeVFA Va(s, Xx). This ensures that the performance difference comes only from different approximation of policy values. Common learning parameters for PPO and PPO-PeVFA are shown in T For each iteration, we update value function approximators first and then the policy with updated values. Such training scheme is used for both PPO and PPO-PeVFA. For evaluation, we evaluate the learning algorithm every 20k time steps, averaging the returns of 10 episodes. Fewer evaluation points are selected and smoothed over neighbors and then plotted in our learning curves below.
Table 2: Common hyperparameter choices of PPO and PPO-PeVFA.
Hyperparameters for PPO & PPO-PeVFA
Policy Learning Rate
10
Value Learning Rate
10-3
Clipping Range Parameter (e)
0.2
GAE Parameter (A)
0.95
Discount Factor (7)
0.99
On-policy Samples Per Iteration
episodes 2000 time steps
Batch Size
128
Actor Epoch
10
Critic Epoch
10
Optimizer
Adam
Details for PPO-PeVFA. For PeVFA, the training process also involves value approximation of historical policies and learning of policy representation. Training details are shown in Table 3 PeVFA (o Xx) is trained every 10 steps with a batch of 64 samples from an experience buffer with size of 200k steps. Policy representation model is trained at intervals of 10 or 20 steps depending on
OPR or SPR adopted. Due to 1k 2k policies are collected in total in each trial, a relatively small batch size of policy is used. For OPR, Random Mask (Figure 12 is performed on all weights and biases of policy network except for the output layer (i.e., mean and log-std). For SPR, two buffers of state-action pairs are maintained for each policy: small one is sampled for calculating SPR and the relatively larger one is sampled for auxiliary training (action prediction).
27

---

Under review as a conference paper at ICLR 2021
Table 3: Training details for PPO-PeVFA, including value approximation of historical policies and policy representation learning. CL is abbreviation for Contrastive Learning and AUX is for auxiliary loss of action prediction. In our experiments, grid search is performed for the best hyperparamter configuration regarding terms with multiple alternatives (i.e., ()).
Value Approximation for Historical Policies
Value Learning Rate
10-3 3
Training Frequency
Every 10 time steps
Batch Size
64 200k (steps)
Experience Buffer Size
Policy Representation Learning
Training Frequency
Every {10, 20) time steps
Policy Num Per Batch
{16. 32)
SPR 8, Pair Num
{200, 500}
CI. Learning Rate
110-9.5.10-
CL Momentum
(5-10- - 10 5-100
CL Mask Ratio for OPR
{0.1, 0.23
CL Sample Ratio for SPR
0.8
AUX Learning Rate
10-3
AUX Batch Size
{128, 256}
F.2 COMPLETE LEARNING CURVES FOR TABLE
Corresponding to Table overall view of learning curves of all variants of PPO-PeVFA well as baseline algorithms are shown in Figure T6] One can refer to Figure 13 for Question I and Figure for Question 2 for clearer comparisons.
14 15
F.3 OTHER EXPERIMENTAL. ANALYSIS
In Figure 17 we analyze the sensitivity of value generalization (experimental results of PPO- PeVFA) with respect to the number of historical policies that PeVFA is trained on. The results show that the performance of PPO-PeVFA is not sensitive relatively large numbers (i.e., 200k, 500k, 1000k) of historical state-action samples (proportional to the number of recent policies) used. However, small number of historical samples may induce less sufficient and stable training of PeVFA, thus results in a slightly less improvement over the vanilla PPO.
F.4 VISUALIZATION OF LEARNED POLICY REPRESENTATION
To answer Question 3 we visualize the learned representation of policies encountered during the learning process.
Visualization Design. We record all policies on the policy improvement path during the learning process of PPO-PeVFA agent. For each trial in experiments in MuJoCo continuous control tasks, about 1k 2k policies are collected. We run trials and 5k 12k policies are collected in total for each task. We also store the policy representation model at intervals for each trial, and use last three checkpoints to compute the representation of each policy collected. For each policy collected during 5 trials, its representation for visualization is obtained by averaging the results of 3 checkpoints of each trial and then concatenating the results from 5 trials. Finally, we plot 2D) embedding of policy representations prepared above through t-SNE (Maaten & Hinton 2008) and Principal Component Analysis (PCA) in sklearr
Results and Analysis. Visualizations of OPR and SPR learned in an end-to-end fashion in HalfCheetah-vl and Ant-vl are in Figure 18 and 19 We use different types of markers to distin- guish policies from different trials to see how policy evolves in representation space from different random initialization. Moreover, we provide two views: performance view and process view, to
html
28

---

Under review as a conference paper at ICLR 2021
fard
pne
145
23
-50
120
304
20
150
13 Tea-Shps B1l0
10
(a) HalfCheetah-vl
(b) Hopper-vl
(c) Walker2d-vl
inp--geia er kand EMg- Pos--grra ee PM (CC0
WCA/P v (32)
(d) Ant-vl
(e) InverreDDubleeemdtuum///
(f) LumarLandeeroomtinuuus-22
02-
Mo AOl $10
(g) InvertedPendulum.vv
Figure 13: Evaluations of PPO-PeVFA with end-to-end (E2E) trained OPR and SPR in MuJoCo continuous control tasks. The results demonstrate the effectiveness of PeVFA and two kinds of policy representation, answering the Question T The results are average returns and the shaded region denotes half standard deviation over 10 trials.
29

---

Under review as a conference paper at ICLR 2021
110
230
300
330
(1650
(a) HalfCheetah-vl
(b) Hopper-vl
(c) Walker2d-v1
Stre 2461
The Dags C41)
1ra 24g115e1
(d) Ant-v1
(c) IivertedDoublceeenddluv-///
(f) Lanarl.anderCnninnnn
Figure 14: Evaluations of PPO-PeVFA with OPR and SPR trained through contrastive learning (CL) in MuJoCo continuous control tasks. The results are average returns and the shaded region denotes half a standard deviation over 10 trials.
310
2012
Po-Agr In Fare
NorT far P1
Pop-err e EM (50
MATT Shi (EE)
(11))
wit (UB)
300
(a) HalfCheetah-vl
(b) Hopper-vl
Walker2d-vl
232
landis
110
p-rhi pors
Aocki 1000
414
1279
58
111
-093
Ponelt C9 (40)
(d) Ant-v1
(e) InverteDDobbeeenntuuun---
(1) LunarLanderrConinnuuu---
Figure 15: Evaluations of PPO-PEVIA with OPR and SPR trained through auxiliary loss of action prediction (AUX) in MuloCo continuous control tasks. The results are average returns and the shaded region denotes half standard deviation over trials.
30

---

Under review as a conference paper at ICLR 2021
HarCChetta--V
Hopper-11
4246
2030
PPO-PAVFA W/ OPN (E2E)
PFO-PUVFA m/ OPA (E2E)
PRO PAUFA W/ SPR (EZE)
PPO POVFA w/ SPR (E2E)
Aveage Return
PRO PEVTA WT SPR ICIL
Average Return
PPO POO-PEVTA m/ SPR (CL)
PRO-PeVTA W/ DPA (ALI))
3805
PRO-PeVIA m/ GPR (AUX)
PEVFA SPR
PeVFA (AUX)
1336
1275
1200
942
150
611
-210
Timr Staps [1e55
Time Steps 1265)
(a) HalfCheetah--1
(b) Hopper-vl
Wwikerdov1
Antov1
2732
6396
PPO
POO
2162
PPO-PWURA w/ Rand PR
1872
PFO-PWFFA w/ Rand PR
2392
PRO-PeUFA (E2E)
3540
PPO-WWFAA SPR (F2E)
PRO PeVTA W/ OPR (CI))
PPO PEVFA w/ OPR (CL)
1781
Boverage Return
2825
-PPVFF
PPO-PaVFA w/ SPA (AUX)
PFO-PNVFA m/ SPR (AUX)
3.0
20.4
12.5
37%
20.0
535
2.
5.0
10.3
37.4
225
Time Steps (1e5)
Time Steps 12e5)
Walker2d-vl
(d) Ant-v1
Invee InveeteeDDuubaPenddillaasss
Lunar Curaraaanddrrrrrnaeusss2
Averege
5385
Auerage
4663
4364
TPO
+2358
PPO
PIO-PWVA Ranc
3000
-2683
-286.
-366.
1563
Time
Time (5e8)
(c) InverteDDoublleenntuunnm
(f) Lunarl andaraarnonrCCnmmuuuss....
Figure 16: An overall view of performance evaluations of different algorithms in MuJoCo contin- uous control tasks. The results are average returns and the shaded region denotes half standard deviation over 10 trials.
see how policies are aligned in representation space regarding performance and 'age' of policies respectively.
Visualization of OPR trained end-to-end fashion is shown in Figure 18 From the performance view, it is obvious that policies of poor and good performances aligned from left right in SNE representation space and are aligned at two distinct directions PCA representation space. An evolvement of policies from different trials be observed in subplot (b) and (d). Thus, policies
31

---

Under review as a conference paper at ICLR 2021
4104
3614
3125
Average
PPO
PPO PeVFA EZE OPR (50K)
PPO-NUVFA EZE OPA (200k, defauit
PPO-PeVIA EZE OPR (1000k)
-302
19.0
Time Steps (2e5)
Results for PPO-PeVFA with End-to-End (E2E) OPR on Ant-vl with respect to the number of historical policies that PeVFA is trained on. The setting varies at the numbers historical samples generated by recent policies different degrees, e.g., 50k denotes 50k steps of state-action samples from recent historical policies used for the training of PeVFA. The results are average returns and the shaded region denotes half standard deviation over 10 trials.
Figure 17:
from different trials are locally continuous; while policies are globally consistent in representation space with respect to policy performance. Moreover, we can observe multimodality for policies with comparable performance. This means that the obtained representation not only reflects optimality information but also maintains the behavioral characteristic of policy.
.35
42
(a) Performance View in HalfChectah-vl
(b) Process View in HalfChectah-vl
192
PA
POL
265
(c) Performance View in Ant-v1
(d) Process View in Ant-vl
Visualizations of end-to-end (E2E) learned Origin Policy Representation (OPR) for policies collected during 5 trials (denoted by different kinds of markers). In total, about 6k policies plotted for HalfCheetah-vl (a-b) and 12k for Ant-vl (c-d), In each subplot, t-SNE and PCA 2D embeddings are at left and right respectively, In performance view, each policy (i.e., marker) is colored by its performance evaluation (averaged return). In process view, each policy is colored by its corresponding iteration ID during GPI process.
Figure 18:
32

---

Under review as a conference paper at ICLR 2021
Parallel OPR, end-to-end trained SPR is visualized in Figure 19 A more obvious multimodality can observed in both t-SNE and PCA space: policies from different trials start from the same region and then diverge during the following learning process. Different from OPR, SPR shows more distinction among different trials since SPR is a more direct reflection of policy behavior (dynamics property mentioned in Sec. D.6p. Another thing is, policies from different trials forms wide 'strands' especially in t-SNE representation space. We conjecture that it is because SPR is more stochastic way to obtain representation as random selected state-action pairs are used.
300
200
98
(a) Performance View in HalfChectah-vl
(b) Process View in HalfChectah-vl
126
POL
DAYD
380
-5
(c) Performance View in Ant-v1
(d) Pracess View in Ant-v1
Figure 19; Visualizations of end-to-end (E2E) learned Surface Policy Representation (SPR) for policies collected during trials (denoted by different kinds of markers). In performance view, each policy (i.e., marker) is colored by its performance evaluation (averaged return). In process view, each policy is colored by its corresponding iteration ID) during GPI process.
33


Reviews:
[{"review_id": "iUi_71E0ASb", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary \n\nThe paper proposes to learn a value function that takes as input both a state and a policy embedding (PeVFA), which is used to design a new version of a generalized policy iteration (GPI) algorithm, named PPO-PeVFA. The authors also introduce a new way of learning policy embeddings and show superior results relative to vanilla RL (PPO) on several MuJoCo tasks. \n\n\nStrengths \n\nI found the proposed idea to be novel and interesting. Considering the value of multiple policies is an interesting and neglected line of work, which could prove useful in many ways for RL, so I commend the authors for taking a step in this direction. I also particularly liked the theoretical analysis, as well as the experiments supporting the claim for local and global generalization. However, I believe there are a number of issues with the experiments and clarity of the paper, which I would like to see addressed.\n \n\nWeaknesses\n\nMy main concern about this paper is the significance of the results and the empirical evaluation. The performance gain from using PeVFA does not seem very significant. After looking carefully at Table 1 and Figure 11 in the appendix, it seems like PPO-PeVFA is typically within a standard deviation of PPO. In addition, some of the reported results for PPO are much lower than the ones reported in the original PPO paper (Schulman et al. 2017). For example, on Hopper-v1 the reported numbers after 1M training sweets are 1600 and 2200, ,while for Walker2d-v1 they are 1500 and 3000 for the authors’ implementation and the original one, respectively. \n\nIn addition, I found the description of PPO-PeVFA to be quite confusing, so I think it can benefit from more details about the training procedure. Does the policy network share any parameters with the PeVFA network? Usually, parameters are shared between the policy and value function to improve learning, but it doesn’t seem like this is the case with PPO-PeVFA. Typically the policy and value function are updated at the same time in PPO but from Algorithm 2, it seems like before a policy update, you first update the PeVFA network with data from the current policy and then use the updated value to update the policy using the PG loss. Is my understanding correct? If this is the case, then it seems like the method is using some kind of privileged information about the current policy so an ablation that uses the same training scheme of first updating the value function (without conditioning on a policy) using the Monte Carlo rollouts from the current policy and then updating the policy should be included. \n\nI also think it would be useful to include comparisons with other baselines that achieve strong results on MuJoCo such as SAC or TD3 or alternatively show that PeVFA can improve over them as well. \n\nDo you have an intuition for why the difference between PeVFA and VFA is more significant for InvertedPendulum compared to Ant (and the other MuJoCo tasks) as illustrated in Figure 3b? \nI’ve found these plots slightly worrying because it seems like the value approximation gains are rather small for most of these tasks. It would be interesting to better understand the relation between these value generalization errors and the performance improvement.\n\nHave you looked into the sensitivity of PeVFA’s generalization with respect to the set (and number) of policies it is trained on? One can imagine training it on the entire history of policies during training or on a subset of these (perhaps the most recent ones). It would be useful to analyze how this choice affects the results. \n\nHave you tried combining the contrastive loss with the auxiliary loss for action action prediction? Is this better than either of them or not?\n\nCan you provide some quantitative metrics for the results in Figure 3a such as the MSE and ranking loss for the train and test values?\n\nWhy doesn’t Table 1 include results for InvertedPendulum? Can you add standard deviation for PPO and Ran PR?\n\n\nMinor Points\n\nTypos: \n“which are beyond the scope of this paper” instead of “which beyond…” in section 3.3\n“In the last section” instead of “in last…” in section 4\n“To answer the above questions” instead of “...answer above...” in section 5\n\n\nRecommendation\n\nWhile the proposed idea is interesting and novel, I believe more work is needed for publication (particularly in the empirical evaluation and algorithm description), so I lean towards rejection at this stage. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "FU1AfFHCYt", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe authors propose PeVFA: a value function able to evaluate the expected return of multiple policies. They do so by extending the conventional value function, allowing it to receive as input the parameter (or a representation) of the policy. The authors study the local generalization property of PeVFA, propose possible ways of encoding the policy parameters and compare traditional PPO with an extended version of PPO using PeVFA. While the idea of generalization among many policies is an interesting topic in RL, there are many theoretical and experimental issues that prevent acceptance. Moreover, the authors do not at all compare their approach to recent work which also uses value functions with policy parameters as input.\n\nReview:\n\nBelow the major problems I found:\n\n- State and action value functions receiving as input the parameters of a policy, which generalize to unseen policies were proposed in a work on Parameter-based Value Functions [1] (PVFs, see version v1 from 16 Jun 2020). Recent work on Policy Evaluation Networks (PENs) proposes policy embedding for a value function receiving as input the parameters of a policy [2]. These works introduce value functions which generalize to many policies. The authors must relate their Surface Policy Representation to the fingerprint mechanism in PENs and introduce their V and Q in the PVF framework.\n\n- Since the state space is not finite, $f_{\\theta}(\\pi)$, the approximation loss of $V_{\\theta}$ should be defined considering an expectation over the state space, which is weighted by a distribution over the states (e.g. the on-policy stationary distribution). Since the loss proposed does not include a distribution over the states, it is not clear if the contraction assumption should hold in expectation or for every possible state. The former would imply that all the results should be stated in a probabilistic framework; the latter would be false in a continuous state setting.\n\n- The authors assume that the class of value function considered can achieve zero approximation error. Where is this assumption used in the proofs and why is it useful?\n\n- Theorem 1 seems quite trivial. It is trivial that if the loss is decreasing in one point and it has some smooth properties, then there exists a close enough point such that the loss decreases also there. Assumption 2 should be stated in a clearer way, differentiating the cases of Lipschitz continuous function, Lipschitz continuous gradient and Lipschitz continuous Hessian more clearly.\n\n- Assuming that $\\gamma_g$ in Corollary 1 is lower than 1 during the training process is quite unreasonable, and the authors do not check if this holds in their experiments. I expect that for Corollary 1 to hold, the learning rate should be extremely small, thus preventing learning.\n\n- In Appendix A.2, there is an additional proof assuming that also $f(\\pi_2)$ is Lipschitz. The authors start from the bound in eq (8), which is strictly less tight than the assumption $f(\\pi_1) \\leq f(\\pi_2)$. By assuming only $f(\\pi_1) \\leq f(\\pi_2)$ one would get the same final condition for $f(\\pi_1)$, except for the term L_0 which would be zero. Hence the condition could be even less restrictive.\n\n- In Corollary 2 it is assumed that the sum of the losses over 2 consecutive policies is lower than the distance between the optimal value function of the two policies. Is there an interpretation of this assumption or is it just a technical requirement to complete the proof? Please discuss why this assumption should hold during the learning process.\n\n- When the policy representation is learned, the problem of mapping the policy parameters to the expected return becomes nonstationary, i.e. the same policy representation over time would be optimally mapped onto different values. How is this addressed in the experiments and how does nonstationarity affect the theoretical claims?\n\n- From the experiments it is not clear if a representation for the policy is necessary. I would have expected to see a comparison between PeVFA with raw policy representation (RPR) as input and PeVFA using the learned representations. The authors should provide strong evidence that RPR is not enough and the policy representation is needed. Furthermore, the policy used is very small (2 layers and 2 neurons per layer). I would expect to see benefits in using a policy representation when the policy is bigger (e.g. 2 hidden layers, 64 or 128 neurons per layer).\n\n- The authors provide no details about the hyperparameters used and about how they tuned their methods. Policy representation learning methods are only in part explained. Without further details, it is difficult to assess if the experiments provided were fair, and it is not possible to reproduce the results.\n\n- In appendix B.1. the transition function is not reported. The state includes sinusoidal terms that do not appear in the reward function. Why is it necessary for the agent to observe these terms? Are not just the x-y coordinate sufficient?\n\n- In Figure 3 and 7 it is not clear why there is only one learning curve for the policy policy and 6 value functions losses in standard PPO.\n\n- Algorithms 1 and 2 make almost no distinction in training when the update is on-policy or off-policy, because the authors consider PeVFA as just a replacement for standard value function when the algorithm is already derived. However, if PeVFA is introduced before deriving the algorithm, the derivation leads to different policy gradient theorems (see [1]). Please discuss this issue.\n\n- Figure 8 represents $\\pi(a|s)$ for each possible a, s. Therefore I would expect that for each s, the integral over A of the curve is 1. However, the area under the curve is much lower than 1. What is the explanation for this?\n\n- Last page in the Appendix is truncated.\n\nI think the most interesting contribution of this paper is the proposed policy representation. I would like to see a revised version published in the future. However, a lot of additional work is needed to address the aforementioned problems in the theory and the missing experimental evidence & implementation details & comparisons to very similar related work. \n\nMinor:\n- The title is grammatically incorrect.\n- \"with a limited parameter space\" -> what does limited mean? Finite? Or that $\\Theta$ is only a subset of the space of value functions?\n- same for \"unlimited\"\n- Appendix C.1 \"Advantage Acotor-Critic\" -> Advantage Actor-Critic\n\n\n[1] Francesco Faccio and Juergen Schmidhuber. Parameter-based Value Functions. arXiv preprint arXiv:2006.09226v1, 2020.\n\n[2] Jean Harb, Tom Schaul, Doina Precup, and Pierre-Luc Bacon. Policy evaluation networks. arXiv preprint arXiv:2002.11833, 2020.\n\n********************\n\nEdit: responses after rebuttal:\n\n> The class of PVFs are developed to be applied and evaluated in the online learning setting.\n\nThis is not true. See for instance off-line and zero-shot learning experiments with PVFs that can learn new policies which do not interact with the environment and still show generalization.\n\n> We think this may also be a reason to the insignificant improvement compared with their baselines in their experiments (their Figure 2), even the inferior results in simple tasks like InvertedPendulum and CartPole. In contrast, our proposed policy representations enable PeVFA to be compatible with normal-scale policy networks and we then demonstrate the superiority of PPO-PeVFA against PPO in standard continuous control task of OpenAI Gym\n\nI would strongly discourage you from claiming this. PPO-PeVFA is built on top of PPO, so it is very hard for your algorithm to perform worse than the baseline. On the other hand, PVFs and PENs are novel algorithms that rely COMPLETELY on the prediction of the value function. It is expected that they might outperform baselines in some environments and be comparable or worse in others. The PSSVF proposed in the work on PVFs is outperforming the baseline ARS (Mania et al. 2018) in all environments but Reacher, even when the policy is a neural net with 2 layers and 64 neurons per layer.\n\nAbout Q2:\n\n1. If the authors are using the $L_{\\infty}$ norm, then the results practically apply only to finite state MDPs. Indeed, this is the case for the papers [2,3,4] cited by the authors in the comment above. However, the experiments proposed in the paper deal with continuous state spaces, so the theoretical results are disconnected.\n\nIt is still not clear what the assumption in Corollary 2 means and if that assumption is met in practice. If the authors cannot justify the assumption, the Corollary should be removed. The novelty of Theorem 1 is not obvious, since it seems to be a quite trivial result from Mathematical Analysis.\n\n> The results show that PeVFA consistently shows lower losses (i.e., closer to approximation target) across all tasks than convention VFA before and after policy evaluation along policy improvement path, which demonstrates Corollary 2\n\nAgain, is the assumption of Corollary 2 met here? If you have a logical rule (or corollary) of the form: assumption -> consequence, and you show that the consequence holds, this does not imply that he assumption is true!\n\nOne main argument in favor of PVFs is that even with a much bigger policy, PVFs are able to outperform the results of PENs. This suggests that Raw Policy Representation (RPR) can be a strong baseline and any work trying to introduce a different policy representation should at least compare to RPR. Without such a baseline it is difficult to assess the benefits of the proposed policy representations. \n\nNote that PeVFAs are PVFs! The PVF formalism already accounts for all kinds of policy representations.\n\nOf course, all of these relations should also be clarified in title and abstract.\n\n\n(Mania et al. 2018): Horia Mania, Aurelia Guy, and Benjamin Recht (2018).  Simple random search of static linear policies is competitive for reinforcement learning.  In Advances in Neural Information Processing Systems,pp. 1800-1809, 2018.\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "A1wjenoBbRU", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper conditions the value function on a representation of the policy. The representation can be based on a batch of state-action pairs or based on the policy filters. When conditioning on the representation of a new policy, the value function can better approximate the value of the new policy. Experiments show benefits on continuous control tasks.\n\nAs mentioned in the paper, conditioning the value function on the policy was considered by some people before. It was unclear how to represent the policy. This paper makes it work and demonstrates clear benefits.\nThe policy representation can be still improved, and this paper can encourage people to try better representations.\n\nThe paper explains the ideas well.\nI would like to see a more detailed description of the Surface Policy Representation (SPR). The Appendix D.5 is missing the review of the related works.\n\nPros:\n- The value conditioned on the policy makes sense.\n- Proposed end-to-end training, contrastive learning and auxiliary distillation to train the policy embedding.\n- Tested raw policy representation, Surface Policy Representation and Origin Policy Representation.\n- Demonstrated benefits in experiments.\n- Visualization of the learned embeddings.\n\nCons:\n- It is unclear how to practically encode policy representations for policies operating on images.\n\nMinor typos:\n- String \"which beyond\" should probably be \"which are beyond\".\n- String \"together with PeVFA end-to-end\" should be \"together with PeVFA is end-to-end\".\n- String \"policies along the policy improvement path naturally provides\" should be \"policies along the policy improvement path naturally provide\".\n- String \"Use PPR\" should probably be \"Use SPR\".", "extra_scores": {}, "extra_text": {}}, {"review_id": "-baeGM-WyJQ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes an interesting idea of using the policy as an input to the value function. This is an interesting idea, moving away from the conventional approaches, and proposing to evaluate the value of a policy, by taking the policy as an input itself. Overall, it is an interesting direction, but there are few major concerns I have about the work, including some overstated claims which are not backed up properly. \n\nComments : \n- The proposed PeVFA framework is interesting, since the value function can take as input many different policies, providing a way for generalization among policies. Based on this, the key algorithmic contribution is to propose generalized policy improvement (GPI) which can guarantee policy improvements among many different policies following the generalized value functions. Previous works have considered taking goal states or other heuristics as input, often for exploration purposes. In contrast, this work suggests that taking policy as input might lead to better generalization, especially in transfer learning situations, as originally proposed by Successor Features (Barreto et al) proposing the GPI algorithm. \n- I do not think, however, that this idea is completely novel. Previous work on policy evaluation networks (Harb et al; arxiv paper from earlier this year) also proposed a similar approach, and this paper does not properly compare or take account of similar ideas from literature. It would be useful if the authors can comment on how their approach is different from Harb et al., given that the framework is almost similar. \n- The key idea is to propose value generalization among policies, such that the value fn of a policy can generalize to a new policy, as outlined in figure 2 and section 3.1. I have a major concern about Assumption 1 being made - I am not fully convinced whether this is a valid assumption for the contraction property to hold in this. I do not think it is a major bottleneck, but to state this assumption makes me question whether the learnt PeVFAs can at all be useful. \n- Theorem 1 is well stated. However, the proof of theorem 1 is not properly justified, and makes the contribution less significant. It would be useful if the authors can expand more on this, especially how the Lipchitz assumptions from Nesterov et al.,  are valid in this case. Otherwise, theorem 1 itself does not add much value for the overall contribution. \n- My major concern with this work is how are PeVFas actually implemted in practice. The authors seem to build up from PPO, which would require the policies to be separately parameterized too. In this case, it would mean that the policy network is fed as input to the value function network itself. How does this approach scale up when the policy network is large? More importantly, how are gradients computed in this case? It is not clear whether this sort of hypernetwork idea is applicable for practical implementation of this approach, and this seems to be a major unavoidable issue with the work. Similar issues also arise in Harb et al., which they describe as the \"fingerprinting\" step. However, I think this is a major bottleneck for any of these works. Although the idea itself has potential, but taking policy as input, and considering parameterized policies might not be the way to go for this? I wonder if this kind of approach may be more preferrable in value based settings? Can the authors comment on this? Figure 4 seems to justify this sort of network architecture, but is not convincing. \n- The paper discusses on the representation learning aspect of the policy. I understand that this is useful, leading to the generalization aspect this work is trying to propose. However, the claims are not made clear and difficult to understand. Is this work suggesting that the policy representations learnt through this framework can help learn invariant representations, which are then further useful for generalization? Some of the experimental demonstrations seem to address on this - however, it is not clear to what extent the invariant representations are indeed meaningful? As a reader, I was expecting the authors to compare and discuss experimentally with successor features and variants, for the GPI algorithm? However, there seem to be no discussions on those?\n- Empirically, it is not clear what the paper is trying to propose. The authors claim to build up from PPO, but the experiments are not well justified for the generalization or transfer learning aspect? What would be the key takeaway from the experimental results? Table 1 is almost meaningless, given that the plots for these results in figure 11 in appendix shows marginal performance improvements. Experiments are only compared with PPO on control tasks, which does not have any generalization aspect per say? What are the authors trying to demonstrate in this case? I would suggest the authors propsoe their approach on simpler tasks, e.g four rooms domains and variants, and show how the proposed framework can be useful to transfer across tasks? Would that be a doable experiment that might add value to the contributions?\n- Another minor issue is that it is not clear whether the algorithm can be implemented online or offline? How are the value functions trained, when the policy is used for rollouts? For example, if policy 1 is used for rollouts to compute the value functions, and doing a policy improvement step, how are these samples re-used when policy 2 is used as input to the value function network? I assume this will make the approach off-policy, since the rollouts under one policy would induce distribution shift compared to the other policy? I am not sure I fully understand how this algorithm can be implemented in practice, and what are the major drawbacks? Can the authors comment on the major difficulties of practical implementation of this idea? It would perhaps be useful if the pseudo-code of the algorithm can be included?\n\nOverall, I think it is an interesting idea; but there are major concerns I have about the work which seem to be unaddressable? My main concern is with the practical implementation of this algorithm (I think I understand the intuitive and overall motivation for this, and why doing this might be useful). The paper claims to take this approach for the first time. However, there seems to be a prior work on a similar idea, which is not addressed in this work either. It would be useful if the authors can clarify on some of these doubts, for me to fully understand the practical significance of this work. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to consider value functions as explicit functions of policies, in order to allow generalization not only on the state(action) space, but also on the policy space.
The initial reviews assessed that the paper was dealing with an important RL topic, but also raised many concerns about the position to previous works (PVN and PVF), the theoretical contributions and the experiments. The authors provided a rebuttal and revision that only partly addressed the initial concerns (check also the review of R3, updated to provide additional feedback following the author’s rebuttal).
The final discussion led to the assessment that the paper is not ready for publication. Remaining concerns include clarity, claims being not fully supported by the experiments, theoretical aspects and missing baselines.

Decision: Reject