Represent Your Own Policies: Reinforcement Learning with Policy-extended Value Function Approximator
Authors: Hongyao Tang
Zhaopeng Meng
Jianye HAO
Chen Chen
Daniel Graves
Dong Li
Hangyu Mao
Wulong Liu
Yaodong Yang
LI Wang
Venue: NeurIPS 2021 Submitted (2021)
Keywords: Reinforcement Learning

Abstract: We study Policy-extended Value Function Approximator (PeVFA) in Reinforcement Learning (RL), which extends conventional value function approximator (VFA) to take as input not only the state (and action) but also an explicit policy representation. Such an extension enables PeVFA to preserve values of multiple policies at the same time and brings an appealing characteristic, i.e., \emph{value generalization among policies}. We formally analyze the value generalization under Generalized Policy Iteration (GPI). From theoretical and empirical lens, we show that generalized value estimates offered by PeVFA may have lower initial approximation error to true values of successive policies, which is expected to improve consecutive value approximation during GPI. Based on above clues, we introduce a new form of GPI with PeVFA which leverages the value generalization along policy improvement path. Moreover, we propose a representation learning framework for RL policy, providing several approaches to learn effective policy embeddings from policy network parameters or state-action pairs. In our experiments, we evaluate the efficacy of value generalization offered by PeVFA and policy representation learning in several OpenAI Gym continuous control tasks. For a representative instance of algorithm implementation, Proximal Policy Optimization (PPO) re-implemented under the paradigm of GPI with PeVFA achieves about 40\% performance improvement on its vanilla counterpart in most environments.

Represent Your Own Policies: Learning with Policy-extendee Value Function Approximator
Anonymous Author(s)
Affiliation
Address
email
Abstract
We study Policy-extended Value Function Approximator (PeVFA) in Reinforce- ment Learning (RL), which extends conventional value function approximator (VFA) take input not only the state (and action) but also explicit policy representation. Such an extension enables PeVFA to preserve values of multi- ple policies the same time and brings an appealing characteristic, i.e., value generalization among policies. We formally analyze the value generalization un- der Generalized Policy Iteration (GPI). From theoretical and empirical lens, we show that generalized value estimates offered by PeVFA may have lower initial approximation eтror to true values of successive policies, which expected improve consecutive value approximation during GPI. Based on above clues, we introduce new form of GPI with PeVFA which leverages the value generalization along policy improvement path. Moreover, we propose representation learning framework for RL policy, providing several approaches learn effective policy em- beddings from policy network parameters or state-action pairs. In our experiments, we evaluate the efficacy of value generalization offered by PeVFA and policy representation learning in several OpenAl Gym continuous control tasks. For a representative instance of algorithm implementation, Proximal Policy Optimization (PPO) re-implemented under the paradigm of GPI with PeVFA achieves about 40% performance improvement on its vanilla counterpart in most environments.
2
4
6
8
10
11
12
13
14
15
17
18
19
Introduction
20
21 Reinforcement Learning (RL) has been widely considered as a promising way to learn optimal policies in many decision-making problems 35
62 16]. One fundamental element of 23 RL value function which defines the long-term evaluation of a policy. With function approximation
31
53
65
47
24 (e.g., deep neural networks), value function approximator (VFA) is able to approximate the values
25 of policy under large and continuous state spaces. As commonly recognized, most RL algorithms can be described Generalized Policy Iteration (GPI) [55). As illustrated on the left of Figure [ 27 at each iteration the VFA is trained to approximate the true values current policy (i.c., policy evaluation), regarding which the policy is further improved (i.e., policy improvement). The value function approximation error hinders the effectiveness policy improvement and then the overall optimality of GPI [5 46). Unfortunately, such enrors inevitable under function approximation. A large number of samples are usually required to ensure high-quality value estimates, resulting in the caample-inefficienee of deep RL algorithms. Therefore, this raises urgent need for more efficient
29
31
32
33 value approximation methods [1
12 25
An intuitive idea to improve the efficiency value approximation is leverage the knowledge 35 the values of previous encountered policies. However, a conventional VFA usually approximates the values of one policy and values learned from old policies are over-written gradually during
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

75
R2
Kg
72
p
y
pit
60 We summarize our main contributions below. 1) We study the value generalization among policies induced by PeVFA. From both theoretical and empirical aspects, we shed the light on the situations where the generalization can be beneficial to the learning along policy improvement path. 2) We Ve, (Xm))
Vo, (tro)
Ve..
Viu
Vas
(Xre
(a,
Vo, (7.2
Improvement
Evaluation
Approximation
Generalization
Figure 1: Generalized Policy Iteration (GPI) with function approximation. Left: GPI with con- ventional value function approximator Vg. Right: GPI with PeVFA (x=) (Sec. 3 where extra generalization steps exist. The subscripts of policy and value function parameters o, 0 denote the iteration number. The squiggle lines represent non-perfect approximation of true values.
37
the learning process. This means that the previously learned knowledge cannot be preserved and utilized with one conventional VFA. Thus, such limitations prevent the potentials to leverage the previous knowledge for future learning. In this paper, we study Policy-extended Value Function Approximator (PeVFA), which additionally takes an explicit policy representation as input in contrast to conventional VFA. Thanks to the policy representation input, P&VFA is able to approximate values for multiple policies and induces value generalization among policies. We formally analyze the generalization of approximate values among policies general form, From both theoretical and empirical lens, we show that the generalized value estimates can be closer to the true values of the successive policy, which can be beneficial to consecutive value approximation along the policy improvement path, called local generalization. Based on above clues, we introduce new form of GPI with PeVFA (the right of Figure I that leverages the local generalization to improve the efficiency of consecutive value approximation along the policy improvement path.
38
39
40
41
43
44
45
46
47
49
One key point of GPI with PeVFA the representation of policy since it determines how VF gen- eralizes the values. For this, we propose framework to learn effective low- www-dieesiionl embedding of RL policy. We use network parameters or state-action pairs as policy data and encode them into low-dimensional embeddings; then the embeddings are trained to capture the effective information through contrastive learning and policy recovery. Finally, we evaluate the efficacy of GPI with PEV and our policy representations. In principle, GPI with PeVFA is general and can be implemented
50
51
52
53
54
55 in different ways. As a practical instance, we re-implement Proximal Policy Optimization (PPO)
56 with PeVFA and propose PPO-PeVFA algorithm. Our experimental results several OpenAl Gym
57 continuous control tasks demonstrate the effectiveness of both value generalization offered by PeVFA and learned policy representations, with about 40% improvement in average returns achieved by 59 our best variants on standard PPO in most tasks.
61
62
63 propose a framework for policy representation learning. To our knowledge, we make the first attempt
64 to learn low-dimensional embedding of over 10k network parameters for an RL policy. 3) We introduce GPI with PeVFA that leverages the value generalization a general form. Our experimental results demonstrate the potential of Pe in deriving practical and more effective RL algorithms.
65
Related Work
67
88
Extensions of Conventional Value Function. Sutton al. [56] propose General Value Functions (GVFs) as general form of knowledge representation of rewards and arbitrary cumulants. Later, conventional value functions are extended take extra inputs for different purposes of generalization. One notable work is Universal Value Function Approximator (UVFA) [45). which is proposed to generalize values among different goals for goal-conditioned RL. UVEA further developed in
89
70
71
72
73
9 and influences the occurrence of other value function extensions in context-based Meta- RL 19 [4] and etc. Most of the above works study how to generalize the policy or value function among extrinsic factors, i.e., environments, tasks and opponents; while we mainly study the value generalization among policies along policy improvement
1,
74
43 29), Hierarchical RL (4 and multiagent RL
76
path, an intrinsic learning process of the agent itself.

---

78
Policy Embedding and Representation. Although not well studied, representation (or embedding)
79
learning for RL policies is involved in few works [8] 14 37. The most common way to learn a policy representation is to extract from interaction experiences. As a representative, Grover et al. [14] propose learning the representation of opponent policy from interaction trajectories with a generative policy recovery loss and a discriminative triplet loss. These losses are later adopted in [64] 42) Another straightforward idea is to represent policy parameters. Network Fingerprint [17 is such a differentiable representation that uses the concatenation of the vectors of action distribution outputted by policy network on a set of probing states. The probing state set is co-optimized along with the primary learning objective, which can be non-trivial especially when the dimensionality of the set is high. Besides, some early attempts in learning low-dimensional embedding of policy parameters are studies in Evolutionary Algorithms [13 44 mainly with the help of VAE [23]. Our work introduce learning framework of policy representation including both above two perspectives.
80
81
83
84
85
87
86
89
90
PVN and PVFs. Recently, several works study the generalization among policy space. Harb et al. 17 propose Policy Evaluation Network (PVN) to directly approximate the distribution of policy ss objective function (T) Epo je" (so)] with initial state So ~ po. PVN takes as input Network Fingerprint (mentioned above) of policy network. After training on a pre-collected set of policies, a random initialized policy can be optimized in a zero-shot manner with the policy gradients of PVN by backpropggiing through the differentiable policy input. We call such gradients CTPI for short below. Similar ideas are later integrated with sasc-specific context learning in multi-tast RL 422. leveraging the generalization among policies and tasks for fast policy adaptation on new tasks. In PVN [17], an early attempt, the generalization among policies is studied with small policy network and simple tasks; besides, the most regular online learning setting is not studied. Concurrent to our work, Faccio and Schmidhuber [TO] propose class of Parameter based Value Functions (PVFs) that take vectorized policy parameters as inputs. Based on PVFs, new policy gradient algorithms are introduced in the form of a combination of conventional policy gradients and GTPI (i.e., by backpropagating through policy parameters in PVFs). Except for zero-shot policy optimization as conducted in PVN, PVFs are also evaluated for online policy learning. Due directly taking parameters as input, PVFs suffer from the curse of dimensionality when the number of parameters is high. Besides, GTPI can rein since policy parameter space are complex and extrapolation generalization error can be large when the value function is only trained on finite policies (usually much fewer than state-action samples) thus further resulting in erroneous policy gradients.
91
93
94
95
97
89
100
101
102
103
104
105
108
be non- ooivial
107
108
109
Our work differs with PVFs from several aspects. First, we make use of learned policy representation rather than policy network parameters. Second, do not resort to GTPI for the policy update in our algorithms but focus on utilizing value generalization for more efficient value estimation in GPI. Furthermore, we shed the light on two important problems how value generalization among
110
111
112
113
policies can happen formally and whether it is beneficial to learning or not - which are neglected in in previous works from both theoretical and empirical lens.
114
3 Policy-extended Value Function Approximator
115
116
In this section, we propose Policy-extended Value Function Approximator (PeVFA), extension of conventional VFA that explicitly takes as input a policy representation. First, we introduce the formulation (Sec. 3.I). then we study value generalization among policies theoretically (Sec. 3.2 along with some empirical evidences (Sec. 3.3). Finally, we derive a new form of GPI (Sec. 3.4)
117
119
120
3.1 Formulation
Consider Markov Decision Process (MDP) defined as (S.A,r, where S is the state space, A is the action space, r is the (bounded) reward function, P is the transition function and y E [0, 1) is the discount factor. A policy π E P P(A))() SS defines the distribution over all actions for each state. The goal of an RL agent is to find optimal policy π² that maximizes the expected long-term discounted return. The state-value function x (s) is defined as the expected discounted return obtained through following the policy from a state v² (s) = E,, 12 ''re+1 |su s) for where rt+1 = r(st, az).
122
123
125
126
127
We use v to denote the vectorized form of value function.
In a general form, we define policy-extended value function V : S x II + R over state and policy space: V(s, = v=(s) (s) for all 8 E S and क E IT. In this paper, we focus on ((s, x) and policy- extended action-value function (ss, a, T) can be obtained similarly. We use ((T) to denote the value
128
129
3

---

ye
(
(,,
Wer
Value, agg..
Volue Apuroximation, o.g. (A(() Value Generationion,c ea. Ya (tov) e4,
Learned
C Unlcarned
Value Space
Generalization
fo(n) (8,(t) - 1'1
(a) Global Generalization
(b) Local Generalization
Figure 2:
Illustrations of value generalization among policies of PeVFA. Each circle denotes value function (estimate) a policy. (a) Global Generalization; values learned from known policies can be generalized to unknown policies. (b) Local Generalization: values of previous policies (e.g., TT) can be generalized to successive policies (e.g., T(+1) along policy improvement path.
vector for all states in the following. The key point is that PeVFA V is able preserve the values of multiple policies. With function approximation, a PeVFA is expected to approximate the values of policies among policy space, i.e., [V"*=en and then enable value generalization among policies.
132
133
134
Formally, given a function g II + X C R" that maps any policy क to an n-dimensional represen- tation Xx ( E X, a PeVFA Va with parameter 8 θ is to minimize the approximation error over all possible states and policies generally:
136
F P.ppo (0, g. IT)
Σ (())) V TENl
where p. p are distributions over policies and states respectively, IIflpp. = (L. [(ds)] is p-weighted -y-norm (26) 46 for any f S + R. The policy distribution of interest depends on the scenario where value generalization value generalization scenarios. In the global generalization scenario, uniform distribution over known policy set may be considered with a general purpose of value generalization for unknown policies. For the specific local generalization scenario along policy improvement path during GPI, sophisticated distribution that adaptively weights recent policies more during the learning process may be suitable in this case. In the following, care more about the local generalization scenario and use uniform state distribution and I2-norm for demonstration. The subscripts are for clarity.
137
139
considered. As illustrated in Figure 2 we provide two
141
143
145
148
omitted and use ||
3.2
Theoretical Analysis on Value Generalization among Policies
148
In this part, we theoretically analyze the value generalization among policies induced by PeVFA. We start from a two-policy case and study whether the value approximation learned for one policy can be generalized to the other one. Later, we study the local generalization scenario (Figure 2(b) and shed the light the superiority of PeVFA for GPI. All the proofs are provided in Appendix
149
150
152
For the convenience of demonstration, we use an identical policy representation function, i.c., Xx n. and define the approximation loss of PeVFA Va for any policy т E 11 as fe (T) = ( V*I > 0. We use the following definitions for a formal description of value approximation process with VeVFA
153
154
155
and local property of loss function fo that influences generalization 40 63) respectively:
156
Definition Value Approximation) We define a value approximation process Pr A
with Pe VFA y-contraction mapping on the approximation loss for policy n, i.e., for 0 Pz) we have (T( S (To(TT) where y E 1).
158
159
Definition 2 (L- (L-Continuity) We call fa is L-continuous at policy π if fe is Lipschitz continuous at g with constant
[0, oc), iee. [so()) ( - fo(m')) S T. d(m=, x²) for xe' E II with some distance metric d for policy space TT.
180
162
With Definition the consecutive value approximation for the policies along policy improvement path during GPI can described as: 0_1 Pro @
0o 0₃ Pe2, as the green arrows illustrated in Figure One may refer to Appendix A. for a discussion the rationality of the two definitions.
163

---

185
To start our analysis, we first study the generalized value approximation loss in a two-policy case where only the value of policy πi is approximated by PeVFA as below:
186
Lemma 1 For P - O, if Sa is L-continuous al n and fo(TI) S So(TT), we have: [ā((T)) S
187
fo (m2) + M(11, n2, L), where MM T2, L) = f. d(m11 T2).
168
Corollary m is conttraannnnn (7g E [0, 1)) for л2 when fo((2)) L.d(x).92))
169
1-7
170
Lemma 1 shows that the post- approximation loss for T2 is upper bounded by generalized contraction of prior loss plus a locality margin term M which is related to IΙ, #2 and the locality property of fg. In general, the form of M depends on the local property assumed. Some higher- order variants are provided in Appendix A.2 For a step further, Corollary reveals the condition
171
172
175
174
where a contraction
value approximation loss for πT is achieved when PeVFA ís only trained to approximate the values of TT. Concretely, such a condition is apt to reach with tighter contraction for policy π is, closer two policies, or smoother approximation loss function fg-
175
178
177
Then we consider the local generalization scenario as illustrated in Figure 2(b) For any iteration t of GPI, the values of current policy सe are approximated by PeVFA, followed by a improved policy T-- whose values are to be approximated in the next iteration. The value generalization from each Te and T can be similarly considered as the two- two-policy case. In addition to the former results, we
178
179
180
181
shed the light on the value generalization loss of PeVTA along policy improvement path below:
Lemma 2 For A-1 Lno, G0 Lr1,
01 P22,
182
with it for each PT., if fo, is La-contimuous at Te for any t 2 O, we have for (T++1) S nifo. (TT) + +Ms where M, = L1 dl( [e, 11-)).
183
Corollary By induction, we have fo, (T++1) I oo vefo, (To)) + E II T Mg + Me.
184
185
The above results indicate that the value generalization loss can be recursively bounded and has a upper bound formed by a repeated contraction on initial loss plus the accumulation of locality margins induced from each local generalization. An infinity-case discussion for Corollary 2 is in Appendix A.5 The next question is whether PeVFA with value generalization among policies is preferable to the conventional VFA. To this end, we introduce a desirable condition which reveals the superiority of VeVFA during consecutive value approximation along the policy improvement path:
186
187
188
190
Theorem During 0-1 Fro, Oo Th
Sti, 0i Frz, VT I|, then fo, (Te+1) S VV, (Te) V=+til.
181
for any t 2 O, if (T.) + (T+1)) S
182
193
Theorem shows that the generalized value estimates Vo, (m++1) can be closer to the true values of policy T than Vo, (TT). Note that Vo, (TT) is the value approximation for ne which is equivalent to the counterpart Ve. for a conventional VFA as value generalization among policies does not exist. To consecutive value approximation along policy improvement path, this means that the value generalization of Pe has the potential to offer closer start points at each iteration. If such closer start points can often exist, we expect PeVFA to be preferable to conventional VFA since value approximation can be more efficient with PeVFA and it in turn facilitates the overall GPI process.
194
185
196
197
198
199
However, the condition in Theorem 1 is not necessarily met in practice. Intuitively, it depends on the locality margins that may be related to function family and optimization method of PeVFA, as well as the scale of policy improvement. We leave these further theoretical investigations for future work. Instead, we empirically examine the existence of such desirable generalizations in the following.
201
202
203
204
3.3 Empirical Evidences
We empirically investigate the value generalization of PeVFA with didactic environments. In this section, PeVFA Ve is parameterized by neural network and we use the concatenation of all weights and biases of the policy network straightforward representation X= for each policy, called Raw Policy Representation (RPR). Experimental details are provided in Appendix B
206
208
209
First, we demonstrate the global generalization (illustrated in Figure 2(a) in continuous 2D Point Walker environment. We build the policy set II with synthetic policies, each of which is a randomly initialized 2-layer lunh-activated neural network with 2 units for each layer. The size of IT is 20k and
210
212
the behavioral diversity of synthetic policies verified (see Figure 7(b) in Appendix). We divide II into training set (i.c., known policies Iog) and testing set (i.c., unseen policies Il,). We rollout the
5

---

3010
111
irm 20hert Pracc loii (atar)
PVFA Safeeti
2014
300
(a) Global Generalization
(b) Local Generalization
Figure 3: Empirical evidences of kinds of generalization of PeVFA. (a) Global generalization: PeVFA shows comparable value estimation performance on testing policy set (red) after learning on training policy set (blue). (b) Local generalization: PeVFA (Vo(x=)) shows lower losses than conventional VFA (V₂) before and after the value approximation training for successive policies along policy improvement path. In (b), the left axis is for approximation loss (lower is better) and the right axis is for average return as reference of the policy learning process (green curve).
214 policies in the environment to collect trajectories. based on which we perform value approximation training. Our results show that PeVEA trained on Πy achieves reasonable generalization performance when evaluating on II The average losses on training and testing set are 1.782 and 2.071 over trials. Figure ((a) shows the value predictions for policies from training and testing set (100 for each).
216
218 Next, we investigate the value generalization along policy improvement path, i.e., local generalization as in Figure 2(b) We use 2-layer 8-unit policy network trained by standard PPO algorithm (50) in MuJoCo continuous control tasks. Parallel the conventional value network V (s) (i.e., VFA) in
220
PPO, we set PeVFA network Va Xar) as reference for the comparison value approximation loss. Compared to Vg, PeVFA Ve(s, Xxx takes RPR as input and approximates the values of all historical policies (m))so)) in addition. We compare the value approximation losses of V. (red) and Vo (blue) before (solid) and after (dashed) updating with on- on-policy samples collected by the improved policy Ti+1 at each iteration. Figure ((b) shows the results for InvertedPendulum- v1 and Ant-v1. Results for all 7 MuJoCo tasks can be found in Appendix [B.2 By comparing approximation losses before updating (red and blue solid curves), we observe that the approximation loss of V (Xm..1) ) is almost consistently lower than that of Va, This means that the generalized value estimates offered by PeVFA are usually closer to the true values of Te+1, demonstrating the consequence arrived Theorem I For the dashed curves, shows that PeVFA Va (Xm(11) can achieve lower approximation loss for TT41 than conventional VFA Voeus after the same number of training with the same on-policy samples. The empirical evidence above indicates that PeVFA can be preferable to the conventional VFA for consecutive value approximation. The generalized value estimates along policy improvement path have the potential to expedite the process of GPI.
222
224
226
228
230
231
232
233
234
3.4
Reinforcement Learning with PeVFA
Based on the results above, we expect to leverage the value generalization of PeVFA to facilitate RL. In Algorithm 1 we propose general description of RI. algorithm under the paradigm of 238 GPI with PeVFA. For each iteration, the interaction experiences of current policy and the policy
236
Algorithm 1 RL under the paradigm of GPI with PeVFA (V(s, is used for demonstration)
Initialize policy To. policy representation model g. PeVFA V-1 and experience buffer 2: for iteration t = 0, L,... do
3: Rollout policy ne in the environment and obtain i trajectories T, = {Tr)
4: Get representation X=, = g(T) for policy x; and add experiences (X Te) in buffer D
5: if t % MM then Update PeVEA V, X for previous policies with data {(x,,) T Update policy representation model g. e.g., with approaches provided in Sec. end if
6:
7:
8:
9:
Update PeVFA V,-1 Xx.) for current policy Xv, and set V, - V--
10:
Update n1 w.r.t V, Xx.) by policy improvement algorithm and set T++1 (- Te
11:
end for

---

Policy Decoder
MLP
(MLF)
Data Source/Input
Policy Encoder
MLP
Auxilory Loss
Policy Farams (OPR)
Contrastive Learning
Unsupervized Leaming
Palicy (5.0) Pairs (SPR)
Trans
MLP
Xm grodients
(End-to-End]
MLP
(S.Xx))
MLP
PeVFA
Figure 4: The framework of policy representation training. Policy network parameters used for OPR or policy state-action pairs used for SPR are fed into policy encoder with permutation-invariant (PI) transformations followed by MLP, producing the representation Xx. Afterwards, X= can be trained by gradients from the value approximation loss of PeVFA (i.e., End-to-End), as well as (optionally) the auxiliary loss of policy recovery or the contrastive learning (i.e., InfoNCE) loss.
239
representation are stored in a buffer (line 3-4). At an interval of M iterations, Pe is trained via value approximation for previous policies with the stored data and the policy representation model is updated according to the method used (line 5-8). This part is unique to PeVFA for preservation and generalization of knowledge of historical policies. Next, value approximation for current policy is performed with PeVFA (line 9). A key difference here is that the generalized value estimates (i.e.. (x=.( are used as start points. Afterwards, a successive policy is obtained from typical policy improvement (line 10). Algorithm I can be implemented in different ways and we propose an instance implemented based on PPO [50] in our experiments later, In the next section, we introduce
240
241
242
243
244
246
our methods for policy representation learning.
4
Policy Representation Learning
248
249
To derive practical deep RL algorithms, one key point is policy representation, i.c., a low-dimensional embedding of RL policy. Intuitively. policy representation influences the approximation and gener- alization of PeVFA. Thus, is of interest to find an effective policy representation based on which the superiority of VFA can be leveraged to improve RL algorithms. To our knowledge, policy representation is not well studied and it remains unclear on how to obtain an effective representation for an RL policy in general case in practice. In previous section, we demonstrate the effectiveness of using policy parameters a naive representation when policy network is small, called RPR. However, usual policy network may have large number of parameters, thus making inefficient and even irrational to use RPR for approximation and generalization [17] [00. More generally, policy parameters of the policy we wish to represent may not be accessible.
250
251
252
253
254
256
258
259 To this end, we propose a general framework of policy representation learning as illustrated in Figure The first thing to consider is data source, i.e., from which we can extract the information for an effective policy representation. Recall that the policy is distribution over state and action space of high dimensionality. The features of such a distribution not directly available. Therefore, we consider two kinds of data source below that indirectly contains the information of policies: 1) Surface Policy Representation (SPR): The first data source is state-action pairs (or trajectories [4] ) since they reflect how policy may behave under such states. This data source is general since no explicit form of policy is assumed. In a geometric view, learning policy representation from state-action pairs can be viewed as capturing the features of policy via scattering sample points the curved surface of policy distribution. 2) Origin Policy Representation (OPR): The other data source is parameters of policy since they determine the underlying form of policy distribution. Such data source is often available during the learning process of deep RL algorithms when policy is parameterized by neural networks. Generally, consider policy network to be an MLP with well represented state features (e.g., features extracted by CNN for pixels or by LSTM for sequences) as input.
260
261
262
254
286
266
270
272
The remaining question is how we extract the policy representation from the data sources mentioned above. As shown in Figure 4 we use permutation-invarian (PI) transformations followed by an MLP to encode the data of policy π into embedding Xxr for both SPR and OPR. For SPR, each
274
7

---

pair of {(s;, ai) is fed into common MLP, followed by a Mean-Redut operation on the outputted features across k. For OPR, we perform PI transformation (similar as done for state-action pairs) inner-layer weights and biases {[w, by) |2_ for each layer first, where h denotes the number of nodes in this layer and wi, is the income weight vector from previous layer and the bias of ith node; then we concatenate encoding of layers and obtain the OPR. A illustrative
278
277
279
280
281
description for the encoding of OPR is in Figure [2 of Appendix.
To train the policy embedding Xxr obtained above, the most straightforward way is to backpropagate the value approximation loss of PeVFA in an End-to- End (E2E) fashion as illustrated on the lower- right of Figure 4 In addition. we provide two self-supervised training losses for both OPR and SPR, as illustrated on the upper-right of Figure The first one is an auxiliary loss (AUX) of policy recovery 14 i.e., to recover the action distributions of π from X= under different states. To be specifie, 286
282
283
284
205
287
an auxiliary policy decoder क (-|s, X=) is trained through behavioral cloning, formally to minimize cross-entropy objective LAUX- = - -E Ra
(aa) (alg(a)s. X=)] For the second one, we propose to train X by Contrastive Learning (CI.) [54 51: policies are encouraged to be close to similar ones (i.e.,
289
positive samples π*), and to be apart from different ones (i.e., negative samples π ) in representation space. For each policy, we construct positive samples by data augmentation on policy data, depending on SPR or OPR considered; and different policies along the policy improvement path naturally provide negative samples for each other. Finally, the embedding Xx is optimized through minimizing xxp(x""Xx-- 41 below: fCL = -B ({--}) log exp(x)Wx,++++3 [x,+)+++-- exp(x.WX,-
291
293
294
the InfoNCE loss
Now, the training of policy representation model in Algorithm| [ can be performed with any com- bination of data sources and training losses provided above. A pseudo-code of the overall policy representation training framework and complete implementation details are provided in Appendix D
285
298
297
Experiments
298
In this section, we conduct experimental study with focus on the following questions:
300
Question Can value generalization offered by PeVTA improve a deep RL algorithm in practice? Question Can our proposed framework to learn effective policy representation?
301
302
Our experiments are conducted in several OpenAI Gym continuous control tasks (one from Box2D and five from MuJoCo) 6 58)
303
All experimental details and curves can be found in Appendix B
304
Al Algorithm Implementation. We use PPO (50 as the basic algorithm and propose a representative
305
implementation of Algorithm [ called PPO-PeVFA. PPO is policy optimization algorithm that left). A value network (s(ss) with parameters ø (i.e., conventional VFA) trained to approximate the value of current policy π.; while is optimized with
306
follows the paradigm of GPI (Figure D
307
respect to a surrogate objective [48] using advantages calculated by V and GAE (49] Compared with original PPO, PPO-Pe VFA makes use of a PeVFA network V Ve(s, Xx) with parameters 0 rather than
308
309
the conventional VFA (s), and follows the training scheme as in Algorithm [ Note PPO-PeVFA uses the same policy optimization method as original PPO and only differs at value approximation.
310
311
312 Baselines and Variants. Except for original PPO as a default baseline, we use another two baselines: 1) PPO-PeVFA with randomly generated policy representation for each policy, denoted by Ran PR; 2) PPO-PeVFA with Raw Policy Representation (RPR). i.e., use the vector of all parameters of policy adopted in PVFs (10]. Our variants of PPO-PeVFF differ at the policy representation used. In total, we consider 6 variants denoted by the combination of the policy data choice (i.e., OPR, SPR) and representation principle choice (i.e., E2E, CL, AUX).
314
315
network as representation
316
317
318 Experimental Details. For all basclines and variants, we use a normal-scale policy network with 2 layers and 64 units for each layer, resulting in over 3k to 10k (c.g., Ant-vl) policy parameters depending on the environments. We do not assume the access to pre-collected policies. Thus the size of policy set increases from 1 (i.e., the initial policy) during the learning process, to about Ik to
320
322 2 for a single trial. The dimensionality of all kinds of policy representation expect for RPR is set
323 to 64. The buffer D maintains recent 200k steps of interaction experience and the policy data of
corresponding policy. The number of interaction step of each trial is 1M for InyDouPend-v and Lu 4M for Ant-vl and 2M for the others.
324
325
326
Results. The overall experimental results are summarized in Table I In Figure we provide aggregated results across all environments expect for InyoouPend-v and LunarLander-v2 (since
8

---

Table 1: Average returns (# half a std) over 10 trials for algorithms. Each result is the maximum evaluation along the training process. Top two values for each environment are bold.
Beschmarks
Orggin Policy Regrssertttion (Ours)
Surface Policy Represestation (Churs)
Favirouments
PPO
PR
RPR
EIF
C1.
AUX
EIE
AUX
HalICheetabb-vl
2470
2325
627.63
3175 517
233.29
3349
3216
Hopper-v1
1226
1097
213.47
51091
2214 550
297.35
2202
2577
217.73
Walker2dowl
1505
1269
317 152
1856
2038
515.1
2044
316.32
1910.57
435.0
2203
881.95
1980
525 54
Ant-u1
2835
2742
2145 406.6 64
3581
185.43
4019
162.47
3784
+ 268.99
3173
184.75
3632
134-27
3397 200003
In-DuuPend-ww
9341
9355
8856
551.90
9357 0.29
9355
2: 0.04
9355 0:0.8
9355
0,89
9356 2: 0.9%
9355
2: 1.42
Lamart.nndee-v2
219
226
-22 + 33.08
238 4: 3.37
239
3.70
254 4 3.47
236
3:13
234 3.33
235 8:5.7
most algorithms achieve near-optimal results), where all returns are normalized by the results of PPO in Table [ Full learning curves are omitted and can be found in Appendix F2
329
330
To Question 1 From Table I we can find that both PPO- PEV w/ OPR (E2E) and PPO-PeVFA w/ SPR (E2E) outperforms PPO in all 6 tasks, and achieve over 20% improvement in Figure [ This demonstrates the effec- tiveness of PeVFA. Moreover, the improvement is further enlarged (to about 40%) by CL and AUX for both OPR and SPR. This indicates that the superiority of PeVFA can be further utilized with better policy representation that offers more suitable space for value generalization.
PPO
331
14
PPO PNAAR hand PM
332
TPO PIOTA on PPM
FPO PAYES GPA (828)
333
PAVEO SP% (22F)
POYPA OTA ICLI
335
(AUX
337
339
To Question 2 In Table L consistent degeneration is observed for PPO-PeVFA w/ Ran PR due to the nega- tive effects on generalization caused by the randomness and disorder of policy representation. This phenomenon seems to be more severe for PPO-PeVFA w/ RPR due to the complexity of high-dimensional parameter space, In contrast, the improvement achieved by our proposed PPO-PeVFA variants shows that effective policy repre- sentation can be learned from policy parameters (OPR) and state-action pairs (SPR) though value approximation loss (i.c., E2E) and further improved when additional self- supervised representation learning is involved as CL and AUX. Overall, OPR slightly outperforms SPR as CL does over AUX. We hypothesize that it is due to the stochas- ticity of state-action pairs which serve as inputs of SPR and training samples for AUX. This reveals the space for future improvement. In addition, we visualize the learned representation in Figure 6 We can observe that policies from different trials are locally continuous and show dif- ferent modes of embedding trajectories due random initialization and optimization; while a global evolvement among trials emerges with respect to policy performance.
S0oza thar mallad)
340
Figure 5: Normalized averaged returns aggregated over 4 MuloCo tasks.
341
342
345
345
347
349
Trai2
351
Trai
353
Figure 6: A t-SNE visualization for representations learned by PPO-PeVFA OPR (E2E) in Ant-v1. In total, 6k poli- cies from 5 trials (denoted by different markers) are plotted, which are colored according to average return.
354
355
356
357
358
359
360
Conclusion and Future Work
381
362
In this paper, we propose Policy-extended Value Function Approximator (PeVFA) and study value generalization among policies. We propose new form of GPI based on PeVFA which is potentially preferable to conventional VFA for value approximation. Morcover, we propose a general framework to learn low-dimensional embedding of RL policy. Our experiments demonstrate the effectiveness of the generalization characteristic of PeVFA and our proposed policy representation learning methods.
364
386
367
Our work opens up some research directions on value generalization among policies and policy representation. A possible future study on the theory of value generalization among policies is to consider the interplay between approximation error, policy improvement and local generalization
368
389
during GPI with PeVFA. Besides, analysis on influence factors of value generalization among policies (e.g., policy representation, architecture of PeVFA) and other utilization of PeVFA are expected. For better policy representation, inspirations on OPR may be got from studies on Manifold Hypothesis of neural network; the selection of more informative state-action pairs for SPR is also worth research.
370
372

---

References
374
[1] M. Andrychowiez, D. Crow, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba. Hindsight experience replay. In NeurlPS, pages 5048-5058, 2017.
378
377
[2] M. Andrychowicz, A. Raichuk, P. Stanezyk, M. Orsini, S. Girgin, R. Marinier, L. Hussenot, M. Geist, O. Pietquin, M. Michalski, S. Gelly, and O. Bachem. What matters in on-policy reinforcement learning? large-scale empirical study. CoRR, (bss2200.059990 2020.
379
[3] I. Amekvist, D. Kragic, and J. A. Stork. VPE: variational policy embedding for transfer reinforcement learning. In ICRA, pages 36-42, 2019.
382
[4] M. G. Bellemare, W. Dabney, and R. Munos. A distributional perspective on reinforcement learning. In ICML, volume 70, pages 449-458, 2017.
384
[5] D. P. Bertsekas and J. N. Tsitsiklis. Neuro-dynamic programming volume 3 of Optimization and neural computation series. 1996.
386
[6] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. CoRR, abs/1606.01 2016.
387
386
[7] T. Chen, S. Kornblith, M. Norouzi, and G. E. Hinton. A simple framework for contrastive learning of visual representations. CoRR, abs/2002.05 05700, 2020.
389
390
[8] L. Engstrom, A. Ilyas, S. Santurkar, Tsipras. F. Janoos, Rudolph, and A. Madry. Implementation matters in deep RL: A case study on PPO and TRPO. In ICLR, 2020.
391
382
B. Eysenbach, Geng, Levine, R. R. Salakhutdinov. Rewriting history with inverse RL: hindsight inference for policy improvement. NeurlPS, 2020.
383
[10] F. Faccio and J. Schmidhuber. Parameter-based value functions. CoRR, abs/2006. 0920000009922,....00 2020.
385
[11] H. Fu, H. Tang. J. Hao. C. Chen, X. Feng, D. Li, and W. Liu. Towards effective context for meta- reinforcement learning: approach based on contrastive learning. CoRR, ahs/2009.1389) 2020.
397
[12] S. Fujimoto, H. v. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In ICMI., 2018.
399
[13] A. Gaier, A. Asteroth, and Mouret. Discovering representations for black-box optimization. GECCO, pages 101-111,20 2020. 2020.
400
401
[14] A. Grover, M. Al-Shedivat, J. K. Gupta, Y. Burda, and Edwards. Learning policy representations in multiagent systems. In JCML, volume 80, pages 1797-1806, 2018.
402
403
[15] T. Haarnoja, A. Zhou, P. Abbeel,
Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In ICML, pages 1856-188 1865, 2018.
404
[16] D. Hafner, T. P. Lillicrap, Ba, and M. Norouzi. Dream control: Learning behaviors by latent imagination, ICLR, 2020.
406
[17] J. Harb, T. Schaul, D. Precup, and P. Bacon. Policy evaluation networks. CoRR, abs/2002.11833 2020.
408
[18] K. Hausman, J. Tobias Springenberg, Z. Wang, N. Heess, and M. A. Riedmiller. Learning an embedding space for transferable robot skills. In ICLR, 2018
410
[19] H. He and J. L. Boyd-Graber. Opponent modeling in deep reinforcement learning. In ICML, volume 48, pages 1804 1813, 2016.
411
412
[20] K. Hc, H. Fan, Y. Wu, S. Xie, and R. B. Girshick. Momentum contrast for unsupervised visual representation lcarning. In CVPR, pages 9726-9735, 2020.
413
414
[21] Igl. G. Farquhar, Luketina, Boehmer, and S. Whiteson. The impact non-stationarity on generalisation in deep reinforvement learning. CoRR, abs/2006.00882. 2020.
415
416
[22] N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In ICLR, 2017.
417
[23] D. P. Kingma and M. Welling. Auto-encoding variational bayes. In ICLR, 2014.
419
[24] 1. Kostrikov, D. Yarats, and R. Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. CoRR, ab642004.13649, 2020.
10

---

[25] A. Kuznetsov, P. Shvechikov, A. Grishin, and D. P. Vetrov. Controlling overestimation bias with truncated mixture of continuous distributional quantile crities. In ICML, volume 119, pages 5556-5566, 2020.
[26] M. G. Lagoudakis and R. Parr. Least-squares policy iteration. J. Mach. Learn. Res., 4:1 107 1149, 2003,
[27]
Q. Lan, Y. Pan, A. Fyshe, and M. White. Maxmin q-learning: Controlling the estimation bias of q-learning. In ICLR, 2020.
425
428
[28] Laskin, K. Lee, A. Stooke, Pinto, P. Abbeel, and A. Srinivas. Reinforeement learning with augmented data. CoRR, abs/2004, 14990, 2020.
427
428
[29] K. Lee. Y. Seo, S. Lee, H. Lee. and J. Shin. Context-aware dynamics model for generalization in model-based reinforcement learning. CoRR, 0hs/2005506800 2020.
429
430
[30] A. Levy, G. D. Konidaris, R. P. Jr., and K. Saenko. Learning multi-level hierarchies with hindsight. In ICLR, 2019.
[31] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Frez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. In ICLR, 2015.
433
434
[32] H. Liu, K. Simonyan, and Y. Yang. DARTS: differentiable architecture search. In ICLR, 2019.
[33] R. Luo, F. Tian, T. Qin, Chen, and T. Liu. Neural architecture optimization. In NeurlPS, pages 7827-7838, 2018.
436
437
[34] L. V. D. Maaten and G. E. Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 2579-2605, 2008.
438
439
[35]
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. A. Riedmiller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglon, H. King. D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature,
442
518(7540):529-53) 2015.
443
[36] V. Mnih, A. P. Badia, M. Mirza, A. Graves, Lillicrap. Harley, Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforvement learning. ICML, 2016.
444
445
[37] O. Nachum, S. Gu, H. Lee, and S. Levine. Data-efficient hierarchical reinforcement learning. CoRR. abs/ 805.08296, 2018.
446
447
[38] O. Nachum, S. Gu, H. Lec, and S. Levine. Near-optimal representation learning for hicrarchical reinforee- ment learning. In ICLR, 2019.
[39] Y. F. Nesterov and B. T. Polyak. Cubic regularization of newton method and its global performance. Math. Program., 177(1): 2006.
450
[40] R. Novak, Bahri, A. Abolañia, Pennington, and J. Sohl-Dickstein. Sensitivity and generalization in neural networks: an empirical study. In ICLR, 2018
452
453
[41] A. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. CoRR, 01//1807707749 2018.
454
455
[42] R. Raileanu, M. Goldstein, A. Szlam, and R. Fergus. Fast adaptation via policy-dynamics value functions.
456
CoRR, abs/2007.02879, 2020.
457
[43] K. Rakelly, A. Zhou, C. Finn, S. Levine, and D. Quillen. Efficient off-policy meta-reinfforcement learning via probabilistic context variables. In ICML, volume 97, pages 5331-5340, 2019.
[44] N. Rakicevic, Cully, and P. Kormushev. Policy manifold search: Exploring the manifold hypothesis for diversity-based neuroevolution. CoRR, 1ass//204.13424., 2021.
460
[45] Т. Schaul, Horgan, K. Gregor, D. Silver. Universal value function approximators. JCML, volume 37, pages 1320- 2015.
462
463
[46] B. Scherrer, M. Ghavamzadeh, Gabillon, B. Lesner, and M. Geist. Approximate modified policy iteration and its application to the game of tetris. J. Mach. Learn. Res., 16:1629-166 1676, 2015.
464
485
[47] J. S Schreck, C. W Coley. and K. JM Bishop. Learning retrosynthetic planning through simulated experience. ACS central science, (000:970-911..0 2019.

---

467
[48] J. Schulman, S. Levine, P. Abbeel. M. I. Jordan, and P. Moritz. Trust region policy optimization. In ICML, pages 1889-1897, 2015.
489
[49] Schulman, P. Moritz, S. Levine, M. I. Jordan, and Abbeel. High-dimensional continuous control using
470
generalized advantage estimation. In ICLR, 2016.
471
[50] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06344) 067377770067,.... 0010066332.1211 2017.
473
[51] Schwarzer, A. Anand, R. Goel, Hjelm, A. C. Courville, and P. Bachman. Data-efficient reinforcement learning with momentum predictive representations. CoRR, 0e5/20007055922 2020.
474
475
[52] D. Silver, G. Lever. N. Heess. T. Degris, D. Wierstra, and M. A. Riedmiller. Deterministic policy gradient algorithms. In ICML, pages 387-395, 2014.
477
[53]
Silver, A. Huang. C. Maddison, A. Guez, Sifre, van den Driessche, Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. P. Lillicrap, M. Leach, K. Kavukcuoglu, T. Gracpel, and D. Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484 2016.
478
479
[54] A. Srinivas, M. Laskin, and P. Abbeel. CURL: contrastive unsupervised representations for reinforcement learning. CoRR, abs/2004.04 04336, 2020.
482
483
[55] R. S. Sutton and A. G. Barto. Reinforcement learning an introduction. Adaptive computation and machine learning. MIT Press, 1998.
485
[56] R. Sutton, Modayil, M. Delp, T. Degris, P. M. Pilarski, White, and D. Precup. Horde: scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. AAMAS, pages 761-768, 2011.
486
487
488
[57] A. Tacchetti, H. F. Song, P. A. M. Mediano, V. Flores Zambaldi, J. Kramár, N. C. Rabinowitz, Th. Graepel, M. Botvinick, and P. W. Battaglia. Relational forward models for multi-agent learning. In ICLR, 2019.
490
[58] E. Todorov, Erez, and Tassa. Mujaco: A physics engine model-based control. 2012 IEEE/RSJ
491
International Conference on Intelligent Robots and Systems, pages
2012.
492
[59] Y. H. Tsai, Y. Wu, R. Salakhutdinov, and L. Morency. Demystifying self-supervised learning: An coomattion-thoreticc. framework. CORR, 00ns/2006.05576, 2020.
494
[60] T. Unterthiner, Keysers, Gelly, O. Bousquet, and I. O. Tolstikhin. Predicting neural network accuracy from weights. CoRR, abs/2002. 1448, 2020.
495
496
[61] H. Hasselt. Double q learninn. In J. D. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. S. Zemel, and A. Culotta, editors, NeurlPS, pages 2613-2621, 2010.
498
[62]
Czarnecki, M. Mathieu, Dudzik, Chung. H. Choi, Powell, T. Ewalds, P. Georgiev, J. Oh, D. Horgan, M. Kroiss, I. Danihelka, A. Huang, L. Sifre, T. Cai, J. Agapiou,
O. Vinyals, I. Babuschkin,
499
500
M. Jaderberg, A. S. Vezhnevets, R. Lehlond, T. Pohlen, V. Dalibard, D. Budden, Y. Sulsky, J. Molloy, T. L.. Paine, C. Gulcehre, 7. Wang, T. Pfaff, Y. Wu, R. Ring, D. Yogatama, D. Wünsch, K. McKinney, O. Smith, T. Schaul, T. Lillicrap, K. Kavukcuoglu, D. Hassabis, C. Apps, and D. Silver. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350-334) 2019.
503
504
[63] H. Wang, N. S. Keskar, C. Xiong, and R. Socher. Identifying generalization properties in neural networks. CoRR, abs/180 htt//80990000 809.07402, 2018.
505
[64] R. Wang, R. Yu. B. An, and Z. Rabinovich. P'rrl: Interactive influence-based hierarchical reinforcement learning. IJCAI, pages 3131-3138, 2020.
507
508
[65] J. You, B. Liu, Z. Ying, V. S. Pande, and J. Leskovec. Graph convolutional policy network for goal-directed molecular graph generation. In NewrlPS 2018, pages 6412-6422, 2018.
509
12

---

Checklist
510
511
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
513
514
(b) Did you describe the limitations your work? [Yes] the future work in Sec.
(c) Did you discuss any potential negative societal impacts of your work? [No] Our work
516
general Reinforcement Learning study. specific practical application considered.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
518
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
521
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either the supplemental material or a URL)? [No] Our experimental environment are public and standard. All the information needed to reproduce our results is provided in the main body and appendix. Code will be available publicly soon.
528
524
525
526
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
527
[Ye]] Partially in main body and all details can be found in the appendix document.
528
(c) Did you report error hars (c.g.. with respect to the random seed after running experiments multiple times)? [Yes]
530
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
532
4. If you are using existing assets (e.g.. code, data, models) or curating/releasing new assets...
(a) If your work uses existing asscts, did you cite the creators? [Yes]
534
(b) Did you mention the license of the assets? [Yes] We use a free education licence for students for
MuJoCo.
536
(c) Did you include any new assets either in the supplemental material or as URL? [No]
537
(d) Did you discuss whether and how consent was obtained from people whose data you're 're us- ing/eurating? [N/A]
539
(e) Did discuss whether the data you are using/curating contains personally identifiable informa- tion offensive content? [N/A]
541
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable?
543
[N/A]
(b) Did you descrihe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
545
546
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
547
13


Reviews:
[{"review_id": "whp1UjCHdQ", "reviewer": "Reviewer_RjEL", "summary": "The authors propose PeVFA, an extension to value functions that include not only states (or state-action pairs), but a representation of the parameters of the policy. They apply their method to PPO and show experimentally that such policy representations help improving over standard PPO. Unfortunately, the theoretical results in the paper seem trivial and disconnected from the rest of the paper. The value functions presented in the paper were already introduced in previous work. The authors compare extensively these methods to their approach, but introduce their PeVFAs separately, creating some confusion on the novelty of the work.\n", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n\n  - I like the extensive comparison between PeVFAs and PBVFs[1]/PENs. I think this work can provide and important contribution in that line of research by providing useful ways of representing the policy parameters. This tackles the main limitations of PBVFs. However, I think the authors should present their work as an improvement over PBVFs (in particular the PSVF), rather than introducing PeVFAs from scratch and then claiming after that this was already done in PBVFs. Note that PBVFs are theoretically sound to receive any representation of the policy, although in their experiments no representation is used.\n   - The theory proposed is a basic analysis result and should be omitted. In particular, if a function is L-continuous and it decreases in a point, then there exists another point close enough such that the function decreases also there. Moreover, the contraction property is never holding in the experiments performed, hence the theoretical results are completely disconnected from the paper.\n  - The assumption in Theorem 1 seems to be a technical step that is needed to prove the theorem itself, but not a reasonable assumption. It states that the sum of the losses of two consecutive policies has to be lower than the distance between the value functions of the two policies. How can one interpret this? \n - In line 229 the authors claim that Figure 3 demonstrates the consequences of Theorem 1. This is not possible, since the assumptions of Theorem 1 are never met.\n- I think the term GPTI is never defined\n- PBVFs are used for offline learning (not online)\n- the citation on PBVFs refers to an old ArXiV submission and not to the version published [1]\n\n\nAgain, the contribution relative to policy representation seems interesting and useful and it should be the main focus, but the paper in its present form is not ready for acceptance. If the authors remove the theoretical part and address my concerns about how they introduce PeVFAs, I will increase the score accordingly.\n\n[1] Francesco Faccio, Louis Kirsch, and J¨urgen Schmidhuber. Parameter-based value functions. In Int. Conf. on Learning Representations (ICLR), Virtual only, May 2021\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "jK9zfaF-GJ", "reviewer": "Reviewer_NFfA", "summary": "This paper studies a class of reinforcement learning methods which utilize a value function which in addition to the state is conditioned on some representation of the current policy. The idea being that this can allow the agent to learn a value function that explicitly generalizes among policies as the policy changes over the course of learning and ultimately improve performance. The approach is named Policy-extended Value Function Approximation (PeVFA).\n\nThey present some theoretical results with essentially show that if we assume that lowering the error in the value function for one policy results in sufficiently lower error for nearby policies, and the policy does not change too much between iterations, we can bound the value error for a policy dependent value function in terms of distance between policies at subsequent iterations. They also show that assuming the policy dependent value function generalizes well it can provide a benefit over a policy independent value function in terms of value error during a policy iteration procedure.\n\nThey further present empirical results using policy dependent value functions with a variety of different techniques for conditioning the value function on the policy. Policy representation techniques tested include: Raw Policy Representation (RPR) which uses all the parameters of the policy network as input to a value approximation, Surface Policy Representation (SPR) which uses a set of state action pairs gathered from running the policy, and Origin Policy Representation (OPR) which again uses the policy network parameters, but preprocessed by a permutation invariant transform. In addition, they test combining these representation strategies with auxiliary losses encouraging policy recovery from eh learned representation.\n\nThey study the use of policy dependent value functions in two settings, a supervised learning like setting where a number of fixed policies are initially evaluated to train the value function and a reinforcement learning setting where a policy is adapted online while learning a policy dependent value function for the policies the arise along the way. Results indicate that in the supervised case the learned value function can generalize to unseen policies.  In the reinforcement learning case, policy dependent value functions appear to confer some performance improvement when used with PPO compared to standard value functions.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The basic idea of somehow learning policy conditioned value functions as a way to more rapidly adapt to a changing policy is interesting and reasonable. A key challenge in implementing this is obviously how to encode a policy in a way that can be input to a neural network. My overall impression is that the empirical aspect of this work was reasonable, but could be improved, while the theory does not seem terribly useful.\n\nHaving checked over the theory in reasonable detail the results appear correct to me. However, the theory is arguably not very useful as it's intuitively obvious that if we assume the value function generalizes between policies meaningfully such that reducing the loss for one policy reduces the loss for nearby policies sufficiently, then such a policy dependent value function function will confer a benefit over a policy independent one. The hard part is how to actually achieve such generalization, and how to encode a policy into a form that is convenient to input into a neural network.\n\nAlso Theorem 1, does not actually use any of the lemmas, which makes it strange to call them lemmas, and seems trivial. Essentially it just applies definitions and the triangle inequality to conclude that if the sum of the value errors for two subsequent policies is smaller then the actual difference between the true values of the policies, then the value error for the second policy will be less than the error in approximating the second policy by the value of the first. The authors claim that this theorem shows that we can get a benefit from using value functions that depend directly on the policy. However, it's clear that such a benefit is possible if the policy dependent value function happens to generalize well even without this theorem, and it's not really clear to me what this theorem adds. Again the hard part is actually ensuring such favourable generalization occurs.\n\nOn the empirical side the results for learning the value of multiple pertrained policies are interesting and seem to demonstrate the possibility of improving generalization to new policies using PeVFA. The proposed algorithm for reinforcement learning seems to rely on growing a buffer containing trajectory data for all past policies and repeatedly retraining on it, which seems fairly restrictive in practice. Nonetheless, it is perhaps a reasonable way to start in order to highlight the potential utility of PeVFA. The results seem to demonstrate the PeVFA can provide an advantage over standard VFA in the settings explored, however aspects of the empirical evaluation could be improved and clarified to make the conclusion more clear.\n\nAccording to the appendix, the added hyperparameters for PPO-PeVFA were tuned using a grid search, but the hyperparameters of the PPO baseline were not which perhaps confers an unfair advantage to PPO-PeVFA. Furthermore, I couldn't find any details on how the shared hyperparameters were selected this should be stated clearly to aid in interpreting the results.\n\n\nMinor Corrections and Suggestions\n=================================\nLine 34 'efficiency value approximation'->efficiency of value approximation'\n\nLemma 1: Is the assumption $f(\\pi)_1\\leq f(\\pi_2)$ really necessary here? It seems a little unnatural and it seems to me you could drop it and modify the result to $\\hat{f}(\\pi_2)\\leq\\gamma f(\\pi_2)+\\gamma \\mathcal{M}(\\pi_2,\\pi_1,\\hat{L})+\\mathcal{M}(\\pi_1,\\pi_2,\\hat{L})$\n\nFor proofs in the appendix: it would help readability if you rewrite the theorems/lemmas before the proof.\n\n\nQuestions for authors\n=====================\nLine 313: What is meant by Ran PR? I can't find any details on this and I don't really know what is meant by \"randomly generated policy representation\". Randomized how exactly?\n\nIt seems quite odd to me that OPR uses permutation invariant transformations. Surely the ordering of the weights and biases is important for a multilayer network, what is the justification for processing it in this way?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are reasonably well covered. I don't see significant potential for negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "YK9G3PY-Lp", "reviewer": "Reviewer_KgSY", "summary": "Update: See the comments. Review score remains at 7.\n\nThe paper considers learning value function approximators that take a policy representation as an input to the value function to allow generalization across different policies. There are several other recent works of this type. The aspect differentiating the current work is the type of policy representation used that allowed the method to scale to policies with 10^5 parameters.\n\nThe paper considered two types of data sources for the policy representations:\n1. State action pairs obtained by the policy.\n2. The policy parameters.\n\nThen, the policy representation is formed by passing the data through a multilayer neural network that embeds the data into a lower dimensional representation.\nThe embedding is fed into the value function as an input, and the network creating the embedding is trained end-to-end to minimize the value function prediction error. Optionally they also added auxiliary losses such as contrastive learning.\n\nExperiments were performed using PPO combined with their new value function and compared to the standard PPO as well as variants taking a random policy representation and also a policy representation with no emebedding. They performed experiments on 6 MuJoCo tasks, and generally the new methods outperformed the baselines.\n\ntSNE plots showed that the policies with similar representations had a similar performance at the task (when comparing the achieved rewards). Moreover, experiments showed that the value prediction error was smaller for the new method compared to a standard value function method, and also that the value function was accurate for a test set of new policies not encountered during training.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper was reasonably well written. The experimental work examined the performance from multiple aspects and showed that their method could deal with the issue of the large parameter space of the policy by embedding it to a smaller space.\n\nI have a question about the state action pair embedding: When you are applying the value function to a new policy, do you first need to gather new data to perform the embedding to compute the value?\n\nOne down-side of the work is that it is not compared to the other recent works doing a similar thing, e.g. \nJ. Harb, T. Schaul, D. Precup, and P. Bacon. \"Policy evaluation networks\"\nor\nF. Faccio and J. Schmidhuber. Parameter-based value functions.\n\nAlthough, these works are quite recent, so maybe it's not expected.\nThe paper could also benefit from fixing some English language errors, but it's not a big issue.\nI would also suggest changing the title: I'm particular confused by why the \"Represent your own policies\" is added in the beginning. Doesn't everyone decide their own representations?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No concern.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "M39TWQyhuCC", "reviewer": "Reviewer_aYpG", "summary": "This paper studies value function approximators that explicitly take the policy as input, which the authors named policy-extended value function approximator (PeVFA). This is different from the common value function approximators which implicitly represent the value of the current policy. The authors discuss two scenarios where PeVFA could provide potential benefits via generalization: global generalization where the policies are sampled from a fixed distribution and local generalization where the policies are sampled from the policy improvement path. The latter is a typical scenario for policy improvement. The authors provide a theoretical analysis of the generalization properties of PeVFA. To apply PeVFA in practice, the authors also investigate several different ways of learning a compact embedding for policies: end-to-end learning, auxiliary loss of policy recovery, and contrastive learning. Empirical study shows that PeVFA improves over the conventional value function approximators on the Mujoco continuous control benchmark. The empirical results also provide some insight into the effectiveness of different policy embedding learning methods.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Things to like:\n* 1. Overall the paper is written in a coherent manner. The motivation is clear. The discussion on how to learn compact embeddings of policies addresses a critical issue for applying PeVFA in practice.\n* 2. The idea of taking the policy as an input to the value function approximator is interesting.\n\nThings that can be improved:\n* 1. I find Theorem 1 not particularly insightful - it basically says \"when PeVFA generalizes well, it helps\". Maybe the authors can make a more detailed comment on Theorem 1 in their response to help me understand its insight.\n* 2. While the local generalization scenario is theoretically possible, it is unclear to me how it would work in practice when the PeVFA is implemented by standard deep neural networks. Intuitively, the policies along the policy improvement path have increasing values. Therefore the PeVFA must \"extrapolate\" to generalize well. However, it is known that standard deep neural networks are not very good at extrapolating without specialized architecture design. In the experiments, the PeVFAs were implemented by standard MLPs, which are not expected to generalize well. Yet they still demonstrated performance improvement. It is possible that my hypothetical argument is wrong, but I would find the empirical results more convincing if the authors can address this question.\n* 3. I find the choice of implementing PeVFA with PPO rather intriguing. Based on Theorem 1, the larger the value difference between consecutive policies during policy iteration, the better the chance PeVFA could offer benefits via generalization. However, like other trust region policy optimization methods, PPO puts a constraint on consecutive policies so that the updated policy cannot deviate too far from the old policy. Intuitively PPO introduces an upper bound on the room for the improvement of PeVFA. Perhaps the authors could provide some explanation on why choosing PPO as the base agent.\n* 4. It is unclear to me how the proposed policy embedding learning methods would scale up to larger policy networks. One possibility is to separate the policy network into a state embedding module and a policy head and only encoding the policy head which is usually smaller. Could the authors comment on how to scale up and perhaps provide some preliminary results?\n* 5. L231-232, \"... PeVFA can achieve lower approximation error than conventional VFA after the same number of training with the same on-policy samples\". What does \"the same number of training\" mean here? Intuitively, PeVFA needs more training, though not necessarily more data, to outperform the conventional VFA because it is learning a more complicated function.\n\nMinor errors and typos:\n* 1. L301, delete \"to\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This work is not directly related to any real-world applications thus is far from making any negative societal impact at the current stage.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agree that the paper tackles an important problem and that the proposed approach has its merits. They also agree that the paper is clear and easy to follow for the most part. The main concern expressed in the reviews and discussions regards the theory, which was considered rather trivial and somewhat disconnected from the rest of the paper. The overall opinion is that the main result builds on fairly strong assumptions that render them somewhat vacuous:  it is intuitively obvious that if the proposed approximator generalizes well across policies this should be useful. The real challenge lies in how to enforce this property in the first place.
 
Given the above, the reviewers suggested during the discussion that the authors remove the theory and present the paper as essentially an empirical argument. Although the authors initially resisted the idea, eventually they agreed to adopt the suggestion. There was some discussion whether the paper without the theory should be accepted or not, as all the reviewers seem to agree that the empirical part of the paper is reasonably well executed and has some interesting insights in it. 
 
We eventually decided to reject the submission for two main reasons. First, the removal of the theory would be a fairly extensive modification to the paper. Second, and more important, it is unclear how much of a contribution the resulting empirical study would be with respect to the literature. The idea of adding an extra dimension to the value function corresponding to policies is quite straightforward and has appeared in the literature multiple times in different forms. Since the idea is already out there, the main challenge that remains to be resolved is how to represent these policies in a way that clearly promotes generalization across policies –especially those policies along the value-improvement path. 
 
This work can be seen as a promising first step towards the goal above, but more work must be done in this direction. We hope the feedback provided can help in defining the next steps of this research.

Decision: Reject