General Policy Evaluation and Improvement by Learning to Identify Few But Crucial States
Authors: Francesco Faccio
Aditya Ramesh
Vincent Herrmann
Jean Harb
Jürgen Schmidhuber
Venue: Submitted to ICLR 2023 (2023)
Keywords: Reinforcement Learning
Off-Policy Reinforcement Learning

Abstract: Learning to evaluate and improve policies is a core problem of Reinforcement Learning (RL). Traditional RL algorithms learn a value function defined for a single policy. A recently explored competitive alternative is to learn a single value function for many policies. Here we combine the actor-critic architecture of Parameter-Based Value Functions and the policy embedding of Policy Evaluation Networks to learn a single value function for evaluating (and thus helping to improve) any policy represented by a deep neural network (NN). The method yields competitive experimental results. In continuous control problems with infinitely many states, our value function minimizes its prediction error by simultaneously learning a small set of  `probing states' and a mapping from actions produced in probing states to the policy's return. The method extracts crucial abstract knowledge about the environment in form of very few states sufficient to fully specify the behavior of many policies. A policy improves solely by changing actions in probing states, following the gradient of the value function's predictions. Surprisingly, it is possible to clone the behavior of a near-optimal policy in Swimmer-v3 and Hopper-v3 environments only by knowing how to act in 3 and 5 such learned states, respectively. Remarkably, our value function trained to evaluate NN policies is also invariant to changes of the policy architecture: we show that it allows for zero-shot learning of linear policies competitive with the best policy seen during training.

Under review as a conference paper at ICLR 2023
GENERAL POLICY EVALUATION AND IMPROVEMENT BY LEARNING TO IDENTIFY FEW BUT CRUCIAL STATES
Anonymous authors Paper under double-blind review
ABSTRACT
Learning to evaluate and improve policies is core problem of Reinforcement Learning (RL). Traditional RL algorithms learn a value function defined for a single policy. A recently explored competitive alternative to learn a single value function for many policies. Here we combine the actor-critic architecture of Parameter-Bass Value Functions and the policy embedding of Policy Evaluation Networks to learn single map from policy parameters to expected return that evaluates (and thus helps to improve) any policy represented by deep neural network (NN). The method yields competitive experimental results. In continuous control problems with infinitely many states, our value function minimizes its prediction error by simultancously learning small set of 'probing states' and a mapping from actions produced in probing states to the policy's return. The method extracts crucial abstract knowledge about the environment in form of very few states sufficient to fully specify the behavior of many policies. A policy improves solely by changing actions in probing states, following the gradient of the value function's 's predictions. Surprisingly. it is possible to clone the behavior of near-optimal policy Swimmer-v3 and H3pper-v3 environments only by knowing how to act in and such learned states, respectively. Remarkably, our value function trained to evaluate NN policies is also invariant to changes of the policy architecture: we show that it allows for zero-shot learning of linear policies competitive with the best policy seen during training.
INTRODUCTION
Policy Evaluation and Policy Improvement are arguably the most studied problems in Reinforcement Learning. They are at the root of actor-critic methods (Konda and Tsitsiklis [000T | [uttoon 1984 Peters and Schaal 2008), which alternate between these two steps iteratively estimate the performance of a policy and using this estimate to learn a better policy. the last few years, they received a lot of attention because they have proven be effective in visual domains (Mnih et al.) 2016 Wu et al.
2017), continuous control problems (Lillicrap et al.) 2013: Haarnoja et al.) 2018 Fujimoto et al.
2018) and applications such roboties (Kober et al. (013). Several ways to estimate value functions
have been proposed, ranging from Monte Carlo approaches, to Temporal Difference methods (Sutton
1984). including the challenging off-policy scenario where the value of policy is estimated without observing its behavior (Precup et al. (001). A limiting feature of value functions is that they are defined for single policy. When the policy is updated, they need keep track of it, potentially losing useful information about old policies. By doing so, value functions typically do not capture any structure over the policy parameter space. While off-policy methods learn single value function using data from different policies, they have no specific mechanism to generalize across policies and usually suffer for large variance (Cortes al. 2010)).
Parameter Based Value Functions (PBVFs) et al., 2021) are promising approach to design value functions that overcome this limitation and generalize over multiple policies. A crucial problem in the application of such value functions is choosing a suitable representation of the policy. Flattening the policy parameters as done in vanilla PBVFs is difficult to scale to larger policies. Here we present an approach that connects PBVFs and policy embedding method called "fingerprint mechanism" by Harb et al. (2020). Using policy fingerprinting allows us to scale PBVFs to handle larger NN policies and also achieve invariance with respect to the policy architecture. Policy fingerprinting was

---

Under review as a conference paper at ICLR 2023
introduced to learn maps from policy parameters to expected return offline and prior to this work was never applied to the online RL setting.
We show in visual classification tasks and in continuous control problems that our approach can identify a small number of critical "probing states" that are highly informative of the policies performance. Our learned value function generalizes across many NN-based policies. It combines the behavior of many bad policies to learn a better policy, and is able to zero-shot learn policies with different architecture. We compare our approach with strong baselines in continuous control tasks: our method is competitive with DDPG (Liliccrpp et al. 2015) and evolutionary approaches.
2 BACKGROUND
We consider an agent interacting with a Markov Decision Process (MDP) Stratonovich (1960); Puterman 20114) M AS P, R no). The state space S C IR and the action space A C I are assumed to be compact sub-spaces. In the MDP framework, at each time-step 4, the agent observes state 8 E 8, chooses action dz E A, transitions into state 8441 with probability P(s Qt), and receives a reward r = R(s,, az). The initial state is chosen with probability S + AA: a function assigning for each
(o(s). The agent's behavior is represented by its policy state probability distribution over the action space. A policy is deterministic when for each state there exists an action a such that a is selected with probability one. Here we consider parametrized policies of the form Ta, where 0 E θ C I are the policy parameters, The return RG is defined as the cumulative discounted reward from time-step e.g. R. = Σk=o 7 [8++++++ 0++k++), where E (0, 1] is the discount factor. The agent's 's performance is measured by the expected return (i.e. the cumulative expected discounted reward) from the initial state: (Ø) = Ex, [Ro]. The state-value function VT (s) Ex, [Ry]s = of a policy Tо is defined as the expected return for being in state 8 and following Ty- Similarly, the action-value function ( a) = E,, [ = 8, a = a) of policy Te is defined as the expected return for being in state 8, taking action a and then following State and action value functions are related by V (s) = [A ma [a) (a)s) x (s, a) da. The expected return can be expressed in terms of the state and action value functions by integration over the initial state distribution:
((0) S 1 Ho(s)V (s) foddd=s-n (HO(S) / To(als) a) da ds.
(1)
The goal of a RL agent is to find the policy parameters 0 that maximize the expected return. Instead of learning single value function for target policy, here we try estimate the value function of any policy and maximize it over the set of initial states.
3 GENERAL POLICY EVALUATION
Recent work focused on extending value functions to allow them to receive the policy parameters as input. This can potentially result in single value functions defined for any policy and methods that can perform direct search in the policy parameters. We begin by extending the state-value function, and define the parameter- based state-value function (PSVF) (Faccio et al., 202T) as the expected return for being in state s and following policy no: ((s, 0) E[Re
s. 0). Using this new definition, we can rewrite the RL objective as [()) = J Mo (o (s, 0) ds. Instead of learning (ss 0) for each state, we focus here on the policy evaluation problem over the set of the initial states of the agent. This is equivalent to trying to model J(O) directly as a differentiable function V(0), which is the expectation of V (s,, 0) over the initial states:
v(+)= == E,- (Vs
= (V po(s) (s J(To).
(2)
V(O) is parameter-bal based start-state value function (PSSVF). We consider the undiscounted case in our setting, so 7 is set to throughout the paper. Once (00) is learned, direct policy search can be performed by following the gradient V (0) to update the policy parameters. This learning procedure can naturally be implemented in the actor-critic framework, where a critic value function the -iterraiiiy uses the collected data to evaluate the policies seen far, and the actor follows the critic's 's direction of improvement to update itself. As in vanilla PSSVF, we inject noise in the policy parameters for exploration. The PSSVF actor-critic framework is reported in Algorithm
2

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Actor critic with PSSVF for V(0)
Input: Differentiable critic V. : θ - R with parameters w; deterministic or stochastic actor m with parameters θ; empty replay buffer D
Output Learned V = V(0)v0, learned Ta = Tg.
Initialize critic and actor weights w, repeat:
Perturb policy: 6' = 0 + e, with Cm o(0,
Generate an episode s0, ao, T1, S1, 01, r2, 8T-1, 4T-1; with policy w
Compute return r = Σk=1
Store (0',r) in the replay buffer D for many steps do:
Sample batch 1 (1r 0'} from D
Update critic by stochastic gradient descent: V B B. (()) end for for many steps do: Update actor by gradient ascent: Vo (O) end for until convergence
Policy fingerprinting (Harb et al., 2020) While the algorithm descrihed above is straightforward and easy to implement, feeding the policy parameters as inputs to the value function remains a challenge. Recently Harb al. (2020) showed that a form of policy embedding can be suitable for this task. Their policy fingerprinting creates lower-dimensional policy representation. It learns a set of K 'probing states' { and an evaluation function U'like the PSSVF. To evaluate a policy To. they first compute the 'probing actions' āe that the policy produces in the probing states. Then the concatenated vector of these actions is given as input U IR K xnd + R. While the learned probing states remain fixed when evaluating multiple policies, the probing actions in such states depend on the policy we are evaluating. The parameters of the value function V are the probing states AND the weights of the MLP Ug that maps the 'probing actions' to the return. When the policy ne deterministic, the probing actions for such policy are the deterministic actions {āz = فe (3x) produced in the probing states
This mechanism has an intuitive interpretation: to evaluate the behavior of an agent, the PSSVF with policy fingerprinting learns set of situations (or states), observes how the agent acts in those situations, and then maps the agent's ' actions to a score. Arguably, this is also how a teacher would evaluate multiple different students by simultaneously learning which questions to ask the students and how to score the student's answers.
Therefore the parameters of the value function (probing states and evaluator function) can be learned by minimizing MSE loss Lv between the prediction of the value function and the observed return. Setting = {Ø, s 5 we retrieve the common notation of (O) for the PSSVF with finger- print mechanism. Given a batch B of data (To.r) E B, the value function optimization problem is:
min Lv = min E (sar))E
(V) (0)) min $..1.... (B
E
[[C,(((((s))......xxx))) (T(k))))) (3)
If the prediction of the value function accurate, policy improvement can be achieved by changing the way a policy acts in the learned probing states in order to maximize the prediction of the value function, like in the original PSSVF.
This process connects to the same interpretation as before: a student (the policy) observes which questions the teacher asks and how the teacher evaluates the student's answers, and subsequently tries to improve in such way to maximize the score predicted by the teacher. This iterative method is depicted in Figure T Note that poorithn[[apppie. directly to this setting. The only distinction is that the probing states part of the learned value function. Throughout this work, with the exception of the MNIST experiments, we consider deterministic policies.
If the policy is stochastic, the prohing actions are the parameters of the output distribution of the policy in such states (the vector of prohahility distribution if the action space is discrete)
3

---

Under review as a conference paper at ICLR 2023
MSE
Lv
Environment
concat.
U
w (0)
T 0
Probing States
Probing
Actions
Figure I: General policy evaluation aims to evaluate any given policy's return based on the policy's actions (referred to as probing actions) the learned probing states. The policy can improved through maximising the prediction of the learned value function via gradient ascent.
4 EXPERIMENTS
This section presents an empirical study of parameter-based value functions (PBVFs) with finger- printing. We begin with demonstration that fingerprinting can learn interesting states in MNIST purely through the designated evaluation task of mapping randomly initialized Convolutional Neural Networks (CNNs) to their expected loss. We also show that such a procedure could be used to con- struct a value function for offline improvement MNIST. Next, we proceed to our main experiments on continuous control tasks in MuloCo (Todorov et al. (2012). Here we show that our approach is competitive with strong baselines like DDPG (Lillierap al. 2013) and ARS (Mania al.) 2018), while it lacks sample efficiency when compared to SAC (Haamojj al. 2018). A strength of our approach is invariance to policy architecture. To illustrate this, we provide results on zero-shot learning of new policy architectures. Thereafter, we present detailed analysis of the learned probing states in various MuJoCo environments. We conclude our study with the surprising observation that very few probing states required clone near-optimal behaviour in certain MuloCo environments. An open-source implementation of our code is provided as supplementary material.
4.1 MOTIVATING EXPERIMENTS ON MNIST
We begin our experimental section with intuitive demonstration of how PBVFs with fingerprinting work, using the MNIST digit classification problem. The policy is CNN, mapping images to probability distribution over digit classes. The environment simulation consists of running a forward pass of the CNN on a batch of data and receiving the reward, which in this case is the negative cross-entropy between the output of the CNN and the labels of the data. The value function learns to map CNN parameters to the reward (the negative loss) obtained during the simulation. Then CNN learns to improve itself only by following the prediction of the value function, without access to the supervised learning loss. These MNIST experiments can be considered as contextual bandit problem, where the initial state (or context) is given by the batch training data sampled and there transition dynamics. We start with randomly initialized CNN and value function and iteratively update them following Algorithm T Using only 10 probing states, we obtain a test set accuracy of 82.5%. When increasing the number of probing states to 50, the accuracy increases to 87%.
Visualization of probing states Figure 2 shows some of the probing states learned by our model, starting from random noise. During learning, we observe the appearance of various digits (sometimes the same digit in different shapes). Since probing states states in which the action of the policy informative about its global behavior, it is intuitive that digits should appear. We emphasize that both the CNNs and the value function are starting from random initializations. The convolutional filters and the probing states are learned using Algorithm|] [ without access to the supervised loss. For more complex datasets like CIFAR 10 method found it difficult to learn meaningful probing states.
4

---

Under review as a conference paper at ICLR 2023
This is possibly due the high variance in the training data given a specific class and highlights limitation of our method.
6
2
3
Figure 2: Samples of probing states learned while training Algorithm I on MNIST.
Offline policy improvement Using this setting, we perform another experiment. We collect one offline dataset ( J.N of N randomly initialized CNN policies and their losses. We constrain the maximum accuracy of these CNNs in the training set to 12%. We then use the dataset to train value function offline. After training, we randomly initialize a new CNN and take many steps of gradient ascent through the fixed value function, obtaining a final CNN whose accuracy is around 65% on the test set. Our experiments show that our value function can combine the behavior of many bad NNs to produce a much better NN in a zero shot manner. We found that also with randomly initialized policies some digits appear probing states, although they are less evident than in the online framework. We include learning curves and probing states for this scenario in Appendix B.1
4.2 MAIN EXPERIMENTS ON MuJoCo
Here we present main evaluation selected continuous control problems from MuloCo (Todorov iet al. 2012). Since our algorithm performs direct search in parameter space, we choose Augmented Random Search (ARS) (Mania al. 2018) as baseline for comparison. Moreover, since our algorithm employs deterministic policies, off off-policy data, and an actor-critic architecture, a natural competitor is the Deep Deterministic Policy Gradient (DDPG) algorithm (Lillicrap et al.) 2015). strong baseline for continuous control. We also compare our method with the ttate-of-the-art Soft Actor- (SAC) (Haarnoja al. 2018).
Implementation details For the policy architecture, we use an MLP with 2 hidden layers and 256 neurons for each layer. We use 200 probing states and later provide analysis of them. Our implementation is based on the public code for Parameter-Bant environments like Hopper and Walker, a bad agent can fail and the episode ends after very few time steps. This results in an excessive number of bad policies in the replay buffer, which can bias learning. Indeed, by the time a good policy is observed, it becomes difficult to use it for training when uniformly sampling experience from the replay buffer. We find that by prioritizing more recent data we are able to achieve a more uniform distribution over the buffer and increase the sample efficiency, We provide an ablation in Appendix B.2 showing the contribution of this component and of policy fingerprinting. Like in the original ARS and PBVF papers (Mania et al.) 2018 Faccio et al. (02T). use observation normalization and remove the survival bonus for the reward. The survival bonus, which provides reward 1 at each time step for remaining alive in Hopper, Walker and Ant, induces a challenging local optimum in parameter space where the agent would learn to keep still.
Value Functions. In some MuJoCo
For DDPG and SAC, we use the default hyperparameters, yielding results on par with the best reported results for the method. For ARS, tune for each enviromment step size, number of population and noise. For our method, we use a fixed set of hyperparameters, with the only exception Ant. In Ant, observe that setting the parameter noise for perturbations 0.05 results in very rare positive
5

---

Under review as a conference paper at ICLR 2023
returns for ARS and PSSVF (after subtracting the survival bonus). Therefore less noise for this environment. We discuss implementation details and hyperparameters in Appendix A
Amt-
HrIECheeisbba
Swotmmer-e3
5000
12530
pood
10000
3r5
1000
7500
ddpg
5930
200
2MM
0,000
0.50M
1.M
2.00
22
U.50M
18M
SOM
Wa
Hopperv33
InertedDouBePennuumm
4.000
4030
10333
4000
2000
1900
2002
0.00M
0boM
$1e sagps
1999 18095
Figure 3: Return as a function of the environment interactions. The solid curve represents the mean (across 20 runs), and the shaded region represents a 95% bootstrapped confidence interval.
Results Figure 3 shows learning curves in terms of expected return (mean and 95% confidence interval) achieved by our algorithm and the baselines across time in the environments. Our algorithm very competitive with DDPG and ARS. It outperforms DDPG in all environments with the exception of HalfCheetah and Walker, and displays faster initial learning than ARS. In the Swimmer environment, DDPG and SAC fails to learn an optimal policy due to the problem of discounting" On the other hand, in HalfCheetah, parameter-based methods take a long time to improve, and the ability of DDPG to give credit to sub-episodes is crucial here learn quickly. Furthermore, the variance of our method's performance is less than DDPG's and comparable to ARS's. Like evolutionary approaches, our method uses only the return as learning data, while ignoring what happens in each state-action pairs. This is limitation of our method and it is evident how PSSVF and ARS are less sample efficient in comparison to SAC in many environments.
In preliminary experiments we tried to learn also function V(so, 0), incorporating the information on the initial state. In practice, we can store the buffer tuples (su. 0, r) consisting of initial state, policy parameters and episodic return. When training the PSSVF (now similar to the PSVF), concatenate the initial state to the probing actions and map the vector of probing actions and initial state to the return. Then policy improvement is achieved by finding the policy parameters that maximize the value function's 's prediction taking an expectation over the initial states sampled from the buffer. The results were very similar those we presented in this section, so we decided to use the more straightforward approach that ignores the initial state and directly maps policy parameters to expected return. It would be also possible to learn a general value function Vss, f) for any state, like in the PSVF algorithm (Faccio et al.) 2021). We leave this as future work.
Comparison to vanilla PSSVF A direct comparison to the standard Saaameter-Basee Value function is unfeasible for large NNs. This is because in the vanilla PSSVE flattened policy parameters directly fed to the value function. In our policy configuration, the flattened vector of policy parameters contains about 70K elements, which is significantly more than 200 x n elements used represent policies with fingerprinting. Nevertheless, we provide direct comparison between the two approaches using a smaller policy architecture which consists of an MLP with 2 hidden layers and 64 neurons per layer. The complete results are provided in Appendix B.2 Our results in this setting show that the fingerprint mechanism could be useful even for smaller policies.
"This is common problem for Temporal Difference methods: the policy optimizing expected return in Swimmer with = 0.99 is sub-optimal when considering the expected return with ? 1. See the ablation in Appendix A.3.1 of (Faccio et al. 2021

---

Under review as a conference paper at ICLR 2023
4.3 ZERO- SHOT LEARNING OF NEW POLICY ARCHITECTURES
Here we show that our method can generalize across policy architectures. We train PSSVF using NN policies as in the main experiments. Then we randomly initialize linear policy and start taking gradient ascent steps through the fixed value function, finding the parameters of the policy that maximizes the value function's 's prediction. In Figure we observe that near-optimal linear policy can be -oot--ee--ee through the value function even if it was trained using policies with different architecture, It achieves an expected return of 345, while the return of best NN used for training was 360. Figure 12 (Appendix) shows results for zero-shot learning deep policies in Swimmer. We notice more variance in the performance, which might be caused by the deep policy overfitting more easily probing-state/probie aacttpiioobin-attiinn pairs during the policy improvement phase.
4.4 FINGERPRINT ANALYSIS
Ablation on number of probing states Our experiments show that learning probing states helps evaluating the performance of many policies, but how many of such probing states are necessary for learning? We run our main experiments again, with fewer probing states, and discover that in many environments, very small number of states is enough to achieve good performance. In particular, we find that the PSSVF with probing states achieves 114 and 2790 final return in Swimmer and Hopper respectively, while Walker needs at least 50 probing states to obtain a return above 2000. In general, 200 probing states represent a good trade-off between learning stability and computational complexity. We compare the performances of PSSVF versions with varying numbers of probing states. We use the same hyperparameters as in the main experiments (see appendix[A.2] apart for the number of probing states. Figure 4 shows that in Hopper and Swimmer 10 probing states are sufficient to learn a good policy, while Walker needs a larger number of probing states to provide stability in learning.
Walker2dw3
Hoppar-v3
9wimer+33
3000
3000
302
200
10
1000
3000
192
50
200
0.504
1,00
1.RMM
2/00
2.50M R0W
0.000
1,00
2.00M
2.50M
3.00
0.000
0.GOM
1.CMM
1.91M
2.00M
2.00M 3.000
Figure 4: Average return of PSSVF with different number of probing states as a function of the number of time steps in the environment. The solid line is the average over 10 independent runs; the shading indicates 95% bootstrapped confidence intervals.
The most surprising result is that randomly initialized policy can learn ncar-optimal behaviors in Swimmer and Hopper by knowing how to act only in 3 (5) such crucial learned states (out of infinitely many in the continuous state space). To verify this, manually select of the 5 learned probing states in Swimmer, and compute the actions of an optimal policy in such states. Then train a new, randomly initialized policy, to just fit these 3 data points minimizing MSE loss. After many gradient steps, the policy obtains a return of 355, compared to the return of 364 of the optimal policy that was used to compute such actions. Figure 15 (in Appendix B.3) includes a detailed analysis of this experiment. The probing actions are the vectors !-0.97, 0.866, 1-0.18, -0.99), (0.86, 0.68). In the plot we notice that when the agent's state is close to the first probing state (bottom plot, depicted in blue), then both components of the actions are close to -1, like the probing action in such state. When the agent's s state is close to the second state (bottom plot, depicted in orange), the first component
200
8.0
3000
Figure 5: Performance of a linear pol- icy (in blue) zero-shot learned (aver- aged over 5 runs, 95% bootstrapped CI). The orange line shows the best perfor- mance of the deep NN when training the PSSVE.
7

---

Under review as a conference paper at ICLR 2023
of the action moves from -1 to 0 (and then to +1) in smooth way, while the second component jumps directly to +1. This behavior is consistent with the second probing action, since the second component is more negative than the first. Notably, although the distance between the agent's 's state and the third probing state (bottom plot, depicted in green) is never close to zero, such a probing state is crucial: it induces the agent take positive actions whenever the other probing states are far away. We observe similar behavior for other environments, although they need more of such states to encode the behavior of an optimal policy. Using similar procedure, we are able train a randomly initialized policy in Hopper achieving 2200 return, using only 5 state-action pairs. We provide a detailed discussion and learning curves for this task in Appendix/BA
Visualization of RL probing states It is possible to visualize the probing states learned by the PSSVF. To understand the behaviour in probing states, we initialize the MuJoCo environment to the learned probing state (when possible) and let it evolve for a few time steps while performing no action. In Appendix B.3 we show the crucial learned probing states of our previous experiment. Additional probing states for all environments can be seen in animated form on the website https cannonyyouus20022.. gi 1o/.
RELATED WORK
There a long history of RL algorithms performing direct search in parameter space or policy space. The most common approaches include evolution strategies, e.g., (Rechenberg 1971, Sehnke et al. 2010, 2008 WWerstra et al. 2014; [Salimans et al.) 2017). They iteratively simulate a population of policies and use the result to estimate a direction of improvement in parameter space. Evolution strategies, however, don't 't reuse data: the information contained the population is lost as soon as an update is performed, making them sumple-inefficient Several attempts have been made to reuse past data, often involving importance sampling (IS) (Zhao et al. (203),, but these methods suffer from high variance of the fitness estimator (Metelli al. 2 Our method directly estimates fitness for each policy observed in the history and makes efficient reuse of past data without involving IS.
Direct search can be facilitated by compressed network search (Koutnik al. 2010 and algorithms that distill the knowledge of an NN into another NN (Schmidhuber 1992). Closely related to our fingerprint embedding is also the concept of Dataset Distillation (Wang al. 2018). However, in our RL setting, learning to distill crucial states from an environment is harder due to the non- differentiability of the environment. Estimating a global objective function is common in control theory, where usually a gaussian process is maintained over the policy parameters. This allows to perform direct policy optimization during the parameter search. Such approaches are often used in the Bayesian optimization framework (Snoek et al. [213]] (012)) where a tractable posterior over the parameter space used drive policy improvements. Despite the soundness of these approaches, they usually employ very small control policies and scale badly with the dimension of the policy parameters. Our method, however, is invariant to policy parametrization.
It is based on recent class of algorithms that were developed to address global estimation and improvement of policies. For Policy Evaluation Networks (PVNs) (Harb et al. (020). an actor-critic algorithm for offline learning through policy fingerprinting was proposed. PVNs focus on the offline RL setting. In PVNs, first a dataset of randomly initialized policies with their returns is collected. Then, once their V(O) with policy fingerprinting is trained, they perform policy improvement through gradient ascent steps on V. Their experimental setting is similar to our MNIST offline demonstration, which we provide just to give an intuition on how policy fingerprinting works. Concurrently, Parameter-Based Value Functions were developed to provide single value functions able to evaluate any policy, given a state, state-action pair, or a distribution over the agent's 's initial states. PBVFs did not use any dimensionality reductions techniques such as the policy fingerprinting mechanism, but demonstrated sample efficiency in the online RL scenario, directly using the flattened parameters of neural network as inputs. They exhibited zero-shot learning for linear policies, but failed when the policy parameters were high-ddmensional. Here, however, we demonstrated that PBVF's with policy fingerprinting mechanisms can be efficient in the online scenario. A minor difference between our approach and PVNs is that PVNs predict a discretized distribution of the return, whereas our approach simply predicts the expected return. Our method can be seen like online version of PVN without some of the tricks used, or like a version of PSSVF where policy fingerprinting is used. Fingerprinting itself is similar to technique for "learning think" (Schmidhuber) 20155 where one
8

---

Under review as a conference paper at ICLR 2023
NN learns to send queries (sequences of activation vectors) into another NN and learns to use the answers (sequences of activation vectors) to improve its performance.
Recent work (Tang al. 2020) learned Parameter- Basee State- Value Functions which, coupled with PPO, improved performance. The authors did not use the value function to directly backpropagate gradients through the policy parameters, but only exploited the general policy evaluation properties of the method. They also proposed two dimensionality reduction techniques. The first, called Surfcce Policy Representation, consists of learning a state-action embedding that encodes possible information from a policy тT. This requires feeding state-action pairs to a common MLP whose output is received as input the value function. The MLP is trained such that it allows for both low prediction error in the value function and low reconstruction error of the action, given a state and the embedding. This method is not differentiable in the policy parameters, therefore it cannot be used for gradient-based policy improvement. The second method, called Origin Policy Representation (OPR), consists of using an MLP that performs laser-wise extraction of features from policy parameters. OPR uses MLPs to take as input direcly the weight matrix of each layer. This approach is almost identical to directly feeding the policy parameters to the value function (they concatenate the state to the last layer of such MLP), and suffers from the curse of dimensionality. Also, OPR was not used to directly improve the policy parameters, but only to provide better policy evaluation.
Alternative strategies to represent policies have been studied in previous work. One such strategy aims to learn a representation function mapping trajectories to a policy embedding through an auto- encoding objective (Grover et al. 2018 Raileanu et al.) 2020). In particular, [rrver et al. (2018) use this idea to model the agent's 's behavior in a multi-agent setting. The approach presented by Raileanu et al. (2020) performs gradient ascent steps finding policy embedding that maximizes the value function's s predicted return. While this maximization through the value function is similar to our setting, relies on representation function (or policy decoder). Our method does not use a decoder and instead directly backpropagates the gradients into the policy parameters for policy improvement. Closer to our fingerprinting setup, [Pacchiano et al. (2020) utilize pairs of states and actions (from the corresponding policy) as policy representation. However, unlike in our approach, the probing states are not Iearned, but sampled from a chosen probing state distribution. Kanervisto et al. (2020) suggest representing policies based on visited states via Gaussian Mixture Models applied to an offline dataset of data from multiple policies. The authors mention that their current version of policy supervectors is intended for analysing policies and is not yet suitable for online optimization. Value functions conditioned on other quantities include vector- valued adaptive critics (Schiihhuber (1991), General Value Functions (Sutton et al.) 201T), and Universal Value Function Approximators (Schaul let al. 2015). Unlike our approach these methods typically generalize over achieving different goals, and are not used generalize across policies.
6 CONCLUSION AND FUTURE WORK
Our approach connects Parameter- Based Value Functions (PBVFs) and the fingerprinting mechanism of Policy Evaluation Networks. It can efficiently evaluate large Neural Networks, is suitable for off-policy data reuse, and competitive with existing baselines for online RL tasks. Zero-shot learning experiments MNIST and continuous control problems demonstrated our method's generalization capabilities. Our value function is invariant to policy architecture changes, and can extract essential knowledge about complex environment by learning small number of situations that are important to evaluate the success of a policy. A randomly initialized policy can learn optimal behaviors in Swimmer (Hopper) by knowing how to act only in 3 (5) such crucial learned states (out of infinitely many in the continuous state space). This suggests that some of the most commonly used RL benchmarks require learn only a few crucial state-action pairs. Our set of learned probing states is instead used to evaluate any policy, while in practice different policies may need different probing states for efficient evaluation. A natural direction for improving this method and scaling it to more complex tasks is to generate probing states in a more dynamic way, or learn to retrieve them directly from the agent's experience in the replay buffer. Like evolutionary approaches and trajectory based RL, our method might suffer high variance in stochastic environments or when the variance of the return over the initial state is high. In such scenario, poor value estimates might prevent policy improvement zero-shot learning. Finally, PBVFs are general framework that also considers value functions that receive states and state-action pairs as input. We plan to investigate how these value functions trained by Temporal Differences (Sutton 1988) behave with policy fingerprinting.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Achiam, J. (2018). Spinning Up in Deep Reinforcement Learning.
Cortes, C., Mansour, Y., and Mohri, M. (2010). Learning bounds for importance weighting. In Advances in neural information processing systems, pages 442-450.
Faccio, F. Kirsch, L., and Schmidhuber, J. (2021). Parameter-based value functions. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Fujimoto, S., Van Hoof, H., and Meger, D. (2018). Addressing function approximation enror in actor-critic methods. arXiv preprint 18X..11802094477
Grover, A., Al-Shedivat, M., Gupta, J., Burda, Y., and Edwards, H. (2018). Learning policy representations in multiagent systems. In International conference muchine learning, pages 1802-1811. PMLR.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, (2018). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. arXiv preprint arXiv:1801.0120
Harb, J., Schaul, T., Precup, D., and Bacon, P.-L. (2020). Policy evaluation networks. arXiv preprint arXiv:2002.11833
Kanervisto, A., Kinnunen, T., and Hautamäki, V. (2020). General characterization of agents by states they visit. arXiv preprint arXiv:2012.01224.
Kober, J., Bagnell, J. A., and Peters, (2013). Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 12(11))223811244
Konda, V. and Tsitsiklis, J. (2001). Actor-critic algorithms. Society for Industrial and Applied Mathematics, 42.
Koutnik, J., Gomez, F., and Schmidhuber, J. (2010). Evolving neural networks in compressed weight space. In Proceedings of the 12th annual conference on Genetic and evolutionary computation, pages 619-626.
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv preprint arXiv: 1509.02971911
Mania, H., Guy, A., and Recht, B. (2018). Simple random search of static linear policies is competitive for reinforcement learning. In Advances in Neural Information Processing Systems, pages 1800- 1809.
Metelli, A. M., Papini, M., Faccio, F., and Restelli, M. (2018). Policy optimization via importance sampling. In Advances Neural Information Processing Systems, pages 5442-5454.
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukeuoglu, K. (2016). Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928-1937. PMLR.
Pacchiano, A., Holder, J., Tang. Y., Choromanski, K., Choromanska, A., and Jordan, M. (2020). Learning to score behaviors for guided policy optimization. In International Conference on Machine Learning, pages 7445-7454. PMLR.
Peters, J. and Schaal, (2008). Natural actor-critic. Neurocomput. 71(7-9): 189-1190.
Precup, D., Sutton, R. S., and Dasgupta, S. (2001). Off-policy temporal difference learning with function approximation. In ICML.
Puterman, M. L. (2014). Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons.
Raileanu, R., Goldstein, M.. Szlam, A., and Fergus. R. (2020). Fast adaptation to environments via pynaiccs value functions. In Proceedings of the 37th International Conference on Machine Learning, pages 7920-79311
10

---

Under review as a conference paper at ICLR 2023
Rechenberg, I. (1971). Evolationssttattegie Optimierung technischer Systeme nach Prinzipien der biologischen Evolution. Dissertation. Published 1973 by Honmman-Holzbog..
Salimans, T., Ho, J., Chen, X., Sidor, S., and Sutskever, (2017). Evolution strategies as scalable alternative to reinforcement learning. arXiv preprint orXiv:/703.03866
Schaul, T., Horgan, D., Gregor, K., and Silver. D. (2015). Universal value function approximators. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning Volume 37, ICML' 15, pages 312-1320 JMLR.org.
Schmidhuber, (1991). Reinforcement learning Markovian and Mon-Markovian environments. In Lippman, D. S., Moody, J. E., and Touretzky, D. S., editors, Advances in Neural Information Processing Systems (NIPS 3), pages 500-506. Morgan Kaufmann.
Schmidhuber, J. (1992). Learning complex, extended sequences using the principle of history compression. Neural Computation, 4(2):234-242.
Schmidhuber, J. (2015). On learning think: Algorithmic information theory for novel com- binations of reinforcement learning controllers and recurrent neural world models. Preprint arXiv:1511.09249.
Sehnke, F., Osendorfer, C., Rückstieß, T., Graves, A., Peters, J., and Schmidhuber, J. (2008). Policy gradients with parameter-based exploration for control. In Kúrková, V., Neruda, R., and Koutník, J., editors, Artificial Neural Networks ICANN 2008, pages 387-396, Berlin, Heidelberg. Springer Berlin Heidelberg.
Sehnke, F., Osendorfer, C., Rückstieß, T., Graves, A., Peters, J., and Schmidhuber, J. (2010).
Parameter-exploring policy gradients. Neural Networks, 23(4):551-559.
Snoek, J., Larochelle, H., and Adams, R. P. (2012). Practical bayesian optimization of machine learning algorithms. In Advances in neural information processing systems, pages 2951-2959.
Snoek, J., Rippel, O., Swersky, K., Kiros, R., Satish, N., Sundaram, N., Patwary, M. M. A., Prabhat, P., and Adams, R. P. (2015). Scalable bayesian optimization using deep neural networks. In Proceedings of the 32nd International Conference on International Conference on Machine Learning Volume 37, ICML'15, page 2171-2180. JMLR.org.
Stratonovich, R. (1960). Conditional Markov processes. Theory of Probability And Its Applications, 5(2):156-178.
Sutton, R. S. (1984). Temporal Credit Assignment in Reinforcement Learning. PhD thesis, University of Massachusetts Amherst.
Sutton, R. S. (1988). Learning predict by the methods of temporal differences. Machine learning. 3(1):9 44.
Sutton, R. S., Modayil, J., Delp. M., Degris, T., Pilarski, P. M.. White, A., and Precup. D. (201 I). Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In The 10th International Conference Autonomous Agents and Multiagent Systems- Volume 2, AAMAS 111, pages 761-768, Richland, SC. International Foundation for Autonomous Agents and Multiagent Systems.
Tang, H., Meng, Z., Hao, J., Chen, C., Graves, D., Li, D., Yu, C., Mao, H., Liu, W., Yang, Y., et al. (2020). What about inputting policy in value function: Policy representation and policy-extended value function approximator. arXiv preprint arXiv:2010.09536.
Todorov, E., Erez, T., and Tassa, (2012). Mujoco: physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033.
Wang. T., Zhu, J.-Y., Torralba, A., and Efros, A. A. (2018). Dataset distillation. arXiv preprint arXiw:1811.10959
Wierstra, D., Schaul, T., Glasmachers, T., Sun, Y., Peters, J., and Schmidhuber, J. (2014). Natural evolution strategies. The Journal of Machine Learning Research, 1519).949-980.
11

---

Under review as a conference paper at ICLR 2023
Wu, Y., Mansimov, E., Grosse, R. B., Liao, S., and Ba, J. (2017). Scalable trust-region method for deep reinforcement learning using kronecker-factoon approximation. Advances in neural information processing systems, 30.
Zhao, T., Hachiya, H., Tangkaratt, V., Morimoto, J., and Sugiyama, M. (2013). Efficient sample reuse in policy gradients with parameter-based exploration. Neural computation, 2516)1152
12

---

Under review as a conference paper at ICLR 2023
A
IMPLEMENTATION DETAILS
A.1 MNIST IMPLEMENTATION
For our experiments with MNIST we adapt the official code for PSSVF CNN policies and the MNIST classification problem.
The policy consists of two convolutional layers with 4 and 8 output channels respectively, 3 kernels and a stride of 1. Each convolutional layer is followed by ReLU activations. The output from the convolutional layers flattened and provided to a fully connected linear layer which outputs the logits for the ten MNIST classes. The logits are fed into categorical distribution; the outputs are interpreted as class probabilities.
Policy architecture:
Value function architecture: MLP with 2 hidden layers and 64 neurons per layer with bias.
ReLU activations.
Batch size for computing the loss: 1024
Batch size for value function optimization: 4
Buffer size: 1000
Loss: Cross entropy
Initialization of probing states: uniformly random in [-0.5, 0.5)
Update frequency: every time new episode is collected
Number of policy updates:
Number of value function updates: 5
Learning rate policy: le-6
Learning rate value function: le-3
Noise for policy perturbation: 0.05
Priority sampling from replay buffer: True, with weights 1/20.8, where r the number of episodes since the data was stored in the buffer
Default PyTorch initialization for all networks.
Optimizer: Adam
A.2 RL IMPLEMENTATION
Here we report the hyperparameters used for PSSVF and the baselines. For PSSVE, we use the open source implementation provided by Faccio et al. (202T). For DDPG and SAC, we use the spinning-up RL implementation (Achiam 2018), whose results are on par with the best reported results. For ARS, we adapt the publicly available implementation (Mania al. 2018) Deep NN policies.
Shared hyperparameters:
Deterministic MLP with 2 hidden layers and 256 neurons per layer with bias. Tanh activations for PSSVF and ARS. ReLu activations for DDPG and SAC. The output layer has Tanh nonlinearity and bounds the action in the action-space limit.
Policy architecture:
Value function architecture: MLP with 2 hidden layers and 256 neurons per layer with bias. ReLU activations for PSSVF and DDPG and SAC.
Initialization for actors and critics: Default PyTorch initialization
Batch size: 128 for DDPG and SAC. 16 for PSSVF
Learning rate actor: 1e-3 for DDPG and SAC; 2e-6 for PSSVF
Learning rate critie: le-3 for DDPG and SAC, 5e-3 for PSSVF
Noise for exploration: 0.05 in parameter space for PSSVF: 0.1 action space for DDPG
Actor's frequency of updates: every episode for PSSVF; every 50 time steps for DDPG and SAC: every batch for ARS
13

---

Under review as a conference paper at ICLR 2023
Critic's frequency of updates: every episode for PSSVF; every 50 time steps for DDPG and SAC
Replay buffer size: 100k for DDPG and SAC; 10k for PSSVE
Optimizer: Adam for PSSVF and DDPG and SAC
Discount factor: 0.99 for DDPG and SAC; for PSSVF and ARS
Survival reward adjustment: True for ARS and PSSVF in Hopper, Walker, Ant; False for DDPG and SAC
Environmental interactions: 300k time steps in InvertedDoubbeeeeddllmm; 3M time steps in all other environments
Tuned hyperparameters:
Step size for ARS: tuned with values {le - 2, le - 3, le 4}
Number of directions and elite directions for ARS: tuned with values in 1], [8, 4), [8, 8], [32, 4], [32, 16), [64, 8], [64, 32]}, where the first element denotes the number of directions and the second element the number of elite directions
Noise for exploration ARS: tuned with values in 10.1, 0.05, 0.025)
Hyperparameters for specific algorithms:
PSSVF:
Number of probing states: 200
Initialization of probing states: uniformly random in [0,
Observation normalization: True
Number of policy updates: 5
Number of value function updates:
Priority sampling from replay buffer: True, with weights 1/x where is the number of episodes since the data was stored in the buffer
ARS:
Observation normalization: True
DDPG and SAC:
Observation normalization: False
Number of policy updates: 50
Number of value function updates: 50
Start-steps (random actions): 10000 time-steps
Update after (no training): 1000 time-steps
Polyak parameter: 0.995
Entropy parameter (SAC): 0.2
A.3 GPU USAGE/ COMPUTATION REQUIREMENTS
Each run of PSSVF in the main experiment takes around 2.5 hours on Tesla P100 GPU. We ran instances of our algorithm for each GPU, We estimate a total 75 node hours to reproduce our main RL. results (20 independent runs for 6 environments).
14

---

Under review as a conference paper at ICLR 2023
B EXPERIMENTAL DETAILS
B. MNIST EXPERIMENTS
Online learning through Algorithm We use PSSVF (Algorithm 1 with the hyperparameters described Appendix Figure shows the performance PSSVF using CNNs on MNIST with 10 and 50 probing states as a function of the number of interactions with the dataset. Each interaction consists of perturbing the current policy with random noise, computing the loss the perturbed policy a batch of data, storing the perturbed policy and its loss, and updating.
50 probing states
2.2
10 probing states
text accurssy
60-
train loss
80
50000
100000 150000
200000
50000
100000
150000
200000
Iterations
Figure 6: On the left: test aceuracy of PSSVF as function of the interactions with the dataset. On the right: loss of the perturbed CNN on the training set. Average over independent runs and 95% bootstrapped confidence interval.
Visualization of learned probing states We plot the evolution of some of the probing states, starting from random noise, until the PSSVF is learned. We consider experiment with 10 probing states and show how they change during learning. This is depicted in Figure where randomly initialized probing states slowly become similar digits.
of the previous
Offline policy improvement This section describes the offline MNIST experiment of the main paper. Here every iteration encompasses the following steps. We perturb randomly initialized CNN with gaussian noise with standard deviation 0.1. Then we compute the loss on a batch of 1024 training data. If the accuracy such batch is below 12% store the CNN and its loss, otherwise we discard the data. At every iteration we also train a PSSVF with 200 probing states, using the data collected (whose accuracy is most 12%). We repeat this for 90000 iterations. Then, randomly initialize new CNN and train it by taking gradient steps through the fixed PSSVF, without further seeing training data. In Figure we plot the performance of the zero-shot learned CNN. Surprisingly, it achieves test accuracy of 65%, although only CNNs with at most 12% accuracy are used in training. From the same figure we also observe that the prediction of the PSSVF quite accurate up to 80 gradient steps, after which the performance degrades. We use learning rate of le 3 for the CNN.
Visualization of learned probing states When training the PSSVF using CNNs whose accuracy is most 12%, we also observe the formation of "numbers" probing states, although they are not as evident as in the online setting. We provide some examples in Figure
B.2 MAIN EXPERIMENTS ON MuJoCo
To measure learning progress, we evaluate each algorithm for episodes every 10000 time steps. We use the learned policy for PSSVF and ARS and the deterministic actor (without action noise) for DDPG. We use 20 independent instances the same hyperparameter configuration for PSSVF and DDPG in all environments. When tuning ARS, we run 5 instances of the algorithm for each hyperparameter configuration. Then we select the best hyperparameter for each environment and сarry out further 20 independent runs. We report the best hyperparameters found for ARS in Table In addition to the learning curves of the main paper in Figure] we report the final return with standard deviation in Table
15

---

Under review as a conference paper at ICLR 2023
Figure 7: From left to right, the 10 probing states learned by the PSSVF using Algorithm column represents 12500 interactions.
Each
Ablation on weighted sampling In Figure [0 show the benefit of using non-uniform sampling from the replay buffer in Hopper and Walker environments. We compare uniform sampling (no weight) to non uniform sampling with weight 1 1/2*, where i E (1.0, 1.1]. and x is the number of episodes since the data stored in the buffer. We achieve the best results in Hopper and Walker for the choice of x = 1.1. It is interesting to take this into consideration when comparing our approach to vanilla PSSVE.
16

---

Under review as a conference paper at ICLR 2023
0.0023
test
value fenciion preditionn
1e11 uccuracy
0.0021
40-
0.0019
30
20
0.0017
150
75
200
gradient steps
gradieen steps
Figure 8: the left: test accuracy random initialized CNN zero-shot learned using a learned PSSVE On the right, the prediction of the performance of the CNN given by the PSSVF and the true performance on the test set. Average over independent runs and 95% bootstrapped c.i.
Figure 9: Samples of probing states learned by the PSSVF using CNNs with at most 12% training set accuracy.
Table 1: Best hyperparameters for ARS
Environment
step size
directions
noise
Walker2d-v3
0.01
[8,8]
0.05
Swimmer-v3
0.01
[8,4]
0.05
HalfCheetah-v3
0.01
[8,4]
0.05
Ant-v3
0.01
[32,16]
0.01
Hopper-v3
0.01
[8,4]
0.05
InverteDDoubleeeenduuummy
0.01
[8,8]
0.025
Table 2: Final return (average over final 20 evaluations)
Environment
PSSVF
ARS
DDPG
SAC
Walker2d-v3
2333 = 343
1488 + 961
2432 + 1330
5287 + 467
Swimmer-v3
349 + 60
342 + 21
129 2
44
HalfCheetah-v3
3067 = 820
2497 + 611
10695 1358
13599 932
Ant-v3
1549 = 240
1697 + 225
466 + 716
5319 + 992
H3pper-v3
2969 + 165
2340 + 199
1634 + 1036
3292 + 345
InvertedDouble
Pendulum-v2
7649 + 2640
4515 = 2733
7377 + 3770
9235 + 227
Moopar-v2
5
2000
2300
No weght
0.50M
LO0N
1.5MM
2.00N
0.50M
1CCM
2.5N
Figure 10: Comparison between algorithm without weighted sampling from the replay buffer and with weight 1/z*, where {1.0, 1.1). Average over 10 independent runs and 95% bootstrapped confidence interval.
17

---

Under review as a conference paper at ICLR 2023
Comparison to vanilla PSSVF Here compare PSSVF with policy fingerprinting to vanilla PSSVF. For vanilla PSSVF, we use the best hyperparameters reported by Faccio et al. (2021) when optimizing policies with 2 hidden layers and 64 neurons per layer and optimizing over the final rewards. Our algorithm uses the policy architecture vanilla PSSVF and the hyperparameters of our main experiments, changing only the learning rate of the policy to 1e 4 and the noise for policy perturbations to 0.1. Figure [T shows that while in Swimmer policy fingerprinting is enough to achieve improvement over vanilla PSSVF, Hopper non-uniform sampling plays an important role. Note that in the vanilla PSSVF paper, learning rates and perturbation noise are tuned for each environment, while our experiments we keep fixed set of hyperparameters for all environments to maintain consistency. We expect the performance of our approach to also improve by selecting hyperparameters separately for each environment.
Swimmer-3
Hopper-y3
400
poswf-ligeepent
200
2000
poovt dingeroii weighted sampling
1.00
0.00M
1.0MM
2.50M
3.00M
Figure 11: Comparison between vanilla PSSVF with weighted sampling and fingerprinting, PSSVF with policy fingerprinting, and our final algorithm that uses also weighted sampling. The solid line is the average over 10 independent runs; the shading indicates 95% bootstrapped confidence intervals.
Zero-shot learning of new policy architectures For this task we use the same hyperparameters as in the main experiments (see Appendix (A.)). We use a learning rate of le to zero-shot learn the linear policy. Figure reports similar results for deep policies.
400
200
200
400
600
800
1000
gradient steps
Figure 12: Performance of deep policy (in blue) zero-shot learned (averaged over 5 runs, 95% bootstrapped CI). The orange line shows the best performance of the deep NN when training the PSSVE.
B.3 FINGERPRINT ANALYSIS
Learning Swimmer with states We are interested in what the smallest amount of state- action pairs we could use to clone an optimal policy. In order to select the 3 transitions we try all combinations of 3 probing states our of that we used to train our PSSVE. When cloning using all 5 probing states, the performance is very similar to the optimal policy. When choosing 4 out of probing states, we notice that the performance highly depends on which probing state is removed, suggesting that some of the learned probing states more important than others. When trying 3 out of probing states this effect is more evident, and many combinations of 3 probing states lead to poor cloning performance. Here report the learning curves for the experiment in the main paper where we a randomly initialized policy using only 3 transitions (see Section 4.4p. These 3 transitions 3 probing states and the corresponding optimal action (probing action) in those states. We can
18

---

Under review as a conference paper at ICLR 2023
in Figure 13 that the MSE loss goes to when fitting the 3 transitions, the return of the policy increases until it almost matches the optimal value. In this experiment we train a PSSVF with 5 probing states following Algorithm for 2.M time steps. We manually select subset of probing states and act in those states using the learned policy. We then fit a new policy over those transition. We use batch size of 3 and learning rate of 2e to fit the policy. The other hyperparameters are the same as in the main experiments (see Appendix A.2)
0.7
retum
200
0.3
100
0.2
0-
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
gradient stops
gradient stops
Figure 13: On the left: return of the policy learned using transitions in Swimmer. On the right, MSE for fitting the 3 transitions. Average over 5 independent runs and 95% bootstrapped confidence interval.
Learning Hopper with states We repeat the same experiment of cloning near-optimal behaviour from a few states in the Hopper environment. Using the action of good policy (whose return is 2450) in probing states, we are able to fit a new policy and obtain final return 2200. We use a batch size of 5 and learning rate of 1e 4 for the randomly initialized policy. All other hyperparameters are like in the Swimmer experiment with 3 transitions. Figure [4]ssoww the learning curve, while Figure [6] relates the behavior of the policy learned using the 5 transitions the distance of the current agent's 's state to the probing states. The probing actions {āx are:
ā1 [0.4859, 0.6492, 00.7818).
ā2 [0.9251, 0.9100, 0.2322).
ä3 [0.0405, 0.0475, 0.9091].
ä.1 [0.2925, -0.4677. -0.1329))
ã5 [0.7578, 0.4327, 00.1521].
We observe similar behavior of the Swimmer experiments (Figure 15). where the action chosen by the agent is similar to the probing action of probing state whenever the agent's state close to the probing state. Although the dynamics in Hopper are more complex than in Swimmer, 5 probing states are enough to make the agent perform non-trivial actions the environment.
0.10
2500
2000
retum
1500
1000
0.04
500
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
gradient steps
gradient steps
Figure 14: On the left: return of the policy learned using 5 transitions in Hopper. On the right, MSE for fitting the 5 transitions. Average over 5 independent runs and 95% bootstrapped confidence interval.
19

---

Under review as a conference paper at ICLR 2023
02
80
100 120
1.0
-0.5
Figure 15: Behavior of the policy learned from 3 probing state-probing action pairs in Swimmer. From top to bottom: each component of the state vector across time steps an environment simulation; each component of the action vector; L2 distance of the current state to each of the 3 probing states used for learning.
10
50
-3
-10
530
200
40
action
0.0
-0.5
50
100
150
200
250
300
350
400
50
100
150
200
250
300
350
400
steps
Figure 16: Behavior of the policy learned from 5 probing state-probing action pairs in Hopper. From top to bottom: each component of the state vector across time steps in an environmental simulation: each component of the action vector; L.2 distance of the current state each of the probing states used for learning.
20

---

Under review as a conference paper at ICLR 2023
Visualization of probing states in RL In Figure 15 show the three probing states of the last experiment on Swimmer. In environments like Hopper and Walker, probing states might not correspond to real state in the environment (e.g. a specific range). We notice that this is usually not the case and that the learned probing states generally correspond to valid environmental states. Moreover, to get closer to certain critical situations over learning. These are states where certain actions have significant effect on the future. In the Ant environment, we notice that all components of the probing state vector from index 28 to 111 learn value of around le - 8. Interestingly, the process of fingerprintig discovers this 'bug' in MuJoCo 2.0.2.2 that all contact forces in Ant to zero. Since these components of the state vector remain constant during the environmental interactions, and are therefore not relevant for learning, the PSSVF learns to set them to zero as well.
components of the probing state outside
observe that probing states tend
Figure 17 shows the evolution of the Swimmer environment from the selected probing states when no action is taken. The 3 probing states reported are those used for the experiment of Figures [3 and T5
Figure 17: From top to bottom: the three learned probing states in Swimmer. From left to right: Evolution of the environment over time steps. The agent is initialized the probing state and performs action.
Figure 18: From top to bottom: the 5 learned probing states on Hopper. From left to right: various time steps in the environment. The agent is initialized in the probing state and performs action.
Figure 18 shows 3 out of the 5 learned probing states Hopper in the experiment of Figures 14 and 16 The other 2 probing states do not correspond to valid states in Hopper and are therefore not visualized. No action is taken from the probing state and the environment allowed to evolve naturally from the probing state. The duration of interaction differs in each row the figure as termination occurs different points from the probing states.
21

---

Under review as a conference paper at ICLR 2023
Figure 19:
Evolution of the environment from probing state when (Top) no actions taken, (Bottom) the first action in the probing state is taken using good policy. Then no action is performed.
Figure 19 supports our hypothesis that some probing states might capture critical scenarios. In the considered probing state from Hopper we see that taking action results in immediate failure as indicated by the shorter span of interaction in the top panel of Figure [9 In contrast, acting for single time-step with successful policy in that situation helps the agent survive and prolongs the interaction (bottom panel of Figure 19
Additional probing states for all environments can be seen in animated form on the website https
/ //anonyyous200522.ccoom wwt.:u.......iiu.....oo.......
C
SOCIETAL IMPACT
Our work makes algorithmic contributions to actor-critic approaches for reinforcement learning and does not focus specific real-world applications. Using PSSVF for offline improvement of policies (as shown in our MNIST experiment) could help mitigate risks from directly applying deep neural network policies to online situations in the real world.
D ENVIRONMENT DETAILS
Mujoco is made available with Apache License 2.0. The MNIST dataset is available through the creative commons license CC BY-SA 3.0.
22


Reviews:
[{"review_id": "ZcGJWTbvHs", "reviewer": "Reviewer_eb1Q", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper \"General Policy Evaluation and Improvement by Learning to Identify Few But Crucial States\" proposes to consider fingerprints based on actions distributions on states, rather than using all policy parameters, for building value functions that generalize over all possible policies. Such policy evaluation functions are very useful for discovering efficient policies, by gradient ascent of policy parameters through the value function, as they do not require interactions to improve policies whenever the value function is general enough. Authors show interesting results.  ", "summary_of_the_review": "I already gave my main concerns above, I have now some questions for the authors: \n    - Is there no catastrophic forgetting during learning, since V has to embed every encountered policies ? Is there not a risk of oscillation between two parameters areas ?  \n    - From my point of view, the experimental part miss a study of probing states dynamics. It would be veryinsightfull to understand how they behave during learning. Do they first collapse in a single state while the value is not well learned for spreading on more useful areas afterwards ? \n   - As PVN is a very related approach, I would have expected to have results of this approach as a baseline, maybe with a version that alternate evaluation and policy improvement\n    - Did authors experiment a version with reward discretization as in PVN ?  \n   - Algorithm1 only learns from feedback over full trajectories. Thus the variance must be high. How would it perform on more difficult stochastic environments ? I suspect that considered tasks are rather simple and this kind of approach would be difficultly transferred on more complex environments. \n\nIn summary, a very easy to read and interesting paper, but from my point of view the proposal is not enough innovative for ICLR, and the experimental analysis not enough insightful. ", "strength_and_weaknesses": "Strengths:\n   - Interesting results : only few states are enough to get useful value functions and policy improvements in many classical settings\n   - Well written paper\n\nWeakness: \n    - I feel the approach is a very straightforward extension of [Harb et al., 2020], as the only novely comes down to simply alternating the evaluation and improvement algorithms from that paper (except a very minor difference on the way of considering rewards - as a regression target rather than classification in discretized bins). \n    - Thus the interest of the paper is mainly experimental. However I feel that not enough analysis is given to well understand the dynamics of the proposal.   \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "H8XI2EiBjbO", "reviewer": "Reviewer_zMK3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors get the parameter-based state-value function (PSVF) Faccio et al., 2021 to work. The motivation is simple, if there's a value function $V:\\mathcal S\\times\\Theta\\to\\mathbb R$ which maps both a state and the parameters of a policy to an accurate cumulative discounted reward estimate, then the policy can be improved by maximizing $V(s,\\theta)$ w.r.t. $\\theta$. The only problem is that for any class of policies to be useful, the parameter space of that class of polices $\\Theta$ is generally quite large (thousands of dimensions) making learning the PSVF V impractical.\n\nThe authors propose to map this large $\\mathcal S\\times\\theta$ to a smaller space with two simple tricks. First, they don't worry about the discounted return will be from a specific state, peruse a general policy evaluation framework to learn the sum of discounted rewards from an average starting state. This isn't novel. The novel and significant step they take is to learn K \"probing states\" $s_1,\\dots,s_K$ and instead of learning the PSVF they estimate (this notation only works for deterministic policies, but the idea works for certain stochastic policies as well) the value of a policy with $V_\\phi(\\pi_\\theta(s_1),\\dots,\\pi_\\theta(s_K))$. Since $K$ is small (generally 200 or less) and the action space is also small, this addresses the curse of dimensionality.\n\nThe authors then do two sets of experiments, one on MNIST and the other on MuJoCo outlining interesting characteristics of their PSSVF method, showing that the probing states learned are meaningful and that the learned value estimation is useful for constructing good policies.\n\n\n\n\n", "summary_of_the_review": "Overall, when reading the paper I felt I learned a significant new tool, one that I feel others in the RL community should learn", "strength_and_weaknesses": "I'll compress \"Strength And Weaknesses\" and \"Clarity, Quality, Novelty And Reproducibility\" into one review, as there's a lot of overlap.\n\n### Strengths:\n- the paper has a clear goal which is quite relevant: learn a single value function for evaluating any policy, and the authors clearly explain how they go about working to this goal, compare to past work on how this goal was achieved, and what they do different, and why this leads to better results.\n- the paper's notation is generally clear, well introduced and easy to follow\n- The experiments show what the method is doing, not just that it gets good results but that the probing states are significant, and that a value function learned even on random policies can be used to generate a good policy.\n- I like that the authors don't claim their method is the best, but compare against more sample efficient methods like SAC.\n\n### Major Concerns:\n\nMy only major concern is that the authors seem to claim to do more than they actually achieve. The claim is to \"learn a single value function for evaluating...any policy\" But they don't do this, they learn an estimate $J(\\theta)$ of the expected return. Worse, they really dress the paper up to make it look like they learn a value function, using $V(\\theta)$ notation for example, and discussing value functions and Q-functions at length in the introduction. While I'm sure their methods could be modified to learn an actual parameter-based state-value $V(s,\\theta)$ I bet it's tricky (since most of the time the estimate of the value function depends on an estimate of the value function).\n\nSo I'd really wish the author's are honest with their readers and say something like \"in order to focus on the main point, the probing state method and not get distracted by the complexities of learning a value function, we focus on the easier task of learning estimate $J(\\theta)$ of the expected return on finite horizon MDPs. And drop anything that distracts from this simpler yet significant goal.\n\nThe authors discuss a tiny bit the step to learning a real value function at the top of page 7, but it's a bit of a farce, since they only learn the value function over initial states, if I remember correctly in MuJoCo initial states are all quite similar, so it's unsurprising that the authors state \"The results were very similar to those we presented in this section\"\n\nFor this reason, I have to say on the correctness that \"Several of the paper’s claims are incorrect or not well-supported,\" but I don't see any reason why this can't be fixed.\n\n### Minor Concerns:\n- I don't find Figure 1 helpful at all. Why are some lines solid, why are some dashed? What's $U_\\phi$? The algorithm has a replay buffer (or storage of policy parameters / rewards) that's not in Figure 1. Either have a helpful figure or no figure at all but not a figure for figure's sake.\n- I think that Eq. 2 is incorrect, but since I think it's a distraction and should be dropped I won't go into detail.\n- in eq. 4, $\\pi_\\theta(s_1)$ is a distribution over actions. You discussed \"parameters of the output distribution of the policy in such states\" but left that out in eq. 4. For eq. 4 to be correct, this mapping from $\\theta$ to distribution parameterization must be included.\n- Please also tell the readers that the estimation of $V(\\theta)$ only works since in MuJoCo the starting states are all very similar, and the MNIST experiment is stateless.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "ADL1bkFiQ6", "reviewer": "Reviewer_yJcN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nThis paper proposes using the policy evaluation networks architecture\n(policy fingerprinting) with the algorithm structure of parameter-based\nvalue functions to learn a single value of the start state that\nconditions on the policy representation. This value function is learned\nthrough online interaction, like parameter-based value functions, but\nscales to larger policies because of policy fingerprinting. The\neponymous finding is that, for some mujoco environments, the number of\nprobing states for policy fingerprinting is surprisingly low while\nfacilitating zero-shot policy learning on different architectures.\n\n", "summary_of_the_review": "It is difficult to rate this submission because, while lacking in\nnovelty, the eponymous finding is rather surprising and interesting. The\nlack of novelty in the algorithm is unavoidable, as it builds on two\ndistinct works that are quite similar. However, I think the experiments\nhave potential to provide more insight. While the results are mostly\npositive, the submission could be strengthened by outlining the limits\nthe approach in negative results. For example, is it more or less\ndifficult to learn to predict the losses when the dataset is more\ncomplicated (omniglot, fashion MNIST, CIFAR). Does this difficulty\ntranslate to difficulty to extrapolate to new architectures, or to\nzero-shot train an architecture to a performance level beyond what was\nseen? There is plenty of opportunity to contextualize the very\ninteresting main finding. As the submission currently stands, I am\nleaning towards weak reject. I am open to increasing my score of some of\nthese points are addressed.\n\n", "strength_and_weaknesses": "# Strengths\n\n-   The main result is very interesting and surprising. The result being\n    that policies in mujoco environments can be represented by observing\n    the actions at a small number of states. It is further surprising\n    that a strong policy can be learned by regressing on the actions at\n    those few states.\n\n-   While there are a few clarity issues, which I outline in the\n    detailed comments below, the overall paper is well-structured and\n    easy-to-follow. Not only is the motivation clear, the contribution\n    is clear as well.\n\n# Weaknesses\n\n-   The main weakness is that the novelty is incremental.\n    Algorithmically, there is little novelty: the proposed algorithm is\n    a combination of the architecture of policy evaluation networks and\n    online algorithm structure of parameter-based value functions. The\n    eponymous finding, that policy evaluation and improvement can be\n    done by identifying few but crucial states, is limited to only two\n    mujoco environments. While the experiments encompass more than this\n    finding, this seems to be the major contribution.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "69iPOj8jQ1_", "reviewer": "Reviewer_xK3x", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work combines two existing approaches: representing a policy based on its behavior in a set of probing states and finding a successful policy via a critic that estimates the return of any input policy.\n\nThe ability to automatically learn probing states is shown via MNIST.\nThe performance on a few continuous control tasks is demonstrated.\nThe ability to use a trained critic to find a successful linear policy is also shown.\n", "summary_of_the_review": "The proposed method is motivated by the difficulty scaling a policy-parameter-based approach. However, this benefit is not supported.\nAdditionally, the overall experiments are limited without sufficient justification.", "strength_and_weaknesses": "This work identifies a promising combination of two existing methods.\nIf probing states are a more efficient representation for complex policies, then this method is of interest.\n\nThe authors claim that \"Flattening the policy parameters ... is difficult to scale to larger policies,\" but do not show the new method scaling.\nUnfortunately, the proposed method is not shown to scale to larger domains. Empirical evaluation is required to confirm favorable scaling, but all of the domains considered can be solved with simple policies.\nI am skeptical that this method scales better than using the policy parameters: a strength of policy gradient methods is that the policy complexity generally scales more slowly than the environment complexity. However, this method requires coverage of the state space, so the number of required crucial states may scale quickly.\nThe key experiment is shown in the appendix (Fig 10) and is quite limited in scope. It specifically uses the environments where few probing states are needed to learn a policy.\n\nMinor Comments:\n- The \"Policy fingerprinting\" paragraph should be divided to more clearly partition novel contributions from existing work.\n- By changing the rewards for Hopper, Walker, and Ant, those environments are no longer being solved (only modified versions of them). Likewise, gamma is part of the MDP; reporting total return (gamma=1) when gamma is 0.99 for only some evaluated methods is incorrect.\n- Sec 4.3: \"trough\" -> \"through\"\n- The limitation of the proposed method to deterministic policies is used to motivate baseline selection, but this limitation is a choice rather than part of the problem setup. To use this line of reasoning, one should also express why deterministic policies are advantageous.\n\nQuestions:\n- How were domains selected for evaluation? Why are only two environments used for Fig 10?\n- What is the behavior of your method with stochastic policies?\n- Can your approach perform well when the policies have different architectures during training?\n- Can vanilla PSSVF be used with weighted sampling?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject