Parameter-Based Value Functions
Authors: Francesco Faccio
Louis Kirsch
Jürgen Schmidhuber
Venue: ICLR 2021 Poster (2021)
Keywords: Reinforcement Learning
Off-Policy Reinforcement Learning

Abstract: Traditional off-policy actor-critic Reinforcement Learning (RL) algorithms learn value functions of a single target policy. However, when value functions are updated to track the learned policy, they forget potentially useful information about old policies. We introduce a class of value functions called Parameter-Based Value Functions (PBVFs) whose inputs include the policy parameters. They can generalize across different policies. PBVFs can evaluate the performance of any policy given a state, a state-action pair, or a distribution over the RL agent's initial states. First we show how PBVFs yield novel off-policy policy gradient theorems. Then we derive off-policy actor-critic algorithms based on PBVFs trained by Monte Carlo or Temporal Difference methods. We show how learned PBVFs can zero-shot learn new policies that outperform any policy seen during training. Finally our algorithms are evaluated on a selection of discrete and continuous control tasks using shallow policies and deep neural networks. Their performance is comparable to state-of-the-art methods.

Published as a conference paper at ICLR 2021
PARAMETER-BASEE VALUE FUNCTIONS
Francesco Faccio, Louis Kirseh & Jürgen Schmidhuber The Swiss Lab IDSIA, USI, SUPSI [francesco, louis, juergen)e litpeennneiiii...oom .c
ABSTRACT
Traditional off-policy actor-critic Reinforcement Learning (RL) algorithms learn value functions of a single target policy. However, when value functions are up- dated to track the learned policy, they forget potentially useful information about old policies. We introduce a class of value functions called Parameter- Value Functions (PBVFs) whose inputs include the policy parameters. They gener- alize across different policies. PBVFs can evaluate the performance of any policy given state, state-action pair, or distribution over the RL agent's s initial states. First we show how PBVFs yield novel off-policy policy gradient theorems. Then we derive off-policy actor-critic algorithms based on PBVFs trained by Monte Carlo or Temporal Difference methods. We show how learned PBVFs can zero- shot learn new policies that outperform any policy seen during training. Finally our algorithms are evaluated on selection of discrete and continuous control tasks using shallow policies and deep neural networks. Their performance is compara- ble to state-of-ther the-art methods.
INTRODUCTION
Value functions are central Reinforcement Learning (RL). For a given policy, they estimate the value of being in a specific state (or of choosing a particular action in a given state). Many RL breakthroughs were achieved through improved estimates of such values, which can be used find optimal policies (Tesauro 1995] Mnih et al., 2015). However, learning value functions of arbitrary policies without observing their behavior the environment not trivial. Such off off-policy learning requires to correct the mismatch between the distribution of updates induced by the behavioral policy and the one we want to learn. Common techniques include Importance Sampling (IS) (Hesterberg) 1988) and deterministic policy gradient methods (DPG) (Silver et al. (2014), which adopt the actor- critic architecture (Sutton 1984; Konda & Tiitsiklis 2001 Peters & Schaal 2008).
Unfortunately, these approaches have limitations. IS suffers from large variance (Cortes al. 2010 Metelli et al.) 2018: Wang et al.) (016) while traditional off-policy actor-critic methods introduce difficult to follow since they involve the gradient of the
off-policy objectives whose gradients action-value function with respect to the policy parameters VoQTT (s, a) (Degris et al. 2012 Silver let al. 2014). This term is usually ignored, resulting in biased gradients for the off-policy objective. Furthermore, off-policy actor-critic algorithms learn value functions of a single target policy. When value functions are updated to track the learned policy, the information about old policies is lost.
We address the problem of generalization across many value functions in the off-policy setting by introducing a class of parameter-base value functions (PBVFs) defined for any policy. PBVT's value functions whose inputs include the policy parameters, the PSSVF (()) PSVF ((s, f), and PAVF Q(s, a. 0). PBVFs can be learned using Monte Carlo (MC) (Metropolis & Ulam 1949) or Temporal Difference (TD) [Sutton] 1988) methods. The PAVF (( 0) leads to novel stochastic and deterministic off-policy policy gradient theorem and, unlike previous approaches, can directly compute V*Q** a). Based on these results, we develop off-policy actor-critic methods and com- pare our algorithms to two strong baselines, ARS and DDPG (Mania et al. 2018 Lillicrap et al. 2013). outperforming them in some environments.
We make theoretical, algorithmic, and experimental contributions: Section introduces the stan- dard MDP setting: Section 3 formally presents PBVFs and derive algorithms for V(Ø), V f) and ((s, a, f); Section 4 describes the experimental evaluation using shallow and deep policies; Sec- tions 5 discuus related and future work. Proofs and derivations can be found in Appendix A.2

---

Published as a conference paper at ICLR 2021
2 BACKGROUND
We consider a Markov Decision Process (MDP) (Stratonovich 1960 Puterman 2014) MM
(S, A. P. R, 7: (o) where at each step agent observes state S. chooses action A, transitions into state s' with probability P(s"|s, a) and receives a reward R(s, a). The agent starts from an initial state, chosen with probability (s(s). It is represented by parametrized stochastic policy To : SS + ((A), which provides the probability of performing action a in state s. θ is the space of policy parameters. The policy is deterministic if for each state there exists tion a such that To (a]s)) = 1. The return RG is defined as the cumulative discounted reward from time step t: R = > k-0 (((R((++1i1 1, 0c+k+1) I)), where T denotes the time horizon and real- valued discount factor. The performance of the agent is measured by the cumulative discounted expected reward (expected return), defined as J (TT)
E,, [Ro]. Given policy Tæ. the state-value function V (s) = Ex, [ = s] is defined as the expected return for being in a state s and express the maximization of the expected cumulative reward in terms of the state-value function J(Tp) = Ss (o(s)) V** (s) ds. The action-value function QT a), which is defined as the expected return for performing action in state s, and following the policy To. is QT (s, a) Ex [Rg = 8, ag a). and it is re- lated to the state-value function by (s) = SA To (a(as)) " da. We define as d=*(s') the discounted weighting of states encountered starting at 80 ~ Ho(s) and following the policy me: d' [s E1=1< "o P(s + s',t, πT) ds, where P(s + te) is the probability of transitioning to s' after t time steps, starting from and following policy [Sutton et al. (1999) showed that, for stochastic policies, the gradient of (TT) does not involve the derivative d and can be expressed in simple form:
following policy Ty. By integrating over the state space S,
VoJ( / (s) V Vono (ato(als) (s, a) da ds.
Similarly, for deterministic policies Silver al. (2014) obtained the following:
VoJ( = Js d d= T (s, ())ama(())) ds.
In off-policy policy optimization, we seek to find the parameters of the policy maximizing performance index JT(Te) using data collected from behavioral policy лTs Here the objective function (T(T)) is typically modified to be the value function of the target policy, integrated over (s) lime P(s: s|so, me). the limiting distribution of states under m. (assuming
Off-policy RL
exists) (Degris et al. 2012) Imani et al., (018)) Wang et al. 2016). Throughout the paper we assume
that the support of dh includes the support fig so that the optimal solution for J, also optimal
for J. Formally, we want to find:
max / d ds) Jo(TT)) max / (SV
/ To(als)("* da...
/s
Unfortunately, in the off- policy setting, the states are obtained from di and not from d hence the gradients suffer from distribution shift (Liu et al., 2019; Nachum et al.] 2019). Moreover, since we have no access to d% term the policy gradient theorem corresponding to the gradient of the action value function with respect to the policy parameters needs to be estimated. This term is usually ignored traditional off-policy policy gradient theorem! stochastic, Degris et al. (0022) showed that:
In particular, when the policy is
(als) (als)
VoJo(To)= Js ( (8)
QQT (8, (,, Vogmool + VoQT* a)) da (4)
(a(s)
(als)
(s)
(
(Q (s, a) Vo log no (als)) ds.
(5)
(a(s)
Analogously, Silver et al.
2014 provided the following approximation for deterministic policies
VaJ((TT)
d (s)
(T (s) a) (s=))s)) + VoQ** (s))))) ds
(6)
d'a ( V Q (s, (s,a))aa= ()))=))()) ds.
With tabular policies, dropping this term still results in convergent algorithm Degris et al. 2012 In the original formulation of Silver et al. 2014) d=d (s) replaced by (s).
2

---

Published as a conference paper at ICLR 2021
Although the term VoQ=o(s, (s, a) is dropped, there might be advantages in using the approximate gradient of J in order to find the maximum of the original RL objective J. Indeed. if we were on-policy, the approximated off-policy policy gradients by Degris et al. (2012;; [iivvr et al. (2014) would revert to the on-policy policy gradients, while an exact gradient for J. would necessarily introduce bias. However, when we off- off-policy, it is not clear whether this would be better than using the exact gradient of J, in order to maximize J. In this work, we assume that Jb can be considered a good objective for off-policy RL and we derive an exact gradient for
PARAMETER-BASHH VALUE FUNCTIONS
In this section, we introduce our parameter-based value functions, the PSSVF ((0), PSVF (s,, O), and PAVF Q(s, a, 0) and their corresponding learning algorithms. First, we augment the state and action-value functions, allowing them to receive as an input also the weights of parametric pol-
icy. The parameter-based state-value function (PSVF) V (ss, 0) - E[R 82 = 8, 0] is defined as the expected return for being in state and following policy parameterized by 0. Similarly, the parameter-based action-valus function (PAVF) (s a, 0) = E[Re|s; = s, a = a. 0] is defined as the expected return for being in state taking action and following policy parameterized by 0. Us- ing PBVFs, the RL objective becomes: ((T)) ) = Ss No (s) V (s, 0) ds. Maximizing this objective leads to on-policy policy gradient theorems that analogous the traditional ones (Sutton al., 1999 Silver et al. 2014):
Theorem 3.1. Let be stochastic. For any Markov Decision Process, the following holds:
V V = Eg-ate (s).o)) [[Q(s, a, V) log =o(a|s))]
(8)
Theorem 3.2. Let be deterministic. Under standard regularity assumptions (Siverr et al. 2014), for any Markov Decision Process, the following holds:
(x)( (Egddr(()) [Vo((s, 0)la=m(()) VoTo(S)]
Parameter- baaeed value functions allow us also to learn function of the policy parameters that directly approximates J(TO). In particular, the parameter-based start-state-value function (PSSVF) defined as:
V(0)= = Es) (V) (V po(s) 0) (TT).
(10)
Off-policy RL
In the off-policy setting, the objective to be maximized becomes:
Jo(TT=)) max / d= (s) ds = max / (s) da
(11)
By taking the gradient of the performance Js with respect to the policy parameters we obtain novel policy gradient theorems. Since 0 is continuous, we need to use function approximators V. (0) s (0)), (s, 0) Re (ss, 0) and Qw(s, a, 8) ~ Q(s, a, 0). Compatible function approximations can be derived to ensure that the approximated value function is following the true gradient. Like in previous approaches, this would result in linearity conditions. However, here we consider nonlinear function approximation and we leave the convergence analysis of linear PB as future work. In episodic settings, we do not have access to d so in the algorithm derivations and in the experiments approximate it by sampling trajectories generated by the behavioral policy. In all cases, the policy improvement step can be very expensive, due to the computation of the arg max over a continuous space θ Actor-critic methods can be derived to solve this optimization problem, where the critic (PBVFs) can be learned using TD or MC methods, while the actor is updated following the gradient with respect to the critie. Although our algorithms on PSSVF and PSVF can be used with both stochastic and deterministic policies, removing the stochasticity of the action-selection process might facilitate learning the value function. All our algorithms make use of a replay buffer.
3.1 PARAMETER-BAE BASED START-STATE- VALUE FUNCTION (())
We first derive the PSSVF (0). Given the original performance index J, and taking the gradient with respect to 0, we obtain:
(oJ( = { VoV ds E Esrwo(s) VoV(s, (oV
(12)
3

---

Published as a conference paper at ICLR 2021
In Algorithm| 1 the critic (f) is learned using MC to estimate the value of any policy 0. The actor is then updated following the direction of improvement suggested by the critic. Since the main application of PSSVF is in episodic task optimize for the undiscounted objective.
Algorithm Actor-critic with Monte Carlo prediction for V(0)
Input: Differentiable critic Ve + R with parameters w; deterministic or stochastic actor To with parameters 0; empty replay buffer D Output Learned V = V(0)v0, learned my = TT* Initialize critic and actor weights w. repeat:
Generate an episode Su, Go, $1, 01, T2.... $T-1, with policy
Compute return Σk= 2
Store (0, r) in the replay buffer
for many steps do:
Sample batch B = ((r, 0)) from D
Update critic by stochastic gradient descent: V (r.0)) (0] end for for many steps do:
Update actor by gradient ascent: Vo (0) end for
until convergence
3.2 PARAMETER-BASER STATE- VALUE FUNCTION V(s, 0)
Learning the value function using MC approaches can be difficult due to the high variance of the estimate. Furthermore, episode-based algorithms like Algorithm ! arre unable to credit good actions in bad episodes. Gradient methods based on TD updates provide a biased estimate of V(s, 0) with much lower variance and can credit actions at each time step. Taking the gradient of Jn (T)) in the PSVF formulation we obtain:
VoJa(TO)) =
E E (8) (8,
(13)
Algorithm (Appendix) the actor-critic architecture, where the critic is learned via TD
3.3 PARAMETER-BAAE BASED ACTION-VALUE FUNCTION Q(s, 0)
The introduction of the PAVF ((s, 0) allows us to derive new policy gradients theorems when using a stochastic or deterministic policy.
Stochastic policy gradients We want to use data collected from some stochastic behavioral policy Ty in order to learn the action-value of a target policy æy. Traditional off-policy actor-critic algo- rithms only approximate the gradient of Jr. since they do not estimate the gradient of the action- value function with respect to the policy parameters VøQT* (s, a) (Degris et al. 2012 Silver et al,, 2014). With PBVFs, we can directly compute this contribution the gradient. This yields an exact policy gradient theorem for Jj:
Theorem 3.3. For any Markov Decision Process, the following holds:
(a)s))
VoJo(To) E Eodd (vxs(()))))
(Q(s,
voQ(s,,,,)
(14)
Algorithm (Appendix) uses an actor-critic architecture and can seen as an extension of Off- PAC (Degris al. 2012) to PAVE.
Alternatives include regenerative method for MC estimation (Rubinstein & Kroese 2016
CCompared to standard methods based on state-value function, we can directly optimize policy following the performance gradient of the PSVE, obtaining policy improvement step in model-free way. Note that the differentiahility of the policy me is never required PSSVF and PSVE.
4

---

Published as a conference paper at ICLR 2021
Deterministic policy gradients Estimating Q(s, a, 0) is in general difficult problem due the stochasticity of the policy. Deterministic policies of the form т : S + AA can help improving the efficiency in learning value functions, since the expectation over the action space is no longer required. Using PBVFs, we can write the performance of a policy Ty as:
Jo(Te)= - - 0) ds= / (s) (s), 0) ds.
d=
(15)
Taking the gradient with respect to 0 we obtain deterministic policy gradient theorem:
Theorem 3.4. Under standard regularity assumptions (Silver et al. 2014), for any Markov Decision Process, the following holds;
(T))(T)) = (2) (o) V a, (s) V VoTo(s) + QoQ 0)), ()mm=moss)]
(16)
Algorithm 4 (Appendix) uses an actor-critic architecture and can be seen as an extension of DPG (Silver et al.) 2014) PAVE. Despite the novel formulation of algorithm [] we decided to avoid the stochasticity of the policy and to implement and analyze only the deterministic PAVE.
4 EXPERIMENTS
Applying algorithms and 4 directly can lead to convergence local optima, due to the lack of exploration. In practice, like in standard deterministic actor-critic algorithms,
noisy version of the current learned policy in order to act in the environment and collect data to encourage exploration. More precisely, at each episode we use Tg with 0 = + t, e ~ (00,, I² instead of m and then store @ in the replay buffer. In our experiments, we report both for our methods as well as the baselines the performance of the policy without parameter noise.
4.1 VISUALIZING PBVFs USING LQRS
We start with an illustrative example that allows us to visualize how PBVFs are learning to estimate the expected return over the parameter space. For this purpose, we use an instance of the ID Linear Quadratic Regulator (LQR) problem and linear deterministic policy with bias. In figure we plot the episodic J(f), the cumulative return that an agent would obtain by acting in the environment using policy To for single episode, and the cumulative return predicted by the PSSVF V(@) for two different times during learning. At the beginning of the learning process, the PSSVF is able provide just local estimation of the performance of the agent, since only few data have been observed. However, after 1000 episodes, it is able to provide a more accurate global estimate over the parameter space. Appendix A.44. contains a similar visualization for PSVF and PAVE, environment details and hyperparameters used.
4.2
MAIN RESULTS
Given the similarities between our PAVF and DPG, Deep Deterministic Policy Gradients (DDPG) is natural choice for the baseline. Additionally, the PSSVF (0)) resembles evolutionary methods as the critic can be interpreted as a global fitness function. Therefore, we decided include in the comparison Augmented Random Search (ARS) which is known for its state-of-the-ar performance using only linear policies in continuous control tasks. For the policy, we use a 2-layer MLP (64,64) with tanh activations and linear policy followed by tanh nonlinearity. Figure 2 shows results for deterministic policies with both architectures. In all the tasks the PSSVF is able to achieve at least the same performance compared ARS, often outperforming it. In the Inverted Pendulum environment, PSVF and PAVF with deep policy are very slow to converge, but they excel in the Swimmer task and MountainCarContinuan DDPG converges quickly to the optimal policy. We conjecture that for this task it is difficult to perform search in parameter space. On the other hand, in MountainCarCCntimmuur the reward is more sparse and DDPG only rarely observes positive reward when exploring in action space. In Appendix [A.4 we include additional results for PSSVF and PSVF with stochastic policies and hyperparameters. We analyze the sensitivity of the algorithms on the choice of hyperparameters in Appendix (A.44.
In Reacher, all PBVFs fail to learn the task, while
"Code htapl//...tppp..o.o..c.//////eee based- Funct inncioons
5

---

Published as a conference paper at ICLR 2021
Optimization after 60 episodes
](0)
(o)
352
32
Return
2957
4893
-3
8962
-5
10000
-5
-3
Optimization after 1000 episodes
((0)
(2)
352
-32
Return
2857
4893
6929
8964
-5
-3
3
-5
81
Figure 1: True episodic return J()) and PSSVF estimation (()) function of the policy pa- rameters at two different stages training. The red arrows represent an optimization trajectory in parameter space. The blue dots represent the perturbed policies used to train V(0).
4.3 ZERO-SHOT LEARNING
In order to test whether PBVFs are generalizing across the policy space, we perform the following experiment with shallow deterministic policies: while learning using algorithm [ we stop training and randomly initialize policies. Then, without interacting with the environment, we train these policies offline, in a zero-shot manner, following only the direction of improvement suggested by (o), whose weights remain frozen. We observe that shallow policies be effectively trained from scratch. Results for PSSVFs Swimmer-v3 are displayed in figure 3 In particular, compare the performance of the policy learned, the best perturbed policy for exploration during training and five policies learned from scratch at three different stages in training. We note that after the PSSVF has been trained for 100,000 time steps interactions with the environment (first snapshot), these policies are already able to outperform both the current policy and any policy seen while training the PSSVE. They achieve an average return of 297, while the best observed return was 225. We include additional results for PSVF and PAVF in different environments, using shallow and deep policies Appendix A.4.2 When using deep policies, we obtain similar results only for the simplest environments. For this task, use the same hyperparameters as in figure 2
4.4 OFFLINE LEARNING WITH FRAGMENTED BEHAVIORS
In last experiment, investigate how PSVTs able to learn in completely offline setting. The goal is to learn a good policy in Swimmer-v3 given a fixed dataset containing 100,000 tran- sitions, without additional environment interactions. Furthermore. the policy generating the data is perturbed every 200 time steps. for a total of 5 policies per episode. Observing only incom- plete trajectories for each policy parameter makes TD bootstrapping harder: order to learn, the PSVF needs to generalize across both the state and the parameter space. Given the fixed dataset, we first train the PSVF, minimizing the TD error. Then, at different stages during learning, we train new shallow deterministic policies. Figure describes this process. We note that at the beginning of training. when the PSVF ((s, 8) has a larger TD error, these policies have poor performance. However, after 7000 gradient updates, they are able achieve reward of 237, before eventually degrading to 167. They outperform the best policy in the dataset used to train the PSVE, whose return is only of 58.

---

Published as a conference paper at ICLR 2021
Shallow policies
90mm
Deep policies
V0nw
Figure 2: Average return of shallow and deep deterministic policies as a function of the number of time steps used for learning (across 20 runs, one standard deviation), for different environments and algorithms. We use the best hyperparameters found when maximizing the average return.
RELATED WORK
There are two main classes of similar algorithms performing search in policy parameter space. Evolutionary algorithms (Wierstra al. 20144) Salimans al. [017]] Mania al.] (2018) iteratively estimate fitness function evaluating the performance of population of policies and then perform gradient ascent in parameter space, often estimating the gradient using finite difference approxima- tion. By replacing the performance of population through a likelihood estimation, evolutionary algorithms become a form of Paramcter Exploring Policy Gradients (Sehnke et al.] 2008, 2010). Our methods are similar to evolution since value function can be seen as fitness. Unlike evolution, however, our approach allows for obtaining the fitness gradient directly and is more suitable for reusing past data. While direct ((0) optimization is strongly related to evolution, more informed algorithms optimize V(s, 0) and Q(s, a, 0). That is, ours both perform a search in policy parameter space AND train the value function and the policy online, without having wait for the ends of trials or episodes.
The second related class of methods involves surrogate functions (Box & Wilson 1951
Booker et al. 1998 MMore Schneider] 1996). They often use local optimizers for generalizing across fitness functions. In particular, Bayesian Optimization (BO) (Snoek et al.) 2012 2015) uses a surrogate function to evaluate the performance of model over a set hyperparameters and follows the uncertainty on the surrogate to query the new data to sample. Unlike BO, we do not build a probabilistic model and we use the gradient the value function instead sample from the posterior to decide which policy parameters
next in the policy improvement step.
The possibility of augmenting the value functions with auxiliary parameters already
7

---

Published as a conference paper at ICLR 2021
20
policy hoamed 290
pakay larred
bed policy huffe
5333
220 -10
0.5 43
2.4 24
05
Figure 3: Policies learned from scratch during training. The plot in the center represents the return of the agent learning while interacting with the environment using Algorithm [ We compare the best noisy policy Tg used for exploration to the policy my learned through the critic. The learning curves in the small plots represent the return obtained by policies trained from scratch following the fixed critic V₂((0 after different time steps of training. The return of the closest policy (1.2 distance) in the replay buffer with respect the policy learned from scratch is depicted in green.
policy parord leameed
10.00
120
Figure 4: Offline learning of PSVF. We plot the mean squared TD error of a PSVF trained using data coming from a set of noisy policies. In the small plots, we compare the return obtained by policies trained from scratch following the fixed critic (s, 0) after different time steps of value function training and the return of the best noisy policy used to train V.
considered in work General Value Functions (Sutton et al. 2011). where the return is defined with respect to an arbitrary reward function. Universal Value Function Approximators (Schaul et al.) 2015) extended this approach to learn a single value function v* (s,,g) representing the value, given possible agent goals In particular, they learn different embeddings for states and goals, exploiting their common structure, and they show generalization our PSVF V(s, 0) is able to generalize to unseen policies, observing data for only few (s, 0) pairs General and Universal Value Functions have not been applied to learn single value function for every possible policy.
new unseen goals. Similarly,
Policy Evaluation Networks (PENs) HHab et al. 2020) are closely related to our work and share the same motivation. PENs focus on the simplest PSSVF ((d) trained without an actor-critic architecture. Like in some of our experiments, the authors show how following the direction of improvement suggested by ((0) leads to an increase in policy performance. They also suggest to explore in future work more complex setting where PSVF V(s, is learned using actor-critic
8

---

Published as a conference paper at ICLR 2021
architecture. Our work directly introduces the PSVF ((s, 0) and PAVF Q(s, a, 0) and presents novel policy gradient theorems for PAVFs when stochastic or deterministic policies are used. There are many differences between approach to learning V()) and theirs. For example, we do not use a fingerprint mechanism (Harb et al. 2020) for embedding the weights of complex policies. Instead, simply parse all the policy weights inputs to the value function, even in the nonlinear case. Fingerprinting may be important for representing nonlinear policies without losing information about their structure and for saving memory required store the weights. |Har et al. (2020) focus on the offline setting. They first use randomly initialized policies to perform rollouts and collect reward from the environment. Then, once V(0) is trained using the data collected, many gradient ascent steps through V yield new, unseen, randomly initialized policies in a zero-shot manner, exhibiting improved performance. They train their value function using small nonlinear policies of one hidden layer and 30 neurons on Swimmer-v3. They evaluate 2000 deterministic policies on 500 episodes each (1 million policy evaluations). achieving a final expected return of R 180 on new policies trained from scratch through V. On the other hand, in our cero-shot learning experiment using a linear PSSVF, after only 100 policy evaluations, we obtain a return of 297. In our main experiments, we showed that a fingerprint mechanism is not necessary for the tasks we analyzed: even when using much bigger 2-layers MLP policy, we are able to outperform the results in PEN. Although Harb et al. (2020) use Swimmer-v3 "to scale up their experiments", our results suggest that Swimmer-v3 does not conclusively demonstrate possible benefits of their policy embedding.
Gradient Temporal Difference (Sutton et al. 2009a bj Maci et al. 2009; 2010; Maci 201T) and Emphatic Temporal Difference methods (Sutton et al.) 2016) were developed to address convergence under on-policy and off-policy (Precup et al.] proximation. The first attempt to obtain a stable off-policy actor-critic algorithm under linear function approximation was called Off-PAC (Degris et al. 2012). where the critic is updated using GTD(A) (Maeil 201T) to estimate the state-value function. This algorithm converges when using tabular policies. However, in general. the actor does not follow the true gradient direction for Js, A paper on DPG (Silver et al.] 2014) extended the Off-PAC policy gradient theorem (Degris) eet al., 2012) deterministic policies. This was coupled with a deep neural network to solve continuous control tasks through Deep Deterministic Policy Gradients (Lilficrap et al.] 2015). [maan] al. (2018) used emphatic weights to derive an exact off-policy policy gradient theorem for Jh. Differently from Off-PAC, they do not ignore the gradient of the action-value function with respect to the policy, which is incorporated in the emphatic weighting: vector that needs to be estimated. Our off- off-policy policy gradients provide an alternative approach that does not need emphatic weights.
2001) learning with function ap-
The widely used off-policy objective function Jn suffers the distribution shift problem. Liu jet al.] (2019) provided an off-policy policy gradient theorem which unbiased for the true RI. objective J(me), introducing term d=/ /d that corrects the mismatch between the states distri- butions. Despite their sound off-policy formulation, estimating the state weighting ratio remains challenging. All our algorithms are based on the off-policy actor-critic architecture. The two algorithms based on Q(s, a, f) can be viewed as analogous to Off-PAC and DPG where the critic is defined for all policies and the actor is updated following the true gradient with respect the critic.
LIMITATIONS AND FUTURE WORK
We introduced PBVFs, a novel class of value functions which receive as input the parameters of a policy and can be used for off-policy learning. We showed that PBVFs are competitive ARS and DDPG (Mania et al., 2018: [illicrap et al. 2015) while generalizing across policies and allowing for zero-shot training an offline setting. Despite their positive results on shallow and deep policies, PBVFs suffer the curse of dimensionality when the number of policy parameters is high. Embed- dings similar to those used PENs (Harb al. 2020) may be useful not only for saving memory and computational time, but also for facilitating search in parameter space. We intend to evaluate the benefits of such embeddings and other dimensionality reduction techniques. We derived off off-policy policy gradient theorems, showing how PBVFs follow the true gradient of the performance Jn. With these results, we plan to analyze the convergence of our algorithms using stochastic approximation techniques (Borkar 2009) and test them environments where traditional methods are known to diverge (Bard) 1995). Finally, we want to investigate how PB VFs applied to supervised learning tasks or POMDPs, can avoid BPTT by mapping the weights of RNN to its loss.
9

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
We thank Paulo Rauber, Imanol Schlag, Miroslav Strupl, Róbert Csordás, Aleksandar Stanié, Anand Gopalakrishnan, Sjoerd Van Steenkiste and Julius Kunze for their feedback. This work was sup- ported by the ERC Advanced Grant (no: 742870). We also thank NVIDIA Corporation for donating a DGX-1 part of the Pioneers of Al Research Award and to IBM for donating a Minsky machine.
REFERENCES
Joshua Achiam. Spinning Up in Deep Reinforcement Learning. 2018.
Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. In Machine Learning Proceedings 1995, pp. 30-37. Elsevier, 1995.
Andrew J. Booker, J. E. Dennis, Paul D. Frank, David B. Serafini, and Virginia Torczon. Optimiza- tion Using Surrogate Objectives on a Helicopter Test Example, pp. 19-58, Birkhäuser Boston, Boston, MA, 1998. ISBN 978-1-4612-1780-0 doi: 10. 1110077978-1-461--77555
Vivek S Borkar. Stochastic opproximation: dynamical systems viewpoint, volume 48. Springer, 2009.
G. E. P. Box and K. B. Wilson. On the experimental attainment of optimum conditions. Journal of the Royal Statistical Society. Series B (Methodological), 13(1):1-45 1951. ISSN 00359246.
Corinna Cortes, Yishay Mansour, and Mchryar Mohri. Learning bounds for importance weighting.
In Advances in neural information processing systems, pp. 442-450, 2010.
Thomas Degris, Martha White, and Richard Sutton. Off-policy actor-critic. Proceedings of the 29th International Coference on International Conference on Machine Learning, ICML' 12, pp- 179-186, USA, 2012. Omnipress. ISBN 1--500--2882---
Jean Harb, Tom Schaul, Doina Precup, and Pierre-Luc Bacon. Policy evaluation networks. arXiv preprint arXiv:2002.1/833 2020.
Timothy Classen Hesterberg. Advances importance sampling. PhD thesis, Stanford University, 1988.
Ehsan Imani, Eric Graves, and Martha White. An off off-policy policy gradient thcorem using emphatic weightings. In Advances in Neural Information Processing Systems, pp. 96-106, 2018.
Max Jaderberg, Wojciech Marian Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, and Koray Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. In Pro- ceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1627-1635. JMLR. org, 2017.
Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Society for Industrial and Applied Math- ematics, 42, 04 2001.
Timothy Lillicrap. Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver. and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint wcXiv:/509.00977 2015.
Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Off-policy policy gradient with 2019.
state distribution correction. arXiv preprint
Hamid Maei. Csaba Szepesvári, Shalabh Bhatnagar, Doina Precup, David Silver, and Richard S. Sutton. Convergent temporal- differeeee learning with arbitrary smooth function approxima- tion. In Proceedings of the 22nd International Conference on Neural Information Processing Systems, NIPS109, pp. 1212, Red Hook, NY, USA, 2009. Curran Associates Inc. ISBN 9781615679119.
Hamid Reza Maei. Gradient temporal rimpora-diiference learning algorithms. PhD thesis, University of Alberta, 2011.
10

---

Published as a conference paper at ICLR 2021
Hamid Reza Maei, Csaba Szepesvári, Shalabh Bhatnagar, and Richard S. Sutton. Toward off-policy learning control with function approximation. In Proceedings of the 27th International Confer- on International Conference on Machine Learning, ICML'10. 10, pp. 719-726, Madison, WI, USA, 2010. Omnipress. ISBN 978 9781605589077.
Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search of static linear policies is competitive for reinforeement learning. In Advances in Neural Information Processing Systems, pp. 1800-1809, 2018.
Alberto Maria Metelli, Matteo Papini, Francesco Faccio, and Marcello Restelli. Policy optimization via importance sampling. In Advances Neural Information Processing Systems, pp. 5442-5454, 2018.
N. Metropolis and S. Ulam. The monte carlo method. J. Am. Stat. Assoc., 44:335, 1949.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 2015,
Andrew W. Moore and Jeff G. Schneider. Memory-based stochastic optimization. D. Touretzky, M. C. Mozer, and M. E. Hasselmo (eds.), Advances in Neural Information Processing Systems 8, pp. 1066-1072. MIT Press, 1996.
Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv:1912.0209: 2019.
Jan Peters and Stefan Schaal. Natural actor-critic. Neurocomput., 71(7-9): 190-1180-1190 March 2008.
ISSN 0925-2312. doi: 10. hetl///jeucomm.0000 11.026.
Doina Precup, Richard S. Sutton, and Sanjoy Dasgupta. Off-policy temporal difference learning
with function approximation. In ICML, 2001.
Martin I. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Reuven Y. Rubinstein and Dirk P. Kroese. Simulation and the Monte Carlo Method. Wiley Publish-
ing, 3rd edition, 2016. ISBN 1118632168.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable alternative reinforcement learning. arXiv preprint 10.3v:/7733038666 2017.
Tom Schaul, Dan Horgan, Karol Gregor, and David Silver. Universal value function approximators. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning Volume 37, ICML' 15, pp. 1312-1320. JMLR.org, 2015.
Jürgen Schmidhuber. Networks adjusting networks. Proceedings of" Distributed Adaptive Neural Information Processing", pp. 197-208. 1990.
Frank Sehnke, Christian Osendorfer. Thomas Rückstießß Alex Graves, Jan Peters, and Jürgen Schmidhuber. Policy gradients with parameter-basee based exploration for control. In Véra Kúrková, Roman Neruda, and Jan Koutník (eds.), Artificial Neural Networks ICANN 2008, pp. 387-396, Berlin, Heidelberg, 2008. Springer Berlin Heidelberg. ISBN 978-3-540-87536.9.
Frank Sehnke, Christian Osendorfer, Thomas Rückstieß, Alex Graves, Jan Peters, and Jürgen Schmidhuber. Parameter-exploring policy gradients. Neural Networks, 23(4)5551-55, May 2010. ISSN 08936080. doi: 10. 10.1011/j wwtp:.eenneee.00..2222
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In Proceedings of the 31st International Conference International Conference Machine Learning Volume 32, ICML' 14, pp. 1-387-1-395. JMLR.org, 2014.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In Advances in neural information processing systems, pp. 2951-2953 2012.

---

Published as a conference paper at ICLR 2021
Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Md. Mostofa Ali Patwary, Prabhat Prabhat, and Ryan P. Adams. Scalable bayesian optimization using deep neural networks. In Proceedings of the 32nd International Conference on International Conference on Machine Learning- Volume 37, ICML' 15, pp. 2171-2180. Jorg. 2015.
RL Stratonovich. Conditional Markov processes. Theory of Probability And Its Applications, 5(2):
156-178, 1960.
Richard Sutton. Temporal Credit Assignment in Reinforcement Learning. PhD thesis, University of Massachusetts Amherst, 1984.
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, (1):9 14,9444,17 1988.
Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Proceedings of the 12th International Conference on Neural Information Processing Systems, NIPS' 99, 1057-1063, Cambridge, MA, USA, 1999. MIT Press.
Richard Sutton, Hamid Maei, and Csaba Szepesvári. convergent o(n) temporal-different algorithm for off-policy learning with lincar function approximation. In Advances in neural in- formation processing systems, pp. 1609-1616, 2009a.
Richard S. Sutton, Hamid Reza Maei, Doina Precup, Shalabh Bhatnagar, David Silver, Csaba Szepesvári, and Eric Wiewiora. Fast gradient-descent methods for temporal-difference learning with linear function approximation. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML '09, pp. 993-1000, New York, NY, USA, 2009b. Association for Computing Machinery. ISBN 9781605585161. doi: (1145//553374.155333
Richard S. Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M. Pilarski, Adam White, and Doina Precup. Horde: A scalable real-time architecture for learning knowledge from unsu- pervised sensorimotor interaction. In The 10th International Conference Autonomous Agents and Multiagent Systems Volume 2, AAMAS '11, pp. 761-768, Richland, SC, 201 1. Interna- tional Foundation for Autonomous Agents and Multiagent Systems. ISBN 0-9826571-6-1 978- 0-9826571-6-4
Richard Sutton, A Rupam Mahmood, and Martha White. An emphatic approach to the problem of off-policy temporal-different learning. The Journal of Machine Learning Research, 17(1): 2603-2631, 2016.
Gerald Tesauro. Temporal difference learning and td id-gammon. Commun. (8():5,800)58-85 March 1995. ISSN 0001-0782, doi: 10. 145/203330.20334..
Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and Ilya Tolstikhin. Predict- ing neural network accuracy from weights. arXiv preprint arXiv:2002. 1448, 2020.
Ziyu Wang, Victor Bapst. Nicolas Heess, Volodymyr Mnih, Remi Munos. Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with experience replay. arXiv preprint arXiv:1611.01224. 2016.
Paul Werbos. Backpropagation through time: what does and how to do it. Proceedings of the IEEE, 78(10):550--1560 1990.
Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, and Jürgen Schmidhuber. Natural evolution strategies. The Journal of Machine Learning Research, 15(1):949-99) 980, 2014.
12

---

Published as a conference paper at ICLR 2021
A APPENDIX
INDEX OF THE APPENDIX
In the following, we briefly recap the contents of the appendix.
Appendix A.
contains additional related works
Appendix
reports all proofs and derivations.
Appendix
iluuttrats implementation details and pseudocode.
Appendix
provides the hyperparameters used in the experiments and further results.
A.1 ADDITIONAL RELATED WORKS
Recent work (Unterthiner et al., 2020) shows how to map the weights of trained Convolutional Neural Network to its accuracy. Experiments show how these predictions allow for performance rankings of neural networks flattened weights as input or using simple statistics. However, these predictions do not guide the training process of CNNs.
new unseen tasks. These maps either learned by taking the
In 1990, adaptive critics trained by TD used to predict the gradients of RNN from its activations (Schmidhuber, 1990), avoiding backpropagation through time (BPTT) (Werbos) 1990.. This idea was later used to update the weights of neural network asynchronously (Jaderberg et al. 2017). In our work, the critic is predicting errors instead of gradients. If applied to POMDPs, or supervised learning tasks involving long time lags between relevant events, the PSSVF could avoid BPTT by viewing the parameters of an RNN as a static object and mapping them to their loss (negative reward).
Additional differences between our work and Policy Evaluation Networks (PENs) (Hrb et al., 2020) concern the optimization problem: do not predict bucket index for discretized
reward, but perform a regression task. Therefore our loss is simply the mean squared error between
the prediction of V(0) and the reward obtained by æo, while their loss (Harb al. 2020 is the KI. divergence between the predicted and target distributions. Both approaches optimize the undiscounted objective when learning (0)).
A.2 PROOFS AND DERIVATIONS
Theorem 3.1. Let me be stochastic, For any Markov Decision Process, the following holds:
VoJ( -Esnat () (og[(s)))))))
Proof. The proof follows the standard approach by Sutton et al. (1999) and we report it for com-
pleteness. We start by deriving expression for Vo V (s, (s:
13

---

Published as a conference paper at ICLR 2021
(oV(s, Vo / (sl -/x Vozoldss)((s,a,0) molls))o(ss da
Vomolals]Q(sa0)) To [((s.a) + P(s'\s..)V da
/ Vomddlss((ssa00=) P(s/ss,a)VoV((s'0) da ( P()l|s,ax
Ve mo |/) / P(s P(s" d''''' 0) ds da' ds' da
P(s + -,,,t, /
Taking the expectation with respect to (o(s) we have:
VoJ(o) Vo / V 0) ds (ss))VoV(ss
mo(s) P((ss - s 1,, / (omo(als)Q(s,a,))) ds
1=
(=) Voro(als) a, 0) dads
Raadd" [(((s, a, V, log (a)s))))
Theorem 3.2. Let ne be deterministic. Under standard regularity assumptions (Silver et al. 2014), for any Markov Decision Process, the following holds:
(TT()))) = (e
QaQ(s, (, ())==) VoTo(s)]
Proof. The proof follows the standard approach by Silver al. 2014) and report it for com- pleteness. We start by deriving an expression for (s)(s,
(oV(s, = VoQ(s, (o(s),)) = Vo R(s (o()) + (os)
Vo
+7 L P(61l + V( mm)= ds' ( + / P(s'ls, (18) la=
=Vono(s)V.
+7 / P(s
(oTo())V.Q((55,00)))) + / P(o/ls,7003))
+7 (o7 () ds'+
+
((s))s.m0(3)) P(s")s,,=0())))VVV ds ds ds'
PPss + 1om ds
1=0
14

---

Published as a conference paper at ICLR 2021
Taking the expectation with respect to so (s) have:
VoJ(0) Vo / (s(s)V(ss
(o(s) vo(s)Voo(ss,0))
/ mo(s) 1 Σ P(s + s 20
a, 0) ao ds' ds
d=d ( (-))()) ds
(s) VOTT(S)V (
Theorem 3.3. For any Markov Decision Process, the following holds:
To T s) (Q(s, a, 0,Vo log +Vo((s,,,))
VoJo(TT)) E sod (s).1=m((11))
(14)
mb(als)
Proof.
VoJo(TO) = Vo dh (s) ds
(17)
de (s) / () dads
V8
(18)
d (8)
a VoTTo(als) + to(
(19)
(a(a)s)
(8)
(a(s)) 00 da (20)
Ta(a
(Q(s, 0)V, (og Vo QoQ 01)
Eood (s)(1))))))))
(21)
TT(a)s)
Theorem 3.4. Under standard regularity assumptions (Sivver (Slveee Process, the following holds:
2014) for any Markov Decision
VoJo(To) = -E (s) [= + VoQ(s, ())a=)))))ol
(16)
Proof.
VoJo(To)
d V
(22)
J
dmh (s) V. [o
VOTO Vol(. ds
(23)
E ( V. V (0 V (a===
(24)
A.3 IMPLEMENTATION DETAILS
A.3.1
In this appendix, we report the implementation details for PSSVE, PSVF, PAVE and the baselines. We specify for each hyperparameter, which algorithms and tasks are sharing them.
Shared hyperparameters:
15

---

Published as a conference paper at ICLR 2021
Deterministic policy architecture (continuous control tasks): We three different deter- ministic policies: a linear mapping between states and actions; a single-layer MLP with 32 neurons and tanh activation; a 2-layers MLP (64,64) with tanh activations. All policies contain a bias term and are followed by tanh nonlinearity in order to bound the action.
Deterministic policy architecture (discrete control tasks): We use three different determin- actions;
istic policies: linear mapping between states and probability distribution a single-layer MLP with 32 neurons and tanh activation; a 2-layers MLP (64,64) with tanh activations. The deterministic action is obtained choosing policies contain a bias term.
arg max To (als). All
Stochastic policy architecture (continuous control tasks): We use three different stochastic policies: a linear mapping: a single-layer MLP with 32 neurons and tanh activation; a 2- layers MLP (64,64) with tanh activations all mapping from states to the mean of a Normal distribution. The variance is inte-inndeendent and parametrized All policies contain a bias term. Actions sampled are given as input to a tanh nonlinearity in order to bound them in the action space.
e2 with diagonal Ω.
Stochastic policy architecture (discrete control tasks): We use three different deterministic policies: linear mapping between states and a probability distribution over actions; a single-layer MLP with 32 neurons and tanh activation; 2-layers MLP (64,64) with tanh activations. All policies contain a bias term.
Policy initialization: all weights and biases are initialized using the default Pytorch initial- ization for PBVFs and DDPG and are set to zero for ARS.
Critic architecture: 2-layers MLP (512,512) with bias and ReLU activation functions for PSVF, PAVF; 2-layers MLP (256,256) with bias and ReLU activation functions for DDPG.
Critic initialization: all weights and biases are initialized using the default Pytorch initial- ization for PBVEs and DDPG.
Batch size: 128 for DDPG, PSVE, PAVE; 16 for PSSVE.
Actor's frequency of updates: every episode for PSSVF: every batch of episodes for ARS; every 50 time steps for DDPG, PSVF, PAVE.
Critic's frequency of updates: every episode for PSSVF; every 50 time steps for DDPG, PSVF, PAVE.
Replay buffer: the size is 100k; data sampled uniformly.
Optimizer: Adam for PBVFs and DDPG.
Tuned hyperparameters:
Number of directions and elite directions for ARS ([directions, elite directions]): tuned with values in [[1, 1], [4, 1), 44, [16, 1], [16, 4]. [16, 16]].
Policy's learning rate: tuned with values in [le - 2, le - 3, le - 4].
Critic's learning rate: tuned with values in [le - 2, 1c 3, 4].
Noise for exploration: the perturbation for the action (DDPG) or the parameter is sampled from (0, ol) with tuned with values in [1, le 1] for PSSVE, PSVE, PAVE; [le 1, le - 2] for DDPG; [1, le - 1, le 2, le 3) for ARS. For stochastie PSSVF and PSVF we include also the value 0, although it almost never results optimal.
Environment hyperparameters:
Environment interactions: IM time steps for Swimmer-v3 and Hopper-v3; 100k time steps for all other environments.
Discount factor for TD algorithms: 0.999 for Swimmer; 0.99 for all other enviromments.
Survival reward in Hopper: True for DDPG, PSVE, PAVE; False for ARS, PSSVE
Aggorich hyperparameters:
Critic's number of updates: 50 for DDPG, 5 for PSVF and PAVE; 10 for PSSVE
16

---

Published as a conference paper at ICLR 2021
Actor's number of updates: 50 for DDPG, 1 for PSVF and PAVE; 10 for PSSVE.
Observation normalization: False for DDPG; True for all other algorithms.
Starting steps DDPG (random actions and no training): first 1%.
Polyak parameter DDPG: 0.995.
PAVF VoQ(s, a, 8) ablation We investigate the effect of the term VoQ(s, a) in the off- policy gradient theorem for deterministic PAVE. We follow the same methodology as in our main experiments to find the optimal hyperparameters when updating using the now biased gradient:
VoJo(TT) B (2
(a) Q a, ())===(()) [oT
(25)
which corresponds to the gradient that DDPG is following. Figure 5 reports the results for Hopper and Swimmer using shallow and deep policies. We observe a sigmificant drop in performance in Swimmer when removing part of the gradient. In Hopper the loss of performance is less significant, possibly because both algorithms tend to converge to the same sub-optimal behavior.
Swimmerv3
Swimmer v3
Hopper
Hopper (64.
200
3000
3000
303-
253
3200
1200
PWVT--
203
150
803
000
0000
102
600
92
200
200
-100
-200
1e6
Ume stops
1.6
1.6
0me alcps
196
Figure 5: Performance of PAVF and biased PAVF (PAVF without the gradient of the action-valus function with respect to the policy parameters) using deterministic policies. We use the hyperparam- eters maximizing average return and report the best hyperparameters found for the biased version in Table I Learning curves are averaged over 20 seeds.
Table I: Table of best hyperparameters for biased PAVFs
Learning rate policy
Policy:
[]
[64,64]
Metrie:
avg
avg
Swimmer-v3
le-3
Ic-4
H3pper-v3
1e-4
le-4
Learning rate critic
Swimmer-v3
le-4
le-4
Hopper-v3
1e-3
le-3
Noise for exploration
Swimmer-v3
1.0
Hopper-v3
0.1
0.1
ARS For ARS, we used the official implementation provided by the authors and we modified in order to use nonlinear policies. More precisely, we used the implementation of ARSv2-t (Mania) et al. 2018), which uses observation normalization, clite directions and an adaptive learning rate based on the standard deviation of the return collected. To avoid divisions by zero, which may happen if all data sampled have the same return, we perform the standardization only in case the standard deviation is not zero. In the original implementation of ARS MMania et al. 2018), the survival bonus for the reward in the Hopper environment is removed to avoid local minima. Since we wanted our PSSVF to be close to their setting, we also applied this modification. We did not remove the survival bonus from all TD algorithms and we did not investigate how this could affect their performance. We provide a comparison of the performance of PSSVF with and without the bonus in figure 6 using deterministic policies.
17

---

Published as a conference paper at ICLR 2021
1800
5.00
Figure 6: Performance of PSSVF with and without the survival bonus for the reward in Hopper-v3 when using the hyperparameters maximizing the average return. Learning curves are averaged over 5 seeds.
DDPG For DDPG, we used the Spinning Up implementation provided by OpenAl (Achiam 2018), which includes target networks for the actor and the critic and learning for fixed set of time steps, called starting steps. We did not include target networks and starting steps in our PBVFs, although they could potentially help stabilizing training. The implementation of DDPG that we used (Achiam 2018) does not use observation normalization. In preliminary experiments we observed that it failed significantly increase or decrease performance, hence we did not use it. Another difference between our TD algorithms and DDPG consists in the number of updates of the actor and the critic. Since DDPG's critic needs to keep track of the current policy, the critic and the actor updated in a nested form, with the first's update depending on the latter and vice versa. Our PSVF and PAVF do not need to track the policy learned, hence, when time to update, we need only to train once the critic for many gradient steps and then train the actor for many gradient steps. This requires less compute. On the other hand, when using nonlinear policies, our PBVFs suffer the curse of dimensionality. For this reason, we profited from using bigger critic. In preliminary experiments, we observed that DDPG's performance did not change significantly through bigger critic. We show differences in performance for methods when removing observation normaliza- tion and when using smaller critic (MLP(266,256)) in figure 7 We observe that the performance is decreasing if observation normalization is removed. However, only for shallow policies in Swimmer and deep policies in Hopper there seems to be significant benefit. Future work will assess when bigger critics help.
Sei m2
Swimmor +3 (64, 64)
Hopper v3 []
Hoppor-v2 [64, 64)
2000
302
2000
smalee noom
300
255
202
5 200
:55
1000
105
300
500-
500
0.0
1.0
0.4
3.6
266
Sie
Swimmerv3 (64, 64)
Happor
1750
2000
2A47 aralle attc nor
250
1500-
200
1566-
350
300
-50
2.5
0.5
1971 10000
106
1000 strgs
246
166
11000 Maas
Figure 7: Learning curves for PSVF and PAVE for different environments and policies removing observation normalization and using smaller critic. We use the hyperparameters maximizing the average return. Learning curves are averaged over 5 seeds. For this ablation we use deterministic policies.
18

---

Published as a conference paper at ICLR 2021
Discounting in Swimmer For TD algorithms, chose fixed discount factor 7 0.99 for all environments but Swimmer-v3. This environment is known to be challenging for TD based algo- rithms because discounting causes the agents to become too short-sighted. We observed that, with the standard discounting, DDPG, PSVF and PAVF were not able to learn the task. However, making the algorithms more far-sighted greatly improved their performance. In figure we report the return obtained by DDPG, PSVF and PAVF for different values of the discount factor in Swimmer when using deterministic policies.
104,
230-
Figure 8: Effeec different choices of the discount factor Swimmer-v3 for PSVE, PAVF and DDPG, with shallow and deep deterministic policies. We use the hyperparameters maximizing the average return. Learning curves are averaged over seeds
A.3.2 PSEUDOCODE
Algorithm 2 Actor-critic with TD prediction for V(8
Input: Differentiable critic Vn S x R with parameters w; deterministic or stochastic actor with parameters 0; empty replay buffer D
Output Learned Vr Rs V(s, 0), learned TT Rs mg-
Initialize critic and actor weights w.
repeat:
Observe state s, take action a = (s)),, observe reward r and next state Store (s, r, s') in the replay buffer if it's time to update then:
for many steps do:
Sample batch B, {(s, r'),s')) from D Update critic by stochastic gradient descent: Vn [B. (e.. (v, (r + (V 0])]
end for
for many steps do:
Sample batch B2 {s)) from D
Update actor by stochastic gradient ascent: Ve [B
E.c B, (s,
end for end until convergence
19

---

Algorithm 4 Deterministic actor-critic with TD prediction for Q(s, a, Published as a conference paper at ICLR 2021
Algorithm 3 Stochastic actor-critic with TD prediction for a,)
Input: Differentiable critic Qn S x A x - R with parameters w; stochastic diffcrentiable actor To with parameters 8; empty replay buffer D
Output Learned Qw = Q(s, a, f), learned Th = ng-
Initialize critic and actor weights w. repeat:
Observe state take action No (s), observe reward and next state s' Store (s, a, 0, r, 8') in the replay buffer if it's time to update then:
for many steps do:
Sample batch B1 = {(s, a, (,,r,''), from D Update critic by stochastic gradient descent:
V Bil (s.a,.r.:")) (0, a,0) (r + 7Q a' aT(s'),)))
end for
for many steps do:
Sample batch BN = {(s a, } from D Update actor by stochastic gradient ascent:
[Ba] (s.0,0)en
E
(a(a])) (Q(s, V) log (a(as)) VoQ(s,,,
[a(a]e)
end for end if until convergence
Input: Differentiable critic Qw x A @ R. with parameters w; differentiable determin- istic actor with parameters 0; empty replay buffer D) Output: Learned QN Qss, a, O). learned my Rz n
Initialize critic and actor weights w,O repeat:
Observe state s, take action a = aa (s). observe reward r and next state Store (s, a, s') in the replay buffer D) it's time to update then:
for many steps do:
Sample batch B, = [ a, r. r.)r) 8')) from D
Update critic by stochastic gradient descent:
V "i end for for many steps do:
E (s...d....)))B (Q, (r + (m Tg(s),
Sample batch Bg {(s)} from D
Update actor by stochastic gradient ascent:
Bal E,EB, (VoTo(s)))) V. a, 0))a=so(s) +
end for end if until convergence
A.4 EXPERIMENTAL DETAILS
A.4.1 LQR
For our visualization experiment, we employ an instance of the Linear Quadratic Regulator. Here, the agent observes a I-D state, corresponding to its position and chooses 1-D action. The transi- tions are s' = s + a and there is a quadratic negative term for the reward: R(s,a) = - a² The agent starts in state $o and acts in the environment for 50 time steps. The state space is bounded [-2,2]. The goal the agent is to reach and remain in the origin. The agent is expected perform small steps towards the origin when it uses the optimal policy. For this task, use
20

---

Published as a conference paper at ICLR 2021
deterministic policy without tanh nonlinearity and we do not observation normalization. Below additional details and plots for different algorithms.
PSSVF We use learning rate of 1e 3 for the policy and 1e 2 for the PSSVF. Weights perturbed every episode using o = 0.3. The policy is initialized with weight 3.2 and bias -3.5. All the other hyperparameters 10,000 policies in the environment with parameters in [-5, 5] x [-5, 5]. (d) is computed by measuring the output of the PSSVF on the same set of policies. Each red arrow in figure reereeett 200 update steps of the policy.
set to their default. The true episodic J(0) is computed by running
PSVF ad PAVE Using the exact same setting, we compare learned (s0. 0) and (s0, (s)))), 0) with the true PSVF and PAYF over the parameter space. Computing the value of the true PSVF and PAVF requires computing the infinite sum of discounted reward obtained by the policy. Here we approximate it by running 10,000 policies in the environment with parameters in [-5, 5, -5] x [-5,5] for 500 time steps. This, setting 7 0.99, provides a good approximation of their true values, since further steps in the environment result in almost zero discounted reward from so. We use a leaming rate of 1e 2 for the policy and 1e for the PSVF and PAVE. Weights are perturbed every episode using every time steps using 2 gradient steps; the PSVF and PAVF are updated every time steps using 10 gradient updates. The critic is I-layer MLP with 64 neurons and tanh nonlinearity.
PSVF and PAVF in LQR environment and
0.5. The policy is updated
In Figures and 10 report J(0), the cumulative discounted reward that agent would obtain by acting in the environment for infinite time steps using policy Tp and the cumulative return predicted by the PSVF and PAVF for different times during learning. Like in the PSSVF experiment, the critic is able improve its predictions over the parameter space. Since in the plots (s,, 0) and Q(s, To(s), 0) are evaluated only in 8y, the results show that PBVFs able to effectively bootstrap the values of future states. Each red arrow in Figures and 10 represents 50 update steps of the policy.
Optimization after 15 episodes
((0)
(50 8)
489
1600
Retum
4471
1
7343
10214
33
13086
-5-
-5
8,
61
Optimization after 100 episodes
J(O)
Va 0)
Return
-4171
7313
10214
33
13086
-5
01
01
Figure 9: True cumulative discounted reward J(0) and PSVF estimation Vo 0) as a function of the policy parameters two different stages in training. The red arrows represent optimization trajectory in parameter space. The blue dots represent the perturbed policies used to train V,, (So, 0).
21

---

Published as a conference paper at ICLR 2021
Optimization after 15 episodes
)(0)
Qu(so, (())))
459
141
1600
Return
4471
7343
10214
33
13086
-5
-5
Optimization after 100 episodes
(())
Qu To (80)
489
141
1600
Return
4471
7343
10214
13086
-5
-3
3
-5
33
61
Figure 10: True cumulative discounted reward (()) and PAVF estimation Qw ne (e(s)) 0) function of the policy parameters at two different stages training. The red arrows represent an optimization trajectory in parameter space. The blue dots represent the perturbed policies used to train Q (8o, (s())),
A.4.2 OFFLINE EXPERIMENTS
Zero-shot learning We evaluate the performance the policies learned from scratch evaluating them with 5 test trajectories every 5 gradient steps. In addition to the results in the main paper, we report in Figures IT and 12 comparison of zero-shot performance between PSSVF, PSVF and PAVF in three different enviromments using deterministic shallow and deep policies (2-layers MLP(64,64)). In this task we use the same hyperparameters found in tables 6 and g One additional hyperparameter needs to be considered: the learning rate of the policies trained from scratch. In Figure 3 of the main paper, we use a tuned learning rate of 0.02 that we found working particularly well for PSSVF in the Swimmer environment. In the additional experiments in Figures 11 and 12, we use a learning rate of 0.05 that we found working well across all policies, environments and algorithms when learning zero-shot.
We observe that, using shallow policies, PBVFs can effectively zero-shot learn policies with performance comparable to the policy learned in the environment without additional tuning for the learning rate. We note the regular presence of a spike in performance followed by a decline due to the policy going to regions of the parameter space never observed. This suggests that there trade-off between exploiting the generalization of the critic and remaining in the part of the parameter space where the critic is accurate. Mcasuring the width of these spikes can be useful for determining the number offline gradient steps to perform in the general algorithm. When using deep policies the results become much worse and zero-shot learned policies performance of the main policy being learned only in simple environments and at beginning of training (eg. MountanCarrConimuon the replay buffer contains policies that are very distant to policies randomly initialized. This might explain why the zero-shot performance is better sometimes the beginning of training (eg. second column). However, since PB VFs in practice perform mostly local off- aff-policy evaluation around the learned policy, this problem is less prone to arise in our main experiments.
recover the
We observe that, when the critic is trained (last column),
Offline learning with fragmented behaviors In this task, data are generated by perturbing a randomly initialized deterministic policy every 200 time steps and using it to act in the environment.
22

---

Published as a conference paper at ICLR 2021
1230
120
6.0MM
1
Figure 11: Shallow policies learned from scratch during training. The plots in the left column repre- sent the return of agents learning while interacting with the environment using different algorithms. The learning curves in the other plots represent the return obtained by policies trained from scratch following the fixed critics after different time steps of training. Zero-shot learning curves are aver- aged over seeds.
We use 0.5 for the perturbations. After the dataset is collected, the PSVF is trained using learning rate of le - 3 with a batch size of 128. When the policy is learned, we use a learning rate of 0.02. All other hyperparameters are default values.
A.4.3 FULL EXPERIMENTAL RESULTS
Methodology In order to ensure a fair comparison of our methods and the baselines, we adopt the following procedure. For each hyperparameter configuration, for each environment and policy architecture, we run 5 instances of the learning algorithm using different seeds. We measure the learning progress by running 100 evaluations while learning the deterministie policy (without action or parameter noise) using 10 test trajectories. We use two metrics to determine the best hyperparam- eters: the average return over policy evaluations during the whole training process and the average return over policy evaluations during the last 20% time steps. For each algorithm, environment and policy architecture, we choose the two hyperparameter configurations maximizing the performance of the two metrics and test them on 20 new seeds, reporting average and final performance in table and respectively.
Figures 13 and 14 report all the learning curves from the main paper and for small non linear policy with 32 hidden neurons.
Stochastic policies We include some results for stochastic policies when using PSSVF and PSVE. Figures 15 and [1)sshow a comparison with the baselines when using shallow and deep policies re- spectively. We observe results sometimes comparable, but often inferior with respect to deterministic policies. In particular, when using shallow policies, PBVTs are able to outperform the baselines in the MountainCar environment, while obtaining comparable performance in CartPole and Inverted- Pendulum. Like in previous experiments, PBVFs fail to learn good policy in Reacher. When using deep policies, the results are slightly different: VBVFs outperform ARS and DDPG in Swimmer, but fail to learn InvertedPendulum. Although the use of stochastic policies can help smoothing the
23

---

Published as a conference paper at ICLR 2021
Samore 2ww.r--0--
-3 34
Deep policies learned from scratch during training. The plots in the left column represent the return of agents learning while interacting with the environment using different algorithms. The learning curves in the other plots represent the return obtained by policies trained from scratch fol- lowing the fixed critics after different time steps of training. Zero-shot learning curves are averaged over seeds.
Figure 12:
objective function and allows the agent exploring in action space, we believe that the lower variance provided by deterministic policies can facilitate learning PBVFs.
A.4.4 SENSITIVITY ANALYSIS
In the following, report the sensitivity plots for all algorithms, for all deterministic policy
20 and 21 show the performance of each algorithm given different hyperparameters tried during training We observe that in gen- eral deep policies are more sensitive and, apart for DDPG, achieve often a better performance than smaller policies. The higher sensitivity displayed by ARS in part caused by the higher number of hyperparameters we tried when tuning the algorithm.
chitectures and environments. In particular, figure 17
18
19
A.4.5
TABLE OF BEST HYPERPARAMETERS
We report for each algorithm, environment, and policy architecture the best hyperparameters found when optimizing for average return or final return in tables and
24

---

Published as a conference paper at ICLR 2021
Table Average return with standard deviation (across 20 seeds) for hypermarameters optimizing the average return during training using deterministic policies. Square brackets represent the number of neurons per layer the policy. []
represents linear policy.
Policy: []
MountainCar Continuous-vl vO
Inverted Pendulum-v2
Reacher -v2
Swimmer -v3
Hopper -v3
ARS
63 + 6
886 + 72
-9.2 0.3
228 + 89
1184 + 345
PSSVF
85 4
944 33
-11.7
259 47
1392 + 287
DDPG
0 +
612 = 169
-8.6
95
629 + 145
PSVF
84 + 20
926 34
-19.7
188 + 71
917 + 249
PAVF
82 21
913 40
-17.0
231 + 56
814 + 223
Policy:[32]
ARS
37 + 11
851 46
0.3
139 78
1003 +
PSSVF
60 33
701 = 138
0..4
189 - 35
707 + 668
DDPG
0
816 - 36
0.3
62
1384 125
PSVF
71 + 25
529 = 281
1.2
226 + 33
864
PAVE
71 27
563 + 228
-00.9
222 28
793 + 322
Policy: [64,64]
ARS
28 8
812 + 239
0.3
129 + 68
964 47
PSSVF
72 + 22
850 + 93
0.2
158 59
922 + 568
DDPG
0 0
834 + 36
-.55
917
767 627
PSVE
80
580 = 107
-.0.7
137 - 38
843 + 282
PAVE
73 + 10
399 = 219
0.5
142 + 26
875 + 136
Policy: []
Acrobot-vl
CartPole-vl
ARS
-161 + 23
476 + 13
PSSVF
137 + 14
443 + 105
PSVF
144 + 25
459 28
Policy:[32]
ARS
296 + 38
395 - 141
PSSVF
-551 + 80
463 + 18
PSVE
270 + 113
413 +
Policy: [64,64]
ARS
-335 + 35
416 + 105
PSSVF
-281 117
452 34
PSVF
-397 + 71
394 + 71
25

---

Published as a conference paper at ICLR 2021
Table 3: Final return with standard deviation (across 20 seeds) for hypermarameters optimizing the final return during training using deterministic policies.
Policy: 0
MountainCar
Inverted
Reacher -v2
Swimmer -v3
Hopper -v3
Continuous-vo
Pendulum-v2
ARS
73 + 5
657 + 477
-8.6 0.5
334 34
1443 + 713
PSSVF
84 + 28
970 = 126
-10.0 1.0
350 8
1560 + 911
DDPG
777 320
-.3
146
152
704 = 234
PSVF
76 + 36
906 + 289
-16.5 1.6
238
+ 107
1067 + 340
PAVF
68 42
950 + 223
-17.2 15.4
298 40
720 + 281
Policy:[32]
ARS
54 + 20
936 + 146
-9.2
239 117
1048 + 68
PSSVF
89 22
816 + 234
294
1204 615
DDPG
0
703 283
0.6
1290 348
PSVF
84 + 31
493 462
-11.3
290 70
1003 572
PAVE
92 + 7
854 + 295
-10.1 = 0.9
307 + 34
967 + 411
Policy: [64,64]
ARS
11 + 30
976 83
0.4
157 + 54
1006 + 47
PSSVF
91 16
898 + 227
1.0.7
224
1412 691
DDPG
0 0 0
943 -
-.44
196 - 151
1437 + 752
PSVF
93 1
1000 =
-10.6
257
1247 344
PAVE
93 + 2
827 267
-10.6 0.4
232
1005 155
Policy: []
Acrobot-vl
CartPole-vl
ARS
126 + 26
499 + 2
PSSVF
-97 + 6
482 + 53
PSVF
100 + 18
500 0
Policy:[32]
ARS
215 + 97
471 + 110
PSSVF
-16 + 33
500 + 0
PSVF
244 151
488 36
Policy: [64,64]
ARS
-182 + 45
492 18
PSSVF
-233 + 139
500 + 0
PSVF
-406 + 51
499 2
26

---

Published as a conference paper at ICLR 2021
Deterministic policies
parp
Hipperspo
200
C00
InvertecTeen Tw +2-04 S4
Tourr Pardiimm
2020
305
1000
60M
2004
00/0
Learning curves representing the average return as function of the number of time steps in the environment (across 20 runs) with different environments and deterministic policy architec- tures. We use the best hyperparameters found while maximizing the average reward for each task. For each subplot, the square brackets represent the number of neurons per policy layer. represents a linear policy.
Figure 13:
27

---

Published as a conference paper at ICLR 2021
Deterministic policies
1xppr--3.p4.0 A4
Precthmn2000
Teutherdg
-Soa CCM
1:300
C.N
1000
1100
3.00
0C30
3.5M2
201
Maii tinoarta sinase mt
2412
5012
2.00
0002
2,50
Figure 14: Learning curves representing the average return function of the number of time steps in the environment (across 20 runs) with different environments and deterministic policy architec- tures. We use the best hyperparameters found while maximizing the final reward for each task. For each subplot, the square brackets represent the number of neurons per policy layer. [] represents linear policy.
Stochastic shallow policies
Hopperd
Beacheed
'90m
Figure 15: Learning curves representing the average return as function of the number of time steps in the environment (across 20 runs) with different environments using stochastic shallow policies. We use the best hy hyperparameters found while maximizing the average reward for each task.
28

---

Published as a conference paper at ICLR 2021
Stochastic deep policies
Happerd2
Reachee
2013
Oweer menii
Figure 16: Learning curves representing the average return as a function of the number of time steps in the environment (across 20 runs) with different environments using stochastic deep policies ([64,64)). We use the best hyperparameters found while maximizing the average reward for each task.
Acsttend- 682
Acribon- +1004
5:90
20:00
50.00
530
C:MM
2010
0.500
3.504
COM
2000
150
70:00
5.0
medan
50-000
20:00
Xe
40:00
Fodo
10100
530
300
C/22M
C-CU
0.00
Figure 17: Sensitivity of PSSVFs using deterministic policies to the choice of the hyperparameter. Performance is shown by percentile using all the learning curves obtained during hyperparameter tuning. The median performance is depicted dark line. For each subplot, the numbers in the square brackets represent the number of neurons per layer of the policy. [] represents a linear policy.
29

---

Published as a conference paper at ICLR 2021
Annneii ER2
Aprocc +1.04 64
7990
630
101
30/70
2:01
3,00
CCOM
CCM
AM
CCM
Carhhin (4)
2070
30:00
530
C.OM
C.SMM
0.80
3.50
.CM
20:00
0.00
70:00
5.0
reday
300
150
10000
30.00
530
0.50M
Figure 18: Sensitivity of PSVFs using deterministic policies to the choice of the hyperparameter. Performance is shown by percentile using all the learning curves obtained during hyperparameter tuning, The median performance is depicted as dark line. For each subplot, the numbers in the square brackets represent the number of neurons per layer of the policy. [] represents a linear policy.
30

---

Published as a conference paper at ICLR 2021
7090
0.MM
3.3M
60OM
CCM
CCO
7-:00
20/00
100
50-00
20/00
5.3
Harpor 0
Prggan
Iappee jut
50.000
150
5.0
reday
125
Figure 19: Sensitivity of PAVEs using deterministic policies to the choice of the hyperparameter. Performance is shown by percentile using all the learning curves obtained during hyperparameter tuning, The median performance is depicted as dark line. For each subplot, the numbers in the square brackets represent the number neurons per layer of the policy.
31

---

Published as a conference paper at ICLR 2021
Se www.CaaaCoooeaaa woo 94
7099
KCOMM
CCM
CCM
7o:00
20/00
-1Pmm
50-00
20/70
530
Iapple
$0.000
6.30
Fedaa
1250 4.300
4.00
Figure 20: Sensitivity of DDPG to the choice of the hyperparameter. Performance is shown by percentile using all the learning curves obtained during hyperparameter tuming. The median perfor- mance is depicted as dark line. For each subplot, the numbers in the square brackets represent the number neurons per layer of the policy.
32

---

Published as a conference paper at ICLR 2021
Acnoeii EX2
Aprotoo 0100 64
7090
630
30/70
2070
30:00
530
3.500
.CM
20:00
5.30
roday
300
150
30.70
5.0
6.5MM 100
Figure 21: Sensitivity of ARS to the choice of the hyperparameter. Performance is shown by percentile using all the learning curves obtained during hyperparameter tuming. The median perfor- mance is depicted as dark line. For each subplot, the numbers in the square brackets represent the number neurons per layer of the policy.
33

---

Published as a conference paper at ICLR 2021
Table 4: Table of best hyperparameters for PSSVFs using deterministic policies
Learning rate policy
Policy:
[32]
[64,64]
Metric:
avg
last
avg
avg
last
Acrobot-vl
le-2
le-3
le-4
le-4
le-4
le-4
MountannCarCnniiuuuu-000
le-2
le-3
le-4
le-4
le-4
le-4
CartPole-vl
le-3
le-3
le-3
le-3
le-4
le-4
Swimmer-v3
le-3
le-3
le-3
le-3
le-2
le-4
InvertedPendulum-22
le-3
le-3
le-3
le-3
le-4
le-4
Reacher-v2
le-4
le-4
le-4
le-4
le-4
le-4
Hopper-v3
le-4
le-4
le-4
le-3
le-4
le-4
Learning rate critic
Acrobot-vI
le-2
le-3
le-2
le-2
le-2
le-2
MountainCarCCntinuussww
le-3
le-2
le-3
le-2
le-2
le-2
CartPole-vl
le-2
le-2
le-3
le-3
le-2
le-2
Swimmer-v3
le-3
le-3
le-2
le-2
le-3
le-2
InvertedPendulum--2
le-2
le-2
le-3
le-2
le-3
le-3
Reacher-v2
le-3
le-3
le-3
le-3
le-4
le-4
Hopper-v3
le-3
1e-3
le-2
le-2
le-2
le-2
Noise for exploration
Acrobot-vI
1.0
1.0
le-1
le-1
le-1
le-1
MountanCCarConinuuss-00
1.0
1.0
le-1
le-1
le-1
le-1
CartPol-vvl
1.0
1.0
1.0
1.0
le-1
le-1
Swimmer-v3
1.0
1.0
1.0
1.0
1.0
le-1
InvertedPendulum--2
1.0
1.0
1.0
le-1
le-1
Reacher-v2
lc-1
le-1
le-1
le-1
le-1
lc-1
Hopper-v3
1.0
1.0
le-1
1.0
le-1
le-1
Table 5: Table of best hyperparameters for ARS
Learning rate policy
Policy:
[32]
[64,64
Metric:
avg
last
avg
last
avg
last
Acrobot-v1
le-2
le-3
le-2
le-2
le-2
le-2
MountinCCrrCnninuuss-000
le-2
le-2
le-2
le-2
le-2
le-2
CartPole-vl
le-2
le-2
le-2
le-2
le-2
le-2
Swimmer-v3
le-2
le-2
le-2
le-2
le-2
le-2
InvertedPendulum-v2
le-2
le-2
le-2
1e-2
le-2
le-2
Reacher-v2
le-2
le-2
le-3
le-2
le-3
le-3
Hopper-v3
le-2
le-2
le-2
le-2
1e-2
lc-2
Number of directions
and elite directions
Acrobot-vl
(4,4)
(4,4)
(.))
(.1)
0.3
833
MounainCarConniuuuss-o00
(1,1)
(1,1)
(64
0.3
08889
CartPole-vl
(4,4)
(4,4)
Swimmer-v3
(1,1)
(1.1)
(11
011
020 (16.4)
InvertedPendulum-v2
(4,4)
(4.4)
44.4)
Reacher-v2
(16,16)
(16,16)
0.3
0.1
Hopper-v3
(4.1)
(4.1)
(1.1)
Noise for exploration
Acrobot-vl v1
le-2
lc-3
le-1
le-1
le-1
le-1
MoumtinCarronniiuuus-o0
le-1
lc-1
le-1
le-1
le-1
le-1
CartPole-vl
le-2
le-2
le-1
le-1
le-2
le-2
Swimmer-v3
le-1
le-1
le-2
le-1
le-l
le-1
InvertedPendulum--2
le-2
le-2
le-2
le-2
le-2
le-2
Reacher-v2
le-2
le-2
le-2
le-2
le-2
le-2
Hopper-v3
le-1
le-l
le-1
le-1
le-1
le-1
34

---

Published as a conference paper at ICLR 2021
Table 6: Table of best hyperparameters for PSVFs using deterministic policies
Learning rate policy
Policy:
[32]
[64,64]
Metric:
avg
last
avg
last
avg
last
Acrobot-vl
le-2
le-2
le-4
le-4
le-
le-2
MountinCarCCotinuuus-00
le-2
le-3
le-2
le-4
1e-3
le-4
CartPole-vl
le-2
le-2
le-2
le-4
le-3
lc-4
Swimmer-v3
le-3
le-3
le-3
le-3
le-3
lc-3
InvertedPendulum-v2
le-2
le-3
le-4
le-4
le-4
le-4
Reacher-v2
le-3
le-2
le-4
le-4
le-4
le-4
Hopper-v3
le-3
le-3
le-4
le-4
le-4
le-3
Learning rate critic
Acrobot-v1
le-3
lc-4
le-2
le-2
le-3
le-2
MountannCrrCntinuuus-000
le-4
le-3
le-2
le-4
le-3
le-3
CartPole-Vl
le-2
le-2
le-2
le-3
le-2
lc-4
Swimmer-v3
le-4
le-4
le-4
le-4
le-4
le-4
InvertedPendulum--22
le-3
le-2
le-3
le-4
le4
le-3
Reacher-v2
le-2
le-2
le-3
le-3
le44
le-4
Hopper-v3
le-2
le-2
le-4
le-4
le-2
le-4
Noise for exploration
Acrobot-vl
1.0
1.0
le-1
le-1
le-1
lc-1
MountainCarCCnninuuusss
1.0
le-1
le-1
1.0
le-1
le-1
CartPole-vl
1.0
1.0
1.0
le-1
le-1
le-1
Swimmer-v3
1.0
1.0
1.0
1.0
1.0
1.0
InvertedPendulum--2
1.0
1.0
le-1
le-1
le-1
le-1
Reacher-v2
1.0
1.0
1.0
1.0
le-1
le-1
Hopper-v3
1.0
1.0
le-1
le-1
le-1
1.0
Table 7: Table of best hyperparameters for PSSVEs and PSVFs using stochastic policies
Algo:
PSSVF
PSVF
Learning rate policy
Policy:
[64,64]
[64,64]
Metric:
avg
avg
avg
avg
Acrobot-vl
lc-2
lc-2
le-2
Ic-3
MountainCarConttiuuoss0000
le-2
le-3
le-2
le-3
CartPole-vl
le-3
le-4
le-2
le-3
Swimmer-v3
le-2
le-4
1e-3
le-4
InvertedPendulum-22
le-3
le-4
le-2
le-3
Reacher-v2
le44
le-3
le-2
le-2
Hopper-v3
le-4
le-4
le-3
le-4
Learning rate critic
Acrobot-vl
le-2
le-4
le4
le-2
MountanCCarCntinuuus-00
le-2
le-2
le-3
le-3
CartPole-vl
le-2
le-3
le-2
le-2
Swimmer-v3
le-2
le-3
le-3
le-4
InvertedPendulum-v2
le-3
le-3
le-3
le-2
Reacher-v2
le-3
le-3
le-3
le-3
Hopper-v3
le-3
le-2
le-2
le-4
Noise for exploration
Acrobot-vl
1.0
1.0
1.0
1.0
MoumtainCarronninuuu--000
1.0
le-1
1.0
le-1
CartPole-vl
1.0
1.0
1.0
le-1
Swimmer-v3
1.0
le-1
1.0
le-1
InvertedPendulum--2
1.0
1.0
1.0
le-1
Reacher-v2
le-1
0.0
1.0
0.0
Hopper-v3
1.0
1e-1
1.0
le-1
35

---

Published as a conference paper at ICLR 2021
Table 8: Table of best hyperparameters for PAVFs using deterministic policies
Learning rate policy
Policy:
[32]
[64,64]
Metric:
avg
last
avg
last
avg
last
MounannCarContiuuusu-00
le-2
1e-3
le-3
le-4
le4
le-4
Swimmer-v3
le-3
le-3
1e-3
le-3
le-3
le-3
InvertedPendulum-22
le-2
le-3
le-3
le-4
le-4
le-4
Reacher-v2
le-3
le-3
le-4
le-4
le4
le-4
Hopper-v3
le-3
le-4
le-4
le-4
le-4
1e-3
Learning rate critic
MounaanCCarContinuuusvdd
le-4
le-4
le-4
le-3
le4
le-3
Swimmer-v3
le-4
le-4
le-4
le-4
le44
le-4
InvertedPendulum--22
le-3
le-2
le-2
le-4
le-2
le-3
Reacher-v2
le-3
le-3
le-3
le-2
le-3
le-3
Hopper-v3
le-4
le-3
le-3
le-2
le-4
le-3
Noise for exploration
MounannCarrCnnimuuus-dd
1.0
le-1
le-1
le-1
le-1
le-1
Swimmer-v3
1.0
1.0
1.0
1.0
1.0
1.0
InvertedPendulum-v2
1.0
1.0
le-1
le-1
le-1
le-1
Reacher-v2
le-1
le-1
le-1
1.0
1.0
1.0
Hopper-v3
1.0
1.0
le-1
1e-1
le-1
1.0
Table 9: Table of best hyperparameters for DDPG
Learning rate policy
Policy:
[32]
[64,64]
Metric:
avg
last
avg
last
avg
last
MommainCarConiiuuuss-000
le-2
le-2
le-2
le-4
le-3
le-3
Swimmer-v3
le-3
le-3
le-2
le-2
le-2
le-2
InvertedPendulum-22
le-4
le-4
le-3
le-3
le-3
le-4
Reacher-v2
le-4
le-3
1e-2
le-2
le-3
le-3
Hopper-v3
le-2
le-2
le-2
le-4
le-2
le-2
Learning rate critic
MountanCarContinuuss000
le-4
le4
le-4
le-3
le-3
le-3
Swimmer-v3
le-3
1e-3
le-3
le-3
le-2
le-3
InvertedPendulum--2
le-3
le-3
le-3
le-4
le-3
le-3
Reacher-v2
le-3
le-3
le-3
le-3
le-3
le-3
Hopper-v3
le43
le-3
le-4
le-4
le-4
le-4
Noise for exploration
MounainCarContnuuuusv00
le-2
le-2
le-2
le-1
le-1
le-1
Swimmer-v3
le-1
le-1
le-2
le-2
le-2
le-1
InvertedPendulum-v2
le-1
le-1
le-2
le-2
le-2
le-2
Reacher-v2
le-1
le-2
le-1
le-1
le-1
le-1
Hopper-v3
le-1
le-1
le-1
le-2
le-1
le-2
36


Reviews:
[{"review_id": "qigPvtUr-z_", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "On page 2, in the background section: the discounted state distribution, what you wrote is not a distribution (doesn't sum to 1). In order to define this $d^{\\pi_\\theta}$ properly, you can multiply everything by $1-\\gamma$. The interpretation is that you \"reset\" in your initial distribution $\\mu_0$ with probability $1 - \\gamma$ at every step, or continue in the discounted stationary distribution with probability $\\gamma$.\n\nIn think that theorem 3.1 is incorrect. I think that this is meant to describe an off-policy setting where we are collecting data from $\\pi_b$ but want the policy gradient for $\\pi_\\theta$. In this case, the importance sampling weight should be $\\frac{d_\\theta(s,a)}{d_b(s,a)}$ not $\\frac{\\pi_\\theta(a|s)}{\\pi_b(a|s)}$ (where $d_b$ is the discounted stationary distribution, see above comment too). Equation 9 follows from the chain rule (because the Q function now depends on $\\theta$ explicitly) using the off-policy formulation in Degris (2012), which is incorrect.\n\nNotes:\n- PVF: to me this acronym is strongly synonymous with Mahadevan's proto-value functions (PVFs), circa 2007. How about \"PBVF\" instead? Maybe I'm old\n\n> we optimize for the undiscounted objective\nthis should be reflected in your notation and problem formulation\n\n> can be used only for episodic tasks\nit doesn't have to. See \"regenerative method\" in Monte Carlo estimation literature ", "extra_scores": {}, "extra_text": {}}, {"review_id": "jsG95bnyqkJ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "— idea:\n\t\nA new class of value functions is introduced where the value function takes the parameters of the policy as input, in addition to its common inputs (state or state-action). The proposed type of value functions, PVFs, are also useful for off-policy learning and generalizing over policies while the common value functions lost their information about previous policies.\n\n— comments:\n   \n1- Regarding the first algorithm, PSSVF, until converging, the data that is stored in the replay buffer does not correspond to a \"reasonable\" policy, unless having a prioritized replay buffer. I am also concerned about it being over fitted to the early policies and not being able to  overcome this. I see in the experiments that using PSSVF, policy is converged but am not convinced about it.\n    \n2- Another concern of mine wrt the proposed PVFs is about their sample efficiency. It would be interesting to see a comparison between DDPG and PVF based methods on their sample efficiency.\n    \n3- In the experiments section (4.2), expect for a few cases, ARS is either the best one or does not differ significantly from PVF based methods. Having this in mind, my question is that have you tried to find the best set of hyperparameters for ARS and DDPG as well as your proposed method? If the answer is no, I would like to see that experiments where ARS and DDPG have their best set of hyperparameters.\n    \n4- I am also interested in seeing results for deep policy zero shot learning. In section 4.3, authors just mention: \"When using deep policies, we obtained similar results only for the simplest environments.\" which is not convincing without showing results.\n    \n5- As mentioned in the last part of the paper, the proposed method hugely suffers from the curse of dimensionality. However, as an initial step, PVFs seem interesting and could be beneficial in terms of learning generalized value functions.\n\n-- minor issues:\n    \nIn the first line of the paragraph above the experiments section (4), starting with \"Algorithm 4 (Appendix) uses an ...\", there is a redundant \"and\". One of them should be removed.\n\n\n\nOverall, I liked the idea presented in the paper and would like to see what their next step would be. But the current version of the paper could benefit from more in depth experiments. I believe the most important weakness of the paper lies in the experiment section. It can be much richer and more insightful.\n\n\n\n[1] Sutton, Richard S., et al. \"Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction.\" The 10th International Conference on Autonomous Agents and Multiagent Systems-Volume 2. 2011.", "extra_scores": {}, "extra_text": {}}, {"review_id": "_0J73CTAQpa", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Update**\n\nI have updated my score to 7.\nOne of the points that was not explained in the original paper was that (ignoring function approximation effects) an optimal solution for $J_b$ (the OffPAC objective) will be optimal also for the original off-policy RL objective $J$ (i.e. estimating the on-policy objective in an unbiased manner from off-policy data). From this point of view, I agree that optimizing $J_b$ directly is an interesting question, despite the fact that the exact gradient for $J_b$ may be less similar to the gradient of $J$ compared to the usually used approximate gradient of $J_b$ that drops the $\\nabla_\\theta Q$ term. It still remains unclear which of the two methods has a theoretical advantage over the other in the function approximation setting (in terms of optimizing for $J$); however, because it is unclear, it is interesting to evaluate the method proposed here and to perform experiments as done in the paper to try to find out which method performs better.\n\nThe results were mixed; however, the evaluation is fairly thorough and some potential advantages of the new methods such as generalization in the $\\theta$ space and zero-shot learning were explained.\n\nThe discussion in the paper is much improved compared to the original version. Also, additional ablation studies such as testing what happens when the $\\nabla_\\theta Q$ term is dropped were added (when $Q$ includes $\\theta$ as an input). Moreover, LQR experiments for $Q(s,a,\\theta)$ and $V(s,\\theta)$ were added in the appendix (the results here do not give as good a match as the $V(\\theta)$ formulation gave, but they are reasonable).\n\n______________________________________________________________\n1. Summarize what the paper claims to contribute. Be positive and\ngenerous.\n\nThey propose to include the policy parameters as an input\nto the value function, so that the value function could generalize\nacross different policies (there are 2 other concurrent works with a similar\nidea, one they have cited and discussed \"Policy evaluation networks\"\nhttps://arxiv.org/abs/2002.11833, another is submitted to ICLR2021 on\nopenreview https://openreview.net/forum?id=V4AVDoFtVM).\nThey put the policy parameters theta as an input to the value function\nin 3 cases $V(\\theta)$ (PSSVF), $V(s, \\theta)$ (PSVF), and $Q(s,a,\\theta)$ (PAVF).\nThey propose new policy gradient theorems for the $V(s,\\theta)$ and\n$Q(s,a,\\theta)$ cases (but I believe these to be theoretically flawed).\n\nThey perform experiments testing $V(\\theta)$ in 2 cases: 4.1) (sanity check\nexperiment) visualizing and testing for correctness on an LQR task,\n4.3) zero-shot learning: after training a policy pi using the $V(\\theta)$ method,\na new policy is reinitialized pi_new and trained from scratch using only the\ntrained $V(\\theta)$ without interacting with the environment. The interesting\nbit was that $\\pi_{new}$ managed to outperform the learned policy during\ndata collection $\\pi$ (this implies that the $V(\\theta)$ function managed to\ngeneralize. It would have been nice to, in addition to $\\pi_{new}$, also\nsee whether $\\pi$ could have been improved by just continuing to optimize\nit without interacting with the environment, but this was not done).\n\nThey tested $V(\\theta)$, $V(s,\\theta)$ and $Q(s,a,\\theta)$ on MuJoCo tasks\ncompared to augmented random search (this is similar to evolution\nstrategies) and to deep deterministic policy gradients (DDPG). And\nthe performance did not change much, and sometimes all the new methods\nfailed when DDPG worked (on the reacher task).\n\nThe final experiment 4.4 was for offline learning with fragmented\nbehaviors, i.e. they do not observe full episode data for a fixed\ntheta, which makes it impossible to learn $V(\\theta)$ directly, but\n$V(s,\\theta)$ can be learned by TD methods (also note that the data is\ncollected from a different behavior policy). Then they test a similar\nzero-shot learning procedure as they did for $V(\\theta)$ at different\nstages of the learning (but as far as I understood, for $V(s,\\theta)$ they sampled\ndata from the replay buffer when training the policy (thus not fully without\ninteracting with the data). Perhaps the authors can clarify this), and\nshow that the newly learned policy can outperform the behavior policy,\nthus demonstrating the generalizability of the method.\n\n\n2. List strong and weak points of the paper. Be as comprehensive as possible.\n\n\\+ The experiment on zero-shot learning is nice to show that the $V(\\theta)$\nfunction can generalize.\n\\+ The paper is clearly written.\n\\+ They discuss a lot of related work.\n\\+ The experimental methodology seemed mostly good and honest, and\nwas explained in detail in the appendix (some nice points: They include\na sensitivity analysis showing quantiles of the performance.\nAlso the final best chosen hyperparameters were evaluated with\n20 new seeds, separate from the 5 seeds used during hyperparameter tuning).\n\n\\- The new policy gradient theorems seemed flawed. Also some discussion\naround off-policy learning seemed incomplete.\n\\- The methods were not shown to experimentally lead to major gains.\n\\- One of the difficulties with searching in parameter space is how\nto deal with large parameter spaces. The two concurrent works considering\n$V(\\theta)$ proposed solutions to this issue by embedding the policy into\na smaller space. In the current work no solution is proposed. The\nexperiments on zero-shot learning using $V(\\theta)$ were only good with\nlow-dimensional linear policies.\n\\- A sanity check experiment on LQR was performed for only $V(\\theta)$ (which was\nthe only one for which the gradient was theoretically sound); it would\nhave been good to do similar experiments for the other ones.\n\\- I would expect $V(s,\\theta)$ to outperform $V(\\theta)$ due to using the\nstate information, but this did not appear to be the case.\n\n\n3. Clearly state your recommendation (accept or reject) with one or two\nkey reasons for this choice.\n\nI recommend rejecting the paper due to the theoretical flaws in the newly\nproposed policy gradient theorems using $V(s,\\theta)$ and $Q(s,a,\\theta)$. Also,\nthe practical advantages of using $V(s,\\theta)$ and $Q(s,a,\\theta)$ were not shown.\n\n\n4. Provide supporting arguments for your recommendation.\n\nThe theoretical issues in this paper start in equation 1. They write:\n\"... we can express the maximization of the expected cumulative reward\nin terms of the state-value function:\"\n\n$J(\\pi_\\theta) = \\int d^{\\pi}(s) V(s) ds,$  (in the paper)\n\nwhere $d(s)$ is the discounted state visitation distribution. However, this\nis not the RL objective. The RL objective would be\n\n$J(\\pi_\\theta) = \\int d^{\\pi}(s) R(s) ds.$    (what it should actually be)\n\nThe authors probably took their objective from the work by\nDegris et al (2012, https://arxiv.org/pdf/1205.4839.pdf);\nhowever, in Degris'12, $d(s)$ is _not_ the discounted\nstate visitation distribution. It is the limiting distribution as\n$t \\to \\infty$, which is a stationary distribution. When $d^{\\pi}(s)$ is stationary,\nthen the two objectives become equivalent: $d(s)$ does not change\nfrom one time step to the next, so the difference between the objectives\nwill be just a $1/(1-\\gamma)$ constant factor. Putting aside this issue,\nprobably the limiting distribution formulation is not realistic as most\nRL researchers consider the episodic setting, so using a discounted\nstate visitation distribution is probably better. However, the newly\nproposed policy gradient theorems do not appear sound for the true RL\nobjective using $R(s)$.\n\nNext, they replace the distribution $d^{\\pi}(s)$ with a\ndistribution $d^{\\pi_b}(s)$ gathered using a behavioral policy (so they are\nworking off-policy). However, they do not apply an importance weighting\ncorrection for the distribution shift, and just ignore the importance\nweights (Note that this is also done by Silver et al (2014) in deterministic\npolicy gradients, and by Lillicrap et al (2015) in DDPG, so it is not that\nstrange per se, as long as it gives better practical performance. However,\nit should at least be acknowledged that the importance weights are being\nignored). Note that they still apply an importance weight on the actions\n($\\pi(a|s)/\\pi_b(a|s)$) once the state is sampled from the data buffer, however,\nthis does not correct for the distribution shift from $d^{\\pi}$ to $d^{\\pi_b}$,\nso the policy gradient computed using such a method will necessarily be biased.\nFor example, see the following works for examples that try to deal with the\ndistribution shift problem:\nMunos et al (2016, https://arxiv.org/abs/1606.02647),\nWang et al (2016, https://arxiv.org/abs/1611.01224),\nGruslys et al (2017, https://arxiv.org/abs/1704.04651)\n\nPutting aside the issue of whether ignoring the distribution shift is OK,\nthe main issues are the new policy gradient theorems derived from this\nformulation. Both the $V(s,\\theta)$ as well as $Q(a,s,\\theta)$ formulations appear\nflawed:\nIn the $V(s,\\theta)$ case they propose the policy gradient:\n\n$\\nabla_\\theta J(\\theta) = \\int d^{\\pi_b}(s) dV(s,\\theta)/d\\theta ~~ds$ in equation 8.\n\nHowever, the true policy gradient is:\n$\\nabla_\\theta J(\\theta) = \\int \\mu(s) dV(s, \\theta)/d\\theta ~~ds,$\nwhere $\\mu(s)$ is the start-state distribution. Actually they wrote\nthis also in equation 7, when they considered the $V(\\theta)$ formulation,\nbut for some reason sampled from $d(s)$ instead for $V(s, \\theta)$ when computing\nthe policy gradient in the $V(s,\\theta)$ formulation.\n\nIn the $Q(a,s,\\theta)$ formulation, they add an extra $dQ/d\\theta$ term to\nthe policy gradient. Their motivation is the following:\n\n$\\nabla_\\theta J(\\theta) = \\int d^{\\pi_b} (dQ(a=\\pi(s,\\theta),s,\\theta)/d\\theta) dads$\n  $   \t= \\int d^{\\pi_b} dQ(a,s,\\theta)/da*da/d\\theta + dQ(a,s,\\theta)/d\\theta~~ dads$\n\nHowever, this derivation stems from the flawed definition of J that is\nnot maximizing the sum of rewards over the trajectory distribution, but\nmaximizing some other objective that sums the value functions at all states\nin the trajectory distribution. My strongest argument for why the original\noff-policy derivations by Degris et al and Silver et al are less flawed is\nthe following:\nIf we are on-policy, i.e. $\\pi_b = \\pi$ and $d^{\\pi_b} = d^{\\pi}$ we would want\nthe off-policy policy gradient theorem to be unbiased, hence it should\nrevert to the standard policy gradient theorem. In the formulations\nof Degris and Silver, this is indeed the case, and these theorems would\nbe unbiased in the on-policy setting. The new theorem in the current\npaper, on the other hand, would have an extra $dQ/d\\theta$ term, which would\nbias the gradient. Therefore, I do not see any good theoretical reason to\nadd this term. Moreover, the practical performance did not improve, so\nthere is little evidence to suggest it as a heuristic either.\n\nIf someone would say that the original policy gradient\ntheorem requires the $dQ/d\\theta$ term, I would urge them to look at the original\nproofs---there is no approximation, these theorems are exact for the true\nRL objective based on maximizing the rewards over the discounted trajectory\ndistribution. The intuition is that the remaining $dQ/d\\theta$ term for the\nremainder of the trajectory from a time-step t is estimated by summing\nthe $dQ/da\\*da/d\\theta$ or $Q\\*dlog/d\\theta$ terms for all the future time-steps.\n\nAnother more minor theoretical issue in the paper is that while the\ntheory considered the discounted state visitation distribution, the\ndiscount factors are not added into the policy gradient in the algorithmic\nsections. This omission is common, and tends to work well as a heuristic\n(but it should at least be mentioned that such an approximation is made).\nSee the following papers for more discussion on this:\nNota and Thomas (2020, https://arxiv.org/abs/1906.07073)\nThomas (2014, http://proceedings.mlr.press/v32/thomas14.html)\n\n\n5. Ask questions you would like answered by the authors to help you clarify your understanding of the paper and provide the additional evidence you need to be confident in your assessment. \n\nHow did the computational times compare? Was there much of an overhead to\nusing the more complicated critics including theta as an input?\n\n\n6. Provide additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.\n\nFor me to change my assessment, first the theoretical issues should be\nfixed or cleared up.\n\nNext, I have some possible suggestions:\n1) Test also $V(s,\\theta)$ on LQR as well as on zero-shot learning while sampling\ns from the initial state distribution $\\mu(s)$. This does not require interacting\nwith the environment (because you never apply any action), and I would consider\nit fair in terms of comparing to $V(\\theta)$. If the learning from the TD error\nis working well, I would expect it to outperform the $V(\\theta)$ formulation\nin the zero-shot task. \n2) Test the parameter value functions using the standard policy gradients\nwithout adding the $dQ/d\\theta$ term. Because you are using $Q(a,s,\\theta)$, there\nmay be some learning to generalize across different policies due to the\ntheta input, so it may outperform the original policy gradients without\nchanging the policy gradient theorem. Actually, it would have been better to\nperform such experiments as an ablation study from the beginning anyhow.\n3) Test $Q(a,s,\\theta)$ also on the LQR task to show it's correctness\n(for example by sampling s from the initial state distribution and computing\nthe action). It may also be nice to test it in the zero-shot task as well.\n4) Perhaps test combinations of the various gradients, for example taking\nthe average of the $V(\\theta)$ gradient with the policy gradient using $Q$\n(i.e. taking the average of two equivalent policy gradients).\n\nIf the above points are convincingly done, I may increase to marginal\naccept. The current contributions are not enough for me to go higher than\nthat: taking away the proposed new policy gradients, the main contribution\nis to add $\\theta$ as an input to $V$ and $Q$, which I think is not enough.\nMoreover, the advantage of adding $\\theta$ as an input was not shown convincingly\nusing compelling evidence. Currently the most compelling evidence is the\nzero-shot task, which shows that there is some generalization happening in\nthe $\\theta$ space; however, what is missing to me, is a demonstration of how\nthis additional generalization helps in solving the original task in a more\ndata-efficient manner. Perhaps interleaving the policy search with longer\nsessions of off-line learning (without any interaction) using $dV/d\\theta$\nto take advantage of the generalization may improve the data-efficiency\nand show the advantage of the new method (exaplaining good practices on how\nto do this may be a useful contribution). I think it would also be important\nto show compelling evidence that including the s input helps in learning\nbetter $V$ and $Q$ functions. Perhaps there are also other ways to better\nshow the advantage of the method.\n\nAnother option may be to change the problem setup, so that the new policy gradient theorems would be more sound. For example, using the original formulation of Degris'12 where $d^{\\pi_b}(s)$ is the limiting distribution as $t \\to \\infty$ would make the new policy gradients correct; however, the standard setup would not correspond to this. One setup that would correspond to this objective is the following: an infinite horizon continuing setting, where the agent is never reset into the initial distribution, but has to continually change the policy to improve. The learning would iterate between running one behavioral policy until it converges to its stationary distribution, then optimizing a new policy while in the off-policy setting, then switching the behavioral policy to this new policy, and repeating the process. In this situation, $d^{\\pi_b}(s)$ can be seen as the initial distribution for the new policy, and in this case the new policy gradient theorems would make sense. My previous argument about wanting the policy gradient theorem to be unbiased in the on-policy case would also be satisfied, because if $d(s)$ is stationary then \nthe $dQ/da\\*da/d\\theta$ and $dQ/d\\theta$ gradients would differ by only a constant factor.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "SngPqjOLQ99", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary:\nThe paper proposes passing the parameters of a policy to the value function attempting to learn estimates of the return for that policy. This allows the value function to generalize across policies and estimate values for arbitrary policies. The paper derives several algorithms for various objectives and value functions, and empirically investigates the deterministic versions.\n\n### Pros:\n- Several new algorithms are proposed\n- The new algorithms can generalize across policies\n- The new algorithms can estimate the value of unseen policies\n\n### Cons:\n- Only the deterministic algorithms are empirically investigated\n- Computation and memory cost seem quite high (the critic takes all of the actor’s parameters as arguments)\n- Empirical results seem mixed\n\n### Decision\nI recommend accepting the paper for publication.\n\nThe paper investigates a simple, interesting, original idea—including the actor’s parameters as inputs to the critic—fairly thoroughly. Several actor-critic algorithms are derived using expressions for the gradient of various performance measures obtained by including the actor’s parameters as inputs to the critic.\n\nThe benefits of doing this are illustrated by some experiments, and the deterministic versions of the new methods are compared with reasonable competitors (DDPG and ARS) in other experiments. Unfortunately the results seem somewhat limited by the number of runs that can be conducted by parameterizing the policies and value functions as neural networks and experimenting on the chosen environments. Overall the empirical results seem mixed; in many environments it’s fine to just disregard the second part of the gradient that is dropped in DDPG and computed by PVFs. However, that’s not the fault of the new algorithms, and there are some environments where not dropping the second part of the gradient is helpful.\n\nThe paper is clearly written for the most part, with the exception of some parts of the related work that are overly terse (i.e., the connection with UVFAs could be expanded). Other parts of the related work seem frankly unrelated (i.e., predicting gradients of RNNs from their inputs in the 90s, and mapping weights of CNNs to their accuracy), and I would recommend removing them in favour of moving the more detailed comparison of PENs and PVFs into the main paper.\n\n### Miscellaneous comments:\n- Grammatical error in the final sentence of the abstract: “Their performance is comparable to the one of state-of-the-art methods”\n- “In practice, like in standard actor-critic algorithms, we use a noisy version of the current learned policy in order to act in the environment and collect data” This should probably read standard deterministic actor-critic algorithms.\n- I was disappointed to see that only the deterministic algorithms were implemented and analysed. Even if the stochastic versions of the algorithm are only demonstrated in a simple linear setting, that would be better than just not investigating them at all.\n- The paper doesn’t mention related work that fixes the Off-PAC policy gradient theorem, which gives an expression for the true gradient of the off-policy objective without requiring PVFs (Imani 2018).\n- Passing the actor’s parameters to the critic seems to necessarily break the requirement of compatible features for the actor to follow the true gradient of performance (Sutton 2000). It might be good to mention this.\n\n### References:\n1. Imani, E., Graves, E., & White, M. (2018). An off-policy policy gradient theorem using emphatic weightings. In Advances in Neural Information Processing Systems (pp. 96-106).\n2. Sutton, R. S., McAllester, D. A., Singh, S. P., & Mansour, Y. (2000). Policy gradient methods for reinforcement learning with function approximation. In Advances in neural information processing systems (pp. 1057-1063).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers generally found the idea interesting and the contribution of the paper significant. I agree, I think this is quite a neat idea to investigate, and the paper is written well and is engaging to read.

I would encourage the authors to take into account all of the reviewer suggestions when preparing the camera-ready version. Of particular importance is the name: I think it's bad form to appropriate a name already used in other prior work (proto-value functions, which are very well known in the RL community), so I think it is very important for the final to change the name to something that does not conflict with an existing technique. Obviously this does not affect my evaluation of the paper, but I trust that the authors will address this feedback (I will check the camera-ready).
Decision: Accept (Poster)