Discovered Policy Optimisation
Authors: Chris Lu
Jakub Grudzien Kuba
Alistair Letcher
Luke Metz
Christian Schroeder de Witt
Jakob Nicolaus Foerster
Venue: NeurIPS 2022 Accept (2022)

Abstract: Tremendous progress has been made in reinforcement learning (RL) over the past decade. Most of these advancements came through the continual development of new algorithms, which were designed using a combination of mathematical derivations, intuitions, and experimentation. Such an approach of creating algorithms manually is limited by human understanding and ingenuity. In contrast, meta-learning provides a toolkit for automatic machine learning method optimisation, potentially addressing this flaw. However, black-box approaches which attempt to discover RL algorithms with minimal prior structure have thus far not outperformed existing hand-crafted algorithms. Mirror Learning, which includes RL algorithms, such as PPO, offers a potential middle-ground starting point: while every method in this framework comes with theoretical guarantees, components that differentiate them are subject to design. In this paper we explore the Mirror Learning space by meta-learning a “drift” function. We refer to the immediate result as Learnt Policy Optimisation (LPO). By analysing LPO we gain original insights into policy optimisation which we use to formulate a novel, closed-form RL algorithm, Discovered Policy Optimisation (DPO). Our experiments in Brax environments confirm state-of-the-art performance of LPO and DPO, as well as their transfer to unseen settings.

Discovered Policy Optimisation
Chris Lu"
Jakub Grudzien Kuba" BAIR, UC Berkeley kuba @ berkeley.edu
Alistair Letcher
FLAIR, University of Oxford
aletcher gi tit.ub.ii
christopher. lu @ww.eeeeex.xcc.....
selletchhr@gmaii.coom
Luke Metz
Christian Schroeder de Witt FLAIR, University of Oxford cs cs@robots.ox.ac.o
Jakob Foerster
Google Brain
FLAIR, University of Oxford
iel.s.mmtzzggmai..e
jakob.foerster @ wetpsstttrennge..eee eng.ox.ac.uut
Abstract
Tremendous progress has been made in reinforcement learning (RL) over the past decade. Most of these advancements came through the continual development of new algorithms, which were designed using combination of mathematical derivations, intuitions, and experimentation. Such an approach of creating algo- rithms manually is limited by human understanding and ingenuity. In contrast, meta-learning provides a toolkit for automatic machine learning method optimi- sation, potentially addressing this flaw. However, black-box approaches which attempt to discover RL algorithms with minimal prior structure have thus far not outperformed existing hand-crafted algorithms. Mirror Learning, which includes RI. algorithms, such as PPO, offers a potential middle-ground starting point: while every method in this framework comes with theoretical guarantees, components that differentiate them are subject to design. In this paper we explore the Mirror Learning space by meta-learning "drift" function. We refer the immediate result as Learnt Policy Optimisation (LPO). By analysing LPO we gain original insights into policy optimisation which RL algorithm, Discovered Policy Optimisation (DPO). Our experiments in Brax environments confirm nate-of-the-e art performance of LPO and DPO, as well their transfer to unseen settings.
formulate novel, closed-form
Introduction
Recent advancements in deep learning have allowed reinforcement learning algorithms [35, RL] to successfully tackle large-scale problems [36, 34]. As a result, great efforts have been put into designing methods that are capable of training neural-network policies in increasingly more complex tasks [33, 31, 24, 9]. Among the most practical such algorithms are TRPO [31] and PPO [32] which known for their performance and stability [2]. Nevertheless, although these research threads have delivered a handful of successful techniques, their design relies on concepts handcrafted by humans, rather than discovered in a scalable learning process. As possible consequence, many of these methods suffer from design flaws, such as the brittleness to hyperparameter settings [31, 12], or lack of robustness guarantees.
However, the most promising alternative approach, algorithm discovery via meta-reiiforcemmnt learning, thus far has been "tough nut to crack",. On the one end of the spectrum, popular "black- box" approaches meta-RL [30, 7, 4] are unable to generalise to tasks outside of their meta-training distribution. On the other end of the spectrum, approaches that attempt to meta-learn algorithms
'Equal Contribution
Work done while FLAIR, University of Oxford
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

starting with more structure [25, 16] typically fail to outperform existing fully handcrafted algorithms. To make matters worse, approaches at both ends of the spectrum lack theoretical guarantees.
Recently Mirror Learning [18], a new theoretical framework, introduced an infinite space of provably correct algorithms, all of which share the same template. In nutshell, Mirror Learning algorithm is defined by four attributes, but in this work we focus on the drift function. A drift function guides the agent's 's update, usually by penalising large changes. Any Mirror Learning algorithm provably achieves monotonic improvement of the return, and converges to an optimal policy Popular RL methods such as TRPO [31] and PPO [32] are instances of this framework.
In this paper, we use meta- meta-learning to discover a new state-of-the-art (SOTA) RL algorithm within the Mirror Learning space. Our algorithm thus inherits theoretical convergence guarantees by construction. Specifically, we parameterise a drift function with a neural network, which we then meta-train using evolution strategies [29, ESJ. Our approach is new middle ground between black- box optimisation and utilising prior structure. The outcome of this meta-training is a specific Mirror Learning algorithm which we name Learnt Policy Optimisation (LPO).
While having a neural network representation of novel, high-performing drift function is a great first step, our next goal is to understand the relevant algorithmic features of this drift function. Our analysis reveals that LPO's drift discovered, for example, optimism about actions that scored low rewards in the past feature referred to rollback [37]. Building upon these insights propose a new, glosed-form algorithm which we name Discovered Policy Optimisation (DPO). We evaluate LPO and DPO in the Brax [8] continuous control environments and Minatar [41, 20] environments, where they obtain superior performance compared to PPO. Importantly, both LPO and DPO generalise environments that were not used for training LPO. To our knowledge, DPO is the first theoretically-sound,, scalable deep RL algorithm that was discovered via meta-learning.
2 Related Work
Over the last few years, researchers have put significant effort into designing and developing algo- rithmic improvements in reinforcement learning. Fujimoto et al. [9] combine DDPG policy training with estimates of pessimistic Bellman targets from separate critic. Hsu et al. [15] stabilise the, previously unsuccessful [32], KL-penalised version of PPO and improve its robustness through novel policy design choices. Haarnoja et al. [13] introduce mechanism that automatically adjusts the temperature parameter of the entropy bonus in SAC. However, none of these hand-crafted efforts succeeds in fully mitigating common RL pathologies, such sensitivity to hyperparameter choices and lack of domain generalisation [4]. This motivates radically expanding the RL algorithm search space through automated means [27].
Popular approaches in meta-RL. have shown that agents can learn to quickly adapt over a pre-specified distribution of tasks. RL2 cquips learning agent with recurrent neural network that retains state across episode boundaries to adapt the agent's behaviour to the current environment [4]. Similarly, a MAML agent meta-lcarns policy parameters which can adapt to a range of tasks with few steps of gradient descent [7]. However, both RL2 and MAML usually only meta-learn across narrow domains and are not expected to generalise well truly unseen environments.
Xu et al. [40] introduce an actor-critie method that adjusts its hyperparameters online using meta- gradients that are updated with every few inner iterations. Similarly, STAC [42] uses implementation techniques from IMPALA [5] and auxiliary loss-guided meta-parameter tuning to further improve on this approach.
Such advances have inspired extending peetaggradient RL techniques to more ambitious objectives, including the discovery of algorithms ab initio. Notably, Oh et al. [25] succeeded in meta-learning an RL algorithm, LPG, that can solve simple tasks efficiently without explicitly relying on concepts such as value functions and policy gradients. Similarly, Evolved Policy Gradients [14, EPG] meta-trains a policy loss network function with Evolution Strategies [29, ES]. Although EPG surpasses PPO in average performance, it suffers from much larger variance [14] and is not expected to perform well on environments with dynamics that differ greatly from the training distribution. MetaGenRL. [17], instead, meta-learns the loss function for deterministic policies which are inherently less affected by estimators' variance [33]. MetaGenRI,, however, fails to improve upon DDPG [21] in terms of performance, despite building up on it. Neither EPG nor MetaGenRL have resulted in the discovery

---

of novel analytical RL algorithms, perhaps due to the limited interpretability of the loss functions learnt. Lastly, Co- Co-Reyes et [3]. Garau et al. [10] and Alet et [1] discover and improve standard RI conventions by evolving, symbolically, algorithms represented graphs, which leads to improved performance in simple tasks. However, none of those guarantees, limiting our certainty of the generality of their abilities. In contrast, our method, LPO, is meta-developed a Mirror Learning space [18], where every algorithm is guaranteed convergence to an optimal policy. As result to this construction, meta-training of LPO is easier than that of methods that learn "from scratch", and achieves great performance across environments. Furthermore, thanks to the clear meta-structure of Mirror Learning, LPO is interpretable, and lets discover new learning strategies. This lets us introduce DPO an efficient algorithm with a flosed-form formulation that exploits the discovered learning concepts.
methods inherit correctness
3 Background
In this section, introduce the essential concepts required comprehend our contribution- the RL and meta-RL problem formulations, as well as the Mirror Learning and Evolution Strategies frameworks for solving them.
3.1 Reinforcement Learning
Formulation We formulate the reinforcement learning (RL) problem as a Markov decision process (MDP) [35] represented by a tuple (S, A, Rt P, 7, which defines the experience of a learning agent as follows: at time step E N, the agent is at state St E S (where So ~ d) and takes an action a, E A according to its stochastic policy x(-(s)). which is member of the policy space II. The environment then emits the reward Rst, az) and transits to the next state S1+1 drawn from the transition function, S1+1 P(-|st, a.). The agent aims to maximise the expected value of the total discounted return,
(T)
E E
Exp Mi
Σ y R(se, at]
(1)
The agent guides its learning process with value functions that evaluate the expected return conditioned on states or state-action pairs
(*(+) ER |r, So = s] (the state value function),
(s, A E IT. So s,
(the state-action value function).
The function that the agent is concerned about most is the advantage function, which computes relative values of actions at different states,
Ax a) & Qx(s,a) - - (s)..
Policy Optimisation In fact, by updating its policy simply maximise the advantage function at every state, the agent is guaranteed to improve its policy, r)(Tnew) 2 (7TTala) [35]. This fact, although requiring a maximisation operation that is intractable in large state-space settings tackled by deep RL (where the policy Ty parameterised by weights 0 of a neural network), has inspired range of algorithms that perform it approximately. For example, A2C [24] updates the policy by step of policy gradient (PG) ascent
B
0k+1 On
Azey (sb, uo) V log Te, (an/ss), a (0,
(3)
estimated from batch of B transitions. Nevertheless, such simple adoptions of generalized policy iteration [35, GPIJ suffer from large variance and instability [43, 33, 32]. Hence, methods that constrain (either explicitly implicitly) the policy update size are preferred [31]. Among the most
popular, well as successful ones, is Proximal Policy Optimization [32. PPO], inspired by trust region learning [31], which updates its policy by maximising the PPO-clip objective,
π (aass)
(als) (als) 1 1 +e) ( (s. a))
Th+1 arg max Ex-per a TEll
min
Art (s, a), clip
(4)
(als)
where the clip(., + e) operator clips (if necessary) the input so that it stays within [1 e, + c] interval. In deep RI, the maximisation oracle Equation (4) approximated by few steps of gradient ascent policy parameters.
3

---

Meta-RL The above approaches to policy optimisation rely on human- -ossannss knowledge, and thus are limited by humans' understanding of the problem. The goal of meta-RL is instead optimise the learning algorithm using machine learning. Formally, suppose that an RI. algorithm alga gg. parameterised by Φ, trains an agent for K iterations. Meta-RI, aims to find the meta-parameter e o such that the expected return of the output policy, ([lgg)) is maximised.
3.2 Mirror Learning
A Mirror Learning agent [18), in addition to value functions, has access to the following operators: the drift function Dx, (s] which, intuitively, evaluates the significance of change from policy m to at state s; the neighbourhood operator (1T) which forms region around the policy mei as well as sampling and drift distributions 3x. (s) and v (s) over states. With these defined, a Mirror Learning algorithm updates agent's s policy by maximising the mirror objective
Mh++1 = arg max BawB.ng [A (s, a)] Eu (D (T]s)] xEX(FE)
(5)
If, for all policies and Tg. the drift function satisfies the following conditions:
1. It is non-negative everywhere and zero at identity Da (T]s) 2 Dxx (Tels) O.
2. Its gradient with respect to т is zero at π = Tk-
then the Mirror Learning algorithm attains the monotonic improvement property, nT(TT+1)) 2 (Tx)) and converges to the optimal return, 77(T))) - ,)(x"), as k + oo [18]. A Mirror Learning agent can be implemented in practice by specifying functional forms of the drift function and neighbourhood operator, and parameterising the policy of the agent with neural network, Ty. As such, the agent approximates the objective in Equation (5) by sample averages, and maximises it with an optimisation method, like gradient ascent. PPO is valid instance of Mirror Learning, with the drift function:
D PPO (T|s) A E EA ReLU T (als) (als)
(aas) T& (als)
ee) Ar, (s, a))
clip
(6)
While it is possible to explicitly constrain the neighbourhood of policy update [31], some algorithms do implicitly. For example, as maximisation oracle of PPO (see Equation (4)) has a form of N steps of gradient ascent with learning rate a and gradient clipping threshold c, it implicitly employs neighbourhood of an Euclidean ball or radius Nac around 8g.
Different Mirror Learning algorithms can differ in multiple aspects such as sample complexity and wall-clock time efficiency [18]. Depending on the setting, different properties may be desirable. In this paper, we optimise for the return of the Wrth iterate, T)(
3.3 Evolution Strategies
Exolution Strategies [28, 29, ES] is backpropagation-fin approach to function optimisation. At their core lies the following identity, which holds for any continuously differentiable function F of Φb. and any positive scalar
V VolEE- + [ee] Eo. - oe)e),
where N(0, I) denotes the standard multivariate normal distribution. By taking the limit a - 0, the gradient on the left-hand side recovers the gradient of (o)( These facts inspire an approach of optimising F' with respect to ó without estimating gradients with backpropagation cinigattioooforro a random sample En, N NO, 1). the vector 1 Σn Ei=1 F(O + dEG)e is an unbiased gradient estimate. To reduce variance of this estimator, antithetic sampling is commonly used [26]. In the context of meta-RL, where é is the meta petamparreeter of an RL algorithm alg the role of F(Ø) played by the
no
average return after the training, F(Ø)= = [[n(TT) ]o)) As oppose to the meta-gradient approaches described in Section 2, ES does not require backpropagation of the gradient through the whole training episode-a a cumbersome procedure which, often approximated by the truncated backpropagation,
introduces bias [38, 39, 25, 6, 23].

---

(a)
(b)
Derivative of PPO Objective
1e-6
1e-6 Derivative of PPO Objective
Advantage
Derivetive wr Reto
Adventaxe
-2 -1
-2-
Ratio
Ratio
Figure I: PPO objective visualisation: (a) is the heat map of the ratio derivative of the PPO objective, and (b) shows slices for fixed advantage values. The algorithm encourages updates towards actions with positive values of the ratio derivative.
4 Methods
Our overall approach is to meta-learn a drift function to perform policy optimisation over a fixed episode length K. Hence, meta-objective is the expected final return,
F()) =E/((T))l]
4.1 Drift Function Network
The drift function that we learn takes form 2 (T)()) = Ea (x)(s), where fa(a) is a fully- connected neural network parameterised by ø. Our drift network is a function of the probability ratio between a candidate and the old policy, r = T[ass) (olss)/ol /TT /T (a|s), and of the advantage = Ar (s, a) (which we assume to be normalised across each batch). To ease learning complicated mappings, include non-linear transformations of these arguments, ultimately forming the following input
F [11 r). (1 +) (1 - 4) (1 - r) A, log(r), log(r)?, log(r) log(r) A].
We perform ablations on different variants of the drift input in Appendix E. In general, we found that the performance of the learned drift funetion seems be robust to different input parameterisations.
In order to guarantee that the neural network is a valid drift function, it must be the case that fo(æ,,,) 0 and (a,,A) model, E 0 whenever r = 1, the former condition is guaranteed by excluding bias terms from the network architecture. To the latter two conditions, we apply the ReLU activation at the last layer with a slight shift, + ReLU(x E), where & 10-6
whenever r 1, and fo( 2 everywhere. As in our
To alleviate the difficulty of the meta-training, the main variant of implementation initialises the drift function near the PPO one (see Section 5.2. We have found that this operation leads to a better performance and generalisation across different environments. Furthermore, such setting directly tests for the optimality of the drift function of PPO. If PPO was optimal, the network could simply learn the zero mapping.
In principle, the drift function not restricted to operate with probability ratios and advantages only; it could also accept other arguments, such algorithm hyperparameters, statistics measuring the progress of training, or even task information. However, many of these vary greatly in the dimensionality scale across different instances types of environments. Furthermore, large number of arguments would impede the analysis of the learnt drift function, and comparison to PPO, which also goals of our work. Hence, work with simple and transferable ratios and normalised advantage estimates. Nevertheless, for specific applications, may be beneficial to consider other, possibly domain-specific, information.
4.2 Meta- Peta-Training the Drift Function Network
The meta-objective we optimise the performance of the learner policy the end of training: F(ø) E [[n(TTx Jalga], where 0g is the th (last) iterate of the RL training under the Mirror
5

---

(a)
(b)
Derivative LPO-Zero Objective
le-5
1.00
Derivative of LPO-Zero Objective
0.75
1.0
Advantage
0.50
Aeventaee
0.25
0.00
Derivative
0.0
-1
00
-0.5
-1.00
Ratii
Ratio
Figure 2: Visualisation of the LPO-Zero (learning from scratch) objective: (a) is the heat map of the ratio derivative of the LPO-Zero objective, and (b) shows its slices for fixed advantage values. The algorithm encourages updates towards actions with positive values of the ratio derivative.
(a)
(b)
Derivative of LPO Objective
1e-6
1e-6 Derivative LPO Objective
Derivetive wrt Retio
Advantage
Adventaxe
-2
Ratio
Ratio
Figure 3: Visualisation of the LPO objective: (a) is the heat is the heat map of the ratio derivative of the LPO objective, and (b) shows its slices for fixed advantage values. The algorithm encourages updates towards actions with positive values of the ratio derivative.
Learning algorithm alg.. The expectation is taken over the randomness the initial parameter t and stochasticity of the environment. We solve this problem using ES with antithetic sampling. At each generation (outer loop iteration), we sample batch of perturbations of Φ, initialise the policy parameters 6g and then train the policy under alg using the drift function's 's parameter Φ, for K iterations. At the end of the inner-loop training, we estimate the return of the final policy Φ0 and use estimate the gradient V (()) as in Equation (7).
We meta-learn the drift function and evaluate policies trained by in the Brax [8] physics simulator environments. Brax is designed to take advantage of parallel computation on accelerators, allowing us to roll out thousands of episodes in parallel, and train entire RL agents within minutes. Thus, is well suited for this type of optimisation problem. Furthermore, the policy optimisation algorithm itself, which lets us train hundreds of RL agents per minute using accelerators. We implement our method on top of the Brax version of PPO, which provides Mirror Searning-frennde code template, keeping the policy architecture and training hyperparameters unchanged. For meta-traing peta-ttaiinng use both evosax [19] and the Learned optimization [22] libraries. For full details of meta- pota-training see Appendix A.
increase its efficiency by vectorising
5 Empirical Studies
consider two different meta-training setups. First, as described Subsection 5.1 we attempt to learn drift function completely from scratch to investigate how similar it is to existing algorithms like PPO. Second, in Subsection 5.2 we ask whether we can learn drift function that successfully generalises to multiple environments, if it is initialised near PPO.

---

5.1 Learning drift functions from scratch
In this setting, fo is a neural network with two hidden layers of size 256 and a ReLU activation function. We meta-train it across 5 Brax environments. We name the resulting algorithm LPO-Zero, and visualise it in Figure 2.
Interestingly, LPO-Zero appears to have learnt a few PPO-like features, as can be observed on Figure 2. For example, it appears have learnt clip the update incentive specific ratio threshold, much like PPO; however, it only does so for negative advantages. Nevertheless, LPO-Zero largely underperforms with respect to LPO, and possibly requires much more training to catch up. We present figures with performance evaluation of LPO-Zero in Appendix B.
5.2 Learning with the PPO Initialisation
In this setting, fa is small neural network, with single hidden layer with 128 neurons, with bias terms removed, and a tanh activation function. Here, we add PPO the output of the last hidden layer, before passing it to the shifted ReLU,
(0 A) = ReLU (Fo (ZC, E ReLU ((r clip(r, 1 + =)) A)):
(8)
where fa((z,,)) is the output of the last hidden layer of the drift network. As such, the resulting drift function similar to that of PPO at initialisation.
Surprisingly, we have found that meta-training in a single environment is sufficient to generate drift functions whose abilities transfer to unseen tasks. Moreover, we found that the learnt drifts generally chose the drift function that was trained on Ant,
display similar characteristics. For readability, whose induced algorithm we refer to as Learnt Policy Optimisation (LPO). Visualisations and results for drift functions trained other environments can be found in the Appendix C.
The results on Figure 6 show that LPO, trained only on Ant, outperforms PPO in unseen environ- ments. Furthermore, the Brax PPO implementation uses different hyperparameters, such as the number of update epochs and the total number of timesteps, for each of the tasks. This means that LPO, which was trained Ant with hyperparameters associated to it, is robust not only against new environments, but also against new hyperparameters, We visualise the derivative of the LPO loss in Figure 3, which enables us to derive an analytical version of it in Section 6.
Entropy in Ant
Analysis of LPO
5.0
PPO
LPO
In this section, we analyse the two key features that are consistently learnt and contribute most to LPO's performance. We then interpret their effect on policy entropy, and the update asymme- try discovered by LPO, through which it differs largely from PPO (recall Figure? for visualisa- tion).
DFC
0.0
entropy
-2.5
-7.5
11..0
12.5
1.5
3.0
Rollback for negative advantage. In the bottom-left quadrant of the heat map, which 1 (negative
steps
corresponds to A < 0 and advantage and decrease in action probability). we observe that the ratio derivative of the LPO objective is positive in a large region, roughly corresponding
Figure 4: Entropy comparison, throughout training on Ant, between PPO (blue), LPO (orange), and DPO (green, see Section 7) across 10 seeds. Error bars denote standard error. While the entropy of all methods decrease throughout training, the entropy of policy learned by both LPO and DPO remain significantly higher than that of PPO.
This implies that actions which fall into this quadrant, although seemingly not appealing. are encouraged to be taken by the agent, which can be interpreted as a form of rollback [37]. Hence, LPOO learns to decrease down to 1 e, but unlike PPO, encourages to stay precisely around that value. By doing so, LPO prevents the agent from giving up on actions that appear poor the moment, and encourages it to keep exploring them at moderate frequency.
7

---

(a)
(b)
Derivative of DPO Objective
1e-6
-e-eDerivative of DPO Objective
Derivetive wrt Redo
Advantage
Adventaxe
-2 -1
-2
Ratio
Ratio
Figure 5: Visualisation of the DPO objective: (a) is the heat is the heat map of the ratio derivative of the DPO objective, and (b) shows its slices for fixed advantage values. Positive values of the derivative encourage updates towards the action.
Cautious optimism for positive advantage. The upper-right quadrant corresponds to A > 0 and r > 1 (positive advantage and increase in action probability), which induced by actions that seem the most appealing to update to. Nevertheless, LPO is cautious in doing so, gradually decreasing the pace of its update towards them, and eventually abstaining from chasing the most extreme advantage values these may come from critic errors. We would like to highlight that this view of LPO on positive advantages in much more sophisticated than that of PPO, which simply removes any incentive from updating towards actions with + e, and thus can be viewed as optimistic relative to PPO.
Implicit entropy maximisation. Together, these two central features of LPO encourage the agent to spread its policy probability mass moderately over all actions, thus leading to larger entropy and allowing for richer exploration. Thus, LPO has implicitly discovered entropy maximisation, which demonstrate in Figure 4. We would like highlight that LPO achieved this with its drift function only, without artificially augmenting the original RL. objective with an entropy bonus [11, 12]
Update asymmetry. LPO learns asymmetric features that respect a natural asymmetry of behaviour change in RL: increasing r for positive advantage A may encourage exploration of newly- found action or strengthen dominant action, whereas decreasing r for negative A will always discourage exploration of that action and strengthen a dominant action. In this context, the two discussed features of I.PO make completely unlike PPO, which clips the update incentives symmetrically around the origin.
Secondary Features. LPO, but not LPO-Zero, appears to consistently learn objectives with gradient spikes around = 1 in the upper left and lower right quadrants. Nevertheless, adding them to analytic model of LPO did not improve performance. We speculate, therefore, that these spikes are mostly artifacts of the network parameterisation.
Discovered Policy Optimisation: A New RL Algorithm Inspired By LPO
In this section, building upon concepts that LPO has discovered, we introduce novel algorithm Discovered Policy Optimization (DPO).
7.1 The Discovered Drift Function Model
Combining the key features identified in Section 6, we construct a closed-form model LPO that can easily be implemented with just few lines of code on top of an existing PPO implementation. name new algorithm Discovered Policy Optimisation (DPO) because we have not derived it-it was instead discovered in the meta-learning process. DPO is a Mirror Learning algorithm, with drift function that takes different functional forms, depending on the sign of advantage A, as dictated by the update asymmetry principle from the previous section. Specifically, we have found

---

4000
200
25
1.5
1G5
29
36
0.
22
8.1
0.6
10
(a) Grasp
(b) Ant
(c) Fetch
0303
2003
-120
666
825
875
3.22
350
t0
(d) Reacher
(c) Humanoid
(f) urSe
560
100
3et
(g) HalfCheetah
(h) Walker2d
Figure 6: Performance comparison between PPO (blue), LPO (orange), and DPO (green) in Brax environments. The curves represent mean evaluation return across 10 random seeds, with error bars showing standard error of the mean. Both LPO and DPO beat PPO across most environments even though they were only meta-trained on Ant, and make use of no hyper parameter tuning. Furthermore, both LPO and DPO are more consistent across runs.
that the (parameter-free) drift function
ReLU((r- 1)4 a tanh((r- fanh(((-1)AAoo)) A A20 |ReLU log(r) A taahh(og(r)44/3) A <<0
[(r, A)
(9)
faithfully reproduces the key features of LPO (cautious optimism and rollback) for appropriate constants a = 2, B 0.6 (see Appendix D for verification of the drift conditions). We visualise DPO in Figure and note that even the "crossing-over" of gradient slices of LPO Figure 3 is faithfully reproduced.
7.2 Results
We compare DPO to PPO and LPO on a variety of Brax environments in Figure 6. We use the PPO implementation provided by Brax, with an addition of advantage normalisation as we observed it to improve the method's performance across the majority of the environments. Our methods also use this implementation technique. We further demonstrate evaluating performance on the Minatar environments [20, 41] in Figure 7.
out-of-distribution performance by
While evaluating DPO, similarly to LPO, we do not re-tune any hyperparameters that were originally selected for PPO Brax. The results on Figure show that DPO matches the performance of LPO and outperforms PPO on the evaluated environments, despite being two-line analytic model of LPO based on two key features. This enables RL practitioners to implement DPO as easily as PPO with performance on par with our best learnt drift function.

---

14
retum
20
10
le7
Breakout-Minatar
(b) Asterix-Minatar
120
average return
100
80
10
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
1.0
367
167
(c) Frreway-MMinatar
(d) Spacelnvaders-Minntie
Figure 7: Performance of PPO, DPO, and LPO on the Minatar environments [20, 41J. We tuned the hyperparameters for PPO and re-used those tuned hyperparameters for DPO and LPO. The curves represent mean evaluation return across 10 random seeds, with error bars showing standard error of the mean.
Conclusion
In this paper, we approached the problem of algorithm discovery by restricting our meta-learning to the space of valid Mirror Learning algorithms. Specifically, we optimised drift function parameterised by neural network, which trained with Evolution Strategies. We consider this work to be an example the new, promising paradigm of RL algorithm discovery. Namely, our strategy was to develop high-perforning RI. algorithm by combining theoretical insights with large-scale computational techniques. a result of the training, we obtained theoretically sound method that named Learnt Policy Optimisation (LPO), which outperforms state-of-the-at baseline (PPO) in unseen environments, and with unseen hyperparameter settings. After analysing the learned features discovered by LPO, we introduced Discovered Policy Optimisation (DPO)-a closed-form approximation to LPO. Our experimental results show that DPO matches LPO in performance and robustness to hyperparameter settings. However, possible weakness of this approach is that it could overfit to specific code-level details of the implementation. For example, LPO built on Brax's 's PPO implementation, which makes numerous design decisions optimise for wall-clock time rather than sample efficiency. In the future, we plan to expand the variety of inputs to the learnt drift function, well parameterise and meta-learn other attributes of Mirror Learning. We expect these advancements to provide more insights into policy optimisation, ultimately resulting in more robust and better performing RL algorithms.
Acknowledgments and Disclosure of Funding
Christian Schroeder Witt is sponsored by the Cooperative AI Foundation. Compute for this project was partially run on Oxford's Advanced Research Cluster (ARC). This work used the Cirrus UK National Tier-2 HPC Service EPCC funded by the University of Edinburgh and EPSRC (EP/P020267/1). This work was also supported by Oracle for Research Cloud Grant (19158657)
10

---

References
[1] Ferran Alet, Martin F. Schneider, Tomás Lozeno-Pérez, and Leslie Pack Kaelbling. Meta- learning curiosity algorithms. CoRR, abs/2003.05325, 2020.
[2] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprini 19Xivv/9/2.000 06680, 2019.
[3] John D Co-Reyes, Yingjie Miao, Daiyi Peng, Esteban Real, Sergey Levine, Quoc V Le, Honglak Lee, and Aleksandra Faust. Evolving reinforcement learning algorithms. arXiv preprint arXiv:2101.0m 033958,, 2021.
[4] Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, llya Sutskever, and Pieter Abbeel. RIS Fast reinforcement learning via slow reinforcement learning. CoRR, abs/1611.02277 2016.
[5] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning, pages 1407-1416. PMLR, 2018.
[6] Xidong Feng, Oliver Slumbers, Ziyu Wan, Bo Liu, Stephen McAleer, Ying Wen, Jun Wang, and Yaodong Yang. Neural auto-curricula in two-player zero-sum games. Advances in Neural Information Processing Systems, 34, 2021.
[7] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In International conference machine learning, pages 1126-1135. PMLR, 2017.
[8] Daniel Freeman, Erik Frey. Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax-a differentiable physics engine for large scale rigid body simulation. arXiv preprint arXiv:2 /06. I 13281, 2021.
[9] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596.
PMLR, 2018.
[10] Juan Jose Garau-Luis, Yingjie Miao, John D Co-Reyes, Aaron Parisi, Jie Tan, Esteban Real, and Alcksandra Faust. Multi-objective evolution for generalizable policy gradient algorithms. arXiv preprint w:Xh::2244.0422. 2022.
[11] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning, pages 1352-1361. PMLR, 2017.
[12] Tuomas Haarnoja, Aurick Zhou, Picter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, pages 1861-1870. PMLR, 2018.
[13] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint 1rXiv:1812.05900 2018.
[14] Rein Houthooft, Yuhua Chen, Phillip Isola, Bradly Stadie, Filip Wolski, OpenAl Jonathan Ho, and Pieter Abbeel. Evolved policy gradients. Advances in Neural Information Processing Systems, 31, 2018
[15] Chloe Ching-Yun Hsu, Celestine Mendler-Dünned and Moritz Hardt. Revisiting design choices in proximal policy optimization. arXiv preprint 2rXiv:2009.10892 2020.
[16] Louis Kirsch and Jürgen Schmidhuber. Meta learning backpropagation and improving Advances in Neural Information Processing Systems, 34, 2021.
[17] Louis Kirsch, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Improving generalization in meta reinforcement learning using learned objectives. arXív preprint arXiv: 91100409999 2019.
[18] Jakub Grudzien Kuba, Christian Schroeder de Witt, and Jakob Foerster. Mirror learning: A unifying framework of policy optimisation. arXiv preprint arXiv:2201.02377 2022.
[19] Robert Tjarko Lange. evosax: Jax-based evolution strategies, 2022.
[20] Robert Tjarko Lange. gymmax: A based reinforcement learning environment library, 2022.
11

---

[21] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: pttp:::09990779 2015.
[22] Luke Metz, C
Daniel Freeman, James Harrison, Niru Maheswaranathan, and Jascha Sohl- Dickstein. Practical tradeoffs between memory, compute, and performance in learned optimizers. arXiv preprint arXiv:2203.178650 2022.
[23]
Luke Metz, C Daniel Freeman, Samuel S Schoenholz, and Tal Kachman. Gradients are not all you need. arXiv preprint arXiv:2
2021.
[24] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. In International conference on machine learning, pages 1928-1937. PMLR, 2016.
(25) Junhyuk Oh, Matteo Hessel, Wojeiech M Czarnecki, Zhongwen Xu, Hado P van Hasselt, Satinder Singh, and David Silver. Discovering reinforcement learning algorithms. Advances in Neural Information Processing Systems, 33:1060-1070, 2020.
[26] Art B Owen. Monte carlo theory, methods and examples (book draft), 2014.
[27] Jack Parker-Holder, Raghu Rajan, Xingyou Song, André Biedenkapp. Yingjie Miao, Theresa Fimer, Baohe Zhang, Vu Nguyen, Roberto Calandra, Aleksandra Faust, Frank Hutter, and Marius Lindauer. Automated reinforcement learning (autorl): A survey and open problems. J. Artif. Intell. Res., 74:517-568, 2022.
[28] Ingo Rechenberg. Exolutionsttaleiicoppiiiiieee biologischen evolution. 1973.
technisher systeme nach prinzipien der
[29] Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies scalable alternative to reinforcement learning. arXiv preprint arXiv: 2017.
[30] Jürgen Schmidhuber. On learning how to learn learning strategies. 1995.
[31] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
[32] John Schulman, Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv, abss/1777.0637,, 2017.
[33] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning. pages 387-395. PMLR, 2014.
[34] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy P. Lillicrap. Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Nat., 55007676):354-359,, 2017.
[35] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. 2018.
[36]
Oriol Vinyals, Igor Babuschkin, Wojeiech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander Sasha Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom Le Paine, Çaglar Gülçehre. Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McK- inney, Oliver Smith, Tom Schaul, Timothy P. Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft II using multi- multi-agent reinforcement learning. Nat., 575(7782):350
2019.
[37] Yuhui Wang, Hao He, and Xiaoyang Tan. Truly proximal policy optimization. In Uncertainty in Artificial Intelligence, pages 1 113-122. PMLR, 2020.
[38] Paul Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE 78(10): 1500-1500,,- 1990.
[39] Yuhuai Wu, Mengye Ren, Renjie Liao, and Roger Grosse. Understanding short-horizon bias in stochastic meta-optimization. arXiv preprint arXiv:1803.00021, 2018.
12

---

[40] Zhongwen Xu, Hado P van Hasselt, and David Silver. Meta-grasn gradient reinforcement learning. Advances neural information processing systems, 31, 2018.
[41] Kenny Young and Tian Tian. Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments. arXiv preprint arXiv: 037660 2019.
[42] Tom Zahavy, Zhongwen Xu, Vivek Veeriah, Matteo Hessel, Junhyuk Oh, Hado van Hasselt, David Silver, and Satinder Singh. A self-tuning actor-critic algorithm. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20 20, Red Hook,
NY, USA, 2020. Curran Associates Inc.
[43] Tingting Zhao, Hirotaka Hachiya, Gang Niu, and Masashi Sugiyama. Analysis and improvement of policy gradient estimation. In NIPS, pages 262-270. Citeseer, 2011.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We discuss this in the Introduc- tion(l), and Discussion (Section 8)
(c) Did you discuss any potential negative societal impacts your work? [N/A] Our work does not introduce any significant new negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We do not violate any of the ethics concerns
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] Our results are empirical.
(b) Did you include complete proofs of all theoretical results? [N/A] Our results are empirical.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed reproduce the main perimental results (either in the supplemental material or as a URL)? [Yes] They are attached in the supplementary material
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] They are in Appendix A
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes] They are in the figures
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] They are in Appendix A
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite all codebases
used.
(b) Did you mention the license the assets? [N/A] We follow the codebase licenses, but do not mention them.
(c) Did you include any new assets either in the supplemental material URL? [Yes] We will include code in the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] We are not using anyone's s data.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] We are not using anyone's 's data.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "hh8AAJKhKq", "reviewer": "Reviewer_H27p", "summary": "This paper studies the learning of a “drift function” (a proximal regularization term in policy optimization) for reinforcement learning (RL). By parameterizing the drift function as a neural network and framing the objective as the expected return after K iterations of RL, the authors can find this drift function through black-box optimization. The authors analyze the structure of the learned drift function and compare it to PPO’s drift function. Inspired by insights on the learned function, the authors also present an analytic drift function that contains many of the same properties as the learned function. Experiments show that policy optimization with both novel regularizers regularly outperform PPO on several Brax continuous control problems.", "questions": "What happens if you try to learn a drift function without the input $x$ from line 179? Does this drift function also give good RL performance?\n\nSome other comments:\\\nLine 137: Is it supposed to be $\\pi_k$ instead of $\\bar\\pi$?\\\nI would suggest moving Sec. 5.1 (Learning drift functions from scratch) to the Appendix since it performs worse than LPO and is not a significant part of the paper.\\\nLine 177: The variable $r$ is already used for the reward. It may make sense to use a different variable.\\\nDespite what the paper says, I did not find any code in the supplementary.\n", "limitations": "The authors acknowledge that they only study the drift function and hope to expand optimization to other parts of the mirror learning objective.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper studies a highly practical problem in the design of surrogate objectives in policy optimization. The proposed approach is sensible, and the results do appear promising. The writing and presentation are also very clear, and it’s easy to read the paper. I especially like the analysis of the LPO function in Section 6, and the insights to its structure (rollback and cautious optimism) make sense for its role in policy optimization.\n\nI think one major weakness of their work is that the evaluation environments are limited. All environments are continuous control problems in the Brax simulator. To assess how well the LPO/DPO drift function generalizes, I highly suggest evaluating the functions in other physics simulators (e.g., MuJoCo and Isaac) and entirely different domains (e.g., Atari and other discrete problems).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NDi8F4vrfv7", "reviewer": "Reviewer_XLZN", "summary": "The paper is based on the concept of Mirror Learning which includes several RL algorithms like PPO, TRPO, MDPO, etc. The work empirically exploits the flexibility of choosing the \"drift\" function in the Mirror Learning domain by using meta-learning. Given a neural network drift function, the authors use Evolutionary Strategies to meta-learn a single drift function (generalizable across environments) where the objective is to maximize the return of the policy trained using an update rule composed of the drift function. This is termed the Learnt Policy Optimization (LPO) which is proposed in two variants: one uses PPO clip augmentation at the last hidden layer output and the other doesn't. Eventually, the authors draw meaningful observations from LPO's analysis and proposed Discovered Policy Optimization (DPO) using those observations. The authors verify the approach by showing that DPO's behavior is close to LPO's behavior. All experiments are conducted in brax-based environments.", "questions": "It would be nice if the following can be justified and included in the paper:\n- Since the overall objective for policy improvement aims to minimize $\\mathbb{E}_{\\pi_k}\\[\\mathcal{D}(\\pi|s)\\]$, I am curious to know where $\\tilde{f}_\\phi$ converges to and how the final drift function $f_\\phi$ is different from the PPO one. This also points towards another interesting phenomenon of DPO (with constants) being an approximated version of PPO with smoothened contours (instead of strict clipping).\n- If there is hyperparameter optimization of PPO (just clip factor) with evolutionary strategies to get the best performance (return) for a particular environment, what does the DPO performance comparison look like? \n- Can the authors add the corresponding heat map of PPO as well in Figure 5?\n- The discovered drift function seems to be cherry-picked. It will be beneficial if the authors add justifications for their choice of the input vector to the drift function during LPO optimization and also in DPO. Why not $(r-1)^n$ in DPO as that would also satisfy the feasibility conditions? \n- For a single environment, can the authors show the variation of the final performance as a function of constants $\\alpha$ and $\\beta$ ?\n- There should be more experiments on multiple benchmarks to justify the claim since this is an improvement of an algorithm. Can you add some analysis of Procgen experiments?\n- Can you add a properly referenced appendix? The references in the paper do not correspond properly to some Appendix sections making it difficult to analyze.\n- Also, add equation numbering.", "limitations": "The authors have acknowledged that the method shown in the presented work is a step towards discovering more robust and high-performing RL algorithms. In general, the work does not has significant limitations but lacks proper justification of choices which makes it unclear how to conduct algorithm discovery in a proper manner. Further, more empirical evidence and experiments are required to complete the contribution.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed method holds some strength by improving over a widely-used baseline using:\n- parametrized drift functions to capture the importance of action distribution ratio and advantage in policy improvement.\n- empirically observed and properly justified learned behavior based on the advantage being positive or negative.\n- well-curated and mechanically optimized/fine-tuned algorithm (DPO) using the discovered drift function model.\n\nHowever, there are the following weaknesses in the proposed framework:\n- there are very less ablations on the choice of structure, for example, the non-linear formulation of $x_{r,A}$ and choice of constants for DPO (overall Appendix E is missing)\n- there is no justification for why an initial drift function of PPO helps. While it is acknowledged that LPO-Zero underperforms (\"possibly\" due to less training), there is no evidence that much more training can help LPO-Zero converge to LPO.\n- the experimental validation is not sufficient to prove the improvement claimed by the contribution. (there are no additional results in Appendix D as mentioned in the paper)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KFUmv9eHc1-", "reviewer": "Reviewer_KEqn", "summary": "\nThis paper proposes a method to learn how to optimize policy under the framework of Mirror Learning. Based on the learning result, they formulate a closed-form policy optimization method for learning the policy.", "questions": "Learning discovery is interesting, and using the fixed learning mechanism is natural. Could that be true if the optimal learning mechanism is different across different environments?\n\nCan you explain why you use the candidate arguments in the equation in Line 180? Are there any other candidates?", "limitations": " ", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The idea of this paper is interesting. They propose a method that lies between human handcraft and learning discovery. The experimental result is interesting.\nI'm not satisfied with the claim and some experimental results (see below for detail).\n\nThe author claim that TRPO and PPO are handcrafted by humans rather than discovered by learning. This is overclaimed to me. \n- The key idea of this paper also follows Mirror Learning, which is also discovered by humans.\n- Furthermore, learning drift functions from scratch underperforms with respect to LPO. Instead, they learn with the PPO initialization, which is more handcrafted.\n\nI'm not satisfied with the experimental result.\n- As the authors claim that discovered by an algorithm is better, and there exits several variants following Mirror Learning, e.g., TRPO and variants of PPO. It's straightforward to compare the learned methods with these mirror-learning-based methods.\n- The experimental results of learning drift functions from scratch underperform LPO and even PPO. Although the authors have also tried learning from PPO initialization, this implies it involves more human handcraft. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept