Discovering Temporally-Aware Reinforcement Learning Algorithms
Authors: Matthew Thomas Jackson
Chris Lu
Louis Kirsch
Robert Tjarko Lange
Shimon Whiteson
Jakob Nicolaus Foerster
Venue: ICLR 2024 poster (2024)
Keywords: Reinforcement Learning
Meta-Learning
Meta-RL
Meta-Optimization
Policy Meta-Optimization
Learned Objective Functions
Primary area: reinforcement learning
TL;DR: Conditioning learned optimizers on agent lifetime leads to an adaptive update and improves generalization performance.

Abstract: Recent advancements in meta-learning have enabled the automatic discovery of novel reinforcement learning algorithms parameterized by surrogate objective functions. To improve upon manually designed algorithms, the parameterization of this learned objective function must be expressive enough to represent novel principles of learning (instead of merely recovering already established ones) while still generalizing to a wide range of settings outside of its meta-training distribution. However, existing methods focus on discovering objective functions that, like many widely used objective functions in reinforcement learning, do not take into account the total number of steps allowed for training, or “training horizon”. In contrast, humans use a plethora of different learning objectives across the course of acquiring a new ability. For instance, students may alter their studying techniques based on the proximity to exam deadlines and their self-assessed capabilities. This paper contends that ignoring the optimization time horizon significantly restricts the expressive potential of discovered learning algorithms. We propose a simple augmentation to two existing objective discovery approaches that allows the discovered algorithm to dynamically update its objective function throughout the agent’s training procedure, resulting in expressive schedules and increased generalization across different training horizons. In the process, we find that commonly used meta-gradient approaches fail to discover such adaptive objective functions while evolution strategies discover highly dynamic learning rules. We demonstrate the effectiveness of our approach on a wide range of tasks and analyze the resulting learned algorithms, which we find effectively balance exploration and exploitation by modifying the structure of their learning rules throughout the agent’s lifetime.

Published as a conference paper at ICLR 2024
DISCOVERING TEMPORALLY-AWARE REINFORCEMENT LEARNING ALGORITHMS
Matthew T. Jackson" University of Oxford
Chris Lu" University of Oxford
Louis Kirsch
The Swiss AI Lab IDSIA
Robert T. Lange Technical University Berlin
Shimon Whiteson University of Oxford
Jakob N. Foerster
University of Oxford
ABSTRACT
Recent advancements in meta-learning have enabled the automatic discovery of novel reinforcement learning algorithms parameterized by surrogate objective functions. To improve upon manually designed algorithms, the parameterization this learned objective function must expressive enough to represent novel principles of learning (instead of merely recovering already established ones) while still generalizing to a wide range of settings outside of its meta-training distribution. However, existing methods focus on discovering objective functions that, like many widely used objective functions in reinforcement learning, do not take into account the total number of steps allowed for training, or "training hori- zon". In contrast, humans use a plethora of different learning objectives across the course of acquiring a new ability. For instance, students may alter their study- ing techniques based on the proximity to exam deadlines and their self-assessed capabilitties. This paper contends that ignoring the optimization time horizon sig- nificantly restricts the expressive potential of discovered learning algorithms. We propose a simple augmentation to two existing objective discovery approaches that allows the discovered algorithm to dynamically update its objective function throughout the agent's training procedure, resulting expressive schedules and increased generalization across different training horizons. In the process, we find that commonly used meta-gradient approaches fail to discover such adaptive objective functions while evolution strategies discover highly dynamic learning rules. We demonstrate the effectiveness our approach on a wide range of tasks and analyze the resulting learned algorithms, which we find effectively balance exploration and exploitation by modifying the structure of their learning rules throughout the agent's 's lifetime.
INTRODUCTION
Advancements in reinforcement learning (RI.) have historically come from handcrafted algorithms derived from human experimentation or theoretical insights (Mnih et al., 2015) [Schulman et al.] 2015). By contrast, recent approaches attempt to discover novel RL algorithms automatically (e.g. Kirsch et al. 2020: Oh et al. 2020: Lu et al., 2022a).. One well-studied feature of handcrafted RL algorithms is the adaptation of the update the total and remaining number of training steps (Auer) 2002 Farsang & Szegletes 2021). This adaptation to perceived learning horizon is even observed in humans, for instance, students using spaced repetition systems with decreasing intervals to maximize learning progress (Smith & Scarf (017).
In this work, take meta-learning approach discover novel RL objective functions that aware of and adapt to the amount of learning time remaining. We modify Learned Policy Gradi- (Oh al., [2020] LPG) and Learned Policy Optimization (Lu al., 2022a LPO), two recent meta-learned objective methods, to directly condition on temporal information about the agent's lifetime. We find the neta-optimization method is a critical component in successfully taking ad- vantage of this information, which we refer to as "lifetime conditioning". Specifically, meta meetiggrdieent
"Equal contribution. Correspondence jojjassonn rowotr.ooooooooooooo christopher. .x..eer.ox.cc..ukk

---

Published as a conference paper at ICLR 2024
approaches using myopic proxy objective (as in LPG) fail learn dynamic update. Instead, investigate the use of Evolution Strategies (Salimans et al. 2017 ES), which computes fitmess from the final return after the entire lifetime of the RL agent. These additions enable both of the methods to discover objective functions that dynamically adapt across the course of learning.
We evaluate the learned objective functions on both in-distribution and auttof-distribution environ- ments, ranging from continuous control tasks to discrete Atari-like settings, over range of training horizons. They significantly improve upon the performance of their non-temporally-wwrr coun- terparts, improving generalization to previously unseen training horizons and environments. Our results highlight the synergistic effects of combining lifetime conditioning with ES when meta- learning temporally-awar objective functions.
Our contributions are summarized follows:
1. We propose temporally- variants of LPG and LPO, named TA-LPG and TA-LPO. We augment their input spaces to contain temporal information, allowing for direct condi- tioning the total and remaining training horizon of the agent (Section 4.1).
2. We compare meta-gradient and evolutionary meta-optimization approaches for discovering non-myopic RL objective functions, finding that meta-gradient approaches fail to learn temporally-awye updates (Section 5.3). Based on this, propose the use of ES for TA- LPG, with a simple adaptation for multi-task meta-learning (Section 4.2).
3. After meta-training, we test the discovered objective functions over a range evalua- tion environments. Our temporally objective functions outperform all baselines we consider, generalizing to environments and training horizons unseen during meta-training (Section 5.2)
4. We analyze the dynamic schedules extracted by the discovered objective functions. We find that they implement dynamic policy importance ratio clipping, in addition, to update and entropy annealing schedules that effectively adapt to the training horizon, thereby balancing exploration and exploitation (Section 5.3).
RELATED WORK
Meta-learning fundamentals Meta-learning (or "learning to learn") aims to discover new prin-
ciples of learning via end-to-end optimization (Schmidhuber) 1987| Thrun & Prati 1998). These
approaches can be grouped into meta-gradient (Finn et al. [2017] et al. 2018) and memory-base based
(in-context learning) approaches (Hochreiter et al.] 2001 Wang al. 2016 Duan al. 2016). In- stead of training agents on a single task instance, meta-learning discovers parameterized learning algorithm components that generalize across task distribution. Two key related considerations arise: First, how flexible should the meta- meta-optimized medium be? Suitable inductive biases con- strain the meta-search space, which can facilitate stable and more data-efficient meta-optimization. However, this comes at the risk of prohibiting truly novel algorithm discovery. Second, how can we avoid overfitting to the meta-training task distribution (Kirsch et al., 2020 Jackson et al. 2023) and rollout horizon (Lange & Sprekeler, 2022) to find truly general-purpose components instead of heuristics? In this work, we argue that both these core problems can be tackled by making the learned algorithm aware of the learning horizon of the agent.
Meta-learning objective functions One specific approach to meta-learning is concerned with the meta-optimization of parameterized objective functions represent learning algorithms (Houthoofi et al. a 2018 Bechtle et al.) 2021). More recently, several advances were achieved in order to facilitate generalization. These include, e.g., MetaGenRL (Kirsch et al. 2020). LPG (Oh et al., 2020). LPO (Lu et al.) 2022a) and GROOVE (Jackson et al. (023). Finally, several attempts have been made obtain interpretable RL algorithm (components) using, for example, symbolic methods (Alet) et al. 2020 Co-Reyes et al.) (021). Here, we leverage LPG and I.OO to demonstrate that further performance improvements can be achieved by giving the algorithm access its current position within its total lifetime.
Meta-optimization with Evolutionary Zeroth-order methods Previously, it has been observed that meta-gradient approaches allow only for short inner loop algorithm unrolls due to memory
2

---

Published as a conference paper at ICLR 2024
constraints and chaotic dynamics (Metz al. 2021 Wu al. 2018). This in turn can lead to short-sightednnes (myopia) of the discovered element. An alternative approach is to perform black- box optimization using evolutionary optimization. This circumvents the computation of higher- order gradients and accommodates full algorithm rollouts. 'Tis has been successfully applied in the context of learning gradient-based (Metz al. [2022) and gradient-free optimizers (Lange al., 2023a b) as well as in-context learning (Kirsch & Schmidhuber 2021), adversarial learning (Lu al. 2022b 2023b) and neural auto-curricula (Feng al. 2021).
BACKGROUND
3.1
META REINFORCEMENT LEARNING
Discovering new reinforcement learning algorithms is a form of meta-reinforcement learn- ing (Schmidhuber 1987 Parker-Holder et al. 2022). However, most meta-reinforcement learning approaches aim to adapt to a small range of tasks (Duan et al. 2016 Wang et al. 2016 Finn et al,
2017 Houthooft et al. 2018 Bechtle et al., 2021 such as varying rewards signals or environment parameterizations. Here, we follow a line of work that aims to meta-learn reinforcement learning algorithms that can learn across wide range of reinforcement learning environments (Kirsch al., 2020 Oh et al.) 2020 Lu et al. (022a)) such as generalizing to robotic control environments of dif- ferent actuators and environment dynamies or from simple grid worlds to robotic control problems.
More formally, in meta-RL we search for learning algorithms that solve reinforcement learning (RL) problems, in the simplest case defined as Markov Decision Processes (MDPs). An MDP is defined by the tuple (S, A, Rt P, 7, d). At each timestep I, an agent takes an action sampled from its policy a₂ ~ (-1ss) (where or E A and & E S). The environment then returns a reward R(se, ar) and samples the next state St+1 according to the transition function St+1 P(- (S., a,). The objective is to find a policy π (parameterized by 8) that maximizes the expected return:
20
(TT) A E [o] B,g To
Σ 7' R(s
In our approach, meta-optimization operates in a space of RL algorithms, represented by meta- parameters ø, which are used to update the policy parameters 6. For example, if Φ parameterizes a loss function Ls that we optimize with gradient descent (using learning rate n1,, then 0x+1 O2 Vogg Let Ia, be the policy after k updates to an initial policy ne, with d. We maximize the expected return after K updates, or at end of agent training, which we define as:
F())= = Fe, [J(Tx))
In our work, we focus on instances of meta-RL that parameterize surrogate loss functions with and apply gradient-based updates to my We focus on two recent instances thereof: Learned Policy Gradients (Oh et al. 2020 LPG) and Learned Policy Optimization (Lu al. 2022a LPO)..
3.2 LEARNED POLICY GRADIENT (LPG)
Learned Policy Gradient (LPG) (Oh et al. 2020 is meta-RL approach that generalizes actor-critic RL (Barto et al., 1983). Rather than training a critic to output a scalar value estimate, it trains a critic that outputs bootstrap vector. The interpretation of this vector is meta-learned by LPG, allowing to encode information beyond value estimation that helps it optimize the inner policy, e.g., state visitation.
LPG is parameterized by an LSTM (Hochreiter & Schmidhuber 1997), ) which iterates over a re- versed sequence of agent transitions. Specifically, compute an update to the joint actor-critic parameters at timestep 4, LPG outputs targets Dt, कe = Us 1.7,,,, Here, the input vector x, is constructed from the transition Te (&t, (et (e; de, 81+1) at timestep by
Sa [re, de, 7, ne (ar|s)), (oo)ss),((())) (o(),, yo (8)+1)),
containing reward rt, ciisoode-erminatioon flag d,, discount factor 7 probability of the chosen action (a (8)), and bootstrap vectors for the current and next states yo ysx)) and (0(++)1). The targets and ă update the critic and policy respectively, giving the update rule
AO x E,, (Vo log (a(ss)) ag Vo Duo [l))]
(4)
3

---

Published as a conference paper at ICLR 2024
LPG is optimized over series of update steps using meta-gras graddggaa,,nss, to maximize the return of the policy at the end of training. Given a distribution of environments and initial agent parameters tu. these meta-gratee -erta-gdadeents computed
A x Eg, V log To, (als))
where Tox is the policy produced from initialization 0g after updating with Up for K steps. This computation requires backpropagation through the agent's entire learning process, making its computation limited by memory constraints. Therefore, a truncated backpropagation window is used in practice, which optimizes the agent over k < K update steps.
3.3 LEARNED POLICY OPTIMISATION (LPO)
Learned Policy Optimisation (Lu et al. 2022a, LPO) is meta-RL method that inherits theoreti- cal convergence guarantees from the Mirror Learning framework (Kuba et al., 2022). In Mirror Learning, agents maximize
Be 14 (s,a)] D aad (Tnew|s)
(6)
where (= a) - E| So = 8, a, = a] (the state-action value function of π) and D is the drift function, which maps from two policies in given state scalar. If D satisfies specific conditions from Kuba et al. (2022),
1. It is non-negative everywhere and zero at identity ne (T|s) 2 Drx (Te|s) = O,
2. Its gradient with respect to π is zero at π Tk.
then the Mirror Learning algorithm provably improves monotonically and converges to the optimal return in the limit. Many existing RL algorithms instances of Mirror Learning. Most notably, PPO (Schulman et al., (017) an instance of Mirror Learning since its clipped objective can be reformulated as drift function.
LPO parameterizes D with a neural network and structures its inputs and network architecture to guarantee the drift function conditions hold. The network does use bias parameters and takes the following vector as input:
(7) where p is the probability ratio between the current policy and the original policy for a given state- action pair (s, a) from the environment p A Xaex (o|s) and A is the estimated advantage of the state-
Ap.A = [[1- p)? (1-p)A, (1 - A, log(p), log(p)?, log(p) log(p)?
(al(a)s)
action pair A, (s. a) (x a) (9, â)]. By removing the bias parameters and mul- tiplying all inputs by (1 p) log(p), this input parameterization guarantees that the drift function conditions hold.
LPO uses evolution strategies (Salimans et al. 2017 ES) to optimize D. ES estimates the gradient of smoothed black-box function from samples. Formally,
V VxEL_Nio.La [x + oe)] (ewN((,,L))
[
- F(x + [
(8)
where F is the objective defined in Equation 2
4 METHOD
4.1
CONDITIONING ON AGENT LIFETIME
We define the current lifetime of the agent n/N, where is the number of environment interac- tions the agent has had so far and N is the total number of environment interactions it will have, ie., the training horizon. By conditioning on this information, we propose two extensions to exist- ing learned-loss meta-RL methods: Temporally-Adaptiv Learned Policy Gradient (TA-LPG) and Temporally-Adaptive Learned Policy Optimization (TA-LPO)).
4

---

Published as a conference paper at ICLR 2024
Temporally-Adaptin LPG We incorporate lifetime conditioning into LPG by appending the agent's 's lifetime and the training horizon to the optimizer input. Specifically, we take the logarithm of the training horizon log(N) due to this input being unbounded. This results in input of
(r. d, y, (T(a)s)) (s), vo(s), N Iog(AN)]
2t
Demporally-Adaptivv LPO To condition LPO on an agent's lifetime, we provide the lifetime as input to D. the drift function network. We follow LPO's implementation and structure the input to guarantee that the theoretical drift function conditions from Equation 6 holdd In particular, we input
n PPpA
P.,A.
pppA
(10)
where Ep.A defined in Equation 7
Since TA-LPO is only meta-trained on a single environment and horizon, we do not include the (N) term since it does not vary across updates.
4.2 META- OPTIMIZATION FOR LIFETIME ADAPTATION
Gradient-free gradient-based meta-optimization A range of meta-RL methods, including LPG, compute the gradient of the meta-parameters in order to update the learned loss function during meta-training (Kirsch et al. 2020 Bechtle et al. 2022]. When meta-optimizing over multiple update steps, these meta- gratdenn approaches require backpropagation through time (Werbos) 1990). This raises range of stability issues, particularly early in meta-training when the update rule is unstable and the inner policy gradient may display chaotic dynamics (Metz et al. 2021). On top of this, the memory requirement severely limits the number of steps that can backpropagated through, requiring the backpropagation window to be truncated to reduced number of steps, typically far below the training horizon. This leads to a myopic proxy objective for meta-gradient methods, which may prevent them from effectively conditioning on an agent's lifetime. We investigate this in Section 5.3
ES is attractive alternative backpropagation baaer based meta- veta-gradients for meta-optimization,, it can optimize through an unlimited number of inner loop update steps with no additional memory requirement. This makes it possible to optimize over the entire lifetime of agent, such that the objective becomes the agent's 's return at the end of training, rather than after a truncated series of updates. ES has previously been used for meta-learning objective functions in the single-task setting Section 3.3). To avoid instabilities in the multi-task setting, we propose a simple adaptation to ES.
Adapting ES to multi-task meta-learning Given a large or continuous distribution tasks E, infeasible to evaluate the expected fitness of each parameter candidate Egag (F.( + ac) across the entire task distribution, where F. is the fitness function on task e. In contrast, evaluating all candidates on an individual task e & each update step leads significant bias in the meta- meta-update, since it is derived from a single task. To balance between these, we could evaluate each candidate e; on distinct task e, E. This balances these factors but can increase update variance, particularly when return is not normalized between tasks.
To solve this, we propose antithetic task sampling, which builds upon antithetic sampling (Geweke) 1988). In antithetic sampling, pairs of candidates x + ae and x - de are evaluated for each sampled noise vector c, reducing update variance in practice. the multi-task setting, this has the form
-
(Ee<e [Fe (x + σe)] Hews (P (x oe)))] 2a
C
(11)
We adapt this method by evaluating each antithetic candidate pair on the same randomly sampled task, giving the estimator
BREE [ 22 (Fa( ac) F -F((x
(12)
In practice, we first apply a rank transformation over the pair's fitness, which is equivalent to select- member. In doing so, we normalize across tasks by returning uniform
ing the fitness from each task. We found this stabilized training and led to higher performance preliminary experiments, so adopted this approach for all multi-task ES optimization.
5

---

Published as a conference paper at ICLR 2024
Dense
Sparse
Long
1.25
0.96
1.00
1.00
A2C- TIC-Nomalized Return
0.90
0.75
TA-LPG
0.75
LPG
Total Train Sleps
0.84
0.50
Standard maze
Sixteen rooms
Small corridor
2
1.25
1.25
1.25
1.00
1.00
1.00
0.75
0.75
0.75
25
0.50
0.50
0.50
2
2
2"
2
2
2
2
11
2
2
Train Step
Figure 1: TA-LPG adapts to variable training horizons. Training curves for LPG and TA-LPG on held-out Grid- World environments from Oh et al. (2020) (top) and Minigrid (bottom), for a range of total train steps. Since TA-LPG adapts the training horizon, we plot individual training curves for each horizon (faded lines) and their final return (bold points), with the color gradient reflecting the horizon for each model. We observe the final return of TA-LPG each horizon is consistently greater than the LPG return at the same point. Return is normalized against an A2C agent trained to convergence and averaged over meta-train and 128 meta-test seeds.
EXPERIMENTS
To investigate the effectiveness of lifetime conditioning, we evaluate TA-LPG and TA-LPO against their non-temporallyy-wwee counterparts. [Section 5.1 describes our experimental setup in more de- tail. In [eection 5.2, we demonstrate improved performance from TA-LPG against LPG over a range of training horizons, before showing improved performance from TA-LPO against LPO out-of- distribution environments and analyze TA-LPG and TA-LPO in Section 5.3
5.1 EXPERIMENTAL SETUP
Model architecture and training We use the reference model hyperparameters for both LPG and LPO (Appendix Ap. As discussed in Section 4.2] we use ES with antithetic task sampling for the meta-optimization of LPG, Secttin 5.3] where we ablate ES against the original gradient-based meta-optinization method. As in the reference work, ES is used for LPO throughout, We implement the entire training process in JAX (Bradbury et al. 2018). using evosax (Lange 2023) for evolution.
Training environments In experiments, follow the meta-training environments originally used Oh et al. (2020) and Lu et al.] (2022a) for LPG and LPO respectively. LPG is meta-trained in multi-task setting over continuous distribution of Grid-Word environments. with variable training horizons per task. In contrast, LPO is meta-trained on a single environment (MinAtar Space Invaders (Young Tian 2019 Lange 2022), with fixed training horizon. For this reason, we train and evaluate LPG for variable horizon adaptation, whilst we evaluate the use of relative time-step information with I.PO.
Testing environments At meta-test time, we evaluate LPO on the MinAtar and Brax (Freeman et al. 2021) evaluation suites, using the PPO byperparameters from PureJaxRL (Lu et al. 2022a). Notably, while the hyperparameters are shared within each suite, they differ significantly between the two. We evaluate LPG on held-out Grid- World configurations from the reference LPG publica- tion and Minigrid CChevalier-ooiseetttee
Boisvert et al. 2023
range of training horizons.

---

Published as a conference paper at ICLR 2024
Brax Performance
Med an
IQM
Mean
Optimality Gap
LPO
PPO
1.02 1.08 1.14
1.02
1.08
1.14
1.20
1.0
1.2
1.3
0.03
0.06 0.09
MinAtar Performance
Med
IOM
Optimallty Gap
TA-LPO
LPO
PPO
1.04 1.12
0.96
1.14
1.0
11
0.04 0.08 0.12
Figure 2: TA-LPO leverages lifetime information and generalizes to wide range of environ- ments. Results of TA-LPO, LPO and PPO on the Brax and MinAtar suites across three seeds. TA-LPO was only meta-trained on Mancelnvader--MnnAan We provide complete training curves in Appendix B
5.2 BENCHMARK EVALUATION
Lifetime conditioning enables adaptation to variable training horizons. In order to investigate the adaptation variable training horizons, we evaluate LPG and TA-LPG over range of horizons on set of held-out Grid-Worlde (Figure I). We observe consistently higher performance in final return from TA-LPG across all horizons. This includes short horizons wwith TA-LPG reaching maximum performance on the "sparse" task in 1/8 of the training steps required for LPG- and long horizons, such as in the "dense" task where TA-LPG continues to improve performance after LPG has plateaued.
Examining the training curves different horizons on each task, we note that the performance of TA-LPG with longer training horizons is typically dominated by TA-LPG with shorter horizons each step. However, these long-howe long-horizon runs then achieve a higher return beyond the short horizon, surpassing their final return. This reflects desired mode of learning discovered by these models, being the sacrifice intra-training performance to produce a superior final return (c.f. exploration exploitation trade-off).
We evaluate TA-LPG against LPG with a heuristic learning rate schedule in Appendie CJ The heuris- tic schedule has insignificant impact LPG's performance, with TA-LPG again outperforming across training horizons. This demonstrates the insufficiency of combining static objective functions with heuristic schedules, in contrast the success of temporally-an aware objective functions.
Lifetime conditioning improves performance on shows that TA-LPO significantly outperforms both LPO and PPO on wide range of tasks from the MinAtar and Brax environment suites. Notably, LPO and TA-LPO were only meta-trained on Space Invaders from the MinAtar suite. While the MinAtar PPO hyperparameters train on 10M environment steps with rollouts of length 128, the Brax hyperparameters train on 50M environment steps with rollouts of length 5, showing that the learned policy objective is robust to different hyper- parameters. Interestingly, the performance of TA-LPO often only exceeds PPO towards the end of training, as seen in /Appendix B, implying that PPO converged too early. Early convergence in RL common failure mode RL algorithms (Nikishin al. 2022 Lyle et al. (022). These results potentially suggest that lifetime awareness could be one approach to addressing this issue.
environments. Figure 2
5.3 ANALYSIS OF LEARNED UPDATE
Adaptation training horizon. In Figure 3 we show the policy entropy and update norm of policies trained with TA-LPG and LPG over a range of training horizons. We observe adaptations from TA-LPG in both of these metrics as the training horizon increases. Policy entropy decays over training with both models, but TA-LPG decays entropy at slower rate the training horizon in- creases, before reaching a similar minimum entropy (beyond horizons 2 steps). This reflects balanced exploration from these models, by maintaining high entropy until late in training, regard-
7

---

Published as a conference paper at ICLR 2024
1.5
12.5
LPG
Policy entropy
1.2
Policy update
0.9
7.5
0.6
5.0
2
2
2
2
2
2
2
2
2
Train step
Train step
Figure 3: Lifetime conditioning enables adaptation to training horizon. Policy entropy and update norm of TA-LPG and LPG over randomly- Grid-Worlds.
less of horizon. In contrast, LPG learns horizon-invariant update, which induces static entropy annealing schedule. Due to this, the maintenance of an exploratory training policy for long-horizon tasks requires the sacrifice of performance on short-horizon tasks. We observe similar behavior when analyzing the policy update norm, with both the average update norm and rate of increase in norm inversely proportional the training horizon.
Adaptation to relative training step. In Figure 4 we follow the analysis used for LPO (Lu et al., 2022a) and visualize the derivative of the TA-LPO objective at different points in training. The æ-axis is the likelihood ratio p. the y-axis is the advantage of a given transition, and the color corre- sponds to the gradient for transition at that point. Positive values (red) increase the likelihood of the given transition (i.e., push the likelihood to the right) and negative values (blue) decrease them. There are two key features of the objective function:
Asymmetric Rollback Schedule. At 0, the lower left quadrant, which corresponds to < and < 1 (a transition with a negative advantage and a relative decrease in action probability) has large positive region. This suggests that the discovered algorithm, instead of clipping gradients for negative advantages, performs rollback (Wang et al. 2020) a more aggressive form of regu- larization that reverses the gradient for extreme ratios. Meanwhile, in the top right quadrant, which corresponds to A 0 and not perform clipping at for positive advantages. These findings align closely with the original LPO analysis (Lu et al. 2022a) and suggest optimistic objective. However, the exact opposite. There is rollback for positive advantage and no clipping for negative advantage.
1, the region is still positive, suggesting that the algorithm does
N, we see This corresponds high level of risk aversion by the algorithm the end of training.
Implicit Entropy Regularization Schedule. At n = 0, the gradient is, on average, more positive than negative, throughout the plot. This corresponds to entropy maximization because it encourages the agent to increase the probability of sampled actions. At r N the opposite holds, which can be viewed as minimizing entropy.
Overall, the analysis suggests that TA-LPO behaves similarly to LPO at n (it is optimistic and maximizes entropy), but over time becomes pessimistic and discourages entropy.
Meta-gradient optimization fails to learn an adaptive update. As outlined in Section 4.2 Oh al. (2020) optimize LPG using meta-gradients, by backpropagating through truncated window of updates. Since this proxy objective is myopic, we hypothesize that LPG is unable to effectively exploit temporal information when optimized with meta--radients. To evaluate this, we trained TA- LPG with meta- gradiintsssstsss and evaluated against the ES-trained method over a distribution of random Woid-Worlss (Figure 5). We observe a lower return from the meta-gradien model at all training horizons, in addition to no sign of horizon adaptation, in contrast to the ES-optimized model. To further analyze the learned update, we plot the L2-norm the agent update when trained with the meta-gradient model over a range of training horizons. Whilst the update norm responds to the rela- tive train step n n/N, the schedule is very similar for each evaluated horizon, with no consistent sign of adaptation. These results demonstrate the inability of meta-gan veeaaggrddeett methods to learn temporal awareness and potentially suggest that truncated meta-optimization is insufficient for discovering long-sighted algorithms.
8

---

Published as a conference paper at ICLR 2024
Derivative
Derivative
Derivative n 1e-6
1
6
01
0
-2
-3
20.00
2,00
2.00
Ratio
Ratio
Retio
Figure 4: TA-LPO learns to switch from optimism to pessimism. A visualization of the derivative of the TA-LPO objective at the beginning (n 0, left), middle (n N/22, center), and end (n N, right) of the training lifetime. The objective at n = 0 appears to be optimistic and maximize entropy while the objective at appears be pessimistic and minimize entropy.
210
0.05
Policy update L2
Total Train Steps
1.5
Relum
2
1.0
0.03
ES
0.5
Meta-gradients
0.02
2
2
0.0
0.2
0.4
0.6
0.8
1.0
Train Step
Relative train step
Figure 5: Meta-gradient TA-LPG fails to adapt to temporal information. Left: Final return of TA-LPG trained with ES and meta-gradient, the meta mntaatrriiing distribution of randomly sampled WorddWordss. Individual training curves are plotted for each horizon (faint lines), but are indistin- guishable for meta-gradient TA-LPG due lack of horizon adaptation. Right: Policy update norm of the meta-gradient TA-LPG model over inner loop training, for a range of training horizons. We observe lower meta-gradient performance across horizons and consistent sign of horizon adapta- tion in update norm.
CONCLUSION
Summary We introduced novel family of parameterized RL objective functions that are tem- porally aware. After meta-training. the learned objective functions demonstrate adaptation over agent's lifetime, resulting in strong generalization to unseen training horizons and environments. Furthermore, we showed that they incorporate adaptive motifs that change throughout learning, with the high-level behavior demonstrating shift from optimism to pessimism over the training process. Finally, we demonstrated evolutionary optimization as a critical factor for discovering RL algorithms capable of effective lifetime conditioning.
Limitations We show that the discovered lemporally-aware algorithms are capable of broad gener- alization both discrete pixel-based environments and continuous control settings well as various time horizons. Nonetheless, for LPG it is difficult to obtain strong (theoretical) robustness guaran- tees. Such robustness would benefit reliable deployment. On the other hand. LPO meta-learns in much more restricted space than LPG and may be less expressive.
Future Work Our contributions highlight the importance of expressivity in discovered RL algo- rithms. Here, we only considered conditioning on temporal information. In the future, believe that further performance gains can be achieved by augmenting algorithms with more information such optimizer statistics, network architectures, and task information. Eventually, may be able to remove these handpicked augmentations and perform RL purely in-context with blackbox meta-learning (Lu et al. 2023al Kirsch al. 2023

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
Matthew Jackson is funded by the EPSRC Centre for Doctoral Training Autonomous Intelligent Machines and Systems, and Amazon Web Services.
ETHICS STATEMENT
We propose method for the automated discovery of time-aware RL algorithms that employ gradient-based learning on flexible meta-learned loss functions. Like in other meta-leaminng ap- proaches, leaming-to-lean automates and thus reduces our knowledge about the learning process. This requires additional monitoring and introspection, such the analysis have provided in Figure 4 The process is aided by evolutionary methods and can potentially result in a system that improves its own performance throughout the course of learning.
REPRODUCIBILITY STATEMENT
The method presented in our paper can be performed with a small modification to an implementa- tion of LPO and LPG. We describe this modification Secction 4 and our hyperparameters in Ap- pendix A Our implementation of LPG, LPO, and their temporally-aware modifications (TA-LPG, TA-LPO) can be found at htttp///gihbb..oom////y//aa/oo//
REFERENCES
Ferran Alet, Martin F Schneider, Tomas Lozano-Perez, and Leslie Pack Kaelbling. Meta-learning curiosity algorithms. arXiv preprint arXiv:2003.05325, 2020.
Peter Auer. Using confidence bounds for exploitation-exxllraaion trade-offs. Journal of Machine Learning Research, 3090v):3 2002.
Andrew G. Barto, Richard S. Sutton, and Charles W. Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics, SMC-13(5):3348846 1983, doi: 10.1109/TSMC.1183..333
Sarah Bechtle, Artem Molchanov, Yevgen Chebotar, Edward Grefenstette, Ludovic Righetti, Gaurav Sukhatme, and Franziska Meier. Meta learning via learned loss. 25th International Conference
Patlern Recognition (ICPR), pp. 4161-4168. IEEE, 2021.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Min Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http:
7/ www.hu..cc////poggggo..ee
Maxime Chevalicr-Boisvertt, Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. CoRR,
abs/2306.13831, 2023.
John D Co-Reyes, Yingjie Miao, Daiyi Peng. Esteban Real, Quoc V Le, Sergey Levine, Honglak Lee, and Aleksandra Faust. Evolving reinforcement learning algorithms. In International Con- ference Learning Representations, 2021.
Yan Duan, John Schulman, Xi Chen, Peter I. Bartlett, llya Sutskever, and Pieter Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.000 1.02799,, 2016.
Mónika Farsang and Luca Szegletes. Decaying clipping range in proximal policy optimization. In 2021 IEEE 15th International Symposiun on Applied Computational Intelligence and Informatics (SACI), pp.
IEEE, 2021.
Xidong Feng, Oliver Slumbers, Ziyu Wan, Bo Liu, Stephen McAleer, Ying Wen, Jun Wang, and Yaodong Yang. Neural auto-curricula. arXiv preprint vrXiv:2106.02744 2021.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leant veta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126-1 135. PMLR, 2017.
10

---

Published as a conference paper at ICLR 2024
C Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax-a differentiable physics engine for large scale rigid body simulation. arXiv preprint 12Xi::2106.1328,, 2021.
John Geweke. Antithetic acceleration of monte carlo integration in bayesian inference. Jour- nal of Econometrics, 38(1):73-89,
1988. ISSN 0304-4076. doi: htts://doi.org//
1016/0004-4006668990007--- URL https://wwwt sc scieiii www. com/science/
Aaticilc/BL/33344078880022
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735 1780, 1997.
Sepp Hochreiter, A Steven Younger, and Peter Conwell. Learning to learn using gradient descent. In Artificial Neural Networks-ICAN 2001: International Conference Vienna, Austria, August 21-25, 2001 Proceedings I, pp. 87-94. Springer, 2001.
Rein Houthooft, Richard Y Chen, Phillip Isola, Bradly C Stadie, Filip Wolski, Jonathan Ho, and Pieter Abbeel. Evolved policy gradients. arXiv preprint arXiv: 2018.
Matthew Thomas Jackson, Minqi Jiang, Jack Parker-Holder, Risto Vuorio, Chris Lu, Gregory Far- quhar, Shimon Whiteson, and Jakob Nicolaus Foerster. Discovering general reinforcement learn- ing algorithms with adversarial environment design. In Thirty-sevnnth Conference Neural Information Processing Systems, 2023.
Louis Kirsch and Jürgen Schmidhuber. Meta learning backpropagation and improving it. Advances in Neural Information Processing Systems, 14:14122-,4434, 2021.
Louis Kirsch, Sjoerd van Steenkiste, and Juergen Schmidhuber. Improving generalization in meta reinforcement learning using learned objectives. In International Conference on Learning Repre- sentations, 2020.
Louis Kirsch, James Harrison, C Freeman, Jascha Sohl-Dickstein, and Jürgen Schmidhuber. To- wards general-purposs purpose in-context learning agents. In NeurlPS 2023 Workshop on Generalization in Planning, 2023.
Jakub Grudzien Kuba, Christian Schroeder de Witt, and Jakob Foerster. Mirror learning: unifying framework of policy optimisation. arXiv preprint arXiv:2201.0233 2022.
Robert Lange, Tom Schaul, Yutian Chen, Chris Lu, Tom Zahavy, Valentin Dalibard, and Sebastian Flennerhag. Discovering attention based genetic algorithms via meta-black-boo optimization. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 929-937, 2023a.
Robert Lange, Tom Schaul, Yutian Chen, Tom Zahavy, Valentin Dalibard, Chris Lu, Satinder Singh, and Sebastian Flennerhag. Discovering evolution strategies via meta-black-boy optimization. In Proceedings ofthe the Companion Conference Genetic and Evolutionary Computation, pp. 29-30, 2023b.
Robert Tjarko Lange gymnax: A jax-based reinforcement learning environment library, 2022b.
URL http://github. com/Rober/TLagggggnnee 2022.
Robert Tjarko Lange. evosax: Jax-based evolution strategies. In Proceedings of the Companion Conference on Genetic and Evolutionary Computation, pp. 659-662, 2023.
Robert Tjarko Lange and Henning Sprekeler. Learning not to learn: Nature versus nurture in silico. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 7290-7299, 2022.
Chris Lu, Jakub Kuba, Alistair Letcher, Luke Metz, Christian Schroeder de Witt, and Jakob Foerster. Discovered policy optimisation. Advances in Neural Information Processing Systems, 35:16455- 16468, 2022a.
Chris Lu. Yannick Schroecker, Albert Gu. Emilio Parisotto, Jakob Foerster, Satinder Singh, and Feryal Behbahani. Structured state space models for in-context reinforcement learning. arXiv preprint arXiv:2003.03982, 2023a.
11

---

Published as a conference paper at ICLR 2024
Chris Lu, Timon Willi, Alistair Letcher, and Jakob Nicolaus Foerster. Adversarial cheap talk. In International Conference on Machine Learning, pp. 22917-22941. PMLR, 2023b.
Christopher Lu, Timon Willi, Christian Schroeder De Witt, and Jakob Foerster. Model-free op- ponent shaping. In International Conference Machine Learning, pp. 14398- 14411. PMLR, 2022b.
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in rein- forcement learning. arXiv preprint arXiv:2204.09560, 2022.
Luke Metz, C Daniel Freeman, Samuel S Schoenholz, and Tal Kachman. Gradients not all you need. arXiv preprint arXiv:2111.0580) 2021.
Luke Metz, James Harrison, Daniel Freeman, Amil Merchant, Lucas Beyer, James Bradbury, Naman Agrawal, Ben Poole, Igor Mordatch, Adam Roberts, et al. Velo: Training versatile learned optimizers by scaling up. arXiv preprint arXiv:2211.09760) 2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, nature, 51 5287540)::529 533, 2015.
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International Conference Machine Learning, pp. 16828-16847. PMLR, 2022.
Junhyuk Oh, Matteo Hessel, Wojciech M Czarnecki, Zhongwen Xu, Hado van Hasselt, Satin- der Singh, and David Silver. Discovering reinforcement learning algorithms. arXiv preprint arXiv:2007.08794 2020.
Jack Parker-Holder, Raghu Rajan, Xingyou Song, André Biedenkapp. Yingjie Miao, Theresa Eimer, Baohe Zhang, Vu Nguyen, Roberto Calandra, Aleksandra Faust, et al. Automated reinforcement learning (autorl): A survey and open problems. Journal of Artificial Intelligence Research, 74: 517-568, 2022.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies scalable alternative to reinforcement learning. arXiv preprint 1rXiv:1703.000 03864, 2017.
Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-... hook. PhD thesis, Technische Universität München, 1987.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMIR, 2015.
John Schulman, Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv, abs/1 700s/170706347, 2017.
Christopher D Smith and Damian Scarf. Spacing repetitions over long timescales: a review and a reconsolidation explanation. Frontiers in Psychology, 8:962, 2017.
Sehastian Thrun and Lorien Pratt. Learning to learn: Introduction and overview. In Learning to learn, pp. 3-17. Springer, 1998.
Jane Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning reinforcement learn. arXiv preprint arXiv: 161 1.05663, 2016.
Yuhui Wang, Hao He, and Xiaoyang Tan. Truly proximal policy optimization. In Uncertainty in Artificial Intelligence, pp. 113-122. PMLR, 2020.
Paul J Werbos. Backpropagation through time: what does and how do it. Proceedings of the IEEE, 1810):1550-11660 1990.
Yuhuai Wu. Mengye Ren, Renjie Liao, and Roger Grosse. Understanding short-horizon bias in stochastic meta-optimization. arXiv preprint arXiv: arXiv:1833.008 1803.0201, 2018.
12

---

Published as a conference paper at ICLR 2024
Zhongwen Xu, Hado P van Hasselt, and David Silver. Meta-gradient reinforcement learning. Ad- vances in neural information processing systems, 31, 2018.
Kenny Young and Tian Tian. Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments. arXiv preprint wxXiv:1903.03176 2019.
13

---

Published as a conference paper at ICLR 2024
A HYPERPARAMETERS
A.1 LPG
Table 1: LPG hyperparameters
Hyperparameter
Value
Optimizer
Adam
Learning rate
le-4
Discount factor
0.99
Number of interactions per agent update
20
Number of parallel lifetimes
512
Number of parallel environments per lifetime
64
Policy entropy coefficient (3p)
0.05
Bootstrap entropy coefficient (3)
0.001
L2 regularization coefficient for Å (3g)
0.005
L2 regularization coefficient for 9 (3g)
0.001
Number of agent updates per optimizer update 5
ES Learning Rate Decay
0.999
ES Learning Rate Limit
le-5
ES Sigma Init
0.003
ES Sigma Decay
1.0
ES Sigma Limit
0.001
14

---

Published as a conference paper at ICLR 2024
A.2 LPO
The drift function is parameterized by a one-layer fully-connected network with hidden layer and 128 hidden units. Meta-training was done 2 A A100 GPUs with synchronous updates.
Table 2: Important parameters for Training LPO and PPO
Hyperparameter
Value
Meta-Evolution
Population Size
64
Number of Hidden Layers
Size of Hidden Layer
128
Number of Generations
128
Centered Ranking
True
ES Sigma Init
0.04
ES Sigma Decay ES Sigma Limit
0.999
0.01
MinAtar
Number of Timesteps
le7
Number of Environments
64
Unroll Length
128
Number of Minibatches
8
Number of Update Epochs
4
Learning Rate
5e-3
Gamma
0.99
Max Grad Norm
8.0 for LPO, 0.5 for PPO
Entropy Coefficient
0.01
Brax
Number of Timesteps
5e7
Number of Environments
2048
Unroll Length
10
Number of Minibatches
32
Number of Update Epochs
4
Learning Rate
3e-4
Gamma
0.99
Max Grad Norm
8.0 for LPO, 0.5 for PPO
Entropy Coefficient
0.0
15

---

Published as a conference paper at ICLR 2024
B TA-LPO RESULTS
0.0
02
0.4
0.6
0.1
1.5
0.6
02
0.6
0.1
1.5
<depe
(a) Asterix
(b) Breakout
PP0
175
LPG
T6.PO
125
520
0.0
0.2
0.6
0.5
1.5
0.0
0.2
0.6
0.5
1.2
(c) Freeway
(d) Spacelnvaders
Figure 6: Performance of PPO, LPO, and TA-LPO the Minatar environments. The curves show the mean evaluation return across random seeds, with error bars showing standard error of the mean. LPO and TA-LPO were meta-trained on SpaceInvaders.
16

---

Published as a conference paper at ICLR 2024
7020
9932
4003
0033
140m
2002
1000
2003
(a) Ant
(h) HalfCheetah
6003
2230
2002
1750
1350
1250
1000
150
4002
(c) Hopper
(d) Humanoid
-300
7020
-350
5903
-005
2003
432
50e5s
Steps
(e) Inverted Double Pendulum
(f) Pusher
mT TP.PO
-20
2003
100
1300
1000
-140
160
302
363
sbece
369
Reacher
(h) Walker2d
Figure 7: Performance comparison between PPO (bluc), LPO (orange), and TA-LPO (green) in Brax environments. The curves show the mean evaluation return across random seeds, with error bars showing standard error of the mean. LPO and TA-LPO were not meta-trained on any of these environments.
17

---

Published as a conference paper at ICLR 2024
C HEURISTIC ANNEALING SCHEDULES
As a heuristic baseline for horizon adaptation, we augment LPG with a learning rate scheduler and compare performance against TA-LPG (Figure 8). As with the original LPG, TA-LPG consistently outperforms the scheduler at all training horizons. Furthermore, we observe that the the performance of LPG with learning rate scheduling is insignificantly different from its original performance (Fig- ure 1). This highlights the failure of the insufficiency of heuristic temporal adaptation methods and the significance of the method learned by TA-LPG.
Sparse
Long
1.25
211
1.00
A2C-Nome IIC-Nommaiized Return
0.90
0.75
0.75
LPG
Total Train Steps
0.84
0.50
Standard maze
Sixteen rooms
Small corridor
2
1.25
1.25
1.25
1.00
1.00
1.00
0.75
0.75
0.75
0.50
0.50
0.50
2
2
2
2
2
2
Train Step
Figure 8: Training curves for LPG with a tuned learning rate scheduler and TA-LPG.
18

---

Published as a conference paper at ICLR 2024
D HYPERPARAMETER SEARCH OVER ENTROPY
hopper
humanoid
walker2d
halfcheetah
ant
700-
+800
XC00
180
1000
6C00
2009
Retum
1290
0400
C195
043
045
6.0
9.00
9065
801
COML 6.0 0/6 6/65
0.00 8.05 641
6.5
06 0.60
0065
9.0
0.6
Entropy Coefficient
Figure 9: Hyperparameter Sweep for Brax environments over the entropy coefficient.
19


Reviews:
[{"review_id": "9elcZFk6cs", "reviewer": "Reviewer_47vA", "summary": "This paper introduces a training horizon to meta-reinforcement learning algorithms to discover objective functions that depend on the learner's lifetime. The basic idea is to add the information about the lifetime to the input vector of the meta-learning algorithm. The authors propose two algorithms, Temporally-Adaptive Learned Policy Gradient (TA-LPG) and TA-Learned Policy Optimization (TA-LPO) by extending LPG and LPO, respectively. Then, the authors found that the evolutionary strategy is more appropriate than meta-gradient approaches to optimize the upper-level objective function.", "questions": "1. The function $U_\\phi (x_t \\mid x_{t+1}, \\ldots, x_T)$ is defined in Section 3.2, but it is unclear because $T$ is not explained. Is it the total number of interactions, denoted by $N$ later?  \n2. Two variables $n/N$ and $\\log N$ are augmented in TA-LPG. Is $n/N = 0$ when $N$ is unbounded? In my understanding, $N$ is determined and fixed before training. Would you explain $N$ and provide an example task? \n3. In TA-LPO, $ \\frac{n}{N} x_{r, A}$ is augmented. It is not straightforward to me because it is proportional to $x_{r, A}$, which means linearly-dependent. Would you explain the problem if $n/N$ and $\\log N$ are added as the authors did in TA-LPG? \n4. In the final paragraph of Section 4.2, the authors mentioned, \"We found this stabilized training and led to higher performance...\" Would you explain \"this stabilized training\" in detail? If it means the rank transformation, it is used in Salimans et al. (2017). Discussing the relation between two equations would be better if the stabilized training means Eq. (12) rather than (11). \n5. I do not fully understand the final paragraph of Section 3.1. In the manuscript, LPG and LPO are selected as the base algorithms. However, the authors mentioned as follows: In our work, we focus on instances of meta-RL that parameterize surrogate loss functions with $\\phi$ and apply gradient-based updates to $\\pi_\\theta$ (Houthooft et al., 2018; Kirsch et al., 2019; Bechtle et al., 2021). Does it mean that three algorithms are implemented somewhere?\n6. Equation (10): Is $x_{r, A}$ a typo of $x_{p, A}$? \nPlease update the reference Kirsch et al. (2019) to Kirsch et al. (2020). Please see https://openreview.net/forum?id=S1evHerYPr", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Based on the proposed idea, the authors implement two meta-reinforcement learning algorithms: TA-LPG and TA-LPO. It implies that the idea can be applied to various algorithms, although input augmentation depends on the algorithms. Systematic experiments show that TA-LPG and TA-LPO outperform LPG and LPO, respectively.", "weaknesses": "Although the experimental results support the authors' hypothesis that conditioning on the lifetime is helpful in meta-learning, there are no theoretical justifications. In addition, the way to incorporate the lifetime depends on the algorithm, and the experimenter carefully has to design augmentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IjlZvBCQXe", "reviewer": "Reviewer_c7mC", "summary": "This work proposes to meta-learn a better objective function for reinforcement learning (RL) tasks by taking into account the information of the task horizon.\nSpecifically, with the help of the time-step information, the proposed algorithms could find a better balance between exploration and exploitation.\nFurthermore, this work shows that meta-gradient methods fail to adapt to different horizons, while evolution strategies can do better in this case.", "questions": "- Section 4.1, after Equation 10, \"Since TA-LPO is only meta-trained on a single environment and horizon, we do not include the $\\log(N)$ term since it does not vary across updates.\" Would including the $\\log(N)$ term decrease the performance?\n- Section 5.2, Paragraph \"Lifetime conditioning improves performance on out-of-distribution environments.\" Figure Figure 2: typo.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea behind this work is simple and effective, supported by solid experiments and detailed analysis.", "weaknesses": "- The idea of incorporating time-step information is not novel in RL, such as this [work](https://proceedings.mlr.press/v80/pardo18a.html), which should be included in the related work.\n\n- Lack of ablation study: In Section 4.1 Equation 9, $n/N$ and $\\log(N)$ are included as part of the input to the agent. It is not clear or argued how good this option is compared with other options, such as $n$ and $N$, $N-n$, or $n/N$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "thJdGiWYB5", "reviewer": "Reviewer_hAap", "summary": "The paper investigates the problem of discovering reinforcement learning algorithms via meta-learning. The paper reveals that incorporating temporal data regarding the agent's lifetime empowers the newly discovered algorithm to dynamically refine its objectives, such as the exploration-exploitation tradeoff, thereby fostering the development of a higher-performing agent. The proposed method can be readily combined with two existing algorithm, LPG and LPO, and demonstrates its effectiveness on a wide variety of benchmarks, including MiniGrid and MinAtar.", "questions": "- Ablation: Could you please provide me a clear understanding of the individual contributions of the two proposed component to the performance improvements?\n- Hyperparameter: Setting the entropy coefficient of PPO to 0.0 in the Brax environment appears to significantly limit exploration in my view and poses questions about the experimental design. To address this and strengthen the validity of your results, I recommend conducting a systematic hyperparameter search on the entropy coefficient of PPO within the Brax environment.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Novelty: Although the proposed method is a simple modification of previous algorithms, the idea of utilizing temporal information is well-motivated and novel.\n- Presentation: The authors meticulously analyze the evolution of the discovered objectives throughout the agent's lifetime, detailed in Section 5.3, with Figures 3 and 4 providing compelling visual representations of these dynamic shifts.\n- Experiment: The authors meticulously design experiments to rigorously assess the generalization ability of the proposed method to different environments or training hyperparameters (e.g. the number of environment interactions).", "weaknesses": "- Novelty: The application of antithetic sampling for gradient estimation, while prevalent in black-box optimization, appears to lack originality.\n- Ablation: The authors do not conduct a comprehensive ablation study comparing the proposed TA-LPG with the standard LPG. Specifically, they introduce two significant modifications, incorporating temporal information and employing antithetic sampling, without isolating the effects of each change.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)