Meta-learning Population-based Methods for Reinforcement Learning
Authors: Johannes Hog
Raghu Rajan
André Biedenkapp
Noor Awad
Frank Hutter
Vu Nguyen
Venue: Accepted by TMLR ()

Abstract: Reinforcement learning (RL) algorithms are highly sensitive to their hyperparameter settings. Recently, numerous methods have been proposed to dynamically optimize these hyperparameters. One prominent approach is Population-Based Bandits (PB2), which uses time-varying Gaussian processes (GP) to dynamically optimize hyperparameters with a population of parallel agents. Despite its strong overall performance, PB2 experiences slow starts due to the GP initially lacking sufficient information. To mitigate this issue, we propose four different methods that utilize meta-data from various environments. These approaches are novel in that they adapt meta-learning methods to accommodate the time-varying setting. Among these approaches, MultiTaskPB2, which uses meta-learning for the surrogate model, stands out as the most promising approach. It outperforms PB2 and other baselines in both anytime and final performance across two RL environment families.

Published in Transactions on Machine Learning Research (03/2025)
Meta-learning Population- based Methods for Reinforcement Learning
Johannes Hog
hogj wtnp:/cccuunffriibrr.. de
University of Freihury, Germany
Raghu Rajan
rajannr@ess.uni-feebbryy de
University of Freibury, Germany
André Biedenkapp
de
University of Freiburg, Germany
Noor Awad
awad@cs awad@cs.uni.freibee uni-freiburff de
University of Freiburg, Germany
Frank Hutter
fh bibes.unifrreibrg.. de
ELLIS Institute Tübhngen, Germany & University of Freihurg, Germany
Vu Nguyen
vu@ieee.org
Amazon, Australia
Reviewed on OpenReview: https: // opw.r/vee...eee...om formm? d9htasefP8
Abstract
Reinforcement lcarning (RL) algorithms are highly sensitive to their hyperparameter set- tings. Recently, numerous methods have been proposed to dynamically optimize these hyperparameters. One prominent approach Population-Basse Bandits (PR2), which uses time-varying Gaussian processes (GP) dynamically optimize hyperparameters with pop- ulation of parallel agents. Despite its strong overall performance, PB2 experiences slow starts due to the GP initially lacking sufficient information. To mitigate this issue, propose four different methods that utilize meta-data from various environments. These approaches are novel in that they adapt meta-learning methods to accommodate the time-varying setting. Among these approaches, MultiTaskPB2, which uses learnnn for the surrogate model, stands out as the most promising approach. outperforms PB2 and other basclines both anytime and final performance across RL environment families. The code to reproduce results is publiely available https::/ https://ggtthne haarb..ooaauuute Samm/attoo/MMeeaP2
1
Introduction
There have been several breakthroughs in Reinforcement Learning (RL) Sutton & Barto 2018 in appli- cations like games (Silver et al.
2016
Berner et al. 2019 Badia et al. 2020) and robotics (Andrychowicz
et al.] 2020; Kalashnikov hyperparameter optimization (HPO),
al.
2018 Lee al.) 2020). crucial component of these breakthroughs is
highly sensitive to the selection of their hyper- parameters (Emer et al.) 2023 Islam et al. 2017 While it is standard practice to employ learning rate scheduler to dynamically adjust the learning rate in Deep Learning GGooffelow et al. 2016), most other lyperparameters are kept fixed during training. This is not a suitable approach for RL where we have non-stationary data distributions that necessitate dynamncaall adjusting all the hyperparameters, not only al.) (022).. These non-stationary data distributions
RL algorithms
the learning rate (Mohan et al. 2023 Parker-Holdee arise due the learning process the RL agent. As the agent's policy is updated, its interactions with the environment change, resulting in the generation of new learning data that differs from previous experiences.

---

Published in Transactions on Machine Learning Research (03/2025)
Copy weights
Performance
Weights
Training
Hyperparameters
Portfolio
explolt
1.
2.
BO explore
PB2
Meta data
Informing
Previous
8
An overview of the different ways we use meta-data to inform PB2. PB2 initializes a population of agents with random hyperparameters and model weights. One of our methods warmstarts the hyperparam- eters using portfolio. The population trained for a fixed number of steps, referred as the perturbation interval. After this, the weights of well-performing agents are copied over poorly performing agents (ex- ploit) and these agents receive new hyperparamelers (explore) through Bayesian Optimization (BO). Some of our methods meta-data data to inform BO, The training cycle then repeats.
Figure 1:
One method that dynamically optimizes hyperparameters PB2 Sarker-Holdes al., 2020a).. which op- timizes the hyperparameters and trains the agents in parallel, taking only the wall clock time of a single training run (but rumning multiple machines). PB2 creates the dynamic hyperparameter schedule by changing the model weights and hyperparameters of poorly performing agents at fixed intervals. The new model weights are copied from randomly selected well-performing agent, and the hyperparameters are adjusted using Bayesian Optimization (arnett,, 2023 Snock et al. 2012) as visualized in Figure [ A shortcoming PB2 that the hyperparameter choices are relatively uninformed during the initial train- ing stages, especially for settings with a small number of examples. We aim address this by utilizing information from past runs through meta-learning.
We adopted comprehensive approach to gain overview of the performance of various methods for the novel application of meta-learning for Bayesian Optimization in time-varying setting. After discussing background (Section 2 and related work (Section 3), we make the following contributions:
We propose four different meta-learning methods for the time-varying setting of PB2 (Section 4 In our first approach, which we call TAFPB2, we apply meta-learning to the acquisition function, while for the second approach, we apply meta-learning the surrogate model using multi-tast GP and call it MultiTaskPB2. In MetaPriorPB2, our third approach, construct prior for the GP based on ensemble from past optimizations. Our fourth approach involves exploring static hyperparameter portfolios to warmstart optimization, aiming to accelerate performance.
We evaluate and compare the performance of our methods with each other and relevant HPO methods and show that MultiTaskPB2 outperforms the other proposed methods and baselines with respect to anytime and final performance (Section 5.2
We investigate a potential synergy of combining our methods by warmstarting TAFPB2, Mul- tiTaskPB2, and MetaPriorPR2. TAFPB2 shows significant. improvement, while MetaPriorPR2's
2

---

Published in Transactions on Machine Learning Research (03/2025)
performance degrades. For MultiTaskPB2, warmstarting has no substantial effect its final per- formance, where it remains one of the top models. (Section 5.4).
We investigate the behavior of our methods and perform ablations. that our models can identify similar tasks (Section [.3). Furthermore, we compare runtimes (Section 5.6h and find that the overhead introduced by our methods is large for simpler environments but becomes negligible for larger environments with longer training times. Additionally, perform an ablation that shows the significant impact of the amount of training the warmstarting portfolio is optimized for. The ablation supports our theoretically motivated default value (Section 5.5
First, we qualitatively confirm
Among our approaches, Mult/TaskPB2, which focuses the surrogate model, stands out the most robust and effective among them. It consistently delivers top performance in both anytime and final performance metries in our experiments and surpasses our baselines.
2
Background
Our method builds upon PB2 Parker-Holder et al. 2020a), which extends Population-Based Training (PBT) JJadeberg et al. 2017). We provide detailed descriptions of both in this section.
Population Based Training: PBT is an evolutionary algorithm that dynamically optimizes hyperpa- rameters by training a population of deep neural network models in parallel. PBT randomly initializes the models and their hyperparameters each which is independently trained by an asynchronous worker. PBT periodically performs exploit and explore steps. We dub the period as the perturbation interval. In the exploit step, the hyperparameters of the top-performing fraction (typically top 20%) of the models are copied over to the poor-performing fraction. Then in the explore step, the hyperparameters of these newly copied models are either perturbed resampled.
Population-based Bandits (PB2): PBT can demand large populations for effective performance, partic- ularly due to its exploratory nature. To mitigate this, PB2 extends PBT by integrating Bayesian Optimiza- tion (BO) (arnettt 2023; Snoek et al. 2012) and Gaussian Processes (GP) [Rasmussen & Williams 2006) for exploration. This adaptation enhances performance efficiency with fewer workers and offers convergence guarantees.
We provide the details on the regular BO and GPs in Appendix A.1 and review the time-varying GP and the selection mechanism used in PB2 in the following.
Using BO in the parallel and time-dependent sctting of PBT requires some adjustment of the standard BO components described in Appendix A.1. The observations that make up the dataset D now additionally contain a time component t. E N that specifies at what perturbation interval the observation was made, and the performance E at the start of the last perturbation interval, i.e., 7,), y) E D. Furthermore, the targets y E now encode the performance change during the last perturbation interval
For the GP surrogate model, PB2 models the similarity between input pairs (41, m1)1), 22,2)) with separable time-varying kernel (Bogunovic et al. 2016)
krv ((1,, n, 21), (12, 72. x2)) krsmm kas ((n an), (1),((2223)))
where KTime (41,42) (1,me(41122) = (1 - w) للعينا is a time kernel with a parameter w E (0,1) that controls how related observations different points in time are. It results in posterior distribution given by
,,t, 7, = krv 7, ( + w/I) -/I
(14,, = ((1, 7, x). (z)) - krv(t, 7, 7,2x) (KTV + 7I1) kTV
by is not part of the original formulation of PB2, but used in the Ray Liaw use our experiments. Adding allows the surrogate model to capture that models with different initial performances different performance increase when having the same hyperparameters.
2018 implementation of PB2, which
3

---

Published in Transactions on Machine Learning Research (03/2025)
where (rv 7: 2), and Krv are induced from the time-varying kernel following Eq. (2) from Appendix A.1
During the explore step, PB2 copies the model (including the neural network weights) of a better performing worker at timestep & with initial performance 7. Then, PB2 performs BO and selects a new configuration
x² by maximizing GP-UCB (Srinivas et al. 2010
GP-UCB (1. D) u((,7,,) + (B,
over x while keeping I and 5 fixed. The pseudo-code for the PB2 algorithm is provided in Appendix A.2
3 Related Work
Our approach builds on population- based approaches and meta-learning approaches. In this section, we present related work under these two categories.
Population-based approaches draw inspiration from Evolutionary Algorithmss (EAs) Bäck 1996.. PBT (Jaderberg al. 2017) has emerged as a prominent method, achieving state-of-the-art results both su- pervised and reinforcement learning (RL). However, the dynamic and evolving nature of RL landscapes has led to the proliferation of PBT variants tailored specifically for RL. PB2 (Parker-Holder et al., 2020a) inte- grates Bayesian Optimization (BO) into PBT, enhancing its adaptability and performance. BG-PBT Waan et al. 2022) extends this concept to include Neural Architecture Search (NAS), enabling simultaneous evo- lution of architectures and hyperparameters. Other variants like PBT for olf-policy RL (Franke et al. 2021) and backtracking-enaar enabled PBT (Zhang et al. 2021) demonstrate improvements in sample efficiency and adaptation to performance fluctuations. Recent developments have focused on addressing the limitations of traditional PBT approaches. Zhang et al. (2021) introduce a PBT variant that allows PBT to backtrack to previously better performing configurations because the performance of RL agents can fall drastically during optimization. Bai & Cheng (2024) and Dalibard & Jaderberg (2021) both try to avoid PBT's greediness by judging population members' fitness based on the rate of improvement rather than the absolute performance. Flajolet et al.] (2022) show that PBT can even be implemented on a single machine without. the need for expensive parallel training on multiple machines. ei al. (2019) and Bai & Cheng (2024) both propose gen- eralized versions of PBT. Parker-Holder et al. (2020b) optimize all the members of the population at once, aiming to adapt the diversity of the population to improve the reward-diversity trade-off. HOOF (Paul
et al. 2019) adapts hyperparameters the fly for single policy gradient agent based an improvement metric for policy gradients to combat PBT's computational expensiveness. These advancements underscore methods in enhancing RL performance across various domains, from algorithmic efficiency to architectural adaptation and beyond. In our approach, we build on top of PB2 because of its efficioncy, and because it uses BO. BO builds a model that can be employed for meta-learning in a variety of ways. Therefore, PB2 lends itself better to exploring meta-learning extensions than other model-free population- based approaches.
the dynamic evolution and broad applicability of population-
Meta-learning approaches (Vanschoren 2019 can be combined with HPO methods to, c.g., transfer good hyperparameter settings across similar problem instances. One prominent research direction in meta- learning is to meta-learn the different components of Bayesian Optimization. Wistuba et al. (018 propose a transfer acquisition function (TAF) that combines the expected improvement on the target task with improvement estimates from similar meta-tasks. This similarity is computed based on meta-features. Feurer et al. 2022b) propose a method called RGPE to determine this similarity implicitly. We adjust the weighting and acquisition function to our setting and use them in our approaches. Other approaches focus on meta- learning the surrogate model. Of these approaches, the ones that use freeze-thaw BO (Swersky et al. 2014], form of fultiifidelity optimization, are most closely related to our time-varying setting. While freeze-thaw BO does not dynamically optimize hyperparancters it builds a surrogate model after partial training to determine which hyperparameter configurat ions are promising. Quick-Tune (Arango el al. 2024) meta-learns a deep kernel GP (Wilson et al.) 2016) based on DyHPO (Wistuba et al. 2022). CMBO (ee et al. 2024), on the other hand, uses meta-learning to train a Prior Fitted Network (uiller et al. 2022) surrogate model on real data. Both of these surrogate models need to be pretrained on large amounts of meta-data, which is why
our surrogate model approach uses more classical Multi-Task GP (Swersky et al. 2013). In contrast to these
4

---

Published in Transactions on Machine Learning Research (03/2025)
approaches, portfolio-based approaches focus on initializing HPO with set of configurations that performed well on similar tasks (Feurer et al. 201 2015a| by. To avoid the computation of meta-features needed to determine similarity, Feurer et al. (2022a) propose to construct a single portfolio that works well across all tasks. This lack of overhead made us build our warmstarting approach on top of Feurer et al. 2022a 's work. Model- Agnostic Meta-Tearning (MAML) (Finn et al., 2017) is a prominent mela-learning technique that differs from the previous approaches by not transferring hyperparameter configurations. Instead, learns model initializations that can be quickly adapted to new problem instances within a few gradient updates. [Banejeee et al. (023)) apply MAML to model-basee RL agents that act principals and incentivize other agents to act in preferred ways.
4 Methods
Meta-learning plays crucial role in enhancing machine learning performance with limited samples. We aim to improve this line of work in PB2 by presenting four meta-learning extensions.
4.1 TAFPB2: Transferring Acquisition Function and Weighting
In our first approach, we consider meta-learning via weighting the acquisition function across related tasks. In particular, we incorporate TAF (Wistuba et al. 2018) with a weighte Gaussian Process ensemble (RGPE) (Feurer et al. 2022b) as the acquisition function for PB2, which we refer to as TAFPB2. Since the original formulation of TAF considers the static setting, we extend it to handle the time-varying setting of PB2 by incorporating the additional inputs given by the interval step i and the initial performance 5. Our formulation treats observations similarly the static case without introducing additional complexity, This results in the development of the tirc-varying TAF function that we present follows:
(aF (,4r1) = w +,,+ F, + Σ w. max mnax(0.11(6 (0,14(1,7:2)
max (BA.))DD441
where i2 are normalized weights (defined subsequently) for the meta-tasks i i= I,...,7 and the target task + We use the El acquisition function (Močkus 1974) with the time-varying component, described in Appendix A.3 the main component for TAF.
We establish a weighting mechanism between a target task and multiple meta-tasks, which we then combine with TAF, as demonstrated in Feurer et al. (2022b). These weights are determined by how accurately surrogate models, derived from the meta-tasks and the target task, characterize the observed configurations. In particular, weights are modeled as the probability that a model fits the target task best based on their ranking loss. This probability is estimated using Monte Carlo estimation with 5 N bootstrap datasets D that are sampled with replacement from D7+1. The loss of each model 1,2,...,7 T each bootstrapped dataset ..2...., S is denoted by 4, This results in weighting
1 (i arg min, (..s)
wy = Σ
(1)
Σ 1j E arg min, 4.,w))
The full procedure to compute the weights can be seen in Section A.4
4.2
MultiTaskPB2: Multi-Task Gaussian Process
For our second meta-learning approach, employ multi-task Gaussian Process (GP) Alvarez et al, 2012). This approach establishes unified model across multiple meta-tasks, enabling observations from different tasks to influence predictions on other tasks based on inferred task similarities.
Our time-varying setting includes multiple meta-tasks, only some which are relevant to the target task. Therefore, we only select from the top h most similar meta-tasks based on the adjusted RGPE weighting from Eq. [.. In addition, we collect only observations from the meta-tasks that lie within window around the time step of interest & since they are more relevant. to the considered time step. Next, we infer
5

---

Published in Transactions on Machine Learning Research (03/2025)
task similarity, which determines how strongly different tasks affect each other within the multi-task GP framework.
Using multi-task GP, we define the dataset D consisting of entries ((m, 7, 2, y) where m E {1, ..., r+1] indicates the task index from which the observation originated. We follow the Linear Model Coregional- ization (LMC) (Journel Huijbregts 1976 Bonilla al.) 2007) define the kernel. This means that our kernel consists of sum of separable kernels as follows
T+1 Σ
kLMC([mI, 21. [m2, 22))
ke km m2 m2)
where := 2t for brevity, the kry are independent kernels with possibly different parameters with 1T+1 coregionalization kernels (m,, m2) = Cm1.m2,4 where the coregionalization matrices (mm.ma:dmmi.mm are symmetric and positive definite. The entries of these coregionalization matrices parameterized and estimated with maximum likelihood resulting in the posterior given by
p(m, z) = kLMM(m, (KIMC + (KKim, o² (m 2) kLMO((m, 2), (m, 2)) KLMC (2)² (KLMC + "m
where kf, y, x), and KLMC are induced from KLMC following Eq. 2 Since we are only interested in predictions the target task, fix
when using it in the acquisition function.
4.3 MetaPriorPB2: Meta-Prior to the Gaussian Process
We propose another variant the meta model using ensemble of the mean predictions μ4 of the meta- GP's the prior mean of the target process fr+1 (instead of using zero mean prior)
Algorithm 1 Portfolio Construction
Input: Datasets D1, 2.,,.., D, portfolio size N Output: Portfolio P of size
I: C C-o 0
b Create portfolio candidates
2: for i= L,.,..,, do
fr+1 ~ GP(m,
where m WiPs
3: I² + hpo(DD)
4: CU
where the weights w, are given by the RGPE in Eq. ① while excluding the target task. In this time- varying setting, the GP posterior distribution (Bo-
5: end for
6: for i 1,,, do
Create and normalize
performance matrix
gunovic et al. 2016) is parameterized by
7:
for x EC do
Pa.s & evaluate(r, D,) 9: end for
8:
μ(=) (Tv(=)"((KK))))) (KTV + +1il1= "y -m) + m(=) o² k2v(22 krv (KTV + - kT)(2),
10: p.d + (p. min ... (max/,,appp. - min p..))
11: end for
P +- 0 Greedily add configurations to portfolio
where [t, x] and m This meta-prior elegantly integrates previous time- varying surrogate models, which are estimated from meta-lasks, according to their relevance for solving the target task.
while |P] < do
14:
x² arg maxxecipppp ET max Paçi P P-PU
15:
16:
end while
4.4 Warm-starting the Population with Portfolio
We consider more classical, but effective, meta-learning technique known as warm-starting, This approach is inspired by the portfolio construction method used in Auto-sklearn 2.0 FFuurer et al. 2022a).
In Algorithm 1 we create a portfolio of hyperparameter configurations which we use initialize the training. First, we construct candidate set of possible portfolio members by searching for the optimal hyperparam- eter configurations all meta-datasets with hpo method, such using random search. Then, we evaluate

---

Published in Transactions on Machine Learning Research (03/2025)
each candidate configuration on all datasets to create performance matrix [Pa,]E,5=1 (line 8). Finally, we build the portfolio by iteratively adding the configuration of the candidate set that maximizes the average best performance of the current portfolio if it has been added to it.
Unlike traditional static HPO approaches to portfolio construction, which typically consider final performance after full training, our method constructs portfolios based on scores at the time step when configurations are first anticipated to change under PB2. Taking the percentage À as the probability that configuration is changed after perturbation interval, the probability that initial configuration changes for the first time after interval i is geometrically distributed with P(i) X(1 X)- Therefore, the expected interval when given initial configuration is changed is the expectation of the geometric distribution
[X]
X
In Section 5.5 we perform an ablation to evaluate the impact of X on algorithm performance.
5
Experiments
This section begins with details experimental setup, followed by the presentation of our rescarch findings that address the following research questions.
RQ1 How do our methods compare in performance standard baselines and each other?
RQ2 Do the RGPE weights corroctly identify similar tasks?
RQ3 How does the warmstarting method influence the performance of other meta-methods?
RQ4 Does the theoretical recommendation for perturbation interval for portfolio construction hold true empirically?
RQ5 How much runtime overhead is introduced by our methods?
5.1
Experimental Setup
We follow the evaluation scheme of the PB2 paper (Parker-Holder et al. 2020a for our experiments and optimize the hyperparameters of Proximal Policy Optimization (PPO) SSchulaan et al. 2017). We now introduce the benchmark enviromments used for meta-learning and elaborate on our evaluation strategies. Appendix B.1 provides a more detailed overview of our experimental setup.
Benchmark environments We evaluated our approaches on two sets of RL environments, employing CARL (Benjamins et al., 2023) to generate several slightly different versions of each. This approach mimics blend of comparable and distinct tasks, enabling to explore the specific tasks methods effectively learn from. The first set of enviromments is classic control (Brockman et al. 2016) specifically mountain car, carl. pole, pendulum, acrobol. We used this cheaper environment to set our methods' hyper-hyperparameli and conduct more compute-intensive experiments. The second set of environments is Brax (Freeman et al. 2021) where CARL allowed us to modify 9 environments (sce Appendix B.1 J. In both sets of environments, we manipulated the gravily across versions to reflect that of five different. planets.
Modification to Ray Implementation We made minor modifications to the Ray implementation (Liaw et al.) 2018) of PB2 and empirically found it to perform better than the original version. This altered version, referred to PB2*, serves as the foundation for our approaches. The key modification that had the most significant impact involves how we manage poorly performing workers after the initial training perturbation period. During this phase, we have performance data from various workers but lack the performance differences necessary for meaningful Bayesian Optimization (BO) steps. Ray's default behavior in such cases to just. exploit, resulting duplicate workers with identical weights and hyperparameters. To address this, we additionally sample entirely new hyperparameter configurations. This approach allows to explore a broader section of the search space and provide better guidance for subsequent BO steps.
7

---

Published in Transactions on Machine Learning Research (03/2025)
3.8
[
Normalized score
Rank (aut of 8)
random search
0.-
PBT
0.4
P82* warm
PB2*
88-
TAFP82
MetaPriorP62
MultiTaskP2
10
15
15
Number iterations
Number of iterations
Brax
cd
Normalized score
Rank (out of 8)
random search
2.6-
MotaPriorPB2
PB2*
P82*
MultiTaskP22
02-
2-
16
15
10
15
Number iterations
Number iterations
Classic control
Figure Method comparison with baselines where PPO was trained on Brax (top) and classic control (bottom). Comparing rewards normalized at each iteration (higher is better) the left and ranking the right (lower is better). Out of all of our methods and baselines, MultiTaskPB2 consistently performs the best. Note that the different values at the first interval for Brax are caused by crashed experiments (see Appendix B.6 These crashes were caused by divergence and are not fixable without changing the hyperparameter ranges.
Evaluation We repeat each of our experiments for 10 seeds and ensure that the initial random config- urations are the same for each method given the same seed. Our population-based algorithms split the training into 15 equally sized perturbation intervals. At each of the perturbation intervals, we evaluate every agent for episodes during training. When evaluating a method, we only look at the performance of the best agent at each timestep and compare these in multiple ways. The first way we compare our methods is by looking at normalized rewards. For each environment variant, normalize the mean rewards of the different methods at each timestep to be between 0 and 1. The second way compare our methods is by ranking the mean rewards instcad of normalizing them. When visualizing the normalized rewards and the rankings, we plot their mean and standard error during the training process. We further pair the rank visualization with a critical difference diagram (Demšar 2006) using the Autorank (Herbold, 2020) Python package to summarize the any-time performance. The diagram displays the mean rank over all timesteps for the methods on vertical line on the right and vertically connects the lines that indicate the method names if there is no statistically significant difference (a 0.05) between their mean ranks.

---

Published in Transactions on Machine Learning Research (03/2025)
5.2 RQ1: Methods Comparison
In this section, we address research question RQ1, examining the performance of the proposed methods in comparison to each other and standard bascline methods. Specifically, we compare the novel meta-lcarning approaches outlined in Sections 4.1
4.4 with: PBT (Jaderberg et al. 2017), PB2 (Parker-Holder et al. 2020a), our improved PB2 variant which we dub PB2*, ; and random search. We use Ray (Liaw et al.] (0018) implementations of the baselines since our methods are implemented on top of Ray. The results of these experiments can be seen in Figure 2 while Appendix B.5 shows additional results for different numbers of workers.
Our experiments show that MultiTasPPB2 overall performs the best over both metrics on both environment families. Additionally, our meta-methods demonstrate statistically significant improvement in anytime performance over PB2. This enhancement in anytime performance appears to primarily stem from the modifications introduced by PB2*. Still, MultiTaskPB2, in particular, was able to use the meta-data to further improve its final performance. In contrast, our other meta-methods do not perform as well. No- tably, TAFPB2 exhibits the lowest final performance in classic control environments. While warmstarting enhances initial performance in classic control, it does not substantially impact final performance and shows minimal effect on Brax. Further analysis in Appendix [35 reveals that a static portfolio fails to provide robust hyperparameter initialization for wide array of different environments like the ones present in Brax. Individual training plots and raw results can be found in Appendix B. and Appendix B.7 respectively.
5.3
RQ2: Similarity Weighting
The methods introduced in Sections 4.1
4.3 namely TAFPB2, MetaPriorPR2, and MultiTaskPP22, use RGPE weights. These weights measure the similarity between the meta-tasks and the target task. In this section, explore the capability of RGPE to identify similar environments. We know which environments are similar to each other due to the way they are constructed. Environments with the same base environment but different gravity should behave more similarly than environments with completely different base envi- ronments. For our visualization, we average each meta-task weight for a given target task over all timesteps and seeds and display the results in matrix form.
Since we identify MultiTaskPB2 as the most promising of our approaches Section 5.2 we will restrict this analysis to MultTaaskPB2. The visualizations for TAFPB2 and MetaPriorPB2 on both Brax and classic control align with the results in this section and can be found in Appendix B.4
The weight matrix for MultiTaskPB2 on Brax is shown in Figure B It has diagonal block structure with varying block sizes. The smaller blocks are made up of environments that share the same base environment but have different, gravity. On average, 34 % of the weight is allocated to them. This is more than three times the 9.5 % of the weight that the average set of environments that uses a different base environment gets allocated. If we look at the base environments of the bigger blocks, we can see that they are related like in the case of humanoid and humanoid slandup (17 % of the total weight) or inverled pendulum and inverted double _pedduuun (20 % of the total weight). In summary, the weights not only identify similarities based on the same base environment but also based related base environments.
5.4 RQ3: Warmstarting the Weighting-baser Meta Methods
Our warmstarting approach and other meta-learning methods are orthogonal and can be combined. In this section, we address RQ3 and analyse how this combination affects performance. For this, we trained MultiTaskPB22, TAFPB2, and MetaPriorPB2 with warmstarted hyperparameters and visualize both the normal and warmstarted training in Figure
We can see that warmstarting improves the initial performance in both environment families, though only marginally on Brax. However, MetaPriorPB2 was negatively affected, with its final performance degrading compared to the vanilla version. One explanation for this could be that the mcta-prior performs poorly because hyperparameter schedules in the meta-data do not match the schodules we encounter when warm- starting our meta-prior. In contrast. to MetaPriorPB2ss decreased performance, TAFPB2 shows a significant
9

---

Published in Transactions on Machine Learning Research (03/2025)
10
balfcheetah earth
halfcheetah neptune
nalrcheetan Saturn
halfcheelah uranus
halfcheetah venus
hopper neptune
hopper earth
hopper uranus
hopper venus
humanoid earth
humanoid neptune
humanoid saturn
humanoid venus
humanoid standup earth
humanold standup nεptune
humanofd standup Satur
humanoid "standup uranus
Target task
in inverted double Fendulum
venus
Weighting
inverted double pendulum
Inverted do pendulum saturn
-00
inverted double pendulum uranus
inverted double penduluu venus
inverted pendulum earth
inverted péndulum neptune
inverted pendulum Betrted_eeddulluul uranus
inverted pendulum venus
pusher earth
pusher neptune
pusher saturn
pusher venus
reacher earth
reacher neptune
reacher saturn
reacher uranus
venus
walker2d
walker24
wa 2d Saturn
walker2d uranus
walker2d venus
halfcheeta earth
neptung
ha salfchestal saturn
hallcheed heheehhhh uranus
/ venus
halfe cetare venus hoppan
hoppel saturn
hopper uranus
humanooo venus
numancia sarth
heptume
humanoo snmaniid wenus humanold
uranus
human oid St St 63rth
commann standup ne reptune comaan
saturo
stand vemun Inverted dduble double penduli rendulum
un earth
pen um neptuse double pendul
Saturn
Inverted p -m uranus
Inverted 'double bendil bendull um venus
sarta
inverted bendulumm uranus
Inverted peeertddeeedddd pendu uum Saturn
_eep_ppee___ee pen um venus
pusher earth
resptune
Saaghee saturn
Qushhrr venus
pracher earth
reacher ne eettunn reachel e saturn
reacher uranus
eeddeaatt earth
walker2g neptune
walket walker2d saturn
walker 2d uranndaaaa uranus
wa 2alker2d venus
he helfcheetan aeaaheeppei
chacher
hopper
stant
douple pendiu
pusher
hop
humanool
inverted
inverted
inverted
inverted
inverted
Meta-task
Figure 3: Average RGPE weights of MultiTaskPB22 for each task and all meta-tasks BRAX. The weights form diagonal block structure. Environments constructed from the same base environments with different gravity are identified as similar. We also see higher weights between different base environments that are related, for example, humanoid and fumanoid standup.
performance improvement. On the other hand, warmstarting had no substantial effect on MultiTaskPP2'' final performance while slightly improving anytime performance in a statistically insignificant way. Overall, outside of TAFPB2, we did not observe a strong positive impact on the final performance. Given that there is no notable difference between the warmstarted TAFPB2 and both MultiTaskPB2 versions, we recommend MultiTaskPB2 to avoid additional complexity.
5.5 RQ4: Portfolio Training Interval
In this section, we investigate the empirical validity of our theoretical recommendation for the number of perturbation intervals train configurations for during portfolio construction, outlined in Section 4.4
10

---

Published in Transactions on Machine Learning Research (03/2025)
4.5
Normalized score
4.0
Rank (out
2
MetaPriorPB2 warm
TAFPB2
MctaPriorF22
3.5
MultTTsskP2
MultTaskP92
TAFP82
10
15
Num ber of iterations
Number of iterations
Brax
2.8-
TAPP82
Rank (outt ()
06-
4.0-
Noma lize
MetaPriorPB2
MetaPriorP22 warm
3.5
644
TAPP62 warm
3.0
MuBITask
82-
10
Number iterations
Number iterations
Classic control
Figure 4: Warmstarting the meta-methods for PPO training on Brax (top) and classic control (bottom). Comparing rewards normalized at each iteration (higher is better) on the left and ranks on the right (lower is better). Warnstarted methods have betier initial performance, but this effect is smaller the more diverse Brax environments. Warmstarted and normal MultitaskPB2 perform the best overall.
We perform an ablation study comparing the performance of PB2* warmstarted with portfolios that were constructed using different numbers of perturbation intervals. We maintain the Ray default value for the fraction of copied workers, À = 0.25. Following our thcoretical argumentation from Scction 4.4 we train for perturbation intervals. We compare this training time with the original full training for 15 intervals and 1, 6, and 8 intervals.
Considering the computational demands of the portfolio generation process, conducted this experiment classic control tasks. The warmstarting was repeated over 10 seeds, and the configuration performance was averaged over 10 seeds during portfolio construction to ensure robustness.
As shown in Figure 5 PB2* warmstarted with the portfolio following our recommendations outperforms other variants. From the third perturbation interval onwards, it achieves the best normalized score and rank, with statistically significant better anytime performance compared all alternatives.
5.6 RQ5: Runtime Comparison
While our methods improve upon PB2, they also introduce an additional computational overhead. In this section, we take a closer look the runtime of the methods we compare in Section 5.2 We ensured efficient training of the methods by matching the number of used CPU cores the number of workers. For classic control experiments, used machines with Intel Xeon Gold 6242 processors at 2.80 GHz, whereas for Brax,
11

---

Published in Transactions on Machine Learning Research (03/2025)
Normalized score
Rank (out of 5)
3.5
PB2*
Paz* warm ft=1
PB2* itw8
2.0
PB2 Itw6
2.5
warm (ours)
2.0
10
15
10
Number of iterations
Number of iterations
Figure 5: Comparing the performance of warmstarting PB2* for PPO training with different portfolios where configuration performance was calculated at different number of perturbation intervals. Rewards that are normalized at each iteration (higher better) are on the left and ranks are on the right (lower is better), results are on classic control. Warmstarting PB2* with the portfolio based on our theoretically motivated recommendation performs best.
4400
20
Perce tagg dfforence from POZZ median
15
dfferuns foom PEC meddan
10
3900
10
'3995
26071
8600
25914
25854
26154
28878
26168
20163
0
"3481
3406
3464
75tal
S19E
3030
2000
-20
MatsPin/PB2
MuRTaauPP2
PB2
PRu wnam
PBT
1AFPB2
randonn Bearch
MelpPtsPP2
2
PB2
wrem
PRizt
PeT
TAFPE2
mediin Setrch
Algorthm
Algodthm
(a) Classic control
(b) Brax
Figure 6: Runtime for training PPO with different HPO methods on classic control and Brax. The overhead introduced by our methods becomes negligible for more complex environments with longer training times such Brax.
we utilized machines with Intel Xeon E5-2630v4 processors at 2.2 GHz. For every method, we calculate the average runtime on each environment and visualize them as a boxplot in Figure 0 We can see that MultiTaskPB2, our pess-rerformig method, introduces the biggest overhead and has the longest runtime. On classic control, the median runtime difference between MultiTaskPR2 and PB2 was 730 seconds, which amounts relative difference of 21%. On Brax the mean runtime difference is similar with 757 seconds, but the relative difference only amounts to 2.9%. The overhead of our methods when learning from up to 44 meta tasks where each task consists of the schedules of combined 40 workers is roughly 12 minutes and becomes negligible when training in harder enviromments that might take multiple days to train. However, it is important to nole that smaller organizations with limited computational resources may not benefit from our methods. Such organizations often cannot afford to run experiments on larger environments where the overhead becomes negligible. For them, the additional computational cost may outweigh the performance improvements, making our methods less practical for their specific needs. Thus, while our approach offers advantages in large-scale settings, may not be suitable for smaller-scale experiments.
12

---

Published in Transactions on Machine Learning Research (03/2025)
6 Conclusion
In this paper, we introduce new meta-learning approaches tailored for dynamic hyperparameter optimiza- tion in reinforcement learning (RL), aimed at enhancing the Population-Bas Bastd Bandits (PB2) algorithm. PB2, leveraging time-varying Gaussian processes (GP), demonstrates strong overall performance but. faces challenges when starting the optimization due to its GPs lacking information at the outset. To address this, we introduced four novel meta-learning methods for PB2 in time-varying environments. Our approaches in- clude TAFPB2, which applies meta-learning to the acquisition function; MultiTaskPB2,, employing multi-task GPs to enhance the surrogate model; MetaPriorPR2, utilizing GP prior based on historical ensemble opti- mizations; and static hyperparameter portfolios for accelerated initialization. Among these, MultiTaskPB2 emerged as particularly promising, surpassing all other hods in both anytime and final performance met- rics across two distinct RL environment families. Our contributions are notable for adapting meta-learning strategics suit the dynamic nature of RL hyperparameter optimization.
While our findings show promise, our methods are constrained by two limitations. They depend on the availability of pertinent meta-data aligned with the hyperparameter search spaces and currently support only continuous hyperparameters. Morcover, implementing our methods entails a consistent computational overhead, which may hinder their suitability for environments with limited training durations. This dis- proportionally affects smaller organizations with limited computational resources that can not train on complex environments. Looking forward, future research could extend our methods to accommodate cate- gorical hyperparamcters, leveraging approaches similar to those explored by Parker-Holdee Holder et al. (2021) for PB2. Exploring more data-intensive meta-learning techniques, such as deep kernels {Araggo et al. 2024)
prior-data fitted networks (PFNs) (Lee et al. 2024), might further enhance performance. Additionally, inte- grating mela-features for configuration selection during warmstarting and adapting methods to domains like Deep Learning, where hyperparameters tend to be static, represent promising avenues for exploration.
Acknowledgments
We ackuowledge funding by the Deutsche Forschuugsgemmiinscaaftt (DFG, German Research Foundation) un- der SFB 1597 (SmallData), grant number 499552394. The authors acknowledge funding through the research network "Responsive and Scalable Learning for Robots Assisting Humans" (ReScaLe) of the University of Freiburg. The ReScaLe project is funded by the Carl Zeiss Foundation. The authors acknowledge support by the state of Baden-Württemmbeee Württemberg through bwHPC and the German Research Foundation (DFG) through grant no INST 39/963-1 FUGG (bwForCluster NEMO). The authors ackuowledge the financial support of the Hector Foundation.
References
Mauricio A. Álvarez, Lorenzo Rosasco, and Neil D. Lawrence. Kernels for vector-valued functions: A review. Foundations and Trendso in Machine Learning, 4(3):195 266, 2012.
Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Józefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schncider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojciech Zaremba. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1), 2020.
Sebastian Pineda Arango, Fabio Ferreira, Arlind Kadra, Frank Hutter, and Josif Grabocka. Quick-tune: Quickly learning which pretrained model to finetune and how. In International Conference on Learning Representations (ICLR (24, 2024.
Thomas Bäck. Evolutionary algorithms in theory and practice: evolution strategies, evolutionary program- ming, genetic algorithms. Oxford University Press, 1996.
Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhao- han Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In Pro- ceedings of the 37th International Conference on Machine Learning (ICML 200, volume 98, pp. 507-517. PMI.R, 2020.
13

---

Published in Transactions on Machine Learning Research (03/2025)
Hui Bai and Ran Cheng. Generalized population-based training for hyperparameter optimization in rein- forcement learning. IEEE Trensactions Emerging Topics in Compulational Intelligence, 2024.
Arundhati Banerjee, Soham Rajesh Phade, Stefano Ermon, and Stephan Zheng. MERMAIDE: Learning to align learners using model-based meta-learning. Transactions on Machine Learning Research, 2023.
Carolin Benjamins, Theresa Eimer, Frederik Schubert, Aditya Mohan, Scbastian Döhler, André Biedenkapp, Bodo Rosenhahn, Frank Hutter, and Marius Lindauer. Contextualize me the case for context in rein- forcement learning. Transactions Machine Learning Research, 2023.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Πya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota with large scale deep reinforcement learning. crXiv:1912.06660 [cs.LG], 2019.
Mija Bogunovic, Jonathan Scarlett, and Volkan Cevher. Fimc-iarying gaussian process bandit optimization. In Artificial Intelligence and Stalistics, volume 51, pp. 314-323. PMLR, 2016.
Edwin V Bonilla, Kian Chai, and Christopher Williams. Multi-Lask gaussian process prediction. In Pro- ceedings of the 20th International Conference on Advances in Neural Information Processing Systems (NewrIPS:07). Curran Associates, 2007.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv:1606.00 01540 [cs.LG), 2016.
Valentin Dalibard and Max Jaderberg. Faster improvement rate population based training, ar 2ix::2099.133000 NEs. 2021.
Janez Demsar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7:1 30, 2006.
Theresa Eimer, Marius Lindauer, and Roberta Raileanu. Hyperparameters in reinforcement learning and how to tune them. In Proceedings of the 40th International Conference Machine Learning (ICML '23), volume 202, pp. 9104-9149. PMLR, 2023.
Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. Efficient and robust automated machine learning. In Proceedings of the 28th International Conference on Advances in Neural Informalion. Processing Systems (NeurIPS'15), pp. 2962 2970. Curran Associates, 2015a.
Matthias Feurer, Jost Springenberg, and Frank Hutter. Initializing bayesian hyperparameter optimization via meta-learning. In Proceedings of the Twenty-ninth AAAI Conference Artificial Intelligence (AAAI'15), pp. 1128 1128-1135. AAAI Press, 2015b.
Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer, and Frank Hutter. Auto-Sklearn 2.0: Hands-free automl via learning Journal of Machine Learning Research, 23(261):1 61, 2022a.
Matthias Feurer, Benjamin Letham, Frank Hutter, and Eytan Bakshy. Practical transfer learning for bayesian optimization. arXiv:1802.022 0221194 [stat.ML), 2022b.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning (ICML'IC) volume 70, pp. 1135. PMLR, 2017.
Arthur Flajolet, Claire Bizon Monroc, Karim Beguir, and Thomas Picrrot. Fast population-based rein- forcement learning on single machine. In Proceedings of the 39th International Conference on Machine Learning (TCML 22), volume 162, pp. 6533 6547. PMLR, 2022.
14

---

Published in Transactions on Machine Learning Research (03/2025)
Jörg K.H. Franke, Gregor Köhler, André Biedenkapp, and Frank Hutter. Sample-effciintt automated deep reinforcement learning. In Proceedings of the International Conference on Learning Represeniations (ICLR'21), 2021.
C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax a differentiable physics engine for large scale rigid body simulation. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks. Curran Associates, 2021.
Roman Garnett. Bayesian Oplimization. Cambridge University Press, 2023.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
Steffen Herbold. Aulorank: A python package for automated ranking of classifiers. Journal of Open Source Software, 5(48):2173, 2020.
Riashat, Islam, Peter Henderson, Maziar Gomrokchi, and Doina Precup. Reproducibility of benchmarked deep reinforcement learning tasks for continuous control. In JCML Workshop on Reproducibility in Machine Learning (RML 17), 2017.
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Population based training of neural networks. arXiv:1711.098.0 [cs.IG). 2017.
Andre G Journel and Charles J Huijbregts. Mining geostatistics. Academic Press, 1976.
Dmitry Kalashnikov, Alex Trpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. Scalable deep reinforcement learning for vision-bol bason robotic manipulation. In Proceedings of the Second Annual Conference on Robot PMLR, 2018.
Learning, pp.
Dong Bok Lee, Acxuan Silvia Zhang, Byungjoo Kim, Junhyeon Park, Juho Lee, Sung Ju Hwang, and Hae Beom Lee. Cost-sensitive multi-fidelity bayesian optimization with transfer of learning curve extrap- olation. In Third International Conference on Automated Machine Learning Workshop Track, 2024.
Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47))eabc5986, 2020.
Ang Li, Ola Spyra, Sagi Perel, Valentin Dalibard, Max Jaderberg, Chenjie Gu, David Budden, Tim Harley, and Pramod Gupta. A generalized framework for population based training. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD'19) 19), pp. 1791 1799. ACM Press, 2019.
Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and Ion Stoica. Tune: A research platform for distributed model selection and training. 1001 18 [cs.L.G), 2018.
Jonas Močkus, On bayesian methods for seeking the extremum. In Optimization Techniques IFIP Technical Conference 1974, pp. 400-404. Springer, 1974.
Aditya Mohan, Carolin Benjamins, Konrad Wienecke, Alexander Dockhorn, and Marius Lindauer. Autorl hyperparameter landscapes. In Proceedings of the Second International Conference on Automated Machine Learning, pp. 13/1- 1121. PMLR, 2023.
Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transformers can do Bayesian inference. Tn Proceedings of the International Conference on. Learning Representations (ICLR'28) 22), 2022.
Jack Holder, Vu Nguyen, and Stephen J Roberts. Provably efficient online hyperparameter optimiza- tion with population- based bandits. In Proceedings of the 33th International Conference on Advances in Neural Information Processing Systems (NeurIPS'20) 200), pp. 17200 17211. Curran Associates, 2020a.
15

---

Published in Transactions on Machine Learning Research (03/2025)
Jack Parker-Holed Holder, Aldo Pacchiano, Krzysztof M Choromanski, and Stephen J Roberts. Effective diversity in population based reinforcement learning. In Proceedings of the 33th International Conference on Advances in Neural Information Processing Systems (NeurIPS'20) pp. 189500-10000 Curran Associates, 2020b.
Jack Parker-Holder, Vu Nguyen, Shaan Desai, and Stephen J Roberts. Tuning mixed input hyperparameters the fly for efficient population based autorl. In Proceedings of the 34th International Conference Advances in Neural Information Processing Systems (NeurIPS'21) pp. 15513-15522 Curran Associates,
2021.
Jack Parker-Holder, Raghu Rajan, Xingyou Song, André Biedenkapp, Yingjie Miao, Theresa Fimer, Baohe Zhang, Vu Nguyen, Roberto Calandra, Aleksandra Faust, Frank Hutter, and Marius Lindauer. Automated reinforcement learning (AutoRL): A survey and open problems. Journal of Artificial Intelligence Research (JAIR), 74:517-568, 2022.
Supratik Paul, Vitaly Kurin, and Shimon Whiteson, Fast efficient hyperparameter tuning for policy gradients. In NeurIPS Workshop on Reinforcement Learning for Real Life Workshop
2019.
Carl Edward Rasmussen and Christopher KI Williams. Gaussion Processes for Machine Learning. The MIT Press, 2006.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. arXiv:1707.06347 [cs.LG], 2017.
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Πya Sutskever, Timothy P. Lillicapp, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529(7588):444 489, 2016.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical Bayesian optimization of machine learn- ing algorithms. In Proceedings of the 26th International Conference on Advances in Neural Information Processing Systems (NeurIPS'12) pp. 2960-2968. Curran Associates, 2012.
Niranjan Srinivas, Andreas Krause, Sham M Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In Proceedings of the 27th International Conference on Machine Learning (ICML (0), pp. 1015-1022. Omnipress, 2010.
Richard S Sutton and Andrew G Barlo. Reinforcement leurning: An introduction. MIT Press, 2018.
Kevin Swersky, Jasper Snoek, and Ryan P Adams. Multi-task bayesian optimization. In Proceedings of the 27th International Conference on Advances in Neural Information Processing Systems (NeurIS'13). Curran Associates, 2013.
Kevin Swersky, Jasper Snoek, and Ryan Prescott arXiv:1406.3896 Istats.ML) 2014.
Adams.
Freeze-thaw Bayesian optimization.
Joaquin Vanschoren. Meta-learning. F. Hutter, L. Kotthoff, and Vanschoren (eds.), Automated Machine Learning: Methods, Systems, Challenges, chapter 2, pp. 35 61. Springer, 2019.
Xingchen Wan, Cong Lu, Jack Parker-Holder Philip J Ball, Vu Nguyen, Binxin Ru, and Michael Osborne. Bayesian generational population-based training. In International Conference on Automated Machine Learning, pp. 14/1 27. PMLR, 2022.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning, In Pro- ceedings of the Nineteenth International Conference on Artificial Intelligence and Statistics (AISTATS'66), volume 51, pp. 370-378. PMLR, 2016.
Martin Wistuba, Nicolas Schilling, and Lars Schmidt- Thiemme. Scalable gaussian process-based transfer surrogates for hyperparameter optimization. Machine Learning, 107(1):43 78, 2018.
16

---

Published in Transactions on Machine Learning Research (03/2025)
Martin Wistuba, Arlind Kadra, and Josif Grabocka. Supervising the multi-fidelity race of hyperparameter configurations. In Proceedings of the 35th International Conference Advances in Neural Information Processing Systems 222,, pp. 13470 13484. Curran Associates, 2022.
Baohe Zhang, Raghu Rajan, Luis Pineda, Nathan Lambert, André Biedenkapp, Kurtland Chua, Frank Hutter, and Roberto Calandra. On the importance of hyperparameter optimization for model-bascd reinforcement learning. In Proceedings of the 24th Intervational Conference Artificial Intelligence and Statistics (AISTATS'21) volume 130, pp.
PMLR, 2021.
17

---

Published in Transactions on Machine Learning Research (03/2025)
A Additional Technical Details
A.1 Bayesian Optimization with GP
Bayesian optimization (BO) (Garnett 2023 Snoek et al. 2012) is popular sample-efficiet black-box opti- mization technique that has been shown to pulperform manual iyperparametlr optimization BO sequen- tially optimizes hyperparameters by constructing a probabilistic surrogate model from these observations to model the expensive target function. The probabilistic model is used to define an acquisition function that balances between exploration and exploitation for selecting the next hyperparameter configuration.
Gaussian Processes (GP) Rasmussen & Williams (006))
popular choice for the surrogate function. A GP is stochastic process consisting of set of random variables where every finite subset is jointly Gaussian. It is fully defined by its mean function m: - IR and positive definite covariance or kernel function k: X' x X - IR. If a function f follows GP, we write
f ~ gP(mm,,))
Given a input X = {a; X) with labels y = {y E R and a new point x E X, the posterior distribution fo at point is Gaussian
f. x x, N (n()),
with the predictive mean (xx) and covariance o² (x); defined in closed form, assuming zero mean for simplicity
u(x) k(x)² (K n11)
111n 'yy
(2)
d²(x) = k(x, x) k(x) (K + +1))~(((x))
where k(z) = [k(2,, x K = (x and 2 0 is a noise term accounting for uncertainty in the data. For a more complete introduction to Gaussian Processes, we recommend the classic textbook by Rasmussen Williams 2006).
A.2 Population Based Bandits (PB2) Algorithm
Algorithm 2 PB2
Input: Maximmm number of perturbation intervals E N, mumber workers W, fractions of workers that exploit and explore per iteration 0.5)
D +- 0, R -
Initialize all workers = 1,2, ..., W with model weights and hyperparameters wn and start training for
T' perturbation intervals.
3: while there is a worker training do
4:
if worker w reaches interval & then
5: e DU [(t, ((t,, Zwo) yw)}
b Update observed data
6:
if worker performance in bottom fraction then
Copy model from random top À fraction of workers timestop with initial performance 7 Search for configuration following Eq. (2)
end if
10: Update set of running configurations R
11: end if
12:
end while
A.3 The expected improvement acquisition function with time-varying component
We consider vane-varying version of Expected Improvement (Močkus) 1974 EI) which takes into account the time-varying index the additional input dimension. This time- vime-narying EI is used with the TAFPB2
18

---

Published in Transactions on Machine Learning Research (03/2025)
method described in Section 4
Ez, D) = (u(t, fr) ob (1))),
(1(1, -)t,7,2))
(1t, 7.0) ( + at,,,, (
o(t,7,z)
where f (aX(( y is the best observed performance improvement at a time step E.
A.4 Banking-Weighter GP Ensemble
In this section, we present the full RGPE weighting procedure from Section 4.1 To calculate the weighting, no meta-features describing the tasks are needed. The weights are based on how well a surrogate model f(z) ((( where = (t, æ), describes the observed configurations D on the target task. This is measured by a pairwise ranking loss between predicted and real performances
D(1,D++1)=
E (#:36): (2):0))ED++)
< (y: < vi)
where 1: false) + {0, 1}: (true + 1, false + 0} and + is the exclusive OR operator (XOR). This loss estimates how well the target model fits the target data leads overfitting since it would predict values for configurations it used for training. To prevent this, the Feurer et al. (2022b) use leave-one-out models (2) ~ (() (z)) that fit Dr+1 \ fZ). Using these models, the pairwise ranking loss for the target task is calculated as:
1 ( (2)) < 1s) (wi < vz))
C D(-1==
Σ (2.,8))
(4)
(ED,)
Using ranking loss over, for example, the mean squared error is preferable, since it better matches the goal of optimizing the hyperparameters. For oplimization, only the ordering of the hyperparameters is relevant. These ranking losses are used to calculate the weights as seen in Algorithm
Algorithm RGPE weighting
Input: Number of bootstrap samples S, observations on the target task Dr+1, meta-models St for 1,2,... ...,, T and the target model fr+1
Output: Model weighting w; for i = 1,2...., +1
1: for I, ,,...,, do
b Calculate losses
2: D < bootstrap(Dr+:)
3: for = TT
4. L (fi, 1++1)
Using Eq g
end for
4+1.8 L [ D
Using Eq.
end for
8: for i= L...,.., + do
Calculate weights
0i < 3 E
(jeargming 4r.)
Σ1
(jearg ming 4..))
10: end for
B Detailed Results
B.1 Detailed Experimental Setup
Search Space For the experiments, we use the same PPO scarch space as Parker-Holder al. (2020a) the only difference being that we fix the batch size to better control the memory usage. The batch size is fixed to 20.000 and 25.000 for classic control and Brax, respectively. The search space is shown in Table 1
19

---

Published in Transactions on Machine Learning Research (03/2025)
Table 1: Hyperparameter ranges PPO.
Hyperparameter
Range
Learning rate
(le-5, 1e-3)
Lambda
(0.9, 0.99)
Clip parameter
(0.1, 0.5)
[d
041
Normalized scare
Rank (out of 8)
random search
TAFP82
PB2
64
MctaP/iorP22
MulttaskPB2
P52* warm
Number of iterations
Number of iterations
Figure 7: Training all methods on classic control for 1,000,000 steps instead of 600,000 with a perturbation interval of 60,000 steps. Comparing rewardsnormalized at each iteration (higher better) on the left and ranks on the right (lower better). Using meta-learning later in training when there enough data to describe the target task is less beneficial. Still, MultiTaskPB22 and warmstarted PB2* remain the top- performing methods for training PPO on classic control.
Hyper-hyperparameeer settings The methods we propose have multiple hyper-lyperparametes whose values we set for the experiments. For the RGPE weight computation, we sampled 100 bootstraps, and for MultiTaskPB2, we used the weights to restrict ourselves the 4 most similar tasks. From each of these meta-task 100 datapoints to train the multi-task GP. During the portfolio construction, we generated the portfolio candidates via random scarch. On classic control, selected the best out of 30 random configurations, and on Brax, we selected the best out of 20. The performance is averaged over 10 seeds.
Environments We used CARL Benjamins et al. 2023 to vary the gravity of the classic control environments (mountain car, cart cart__oooe pole, pendulum, acrobot) and the Brax environments (humanoidd halfcheetah, hopper, humanoid standup, inverted double douull__peedeeeddeeeeeeeeeeeeeee _peddulme inverted peedduuuu_ pusher, reacher, and walker2d). Since CARL does allow instead altered the mass of the robots in such environments by multiplying with the relative gravity of the planets with respect to Earth. The different gravity values can be seen in Table 2 For classic control, we trained the agents on each environment for 600,000 steps with perturbation interval of 40,000. The agents on the Brax environments were each trained for 3,000,000 steps with a perturbation interval 200,000. We also looked at increasing the number of training steps and trained classic control for 1,000,000 steps in Figure Such an ablation was too expensive for Brax.
vary the gravity in all of the classic control environments,
Table 2:
Gravity variation for the RL environments. Values retrieved from https:/nsssdc.gst
nasa. gww/ppaaetarr/aacttheee///aa html
Planet
Uranus
Venus
Saturn
Earth
Neptune
Gravity
8.7
8.9
9.0
9.81
11.0
20

---

Published in Transactions on Machine Learning Research (03/2025)
6.5
0.49
0.55
0.50
Combined seeds
Combined seads
0.45
10
0.46
0.61
0.21
0.40
10
3.5
6.7
5.00
0.35
0.30
0.52
0.49
0.36
4.5
4,7
5.8
335
MetaPriorFB2
MuttiassPB2
TAFPB2
MetaPriorP22
MultTTskPP2 Algorithm
TAFP92
Algorithm
(a) Normalized performance
(b) Average rank
Figure 8: Performance at the end of training PPO on classic control for our meta-methods based on how many seeds were used to make up the meta-data. On the left normalized performance (higher is better) and on the right is rank (lower better).
B.2 Meta-data Requirements
In our experiments, we combined runs of PB2, using ten different seeds to create the meta-data. In this section, we aim to determine whether this amount of meta-data is sufficient and if there is any benefit to using more seeds. To explore this, we examine the performance TAFPB2, MetaPriorPB2, and MultiTaskPP2 when the mela-data consists of combinations of live, ten, and fifteen runs.
The results, presented in Table reveal no significant trend in the data. Ofall the methods, MultiTaskPPB2 using meta-data that combined the runs of 10 seeds, performed the best. However, it is possible that MultiTaskPP33 might benefit from more meta-data if we adjust its hyper-hyperparameter., especially the number of points that are input to the multi-task GP from each meta-task.
B.3 General vs Individual Portfolios
When analyzing the warmstarted and vanilla versions PB2* in Section 5.2 we observe that warmstarting does not significantly affect performance on Brax, but it did enhance initial performance on classic control. This difference may be due the broader range of environments that Brax encompasses, making harder for general portfolio optimize effectively. Using meta-features select an individual portfolio for each task is way to avoid this limitation. In this section, compare the performance of individual portfolios with our general foltoolio. We employ the same algorithm outlined in Section 4.4 to construct individual portfolios, but we limit the performance matrix to environments sharing the same base environment, thereby avoiding the use of meta-features.
Figure g shows the visualizations of the two portfolio types on Brax and classic control. Note that our general portfolio approach, we also have different portfolios for cach environment; these portfolios are not optimized for the given environment but are result of our leave-one-out strategy that excludes the that
performance of all configurations
the target environment. In both environment families, the general portfolios often do not contain the best configurations. Examples include pendulum on classic control and walker2d, hopper, and inwerted double pendulum on Brax. In contrast, the individual portfolios encompass most. of the top-performing configurations.
The impact of these differences portfolios on HPO performance is illustrated in Figure 10 We compare PB2* warmstarted with our individual portfolio to PB2* warmstarted with our general portfolio, PB2* and overall best approach Multi PB2. On classic control, using the individual portfolio results in
21

---

Published in Transactions on Machine Learning Research (03/2025)
1.0
1.0
Contpuraton
0.6
28
22
24
X
0.0
hopp salum
Hamma eart
humonne caturn
manoo vene
humeen in Satun
Intetsse gra pou seeete wete invette buiite
Tirdii diaily pordd votte
theeri somou nutoon
inve mndette pordil aifh vatter
plahee uranas
reacche mecuun
ool catumn
numone coanta
ca Dole oath acrdbo
car 200 regture
oi Lolk catum
cart D0 pe useniu or orupple vento
munain of earth
mOUTE saturn
pendumm astur
peddilum unmoo peodutu venus
dum
Envirnments
Ematronmants
(a) General portfolios on Brax
(b) General portfolios on classic control
0.5
0.8
10-
Confprrttio Cinfgrrttin
15-
25
0.4
20-
NomeRtd
-0.2
hale seree wert
lespur uu
nummnne ook
to ox in-
hume akande ti seture
Hummit ssntii west
proton potiol rolture
pow endtr pents un va peectee and pinndlli yorid
In mouuttu a-tuu sodiure invat ganto sakun
Inverta malai ye
posher acturre
pushur sutumn
nicher seaup
meat secure
naicht cotum
wes beez 2 setture
weteerdd ueou
no ko un
human gaant
reather
numttnd soii
bu
aumoen carth
ont pole nemune
cift 201 aruu-
catt jite tramaa
CILL Le ven
mourta Gud wathh
penduum masune
pandulum sarum proditum ueamaa
(c) Individual portfolios Brax
(d) Individual portfolios classic control
Figure 9: Portfolio performance matrix for Brax (left)) and classic control (right). Red crosses signal that the marked configuration is part of the portfolio for the environment. We observe that the portfolios do not contain the best configurations for multiple environments (e.g. pendulum on classic control and walker2d, inwerted double pendulum on Brax). This is remedied by constructing portfolios based on the performance of the configurations on the same base environment (bottom).
even greater initial performance boost compared to the general portfolio, this does not translate into better final performance. On Brax, warmstarting with the individual portfolio leads to a significant improvement initial performance that continues through the optimization process, achieving the best final performance among all approaches. These findings suggest that future studies should investigate the potential of using meta-features warmstart population-assed optimization methods.
22

---

Published in Transactions on Machine Learning Research (03/2025)
1.0
35
cd
PB22
PB2* warm general
NuttbakF822
Rank (out of
2.5
Comalize
08
2.0
U
P82* warm individual
Number teerations
Number iterations Brax
cd
3.5
Normalized Score
PE2
Rank lout of
Lande...
PB2* wam general
2.0
PB2 Individual
0.
15 10
20
Number iterations
Number iterations
Classic control
Figure 10: Performance comparison between using individual and general portfolios for warmstarting on Brax (top) and classic control (bottom). Warmstarting with individual portfolio is statistically significantly better than all other methods for anytime performance. On Brax individual portfolios are also the best final performers. On the other hand, on classic control, individual portfolios have worse final performance compared to a general portfolio.
B.4 Additional Similarity Weighting Results
In Section 5.3 we analyze the behavior of MultiTaskPB2 by examining the distribution of RGPE weights on Brax. Here, we provide complementary visualizations for classic control environments and for the MetaPri- orPB2 and TAFPB2 methods. The visualizations for MctaPriorPB2 on Brax are shown in Figure [1] and those for TAFPB2 are shown in Figure 12 The results for all three methods on classic control environments are presented in Figure 13
23

---

Published in Transactions on Machine Learning Research (03/2025)
10
balfcheetah earth
halfcheetah neptune
nalrcheetan Saturn
halfcheelah uranus
halfcheetah venus
hopper neptune
hopper earth
hopper uranus
hopper venus
humanoid earth
humanoid neptune
humanoid saturn
humanoid venus
humanoid standup earth
humanold standup nεptune
humanofd standup Satur
humanoid "standup uranus
Target task
in inverted double Fendulum
venus
Weighting
inverted double oendulum
inverted do pendulum saturn
-00
inverted double pendulum uranus
inverted double penduluu venus
inverted pendulum earth
inverted péndulum neptune
inverted pendulum letrted_eendlllill uranus
inverted pendulum venus
pusher earth
pusher neptune
pusher saturn
pusher venus
reacher earth
reacher neptune
reacher saturn
reacher uranus
venus
walker2d
walker24
wa 2d Saturn
walker2d uranus
walker2d venus
halfcheeta earth
helfe helfchhetat Reptune
ha halfchestal saturn
iificeeeahh uranus
halfe deticeetttr venus
hoppe ventahveeuus
hoppel saturn
hopper uranus
humarooo venus
numanoia sarth
heptune
humanold uranus
humano unaaoid venus
St Stskaup 6arth
mommann standup ne reptune comaaa
saturn
stand uranus inverted dduble double pendulic
bendulum unn earth
pendu rendulmm neptuse double pendulum Saturn Inverted 'duuble bendulleeeeullle pendul-m uranus douple pendulumm venus
Inverted penaum neptune inverted
dnvveredd peneeteddeeddddd pendullde ul saturn
inverted benduluum uranus
inverted pen um venus
repttun pusher
pusher earth
Saaghee saturn
pushee venus
pracher earth
reacher nerreeeee heetunee reacher e saturn
chaccedd uranus
earth
walker22 neptune
walked walkerdd saturn
walker aalkkrr2d uranus wa 2eee2d venus
hopper
reacher
start
huran oid
humanool
inverted
inverted
inverted
Meta-task
Figure 11: Average RGPE weights of MetaPriorPB2 for each task and all mela-lasks on Brax. The results are in line with MetaPriorPB2..
B.5 Ablation over the Number of Workers
In our other experiments, we consistently used four parallel workers. However, in this section, we expand our analysis to consider the use of eight and twelve workers. We compare our methods and the baselines for these different numbers of workers Figure 14
Our results show that MultiTaskPB2 performs well overall, maintaining strong performance across different worker counts. MetaPriorPB2 shows improvement and becomes comparable to MultiaakkPB2 the number of workers increases. Notably, when utilizing twelve workers, the warmstarting approach demonstrates the strongest performance.
24

---

Published in Transactions on Machine Learning Research (03/2025)
10
balfcheetah earth
halfcheetah neptune
nalrcheetan Saturn
halfcheelah uranus
halfcheetah venus
hopper neptune
hopper earth
hopper uranus
hopper venus
humanoid earth
humanoid neptune
humanoid saturn
humanoid venus
humanoid standup earth
humanold standup nεptune
humanofd standup Satur
humanoid "standup uranus
Target task
in inverted double Fendulum
venus
Weighting
inverted double oendulum
Inverted do pendulum saturn
-00
inverted double pendulum uranus
inverted double penduluu venus
inverted pendulum earth
inverted péndulum neptune
inverted pendulum Letrted_eeddllllll uranus
inverted pendulum venus
pusher earth
pusher neptune
pusher saturn
pusher venus
reacher earth
reacher neptune
reacher saturn
reacher uranus
venus
walker2d
walker2d
wa balker22 Saturn
walker2d uranus
walker2d venus
halfcheeta earth
helfe helfcheetea neptume
ha halfchestal saturn
malfcheetal heetehhhh uranus
/ ventu
cetan venus hoppan
hoppel saturn
hopper uranus
hopper venus
numanoid sarth
heptune
humanold uranus
humano sumanoid venus
huran nid St Strkaup 03rth
commann standup ne reptune commaao
saturn
stand uranus
pendulum neptune
pendu renduumm un earth double pendu
Saturn
Inverted 'duuble bendue bendul -endul-m uranus douple menduluu Inverted
venus
inverted periddeepel penaa neptune
Inverted pend ulm saturn
uranus
inverted pww.eenduulnnnnne um venus
reptuun pusher
siseer earth
pusher Saahhi saturn
pushee venus
pracher earth
heptune
Teacher Satur saturn
reacher uranus
earth
walkerr2a neptune
walket walkerdd saturn
walk 2orrrrd uranurrann uranus wa leeeed venus
reacher ne
hop hoper
humarolo
stant
dduble double pendulic
inverted Bendull
halfe
humanol
inverted
inverted
inverted
inverted
inverted
Meta-task
Figure 12: Average RGPE weights of TAFPB2 for each task and all meta-tasks on Brax. The results are in line with MetaPriorPB2. Note that the diagonal is not filled with nan's since TAFPB2 computes how well a GP trained on data from the target task fits the target task.
B.6 Crashed Runs on Brax
During our experiments, some training runs crashed on Brax due to divergence issues. The primary cause of these crashes is the choice of our hyperparameter ranges, which were designed for classic control by Parker-Holder et al. (2020a) and are less suitable for Brax.
Table presents a summary of the crashed runs. PB2 was the most affected, with 27 total crashes, while random search, MultiTaskPB2, and MetaPriorPB2 experienced the fewest crashes, with 3, 3, and crashes, respectively. The lower crash rate for random search is due to its use of constant hyperparameters. Multi-
25

---

Published in Transactions on Machine Learning Research (03/2025)
STONE
(a) MuluiaaskPB2
(b)
(c) TAFPB2
Figure 13: Average RGPE weights of TAFPB2, MetaPriorPB2,, and MultiTaskPB2 for each task and all meta-tasks classic control. Like for Brax, the weights form diagonal block structure. Bigger blocks be seen between acrobot and cort_pole and to degree between cart cart_pole and pendulum.
TaskPB2 and MetaPriorPB22s ability to avoid detrimental hyperparameter regions showcases their robust- ness, supporting the reliability of our approaches.
26

---

Published in Transactions on Machine Learning Research (03/2025)
2.-
PB2
Normalized score
Rank (out of 8)
random search
2.-
TAFPB2
Bi MetaPriorPB2
244
PB2*
MultTask222
22-
10
15
15
Number iterations
Number of iterations
(a) workers
0.6.
2.
Normalized score
5.8
Rank (out of 8)
random search
PBT
05-
844
MuP2
MetaPriorPa2
2.-
52-
03
16
15
10
15
Number iterations
Number iterations
(b) workers
6-
3.6
Normalized score
Rank (out of 8)
random search
P82 PB2_12
0.4
PB2*
MultiTaskB22
MctaPriorPB2
02-
014
10
10
Number iterations
Number iterations
(c) workers
Figure 14: Ablation over the number of workers on classic control. The top shows results for 4 workers, the middle for and the bottom for 12 workers. MultiTaskPB2 shows strong performance for all numbers of workers while warmstarting PB2* and MetaPriorPB2 have relative benefit in performance with increased number of workers.
27

---

Published in Transactions on Machine Learning Research (03/2025)
Table 3: Crashed runs algorithms
Algorithm Environment
MetaPriorPB2
MukiTTskPE2
PB2
PE22
PB2"
PET
TAFPB2
raadom search
halfeheetah earth
0
U
hallcheetah neptune
0
halfchoetah saturn
0
halfcheetah uramus
0
0
halfchettah venus
0
0
hopper earth
0
hopper neptune
0
0
hopper saturn
hopper uranus
hopper venus
hunanoid earth
humanoid neptune
humanoid saturn
humanoid urams
0
humanoid venus
0
humanoid standup earth
0
humanoid standup neptune
0
humanoid standuu saturn
0
humanoid standup uranus
0
humanoid standup venus
0
invertod_double__ntt dronble_pendulmm earth
coouble_pendulan
neptune
0
inverted double pendulum
inverted double pendulum
inverted_dde eveeted_ddobbee_pg domble_pendulung
lnverted pendulum earth
0
inverted _enduluu_enm neptune
0
inverted ppentee_peeuuuum saturn
0
inverted pendulum uranss
0
inverted
inverted_penduurre pendulum venns
0
2
2
pusher earth
0
pusher neptume
0
pusher saturn
0
pusher uranus
pusher venus
reacher earth
reacher neptume
reacher saturn
rescher uranus
0
reacher vemus
0
walker2d earth
0
walker2d neptume
0
0
walker2d saturn
0
walker2d uramus
0
walker2d venus
Sum
27
9
3
B.7 Raw Results
In this section, we display the raw rewards at the end of training for all environments and methods used in our main mothod comparison (Section 5.2). The rewards for classic control environments are detailed Table and those for Brax environments listed in Table
28

---

Published in Transactions on Machine Learning Research (03/2025)
Table 4: Average reward and standard error at the end of training on classic control.
algorithm datases
MetaPrionPP2 MultiTasPP22
PD2
PD2
PD2 worm
PBT
TAFP12
tandom search
acrobot earth
-79.40+1.288
-79.90+2.27
-8011+1161
-78.18+1.04
-70.8++1.27
-76.255+-065
-80.95+2.44 49
-81.83+3-13
acrohot neptune
-89.46:11.00
-96.80+1.06
-88.19 1.28
1.3.50
.88.07=1.50
-85.2141.00
-87.1711.17
-88.294 29..2942.05
acrobot saturn
1.77
-718551177
-74.0111.43
-74.8112.56
-74.2111.04
-T2.96:11.19
174.1910.98
-73. -73.1611.95
acrohot uranss
-76.71+5.55
-71.2842.66
-68.76 1.90
-7,.41+2.25
-68.75.11.50
.69.90 00.99011.72
-70.33+1.4 45
-71.76+1.56
acrobot veaus
.3..9323.22
172.731199
-74.31
-77.18
.72.8911.66
-75.244:2 11
71..1111.05
porl_ earth
500.00:00000
300.00:0.000
500.00 000.00120000 00
500 000.0010:000
500.0020.000
500 00000000.00
500,0000.00
000.00000000
cart. pole neptuae
300.00.0.00
5.0.0000.00
500.00.0.00
500.0010.000
500.0010.00
500.00.0.00
500,0000.00
500.00
pole sateru
500.00:0.000
500,00:0.00
199, 00048220.52 52
500.00000000
500.00000000
500,0000.00
500,0000.00
500
port_pole unams
300.00+0.00
500.0000.00
500000+0000
500.00+0.00
300.00+0.00
500.0000.00
50000000.00
500-00-0.00
pole
500.00:0.000
500,0000.00
500.0020.000
500.0020.000
500.0000.000
500.00:0.00
500,00:0.00
500.0000.00
mountsin__er par earth
-131126+12.16
184113712.37
-139.777418.22 24
11.5005+33.33
-100048-+11544
-187-77+11.7 .70
-146-80+12.3 32
-145 146-30+9.00
mountain car neptune
-144.81=10.10
10.11
-144.00#10.4
-140,25=10.44
-114.13:+2.11
-149.98+ 10.50
-146.49#9.64
-152.21+9.77
mountain par saturn
-131.11=12.31
-131 02+12.67
-141,82+10.99
-141.22+11.77
-98.10++1144
-141.14+70.64
-140-23+12.01
-138 30++ 13.30
mountain, czr urams
-127.83==2.444
-133. 7281192 80
1135,94= 16
-132.93+1222 26
-98.51
00..1++-2.97 2.997
-137 0077088 12.46
-134.29+ 12.14
-129.
13+ 13.12
mountain oar venus
13.005112.37
-133.00 12.400
131665112
155.30
.95.05
-139.8 1.0.811:1110
-140
-10.
pendulum earth
242.98==1.50
-215.39:-33.16
1216,33+13.00
120.577=1..37
$238.17=288. 75
-238 10..11+#11.0 .94
-268. 12++-22.99 92
-220.73+ 21
pendulum aeptune
256.10:299.66
220.74 12.2001118.21
-211-48128.77
85
-295.98
220.733.26.21
pendulum saturn
227.011:26.29
-213.02=21.14
-116.83113.00 00
-219,51=19.46 46
-238
1-:228 78
28:2211125.. 10
-286.01
286.011151.99 95
-20 0000.73:26.22. 21
pendulum uramus
236.29-20.66
22.7.74220.71
-24442211495
118.488:19.54 59.54
1288177
252 24..499
-27
220. 220.731.26.2
pendulum venus
181.11
-212.04=21.48
-216.83213.00
188.18:19.54
-238
083.088287777 79
12:6911:9.44 13
280.7411-9.11 15
-220 220.73126.21 26.21
Table 5: Average reward and standard error at the end of training on Brax.
alpetthi
Pizz"
PITT
TAFPII
fotdlo seach
disowut
Inlktrettk eorth
1417.00:554.0
LEB 7T GL
13006F9:66..7 .7
1206-5
1201 17:664.7
1256 77:500 75
1198 SUS-5.277
balkhertok teptune
1447.00 74T.599:77.66 65
145 035.66663.88 55
1:10 35 71
1401 05--0155
1277 20-3200-55 .7
1225 5..-5-5-551 14
1406 02:35 07
1255. 203-20---.0000 302
halkhertsh sarrn
1351 J88 106.11
1510 730-184133 W5
1998 033866--4077 0T
756105-100955 10.90
1100 050-05555553
1533 104-2 509
1952 85- 120 70
1185 38--57.01
halkchettk TEar
1211 109 14
1212.56199-4
1238 33 12- 13
11501.33 103.00
1161 52:19939
1262. 79 120.31
1198 47 111 37
1973. 16- -51.02
hallchetth wane
1320 26 151055
1513.721 222-45550 91
1225. 72. 10100000
225-17.000 10..000 85
1150 17119985
1281: 121.933000 76
1352- 162 50.000 10.05
1110. (110.-5A.1 15.770
hopese earth
050.21-220
72805511 19:00
B85 00.34411
652 03:221 223
690 07 28 01
752 252.8-4:233222 23:222222222 22
F1LAR 22. 77
500 000.35-..44.40
hopese peptuns
d50 18:225
680-45:222.00 20
B75 (012::
0F1 1532
455 02: 622
724. 19:3 305
Di:: 6.06
DRS 4.2.1 10-52
loyere astura
745, 745.900:332.30
710004:22222 27
723.022155000
718.51 2:222222000 47
702 20:200 30
411-0.2-0000 00
20
T2R 228.005330.55
loyort uraus
735 (55-91:200 (12
724002255.. 18
702. 17
702. 34.002
07T 64
80
hopper verna
738
715
710
Inmestdd sorth
ST9 200.611
418
Iomannd nrpenne
368
361
50
Iomaned katznn
410
601100.00
431
120
402 10.00
40S 51.00.0
04
412
27
12.751=10.00
2624 70
74
26
1807) 200730092258000 06
160 99
15144
264-302
10840
27
24002 37:
steadiy aeptune
147500004223100
15509.00 100.33
10266
15490.776-110
155022
34753 OZ:
Inmotoi
stendip
15009:964-144.0 35
15U25
10836.014
10-
stendop
264-344
15071 12
15477
dabble perdulum
285.0
danble
reptine
2051.25:8111
Sauble pershila
2576.911:01
Sauble pezzhilu
aesctol Banbl pezaiiaa
227 42
lesclol
carih
lecclol pesdulin
542.74556..
THE
27:344
tecclol pendulim
45
tevetel pendutun
437
56
30850
foveted peodutmm
550 08-00-3-0000
15,
18
praber cazta
-1512 3542778..
1112.88:22244 24
JUT2 90: 101 52
.1744 1.3.335411.666 16
201 ait 155 10
.522 08 121
-800 27 175.
prilet oestmm
-1090 77
1842 2012 00.33
JO5S 70 105
.442 16:
.1441
441.9655443.00 56
JSA 11 145.01
priler slture
-1361 241.24422155 8.50
-78 08:78.077 07
1142 24:2222222222 4T
10G4 19:1 17 18
2702 20: 10
.551 72 101 44
1500
pislet unaus
-1004 3100154451455
1022 270 45
11
.85 22 40
- 227 075.9951155 30
.953 11 157 32
122
OI 200 12
prster wents
-1165 77:100888 59
1001 11391
1T7. 1775055550 52
-200 51: 126
000. 720 161.02
-11507 11-20000000 30
1554
20
reecher oarth
-157 30
-197 0518-55 57
.T6 19-55.0 50
110 75-7.77
-113 51-9 80
1377 115 19911000
-125 30-155
40 AI $.11
wocher repiine
-120 37- 5.00
-117221110.5
.197 19-11 1135
-117 15- 10.00
-97679- 6.00
-12 09 11 10
775.22--4 4-227
nocher sotum
-135 01 1.87
-133.5117.37
-138 13 4.37
-150 16.13
-150 5.79
-110 1651
wocher srans
-128 30.19:2
-111.46111.992
-137.90.122 22.00
-125 1005
-100
100 100.000000
-122
47.555
aucher veuse
-128 22222222200 2.22
-135.55177.0 59
-135.24:7711 100
-120 21.00
-122 12.27
-12 00.00011000
-12% 01.0.00
-786-922:4440
vilkersi woth
535 535.53::36.11 11
86408-291.36
CF 20F:01123..2 10
550 54:000
472 20:21 10
582 12351 /6
584 404.422-41.75
470 470.56-55.00
valler1 neptume
477.85:14.71
40.07:17.55
ui2 00222222...11 10
458.00 068.012:6.11 19
440 00031:51111
502 0022022200.14 18
di0. 7.127711 10
vallerai setien
344. 2411444
1.0:21:110:0
501101122 52
0180000 1444
500 21:31:111 125
ton 201.252
578, (78.67257.11 118
490 200.1 122
valler2 unnus
608. 008.93240.200
112.404657.5 29
til 00111.111.
070 88 255
47% 375.61:22200 07
tun 300.ii220..0 55
Geb t0 t6.
walkerhl venus
330 27.54
558. 0555044:33.66
515:006530.71
01..5 025.84128813 18
47t, 700770:30100 700
543.00631..0 30
B.8 Individual Training Plots
In this section, we display the individual training plots for all base environments with the default earth gravity. We show the mean reward and standard error. Table 16 contains the training plots for Brax, and Table contaiis the plots for classic control.
29

---

Published in Transactions on Machine Learning Research (03/2025)
139
1000
422-
6.0
430
Karfbudd
082
For
bon
191
Tea Kamss
(a) halfcheetah
(b) hopper
(c) humanoid
Score ower Dilased hurmeress stareuu earh
Sonn over Dalaner, insetted deste c.sertuure sarts
over Dutaed twerieeeeentturee
3093
3889
330
24.00
793-
2000
32090
991-
700
1109
(d) humanoid standup
(e) inverted double pendulum
(f) inverted pendulum
Score over Time on Daramet pubter sarth
over Time on Docamst mecher
over Outarok watourze
F$27
-222
-1290
-100
537-
-100
-1483
Nat 32
-1000
-500
150
-00
332
-229
32
0
12
10
32
(x) pusher
(h) reacher
walker2d
Figure 15: Training plots with earth gravity (default value) for all Brax environments. MultiTaskPB2 is consistently one of the top-performing methods, showing its robustness. It is notable that warmstarted PB2* is the best-performing method on some environments (e.g. humanoid and inverted pendulum) while being the worst performing on others (e.g. reacher and 2alker2d).
30

---

Published in Transactions on Machine Learning Research (03/2025)
Score over Time on Dataset acrobol earth
Score over Time on Dataset cart_polf _apleeeeeeeeoo earth
200
Reward
250
MctaPriorPB2
MutllasP922
MultIaasPB2
-350
PB2* warm
P02* warm
TAFP82 PB2
TAFP82
4000
random search
random search
10
14
10
14
Steps
Steps
(a) acrobot
(b) cart pole
on Dataset mountain
pendulum earth
MetaPriorPB2
P62
-120
PB2
PBT
Reward
-140
random
Reward
-000-
MetaPrionPU
MultmaskPe2
800
PS2
-1000
TAFP82
-200
10
10
Time Steps
Time Steps
mountain car
(d) pendulum
Figure 16: Training plots with earth gravity (default value) for all classic control environments. Multi- TaskPB2 is consistently of the top-performing methods, showing its robustness. Warmstarted PB2*, the other hand, performs inconsistently. In mountain car, it performs exceptionally well, while it performs poorly on acrobot.
31


Reviews:
[{"review_id": "EYF334Us2v", "reviewer": "Reviewer_bF8L", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper studies an interesting problem of incorporating prior information into population based bandit problems. The paper also offers a few rather intuitively sensible methods. The technical contributions are of interest to the general ML community and especially interesting for population based training practitioners.\n\nThe main weakness might lie in that it's a bit hard to tell apart the relative strengths of the proposed four methods. The paper rather feels like a bag of tricks that can accelerate population based training with hyper-parameter optimization. The connection between different methods feels rather less coherent, and the paper feels like just enumerating different possibilities without stressing enough on the common insights.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a few ways to incorporate prior information into population based training or population based bandits, where hyper-parameters of the training system are optimized over the course of training. Compared to baseline methods, they propose ways to incorporate prior information that helps avoid the initial warmup slowness and accelerate learning. They show improvements in a number of benchmarks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# ==== figure 2 results ====\n\nIn Figure 2, the baseline methods (random and PB2) do not seem to have significant score improvements over the course of training. In this case, the random search does not seem to improve over iterations at all while the PB2 baseline seems to improve a little bit during the whole process, with an initial dive in performance. How should we interpret the dive in performance at the beginning of PB2, is this related to the warm-up motivation?\n\nIf the random parameter search does not improve the new iterates' performance, it should still improve the cumulative performance, that is the best hyper-parameter config located by different algorithms thus far in the iterations. Does that sound reasonable?\n\nThough the paper claims that MultiTaskPB2 performs the best, it is still not very clear if it is indeed the statistically significantly the best approach among the four proposed new methods. The error bars from different techniques rather overlap with each other quite a lot, making the statement less clear.\n\n# === Fig 14 ===\n\nIt feels like Fig 14 paints a more comprehensive comparison across different methods, where it seems that even random search obtains reasonable performance in a few environments such as pusher and reacher, where random search performs the best. What is the special property of such environment that makes random search stand out in such a case, where deliberate hyper-parameter search techniques do not well as well?\n\n# ==== nit comments ====\n\nFigure 6 runtime comparison can be more meaningful if put in relative values rather than absolute values.\n\nAcross Fig 4-5 it feels like the ranking curves are a bit redundant given the normalized scores, and the results might be more presented using tables rather than plots.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "W4EnMO3vrH", "reviewer": "Reviewer_6hor", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- Hyperparameter optimization algorithms for RL is an important research question and can improve practical performance of RL algorithms. \n- In this paper, authors have clearly described the prior work and how they extend prior frameworks by addressing some of their limitations. \n- Ablation experiments have been performed to validate the different components of the proposed algorithms, eg. the weighting mechanism used to combine different meta-tasks, or probability of configuration change at perturbation intervals. \n\n\nWeaknesses: \n\n- The success of the methods heavily relies on the availability of relevant and well-aligned tasks which can be used for meta-learning. Although the RGPE heat maps seem intuitive, no quantitative metric is provided for task-similarity correctness. \n\n- The overhead introduced by MultiTaskPB2 is significant for smaller environments. The paper minimizes this issue for complex tasks, but practical implications for researchers with limited computational resources should be highlighted more explicitly.\n\n- There is substantial variance in normalized score and rank over iterations, particularly in the classic control environments in Fig 2. This is also true in Fig 4 and Fig 14 for Brax. This makes it difficult to agree to the paper's claims about superior performance of the proposed algorithms.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose four meta-learning methods tailored for Population-Based Bandits (PB2) to dynamically optimize hyper parameters in RL algorithms. PB2 uses time-varying Gaussian Processes (GPs) and Bayesian Optimization (BO) to make explore-exploit decisions in the hyper parameter search space. But the GP initially lacks sufficient information to make good hyper parameter choices especially in the initial training stages with small number of examples. Instead, the algorithms proposed in this paper introduce different ways to leverage meta-learning over different RL tasks to address this drawback of PB2. \n- **TAFPB2**: Enhances the BO acquisition function using meta-learning.\n- **MultiTaskPB2**: Employs multi-task Gaussian processes for improved surrogate modeling.\n- **MetaPriorPB2**: Constructs a GP prior using meta-learned ensemble.\n- **Warmstarting**: Leverages static hyperparameter portfolios to accelerate optimization.\n\nExperiments demonstrate that MultiTaskPB2 outperforms PB2 and other baselines in both anytime and final performance across two RL environment families, classic control and Brax.\nThe code implementation is shared for reproducibility.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weaknesses section.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "f3mNz7tTnS", "reviewer": "Reviewer_xanU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Proposed methods demonstrate strong results vs baselines on experimented domains (under certain computational contrains)\n- Clear approaches of playing with various levers of BO/GPs to enhance PB2\n- Good acknowledgement and awareness of limitations (meta-data alignment and continuous hyperparams, computationally expensive vs baselines)\n- Provided open-source code to reproduce experiments\n\nQuestions/Improvement Points:\n- The contributions section in the introduction should be more detailed and clarified, and summarise your findings/results. For example:  --> “We investigate potential synergies of combining our methods”. What are the synergies? What was the summary/result of this? Good/bad/no-use?\n--> Similarly “We investigate the behaviour ……”. What behaviour? Ablation for hyperparamter? So is it sensitive/not sensitive etc.? \n- Could you expand why “BO naturally lends itself to meta-learning” in the related work section?\n- Why only 15 iterations of the algorithm? Both the baseline algorithms still seem to be improving/have not converged. If there is a particular reason, you should explain why just 15 iterations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes improving population based learning using meta-learning methods. In particular, it builds off PB2, also a meta-learning algorithms for Population based training that uses Gaussian Processes. The authors propose 4 enhancements based on using different elements/improvements of GPs. Critically it utilises some meta-data from various environments to help with improving the efficiency.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "No major changes but some clarifications as commented above in the “Questions/Improvement points”.", "broader_impact_concerns": "No major concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were mostly positive about the paper in their initial reviews and they acknowledged the rebuttal has resolved most of the raised concerns. Thus, I am proposing to accept the paper as is. However, if the authors are interested in improving their manuscript for the camera ready version, reviewers' suggestions include:
- Providing "a more convincing argument in support of the variance in the plots which makes it hard to distinguish among the different proposed methods";
- "The empirical setups can be made more solid, such as the ablation against why random search turns out to well perform in certain cases, and justifies whether more sophisticated approaches are needed at all".

Although not all of the reviewers provided a positive final recommendation, I think this can be a valuable contribution to the TMLR community. A wide adoption of the proposed method may instead be hindered by a significant computational burden to integrate it, which the authors acknowledged in their conclusions.
Decision: Accept (poster)